1. Round-robin (the default) – This algorithm selects first one app server; then the next; then the next; until it loops back around. Since the actual marker for what is “the next” is kept in the proxy for the PowerPivot service application, the net-effect of this in practice is that the selection looks different from what you would expect. The behavior is a lot closer to random rather than sequential. Being random, and with a low number of servers to pick from (for example, 2), you should naturally expect that one app server might seem to be biased. Add more servers and you will see less bias.
2. Health-based – This is the one that most large shops will likely choose. The idea behind health-based is that the system will decide which is the ‘best’ app server for a machine. So when the allocation appears bias, it seems like health-based isn’t working. In reality, health-based may be doing exactly what was intended. If all machines have memory available (i.e. none of them are under memory pressure), then health-based uses CPU to break any ties, i.e. which ever CPU has the most CPU free wins. Let’s take an example:Suppose you have two machines “A” and “B”. Both are running PowerPivot. They both have 32GB on them with four quad-core processors. Rather than being dedicated to PowerPivot, “A” also doubles as the backend app server for Excel Services. Likewise, “B” doubles as the backend app server for PerformancePoint Services. So long as Excel Services and PerformancePoint consume similar CPU time, then databases will be loaded back and forth between “A” and “B” as one is more lightly loaded than the other. However, if PerformancePoint is lightly used and “B” is consistently less loaded (CPU-wise), then you will see PowerPivot databases being allocated to “B” until it becomes under memory pressure, and then allocation will shift to “A”.

All-in-all, health-based is still the best algorithm for large shops. Remember that the PowerPivot engine is an in-memory system. Our first goal is to get databases allocated wherever memory is available – balancing across the farm is not a priority.

Enjoy.

Make sure to run this executable as the service account under which Excel Calculation Service is running. If you are not sure what account that is, first go to Central Admin’s “Security” page:

Then under “General Security” chose “Configure Service Accounts”:

In the drop down list on the right side you are looking for a “Service Application Pool” which contains your “Excel Services Application Web Service Application” (in the middle list). The account at the bottom would be the account you want to make sure you run the test application as to accurately simulate what ECS is doing when you are actually using SharePoint. Using the wrong account might give misleading results.

You must provide the User Principle Name (UPN) of the interactive user to the test application. All users have an implicit UPN which can be expressed as <user>@<domain> (I would be “leegr@redmond”). You may also have been given an explicit UPN which might look slightly different (although the implicit UPN would still work). If you are concerned that you are not using the right UPN, you can dig through the ULS log to find the UPN associated with the failure. This is the log entry that I got when I turned the c2wts service off:

SPSecurityContext.WindowsIdentity: Could not retrieve a valid windows identity for NTName=’REDMOND\leegr’, UPN=’leegr@microsoft.com’. UPN is required when Kerberos constrained delegation is used.

Note that the bit about “Kerberos” can be ignored because we do not require Kerberos constrained delegation to work (and neither does c2wts … it just returns a limited token in this case which is fine for us). Also, your log entry might look a bit different since I don’t know if they output different things to the log based on the exception type at this level.

To compile this code, you need to link to:

• Microsoft.IdentityModel
• System
• System.Core
• System.IdentityModel
• System.ServiceModel

using System;
using System.Security.Principal;
using System.ServiceModel;
using System.ServiceModel.Security;
using Microsoft.IdentityModel.WindowsTokenService;

namespace C2WTSTest
{
    class Program
    {
        static void OutputUsage()
        {
            Console.WriteLine(“Usage:”);
            Console.WriteLine(“/tc2wtstest.exe <upn>”);
            Console.WriteLine(“/tExample: c2wtstest.exe dwickert@redmond”);
        }

        static void Main(string[] args)
        {
            if ((args.Length != 1) || (string.Compare(args[0], “/?”) == 0) || (string.Compare(args[0], “-?”) == 0) || (string.Compare(args[0], “?”) == 0))
            {
                OutputUsage();
                return;
            }

            string upn = args[0];

            WindowsIdentity windowsIdentity = null;
            if (!String.IsNullOrEmpty(upn))
            {
                try
                {
                    Console.WriteLine(“Attempting to acquire windows identity for upn: ‘{0}’”, upn);
                    windowsIdentity = S4UClient.UpnLogon(upn);
                }
                catch (SecurityAccessDeniedException)
                {
                    Console.WriteLine(“Could not map the upn claim to a valid windows identity. Security Access Denied”);
                    return;
                }
                catch (EndpointNotFoundException)
                {
                    Console.WriteLine(“Could not map the upn claim to a valid windows identity because the c2wts service was unavailable”);
                    return;
                }
                catch (FaultException e)
                {
                    Console.WriteLine(“Could not map the upn claim to a valid windows identity because the c2wts service returned a fault”);
                    Console.WriteLine(e.ToString());
                    return;
                }
                catch (Exception e)
                {
                    Console.WriteLine(“Could not map the upn claim to a valid windows identity because of an unexpected exception”);
                    Console.WriteLine(e.ToString());
                    return;
                }
            }
            else
            {
                throw new Exception(“No UPN claim found”);
            }

            using (WindowsImpersonationContext ctxt = windowsIdentity.Impersonate())
            {
                Console.WriteLine(“Successfully acquired token and impersonated user: ‘{0}’”, WindowsIdentity.GetCurrent().Name);
            }

        }
    }
}

HTH
Lee

PowerPivot provides on-demand loading of embedded databases. What this means is that when a client application makes a connection to a workbook (by placing “server=http://sharepoint_server/site/subsite/doclib/workbook.xlsx;” or “data source=http://…/workbook.xlsx;” on the connection string), then the system looks to see if the workbook’s embedded data has already been loaded. If so, the connection request is routed to that PowerPivot app server. If the embedded data has not yet be located then the system has to go through what we call Allocation. Allocation is the process of deciding which app server to use to load the data and to load it into that machine’s AS Engine instance (we only allow one AS instance per machine – called <server>\GeminiBI in CTP3 and <server>\PowerPivot in RTM).

PowerPivot ships with two algorithms:

1. Round-robin – where the system just picks the next server in the PowerPivot System Service instance list (using the SP object model – remember PowerPivot is a shared service in SharePoint).
2. Health-based – where the system polls all of the PowerPivot System Service instances to determine their health and uses the healthiest server as the location for the ‘to-be-loaded database’.

Notice that round-robin is the default. Round robin is particular good if need to minimize the number of messages flowing through your system as it does not poll any of the servers – it just picks the next one and uses it. Since it does not check to see if that system is the ‘best’ from a memory or CPU perspective, round robin is best used when there is no memory pressure on the servers. It is NOT a good algorithm under heavy load. The heavy load use case was the design point for health-based allocation.

Health-based uses the current CPU and memory usage of the AS instance on each machine to determine the healthiest. But for our purposes the other thing that it does, if it cannot find a server that has memory free (CPU is only used as a tie-breaker), then health-based looks to see if there are any inactive databases that can be unloaded to make memory available for the target database (the embedded data in the workbook). An ‘inactive’ database is those that have no currently opened connections to it. Health-based scans the list of inactive databases in oldest-to-latest last accessed order; until the target amount of memory (1.5X the size of the workbook) is found. Thus multiple databases may be unloaded. The PSS sends XMLA DETACH commands to each database, it loads the target workbook, and then the connection request is forwarded to the right server.

So . . that is what happens. But who is actually doing this work? That is the “geek’ie” part of this post. Rather than using the PSS and having to create a “master” PSS (which would give us a single failure point), we elected to do this work in a distributed manner. Here are the components:

The allocation algorithms run in the WCF proxy code. When a request comes in (a remote procedure call to a method in the PSS), the proxy looks to see if it knows where the target database is located. If the proxy has seen the database before, then it forwards the request to the PSS for that respective endpoint. If the proxy has not yet seen the target database it picks one of the PSS web services at random and invokes a method on it to see if the database is located in the PowerPivot service application database (the so-called instance map, which is a record of what databases are loaded and cached (detached) on each AS engine. There is a 10-minute background thread in the PSS that keeps the local AS engine in-synch with the instance map. The randomly chosen PSS returns the instance map information. If the target database is in the instance map then the proxy forwards the request to that PSS endpoint. And everything is good.

If the target database is not in the instance map information, then the proxy looks to see what allocation algorithm to use. If it is round robin, then the proxy just picks the next server in the PSS instance list. Since there could be many PSS proxies in the farm (e.g. many WFEs, many web applications that might have different IIS application pools, ECS transports, and other backend services that access PowerPivot via msolap, ADOMD.NET, AMO which in turn use the Channel Transport), the actual allocation of servers is likely to appear random or heavily favoring one server. We purposely did not synchronize the picking of the ‘next’ server because that would have placed a hot spot on that particular resource. Round-robin is supposed to be a fast, minimal algorithm – so having it become random over time is considered OK.

If the allocation algorithm is health-based, then the proxy polls each PSS gathering information about CPU, memory and the status and size of memory taken by each database on the servers. The proxy then decides which is the ‘best’ server and what databases to unload (detach) to be able to load the database on that server. The proxy then calls the server to perform the load and forwards the request to that particular machine’s PSS endpoint. 

At any point, in the entire algorithm the proxy is prepared to handle that the ultimate target endpoint that it is given, for example, from its own cache, or from the instance map information, might be wrong and the RPC call may return that the database is not located where the proxy believes it to be. If this happens, the proxy reties. It keeps retrying the number of PSS instances are on the farm. If the proxy is still chasing its tail through all of the pointers and caches in the system, then it gives up and returns an “unable to load database” error to the caller.

So, in summary, i wanted to go over this not to explain allocation itself, but more to make the point that rather than say one simple thing — “When in doubt, turn on health-based allocation – and let it proactively unload databases.” While not directly tied to inactivity, it is the single biggest ‘tuning’ option that you have to clearing resources in a high memory use configuration.

Enjoy.

Here is a quick list of most likely causes for this error and how to debug and fix them (in order based on my experience of likelihood to be the root cause):

• Is the Claims to Windows Token Service started on the server running Excel Calculation Service (ECS)? For detailed information on the Claims to Windows Token Service (c2wts), you can read the msdn article. This service is turned on when doing a “New Farm” installation of PowerPivot, but if you do an “Existing Farm” installation, or have Excel Calculation Service (ECS) running on a different machine, the Claims to Windows Token Service might not be started. Check to make sure that this service is running on every server on which ECS is running. You can validate this via the “Services on Server” option in SharePoint’s Central Administration web site but it is also important to make sure that this is running from Service Control Manager (SCM accessed via services.msc). There is a known issue after reboot where the c2wts fails to start because of an unexpressed startup dependency on the crypto service. I will add a link to the KB when it is available but until then you can add the dependency manually from SCM or from an administrative command prompt with “sc.exe config c2wts depend= cryptsvc”.This will prevent the problem from reappearing after your next reboot. As per http://powerpivotgeek.com/2010/01/18/why-you-shouldnt-stop-start-analysis-services-from-scm-when-running-in-sharepoint-integration-mode/, you should not manage SharePoint services from SCM, however SCM is the truth when it comes to whether a service is running and so you should always double check by looking at the state in SCM. If SharePoint indicates that the service is started but it is not actually running per SCM, it is safe to start it from SCM (alternatively you could stop and start it from SharePoint Central Administration). As well as simply managing this service, SharePoint configures the security permissions for this service automatically as part of their setup so that all SharePoint Shared Services (which includes ECS) can use it. If you find that the service is stopped on the machine running ECS, start it. After starting this service, you should not need to do any type of  IisReset to see the system start working.
• Is your machine connected to the network? Dave has written a good blog on this also (http://powerpivotgeek.com/2009/11/06/taking-your-server-off-the-network/). If you are actually trying to run a PowerPivot demo with a machine which is not on the network, follow the steps in Dave’s blog to configure ECS to use a set of stored credentials for the PowerPivot datasource. If you are not doing this on purpose, then connect back to the network.
• The final cause would be that for some reason other than network connectivity, the Claims to Windows Token Service is not able to convert the SAML claims token to a Windows User security token. Dave provides a lot of details on these potential issues. Some quick questions to ask yourself:
  • Is the client user account (the logged in user who is browsing the workbook in IE) a domain account? If the account is a local machine account, then the Claims to Windows Token service will not be able to retrieve a Windows user security token. We do not support this scenario in V1 of PowerPivot. Interactive users must be domain users. For demo purposes in a bind you might try the workaround Dave provided for taking the server off of the network, but I have not personally tested it in this case.
  • Is the client user account in a different domain than the SharePoint servers? This is completely supported, but there must be a trust relationship established between the two domains. You could verify if a missing trust issue is causing your problems by logging in as a user account in the same domain as the SharePoint servers and try interacting with the workbooks (note that you had to have given that user access to the workbook). If it works for users in the same domain but not for users in other domains, it might be an issue with cross domain trust. Contact your domain admin to figure out what the relationships are setup as.
  • What account is the Claims to Windows Token Service running as? By default it is configured to run as Local System, and I am not aware of the reasons for changing this configuration (the msdn article also refers to the fact that it should be running as Local System). While there might be a good reason for trying to change it, it is possible that the person who altered it did not understand the implications of this change. You should probably track down the person who changed it and get an understanding of why. If you have permissions, switch it back to Local System and try the scenario again. If it works, you will need to determine why it was changed in the first place.
  • If you have gotten this far and none of the above have solved your issue, then there is the possibility that you have some custom AD configuration which is causing the issue. Dave points out one possibility:
    

    The account being used as the Excel Services service account must have AD rights to be able to query the object. One place where we know this restriction comes into play is if you have configured your domain controller to have a subgroup under “Users”, e.g. “Service Accounts”, which is a separate AD group that derives from “Users” –> but I am sure that there are more. AD rights for service accounts is a common problem across all of SharePoint.

    What Dave describes is one possibility where an AD configuration could cause this issue. As we discover more potential AD configurations that could cause this issue, we will try to update this list. If you are comfortable building your own test application and have gotten this far without figuring it out (and feel very confident that it is not #1), you can try running the test application we have posted here to manually test your ability to acquire a Windows Identity.

Hopefully this list will help you quickly debug and fix this issue on your system. HTH

Lee