First, why is this a problem? After all, as you can see in [...]]]> I am inspired by a recent post from a colleague about the various issues that can come up with Excel Services delegation (see a Denny Lee’s blog here: http://dennyglee.com/2009/11/18/troubleshooting-powerpivot-excel-services-connectivity/) and I wanted to take it a bit further (and maybe a bit ‘geekie’-er)

First, why is this a problem? After all, as you can see in Denny’s post, you can see the workbook and you even have a thumbnail for it in the Gallery. What’s up here? The core of the problem is that unless you’ve set the connection to refresh when you first open the workbook, Excel Services uses its pivot cache to construct the pivot table and slicers. It is only if you manually refresh the connection, or click on a slicer, that you make an actual connection to the embedded data. Until then you are just looking at cached information. Until you click on a slicer, you don’t really know if Excel Services is working – so a strong recommendation that I make to any person doing a validating their installation is to “ALWAYS CLICK ON A SLICER” if you want to make sure that your installation is working properly.

Ok, so now we’ve hit the problem. And you get the dredged “An error occurred during an attempt to establish a connection to the external data source.

The issue is (get your geek-armor ready) is that when accessing data PowerPivot looks like just another data source to both Excel desktop and Excel Services. Prior to accessing the data, if using Windows authentication, Excel Services needs to impersonate the user on the calling thread. But, in a claims-aware world, the only ‘identity’ that Excel Services has is the claims token. When the user connects (in whatever authentication method the SharePoint Web Application allows), the first thing that SharePoint does on the web front end is to translate the authentication method’s user identity to a claims token. And it is that claims token that is passed around within the farm. Remember this dialog box within Excel desktop:

The setting tells Excel Services what kind of lookup to perform when a new connection is started:

• ‘None’ means to use the Unattended Execution Account specified for the Excel Services service application being used by this connection. The username and password are retrieved from Secure Store. Using these credentials, Excel Services does a Windows logon and then it calls the data provider (the msolap OLEDB provider in the case of PowerPivot).
• ‘SSS’ means that Excel Services should access its Secure Store service application. The username and password are retrieved from Secure Store using this Application ID. Using these credentials, Excel Services does a Windows logon and then it calls the data provider Using and Excel Services does a Windows logon and then calls the data provider.
• ‘Windows Authentication’ (which is both the default and our case today) means that Excel Services should use the interactive user’s Windows identity. The original Windows token for the user is looked up, Excel Services impersonates that user on the calling thread and then calls the data provider.

In the case of Windows Authentication, to perform the lookup, Excel Services uses the “Geneva to Windows Token Service” (GTS) provided by SharePoint. GTS takes the claims token of the caller and translates it to the Windows identity of the caller (the underlying Windows API that is uses for this is S4U (see here: xx). Unfortunately S4U does have its restrictions, and those restrictions are the heart of the “Cannot Delegate” error message that we are seeing. GTS requires:

1. A domain controller must be available to validate the logon. GTS cannot use cached credentials. It has to validate the login token on every connection. This obviously has performance implications, but fortunately it isn’t on very query; but just when the connection is established. This is easy to see (and it was the way that I generated the error message box above) –> just unplug the network from your laptop. You see that you can use SharePoint and Excel Services for everything using cached credentials until you go to Excel Services and try to connect to any data source (PowerPivot included) using Windows authentication.
2. The server must be a member of the same domain as the caller; or there must be a two-way trust relationship between the domains. This means that a common Windows 2000 domain architecture cannot be used by GTS.
3. The caller cannot be a local machine account. GTS only understands how to talk to domain controllers.
4. The account being used as the Excel Services service account must have AD rights to be able to query the object. One place where we know this restriction comes into play is if you have configured your domain controller to have a subgroup under “Users”, e.g. “Service Accounts”, which is a separate AD group that derives from “Users” –> but I am sure that there are more. AD rights for service accounts is a common problem across all of SharePoint.

So – this is happening to you – you cannot delegate credentials. What are your options? First, you could fix the problem so the restriction no longer holds, e.g. you could establish a two-way trust between the domains rather than a one-way trust, but this is likely not a doable approach because you probably have business justifications for why the configuration was done this way. A second alternative is that you could switch to “None” as the authentication, this is how the “off-the-network” blog entry that I wrote, see http://powerpivotgeek.com/2009/11/06/taking-your-server-off-the-network/. This is straightforward and easy to implement, but it lacks the strong security enforcement of Windows authentication and it is a more general account. So there will likely be business issues with that approach also.

From time to time, particular from knowledgeable SharePoint users coming up to speed with PowerPivot, I get the question: “PowerPivot have ‘cross-farm’ support”. As you can see from the title of this post, we don’t support it – and in this “A Peek Inside” I hope to explain why.

First, what is SharePoint ‘cross-farm’ support [...]]]>

From time to time, particular from knowledgeable SharePoint users coming up to speed with PowerPivot, I get the question: “PowerPivot have ‘cross-farm’ support”. As you can see from the title of this post, we don’t support it – and in this “A Peek Inside” I hope to explain why.

First, what is SharePoint ‘cross-farm’ support and why is it important. In large, complex SharePoint configurations a common requirement is to specialize servers or farms of servers to specific services. A good example of this approach is to have a separate farm dedicated to Search. Rather than having each end-user farm host its own Search service, the idea is to get better scale through specialization. Content crawling is done remotely; the indexes are kept remotely; and the Search results are calculated remotely. End users connect to the content farms (so-called because that is where the content is stored), but the content farm reaches out to specialized servers/farms for other services. Example of these services is: Search, Personalization, Business Data Catalog, Portal Usage reporting – coming in SharePoint 2010 are lots more . . .

Here is an example:

So why can’t we put PowerPivot out on one of the specialized farms. At a cursory level this sounds like a good thing. You can could share PowerPivot servers across the whole enterprise rather than having to replicate them within each of the content farms (where the workbooks are stored). Sounds like a great idea. Unfortunately however, it isn’t technically possible. For those services such as PowerPivot and Excel Services that rely on on access to the content, there is no way for the remote service to reach back into the content farm to access data. That is OK as far as it goes, but let’s get geekie and dig a bit deeper.

If you take a look at the kinds of services which can be spun out to specialized servers. These have several interesting characteristics, i.e. they are self-contained, and independent of the content itself.

BDC –> Obviously getting access to transactional systems is totally unrelated to the content (.docx, pptx, xlsx, etc.) that is stored in the content farms. Typically you are using BDC servers to lookup line of business reference data, e.g. customer master lists, product catalogs, transactional data, etc. None Typically this means lookups based on keys, e.g. give me the reference data for customer #42, or give me the reference data for product “abc”, or give me the POS data for transaction #291129211. As there are no references to the content, the BDC service can easily be deployed in its own server farm where it acts as a front-end to underlying corporate databases.

Personalization –> While personalization is related to a certain kind of content, i.e. the SharePoint People and Groups that would be used across various farms, it is also very self-contained. The keys are passed into the service for data lookups, but the actual personalized information is fully  contained within the service itself and it has no dependency on the content.

Search –> At a first glance, it seems like Search should not be a good candidate for server farms. After all, Search is all about the content. Isn’t it?? As it turns out, it is about the content, but not during the actual lookup process. The Search farm contains the crawlers that are indexing the various SharePoint content farms (and from that point of view, it is all about the content), but it also crawls other sources as well. As it turns out, what the Search really needs at runtime isn’t the content – it is the indexes about the content. As these are kept self-contained within the Search farm, it turns out that Search is a good candidate for a server farm because at run time, what is used isn’t the content itself (there is no need for Search to reach back into the content farms), but rather access to the indexes that matters.

So when we were deciding if PowerPivot would be a candidate for cross-farm specialization, we had to look at the need for access to content. And this brings us to the “geek’est” (is that a real word??) part of the post. How do you programmatically access content in SharePoint. There are two different aspects to the problem: getting access to content if data is within the farm; and getting access if data is outside the farm. If the data is outside the farm, then there are 3 ways:

1. Using the web services APIs that SharePoint exposes (flexible but very inefficient at large data sizes)
2. Using a standard HTTP GET against the SharePoint URL for where the content is stored, e.g. (this is what Search crawlers use)
            /site/subsite/doclib/file.xlsx">/site/subsite/doclib/file.xlsx">http://<sp_server>/site/subsite/doclib/file.xlsx
3. Using WebDAV that allows you to access content from the SharePoint farm as if it was a large, single file share, e.g.
            \site\subsite\doclib\file.xlsx">\site\subsite\doclib\file.xlsx">\\<sp_server>\site\subsite\doclib\file.xlsx

Unfortunately none of those techniques work if you are within the farm because firewall rules do not allow backend servers to loopback into the front-end servers. If the data is within the farm then there is only one way to access content: use the SharePoint object model (aka SP ‘binary OM’) , e.g. SPFile.OpenBinaryStream. This is the preferred way of getting data as it is very efficient and accesses the SharePoint content databases directly. It is an order of magnitude (or more) faster than using the outside the farm APIs, even if loopback was allowed.

Now here’s the problem. The binary OM only allows you to access content from within the farm. You cannot reach out an retrieve data from a different farm. In the case of a remote PowerPivot server farm, it cannot use the binary OM to access the content farm – except if using the ‘outside the farm’ APIs – which is very inefficient and would not perform anywhere near fast enough. Thus for those services that are “content focused”, such as Excel Services and PowerPivot, you have no option but to build their app servers as part of the content farm; cross-farm support is not possible or feasible until there are considerable changes made to the SharePoint binary OM.

One of the things that users just kind of glance over, but don’t realize the implication, is the fact that PowerPivot is a copy of the data. If [...]]]> Ok. This post will be a bit complicated but stick with me. Hopefully, in the end, all will be clear. And the geek in you will love it.

One of the things that users just kind of glance over, but don’t realize the implication, is the fact that PowerPivot is a copy of the data. If you haven’t already, let me suggest that you read my "Where’s the beef?" posting. In that posting I talked about the fact that data itself is pulled into the workbook when you save it. When you click any of these buttons:

The import process runs. From then on, the data can start change and shift away from the values that is stored in memory and ultimately in the workbook. The data is ‘real-time’ only when the import is running; afterwards all calculations, pivot and slice is driven by the stored data. On the client this is clear because we have the ‘Refresh’ button (and its options) that provide refresh on the client. But how about the server?? Well, that is the core of this posting. Let’s take a closer look at it. We will start at the menu items for the Excel Services rendering of the workbook. Notice the options here:

The question is “What does it mean to ‘refresh’ the connection?” The answer to that is that it depends on the data provider. For virtually every OLEDB and ODBC provider that Excel Services uses, ‘refreshing a connection’ means going out to the data source and re-querying the data source for its data. SQL Server RDBMS, Oracle, Teradata, virtually to everyone it means refreshing that Excel Services data. And it means that in PowerPivot also, but in PowerPivot where is the data stored? (You know the answer this already, don’t you). The data is in the workbook. Has the workbook changed since you last opened the .xlsx file? Well, I suppose it might have – and in which case, refreshing the connection might bring in new data. But in the vast, vast number of cases, refreshing the PowerPivot table means just re-reading the data that Excel Services already has. In most cases, it has absolutely no effect at all.

To really drive this home, let’s shift into super-geek mode and drill down into the workbook itself. I will go back to the workbook in the first screen shot and first click on the Connections option in the Data ribbon. Notice that there is a connection that has been defined behind my back in the workbook. It is called “Sandbox” which by the way was the name of our system prior to Gemini and prior to PowerPivot. I didn’t create that connection. It was created for me when the PowerPivot Excel add in was first started. This is the connection which is actually interfacing to the in-memory database. Now let’s drilldown further into the “Sandbox” connection and look at its connection string. WOW! The “Data Source=” property, which would normally point to the server for where the database is stored, instead points to “$Embedded$” – What’s that??

$Embedded$ is the magic tag that tells PowerPivot for SharePoint that the data does not come from some server somewhere – instead the data comes from the workbook itself. One of the new OLEDB interfaces created for PowerPivot is a property that Excel Services sets which contains the URL for the workbook that Excel Services is opening. The msolap OLEDB provider takes that URL and replaces the $Embedded$ string with the URL itself –> and thus the infrastructure will read its data from the workbook itself.

But – and this is the critical “BUT” – notice that the embedded content never changes. After you upload a workbook, that workbook doesn’t change on its own. Thus neither does the data. Remember the data is a copy of the data that is embedded in a workbook. If Excel Services refreshes it, the ECS calc engine gets the same data over and over again. The SSAS database embedded in the workbook hasn’t changed – so the data refresh is a nop – it never changes. Refreshing a connection to an embedded PowerPivot database doesn’t refresh anything. You get the same data over and over again.

So, how does the workbook data get refreshed? After all, there must be some way to do it . . . In fact, there are two ways:

1. Bring the workbook down on the client and refresh the data in the workbook. Then re-publish the workbook back to the same location in SharePoint. New data is automatically given to Excel Services and existing connections.
2. Use the data refresh facility, see the data refresh posting and detailed steps posting for more information. In this case the PowerPivot System Service will reach out and pull in new data into the workbook. A new version of the workbook is created and new data is automatically give to Excel Service and existing connections.

And before you ask, No, PowerPivot V1 has no option to monitor the data in real-time and update its data in-memory as the source data changes. The workbook captures the data at a point in time – and then users work with that data. There are no provisions for real-time access to data while doing analytics / calculations / pivot table operations.