So . . . here I am playing around with PowerPivot to get ready for a demo. As this is a group from the SharePoint dev team (actually SharePoint Online), I wanted to include some cool SharePoint functionality [...]]]> Updated 28-Oct-2010: Some folks are reporting that an iisreset is needed to make the updates visible to SharePoint.

So . . . here I am playing around with PowerPivot to get ready for a demo. As this is a group from the SharePoint dev team (actually SharePoint Online), I wanted to include some cool SharePoint functionality into the demo. Trying to be cool and ‘wow’ them, I decide to use a SharePoint list as a data source. I want to show the data mashup capabilities of PowerPivot so I have the bulk of my demo come from the Contoso sample database (the three product catalog tables) and the 4 million row Sales Fact table.

I create my SP list by extracting the 11 distinct manufacturers – and I assign them to a ‘shipper’ that I made up.

When I go to SharePoint an click on:

PowerPivot comes up but it returns an error:

"For security reasons DTD is prohibited in this XML document. To enable DTD processing set the ProhibitDtd property on XmlReaderSettings to false and pass the settings into XmlReader.Create method."

So it turns out that I forgot (it happens to the best of us) to install the ADO.NET Data Services 3.5 SP1, located here: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=3e102d74-37bf-4c1e-9da6-5175644fe22d

Which I would have known if I had taken my own advice and read BOL beforehand. Read the “New server” installation notes here: http://msdn.microsoft.com/en-us/library/ee210708.aspx#nextsteps

So the moral of the story is: RTFM

After installing ADO.NET Data Services on my SharePoint site, I was able to bring up the data feed in PowerPivot – setup the relationship between the Manufacturer field in the SharePoint list and the Manufacturer field in the Contoso database and I was off and running . . . I also setup daily data refresh job . . . and I could see the Gallery snapshot changing as I entered new data into the SharePoint list . . WOW!

Enjoy!

Enjoy!

Microsoft does not recommend or support the use of Office applications in a server-side environment (see http://support.microsoft.com/kb/257757 ). In the case of PowerPivot, the data refresh facility is implemented within an NT Service and, as such, falls under this restriction. In this article we will use the ACE provider to give you more control [...]]]> Updated: 18-Aug-2010

Microsoft does not recommend or support the use of Office applications in a server-side environment (see http://support.microsoft.com/kb/257757 ). In the case of PowerPivot, the data refresh facility is implemented within an NT Service and, as such, falls under this restriction. In this article we will use the ACE provider to give you more control over importing text files. So long as the importing of data is done using the PowerPivot for Excel add-in, then you in a supported environment. However you should not be refreshing the data on the server as that uses the ACE provider in an unsupported environment.

Question: By default PowerPivot has two limitations when working with text files (using the Office ACE OLE DB provider): (1) will only import data from files with an extension of .txt, .tab, and .csv; and (2) it will only recognize tab, comma, semicolon, space, colon, and vertical bar and column delimiters. Is there a way to change PowerPivot’s configuration so that it will recognize other file extensions or column delimiters?

Answer: Yup — that is a technical term.  You can use a schema.ini file in the same directory as the imported text file to specify all kinds of things that aren’t available through the PowerPivot UI, including fixed column rather than delimited text file format.  Here’s a web page with the info:

http://msdn.microsoft.com/en-us/library/ms709353(VS.85).aspx

(Thanks to Ron Pihlgren for this solution. Ron is the PowerPivot client test lead – and his blog is included in my blogroll.)

Enjoy.

Microsoft does not recommend or support the use of Office applications in a server-side environment (see http://support.microsoft.com/kb/257757 ). In the case of PowerPivot, the data refresh facility is implemented within an NT Service and, as such, falls under this restriction. If you are using Office data within a PowerPivot workbook (e.g. importing the data [...]]]> Updated: 18-Aug-2010

Microsoft does not recommend or support the use of Office applications in a server-side environment (see http://support.microsoft.com/kb/257757 ). In the case of PowerPivot, the data refresh facility is implemented within an NT Service and, as such, falls under this restriction. If you are using Office data within a PowerPivot workbook (e.g. importing the data from flat files, Excel workbooks, or MS Access databases) and you need to automate the refreshing of that data, then Microsoft recommends that you move the data to a supported server-based, multi-user environment, such as a relational database. That being said, users may still try to use the ACE provider and this information is designed to ease their pain in the process.

Ok. I’ve promised some best practices – here is my first one.

What is the 2010 Office ACE provider and why is this important to PowerPivot? The 2010 Office System Driver for Data Connectivity Components (aka the Office ACE provider) is a OLE DB provider that can be used to read data from and write data to Office 2010 system files such as Microsoft Access Beta (mdb and accdb) files and Microsoft Excel 2010 (xls, xlsx, and xlsb) files. You can also use it to access text files. The provider “Microsoft.ACE.OLEDB.14.0” which includes both 32-bit and 64-bit versions, is available for download here: ( http://www.microsoft.com/downloads/details.aspx?familyid=C06B8369-60DD-4B64-A44B-84B371EDE16D&displaylang=en ). PowerPivot uses the ACE provider as part of a data refresh job to import data.

In this blog, I would like to talk about two important topics concerning ACE: (a) how to get it and use it; and (b) what are some of the best practices that we’ve recently encountered when using it internally for some dogfood servers.

First, how do you get and use the ACE provider:

• You must manually download and install the ACE provider on each PowerPivot app server where PowerPivot for SharePoint is installed. Since you’ve used this OLE DB provider to import data on the PowerPivot Excel add-on wizard, if you in intend to use it for data refresh on your SharePoint farm, then the same OLE DB provider must be on the PowerPivot app server.
• Since SharePoint runs only under 64-bit servers – this means that you must use the 64-bit version of the ACE provider. Your client can be 32-bit, but the SharePoint server forces the use of the 64-bit version on the PowerPivot app server.
• The actual target file must be located on a shared location such as a file share, or some other location that can be reached by both the client import data wizard, and the server-side, SharePoint PowerPivot app server. It cannot be located on the client’s local desktop, e.g. C:\folder\source.xlsx.

So, now you’ve installed it, what is it like when you use it in production? In general, it runs as advertised, but we’ve found two issues that those folks doing deployments should be aware of:

Issue #1 – The ACE provider creates temporary files (copies) from the source data files. The temp files are created after the data refresh job has impersonated the user specified in the job schedule. Since normally the temp folder is ACL’ed to only allow the PowerPivot System service account access, the file creation may fail. If using the NEW SERVER configuration (the single all-in-1 server installation), then the system uses the same service account for the PowerPivot System service account as it uses the PowerPivot unattended account. So if you click on “Connect using the credentials already stored inside this workbook” then things work OK. But if you specify custom credentials, then it is likely that the ACE provider cannot create a file in the TEMP folder, then an error is returned to the data refresh job:

“OLE DB or ODBC error: Unspecified error. A connection could not be made to the data source with the DataSourceID of <guid>, Name of <name>. An error occurred while processing the ‘<sheet name>’ table. The operation has been cancelled.”

To solve this problem, we need to add an ACL to the PowerPivot System service account’s TEMP folder that allows users read+write access. We don’t need all accounts, just those who are specified in the custom credentials section of the data refresh job that is using the ACE provider. The account is the one specified here:

To solve the issue: (on each PowerPivot app server) Use Windows Explorer, running in administrator mode, to modify the “C:\Users\<PowerPivot System service account>\AppData\Local\Temp” folder by adding an ACL that gives the accounts that will be using the ACE provider limited access, i.e. Read+Write.

As you would expect this is a common Windows management issue. There are several approaches to use: (in increasing better ‘best practices’ order)

• Give “Everyone” read+write permission – this is the easiest, but least secure. Typically the C:\Users subfolders are shared out over the network, so any penetrators would have to actually be running on the app server, so this isn’t too bad, but the Everyone group contains lots and lots of accounts.
• Give “<domain>\Authenticated Users” read+write permission – this is better, but provides access to accounts that are not using your SharePoint farm. Clearly this is a large population – maybe even still too large.
• You might already have domain groups that you are using to control access to your SharePoint farm. If those groups exist, re-use them here and give those groups read+write access. This approach limits access to just SharePoint users, but users still might not be using ACE and they have access.
• Create a new local group, called “ACE Users” – Initially this group is empty; as users encounter problems, have your machine administrators add users to this group on request; in the best of all worlds, you would have a provisioning system so user can request access and have the system auto-remove them periodically.

Anyway, somehow regardless of what technique you use, the TEMP folder ACLs must be changed to allow access.

Issue #2 – To actually access the remote files the ACE provider uses SMB via the UNC name (typically in the form: \\server\share\folder\workbook.xlsx)

If the target location is a true file share, then things are OK. If however the target location is a SharePoint farm acting as a file share (using WebDAV protocol), then additional OS components must be installed. Typically this is done because users are storing Access databases, Excel files, or text files that are stored back on the same SharePoint farm as the source workbook (a kind of a ‘loopback’ for data access). If wish to provide this capability to end-users, then you must install the “Desktop Experience” role to the PowerPivot app servers.

If you don’t have “Desktop Experience” added as a server role and a data refresh job attempts to use the ACE provider to access a remote SharePoint farm via its UNC name, then the job will fail with:

OLE DB or ODBC error: Failure creating file.; 3436. A connection could not be made to the data source with the DataSourceID……”

The Desktop Experience role is not by default installed on a server OS and thus your administrator will have to add it as a role to the server. Again, this is not needed if the target files are located on true file shares, or equivalent. It is only needed if the app server is going to be referencing a SharePoint farm (either itself or a different remote farm) using a UNC name.

Enjoy.

One of the things that users just kind of glance over, but don’t realize the implication, is the fact that PowerPivot is a copy of the data. If [...]]]> Ok. This post will be a bit complicated but stick with me. Hopefully, in the end, all will be clear. And the geek in you will love it.

One of the things that users just kind of glance over, but don’t realize the implication, is the fact that PowerPivot is a copy of the data. If you haven’t already, let me suggest that you read my "Where’s the beef?" posting. In that posting I talked about the fact that data itself is pulled into the workbook when you save it. When you click any of these buttons:

The import process runs. From then on, the data can start change and shift away from the values that is stored in memory and ultimately in the workbook. The data is ‘real-time’ only when the import is running; afterwards all calculations, pivot and slice is driven by the stored data. On the client this is clear because we have the ‘Refresh’ button (and its options) that provide refresh on the client. But how about the server?? Well, that is the core of this posting. Let’s take a closer look at it. We will start at the menu items for the Excel Services rendering of the workbook. Notice the options here:

The question is “What does it mean to ‘refresh’ the connection?” The answer to that is that it depends on the data provider. For virtually every OLEDB and ODBC provider that Excel Services uses, ‘refreshing a connection’ means going out to the data source and re-querying the data source for its data. SQL Server RDBMS, Oracle, Teradata, virtually to everyone it means refreshing that Excel Services data. And it means that in PowerPivot also, but in PowerPivot where is the data stored? (You know the answer this already, don’t you). The data is in the workbook. Has the workbook changed since you last opened the .xlsx file? Well, I suppose it might have – and in which case, refreshing the connection might bring in new data. But in the vast, vast number of cases, refreshing the PowerPivot table means just re-reading the data that Excel Services already has. In most cases, it has absolutely no effect at all.

To really drive this home, let’s shift into super-geek mode and drill down into the workbook itself. I will go back to the workbook in the first screen shot and first click on the Connections option in the Data ribbon. Notice that there is a connection that has been defined behind my back in the workbook. It is called “Sandbox” which by the way was the name of our system prior to Gemini and prior to PowerPivot. I didn’t create that connection. It was created for me when the PowerPivot Excel add in was first started. This is the connection which is actually interfacing to the in-memory database. Now let’s drilldown further into the “Sandbox” connection and look at its connection string. WOW! The “Data Source=” property, which would normally point to the server for where the database is stored, instead points to “$Embedded$” – What’s that??

$Embedded$ is the magic tag that tells PowerPivot for SharePoint that the data does not come from some server somewhere – instead the data comes from the workbook itself. One of the new OLEDB interfaces created for PowerPivot is a property that Excel Services sets which contains the URL for the workbook that Excel Services is opening. The msolap OLEDB provider takes that URL and replaces the $Embedded$ string with the URL itself –> and thus the infrastructure will read its data from the workbook itself.

But – and this is the critical “BUT” – notice that the embedded content never changes. After you upload a workbook, that workbook doesn’t change on its own. Thus neither does the data. Remember the data is a copy of the data that is embedded in a workbook. If Excel Services refreshes it, the ECS calc engine gets the same data over and over again. The SSAS database embedded in the workbook hasn’t changed – so the data refresh is a nop – it never changes. Refreshing a connection to an embedded PowerPivot database doesn’t refresh anything. You get the same data over and over again.

So, how does the workbook data get refreshed? After all, there must be some way to do it . . . In fact, there are two ways:

1. Bring the workbook down on the client and refresh the data in the workbook. Then re-publish the workbook back to the same location in SharePoint. New data is automatically given to Excel Services and existing connections.
2. Use the data refresh facility, see the data refresh posting and detailed steps posting for more information. In this case the PowerPivot System Service will reach out and pull in new data into the workbook. A new version of the workbook is created and new data is automatically give to Excel Service and existing connections.

And before you ask, No, PowerPivot V1 has no option to monitor the data in real-time and update its data in-memory as the source data changes. The workbook captures the data at a point in time – and then users work with that data. There are no provisions for real-time access to data while doing analytics / calculations / pivot table operations.