Enjoy!

As the steps that we do to accomplish this ‘parallelization’ isn’t talked too much in the regular BOL (there is a bit, but mostly around the setting dialog boxes), I thought that it would make a good blog posting.

First, let’s start off with the basic PowerPivot overall architecture: (I will let other postings cover the details; here is a basic overview)

• PowerPivot for SharePoint has two shared services: one service called the PowerPivot System service is responsible for all back-end PowerPivot work (except the Engine itself); and we have a second shared service that is (you guessed it), the SSAS Engine itself. The Engine service is the regular SSAS Windows service wrapped as a SharePoint shared service. It is set to Vertipaq mode and is ready for loading, querying and processing of embedded PowerPivot datasets that are part of your Excel workbooks. When you install PowerPivot you get both services installed on the SharePoint app server. Across the SharePoint farm, you have “x’” pairs of services (i.e. where you have installed PowerPivot for SharePoint). This might be all app servers in the farm; one app server in the farm; or anywhere in-between. As an administrator you get to decide which app servers will be providing PowerPivot services.
• Associated with the PowerPivot System service is a service application database. There is one service application database per PowerPivot System service application. Here is how you find out which RDBMS is created:

  

  

  The service application database holds:

1. The ‘instance map’ which tells the system which app server has a given workbook’s data is loaded, or cached
2. The data refresh schedules that have been created by end-users. It also contains the ‘run queue’ for those data refresh jobs that are now runable, or being ran.
3. Lastly, the service application database contains data refresh history, i.e. when jobs ran and their status (success, failure, or informational messages).
• There is no “Master” PowerPivot app server. All of our code assumes that each PowerPivot app server is independent and is always looking for work. When a data refresh job is put into the run queue, any of the PowerPivot machines can pick up the work and assign it to themselves. Where we need to control concurrency (since each service instance is independent of each other), we use locking capabilities of the SQL RDBMS for the service application database, e.g. we place write-locks on the run queue table to ensure that no other server attempts to update the run queue at the same time.

To configure data refresh jobs, there are two Engine service instance properties. To see these settings, run Central Administration and click on the SSAS Engine service instance. In this case, here is a server in my office at Microsoft. I went to Central Admin “Services on Server” for the machine DWICKERT-RTM and clicked on the “SQL Server Analysis Services” service at the bottom:

And here are the data refresh settings for the SSAS Engine service instance on DWICKERT-RTM:

The first checkbox tells the system that DWICKERT-RTM can be used for querying, i.e. loading read-only databases on the machine. If the checkbox was not enabled (unckecked), then this machine would have been dedicated to data refresh. The second checkbox is a similar setting for data refresh. If neither is enabled, then the instance will not be used. It is installed and configured, but it won’t be used. The default setting is that both options are enabled, i.e. that a server can be used for both querying and data refresh – but if you want to change it and dedicate the server for one or the other activity, here is where you set the role.

If you have enabled data refresh on this machine then you get to decide how many workbooks can be concurrently refreshing at the same time. We call these data refresh units of work a slot. In the case above we have enabled 4 slots on DWICKERT-RTM. The default setting is the amount of memory divided by 4GB (thus a 16GB machine should result in default setting of 4 concurrent jobs) – although there is a current bug in the system where the setting is always set to 1 regardless of how large the server memory is. The maximum value is the number of CPUs. To get the most use from your machine resources, I strongly recommend that you set the maximum concurrency if the machine is dedicated to data refresh. If you are running on a quad core machine, then the maximum number is 4. The way the system is designed is a bit complication. The UI allows you to enter any value you wish for this dialog box (e.g. you could enter 100 instead of 4 if you wished), but when the system goes to actually run the data refresh jobs, it will generate errors if the maximum concurrent jobs is larger than the number of cores on the machine. While this is not the best behavior, we would have like to stop you from entering a wrong number in the dialog box right up front, but given the fact that it is difficult to monitor remote machines, this was the most effective approach.

So . . . now we have things running, we are all done, Right?? Well as it turns out, No. The one remaining thing to talk about is how a data refresh job gets ‘kicked off’ to begin with. The “kicker” is the PowerPivot Data Refresh Timer job: (again, running Central Admin, PowerPivot Management Dashboard, here it is:)

As you can see, the timer job runs once a minute. This means that the timer job calls into the PowerPivot System service (on each machine where data refresh is enabled) giving it a ‘kick’ to take a look at its scheduled jobs every minute. The PowerPivot System service looks to see if any of the scheduled jobs is now ‘runable’ and if so, the job is placed in the run queue. At the end of each timer job (i.e. each minute), the PowerPivot System service looks to see if there are any runable jobs waiting, and if there are any ‘slots’ available for this machine. If so, the PowerPivot System service starts the refresh process.

All SharePoint interactions, i.e. reading the workbook from the SharePoint content database, saving the file back to the content database, are done in the same calling thread from the RPC call from the timer job into the PowerPivot System service, but when the job is ready to actually do the SSAS processing (where the Engine goes out and refreshes the cube data from the original data sources), then that work is requested in parallel on separate threads within the PowerPivot System service (one thread per ‘slot’). Remember that the actual processing is done by the local Engine service. Remember that in PowerPivot for SharePoint, the PowerPivot System service and the local SSAS Engine instance are always installed and operate as pairs. The PowerPivot System service acts as a ‘gatekeeper’ for the local Engine service. We never have the situation where the data refresh is done by a PowerPivot System service, but executed on a remote Engine service. The local Engine is all that PowerPivot System service knows (or cares) about.

So, like many things in PowerPivot, while data refresh seems simple and straightforward, there is actually a fair amount of technology underneath.

I hoped you enjoyed the geeky tour.

Enjoy!

So where to begin . . . Being a geek, let’s start off with a technical topic. When you are running in a 64-bit process, then you can only use dlls and other executables that are build as native 64-bit applications. Therefore whatever our choice, 32-bit or 64-bit, the ‘bit-ness’ of our approach has to hold for the entire process. For example, if we pick 64-bit, then we need the 64-bit version of Excel 2010, we need the 64-bit versions of any add-ins, such as PowerPivot for Excel, and we need 64-bit versions of any OLE DB providers or ODBC drivers (using the 64-bit version of the OLE DB Provider for ODBC here). So let’s keep that in mind.

Ok, here we go:

Pro for 32-bit:

• It does not require an upgrade to a new OS. While yes, Windows 64-bit is still Windows, there are subtle differences between them.
• Likewise, it does not require new PCs. Depending on what your PCs are, they might not be capable of running 64-bit or may not have the capacity to make use of the 64-bit address space, e.g. they might only have 2GB of memory.
• There is better availability of add-ins and other components on 32-bit. Your existing add-ins will continue to work. For example, if we want to run the Microsoft Communicator add-in to Outlook 2010, then there is only a 32-bit version available. The installation of 32-bit verses 64-bit is at the Office “level” – you cannot have a 32-bit version of Outlook and a 64-bit version of Excel.

An excellent example of this limitation is that the Data Mining add-in for Excel is only available as a 32-bit version. Therefore if you need to run the data mining add-in then you must install Excel 32-bit. The larger dataset and performance improvements of 64-bit are not available to you.

• Likewise there is better availability of data providers on 32-bit. This can be important, particularly if you are using the PowerPivot mashup capability, then being able to use older legacy ODBC drivers is important. Some of those ODBC drivers are available in 64-bit versions (for example, there is a 64-bit version of the SQL Server OLE DB provider), but not all of them. Remember that you cannot mix ‘bit-ness’. If you install the 64-bit version of Excel, then you must use the 64-bit versions of any data providers. But this can be a double edged sword. Data refresh on the server is a 64-bit only environment. Therefore you could easily back yourself back into a corner by selecting a data provider that is available in 32-bit on the PowerPivot for Excel client – but is not available for the 64-bit server and thus you cannot use the data refresh facility on the server. The workaround for a situation like this can be problematic – it is to install a 32-bit version of SSRS and use the data feeds feature to get access to the data. The 32-bit server running Reporting Services is the ‘link’ via to the 32-bit provider. Or you could use a linked server in a similar 64-bit server connecting through a 32-bit server. Kind of a hack if you ask me.
• Lastly, an advantage of 32-bit is that you can run Office 2007 and Office 2010 side-by-side on the same 32-bit desktop. This is not possible with 64-bit because there is no 64-bit version of Office 2007 (we need a common registry hive to have the side-by-side work).

Pros for 64-bit:

• Let’s start with the biggest advantage first – you can handler larger datasets with 64-bit. You can save an Excel workbook with embedded PowerPivot data in it up to 2GB of embedded data. This represents 4GB of in-memory data. In the 32-bit world, you will find that the largest Excel workbook with embedded PowerPivot data is about 300-400MB. This is because in the 32-bit world, the 2GB virtual address space has to handle Excel, plus any add-ins including the PowerPivot for Excel add-in, plus the PowerPivot vertipaq in-memory engine, plus its data structures, dlls, etc . . . thus you will find the largest in-memory data is about 600-800MB – and that translates to a workbook of 300-400MB. Now, it may be that this is large enough for all of your user data, and you may even have a set a lower limit for the SharePoint maximum file size upload. Out-of-the-box PowerPivot for SharePoint in a “New Server” configuration sets the maximum file upload to 200MB. But if you want to go larger, you may quickly find that 32-bit just isn’t large enough for the datasets that end-users need.
• There are some performance gains at the OS level as applications grow beyond 2GB. While this might not be an issue with most desktop applications such as Office, it is quite apparent in larger applications that push the computing power of the desktop – and those benefit from not having to page virtual address space to disk as its memory usage increases. No knowing if your desktop users might be using one of these ‘power’ applications on their desktop it is hard to establish if they will see a performance gain in going to 64-bit, but there are some apps that do.

Thus you have some interesting choices ahead of yourself as we all walk the path of the Brave New World of 64-bit on the desktop. Let’s just hope that we can have some fun while we struggle with it.

Enjoy.