SharePoint 2010 Upgrade Tips and Dreaded User Profile Synchronization Service Stuck on Starting…

SharePoint 2010 Upgrade Tips:

I have been working with SharePoint since way back in 2000 when I started at Microsoft, with every release of the product the upgrade process has been getting easier. If you are upgrading only content databases, no significant customizations are detected, and are not planning on upgrading MySites then upgrading SP2007 to SP2010 is very straight-forward using the database attach method, virtually bullet-proof from  my experience over a multitude of SP2010 upgrades both small and large. If however significant customizations are detected (covering in more detail in subsequent post) have been deployed and/or you are upgrading MySites (focus of this post) then a successful migration requires a solid assessment and planning to limit impact to the business. I always strive to limit the impact of the upgrade on the business keeping the production upgrade in as short of a window as possible: “Everyone leaves on Friday with SP2007 and comes in on Monday with SP2010″:)   To that end sharing a few of the lessons learned during recent SharePoint 2010 upgrades for those of you in the planning stages of an upgrade.  Not a comprehensive guide, for a more comprehensive list of upgrade planning steps the place to start is the SharePoint 2010 Upgrade documentation on TechNet.

Tip #1: Perform Database Attach as the upgrade approach: Database attach is the best upgrade approach providing you put the necessary time into planning the upgrade including planning for several test upgrades particularly if you are upgrading MySites as well. Also provides a viable rollback plan if production upgrade fails-reset the databases to read/write while you address upgrade issues and schedule another pass-live to fight another day:)

Tip #2: Perform Test Upgrades and document results: Always perform test upgrades on a virtual machine in advance of production upgrade to resolve issues and verify a successful upgrade path.

  • Document: Document successful  upgrade path based on test upgrades, including environment variables and PowerShell  scripts with the assumption you could follow step-by-step on production      upgrade. Great documentation to provide IT for future reference as a runbook.
  • Test on production hardware and configuration: Consider a test upgrade pass on the new SP2010 production environment prior to actual production upgrade to identify and address environment variables prior to actual production      upgrade weekend i.e. web part caching issues in production environment  where NLB is configured, web.config modifications, SMTP out-bound and  in-bound email,  Alternate Access Mappings, IIS Bindings, all general configurations. You do not want to address these during actual production upgrade for release!
  • Hint: Plan for available free disk space of 3-times the size of the content database or  user profile database you are planning to upgrade. When upgrading a database exceeding 100GB expect the upgrade process to take some time. I was upgrading a content database of 100GB in one scenario and noticed the upgrade progress stayed at .30% complete for some time, after a duration of 2 hours it eventually upgraded so don’t end the process too soon. Monitor the SharePoint upgrade log file if you suspect it’s taking too long.

Tip #3: Provide adequate time to identify and address customizations:

  • STSADM Pre-Upgrade Checker and PowerShell Test-SPContentDatabase are  required. If you are not familiar with STSADM already then could be rough going:) Overall, being proficient in PowerShell is a pre-requisite skill  for SharePoint 2010 upgrade.
  • Consider how best to deal with the Fab 40 templates and provide ample time if you need  to provide support in SP2010 for existing Fab 40 sites.
  • Determine the need to upgrade InfoPath form administrator templates in Central Administration Form Template Library as these may have to be upgraded manually and there could be custom dlls that need to be migrated as well.
  • Plan for addressing custom applications, services, and custom and/or third-party web-parts.
  • Transfer the customizations into the test upgrade environment and test thoroughly.
  • Consider transferring customizations you plan to keep to production environment prior to actual production upgrade and test against production environment to eliminate environmental variables to address during production upgrade i.e. web part caching issues in environment where NLB is configured, web.config modifications, SMTP out-bound and in-bound email, Alternate Access Mappings, IIS Bindings, and all general configurations.

Tip #4: MySite Upgrades:

  • For MySite upgrades do not forget to create or upgrade the MySite host and set MySite      host URL in Profile Service Application prior to upgrading MySites.
  • Create a new web application to host MySites when possible,  I have found no issues with deploying a new site collection for the MySite host rather than upgrading the SP2007 MySite host with the exception of scenario where the MySite host was originally created in the default web application on port 80. Recommend hosting the MySites site collections as part of a separate web application providing a level of isolation for MySites.
  • Use PowerShell to upgrade MySite SSP and provision UPS application pool etc. Ensure you reference the correct Shared Services Database typically SharedServicesDB etc. rather than the MySite content database when activating User Profile Service Application.

Tip #5: Plan to address the Dreaded User Profile Synchronization Service not starting… You have successfully upgraded your content databases via the recommended database attach upgrade and proceeded to the MySite upgrade process. After completing the upgrade of the profile db following the steps on TechNet to the letter and in preparation had reviewed the UPS Architecture when you advance to the step to configure your profile service application settings i.e. to set the MySite host link SharePoint on the Profile Service Application management link it returns an error page instead of the expected  administration page. Potential cause: the user profile synchronization service is not started.  On more than one occasion I noticed the profile synchronization service was stuck on starting in the manage services administration page for more than 10 minutes after provisioning and activating the UPS and until the service is started you will not be able to access the manage profile service application page.

Resolution: There have been many posts dedicated to resolving the profile sync stuck on starting. Here is one of my favorites http://www.harbar.net/articles/sp2010ups2.aspx that includes significant detail and troubleshooting steps.  TechNet article http://technet.microsoft.com/en-us/library/gg750253.aspx   Recommend reviewing these posts in cases where following the steps I outline below are not successful. There are cases where permissions have not been set properly in AD, NetBIOS issue, or security access issues with the FIM Services themselves that will require access to both SharePoint, SQL, and Active Directory to resolve.  In such cases I involve IT Admin/IT Services team and in some cases contacting Microsoft support. In one case with MS support assistance uncovered changes made to the AD containers were causing synchronization errors.

Quick steps to try before getting too deep into the weeds that has worked for me over the course of several SP2010 Upgrades.

  • Ensure you have installed SharePoint latest SharePoint 2010 cumulative updates http://technet.microsoft.com/en-us/office/sharepointserver/bb735839
  • Ensure the ForeFront Identity Management Services are enabled and set to manual start and configure to run under your service account. DO NOT START YOURSELF-the user profile service will attempt to configure and start these services automatically.
  • Start the Profile Service application from Central Administration (in addition to the Profile Synchronization Service)
  • Reboot the SharePoint server.
  • Verify FIM  services are started. Hint: You can view FIM services in action using the FIM Client located at C:\Program Files\Microsoft Office Servers\14.0\Synchronization Service\UIShell\miisclient.exe
  • Verify the Profile Synchronization Service and User Profile Service status are both reporting as “Started” and no longer “Starting”
  • If the Profile Synchronization Service status is still stuck on starting, try setting the FIM Services to “Automatic-Delayed Start” then Reboot once more.
  • At this point your profile synchronization service status should be reporting as started, review errors in event log, and if successful you will be able to access the Profile Sync Settings page and verify the user profiles have been successfully imported, configure your MySite settings etc.
  • Hint: Do not be shy about deleting the UPS application and re-activating using PowerShell from scratch once more to ensure all configurations are correct.
  • Consider turning off the SMTP service to disable email during user profile upgrade to avoid emails being sent out to Managers from the MySite Cleaner Timer Job.

Summary:

Performing database attach upgrade is a great approach for planning an upgrade that limits the overall impact to the business and provides best roll-back plan. Plan for several test upgrade passes to get to a successful upgrade pass, particularly if upgrading customizations and/or MySites. Document the test results as part of a step-by-step upgrade guide that includes each step, environmental variables i.e. server names, service accounts, database names, general configurations, list of customizations, and accompanying PowerShell scripts will help make the actual production upgrade go smoothly.  Also consider the additional step of performing a test upgrade on the planned production environment to identify and address environmental variables early. Plan out the production upgrade tasks, activities day by day, including all tasks, resources, and estimated time necessary for copying databases from production, estimate for performing database upgrade, and allocated time/resources for performing post-upgrade testing.

Most importantly, bring coffee.

Rod Stagg
SharePoint Practice Manager and Architect http://www.rstagg.com