Important: Be careful when installing WSS August Cumulative Update

Last week we discovered a problem with the WSS August CU.

Due to schema and stored procedure changes coming with the August CU databases with a patch level older than the August CU cannot be upgraded to August CU through database attach method.

That means that you need to be careful when upgrading your WSS or MOSS farm to August CU as the usual recommended steps to detach the content database before installing the hotfix will cause problems later when you are going to reattach the databases. In case you need to install the August CU now to resolve an issue that is covered by the fix you should avoid detaching the content databases during the hotfix installation. If the databases are attached the hotfix installation will correctly upgrade the database when running PSCONFIG.

In case you need to detach the databases it would be required to upgrade the database in a separate farm before attaching them back to the production database. E.g. install a test farm with the previous build level the databases, attach the databases there and then upgrade the test farm to August CU. Afterwards you can detach the upgraded databases from the test farm and attach them back to the production farm.

Also be aware that this will also affect database backups taken before installing August CU.

The product group is currently actively investigating this issue to identify a more elegant method to avoid this problem.

5 Reasons SharePoint 2010 PreUpgradeCheck is better than Prescan

1. PreUpgradeCheck is has a better engine, it’s based on a best practices analyzer – Prescan had the job of making sure the database was consistent and wouldn’t fail when you upgraded.  It did a decent job of finding really bad things, but commonly would miss things that would matter.  Ask people why they hated in place upgrade in going from 2003 to 2007.  Why?  Cause it would fail for a ton of reasons that weren’t included in the prescan.  PreUpgradeCheck has an extensible rule base and can get better by simply adding rules.  The rules themselves are easily influenced.  You also have tons more detail and insight around your configuration and insight.  We also didn’t have workflows and content types and features to worry about in 2003 upgrade to 2007.

2. PreUpgradeCheck is read-only – For many this will be the number one reason they favor the likes of PreUpgradeCheck.  Prescan would “Prep” your database or even think about fixing things with your environment.  Really it simpy flipped a bit, but when the bit didn’t get flipped the upgrade would fail.  Now there’s no bit to flip.  BUT, if you forget to run the PreUpgradeCheck, you’ll find it’s not the end of the world.  Being read only most people will feel much more comfortable about running it in production.  YES!  Run it in production.  You need the data that it provides.  Sure you don’t have to start there, in fact I agree don’t start there, but get comfortable with the tool and run it not once but OFTEN!

3. PreUpgradeCheck Reporting Rocks!  Gives you not just a log and screen output, but an XML report and HTM report, and verbose log of all of the rules with a again verbose database and site collection detail that goes way beyond.  I love the idea of comparing the Local vs. Farm reports and comparing server to server reports.  I hope to hear how from small to large people are really leveraging this tool to keep things more synchronized than ever.  You’ll see there’s even more reason to leverage this reporting.  If you’re running PreUpgradeCheck once you’ve missed the point.  Even MS IT runs PreUpgradeCheck way before the upgrade, the hours or days before, and uses it as a cross check against any changes that may have happened.  Serious checks and balances.

4. PreUpgradeCheck tells you about the state of your farm – You did get some insight into your site collections in Prescan don’t get me wrong… from ghosting to site definitions, but you’ll see 10X the detail in PreUpgradeCheck.  I hope people realize the sweetness and insight.  It wouldn’t surprise me at all if people hit their last PreUpgradeCheck report after a disaster.  What else do you have that has all of your SharePoint webapps, your databases, your SSP config, and your AAMs, (seriously where do you have your AAMs written)  Why isn’t PreUpgradeCheck report part of your DR Plan???? IT SHOULD BE!

5. PreUpgradeCheck is a native STSADM command and just as risky or unrisky as any other stsadm command - Presan was an insane install.  You had to download the bits and install them somewhere or find a 2007 install to grab the prescan executable.  It took about a year to even get it as a download center package.  Sad. What a mess!  Some customers were very scared of the tool.  Don’t let that fear from prescan carry over.  The mantra of the upgrade team… *DO NO HARM.*   With PreUpgradeCheck it’s on every farm that is running SP2 update or later.  Great stuff!  As a consultant you aren’t adding bits to run the check, it’s all native.  

As you can tell, there are a ton of reasons you should be running PreUpgradeCheck.  I don’t need to dis the PRESCAN to convince you, but those having any hesitation, this may help.

SharePoint Administrators Toolbox

So there has been demand for a page for all the tools SharePoint Administrators use, so please edit this page and add the tool you use with a short description

Healthcheck

SharePoint Best Practice Analayser

Tool developed by Microsoft that can be used
Best Practices Analyzer

Microsoft SharePoint Administration Toolkit

The toolkit includes SharePoint Diagnostics Tool, the Permissions Reporting Tool, the Quota Management command, and Security Configuration Wizard Manifests.
Microsoft SharePoint Administration Toolkit

Performance

Perfmon

Comes free with Windows OS (permon.exe) and used to monitor various counters.

STSADM

Gary LaPoint STSADM add-ons

This guy has written tonnes of STSADM extra functions that allow administrators to do things that can't be done in the UI and only in the object model...no longer need to rely on Programmers so much.
SharePoint Automation

Logging

SharePoint Logging Spy

Check out this tool for watching the SharePoint logs in real time. http://sharepointloggingspy.codeplex.com/

SharePoint Indexing Performance Tuning Tips

There are many factors involved in the SharePoint crawling process that can impact indexing performance. There are also some steps you can take to improve that. Here are the common causes and their resolution:

• Indexing Performace is set at reduced - common mistake on the configuration screen for the index service. See Central Administration > Operations > Services on Server > Office SharePoint Server Search Service Settings and set to Maximum.
• Number of Connections - by default the indexer will run a limited number of simultaneous threads (6 usually) per host. This can be increased manually by adding specific Crawler Impact Rules for each host. You can really improve speed by setting a large file server up to 64 connections. This number is just a suggestion btw to SharePoint, it also looks at other factors like the number of processors (8 * #procs). And also watch your network for bottlenecks and those pesky RPC errors you may get in your logs (dial it back of you see those)
• Crawled systems are slow or hosted on remote networks. - not a lot to be done here, except by moving those files closer.
• Overlapping Crawls - SharePoint gives priority to the first running crawl so that if you already are indexing one system it will hold up the indexing of a second and increase crawl times.
  • Solution: Schedule your crawl times so there is no overlap. Full crawls will take the longest so run those exclusively.
• IFilter Issues - the Adobe PDF IFilter can only filter one file at a time and that will slow crawls down, and has a high reject rate for new PDFs
  • Solution: Use a retail PDF filter from pdflib.com or Foxit
• Not enought Memory Allocated to Filter Process - an aspect of the crawling process is then the filtering deamons use up to much memory (mssdmn.exe) they get automatically terminated and restarted. There is of course a windup time when this happend and can slow down your crawling. The current default setting is pretty low (around 100M) so is easy to trip when filter large files. You can and should increase the memory allocation by adjusting the following registry keys
  • HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\12.0\Search\Global\Gathering Manager: set DedicatedFilterProcessMemoryQuota = 200000000 Decimal
  • HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\12.0\Search\Global\Gathering Manager: set FilterProcessMemoryQuota = 200000000 Decimal
• Bad File Retries - there is a setting in the registry that controls the number of times a file is retried on error. This will severly slow down incremental crawls as the default is 100. This retry count can be adjust by this key:
  • HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\12.0\Search\Global\Gathering Manager: set DeleteOnErrorInterval = 4 Decimal
• General Architecture Issues - Ensure that you have at least 2 Gig of free memory available before your crawl even starts and that you have at least 2 real processors available.
• Disk Health - the nature of the indexing process causes extensive fragmentation of the file system for both the index server and the database server. Schedule defrags routinely and after all full crawls. Ensure you have enough diskspace always.
• Run 64 bit OS - school is still out on this one, i personally haven't seen much difference as long as there is enough memory and the same processor types, but MS recommends this for large deployments.
• Proper SQL Server configuration (new) - For large (>5 Million) item indexing you will need to plan ahead for the correct SQL Server configuration in order to scale to these numbers. There is one table in particular that grows at 40x the number of items and can severely hinder peformance if you do not treat your SQL Environment like a very large data wharehouse. Here are my recommendations based on experience:

1. Raid 10 Direct Attached Storage Only – minimum 4 arrays – 16 disks

2. Multiple File Groups- pre-allocate all database files and partition on dedicated separate arrays and assign 1-1 for:

a. Indexes for SharedServices1_search db

b. Temp and system databases/tables

c. transaction log for SharedServices1_search db

d. Table content for SharedServices1_search 

i. For every 5 million items have additional file from dedicated drive in dedicated file group. Content and load is spread across and will improve performance

   3. When intially crawling, be sure to pause your crawls every day or so and rebuild/reorg the indexes on the SharedSevice1_search_db database (especially the indexes on MSSDocProps table)

NOTE: It is a good idea to open up perfmon and look at the gatherer stats while indexing. There is a statistic called Performance Level and this reflects the actual level that the indexer is running at where 5 is max and 3 is reduced. Even if you set everything to max the indexer may decide to run at reduced anyways based an some unknown factors.

This is a good read too: http://technet.microsoft.com/en-us/library/cc262574.aspx (Estimate performance and capacity requirements for search environments)

Addendum (7/28/2009)

Here is another good read from MS (http://technet.microsoft.com/en-us/library/cc850696.aspx (Best practices for Search in Office SharePoint Server)