Home - Chris Gideon

9/3/2009

Deployment has an ASP.NET web part that is calling into a COM+ based application. Rather than use Kerberos Delegation the web part calls LogonUser(). This results in ~200-300 NTLM Auth/second. The company's infrastructure is based on x86 DC's with 14 domains. They had one outage after another which manifested as white screens in IE when accessing SharePoint. I captured a memory dump of W3WP.exe and found that all the active threads had been waiting for quite some time on RPCSS. Went back to the server captured a memory dump of the RPCSS process. It was waiting on LSASS. Captured a memory dump of the whole machine to look at LSASS, it was waiting on a response from two auth requests with several in the semaphore queued up waiting to be authenticated. Learning's from this experience. If you have 14 domains and x86 DC's make sure you have DC's or GC's (preferred if you are in Native mode or greater) close to the machines generating the authentication. Second, it's easier to view the netlogon.log file to find this problem. Third, now it's easier to enable the Netlogon performance counters to diagnose this problem. This happened before they were created. Fourth, if the DC's can be profiled to ensure they can handle the load increase MaxConcurrentAPI on the SharePoint boxes and the DC's servicing SharePoint so that the request doesn't bottleneck on the trust. Fifth, if you use LogonUser() understand the consequences and consider Kerberos instead.

Second story:
SharePoint farm is in an environment where the GC's are x64 with lots of ram. They also had 8 domains around the world. There are 35,000 (150,000 total) users hitting the farm on average due to the heavy collaborative nature of the farm we saw 3000-5000 NTLM Auth/second. The DC's were fine from a utilization perspective. However, we were seeing a bottleneck on SharePoint servers in the form of White Screens or spinning globes. Checked the Netlogon.log and found we were stalling on MaxConcurrentAPI. We bumped it slowly on the SharePoint servers and the problem still occurred but slower to get there. Moved the troubleshooting to the DC's and found that the trust was the bottleneck. Increased MaxConcurrentAPI on the DC's. The problem took longer to surface. Finally had enough data to see that the problem almost always surfaced on Wednesdays. Discovered that all the local DC's were being rebooted on Tuesday night. Secure Channels were being established with DC's over the WAN.Moral of the story, MaxConcurrentAPI modifications are not a cure all. A solid Domain architecture and DC placement are critical to success under high volumes.

Third story, why some don't like messing with MaxConcurrentAPI.
A colleague of mine that had little experience with Domains called me after bumping MaxConcurrentAPI to it's highest value across SharePoint Servers and DC's. He was seeing the DC's spike in utilization to the point where they were sending back and RPC Server too busy error. When I asked if he had first profiled the DC's to see if they could handle such a jump in load he said no. Remember when you bump MaxConcurrentAPI you are increasing load on the DC's by multiples. For example, if I have a default of 2 threads doing concurrent authentication and I bump it to 4 I have doubled the load from that server to my DC if the traffic is constant. In this customer's case it was caused by Virtualized x86 DC's with really poor allocations to RAM and bad disk planning. Therefore always test, test, test before increasing MaxConcurrentAPI. Having spent years in AD I assume people will do this but it's not always the case.

Now take those stories and multiply them by dozens of times. This is why I became such a Kerberos advocate because I got tired of working with plumbing created back in 1995. Kerberos is barely faster over long sessions as Spence Harbar and Bob Fox have proven through a great deal of testing. However, it's a life savor in high authentication environments with a distributed DC infrastructure containing several domains. The clients Authenticate before contacting SharePoint. In my next post I will address PAC validation, why I have had to go to extrodinary lengths to turn it on etc.

Posted at 7:49 AM by cgideon | Permalink | Email this Post | Comments (0)

8/26/2009

Where my posts come from...

Sometimes when I make a post there are thoughts or experiences that don’t make it into the post. This comes from being highly randomized in my job and not dedicating the time to blogging that others do. This alone has made me call into question whether or not I should even bother with blogging.

Some background, I used to be a Senior Premier Field Engineer for SharePoint servicing some of the world’s largest deployments. Prior to that I was in our Critical Problem Resolution team for developer Internet products (IIS, ASP.net etc) and further back still was a Solution Integration Engineer (SIE). Most of my posts come from that experience as I now blog about SharePoint. I done lots of other things in Microsoft including Active Directory and Metadirectory. When I post something it’s because I have personally experienced a challenge with something and I don’t want anyone else to go through it again. Because of this view point and the customers I support there is a tendency on my part to see things in high volumes or various levels of complexity because of their multi-national/geographical distribution. In other words, I have a tendency to make generalizations in my posts that may be practices that only large deployments need be concerned.

My promise: I will cleanup and add context to some of my earlier blog posts or make clarifications. In addition, I will try to point out when a practice only makes sense for a large deployment.

Posted at 7:24 AM by cgideon | Permalink | Email this Post | Comments (0)

5/22/2009

Important SP2 Information

Please take a moment to review the information on SP2 posted by the product team.

Attention: Important Information on Service Pack 2

Posted at 5:22 AM by cgideon | Permalink | Email this Post | Comments (0)

5/19/2009

Kerberos and SharePoint

Spence Harbar, SharePoint MCM and Bob Fox, SharePoint MVP, have some great information on the peformance and scalability of Kerberos vs NTLM. Here is a link:

http://www.harbar.net/archive/2009/02/19/sharepoint-best-practices-presentations.aspx

Take a look at slide 17 in the Kerberos presentation.

Posted at 8:47 AM by cgideon | Permalink | Email this Post | Comments (0)

4/28/2009

SP Pre-upgrade Checker

With the release of SP2 today there are now several reports that can be run to find some pretty important information about your farm. In addition it starts the process of getting your farm into a healthy state for eventual upgrade for SP2010. Check out these KB's for more details.

960577 List of all Windows SharePoint Services and SharePoint Server Pre-Upgrade Checker knowledge base articles
http://support.microsoft.com/default.aspx?scid=kb;EN-US;960577

There are several more KB's coming on the subject. But for now I encourage you to get familiar with the Pre-upgrade checker.

Posted at 6:06 AM by cgideon | Permalink | Email this Post | Comments (0)

2/6/2009

SPDiag is here!

After considerable work from a lot of people in Microsoft the work of troubleshooting SharePoint issues is much easier. SPDiag has been released.

http://blogs.msdn.com/sharepoint/archive/2009/02/05/sharepoint-diagnostics-spdiag-tool-v1-0-for-sharepoint-products-and-technologies.aspx

Posted at 8:14 AM by cgideon | Permalink | Email this Post | Comments (0)

12/11/2008

Where have I been?

It's been months since my last post. What has kept me from blogging?

A newborn daughter and the lack of sleep that such a wonderful change brings.
Working on the SharePoint Master Certification with some of the greatest SharePoint minds in the business.
Adjusting to a new job that is completely different that what I am used to.
Trying to learn as much as I can about configuring proper storage for SharePoint and many other things.

In otherwords its been a very busy several months. I am adapting and plan on contributing regularly going foward.

Posted at 1:15 PM by cgideon | Permalink | Email this Post | Comments (0)

Search Performance

I realize this is simply pointing to another blog and many have dinged me for it in the past. However, I continue to see amazing results just by following the guidance in Dan Blood's post on the Enterprise Search Blog.

SQL File groups and Search

http://blogs.msdn.com/enterprisesearch/archive/2008/09/16/sql-file-groups-and-search.aspx

Thanks for the great work Dan!

Posted at 12:57 PM by cgideon | Permalink | Email this Post | Comments (0)

7/2/2008

File Level Antivirus and SharePoint

I have seen countless issues induced by not excluding the right directories in a file level antivirus. I am happy to say we have an article on the subject now. Please take it into consideration when deploying WSSv3 or MOSS.

952167 Folders may have to be excluded from antivirus scanning when you use a file-level antivirus program in Windows SharePoint Services 3.0 or in SharePoint Server 2007
http://support.microsoft.com/default.aspx?scid=kb;EN-US;952167

Posted at 1:09 PM by NORTHAMERICA\cgideon | Permalink | Email this Post | Comments (0)

6/6/2008

Hotfixes and SP1

Updating 12/11/08

For deploying fixes in general here are the proper steps.

Deploy software updates for Office SharePoint Server 2007

http://technet.microsoft.com/en-us/library/cc263467(TechNet.10).aspx

For a green field deployment I recommend you slip stream as per Create an installation source that includes software updates (Office SharePoint Server 2007)

http://technet.microsoft.com/en-us/library/cc261890(TechNet.10).aspx

What should I have in my slipstream to get the latest fixes?

957691 Description of the Windows SharePoint Services 3.0 hotfix package (Sts.msp): October 28, 2008
http://support.microsoft.com/default.aspx?scid=kb;EN-US;957691
956612 Description of an update for Windows SharePoint Services 3.0 that applies to daylight saving time (DST) changes
http://support.microsoft.com/default.aspx?scid=kb;EN-US;956612
957693 Description of the SharePoint Server 2007 hotfix package (Coreserver.msp): October 28, 2008
http://support.microsoft.com/default.aspx?scid=kb;EN-US;957693
958567 Description of the SharePoint Server 2007 hotfix package (Coreservermui.msp): October 28, 2008
http://support.microsoft.com/default.aspx?scid=kb;EN-US;958567
955586 Description of the SharePoint Server 2007 post-2007 Microsoft Office servers Service Pack 1 hotfix package: July 23, 2008
http://support.microsoft.com/default.aspx?scid=kb;EN-US;955586
958569 Description of the SharePoint Server 2007 hotfix package (Dlc.msp): October 28, 2008
http://support.microsoft.com/default.aspx?scid=kb;EN-US;958569
955937 MS08-057: Description of the security update for Excel Services in Microsoft Office SharePoint Server 2007: October 14, 2008
http://support.microsoft.com/default.aspx?scid=kb;EN-US;955937
957694 Description of the Forms Server 2007 hotfix package (Ifswfe.msp): October 28, 2008
http://support.microsoft.com/default.aspx?scid=kb;EN-US;957694
951597 MS08-069: Description of the security update for the 2007 Office servers: November 11, 2008
http://support.microsoft.com/default.aspx?scid=kb;EN-US;951597
957175 MS08-077: Vulnerability in Microsoft Office SharePoint Server could cause elevation of privilege
http://support.microsoft.com/default.aspx?scid=kb;EN-US;957175

Do I have to PSConfig after each fix?

No, you can deploy SP1 for both MOSS and WSSv3 plus all the fixes and PSCONFIG one time.

Hope this helps,

Chris

Posted at 10:47 AM by NORTHAMERICA\cgideon | Permalink | Email this Post | Comments (0)

RSS Feed

1 - 10

Chris Gideon