Capacity Planning Testing for SharePoint 2007 Steve Peschka Senior Architect Microsoft OFC319 Complete an evaluation on CommNet and enter to
win! Session Objectives and Takeaways Session Objective: Learn how to determine the real throughput requirements for a farm Understand how to do capacity planning testing on your SharePoint farm NOT a primer on how to use VSTT 2008
Learn some of the different ways in which you can increase capacity in your farm Agenda Know the goal of your testing Know what to measure Determine the throughput requirements for your farm Create the test environment Create the tests and custom tools Run tests and analyze results
Know the Goal of Your Testing Determine what you want the end result to be what are you hoping to prove? While this seems like a simple concept, many people that are doing testing for the first time have a difficult time being very specific in answering this question. Know the Goal of Your Testing Measuring RPS? Most common
Measuring specific operations? Impact on or impact by normal ops Measuring how much / how fast content can be indexed? Measuring performance of individual pages TTLB for pages with custom code clients over latent networks Typically one of two categories: capacity planning verification yes this will work!
proof of concept how much can I do? Know What to Measure Knowing the goal of your testing will make it easier for you to decide which metrics to capture and what thresholds you need to meet for each metric when running your tests. Know What to Measure RPS most tests are based on how many requests you can service. Requests Per Second
is the best number for a test like that. RPS can be used for measuring how many pages are delivered as well as things like how many searches are executed. Know What to Measure Page Time also known as Time To Last Byte (TTLB), this tells you how long it takes to deliver a page back to the client. Often you will use this value in conjunction with RPS; for example, our farm needs to deliver 100 RPS and pages should
load within 5 seconds. Know what to measure Crawl Time for measuring crawl performance you are likely to measure the overall time the crawl takes, your corpus size and how many documents are being indexed per second. Document indexing rate is a little more complicated. Office Server Search Indexer Catalogs Documents Filtered gives you the total number of documents that have been indexed per content source Office Server Search Gatherer Projects Document Add Rate gives
you the number of items indexed per second per content source Office Server Search Gatherer Documents Filtered Rate gives you the overall number of items being indexed per second Determine the Throughput Requirements for Your Farm The rest of this section will focus on testing for farm throughput, or RPS, since it is the most common scenario by a wide margin. Determining the RPS needed to support your farm can be a complicated process.
Determine the Throughput Requirements for Your Farm Rule #1: Number of users means nothing! Rule #2: Number of users means nothing!! Rule #3: Number of users means nothing!!! I can support 100,000 users on a single server farm running on my laptop if they each average 1 request every 12 hours 100k requests over 12 hours is 2 RPS
Determine the Throughput Requirements for Your Farm First determine number of RPS needed Use historical data if possible (IIS logs and Log Parser, Web Trends, etc.) Otherwise, start with number of users Divide users by usage profile What percent are Light (20), typical (36), heavy (60), or extreme (120) Multiply number users by number ops per usage profile
Factor in peak concurrency Determine the Throughput Requirements for Your Farm Example: Contoso has 80k employees; up to 40k may be at work during any 8 hour window 80k users, 40k active, concurrency 5% to 10% peak 10% light, 70% typical, 15% heavy, 5% extreme (10% light x 40k) x 20 RPH = 80,000 RPH (70% typical x 40k) x 36 RPH = 1,008,000 RPH
(15% heavy x 40k) x 60 RPH = 360,000 RPH (5% extreme x 40k) x 120 RPH = 240,000 RPH 1,688,000 / 3600 (seconds per hour) = 469 RPS 469 x 10% peak = 46.9 RPS required Determine the throughput requirements for your farm Now you know RPS required, but what should those requests be doing? This determines your test mix
Again, look to any historical info you can get if possible Otherwise, as in most cases, start making educated guesses Start with test mixes in Planning for Software Boundary documents on TechNet Create the Test Environment Once youve decided what your test goals are, you need to design the environment and tests that are going to be used to execute your tests. This includes
not only the SharePoint environment, but also the operational environment. Create the Test Environment Plan and define what your infrastructure is going to look like AD Forest and Domain Users how many test accounts? How will you create them? What groups are they in? Do you have enough DCs for the authentication load
Monitor the lssas.exe on DCs DNS Load Balancing Remember, names are only resolved once A test agent will route all requests for a namespace to the same IP address once its initially resolved Create the Test Environment Test Servers VSTT Controller
VSTT Agent(s) Use a separate SQL Server for VSTT results so it doesnt impact the SQL Server used for MOSS Turn off anti-virus software on load test controller and agents. Make sure all network settings are configured correctly, names are resolvable, etc. Watch out for bottlenecks on your agents and controller Create the Test Environment
SharePoint farm configuration Depending on test, you may want to stop the timer and admin service, as well as profile imports and crawls Enabled BLOB and/or output cache, if appropriate Use a time-based expiration for object cache All pages should be published; nothing checked-out Make sure navigation is realistic and not overly huge Make sure you test a wide sample of pages Stop anti-virus software (unless you are trying to measure performance when using SharePoint integrated AV) Each Write scenario will change the content database; it should be
restored from backup after a test run that includes writes Create the Test Environment Other SharePoint tasks: How many users are you going to have, and in what roles How are you going to populate those SharePoint roles with AD users and groups Do you need audiences Do you need profile imports Do you need search content sources
Are you going to have to import profiles and crawl content prior to running your tests? If so, how long will that take? Create the Test Environment Make sure you have adequate sample data A very common stumbling block Have enough content for a reasonable search corpus. Uploading the same document many times sometimes hundreds
or thousands of times with different names each time CAN HURT YOU You will need tools in all likelihood to populate sample data, such as those at: http://www.codeplex.com/sptdatapop You almost always end up writing additional tools for other data population tasks Create the Tests and Custom Tools The next step is to create the actual tests that are going to be run. Often times tests are data driven,
meaning test parameters are read from a database or CSV file rather than a static value so that the entire test site is hit over the course of a test. That may mean that additional custom tools are developed and used to capture all of the unique sites, pages, lists, and items in the site so that they can be plugged into your tests. Web Tests Best Practices After recording a test change hardcoded values (urls, params, etc.) to pull from data source
Always have validation rules (error.aspx is 200) Use multiple users and user roles The system behaves differently per user role Avoid using farm admin (it does not benefit caching) Model client apps such as RSS, Outlook, OneNote Dont forget time-based operations (RSS, Outlook sync, etc.) Web Tests Best Practices
Parse Dependent Requests Yes or No? Rarely, analyze the percentage of first-time users When set to True, your bottleneck is most likely going to be WFE disk Internal MS properties analysis tell us 30% of total requests are for resources (js, css, etc.) You can model this in a separate web test Always test your web test Does it work for all URLs? Does it work with all users?
Load Test Best Practices Validate your test mix; a bad mix can make the results unreasonably fast or slow Start in a well known system state Restore a backup before each run Be sure to update stats and defrag indexes Do iisreset Define warmup period for the test; the first request after iisreset does a lot of things
Think times Yes or No? PG hasnt found evidence that think times have different impact on results (exception: IPFS) Virtually impossible to model correctly If Yes, youll need to have higher user load to generate high RPS. It can stress the agents Sample Tests and Data Population http://www.codeplex.com/sptdatapop Was published by PG based in internal tool
Sample tests cover many basic SharePoint operations Use as a reference to build new tests Data population tool defines hierarchical data in XML format Nest the tag in the same way as the product OM (e.g. SPSite->SPWeb->SPList->SPListItem..; Sites>Webs->Lists->ListItems..) Other Tools you May Need Script to create users in Active Directory Tool to scrape a list of webs, lists, libraries, list items and
documents and stick in a CSV or database for use in webtests Tool to upload sample corpus into the site sptdatapop tools May also be able to use your upload document webtest Tool to create webs, lists, libraries, list items, etc. sptdatapop tools Tool to create My Sites May be able to use the sptdatapop tools
Tool to create a list of users and passwords to run tests as demo Add a WebTest, Run a Load Test Run Tests and Analyze Results Your tests have been created, now you are running your tests. What are some of the bottlenecks you can look for? What are ways you can scale your farm up further?
Questions to Ask Yourself Where is the bottleneck? theres always going to be one And is it where I expect it to be? Can performance be improved? How? Is throughput spiky? Are there any strange patterns? Are there many errors? Unexpected resources getting downloaded? Any Load Balancing Issues?
Investigation Tecniques Ensure correct configuration (HW, SharePoint Settings, Load Balancer, etc.) Divide and conquer Run a simple litmus scenario (e.g. homepage) Isolate workloads / operations / pages Reduce farm topology Read vs. Read-Write scenarios Investigating with VSTT
Use Tables view to: Find error rate of individual transactions TTLB per operation Look for any suspicious responses Use Graph view to: Look for RPS and TTLB Patterns Analyze perf counters and Correlate behaviors Analyzing Perf Counters Which machine/s cause the bottleneck?
CPU / Memory / Disk IO / Network Then you can zoom in according to machine role Start with the VSTT default counter sets Add SharePoint / Search / Excel specific ones Patterns and Behaviors The SQL Private Bytes behavior is normal WFE memory is stable We expect this level of spikiness in CPU
Patterns and Behaviors WFE CPU went down across all machines SQL CPU went up SQL Lock Wait Time went up Hence the bottleneck is in SQL demo Reviewing Test Results
Typical Data Related Problems Large lists Lots of Web parts importing non-cached data from various places Cross-list queries and CBQ Web parts Too deep site structures Too many sites in a site collection Too many site collections in a Content DB Too many ACLs Unrealistic org structure showing on MySite
Scaling Points Steves big list of general ideas to consider for scaling once your capacity is understood Pages with custom code maybe code should be rewritten Are my objects within recommended guidelines (# of app pools, web applications, site collections, sites collections per content database, # of items in lists and libraries, etc.) Is tempDB on its own drive? If not, why not? Is the searchDB on its own drive? If not, why not? Are data files and log files on separate spindles? Should I put the configDB on its own drive or SQL Server
Should I put the SSP database on its own drive or SQL Server Should I put the search database on its own drive or SQL Server Should I add additional SQL Servers to host content databases For any of these databases should I add additional data files to a database up to one data file per CPU core, spread across multiple drives In really large implementations or where there are lots of write operations, SQL disk design is critical! Scaling Points (cont.) Should I split my load up by function into multiple farms Am I running x64 (just say YES!!!)
Am I virtualized should I be What does the host look like What does the guest look like Are you using caching Output cache Blob cache Object cache Are you monitoring the object cache and sizing accordingly? SharePoint Publishing Cache Publishing Cache Hit Ratio; select your instance and work on getting around 90% cache hit rate or raising memory allocated to object cache if you are not. Note
however that a site with a lot of read and write activity should expect to have a lower cache hit ratio, so dont over extend this number. Have I over utilized per-item security How big are my pages Fiddler is your friend! Am I using SSL? If so are you using sticky sessions so you dont have to re-negotiate on each page Scaling Points (cont.) Do you have enough network bandwidth to support the traffic? Do you have two VLANs for your farm one for the page traffic and a second backend channel for your servers to talk to each
other and SQL? Is the communication actually occurring on the appropriate VLAN? Do you have a dedicated WFE that is not in your end user pool that is used for indexing Resources www.microsoft.com/teched www.microsoft.com/learning Sessions On-Demand & Community
Microsoft Certification & Training Resources http://microsoft.com/technet http://microsoft.com/msdn Resources for IT Professionals Resources for Developers
www.microsoft.com/learning Microsoft Certification and Training Resources Track Resources Spin the Wheel: Attend SharePoint breakout sessions Mon-Thurs and collect the picture of the day. Come by the booth for a chance to spin the SharePoint wheel. Collect all 4 pictures and enter to win a Microsoft Arc Mouse, drawing 11:30am on Friday, game cards at the booth. Product Info http://www.microsoft.com/SharePoint SharePoint Conference 2009 www.mssharepointconference.com
Dev Resources SharePoint MSDN Web Site: http://msdn.microsoft.com/sharepoint SharePoint Developer Resources: http://mssharepointdeveloper.com/ ITPro Resources SharePoint Tech Center: http://technet.microsoft.com/en-us/office/sharepointserver/ SharePoint Best Practices: http://technet.microsoft.com/en-us/office/sharepointserver/bb736746.aspx Collect the SharePoint Session Picture of the Day Attend any SharePoint Breakout Session day 1 4 and
check the last slide for a daily picture. Come to the TLC yellow area, SharePoint booths OFC 4 & 5 to spin the wheel for a chance to win a prize. The wheel only spins when the SharePoint spotlight is flashing. One spin per person. Thursdays Picture is: Collect all four daily SharePoint pictures and enter to win a Microsoft ARC Mouse. Game cards can be picked up at the booth.
Drawing for Mice on Friday at the wheel at 11:30AM. SharePoint Pie Attend an Interactive Theater Session and enter to win a Microsoft ARC Mouse 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
keith stobie Last modified by: omnamashivaya Created Date: 2/12/2002 3:02:53 AM Document presentation format: On-screen Show (4:3) Company: Microsoft Corporation Other titles: Arial Wingdings Harrington 宋体 Times New Roman Italic Zapf Dingbats Digital Dots 1_Digital Dots Abstract State Machine Outline...
Feasible in higher density corridors, routes with strong anchors on both ends and park-and-ride based peak-period commuter service. 16 - 30. 2 - 4. ... Operator has made a policy decision to emphasize coverage over cost-efficiency.
myPascoConnect is a single sign on launchpad for many common online resources you can use in your daily activities at school or home! You'll be prompted to agree to the updated policies for this school year.
Hydropower- going over dam. Ball flying through air. You walking. 8. SOUND ENERGY. Energy of longitudinal waves. Examples are. Music. Talking. Waves travel into ear, hits eardrum that vibrates hairs and tiny bones in ear, then sends signals to brain....
I knew I wanted to be a nurse when . I was a 20-year-old undergraduate student and saw the professionalism and dedication of the Air Force nurses caring for my closest friend, who experienced severe complications after giving birth. When...
Teacher directions follow the Teacher's Guides, with added information to make the content more concrete, to delve deeper into a concept, or to include SDAIE techniques. There is not really any prep for teachers except for reading over the implementation...