Original Link: https://www.anandtech.com/show/797
The XP Transition – Feel the Pain
by Randall Kennedy on July 10, 2001 9:06 PM EST- Posted in
- IT Computing
Introduction
It’s an old story. You spend countless hours and many irreplaceable budget dollars upgrading your Windows PC infrastructure only to discover that Microsoft’s latest and greatest is also the company’s slowest and fattest. In fact, the new software is so bloated that it’s burying your once state of the art PCs. End users are furious. The CIO wants answers. Suddenly, the daunting task of maintaining your enterprise client software becomes an impossible race against time in which depreciating hardware assets are overtaken by that most insidious of phenomena: OS and Application “Inflation.”
Compounding the problem are the confusing and often times contradictory statements mad by the vendors themselves. On the one hand, they like to tout all of the exciting “bells and whistles” that their new version includes. Yet at the same time they claim that it will “run just fine” on your current hardware. Savvy veterans of the upgrade treadmill, they are loath to admit that their spiffy new features demand more computing power – to do so might jeopardize sales. Case in point: Those ridiculously unrealistic “System Requirements” that accompany virtually every major Windows or Office upgrade.
In the end, customers are forced to fend for themselves, often without the prerequisite tools. Here the mainstream benchmark developer community has really dropped the ball. While tools from BAPCo and ZDBOP are able to provide us with performance figures from various hardware configurations, they aren't able to do the same for various software configurations. You can't run SYSMark 2001 and compare Office 2000 to Office XP, although it works just fine for comparing a 1.4GHz Athlon to a 1.0GHz Athlon.
Clearly, something must be done to bring the cross-generational performance issue to light. Hence our motivation in developing this article: The need for reliable, objective metrics data that illustrate the performance impact of a major OS and application upgrade. Our goals for the project were straightforward: Measure the performance of a common test script as executed against three major OS and two major application suite revisions; and translate the results into a practical, tangible hardware platform performance recommendation for new PC purchases.
Setup & Methodology
The first step was to select the target platforms. In order to give ourselves the greatest freedom when designing our test scenarios, we decided to forgo the various DOS-based Windows incarnations and focus exclusively on enterprise-caliber platforms based on the Windows NT kernel. This narrowed our field to Windows NT Workstation 4.0, Windows 2000 Professional and Windows XP (Whistler Beta 2 release - see our test note for why we didn't use RC1). By selecting only true 32-bit OS platforms we were able to construct a much more sophisticated simulation scenario than would be possible under the system resources-constrained Win9x architecture.
Next, we chose the application suites where once again our selection list was narrowed for us. Out of the various flavors and versions of Microsoft Office, only Office 2000 and the recently released Office XP include a comprehensive object model suitable for scripting. This, combined with their common application set (Word, Excel, PowerPoint), made a generational study of Office 2000 vs. Office XP the logical choice.
Finally, we had to settle on a test script. In this case, the choice was easy: OfficeBench 2001. Only CSA Research’s OfficeBench 2001 script could provide the kind of application and OS flexibility that our project required – application flexibility in that it works with both versions of Office; and OS flexibility in that it will install on all three of our target operating systems. Add to this the ability to apply concurrent workloads to the test mix, and we had the perfect tool for the job.
Of course, we still needed to decide on a common hardware test bed. In an effort to stay as mainstream as possible, we chose a genuine Intel D850GB motherboard with a healthy serving (256MB) of PC800 RDRAM, ATI Radeon DDR (64MB) video, Intel Pro 100+ Management NIC and driven by a Pentium 4 CPU running at 1.5GHz. It’s not the fastest system on the planet, but it is what most OEMs have been pitching to their enterprise IT customers as the current state of the art in PC design. Throw-in a set of 3 identical 7200RPM UDMA/66 EIDE disks (one for each OS version – alternated between test sessions), and we had what we felt to be a fairly representative PC test bed. Note that the performance standings wouldn’t have changed had we used an AMD Athlon platform.
To fully explore the issue of generational OS and application performance we decided to run 2 different test scenarios against each of the 6 possible Microsoft Windows and Office combinations, for a total of 12 unique data points. The test scenarios themselves corresponded to two different OfficeBench 2001 loading levels:
Baseline Scenario – Involved booting the PC and running the OfficeBench 2001 Test Script by itself with none of the background Loading Tasks active on the system. In such a configuration, the scenario does little to tax the underlying OS and hardware multitasking capability.
Light Multitasking – Similar to the Baseline scenario except that this time the Loading Tasks are active at their lowest setting. Given the nature of the Load Simulators in OfficeBench 2001 – Database, Workflow and Multimedia – this translates into a moderately active knowledge worker environment where multiple, concurrent applications are running on the system and competing with the Test Script for CPU time and other resources.
In each scenario we began executed a single, “caching” run of the Test Script in order to force the OS to cache the various file images, eliminating application startup time as a factor. We then executed the scenarios in a 10-iteration loop, with OfficeBench 2001 automatically starting/stopping the Loading Tasks and recording the completion times for the individual test runs.
Other key steps included disabling all font smoothing and interface animation, running the disk defragmenter on the hard disk prior to testing and maintaining a constant color depth/display resolution (1024x768x16bpp at 75Hz). All OS installations were configured as Windows 2000 domain clients and connected via a full-duplex, 100Mbps Ethernet backbone.
The Test
Author's Note: By the time you read this, Release Candidate 1 of Windows XP will be shipping to Preview Program participants and beta testers. Though we had the RC1 code in time to re-test for this article, our efforts to do so were thwarted by an unexpected incompatibility between the new Windows build and the Intel UDMA drivers we used under Beta 2. This made it impossible for us to conduct a true, apples-to-apples comparison of RC1 vs. the other platforms. However, we did run the same set of tests using the native Microsoft DMA drivers, and from what we saw using this nearly identical configuration, it doesn't appear that RC1 will offer *any* significant performance boost over Beta 2. So our analysis here still stands. If performance changes significantly we will reinvestigate this issue if/when it does.
Windows NT4 / 2000 / XP Test System |
||||||
Hardware |
||||||
CPU(s) | Intel Pentium 4 1.5GHz | |||||
Motherboard(s) | Intel D850GB (BIOS Revision 86A.0058.P12) | |||||
Memory |
256MB PC800 Samsung RDRAM |
|||||
Hard Drive |
Maxtor
51536U3 (Ultra ATA/66 - 7200RPM) |
|||||
CDROM |
N/A |
|||||
Video Card(s) |
ATI Radeon DDR 64MB |
|||||
Ethernet |
Intel Pro 100+ Management Adapter (Driver v5.41) |
|||||
Software |
||||||
Operating System |
Windows
NT Workstation 4.0 Service Pack 6.0a |
|||||
Video Drivers |
|
|||||
Benchmarking Applications |
||||||
IT/Enterprise |
CSA
Research OfficeBench 2001
|
Performance
When we first ran the Baseline Scenario we thought for certain we were in for a major upset. According to OfficeBench 2001, our Windows NT Workstation 4.0 configuration was a solid 60% faster than its nearest competitor, Windows 2000 Professional. And while the gap closed significantly when upgraded the boxes to Office XP, the venerable Windows NT was still 10% faster than the more sophisticated Windows 2000 and a full 26% faster than Windows XP (Beta 2).
The tables turned, however, when the focus shifted to Multitasking performance. Here the various optimizations in the Windows 2000 kernel allowed it to pull ahead of Windows NT 4.0 by 8% when running against Office 2000. The same tests yielded a delta of 7% when Office XP was used. And while the gap between Windows XP and Windows NT 4.0 closed significantly under this scenario, the new OS still trailed its immediate predecessor, Windows 2000, by 11-12% across both Office versions.
Interesting numbers, to be sure. However, the real story only becomes apparent when you compare the data on a combined OS/Application generation basis. Viewed in this light, the move from Windows 2000/Office 2000 to Windows XP/Office XP translates into a performance hit of from 30-37%, depending on the workloads involved. This is a huge drop in overall system throughput and a chilling testament to just how bloated the Windows/Office code base has become.
Disclaimer: The above results are based on an analysis of the Beta 2 build of Windows XP Professional. Microsoft will no doubt incorporate significant optimizations as we draw closer to the RTM date. Still, 37% is an enormous delta, and it has been our experience that late build OS optimizations are worth, at best, a 10-15% improvement over an otherwise polished Beta.
The net result is that customers will need to budget for a 25-30% performance loss per client PC when transitioning from Windows 2000/Office 2000 to Windows XP/Office XP. Put into real-world terms, this represents the equivalent of a 500MHz performance delta – i.e. you’d need to crank-up the processor clock on our test bed by 33% to 2GHz in order to compensate for performance lost to OS and application “inflation.”
Final Words
The XP transition is going to be a painful one. In fact, the double-whammy of OS and application code bloat could prove to be one of those unplanned “budget busters” that diverts funds away from critical projects and constricts IT spending months into the future. And since the performance loss we observed could not be linked to any specific configuration or tuning error (we used identical hardware and, whenever possible, the same device driver revisions), we must conclude that the phenomena are intrinsic to the OS and that the only viable solution is to upgrade the hardware.
If there’s a moral to the story it’s this: No matter how much vendors try to disguise their real-world system requirements, you never get “something for nothing.” In this case, the price of business productivity “bliss” is an increased appetite for CPU cycles. Veteran IT professionals should already know this (again, it’s an old story), so shame on anyone caught off-guard by a vendor’s misleading and/or unrealistic “system requirements.”
Bottom Line: If you plan to upgrade to Windows XP/Office XP, and if you’ve already qualified new PC platforms based on your experience with Windows 2000/Office 2000, you’ll need to revise your minimum system performance levels upwards by 25-30%. That should give you enough headroom to compensate for the more “portly” XP combination.