Quest Software Benchmark Factory

We mentioned that the benchmarks we previously used are no longer useful, as we did not have the I/O capacity required to support them. We went looking for alternative benchmarks, and stumbled upon Benchmark Factory from Quest Software. Below is a description of the product and the benchmarks we used in this article.

Benchmark Factory for Databases is a performance and code scalability testing tool that simulates users and transactions on the database and replays a production or synthetic workload in non-production environments. This enables organizations to validate database scalability as user loads increase, application changes are made, and platform changes are implemented. Benchmark Factory is available for Oracle, SQL Server, DB2, Sybase, MySQL and other databases via ODBC and Native connectivity.

Benchmark Factory provides many tests you can run, and has a very nice and customizable metric reporting engine. We decided to run the AS3AP test, and the Scalable Hardware CPU, Reads, and Mixed tests. Here is what Quest's help file says about these tests:

AS3AP

The AS3AP benchmark is an American National Standards Institute (ANSI) Structured Query Language (SQL) relational database benchmark. The AS3AP benchmark provides the following features:
  • Tests database processing power
  • Built-in scalability and portability that tests a broad range of database systems
  • Minimizes effort in implementing and running benchmark tests
  • Provides a uniform metric and straightforward interpretation of benchmark results
Systems tested with the AS3AP benchmark must support common data types and provide a complete relational interface with basic integrity, consistency, and recovery mechanisms. The AS3AP benchmark can test systems ranging from a single-user microcomputer Database Management System (DBMS) to a high-performance parallel or distributed database.

Scalable Hardware

The Scalable Hardware benchmark measures relational database systems. This benchmark is a subset of the AS3AP benchmark and tests the following:
  • CPU
  • Disk
  • Network
It can also test any combination of the above three entities

We run three iterations of each load point, and then average the results. We also monitor deviations to ensure they are within an acceptable range. We like to see a max deviation of +/- 3%.

Choosing the contenders

In previous articles, we've been asked to explain why we chose the parts we did for an article. For this article Intel sent us their 3.0 GHz Harpertown CPUs. We requested the 3.0 GHz Clovertown CPUs, which are 120 Watt TDP parts, to allow us to do a clock to clock comparison of Harpertown to Clovertown. We also tried to get Harpertown 2.66 GHz or 2.5 GHz CPUs but none were available. These would have provided us with the closest cost comparison to the Opteron 2350's, but it was not possible. We resourcefully acquired two of AMDs newest Opteron 2350's and we requested the Opteron 2222 3.0 GHz Opteron CPUs, which are the highest clock in the 95 Watt TDP envelope. We did review the results of the Opteron 2224SE 3.2 GHz 119W TDP CPUs but their performance was only marginally better than the 2222's and their performance/watt was consistently lower and thus we concluded of less of interest for this article.

AMD Quad-Core Opteron (Barcelona) Test Setup
Comments Locked

77 Comments

View All Comments

  • Jason Clark - Tuesday, September 18, 2007 - link

    Ross and I did not have 2.5Ghz, it was nearly impossible just getting ahold of 2.0GHz.... We'd run it if we had it :)
  • Regs - Wednesday, September 19, 2007 - link

    Hi Derek Johan De Gelas mentioned that had 2.5 GHz part in your tech labs. Can we expect a preview of that soon?
  • Regs - Wednesday, September 19, 2007 - link

    Whoops, I confused you Jason for Derek Wilson.
  • Viditor - Tuesday, September 18, 2007 - link

    Which stepping of Barcelona were you using (it wasn't in the test setup and has become an issue of late)

    Cheers
  • firewolfsm - Tuesday, September 18, 2007 - link

    This review seems biased. If you want to run only the 2GHz part at least calculate the performance per clock because it looks like Barcelona has Intel beat in a lot of the benchmarks. Meaning 2.5GHz would be much more competitive.
  • firewolfsm - Tuesday, September 18, 2007 - link

    Sorry for the double...
  • GlassHouse69 - Tuesday, September 18, 2007 - link

    Well. It looks like AMD wins. Money is everything. Oh sure, there will be some geek who will say, "Money is no object, only getting the workload done as fast as possible." that geek would be wrong. Amazing how this WOULDNT start a price war. 400 vs.... 2.5-3x that much? You could put four on a board and start rocking in the free world.

    Makes me happy about Phenom. Imagine a 190 dollar quad that isnt intel? something to buy finally.

Log in

Don't have an account? Sign up now