The TPC defines its benchmarks in terms of business-world transactions. For example, a typical transaction as defined by the TPC includes the updating to a database system for such things as inventory control (goods), airline reservations (services), or banking (money). Today, the two primary benchmarks are TPC-C and TPC-D.

TPC-C is an OLTP benchmark. This benchmark involves a mix of five concurrent transactions of different types and complexity, which are either executed online or queued for deferred execution. The database comprises nine types of records with a wide range of record and population sizes. TPC-C is measured in transactions per minute (tpm).

TPC-C simulates a complete computing environment where a population of terminal operators executes transactions against a database. The benchmark is centered on the principal activities (transactions) of an order-entry environment. These transactions include entering and delivering orders, recording payments, checking the status of orders, and monitoring the level of stock at the warehouses. While the benchmark portrays the activity of a wholesale supplier, TPC-C is not limited to the activity of any particular business segment but, rather, represents any industry that must manage, sell, or distribute a product or service.

The TPC-D benchmark, the newest one from the TPC, represents a broad range of decision-support applications that require complex, long-running queries against large, complex data structures. Real-world business questions were written against this model, resulting in 17 complex queries.

For most businesses today, the TPC-C benchmark probably provides the best means for comparing performance of one system to another in a real-world business environment. As more and more businesses begin to use decision-support applications, the TPC-D benchmark will become more important.

Within Rochester, we use the TPC-C benchmark for our performance measurements. The TPC-C measurement process is very involved, requiring a fairly elaborate setup to measure a single system configuration. The TPC also requires that all results be verifiable, and it has an extensive process to ensure that all TPC-C measurements are valid and reproducible. This is as it should be, but it does mean we cannot do this for every possible AS/400 configuration. Instead, we measure selected systems in our product line and submit these measurements to the TPC for verification and publication. These are the TPC-C numbers you see published in various system-performance comparisons.

We want to provide performance information for our customers about every one of our systems, so we do measurements for several configurations with a workload that is based on a version of TPC-C. A couple of years ago, we selected a version of TPC-C and froze the definition to provide us with a consistent benchmark across all our systems. We cannot publish these numbers as TPC-C measurements because the TPC-C benchmark has continued to evolve. Instead, we publish this performance information for all systems based on another rating system we call the Commercial Processing Workload (CPW). As a rule of thumb, multiplying CPW performance numbers by 10 will give a rough approximation of TPC-C performance numbers. Actual measurements of the current TPC-C benchmark typically give slightly higher numbers.

How Do We Compare?

New CPW and TPC-C performance numbers for AS/400e series models are published at every release. For comparison purposes, let’s look at the performance numbers originally measured at the V4R1 announcement (August 1997) for the 12-way systems. At the time this is being written, the final AS/400e series numbers have not been certified and are, therefore, subject to change. Table 12.1 shows the TPC-C performance numbers of the top five systems at the time of the V4R1 announcement, including my estimate of where the 12-way AS/400 fits in this list. Of course, the actual numbers and the positions on the list are subject to change as various vendors introduce new system models and we remeasure existing systems for this benchmark.

You can obtain current TPC-C performance numbers from several places, including the Web site for the Transaction Processing Performance Council (http://www.tpc.org). I personally like to visit the Web site of IDEAS International (http://www.ideasinternational.com), from Sydney, Australia, a company that specializes in the delivery of comparative information for computer systems via the Internet. It is also a member of the TPC council and participates in the definition of the benchmarks. IDEAS International publishes comparative benchmark results for many different systems on its Web page. I like to look at its Top 20 for TPC-C.

The information in Table 12.1 is interesting from a couple of perspectives. First, the tpm numbers for the 12-way AS/400 are many, many times higher than the previous TPC-C numbers published for any AS/400. This single system is now equal to the biggest single system IBM makes, including mainframes. The only larger IBM system configurations are clusters. The second point demonstrated here is the efficiency of the AS/400 design — and implications about what the future holds for this design. To understand this, we need look at the systems that outperform the AS/400 on the TPC-C benchmark.

The highest tpm rating in Table 12.1 belongs to a 24-way Sun Ultra Enterprise server, which has 24 UltraSPARC processors, each cycling at 250 MHz. Considering it has twice as many processors, each running at twice the speed, we should expect the Sun system to trounce the AS/400, but it delivers only about a quarter more transactions. Notice that the 12-way AS/400 outperforms the Sun Ultra Enterprise 16-way server on the list. Even so, the Sun system is still a fairly efficient design when compared with others on the list.