Re: [SAGE] SPEC CPU Benchmark

[ntwrkd <ntwrkd@xxxxxxxxx>]

After some testing with SPEC, I found that it is ok getting a general
idea of system performance, but as many have expressed it can be best
summed up by Doug's comment:

> Run *YOUR* code on the cpus in question, or a simplified version of it that
> is mathematically and structurally equivalent. Compare the results.

Perhaps any SPEC CPU engineers on the list may disagree?

On Sat, Oct 31, 2009 at 7:47 AM, Doug Hughes <doug@xxxxxxx> wrote:
> ntwrkd wrote:
>>
>> I'm trying to gather a baseline of performance testing the
>> CPU/OS/Compiler on a Linux system and I seemingly made the mistake of
>> purchasing the SPEC Benchmark. I don't know if there are any Intel
>> folk on the list but there seem to be a few problem with this
>> benchmark (granted that I know it is a respected standard):
>> - Lack of up-to date documentation
>> - Errors in documentation
>> - Requirements on a Fortran Compiler for integer testing
>> - No real way to translate results other than with a previous baseline
>> or another systems publicly published results.
>>
>>
>> I'm thinking that my own tests will be better off by using a more
>> real-world test, such as running a compute intensive process and
>> recording the results with sysstat.
>> This leaves me with the following questions:
>> - Does anyone use SPEC anymore?
>> - Are these shared concerns listed above with this benchmark?
>> - Any suggestions for alternatives?
>>
>>
>
> All the suggestions I have for benchmark involve getting a load that matches
> what you want to do in reality. If SPEC doesn't match your reality, it's of
> very little use. Even if you're doing pure floating point, there are a lot
> of different ways to do it. How many floating point operations can you have
> running in parallel on the different FP units (depends on your code)? SSE vs
> non-SSE offloading. is it 32 bit or 64 bit multiplication or is it division?
> All of these things make a big difference.
>
> Also, there are the memory effects, which cannot be ignored. Different
> architectures and chipsets handle memory copies VERY differently. We've
> noticed that a memory DIMM that is the same speed as all of the other dimms
> with measured with qcdstream --sse --mem (this is a good test by the way, I
> recommend it), runs 30% slower when run on a particular version of an MPI
> code test we use in HPC (inter and intra core copies for IPC).
>
> Run *YOUR* code on the cpus in question, or a simplified version of it that
> is mathematically and structurally equivalent. Compare the results.
>
> For our HPC results, the Nehalem architecture does much better at memory
> operations by a long shot then anything else for a certain HPC benchmark,
> but we have another app that runs as a file server where just more less
> expensive memory is a bigger win, so we went with AMD Istanbul with more
> cores and cheaper DDR2 dimms. Some jobs require the highest clock you can
> get because the marginal cost of the license allocation far exceeds that of
> the CPU, whereas others require higher throughput so having more less
> expenive, slightly slower CPUs is a bigger win.
>
> Measure cost per unit of speed of your code and decide whether more and
> slower or fewer and faster is your best bet.
>