Athlon 64 Vs. Pentium 4 article: On the Justification for Quake3 as a CPU Benchmark

[I sent this letter to Hardocp, Tomshardware, and Anandtech. Comments
welcome -- rms]

Hi, this is about your Athlon 64 Vs. Pentium 4 article, specifically the use
of Quake3 as a CPU benchmark when comparing AMD vs. Intel cpus, as shown on
this page

http://www.hardocp.com/article.html?art=NTI0LDU=

http://www.anandtech.com/cpu/showdoc.html?i=1884&p=13

http://www.tomshardware.com/cpu/2003...ngl_benchmarks

Let me say the article is great, no complaints there. I know it takes alot
of work to produce these articles.

Now, I see two reasons for using a game as a cpu benchmark:

1) It presents a fair (emphasis on the word 'fair') comparison of the
competing cpu architectures and scaling issues.

2) The game itself is of current interest to the community.

In your article you already concede 2). Quake3 itself is not relevant as a
game to anybody, and framerates in the 400's make the results meaningless to
gamers. Quake3-derived games are another matter, and are still popular and
certainly relevant. More on these later.

I believe there is strong evidence that Quake3 does not provide a fair
benchmark for comparing *modern* (AthlonXP and possibly Athlon64 as well)
AMD cpus vs Intel cpus. The reason being (and let me emphasize that I don't
know this as an verified fact, I'm going on what a couple of programmers
involved with helping AMD produce optimized game code have told me) that the
Quake3 cpu recognition code does not recognize the AthlonXP as an
SSE-capable cpu. Not only that, but the 3DNow code in Quake3 is apparently
non-functional for this cpu.

The politics and history behind this are interesting, but probably boil down
to the AthlonXP being released well after Quake3, and Carmack being rightly
uninterested in patching an old game.

If this is true, you are benchmarking two equally SSE-capable cpus against
each other, using a game engine which enables SSE for the Intel cpu and
*disables* SSE for the AMD cpu (apparently there's no simple way to force
SSE recognition either that works), for no valid reason, other than the game
is too old to know about the AMD cpu's capabilities. What would be even
worse is if this same recognition problem carries over to the Athlon64 (I
have no word on this) and to newer Quake3-based games.

Again, assuming this is true, it removes any rationale for using a 3-year
old game that: a) few people play, b) which gives ridiculously high scores,
and which c) unfairly handicaps AMD cpus; as a benchmark to be used
specifically in comparing AMD cpus vs their Intel competitors in articles
such as this one.

So. Here are the recommendations I, as an interested Hardocp/Anandtech/Toms
reader (and admitted AMD fan) am making to you and your site:

1) Investigate this matter further, and write an article discussing it. And
in particular discuss the relevance of this cpu issue to current
Quake3-based games. Assuming there is in fact an Intel bias to Quake3-based
benchmarking I think people would be very interested to learn about it.
Apparently the SSE issue does indeed carry over to later games.

1) Assuming there is a bias, discontinue using Quake3 as a cpu benchmark,
and especially discontinue it's use when comparing AMD vs Intel cpus. The
game will never be patched to fix this issue, and using 3rd party fixes
noone cares about is more or less pointless too. I'm referring to the dlls
on this page:

http://speedycpu.dyndns.org/opt/

This guy is one of the programmers I referred to earlier, and he tells me
the dlls do not enable SSE where it really matters anyway. The other was a
student working at AMD writing assembly 3DNow code. The best solution is
simply to retire this benchmark, just as Q1 and Q2 were retired.

rms

rms wrote:

[I sent this letter to Hardocp, Tomshardware, and Anandtech. Comments
welcome -- rms]

Hi, this is about your Athlon 64 Vs. Pentium 4 article, specifically the use
of Quake3 as a CPU benchmark when comparing AMD vs. Intel cpus, as shown on
this page

http://www.hardocp.com/article.html?art=NTI0LDU=

http://www.anandtech.com/cpu/showdoc.html?i=1884&p=13

http://www.tomshardware.com/cpu/2003...ngl_benchmarks

Let me say the article is great, no complaints there. I know it takes alot
of work to produce these articles.

Now, I see two reasons for using a game as a cpu benchmark:

1) It presents a fair (emphasis on the word 'fair') comparison of the
competing cpu architectures and scaling issues.

2) The game itself is of current interest to the community.

In your article you already concede 2). Quake3 itself is not relevant as a
game to anybody, and framerates in the 400's make the results meaningless to
gamers. Quake3-derived games are another matter, and are still popular and
certainly relevant. More on these later.

I believe there is strong evidence that Quake3 does not provide a fair
benchmark for comparing *modern* (AthlonXP and possibly Athlon64 as well)
AMD cpus vs Intel cpus. The reason being (and let me emphasize that I don't
know this as an verified fact, I'm going on what a couple of programmers
involved with helping AMD produce optimized game code have told me) that the
Quake3 cpu recognition code does not recognize the AthlonXP as an
SSE-capable cpu. Not only that, but the 3DNow code in Quake3 is apparently
non-functional for this cpu.

The politics and history behind this are interesting, but probably boil down
to the AthlonXP being released well after Quake3, and Carmack being rightly
uninterested in patching an old game.

If this is true, you are benchmarking two equally SSE-capable cpus against
each other, using a game engine which enables SSE for the Intel cpu and
*disables* SSE for the AMD cpu (apparently there's no simple way to force
SSE recognition either that works), for no valid reason, other than the game
is too old to know about the AMD cpu's capabilities. What would be even
worse is if this same recognition problem carries over to the Athlon64 (I
have no word on this) and to newer Quake3-based games.

Again, assuming this is true, it removes any rationale for using a 3-year
old game that: a) few people play, b) which gives ridiculously high scores,
and which c) unfairly handicaps AMD cpus; as a benchmark to be used
specifically in comparing AMD cpus vs their Intel competitors in articles
such as this one.

So. Here are the recommendations I, as an interested Hardocp/Anandtech/Toms
reader (and admitted AMD fan) am making to you and your site:

1) Investigate this matter further, and write an article discussing it. And
in particular discuss the relevance of this cpu issue to current
Quake3-based games. Assuming there is in fact an Intel bias to Quake3-based
benchmarking I think people would be very interested to learn about it.
Apparently the SSE issue does indeed carry over to later games.

1) Assuming there is a bias, discontinue using Quake3 as a cpu benchmark,
and especially discontinue it's use when comparing AMD vs Intel cpus. The
game will never be patched to fix this issue, and using 3rd party fixes
noone cares about is more or less pointless too. I'm referring to the dlls
on this page:

http://speedycpu.dyndns.org/opt/

This guy is one of the programmers I referred to earlier, and he tells me
the dlls do not enable SSE where it really matters anyway. The other was a
student working at AMD writing assembly 3DNow code. The best solution is
simply to retire this benchmark, just as Q1 and Q2 were retired.

rms



I would have to agree, Quake III as a benchmarking tool should have gorn
out the window when support for newer hardware failed to materialise.

Personaly, I think that all these people using Quake III for
benchmarking just like it for nostalgia reasons. It's like
installing Windows 3.1 on an Opteron to see what the hardware can do..

Irrelevent since Q3 engines are used in current games. It doesn't matter
whether the games are "fair" or not, although I think they should use a game
that DOES support SSE just to balance out reviews, because in the future
(what most people buy CPU's for), the Athlon 64 is going to be supported by
alot of games.

rms wrote:
[I sent this letter to Hardocp, Tomshardware, and Anandtech. Comments
welcome -- rms]

Hi, this is about your Athlon 64 Vs. Pentium 4 article, specifically the
use of Quake3 as a CPU benchmark when comparing AMD vs. Intel cpus, as
shown on this page

http://www.hardocp.com/article.html?art=NTI0LDU=

http://www.anandtech.com/cpu/showdoc.html?i=1884&p=13


http://www.tomshardware.com/cpu/2003...ngl_benchmarks

Let me say the article is great, no complaints there. I know it takes alot
of work to produce these articles.

Now, I see two reasons for using a game as a cpu benchmark:

1) It presents a fair (emphasis on the word 'fair') comparison of the
competing cpu architectures and scaling issues.

When playing Quake3.

2) The game itself is of current interest to the community.

Not really.

In your article you already concede 2). Quake3 itself is not relevant as a
game to anybody, and framerates in the 400's make the results meaningless
to gamers. Quake3-derived games are another matter, and are still popular
and certainly relevant. More on these later.

Indeed, but that depends on what newer features are added. If it's merely
polygon count and more lights in the sty;le of the original then yeah, but I
suspect far more is going on that in some games. As soon as you start using
features (especially newer GPU intensive ones) the old game engine no longer
reflects the current engine.

I believe there is strong evidence that Quake3 does not provide a fair
benchmark for comparing *modern* (AthlonXP and possibly Athlon64 as well)
AMD cpus vs Intel cpus. The reason being (and let me emphasize that I
don't know this as an verified fact, I'm going on what a couple of
programmers involved with helping AMD produce optimized game code have
told me) that the Quake3 cpu recognition code does not recognize the
AthlonXP as an SSE-capable cpu. Not only that, but the 3DNow code in
Quake3 is apparently non-functional for this cpu.

Essentially, yeah. I'm assuming the Athlon XP is being recognised as an
Athlon, which did not have SSE.

The politics and history behind this are interesting, but probably boil
down to the AthlonXP being released well after Quake3, and Carmack being
rightly uninterested in patching an old game.

Indeed.

If this is true, you are benchmarking two equally SSE-capable cpus against
each other, using a game engine which enables SSE for the Intel cpu and
*disables* SSE for the AMD cpu (apparently there's no simple way to force
SSE recognition either that works), for no valid reason, other than the
game is too old to know about the AMD cpu's capabilities.

This is only relevent if you are using Quake3 to benchmark the architectures
and attempting to use that comparison as a way of predicting performance
elsewhere.

What would be
even worse is if this same recognition problem carries over to the
Athlon64 (I have no word on this) and to newer Quake3-based games.

Indeed. If however it does carry over (They do not use SSE code on
AthlonXPs) then Quake3 may be considered as a suitable benchmark.

Again, assuming this is true, it removes any rationale for using a 3-year
old game that: a) few people play, b) which gives ridiculously high
scores, and which c) unfairly handicaps AMD cpus; as a benchmark to be
used specifically in comparing AMD cpus vs their Intel competitors in
articles such as this one.

Exactly. Using a 3 year old game to indicate the performance on any newer
features is useless, pointless and unnecessary.

So. Here are the recommendations I, as an interested
Hardocp/Anandtech/Toms reader (and admitted AMD fan) am making to you and
your site:

1) Investigate this matter further, and write an article discussing it.
And in particular discuss the relevance of this cpu issue to current
Quake3-based games. Assuming there is in fact an Intel bias to
Quake3-based benchmarking I think people would be very interested to
learn about it. Apparently the SSE issue does indeed carry over to later
games.

Good plan.

1) Assuming there is a bias, discontinue using Quake3 as a cpu benchmark,
and especially discontinue it's use when comparing AMD vs Intel cpus. The
game will never be patched to fix this issue, and using 3rd party fixes
noone cares about is more or less pointless too. I'm referring to the dlls
on this page:

http://speedycpu.dyndns.org/opt/

This guy is one of the programmers I referred to earlier, and he tells me
the dlls do not enable SSE where it really matters anyway. The other was a
student working at AMD writing assembly 3DNow code. The best solution is
simply to retire this benchmark, just as Q1 and Q2 were retired.


Since the game uses few features that newer games use, you cannot use it as
an indicator of performance in the newer games. I don;t think that this is
the reason they use it... I suspect the reason they include Q3 benchamrks is
because everyboady had Q3, many still have it, many know what their old
cards could do and are interested to find out what the new cards can do.
400FPS is insane and shows you just how far cards have come since the game
release, in terms of raw feature speed.

It's much harder to compare the performance in terms of image quality than
it is in terms of FPS. Including an old game that excludes much of the
eye-candy allows easy direct comparison of cards over 2-3 generations. You
also can't really make any comparisons of video cards from scores 2 years
ago as the increase on CPU speed, memory bandwidth etc are likely to make a
difference.

It all comes down to what you think the benchmark is telling you. If you
take the facts (Quake3 on hardware x) fine. If you try to say much else,
you need to be very careful and in most cases you simply cannot say very
much else than hardware x plays it at this speed.

Ben
--
I'm not just a number. To many, I'm known as a String...

magnulus wrote:
Irrelevent since Q3 engines are used in current games. It doesn't matter
whether the games are "fair" or not, although I think they should use a game
that DOES support SSE just to balance out reviews, because in the future
(what most people buy CPU's for), the Athlon 64 is going to be supported by
alot of games.




Well perhaps they should just use an updated engine, that includes
optimisations for both AMD and Intel CPU's.

It's not irrelivant, because those modern games that are built on the
Quake III engine today, surely include 'NEW' code for both Intel and AMD
CPU extensions. Quake III was never updated officialy for AMD CPU's and
that is why 'The Original Product "Quake III"' is flawed for
benchmarking, on todays PC's. It's like running Windows 3.1 on an
Opteron as I said earlier

"Minotaur" wrote in message
...
It's not irrelivant, because those modern games that are built on the
Quake III engine today, surely include 'NEW' code for both Intel and AMD
CPU extensions. Quake III was never updated officialy for AMD CPU's and
that is why 'The Original Product "Quake III"' is flawed for
benchmarking, on todays PC's. It's like running Windows 3.1 on an
Opteron as I said earlier


No, nobody working on Quake 3 engine games is working at modding Q3 like
that. You're talking about recompiling dll's and exe's. If one exists, I
don't know of one. Wolfenstein and Jedi Knight II both run better on Intel
chips becaues they still have the same "flaw" that Q3 has.

I just wish they'd stop benchmarking with Q3, period. Almost nobody has a
problem anymore running Q3 games, so any new card will theoretically run
just fine on them. The whole paradigm used to write the game is getting old
too, in the future texturing ability might very well be irrelevent next to
computational power. And it really doesn't matter if you are getting 300
FPS or 200 FPS, because you're eye won't see it anyways and VSync will cap
it at the refresh (and running with vsync off is just stupid).

"rms" wrote in message
m...
1) Assuming there is a bias, discontinue using Quake3 as a cpu benchmark,
and especially discontinue it's use when comparing AMD vs Intel cpus. The
game will never be patched to fix this issue, and using 3rd party fixes
noone cares about is more or less pointless too. I'm referring to the dlls
on this page:

http://speedycpu.dyndns.org/opt/

This guy is one of the programmers I referred to earlier, and he tells me
the dlls do not enable SSE where it really matters anyway. The other was a
student working at AMD writing assembly 3DNow code. The best solution is
simply to retire this benchmark, just as Q1 and Q2 were retired.

I'm not sure why 3DNow code should be slower than SSE on Athlon XP's. I mean
the only time it should be slower is when switching tasks and you have save
the FPU stack to memory, and that's only if you use the traditional Intel
way of saving MMX to memory through the FPU.

Looking back historically, prior to the time that MMX was introduced to the
public during the Pentium MMX and K6 days, AMD and Intel were working on
separate implementations of SIMD instructions. Just prior to introduction,
AMD licensed the MMX instruction set from Intel and quickly reworked their
own SIMD to use the Intel instructions. Intel's version made use of the FPU
registers directly, while AMD's version used separate registers than the FPU
stack but aliased them to the FPU stack. AMD retained some extra
instructions to allow for quick saves of MMX separate from the FPU save
mechanism; and it was even conceivable to be working on FPU and MMX
simultaneously on an AMD, because of AMD's different approach to MMX. This
carried on later to 3DNow, which is basically an extension to MMX.

Yousuf Khan