Opteron, the chip for supercomputers

Looks like the AMD Opteron is quickly becoming the favoured chip for
large supercomputing applications, only a few months after
introduction. Their latest announcement involves Los Alamos National
Lab in New Mexico, which will build two separate computing clusters
from these chips, one called Lightening and one called Orange.

This follows supercomputing announcements from a Japanese institution
using IBM Opteron servers. An announcement from Dawning of China. And
the first announcement of all was one from Cray for the Red Storm
computer at Sandia National Lab.

Lightening and Orange will use 2816 and 512 Opteron processors
respectively. The Orange cluster will also be connected over
Infiniband connections.

All of the supercomputers announced so far seem to use Linux as their
operating systems. Most of the supercomputers are the distributed
cluster-style supercomputer model. The Cray/Sandia Red Storm will be a
proprietary integrated solution.

Yousuf Khan

http://news.com.com/2100-1008_3-5063722.html?tag=cd_mh

On 14 Aug 2003 10:08:24 -0700, (Yousuf Khan) wrote:

Looks like the AMD Opteron is quickly becoming the favoured chip for
large supercomputing applications, only a few months after
introduction. Their latest announcement involves Los Alamos National
Lab in New Mexico, which will build two separate computing clusters
from these chips, one called Lightening and one called Orange.

This follows supercomputing announcements from a Japanese institution
using IBM Opteron servers. An announcement from Dawning of China. And
the first announcement of all was one from Cray for the Red Storm
computer at Sandia National Lab.

Lightening and Orange will use 2816 and 512 Opteron processors
respectively. The Orange cluster will also be connected over
Infiniband connections.

All of the supercomputers announced so far seem to use Linux as their
operating systems. Most of the supercomputers are the distributed
cluster-style supercomputer model. The Cray/Sandia Red Storm will be a
proprietary integrated solution.

Yousuf Khan

http://news.com.com/2100-1008_3-5063722.html?tag=cd_mh

IOW what they were originally going to use, did Intel loose an
Itanium/Xeon sale here? Just wondering if these little wins are hurting
Intel more then they are helping AMD. ;p

Ed

"Yousuf Khan" wrote in message
om...
Looks like the AMD Opteron is quickly becoming the favoured chip for
large supercomputing applications, only a few months after
introduction.

Indeed. When building supercomputers out of large numbers of
"desktop" CPUs, the limiting factor in speed is CPU
intercommunication. Providing such intercommunication is very
expensive, usually.

The reason for so many Opteron wins is that up to eight Opterons can
communicate very well without the addition of any additional glue
logic. And the bandwidth into memory is eight times the bandwidth
into memory of one microprocessor.

This means that Opteron supercomputers will be based on groups of 8
CPUs, while P4 supercomputers (if such exist at all) will be based on
single CPUs. This is why no P4-based supercomputers have been
announced recently.

Of course, supercomputers, even with thousands of CPUs each, are a
small niche market in the overall scheme of things. At this time,
Intel is selling far more P4s than AMD is selling Opterons. I imagine
that Intel is quite comfortable with the situation.

On Fri, 15 Aug 2003 03:24:22 GMT, "Felger Carbon"
wrote:

"Yousuf Khan" wrote in message
. com...
Looks like the AMD Opteron is quickly becoming the favoured chip for
large supercomputing applications, only a few months after
introduction.

Indeed. When building supercomputers out of large numbers of
"desktop" CPUs, the limiting factor in speed is CPU
intercommunication. Providing such intercommunication is very
expensive, usually.


Didn't have to be that way. Wouldn't be that way if Intel hadn't
abandoned Infiniband. As far as I can tell, Intel is now headed in a
completely different direction with "RDMA over ethernet."

The reason for so many Opteron wins is that up to eight Opterons can
communicate very well without the addition of any additional glue
logic. And the bandwidth into memory is eight times the bandwidth
into memory of one microprocessor.


Once you get outside the group of eight, you've still got the
expensive interconnect problem. Last time I looked, there was only a
four-way hyptransport switch available on the market, which means, at
the moment, you have to go to Infiniband or to a proprietary protocol.

This means that Opteron supercomputers will be based on groups of 8
CPUs, while P4 supercomputers (if such exist at all) will be based on
single CPUs. This is why no P4-based supercomputers have been
announced recently.


The real leverage being that you only have to have one expensive
interconnect port for 8 CPUS, and since the interconnect is much more
expensive per connection (gateway + switch port) than the CPU, the
Opteron has a huge cost advantage.

Of course, supercomputers, even with thousands of CPUs each, are a
small niche market in the overall scheme of things. At this time,
Intel is selling far more P4s than AMD is selling Opterons. I imagine
that Intel is quite comfortable with the situation.


Apparently. Intel wants to be selling network silicon, and the
networking silicon it intends to sell, apparently, is ethernet. When
Intel is ready to go with its own 10G ethernet silicon, it will
provide a high-bandwidth path to connect its CPU to the outside world,
and not a moment sooner.

The unfortunate reality for those wanting to build a supercomputer out
of commodity parts is that there won't be a commodity interconnect
until Intel is ready to put one on the market. That leaves a market
niche for Opteron, but it's not good news for the world of high-speed
computation.

RM

"Ed" wrote in message
...
IOW what they were originally going to use, did Intel loose an
Itanium/Xeon sale here? Just wondering if these little wins are hurting
Intel more then they are helping AMD. ;p

I don't know, I don't think they're affecting Intel directly, since some
supercomputers are still being made with their processors. Dell just
announced a supercomputing win somewhere using Xeons. And the Lightening
computer that will go into place at Los Alamos will be a backup
supercomputer to an HP-built Itanium unit. Let's see if I can't find the
links, somewhere.

Yousuf Khan

It probably hurts Intel more than it helps AMD. AFAIK, AMD doesn't
care about the "strategic" value of a LANL win, but Intel has a small
team devoted to it, and you can bet they feel the pain. Of course,
it's not even a blip to Intel revenues as a whole.

Intel used to have an entire Division (Supercomputing Systems
Division) that designed, built, sold, and gave direct customer
end-user support for parallel supercomputing systems. Intel Corporate
didn't even bat an eye when they unceremoniously shut them down (ca.
1996).

"Yousuf Khan" wrote:
"Ed" wrote in message
.. .
IOW what they were originally going to use, did Intel loose an
Itanium/Xeon sale here? Just wondering if these little wins are hurting
Intel more then they are helping AMD. ;p

I don't know, I don't think they're affecting Intel directly, since some
supercomputers are still being made with their processors. Dell just
announced a supercomputing win somewhere using Xeons. And the Lightening
computer that will go into place at Los Alamos will be a backup
supercomputer to an HP-built Itanium unit. Let's see if I can't find the
links, somewhere.

Yousuf Khan

On Fri, 15 Aug 2003 22:11:37 GMT, Free
wrote:

It probably hurts Intel more than it helps AMD. AFAIK, AMD doesn't
care about the "strategic" value of a LANL win, but Intel has a small
team devoted to it, and you can bet they feel the pain. Of course,
it's not even a blip to Intel revenues as a whole.


No, but it's a win they would have liked to have had for Itanium.
There's a really creepy book from Intel Press, the title of which I
have suppressed, that lays out their expected adoption curve for
Itanium, and technical users are supposed to be the earliest adopters.

Itanium *is* one hell of a floating point chip, but maybe they've been
so busy dealing with problems in the processor core that they haven't
had enough time to think about how they were going to hook it to
anything.

RM

On Fri, 15 Aug 2003 01:05:07 -0400, Robert Myers
wrote:

On Fri, 15 Aug 2003 03:24:22 GMT, "Felger Carbon"
wrote:

"Yousuf Khan" wrote in message
.com...
Looks like the AMD Opteron is quickly becoming the favoured chip for
large supercomputing applications, only a few months after
introduction.

Indeed. When building supercomputers out of large numbers of
"desktop" CPUs, the limiting factor in speed is CPU
intercommunication. Providing such intercommunication is very
expensive, usually.


Didn't have to be that way. Wouldn't be that way if Intel hadn't
abandoned Infiniband. As far as I can tell, Intel is now headed in a
completely different direction with "RDMA over ethernet."

I'm not sure that Intel had a lot of choice in the matter of Infiniband.
Was it Microsoft who "abandoned" it first... or was it the LAN mfrs... or
was it somebody else? A lot of people got to gloat over it anyway.

Rgds, George Macdonald

"Just because they're paranoid doesn't mean you're not psychotic" - Who, me??

On Mon, 18 Aug 2003 07:30:43 GMT,
(George Macdonald) wrote:

On Fri, 15 Aug 2003 01:05:07 -0400, Robert Myers
wrote:

On Fri, 15 Aug 2003 03:24:22 GMT, "Felger Carbon"
wrote:

"Yousuf Khan" wrote in message
e.com...
Looks like the AMD Opteron is quickly becoming the favoured chip for
large supercomputing applications, only a few months after
introduction.

Indeed. When building supercomputers out of large numbers of
"desktop" CPUs, the limiting factor in speed is CPU
intercommunication. Providing such intercommunication is very
expensive, usually.


Didn't have to be that way. Wouldn't be that way if Intel hadn't
abandoned Infiniband. As far as I can tell, Intel is now headed in a
completely different direction with "RDMA over ethernet."

I'm not sure that Intel had a lot of choice in the matter of Infiniband.
Was it Microsoft who "abandoned" it first... or was it the LAN mfrs... or
was it somebody else? A lot of people got to gloat over it anyway.


I tend to be skeptical of any theory that has Intel and Microsoft on
different pages, since they both have so much to lose by having a
wedge driven between them.

Intel wasn't even close to having market-competitive infiniband
silicon ready when all this came down, and I think it, not without
consultation with Microsoft, decided it wanted to change the rules.

I think that when Intel saw how much control over communications with
Intel processors they would lose, and along with it control over the
market for vast quantities of silicon, if infiniband performed a
similar function for Intel processors as hypertransport is performing
for AMD processors, they pulled the plug on it. They were overtaken
by competitors on their own technology, so they killed their own
technology. At least that's how I read it.

Intel has taken what seem like some very odd decisions, the net effect
of which is to leave rather high-performing single CPU's floating out
there without a cost-effective way of cooperating with one another.
The point that Felger Carbon made about the effective memory bandwidth
of an 8-way Opteron cluster is potentially devastating for Intel,
especially when you understand that, as compared to an Opteron
cluster, no current Intel processor can compete on latency *or*
bandwidth. That wouldn't be so much of a problem for Intel if they
gave OEM's a way to cobble together their own NUMA architecture
(Infiniband would have done that). Whatever are they thinking about?

RM

"David Schwartz" wrote in message
...
And the oddest part is that Intel's 64-bit plans don't seem to include
a
high-speed interconnect either. Does anyone really want a 4-CPU Itanium-2
box where they all share a single FSB to a single RAM port? A single
shared
6.4GB/s bus for memory and inter-CPU communication (4xItanium-2) versus 4
independent memory channels, 6 dedicated 6.4GB/s CPU interlinks, and 4
leftover 6.4GB/s ports to I/O (4xOpteron) is no contest.

I think the Itanium philosophy is firmly set in a previous era. When Itanium
was on the drawing boards (circa '95), the target competition was big-iron
64-bit RISC servers, with maybe upto 8 CPUs in a box (16 at the
highest-end). These CPUs of that era also usually had shared bus memory
architectures within a systemboard, with perhaps a simple proprietary
crossbar connection between system boards that the server manufacturer
probably spent considerable amounts of their own R&D money to develop. Each
system board likely never held more than 2 CPUs, while the system boards
themselves had expensive proprietary links connecting them. You could
isolate and minimize CPU contention issues within each system board.

Given that era of computing, the Itanium would be the indisputed king of the
hill of those processors. Of course since that time, all of those RISC CPUs
have added new generations with new tricks up their sleaves that add
immensely to both speed and redundancy.

Yousuf Khan