Help, 64 bit Linux only sees 3.5G of RAM on an A64 X2

Just got an Athlon 64 X2 system with 4G of RAM. The motherboard is an MSI
K8N Neo4 Platinum (NForce 4). I'm using 64 bit Fedora Core 3 with a
2.6.11.12 kernel compiled for the Athlon 64.

The BIOS has a switch called S/W memory hole remapping, when OFF the BIOS
sees 3.5G of RAM as does Linux. When ON the BIOS sees all 4G of RAM but
Linux sees only 3G. I've given a mem=4096M to GRUB,

title Fedora Core (2.6.11.12)
root (hd0,0)
kernel /boot/vmlinuz-2.6.11.12 ro root=LABEL=/1 mem=4096M rhgb quiet
initrd /boot/initrd-2.6.11.12.img

This doesn't help, is my grub config correct?

With the memory hole remapping ON the mem switch is required, if it's not
there I get a kernel panic.

Is there another switch that I need? Is there a kernel config switch that
I need to throw.

BTW this a custom kernel based on the .config file for the FC3 2.6.12
kernel. The current FC3 2.6.12 kernel panics on boot because they left our
the nv_sata driver. I built a standard 2.6.12.3 kernel but it panics
because of a problem with the Silicon Image RAID driver, the 2.6.11.12
kernel boots fine, however I had to install Nvidia's networking drivers to
get the ethernet controller to work, the kernel 2.6.11.12 drivers don't
seem to work although the drivers in the old 2.6.9 kernel that comes with
FC3 work fine.

General Schvantzkoph wrote:
Just got an Athlon 64 X2 system with 4G of RAM. The motherboard is an MSI
K8N Neo4 Platinum (NForce 4). I'm using 64 bit Fedora Core 3 with a
2.6.11.12 kernel compiled for the Athlon 64.

Cool!

Is there another switch that I need? Is there a kernel config switch that
I need to throw.

You bet there is!

BTW this a custom kernel based on the .config file for the FC3 2.6.12
kernel. The current FC3 2.6.12 kernel panics on boot because they left our

You're doing WHAT?!?!? You're using 2.6.11 with a config lifted from a
broken 2.6.12?!?!? What were you thinking?!?!?

You need to find an old config for 2.6.11 and make menuconfig from it,
customizing as necessary. If you can't find such a file, you should
start from scratch or even reinstall the 2.6.11 source package.

The config for 2.6.11 contains a line called CONFIG_NOHIGHMEM. In
menuconfig, this appears as a list of three options: off, 4GB, 64GB.
This is deprecated in 2.6.12 so you have essentially had it set to off
(because it was missing altogether).

You should never try to use a config file from one version with another
version. Do "make oldconfig" and it will walk you through the upgrade
from one kernel version to another. Unfortunately, "make oldconfig"
just tells you the name of the new features added to the upgraded
kernel; it doesn't explain what they mean. With a little slight of
hand, you can open a copy of the new kernel version's config file in
"make menuconfig" so that you'll have explanations of all the new
features. This will let you do "make oldconfig" in another terminal and
you'll know how to answer the new lines. When you're done, you will
have an upgraded config file appropriate for the new kernel version, but
retaining all of your old settings (if they still apply in the new version).

Since you have 4GB and that is an edge condition, you will want to test
a kernel built with CONFIG_NOHIGHMEM set at 4GB. If it still reports
less than 4GB, you may need to try setting it to 64GB. I know it sounds
like overkill but what can you do.

Well, actually there is something you can do. Upgrade to 2.6.12 and get
it working. I'm pretty sure this HIGHMEM stuff is handled automatically
by the more recent kernels.

On Thu, 28 Jul 2005 12:42:33 -0400, joe wrote:

General Schvantzkoph wrote:
Just got an Athlon 64 X2 system with 4G of RAM. The motherboard is an MSI
K8N Neo4 Platinum (NForce 4). I'm using 64 bit Fedora Core 3 with a
2.6.11.12 kernel compiled for the Athlon 64.

Cool!

Is there another switch that I need? Is there a kernel config switch that
I need to throw.

You bet there is!

BTW this a custom kernel based on the .config file for the FC3 2.6.12
kernel. The current FC3 2.6.12 kernel panics on boot because they left our

You're doing WHAT?!?!? You're using 2.6.11 with a config lifted from a
broken 2.6.12?!?!? What were you thinking?!?!?

You need to find an old config for 2.6.11 and make menuconfig from it,
customizing as necessary. If you can't find such a file, you should
start from scratch or even reinstall the 2.6.11 source package.

The config for 2.6.11 contains a line called CONFIG_NOHIGHMEM. In
menuconfig, this appears as a list of three options: off, 4GB, 64GB.
This is deprecated in 2.6.12 so you have essentially had it set to off
(because it was missing altogether).

You should never try to use a config file from one version with another
version. Do "make oldconfig" and it will walk you through the upgrade
from one kernel version to another. Unfortunately, "make oldconfig"
just tells you the name of the new features added to the upgraded
kernel; it doesn't explain what they mean. With a little slight of
hand, you can open a copy of the new kernel version's config file in
"make menuconfig" so that you'll have explanations of all the new
features. This will let you do "make oldconfig" in another terminal and
you'll know how to answer the new lines. When you're done, you will
have an upgraded config file appropriate for the new kernel version, but
retaining all of your old settings (if they still apply in the new version).

Since you have 4GB and that is an edge condition, you will want to test
a kernel built with CONFIG_NOHIGHMEM set at 4GB. If it still reports
less than 4GB, you may need to try setting it to 64GB. I know it sounds
like overkill but what can you do.

Well, actually there is something you can do. Upgrade to 2.6.12 and get
it working. I'm pretty sure this HIGHMEM stuff is handled automatically
by the more recent kernels.

I've built a 32 bit 2.6.11.12 kernel with highmem on, it does the same
thing as the 64 bit kernel. The 2.6.12 kernel has a broken driver for the
Silicon Image SATA raid chip which is on my motherboard. I've been trying
to get a 2.6.13.rc3 kernel to boot. The 2.6.13 kernel has a bunch of new
memory modes, flat and a couple of NUMA modes that have different
mechanisms for handling discontinuous memory. I tried the NUMA kernel,
wouldn't boot at all, it gets to the booting kernel line and freezes. I'm
now building a non-NUMA kernel with the flat memory model. The X2 is a
shared memory multi-processor, it's not NUMA, but architecturally it's
nearly identical to two Opterons so I'm surprised that the NUMA mode
doesn't boot but then again the BIOS is basically a uniprocessor BIOS so
maybe that explains it.

I have another data point. I ran Memtest86. It has a couple of ways of
sizing memory. The default mechanism is to ask the BIOS, in that mode it
sees 4G if the BIOS has memory hole remapping turned on, and 3.5G if
off. It also has a memory probe method. With the memory probe method it
gets exactly the same results as Linux, 3.5G if memory hole mapping is
OFF, 3G if it's ON. Memtest86 crashes if you run the test using the memory
probe method, it runs fine on all 4G with no errors if you use the BIOS
method of memory sizing.

On Wed, 27 Jul 2005 15:19:06 -0400, General Schvantzkoph wrote:

I got some e-mail from MSI, it's a design defect and they aren't going to
fix it. Anyone wanting 4G or more of RAM should pick another brand, the
MSI K8N Neo4 can only see 3.5G.

Here is the notice on their website.

Due to the South Bridge resource deployment, the system density will only
be detected up to 3+ GB (not full 4GB) when each DIMM is installed with an
1GB memory module

Hi, the chipset and cpu are probably reserving part of high memory, its
always been that way with 4G of memory.

doug
www.lynncomp.com

"General Schvantzkoph" wrote in message
news
Just got an Athlon 64 X2 system with 4G of RAM. The motherboard is an MSI
K8N Neo4 Platinum (NForce 4). I'm using 64 bit Fedora Core 3 with a
2.6.11.12 kernel compiled for the Athlon 64.

The BIOS has a switch called S/W memory hole remapping, when OFF the BIOS
sees 3.5G of RAM as does Linux. When ON the BIOS sees all 4G of RAM but
Linux sees only 3G. I've given a mem=4096M to GRUB,

title Fedora Core (2.6.11.12)
root (hd0,0)
kernel /boot/vmlinuz-2.6.11.12 ro root=LABEL=/1 mem=4096M rhgb quiet
initrd /boot/initrd-2.6.11.12.img

This doesn't help, is my grub config correct?

With the memory hole remapping ON the mem switch is required, if it's not
there I get a kernel panic.

Is there another switch that I need? Is there a kernel config switch that
I need to throw.

BTW this a custom kernel based on the .config file for the FC3 2.6.12
kernel. The current FC3 2.6.12 kernel panics on boot because they left our
the nv_sata driver. I built a standard 2.6.12.3 kernel but it panics
because of a problem with the Silicon Image RAID driver, the 2.6.11.12
kernel boots fine, however I had to install Nvidia's networking drivers to
get the ethernet controller to work, the kernel 2.6.11.12 drivers don't
seem to work although the drivers in the old 2.6.9 kernel that comes with
FC3 work fine.

On Sun, 31 Jul 2005 14:36:03 +0200, Hari Seldon wrote:

Il Sat, 30 Jul 2005 22:50:39 -0400, Doug Lynn ha scritto

Hi, the chipset and cpu are probably reserving part of high memory, its
always been that way with 4G of memory.

It's a 64 bit system, the statement doesn't make any sense.

Moreover, the OP discovered what went wrong. It's board's fault. Msi
admitted they did something wrong, that mb won't support more than
3.5GB.

Right. The AMD64 has a 64 bit Virtual Address and more importantly it has
a 40 bit physical address so it makes no sense to map the chipset into the
same physical space as the RAM. Obviously this is a solvable problem
because Opteron systems support 8G per processor. I'm guessing that this
is really a BIOS problem because the Award BIOS was developed for 32 bit
systems and they haven't gotten around to rewriting it for 64 bit chips.
The memory controller on the A64 is on the CPU not in the chipset so I
don't see what they could have done to actually block access to all 4G.
The RAM could be mapped anywhere in the 40 bit physical address space so
there is no excuse for mapping it into the same space as the bridge chip.

On Mon, 01 Aug 2005 00:16:48 +0200, Hari Seldon wrote:

Il Sun, 31 Jul 2005 08:54:48 -0400, General Schvantzkoph ha scritto


Hi, the chipset and cpu are probably reserving part of high memory, its
always been that way with 4G of memory.

It's a 64 bit system, the statement doesn't make any sense.

Moreover, the OP discovered what went wrong. It's board's fault. Msi
admitted they did something wrong, that mb won't support more than
3.5GB.

Right. The AMD64 has a 64 bit Virtual Address and more importantly it has
a 40 bit physical address so it makes no sense to map the chipset into the
same physical space as the RAM. Obviously this is a solvable problem
because Opteron systems support 8G per processor. I'm guessing that this
is really a BIOS problem because the Award BIOS was developed for 32 bit
systems and they haven't gotten around to rewriting it for 64 bit chips.
The memory controller on the A64 is on the CPU not in the chipset so I
don't see what they could have done to actually block access to all 4G.
The RAM could be mapped anywhere in the 40 bit physical address space so
there is no excuse for mapping it into the same space as the bridge chip.
Actually there still is a part of northbridge: the one necessary for the
agp/pci-e video interface is still there. Could *this* one be the culprit
somehow? Or the ACPI interface?
I'm not quite convinced about the bios part. Does Opteron have something
totally different from the usual award/ami/whatever stuff?

Server cards have had to deal with 4G for a long time because the
physical address space on a Xeon is 36 bits, so they must have solved the
problem there. So you would expect a server BIOS to be able to
accommodate a large address space. 4G on desktop systems is brand new
although it can't be a surprise to anyone designing a chipset or writing
a BIOS.

As for the physical address of each DIMM it can be anywhere in the 40 bit
space as long as the OS knows about it. The memory management code in the
OS takes care of mapping virtual memory to physical memory and as already
stated the MTU entries in the Athlon64 accommodate a 40 bit address. Even
the legacy modes accommodate 36 bits because that's what the X86s have
used for years. Also the memory controller is on the CPU, it's completely
divorced from the bridge chip so how is it possible for there to be
anything on the board that prevents the BIOS from recognizing all 4G and
more importantly passing the info to the OS.

The problem could be a Linux issue. The BIOS does see all 4G if you use
memory hole mapping. However in this mode Linux sees only 3G, it also
requires a mem= statement in order to boot. I've tried lots of kernels
including the latest release candidate for 2.6.12, they all behave
identically. I don't own a copy of XP so I have no idea if it can handle
memory hole mapping. I'd be curious to hear if anyone with a 4G XP system
has the same issue.

General Schvantzkoph wrote in
news
On Mon, 01 Aug 2005 00:16:48 +0200, Hari Seldon wrote:

Il Sun, 31 Jul 2005 08:54:48 -0400, General Schvantzkoph ha scritto


Hi, the chipset and cpu are probably reserving part of high
memory, its always been that way with 4G of memory.

It's a 64 bit system, the statement doesn't make any sense.

Moreover, the OP discovered what went wrong. It's board's fault.
Msi admitted they did something wrong, that mb won't support more
than 3.5GB.

Right. The AMD64 has a 64 bit Virtual Address and more importantly
it has a 40 bit physical address so it makes no sense to map the
chipset into the same physical space as the RAM. Obviously this is a
solvable problem because Opteron systems support 8G per processor.
I'm guessing that this is really a BIOS problem because the Award
BIOS was developed for 32 bit systems and they haven't gotten around
to rewriting it for 64 bit chips. The memory controller on the A64
is on the CPU not in the chipset so I don't see what they could have
done to actually block access to all 4G. The RAM could be mapped
anywhere in the 40 bit physical address space so there is no excuse
for mapping it into the same space as the bridge chip.
Actually there still is a part of northbridge: the one necessary for
the agp/pci-e video interface is still there. Could *this* one be the
culprit somehow? Or the ACPI interface?
I'm not quite convinced about the bios part. Does Opteron have
something totally different from the usual award/ami/whatever stuff?

Server cards have had to deal with 4G for a long time because the
physical address space on a Xeon is 36 bits, so they must have solved
the problem there. So you would expect a server BIOS to be able to
accommodate a large address space. 4G on desktop systems is brand new
although it can't be a surprise to anyone designing a chipset or
writing a BIOS.

As for the physical address of each DIMM it can be anywhere in the 40
bit space as long as the OS knows about it. The memory management code
in the OS takes care of mapping virtual memory to physical memory and
as already stated the MTU entries in the Athlon64 accommodate a 40 bit
address. Even the legacy modes accommodate 36 bits because that's what
the X86s have used for years. Also the memory controller is on the
CPU, it's completely divorced from the bridge chip so how is it
possible for there to be anything on the board that prevents the BIOS
from recognizing all 4G and more importantly passing the info to the
OS.

The problem could be a Linux issue. The BIOS does see all 4G if you
use memory hole mapping. However in this mode Linux sees only 3G, it
also requires a mem= statement in order to boot. I've tried lots of
kernels including the latest release candidate for 2.6.12, they all
behave identically. I don't own a copy of XP so I have no idea if it
can handle memory hole mapping. I'd be curious to hear if anyone with
a 4G XP system has the same issue.




I have the 4400+, 4GB PC3200, Abit Fatal1ty SLi, XP x64.
I see 4.00GB displayed in Windows properties.

Running XP 32-bit I only see 2.75GB of my 4GB. Is your Linux 32-bit?

On Mon, 01 Aug 2005 16:09:44 -0500, dead kitty wrote:

General Schvantzkoph wrote in
news
On Mon, 01 Aug 2005 00:16:48 +0200, Hari Seldon wrote:

Il Sun, 31 Jul 2005 08:54:48 -0400, General Schvantzkoph ha scritto


Hi, the chipset and cpu are probably reserving part of high
memory, its always been that way with 4G of memory.

It's a 64 bit system, the statement doesn't make any sense.

Moreover, the OP discovered what went wrong. It's board's fault.
Msi admitted they did something wrong, that mb won't support more
than 3.5GB.

Right. The AMD64 has a 64 bit Virtual Address and more importantly
it has a 40 bit physical address so it makes no sense to map the
chipset into the same physical space as the RAM. Obviously this is a
solvable problem because Opteron systems support 8G per processor.
I'm guessing that this is really a BIOS problem because the Award
BIOS was developed for 32 bit systems and they haven't gotten around
to rewriting it for 64 bit chips. The memory controller on the A64
is on the CPU not in the chipset so I don't see what they could have
done to actually block access to all 4G. The RAM could be mapped
anywhere in the 40 bit physical address space so there is no excuse
for mapping it into the same space as the bridge chip.
Actually there still is a part of northbridge: the one necessary for
the agp/pci-e video interface is still there. Could *this* one be the
culprit somehow? Or the ACPI interface?
I'm not quite convinced about the bios part. Does Opteron have
something totally different from the usual award/ami/whatever stuff?

Server cards have had to deal with 4G for a long time because the
physical address space on a Xeon is 36 bits, so they must have solved
the problem there. So you would expect a server BIOS to be able to
accommodate a large address space. 4G on desktop systems is brand new
although it can't be a surprise to anyone designing a chipset or
writing a BIOS.

As for the physical address of each DIMM it can be anywhere in the 40
bit space as long as the OS knows about it. The memory management code
in the OS takes care of mapping virtual memory to physical memory and
as already stated the MTU entries in the Athlon64 accommodate a 40 bit
address. Even the legacy modes accommodate 36 bits because that's what
the X86s have used for years. Also the memory controller is on the
CPU, it's completely divorced from the bridge chip so how is it
possible for there to be anything on the board that prevents the BIOS
from recognizing all 4G and more importantly passing the info to the
OS.

The problem could be a Linux issue. The BIOS does see all 4G if you
use memory hole mapping. However in this mode Linux sees only 3G, it
also requires a mem= statement in order to boot. I've tried lots of
kernels including the latest release candidate for 2.6.12, they all
behave identically. I don't own a copy of XP so I have no idea if it
can handle memory hole mapping. I'd be curious to hear if anyone with
a 4G XP system has the same issue.




I have the 4400+, 4GB PC3200, Abit Fatal1ty SLi, XP x64.
I see 4.00GB displayed in Windows properties.

Running XP 32-bit I only see 2.75GB of my 4GB. Is your Linux 32-bit?

I'm running 64 bit Linux, latest kernel. I've also tried a 32 bit large
mem kernel. Do you have memory hole mapping turned on in your BIOS?

General Schvantzkoph writes:


I have another data point. I ran Memtest86. It has a couple of ways of
sizing memory. The default mechanism is to ask the BIOS, in that mode it
sees 4G if the BIOS has memory hole remapping turned on, and 3.5G if
off. It also has a memory probe method. With the memory probe method it
gets exactly the same results as Linux, 3.5G if memory hole mapping is
OFF, 3G if it's ON. Memtest86 crashes if you run the test using the memory
probe method, it runs fine on all 4G with no errors if you use the BIOS
method of memory sizing.


"Memory Hole Mapping"

The northbridge on amd64 processors supports a concept called
memory hoisting. The Memory Mapped I/O space in 32-bit environments
is often in the upper 512MB of the 32-bit address space. When you
have 4GB of DRAM, the upper 512MB of the DRAM conflicts with the
MMIO range, so the BIOS can set a bit in the northbridge configuration
that will "hoist" the last 512MB of DRAM and re-base it at 4GB (which
makes the highest physical DRAM address (or Top Of Memory in amd-ese)
4.5GB, with a hole (occupied by the MMIO space) from 3.5GB to 4.0GB).

Memtest will crash using the probe method because it is writing to
MMIO space (i.e pci device configuration and operation registers) between
3.5GB and 4.0GB.

64-bit linux only seeing 3.0GB when hoisting is enabled sounds like
either a bug in the linux sizing code, or a bug in the BIOS when it exposes
the memory configuration through ACPI and/or E820.

A 32-bit kernel would need to use PAE to access the high 512MB when
hoisting is enabled - config for 64G should enable PAE support.

The early dmesg(1m) output should show the values of the E820 BIOS memory
map that the kernel operates from, can you post that?

scott