Enabling ECC memory in Linux without BIOS support

I build computers for reliability and low(er) power; I've been doing so long before the somewhat recent green kick. In particular, I want ECC memory, and a lot of it, and a good power supply. I don't care about CPU speed or the video card. I like to leave my linux box up for months, even a year. And ECC memory is necessary for this. I used to have to buy specific chipsets for Intel processors, but in the past 3 years I have chosen AMD processors solely largely because they support ECC. The AMD Athlon CPUs have a built-in memory controller and it has supported unbuffered ECC RAM all this time. So any motherboard is largely fine... or so I thought.

I finally assembled my new system with a Phenom II X4 and a lovely Gigabyte GA-MA785GM-US2H MB with nice copper wiring and good capacitors. I chose this M/B since it has the latest AMD 785G video, and it supports DDR2 which was cheaper than DDR3 when buying ECC RAM (I've been buying Kingston ECC RAM, and for this system it was 8G of KVR533D2E4K2/4G since it was amazingly cheap.). But this stupid mother ***** does not support ECC in the BIOS, which is a bit odd as the CPU talks to the memory directly. Apparently Gigabyte does not provide for this in their BIOS settings http://forums.amd.com/forum/messageview.cfm?catid=21&threadid=123883, see the response from Gigabyte.

I had the following fails:

1. Running memtest86+ v4.10, the memory is not recognized as ECC. Argh.
2. Flashing the latest BIOS for this M/B did not help. Argh.
3. I tried adding the kernel boot parameter to GRUB ecc_enable_override, but that did not work. Argh.

To make a long story short, the solution is that you can force the Linux kernel module that enables ECC to load via:

% modprobe -v amd64_edac_mod ecc_enable_override=1

To verify that the ECC was turned on run

% dmesg | grep -i edac

And you should see something like:

[ 658.399849] EDAC amd64_edac: Ver: 3.3.0 Sep 19 2010
[ 658.400082] EDAC amd64: This node reports that Memory ECC is currently disabled, set F3x44[22] (0000:00:18.3).
[ 658.400102] EDAC amd64: Forcing ECC checking on!
[ 658.400198] EDAC MC: F10h CPU detected
[ 658.400230] EDAC MC: DCT0 chip selects:
[ 658.400236] EDAC MC: 0: 1024MB 1: 1024MB
[ 658.400242] EDAC MC: 2: 1024MB 3: 1024MB
[ 658.400246] EDAC MC: 4: 0MB 5: 0MB
[ 658.400251] EDAC MC: 6: 0MB 7: 0MB
[ 658.400254] EDAC MC: DCT1 chip selects:
[ 658.400259] EDAC MC: 0: 1024MB 1: 1024MB
[ 658.400263] EDAC MC: 2: 1024MB 3: 1024MB
[ 658.400267] EDAC MC: 4: 0MB 5: 0MB
[ 658.400271] EDAC MC: 6: 0MB 7: 0MB
[ 658.400333] EDAC amd64: This node reports that DRAM ECC is currently Disabled; ENABLING now
[ 658.400339] EDAC amd64: Hardware accepted DRAM ECC Enable
[ 658.401685] EDAC MC0: Giving out device to 'amd64_edac' 'Family 10h': DEV 0000:00:18.2
[ 658.401731] EDAC PCI0: Giving out device to module 'amd64_edac' controller 'EDAC PCI controller': DEV '0000:00:18.2' (POLLED)

The Linux modules that deal with ECC are labelled "enad". Some other commands you can run, are lsmod (to verify the amd enad module is loaded) and dmidecode --type memory (to see how the BIOS is reporting memory, which shows non-ECC RAM in this particular case).

What shocks me about the falling prices of tech

Technology gets better and cheaper over time. The advances in computer speed, memory size, hard drive size, networking speed are revolutionary but to some degree expected. While it is still a bit hard to stomach, 32 core machines and 5TB hard drives will arrive.

But the cutting edge stuff like a mobile phone that can surf the web and run for 3+ hours still costs decent money.

What shocks me is you can get a low-end CPU and low-end M/B (motherboard) for $29 ($39 with $10 rebate at Fry's). These days, a low end M/B come standard with audio, networking and video. It supports SATA and USB. The CPU and the integrated video is better, especially cooler, than the best you could buy in 2002 for any amount of money. It's akin to saying wait 8 years and the phone they give away for $5 will be better and smaller than the top end iPhone or Android phone.

The kicker is the price. At $29 this means the M/B and the CPU collectively cost less than this. I can imagine the CPU costs say $8, after all it is just a piece of silicon and some packaging (yes, I'm greatly oversimplifying). But a M/B is a big (from a tech standpoint) item of lots of items. There are at least 100 parts that need to be assembled. The chipset alone is as complicated as the CPU. At this price level there are no margins. Which means this level of technology is so well understood anybody can do it for peanuts.

Amazing.

The passing of the torch from laptops to cellphones

For over a decade, people have been predicting the coming big thing would be a mobile computer you have with you all the time. It is obvious, the smartphone is this device. The Apple iPhone was the breakthrough device.

What became obvious to me a year ago (shame on me for not blogging about it then), was that this shift has already happened. Yep it will be a completely done deal in another year.

The existing competition for a mobile computer is well a mobile "computer", aka the laptop. And while laptops still outsell smart phones, the epiphany was realizing that a high-end smart phones cost more than the median laptop. And people are lining up in droves to buy them.

The indicators are various.

1. Price is the ultimate indicator of worth. Including the cost of a data plan, a smart phone is much more expensive than a laptop over 3 years.
2. The developer mindset has changed. The coolest apps to write are for mobile devices. Traditional desktop devices are passe.
3. My own behaviour. I check my email as much on my high-end Android phone as much as on my computer. It is so easy.
   

The end of public boredom as we know it

We've all passed the time in a line for the DMV, waiting for a bus/train, or even sitting solo at lunch. But this semi-lonely, sem-bored, semi-waste of times has ended. Thanks to the smartphone.

Now you can look busy, intently surfing the web, checking your idle calendar and continually verifying you still don't have any new email.

Be SMART, get advance warning that your hard drive is starting to fail

All modern hard drives support S.M.A.R.T. in which the hard drive (HD) runs self tests and reports on the status of internal metrics (seek errors, block errors, temperature, being dropped, etc). If a HD starts to do poorly in some of the internal metrics, it's one sign the drive might be starting to fail. Unfortunately predicting when the drive will actually failure is impossible right now, akin to predicting earthquakes. But it does give you some advance warning to redouble the backup efforts or to switch out that drive if it holds valuable data.

There exist many tools to access the SMART interface depending on your OS. The key thing is that the HD itself does all the work. The tools just provide access to the HD.

The beauty of SMART is that

• you can run the SMART tests on a running system on a mounted HD in use! Since the HD does the SMART scheduling internally, it can figure out when and how to continue with it's own test while getting normal OS requests.
• it's pretty easy to do a manual scan. For Linux, download the smartctl software and then start issuing commands. There's a lot of documentation on the web.
• with a bit of work you can setup regular SMART background tests and have alerts sent when a HD falls below the built in "failure" thresholds.

The main drawbacks are

• a thorough "long" test takes several hours. On a 1.5TB drive, it takes 4-8 hours.
• you need to have the HD hooked up directly via its native interface, namely PATA or SATA. If a drive is hooked up via a USB enclosure, smartctl will claim the drive does not support SMART. Ugh. (I assume eSATA will work, as this is fundamentally SATA).

I'm not going to repeat all the information out there. But here's some decent links.

• A great article on SMART geared toward Linux.
• The wikipedia article on SMART, with many more links.

System tools in Linux and Unix

These keep changing over the years, but here's a quick run down

Hardware related:

ethtool, mii-tool: view / set ethernet device settings and the MII settings

lspci: show PCI devices

lshw: show hardware

lscpu, lshal, lsusb: show CPU, HAL and USB information. Note the USB information is often the USB controllers on your M/B, not the devices actually hooked up.

Linux Kernel

lsmod , rmmod, modprobe: manage Linux kernel modules (which how most device drivers are loaded)

lsof, fuser: show processes using files/directories/file systems. Very useful if you want to unmount a disk, say a removable drive, and you are told the device is still in use. Note that if you run the samba SMB/CIFS file server, it will continue to use a filesystem if at anytime in the past a client used that file system. I restart samba in this case.

mount, umount: mount and unmount filesystems. A trick I often use is to remount a filesystem as read-only or read-write via

mount -o remount,ro /dev/... 

or 

mount -o remount,rw /dev/...

which is much faster than unmounting and remounting.

Changing the DHCP (or any other service) config file in Ubuntu 9.10

This applies to Ubuntu 9.10 Karmic Koala. I did not have this problem in 8.10 Ibex.
I keep my system config files such as for dhcpd (DHCP daemon) in a separate directory with all my system tweaks. I then set up a symbolic link from /etc/dhcpd3/dhcpd.conf --> my config.

But when I started the dhcpd server via /etc/init.d/dhcpd3-server restart
I kept getting the error message "Can't open /etc/dhcp3/dhcpd.conf: Permission denied"

Moving the config to /tmp also did not work, as tested using the -t flag, via
dhcpd3 -t -cf /tmp/dhcpd.conf

Finally, after some hunting, aka Googling, I found the problem was Apparmor, which restricts the files and directories that various services can use. So I added the following line to /etc/apparmor.d/usr.sbin.dhcpd3, where the dir should be the physical dir you get from cd -P in bash, namely remove all symlinks:

...
/dir/holding/my/dhcpd/config/** r
...

I then restarted apparmor via /etc/init.d/apparmor restart

And now I'm golden.