{"id":324,"date":"2008-09-09T19:46:13","date_gmt":"2008-09-09T23:46:13","guid":{"rendered":"http:\/\/nuxx.net\/blog\/?p=324"},"modified":"2008-09-12T20:26:49","modified_gmt":"2008-09-13T00:26:49","slug":"cpu-2-machine-check-exception-4-bank-4-f61c2001ba080813","status":"publish","type":"post","link":"https:\/\/nuxx.net\/blog\/2008\/09\/09\/cpu-2-machine-check-exception-4-bank-4-f61c2001ba080813\/","title":{"rendered":"CPU 2: Machine Check Exception: 4 Bank 4: f61c2001ba080813"},"content":{"rendered":"

<\/p>\n\n\n
\"A<\/a><\/td>\n<\/tr>\n<\/table>\n

<\/center><\/p>\n

In testing my server banstyle.nuxx.net<\/a> has had its first real set of errors \/ failures. This is a good thing.<\/p>\n

First, last night I started getting SMART warnings about bad blocks on ad6<\/tt>, which is the second hard drive. So today I just went ahead and ordered up a pair of ST3500320AS<\/a> 500GB disks and a 3ware 8006-2LP<\/a>, the same as is used in my current server.<\/p>\n

Note the sdb<\/tt> errors, which are consistent with the other errors I’d been seeing indicating a bad block on the second hard disk.<\/p>\n

Second, I came home today and found my server hung while running Breakin<\/a>, displaying the error CPU 2: Machine Check Exception: 4 Bank 4: f61c2001ba080813 TSC 2561d00c4ef7 ADDR ce19fd00<\/tt>. So, at least I’ve got some place to look for what else might be the issue.<\/p>\n

This error was decoded by AMD’s the AMD Machine Check Analysis Tool (MCAT)<\/a>. Since the machine contains Opteron 885<\/a> CPUs which are in the AMD K10<\/a> family, the \/gh<\/tt> flag is used:<\/p>\n

C:\\Program Files\\AMD\\MCat>mcat \/gh \/cmd 4 0xf41c2000ba080a13 0xce19fd00 0x2561d00c4ef7<\/tt>
\nProcessor Number  : 0<\/tt>
\nBank Number       : 4<\/tt>
\nTime Stamp    (0x): 00000000 00000000<\/tt>
\nError Status  (0x): F41C2000 BA080A13<\/tt>
\nError Address (0x): 00000000 CE19FD00<\/tt>
\nError Misc    (0x): 00002561 D00C4EF7<\/tt>
\nStatus Bit Decode :<\/tt>
\n   Uncorrectable ECC error<\/tt>
\n   Error address valid<\/tt>
\n   Error enable<\/tt>
\n   Error uncorrected<\/tt>
\n   Error overflow<\/tt>
\n   Error valid<\/tt>
\nError Code    (0x): 0A13<\/tt>
\n   Error Type - Bus<\/tt>
\n   Participation Processor (PP) - Local node responded to the request (RES)<\/tt>
\n   Timeout (T) - Request did not time out<\/tt>
\n   Memory Transaction Type (RRRR) - Generic read (RD)<\/tt>
\n   Memory or IO (II) - Memory Access (MEM)<\/tt>
\n   Cache Level (LL) - Generic, includes L3 cache (LG)<\/tt>
\nBank 4 North Bridge Errors:<\/tt>
\n   ECC Error - DRAM ECC error detected in the NB.<\/tt>
\n   Error address at 3297 MB rage<\/tt>
\n   Syndrome  (0x): BA38<\/tt>
\n   Address decode: 00000000CE19FD00<\/tt>
\n      Node ID: 5<\/tt>
\n      Channel Select: 0<\/tt>
\n      Chip Select: 0<\/tt><\/p>\n

I then swapped the first and fourth DIMMs which are connected to the first CPU, then received this error after a few more hours of running Breakin:<\/p>\n

C:\\Program Files\\AMD\\MCat>mcat \/gh \/cmd 4 0xf41c2000ba080a13 0x8589bd00 0x147c1f963903<\/tt>
\nProcessor Number  : 0<\/tt>
\nBank Number       : 4<\/tt>
\nTime Stamp    (0x): 00000000 00000000<\/tt>
\nError Status  (0x): F41C2000 BA080A13<\/tt>
\nError Address (0x): 00000000 8589BD00<\/tt>
\nError Misc    (0x): 0000147C 1F963903<\/tt>
\nStatus Bit Decode :<\/tt>
\n   Uncorrectable ECC error<\/tt>
\n   Error address valid<\/tt>
\n   Error enable<\/tt>
\n   Error uncorrected<\/tt>
\n   Error overflow<\/tt>
\n   Error valid<\/tt>
\nError Code    (0x): 0A13<\/tt>
\n   Error Type - Bus<\/tt>
\n   Participation Processor (PP) - Local node responded to the request (RES)<\/tt>
\n   Timeout (T) - Request did not time out<\/tt>
\n   Memory Transaction Type (RRRR) - Generic read (RD)<\/tt>
\n   Memory or IO (II) - Memory Access (MEM)<\/tt>
\n   Cache Level (LL) - Generic, includes L3 cache (LG)<\/tt>
\nBank 4 North Bridge Errors:<\/tt>
\n   ECC Error - DRAM ECC error detected in the NB.<\/tt>
\n   Error address at 2136 MB rage<\/tt>
\n   Syndrome  (0x): BA38<\/tt>
\n   Address decode: 000000008589BD00<\/tt>
\n      Node ID: 5<\/tt>
\n      Channel Select: 0<\/tt>
\n      Chip Select: 0<\/tt><\/p>\n

I’ve got interleaving enabled and I suspect that with this they are being used in some sort of balanced fashion, round-robin or something like that. With this, I would have suspected that the error would then have moved from the upper 2048 MB of that CPU’s RAM to the lower 2048 MB, but it didn’t. It’s very likely that I’m wrong in this thinking, though.<\/p>\n

So, tomorrow morning before work I’ll turn off interleaving, run the test again, and try to narrow it down to a single DIMM. If I can do that I’ll then move it over to the other CPU and see if it moves. Hopefully the error will appear in a manner which I can more easily narrow down to a single DIMM.<\/p>\n

This third MCE, with Bank Interleaving set to Auto and Node Interleaving set to Disabled<\/a> has resulted in the following:<\/p>\n

C:\\Program Files\\AMD\\MCat>mcat \/gh \/cmd 4 0xf61c2001ba080813 0x1b7ae9d00 0x4e7b5766b77b<\/tt>
\nProcessor Number  : 0<\/tt>
\nBank Number       : 4<\/tt>
\nTime Stamp    (0x): 00000000 00000000<\/tt>
\nError Status  (0x): F61C2001 BA080813<\/tt>
\nError Address (0x): 00000001 B7AE9D00<\/tt>
\nError Misc    (0x): 00004E7B 5766B77B<\/tt>
\nStatus Bit Decode :<\/tt>
\n   Error associated with CPU core 0<\/tt>
\n   Uncorrectable ECC error<\/tt>
\n   Processor context corrupt<\/tt>
\n   Error address valid<\/tt>
\n   Error enable<\/tt>
\n   Error uncorrected<\/tt>
\n   Error overflow<\/tt>
\n   Error valid<\/tt>
\nError Code    (0x): 0813<\/tt>
\n   Error Type - Bus<\/tt>
\n   Participation Processor (PP) - Local node originated the request (SRC)<\/tt>
\n   Timeout (T) - Request did not time out<\/tt>
\n   Memory Transaction Type (RRRR) - Generic read (RD)<\/tt>
\n   Memory or IO (II) - Memory Access (MEM)<\/tt>
\n   Cache Level (LL) - Generic, includes L3 cache (LG)<\/tt>
\nBank 4 North Bridge Errors:<\/tt>
\n   ECC Error - DRAM ECC error detected in the NB.<\/tt>
\n   Error address at 7034 MB rage<\/tt>
\n   Syndrome  (0x): BA38<\/tt><\/p>\n

Tomorrow I shall try to duplicate this result and memory location. If I can, I’ll begin shuffling parts around to see if the failure address moves. If I can’t, I’ll investigate the CPU.<\/p>\n

This fourth MCE<\/a> was generated with the same settings as the third, except with different disks (Seagate 500GB) and a new RAID controller (3ware 8006-2LP in JBOD mode):<\/p>\n

C:\\Program Files\\AMD\\MCat>mcat \/gh \/cmd 4 0xf41c2000ba080a13 0x1250f2d00 0xe0eb9e12d8e<\/tt>
\nProcessor Number  : 0<\/tt>
\nBank Number       : 4<\/tt>
\nTime Stamp    (0x): 00000000 00000000<\/tt>
\nError Status  (0x): F41C2000 BA080A13<\/tt>
\nError Address (0x): 00000001 250F2D00<\/tt>
\nError Misc    (0x): 00000E0E B9E12D8E<\/tt>
\nStatus Bit Decode :<\/tt>
\n   Uncorrectable ECC error<\/tt>
\n   Error address valid<\/tt>
\n   Error enable<\/tt>
\n   Error uncorrected<\/tt>
\n   Error overflow<\/tt>
\n   Error valid<\/tt>
\nError Code    (0x): 0A13<\/tt>
\n   Error Type - Bus<\/tt>
\n   Participation Processor (PP) - Local node responded to the request (RES)<\/tt>
\n   Timeout (T) - Request did not time out<\/tt>
\n   Memory Transaction Type (RRRR) - Generic read (RD)<\/tt>
\n   Memory or IO (II) - Memory Access (MEM)<\/tt>
\n   Cache Level (LL) - Generic, includes L3 cache (LG)<\/tt>
\nBank 4 North Bridge Errors:<\/tt>
\n   ECC Error - DRAM ECC error detected in the NB.<\/tt>
\n   Error address at 4688 MB rage<\/tt>
\n   Syndrome  (0x): BA38<\/tt><\/p>\n

This fifth MCE<\/a> was generated with Bank and Node Interleaving both disabled:<\/p>\n

C:\\Program Files\\AMD\\MCat>mcat \/gh \/cmd 4 0xf41c2000ba080a13 0x1e224fd00 0x48e38a5ecb7<\/tt>
\nProcessor Number  : 0<\/tt>
\nBank Number       : 4<\/tt>
\nTime Stamp    (0x): 00000000 00000000<\/tt>
\nError Status  (0x): F41C2000 BA080A13<\/tt>
\nError Address (0x): 00000001 E224FD00<\/tt>
\nError Misc    (0x): 0000048E 38A5ECB7<\/tt>
\nStatus Bit Decode :<\/tt>
\n   Uncorrectable ECC error<\/tt>
\n   Error address valid<\/tt>
\n   Error enable<\/tt>
\n   Error uncorrected<\/tt>
\n   Error overflow<\/tt>
\n   Error valid<\/tt>
\nError Code    (0x): 0A13<\/tt>
\n   Error Type - Bus<\/tt>
\n   Participation Processor (PP) - Local node responded to the request (RES)<\/tt>
\n   Timeout (T) - Request did not time out<\/tt>
\n   Memory Transaction Type (RRRR) - Generic read (RD)<\/tt>
\n   Memory or IO (II) - Memory Access (MEM)<\/tt>
\n   Cache Level (LL) - Generic, includes L3 cache (LG)<\/tt>
\nBank 4 North Bridge Errors:<\/tt>
\n   ECC Error - DRAM ECC error detected in the NB.<\/tt>
\n   Error address at 7714 MB rage<\/tt>
\n   Syndrome  (0x): BA38<\/tt><\/p>\n

The sixth MCE<\/a> was generated under the same conditions as #5, with both Node and Bank Interleaving disabled:
\nC:\\Program Files\\AMD\\MCat>mcat \/gh \/cmd 4 0xf61c2001ba080813 0x1a78d9d00 0x42f9116690d7<\/tt>
\nProcessor Number  : 0<\/tt>
\nBank Number       : 4<\/tt>
\nTime Stamp    (0x): 00000000 00000000<\/tt>
\nError Status  (0x): F61C2001 BA080813<\/tt>
\nError Address (0x): 00000001 A78D9D00<\/tt>
\nError Misc    (0x): 000042F9 116690D7<\/tt>
\nStatus Bit Decode :<\/tt>
\n   Error associated with CPU core 0<\/tt>
\n   Uncorrectable ECC error<\/tt>
\n   Processor context corrupt<\/tt>
\n   Error address valid<\/tt>
\n   Error enable<\/tt>
\n   Error uncorrected<\/tt>
\n   Error overflow<\/tt>
\n   Error valid<\/tt>
\nError Code    (0x): 0813<\/tt>
\n   Error Type - Bus<\/tt>
\n   Participation Processor (PP) - Local node originated the request (SRC)<\/tt>
\n   Timeout (T) - Request did not time out<\/tt>
\n   Memory Transaction Type (RRRR) - Generic read (RD)<\/tt>
\n   Memory or IO (II) - Memory Access (MEM)<\/tt>
\n   Cache Level (LL) - Generic, includes L3 cache (LG)<\/tt>
\nBank 4 North Bridge Errors:<\/tt>
\n   ECC Error - DRAM ECC error detected in the NB.<\/tt>
\n   Error address at 6776 MB rage<\/tt>
\n   Syndrome  (0x): BA38<\/tt><\/p>\n","protected":false},"excerpt":{"rendered":"

In testing my server banstyle.nuxx.net has had its first real set of errors \/ failures. This is a good thing. First, last night I started<\/p>\n

Continue readingCPU 2: Machine Check Exception: 4 Bank 4: f61c2001ba080813<\/span><\/a><\/div>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[],"class_list":["post-324","post","type-post","status-publish","format-standard","hentry","category-computers","entry"],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/nuxx.net\/blog\/wp-json\/wp\/v2\/posts\/324","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nuxx.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nuxx.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nuxx.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/nuxx.net\/blog\/wp-json\/wp\/v2\/comments?post=324"}],"version-history":[{"count":19,"href":"https:\/\/nuxx.net\/blog\/wp-json\/wp\/v2\/posts\/324\/revisions"}],"predecessor-version":[{"id":327,"href":"https:\/\/nuxx.net\/blog\/wp-json\/wp\/v2\/posts\/324\/revisions\/327"}],"wp:attachment":[{"href":"https:\/\/nuxx.net\/blog\/wp-json\/wp\/v2\/media?parent=324"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nuxx.net\/blog\/wp-json\/wp\/v2\/categories?post=324"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nuxx.net\/blog\/wp-json\/wp\/v2\/tags?post=324"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}