blob: 5029885245192cd6cdc1b1aeeb98b8b71b3fcabd [file] [log] [blame]
Alan Coxda9bb1d2006-01-18 17:44:13 -08001EDAC - Error Detection And Correction
Borislav Petkov043b4312015-06-19 11:47:17 +02002=====================================
Doug Thompson87f24c32007-07-19 01:50:34 -07003
Borislav Petkove34217c2015-11-26 14:12:56 +01004"bluesmoke" was the name for this device driver when it
5was "out-of-tree" and maintained at sourceforge.net -
6bluesmoke.sourceforge.net. That site is mostly archaic now and can be
7used only for historical purposes.
8
9When the subsystem was pushed into 2.6.16 for the first time, it was
10renamed to 'EDAC'.
Doug Thompson87f24c32007-07-19 01:50:34 -070011
Borislav Petkov043b4312015-06-19 11:47:17 +020012PURPOSE
13-------
Doug Thompson87f24c32007-07-19 01:50:34 -070014
Borislav Petkov043b4312015-06-19 11:47:17 +020015The 'edac' kernel module's goal is to detect and report hardware errors
16that occur within the computer system running under linux.
Doug Thompson87f24c32007-07-19 01:50:34 -070017
18MEMORY
Borislav Petkov043b4312015-06-19 11:47:17 +020019------
Doug Thompson87f24c32007-07-19 01:50:34 -070020
Borislav Petkov043b4312015-06-19 11:47:17 +020021Memory Correctable Errors (CE) and Uncorrectable Errors (UE) are the
22primary errors being harvested. These types of errors are harvested by
23the 'edac_mc' device.
Alan Coxda9bb1d2006-01-18 17:44:13 -080024
25Detecting CE events, then harvesting those events and reporting them,
Borislav Petkov043b4312015-06-19 11:47:17 +020026*can* but must not necessarily be a predictor of future UE events. With
27CE events only, the system can and will continue to operate as no data
28has been damaged yet.
Alan Coxda9bb1d2006-01-18 17:44:13 -080029
Borislav Petkov043b4312015-06-19 11:47:17 +020030However, preventive maintenance and proactive part replacement of memory
31DIMMs exhibiting CEs can reduce the likelihood of the dreaded UE events
32and system panics.
33
34OTHER HARDWARE ELEMENTS
35-----------------------
Doug Thompson87f24c32007-07-19 01:50:34 -070036
37A new feature for EDAC, the edac_device class of device, was added in
38the 2.6.23 version of the kernel.
39
40This new device type allows for non-memory type of ECC hardware detectors
41to have their states harvested and presented to userspace via the sysfs
42interface.
43
Borislav Petkov043b4312015-06-19 11:47:17 +020044Some architectures have ECC detectors for L1, L2 and L3 caches,
45along with DMA engines, fabric switches, main data path switches,
46interconnections, and various other hardware data paths. If the hardware
47reports it, then a edac_device device probably can be constructed to
48harvest and present that to userspace.
Doug Thompson87f24c32007-07-19 01:50:34 -070049
50
51PCI BUS SCANNING
Borislav Petkov043b4312015-06-19 11:47:17 +020052----------------
Alan Coxda9bb1d2006-01-18 17:44:13 -080053
Borislav Petkov043b4312015-06-19 11:47:17 +020054In addition, PCI devices are scanned for PCI Bus Parity and SERR Errors
55in order to determine if errors are occurring during data transfers.
Doug Thompson87f24c32007-07-19 01:50:34 -070056
Alan Coxda9bb1d2006-01-18 17:44:13 -080057The presence of PCI Parity errors must be examined with a grain of salt.
Borislav Petkov043b4312015-06-19 11:47:17 +020058There are several add-in adapters that do *not* follow the PCI specification
Alan Coxda9bb1d2006-01-18 17:44:13 -080059with regards to Parity generation and reporting. The specification says
60the vendor should tie the parity status bits to 0 if they do not intend
61to generate parity. Some vendors do not do this, and thus the parity bit
62can "float" giving false positives.
63
Borislav Petkov043b4312015-06-19 11:47:17 +020064There is a PCI device attribute located in sysfs that is checked by
65the EDAC PCI scanning code. If that attribute is set, PCI parity/error
66scanning is skipped for that device. The attribute is:
Doug Thompson87f24c32007-07-19 01:50:34 -070067
68 broken_parity_status
69
Borislav Petkov043b4312015-06-19 11:47:17 +020070and is located in /sys/devices/pci<XXX>/0000:XX:YY.Z directories for
Doug Thompson87f24c32007-07-19 01:50:34 -070071PCI devices.
72
Alan Coxda9bb1d2006-01-18 17:44:13 -080073
Borislav Petkov043b4312015-06-19 11:47:17 +020074VERSIONING
75----------
Alan Coxda9bb1d2006-01-18 17:44:13 -080076
Doug Thompson87f24c32007-07-19 01:50:34 -070077EDAC is composed of a "core" module (edac_core.ko) and several Memory
Borislav Petkov043b4312015-06-19 11:47:17 +020078Controller (MC) driver modules. On a given system, the CORE is loaded
79and one MC driver will be loaded. Both the CORE and the MC driver (or
80edac_device driver) have individual versions that reflect current
81release level of their respective modules.
Doug Thompson87f24c32007-07-19 01:50:34 -070082
Borislav Petkov043b4312015-06-19 11:47:17 +020083Thus, to "report" on what version a system is running, one must report
84both the CORE's and the MC driver's versions.
Alan Coxda9bb1d2006-01-18 17:44:13 -080085
86
87LOADING
Borislav Petkov043b4312015-06-19 11:47:17 +020088-------
Alan Coxda9bb1d2006-01-18 17:44:13 -080089
Borislav Petkov043b4312015-06-19 11:47:17 +020090If 'edac' was statically linked with the kernel then no loading
91is necessary. If 'edac' was built as modules then simply modprobe
92the 'edac' pieces that you need. You should be able to modprobe
93hardware-specific modules and have the dependencies load the necessary
94core modules.
Alan Coxda9bb1d2006-01-18 17:44:13 -080095
96Example:
97
98$> modprobe amd76x_edac
99
100loads both the amd76x_edac.ko memory controller module and the edac_mc.ko
101core module.
102
103
Borislav Petkov043b4312015-06-19 11:47:17 +0200104SYSFS INTERFACE
105---------------
Alan Coxda9bb1d2006-01-18 17:44:13 -0800106
Borislav Petkov043b4312015-06-19 11:47:17 +0200107EDAC presents a 'sysfs' interface for control and reporting purposes. It
108lives in the /sys/devices/system/edac directory.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800109
Borislav Petkov043b4312015-06-19 11:47:17 +0200110Within this directory there currently reside 2 components:
Alan Coxda9bb1d2006-01-18 17:44:13 -0800111
112 mc memory controller(s) system
Doug Thompson49c0dab72006-07-10 04:45:19 -0700113 pci PCI control and status system
Alan Coxda9bb1d2006-01-18 17:44:13 -0800114
115
Borislav Petkov043b4312015-06-19 11:47:17 +0200116
Alan Coxda9bb1d2006-01-18 17:44:13 -0800117Memory Controller (mc) Model
Borislav Petkov043b4312015-06-19 11:47:17 +0200118----------------------------
Alan Coxda9bb1d2006-01-18 17:44:13 -0800119
Borislav Petkov043b4312015-06-19 11:47:17 +0200120Each 'mc' device controls a set of DIMM memory modules. These modules
121are laid out in a Chip-Select Row (csrowX) and Channel table (chX).
122There can be multiple csrows and multiple channels.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800123
Borislav Petkov043b4312015-06-19 11:47:17 +0200124Memory controllers allow for several csrows, with 8 csrows being a
125typical value. Yet, the actual number of csrows depends on the layout of
126a given motherboard, memory controller and DIMM characteristics.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800127
Borislav Petkov043b4312015-06-19 11:47:17 +0200128Dual channels allows for 128 bit data transfers to/from the CPU from/to
129memory. Some newer chipsets allow for more than 2 channels, like Fully
130Buffered DIMMs (FB-DIMMs). The following example will assume 2 channels:
Alan Coxda9bb1d2006-01-18 17:44:13 -0800131
132
133 Channel 0 Channel 1
134 ===================================
135 csrow0 | DIMM_A0 | DIMM_B0 |
136 csrow1 | DIMM_A0 | DIMM_B0 |
137 ===================================
138
139 ===================================
140 csrow2 | DIMM_A1 | DIMM_B1 |
141 csrow3 | DIMM_A1 | DIMM_B1 |
142 ===================================
143
144In the above example table there are 4 physical slots on the motherboard
145for memory DIMMs:
146
147 DIMM_A0
148 DIMM_B0
149 DIMM_A1
150 DIMM_B1
151
Borislav Petkov043b4312015-06-19 11:47:17 +0200152Labels for these slots are usually silk-screened on the motherboard.
153Slots labeled 'A' are channel 0 in this example. Slots labeled 'B' are
154channel 1. Notice that there are two csrows possible on a physical DIMM.
155These csrows are allocated their csrow assignment based on the slot into
156which the memory DIMM is placed. Thus, when 1 DIMM is placed in each
157Channel, the csrows cross both DIMMs.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800158
159Memory DIMMs come single or dual "ranked". A rank is a populated csrow.
160Thus, 2 single ranked DIMMs, placed in slots DIMM_A0 and DIMM_B0 above
161will have 1 csrow, csrow0. csrow1 will be empty. On the other hand,
Dave Petersonf3479812006-03-26 01:38:53 -0800162when 2 dual ranked DIMMs are similarly placed, then both csrow0 and
Alan Coxda9bb1d2006-01-18 17:44:13 -0800163csrow1 will be populated. The pattern repeats itself for csrow2 and
164csrow3.
165
Borislav Petkov043b4312015-06-19 11:47:17 +0200166The representation of the above is reflected in the directory
167tree in EDAC's sysfs interface. Starting in directory
Alan Coxda9bb1d2006-01-18 17:44:13 -0800168/sys/devices/system/edac/mc each memory controller will be represented
Raoul Bhatia5989f112010-11-25 17:32:47 +0100169by its own 'mcX' directory, where 'X' is the index of the MC.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800170
171
172 ..../edac/mc/
173 |
174 |->mc0
175 |->mc1
176 |->mc2
177 ....
178
179Under each 'mcX' directory each 'csrowX' is again represented by a
Raoul Bhatia5989f112010-11-25 17:32:47 +0100180'csrowX', where 'X' is the csrow index:
Alan Coxda9bb1d2006-01-18 17:44:13 -0800181
182
183 .../mc/mc0/
184 |
185 |->csrow0
186 |->csrow2
187 |->csrow3
188 ....
189
Borislav Petkov043b4312015-06-19 11:47:17 +0200190Notice that there is no csrow1, which indicates that csrow0 is composed
191of a single ranked DIMMs. This should also apply in both Channels, in
192order to have dual-channel mode be operational. Since both csrow2 and
193csrow3 are populated, this indicates a dual ranked set of DIMMs for
194channels 0 and 1.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800195
196
Borislav Petkov043b4312015-06-19 11:47:17 +0200197Within each of the 'mcX' and 'csrowX' directories are several EDAC
198control and attribute files.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800199
Alan Coxda9bb1d2006-01-18 17:44:13 -0800200
Borislav Petkov043b4312015-06-19 11:47:17 +0200201'mcX' directories
202-----------------
Alan Coxda9bb1d2006-01-18 17:44:13 -0800203
204In 'mcX' directories are EDAC control and attribute files for
Mauro Carvalho Chehab8b6f04c2012-04-17 08:53:34 -0300205this 'X' instance of the memory controllers.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800206
Mauro Carvalho Chehab8b6f04c2012-04-17 08:53:34 -0300207For a description of the sysfs API, please see:
Rami Rosen3aae9ed2015-06-19 09:18:34 +0300208 Documentation/ABI/testing/sysfs-devices-edac
Alan Coxda9bb1d2006-01-18 17:44:13 -0800209
210
Mauro Carvalho Chehab032d0ab2016-10-27 10:00:46 -0200211``dimmX`` or ``rankX`` directories
212----------------------------------
213
214The recommended way to use the EDAC subsystem is to look at the information
215provided by the ``dimmX`` or ``rankX`` directories [#f5]_.
216
217A typical EDAC system has the following structure under
218``/sys/devices/system/edac/``\ [#f6]_::
219
220 /sys/devices/system/edac/
221 ├── mc
222 │   ├── mc0
223 │   │   ├── ce_count
224 │   │   ├── ce_noinfo_count
225 │   │   ├── dimm0
226 │   │   │   ├── dimm_dev_type
227 │   │   │   ├── dimm_edac_mode
228 │   │   │   ├── dimm_label
229 │   │   │   ├── dimm_location
230 │   │   │   ├── dimm_mem_type
231 │   │   │   ├── size
232 │   │   │   └── uevent
233 │   │   ├── max_location
234 │   │   ├── mc_name
235 │   │   ├── reset_counters
236 │   │   ├── seconds_since_reset
237 │   │   ├── size_mb
238 │   │   ├── ue_count
239 │   │   ├── ue_noinfo_count
240 │   │   └── uevent
241 │   ├── mc1
242 │   │   ├── ce_count
243 │   │   ├── ce_noinfo_count
244 │   │   ├── dimm0
245 │   │   │   ├── dimm_dev_type
246 │   │   │   ├── dimm_edac_mode
247 │   │   │   ├── dimm_label
248 │   │   │   ├── dimm_location
249 │   │   │   ├── dimm_mem_type
250 │   │   │   ├── size
251 │   │   │   └── uevent
252 │   │   ├── max_location
253 │   │   ├── mc_name
254 │   │   ├── reset_counters
255 │   │   ├── seconds_since_reset
256 │   │   ├── size_mb
257 │   │   ├── ue_count
258 │   │   ├── ue_noinfo_count
259 │   │   └── uevent
260 │   └── uevent
261 └── uevent
262
263In the ``dimmX`` directories are EDAC control and attribute files for
264this ``X`` memory module:
265
266- ``size`` - Total memory managed by this csrow attribute file
267
268 This attribute file displays, in count of megabytes, the memory
269 that this csrow contains.
270
271- ``dimm_dev_type`` - Device type attribute file
272
273 This attribute file will display what type of DRAM device is
274 being utilized on this DIMM.
275 Examples:
276
277 - x1
278 - x2
279 - x4
280 - x8
281
282- ``dimm_edac_mode`` - EDAC Mode of operation attribute file
283
284 This attribute file will display what type of Error detection
285 and correction is being utilized.
286
287- ``dimm_label`` - memory module label control file
288
289 This control file allows this DIMM to have a label assigned
290 to it. With this label in the module, when errors occur
291 the output can provide the DIMM label in the system log.
292 This becomes vital for panic events to isolate the
293 cause of the UE event.
294
295 DIMM Labels must be assigned after booting, with information
296 that correctly identifies the physical slot with its
297 silk screen label. This information is currently very
298 motherboard specific and determination of this information
299 must occur in userland at this time.
300
301- ``dimm_location`` - location of the memory module
302
303 The location can have up to 3 levels, and describe how the
304 memory controller identifies the location of a memory module.
305 Depending on the type of memory and memory controller, it
306 can be:
307
308 - *csrow* and *channel* - used when the memory controller
309 doesn't identify a single DIMM - e. g. in ``rankX`` dir;
310 - *branch*, *channel*, *slot* - typically used on FB-DIMM memory
311 controllers;
312 - *channel*, *slot* - used on Nehalem and newer Intel drivers.
313
314- ``dimm_mem_type`` - Memory Type attribute file
315
316 This attribute file will display what type of memory is currently
317 on this csrow. Normally, either buffered or unbuffered memory.
318 Examples:
319
320 - Registered-DDR
321 - Unbuffered-DDR
322
323.. [#f5] On some systems, the memory controller doesn't have any logic
324 to identify the memory module. On such systems, the directory is called ``rankX`` and works on a similar way as the ``csrowX`` directories.
325 On modern Intel memory controllers, the memory controller identifies the
326 memory modules directly. On such systems, the directory is called ``dimmX``.
327
328.. [#f6] There are also some ``power`` directories and ``subsystem``
329 symlinks inside the sysfs mapping that are automatically created by
330 the sysfs subsystem. Currently, they serve no purpose.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800331
Borislav Petkov043b4312015-06-19 11:47:17 +0200332'csrowX' directories
333--------------------
334
335When CONFIG_EDAC_LEGACY_SYSFS is enabled, sysfs will contain the csrowX
336directories. As this API doesn't work properly for Rambus, FB-DIMMs and
337modern Intel Memory Controllers, this is being deprecated in favor of
338dimmX directories.
Mauro Carvalho Chehab8b6f04c2012-04-17 08:53:34 -0300339
Alan Coxda9bb1d2006-01-18 17:44:13 -0800340In the 'csrowX' directories are EDAC control and attribute files for
Raoul Bhatia5989f112010-11-25 17:32:47 +0100341this 'X' instance of csrow:
Alan Coxda9bb1d2006-01-18 17:44:13 -0800342
343
344Total Uncorrectable Errors count attribute file:
345
346 'ue_count'
347
348 This attribute file displays the total count of uncorrectable
349 errors that have occurred on this csrow. If panic_on_ue is set
350 this counter will not have a chance to increment, since EDAC
351 will panic the system.
352
353
354Total Correctable Errors count attribute file:
355
356 'ce_count'
357
358 This attribute file displays the total count of correctable
Borislav Petkov043b4312015-06-19 11:47:17 +0200359 errors that have occurred on this csrow. This count is very
360 important to examine. CEs provide early indications that a
361 DIMM is beginning to fail. This count field should be
362 monitored for non-zero values and report such information
363 to the system administrator.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800364
365
366Total memory managed by this csrow attribute file:
367
368 'size_mb'
369
Rami Rosen3aae9ed2015-06-19 09:18:34 +0300370 This attribute file displays, in count of megabytes, the memory
Dave Petersonf3479812006-03-26 01:38:53 -0800371 that this csrow contains.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800372
373
374Memory Type attribute file:
375
376 'mem_type'
377
378 This attribute file will display what type of memory is currently
379 on this csrow. Normally, either buffered or unbuffered memory.
Doug Thompson49c0dab72006-07-10 04:45:19 -0700380 Examples:
381 Registered-DDR
382 Unbuffered-DDR
Alan Coxda9bb1d2006-01-18 17:44:13 -0800383
384
385EDAC Mode of operation attribute file:
386
387 'edac_mode'
388
389 This attribute file will display what type of Error detection
390 and correction is being utilized.
391
392
393Device type attribute file:
394
395 'dev_type'
396
Doug Thompson49c0dab72006-07-10 04:45:19 -0700397 This attribute file will display what type of DRAM device is
398 being utilized on this DIMM.
399 Examples:
400 x1
401 x2
402 x4
403 x8
Alan Coxda9bb1d2006-01-18 17:44:13 -0800404
405
406Channel 0 CE Count attribute file:
407
408 'ch0_ce_count'
409
410 This attribute file will display the count of CEs on this
411 DIMM located in channel 0.
412
413
414Channel 0 UE Count attribute file:
415
416 'ch0_ue_count'
417
418 This attribute file will display the count of UEs on this
419 DIMM located in channel 0.
420
421
422Channel 0 DIMM Label control file:
423
424 'ch0_dimm_label'
425
426 This control file allows this DIMM to have a label assigned
427 to it. With this label in the module, when errors occur
428 the output can provide the DIMM label in the system log.
429 This becomes vital for panic events to isolate the
430 cause of the UE event.
431
432 DIMM Labels must be assigned after booting, with information
433 that correctly identifies the physical slot with its
434 silk screen label. This information is currently very
435 motherboard specific and determination of this information
436 must occur in userland at this time.
437
438
439Channel 1 CE Count attribute file:
440
441 'ch1_ce_count'
442
443 This attribute file will display the count of CEs on this
444 DIMM located in channel 1.
445
446
447Channel 1 UE Count attribute file:
448
449 'ch1_ue_count'
450
451 This attribute file will display the count of UEs on this
452 DIMM located in channel 0.
453
454
455Channel 1 DIMM Label control file:
456
457 'ch1_dimm_label'
458
459 This control file allows this DIMM to have a label assigned
460 to it. With this label in the module, when errors occur
461 the output can provide the DIMM label in the system log.
462 This becomes vital for panic events to isolate the
463 cause of the UE event.
464
465 DIMM Labels must be assigned after booting, with information
466 that correctly identifies the physical slot with its
467 silk screen label. This information is currently very
468 motherboard specific and determination of this information
469 must occur in userland at this time.
470
Alan Coxda9bb1d2006-01-18 17:44:13 -0800471
Borislav Petkov043b4312015-06-19 11:47:17 +0200472
473SYSTEM LOGGING
474--------------
475
476If logging for UEs and CEs is enabled, then system logs will contain
477information indicating that errors have been detected:
Alan Coxda9bb1d2006-01-18 17:44:13 -0800478
Doug Thompson49c0dab72006-07-10 04:45:19 -0700479EDAC MC0: CE page 0x283, offset 0xce0, grain 8, syndrome 0x6ec3, row 0,
Alan Coxda9bb1d2006-01-18 17:44:13 -0800480channel 1 "DIMM_B1": amd76x_edac
481
Doug Thompson49c0dab72006-07-10 04:45:19 -0700482EDAC MC0: CE page 0x1e5, offset 0xfb0, grain 8, syndrome 0xb741, row 0,
Alan Coxda9bb1d2006-01-18 17:44:13 -0800483channel 1 "DIMM_B1": amd76x_edac
484
485
486The structure of the message is:
487 the memory controller (MC0)
488 Error type (CE)
489 memory page (0x283)
490 offset in the page (0xce0)
491 the byte granularity (grain 8)
492 or resolution of the error
493 the error syndrome (0xb741)
494 memory row (row 0)
495 memory channel (channel 1)
496 DIMM label, if set prior (DIMM B1
497 and then an optional, driver-specific message that may
498 have additional information.
499
Borislav Petkov043b4312015-06-19 11:47:17 +0200500Both UEs and CEs with no info will lack all but memory controller, error
501type, a notice of "no info" and then an optional, driver-specific error
502message.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800503
504
Alan Coxda9bb1d2006-01-18 17:44:13 -0800505PCI Bus Parity Detection
Borislav Petkov043b4312015-06-19 11:47:17 +0200506------------------------
Alan Coxda9bb1d2006-01-18 17:44:13 -0800507
Borislav Petkov043b4312015-06-19 11:47:17 +0200508On Header Type 00 devices, the primary status is looked at for any
509parity error regardless of whether parity is enabled on the device or
510not. (The spec indicates parity is generated in some cases). On Header
511Type 01 bridges, the secondary status register is also looked at to see
512if parity occurred on the bus on the other side of the bridge.
Alan Coxda9bb1d2006-01-18 17:44:13 -0800513
514
515SYSFS CONFIGURATION
Borislav Petkov043b4312015-06-19 11:47:17 +0200516-------------------
Alan Coxda9bb1d2006-01-18 17:44:13 -0800517
518Under /sys/devices/system/edac/pci are control and attribute files as follows:
519
520
521Enable/Disable PCI Parity checking control file:
522
523 'check_pci_parity'
524
525
526 This control file enables or disables the PCI Bus Parity scanning
527 operation. Writing a 1 to this file enables the scanning. Writing
528 a 0 to this file disables the scanning.
529
530 Enable:
531 echo "1" >/sys/devices/system/edac/pci/check_pci_parity
532
533 Disable:
534 echo "0" >/sys/devices/system/edac/pci/check_pci_parity
535
536
Arthur Jones327dafb2008-07-25 01:49:10 -0700537Parity Count:
538
539 'pci_parity_count'
540
541 This attribute file will display the number of parity errors that
542 have been detected.
543
544
Borislav Petkov043b4312015-06-19 11:47:17 +0200545
Arthur Jones327dafb2008-07-25 01:49:10 -0700546MODULE PARAMETERS
Borislav Petkov043b4312015-06-19 11:47:17 +0200547-----------------
Arthur Jones327dafb2008-07-25 01:49:10 -0700548
549Panic on UE control file:
550
551 'edac_mc_panic_on_ue'
552
553 An uncorrectable error will cause a machine panic. This is usually
554 desirable. It is a bad idea to continue when an uncorrectable error
555 occurs - it is indeterminate what was uncorrected and the operating
556 system context might be so mangled that continuing will lead to further
557 corruption. If the kernel has MCE configured, then EDAC will never
558 notice the UE.
559
560 LOAD TIME: module/kernel parameter: edac_mc_panic_on_ue=[0|1]
561
562 RUN TIME: echo "1" > /sys/module/edac_core/parameters/edac_mc_panic_on_ue
563
564
565Log UE control file:
566
567 'edac_mc_log_ue'
568
569 Generate kernel messages describing uncorrectable errors. These errors
570 are reported through the system message log system. UE statistics
571 will be accumulated even when UE logging is disabled.
572
573 LOAD TIME: module/kernel parameter: edac_mc_log_ue=[0|1]
574
575 RUN TIME: echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ue
576
577
578Log CE control file:
579
580 'edac_mc_log_ce'
581
582 Generate kernel messages describing correctable errors. These
583 errors are reported through the system message log system.
584 CE statistics will be accumulated even when CE logging is disabled.
585
586 LOAD TIME: module/kernel parameter: edac_mc_log_ce=[0|1]
587
588 RUN TIME: echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ce
589
590
591Polling period control file:
592
593 'edac_mc_poll_msec'
594
595 The time period, in milliseconds, for polling for error information.
596 Too small a value wastes resources. Too large a value might delay
597 necessary handling of errors and might loose valuable information for
598 locating the error. 1000 milliseconds (once each second) is the current
599 default. Systems which require all the bandwidth they can get, may
600 increase this.
601
602 LOAD TIME: module/kernel parameter: edac_mc_poll_msec=[0|1]
603
604 RUN TIME: echo "1000" > /sys/module/edac_core/parameters/edac_mc_poll_msec
605
Alan Coxda9bb1d2006-01-18 17:44:13 -0800606
607Panic on PCI PARITY Error:
608
609 'panic_on_pci_parity'
610
611
Rami Rosen3aae9ed2015-06-19 09:18:34 +0300612 This control file enables or disables panicking when a parity
Alan Coxda9bb1d2006-01-18 17:44:13 -0800613 error has been detected.
614
615
Arthur Jones327dafb2008-07-25 01:49:10 -0700616 module/kernel parameter: edac_panic_on_pci_pe=[0|1]
Alan Coxda9bb1d2006-01-18 17:44:13 -0800617
618 Enable:
Arthur Jones327dafb2008-07-25 01:49:10 -0700619 echo "1" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe
Alan Coxda9bb1d2006-01-18 17:44:13 -0800620
621 Disable:
Arthur Jones327dafb2008-07-25 01:49:10 -0700622 echo "0" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe
Alan Coxda9bb1d2006-01-18 17:44:13 -0800623
624
625
Borislav Petkov043b4312015-06-19 11:47:17 +0200626EDAC device type
627----------------
Doug Thompson87f24c32007-07-19 01:50:34 -0700628
629In the header file, edac_core.h, there is a series of edac_device structures
630and APIs for the EDAC_DEVICE.
631
632User space access to an edac_device is through the sysfs interface.
633
634At the location /sys/devices/system/edac (sysfs) new edac_device devices will
635appear.
636
637There is a three level tree beneath the above 'edac' directory. For example,
638the 'test_device_edac' device (found at the bluesmoke.sourceforget.net website)
639installs itself as:
640
641 /sys/devices/systm/edac/test-instance
642
643in this directory are various controls, a symlink and one or more 'instance'
Carlos Garciac98be0c2014-04-04 22:31:00 -0400644directories.
Doug Thompson87f24c32007-07-19 01:50:34 -0700645
646The standard default controls are:
647
648 log_ce boolean to log CE events
649 log_ue boolean to log UE events
650 panic_on_ue boolean to 'panic' the system if an UE is encountered
651 (default off, can be set true via startup script)
652 poll_msec time period between POLL cycles for events
653
654The test_device_edac device adds at least one of its own custom control:
655
656 test_bits which in the current test driver does nothing but
657 show how it is installed. A ported driver can
658 add one or more such controls and/or attributes
659 for specific uses.
660 One out-of-tree driver uses controls here to allow
661 for ERROR INJECTION operations to hardware
662 injection registers
663
664The symlink points to the 'struct dev' that is registered for this edac_device.
665
666INSTANCES
Borislav Petkov043b4312015-06-19 11:47:17 +0200667---------
Doug Thompson87f24c32007-07-19 01:50:34 -0700668
669One or more instance directories are present. For the 'test_device_edac' case:
670
671 test-instance0
672
673
674In this directory there are two default counter attributes, which are totals of
675counter in deeper subdirectories.
676
677 ce_count total of CE events of subdirectories
678 ue_count total of UE events of subdirectories
679
680BLOCKS
Borislav Petkov043b4312015-06-19 11:47:17 +0200681------
Doug Thompson87f24c32007-07-19 01:50:34 -0700682
683At the lowest directory level is the 'block' directory. There can be 0, 1
684or more blocks specified in each instance.
685
686 test-block0
687
688
689In this directory the default attributes are:
690
691 ce_count which is counter of CE events for this 'block'
692 of hardware being monitored
693 ue_count which is counter of UE events for this 'block'
694 of hardware being monitored
695
696
697The 'test_device_edac' device adds 4 attributes and 1 control:
698
699 test-block-bits-0 for every POLL cycle this counter
700 is incremented
701 test-block-bits-1 every 10 cycles, this counter is bumped once,
702 and test-block-bits-0 is set to 0
703 test-block-bits-2 every 100 cycles, this counter is bumped once,
704 and test-block-bits-1 is set to 0
705 test-block-bits-3 every 1000 cycles, this counter is bumped once,
706 and test-block-bits-2 is set to 0
707
708
709 reset-counters writing ANY thing to this control will
710 reset all the above counters.
711
712
Rami Rosen3aae9ed2015-06-19 09:18:34 +0300713Use of the 'test_device_edac' driver should enable any others to create their own
Doug Thompson87f24c32007-07-19 01:50:34 -0700714unique drivers for their hardware systems.
715
716The 'test_device_edac' sample driver is located at the
717bluesmoke.sourceforge.net project site for EDAC.
718
Borislav Petkov043b4312015-06-19 11:47:17 +0200719
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300720NEHALEM USAGE OF EDAC APIs
Borislav Petkov043b4312015-06-19 11:47:17 +0200721--------------------------
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300722
723This chapter documents some EXPERIMENTAL mappings for EDAC API to handle
724Nehalem EDAC driver. They will likely be changed on future versions
725of the driver.
726
727Due to the way Nehalem exports Memory Controller data, some adjustments
728were done at i7core_edac driver. This chapter will cover those differences
729
Rami Rosen3aae9ed2015-06-19 09:18:34 +03007301) On Nehalem, there is one Memory Controller per Quick Patch Interconnect
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300731 (QPI). At the driver, the term "socket" means one QPI. This is
732 associated with a physical CPU socket.
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300733
734 Each MC have 3 physical read channels, 3 physical write channels and
Masanari Iidac94bed8e2012-04-10 00:22:13 +0900735 3 logic channels. The driver currently sees it as just 3 channels.
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300736 Each channel can have up to 3 DIMMs.
737
738 The minimum known unity is DIMMs. There are no information about csrows.
Rami Rosen3aae9ed2015-06-19 09:18:34 +0300739 As EDAC API maps the minimum unity is csrows, the driver sequentially
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300740 maps channel/dimm into different csrows.
741
Lucas De Marchi25985ed2011-03-30 22:57:33 -0300742 For example, supposing the following layout:
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300743 Ch0 phy rd0, wr0 (0x063f4031): 2 ranks, UDIMMs
744 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
745 dimm 1 1024 Mb offset: 4, bank: 8, rank: 1, row: 0x4000, col: 0x400
746 Ch1 phy rd1, wr1 (0x063f4031): 2 ranks, UDIMMs
747 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
748 Ch2 phy rd3, wr3 (0x063f4031): 2 ranks, UDIMMs
749 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
750 The driver will map it as:
751 csrow0: channel 0, dimm0
752 csrow1: channel 0, dimm1
753 csrow2: channel 1, dimm0
754 csrow3: channel 2, dimm0
755
756exports one
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300757 DIMM per csrow.
758
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300759 Each QPI is exported as a different memory controller.
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300760
Rami Rosen3aae9ed2015-06-19 09:18:34 +03007612) Nehalem MC has the ability to generate errors. The driver implements this
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300762 functionality via some error injection nodes:
763
764 For injecting a memory error, there are some sysfs nodes, under
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300765 /sys/devices/system/edac/mc/mc?/:
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300766
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300767 inject_addrmatch/*:
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300768 Controls the error injection mask register. It is possible to specify
769 several characteristics of the address to match an error code:
770 dimm = the affected dimm. Numbers are relative to a channel;
771 rank = the memory rank;
772 channel = the channel that will generate an error;
773 bank = the affected bank;
774 page = the page address;
775 column (or col) = the address column.
776 each of the above values can be set to "any" to match any valid value.
777
778 At driver init, all values are set to any.
779
780 For example, to generate an error at rank 1 of dimm 2, for any channel,
781 any bank, any page, any column:
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300782 echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm
783 echo 1 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300784
785 To return to the default behaviour of matching any, you can do:
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300786 echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm
787 echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300788
789 inject_eccmask:
790 specifies what bits will have troubles,
791
792 inject_section:
793 specifies what ECC cache section will get the error:
794 3 for both
795 2 for the highest
796 1 for the lowest
797
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300798 inject_type:
799 specifies the type of error, being a combination of the following bits:
800 bit 0 - repeat
801 bit 1 - ecc
802 bit 2 - parity
803
804 inject_enable starts the error generation when something different
805 than 0 is written.
806
807 All inject vars can be read. root permission is needed for write.
808
809 Datasheet states that the error will only be generated after a write on an
810 address that matches inject_addrmatch. It seems, however, that reading will
811 also produce an error.
812
813 For example, the following code will generate an error for any write access
814 at socket 0, on any DIMM/address on channel 2:
815
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300816 echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/channel
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300817 echo 2 >/sys/devices/system/edac/mc/mc0/inject_type
818 echo 64 >/sys/devices/system/edac/mc/mc0/inject_eccmask
819 echo 3 >/sys/devices/system/edac/mc/mc0/inject_section
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300820 echo 1 >/sys/devices/system/edac/mc/mc0/inject_enable
821 dd if=/dev/mem of=/dev/null seek=16k bs=4k count=1 >& /dev/null
822
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300823 For socket 1, it is needed to replace "mc0" by "mc1" at the above
824 commands.
825
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300826 The generated error message will look like:
827
828 EDAC MC0: UE row 0, channel-a= 0 channel-b= 0 labels "-": NON_FATAL (addr = 0x0075b980, socket=0, Dimm=0, Channel=2, syndrome=0x00000040, count=1, Err=8c0000400001009f:4000080482 (read error: read ECC error))
829
8303) Nehalem specific Corrected Error memory counters
831
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300832 Nehalem have some registers to count memory errors. The driver uses those
833 registers to report Corrected Errors on devices with Registered Dimms.
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300834
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300835 However, those counters don't work with Unregistered Dimms. As the chipset
836 offers some counters that also work with UDIMMS (but with a worse level of
837 granularity than the default ones), the driver exposes those registers for
838 UDIMM memories.
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300839
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300840 They can be read by looking at the contents of all_channel_counts/
Mauro Carvalho Chehab31983a02009-08-05 21:16:56 -0300841
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300842 $ for i in /sys/devices/system/edac/mc/mc0/all_channel_counts/*; do echo $i; cat $i; done
843 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm0
844 0
845 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm1
846 0
847 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm2
848 0
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300849
850 What happens here is that errors on different csrows, but at the same
851 dimm number will increment the same counter.
852 So, in this memory mapping:
853 csrow0: channel 0, dimm0
854 csrow1: channel 0, dimm1
855 csrow2: channel 1, dimm0
856 csrow3: channel 2, dimm0
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300857 The hardware will increment udimm0 for an error at the first dimm at either
858 csrow0, csrow2 or csrow3;
859 The hardware will increment udimm1 for an error at the second dimm at either
860 csrow0, csrow2 or csrow3;
861 The hardware will increment udimm2 for an error at the third dimm at either
862 csrow0, csrow2 or csrow3;
Mauro Carvalho Chehabc3444362009-09-05 05:10:15 -0300863
8644) Standard error counters
865
866 The standard error counters are generated when an mcelog error is received
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300867 by the driver. Since, with udimm, this is counted by software, it is
Rami Rosen3aae9ed2015-06-19 09:18:34 +0300868 possible that some errors could be lost. With rdimm's, they display the
Mauro Carvalho Chehab35be9542009-09-24 17:28:50 -0300869 contents of the registers
Borislav Petkov043b4312015-06-19 11:47:17 +0200870
Aravind Gopalakrishnan6b7464b2015-09-28 06:44:31 -0500871AMD64_EDAC REFERENCE DOCUMENTS USED
872-----------------------------------
873amd64_edac module is based on the following documents
874(available from http://support.amd.com/en-us/search/tech-docs):
875
8761. Title: BIOS and Kernel Developer's Guide for AMD Athlon 64 and AMD
877 Opteron Processors
878 AMD publication #: 26094
879 Revision: 3.26
880 Link: http://support.amd.com/TechDocs/26094.PDF
881
8822. Title: BIOS and Kernel Developer's Guide for AMD NPT Family 0Fh
883 Processors
884 AMD publication #: 32559
885 Revision: 3.00
886 Issue Date: May 2006
887 Link: http://support.amd.com/TechDocs/32559.pdf
888
8893. Title: BIOS and Kernel Developer's Guide (BKDG) For AMD Family 10h
890 Processors
891 AMD publication #: 31116
892 Revision: 3.00
893 Issue Date: September 07, 2007
894 Link: http://support.amd.com/TechDocs/31116.pdf
895
8964. Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 15h
897 Models 30h-3Fh Processors
898 AMD publication #: 49125
899 Revision: 3.06
900 Issue Date: 2/12/2015 (latest release)
901 Link: http://support.amd.com/TechDocs/49125_15h_Models_30h-3Fh_BKDG.pdf
902
9035. Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 15h
904 Models 60h-6Fh Processors
905 AMD publication #: 50742
906 Revision: 3.01
907 Issue Date: 7/23/2015 (latest release)
908 Link: http://support.amd.com/TechDocs/50742_15h_Models_60h-6Fh_BKDG.pdf
909
9106. Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 16h
911 Models 00h-0Fh Processors
912 AMD publication #: 48751
913 Revision: 3.03
914 Issue Date: 2/23/2015 (latest release)
915 Link: http://support.amd.com/TechDocs/48751_16h_bkdg.pdf
916
Borislav Petkov043b4312015-06-19 11:47:17 +0200917CREDITS:
918========
919
920Written by Doug Thompson <dougthompson@xmission.com>
9217 Dec 2005
92217 Jul 2007 Updated
923
924(c) Mauro Carvalho Chehab
92505 Aug 2009 Nehalem interface
926
927EDAC authors/maintainers:
928
929 Doug Thompson, Dave Jiang, Dave Peterson et al,
930 Mauro Carvalho Chehab
931 Borislav Petkov
932 original author: Thayne Harbaugh