nixos/rasdaemon: init with 0.6.7 #85039

evils · 2020-04-11T22:54:57Z

Motivation for this change

closes #42592
based on #73149

Things done

packaged rasdaemon
wrote rasdaemon module
wrote rasdaemon module test

Status

help wanted

amaxine · 2020-04-12T09:06:36Z

determine what config options are needed

CONFIG_EDAC_DEBUG is needed for testing.

DIMM labels

The format is:

Vendor: <vendor-name>
  Product: <product-name>
  Model: <model-name>
    <label>:  <mc>.<top>.<mid>.<low>

Have never actually bothered decoding these, but a rough example would be:

Vendor: Dell Inc.
  Product: PowerEdge R740
  Model: 08D89F
    DIMM_A1:  0.0.0; DIMM_A2:  0.1.0;

Though those labels are nonsensical, but they do provide output:

ras-mc-ctl --print-labels
LOCATION                            CONFIGURED LABEL     SYSFS CONTENTS      
mc0 channel 0 slot 0                DIMM_A1              CPU_SrcID#0_MC#0_Chan#0_DIMM#0
mc0 channel 1 slot 0                DIMM_A2              CPU_SrcID#0_MC#0_Chan#1_DIMM#0

Both Product and Model can be a comma delimited list.

I hope this helps a bit.

evils · 2020-04-12T23:52:38Z

determine what config options are needed

CONFIG_EDAC_DEBUG is needed for testing.

that's mostly a note to myself, the dependencies need some config too and i should summarize them for when i get around to writing the nixos test

DIMM labels

The format is:

I wasn't sure if the content at /etc/ras/dimm_labels.d/ was indeed that of their example labels or some generated output because the example ones don't get installed (at least on debian)

I'll probably link to the dell file and any user supplied entries in /etc/ras/dimm_labels.d/
and to a mainboard definition in /etc/ras/mainboard, which unlike what the man page says, seems to be located there
(everything related to labels and mainboard seems to be reused from edac-utils, hence the remnant /etc/edac/ from edac-ctl(8))

@maxeaubrey thanks for the --print-labels output, it clarifies what the "label" numbers mean, could you show me the output of sudo ras-mc-ctl --guess-labels?

amaxine · 2020-04-13T09:01:12Z

The provided examplels are not useful without models (or a mainboard file to overwrite the name, I guess? Haven't actually double checked the code, or verified how that works in practice), which is why I assume they do not get installed - they do not work as is in my experience at least.

I've not seen --guess-labels work frequently, if at all, but here's a Dell example:

memory stick 'A1' is located at 'Not Specified'            
memory stick 'A2' is located at 'Not Specified'            
memory stick 'A3' is located at 'Not Specified'                                
memory stick 'A4' is located at 'Not Specified'            
memory stick 'A5' is located at 'Not Specified'            
memory stick 'A6' is located at 'Not Specified'               
memory stick 'A7' is located at 'Not Specified'           
memory stick 'A8' is located at 'Not Specified'           
memory stick 'A9' is located at 'Not Specified'           
memory stick 'A10' is located at 'Not Specified'              
memory stick 'A11' is located at 'Not Specified'           
memory stick 'A12' is located at 'Not Specified'                         
memory stick 'B1' is located at 'Not Specified'                        
memory stick 'B2' is located at 'Not Specified'                                
memory stick 'B3' is located at 'Not Specified'            
memory stick 'B4' is located at 'Not Specified'            
memory stick 'B5' is located at 'Not Specified'               
memory stick 'B6' is located at 'Not Specified'            
memory stick 'B7' is located at 'Not Specified'            
memory stick 'B8' is located at 'Not Specified'            
memory stick 'B9' is located at 'Not Specified'            
memory stick 'B10' is located at 'Not Specified'
memory stick 'B11' is located at 'Not Specified'
memory stick 'B12' is located at 'Not Specified'

And here's a somewhat more useful Supermicro example.

memory stick 'P1_DIMMA1' is located at 'P0_Node0_Channel0_Dimm0'
memory stick 'NO DIMM' is located at 'NO DIMM'
memory stick 'P1_DIMMB1' is located at 'P0_Node0_Channel1_Dimm0'
memory stick 'NO DIMM' is located at 'NO DIMM'
memory stick 'P1_DIMMC1' is located at 'P0_Node0_Channel2_Dimm0'
memory stick 'NO DIMM' is located at 'NO DIMM'
memory stick 'NO DIMM' is located at 'NO DIMM'
memory stick 'NO DIMM' is located at 'NO DIMM'
memory stick 'P2_DIMME1' is located at 'P1_Node1_Channel0_Dimm0'
memory stick 'NO DIMM' is located at 'NO DIMM'
memory stick 'P2_DIMMF1' is located at 'P1_Node1_Channel1_Dimm0'
memory stick 'NO DIMM' is located at 'NO DIMM'
memory stick 'P2_DIMMG1' is located at 'P1_Node1_Channel2_Dimm0'
memory stick 'NO DIMM' is located at 'NO DIMM'
memory stick 'NO DIMM' is located at 'NO DIMM'
memory stick 'NO DIMM' is located at 'NO DIMM'

I've not seen Dell/HP servers return more useful info than that, though that's with the caveat of only trying a limited set of Dell and HP servers.

evils · 2020-04-23T02:03:49Z

i consider the state of rasdaemon and nixos/rasdaemon "minimally viable"
if anyone wants to merge it i'm willing to separate the testing commits to a separate PR

stale · 2020-11-03T13:59:14Z

I marked this as stale due to inactivity. → More info

Luflosi · 2021-01-30T18:36:25Z

What's missing for this to get merged?

evils · 2021-01-30T20:07:33Z

for just nixos/rasdaemon, i think just some minor cleanup to get the tests out of the branch
and i'd have to switch it to lib instead of using stdenv.lib
(and i've got a local change to github for the source since there was an outage at infradead at some point)

for the tests, i'm not sure, i'll see about that

evils · 2021-02-02T02:01:35Z

changed almost everything a bit

switched to rasdaemon to v0.6.6-18-gc329012
v0.6.6 introduced Corrected Error Preemptive Failure Analysis, a feature to disable memory pages with recurring errors
the configuration for when a page is disabled is set via /etc/sysconfig/rasdaemon which is hard coded in the latest release
therefore, moved to latest master commit (though it's an intermediate commit that makes this configurable)

edac and aer tests look like they may work, except the machine features required for them are not present in the nixosTest VM

the mce test seems to assume mcelog is being used,
mce-test may be applicable for this application, unfortunately the package doesn't yield an output
(compilation failure without nix build failure...)
i don't recall this happening the last time i looked at it...

i'll try to run the tests on bare metal, though that may take me a while to get around to
if anyone knows how to get them working in a VM, that'd be nice.

evils · 2021-10-07T00:50:55Z

added release notes entry to the module init commit
missed because this PR predates the current CONTRIBUTING.md

evils · 2021-10-07T01:22:22Z

rebased on master where literalExample is deprecated (and switched to using literalExpression instead)
(also removed a trailing semicolon in the extraModules example)

tomberek · 2021-10-08T07:06:54Z

@SuperSandro2000 are the style suggestions here a blocking issue?

Often Sando's reviews are for style and consistency, but not strictly blocking. Merging of a new module usually requires a more functional review and someone familiar with the service or software in question.

tomberek · 2021-10-08T07:18:12Z

FWIW: software builds and runs. Test has the following output:

vm-test-run-rasdaemon> machine # [    6.275189] rasdaemon[892]: rasdaemon: ras:mc_event event disabled
vm-test-run-rasdaemon> machine # [    6.278076] rasdaemon[892]: rasdaemon: ras:aer_event event disabled
vm-test-run-rasdaemon> machine # [    6.280894] rasdaemon[892]: rasdaemon: mce:mce_record event disabled
vm-test-run-rasdaemon> machine # [    6.281696] rasdaemon[892]: rasdaemon: ras:extlog_mem_event event disabled
vm-test-run-rasdaemon> machine # [    6.285062] rasdaemon[892]: rasdaemon: ras:non_standard_event event disabled
vm-test-run-rasdaemon> machine # [    6.287613] rasdaemon[892]: rasdaemon: devlink:devlink_health_report event disabled
vm-test-run-rasdaemon> machine # [    6.290071] rasdaemon[892]: rasdaemon: block:block_rq_complete event disabled
vm-test-run-rasdaemon> machine # [    6.291779] rasdaemon[892]: rasdaemon: Can't write to set_event
vm-test-run-rasdaemon> machine # [    6.293090] dhcpcd[878]: eth0: deleting route to 10.0.2.0/24
vm-test-run-rasdaemon> machine # [    6.295242] systemd[1]: Stopping the RAS logging daemon...
vm-test-run-rasdaemon> machine # [    6.296357] rasdaemon[671]: rasdaemon: Recevied signal=15
vm-test-run-rasdaemon> machine # [    6.297358] rasdaemon[671]: overriding event (1180) ras:mc_event with new print handler
vm-test-run-rasdaemon> machine # [    6.298382] rasdaemon[671]: overriding event (1177) ras:aer_event with new print handler
vm-test-run-rasdaemon> machine # [    6.299379] rasdaemon[671]: overriding event (1178) ras:non_standard_event with new print handler
vm-test-run-rasdaemon> machine # [    6.299974] rasdaemon[671]: overriding event (110) mce:mce_record with new print handler
vm-test-run-rasdaemon> machine # [    6.300584] rasdaemon[671]: overriding event (1181) ras:extlog_mem_event with new print handler
vm-test-run-rasdaemon> machine # [    6.301494] rasdaemon[671]: overriding event (1265) net:net_dev_xmit_timeout with new print handler
vm-test-run-rasdaemon> machine # [    6.302305] rasdaemon[671]: overriding event (1274) devlink:devlink_health_report with new print handler
vm-test-run-rasdaemon> machine # [    6.302938] rasdaemon[671]: overriding event (1039) block:block_rq_complete with new print handler
vm-test-run-rasdaemon> machine # [    6.303781] rasdaemon[671]: Calling ras_mc_event_opendb()
vm-test-run-rasdaemon> machine # [    6.304270] rasdaemon[671]: Calling ras_mc_event_closedb()
vm-test-run-rasdaemon> machine # [    6.305084] ahx2l32ys40zhm4v6aca3k2mzkzi2p1i-audit-stop[894]: No rules
vm-test-run-rasdaemon> machine # [    6.305827] dhcpcd[878]: eth0: deleting default route via 10.0.2.2
vm-test-run-rasdaemon> machine # [    6.306822] systemd[1]: reload-systemd-vconsole-setup.service: Deactivated successfully.
vm-test-run-rasdaemon> machine # [    6.307859] ahx2l32ys40zhm4v6aca3k2mzkzi2p1i-audit-stop[900]: enabled 0
vm-test-run-rasdaemon> machine # [    6.308417] ahx2l32ys40zhm4v6aca3k2mzkzi2p1i-audit-stop[900]: failure 1
vm-test-run-rasdaemon> machine # [    6.308972] ahx2l32ys40zhm4v6aca3k2mzkzi2p1i-audit-stop[900]: pid 0
vm-test-run-rasdaemon> machine # [    6.309513] ahx2l32ys40zhm4v6aca3k2mzkzi2p1i-audit-stop[900]: rate_limit 0
vm-test-run-rasdaemon> machine # [    6.310129] ahx2l32ys40zhm4v6aca3k2mzkzi2p1i-audit-stop[900]: backlog_limit 64
vm-test-run-rasdaemon> machine # [    6.310790] ahx2l32ys40zhm4v6aca3k2mzkzi2p1i-audit-stop[900]: lost 0
vm-test-run-rasdaemon> machine # [    6.311399] ahx2l32ys40zhm4v6aca3k2mzkzi2p1i-audit-stop[900]: backlog 0
vm-test-run-rasdaemon> machine # [    6.311963] ahx2l32ys40zhm4v6aca3k2mzkzi2p1i-audit-stop[900]: backlog_wait_time 60000

evils · 2021-10-08T08:41:50Z

rebased onto master due to release notes merge conflict
for the sake of easing future rebases i've separated the release notes update into its own commit

nixos/modules/services/hardware/rasdaemon.nix

evils · 2021-10-17T22:25:52Z

fixed typo in the ras-mc-ctl service (ExectStart -> ExecStart)
rebased onto latest nixos-unstable
resolved another release notes conflict

danderson

Testing tonight (+8-10h from now) on a couple of Xeon systems and a 5th gen Ryzen. Thanks for putting this together!

nixos/modules/services/hardware/rasdaemon.nix

danderson · 2021-10-20T03:47:00Z

nixos/modules/services/hardware/rasdaemon.nix

+        # edac_core and amd64_edac should get loaded automatically
+        # i7core_edac may not be, and may not be required, but should load successfully
+        "edac_core"
+        "amd64_edac"


amd64_edac here makes initrd building fail for me when I set hardware.rasdaemon.testing = true. AFAICT, CONFIG_EDAC_AMD64 isn't enabled on the standard NixOS kernels, so initrd building fails when it tries to incorporate it. Tried with both a 4.19 and a 5.10 kernel, both failed in the same way.

Doing another build run now with a kernel patch to enable EDAC_AMD64, to see if that fixes it.

Doesn't seem to have fixed it :( AFAICT amd64_edac just doesn't get built for me, and as a result building the initrd fails down the road.

odd, besides my generated hardware-configuration.nix having kvm-amd, i don't think there's anything AMD specific in my config (nixos-unstable on ryzen)

If hydra doesn't complain while building, I think it's fine to assume my system is wonky somehow. This is only relevant when building for fault injection testing, so, meh.

hmm, at some point i had to change the module name from amd64_edac_mod to amd64_edac
i'm not sure when/where that changed, but maybe it's still the old name for you?

nixos/modules/services/hardware/rasdaemon.nix

danderson · 2021-10-20T04:22:27Z

Checked linux 4.19, 5.10 and 5.14:

devlink works on 5.10 and 5.14, the subsystem doesn't exist in 4.x.
memory_failure_event doesn't work on any of the three, afaict because CONFIG_MEMORY_FAILURE isn't enabled in the NixOS kernel config. Despite the name, this isn't logging events related to ECC errors, but events related to the kernel trying to survive uncorrectable ECC errors (e.g. by realizing the error is in a harmless buffer cache page it can dump and quarantine).

In both cases, the errors are benign: rasdaemon is correctly reporting that some kernels are missing some features that it could monitor, if you enabled them. But it keeps logging everything else it can.

So, LGTM on my machines. It can see a whole bunch of hardware stuff, maps my DIMM layout correctly, and generally works as well as the running kernel allows it to. I look forward to seeing this merged so I can enable it on my servers :)

and passthru test in the package

needed for mce-test

evils · 2021-10-29T00:03:51Z

resolved merge conflict due to release notes

Luflosi

I have been using this module for over half a year and it LGTM.

cyplo · 2021-10-30T10:24:02Z

I think it needs to be @SuperSandro2000 to approve as they requested changes before ?

SuperSandro2000 · 2021-10-30T13:15:04Z

I think it needs to be @SuperSandro2000 to approve as they requested changes before ?

We don't handle that like this. I don't have the time to double check if my suggestions where applied all the time. If someone else thinks the PR is in a good state and can be merged he/she can go ahead and merge it.

ofborg bot added 6.topic: nixos 8.has: module (update) labels Apr 11, 2020

evils force-pushed the rasdaemon branch 2 times, most recently from 7d80a8d to 095983e Compare April 11, 2020 23:23

ofborg bot added 8.has: package (new) 10.rebuild-darwin: 0 10.rebuild-linux: 11-100 labels Apr 11, 2020

evils force-pushed the rasdaemon branch 2 times, most recently from 8d83494 to 949ef01 Compare April 12, 2020 00:45

ofborg bot added 11.by: package-maintainer 10.rebuild-linux: 1-10 10.rebuild-linux: 11-100 and removed 10.rebuild-linux: 11-100 10.rebuild-linux: 1-10 labels Apr 12, 2020

evils force-pushed the rasdaemon branch 2 times, most recently from 90045ff to d217c09 Compare April 12, 2020 01:08

evils force-pushed the rasdaemon branch from d217c09 to bde6739 Compare April 23, 2020 01:34

evils force-pushed the rasdaemon branch from bde6739 to 4bad4cb Compare May 7, 2020 08:57

stale bot added the 2.status: stale label Nov 3, 2020

stale bot removed the 2.status: stale label Jan 30, 2021

evils force-pushed the rasdaemon branch 2 times, most recently from 61c34b5 to b73c012 Compare February 2, 2021 01:44

evils force-pushed the rasdaemon branch from 2253dc2 to 68bd338 Compare October 7, 2021 00:49

github-actions bot added 8.has: changelog 8.has: documentation labels Oct 7, 2021

evils force-pushed the rasdaemon branch from 68bd338 to 8039522 Compare October 7, 2021 01:21

evils force-pushed the rasdaemon branch from 8039522 to 77c09d0 Compare October 8, 2021 08:41

roberth reviewed Oct 9, 2021

View reviewed changes

nixos/modules/services/hardware/rasdaemon.nix Outdated Show resolved Hide resolved

evils force-pushed the rasdaemon branch from 77c09d0 to ac9e94d Compare October 9, 2021 12:57

evils force-pushed the rasdaemon branch from ac9e94d to 26fb7f0 Compare October 17, 2021 22:24

roberth mentioned this pull request Oct 19, 2021

rasdaemon: init at 0.6.7. #142181

Closed

12 tasks

danderson reviewed Oct 19, 2021

View reviewed changes

nixos/modules/services/hardware/rasdaemon.nix Show resolved Hide resolved

danderson reviewed Oct 20, 2021

View reviewed changes

evils added 6 commits October 29, 2021 02:01

rasdaemon: init at 0.6.7

2758b9c

error-inject: init mce, edac and aer

7af4a40

nixos/rasdaemon: init module

f3e9d48

nixos/rl-2111: add new service: rasdaemon

09be1fe

nixosTests.rasdaemon: init module test

4392ff7

and passthru test in the package

linuxPackages.vm-tools: init

8463ac9

needed for mce-test

evils force-pushed the rasdaemon branch from 26fb7f0 to 8463ac9 Compare October 29, 2021 00:03

Luflosi approved these changes Oct 29, 2021

View reviewed changes

tomberek merged commit 27ba20d into NixOS:master Oct 31, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos/rasdaemon: init with 0.6.7 #85039

nixos/rasdaemon: init with 0.6.7 #85039

evils commented Apr 11, 2020 •

edited

Loading

amaxine commented Apr 12, 2020 •

edited

Loading

evils commented Apr 12, 2020 •

edited

Loading

amaxine commented Apr 13, 2020

evils commented Apr 23, 2020

stale bot commented Nov 3, 2020

Luflosi commented Jan 30, 2021

evils commented Jan 30, 2021 •

edited

Loading

evils commented Feb 2, 2021 •

edited

Loading

evils commented Oct 7, 2021

evils commented Oct 7, 2021 •

edited

Loading

tomberek commented Oct 8, 2021

tomberek commented Oct 8, 2021

evils commented Oct 8, 2021

evils commented Oct 17, 2021

danderson left a comment

danderson Oct 20, 2021

danderson Oct 20, 2021

evils Oct 29, 2021

danderson Oct 29, 2021

evils Oct 29, 2021

danderson commented Oct 20, 2021 •

edited

Loading

evils commented Oct 29, 2021

Luflosi left a comment

cyplo commented Oct 30, 2021

SuperSandro2000 commented Oct 30, 2021

nixos/rasdaemon: init with 0.6.7 #85039

nixos/rasdaemon: init with 0.6.7 #85039

Conversation

evils commented Apr 11, 2020 • edited Loading

Motivation for this change

Things done

Status

amaxine commented Apr 12, 2020 • edited Loading

evils commented Apr 12, 2020 • edited Loading

amaxine commented Apr 13, 2020

evils commented Apr 23, 2020

stale bot commented Nov 3, 2020

Luflosi commented Jan 30, 2021

evils commented Jan 30, 2021 • edited Loading

evils commented Feb 2, 2021 • edited Loading

evils commented Oct 7, 2021

evils commented Oct 7, 2021 • edited Loading

tomberek commented Oct 8, 2021

tomberek commented Oct 8, 2021

evils commented Oct 8, 2021

evils commented Oct 17, 2021

danderson left a comment

Choose a reason for hiding this comment

danderson Oct 20, 2021

Choose a reason for hiding this comment

danderson Oct 20, 2021

Choose a reason for hiding this comment

evils Oct 29, 2021

Choose a reason for hiding this comment

danderson Oct 29, 2021

Choose a reason for hiding this comment

evils Oct 29, 2021

Choose a reason for hiding this comment

danderson commented Oct 20, 2021 • edited Loading

evils commented Oct 29, 2021

Luflosi left a comment

Choose a reason for hiding this comment

cyplo commented Oct 30, 2021

SuperSandro2000 commented Oct 30, 2021

evils commented Apr 11, 2020 •

edited

Loading

amaxine commented Apr 12, 2020 •

edited

Loading

evils commented Apr 12, 2020 •

edited

Loading

evils commented Jan 30, 2021 •

edited

Loading

evils commented Feb 2, 2021 •

edited

Loading

evils commented Oct 7, 2021 •

edited

Loading

danderson commented Oct 20, 2021 •

edited

Loading