nixos/vector: add module #103393

happysalada · 2020-11-11T03:51:19Z

Motivation for this change

add the vector service on nixos

Things done

I took inspiration from https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/monitoring/loki.nix
Not sure exactly what all the options should be.

I'm testing with the following in my configuration

    vector = {
      enable = true;
      configuration = {
        sources.input.type = "journald";

        transforms.level = {
          type = "lua";
          inputs = ["input"];
          version = "2";

          hooks = {
            process = ''
            function (event, emit)
              message = event.log.message
              level = string.match(message, "le?ve?l=(%w+)")
              msg = string.match(message, "me?ss?a?ge?=(.+)")
              if level then
                event.log.level = level
                event.log.message = msg
              end
              -- Very important! Emit the processed event.
              emit(event)
            end
            '';
          };
        };


        sinks = {

          file = {
            type = "file";
            inputs = ["level"];
            path = "./logs/day-%d.log";
            encoding = {
              codec = "ndjson";
            };
          };


          loki = {
            endpoint = "http://localhost:3100";
            inputs = ["level"]; 
            type = "loki";

            # Labels
            labels = {
              level = "{{ level }}";
            };
          };
        };
      };

The corresponding generated vector.toml file looks correct

[sinks]
[sinks.file]
inputs = ["level"]
path = "./logs/day-%d.log"
type = "file"

[sinks.file.encoding]
codec = "ndjson"

[sinks.loki]
endpoint = "http://localhost:3100"
inputs = ["level"]
type = "loki"

[sinks.loki.labels]
level = "{{ level }}"

[sources]
[sources.input]
type = "journald"

[transforms]
[transforms.level]
inputs = ["input"]
type = "lua"
version = "2"

[transforms.level.hooks]
process = "function (event, emit)\n  message = event.log.message\n  level = string.match(message, \"le?ve?l=(%w+)\")\n  msg = string.match(message, \"me?ss?a?ge?=(.+)\")\n  if level then\n    event.log.level = level\n    event.log.message = msg\n  end\n  -- Very important! Emit the processed event.\n  emit(event)\nend\n"

However I get an error starting up vector

Nov 11 03:35:55.482 ERROR vector: Configuration error: "/nix/store/pqjw0696125zylh3912f289rpwblsnm8-vector.toml": unknown variant `file`, there are no variants for key `sinks.file`

I can't figure out why. I'm currently using a similar vector.toml file with a docker setup and I don't have this error.

@thoughtpolice if you have any thoughts on this, I'm happy to try them.

aanderse

Hi there. Thanks for contributing a new module! I have reviewed your contribution and hope you will find my comments helpful. If you have any questions, or need any help with my comments please let me know.

I haven't got a chance to poke around with your code to understand the error (yet?). If I can find some time I will try.

aanderse · 2020-11-11T11:55:50Z

nixos/modules/services/logging/vector.nix

+
+in {
+  options.services.vector = {
+    enable = mkOption {


By convention most people use mkEnableOption here.

aanderse · 2020-11-11T11:57:49Z

nixos/modules/services/logging/vector.nix

+      '';
+    };
+
+    package = mkOption {


There is only one version of vector packaged in nixpkgs and it has no variation. I can't imagine this option will ever be used.

aanderse · 2020-11-11T11:58:33Z

nixos/modules/services/logging/vector.nix

+      '';
+    };
+
+    configuration = mkOption {


By convention we generally call this option settings.

aanderse · 2020-11-11T12:03:32Z

nixos/modules/services/logging/vector.nix

+      '';
+    };
+
+    user = mkOption {


I don't think this option is very useful. Maybe you should remove it.

aanderse · 2020-11-11T12:03:51Z

nixos/modules/services/logging/vector.nix

+      '';
+    };
+
+    group = mkOption {


aanderse · 2020-11-11T12:05:05Z

nixos/modules/services/logging/vector.nix

+      group = cfg.group;
+      isSystemUser = true;
+    };
+    systemd.services.vector = {


You should consider using the unit provided by upstream instead.

happysalada · 2020-11-12T01:25:32Z

@aanderse Thanks a lot for your comments!
I think I have addressed them all, let me know if I missed something.

I have tested again after the requested changes and I still get the same configuration error.
I'm not sure how to move forward. It seems the file is read, but that the configuration is not recognized.
I'm putting here a link that they have referenced to install vector on nixos
https://vector.dev/docs/setup/installation/operating-systems/nixos/
I tried with their default configuration file that they give and I still get the configuration error.
There is something that I can't see that I'm missing.

thoughtpolice · 2020-11-12T19:57:59Z

I think the error you're seeing with file etc is a result of #101778, which should be merged and fixed. Can you test this if you get a chance? Thanks.

happysalada · 2020-11-13T00:52:23Z

@thoughtpolice amazing, updating the channel fixed the initial problem, thanks!

now I get this new error.

Nov 13 00:35:44.887 ERROR vector: Configuration error: "/nix/store/qiw20ic94pzsklar8cbh336dpxj0va8j-vector.toml": unknown variant `journald`, expected one of `docker`, `file`, `generator`, `http`, `internal_metrics`, `kafka`, `logplex`, `prometheus`, `socket`, `splunk_hec`, `statsd`, `stdin`, `syslog`, `vector` for key `sources.input.type`

After checking at journald, vector should be added in a supplementary group if it wants to access journald data. I thought that might be the problem. I added a feature flag for journaldAccess. However even after trying that, I still run into the same error
https://vector.dev/docs/reference/sources/journald/#user-permissions

I checked into the default sources included, and it looks like journald is there
https://github.com/timberio/vector/blob/0f63b7efa02325d97a18ba5ed548c6ded8004131/Cargo.toml#L297
doing a git blame shows it was added 9 months ago
https://github.com/timberio/vector/blame/0f63b7efa02325d97a18ba5ed548c6ded8004131/Cargo.toml#L297

I'm not sure why this is happening. I'm open to test things if anybody has ideas.

happysalada · 2020-11-18T02:24:12Z

@johanot I see that you have been using vector recently. I'm trying to get a service merged to nixos. Have you been using it with journald?
The journald source seems to be give a configuration error. You might have an opinion on that.

happysalada · 2020-11-18T02:51:19Z

It looks like the feature is overridable in the package. I don't know how to build it with a custom set of features. If anybody knows, I'm interested in trying with different feature sets. (at least with the default features) just to see if it changes anything.

happysalada · 2020-11-20T22:47:20Z

@johanot Thanks again for that fix, it enabled us to move to the next step!

The next error is the following

Nov 20 22:12:39 nixos-cloud-test vector[6138]: Nov 20 22:12:39.808 ERROR vector::topology: Configuration error: Source "input": data_dir "/var/lib/vector/" does not exist

After looking a little bit, vector needs to have a writable folder
https://vector.dev/docs/reference/sources/journald/#checkpointing
by default it's "/var/lib/vector".
There are two ways to configure it apparently. I think as a first step, it would be better to just enable the global config to be changeable.
I was thinking of writing something like
dataDir = attrByPath [ "settings" "dataDir" ] "/var/lib/vector" cfg;
and then in the serviceConfig
ExecStartPre = "${pkgs.coreutils}/bin/mkdir -p ${dataDir}";
but of course then I get

Nov 20 22:39:07 nixos-cloud-test mkdir[9467]: /nix/store/ffw9vziqw2p4rpm2zwgi5r3ymynd54d8-coreutils-8.32/bin/mkdir: cannot create directory ‘/var/lib/vector’: Permission denied

I'm wondering what is the way to do this usually on nix? Is using the "/var/lib/vector" folder a bad idea?
should another folder be used?
Is using the ExecStartPre the right place to mkdir -p?
@aanderse perhaps you have an opinion on this.

aanderse · 2020-11-21T20:12:02Z

StateDirectory = "vector"; can create /var/lib/vector for you. I'm a little bit surprised the upstream unit doesn't include this directive. You might consider a PR to add it for them 🤷‍♂️

johanot · 2020-11-21T20:30:35Z

@aanderse

I'm a little bit surprised the upstream unit doesn't include this directive.

I had the pleasure of working with a Debian 9 box a couple of days ago, which is not eol until 2022. Unfortunately, Debian 9 ships with systemd 232, and StateDirectory was introduced in systemd 235. Since Vector ships deb-packages, they might simply be trying to be compatible with Debian stable - even though 232 is from 2016 :/.

johanot · 2020-11-21T20:43:43Z

@happysalada Good job working on an upstream module! 👍

Have you considered using vector validate for buildtime validation of Vector config?

I do something like this in my private module:

validateConfig = file: pkgs.runCommand "validate-vector-conf" {} ''
    ${pkgs.vector}/bin/vector validate --no-topology --no-environment "${file}"
    ln -s "${file}" "$out"
  '';

ExecStart="${pkgs.vector}/bin/vector --config ${validateConfig conf}"

happysalada · 2020-11-23T03:37:32Z

@johanot thanks for the validation, great idea!

@aanderse thanks for the stateDirectory!

Now I don't have any more failures on start. I still have a problem with the file sink though.
With the file sink you need to define a path to send the logs to a file. The idea is to keep a copy for backup for example.
I got the following error
Nov 23 02:50:30.096 ERROR vector::sinks::file: Unable to open the file. path=b\"day-23.log\" error=Permission denied (os error 13)
I tried the following paths with the same outcome
./logs/day-%d.log, day-%d.log, /root/day-%d.log
having the path be /var/lib/vector/day-%d.log worked of course. Is that the cannonical behaviour (writing files there)?
If using that folder is what makes the most sense, then I think this is ready to merge.

happysalada · 2020-11-27T11:22:41Z

@GrahamcOfBorg build vector

happysalada · 2020-11-27T11:23:45Z

@aanderse @thoughtpolice I think this is ready to merge when you have a moment.

happysalada · 2020-11-28T01:47:36Z

@GrahamcOfBorg build eval

happysalada · 2020-11-28T01:48:14Z

@GrahamcOfBorg eval

happysalada · 2020-11-28T01:49:37Z

@GrahamcOfBorg eval-nixpkgs-tarball

happysalada · 2020-11-28T01:51:04Z

@GrahamcOfBorg eval nixpkgs tarball

thoughtpolice · 2020-11-29T12:10:06Z

@happysalada This looks fine and I'm happy to merge it. Do you think we could add a test for this as well? It would help ensure regressions don't go unnoticed. (If you don't want to add one in this PR so it can just go in, please say so and we can merge it and you can just follow up.) Thanks.

nixos/modules/services/logging/vector.nix

happysalada · 2020-11-30T03:05:43Z

@thoughtpolice I added a file sink test.
I based the script based on rsyslogd that @aanderse did.
About adding a maintainer, I didn't know If I should add myself as a maintainer in the same PR. So I put you as a first step.
I've never been a maintainer of nixpkgs, so I thought, adding myself as a maintainer would have to be a separate PR.

Of course if you have other tests in mind, let me know and I can try to create scenarii for them. I couldn't think of something other than the file sink.

srhb · 2020-11-30T06:20:36Z

@happysalada Adding yourself as a maintainer in this PR is both allowed and appreciated -- just make it a separate commit :)

happysalada · 2020-11-30T06:45:33Z

@srhb Thanks for the heads up. I'll squash my commits and will also add myself in a separate commit.

happysalada · 2020-12-02T03:53:19Z

@thoughtpolice When you have some time, can you have a look at this again?

thoughtpolice · 2020-12-02T09:30:06Z

@happysalada Thanks for the ping, this looks good!

(I'll go ahead and hit the merge button — I think all of @aanderse's comments have been addressed, just GitHub didn't pick up on them. Thanks again!)

ofborg bot added 6.topic: nixos 8.has: module (update) 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 10.rebuild-linux: 1 labels Nov 11, 2020

aanderse requested changes Nov 11, 2020

View reviewed changes

happysalada requested a review from aanderse November 12, 2020 01:17

happysalada mentioned this pull request Nov 18, 2020

NixOS: Configuration error: unknown variant journald vectordotdev/vector#5097

Closed

johanot mentioned this pull request Nov 18, 2020

vector: include the "unix" cargo feature if targetPlatform isUnix #104179

Merged

10 tasks

thoughtpolice approved these changes Nov 29, 2020

View reviewed changes

aanderse reviewed Nov 29, 2020

View reviewed changes

nixos/modules/services/logging/vector.nix Outdated Show resolved Hide resolved

happysalada requested a review from aanderse November 30, 2020 03:05

happysalada requested a review from thoughtpolice November 30, 2020 03:05

nixos/vector: add module

627dfec

happysalada force-pushed the add_vector branch from 98d395f to 627dfec Compare November 30, 2020 07:22

add happysalada as maintainer

85767db

thoughtpolice approved these changes Dec 2, 2020

View reviewed changes

thoughtpolice merged commit 652ac69 into NixOS:master Dec 2, 2020

happysalada deleted the add_vector branch April 28, 2023 19:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos/vector: add module #103393

nixos/vector: add module #103393

happysalada commented Nov 11, 2020

aanderse left a comment

aanderse Nov 11, 2020

aanderse Nov 11, 2020

aanderse Nov 11, 2020

aanderse Nov 11, 2020

aanderse Nov 11, 2020

aanderse Nov 11, 2020

happysalada commented Nov 12, 2020

thoughtpolice commented Nov 12, 2020

happysalada commented Nov 13, 2020

happysalada commented Nov 18, 2020

happysalada commented Nov 18, 2020

happysalada commented Nov 20, 2020

aanderse commented Nov 21, 2020

johanot commented Nov 21, 2020

johanot commented Nov 21, 2020

happysalada commented Nov 23, 2020

happysalada commented Nov 27, 2020

happysalada commented Nov 27, 2020

happysalada commented Nov 28, 2020

happysalada commented Nov 28, 2020

happysalada commented Nov 28, 2020

happysalada commented Nov 28, 2020

thoughtpolice commented Nov 29, 2020

happysalada commented Nov 30, 2020

srhb commented Nov 30, 2020

happysalada commented Nov 30, 2020

happysalada commented Dec 2, 2020

thoughtpolice commented Dec 2, 2020

nixos/vector: add module #103393

nixos/vector: add module #103393

Conversation

happysalada commented Nov 11, 2020

Motivation for this change

Things done

aanderse left a comment

Choose a reason for hiding this comment

aanderse Nov 11, 2020

Choose a reason for hiding this comment

aanderse Nov 11, 2020

Choose a reason for hiding this comment

aanderse Nov 11, 2020

Choose a reason for hiding this comment

aanderse Nov 11, 2020

Choose a reason for hiding this comment

aanderse Nov 11, 2020

Choose a reason for hiding this comment

aanderse Nov 11, 2020

Choose a reason for hiding this comment

happysalada commented Nov 12, 2020

thoughtpolice commented Nov 12, 2020

happysalada commented Nov 13, 2020

happysalada commented Nov 18, 2020

happysalada commented Nov 18, 2020

happysalada commented Nov 20, 2020

aanderse commented Nov 21, 2020

johanot commented Nov 21, 2020

johanot commented Nov 21, 2020

happysalada commented Nov 23, 2020

happysalada commented Nov 27, 2020

happysalada commented Nov 27, 2020

happysalada commented Nov 28, 2020

happysalada commented Nov 28, 2020

happysalada commented Nov 28, 2020

happysalada commented Nov 28, 2020

thoughtpolice commented Nov 29, 2020

happysalada commented Nov 30, 2020

srhb commented Nov 30, 2020

happysalada commented Nov 30, 2020

happysalada commented Dec 2, 2020

thoughtpolice commented Dec 2, 2020