zeek: add zeek module and fix zeekctl to deploy it. #86494

GTrunSec · 2020-05-01T19:38:27Z

Motivation for this change

Re-open PR by #86435.

Things done

GTrunSec · 2020-05-01T20:17:44Z

TEST service.zeek setting

  services.zeek = {
    enable = true;
    standalone = true;
    interface = "eno1";
    listenAddress = "localhost";
    privateScript = ''
    @load /home/gtrun/project/hardenedlinux-zeek-script/scripts/zeek-query.zeek
    '';
  };

TEST systemctl --user status zeek.service (https://github.com/hardenedlinux/debian-nix-manager/blob/master/modules/zeek.nix) for home-manager.

● zeek.service
   Loaded: loaded (/nix/store/w4sr0hzjsnj38sd3x3znm2i9043w1yxx-zeek.service/zeek.service; linked; vendor preset: enabled)
   Active: active (running) since Sat 2020-05-02 00:00:29 EDT; 1min 10s ago
  Process: 12297 ExecStartPre=/nix/store/kgp3vq8l9yb8mzghbw83kyr3f26yqvsz-bash-4.4-p23/bin/bash /nix/store/bibwqxc59ggqsl06lplng7wcfswgyv
 Main PID: 12320 (bash)
   CGroup: /user.slice/user-1000.slice/user@1000.service/zeek.service
           ├─12320 /nix/store/kgp3vq8l9yb8mzghbw83kyr3f26yqvsz-bash-4.4-p23/bin/bash /nix/store/zrnjfvn6hs93ri346n5s6zwndbs8chnn-zeek-one
           ├─12384 /nix/store/kgp3vq8l9yb8mzghbw83kyr3f26yqvsz-bash-4.4-p23/bin/bash /var/lib/zeek/scripts/run-zeek -1 -i enp0s3 -U .stat
           ├─12390 /nix/store/vh1f9s48sjaw0i0fy1d5bq01ggb5j1l1-zeek-3.0.5/bin/zeek -i enp0s3 -U .status -p zeekctl -p zeekctl-live -p sta
           └─12418 sleep infinity
lines 1-10/10 (END)

PATH env_vars check

==== .cmdline
-i enp0s3 -U .status -p zeekctl -p zeekctl-live -p standalone -p local -p zeek local.zeek zeekctl zeekctl/standalone zeekctl/auto

==== .env_vars
PATH=/nix/store/vh1f9s48sjaw0i0fy1d5bq01ggb5j1l1-zeek-3.0.5/bin:/var/lib/zeek/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/test/.nix-profile/bin/
ZEEKPATH=/var/lib/zeek/spool/installed-scripts-do-not-touch/site::/var/lib/zeek/spool/installed-scripts-do-not-touch/auto:/nix/store/vh1f9s48sjaw0i0fy1d5bq01ggb5j1l1-zeek-3.0.5/share/zeek:/nix/store/vh1f9s48sjaw0i0fy1d5bq01ggb5j1l1-zeek-3.0.5/share/zeek/policy:/nix/store/vh1f9s48sjaw0i0fy1d5bq01ggb5j1l1-zeek-3.0.5/share/zeek/site
CLUSTER_NODE=

==== .status
RUNNING [net_run]

sudo systemctl status zeek.service

GTrunSec · 2020-05-02T07:17:14Z

@tobim @marsam @pSub. Sorry, I can not deploy the Zeekctl on NixOS. I still got an error output like this
Error: cannot create working directory for zeek (https://github.com/zeek/zeekctl/blob/7e65a34905ec9684c442da5f737fe75beb94aae6/ZeekControl/control.py#L175

if you have time do a PR test or debug this issue. I really confuse what happed about this error outpout. I can deploy it and works well on other Linux which installed nix.

tobim · 2020-05-02T12:46:11Z

@GTrunSec I appreciate you put in so much work into that. I hope to be able to free up some time to help debug this during the next couple of days.

I suspect the error you get is a result of zeekctl trying to create a folder in the PREFIX where zeek is installed, which in case of nixos is in the store, which is mounted read-only by default.

GTrunSec · 2020-05-02T22:39:41Z

@tobim first of all, Thank you so much to help me debug it.

I think this issue from Spool/zeek could not be created.

invalid Spool/zeek hardlink to logs/current

It did not generate Zeek directory under Spool dir

let us look at the zeekctl testing what it did.

Create Zeek's working directory, chdir to that dir, and create rotated logs
mkdir $ZEEKCTL_INSTALL_PREFIX/spool/zeek
cd $ZEEKCTL_INSTALL_PREFIX/spool/zeek
cat > ${origtestlog} << EOF
This is a test zeekctltest.log
EOF

Start process.

verify that these files don't already exist before the start command runs
test ! -e $ZEEKCTL_INSTALL_PREFIX/spool/zeek
test ! -e $ZEEKCTL_INSTALL_PREFIX/spool/stats.log
zeekctl start > start.out
the start command creates some files (these are NOT created by zeek itself)
while read line; do
test -e $ZEEKCTL_INSTALL_PREFIX/$line
done << EOF
spool/zeek/.cmdline
spool/zeek/.env_vars
spool/zeek/.pid
spool/zeek/.startup
spool/zeek/stderr.log
spool/zeek/stdout.log

On Debian OS I got all of the outputs in here.
Valid Zeek directory and hard link.

GTrunSec · 2020-05-03T08:33:14Z

I even can not deploy the standalone mode of Zeek which installed without the nix-store on NixOS.

test nix-shell
https://github.com/hardenedlinux/debian-nix-manager/blob/master/pkgs/zeek/zeek-shell/zeek-shell.nix
nix-shell zeek-shell.nix --command "make -j8 all && sudo make install"

@tobim if you can deploy it whatever without nix-store or not. Please let me know that. (I can surely no any dynamic files will be generated in nix-store.) Also, you can check the config file which in /var/lib/zeek/spool/zeekctl-config.sh. All of the dynamic file options have been removed to /var/lib/zeek path.

tobim · 2020-05-03T11:15:22Z

It looks like zeekctl is trying to run ['mkdir', '-p', '/var/lib/zeek/spool/zeek'] through ssh, as if the host IP belongs to a worker node, even when using type=standalone in node.cfg.

GTrunSec · 2020-05-04T07:55:15Z

@tobim

I have no idea how to fix this issue for Zeekctl.

if we did not solve the Zeekctl problem which running on NixOS(sandbox mode). I'm going to write a Systemd of Zeek for NixOS that can be running with some arguments of the interface instead of zeekctl to deploy it.

One of another solution which Without Zeekctl deploy
systemd workindDirecotry /var/lib/zeek/logs

../scripts/run-zeek -1 -i eno1 -U .status -p zeekctl -p zeekctl-live -p standalone -p local -p zeek local.zeek zeekctl zeekctl/standalone zeekctl/auto

GTrunSec · 2020-05-08T03:15:07Z

@tobim @marsam @pSub. ready to take a test for the maintainer. everything goes well.
Zeek's module test setting:

services.zeek = {
  enable = true;
  standalone = true;
  interface = "eno1";
  listenAddress = "localhost";
  privateScript = ''                                                            
  @load /home/gtrun/project/hardenedlinux-zeek-script/scripts/zeek-query.zeek   
  '';
};

check privateScript loading Status:

more /var/lib/zeek/logs/current/loaded_scripts.log      
  /home/gtrun/project/hardenedlinux-zeek-script/scripts/zeek-query.zeek
/nix/store/g4al63qrpx8ckf6bkimam6w3wps1imav-zeek-3.0.5/share/zeek/zeekctl/standalone.zeek
  /var/lib/zeek/spool/installed-scripts-do-not-touch/auto/standalone-layout.zeek
  /nix/store/g4al63qrpx8ckf6bkimam6w3wps1imav-zeek-3.0.5/share/zeek/policy/misc/trim-trace-file.zeek
/nix/store/g4al63qrpx8ckf6bkimam6w3wps1imav-zeek-3.0.5/share/zeek/zeekctl/auto.zeek
  /var/lib/zeek/spool/installed-scripts-do-not-touch/auto/local-networks.zeek

sudo systemctl status zeek.service

zeek.service - Zeek Daemon
     Loaded: loaded (/nix/store/nal9g7ih8idk27pajvdxbbid4y18mgsc-unit-zeek.service/zeek.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2020-05-07 20:07:36 PDT; 1min 6s ago
    Process: 7985 ExecStartPre=/nix/store/xcs2xvn398hgsp0gz3slhlv02qljnhm0-unit-script-zeek-pre-start (code=exited, status=0/SUCCESS)
   Main PID: 8034 (bash)
         IP: 0B in, 0B out
      Tasks: 18 (limit: 4915)
     Memory: 82.4M
        CPU: 18.158s
     CGroup: /system.slice/zeek.service
             ├─8034 /nix/store/kgp3vq8l9yb8mzghbw83kyr3f26yqvsz-bash-4.4-p23/bin/bash /nix/store/zkrr30qd8acjdnwk89gp3fa6jjss4rdg-zeek-o>
             ├─8148 /nix/store/kgp3vq8l9yb8mzghbw83kyr3f26yqvsz-bash-4.4-p23/bin/bash /var/lib/zeek/scripts/run-zeek -1 -i eno1 -U .stat>
             └─8158 /nix/store/g4al63qrpx8ckf6bkimam6w3wps1imav-zeek-3.0.5/bin/zeek -i eno1 -U .status -p zeekctl -p zeekctl-live -p sta>

May 07 20:07:36 nixos bash[8042]: generating standalone-layout.zeek ...
May 07 20:07:36 nixos bash[8042]: generating local-networks.zeek ...
May 07 20:07:36 nixos bash[8042]: generating zeekctl-config.zeek ...
May 07 20:07:36 nixos bash[8042]: generating zeekctl-config.sh ...
May 07 20:07:36 nixos bash[8148]: max memory size         (kbytes, -m) unlimited
May 07 20:07:36 nixos bash[8148]: data seg size           (kbytes, -d) unlimited
May 07 20:07:36 nixos bash[8148]: virtual memory          (kbytes, -v) unlimited
May 07 20:07:36 nixos bash[8148]: core file size          (blocks, -c) unlimited
May 07 20:07:36 nixos bash[8158]: listening on eno1
May 07 20:07:46 nixos bash[8158]: 1588907266.826376 warning in /nix/store/g4al63qrpx8ckf6bkimam6w3wps1imav-zeek-3.0.5/share/zeek/base/mi

GTrunSec · 2020-05-08T05:59:07Z

fixed archive logs (paths)

/var/lib/zeek/logs                                                                                                                       
├── 2020-05-07                                                                                                                           
│   ├── conn.23:57:24-00:00:00.log.gz                                                                                                    
│   ├── conn-summary.23:57:24-00:00:00.log.gz                                                                                            
│   ├── dns.23:57:40-00:00:00.log.gz                                                                                                     
│   ├── files.23:58:54-00:00:00.log.gz                                                                                                   
│   ├── http.23:58:54-00:00:00.log.gz                                                                                                    
│   ├── known_services.23:57:30-00:00:00.log.gz                                                                                          
│   ├── loaded_scripts.23:57:21-00:00:00.log.gz
│   ├── packet_filter.23:57:21-00:00:00.log.gz
│   ├── reporter.23:57:31-00:00:00.log.gz
│   ├── ssh.23:58:07-00:00:00.log.gz
│   ├── ssl.23:57:29-00:00:00.log.gz
│   ├── stats.23:57:21-00:00:00.log.gz
│   └── weird.23:57:21-00:00:00.log.gz
└── current
    ├── capture_loss.log
    ├── conn.log
    ├── dns.log

GTrunSec · 2020-05-09T02:36:43Z

add Option: to override some arguments that alternative installs Zeek's plugin or not.

test package of Zeek setting:

 services.zeek = {
   enable = true;
   standalone = true;
   interface = "eno1";
   listenAddress = "localhost";
   package = pkgs.zeek.override{ KafkaPlugin = true; PostgresqlPlugin = true;};
   privateScript = ''                                                          
   @load /home/gtrun/project/hardenedlinux-zeek-script/scripts/zeek-query.zeek 
   '';
 };

GTrunSec · 2020-05-14T02:08:48Z

Does anyone give me some feedback about this PR? let me know that to keep going deep work for this module on NixOS or not. Make sure I am doing the right direction on the way. @marsam @pSub @tobim Thanks. Sorry to bother you.

GTrunSec · 2020-10-05T23:23:13Z

@ryantm resolved the merge conflict. Thanks for reviewing.

stale · 2021-07-21T01:12:10Z

I marked this as stale due to inactivity. → More info

koalaeagle · 2022-09-02T03:21:46Z

Seems like this is exactly what I was looking for on NixOS. Still learning NixOS, but appreciate the effort trying to sort out a somewhat complicated package/module.

ofborg bot requested review from pSub, tobim and marsam May 1, 2020 19:49

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 10.rebuild-linux: 1-10 10.rebuild-linux: 1 labels May 1, 2020

ofborg bot added 6.topic: nixos 8.has: module (update) and removed 10.rebuild-linux: 1 labels May 2, 2020

GTrunSec mentioned this pull request May 28, 2020

Building and Deploying Zeek with Sandbox AlkenePan/awesome-bro#3

Open

ryantm added 2.status: merge conflict and removed 2.status: merge conflict labels Oct 3, 2020

GTrunSec force-pushed the zeekctl branch from c54403c to b9869bf Compare October 5, 2020 22:56

ofborg bot added 6.topic: nixos and removed 6.topic: nixos 8.has: module (update) 2.status: merge conflict labels Oct 5, 2020

ofborg bot added 8.has: module (update) 10.rebuild-linux: 1 labels Oct 5, 2020

zeek: enable zeekctl and fix zeekctl patch

2d3aa72

ofborg bot removed the 10.rebuild-linux: 1 label Oct 5, 2020

GTrunSec force-pushed the zeekctl branch from dfb503d to 2d3aa72 Compare October 5, 2020 23:13

update zeek services module

9778f6e

ofborg bot added 6.topic: nixos 8.has: module (update) and removed 6.topic: nixos 8.has: module (update) labels Oct 5, 2020

ofborg bot added 10.rebuild-linux: 1 and removed 10.rebuild-linux: 1 labels Oct 5, 2020

SuperSandro2000 added the 2.status: merge conflict label Jan 18, 2021

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 21, 2021

GTrunSec closed this Jul 21, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

zeek: add zeek module and fix zeekctl to deploy it. #86494

zeek: add zeek module and fix zeekctl to deploy it. #86494

GTrunSec commented May 1, 2020 •

edited

GTrunSec commented May 1, 2020 •

edited

GTrunSec commented May 2, 2020 •

edited

tobim commented May 2, 2020

GTrunSec commented May 2, 2020 •

edited

GTrunSec commented May 3, 2020 •

edited

tobim commented May 3, 2020

GTrunSec commented May 4, 2020 •

edited

GTrunSec commented May 8, 2020

GTrunSec commented May 8, 2020 •

edited

GTrunSec commented May 9, 2020 •

edited

GTrunSec commented May 14, 2020

GTrunSec commented Oct 5, 2020

stale bot commented Jul 21, 2021

koalaeagle commented Sep 2, 2022

zeek: add zeek module and fix zeekctl to deploy it. #86494

zeek: add zeek module and fix zeekctl to deploy it. #86494

Conversation

GTrunSec commented May 1, 2020 • edited

Motivation for this change

Things done

GTrunSec commented May 1, 2020 • edited

GTrunSec commented May 2, 2020 • edited

tobim commented May 2, 2020

GTrunSec commented May 2, 2020 • edited

GTrunSec commented May 3, 2020 • edited

tobim commented May 3, 2020

GTrunSec commented May 4, 2020 • edited

GTrunSec commented May 8, 2020

GTrunSec commented May 8, 2020 • edited

GTrunSec commented May 9, 2020 • edited

GTrunSec commented May 14, 2020

GTrunSec commented Oct 5, 2020

stale bot commented Jul 21, 2021

koalaeagle commented Sep 2, 2022

GTrunSec commented May 1, 2020 •

edited

GTrunSec commented May 1, 2020 •

edited

GTrunSec commented May 2, 2020 •

edited

GTrunSec commented May 2, 2020 •

edited

GTrunSec commented May 3, 2020 •

edited

GTrunSec commented May 4, 2020 •

edited

GTrunSec commented May 8, 2020 •

edited

GTrunSec commented May 9, 2020 •

edited