url-bot-rs: init at 0.3.1; nixos/url-bot-rs: init #105315

mweinelt · 2020-11-29T03:52:41Z

Motivation for this change

Add some sorely needed metadata to IRC conversations.

<@hexa-> https://github.com/NixOS/nixpkgs/pull/105315
<urlmetadata> ⤷ url-bot-rs: init at 0.3.1; nixos/url-bot-rs: init by mweinelt · Pull Request #105315 · NixOS/nixpkgs · GitHub

Things done

  NAME                                                        DESCRIPTION                                                                    EXPOSURE
✗ PrivateNetwork=                                             Service has access to the host's network                                            0.5
✓ User=/DynamicUser=                                          Service runs under a transient non-root user identity                                  
✓ CapabilityBoundingSet=~CAP_SET(UID|GID|PCAP)                Service cannot change UID/GID identities/capabilities                                  
✓ CapabilityBoundingSet=~CAP_SYS_ADMIN                        Service has no administrator privileges                                                
✓ CapabilityBoundingSet=~CAP_SYS_PTRACE                       Service has no ptrace() debugging abilities                                            
✗ RestrictAddressFamilies=~AF_(INET|INET6)                    Service may allocate Internet sockets                                               0.3
✓ RestrictNamespaces=~CLONE_NEWUSER                           Service cannot create user namespaces                                                  
✓ RestrictAddressFamilies=~…                                  Service cannot allocate exotic sockets                                                 
✓ CapabilityBoundingSet=~CAP_(CHOWN|FSETID|SETFCAP)           Service cannot change file ownership/access mode/capabilities                          
✓ CapabilityBoundingSet=~CAP_(DAC_*|FOWNER|IPC_OWNER)         Service cannot override UNIX file/IPC permission checks                                
✓ CapabilityBoundingSet=~CAP_NET_ADMIN                        Service has no network configuration privileges                                        
✓ CapabilityBoundingSet=~CAP_SYS_MODULE                       Service cannot load kernel modules                                                     
✓ CapabilityBoundingSet=~CAP_SYS_RAWIO                        Service has no raw I/O access                                                          
✓ CapabilityBoundingSet=~CAP_SYS_TIME                         Service processes cannot change the system clock                                       
✗ DeviceAllow=                                                Service has a device ACL with some special devices                                  0.1
✗ IPAddressDeny=                                              Service does not define an IP address allow list                                    0.2
✓ KeyringMode=                                                Service doesn't share key material with other services                                 
✓ NoNewPrivileges=                                            Service processes cannot acquire new privileges                                        
✓ NotifyAccess=                                               Service child processes cannot alter service state                                     
✓ PrivateDevices=                                             Service has no access to hardware devices                                              
✓ PrivateMounts=                                              Service cannot install system mounts                                                   
✓ PrivateTmp=                                                 Service has no access to other software's temporary files                              
✓ PrivateUsers=                                               Service does not have access to other users                                            
✓ ProtectClock=                                               Service cannot write to the hardware clock or system clock                             
✓ ProtectControlGroups=                                       Service cannot modify the control group file system                                    
✓ ProtectHome=                                                Service has no access to home directories                                              
✓ ProtectKernelLogs=                                          Service cannot read from or write to the kernel log ring buffer                        
✓ ProtectKernelModules=                                       Service cannot load or read kernel modules                                             
✓ ProtectKernelTunables=                                      Service cannot alter kernel tunables (/proc/sys, …)                                    
✓ ProtectSystem=                                              Service has strict read-only access to the OS file hierarchy                           
✓ RestrictAddressFamilies=~AF_PACKET                          Service cannot allocate packet sockets                                                 
✓ RestrictSUIDSGID=                                           SUID/SGID file creation by service is restricted                                       
✓ SystemCallArchitectures=                                    Service may execute system calls only with native ABI                                  
✓ SystemCallFilter=~@clock                                    System call allow list defined for service, and @clock is not included                 
✓ SystemCallFilter=~@debug                                    System call allow list defined for service, and @debug is not included                 
✓ SystemCallFilter=~@module                                   System call allow list defined for service, and @module is not included                
✓ SystemCallFilter=~@mount                                    System call allow list defined for service, and @mount is not included                 
✓ SystemCallFilter=~@raw-io                                   System call allow list defined for service, and @raw-io is not included                
✓ SystemCallFilter=~@reboot                                   System call allow list defined for service, and @reboot is not included                
✓ SystemCallFilter=~@swap                                     System call allow list defined for service, and @swap is not included                  
✗ SystemCallFilter=~@privileged                               System call allow list defined for service, and @privileged is included             0.2
✗ SystemCallFilter=~@resources                                System call allow list defined for service, and @resources is included              0.2
✓ AmbientCapabilities=                                        Service process does not receive ambient capabilities                                  
✓ CapabilityBoundingSet=~CAP_AUDIT_*                          Service has no audit subsystem access                                                  
✓ CapabilityBoundingSet=~CAP_KILL                             Service cannot send UNIX signals to arbitrary processes                                
✓ CapabilityBoundingSet=~CAP_MKNOD                            Service cannot create device nodes                                                     
✓ CapabilityBoundingSet=~CAP_NET_(BIND_SERVICE|BROADCAST|RAW) Service has no elevated networking privileges                                          
✓ CapabilityBoundingSet=~CAP_SYSLOG                           Service has no access to kernel logging                                                
✓ CapabilityBoundingSet=~CAP_SYS_(NICE|RESOURCE)              Service has no privileges to change resource use parameters                            
✓ RestrictNamespaces=~CLONE_NEWCGROUP                         Service cannot create cgroup namespaces                                                
✓ RestrictNamespaces=~CLONE_NEWIPC                            Service cannot create IPC namespaces                                                   
✓ RestrictNamespaces=~CLONE_NEWNET                            Service cannot create network namespaces                                               
✓ RestrictNamespaces=~CLONE_NEWNS                             Service cannot create file system namespaces                                           
✓ RestrictNamespaces=~CLONE_NEWPID                            Service cannot create process namespaces                                               
✓ RestrictRealtime=                                           Service realtime scheduling access is restricted                                       
✓ SystemCallFilter=~@cpu-emulation                            System call allow list defined for service, and @cpu-emulation is not included         
✓ SystemCallFilter=~@obsolete                                 System call allow list defined for service, and @obsolete is not included              
✓ RestrictAddressFamilies=~AF_NETLINK                         Service cannot allocate netlink sockets                                                
✗ RootDirectory=/RootImage=                                   Service runs within the host's root directory                                       0.1
✓ SupplementaryGroups=                                        Service has no supplementary groups                                                    
✓ CapabilityBoundingSet=~CAP_MAC_*                            Service cannot adjust SMACK MAC                                                        
✓ CapabilityBoundingSet=~CAP_SYS_BOOT                         Service cannot issue reboot()                                                          
✓ Delegate=                                                   Service does not maintain its own delegated control group subtree                      
✓ LockPersonality=                                            Service cannot change ABI personality                                                  
✓ MemoryDenyWriteExecute=                                     Service cannot create writable executable memory mappings                              
✓ RemoveIPC=                                                  Service user cannot leave SysV IPC objects around                                      
✓ RestrictNamespaces=~CLONE_NEWUTS                            Service cannot create hostname namespaces                                              
✓ UMask=                                                      Files created by service are accessible only by service's own user by default          
✓ CapabilityBoundingSet=~CAP_LINUX_IMMUTABLE                  Service cannot mark files immutable                                                    
✓ CapabilityBoundingSet=~CAP_IPC_LOCK                         Service cannot lock memory into RAM                                                    
✓ CapabilityBoundingSet=~CAP_SYS_CHROOT                       Service cannot issue chroot()                                                          
✓ ProtectHostname=                                            Service cannot change system host/domainname                                           
✓ CapabilityBoundingSet=~CAP_BLOCK_SUSPEND                    Service cannot establish wake locks                                                    
✓ CapabilityBoundingSet=~CAP_LEASE                            Service cannot create file leases                                                      
✓ CapabilityBoundingSet=~CAP_SYS_PACCT                        Service cannot use acct()                                                              
✓ CapabilityBoundingSet=~CAP_SYS_TTY_CONFIG                   Service cannot issue vhangup()                                                         
✓ CapabilityBoundingSet=~CAP_WAKE_ALARM                       Service cannot program timers that wake up the system                                  
✓ RestrictAddressFamilies=~AF_UNIX                            Service cannot allocate local sockets                                                  

→ Overall exposure level for url-bot-rs.service: 1.3 OK 🙂

@nuxeh

mweinelt · 2020-11-29T03:56:45Z

I'd totally be down for writing a test, but we'd need to spin up some IRCd. If someone can recommend an IRCd that I can spin up with just a few lines of config, that'd be great.

SuperSandro2000 · 2020-11-29T05:05:45Z

Patch to make it build on darwin:

diff --git a/pkgs/applications/networking/irc/url-bot-rs/default.nix b/pkgs/applications/networking/irc/url-bot-rs/default.nix
index 97445029dce5..5fe7f631bc2b 100644
--- a/pkgs/applications/networking/irc/url-bot-rs/default.nix
+++ b/pkgs/applications/networking/irc/url-bot-rs/default.nix
@@ -1,9 +1,11 @@
-{ lib
+{ stdenv
 , rustPlatform
 , fetchFromGitHub
 , pkg-config
 , openssl
 , sqlite
+, libiconv
+, Security
 }:
 
 rustPlatform.buildRustPackage rec {
@@ -21,13 +23,13 @@ rustPlatform.buildRustPackage rec {
 
   nativeBuildInputs = [ pkg-config ];
 
-  buildInputs = [ openssl sqlite ];
+  buildInputs = [ openssl sqlite ] ++ stdenv.lib.optionals stdenv.isDarwin [ libiconv Security ];
 
   preBuild = ''
     export HOME=$TMPDIR
   '';
 
-  meta = with lib; {
+  meta = with stdenv.lib; {
     description = "Minimal IRC URL bot in Rust";
     homepage = "https://github.com/nuxeh/url-bot-rs";
     license = licenses.isc;
diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index 4df81a9efa63..0a8666e369a7 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -7901,7 +7901,9 @@ in
 
   uriparser = callPackage ../development/libraries/uriparser {};
 
-  url-bot-rs = callPackage ../applications/networking/irc/url-bot-rs {};
+  url-bot-rs = callPackage ../applications/networking/irc/url-bot-rs {
+    inherit (darwin.apple_sdk.frameworks) Security;
+  };
 
   urlscan = callPackage ../applications/misc/urlscan { };

nuxeh · 2020-11-29T11:32:54Z

This is awesome, thanks for taking the time to package url-bot-rs!

@mweinelt, totally agree about testing, this is something I've been thinking about for a long time, but have no personal experience with nix tests, so wasn't sure about the correct way to do it.

For my own testing I tend to use inspircd, since it can simply be run on the command line to give you a working ircd without having to play around with a configuration too much, if at all, and out of all the ircds i tried it seemed by far the easiest in this regard. I had guessed it could be easy enough to run it in a nix-shell for testing, but hadn't thought much further than that. However, it's not in nixpkgs, although I do have a derivation written for it that hasn't been submitted anywhere yet. Maybe this could be a good reason to get inspircd submitted too.

aanderse

@mweinelt exceptional work as always! I left a few optional suggestions that are worth bonus points, if you're interested.

nixos/modules/services/networking/url-bot-rs.nix

mweinelt · 2020-11-29T20:48:08Z

Added StateDirectory, so the optional sqlite database backend has a place to store its state.

nuxeh · 2020-11-30T20:37:52Z

@mweinelt I made a PR mweinelt#4 against this branch to update the module and the config generation to make it feature complete, allowing a number of things to be configured, but primarily for connecting to multiple networks and enabling other additional features on a per-network basis.

Also, tested thoroughly using nixos-shell.

SuperSandro2000 · 2020-12-27T19:34:15Z

Result of nixpkgs-review pr 105315 run on x86_64-linux 1

1 package blacklisted:

tests.nixos-functions.nixos-test

1 package built:

url-bot-rs

stale · 2021-06-26T00:12:04Z

I marked this as stale due to inactivity. → More info

nuxeh · 2021-06-26T09:44:42Z

Would like to get this merged. I see the benefit of automatic toml generation from expressions - seems to be the modern way, while also feel an ergonomic version as I proposed in the PR i made, is also nice to have, even if the implementation is more complex, it's a conundrum for me.

However, one essential feature as I see it is being able to specify multiple networks/connections in the nix. If I can get it to work I might suggest updates to this PR to do this, using the toml generation.

nuxeh · 2021-06-26T09:47:43Z

Btw... it's currently not possible to connect to multiple networks, since the expression will only generate a single configuration, but with the way url-bot-rs is designed, you currently need multiple configuration files to do this.

mweinelt · 2021-06-26T10:25:22Z

Sorry, I was a bit overwhelmed at the time when you put up that huge pull request to my fork and was thinking about how to reconcile that with RFC42 (https://github.com/NixOS/rfcs/blob/master/rfcs/0042-config-option.md) and that is still an open question to me.

nuxeh · 2021-06-26T15:16:39Z

Makes total sense, i felt conflicted about the PR myself. Have never seen this RFC, but glad there are some guidelines in this area, otherwise it does seem to be a bit open ended how you implement modules.

I think it certainly seems that the expression to toml thing seems like the way to go, I'll see if I can knock up something to allow multiple configs using it.

Something like services.url-bot-rs.configs."name" = {};. Since I guess you can have an attribute set of the to-toml type i guess that should be straight forward.

pkgs/applications/networking/irc/url-bot-rs/default.nix

nixos/modules/services/networking/url-bot-rs.nix

pkgs/applications/networking/irc/url-bot-rs/default.nix

mweinelt · 2021-12-13T18:47:30Z

From mweinelt#4 (comment)

The key thing it addresses, however, which I do think is essential (at least for my uses), is the facility to specify connections to multiple servers. As a service this is essential for anything more than a simple single irc network deployment.

Is there a good example of a module that supports multiple instances? From what I remember people always just went with NixOS containers to side-step this issue.

nuxeh · 2021-12-13T19:00:52Z

@mweinelt i'm sure i have come across it many times, although the only one i can think of off the top of my head is nginx, for example. However I'm sure all of the cases I have seen are "old school" nix modules.

Since url-bot-rs natively supports multiple connections with the same binary, it seems wasteful to spawn multiple copies, also in containers, to me.

However, another way to sidestep this would be for url-bot-rs to accept a single config containing multiple networks, in a future release. I'd be quite happy to do that, it seems like a simpler option overall, and it would be better to get this merged sooner rather than later!

mweinelt · 2021-12-13T19:04:58Z

Sounds good to me.

pkgs/applications/networking/irc/url-bot-rs/default.nix

mweinelt · 2021-12-14T19:20:54Z

The 0.3.2 update chokes reproducibly on a test:

---- plugins::vimeo::tests::evaluate stdout ----
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 98, kind: AddrInUse, message: "Address already in use" }', src/plugins/vimeo.rs:124:56
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'plugins::vimeo::tests::evaluate' panicked at 'called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Request, url: Url { scheme: "http", host: Some(Ipv4(127.0.0.1)), port: Some(28286), path: "/videos/53603603", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })) }', src/plugins/vimeo.rs:135:64


failures:
    plugins::vimeo::tests::evaluate

test result: FAILED. 54 passed; 1 failed; 2 ignored; 0 measured; 0 filtered out; finished in 22.39s

nuxeh · 2021-12-14T20:21:40Z

@mweinelt 0.3.2 hasn't changed, it's just outstanding stuff that hasn't had a release yet. But indeed, have noticed this too, I think it is intermittent. Perhaps something about the port it uses in the test.

I have multi configs in nuxeh/url-bot-rs#420 (!) which appears to work, just having a final look over it, and still some tests to sort out, but I think this could be the basis of a new version.

mweinelt · 2021-12-14T21:12:11Z

https://github.com/nuxeh/url-bot-rs/blob/1ccfadcf79b75ec186b7052fb06f028fc38edc3e/src/plugins/vimeo.rs#L119-L124
https://github.com/nuxeh/url-bot-rs/blob/bade0f41fc976be4e370d53b5247824b88b64a63/src/http.rs#L619-L627

These two want to bind to the same port, maybe that is the problem?

nuxeh · 2021-12-15T00:32:52Z

So now there's https://github.com/nuxeh/url-bot-rs/releases/tag/v0.4.0-rc.1. Still think a bit more testing is needed which I can't do till tomorrow, but it's a start at least.

Wondering if a HashMap TOML serialisation might map better to a Nix expression, rather than a vector, maybe I'll play around with that some more, also.

nuxeh · 2021-12-15T12:25:29Z

https://github.com/nuxeh/url-bot-rs/releases/tag/v0.4.0-rc.2 is a hash map, think this is much better. Also, incase it tickles anyone's fancy I'll work on getting shell completion generation quite soon.

url-bot-rs is a minimal IRC bot that looks up URLs posted on IRC and extracts title and other metadata and relays it back to the IRC channel it was originally mentioned.

nuxeh · 2022-01-28T11:48:59Z

@mweinelt I made mweinelt#9, which is the simplest method i could come up with to allow both the existing module behaviour, and multiple network configurations which are available in url-bot-rs since v0.4.0, and by now have been reasonably well tested.

Would be interested to hear what you think. I hope it doesn't obfuscate things too much - i'm thinking mainly that the example I've added for the settings option is a multiple network configuration, which could be confusing, if it were possible to have multiple examples, i'd put a single network there too

mweinelt · 2022-01-28T11:55:49Z

I hope we can update the the module, so we can make the config look like services.url-bot-rs.settings.<name>.connection.server, where name would be the network instance.

That would be an types.attrsOf types.submodule construction IIRC.

ofborg bot added 6.topic: nixos 8.has: module (update) 8.has: package (new) 11.by: package-maintainer 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 10.rebuild-linux: 1-10 labels Nov 29, 2020

aanderse reviewed Nov 29, 2020

View reviewed changes

nixos/modules/services/networking/url-bot-rs.nix Show resolved Hide resolved

nixos/modules/services/networking/url-bot-rs.nix Outdated Show resolved Hide resolved

mweinelt force-pushed the url-bot-rs branch from b2b37b8 to 7bef954 Compare November 29, 2020 15:47

mweinelt mentioned this pull request Nov 29, 2020

Submit Nix packages to Nixpkgs/NUR nuxeh/url-bot-rs#342

Open

mweinelt force-pushed the url-bot-rs branch from 7bef954 to 02cedc1 Compare November 29, 2020 20:44

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 26, 2021

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 26, 2021

andir reviewed Dec 13, 2021

View reviewed changes

pkgs/applications/networking/irc/url-bot-rs/default.nix Outdated Show resolved Hide resolved

mweinelt force-pushed the url-bot-rs branch from 02cedc1 to 7d7a700 Compare December 13, 2021 16:59

aanderse reviewed Dec 13, 2021

View reviewed changes

nixos/modules/services/networking/url-bot-rs.nix Outdated Show resolved Hide resolved

andir reviewed Dec 13, 2021

View reviewed changes

pkgs/applications/networking/irc/url-bot-rs/default.nix Outdated Show resolved Hide resolved

mweinelt force-pushed the url-bot-rs branch from 7d7a700 to 3aaca76 Compare December 13, 2021 17:34

aanderse mentioned this pull request Dec 13, 2021

url-bot-rs: update module to make feature-complete mweinelt/nixpkgs#4

Closed

10 tasks

mweinelt commented Dec 13, 2021

View reviewed changes

pkgs/applications/networking/irc/url-bot-rs/default.nix Show resolved Hide resolved

mweinelt force-pushed the url-bot-rs branch from 1378085 to 4e5f0ca Compare December 14, 2021 02:32

mweinelt force-pushed the url-bot-rs branch 3 times, most recently from 690c9a8 to 8c9c4c9 Compare January 24, 2022 14:53

mweinelt added 2 commits January 27, 2022 01:44

url-bot-rs: init at 0.4.0

ce1a4ca

nixos/url-bot-rs: init

dcb12dc

url-bot-rs is a minimal IRC bot that looks up URLs posted on IRC and extracts title and other metadata and relays it back to the IRC channel it was originally mentioned.

mweinelt force-pushed the url-bot-rs branch from 8c9c4c9 to dcb12dc Compare January 27, 2022 00:44

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 30, 2022

wegank added the 2.status: merge conflict label Mar 20, 2024

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 20, 2024

wegank marked this pull request as draft March 20, 2024 22:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

url-bot-rs: init at 0.3.1; nixos/url-bot-rs: init #105315

url-bot-rs: init at 0.3.1; nixos/url-bot-rs: init #105315

mweinelt commented Nov 29, 2020 •

edited

mweinelt commented Nov 29, 2020

SuperSandro2000 commented Nov 29, 2020

nuxeh commented Nov 29, 2020

aanderse left a comment

mweinelt commented Nov 29, 2020 •

edited

nuxeh commented Nov 30, 2020 •

edited

SuperSandro2000 commented Dec 27, 2020

stale bot commented Jun 26, 2021

nuxeh commented Jun 26, 2021

nuxeh commented Jun 26, 2021

mweinelt commented Jun 26, 2021 •

edited

nuxeh commented Jun 26, 2021

mweinelt commented Dec 13, 2021

nuxeh commented Dec 13, 2021

mweinelt commented Dec 13, 2021

mweinelt commented Dec 14, 2021

nuxeh commented Dec 14, 2021

mweinelt commented Dec 14, 2021

nuxeh commented Dec 15, 2021

nuxeh commented Dec 15, 2021

nuxeh commented Jan 28, 2022

mweinelt commented Jan 28, 2022 •

edited

url-bot-rs: init at 0.3.1; nixos/url-bot-rs: init #105315

Are you sure you want to change the base?

url-bot-rs: init at 0.3.1; nixos/url-bot-rs: init #105315

Conversation

mweinelt commented Nov 29, 2020 • edited

Motivation for this change

Things done

mweinelt commented Nov 29, 2020

SuperSandro2000 commented Nov 29, 2020

nuxeh commented Nov 29, 2020

aanderse left a comment

Choose a reason for hiding this comment

mweinelt commented Nov 29, 2020 • edited

nuxeh commented Nov 30, 2020 • edited

SuperSandro2000 commented Dec 27, 2020

stale bot commented Jun 26, 2021

nuxeh commented Jun 26, 2021

nuxeh commented Jun 26, 2021

mweinelt commented Jun 26, 2021 • edited

nuxeh commented Jun 26, 2021

mweinelt commented Dec 13, 2021

nuxeh commented Dec 13, 2021

mweinelt commented Dec 13, 2021

mweinelt commented Dec 14, 2021

nuxeh commented Dec 14, 2021

mweinelt commented Dec 14, 2021

nuxeh commented Dec 15, 2021

nuxeh commented Dec 15, 2021

nuxeh commented Jan 28, 2022

mweinelt commented Jan 28, 2022 • edited

mweinelt commented Nov 29, 2020 •

edited

mweinelt commented Nov 29, 2020 •

edited

nuxeh commented Nov 30, 2020 •

edited

mweinelt commented Jun 26, 2021 •

edited

mweinelt commented Jan 28, 2022 •

edited