rspamd: add hyperscan, drop obsoleted dependencies #49915

avnik · 2018-11-08T16:07:31Z

Motivation for this change

Add hyperscan support on x86_64 (and package hyperscan as well)
Drop obsoleted gmime dependency
Add options to enable fann/gd/openblas

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

Mic92 · 2018-11-08T16:35:47Z

@GrahamcOfBorg build rspamd

GrahamcOfBorg · 2018-11-08T16:35:56Z

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: rspamd

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-11-08T16:41:03Z

Success on aarch64-linux (full log)

Attempted: rspamd

Partial log (click to expand)

shrinking /nix/store/24z497389g4m5g7k0dd9842kwnym4zql-rspamd-1.8.1/lib/rspamd/librspamd-actrie.so
shrinking /nix/store/24z497389g4m5g7k0dd9842kwnym4zql-rspamd-1.8.1/bin/rspamadm
shrinking /nix/store/24z497389g4m5g7k0dd9842kwnym4zql-rspamd-1.8.1/bin/rspamc
shrinking /nix/store/24z497389g4m5g7k0dd9842kwnym4zql-rspamd-1.8.1/bin/rspamd
gzipping man pages under /nix/store/24z497389g4m5g7k0dd9842kwnym4zql-rspamd-1.8.1/share/man/
strip is /nix/store/p9akxn2sfy4wkhqdqa3li97pc6jaz3r1-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/24z497389g4m5g7k0dd9842kwnym4zql-rspamd-1.8.1/lib  /nix/store/24z497389g4m5g7k0dd9842kwnym4zql-rspamd-1.8.1/bin
patching script interpreter paths in /nix/store/24z497389g4m5g7k0dd9842kwnym4zql-rspamd-1.8.1
/nix/store/24z497389g4m5g7k0dd9842kwnym4zql-rspamd-1.8.1/bin/rspamd_stats: interpreter directive changed from "/usr/bin/env perl" to "/nix/store/bzgcpj9fijmlxgjaa72fjys7m3ycijy8-perl-5.28.0/bin/perl"
checking for references to /build in /nix/store/24z497389g4m5g7k0dd9842kwnym4zql-rspamd-1.8.1...

GrahamcOfBorg · 2018-11-08T17:36:16Z

Timed out, unknown build status on x86_64-linux (full log)

Attempted: rspamd

Partial log (click to expand)

[ 55%] Building C object CMakeFiles/hs_exec_corei7.dir/src/nfa/limex_64.c.o
[ 55%] Building C object CMakeFiles/hs_exec_corei7.dir/src/nfa/limex_simd128.c.o
[ 55%] Building C object CMakeFiles/hs_exec_corei7.dir/src/nfa/limex_simd256.c.o
[ 55%] Building C object CMakeFiles/hs_exec_corei7.dir/src/nfa/limex_simd384.c.o
[ 55%] Building C object CMakeFiles/hs_exec_corei7.dir/src/nfa/limex_simd512.c.o
[ 56%] Building C object CMakeFiles/hs_exec_corei7.dir/src/nfa/mcclellan.c.o
[ 56%] Building C object CMakeFiles/hs_exec_corei7.dir/src/nfa/mcsheng.c.o
building of '/nix/store/5x8m26fj9z8vsrhk6rf0170x8an0pgvc-hyperscan-5.0.0.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/zcms8z88jmh5hiavbdhsnsbjkamb3ll5-rspamd-1.8.1.drv': 1 dependencies couldn't be built
error: build of '/nix/store/zcms8z88jmh5hiavbdhsnsbjkamb3ll5-rspamd-1.8.1.drv' failed

avnik · 2018-11-08T22:06:59Z

@Mic92 hyperscan build timed out. Probably we can set it off by default -- is useful optimization on high volumes, but can take hours to build (although it depends on cmake and boost only, and shouldn't rebuild often)

avnik · 2018-11-08T23:11:16Z

@GrahamcOfBorg build rspamd

pkgs/development/libraries/hyperscan/default.nix

c0bw3b · 2018-11-23T18:28:46Z

What's the rationale to build rspamd with jemalloc? It does not seem to be the upstream default.

(also: v1.8.2 is out)

pkgs/servers/mail/rspamd/default.nix

avnik · 2018-11-23T18:47:17Z

@c0bw3b build with jemalloc is author suggestion. Source package have very conservative defaults, to reduce build dependencies. Author himself build deb/rpm/freebsd packages with jemalloc (and hyperscan on x86_64). Hyperscan looks good on runtime, but take ~1.5 hours to build, so I move it to option.

other suggestions taken in acocunt, will update PR today later or tomorrow

c0bw3b · 2018-11-23T19:16:21Z

Ok for building with jemalloc. Thanks for explaining.
The hyperscan build time is less of an issue. Once built it will be cached.
(I'm also building it locally to check).

andir

Added a few nits that are mostly about style not functionality. But mostly looks fine 👍

andir · 2018-11-23T19:11:01Z

pkgs/development/libraries/hyperscan/default.nix

+, boost
+}:
+
+assert stdenv.isx86_64;  


should this not be moved towards a platform restriction in the meta attribute?

pkgs/development/libraries/hyperscan/default.nix

c0bw3b · 2018-11-23T22:21:27Z

No problem for building hyperscan on x64-linux

Result in /[...]/nox-review-_kylyhod
total 0
lrwxrwxrwx 1 zzzz users 56 23 nov.  21:23 result -> /nix/store/yn5rqbk5spf3s52khk76szjbk2b81134-rspamd-1.8.1
lrwxrwxrwx 1 zzzz users 59 23 nov.  21:23 result-2 -> /nix/store/d4m0im5a8dnh6gb2cqr6vlpnn59slnnm-hyperscan-5.0.0

avnik · 2018-11-24T01:23:11Z

Added 1.8.2 upgrade, and included all clarifications, except some I have mentioned above.

@Mic92 @fpletz I'd like your opinion, about nessesary to build hyperscan by default

c0bw3b · 2018-12-02T21:14:52Z

pkgs/top-level/all-packages.nix

@@ -3349,6 +3349,8 @@ with pkgs;

  hylafaxplus = callPackage ../servers/hylafaxplus { };

+  hyperscan = callPackage ../development/libraries/hyperscan { };


This would be better located between hyena and icu58 in the ### DEVELOPMENT / LIBRARIES section.
(nitpick)

c0bw3b · 2018-12-02T21:15:54Z

Probably going to timeout on x64-linux but I'd like to check aarch64 build so
@GrahamcOfBorg build rspamd

GD is experimental and never-enabled-by-default.

(by upstream author's suggestion)

Also update homepage link, and fetch from org's github, instead author's private fork.

avnik · 2018-12-02T22:14:46Z

On Sun, Dec 02, 2018 at 01:14:55PM -0800, Renaud wrote: > @@ -3349,6 +3349,8 @@ with pkgs; hylafaxplus = callPackage ../servers/hylafaxplus { }; + hyperscan = callPackage ../development/libraries/hyperscan { }; This would be better located between `hyena` and `icu58` in the `### DEVELOPMENT / LIBRARIES` section. (nitpick)

done. And rebased on today's master

c0bw3b · 2018-12-02T22:33:19Z

@GrahamcOfBorg build rspamd

c0bw3b

Rebuilt locally on NixOS

Result in /[...]/nox-review-3hgirszu
total 0
lrwxrwxrwx 1 zzzz users 56  3 déc.  01:41 result -> /nix/store/xgmfcngzaqf7wv8hzpr5wg3fhp8j2csp-rspamd-1.8.2
lrwxrwxrwx 1 zzzz users 59  3 déc.  01:41 result-2 -> /nix/store/9bnflfbbkylhyg8s3vww5mw1fwjkllky-hyperscan-5.0.0

GrahamcOfBorg added 8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Nov 8, 2018

avnik force-pushed the rspamd branch from d7f00c4 to d6a01d9 Compare November 8, 2018 23:10

c0bw3b reviewed Nov 23, 2018

View reviewed changes

pkgs/development/libraries/hyperscan/default.nix Outdated Show resolved Hide resolved

c0bw3b reviewed Nov 23, 2018

View reviewed changes

pkgs/servers/mail/rspamd/default.nix Outdated Show resolved Hide resolved

andir reviewed Nov 23, 2018

View reviewed changes

avnik force-pushed the rspamd branch from d6a01d9 to 7544106 Compare November 24, 2018 01:16

GrahamcOfBorg added the 8.has: clean-up label Nov 24, 2018

avnik force-pushed the rspamd branch from 7544106 to 58edf9c Compare December 2, 2018 02:06

c0bw3b reviewed Dec 2, 2018

View reviewed changes

avnik added 7 commits December 3, 2018 00:13

rspamd: drop gmime dependency, it is not used anymore

ff5eba9

rspamd: Make libfann and libgd optional

1b59c4a

GD is experimental and never-enabled-by-default.

rspamd: build with jemalloc

e77fa36

(by upstream author's suggestion)

rspamd: enable openblas support for neural networks

12a5b53

hyperscan: init at 5.0.0

3e1acb0

rspamd: enable hyperscan

b73b022

rspamd: 1.8.1 -> 1.8.2

2fd6de3

Also update homepage link, and fetch from org's github, instead author's private fork.

avnik force-pushed the rspamd branch from 58edf9c to 2fd6de3 Compare December 2, 2018 22:13

c0bw3b approved these changes Dec 3, 2018

View reviewed changes

c0bw3b merged commit 550b275 into NixOS:master Dec 3, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rspamd: add hyperscan, drop obsoleted dependencies #49915

rspamd: add hyperscan, drop obsoleted dependencies #49915

avnik commented Nov 8, 2018

Mic92 commented Nov 8, 2018

GrahamcOfBorg commented Nov 8, 2018

GrahamcOfBorg commented Nov 8, 2018

GrahamcOfBorg commented Nov 8, 2018

avnik commented Nov 8, 2018

avnik commented Nov 8, 2018

c0bw3b commented Nov 23, 2018

avnik commented Nov 23, 2018

c0bw3b commented Nov 23, 2018

andir left a comment •

edited

andir Nov 23, 2018

c0bw3b commented Nov 23, 2018

avnik commented Nov 24, 2018

c0bw3b Dec 2, 2018

c0bw3b commented Dec 2, 2018

avnik commented Dec 2, 2018 via email

c0bw3b commented Dec 2, 2018

c0bw3b left a comment

		@@ -3349,6 +3349,8 @@ with pkgs;

		hylafaxplus = callPackage ../servers/hylafaxplus { };

		hyperscan = callPackage ../development/libraries/hyperscan { };

rspamd: add hyperscan, drop obsoleted dependencies #49915

rspamd: add hyperscan, drop obsoleted dependencies #49915

Conversation

avnik commented Nov 8, 2018

Motivation for this change

Things done

Mic92 commented Nov 8, 2018

GrahamcOfBorg commented Nov 8, 2018

GrahamcOfBorg commented Nov 8, 2018

GrahamcOfBorg commented Nov 8, 2018

avnik commented Nov 8, 2018

avnik commented Nov 8, 2018

c0bw3b commented Nov 23, 2018

avnik commented Nov 23, 2018

c0bw3b commented Nov 23, 2018

andir left a comment • edited

Choose a reason for hiding this comment

andir Nov 23, 2018

Choose a reason for hiding this comment

c0bw3b commented Nov 23, 2018

avnik commented Nov 24, 2018

c0bw3b Dec 2, 2018

Choose a reason for hiding this comment

c0bw3b commented Dec 2, 2018

avnik commented Dec 2, 2018 via email

c0bw3b commented Dec 2, 2018

c0bw3b left a comment

Choose a reason for hiding this comment

andir left a comment •

edited