ntl: 9.11.0 -> 11.2.1 #43679

timokau · 2018-07-17T17:01:22Z

Motivation for this change

ntl hasn't been updated in a while. So I'm doing that and adding myself
as the maintainer. I'm also adding some options and pinning the sage
dependency, since it is unfortunately not compatible with the latest ntl
yet.

I've also enabled the tests, since they don't take terribly long and are
worth the time in my opinion.

@7c6f434c (for the flint change)

The overrides in sages default.nix are really annoying to maintain (as seen in this patch). There has to be a better way than to "wire everything up" manually. Basically an overlay within nixpkgs.

@matthewbauer git log shows that you worked with overlays recently for the musl package set. Do you know if I could apply something similar for this purpose?

I basically want to pin some of sages dependencies. But since there are so many dependencies, its important that those overrides apply recursively.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

7c6f434c · 2018-07-17T18:46:19Z

Looks OK; I do not personally care about flint beyond its use for sage; overlays might be a nice solution.

GrahamcOfBorg · 2018-07-18T08:55:17Z

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: ntl

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-07-18T09:50:46Z

Success on aarch64-linux (full log)

Attempted: ntl

Partial log (click to expand)

more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/mnr81n5rqlqylckcr1qxnj9jwr43rxab-ntl-11.2.1
shrinking /nix/store/mnr81n5rqlqylckcr1qxnj9jwr43rxab-ntl-11.2.1/lib/libntl.so.38.0.0
strip is /nix/store/7iyn7gn33i7xxjgmwf25k20246y6nd9d-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/mnr81n5rqlqylckcr1qxnj9jwr43rxab-ntl-11.2.1/lib
patching script interpreter paths in /nix/store/mnr81n5rqlqylckcr1qxnj9jwr43rxab-ntl-11.2.1
checking for references to /build in /nix/store/mnr81n5rqlqylckcr1qxnj9jwr43rxab-ntl-11.2.1...
/nix/store/mnr81n5rqlqylckcr1qxnj9jwr43rxab-ntl-11.2.1

GrahamcOfBorg · 2018-07-18T09:55:33Z

Failure on x86_64-linux (full log)

Attempted: ntl

Partial log (click to expand)

libtool: link: g++ -I../include -I. -g -O2 -pthread -o .libs/mat_lzz_pTest mat_lzz_pTest.cpp  ./.libs/libntl.so -L/nix/store/k15acciycff39mifhdrar2i9q0gd0bkh-ntl-11.2.1/lib -L/nix/store/5fmjd54c5g7a41b0f38r5i4qnjdl8ln6-gf2x-1.2/lib /nix/store/jvz2yldc7kcfxrfwi44dys82rf9qgwwl-gmp-6.1.2/lib/libgmp.so /nix/store/5fmjd54c5g7a41b0f38r5i4qnjdl8ln6-gf2x-1.2/lib/libgf2x.so /nix/store/i5znwsdi9yybfydd1kbdrsq5s4phjm93-gcc-7.3.0-lib/lib/../lib64/libstdc++.so -lm -pthread -Wl,-rpath -Wl,/nix/store/k15acciycff39mifhdrar2i9q0gd0bkh-ntl-11.2.1/lib -Wl,-rpath -Wl,/nix/store/jvz2yldc7kcfxrfwi44dys82rf9qgwwl-gmp-6.1.2/lib -Wl,-rpath -Wl,/nix/store/5fmjd54c5g7a41b0f38r5i4qnjdl8ln6-gf2x-1.2/lib -Wl,-rpath -Wl,/nix/store/i5znwsdi9yybfydd1kbdrsq5s4phjm93-gcc-7.3.0-lib/lib/../lib64
make[1]: Leaving directory '/build/ntl-11.2.1/src'
running mat_lzz_pTest

seed=772080562
testing multiplication....................................................................................................
testing inversion....................................................................................................
testing solve.....................................................[singular]...............................................
building of '/nix/store/4yqnn871nv88lyfk2349q1b8fv9m917v-ntl-11.2.1.drv' timed out after 3600 seconds
error: build of '/nix/store/4yqnn871nv88lyfk2349q1b8fv9m917v-ntl-11.2.1.drv' failed

ntl hasn't been updated in a while. So I'm doing that and adding myself as the maintainer. I'm also adding some options and pinning the sage dependency, since it is unfortunately not compatible with the latest ntl yet. I've also enabled the tests, since they don't take terribly long and are worth the time in my opinion.

timokau · 2018-07-18T13:10:26Z

Looks OK; I do not personally care about flint beyond its use for sage; overlays might be a nice solution.

Great! Can you merge this then? I think the less-than-optimal override situation is a different issue (although I'd gladly discuss options here).

GrahamcOfBorg · 2018-07-18T13:19:17Z

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: ntl

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-07-18T13:19:27Z

Success on aarch64-linux (full log)

Attempted: ntl

Partial log (click to expand)

/nix/store/mnr81n5rqlqylckcr1qxnj9jwr43rxab-ntl-11.2.1

GrahamcOfBorg · 2018-07-18T13:31:19Z

Success on x86_64-linux (full log)

Attempted: ntl

Partial log (click to expand)

more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/k15acciycff39mifhdrar2i9q0gd0bkh-ntl-11.2.1
shrinking /nix/store/k15acciycff39mifhdrar2i9q0gd0bkh-ntl-11.2.1/lib/libntl.so.38.0.0
strip is /nix/store/90vmpr41dzsx350k5argycaf693hnl1l-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/k15acciycff39mifhdrar2i9q0gd0bkh-ntl-11.2.1/lib
patching script interpreter paths in /nix/store/k15acciycff39mifhdrar2i9q0gd0bkh-ntl-11.2.1
checking for references to /build in /nix/store/k15acciycff39mifhdrar2i9q0gd0bkh-ntl-11.2.1...
/nix/store/k15acciycff39mifhdrar2i9q0gd0bkh-ntl-11.2.1

timokau · 2018-07-18T18:34:59Z

Follow-up to the overlay issue in #43755.

matthewbauer · 2018-07-19T03:53:44Z

Looks like this introduces an issue:

https://hydra.nixos.org/build/77748343

7c6f434c · 2018-07-19T08:54:05Z

Illegal instruction? I have a bad feeling…

timokau · 2018-07-19T11:54:29Z

Thanks for telling me (what happened to hydra mailing maintainers on failures?)

I can imagine that it has something to do with the "tune" variable. I've only tested the x86 option and only on one cpu. Is it possible to determine which architecture that hydra runner is using?

@GrahamcOfBorg build ntl

GrahamcOfBorg · 2018-07-19T11:54:47Z

Success on aarch64-linux (full log)

Attempted: ntl

Partial log (click to expand)

these paths will be fetched (1.47 MiB download, 11.39 MiB unpacked):
  /nix/store/bnyjzdc5bq11g8h9gf8sc86alhhjrikv-gf2x-1.2
  /nix/store/mnr81n5rqlqylckcr1qxnj9jwr43rxab-ntl-11.2.1
copying path '/nix/store/bnyjzdc5bq11g8h9gf8sc86alhhjrikv-gf2x-1.2' from 'https://cache.nixos.org'...
copying path '/nix/store/mnr81n5rqlqylckcr1qxnj9jwr43rxab-ntl-11.2.1' from 'https://cache.nixos.org'...
/nix/store/mnr81n5rqlqylckcr1qxnj9jwr43rxab-ntl-11.2.1

GrahamcOfBorg · 2018-07-19T11:54:56Z

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: ntl

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-07-19T11:54:57Z

Success on x86_64-linux (full log)

Attempted: ntl

Partial log (click to expand)

/nix/store/k15acciycff39mifhdrar2i9q0gd0bkh-ntl-11.2.1

timokau · 2018-07-19T20:59:16Z

@victorshoup (hoping that this is the correct github handle for the ntl author)

As I recently emailed you, I'm trying to (well actually did) upgrade ntl for the nix package manager. Unfortunately our build servers are reporting an illegal instruciton during testing:

running tests...............make: *** [makefile:405: check] Illegal instruction

While the build worked fine locally and on our CI. I'm using TUNE=x86 in this case. The documentation says that that "chooses options that should be well suited for most x86 platforms". Could the "most" in that sentence be the cause for that error? Or can you imagine some other cause?

That build server is x86_64-linux. I think its an intel e5620, but I'm not sure about that. If you think its relevant I'll have to see if I can find out.

The other configure flags I'm using are (with $(out) and ${gf2x} expanded to the proper directories):

DEF_PREFIX=$(out)
SHARED=on
NATIVE=off
NTL_GF2X_LIB=on
GF2X_PREFIX=${gf2x}

victorshoup · 2018-07-19T21:41:03Z

Hi, So I imagine the problem is the following. By default, NTL compiles with -march=native, so it will generate instructions that may only be valid for the machine on which it is compiled. You may have to configure with NATIVE=off to make the binary work across different x86 architectures. While that should work, you should note that many users who use these binaries will be using very suboptimal builds. I *strongly* encourage you to provide and encourage users to build NTL from source, especially if they want truly high performance.

…

-Victor

On Jul 19, 2018, at 4:59 PM, Timo Kaufmann ***@***.***> wrote: @victorshoup (hoping that this is the correct github handle for the ntl author) As I recently emailed you, I'm trying to (well actually did) upgrade ntl for the nix package manager. Unfortunately our build servers are reporting an illegal instruciton during testing: running tests...............make: *** [makefile:405: check] Illegal instruction While the build worked fine locally and on our CI. I'm using TUNE=x86 in this case. The documentation says that that "chooses options that should be well suited for most x86 platforms". Could the "most" in that sentence be the cause for that error? Or can you imagine some other cause? That build server is x86_64-linux. I think its an intel e5620, but I'm not sure about that. If you think its relevant I'll have to see if I can find out. The other configure flags I'm using are (with $(out) and ${gf2x} expanded to the proper directories): DEF_PREFIX=$(out) SHARED=on NATIVE=off NTL_GF2X_LIB=on GF2X_PREFIX=${gf2x} — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

timokau · 2018-07-19T21:51:49Z

So I imagine the problem is the following. By default, NTL compiles with -march=native, so it will generate instructions that may only be valid for the machine on which it is compiled. You may have to configure with NATIVE=off to make the binary work across different x86 architectures.

I am using `NATIVE=off` (see the list of configure flags in my last post) and the tests are running on the same machine as the build. So I don't think that is the issue.

While that should work, you should note that many users who use these binaries will be using very suboptimal builds. I *strongly* encourage you to provide and encourage users to build NTL from source, especially if they want truly high performance.

I do offer a `tune` option that will do that (and build from source). *Encouraging* to build from source kind of goes against the point of a package manager however (except if you are gentoo). And using tuning / non-native by default is especially against the point of nix (which tries to be reproducible). But the option is there for people who need it.

victorshoup · 2018-07-19T23:52:30Z

Ok. You'll have to send me more details. What's the drv file you sent me? I don't know what a drv file is. That said, I really doubt this is an ntl issue. First, ntl itself does use a few inline asm instructions, but these are very, very old instruction types and have not changed in many years. So that means something is configured wrong in the compiler or in some of the libraries that ntl uses. For example, perhaps the problem is with gmp. Anyway, send me detailed logs. You could also try running the test program that crashes in gdb and backtrace. That's probably the quickest way to figure out what's going on.

…

On Jul 19, 2018, at 5:51 PM, Timo Kaufmann ***@***.***> wrote: > So I imagine the problem is the following. > By default, NTL compiles with -march=native, so it will generate instructions > that may only be valid for the machine on which it is compiled. > > You may have to configure with NATIVE=off to make the binary > work across different x86 architectures. I am using `NATIVE=off` (see the list of configure flags in my last post) and the tests are running on the same machine as the build. So I don't think that is the issue. > While that should work, you should note that many users who use these > binaries will be using very suboptimal builds. > > I *strongly* encourage you to provide and encourage users to build NTL from source, > especially if they want truly high performance. I do offer a `tune` option that will do that (and build from source). *Encouraging* to build from source kind of goes against the point of a package manager however (except if you are gentoo). And using tuning / non-native by default is especially against the point of nix (which tries to be reproducible). But the option is there for people who need it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

timokau · 2018-07-20T12:00:36Z

Ok. You'll have to send me more details. What's the drv file you sent me? I don't know what a drv file is.

The naming of the file is a bit confusing, but that is the build log of ntl containing the failure.

That said, I really doubt this is an ntl issue. First, ntl itself does use a few inline asm instructions, but these are very, very old instruction types and have not changed in many years. So that means something is configured wrong in the compiler or in some of the libraries that ntl uses. For example, perhaps the problem is with gmp.

That is possible. Its hard to debug right now because I cannot reproduce the problem locally and don't have direct access to the builder that had the problem. It is just one builder in our build cluster.

I'll have to try to find someone that can reproduce the failure locally.

Anyway, send me detailed logs.

The build log is the file with the confusing name. Are there other logs that might be helpful?

You could also try running the test program that crashes in gdb and backtrace. That's probably the quickest way to figure out what's going on.

Again the above issue :/ I'll try to find someone.

@7c6f434c @matthewbauer did you try the build locally?

victorshoup · 2018-07-20T15:37:24Z

I don't know how to open a dry file (and neither does my computer). Could you send me a build log that is a plain text file? Thanks.

…

On Jul 20, 2018, at 8:00 AM, Timo Kaufmann ***@***.***> wrote: Ok. You'll have to send me more details. What's the drv file you sent me? I don't know what a drv file is. The naming of the file is a bit confusing, but that is the build log of ntl containing the failure. That said, I really doubt this is an ntl issue. First, ntl itself does use a few inline asm instructions, but these are very, very old instruction types and have not changed in many years. So that means something is configured wrong in the compiler or in some of the libraries that ntl uses. For example, perhaps the problem is with gmp. That is possible. Its hard to debug right now because I cannot reproduce the problem locally and don't have direct access to the builder that had the problem. It is just one builder in our build cluster. I'll have to try to find someone that can reproduce the failure locally. Anyway, send me detailed logs. The build log is the file with the confusing name. Are there other logs that might be helpful? You could also try running the test program that crashes in gdb and backtrace. That's probably the quickest way to figure out what's going on. Again the above issue :/ I'll try to find someone. @7c6f434c @matthewbauer did you try the build locally? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

timokau · 2018-07-20T15:45:48Z

That is a plain text file. It really shouldn't have the drv extension.

At least for me, the browser just opens it. Otherwise you should be able to open it with any text editor.

timokau · 2018-07-20T15:46:19Z

You should be able to just rename it to txt or similar.

victorshoup · 2018-07-20T16:01:26Z

OK. Firefox opened it for me (but Vim shows unprintable gobbledygook). Anyway, everything looks fin, until the Illegal instruction. I guess the only way forward is to 1) find the platform where this happens 2) instead of running "make check", run: make QuickTest gbb ./QuickTest 3) run it (with no args), and the do a backtrace from the Illegal instruction. Like I said, in principle, this should not be an NTL issue. But...never say never, as they say...

…

On Jul 20, 2018, at 11:45 AM, Timo Kaufmann ***@***.***> wrote: That is a plain text file. It really shouldn't have the drv extension. At least for me, the browser just opens it. Otherwise you should be able to open it with any text editor. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

timokau · 2018-07-20T16:21:11Z

OK. Firefox opened it for me (but Vim shows unprintable gobbledygook).

Oh, you're right. Weird. I don't know which black magic is involved here (right click -> save page on firefox works and vim opens that).

Anyway, everything looks fin, until the Illegal instruction. I guess the only way forward is to 1) find the platform where this happens 2) instead of running "make check", run: make QuickTest gbb ./QuickTest 3) run it (with no args), and the do a backtrace from the Illegal instruction.

Looks like we can't do anything until someone can reproduce that error. Maybe the gmp maintainers have an idea @peti @vrthra?

Like I said, in principle, this should not be an NTL issue. But...never say never, as they say...

Thanks for helping trouble shoot it anyway :)

GrahamcOfBorg added 10.rebuild-darwin: 1-10 10.rebuild-linux: 11-100 labels Jul 18, 2018

timokau force-pushed the ntl-11.2.1 branch from cb8f4f8 to cad4465 Compare July 18, 2018 12:59

7c6f434c merged commit 877899f into NixOS:master Jul 18, 2018

timokau deleted the ntl-11.2.1 branch July 18, 2018 15:39

timokau mentioned this pull request Jul 18, 2018

[WIP] sage: use proper overlays #43755

Draft

9 tasks

vcunat mentioned this pull request Aug 14, 2018

staging-next #44867

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ntl: 9.11.0 -> 11.2.1 #43679

ntl: 9.11.0 -> 11.2.1 #43679

timokau commented Jul 17, 2018

7c6f434c commented Jul 17, 2018

GrahamcOfBorg commented Jul 18, 2018

GrahamcOfBorg commented Jul 18, 2018

GrahamcOfBorg commented Jul 18, 2018

timokau commented Jul 18, 2018

GrahamcOfBorg commented Jul 18, 2018

GrahamcOfBorg commented Jul 18, 2018

GrahamcOfBorg commented Jul 18, 2018

timokau commented Jul 18, 2018

matthewbauer commented Jul 19, 2018

7c6f434c commented Jul 19, 2018

timokau commented Jul 19, 2018

GrahamcOfBorg commented Jul 19, 2018

GrahamcOfBorg commented Jul 19, 2018

GrahamcOfBorg commented Jul 19, 2018

timokau commented Jul 19, 2018

victorshoup commented Jul 19, 2018 via email

timokau commented Jul 19, 2018 via email

victorshoup commented Jul 19, 2018 via email

timokau commented Jul 20, 2018

victorshoup commented Jul 20, 2018 via email

timokau commented Jul 20, 2018

timokau commented Jul 20, 2018

victorshoup commented Jul 20, 2018 via email

timokau commented Jul 20, 2018 via email

ntl: 9.11.0 -> 11.2.1 #43679

ntl: 9.11.0 -> 11.2.1 #43679

Conversation

timokau commented Jul 17, 2018

Motivation for this change

Things done

7c6f434c commented Jul 17, 2018

GrahamcOfBorg commented Jul 18, 2018

GrahamcOfBorg commented Jul 18, 2018

GrahamcOfBorg commented Jul 18, 2018

timokau commented Jul 18, 2018

GrahamcOfBorg commented Jul 18, 2018

GrahamcOfBorg commented Jul 18, 2018

GrahamcOfBorg commented Jul 18, 2018

timokau commented Jul 18, 2018

matthewbauer commented Jul 19, 2018

7c6f434c commented Jul 19, 2018

timokau commented Jul 19, 2018

GrahamcOfBorg commented Jul 19, 2018

GrahamcOfBorg commented Jul 19, 2018

GrahamcOfBorg commented Jul 19, 2018

timokau commented Jul 19, 2018

victorshoup commented Jul 19, 2018 via email

timokau commented Jul 19, 2018 via email

victorshoup commented Jul 19, 2018 via email

timokau commented Jul 20, 2018

victorshoup commented Jul 20, 2018 via email

timokau commented Jul 20, 2018

timokau commented Jul 20, 2018

victorshoup commented Jul 20, 2018 via email

timokau commented Jul 20, 2018 via email