Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Staging next #104781

Merged
merged 90 commits into from Nov 30, 2020
Merged

Staging next #104781

merged 90 commits into from Nov 30, 2020

Conversation

FRidh
Copy link
Member

@FRidh FRidh commented Nov 24, 2020

Motivation for this change

https://hydra.nixos.org/job/nixpkgs/staging-next/unstable#tabs-constituents

With cross-compilation of Python extension modules.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

andir and others added 30 commits November 1, 2020 19:40
This adds -frandom-seed to each compiler invocation in stdenv. The
object here is to make the compierl invocations produce the same output
every time they are called (for the same derivation). When the
-frandom-seed option is not set the compiler will use a combination of
random numbers (in GCC's case from /dev/urandom) and the durrent time to
produce a "random" input per file. This can (among other things) lead to
different ordering of symbols in the produced object files.

For reason of reproducibility we prefer having the same derivation
produce the exact same outputs. This is not a silver bullet but one way
to tame the compiler.
Sensitive to exact parser output
This reverts commit 0fc5e60.

For some reason the v1.0.3 PyPI tarball ships with tests that expect pytest-testmon master instead of the v1.0.3 code (tarpas/pytest-testmon#158), causing these tests to fail against 1.0.3 itself.
stdenv: introduce -frandom-seed
See https://www.redhat.com/sysadmin/fedora-31-control-group-v2 for
details on why this is desirable, and how it impacts containers.

Users that need to keep using the old cgroup hierarchy can re-enable it
by setting `systemd.unifiedCgroupHierarchy` to `false`.

Well-known candidates not supporting that hierarchy, like docker and
hidepid=… will disable it automatically.

Fixes #73800
This gets automatically disabled by docker if the docker backend is
used, but the bundled containerd also doesn't seem to support cgroupsv2,
so disable it explicitly here, too.
For now, testing IO Accounting is skipped, as it seems to be either
broken, or hard to reproduce in a VM.
Please do not use tarballs.nixos.org in src URLs.  tarballs.nixos.org is
a cache, not an authority.

This patch differs from the one in tarballs.nixos.org only in source code
comments.
@r-burns
Copy link
Contributor

r-burns commented Nov 24, 2020

I think the sysconfigdata changes are breaking python37Packages.bootstrapped-pip - hydra log:

  File "/nix/store/i8sy0wp598x46ny7jp26ga45sn0nrmf9-python3-3.7.9/lib/python3.7/sysconfig.py", line 422, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ModuleNotFoundError: No module named '_sysconfigdata__linux_x86_64-linux-gnu'

@FRidh
Copy link
Member Author

FRidh commented Nov 24, 2020

Sigh. This was the issue spidermonkey_78 had before my fix. That got fixed by using a symlink, and now this one complains with a symlink. OK, next patch, a copy instead. Yet another stdenv rebuild...

Also just now aborted all builds on staging-next.

…ling extension modules"""

This still does not function without issues. E.g., bootstrapped-pip fails with Python 3.6 and 3.7
as well as 3.8 on 32-bit.

Because this is a stdenv-rebuild it needs to be tested significantly better

This reverts commit 6100bc2.
@FRidh
Copy link
Member Author

FRidh commented Nov 25, 2020

Reverted the Python cross commit again in 0a12b8a.

@FRidh
Copy link
Member Author

FRidh commented Nov 26, 2020

Python namespace hook is causing trouble @jonringer

Enforcing PEP420 namespace: azure.keyvault
/nix/store/szhb6132xmkq3q3bn9zv0fynmcfykjml-findutils-4.7.0/bin/find: ‘/nix/store/0v1llc4mv94qsvixjfasiavkk63hk5zk-python3.7-azure-keyvault-certificates-4.2.1/lib/python3.7/site-packages/azure/__pycache__/’: No such file or directory
rm: missing operand

@jonringer
Copy link
Contributor

jonringer commented Nov 26, 2020

Ah, I originally, had a if [ -d "$pycachePath"]; then in there, guess I forgot to add is back when changing the find call to support whitespace

@jonringer
Copy link
Contributor

jonringer commented Nov 26, 2020

fixed:

$ nix-build -E 'with import ./. { config = {}; overlays = []; }; python3.withPackages (ps: with ps; [ jaraco_functools jaraco_collections])'
these derivations will be built:
  /nix/store/f86c3fslhirg7l5wfpxszhmm5crcg86y-python3-3.8.6-env.drv
building '/nix/store/f86c3fslhirg7l5wfpxszhmm5crcg86y-python3-3.8.6-env.drv'...
created 234 symlinks in user environment
/nix/store/vpivpif8aapvr5fmaivi8m1akaam7brj-python3-3.8.6-env
[13:49:35] jon@nixos ~/projects/nixpkgs (staging-next)
$ nix-build -A azure-cli
/nix/store/qm8fsm79x5cycqs702gjch2w20r9fzik-python3.7-azure-cli-2.14.2

building ceph now, but it should work... takes a while since ceph has ~2000 build dependencies

$ nix-store -q --requisites $(nix-instantiate -A ceph) | grep drv | wc -l
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
1987

ceph builds as well now:

[15:16:34] jon@nixos ~/projects/nixpkgs (staging-next)
$ nix-build -A ceph
/nix/store/1ringwxafvswm7ppj11f4dy0hjmrrbk4-ceph-15.2.5

@FRidh FRidh merged commit 9a63b3d into master Nov 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet