Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stdenv, treewide: Better configure platforms #60166

Draft
wants to merge 2 commits into
base: staging
Choose a base branch
from

Conversation

Ericson2314
Copy link
Member

Motivation for this change
  1. Use @lheckemann's trick of an env var, so we don't screw up non-autoconf.
  2. Always pass build_alias, so that building x86_32 on x86_64, or aarch32 on aarch64, etc should work.
  3. Remove all configurePlatforms = [ ]; now that they are not needed.

CC @volth, can you confirm this mostly works? We can combine with any package-specific fixes needed from #60131.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Assured whether relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@Ericson2314 Ericson2314 changed the title Better configure platforms stdenv, treewide: Better configure platforms Apr 24, 2019
@Ericson2314 Ericson2314 changed the base branch from master to staging April 24, 2019 15:33
(stdenv.hostPlatform != stdenv.buildPlatform)
[ "build" "host" ]
"host"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why leave out host?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it forced some cross-specific logic, even if both were the same. Don't want to go that far yet :).

@@ -133,7 +133,6 @@ stdenv.mkDerivation rec {
++ (with darwin.apple_sdk.frameworks; optionals stdenv.isDarwin [ Cocoa OpenGL ])
;

configurePlatforms = [ ];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't these fail on the build flag?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But it's a harmless env var now!

@grahamc
Copy link
Member

grahamc commented Apr 24, 2019

Not sure if you've seen it yet @Ericson2314 but each ofborg run now comes with an evaluation performance report: https://github.com/NixOS/nixpkgs/pull/60166/checks?check_run_id=109946398 since you do some pretty core stuff, I think this report is especially relevant to you. Check it out?

not saying your changes here need rethinking due to the results, just keep it in mind and check it out when submitting changes to see how it impacts evaluation

@grahamc
Copy link
Member

grahamc commented Apr 24, 2019

I hope not. I chose not to make any labels or value judgement around the results of the performance report, specifically to avoid pushing people to write ugly code to make the mark go away.

@matthewbauer
Copy link
Member

For make-derivation it's pretty important to look at performance. optionalAttrs is best to avoid as much as possible here. Nix has to evaluate those checks every timne.

Copy link
Member

@matthewbauer matthewbauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't work for build scripts that support build, host and target but don't know about the *_alias flag.

@Ericson2314
Copy link
Member Author

@grahamc thanks for the heads up! I errored on the side of smaller derivations but more //. nrOpUpdates makes me look like I made a bad choice, and per @volth's point more null is actually the cleaner code anyways, so I'll gladly remove my premature optimization.

@grahamc
Copy link
Member

grahamc commented Apr 24, 2019

Cool! No worries, I hadn't even looked at your PR, or the result of the performance report ;) thank you!

Then we don't run the risk of stomping over non-autotools.
This isn't needed now that we use env vars.
@Ericson2314
Copy link
Member Author

Ericson2314 commented Apr 24, 2019

@volth Hmm does that mean everything all the way down is screwed up, or I just need to cherry-pick your perl fix? Perl doesn't use autoconf so I wouldn't expect build_alias and friends to work.

@Ericson2314
Copy link
Member Author

I thought I used your PR's base-branch. Maybe I need to use something else to get a) the right perl version, b) your and @matthewbauer's -m flags?

@Ericson2314
Copy link
Member Author

Also, is there a way to turn of the x86_32 hacks in Nix so I can decently repo building i686-linux?

@Ericson2314
Copy link
Member Author

OK I'll see why stdenv is screwed up.

@Ericson2314
Copy link
Member Author

Ericson2314 commented Apr 24, 2019

x86_64 python did build correctly, so the bootstrap tools referece does seem to be an aarch32-specific (or alt-mode-specific) problem.

@grahamc
Copy link
Member

grahamc commented Apr 25, 2019

He can request access any time he'd like :) https://github.com/nix-community/aarch64-build-box#want-access

@Ericson2314
Copy link
Member Author

@grahamc I read the README and now I'm properly scared :)

@Ericson2314
Copy link
Member Author

What do you all think we merge this cause it's a step in the right direction?

@Ericson2314
Copy link
Member Author

@volth you saying this is worse than 19.03? I figured it was better, just not sufficient to fix aarch32.

@lheckemann
Copy link
Member

I read the README and now I'm properly scared :)

Nothing to be really scared about, just a warning not to (mis)place any trust in the community box. I can highly recommend using it for stuff like this though! Hell, I've used it for non-aarch64-specific stuff as well just because of the oomph it has :)

What do you all think we merge this cause it's a step in the right direction?

I'm generally in favour. My main remaining concern is whether the *_alias env vars are actually a documented/supported interface in autotools and if they won't break behind our backs with some new autotools release. It's not a major issue though, since that's only a hypothetical situation, we can fix it by reverting this change, and I don't think it's that likely to happen.

@matthewbauer
Copy link
Member

This needs much more testing IMO

@Ericson2314
Copy link
Member Author

https://www.gnu.org/software/autoconf/manual/autoconf-2.65/html_node/Canonicalizing.html this is the documentation for *_alias. It does mention the read direction, but not the write direction.

@lheckemann
Copy link
Member

@matthewbauer That's not a very helpful comment. What sort of testing is missing in your view? It's targeted at staging, where hydra will (once the PR is merged) rebuild everything including the nixos tests. If that's not good enough for you, please be more specific about what you think we need in addition.

@yorickvP
Copy link
Contributor

When building armv7l on an aarch64 machine, I'm getting

builder for '/nix/store/s4n4a9s9zg0m2v2239zfay713rwkw09q-expand-response-params.drv' failed with exit code 1; last 9 log lines:
  unpacking sources
  patching sources
  configuring
  no configure script, doing nothing
  building
  Assembler messages:
  Error: unknown architecture `armv7-a'

  Error: unrecognized option -march=armv7-a
builder for '/nix/store/9v1pkwrhfwhg6wvgr3ka9krw04caikpd-patchelf-0.9.drv' failed with exit code 77; last 10 log lines:
  checking for a thread-safe mkdir -p... /nix/store/65sxp86ypi3ca914pq069blazwq9qzr1-bootstrap-tools/bin/mkdir -p
  checking for gawk... gawk
  checking whether make sets $(MAKE)... yes
  checking whether make supports nested variables... yes
  checking for style of include used by make... GNU
  checking for gcc... gcc
  checking whether the C compiler works... no
  configure: error: in `/build/patchelf-0.9':
  configure: error: C compiler cannot create executables
  See `config.log' for more details
cannot build derivation '/nix/store/jmsghzfyy114fsxpy046jmpb1nwz52x2-bootstrap-stage3-stdenv-linux.drv': 1 dependencies couldn't be
built

@FRidh FRidh added this to Needs review in Staging Oct 24, 2019
@FRidh FRidh moved this from Needs review to New in Staging Oct 24, 2019
@FRidh FRidh moved this from New to Not ready in Staging Oct 24, 2019
@stale
Copy link

stale bot commented Jun 1, 2020

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

  1. Search for maintainers and people that previously touched the
    related code and @ mention them in a comment.
  2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
    irc.freenode.net.

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 1, 2020
@lheckemann
Copy link
Member

@yorickvP's error isn't a regression AFAIK since I guess builds didn't even get that far when natively building armv7 software on aarch64. Is there anything besides the merge conflicts that prevents this from being merged?

@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 2, 2020
@stale
Copy link

stale bot commented Jun 9, 2021

I marked this as stale due to inactivity. → More info

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 9, 2021
@Artturin
Copy link
Member

Rebased #240637

@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 30, 2023
@Artturin Artturin marked this pull request as draft June 30, 2023 18:04
@wegank wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Staging
  
WIP
Development

Successfully merging this pull request may close these issues.

None yet

8 participants