[RFC 0031] Support ppc64(le) architecture #31
Conversation
|
I am willing to donate significant on-going build-farm resources on POWER9 hardware (multiple Raptor CS dual 18-core Talos 2 machines with around 128GiB of RAM each) to this effort on a Best Effort workload priority basis. I am also willing to provide shell accounts for interactive debugging sessions and/or other platorm testing/debugging uses to NixOS/Nix contributors. |
|
Not sure if needed, but I'd be happy to host bootstrap tarballs, assuming they are known to at least somewhat work ;). This would match how the musl tarballs are provided today.... for better or for worse. Just bug me if that'd be useful and I'll add it to the list! But hosting is easy these days but thought I'd offer :). Can't test things really (other than QEMU haha) but generally in favor of this! I don't think adding another URL to the rust bootstrap expression is a big deal, but I suppose someone will need to generate them. I take it upstream doesn't do so already? Do you know the story re:GHC? |
rfcs/0030-ppc64-platforms.md
Outdated
| * mesa: currently assumes that any non-ARM system is x86. Requires restructuring the driver selection, but to the author's knowledge (since no actual graphical software was tested yet) it should not require any changes to the actual build description. ([nixpkgs#45474](https://github.com/NixOS/nixpkgs/pull/45474)) | ||
| * strace: also assumes that any non-ARM system is x86 and so tries to use the m32 personality. Requires checking the available machine personalities for all non-x86 architectures. ([nixpkgs#45472](https://github.com/NixOS/nixpkgs/pull/45472)) | ||
| * TeX Live: builds LuaJIT as part of mfluajit, which doesn't work because LuaJIT doesn't support 64-bit Power architectures yet; seems to be possible to disable on that platform without too much impact. ([nixpkgs#45475](https://github.com/NixOS/nixpkgs/pull/45475)) | ||
| * OpenBLAS: pulled in by LVM. Needs a target description for ppc64le, but no changes to the build itself. <!-- I ran into a problem with a broken test a while ago, but it seems to be fine right now --> |
There was a problem hiding this comment.
OpenBLAS needed by LVM sounds very strange. LLVM maybe? It doesn't seem to be so on x86_64, but I wouldn't be surprised if POWER is different enough.
There was a problem hiding this comment.
I think it is at least (transitively). It was definitely pulled in by something related to file systems.
There was a problem hiding this comment.
I'll boot up my POWER box real quick and check what it was …
There was a problem hiding this comment.
Strange. It seems it's not required anymore for the system derivation (maybe the dependency was broken somewhere along the line since last month?). Should I remove the bullet point then?
There was a problem hiding this comment.
Well, if you want scientific calculations (Sage, Julia…) you will need OpenBLAS soon, but you might want to deprioritise it towards the end of the list.
I am not a ppc64 user (neither current nor near future), so I am not in position to give advice on priorities.
There was a problem hiding this comment.
OK, I guess I will put it after Rust, as something like "Some packages, like OpenBLAS, will need target descriptions for these platforms, but generally don't require changes to the actual build process."
|
For what it's worth, I'm in favor of this without hydra support (yet) and machine(s) from @justinlynn for ofborg, so that we can try to get things to actually work there. (I'm not sure of ofborg's security infrastructure, maybe it'd require approval from @LnL7 if it allows @justinlynn to trigger builds for darwin) And why not machines from @justinlynn on cachix, but I'm not sure it'd be possible to integrate them with hydra for trust reasons, as it'd distribute binaries in this case :) |
|
@dtzWill re: GHC on POWER9 -- I have a GHC bin-dist build working (there are a few small test failures, but they're probably not severe issues) w/ stack but need to upstream various fixes for the architecture. GHC itself works but the stack tooling needs patching (granted, it's fairly straightforward). |
|
re: ofBorg and POWER9: ppc64 will run Linux with normal namespace-based sandboxing, so trusted status is not needed. Unless there are some problems with Linux namespaces on these architectures… |
|
@dtzWill I haven't looked into Haskell myself, but looking at @justinlynn 's comment it seems to be manageable. Re: Rust, I know upstream are running some builds for ppc64(le) … (whether they can be used as bootstrap, I don't know) Generally I'd really like for Rust to be built from source, but maybe that should be a separate RFC? |
|
@7c6f434c I was meaning, as they're all connected to the same AMQP, maybe a builder can send commands to another builder, and then justinlynn could send commands to LnL7's builder, which would require LnL7 to trust justinlynn :) @CrystalGamma I think that, whether or not it deserves a RFC, building Rust from source is a matter that should not be discussed in this RFC, as RFCs that are not completely consensual have a high likelihood not to go anywhere :) |
|
@Ekleog so remove that sentence, too? I'm not too keen on spending a lot of time on a solution (binary bootstrap) I don't want to use, though … :( |
|
I'm not sure we should support POWER9 at this time. Nixpkgs and NixOS are being drawn in a large number of directions, and I think we might should focus on more pressing issues (memory use, existing architectures we do support) before expanding its scope. (edited to say more:) Adding more architectures is a significant investment in tooling and support and code complexity. If we support many architectures badly, we have a operating system which provides a lot of poor experiences. Additionally, adding it to ofborg means it will now send more build messages which some people already consider spammy. To make matters worse, package failures on architectures people can't easily debug on are confusing and sends a very unclear signal. |
|
@grahamc I think a distinction needs to be made between official support and agreeing to merge patches that provide unofficial support. Certainly, it would be wholly premature for NixOS as an entity to provide official builds or support for ppc64le. There are other architectures which can currently be built but for which official builds aren't provided. OTOH, patches which make ppc64le work should be merged if people submit them. There are people willing to do the work here; moreover, the rate of change of nixpkgs is fast enough that trying to maintain these patches out-of-tree has caused difficulty. IMO, adding ppc64le support to the extent that persons interested can build it themselves is unlikely to negatively affect people working on nixpkgs who don't care about ppc64le support, or, to the extent that it does affect those people, it will be in a way that enhances the ability of nixpkgs to target multiple platforms and architectures generally. Moreover, there doesn't need to be a commitment not to break ppc64le (e.g. someone who doesn't test on ppc64le merges a patch that breaks it) at this stage; it could be organised as a best-effort initiative by people interested in ppc64le, with no requirement that patches not break ppc64le and subsequent fixes for ppc64le maintained by people interested in it. What I think people interested in a ppc64le port need at this time is a willingness by nixpkgs to merge ppc64le patches, not to make any sort of official commitment or impose CI requirements. This RFC was specifically created because it was requested in response to such a PR; the ability to get such PRs merged is all that's really necessary right now. Discussing anything else seems premature. |
|
I think @grahamc makes a good point that in the current state of affairs ofBorg support for POWER9 is indeed increasing annoyance and premature, so the discussion of ofborg in context of POWER9 should be delayed for a long time. (I do not agree with the point about build errors on inaccessible architectures as long as there is Darwin support — in the case of Darwin I actually don't know how to run a build locally, in the case of Aarch64/POWER9/RISC-V I only need someone to publish a bootable Nix-containing Linux image for QEMU) Specifying that there is no expectation to avoid breaking Re: RAM requirements — the work is not fully fungible in that case, POWER9 support seems to attract new contributors and people already working on cross-builds and multi-target support. RAM usage requires (probably) something about NixOS module system. Not everyone risks to touch the module system, and one needs quite a bit of reputation to get the necessary reviewer efforts from the few and busy people who are expected to review fundamental changes in the module system. (disclaimer: I gave up on NixOS but stay fully dependent on Nixpkgs, so my interest is about Nixpkgs being nice, flexible, and preferably not shattered into a dozen of forks with various subsets of fixes) |
|
@CrystalGamma re: Rust: I think any solution would include you publishing a Rust binary suitable for bootstrapping on POWER9, be it via cross-bootstrapping or |
|
I don't personally see having POWER support in nixpkgs having a big effect on maintainability given it should be a mature architecture by now (though maybe not as popular as say, aarch64). I expect it will need way less amounts of patches/tweaks than RISC-V, and perhaps even less than aarch64. Regarding rust, according to https://forge.rust-lang.org/platform-support.html ppc64le (unless I mixed this up with some other architecture) is in the same support tier as both aarch64 and armv7 which means they do provide prebuilt compilers, so if all goes well getting it to work is like 5 lines of changes, similar to what was done in NixOS/nixpkgs@222f905 |
|
My prediction for the future: this RFC will go the same route as #23 where the RFC gets stalled and then merged because some people will have added POWER9 support into nixpkgs. I don't know if it's good or bad but it reflects the anarchist approach our community has to software development. In the absence of clear leadership and direction the next best thing to do is probably to nurture the ground for independent activities to happen. |
|
I think that reflects our current RFC process until #18 moves forward: If there are absolutely no disapprovals after some time, just do it. If there are any disapprovals, the RFC is essentially doomed for eternal limbo. Regarding this RFC I think "just do it" is fine: As long as any changes are clearly commented with explanations about their purpose and we don't make any commitments, it would be easy to remove support specific workarounds again if momentum dies. I don't see any major cost to this. |
|
@CrystalGamma It looks like this should get renamed to RFC 31 |
CrystalGamma
force-pushed
the
master
branch
from
d97dd8a
to
244b30e
Compare
|
@ryantm I suppose. The previous PR didn't carry a number in the title, so I thought it was an amendmend to an existing RFC. |
CrystalGamma
changed the title
|
Per meeting of the @NixOS/rfc-steering-committee this RFC is open for shepherding nominations. |
|
OK. Anything I as author need to do (except reading/updating the RFC again, since it's been a while)? |
|
You can nominate people, you think would make sense as shepherds. Otherwise obviously feel free to improve on the RFC anytime or incorporate discussion items and as soon as shepherds are chosen, cooperate with them on finishing it up! |
|
This pull request has been mentioned on Nix community. There might be relevant details there: https://discourse.nixos.org/t/rfcs-31-and-34-open-for-shepherd-nominations/1965/1 |
Mic92
added
the
status: open for nominations
|
RFC steering committee nominates @grahamc, @ryantm, @Ericson2314, and @zimbatm as shepherds, as they should cover the interests of ofborg, hydra, stdenv, and general package maintainership well. Ultimately this is up to the shepherd team and the RFC author, but we recommend shifting this RFC to lay out the general criteria/process for accepting a new architecture as supported (or removing one from the supported list). Please let us know if you accept! |
|
I accept the nomination. |
|
happy to help! |
|
@shlevy did you get any reply from @grahamc or @Ericson2314 ? |
There was a problem hiding this comment.
One thing I would really like to see in nixpkgs is to have a clear separation between the official and unofficial systems.
- clearly separate the official and non-official stdenv and bootstraps in the tree hierarchy
- add a "allowUnsupportedSystem" flag in the nixpkgs config. Similarly to allowUnfree, it would abort the evaluation if the target system is not part of the offically-supported list of systems.
Right now there are bootstrap files hosted all over the Internet and it's not good.
| Bringing support for ppc64(le) platforms will be done as follows: | ||
|
|
||
| The first step for implementing support is extending lib/ to recognize powerpc64(le) as a CPU architecture and allowing bootstrap files for the architecture to be (cross-)built and a stdenv based on those bootstrap files to be built. | ||
| This has already been merged in [nixpkgs#45340](https://github.com/NixOS/nixpkgs/pull/45340). |
There was a problem hiding this comment.
The more architectures we support, the harder it's going to be to upgrade gcc and other heavily-patched packages. Maintainers don't have access to the more esoteric platforms.
How do you propose that patches be handled if for example gcc has to be upgraded quickly for security-related reasons?
There was a problem hiding this comment.
Maybe lib/systems/platforms.nix should be amended with an extra "officiallySupported" flag.
|
|
||
| Some packages, like OpenBLAS, will need target descriptions for these platforms, but many don't require changes to the actual build process. | ||
|
|
||
| Support for Big Endian ppc64 might require changes in a few packages, though the majority of changes are assumed to be upstreamable (see [Unresolved Questions][unresolved]). |
There was a problem hiding this comment.
How about keeping track of this progress in the nixos.wiki ?
While these information give an idea of how much work is left to do, the RFC should also cover other aspects of adding new platforms like the added cost of nix evaluation and maintenance overhead that it creates inside of nixpkgs. I would like to see things addressed like:
- what are the rules of package maintainers in interaction with the non-official platforms?
- where is the list of official and unofficial platforms listed?
- what are the rules for adding and dropping a platform?
- for non-official platforms, are they keeping their own build farm? if yes, how can this be listed?
- where should the stdenv bootstrap files be stored?
There was a problem hiding this comment.
One thing that I would like to see explored, is if it's possible to store the stdenv out of the main nixpkgs tree, along with the binary cache and all the platform-specific information. Using the @nix-community organisation would be OK for that type of activity if you want.
There was a problem hiding this comment.
One thing that I would like to see explored, is if it's possible to store the stdenv out of the main nixpkgs tree
Would this make Continuous Integration less easy?
One thing I'd definitely want to have as a maintainer is swift feedback on whether changes I make break things for some architecture.
There was a problem hiding this comment.
how would you be notified that your change broke an architecture that is not checked by ofborg and the main hydra?
let's say I am the maintainer of POWER, I can:
- add power to the main nixpkgs architectures
- create my own nixpkgs-power repo. that repo contains the stdenv, overrides and power-specific packages
- create my own hydra instance that runs whenever nixpkgs master or nixpkgs-power changes => and publishes my binary cache
potentially as the POWER maintainer I could also report breakage issues with links from my hydra instance
|
|
||
| If a new Rust bootstrap is created for ppc64le, as long as it isn't adopted for the other platforms, it will be a second rustc bootstrap that may cause maintenance effort. | ||
|
|
||
| If a build infrastructure of ppc64le systems should be established, it will cause maintenance effort as well as costs for either procurement of hardware, or use of public cloud services (e. g. IntegriCloud), unless an organisation is willing to sponsor build machines. |
There was a problem hiding this comment.
Did you try and talk to IntegriCloud if they are willing to sponsor hardware for open source projects? I can arrange for the NixOS Foundation to talk to them if that helps.
One thing that you have to gage is how much people are interested in using this architecture, and how much effort you are willing to put into that project. This is really on your side I'm afraid, but it would be good to know. Obviously if the port is successful it might attract more users in the future and this hard to foresee. Do you already have a sense of today's user base that you would have?
| # Unresolved questions | ||
| [unresolved]: #unresolved-questions | ||
|
|
||
| Should the nixpkgs/NixOS project host bootstrap files for ppc64le? |
There was a problem hiding this comment.
The informal NixOS hardware team can provide some file hosting if necessary but it can't look official.
|
|
||
| Should the nixpkgs/NixOS project host bootstrap files for ppc64le? | ||
|
|
||
| Should Big Endian be in-scope for this RFC? |
There was a problem hiding this comment.
Probably not, one thing at a time.
|
I should have posted this ages ago, but yes I support the nomination to be part of the shepherd team :) however, I can't commit to be its leader at this time. |
|
I like the idea of Official vs Unofficial. Then people could start experimentally adding support for architectures without affecting other people as much. Based on @zimbatm's feedback, it seems like he wants to go in the direction of making this RFC about supported architectures generally instead of specifically ppc64. @CrystalGamma Are you willing to accept this change of scope on your RFC? |
|
I like the idea of Official vs Unofficial. Then people could start experimentally adding support for architectures without affecting other people as much. Based on @zimbatm's feedback, it seems like he wants to go in the direction of making this RFC about supported architectures generally instead of specifically ppc64. @CrystalGamma Are you willing to accept this change of scope on your RFC?
Just in case — I tried to summarise the questions to answer for proper support tiers and there are a lot of details we might want to fill in for a full RFC (I currently maintain a list of concerns at https://gist.github.com/7c6f434c/431c8233793571cad313e77356d2f7db in a gist).
I am not sure we want to pivot this RFC in a true and full solution to support tiers; maybe just say that the support is unofficial, no testing promises are made, and for the initial time ppc64-specific fixes are accepted either when they are generic cleanups with potential to benefit unrelated minority platforms, or when they are small enough and change a package that naturally depends on architecture (like compiler having a list of compilation target names).
|
|
@ryantm I would be OK with 'retargeting' this RFC to be more about how to deal with new platforms/architectures in general. That said, I don't have sufficient maintenance experience to actually make any useful technical suggestions. |
|
a few things I suggest can be done independently from this RFC:
|
|
* create a PR to add a "supported" flag to all supported architectures
I am afraid adding a «supported» flag would require determining what «supported» means — and I think we already have some sliding scale, just without codifying it.
|
|
exactly, that's why it's a good idea to add the flag and force us to clarify things. The most supported platform is "x86_64-linux" and we can start with that. |
|
pkgs/top-level/release.nix says: |
|
Right, so the flag is already there but definition is not! (On the other hand, as I mentioned in my gist, the system name doesn't fully describe the platform — we have code for |
|
@NixOS/rfc-steering-committee is ready to move this forward if @Ericson2314 thumbs up means we have the following shepherd team: @ryantm @zimbatm @grahamc @Ericson2314 |
|
Yeah I can shepherd. |
|
Who would like to be the leader (so far, Graham declined)? |
|
@ryantm @Ericson2314 Are either of you available to lead the shepherd team? |
|
@shlevy I think @Ericson2314 would be more appropriate given his knowledge of cross-compilation. But if he isn't up for it, I can try to do it. |
|
Sure I can do that. Hopefully I'll learn more about the process first hand with #13 meeting soon. |
Mic92
added
status: in discussion
and removed
status: open for nominations
|
@ryantm @zimbatm @grahamc @Ericson2314 We just discussed in the RFC steering committee whether this RFC should be kept open, or whether it makes sense to close it in light of RFC #46 (Platform Support Tiers), which addresses the more general issue of platform support. What do you think? |
|
Yes. I approved of the early plan of changing this to be like #46 had that one not been opened anyways. |
|
I second the proposal to close this PR in favor of #46 because
|
|
👍 for superseding this RFC with #46 |
|
Yes, let's close this. |
Add toLower to shard
In nixpkgs#45474, @edolstra made me aware that this warrants an RFC.
@dtzWill and @Ericson2314 had displayed interest earlier in nixpkgs#45340 (maybe one of you is interested in co-authoring?).
rendered view