Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

llvmPackages_10: init at rc2 #78482

Merged
merged 1 commit into from Feb 25, 2020
Merged

llvmPackages_10: init at rc2 #78482

merged 1 commit into from Feb 25, 2020

Conversation

ggreif
Copy link
Contributor

@ggreif ggreif commented Jan 25, 2020 •
edited

Reaching out to @dtzWill for pointers.

Miscellaneous:

  • needs to be changed to target the staging branch? Not needed, no mass rebuilds will happen.
  • check which patches are still necessary (and why) -- All needed.
  • optionally: add libcompression as a dependency? (Nope, non-free.)

My build system is a MacBook Pro with High Sierra (macOS 10.13.6).

Motivation for this change

Get on the llvm-10 bandwagon, learn how to do nix stuff.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@DieGoldeneEnte
Copy link
Contributor

DieGoldeneEnte commented Jan 25, 2020 •
edited

The build failed for me, because of hash-mismatches when downloading using fetchBranchSub. I don't now how often the archive is updated, but maybe using git to get a commit of the release/10.x branch would be more stable until we have a release?

hash mismatch in fixed-output derivation '/nix/store/gkixca1c9jcqs0qwj0pns4gfkycjn117-Xlldb-11-init.tar.gz':
  wanted: sha256:02gb3fbz09kyw8n71218v5v77ip559x3gqbcp8y3w6n3jpbryywa
  got:    sha256:1cjsvyrq759hmp4cki2l2q62pdasqmjcpqkb5rdgx0rpmnrja1i2
hash mismatch in fixed-output derivation '/nix/store/q5pwliz6wa7mhxqa949271k15s4mgz8h-Xllvm-11-init.tar.gz':
  wanted: sha256:1igkyhwxffgqiglqgdprnr56hxjazzqpk15sy00i3hiwzj03issv
  got:    sha256:1gvhzk76iar7jqn7h7yq9snmzwinin86rkf9c7q16mpjxym7wq5q
hash mismatch in fixed-output derivation '/nix/store/6hw1xaib1fdv4a109h67vf3j1bppnxzp-Xlibcxx-11-init.tar.gz':
  wanted: sha256:0hca16in5zxvq7d6x82gjfdvc2xjqjw7parskq5dh6acr5jrlsvv
  got:    sha256:0ci72m76afxqi5mc68kb6y7zs25zrx4z3pffbk8cpvlyb26pkaj9
hash mismatch in fixed-output derivation '/nix/store/qq1n46yzj907m06m9ni32x07yf4f7p8j-Xpolly-11-init.tar.gz':
  wanted: sha256:1prjbpk6q34739ajvaayw5ygm0bnmcbq5fpa7dc210191rz2gcrf
  got:    sha256:16qzvbgcy2nl16c4mnisx6mr79j2xq6bkam7i7b9j864vmjymj7j
cannot build derivation '/nix/store/53175332vc7zrac00g20arq7dqli2yw9-llvm-10.0.0-branch.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/b9pfqhsl59n3cl4m59kbgblip431ap3y-llvm-10.0.0-branch.drv': 2 dependencies couldn't be built
hash mismatch in fixed-output derivation '/nix/store/7cwgvmvkxhis2zasypg9ns020w0gc9dk-Xclang-11-init.tar.gz':
  wanted: sha256:0ikq6pyx9631l3aiibwilsrxqg89bdwj68hmg5czhgmb8jda5x38
  got:    sha256:023vnakljb8858r040rxm3cbykibxi7bihnjzdg1ppjz3v8gqfnx
hash mismatch in fixed-output derivation '/nix/store/vpcs0lvppim1fn92h3c2kjn28wakaqmb-Xclang-tools-extra-11-init.tar.gz':
  wanted: sha256:0mybrjr6yyrzzn8wyyzhq8l9k2bc2x8yradli4zwmlm8brpwkmby
  got:    sha256:189l0b4lhfavjf1nb6w4dljz9l5dr8739xvlfwxshkq33jg1i03s
cannot build derivation '/nix/store/8hasjj24r426jmghi1wjbmzpyfkcn81k-clang-10.0.0-branch.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/bi0wb2divz77lr2wnp5mwa73k24yc41j-clang-10.0.0-branch.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/2cil3kdvzw1shm405vsbnrq900jngshw-lldb-10.0.0-branch.drv': 3 dependencies couldn't be built
hash mismatch in fixed-output derivation '/nix/store/v5l3v9pngqqqv5h75bli2dc8km71sff0-Xlld-11-init.tar.gz':
  wanted: sha256:10hckfxpapfnh6y9apjiya2jpw9nmbbmh8ayijx89mrg7snjn9l6
  got:    sha256:03c0m4dn47klg7r3y5p130y5ml585sbwdlyymv4gi4wvilqzzzlf
cannot build derivation '/nix/store/b0pr94wgy33skrrv3szlkwkjc98bxlix-lld-10.0.0-branch.drv': 2 dependencies couldn't be built
hash mismatch in fixed-output derivation '/nix/store/y85ywnqgh2a423bdbaxq0ih7bci52b1p-Xcompiler-rt-11-init.tar.gz':
  wanted: sha256:1j13ms1hsv8270dl5c714c6nlzf37951n5qpkgmf5wrg2dsql3df
  got:    sha256:1bkh1b0685kg6qqwvn36dnlp508978cwlqr159qi623w0mjrvvsk
cannot build derivation '/nix/store/n5283ixsq7zpa03s0mapda2nii9inkql-compiler-rt-10.0.0-branch.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/80cdwpr225q23g4pwjpcjp39nns990r5-clang-wrapper-10.0.0-branch.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/jqrcv836jkjbpz07m5365drp5xq5kd3y-clang-wrapper-10.0.0-branch.drv': 2 dependencies couldn't be built
hash mismatch in fixed-output derivation '/nix/store/rrwln0f9gyb9bkas4qc158lyrxv0ngyl-Xlibcxxabi-11-init.tar.gz':
  wanted: sha256:09df7f99wj9z61w6rx2v58nhyvzviwpjrf62vhsxrds3g8xwwi8r
  got:    sha256:1prxa8na60h04768bd3hb4xm90if4k3jzn51gbpb1lg8zs15qa1g
hash mismatch in fixed-output derivation '/nix/store/b011s3njjpf2z98lz76m1g3ddf1ixl5q-Xlibunwind-11-init.tar.gz':
  wanted: sha256:1wb02ha3gl6p0a321hwpll74pz5qvjr11xmjqx62g288f1m10njk
  got:    sha256:0wnn89qnkpd52cklp0qyn0vhpmb5h3c8nl1d5xfxi2pisvkwbnh9

@ggreif
Copy link
Contributor Author

ggreif commented Jan 25, 2020 •
edited

Possibly the tar (resp. gzip) programs don't produce reproducible output when used on different platforms. I get completely different hashes on the Mac. Maybe timestamps?

@DieGoldeneEnte I'll have to overhaul the unpacking still, so I'd prefer not to fix these right now, maybe there is a small perl script somewhere that parses the error and perl -pi -e s///g-s the gotten hashes in?

EDIT: I am writing such a script.

$ cat fix-wanted.awk
# hash mismatch in fixed-output derivation '/nix/store/gkixca1c9jcqs0qwj0pns4gfkycjn117-Xlldb-11-init.tar.gz':
#   wanted: sha256:02gb3fbz09kyw8n71218v5v77ip559x3gqbcp8y3w6n3jpbryywa
#   got:    sha256:1cjsvyrq759hmp4cki2l2q62pdasqmjcpqkb5rdgx0rpmnrja1i2
# invoke `awk -f fix-wanted.awk -F: 〈mismatch-log〉 > script.sh && sh script.sh`

/wanted: / { wanted = $3 }
/got: / { got = $3; print "git grep", wanted, "|", "awk -F: '{print $1}' | xargs perl -pi -e s/"wanted"/"got"/g" }

@ggreif ggreif mentioned this pull request Jan 26, 2020
Closed
ggreif added a commit to ggreif/nixpkgs that referenced this pull request Jan 28, 2020
@ggreif
fix up hashes as suggested by @DieGoldeneEnte
85cfc31 
see NixOS#78482 (comment)
@ggreif ggreif force-pushed the master branch 2 times, most recently from cc16213 to 604d50a Compare January 31, 2020 09:09
@ggreif ggreif marked this pull request as ready for review February 1, 2020 17:31
@ggreif
Copy link
Contributor Author

ggreif commented Feb 3, 2020

@matthewbauer @dtzWill ping for a quick review. I can quickly iterate on this.

@DieGoldeneEnte
Copy link
Contributor

I had a quick look at the missing lldb.1 file. If I interpret it correctly the man page for lldb now needs to be build with sphinx like it is already done for llvm. If you want I could try to get man pages for lldb working, once I find some time.

@ggreif
Copy link
Contributor Author

ggreif commented Feb 4, 2020 via email •
edited

@DieGoldeneEnte
Copy link
Contributor

It is there in the RC1

I could only find lldb.rst, but not lldb.1 (in the repository and the lldb-archive).
The build also fails for me with

install: cannot stat '../docs/lldb.1': No such file or directory

when building via nix-review or nix-build on your fork

@ggreif
Copy link
Contributor Author

ggreif commented Feb 4, 2020 •
edited

Sorry, you are probably right then. I didn't look inside the tarball, I just assumed that they must be there if there were there for the GitHub 9.0.1 release too. But in fact, lldb didn't even build for 9.0.1!

So maybe we'll need pandoc to convert .rst to man pages.

Something like

pandoc -s -t man ../docs/man/lldb.rst -o $out/share/man/man1/lldb.1

@DieGoldeneEnte
Copy link
Contributor

DieGoldeneEnte commented Feb 4, 2020 •
edited

something like

"-DLLVM_BUILD_DOCS=ON"
"-DLLVM_ENABLE_SPHINX=ON"
"-DSPHINX_OUTPUT_MAN=ON"
"-DSPHINX_OUTPUT_HTML=OFF"
"-DSPHINX_WARNINGS_AS_ERRORS=OFF"

should be enough (copied from llvm.nix, so some adjustments are likely needed) together with adding the needed dependencies. The cmake code can be found in lldb/docs/CMakeLists.txt .

@ggreif
Copy link
Contributor Author

ggreif commented Feb 4, 2020 •
edited

@DieGoldeneEnte added some build steps, but share/man is still not populated. It is a bit hard to debug when nix-build immediately removes everything... :-(

EDIT: found --keep-failed

@ggreif
Copy link
Contributor Author

ggreif commented Feb 5, 2020

@DieGoldeneEnte Okay, I think I cracked it. PTAL.

@DieGoldeneEnte
Copy link
Contributor

@ggreif I think it would be usefull to not build manpages per default like for llvm and add a version with manpages. This is how it is done for clang/llvm to reduce dependencies (see #26756). The manpage-dependencies should also be optional.
I will take a closer look later.

@DieGoldeneEnte DieGoldeneEnte mentioned this pull request Feb 16, 2020
Merged
@DieGoldeneEnte
Copy link
Contributor

It build with rc2. I noticed some warnings when building the clang documentation, but this is an upstream problem.
@ggreif I had another look at the substituteStream-warning you get. I believe we can run the problematic substituion only on non-darwin devices. While I don't know exactly how thefile is generated or how darwin handles shared librries, the substituteInPlace that is only run on darwin seems to have the same same function.
To test this, could you look into the lib output of llvm and check whether all files called libLLVM-* are symlinks to libLLVM.dylib. If this is the case the warning is no problem.

@ggreif
Copy link
Contributor Author

ggreif commented Feb 16, 2020

@DieGoldeneEnte, I don't see what you are at. I did

$ find /nix/store/p40b1xr5amm162hgy3l180i01i5plni9-llvm-10.0.0rc2 -name "libLLVM-*"

[nix-shell:~/nixpkgs]$ find /nix/store/p40b1xr5amm162hgy3l180i01i5plni9-llvm-10.0.0rc2 -name "libLLVM*"
/nix/store/p40b1xr5amm162hgy3l180i01i5plni9-llvm-10.0.0rc2/lib/libLLVMCFGuard.a
/nix/store/p40b1xr5amm162hgy3l180i01i5plni9-llvm-10.0.0rc2/lib/libLLVMOrcError.a
/nix/store/p40b1xr5amm162hgy3l180i01i5plni9-llvm-10.0.0rc2/lib/libLLVMMipsInfo.a
/nix/store/p40b1xr5amm162hgy3l180i01i5plni9-llvm-10.0.0rc2/lib/libLLVMGlobalISel.a
/nix/store/p40b1xr5amm162hgy3l180i01i5plni9-llvm-10.0.0rc2/lib/libLLVMAnalysis.a
<snip>

$ find /nix/store/p40b1xr5amm162hgy3l180i01i5plni9-llvm-10.0.0rc2 -name "*dylib"
/nix/store/p40b1xr5amm162hgy3l180i01i5plni9-llvm-10.0.0rc2/lib/libRemarks.dylib
/nix/store/p40b1xr5amm162hgy3l180i01i5plni9-llvm-10.0.0rc2/lib/libLTO.dylib

There is no libLLVM-*.dylib around.

@ggreif
Copy link
Contributor Author

ggreif commented Feb 16, 2020 •
edited

Maybe we should do .dylibs one day.

@DieGoldeneEnte
Copy link
Contributor

@ggreif You have to look in the llvm-10.0.0rc2-lib output (not sure what the best way to find it is, I just search in /nix/store/), since libLLVM-* gets moved there.

@ggreif
Copy link
Contributor Author

ggreif commented Feb 16, 2020 via email

@DieGoldeneEnte
Copy link
Contributor

LGTM and should be mergeable after squashing.

@ggreif
Copy link
Contributor Author

ggreif commented Feb 16, 2020 •
edited

@DieGoldeneEnte Successfully built on Darwin and squashed. Also rebased to master. PTAL.

@ggreif ggreif changed the title llvmPackages_10: init llvmPackages_10: init at rc2 Feb 17, 2020
@ggreif
Copy link
Contributor Author

ggreif commented Feb 18, 2020

@matthewbauer @dtzWill ping.

@DieGoldeneEnte
Copy link
Contributor

LGTM, only thing I noticed is the homepages are all http instead of https. Do we change this here or create an extra PR for all versions?

@ggreif
Copy link
Contributor Author

ggreif commented Feb 18, 2020 via email

@basvandijk
Copy link
Member

@GrahamcOfBorg build clang_10 lld_10 lldb_10 llvm_10

@ggreif
llvmPackages_10: copy llvmPackages_9
f111c6f 
* starting with rc2
* make `lldb` compilable again on Darwin
* separate out manpage creation for `lldb` into a new derivation
* minor tweaks to the patching of sources,
  some of which are backportable to earlier versions
@basvandijk
Copy link
Member

I've talked with @ggreif offline that we shouldn't update llvmPackages_latest from 9 to 10-rc2. We can update it to 10 when it gets released.

So now this PR just adds llvmPackages_10 and doesn't change existing attributes. So I'll go ahead and merge it.

@basvandijk basvandijk merged commit b63491f into NixOS:master Feb 25, 2020
@DieGoldeneEnte
Copy link
Contributor

Thanks @basvandijk.
Not updating llvmPackages_latest for now is probably a good idea.

Just as a reference (in case a upgrade to llvm10 breaks a package using polly): There might be some changes to how compilation using polly will work with llvm_10 (see 45001 or llvm-project#120). The change is currently in llvm10-rc2, but might be reverted.

@ggreif ggreif mentioned this pull request Feb 25, 2020
Merged
10 tasks
dtzWill pushed a commit to dtzWill/nixpkgs that referenced this pull request Feb 26, 2020
@basvandijk @dtzWill
Merge pull request NixOS#78482 from ggreif/master
bc59456 
llvmPackages_10: init at rc2

(cherry picked from commit b63491f)
@primeos primeos mentioned this pull request Feb 26, 2020
Merged
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DieGoldeneEnte DieGoldeneEnte DieGoldeneEnte left review comments

@dtzWill dtzWill Awaiting requested review from dtzWill

@lovek323 lovek323 Awaiting requested review from lovek323

@7c6f434c 7c6f434c Awaiting requested review from 7c6f434c

@matthewbauer matthewbauer Awaiting requested review from matthewbauer

Assignees
No one assigned
Labels
8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@ggreif @DieGoldeneEnte @basvandijk