Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

llvmPackages_10: rc5 -> release #83350

Merged
merged 1 commit into from Apr 1, 2020
Merged

Conversation

DieGoldeneEnte
Copy link
Contributor

@DieGoldeneEnte DieGoldeneEnte commented Mar 25, 2020

Motivation for this change

LLVM 10.0.0 was released (http://lists.llvm.org/pipermail/llvm-dev/2020-March/140233.html)

@ggreif I fixed the hashes, hopefully you haven't already :)

Things done

updated hashes and url for release code
I checked with nix-review, only chromiumDev depends on this, but it doesn't build even without this commit, because of patches.

  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@DieGoldeneEnte
Copy link
Contributor Author

@volth This doesn't build for me, because the commit is not found ("reference is not a tree"). How can I regenerate the vendor file (./pkgs/applications/networking/browsers/chromium-git/vendor-82.0.4082.1.nix in this case)?

@@ -6,15 +6,14 @@

let
release_version = "10.0.0";
candidate = "rc5";
version = "10.0.0${candidate}"; # differentiating these is important for rc's
version = release_version; # differentiating these is important for rc's
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is confusing now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would you write it? We need a comment to make clear, why we have two variables with the same value. I just kept it this way, because this is consistent with the other llvm versions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Never mind then. Or take my suggestion.

Suggested change
version = release_version; # differentiating these is important for rc's
version = release_version; # differentiating these (variables) is important for rc's

I'll look into the history when 11-rc1 comes along, anyway.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should have had version = "${release_version}${candidate}". Too late now :-)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added it into the commit, it's a lot clearer what is meant this way. Thanks!

@ggreif
Copy link
Contributor

ggreif commented Mar 25, 2020

I use a script to fix the hashes:

$ cat fix-wanted.awk 
# hash mismatch in fixed-output derivation '/nix/store/gkixca1c9jcqs0qwj0pns4gfkycjn117-Xlldb-11-init.tar.gz':
#   wanted: sha256:02gb3fbz09kyw8n71218v5v77ip559x3gqbcp8y3w6n3jpbryywa
#   got:    sha256:1cjsvyrq759hmp4cki2l2q62pdasqmjcpqkb5rdgx0rpmnrja1i2

# invoke `awk -f fix-wanted.awk -F: <mismatch-log>` > script.sh && sh script.sh
# or build like this: `nix-build -j8 -A llvmPackages_10 |& tee /dev/tty | awk -f fix-wanted.awk -F: | sh`

/wanted: / { wanted = $3 }
/got: / { got = $3; print "git grep", wanted, "|", "awk -F: '{print $1}' | xargs perl -pi -e s/"wanted"/"got"/g" }

Building with this command line: nix-build -j8 -A llvmPackages_10 |& tee /dev/tty | awk -f fix-wanted.awk -F: | sh

@ggreif
Copy link
Contributor

ggreif commented Mar 25, 2020

@DieGoldeneEnte can you link to http://lists.llvm.org/pipermail/llvm-dev/2020-March/140233.html from the description?

updated hashes and url
updated comment regarding version/release_version to make it clearer
@DieGoldeneEnte
Copy link
Contributor Author

@ggreif added it to the PR-text.

@primeos
Copy link
Member

primeos commented Mar 30, 2020

Just a quick note so that we don't forget about it: After this PR is merged we should also backport LLVM 10 to 20.03 as Chromium 83 (82 was cancelled) depends on it and will be released ~May 19, 2020 (schedule).

And regarding chromiumDev: I fixed the build in the meantime, but it requires a1b4bfe, which isn't in master yet (but feel free to ignore chromiumDev, I can test it again after the next update).

@primeos primeos added the 9.needs: port to stable A PR needs a backport to the stable release. label Mar 30, 2020
@ggreif
Copy link
Contributor

ggreif commented Apr 1, 2020

@basvandijk @dtzWill @matthewbauer @lovek323 @7c6f434c this is a pretty boring bump, can this be merged?

@basvandijk
Copy link
Member

@GrahamcOfBorg build llvm

@basvandijk basvandijk merged commit d44a752 into NixOS:master Apr 1, 2020
@ggreif
Copy link
Contributor

ggreif commented Apr 1, 2020

@GrahamcOfBorg build llvm_10

DieGoldeneEnte pushed a commit to DieGoldeneEnte/nixpkgs that referenced this pull request Apr 2, 2020
Backport of llvmPackages_10 since Chromium 83 will depend on it.
See NixOS#83350 (comment)

llvmPackages_10: copy llvmPackages_9

* starting with rc2
* make `lldb` compilable again on Darwin
* separate out manpage creation for `lldb` into a new derivation
* minor tweaks to the patching of sources,
  some of which are backportable to earlier versions

(cherry picked from commit f111c6f)

llvmPackages_10: rc2 -> rc3

http://lists.llvm.org/pipermail/llvm-dev/2020-March/139729.html

Additionally cherry-picked 3 commits from `llvm-project/master`:
- llvm/llvm-project@d21664c
- llvm/llvm-project@3a0f6e6
- llvm/llvm-project@87dac7d

such that clang can automatically pick up the polly plugin from the
`llvm-polly` build.

(cherry picked from commit 3a84353)

llvmPackages_10: rc3 -> rc4

Only needed to update hashes and the version.
Updated comment for extension handling patch

(cherry picked from commit 0ec3f4e)

llvmPackages_10: removed extra polly-build

There is no good reason to have separate builds for polly and no-polly
versions. wwwwwThe reason for the two versions was (as far as I can
tell) to defer rebuilds (see ed60483).
Polly is now enabled by default.

(cherry picked from commit e9aa877)

llvmPackges_10: rc4 -> rc5

updated versino and hashes for new rc

(cherry picked from commit cdee144)

llvmPackages_10: rc5 -> release

updated hashes and url
updated comment regarding version/release_version to make it clearer

(cherry picked from commit 4665b2a)
@ggreif
Copy link
Contributor

ggreif commented May 8, 2020

@DieGoldeneEnte quick reality check: can you successfully start LLDB and run the script command in it on Linux? I got something saying that the No module named '_posixsubprocess' module is missing.

On macOS all is fine:

[nix-shell:~/motoko/wasmtime]$ lldb
(lldb) script
Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.
>>> 

@DieGoldeneEnte
Copy link
Contributor Author

@ggreif It works for me

[nix-shell:~]$ lldb
(lldb) script
Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.
>>> ^D
now exiting InteractiveConsole...
(lldb) version
lldb version 10.0.0
(lldb) ^D

My output for nix-shell -p nix-info --run "nix-info -m":

 - system: `"x86_64-linux"`
 - host os: `Linux 5.5.9, NixOS, 20.09pre217537.d96bd3394b7 (Nightingale)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.3`
 - channels(holger): `""`
 - channels(root): `"nixos-20.09pre223023.fce7562cf46, old-18.09.2574.a7e559a5504, unstable"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

@ggreif
Copy link
Contributor

ggreif commented May 8, 2020

I get

[nix-shell:~/motoko]$ lldb
(lldb) script
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/nix/store/2fkjplplm20jhx6ybq4vqj391il7pf51-lldb-10.0.0/lib/python3.7/site-packages/lldb/__init__.py", line 101, in <module>
    import six
  File "/nix/store/2fkjplplm20jhx6ybq4vqj391il7pf51-lldb-10.0.0/lib/python3.7/site-packages/six.py", line 644, in <module>
    import struct
  File "/usr/lib/python3.7/struct.py", line 13, in <module>
    from _struct import *
ModuleNotFoundError: No module named '_struct'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 12, in <module>
    import subprocess, tempfile, os.path, re, pwd, grp, os, time, io
  File "/usr/lib/python3.7/subprocess.py", line 152, in <module>
    import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'

Original exception was:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/nix/store/2fkjplplm20jhx6ybq4vqj391il7pf51-lldb-10.0.0/lib/python3.7/site-packages/lldb/__init__.py", line 101, in <module>
    import six
  File "/nix/store/2fkjplplm20jhx6ybq4vqj391il7pf51-lldb-10.0.0/lib/python3.7/site-packages/six.py", line 644, in <module>
    import struct
  File "/usr/lib/python3.7/struct.py", line 13, in <module>
    from _struct import *
ModuleNotFoundError: No module named '_struct'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'run_one_line' is not defined
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 12, in <module>
    import subprocess, tempfile, os.path, re, pwd, grp, os, time, io
  File "/usr/lib/python3.7/subprocess.py", line 152, in <module>
    import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'

Original exception was:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'run_one_line' is not defined
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'run_one_line' is not defined
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 12, in <module>
    import subprocess, tempfile, os.path, re, pwd, grp, os, time, io
  File "/usr/lib/python3.7/subprocess.py", line 152, in <module>
    import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'

Original exception was:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'run_one_line' is not defined
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'run_one_line' is not defined
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 12, in <module>
    import subprocess, tempfile, os.path, re, pwd, grp, os, time, io
  File "/usr/lib/python3.7/subprocess.py", line 152, in <module>
    import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'

Original exception was:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'run_one_line' is not defined
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'run_one_line' is not defined
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 12, in <module>
    import subprocess, tempfile, os.path, re, pwd, grp, os, time, io
  File "/usr/lib/python3.7/subprocess.py", line 152, in <module>
    import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'

Original exception was:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'run_one_line' is not defined
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'run_one_line' is not defined
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 12, in <module>
    import subprocess, tempfile, os.path, re, pwd, grp, os, time, io
  File "/usr/lib/python3.7/subprocess.py", line 152, in <module>
    import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'

Original exception was:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'run_one_line' is not defined
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'run_one_line' is not defined
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 12, in <module>
    import subprocess, tempfile, os.path, re, pwd, grp, os, time, io
  File "/usr/lib/python3.7/subprocess.py", line 152, in <module>
    import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'

Original exception was:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'run_one_line' is not defined
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'run_python_interpreter' is not defined
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 12, in <module>
    import subprocess, tempfile, os.path, re, pwd, grp, os, time, io
  File "/usr/lib/python3.7/subprocess.py", line 152, in <module>
    import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'

Original exception was:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'run_python_interpreter' is not defined
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'lldb' is not defined
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 12, in <module>
    import subprocess, tempfile, os.path, re, pwd, grp, os, time, io
  File "/usr/lib/python3.7/subprocess.py", line 152, in <module>
    import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'

Original exception was:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'lldb' is not defined
(lldb) 

Maybe we can track this down?

[nix-shell:~/motoko]$ nix-shell -p nix-info --run "nix-info -m"
these paths will be fetched (0.00 MiB download, 0.00 MiB unpacked):
  /nix/store/8gygka3jiyh5gs936s2jy2i5dxk3s18s-nix-info
copying path '/nix/store/8gygka3jiyh5gs936s2jy2i5dxk3s18s-nix-info' from 'https://cache.nixos.org'...
 - system: `"x86_64-linux"`
 - host os: `Linux 5.3.0-46-generic, Ubuntu, 19.10 (Eoan Ermine)`
 - multi-user?: `no`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.4`
 - channels(ggreif): `"nixpkgs-20.09pre221322.8686922e68d"`
 - nixpkgs: `/home/ggreif/.nix-defexpr/channels/nixpkgs`

@ggreif
Copy link
Contributor

ggreif commented May 8, 2020

Looks like the lldb installed by nix picks up the python3 from the path. This seems odd. It picks up

ggreif@vm-ggreif:~/llvm-project/lldb$ which python3
/usr/bin/python3
ggreif@vm-ggreif:~/llvm-project/lldb$ python3 --version
Python 3.7.5

on my machine.

After doing nix-env -i python3-3.7.7 executing script in lldb started working for me.

So I guess we should specify a python3-dependency into the nix-expression for lldb. What do you think? @DieGoldeneEnte

@ggreif
Copy link
Contributor

ggreif commented May 8, 2020

I guess this should work in a derivation recipe: printf 'script\nexit\n' | lldb > $out.

and strangely, the following derivation builds just fine in my nix-repl;

stdenv.mkDerivation {
  name = "lldb-scriptable";
  buildInputs = [ coreutils lldb_10 ];
  installPhase = "export PATH=$coreutils/bin; printf 'script\nexit\n' | ${lldb_10}/bin/lldb > $out";
  phases = [ "installPhase" ];
}

@DieGoldeneEnte
Copy link
Contributor Author

Looks like the lldb installed by nix picks up the python3 from the path.
I disagree. I get different results for the system-python and lldb-python (because I didn't rebuild my system in a while):

[nix-shell:~]$ python3
Python 3.7.6 (default, Dec 18 2019, 19:23:55) 
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys;sys.version
'3.7.6 (default, Dec 18 2019, 19:23:55) \n[GCC 9.2.0]'
>>> 
[nix-shell:~]$ lldb 
(lldb) script 
Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.
>>> import sys;sys.version
'3.7.7 (default, Mar 10 2020, 06:34:06) \n[GCC 9.3.0]'
>>> ^D
now exiting InteractiveConsole...
(lldb) ^D

You can try out this commit, it should set the python path explicitly during building, in case it finds a different version fist.

From ee3ccc14b51179b5a21218d5e2c31a0dec78e99c Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Holger=20W=C3=BCnsche?= <holger.o.wuensche@t-online.de>
Date: Sat, 9 May 2020 19:59:12 +0200
Subject: [PATCH] lldb_10: explicitly fixed python path

lldb might pick wrong python during installation. This chage explicitly
tells lldb which path to use. LLDB_RELOCATABLE_PYTHON set to its default
value.
---
 pkgs/development/compilers/llvm/10/lldb.nix | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/pkgs/development/compilers/llvm/10/lldb.nix b/pkgs/development/compilers/llvm/10/lldb.nix
index 90716f67f83..9e8a7006c28 100644
--- a/pkgs/development/compilers/llvm/10/lldb.nix
+++ b/pkgs/development/compilers/llvm/10/lldb.nix
@@ -50,6 +50,8 @@ stdenv.mkDerivation (rec {
     "-DLLDB_CODESIGN_IDENTITY=" # codesigning makes nondeterministic
     "-DClang_DIR=${clang-unwrapped}/lib/cmake"
     "-DLLVM_EXTERNAL_LIT=${lit}/bin/lit"
+    "-DPYTHON_HOME=${python3}"
+    "-DLLDB_RELOCATABLE_PYTHON=0"
   ] ++ stdenv.lib.optionals stdenv.isDarwin [
     "-DLLDB_USE_SYSTEM_DEBUGSERVER=ON"
   ] ++ stdenv.lib.optionals enableManpages [
-- 
2.25.1

stigok pushed a commit to stigok/nixpkgs that referenced this pull request Jun 12, 2020
Backport of llvmPackages_10 since Chromium 83 will depend on it.
See NixOS#83350 (comment)

llvmPackages_10: copy llvmPackages_9

* starting with rc2
* make `lldb` compilable again on Darwin
* separate out manpage creation for `lldb` into a new derivation
* minor tweaks to the patching of sources,
  some of which are backportable to earlier versions

(cherry picked from commit f111c6f)

llvmPackages_10: rc2 -> rc3

http://lists.llvm.org/pipermail/llvm-dev/2020-March/139729.html

Additionally cherry-picked 3 commits from `llvm-project/master`:
- llvm/llvm-project@d21664c
- llvm/llvm-project@3a0f6e6
- llvm/llvm-project@87dac7d

such that clang can automatically pick up the polly plugin from the
`llvm-polly` build.

(cherry picked from commit 3a84353)

llvmPackages_10: rc3 -> rc4

Only needed to update hashes and the version.
Updated comment for extension handling patch

(cherry picked from commit 0ec3f4e)

llvmPackages_10: removed extra polly-build

There is no good reason to have separate builds for polly and no-polly
versions. wwwwwThe reason for the two versions was (as far as I can
tell) to defer rebuilds (see ed60483).
Polly is now enabled by default.

(cherry picked from commit e9aa877)

llvmPackges_10: rc4 -> rc5

updated versino and hashes for new rc

(cherry picked from commit cdee144)

llvmPackages_10: rc5 -> release

updated hashes and url
updated comment regarding version/release_version to make it clearer

(cherry picked from commit 4665b2a)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants