wasmtime: 0.21.0 -> 0.32.1 #109571

ggreif · 2021-01-16T21:52:24Z

Motivation for this change

Things done

ggreif · 2021-01-16T21:55:49Z

SuperSandro2000 · 2021-01-17T08:33:44Z

pkgs/development/interpreters/wasmtime/default.nix


  nativeBuildInputs = [ python cmake clang ];
  buildInputs = [ llvmPackages.libclang ] ++
   lib.optionals stdenv.isDarwin [ darwin.apple_sdk.frameworks.Security ];
  LIBCLANG_PATH = "${llvmPackages.libclang}/lib";

-  doCheck = true;
+  doCheck = false;


Why did you disable tests?

Because, sadly, the test traps::parse_dwarf_info is failing (see 7e8cc4f). I didn't come around analysing it yet.

Here is a more verbose output (captured from my Mac, testing today's dev version of wasmtime):

---- traps::parse_dwarf_info stdout ---- thread 'traps::parse_dwarf_info' panicked at 'rustc failed: exit code: 1 error[E0463]: can't find crate for `std` | = note: the `wasm32-wasi` target may not be installed error: aborting due to previous error For more information about this error, try `rustc --explain E0463`. ', tests/all/traps.rs:511:5

Can you add that as a comment?

SuperSandro2000 · 2021-01-17T08:34:13Z

pkgs/development/interpreters/wasmtime/default.nix

  buildInputs = [ llvmPackages.libclang ] ++
   lib.optionals stdenv.isDarwin [ darwin.apple_sdk.frameworks.Security ];


Suggested change

buildInputs = [ llvmPackages.libclang ] ++

lib.optionals stdenv.isDarwin [ darwin.apple_sdk.frameworks.Security ];

buildInputs = [ llvmPackages.libclang ]

++ lib.optionals stdenv.isDarwin [ Security ];

Please inherit this in top-level and add it as an input.

Hmm, any good reason why exposing the Security framework on the top level is beneficial? After all it is only needed on Darwin (and once), not unconditionally. When grep-ping through the sources, I see all three styles (and this file was not originally written by me). I have checked in a compromise 1fb2969, which is using with and shortens the list, while opening the frameworks only in the Darwin case.

When grep-ping through the sources

That usually include a lot of old packages which should be changed, too.

Hmm, any good reason why exposing the Security framework on the top level is beneficial?

I don't know the reason right now but darwin like pkgs shouldn't be in inputs. Using with does not solve that.

SuperSandro2000 · 2021-01-17T10:53:11Z

This is a semi-automatic executed nixpkgs-review which does not build all packages (e.g. lumo, tensorflow or pytorch)
If you find some bugs or got suggestions for further things to search or run please reach out to SuperSandro2000 on IRC.

Result of nixpkgs-review pr 109571 run on x86_64-darwin 1

1 package built:

wasmtime

siraben · 2021-08-02T10:46:51Z

Result of nixpkgs-review pr 109571 run on x86_64-linux 1

1 package failed to build:

wasmtime

risicle · 2021-09-19T17:52:43Z

https://nvd.nist.gov/vuln/detail/CVE-2021-39219 and https://nvd.nist.gov/vuln/detail/CVE-2021-39216 make this suddenly more pressing

siraben · 2021-09-20T06:43:41Z

Result of nixpkgs-review pr 109571 run on x86_64-darwin 1

1 package failed to build:

wasmtime

siraben · 2021-09-20T16:33:56Z

$ cat ~/.cache/nixpkgs-review/pr-109571/logs/wasmtime.log  | ix

http://ix.io/3zsX

risicle · 2021-09-20T18:14:45Z

In case anyone's coming up dry figuring out what posish is: https://github.com/bytecodealliance/rsix

rsix (formerly known as posish) provides efficient memory-safe and I/O-safe wrappers to POSIX-like, Unix-like, and Linux syscall APIs, with configurable backends.

risicle · 2021-09-20T18:59:06Z

Bumping straight to 0.30.0 (like we essentially have to anyway) successfully builds for me on linux, but the checkPhase fails building with the rusty_v8 component trying to download a binary:

error: failed to run custom build command for `rusty_v8 v0.27.0`

Caused by:
  process didn't exit successfully: `/build/source/target/release/build/rusty_v8-c7fdf706d2e89b97/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-changed=src/binding.cc
  download lockfile: "/build/source/target/x86_64-unknown-linux-gnu/release/build/lib_download.fslock"
  static lib URL: https://github.com/denoland/rusty_v8/releases/download/v0.27.0/librusty_v8_release_x86_64-unknown-linux-gnu.a
  cargo:rustc-link-search=/build/source/target/x86_64-unknown-linux-gnu/release/gn_out/obj
  Downloading https://github.com/denoland/rusty_v8/releases/download/v0.27.0/librusty_v8_release_x86_64-unknown-linux-gnu.a
  Downloading https://github.com/denoland/rusty_v8/releases/download/v0.27.0/librusty_v8_release_x86_64-unknown-linux-gnu.a...
  <urlopen error [Errno -2] Name or service not known>
  Retrying in 5 s ...

Looks like something deno has already had to deal with.

Edit: appears that rusty_v8 is only needed for the fuzzing-related parts of the tests and I wouldn't imagine they're any use to us anyway: https://github.com/bytecodealliance/wasmtime/blob/51131a3accb66aa7d94446b1652c2e3d968e7180/crates/fuzzing/Cargo.toml#L30

risicle · 2021-09-20T19:10:27Z

Also seems that versions prior to 0.27.0 are vulnerable to https://nvd.nist.gov/vuln/detail/CVE-2021-32629

happysalada · 2021-09-21T06:56:20Z

The futimens api it seems is only available on macos 10.13 or newer it seems.
ziglang/zig#5023
I think removing x86_64-darwin from the platforms makes sense here.
The aarch64-linux failure appears to be related to some tests, perhaps we can just disable those tests ?

avdv · 2021-10-29T08:16:20Z

Also see #143013.

Would need to bump to 0.30.0 as @risicle mentioned already.

ggreif · 2021-12-16T18:11:27Z

@GrahamcOfBorg build wasmtime

mohe2015 · 2021-12-21T15:01:01Z

error: failed to run custom build command for `v8 v0.33.0`

Caused by:
  process didn't exit successfully: `/build/source/target/release/build/v8-660f93a2f41d4306/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-changed=src/binding.cc
  cargo:rerun-if-env-changed=CARGO
  cargo:rerun-if-env-changed=CARGO_MANIFEST_DIR
  cargo:rerun-if-env-changed=CARGO_PKG_VERSION
  cargo:rerun-if-env-changed=CCACHE
  cargo:rerun-if-env-changed=CLANG_BASE_PATH
  cargo:rerun-if-env-changed=DENO_TRYBUILD
  cargo:rerun-if-env-changed=DOCS_RS
  cargo:rerun-if-env-changed=GN
  cargo:rerun-if-env-changed=GN_ARGS
  cargo:rerun-if-env-changed=HOST
  cargo:rerun-if-env-changed=NINJA
  cargo:rerun-if-env-changed=OUT_DIR
  cargo:rerun-if-env-changed=PROFILE
  cargo:rerun-if-env-changed=RUSTY_V8_ARCHIVE
  cargo:rerun-if-env-changed=RUSTY_V8_MIRROR
  cargo:rerun-if-env-changed=SCCACHE
  cargo:rerun-if-env-changed=TARGET
  cargo:rerun-if-env-changed=V8_FORCE_DEBUG
  cargo:rerun-if-env-changed=V8_FROM_SOURCE
  download lockfile: "/build/source/target/x86_64-unknown-linux-gnu/release/build/lib_download.fslock"
  static lib URL: https://github.com/denoland/rusty_v8/releases/download/v0.33.0/librusty_v8_release_x86_64-unknown-linux-gnu.a
  cargo:rustc-link-search=/build/source/target/x86_64-unknown-linux-gnu/release/gn_out/obj
  Downloading https://github.com/denoland/rusty_v8/releases/download/v0.33.0/librusty_v8_release_x86_64-unknown-linux-gnu.a
  Downloading https://github.com/denoland/rusty_v8/releases/download/v0.33.0/librusty_v8_release_x86_64-unknown-linux-gnu.a...
  <urlopen error [Errno -2] Name or service not known>
  Retrying in 5 s ...
  Downloading https://github.com/denoland/rusty_v8/releases/download/v0.33.0/librusty_v8_release_x86_64-unknown-linux-gnu.a...
  <urlopen error [Errno -2] Name or service not known>
  Retrying in 10 s ...
  Downloading https://github.com/denoland/rusty_v8/releases/download/v0.33.0/librusty_v8_release_x86_64-unknown-linux-gnu.a...
  <urlopen error [Errno -2] Name or service not known>
  Retrying in 20 s ...
  Downloading https://github.com/denoland/rusty_v8/releases/download/v0.33.0/librusty_v8_release_x86_64-unknown-linux-gnu.a...
  <urlopen error [Errno -2] Name or service not known>
  Python downloader failed, trying with curl.

  --- stderr
  Traceback (most recent call last):
    File "./tools/download_file.py", line 64, in <module>
      sys.exit(main())
    File "./tools/download_file.py", line 59, in main
      DownloadUrl(args.url, f)
    File "./tools/download_file.py", line 45, in DownloadUrl
      raise e
  urllib2.URLError: <urlopen error [Errno -2] Name or service not known>
  thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', /build/was>
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed

mohe2015 · 2021-12-21T15:01:17Z

Seems like a dependency tries to download stuff

this is supposed to improve testing frobbed from dfinity/motoko#3041

SuperSandro2000 · 2022-01-12T12:46:52Z

pkgs/development/interpreters/wasmtime/default.nix


  nativeBuildInputs = [ python cmake clang ];
  buildInputs = [ llvmPackages.libclang ] ++
   lib.optionals stdenv.isDarwin [ darwin.apple_sdk.frameworks.Security ];
  LIBCLANG_PATH = "${llvmPackages.libclang.lib}/lib";

+  configurePhase = ''


This should likely be preConfigure. If you want to skip configurePhase use dontConfigure instead.

ereslibre · 2022-03-30T17:51:54Z

This build is not hermetic, this is why it's failing. The issue comes from rusty_v8 that tries to download first v8, then gn. The logic is at https://github.com/denoland/rusty_v8/blob/v0.33.0/build.rs.

I have tried to apply this patch on top of this expression.

Now the problem is that the gn version in the nix tree is not new enough to parse the //v8/BUILD.gn bundled on the v8 crate:

  gn gen --root=/build/wasmtime-0.32.1-vendor.tar.gz/v8 /build/source/target/x86_64-unknown-linux-gnu/release/gn_out
  running: "/nix/store/ymarl788sscc8rcgr7hnx8lpcqi9x3b5-gn-unstable-2020-03-09/bin/gn" "--root=/build/wasmtime-0.32.1-vendor.tar.gz/v8" "gen" "/build/source/target/x86_64-unknown-linux-gnu/release/gn_out" "--args=is_debug=false v8_enable_handle_zapping=false clang_base_path=\"/nix/store/pv4c3d549ifd5lgpm4s5myl8arkd93b2-clang-wrapper-7.1.0\" treat_warnings_as_errors=false clang_use_chrome_plugins=false"
  ERROR at //v8/BUILD.gn:3456:16: Unknown function.
      sources += filter_include(v8_third_party_heap_files, [ "*.h" ])
                 ^-------------
  See //BUILD.gn:9:5: which caused the file to be included.
      "//v8:v8",
      ^--------

  --- stderr
  thread 'main' panicked at '
  command did not execute successfully, got: exit status: 1

  build script failed, must exit now', /build/wasmtime-0.32.1-vendor.tar.gz/v8/build.rs:620:3
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed
error: builder for '/nix/store/zpj3bss2f6l07vl4zqsbqcacp7kw9kw9-wasmtime-0.32.1.drv' failed with exit code 101

Is there a way in which I can use a specific gn version (a newer one) that includes filter_include and filter_exclude? Not sure what an acceptable practice would that be on nixpkgs...

I disregard the option of bumping gn itself given:

nixpkgs/pkgs/development/tools/build-managers/gn/default.nix

Lines 4 to 5 in e01c412

    
           # Note: Please use the recommended version for Chromium, e.g.: 
        
           # https://git.archlinux.org/svntogit/packages.git/tree/trunk/chromium-gn-version.sh?h=packages/gn

ereslibre · 2022-03-30T18:12:36Z

More information, it's the fuzzing crate the one that depends on v8:

https://github.com/bytecodealliance/wasmtime/blob/309002b02183e949c5977e998cd4ae6a66da7d84/crates/fuzzing/Cargo.toml#L25-L30

ereslibre · 2022-03-31T17:09:01Z

@ggreif: you can cherry-pick this commit on top of this PR: 60e1a92

Instead of disabling tests, I expose an environment variable to specify where the V8 archive is, so that rusty_v8 dependency of wasmtime can skip downloading the static library, what breaks hermetism during build. Instead, it uses the provided path.

ereslibre · 2022-03-31T17:09:32Z

Result:

test result: ok. 651 passed; 0 failed; 10 ignored; 0 measured; 0 filtered out; finished in 2.98s

     Running tests/host_segfault.rs (target/x86_64-unknown-linux-gnu/release/deps/host_segfault-dfa84a10b5c36f5e)
     Running tests/rlimited-memory.rs (target/x86_64-unknown-linux-gnu/release/deps/rlimited_memory-b950922710447172)

running 1 test
test custom_limiter_detect_os_oom_failure ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Finished cargoCheckHook
installing
Executing cargoInstallHook
Finished cargoInstallHook
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/2033v0j18n0psbncxh3j6vgd5spi6h96-wasmtime-0.32.1
shrinking /nix/store/2033v0j18n0psbncxh3j6vgd5spi6h96-wasmtime-0.32.1/bin/wasmtime
strip is /nix/store/xiq6j4jsyj351p8q3yw9cg1hdqp9m685-gcc-wrapper-10.3.0/bin/strip
stripping (with command strip and flags -S) in /nix/store/2033v0j18n0psbncxh3j6vgd5spi6h96-wasmtime-0.32.1/bin
patching script interpreter paths in /nix/store/2033v0j18n0psbncxh3j6vgd5spi6h96-wasmtime-0.32.1
checking for references to /build/ in /nix/store/2033v0j18n0psbncxh3j6vgd5spi6h96-wasmtime-0.32.1...
/nix/store/2033v0j18n0psbncxh3j6vgd5spi6h96-wasmtime-0.32.1

happysalada · 2022-04-02T02:18:16Z

@ereslibre do you want to make a separate pull request with your working build ? (I'd be happy to help you get it merged).

ereslibre · 2022-04-02T06:11:17Z

@happysalada I was thinking in doing so and bumping to the latest available version but I wanted to give some room for this PR to be merged since it was already open. I will open the one superseding this one at the end of my day. Thank you!

ereslibre · 2022-04-03T19:04:07Z

Closing as #166965 merged.

ggreif force-pushed the master branch from fcdf312 to 99bb754 Compare January 16, 2021 21:54

ofborg bot requested a review from matthewbauer January 16, 2021 22:04

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 10.rebuild-linux: 1-10 10.rebuild-linux: 1 labels Jan 16, 2021

SuperSandro2000 reviewed Jan 17, 2021

View reviewed changes

ggreif force-pushed the master branch from 1fb2969 to 44ac908 Compare July 15, 2021 11:45

ofborg bot added the 2.status: merge conflict label Jul 15, 2021

ggreif force-pushed the master branch from 44ac908 to ad1229d Compare July 15, 2021 12:13

ofborg bot removed the 2.status: merge conflict label Jul 15, 2021

ggreif force-pushed the master branch from ad1229d to de2355a Compare July 15, 2021 12:25

ggreif changed the title ~~wasmtime: 0.21.0 -> 0.22.0~~ wasmtime: 0.21.0 -> 0.28.0 Jul 15, 2021

risicle added the 1.severity: security label Sep 19, 2021

nomeata mentioned this pull request Dec 16, 2021

error: Found argument '--disable-cache' which wasn't expected, or isn't valid in this context dfinity/motoko-base#319

Closed

ggreif added 3 commits December 16, 2021 19:09

wasmtime: 0.21.0 -> 0.22.0

539eb3d

wasmtime: 0.22.0 -> 0.28.0

5243cda

wasmtime: 0.28.0 -> 0.32.0

969fcef

ggreif force-pushed the master branch from 0695726 to 969fcef Compare December 16, 2021 18:10

ggreif changed the title ~~wasmtime: 0.21.0 -> 0.28.0~~ wasmtime: 0.21.0 -> 0.32.0 Dec 16, 2021

ggreif mentioned this pull request Jan 4, 2022

utimensat not available on all macOS versions bytecodealliance/rustix#157

Closed

paulyoung mentioned this pull request Jan 7, 2022

nix-shell failure dfinity/motoko#3041

Open

ggreif added 2 commits January 7, 2022 10:55

apply a workaround

387d37c

this is supposed to improve testing frobbed from dfinity/motoko#3041

bump to 0.32.1

c405fb5

ggreif changed the title ~~wasmtime: 0.21.0 -> 0.32.0~~ wasmtime: 0.21.0 -> 0.32.1 Jan 10, 2022

SuperSandro2000 reviewed Jan 12, 2022

View reviewed changes

ereslibre mentioned this pull request Apr 2, 2022

wasmtime: 0.21.0 -> 0.35.2 #166965

Merged

13 tasks

ereslibre closed this Apr 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wasmtime: 0.21.0 -> 0.32.1 #109571

wasmtime: 0.21.0 -> 0.32.1 #109571

ggreif commented Jan 16, 2021 •

edited

ggreif commented Jan 16, 2021 •

edited

SuperSandro2000 Jan 17, 2021

ggreif Jan 17, 2021 •

edited

ggreif Jan 17, 2021

SuperSandro2000 Jan 17, 2021

SuperSandro2000 Jan 17, 2021

ggreif Jan 17, 2021 •

edited

SuperSandro2000 Jan 17, 2021

SuperSandro2000 commented Jan 17, 2021

siraben commented Aug 2, 2021

risicle commented Sep 19, 2021 •

edited

siraben commented Sep 20, 2021

siraben commented Sep 20, 2021

risicle commented Sep 20, 2021

risicle commented Sep 20, 2021 •

edited

risicle commented Sep 20, 2021

happysalada commented Sep 21, 2021

avdv commented Oct 29, 2021

ggreif commented Dec 16, 2021 •

edited

mohe2015 commented Dec 21, 2021

mohe2015 commented Dec 21, 2021

SuperSandro2000 Jan 12, 2022

ereslibre commented Mar 30, 2022 •

edited

ereslibre commented Mar 30, 2022 •

edited

ereslibre commented Mar 31, 2022 •

edited

ereslibre commented Mar 31, 2022

happysalada commented Apr 2, 2022

ereslibre commented Apr 2, 2022

ereslibre commented Apr 3, 2022

		buildInputs = [ llvmPackages.libclang ] ++
		lib.optionals stdenv.isDarwin [ darwin.apple_sdk.frameworks.Security ];

wasmtime: 0.21.0 -> 0.32.1 #109571

wasmtime: 0.21.0 -> 0.32.1 #109571

Conversation

ggreif commented Jan 16, 2021 • edited

Motivation for this change

Things done

ggreif commented Jan 16, 2021 • edited

SuperSandro2000 Jan 17, 2021

Choose a reason for hiding this comment

ggreif Jan 17, 2021 • edited

Choose a reason for hiding this comment

ggreif Jan 17, 2021

Choose a reason for hiding this comment

SuperSandro2000 Jan 17, 2021

Choose a reason for hiding this comment

SuperSandro2000 Jan 17, 2021

Choose a reason for hiding this comment

ggreif Jan 17, 2021 • edited

Choose a reason for hiding this comment

SuperSandro2000 Jan 17, 2021

Choose a reason for hiding this comment

SuperSandro2000 commented Jan 17, 2021

siraben commented Aug 2, 2021

risicle commented Sep 19, 2021 • edited

siraben commented Sep 20, 2021

siraben commented Sep 20, 2021

risicle commented Sep 20, 2021

risicle commented Sep 20, 2021 • edited

risicle commented Sep 20, 2021

happysalada commented Sep 21, 2021

avdv commented Oct 29, 2021

ggreif commented Dec 16, 2021 • edited

mohe2015 commented Dec 21, 2021

mohe2015 commented Dec 21, 2021

SuperSandro2000 Jan 12, 2022

Choose a reason for hiding this comment

ereslibre commented Mar 30, 2022 • edited

ereslibre commented Mar 30, 2022 • edited

ereslibre commented Mar 31, 2022 • edited

ereslibre commented Mar 31, 2022

happysalada commented Apr 2, 2022

ereslibre commented Apr 2, 2022

ereslibre commented Apr 3, 2022

ggreif commented Jan 16, 2021 •

edited

ggreif commented Jan 16, 2021 •

edited

ggreif Jan 17, 2021 •

edited

ggreif Jan 17, 2021 •

edited

risicle commented Sep 19, 2021 •

edited

risicle commented Sep 20, 2021 •

edited

ggreif commented Dec 16, 2021 •

edited

ereslibre commented Mar 30, 2022 •

edited

ereslibre commented Mar 30, 2022 •

edited

ereslibre commented Mar 31, 2022 •

edited