Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nar-serve: init at 0.3.0 #95420

Merged
merged 2 commits into from Nov 6, 2020
Merged

nar-serve: init at 0.3.0 #95420

merged 2 commits into from Nov 6, 2020

Conversation

Rizary
Copy link
Contributor

@Rizary Rizary commented Aug 14, 2020

@zimbatm I need your input regarding the test module.

Motivation for this change

initialize nar-serve and add the test for nar-serve

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@ofborg ofborg bot requested a review from kalbasit August 14, 2020 12:31
@freezeboy
Copy link
Contributor

There are AWS tokens in this PR ...

@zimbatm
Copy link
Member

zimbatm commented Aug 14, 2020 •
edited

they are fake ones (borrowed from the minio.nix test)

@zimbatm
Copy link
Member

zimbatm commented Aug 14, 2020

@Rizary I improved the tests a bit but it's still failing (see the commit message)

@Rizary Rizary requested a review from zimbatm August 14, 2020 16:37
@zimbatm
Copy link
Member

zimbatm commented Aug 14, 2020

@ofborg test nar-serve

1 similar comment
@zimbatm
Copy link
Member

zimbatm commented Aug 14, 2020

@ofborg test nar-serve

zimbatm
zimbatm reviewed Aug 17, 2020
View reviewed changes
nixos/modules/services/networking/nar-serve.nix Outdated Show resolved Hide resolved
nixos/modules/services/networking/nar-serve.nix Outdated
Comment on lines 31 to 34
- HttpBinaryCacheStore (http://, https://) for binary caches via HTTP or HTTPS
- S3BinaryCacheStore (s3://) for binary caches stored in Amazon S3
- LegacySSHStore (ssh://) for implement remote builds and nix-copy-closure
- SSHStore (ssh-ng://) for arbitrary Nix operations on a remote machine via the same protocol used by nix-daemon
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nar-serve doesn't support the ssh protocols

Suggested change
- HttpBinaryCacheStore (http://, https://) for binary caches via HTTP or HTTPS
- S3BinaryCacheStore (s3://) for binary caches stored in Amazon S3
- LegacySSHStore (ssh://) for implement remote builds and nix-copy-closure
- SSHStore (ssh-ng://) for arbitrary Nix operations on a remote machine via the same protocol used by nix-daemon
- http://, https:// for binary caches via HTTP or HTTPS
- s3:// for binary caches stored in Amazon S3
- gs:// for binary caches stored in Google Cloud Storage

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does nar-serve reimplement the store implementations? Seems better to use commands like nix cat-nar --store <...> since then it would support all stores automatically.

Copy link
Member

@zimbatm zimbatm Aug 17, 2020 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a pure Go implementation at the moment. Having our own implementation will allow to add support for on-disk caching for example (that doesn't necessarily unpack the NAR file).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also allowed us to easily add Google Cloud Storage as a supported backend.

@Rizary Rizary force-pushed the rizary branch 3 times, most recently from b5260cc to 0da57e6 Compare August 18, 2020 14:51
@zimbatm zimbatm changed the title nar-serve: init at 0.1.0 nar-serve: init at 0.2.0 Aug 18, 2020
@zimbatm
Copy link
Member

zimbatm commented Aug 18, 2020

Result of nixpkgs-review pr 95420 1

1 package built:
- nar-serve

@zimbatm
Copy link
Member

zimbatm commented Aug 18, 2020

@ofborg test nar-serve

@zimbatm
Copy link
Member

zimbatm commented Aug 18, 2020 

$ nix-build -A nixosTests.nar-serve
<snip>
server: must succeed: curl -o /bin/hello -sf http://localhost:8080/nix/store/ab1pfk338f6gzpglsirxhvji4g9w558i/bin/hello
server # [   18.692917] nar-serve[717]: 3 [ab1pfk338f6gzpglsirxhvji4g9w558i bin hello]
server # [   18.696662] nar-serve[717]: Fetching the narinfo: ab1pfk338f6gzpglsirxhvji4g9w558i.narinfo from: https://cache.nixos.org
server # [   18.711084] nar-serve[717]: [negroni] 2020-08-18T16:14:00Z | 500 |   36.709693ms | localhost:8080 | GET /nix/store/ab1pfk338f6gzpglsirxhvji4g9w558i/bin/hello
server: output: 
error: 
Traceback (most recent call last):
  File "/nix/store/hmali2zlviy4smsq3cpz527dn16jy9br-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 894, in run_tests
    exec(tests, globals())
  File "<string>", line 1, in <module>
  File "<string>", line 25, in <module>
  File "/nix/store/hmali2zlviy4smsq3cpz527dn16jy9br-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 421, in succeed
    raise Exception(
Exception: command `curl -o /bin/hello -sf http://localhost:8080/nix/store/ab1pfk338f6gzpglsirxhvji4g9w558i/bin/hello` failed (exit code 22)
cleaning up
killing server (pid 9)
(0.00 seconds)
error: --- Error ----------------------------------------------------------------------------------------- nix-build
error: --- Error --- nix-daemon
builder for '/nix/store/w63vq84p6aa97f50z37l40kqcfh9ryjd-vm-test-run-nar-serve.drv' failed with exit code 1; last 10 log lines:
<snip>

@SuperSandro2000
Copy link
Member

What is the status of this PR?

@zimbatm
Copy link
Member

zimbatm commented Oct 22, 2020

@ofborg test nar-serve

@zimbatm zimbatm changed the title nar-serve: init at 0.2.0 nar-serve: init at 0.3.0 Oct 24, 2020
@zimbatm
Copy link
Member

zimbatm commented Oct 24, 2020

still stuck on the NixOS test failing with a 500 error

@SuperSandro2000
Copy link
Member

@zimbatm Seems to be green now.

@zimbatm
Copy link
Member

zimbatm commented Oct 24, 2020

It's because the NixOS test is not being executed by ofborg.

@zimbatm zimbatm force-pushed the rizary branch 2 times, most recently from b8ffc92 to ea7c46e Compare November 6, 2020 14:40
@zimbatm
Copy link
Member

zimbatm commented Nov 6, 2020

@ofborg test nar-serve

@zimbatm zimbatm merged commit 038497d into NixOS:master Nov 6, 2020
@zimbatm zimbatm deleted the rizary branch November 6, 2020 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zimbatm zimbatm zimbatm left review comments

@edolstra edolstra edolstra left review comments

@kalbasit kalbasit Awaiting requested review from kalbasit

Assignees
No one assigned
Labels
6.topic: nixos 8.has: module (update) 8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 10.rebuild-linux: 1-10 11.by: package-maintainer
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@Rizary @freezeboy @zimbatm @SuperSandro2000 @edolstra