Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add support for specifying 'compression-level' for NAR's #2276

Closed
wants to merge 1 commit into from
Closed

add support for specifying 'compression-level' for NAR's #2276

wants to merge 1 commit into from

Conversation

dtzWill
Copy link
Member

@dtzWill dtzWill commented Jul 5, 2018

As requested on #2255.

Lightly tested, but could probably use a bit more vetting :).

May be useful to also add:

  • Documentation
  • New tests

I'll probably put together at least some test for this,
but help with documentation (what/where/how?) would be appreciated.

As an aside, as I was writing the "description" for Setting's that
are part of BinaryCacheStore I wondered:

Are these flags visible somewhere to the user? I couldn't find any way
to get the store-oriented flags to be listed (with or without descriptions/values),
is this currently possible?

@dtzWill dtzWill mentioned this pull request Jul 5, 2018
Closed
@dtzWill
Copy link
Member Author

dtzWill commented Sep 27, 2018

Rebased to resolve conflicts and work with compressionsink introduction.

@tomberek tomberek mentioned this pull request Jan 4, 2020
Open
@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-slowdown-handling-large-files-derivations/5310/3

@tomberek
Copy link
Contributor

tomberek commented Jan 5, 2020 

[tom@tom:~/nix]$ dd if=/dev/urandom of=400MB.rand bs=4096 count=$( expr 100 \* 1024 )
102400+0 records in
102400+0 records out
419430400 bytes (419 MB, 400 MiB) copied, 2.12592 s, 197 MB/s

[tom@tom:~/nix]$ time nix copy --to "file://store?parallel-compression=true" /nix/store/7sy62s
nicdvfb7g0q5spi1y7gpkay6vz-400MB.rand
warning: dumping very large path (> 256 MiB); this may run out of memory
[1 copied (400.0 MiB)]

real    0m19.810s
user    3m29.199s
sys     0m2.083s

[tom@tom:~/nix]$ time ./result-nix/bin/nix copy --to "file://store?compression-level=0&paralle
l-compression=true" /nix/store/7sy62snicdvfb7g0q5spi1y7gpkay6vz-400MB.rand
warning: dumping very large path (> 256 MiB); this may run out of memory
[1 copied (400.0 MiB)]

real    0m10.548s
user    2m12.592s
sys     0m0.835s

[tom@tom:~/nix]$ time ./result-nix/bin/nix copy --to "file://store?compression-level=9&parallel-compression=true" /nix/store/7sy62snicdvfb7g0q5spi1y7gpkay6vz-400MB.rand
warning: dumping very large path (> 256 MiB); this may run out of memory
[1 copied (400.0 MiB)]

real    1m25.031s
user    2m48.845s
sys     0m1.413s

[tom@tom:~/nix]$ time ./result-nix/bin/nix copy --to "file://store?compression-level=6&paralle
l-compression=true" /nix/store/7sy62snicdvfb7g0q5spi1y7gpkay6vz-400MB.rand
warning: dumping very large path (> 256 MiB); this may run out of memory
[1 copied (400.0 MiB)]

real    0m19.636s
user    3m30.792s
sys     0m2.007s

Light testing seems to function as expected.

  • compression-level=0&parallel-compression=true
  • compression-level=6&parallel-compression=true
  • compression-level=1&parallel-compression=false
  • compression-level=1&parallel-compression=false&compression=bzip2

Seems to work well. @dtzWill Can this be revived?

Documentation of the various store URI options is still an open issue: #1556

@stale
Copy link

stale bot commented Feb 13, 2021

I marked this as stale due to inactivity. → More info

@stale stale bot added the stale label Feb 13, 2021
@yorickvP
Copy link
Contributor

Still important to me

@stale stale bot removed the stale label Feb 13, 2021
@yorickvP yorickvP mentioned this pull request Mar 10, 2021
Merged
5 tasks
@stale
Copy link

stale bot commented Aug 13, 2021

I marked this as stale due to inactivity. → More info

@stale stale bot added the stale label Aug 13, 2021
tomberek added a commit to tomberek/nix that referenced this pull request Oct 12, 2021
@tomberek
Add compression level for NARs
03bb8f8 
Based off on @dtzWill's NixOS#2276
@tomberek tomberek mentioned this pull request Oct 12, 2021
Merged
@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/speed-up-nix-copy/15884/3

dramforever pushed a commit to dramforever/nix that referenced this pull request Nov 27, 2021
@tomberek @dramforever
Add compression level for NARs
4fd0123 
Based off on @dtzWill's NixOS#2276
@tomberek
Copy link
Contributor

Fixed in #5372

@tomberek tomberek closed this Feb 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
stale
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@dtzWill @nixos-discourse @tomberek @yorickvP