Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

libstore/filetransfer: set CURLOPT_ACCEPT_ENCODING #4191

Closed
wants to merge 1 commit into from
Closed

libstore/filetransfer: set CURLOPT_ACCEPT_ENCODING #4191

wants to merge 1 commit into from

Conversation

puckipedia
Copy link
Contributor

This allows Nix to use transport-level compression, where supported.

As documented on https://curl.haxx.se/libcurl/c/CURLOPT_ACCEPT_ENCODING.html, setting an empty string will use libcurl's default.

Also, no tests ran, because nix-build failed:

ran test tests/gc.sh... [FAIL]
    ++ nix-instantiate dependencies.nix
    gc.sh: line 3: nix-instantiate: command not found
    + drvPath=

@puckipedia
libstore/filetransfer: set CURLOPT_ACCEPT_ENCODING
c9dd6cc 
This allows Nix to use transport-level compression, where supported.
@edolstra
Copy link
Member

edolstra commented Nov 2, 2020

This breaks nix log:

$ nix log /nix/store/nq8gcjq461ijxj2s28xjikyjan6xyq90-glibc-2.27.drv > /dev/null
warning: error: --- FileTransferError ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
unable to download 'https://cache.nixos.org/log/nq8gcjq461ijxj2s28xjikyjan6xyq90-glibc-2.27.drv': HTTP error 200 ('') (curl error: Unrecognized or bad HTTP Content or Transfer-Encoding); retrying in 290 ms
warning: error: --- FileTransferError ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
unable to download 'https://cache.nixos.org/log/nq8gcjq461ijxj2s28xjikyjan6xyq90-glibc-2.27.drv': HTTP error 200 ('') (curl error: Unrecognized or bad HTTP Content or Transfer-Encoding); retrying in 686 ms
warning: error: --- FileTransferError ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
unable to download 'https://cache.nixos.org/log/nq8gcjq461ijxj2s28xjikyjan6xyq90-glibc-2.27.drv': HTTP error 200 ('') (curl error: Unrecognized or bad HTTP Content or Transfer-Encoding); retrying in 1103 ms
warning: error: --- FileTransferError ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
unable to download 'https://cache.nixos.org/log/nq8gcjq461ijxj2s28xjikyjan6xyq90-glibc-2.27.drv': HTTP error 200 ('') (curl error: Unrecognized or bad HTTP Content or Transfer-Encoding); retrying in 2812 ms
error: --- FileTransferError ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
unable to download 'https://cache.nixos.org/log/nq8gcjq461ijxj2s28xjikyjan6xyq90-glibc-2.27.drv': HTTP error 200 ('') (curl error: Unrecognized or bad HTTP Content or Transfer-Encoding)

@edolstra
Copy link
Member

edolstra commented Nov 2, 2020

That file is brotli-compressed BTW. Curl gives the following warning:

curl: Unrecognized content encoding type. libcurl understands deflate, gzip content encodings.

Note that FileTransfer already does its own decompression so Accept-Encoding may interfere with that.

@puckipedia
Copy link
Contributor Author

puckipedia commented Nov 2, 2020 •
edited

This is caused by a misconfiguration of the object's properties:

content-encoding: br
content-type: text/plain; charset=utf-8

So curl's behavior is correct here.

@puckipedia
Copy link
Contributor Author

Or, well. Technically it is correct, and modern browsers will accept it, but older curl doesn't understand it (7.57.0 and up do br natively). It'd be possible to set Accept-Encoding manually, or just upgrade curl and drop the FileTransfer decompression altogether.

@edolstra
Copy link
Member

edolstra commented Nov 2, 2020

How is it misconfigured? The file is encoded using Brotli.

@puckipedia
Copy link
Contributor Author

See my reply immediately after. I misparsed it as the nix log files being inherently encoded as brotli, since i've not seen them in another format otherwise.

@stale
Copy link

stale bot commented Jun 2, 2021

I marked this as stale due to inactivity. → More info

@stale stale bot added the stale label Jun 2, 2021
@stale
Copy link

stale bot commented Jun 19, 2022

I closed this issue due to inactivity. → More info

@stale stale bot closed this Jun 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@edolstra edolstra Awaiting requested review from edolstra

Assignees
No one assigned
Labels
stale
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@puckipedia @edolstra