Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

treewide: http -> https sources #42676

Merged
merged 3 commits into from Jun 28, 2018
Merged

treewide: http -> https sources #42676

merged 3 commits into from Jun 28, 2018

Conversation

infinisil
Copy link
Member

This updates the source urls of all top-level packages from http to https where possible. This is done using https://github.com/Infinisil/nix-url-verify which is a mess right now, but the basic functionality works. While it will be able to change urls in regards to redirects too, I think http -> https is a good start. This is in the sense of the recent #42665 by @ryantm

What my program does is:

  • Go through all src urls of top-level attributes
  • If they are using http, make a request to the https version
  • If the request succeeds (2xx status code), the url gets updated

This does not verify the hashes, which shouldn't be needed anyways. Because the hashes stay the same, this also shouldn't trigger any builds.

Here is a list of all attributes whose source url has been changed: https://gist.github.com/d64724f7c284ce8dd4fed9bb0748dde6

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@infinisil
treewide: http -> https sources
a76e42e 
This updates the source urls of all top-level packages from http to
https where possible.
@infinisil infinisil requested a review from peti as a code owner June 27, 2018 20:52
infinisil
infinisil commented Jun 27, 2018
View reviewed changes
pkgs/tools/misc/buildtorrent/default.nix Outdated
@@ -6,7 +6,7 @@ stdenv.mkDerivation rec {
name = "buildtorrent-${version}";

src = fetchurl {
url = "http://mathr.co.uk/blog/code/${name}.tar.gz";
url = "http://mathr.https://mathr.co.uk/blog/code/z";.tar.gz
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It somehow messed up here.. Not sure what happened

@infinisil
Copy link
Member Author

I figured it out, that file used tabs instead of spaces, which apparently can trip up hnix, ping @jwiegley

@GrahamcOfBorg eval

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: buildtorrent

Partial log (click to expand)

these paths will be fetched (0.01 MiB download, 0.03 MiB unpacked):
  /nix/store/gafxh5m75by744rbkf70rjxfficrgp51-buildtorrent-0.8
copying path '/nix/store/gafxh5m75by744rbkf70rjxfficrgp51-buildtorrent-0.8' from 'https://cache.nixos.org'...
/nix/store/gafxh5m75by744rbkf70rjxfficrgp51-buildtorrent-0.8

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: buildtorrent

Partial log (click to expand)

these paths will be fetched (0.02 MiB download, 0.04 MiB unpacked):
  /nix/store/zx76nmj0lgzsv4ss8nh3jnj44v0bicb2-buildtorrent-0.8
copying path '/nix/store/zx76nmj0lgzsv4ss8nh3jnj44v0bicb2-buildtorrent-0.8' from 'https://cache.nixos.org'...
/nix/store/zx76nmj0lgzsv4ss8nh3jnj44v0bicb2-buildtorrent-0.8

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: buildtorrent

Partial log (click to expand)

/nix/store/zx76nmj0lgzsv4ss8nh3jnj44v0bicb2-buildtorrent-0.8

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: buildtorrent

Partial log (click to expand)

/nix/store/gafxh5m75by744rbkf70rjxfficrgp51-buildtorrent-0.8

@xeji
Copy link
Contributor

xeji commented Jun 28, 2018

Thanks!
LGTM. Please fix the conflicts so we can merge this.

@grahamc
Copy link
Member

grahamc commented Jun 28, 2018

@LnL7 asks a great question -- can the bootstrap fetchurl (ie: pre-stdenv) support HTTPS URLs?

@infinisil
Copy link
Member Author

infinisil commented Jun 28, 2018 •
edited

@grahamc I'll test this in a few hours by doing a nix-build -A stdenv --option substitute false, fixing conflicts after that.

As a side effect of the merging of this PR, a whole lot of @r-ryantm updates will result in merge conflicts heh

@ryantm
Copy link
Member

ryantm commented Jun 28, 2018 •
edited

We can just close all of @r-ryantm's conflicted PRs and re-make them.

@infinisil
Copy link
Member Author

nix-build -A stdenv --option substitute false --check built without problems. Not sure if that actually recursively does the build though, I think it does.

I resolved the conflicts as well now.

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: buildtorrent

Partial log (click to expand)

these paths will be fetched (0.01 MiB download, 0.03 MiB unpacked):
  /nix/store/gafxh5m75by744rbkf70rjxfficrgp51-buildtorrent-0.8
copying path '/nix/store/gafxh5m75by744rbkf70rjxfficrgp51-buildtorrent-0.8' from 'https://cache.nixos.org'...
/nix/store/gafxh5m75by744rbkf70rjxfficrgp51-buildtorrent-0.8

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: buildtorrent

Partial log (click to expand)

/nix/store/zx76nmj0lgzsv4ss8nh3jnj44v0bicb2-buildtorrent-0.8

@xeji
Copy link
Contributor

xeji commented Jun 28, 2018

Let's merge before new conflicts come up, and take it from there.

@xeji xeji merged commit 57bccb3 into NixOS:master Jun 28, 2018
@infinisil
Copy link
Member Author

Alright, nice :D

This was referenced Jun 28, 2018
Closed
Closed
gavinrogers pushed a commit to gavinrogers/nixpkgs that referenced this pull request Jun 28, 2018
@infinisil
treewide: http -> https sources (NixOS#42676)
b11e487 
* treewide: http -> https sources

This updates the source urls of all top-level packages from http to
https where possible.

* buildtorrent: fix url and tab -> spaces
@infinisil infinisil deleted the https branch June 29, 2018 11:39
zimbatm added a commit to zimbatm/nixpkgs that referenced this pull request Jun 30, 2018
@zimbatm
Revert "treewide: http -> https sources (NixOS#42676)"
8701be1 
This reverts commit 57bccb3.
bjornfor added a commit that referenced this pull request Aug 8, 2018
@bjornfor
cgit: 1.1 -> 1.2.1
0689d27 
Fixes CVE-2018-14912.

(cherry picked from commit 970d5ef)

Trivial conflict from commit 57bccb3
("treewide: http -> https sources (#42676)") on master branch.
bjornfor pushed a commit that referenced this pull request Aug 17, 2018
@bjornfor
lighttpd: 1.4.49 -> 1.4.50
7d829c1 
Semi-automatic update generated by
https://github.com/ryantm/nixpkgs-update tools. This update was made
based on information from
https://repology.org/metapackage/lighttpd/versions.

(cherry picked from commit 6e1d3cb)

Fix trivial conflict from commit 57bccb3
("treewide: http -> https sources (#42676)") on master branch.
@infinisil infinisil mentioned this pull request Nov 24, 2018
Closed
@infinisil infinisil mentioned this pull request Apr 22, 2019
Merged
10 tasks
matthewbauer pushed a commit that referenced this pull request May 25, 2019
multipath-tools: fix download URL
3c66a05 
The URL was mistakenly truncated in the following commit:
    57bccb3 treewide: http -> https sources (#42676)
peti pushed a commit that referenced this pull request May 27, 2019
@peti
multipath-tools: fix download URL
c7d0fb7 
The URL was mistakenly truncated in the following commit:
    57bccb3 treewide: http -> https sources (#42676)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@peti peti Awaiting requested review from peti

Assignees
No one assigned
Labels
6.topic: printing 10.rebuild-darwin: 1-10 10.rebuild-darwin-stdenv 10.rebuild-linux: 1-10 10.rebuild-linux-stdenv
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@infinisil @GrahamcOfBorg @xeji @grahamc @ryantm