Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mariadb: provide multiple src url mirrors #41871

Merged
merged 1 commit into from Jun 24, 2018
Merged

mariadb: provide multiple src url mirrors #41871

merged 1 commit into from Jun 24, 2018

Conversation

bhipple
Copy link
Contributor

@bhipple bhipple commented Jun 11, 2018 •
edited

Both mariadb.com and mariadb.org appear to be canonical upstream sources, and
have identical sha256 hashes at these URLs.

For the third-party mirrors, we should provide multiple for better availability,
and also note that some users may not be able to process FTP traffic due to
network security settings.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

@bhipple
Copy link
Contributor Author

bhipple commented Jun 11, 2018

@GrahamcOfBorg build mariadb mariadb.connector-c

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: mariadb, mariadb.connector-c

Partial log (click to expand)

these paths will be fetched (29.11 MiB download, 220.60 MiB unpacked):
  /nix/store/a0wayl52sqsv7zgsy8nkz4k4qb1w1693-mariadb-10.2.15
  /nix/store/il1c2yp4najjqqxy6sqbcn2y3s79nzw1-judy-1.0.5
copying path '/nix/store/il1c2yp4najjqqxy6sqbcn2y3s79nzw1-judy-1.0.5' from 'https://cache.nixos.org'...
copying path '/nix/store/a0wayl52sqsv7zgsy8nkz4k4qb1w1693-mariadb-10.2.15' from 'https://cache.nixos.org'...
/nix/store/a0wayl52sqsv7zgsy8nkz4k4qb1w1693-mariadb-10.2.15
/nix/store/kj5j2mv6f16a0fx7sdfbbill0slhndsm-mariadb-connector-c-2.3.5

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: mariadb, mariadb.connector-c

Partial log (click to expand)

these paths will be fetched (31.44 MiB download, 225.65 MiB unpacked):
  /nix/store/w5g9y09cnbjny9gfxj00sqrdx5wkwmmy-mariadb-10.2.15
copying path '/nix/store/w5g9y09cnbjny9gfxj00sqrdx5wkwmmy-mariadb-10.2.15' from 'https://cache.nixos.org'...
/nix/store/w5g9y09cnbjny9gfxj00sqrdx5wkwmmy-mariadb-10.2.15
/nix/store/n3s70fxzhpgxwsylxqr9739da6wvfj3q-mariadb-connector-c-2.3.5

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: mariadb

Partial log (click to expand)

/nix/store/w5g9y09cnbjny9gfxj00sqrdx5wkwmmy-mariadb-10.2.15

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: mariadb

Partial log (click to expand)

/nix/store/a0wayl52sqsv7zgsy8nkz4k4qb1w1693-mariadb-10.2.15

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: mariadb

Partial log (click to expand)

/nix/store/w5g9y09cnbjny9gfxj00sqrdx5wkwmmy-mariadb-10.2.15

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: mariadb

Partial log (click to expand)

/nix/store/a0wayl52sqsv7zgsy8nkz4k4qb1w1693-mariadb-10.2.15

@Mic92
Copy link
Member

Mic92 commented Jun 12, 2018

Are the mirrors particular unstable that we need them?

@xeji
Copy link
Contributor

xeji commented Jun 12, 2018

I don't see why this is necessary for a single package unless the standard download site fails really often. After all the source is only downloaded once and then cached until the hash changes...

Also there's a trust issue with non-official mirrors because of the "trust on first use" process most people use for updating source hashes.

@bhipple
Copy link
Contributor Author

bhipple commented Jun 12, 2018 •
edited

I had a few hours where I wasn't able to access mariadb.org, but could access mariadb.com -- they both appear accessible now, though.

On the connector-c one, I happen to have some machines on a corporate network that doesn't allow FTP traffic from untrusted sites, because the connect-back port gets blocked. The mirrors appear to be stable, but if we want to just have 1 of them it'd be nice to pick an HTTPS mirror instead of FTP/SSH. Same issue as in #41761 and #41774

I was using an overlay for a bit that used the digitalocean mirror (https://downloads.mariadb.org/interstitial/connector-c-2.3.6/mariadb-connector-c-2.3.6-src.tar.gz/from/http%3A//mirror.nodesdirect.com/mariadb/?serve), but I noticed it was down several times, which is why I added 3 of them. It's a little awkward because of the long, version-specific prefix, so it doesn't fit well into mirrors.nix unfortunately.

FWIW while these are third-party mirrors, they're listed on the official download site:
https://downloads.mariadb.org/connector-c/2.3.6/

@Mic92
Copy link
Member

Mic92 commented Jun 14, 2018 •
edited

Having both mariadb.org/mariadb.com should be fine. For the other mirrors switching from ftp to https sounds also like a good idea.

@bhipple
Copy link
Contributor Author

bhipple commented Jun 18, 2018

Ok, should we merge this as-is, or would you like me to delete some of the connector-c mirrors and just leave 1 HTTPS one?

@bhipple
mariadb: provide multiple src url mirrors
3bb3055 
Both mariadb.com and mariadb.org appear to be canonical upstream sources, and
have identical sha256 hashes at these URLs.

For the third-party mirrors, if we are just going to provide one URL, it should
be an HTTPS url instead of FTP or SSH, which many users may not be able to
access due to network security configurations.
@bhipple
Copy link
Contributor Author

bhipple commented Jun 24, 2018 •
edited

Ok, I left the mariadb.com and mariadb.org mirrors, but for connector-c I've swapped it to rely on just one HTTPS mirror from the list of official mirrors. Also rebased to fix merge conflicts with a connector-c update.

@bhipple
Copy link
Contributor Author

bhipple commented Jun 24, 2018

@GrahamcOfBorg build mariadb mariadb.connector-c

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: mariadb, mariadb.connector-c

Partial log (click to expand)

these paths will be fetched (31.58 MiB download, 226.27 MiB unpacked):
  /nix/store/ms6af062lws30lq4p1ix40d9nq346040-mariadb-10.2.15
  /nix/store/ychaas7qmvk71xxms1pp69a44fbmhcgi-judy-1.0.5
copying path '/nix/store/ychaas7qmvk71xxms1pp69a44fbmhcgi-judy-1.0.5' from 'https://cache.nixos.org'...
copying path '/nix/store/ms6af062lws30lq4p1ix40d9nq346040-mariadb-10.2.15' from 'https://cache.nixos.org'...
/nix/store/ms6af062lws30lq4p1ix40d9nq346040-mariadb-10.2.15
/nix/store/ky9zcgf91j4i50z7l8myg4l1l4wzabbm-mariadb-connector-c-2.3.6

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: mariadb, mariadb.connector-c

Partial log (click to expand)

these paths will be fetched (29.09 MiB download, 220.60 MiB unpacked):
  /nix/store/32ks8vmhfml55nf435hd48pm9158a3cs-judy-1.0.5
  /nix/store/zj35a34b9xq2223zvalzydnjx1x5rkqc-mariadb-10.2.15
copying path '/nix/store/32ks8vmhfml55nf435hd48pm9158a3cs-judy-1.0.5' from 'https://cache.nixos.org'...
copying path '/nix/store/zj35a34b9xq2223zvalzydnjx1x5rkqc-mariadb-10.2.15' from 'https://cache.nixos.org'...
/nix/store/zj35a34b9xq2223zvalzydnjx1x5rkqc-mariadb-10.2.15
/nix/store/4vv9kk7358maklx6dckvbgh2k6i2bkb6-mariadb-connector-c-2.3.6

@bhipple
Copy link
Contributor Author

bhipple commented Jun 24, 2018 •
edited

After all the source is only downloaded once and then cached until the hash changes...

@xeji this is true for anyone using the hydra binary cache, but it's not true for users who aren't -- either because they have a non-root installation in a non-standard store path, or they prefer to build from source to avoid trusting third-party binaries. For these users, fragile source urls are somewhat of an annoyance, because they more frequently rely on them.

It looks like the content-addressed tarballs.nixos.org mirror doesn't automatically mirror source tarballs as they're added, since they're often missing. I see some maintainers perl scripts for mirroring, so I assume it's a manual process?

At any rate, when upstream gives us N equivalent mirror sites, it's somewhat nice to allow swapping between them, as we do in mirrors.nix. If nothing else, preferring an HTTPS one over FTP when possible is nice :)

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: mariadb

Partial log (click to expand)

these paths will be fetched (31.58 MiB download, 226.27 MiB unpacked):
  /nix/store/ms6af062lws30lq4p1ix40d9nq346040-mariadb-10.2.15
  /nix/store/ychaas7qmvk71xxms1pp69a44fbmhcgi-judy-1.0.5
copying path '/nix/store/ychaas7qmvk71xxms1pp69a44fbmhcgi-judy-1.0.5' from 'https://cache.nixos.org'...
copying path '/nix/store/ms6af062lws30lq4p1ix40d9nq346040-mariadb-10.2.15' from 'https://cache.nixos.org'...
/nix/store/ms6af062lws30lq4p1ix40d9nq346040-mariadb-10.2.15

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: mariadb

Partial log (click to expand)

/nix/store/zj35a34b9xq2223zvalzydnjx1x5rkqc-mariadb-10.2.15

@xeji
Copy link
Contributor

xeji commented Jun 24, 2018

It looks like the content-addressed tarballs.nixos.org mirror doesn't automatically mirror source tarballs as they're added, since they're often missing. I see some maintainers perl scripts for mirroring, so I assume it's a manual process?

Sorry, I don't know.
Anyway, LGTM now.

@xeji xeji merged commit 5e451e4 into NixOS:master Jun 24, 2018
@bhipple
Copy link
Contributor Author

bhipple commented Jun 24, 2018

Thanks @xeji!

@bhipple bhipple deleted the fix/mariadb-src-url branch July 21, 2018 17:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
10.rebuild-darwin: 0 10.rebuild-linux: 0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@bhipple @GrahamcOfBorg @Mic92 @xeji