New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
google cloud storage support #3021
Conversation
@volth That looks like a pretty massive dependency (and it pulls in Java). Probably not something we want to depend on in Nix. As an aside we probably should move S3/GCS support into separate plugins to reduce Nix's closure size. |
|
||
namespace nix { | ||
|
||
class GCSBinaryCacheStore : public BinaryCacheStore |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This class and header file don't appear to be used anywhere except as a base for GCSBinaryCacheStoreImpl
, so you could just get rid of them.
@andir Can you make a PR to add |
Yes, will do during the day. I also have a few WIP changes locally that
handle large (slow) file uploads. I'll push them as well.
…On Mon, 12 Aug 2019, 12:16 Eelco Dolstra, ***@***.***> wrote:
@andir <https://github.com/andir> Can you make a PR to add
google-cloud-cpp to Nixpkgs?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3021?email_source=notifications&email_token=AAE365HUQPPK2EX24NVANU3QEEZ6PA5CNFSM4IIXKKL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4CDHFY#issuecomment-520369047>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAE365BYDHA2SO3RFHUVILDQEEZ6PANCNFSM4IIXKKLQ>
.
|
@edolstra I added those remarks, rebased onto master and added a bit of rough documentation. Let me know if there is anything else that has to be done. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some comments.
We probably also want to add a doc/manual/packages/gcs-substituter.xml
, and a line to doc/manual/release-notes/…
.
{ | ||
PathSet paths; | ||
|
||
// FIXME: is this really needed for binary caches? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should probably be replied before merging.
Really excited to see this! I'll give the current code a test run on some GCS-based bucket later this week. |
<para>Nix has built-in support for storing and fetching store paths | ||
from Google Cloud Storage.</para> | ||
<para>For a GCS bucket with the name <literal>example-nix-cache</literal> the URI is <uri>gs://example-nix-cache</uri>.</para> | ||
<para>Nix will use the environment variable <literal>GOOGLE_APPLICATION_CREDENTIALS</literal> to discover the credentials JSON file.</para> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should probably write we do authentication as documented at https://cloud.google.com/storage/docs/authentication#libauth - which also includes automatic credential management in Google App Engine and Google Cloud engine via their metadata server.
GOOGLE_APPLICATION_CREDENTIALS
is another way to do it, but this reads as if it's the only way to do it. I'd still keep the example with GOOGLE_APPLICATION_CREDENTIALS
however.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
-
nix copy --to gs://bucketname
is working fine -
nix-build --substituters gs://bucketname
isn't supported
Is that a requirement? The way that is currently implement for S3 looks like an ugly hack. |
I don't think it's a requirement. I was assuming that registering the store would make it work everywhere so I was surprised when it didn't work. Looking at the S3 code I see that it's not that clean. |
I gave this a quick run on a CI pipeline, using the
The derivations are 1.8MiB and 700KiB in size, and that's running from a GCP instance, so I highly doubt too slow transfer speeds. However, I'm not sure what else can be the issue here… |
I noted a lot of CPU usage during However, after some tracing, this seems to be due to extensive polling inside google-cloud-sdk-cpp while uploading to GCS: I did a snapshot backtrace while
|
With #2733 fixed by bumping to curl 7.68.0, this might be worth another try. |
I propose closing this - this has bitrotten a bit, isn't actively being worked on, and @edolstra mentioned both the S3 and GCS support should eventually be moved into separate plugins. |
Yes, though I think it would make sense to keep them in the |
@edolstra can this be closed? |
This was only to enable NixOS/nix#3021, which has bitrotted sufficiently in the meantime.
I started adding support for Google Cloud Storage (GCS) to Nix.
This is very similar to the AWS S3 support. Unlike AWS there is no default path for the credentials. The environment variable
GOOGLE_APPLICATION_CREDENTIALS
must point to a standard Google credentials JSON file. The store URI must have the formatgs://bucketName
.I have not thoroughly tested everything just yet. It seems to work fine so far. I uploaded a bunch of larger system closures (11-20GB) and some pandoc closures without problems.
Initially I did run into timeout issues but eventually switched to uploading larger files in chunks (37297df ).
This work depends on yet to be merged
nixpkgs
changes that introduce the required dependencies. (https://github.com/andir/nixpkgs/tree/google-cloud-cpp).I will continue work on this during the coming days and welcome early feedback.
This change is