Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3: Ensure connection to the well-defined region #14

Closed
wants to merge 1 commit into from

Conversation

RaitoBezarius
Copy link
Member

Currently, if you try to create a S3 Bucket out of a "default region" (US/EU), you'll get an error because self.region is equal to None during creation, we should use the definition rather and store defn.region in self.region, this PR fixes the behavior.

I am open to all pointers to make this solution more NixOps-y.

@AmineChikhaoui
Copy link
Member

if you try to create a S3 Bucket out of a "default region" `

I'm not sure what does this mean exactly ? What would be the value of region ?

@RaitoBezarius
Copy link
Member Author

if you try to create a S3 Bucket out of a "default region" `

I'm not sure what does this mean exactly ? What would be the value of region ?

When you don't specify a region, AWS for compat reasons will suppose you're in us-east-1 AFAIK, if region = "US", it'll set also us-east-1, if you set region = "EU", it'll set eu-west-1 IIRC.

Anyway, for backward compatibility, AWS will gladly take no region and give you the "default" regional S3, which is in most of the case us-east-1.

So when you go out of the default region, it will break because you cannot create from a regional S3 an S3 Bucket outside of the specified region which what the current code does:

  • connect opens a session in the self.region region
  • create uses the session to create a S3 Bucket in the defn.region region.

@AmineChikhaoui
Copy link
Member

I still don't see how the issue happens exactly. region is required in the configuration of S3 (there is no default value) so if you create a nixops expression, you would need to set the value for region first.

Maybe to make it easier, what would be the steps to reproduce the issue you're describing ?

@RaitoBezarius
Copy link
Member Author

I still don't see how the issue happens exactly. region is required in the configuration of S3 (there is no default value) so if you create a nixops expression, you would need to set the value for region first.

Maybe to make it easier, what would be the steps to reproduce the issue you're describing ?

Try to create an S3 Bucket in the region us-west-1 for example, AFAIK, it'll fail, you don't need to create anything else.

Sure, defn.region is always set, but self.region is not. The bug is that we connect using self.region rather than defn.region or that self.region is not properly populated.

I'm not used to how self.*** vars are populated, what is their lifecycle, when is the first method called, when is the first time that they receive a defn object, etc.

So my easy fix is just to always use defn as an only source of truth which makes sense to me.

Comment on lines 197 to +204
if region == "eu-west-1": return "EU"
elif region == "us-east-1": return "US"
else: return region

def region_name_to_region(region):
if region == "EU": return "eu-west-1"
elif region == "US": return "us-east-1"
else: return region
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if possible we should get rid of all these US/us-east-1, EU/eu-west-1 default stuff and always specify the region in the boto3 request since it is already required. cc @AmineChikhaoui

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@PsyanticY I can do it in this PR if you want.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AmineChikhaoui should decide that ^^

@AmineChikhaoui
Copy link
Member

Try to create an S3 Bucket in the region us-west-1 for example, AFAIK, it'll fail, you don't need to create anything else.

I was able to create a bucket in us-west-1 without any issue.

@RaitoBezarius
Copy link
Member Author

Try to create an S3 Bucket in the region us-west-1 for example, AFAIK, it'll fail, you don't need to create anything else.

I was able to create a bucket in us-west-1 without any issue.

Okay, I'll come up with a repro case then.

@NorfairKing
Copy link

NorfairKing commented Jan 6, 2020

@RaitoBezarius @AmineChikhaoui While you're looking into this, please look at the related https://github.com/NixOS/nixops/issues/1215

@NorfairKing
Copy link

It turns out that this fixes NixOS/nixops#1215. Please merge it !

@RaitoBezarius
Copy link
Member Author

@NorfairKing I didn't have any time to reproduce the bug, do you have some repro case at the hand?

@NorfairKing
Copy link

NorfairKing commented Jan 22, 2020

Here you go:

resources.s3Buckets.myBucket = {
        name = "mySuperUniqueBucketNameaeusnthaouenthu";
        versioning = "Suspended"; # Probably not even necessary.
        region = "eu-west-1";
        inherit accessKeyId;
};

@NorfairKing
Copy link

@RaitoBezarius I'd like to get this merged.

@RaitoBezarius
Copy link
Member Author

I'd love to merge it, @NorfairKing — but @AmineChikhaoui would need to reproduce it.

@tewfik-ghariani
Copy link
Contributor

For the example you've mentioned @NorfairKing , the issue was related to the bucket name containing upper case characters ( mySuperUniqueBucketNameaeusnthaouenthu )

Bucket names can consist only of lowercase letters,

Ref : https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html#bucketnamingrules

For the other example referenced by #88 , the bucket backup is definitely not unique and it must have been created in a different region than eu-west-1 as explained there

@NorfairKing
Copy link

@tewfik-ghariani So it looks like my examples weren't good.
Do you mean to say that you cannot reproduce my problem?

@tewfik-ghariani
Copy link
Contributor

@NorfairKing I mean that I noticed this could potentially be a false problem. Could you give it one more try while abiding to the bucket naming requirements?

@NorfairKing
Copy link

NorfairKing commented May 10, 2020

@tewfik-ghariani Not really, I aleady started using the fork and that fixes my problem, so I definitely use a sensible bucket name right now. I'd prefer not to revert my setup and create additional buckets.
I just don't remember whether not using the fork still went wrong with a good bucket name. (It's been almost a year.)

@NorfairKing
Copy link

Could we please merge this at some point? @RaitoBezarius

@RaitoBezarius
Copy link
Member Author

I will close this PR as I believe it has too much diverged from the source tree ; if the bug still remains, feel free to start a new PR based on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants