New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BREAKING - Replace IamPolicyLambdaExecution with inline policies and added ManagedPolicyArns to fix VPC permissions #2983
Conversation
Rockin'! This has been needed for a very long while! |
if (!this.serverless.service.getAllFunctions().length) { | ||
return BbPromise.resolve(); | ||
} | ||
|
||
let anyFunctionHasNoRole = false; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pmuens @eahefnawy @mthenw I updated this logic as it seemed to be broken, or my brain didn't compute 😊
@pmuens @eahefnawy @mthenw I added the |
@serverless/team Successfully tested all stack create/update cases against actual stacks. This PR is now 100% ready for review 👍 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just did a first dive into the code. Really, really great work! Love the refactoring and the inline policies!
Tested it with a simple setup and it works as nothing has changed.
Even ran the complex-integration-test-suite
against it and it succeeded.
However this PR needs far more attention / deep dives with different configurations as it touches mission critical parts.
I'll test it more thoroughly and encourage everyone who reads this to do as well.
Great work @nicka 🥇
@pmuens Any news on this? |
@nicka thanks for getting back. This PR is a great move towards v2 so we'll focus on this one quite soon! 👍 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just gave it another spin today. Deployed a simple service, a more complex service (with multiple event sources) and ran the complex-integration-test-suite
again. Couldn't break it 👍
Does anyone know a very complex, yet easy to deploy open source Serverless project / service we can use to test it against this?
I'll look through the code again tomorrow...
Great job @nicka 💯 👍
Hey @nicka we finalized the decision to include this PR in v1.8! Could you maybe look at the conflicts and resolve them 😬 . That would be super awesome 💯 Sorry for any inconveniences. Let us know if you need any help and thanks in advance! 👍 |
@pmuens Sure thing, I'll try my best to finish this by the end of this week. |
I'm working behind a company firewall and when I run serverless deploy -v I'm getting WARNING: You are running v1.7.0. v1.8.0 will include the following breaking changes:
- Will replace IamPolicyLambdaExecution resource with inline policies -> https://git.io/vDilm
- "sls info" will output the short function name rather than the lambda name -> https://git.io/vDiWx
You can opt-out from these warnings by setting the "SLS_IGNORE_WARNING=*" environment variable. I just installed serverless npm module, why am I getting a old module to begin with? |
You are not getting an old version. This is a warning that the next version will have breaking changes to give you time to adjust. |
…polices within the IamRoleLambdaExecution resource
…t.js Improved tests for lib/plugins/aws/deploy/compile/functions/index.test.js wip
@pmuens @mthenw @erikerikson Rebased on top of master (the merge conflicts where hell), def needs some love and testing before merging! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@nicka testing this now...I requested one small but essential change.
Thanks a lot! and sorry for the delay! 🙌
[this.provider.naming.getRoleLogicalId()]: iamRoleLambdaExecutionTemplate, | ||
} | ||
); | ||
|
||
this.serverless.service.getAllFunctions().forEach((functionName) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This code block must be at the very beginning of the merge()
method, otherwise bug #3174 will resurface. In other words, now that we always use log group resources, they must be merged regardless of whether we're using custom or default roles.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure if I completely follow because of all the prior checks:
// resolve early if no functions are provided
// resolve early if provider level role is provided
// resolve early if all functions contain a custom role
If we're talking purely about Log groups and assuming the user is providing his own IAM statement the create logs code:
this.serverless.service.getAllFunctions().forEach((functionName) => {
const functionObject = this.serverless.service.getFunction(functionName);
const logGroupLogicalId = this.provider.naming
.getLogGroupLogicalId(functionName);
const newLogGroup = {
[logGroupLogicalId]: {
Type: 'AWS::Logs::LogGroup',
Properties: {
LogGroupName: this.provider.naming.getLogGroupName(functionObject.name),
},
},
};
_.merge(this.serverless.service.provider.compiledCloudFormationTemplate.Resources,
newLogGroup);
});
might come between:
// resolve early if no functions are provided
// HEREEEEEE
// resolve early if provider level role is provided
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@nicka yep exactly ... what we're trying to avoid is to resolving early before getting the chance to merge the LogGroup, which must be merged regardless of the what role we're using.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, I'll update the flow 👍
] | ||
} | ||
} | ||
] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is probably taken from the legacy policy json file, but do you see any reason why we have two seperate statements? we can just have 1 statement for all 3 permissions (CreateLogGroup
, CreateLogStream
& PutLogEvents
), since after all they're all affecting the same resources/functions. Actually, I don't think we need the logs:CreateLogGroup
permission anymore since we're now explicitly creating it via CF.
This has probably should've been done earlier and not really critical, but do you see any downside to that? if not let's make those changes 😊
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes total sense to update, especially if were ALWAYS creating the log groups. Will only keep CreateLogStream
and PutLogEvents
then.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might still need two because of /*
within IAM. What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
aaah right I think you're right! 😊
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for the delay, switched MacBook's etc. Will work on the updates.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@eahefnawy using two was important because the use of star would give creation rights to any log group with the same prefix. but as you note, it is legacy that can be removed now that explicit log group creation happens.
There's a bit more paranoia not in there though... Here is our latest:
- Effect: Allow
Action:
- logs:CreateLogStream
Resource:
- Fn::GetAtt:
- TheLogGroup
- Arn
- Effect: Allow
Action:
- logs:PutLogEvents
Resource:
- Fn::Join:
- ":"
- - Fn::GetAtt:
- TheLogGroup
- Arn
- "*"
This way, the lambda can't create log streams in any similarly-prefixed-but-not-the-log-group-that-has-been-created while still being able to create in the created log group and then write to any such log stream.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
small surface space but since we're choosing this for all users and we can do it once and be done... seems worth the energy.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and... pretty sure it's my fault it's not already correct. sorry for that
@nicka added another comment. Sorry for the hassle 🙌 |
…p resources are created through CloudFormation by default
@eahefnawy Updated! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @nicka ... G2M 🙌
My stacks aren't complex, but I confirm this fixes the problem for me for my projects that functions in a VPC during initial creation! +1 here, good job, can't wait for it in next release. Cheers! |
Woehoe!!! |
What did you implement:
Closes #1786
Closes #2780
Closes #2830
How did you implement it:
IamPolicyLambdaExecution
resource statements towards inline polices within theIamRoleLambdaExecution
resource to fix CloudFormation dependency issues. As a bonues, we save an extra CloudFormation resource.ManagedPolicyArns
property toIamRoleLambdaExecution
resource when VPC config is provided.IamRoleLambdaExecution
resource.NOTE: This might be a breaking change since users could be referecing to the
IamPolicyLambdaExecution
policy which this PR removes.Research
Adding VPC config to an existing stack still occasionally fails with the ec2 permissions issue. As a solution to this I will try to work with ManagedPolicyArns. Each account should come with a default
arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
by AWS which could be used within theManagedPolicyArns
property.Update 1:
The
ManagedPolicyArns
with the defaultAWSLambdaVPCAccessExecutionRole
works!!!Update 2:
ManagedPolicyArns
has been added as a default when vpc config is provided.How can we verify
Todos:
Is this ready for review?: YES