CDK

Cross-region pull for a lambda function

Graham HayLeave a comment

Having got x-acct pull working for my lambda function, in staging, I had foolishly assumed that running the same CDK script for prod would be easy (as that is the same account where the ECR lives).

Instead the build was failing, with a confusing message:

14:52:31  MyStack |  4/11 | 1:52:29 PM | CREATE_FAILED        | AWS::Lambda::Function                       | MyLambda (MyLambda...) Resource handler returned message: "Source image ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/REPO:latest is not valid. Provide a valid source image. (Service: Lambda, Status Code: 400, Request ID: ...) (SDK Attempt Count: 1)" (RequestToken: ..., HandlerErrorCode: InvalidRequest)

Obviously, the ECR uri is valid, or it wouldn’t be working in the other account. I assumed it was permissions related, but the permissions I had added for x-acct seemed to be a superset of the permissions necessary within the same account.

When I tried to create the lambda in the console, a slightly more useful error was returned. It seems that Lambda is unable to pull from ECR in another region (even though Fargate has no trouble). The easiest solution to this, is to enable replication.

You can do this in the cloudformation to create the repo:

Resources:
    RepositoryReplicationConfig:
        Type: AWS::ECR::ReplicationConfiguration
        Properties:
            ReplicationConfiguration: 
                Rules:
                    - Destinations:
                        - Region: ...
                          RegistryId: ... (account id)

Cross-account pull for a Lambda function

Graham HayLeave a comment

I have been trying to set up a Lambda function, using the CDK; that uses a docker image, from a different account (because reasons). It felt like I was stuck in a chicken & egg situation, where the IAM role to be used was created by the stack, which then failed (because it couldn’t pull the image) and rolled back; deleting the role.

I tried using a wildcard, for the principal:

                Statement:
                -
                    Sid: AllowCrossAccountPull
                    Effect: Allow
                    Principal:
                        AWS: "arn:aws:iam::$ACCOUNT_ID:role/$STACK-LambdaServiceRole*"

but that was rejected:

Resource handler returned message: "Invalid parameter at 'PolicyText' failed to satisfy constraint: 'Principal not found'

After some digging around in the CDK source code, I was able to create the role first; and set up the cross account permissions before creating the lambda. But I was still getting the same error:

Resource handler returned message: "Lambda does not have permission to access the ECR image. Check the ECR permissions. (Service: Lambda, Status Code: 403, ...

At this point, I did what I should have done originally, and actually read the docs. It turns out that like Fargate tasks use a separate role to start the task, and execute it, so does Lambda. But in this case, one role is played by the service itself.

After a bit more flopping around, I finally had something that worked 🥳

                Statement:
                -
                    Sid: CrossAccountPermission
                    Effect: Allow
                    Principal:
                        AWS: "arn:aws:iam::$ACCOUNT_ID:root"
                    Action:
                        - "ecr:BatchGetImage"
                        - "ecr:GetDownloadUrlForLayer"
                -
                    Sid: LambdaECRImageCrossAccountRetrievalPolicy
                    Effect: Allow
                    Principal:
                        Service: "lambda.amazonaws.com"
                    Action:
                        - "ecr:BatchGetImage"
                        - "ecr:GetDownloadUrlForLayer"