Summary of Issue:
We are using servereless frame work with runtime python3.7 we are using the plugin servereless-python-requirements. We are creating local python modules to have helper classes. These modules are imported and used by lambda functions and used by other local python module methods that are not lambda functions. We are doing this for testability and cleanliness. These local python modules when called by a lambda function works only when the serverelss.yml is configured with,‘module’ and ‘include’(see .yml below). Our issues (see error message below) arrises when we have a lambda function call a local python module function, that is not a lambda function, and that python module function tries to import a local python module outside of its own folder. The error you will view below occurs when the lambda function ‘marketoOrgCreate’ calls a non-lambda method in the connection.py folder which is trying to import a local class payload.
What I have included in this request for help
Below you will find the error message, a sample of the folder structure of the project, a sample of the serverless.yml, and samples of the folders with local imports. Thank you for your thoughts and direction in advance.
Error:
$ sls invoke -f marketoOrgCreate -l
"errorMessage": "attempted relative import beyond top-level package",
"errorType": "ValueError",
"stackTrace": [ ... <lots of read out> ....
" File \"/var/task/marketo/connection.py\", line 11, in <module>\n from ..resource.payload
import Payload\n"
Folder Structure Sample:
publication-service-2
publication-service
__init__.py
serverless.yml
requirements.txt
lambdaFunc
organization.py
resource
__init__.py
payload.py
requirements.txt
marketo
__init__.py
connection.py
requirements.txt
servereless.yml (with private stuff removed)
service: publication-service
app: publication-service-app
provider:
name: aws
runtime: python3.7
profile: serverless-admin
region: us-east-2
environment:
KMS: <some long kms>
iamRoleStatements:
- Effect: 'Allow'
Action:
- 'ssm:GetParameters'
- 'ssm:GetParameter'
- 'ssm:PutParameter'
- 'ssm:DeleteParameters'
Resource: <some long arn for our ssm>
- Effect: 'Allow'
Action:
- 'kms:Encrypt'
- 'kms:Decrypt'
Resource: <some long arn for our kms>
- Effect: Allow
Action: lambda:InvokeFunction
Resource: "*"
plugins:
- serverless-python-requirements
- serverless-offline-python
custom:
pythonRequirements:
dockerizePip: true
package:
individually: true
exclude:
- .circleci/**
- .pytest_cache/**
- .coverage
functions:
marketoOrgCreate:
handler: organization.create
module: publication-service/lambdaFunc
include:
- publication-service/lambdaFunc/organization.py
Sample organization.py import that does work I believe because organization holds the lambda function
from resource.payload import Payload
Sample import statement that is causing error message from
publication-service/marketo/connection.py:
#python AWS SDK library
import boto3
#local
from ..resource.payload import Payload
I found the solution:
I was missing a keyword ‘package’ above the ‘include’ statement in servereless.yml and that enables me to include the local modules. Below you will see the updated servereless.yml and the updated import statement in connection.py. These are the only changes needed. Note that the package include statement can be put at the function level or at the package level depending on how many of your lambda functions should have access to the included local python modules. I add my change at the function level.
servereless.yml
service: publication-service
app: publication-service-app
provider:
name: aws
runtime: python3.7
profile: serverless-admin
region: us-east-2
environment:
KMS: <some long kms>
iamRoleStatements:
- Effect: 'Allow'
Action:
- 'ssm:GetParameters'
- 'ssm:GetParameter'
- 'ssm:PutParameter'
- 'ssm:DeleteParameters'
Resource: <some long arn for our ssm>
- Effect: 'Allow'
Action:
- 'kms:Encrypt'
- 'kms:Decrypt'
Resource: <some long arn for our kms>
- Effect: Allow
Action: lambda:InvokeFunction
Resource: "*"
plugins:
- serverless-python-requirements
- serverless-offline-python
custom:
pythonRequirements:
dockerizePip: true
package:
individually: true
exclude:
- .circleci/**
- .pytest_cache/**
- .coverage
functions:
marketoOrgCreate:
handler: organization.create
module: publication-service/lambdaFunc
package:
include:
- publication-service/lambdaFunc/organization.py
- publication-service/lambdaFunc/resource/payload.py
connection.py
from marketo.resource.payload import Payload
Does it work with local invoke? I designed my packages similarly to yours, however when I do local invoke doesn’t find the packages. Did you test it with the --docker option? If yes how do you pass custom events?
Thanks.