3

When setting up a lambda written in python, you can include libraries (json, pymysql, pandas, etc.) either using a requirements.txt file, or by setting up a layer. Does anyone know which option works faster at run time?

For example, is it the case that requirements.txt will cause pip install execution (slow to start) whereas a layer "has the library already there" and is therefore faster to launch? (But then again, if that's the case, and you want the most recent version, how does AWS accomplish that? it must run pip install.)

I am aware of other benefits of layers (core reuse), my question is limited to the above.

Thank you!

0

1 Answer 1

4

The information in your question is incorrect. You can't include a requirements.txt file in a Lambda function. AWS Lambda does not look at your rquirements.txt file and do anything with it. AWS Lambda will not download and install dependencies for you just because you bundled a requirements.txt file. AWS Lambda expects all dependencies to be included in your Lambda function's deployment package, or in the attached Lambda layers.

For example, is it the case that requirements.txt will cause pip install execution (slow to start) whereas a layer "has the library already there" and is therefore faster to launch? (But then again, if that's the case, and you want the most recent version, how does AWS accomplish that? it must run pip install.)

AWS Lambda doesn't actually do any of that. So that's not a concern.

Your actual choices are between attaching Lambda layers, or installing all the requirements in your requirements.txt file locally when you are building your Lambda function, and including all those dependencies in the function's deployment zip file.


I am aware of other benefits of layers (core reuse), my question is limited to the above.

Code reuse is the main benefit of layers. Often it can be difficult to include certain dependences in your Python Lambda deployment, if those dependencies include binary executables, which have to be built for AWS Linux to run on AWS Lambda. Pandas is a common Python library where you run into this issue. So building that dependency once as a Layer, and then reusing it, or even pulling in that dependency as a layer that someone else has already built for you, is a common reason to use Lambda layers.

Sign up to request clarification or add additional context in comments.

4 Comments

Thanks Mark, very helpful. My lambda works without any layer, without the libs, only a requirements.txt file. What gives?
I can't tell you what gives without knowing a lot more about how you deploy, what libraries are in the requirements.txt file, etc. If you are using something like the serverless framework or AWS SAM to deploy your app, it is zipping the libs for you during deployment.
Ah! I am using SAM so then that's what happens. For common libs (json, datetime, etc.) is it your recommendation to use this method, or create layers? Thanks again
Libs like json and datetime are built into Python. Those are already available in the AWS Lambda runtime environment without needing to install anything, just like they are available on your local computer as soon as you install Python. AWS also provides the AWS SDK for Python (Boto3) so you don't have to worry about bundling that either. For others I would recommend using your current method and only look into Layers when you hit a dependency that is not easy to include (like Pandas).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.