torchlambda

torchlambda is a tool to deploy PyTorch models on Amazon's AWS Lambda using AWS SDK for C++ and custom C++ runtime.

Using statically compiled dependencies whole package is shrunk to only 30MB.

Due to small size of compiled source code users can pass their models as AWS Lambda layers. Services like Amazon S3 are no longer necessary to load your model.

torchlambda has it's PyTorch & AWS dependencies always tested & up to date because of daily continuous deployment runs.

Docs	Deployment	Package	Python	PyTorch	Docker	CodeBeat	Images

:heavy_check_mark: Why should I use torchlambda?

• Lightweight & latest dependencies - compiled source code weights only 30MB. Previous approach to PyTorch network deployment on AWS Lambda (fastai) uses outdated PyTorch (1.1.0) as dependency layer and requires AWS S3 to host your model. Now you can only use AWS Lambda and host your model as layer and PyTorch master and latest stable release are supported on a daily basis.
• Cheaper and less resource hungry - available solutions run server hosting incoming requests all the time. AWS Lambda (and torchlambda) runs only when the request comes.
• Easy automated scaling usually autoscaling is done with Kubernetes or similar tools (see KubeFlow). This approach requires knowledge of another tool, setting up appropriate services (e.g. Amazon EKS). In AWS Lambda case you just push your neural network inference code and you are done.
• Easy to use - no need to learn new tool. torchlambda has at most 4 commands and deployment is done via YAML settings. No need to modify your PyTorch code.
• Do one thing and do it well - most deployment tools are complex solutions including multiple frameworks and multiple services. torchlambda focuses solely on inference of PyTorch models on AWS Lambda.
• Write programs to work together - This tool does not repeat PyTorch & AWS's functionalities (like aws-cli). You can also use your favorite third party tools (say saws, Terraform with AWS and MLFlow, PyTorch-Lightning to train your model).
• Test locally, run in the cloud - torchlambda uses Amazon Linux 2 Docker images under the hood & allows you to use lambci/docker-lambda to test your deployment on localhost before pushing deployment to the cloud (see Test Lambda deployment locally tutorial).
• Extensible when you need it - All you usually need are a few lines of YAML settings, but if you wish to fine-tune your deployment you can use torchlambda build --flags (changing various properties of PyTorch and AWS dependencies themselves). You can also write your own C++ deployment code (generate template via torchlambda template command).
• Small is beautiful - 3000 LOC (most being convenience wrapper creating this tool) make it easy to jump into source code and check what's going on under the hood.

:house: Table Of Contents

• Installation
• Tutorials
  • ResNet18 deployment on AWS Lambda
  • Test Lambda deployment locally
  • base64 image encoding
• Commands
  • settings
  • template
  • build
  • layer
• YAML settings file reference
• C++ code

:page_with_curl: Benchmarks

Benchmarks can be seen in BENCHMARKS.md file and are comprised of around ~30000 test cases.

Results are divided based on settings used, model type, payload, AWS Lambda timing etc. Below is an example of how inference performance changes due to higher resolution images and type of encoding:

	128x128	256x256	512x512	1024x1024
base64	120.622	165.184	311.129	995.249
byte	133.315	203.628	498.391	1738.97
char	128.331	209.306	517.482	1822.56
short	135.859	207.389	497.818	1740.91
int	133.42	216.163	519.502	1783.02
long	126.979	228.497	516.98	1760.93
float	135.825	223.045	515.245	1802.25
double	137.281	209.267	536.959	1811.83

Clearly the bigger image, the more important it is to use base64 encoding. For all results and description click here.

:question: Contributing

If you find an issue or you think some functionality may be useful to you, please open new Issue or create Pull Request with your changes, thanks!