Python native libraries compatibility with AWS Lambda

Lately, I’ve been digging into Couchbase and noticed so many people troubled with getting Couchbase Python SDK working in AWS Lambda environment. Let me shortly guide you through the process for getting it done for Couchbase, but also for other native libraries.

I’m not going to cover automating code deployment with SAM, CDK or CloudFormation, you can find tons of tutorials for it. I’ll make it simple and just upload working deployment package (.zip) directly to Lambda console.

Installing Couchbase Python SDK

The most important thing to understand is that Couchbase Python SDK is a native library, meaning it’s compiled specifically for a particular operating system. In simple terms, if you install it on MacOS, it’ll be compiled for MacOS and won’t work well in AWS Lambda, which is linux-based – Lambda uses Amazon Linux 2 environment. In such case, you might encounter following error:

[ERROR] Runtime.ImportModuleError: Unable to import module 'index': /var/task/couchbase/pycbc_core.so: invalid ELF header

If you’re on mac and wonder, arch -arm64 pip install couchbase will not help. MacOS uses it’s own implementation of C standard library, which, apparently, results in some incompatibilities in AWS Lambda environment.

In my case, I use ARM64 Lambda architecture. I’ve seen some people looking for compatible Docker images, building dependencies inside, uploading to Lambda. Sure, that works, but why bother?

pip allows to select target architecture and it turns out you can just use a wheel that is compatible with Lambda – manylinux2014_aarch64.
If you’re using x86_64 – use manylinux2014_x86_64.

Now, all you have to do is to navigate to your deployment package folder and run:

pip install --platform manylinux2014_aarch64 --target=. --implementation cp --python-version 3.12 --only-binary=:all: --upgrade couchbase

Once installed, just wrap up your code and installed dependencies into .zip archive and upload to your Lambda.

By the way, this is very well explained in AWS docs: https://docs.aws.amazon.com/lambda/latest/dg/python-package.html#python-package-native-libraries

Of course, this works well with other native libraries, too. No need to spin up docker containers, no need to waste your time researching. If you’re using docker containers for executing pipeline scripts, either make sure to choose compatible one, or run your pip install command like above. That’s it.