.NET Core 3.1 on AWS Lambda
This is going to be the first entry in a series of (probably) four blog posts regarding the support of .NET Core 3.1 as an AWS Lambda runtime that was announced back in November 2019 during AWS re:Invent 2019 and hit the shelves this past March.
Motivation
If you are developing and maintaining serverless .NET Core applications on AWS Lambda then this release is important to you for two reasons:
- It includes several new (introduced with .NET Core 3) opportunities for performance improvements; specifically targeted at improving cold load times,
- .NET Core 3.1 is a Long-Term Support (LTS) release by Microsoft, which means your functions will run without any framework-related hiccups or AWS deprecating Lambda's support of the runtime till 2023, 3 years from now.
This blog series will focus on the first point above as well as some of the obstacles encountered (and how to overcome them) while upgrading our own serverless applications from .NET Core 2.1 to 3.1 in AWS Lambda these past couple of months.
Before I jump into specifics though, there's something y'all need to know first...
Upgraded Lambda instance container
The AWS Lambda team recently adopted a new Amazon Linux image, citing improvements in lambda invocation performance, for all their containers going forward; all the latest runtimes (.NET Core 3.1 among them) run on this new Linux distro, promptly called Amazon Linux 2 while previous versions of runtimes that are still supported (think .NET Core 2.1 for instance) are still running on Amazon Linux. Further information on AWS Lambda runtimes can be found on the relevant documentation.
Great, right? Everyone loves shaving a few extra milliseconds off cold load invocations. The rub is that the flavor of this new Linux distro that the AWS Lambda team adopted, is a seriously slimmed down version of the previous generation. It lacks almost all of the networking and utility libraries commonly found in *nix distros (we found this out the hard way as I will explain in this post).
Now if the dependencies of your .NET Core 3.1 project all adhere to .NET Standard 2.1, you have nothing to worry about. However, not all of us are in this envious position. For example, I've recently had to upgrade an app that had a networking library dependency stuck on .NET Standard 1.3 and that's where it gets interesting.
Replacing said dependency would be a process with an uncertain outcome as it is business critical and there are neither alternatives available nor plans by the publisher for further active development on it. It should also be noted that we had no real idea of what we were about to get ourselves into, because we weren't aware of the change in the underlying Operating System that Lambda was running at since this was our first attempt at upgrading one of our existing apps deployed there from 2.1 to 3.1.
You don't know what you don't know. - the Socratic paradox
In retrospective, it might've been possible to get our hands dirty in-house and upgrade said library, which is OSS with an MIT license, to .NET Standard 2.1 had we known the troubles we'll have with getting this to run on Amazon Linux 2.
This library, even though compiled for .NET Standard 1.3 (not .NET Standard 2.0 as it theoretically should've been to be compatible with .NET Core 2.x) was working as intended as part of this .NET Core 2.1 lambda serverless application running on Amazon Linux.
The serverless flavor of Amazon Linux 2 though, ships without any of the native *nix libraries that this library requires. We upgraded all the dependencies as well as followed through on the other optimizations that I will be going over on the next instalments of this blog series and deployed the app on our staging AWS Lambda environment. The CloudWatch log group of this app immediately started throwing an exception regarding missing native libraries and the stack trace made it clear who the culprit was.
To get to the bottom of this issue, I had to (for the first time ever even though we have 30+ functions/apps running on Lambda and we've started using it some 3 years ago) create a local environment that exactly matches the .NET Core 3.1 + Amazon Linux 2 Lambda runtime to try and replicate the issue in order to improve the our understanding of the error. I used the .NET Core 3.1 Lambda Docker image provided by lambci and it worked wonderfully, giving me an accurate picture of the differences between Amazon Linux & Amazon Linux 2.
lambci
Without further ado, this boils down to the following: Amazon Linux has all the necessary native libraries in it's lib64 directory whereas Amazon Linux 2 only has a subset of those.
How to make sure your non-.NET Standard 2.1 dependencies work reliably in a .NET Core 3.1 Lambda function:
- Create an AWS Lambda Layer that includes the necessary for your use case .so files that are missing from Amazon Linux 2 /lib64 directory (I've taken those directly out of the Docker instance above for example). This will place all those files in the /opt directory of the executing Lambda instance container,
- Add this new AWS Lambda Layer to your .NET Core 3.1 project's serverless.template configuration file under Resources > Function > Properties like this:
"Layers": ["arn:aws:lambda:us-east-1:accountnumber:layer:name:version"]and finally, - In the same serverless.template configuration file, under Resources > Function > Properties > Environment > Variables, instruct the runtime to look for library dependencies in /opt (which is where we placed the required dependencies with the Layer we created in the previous steps) on top of the additional directories by adding an additional key/value pair as follows:
"LD_LIBRARY_PATH": "/opt:/var/lang/bin/shared/Microsoft.NETCore.App/3.1.1:/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib". One caveat is that you cannot just do /opt here; instead think of this as a Union type of operation, you have to add /opt in front of the directories the runtime already looks for dependencies by default.
Ultimately, your Lambda app's Amazon Linux 2 relevant directories will look something like this (and this is by no means an exhaustive list of the missing .so libraries between Amazon Linux & Amazon Linux 2, just the ones we had to use for our particular use case):
/lib64 | /var/runtime | /usr/lib64 | /opt (the layer) |
|---|---|---|---|
librt.so.1 libpthread.so.0 libc.so.6 liblttng-ust.so.0 liblttng-ust-tracepoint.so.0 libc.so.6 libstdc++.so.6 libm.so.6 libgcc_s.so.1 libdl.so.2 libicuuc.so.50 libicudata.so.50 libicui18n.so.50 libssl.so.10 libgssapi_krb5.so.2 libkrb5.so.3 libcom_err.so.2 libk5crypto.so.3 libcrypto.so.10 libz.so.1 libkrb5support.so.0 libkeyutils.so.1 libresolv.so.2 libselinux.so.1 libpcre.so.1 libnss_files.so.2 libnss_dns.so.2 | liblambdaio.so liblambdalog.so liblambdaipc.so liblambdaruntime.so | libc.so liblibc.so Crypt32.dll.so libCrypt32.dll.so Crypt32.dll libssl.so.1.1 libssl.so.1.0.2 libssl.so.1.0.0 libnss_sss.so.2 | libcurl.so.4 libnghttp2.so.14 libidn2.so.0 libssh2.so.1 libldap-2.4.so.2 liblber-2.4.so.2 libunistring.so.0 libsasl2.so.3 libssl3.so libsmime3.so libnss3.so libnssutil3.so libplds4.so libplc4.so libnspr4.so libcrypt.so.1 |
Next steps
That's it, you're now officially breaking the laws of software physics(?)!
Over the course of this blog series, I'll cover the specifics of each of the new in .NET Core 3.x features that provide opportunities for improving the performance of an AWS Lambda app, compared to .NET Core 2.1.
- Replacing the default Lambda serialization library Amazon.Lambda.Serialization.Json.JsonSerializer which is based on Json.Net with Amazon.Lambda.Serialization.SystemTextJson which is a new implementation based on System.Text.Json,
- Using a new
dotnet publishflag calledReadyToRunwhich will enable the compilation of your application assemblies in an ahead-of-time (AOT) fashion ultimately reducing the amount of work the just-in-time (JIT) compiler has to do and thus resulting in performance improvements and - Using a feature called Runtime Package Store when publishing new Lambdas, allows for bundling up your function's dependencies separately so that the dependencies don't have to be uploaded every time you changed your code and deployed a new Lambda function version.
p.s. #BlackLivesMatter ✊🏿
