Creating Custom AWS Lambda Runtime using Lambda Layer

Over the weekend, I spent some time experimenting with the new Runtime API and Lambda Layer that AWS announced last week. The goal was to create a custom runtime that I could use to start writing Lambda functions in PHP. Following is a brief summary of what I learned in the process and steps to create your own custom runtime.

Source for the code samples used in this post: https://github.com/pagnihotry/PHP-Lambda-Runtime

Getting started

There are a couple of offerings from AWS that you would need to be familiar with for creating a custom runtime and to add support for any language on Lambda:

1. Lambda Layer - AWS allows users to bundle some files (code or libraries) separate from the main function. This is called Lambda Layer. The advantage of using Layer is that you can create it and reuse in as many functions as you like, keeping the function itself small and enabling easy reuse of the common code or libraries.
2. Runtime API - This is the API that allows a Layer (or Function) to interact with AWS Lambda like send response or error to it or check if a new event is available to process. More on Runtime API: AWS Lambda Runtimes.

Combining these 2, you can add re-usable support for pretty much any language or possibly create custom runtimes with baked in libraries, etc. that you can use in multiple functions or distribute to your users.

Note: You can add multiple layers in a function and specify the order in which you would like to process them.

Adding Custom Runtime

To add a custom runtime that can be shared across functions, you will first need to create a Layer and then implement the logic to process the incoming invocations.

While creating a custom runtime, Lambda expects certain files and logic to be present in the uploaded zip file. It expects a file called bootstrap with permissions 755. This file should either implement or start an endless loop fetching incoming invocations of Lambda from the Runtime API and report the response or errors to it.

Once the instace serving Lambda starts, bootstrap is moved to /opt/bootstrap. I also noticed that it moved all the files I had included in my zip file to the /opt directory. The bootstrap file is executed exactly once - at the startup time. Hence, the endless loop is needed to keep processing the incoming requests. If the runtime stops, you would see your Lambda functions returning an error.

Available environment variables

There are certain environment variables which are available to the Layer that I have used in my implementation:

1. LAMBDA_TASK_ROOT - The local directory containing the function code
2. AWS_LAMBDA_RUNTIME_API - HOST:PORT for the runtime API
3. _HANDLER - The function handler defined in the Lambda config

A full list of all the environment variables that are available can be found here: Lambda Execution Environment and Available Libraries

bootstrap

The first step for making your own Layer will be to put together the required files. As mentioned earlier in this post, you will need a bootstrap file. In my setup, the bootstrap file executes a bundled PHP script using a static PHP executable that I compiled from source on alinux 2017.03.

bootstrap (https://github.com/pagnihotry/PHP-Lambda-Runtime/blob/master/runtime/bootstrap)

#!/bin/sh

#go into the source directory
cd $LAMBDA_TASK_ROOT

#execute the runtime
/opt/php /opt/runtime.php

Note: More time you spend in initialization step of your bootstrap, higher will be the cold start time of your Lambda function.

runtime.php

The runtime.php file contains the logic to curl the Runtime API, checking for available Lambda invocations and then calling the handler with the payload data followed by reporting the results to the Runtime API.

When a Lambda function is invoked, the payload becomes available on the /runtime/invocation/next endpoint of the API. From here on, the runtime code fetches the event, runs the handler set in the Lambda configuration and reports the response to /runtime/invocation/<AwsRequestId>/response or the error to /runtime/invocation/<AwsRequestId>/error. Once the response or error endpoints are hit, Lambda will return/display success or error to the invoking service.

If for some reason, error or response endpoints do not get a hit, I noticed that this loop keeps running invoking the handler multiple times till the maximum execution time of the Lambda is reached and then returns with an error. You can try it out by commenting flushResponse call in the provided example.

runtime.php (https://github.com/pagnihotry/PHP-Lambda-Runtime/blob/master/runtime/runtime.php)

$lambdaRuntime = new LambdaRuntime();
$handler =  $lambdaRuntime->getHandler();

//Extract file name and function
list($handlerFile , $handlerFunction) = explode(".", $handler);

//Include the handler file
require_once($handlerFile.".php");

//Poll for the next event to be processed

while (true) {

    //Get next event
    $data = $lambdaRuntime->getNextEventData();

    //Check if there was an error that runtime detected with the next event data
    if(isset($data["error"]) && $data["error"]) {
        continue;
    }

    //Process the events
    $eventPayload = $lambdaRuntime->getEventPayload();
    //Handler is of format Filename.function
    //Capture stdout
    ob_start();
    //Execute handler
    $functionReturn = $handlerFunction($eventPayload);
    $out = ob_get_clean();
    $lambdaRuntime->addToResponse($out.$functionReturn);
    //Report result
    $lambdaRuntime->flushResponse();

}

The layout of the zip file uploaded to the Layer:

.
├── bootstrap
├── php
└── runtime.php

Note all the files are at the root. Not all files are required to be there except for bootstrap. Lambda expects the bootstrap file to be in the root of the uploaded zip.

Creating the Layer in AWS

Once the code and file are in place, creating a new Layer can be done using the CLI or the UI.

From the UI

1. Navigate to AWS Lambda and Create a new Layer
2. Upload the zip file
3. Leave the Compatible runtimes empty
4. Press Create

AWS CLI

aws lambda publish-layer-version --layer-name <name of the layer> --zip-file fileb://<path to the zip file>

Adding a function that uses the custom Layer

Following are the steps to create the function that uses a custom runtime from the Layer created in the section above.

From the UI

1. Select Author from scratch
2. Select Use custom runtime in function code or layer
3. Select Layers
4. Click on Add a layer
5. Select Provide a layer version ARN
6. Enter the ARN with the version from the previous Layer and click Add
7. Save the function

AWS CLI

Create the function: aws lambda create-function --function-name <name fo the function> --zip-file fileb://<path to the zip file> --handler function.handler --runtime provided --role <ARN of the IAM role>

Update the function: aws lambda update-function-configuration --function-name <name fo the function> --layers <full ARN of the Layer with version>

Once you have followed all the steps outlined above, you should have a ready to go Lambda Layer to be used in your functions. As you can see, AWS has made it pretty easy to add support for new languages and bundle reusable shared code. There are some limitations that you may run into along the way like the upper limit for the total size of Lambda unzipped and including all the layers is still 250MB. More on limits for Lambda - https://docs.aws.amazon.com/lambda/latest/dg/limits.html