Serverless Deployment for ML Model Inference

AWS Lambda with HTTP API

1 Context

We provide a minimum step-by-step working example using the Serverless Framework, to deploy a machine learning model predictor written by Python, with AWS Lambda and API Gateway.

2 Problem Statement

We’ve trained a machine learning model in Python. We want to serve it over the Internet with an API endpoint for realtime prediction. But we don’t want to host or provision a server to run the code.

3 Prerequisite

Node is required. We recommend to use nvm to manage node.

Once node is ready on our system try to install serverless:

npm install -g serverless

We also need Docker in order to build the image for packaging.

Also, apparently, we need an AWS account. :)

4 Workflow

4.1 Create Project Template

Run

serverless create --template aws-python3 --name lambda_http_api --path lambda_http_api

which on success will give the following message:

Serverless: Generating boilerplate...
Serverless: Generating boilerplate in "/Users/kyle.c/k9/notebooks/data_eng/serverless/lambda_http_api"
 _______                             __
|   _   .-----.----.--.--.-----.----|  .-----.-----.-----.
|   |___|  -__|   _|  |  |  -__|   _|  |  -__|__ --|__ --|
|____   |_____|__|  \___/|_____|__| |__|_____|_____|_____|
|   |   |             The Serverless Application Framework
|       |                           serverless.com, v2.44.0
 -------'

Serverless: Successfully generated boilerplate for template: "aws-python3"

This will create two important files:

• A templated Python script as the Lambda entry point
• A templated serverless.yml to configure our deployment

4.2 Serverless Configuration

Let’s edit the serverless.yml to be something like the following:

service: predict

frameworkVersion: '2'

provider:
  name: aws
  region: ap-southeast-1
  runtime: python3.7
  lambdaHashingVersion: 20201221

functions:
  predict:
    handler: handler.predict
    events:
      - httpApi:
          path: /predict
          method: post
    package:
      patterns:
        - lgb.model

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerizePip: non-linux
    dockerExtraFiles:
      - /usr/lib64/libgomp.so.1
    zip: true

Several important notes:

• the handler specifies the function entry point, in this case a function named predict in the module handler
• the events: httpApi: defines our Lambda function to be exposed to an HTTP API endpoint
• the package: section will include/exclude any file that is dependent by the functions
• the plugins: section specify additional npm packages that will help us package the service
• the custom: pythonRequirements: dockerizePip: non-linux specifies that we want to prepare dependencies using Docker only when we are on a non-linux host OS
• the custom: pythonRequirements: zip: true reduces the deployment size

We can also set dockerizePip: true to always use Docker for dependency preparation. Be aware that our final deployment of Lambda will still have package type to be Zip instead of Image. Here dockerizePip simply means that we want to prepare the dependency using a Linux environment even if we are not on a Linux machine. This makes sense since the Lambda is going to be running on a Linux machine that is basically different from our local environment. By default serverless will use a Docker image that is as close as the Lambda running environment, if not entirely identical.¹

The dockerExtraFiles configuration is to fix the problem of lightgbm’s extra dependency.²

4.2.1 Deal with Size Limitation

Our Lambda use lightgbm which further depends on two very big packages: scikit-learn and scipy. Without using the zip: true trick we are not able to manage the overall package size under 250 MB.

The caveat is that we will need to introduce this piece of code:

try:
  import unzip_requirements
except ImportError:
  pass

to the beginning of our handler module.

There are two other ways to deal with the size problem:

• Use Lambda layer
• Use Image package type, which has a much larger file size limitation

To keep things simple we are not exploring these other approaches in this notebook.³

4.2.2 ML Model Dependency

Ideally, model file should be loaded from a versioned repository (such as AWS S3). But in this example just to demonstrate the file dependency layer and also for simplicity, we put a static model file and use package: section to include it.

4.2.3 Python Package Dependency

The Python environment running AWS Lambda by default comes with very limited packages installed. Common data science packages such as numpy, pandas, or scikit-learn are not available. The serverless framework helps us easily nail it by the serverless-python-requirements plugin.

To do so, we need to install and maintain the npm package for our project:

npm init
npm install --save serverless-python-requirements

Or for the minimalist we can also simply run:

serverless plugin install -n serverless-python-requirements

This will generate a minimum package.json and also lock file, along with the package installation, in the meantime automatically update our serverless.yml for the plugins section.

Now the only thing left is to prepare a conventional requirements.txt file under our project that locks in the dependent Python packages. The serverless-python-requirements package will automatically prepare the dependencies based on the requirement file.

4.3 Implement the Function

For demo purpose, we use the IRIS data to train a very simple gradient boosting model and save it to lgb.model.

This is the training script that outputs the model:

#!/usr/bin/env python
'''Train a toy model using the iris dataset.'''

import lightgbm as lgb
from sklearn.datasets import load_iris


iris = load_iris()
data_train = lgb.Dataset(iris.data, iris.target, feature_name=iris.feature_names)

params = {
  'boosting_type': 'gbdt',
  'objective': 'multiclass',
  'num_class': 3,
  'metric': ['multi_logloss', 'multi_error'],
  'max_depth': 3,
  'num_leaves': 3,
  'learning_rate': .1,
}

bst = lgb.train(
  params,
  data_train,
  num_boost_round=10,
  valid_sets=[data_train],
)

bst.save_model('lambda_http_api/lgb.model')

Now let’s edit the handler.py created in the boilerplate:

try:
  import unzip_requirements
except ImportError:
  pass

import json

import numpy as np
import lightgbm as lgb


model = lgb.Booster(model_file='lgb.model')
label_names = np.array(['setosa', 'versicolor', 'virginica'])


def predict(event, context):

  body = json.loads(event['body'])  # assuming a json string
  x = [[body["sepal length"], body["sepal width"], body["petal length"], body["petal width"]]]
  yhat = model.predict(x)
  label = label_names[np.argmax(yhat, axis=1)]

  response = {
    'statusCode': 200,
    'body': json.dumps({
      'proba': yhat[0].tolist(),
      'label': label[0],
    }),
  }

  return response

4.4 Test with Local Invocation

Before we do deployment, we can invoke the function locally to see if it works:

# note that the body content is a json string
serverless invoke local -f predict --data '{"body": "{\"sepal length\": 6, \"sepal width\": 3, \"petal length\": 5, \"petal width\": 2}"}'

which, on success, should return the following response:

{
    "statusCode": 200,
    "body": "{\"proba\": [0.11376620900197651, 0.17004835386299028, 0.7161854371350332], \"label\": \"virginica\"}"
}

4.5 Deploy

Now we are ready to deploy the service. Simply run:

serverless deploy  # make sure your AWS credential is available in the shell

It will take a while until deployment finished.

4.6 Smoke Test

Now let’s actually hit the endpoint with curl POST method:

curl -H "Content-Type: application/json" \
    --data '{"sepal length": 6, "sepal width": 3, "petal length": 5, "petal width": 2}' \
    https://<api-id>.execute-api.${AWS_REGION}.amazonaws.com/predict

which, on success, should return the following response:

{"proba": [0.11376620900197651, 0.17004835386299028, 0.7161854371350332], "label": "virginica"}

Mission accomplished!

4.7 Destroy

To remove the entire deployment stack, simply run:

serverless remove

---

1. https://github.com/lambci/docker-lambda↩︎

2. Related discussion: https://stackoverflow.com/questions/61717991/xgboost-library-libxgboost-so-could-not-be-loaded↩︎

3. Indeed, scikit-learn is not really needed for lightgbm. So another workaround is to use the noDeploy: option to exclude it manually.↩︎