Context
We provide a minimum step-by-step working example using the Serverless Framework, to deploy a machine learning model predictor written by Python, with AWS Lambda and API Gateway.
Prerequisite
Node
is required. We recommend to use nvm
to manage node
.
Once node
is ready on our system try to install serverless
:
npm install -g serverless
We also need Docker in order to build the image for packaging.
Also, apparently, we need an AWS account. :)
Workflow
Create Project Template
Run
serverless create --template aws-python3 --name lambda_http_api --path lambda_http_api
which on success will give the following message:
Serverless: Generating boilerplate...
Serverless: Generating boilerplate in "/Users/kyle.c/k9/notebooks/data_eng/serverless/lambda_http_api"
_______ __
| _ .-----.----.--.--.-----.----| .-----.-----.-----.
| |___| -__| _| | | -__| _| | -__|__ --|__ --|
|____ |_____|__| \___/|_____|__| |__|_____|_____|_____|
| | | The Serverless Application Framework
| | serverless.com, v2.44.0
-------'
Serverless: Successfully generated boilerplate for template: "aws-python3"
This will create two important files:
- A templated Python script as the Lambda entry point
- A templated
serverless.yml
to configure our deployment
Serverless Configuration
Let’s edit the serverless.yml
to be something like the following:
service: predict
frameworkVersion: '2'
provider:
name: aws
region: ap-southeast-1
runtime: python3.7
lambdaHashingVersion: 20201221
functions:
predict:
handler: handler.predict
events:
- httpApi:
path: /predict
method: post
package:
patterns:
- lgb.model
plugins:
- serverless-python-requirements
custom:
pythonRequirements:
dockerizePip: non-linux
dockerExtraFiles:
- /usr/lib64/libgomp.so.1
zip: true
Several important notes:
- the
handler
specifies the function entry point, in this case a function named predict
in the module handler
- the
events: httpApi:
defines our Lambda function to be exposed to an HTTP API endpoint
- the
package:
section will include/exclude any file that is dependent by the functions
- the
plugins:
section specify additional npm
packages that will help us package the service
- the
custom: pythonRequirements: dockerizePip: non-linux
specifies that we want to prepare dependencies using Docker only when we are on a non-linux host OS
- the
custom: pythonRequirements: zip: true
reduces the deployment size
We can also set dockerizePip: true
to always use Docker for dependency preparation. Be aware that our final deployment of Lambda will still have package type to be Zip
instead of Image
. Here dockerizePip
simply means that we want to prepare the dependency using a Linux environment even if we are not on a Linux machine. This makes sense since the Lambda is going to be running on a Linux machine that is basically different from our local environment. By default serverless
will use a Docker image that is as close as the Lambda running environment, if not entirely identical.
The dockerExtraFiles
configuration is to fix the problem of lightgbm
’s extra dependency.
Deal with Size Limitation
Our Lambda use lightgbm
which further depends on two very big packages: scikit-learn
and scipy
. Without using the zip: true
trick we are not able to manage the overall package size under 250 MB.
The caveat is that we will need to introduce this piece of code:
try:
import unzip_requirements
except ImportError:
pass
to the beginning of our handler module.
There are two other ways to deal with the size problem:
- Use Lambda layer
- Use
Image
package type, which has a much larger file size limitation
To keep things simple we are not exploring these other approaches in this notebook.
ML Model Dependency
Ideally, model file should be loaded from a versioned repository (such as AWS S3). But in this example just to demonstrate the file dependency layer and also for simplicity, we put a static model file and use package:
section to include it.
Python Package Dependency
The Python environment running AWS Lambda by default comes with very limited packages installed. Common data science packages such as numpy
, pandas
, or scikit-learn
are not available. The serverless
framework helps us easily nail it by the serverless-python-requirements
plugin.
To do so, we need to install and maintain the npm
package for our project:
npm init
npm install --save serverless-python-requirements
Or for the minimalist we can also simply run:
serverless plugin install -n serverless-python-requirements
This will generate a minimum package.json
and also lock file, along with the package installation, in the meantime automatically update our serverless.yml
for the plugins
section.
Now the only thing left is to prepare a conventional requirements.txt
file under our project that locks in the dependent Python packages. The serverless-python-requirements
package will automatically prepare the dependencies based on the requirement file.
Implement the Function
For demo purpose, we use the IRIS data to train a very simple gradient boosting model and save it to lgb.model
.
This is the training script that outputs the model:
#!/usr/bin/env python
'''Train a toy model using the iris dataset.'''
import lightgbm as lgb
from sklearn.datasets import load_iris
iris = load_iris()
data_train = lgb.Dataset(iris.data, iris.target, feature_name=iris.feature_names)
params = {
'boosting_type': 'gbdt',
'objective': 'multiclass',
'num_class': 3,
'metric': ['multi_logloss', 'multi_error'],
'max_depth': 3,
'num_leaves': 3,
'learning_rate': .1,
}
bst = lgb.train(
params,
data_train,
num_boost_round=10,
valid_sets=[data_train],
)
bst.save_model('lambda_http_api/lgb.model')
Now let’s edit the handler.py
created in the boilerplate:
try:
import unzip_requirements
except ImportError:
pass
import json
import numpy as np
import lightgbm as lgb
model = lgb.Booster(model_file='lgb.model')
label_names = np.array(['setosa', 'versicolor', 'virginica'])
def predict(event, context):
body = json.loads(event['body']) # assuming a json string
x = [[body["sepal length"], body["sepal width"], body["petal length"], body["petal width"]]]
yhat = model.predict(x)
label = label_names[np.argmax(yhat, axis=1)]
response = {
'statusCode': 200,
'body': json.dumps({
'proba': yhat[0].tolist(),
'label': label[0],
}),
}
return response
Test with Local Invocation
Before we do deployment, we can invoke the function locally to see if it works:
# note that the body content is a json string
serverless invoke local -f predict --data '{"body": "{\"sepal length\": 6, \"sepal width\": 3, \"petal length\": 5, \"petal width\": 2}"}'
which, on success, should return the following response:
{
"statusCode": 200,
"body": "{\"proba\": [0.11376620900197651, 0.17004835386299028, 0.7161854371350332], \"label\": \"virginica\"}"
}
Deploy
Now we are ready to deploy the service. Simply run:
serverless deploy # make sure your AWS credential is available in the shell
It will take a while until deployment finished.
Smoke Test
Now let’s actually hit the endpoint with curl
POST method:
curl -H "Content-Type: application/json" \
--data '{"sepal length": 6, "sepal width": 3, "petal length": 5, "petal width": 2}' \
https://<api-id>.execute-api.${AWS_REGION}.amazonaws.com/predict
which, on success, should return the following response:
{"proba": [0.11376620900197651, 0.17004835386299028, 0.7161854371350332], "label": "virginica"}
Mission accomplished!
Destroy
To remove the entire deployment stack, simply run:
serverless remove
LS0tCnRpdGxlOiAiU2VydmVybGVzcyBEZXBsb3ltZW50IGZvciBNTCBNb2RlbCBJbmZlcmVuY2UiCnN1YnRpdGxlOiAiQVdTIExhbWJkYSB3aXRoIEhUVFAgQVBJIgphdXRob3I6Ci0gbmFtZTogS3lsZSBDaHVuZwogIGFmZmlsaWF0aW9uOgpkYXRlOiAiYHIgZm9ybWF0KFN5cy50aW1lKCksICclZCAlYiAlWScpYCBMYXN0IFVwZGF0ZWQgKDA1IEp1bmUgMjAyMSBGaXJzdCBVcGxvYWRlZCkiCm91dHB1dDoKICBodG1sX25vdGVib29rOgogICAgaGlnaGxpZ2h0OiB0ZXh0bWF0ZQogICAgbnVtYmVyX3NlY3Rpb25zOiB5ZXMKICAgIHRoZW1lOiBsdW1lbgogICAgdG9jOiB5ZXMKICAgIHRvY19kZXB0aDogMwogICAgdG9jX2Zsb2F0OiB5ZXMKICAgIGluY2x1ZGVzOgogICAgICBpbl9oZWFkZXI6IC90bXAvbWV0YV9oZWFkZXIuaHRtbAogIGNvZGVfZG93bmxvYWQ6IHRydWUKICBjb2RlX2ZvbGRpbmc6ICJub25lIgotLS0KCmBgYHtyIG1ldGEsIGluY2x1ZGU9RkFMU0V9CmxpYnJhcnkobWV0YXRoaXMpCgojIEFkZCBvcGVuIGdyYXBoIG1ldGEuCm1ldGEoKSAlPiUKICBtZXRhX2Rlc2NyaXB0aW9uKAogICAgIkEgZGF0YSBzY2llbmNlIG5vdGVib29rIGFib3V0IHNlcnZlcmxlc3MgZGVwbG95bWVudCBmb3IgQVdTIExhbWJkYSB3aXRoIEhUVFAgQVBJLiIKICApICU+JQogIG1ldGFfdmlld3BvcnQoKSAlPiUKICBtZXRhX3NvY2lhbCgKICAgIHRpdGxlPSJTZXJ2ZXJsZXNzIERlcGxveW1lbnQ6IEFXUyBMYW1iZGEgd2l0aCBIVFRQIEFQSSIsCiAgICB1cmw9Imh0dHBzOi8vZXZlcmRhcmsuZ2l0aHViLmlvL2s5L25vdGVib29rcy9kYXRhX2VuZy9zZXJ2ZXJsZXNzL2xhbWJkYV9odHRwX2FwaS5uYi5odG1sIiwKICAgIGltYWdlPSJodHRwczovL2V2ZXJkYXJrLmdpdGh1Yi5pby9rOS9hc3NldHMvc2VydmVybGVzc19sb2dvLnBuZyIsCiAgICBvZ190eXBlPSJhcnRpY2xlIiwKICAgIG9nX2F1dGhvcj0iS3lsZSBDaHVuZyIsCiAgICB0d2l0dGVyX2NhcmRfdHlwZT0ic3VtbWFyeSIKICApCgpjb250ZW50cyA8LSBjKCkKCiMgQWRkIEdpdGh1YiBjb3JuZXIuCmdpdGh1Yl9jb3JuZXJfc3ZnIDwtICIuLi8uLi8uLi9hc3NldHMvZ2l0aHViX2Nvcm5lci5odG1sIgpnaXRodWJfY29ybmVyX2NvbmYgPC0gbGlzdChnaXRodWJfbGluaz0iaHR0cHM6Ly9naXRodWIuY29tL2V2ZXJkYXJrL2s5L3RyZWUvbWFzdGVyL25vdGVib29rcy9kYXRhX2VuZy9zZXJ2ZXJsZXNzIikKY29udGVudHMgPC0gYyhjb250ZW50cywgc3RyaW5ncjo6c3RyX2ludGVycChyZWFkTGluZXMoZ2l0aHViX2Nvcm5lcl9zdmcpLCBnaXRodWJfY29ybmVyX2NvbmYpKQoKbWV0YV9oZWFkZXJfZmlsZSA8LSBmaWxlKCIvdG1wL21ldGFfaGVhZGVyLmh0bWwiKQp3cml0ZUxpbmVzKGNvbnRlbnRzLCBtZXRhX2hlYWRlcl9maWxlKQpjbG9zZShtZXRhX2hlYWRlcl9maWxlKQpgYGAKCiMgQ29udGV4dAoKV2UgcHJvdmlkZSBhIG1pbmltdW0gc3RlcC1ieS1zdGVwIHdvcmtpbmcgZXhhbXBsZSB1c2luZyB0aGUgW1NlcnZlcmxlc3MgRnJhbWV3b3JrXShodHRwczovL2dpdGh1Yi5jb20vc2VydmVybGVzcy9zZXJ2ZXJsZXNzKSwKdG8gZGVwbG95IGEgbWFjaGluZSBsZWFybmluZyBtb2RlbCBwcmVkaWN0b3Igd3JpdHRlbiBieSBQeXRob24sCndpdGggW0FXUyBMYW1iZGFdKGh0dHBzOi8vYXdzLmFtYXpvbi5jb20vbGFtYmRhLz90cmtDYW1wYWlnbj1hY3FfcGFpZF9zZWFyY2hfYnJhbmQmc2NfY2hhbm5lbD1wcyZzY19jYW1wYWlnbj1hY3F1aXNpdGlvbl9TRyZzY19wdWJsaXNoZXI9R29vZ2xlJnNjX2NhdGVnb3J5PUNsb3VkJTIwQ29tcHV0aW5nJnNjX2NvdW50cnk9U0cmc2NfZ2VvPUFQQUMmc2Nfb3V0Y29tZT1hY3Emc2NfZGV0YWlsPSUyQmFtYXpvbiUyMCUyQmNsb3VkJTIwJTJCc2VydmljZXMmc2NfY29udGVudD17YWRncm91cH0mc2NfbWF0Y2h0eXBlPWImc2Nfc2VnbWVudD00NzY5OTQ0MTIwMTMmc2NfbWVkaXVtPUFDUS1QfFBTLUdPfEJyYW5kfERlc2t0b3B8U1V8Q2xvdWQlMjBDb21wdXRpbmd8U29sdXRpb258U0d8RU58U2l0ZWxpbmsmc19rd2NpZD1BTCE0NDIyITMhNDc2OTk0NDEyMDEzIWIhIWchISUyQmFtYXpvbiUyMCUyQmNsb3VkJTIwJTJCc2VydmljZXMmZWZfaWQ9Q2owS0NRanc1UEdGQmhDMkFSSXNBSUZJTU5jeU9MWGRWOFpHTjltVFJpZVRHOEhMUk5jQW9JRTNqa1dIV3I5NUZuWHFfZzZWNHB3NlZBQWFBaGN4RUFMd193Y0I6RzpzJnNfa3djaWQ9QUwhNDQyMiEzITQ3Njk5NDQxMjAxMyFiISFnISElMkJhbWF6b24lMjAlMkJjbG91ZCUyMCUyQnNlcnZpY2VzKSBhbmQgW0FQSSBHYXRld2F5XShodHRwczovL2F3cy5hbWF6b24uY29tL2FwaS1nYXRld2F5LykuCgojIFByb2JsZW0gU3RhdGVtZW50CgpXZSd2ZSB0cmFpbmVkIGEgbWFjaGluZSBsZWFybmluZyBtb2RlbCBpbiBQeXRob24uCldlIHdhbnQgdG8gc2VydmUgaXQgb3ZlciB0aGUgSW50ZXJuZXQgd2l0aCBhbiBBUEkgZW5kcG9pbnQgZm9yIHJlYWx0aW1lIHByZWRpY3Rpb24uCkJ1dCB3ZSBkb24ndCB3YW50IHRvIGhvc3Qgb3IgcHJvdmlzaW9uIGEgc2VydmVyIHRvIHJ1biB0aGUgY29kZS4KCiMgUHJlcmVxdWlzaXRlCgpbYE5vZGVgXShodHRwczovL25vZGVqcy5vcmcvZW4vKSBpcyByZXF1aXJlZC4KV2UgcmVjb21tZW5kIHRvIHVzZSBbYG52bWBdKGh0dHBzOi8vZ2l0aHViLmNvbS9udm0tc2gvbnZtKSB0byBtYW5hZ2UgYG5vZGVgLgoKT25jZSBgbm9kZWAgaXMgcmVhZHkgb24gb3VyIHN5c3RlbSB0cnkgdG8gaW5zdGFsbCBgc2VydmVybGVzc2A6CgpgYGAKbnBtIGluc3RhbGwgLWcgc2VydmVybGVzcwpgYGAKCldlIGFsc28gbmVlZCBbRG9ja2VyXShodHRwczovL3d3dy5kb2NrZXIuY29tLykgaW4gb3JkZXIgdG8gYnVpbGQgdGhlIGltYWdlIGZvciBwYWNrYWdpbmcuCgpBbHNvLCBhcHBhcmVudGx5LCB3ZSBuZWVkIGFuIEFXUyBhY2NvdW50LiA6KQoKIyBXb3JrZmxvdwoKIyMgQ3JlYXRlIFByb2plY3QgVGVtcGxhdGUKClJ1bgoKYGBgCnNlcnZlcmxlc3MgY3JlYXRlIC0tdGVtcGxhdGUgYXdzLXB5dGhvbjMgLS1uYW1lIGxhbWJkYV9odHRwX2FwaSAtLXBhdGggbGFtYmRhX2h0dHBfYXBpCmBgYAoKd2hpY2ggb24gc3VjY2VzcyB3aWxsIGdpdmUgdGhlIGZvbGxvd2luZyBtZXNzYWdlOgoKYGBgClNlcnZlcmxlc3M6IEdlbmVyYXRpbmcgYm9pbGVycGxhdGUuLi4KU2VydmVybGVzczogR2VuZXJhdGluZyBib2lsZXJwbGF0ZSBpbiAiL1VzZXJzL2t5bGUuYy9rOS9ub3RlYm9va3MvZGF0YV9lbmcvc2VydmVybGVzcy9sYW1iZGFfaHR0cF9hcGkiCiBfX19fX19fICAgICAgICAgICAgICAgICAgICAgICAgICAgICBfXwp8ICAgXyAgIC4tLS0tLS4tLS0tLi0tLi0tLi0tLS0tLi0tLS18ICAuLS0tLS0uLS0tLS0uLS0tLS0uCnwgICB8X19ffCAgLV9ffCAgIF98ICB8ICB8ICAtX198ICAgX3wgIHwgIC1fX3xfXyAtLXxfXyAtLXwKfF9fX18gICB8X19fX198X198ICBcX19fL3xfX19fX3xfX3wgfF9ffF9fX19ffF9fX19ffF9fX19ffAp8ICAgfCAgIHwgICAgICAgICAgICAgVGhlIFNlcnZlcmxlc3MgQXBwbGljYXRpb24gRnJhbWV3b3JrCnwgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgIHNlcnZlcmxlc3MuY29tLCB2Mi40NC4wCiAtLS0tLS0tJwoKU2VydmVybGVzczogU3VjY2Vzc2Z1bGx5IGdlbmVyYXRlZCBib2lsZXJwbGF0ZSBmb3IgdGVtcGxhdGU6ICJhd3MtcHl0aG9uMyIKYGBgCgpUaGlzIHdpbGwgY3JlYXRlIHR3byBpbXBvcnRhbnQgZmlsZXM6CgotIEEgdGVtcGxhdGVkIFB5dGhvbiBzY3JpcHQgYXMgdGhlIExhbWJkYSBlbnRyeSBwb2ludAotIEEgdGVtcGxhdGVkIGBzZXJ2ZXJsZXNzLnltbGAgdG8gY29uZmlndXJlIG91ciBkZXBsb3ltZW50CgojIyBTZXJ2ZXJsZXNzIENvbmZpZ3VyYXRpb24KCkxldCdzIGVkaXQgdGhlIGBzZXJ2ZXJsZXNzLnltbGAgdG8gYmUgc29tZXRoaW5nIGxpa2UgdGhlIGZvbGxvd2luZzoKCmBgYHltbApzZXJ2aWNlOiBwcmVkaWN0CgpmcmFtZXdvcmtWZXJzaW9uOiAnMicKCnByb3ZpZGVyOgogIG5hbWU6IGF3cwogIHJlZ2lvbjogYXAtc291dGhlYXN0LTEKICBydW50aW1lOiBweXRob24zLjcKICBsYW1iZGFIYXNoaW5nVmVyc2lvbjogMjAyMDEyMjEKCmZ1bmN0aW9uczoKICBwcmVkaWN0OgogICAgaGFuZGxlcjogaGFuZGxlci5wcmVkaWN0CiAgICBldmVudHM6CiAgICAgIC0gaHR0cEFwaToKICAgICAgICAgIHBhdGg6IC9wcmVkaWN0CiAgICAgICAgICBtZXRob2Q6IHBvc3QKICAgIHBhY2thZ2U6CiAgICAgIHBhdHRlcm5zOgogICAgICAgIC0gbGdiLm1vZGVsCgpwbHVnaW5zOgogIC0gc2VydmVybGVzcy1weXRob24tcmVxdWlyZW1lbnRzCgpjdXN0b206CiAgcHl0aG9uUmVxdWlyZW1lbnRzOgogICAgZG9ja2VyaXplUGlwOiBub24tbGludXgKICAgIGRvY2tlckV4dHJhRmlsZXM6CiAgICAgIC0gL3Vzci9saWI2NC9saWJnb21wLnNvLjEKICAgIHppcDogdHJ1ZQpgYGAKClNldmVyYWwgaW1wb3J0YW50IG5vdGVzOgoKLSB0aGUgYGhhbmRsZXJgIHNwZWNpZmllcyB0aGUgZnVuY3Rpb24gZW50cnkgcG9pbnQsIGluIHRoaXMgY2FzZSBhIGZ1bmN0aW9uIG5hbWVkIGBwcmVkaWN0YCBpbiB0aGUgbW9kdWxlIGBoYW5kbGVyYAotIHRoZSBgZXZlbnRzOiBodHRwQXBpOmAgZGVmaW5lcyBvdXIgTGFtYmRhIGZ1bmN0aW9uIHRvIGJlIGV4cG9zZWQgdG8gYW4gSFRUUCBBUEkgZW5kcG9pbnQKLSB0aGUgYHBhY2thZ2U6YCBzZWN0aW9uIHdpbGwgaW5jbHVkZS9leGNsdWRlIGFueSBmaWxlIHRoYXQgaXMgZGVwZW5kZW50IGJ5IHRoZSBmdW5jdGlvbnMKLSB0aGUgYHBsdWdpbnM6YCBzZWN0aW9uIHNwZWNpZnkgYWRkaXRpb25hbCBgbnBtYCBwYWNrYWdlcyB0aGF0IHdpbGwgaGVscCB1cyBwYWNrYWdlIHRoZSBzZXJ2aWNlCi0gdGhlIGBjdXN0b206IHB5dGhvblJlcXVpcmVtZW50czogZG9ja2VyaXplUGlwOiBub24tbGludXhgIHNwZWNpZmllcyB0aGF0IHdlIHdhbnQgdG8gcHJlcGFyZSBkZXBlbmRlbmNpZXMgdXNpbmcgRG9ja2VyIG9ubHkgd2hlbiB3ZSBhcmUgb24gYSBub24tbGludXggaG9zdCBPUwotIHRoZSBgY3VzdG9tOiBweXRob25SZXF1aXJlbWVudHM6IHppcDogdHJ1ZWAgcmVkdWNlcyB0aGUgZGVwbG95bWVudCBzaXplCgpXZSBjYW4gYWxzbyBzZXQgYGRvY2tlcml6ZVBpcDogdHJ1ZWAgdG8gYWx3YXlzIHVzZSBEb2NrZXIgZm9yIGRlcGVuZGVuY3kgcHJlcGFyYXRpb24uCkJlIGF3YXJlIHRoYXQgb3VyIGZpbmFsIGRlcGxveW1lbnQgb2YgTGFtYmRhIHdpbGwgc3RpbGwgaGF2ZSBwYWNrYWdlIHR5cGUgdG8gYmUgYFppcGAgaW5zdGVhZCBvZiBgSW1hZ2VgLgpIZXJlIGBkb2NrZXJpemVQaXBgIHNpbXBseSBtZWFucyB0aGF0IHdlIHdhbnQgdG8gcHJlcGFyZSB0aGUgZGVwZW5kZW5jeSB1c2luZyBhIExpbnV4IGVudmlyb25tZW50IGV2ZW4gaWYgd2UgYXJlIG5vdCBvbiBhIExpbnV4IG1hY2hpbmUuClRoaXMgbWFrZXMgc2Vuc2Ugc2luY2UgdGhlIExhbWJkYSBpcyBnb2luZyB0byBiZSBydW5uaW5nIG9uIGEgTGludXggbWFjaGluZSB0aGF0IGlzIGJhc2ljYWxseSBkaWZmZXJlbnQgZnJvbSBvdXIgbG9jYWwgZW52aXJvbm1lbnQuCkJ5IGRlZmF1bHQgYHNlcnZlcmxlc3NgIHdpbGwgdXNlIGEgRG9ja2VyIGltYWdlIHRoYXQgaXMgYXMgY2xvc2UgYXMgdGhlIExhbWJkYSBydW5uaW5nIGVudmlyb25tZW50LAppZiBub3QgZW50aXJlbHkgaWRlbnRpY2FsLl5baHR0cHM6Ly9naXRodWIuY29tL2xhbWJjaS9kb2NrZXItbGFtYmRhXQoKVGhlIGBkb2NrZXJFeHRyYUZpbGVzYCBjb25maWd1cmF0aW9uIGlzIHRvIGZpeCB0aGUgcHJvYmxlbSBvZiBgbGlnaHRnYm1gJ3MgZXh0cmEgZGVwZW5kZW5jeS5eWwpSZWxhdGVkIGRpc2N1c3Npb246IGh0dHBzOi8vc3RhY2tvdmVyZmxvdy5jb20vcXVlc3Rpb25zLzYxNzE3OTkxL3hnYm9vc3QtbGlicmFyeS1saWJ4Z2Jvb3N0LXNvLWNvdWxkLW5vdC1iZS1sb2FkZWRdCgojIyMgRGVhbCB3aXRoIFNpemUgTGltaXRhdGlvbgoKT3VyIExhbWJkYSB1c2UgYGxpZ2h0Z2JtYCB3aGljaCBmdXJ0aGVyIGRlcGVuZHMgb24gdHdvIHZlcnkgYmlnIHBhY2thZ2VzOiBgc2Npa2l0LWxlYXJuYCBhbmQgYHNjaXB5YC4KV2l0aG91dCB1c2luZyB0aGUgYHppcDogdHJ1ZWAgdHJpY2sgd2UgYXJlIG5vdCBhYmxlIHRvIG1hbmFnZSB0aGUgb3ZlcmFsbCBwYWNrYWdlIHNpemUgdW5kZXIgMjUwIE1CLgoKVGhlIGNhdmVhdCBpcyB0aGF0IHdlIHdpbGwgbmVlZCB0byBpbnRyb2R1Y2UgdGhpcyBwaWVjZSBvZiBjb2RlOgoKYGBgcHl0aG9uCnRyeToKICBpbXBvcnQgdW56aXBfcmVxdWlyZW1lbnRzCmV4Y2VwdCBJbXBvcnRFcnJvcjoKICBwYXNzCmBgYAoKdG8gdGhlIGJlZ2lubmluZyBvZiBvdXIgaGFuZGxlciBtb2R1bGUuCgpUaGVyZSBhcmUgdHdvIG90aGVyIHdheXMgdG8gZGVhbCB3aXRoIHRoZSBzaXplIHByb2JsZW06CgotIFVzZSBMYW1iZGEgbGF5ZXIKLSBVc2UgYEltYWdlYCBwYWNrYWdlIHR5cGUsIHdoaWNoIGhhcyBhIG11Y2ggbGFyZ2VyIGZpbGUgc2l6ZSBsaW1pdGF0aW9uCgpUbyBrZWVwIHRoaW5ncyBzaW1wbGUgd2UgYXJlIG5vdCBleHBsb3JpbmcgdGhlc2Ugb3RoZXIgYXBwcm9hY2hlcyBpbiB0aGlzIG5vdGVib29rLl5bCkluZGVlZCwgYHNjaWtpdC1sZWFybmAgaXMgbm90IHJlYWxseSBuZWVkZWQgZm9yIGBsaWdodGdibWAuIFNvIGFub3RoZXIgd29ya2Fyb3VuZCBpcyB0byB1c2UgdGhlIGBub0RlcGxveTpgIG9wdGlvbiB0byBleGNsdWRlIGl0IG1hbnVhbGx5Ll0KCiMjIyBNTCBNb2RlbCBEZXBlbmRlbmN5CgpJZGVhbGx5LCBtb2RlbCBmaWxlIHNob3VsZCBiZSBsb2FkZWQgZnJvbSBhIHZlcnNpb25lZCByZXBvc2l0b3J5IChzdWNoIGFzIEFXUyBTMykuCkJ1dCBpbiB0aGlzIGV4YW1wbGUganVzdCB0byBkZW1vbnN0cmF0ZSB0aGUgZmlsZSBkZXBlbmRlbmN5IGxheWVyIGFuZCBhbHNvIGZvciBzaW1wbGljaXR5LAp3ZSBwdXQgYSBzdGF0aWMgbW9kZWwgZmlsZSBhbmQgdXNlIGBwYWNrYWdlOmAgc2VjdGlvbiB0byBpbmNsdWRlIGl0LgoKIyMjIFB5dGhvbiBQYWNrYWdlIERlcGVuZGVuY3kKClRoZSBQeXRob24gZW52aXJvbm1lbnQgcnVubmluZyBBV1MgTGFtYmRhIGJ5IGRlZmF1bHQgY29tZXMgd2l0aCB2ZXJ5IGxpbWl0ZWQgcGFja2FnZXMgaW5zdGFsbGVkLgpDb21tb24gZGF0YSBzY2llbmNlIHBhY2thZ2VzIHN1Y2ggYXMgYG51bXB5YCwgYHBhbmRhc2AsIG9yIGBzY2lraXQtbGVhcm5gIGFyZSBub3QgYXZhaWxhYmxlLgpUaGUgYHNlcnZlcmxlc3NgIGZyYW1ld29yayBoZWxwcyB1cyBlYXNpbHkgbmFpbCBpdCBieSB0aGUgYHNlcnZlcmxlc3MtcHl0aG9uLXJlcXVpcmVtZW50c2AgcGx1Z2luLgoKVG8gZG8gc28sCndlIG5lZWQgdG8gaW5zdGFsbCBhbmQgbWFpbnRhaW4gdGhlIGBucG1gIHBhY2thZ2UgZm9yIG91ciBwcm9qZWN0OgoKYGBgCm5wbSBpbml0Cm5wbSBpbnN0YWxsIC0tc2F2ZSBzZXJ2ZXJsZXNzLXB5dGhvbi1yZXF1aXJlbWVudHMKYGBgCgpPciBmb3IgdGhlIG1pbmltYWxpc3Qgd2UgY2FuIGFsc28gc2ltcGx5IHJ1bjoKCmBgYApzZXJ2ZXJsZXNzIHBsdWdpbiBpbnN0YWxsIC1uIHNlcnZlcmxlc3MtcHl0aG9uLXJlcXVpcmVtZW50cwpgYGAKClRoaXMgd2lsbCBnZW5lcmF0ZSBhIG1pbmltdW0gYHBhY2thZ2UuanNvbmAgYW5kIGFsc28gbG9jayBmaWxlLAphbG9uZyB3aXRoIHRoZSBwYWNrYWdlIGluc3RhbGxhdGlvbiwKaW4gdGhlIG1lYW50aW1lIGF1dG9tYXRpY2FsbHkgdXBkYXRlIG91ciBgc2VydmVybGVzcy55bWxgIGZvciB0aGUgYHBsdWdpbnNgIHNlY3Rpb24uCgpOb3cgdGhlIG9ubHkgdGhpbmcgbGVmdCBpcyB0byBwcmVwYXJlIGEgY29udmVudGlvbmFsIGByZXF1aXJlbWVudHMudHh0YCBmaWxlIHVuZGVyIG91ciBwcm9qZWN0IHRoYXQgbG9ja3MgaW4gdGhlIGRlcGVuZGVudCBQeXRob24gcGFja2FnZXMuClRoZSBgc2VydmVybGVzcy1weXRob24tcmVxdWlyZW1lbnRzYCBwYWNrYWdlIHdpbGwgYXV0b21hdGljYWxseSBwcmVwYXJlIHRoZSBkZXBlbmRlbmNpZXMgYmFzZWQgb24gdGhlIHJlcXVpcmVtZW50IGZpbGUuCgojIyBJbXBsZW1lbnQgdGhlIEZ1bmN0aW9uCgpGb3IgZGVtbyBwdXJwb3NlLAp3ZSB1c2UgdGhlIElSSVMgZGF0YSB0byB0cmFpbiBhIHZlcnkgc2ltcGxlIGdyYWRpZW50IGJvb3N0aW5nIG1vZGVsIGFuZCBzYXZlIGl0IHRvIGBsZ2IubW9kZWxgLgoKVGhpcyBpcyB0aGUgdHJhaW5pbmcgc2NyaXB0IHRoYXQgb3V0cHV0cyB0aGUgbW9kZWw6CgpgYGBweXRob24KIyEvdXNyL2Jpbi9lbnYgcHl0aG9uCicnJ1RyYWluIGEgdG95IG1vZGVsIHVzaW5nIHRoZSBpcmlzIGRhdGFzZXQuJycnCgppbXBvcnQgbGlnaHRnYm0gYXMgbGdiCmZyb20gc2tsZWFybi5kYXRhc2V0cyBpbXBvcnQgbG9hZF9pcmlzCgoKaXJpcyA9IGxvYWRfaXJpcygpCmRhdGFfdHJhaW4gPSBsZ2IuRGF0YXNldChpcmlzLmRhdGEsIGlyaXMudGFyZ2V0LCBmZWF0dXJlX25hbWU9aXJpcy5mZWF0dXJlX25hbWVzKQoKcGFyYW1zID0gewogICdib29zdGluZ190eXBlJzogJ2diZHQnLAogICdvYmplY3RpdmUnOiAnbXVsdGljbGFzcycsCiAgJ251bV9jbGFzcyc6IDMsCiAgJ21ldHJpYyc6IFsnbXVsdGlfbG9nbG9zcycsICdtdWx0aV9lcnJvciddLAogICdtYXhfZGVwdGgnOiAzLAogICdudW1fbGVhdmVzJzogMywKICAnbGVhcm5pbmdfcmF0ZSc6IC4xLAp9Cgpic3QgPSBsZ2IudHJhaW4oCiAgcGFyYW1zLAogIGRhdGFfdHJhaW4sCiAgbnVtX2Jvb3N0X3JvdW5kPTEwLAogIHZhbGlkX3NldHM9W2RhdGFfdHJhaW5dLAopCgpic3Quc2F2ZV9tb2RlbCgnbGFtYmRhX2h0dHBfYXBpL2xnYi5tb2RlbCcpCmBgYAoKTm93IGxldCdzIGVkaXQgdGhlIGBoYW5kbGVyLnB5YCBjcmVhdGVkIGluIHRoZSBib2lsZXJwbGF0ZToKCmBgYHB5dGhvbgp0cnk6CiAgaW1wb3J0IHVuemlwX3JlcXVpcmVtZW50cwpleGNlcHQgSW1wb3J0RXJyb3I6CiAgcGFzcwoKaW1wb3J0IGpzb24KCmltcG9ydCBudW1weSBhcyBucAppbXBvcnQgbGlnaHRnYm0gYXMgbGdiCgoKbW9kZWwgPSBsZ2IuQm9vc3Rlcihtb2RlbF9maWxlPSdsZ2IubW9kZWwnKQpsYWJlbF9uYW1lcyA9IG5wLmFycmF5KFsnc2V0b3NhJywgJ3ZlcnNpY29sb3InLCAndmlyZ2luaWNhJ10pCgoKZGVmIHByZWRpY3QoZXZlbnQsIGNvbnRleHQpOgoKICBib2R5ID0ganNvbi5sb2FkcyhldmVudFsnYm9keSddKSAgIyBhc3N1bWluZyBhIGpzb24gc3RyaW5nCiAgeCA9IFtbYm9keVsic2VwYWwgbGVuZ3RoIl0sIGJvZHlbInNlcGFsIHdpZHRoIl0sIGJvZHlbInBldGFsIGxlbmd0aCJdLCBib2R5WyJwZXRhbCB3aWR0aCJdXV0KICB5aGF0ID0gbW9kZWwucHJlZGljdCh4KQogIGxhYmVsID0gbGFiZWxfbmFtZXNbbnAuYXJnbWF4KHloYXQsIGF4aXM9MSldCgogIHJlc3BvbnNlID0gewogICAgJ3N0YXR1c0NvZGUnOiAyMDAsCiAgICAnYm9keSc6IGpzb24uZHVtcHMoewogICAgICAncHJvYmEnOiB5aGF0WzBdLnRvbGlzdCgpLAogICAgICAnbGFiZWwnOiBsYWJlbFswXSwKICAgIH0pLAogIH0KCiAgcmV0dXJuIHJlc3BvbnNlCmBgYAoKIyMgVGVzdCB3aXRoIExvY2FsIEludm9jYXRpb24KCkJlZm9yZSB3ZSBkbyBkZXBsb3ltZW50LAp3ZSBjYW4gaW52b2tlIHRoZSBmdW5jdGlvbiBsb2NhbGx5IHRvIHNlZSBpZiBpdCB3b3JrczoKCmBgYAojIG5vdGUgdGhhdCB0aGUgYm9keSBjb250ZW50IGlzIGEganNvbiBzdHJpbmcKc2VydmVybGVzcyBpbnZva2UgbG9jYWwgLWYgcHJlZGljdCAtLWRhdGEgJ3siYm9keSI6ICJ7XCJzZXBhbCBsZW5ndGhcIjogNiwgXCJzZXBhbCB3aWR0aFwiOiAzLCBcInBldGFsIGxlbmd0aFwiOiA1LCBcInBldGFsIHdpZHRoXCI6IDJ9In0nCmBgYAoKd2hpY2gsIG9uIHN1Y2Nlc3MsIHNob3VsZCByZXR1cm4gdGhlIGZvbGxvd2luZyByZXNwb25zZToKCmBgYAp7CiAgICAic3RhdHVzQ29kZSI6IDIwMCwKICAgICJib2R5IjogIntcInByb2JhXCI6IFswLjExMzc2NjIwOTAwMTk3NjUxLCAwLjE3MDA0ODM1Mzg2Mjk5MDI4LCAwLjcxNjE4NTQzNzEzNTAzMzJdLCBcImxhYmVsXCI6IFwidmlyZ2luaWNhXCJ9Igp9CgpgYGAKCiMjIERlcGxveQoKTm93IHdlIGFyZSByZWFkeSB0byBkZXBsb3kgdGhlIHNlcnZpY2UuClNpbXBseSBydW46CgpgYGAKc2VydmVybGVzcyBkZXBsb3kgICMgbWFrZSBzdXJlIHlvdXIgQVdTIGNyZWRlbnRpYWwgaXMgYXZhaWxhYmxlIGluIHRoZSBzaGVsbApgYGAKCkl0IHdpbGwgdGFrZSBhIHdoaWxlIHVudGlsIGRlcGxveW1lbnQgZmluaXNoZWQuCgojIyBTbW9rZSBUZXN0CgpOb3cgbGV0J3MgYWN0dWFsbHkgaGl0IHRoZSBlbmRwb2ludCB3aXRoIGBjdXJsYCBQT1NUIG1ldGhvZDoKCmBgYApjdXJsIC1IICJDb250ZW50LVR5cGU6IGFwcGxpY2F0aW9uL2pzb24iIFwKICAgIC0tZGF0YSAneyJzZXBhbCBsZW5ndGgiOiA2LCAic2VwYWwgd2lkdGgiOiAzLCAicGV0YWwgbGVuZ3RoIjogNSwgInBldGFsIHdpZHRoIjogMn0nIFwKICAgIGh0dHBzOi8vPGFwaS1pZD4uZXhlY3V0ZS1hcGkuJHtBV1NfUkVHSU9OfS5hbWF6b25hd3MuY29tL3ByZWRpY3QKYGBgCgp3aGljaCwgb24gc3VjY2Vzcywgc2hvdWxkIHJldHVybiB0aGUgZm9sbG93aW5nIHJlc3BvbnNlOgoKYGBgCnsicHJvYmEiOiBbMC4xMTM3NjYyMDkwMDE5NzY1MSwgMC4xNzAwNDgzNTM4NjI5OTAyOCwgMC43MTYxODU0MzcxMzUwMzMyXSwgImxhYmVsIjogInZpcmdpbmljYSJ9CmBgYAoKTWlzc2lvbiBhY2NvbXBsaXNoZWQhCgojIyBEZXN0cm95CgpUbyByZW1vdmUgdGhlIGVudGlyZSBkZXBsb3ltZW50IHN0YWNrLApzaW1wbHkgcnVuOgoKYGBgCnNlcnZlcmxlc3MgcmVtb3ZlCmBgYAo=