This blog has explained the following concepts for serverless applications so far:

The third blog in serverless series will explain how to create a simple microservice using Amazon API Gateway, AWS Lambda and Couchbase.

Read previous blogs for more context on AWS Lambda.

Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. Amazon API Gateway handles all the tasks involved in accepting and processing up to hundreds of thousands of concurrent API calls, including traffic management, authorization and access control, monitoring, and API version management.

Here are the key components in this architecture:

serverless-microservice

  • The client could be curl, AWS CLI, Postman client or any other tool/API that can invoke a REST endpoint.
  • API Gateway is used to provision APIs. The top level resource is available at path /books. HTTP GET and POST methods are published for the resource.
  • Each API triggers a Lambda function. Two Lambda functions are created, book-list function for listing all the books available and book-create function to create a new book.
  • Couchbase is used as a persistence store in EC2. All the JSON documents are stored and retrieved from this database.

Let’s get started!

Create IAM Role

IAM roles will have policies and trust relationships that will allow this role to be used in API Gateway and execute Lambda function.

Let’s create a new IAM role:

aws iam create-role 
--role-name microserviceRole 
--assume-role-policy-document file://./trust.json

--assume-role-policy-document defines the trust relationship policy document that grants an entity permission to assume the role. trust.json is at github.com/arun-gupta/serverless/blob/master/aws/microservice/trust.json and looks like:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "lambda.amazonaws.com",
          "apigateway.amazonaws.com"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

This trust relationship allows Lambda functions and API Gateway to assume this role during execution.

Associate policies with this role as:

aws iam put-role-policy 
--role-name microserviceRole 
--policy-name microPolicy 
--policy-document file://./policy.json

policy.json is at github.com/arun-gupta/serverless/blob/master/aws/microservice/policy.json and looks like:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:*"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "apigateway:*"
      ],
      "Resource": "arn:aws:apigateway:*::/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "execute-api:Invoke"
      ],
      "Resource": "arn:aws:execute-api:*:*:*"
    },
    {
      "Effect": "Allow",
      "Action": [
          "lambda:*"
      ],
      "Resource": "*"
    }
  ]
}

This generous policy allows any permissions over logs generated in CloudWatch for all resources. In addition it allows all Lambda and API Gateway permissions to all resources. In general, only the required policy would be given to specific resources.

Create Lambda Functions

Detailed steps to create Lambda functions are explained in Serverless FaaS with AWS Lambda and Java. Let’s create the two Lambda functions as required in our case:

aws lambda create-function 
--function-name MicroserviceGetAll 
--role arn:aws:iam::598307997273:role/microserviceRole 
--handler org.sample.serverless.aws.couchbase.BucketGetAll 
--zip-file fileb:///Users/arungupta/workspaces/serverless/aws/microservice/microservice-http-endpoint/target/microservice-http-endpoint-1.0-SNAPSHOT.jar 
--description "Microservice HTTP Endpoint - Get All" 
--runtime java8 
--region us-west-1 
--timeout 30 
--memory-size 1024 
--environment Variables={COUCHBASE_HOST=ec2-52-53-193-176.us-west-1.compute.amazonaws.com} 
--publish

Couple of key items to note in this function are:

  • IAM role microserviceRole created in previous step is explicitly specified here
  • Handler is org.sample.serverless.aws.couchbase.BucketGetAll class. This class queries the Couchbase database defined using the COUCHBASE_HOST environment variable.

Create the second Lambda function:

aws lambda create-function 
--function-name MicroservicePost 
--role arn:aws:iam::598307997273:role/microserviceRole 
--handler org.sample.serverless.aws.couchbase.BucketPost 
--zip-file fileb:///Users/arungupta/workspaces/serverless/aws/microservice/microservice-http-endpoint/target/microservice-http-endpoint-1.0-SNAPSHOT.jar 
--description "Microservice HTTP Endpoint - Post" 
--runtime java8 
--region us-west-1 
--timeout 30 
--memory-size 1024 
--environment Variables={COUCHBASE_HOST=ec2-52-53-193-176.us-west-1.compute.amazonaws.com} 
--publish

The handler for this function is org.sample.serverless.aws.couchbase.BucketPost class. This class creates a new JSON document in the Couchbase database identified by COUCHBASE_HOST environment variable.

The complete source code for these classes is at github.com/arun-gupta/serverless/tree/master/aws/microservice/microservice-http-endpoint.

API Gateway Resource

Create an API using Amazon API Gateway and Test It and Build an API to Expose a Lambda Function provide detailed steps and explanation on how to use API Gateway and Lambda Functions to build powerful backend systems. This blog will do a quick run down of the steps in case you want to cut the chase.

Let’s create API Gateway resources.

  1. The first step is to create an API:
    aws apigateway 
    create-rest-api 
    --name Book

    This shows the output as:

    {
        "name": "Book", 
        "id": "lb2qgujjif", 
        "createdDate": 1482998945
    }

    The value of id attribute is API ID. In our case, this is lb2qgujjif.

  2. Find ROOT ID of the created API as this is required for the next AWS CLI invocation:
    aws apigateway get-resources --rest-api-id lb2qgujjif

    This shows the output:

    {
        "items": [
            {
                "path": "/", 
                "id": "hgxogdkheg"
            }
        ]
    }

    Value of id attribute is ROOT ID. This is also the PARENT ID for the top level resource.

  3. Create a resource
    aws apigateway create-resource 
    --rest-api-id lb2qgujjif 
    --parent-id hgxogdkheg 
    --path-part books

    This shows the output:

    {
        "path": "/books", 
        "pathPart": "books", 
        "id": "vrpkod", 
        "parentId": "hgxogdkheg"
    }

    Value of id attribute is RESOURCE ID.

API ID and RESOURCE ID are used for subsequent AWS CLI invocations.

API Gateway POST Method

Now that the resource is created, let’s create HTTP POST method on this resource.

  1. Create a POST method

    aws apigateway put-method 
    --rest-api-id lb2qgujjif 
    --resource-id vrpkod 
    --http-method POST 
    --authorization-type NONE

    to see the response:

    {
        "apiKeyRequired": false, 
        "httpMethod": "POST", 
        "authorizationType": "NONE"
    }
  2. Set the Lambda function as the destination of the POST method:

    aws apigateway put-integration 
    --rest-api-id lb2qgujjif 
    --resource-id vrpkod 
    --http-method POST 
    --type AWS 
    --integration-http-method POST 
    --uri arn:aws:apigateway:us-west-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-1:<act-id>:function:MicroservicePost/invocations

    Make sure to replace <act-id> with your AWS account id. API ID and RESOURCE ID from previous section are used here as well. --uri is used to specify the URI of integration input. The format of the URI is fixed. This CLI will show the result as:

    {
        "httpMethod": "POST", 
        "passthroughBehavior": "WHEN_NO_MATCH", 
        "cacheKeyParameters": [], 
        "type": "AWS", 
        "uri": "arn:aws:apigateway:us-west-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-1:<act-id>:function:MicroservicePost/invocations", 
        "cacheNamespace": "vrpkod"
    }
  3. Set content-type of POST method response:

    aws apigateway put-method-response 
    --rest-api-id lb2qgujjif 
    --resource-id vrpkod 
    --http-method POST 
    --status-code 200 
    --response-models "{"application/json": "Empty"}"

    to see the response:

    {
        "responseModels": {
            "application/json": "Empty"
        }, 
        "statusCode": "200"
    }
  4. Set content-type of POST method integration response:

    aws apigateway put-integration-response 
    --rest-api-id lb2qgujjif 
    --resource-id vrpkod 
    --http-method POST 
    --status-code 200 
    --response-templates "{"application/json": "Empty"}"

    to see the response:

    {
        "statusCode": "200", 
        "responseTemplates": {
            "application/json": "Empty"
        }
    }
  5. Deploy the API

    aws apigateway create-deployment 
    --rest-api-id lb2qgujjif 
    --stage-name test

    to see the response

    {
        "id": "9wi991", 
        "createdDate": 1482999187
    }
  6. Grant permission to allow API Gateway to invoke Lambda Function:

    aws lambda add-permission 
    --function-name MicroservicePost 
    --statement-id apigateway-test-post-1 
    --action lambda:InvokeFunction 
    --principal apigateway.amazonaws.com 
    --source-arn "arn:aws:execute-api:us-west-1:<act-id>:lb2qgujjif/*/POST/books"

    Also, grant permission to the deployed API:

    aws lambda add-permission 
    --function-name MicroservicePost 
    --statement-id apigateway-test-post-2 
    --action lambda:InvokeFunction 
    --principal apigateway.amazonaws.com 
    --source-arn "arn:aws:execute-api:us-west-1:<act-id>:lb2qgujjif/test/GET/books"
  7. Test the API method:

    aws apigateway test-invoke-method 
    --rest-api-id lb2qgujjif 
    --resource-id vrpkod 
    --http-method POST 
    --path-with-query-string "" 
    --body "{"id": "1", "bookname": "test book", "isbn": "123", "cost": "1.23"}"

    to see the response:

    {
        "status": 200, 
        "body": "Empty", 
        "log": "Execution log for request test-requestnThu Dec 29 08:16:05 UTC 2016 : Starting execution for request: test-invoke-requestnThu Dec 29 08:16:05 UTC 2016 : HTTP Method: POST, Resource Path: /booksnThu Dec 29 08:16:05 UTC 2016 : Method request path: {}nThu Dec 29 08:16:05 UTC 2016 : Method request query string: {}nThu Dec 29 08:16:05 UTC 2016 : Method request headers: {}nThu Dec 29 08:16:05 UTC 2016 : Method request body before transformations: {"id": "1", "bookname": "test book", "isbn": "123", "cost": "1.23"}nThu Dec 29 08:16:05 UTC 2016 : Endpoint request URI: https://lambda.us-west-1.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-west-1:598307997273:function:MicroservicePost/invocationsnThu Dec 29 08:16:05 UTC 2016 : Endpoint request headers: {x-amzn-lambda-integration-tag=test-request, Authorization=****************************************************************************************************************************************************************************************************************************************************************************************************************************************c8bb85, X-Amz-Date=20161229T081605Z, x-amzn-apigateway-api-id=lb2qgujjif, X-Amz-Source-Arn=arn:aws:execute-api:us-west-1:598307997273:lb2qgujjif/null/POST/books, Accept=application/json, User-Agent=AmazonAPIGateway_lb2qgujjif, Host=lambda.us-west-1.amazonaws.com, X-Amz-Content-Sha256=559d0296d96ec5647eef6381602fe5e7f55dd17065864fafb4f581d106aa92f4, X-Amzn-Trace-Id=Root=1-5864c645-8494974a41a3a16c8d2f9929, Content-Type=application/json}nThu Dec 29 08:16:05 UTC 2016 : Endpoint request body after transformations: {"id": "1", "bookname": "test book", "isbn": "123", "cost": "1.23"}nThu Dec 29 08:16:10 UTC 2016 : Endpoint response body before transformations: "{"cost":"1.23","id":"1","bookname":"test book","isbn":"123"}"nThu Dec 29 08:16:10 UTC 2016 : Endpoint response headers: {x-amzn-Remapped-Content-Length=0, x-amzn-RequestId=0b25323b-cd9f-11e6-8bd4-292925ba63a9, Connection=keep-alive, Content-Length=78, Date=Thu, 29 Dec 2016 08:16:10 GMT, Content-Type=application/json}nThu Dec 29 08:16:10 UTC 2016 : Method response body after transformations: EmptynThu Dec 29 08:16:10 UTC 2016 : Method response headers: {X-Amzn-Trace-Id=Root=1-5864c645-8494974a41a3a16c8d2f9929, Content-Type=application/json}nThu Dec 29 08:16:10 UTC 2016 : Successfully completed executionnThu Dec 29 08:16:10 UTC 2016 : Method completed with status: 200n", 
        "latency": 5091, 
        "headers": {
            "X-Amzn-Trace-Id": "Root=1-5864c645-8494974a41a3a16c8d2f9929", 
            "Content-Type": "application/json"
        }
    }

    Value of status attribute is 200 and indicates this was a successful invocation. Value of log attribute shows the log statement from CloudWatch Logs. Detailed logs can also be obtained using aws logs filter-log-events --log-group /aws/lambda/MicroservicePost.

    This command stores a single JSON document in Couchbase. This can be easily verified using the Couchbase CLI Tool cbq.

  8. Connect to the Couchbase server as:

    cbq -u Administrator -p password -e="http://<COUCHBASE_HOST>:8091"

    Create a primary index on default bucket as this is required to query the bucket with no clauses:

    cbq> create primary index default_index on default;
    {
        "requestID": "13b539f9-7fff-4386-92f4-cea161a7aa08",
        "signature": null,
        "results": [
        ],
        "status": "success",
        "metrics": {
            "elapsedTime": "1.917009047s",
            "executionTime": "1.916970061s",
            "resultCount": 0,
            "resultSize": 0
        }
    }
  9. Write a N1QL query to access the data:

    cbq> select * from default limit 10;
    {
        "requestID": "d7b1c3f9-6b4e-4952-9a1e-9faf5169926e",
        "signature": {
            "*": "*"
        },
        "results": [
            {
                "default": {
                    "bookname": "test",
                    "cost": "1.23",
                    "id": "1",
                    "isbn": "123"
                }
            }
        ],
        "status": "success",
        "metrics": {
            "elapsedTime": "24.337755ms",
            "executionTime": "24.289796ms",
            "resultCount": 1,
            "resultSize": 175
        }
    }

    The results show the JSON document that was stored by our Lambda function.

API Gateway GET Method

Let’s create HTTP GET method on the resource:

  1. Create a GET method:

    aws apigateway put-method 
    --rest-api-id lb2qgujjif 
    --resource-id vrpkod 
    --http-method GET 
    --authorization-type NONE
  2. Set correct Lambda function as destination of GET:

    aws apigateway put-integration 
    --rest-api-id lb2qgujjif 
    --resource-id vrpkod 
    --http-method GET 
    --type AWS 
    --integration-http-method POST 
    --uri arn:aws:apigateway:us-west-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-1:598307997273:function:MicroserviceGetAll/invocations
  3. Set content-type of GET method response:

    aws apigateway put-method-response 
    --rest-api-id lb2qgujjif 
    --resource-id vrpkod 
    --http-method GET 
    --status-code 200 
    --response-models "{"application/json": "Empty"}"
  4. Set content-type of GET method integration response:

    aws apigateway put-integration-response 
    --rest-api-id lb2qgujjif 
    --resource-id vrpkod 
    --http-method GET 
    --status-code 200 
    --response-templates "{"application/json": "Empty"}"
  5. Grant permission to allow API Gateway to invoke Lambda Function

    aws lambda add-permission 
    --function-name MicroserviceGetAll 
    --statement-id apigateway-test-getall-1 
    --action lambda:InvokeFunction 
    --principal apigateway.amazonaws.com 
    --source-arn "arn:aws:execute-api:us-west-1:598307997273:lb2qgujjif/*/GET/books"
  6. Grant permission to the deployed API:

    aws lambda add-permission 
    --function-name MicroserviceGetAll 
    --statement-id apigateway-test-getall-2 
    --action lambda:InvokeFunction 
    --principal apigateway.amazonaws.com 
    --source-arn "arn:aws:execute-api:us-west-1:598307997273:lb2qgujjif/test/GET/books"
  7. Test the method:

    aws apigateway test-invoke-method 
    --rest-api-id lb2qgujjif 
    --resource-id vrpkod 
    --http-method GET

    to see the output:

    {
        "status": 200, 
        "body": "Empty", 
        "log": "Execution log for request test-requestnSat Dec 31 09:07:48 UTC 2016 : Starting execution for request: test-invoke-requestnSat Dec 31 09:07:48 UTC 2016 : HTTP Method: GET, Resource Path: /booksnSat Dec 31 09:07:48 UTC 2016 : Method request path: {}nSat Dec 31 09:07:48 UTC 2016 : Method request query string: {}nSat Dec 31 09:07:48 UTC 2016 : Method request headers: {}nSat Dec 31 09:07:48 UTC 2016 : Method request body before transformations: nSat Dec 31 09:07:48 UTC 2016 : Endpoint request URI: https://lambda.us-west-1.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-west-1:598307997273:function:MicroserviceGetAll/invocationsnSat Dec 31 09:07:48 UTC 2016 : Endpoint request headers: {x-amzn-lambda-integration-tag=test-request, Authorization=******************************************************************************************************************************************************************************************************************************************************************************************************6de147, X-Amz-Date=20161231T090748Z, x-amzn-apigateway-api-id=lb2qgujjif, X-Amz-Source-Arn=arn:aws:execute-api:us-west-1:598307997273:lb2qgujjif/null/GET/books, Accept=application/json, User-Agent=AmazonAPIGateway_lb2qgujjif, X-Amz-Security-Token=FQoDYXdzEHEaDEILpsKTo45Ys1LrFCK3A+KOe5HXOSP3GfVAaRYHe1pDUJGHL9MtkFiPjORLFT+UCKjRqE7UFaGscTVG6PZXTuSyQev4XTyROfPylCrtDomGsoZF/iwy4rlJQIJ7elBceyeKu1OVdaT1A99PVeliaCAiDL6Veo1viWOnP+7c72nAaJ5jnyF/nHl/OLhFdFv4t/hnx3JePMk5YM89/6ofxUEVDNfzXxbZHRpTrG/4TPHwjPdoR5i9dEzWMU6Eo5xD4ldQ/m5B3RmrwpaPOuEq39LhJ8k/Vzo+pAfgJTq5ssbNwYOgh0RPSGVNMcoTkCwk0EMMT5vDbmQqZ2dW1a1tmQg9N2xR+QQy+RKMFgO5YY8fMxHnRSdMuuipxl79G1pktc [TRUNCATED]nSat Dec 31 09:07:48 UTC 2016 : Endpoint request body after transformations: nSat Dec 31 09:07:53 UTC 2016 : Endpoint response body before transformations: "[{"default":{"cost":"1.23","id":"1","bookname":"test book","isbn":"123"}}]"nSat Dec 31 09:07:53 UTC 2016 : Endpoint response headers: {x-amzn-Remapped-Content-Length=0, x-amzn-RequestId=99ab09b2-cf38-11e6-996f-f5f07af431af, Connection=keep-alive, Content-Length=94, Date=Sat, 31 Dec 2016 09:07:52 GMT, Content-Type=application/json}nSat Dec 31 09:07:53 UTC 2016 : Method response body after transformations: EmptynSat Dec 31 09:07:53 UTC 2016 : Method response headers: {X-Amzn-Trace-Id=Root=1-58677564-66f1e96642b16d2db703126e, Content-Type=application/json}nSat Dec 31 09:07:53 UTC 2016 : Successfully completed executionnSat Dec 31 09:07:53 UTC 2016 : Method completed with status: 200n", 
        "latency": 4744, 
        "headers": {
            "X-Amzn-Trace-Id": "Root=1-58677564-66f1e96642b16d2db703126e", 
            "Content-Type": "application/json"
        }
    }

    Once again, 200 status code shows a successful invocation. Detailed logs can be obtained using aws logs filter-log-events --log-group /aws/lambda/MicroservicePost.

This blog only shows one simple POST and GET methods. Other HTTP methods can be very easily included in this microservice as well.

API Gateway and Lambda References

Microservice using AWS API Gateway, AWS Lambda and Couchbase

Profile photo of Arun Gupta
About The Author
- Arun Gupta is the vice president of developer advocacy at Couchbase. He has been building developer communities for 10+ years at Sun, Oracle, and Red Hat. He has deep expertise in leading cross-functional teams to develop and execute strategy, planning and execution of content, marketing campaigns, and programs. Prior to that he led engineering teams at Sun and is a founding member of the Java EE team. Gupta has authored more than 2,000 blog posts on technology. He has extensive speaking experience in more than 40 countries on myriad topics and is a JavaOne Rock Star. Gupta also founded the Devoxx4Kids chapter in the US and continues to promote technology education among children. An author of a best-selling book, an avid runner, a globe trotter, a Java Champion, and a JUG leader, he is easily accessible at @arungupta.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>