Sample AWS Kinesis Firehose CloudWatch Log HTTP Endpoint Payload

I couldn’t find an example payload for CloudWatch Logs data delivered by a Kinesis Firehose stream to a Lambda function URL, so I set one up.

CloudWatch Logs to Kinesis Firehose Delivery Stream to AWS Lambda

Sample payloads appear below, as well as a walkthrough of the process I used to create the data stream.

Use Case

I need to deliver API usage data to Stripe for billing, so I want to subscribe a Lambda function to a CloudWatch Logs log group containing access logs. The relevant API endpoint is rate limited, so rather than subscribe the Lambda function directly to the CloudWatch Logs log group, I want to go through a Kinesis Firehose Delivery Stream for explicit batching to manage that rate limit.

CloudWatch Logs Setup

The CloudWatch Logs log group in question contains access logs from API Gateway using the following template:

{
  "requestId": "$context.requestId",
  "extendedRequestId": "$context.extendedRequestId",
  "ip": "$context.identity.sourceIp",
  "caller": "$context.identity.caller",
  "user": "$context.identity.user",
  "requestTime": "$context.requestTime",
  "httpMethod": "$context.httpMethod",
  "resourcePath": "$context.resourcePath",
  "status": "$context.status",
  "protocol": "$context.protocol",
  "responseLength": "$context.responseLength"
}

Kinesis Firehose Setup

The Kinesis Firehose has the following configuration:

Record Transformation — Off
Destination — HTTP Endpoint
HTTP Endpoint URL — The Lambda function URL of the below Lambda function
Content Encoding — Disabled
Buffer Hints — Buffer size 1 MiB, Buffer interval 60 seconds
Parameters — 1 parameter: alpha=bravo
Access Key — foxtrot

Lambda Function Setup

The Lambda function is configured with the Python 3.12 runtime and the following source code:

from json import loads, dumps
from time import time
from math import floor
from datetime import datetime

def lambda_handler(event, context):
    print(dumps(event))
    
    body = loads(event["body"])
    now = datetime.utcfromtimestamp(floor(time()))

    return {
        'statusCode': 200,
        'body':dumps({
            'requestId': body["requestId"],
            'timestamp': floor(now.timestamp())
        })
    }

CloudWatch Logs Subscription Filter

The Subscription Filter that connects the CloudWatch Logs log group to the Kinesis Firehose has the following configuration:

Destination — Current Account, with the above Kinesis Firehose delivery stream selected
Grant Permission — A bespoke role, given below
Log format — JSON, with an empty Subscription filter pattern and the name “unfiltered”

The bespoke IAM role includes the AmazonKinesisFirehoseFullAccess policy, which has the following definition:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "firehose:*"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

Of course, this particular configuration is unsuitable for production use, but is fine for this demo use case.

Example Lambda Function URL Payload

Using the above configuration, the Lambda function prints the following payload to CloudWatch logs, lightly modified for privacy:

{
  "version": "2.0",
  "routeKey": "$default",
  "rawPath": "/",
  "rawQueryString": "",
  "headers": {
    "content-length": "2594",
    "x-amzn-tls-version": "TLSv1.2",
    "x-forwarded-proto": "https",
    "x-amz-firehose-common-attributes": "{\"commonAttributes\":{\"alpha\":\"bravo\"}}",
    "x-forwarded-port": "443",
    "x-forwarded-for": "3.14.83.175",
    "x-amzn-tls-cipher-suite": "ECDHE-RSA-AES128-GCM-SHA256",
    "x-amz-firehose-request-id": "0f9318bc-b9c0-4074-bfa5-8ea3715efedc",
    "x-amzn-trace-id": "Root=1-65958505-5b3afda535f32dae553a4a58",
    "x-amz-firehose-access-key": "foxtrot",
    "x-amz-firehose-protocol-version": "1.0",
    "host": "kmiys23k26f7ukrc5ke5kq4uxxxxxxxx.lambda-url.us-east-2.on.aws",
    "content-type": "application/json",
    "x-amz-firehose-source-arn": "arn:aws:firehose:us-east-2:123456789012:deliverystream/deleteme-logs-test",
    "user-agent": "Amazon Kinesis Data Firehose Agent/1.0"
  },
  "requestContext": {
    "accountId": "anonymous",
    "apiId": "kmiys23k26f7ukrc5ke5kq4uxxxxxxxx",
    "domainName": "kmiys23k26f7ukrc5ke5kq4uxxxxxxxx.lambda-url.us-east-2.on.aws",
    "domainPrefix": "kmiys23k26f7ukrc5ke5kq4uxxxxxxxx",
    "http": {
      "method": "POST",
      "path": "/",
      "protocol": "HTTP/1.1",
      "sourceIp": "3.14.83.175",
      "userAgent": "Amazon Kinesis Data Firehose Agent/1.0"
    },
    "requestId": "b9ac4b3a-24b7-41de-bb66-5d7e8d046e78",
    "routeKey": "$default",
    "stage": "$default",
    "time": "03/Jan/2024:16:02:13 +0000",
    "timeEpoch": 1704297733192
  },
  "body": "{\"requestId\":\"0f9318bc-b9c0-4074-bfa5-8ea3715efedc\",\"timestamp\":1704297733161,\"records\":[{\"data\":\"H4sIAB2LlWUAA1VRXW/TMBR951cgv9I0/qzjSDxUottATBSahyEyIdd220hpbBJndKr637nzUqRd+cXnnHvu1xkd3TDovaueg0Ml+rSslr/vV5vN8naFZsj/7VwPMKGMi4UsFCYU4Nbvb3s/BmByd9LH0LpchyYPvbe5NgYsM9AMr9JN7J0+gpbynbKaLISyjFNBpBViu9PiNAXIh3E7mL4JsfHdTdNG1w+o/IXGbpc+zqLH5Ll6cl18oc6osWDNCowlwVgpBb5S4YWgnBZMCSYlKQTmBcGSEVoQIrEQeAooGRtYQYQhUAkUp0oyyBV4dl0N2J9r1Ls/I+g+2xqVNbIaWlfaZAqqZVyZbbZlZpFp5XZQtFBQskazGrlTdJ119seb9O/ZjXq4y0Tz06we1PJjkjYhcYTNRTEnhM8FT7jRbev6xGUJGIc336mzCuZIKGb5F93lFFNeElEKePT9h5dpk/wQY7h38eBfW1l/21STzeDH3ri1jofE5E/k/3WvZB4SC3JYWRyHJKSTM5w/euPbBN5V1Tonc3L1Dr4b3FfX7Sd3Iqms0eUdujxe/gENzCHOhgIAAA==\"},{\"data\":\"H4sIAB2LlWUAA1VRXW/TMBR951cgv9I0/qzjSDxUottATBSahyEyIdd220hpbBJndKr637nzUqRd+cXnnHvu1xkd3TDovaueg0Ml+rSslr/vV5vN8naFZsj/7VwPMKGMi4UsFCYU4Nbvb3s/BmByd9LH0LpchyYPvbe5NgYsM9AMr9JN7J0+gpbynbKaLISyjFNBpBViu9PiNAXIh3E7mL4JsfHdTdNG1w+o/IXGbpc+zqLH5Ll6cl18oc6osWDNCowlwVgpBb5S4YWgnBZMCSYlKQTmBcGSEVoQIrEQeAooGRtYQYQhUAkUp0oyyBV4dl0N2J9r1Ls/I+g+2xqVNbIaWlfaZAqqZVyZbbZlZpFp5XZQtFBQskazGrlTdJ119seb9O/ZjXq4y0Tz06we1PJjkjYhcYTNRTEnhM8FT7jRbev6xGUJGIc336mzCuZIKGb5F93lFFNeElEKePT9h5dpk/wQY7h38eBfW1l/21STzeDH3ri1jofE5E/k/3WvZB4SC3JYWRyHJKSTM5w/euPbBN5V1Tonc3L1Dr4b3FfX7Sd3Iqms0eUdujxe/gENzCHOhgIAAA==\"}]}",
  "isBase64Encoded": false
}

Readers should take particular note of the following:

The location and format of the Kinesis Firehose parameters (HTTP request header x-amz-firehose-common-attributes: {"commonAttributes":{"alpha":"bravo"}})
The location and format of the access key (HTTP request header x-amz-firehose-access-key: foxtrot)

HTTP Request Body

The logical HTTP request body (excluding the Lambda HTTP request “envelope”) looks like this:

{
  "requestId": "0f9318bc-b9c0-4074-bfa5-8ea3715efedc",
  "timestamp": 1704297733161,
  "records": [
    {
      "data": "H4sIAB2LlWUAA1VRXW/TMBR951cgv9I0/qzjSDxUottATBSahyEyIdd220hpbBJndKr637nzUqRd+cXnnHvu1xkd3TDovaueg0Ml+rSslr/vV5vN8naFZsj/7VwPMKGMi4UsFCYU4Nbvb3s/BmByd9LH0LpchyYPvbe5NgYsM9AMr9JN7J0+gpbynbKaLISyjFNBpBViu9PiNAXIh3E7mL4JsfHdTdNG1w+o/IXGbpc+zqLH5Ll6cl18oc6osWDNCowlwVgpBb5S4YWgnBZMCSYlKQTmBcGSEVoQIrEQeAooGRtYQYQhUAkUp0oyyBV4dl0N2J9r1Ls/I+g+2xqVNbIaWlfaZAqqZVyZbbZlZpFp5XZQtFBQskazGrlTdJ119seb9O/ZjXq4y0Tz06we1PJjkjYhcYTNRTEnhM8FT7jRbev6xGUJGIc336mzCuZIKGb5F93lFFNeElEKePT9h5dpk/wQY7h38eBfW1l/21STzeDH3ri1jofE5E/k/3WvZB4SC3JYWRyHJKSTM5w/euPbBN5V1Tonc3L1Dr4b3FfX7Sd3Iqms0eUdujxe/gENzCHOhgIAAA=="
    },
    {
      "data": "H4sIAB2LlWUAA1VRXW/TMBR951cgv9I0/qzjSDxUottATBSahyEyIdd220hpbBJndKr637nzUqRd+cXnnHvu1xkd3TDovaueg0Ml+rSslr/vV5vN8naFZsj/7VwPMKGMi4UsFCYU4Nbvb3s/BmByd9LH0LpchyYPvbe5NgYsM9AMr9JN7J0+gpbynbKaLISyjFNBpBViu9PiNAXIh3E7mL4JsfHdTdNG1w+o/IXGbpc+zqLH5Ll6cl18oc6osWDNCowlwVgpBb5S4YWgnBZMCSYlKQTmBcGSEVoQIrEQeAooGRtYQYQhUAkUp0oyyBV4dl0N2J9r1Ls/I+g+2xqVNbIaWlfaZAqqZVyZbbZlZpFp5XZQtFBQskazGrlTdJ119seb9O/ZjXq4y0Tz06we1PJjkjYhcYTNRTEnhM8FT7jRbev6xGUJGIc336mzCuZIKGb5F93lFFNeElEKePT9h5dpk/wQY7h38eBfW1l/21STzeDH3ri1jofE5E/k/3WvZB4SC3JYWRyHJKSTM5w/euPbBN5V1Tonc3L1Dr4b3FfX7Sd3Iqms0eUdujxe/gENzCHOhgIAAA=="
    }
  ]
}

The individual log entries are base64-encoded GZIP data. The first entry reads like this after decoding, again lightly modified for privacy:

{
  "messageType": "DATA_MESSAGE",
  "owner": "123456789012",
  "logGroup": "/example/api/prod/access-logs",
  "logStream": "24f9da1659d342517d55bfa5xxxxxxxx",
  "subscriptionFilters": [
    "unfiltered"
  ],
  "logEvents": [
    {
      "id": "38007100999517790652428395377185048107312811705500000000",
      "timestamp": 1704297352450,
      "message": "{\"requestId\":\"da55b9ac-9177-49cb-b3c6-a9ef39589850\",\"extendedRequestId\":\"Q-F9XH-5iYcEX9A=\",\"ip\":\"13.58.114.54\",\"caller\":\"-\",\"user\":\"-\",\"requestTime\":\"03/Jan/2024:15:55:52 +0000\",\"httpMethod\":\"POST\",\"resourcePath\":\"/v1/example/resource/path\",\"status\":\"200\",\"protocol\":\"HTTP/1.1\",\"responseLength\":\"1727\"}\n"
    }
  ]
}

Readers should take particular note of the following:

The unfiltered value of the subscriptionFilter property corresponds to the name of the filter pattern given in the Subscription Filter.

Individual Log Entries

Individual log entries are formatted as follows:

Conclusion

The above gives (generally) reasonable configuration hints for connecting a CloudWatch Logs log group to a Lambda function via a Kinesis Firehose Delivery Stream and a Lambda function URL. Of course, it’s possible to connect a CloudWatch Logs log group to a Lambda function directly, but for use cases that benefit from explicit batching — for example, to manage a rate limit in a downstream API — this approach works well.

Pricing is generally reasonable, although users should be aware of the following:

Individual log entries being delivered are rounded up to the nearest 5KB for billing purposes.
Data delivery from Kinesis Firehose to Lambda function URLs go over the public internet, so incur egress costs of $0.09/GB. For access logs, this cost is generally pretty reasonable.

There is additional useful information in the Kinesis Firehose Appendices, including OpenAPI spec snippets for some of these request types.