Sending AWS Activity Logs via SNS in JSON Format Using CloudWatch Logs Subscription Filters and Lambda
Architecture
Introduction
In this blog post, we’ll dive into how to send AWS activity logs to your email in JSON format using Amazon SNS, Lambda, and CloudWatch Logs. Instead of relying on CloudWatch Logs Metric Filters, we’ll utilize Subscription Filters to match specific patterns in the logs.
This approach ensures you receive detailed information about the actual activities occurring within your AWS environment. We’ll walk you through the setup process, provide Lambda code examples, and demonstrate how to configure everything for seamless integration and efficient monitoring.
Prerequisites
1. Before we begin, ensure you have the latest version of the AWS CLI installed or updated.
Create the execution role
2. First, create an execution role that grants your Lambda function permission to access AWS resources:
aws iam create-role --role-name lambda-ex --assume-role-policy-document '{"Version": "2012-10-17","Statement": [{ "Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]}'3. Next, attach the AWSLambdaBasicExecutionRole policy to the role:
aws iam attach-role-policy --role-name lambda-ex --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole4. Next, attach an inline policy to the role to allow SNS publish. Replace the Resource value with your specific SNS ARN.
aws iam put-role-policy --role-name lambda-ex --policy-name allowSNSPublish --policy-document '{"Version": "2012-10-17","Statement": [{ "Effect": "Allow", "Action": "sns:Publish", "Resource": "arn:aws:sns:region:123456789012:Alarm"}]}'Set Up Subscription Filters with AWS Lambda
5. Create a package.json file with the following content:
{
"name": "aws-log-email",
"version": "1.0.0",
"author": "Your Name",
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/client-sns": "^3.370.0"
},
"scripts": {
"integration-test": "vitest run **/*.integration.test.js"
},
"type": "module",
"devDependencies": {
"@aws-sdk/client-sqs": "^3.382.0",
"vitest": "^1.6.0"
}
}6. Create a file named snsClient.js to import the AWS SNS SDK:
import { SNSClient } from "@aws-sdk/client-sns";
// The AWS Region can be provided here using the `region` property. If you leave it blank
// the SDK will default to the region set in your AWS config.
export const snsClient = new SNSClient({});7. Create a file named index.js and update the topicArn value:
import { PublishCommand } from "@aws-sdk/client-sns";
import { snsClient } from "./snsClient.js";
import zlib from 'zlib';
export const handler = function (input, context) {
let payload = Buffer.from(input.awslogs.data, 'base64');
zlib.gunzip(payload, function (e, result) {
if (e) {
console.error("Error unzipping payload", e);
context.fail(e);
} else {
try {
result = JSON.parse(result.toString());
console.log("Event Data:", JSON.stringify(result, null, 2));
if (result.logEvents && result.logEvents.length > 0) {
const message = JSON.stringify(JSON.parse(result.logEvents[0].message), null, 4);
const topicArn = "arn:aws:sns:ap-south-1:123456789012:Alarm";
publish(message, topicArn)
.then(response => {
console.log(`Message sent to the topic ${topicArn}`);
console.log("Response:", response);
context.succeed();
})
.catch(err => {
console.error("Error publishing to SNS", err);
context.fail(err);
});
} else {
console.log("No log events found");
context.succeed();
}
} catch (err) {
console.error("Error parsing JSON", err);
context.fail(err);
}
}
});
};
export const publish = async (
message = "Hello from SNS!",
topicArn = "TOPIC_ARN",
) => {
try {
const response = await snsClient.send(
new PublishCommand({
Message: message,
TopicArn: topicArn,
})
);
console.log(response);
return response;
} catch (error) {
console.error("Error sending SNS message", error);
throw error;
}
};8. Zip the above files to be used in Lambda:
zip function.zip package.json snsClient.js index.jsCreate a lambda function
9. Create a Lambda function (update the account number in the role ARN):
aws lambda create-function \
--function-name sendJSONEmail \
--zip-file fileb://function.zip \
--role arn:aws:iam::123456789012:role/lambda-ex \
--handler index.handler \
--runtime nodejs20.x10. Grant CloudWatch Logs permission to invoke your Lambda function (update source-arn with the log group name):
aws lambda add-permission \
--function-name "sendJSONEmail" \
--statement-id "sendJSONEmail" \
--principal "logs.amazonaws.com" \
--action "lambda:InvokeFunction" \
--source-arn "arn:aws:logs:region:123456789012:log-group:TestLambda:*" \
--source-account "123456789012"Create a Subscription Filter
11. Create a subscription filter to process the log group (replace placeholders with your account and log group details):
aws logs put-subscription-filter \
--log-group-name LogGroupName \
--filter-name demo \
--filter-pattern "" \
--destination-arn arn:aws:lambda:region:123456789012:function:sendJSONEmailBy following these steps, you can set up an efficient system to monitor and receive detailed AWS activity logs via email in JSON format.
