Collecting Amazon CloudFront logs using the Lambda Function

CloudFront logs contain information about every user request that CloudFront receives, helping you to identify usage patterns across the web properties served by Cloudfront. You can now collect your Cloudfront logs stored in S3 buckets and send them to Site24x7 for monitoring via the Lambda Function. This document will discuss how to configure your Site24x7 account to enable log monitoring for Cloudfront logs. Learn more about log management with Site24x7.

Create a Log Profile

To collect the CloudFront logs you will first need to create a Log Profile. Navigate to Admin > AppLogs > Log Profile > Add Log Profile, and follow the instructions below:

1. Profile Name: Enter a name for your Log Profile.
2. Log Type: Choose CloudFront logs. If you haven't enabled CloudFront logs in your AWS account, please follow the instructions given here.
3. Log Source: Choose Amazon Lambda.
4. Timezone: Select a timezone for your logs.
5. Click Save.
6. Configure the Lambda function as described here.

AWS setup

1. Get the Lambda code

Use this link to obtain the code required for the Lambda Function:

https://github.com/site24x7/applogs-aws-lambda/blob/master/s3/s3-sender.py

2. Configure the Lambda Function

• Choose Lambda from the Services drop-down list, and choose Create Function. Select Author from scratch, define a name for the function, and choose Python 3.7 as the Runtime. 

• Permissions: You can choose an existing IAM role or create a new role from the AWS Policy Template. From the Policy Template drop-down select Amazon S3 Object Read-only permission, and enter a role name. You also have the option to create a new user role and extend permission to other services as well.
• Add triggers: Scroll down to choose S3 Bucket. Any log file added to the S3 bucket will be sent to Site24x7 by the Lambda Function. 
  
• Configure Triggers
  • Bucket:Enter the name of the S3 bucket from which logs will be collected.
  • Event type: Choose All object create events.
  • Click Add.
• In the window that opens, click on the Lambda Function as shown:
  
• Scroll to the editor, and place the code provided in the link below:

https://github.com/site24x7/applogs-aws-lambda/blob/master/s3/s3-sender.py

• After entering the code, navigate to the Site24x7 web client, select Admin > Applogs > Log Profile, then select the created Log Profile, and copy the code that appears on the screen as the input for the variable logTypeConfig. 
  
• Paste this code under Environment Variable with the field name logTypeConfig in the AWS console. 

Configure CloudFront logging

From the Services drop-down, choose CloudFront, select the required CloudFront Distribution, and choose Distribution Settings. Click on Edit, enable Logging, and choose the name of the S3 bucket to collect the logs from.

CloudFront logs dashboard

AppLogs creates an exclusive dashboard for every Log Type, and shows a few widgets by default. Here's a list of the widgets available in the CloudFront logs dashboard:

• Total Requests
• Average Response Time
• Average Bytes Send
• Bytes Received
• Failed Requests
• Top 20 Failed Requests
• User Agent Stats
• Request Trend
• Status Code Stats
• Response Time Stats
• Top 50 Successful Requests
• Failed Requests by Source (User Agent)

Related log types

• S3 logs
• CloudTrail logs
• VPC Flow logs