Introduction

AWS and Elastic have had a complicated relationship since Elastic's inception, and in the last several years things have only heated up. In 2019, Elastic sued AWS for trademark infringement over the Elasticsearch Service currently offered by Amazon. With all of this complexity surrounding the two companies, no developer can be blamed for being confused by exactly how to use both tools and more importantly, how to directly integrate application logs from AWS with the storage service provided by Elastic Cloud.

Here at NextLink Labs, we enjoy using AWS for most of our DevOps needs, but we still prefer hosting logs in Elastic Cloud especially with how sophisticated Elastic SIEM has become. Unfortunately, bringing logs from AWS into Elastic Cloud isn’t trivial, and it involves more work than you might expect using two hugely popular tools.

Why are logs important?

Software developers know that logs are important to help with tracking production bugs and monitoring past user activity. Application logs are also critically important for compliance frameworks, and the timescale for retaining logs for these frameworks is years. For example, HIPAA the compliance framework for healthcare applications requires six years of application logs and PCI DSS the framework for payment data requires one year of logs.

The national focus on cybersecurity that we’re seeing has put a renewed importance on these compliance frameworks, and we believe all companies should be moving towards DevSecOps maturity by maintaining application logs for at least 12 months.

When retaining logs for this amount of time, it's critical to use a service like Elastic Cloud for both organizational reasons and to keep costs down by using technologies like cold and frozen storage. Our preferred method for importing logs into Elastic Cloud is using an AWS Lambda, and we’ll describe each step involved with this process below.

Integrating CloudWatch with Elastic Cloud

1. Setting up Elastic Cloud

Once you have created an account with Elastic Cloud, you'll be prompted to create the deployment itself. We recommend creating the deployment wherever your other infrastructure is located, so for this tutorial we set up our deployment inside AWS.

After creating the deployment, you'll be provided a username and password for your elastic user. Save this information in a place you can access it later. The other two pieces of information you will need are your endpoint URL and port number. This information is retrievable from the homepage of your deployment by clicking on the "Copy Endpoint" button for your Elasticsearch application. If your deployment is hosted in AWS, the endpoint URL should have this format foobar.us-east-1.aws.found.io and the port number should be 9243.

2. Setting up CloudWatch Log Group

With the initial Elastic Cloud set up, you should verify that your application logs are coming in to CloudWatch correctly. The AWS Lambda we create in the next step will use the CloudWatch log group name by default, so ensure your log group names are valid Elastic Cloud index names and that the log group names align with what types of logging they contain.

Additionally, the AWS Lambda we setup will only import logs created after it's created. If you have a large number of historic logs in CloudWatch that you would like to import into Elastic Cloud, we would recommend creating an export task in CloudWatch and manually uploading those past logs into Elastic Cloud.

3. Creating the AWS Lambda Function

Now with Elastic Cloud and CloudWatch correctly set up, we can create the Lambda function itself.

The function we use is forked from an older template from Sebastian Tschan. The full function can be found here. After reviewing the code, copy and paste everything into a new Lambda function. Then add each of the four environment variables you stored when setting up Elastic Cloud username, password, hostname, and port number.

If the environment variables have been configured correctly, the function should send logs into Elastic Cloud now. You can test at this page by either hardcoding an example payload into the function or by setting up a test event through AWS’s testing setup. After you have fired your test event, you can check if the logs were stored correctly in Elastic Cloud by checking Kibana Discover and confirming the log data is visible at the timestamp from the log payload.

4. Attaching a Trigger to the Lambda Function

Finally, with the Lambda function written and confirmed to be sending logs to Elastic Cloud we can attach the automated trigger. For most simple applications, we attach the CloudWatch log group directly to the Lambda function as the trigger. This will fire the Lambda every time that logs are created by the application and send each one individually to Elastic Cloud. With this trigger set up, you can attach multiple CloudWatch log groups to the same Lambda, and they will all be filterable inside Elastic Cloud using log group name.

If you have a more popular app with a large amount of logs being created per minute, you should consider using a more sophisticated method to send logs over to Elastic Cloud. The next level up would be a tool like AWS SQS which can group a large number of logs at a time, parse all of them, and then send them over to Elastic Cloud as a group. If even SQS is not able to handle your data transfer, then consider using a tool like Amazon Kinesis Data Firehose instead of Lamda altogether.

Conclusion

Application logging and storage is only becoming more important for cybersecurity and auditing purposes. For these use cases, a sophisticated outside tool like Elastic Cloud is all but required to ensure your logs can still be searched efficiently while keeping costs low with cheap storage tiers and precise lifecycle policies.

This guide will give you the start you need to integrate your existing AWS CloudWatch logs with Elastic Cloud. If you have a more sophisticated logging set up that you need help to get started or if you're just looking for DevSecOps advice, please reach out to us and we can provide the assistance you need.