Downloading AWS LB Log Files Output to S3 Every 5 Minutes in Bulk by Time Range

Overview

When you have configured LB logs to be stored in S3 on AWS, and you want to collect all the logs from the time range when an incident occurred, I use awscli to retrieve the logs in bulk.

I've summarized the procedure for that as a memo.

Prerequisites

• Install awscli

The execution environment for this case is as follows.

macOS%$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.12.6
BuildVersion:   16G1036

macOS%$ aws --version
aws-cli/1.15.50 Python/3.7.0 Darwin/16.7.0 botocore/1.10.49

Downloading the Log Files

Example) Download the ALB log files for the 2 PM hour on September 12, 2018

// Download the log files for the 2 PM hour on September 12, 2018
macOS%$ aws s3 --profile <profile> cp s3://<log bucket name>/<lb name>/AWSLogs/123456789012/elasticloadbalancing/ap-northeast-1/2018/09/12/ . --recursive --exclude "*" --include "*20180912T05*"

// Decompress the log files
gunzip *.gz

// Combine the logs into a single file
cat *.log > all.log

// Collect entries with an HTTP code of 50x into 50x.log
awk '{if($9 ~ 50) print $0}' all.log > 50x.log

`--profile <profile>` is the setting used when you have configured it with `aws configure --profile <profile>`.
If you use the default, specifying `--profile <profile>` is not necessary.

That's all.