Building a Development Environment with docker-compose — Visualizing Nginx Access Logs (LTSV) with Fluentd + Elasticsearch + Kibana

Overview

We’ll build a development environment with docker-compose on top of the Vagrant environment we set up previously.

Vagrant (Ubuntu) に Docker, Docker Compose インストール - 長生村本郷Engineers'Blog

以下に移行しました。 kenzo0107.github.io

Continuing from last time, we’ll visualize Nginx access logs with Elasticsearch + Fluentd + Kibana.
App

Quick Setup Steps

macOS% $ git clone https://github.com/kenzo0107/vagrant-docker
macOS% $ cd vagrant-docker
macOS% $ vagrant up
macOS% $ vagrant ssh
vagrant% $ cd /vagrant/nginx-efk

// We don't use -d (detached mode) so that the startup status of each container is visible in the logs.
vagrant% $ docker-compose up
...
...

docker-compose Configuration

Everything is collected on Git.

vagrant-docker/docker/nginx-efk at master · kenzo0107/vagrant-docker

Docker on Vagrant(ubuntu). Contribute to kenzo0107/vagrant-docker development by creating an account on GitHub.

├── docker-compose.yml
├── fluentd
│   ├── conf
│   │   ├── conf.d
│   │   │   └── nginx.log.conf
│   │   └── fluent.conf
│   └── Dockerfile
└── nginx
    └── conf
        └── nginx.conf

Key Points

• The Nginx log directory is set as a volume and synced with the host side. On the fluentd side it’s also set as a volume and tailed.

Here’s the overall picture.

Verifying Nginx Startup from the Browser

When you access http://192.168.35.101/ from your browser,
you’ll see the Nginx Welcome page.

After the earlier docker-compose up, you’ll see logs like the following,
which show that fluentd is capturing the Nginx access logs.

Accessing Kibana

When you access http://192.168.35.101:5601 from your browser,
the Kibana page is displayed.

1. Index name or pattern

   • Specify fluentd-*

2. Time-field name

   • Specify @timestamp

3. Click the Create button

4. Click Discover from the left menu

Checking Logs from macOS

Naturally, since macOS and vagrant are synced,
you can tail the logs from macOS as well.

macOS%$ tail -f <path/to/vagrant-docker>/docker/nginx-efk/_log/nginx/access.log

That’s all.
I hope you find it helpful.