InfrastructureAWSGoDatabaseMonitoringRaspberryPIPythonTerraformGitAIDIYData Analytics
Catalogue
Sakura VPS fluentd + elasticsearch + kibana3
Posted 2015-03-01Monitoring6 minutes read (About 859 words)

Sakura VPS fluentd + elasticsearch + kibana3

🌐 日本語で読む

Installing ElasticSearch

Official site: http://www.elasticsearch.org/blog/apt-and-yum-repositories/

Add the yum repository

1
rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch

Add the elasticsearch repository configuration file

1
2
3
4
5
6
7
8
cat >> /etc/yum.repos.d/elasticsearch.repo <<'EOF'
[elasticsearch-1.0]
name=Elasticsearch repository for 1.0.x packages
baseurl=http://packages.elasticsearch.org/elasticsearch/1.0/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1
EOF

Install java and elasticsearch

1
yum install elasticsearch java-1.7.0-openjdk

Configure the module to start automatically on server boot

1
chkconfig elasticsearch on

Start elasticsearch

1
service elasticsearch start

Operation test

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
curl -X GET http://localhost:9200/

// response
{
  "status" : 200,
  "name" : "Hydron",
  "version" : {
    "number" : "1.0.3",
    "build_hash" : "61bfb72d845a59a58cd9910e47515665f6478a5c",
    "build_timestamp" : "2014-04-16T14:43:11Z",
    "build_snapshot" : false,
    "lucene_version" : "4.6"
  },
  "tagline" : "You Know, for Search"
}

Installing Kibana

Add the kibana user

1
useradd kibana

Set the password

1
2
3
4
5
6
passwd kibana
Changing password for user kibana.
New password: [enter password]
BAD PASSWORD: it is based on a dictionary word
Retype new password: [enter password again]
passwd: all authentication tokens updated successfully.

Set permissions for kibana

1
chmod +x /home/kibana

kibana

1
su - kibana

Download kibana3

1
curl -LO https://download.elasticsearch.org/kibana/kibana/kibana-3.0.0milestone5.tar.gz

Extract the module

1
tar zxvf kibana-3.0.0milestone5.tar.gz

Set up the symbolic link

1
ln -s /home/kibana/kibana-3.0.0milestone5 ./kibana

Edit the kibana config

  • /home/kibana/kibana/config.js
1
2
// Configure as follows
elasticsearch: "http://(domain)/es/",

Leave the kibana user

1
exit

Configure /es/ as a reverse proxy for connecting to Elasticsearch

1
2
3
4
htdigest -c /etc/httpd/conf/htdigest "Required authentication" (the ID you want to set for Basic authentication)
Adding password for okochang in realm Required authentication.
New password: [enter password] (the password you want to set for Basic authentication)
Re-type new password: [enter password] (the password you want to set for Basic authentication)

vim /etc/httpd/conf.d/vhosts.conf

Syntax-check the configuration file

1
httpd -t

▼ Result

1
Syntax OK

Restart httpd

1
service httpd restart

kibana admin screen

1
http://(domain)/#/dashboard/file/default.json

If it displays as shown below, you have succeeded.

  • fluent-plugin-elasticsearch

Install gcc, gcc-c++

1
yum install gcc gcc-c++ libcurl-devel

Install fluent-plugin-elasticsearch

1
/usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-elasticsearch --no-ri --no-rdoc
1
vim /etc/td-agent/td-agent.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
## Input
<source>
  type tail
  path /var/log/httpd/access_log
  format /^(?<date>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \w{3}) (?<processing_time>[^ ]*) (?<remote>[^ ]*) (?<user>[^ ]*) \[(?<method>.*)\] (?<status>[^ ]*) (?<size>[^ ]*) \[(?<referer>[^ ]*)\] \[(?<agent>.*)\]/
  pos_file /var/log/td-agent/tmp/apache.access.log.pos
  tag apache.access
</source>

## Output
<match apache.access>
  type copy
  <store>
    type file
    path /var/log/td-agent/apache.access
    time_slice_format %Y%m%d
    time_format %Y%m%dT%H%M%S%z
  </store>
  <store>
    type forward
    send_timeout 60s
    recover_wait 10s
    heartbeat_interval 1s
    <server>
      name (fluentd server)
      host (fluentd server IP)
      port (Port)
    </server>
  </store>
  <store>
    type elasticsearch
    host (elasticsearch server IP)
    port (Port)
    type_name access_log
    logstash_format true
    logstash_prefix apache_access
    logstash_dateformat %Y%m
    flush_interval 10s
  </store>
</match>

Preparation before installing fluentd

  • Edit the per-user resource limit file
  • /etc/security/limits.conf

Append the following

1
2
root soft nofile 65536
root hard nofile 65536

▼ Resource attributes

noprocMaximum number of processes
nofileMaximum number of files that can be opened
maxloginMaximum number of logins
dataMaximum data size
fsizeMaximum file size
asMaximum memory address space size
priorityPriority of the user's processes
stackMaximum stack of the user's processes
rssMemory size of the user's processes
coreMaximum core file size

Configure kernel parameters

  • /etc/sysctl.conf
1
2
3
4
// Append the following
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240    65535

Reboot

1
reboot

Apache configuration

Use Apache’s access log as the test log file

1
grep "custom" /etc/httpd/conf/httpd.conf

▼ Result

1
2
LogFormat "%{%Y-%m-%d %T %Z}t %D %a %u [%r] %s %b [%{Referer}i] [%{User-Agent}i]" custom
CustomLog logs/access_log custom

Fix the log directory permissions so td-agent can access them

1
chmod 755 /var/log/httpd

Installing td-agent

1
curl -L http://toolbelt.treasuredata.com/sh/install-redhat.sh | sh

td-agent configuration

  • /etc/td-agent/td-agent.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
<match log.**>
  # Use fluentd-plugin-elasticsearch
  type elasticsearch

  # Format the index for use with Kibana
  logstash_format true

  # Specify the index prefix
  logstash_prefix demo-log

  # Destination Elasticsearch
  hosts localhost:9200

  # Specify the document type when writing to Elasticsearch
  type_name application-log

  # buffer settings - use a memory buffer
  buffer_type memory

  # Chunk size 1MB
  buffer_chunk_limit 1m

  # Maximum chunk queue size 128
  buffer_queue_limit 128

  # Flush the buffer every specified number of seconds - issues a write request to Elasticsearch every specified number of seconds
  flush_interval 2s

  # Maximum number of retries when a flush fails
  retry_limit 17
</match>
1
2
3
4
5
6
7
8
// Create the tmp directory
mkdir /var/log/td-agent/tmp
// Fix the owner
chown td-agent.td-agent /var/log/td-agent/tmp
// Configure startup at server boot
chkconfig td-agent on
// Start
service td-agent start

References

Sakura VPS fluentd + elasticsearch + kibana3

https://kenzo0107.github.io/en/2015/03/01/fluentd-elasticsearch-kibana-sakura-vps/

Author

Kenzo Tanaka

Posted on

2015-03-01

Licensed under

Like this article? Support the author with

×