Install and Setup TIG Stack on Ubuntu 20.04

In this tutorial, we are going to learn how to install and setup TIG Stack on Ubuntu 20.04.

TIG stack is a group of powerful open-source monitoring tools, Telegraf, InfluxDB and Grafana where;

• Telegraf is an open-source server agent for collecting and sending metrics and events from databases, systems, and IoT sensors.
• InfluxDB is an open-source time series database and provides datastore for metrics, events, and real-time analytics.
• Grafana is a data visualization and monitoring tool and supports time series datastores such as Graphite, InfluxDB, Prometheus, Elasticsearch.

TIG stack can be used for monitoring system metrics such as memory, disk, logged in users, system load, swap usage, system uptime, system processes. 

Installing and Setup TIG Stack on Ubuntu 20.04

In order to install TIG stack, you need to install and setup each individual component of the stack, Telegraf, InfluxDB, Grafana.

Run System Update
To begin with, ensure that your system packages are up-to-date;

apt update
apt upgrade

Install Telegraf on Ubuntu 20.04

You can install Telegraf on Ubuntu 20.04 either by downloading the DEB package file or directly from the InfluxData repos.

Install Telegraf using DEB Package File

To install Telegraf using the DEB binary, grab the binary installer from the InfluxData downloads page. You can simply obtain the link to binary installer and pull it with wget;

wget https://dl.influxdata.com/telegraf/releases/telegraf_1.14.3-1_amd64.deb

Once you have the binary downloaded, you can install it as follows

dpkg -i telegraf_1.14.3-1_amd64.deb

Install Telegraf from InfluxData Repos

If you want to install it from the InfluxData repos to ensure seamless updates whenever there are new releases, simply create the InfluxData repos as follows;

wget -qO- https://repos.influxdata.com/influxdb.key | sudo apt-key add -
source /etc/lsb-release
echo "deb https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
apt update
apt install telegraf

Running Telegraf on Ubuntu 20.04

Once the installation is done, you can start and enable Telegraf to run on system boot (This is usually done automatically during installation);

systemctl enable --now telegraf

Install InfluxDB on Ubuntu 20.04

Similarly, InfluxDB can be installed using DEB package file or from the InfluxData repository.

Install InfluxDB using DEB Package File. The binary package can be downloaded from InfluxData downloads page.

wget https://dl.influxdata.com/influxdb/releases/influxdb_1.8.0_amd64.deb
dpkg -i influxdb_1.8.0_amd64.deb

Install InfluxDB from InfluxData Repos

As of this writing, InfluxData repos for Ubuntu 20.04 do not provide InfluxDB package. However, InfluxDB package is available on the Focal Universe repos but it is not the latest version as of this writing. Hence, we prefer to install InfluxDB on Ubuntu 20.04 using the debian binary package as described above.

Start and enable InfluxDB to run on system boot;

systemctl enable --now influxdb

You can check the status by running the command below;

systemctl status influxdb

Install Grafana on Ubuntu 20.04

Same to Telegraf and InfluxDB, you can install Grafana on Ubuntu 20.04 using the debian package file or directly from Grafana repos.

Install Grafana using Debian Package File

To install Grafana using the debian package file, download the latest stable release version of Grafana from Grafana downloads page. The latest stable release version is usually selected by default;

apt install -y adduser libfontconfig1
wget https://dl.grafana.com/oss/release/grafana_7.0.1_amd64.deb
dpkg -i grafana_7.0.1_amd64.deb

Install Grafana From Grafana Repos

In this tutorial, we are installing the opensource version of Grafana. Create the repo and install Grafana as follows;

wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
apt update -y
apt install grafana

Running Grafana

systemctl daemon-reload
systemctl enable --now grafana-server

To check the status;

systemctl status grafana-server
● grafana-server.service - Grafana instance
Loaded: loaded (/lib/systemd/system/grafana-server.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2020-05-30 10:00:09 UTC; 3s ago
Docs: http://docs.grafana.org
Main PID: 6325 (grafana-server)
Tasks: 8 (limit: 2282)
Memory: 7.8M
CGroup: /system.slice/grafana-server.service
└─6325 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/var/run/grafana/grafana-server.pid --packaging=deb cfg:default.paths.logs=/var/lo>

May 30 10:00:10 ubuntu20 grafana-server[6325]: t=2020-05-30T10:00:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email"
...

Configuring TIG Stack on Ubuntu 20.04

Once all the components of the stack are in place, proceed with configuration.

Configure InfluxDB to Store System Metrics Collected by Telegraf

Create InfluxDB database for storing the time series metrics to be collected by the Telegraf agent. Connect to InfluxDB using the influx client command.

influx
Connected to http://localhost:8086 version 1.8.0
InfluxDB shell version: 1.8.0
>

Next, create InfluxDB database. In this demo, we create a database called telegraf and a database user called telegraf.

create database telegraf
create user telegraf with password 'myP@SSword'
grant all on telegraf to telegraf

You can list the created databases;

show databases
name: databases
name
----
telegraf
_internal

To list users;

show users
user admin
---- -----
telegraf false

Exit InfluxDB and verify the connection to the database using the credentials created above;

influx -username 'telegraf' -password '' -database telegraf

Enter the password. If all is well, you should get InfluxDB prompt;

password: myP@SSword
Connected to http://localhost:8086 version 1.8.0
InfluxDB shell version: 1.8.0
>

Configuring Telegraf to Collect System Metrics

It is a plugin-driven server agent for collecting and sending metrics and events from databases, systems, and IoT sensors. All metrics are gathered from the declared inputs, and sent to the declared outputs. It is basically made up of four distinct plugin types:

• Input Plugins collect metrics from the system, services
• Processor Plugins transform, decorate, and/or filter metrics
• Aggregator Plugins create aggregate metrics (e.g. mean, min, max, quantiles, etc.)
• Output Plugins write metrics to various destinations such as InfluxDB for our case.

/etc/telegraf/telegraf.conf is the default Telegraf configuration file.

In this tutorial, we will configure TIG stack monitor system memory usage, system processes, disk usage, system load, system uptime and logged in users. As such, we will generate our custom Telegraf configuration with the above specified metrics on the input filter. Custom Telegraf configuration file can be generated with telegraf command.

Create a backup of the original Telegraf configuration file.

mv /etc/telegraf/telegraf.conf{,.old}

Generate the Custom Telegraf for the specified metrics;

telegraf config -input-filter cpu:mem:swap:system:processes:disk -output-filter influxdb > /etc/telegraf/telegraf.conf

Read more about Telegraf input plugins on Telegraf Plugins Github page.

Once the configuration file is generated, edit it to define the connection details to InfluxDB metrics database;

vim /etc/telegraf/telegraf.conf
Replace the InfluxDB database connection details accordingly.

###############################################################################
# OUTPUT PLUGINS #
###############################################################################

# Configuration for sending metrics to InfluxDB
[[outputs.influxdb]]
urls = ["http://127.0.0.1:8086"]
database = "telegraf"
username = "telegraf"
password = "myP@SSword"

This is how our configuration file looks like with no comment lines;

grep -v "^\s*[#\;]\|^\s*$" /etc/telegraf/telegraf.conf
[global_tags]
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = ""
hostname = ""
omit_hostname = false
[[outputs.influxdb]]
urls = ["http://127.0.0.1:8086"]
database = "telegraf"
username = "telegraf"
password = "myP@SSword"
[[inputs.cpu]]
percpu = true
totalcpu = true
collect_cpu_time = false
report_active = false
[[inputs.disk]]
ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]
[[inputs.diskio]]
[[inputs.kernel]]
[[inputs.mem]]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]

For further configurations of Telegraf, refer to Telegraf administration configuration page.

Restart Telegraf after modifications;

systemctl restart telegraf

Recheck the status;

systemctl status telegraf
● telegraf.service - The plugin-driven server agent for reporting metrics into InfluxDB
Loaded: loaded (/lib/systemd/system/telegraf.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2020-05-30 16:43:07 UTC; 8s ago
Docs: https://github.com/influxdata/telegraf
Main PID: 4776 (telegraf)
Tasks: 10 (limit: 2282)
Memory: 10.1M
CGroup: /system.slice/telegraf.service
└─4776 /usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d

May 30 16:43:07 ubuntu20 systemd[1]: Started The plugin-driven server agent for reporting metrics into InfluxDB.
May 30 16:43:07 ubuntu20 telegraf[4776]: 2020-05-30T16:43:07Z I! Starting Telegraf 1.14.3
May 30 16:43:07 ubuntu20 telegraf[4776]: 2020-05-30T16:43:07Z I! Loaded inputs: swap system cpu disk diskio kernel mem processes
May 30 16:43:07 ubuntu20 telegraf[4776]: 2020-05-30T16:43:07Z I! Loaded aggregators:
May 30 16:43:07 ubuntu20 telegraf[4776]: 2020-05-30T16:43:07Z I! Loaded processors:
May 30 16:43:07 ubuntu20 telegraf[4776]: 2020-05-30T16:43:07Z I! Loaded outputs: influxdb
May 30 16:43:07 ubuntu20 telegraf[4776]: 2020-05-30T16:43:07Z I! Tags enabled: host=ubuntu20
May 30 16:43:07 ubuntu20 telegraf[4776]: 2020-05-30T16:43:07Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"ubuntu20", Flush Interval:10s

Start Telegraf and Verify If data appears

Telegraf enables you to run configuration tests outputing metrics to stdout. To run a general test for all metrics defined;

telegraf --config /etc/telegraf/telegraf.conf --test

To run configuration test for a specific metric for example, system input plugin;

telegraf -test -config /etc/telegraf/telegraf.conf --input-filter system
2020-05-30T11:07:42Z I! Starting Telegraf 1.14.3
> system,host=ubuntu20 load1=0.01,load15=0,load5=0.02,n_cpus=2i,n_users=2i 1590836862000000000
> system,host=ubuntu20 uptime=9206i 1590836862000000000
> system,host=ubuntu20 uptime_format=" 2:33" 1590836862000000000

Configure Grafana to Display InfluxDB/Telegraf Metrics

Accessing Grafana Web Interface: To access Grafana externally, you need to open port 3000/tcp on UFW if it is running.

ufw allow 3000/tcp

You can now access Grafana web Interface, http://server-IP-or-Hostname:3000. On the login page, type admin for the username and password.

Reset the password and proceed to Grafana web Interface.

Add InfluxDB Data source

To add your data source, click the either of the highlighted sections as in the screenshot below.

Search for and select InfluxDB data source.

Set the Name and URL of the InfluxDB data source. You can leave it as http://localhost:8086 for local connection.

To test the connection to the InfluxDB database, click Save&Test button.

Create Grafana Telegraf Dashboard

You can either create your own dashboards for Telegraf are find any that suit your needs from the already created Grafana community dashboards. In this tutorial, we are using the community Telegraf-system metrics dashboard created by user, jmutai.

To import the dashboard, click the + (plus) sign on the left panel of Grafana UI. You can either import using the dashboard ID or JSON file provided on specific Grafana dashboard page.

Click Load to load the dashboard.

Once the dashboard loads, select the InfluxDB data source and import the dashboard. You should now be able to see various dashboards detailing various system metrics;

Ref:

Install and Setup TIG Stack on Ubuntu 20.04