In this tutorial, we are going to learn how to install and setup TIG Stack on Ubuntu 20.04.
TIG stack is a group of powerful open-source monitoring tools, Telegraf, InfluxDB and Grafana where;
- Telegraf is an open-source server agent for collecting and sending metrics and events from databases, systems, and IoT sensors.
- InfluxDB is an open-source time series database and provides datastore for metrics, events, and real-time analytics.
- Grafana is a data visualization and monitoring tool and supports time series datastores such as Graphite, InfluxDB, Prometheus, Elasticsearch.
TIG stack can be used for monitoring system metrics such as memory, disk, logged in users, system load, swap usage, system uptime, system processes.
Installing and Setup TIG Stack on Ubuntu 20.04
In order to install TIG stack, you need to install and setup each individual component of the stack, Telegraf, InfluxDB, Grafana.
Run System Update
To begin with, ensure that your system packages are up-to-date;
apt update apt upgrade
Install Telegraf on Ubuntu 20.04
You can install Telegraf on Ubuntu 20.04 either by downloading the DEB package file or directly from the InfluxData repos.
Install Telegraf using DEB Package File
To install Telegraf using the DEB binary, grab the binary installer from the InfluxData downloads page. You can simply obtain the link to binary installer and pull it with wget;
wget https://dl.influxdata.com/telegraf/releases/telegraf_1.14.3-1_amd64.deb
Once you have the binary downloaded, you can install it as follows
dpkg -i telegraf_1.14.3-1_amd64.deb
Install Telegraf from InfluxData Repos
If you want to install it from the InfluxData repos to ensure seamless updates whenever there are new releases, simply create the InfluxData repos as follows;
wget -qO- https://repos.influxdata.com/influxdb.key | sudo apt-key add - source /etc/lsb-release echo "deb https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list apt update apt install telegraf
Running Telegraf on Ubuntu 20.04
Once the installation is done, you can start and enable Telegraf to run on system boot (This is usually done automatically during installation);
systemctl enable --now telegraf
Install InfluxDB on Ubuntu 20.04
Similarly, InfluxDB can be installed using DEB package file or from the InfluxData repository.
Install InfluxDB using DEB Package File. The binary package can be downloaded from InfluxData downloads page.
wget https://dl.influxdata.com/influxdb/releases/influxdb_1.8.0_amd64.deb dpkg -i influxdb_1.8.0_amd64.deb
Install InfluxDB from InfluxData Repos
As of this writing, InfluxData repos for Ubuntu 20.04 do not provide InfluxDB package. However, InfluxDB package is available on the Focal Universe repos but it is not the latest version as of this writing. Hence, we prefer to install InfluxDB on Ubuntu 20.04 using the debian binary package as described above.
Start and enable InfluxDB to run on system boot;
systemctl enable --now influxdb
You can check the status by running the command below;
systemctl status influxdb
Install Grafana on Ubuntu 20.04
Same to Telegraf and InfluxDB, you can install Grafana on Ubuntu 20.04 using the debian package file or directly from Grafana repos.
Install Grafana using Debian Package File
To install Grafana using the debian package file, download the latest stable release version of Grafana from Grafana downloads page. The latest stable release version is usually selected by default;
apt install -y adduser libfontconfig1 wget https://dl.grafana.com/oss/release/grafana_7.0.1_amd64.deb dpkg -i grafana_7.0.1_amd64.deb
Install Grafana From Grafana Repos
In this tutorial, we are installing the opensource version of Grafana. Create the repo and install Grafana as follows;
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add - add-apt-repository "deb https://packages.grafana.com/oss/deb stable main" apt update -y apt install grafana
Running Grafana
systemctl daemon-reload systemctl enable --now grafana-server
To check the status;
systemctl status grafana-server ● grafana-server.service - Grafana instance Loaded: loaded (/lib/systemd/system/grafana-server.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2020-05-30 10:00:09 UTC; 3s ago Docs: http://docs.grafana.org Main PID: 6325 (grafana-server) Tasks: 8 (limit: 2282) Memory: 7.8M CGroup: /system.slice/grafana-server.service └─6325 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/var/run/grafana/grafana-server.pid --packaging=deb cfg:default.paths.logs=/var/lo> May 30 10:00:10 ubuntu20 grafana-server[6325]: t=2020-05-30T10:00:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" ...
Configuring TIG Stack on Ubuntu 20.04
Once all the components of the stack are in place, proceed with configuration.
Configure InfluxDB to Store System Metrics Collected by Telegraf
Create InfluxDB database for storing the time series metrics to be collected by the Telegraf agent. Connect to InfluxDB using the influx client command.
influx Connected to http://localhost:8086 version 1.8.0 InfluxDB shell version: 1.8.0 >
Next, create InfluxDB database. In this demo, we create a database called telegraf and a database user called telegraf.
create database telegraf create user telegraf with password 'myP@SSword' grant all on telegraf to telegraf
You can list the created databases;
show databases name: databases name ---- telegraf _internal
To list users;
show users user admin ---- ----- telegraf false
Exit InfluxDB and verify the connection to the database using the credentials created above;
influx -username 'telegraf' -password '' -database telegraf
Enter the password. If all is well, you should get InfluxDB prompt;
password: myP@SSword Connected to http://localhost:8086 version 1.8.0 InfluxDB shell version: 1.8.0 >
Configuring Telegraf to Collect System Metrics
It is a plugin-driven server agent for collecting and sending metrics and events from databases, systems, and IoT sensors. All metrics are gathered from the declared inputs, and sent to the declared outputs. It is basically made up of four distinct plugin types:
- Input Plugins collect metrics from the system, services
- Processor Plugins transform, decorate, and/or filter metrics
- Aggregator Plugins create aggregate metrics (e.g. mean, min, max, quantiles, etc.)
- Output Plugins write metrics to various destinations such as InfluxDB for our case.
/etc/telegraf/telegraf.conf is the default Telegraf configuration file.
In this tutorial, we will configure TIG stack monitor system memory usage, system processes, disk usage, system load, system uptime and logged in users. As such, we will generate our custom Telegraf configuration with the above specified metrics on the input filter. Custom Telegraf configuration file can be generated with telegraf command.
Create a backup of the original Telegraf configuration file.
mv /etc/telegraf/telegraf.conf{,.old}
Generate the Custom Telegraf for the specified metrics;
telegraf config -input-filter cpu:mem:swap:system:processes:disk -output-filter influxdb > /etc/telegraf/telegraf.conf
Read more about Telegraf input plugins on Telegraf Plugins Github page.
Once the configuration file is generated, edit it to define the connection details to InfluxDB metrics database;
vim /etc/telegraf/telegraf.conf Replace the InfluxDB database connection details accordingly. ############################################################################### # OUTPUT PLUGINS # ############################################################################### # Configuration for sending metrics to InfluxDB [[outputs.influxdb]] urls = ["http://127.0.0.1:8086"] database = "telegraf" username = "telegraf" password = "myP@SSword"
This is how our configuration file looks like with no comment lines;
grep -v "^\s*[#\;]\|^\s*$" /etc/telegraf/telegraf.conf [global_tags] [agent] interval = "10s" round_interval = true metric_batch_size = 1000 metric_buffer_limit = 10000 collection_jitter = "0s" flush_interval = "10s" flush_jitter = "0s" precision = "" hostname = "" omit_hostname = false [[outputs.influxdb]] urls = ["http://127.0.0.1:8086"] database = "telegraf" username = "telegraf" password = "myP@SSword" [[inputs.cpu]] percpu = true totalcpu = true collect_cpu_time = false report_active = false [[inputs.disk]] ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"] [[inputs.diskio]] [[inputs.kernel]] [[inputs.mem]] [[inputs.processes]] [[inputs.swap]] [[inputs.system]]
For further configurations of Telegraf, refer to Telegraf administration configuration page.
Restart Telegraf after modifications;
systemctl restart telegraf
Recheck the status;
systemctl status telegraf ● telegraf.service - The plugin-driven server agent for reporting metrics into InfluxDB Loaded: loaded (/lib/systemd/system/telegraf.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2020-05-30 16:43:07 UTC; 8s ago Docs: https://github.com/influxdata/telegraf Main PID: 4776 (telegraf) Tasks: 10 (limit: 2282) Memory: 10.1M CGroup: /system.slice/telegraf.service └─4776 /usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d May 30 16:43:07 ubuntu20 systemd[1]: Started The plugin-driven server agent for reporting metrics into InfluxDB. May 30 16:43:07 ubuntu20 telegraf[4776]: 2020-05-30T16:43:07Z I! Starting Telegraf 1.14.3 May 30 16:43:07 ubuntu20 telegraf[4776]: 2020-05-30T16:43:07Z I! Loaded inputs: swap system cpu disk diskio kernel mem processes May 30 16:43:07 ubuntu20 telegraf[4776]: 2020-05-30T16:43:07Z I! Loaded aggregators: May 30 16:43:07 ubuntu20 telegraf[4776]: 2020-05-30T16:43:07Z I! Loaded processors: May 30 16:43:07 ubuntu20 telegraf[4776]: 2020-05-30T16:43:07Z I! Loaded outputs: influxdb May 30 16:43:07 ubuntu20 telegraf[4776]: 2020-05-30T16:43:07Z I! Tags enabled: host=ubuntu20 May 30 16:43:07 ubuntu20 telegraf[4776]: 2020-05-30T16:43:07Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"ubuntu20", Flush Interval:10s
Start Telegraf and Verify If data appears
Telegraf enables you to run configuration tests outputing metrics to stdout. To run a general test for all metrics defined;
telegraf --config /etc/telegraf/telegraf.conf --test
To run configuration test for a specific metric for example, system input plugin;
telegraf -test -config /etc/telegraf/telegraf.conf --input-filter system 2020-05-30T11:07:42Z I! Starting Telegraf 1.14.3 > system,host=ubuntu20 load1=0.01,load15=0,load5=0.02,n_cpus=2i,n_users=2i 1590836862000000000 > system,host=ubuntu20 uptime=9206i 1590836862000000000 > system,host=ubuntu20 uptime_format=" 2:33" 1590836862000000000
Configure Grafana to Display InfluxDB/Telegraf Metrics
Accessing Grafana Web Interface: To access Grafana externally, you need to open port 3000/tcp on UFW if it is running.
ufw allow 3000/tcp
You can now access Grafana web Interface, http://server-IP-or-Hostname:3000. On the login page, type admin for the username and password.
Reset the password and proceed to Grafana web Interface.
Add InfluxDB Data source
To add your data source, click the either of the highlighted sections as in the screenshot below.
Search for and select InfluxDB data source.
Set the Name and URL of the InfluxDB data source. You can leave it as http://localhost:8086 for local connection.
To test the connection to the InfluxDB database, click Save&Test button.
Create Grafana Telegraf Dashboard
You can either create your own dashboards for Telegraf are find any that suit your needs from the already created Grafana community dashboards. In this tutorial, we are using the community Telegraf-system metrics dashboard created by user, jmutai.
To import the dashboard, click the + (plus) sign on the left panel of Grafana UI. You can either import using the dashboard ID or JSON file provided on specific Grafana dashboard page.
Click Load to load the dashboard.
Once the dashboard loads, select the InfluxDB data source and import the dashboard. You should now be able to see various dashboards detailing various system metrics;
Ref: