Long gone are the times where “the” database was single Relational Database Management System installed typically on the most powerful server in the datacenter. The monsters handled anything the business required. Today databases run on commodity hardware, they are also more sophisticated in terms of the high availability and specialized to handle particular types of data. Specialization allows them to achieve much better performance – everything is optimized to deal with a particular kind of data: optimizer, storage engine, even language doesn’t have to be SQL, like it used to be in the past. It can be SQL-based with some extensions allowing for more efficient data manipulation, or it can be something totally new.
Today we have analytical, columnar databases like ClickHouse or MariaDB AX, we have big data platforms like Hadoop, NoSQL solutions like MongoDB or Cassandra, key-value datastores like Redis. This blog is going to talk about Time-Series databases like Prometheus or InfluxDB.
A time series database (TSDB) is a database optimized for time-stamped or time series data. Time series data are simply measurements or events that are tracked, monitored, down sampled, and aggregated over time. This could be server metrics, application performance monitoring, network data, logs, sensor data, events, clicks, trades in a market, and many other types of analytics data.
As can be seen from the ranking of the most popular database management systems on DB-Engines, the popularity of TSDBs has maintained a high growth rate over the last two years.
A Time Series Database has a very specific data model to allow it to be so efficient. A data model of time series data mainly consists of the following parts:
- Subject: The subject to be measured. A subject may have attributes with multiple dimensions. Taking server status monitoring as an example, the measured subject is a server, and its attributes may include the cluster name, host name, custom user labels, and so on.
- Measurements: A subject may have one or more measurements, each corresponding to a specific metric. In the case of the server status monitoring, the measured metrics may include CPU usage and IOPS. The value of CPU usage is a percentage, and the value of IOPS is the number of I/Os during the measurement period.
- Timestamp: The measurement report is always attached with a timestamp attribute to indicate the time.
One of the newest areas that has benefited from TSDBs is logging, in particular, the Loki project from Grafana Labs. Not happy with any of the open-source solutions, Grafana engineers started speaking to people and noticed that A LOT of people had the same issues. In fact, I’ve come to realize that lots of developers and operations folk still SSH and grep/tail the logs on machines even today! The commercial and open source solutions they were using were either too pricey or not stable enough. In fact, people were being asked to log less which we think is an anti-pattern for logs.
The conclusion was to build a tool where:
- Logs should be cheap. Nobody should be asked to log less.
- Easy to operate and scale
- Metrics, logs (and traces later) need to work together
Time-series data analysis is getting more and more traction recently, it becomes more and more critical for business operations. Luckily, given the large number of offerings, both open source and commercial, it is quite likely that you can find a tool which will suit your needs.
In order to keep a NOC and SOC running at peak efficiency, they need the best in tooling that can give them as close to complete visibility as possible and allow them to quickly query multiple data sources to root cause an issue. With all of this in mind, learn about next gen tooling in this free white paper.
Author Bio: Joe Goldberg is the Senior Cloud Program manager at CCSI. Over the past 15+ years, Joe has helped companies to design, build out, and optimize their network and data center infrastructure. As a result of his efforts, major gains in ROI have been realized through virtualization, WAN implementation, core network redesigns, and the adoption of cloud services. Joe is also ITIL certified.
