What is the usecase of multi tier db engine architecture?

[yasharne]

Hi
I read the document about the tiers in dbengine: https://learn.netdata.cloud/docs/architecture/database-engine#tiers
But I don't get what is the benefit of using it, do I get less response time when querying older data? or does it help with storage management?
Can you give some samples or use cases for this feature?

[Ferroin]

The primary use case is storing older data without needing absurd amounts of disk space to do so.

Each tier is storing lower resolution data then the next higher tier, so with a given amount of space it can store data for the same number of metrics for a much longer period of time.

Without tiering, storing a year of data for a few thousand metrics would take at least hundreds of gigabytes of disk space, if not multiple terabytes, which is unreasonable for most single nodes, and completely non-viable for large collections of nodes.

With tiering, it can be done in a few dozen gigabytes of space tops, while still providing good enough resolution for looking at long term trends in the data.

The whole set up is designed to take advantage of the fact that you generally don’t need high resolution for long-term data, which is a result of two factors:

• In general, you need the same number of data points for each metric for a graph independent of whether you’re looking at the last ten minutes of data, the last hour of data, the last day of data, or the last year of data (put differently, the number of points required for a specific metric on a specific graph is usually solely dependent on the way you are rendering the graph itself, because you have a fixed amount of space to plot the data in).
• When looking at long time periods, you’re almost always looking at long-term trends, so you rarely care about having per-second resolution when actually viewing the data (even if you do care about having it when the data was collected).

It also has a secondary benefit that queries over long time intervals can be processed faster (lower latency for a response, and lower load on the system to construct the response), because there is less data to be processed.

RRDTool, Graphite’s Whisper, and most other TSDB software supports essentially the same type of thing, they just use different terminology for it. RRDTool talks about ’consolidation’ and RRAs, some other tools use the term ‘progressive thinning’, etc.