Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add constant page type #17217

Closed
wants to merge 2 commits into from
Closed

Conversation

vkalintiris
Copy link
Contributor

Summary

Adds a constant page type that stores efficiently pages that contain the same value.

Experimental results show that we achieve the same memory reduction as the gorilla page type, ie. around 50-75% of the pages are of constant type. However: (a) the implementation is far simpler, (b) it's easy to extend the concept to non-zero tiers (whereas the gorilla implementation is not applicable to non-zero tiers).

Test Plan
  • CI jobs
  • Running agent on parent 3.

@thiagoftsm
Copy link
Contributor

Hello @vkalintiris ,

When I run this PR using db.mode = ram everything is working as expected, but when I shift to db.mode = dbegnine, I observed this SIGABRT:

0x00007ffff6c96aab in pthread_kill@@GLIBC_2.34 () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff6c96aab in pthread_kill@@GLIBC_2.34 () from /lib64/libc.so.6
#1  0x00007ffff6c42e12 in raise () from /lib64/libc.so.6
#2  0x00007ffff6c2849f in abort () from /lib64/libc.so.6
#3  0x0000000000414f68 in netdata_logger_fatal (
    file=file@entry=0xb67ac0 "/home/thiago/Netdata/tests_netdata/src/database/engine/page.c", 
    function=function@entry=0xbb0f10 <__FUNCTION__.1> "pgdc_get_next_point", line=line@entry=714, 
    fmt=fmt@entry=0xb68090 "Wrong expected cursor position")
    at /home/thiago/Netdata/tests_netdata/src/libnetdata/log/log.c:2351
#4  0x00000000006cf95a in pgdc_get_next_point (pgdc=<optimized out>, expected_position=<optimized out>, 
    sp=sp@entry=0x7fffe1bdc400) at /home/thiago/Netdata/tests_netdata/src/database/engine/page.c:714
#5  0x00000000006c9190 in rrdeng_load_metric_next (seqh=0x51963e8)
    at /home/thiago/Netdata/tests_netdata/src/database/engine/rrdengineapi.c:886
#6  0x00000000004a207b in storage_engine_query_next_metric (seqh=<optimized out>)
    at /home/thiago/Netdata/tests_netdata/src/database/rrd.h:592
#7  rrd2rrdr_query_execute (ops=0x51962c0, dim_id_in_rrdr=0, r=0x5196090)
    at /home/thiago/Netdata/tests_netdata/src/web/api/queries/query.c:1662
#8  rrd2rrdr (owa=owa@entry=0x5196050, qt=qt@entry=0x5197060)
    at /home/thiago/Netdata/tests_netdata/src/web/api/queries/query.c:3532
#7  rrd2rrdr_query_execute (ops=0x51962c0, dim_id_in_rrdr=0, r=0x5196090)
    at /home/thiago/Netdata/tests_netdata/src/web/api/queries/query.c:1662
#8  rrd2rrdr (owa=owa@entry=0x5196050, qt=qt@entry=0x5197060)
    at /home/thiago/Netdata/tests_netdata/src/web/api/queries/query.c:3532
#9  0x00000000004a83ba in rrd2rrdr_legacy (owa=owa@entry=0x5196050, st=st@entry=0x2eb7220, 
    points=points@entry=1, after=after@entry=-60, before=before@entry=0, 
    group_method=group_method@entry=RRDR_GROUPING_SUM, resampling_time=0, options=16809984, 
    dimensions=0x1927f08 "unmatched", group_options=0x0, timeout_ms=0, tier=0, 
    query_source=QUERY_SOURCE_HEALTH, priority=STORAGE_PRIORITY_SYNCHRONOUS)
    at /home/thiago/Netdata/tests_netdata/src/web/api/queries/query.c:3429
#10 0x00000000004bc459 in rrdset2value_api_v1 (st=0x2eb7220, wb=wb@entry=0x0, n=n@entry=0x25a0d00, 
    dimensions=0x1927f08 "unmatched", points=points@entry=1, after=after@entry=-60, before=0, 
    group_method=RRDR_GROUPING_SUM, group_options=0x0, resampling_time=0, options=16809984, 
--Type <RET> for more, q to quit, c to continue without paging--
    db_after=0x25a0d48, db_before=0x25a0d50, db_points_read=0x0, db_points_per_tier=0x0, 
    result_points_generated=0x0, value_is_null=0x7fffe1bdcb6c, anomaly_rate=0x0, timeout=0, tier=0, 
    query_source=QUERY_SOURCE_HEALTH, priority=STORAGE_PRIORITY_SYNCHRONOUS)
    at /home/thiago/Netdata/tests_netdata/src/web/api/formatters/rrd2json.c:80
#11 0x00000000004fafc7 in health_event_loop ()
    at /home/thiago/Netdata/tests_netdata/src/health/health_event_loop.c:366
#12 0x00000000004fc733 in health_main (ptr=0xdf6130)
    at /home/thiago/Netdata/tests_netdata/src/health/health_event_loop.c:768
#13 0x000000000085793d in netdata_thread_init (ptr=0x1827f70)
    at /home/thiago/Netdata/tests_netdata/src/libnetdata/threads/threads.c:285
#14 0x00007ffff6c94d57 in start_thread () from /lib64/libc.so.6
#15 0x00007ffff6d196a8 in clone3 () from /lib64/libc.so.6

I do not have these issues with our current master branch.

Best regards!

@vkalintiris
Copy link
Contributor Author

Thanks for reporting the issue @thiagoftsm! The latest commit should fix the error you encountered.

@thiagoftsm
Copy link
Contributor

thiagoftsm commented Mar 22, 2024

Thanks for reporting the issue @thiagoftsm! The latest commit should fix the error you encountered.

After last commit, everything is working as expected, I am going to test streaming during a few hours to be sure I did not miss anything. 🤝

Copy link
Contributor

@thiagoftsm thiagoftsm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After last commit, everything is working as expected; LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants