Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support time based retention #17413

Merged
merged 33 commits into from
Jun 7, 2024
Merged

Support time based retention #17413

merged 33 commits into from
Jun 7, 2024

Commits on Jun 6, 2024

  1. Add charts 

    Rename options
Global backfill
Metadata calculation (percentage)
Retention timer
Adjust time
Calculate iterations
Update every is less than 60 and divisor of 60
    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    12ee96f View commit details
    Browse the repository at this point in the history

  2. Add option to specify time retention 

    dbengine tier x retention days
Switch to dbengine tier x disk space MB
Fix grouping iterations
    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    2cde1a5 View commit details
    Browse the repository at this point in the history

  3. Allow disk space to be 0 (unlimited) 

    Assume retention to be the one specified with time
Sane value for maximum datafile target size if max disk space is 0
    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    5134d9d View commit details
    Browse the repository at this point in the history

  4. Test configured disk space to be total free space - 10% 

    Rework human readable retention and expected retention in nodes_instances api
    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    7eb0afd View commit details
    Browse the repository at this point in the history

  5. Further fix on human readable retention 

    If no time restriction is specified, use disk space calculated one
    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    01356e6 View commit details
    Browse the repository at this point in the history

  6. Remove dbengine parallel initialization option 

    Remove duplicate, commented out code
    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    1f5874c View commit details
    Browse the repository at this point in the history

  7. Allow tier disk space to be 0 in which case the current disk space av… 

    …ailable will be used for calculations
    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    688cdf9 View commit details
    Browse the repository at this point in the history

  8. Proper calculation of iterations 

    Support seconds in human duration representation
Remove commented out code
Create tiers as needed
    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    90de153 View commit details
    Browse the repository at this point in the history

  9. Function to return sqlite database space 

    Compile with disable-ml properly
Do proper time retention check
Temporary additional info in node_instances api
    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    c7b72f2 View commit details
    Browse the repository at this point in the history

  10. Do not account for metadata size if old settings are detected

    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    a44d3db View commit details
    Browse the repository at this point in the history

  11. Consider regacy tier multihost disk space MB settings as well

    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    4588cec View commit details
    Browse the repository at this point in the history

  12. Adjust time retention calculation

    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    358fe53 View commit details
    Browse the repository at this point in the history

  13. Cleanup / allow 5% free disk space when all disk is to be used just t… 

    …o be safe
    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    73c4299 View commit details
    Browse the repository at this point in the history

  14. Update some defaults

    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    d9f339c View commit details
    Browse the repository at this point in the history

  15. Use default 1024 MB for each tier 

    Add new dbengine tier 0 multihost disk space MB
Time based retention defaults to disabled if all parameters are default
    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    10245f1 View commit details
    Browse the repository at this point in the history

  16. Update src/database/engine/rrdengine.c 

    Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud>
    @stelfrag @ilyam8
    stelfrag and ilyam8 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    b257cb9 View commit details
    Browse the repository at this point in the history

  17. Switch retention days to integer

    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    61ac802 View commit details
    Browse the repository at this point in the history

  18. Respect "dbengine multihost disk space MB" setting for tier 0 if "dbe… 

    …ngine tier 0 multihost disk space MB" is not specified
    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    6478cef View commit details
    Browse the repository at this point in the history

  19. Add dbengine_tier label

    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    fd5856f View commit details
    Browse the repository at this point in the history

  20. Fix retention_percentage check

    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    aa4463a View commit details
    Browse the repository at this point in the history

  21. Change to tier

    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    6ff8431 View commit details
    Browse the repository at this point in the history

  22. Time % calculation includes first datafile

    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    ba62e60 View commit details
    Browse the repository at this point in the history

  23. Add disk space used by metadata in node_instances 

    Calculated disk space matches the calculated used for datafile rotation
Fix percentage contribution calculation of metadata to each tier
    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    5afba82 View commit details
    Browse the repository at this point in the history

  24. Revent changes to node_instances API

    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    f209cd7 View commit details
    Browse the repository at this point in the history

  25. Do not take into account metadata diskspace usage for now

    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    71fabbd View commit details
    Browse the repository at this point in the history

  26. Add note for the disk space usage calculation in dbengine_tier_retent… 

    …ion charts vs space reported in /api/v2/node_instances
    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    11b44d3 View commit details
    Browse the repository at this point in the history

  27. Restore update every checks 

    RRDENG_MIN_DISK_SPACE_MB 256 MB (from 64)
dbengine multihost disk space MB maps to dbengine tier 0 disk space MB
Disk space for tiers "dbengine tier X disk space MB"
    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    178143f View commit details
    Browse the repository at this point in the history

  28. Check backfill option before tier disk space config to reorder it in … 

    …netdata.conf output
    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    3387c00 View commit details
    Browse the repository at this point in the history

  29. Check tier iteration option before the disk space config per tier

    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    aaee15c View commit details
    Browse the repository at this point in the history

  30. Check if legacy dbengine multihost disk space MB option is set before… 

    … migration of options

Rename options
    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    044669e View commit details
    Browse the repository at this point in the history

  31. Fix warning

    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    8c07067 View commit details
    Browse the repository at this point in the history

  32. Allow dbengine tier 0 disk space MB to be zero and not default to 256… 

    … MB.

If the value is non zero for tier 0, it must be at least 256 MB
    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    7164004 View commit details
    Browse the repository at this point in the history

  33. Check for options in global section

    @stelfrag
    stelfrag committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    c2a00d7 View commit details
    Browse the repository at this point in the history