Netdata displays "[x]" in chart sidebar instead of "all" with new custom python collector

[je2555]

I've written a quick collector for disk quota info, which is working nicely. It dynamically gathers quota info from all users. I'm happy to contribute it if you think there's a need. Sorry, I don't know Go so I can't write it in that.

My final problem is with the presentation in the dashboard. The netdata supplied collectors like diskspace are able to title their charts with "all" which is also displayed in the chart sidebar for navigation. I can't see how to replicate that? My collector simply displays "[x]" in the sidebar. I'm sure there's an obvious way to do it, something to do with the chart definition or CLABELS?

I'm kinda struggling to follow the docs because they seem to be written for pre Netdata v1.39 / Charts v30 / NIDL which is mildly confusing.

[andrewm4894]

Can you share your code for your collector?

Would be cool to make a PR to contribute it here

https://github.com/netdata/community/tree/main/collectors/python.d.plugin

[andrewm4894]

It could be that maybe your chart definition is missing this stuff.

netdata/collectors/python.d.plugin/example/example.chart.py

Line 18 in 200a90f

'options': [None, 'A random number', 'random number', 'random', 'random', 'line'],

I'm not sure this has changed much recently in how it's handled etc.

Anyway if you can share your code in some way I think I should be able to help.

[je2555]

Hmm, I do have that stuff, but I change the 4th item depending on the user. Here's my code, hopefully you can spot the issue!

# -*- coding: utf-8 -*-
# Description: disk quota netdata python.d module
# Author: je2555
# SPDX-License-Identifier: GPL-3.0-or-later

import csv
from bases.FrameworkServices.ExecutableService import ExecutableService

# You must add this command to the netdata user in sudoers file, eg:
# netdata    ALL=(root)    NOPASSWD: /sbin/repquota --output=csv /
QUOTA_COMMAND = 'sudo repquota --output=csv /'

# Dynamically generate a series of charts per user similar to these

ORDER = [
#    'blocks',
#    'files',
]

CHARTS = {
#    'blocks': {
#        'options': [None, 'Disk Block Quota', 'KB', 'family', 'diskquota.blocks', 'area'],
#        'lines': [
#            ['blocks_used', None, 'absolute'],
#            ['blocks_softlimit', None, 'absolute'],
#            ['blocks_hardlimit', None, 'absolute'],
#        ]
#    },
#    'files': {
#        'options': [None, 'Disk File Quota', 'files', 'family', 'diskquota.files', 'area'],
#        'lines': [
#            ['files_used', None, 'absolute'],
#            ['files_softlimit', None, 'absolute'],
#            ['files_hardlimit', None, 'absolute'],
#        ]
#    },
}

def quota_chart_template(user):
    """
    Build chart config for a user
    :return: tuple
    """
    order = [
        user + '_blocks',
        user + '_files',
    ]

    charts = {
        order[0]: {
            'options': [None, 'Disk Block Quota', 'KB', user, 'diskquota.blocks', 'area'],
            'lines': [
                [user + '_blocks_used', None, 'absolute'],
                [user + '_blocks_softlimit', None, 'absolute'],
                [user + '_blocks_hardlimit', None, 'absolute'],
            ]
        },
        order[1]: {
            'options': [None, 'Disk File Quota', 'files', user, 'diskquota.files', 'area'],
            'lines': [
                [user + '_files_used', None, 'absolute'],
                [user + '_files_softlimit', None, 'absolute'],
                [user + '_files_hardlimit', None, 'absolute'],
            ]
        },
    }

    return order, charts


class Service(ExecutableService):
    def __init__(self, configuration=None, name=None):
        ExecutableService.__init__(self, configuration=configuration, name=name)
        self.order = ORDER
        self.definitions = CHARTS
        self.command = QUOTA_COMMAND

    def _get_data(self):
        """
        Format data received from shell command
        :return: dict
        """
        raw = self._get_raw_data()
        data = csv.DictReader(raw)

        active_charts = self.charts.active_charts()

        result = {}
        for row in data:
            user = row['User']

            # Create charts that don't exist yet
            if user + '_blocks' not in active_charts:
                order, charts = quota_chart_template(user)

                for chart_name in order:
                    params = [chart_name] + charts[chart_name]['options']
                    dimensions = charts[chart_name]['lines']

                    new_chart = self.charts.add_chart(params)
                    for dimension in dimensions:
                        new_chart.add_dimension(dimension)

            # Prepare and submit the gathered data
            result.update({
                user + '_blocks_used': int(row['BlockUsed']),
                user + '_blocks_softlimit': int(row['BlockSoftLimit']),
                user + '_blocks_hardlimit': int(row['BlockHardLimit']),
                user + '_files_used': int(row['FileUsed']),
                user + '_files_softlimit': int(row['FileSoftLimit']),
                user + '_files_hardlimit': int(row['FileHardLimit']),
            })
        return result

We could have done it with Pandas but we wanted it to look neat in the interface, and we'd have needed an external script to write the quota CSV to a file for Pandas to read.

[ilyam8]

but I change the 4th item depending on the user

This is a problem. The "all" thing was/is a temporary workaround and implemented (hard-coded for some metrics). Having instance-specific data in "family" is not correct. Consider changing family, e.g. "blocks", "files".

[je2555]

I see, I was under the impression I had to change that for each instance, but now I'm looking at it again, I'm guessing that each entry in the ORDER list is a separate instance?

While I'm at it is there any way to change the name of the plugin in the interface to something other than "diskquota" taken from the filename?

[ilyam8]

Just change the chart family (user)

'options': [None, 'Disk Block Quota', 'KB', user, 'diskquota.blocks', 'area'],
...
'options': [None, 'Disk File Quota', 'files', user, 'diskquota.files', 'area'],

=>

'options': [None, 'Disk Block Quota', 'KB', "blocks", 'diskquota.blocks', 'area'],
...
'options': [None, 'Disk File Quota', 'files', "files", 'diskquota.files', 'area'],

---

other than "diskquota"

Not really right now.

[je2555]

Another question:

How can I create health alarms for this? I would want to use a template so I don't have to create an individual alarm for each user. I've tried a few different approaches but I can't seem to make them work. Here's one I thought was closest:

 template: diskquota_blocks_used
       on: diskquota.blocks
    class: Utilization
     type: System
component: Disk
    hosts: *
 families: *
    every: 10s 
   lookup: max -10s foreach *_blocks_used
    units: KB

 template: diskquota_blocks_hardlimit
       on: diskquota.blocks
    class: Utilization
     type: System
component: Disk
    hosts: *
 families: *
    every: 10s 
   lookup: max -10s foreach *_blocks_hardlimit
    units: KB

 template: diskquota_blocks_remaining
       on: diskquota.blocks
    class: Utilization
     type: System
component: Disk
    hosts: *
 families: *
    every: 10s 
     calc: (($diskquota_blocks_hardlimit - $diskquota_blocks_used) / $diskquota_blocks_hardlimit) * 100 
    units: %
     warn: $this < (($status >= $WARNING)  ? (10) : (5))
     crit: $this < (($status == $CRITICAL) ? (5) : (2))
    delay: up 1m down 15m multiplier 1.5 max 1h
     info: 
       to: silent

But the calc at the end doesn't work, it just comes out at 0%. I've also tried:

     calc: (($blocks_hardlimit - $blocks_used) / $blocks_hardlimit) * 100 

and:

     calc: (($*_blocks_hardlimit - $*_blocks_used) / $*_blocks_hardlimit) * 100 

But because the family is now either blocks or files and not the username, I don't think it's as easy as I thought it was going to be.