High CPU from go.d/logind

[TheOldMan2000]

Hi,

I recently came back to Netdata after a while away and am loving all the improvements. I've noticed that Netdata is using about 56% constant CPU resources, which isn't so great. Specifically, it seems to be the Go.d plugin/collector. I'm running the latest v1.44.0-122-nightly version on Alma Linux 8 (Virtuozzo) with 8 CPU cores and 8GB RAM. I tried switching off the Machine Learning and it made no difference (screenshot attached). The Netdata set-up is almost stock, aside from adding the login details for several Redis pool instances. Is it a known issue at the moment?

The other thing I've noticed is that whereas before I could use Netdata on my iPad (128MB 2017 model) and could scroll several pages down before it white screened, now it is just a few charts. So I wondered if you had any plans for an App at all?

Many thanks.

[ilyam8]

Hi. Your screenshot doesn't say much, I think you will need to debug it yourself:

• find used go.d modules
• disable them one by one in go.d.conf and see which one is the problem

There is only one known issue - go.d/ping in privileged mode (default).

[TheOldMan2000]

Thanks @ilyam8
Didn't Netdata used to provide a chart at the very bottom of the sidebar menu for itself, showing the stats for each module (in this case the Go.d collector)? I can't find it now perhaps it's gone in the Cloud version but that would be a perfect job for Netdata to display and then you could see at a glance which modules are using the most resources CPU, RAM etc.

[TheOldMan2000]

Thank you! That worked and it output:

{
  "plugin": "go.d",
  "module": "web_log"
}
{
  "plugin": "go.d",
  "module": "powerdns_recursor"
}
{
  "plugin": "go.d",
  "module": "redis"
}
{
  "plugin": "go.d",
  "module": "mysql"
}
{
  "plugin": "go.d",
  "module": "logind"
}
{
  "plugin": "go.d",
  "module": "apache"
}
{
  "plugin": "go.d",
  "module": "systemdunits"
}

I'll try disabling each in turn.

[ilyam8]

I suggest starting with logind.

[ilyam8]

Ah, found the problem - #15930

[ilyam8]

What is systemd version on Alma8?

systemd --version

I think we need to disable go.d/logind by default for this version.

[TheOldMan2000]

Nice find, I tried disabling Web Log first and it made no difference. Then I saw your message and I re-enabled it and disabled logind and bingo! The CPU usage dropped from 55-75% CPU down to 0.3-1%.

What is systemd version on Alma8?

# systemctl --version
systemd 239 (239-78.el8)
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy

My Netdata is: netdata v1.44.0-126-nightly

FYI:

# uname -a; grep -HvE "^#|URL" /etc/*release
Linux vpsxxx.inmotionhosting.com 4.18.0 #1 SMP Wed Jul 12 12:00:44 MSK 2023 x86_64 x86_64 x86_64 GNU/Linux
/etc/almalinux-release:AlmaLinux release 8.9 (Midnight Oncilla)
/etc/centos-release:AlmaLinux release 8.9 (Midnight Oncilla)
/etc/os-release:NAME="AlmaLinux"
/etc/os-release:VERSION="8.9 (Midnight Oncilla)"
/etc/os-release:ID="almalinux"
/etc/os-release:ID_LIKE="rhel centos fedora"
/etc/os-release:VERSION_ID="8.9"
/etc/os-release:PLATFORM_ID="platform:el8"
/etc/os-release:PRETTY_NAME="AlmaLinux 8.9 (Midnight Oncilla)"
/etc/os-release:ANSI_COLOR="0;34"
/etc/os-release:LOGO="fedora-logo-icon"
/etc/os-release:CPE_NAME="cpe:/o:almalinux:almalinux:8::baseos"
/etc/os-release:
/etc/os-release:ALMALINUX_MANTISBT_PROJECT="AlmaLinux-8"
/etc/os-release:ALMALINUX_MANTISBT_PROJECT_VERSION="8.9"
/etc/os-release:REDHAT_SUPPORT_PRODUCT="AlmaLinux"
/etc/os-release:REDHAT_SUPPORT_PRODUCT_VERSION="8.9"
/etc/redhat-release:AlmaLinux release 8.9 (Midnight Oncilla)
/etc/system-release:AlmaLinux release 8.9 (Midnight Oncilla)

Netdata Build Info:

# netdata -W buildinfo
Packaging:
    Netdata Version ____________________________________________ : v1.44.0-126-nightly
    Installation Type __________________________________________ : binpkg-rpm
    Package Architecture _______________________________________ : x86_64
    Package Distro _____________________________________________ :  
    Configure Options __________________________________________ : dummy-configure-commandDefault Directories:
    User Configurations ________________________________________ : /etc/netdata
    Stock Configurations _______________________________________ : /usr/lib/netdata/conf.d    Ephemeral Databases (metrics data, metadata) _______________ : /var/cache/netdata
    Permanent Databases ________________________________________ : /var/lib/netdata
    Plugins ____________________________________________________ : /usr/libexec/netdata/plugins.d
    Static Web Files ___________________________________________ : /usr/share/netdata/web
    Log Files __________________________________________________ : /var/log/netdata
    Lock Files _________________________________________________ : /var/lib/netdata/lock
    Home _______________________________________________________ : /var/lib/netdata
Operating System:
    Kernel _____________________________________________________ : Linux
    Kernel Version _____________________________________________ : 4.18.0
    Operating System ___________________________________________ : unknown
    Operating System ID ________________________________________ : unknown
    Operating System ID Like ___________________________________ : unknown
    Operating System Version ___________________________________ : unknown
    Operating System Version ID ________________________________ : 8.9
    Detection __________________________________________________ : unknown
Hardware:
    CPU Cores __________________________________________________ : 8
    CPU Frequency ______________________________________________ : 3200000000
    RAM Bytes __________________________________________________ : 8589934592
    Disk Capacity ______________________________________________ : 0
    CPU Architecture ___________________________________________ : x86_64
    Virtualization Technology __________________________________ : none
    Virtualization Detection ___________________________________ : systemd-detect-virt
Container:
    Container __________________________________________________ : openvz
    Container Detection ________________________________________ : systemd-detect-virt
    Container Orchestrator _____________________________________ : none
    Container Operating System _________________________________ : AlmaLinux
    Container Operating System ID ______________________________ : almalinux
    Container Operating System ID Like _________________________ : rhel centos fedora
    Container Operating System Version _________________________ : 8.9 (Midnight Oncilla)
    Container Operating System Version ID ______________________ : 8.9
    Container Operating System Detection _______________________ : /etc/os-release
Features:
    Built For __________________________________________________ : Linux
    Netdata Cloud ______________________________________________ : YES
    Health (trigger alerts and send notifications) _____________ : YES
    Streaming (stream metrics to parent Netdata servers) _______ : YES
    Back-filling (of higher database tiers) ____________________ : YES
    Replication (fill the gaps of parent Netdata servers) ______ : YES
    Streaming and Replication Compression ______________________ : YES (zstd gzip)
    Contexts (index all active and archived metrics) ___________ : YES
    Tiering (multiple dbs with different metrics resolution) ___ : YES (5)
    Machine Learning ___________________________________________ : YES
Database Engines:
    dbengine ___________________________________________________ : YES
    alloc ______________________________________________________ : YES
    ram ________________________________________________________ : YES
    map ________________________________________________________ : YES
    save _______________________________________________________ : YES
    none _______________________________________________________ : YES
Connectivity Capabilities:
    ACLK (Agent-Cloud Link: MQTT over WebSockets over TLS) _____ : YES
    static (Netdata internal web server) _______________________ : YES
    h2o (web server) ___________________________________________ : YES
    WebRTC (experimental) ______________________________________ : NO
    Native HTTPS (TLS Support) _________________________________ : YES
    TLS Host Verification ______________________________________ : YES
Libraries:
    LZ4 (extremely fast lossless compression algorithm) ________ : NO
    ZSTD (fast, lossless compression algorithm) ________________ : YES
    zlib (lossless data-compression library) ___________________ : YES
    protobuf (platform-neutral data serialization protocol) ____ : YES (system)
    OpenSSL (cryptography) _____________________________________ : YES
    libdatachannel (stand-alone WebRTC data channels) __________ : NO
    JSON-C (lightweight JSON manipulation) _____________________ : YES
    libcap (Linux capabilities system operations) ______________ : NO
    libcrypto (cryptographic functions) ________________________ : YES
Plugins:
    apps (monitor processes) ___________________________________ : YES
    cgroups (monitor containers and VMs) _______________________ : YES
    cgroup-network (associate interfaces to CGROUPS) ___________ : YES
    proc (monitor Linux systems) _______________________________ : YES
    tc (monitor Linux network QoS) _____________________________ : YES
    diskspace (monitor Linux mount points) _____________________ : YES
    freebsd (monitor FreeBSD systems) __________________________ : NO
    macos (monitor MacOS systems) ______________________________ : NO
    statsd (collect custom application metrics) ________________ : YES
    timex (check system clock synchronization) _________________ : YES
    idlejitter (check system latency and jitter) _______________ : YES
    bash (support shell data collection jobs - charts.d) _______ : YES
    debugfs (kernel debugging metrics) _________________________ : YES
    cups (monitor printers and print jobs) _____________________ : YES
    ebpf (monitor system calls) ________________________________ : YES
    freeipmi (monitor enterprise server H/W) ___________________ : YES
    nfacct (gather netfilter accounting) _______________________ : NO
    perf (collect kernel performance events) ___________________ : YES
    slabinfo (monitor kernel object caching) ___________________ : YES
    Xen ________________________________________________________ : NO
    Xen VBD Error Tracking _____________________________________ : NO
    Logs Management ____________________________________________ : YES
Exporters:
    AWS Kinesis ________________________________________________ : NO
    GCP PubSub _________________________________________________ : NO
    MongoDB ____________________________________________________ : YES
    Prometheus (OpenMetrics) Exporter __________________________ : YES
    Prometheus Remote Write ____________________________________ : YES
    Graphite ___________________________________________________ : YES
    Graphite HTTP / HTTPS ______________________________________ : YES
    JSON _______________________________________________________ : YES
    JSON HTTP / HTTPS __________________________________________ : YES
    OpenTSDB ___________________________________________________ : YES
    OpenTSDB HTTP / HTTPS ______________________________________ : YES
    All Metrics API ____________________________________________ : YES
    Shell (use metrics in shell scripts) _______________________ : YES
Debug/Developer Features:
    Trace All Netdata Allocations (with charts) ________________ : NO
    Developer Mode (more runtime checks, slower) _______________ : NO

Hope this helps!