Losing monitor for MongoDB for unknown reason

[kj415j45]

I've been using Netdata for quite a while. But it seems that Netdata suddenly lost the connection to my MongoDB instances in 7th, March.
I did nothing around that time.

---

Netdata has been configured as auto-updated and now runs on v1.44.0-502-nightly version.
There's no additional setup for Netdata since installed.

MongoDB is running on 7.0.1 (mongodb/mongodb-community-server:latest). Configured as Replica Set.

[ilyam8]

Hi. Can you show ps faxu | grep mongo

[kj415j45]

Sure, here's the output

root     3223944  0.0  0.1   6480  2372 pts/0    S+   15:39   0:00          \_ grep --color=auto mongo
systemd+ 3112655  0.0  0.7  19340 12844 ?        Ss   11:03   0:00  \_ python3 /usr/local/bin/docker-entrypoint.py mongod --replSet [REDACTED]
systemd+ 3112713  0.9 12.7 3232480 222988 ?      Sl   11:03   2:31      \_ mongod --replSet [REDACTED] --bind_ip_all

[REDACTED] is the repl set name.

[ilyam8]

Did you try debug mode? Share its output

[kj415j45]

Tried to investigate that before, but I don't know where's the problem.

DBG godplugin/main.go:145 plugin: name=go.d, version=1.44.0.502 component=agent
DBG godplugin/main.go:147 current user: name=netdata, uid=116 component=agent
INF godplugin/main.go:151 env HTTP_PROXY '', HTTPS_PROXY '' component=agent
INF agent/agent.go:145 instance is started component=agent
INF agent/setup.go:23 loading config file component=agent
DBG agent/setup.go:31 looking for 'go.d.conf' in [/etc/netdata /usr/lib/netdata/conf.d] component=agent
INF agent/setup.go:38 found '/etc/netdata/go.d.conf component=agent
INF agent/setup.go:45 config successfully loaded component=agent
INF agent/agent.go:149 using config: enabled 'true', default_run 'true', max_procs '0' component=agent
INF agent/setup.go:50 loading modules component=agent
INF agent/setup.go:73 enabled/registered modules: 1/77 component=agent
INF agent/setup.go:79 building discovery config component=agent
DBG agent/setup.go:126 looking for 'mongodb.conf' in [/etc/netdata/go.d /usr/lib/netdata/conf.d/go.d] component=agent
DBG agent/setup.go:142 found '/usr/lib/netdata/conf.d/go.d/mongodb.conf component=agent
INF agent/setup.go:147 dummy/read/watch paths: 0/1/0 component=agent
INF discovery/manager.go:116 registered discoverers: [file discovery: [file reader] service discovery] component="discovery manager"
DBG agent/setup.go:165 looking for 'vnodes/' in [/etc/netdata /usr/lib/netdata/conf.d] component=agent
INF agent/setup.go:177 found '/usr/lib/netdata/conf.d/vnodes' (0 vhosts) component=agent
INF discovery/manager.go:61 instance is started component="discovery manager"
INF functions/manager.go:50 instance is started component="functions manager"
INF jobmgr/manager.go:87 instance is started component="job manager"
DBG functions/ext.go:15 registering function 'config' component="functions manager"
CONFIG go.d:collector:mongodb create accepted template /collectors/jobs internal 'internal' 'add schema enable disable test' 0x0000 0x0000

INF file/discovery.go:69 instance is started component=discovery discoverer=file
INF file/read.go:48 instance is started component=discovery discoverer=file
INF sd/sd.go:66 instance is started component="service discovery"
INF file/read.go:49 instance is stopped component=discovery discoverer=file
DBG jobmgr/manager.go:122 received configs: 0/+0/-0 ('/usr/lib/netdata/conf.d/go.d/mongodb.conf') component="job manager"
INF pipeline/pipeline.go:105 instance is started component="service discovery" pipeline="network listeners"
INF pipeline/accumulator.go:92 discoverer 'sd:net_listeners' exited before ctx done component="service discovery" pipeline="network listeners"
INF pipeline/accumulator.go:61 all discoverers exited before ctx done component="service discovery" pipeline="network listeners"
DBG pipeline/pipeline.go:126 received 1 target groups component="service discovery" pipeline="network listeners"
DBG pipeline/pipeline.go:141 processing group 'discoverer=net_listeners,host=localhost,file=/usr/lib/netdata/conf.d/go.d/sd/net_listeners.conf' with 9 target(s) component="service discovery" pipeline="network listeners"
DBG pipeline/compose.go:85 created 1 config(s) for target 'udp_323_4918513195805259991'
INF pipeline/pipeline.go:124 instance is stopped component="service discovery" pipeline="network listeners"
DBG jobmgr/manager.go:122 received configs: 1/+1/-0 ('discoverer=net_listeners,host=localhost,file=/usr/lib/netdata/conf.d/go.d/sd/net_listeners.conf') component="job manager"
CONFIG go.d:collector:chrony:local create accepted job /collectors/jobs discovered 'discoverer=net_listeners,host=localhost,file=/usr/lib/netdata/conf.d/go.d/sd/net_listeners.conf' 'schema get enable disable update restart test' 0x0000 0x0000

WRN jobmgr/dyncfg.go:380 dyncfg: enable: module chrony job local: failed to apply config: can not find chrony module component="job manager"
CONFIG go.d:collector:chrony:local status failed

[ilyam8]

Ah, I didn't notice this right away - your mongodb instance is a Docker container.

v1.44.0-502-nightly version

The issue should be fixed if you update to the latest nightly.

[ilyam8]

@kj415j45

We recently changed go.d.plugin service discovery.

Old way: portscan kind of approach. Have hundreds of data collection jobs defined and try to start them on every start/restart.

New:

• discover application during runtime:
  • host: find open ports by reading /proc/net/{tcp,tcp6,udp,udp6}.
  • docker: query Docker engine.
• identify applications (find which collector we need to use) and create data collection jobs config.

For docker, we use image names to identify applications. And I didn't know there was mongodb/mongodb-community-server.

Fixed in #17173. It will be in tomorrow's nightly release.

[kj415j45]

Thanks for the quick patch. 👍

You might want to add another image name to that list too. mongodb-enterprise-server

[ilyam8]

@kj415j45 I decided to improve detection by considering the exposed port too. If the port is well known (e.g. IANA registered) - you don't need to check the image name. So any container will be identified as mongo as long as it exposes 27017.

Fixed in #17174