Duplicated log entries

[maxfliri]

I'm seeing a lot of duplicated lines since upgrading to v3.8.0. As an example, this line was reported multiple times over about a week of time:

If I print the container logs using kubectl, I see the same line only once:

[2023-03-28T12:19:46Z INFO  aws_credential_types::cache::lazy_caching] credentials cache miss occurred; retrieved new AWS credentials (took 410.669µs)

I'm running logdna-agent v3.8.0 in kubernetes. If I revert to the version I was using before (v2.2.4), the problem goes away.

[dkhokhlov]

hi @maxfliri
could you collect "bad case" logs from agent running with this env var RUST_LOG=debug? full log.

[maxfliri]

hi @dkhokhlov

I tried to run the agent with RUST_LOG=debug in our testing environment, but unfortunately it keeps crashing and restarting when I do so. I think the agent is somehow crashing under the load of its own logs. I tried to exclude the agent's own logs with LOGDNA_EXCLUSION_RULES="/var/log/containers/logdna-agent*.log", but the agent keeps crashing anyway.

[dkhokhlov]

fix PR:
#496

there is also another fix in development now for the same issue. cc: @jakedipity

[jakedipity]

To clarify, that PR fixes a bug we've identified where the agent will re-ingest logs everytime it's internal tailing system is rebuilt (by default every 6 hours). This bug has been around since before 3.6, but was exacerbated by the regular restarts introduced in 3.7.

It's very likely that is causing the issue you're seeing, but there's not enough information to say for certain.

@maxfliri The agent crashing is interesting, if possible could you get us the log output? I don't see the same behavior when enabling debug logs.