on_deployment_update + mention_enricher creates multiple alerts.

[kristeey]

Describe the bug
Triggers: on_deployment _update
actions: resource_babysitter and mention_enricher
sinks: Slack

Using on_deployment _update + resource_babysitter alone works just fine, but when used alongside mention_enricher produce multiple (i.e. four per pod) "empty" alerts as well as the actual "configured" alert. The "empty" alerts also triggers whenever there is any change to a deployment, even if the change is not in the field_to_monitor defined in resource_babysitter.

I have narrowed it down to being related to the trigger on_deployment_update trigger not working well with mention_enricher action. I have also tested that on_pod_update gives the same outcome. on_pod_create however will only create a single triggering resulting in a single alert.

I guess this has something to do with that a single change in a deployment will actually aggregate multiple changes to the deployment manifest. I.e. annotations.deployment.kubernetes.io/revision, kubectl.kubernetes.io/last-applied-configuration, creationTimestamp, generation e.t.c. And the mention_enricher action will be triggered for each of these changes instead of combining these changes into a single trigger?

Additionally: Alerts will be created even though the change does contain the specific field in specified in the resource_babysitter, but in this case we do not get the "configured" alert as expected. Removing the resource_babysitter action all together while still leaving the mention_enricher action in will give the same result.

To Reproduce
1. Configure Robusta helm chart:

Custom Playbook:

    - triggers:
        - on_deployment_update: {}
      actions:
        - resource_babysitter:
            fields_to_monitor:
            - spec.template.spec.containers[0].image 
        - mention_enricher:
            static_mention:
            - @kristian
            message_template: $mentions

Sink:

    - slack_sink:
        api_key: '{{ env.SLACK_API_KEY }}'
        default: true
        name: main_slack_sink
        slack_channel: tmp-robusta-test

2. Apply a Dummy deployment to the cluster:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dummy-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx

apply to cluster using kubectl apply -f dummy-deployment.yaml.

3. Manually change the container image ...
   to e.g. nginx:latest using kubectl edit deployment dummy-deployment

Expected behavior
Get a single alert in the tmp-robusta-test slack channel that is enriched with the slack mention @kristian, when the image of the dummy deployment is changed.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context

[aantn]

Hi @kristeey,
I am guessing that the extra messages are for changes which matched the on_deployment_update trigger but where the babysitter did nothing because the change was not to the field spec.template.spec.containers[0].image . In that case, only the mention_enricher did something.

This obviously is not the desired behaviour and we have a fix for it coming soon! (We are moving the filtering functionality to the trigger itself - on_deployment_update - so that it will impact all actions in the playbook.)