Argocd manifest diff shows no differences even though image was updated and desired is different then live.

[abhay-apella]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug
Argocd is not updating the live manifest to the desired manifest even though there is a update to the spec image tag. Our environment is:

1. Make a commit to git, create a PR and merge to main.
2. We build the container and then upload it to our artificatory.
3. We use argocd-image-updater to update using digest strategy
4. The desired manifest gets updated and correctly reflects the new image digest.

However, even though we can manually check and see the Live and Desired manifests:
Desired:

spec:
      containers:
        - args:
             ...
          image: >-
            us-central1-docker.pkg.dev/<our artifact repo>:main@sha256:39491a679c6b1d8e02ba5c9674ff2b3a371a566bf4f49743e52430d44878de70

Live:

spec:
      containers:
        - args:
             ...
          image: >-
            us-central1-docker.pkg.dev/<our artifact repo>:main@sha256:7ac98a97c4473f4505e8cf6f760650b62d17f3466192dbc51da7d04edd366d5e

However, the diff of the manifests shows as empty. The other odd thing is that we apply the same set of configuration across multiple apps and our setup consistently works.
In this case we see:

1. Logs from argocd-image-updater that it attempted to deploy the new version
2. We see the Live manifest being mostly updated with the correct new labels and variables that we set.
3. The spec.containers.image is still the old tag.

We also tried to:

1. To disable RespectIgnoreDifferences, which worked the first time to get the image to update but not subsequent times. Our ignore configuration is:

metadata:
  annotations: {}
syncPolicy:
  syncOptions:
    - "CreateNamespace=true"
    - "RespectIgnoreDifferences=true"

ignoreDifferences:
  - group: networking.istio.io
    kind: VirtualService
    jqPathExpressions:
      - >-
        .spec.http[].route[] | select(.destination.host |
        match("stable|canary")) | .weight

2. Manually set values through argocd image updater and through argocd which would not trigger manifest updates.

The only consistent way we can get the deployment to work is to delete the deploy in argocd and then get it to update the deployed image.

Reading this: https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/ it seems it should be merging the updates but it is not consistent. This also did not seem to be the issue: #14838

To Reproduce
We are not sure how to reproduce this consistently.

Expected behavior

We would expect diff to show the desired manifest updates and then be applied to the live manifest globally as it does not occur in all of our apps, but does occur on these steps:

1. Make a commit to git, create a PR and merge to main.
2. We build the container and then upload it to our artificatory.
3. We use argocd-image-updater to update using digest strategy
4. The desired manifest gets updated and correctly reflects the new image digest.

Screenshots

Version
We apply 2.10.5 on cluster install.

• https://raw.githubusercontent.com/argoproj/argo-cd/v2.10.5/manifests/ha/install.yaml

Logs

Paste any relevant application logs here.

[todaywasawesome]

@abhay-apella does it catch the diff if you do a hard refresh manually? I suspect you're having an issue with the cache.

[abhay-apella]

@abhay-apella does it catch the diff if you do a hard refresh manually? I suspect you're having an issue with the cache.

We also tried clicking refresh, sync, and attempted manually updating values supplied. It only updated when we manually changed the spec.containers.image value or deleted the deploy. Would there be another thing to try to check if its a caching issue?

[stephaneetje]

Hello, we are facing the same issue after upgrading from helm chart version 6.7.18 to 6.8.1.
Clicking on Hard refresh does make argocd see the diff to sync. Without manually doing a hard refresh, changes on image tag are never seen.

This bug looks like it came with argocd version 2.11.0