Unified Alerting: contact-points/export/ returns EOF

[andrejshapal]

What happened?

Hello,
When trying to get manifest of contact points I am getting EOF (any option in UI and via API).
https://.../api/v1/provisioning/contact-points/export/
{"message":"EOF","traceID":""}
At the same time, the rest of simmilar exports (dashboard, alert rule) works without any issues.

I turned on debug logging, but did not find anything related.
The errors are:

2024-05-17 09:54:54.899	
{
  "error": "EOF",
  "level": "error",
  "logger": "context",
  "msg": "EOF",
  "orgId": 1,
  "remote_addr": "84.237.247.18",
  "t": "2024-05-17T09:54:54.899384343Z",
  "traceID": "",
  "uname": "andrej",
  "userId": 138
}

2024-05-17 09:54:54.899	
{
  "duration": "3.975834ms",
  "handler": "/api/v1/provisioning/contact-points/export",
  "level": "error",
  "logger": "context",
  "method": "GET",
  "msg": "Request Completed",
  "orgId": 1,
  "path": "/api/v1/provisioning/contact-points/export/",
  "referer": "https://appsupport.grafana-dev.domain.com/alerting/notifications?alertmanager=grafana&search=",
  "remote_addr": "1.2.3.4",
  "size": 30,
  "status": 500,
  "status_source": "server",
  "t": "2024-05-17T09:54:54.899445268Z",
  "time_ms": 3,
  "uname": "andrej",
  "userId": 138
}

At the same time, alerts are working without any issues.
I have tried 10.4.1, 10.4.4 and 11.0.0. All had the same issue.

At the same time, I have instance on 10.4.1 which don't have this issue. The difference between instances: one is deployed with grafana operator and uses postgres backend. The one without issues has sqllite.
But I did not find any difference in db payload:

The config in general is the same in both grafanas except db part.

config:
  log:
    mode: "console"
  log.console:
      level: "debug"
      format: "json"
  auth:
    disable_login_form: "true"
  auth.basic:
    enabled: "true"
  auth.proxy:
      enabled: "true"
      header_property: email
      header_name: X-Pomerium-Claim-Email
      headers: Role:X-Pomerium-Role
      auto_sign_up: "true"
      sync_ttl: "60"
      enable_login_token: "false"
  users:
    auto_assign_org_role: Editor
  alerting:
    enabled: "false"
  unified_alerting:
    enabled: "true"
  dataproxy:
    logging: "false"
    timeout: "300"
  feature_toggles:
    enable: editPanelCSVDragAndDrop, enableDatagridEditing, groupToNestedTableTransformation
  security:
    admin_user: root
    admin_password: secret
  database:
    type: postgres
    host: db-grafana.monitoring.svc.cluster.local
    name: app_dev
    user: app_dev_owner_user
    ssl_mode: require
  dashboards:
    default_home_dashboard_path: /var/lib/grafana/home-dashboard/home.json
  date_formats:
    default_timezone: UTC
  server:
    root_url: https://appsupport.grafana-dev.domain.com/
    router_logging: "false"

I did check db logs for errors and did not find any.

What did you expect to happen?

/api/v1/provisioning/contact-points/export should return manifest

Did this work before?

This is my first usage of this functionality.

How do we reproduce it?

I am not sure. If the issue is in postgres, then it should be replicatable just switching be to psql.

application_support_dev=# SHOW server_version;
-[ RECORD 1 ]--+---------------------------------
server_version | 14.7 (Ubuntu 14.7-1.pgdg22.04+1)

But this looks like something wrong in code (serialization maybe) because simmilar api endpoints works without any issues.

Is the bug inside a dashboard panel?

No response

Environment (with versions)?

No response

Grafana platform?

Kubernetes

Datasource(s)?

No response

[jim-barber-he]

I've also hit this problem on version 10.4.1.
I'm also a first time user of this API. Exports of the other alerting resources are working fine, it's only the contact-points that are not working.
Leaving /export off the end lists the contact-points in a non-export format fine...

[andrejshapal]

I've also hit this problem on version 10.4.1. I'm also a first time user of this API. Exports of the other alerting resources are working fine, it's only the contact-points that are not working. Leaving /export off the end lists the contact-points in a non-export format fine...

Which backend do you use?

[jim-barber-he]

We're just using the sqlite backend.

[jim-barber-he]

I got this export to work by adding the ?decrypt=true parameter...
/api/v1/provisioning/contact-points/export?decrypt=true

[tonypowa]

Thank you for reporting this issue @andrejshapal
I have reproduced it from my end using the default Grafana db sqlite in version 10.4.1
Forwarding issue to the alerting squad

Test in v10.2.1 (no issue)

apiVersion: 1
contactPoints:
    - orgId: 1
      name: email receiver
      receivers:
        - uid: xqVk
          type: email
          settings:
            addresses: <example@email.com>
          disableResolveMessage: false
    - orgId: 1
      name: paco_test
      receivers:
        - uid: 5A_
          type: email
          settings:
            addresses: 
            message: '{{ $variable }}'
            singleEmail: false
          disableResolveMessage: false
    - orgId: 1
      name: test
      receivers:
        - uid: c8ad90db
          type: telegram
          settings:
            bottoken: '[REDACTED]'
            chatid: "-"
            disable_notification: false
            disable_web_page_preview: false
            protect_content: false
          disableResolveMessage: false

Test in v10.4.1 (issue can be reproduced)

{
    "message": "EOF",
    "traceID": ""
}