No login possible after the upgrade of PostgreSQL from 15 to 16.1

[christianlupus]

Playbook Configuration:

My vars.yml file looks like this:

# The bare domain name which represents your Matrix identity.
# Matrix user ids for your server will be of the form (`@user:<matrix-domain>`).
#
# Note: this playbook does not touch the server referenced here.
# Installation happens on another server ("matrix.<matrix-domain>").
#
# Example value: example.com
matrix_domain: aes-sb.de

# This is something which is provided to Let's Encrypt when retrieving SSL certificates for domains.
#
# In case SSL renewal fails at some point, you'll also get an email notification there.
#
# If you decide to use another method for managing SSL certifites (different than the default Let's Encrypt),
# you won't be required to define this variable (see `docs/configuring-playbook-ssl-certificates.md`).
#
# Example value: someone@example.com
devture_traefik_config_certificatesResolvers_acme_email: certs@aes-sb.de

# A shared secret (between Coturn and Synapse) used for authentication.
# You can put any string here, but generating a strong one is preferred (e.g. `pwgen -s 64 1`).
matrix_coturn_turn_static_auth_secret: "=== Redacted ==="

# A secret used to protect access keys issued by the server.
# You can put any string here, but generating a strong one is preferred (e.g. `pwgen -s 64 1`).
matrix_synapse_macaroon_secret_key: "=== Redacted ==="

matrix_coturn_turn_external_ip_address: "51.38.113.161"

matrix_synapse_max_upload_size_mb: 25

matrix_synapse_enable_registration: true
matrix_synapse_registrations_require_3pid:
  - email
matrix_ma1sd_container_labels_matrix_client_3pid_registration_enabled: true
matrix_synapse_auto_join_rooms:
  - "#aes:aes-sb.de"
matrix_ma1sd_verbose_logging: true
matrix_ma1sd_configuration_extension_yaml: |
  register:
    policy:
      threepid:
        email:
          domain:
            whitelist:
              - '*aut.uni-saarland.de'

matrix_synapse_federation_domain_whitelist:
  - aes-sb.de

#matrix_synapse_ext_spam_checker_synapse_simple_antispam_enabled: true

#matrix_synapse_turn_uris: []
#matrix_synapse_turn_shared_secret: ""
#matrix_synapse_turn_allow_guests: False

#matrix_synapse_email_enabled: false
#matrix_synapse_email_smtp_host: ""
#matrix_synapse_email_smtp_port: 587
#matrix_synapse_email_smtp_require_transport_security: false
#matrix_synapse_email_notif_from: "Matrix <matrix@{{ matrix_domain }}>"
#matrix_synapse_email_client_base_url: "https://{{ matrix_server_fqn_riot }}"

matrix_client_elements_themes_enabled: true
matrix_server_fqn_element: "riot.{{ matrix_domain }}"

exim_relay_sender_address: "matrix@{{ matrix_domain }}"
exim_relay_relay_use: true
exim_relay_relay_host_name: "=== Redactd ==="
exim_relay_relay_host_port: 587
exim_relay_relay_auth: true
exim_relay_relay_auth_username: "matrix@aes-sb.de"
exim_relay_relay_auth_password: "=== Redacted ==="

matrix_user_uid: 900
matrix_user_gid: 900

matrix_dimension_enabled: true
matrix_dimension_admins: ['@=== Redacted ===:aes-sb.de']
matrix_dimension_access_token: "=== Redacted ==="

jitsi_enabled: false
#jitsi_enabled: true

jitsi_jicofo_component_secret: "=== Redacted ==="
jitsi_jicofo_auth_password: "=== Redacted ==="
jitsi_jvb_auth_password: "=== Redacted ==="
jitsi_jibri_recorder_password: "=== Redacted ==="
jitsi_jibri_xmpp_password: "=== Redacted ==="

# We only need this temporarily - until Jitsi integration in riot-web is finalized.
# Remove this line in the future, to switch back to a stable riot-web version.
#matrix_riot_web_docker_image: "vectorim/riot-web:develop"

matrix_appservice_slack_enabled: false
matrix_appservice_slack_control_room_id: "!xxxxxxxx:aes-sb.de"

matrix_mautrix_whatsapp_enabled: true

matrix_admin: "@christianwolf:{{ matrix_domain }}"

matrix_synapse_ext_password_provider_shared_secret_auth_enabled: true
matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret: "=== Redacted ==="

matrix_mautrix_telegram_enabled: true
matrix_mautrix_telegram_api_id: 1354678
matrix_mautrix_telegram_api_hash: "=== Redacted ==="

matrix_appservice_webhooks_enabled: false
matrix_appservice_webhooks_api_secret: '=== Redacted ==='
#matrix_appservice_webhooks_log_level: '<log_level>'

matrix_mautrix_facebook_enabled: true
matrix_synapse_configuration_extension_yaml: |
  enable_group_creation: true
  group_creation_prefix: "unofficial/"

matrix_mautrix_facebook_configuration_extension_yaml: |
  bridge:
    community_template: "unofficial/facebook_{localpart}={server}"

matrix_mautrix_whatsapp_configuration_extension_yaml:
  # Your custom YAML configuration goes here.
  # This configuration extends the default starting configuration (`matrix_mautrix_whatsapp_configuration_yaml`).
  #
  # You can override individual variables from the default configuration, or introduce new ones.
  #
  # If you need something more special, you can take full control by
  # completely redefining `matrix_mautrix_whatsapp_configuration_yaml`.
  bridge:
    displayname_template: "{{ '{{if .Name}}{{.Name}}{{else}}{{.Jid}}{{end}} {{if .Notify}}({{.Notify}}) {{end}}(WA)' }}"

    community_template: "{{ 'unofficial/whatsapp_{{.Localpart}}={{.Server}}' }}"
    history_sync:
      backfill: true
      request_full_sync: true

devture_postgres_connection_password: '=== Redacted ==='

matrix_coturn_turn_udp_min_port: 49152
matrix_coturn_turn_udp_max_port: 49252

matrix_homeserver_generic_secret_key: "{{ matrix_synapse_macaroon_secret_key }}"

matrix_playbook_reverse_proxy_type: playbook-managed-traefik

matrix_bot_matrix_reminder_bot_enabled: true
matrix_bot_matrix_reminder_bot_matrix_user_password: "=== Redacted ==="
# Adjust this to your timezone
matrix_bot_matrix_reminder_bot_reminders_timezone: Europe/Berlin

Matrix Server:

• OS: Archinux
• Architecture amd64

Ansible:
If your problem appears to be with Ansible, tell us:

I am running the following Ansible version:

ansible [core 2.16.4]
  config file = /home/christian/tmp/matrix-docker-ansible-deploy/ansible.cfg
  configured module search path = ['/home/christian/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.11/site-packages/ansible
  ansible collection location = /home/christian/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.11.8 (main, Feb 12 2024, 14:50:05) [GCC 13.2.1 20230801] (/usr/bin/python)
  jinja version = 3.1.3
  libyaml = True

I run this on a separate machine (also Archlinux).

Problem description:

I had an installation mainly based on fe9b72e. It was quite dated and I wanted to update the complete playbook.
I thus merged my local changes with the upstream master (c89e437 at the time being).

It seems to work in the first glance (I can browse the web frontend) but I am presented in the terminal with the information that a postgres update was needed. I carry out the update by means of just run-tags upgrade-postgres.

This process runs some time and at least terminates without any error message. However, I am no longer able to log into the matrix server. the password is no longer accepted.

All I see in the logs of synapse:

Apr 14 16:33:01 vps808958 matrix-synapse[116697]: 2024-04-14 14:33:01,632 - shared_secret_authenticator - 102 - INFO - POST-18 - Authenticating user `== Redacted ==` with login type `m.login.password`
Apr 14 16:33:01 vps808958 matrix-synapse[116697]: 2024-04-14 14:33:01,633 - shared_secret_authenticator - 113 - INFO - POST-18 - Bad hmac value for user: @== Redacted ==:aes-sb.de
Apr 14 16:33:01 vps808958 matrix-synapse[116697]: 2024-04-14 14:33:01,642 - synapse.handlers.auth - 1079 - WARNING - POST-18 - Attempted to login as @== Redacted ==:aes-sb.de but they do not exist

As I did not change any configuration but just updated postgres, I expect the login to be possible and I can continue to use the Synapse server. However, I am locked out (with the admin user) from my server.

I reread the README looking for any breaking changes. I did not find anything obvious. Apart from that, I am a bit lost on the problem. I miss a way to debug this as I do not know where to look, to be honest.

Additional context

Side remark:I ran into an issue while upgrading the DB. It failed due to storage restrictions. The postgres daemon was killed as no consistent data was present. I removed the failed data, increased the quota, restored the old data, and retried. Then it went through smoothly