Empty payload after version upgrade

[mautini]

We have a Qdrant cluster installed using the official Helm Chart with 3 nodes. After the upgrade from 1.7 to 1.8 (rolling upgrade, one node at a time), I noticed that for some points in our collections, payloads were missing on some nodes. By hitting the /collections/<collection_name>/points/ endpoint, sometimes I got a payload and sometimes not, depending on the node that responded.

• After deleting and reindexing all our data, the problem disappeared.
• I never noticed this issue before, whether during an upgrade or not.
• Our payloads are stored on disk with indexing for some fields.
• I don't have a precise count / % but it didn't affected a lot of points.

Current Behavior

After an upgrade, some points in our collections had an empty payload. This issue was present only on some nodes in our cluster.

Steps to Reproduce

I was not able to reproduce it. It would require:

• Setting up a complete cluster
• Indexing a lot of points with payloads (we have more than 600k points)
• Upgrading the cluster
• Checking each node and each point to see if the payload is now empty

Expected Behavior

All points retain their payload on each node after an upgrade.

Possible Solution

Reindexing all the data after the upgrade solved the issue, but it’s not very convenient.

Context (Environment)

Our search was affected as we use the payload for filtering.

I know that my bug report is very vague and I’m pretty sure this issue isn’t trivial to solve. Do you have any advice or suggestions on what I can do during the next upgrade to avoid this issue or provide more information (logs?) about it?

Thanks,

[generall]

Hey @mautini, I would recommend to use the latest version of the engine, as it contains some improvements for consistency, which might be relevant in your case

[mautini]

Thanks for your response.

I plan to migrate to the latest Qdrant version this week, so I’ll monitor closely to see if we encounter the same issue and let you know.

[mautini]

I just upgraded on 1.9.2 and I can confirm that all points have payloads, so the issue seems to be fixed with the latest version (3 nodes, shard_number: 3, replication_factor: 2)

I close the issue, thanks!

[vladimircape]

I have v1.9.2 and also a lot of Points was with empty payload and vector
example
ScoredPoint(id=2042701, version=32070, score=0.6048526, payload={}, vector=None, shard_key=None), ScoredPoint(id=2042670, version=32069, score=0.6039498, payload={}, vector=None, shard_key=None), ScoredPoint(id=2042702, version=32070, score=0.6039498, payload={}, vector=None, shard_key=None), ScoredPoint(id=2042686, version=32069, score=0.6039498, payload={}, vector=None, shard_key=None), ScoredPoint(id=2042789, version=32071, score=0.60169375, payload={}, vector=None, shard_key=None), ScoredPoint(id=2042799, version=32071, score=0.60169375, payload={}, vector=None, shard_key=None),
and if they are empty WHY SCORE such big 0.6