Need Efficient Solution for Slow Collection Restoration in Qdrant

[ruben-quilez]

Hi!
I am currently using Qdrant for vector storage and have encountered a critical issue.

When I take a snapshot of a collection, the process is quick and the download is fast. However, when I attempt to upload and restore the collection, the process becomes extremely slow. For example, I tested this with a collection that is several gigabytes in size, and it took hours to complete the upload.

If for any reason (due to any problem), I need to recreate a collection, it takes an excessive amount of time, leaving my service unavailable for an unacceptably long period.

I have been following the documentation provided here: https://qdrant.tech/documentation/concepts/snapshots/. Is there any faster method available?

This is a critical issue for us, and without a resolution, it significantly undermines the reliability of Qdrant for our operations. I would greatly appreciate our solution to this problem.

Thank you!

Rubén

[generall]

hey @ruben-quilez, how do you upload a snapshot? Do you upload it from your local machine or some server?

[ruben-quilez]

From my local machine to some server. With this method:

def recover_snapshot_from_file(self, file_path: str) -> dict:
        """
        Recovers a collection from a snapshot file.
        
        Args:
            file_path (str): The local path of the snapshot file.
        
        Returns:
            dict: The response from the server.
        """
        url = f"{self.qdrant_url}/collections/{self.collection_name}/snapshots/upload?priority=snapshot"
        headers = {
            'api-key': self.api_key,
            'Content-Type': 'multipart/form-data'
        }
        files = {'snapshot': open(file_path, 'rb')}
        response = requests.post(url, headers=headers, files=files, verify=False)
        response.raise_for_status()
        return response.json()

[ruben-quilez]

One of the main reasons for using snapshots is to be able to restore a backup quickly. In the specific case I mentioned, I set up another Qdrant instance on a different machine and server. My goal is to "migrate" the collections from one instance to another.

[generall]

From my local machine to some server.

Might it be, that your internet connection is just asymmetric for download and upload? It is usually the case

[ruben-quilez]

The same thing happens when I upload them to a local machine (from local also)

[ruben-quilez]

In any case, the current collection size is 1.2 GB and contains 199,628 points. This shouldn't be a problematic size. In the future, we plan to scale up significantly, handling about 100 times this amount.

[adantart]

I confirm the same issue.
Restoring a 2Gb collection (with about 300.000 points) even into local is a nightmare.
I'm really worried the day I had to restore

[adantart]

I confirm this:

enabling api_key and tls, 2Gb of restoration takes about 20 minutes
disabling it, it takes 15 seconds.
Definetely, there must be a bug in the TLS handshake uploading loop ...

Please, Qdrant, take a look !

[generall]

thanks for reporting, we will try to reproduce on our side

[tellet-q]

@ruben-quilez Thank you for the code snippet. I've noticed that you explicitly pass 'Content-Type': 'multipart/form-data' header into the request. Could you please avoid doing that and see if the issue still reproduces? So, the final code will look like this:

        headers = {
            'api-key': self.api_key
        }
        files = {'snapshot': open(file_path, 'rb')}
        response = requests.post(url, headers=headers, files=files, verify=False)

I tried to reproduce your issue on our side. In my case everything works fast with or without TLS if I don't provide the header, but when I do the upload takes ages. It's not Qdrant's issue. The requests library provides multipart/form-data header and calculates boundary value on its own when files arg is provided. If user provides header explicitly then the library does not calculate the boundary value and that may lead to a longer upload time.

[Faisalee50]

hi @ruben-quilez , I am getting issue when i vector count goes above 300k+, it comes automatically to zero, without having any log on qdrant server, can you please tell me what is the size of your data ? and did you get any kind of this behaviour? and what is your docker compose configuration , if you can share will be very helpful