Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pre-ensure that downloading a file wont exceed the configured/available disk space #196

Open
zeeshanakram3 opened this issue May 28, 2023 · 0 comments
Labels
enhancement New feature or request YPP

Comments

@zeeshanakram3
Copy link
Contributor

Problem

A few definitions first:

  • Total Disk Space: Total disk space available on the machine (instance had 200 GB).
  • spaceAvailable: Total disk space configured to be made available for YT-synch service(configured to 100GB)
  • spaceUsed: Space already used by all the downloaded videos, would be less than spaceAvailable

Right now youtube-synch downloads videos in a batch (of 50, which is configurable) for postprocessing, i.e. merging audio/video, calculating hash, etc. And before start downloading the batch it only ensures that some space is available (i.e. spaceUsed < spaceAvailable)

The problem with the above approach is that the size of the prepared batch might actually exceed not only the availableSpace but all the system disk space too, and it could lead to service crash.

Fix

Pre-calculate the size of the batch & only queue it for downloading if the space would not exceed even after all the videos have been downloaded. i.e. (spaceUsed + batchSize <= spaceAvailable)

@dmtrjsg dmtrjsg added the YPP label Jun 7, 2023
@dmtrjsg dmtrjsg mentioned this issue Sep 6, 2023
6 tasks
@dmtrjsg dmtrjsg added the ypp2.0 label Sep 13, 2023
@dmtrjsg dmtrjsg added enhancement New feature or request and removed ypp2.0 labels Sep 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request YPP
Projects
None yet
Development

No branches or pull requests

2 participants