Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CUDA out of memory #18

Open
zhibaishouheilab opened this issue Nov 29, 2022 · 2 comments
Open

CUDA out of memory #18

zhibaishouheilab opened this issue Nov 29, 2022 · 2 comments

Comments

@zhibaishouheilab
Copy link

Hello!
Thanks for your excellent work! I'm just learning the pytorch and unet network. When I ran your scripts I got the following error:
OutOfMemoryError: CUDA out of memory. Tried to allocate 226.00 MiB (GPU 0; 4.00 GiB total capacity; 3.24 GiB already allocated; 0 bytes free; 3.27 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
The error was got in the step of training. And there were no other process hobbling the gpu.
Thanks for your reply.

@shahzad-ali
Copy link

The provided information is too generic. It simply means your GPU memory is low and cannot fit the model and its parameters during the training process. You may try

(ordered list)

  • lowering the batch size
  • using a smaller number of features maps (e.g., 32 instead of 64 in Conv2d-1; make subsequent changes accordingly)
  • reducing the input image size either by using patches or downscaling the entire input image
  • upgrading to a better GPU with more memory

Good luck!

@aymuos15
Copy link

aymuos15 commented May 8, 2024

#19

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants