[Misc]: can vllm support long content inference like 800k

[yunll]

Anything you want to discuss about vllm.

we will finetune a 70B model that support long content with 800k, can vllm support to inference this model?

[mgoin]

Hi @yunll, yes if you have enough GPU memory available for a context length that large it will run. I tested Mistral 128k last week and was able to use its full length. Model for reference: https://huggingface.co/NousResearch/Yarn-Mistral-7b-128k