-
Notifications
You must be signed in to change notification settings - Fork 5.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[serve] vllm example to serve llm models #45430
Conversation
Signed-off-by: can <can@anyscale.com>
Signed-off-by: can <can@anyscale.com>
4c067d7
to
3c8ca11
Compare
Signed-off-by: can <can@anyscale.com>
@MicroCheck linux://doc:source/serve/doc_code/distilbert linux://doc:source/serve/doc_code/object_detection linux://doc:source/serve/doc_code/stable_diffusion Signed-off-by: can <can@anyscale.com>
@MicroCheck //doc:source/serve/doc_code/distilbert //doc:source/serve/doc_code/object_detection //doc:source/serve/doc_code/stable_diffusion Signed-off-by: can <can@anyscale.com>
@akshay-anyscale, @edoakes i managed to create an environment for the test to run but it fails for some other reasons https://buildkite.com/ray-project/microcheck/builds/237#018f8c35-e5a1-443d-8cf9-bbb481af6c1e/177-2429; if this makes sense feel free to change this pr, thankkks |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this just a PoC, or is this intended to get merged?
@aslonnie intended to get merged, but will need serve folks to pick up and finish the job ;) |
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Pushed a commit to change the dtype, hopefully that fixes things. |
Signed-off-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com>
Signed-off-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com>
Is ray-llm going to be deprecated and this example will be the recommended way to run vllm on Ray? |
Signed-off-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com>
Signed-off-by: can <can@anyscale.com>
Signed-off-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com>
Signed-off-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com>
@angelinalg do you mind help review the doc content pieces, thankks |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
some style nits.
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
Adds a documentation example using vLLM to serve LLM models on Ray Serve. This is a copy of ray-project#45325 + add a build environment for ray serve + vllm. Test: - CI --------- Signed-off-by: can <can@anyscale.com> Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com> Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Co-authored-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Adds a documentation example using vLLM to serve LLM models on Ray Serve. This is a copy of ray-project#45325 + add a build environment for ray serve + vllm. Test: - CI --------- Signed-off-by: can <can@anyscale.com> Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com> Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Co-authored-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Adds a documentation example using vLLM to serve LLM models on Ray Serve. This is a copy of ray-project#45325 + add a build environment for ray serve + vllm. Test: - CI --------- Signed-off-by: can <can@anyscale.com> Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com> Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Co-authored-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Adds a documentation example using vLLM to serve LLM models on Ray Serve. This is a copy of ray-project#45325 + add a build environment for ray serve + vllm. Test: - CI --------- Signed-off-by: can <can@anyscale.com> Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com> Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Co-authored-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: gchurch <gabe1church@gmail.com>
Adds a documentation example using vLLM to serve LLM models on Ray Serve.
This is a copy of #45325 + add a build environment for ray serve + vllm.
Test: