如何调整参数,让知识库问答接口支持更多的并发 #3989
-
如何调整参数,让知识库问答接口支持更多的并发。普通模式和使用vllm加速模式下,分别如何调整。 vllm主要参数:
|
Beta Was this translation helpful? Give feedback.
Replies: 2 comments
-
To enhance the concurrency support of your knowledge base Q&A interface, especially under VLLM acceleration, consider the following parameter adjustments:
These adjustments should be made with consideration to your hardware's capabilities and the specific requirements of your application. Testing changes in a controlled environment before production deployment is crucial to ensure system stability and performance.
|
Beta Was this translation helpful? Give feedback.
-
现在没有并发是因为langchain没有并发啊,没有做并发管理所以就是纯粹是异步代码而已,改参数没用的,要改整个框架 |
Beta Was this translation helpful? Give feedback.
现在没有并发是因为langchain没有并发啊,没有做并发管理所以就是纯粹是异步代码而已,改参数没用的,要改整个框架