Mistral rs python binding error #323
Comments
|
Thank you for reporting this. Can you please try to run an example from the command line, for example the below, which loads a GGUF model as configurable in the .toml file here:
I think that if this works, we can isolate the issue with the Python bindings and not a general issue.
You can run |
|
Thank you for your response. I tried running: But I get the below error: .\mistralrs_server : The term '.\mistralrs_server' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is
I am guessing I am either not running it from the right directory or I need to add something to PATH before running it. Can you tell me where I must run you |
|
Ok, sorry I wasn't more clear. If you go to the |
|
I was able to run the command. I got an error, this is my logs from running the command : cargo run --release --features cuda -- -i toml -f toml-selectors/gguf.toml Updating crates.io index |
|
@shresht8, sorry for the wait. coreylowman/cudarc#240 should fix this, I'll let you know when it gets merged. |
|
Hi @shresht8, I think it should be fixed now. Can you please try it again after a |
Bug:
I am attempting to run mistral rs for inference for my own GGUF files but before that I wanted to test with the example given in the documentation. I installed mistral rs using ubuntu 22.04 since I have a windows machine with RTX 4070 GPU. I installed the dependencies required as well using ubuntu such as the libssl-dev and pkg-config. Once that was done I went into my conda environment and ran the pip install mistralrs-cuda command to install the package. It ran successfully without any errors so I assumed the installation was complete. Is there any verification process for the installation?
Once the installation was complete, I successfully imported the packages and then attempted to run the example given in the documentation given for the python bindings:
runner = Runner(
which=Which.GGUF(
tok_model_id="mistralai/Mistral-7B-Instruct-v0.1",
quantized_model_id="TheBloke/Mistral-7B-Instruct-v0.1-GGUF",
quantized_filename="mistral-7b-instruct-v0.1.Q4_K_M.gguf",
tokenizer_json=None,
repeat_last_n=64,
)
)
and got the below error:
PanicException Traceback (most recent call last)
Cell In[2], line 1
----> 1 runner = Runner(
2 which=Which.GGUF(
3 tok_model_id="mistralai/Mistral-7B-Instruct-v0.1",
4 quantized_model_id="TheBloke/Mistral-7B-Instruct-v0.1-GGUF",
5 quantized_filename="mistral-7b-instruct-v0.1.Q4_K_M.gguf",
6 tokenizer_json=None,
7 repeat_last_n=64,
8 )
9 )
PanicException: called
Result::unwrap()on anErrvalue: LoadLibraryExW { source: Os { code: 126, kind: Uncategorized, message: "The specified module could not be found." } }I attempted to run the actual mistral model as well (not the GGUF version) like this:
runner = Runner(
which=Which.Plain(
model_id="mistralai/Mistral-7B-Instruct-v0.1",
arch=Architecture.Mistral,
tokenizer_json=None,
repeat_last_n=64,
)
)
But I got the same error as above. Can anyone help me understand the cause of this error?
Latest commit
commit 455653c (HEAD -> master, origin/master, origin/HEAD)
The text was updated successfully, but these errors were encountered: