-
Notifications
You must be signed in to change notification settings - Fork 392
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Phi3 support #1826
Comments
optimum/optimum/utils/normalized_config.py Line 254 in 56aabbe
Add |
patching the from transformers import AutoTokenizer
from optimum.exporters import TasksManager
from optimum.exporters.onnx import main_export
from optimum.onnxruntime import ORTModelForCausalLM
from optimum.utils import NormalizedConfigManager
TasksManager._SUPPORTED_MODEL_TYPE["phi3"] = TasksManager._SUPPORTED_MODEL_TYPE["phi"]
NormalizedConfigManager._conf["phi3"] = NormalizedConfigManager._conf["phi"]
# output = "phi3_onnx"
# main_export(
# model_name_or_path="microsoft/Phi-3-mini-4k-instruct",
# task="text-generation-with-past",
# trust_remote_code=True,
# output=output,
# )
model = ORTModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct", trust_remote_code=True, export=True)
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
inputs = tokenizer(["Hello, my dog is cute"], return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True)) |
Although I had following patch applied, it still cann't export Phi-3 ONNX model. commit db6db6f
My command is as follows: The output and error are as following: |
With the PR #1841 submitted by the ORT team, we will be able to load onnx checkpoints(eg. For phi3 small, the team will wait until it becomes a part of a stable transformers release (it is using remote code now). |
Feature request
Microsoft's new phi3 mode, in particular the 128K context mini model, is not supported by Optimum export.
Error is:
"ValueError: Trying to export a phi3 model, that is a custom or unsupported architecture, but no custom export configuration was passed as
custom_export_configs
. Please refer to https://huggingface.co/docs/optimum/main/en/exporters/onnx/usage_guides/export_a_model#custom-export-of-transformers-models for an example on how to export custom models. Please open an issue at https://github.com/huggingface/optimum/issues if you would like the model type phi3 to be supported natively in the ONNX export."Motivation
Phi3-mini is potentially very significant as it has a large context but a small size. This could be used in lots of scenarios if it has good performance.
Your contribution
Unlikely I could do a PR as ONNX work is not my forte.
The text was updated successfully, but these errors were encountered: