You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When I attempt using the Unsloth Phi-4-mini-instruct model from HuggingFace it throws an access violation on llama_decode in SafeLLamaContextHandle. I am using the InstructExecutor example from the documentation. Inference works with other GGUF models, so I'm wondering why this particular one doesn't? It's only 25 days old, so I don't think the model age is the issue.
public DecodeResult Decode(LLamaBatch batch)
{
if (batch.TokenCount == 0)
{
return DecodeResult.Ok;
}
lock (GlobalInferenceLock)
{
LLamaNativeBatch batch2;
using (batch.ToNativeBatch(out batch2))
{
return (DecodeResult)llama_decode(this, batch2); // Access violation here
}
}
}
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
When I attempt using the Unsloth Phi-4-mini-instruct model from HuggingFace it throws an access violation on llama_decode in
SafeLLamaContextHandle
. I am using the InstructExecutor example from the documentation. Inference works with other GGUF models, so I'm wondering why this particular one doesn't? It's only 25 days old, so I don't think the model age is the issue.Beta Was this translation helpful? Give feedback.
All reactions