Is dspy.OllamaLocal here to stay?

[NumberChiffre]

Hey guys,

I've missed out the latest updates over the past 1-2 months. I remember update from v2.5 that we will remove dspy.OllamaLocal and all else to be replaced by dspy.LM which uses litellm under the hood.

For those of us using local models, it seems we still need dspy.OllamaLocal since the newest llama and qwen models are not supported from litellm: https://docs.litellm.ai/docs/providers/ollama#ollama-models

Even for the local models that are supported, the structured format fails frequently with dspy.LM, while dspy.OllamaLocal works for most cases.

Could you share any forward guidance for those of us relying on local models for your roadmap?

[okhat]

Hey @NumberChiffre ! LiteLLM supports any Ollama model. I use Llama 3.2 on Ollama.

What are you seeing in terms of failure cases?

import dspy
lm = dspy.LM('ollama_chat/llama3.2', api_base='http://localhost:11434', api_key='')
dspy.configure(lm=lm)

[okhat]

@NumberChiffre Maybe you passed ollama/ instead of ollama_chat/ to LM?

[NumberChiffre]

@okhat Thanks for the suggestion, actually it worked with ollama as provider for llama3.2, the problem is it fails with structured output which refers to this issue

[okhat]

@NumberChiffre Please try ollama_chat. I think litellm makes a big distinction between the two. I doubt it'll fail often with structured outputs, but let me know if it does and if you have an example!

[NumberChiffre]

@okhat Yeah it doesn't fail frequently, I haven't run enough tests yet. Could there be any permanent fix to this issue: #1539

[okhat]

If you have a self-contained failure (even one is helpful), like inputs + signature -> fail, we'd love to take a look. Especially with ollama_chat/* since ollama/* is probably just not processing the inputs correctly based on a few anecdotes from other users.