Question regarding 2.3's LLM fallback

koinsxc
Oct 2026
5 posts
0 likes
Member
Question regarding 2.3's LLM fallback
Posted Sun Oct 25th 02:04

Specifically I'm looking for assistance with the following from the release notes

LLM fallback now has web search and web fetch functionality, allowing you to get the latest sports scores, current events, etc. You need to get a free API key from https://ollama.com/ and enter it in the Settings menu of the localhost web interface.

Is there documentation on what setup is involved for getting this working? I'm on 2.3.2 and I have an Ollama API key in the LLM credentials section. I've tried switching between Providers (though Ollama seems the intuitive answer) and using / not using the LLM trigger phrase to try to force route questions to web search, but I haven't had any luck. If someone has gotten that route working, could you share what you did to get there? Is it pairing the LLM fallback with custom intent sentences, or something else?

Thank you!

Matt
Apr 2025
94 posts
0 likes
Member
Posted Sun Oct 25th 08:50

Does LLM itself work? If you ask something like what's the distance to the sun, does it correctly hit the LLM?

You said you're using Ollama. Does Ollama have access to the internet?

Send in a message such as "is it going to rain in Berlin later today?". Then visit the Settings -> Apps menu of Home Assistant, open Sophia NLU, and click the Log button. Do you see any lines there saying web fetch tool called?

I can't quite tell exactly what the issue would be from your message, but web search and fetch does work, and confirmed by multiple customers. Let me know on the above, and I'll do my best to assist further.

koinsxc
Oct 2026
5 posts
0 likes
Member
Posted Sun Oct 25th 19:51

Maybe I'm not understanding the intended setup for that feature based on your questions. With Ollama selected, when I ask about rain in berlin I get an unknown intent HassError (shown below).

text: is it going to rain in berlin today
  intent HassError:
    error: Unknown intent HassError
response: an unexpected error occurred: Unknown intent HassError

With none selected, I get something slightly different

text: is it going to rain in berlin today
  intent HassRespond:
response: sorry, can you say that differently?

Am I supposed to be running a model to fall back to? If so, what is the purpose of the API key that the instructions said I'd need. Apologies because I can tell I'm missing something pretty obvious but I don't see what it is

Matt
Apr 2025
94 posts
0 likes
Member
Posted Sun Oct 25th 22:37

Hmmm... it's not even hitting the LLM for some reason.

In the Settings menu of the Sophia web interface set the API URL to: http://127.0.0.1:11434/api/chat"

Ensure the IP address is correct for your setup and that Sophia can connect to it.

In the next patch update I'll include a fix that speaks the error when LLM fallback failed. I added that in previously, but obviously it's not working. Will ensure it's fixed for the next upgrade.

Matt
Apr 2025
94 posts
0 likes
Member
Posted Mon Oct 26th 02:54

Patch to v2.3.4 is now out. Please upgrade the app and the speech response will now include the error message as to why the LLM is failing.

Let me know how it goes.

koinsxc
Oct 2026
5 posts
0 likes
Member
Posted Mon Oct 26th 16:06

We're getting somewhere! It still errors but now I can see why

[2026-10-26 12:01:54] text: will it rain in athens today
    intent HassRespond:
  response: i ran into a problem: Unable to connect to server at localhost:11434

[2026-10-26 12:02:19] text: will it rain in athens today
    intent HassRespond:
  response: oops, something went wrong: Unable to connect to server at 127.0.0.1:11434

FWIW this setup is HA on a Raspberry Pi - I'm not running Ollama on the Pi unless it's packaged up in the new versions of Sophia. Am I supposed to be doing that?

Matt
Apr 2025
94 posts
0 likes
Member
Posted Mon Oct 26th 18:02

Nope, Ollama can be on a different machine than HA.

You simply need to figure out the local IP of the machine Ollama is on. It'll be something like 192.168.0.24 or maybe 10.0.1.4, or something.

If you're running Linux (or Mac I think) it's /sbin/ifconfig to find out. Windows, I have no idea, ask Claude.

Once you have the IP address goto the Settings menu of the Sophia web interface and update the LLM API URL. Change the 127.0.0.1 out with the new IP address, and it should begin working fine.

Let me know how it goes.

koinsxc
Oct 2026
5 posts
0 likes
Member
Posted Mon Oct 26th 21:59

Gotcha. Before I do that, and I know this is probably a dumb question, but what is the API key from https://ollama.com/ for (from the release notes)?

Is the idea that I have an LLM running that has web search as a tool, and it just happens to be that you chose Ollama? I ask because I'd consider running a model that isn't Ollama and relying on Ollama just for the web search part, but I'm not sure if that is intended to work.

Thank you for your guidance!

Matt
Apr 2025
94 posts
0 likes
Member
Posted Mon Oct 26th 22:56

Yep, that's fine. You can use any LLM rovider you wish, it's just that it will still use Ollama's API for any web search calls. You definitely don't need to use Ollama as your LLM provider in order for web search to work though.

The reason I chose Ollama for web search is because API keys are free and don't require a credit card, whereas Brave search requires a credit card to get a API key.

Quick Reply

You must be logged in to reply to this thread.

Login Register