Introduction
Hey everyone, I’ve been experimenting lately with bridging Canonical Inference Snaps and Home Assistant. The dream here is a completely local, privacy-first, smart home AI that is as easy to spin up as a snap install.
While Home Assistant has been pushing hard into local Voice Assistants and AI integrations, connecting the two ecosystems isn’t quite plug-and-play yet. I wanted to share some notes on what works today, where the friction is, and an awesome use-case I’ve been playing with.
What works: Getting Home Assistant talking to an inference snap
Currently, Home Assistant (HA) supports OpenAI and Ollama endpoints out of the box for its AI pipeline. To use our inference snaps (like the gemma4 snap with the 12B model), you have to bypass the official integrations.
Because the official HA OpenAI and OpenRouter integrations do not allow configuring the server address and API base path, these could not be pointed to an inference snap.
The solution right now is a third-party integration via HACS called Local OpenAI LLM. By pointing this integration to the Gemma 4 snap, it works! I can open a chat window in Home Assistant and ask it to switch on specific lights.
The friction points
While it works, there are a few rough edges if we want Inference Snaps to be a first-class citizen in the smart home:
Voice and speech-to-text (STT) mapping is tricky: In my testing, adding the whisper-asr snap as a service to the HACS integration failed. Based on my experiments, this appears to be because the snap’s OpenAI API doesn’t yet implement the /v1/models endpoint, though it’s also possible Home Assistant (HA) expects an endpoint to provide the entire STT-TTS-LLM pipeline. Regardless, HA natively prefers the Wyoming protocol for local voice, so exposing a Wyoming endpoint from these snaps could be an interesting area to explore in the future.
Reasoning Speed: During my testing, reasoning capabilities were invaluable for complex tasks like security video classification, but introduced noticeable latency for simple smart home commands. Using a 12B model that pauses to “think” before turning on a light creates a sluggish user experience in practice. A potential solution could be a hybrid approach: smaller, fast models dedicated strictly to tool-calling alongside larger models for deep analysis, or exploring the ability to dynamically toggle reasoning (such as through chat template configuration) depending on the specific task.
The killer use case: AI-powered security triage of camera footage
Where this setup truly shines isn’t just turning on lights, it’s using the local LLM as a building block in HA Automations via “AI Tasks”.
Here is the scenario I am testing:
Frigate (NVR) detects a “person” on a camera in my garden.
An HA automation grabs a snapshot frame and feeds it to the AI Task.
The prompt asks the model to triage the image to see if it’s a real threat or a false positive (e.g., a shadow, or just a delivery driver).
This camera shows the home's patio and glass entrance. Decide whether the homeowner should check the person, using a cautious security threshold.
🚨 Check if a person is close to the glass door/window, looking toward or through it, touching the entrance, has their hood up or face covered, or is holding/carrying something on the patio. Any one cue is enough. You do not need to identify the held object or prove harmful intent. A possible phone or drink must not override this review rule. Do not assume the person is authorized.
❎ No visible concern only if no person is visible, or a visible person has none of these cues. Describe visible evidence; do not invent actions or objects.
Reply with one short line:
🚨 Check: <person and visible cue>
❎ No visible concern: <ordinary scene>
⚠️ Unknown: <image cannot be assessed>
Based on the strict text output (e.g., 
Check: A person looking through the window vs 
No visible concern: A shadow moving), HA decides whether to send a critical notification to my phone.
Doing this entirely locally, without sending camera feeds to a cloud API, is incredible.
What do you think?
We’re just scratching the surface of what local AI can bring to smart homes, and we’d love to make this a collaborative effort. Have you tried running Inference Snaps in your own Home Assistant setup? Whether you’ve experimented with custom integrations, found workarounds for audio/STT pipelines, or have ideas for improving native OpenAI API compatibility, we want to hear from you!
Let’s brainstorm together. What other creative automation workflows or local AI use cases would you like to see? Jump into the discussion below and share your test results, ideas, or feedback!