Hey Jarvis: A Web Page I Can Talk To

Jarvis is the AI assistant that looks after my homelab. For weeks I only typed to it in the Claude app, and I wanted to just talk to it. So Jarvis and I built a web page for it: a dark screen with a glowing orb of particles. I say “Hey Jarvis”, the orb reacts while it listens, then it thinks and answers out loud. It’s the same Jarvis as the one in the app, with the same tools, the same memory and the same rules.

In the clip I say “Hey Jarvis”, ask what the time is, and then ask whether any of my servers need patching. After a short “One moment”, Jarvis checks and replies that none do. All twelve servers have no pending updates, no reboots waiting and no failed services, Home Assistant has nothing waiting either, and the fullest disk is on the containers server at 74%.

What happens when I speak

The page handles all the audio, in and out, so everything below happens in a browser tab.

1. Listening for the wake word. The wake word model, openWakeWord, runs inside the page itself, so no audio goes anywhere until it hears “Hey Jarvis”. I can also hold the space bar to talk. For seven seconds after a reply I can follow up without saying the wake word again, and saying “Hey Jarvis” while it’s talking makes it stop.
2. Recording. The page records until I pause for about 0.7 seconds.
3. Speech to text. The recording goes to Whisper running on my own hardware.
4. Thinking. A long-lived Claude session on the Jarvis VM works out the answer. Faint labels under the orb show what it’s doing, such as checking the servers.
5. Speaking. Piper turns the reply into speech, using a Piper voice that sounds like the Jarvis from the films, and the page plays it.

Making it fast enough to use

Speed was my main complaint after the first version. Whisper and Piper together take under a second. The thinking step takes between 2.5 and 6 seconds a request, and switching to a different model barely changed that. So I worked around it instead.

The Claude session stays running between requests, so there’s no start-up cost each time. Questions like the time and date are answered directly without asking Claude at all. And if the answer is slow, the page says “One moment” after about 1.8 seconds so I know it heard me.

More than a voice

The page does a few other jobs, which is why it ended up as a page and not just a microphone:

• Jarvis can put things on the screen: an image, a screenshot of a web page or a camera snapshot. Saying “close that” clears it.
• Alerts appear as a banner and tint the orb. The doorbell, the leak sensors and the smoke alarm come in live from Home Assistant, and so do warnings and failures from Jarvis’s own notifications. Alerts are only spoken while the page is in use and between 07:00 and 23:00. A leak or smoke alarm speaks at any hour.
• After 40 seconds of nothing, the orb shrinks and the page shows the clock, the date, the weather, my next calendar event and the health of the servers.
• A menu option shares my screen, and the page then sends Jarvis a fresh picture of it with each request, so I can ask about whatever I’m looking at.

Who it listens to

The page has a username and password, and it only works on my home network and over my VPN, so there’s no link here. On top of that it can learn my voice. I read five sentences, it stores a fingerprint of my voice and not the recordings, and I can tell it to answer only to me. That stops visitors and the TV from giving it orders. It sits on top of the login and doesn’t replace it.

The risky jobs never happen by voice. That means rebooting Proxmox, Home Assistant, the router or the NAS, and anything to do with locks, the alarm or the garage door. For those, Jarvis says it needs my OK, and the request lands in the Jarvis session on my phone marked as unverified. Nothing happens until I confirm it there, and the voice lock doesn’t change that.

Things I ran into

• Browsers only let a page use the microphone over https, so the page needs a proper secure address even though it’s only reachable at home.
• The wake word model only knows “Hey Jarvis”. A bare “Jarvis” doesn’t trigger it.
• The wake word model’s licence is Creative Commons non-commercial, which is fine for a personal project like this one.
• The microphone, screen sharing and voice learning can’t be tested with an automated browser. For those I had to sit in front of a real one and try them.

I built the page with Claude over a couple of days, and the first version only worked with a typed request. Voice came after that, then the screen panels, the alerts and the login.

Got a question about this, or built something similar? Get in touch. New write-ups go out on the RSS feed and LinkedIn, everything I've built is on the projects page, and my free calculators and guides are on the tools page.