Adding Face Recognition to My Smart Home with Frigate

As my smart home has grown more capable, one of the last “sci-fi” gaps has been true visual understanding: not just that someone is at the door, but who it is, what they might want, and how the house should respond. I already use sensors, presence detection, and complex Home Assistant automations inside my house, but cameras, which I only have outside the home, have mostly been passive. They are great for recording in case an incident happens but haven’t driven any intelligent behavior.

Frigate, an open-source NVR with powerful object detection, gave me the foundation I needed to change that. This project has been about layering face recognition and video-driven automations on top of my existing system so the house can react more intelligently to what it sees around it.

Goals

My primary goal was to develop an even faster way for me to be let back in my house after a walk. I already have connected a fingerprint sensor to my door locks, but the process is slow and has a high false-negative rate. I wanted to build something that would automatically unlock the doors as I walk up to the doorbell.

Some other goals I had include:

• Announcing which of my friends or family is at the door based off their face eg “John is ringing the doorbell, would you like me to let them in”

• Automations based on the presence of a car in my driveway (TBD)

• Delivery person detection triggering a different set of actions than a known friend or solicitor

Preparing the Hardware

For the hardware, I decided not to build a custom PC or try to run on Raspberry Pi and instead repurposed an off-the-shelf Intel NUC as the brains behind Frigate. The NUC form factor is perfect for this kind of always-on workload: small enough to disappear into my rack, but powerful enough to handle multiple camera streams, in particular, encoding / decoding video. Out of the box it wasn’t optimized for real-time video inference. To fix that, I opened up the NUC, pulled the stock Wi-Fi card, and replaced it with an NVMe Coral TPU module. That single change moves the heavy lifting of object and face detection off the CPU and onto dedicated silicon designed for exactly this kind of workload. The result is a much smoother system: detections come back faster and at higher frame rate. Drivers were a little tricky, but I eventually got it up and running and passed the TPU through to Docker for Frigate.

Wrestling with Cameras and RTSP Streams

On paper, this all sounds straightforward: point cameras at the right places, feed their streams into Frigate, add face recognition, and wire the events into Home Assistant. In practice, the first real challenge showed up at the camera layer.

My TP-Link cameras only support a single RTSP stream at a time. This was not obvious at first and took a lot of troubleshooting to discover. One camera stream is fine if you feed them to one system, but I was already using my cameras with UniFi Protect for recording and archiving. Adding Frigate meant suddenly needing two clients for the same video feed: UniFi Protect for centralized recording and Frigate for real-time object and face detection.

RTSP, unfortunately, doesn’t magically duplicate itself. With only one stream available, I had to make a tradeoff. Instead of recording into Unifi Protect, I attached a separate NVR-grade external HDD to frigate specifically to record those streams. This means that some of my camera streams are recorded on Unifi Protect and others on Frigate, but I feel the compromise is worth it. Frigate’s real-time analysis was more important than having everything under a single viewing app, especially because Unifi Protect does not support motion tagging or audio for third party RSTP streams.

Building a Face Library

With the camera feeds routed properly into Frigate, the next step was building a face library. The concept is simple: I train the model with my face and faces of trusted guests, then fine tune it by tagging people as they visit the door with the doorbell’s unique camera angle and lighting situation.

I am deliberately limiting the face library to a small set of trusted identities. Everyone else is treated generically as “person at the door,” which is usually sufficient. I am not trying to build a large-scale, long-term archive of faces: just enough to make the house feel smarter and more welcoming.

These events then flow into Home Assistant and drive announcements and actions. For example:

If doorbell_pressed and person_recognized: Freddy’s Mom → start conversation “Mom is at the door, would you like me to let her in?” on the nearest speakers

Safe Auto-Unlocking on My Face

The most tempting (and dangerous) automation is “If the camera sees Freddy, unlock the door.” Used naively, this is a bad idea. A static face photo, or a photo displayed on a phone could trick the system.

To mitigate that, my design for auto-unlocking is layered:

1. Face match

    Frigate/face recognition must identify me with high confidence.

2. Device presence

   My phone and watch must also be within a small GPS radius

3. Contextual checks

   I have a few other tricks up my sleeve to verify that it is me at the door, but I’m not going to share them with the internet

If my face shows up but there is a failure on the other checks, the door remains locked, a video clip is archived from all cameras and a notification is sent to my phone.

Reflections and Next Steps

I have a camera in my garage, and I have already been using it to track if the car is there or not, although I haven’t quite figured out what the automation should do. I already have a locally-hosted vision language model (VLM) running on my own hardware, and Frigate’s ability to pass camera stills from security cameras to a VLM will be a key capability unlock for building out a smarter home. Some ideas I already have are around guessing the intent of someone at the front door, but other ideas, such as identifying if I have taken out the trash, or identifying if someone is blocking my driveway can be useful tools as well. A log of animals visiting my garden could also be a fun experiment.