Ambilight is a discontinued feature on early Phillips flat panel TVs that used rear-facing fluorescent tubes to illuminate a wall behind the TV. The eight lights would change color to imitate the average of the edge pixels in their eighth of the screen’s perimeter. The effect of this is that the area behind the screen appeared to be an extension of the displayed image.
I decided to create my own version of system using 86 individually addressable LEDs (thus increasing the spatial resolution) powered by an overclocked Raspberry Pi. This project took a long time to complete in part because of delays shipping hard-to-find parts, but also because of difficulty in reading the signal over a USB connection, analyzing it, and driving the LEDs fast enough to avoid noticeable latency. This required testing multiple video capture chipsets (some of which could compress data before sending it over USB), overclocking the Raspberry Pi (and attaching heat sinks to compensate), and implementing color-correction filters to compensate for quality issues in the HDMI to AV conversion.
Video is first aggregated to a single device (in this case, an Xbox One (although a HDMI switch could work just as well). The final HDMI signal is then split in an HDMI splitter, and one of the outgoing signals goes through a HDMI to AV converter (the other signal goes to the screen). The AV converter substantially reduces image quality, but since we only care about edge pixels, it doesn't matter (color inaccuracies have to be compensated for, however). This AV signal can then be read using a USB video capture device, compressed, and sent to the Raspberry Pi that analyzes the edge pixels and drives the LEDs.