How do you track someone's breathing in realtime, without attaching anything to their body or even requiring them to touch anything at all?

We solved this using a combination of thermal cameras and machine learning. In infrared, it's easy to spot the temperature change in a person's nostrils when they breathe in and out; the tricky part was getting a computer to recognise this phenomenon and reliably classify "inhale" and "exhale" states a few times a second.

I used the popular YOLO Real-time Object Detection system as a basis. We captured videos of "people breathing" in our offices and produced thousands of black-and-white frames to serve as training data. The manual labelling process took about 3 days, and training on a RTX 2070 took about 2 hours.

The model performed exceptionally well, and with some conversion into TensorRT format, I managed to get it running at a reliable 9 frames per second (the maximum refresh rate of the camera we had) on relatively inexpensive Nvidia Jetson Nano single-board computers. These tiny, low power machines feature specialised Edge-AI processing hardware and a custom Linux distribution.

In parallel, all kinds of other software had to be built (in React/TypeScript, Rust, NodeJS) and hardware integrated (moveable DMX fixtures, projectors, iPads and a Mac Mini). The installation had to run completely unattended in the store for about 6 weeks, so testing and remote monitoring was essential.

To service this project's lighting needs, I wrote my own DMX fixture control software, which you can read about here.

Two GLP Impression X5 "wash" lights and two Cameo Opus S5 spots were controlled by a single NodeJS service which could trigger scenes as well as adjust individual parameters based on realtime breathing information. It felt magical to stand inside the sphere and have the lights pulse in and out in time with your breathing.

More details on this project at https://random.studio/projects/a-sensorial-store-takeover-for-nike-house-of-innovation