smart mirror with voice, gesture control and a full OS to go with it

I built a smart mirror with voice, gesture control and a full OS to go with it

Finished product

This is how it looks like! Video here: www.youtube.com/watch?v=8swdfbnZn9E

I built a smart mirror that does voice recognition with a built-in microphone and gesture recognition with two distance sensors on each side of the frame.

I also made a small operating system that you can install apps on and it's really easy for developers to build apps for. The software I built runs on Electron (a web app engine) and provides APIs for developers to access the sensors and the microphones. However I have not published it yet because it needs some refinements.

Features:

– Voice control: When you clap your hands a microphone detects it and the voice recognition starts. You can say commands to launch apps such as "shakira video" to launch YouTube with a video of shakira, "play the radio" to launch the radio app, "show me the news" for the news app or "weather" for the weather.

– Gesture control: You can use gesture control in apps to navigate through the content or to play games. For example if you search for "pictures of cats" you can navigate through the pictures.

– Smartphone remote control: With a smartphone app I made you can see the list of all the installed apps and launch and interact with them.

The whole project cost about 400€. Here you can see some slides with more info: docs.google.com/presentation/d/1g7epekWB7_d-9sFHxVkCuLePXaNvUdDchLt7i3u3QNQ/edit?usp=sharing

"Blueprints"

I borrowed the one-way mirror from my University so I had to design the hardware to be easy to take apart. I made two parts: the frame, which holds the glass and the sesnors¡, and the back support, which holds the TV, the microphone, the Raspberry Pi and the power adapters. The back is hung on to a wall and then the frame is hung on to the back.
Building the frame and back

For the back support I used a wooden square and I cut a hole in the middle to put the TV support. Luckily , for the frame, we found an old wooden frame which had a nice size for our glass. We only had to paint it and drill four holes for the sensors.
TV

I used an LG 24" TV
Back support

I ended up cutting more parts than needed.
Testing the distance sensors

HC-SR04 sensors. Not the best but they're cheap and work.
Soldering the sensors

I soldered the sensors to a board to be able to easlily integrate them with the frame.
Drilling the holes for the sensors

Fitting the sensors

Frame

This is the final painted frame mounted on the back support. The glass is still missing here.
FInished back support

This is the finished back support which hangs on to a wall.
Finished frame

This is the back of the frame completely finished with the sensors and glass. The glass is held by the top and bottom wooden parts which are screwed to the frame. It was done this way to be able to remove the glass easily.
Raspberry Pi

The software runs on a Raspberry Pi. It can run on the RPi2 and the 3.
Developing

I developed all the software on the Raspberry Pi itself. Using geany. After all it's only Javascript, HTML, CSS and some Python.
News app

Smartphone remote app

Shows all the apps that are installed on the mirror and lets you launch them. Also lets you use the phones microphone for voice control.
Early tests

Youtube app

Source : itsjoui from Reddit