Jarvis

An interactive, gesture-controlled 3D hologram. Built with OpenCV, Mediapipe, Websockets, Flask, Next.js, and various Poimandres React packages for Three.js.

Click to view our Video Demonstration

Running Locally

It works just fine on any regular screen (without the hologram effect, of course). But feel free to build a Pepper's Ghost hologram if you'd like.

Cloning the repository the local machine.

git clone https://github.com/ishaan1013/jarvis

Backend setup

• Set up a webcam facing upwards, in front of your screen and a little bit below.
• Ensure OpenCV recognizes your webcam in backend/camera.py
• 🐍 Run the flask app, which will be available at http://localhost:8000.

flask run

Running the frontend app

npm install

Then, run the application in the command line and it will be available at http://localhost:3000.

npm run dev

Notes

• Viewing and manipulating 3D models can be resource-intensive.
• Voice control uses the WebSpeechAPI which currently works best on Chrome, Microsoft Edge, or Safari 14.1