Gaming with disabilities: Face-tracking game controllers


Games can be challenging for people with disabilities. Even playing simple games can be difficult for them because most game controllers are not designed with their needs in mind.

In this project, I will try to develop a game controller that can be used with face/head movements, it is hands-free design and requires minimal input to play simple online games/browser games.

Demonstration video

facial movement control
Face tracking using emojis on OLED displays!

Operating principle

In a nutshell, the device works as follows:

  • Find faces using body sensors
  • Sensing the face position in the field of view through I2C
  • Emulate keyboard as USB HID using microcontroller
  • Send key input to the console to control the game

The human sensor is a camera module with a pre-programmed microcontroller with a computer vision facial recognition model.

Hardware construction

The prototype was constructed based on the following schematic

Schematic diagram

prototype hardware

This is the actual hardware prototype

Hardware description (material list)

Human Sensors Useful sensors: As the name suggests, human body sensors are sensors that use computer vision to detect human faces. It is pre-programmed with a face detection algorithm that can track the movement of faces within the field of view. The human body sensor does not allow access to the original image data, but returns information about whether a face is detected, the number of faces, the position of the face, etc. through the I2C interface.

That’s why it’s easy to use microcontrollers without knowing the ins and outs of computer vision or facial recognition artificial intelligence.

White LED light with diffuser: Human body sensors require sufficient lighting on the user’s face to function properly. That’s why I added two white LEDs with diffusers. They are driven by PWM pins to adjust brightness.

Make sure there is enough light on the face of the person or the camera will lose face tracking

Beetle board (Atmega32U4 mcu):This is a very small form factor Arduino Leonardo equivalent board with an Atmega32U4 microcontroller. It is used to drive OLED display 1306, read human body sensor data through I2C interface, drive servo motor angle, drive LED light brightness, use ADC to sense phototransistor beam interruption, drive vibration motor, and simulate HID USB keyboard on the host PC And send key commands according to the face position in front of the human body sensor. Basically this MCU runs the entire device.

OLED display: Small, low-cost, I2C-based OLED display for user interface with 1306 display controller. This display is used to show the location of faces in the field of view and also notifies the user if a face is not detected.

I find this display very useful for debugging code when using the human sensor.

Point laser diode:These are tiny LED-based semiconductor laser diodes that produce a monochromatic beam of red light. Here a laser beam is used to illuminate a phototransistor (an LDR or diode would also work). Plans to develop two “zero-pressure” air buttons. The air button simply interrupts the laser beam with a finger or any object to sense user input.

Optoelectronic transistor: Two phototransistors are used in this device to create an Air Tap button (basically interrupting a light beam to press a switch). These phototransistors are connected to the two ADC inputs of the microcontroller. When the laser beam is interrupted, the transistor is almost turned off, causing the ADC to sense the Gnd potential. This change is registered as the user “pressed the button”

Vibration motor: I added two vibration motors for tactile feedback because the optical interrupt button (where the laser beam and phototransistor is interrupted by a finger or object) doesn’t have any sense of force and the tactile sensation would give the feeling that something is happening.

Servo motor: The purpose of this servo motor is to initially align the camera with the center of the face. Although the hardware is connected, I haven’t implemented this functionality in the firmware yet.

Programming and Firmware

Download Arduino IDE

Download the latest Arduino IDE from here. It is recommended to use IDE version 2.1 and latest for better compatibility, but any version 1.8+ or higher will work.

Arduino integrated development environment

Select “Arduino Leonard”

Tools > Board > Arduino AVR Board > Leonardo

Add the library to the Arduino IDE

This project requires the following libraries, please make sure you have them included in the Arduino IDE. Details are available at the GitHub link

U8G2 library for display drivers

Useful sensor library for human body sensors

Next, Compile and upload the code* (attached below) to the Beetle board and be ready to test the concept.


Here are some test videos and yes, you guessed it! Since the polling rate of the camera on face tracking is only 7 Hz, there is a bit of latency!

Nonetheless, the idea is what I’m trying to demonstrate here, and implementing it on a single board computer with a faster camera would solve the latency issue!

refer to

in conclusion

This project is far from complete. I was rushing this at the last minute. This firmware is still under development. Don’t think of it as a finished project, think of it as a prototype concept.

I would like to make a more complex game controller that can accept other forms of input from the user, such as making certain sounds with the mouth (and detecting them via voice recognition) as game control input. Developing game controllers for people with disabilities is not easy because everyone’s needs are different.

Let’s keep exploring new ideas and try them, wonderful things may happen and we’ll have the ultimate gaming controller that everyone can use

Source link


Your email address will not be published. Required fields are marked *