Generative AI, Shaders and Eyetracking input
This project explored how eye-tracking technology could enhance player focus and immersion in a gaming environment. Using the Unity engine and Tobii Eye-tracker, the goal was to investigate whether dynamically adjusting sound and lighting based on where the player was looking could improve attention during a multi-conversation scene. Set in a murder mystery scenario, the player needed to gather clues by listening to various conversations happening simultaneously. Generative AI was used to craft the dialogues and voices for the characters. The experiment showed that participants in the adaptive group (where sound and lighting followed their gaze) were able to focus more on individual conversations, compared to the control group, which frequently shifted attention between multiple speakers.
Keyfeatures:
- Adaptive Immersion: The system used real-time eye-tracking data to dynamically adjust the sound and lighting around the conversation the player was focusing on, enhancing their immersion and attention.
- Generative AI-driven Storytelling: AI was leveraged to generate unique dialogue scripts and text-to-speech voices for the characters, allowing for dynamic and diverse conversations.
- Cinematic and Shader Effects: The experience opened with a custom cinematic intro, featuring dramatic special effects and sound design to set the tone for the mystery, fully immersing players from the start.
My Role & Skills Used
In this project, I took on multiple creative and technical roles. I was responsible for writing and shaping the story using generative AI, which allowed for dynamic and unique interactions with the NPCs. I also created and implemented shaders that enhanced the visual atmosphere of the game, ensuring the lighting and environmental effects complemented the story and gameplay. Additionally, I directed and developed the intro cinematic, incorporating special effects and sound design to captivate the player and set up the mood for the game. My sound design work extended throughout the gameplay, adding layers of auditory feedback that heightened the player’s sense of immersion.
