Dynamic Adaptation of Graphical User Interfaces in Augmented Reality Based on Environmental Factors: Enhancing User Experience and Safety

Abstract

Augmented Reality (AR) has been experiencing great steps in further blending real life and virtual environments, by bringing additional information and virtual controls to real world scenarios. As such, adequate and responsive positioning and integration of virtual elements are fundamental in AR to provide users a seamless experience in blending virtual elements to reality. This project aims to develop an Augmented Reality solution that makes graphical user interfaces (GUI), such as interactive panels, respond dynamically to their environment. The solution automatically adjusts the positioning and appearance of graphical interfaces based on conditions from the environment, such as changes in lighting, presence of important objects (such as warning signs), to the presence of people, and to the presence of possible safety risks to the user. To achieve this, multiple techniques in the area of computer vision have been used for identifying and classifying objects detected, while a Generative Artificial Intelligence (AI) model is used to interpret more nuanced contextual data, such as a user text input. This allows interfaces to adjust themselves to not block the vision of points of interest from the user, adjust colors and contrast for legibility and visual comfort. The development of the solution has been guided by user testing to ensure effectiveness and an intuitive experience. As of this report, a prototype has been developed that can adjust the positioning of the GUI to avoid occluding specific classes based on a text input. Additionally, it modifies the color of the GUI elements to complement the dominant colors behind GUIs in the camera image.