Yale-NUS students develop AudioReality project to empower blind community

Trishan Jerome Montoya (Class of 2024), Ryan Lee (Class of 2025), Joseph Goh, and Andrew Siow (both from Class of 2024) (left to right). Image provided by Joseph Goh.

A team of Yale-NUS students recently won first place in the Hack for Good 2023 Hackathon. Organised by the Developer Student Clubs at the National University of Singapore (NUS) Computing, the competition inspires participants to use programming skills and technology for social causes. After a week-long competition, Ryan Lee (Class of 2025), Joseph Goh, Andrew Siow, and Trishan Jerome Montoya (all from Class of 2024) presented their final project on 18 February 2023 to a panel of judges from various non-profit organisations.

Together, they developed a mobile app called AudioReality which determines the geolocation of an individual and describes their surroundings based on Google Streetview. Targeted towards the blind community, the app generates a pleasant literary description of the surroundings, enabling blind people to enjoy “sightseeing” in their own unique way and differentiating the app from other practical navigational tools.

The team was inspired to improve the well-being of the blind community in a way that would allow them to have fun. “My experience at Yale-NUS has taught me the significance of creating enjoyable experiences even if they may not have immediate practical benefits, but are still crucial for enhancing our overall well-being,” Ryan said. “Currently, the software space is saturated with apps that help the blind community in practical ways, and we wanted to complement them with our app,” Joseph added.

The judges were also impressed with the AudioReality project’s angle on leisure and how the team emphasised the word “enjoy” throughout. “I believe this speaks to the unique benefit of pursuing a tech major in a liberal arts and sciences college. We are positioned in a place that allows us to blend the arts and sciences together,” Joseph explained.

Though the team are all Mathematical, Computational, and Statistical Sciences (MCS) majors, their unique backgrounds contributed to the project. For example, Joseph and Ryan were Vice-Presidents of Yale-NUS Tech, a student organisation for students interested in the technology industry, giving them prior knowledge of how technology can be incorporated into all kinds of fields. Andrew, as an MCS major on the data science track, was familiar with advanced image recognition software and conversation models, while Trishan had a keen interest in psychology and literature to help drive the project.

Having to come up with the product within a week was tough and Joseph shared, “Ryan and Trishan had to learn app development and pick up an entire new programming language within a week.” Despite the challenges, they were collectively driven by a desire to bring their solution to reality.

“I learnt quite a bit about useful APIs and tools that can help us do a lot without having to spend too much effort training our own models,” Andrew said. The project and the competition gave each member invaluable experience for their future academic and professional lives.

Sharing about future plans for AudioReality, Joseph shared, “Though we currently do not have the capacity to continue developing the project, one prominent feature we envision the app having is a database of Singaporean poems and short stories. When users walk around Singapore, AudioReality would also read relevant poems to enhance the literary experience.” Moreover, he believes that the app and software can eventually be scaled up and use real-time images from devices such as smart glasses.

After the competition, the team also had a discussion with one of the judges who works for SG Enable, a non-profit organisation with a goal to empower persons with disability in Singapore. “He suggested we post our project on www.ideable.sg and write some documentation for it so that available developers can take it from there and continue working on it if they want,” Joseph explained.