Harnessing the Power of AI to Enhance Communication for People with Hearing Loss
Xiao Wei (third from the right) and his team at Tencent's Ethereal Audio Lab.
Xiao Wei is an audio engineer who is passionate about developing algorithms to enhance real-time, end-to-end digital communications for video and audio conferencing tools.
Wei and his colleagues at Tencent’s Ethereal Audio Lab (TEALab) believe that the open-source technology behind the popular video conferencing platform Tencent Meeting can make life easier for people around the world to run their businesses and stay in touch with loved ones, especially when meeting in person is not as flexible as before.
Investigating further, they realized that improvements in audio quality and noise reduction can fulfil Tencent’s mission of tech for good in other ways by helping people who are hard of hearing.
You Probably Know Someone with Hearing Loss
According to the World Health Organization (WHO), nearly 1.5 billion people live with hearing loss worldwide, with the number expected to rise to 2.5 billion people by 2050. Of them, at least 700 million are expected to need hearing aids.
The consequences of hearing loss can be significant. It can affect one's learning and development skills, employment, interpersonal relationships, and participation in society. They can lead to loneliness, isolation, and mental health problems.
Thankfully, hearing loss can be mitigated by using assistive hearing technology, such as hearing aids or cochlear implants.
Destigmatizing Hearing Loss and Encouraging Early Detection
If you are short- or long-sighted, it will usually be picked up early and glasses are routinely prescribed to correct your vision. Yet there is embarrassment about wearing hearing aids because it is seen as a disability.
Shifting the awareness takes time and requires education. TEALab partnered with China’s National Ear, Nose and Throat Research Center to launch a simple online audiometry tool. The primary screening tool incorporates elements of the Chinese zodiac, and a speech intelligibility test based on noise scenarios, making it easy for users to assess their hearing health.
Applying Audio AI to Help People with Hearing Loss
When Wei and his colleagues at TEALab first started adapting communication technology to help people with hearing loss, they worked to eliminate or suppress background noise. Yet the feedback from test users indicated that the intelligibility is even worse when we only reduce the background noise arbitrarily.
One day, Wei was speaking to an expert who pointed out that speech intelligibility is the most important thing from an audiology perspective. It reminded him that what people with hearing loss want to hear is speech and that they must enhance speech and preserve important sound components, rather than fighting against noise.
Drawing from the AI enhancement technology and deep learning models used for Tencent Meeting, engineers developed a new method to protect speech components before undertaking noise reduction, leading to intelligibility improvement under different noise conditions.
A researcher at Tencent’s Ethereal Audio Lab collects outdoor noises.
Harnessing Audio AI Through Smartphones
Smartphones have become a vital component of audio AI technology. They are essential to capturing and processing sound before transmitting it via Bluetooth to a hearing device such as cochlear implants or hearing aids.
For Wei, the challenge has been to design algorithms to enhance or reduce noise to suit the environment and user’s hearing profiles. He likens it to how a person who wears glasses needs the right prescription lens. Now with the app, people with hearing loss can personalize the speech enhancement noise reduction without having to visit a doctor.
TEALab has also found a cost-effective solution where people with mild hearing loss can use headphones rather than specialized cochlear implants or hearing aids. For example, when users are in a classroom or lecture hall, they can place their smartphone on a desk near the speaker. Using a free app, the phone picks up and processes the speech, then transmits it to the user’s headphones via Bluetooth. Additionally, the transcription is displayed on the screen, to assist the people to understand.
Prioritizing the Elderly
The elderly are most severely impacted by hearing loss. Up to a third of people over 65 years old globally have at least moderate hearing loss, which gets worse as they get older. Studies indicate that hearing loss is associated with dementia and depression.
To gauge the effectiveness of their technology, Wei’s team worked with potential users to observe real-life scenarios in busy or noisy places such as markets. For example, when a user opens an app using TEALab’s listening technology to converse with a store clerk, the app processes the staff’s speech and converts it into texts on their smartphone.
Also, if an elderly user wants to call his family or friends, a Weixin Mini Program can be activated to listen to the call and display the subtitles on the mobile phone screen in real time. In this scenario, both subtitles assisted listening and voice-assisted listening features come into play.
Finding Solutions to Problems Through Cross-sector Collaboration
Cross-sector collaborations are an integral part of what TEALab does. Their engineers frequently liaise with experts from different industries, who come with challenges that they haven’t been able to solve. Equipment manufacturers also call TEALab often for technical support. With such a depth of audio AI algorithms and deep learning capabilities, the team works to adapt the technology to benefit users and empower collaborators.
“Applying one algorithm to another field effectively isn’t an easy task,” says Wei. “It depends on the solid foundation of technology and the technicians’ insight into new application fields. They also have to establish connection points between the technology and application. Most importantly, you need to identify and understand the problems from the user’s perspective.”
