Researchers at Northwestern University in Illinois have developed a wearable device that continuously tracks how much people use their voices, alerting them to overuse before vocal fatigue and potential injury set in.

The battery-powered, wireless device and accompanying algorithms could be a game-changer for professional singers, teachers, politicians, call-centre workers, coaches and anyone who relies on their voices to communicate effectively and make a living. It also could help clinicians remotely and continuously monitor patients with voice disorders throughout their treatment.

Developed by an interdisciplinary team of materials scientists, biomedical engineers, opera singers and a speech-language pathologist, the research behind the technology was published last month in the Proceedings of the National Academy of Sciences.

The soft, flexible, postage-stamp-sized device comfortably adheres to the upper chest to sense the subtle vibrations associated with talking and singing. From there, the captured data are instantaneously streamed via Bluetooth to the users’ smartphone or tablet so they can monitor their vocal activities in real time throughout the day and measure cumulative total vocal usage. Custom machine-learning algorithms distinguish the difference between speaking and singing, enabling singers to track each activity separately.

With the app, users can set their personalised vocal thresholds. When they near that threshold, their smartphone, smartwatch or an accompanying device on the wrist provides real-time haptic feedback as an alert. Then, they can rest their voices before pushing it too far.

“The device precisely measures the amplitude and frequency for speaking and singing,” said Northwestern’s John Rogers, a bioelectronics pioneer who led the device’s development. “Those two parameters are most important in determining the overall load that’s occurring on the vocal folds. Being aware of those parameters, both at a given instant and cumulatively over time, is essential for managing healthy patterns of vocalisation.”

Northwestern’s Theresa Brancaccio, a voice expert who co-led the study, added: “It’s easy for people to forget how much they use their voice. Seasoned classical singers tend to be more aware of their vocal usage because they have lived and learned. But some people – especially singers with less training or people, like teachers, politicians and sports coaches, who must speak a lot for their jobs – often don’t realise how much they are pushing it. We want to give them greater awareness to help prevent injury.”

For the millions of people who make their livings by speaking or singing, vocal fatigue is a constant, looming threat. The common condition occurs when overused vocal folds swell, making the voice sound raspy and lose endurance. Vocal fatigue negatively affects singers, in particular, altering their abilities to sing clearly or hit the same notes as their healthy voice can. At best, one short period of vocal fatigue can briefly interrupt a singer’s plans. At worst, it can lead to enough damage to derail a career.

Lack of awareness is the underlying problem. People rarely make the connection between vocal activities and how those activities affect their voices. Most people don’t notice they are overusing their voices until hoarseness has already set in.

“What leads people into trouble is when events stack up,” Brancaccio said. “They might have rehearsals, teach lessons, talk during class discussions and then go to a loud party, where they have to yell over the background noise. Then, throw a cold or illness into the mix. People have no idea how much they are coughing or clearing their throats. When these events stack up for days, that can put major stress on the voice.”

Separately, Rogers, in collaboration with researchers at the Shirley Ryan AbilityLab, had developed a wireless wearable device to track swallowing and speech in stroke patients. The bandage-like sensor measures swallowing abilities and speech patterns to monitor stroke patients’ recovery processes. In the early weeks of the Covid-19 pandemic, Rogers’ team modified the technology to monitor coughing, as a key symptom of the illness.

“I wanted to gather more data and make our tracking system more precise and more accurate,” Brancaccio said. “So, I reached out to John to see if his sensors could help us gather more information.”

Rogers said: “I thought it was a great opportunity for us to extend our technologies beyond our very important, but narrowly targeted, uses in health care to something that might capture a broader population of users. Anyone who uses their voice extensively could benefit.”

The pair also partnered with speech pathologist and voice expert Aaron Johnson to explore how the devices could be used to evaluate and monitor treatment for patients with vocal disorders. Johnson, who co-directs the NYU Langone’s Voice Center, said the small, wireless device could help track patients’ voices in the real world, outside of a clinical setting.

“A key part of voice therapy is helping people change how and how much they use their voice,” said Johnson, study co-author and associate professor in the department of otolaryngology at NYU Grossman School of Medicine. “This device will enable patients and their clinicians to understand voice use patterns and make adjustments in vocal demand to reduce vocal fatigue and speed recovery from voice disorders. Generalising vocal techniques and exercises from therapy sessions into daily life is one of the most challenging aspects of voice therapy, and this device could greatly enhance that process.”

The team modified Rogers’ existing devices to measure vocal load precisely over time. That includes frequency, volume, amplitude, duration and time of day. Like Rogers’ previous devices for Covid-19 and stroke patients, the new device also senses vibrations rather than recording audio. This lets the device detect vocal activity precisely from the user, rather than the ambient noise surrounding them.

The biggest challenge was to develop algorithms capable of distinguishing speaking from singing. To overcome this, Brancaccio recruited voice and opera students to undertake singing exercises to train the machine-learning algorithms. A team of classical singers with vocal ranges from bass to soprano wore the devices while humming, singing staccato scales and songs, reading and more. Each singer generated 2500 one-second-long windows of singing and 2500 one-second-long windows of speaking.

The resulting algorithm can separate singing from speaking with more than 95% accuracy. And, when used in a choir setting, the device captures only data from the wearer and not noise from nearby singers.

“Prolonged talking is one of the most fatiguing activities for people who are training to become professional singers,” Brancaccio said. “By separating singing and speaking, it can help people develop more awareness around how much they are speaking. There is evidence that even brief 15- to 20-minute periods of total silence interspersed throughout the day can help vocal fold tissues recover and repair.”

To use the device, the wearer simply adheres it to the sternum, below the neck, and syncs the device with the accompanying app. Rogers’ team is working on a method to personalise vocal budgets for each user. Here, users will press a button in the app if they experience vocal discomfort at any point during the day, effectively capturing the instantaneous and cumulative vocal load at the time. These data can serve as a personalised threshold for vocal fatigue. When the user nears or exceeds their personalised threshold, a haptic device will vibrate as an alert.

Similar in size and form to a wristwatch, this haptic device includes multiple motors that can activate in different patterns and with varying levels of intensity to convey different messages. Users can also monitor a graphical display within the app, which splits information into speaking and singing categories.

“It uses Bluetooth, so it can talk to any device that has a haptic motor embedded,” Rogers said. “So, you don’t have to use our wristband. You could just leverage a standard smart watch for haptic feedback.”

Although other vocal-monitoring devices exist, those use big collars, tethering wires and bulky equipment. Some also use embedded microphones to capture audible vocal data, leading to privacy concerns.

“Those don’t work for continuous monitoring in a real environment,” Brancaccio said. “Instead of wearing cumbersome, wired equipment, I can stick on this soft, wearable device. Once it’s on, I don’t even notice it. It’s super light and easy.” 

Because Rogers’ previous devices capture body temperature, heart rates and respiratory activity, the researchers included those capabilities in the vocal-monitoring device. They believe these extra data will help explore fundamental research questions concerning vocal fatigue.

“This is more speculative, but it might be interesting to see how physical activity affects vocal fatigue,” Rogers said. “If someone is dancing while singing, is that more stressful on the vocal folds compared with someone who is not physically exerting themselves? Those are the kinds of questions we can ask and quantitatively answer.” 

In the meantime, Brancaccio is excited for her students to have a tool that can help prevent injury. She hopes others including non-singers will see the benefit to keeping their vocal cords healthy.

“Your voice is part of your identity, whether you are a singer or not,” she said. “It’s integral to daily life, and it’s worth protecting.”

The study was supported by the Querrey Simpson Institute for Bioelectronics at Northwestern University.