Christopher Raphael, a professor computer science at the School of Informatics, Computing, and Engineering, has built a career around the blending of computers and music.
Now, he’s going global.
Raphael recently demonstrated the power of the Informatics Philharmonic, a system in which a computer-driven orchestra follows and learns to play along with a soloist in a concerto-like setting, to several top conservatories in China, including The Central Conservatory in Beijing, the Shanghai Conservatory, and the Xinghai Conservatory in Gauangzhou.
Raphael traveled to China with a former Ph.D. student, Yushen Han, who was instrumental in making the event a success. The trip has already yielded dividends. SICE and Raphael have entered into a collaboration with the Xinghai Conservatory to install the Info Phil system on the conservatory’s faculty computers, and Xinghai has committed to using it widely and intensively in teaching, concerts, and master classes. The partnership is expected to yield quality, real-world feedback that can be used to improve the system and provide suggestions for future development.
“China has announced a nation-wide commitment to Artificial Intelligence, which is the first such commitment since it made a similar one to steel in the late 1950's or so,” Raphael said. “This is significant since it allows the Conservatory and SICE to be a part of what is cutting edge in China today. This is an especially unusual experience for a music conservatory, and Xinghai was very excited to have a chance to play this role.”
The Info Phil system computes a real-time score match using a hidden Markov model, generates the output audio by phase-vocoding a preexisting audio recording, and providing a link between the two by predicting future notes based on what has been played prior. The result is a soloist who is accompanied by a virtual orchestra that learns on the fly and adjusts to the style and tempo of the human player.
Raphael worked with a number of faculty at the various conservatories using different instruments, including the viola, violin, flute, cello, horn, bassoon, and trumpet. Xinghai’s high level of musical sophistication will make it a demanding user of the system, which will allow it to create feedback that will address the core technology and the subtleties of developing an interface that is appropriate and natural for musicians.
“This sort of collaboration shows the way informatics can work to solve real-world problems,” said Raj Acharya, Dean of SICE. “Using machine learning as a tool for music is a great example of the ways our faculty use unique pathways to find solutions, and this partnership will help strengthen the bonds between SICE and the vast cultural and intellectual resources in China.”
The Info Phil system was developed nearly a decade ago thanks to a $450,000 grant from the National Science Foundation, and Raphael has been improving the system ever since. It has allowed soloists to work on their craft with the backing of a full virtual orchestra and has allowed musicians to develop their skills in the comfort of their own home.
“This is all part of an ongoing effort of mine to get the system out into the musical world to be used, known, and appreciated by serious practicing musicians,” Raphael said. “We have ongoing discussions with the Central Conservatory and the Shanghai Conservatory, and there’s a fourth conservatory that has expressed interest. We hope this will be part of a groundswell of interest and attention for the system.”
For more information on music informatics at SICE, visit our website.