AI Tackles Piano Playing and Unfolds Sealed 17th Century Letters
Researchers use computer vision to recreate piano music from silent video and deploy advanced algorithms to virtually unfold fragile historical letters. These creative applications show how artificial intelligence extends into arts and humanities.
Researchers at the University of Washington develop a computer vision system called Audeo that watches silent overhead video of piano playing and translates the hand movements into recognizable music. The system first identifies the sequence of key presses from the video frames, then adds musical expression by estimating the length and strength of the presses, and finally outputs the result through a MIDI synthesizer. The researchers express surprise that the algorithm successfully imagines the sound occurring between video frames to create a surprisingly musical performance.
In another fascinating intersection of AI and the humanities, an MIT team develops a computational method to read centuries-old letters without physically opening them. These 17th-century letters are locked through intricate folding and sealing techniques that make physical unfolding risky and potentially destructive. Instead of touching the delicate documents, the researchers X-ray the sealed letters and apply an advanced algorithm to separate the densely packed layers of paper in the resulting imagery.
The virtual unfolding algorithm performs impressively despite the extreme thinness of the paper and the tiny gaps between layers that sometimes fall below the scan's resolution. While the researchers admit this specific technique stretches the strict definition of machine learning, the breakthrough provides a safe way to access previously hidden historical texts. This innovative approach shows promise for deciphering many other types of fragile, folded documents that simple X-ray techniques cannot easily unravel.