Audio to sheet music

A piano score on two staves: the right hand on the treble staff and the left hand on the bass staff, each written as one line of notes and chords. It has a key signature, a time signature and a tempo mark, notes from whole notes down to 16ths and eighth-note triplets, ties across bar lines, and a pickup bar when the music starts before the first beat.

It's written to be read, not to copy every millisecond of the performance: notes that the pedal holds over the next ones end where the next note in that hand starts, a key let go a moment early still gets its full value, and nothing shorter than a 16th becomes a rest.

• A piano check listens first. The transcriber writes any pitched sound as piano notes, so a recording without a piano is turned away instead of becoming a page of nonsense.
• A transcription model trained on hundreds of hours of piano (Kong et al., 2021) hears when each of the 88 keys is pressed and let go, to about a hundredth of a second, and how hard.
• PM2S (Liu et al., 2022), a model that reads a performance the way a musician does, finds the beats, the downbeats, which hand plays each note and the key signature.
• Rules from engraving practice turn that into notation: each beat is snapped to 16ths, or to triplets only where the beat really is in threes, chords played a little apart become one chord, and every chord is made playable by one hand.

• Solo piano works best, acoustic or digital. A voice over the piano is fine, but it may add a few stray notes.
• Record close to the piano in a quiet room: a phone on the music stand is enough. Heavy room echo blurs the notes.
• A steady tempo helps the bar lines land in the right place. Strong rubato, fermatas and big tempo changes make the bars drift.
• Trim talking and long silence from the start and end of the recording.

The notes are usually right; the rhythm is the hard part. A piece can come out with some bar lines a beat off, or in a neighbouring meter (2/4 for 4/4, 4/4 for a slow 3/4), and fast ornaments and grace notes come out as plain short notes. Play the first few bars against the score before you trust it.

To correct it, download the MusicXML and open it in MuseScore, which is free, or in Sibelius, Finale or Dorico: every note is editable there. The MIDI file opens in any DAW.

We tested it on seven freely licensed recordings by real pianists, each against its printed score, through a phone-quality copy of each recording (the originals score within two points of it):