Runtime latency control for streaming guitar tablature transcription through constant-Q buffer truncation

What the system does

The system listens to a guitar and writes tablature, the string and fret of each note, while the guitarist plays. Most of its delay comes from the analysis buffer of its constant-Q front end, which must hold the long filter of the lowest note. We shorten this buffer at start-up by truncating the filters, so one trained model runs with a 216 ms buffer or a 16 ms buffer without retraining. A shorter buffer answers sooner but loses some accuracy on the lowest notes. The paper also studies two other ways to lower the delay, training the model with a shorter analysis window and raising the lowest analysed frequency, and reports an estimated pluck-to-display latency budget on a laptop.

Supplementary videos

Both videos show the same performances recorded through the live system in four configurations, with the four tablature displays stacked on one time axis. The performances are two clips from the GuitarSet test split (jazz comping and a funk solo, each at normal speed and then at quarter speed) and a performance of Romanza on a Yamaha SILENT Guitar that the model had never heard. The system runs on a laptop’s integrated GPU (Intel Core Ultra 7 265H with Arc 140T, Windows 11, plugged in, power mode “Best performance”).

How to read a band. Time runs from right to left, and the green line at the right edge is now. An outlined ring is a reference note, drawn when it sounds. A filled marker is the model’s prediction, drawn when the model outputs it, so the line between a ring and its marker is that note’s delay. Green means the right string and fret, amber the right pitch on another string, and red a wrong or missed note. The Romanza performance has no reference notes, so it shows predictions only, and there the marker colours indicate strings, not correctness.

Video 1: training window, buffer and chunk size

The top band is the model of the paper’s main experiments. Each band below it changes one setting.

The training window is fixed when the model is trained. The buffer and the audio chunk are chosen at start-up. With the shorter training window, the model’s note-level tablature F-measure on the test split was lower by at most 0.018.

Video 2: four models at the 16 ms buffer

All four bands use the 16 ms buffer and 25 ms audio chunks. Raising the lowest analysed frequency shortens the training window and the delay, but the model then no longer sees the lowest notes directly.

At this buffer, the four models’ note-level tablature F-measures on the test split differed by at most 0.02 in each test condition.

Notes

• Each estimated delay adds the audio delay calculated from the buffer, the frame alignment and the audio chunk, the median inference time measured on the laptop GPU with the viewer open and disconnected, and a display time estimated in an earlier session. It leaves out the host’s audio path, scheduling delays and queueing, so the delays seen in the videos can be somewhat longer.
• The two GuitarSet clips were chosen for a clear picture. The paper reports accuracy on all 120 test clips at every buffer.
• Both videos run 2:06. Chapters: 0:16 jazz comping, 0:40 at quarter speed, 0:52 funk solo, 1:14 at quarter speed, 1:26 Romanza, 1:55 summary.

Reference

A. A. Nugraha, N. Iino, K. Yoshii, and M. Hamanaka, “Runtime latency control for streaming guitar tablature transcription through constant-Q buffer truncation,” APSIPA Transactions on Signal and Information Processing, under review.