kobimusic

introducing dettato

A low-poly pink bird, drawn ten times in a loop, listening

we're introducing Dettato, a SOTA audio to MIDI transcription model that significantly beats MuScriptor 1.3B - the next best open model - and scores a bit higher than mirelo's closed source model 'Mirelo A2M v1.1 Pro' on their published benchmarks. all at just 20m parameters - roughly 65x smaller than muscriptor. Additionally, in training the model, we trained on absolutely no copyrighted recordings. the model is open source and you can use its outputs for commercial purposes.

the composite, and the measures it is the mean of, for each system
systemcompositeonsetsframesoffsetsdrumsparameters
dettato60.870.879.140.952.520M
Mirelo A2M v1.1 Pro56.167.175.740.141.4not published
MuScriptor47.457.071.231.430.11.3B
YourMT3+45.758.263.930.530.046M
sets used: PHENICX, RWC classical, RWC royalty-free
F1 per set: the mean of onset, frame and offset
F1 per set: the mean of onset, frame and offset
setdettatomirelo A2M v1.1 promuscriptor 1.3Byourmt3+
Bach1091.3not reported54.757.9
Dagstuhl ChoirSet60.8not reported35.528.0
PHENICX-Anechoic64.860.954.444.8
RWC Popular54.0not reported44.336.1
RWC Classical64.465.858.462.4
RWC Genre48.7not reported42.635.6
RWC Jazz55.7not reported52.047.1
RWC Royalty-Free61.656.246.845.4
Slakh2100 (synthetic)68.4not reported51.1not reported

disclaimer with the benchmarks: it's difficult to gauge the true accuracy of the model for your use-case without trying it yourself. one thing my model performs poor on for instance, is splitting instruments correctly. In addition, this just spits out performance MIDI with a beat aligned. If you're looking for something that you can actually play(e.g sheet music), i'd reccommend songscription or klangio - both are excellent tools by great companies that push the envelope of what's possible in this field forward.

loading
0:00 / 0:00

sousa, the stars and stripes forever: “the president's own” united states marine band, from the complete marches of john philip sousa, vol. 3. a us government recording, public domain. download the MIDI

upload your own example(or pick a demo below)

the driving goal of kobimusic: to help musicians, not replace them. a part of that goal means following copyright law. we trained dettato on a mix of synthetic instrument recordings and CC solo instruments mixed in using the URMP dataset. the rest of the instruments morph together ~20 soundfonts, using the 'soundfont babies' technique i published. no copyrighted recordings were used.

I began working on dettato about 10 months ago, but scrapped what I had in about April. The reason why was that I treated AMT as a data problem - but in reality, the bottleneck is architecture. There are a lot of parellels that you can draw for OMR (first model i made) and AMT. both traditionally were tackled early on using algorithms. later, in both fields they plugged in giant LLM components. These models would naturally become overfit to synthetic data, since globally the problems of AMT and OMR become different than if you treat it locally. But, for both you can reframe it as a local problem. For OMR, using a detector model. For AMT, a pitch-lane model.

overall, i hope both open and closed weight music retrieval companies can learn from this. The constraint of not using copyrighted recordings doesn't hold you back, but actually encourages innovation, and it's likely you'll end up with a better model in the end.