The test behind the accuracy chart.
We wanted a test HomeNote had no home-field advantage in: songs it had never heard, labelled by people who had nothing to do with HomeNote, with every tool run the same way on the same computer. Here is exactly what we did, and every answer, so you can check it.
The songs
684 full songs from the Free Music Archive, keyed by music experts in the FMAK dataset (Stella Wong, ISMIR 2023; zenodo.org/records/10719860). None of them were used to train, tune or test HomeNote before this test.
- 342 hip-hop songs: every FMAK song tagged hip-hop, rap, alternative or abstract hip-hop, or hip-hop beats.
- 342 songs from everything else, spread evenly across rock, electronic, folk, pop, instrumental, classical, experimental, international, jazz, country, R&B, blues and older recordings (smaller genres gave all they had). Picked at random with a fixed seed.
- The answer key is the experts' label. A tool is right only if it names that exact key.
The setup
- One computer for everything: a 2017 iMac Pro running Windows (Boot Camp), with the same audio files.
- HomeNote: the Windows release build, given each whole song.
- Auto-Key 2 (Antares; its detection engine is zplane's TONART v3): its File Mode in FL Studio, which analyzes the whole file. The key was read off its display, with a check that the display showed the right file name before each answer was recorded.
- libkeyfinder: the open-source engine inside Mixxx and KeyFinder, built from source with its default settings, given each whole song.
Results
| HomeNote | Auto-Key 2 | libkeyfinder | |
|---|---|---|---|
| Exact key, all 684 songs | 67.7% | 58.5% | 53.9% |
| Exact key, hip-hop half | 62.9% | 53.2% | 53.2% |
| Exact key, other half | 72.5% | 63.7% | 54.7% |
| Main key or its relative (same notes, same Auto-Tune correction) | 73.1% | 64.5% | 60.7% |
| Right key shown on screen | 83.8% in its top 3 | 64.5% main or relative switch | 53.9% one answer |
| Standard partial-credit score (MIREX) | 72.0 | 65.8 | 61.9 |
How sure we are: resampling the songs 3,000 times, HomeNote's lead over Auto-Key 2 on exact key stays between +6.1 and +12.4 points (95% range). Song by song, HomeNote was right on 89 songs Auto-Key 2 missed, and Auto-Key 2 was right on 26 HomeNote missed.
What this doesn't show
- One label per song. Some songs are honestly ambiguous, and an expert's call decides "right" for every tool alike.
- Auto-Key 2 was tested in File Mode only, its whole-file analysis. Its Listen Mode wasn't tested.
- Accuracy isn't everything: Auto-Key 2 sends its settings straight into Auto-Tune, and HomeNote doesn't.
- A test set we built ourselves on short vocal loops is left out until we've confirmed none of it overlaps HomeNote's training material.
Check it yourself
- homenote-key-test-songs.csv FMA track id, artist and title, genre, the experts' key and the song's licence, for all 684 songs
- homenote-key-test-results.csv every tool's answer for every song
The audio itself comes from the public FMAK and FMA datasets; each song keeps its artist's Creative Commons licence, so we link to it rather than re-share it.
Auto-Tune and Auto-Key are trademarks of Antares Audio Technologies. HomeNote is an independent project, not affiliated with Antares.