ISMIR - MIREX Panel Discussion
Stephen Downie presents the MIREX session Statistics for 2009:
26 tasks
138 participants
289 evaluation runs
Results are now published: http://music-ir.org/r/09results This year, new datasets:
MIR 1K
Back Chorales
Chord and Segmentation datasets
Mood dataset
Tag-a-Tune
Evalutron 6K - Human evaluations - this year, 50 graders / 7500 possible grading events. What's Next?
End-to-End systems and tasks
Qualitative assessments
Possible Journal 'Special Issue'
MIREX 2010 early start: http://www.music-ir.org/mirex/2010/index.php/Main_Page
Suggestions to model after the ACM MM Grand challenge
Issues about MIREX
Rein in the parameter explosion
Not rigorously tested algorithms
Hard-coded parameters, path-separators, etc
Poorly specified data inputs/outputs
Dynamically linked libraries
Windows submissions
Pre-compiled Matlab/MEX Submissions
The 'graduation' problem - Andreas and Cameron will be gone in summer.
Long discussion with people opining about tests, data. Ben Fields had a particularly good point about trying to make MIREX better reflect real systems that draw upon web resources.