CROHME 2016 results will be submitted directly online by participants. Submitted results may be updated for each task, and multiple times if necessary. NOTE: Each competition task has a 'test' version used for computing competition results, along with a 'validation' version for error and sanity-checking.
You will need to create an account on the submission system. Select the 'Register' link at top-right, and then provide a participant id and password. This will create an account for your team that can be used to view results for your submissions to the various tasks.
Have your recognition system produce results in label graph (.lg) format (see the LgEval library README for details). This is a .csv file providing the segmentation of strokes into symbols, classification of symbols, and labeling of spatial relationships between symbols. We recommend using the Object-Relationship (OR) .lg file format. This format allows stroke groups (e.g., symbols) to be named, and then relationship to be defined directly between stroke groups. Earlier .lg formats (e.g. Node-Edge (NE) files) will also work.
The symbol classification tasks use a different .csv file format. An example is shown below. Each line starts with an identifier for the .inkml file associated with a symbol. Remaining entries on a line are symbol classes, provided in decreasing order of confidence. For the first symbol in the example below, class 'a' has the highest confidence, followed by class 'b,' etc. Our evaluation tools consider up to the top-10 classes for each symbol.
MfrDB3907_85801, a, b, c, d, e, f, g, h, i, j MfrDB3907_85802, 1, |, l, COMMA, junk, x, X, \times
For each task that you want to submit results for, you will need to create a .zip file of all the .lg files. The .lg files should all be in a single directory. Then log into your user account, and click on the Upload link at the top of this page. Then:
Please Note: computing results may take several minutes, depending on the server load, complexity of the task, etc. While results are being computed, a message will be displayed. Refresh the web page to check for when results have been updated.
Submission is the same as for Tasks 1, 3 and 4, but for each subtask, you need to instead upload a .csv file containing classification results as given in the format shown above. Note: Submissions should be a single .csv file, with results for all symbols.
Once results have been computed, visible results will differ depending on the task that you submit to.
Included are recall, precision and f-measure metrics, along with error metrics (e.g. false positives and false negatives for symbol segmentation). Recognition rates (i.e. percentage of GT targets recognized correctly) are represented by Recall. Percentage of correct classifications for correct detections is shown using Class/Det.
If you wish to update your results for a given task (e.g., after correcting bugs, or improving performance in some way), you may submit new results by repeating the process described in Step 3.
For each task, participants will be ranked by their submission with the highest recognition rate, after any corrections to ground truth have been made by the competition organizers. It is quite likely that corrections to ground truth will be found during or after the results submission period for the competition. During this time your recognition rates and other results may change as a result of corrections made, and these may be reflected in the interface.
From your participant account, you may download previously uploaded submissions, in case you accidentally lose track of them.
For the validation tasks, the interface allows recognition errors to be viewed in .pdf-rendered dot graphs, with errors shown in red. You may select to have these visualizations produced as Symbol Layout Trees, or Stroke Label Graphs, which will be generated and stored in a .zip file that you can download.
Confusion histograms simultaneously count and visualize errors from symbol segmentation, clasification, and parsing at the stroke level. A description of confusion histograms and their contents may be found in the paper below. Errors are presented in decreasing order of frequency. For example, specific errors for segmenting and classifying the symbol 'x' written using one vs. two strokes may be easily seen using confusion histograms. We also provide confusion histograms for pairs of symbols with a spatial relationship in ground truth, such as 'x squared' and '2x.'
All files with errors may be identified by clicking on check boxes shown in the generated .html files, and then exporting the list of files as text using the button at the top of the page.
H. Mouchère, R. Zanibbi, U. Garain and C. Viard-Gaudin. (2016) Advancing the State-of-the-Art for Handwritten Math Recognition: The CROHME Competitions, 2011-2014. Int'l Journal on Document Analysis and Recognition, 19(2): 173-189. The penultimate version of this paper is available online: www.cs.rit.edu/~rlaz/publications.html.
If you have questions, please email competition organizers Richard Zanibbi and Harold Mouchère: