** Submission is now closed. **

We thank all who participated in CROHME 2016. We plan to make this page active again in the near future so that people can submit additional runs on their own.

CROHME 2016 Submission Instructions

CROHME 2016 results will be submitted directly online by participants. Submitted results may be updated for each task, and multiple times if necessary. NOTE: Each competition task has a 'test' version used for computing competition results, along with a 'validation' version for error and sanity-checking.

Submitting Results

1. Create an Account

You will need to create an account on the submission system. Select the 'Register' link at top-right, and then provide a participant id and password. This will create an account for your team that can be used to view results for your submissions to the various tasks.

2. Produce Result Files

Tasks 1, 3, and 4 (Formulas, Structure, and Matrices)

Have your recognition system produce results in label graph (.lg) format (see the LgEval library README for details). Participants need to produce one .lg file for each test file. Label graph files are .csv files providing the segmentation of strokes into symbols, classification of symbols, and labeling of spatial relationships between symbols. We recommend using the Object-Relationship (OR) .lg file format. This format allows stroke groups (e.g., symbols) to be named, and then relationship to be defined directly between stroke groups. Earlier .lg formats (e.g. Node-Edge (NE) files) will also work.

Tasks 2a and 2b (Symbol Classification)

The symbol classification tasks use a different .csv file format, with a single .csv file providing the classification results for all input files. An example .csv classification result file is shown below.

MfrDB3907_85801, a, b, c, d, e, f, g, h, i, j
MfrDB3907_85802, 1, |, l, COMMA, junk, x, X, \times
Each line starts with a symbol identifier, which is defined for each .inkml file using the tag:
<annotation type="UI">
Remaining entries on a line are symbol classes, provided in decreasing order of confidence. For the first symbol in the example below, class 'a' has the highest confidence, followed by class 'b,' etc.

3. Upload Results

Tasks 1, 3 and 4 (Formulas, Structure, Matrices)

For each task that you want to submit results for, you will need to submit all .lg files. We recommend uploading a .zip file containing all .lg files for a task; multiple .zip and/or .lg files may be selected for upload. Log into your user account, and click on the Upload link at the top of the page. Then:

  1. Select the appropriate task
  2. Click on "Choose Files"
  3. Select your .zip and/or .lg file(s) for submission.
  4. IMPORTANT: Click on "Upload Files" to submit your results for evaluation.

Please Note: computing results may take several minutes, depending on the server load, complexity of the task, etc. While results are being computed, a message will be displayed. Refresh the web page to check for when results have been updated.

Tasks 2a and 2b (Symbol Classification)

Submission is the same as for Tasks 1, 3 and 4, but for each subtask, you need to instead upload a single .csv file in the classification results format shown above. Submissions should be a single .csv file, with results for all symbols.

4. View Results

Once results have been computed, visible results will differ depending on the task that you submit to; detailed results are not visible for the new test sets for the 'main' tasks (tasks 1, 2a, 2b, 3 and 4).

For the other official and validation tasks, we provide detailed error information computed using the LgEval and CROHMELib libraries, including:
  1. Summary: a text file with results for
    • Stroke level (Primitives)
    • Symbol level (Objects), and
    • Expression or Symbol level (Files)

    Included are recall, precision and f-measure metrics, along with error metrics (e.g. false positives and false negatives for symbol segmentation). Recognition rates (i.e. percentage of GT targets recognized correctly) are represented by Recall. Percentage of correct classifications for correct detections is shown using Class/Det.

  2. confMat: Stroke-level confusion matrix.
  3. confMat1: Confusion histograms for individual symbols in ground truth (see below).
  4. confMat2: Confusion historgrams for pairs of symbols in ground truth (see below).
  5. symbolResults: Detailed information regarding classification results.

5. Updating Results

If you wish to update your results for a given task (e.g., after correcting bugs, or improving performance in some way), you may submit new results by repeating the process described in Step 3.

Important Notes

Ranking Participant Submissions

For each task, participants will be ranked by their submission with the highest recognition rate, after any corrections to ground truth have been made by the competition organizers. It is quite likely that corrections to ground truth will be found during or after the results submission period for the competition.

Participants may be asked to re-submit results to a 'new' task if ground-truth is updated.

Recovering Submitted Files

From your participant account, you may download previously uploaded submissions, in case you accidentally lose track of them.

Error Visualizations

File-Level Error Visualization

For the formula parsing validation tasks, the interface allows recognition errors to be viewed in .pdf-rendered dot graphs, with errors shown in red. You may select to have these visualizations produced as Symbol Layout Trees, or Stroke Label Graphs, which will be generated and stored in a .zip file that you can download.

Confusion Histograms

Confusion histograms simultaneously count and visualize errors from symbol segmentation, clasification, and parsing at the stroke level. A description of confusion histograms and their contents may be found in the paper below. Errors are presented in decreasing order of frequency. For example, specific errors for segmenting and classifying the symbol 'x' written using one vs. two strokes may be easily seen using confusion histograms. We also provide confusion histograms for pairs of symbols with a spatial relationship in ground truth, such as 'x squared' and '2x.'

All files with errors may be identified by clicking on check boxes shown in the generated .html files, and then exporting the list of files as text using the button at the top of the page.

H. Mouchère, R. Zanibbi, U. Garain and C. Viard-Gaudin. (2016) Advancing the State-of-the-Art for Handwritten Math Recognition: The CROHME Competitions, 2011-2014. Int'l Journal on Document Analysis and Recognition, 19(2): 173-189.

The penultimate version of this paper is available online: www.cs.rit.edu/~rlaz/publications.html.


If you have questions, please email competition organizers Richard Zanibbi and Harold Mouchère:



We thank Yi Huang, who created the main design for this system, and implemented the system core for his MSc project. We also thank the systems administrators at RIT CS (James Craig and Sam Waters) for helping set up the web server.