CROHME 2016 Submission Instructions

CROHME 2016 results will be submitted directly online by participants. Submitted results may be updated for each task, and multiple times if necessary. NOTE: Each competition task has a 'test' version used for computing competition results, along with a 'validation' version for error and sanity-checking.


Submitting Results

1. Create an Account

You will need to create an account on the submission system. Select the 'Register' link at top-right, and then provide a participant id and password. This will create an account for your team that can be used to view results for your submissions to the various tasks.

2. Produce Result Files

Tasks 1, 3, and 4 (Formulas, Structure, and Matrices)

Have your recognition system produce results in label graph (.lg) format (see the LgEval library README for details). This is a .csv file providing the segmentation of strokes into symbols, classification of symbols, and labeling of spatial relationships between symbols. We recommend using the Object-Relationship (OR) .lg file format. This format allows stroke groups (e.g., symbols) to be named, and then relationship to be defined directly between stroke groups. Earlier .lg formats (e.g. Node-Edge (NE) files) will also work.

Tasks 2a and 2b (Symbol Classification)

The symbol classification tasks use a different .csv file format. An example is shown below. Each line starts with an identifier for the .inkml file associated with a symbol. Remaining entries on a line are symbol classes, provided in decreasing order of confidence. For the first symbol in the example below, class 'a' has the highest confidence, followed by class 'b,' etc. Our evaluation tools consider up to the top-10 classes for each symbol.

MfrDB3907_85801, a, b, c, d, e, f, g, h, i, j
MfrDB3907_85802, 1, |, l, COMMA, junk, x, X, \times

3. Upload Results

Tasks 1, 3 and 4 (Formulas, Structure, Matrices)

For each task that you want to submit results for, you will need to create a .zip file of all the .lg files. The .lg files should all be in a single directory. Then log into your user account, and click on the Upload link at the top of this page. Then:

  1. Select the appropriate task
  2. Click on "Choose Files"
  3. Select your zip file of .lg results.
  4. IMPORTANT: Click on "Upload Files" to submit your results for evaluation.

Please Note: computing results may take several minutes, depending on the server load, complexity of the task, etc. While results are being computed, a message will be displayed. Refresh the web page to check for when results have been updated.

Tasks 2a and 2b (Symbol Classification)

Submission is the same as for Tasks 1, 3 and 4, but for each subtask, you need to instead upload a .csv file containing classification results as given in the format shown above. Note: Submissions should be a single .csv file, with results for all symbols.

4. View Results

Once results have been computed, visible results will differ depending on the task that you submit to.

  1. Official CROHME 'Test' Task Results: Only the expression or symbol rate may be displayed.
  2. Validation Tasks: These provide detailed error information computed using the LgEval and CROHMELib libraries, including:
    1. Summary: a text file with results for
      • Stroke level (Primitives)
      • Symbol level (Objects), and
      • Expression or Symbol level (Files)

      Included are recall, precision and f-measure metrics, along with error metrics (e.g. false positives and false negatives for symbol segmentation). Recognition rates (i.e. percentage of GT targets recognized correctly) are represented by Recall. Percentage of correct classifications for correct detections is shown using Class/Det.

    2. confMat: Stroke-level confusion matrix.
    3. confMat1: Confusion histograms for individual symbols in ground truth (see below).
    4. confMat2: Confusion historgrams for pairs of symbols in ground truth (see below).

5. Updating Results

If you wish to update your results for a given task (e.g., after correcting bugs, or improving performance in some way), you may submit new results by repeating the process described in Step 3.


Important Notes

Ranking Participant Submissions

For each task, participants will be ranked by their submission with the highest recognition rate, after any corrections to ground truth have been made by the competition organizers. It is quite likely that corrections to ground truth will be found during or after the results submission period for the competition. During this time your recognition rates and other results may change as a result of corrections made, and these may be reflected in the interface.

Recovering .zip Output Files

From your participant account, you may download previously uploaded submissions, in case you accidentally lose track of them.


Error Visualizations

File-Level Error Visualization

For the validation tasks, the interface allows recognition errors to be viewed in .pdf-rendered dot graphs, with errors shown in red. You may select to have these visualizations produced as Symbol Layout Trees, or Stroke Label Graphs, which will be generated and stored in a .zip file that you can download.

Confusion Histograms

Confusion histograms simultaneously count and visualize errors from symbol segmentation, clasification, and parsing at the stroke level. A description of confusion histograms and their contents may be found in the paper below. Errors are presented in decreasing order of frequency. For example, specific errors for segmenting and classifying the symbol 'x' written using one vs. two strokes may be easily seen using confusion histograms. We also provide confusion histograms for pairs of symbols with a spatial relationship in ground truth, such as 'x squared' and '2x.'

All files with errors may be identified by clicking on check boxes shown in the generated .html files, and then exporting the list of files as text using the button at the top of the page.

H. Mouchère, R. Zanibbi, U. Garain and C. Viard-Gaudin. (2016) Advancing the State-of-the-Art for Handwritten Math Recognition: The CROHME Competitions, 2011-2014. Int'l Journal on Document Analysis and Recognition, 19(2): 173-189.

The penultimate version of this paper is available online: www.cs.rit.edu/~rlaz/publications.html.

Contact

If you have questions, please email competition organizers Richard Zanibbi and Harold Mouchère:

rlaz@cs.rit.edu
harold.mouchere@univ-nantes.fr