Matching the groundtruth with the hypothesis baselines for CER / WER

[icarl-ad]

Hi,

I was wondering how the baselines (or the transcription of baselines) of the groundtruth and hypothesis are matched? This is obviosly important for the calculation of KPIs.

If e.g. the levenshtein distance is applied, how does this work with an extremly bad transcription or baselines that occur only in groundtruth or hypothesis?

Thank you for your help!

[mikegerber]

(Baselines is not the right term here, so I am using the term "textlines")

dinglehopper matches the extracted text in order (e.g. ReadingOrder for PAGE documents, or in document order in ALTO, text files etc.). It uses the shortest alignment as per the Levenshtein algorithm.

If you have lines that only appear on one side, there will be no match (or a bad match). An example (from the README):

https://raw.githubusercontent.com/qurator-spk/dinglehopper/master/.screenshots/dinglehopper.png

If you look at the second line, there's no match on the left (GT) side and the extra text on the right side (OCR) is counted as an error.

[icarl-ad]

Does that mean that the levenshtein distance is calculated from every textline of the ground truth to every textline of the hypothesis and based on the smallest distance the allignment is made?

[mikegerber]

No, the alignment/calculating the distance works on the full concatenated text, not on single textlines.

[mikegerber]

Note that calculating the distance requires alignment, and so is essentially the same thing.

[icarl-ad]

Can you elaborate on that? I don´t really understand how calculating the levenshtein distance on the whole transcription would be helpful, escpecially since it sounds like the alignment of the textlines has been set beforehand. Thank you so much again!

[mikegerber]

1. Both texts are extracted, in the order defined in the files. At this point you do not have any textlines anymore, but a series of characters (including newline characters).
2. Then the Levenshtein distance is calculated.

Calculating the Levenshtein distance involves aligning the two texts, so you don't get "a distance", but "the distance" = the shortest series of operations to get from one text to the other.

[mikegerber]

@icarl-ad mailed me about this.

There seems to be a fundamental misunderstanding: There are no textlines matched. dinglehopper does not work with lines.

1. The text is extracted, in reading order, as defined by the input data.
2. The resulting sequences of characters are compared using the Levenshtein algorithm (yields differences and distance aka CER) as implemented by the RapidFuzz library. (Words are extracted from this text, too, and then, sequences of words are compared.)