Open-Set Scene Text Recognition

How can we read characters that we see around us on a daily basis, such as on buildings and signs (aka Scene text)? How can we recognize the letters if they are ones we have not seen before (aka Open-set)?

Rather than retraining a model everytime we manually find unseen characters in the datastream after user feedbacks, we are developing open-set text recognition models that can automatically reject new labels and adapt to recognize them without compulsory model retraining. We also want to be able to recognize words written horizontally and words written vertically.

Open-set Text Recognition (OSTR) is an upgrade to how AI reads text, moving away from "fixed" alphabets to a more flexible, human-like way of learning.

1. The Problem: "Silent Errors"

Standard AI models are usually trained on a fixed set of characters (like A-Z). If they see something new—like a math symbol or a foreign character—they don't say "I don't know." Instead, they "silently" guess the closest thing they recognize (e.g., mistaking a $\theta$ for a "0").

2. Step One: Detection (Spotting the New)

The first goal of OSTR is Anomaly Detection. The model must be smart enough to realize: "Wait, I’ve never seen this character before," rather than just making a mistake.

3. Step Two: Adaptation (Learning on the Fly)

Usually, to teach an AI a new character, you have to retrain the whole thing, which is slow and expensive. OSTR aims to:

• Use side-information (like a description or a single example).
• Incrementally learn the new character immediately without needing a full system reboot.

In short: OSTR is about creating an AI that can recognize when it’s looking at something unfamiliar and then learn what that new thing is instantly.

We also have models that in addition to spotting and recognizing unseen characters, can handle the diversity in writing direction. This uses a mixture of expert structure.