A Platform for Large Scale Auto Annotation of Scanned Documents Featuring Real-Time Model Building and Model Pooling

Abstract

Document digitization is an active area of research especially involving handwritten manuscripts. While the most common use cases involve digital libraries, there are other important applications in the area of electronic health records where handwritten text is predominant in developing worlds. The state-of-the-art approaches are domain-specific, and scaling across domains is still an open research problem. We report here a platform for real-time annotation and training of sub-region models in scanned documents using model pools and plug-n-play of annotation services. Given a document, sub-regions are annotated with textual labels. The textual regions themselves may correspond to characters or words or any other pattern of interest. For a given sub-region category, several sub-regions may be present in a given page or across pages. In the proposed system, a user needs to annotate only some of the sub-regions. A convolutional neural network (CNN) model is built for each of the sub-region categories, and named sets or pools of such models are prepared for application on any new document. We observe that a sub-region label may be provided by an existing optical character recognition system instead of a human annotator. In this regard, we have provisioned annotation as a service where any third-party system can be integrated into a plug-n-play mechanism. The state-of-the-art systems focused on having a pre-trained monolithic model which suffers from the problem of catastrophic forgetting when new sub-region classes are added over time. In our approach, due to sub-region specific models, the previous data models are not touched and hence providing a truly incremental learning solution. We have carried out the validation by choosing handwritten data sets belonging to different languages such as Devanagari, Kannada, Telugu, English that span diverse text patterns and the models produced by our sub-region detection algorithm were evaluated on documents containing hundreds of handwritten scripts by several authors. With respect to the performance of our models on the validation data sets, we found mAP scores for different data sets as follows: Devanagari words (96.18); Telugu words (93.20); Devanagari letters (100); Kannada letters (99.83); Tesseract English word-level annotations (90). We have also presented a single page annotation as proof of concept for annotation as a service for Kannada, Telugu, Malayalam, and English recognition to learn from Tesseract annotations.