Text Extraction from Videos – Techniques and Challenges

There have been various research efforts to find out the most efficient method of text extraction from complex videos and texts. The process of extraction only happens after detection and shot segmentation methods have been proposed as a solution to a variety of problem similar to that, that arise in the instances of detection and localization phases.

Before moving further, here’s an interesting piece of read on Text Extraction And The Role It Plays In Digital Transformation.

Videos as compared to still images usually always have a lower resolution and the text on screen, on different frames have poor contrast with the changing background. Optical Character Recognition (OCR solutions) is what is used to identify different typesets automatically through an ocular method. Similar to that of how a human eye “sees” and “processes” images in the brain, and how there’s a variety of factors that affect how these signals are processes, OCR works at identifying alphabets, numeric, special characters or punctuations present in digital media without any human touch. This is done by putting parallelly the extracted text with the library of text models in the software.

Extracting text from video deals with three main tasks- detection, localization and segmentation. Digital videos consist of two kinds of texts, namely- scene text: the text that is caught during the making of the video. An example is that of text written on buildings, banners or billboards. It is also called graphic text. Artificial text is the kind that is added to videos by editing software. It can appear at a particular position on the screen depending on whether it is a name card, subtitles or other information about the video. This is also called caption text or superimposed text.

The temporal nature of videos and elements amplify the problem of efficiently extracting text. The text could appear in the video for a few seconds, whilst in some videos the text could change and shift in non-linear and complex ways. They could get bigger, or reduce size, have different font colours and some text strings could break and join together in different frames of the video. The orientation, font style coupled with special effects and camera movements, can make the task of locating and extracting specific elements of text from a video a gruelling task. Assembling the necessary text regions into “text instances” is referred to as text localization.

Also Read: Extracting Text from Machine Drawing and Nested Tables – A teX.ai Success Story

Most text segmentation and recognition work is done on high resolution media and as previously mentioned, video frames have a relatively low quality of resolution. This in turn makes it so that the viewer experiences blurry effects due to lossy compression. A single frame can contain a variety of subjects with text on it and the orientation of the overlay text can also get very tasking as text may need to be of different sizes and colours. The segmentation circuit works upon a certain few bounding box which are determined in earlier stages of the system wherein the output is a binary image of the corresponding text in each box. The text pixels are coloured white and background pixels are colours black. The process of development of this module has been generalized to the extent that it can detect both artificial text and scene text that occur naturally on the screen. In particular, for the scene text to be proper delineated, the algorithm should also have capabilities to segment text from low lit videos or ones with uneven contrast and exposures.

• When human beings read a page of another language, we are able to identify some similar looking characters, but not ascertain the words and their meanings exactly. Similarly, OCR technology are usually able to interpret numerical section but not the alphanumerical ones with the same ease.
• There are also a lot of similarities when it comes to alphabets and numeric, for example, when examining a thread of letters and numbers, there is very little visible difference between a numeral “0” and a capital “O”. The human brain is able to contextualize the same and read back the sentence and determine the accurate meaning, but this practice is a lot more complicated for a machine.
• The human eye relies on contrast to read text and differentiate between negative space and graphics. It becomes a difficult task when the contrast is not enough or if the words are printed over each other- similarly, the OCR algorithm needs to be programmed in a way that only relevant data is picked up, which is a strenuous task for OCR developer.

A study of a variety of literature reveals that there is no video extraction system that is completely reliable. It has also been discussed that not one factor plays into the process of extraction, but multiple ones. Most of the algorithms are made for extracting text out of complex coloured images and just evolved into extracting for video data. Unspecified text colour, unknown text size, unconstrained background elements, colour bleeding, low contrast and low light instances are all some of the few challenges to successfully and efficiently be able to extract data from videos. The goal for the future should be to develop a text extraction software for videos that has a high recognition rate, recognizes a plethora of compound and special characters, and be able to use Artificial Intelligence to detect patterns to contextualize data. That will certainly be a subject of significance in the future.

Want to transform your business with proper decision-making? Choose teX-Ai, a trustable text analytics solution provider.