Unveiling Handwritten Text: A Guide to Masking Handwritten Content in Images

By Durgesh Rathod, WebApps Lead, SourceFuse

handwritten text masking

In today’s digital age, extracting text from images has become a common practice. However, when it comes to handwritten text, the task becomes more challenging due to variations in handwriting styles and complexities. In this guide, we’ll explore how to mask handwritten text in images using Python and Amazon Textract.

Understanding Handwritten Text Detection:

Amazon Textract is a powerful Machine learning service that can detect both printed and handwritten text in images. The provided Python code demonstrates how to leverage Textract to identify handwritten and printed text in an image. Let’s break down the key components of the code:

1. Initializing Textract Client:

The boto3 library is used to initialize the Textract client with the necessary AWS credentials and region information.

Install the necessary packages

$ pip install boto3 pillow

Set your AWS secrets in your environment variables by running below command in your terminal

export AWS_ACCESS_KEY_ID=<your_aws_access_key>
export AWS_SECRET_ACCESS_KEY=<your_aws_secret_access_key>
export AWS_REGION=<your_aws_region>

Import the libraries

import boto3
from PIL import Image, ImageDraw
import os

2. Detecting Text in the Image:

The detect_handwritten_and_printed_text method is invoked to detect text in the provided image. Textract returns a JSON response containing information about each detected block of text, including its bounding box coordinates.

text segmentation

3. Extracting Bounding Box Coordinates:

The code iterates through the response blocks to identify handwritten and printed text. For each text block, the bounding box coordinates are extracted, specifying the location of the text in the image.

def detect_handwritten_and_printed_text(image_path):
    # Initialize Textract client
    textract_client = boto3.client(
        "textract",
        aws_access_key_id=os.environ.get("AWS_ACCESS_KEY_ID"),
        aws_secret_access_key=os.environ.get("AWS_SECRET_ACCESS_KEY"),
        region_name=os.environ.get("AWS_REGION"),
    )
    # Read the image
    with open(image_path, "rb") as image_file:
        image_bytes = image_file.read()
    # Call Textract API to detect text
    response = textract_client.detect_document_text(Document={"Bytes": image_bytes})
    # Extract bounding box coordinates for handwritten text
    handwriting_text_rects = []
    printed_text_rects = []
    for item in response["Blocks"]:
        if "Text" in item:
            text_type = item.get("TextType", "")
            if text_type == "HANDWRITING" or (
                text_type == "PRINTED"
                and "Geometry" in item
                and "BoundingBox" in item["Geometry"]
            ):
                bbox = item["Geometry"]["BoundingBox"]
                width, height = Image.open(image_path).size
                left = width * bbox["Left"]
                top = height * bbox["Top"]
                right = left + (width * bbox["Width"])
                bottom = top + (height * bbox["Height"])
                if text_type == "HANDWRITING":
                    handwriting_text_rects.append((left, top, right, bottom))
                else:
                    printed_text_rects.append((left, top, right, bottom))
    return handwriting_text_rects, printed_text_rects

4. Handling Overlapping Text:

To accurately mask handwritten text without obscuring printed text, the code implements a method to determine overlapping regions between handwritten and printed text.

The is_overlap function is a fundamental part of the process for finding intersections between two rectangles. This function takes two rectangle objects (rect1 and rect2) as input and determines whether they overlap or not. It achieves this by comparing the coordinates of the rectangles' corners. If any of the conditions for non-overlapping are met, the function returns True, indicating that the rectangles do not overlap. Otherwise, it returns False, indicating an overlap between the rectangles

def is_overlap(rect1, rect2):
    """
    Check if two rectangles overlap.
    """
    return not (
        rect1[2] <= rect2[0]
        or rect1[0] >= rect2[2]
        or rect1[1] >= rect2[3]
        or rect1[3] <= rect2[1]
    )

5. Finding Intersection Areas:

The find_intersections_and_cropout_printed_text_rects function is responsible for identifying overlapping regions between two sets of rectangles and cropping out the printed rectangles accordingly. This function takes two lists of rectangles as input: handwritten_text_rects, which contains rectangles representing regions to be masked, and printed_text_rects, which contains rectangles representing regions not to be masked.

The function iterates through each rectangle in handwritten_text_rectsand checks if it overlaps with any rectangle in printed_text_rects. If an overlap is found, it divides the area of the rectangle into non-overlapping parts by comparing it with rectangles in printed_text_rects. It then adds the non-overlapping parts to the to_be_masked_rects list.

def find_intersections_and_cropout_printed_text_rects(
    handwritten_text_rects, printed_text_rects
):
    to_be_masked_rects = []
    for to_paint_rect in handwritten_text_rects:
        entirely_overlapped = False
        for not_to_paint_rect in printed_text_rects:
            if (
                to_paint_rect[0] >= not_to_paint_rect[0]
                and to_paint_rect[1] >= not_to_paint_rect[1]
                and to_paint_rect[2] <= not_to_paint_rect[2]
                and to_paint_rect[3] <= not_to_paint_rect[3]
            ):
                entirely_overlapped = True
                break
        if entirely_overlapped:
            continue
        remaining_area = [to_paint_rect]
        for not_to_paint_rect in printed_text_rects:
            new_remaining_area = []
            for area in remaining_area:
                if is_overlap(area, not_to_paint_rect):
                    # If there's an overlap, split the area and keep the non-overlapping parts
                    left = max(area[0], not_to_paint_rect[0])
                    top = max(area[1], not_to_paint_rect[1])
                    right = min(area[2], not_to_paint_rect[2])
                    bottom = min(area[3], not_to_paint_rect[3])
                    if left < right and top < bottom:
                        # Non-overlapping area on the left
                        if area[0] < left:
                            new_remaining_area.append((area[0], area[1], left, area[3]))
                        # Non-overlapping area on the right
                        if right < area[2]:
                            new_remaining_area.append(
                                (right, area[1], area[2], area[3])
                            )
                        # Non-overlapping area on the top
                        if area[1] < top:
                            new_remaining_area.append((left, area[1], right, top))
                        # Non-overlapping area on the bottom
                        if bottom < area[3]:
                            new_remaining_area.append((left, bottom, right, area[3]))
                    else:
                        # If the cropped area is degenerate, skip it
                        new_remaining_area.append(area)
                else:
                    # If there's no overlap, keep the original area
                    new_remaining_area.append(area)
            remaining_area = new_remaining_area
        # After handling all intersections, add the remaining areas to the painted area
        to_be_masked_rects.extend(remaining_area)
    return to_be_masked_rects

6. Creating Masked Image:

After identifying the intersecting regions, a new image is created where the handwritten text regions are masked with a white box, effectively concealing them from view.

def create_mask_bounding_rects(image_path, to_be_masked_rects):
    image_name = image_path.split("/")[-1]
    image_name
    # Open the image
    image = Image.open(image_path)
    # Convert to grayscale
    image = image.convert("L")
    # Mask the area with white box
    draw = ImageDraw.Draw(image)
    for box in to_be_masked_rects:
        draw.rectangle(box, fill="white")
    # Save or display the image
    image.save(f"./{image_name}")

Example Usage

The provided code snippet demonstrates how to utilize the functions for masking handwritten text in an image. By specifying the path to the image file, the script detects handwritten and printed text, creates a mask, and saves the resulting image.

if __name__ == "__main__":
    image_path = "/path/to/your/image.jpg"
    handwriting_bounding_rects, printed_text_rects = (
        detect_handwritten_and_printed_text(image_path)
    )
    to_be_masked_rects =   find_intersections_and_cropout_printed_text_rects(
        handwriting_bounding_rects, printed_text_rects
    )
    create_mask_bounding_rects(image_path, to_be_masked_rects)
    )

$ python main.py

Output:

We mask out the handwritten text while trying to prevent the overlapping printed text from being masked. In the output image we can see some handwritten text is not masked. It is so because AWS Textract didn’t labeled it as a handwritten text. It is important to recognize that the accuracy will also depend on the quality of images being used.

masked handwritten text

Conclusion

Masking handwritten text in images is a crucial task in various applications, including privacy protection and document analysis. With the capabilities of Amazon Textract and the Python programming language, developers can efficiently tackle this challenge. By following the guidelines outlined in this guide, you can seamlessly integrate handwritten text masking into your image processing pipelines. Unlock the potential of your image data while preserving privacy and confidentiality.

We would love to hear from you! LET’S TALK!

Contact Us