Stamp2vec

This project is WIP. Stamp2vec is a project which aims to build a stamp detection and feature extraction of stamps. The goal is to provide various end-to-end pipelines for stamp detection and stamp feature extraction. It contains implementation of papers [1], [2] and uses various frameworks [3], [4].

Methodology

The stamp recognition system follows the following steps:

Detection: One of the available YOLO models predicts bounding boxes of stamps on the image.

Segmentation (optional): Segmentation model highlight all the pixels belonging to a stamp, to get a better embedding on the next step.

Embeddings: One of the available embeddings models returns an array of values that best describes a particular stamp.

Classification: Embeddings of different stamps can be compared using methods such as cosine similarity.

Usage

Install needed libraries

pip install -r requirements.txt

One of the goals of the project is to provide simple interface for using trained models. Therefore, this project incorporates various pipelines written for each model which includes pre-processing, inference and postprocessing.

#import pipeline from the module
from pipelines.detection.yolo_stamp import YoloStampPipeline
from PIL import Image
#initialize it with a path to a huggingface space
pipe = YoloStampPipeline.from_pretrained("stamps-labs/yolo-stamp")
#load PIL image
img = Image.open("dataset_generation/images/images/img9.png")
#Print the results of inference (coordinates in this case)
print(pipe(img))

Features

1. A deployed app on HuggingFace with simple inference to test models. Functionallity includes:
   

   1. Showing bounding boxes of stamps
      
   2. Get stamps embeddings
      
   3. (optional) Get segmented stamps
      
   4. Show cosine similarity between all stamps on the document
      

2. No 3rd party libraries for inference.
   Lots of different libraries and frameworks are bad for versioning and usage. We had to overcome challenge of using less libraries for usage of this project therefore we have used torch JIT model compiler in order to make this project as lightweight as possible.

3. Pipelines for ML models.
   Pipelines (referenced above) are:
   

   1. Easy to use
      
   2. Includes post- and pre-processing
      
   3. Allows storing models on dedicated server
      
   4. Backwards compatibility
      

4. Various model presets During development of this project we trained and experimented with multiple different architectures as well as implemented a few papers in order to achieve the best results in terms of quality and performance. Thus, we have different models for each use case:

   1. Lightweight models for fast inference (Yolo-stamp, VAE)
   2. Heavier models for best results (YOLO v8, dino-vits8)

Resources

This project contains implementation of "Fast and Accurate Deep Learning Model for Stamps Detection for Embedded Devices" [1] and incorporates techniques mentioned in "TVAE: Triplet-Based Variational Autoencoder using Metric Learning" [2].
This project also uses Open-Metric-Learning framework [3] for training the Deep metric learning models with Triplet Loss and Ultralytics framework [4] for YoloV8 training.

Additional information

All project artifacts(models, spaces, datasets*) can be accessed on HuggingFace classroom stamps-labs
*in future

References

[1] Gayer, A., Ershova, D. & Arlazarov, V. Fast and Accurate Deep Learning Model for Stamps Detection for Embedded Devices. Pattern Recognit. Image Anal. 32, 772–779 (2022). https://doi.org/10.1134/S1054661822040046
[2] Ishfaq, H., Hoogi, A., & Rubin, D.L. (2018). TVAE: Triplet-Based Variational Autoencoder using Metric Learning. ArXiv, abs/1802. 04403. https://doi.org/10.48550/arXiv.1802.04403
[3] https://github.com/OML-Team/open-metric-learning
[4] https://github.com/ultralytics/ultralytics
[5] https://github.com/pytorch/pytorch
[6] https://github.com/gradio-app/gradio
[7] https://github.com/tiangolo/fastapi
[8] https://huggingface.co/