1. Motivation

I have completed the first version of this project in summer 2017 and since then wanted to create a better version. There were several possible improvements in my mind: using other frameworks, recognition of several digits and separating non-digits. Alas, I lacked skills and free time to do this. Finally at the end of 2018 an opportunity appeared: I was taking part in a Pytorch challenge course from Udacity and could do a side project. This app is that side project. It works using Pytorch and cv2.

2. Development

It was possible to use the old version of the app and make changes to it. Thanks to the app I had more than 6000 handwritten images collected, but I had to check their labels and make corrections when necessary. After this is was necessary to choose an approach to digit detection and separation of digits and non-digits. I have tried several approaches including SVM for classifying object into digits and non-digits. I contemplated using siamese networks for this and yolo for digit detection, but in the end it turned out that simpler approaches work good enough.

Model training

Model training

3. Pytorch and cv2

My final solution looks like this: cv2 for image detection and CNN in Pytorch on 11 classes to make predictions. At first I use cv2 to find separate contours on the image and to draw bounding rectangles on them. This way I can separate all drawn object on the image and work with each of them. I have trained a CNN with 2 convolutional layers on 11 classes (10 digits and "other") with SGD optimizer and CosineAnnealingLR. It was somewhat tricky to train considering number of samples in "other" category was quite low. Still I achieved accuracy of ~89% on validation data.

4. Other things

I contemplated adding a possibility of training model on additional data, but dropped this idea - training on several digits could lead to overfitting. Maybe I'll retrain model manually once per month on newly acquired samples. One fun feature was added - showing number of predictions made since the app is active. Sadly apps on Heroku on free plan go to sleep after 30 minutes of inactivity.

Confusion matrix for validation

Confusion matrix