Project description

The general workflow is the following: user draws a digit and presses the button "Predict". Javascript encodes image into 64 bit encoded JPG URL, which is passed into Python. Image is decoded and processed by cv2, which tries to find separate objects in the image. Then model makes predictions for each object. These predictions are passed back into Javascript and it shows them on the page.

Drawing the digits

Functions to draw on canvas are defined in draw.js. It is possible to draw with mouse on PC and with touches on mobile devices. Button "Clear" clears the canvas and hides unnecessary elements of interface. Pressing button "Predict" sends the data to python script (64 bit encoded JPG URL) with ajax request.

Receiving the encoded image and predicting labels

The encoded image is passed into main.py. It is decoded and is sent to functions.py. Weights of CNN are loaded when the app is initialized, so predictions can be done quickly. The weights are kept in the app on Heroky. Then the image is processed:

After this CNN model makes prediction for each object, the result is either a number or '-' for non-digits.