OCR Updates

[ethanmick]

An overarching issue that encapsulates the scope of what I want the next OCR app to be. In general, I want to streamline the OCR'ing so I need to jump out to external programs and processes less. Ideally, not at all. I'd also like this version to be compatible with other people really using it to OCR recipes, and have it "just work" as expected.

Requirements

This update will need a new uploads table for tracking User Uploaded files that need to be OCR'ed. Something like:

-- Not Final
CREATE TABLE uploads (
  id SERIAL PRIMARY KEY,
  user_id integer NOT NULL,
  status integer NOT NULL,
  file_location varchar NOT NULL,
  -- The created recipe from this file
  recipe_id integer NULL,
  -- Who OCR'ed this file
  signed_off_by integer NULL,
  -- Note/Communication
  user_notes varchar NULL,
  platezero_notes varchar NULL,
  error_reason varchar NULL,
  created_at timestamp without time zone NOT NULL DEFAULT now(),
  updated_at timestamp without time zone,
  deleted_at timestamp without time zone
);

Files uploaded are given a PENDING status and can be viewed in the user profile. When OCR'ed, the record is updated and connects the original asset to the new recipe, and gives credit to the PlateZero person who did the OCR'ing. If any issue occurs during the process, we can try to communicate back to the person (maybe email is fine), or if no recipe was found, a reason is given and the import "fails".

Flow

On startup, login to PlateZero (or remembers your session) and checks if you are authorized to OCR recipes. Adds a Menu Icon to the MacOS bar that "lights up" when new recipes have been uploaded that need OCRing.

Main page is the list of recipes that need to be OCR'ed. Grouped by user, but remove the user_id for another identifier to increase anonymity. You can use Quick Look to take a... quick look at a file before OCRing. You can archive and link to a recipe or archive and write an error message.

When selecting a recipe to OCR, it brings you to the OCR step. You can always go back to the main list of recipes from this page. You can also select another image to load into the OCR page if a recipe spans multiple images. OCRing works mostly as it currently does, except with the new upgraded Ingredient/Procedure lines. Add in Preheats (try to guess from procedure line text still). Add in Image.

When finished, add in a new step for "preview" the recipe. Show the details as they will roughly appear on PlateZero. The user can press "back" and go back and correct any details. When the recipe looks correct, the user can:

1. Create recipe & send email
2. Create recipe and delay sending email
3. Cancel and go back to recipe selection

Option 2 adds a new "bucketing" feature, allowing you to OCR multiple recipes by the same user. When you decide to bucket recipes, it delays sending an email until you have reached a stopping point and then you send one email containing the links to all the recipes you just OCR'ed. (Bucketing only delays sending the email, the recipes are still created immediately after the OCRing).

[ethanmick]

@benburwell Am I missing anything?

[ethanmick]

A few other small features.

• Reload image from disk (Allowing for external manipulation and reloading)
• Open image in a given external program (right click, open as)
• Expand recipe sidebar to take up more room while OCRing
• Rotate in varied amounts
• Allow for any/more convert commands
• Preview/change email before sending

[benburwell]

This looks pretty good. A few high-level thoughts: 1. How are you thinking of handling non-image file uploads, e.g. Word documents? Convert to images and then OCR? Or are you thinking of using something more along the lines of pandoc and simply selecting text and skipping the Vision API? 2. I'm really not convinced that this should be an Electron app rather than just part of PlateZero. I feel like this just makes it unnecessarily complicated, especially when we start wanting other people to participate in the OCR process. With an electron app, we need to now manage the process of getting it installed for various people, auto updates, yet another auth flow, harder to share code with the website (e.g. the recipe editor), etc. The advantages to using an Electron app are some of the native integration features as you mentioned, like the menu bar icon, and being able to call other binaries to do work, like imagemagick, but I can't really think of any others. Even the latter is a double-edged sword, as it will lead to compatibility issues across different OSes etc. And now that we have imageproxy.platezero.com, which has the ability to do format conversion, resizing, rotation, cropping, etc, I think we could do everything that we're using imagemagick for fairly easily in the browser. (see https://godoc.org/willnorris.com/go/imageproxy#ParseOptions) More specific feedback inline:

CREATE TABLE uploads ( id SERIAL PRIMARY KEY, user_id integer NOT NULL, status integer NOT NULL, file_location varchar NOT NULL, -- The created recipe from this file recipe_id integer NULL, -- Who OCR'ed this file signed_off_by integer NULL, -- Note/Communication user_notes varchar NULL, platezero_notes varchar NULL, error_reason varchar NULL, created_at timestamp without time zone NOT NULL DEFAULT now(), updated_at timestamp without time zone, deleted_at timestamp without time zone );

This pretty much looks correct. I might swap out `user_id` for `uploader_id` or something that makes it a little more explicit as to which user it's talking about. I'm not sure that "signed off" is the correct term, "signed off" sounds more like the person who approved the OCR from a quality assurance perspective, but it sounds like you're talking about the person who actually did the OCRing. Also I think we want to continue to use the pattern of appending `_id` to foreign keys so that when we construct the Upload model and UploadJSON, we store e.g. the user's ID in `user_id` and the embedded User model in `user`. Related: are we assuming that everyone who will be processing uploads has a PZ account? I think that is OK, but we should make this explicit if that is the case.

When OCR'ed, the record is updated and connects the original asset to the new recipe, and gives credit to the PlateZero person who did the OCR'ing.

I'm a bit curious as to what your thoughts are here. What do you mean by "gives credit"?

If any issue occurs during the process, we can try to communicate back to the person (maybe email is fine), or if no recipe was found, a reason is given and the import "fails".

I don't think we should have any sort of back-and-forth flow. If we can't figure out what to do with whatever they gave us, we should just fail the upload, probably by sending them an email and attaching the file they uploaded (or providing a link to it) so they can remember what it was, and explain what the problem we had was. Then, to correct it, they can just re-upload it.

On startup, login to PlateZero (or remembers your session) and checks if you are authorized to OCR recipes.

I suppose we will add a column to the users table for this.

Grouped by user, but remove the user_id for another identifier to increase anonymity.

Perhaps; I like this in general, but suspect it might be a little complicated to implement correctly. If I'm wrong, then let's do it. I think this will still be a relatively internal tool, so I'm not too concerned with this yet, but I agree it's something we'll want if we open this up to truly random people (whether contractors or PZ users).

You can use Quick Look to take a... quick look at a file before OCRing.

Do you literally mean macOS's quicklook service? That would severely limit who can use the app, or at least this feature. Thumbnails might be nice either instead of or in addition to this.

You can also select another image to load into the OCR page if a recipe spans multiple images.

I find a more natural flow to be selecting multiple images to OCR together in the first step. E.g. in the list view, put checkboxes or something next to each uploaded file and have a button to "OCR selected" which would bring me to the OCR step with all the images I selected available to grab text from. I sort of think we want to do both, actually: select multiple images from the get-go, and have the ability to pick more images once you've already started.

When finished, add in a new step for "preview" the recipe. Show the details as they will roughly appear on PlateZero. The user can press "back" and go back and correct any details.

Can this simply be handled by using the same components as the recipe editor which already show a side-by-side preview?

When the recipe looks correct, the user can: 1. Create recipe & send email 2. Create recipe and delay sending email 3. Cancel and go back to recipe selection Option 2 adds a new "bucketing" feature, allowing you to OCR multiple recipes by the same user. When you decide to bucket recipes, it delays sending an email until you have reached a stopping point and then you send one email containing the links to all the recipes you just OCR'ed. (Bucketing only delays sending the email, the recipes are still created immediately after the OCRing).

This sort of feels like something that should be handled automatically, not by the person doing the OCRing. I think we could do something like: * When a recipe is OCRd, check the email queue for an existing email being sent to the same user. If there's a queued email, update it with the additional recipe. Otherwise, add a new email to the queue. * x minutes (30?) after an email has been last touched, or y hours after it was first queued (4?), send the email. In practice, this would probably be implemented as something like a cron job which checks uploads for their `finished_at` timestamp and `email_sent_at` timestamps so that we don't need to handle the very stateful process of "wait for 30 minutes" which could be interrupted by app restarts etc.