Enhance pending documents API

[allanlasser]

Problems

• We have to poll for changes to processing state. If the API provided a streaming response, we could avoid hitting the endpoint multiple times.
• Once the document is done processing, it’s removed from the results. Because we’re polling for data, once the document is done, it disappears from our processing list.
• When dispatching a bulk reprocess from the API, all those documents will appear in the web indicator
• Get pending status for a single document #230
• Handle pending documents that get stuck in processing #211

Proposed Solutions

• Document processing jobs are inferred by filtering documents for a “pending” status. We should consider creating a new table for processing jobs, that track details like:
  • start time and end time
  • a value to track job progress
  • relation to the document, or documents, involved in the job
  • track the client origin of a processing job: did it come from the web client, an API client, someplace else
  • optionally, some kind of human-readable summary or description of the job: “Applying OCR to 15 documents”
  • a boolean “unread” or “dismissed” status, so that clients know which completed jobs to display to users
• Provide date and origin filters for processing jobs, so that the client can make a request like “send me all pending or non-dismissed completed jobs created from the frontend client in the last week”
• Provide a streaming endpoint so that the client only has to make one request for the list of processing documents, instead of continuously polling the endpoint.

[eyeseast]

Another thing this would let us do: If we're tracking start and end time for processing jobs, we could investigate documents that take an unusually long time to process and figure out why.

[allanlasser]

• We could also have a Celery task that checks the table of jobs for stuck processing tasks.
• We would need a retention policy for the table, so that we prune out old jobs and prevent the database from growing to an unmanageable size.
• We'd want a link to the document file that resulted from the task as fallback for failed jobs (Error in grafting ocr'd text back into PDF #336).