How to make a Job execute after another Job finishes?

[blikblum]

I have an application that is needing some complex orchestration and i'm evaluating Sidequest.

This app scrapes medical records from a health provider website

It access insurer website to retrieve information about the medical records

Than, saves back the information to provider website

It started with only one insurer, now it has four insurers and the architecture is showing its limit since is doing the work, one insurer after the other.
For each insurer: read in provider -> read in insurer -> save in provider

I need to change to the following architecture:

1. read all insurers in provider site (one job)
2. read in each provider site (multiple jobs)
3. save all insurers in provider site (one job)

The jobs in step 2 must be executed after job 1 finished
The jobs in step 2 can be run in parallel
The job in step 3 must be executed after all jobs in step 2 finished

Is there a way to accomplish that with Sidequest?

As far as i know queue priority will not work because, even if i set job 3 to a queue with lower priority, as soon as a worker is available it will start the job

[blikblum]

Having a way to know when a job is finished is enough to my use case.

I can queue the read records job, on finished queue the read insurer jobs. When all finished i queue the save record job

[GiovaniGuizzo]

Hey @blikblum.

It's somewhat possible.

You can enqueue jobs from within other jobs, so what you can do is:

Start your job number 1
Then within job number 1, when it succeeds and before returning, call Sidequest.enqueue() to enqueue your multiple jobs from step number 2.
Here it's a bit trickier, because there is no easy way to know from within the parallel jobs if they all have completed.

You can, however, fetch the list of all jobs of a given type using Sidequest.job.list or fetch a specific one using Sidequest.job.get.

What I suggest:

Create a new job called ProviderJobChecker that receives an array of job IDs containing all of the parallel jobs IDs. This should be enqueued right after you enqueue all the parallel jobs in the first job. The return type of Sidequest.enqueue has the job ID, so you can create that array upon enqueueing them.

Inside this job, you check if all the jobs from the ID array are completed using Sidequest.job.get(jobId). If they are, only then you enqueue the third job. If there is at least one that is not completed, you can return this.snooze(delayInMs) from within ProviderJobChecker's run method to snooze it for a given amount of time - this will postpone its execution for retrying in the future, hopefully after all the parallel jobs have completed. If you keep snoozing it, it will keep being postponed.

Inside it, you can even do some other shenanigans to check if one of the parallel jobs has permanently failed and skip it in that case, or notify you by email or other ways about failures.

This is not ideal and a workflow would be better, but workflows are something we will implement only on Sidequest pro :)

[blikblum]

I was thinking in using node-cron to poll the jobs status more or less what your suggested approach does. At first look, an advantage of using node-cron is not running an extra worker thread

[GiovaniGuizzo]

I see. Well, the advantage of using Sidequest for this additional polling job would be easy tracking of status and all. Bear in mind that it will get into the job pool and Sidequest will respect your concurrency settings.