Best way to queue a 1000 jobs?

[drewbaker]

I’m curious what you’d recommend as the best way to queue a 1000 jobs on a user action. Would you await .build() and just loop each job? Use Promise.all() in batches? Or is there some way to queue a lot of jobs together in a single function call to Sidequest?

I’ve found that if I loop through a big array of jobs and await a build() in the loop that some jobs (like 10 out of 1000) won’t be placed in the queue with no failure or error thrown.

My use case is something like an aggregated search… user enters one thing and we go off and search a 1000 different sources that may take 5s to 30s to each respond.

[GiovaniGuizzo]

some jobs (like 10 out of 1000) won’t be placed in the queue

This is a bit concerning. Sidequest would throw if it cannot insert a job in the DB. I suspect that there is something going on.

I have done await in a loop with hundreds of thousands of jobs already. It takes some time, but it works well. For 1000 jobs, maybe you can try no await and then await the promise array.

Test it and check what's best :)