tj_lavin_delayed_jobs_example

This repo is to show an example Lucky app using the tj_lavin shard.

Delayed job issues

The problem I'm running into is that with jobs using the x-delay header, the job is not always coming back in the appropriate amount of time.

My initial assumption was the by using the dead-letter queue, the job would get rescheduled with no delay and the highest priority once it was ready. With this repo, you can see that MyWorker is schedule to run after 5 seconds.

Now in local development, this works just fine, but in production, that time looks to be variable. I have multiple machines running multiple workers and they are able to keep up with all of the jobs, so I don't think it's a resource issue.

I've been testing by using the management UI and publishing jobs directly to each exchange. The way the tj_lavin shard works is to create one main queue and the delayed queue following this documentation. When I publish a message into the main/direct queue, the job excecutes immediately. When I publish to the delayed queue, in production, the job eventually comes in. It's never quite the delay I set with x-delay.

The example I'm trying to test is, we enqueue a job that needs to run 45 seconds later after a given event happens. Watching the logs, we've observed that the job runs anywhere up to several minutes later. This is with using the management UI and code that is running in our production setup.

Here is an example of what I'm putting in the UI to publish a message to the delayed exchange.

Points of interest in the repo relating to the tj_lavin setup.

• https://github.com/russ/tj_lavin_delayed_jobs_example/blob/master/src/worker.cr
• https://github.com/russ/tj_lavin_delayed_jobs_example/blob/master/src/workers/my_worker.cr
• https://github.com/russ/tj_lavin_delayed_jobs_example/blob/master/src/actions/home/schedule_job.cr

Running the example

Start the docker containers:

docker compose up

Then schedule a job:

curl http://localhost:3000/schedule_job