Pre-note: This is an old article from when I headed Isoscribe. I’m reposting it here, because I thought it was somewhat cool.
So, let me walk you through a problem
We have various jobs that need to run every so often. For example, we need to
generate a sitemap. Let’s walk through
how we would do this, shall we?
- We could have a dedicated server generate sitemaps for us.
- We could have each individual server generate sitemaps at random or specified times
- We could have an API route that generates a sitemap
- We could create an auto negotiating cron system
Let’s analyze these options, shall we?
For the first one. It is arguably the easiest to implement and manage. However, what if that dedicated server goes down? Then we won’t get sitemaps. Or, that single server will be very expensive for a minuscule task.
How about the second? Well, that would mean a lot of redundancy of tasks, and it could result in us paying for more servers than we need.
The third? Well, that runs into the problem of the first. How do we call that API route? When do we call it? What calls it?
Which brings us to the fourth. Arguably the most complex, however, very possibly the most rewarding. And that’s how Isoscribe runs internal tasks.
But how does it really work?
When Isoscribe wants to generate a sitemap, we go through a few steps. First, in our database, we have a list of cron “tasks,” each with an associated ID, timing, and data on the last time to complete the task.
Each Isoscribe server, when it starts, generates a unique ID. A background thread will look through the crontab, and figure out which ones need to be run. If there are open tasks, it rapidly inserts its own ID into the crontab, effectively “claiming” the task for itself. It then checks the field a second time, to ensure it has not been overwritten. If it still “owns” the task, it will begin running. When the task completes, it will mark it as “completed” in the database, along with a timestamp for when it was completed.
These cronjobs are distributed, and exist across all of our servers, running 24/7. Why run this way? It means that our infrastructure can have even more resilience than we had before, and no single piece can go down to take all of our servers offline.
Wow, that’s got to be some complicated code, right?
Actually, not really. It ends up being a single insert query, a couple ifs, a datetime comparison, and a get query. It’s not that bad - however, we did have to use a fancy to make this work.
Hope you enjoyed the little write up on our distributed crontab.
– E