Taskqueues tips

I wrote a post listing a few tips to write a taskqueue while I was working at Drivy. You can find it here : https://drivy.engineering/taskqueues-tips/.

As a TLDR; this post suggests :

  • Writing re-entrant idempotent tasks that require the least number of arguments possible
  • Avoid class-level calls
  • Avoid mutable instance variables
  • Specialize your workers for more efficiency
  • Queues should be designed by speed and urgency, not semantically
  • Read carefully your message broker’s doc (Redis, RabbitMQ …) and monitor it
  • Also monitor your queues SLAs, your worker’s memory and exceptions, and your CRON
  • Check your DB connection pool

I also spoke about this in a meetup talk, you can find the slides here : https://adipasquale.github.io/taskqueues-slides-2015