Shameless plug: Our team at Close has been inspired by RQ when we created TaskTiger – https://github.com/closeio/tasktiger. We’ve been running it in production for a few years now, processing ~4M tasks a day and it’s been a wonderful tool to work with. Curious to hear what y’all think!
Curious. What's the ROI on developing a task queue system for 50 req/sec?
Please don't take my question negatively. I acknowledge that not everything is about ROI.
For example I often let my team scratch their own itches by developing tools just for the sake of having fun and learning new toys. Perhaps this is the case with TaskTiger?
It was definitely fun to work on it :) That said, I think we've also seen quite a big positive ROI as well.
It is true that we've spent quite some time working on TaskTiger where we could've used an off-the-shelf task queue system. However, since we've created it, it allowed us to develop new features a lot faster and spend a lot less time debugging issues.
We were using Celery before and it was a big pain with memory leaks, periodic tasks not working as expected, some tasks being lost for no apparent reason, and poor visibility into both aggregate statistics and individual tasks. We spent too much time debugging Celery's issues instead of working on our core business. Here's some more context: https://github.com/closeio/tasktiger/issues/141#issuecomment...
Advanced features like scheduled tasks, unique queues, and task locks have helped us move faster, too.
We tried Tasktiger but ran into zombie tasks and some small but annoying bugs while testing. Around that same time, we found Dramatiq and currently have Dramatiq + Celery<4.0 in use.
Interesting, we haven't had issues with zombie tasks at all (and we DID have issues with them when using Celery). Did you manage to find out what was causing them?
> and some small but annoying bugs while testing
Do you recall any details?
Appreciate you trying TaskTiger, even if you've moved on since!
* RabbitMQ scales messages without needing tons of RAM
* I don't have to decide between not persisting messages to disk with Redis vs only using half the machine's RAM [1]
* Queue and task visibility is better in RabbitMQ
* Support for purging all tasks in a queue
* Tasktiger had lower throughput than Celery and Dramatiq, maybe needs lazy-forking?
Things I did like about Tasktiger:
* no feature bloat and it's possible to actually read it's source code[2].
[1]: To enable persisting to disk (Redis fork + save snapshotting) you must limit Redis to only using half the available RAM on a machine.
[2]: Celery is split up into multiple convoluted, bloated, difficult to read repos.
Both, but unfortunately still mostly using Celery 3.2. Never had any problems with zombie tasks, since that's related to how you're storing tasks and queues and RabbitMQ is excellent at that. Zombie tasks are never a problem when using RabbitMQ.
Especially never had problems with zombie workers. That wouldn't be a problem with your choice of queue library, more an infra problem.
Interesting. At the moment I'm noticing some zombie workers popping up here and there with RQ and was thinking of migrating soon to Celery or Dramatiq. I think I'll experiment a bit and decide between both (but Celery seems like the more popular option, so I'm a bit biased for going for that).
