Can celery celerybeat use a Database Scheduler without Django?

Question

I have a small infrastructure plan that does not include Django. But, because of my experience with Django, I really like Celery. All I really need is Redis + Celery to make my project. Instead of using the local filesystem, I'd like to keep everything in Redis. My current architecture uses Redis for everything until it is ready to dump the results to AWS S3. Admittedly I don't have a great reason for using Redis instead of the filesystem. I've just invested so much into architecting this with Docker and scalability in mind, it feels wrong not to.

score 7 · Answer 1 · answered Feb 27 '15 at 17:09

7

I was searching for a non-Django database scheduler too a while back, but it looked like there's nothing else. So I took the Django scheduler code and modified it to use SQLAlchemy. Should be even easier to make it use Redis instead.

answered Feb 27 '15 at 17:09

tuomur

6,888
34
37

Is your solution general enough that it might be useful to others? It looks like I'm gonna have to do the same thing for our Pyramids/SQLAlchemy system. – Chris W. Oct 21 '15 at 21:49
4

Pushed to [github](https://github.com/tuomur/celery_sqlalchemy_scheduler). Hope it helps :) – tuomur Oct 22 '15 at 06:29
wow thanks! even if it doesn't help, I appreciate your response! – Chris W. Oct 23 '15 at 17:08
I did figure it out. Thanks for taking the time to share this. It inspired me to come back and share my own findings in an answer. I see we have a few votes so it's nice to know it helped the community at large. – Bruno Bronosky Oct 05 '16 at 19:26

score -1 · Answer 2 · answered Oct 05 '16 at 19:23

It turns out that you can!

First I created this little project from the tutorial on celeryproject.org.

That went great so I built a Dockerized demo as a proof of concept.

Things I learned from this project

Docker
- using --link to create network connections between containers
- running commands inside containers
Dockerfile
- using FROM to build images iteratively
- using official images
- using CMD for images that "just work"
Celery
- using Celery without Django
- using Celerybeat without Django
- using Redis as a queue broker
- project layout
- task naming requirements
Python
- proper project layout for setuptools/setup.py
- installation of project via pip
- using entry_points to make console_scripts accessible
- using setuid and setgid to de-escalate privileges for the celery deamon

Can celery celerybeat use a Database Scheduler without Django?

2 Answers2

It turns out that you can!

Things I learned from this project

Linked