Celery

6 Notes
+ Django Celery with django-celery-results extension (Nov. 11, 2016, 9:07 a.m.)

pip install celery pip install django_celery_results pip install django_celery_beat ------------------------------------------------ # project/project/celery.py from __future__ import absolute_import, unicode_literals import os from celery import Celery os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings') app = Celery('project') app.config_from_object('django.conf:settings', namespace='CELERY') app.autodiscover_tasks() @app.task(bind=True) def debug_task(self): print('Request: {0!r}'.format(self.request)) ------------------------------------------------ # project/project/__init__.py from __future__ import absolute_import, unicode_literals from .celery import app as celery_app __all__ = ['celery_app'] ------------------------------------------------ project/project/tasks.py from __future__ import absolute_import from celery import shared_task @shared_task def begin_ping(): return 'hi' ------------------------------------------------ # settings.py INSTALLED_APPS = ( 'celery', 'django_celery_results', 'django_celery_beat', ) CELERY_RESULT_BACKEND = 'django-db' ------------------------------------------------ python manage.py migrate django_celery_results python manage.py migrate django_celery_beat ------------------------------------------------ apt install rabbitmq-server For running it: rabbitmq-server ------------------------------------------------ Run these two commands in separated activated virtualenvs: celery -A project beat -l info -S django celery -A project worker -l info The "celery -A project beat -l info -S django" is for "DatabaseScheduler" which gets the schedules from Django admin panel. You can use "celery -A project beat -l info" which is for "PersistentScheduler" which gets the schedules from scripts in the tasks. For having the schedules from Admin panel, refer to the link "Intervals" and define a suitable interval. Then follow the link "Periodic tasks" and select the defined interval in the "Interval" dropdown list. ------------------------------------------------

+ Celery and RabbitMQ with Django (Oct. 14, 2018, 9:24 a.m.)

1- pip install Celery -------------------------------------------------------------- 2- apt-get install rabbitmq-server -------------------------------------------------------------- 3- Enable and start the RabbitMQ service systemctl enable rabbitmq-server systemctl start rabbitmq-server -------------------------------------------------------------- 4- Add configuration to the settings.py file: CELERY_BROKER_URL = 'amqp://localhost' -------------------------------------------------------------- 5- Create a new file named celery.py in your app: import os from celery import Celery os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mysite.settings') app = Celery('mysite') app.config_from_object('django.conf:settings', namespace='CELERY') app.autodiscover_tasks() -------------------------------------------------------------- 6- Edit the __init__.py file in the project root: from .celery import app as celery_app __all__ = ['celery_app'] -------------------------------------------------------------- 7- Create a file named tasks.py inside a Django app: from celery import shared_task @shared_task def my_task(x, y): return x, y -------------------------------------------------------------- 8- In views.py from .tasks import my_task my_task.delay(x, y) Instead of calling the "my_task" directly, we call my_task.delay(). This way we are instructing Celery to execute this function in the background. -------------------------------------------------------------- 9- Starting The Worker Process: Open a new terminal tab, and run the following command: celery -A mysite worker -l info --------------------------------------------------------------

+ Periodic Tasks from tasks.py (Oct. 14, 2018, 9:54 a.m.)

import datetime from celery.task import periodic_task @periodic_task(run_every=datetime.timedelta(minutes=5)) def myfunc(): print 'periodic_task'

+ Periodic Tasks from settings.py (Oct. 14, 2018, 10:23 a.m.)

CELERYBEAT_SCHEDULE = { 'add-every-30-seconds': { 'task': 'tasks.add', 'schedule': timedelta(seconds=30), 'args': (16, 16) }, }

+ Running tasks in shell (Oct. 11, 2018, 10:19 a.m.)

celery -A project_name beat celery -A cdr worker -l info

+ Daemon Scripts (Sept. 29, 2015, 11:09 a.m.)

These scripts are needed when you want to run the worker as a daemon. The first is used for seeing the output of running tasks. For example, I had something printed in the console, from within the task, and I could see the output (the printed string) in this terminal. The second is for firing up / starting the tasks. 1- Create a file /etc/supervisor/conf.d/celeryd.conf with this content: [program:celery] ; Set full path to celery program if using virtualenv command=/home/mohsen/virtualenvs/django-1.7/bin/celery worker -A cdr --loglevel=INFO directory=/home/mohsen/websites/cdr/ user=nobody numprocs=1 stdout_logfile=/var/log/celery/worker.log stderr_logfile=/var/log/celery/worker.log autostart=true autorestart=true startsecs=10 ; Need to wait for currently executing tasks to finish at shutdown. Increase this if you have very long running tasks. stopwaitsecs = 600 ; When resorting to send SIGKILL to the program to terminate it send SIGKILL to its whole process group instead, taking care of its children as well. killasgroup=true ; if rabbitmq is supervised, set its priority higher ; so it starts first priority=998 -------------------------------------------------------------------------------------------- 2- Create a file /etc/supervisor/conf.d/celerybeat.conf with this content: [program:celerybeat] ; Set full path to celery program if using virtualenv command=/home/mohsen/virtualenvs/django-1.7/bin/celery beat -A cdr ; remove the -A myapp argument if you are not using an app instance directory=/home/mohsen/websites/cdr/ user=nobody numprocs=1 stdout_logfile=/var/log/celery/beat.log stderr_logfile=/var/log/celery/beat.log autostart=true autorestart=true startsecs=10 ; if rabbitmq is supervised, set its priority higher so it starts first priority=999