dbstorage/dbapp/CELERY_SETUP.md

Celery Setup and Testing Instructions

Prerequisites

Make sure you have Redis running (it's already configured in your docker-compose.yaml):

# Start Redis and other services
cd /home/vesemir/DataStorage
docker-compose up -d redis

Installing Dependencies

pip install -r requirements.txt

Database Setup

Since we're using django-celery-results and celery-beat, you need to run migrations:

python manage.py migrate

This will create the necessary tables for storing Celery results and managing periodic tasks.

Running Celery

1. Start Celery Worker

# From the dbapp directory
cd /home/vesemir/DataStorage/dbapp

# Run with development settings
python -m celery -A dbapp worker --loglevel=info

# Or with environment variable
DJANGO_SETTINGS_MODULE=dbapp.settings.development celery -A dbapp worker --loglevel=info

2. Start Celery Beat (for periodic tasks)

# From the dbapp directory
cd /home/vesemir/DataStorage/dbapp

# Run with development settings
python -m celery -A dbapp beat --loglevel=info

# Or with environment variable
DJANGO_SETTINGS_MODULE=dbapp.settings.development celery -A dbapp beat --loglevel=info

3. Start Flower (Optional - for monitoring)

# Install flower if not already installed
pip install flower

# Run flower to monitor tasks
celery -A dbapp flower

Testing Celery

Method 1: Using Django Shell

cd /home/vesemir/DataStorage/dbapp
python manage.py shell

# In the Django shell
from mainapp.tasks import test_celery_connection, add_numbers
from lyngsatapp.tasks import fill_lyngsat_data_task

# Test simple connection
result = test_celery_connection.delay("Test message!")
print(result.id)  # Task ID
print(result.get(timeout=10))  # Wait for result and print

# Test addition
result = add_numbers.delay(5, 7)
print(result.get(timeout=10))

# Check task state
print(result.state)  # Should be 'SUCCESS'
print(result.ready())  # Should be True
print(result.successful())  # Should be True

Method 2: Using Django Management Command

Create a management command to test:

mkdir -p dbapp/management/commands

Create /home/vesemir/DataStorage/dbapp/dbapp/management/commands/test_celery.py:

from django.core.management.base import BaseCommand
from mainapp.tasks import test_celery_connection, add_numbers


class Command(BaseCommand):
    help = 'Test Celery functionality'

    def handle(self, *args, **options):
        self.stdout.write('Testing Celery connection...')
        
        # Test simple task
        result = test_celery_connection.delay("Hello from test command!")
        self.stdout.write(f'Task ID: {result.id}')
        
        # Wait for result
        task_result = result.get(timeout=10)
        self.stdout.write(self.style.SUCCESS(f'Task result: {task_result}'))
        
        # Test math task
        math_result = add_numbers.delay(10, 20)
        sum_result = math_result.get(timeout=10)
        self.stdout.write(self.style.SUCCESS(f'10 + 20 = {sum_result}'))
        
        self.stdout.write(self.style.SUCCESS('All tests passed!'))

Then run:

python manage.py test_celery

Troubleshooting

Common Issues

1. Connection Error with Redis: Make sure Redis is running

   docker-compose up -d redis
   

2. Module Not Found Errors: Ensure all dependencies are installed

   pip install -r requirements.txt
   

3. Settings Module Error: Make sure DJANGO_SETTINGS_MODULE is set properly

   export DJANGO_SETTINGS_MODULE=dbapp.settings.development
   

4. Database Tables Missing: Run migrations

   python manage.py migrate
   

Debugging

Check if Celery can connect to Redis:

# Test Redis connection
redis-cli ping

Check Celery configuration:

# In Django shell
from django.conf import settings
print(settings.CELERY_BROKER_URL)
print(settings.CELERY_RESULT_BACKEND)

Environment Variables

Make sure your .env file contains:

CELERY_BROKER_URL=redis://localhost:6379/0
DJANGO_SETTINGS_MODULE=dbapp.settings.development

Running in Production

For production, ensure you have:

1. A production Redis instance
2. Proper security settings
3. Daemonized Celery workers

Example systemd service file for Celery worker (save as /etc/systemd/system/celery.service):

[Unit]
Description=Celery Service
After=network.target

[Service]
Type=forking
User=www-data
Group=www-data
EnvironmentFile=/path/to/your/.env
WorkingDirectory=/home/vesemir/DataStorage/dbapp
ExecStart=/path/to/your/venv/bin/celery -A dbapp worker --loglevel=info --pidfile=/var/run/celery/worker.pid --logfile=/var/log/celery/worker.log
ExecReload=/bin/kill -HUP $MAINPID
KillSignal=SIGTERM
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target