dbstorage/dbapp/CELERY_SETUP.md

# Celery Setup and Testing Instructions

## Prerequisites

Make sure you have Redis running (it's already configured in your docker-compose.yaml):

```bash
# Start Redis and other services
cd /home/vesemir/DataStorage
docker-compose up -d redis
```

## Installing Dependencies

```bash
pip install -r requirements.txt
```

## Database Setup

Since we're using django-celery-results and celery-beat, you need to run migrations:

```bash
python manage.py migrate
```

This will create the necessary tables for storing Celery results and managing periodic tasks.

## Running Celery

### 1. Start Celery Worker

```bash
# From the dbapp directory
cd /home/vesemir/DataStorage/dbapp

# Run with development settings
python -m celery -A dbapp worker --loglevel=info

# Or with environment variable
DJANGO_SETTINGS_MODULE=dbapp.settings.development celery -A dbapp worker --loglevel=info
```

### 2. Start Celery Beat (for periodic tasks)

```bash
# From the dbapp directory
cd /home/vesemir/DataStorage/dbapp

# Run with development settings
python -m celery -A dbapp beat --loglevel=info

# Or with environment variable
DJANGO_SETTINGS_MODULE=dbapp.settings.development celery -A dbapp beat --loglevel=info
```

### 3. Start Flower (Optional - for monitoring)

```bash
# Install flower if not already installed
pip install flower

# Run flower to monitor tasks
celery -A dbapp flower
```

## Testing Celery

### Method 1: Using Django Shell

```bash
cd /home/vesemir/DataStorage/dbapp
python manage.py shell
```

```python
# In the Django shell
from mainapp.tasks import test_celery_connection, add_numbers
from lyngsatapp.tasks import fill_lyngsat_data_task

# Test simple connection
result = test_celery_connection.delay("Test message!")
print(result.id)  # Task ID
print(result.get(timeout=10))  # Wait for result and print

# Test addition
result = add_numbers.delay(5, 7)
print(result.get(timeout=10))

# Check task state
print(result.state)  # Should be 'SUCCESS'
print(result.ready())  # Should be True
print(result.successful())  # Should be True
```

### Method 2: Using Django Management Command

Create a management command to test:

```bash
mkdir -p dbapp/management/commands
```

Create `/home/vesemir/DataStorage/dbapp/dbapp/management/commands/test_celery.py`:

```python
from django.core.management.base import BaseCommand
from mainapp.tasks import test_celery_connection, add_numbers


class Command(BaseCommand):
    help = 'Test Celery functionality'

    def handle(self, *args, **options):
        self.stdout.write('Testing Celery connection...')
        
        # Test simple task
        result = test_celery_connection.delay("Hello from test command!")
        self.stdout.write(f'Task ID: {result.id}')
        
        # Wait for result
        task_result = result.get(timeout=10)
        self.stdout.write(self.style.SUCCESS(f'Task result: {task_result}'))
        
        # Test math task
        math_result = add_numbers.delay(10, 20)
        sum_result = math_result.get(timeout=10)
        self.stdout.write(self.style.SUCCESS(f'10 + 20 = {sum_result}'))
        
        self.stdout.write(self.style.SUCCESS('All tests passed!'))
```

Then run:
```bash
python manage.py test_celery
```

## Troubleshooting

### Common Issues

1. **Connection Error with Redis**: Make sure Redis is running
   ```bash 
   docker-compose up -d redis
   ```

2. **Module Not Found Errors**: Ensure all dependencies are installed
   ```bash
   pip install -r requirements.txt
   ```

3. **Settings Module Error**: Make sure DJANGO_SETTINGS_MODULE is set properly
   ```bash
   export DJANGO_SETTINGS_MODULE=dbapp.settings.development
   ```

4. **Database Tables Missing**: Run migrations
   ```bash
   python manage.py migrate
   ```

### Debugging

Check if Celery can connect to Redis:

```bash
# Test Redis connection
redis-cli ping
```

Check Celery configuration:
```python
# In Django shell
from django.conf import settings
print(settings.CELERY_BROKER_URL)
print(settings.CELERY_RESULT_BACKEND)
```

### Environment Variables

Make sure your `.env` file contains:

```
CELERY_BROKER_URL=redis://localhost:6379/0
DJANGO_SETTINGS_MODULE=dbapp.settings.development
```

## Running in Production

For production, ensure you have:

1. A production Redis instance
2. Proper security settings
3. Daemonized Celery workers

Example systemd service file for Celery worker (save as `/etc/systemd/system/celery.service`):

```
[Unit]
Description=Celery Service
After=network.target

[Service]
Type=forking
User=www-data
Group=www-data
EnvironmentFile=/path/to/your/.env
WorkingDirectory=/home/vesemir/DataStorage/dbapp
ExecStart=/path/to/your/venv/bin/celery -A dbapp worker --loglevel=info --pidfile=/var/run/celery/worker.pid --logfile=/var/log/celery/worker.log
ExecReload=/bin/kill -HUP $MAINPID
KillSignal=SIGTERM
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
```