Django models not being saved to database when updated using Celery and when deployed to Heroku-CodePudding

So the title is a bit long winded but i think explains my issue pretty well.

I have a Django project that updates models every 60 seconds by scraping currency data from another website using Celery and RabbitMQ. When i run the celery worker locally it works perfectly, however when i deploy to Heroku the models do not update.

When i check the logs (heroku logs --tail) It shows that the tasks are running but they aren't updating the database models. I beleive this must be due to some configuration error in my settings.py file but i have tried numerous solutions and nothing has worked. I have also tried changing to use Redis instead and have experienced the same problem (it runs fine in the logs but does not actually update the database).

Here is what i believe to be the relevant code that could be causing the problem.

Settings.py (when using rabbitMQ):

CELERY_BROKER_URL = 'rabbitMQ_url_given_from_heroku'
BROKER_URL = os.environ.get("CELERY_BROKER_URL", "django://")
BROKER_POOL_LIMIT = 1
BROKER_CONNECTION_MAX_RETRIES = None

CELERY_TASK_SERIALIZER = "json"
CELERY_ACCEPT_CONTENT = ["json", "msgpack"]

Settings.py (when using Redis):

CELERY_BROKER_URL = 'Redis_url_given_from_heroku'
CELERRY_RESULT_BACKEND = 'django-db'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'UTC'

celery.py

import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'cryptoApi.settings')

app = Celery('cryptoApi')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

Procfile

web: gunicorn cryptoApi.wsgi --log-file -
worker: celery -A cryptoApi worker -l info

Any input would be greatly appreciated

CodePudding user response：

I suspect you're using SQLite as your database. That won't work as expected on Heroku.

However, most questions about this present differently: they look like they're working but then data is lost some time later, when new code is deployed or the next day after a dyno restart.

The reason you're not seeing new data at all is that workers run on dedicated dynos, separate from your web dynos, and each dyno will have its own SQLite database. Whatever data you save to one won't be visible on the others.

You'll have to switch to a client-server data store. Heroku's own Postgres service is usually a good starting point.