添加链接
link管理
链接快照平台
  • 输入网页链接,自动生成快照
  • 标签化管理网页链接

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement . We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When using django-celery-results with the Postgres backend, it's possible to reach the maximum integer value for the primary key column, and this causes every task to fail.

We're using Python 3.9.7, Django 2.2.28, Django Celery Results 2.2.0, Postgres 12.

Traceback:

TaskResult.DoesNotExist: TaskResult matching query does not exist.
  File "django/db/models/query.py", line 538, in get_or_create
    return self.get(**kwargs), False
  File "django/db/models/query.py", line 406, in get
    raise self.model.DoesNotExist(
SequenceGeneratorLimitExceeded: nextval: reached maximum value of sequence "django_celery_results_taskresult_id_seq" (2147483647)
  File "django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
DataError: nextval: reached maximum value of sequence "django_celery_results_taskresult_id_seq" (2147483647)
  File "billiard/pool.py", line 1796, in safe_apply_callback
    fun(*args, **kwargs)
  File "celery/worker/request.py", line 571, in on_failure
    self.task.backend.mark_as_failure(
  File "celery/backends/base.py", line 171, in mark_as_failure
    self.store_result(task_id, exc, state,
  File "celery/backends/base.py", line 482, in store_result
    self._store_result(task_id, result, state, traceback,
  File "django_celery_results/backends/database.py", line 66, in _store_result
    self.TaskModel._default_manager.store_result(
  File "django_celery_results/managers.py", line 46, in _inner
    return fun(*args, **kwargs)
  File "django_celery_results/managers.py", line 168, in store_result
    obj, created = self.using(using).get_or_create(task_id=task_id,
  File "django/db/models/query.py", line 541, in get_or_create
    return self._create_object_from_params(kwargs, params)
  File "django/db/models/query.py", line 575, in _create_object_from_params
    obj = self.create(**params)
  File "django/db/models/query.py", line 422, in create
    obj.save(force_insert=True, using=self.db)
  File "django/db/models/base.py", line 743, in save
    self.save_base(using=using, force_insert=force_insert,
  File "django/db/models/base.py", line 780, in save_base
    updated = self._save_table(
  File "django/db/models/base.py", line 873, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "django/db/models/base.py", line 910, in _do_insert
    return manager._insert([self], fields=fields, return_id=update_pk,
  File "django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "django/db/models/query.py", line 1186, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "cachalot/monkey_patch.py", line 37, in inner
    return original(compiler, *args, **kwargs)
  File "cachalot/monkey_patch.py", line 113, in inner
    return original(write_compiler, *args, **kwargs)
  File "django/db/models/sql/compiler.py", line 1377, in execute_sql
    cursor.execute(sql, params)
  File "cachalot/monkey_patch.py", line 137, in inner
    return original(cursor, sql, *args, **kwargs)
  File "django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)

The fix was to manually ALTER SEQUENCE and ALTER COLUMN to bigint. After this, tasks continued to work as expected.

Although our problem is now solved, I'm not sure if there was a way to prevent this, or at least handle the issue in a better way.

I've changed the default primary keys to BigAutoField in #426

It feels like the primary keys of these models shouldn't be integers, but rather some sort of UUID. That seems like a much difficult change to make as a third party package.

I don't know what mechanisms Django migrations gives that can be applied here. Doing a data migration on tables with millions of rows might take a while to run, so that seems out of question...