Commit 87333ed
committed
Standardize how exporter return vals work
This commit makes several changes to standardize and simplify how
exporters' return values (e.g. from `export_records` and
`delete_records`) work and are handled in Celery tasks. These changes
will break any code based on previous versions of exporters.
1. Exporter methods are now assumed to be run in parallel, always. Each
no longer has access to a cumulative `vals` kwarg (which otherwise
would theoretically contain a running accumulation of vals from previous
chunks of the same batch). They can return whatever they like, and the
Celery tasks keep a list of all returned values. The complete set of
return values is then passed to the exporter's `final_callback` method
at the end of a job.
2. Added a `compile_vals` method to Exporter classes, where you can add
custom code for how the return values for a particular Exporter subclass
need to be handled. Celery tasks used as callbacks (i.e.
`delegate_batch` and `do_final_cleanup`) call that method on accumulated
lists of return values in order to consolidate the return values as the
job runs, to help better manage memory usage. The final compiled version
is passed to the exporter's `final_callback` method.
3. Error callbacks defined in Celery tasks (via `link_error`) now try to
collect the return values from a batch where one or more chunks raised
errors. When a chunk raises an error, instead of passing the results to
the error callback (as Celery does with a regular callback), it passes
the task ID(s) for the task(s) that had errors. When this happens, we
have to instantiate a celery.result.AsyncResult object for the batch
that raised errors and compile the list of results manually. This
ensures that non-fatal errors during processing will not lead to missing
return values. (This was a bug introduced by the `improve-tasks` branch
changes. Fortunately errors are exceedingly rare, and our production
exporters do not (yet) rely on return values, mostly. This will change
in a future update, though, so it's important to fix it now.)1 parent 8e8cd28 commit 87333ed
5 files changed
Lines changed: 316 additions & 254 deletions
File tree
- django/sierra
- export
- shelflist
0 commit comments