Skip to content

Commit 87333ed

Browse files
committed
Standardize how exporter return vals work
This commit makes several changes to standardize and simplify how exporters' return values (e.g. from `export_records` and `delete_records`) work and are handled in Celery tasks. These changes will break any code based on previous versions of exporters. 1. Exporter methods are now assumed to be run in parallel, always. Each no longer has access to a cumulative `vals` kwarg (which otherwise would theoretically contain a running accumulation of vals from previous chunks of the same batch). They can return whatever they like, and the Celery tasks keep a list of all returned values. The complete set of return values is then passed to the exporter's `final_callback` method at the end of a job. 2. Added a `compile_vals` method to Exporter classes, where you can add custom code for how the return values for a particular Exporter subclass need to be handled. Celery tasks used as callbacks (i.e. `delegate_batch` and `do_final_cleanup`) call that method on accumulated lists of return values in order to consolidate the return values as the job runs, to help better manage memory usage. The final compiled version is passed to the exporter's `final_callback` method. 3. Error callbacks defined in Celery tasks (via `link_error`) now try to collect the return values from a batch where one or more chunks raised errors. When a chunk raises an error, instead of passing the results to the error callback (as Celery does with a regular callback), it passes the task ID(s) for the task(s) that had errors. When this happens, we have to instantiate a celery.result.AsyncResult object for the batch that raised errors and compile the list of results manually. This ensures that non-fatal errors during processing will not lead to missing return values. (This was a bug introduced by the `improve-tasks` branch changes. Fortunately errors are exceedingly rare, and our production exporters do not (yet) rely on return values, mostly. This will change in a future update, though, so it's important to fix it now.)
1 parent 8e8cd28 commit 87333ed

5 files changed

Lines changed: 316 additions & 254 deletions

File tree

0 commit comments

Comments
 (0)