Currently the catalog table, when being distributed, sends one file per HBee. If the target files are small, it would be more efficient to process multiple files in a single cloud function.
- Verify that the parquet table works correctly with multiple files (can processing be done in parallel? or is their an other way to schedule downloads that minimizes wait time?)
- Implement the splitting login according to file size
- Find sensible file size thresholds to govern groupings
Currently the catalog table, when being distributed, sends one file per HBee. If the target files are small, it would be more efficient to process multiple files in a single cloud function.