We need a standard way to shard large data (e.g., medicare enrollment). Right now I do it using rg as in https://github.com/NSAPH/data_requests/blob/master/request_projects/exp_covar_health_merge_2016_april2019/code/1_prep_health_data_to_fst.R for example. Not sure if this approach can be generalized or if we need to develop something else, but we badly need better tooling for this.
We need a standard way to shard large data (e.g., medicare enrollment). Right now I do it using
rgas in https://github.com/NSAPH/data_requests/blob/master/request_projects/exp_covar_health_merge_2016_april2019/code/1_prep_health_data_to_fst.R for example. Not sure if this approach can be generalized or if we need to develop something else, but we badly need better tooling for this.