EPIC 2: speed up eDisGo runtime on large grids (#671)#680
Open
joda9 wants to merge 8 commits into
Open
Conversation
Computing an exact geopy geodesic to every candidate bus per charging park is O(parks x buses) of iterative Karney calls and dominated wall time on large grids (e.g. 32064); small grids were unaffected. Replace the per-bus loop with a vectorised equirectangular nearest-neighbour search (preserves NN ordering at grid-district scale) and compute the exact geodesic only for the winning bus, so the returned distance -- used as the connection line length -- stays exact. The dropped bus_target[dist] side-effect column was unused by all callers (components.py, topology.py, edisgo.py, generators_import.py). Verified numerically identical: 200 random grid-district-scale trials show 0 nearest-neighbour mismatches vs. the old method and winner distances matching the exact geodesic to <1e-9 km. Part of #671.
…674) Replace the O(n^2) per-building pd.concat loop in get_residential_electricity_profiles_per_building with a single sparse incidence-matrix product (profiles @ incidence) followed by a per-building factor scaling. The previous concat-in-loop copied the whole growing frame on every iteration and was the dominant cost on large grids with thousands of buildings. Building columns are sorted to match the previous groupby(sort=True) order and the trailing dropna(axis=1) preserves the drop-on-NaN-factor behaviour, so the returned profiles are unchanged apart from negligible floating-point reordering noise (~1e-18). Type-hint the adjusted function and add scipy as a dependency. Part of #671.
Build the upper power, upper energy and lower energy bands without the per-charging-process Python loop. Each process adds a constant value to a contiguous range of time steps; encoding every range as two point updates on a difference array lets all processes be applied at once with a few numpy.add.at calls, and a single cumsum per band reconstructs the values. This removes the iterrows loop that dominated runtime on large grids (~9x faster on a 300k-process / 2000-charging-point / 35040-step case) and avoids the per-row pandas label-slice assignments that could intermittently segfault in _setitem_with_indexer. Band values are unchanged apart from negligible floating-point reordering noise (~1e-13). Type-hint the method and document the algorithm in the docstring Notes section. Part of #671.
…g_demand (#675) Record the drawn charging park and charging point per (car_id, destination) and write them back to charging_processes_df in a single vectorised step after the loop, instead of once per car. The previous per-car write-back matched car_id and destination against the whole frame for every car -- O(cars x rows) and the dominant cost on large grids; collecting the assignment and applying it once is O(rows). The per-car nominal-capacity lookup is likewise replaced by a single groupby. The weighted random draws and the capacity bookkeeping are left untouched, so the resulting assignment is bit-identical (verified against the original loop on synthetic data). This intentionally preserves the pre-existing home-branch bug where capacity is booked to a stale charging_park_id; that is tracked separately in #679. Type-hint the function and document the approach in the docstring Notes section. Part of #671.
scipy was added as a direct dependency in setup.py but was missing from rtd_requirements.txt, eDisGo_env.yml and eDisGo_env_dev.yml, which also list the other direct dependencies. Add it there with the same constraint (scipy < 1.18.0). Part of #671.
In the dumb and reduced branches, collect each charging park's load series and add them to loads_active_power in a single add_component_time_series call after the loop, instead of one call per park. Each call concatenates onto the growing time-series frame, so the previous per-park add was O(parks^2) and dominated the runtime on grids with many charging parks. The residual branch already builds and adds its columns in one go and is left unchanged. harmonize_charging_processes_df does not read the load time series, so moving the write out of the loop is safe; the resulting time series are identical (concatenation of independent columns in the same order). Part of #671.
…_building (#678) Collect the per-grid CTS profiles in a list and concatenate them in a single pd.concat after the loop, instead of concatenating onto a growing frame on every iteration (O(grids^2)). None results (grids without a substation profile) are dropped explicitly, matching the previous behaviour where the per-iteration pd.concat silently ignored them. The returned profiles are unchanged. Profiling the rest of the set_time_series_active_power_predefined umbrella (get_cts_profiles_per_grid, the industrial and feed-in importers) showed no further concat-in-loop or per-row pattern; the remaining cost is dominated by the database queries and the explode/pivot of the imported series. Part of #671.
Follow-up to the EPIC 2 audit (#671): a codebase-wide sweep for super-linear accumulation and per-row Python work on pandas/graph objects. All changes are output-identical apart from negligible floating-point reordering noise where noted. - io/powermodels_io.py: replace the O(n^2) per-component lookup in (linear name scan + per-call frame rebuild) with a cached and hoist the loop-invariant generator-type masks out of - tools/pseudo_coordinates.py: rewrite the per-node searches in as single single-source traversals (+ deque) - flex_opt/check_tech_constraints.py: collect per-grid/per-station frames and concat once in , and - flex_opt/battery_storage_operation.py: replace the per-time-step in with a numpy array loop (sequential SoE recursion unchanged) and defer the per-storage concat to a single call - flex_opt/reinforce_measures.py: batch the per-station transformer additions/removals into single concat/drop operations after the loop - network/timeseries.py: vectorise the time-series setters ( -> broadcast multiply) - io/generators_import.py, io/heat_pump_import.py, io/storage_import.py: batch the per-component grid-integration loops (single concat + Index build) - tools/tools.py: vectorise ; replace the per-bus shortest-path loops in with single-source traversals (existing LV metric preserved pending #681) - tools/powermodels_io.py: pull the time-series frames to numpy arrays once in / - flex_opt/costs.py, edisgo.py, tools/spatial_complexity_reduction.py, tools/preprocess_pypsa_opf_structure.py: vectorise/defer the remaining growing-frame concats and per-bus shortest-path loops Verified output-identical with synthetic harnesses replaying old vs new logic (max abs diff 0.0; battery checked against the previous implementation) and benchmarked at realistic scale. Part of #671 closes #682
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Umbrella PR for EPIC 2 (#671) — speed up the eDisGo runtime on large grids.
The recurring root cause across the codebase was super-linear accumulation:
pd.concat/add_component_time_seriesinside per-element loops (O(n²)) andper-row pandas work in Python loops. Each sub-issue removes one such hotspot.
Unless noted, the outputs are unchanged (numerically identical, or identical up
to negligible floating-point reordering noise).
Changes per sub-issue
tools.geo.find_nearest_busnetwork.electromobility.get_flexibility_bandsiterrowsloop; bands built via difference arrays + a singlecumsum(vectorisednumpy.add.at)io.timeseries_import.get_residential_electricity_profiles_per_buildingpd.concatwith a single sparse matrix product (incidence × profiles); addsscipydependencyio.electromobility_import.distribute_private_charging_demandgroupbyflex_opt.charging_strategies.charging_strategydumb/reducedbranches (oneadd_component_time_seriesinstead of O(parks²) concat);residualalready batchedio.timeseries_import.get_cts_profiles_per_buildingbus_idloopNot in this PR
pm_optimize) runtime: Julia startup & solve (investigation) #676 (OPFpm_optimize) — investigation only. No algorithmic O(n²) bug;the runtime is the MISOCP solve plus Julia JIT cold-start. The real levers
(a PackageCompiler sysimage / persistent Julia worker, parallelism across
grids) are build/deployment infrastructure decisions, not an output-preserving
code change, and cannot be verified without a Gurobi-licensed run. Deferred.
distribute_private_charging_demandnested loops (proposed) #675 (thehomebranch ofdistribute_private_charging_demandbooks capacity to a stalecharging_park_id). It is intentionally preserved here to keep 2.4 — Speed updistribute_private_charging_demandnested loops (proposed) #675'soutput identical, and is filed separately as a
[BUG]issue to be fixed on itsown (the fix changes the resulting assignment).
Verification
old vs new logic; numerical identity or float-reorder noise on the order of
1e-13…1e-18).
test_geo,test_electromobility(flexibility bands),
test_timeseries_import(residential + CTS, against theOEP DB),
test_electromobility_import(distribute_charging_demand),test_charging_strategy.Checklist
pre-commithooks (ruff; oneunrelated pre-existing E501 in
electromobility_import.pyleft untouched)(new
find_nearest_bustests; the other refactors are output-preserving andcovered by existing tests + equivalence harnesses)
tests are environment-dependent and unaffected by these changes)
setup.py,rtd_requirements.txt,eDisGo_env.ymlandeDisGo_env_dev.yml(scipyfor 2.3 — Remove O(n²) concat in residential demand-profile assembly #674)Closes #672, #673, #674, #675, #677, #678. Part of #671.