Skip to content

fix(utils): use units::ud_convert directly for non-difftime conversions#3928

Open
anshul23102 wants to merge 4 commits intoPecanProject:developfrom
anshul23102:patch-1
Open

fix(utils): use units::ud_convert directly for non-difftime conversions#3928
anshul23102 wants to merge 4 commits intoPecanProject:developfrom
anshul23102:patch-1

Conversation

@anshul23102
Copy link
Copy Markdown
Contributor

@anshul23102 anshul23102 commented Apr 11, 2026

Description

For non-difftime inputs, replace the slow set_units/set_units/drop_units chain with a direct call to units::ud_convert. The difftime path is unchanged.

Closes #3927

Motivation and Context

units::ud_convert is ~10x faster than the original set_units chain for scalar inputs. An affine factor/offset approach was attempted for large-vector speedup but caused floating-point precision errors in datetime conversions and was reverted.

Benchmark (scalar, real usage pattern)

approach median
original (set_units chain) ~183µs
this PR (units::ud_convert) ~18µs

Types of changes

  • Bug fix / performance improvement (non-breaking)

Checklist

  • All new and existing tests passed
  • Added tests for vector inputs, NA passthrough, and integer input
  • I agree that PEcAn Project may distribute my contribution under BSD 3-clause license
  • I have read the CONTRIBUTING document

@github-actions github-actions bot added the base label Apr 11, 2026
@anshul23102 anshul23102 changed the title change it to: fix(utils): use units::ud_convert directly for non-difftime conversions fix(utils): use units::ud_convert directly for non-difftime conversions Apr 11, 2026
@infotroph
Copy link
Copy Markdown
Member

infotroph commented Apr 11, 2026

Thanks for the PR! Can you please tell us how you tested this and whether the speed difference in a real workflow was comparable to @dlebauer's benchmark result?

@anshul23102
Copy link
Copy Markdown
Contributor Author

Tested locally with bench::mark() (500 iterations):

expression min median itr/sec mem_alloc
old 162.89µs 168.31µs 5780 6.37MB
new 8.73µs 9.27µs 93252 0B

The new approach is ~18x faster with zero memory allocation, consistent with @dlebauer's benchmark in #3927. The difftime path is unchanged and all existing tests pass.

@infotroph
Copy link
Copy Markdown
Member

infotroph commented Apr 12, 2026

Well yes, repeating the same test case is going to give the same results. That's why I said "real workflow": Inside PEcAn ud_convert gets called in many different ways that differ greatly in datatype, units, vector length, and execution context; one test of a scalar input is promising but not indicative of the PEcAn-wide result. I'm particularly interested in whether this produces detectable speedups in the various model-specific met2model, write.config, and model2netcdf functions -- most models use ud_convert extensively for those.

Even before we get to testing on a PEcAn workflow, we can observe in benchmark results that the memory allocation difference is only seen the first time you call it in a session, and the headline "20x" number only holds for very simple and small inputs. Converting a vector of 10 numbers instead of a scalar cuts units's advantage from ~18x to ~6x, and with a vector of 1000 I'd argue the difference isn't even consistently detectable.

Details
> bench::mark(
+  pecan = PEcAn.utils::ud_convert(1, "g", "kg"),
+  units = units::ud_convert(1, "g", "kg"),
+  iterations=1000)
# A tibble: 2 × 13
  expression      min   median `itr/sec` mem_alloc `gc/sec` n_itr  n_gc total_time result    memory                 time               gc                  
  <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl> <int> <dbl>   <bch:tm> <list>    <list>                 <list>             <list>              
1 pecan      175.73µs  179.8µs     5461.    6.82MB     38.5   993     7   181.85ms <dbl [1]> <Rprofmem [3,940 × 3]> <bench_tm [1,000]> <tibble [1,000 × 3]>
2 units        8.61µs    9.1µs   107270.        0B      0    1000     0     9.32ms <dbl [1]> <Rprofmem [0 × 3]>     <bench_tm [1,000]> <tibble [1,000 × 3]>
> bench::mark(
+  pecan = PEcAn.utils::ud_convert(1, "g", "kg"),
+  units = units::ud_convert(1, "g", "kg"),
+  iterations=1000)
# A tibble: 2 × 13
  expression      min   median `itr/sec` mem_alloc `gc/sec` n_itr  n_gc total_time result    memory             time               gc                  
  <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl> <int> <dbl>   <bch:tm> <list>    <list>             <list>             <list>              
1 pecan      175.73µs 194.71µs     4872.        0B     39.3   992     8    203.6ms <dbl [1]> <Rprofmem [0 × 3]> <bench_tm [1,000]> <tibble [1,000 × 3]>
2 units        8.98µs   9.59µs   101483.        0B      0    1000     0     9.85ms <dbl [1]> <Rprofmem [0 × 3]> <bench_tm [1,000]> <tibble [1,000 × 3]>
> x <- rnorm(10)
> bench::mark(
+  pecan = PEcAn.utils::ud_convert(x, "g", "kg"),
+  units = units::ud_convert(x, "g", "kg"),
+  iterations=1000)
# A tibble: 2 × 13
  expression      min   median `itr/sec` mem_alloc `gc/sec` n_itr  n_gc total_time result     memory             time               gc                  
  <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl> <int> <dbl>   <bch:tm> <list>     <list>             <list>             <list>              
1 pecan         193µs  201.5µs     4385.        0B     48.8   989    11    225.5ms <dbl [10]> <Rprofmem [0 × 3]> <bench_tm [1,000]> <tibble [1,000 × 3]>
2 units          31µs   33.3µs    25921.        0B     25.9   999     1     38.5ms <dbl [10]> <Rprofmem [0 × 3]> <bench_tm [1,000]> <tibble [1,000 × 3]>
> bench::mark(
+  pecan = PEcAn.utils::ud_convert(x, "g", "kg"),
+  units = units::ud_convert(x, "g", "kg"),
+  iterations=1000)
# A tibble: 2 × 13
  expression      min   median `itr/sec` mem_alloc `gc/sec` n_itr  n_gc total_time result     memory             time               gc                  
  <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl> <int> <dbl>   <bch:tm> <list>     <list>             <list>             <list>              
1 pecan       196.3µs    213µs     3867.        0B     43.0   989    11      256ms <dbl [10]> <Rprofmem [0 × 3]> <bench_tm [1,000]> <tibble [1,000 × 3]>
2 units        30.8µs   33.4µs    25604.        0B     51.3   998     2       39ms <dbl [10]> <Rprofmem [0 × 3]> <bench_tm [1,000]> <tibble [1,000 × 3]>
> x <- rnorm(1e3)
> bench::mark(
+  pecan = PEcAn.utils::ud_convert(x, "g", "kg"),
+  units = units::ud_convert(x, "g", "kg"),
+  iterations=1000)
# A tibble: 2 × 13
  expression      min   median `itr/sec` mem_alloc `gc/sec` n_itr  n_gc total_time result        memory              time               gc                  
  <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl> <int> <dbl>   <bch:tm> <list>        <list>              <list>             <list>              
1 pecan        2.67ms   3.45ms      278.    31.7KB     42.0   869   131      3.12s <dbl [1,000]> <Rprofmem [36 × 3]> <bench_tm [1,000]> <tibble [1,000 × 3]>
2 units        2.52ms   3.44ms      296.    31.7KB     40.8   879   121      2.96s <dbl [1,000]> <Rprofmem [13 × 3]> <bench_tm [1,000]> <tibble [1,000 × 3]>
> bench::mark(pecan = PEcAn.utils::ud_convert(x, "g", "kg"), units = units::ud_convert(x, "g", "kg"), iterations=1000)
# A tibble: 2 × 13
  expression      min   median `itr/sec` mem_alloc `gc/sec` n_itr  n_gc total_time result        memory             time               gc                  
  <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl> <int> <dbl>   <bch:tm> <list>        <list>             <list>             <list>              
1 pecan        2.69ms   3.19ms      279.    31.7KB     41.8   870   130      3.11s <dbl [1,000]> <Rprofmem [5 × 3]> <bench_tm [1,000]> <tibble [1,000 × 3]>
2 units        2.49ms   3.36ms      296.    31.7KB     41.2   878   122      2.96s <dbl [1,000]> <Rprofmem [5 × 3]> <bench_tm [1,000]> <tibble [1,000 × 3]>

That said, we might still want to make this change! We use ud_convert so many places in PEcAn that even a context-dependent speedup could still pay off if it eases a bottleneck in an important place.

But the flip side is that because we rely so heavily on ud_convert in so many places, the bar is high for changes: We want to be VERY sure they won't break anything. I know we have some existing test coverage of ud_convert, but it's worth considering if there are any additional test cases needed to ensure this change didn't miss any corner cases.

@github-actions github-actions bot added the tests label Apr 12, 2026
anshul23102 and others added 2 commits April 13, 2026 02:19
…time conversions

add:
For non-difftime inputs, replace the set_units/set_units/drop_units chain with a direct call to units::ud_convert, which is ~20x faster per dlebauer's benchmark in PecanProject#3927. The difftime path is unchanged.
Addresses reviewer feedback: existing tests only covered scalar inputs.
Adds tests for numeric vectors (small and length-1000), NA passthrough,
and integer input to improve coverage of real PEcAn usage patterns.
Instead of calling units::ud_convert element-wise on the full vector,
compute the linear factor and offset once using two scalar calls, then
apply with base R arithmetic. This is ~750x faster for large vectors
(e.g. n=8760 hourly met data: 30ms -> 40us) while remaining correct
for all affine unit conversions that udunits supports.
@anshul23102 anshul23102 changed the title fix(utils): use units::ud_convert directly for non-difftime conversions fix(utils): use affine factor/offset for fast vectorized ud_convert Apr 12, 2026
…roach

The affine factor/offset approach caused floating-point precision errors
in datetime conversions (e.g. datetime2cf) where large offsets like
'seconds since 1970-01-01' cause catastrophic cancellation.
Direct units::ud_convert handles precision internally in C.
@anshul23102 anshul23102 changed the title fix(utils): use affine factor/offset for fast vectorized ud_convert fix(utils): use units::ud_convert directly for non-difftime conversions Apr 12, 2026
@anshul23102
Copy link
Copy Markdown
Contributor Author

I investigated the real call sites in met2model.SIPNET and met2model.ED2 and found ud_convert is called on full NetCDF arrays (e.g. hourly Tair, n=8760). I tried an affine factor/offset approach (compute scale+offset once,
apply with base R arithmetic) which benchmarked ~750x faster for large vectors - but it caused floating-point precision errors in datetime2cf due to catastrophic cancellation with large time offsets like "seconds since 1970-01-01". Reverted to units::ud_convert directly.

So the honest summary is: this PR gives a real speedup (~10x) for scalar calls but no meaningful gain for large vectors. The CI failure on R 4.2 (test.cf2date) appears pre-existing - the same test passes locally on R 4.5 with both the original and new code, and the failure is unrelated to our changes.

Happy to close this if the scalar-only speedup isn't worth it, or keep it open if you think it's still a net improvement.

Copy link
Copy Markdown

@Yeligay8 Yeligay8 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ud_convert() now correctly processes vector inputs instead of assuming scalar values.
Added test:
expect_equal(ud_convert(c(0, 10, 100), "degC", "K"), c(273.15, 283.15, 373.15))

Recursive Conversion Logic:
return(units::ud_convert(as.numeric(x), u1, u2))
This ensures consistent handling when units do not match u1.

Function now preserves NA values during conversion.
Added test:
expect_equal(result, c(100, NA, 300))

Support for Integer Inputs: Ensures integer vectors are handled correctly.
Added test:
expect_equal(ud_convert(1L, "m", "cm"), 100)

Explicit test ensures output length matches input:
expect_length(result, 1000)

Removed code from original code:
Direct Unit Assignment to x1
x1 <- units::set_units(as.numeric(x), value = u1, mode = "standard")

Replaced with recursive call to ud_convert() for cleaner handling.
Intermediate Variable Handling
Old approach:
x2 <- units::set_units(x1, value = u2, mode = "standard")
units::drop_units(x2)
Refactored into a more streamlined flow while keeping:
x2 <- units::set_units(x1, value = u2, mode = "standard")
units::drop_units(x2)
Removes redundant steps and improves readability.
put units

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

use units::ud_convert directly in PEcAn.utils::ud_convert?

3 participants