Repository navigation
Improve accuracy of log2mexp near log(2). #139
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -262,7 +262,7 @@ This is also called the ["softplus"](https://en.wikipedia.org/wiki/Rectifier_(ne | |
| transformation, being a smooth approximation to `max(0,x)`. Its inverse is [`logexpm1`](@ref). | ||
|
|
||
| This is also called the ["softplus"](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) | ||
| transformation (in its default parametrization, see [`softplus`](@ref)), being a smooth approximation to `max(0,x)`. | ||
| transformation (in its default parametrization, see [`softplus`](@ref)), being a smooth approximation to `max(0,x)`. | ||
|
|
||
| See: | ||
| * Martin Maechler (2012) [“Accurately Computing log(1 − exp(− |a|))”](http://cran.r-project.org/web/packages/Rmpfr/vignettes/log1mexp-note.pdf) | ||
|
|
@@ -364,6 +364,27 @@ Return `log(2 - exp(x))` evaluated carefully. | |
| """ | ||
| log2mexp(x::Real) = log1p(-expm1(x)) | ||
|
|
||
| # Split log(2) into hi + lo, where hi = T(log 2) and lo is the rounding residual in T. | ||
| Base.@assume_effects :foldable function _log2_hilo(::Type{T}) where {T<:AbstractFloat} | ||
| setprecision(BigFloat, 4 * precision(T) + 64) do | ||
| l2 = log(BigFloat(2)) | ||
| hi = T(l2) | ||
| lo = T(l2 - BigFloat(hi)) | ||
| return hi, lo | ||
| end | ||
| end | ||
|
|
||
| function log2mexp(x::T) where {T<:Union{Float16,Float32,Float64}} | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Restricting this to |
||
| if x > T(0.5) | ||
| log2_hi, log2_lo = _log2_hilo(T) | ||
| d = (log2_hi - x) + log2_lo # log(2) - x, accurate to ~eps relative | ||
| d < 0 && return T(NaN) # x > log(2): outside the domain | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This changes the behaviour for x > log 2 from |
||
| return log(-2 * expm1(-d)) # d == 0 gives -Inf | ||
| else | ||
| return log1p(-expm1(x)) # generic method, fine away from log(2) | ||
| end | ||
| end | ||
|
|
||
| """ | ||
| $(SIGNATURES) | ||
|
|
||
|
|
@@ -391,8 +412,8 @@ _logexpm1(x::BigFloat) = x > 1 ? x + log1p(-exp(-x)) : log(expm1(x)) | |
| """ | ||
| $(SIGNATURES) | ||
|
|
||
| The generalized `softplus` function (Wiemann et al., 2024) takes an additional optional parameter `a` that control | ||
| the approximation error with respect to the linear spline. It defaults to `a=1.0`, in which case the softplus is | ||
| The generalized `softplus` function (Wiemann et al., 2024) takes an additional optional parameter `a` that control | ||
| the approximation error with respect to the linear spline. It defaults to `a=1.0`, in which case the softplus is | ||
| equivalent to [`log1pexp`](@ref). | ||
|
|
||
| See: | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -205,6 +205,11 @@ end | |
| @test @inferred(log2mexp(T(0))) isa T | ||
| @test iszero(log2mexp(T(0))) | ||
| @test log2mexp(-T(1)) ≈ log(2 - exp(-T(1))) | ||
| for m in 1:5 | ||
| x = log(T(2)) | ||
| x -= ldexp(eps(x), m) | ||
| @test log2mexp(x) ≈ T(log2mexp(BigFloat(x))) | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| end | ||
| end | ||
| end | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
:foldableasserts:consistent,:effect_freeand:terminates, but this allocates BigFloats with finalizers and calls MPFR undersetprecision. If const-eval doesn't happen, this runs on every call. If the assumption is wrong, it's UB. Hard-coded constants (like_log1pexp_thresholds_*) avoid all of this. Also:log2in the name suggests the base-2 logarithm; maybe_logtwo_*? And 2p + a few guard bits is enough, rather than 4p + 64.