Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/channels.rst
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Measures

* ``mi`` is :math:`I[X : Y]` at the observed input distribution: the information
the edge carries as the source actually drives it. It uses the network's estimator
(Miller--Madow by default :cite:`Miller1955`), or
(Chao--Wang--Jost by default :cite:`Chao2013`), or
``ChannelSettings(estimator=...)``.
* ``capacity`` is :math:`C = \max_{p(x)} I[X : Y]` of the estimated channel,
computed by the Blahut--Arimoto algorithm :cite:`Blahut1972,Arimoto1972`
Expand Down
27 changes: 25 additions & 2 deletions docs/layers.rst
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,31 @@ sampling noise. :func:`infoflow.crossfit.crossfit_flows` therefore separates the
two in the style of double machine learning :cite:`Chernozhukov2018`: contexts are
merged (exactly when their predictive distributions coincide, statistically
otherwise, as CSSR merges histories :cite:`Shalizi2004`), the channel is fit on
one fold and evaluated with the Miller--Madow estimator :cite:`Miller1955` on the
other, and the folds are swapped and averaged.
one fold and evaluated on the other with the Chao--Wang--Jost entropy estimator
:cite:`Chao2013`, and the folds are swapped and averaged.

``infer_multiplex(estimator=...)`` (and :func:`~infoflow.crossfit.crossfit_flows`)
takes any of dit's discrete entropy estimators, applied to every entropy term of the
evaluation table (``'miller_madow'`` :cite:`Miller1955` was the default before
Chao--Wang--Jost). Except for the plug-in and Miller--Madow, the intrinsic flow is
evaluated at the channel hardened to its most likely output per context, since these
estimators need integer counts.
On the sofic processes with exact flows (``validation/dit_estimators.py``; copy, XOR,
shared, and mixed edges with up to 54 contexts, 20 replicates), the Chao--Wang--Jost
estimator :cite:`Chao2013` lowered the cross-fitted layer RMSE against Miller--Madow
from 0.047 to 0.035 bits at 300 samples and from 0.019 to 0.014 at 1000 (0.007 for
both at 3000), and the synergistic layer's bias at 54 contexts from 0.081 to 0.033
bits; NSB :cite:`Nemenman2002` was slightly more accurate still (0.030, 0.013) but
costs thousands of times more per entropy. On real iEEG with the source shuffled
(every true layer zero), Miller--Madow left 0.0010 bits of spurious synergy per edge,
Chao--Wang--Jost 0.0004 and NSB 0.0002. Shrinkage, Laplace, Bonachela and the
asymptotic NSB estimators were worse than Miller--Madow. Against Miller--Madow
layers, Chao--Wang--Jost raised the recall of significant intrinsic flow on random
VAR, logistic-map and Lorenz-96 networks at unchanged precision and left NetSim and
CausalRivers unchanged (``validation/new_defaults.py``; ``reports/dit_estimators.md``).
It costs about 13 times as much per entropy as Miller--Madow; whole networks took
about 1.5 times as long. :func:`~infoflow.compare.compare_networks` still evaluates
with Miller--Madow.

Statistics
----------
Expand Down
33 changes: 33 additions & 0 deletions docs/references.bib
Original file line number Diff line number Diff line change
Expand Up @@ -1070,3 +1070,36 @@ @article{Grigorescu2024
year = {2024},
doi = {10.1109/TIT.2024.3411919}
}

@article{Chao2013,
author = {Chao, Anne and Wang, Y. T. and Jost, Lou},
title = {Entropy and the species accumulation curve: a novel entropy estimator via discovery rates of new species},
journal = {Methods in Ecology and Evolution},
volume = {4},
number = {11},
pages = {1091--1100},
year = {2013},
doi = {10.1111/2041-210X.12108}
}

@inproceedings{Nemenman2002,
author = {Nemenman, Ilya and Shafee, Fariel and Bialek, William},
title = {{Entropy and inference, revisited}},
booktitle = {Advances in Neural Information Processing Systems 14},
pages = {471--478},
publisher = {MIT Press},
year = {2002},
eprint = {physics/0108025},
archivePrefix = {arXiv}
}

@article{Zhang2012,
author = {Zhang, Zhiyi},
title = {Entropy estimation in {T}uring's perspective},
journal = {Neural Computation},
volume = {24},
number = {5},
pages = {1368--1389},
year = {2012},
doi = {10.1162/NECO_a_00266}
}
12 changes: 12 additions & 0 deletions docs/skeleton.rst
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,18 @@ shuffles when there are trials, otherwise circular, block, or local permutations
Selection and estimation can use disjoint halves of the data (``holdout``) to
avoid post-selection bias in the layer estimates.

``SkeletonSettings(entropy_estimator=...)`` computes the plug-in estimator's
conditional mutual information with any of dit's discrete entropy estimators instead.
The permutation tests stay valid; only their power changes. On random VAR and
coupled-logistic networks (``validation/dit_estimators_selection.py``; 10 nodes, 500
and 2000 samples, five networks each, the same networks for every estimator),
Miller--Madow raised recall by 0.05 over the plug-in on both kinds (better on 7 of
10 networks, worse on none) at the same cost and without more false positives on
VAR; Chao--Wang--Jost :cite:`Chao2013` raised it most (0.06 on VAR, 0.13 on
logistic maps) at two to nine times the cost, with 0.4 more false positives per
logistic network and a lower edge-score AUROC; Zhang :cite:`Zhang2012` matched
Miller--Madow at two to eight times the cost.

For continuous data, ``SkeletonSettings(estimator="ksg")`` selects parents with the
KSG conditional mutual information on the raw values :cite:`Kraskov2004,Frenzel2007`,
as IDTxl does. Conditioning on a parent continuously removes its influence where bins
Expand Down
9 changes: 6 additions & 3 deletions infoflow/auto.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
from .checkpoint import Checkpoint
from .data import DiscreteData, as_trials
from .embedding import Embedding
from .measures import DEFAULT_LAYER_ESTIMATOR
from .network import estimate_edge, infer_multiplex
from .parallel import thread_map
from .selection import SkeletonSettings, select_parents
Expand Down Expand Up @@ -512,7 +513,7 @@ def _ksg_seconds(data, embeddings, target, settings, cheap):
return per_estimate * evaluations


def _edge_seconds(data, skeletons, n_resamples):
def _edge_seconds(data, skeletons, n_resamples, estimator):
"""
Measured seconds per edge of layer estimation at `n_resamples`, from one pilot parent.
"""
Expand All @@ -522,7 +523,7 @@ def _edge_seconds(data, skeletons, n_resamples):
w = base + [v for v in sk.sources if v[0] != p]
trial = 50
start = time.perf_counter()
estimate_edge(data, sk.target, sk.source_variables(p), w, "miller_madow", trial, trial, prng=0)
estimate_edge(data, sk.target, sk.source_variables(p), w, estimator, trial, trial, prng=0)
return (time.perf_counter() - start) * n_resamples / trial
return 0.0

Expand Down Expand Up @@ -719,7 +720,9 @@ def pilot_run(t):
"prescreen": cheap["prescreen"],
"parents_per_target": parents,
"shared_per_target": shared,
"edge_seconds": _edge_seconds(discrete, pilot_skeletons, n_resamples),
"edge_seconds": _edge_seconds(
discrete, pilot_skeletons, n_resamples, pinned_infer.get("estimator", DEFAULT_LAYER_ESTIMATOR)
),
}
# Preprocessing, the linearity test, and the pilot are already spent.
elapsed = time.perf_counter() - started
Expand Down
2 changes: 1 addition & 1 deletion infoflow/benchmarks.py
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ def _estimate(x, y, z, estimator, kind, bins, k, rng):

data = np.column_stack([x, y, z]).astype(float)
crvs = list(range(2, data.shape[1])) or None
return float(total_correlation_ksg(data, [[0], [1]], crvs, k=k, prng=rng))
return float(total_correlation_ksg(data, [[0], [1]], crvs, k=k, rng=rng))
if kind == "gaussian":
x, y, z = _bin(x, bins), _bin(y, bins), _bin(z, bins) if z.shape[1] else z
yc, _ = dense_codes(y)
Expand Down
6 changes: 3 additions & 3 deletions infoflow/channels.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
import xarray as xr

from .machines import _aligned, _as_data
from .measures import cmi_from_joint
from .measures import DEFAULT_LAYER_ESTIMATOR, cmi_from_joint

__all__ = ("ChannelLayer", "ChannelSettings", "edge_channel", "infer_channels")

Expand All @@ -38,7 +38,7 @@ class ChannelSettings:

Attributes
----------
estimator : {'miller_madow', 'plugin'}, None
estimator : str, None
Estimator of the mutual information (None: the network's). The capacity is
always the plug-in capacity of the estimated channel.
include_shared_candidates : bool
Expand Down Expand Up @@ -151,7 +151,7 @@ def infer_channels(data, network, settings=None):
settings = ChannelSettings()
elif isinstance(settings, dict):
settings = ChannelSettings(**settings)
estimator = settings.estimator or network.settings.get("estimator", "miller_madow")
estimator = settings.estimator or network.settings.get("estimator", DEFAULT_LAYER_ESTIMATOR)
names = network.names
data = _as_data(data, names)
P = data.n_processes
Expand Down
12 changes: 7 additions & 5 deletions infoflow/crossfit.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,11 @@

from .context import apply_merge, merge_contexts_statistical
from .measures import (
DEFAULT_LAYER_ESTIMATOR,
EdgeFlow,
_degrade,
cmi_from_joint,
dense_codes,
evaluation_table,
intrinsic_flow,
joint_table,
layers_from_values,
Expand All @@ -42,15 +43,15 @@ def _evaluate(table_b, mapping, Q, estimator):
n = merged_b.sum()
te = cmi_from_joint(table_b, estimator)
tdmi = cmi_from_joint(table_b.sum(axis=2, keepdims=True), estimator)
iif = cmi_from_joint(_degrade(merged_b, Q), estimator, n=n)
iif = cmi_from_joint(evaluation_table(merged_b, Q, estimator), estimator, n=n)
return te, tdmi, min(iif, te, tdmi)


def crossfit_flows(
y,
x,
w,
estimator="miller_madow",
estimator=DEFAULT_LAYER_ESTIMATOR,
merge=True,
merge_alpha=0.01,
gap=0,
Expand All @@ -66,8 +67,9 @@ def crossfit_flows(
y, x, w : array_like
The target present, source past, and context of each aligned realization
(1D codes or 2D rows of joint values); `w` may be None.
estimator : {'miller_madow', 'plugin'}
The evaluation estimator on the held-out fold.
estimator : str
The evaluation estimator on the held-out fold (see
:func:`~infoflow.measures.cmi_from_joint`).
merge : bool
Merge statistically indistinguishable context values before fitting.
merge_alpha : float
Expand Down
16 changes: 13 additions & 3 deletions infoflow/layers.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,17 @@

from .context import apply_merge, merge_contexts_statistical
from .crossfit import crossfit_flows
from .measures import EdgeFlow, _degrade, cmi_from_joint, dense_codes, intrinsic_flow, joint_table, layers_from_values
from .measures import (
DEFAULT_LAYER_ESTIMATOR,
EdgeFlow,
_degrade,
cmi_from_joint,
dense_codes,
evaluation_table,
intrinsic_flow,
joint_table,
layers_from_values,
)

__all__ = (
"BAYES_OPTIONS",
Expand Down Expand Up @@ -90,7 +100,7 @@ def _layers_on_table(table, mapping, n_clusters, estimator, initial, prng):
_, Q = intrinsic_flow(merged, initial=initial, prng=prng)
te = cmi_from_joint(table, estimator)
tdmi = cmi_from_joint(table.sum(axis=2, keepdims=True), estimator)
iif = min(cmi_from_joint(_degrade(merged, Q), estimator, n=n), te, tdmi)
iif = min(cmi_from_joint(evaluation_table(merged, Q, estimator), estimator, n=n), te, tdmi)
te, tdmi, iif, syn, shared, _ = layers_from_values(te, tdmi, iif)
return {"te": te, "tdmi": tdmi, "intrinsic": iif, "synergistic": syn, "shared": shared}

Expand Down Expand Up @@ -132,7 +142,7 @@ def layer_statistics(
n_null=100,
confidence=0.95,
mean_block_length=None,
estimator="miller_madow",
estimator=DEFAULT_LAYER_ESTIMATOR,
merge=True,
merge_alpha=0.01,
gap=0,
Expand Down
63 changes: 54 additions & 9 deletions infoflow/measures.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@

_LOG2 = np.log(2)

#: Estimator of the flow layers unless one is given: least layer error for its cost in
#: ``validation/dit_estimators.py`` and ``validation/new_defaults.py`` :cite:`Chao2013`.
DEFAULT_LAYER_ESTIMATOR = "chao_wang_jost"


def dense_codes(values):
"""
Expand Down Expand Up @@ -68,6 +72,21 @@ def _entropy_terms(q):
return float(-np.sum(q * np.log(q)))


def _dit_entropy(counts, estimator, K):
"""
Entropy in bits of a count array with a dit estimator and `K` possible outcomes.
"""
from dit.inference.estimators import ENTROPY_ESTIMATORS

if estimator not in ENTROPY_ESTIMATORS:
raise ValueError(f"Unknown estimator {estimator!r}; choose from {sorted(ENTROPY_ESTIMATORS)}.")
counts = counts[counts > 0]
if len(counts) <= 1:
# One observed outcome (e.g. a context with a single value); some estimators reject it.
return 0.0
return float(ENTROPY_ESTIMATORS[estimator](counts, counts.sum(), K)) / _LOG2


def cmi_from_joint(p, estimator="plugin", n=None):
"""
:math:`I[Y : X \\mid W]` in bits from a table ``p[y, x, w]`` (counts or probabilities).
Expand All @@ -77,28 +96,52 @@ def cmi_from_joint(p, estimator="plugin", n=None):
p : np.ndarray
Joint table over (y, x, w). With a single context value this is
:math:`I[Y : X]`.
estimator : {'plugin', 'miller_madow'}
``'miller_madow'`` adds each entropy term's :math:`(K - 1) / 2N` bias
correction :cite:`Miller1955`, with :math:`K` the number of occupied
cells. It needs the sample size `n` (taken from ``p.sum()`` for counts).
estimator : str
``'plugin'``; ``'miller_madow'``, which adds each entropy term's
:math:`(K - 1) / 2N` bias correction :cite:`Miller1955`, with :math:`K` the
number of occupied cells (it needs the sample size `n`, taken from ``p.sum()``
for counts); or any other of dit's discrete entropy estimators
(:data:`dit.inference.estimators.ENTROPY_ESTIMATORS`), applied to each entropy
term of the count table with the product of its axes' sizes as the alphabet.
Those need integer counts: pass counts, or probabilities with `n`.
n : float, None
The number of samples behind a probability table.
"""
total = p.sum()
if total <= 0:
return 0.0
n = total if n is None else n
if estimator not in ("plugin", "miller_madow"):
c = np.rint(p * (n / total))
Ky, Kx, Kw = p.shape
return (
_dit_entropy(c.sum(axis=0).ravel(), estimator, Kx * Kw)
+ _dit_entropy(c.sum(axis=1).ravel(), estimator, Ky * Kw)
- _dit_entropy(c.ravel(), estimator, Ky * Kx * Kw)
- _dit_entropy(c.sum(axis=(0, 1)), estimator, Kw)
)
q = p / total
q_xw, q_yw, q_w = q.sum(axis=0), q.sum(axis=1), q.sum(axis=(0, 1))
value = _entropy_terms(q_xw) + _entropy_terms(q_yw) - _entropy_terms(q) - _entropy_terms(q_w)
if estimator == "miller_madow":
k = lambda a: np.count_nonzero(a > 1e-12) # noqa: E731
value += ((k(q_xw) - 1) + (k(q_yw) - 1) - (k(q) - 1) - (k(q_w) - 1)) / (2 * n)
elif estimator != "plugin":
raise ValueError(f"Unknown estimator {estimator!r}; use 'plugin' or 'miller_madow'.")
return value / _LOG2


def evaluation_table(table, Q, estimator):
"""
The table on which `estimator` evaluates the intrinsic flow at channel `Q`.

Plug-in and Miller--Madow evaluate the degraded table itself; dit's other
estimators need integer counts, so the channel is hardened to its most likely
output per context (the minimizing channel is usually close to deterministic).
"""
if estimator in ("plugin", "miller_madow"):
return _degrade(table, Q)
return _degrade(table, np.eye(Q.shape[1])[Q.argmax(axis=1)])


def _degrade(p, Q):
"""
``q[y, x, z] = sum_w p[y, x, w] Q[w, z]``.
Expand Down Expand Up @@ -268,8 +311,10 @@ def edge_flows(y, x, w, estimator="plugin", bound=None, restarts=3, prng=None):
The source's past (1D codes or a 2D array whose rows are joint values).
w : array_like or None
The context (1D codes or 2D rows); None for an empty context.
estimator : {'plugin', 'miller_madow'}
Applied to TE, TDMI, and the intrinsic flow at the minimizing channel.
estimator : str
``'plugin'``, ``'miller_madow'``, or another of dit's discrete entropy
estimators (:func:`cmi_from_joint`), applied to TE, TDMI, and the intrinsic
flow at the minimizing channel (:func:`evaluation_table`).
bound, restarts, prng
Passed to :func:`intrinsic_flow`.

Expand All @@ -285,7 +330,7 @@ def edge_flows(y, x, w, estimator="plugin", bound=None, restarts=3, prng=None):
iif_plugin, Q = intrinsic_flow(table, bound=bound, restarts=restarts, prng=prng)
te = cmi_from_joint(table, estimator)
tdmi = cmi_from_joint(table.sum(axis=2, keepdims=True), estimator)
iif = cmi_from_joint(_degrade(table, Q), estimator, n=n)
iif = cmi_from_joint(evaluation_table(table, Q, estimator), estimator, n=n)
te, tdmi, iif, syn, shared, clipped = layers_from_values(te, tdmi, iif)
raw = {
"te": cmi_from_joint(table),
Expand Down
Loading
Loading