Resampling Inference¶
Permutation tests and multiple-comparison corrections. Deliberately not in
modverif.spatial — nothing here is spatial, and the next code to need a
family-wise correction is as likely to be a teleconnection analysis as a variogram.
Permutation tests here are one-sided, and the side is yours to choose
The default, side='greater', asks "is the observed statistic unusually high?" — which
suits Moran's I and variance-explained. A test against a negative prior — "is this
correlation unusually low?" — needs side='less'.
Calling the default on a lower-sided test reports very nearly its complement: a wrong answer that looks entirely reasonable, and one no amount of correction downstream will catch.
There is no two-sided option. Every test this serves has a directional prior, and choosing a side after seeing the data is precisely what a p-value cannot survive.
Why the add-one correction¶
permutation_pvalue counts the observed statistic as one of
its own null draws — (extreme + 1) / (n_null + 1). Without it, a p-value of exactly zero is
reportable, and that is never true: it only means the null sample was too small to resolve the tail.
With 999 permutations the smallest honest p-value is 0.001.
Holm, and when not to use it¶
holm_adjust controls the family-wise error rate — the chance of
any false positive across the whole family. It is uniformly more powerful than plain Bonferroni at
the same guarantee, so there is no reason to prefer Bonferroni.
Use it when a single false positive would change the conclusion — scanning distance bands for spatial structure, say, where one spurious band is enough to claim structure that is not there. When the question is instead "what fraction of my many discoveries are false?", a false-discovery-rate procedure is the right tool and Holm is needlessly conservative.
Eta-squared is an effect size, not evidence
eta_squared rises mechanically with the number of groups —
one group per point explains any field perfectly. Always pair it with a permutation test
(shuffle values across fixed labels) or with out-of-sample skill before concluding a grouping is
real.
API¶
Resampling inference and multiple-comparison corrections.
Deliberately not in modverif.spatial: nothing here is spatial. A permutation p-value, a
family-wise error correction and a variance-explained ratio are general statistical tools, and the
next code to need them is as likely to be a teleconnection analysis as a variogram. Burying them in a
module named for one application is how a second consumer ends up writing its own copy.
WARNING:
Permutation tests are one-sided here, and the side is the caller's to choose. The default
('greater') asks "is the observed statistic unusually high?", which suits Moran's I and
variance-explained. A test against a negative prior -- "is this correlation unusually low?" --
needs side='less', and calling the default on it silently reports very nearly its complement.
eta_squared(z, labels)
¶
Fraction of a field's variance explained by group means (between-SS / total-SS).
The natural effect size for "do these groups capture anything?" -- but on its own it is not evidence, because eta-squared rises mechanically with the number of groups. Enough clusters will explain any field perfectly. Pair it with a permutation test (shuffle the values across fixed group labels) or with out-of-sample skill before concluding the grouping is real.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
z
|
array - like
|
Field values. |
required |
labels
|
array - like
|
Group label per value; any hashable labels, not necessarily contiguous integers. |
required |
Returns:
| Type | Description |
|---|---|
float
|
Ratio in |
holm_adjust(pvals)
¶
Holm--Bonferroni step-down adjusted p-values, returned in the input order.
Controls the family-wise error rate: the probability of any false positive across the whole family. Uniformly more powerful than plain Bonferroni at the same guarantee, so there is no reason to prefer Bonferroni.
Use this when a single false positive would change the conclusion -- for example when scanning distance bands for spatial structure, where one spurious band is enough to claim structure that is not there. When the question is instead "what fraction of my many discoveries are false?", a false-discovery-rate procedure is the better tool and this is needlessly conservative.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pvals
|
array - like
|
Unadjusted p-values. |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
Adjusted p-values, aligned to the input order, each capped at 1.0 and non-decreasing in the sorted order (the step-down monotonicity that makes the family interpretable). |
permutation_pvalue(observed, null, side='greater')
¶
One-sided permutation p-value with the add-one correction.
The +1 in both numerator and denominator counts the observed statistic as one of its own
null draws. Without it a p-value of exactly 0 is reportable, which is never true -- it just means
the null sample was too small to resolve the tail.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
observed
|
float
|
The statistic computed on the real data. |
required |
null
|
array - like
|
Statistics from the permuted replicates. Non-finite entries are dropped, and the denominator counts only what survived -- a replicate whose statistic failed to compute is not evidence either way, and leaving it in the denominator deflates the p-value. |
required |
side
|
(greater, less)
|
|
'greater'
|
Returns:
| Type | Description |
|---|---|
float
|
p-value in |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Notes
A non-finite observed is refused rather than scored. Every comparison against NaN is
False, so a NaN statistic would score zero extremes and return the smallest reportable
p-value -- on both sides simultaneously. That is the most dangerous possible failure for
this function: a statistic that could not be computed reported as maximally significant.
Callers whose statistic can legitimately fail (too few samples, say) must check before calling.