Out-of-Sample Assessment

Does a candidate predictor of model bias still work where you did not measure?

That is a different question from "does it fit?", and it is the only one that matters when the point of a bias correction is to reach the ungauged terrain between the gauges. A predictor fitted and scored on the same points will nearly always look useful.

Predicting a point from a group that contains it is not out-of-sample

loo_cluster_pred recomputes each group's mean without the point being predicted. Including it is the standard way to make a regionalisation look skilful when it is merely descriptive — the point is partly predicting itself, and the effect is largest for exactly the small groups a clustering tends to produce.

A singleton group falls back to the leave-one-out global mean, because a group of one carries no information about itself.

The baseline matters as much as the method

loo_global_mean is the aspatial baseline — "correct everything by the same amount". Any regionalisation, covariate or clustering has to beat this to have earned its complexity, and a surprising amount of apparent skill disappears when it is measured against a leave-one-out global mean rather than against no correction at all.

Turn two sets of residuals into one number with compute_residual_skill_score — positive means the method beat the baseline, negative means the baseline won, which is the outcome worth reporting rather than hiding.

Nothing here is spatial

loo_cluster_pred takes values and labels. The labels may come from a spatial clustering (modverif.spatial.best_kmeans) or from anything else — land-use class, elevation band, forecast regime. That is why these live apart from the structure diagnostics.

API

Out-of-sample assessment of candidate predictors for a scattered error field.

The question here is not "does this predictor fit?" but "does it still work where you did not measure?" -- which is the only version that matters when the point of a bias correction is to reach the ungauged terrain between the gauges.

Nothing in this module is spatial. loo_cluster_pred takes values and labels; the labels may come from a spatial clustering (modverif.spatial.best_kmeans) or from anything else. Keeping these apart from the structure diagnostics is what makes them findable by the next consumer, who may have no variogram in sight.

Pair the residuals these produce with modverif.metrics.compute_residual_skill_score, which is the common way to turn two sets of residuals into a single "did it beat the baseline?" number.

cluster_holdout(pred_fn, values, labels)

Leave-one-group-out: predict each held-out group from all the others.

Much harsher than leave-one-out, and much closer to the real question. Leave-one-out leaves a point surrounded by its own neighbours; leaving out a whole region asks whether the relationship transfers to somewhere it was never fitted. A regionalisation that scores well under leave-one-out and badly here is describing the gauges, not the field.

Parameters:

Name	Type	Description	Default
pred_fn	callable	pred_fn(train_mask, held_mask) -> predictions for the held points. Taking a callback rather than a fixed model is what lets the same protocol score a covariate regression, a group mean, or anything else, against the identical baseline.	required
values	ndarray	Field being predicted.	required
labels	ndarray	Group label per point.	required

Returns:

Type	Description
float	Skill against the training-set mean of each fold.

holdout_high(covariate, values, q)

Train on the low end of a covariate and predict the high end -- an extrapolation test.

This is the decisive check when the operational claim is "correct the bias where we have no measurements", and the ungauged places sit at the extreme of the covariate. Rain gauges cluster in valleys, so a correction applied to ridges extrapolates the elevation slope beyond any support in the data. Leave-one-out cannot detect that; withholding the top of the range can.

A negative skill here is the informative result: it says the relationship, however significant in-sample, does not reach the places the correction was wanted for.

Parameters:

Name	Type	Description	Default
covariate	ndarray	Predictor per point. The highest q fraction is withheld.	required
values	ndarray	Field being predicted.	required
q	float	Fraction to withhold; at least 2 points are always held out.	required

Returns:

Name	Type	Description
skill	float	modverif.metrics.compute_residual_skill_score of the fitted prediction against the training-set mean -- the honest baseline, since a model that cannot beat "assume the training average" has bought nothing.
diagnostics	tuple	(slope, intercept, slope_lo, slope_hi, max_train_covariate, held_indices). The fourth and fifth elements are what make the extrapolation visible: the slope's confidence interval, and the covariate value beyond which the prediction has no support at all.

loo_cluster_pred(z, labels)

Leave-one-out prediction from cluster means.

Each point is predicted from its own cluster's mean computed without it. Including the point in its own predictor is the standard way to make a regionalisation look skilful when it is merely descriptive.

Parameters:

Name	Type	Description	Default
z	ndarray	Field values.	required
labels	ndarray	Cluster label per point.	required

Returns:

Type	Description
ndarray	Prediction per point. A singleton cluster falls back to the leave-one-out global mean, since a cluster of one carries no information about itself.

loo_global_mean(z)

Leave-one-out global mean -- the aspatial baseline any regionalisation must beat.

Parameters:

Name	Type	Description	Default
z	ndarray	Field values.	required

Returns:

Type	Description
ndarray	Mean of all other points, per point.

loo_theilsen(covariate, values)

Leave-one-out Theil--Sen prediction of values from a scalar covariate.

Refits the regression n times, each time omitting the point being predicted. Theil--Sen rather than least squares because a bias field's relationship to a covariate is usually driven by the bulk of the points, and a handful of extreme stations should not set the slope.

Parameters:

Name	Type	Description	Default
covariate	ndarray	Predictor per point -- elevation, distance to coast, whatever is being assessed.	required
values	ndarray	Field being predicted, typically log(model / obs).	required

Returns:

Type	Description
ndarray	Out-of-sample prediction per point.

Notes

Leave-one-out still shares most of the training set between folds, so it is the mildest honest test available. It says nothing about whether the relationship extrapolates beyond the range the points cover -- for that, use holdout_high.