Evaluation & Search¶
Scoring point, interval, and class-probability forecasts; aggregation modes; time-weighted and multi-vintage scoring; custom scorers; and hyperparameter search with GridSearchCV and RandomizedSearchCV.
-
How to Aggregate Scorer Results
Demonstrate all scorer aggregation strategies (stepwise, vintagewise, componentwise, groupwise, coveragewise, all) on panel data with weighted group aggregation.
-
How to Score Class-Probability Forecasts
Evaluate categorical forecasts with LogLoss, BrierScore, and Accuracy. Covers per-timestep scoring, aggregation modes, and reliability diagrams.
-
How to Use Conformity Scorers
Compare Residual, AbsoluteResidual, GammaResidual, and AbsoluteGammaResidual conformity scorers with coverage/width analysis and DistanceSimilarity interaction.
-
Cross-Validation for Time Series
Evaluate forecasters with cross_val_score, cross_validate, and cross_val_predict using temporal splitters.
-
How to Create a Custom Scorer
Implement a custom point scorer with aggregation, panel support, and systematic testing.
-
How to Run Hyperparameter Search
Tune forecaster hyperparameters with GridSearchCV and RandomizedSearchCV using temporal cross-validation splitters and result scatter visualisation.
-
How to Evaluate Interval Forecasts
Evaluate prediction intervals with EmpiricalCoverage, IntervalScore, MeanIntervalWidth, PinballLoss, and CalibrationError across coverage levels.
-
How to Search Interval Forecaster Hyperparameters
Tune interval forecaster parameters directly with interval metrics in GridSearchCV, including mixed point+interval multimetric search.
-
How to Search with Multiple Metrics
Evaluate hyperparameter configurations against multiple metrics simultaneously with dict-of-scorers, refit strategies, and Pareto-optimal selection.
-
How to Score Multi-Vintage Forecasts
Generate multi-vintage predictions with observe_predict, score per step and per vintage, and visualize with heatmap, per-step, and per-vintage plots.
-
How to Use Point Forecast Metrics
Compare MAE, MAPE, MASE, RMSE, and other point metrics across multiple forecasters with componentwise and groupwise aggregation.
-
How to Score with Time-Weighted Metrics
Apply exponential decay, linear decay, and seasonal emphasis weighting to forecast evaluation, prioritising recent or periodic time steps.