class: center, middle, inverse, title-slide .title[ # Model Combinations<br> through revised<br> base rates ] .author[ ### Anastasios Panagiotelis ] .institute[ ### University of Sydney ] --- <script type="text/x-mathjax-config"> MathJax.Hub.Config({ TeX: { Macros: { bA: "{\\mathbf{A}}", bC: "{\\mathbf{C}}", bv: "{\\mathbf{v}}", bw: "{\\mathbf{w}}", bx: "{\\mathbf{x}}", bX: "{\\mathbf{X}}", by: "{\\mathbf{y}}", bY: "{\\mathbf{Y}}", bw: "{\\mathbf{w}}", bW: "{\\mathbf{W}}", bK: "{\\mathbf{K}}", bz: "{\\mathbf{z}}", bU: "{\\mathbf{U}}", bS: "{\\mathbf{S}}", bff: "{\\mathbf{f}}", bD: "{\\mathbf{D}}", bL: "{\\mathbf{L}}", calL: "{\\mathcal{L}}", calM: "{\\mathcal{M}}", bbR: "{\\mathbb{R}}", bB: "{\\mathbf{B}}", bbb: "{\\mathbf{b}}", bLambda: "{\\boldsymbol{\\Lambda}}", bDelta: "{\\boldsymbol{\\Delta}}", bH: "{\\mathbf{H}}", bI: "{\\mathbf{I}}", biota: "{\\boldsymbol{\\iota}}", bN: "{\\mathbf{N}}", bQ: "{\\mathbf{Q}}" } } }); </script> # Co-authors .pull-left[ <img src="img/Fotis.jpeg" width="300" /> Fotios Petropoulos] .pull-right[ <img src="img/Vaggelis.jpeg" width="300" /> Evangelos Spiliotis] --- class: inverse, center, middle # A motivation --- # A problem for undergrads - An algorithm detects fraud -- - The algorithm has an accuracy of 90% -- + If fraud is committed the algorithm detects fraud 90% of the time -- + If fraud is not committed the algorithm detects no fraud 90% of the time. -- - You apply the algorithm to an individual and the algorithm detects fraud. - What is the probability that the individual has truly committed fraud? --- # A problem for forecasters - An information criterion (IC) selects whether a forecasting model is 'correct'. - The (IC) has an accuracy of 90% + If model truly correct the IC selects it 90% of the time + If model is not truly correct, the algorithm selects another model 90% of the time. - You apply the IC to a time series and the algorithm selects the model. - What is the probability that selected model is the correct model? --- # Setting - Setting is multivariate data -- + Macroeconomics (FRED data) + Retail (SKU) + M forecasting competitions -- - Will focus on univariate models however... -- - ... still want to cross learn. -- - Can the behavior of selection criteria across a set of 'reference' series improve forecast selection and combination? --- class: inverse, center, middle # The method --- # Selected and Correct - Information criteria include AIC and BIC - "Correct" model is one that minimises forecast error (MAE or RMSE) - Construct a contingency table of selected/correct models. - Compute the following probabilities + p(C|S): Precision + p(S|C): Sensitivity - Sensitivity = Precision if all models are equally likely to be the correct model a priori. --- # Computing the Table - Fit all models -- + Compute selection criterion + Determine selected model -- - Forecast using all models -- + Evaluate out of sample criterion + Determine "correct" model -- - Repeat for all series --- # (Part of) a cross tab | | ARMA(1,1)| SAR(1)| SMA(1)| Total| |:---------|---------:|------:|------:|-----:| |ARMA(1,1) | 25| 0| 1| 26| |SAR(1) | 0| 34| 2| 36| |SMA(1) | 0| 7| 0| 7| |Total | 25| 41| 3| 69| - This is a simulation where all series are ARMA(1,1) or SAR(1). - Sometimes we select SMA(1), but cross learning leads the correct model. --- # Precision <table> <thead> <tr> <th style="text-align:left;"> </th> <th style="text-align:right;"> ARMA(1,1) </th> <th style="text-align:right;"> SAR(1) </th> <th style="text-align:right;"> SMA(1) </th> <th style="text-align:right;"> Total </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> ARMA(1,1) </td> <td style="text-align:right;"> 25 </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 26 </td> </tr> <tr> <td style="text-align:left;font-weight: bold;"> SAR(1) </td> <td style="text-align:right;font-weight: bold;"> 0 </td> <td style="text-align:right;font-weight: bold;"> 34 </td> <td style="text-align:right;font-weight: bold;"> 2 </td> <td style="text-align:right;font-weight: bold;"> 36 </td> </tr> <tr> <td style="text-align:left;"> SMA(1) </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 7 </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 7 </td> </tr> <tr> <td style="text-align:left;"> Total </td> <td style="text-align:right;"> 25 </td> <td style="text-align:right;"> 41 </td> <td style="text-align:right;"> 3 </td> <td style="text-align:right;"> 69 </td> </tr> </tbody> </table> -- - Suppose SAR(1) selected. - Consider one row and normalise. - Weights w=(0, 0.944, 0.056) proxy `\(p(C|S)\)`. --- # Sensitivity <table> <thead> <tr> <th style="text-align:left;"> </th> <th style="text-align:right;"> ARMA(1,1) </th> <th style="text-align:right;"> SAR(1) </th> <th style="text-align:right;"> SMA(1) </th> <th style="text-align:right;"> Total </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> ARMA(1,1) </td> <td style="text-align:right;"> 25 </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 26 </td> </tr> <tr> <td style="text-align:left;"> SAR(1) </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 34 </td> <td style="text-align:right;"> 2 </td> <td style="text-align:right;"> 36 </td> </tr> <tr> <td style="text-align:left;"> SMA(1) </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 7 </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 7 </td> </tr> <tr> <td style="text-align:left;"> Total </td> <td style="text-align:right;"> 25 </td> <td style="text-align:right;"> 41 </td> <td style="text-align:right;"> 3 </td> <td style="text-align:right;"> 69 </td> </tr> </tbody> </table> -- - Apply Bayes' Rule to get `\(p(S|C)\)` -- - If SAR(1) selected w=(0, 0.554, 0.446) -- - May be useful if assume uniform priors of 'correct' model for new series --- class: inverse, center, middle # Applications --- #Fred data - Consider monthly FRED data (Ng and McCracken data). -- - Forecast 115 series. -- - Use a training sample of `\(T=60\)` and rolling window of length `\(W_{in}=30\)` to construct cross tab. -- - Then wrap this inside another rolling window of size `\(W_{out}=90\)`. -- - Evaluation based on MASE. -- - Consider 16 (Seasonal) ARMA models --- # Benchmarks - Selection - Best model according to criterion (criterion-select) - Most common correct model overall (aggregate-select) -- - Combination - Equal weights - Criterion specific weights `\(w_i\propto exp(S_i)\)` - Regularised weights (Diebold and Shin, 2019) --- # Selection Results <table> <thead> <tr> <th style="text-align:left;"> Method </th> <th style="text-align:right;"> MASE </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> Aggregate-Select </td> <td style="text-align:right;"> 0.553 </td> </tr> <tr> <td style="text-align:left;font-weight: bold;"> Criterion-Select </td> <td style="text-align:right;font-weight: bold;"> 0.530 </td> </tr> <tr> <td style="text-align:left;"> Precision-Select </td> <td style="text-align:right;"> 0.537 </td> </tr> <tr> <td style="text-align:left;"> Sensitivity-Select </td> <td style="text-align:right;"> 0.544 </td> </tr> </tbody> </table> For selection in this example, cross learning doesn't help. --- #Combination Results <table> <thead> <tr> <th style="text-align:left;"> Method </th> <th style="text-align:right;"> MASE </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> Equal </td> <td style="text-align:right;"> 0.532 </td> </tr> <tr> <td style="text-align:left;"> Regularised </td> <td style="text-align:right;"> 0.533 </td> </tr> <tr> <td style="text-align:left;"> Criterion-Average </td> <td style="text-align:right;"> 0.527 </td> </tr> <tr> <td style="text-align:left;font-weight: bold;"> Precision-Average </td> <td style="text-align:right;font-weight: bold;"> 0.525 </td> </tr> <tr> <td style="text-align:left;"> Sensitivity-Average </td> <td style="text-align:right;"> 0.527 </td> </tr> </tbody> </table> For combination in this example, cross learning can help. --- # M Competition - Data from M, M3 and M4. -- - Yearly, Quarterly and Monthly frequency. -- - Consider 15 clases of ETS model. -- - Total of 95,434 series. --- # Selection Results <table> <thead> <tr> <th style="text-align:left;"> Method </th> <th style="text-align:right;"> Yearly </th> <th style="text-align:right;"> Quarterly </th> <th style="text-align:right;"> Monthly </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> Aggregate-Select </td> <td style="text-align:right;"> 3.512 </td> <td style="text-align:right;"> 1.192 </td> <td style="text-align:right;"> 0.988 </td> </tr> <tr> <td style="text-align:left;"> Criterion-Select </td> <td style="text-align:right;"> 3.412 </td> <td style="text-align:right;font-weight: bold;"> 1.166 </td> <td style="text-align:right;"> 0.949 </td> </tr> <tr> <td style="text-align:left;"> Precision-Select </td> <td style="text-align:right;"> 3.490 </td> <td style="text-align:right;"> 1.184 </td> <td style="text-align:right;"> 0.985 </td> </tr> <tr> <td style="text-align:left;"> Sensitivity-Select </td> <td style="text-align:right;font-weight: bold;"> 3.309 </td> <td style="text-align:right;"> 1.174 </td> <td style="text-align:right;font-weight: bold;"> 0.948 </td> </tr> </tbody> </table> For selection in this example, sensitivity select works well for yearly and quarterly data. --- # Combination Results <table> <thead> <tr> <th style="text-align:left;"> Method </th> <th style="text-align:right;"> Yearly </th> <th style="text-align:right;"> Quarterly </th> <th style="text-align:right;"> Monthly </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> Equal </td> <td style="text-align:right;"> 3.231 </td> <td style="text-align:right;"> 1.174 </td> <td style="text-align:right;"> 0.948 </td> </tr> <tr> <td style="text-align:left;"> Criterion-Average </td> <td style="text-align:right;"> 3.351 </td> <td style="text-align:right;"> 1.152 </td> <td style="text-align:right;"> 0.942 </td> </tr> <tr> <td style="text-align:left;"> Precision-Average </td> <td style="text-align:right;"> 3.247 </td> <td style="text-align:right;font-weight: bold;"> 1.147 </td> <td style="text-align:right;font-weight: bold;"> 0.160 </td> </tr> <tr> <td style="text-align:left;"> Sensitivity-Average </td> <td style="text-align:right;font-weight: bold;"> 3.212 </td> <td style="text-align:right;"> 1.155 </td> <td style="text-align:right;"> 0.922 </td> </tr> </tbody> </table> For combination in this example, cross learning helps at all frequencies. --- # Size of Reference Set <img src="img/ss.png" width="2340" /> --- # Conclusions - This is a simple way to do cross learning. -- - Can achieve modest gains for a small number of variables (~100). -- - For 1k-100k variables, our combinations methods are + Outperforming benchmarks, + Better than corresponding selection methods and + Robust to size and choice of 'reference' set --- class: inverse, middle, center # Questions ?