Recursive Feature Elimination vs LASSO A Performance Analysis in High-Dimensional Time Series Data

I've been wrestling with a familiar demon lately: feature selection in high-dimensional time series forecasting. When you're staring down thousands of potential predictors, trying to build a model that generalizes rather than just memorizes noise, the choice of selection method becomes anything but trivial. We’re often dealing with financial tick data, sensor readings from industrial machinery, or perhaps dense genomic sequences over time—scenarios where the curse of dimensionality bites hard and fast.

The traditional approach often leans toward regularization techniques, and LASSO, with its $\ell_1$ penalty, is the go-to for producing sparse models by driving less important coefficients exactly to zero. But then there's Recursive Feature Elimination (RFE), a wrapper method that iteratively prunes features based on the performance of a chosen estimator, usually requiring repeated model fitting. I wanted to put these two fundamentally different philosophies head-to-head specifically when the data carries temporal dependence. Does the iterative, performance-driven pruning of RFE handle autocorrelated noise better than the inherent sparsity bias of LASSO when applied to time series? It's a practical question that separates theoretical preference from actual predictive accuracy when the stakes are high.

Let's consider LASSO first. Its mechanism is elegantly straightforward: it adds the sum of the absolute values of the coefficients to the loss function. This pushes the model toward simplicity, which is usually desirable when you suspect many features are irrelevant or highly redundant, a common situation in high-frequency data streams. However, when features are highly correlated—a near certainty in time series where lagged variables are common—LASSO has a known tendency to arbitrarily select only one variable from the correlated group and zero out the others. This selection is not necessarily based on which feature is *truly* the most predictive in the long run, but rather which one happens to win the initial coefficient estimation lottery. For time series where the predictive signal might be subtly distributed across several highly coupled predictors, this aggressive zeroing-out can lead to discarding valuable, albeit redundant, information. I’ve seen this manifest as models that perform well on the training window but exhibit sudden drops in out-of-sample stability when the underlying data generating process shifts slightly.

Now, let's turn our attention to RFE. This method starts with all features, trains the base estimator (say, a linear model or even a Support Vector Machine), ranks the features based on coefficient magnitude or feature importance scores derived from that estimator, and then systematically removes the least important one. This loop repeats until the desired number of features remains, or until performance plateaus. The critical difference here is that RFE is constantly optimizing against a performance metric—like cross-validation error—at each stage of elimination, rather than just minimizing a penalized objective function. This means RFE inherently adapts to the specific predictive landscape of the data subset being evaluated at that iteration. If two correlated features offer slightly different predictive power across different temporal segments, RFE’s iterative evaluation is more likely to retain both, or at least one, based on demonstrated predictive utility rather than just regularization strength. My initial hypothesis, which the recent tests seem to support, is that RFE maintains a slight edge in scenarios where the feature space is dense with weak but non-zero predictive signals spread across correlated variables, which is the bread and butter of complex time series modeling. The computational burden of RFE, of course, scales poorly with the number of features and observations, but in many modern computing environments, the gain in stability can justify the extra fitting time.