Feature Engineering for Machine Learning: From Raw Data to Predictive Power

Orientation

Feature engineering is the process of transforming raw observations into numeric or structured inputs that machine‑learning models can consume effectively. It matters whenever the raw signal does not directly map to the predictive problem at hand—high‑cardinality IDs, sparse logs, unstructured text, mixed data types, or temporal sequences all require deliberate shaping before model training. This page walks through concrete, repeatable ways to engineer those inputs, organized by the practical problem you’re solving rather than by algorithm taxonomy.

Deriving numeric values from timestamps, logs, and strings

Many real‑world pipelines begin with timestamps, event logs, or free‑form text that must be converted to usable numbers before any model can be fitted.

Extracting date‑time components with pandas

import pandas as pd


s = pd.Series(["2024-03-15T10:23:45Z", "2024-03-15T14:08:12Z", "2024-03-16T08:11:03Z"])

# Parse to datetime
dt = pd.to_datetime(s, utc=True)

# Derive hour, dayofweek, month as new features
features = pd.DataFrame({
    "hour": dt.dt.hour,
    "dow": dt.dt.dayofweek,
    "month": dt.dt.month
})

The pd.to_datetime function accepts format for strict parsing, utc=True to attach timezone info, and errors='coerce' to turn unparseable values into NaT rather than raising. If you omit errors, a single malformed entry will abort the whole call with ValueError: unconverted data remains: ....

Parsing structured fields from free‑form logs

When a log line contains key‑value pairs such as user_id=42 action=click timestamp=1710472425, a single str.extract can pull multiple columns at once:

line = "user_id=42 action=click timestamp=1710472425"
m = pd.Series([line]).str.extract(
    r"(user_id=(\d+))\s+(action=(\w+))\s+(timestamp=(\d+))",
    expand=True
)
m.columns = ["raw_user","user","raw_action","action","raw_ts","ts"]
m["user"] = m["user"].astype(int)
m["ts"] = pd.to_numeric(m["ts"])

The regex groups are captured verbatim; the expand=True flag returns a DataFrame rather than a nested tuple. If the pattern does not match a line, the resulting cells are NaN—no exception is raised, which is useful for batch processing uneven log sources.

Encoding categorical and text data

Categorical and textual fields are the most common source of engineering friction. Choosing the right encoding method silently changes model behavior.

One‑hot encoding with scikit‑learn’s OneHotEncoder

from sklearn.preprocessing import OneHotEncoder

cat = pd.DataFrame({"color": ["red", "blue", "green", "blue", "red"]})
enc = OneHotEncoder(handle_unknown="use_encoded_value", unknown_value=-1, sparse_output=False)
encoded = enc.fit_transform(cat)

# enc.categories_ tells you the learned ordering
print(enc.categories_)
# → [['blue' 'green' 'red']]
print(encoded)
# → [[1. 0. 0.]
#    [0. 0. 1.]
#    [0. 1. 0.]
#    [0. 0. 1.]
#    [1. 0. 0.]]

The handle_unknown="use_encoded_value" flag is the key difference from the default error mode: when a test‑time category was absent in training, instead of raising ValueError: Unknown categories encountered in new data when handle_unknown='error', the encoder substitutes unknown_value (here, -1) and produces a row of zeros. This avoids pipeline failures during online inference but introduces a sentinel feature that the model must learn to interpret.

If sparse_output=True (the default prior to scikit‑learn 1.2), the result is a sparse CSR matrix. In versions 1.2+, the default shifted to sparse_output=False; passing sparse=True explicitly still works but triggers a deprecation warning.

Ordinal encoding when order is meaningful

from sklearn.preprocessing import OrdinalEncoder

size = pd.DataFrame({"size": ["S", "M", "L", "XL"]})
enc = OrdinalEncoder(categories=[["S", "M", "L", "XL"]])
enc.fit_transform(size)
# → [[0.]
#    [1.]
#    [2.]
#    [3.]]

OrdinalEncoder imposes a numeric ordering on the categories. Use it only when the categories genuinely have a monotonic relationship (e.g., size, education level, risk grades). Applying it to nominal categories (e.g., country, product type) can mislead tree‑based models into treating the numeric gap as meaningful distance.

Text featureization basics

For free‑form text, a common first step is TF‑IDF featurization without leaving the engineering layer:

from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
    "red blue green",
    "blue green yellow",
    "red green yellow blue",
]
vec = TfidfVectorizer(max_features=3, ngram_range=(1, 2))
X = vec.fit_transform(corpus)
# X is a sparse matrix of shape (3, <n_features>)
print(vec.get_feature_names_out())
# → ['blue' 'green' 'red' 'blue green' 'green yellow' 'red blue' 'red green' 'green yellow blue' 'red green yellow' 'red blue green']

max_features caps the vocabulary size to control dimensionality. ngram_range=(1, 2) captures both unigrams and bigrams, which often captures semantic phrases like "red blue" without needing a separate phrase‑segmentation step. The output column names are available via get_feature_names_out(), which can be aligned with a downstream model's feature names via pd.DataFrame(X.toarray(), columns=vec.get_feature_names_out()).

Constructing interaction and derived features

Once individual columns are in numeric form, the next engineer question is whether combinations carry signal.

Polynomial and interaction features

from sklearn.preprocessing import PolynomialFeatures

X = pd.DataFrame({"x1": [1, 2, 3, 4], "x2": [10, 20, 30, 40]})
poly = PolynomialFeatures(degree=2, interaction_only=True, include_bias=False)
poly.fit_transform(X)
# → columns: ["x1" "x2" "x1 x2"]

interaction_only=True restricts the output to product terms between distinct features, omitting pure squares (x12, x22). This is often sufficient for capturing pairwise synergies while keeping the column count manageable. Setting degree=3 or higher quickly explodes the feature space: with 4 original columns, degree=3 yields \(\binom{4+3-1}{3} = 20\) columns, and with interactions enabled the count grows further. A rule of thumb: inspect poly.n_output_features_ before committing to a high degree.

Domain‑specific interaction example

If you have "average session duration" and "number of pages viewed," the ratio pages / duration may be more predictive than either raw column:

df["engagement_rate"] = df["pages_viewed"] / df["session_duration_seconds"]

Division by zero will produce inf or NaN depending on pandas settings. A safe pattern:

df["engagement_rate"] = df["pages_viewed"] / df["session_duration_seconds"].replace(0, pd.NA)
df["engagement_rate"] = df["engagement_rate"].fillna(0)

This silently replaces undefined ratios with zero, which is often acceptable if a zero duration implies no pages were viewed. If the business logic demands a different sentinel, replace 0 with a domain‑chosen constant.

Dimensionality reduction and feature selection within engineering

When the engineered corpus grows beyond a few dozen columns, reduction or selection becomes necessary before model training.

Univariate feature selection based on statistical tests

from sklearn.feature_selection import SelectKBest, f_classif

X = pd.DataFrame({
    "num_1": [1.0, 2.0, 3.0, 4.0, 5.0],
    "num_2": [10, 20, 30, 40, 50],
    "num_3": [0.1, 0.2, 0.3, 0.4, 0.5],
})
y = pd.Series([0, 0, 1, 1, 1])

selector = SelectKBest(f_classif, k=2)
X_selected = selector.fit_transform(X, y)
print(selector.get_support())       # → [True, False, True]
print(selector.scores_)            # → scores for each original feature

f_classif (ANOVA F‑value) tests each feature’s ability to distinguish between classes. SelectKBest keeps the top‑k. If k exceeds the number of features, sklearn raises ValueError: k must be <= X.shape[1]. The selector’s transform method can later be applied to hold‑out or production data using the same get_support_ mask.

Variance threshold pruning

from sklearn.feature_selection import VarianceThreshold

X = pd.DataFrame({
    "constant": [1, 1, 1, 1, 1],
    "varied": [0, 1, 2, 3, 4],
    "sparse": [0, 0, 1, 0, 0],
})
selector = VarianceThreshold(threshold=1.0)
X_selected = selector.fit_transform(X)
# → columns "varied" and "sparse" retained; "constant" dropped (zero variance)

VarianceThreshold removes features with variance below the specified threshold. A common pitfall: applying it to already‑encoded one‑hot features where many columns have tiny but non‑zero variance due to rare categories. Those rare columns may be informative (e.g., a " VIP" flag) and should be excluded from the variance filter before transformation, or the threshold should be set conservatively low (e.g., 1e-6).

Sharp edges: what silently fails, which options are confused for each other, what you only learn after it breaks in production

Feature leakage through engineering

The most dangerous silent failure occurs when engineering steps use information from outside the training fold. If you compute a feature like "time since last purchase" using the full dataset’s max timestamp before splitting, every row in both train and test gets a value anchored to the global latest date. The model learns a correlation that will not hold at inference time, and performance drops sharply when deployed against fresh data. Always split before engineering, or wrap engineering in a Pipeline that fits only on training data.

OneHotEncoder with sparse_output=False and memory pressure

In high‑cardinality settings (e.g., 10k+ unique categories), a dense one‑hot matrix can exceed RAM. The default in scikit‑learn <1.2 was sparse_output=True, producing a CSR matrix that saves memory but cannot be indexed with standard DataFrame operations without .toarray(). In 1.2+, the default flipped. If you are on an older version and need dense output, call .toarray() explicitly and monitor usage, or switch to hashing tricks (HashingEncoder‑style) that map categories to fixed‑width vectors without blowing up column count.

PolynomialFeatures interaction explosion with missing values

If any input column contains NaN, PolynomialFeatures will propagate NaN through every product term, potentially creating an all‑NaN column that breaks downstream estimators (e.g., LinearRegression raises LinAlgError: Singular matrix). Impute missing values before engineering, or use SimpleImputer inside a Pipeline before PolynomialFeatures. A safe pattern:

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import PolynomialFeatures

num_pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("poly", PolynomialFeatures(degree=2, interaction_only=True)),
])

Ordinal vs. one‑hot confusion in tree models

Tree‑based models (RandomForest, XGBoost, LightGBM) handle integer‑encoded categories differently than linear models. Ordinal encoding imposes an arbitrary order that the splitter may exploit, sometimes improving performance, sometimes degrading it. One‑hot encoding avoids order but increases dimensionality. A practical compromise is target‑encoding (computing the mean target per category) with smoothing to avoid overfitting, but this introduces its own leakage risk if computed on the full dataset. Always validate encoding choice with cross‑validation, not a single train‑test split.

TfidfVectorizer vocabulary drift between train and test

If the test corpus contains n‑grams absent from training, TfidfVectorizer fitted on train will produce zero columns for those terms at transform time, silently dropping features. Conversely, if you fit_transform on the combined train+test set, you leak vocabulary information. The correct workflow: fit on train, then transform on test. If you need to capture previously unseen terms, consider hashing vectorizers (sklearn.feature_extraction.text.HashingVectorizer) which map to a fixed‑size vector regardless of vocabulary, at the cost of interpretability.

Pipeline ordering gotchas

When chaining ColumnTransformer with OneHotEncoder and PolynomialFeatures, the order of transformers matters. ColumnTransformer applies each branch to its designated columns, then concatenates results. If you place PolynomialFeatures inside a branch that already one‑hots, you’ll get interaction terms between one‑hot indicators, which are mostly redundant (e.g., x1_x2 where both are binary and mutually exclusive). A common fix is to apply polynomial expansion only to numeric branches, keeping categorical branches as‑is or using target‑encoded summaries.

Production‑side encoding mismatches

In serving infrastructure, the same OneHotEncoder object must be reused with transform (not fit_transform). If the serving code calls fit_transform on incoming batches, each request re‑learns categories, resetting the encoder and producing inconsistent feature vectors. Persist the encoder (e.g., joblib.dump(enc, "enc.joblib")) and load it in the service, calling enc.transform(new_data). Similarly, TfidfVectorizer’s IDF statistics must be frozen; re‑fitting on live data will shift the IDF weights and change feature magnitudes mid‑stream.

These edges are not abstract—they surface as ValueError, LinAlgError, silent accuracy drops, or mysteriously drifting feature importances. Catching them requires disciplined pipeline construction, explicit train/test separation before any feature computation, and version‑pinned serialization of all fitted transformers.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.