Question

You are building a Spark ML pipeline to predict customer churn. One categorical feature, 'zip_code', has approximately 12,000 distinct values. Your teammate proposes applying StringIndexer followed by OneHotEncoder to this column, as is done for the other low-cardinality categorical features. What is the primary concern with applying one-hot encoding directly to 'zip_code', and what is a more appropriate approach?

Accepted Answer

One-hot encoding will produce ~12,000 sparse columns, drastically increasing dimensionality and training cost; consider target/frequency encoding or grouping rare categories instead.

Answer

One-hot encoding cannot be applied to numeric-looking strings like zip codes, so the column must first be cast to an integer type before encoding.

Answer

One-hot encoding will silently drop the zip_code column because Spark limits categorical columns to 1,000 levels; use StringIndexer alone with no further steps.

Answer

One-hot encoding introduces data leakage for high-cardinality features, so the column should be scaled with StandardScaler after indexing.