ML Workflows

A data scientist is preparing a customer churn dataset. The 'age' feature is continuous and highly right-skewed, and the team wants to convert it into categorical bins so that each bin contains roughly the same number of customers, which will help a downstream tree ensemble handle the skew more evenly. Which binning strategy should they apply?

During exploratory data analysis on a Spark DataFrame of customer records, a data scientist computes summary statistics with df.describe() and notices that the numeric column 'account_region_code' has a standard deviation of 0.0 and identical min, max, and mean values across all 500,000 rows. What is the most appropriate action to take before model training?

A data scientist is preparing features for a linear regression model that predicts house prices. During EDA, they compute a Pearson correlation matrix and notice that 'total_square_feet' and 'number_of_rooms' have a correlation coefficient of 0.94 with each other, while each correlates around 0.60 with the target price. What is the most appropriate action to take before training the linear model?

During exploratory data analysis on a housing dataset, an analyst computes a covariance matrix between 'square_footage' (measured in thousands of square feet) and 'lot_size' (measured in acres). The covariance value is 45.2, and the analyst wants to determine the strength of the linear relationship between the two features to decide whether one is redundant. Why is the correlation coefficient a more appropriate metric than the raw covariance value here?

A data scientist runs df.describe() on a Spark DataFrame of loan applications before modeling. For the 'annual_income' column, the summary shows: count = 48,500 (out of 50,000 rows), min = -1, mean = 62,300, 50% (median) = 55,000, max = 9,999,999. Which single observation from these statistics most strongly indicates a data quality problem that should be investigated before feature engineering?

A data scientist is building a scikit-learn pipeline to preprocess a dataset before training a logistic regression model. The dataset contains numeric features with widely different ranges and several rows with missing values in those numeric columns. The scientist wants to (1) impute missing numeric values with the column median and (2) standardize the features. To avoid data leakage and produce correct results, in what order should these steps be arranged inside the pipeline, and how should they be fit?

During exploratory data analysis on a Spark DataFrame of loan applications, a data scientist wants to flag outliers in the 'annual_income' column using the interquartile range (IQR) method. They compute Q1 (25th percentile) = 40,000 and Q3 (75th percentile) = 90,000. Using the standard IQR rule, which values should be considered outliers?

A data scientist is preparing a numeric feature, annual_income, that contains missing values in a customer churn dataset. They plan to impute the missing values with the median before training a model. They have already split the data into training and test sets. To avoid data leakage while imputing, what is the correct approach?

A data scientist is preparing a customer churn dataset in a Spark DataFrame. The categorical column 'preferred_channel' has about 8% missing values, and the business confirms these are missing at random with no meaningful 'unknown' category. The scientist wants to impute the missing values before one-hot encoding, while avoiding data leakage. Which approach is most appropriate?

You are building a Spark ML pipeline to predict customer churn. One categorical feature, 'zip_code', has approximately 12,000 distinct values. Your teammate proposes applying StringIndexer followed by OneHotEncoder to this column, as is done for the other low-cardinality categorical features. What is the primary concern with applying one-hot encoding directly to 'zip_code', and what is a more appropriate approach?

A data scientist fits a scikit-learn OneHotEncoder on a training set containing a categorical 'region' column with values {'North', 'South', 'East'}. During batch inference, the incoming data contains a new value 'West' that was never seen during training. With the encoder's default configuration, what happens, and what is the recommended fix so inference does not fail?

A data scientist is preparing a feature named 'shirt_size' with categories 'S', 'M', 'L', and 'XL' for a gradient-boosted tree model. They want the encoding to preserve the natural order of the sizes while producing a single numeric column. Which feature engineering approach best fits this requirement?

A data scientist is preparing a housing price dataset for a regression model. During EDA, they find that about 3% of the 'lot_area' values are extreme outliers caused by legitimate large commercial lots, not data-entry errors. The dataset is relatively small (2,000 rows), and dropping these rows would lose meaningful information. The scientist wants to reduce the influence of these extreme values on model training while retaining every observation. Which approach best meets this goal?

During exploratory data analysis on a housing price dataset, a data scientist computes a Pearson correlation matrix and finds that 'living_area_sqft' and 'total_rooms' have a correlation coefficient of 0.94. They plan to train a linear regression model. What is the most appropriate action to take based on this finding?

A data scientist splits a Spark DataFrame into training and test sets using randomSplit([0.8, 0.2]). A colleague runs the exact same notebook code on the same cluster the next day but reports getting different rows in each split, causing model metrics to differ. What is the most likely cause, and how should it be fixed?

A data scientist is building a RandomForestClassifier in a Databricks notebook and needs the entire pipeline to be fully reproducible across reruns. They already pass a fixed integer to train_test_split's random_state. On repeated runs, the train/test split is identical, but the trained model still produces slightly different predictions each time. What is the most likely cause and correct fix?

A data scientist is preparing a dataset of customer features for a logistic regression model. Their preprocessing code applies a StandardScaler to the entire dataset first, and only afterward splits the data into training and test sets using train_test_split. During review, a colleague flags this as a methodological problem. What is the primary issue with this workflow ordering?

During exploratory data analysis of a housing dataset, a data scientist computes summary statistics on the 'lot_area' feature and finds that the mean (18,400) is far larger than the median (9,600), with a maximum value of 1,200,000. A histogram shows a long right tail. The scientist wants to reduce the impact of this heavy right skew before training a linear regression model, without discarding any rows. Which approach best addresses the skew while preserving all observations?

During exploratory data analysis on a Spark DataFrame, a data scientist examines the relationship between a house's 'age' and its 'maintenance_cost'. A scatter plot shows that maintenance cost consistently increases as age increases, but the growth is clearly curved rather than a straight line. The scientist computes the Pearson correlation coefficient and gets a value of 0.61, which seems to understate how strong the relationship looks visually. Which action will best quantify the strength of this monotonic but nonlinear relationship?

A data scientist is preparing a numeric feature 'annual_income' for a linear regression model. Exploratory analysis shows the feature has several extreme high-value outliers that were confirmed to be legitimate (not data errors). The team wants to scale the feature so that these outliers do not disproportionately compress the majority of the data into a narrow range. Which scaling approach best addresses this concern?