Question

A data scientist is building a multiclass classifier to route support tickets into 8 categories. Two categories represent 85% of the tickets, while the remaining six are rare. Using a random 80/20 train/test split, the scientist notices that the reported test accuracy varies significantly between different runs, and some rare categories occasionally have zero examples in the test set. Which change to the data-splitting strategy will produce the most reliable and consistent evaluation across runs?

Accepted Answer

Use a stratified split that preserves the class distribution of each category in both the training and test sets

Answer

Increase the test set size to 50% so that rare categories are more likely to appear

Answer

Shuffle the dataset more times before performing the random split

Answer

Remove the six rare categories and train only on the two majority categories