Question

A data scientist is training a linear model with gradient descent on a dataset where one feature ranges from 0 to 1 and another ranges from 0 to 1,000,000. During training, the loss oscillates and takes many epochs to converge, and reducing the learning rate slows progress to a crawl without fixing the instability. What is the MOST likely cause and the appropriate remedy?

Accepted Answer

The features are on vastly different scales, causing an elongated, ill-conditioned loss surface; standardize or normalize the features before training.

Answer

The model is overfitting; add L2 regularization and reduce the number of training epochs.

Answer

The training data has too few samples; collect more data before retraining the model.

Answer

The model is too simple for the data; switch to a deep neural network with additional hidden layers.