Question

A data scientist is training a deep neural network with a fixed learning rate using a SageMaker training job. Training loss decreases quickly at first and then oscillates around a value without improving further for many epochs. The validation loss also stops improving at the same plateau. Which adjustment is MOST likely to help the model continue improving beyond this plateau?

Accepted Answer

Apply a learning rate schedule that decays the rate when the loss stops improving (e.g., reduce-on-plateau)

Answer

Add L2 regularization to the loss function to penalize large weights

Answer

Increase the batch size to smooth out the gradient estimates

Answer

Reduce the number of hidden layers to lower model capacity