Question

A data scientist is training a deep neural network on a sparse text-classification dataset using SageMaker. Training with plain stochastic gradient descent (SGD) at a fixed learning rate converges very slowly, and different features clearly need different effective learning rates because some tokens appear frequently while others are rare. The team wants an optimizer that adapts the per-parameter learning rate automatically to speed up convergence without extensive manual learning-rate tuning. Which optimizer change best addresses this need?

Accepted Answer

Switch to the Adam optimizer, which maintains per-parameter adaptive learning rates using running estimates of first and second moments of the gradients.

Answer

Keep SGD but increase the global fixed learning rate by 10x to force faster convergence across all parameters.

Answer

Keep SGD and add L2 weight decay to the loss so that rare-feature weights receive larger updates.

Answer

Switch to plain batch gradient descent computed over the full dataset each step to smooth the gradient signal.