Question

Your team deployed an INT8 post-training quantized version of a sentiment classification model to reduce serving latency and cost. After deployment, accuracy dropped from 94% to 87%, which is below the acceptable 92% threshold, while the FP32 model met the threshold. You want to keep the smaller INT8 footprint and low latency benefits but recover most of the lost accuracy. What is the most effective next step?

Accepted Answer

Retrain the model using quantization-aware training (QAT) so the model learns to compensate for quantization effects during training, then re-export to INT8.

Answer

Revert to the FP32 model permanently and abandon quantization, since INT8 inherently cannot preserve accuracy for text models.

Answer

Increase the number of serving replicas so more compute is available to offset the accuracy loss from quantization.

Answer

Apply post-training FP16 quantization instead, since FP16 always yields the same latency and cost profile as INT8.