One Size Does Not Fit All: Setting Inference Depth from the Questions a Deployment Actually Asks
It is found that only customizing the threshold for exiting early is worthwhile: calibrating it per deployment raised exit rates by up to 59 percentage points across three models, and by more than 10 points on most corpora tested, while other techniques showed far less impact.