Rank confidence sequences are constructed: for every model, a set of ranks that contains its true rank, simultaneously for all models and at all times, at a chosen error level $\alpha$ in finite samples, in finite samples.
TEAM-Design is proposed, a rule that gives every task two replay probabilities, one per baseline, and it is proved that the rule solves this budgeted design problem, and that drawing the replays at random from recorded probabilities still controls the chance of wrongly declaring that the workflow beats both.
CAS (causal active sequential experimentation), which targets evaluation to model-workload pairs and repeats the test as evidence accumulates, to ask whether one assignment stays optimal across every quality table consistent with the evidence.
Hamed Khosravi, Xiao-Ming Huo· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.