| |||||||||||
|
Evals Researcher, Rewards and Environments
Location: US-CA-San Francisco Jobcode: 975186811 Share Job | Report Job
About the Role: Evals Researcher, Rewards and Environments, San Francisco (hybrid) Post-training and evaluation lab, profitable at four months, San Francisco You would build the evaluation systems that decide which data is worth training on and whether the models and agents are actually getting better. About the Company Our company builds theevaluation environments and post-training infrastructure that make long-horizon enterprise agents trainable, then uses that stack to train and deploy custom models for very large technology companies and government customers. Under four months in, ten people, already profitable, on angel and institutional money and no priced round. What you would do: - Rank training value before compute is spent: work out which tasks in a dataset are worth training on and build a system that ranks every task by expected training value. - Build scoring that runs constantly and cheaply, tuned to each customer's definition of a good outcome rather than generic correctness. - Catch drift before it is a problem: notice when real customer requests have moved far enough from the test set that it no longer describes the job, and rebuild the evals to match. - Design rewards: checkers a model cannot game, model-as-judge and rubric graders, and evaluation of long-horizon agent trajectories. - Sit at the centre of the post-training loop: decide before compute is spent, and say afterwards whether a change actually helped. Hard requirements from the JD
Fill in the form, we will contact you...
|