Together AI adds endpoint-level A/B testing for production models

Together AI has added production A/B experiments that split an endpoint’s live traffic between a single control deployment and one or more model variants. Published August 17, the company’s production testing guide says teams can compare model behavior with real users after shadow validation while keeping experiment routing out of application code.

That routing sits behind the existing endpoint, so clients keep calling the same endpoint name, API and credentials. The controls belong to Together AI’s wider production inference platform, which also covers infrastructure arrangements such as its Together AI inference deployment with IBM.

Shadow traffic surfaces operational issues without showing candidate responses to users. A production A/B experiment goes further, putting those responses in front of a controlled cohort so teams can measure product outcomes.

Each experiment holds exactly one control and as many as 20 variants, according to Together AI. Traffic shares are integers that must total 100, and they stay independent of replica counts. The control remains part of the endpoint’s base traffic split; experiment variants must carry zero base weight, so the experiment alone determines their exposure.

Requests that carry a stable sampling key, such as a user identifier, can keep the same cohort assignment. Without that key, assignment is effectively random for each request. Responses identify the deployment that served them, Together AI says, so customers can pair deployment-level latency, error and throughput data with their own signals such as ratings, retries or task completion. The materials do not say the platform calculates application-specific success metrics or statistical significance.

In a company-run test at three requests per second, Together AI reported control-to-variant splits of 95.3% to 4.7% across 1,330 requests, 79.2% to 20.8% across 1,348 requests, and 50.2% to 49.8% across another 1,348 requests. Those results were not independently reproduced. The company said it waited about 75 seconds after updates before measuring, and that routing changes propagate on roughly the same 30-to-60-second timescale as traffic-split changes.

Changing exposure means replacing the full experiment member list, and Together AI says etag checks reject stale concurrent updates rather than silently overwriting them. Its documented promotion path runs a blue-green rollout from the control to the selected variant, then deletes the experiment. Deletion removes experiment routing and returns traffic control to the endpoint’s base split; in the company test, 360 consecutive requests reached the control afterward.