OpenAI Batch API: buying the catalogue update instead of the GPU hour

A catalogue has 100,000 descriptions waiting for cleaner categories and short summaries. Nobody needs the answer to the first record in three seconds. They need a complete, usable dataset before the next publishing cycle. That distinction changes the compute purchase.

OpenAI’s Batch API packages inference around deferred delivery. The supplier’s April 23, 2024 enterprise announcement described a 50% discount for asynchronous workloads, including classification and evaluation. This is an established service, not a fresh November launch. The useful question is what a builder must do to turn that discount into a dependable production result. S1

A workload you can picture

OpenAI’s cookbook, dated April 24, 2024, demonstrates extracting movie categories and one-sentence summaries from descriptions. It also includes image captioning. The page is now explicitly marked archived, so treat it as a concrete workflow illustration rather than a current implementation recipe or performance benchmark. S2

The catalogue example here is PCT’s own planning case. Assume the operator has rights to process the text and a fixed category vocabulary. Each record can be processed independently. The output should contain the record identifier, permitted categories and a short description supported by the input. No production catalogue or customer order was inspected for this article.

This is attractive builder territory because the job is easy to make visible. You can count the records, inspect samples, test the category rules and decide whether the result is good enough before committing the whole collection. A cheaper run that produces inconsistent labels still leaves an expensive cleanup job.

A batch needs a receipt

The current guide permits up to 50,000 requests per batch and a maximum 200 MB input file. It also imposes model-specific queued-input-token limits. Our 100,000-record case therefore needs at least two batches by request count alone; file size and account limits may require more. Splitting the work does not create extra quota.

Outputs may arrive in a different order from inputs. The guide tells developers to join them using custom identifiers, and failed requests have a separate error file. A practical receipt should reconcile every submitted identifier with a successful response, an error or unresolved work. S3

That receipt is valuable beyond avoiding a spreadsheet accident. It lets the operator rerun only the records needing attention and preserve accepted results. Replaying an entire collection because one slice failed is an avoidable cost, and can produce inconsistent versions of records that were already usable.

PCT would also keep the prompt version, category vocabulary, chosen model version and acceptance checks beside the output. A clean audit trail makes it possible to explain why a record changed when the next catalogue refresh arrives.

Put numbers on the choice

Here is illustrative arithmetic, not an OpenAI quote. Assume 100,000 requests, each with 2,000 input tokens and 200 output tokens. That is 200 million input tokens and 20 million output tokens. Suppose the eligible synchronous route costs $1 per million input tokens and $4 per million output tokens. Its token bill would be $280: $200 for input and $80 for output.

Apply the documented 50% Batch discount to those assumed rates and the token estimate becomes $140. The gross saving is $140, before taxes, preparation, evaluation, storage, retries or engineering work. The token counts include the prompt, not just the catalogue description, and no cache discount or tool use is assumed. Actual model selection still requires quality and eligibility checks. S4

Now suppose this particular implementation requires three additional staff hours at an assumed internal cost of $75 an hour. That adds $225. For one run, the batch workflow costs $365 against the $280 baseline under these assumptions. It has not paid back its incremental setup effort. After two equivalent runs, token savings of $280 exceed that one-time cost by $55, assuming no extra ongoing overhead.

There is a good commercial lesson here. A 50% token discount does not mean a 50% reduction in project cost. Repeat frequency, existing tooling and the amount of review required can matter more than the posted rate. An established batch pipeline may need little additional setup; a one-off migration may need substantially more than this example.

The deadline and retention tests

The current FAQ specifies a 24-hour processing window, but also describes expiration: unfinished work is canceled, completed results are returned and completed work is charged. Treat partial completion as an outcome the application must handle. Submitting at the last possible moment leaves no room for correction or another attempt. S4

For a publishing pipeline, PCT would choose an internal submission cutoff earlier than the actual deadline and define what happens if some records remain missing. The operator might retain the previous accepted description or hold those records out of the release. That is a product decision to make before the run, not an emergency policy to improvise afterward.

Data policy can disqualify the route entirely. OpenAI’s FAQ states that zero data retention does not apply to Batch, even for organizations with that setting enabled. Inputs and outputs follow the applicable retention policies. Buyers with strict requirements must verify the actual service terms and configuration before uploading anything. S4

What the market can learn

This service sells a different delivery condition for model inference. It does not give the buyer a transferable reservation on named GPUs, and the published discount does not reveal a market-clearing price for physical capacity. PCT’s interpretation is narrower: some workloads can trade immediate response for lower token costs without managing their own training checkpoints or GPU fleet.

The worthwhile experiment is a small, rights-cleared sample with explicit acceptance criteria and a complete result ledger. Measure accepted records, elapsed time, review effort and actual charges. The opportunity is real enough to test. Whether it is economical for a specific catalogue remains unproven until those measurements replace the assumptions.