A practical guide to the OpenAI Batch API: What it is and when to use it

Ever tried to process a huge amount of data with an AI, only to get stopped dead in your tracks by rate limits? It's a common headache. Big AI jobs can be slow, surprisingly expensive, and they often hog the API quota that your real-time, user-facing apps desperately need.

This is exactly the problem the OpenAI Batch API was built to solve. It’s a tool designed for large-scale tasks that aren't time-sensitive. It lets you bundle thousands of requests, send them off in one go, and get them processed asynchronously at a serious discount.

In this guide, we'll walk through what the Batch API is, where it makes the most sense to use it, how to get it working step-by-step, and what its limitations are. By the end, you'll have a clear idea of whether it's the right tool for your next big project.

What is the OpenAI Batch API?

At its core, the OpenAI Batch API lets you package up a ton of API requests into a single file, upload it, and get all the results back within 24 hours. The key here is that it's asynchronous.

A standard API call is synchronous: you send a request and hang on for a response, which usually comes back in seconds. This is great for things like chatbots where you need an immediate answer. The Batch API is different. You send your big job off into the void and then check back later to collect the results.

This trade-off for speed comes with a couple of pretty great perks:

• Big Savings: You get a 50% discount on the standard API price for most models. When you're processing a lot of data, that adds up fast.

• Higher Rate Limits: The Batch API has its own, much larger, rate limit. This means you can run your massive background jobs without slowing down or crashing your main applications.

Here’s a quick breakdown of the differences:

Key benefits and use cases for the OpenAI Batch API

So we know what it is, but when should you actually use it? The perks go beyond just saving money and avoiding rate limits; this API makes some projects possible that would have been a nightmare before.

Seriously cut down your costs

Let's be honest, the 50% discount on input and output tokens is the main event here. If your work involves chewing through millions of tokens for data classification or content creation, that discount can be the difference between a project being wildly expensive and actually affordable. Put it this way: if a job would normally cost you $1,000 in API credits, the Batch API gets it done for $500.

Stop background jobs from crashing your main services

If you’re running an app that your users depend on, the last thing you want is a massive internal data job eating up your API quota and causing slowdowns. Because the Batch API runs on a separate quota, you can let your heavy-lifting tasks run in the background without any risk. It’s like having a dedicated lane on the highway for your big trucks, keeping the main road clear for everyone else.

Ideal scenarios for asynchronous processing

The Batch API is your best friend in any situation where you have a lot of work to do and you don't need the answers right this second. Here are a few common scenarios where it really shines:

• Bulk data processing: Got a year's worth of customer support tickets to categorize? Thousands of legal documents to summarize? A mountain of user feedback to analyze for sentiment? This is the tool for that.

• Offline content generation: Imagine you need to generate 10,000 product descriptions for a new online store or create thousands of personalized email drafts for a marketing campaign. The Batch API can handle these tasks without breaking a sweat.

• Model evaluations: When you're testing out a new prompt or fine-tuning a model, you have to run it against a ton of examples to see how well it performs. The Batch API makes this process consistent and much cheaper.

How to use the OpenAI Batch API: A step-by-step walkthrough

While the Batch API is powerful, it’s not a point-and-click solution. It takes a bit of setup and code to get things moving. Here’s a full walkthrough of how to do it using Python.

Step 1: Prepare your batch file in JSONL format

First things first, you need to create a JSON Lines file (with a ".jsonl" extension). It’s just a plain text file where every single line is a complete JSON object that represents one API request.

Each line in the file needs three specific things:

• "custom_id": A unique ID you make up to keep track of each request. You'll need this later to match the output to your original input, so don't skip it!

• "method": The HTTP method, which for now is always "POST".

• "url": The API endpoint you're calling, like "/v1/chat/completions".

Here’s an example of what one line for a chat completion request would look like:

{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4o-mini", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the capital of France?"}]}}  

Your file will just be a long list of these, one after another, each with its own "custom_id" and prompt.

Step 2: Upload your file

With your file ready, you need to upload it to OpenAI's file storage. You'll use the files API endpoint for this and make sure to tell it the file's purpose is for "batch" processing.

Here's the Python code for that:

from openai import OpenAI  

client = OpenAI()  

batch_input_file = client.files.create(  

  file=open("your_batch_file.jsonl", "rb"),  

  purpose="batch"  

)  

This function will give you back a file object with an ID, which you'll need for the next step.

Step 3: Create and kick off the batch job

Now you can officially create the batch job. You'll use the "input_file_id" you just got and specify the endpoint. The "completion_window" is currently locked at "24h", so that's your only option.

batch_job = client.batches.create(  

  input_file_id=batch_input_file.id,  

  endpoint="/v1/chat/completions",  

  completion_window="24h"  

)  

And just like that, the job is off and running on OpenAI's end.

Step 4: Monitor the job status

Since this is all happening in the background, you'll need to check in on the job's status. It can be "validating", "in_progress", "completed", "failed", or "expired". You can check on it by polling the API with the job's ID.

Here's a simple Python loop that checks the status every 30 seconds:

import time  

while True:  

  batch_job = client.batches.retrieve(batch_job.id)  

  print(f"Job status: {batch_job.status}")  

  if batch_job.status in ["completed", "failed", "cancelled"]:  

    break  

  time.sleep(30)  

Step 5: Download and use your results

Once the job status switches to "completed", the batch object will have an "output_file_id" for the successful requests and an "error_file_id" for any that didn't make it. You can download these files using their IDs.

if batch_job.output_file_id:  

  result_file = client.files.content(batch_job.output_file_id)  

  # Save the content to a local file  

  with open("results.jsonl", "wb") as f:  

    f.write(result_file.content)  

The results file comes back in the same JSONL format. Each line will have the "custom_id" you set up in step one, making it easy to connect each answer to the original question.

Understanding OpenAI Batch API pricing and limitations

The Batch API is a great tool, but it's good to know the costs and trade-offs before you build a whole project around it.

How OpenAI Batch API pricing works

The pricing is refreshingly simple: you pay 50% of the normal rate for whichever model you use. This discount applies to both the input tokens you send and the output tokens you get back.

Here’s a quick look at the savings for a few popular models.

Heads up: Prices are per 1 million tokens. They can change, so it's always smart to check the official OpenAI pricing page for the most current info.

Common limitations and challenges

While the API is powerful, it does come with some strings attached.

• The 24-hour wait: This is the big one. The Batch API is strictly for things that aren't urgent. If you need results in a few minutes or even a couple of hours, this isn't the right tool. Think of the 24-hour window as a deadline, not a loose estimate.

• It requires developer work: Using the Batch API isn't a simple, out-of-the-box experience. It takes a real engineering effort to build and maintain the whole process. Your team will have to write code to create the JSONL files, manage uploads, check job statuses, handle failures, and process the results.

• Troubleshooting can be a pain: When a huge batch job fails, figuring out why can be a headache. The error files aren't always super helpful, which can lead to a lot of trial and error while you burn through time and credits.

• An alternative for support teams: For businesses that want to automate support tasks, like analyzing old Zendesk tickets or creating help articles from Confluence docs, building a custom solution with the Batch API is a pretty heavy lift. A tool like eesel AI is built to handle this stuff for you. It connects to your helpdesk and knowledge bases, learns from your data, and gets you up and running in minutes. You get all the benefits of large-scale AI processing without the months of engineering work.

The bottom line: Is the OpenAI Batch API right for you?

So, what's the verdict? The OpenAI Batch API is a fantastic, money-saving tool for developers who need to run big, non-urgent AI jobs and have the technical team to manage the entire workflow. It’s built for scale and efficiency, as long as you can wait for your results.

The trade-off is pretty clear: you get a huge discount and higher rate limits, but you give up speed and simplicity. If you need answers in real time, or if you don't have developers ready to build and maintain a custom pipeline, the Batch API probably isn't the best fit.

For teams that are specifically looking to automate customer support, a purpose-built platform is a much faster and more direct route. With eesel AI, you can connect your tools, see how an AI agent would perform on thousands of your past tickets, and launch it, all from a simple dashboard.

Ready to see what support automation can do for you?

Try eesel AI for free and find out how quickly you can start reducing your ticket queue and freeing up your team.