n8n Split in Batches: Processing Large Datasets Efficiently

Why Your n8n Workflow Chokes on Large Datasets

You built a workflow that pulls three thousand rows from a database, enriches each one with an API call, and writes the result to a CRM. It works perfectly in testing with ten rows. Then you run it against the real dataset and it either times out, gets rate-limited by the third-party API, or silently drops items somewhere in the middle. If this sounds familiar, you have run into the single most common scaling problem in n8n: trying to process everything in one giant pass instead of breaking it into batches.

n8n, by default, is eager. When a node receives an array of a thousand items, it tries to process all of them in the same execution cycle, as fast as the node and the downstream API allow. That is fantastic for small payloads and terrible for anything at production scale. APIs impose rate limits. Servers have memory ceilings. Webhooks and HTTP requests have timeout windows. When you push too much data through a single node execution, you hit all three walls at once.

The fix is batching: processing your dataset in smaller, controlled chunks, one batch at a time, with pauses, retries, and checkpoints built in. n8n has a dedicated node for exactly this purpose, historically called Split in Batches and now labeled Loop Over Items in the newer editor UI. Whether you see one name or the other in your instance, the underlying mechanism is identical, and understanding it well is one of the highest-leverage skills you can build as an n8n automation engineer. This article walks through how the node actually works, why naive loops fail, and how to design batch-processing workflows that hold up under real-world data volume.

What the Split in Batches Node Actually Does

The Split in Batches node (renamed Loop Over Items in recent versions) takes an incoming array of items and divides it into groups of a fixed size that you specify. Instead of handing your entire dataset to the next node in one shot, it releases one batch at a time, waits for that batch to finish flowing through the rest of the workflow, and then pulls the next batch on the following loop iteration.

Structurally, the node has two outputs:

• Loop output: fires for every batch as long as items remain to be processed. You connect this to the nodes that should run per batch.
• Done output: fires exactly once, after the very last batch has been processed. You connect this to whatever should happen after the entire dataset is finished, like a summary email or a final database write.

The key architectural pattern is that you connect the last node in your per-batch processing chain back to the Split in Batches node itself, forming a loop. Each time execution returns to the node, it checks whether there are more items left in the queue. If yes, it emits the next batch through the Loop output. If no, it emits through Done and the loop terminates.

This is fundamentally different from a for loop in a traditional programming language, where iteration happens within a single function call and the stack frame persists. In n8n, each loop pass is a distinct execution path through the graph, which means every node in the loop actually runs once per batch, with all of n8n's usual node execution behavior (retries, error handling, logging) applying independently to each pass.

Setting Up Your First Batching Workflow

Let's build a concrete example: fetching 500 customer records from a database, enriching each with a call to an external validation API, and writing results back. Here is the skeleton structure.

1. Trigger node (Manual Trigger or Schedule Trigger) starts the workflow.
2. Database node (Postgres, MySQL, or similar) fetches all 500 rows in a single query.
3. Split in Batches node receives the 500 items and is configured with a batch size, say 10.
4. HTTP Request node calls the external API, but now only receives 10 items per execution pass instead of 500.
5. Set node or database write node processes the enriched result for that batch.
6. A connection loops back from the last node to the Split in Batches node, closing the cycle.
7. Done output connects to a final node, like a Slack notification saying "all 500 records processed."

The critical configuration is the Batch Size parameter on the node itself. This single number controls how many items are released per loop iteration. Getting this number right is mostly a matter of understanding your bottleneck, which we'll cover in the next section.

Here's what the relevant part of the node's JSON configuration looks like when you export the workflow:

{
  "parameters": {
    "batchSize": 10,
    "options": {}
  },
  "name": "Loop Over Items",
  "type": "n8n-nodes-base.splitInBatches",
  "typeVersion": 3,
  "position": [680, 300]
}

Note the type field still references splitInBatches internally even in versions where the UI label reads "Loop Over Items" — this is a naming change at the display layer, not a new node type, so older workflow JSON exports remain fully compatible.

Choosing the Right Batch Size

There is no universal correct batch size. It depends entirely on what you are bottlenecked by. Work through these questions before you pick a number:

• What is the API rate limit? If a third-party service allows 60 requests per minute and your HTTP Request node fires once per item, a batch size of 10 with a short delay between batches keeps you comfortably under the limit. A batch size of 100 will get you throttled or banned.
• How much memory does each item consume? If you're processing large binary files (PDFs, images, CSV exports) rather than lightweight JSON rows, smaller batches (2 to 5) prevent n8n's execution memory from ballooning and crashing the workflow.
• What is your timeout window? Self-hosted n8n instances have configurable execution timeouts, but cloud plans and webhook-triggered workflows often have hard ceilings. If each item takes 2 seconds to process and your timeout is 300 seconds, you need to keep total batch processing time well under that limit, which means smaller batches with more loop iterations rather than one enormous batch.
• Does the downstream system support bulk operations? If you're writing to a database that accepts bulk inserts, a larger batch size (50 to 200) is often more efficient because you reduce the number of round trips. If you're calling a REST API that only accepts one record per request, the batch size mostly controls concurrency and pacing rather than throughput.

A practical starting point for API-bound workflows is a batch size between 5 and 20, combined with a Wait node inserted into the loop to throttle pacing. For database-bound workflows without external rate limits, batch sizes of 50 to 100 usually perform well. Always start conservative and increase gradually while watching your execution logs for errors or slowdowns.

Combining Split in Batches with Rate Limiting

Batching solves the "how much data per cycle" problem, but it does not automatically solve pacing. If your batch size is 10 and each loop iteration completes in half a second, you can still fire off enough requests per minute to trip an API's rate limiter. This is where the Wait node becomes essential.

Insert a Wait node inside the loop, right before or after your HTTP Request node, configured to pause for a fixed duration (say, 2 seconds) or until a specific condition is met. This turns your batch loop into a throttled pipeline: process a batch, pause, process the next batch, pause again, until the Done output fires.

// Example: calculating a safe delay based on API rate limit
// If limit is 60 requests/minute and batchSize is 10:
const requestsPerMinute = 60;
const batchSize = 10;
const batchesPerMinute = requestsPerMinute / batchSize;
const delaySeconds = 60 / batchesPerMinute;
// delaySeconds = 10 -> wait 10 seconds between each batch

You can compute this delay dynamically in a Code node upstream of the Wait node, then pass it in as an expression, so that if the API's rate limit ever changes, you only need to update one variable rather than manually recalculating pacing throughout the workflow.

For APIs that return explicit rate-limit headers (like X-RateLimit-Remaining or Retry-After), a more robust pattern is to check those headers after each HTTP Request call inside an IF node, and route to a longer Wait period only when you're close to the limit, rather than always pausing the same fixed amount regardless of actual usage. This keeps your workflow running as fast as safely possible instead of always assuming the worst case.

Handling Errors Inside a Batch Loop

Long-running batch workflows are exactly where you cannot afford to let one bad record kill the entire run. If item 340 out of 500 has malformed data and throws an error in your HTTP Request node, a naive workflow stops dead, and you have no clean record of what succeeded and what didn't.

The fix is to combine batching with n8n's built-in error handling settings on individual nodes:

• Enable Continue On Fail (sometimes labeled "Continue on Error" depending on version) on nodes inside the loop that call external services. This lets the workflow keep running even if that specific node throws, and the error gets passed downstream as part of the item's data instead of halting execution.
• Add an IF node immediately after the risky node to check whether the response contains an error field, and route failed items to a separate branch, typically one that logs the failure to a Google Sheet, database table, or Slack channel.
• Consider a retry counter pattern: attach a field to each item tracking how many times it has been retried, and if a batch fails, route it back through the same HTTP Request node up to a maximum of 2 or 3 attempts before giving up and logging it as a permanent failure.

This is the difference between a workflow that is merely functional and one that is genuinely production-grade. Anyone can build a happy-path automation. The real engineering work is deciding what happens when the 340th record out of 10,000 doesn't behave the way you expected, and making sure the other 9,999 keep processing regardless.

Split in Batches vs. Native Node Batching

It's worth clarifying a point of confusion that trips up a lot of people new to n8n: some nodes have their own built-in batching parameters, separate from the Split in Batches node entirely. The HTTP Request node, for instance, has a Batching option under its settings that controls how many requests fire in parallel and how long to wait between them, without requiring a separate loop node at all.

So when should you use the dedicated Split in Batches node versus a node's native batching settings?

• Use a node's native batching option (when available) if your entire workflow is just "call this one API for every item in the list," since it requires less workflow wiring and n8n handles the looping internally.
• Use the Split in Batches node when you need multiple nodes to execute per batch, when you need custom logic between batches (like the Wait node pattern above, or conditional branching), or when you need to aggregate results across batches before moving to the next stage.
• Use the Split in Batches node when you need fine control over loop state, such as tracking a running total, checking a counter against a threshold, or making decisions based on how many batches have already completed. The node exposes context data about the current loop that you can reference in a Code node.

In practice, most non-trivial production workflows end up using the dedicated Loop Over Items node because real-world processing rarely involves just a single API call per item. There is almost always a transformation step, a validation step, or a conditional branch in between.

Monitoring and Debugging Batch Workflows

Batch loops that run for a long time create a specific debugging challenge: when something goes wrong, you need to know which batch and which item caused it, not just that "the workflow failed." A few practices make this much easier:

• Log batch progress explicitly. Add a Code node or a lightweight HTTP call to an external logging service (or even just a Google Sheets append) at the start of each loop pass, writing out the current batch number and item count. When you're debugging a failure at 2 AM, this tells you exactly how far the workflow got.
• Use the execution list generously. n8n's execution history shows you each run, and for looped workflows, you can inspect the data at each node for each pass. Get comfortable clicking into individual node executions rather than assuming the aggregate result tells the whole story.
• Name your loop nodes descriptively. "Loop Over Items" as a default name gets confusing fast if your workflow has three separate loops for three separate datasets. Rename each instance to something like "Batch: Enrich Customers" or "Batch: Validate Emails" so your execution logs are actually readable.
• Add a final reconciliation step. After the Done output fires, compare the count of items you started with against the count of items that successfully completed, and flag any discrepancy. This catches silent failures that Continue On Fail might otherwise hide from you.

Building this kind of observability into your batch workflows from the start saves enormous amounts of time later, especially once these workflows are running unattended on a schedule and nobody is watching them execute in real time.

Common Mistakes to Avoid

A few patterns show up repeatedly when people are new to batching in n8n, and they're worth calling out directly so you can avoid them:

• Forgetting to loop back to the Split in Batches node. If you don't wire the last node in your per-batch chain back to the loop node, it will only ever process the first batch and then stop, which looks like a bug in the node itself but is actually just a missing connection.
• Setting batch size to 1 by default "to be safe." This works, but it's often needlessly slow. If your bottleneck is API rate limits and the limit allows 20 requests per minute, processing one item at a time with a wait after each one wastes most of your available throughput. Match the batch size to your actual constraint.
• Putting the trigger's full dataset fetch inside the loop. The data-fetching node (your database query, your initial API pull) should happen once, before the Split in Batches node, not inside the loop itself. If you accidentally put it inside the loop, you'll re-fetch the entire dataset on every single batch pass, which is both wasteful and can produce inconsistent results if the underlying data changes mid-run.
• Ignoring the Done output entirely. It's tempting to just let the workflow "finish" whenever the loop stops, but explicitly wiring the Done output to a summary or notification step gives you a clear, reliable signal that the entire dataset was processed, rather than inferring it from the absence of further activity.
• Not testing with a subset first. Before running a batch workflow against 10,000 real records, test it against 20 to 30 representative rows, including a few edge cases like empty fields or unusual characters. This catches most configuration mistakes cheaply, before they cost you API quota or processing time against the full dataset.

Wrapping Up

Split in Batches, now labeled Loop Over Items in current versions of n8n, is one of the most important nodes to master if you plan to build automations that operate on real-world data volumes rather than toy examples. The core idea is simple: stop trying to process everything at once, and instead release your dataset in controlled, sized chunks that respect the rate limits, memory ceilings, and timeout windows of every system your workflow touches. Combine it with a Wait node for pacing, proper error handling for resilience, and clear logging for observability, and you have a pattern that scales from a few hundred records to millions without falling over.

The gap between a workflow that works in a demo and one that survives contact with production data is almost always about exactly this kind of detail: batching, pacing, and error recovery. These are the skills that separate someone who can drag a few nodes onto a canvas from someone who can be trusted to automate a business-critical process unattended.

If you want to go deeper into building resilient, production-grade automations like this, including how batching interacts with AI agent workflows, memory, and tool calls inside n8n, check out the n8n AI Agent Tutorial course on teachyou.ai, where we build these patterns from scratch on real datasets rather than toy demos.