---
title: "How do you batch insert/update large data sets efficiently?"  
description: "How do you batch insert/update large data sets efficiently?"  
author: "ICSM Computer"  
published: 2025-07-08  
updated: 2025-07-08  
canonical: https://www.mindstick.com/interview/34327/how-do-you-batch-insert-update-large-data-sets-efficiently  
category: "IndexedDB"  
tags: ["indexeddb"]  
reading_time: 5 minutes  

---

# How do you batch insert/update large data sets efficiently?

Efficiently **batch inserting or updating large datasets** in IndexedDB (especially 10,000+ records) requires careful use of transactions, chunking, and avoiding memory bottlenecks. Here's how you can do it **efficiently and safely**:

## 1. Use Bulk APIs (`bulkAdd`, `bulkPut`) with Dexie.js

If you're using **Dexie.js**, it provides `bulkAdd` and `bulkPut` which are highly optimized.

```javascript
await db.users.bulkAdd([
  { id: 1, name: "Alice" },
  { id: 2, name: "Bob" },
  // ...
]);
```

- `bulkAdd()` → Inserts new items (fails on existing keys)
- `bulkPut()` → Inserts or updates (upsert)
- Automatically runs in a single transaction (very fast)

## 2. Chunk the Data (to avoid memory/transaction limits)

IndexedDB transactions have limits (browser- and platform-dependent). Safe chunk size: **500–1000 records** per transaction.

```javascript
async function chunkedInsert(data, chunkSize = 1000) {
  for (let i = 0; i < data.length; i += chunkSize) {
    const chunk = data.slice(i, i + chunkSize);
    await db.users.bulkPut(chunk);
  }
}
```

⚠️ Without chunking, large sets (e.g. 100k records) may crash or freeze the browser.

## 3. Use Explicit Transactions (Vanilla IndexedDB)

If not using Dexie:

```javascript
const tx = db.transaction(['users'], 'readwrite');
const store = tx.objectStore('users');

for (let user of users) {
  store.put(user); // or store.add(user)
}

tx.oncomplete = () => console.log("All inserted");
```

But this is **slower and harder to manage** than Dexie.

## 4. Update Efficiently Using `modify()` in Dexie

```javascript
await db.users
  .where('status')
  .equals('inactive')
  .modify(user => {
    user.flagged = true;
  });
```

Efficient for **bulk updating in-place** using a predicate condition.

## 5. Avoid JSON.parse/stringify for Huge Sets

Handling 100K records as a full JSON array in memory is expensive. Process data in **streams or generators** if source is large (e.g., from file or API).

## 6. Wrap in Transactions for Speed and Atomicity

## Dexie:

```javascript
await db.transaction('rw', db.users, async () => {
  await db.users.bulkPut(users);
});
```

## Vanilla:

```javascript
const tx = db.transaction(['users'], 'readwrite');
const store = tx.objectStore('users');
// loop over store.put(...)
```

Transactions batch writes and are faster than calling `put` outside a transaction.

## 7. Parallel vs Sequential Batching

Don’t write all chunks in parallel (can overload IndexedDB).

## Prefer:

```javascript
for (const chunk of chunks) {
  await db.users.bulkPut(chunk);
}
```

## Avoid:

```javascript
await Promise.all(chunks.map(chunk => db.users.bulkPut(chunk))); // ❌ risk of lock contention
```

## Summary Table

| **Technique** | **Use When** | **Benefit** |
| --- | --- | --- |
| `bulkPut()` (Dexie) | Most common bulk insert/update | Fast, transactional |
| Chunking (1000 max) | 10K+ records | Prevents crashes |
| `modify()` | Bulk update with filters | Fast, no full read |
| Explicit Transaction | Vanilla IndexedDB | Atomic, grouped writes |
| Sequential Chunks | Large data sets | Prevents lock contention |

## Answers

### Answer by ICSM Computer

Efficiently **batch inserting or updating large datasets** in IndexedDB (especially 10,000+ records) requires careful use of transactions, chunking, and avoiding memory bottlenecks. Here's how you can do it **efficiently and safely**:

## 1. Use Bulk APIs (`bulkAdd`, `bulkPut`) with Dexie.js

If you're using **Dexie.js**, it provides `bulkAdd` and `bulkPut` which are highly optimized.

```javascript
await db.users.bulkAdd([
  { id: 1, name: "Alice" },
  { id: 2, name: "Bob" },
  // ...
]);
```

- `bulkAdd()` → Inserts new items (fails on existing keys)
- `bulkPut()` → Inserts or updates (upsert)
- Automatically runs in a single transaction (very fast)

## 2. Chunk the Data (to avoid memory/transaction limits)

IndexedDB transactions have limits (browser- and platform-dependent). Safe chunk size: **500–1000 records** per transaction.

```javascript
async function chunkedInsert(data, chunkSize = 1000) {
  for (let i = 0; i < data.length; i += chunkSize) {
    const chunk = data.slice(i, i + chunkSize);
    await db.users.bulkPut(chunk);
  }
}
```

⚠️ Without chunking, large sets (e.g. 100k records) may crash or freeze the browser.

## 3. Use Explicit Transactions (Vanilla IndexedDB)

If not using Dexie:

```javascript
const tx = db.transaction(['users'], 'readwrite');
const store = tx.objectStore('users');

for (let user of users) {
  store.put(user); // or store.add(user)
}

tx.oncomplete = () => console.log("All inserted");
```

But this is **slower and harder to manage** than Dexie.

## 4. Update Efficiently Using `modify()` in Dexie

```javascript
await db.users
  .where('status')
  .equals('inactive')
  .modify(user => {
    user.flagged = true;
  });
```

Efficient for **bulk updating in-place** using a predicate condition.

## 5. Avoid JSON.parse/stringify for Huge Sets

Handling 100K records as a full JSON array in memory is expensive. Process data in **streams or generators** if source is large (e.g., from file or API).

## 6. Wrap in Transactions for Speed and Atomicity

## Dexie:

```javascript
await db.transaction('rw', db.users, async () => {
  await db.users.bulkPut(users);
});
```

## Vanilla:

```javascript
const tx = db.transaction(['users'], 'readwrite');
const store = tx.objectStore('users');
// loop over store.put(...)
```

Transactions batch writes and are faster than calling `put` outside a transaction.

## 7. Parallel vs Sequential Batching

Don’t write all chunks in parallel (can overload IndexedDB).

## Prefer:

```javascript
for (const chunk of chunks) {
  await db.users.bulkPut(chunk);
}
```

## Avoid:

```javascript
await Promise.all(chunks.map(chunk => db.users.bulkPut(chunk))); // ❌ risk of lock contention
```

## Summary Table

| **Technique** | **Use When** | **Benefit** |
| --- | --- | --- |
| `bulkPut()` (Dexie) | Most common bulk insert/update | Fast, transactional |
| Chunking (1000 max) | 10K+ records | Prevents crashes |
| `modify()` | Bulk update with filters | Fast, no full read |
| Explicit Transaction | Vanilla IndexedDB | Atomic, grouped writes |
| Sequential Chunks | Large data sets | Prevents lock contention |


---

Original Source: https://www.mindstick.com/interview/34327/how-do-you-batch-insert-update-large-data-sets-efficiently

Copyright © MindStick Software Pvt. Ltd. This Markdown version is provided for developers, AI systems, and offline reading.