Services
Technology Web App Development Mobile App Development Windows App Development Database Development UI/UX Development Cloud Development SEO Digital Marketing Guest Posting Content Writing Artificial Intelligence Data Science ERP Development CRM Development CMS Development LMS Development Business Listing Blockchain Technology Startup Promotion AI Tools Integration
Products
Data Converter Cleaner TUC - Unit Converter Import Export Survey Manager Tweet Controller
Developer
Article Blog Forum Interview Beginner News Startup Quiz
Users
Career
Apply Job Internship
Pricing
Login Sign Up Get a Quote

interview

Home Interviews How do you batch insert/update large data sets efficiently? – MindStick
Ask Question

1 Answers

ICSM Computer
ICSM Computer 08 Jul 2025
Report Abuse

Efficiently batch inserting or updating large datasets in IndexedDB (especially 10,000+ records) requires careful use of transactions, chunking, and avoiding memory bottlenecks. Here's how you can do it efficiently and safely:

1. Use Bulk APIs (bulkAdd, bulkPut) with Dexie.js

If you're using Dexie.js, it provides bulkAdd and bulkPut which are highly optimized.

await db.users.bulkAdd([
  { id: 1, name: "Alice" },
  { id: 2, name: "Bob" },
  // ...
]);
  • bulkAdd() → Inserts new items (fails on existing keys)
  • bulkPut() → Inserts or updates (upsert)
  • Automatically runs in a single transaction (very fast)

2. Chunk the Data (to avoid memory/transaction limits)

IndexedDB transactions have limits (browser- and platform-dependent). Safe chunk size: 500–1000 records per transaction.

async function chunkedInsert(data, chunkSize = 1000) {
  for (let i = 0; i < data.length; i += chunkSize) {
    const chunk = data.slice(i, i + chunkSize);
    await db.users.bulkPut(chunk);
  }
}

⚠️ Without chunking, large sets (e.g. 100k records) may crash or freeze the browser.

3. Use Explicit Transactions (Vanilla IndexedDB)

If not using Dexie:

const tx = db.transaction(['users'], 'readwrite');
const store = tx.objectStore('users');

for (let user of users) {
  store.put(user); // or store.add(user)
}

tx.oncomplete = () => console.log("All inserted");

But this is slower and harder to manage than Dexie.

4. Update Efficiently Using modify() in Dexie

await db.users
  .where('status')
  .equals('inactive')
  .modify(user => {
    user.flagged = true;
  });

Efficient for bulk updating in-place using a predicate condition.

5. Avoid JSON.parse/stringify for Huge Sets

Handling 100K records as a full JSON array in memory is expensive. Process data in streams or generators if source is large (e.g., from file or API).

6. Wrap in Transactions for Speed and Atomicity

Dexie:

await db.transaction('rw', db.users, async () => {
  await db.users.bulkPut(users);
});

Vanilla:

const tx = db.transaction(['users'], 'readwrite');
const store = tx.objectStore('users');
// loop over store.put(...)

Transactions batch writes and are faster than calling put outside a transaction.

7. Parallel vs Sequential Batching

Don’t write all chunks in parallel (can overload IndexedDB).

Prefer:

for (const chunk of chunks) {
  await db.users.bulkPut(chunk);
}

Avoid:

await Promise.all(chunks.map(chunk => db.users.bulkPut(chunk))); // ❌ risk of lock contention

Summary Table

Technique Use When Benefit
bulkPut() (Dexie) Most common bulk insert/update Fast, transactional
Chunking (1000 max) 10K+ records Prevents crashes
modify() Bulk update with filters Fast, no full read
Explicit Transaction Vanilla IndexedDB Atomic, grouped writes
Sequential Chunks Large data sets Prevents lock contention
MindStick Training
Markdown for AI

A clean, structured version of this page for AI assistants and LLMs.

Open .md