High-Throughput Parallel Fleet Scraping

How AI agents scale data harvesting using two-dimensional parallelism: spreading workloads across N Chrome profiles and M concurrent tabs with fault-tolerant state persistence.

1. The Two-Dimensional Fan-Out Architecture

Standard browser automations lock execution to a single active tab or single profile. In WebBridge, every tab maintains an independent CDP connection through the MV3 extension:

Fleet Concurrency = (N Profiles) × (M Tabs per Profile)

Example Configuration:
  • Profile 1 ("Work"):     Tab 1 (s1) ║ Tab 2 (s1) ║ Tab 3 (s1) ║ Tab 4 (s1)
  • Profile 2 ("Personal"): Tab 1 (s2) ║ Tab 2 (s2) ║ Tab 3 (s2) ║ Tab 4 (s2)
  • Profile 3 ("Research"): Tab 1 (s3) ║ Tab 2 (s3) ║ Tab 3 (s3) ║ Tab 4 (s3)
  Total Concurrency: 12 live pages evaluating in parallel

2. The Resumable .jsonl Ledger Pattern

When running large scraping jobs, never buffer results in memory. Append output incrementally to a JSON Lines (.jsonl) file after every page evaluation. If the process is paused or aborted, resume directly from the remaining items:

import fs from 'node:fs';
import { WebBridgeClient } from 'agent-webbridge';

// Read previously processed URLs from disk
function loadDoneUrls(logFile) {
  if (!fs.existsSync(logFile)) return new Set();
  const lines = fs.readFileSync(logFile, 'utf8').trim().split('\n');
  return new Set(lines.map(line => {
    try { return JSON.parse(line).url; } catch { return null; }
  }).filter(Boolean));
}

// Write result incrementally
function appendResult(logFile, data) {
  fs.appendFileSync(logFile, JSON.stringify(data) + '\n');
}

3. Production Implementation (Node.js)

import { WebBridgeClient } from "agent-webbridge";

const TARGET_URLS = [
  "https://news.ycombinator.com/item?id=39000001",
  "https://news.ycombinator.com/item?id=39000002",
  "https://news.ycombinator.com/item?id=39000003",
  "https://news.ycombinator.com/item?id=39000004",
];

async function harvestUrl(client, url) {
  // Open tab specifically inside this session
  const nav = await client.navigate(url, { newTab: true });
  
  // Extract data using in-page evaluate
  const result = await client.evaluate(`(() => {
    const title = document.querySelector('.titleline a')?.innerText || '';
    const score = document.querySelector('.score')?.innerText || '0';
    return { title, score };
  })()`);

  // Close tab immediately after extraction to keep memory clean
  await client.closeTab();

  return { url, ...result };
}

async function runFleet() {
  const client = new WebBridgeClient({ profile: "Work", session: "harvest-job" });

  // Concurrently process in batches of 4
  const CONCURRENCY = 4;
  for (let i = 0; i < TARGET_URLS.length; i += CONCURRENCY) {
    const chunk = TARGET_URLS.slice(i, i + CONCURRENCY);
    const results = await Promise.all(chunk.map(url => harvestUrl(client, url)));
    console.log(`Processed batch of ${results.length} items`);
  }

  // Cleanup tab group
  await client.closeSession();
}

runFleet().catch(console.error);

4. Defensive Egress IP Pacing

Because all local Chrome profiles share the machine's egress IP address:

• Distinct Domains: Fan-out delivers maximum throughput when scraping across different platforms or separate domains.
• Single Domain: When targeting a single rate-limited host (e.g., Google or LinkedIn), keep concurrency to 2–3 tabs and insert random jitters (500ms–1500ms) to avoid triggering IP-level CAPTCHA challenges.