Browser-Extension Fallback and No-Scrape Parity for County Data Collection

Full report

On this page

Core architecture

The cleanest way to design this workflow is to treat browser-extension capture as a transport variant of the same logical source, not as a different source. In practice, that means source_key should identify the business source such as “county civil docket detail” or “county assessor parcel detail,” while transport_kind identifies how evidence was collected: direct_scraper, browser_extension_dom, or browser_extension_network. That separation lets a blocked direct source route to a same-key fallback without changing downstream normalization, validation, or report assembly. JSON Schema is a good fit for both the evidence envelope and the final normalized report because it is specifically intended to enforce JSON consistency, validity, and interoperability.

A Chromium MV3 extension is a good technical base for this because content scripts can read the DOM of the active page, the extension can pass JSON-serializable messages between content scripts and the service worker, and activeTab grants temporary access only after an explicit user action. That permission model is especially useful for an operator-driven workflow because it narrows host access to the page the operator actively chose, instead of giving the extension broad standing access to every site all the time.

For reliability, the extension should behave like a task-bound capture instrument, not like a generic page saver. Each lead run should create one or more expected source tasks, each task should bind to a county, a source_key, an expected URL template, and required normalized outputs, and the extension should only allow submission when the current page satisfies those constraints. The page binding should use documentId from chrome.webNavigation, because Chrome explicitly distinguishes a frame from the document it hosts and notes that documentId changes when a frame navigates to a new document. That is a much more precise guard against stale or wrong-page captures than tabId alone.

The recommended collection stack is therefore a layered one. Use content scripts for visible DOM, text, and page metadata; use MAIN-world injection only where you must read page-owned JS state or intercept in-page fetch/XHR behavior; use chrome.debugger plus the Chrome DevTools Protocol only for counties where the normalized result depends on structured network responses that are not safely or reliably recoverable from the DOM. This distinction matters because chrome.webRequest exposes request lifecycle events and response headers, but not response bodies, while the DevTools Protocol Network domain explicitly exposes request bodies and response bodies.

One design choice should be avoided as a primary data path: chrome.devtools.network. Chrome’s own docs say it reads what is shown in the DevTools Network panel, and if DevTools is opened after the page loads, some requests may already be missing. MDN also notes that the listener only starts firing after the browser’s network panel has been activated at least once. That makes it useful for troubleshooting or support, but too fragile for a production capture pipeline that has to be deterministic and auditable.

Browser-extension data contract

The extension contract should use two schemas. The first is an evidence envelope for transport-specific capture data. The second is the final normalized lead report schema shared by both direct and extension paths. This keeps transport noise out of the business object while still preserving raw evidence and provenance. Use RFC 3339 UTC timestamps everywhere, because RFC 3339 is the Internet timestamp profile built for interoperable protocol events.

Because MV3 service workers are event-driven and extension state should survive worker restarts, do not treat service-worker globals as authoritative run state. Chrome recommends extension storage for service-worker use, and storage.session is explicitly in-memory and cleared on restart, disable, reload, or update. That makes it appropriate for ephemeral “current run” state, but not for durable evidence. For larger capture bundles such as full HTML, response bodies, screenshots, or HAR-like objects, use IndexedDB, which MDN describes as a client-side store for significant amounts of structured data, including files and blobs. Also avoid ordinary web storage for core data because Chrome does not recommend it for extension workflows and notes that service workers cannot use it.

The evidence envelope should be canonicalized and hashed. The practical pattern is: compute per-artifact SHA-256 digests on the client for html, text, screenshots, and captured response bodies; upload the raw artifacts immediately; then canonicalize the entire JSON envelope server-side using RFC 8785 JSON Canonicalization Scheme and sign or hash-chain the server copy for tamper evidence. MDN’s SubtleCrypto.digest() is suitable for the client digest step, but it is not a streaming API, so very large bodies should also be hashed server-side after upload. NIST recommends digital signatures and secure timestamps to protect audit trails from undetected modification, and OWASP recommends logs that provide a chronological, independently verifiable trail.

A proposed evidence contract is below. It is intentionally opinionated: the operator never chooses source_key directly during capture; the run/task assigns it, and the extension only submits evidence against that assignment.

{
  "schema_version": "be_capture_v1",
  "transport_kind": "browser_extension",
  "capture_id": "uuid",
  "lead_run_id": "uuid",
  "capture_task_id": "uuid",
  "county_id": "tx_travis",
  "source_key": "tx_travis_civil_docket",
  "source_template_version": "2026-07-01",
  "operator": {
    "operator_id": "user-123",
    "operator_session_id": "uuid"
  },
  "extension": {
    "extension_version": "1.12.4",
    "browser_family": "chromium",
    "browser_version": "138.0.x",
    "capture_started_at_utc": "2026-07-08T18:21:11Z",
    "capture_submitted_at_utc": "2026-07-08T18:21:19Z"
  },
  "page_binding": {
    "tab_id": 412,
    "frame_id": 0,
    "document_id": "uuid",
    "committed_url": "https://county.example.gov/case/123",
    "final_url": "https://county.example.gov/case/123",
    "page_title": "Case Detail",
    "page_committed_at_utc": "2026-07-08T18:20:54Z",
    "last_history_change_at_utc": null
  },
  "validation": {
    "url_template_id": "county_case_detail_v3",
    "url_match_passed": true,
    "url_extracted": {
      "county_hint": "travis",
      "case_number": "C-1-CV-24-001234"
    },
    "page_classifier_passed": true,
    "required_normalized_fields": ["case_number", "filing_date"],
    "required_normalized_fields_present": true
  },
  "artifacts": {
    "dom_html": {
      "content_type": "text/html",
      "sha256": "hex",
      "bytes": 182344,
      "value": "<html>...</html>"
    },
    "visible_text": {
      "sha256": "hex",
      "bytes": 17280,
      "value": "Case Number ..."
    },
    "network": [
      {
        "request_key": "case_detail_json",
        "url": "https://county.example.gov/api/case/123",
        "method": "GET",
        "status": 200,
        "mime_type": "application/json",
        "response_sha256": "hex",
        "response_body": "{\"caseNumber\":\"...\"}"
      }
    ],
    "screenshot": {
      "sha256": "hex",
      "mime_type": "image/png",
      "bytes": 248102,
      "value_base64": "..."
    }
  },
  "provenance": {
    "client_payload_sha256": "hex",
    "server_canonical_sha256": "hex",
    "previous_event_hash": "hex-or-null"
  }
}

For a multi-page run, do not create a separate business report per page. Instead, create one lead run with multiple source instances. Each instance should be keyed by the same lead_run_id and have its own capture_task_id, source_key, and page_key. That supports a docket page plus a property page in the same run while preserving independent validation and provenance.

For naming, nested structures are much safer than flat arbitrary keys. The recommended convention is:

• raw evidence under sources.<source_key>.pages.<page_key>...
• transport metadata under sources.<source_key>.pages.<page_key>.capture...
• extracted raw page fields under sources.<source_key>.pages.<page_key>.extracted...
• normalized business output only under normalized...

If you also need warehouse-friendly flattened columns, generate them mechanically as src__{source_key}__{page_key}__{field} and norm__{field}. That keeps raw-source scope and normalized scope visibly distinct, and it avoids transport-specific field names leaking into the final report shape.

Source URL validation and stale-data controls

Source URL validation should be two-stage. The extension performs a fast client-side guard so the operator gets immediate feedback. The backend then performs authoritative validation before evidence is accepted. For the client-side guard, WebExtension match patterns are useful for broad host/path allowlisting, and URLPattern is useful for richer template matching and parameter extraction. However, MDN notes that URLPattern only became “Baseline newly available” in late 2025 and may not work in older browsers or devices, so the backend should still be the final source of truth using its own canonical URL parser and compiled templates.

A strong URL template should validate more than hostname. At minimum, each template should define: scheme rules, exact or wildcard host rules, path template or regex, required query parameters, forbidden query parameters, optional page-title regexes, required DOM markers, and extraction rules for county or entity identifiers. This is where wrong-county and wrong-source-key mistakes are eliminated. If the template extracts county_hint=travis or parcel_id=R123456, that extracted value should be compared to the run context before submission is allowed.

For stale-data prevention, the key control is to bind submission to the currently committed document, not merely to the tab. Chrome’s webNavigation documentation explains that documentId is unique per document and changes when a frame navigates to a new document. It also exposes onCommitted, onDOMContentLoaded, onCompleted, onHistoryStateUpdated, and onReferenceFragmentUpdated, which lets the extension detect both normal navigations and SPA-style URL changes. A solid implementation should record the documentId, committed URL, and time at preview time, then refuse submission if any of those change before upload.

URL validation alone is still not enough for county sites that use common shells, search pages, or multi-step apps. The backend should therefore run a page classifier after upload. That classifier can be very simple at first: required title patterns, required DOM markers, required normalized preview fields, and optional content fingerprints. If the template expects a case-detail page but the page only yields a search-results grid and no case_number, the capture should fail even if the URL matched. JSON Schema can enforce those required outputs formally, and this is the single best defense against “right site, wrong page” submissions.

The operational rejection rules should be strict:

• reject when source_key on the task and payload differ
• reject when URL template match fails on the backend even if the client allowed it
• reject when documentId changed between preview and submit
• reject when a history-state or fragment update occurred after preview on templates marked url_changes_are_material
• reject when required normalized fields are absent or violate schema
• reject when freshness SLA is exceeded, such as “capture must be submitted within 5 minutes of page commit”
• soft-flag, rather than immediately reject, when the artifact hash duplicates a recent successful capture in the same run

These rules are anchored in Chrome’s document-aware navigation model and in schema-based validation, which together are much more reliable than a permissive “URL looked right” approach.

Operator UX and permission model

The best operator experience is a guided checklist, not a blank popup. The extension should open on the current tab and show: county, logical source, expected page label, URL match status, document stability status, freshness timer, required normalized outputs, and remaining pages for the run. The “Capture” button should stay disabled until all mandatory checks pass. That design matches Chrome’s activeTab model well, because access is intentionally tied to an explicit user gesture, and it naturally reduces accidental capture on the wrong page.

Use narrow permissions wherever you can. Chrome’s documentation recommends optional permissions for optional features, and runtime permission requests are preferable when they let users understand why a grant is needed. For an internal operator extension, the baseline permission set should usually be activeTab, scripting, storage, and webNavigation, with curated host permissions only if needed for passive page detection or non-click workflows. If you manage browsers centrally, Chrome’s storage.managed can also hold policy-defined read-only settings, which is useful for county template catalogs, allowed transport modes, and environment pins.

There is an important permission tradeoff around network-body capture. Chrome explicitly says the debugger permission cannot be specified as optional, which means an extension that needs chrome.debugger will always carry that elevated capability at install time. Since chrome.debugger is the most robust way to instrument network interaction and the DevTools Protocol Network domain can return request bodies and response bodies, many teams should split this into two delivery tiers: a standard operator extension for DOM/text capture, and an elevated operator-only build for counties that require network-body capture. That separation gives you a much better least-privilege posture and a cleaner audit story.

When structured-response capture is necessary, prefer one of two paths. First choice: use chrome.debugger with CDP Network.getResponseBody and Network.getRequestPostData, because those are official network-body mechanisms. Second choice: inject a very small MAIN-world bridge only for the specific county template and only to read page-owned data or intercept page-level fetch/XHR that the page already uses. Chrome’s scripting API explicitly distinguishes the extension’s isolated world from the page’s main world, so using ExecutionWorld: "MAIN" should be deliberate and template-scoped, not the default.

A screenshot should be treated as human QA evidence, not as the primary extraction medium. Chrome warns that captureVisibleTab is expensive and limits how often it can be called; the API is best used to anchor operator review or explain a failed classifier, not to drive routine normalization. The normalized report should come from HTML, visible text, and structured responses, not from image-first workflows.

Direct and extension parity model

Direct-vs-extension parity becomes much easier when you make logical source schema the stable unit and transport the replaceable unit. In other words, a county docket source should have one source_key, one normalization contract, one required output schema, and multiple transport variants underneath it. The normalizer should operate on the same intermediate evidence shape regardless of transport, and the lead report should be emitted against one shared JSON Schema.

A practical template model looks like this:

logical_source:
  source_key: tx_travis_civil_docket
  county_id: tx_travis
  normalized_contract: lead_docket_v3
  required_normalized_fields:
    - case_number
    - filing_date
    - parties
  transport_variants:
    - kind: direct_scraper
      enabled: true
      template_id: tx_travis_civil_docket_direct_v7
      route_priority: 1
    - kind: browser_extension
      enabled: true
      template_id: tx_travis_civil_docket_capture_v2
      route_priority: 2
      required_pages:
        - docket_detail
      capture_channels:
        - dom_html
        - visible_text
        - network.case_detail_json

The routing policy then becomes straightforward. When direct collection succeeds, you use it. When it fails with a routable status such as technical_block, 429, challenge_page, session_required, or compliance_manual_only, the scheduler looks for an enabled browser-extension transport variant with the same source_key and opens a human capture task for that variant. The report shape does not change, and downstream consumers do not need separate parsers for “extension counties.” They only see the same normalized contract plus richer provenance.

The policy catalog should also separately track whether a source is direct-allowed, extension-preferred, extension-only, or denied. Robots.txt can be one input, but RFC 9309 is explicit that robots rules are requests to crawlers, not access authorization. Recent legal commentary also notes that even where CFAA theories are narrower, contract and Terms-of-Service claims remain important, and organizations should maintain a data-rights register tied to sources and pipelines. The practical product takeaway is that transport policy should be a maintained compliance field, not something engineers infer solely from whether a request technically works.

This also answers how to distinguish browser-extension payload templates from direct templates that require extension fallback. Do not model those as unrelated templates. Model them as separate transport variants attached to the same logical source schema. That gives you same-key fallback, same required normalized fields, and a single place to evolve field mappings over time.

Failure rules and parity test matrix

The end-to-end test strategy should focus less on “did the extension save a page?” and more on contract correctness, routing correctness, and normalized parity. Every fixture should include raw evidence, validation expectations, and final normalized expectations. Audit data should be validated too, because NIST and OWASP both emphasize that high-value transactions need an independently reviewable audit trail, not just business output.

The most important parity assertion is this: for every logical source contract, the normalized output must be compared across transports using the same schema, same required fields, same null handling, same enumerations, and same provenance keys. Direct and extension pipelines should differ in evidence acquisition only. They should not diverge in field naming, report shape, or completeness semantics. JSON Schema contract tests and golden normalized fixtures are the right place to lock that down.

A compact set of implementation rules follows from the research:

Use one logical source_key across direct and extension transports. Bind every capture to a task, county, and document. Validate URL server-side, classify page content, and verify required normalized outputs before acceptance. Store raw evidence separately from normalized output. Keep extension state ephemeral in storage.session, large artifacts in IndexedDB, and transport configuration in signed backend config or managed policy. Prefer activeTab and narrow permissions for the default operator workflow. Reserve chrome.debugger for the minority of counties that truly require network-body evidence, because that permission cannot be optional. And never let “browser-extension county” become a different downstream contract from “direct county”; parity is a schema discipline, not just a routing feature.