Self-Healing Source Template Compatibility for a U.S. County Docket and Property Lead Platform

Full report

On this page

Design position

The most durable design is not a smarter scraper in isolation. It is a provenance-first extraction platform that separates acquisition, parsing, normalization, validation, and rollout, then lets AI repair only the narrowest safe layer of the system. That position is strongly supported by older wrapper-maintenance research, which found that resilient repair depends on preserved page features, wrapper verification, and bounded reinduction rather than blind selector substitution, and by modern declarative connector ecosystems that separate source configuration, schemas, and acceptance testing from runtime execution.

For your county docket and property platform, “maximum compatibility” should mean four things at once. First, the system must survive ordinary presentation drift such as DOM reordering, renamed fields, button text changes, pagination changes, and transport changes from HTML to JSON or browser-captured XHR. Second, it must fail loudly and classifiably when the site moves into a blocking, challenge, or policy-controlled state, rather than retrying forever. Third, AI-assisted repair must emit bounded, reviewable patches instead of silently mutating runtime behavior. Fourth, every property-value or address-bearing output must retain source provenance that lets an operator assess trustworthiness later. Those principles align with W3C PROV’s view that provenance is what allows judgments about quality, reliability, and trustworthiness, with NIST guidance that AI systems need documented oversight and review processes, and with SRE guidance that risky changes should be canaried before broad activation.

Recommended architecture

The architecture should be organized as a layered extraction pipeline with a single canonical evidence model in the middle:

A key design choice is the canonical evidence envelope. Direct HTTP clients, Playwright-based browser automation, Chrome extension capture, HAR import, and CDP traces should all emit the same internal structure: request URL, method, headers, redirected-from chain, final URL, response status, content type, body reference, timing, and whether the artifact came from direct fetch, browser automation, or extension capture. That is practical because Playwright can track and modify browser HTTP/HTTPS traffic including XHR and fetch; Chrome extensions can observe and analyze request lifecycles through chrome.webRequest; Chrome DevTools Protocol’s Network domain exposes headers, bodies, timings, and redirect-related data; and HAR export is a JSON object with a known field structure.

That common envelope is what preserves compatibility between direct scraping and browser-extension capture. The parser should never care whether a county search result came from a raw GET, a browser fetch, or an exported HAR. It should consume a canonical evidence object plus optional rendered DOM snapshot. This is conceptually similar to how declarative connector ecosystems separate actor configuration, discovery, schema, streams, catalog, and state from transport details. Airbyte’s protocol models sources, catalogs, streams, and configured streams explicitly; Singer likewise separates SCHEMA, RECORD, and STATE messages.

Operationally, the system should also emit observability and lineage events around every run. OpenTelemetry gives you a standard vocabulary for HTTP spans, metrics, and logs, while OpenLineage gives you run IDs, dataset facets, custom facets, and input/output statistics. That makes every county run queryable by county, template version, transport mode, response type, row counts, missingness, and repair candidate.

Template schema recommendations

Use JSON Schema Draft 2020-12 as the authoritative template-definition contract, then store changes as RFC 6902 JSON Patch documents. JSON Schema 2020-12 is suitable because it is expressly about validating JSON documents and includes mature features such as unevaluatedProperties, unevaluatedItems, and dynamic references; JSON Patch gives you a standard partial-update format for reviewable configuration changes. For value selection inside JSON responses, use RFC 9535 JSONPath for expressive extraction and RFC 6901 JSON Pointer for deterministic exact paths. For CSV, make the parser RFC 4180-aware because quoting, multiline fields, and delimiter interpretations vary in the wild.

A county template should have the following top-level sections:

Identity and governance

This section should include sourceId, county, state, sourceFamily, allowedDomains, robotsPolicyStatus, contactHint, authMode, activationState, compatibilityClass, and riskTier. The point is to let policy and rollout decisions happen before any request is sent. Robots rules should be treated as crawler guidance rather than authentication, because the Robots Exclusion Protocol explicitly controls how crawlers are requested to access content and is not access authorization.

Acquisition plan

This section should declare a finite acquisition graph:

• entrypoints
• requestTemplates
• navigationMode
• paginationStrategy
• detailStrategy
• renderMode such as direct_http, browser_automation, extension_capture, file_download
• captureTargets for matching XHR/fetch endpoints
• retryPolicyRef
• blockingClassifierRef

The acquisition plan should be declarative, not executable code. Airbyte’s declarative-manifest approach and YAML reference are useful precedents here: configuration names the components and interpolation values, while platform code supplies the executor.

Extraction plan by content type

The schema should support typed extractors, not one generic selector bag:

• html: resilient locator chains, anchors, landmarks, table region rules
• json: JSONPath selectors, JSON Pointer fallbacks, array iteration rules
• csv: header detection, delimiter configuration, quoting mode, row filters
• pdf: page selection, table zone hints, text anchors, repeated-section rules
• networkPayload: parser bound to a captured endpoint instead of rendered HTML

For HTML specifically, prefer a selector ladder in this order: stable data attributes/test IDs if present, then role/label/title/text-driven locators, then constrained CSS/XPath as a last resort. Playwright explicitly calls test IDs the most resilient choice and warns that CSS/XPath should be used only when you absolutely must; it also treats locators as the core unit of auto-waiting and retryability.

Normalization and field mapping

Separate rawFields from normalizedFields. A raw field might be assd_val, marketValue, appraised, or total_value; a normalized output is always PropertyValue. This is where schema-drift compatibility matters. Avro’s alias model is a good pattern: fields and types may have aliases, and aliases can map old names to new ones during schema evolution. Confluent’s compatibility documentation is also helpful conceptually because it frames changes in backward, forward, full, and transitive terms and explains how optional fields and defaults affect safe evolution.

I recommend that every normalized field definition support:

• required
• aliases
• sourceCandidates
• type
• coercion
• nullPolicy
• qualityRules
• evidenceRequirements

For property data, make the provenance rule explicit:

• If PropertyValue is present, then PropertyValueEvidencePageUrl is required.
• If normalized address fields are present, then AddressDataSourcePageUrl is required.
• If both come from different pages, retain both URLs.
• Also retain EvidenceContentType, EvidenceRetrievedAtUtc, and EvidenceFinalUrl.

That requirement is not cosmetic. W3C PROV treats provenance as the information needed to assess quality, reliability, and trustworthiness, and the PROV model is intentionally extensible so domain-specific provenance can be embedded.

State and resume semantics

Templates should explicitly define what progress is resumable. Singer’s STATE and bookmark pattern is a good precedent: extraction state is separate from schema and record messages, and bookmarks track the last known position per stream. For county sources, state should include request cursors, page numbers, seen docket numbers, last-export timestamp, next-detail URL, and capture mode.

Drift detection and failure taxonomy

Classic wrapper-maintenance research is still highly relevant here. Two durable ideas stand out. First, wrapper verification should detect both “nothing extracted” and “extracted but wrong.” Second, repair works better when it exploits preserved features such as syntactic patterns, annotations, hyperlinks, page templates, data position, and surrounding context rather than only replaying brittle DOM paths. One paper specifically highlights preserved page features for repair; another divides maintenance into wrapper verification and reinduction; and the template-finding algorithm in the JAIR work identifies repeated page structure as a reusable signal for locating changed data slots.

The platform should classify failures into distinct machine-actionable categories:

The supporting signals should come from both structural fingerprints and data-quality fingerprints. Structural fingerprints include response status class, content type, DOM subtree signatures, stable anchor texts, table headers, repeated semantic blocks, and payload schemas. Data-quality fingerprints include record counts, nullness rates, field-shape assertions, identity cardinality, and the fraction of rows that carry required evidence URLs. Great Expectations is useful here because it frames missingness and volume as first-class data-quality problems and provides row-count and non-null assertions; OpenLineage’s input/output statistics facets provide row counts, file counts, and sizes that help compare expected and observed output.

Blocking pages deserve their own branch because they are not normal extraction failures. HTTP 429 explicitly means rate limiting and may include Retry-After; AWS and Azure guidance both recommend bounded retries with backoff for transient faults and warn against unlimited retries; Azure’s retry-storm antipattern explicitly calls out avoiding while(true) loops; and 403/challenge pages often represent WAF, bot-management, or challenge interstitials rather than broken selectors. Cloudflare documents that challenge pages evaluate browser signals, may require JavaScript or interaction, and can be triggered by WAF, bot management, or rate limiting.

That leads to an important rule: do not send selector-repair jobs when the body is a challenge page. Repairing selectors against a challenge page teaches the system the wrong document.

AI-assisted repair and validation rules

AI repair should be treated as a proposal system, not an autonomous production editor. NIST’s AI RMF says processes for human oversight should be defined and documented, roles and responsibilities for human-AI configurations should be clear, and relevant governance should include retention of testing, evaluation, validation, and verification history. The GenAI profile also recommends sharing pre-deployment test results with release-approval authorities and reviewing and verifying sources and citations during pre-deployment and ongoing monitoring.

I recommend that the repair engine take as input only:

• the prior template version
• recent successful artifacts
• the failed run’s canonical evidence envelope
• DOM excerpts or payload samples
• data-quality deltas
• county policy metadata
• allowed patch scopes

And it should output only one of three things:

• NoChange
• CandidatePatch
• EscalateHuman

When it produces a patch, the patch should be an RFC 6902 JSON Patch document plus structured justification: changed paths, confidence score, evidence references, expected impact, and risk flags. That gives operators small, reviewable diffs instead of whole-template rewrites.

Safe repair boundaries

AI may propose changes to:

• alternate selectors inside the same field extractor
• field aliases
• JSONPath or JSON Pointer fallbacks
• search/detail row anchors
• pagination selectors within the same host and acquisition graph
• parser choice among already-approved extractor types for that source

AI should not auto-change:

• allowed domains
• auth mode or credentials
• request method from read-safe to state-changing
• body semantics for forms or APIs unless already modeled as alternatives
• compliance settings
• evidence-field requirements
• county identity or source ownership

Those constraints follow both common sense and the cited governance guidance on documented oversight, accountability, and review.

Validation before save, activation, or runtime use

Every candidate patch should pass three gates.

Static gate. Validate the patched template against your JSON Schema meta-schema, reject unknown top-level fields except in approved extension namespaces, ensure patch paths are within allowed scopes, and ensure all normalized outputs still satisfy required evidence and typing rules. Use backward/full compatibility semantics when comparing required fields, aliases, and content models. Avro aliases and Confluent compatibility concepts are especially useful here for distinguishing safe renames from breaking schema changes.

Execution gate. Run the patched template against a fixed corpus of historical artifacts and one or more fresh live pulls. Require minimum extraction success and zero violations on hard invariants such as required evidence URLs, county identity, and strongly typed key fields. Airbyte’s acceptance-test philosophy is a useful precedent: connectors should respond correctly to valid and invalid inputs and meet a minimum quality bar before they are trusted.

Rollout gate. Save the candidate inactive, then run it in shadow mode against a subset of traffic, then canary it before full promotion. Google SRE’s canary guidance is directly applicable here: canarying detects defects quickly while limiting impact.

Recommended retry and repair limits

The limits below are a design recommendation derived from transient-fault guidance, retry-storm warnings, 429 semantics, and circuit-breaker guidance.

Admin review workflow and observability

A strong admin workflow should look like this:

Draft generation. The system stores the failed run, evidence artifacts, old template version, candidate patch, evaluation metrics, and risk flags.

Review. An operator sees a side-by-side diff, field-level impact summary, extraction previews, row-count delta, nullness delta, and evidence-URL compliance delta.

Shadow execution. The candidate runs against historical and fresh evidence but produces no published records.

Canary activation. The candidate serves a small percentage of scheduled jobs for that county or source family.

Promotion or rollback. If metrics stay inside bounds, promote; otherwise revert immediately to the prior version.

This is exactly the kind of change-management discipline that NIST SSDF encourages: follow change-management processes, audit unexpected changes, archive release files and supporting provenance data, and periodically review defined roles and responsibilities.

For logging and audit, use three linked identifiers on every event:

• templateVersionId
• runId
• patchId

OpenLineage run facets are useful because every run has a uniquely identifiable run ID, while custom facets and dataset facets let you attach domain-specific metadata. OpenTelemetry semantic conventions should back the lower-level HTTP client and browser telemetry so HTTP requests, statuses, durations, headers, and retry behavior are traceable consistently. W3C PROV is the right conceptual model for long-term provenance because it separates entities, activities, agents, derivations, and bundles of provenance.

A good review screen should surface risk in plain terms:

• “Selector-only change, same response schema, same evidence URLs”
• “JSON key rename, alias added, backward compatible”
• “Pagination model changed from page links to cursor token”
• “Challenge page detected, no repair attempted”
• “Property values extracted but evidence URLs dropped”

That last case should be a hard stop. The platform should not publish a PropertyValue without its supporting evidence URL because provenance is part of trustworthiness, not optional ornamentation.

Test matrix and failure taxonomy

The platform should maintain a template acceptance matrix that is broader than “did selectors match.” The matrix below is a recommended minimum.

The rationale for this broad matrix comes from multiple directions: wrapper-maintenance literature shows that failures include both total extraction failure and subtly wrong extraction; Great Expectations emphasizes missingness and volume anomalies; Airbyte formalizes connector acceptance testing; and OpenLineage supports recording statistics and assertions attached to runs and datasets.

A concise operational failure taxonomy for the admin console should distinguish at least these states:

Risks and mitigations

The main architectural risk is over-trusting self-healing. Runtime healing systems can reduce maintenance burden, but unbounded healing can normalize bad data, teach on blocking pages, or silently strip provenance. The mitigation is to constrain repair scope, require hard invariants on evidence fields, and route all nontrivial changes through validation plus human-reviewed rollout. That governance stance is consistent with NIST AI RMF oversight guidance and with long-standing wrapper-maintenance literature, which treats verification and repair as separate phases.

Another risk is retry amplification. County sites are often fragile, rate-limited, or protected by WAFs. The mitigation is not more retries; it is bounded retries with backoff and jitter for genuinely transient faults, respect for Retry-After, and circuit-breaker behavior for persistent errors. Retry storms are explicitly called out as an antipattern by Azure, and 429 is explicitly defined as rate limiting by the IETF.

A more subtle risk is capture divergence between browser-extension evidence and direct scraping evidence. If the two flows produce different internal models, you will duplicate every parser and every repair. The mitigation is the canonical evidence envelope described earlier, backed by standards and browser instrumentation APIs that already expose the necessary transport details.

The last major risk is evidence decay. County sites change, records disappear, and URLs redirect. The mitigation is to keep both an evidence URL and an archived content reference for each critical extracted value, version your templates and patches, and retain provenance and TEVV history. W3C PROV, OpenLineage, and NIST guidance all support this direction.

In practical terms, the recommended end state is this: a declarative county-source platform where transport capture is unified, extractors are typed, selectors are ranked rather than singular, compatibility is modeled explicitly, provenance is mandatory for property values, AI emits bounded patches instead of live mutations, and every change is validated, shadowed, canaried, diffed, and reversible. That is the highest-compatibility design pattern in this problem space because it treats website drift as a governed compatibility problem, not just a scraper-maintenance problem.