The Purchased Black Box: Vendor Secrecy, Public Procurement, Trade Secrets, and the Non-Delegable Duty to Explain

Full report

On this page

1. Executive Summary

As administrative and corporate governance rapidly digitizes, public and private institutions increasingly procure proprietary algorithmic systems to allocate resources, assess risk, screen employment candidates, and enforce the law. This technological modernization, however, has precipitated a profound crisis in administrative and constitutional law. The procurement of "black-box" decision-making tools—developed by private vendors and shielded by aggressive intellectual property and trade secret protections—creates a structural conflict between commercial confidentiality and the foundational tenets of due process, public records transparency, and algorithmic accountability. The central inquiry of this research is how an institution can procure a proprietary algorithmic system while preserving meaningful explanation, evidence access, independent testing, correction, portability, and public accountability. Comprehensive legal and technical analysis indicates that current procurement frameworks are drastically underequipped to manage the nuances of algorithmic deployment. Vendors routinely invoke trade secret privileges under state and federal statutes to block discovery in criminal proceedings, thwart public records requests, and prevent independent adversarial audits of their machine learning models1. Consequently, institutions are delegating highly consequential human judgments to automated systems without securing the requisite legal mechanisms to explain or contest those judgments. To resolve this escalating conflict, this report establishes a Non-Delegable Accountability Principle, asserting that the constitutional, statutory, and administrative duties of an institution cannot be contracted away through vendor confidentiality clauses or End User License Agreements. Through an exhaustive examination of algorithmic procurement across ten distinct sectors—ranging from criminal courts to public infrastructure—and the development of nine bespoke legal and technical frameworks, this report provides a comprehensive blueprint for preserving algorithmic due process. By adapting traditional software escrow into a modernized Evidence Escrow model, and by restructuring the public procurement lifecycle to mandate transparency, institutions can harness advanced computational tools without sacrificing the fundamental rights of the populations they serve.

2. Why Institutions Purchase Black-Box Systems

Institutions procure proprietary algorithmic systems to manage unprecedented volumes of data, reduce administrative backlogs, and ostensibly eliminate human bias from complex decision-making processes. The appeal of vendor-supplied systems over in-house development lies in the rapid deployment of mature, commercial-off-the-shelf technologies. The Federal Acquisition Regulation and state-level equivalents historically incentivize the procurement of commercial items to optimize taxpayer expenditures, but these legacy frameworks were designed for physical goods and static software, not dynamic, probabilistic machine-learning models3. The systemic failure to adapt procurement laws to artificial intelligence has resulted in the unchecked proliferation of opaque systems across critical sectors. The deployment of these black-box systems spans every facet of public and private life, frequently carrying profound civil rights implications. In policing, law enforcement agencies purchase facial recognition software, predictive policing models, and video analytics tools to dictate patrol routes and identify suspects. Vendors routinely refuse to disclose the training data or algorithmic weights, citing trade secrets, which effectively prevents defendants from scrutinizing the digital evidence used against them4. Similarly, within the judicial system, courts license pretrial risk assessment instruments to predict recidivism. Judges utilize these proprietary scores for bail and sentencing determinations, despite well-documented racial disparities and the vendor's categorical refusal to open the source code to adversarial defense scrutiny5. Administrative agencies managing public benefits heavily utilize automated fraud detection systems to flag unemployment and welfare claims. When these machine-learning systems trigger mass automated denials, the procuring agencies often cannot explain the specific variables that caused an individual claimant's denial because the vendor treats the underlying anomaly detection logic as proprietary intellectual property7. In the education sector, public school districts deploy vendor-managed student monitoring software to scan student emails, documents, and web traffic for self-harm or violence risks. The algorithms determining what constitutes a behavioral "red flag" remain opaque, leaving students and parents unable to challenge inaccurate disciplinary triggers that may follow the student for years8. In the private sector, human resources departments purchase automated employment decision tools to screen resumes, analyze video interviews, and rank candidates. While certain jurisdictions have attempted to mandate bias audits for these tools, the underlying mechanisms of the artificial intelligence systems often remain complete black boxes to the applicants being rejected10. Healthcare institutions similarly procure triage and population-health algorithms to allocate intensive care management resources. The landmark 2019 Obermeyer study published in Science revealed that a widely utilized commercial healthcare algorithm severely underestimated the health needs of Black patients because the vendor's model relied on healthcare costs as a proxy for healthcare needs—a systemic flaw hidden for years behind vendor opacity and restrictive procurement contracts12. Financial regulators and tax authorities procure anomaly detection systems for fraud detection, justifying algorithmic opacity under the premise that revealing the decision logic would allow malicious actors to "game" the system. However, this absolute secrecy prevents legitimate businesses from understanding why their accounts have been frozen or their transactions flagged. In immigration, visa processing and border enforcement increasingly rely on continuous evaluation risk-scoring algorithms to flag applications for fraud or security concerns. The lack of explainability leads to administrative delays and deportations that cannot be meaningfully appealed under standard administrative procedures15. The housing market has been similarly automated, with landlords relying on automated tenant screening services that scrape eviction records and criminal databases to generate proprietary risk scores. Errors in identity matching are notoriously difficult for applicants to correct because the data brokers shield their probabilistic matching algorithms from consumer review. Finally, public infrastructure relies on automated license plate readers and biometric identification for access control, smart tolling, and transit bans. False positives in these systems result in unwarranted financial penalties or the loss of access to public goods without a clear, evidence-based mechanism for citizen redress. In response to increasing demands for accountability across these ten sectors, vendors frequently offer Model Cards, summary accuracy statistics, or controlled demonstrations. These artifacts, however, are fundamentally insufficient for individualized contestability. A Model Card provides global metrics—such as an overall accuracy rate or aggregated demographic parity—but it cannot explain why a specific individual received an adverse output16. Demonstrations and application programming interface (API) access allow for basic input-output testing, but they do not expose the environmental variables, historical hyperparameter shifts, or specific data weights that were active at the exact microsecond the contested decision was rendered. Consequently, summary statistics provide the illusion of transparency while failing to satisfy the legal requirements of due process.

3. Trade Secrets and Public Accountability

The escalating tension between algorithmic accountability and intellectual property rights is rooted in the strategic weaponization of trade secret law by government vendors. Technology companies assert that revealing the internal mechanics of their algorithms—including source code, training data schemas, and mathematical decision trees—would destroy their commercial viability and allow competitors to appropriate their innovations. The legal mechanics of vendor secrecy are robustly protected under state and federal law. Under the Illinois Trade Secrets Act and the Uniform Trade Secrets Act, a trade secret is defined as information that derives independent economic value from not being generally known and is subject to reasonable efforts to maintain its secrecy (Jurisdiction: Illinois; Date: 2024; Status: Active; Authority: 765 ILCS 1065/2(d))2. Vendors codify these "reasonable efforts" through aggressive confidentiality clauses and sweeping indemnification waivers in public procurement contracts. When a journalist, academic researcher, or affected citizen files a public records request, the government agency inevitably invokes state equivalents of the Freedom of Information Act to deny the request. For example, Illinois law exempts trade secrets and commercial financial information from public disclosure if the information is furnished under a claim that it is proprietary and its disclosure would cause competitive harm (Jurisdiction: Illinois; Date: 2024; Status: Active; Authority: 5 ILCS 140/7(1)(g))1. However, public bodies bear the burden of proving by clear and convincing evidence that such exemptions apply, a standard frequently ignored in the rush to protect vendor relationships1. The reliance on trade secret evidentiary privileges in criminal and administrative proceedings represents a profound crisis for constitutional due process. Criminal defendants possess a constitutional right under the Due Process Clause to access exculpatory and impeachment evidence4. Yet, when law enforcement utilizes proprietary algorithms, vendors intervene to quash defense subpoenas, arguing that source code disclosure constitutes irreparable harm to their intellectual property6. This creates a severe asymmetry: the state can utilize digital evidence to deprive a citizen of liberty, but the citizen cannot scrutinize the digital witness testifying against them. Legal scholars argue that this application of the trade secret privilege is ahistorical, harmful to defendants, and unnecessary, as protective orders can sufficiently safeguard the vendor's commercial interests while allowing defense experts to examine the code19. The jurisprudence surrounding this conflict remains highly fractured. The seminal Wisconsin Supreme Court case State v. Loomis upheld the use of the proprietary COMPAS risk assessment algorithm in sentencing, concluding that the defendant's inability to inspect the algorithm's source code did not violate due process because the score was ostensibly advisory6. Conversely, in State v. Pickett, the New Jersey Appellate Division recognized the indispensable nature of source code for effective cross-examination, ordering the disclosure of the TrueAllele DNA software's source code under a strict protective order (Jurisdiction: New Jersey; Date: 2021; Status: Active; Authority: 468 N.J. Super. 340\)20. To resolve this conflict and provide standardized guidance for procurement officers and administrative law judges, institutions must adopt a Trade Secret versus Due Process balancing framework. This framework explicitly weighs the life, liberty, and property interests of the affected individual against the commercial interests of the software vendor.

4. Procurement Lifecycle and Contractual Governance

The failure of algorithmic accountability is fundamentally a failure of contract drafting. When agencies procure artificial intelligence systems under standard IT templates designed for word processors and database hosting, they inadvertently delegate sovereign functions to private entities without attaching the necessary transparency hooks. Federal guidance, such as the Office of Management and Budget Memorandum M-24-10, explicitly mandates that agencies implementing "rights-impacting" and "safety-impacting" artificial intelligence conduct pre-deployment risk assessments, maintain ongoing monitoring, and ensure human oversight (Jurisdiction: Federal; Date: 2024; Status: Active; Authority: OMB M-24-10)15. However, these sweeping federal mandates cannot be satisfied if the underlying commercial contract prohibits the agency from accessing the necessary technical artifacts. Recent audits indicate that federal agencies face significant challenges in acquiring generative artificial intelligence tools because existing procurement regulations conflict with the rapid evolution and inherent opacity of these systems24. To address this structural failure, this report develops the Non-Delegable Accountability Principle. This principle dictates that an institution vested with the authority to make decisions impacting the legal rights, physical liberty, or socio-economic status of an individual cannot contractually delegate the cognitive logic of that decision to a third party without retaining the unfettered ability to explain, audit, and contest the specific mechanics of every individual outcome. If a vendor refuses to grant the agency the technical access required to fulfill its administrative or constitutional duties, the agency is legally prohibited from procuring the system. Vendor opacity does not shield agency decisions from judicial review under the Administrative Procedure Act; the procuring agency remains the legally responsible actor15. To operationalize this principle, institutions must embed a Public Algorithm Procurement Checklist into every phase of the procurement lifecycle, from the initial Request for Proposals to final contract execution.

5. Evidence Access and Escrow

When an algorithmic decision is contested—whether by a criminal defendant, a rejected job applicant, or a denied benefits recipient—meaningful review requires access to the exact state of the algorithmic system at the precise moment the decision was rendered. Vendors frequently argue that providing the input data and the resulting output should be sufficient to test for bias or error. This assertion is mathematically and legally fallacious. Machine learning models, particularly deep neural networks, are highly non-linear. Output-only review cannot detect whether the system relied on an impermissible proxy variable, such as utilizing a zip code as a proxy for race in a housing or sentencing algorithm. Furthermore, relying solely on post-hoc explainability techniques provides only approximations of the model's behavior, which can be fragile and legally insufficient for proving intent or causation in a discrimination claim. To bridge the gap between valid trade secret concerns and the absolute necessity of due process, this report proposes the Evidence Escrow model, heavily adapted from traditional software escrow agreements utilized in corporate acquisitions and enterprise software licensing25. Traditional software escrow secures static source code to ensure business continuity if a vendor goes bankrupt27. An Evidence Escrow extends this concept dramatically, securing the "intelligence" of the artificial intelligence system to ensure legal continuity, algorithmic accountability, and civil rights compliance25. The mechanics of the Evidence Escrow model are highly specified. First, the vendor (acting as the Depositor) routinely deposits the source code, training data schemas, model weights, hyperparameters, and the full build environment into a secure, air-gapped vault managed by a neutral third party (the Escrow Agent)25. Second, the system requires a rigorous Intellectual Property Audit Trail. Every update pushed to the live production environment must be simultaneously mirrored in the escrow with a cryptographic hash, ensuring that investigators can roll back the model to the exact microsecond a contested decision was made29. Third, unlike commercial escrow triggered by insolvency, an Evidence Escrow is activated by specific legal release triggers. These include a valid subpoena in a civil or criminal proceeding, an official inquiry by an Inspector General, or a mandate for an annual independent bias audit, such as those required by New York City's Local Law 14410. Finally, upon a trigger event, the Escrow Agent does not publish the code publicly. Instead, access is granted within a secure, monitored computational environment strictly to authorized defense experts or auditors under a rigorous judicial Protective Order, satisfying the vendor's trade secret requirements while upholding the citizen's right to due process. Access to algorithmic artifacts must be strictly tiered based on the stakeholder's legal mandate. The Vendor Audit-Access Matrix defines these operational parameters to prevent over-disclosure while ensuring accountability.

6. Vendor Replacement and Lock-In

A critical and often overlooked vulnerability in algorithmic procurement is the phenomenon of vendor lock-in. When a public agency attempts to replace a failing, expensive, or biased algorithmic vendor, they frequently discover that the vendor holds the historical data and model outputs hostage, claiming them as proprietary derivatives of their intellectual property. If an agency's contract fails to secure these logs, the replacement of the vendor results in the total evaporation of administrative records, crippling subsequent longitudinal audits, civil rights investigations, or federal oversight. Federal acquisition regulations, such as FAR 52.227-14, govern the rights to technical data and computer software, often granting the government limited or restricted rights in software developed at private expense3. Vendors heavily leverage these clauses to claim that the inferences generated by their models—even when analyzing government data—belong exclusively to the vendor. To mitigate this extortionate practice, procurement contracts must explicitly delineate that the procuring institution retains absolute sovereignty over the raw input data, the generated inferences, and the historical decision logs. This requirement is codified in the Termination and Data Return protocol. This protocol dictates that upon contract expiration or termination for cause (such as failing a mandated bias audit), the vendor must return all institutional data and decision logs in an interoperable, non-proprietary format (e.g., JSON, CSV). Furthermore, institutions must enforce the Successor Vendor Continuity standard. This standard requires the outgoing vendor to provide reasonable technical assistance and data dictionaries to map historical outputs into the successor's architecture, ensuring that the agency's records management and legal retention obligations are uninterrupted. Following termination, the vendor must execute a certified deletion protocol, providing cryptographic proof that institutional data is no longer being used to train the vendor's future commercial models. This prevents public data from being indefinitely strip-mined for private intellectual property enrichment.

7. Ten Fictional Procurement Scenarios

The theoretical risks of vendor secrecy manifest violently in practice. The following ten scenarios illustrate the friction between trade secrets, procurement, and accountability, strictly applying the required analytical parameters to demonstrate the catastrophic failure of standard procurement models.

8. Model Contract Protections

To prevent the catastrophic failures outlined in the scenarios above, institutions must fundamentally overhaul their procurement templates. The No-Unexplainable-Adverse-Action rule must be established as the non-negotiable cornerstone of every algorithmic contract: The institution shall not deploy any system that generates an adverse action against a human being if the vendor prohibits the institution from obtaining and disclosing the exact computational logic that produced that specific action. To enforce this rule, the procurement contract must explicitly and exhaustively govern the following eleven operational domains:

1. Data Ownership: The procuring institution retains absolute, unencumbered ownership of all raw input data, generated outputs, risk scores, and metadata. The vendor is granted a limited, revocable license strictly for operating the service, explicitly prohibiting the vendor from utilizing the agency's data to train distinct commercial models for resale.

2. Model Updates: The Model Update Notice rule mandates that vendors cannot push "silent" updates to models affecting rights or safety. Vendors must provide 30 days' advance notice of any changes to model architecture, feature weights, or training data schemas. This notice must be accompanied by an updated bias impact assessment proving that the update does not violate statistical parity thresholds, such as the four-fifths rule recognized under federal guidelines and local laws11.

3. Subcontractors: Vendors must fully disclose all third-party artificial intelligence models (e.g., external Large Language Model APIs) or data brokers utilized within their software stack. The primary vendor bears strict liability for the opacity, biases, or security failures of its subcontractors.

4. Audit Access: The contract must explicitly waive trade secret defenses against authorized civil rights audits, inspector general reviews, and court-ordered defense discovery, channeling all such access through the secure Evidence Escrow model.

5. Incident Reporting: Vendors must report any discovered algorithmic anomalies, data drifts, cybersecurity breaches, or disparate impacts to the institution within 48 hours of detection, subject to liquidated damages for failure to report.

6. Correction Propagation: If a citizen successfully appeals an algorithmic decision, the vendor must have a contractual Service Level Agreement to ingest that correction and update the model's localized weights within 72 hours, preventing the system from recursively generating the same error.

7. Termination: The institution reserves the unilateral right to terminate the contract for cause if the system fails a bias audit, if the vendor refuses a lawful transparency request, or if the vendor breaches the escrow deposit requirements.

8. Data Return: Governed by the Termination and Data Return protocol, the vendor must return all data in a standard, interoperable format (such as JSON or XML) within 30 days of termination, at no additional cost to the procuring institution.

9. Deletion: Following the return of data, the vendor must provide cryptographically signed certification that all institutional data, including data absorbed into active training sets, has been permanently purged from the vendor's servers and cloud infrastructure.

10. Model Retirement: The contract must outline a safe sunset period, maintaining the Evidence Escrow for a minimum of five years post-retirement to allow for historical legal challenges, longitudinal studies, and delayed administrative appeals.

11. Successor Vendors: The vendor is contractually obligated to provide API documentation, data dictionaries, and reasonable technical engineering support to migrate the institution's operations to a new vendor without sabotage, data degradation, or extortionate transition fees.

9. Counterarguments

It is intellectually necessary and practically vital to present the vendor's perspective fairly. The private sector drives the vast majority of artificial intelligence innovation. If governments force technology companies to completely open-source their algorithms, companies will simply refuse to bid on public contracts, depriving the public sector of life-saving technology—such as advanced medical triage, highly accurate climate modeling, or cybersecurity threat detection. The Cybersecurity Imperative posits that total transparency is a severe security hazard. In fields like tax fraud detection, cyber-defense, or critical infrastructure access, revealing the exact parameters and threshold weights of an anomaly detection algorithm provides malicious actors with a literal roadmap to bypass the system. Federal regulations intentionally grant the government "restricted rights" to commercial computer software specifically to protect these proprietary and security-sensitive assets from public exposure via records requests30. Furthermore, the Intellectual Property Reality dictates that training a sophisticated machine learning model requires millions of dollars in computational power and specialized data acquisition. The resulting model weights, hyperparameter configurations, and training schemas are the literal manifestation of that immense financial investment. Under statues such as the Illinois Trade Secrets Act, this represents a classic, protectable trade secret (Jurisdiction: Illinois; Date: 2024; Status: Active; Authority: 765 ILCS 1065/2)2. Stripping this protection devalues the company, penalizes innovation, hurts investors, and stifles the research and development cycle. The frameworks proposed in this report—specifically the Evidence Escrow model and the Vendor Audit-Access Matrix—fully concede and accommodate these points. They do not demand that source code or model weights be published on the open internet or handed to commercial competitors. Instead, they demand that the code be securely auditable by authorized officials and legal defense teams under strict Non-Disclosure Agreements and judicial protective orders. A vendor's right to protect its capital investment does not, and constitutionally cannot, supersede a citizen's fundamental right to due process or a government's sovereign duty to explain its actions. Security through obscurity is a brittle defense paradigm; accountability through controlled, secure transparency represents the necessary equilibrium.

10. Open Questions

The intersection of public procurement, intellectual property, and algorithmic accountability remains fluid. Several critical questions remain unresolved by current jurisprudence and administrative law: First, what is the technical scalability of the Evidence Escrow model for continuously learning systems? While escrowing a static algorithm is straightforward, applying the same model to reinforcement learning algorithms—where the mathematical weights change millions of times a day based on continuous data ingestion—presents profound version-control challenges. Second, regarding judicial and defense competence, even if a vendor provides full source code and model weights to a public defender, do underfunded defense agencies possess the computational resources and expert witness budgets required to actually parse a multi-billion-parameter neural network? Releasing code without funding the computational expertise to interpret it provides only a pyrrhic victory for due process. Third, how do individuals establish Article III standing in federal court to challenge an opaque system? If a citizen suspects an artificial intelligence system wrongfully denied them a benefit, establishing standing requires an injury-in-fact caused by the defendant. If the vendor's opacity prevents the citizen from knowing exactly why they were denied, they face a Kafkaesque barrier to initiating litigation in the first place. Addressing these questions will require a paradigm shift across the legal and technical professions. Society must move away from the presumption that computational technology is inherently objective and infallible, and recognize that an unexplainable algorithm deployed by the state is legally indistinguishable from an arbitrary autocrat.

Web-Ready Content

150-Word Public Explanation

When the government or a hospital makes a decision that changes your life—like denying you bail, rejecting a job application, or withholding healthcare—you have a fundamental right to know exactly why. Increasingly, these decisions are made by automated AI systems purchased from private companies. However, when citizens demand an explanation, the government often claims it cannot tell them because the private company considers the algorithm's internal logic a "trade secret." This report investigates this growing crisis of accountability. We argue that public institutions cannot contract away their constitutional duties. We provide actionable solutions, including an "Evidence Escrow" system, ensuring that companies can protect their true intellectual property from competitors without stripping citizens of their right to due process, independent audits, and fair treatment under the law. No one should lose their liberty or livelihood to a purchased black box.

Ten Key Findings

1. Due Process is Being Outsourced: Institutions are delegating consequential decisions to AI without securing the contractual right to explain those decisions to affected individuals.

2. Trade Secrets are Weaponized: Vendors routinely abuse trade secret laws (like 765 ILCS 1065\) and FOIA exemptions to block criminal discovery, civil rights audits, and media scrutiny.

3. Source Code is Insufficient: Disclosing source code alone does not explain AI behavior; auditors need model weights, training data schemas, and hyperparameters.

4. Model Cards are Not Due Process: Global accuracy statistics and "Model Cards" provide averages, but they cannot provide the individualized contestability required by law.

5. Evidence Escrow Solves the Impasse: Adapting software escrow to hold AI model weights and code securely allows for defense audits without exposing IP to market competitors.

6. Vendor Lock-In is a Data Trap: Poorly drafted contracts leave agencies unable to switch vendors without losing years of historical data and administrative records.

7. Silent Updates Cause Chaos: Vendors frequently push unannounced model updates that alter decision weights, instantly shifting policy outcomes without agency approval.

8. Federal Mandates Require Change: New regulations (like OMB M-24-10 and NYC Local Law 144\) mandate transparency, rendering standard IT procurement contracts legally non-compliant.

9. Automation Bias Amplifies Harm: Because agencies cannot see the logic, human operators default to trusting the machine, effectively rubber-stamping biased algorithmic outputs.

10. The Non-Delegable Duty: A public or quasi-public institution retains ultimate constitutional and administrative liability for automated adverse actions, regardless of vendor indemnification clauses.

Ten FAQs

1\. What is an algorithmic "black box"? A black box is an AI or algorithmic system where the inputs and outputs are visible, but the internal mathematical logic determining how the inputs are transformed into outputs is hidden or too complex to easily interpret. 2\. Why do vendors claim their algorithms are trade secrets? Vendors spend millions developing the data structures, weights, and logic of their AI. They argue that revealing this information would allow competitors to steal their innovations, destroying their business model. 3\. Can the government just demand the source code? Not always. Standard procurement contracts often explicitly state the software is commercial off-the-shelf (COTS) and the intellectual property remains with the vendor, legally binding the government to secrecy. 4\. How does this affect criminal justice? Algorithms are used for facial recognition, DNA mixture analysis, and predicting recidivism. If the defense cannot inspect the algorithm for errors or biases, the defendant is denied their constitutional right to confront the evidence against them. 5\. What is NYC Local Law 144? It is a New York City law requiring employers to conduct annual independent bias audits of automated employment decision tools (AEDTs) and publish a summary of the results. 6\. What is an Evidence Escrow? It is a legal and technical agreement where a neutral third party holds a secure copy of the AI's source code, weights, and training data. It is only released to authorized auditors or defense experts under a strict court order. 7\. Why are "Model Cards" not enough? Model Cards are like a nutrition label for AI—they give general information about the model's design and overall accuracy. However, they cannot tell a specific person exactly why their individual application was rejected. 8\. What is the "Non-Delegable Accountability Principle"? The legal concept that a government agency cannot transfer its constitutional duty to provide due process and explain its decisions to a private technology vendor. 9\. Can't we just have a human review the AI's decision? In theory, yes. In practice, human operators often suffer from "automation bias"—they blindly trust the computer. Furthermore, if the human doesn't know why the AI made the decision, their review is meaningless. 10\. What happens if a public agency fires a bad AI vendor? Without strong contract protections, the agency might lose all its historical data, be barred from exporting its records, or face operational collapse, leading to dangerous "vendor lock-in."

Glossary of Twenty-Five Terms

1. Algorithmic Accountability: The concept that organizations must be responsible for the outcomes of their automated systems, including proving they are fair and accurate.

2. Algorithmic Impact Assessment (AIA): A formal pre-deployment evaluation to identify potential risks, biases, and civil rights impacts of an AI system.

3. Automation Bias: The psychological tendency of humans to favor suggestions from automated decision-making systems and ignore contradictory information made without automation.

4. Black Box: A system whose internal workings are hidden from the user, regulators, or affected parties.

5. Brady Obligation: A prosecutor's constitutional duty to disclose exculpatory evidence to a criminal defendant.

6. Commercial Off-The-Shelf (COTS): Standard, pre-packaged software purchased by the government, often protected by restrictive IP licensing.

7. Data Drift: The phenomenon where the real-world data an AI encounters changes over time, causing the model's accuracy to degrade.

8. Disparate Impact: When a seemingly neutral algorithmic policy disproportionately affects a protected class (e.g., race, gender) negatively.

9. Due Process: The constitutional guarantee that a person will receive fair, explainable, and contestable procedures before being deprived of life, liberty, or property.

10. Evidence Escrow: A secure, third-party vault holding AI source code and weights, accessible only via a legal trigger (like a subpoena) for auditing.

11. Explainability: The degree to which the internal mechanics of a machine or deep learning system can be explained in human terms.

12. FOIA Exemption: A legal provision (e.g., 5 ILCS 140/7(1)(g)) allowing the government to withhold public records, often used to protect vendor trade secrets.

13. Hyperparameters: The external configuration variables set by developers before training a machine learning model, heavily influencing how the model learns.

14. Impact Ratio: A metric used in bias audits to compare the selection rate of a protected group against the highest-selected group.

15. Local Law 144: A NYC law mandating bias audits for automated employment decision tools.

16. Model Card: A document detailing the performance characteristics, intended use, and limitations of a machine learning model.

17. Model Weights: The mathematical values inside a neural network that determine how much influence one piece of data has over the final decision.

18. Non-Delegable Duty: A legal obligation that cannot be transferred to a third party via contract.

19. OMB M-24-10: A federal memo mandating governance and risk management for rights-impacting AI used by federal agencies.

20. Protective Order: A court order limiting the disclosure of sensitive information (like source code) to specific experts to prevent public release.

21. Proxy Variable: A seemingly neutral data point (like zip code) that strongly correlates with a protected characteristic (like race) and can introduce bias.

22. Red-Teaming: The practice of rigorously challenging an AI system by adopting an adversarial approach to uncover flaws, biases, or security risks.

23. Rights-Impacting AI: An automated system whose output directly affects a person's civil rights, liberties, or access to essential services.

24. Trade Secret: Proprietary information deriving economic value from not being generally known, actively protected by its owner.

25. Vendor Lock-In: A situation where an institution is so reliant on a vendor's proprietary technology and data formats that transitioning to a competitor is practically impossible.

Procurement Red-Flag Checklist

Model Clause Checklist (Plain Language)

Six Warning Callouts

WARNING: Procuring a system that provides a "risk score" without an explanation of the contributing factors legally exposes the institution to due process and Administrative Procedure Act violations.

WARNING: Relying solely on a vendor's self-published "Accuracy Report" is insufficient for civil rights compliance. Independent, adversarial red-teaming is required.

WARNING: "Source Code" is not enough. If the contract grants access to the code but not the model weights and training data schemas, the AI remains a black box.

WARNING: Do not accept cross-examination of the human operator as a substitute for algorithmic transparency. An operator cannot testify to mathematical functions they cannot see.

WARNING: Permitting a vendor to retain derived data post-termination effectively turns public taxpayer records into perpetual private assets.

WARNING: Federal guidelines (OMB M-24-10) are creating a baseline standard of care. Ignorance of a vendor's internal AI mechanics is no longer a valid legal defense for federal or federally funded agencies.

Procurement-to-Retirement Lifecycle Diagram

\[1. PRE-PROCUREMENT\] └── Algorithmic Impact Assessment (AIA) └── Is it Rights-Impacting? (Yes/No) └── If Yes \-\> Mandate Explainability Requirements in RFP. \[2. CONTRACTING\] ├── Establish Evidence Escrow (Tri-party Agreement). ├── Embed Model Update Notice Rule. ├── Enforce Data Ownership & Right-to-Audit Clauses. └── Reject Absolute Trade-Secret Shields. \[3. DEPLOYMENT & OPERATION\] ├── Vendor Deposits Code & Weights into Escrow. ├── System Flags/Scores Individuals. └── Agency Provides Meaningful Notice to Affected Individuals. \[4. CONTESTABILITY & AUDIT\] ├── Individual Appeals Decision / Files Lawsuit. ├── TRIGGER EVENT: Court Order or Statutory Audit. └── Escrow Agent Grants Secure, NDA-Bound Access to Experts. \[5. RETIREMENT & SUCCESSION\] ├── Contract Terminates. ├── Vendor Executes Data Return Protocol (Interoperable Format). ├── Vendor Certifies Deletion of Agency Data. └── Escrow Maintained for 5 Years for Historical Legal Claims.

Metadata and Search-Intent Recommendations

JSON { "Title": "The Purchased Black Box: Vendor Secrecy, Public Procurement, and Algorithmic Due Process", "Meta\_Description": "An exhaustive analysis of how public institutions can procure proprietary AI systems while preserving due process, independent audit rights, and overcoming vendor trade-secret claims.", "Primary\_Keywords": \[ "algorithmic accountability", "AI public procurement", "trade secret due process", "Evidence Escrow model", "vendor lock-in", "OMB M-24-10 compliance", "NYC Local Law 144", "criminal discovery algorithms" \], "Search\_Intent": "Informational / Policy-making. Targets government procurement officers, legal counsel, administrative law judges, and civil rights advocates seeking actionable frameworks to draft safe AI procurement contracts and navigate IP vs. due process conflicts.", "Target\_Audience": "Public procurement officials, administrative law attorneys, cybersecurity auditors, policymakers.", "URL\_Slug": "/research/purchased-black-box-algorithmic-procurement-trade-secrets" }

Works cited

1. Disclosure of Terms of Lease Agreements for the Rental of Public, https://illinoisfoia.org/opinions/doc-49/

2. (765 ILCS 1065/) Illinois Trade Secrets Act., https://www.ilga.gov/Legislation/ILCS/Articles?ActID=2239\&ChapterID=62\&Chapter=PROPERTY\&MajorTopic=RIGHTS%20AND%20REMEDIES\&Print=True

3. Tech and GovCon: Navigating Intellectual Property Rights Under, https://www.pbmares.com/tech-and-govcon-navigating-intellectual-property-rights-under-government-contracts/

4. Safeguarding Brady Against the Rise of Trade Secrecy in Policing, https://michiganlawreview.org/journal/the-missing-algorithm/

5. Injustice Ex Machina: Predictive Algorithms in Criminal Sentencing, https://www.uclalawreview.org/injustice-ex-machina-predictive-algorithms-in-criminal-sentencing/

6. THE WEAPONIZATION OF TRADE SECRET LAW, https://columbialawreview.org/wp-content/uploads/2024/04/April-2024-7-Chan.pdf

7. criminal justice technology and the regulatory sandbox: toward, https://lawreview.law.pitt.edu/ojs/lawreview/article/download/959/627/1937

8. Examples of Abuse Timeline \- Privacy International, https://privacyinternational.org/abusetimeline

9. Goguardian stories at Techdirt., https://www.techdirt.com/company/goguardian/

10. NYC Bias Audit Law: A Comprehensive Guide, https://www.nycbiasaudit.com/

11. Bias Auditing and Fairness Testing: A Practical Method, https://www.aigovernancecore.com/blog/bias-auditing-fairness-testing

12. Critical AI Health Literacy as Liberation Technology: A New Skill for, https://nam.edu/perspectives/critical-ai-health-literacy-as-liberation-technology-a-new-skill-for-patient-empowerment/

13. Reducing Racial Biases within Healthcare Applications of Arti icial, https://www.sciencepolicyjournal.org/uploads/5/4/3/4/5434385/harve\_etal\_jspg-25-1.pdf

14. The Impact of Health Care Algorithms on Racial and Ethnic Disparities, https://www.acpjournals.org/doi/10.7326/M23-2960

15. AI in U.S. Administrative Law: Agency Rulemaking, APA, and, https://ailegalauthority.com/ai-administrative-law-us/

16. (PDF) Standardizing AI Model Transparency in U.S. Federal, https://www.researchgate.net/publication/412918992\_Standardizing\_AI\_Model\_Transparency\_in\_US\_Federal\_Agencies\_A\_Framework\_for\_Implementing\_AI\_Model\_Cards\_and\_Data\_Provenance\_Auditing

17. Digital dragnets: Examining the government's access to your, https://www.brookings.edu/articles/digital-dragnets-examining-the-governments-access-to-your-personal-data/

18. Who Framed Roger Rabbit? Probably the Secret Codes, https://scholar.smu.edu/cgi/viewcontent.cgi?article=1051\&context=smulrforum

19. AI in Criminal Justice (Chapter 13\) \- The Cambridge Handbook of AI, https://www.cambridge.org/core/books/cambridge-handbook-of-ai-and-technologies-in-courts/ai-in-criminal-justice/E05BCB455FD3BC7C7DBD1386CD067B6E

20. Suspecting with Data, https://scholarship.law.umn.edu/cgi/viewcontent.cgi?article=1164\&context=minnlrev

21. Reference Guide on Forensic Feature Comparison Evidence \- NCBI, https://www.ncbi.nlm.nih.gov/books/NBK621601/

22. Policy Alignment — OMB M-24-10, NIST AI RMF, EU AI Act | Evigov, https://www.evigov.com/policy-alignment

23. Department of Veterans Affairs Compliance Plan for OMB Memor, https://department.va.gov/ai/department-of-veterans-affairs-compliance-plan-for-omb-memorandum-m-24-10/

24. Generative AI Use and Management at Federal Agencies \- GAO, https://files.gao.gov/reports/GAO-25-107653/index.html

25. What is Software Escrow? The 2026 Guide to SaaS and AI Continuity, https://www.escode.com/resources/what-is-software-escrow/

26. Software Escrow – Its Evolution, Use in Legal Frameworks and The, https://www.scl.org/software-escrow-its-evolution-use-in-legal-frameworks-and-the-influence-of-ai/

27. Why Do We Need Software Escrow? \- Infosec Institute, https://www.infosecinstitute.com/resources/application-security/why-do-we-need-software-escrow/

28. What Are Software Escrow Agreements? | ESCROWSURE, https://escrowsure.com/what-are-software-escrow-agreements/

29. Understanding Intellectual Property Audit Trails and Their Importance, https://www.castlercode.com/learning-hub/understanding-intellectual-property-audit-trails-and-their-importance

30. Government Purpose Rights in Technical Data and Software, https://www.pilieromazza.com/government-purpose-rights-in-technical-data-and-software-developed-with-dod-funding/

31. 48 CFR 52.227-14 \-- Rights in Data—General. \- eCFR, https://www.ecfr.gov/current/title-48/chapter-1/subchapter-H/part-52/subpart-52.2/section-52.227-14