Development / adjudication set
Used to understand error classes, refine deterministic rules, and document reasons for acceptance, rejection, or unresolved status. Performance on this set is descriptive, not out-of-sample evidence.
Shared validation · Sequence 03
Matching Quality Benchmark
A common protocol for distinguishing rule development, frozen-holdout evaluation, and a validated linkage release across both accounting–labor data resources.
01 · Evaluation architecture
Development is not holdout, and holdout is not release
These labels identify different evidence roles. Adjudicated cases used to design rules cannot also provide an out-of-sample reliability claim. A validated release adds provenance, reproducibility, reconciliation, and independent verification to the frozen-holdout result.
Frozen holdout
Selected before final tuning, stratified across material confidence and error classes, and evaluated without changing the matching rules. This is the primary out-of-sample evidence object.
Validated release
Requires a sufficiently broad holdout plus documented disagreement resolution, tier-specific metrics, versioned checksums, a reproducible evaluation command, and fresh reconciliation of public metrics.
Ground-truth boundary
A model score is not a label. Candidate-generation recall is reported only for cases with an independently known link, and a partial estimate is never described as overall recall.
02 · Current evidence
Public benchmark evidence register
The table states the current public evidence ceiling directly. “Not measured” means that no defensible estimate is available; it is not a zero. “Not released” means that the required public evidence object has not been verified for this release.
| Evidence object | Public state | What the current release supports |
|---|---|---|
| Development/adjudication set | Not released | No public labeled case set is available for rule refinement or error analysis. |
| Frozen holdout | Not released | No sequestered evaluation set supports an out-of-sample claim. |
| Second-reviewer agreement | Not yet measured | No public double-review evidence is available. |
| Precision and false-match rate | Not yet measured | No accepted-link reliability estimate is published. |
| Candidate-generation recall | Not yet measured | No set of independently known production links is released. |
| Unresolved and no-candidate rates | Not yet measured | No production denominator has been publicly reconciled. |
| Temporal consistency | Not yet measured | No adjacent-year production evaluation is published. |
| Public-safe error taxonomy | Available | The required error classes and adjudication questions are documented without licensed rows. |
6 evidence objects shown.
| Evaluation object | Measure | Current state | Required before release |
|---|---|---|---|
| Production exact or identifier-supported accepted links | Precision and false-match rate | Not yet measured | A release-specific exact/identifier rule, not supplied by the generic reference package, plus adjudicated accepted links in a frozen holdout. |
| Rule-based accepted links | Precision by rule and parent/subsidiary error rate | Not yet measured | Rule-stratified labels and documented organizational-level decisions. |
| Fuzzy candidates sent to review | Top-1 versus top-k coverage and false-negative categories | Not yet measured | Known-link cases plus retained candidate sets. |
| Unresolved multiple candidates | Unresolved rate and ambiguity taxonomy | Not yet measured | A reconciled production denominator and adjudication reasons. |
| Adjacent-year accepted links | Temporal consistency and stale-link rate | Not yet measured | Versioned company-year links with valid intervals. |
| No-candidate source entities | No-candidate rate and candidate-generation recall | Not yet measured | Independently known links or a documented audit sample. |
03 · Metric discipline
Each metric answers a different error question
Coverage alone cannot establish linkage quality. The benchmark separates accepted-link correctness, candidate-set completeness, unresolved cases, collision structure, and adjacent-year coherence.
Precision / false-match rate
Among adjudicated accepted links, the share that are correct and incorrect. It does not measure how many true links were never generated.
Candidate-generation recall
Among cases with an independently known link, the share for which the correct target appears in the candidate set. It is distinct from final accepted-link recall.
Unresolved and no-candidate rates
The shares of eligible source entities that retain multiple plausible candidates or receive none. Their denominator must be a reconciled production population.
Top-1 versus top-k coverage
Whether the correct target is ranked first or merely retained among reviewable alternatives. This diagnoses ranking separately from blocking.
Many-to-one and one-to-many rates
How often mapping structure concentrates or branches. These rates require interpretation because valid subsidiaries can resemble over-collapsing.
Temporal consistency
Whether adjacent-year links follow documented name, ownership, and identifier histories rather than switching without support or persisting after validity ends.
04 · Error analysis
A taxonomy for false positives and false negatives
Adjudication records the reason for a decision, not only a binary label. The same taxonomy supports sampling, reviewer instructions, metric stratification, and release notes.
Legal-name variationNames
Suffixes, punctuation, abbreviations, and branding differ while the underlying entity remains the same.
Abbreviation and token lossNames
Short names or acronyms remove the tokens needed to distinguish firms.
Parent versus subsidiaryCorporate structure
The source employer and target company refer to different levels of the same corporate group.
Merger, acquisition, or spin-offCorporate structure
A relationship is valid only for part of the time series or changes organizational meaning.
Historical name changeTime
A valid name at one date becomes stale after a rename or reorganization.
Stale temporal linkTime
A candidate matches on name but falls outside the target identifier's valid interval.
Duplicate source entitiesCollisions
Multiple source records represent the same employer or business unit.
Common-name collisionCollisions
Several unrelated entities share a short or generic name.
Many-to-one concentrationCollisions
Multiple source entities map to one target and may represent either valid subsidiaries or over-collapsing.
One-to-many ambiguityCollisions
One source entity has several plausible targets without enough evidence for deterministic acceptance.
Geography or domain conflictSource coverage
Name similarity conflicts with location or website evidence.
Source missingnessSource coverage
A required blocking or validation field is unavailable, producing no candidate or lower-confidence review.
05 · Maturity gates
Labels advance only with evidence
The case counts and performance conditions below are release requirements, not claims about the current prototype. If a benchmark fails, lacks a predeclared intended-use criterion, or leaves material false-match risk unexplained, the resource retains the less mature label.
Claim-specific acceptance rule
A validated label requires more than a sufficiently large holdout. Before unblinding, each public reliability claim must state its intended use, metric, denominator, uncertainty rule, and minimum acceptable performance. A material tier or error class that fails that rule must be disclosed or must prevent the stronger label.
- Method or source authority remains incomplete.
- Validation evidence is absent, private, or insufficient.
- No reliability statement beyond directly released evidence.
- At least 150 adjudicated source decisions for the linkage.
- At least 50 cases frozen before final rule tuning.
- Material confidence tiers and error classes represented.
- Second review for at least 20% of the holdout, or a visible single-reviewer limitation.
- Holdout precision with uncertainty by material tier, plus reconciled unresolved and no-candidate rates.
- At least 300 adjudicated source decisions and 100 frozen holdout cases.
- All material error classes represented and disagreements resolved.
- Exact, rule-based, fuzzy, and manual-review tiers evaluated separately.
- Predeclared intended-use criteria satisfied by the holdout result and its uncertainty.
- No material tier or error class contradicts the claimed use without a visible limitation.
- Versioned checksum, reproducible evaluation command, and fresh public-metric reconciliation.
06 · Reproducibility
Build the benchmark without exposing licensed rows
The public package supplies the error vocabulary, generic schema, synthetic candidates, scoring demonstration, diagnostics, and release manifest. Production adjudication notes and licensed identifiers remain private; public outputs must be aggregate or independently public-safe.
Markdown
Methodology
Measurement scope, matching stages, evidence ceiling, validation gates, and disclosure boundary.
Download MethodologyCSV
Synthetic example
Reserved synthetic identifiers and .example domains for testing the reference implementation.
Download Synthetic exampleJSON Schema
Public schema
Generic input and candidate-output fields without licensed values.
Download Public schemaRunnable npm package (.tgz)
Reference implementation
Self-contained TypeScript source, package metadata, tests, license, and synthetic inputs; extract it, run npm install, then npm test.
Download Reference implementationJSON
Release manifest
Allowlisted files, byte sizes, media types, and SHA-256 checksums.
Download Release manifestCFF
Citation file
Machine-readable citation metadata for the prototype method release.
Download Citation fileText
License
License for original public-safe code and artifacts only; source-dataset rights are excluded.
Download LicenseMarkdown
Changelog
Version history and explicit statements of what each release does not contain.
Download ChangelogCitation
Yang, Hongye Chris. 2026. Open Accounting–Labor Data Linkage Hub: Public-Safe Entity-Linkage Methods and Validation Protocol, version 0.1.0.