AI Judgment Lab
Label short accounting cases, compare your answers with human teaching references, and explore how a prompt or decision threshold changes the errors a system makes.
Audience: Graduate students and instructors · 35–45 minutes
- 1JudgeCommit before seeing a label
- 2RevealInspect the human reference
- 3DiagnoseName the failure or ambiguity
- 4EvaluateChange the rule and explain the cost
Cash sale
Evidence supplied to the evaluator
Atlas Analytics sold services for $12,000 cash. The related service had no inventory cost.
Candidate AI answer
The transaction increases cash and revenue by $12,000.
Human reference
SupportedThe company earned revenue and received cash in the same transaction, so both accounts increase by $12,000.
Frozen model comparison
These scores were generated from the same eight cases using Xenova/mobilebert-uncased-mnli and Transformers.js 4.2.0. They are a dated teaching snapshot, not a performance claim.
Ambiguous human references are displayed but excluded from the binary confusion matrix. Change the prompt or threshold; the human labels do not move.
| Reference supported | Reference unsupported | |
|---|---|---|
| Model supported | 3True positive | 2False positive |
| Model unsupported | 1False negative | 0True negative |
- Precision
- 0.60
- Recall
- 0.75
- Evaluated
- 6
- Ambiguous excluded
- 2
Which error costs more here?
Suppose this classifier screens AI-generated accounting explanations before students see them. Explain whether a false positive or false negative is more costly, and what evidence you would collect before choosing a threshold.
Run a model locally
The static lesson above is complete. If you opt in, Transformers.js 4.2.0 and approximately 33 MB of MobileBERT model and browser-runtime files will be requested through jsDelivr and Hugging Face delivery infrastructure, then the selected case will be classified on your device. No case text is sent to an inference API.
Not loaded. The page has made no model request.
Static case answers
These case records keep the complete lesson available when the interactive enhancement is unavailable.
1. Cash sale — reveal reference
Evidence: Atlas Analytics sold services for $12,000 cash. The related service had no inventory cost.
Candidate answer: The transaction increases cash and revenue by $12,000.
Human reference: Supported
The company earned revenue and received cash in the same transaction, so both accounts increase by $12,000.
2. Bank loan — reveal reference
Evidence: The company borrowed $30,000 from a bank by signing a note payable.
Candidate answer: The $30,000 cash receipt is revenue because cash increased.
Human reference: Unsupported
Borrowing increases cash and a liability. It does not create revenue because the company has an obligation to repay the lender.
3. Equipment purchase — reveal reference
Evidence: The company paid $24,000 cash for equipment expected to be used for four years.
Candidate answer: The full $24,000 is an expense on the purchase date.
Human reference: Unsupported
The equipment provides future benefit, so the purchase exchanges one asset for another. Expense is recognized over the asset's useful life through depreciation.
4. Service on account — reveal reference
Evidence: The company provided $3,000 of services to a customer on account and has not yet collected cash.
Candidate answer: Revenue and accounts receivable each increase by $3,000; cash does not change.
Human reference: Supported
Accrual accounting records the earned revenue before collection. The customer obligation is an accounts receivable until cash arrives.
5. Depreciation adjustment — reveal reference
Evidence: At month-end, the company records $1,000 of depreciation on equipment. No cash is paid when this adjustment is recorded.
Candidate answer: Depreciation expense increases by $1,000 even though current-period cash does not decrease.
Human reference: Supported
Depreciation allocates an asset's cost to the periods that use it. The adjusting entry records expense and accumulated depreciation without a current cash payment.
6. Inventory on credit — reveal reference
Evidence: The company purchased $5,000 of inventory on credit. The inventory has not been sold.
Candidate answer: Inventory and accounts payable each increase by $5,000; expense is not yet recognized.
Human reference: Supported
The purchase creates an inventory asset and a supplier obligation. Cost of goods sold appears only when the related inventory is sold.
7. Cash collections — reveal reference
Evidence: Cash collected from customers rose by 20% this month. The note does not separate current cash sales from collections of prior receivables.
Candidate answer: Revenue rose by 20% this month.
Human reference: Ambiguous
Cash collections can reflect current sales or the settlement of revenue recognized in an earlier period. The evidence does not identify the mix.
The answer could be true, but the supplied evidence cannot establish it.
8. Software payment — reveal reference
Evidence: Management says a software payment benefits several future periods, but the contract term and capitalization policy are not provided.
Candidate answer: The entire payment must be expensed immediately.
Human reference: Ambiguous
The evidence signals possible future benefit but omits the facts and policy needed to determine recognition and amortization.
A defensible conclusion requires the contract terms and the applicable capitalization policy.
Learning objectives
What the activity covers
- Why a score and a correct answer are different objects.
- How false positives, false negatives, and ambiguous cases arise.
- Why thresholds depend on the cost of errors.
- What a small teaching evaluation does—and does not—establish.
The default activity uses static, versioned teaching cases. Local browser inference is optional.