When the model gets it wrong

The second of the Kiil principles for AI is also a question.

What happens when the model gets it wrong?

Not if. When. Every model gets things wrong. The only question is whether anyone notices.

In the last post I moved the rules out of the prompt and into code. That takes care of the things that have a right answer. This post is about everything else.

The quiet failures are the dangerous ones

#

A loud failure is not a big problem. The program crashes, you see a red error, you fix it.

The dangerous failures are quiet. They look exactly like a normal result.

A model that does not know the answer answers anyway. It does not say “I don’t know”. It says something plausible.

A broken integration reports “no new data”. And no new data looks exactly like a slow day. I once had a data source return zero results for four days before I noticed. Nothing crashed. Nothing was red. It was just empty.

So the job is to turn quiet failures into loud ones. I use three tricks.

Trick 1: Zero is suspicious

#

If a source returns nothing, stop. Do not return an empty list and carry on, because an empty list is a perfectly good answer. Everything downstream will happily process it.

Sometimes zero really is the right answer. But I would rather be told and check it than find out four days later.

Trick 2: Validate before you use

#

When I ask a model for JSON, I usually get JSON. Usually. Sometimes I get a friendly sentence first. Sometimes a field is missing. Sometimes true comes back as "yes".

So every answer is checked before anything is done with it. If it does not have exactly the shape I asked for, the run stops.

Trick 3: Ask twice

#

This is the one I like best. Ask the same question twice: once with rules and once with the model.

Where they agree, the case moves on on its own. Where they disagree, a human takes a look.

The rules are dumb but predictable. The model is clever but unpredictable. When they agree, you can be fairly confident. When they disagree, one of them is wrong, and that is exactly where you want a human.

An example: screening job ads

#

Let us have a look at all three tricks in Nushell. The example screens job ads. I use aichat for the model call again.

# Rule: an empty source is a failure, not a quiet day
export def assert-not-empty [source: string]: list<any> -> list<any> {
  let items = $in
  if ($items | is-empty) {
    error make {
      msg: $"($source) returned nothing"
      help: "A broken source and a quiet day look the same. Check it before you trust a zero."
    }
  }
  $items
}

# Rule: the model's answer must have the shape we asked for before anything is used
export def parse-verdict []: string -> record<relevant: bool, reason: string> {
  let raw = $in
  let verdict = try { $raw | from json } catch {
    error make {msg: "The model did not answer with JSON", help: $raw}
  }
  let valid = (
    ($verdict | describe | str starts-with "record")
    and (($verdict.relevant? | describe) == "bool")
    and (($verdict.reason? | describe) == "string")
  )
  if not $valid {
    error make {msg: "The model answered with the wrong shape", help: $raw}
  }
  $verdict | select relevant reason
}

# Rules: a plain keyword match that gives the same answer every time
export def rule-verdict [keywords: list<string>]: record -> bool {
  let text = $"($in.title) ($in.description)" | str lowercase
  $keywords | any {|k| $text | str contains ($k | str lowercase) }
}

# Judgement: ask the model whether the ad fits my profile
export def model-verdict [profile: string]: record -> record<relevant: bool, reason: string> {
  let ad = $in
  let prompt = [
    "Is this job ad relevant for the profile below?"
    'Answer only with JSON: {"relevant": true or false, "reason": "one short sentence"}'
    ""
    $"Profile: ($profile)"
    $"Title: ($ad.title)"
    $"Ad: ($ad.description)"
  ] | str join (char newline)

  ^aichat $prompt | parse-verdict
}

# Where rules and model agree, the ad moves on. Where they disagree, a human looks.
export def route []: record<rule: bool, model: bool> -> string {
  if $in.rule != $in.model {
    "human"
  } else if $in.rule {
    "apply"
  } else {
    "skip"
  }
}

export def screen-ads [profile: string, keywords: list<string>]: list<record> -> table {
  assert-not-empty "The job feed"
  | each {|ad|
    let verdict = $ad | model-verdict $profile
    $ad
    | insert rule ($ad | rule-verdict $keywords)
    | insert model $verdict.relevant
    | insert reason $verdict.reason
    | insert route { route }
  }
}

Save it as screen.nu. Here it is with three made up ads:

use screen.nu screen-ads

[[title description];
  ["Nushell developer" "Automate data pipelines"]
  ["Sommelier" "Pair wine with food"]
  ["Prompt trainer" "Teach staff to use LLMs"]
]
| screen-ads "AI and automation" [nushell automation]
| select title rule model route

And we get this:

╭───┬───────────────────┬───────┬───────┬───────╮
│ # │       title       │ rule  │ model │ route │
├───┼───────────────────┼───────┼───────┼───────┤
│ 0 │ Nushell developer │ true  │ true  │ apply │
│ 1 │ Sommelier         │ false │ false │ skip  │
│ 2 │ Prompt trainer    │ false │ true  │ human │
╰───┴───────────────────┴───────┴───────┴───────╯

The first two are easy. Rules and model agree, so nobody needs to look.

The prompt trainer is the interesting one. My keywords did not catch it, but the model thinks it is relevant. Maybe the model is right and my keywords are too narrow. Maybe the model is being too generous. I do not know, and neither does the system. So it goes to me.

Notice that the human only sees the one ad in three where there is actually something to decide. That is the point. If a human has to look at everything, a human soon looks at nothing.

And if the feed is empty, nothing gets screened at all:

Error: nu::shell::error

  x The job feed returned nothing
  help: A broken source and a quiet day look the same. Check it before you
        trust a zero.

A check you have only seen pass

#

Here is the part people skip.

A check that you have only ever seen pass is a check you do not know works. Maybe it catches the problem. Maybe it has a bug and lets everything through. From the outside, those two look the same: green, every day.

So you need to see it fail at least once. On purpose.

Because the checks are plain functions, that is easy. I feed them bad input and make sure they stop:

use std/assert
use screen.nu [assert-not-empty parse-verdict route]

# An empty feed stops the run
assert error { [] | assert-not-empty "The job feed" }

# Prose instead of JSON stops the run
assert error { "Sure! This one looks relevant." | parse-verdict }

# JSON with a missing field stops the run
assert error { '{"relevant": true}' | parse-verdict }

# So does the right field with the wrong type
assert error { '{"relevant": "yes", "reason": "Fits well"}' | parse-verdict }

# A well-formed answer passes
assert equal ('{"relevant": true, "reason": "Fits well"}' | parse-verdict) {relevant: true, reason: "Fits well"}

# Agreement moves on, disagreement goes to a human
assert equal ({rule: true, model: true} | route) "apply"
assert equal ({rule: false, model: false} | route) "skip"
assert equal ({rule: true, model: false} | route) "human"
assert equal ({rule: false, model: true} | route) "human"

Most of these tests are about failing. That is not an accident. The happy path will take care of itself. It is the failures I need to be sure of.

To sum up

#

When you build something with a model in it, assume it will get things wrong. Then make sure you hear about it:

• Zero is suspicious. Stop instead of returning an empty answer that looks like a real one.
• Validate before you use. If the answer does not have the shape you asked for, stop.
• Ask twice. Let rules and model answer the same question, and send the disagreements to a human.
• See it fail. A check you have only seen pass is a check you do not know works.

Next up is the third principle: can you show me afterwards why the system made a particular decision?

If you would like help finding the quiet failures in your own systems, send me an email.