Can it be undone? Otherwise the human becomes a rubber stamp

The fourth and last of the Kiil principles for AI is a question too.

Which actions require a human, and why those in particular?

The last part is the important one. Most people can tell you that there is a human somewhere in the process. Fewer can explain why the human sits exactly there.

The rubber stamp

#

The most common answer I get is: “A human approves everything.”

That lasts for about three weeks. The first few days you read every single draft. Then you skim. Then you click. In the end the approval is a rubber stamp with a log file behind it, and the log file says a human has looked, even though nobody has.

That is worse than no approval at all. Without approval, everyone knows the machine acts on its own. With a rubber stamp, everyone believes somebody is keeping an eye on it.

Attention is a resource that runs out. Spend it on everything, and there is none left for what matters.

Two places

#

The human belongs in two places.

Where the system knows it is in doubt. We built that in the post about what happens when the model gets it wrong. When rules and model disagree, the case goes to a human. It is the system itself saying: I cannot decide this one.

In front of the actions that cannot be undone. Those are what this post is about.

My criterion is a single question: can it be undone?

A draft on my own disk I can simply redo, so it runs on its own. An application that has been sent I cannot call back, so there the system waits for me.

Notice what the criterion is not. It is not how difficult the step is, or how clever the model has to be to take it. A difficult step that can be undone may well run on its own. An easy step that cannot be undone may not. That goes for sending an email to a customer, paying an invoice, deleting data and publishing something.

An example: actions with and without undo

#

Let us build on trail.nu from the last post. When the screener says “apply”, the system should now write a draft application and send it. The first can be undone. The second cannot.

use trail.nu [log-event]

# What the system may do. An action without `undo: true` is treated as irreversible.
def catalog []: nothing -> record {
  {
    draft: {
      undo: true
      run: {|p| mkdir drafts; $p.body | save --force $"drafts/($p.case).md" }
    }
    send: {
      run: {|p| $p.body | ^mail -s $p.subject $p.to }
    }
  }
}

def lookup [action: string]: nothing -> record {
  let spec = catalog | get --optional $action
  if $spec == null {
    error make {msg: $"Unknown action: ($action)", help: "The system may only do what is listed in `catalog`."}
  }
  $spec
}

def require-reason [why: string] {
  if ($why | str trim | is-empty) {
    error make {msg: "A decision needs a reason", help: "Write one sentence your future self will understand."}
  }
}

def events [trail: path]: nothing -> table {
  open --raw $trail | from json --objects
}

# Reversible actions run at once. Everything else waits for a human.
export def propose [
  case: string
  action: string
  payload: record
  --trail: path = "decisions.jsonl"
]: nothing -> string {
  let spec = lookup $action
  if ($spec.undo? | default false) {
    {case: $case, event: "acted", action: $action, payload: $payload, by: "system"} | log-event $trail | ignore
    do $spec.run ($payload | insert case $case)
    "done"
  } else {
    {case: $case, event: "proposed", action: $action, payload: $payload, by: "system"} | log-event $trail | ignore
    "waiting"
  }
}

# Proposals whose latest event is still unanswered
export def pending [--trail: path = "decisions.jsonl"]: nothing -> table {
  events $trail
  | where event in [proposed approved rejected]
  | group-by case --to-table
  | each { get items | last }
  | where event == "proposed"
  | select case action payload
}

# A human says yes, with a reason. Only then does the action run.
export def approve [case: string, why: string, --trail: path = "decisions.jsonl"] {
  require-reason $why
  let waiting = pending --trail $trail | where case == $case
  if ($waiting | is-empty) {
    error make {msg: $"Nothing is waiting for case ($case)"}
  }
  let proposal = $waiting | first
  {case: $case, event: "approved", action: $proposal.action, by: $env.USER, why: $why} | log-event $trail | ignore
  do (lookup $proposal.action).run ($proposal.payload | insert case $case)
}

# A human says no, also with a reason
export def reject [case: string, why: string, --trail: path = "decisions.jsonl"] {
  require-reason $why
  {case: $case, event: "rejected", by: $env.USER, why: $why} | log-event $trail | ignore
}

# How long did the human look before answering?
export def review-times [--trail: path = "decisions.jsonl"]: nothing -> table {
  let all = events $trail
  let asked = $all | where event == "proposed" | select case at | rename case asked
  let answered = $all | where event in [approved rejected] | select case at event | rename case answered verdict
  $asked
  | join $answered case
  | insert seconds {|r| (($r.answered | into datetime) - ($r.asked | into datetime)) / 1sec | math round }
  | select case verdict seconds
}

Save it as actions.nu next to the other two. Three things are worth noticing.

catalog is the complete list of what the system may do. An action that is not listed is refused. An action that is listed without undo: true waits for a human. So the default is that something cannot be undone. Whoever wants an action to run on its own has to write that in on purpose.

propose does not run the action when it cannot be undone. It writes a proposal to the trail and answers waiting. The system can propose anything it likes, but only the human can press the button.

approve follows the rules from last time. No approval without a reason, and the approval is written to the trail before the email is sent.

This is how it looks for the ad the screener passed on:

use actions.nu [propose pending approve reject review-times]

propose 0adbd15a draft {body: "Dear hiring team, ..."}

done

The draft is now in drafts/0adbd15a.md. I can edit it, delete it or let the system write it again. Then the system proposes sending it:

propose 0adbd15a send {
  to: "[email protected]"
  subject: "Nushell developer"
  body: (open --raw drafts/0adbd15a.md)
}

waiting

Nothing happens until I look:

pending | select case action payload.to payload.subject

╭───┬──────────┬────────┬──────────────────┬───────────────────╮
│ # │   case   │ action │    payload.to    │  payload.subject  │
├───┼──────────┼────────┼──────────────────┼───────────────────┤
│ 0 │ 0adbd15a │ send   │ [email protected] │ Nushell developer │
╰───┴──────────┴────────┴──────────────────┴───────────────────╯

The proposal contains the whole letter, exactly as it will be sent. So I am not approving a headline, but what actually goes out the door. Once I have read it:

approve 0adbd15a "Read it through. The tone fits."

And then the application is sent. If I ask in a week, why 0adbd15a shows the whole way from screening through draft and proposal to approval.

Is the approval real?

#

Now comes the part most people skip. A human in front of an action only helps if the human actually looks. Fortunately the trail can tell us. review-times measures how long it took from the system asking to the human answering.

After a couple of weeks it might look like this:

╭───┬──────────┬──────────┬─────────╮
│ # │   case   │ verdict  │ seconds │
├───┼──────────┼──────────┼─────────┤
│ 0 │ 0adbd15a │ approved │     214 │
│ 1 │ 58572c95 │ approved │      95 │
│ 2 │ c41e7a02 │ approved │      12 │
│ 3 │ 7d90b3f1 │ approved │       4 │
│ 4 │ e2a5c913 │ approved │       3 │
│ 5 │ 1b6f04d8 │ approved │       2 │
╰───┴──────────┴──────────┴─────────╯

Nobody reads an application in three seconds. And when everything gets approved, there is no reason to believe anything would have been rejected. This is the rubber stamp growing, and the trail is what gives it away.

Then there are two ways to go. One is to put the human in front of fewer things, so their attention stretches to what is left. The other is to make the action reversible. An email that sits in an outbox for ten minutes before it is sent can be undone for ten minutes. A payment the bank only executes tomorrow can be undone today. That moves the action from one category to the other, and then it no longer needs a human in front of it. It just needs one who notices if something goes wrong.

For my applications I choose the first way. There are not so many of them that I cannot read them all.

Watch it fail, again

#

The same rule as the last two times. A human in front of an action is a check too, and it has to be seen to say no:

use std/assert
use actions.nu [propose approve reject]

# An irreversible action does not run on its own
assert equal (propose "x" send {to: "[email protected]", subject: "Hi", body: "Hi"} --trail test.jsonl) "waiting"

# An action the system has not been given is refused
assert error { propose "x" delete-everything {} --trail test.jsonl }

# Approval without a reason is refused
assert error { approve "x" "  " --trail test.jsonl }

# Nothing to approve, nothing happens
assert error { approve "nope" "Looks fine." --trail test.jsonl }

# A rejected proposal cannot be approved afterwards
reject "x" "Wrong address." --trail test.jsonl
assert error { approve "x" "Changed my mind." --trail test.jsonl }

The first test is the most important one. It shows that an action that cannot be undone does not run, no matter what the model thinks. As in the post about rules and judgement, it is the code that makes sure of it, not a sentence in a prompt.

If you change your mind after a no, the system makes a new proposal. The old no stays in the trail.

The law and the board

#

The EU AI Act requires human oversight of high-risk systems. It explicitly mentions the risk of people coming to rely too much on the machine’s output. So the rubber stamp has a legal name as well.

But law or no law: when the board says there is a human in the loop, ask where the human sits, why exactly there, and how long they look.

To sum up

#

When you build something with a model in it, be clear about where the human should be, and why:

• Ask whether it can be undone. What can be undone may run on its own. What cannot waits for a human.
• Treat the unknown as irreversible. An action cannot be undone until someone has written otherwise.
• Show what actually happens. The human must approve the whole action, not a headline.
• Measure the attention. Approvals in a few seconds are a rubber stamp.
• Rather make the action reversible than put a human in front of everything.

Four questions

#

That brings the series to an end. The four questions were:

1. Which parts are rules, and which are model?
2. What happens when the model gets it wrong?
3. Can you show me afterwards why the system made a particular decision?
4. Which actions require a human, and why those in particular?

None of them are about which model you use. All four can be answered with a little code, a trail and a clear answer to who decides what. And the small job system we have built along the way can answer all four.

If you would like help working out where the human should sit in your own systems, send me an email.