The third of the Kiil principles for AI is a question as well.
Can you show me afterwards why the system made a particular decision?
Note the word system. I am not asking what the model answered. The model is one part of the decision. The rules, the data and the person who signed off on it are the rest.
In the last post I built a small job ad screener where rules and model answer the same question, and a human takes a look when they disagree. It works fine. But ask it a week later why a particular ad was skipped, and it has no answer.
A log is not a trail #
Most AI systems I see log prompts and responses. That is good, but it is a transcript of a conversation. It tells you what was said, and not what happened next.
A trail follows the decision. For every case I want to be able to see:
- which case it was
- what data went in
- which rules fired
- what the model was told, and what it answered
- what the outcome was
- who decided it, and why
It sounds like bureaucracy until a customer complains or the auditor calls. Then it is the only thing that counts.
In my case, the trail is what revealed that an automatic reminder was about to chase me for something I had already done. The model had done nothing wrong. Without the trail I could only have seen that the reminder was sent. With it, I could see what it was based on.
Three rules for a trail #
I stick to three rules.
Write before you act. The decision goes into the trail before anything is done with it. If it cannot be written, nothing happens. A system that acts without being able to remember it is worse than a system that stands still.
Append, never edit. The trail is a list of events. If someone changes their mind, that is a new event. The old one stays.
Humans are events too. When a person approves or rejects something, it is written down the same way as the machine’s decisions, with a name and a reason. No reason, no decision.
An example: the screener gets a trail #
Let us build on screen.nu from the last post. Every case gets a short id, and every decision becomes one line of JSON in the file decisions.jsonl.
use screen.nu [assert-not-empty parse-verdict route]
# Rules: return the keywords that matched, so the trail can say which rule fired
export def rule-hits [keywords: list<string>]: record -> list<string> {
let text = $"($in.title) ($in.description)" | str lowercase
$keywords | where {|k| $text | str contains ($k | str lowercase) }
}
# Judgement: keep the prompt and the raw answer, not just the verdict
export def model-call [profile: string]: record -> record {
let ad = $in
let prompt = [
"Is this job ad relevant for the profile below?"
'Answer only with JSON: {"relevant": true or false, "reason": "one short sentence"}'
""
$"Profile: ($profile)"
$"Title: ($ad.title)"
$"Ad: ($ad.description)"
] | str join (char newline)
let answer = ^aichat $prompt
{prompt: $prompt, answer: $answer, verdict: ($answer | parse-verdict)}
}
# One event per line. If the line cannot be written, the run stops before anything happens.
export def log-event [trail: path]: record -> record {
let fields = $in
let event = {at: (date now | format date "%+")} | merge $fields
$"($event | to json --raw)\n" | save --append $trail
$event
}
export def screen-ads [
profile: string
keywords: list<string>
--trail: path = "decisions.jsonl"
]: list<record> -> table {
assert-not-empty "The job feed"
| each {|ad|
let case = $ad | to json --raw | hash sha256 | str substring 0..7
let hits = $ad | rule-hits $keywords
let call = $ad | model-call $profile
let route = {rule: ($hits | is-not-empty), model: $call.verdict.relevant} | route
{
case: $case
event: "screened"
input: $ad
rules_fired: $hits
prompt: $call.prompt
answer: $call.answer
outcome: $route
by: "system"
why: $call.verdict.reason
} | log-event $trail | ignore
{case: $case, title: $ad.title, route: $route}
}
}
# A human decision is an event too: who decided, what and why
export def decide [
case: string
outcome: string
why: string
--trail: path = "decisions.jsonl"
] {
if ($why | str trim | is-empty) {
error make {msg: "A decision needs a reason", help: "Write one sentence your future self will understand."}
}
{case: $case, event: "decided", outcome: $outcome, by: $env.USER, why: $why}
| log-event $trail
| ignore
}
# Everything that happened to one case, in order
export def why [case: string, --trail: path = "decisions.jsonl"]: nothing -> table {
open --raw $trail | from json --objects | where case == $case
}Save it as trail.nu next to screen.nu. Two things are worth pointing out.
rule-hits returns the keywords that matched, not just true or false. That way the trail can say which rule fired.
model-call keeps the whole prompt and the raw answer. If I change the prompt later, I can still see what the model was told back then.
We run the same three ads as last time:
use trail.nu [screen-ads decide why]
[[title description];
["Nushell developer" "Automate data pipelines"]
["Sommelier" "Pair wine with food"]
["Prompt trainer" "Teach staff to use LLMs"]
]
| screen-ads "AI and automation" [nushell automation]╭───┬──────────┬───────────────────┬───────╮
│ # │ case │ title │ route │
├───┼──────────┼───────────────────┼───────┤
│ 0 │ 0adbd15a │ Nushell developer │ apply │
│ 1 │ 9981e976 │ Sommelier │ skip │
│ 2 │ 58572c95 │ Prompt trainer │ human │
╰───┴──────────┴───────────────────┴───────╯The Prompt trainer ad comes to me again. I look at it and make up my mind:
decide 58572c95 apply "Close enough to what I do."And a week later, when I have forgotten all about it:
why 58572c95 | select event outcome by why╭───┬──────────┬─────────┬────────┬───────────────────────────────────╮
│ # │ event │ outcome │ by │ why │
├───┼──────────┼─────────┼────────┼───────────────────────────────────┤
│ 0 │ screened │ human │ system │ LLM training is close to AI work. │
│ 1 │ decided │ apply │ lk │ Close enough to what I do. │
╰───┴──────────┴─────────┴────────┴───────────────────────────────────╯That is the whole story. The system was unsure because the model thought the ad was relevant and my keywords did not match. Then I decided it should go through, and wrote down why. Behind each line there is also the ad’s text, the prompt and the model’s raw answer, if I want to dig deeper.
See it fail, again #
The last post ended with the point that a check you have only seen pass is a check you do not know works. That goes for the trail too. So I test that it stops when it should:
use std/assert
use trail.nu [screen-ads log-event decide why]
# A trail that cannot be written stops the run
assert error { {case: "x"} | log-event /no/such/dir/decisions.jsonl }
# A decision without a reason is refused
assert error { decide "x" "apply" " " --trail test.jsonl }
# Every screened ad leaves an event that can be found again
let screened = [{title: "Sommelier", description: "Pair wine with food"}]
| screen-ads "AI" [nushell] --trail test.jsonl
assert equal (why $screened.0.case --trail test.jsonl | get event) ["screened"]The first test is the most important one. If the trail cannot be written, you get an error, and no ad moves on. The second makes sure “because” is never empty.
The law and the board #
The EU AI Act requires logging for high-risk systems. But law or no law, management cannot take responsibility for something it cannot see. And it is easier to build the trail from the start than to bolt it on later, when someone asks.
To sum up #
When you build something with a model in it, make sure you can answer afterwards why the system did what it did:
- Follow the decision. Keep the case, the input, the rules, the prompt, the answer, the outcome and who decided.
- Write before you act. If the decision cannot be written, nothing happens.
- Append, never edit. A change of mind is a new event.
- Require a reason. Also when a human is the one deciding.
Next up is the fourth and last principle: which actions require a human, and why those in particular?
If you would like help giving your own systems a trail, send me an email.