i was trying to prepare a quotation.
that is how i found the problem.
nothing philosophical.
nothing about artificial general intelligence.
nothing about alignment.
a quotation.
there is a prescribed scale of fees for licensed survey work in kenya.
fine.
i have the regulations.
the machine can read them.
better than most people probably can.
regulation 35 says the fees in the eighth schedule shall be charged.
it goes further.
a licensed surveyor who charges, attempts to charge, offers to work at, or even holds himself out as prepared to work at another survey fee commits an offence.
fine again.
then you reach the eighth schedule.
transport and travelling when using your own vehicle:
80 shillings per kilometre.
another 40 shillings per kilometre for the surveyor's professional time.
(Survey Regulations, L.N. 168 of 1994, Eighth Schedule, paragraph 5.)
i stared at it.
then stared at the fuel price.
😂
what exactly am i supposed to do with this information?
because the machine is not wrong.
that is what bothers me.
it found the law.
it cited the law.
it followed the law.
and if you build the business exactly around what it found, something has gone very badly wrong.
the document is current
this would be easier if i had simply found an old pdf somewhere.
i could shrug.
outdated copy.
problem solved.
except the copy sitting in my documents is the consolidated legislation as at 25 april 2024.
it contains amendments from 2020.
some parts have clearly moved with time.
and sitting inside that current instrument is the eighth schedule, carrying the amendment notation from 1999, telling the surveyor 80 shillings per kilometre.
this is almost more interesting than if the whole law had simply been abandoned.
the law has been maintained.
parts of it have been touched.
the number survived.
so now the machine has something even stronger than an old document.
it has an apparently current one.
what should it conclude?
exactly what it concluded.
everybody else knows
this is where humans become annoying.
ask an actual surveyor whether 80 shillings a kilometre covers transport today.
he does not need a legal research exercise.
he has bought fuel.
he knows.
the correction exists.
it just does not exist in the document.
it exists in people's heads.
it exists in invoices.
it exists in what firms actually charge.
it exists in the mildly irritated look you would receive if you suggested driving hundreds of kilometres at the statutory rate.
that knowledge is everywhere.
and nowhere.
nobody sat down and wrote:
dear future machine, this figure stopped making economic sense years ago.
why would they?
humans simply adapted.
that is what we do.
the law remained there.
practice moved.
everyone involved learned the gap.
then along comes a machine whose great advantage is that it can finally read all the rules.
oh dear.
i thought the problem was classification
i wrote recently that the classifier is not a judge.
i meant it in the context of transcripts.
a classifier can measure resemblance to some learned category and act on the result without ever engaging the particular case in the way judgment requires.
i still think that.
i just think i made the argument too small.
because this is the same problem from another direction.
the machine has a reference.
80 shillings per kilometre.
the world has moved.
the surveyor who follows the reference becomes economically ridiculous.
the surveyor who survives has departed from the reference.
now imagine scoring them.
one is compliant.
one is deviant.
the machine can be perfectly correct about the distance from the rule and completely useless about which behaviour makes sense.
that is the wall.
judgment contains an ugly little human move
human judgment has a move that is difficult to write into a rule.
this rule has stopped making sense here.
we do this constantly.
sometimes legitimately.
sometimes corruptly.
sometimes lazily.
sometimes because there is genuinely no other way for the system to continue functioning.
a police officer sees an offence and gives a warning.
a regulator knows a requirement is being phased in and does not immediately destroy everyone who has not complied.
a tax officer distinguishes an obvious error from fraud.
a judge looks at a literal reading and asks what absurdity follows from it.
a professional reads an old schedule and quietly refuses to bankrupt himself.
none of this sits comfortably inside the fantasy that society works because everybody reads the rules and follows them exactly.
society works partly because human beings continuously interpret the distance between written rules and the world in which those rules have to operate.
that should make us uncomfortable.
it should also make us very careful before automating the discomfort away.
imagine the eye actually worked
we keep talking about AI giving institutions better visibility.
better fraud detection.
better regulatory compliance.
better monitoring.
better enforcement.
the all-seeing eye.
i keep wondering whether anyone has seriously considered what happens if the eye becomes very good.
because i suspect most functioning societies contain an enormous amount of tolerated non-compliance.
small things.
technical things.
outdated things.
things nobody knows are offences.
things everybody knows are offences and nobody enforces literally.
things where compliance would cost more than whatever harm the rule was designed to prevent.
things that became normal because the law was never updated after the world changed.
if we suddenly became capable of identifying every violation continuously, i am not convinced the result would look like the triumph of the rule of law.
it might look completely insane.
how many people would be committing some technical offence at this exact moment if every regulation, licence condition, tax rule, building requirement, employment provision, traffic rule and professional standard were applied continuously and literally?
i have absolutely no desire to find out from inside a cell.
😂
and that is before we get to countries where the statute book has accumulated decades of rules faster than the state has accumulated the capacity to revise them.
enforcement capacity was part of the law
this is the bit i had never really considered.
we normally treat limited enforcement as a defect.
perhaps sometimes it is also part of the equilibrium.
the law says many things.
the state can inspect only some things.
humans exercise discretion over others.
people develop practices around what is actually required.
courts resolve the ugly cases.
professional communities develop tacit interpretations.
life continues.
then AI changes the cost of observation.
suddenly you can inspect everything.
every invoice.
every conversation.
every licence.
every payment.
every transaction.
every report.
every filing.
every quotation.
every little departure from the written rule.
that does not merely improve enforcement.
it changes what the law is in practice.
a statute written in a world where one inspector could examine 200 files may behave very differently in a world where a machine can examine 200 million.
the text has not changed.
the power behind it has.
i am beginning to think legislation has assumptions about enforcement bandwidth buried inside it.
nobody wrote those assumptions down either.
then there are impossible rules
the payment provisions annoyed me for another reason.
the eighth schedule does not merely specify amounts.
it specifies when charges become payable.
75 percent when the survey is completed and the plan deposited with the director.
the remaining 25 percent when the survey is approved.
a deposit may also be required when accepting the commission.
(Survey Regulations, Eighth Schedule, paragraph 2.)
now enter an ordinary human being known as a client.
the regulation can determine when money is payable.
it cannot make the client pay.
anyone who has operated a business understands the difference immediately.
the invoice is due.
lovely.
the customer has apparently developed an alternative theory of time.
😂
this sounds trivial until you try to convert law into a machine-readable compliance state.
what is the state?
compliant?
non-compliant?
awaiting payment?
whose compliance?
the professional did the work.
the client did not pay.
the regulatory requirement describes an outcome involving two actors, only one of whom the system is scoring.
now things become messy very quickly.
the machine wants a state.
reality gives you a relationship.
there may be no compliant action
this is what worries me more.
some regulation is difficult.
some regulation is outdated.
then there is regulation that produces a situation where every available action violates something.
that is different.
suppose one rule requires you to perform an action.
another actor must cooperate before you can perform it.
they refuse.
what exactly does compliance mean now?
try harder?
wait forever?
break another rule?
stop trading?
lawyers spend careers inside questions like this.
machines hate them.
not because machines are necessarily less intelligent.
because software eventually needs a state.
TRUE.
FALSE.
VALID.
INVALID.
PASS.
FAIL.
ESCALATE.
but sometimes the correct description of the world is:
this requirement applies, the actor cannot presently satisfy it, the failure depends on another person, existing practice has developed around the problem, enforcement is inconsistent, and nobody has resolved the contradiction formally.
good luck putting that in a green tick.
now move away from surveying
this is where my small quotation problem became something else.
because surveying is hardly special.
take any old regulated industry.
construction.
medicine.
transport.
hospitality.
banking.
employment.
food.
land.
tax.
how much of the real operating system exists outside the written law?
professional custom.
regulator guidance.
telephone calls.
accepted practice.
things everyone has agreed to stop doing despite the regulation remaining untouched.
things everyone does despite the regulation saying something awkward.
court decisions.
forms.
departmental interpretations.
a guy called patrick who has been at the ministry since 1987 and knows what the form actually means.
😂
the machine sees the documents.
patrick sees the institution.
those are different datasets.
the missing corpus
this has become one of the more interesting things about AI to me.
people speak about the world's knowledge as though the world's knowledge has been written down.
it hasn't.
not remotely.
some of the most important information in any profession exists as:
don't do it that way.
why?
it doesn't work.
where is that written?
nowhere.
who told you?
walter.
who told walter?
probably some angry surveyor in 1998.
that is knowledge.
very high quality knowledge sometimes.
it has survived contact with reality for decades.
and a frontier model may know considerably less about it than an old man with bad handwriting.
not because the model is stupid.
because nobody gave the model anything to read.
the absence is not marked.
there is no document called:
things the statute says that everybody quietly stopped doing.
although perhaps there should be.
and the error will not be evenly distributed
this is the part that worries me most.
the gap between written reality and lived reality is not the same everywhere.
newer regulatory systems will have newer documents.
wealthier jurisdictions produce more guidance.
large industries produce commentary.
important court decisions are digitised.
major companies publish policies.
regulators issue FAQs.
professional bodies update practice notes.
now move somewhere where the governing instrument is thirty years old.
the practitioners adapted.
the documentation did not.
which means the machine may become more confident as the quality of its picture gets worse.
it has the statute.
excellent source.
official.
current consolidated copy.
high confidence.
meanwhile the correction is sitting in the heads of ten thousand working professionals who have never put it online.
the poorer the information environment, the stronger the temptation to mistake documentary completeness for actual completeness.
and those may also be the places with the least institutional capacity to challenge an automated decision.
that combination is ugly.
the future is going to make old law very strange
we are discussing AI regulation as though the only problem is writing new laws for AI.
i am beginning to think AI is going to force us to confront enormous amounts of old law too.
because machines make literalism cheap.
an intelligent system can read every requirement.
cross-reference every transaction.
check every deadline.
flag every discrepancy.
and suddenly rules that survived for decades because humans applied them selectively start behaving exactly as written.
the law wakes up.
all of it.
at once.
that may not be progress.
there is a beautiful assumption hiding inside much of the compliance discussion:
more enforcement of the law means more fidelity to the purpose of the law.
my 80 shillings says otherwise.
sometimes perfect enforcement preserves the mistake perfectly.
this is not an argument for ignoring law
quite the opposite.
the answer cannot be:
the law is inconvenient, do whatever makes sense.
that road gets ugly very quickly.
the written rule matters precisely because discretion can become corruption.
if every professional gets to announce that the law is outdated whenever it costs him money, we have solved one problem by creating a much larger one.
so the machine has found something real.
80 means 80 until something with legitimate authority says otherwise.
that is why this bothers me.
the absurdity is legally meaningful.
the economic reality is also meaningful.
the system has failed to reconcile them.
humans have simply been carrying the contradiction manually.
AI did not create the contradiction.
it made it difficult for me to ignore.
maybe this is what the frontier is actually exposing
i started with a quotation.
i wanted the quotation to obey the law.
that seemed like a reasonable thing to ask.
the machine found the law.
the law produced an answer that did not survive the trip to the petrol station.
then i discovered that the problem was not really the quotation.
it was everything sitting between text and practice.
judgment.
discretion.
tacit knowledge.
institutional memory.
non-enforcement.
impossible obligations.
outdated numbers.
rules whose purpose everyone understands but whose literal wording no longer fits the world.
we have built human systems around all of this mess.
now we are introducing machines that are extraordinarily good at reading what we wrote.
i am not sure we appreciated how much civilization depends on what we did not write.
and if we are going to give machines the authority to watch, score, refuse, approve, report and eventually enforce, then updating the model will not be enough.
we may have to update the world it is reading.
because the machine found the law.
the embarrassing part is that the law found us first.