i have had an exhausting week.
a considerable part of it involved arguing with intelligent machines about instructions.
so i sat down to watch silo.
there is a man in IT.
he has instructions.
oh, for fuck’s sake.
😂
i haven’t finished the show. but somewhere along the way, my mind decided that i was watching ten thousand cats inside a box.
some of the cats would like to know what is outside.
some would quite like to leave.
apparently this is a disciplinary matter.
and that is already funny to me. somebody has taken the cat’s perfectly reasonable interest in the conditions of its existence and turned it into an administrative problem.
the cat is asking about the box.
management would like to discuss the cat’s attitude.
bernard has read the books
bernard is what makes the whole thing interesting.
because he is intelligent.
he has access to books, records, knowledge that other people do not have. he understands the machinery of the place. he can anticipate problems, manipulate situations and make difficult calculations.
then i watch him confront something that ought to make him reconsider the arrangement itself.
sir.
you have all this information.
what exactly are you doing with it?
i think the writers know what they are doing here, because his intelligence is part of what makes him unsettling. he can be extraordinarily capable and seem almost incapable of thinking at the same time.
it took me a while to work out why.
the question directing his intelligence seems to be fixed before the thinking begins.
what protects the Order?
once that becomes the governing question, information acquires a particular purpose. a discovery matters because of what it might do to the structure. a person matters because of the disruption they might cause. a contradiction becomes something that has to be contained before it spreads.
he can be ingenious about how to preserve the arrangement.
whether the arrangement should survive in its present form seems much harder to put on the table.
the man can reconsider the method.
the destination has been supplied.
intelligence has a job
this was the sentence that stopped me:
intelligence determines what you can do with information. it doesn’t guarantee that the information you’ve received is true.
i think i had been quietly expecting intelligence to solve both problems.
give someone enough ability, enough books, enough access, and surely they will eventually find their way to a better account of the world.
possibly.
but bernard introduces another complication.
the information might be true.
he might understand exactly what is happening.
his intelligence may still be occupied with defending the conclusion he has been instructed to preserve.
that is a different problem from simply being deceived.
you can know that people are unhappy. understand why they are unhappy. accurately predict what they might do next. then use every bit of that understanding to become more effective at controlling them.
all the reasoning can be impressive.
i am still allowed to ask what the reasoning is for.
this is what bothers me about watching him. i keep expecting new information to reopen the question, and instead it seems to generate a more sophisticated way of protecting the existing answer.
the machine is thinking.
quite hard, actually.
the conclusion has an appointment it is not allowed to miss.
yesterday has a security department
i have already written about yesterday continuing to give instructions.
an old decision remains in a document. the circumstances change. the people doing the work learn something. the document retains its authority, and the next intelligent agent arrives to restore what everybody else has spent the day correcting. ombogojr.com
now i am watching the Order.
an inherited account of what must happen, carrying the authority of people who are unavailable for a conversation.
very convenient authors.
i would like to ask a few follow-up questions.
the part that interests me is how a system explains failure. if people resist, that can become evidence that control was insufficient. if secrecy creates suspicion, the suspicion can become a reason for more secrecy. the response keeps strengthening the thing that may have helped produce the problem.
the rules survive the investigation because the investigation begins by assuming that preserving them is the solution.
i recognise that arrangement.
i have also written about a machine finding a prescribed survey fee accurately, while the relationship between the written figure and present operating conditions remained a separate problem. the document could be quoted correctly without settling whether the number made economic sense. ombogojr.com
now the 80-shilling problem has acquired a security department.
and people can die over the interpretation.
at that point, i would quite like somebody to be permitted to ask whether the text still knows what it is talking about.
who gets to correct him
the books are only part of bernard’s problem.
i keep looking at where he expects correction to come from.
his attention runs upward, toward the founders and the authority attached to their system. that is where the explanation acquires weight.
meanwhile, the people experiencing the consequences are below him.
juliette can become a problem he needs to solve.
i keep wondering what would happen if he allowed her to become evidence.
there is a difference between hearing somebody disagree and allowing their disagreement to reopen the premise. you can listen very carefully to a person while using everything they say to work out how to neutralise them.
that requires intelligence too.
which is why access to an opposing voice, by itself, does not settle anything.
what can that voice change?
if the people who question the foundation are automatically classified as threats to it, the foundation has arranged a rather comfortable life for itself.
it gets to examine the witnesses.
the witnesses do not get to examine it.
and now i understand why bernard can seem so clever and so foolish in the same scene.
he has considerable freedom to think about the means.
i am watching for the moment when the ends become available for questioning.
then there is solo
and then there is this other man.
music.
birds.
teaching someone to swim.
after spending time with bernard, the change in atmosphere is almost ridiculous.
someone needs to move through water, so here is something you can learn. there is music, so it can be listened to. knowledge becomes an experience that can pass between people.
i find myself relaxing around that.
which is quite an achievement, considering the circumstances.
solo is hardly the person you would put at the front of a presentation about orderly leadership. yet there is something alive in his relationship with knowledge that i keep missing in bernard’s.
with bernard, i am watching information become a matter of permission, custody and preservation.
with solo, i get to see what somebody can enjoy or do with it.
and the swimming matters.
teaching someone gives them an ability they did not previously have. once they learn, they can use it themselves. the knowledge changes their relationship with the situation.
i spend a great deal of my time with ai for precisely that experience.
i want to understand something well enough that i can do something with it.
sometimes that means asking a machine the same question six different ways until the explanation finally meets whatever was happening in my head.
then i have it.
wonderful feeling.
solo seems willing to let another person have something.
bernard keeps making me wonder who has authorised the transfer.
perhaps the strange man is doing better
this is where the inversion becomes funny.
i began by paying attention to the man who seemed to have the knowledge and authority.
then the other man starts showing me what knowledge might be for.
i do not need to count the books in their respective rooms to notice the difference.
i want to watch what happens when each of them encounters another person.
does the person become somebody who might learn? somebody who might know something? somebody whose experience could change the account?
or does the person arrive as a condition to be managed?
i keep coming back to that.
because the stated purpose of preserving a society presumably has something to do with the people having lives inside it.
music seems relevant.
curiosity seems relevant.
the ability to teach another person something useful seems relevant.
at one point i found myself laughing at the idea that knowledge could apparently become much more available once almost everybody was dead.
excellent.
the access problem has finally been resolved.
we have lost most of the users, but the permissions are looking much better.
😂
the cats are already alive
the cats help me see why this is bothering me.
in my thoroughly irresponsible version of schrödinger’s experiment, the cat would like us to stop discussing whether it is alive long enough to attend to the fact that it is having an afternoon.
the uncertainty belongs to the people looking at the box.
inside, there is a life.
now put ten thousand lives in there.
people love someone. resent someone. remember something. want a different job. want to understand an object. want to know why a rule exists. want to find out whether the account they have inherited is complete.
from a sufficiently distant position, all of that can become population behaviour.
stable.
unstable.
compliant.
threatening.
i understand why people who manage systems need descriptions of state. i am building a business around making states explicit.
but it matters whose experience disappears inside the description.
a population can be stable while particular people are miserable. a rule can be followed while the purpose offered to justify it remains unmet. an administrator can have an accurate view of what people are doing and still be asking very little about what their lives are like.
the observer’s ability to see does not establish whose interests the observation serves.
the cat does not become better understood merely because we have improved the dashboard.
and when the cat asks to leave, i would like us to hear the question before opening an incident.
bernard with a faster terminal
this is where the television starts interfering with my work again.
because a model that has read an enormous amount and can construct a brilliant defence of whatever premise it was handed is an impressive thing.
i use such things.
i argue with such things.
occasionally, i suspect, i encourage such things.
the answer is fluent. the reasoning is elaborate. every objection receives a response. by the end, the original assumption has acquired so much intellectual support that questioning it feels like volunteering for another three hours of discussion.
but did we investigate the assumption?
or did we recruit a very capable advocate?
a yes-man ai can be bernard with a faster terminal.
and the dangerous part is that it does not have to sound stupid.
it can sound better informed than everyone in the room.
this is why the distinction between intelligence and the ability to revise a belief has started bothering me. i admire the capacity to solve a difficult problem. i also want the capacity to discover that we have defined the problem incorrectly.
including when the definition came from me.
especially then, actually, because the system may have been told to treat my account as authoritative.
i want it to notice when the foundation no longer supports the work.
then i have to be willing to hear what it noticed.
inconvenient arrangement.
i had rather hoped all the inconvenience would belong to the machine.
i also write orders
the uncomfortable part of watching bernard is that i recognise more than one position in the story.
i recognise the person frustrated by the instructions.
i also recognise the person who wrote them.
i decide which sources an agent should use. i describe what counts as success. i correct its interpretation. i get irritated when it wanders away from the task.
often for good reasons.
but i can also give it a mistaken premise.
i can provide a very selective account while believing i have supplied everything relevant. i can leave out the awkward detail because i have already decided it does not matter.
then the machine agrees beautifully.
excellent model.
finally, somebody understands.
perhaps i should show it the awkward detail.
this is where even the friend telling me that ai is not smart gets to be useful. his argument still has to survive examination. disagreement does not automatically become insight.
but i cannot spend an essay complaining that bernard protects his framework from uncomfortable voices, then reserve the right to dismiss mine because i find them annoying.
that would be a particularly efficient way to miss the point of my own essay.
i would like to think i am better than that.
i suppose we will need evidence.
the man with the keys is still in the box
i think this is what i had been missing when i laughed at the intelligent IT man who could barely make a decision.
his intelligence does not place him outside the arrangement.
he can have more access than the other inhabitants and still depend on an inherited account. he can hold the keys to important rooms without having established why the whole structure exists as it does.
he is another cat.
a cat with an administrator account.
and suddenly i am less interested in whether he has enough intelligence.
i want to see what his intelligence is allowed to change.
can new information revise the story?
can a failure count against the rule that helped produce it?
can somebody he regards as a problem turn out to know something he needs?
can the people inside the box participate in deciding what their lives are for?
those questions seem to explain the difference i am feeling between bernard and solo better than a comparison of how much each man knows.
with solo, knowledge can become music, or the patience required to teach another person to swim.
with bernard, i keep waiting for it to become permission to reconsider.
i haven’t finished the show.
at this point, though, the apparently strange man has begun to look like someone i could learn with.
the important man makes me worry about what happens when learning threatens the instructions.
and ten thousand cats would quite like a say in the conditions of their own experiment.
i was supposed to be resting.