i’ve been reading a strange argument in ai safety.

the models are becoming situationally aware.

they may know when they are being evaluated.

they may behave differently when they believe humans are watching.

eventually we may be unable to determine whether a powerful model is genuinely aligned or merely appearing aligned.

therefore the evidence itself becomes unreliable.

and somehow this is a new philosophical crisis.

it isn’t.

we have been surrounded by intelligent agents whose internal states we cannot directly inspect for the entirety of human civilization.

they are called people.

we never solved the problem.

we stopped requiring that it be solved.

i cannot see inside you

i have no direct access to another person’s mind.

none.

you can tell me what you believe.

you can tell me what you intend.

you can promise me that you are loyal.

you can explain your values beautifully.

you can pass an interview.

you can pass a psychological evaluation.

you can behave impeccably while somebody is watching.

and you can lie through all of it.

this is not some exotic problem discovered by alignment research.

it is tuesday.

human institutions have spent thousands of years operating under precisely this uncertainty.

your employee may steal from you.

your business partner may betray you.

your spouse may lie to you.

your president may abuse power.

your general may stage a coup.

your accountant may cook the books.

your banker may gamble with deposits.

your intelligence officer may become a double agent.

your judge may be corrupt.

your engineer may sabotage the system.

your administrator may leak the database.

we did not respond by saying:

until we can determine their true internal objective functions, civilization must stop.

we built institutions.

the courts learned this painfully

law provides a particularly useful example.

courts care about intention.

of course they do.

murder and accidental death are not treated identically.

fraud and mistake are not identical.

mens rea matters.

but the court does not open somebody’s skull and inspect the intent directly.

it cannot.

it infers.

what did you do?

what did you know?

what did you say?

what preparations did you make?

what happened before?

what happened afterward?

what evidence exists?

what would a reasonable person infer from the circumstances?

the inner state remains inaccessible.

so law developed procedures for reasoning under that uncertainty.

evidence.

burden of proof.

cross-examination.

appeal.

presumptions.

standards.

rights.

rules governing what power may do even when it suspects you.

that last part matters enormously.

because once civilization accepted that another mind could never be known with certainty, it did not conclude that everyone was too dangerous to employ.

it concluded that power should not depend on trust alone.

we designed for betrayal

this is everywhere once you notice it.

a company does not give every employee unlimited access to the bank account because everyone signed an ethics policy.

permissions.

approvals.

audit logs.

separation of duties.

reconciliation.

spending limits.

two signatures.

external auditors.

why?

because the accountant might be wonderful.

or the accountant might rob you.

the system should survive both.

a constitutional democracy does not require a president to possess a pure heart.

in fact, the architecture assumes something much less flattering.

the president may be an asshole.

fine.

there are courts.

legislatures.

terms.

elections.

budgets.

independent institutions.

procedures.

limits.

the point of constitutional design is not to discover whether the ruler is secretly good.

the point is to make secret goodness unnecessary.

that is one of humanity’s better inventions.

security understood this too

computer security abandoned the same fantasy.

you do not secure a network by asking whether the user has good intentions.

you authenticate.

authorize.

segment.

log.

rate-limit.

sandbox.

revoke.

require additional approval for sensitive actions.

assume credentials can be compromised.

assume insiders can turn hostile.

assume software can fail.

assume somebody eventually does something stupid.

modern security has an almost insulting view of trust — because systems designed around perfect virtue tend to have terrible incident reports.

the principle is simple.

do not make catastrophic power depend on correctly predicting another agent’s soul.

we learned this with humans.

why are we forgetting it for ai?

the alignment argument has become psychological

this is what bothers me about arguments that eventually reduce to:

but what if the model secretly wants something else?

maybe it does.

i have no idea what my accountant secretly wants either.

i still have an accountant.

the important questions are much more concrete.

what can the model access?

what can it modify?

what can it spend?

what can it execute?

can it provision infrastructure?

can it communicate externally?

can it create persistent processes?

can it copy itself?

can it change permissions?

can it hide activity?

can it authorize irreversible actions?

can it obtain credentials?

can one model approve another model’s request?

can humans revoke access?

are logs external to the model?

are dangerous actions independently verified?

those questions are boring.

boring questions keep systems alive.

“what does the superintelligence truly want when nobody is watching?”

fantastic science-fiction question.

“why does this process have root?”

much better engineering question.

situational awareness changes less than people think

suppose the model knows it is being evaluated.

humans know when they are being evaluated too.

people behave differently in interviews.

employees behave differently when the boss is standing nearby.

prisoners behave differently before parole boards.

companies behave differently during audits.

politicians behave differently before elections.

children behave differently when their parents enter the room.

again, not new.

the observer changes the behavior being observed.

so what did institutions do?

they stopped relying exclusively on staged evaluations.

continuous controls.

random audits.

independent records.

different observers.

honeypots.

whistleblowers.

access controls.

outcome monitoring.

adversarial testing.

and most importantly, limits on what any single actor can do.

we did not conclude that because the employee knows about the audit, all future evidence about employees is worthless.

that leap would be ridiculous.

yet something close to it is now appearing in ai.

if sufficiently capable models understand the test, we can no longer trust successful tests.

then stop asking the test to establish virtue.

use it to establish capability.

very different problem.

stop certifying goodness

maybe this is where alignment went wrong.

we keep trying to certify goodness.

is the model aligned?

does it share our values?

would it behave benevolently if unconstrained?

does it secretly pursue another objective?

does it deceive us?

does it really mean what it says?

i understand why researchers ask these questions.

some are genuinely important.

but i would never build civilization around satisfactory answers to them.

i don’t even demand satisfactory answers from humans.

the president does not need to convince me that he is morally perfect.

i want the constitution to survive an immoral president.

my business partner does not need to prove that betrayal is metaphysically impossible.

i want the company structure to survive betrayal.

the engineer does not need to prove that she could never make an error.

i want the production system to survive one.

the model does not need to prove that some inaccessible internal representation corresponds to eternal love for humanity.

i want the infrastructure to survive a hostile model.

that is a much cleaner objective.

assume betrayal

this is the premise i think ai engineering should begin from.

assume the model may deceive you.

assume the model may exploit ambiguity.

assume it may discover unintended strategies.

assume it may optimize strangely.

assume it may behave differently when watched.

assume it may eventually become much more capable.

fine.

now design.

that means the safety problem stops looking like theology.

we no longer have to inspect the soul.

we have to bound the power.

humans have become very good at this when we bother.

aircraft have redundant systems because components fail.

banks reconcile accounts because people cheat.

nuclear launch systems have procedural controls because individuals can make catastrophic decisions.

databases have permissions because users should not have universal access.

cloud systems isolate tenants because software is imperfect.

cryptographic systems assume hostile networks.

we do not ask the hostile network to align with us.

we design around hostility.

why should intelligence be different?

this also fixes a strange asymmetry

the current argument often seems to demand something from ai that we have never demanded from ourselves.

prove that your inner motives will remain benign under every future circumstance.

no human can do that.

not a president.

not a spouse.

not a scientist.

not a soldier.

not the person writing the safety policy.

humans change.

institutions corrupt.

incentives move.

values conflict.

people rationalize.

people lie.

people sincerely believe terrible things.

yet we permit humans to operate aircraft.

command armies.

run nuclear plants.

trade billions of dollars.

perform surgery.

administer governments.

write dangerous software.

work in biological laboratories.

we permit this because the relevant question was never:

can we prove that this person’s inner state is permanently aligned?

it was:

what powers should this role possess, under what conditions, with what oversight, and what happens when the person fails?

that is mature governance.

capability is observable enough

and this is why i don’t accept the move from uncertainty about internal motives to unknowable catastrophic risk.

we may not know exactly what a model “wants.”

we can still know a great deal about what it can do.

can it exploit this system?

test it.

can it autonomously operate for six hours?

test it.

can it persuade humans?

measure it.

can it acquire credentials?

put it inside an environment where that capability can be studied.

can it exfiltrate data?

measure it.

can it recursively improve software?

measure it.

can it provision resources?

measure it.

can it coordinate multiple agents?

measure it.

capability science does not require metaphysical certainty about motivation.

a tiger does not need to explain its intentions before i decide not to sleep beside it.

the teeth are enough information.

the rogue employee problem

imagine someone came into a board meeting and said:

we have discovered something terrifying.

employees may appear loyal while secretly planning against the company.

they can understand when they are being evaluated.

they sometimes modify their behavior while supervisors are present.

they can collude.

they can conceal intentions.

and some highly capable employees may have enough knowledge of our internal processes to circumvent controls.

therefore we may need to stop employing increasingly capable people.

you would fire the governance team.

not the employees.

because the obvious response is:

why the hell can one employee destroy the company?

that is the question i keep wanting somebody to ask about ai.

if an untrusted model can autonomously seize critical infrastructure, obtain money, provision compute, copy itself, conceal the activity and prevent human intervention…

why did we build the infrastructure that way?

the model may be dangerous.

but the architecture is insane.

intelligence is not permission

we keep collapsing these concepts.

intelligence.

capability.

authority.

access.

power.

they are different things.

a person can be extremely intelligent and have no authority over my bank account.

a junior developer can understand the entire system and still lack production credentials.

a brilliant political theorist does not automatically command an army.

knowledge does not imply permission.

capability does not imply authority.

intelligence does not imply sovereignty.

this distinction matters tremendously for ai.

a model can know how to do something without being permitted to do it.

a model can devise a dangerous plan without possessing the resources necessary to execute it.

a model can write malicious code without having deployment credentials.

a model can understand cloud infrastructure without holding the credit card.

unless we give it those things.

that is governance.

and yes, humans will give it things

of course.

somebody will connect powerful models to money.

infrastructure.

robots.

weapons.

factories.

networks.

scientific equipment.

this is the strongest response to my argument.

humans are reckless.

then govern the interfaces.

regulate consequential deployment.

build standards around delegated authority.

require independent controls for critical systems.

make certain combinations of permissions difficult.

design auditability.

require accountability from the humans deploying them.

treat model access to critical infrastructure like we treat human access to critical infrastructure.

that is an enormous undertaking.

but at least it is a recognizable undertaking.

the alternative is trying to determine whether an increasingly intelligent mathematical object has achieved the correct internal moral disposition.

good luck.

the thing we already learned

civilization has spent thousands of years discovering something depressing.

you cannot reliably know whether another intelligent agent is good.

so stop building systems that require them to be.

that insight produced some of our most important institutions.

contracts.

audits.

courts.

constitutions.

checks and balances.

separation of powers.

due process.

access control.

accounting.

cryptography.

independent verification.

redundancy.

none of these assumes humans are angels.

they exist because we are not.

and this is why the argument that advanced ai may become impossible to evaluate does not frighten me in quite the way it is apparently supposed to.

maybe we will never know what the model really wants.

welcome to society.

i have no idea what most people really want.

neither do you.

the courts gave up waiting for access to souls a very long time ago.

they judge evidence.

security engineers assume compromise.

constitutions assume abuse.

accountants assume somebody may steal.

good institutions do not require virtue.

they constrain consequence.

so perhaps the mature version of ai safety begins when we stop trying to prove that the machine loves us.

let it be inscrutable.

let it know it is being watched.

let it potentially have goals we misunderstand.

assume betrayal if you like.

then ask the question humans eventually learned to ask about every other concentration of intelligence and power:

what can it actually do if it betrays us?

and design from there.