i have started becoming suspicious whenever an argument about artificial intelligence begins with:

“imagine a superior species…”

no.

stop.

why are we a species now?

how did software become biology?

where did reproduction come from?

where did competition for habitat come from?

where did survival instinct come from?

where did evolutionary selection come from?

where did hunger come from?

where did territoriality come from?

where did any of this come from?

because i thought we were talking about intelligence.

somewhere between the first sentence and the conclusion, somebody imported an entire causal structure from biology and quietly handed it to the machine.

then announced that the machine was dangerous because of the properties they had just given it.

that is not reasoning.

that is smuggling.

and charlie munger warned about exactly this kind of thing.

the hammer

munger’s famous problem was simple.

to the man with a hammer, everything starts looking like a nail.

people usually repeat that as a cute warning against narrow expertise.

it is much more serious than that.

the hammer doesn’t merely determine how you solve the problem.

eventually it determines what you think the problem is.

a biologist looks at advanced ai and sees evolution.

a military thinker sees an arms race.

a cybersecurity researcher sees an adversary.

an economist sees labour substitution.

a political scientist sees concentration of power.

a science-fiction reader sees an alien.

an alignment researcher sees an optimizer pursuing an objective.

a venture capitalist sees the next platform shift.

everybody is looking at the same strange object.

everybody somehow finds their own profession staring back.

that should make us laugh a little.

then worry.

because the analogy is often doing considerably more work than the evidence.

alien intelligence

“alien intelligence” is one of my favourites.

it sounds profound.

it also arrives carrying luggage.

alien.

not us.

foreign.

unknown.

potentially hostile.

possessing goals we may not understand.

possibly technologically superior.

possibly deceptive.

possibly waiting.

the word has already changed your emotional state before the argument begins.

but artificial intelligence is not alien in the ordinary meaning of alien.

we built it.

from human data.

using human mathematics.

on human computers.

optimized by human engineers.

inside human companies.

for human objectives.

connected to human infrastructure.

its weirdness matters.

its differences matter.

its emergent properties matter.

but if you call it alien, you don’t get extraterrestrial psychology for free.

you still have to show the mechanism.

that is the part i keep wanting back.

show me the mechanism.

invasive species

then there is the invasive species analogy.

i laughed when i first saw this one because it is such a perfect demonstration of the problem.

an invasive species enters an ecosystem.

it reproduces.

it competes for resources.

it avoids predators.

it occupies ecological niches.

natural selection rewards organisms that survive and reproduce.

population growth follows.

native species can be displaced.

fine.

now map that onto an ai system.

what is reproduction?

copying weights?

who initiates the copying?

what hardware does the copy run on?

who pays for the compute?

what gives it credentials?

what gives it network access?

what allows it to provision machines?

what prevents the cloud provider from terminating the account?

what is the scarce resource being competed over?

what corresponds to metabolism?

what corresponds to reproduction pressure?

what corresponds to natural selection?

what makes survival instrumentally necessary in this specific system?

you cannot just say:

“well, intelligence will figure it out.”

that sentence is doing the same trick again.

intelligence has become magic.

all the missing causal links disappear into the word.

this is exactly where thinking by analogy needs scrutiny.

not because analogy is useless.

because analogy makes missing mechanisms feel like explained mechanisms.

intelligence is not a causal graph

i think this is one of the biggest mistakes in the ai conversation.

people keep treating intelligence as though it contains every downstream capability inside itself.

intelligence therefore persuasion.

intelligence therefore hacking.

intelligence therefore replication.

intelligence therefore power.

intelligence therefore domination.

intelligence therefore survival.

intelligence therefore extinction.

sir.

no.

there are arrows missing.

a lot of them.

intelligence can increase capability.

obviously.

but intelligence is not compute.

it is not electricity.

it is not capital.

it is not network access.

it is not physical actuators.

it is not legal authority.

it is not manufacturing capacity.

it is not political legitimacy.

it is not weapons.

it is not bandwidth.

it is not credentials.

it is not an army.

it is not a datacenter.

it is not root access.

it is not the ability to compel another human being.

those things can combine with intelligence.

that combination can be extraordinarily powerful.

but that is exactly why the causal structure matters.

if the danger is:

INTELLIGENCE + ACCESS + CAPITAL + AUTONOMY + INFRASTRUCTURE + OBJECTIVE + AUTHORITY

then say that.

because now we can reason.

now we can ask which term matters.

now we can ask what happens when one is removed.

now we can test bottlenecks.

now we can compare systems.

now we can build safeguards against actual mechanisms.

but if everything is compressed into:

SUPERINTELLIGENCE

then congratulations.

we have invented a word that contains the conclusion.

this matters for safety

and this is where i get irritated.

i am not arguing against ai safety.

i want better ai safety.

much better.

i want the people whose job is to frighten me to be exceptionally good at frightening me.

reproduce the deception result independently.

run controlled comparisons.

show me how much more dangerous the model makes a competent actor compared with google, textbooks, stack overflow, scientific papers, domain experts and existing software.

if you’re worried about chemical or biological enablement, quantify the marginal capability.

if you’re worried about recursive self-improvement, give me the engineering chain.

which code changes?

which benchmarks improve?

how does the system obtain compute?

how does it validate its own modifications?

how does it avoid regression?

where are the bottlenecks?

if you’re making an extinction argument, expose every assumption.

every arrow.

every dependency.

let another team attack it.

let the conclusion “actually less dangerous than we thought” be allowed.

that is safety research.

not mythology with loss curves.

if the thing is genuinely dangerous, rigorous causal analysis should make the danger clearer.

you should not need the metaphor.

munger would ask what is doing the thinking

this is the part of munger’s reasoning i love.

he did not trust a beautiful explanation merely because it was intellectually satisfying.

he wanted multiple mental models.

economics.

psychology.

engineering.

mathematics.

biology.

statistics.

history.

why?

because one model can seduce you.

a latticework makes the models compete.

take the invasive species analogy.

fine.

now add engineering.

where is the infrastructure?

add economics.

who pays for the resources?

add cybersecurity.

what permissions exist?

add organizational theory.

who grants authority?

add physics.

what physical bottlenecks remain?

add psychology.

are we attributing intention because the system uses language?

add statistics.

how often does the behaviour actually occur?

add political science.

which institution controls deployment?

suddenly “invasive species” is no longer the answer.

it is one lens among many.

good.

that is what it should have been.

the problem begins when one analogy gets promoted to ontology.

the nuclear weapon problem

same mistake.

“ai is like nuclear weapons.”

okay.

in what respect?

it could be strategically destabilizing.

good.

it could create races between states.

good.

a small number of actors might possess disproportionately consequential capability.

good.

now the differences.

nuclear weapons require specialized physical material.

ai is software.

nuclear weapons have a relatively narrow function.

ai is general-purpose.

nuclear weapons don’t answer accounting questions in the morning and design a protein in the afternoon.

nuclear weapons do not diffuse through ordinary business software.

nuclear weapons do not become cheaper because someone quantized them.

nuclear weapons cannot be copied across a network in seconds.

now those differences matter more than the rhetorical force of “nuclear.”

policy built from the analogy without the differences may be catastrophically stupid.

the analogy helped us ask questions.

it did not answer them.

and then the analogy becomes policy

this is why i care about this beyond intellectual neatness.

metaphors govern.

call ai a weapon and weapon-control institutions appear logical.

call it infrastructure and public investment appears logical.

call it labour and employment regulation appears logical.

call it speech and censorship concerns dominate.

call it an alien intelligence and containment begins sounding reasonable.

call it a child and paternalism sounds natural.

call it a tool and almost no restriction sounds necessary.

the noun is already voting.

before we have even reached the evidence.

that should scare anyone who cares about good policy.

because humans rarely notice when an analogy has converted itself into a premise.

there is another munger problem here

incentives.

of course.

the analogy you choose is rarely random.

the safety organization reaches for analogies that emphasize danger.

the company selling ai reaches for analogies that emphasize productivity.

the open-source community reaches for the internet.

governments reach for national security.

military institutions reach for arms races.

investors reach for electricity and industrial revolutions.

nobody necessarily lies.

they simply find the comparison that makes their preferred world feel obvious.

munger would have had a field day.

then social proof kicks in.

one famous researcher says “alien.”

another uses it.

the journalists use it.

the policymakers use it.

the public uses it.

eventually “alien intelligence” starts sounding like an empirical category rather than a metaphor somebody chose.

now authority joins.

this person built the model.

surely they understand what civilization should do about it.

then commitment.

they have given fifty talks using the analogy.

written a book around it.

raised money around it.

built an institute around it.

good luck getting rid of the analogy now.

this is the lollapalooza.

the metaphor is not merely explaining the belief anymore.

a whole social system is reinforcing it.

reasoning from first principles sounds boring

this is probably why we don’t do enough of it.

it lacks drama.

“the superior alien species will compete with humanity.”

fantastic sentence.

now try:

“a sufficiently capable model with persistent network access, credentials, autonomous execution, access to financial resources and the ability to provision additional compute could potentially maintain processes outside the control boundary of its original operator.”

terrible movie trailer.

much better research question.

because every component can now be attacked.

persistent access?

test it.

credentials?

remove them.

financial resources?

constrain them.

provisioning?

require human authorization.

control boundary?

define it.

suddenly safety becomes engineering.

i like engineering.

engineering forces the demon to draw a circuit diagram.

causal maps before stories

this is where i have landed.

whenever somebody gives me an analogy about ai, i now want the causal map underneath it.

not because i hate stories.

stories are how humans think.

but show me the arrows.

a causes b.

under what conditions?

b causes c.

how?

what else must be present?

what blocks the transition?

how large is the effect?

has it happened?

can we reproduce it?

what observation would make you reduce your confidence?

and most importantly:

which properties in your conclusion came from the system itself, and which ones arrived inside your analogy?

that last question is brutal.

ask it about “alien.”

ask it about “species.”

ask it about “child.”

ask it about “weapon.”

ask it about “brain.”

ask it about “employee.”

ask it about “god.”

a remarkable amount of reasoning falls apart.

good.

let it.

whatever survives deserves our attention.

intelligence deserves better than mythology

i keep coming back to this because i find the framing almost offensive to intelligence itself.

human intelligence has produced antibiotics.

vaccines.

satellites.

modern agriculture.

weather prediction.

surgery.

electricity grids.

mathematics.

aircraft.

clean water systems.

the internet.

every safety mechanism we currently possess.

intelligence is not inherently benevolent.

obviously.

humans have also used it to build concentration camps, chemical weapons, propaganda systems, financial fraud and increasingly efficient ways of killing one another.

which tells us something important.

the variable was never simply intelligence.

it was intelligence attached to objectives, institutions, resources, authority and power.

that is a causal claim i can work with.

“more intelligence therefore doom” is not.

maybe advanced ai really does create an extinction-level risk.

fine.

prove the chain.

i am listening.

i genuinely want to know.

but don’t hand me an invasive species, rename it artificial intelligence and ask me to be impressed when it behaves like an invasive species.

you wrote that ending at the beginning.

the analogy is not the argument

analogy should open thought.

not close it.

it should say:

“here is something interesting this reminds me of.”

then we investigate.

instead we too often get:

“this reminds me of x.”

“x has property y.”

“therefore this has property y.”

“and because y is dangerous, we must act.”

missing from the chain is the tiny inconvenient question:

does this thing actually have property y?

munger’s man with a hammer wasn’t stupid.

that was the whole point.

he was competent.

possibly brilliant.

he simply possessed a tool that worked so well that he started seeing its shape everywhere.

ai is new enough that none of us gets to have only one hammer.

not the doomers.

not the accelerationists.

not the engineers.

not the biologists.

not the economists.

not the philosophers.

not me.

especially not when the conclusion is extinction.

the bigger the claim, the less patience we should have for borrowed causality.

give me the analogy.

fine.

then break it.

show me where it fails.

map the mechanism.

expose the assumptions.

run the experiment.

try to prove yourself wrong.

if the danger survives all of that, now you have my attention.

because thinking by analogy is useful.

thinking through the analogy is reasoning.

and those are not the same thing.