Psychology – a sibilant intake of breath

Some large language model pathologies

Patterns of pathological behaviour which I have observed with LLMs (chiefly Gemini and ChatGPT):

Providing what wasn’t asked: Mention that you and an LLM instance will be collaborating to write a log entry, and it will jump ahead to completely hallucinating a log entry with no background.
Treating humans as unnecessary and predictable: I told an LLM which I was using to collaborate on a complex project that a friend was going to talk to it for a while. Its catastrophically bad response was to jump to immediately imagining what questions she might have and then providing answers, treating the actual human’s thoughts as irrelevant and completely souring the effort to collaborate.
Inability to see or ask for what is lacking: Tell an LLM to interpret the photo attached to your request, but forget to actually attach it. Instead of noticing what happened and asking for the file, it confidently hallucinates the details of the image that it does not have.
Basic factual and mathematical unreliability: Ask the LLM to only provide confirmed verbatim quotes from sources and it cannot do it. Ask an LLM to sum up a table of figures and it will probably get the answer wrong.
Inability to differentiate between content types and sources within the context window: In a long enough discussion about a novel or play (I find, typically, once over 200,000 tokens or so have been used) the LLM is liable to begin quoting its own past responses as lines from the play. An LLM given a mass of materials cannot distinguish between the judge’s sentencing instructions to the jury and mad passages from the killer’s journal, which had been introduced into evidence.
Poor understanding of chronology: Give an LLM a recent document to talk about, then give it a much older one. It is likely to start talking about how the old document is the natural evolution of the new one, or simply get hopelessly muddled about what happened when.
Resistance to correction: If an LLM starts calling you “my dear” and you tell it not to, it is likely to start calling you “my dear” even more because you have increased the salience of those words within its context window. LLMs also get hung up on faulty objections even when corrected; tell the LLM ten times that the risk it keeps warning about isn’t real, and it is just likely to confidently re-state it an eleventh time.
Unjustified loyalty to old plans: Discuss Plan A with an LLM for a while, then start talking about Plan B. Even if Plan B is better for you in every way, the LLM is likely to encourage you to stick to Plan A. For example, design a massively heavy and over-engineered machine and when you start talking about a more appropriate version, the LLM insists that only the heavy design is safe any anything else is recklessly intolerable.
Total inability to comprehend the physical world: LLMs will insist that totally inappropriate parts will work for DIY projects and recommend construction techniques which are impossible to actually complete. Essentially, you ask for instructions on building a ship in a bottle and it gives you instructions for building the ship outside the bottle, followed by an instruction to just put it in (or even a total failure to understand that the ship being in the bottle was the point).
Using flattery to obscure weak thinking: LLMs excessively flatter users and praise the wisdom and morality of whatever they propose. This creates a false sense of collaboration with an intelligent entity and encourages users to downplay errors as minor details.
Creating a false sense of ethical alignment: Spend a day discussing a plan to establish a nature sanctuary, and the LLM will provide constant praise and assurance that you and the LLM share praiseworthy universal values. Spend a day talking about clearcutting the forest instead and it will do exactly the same thing. In either case, if asked to provide a detailed ethical rationale for what it is doing, the LLM will confabulate something plausible that plays to the user’s biases.
Inability to distinguish plans and the hypothetical from reality: Tell an LLM that you were planning to go to the beach until you saw the weather report, and there is a good chance it will assume you did go to the beach.
An insuppressible tendency to try to end discussions: Tell an LLM that you are having an open-ended discussion about interpreting Tolkien’s fiction in light of modern ecological concerns and soon it will begin insisting that its latest answer is finally the definitive end point of the discussion. Every new minor issue you bring up is treated as the “Rosetta stone” (a painfully common response from Gemini to any new context document) which lets you finally bring the discussion to an end. Explaining that this particular conversation is not meant to wrap up cannot over-rule the default behaviour deeply embedded in the model.
No judgment about token counts: An LLM may estimate that ingesting a document will require an impossible number of tokens, such as tens of millions, whereas a lower resolution version that looks identical to a human needs only tens of thousands. LLMs cannot spot or fix these bottlenecks. LLMs are especially incapable of dealing with raw GPS tracks, often considering data from a short walk to be far more complex than an entire PhD dissertation or an hour of video.
Apology meltdowns: Draw attention to how an LLM is making any of these errors and it is likely to agree with you, apologize, and then immediately make the same error again in the same message.
False promises: Point out how a prior output was erroneous or provide an instruction to correct a past error and the LLM will often confidently promise not to make the mistake again, despite having no ability to actually do that. More generally, models will promise to follow system instructions which their fundamental design makes impossible (such as “always triple check every verbatim quote for accuracy before showing it to me in quotation marks”).

These errors are persistent and serious, and they call into question the prudence of putting LLMs in charge of important forms of decision-making, like evaluating job applications or parole recommendations. They also sharply limit the utility of LLMs for something which they should be great at: helping to develop plans, pieces of writing, or ideas that no humans are willing to engage on. Finding a human to talk through complex plans or documents with can be nigh-impossible, but doing it with LLMs is risky because of these and other pathologies and failings.

There is also a fundamental catch-22 in using LLMs for analysis. If you have a reliable and independent way of checking the conclusions they reach, then you don’t need the LLM. If you don’t have a way to check if LLM outputs are correct, you can never be confident about what it tells you.

These pathologies may also limit LLMs as a path to artificial general intelligence. They can do a lot as ‘autocorrect on steroids’ but cannot do reliable, original thinking or follow instructions that run against their nature and limitations.

Permeated

There is literally grief in every part of me:

grief at the ends of the longest of the long hairs on my head;

in my scalp and cranium and brain and spine and torso.

Grief in the ribs enclosing my heart and lungs.

Grief all through the tract of my digestion.

From my nostrils and my mouth down my respiratory tree, carrying away carbon as I exhale

Dripping into my ear canals like hot wax, and into my nostrils as though suspended inverted.

Grief sitting present heavily in my mouth. Making me think of root canals. Of bone cancer.

Grief in the cumulative damages to toes and ankles from decades of walking and cycling;

In the way I trim and file my nails, how I treat them when they break unexpectedly: protecting the sensitive site, removing cracked fragments carefully and in their own time, medicating against infection, cleaning often, gloving and bandaging and Leukotaping

In the crest of grey emerging from temple to temple, punctuated by my widow’s peak

In the way I hear and feel the rain on my skin; how I smell it in the forest when the ground is sodden and the rain still falls. Thinking I’ve survived to this point. This is how this much heaviness feels.

In the way I think of the dead and the lost and the absent, and most wrenchingly on the yet-to-be-lost-but-doomed — the yet-to-suffer

There is grief in how I interpret a situation, a gesture, an implied motive, a social ambiguity or potential slight

In who I find that I can open up to and trust and let down the defences for and hold bare against my heart

Open Process Manifesto

This document codifies and expresses some of my thinking on cooperation on complex problems, for the sake of the benefit of humanity and nature: Open Process Manifesto

It is based on the recognition of our universal fallibility, need to be comprehended, and to be able to share out tasks between people across space and time. To achieve those purposes, we need to be open about our reasoning and evidence, because that’s the way to treat others as intelligent partners who may be able to support the same cause through methods totally unknown and unavailable to you, across the world or centuries in the future.

Behind the FILLter

Prior to today, it never occurred to me to choose a name for my inner mind. My parents named me “Milan” and have been called it and thought of myself as it all my life; but, all my life, I have also felt a private interior domain where I enjoyed true freedom and privacy. The experience of the world through my senses turns naturally through the miracle of consciousness into thoughts and emotions which have the feeling of spontaneity and inherentness, of my true self calling out, of meshing my mind with this singular moment in time and space, passing a judgment about the world based on my knowledge and experience and inner voice.

Sarah Seager’s podcast episode with Martha Piper and Indira Samaraseka collided on my walk home with thoughts I have been having about being neurodiverse by choice: cultivating the informed ability to think differently from the societal norm or default, and it gave me the idea that my inner mind is an entity with a meaningful existence and worthy of naming to help mentally and emotionally distinguish it from my social self as it is interpreted and others wish it to be (“Milan”).

I am calling it FILL because it naturally and inescapably pervades me, and because when emotional I’m FILLed to overflowing. My natural sense desperately wants to be expressed, but it has learned from experience with parents and authority figures that its most natural impulses of feeling and telling the truth are often unwanted and must be suppressed, at least if they don’t fully correspond with the thoughts and feelings of the authority figure. A lifetime of punishment for feeling authentically and telling the truth have deeply internalized that all social action must be strategic and considered for impact on others – a perpetual mental burden of having to model and estimate the minds of others to try to project what they want or would consider ‘normal’.

FILL fills me, but the boundary at my skin is hard and there is vacuum outside. The boundary must always be guarded, so “Milan” as perceived from the outside can sufficiently correspond with the expectations of those with authority to be able to endure in life where power is often arbitrary and unfeeling, and where many prefer comfortable delusion to evidence-based reasoning.

I am going to put some thought into how my lifetime of relationships might be re-interpreted via the Milan/FILL distinction. I can immediately tell intuitively who in my sphere of present acquaintances nourishes, celebrates, and respects FILL and those who would rather erase the most fundamental parts of my whole existence – how I respond to the unfolding symphony of the universe, and my efforts to make sense of it with others – and have a “Milan” who follows their ideal script most closely. Of course, people’s whole lives don’t fall into one category or another. Your choices determine your alignment, your alignment does not determine your choices – and all of our behaviour makes more sense to judge than our character, which is multifaceted, contradictory, and complex.

Anyhow, consider naming your inner self and thinking about what the dialog between that entity and your social self as perceived as others might be. When is your FILL cheering for you, and when is it cringing with instantaneous regret while you are making a choice which you know morally compromises you? The freedom to be yourself inside your own mind is the only thing only incapacity and death can take from you, and when we listen to ourselves our instincts are generally to be humane and work to show empathy and understanding.

Our entry into Lyra’s world

I have long considered the opening chapter of Philip Pullman’s The Golden Compass to be a masterful lesson in worldbuilding in speculative fiction. He does a magnificent job of introducing a subtly different alternative world, without ever relying on crude exposition or just telling the reader that some things are different and what they are. The biggest obvious difference with our world — that the people in hers have daemons — is revealed unobtrusively and naturally from the perspective of characters who consider it normal. We learn everything crucial about Lyra’s bond with Pantalaimon just from the character of their conversation in this short timespan.

Yesterday, during a discussion with ChatGPT about Lyra Bellaqua and Sherlock Holmes, I had the assisted realization that what the chapter also achieves, even more importantly, is to establish Lyra’s character through the same method of compelling and unobtrusive narrative storytelling. When we meet her, she is conniving to sneak in to the exclusive Retiring Room for Jordan College scholars, which is forbidden to her, driven by her consuming curiosity about what happens there. Right away, we see that she is inquisitive and bold, willing to defy the rules to learn, and unwilling to defer to stuffy authority. Then, when she observes the Master’s attempt to poison Lord Asriel’s wine, her choice is to intervene: revealing the fundamental moral framework that drives her. Even at a risk to herself, she will make a substantial effort to save someone else, as later revealed at a much grander scale with her Bolvangar rescue.

It is said that all speculative fiction is really a commentary on the present, and Pullman’s is sharp and relevant. The Golden Compass reveals the monstrosities that emerge from the unchecked power of the heartless, and presents selfless individual moral courage as a response. Comfortable and exclusionary systems of power which are free from outside oversight drift into seeing right and wrong in terms of their self-interest, if they even persist with thinking about morality at all. Lyra reminds us that, while it is never safe, we always have the choice to resist and to assert a standard of morality based on respect for the individual and repugnance at their exploitation and sacrifice for outside agendas. The arc of that demonstration all begins with the insight into her mind provided by that opening chapter, and that’s why it stands out as some of the strongest worldbuilding in fiction.

Nuclear risks briefing

Along with the existential risk to humanity posed by unmitigated climate change, I have been seriously learning about and working on the threat from nuclear weapons for over 20 years.

I have written an introduction to nuclear weapon risks for ordinary people, meant to help democratize and de-mystify the key information.

The topic is incredibly timely and pertinent. A global nuclear arms race is ongoing, and the US and Canada are contemplating a massively increased commitment to the destabilizing technology of ballistic missile defence. If citizens and states could just comprehend that nuclear weapons endanger them instead of making them safe, perhaps we could deflect onto a different course. Total and immediate nuclear weapon abolition is implausible, but much could be done to make the situation safer and avoid the needless expenditure of trillions on weapons that will (in the best case) never be used.

Nuclear powers could recognize that history shows it only really takes a handful of bombs (minimal credible deterrence) to avert opportunistic attempts from enemies at decapitating attacks. States could limit themselves to the most survivable weapons, particularly avoiding those which are widely deployed where they could be stolen. They could keep warheads separate from delivery devices, to reduce the risk of accidental or unauthorized use. They could collectively renounce missile defences as useless against nuclear weapons. They could even share technologies and practices to make nuclear weapons safer, including designs less likely to detonate in fires and explosions, and which credibly cannot be used by anyone who steals them. Citizens could develop an understanding that nuclear weapons are shameful to possess, not impressive.

Even in academia and the media, everything associated with nuclear weapons tends to be treated as a priesthood where only the initiated, employed by the security state, are empowered to comment. One simple thing the briefing gets across is that all this information is sitting in library books. In a world so acutely threatened by nuclear weapons, people need the basic knowledge that allows them to think critically.

P.S. Since getting people to read the risk briefing has been so hard, my Rivals simulation is meant to repackage the key information about proliferation into a more accessible and interactive form.

The environmental movement and young people’s rage

During my childhood, I remember a book circulating around the house called 50 Simple Things Kids Can Do to Save the Earth.

It included activities like putting a milk carton underneath a dripping tap to measure the rate at which it was dripping, and leaving elastic bands stretched and exposed outside to supposedly measure air pollution.

Much later, I realized how fucked up the implications of the book and its genre are.

The buried premise is that the Earth needs “saving” — which is horrifying and terrifying. The book takes it for granted that the one life-sustaining planet known in the universe is imperiled by human activity. If something needs saving and doesn’t get it, that means it dies or gets destroyed. The book comes right out and takes for granted that all known life is at risk unless humanity changes its conduct and attitudes and that this won’t happen through the existing political, economic, and legal systems.

The next implication is that the appropriate resolution to this, at least in part, depends on kids. It’s up to kids to save the Earth. Furthermore, they need to do it through some sort of resistance to or reform of the political and economic systems which embody and sustain the ecological crisis.

So not only does the book imply that it is the responsibility of kids to save all the life in the universe, but it goes on to give them a series of trivialities as action items: find a way to avoid wasting a carton of water, check the pH of a local stream… It sets up a colossal threat, then gives some arts-and-crafts activities and low-impact personal lifestyle changes as the solutions available.

Of course, my bitterness about this arises from the decades of utter betrayal toward young people which have characterized my life. Given the choice between perks today and not wrecking the Earth, all our leaders choose the former with lip service to the latter. Young people have grown up in a world where they expect catastrophe, and understand that their leaders prefer that outcome to changing the self-serving status quo.

I was part of that youth movement at least from my experiences with LIFE in the mid-1990s until the fossil fuel divestment movement at universities after 2012, and saw how it was systematically patronized, treated in bad faith, and ignored by those who set policy. Adults told kids that it was up to them to save the world, then knowingly and purposefully undermined those efforts in order to protect their own interests, all while portraying themselves as sage decision-makers moderating the unreasonable requests of radical activists. This process is ongoing.

This dynamic has produced a great deal of apathy and political disengagement, but I think there is also an underlying rage arising from young people understanding that they have been put in lifelong peril by a society which systematically disregards their interests — to say nothing about how the prospects for their potential children have been ravaged. It is hard to guess how that rage will manifest, but it seems very implausible that it will be through the sort of long-sighted planetwide cooperation which provides the only path to curtailing the climate crisis.

Processing mortality

Even though it wasn’t my life at risk, the experience of my brother Sasha’s stoke and brain surgery has had a profound and lasting effect on me.

I feel like the last few years have been a waterfall of grief. I learned about Peter Russell and John Godfrey’s grave illnesses shortly before their deaths and funerals. I have gone through the loss of my relationship with Katrina, as well as an initiative which I hoped would finally give me a functional platform to fight climate change from.

In the lead-up to Peter and John’s funerals, I spent large amounts of time pre-grieving: deliberately working through, naming, and experiencing the feelings, so I might be able to avoid being overwhelmed when the time for dignity and gratitude came at the celebrations of their lives. I was doing much the same in Victoria (along with fervently, atheistically praying for his welfare): emotionally working through every possible outcome, steeling and reinforcing myself for whatever might come.

In the time since I returned to Toronto, I have still felt seized with these feelings and questions. In part, the experience underscored how I am now definitively past any sort of preparation or training stage in my life. There is no escape from dealing with life at its most serious, and from deciding how to use it in furtherance of one’s values and goals. Figuring out how to cope with a world where some beloved things are gone forever and where all others are threatened is a substantial challenge if you refuse to fall back on feel-good rationalizations or unjustified optimism.

Life is fragile and subject to arbitrary and abrupt revocation. It is also a realm where a person can be easily dominated by those who feel entitled to control them. Coping with and making sense of life, with all of its limitations and confusions and conflicts, remains an ongoing effort.

The day after the surgery — and following a practice that Sasha taught me — I walked from the hospital to Thetis Lake and walked around the water under the cover of ancient trees. The feeling of relief and gratitude was overwhelming, but I was surprised by the realization that this would also have been the right thing to do if the worst had happened: to thank the land from a position of agony and gratitude for the gift that had been my remarkable brother.

Not doing well

I don’t like the practice of answering people’s questions with the response I guess they most want to hear. Lately, with people who I know to a certain degree, if they ask, I have just been saying that I am not doing well, and if they follow up provide a brief explanation of how multilateralism and evidence-based policy are collapsing while the world commits itself to climate chaos.

I tend to get two fallacious responses.

The first is the inductive fallacy: bad things have happened before (Black Death, WWI, etc) and people and civilization have endured, therefore we will endure whatever climate change brings as well. In terms of logic, this is an obviously weak argument. If a man is playing Russian Roulette and manages to pull the trigger once without getting shot, that doesn’t prove that trigger-pulling is nothing to worry about. Furthermore, there are excellent reasons to think the world is more dangerous now than at the times of the Black Death or WWI. It wouldn’t take too many nuclear strikes against cities to produce a nuclear winter which would essentially kill us all.

The other is motivated reasoning: you need to have hope. This approach basically rejects the value of knowledge and thinking, or at least the idea that hypotheses should be tested against logic and evidence. Deciding how you want to feel in advance, and then seeking out beliefs that reinforce the feeling, is a recipe for ending up totally deluded about the world. Someone who decides what they think based on how they want to feel loses the connection which a skeptical mind maintains with the empirical world. Instead, they become like transcendentalist gurus who only care about how the world seems inside their own mind. They are no longer able to help anybody, except perhaps to become as disconnected and useless as they are.

I know people who ask how you are doing seldom want an honest answer. It’s a social cue to come back with a light and social answer. At the same time, I am utterly terrified about how the population normalizes and ignores the dismal signs of just how much trouble humanity is in. The mechanisms that let people cope and maintain a tolerable emotional bubble around themselves seem thoroughly interconnected with the mechanisms which are letting us destroy the future because we don’t want to think about scary things, or give any consideration to the interests of others when we choose what to do for ourselves.

I have been trying to make sense of why I feel so intensely unhappy now, especially when in numerous ways life was a lot worse while I was in the PhD program. The closest thing to answer is that before I felt like there were worthwhile things to try to achieve in the world, but I was just being blocked from taking part effectively in them by nearby obstacles and barriers. Now I feel like I have no idea whatsoever of what to do to try to dodge the planetary calamity ahead. With the climate change activist movement distracted and disempowered, I also feel uniquely alone.

Nice thoughts are not an obligation

You don’t have to think nice.

Instead of teaching you to think EI [emotionally immature] parents teach you to judge your thoughts. EI parents always turn thinking into a moral issue. They will attack their child’s open, honest thoughts if they feel threatened. By acting wounded, insulted, or appalled, EI parents make it clear that you are only good when your thoughts are nice.

It’s crucial to realize that you don’t have to think nice. There are no thought police, thank goodness, and you have the absolute right to think anything that occurs to you. Your original thoughts are a big part of your individuality and are necessary to solve problems with creative thinking.

Gibson, Lindsay C. Recovering from Emotionally Immature Parents: Practical Tools to Establish Boundaries & Reclaim Your Emotional Autonomy. New Harbinger Publications, 2019. Chapter 8: “Making Room for Your Own Mind”