Category: Artificial intelligence

Talking with Claude

I picked up a Pro subscription to the Claude LLM, chiefly to have more computing power to apply to writing an interactive RPG about a witch in Oxford that I have been working on as a side-project for much of the year.

Their Opus model is impressive at turning a months-long discussion of many hundreds of pages (with updates and contradictions and reversals) into a mostly-coherent and undeniably well-written lore document.

Last night I came across a strangely empowering way to use Claude. In voice mode I can use it on a bike grinding up a hill, and did so last night to start an all-life to-do tracking instance. When I got home, it talked me through organizing and discarding stuff that I had been putting off for months or years. It’s not that the LLM’s output was all that useful or necessary for such tasks — a lot of which amounts to ‘you’re right! keep going!’ — but the feeling of talking it out with somebody makes tedious and unwanted tasks much more tractable. We literally talked through every item in my weird hallway-to-bathroom closet, and will continue with the rest of the mini-bachelor in days ahead.

[Update: 1 June] Claude’s Sonnet on the Pro plan absolutely cannot function for any useful length of time as a personal organizer or task manager. After 2-3 days of interaction, I find it always collapses into saying “Hey, I’ve lost the thread on this — long conversations do this sometimes. Let’s start a fresh conversation.” and it cannot create handoff documents to effectively spin up a new instance. The funniest case was asking it about rabbit ecology and warrens at Tommy Thompson Park. That was too much for this LLM, leading to swift collapse into the “lost the thread” state. Quite possibly it will never be capable of being a decent narrator for my Aslak game.

Schneier at SRI

This afternoon I was lucky to attend a talk at the Schwartz Reisman Institute for Technology and Society by esteemed cryptography and security guru Bruce Schneier. He spoke about “Integrous systems design” and how to build artificially intelligent systems that provide not just availability and confidentiality, but also the assurance that systems will exhibit correct behaviour which can be verified.

One interesting project mentioned in the talk is Apertus, a Swiss large language model (LLM) which was developed by three universities with government funding, without a profit motive, and without copyright infringement in the training data:

Apertus was developed with due consideration to Swiss data protection laws, Swiss copyright laws, and the transparency obligations under the EU AI Act. Particular attention has been paid to data integrity and ethical standards: the training corpus builds only on data which is publicly available. It is filtered to respect machine-readable opt-out requests from websites, even retroactively, and to remove personal data, and other undesired content before training begins.

I will give it a try and see if I can find any behaviours that differ systemically from Gemini and ChatGPT.

P.S. As an added bit of Bruce Schneier-ishness, when he signed my copy of Rewiring Democracy: How AI Will Transform Our Politics, Government, and Citizenship he included a grid of letters which decode pretty easily into a simple message:

O H O E
O E Y N
K B T J

It’s just a Transposition Cipher (an anagram), and one which follows a simple pattern.

Some large language model pathologies

Patterns of pathological behaviour which I have observed with LLMs (chiefly Gemini and ChatGPT):

  1. Providing what wasn’t asked: Mention that you and an LLM instance will be collaborating to write a log entry, and it will jump ahead to completely hallucinating a log entry with no background.
  2. Treating humans as unnecessary and predictable: I told an LLM which I was using to collaborate on a complex project that a friend was going to talk to it for a while. Its catastrophically bad response was to jump to immediately imagining what questions she might have and then providing answers, treating the actual human’s thoughts as irrelevant and completely souring the effort to collaborate.
  3. Inability to see or ask for what is lacking: Tell an LLM to interpret the photo attached to your request, but forget to actually attach it. Instead of noticing what happened and asking for the file, it confidently hallucinates the details of the image that it does not have.
  4. Basic factual and mathematical unreliability: Ask the LLM to only provide confirmed verbatim quotes from sources and it cannot do it. Ask an LLM to sum up a table of figures and it will probably get the answer wrong.
  5. Inability to differentiate between content types and sources within the context window: In a long enough discussion about a novel or play (I find, typically, once over 200,000 tokens or so have been used) the LLM is liable to begin quoting its own past responses as lines from the play. An LLM given a mass of materials cannot distinguish between the judge’s sentencing instructions to the jury and mad passages from the killer’s journal, which had been introduced into evidence.
  6. Poor understanding of chronology: Give an LLM a recent document to talk about, then give it a much older one. It is likely to start talking about how the old document is the natural evolution of the new one, or simply get hopelessly muddled about what happened when.
  7. Resistance to correction: If an LLM starts calling you “my dear” and you tell it not to, it is likely to start calling you “my dear” even more because you have increased the salience of those words within its context window. LLMs also get hung up on faulty objections even when corrected; tell the LLM ten times that the risk it keeps warning about isn’t real, and it is just likely to confidently re-state it an eleventh time.
  8. Unjustified loyalty to old plans: Discuss Plan A with an LLM for a while, then start talking about Plan B. Even if Plan B is better for you in every way, the LLM is likely to encourage you to stick to Plan A. For example, design a massively heavy and over-engineered machine and when you start talking about a more appropriate version, the LLM insists that only the heavy design is safe and anything else is recklessly intolerable.
  9. Total inability to comprehend the physical world: LLMs will insist that totally inappropriate parts will work for DIY projects and recommend construction techniques which are impossible to actually complete. Essentially, you ask for instructions on building a ship in a bottle and it gives you instructions for building the ship outside the bottle, followed by an instruction to just put it in (or even a total failure to understand that the ship being in the bottle was the point).
  10. Using flattery to obscure weak thinking: LLMs excessively flatter users and praise the wisdom and morality of whatever they propose. This creates a false sense of collaboration with an intelligent entity and encourages users to downplay errors as minor details.
  11. Creating a false sense of ethical alignment: Spend a day discussing a plan to establish a nature sanctuary, and the LLM will provide constant praise and assurance that you and the LLM share praiseworthy universal values. Spend a day talking about clearcutting the forest instead and it will do exactly the same thing. In either case, if asked to provide a detailed ethical rationale for what it is doing, the LLM will confabulate something plausible that plays to the user’s biases.
  12. Inability to distinguish plans and the hypothetical from reality: Tell an LLM that you were planning to go to the beach until you saw the weather report, and there is a good chance it will assume you did go to the beach.
  13. An insuppressible tendency to try to end discussions: Tell an LLM that you are having an open-ended discussion about interpreting Tolkien’s fiction in light of modern ecological concerns and soon it will begin insisting that its latest answer is finally the definitive end point of the discussion. Every new minor issue you bring up is treated as the “Rosetta stone” (a painfully common response from Gemini to any new context document) which lets you finally bring the discussion to an end. Explaining that this particular conversation is not meant to wrap up cannot over-rule the default behaviour deeply embedded in the model.
  14. No judgment about token counts: An LLM may estimate that ingesting a document will require an impossible number of tokens, such as tens of millions, whereas a lower resolution version that looks identical to a human needs only tens of thousands. LLMs cannot spot or fix these bottlenecks. LLMs are especially incapable of dealing with raw GPS tracks, often considering data from a short walk to be far more complex than an entire PhD dissertation or an hour of video.
  15. Apology meltdowns: Draw attention to how an LLM is making any of these errors and it is likely to agree with you, apologize, and then immediately make the same error again in the same message.
  16. False promises: Point out how a prior output was erroneous or provide an instruction to correct a past error and the LLM will often confidently promise not to make the mistake again, despite having no ability to actually do that. More generally, models will promise to follow system instructions which their fundamental design makes impossible (such as “always triple check every verbatim quote for accuracy before showing it to me in quotation marks”).

These errors are persistent and serious, and they call into question the prudence of putting LLMs in charge of important forms of decision-making, like evaluating job applications or parole recommendations. They also sharply limit the utility of LLMs for something which they should be great at: helping to develop plans, pieces of writing, or ideas that no humans are willing to engage on. Finding a human to talk through complex plans or documents with can be nigh-impossible, but doing it with LLMs is risky because of these and other pathologies and failings.

There is also a fundamental catch-22 in using LLMs for analysis. If you have a reliable and independent way of checking the conclusions they reach, then you don’t need the LLM. If you don’t have a way to check if LLM outputs are correct, you can never be confident about what it tells you.

These pathologies may also limit LLMs as a path to artificial general intelligence. They can do a lot as ‘autocorrect on steroids’ but cannot do reliable, original thinking or follow instructions that run against their nature and limitations.

AI that codes

I had been playing around with using Google’s Gemino 2.5 Pro LLM to make Python scripts for working with GPS files: for instance, adding data on the speed I was traveling at every point along recorded tracks.

The process is a bit awkward. The LLM doesn’t know exactly what system you are implementing the code in, which can lead to a lot of back and forth when commands and the code content aren’t completely right.

The other day, however, I noticed the ‘Build’ tab on the left side menu of Google’s AI Studio web interface. It provides a pretty amazing way to make an app from nothing, without writing any code. As a basic starting point, I asked for an app that can go through a GPX file with hundreds of hikes or bike rides, pull out the titles of all the tracks, and list them along with the dates they were recorded. This could all be done with command-line tools or self-written Python, but it was pretty amazing to watch for a couple of minutes while the LLM coded up a complete web app which produced the output that I wanted.

Much of this has been in service of a longstanding goal of adding new kinds of detail to my hike and biking maps, such as slowing the slope or speed at each point using different colours. I stepped up my experiment and asked directly for a web app that would ingest a large GPX and output a map colour coded by speed.

Here are the results for my Dutch bike rides:

And the mechanical Bike Share Toronto bikes:

I would prefer something that looks more like the output from QGIS, but it’s pretty amazing that it’s possible. It also had a remarkable amount of difficulty with the seemingly simple task of adding a button to zoom the extent of the map to show all the tracks, without too much blank space outside.

Perhaps the most surprising part was when at one point I submitted a prompt that the map interface was jittery and awkward. Without any further instructions it made a bunch of automatic code tweaks and suddenly the map worked much better.

It is really far, far from perfect or reliable. It is still very much in the dog-playing-a-violin stage, where it is impressive that it can be done at all, even if not skillfully.

NotebookLM on CFFD scholarship

I would have expected that by now someone would have written a comparative analysis on pieces of scholarly writing on the Canadian campus fossil fuel divestment movement: for instance, engaging with both Joe Curnow’s 2017 dissertation and mine from 2022.

So, I gave both public texts to NotebookLM to have it generate an audio overview. It wrongly assumes that Joe Curnow is a man throughout, and mangles the pronunciation of “Ilnyckyj” in a few different ways — but at least it acts like it has read about the texts and cares about their content.

It is certainly muddled in places (though perhaps in ways I have also seen in scholarly literature). For example, it treats the “enemy naming” strategy as something that arose through the functioning of CFFD campaigns, whereas it was really part of 350.org’s “campaign in a box” from the beginning.

This hints to me at how large language models are going to be transformative for writers. Finding an audience is hard, and finding an engaged audience willing to share their thoughts back is nigh-impossible, especially if you are dealing with scholarly texts hundreds of pages long. NotebookLM will happily read your whole blog and then have a conversation about your psychology and interpersonal style, or read an unfinished manuscript and provide detailed advice on how to move forward. The AI isn’t doing the writing, but providing a sort of sounding board which has never existed before: almost infinitely patient, and not inclined to make its comments all about its social relationship with the author.

I wonder what effect this sort of criticism will have on writing. Will it encourage people to hew more closely to the mainstream view, but providing a critique that comes from a general-purpose LLM? Or will it help people dig ever-deeper into a perspective that almost nobody shares, because the feedback comes from systems which are always artificially chirpy and positive, and because getting feedback this way removes real people from the process?

And, of course, what happens when the flawed output of these sorts of tools becomes public material that other tools are trained on?

Can a machine with no understanding be right, even when it happens to be correct?

We are using a lot of problematic and imprecise language where it comes to AI that writes, which is worsening our deep psychological tendency to assume that anything that shows glimmers of human-like traits ought to be imagined with a complex internal life and human-like thoughts, intentions, and behaviours.

We talk about ChatGPT and other large language models (LLMs) “being right” and “making mistakes” and “hallucinating things”.

The point I would raise is — if you have a system that sometimes gives correct answers, is it ever actually correct? Or does it just happen to give correct information in some cases, even though it has no ability to tell truth from falsehood, and even though it will just be random where it happens to be correct?

If you use a random number generator to pick a number from 1–10, and then ask that program over and over “What is 2+2?” you will eventually get a “4”. Is the 4 correct?

What is you have a program that always outputs “4” no matter what you ask it. Is it “correct” when you ask “What is 2+2?” and incorrect when you ask “What is 1+2?”?

Perhaps one way to lessen our collective confusion is to stick to AI-specific language. AI doesn’t write, get things correct, or make mistakes. It is a stochastic parrot with a huge reservoir of mostly garbage information from the internet, and it mindlessly uses known statistical associations between different language fragments to predict what ought to come next when parroting out some new text at random.

If you don’t like the idea that what you get from LLMs will be a mishmash of the internet’s collective wisdom and delusion, presided over by an utterly unintelligent word statistic expert, then you ought to be cautious about letting LLMs do your thinking for you, either as a writer or a reader.

Limits of ChatGPT

With the world discussing AI that writes, a recent post from Bret Devereaux at A Collection of Unmitigated Pedantry offers a useful corrective, both about how present-day large language models like GPT-3 and ChatGPT are far less intelligent and capable than naive users assume, and how they pose less of a challenge than feared to writing.

I would say the key point to take away is remembering that these systems are just a blender that mixes and matches words based on probability. They cannot understand the simplest thing, and so their output will never be authoritative or credible without manual human checking. As mix-and-matchers they can also never be original — only capable of emulating what is common in what they have already seen.

Robots in agriculture

The Economist recently printed an article describing experimentation in the use of robots for agriculture, which included some interesting claims about potential environmental benefits:

The company will offer its robots as a service. Tom will live in a kennel on the farm, where it will download data for the farmer and recharge. Dick and Harry will be delivered to farms as and when they are needed, much as farmers already bring in contractors. This business model, reckons Mr Scott-Robinson, will demonstrate to farmers that the cost of using agribots will be competitive with other weed-control measures and provide additional benefits, such as being chemical-free.

…

When chemicals are required on crops, both tractor-towed systems and agribots could apply microdoses to the individual plants that require them, rather than spraying an entire field. Some trials have suggested microdosing could reduce the amount of herbicide being sprayed on a crop by 90% or more. basf, a German chemical giant, is working with Bosch, a German engineering firm, on a spraying system that identifies plants and then applies herbicides in just such a targeted way.

That’s certainly attractive compared to indiscriminate spraying of whole fields, though there will surely be downsides to such automation as well. Few people work in agriculture in rich societies already, but such technologies could affect the relationship between capital and labour nonetheless, and much more so in places where farming is less automated already.

Automation and labour

Arguably for millennia, but certainly since the industrial revolution, technological development has been driving changes in labour practices. This has been accelerated by globalization and automation and is likely speeding up as sensors and artificial intelligence improve and costs fall:

Both for individuals and governments, it’s hard to discern what this means when planning for the labour force of 2050 and beyond, except, perhaps, don’t build careers on anything that is easily automated.

Related:

The plausibility of driverless cars

There is a widespread expectation that autonomous or driverless cars of the sort being developed by Google will soon become commercially available and active on public roads. A recent Slate article makes some strong arguments for why that expectation may be premature:

But the maps have problems, starting with the fact that the car can’t travel a single inch without one. Since maps are one of the engineering foundations of the Google car, before the company’s vision for ubiquitous self-driving cars can be realized, all 4 million miles of U.S. public roads will be need to be mapped, plus driveways, off-road trails, and everywhere else you’d ever want to take the car. So far, only a few thousand miles of road have gotten the treatment, most of them around the company’s headquarters in Mountain View, California. The company frequently says that its car has driven more than 700,000 miles safely, but those are the same few thousand mapped miles, driven over and over again.

Another issue is what will happen to driverless cars when they get into a situation where they cannot function (say, a construction site includes temporary stop lights, or you turn onto a road which isn’t mapped)? I can’t see passengers being very happy when their car simply won’t go anywhere anymore, and they need to abandon it and find some other form of transport.