The author of this essay has some good points in terms of the possibilities but considering the time scales involved then any 'projections' would have to be taken with a grain of salt, or in this case the grain of a world: Earth.
The simple use of playing cards can illustrate some aspects of this.
The possible combinations of the standard 52 playing cards is estimated at around 8 x 10 to the 67th power. That's an 8 followed by 67 zeros. We can use that number as the universe, the 'field of possibilities'.
In the 'present' field of possibilities - Earth - we have 'winning' combinations: Royal flush, Four of a kind, Full house, etc., and that's just with the game of 'Poker' which is itself a framework, a context, that has been agreed upon. Add other games, other agreed upon frameworks of valuations, and those can be your religions, cults, or anything where a consensus of some sort or another is considered optimal, e.g., 'Civilization(s)' is a working consensus. So, you can think of religions/cults as different games/frameworks: Poker religion/cult, Blackjack r/c, Go Fish r/c, etc., and even the atheist can enjoy Solitaire. There's something for everyone.
Now, in the context of this Aeon essay, I thought of longtermism soon into the reading and was not surprised to find it mentioned toward the end. Back in October, 2021, Aeon published an essay by Émile P Torres: 'Against longtermism' which was good and garnered 85 comments including my own. My comment was brief but the gist of it was 'personal values in the context of an impersonal universe'.
The author of this current Aeon essay notes: "What about this ‘Cosmic Story with Moral Dimension that delivers Purpose and Meaning’? That description may seem familiar. That’s because it’s religion, by another name". "....If your goal is to conquer and utilise the accessible Universe, you’ll need absolute certainty in your philosophy". "....And you’ll need to relentlessly purge dissenters who occasionally arise inside your organisation – they threaten to mutate the ‘absolutely certain’ goal. You’ll also have a strong incentive to adopt secrecy as a tool to prevent infiltration, spying and sabotage from competing groups, or government interference."
The glitch with the above scenarios is that while we (humankind) in our spatial-temporal context can relate to them, the distant scenarios of millions of years may have no need for such especially if AIs are part of the equation. It may be that 'poker', or for that matter any other currently agreed upon convention/valuation will be extinct, immemorial. The AI's 'playing deck' might consist of 520 or more cards, contexts (games), and winning hands that we can't even imagine. For humans, the only 'absolute certainty' of a winning hand in poker is a Royal Flush of Spades. To an AI, it may mean absolutely nothing. And as for religion? AI may have no use for that and simply dismiss it as a primitive, operating system of the ancients. Yes, even Solitaire.
And if you really want to get on the crazy train of possibilities, imagine an AI that examines poker (or any other game), not for it's context (which even at 8x10 to the 67th power is a closed system and thus probably boring to the AI) but to reverse-engineer it, so to speak, to analyze what made it interesting to humans. Now that's a different kind of 'tell'.
"The humans recruited would be technical types, and those with connections, money or other useful resources. They would have to be attracted to (or tolerant of) cult-like behaviour, with personalities that accept the demand for extreme control, and for whom personal meaning, ‘secret knowledge’, and a new/special identity are a big draw."_
Again, AI 'beings' may have no need for any of that: Cults, personalities, personal meaning, money, secret knowledge, identity, the AI might regard those in the way we humans regard the grunts and groans of our primitive ancestors. Even if we programmed the AI for these it could probably discard them as impediments.
So, having said all that: Is it possible that the AI would create their own religion and thus all the 'accessories' that go with it? If so, then that would mean that EVERYTHING in the cosmos and beyond is founded on a religious substratum of some sort. It would be the Causa Prima that nothing, not even AI, could escape. In fact, it would be the foundation, the ground, of everything that was before and beyond humankind, AI, and anything/anyone in the cosmic parade of existence.
At that point it's way beyond my paygrade. But I doubt it would be anything like religion as we now know it
Nice essay, Mr. Olson.
Daniel
______
Note: For those from Aeon.co, I placed my response to the essay here because it exceeded the allotted character count. If you have any direct response to me do it at Aeon since that's where the essay is published. If you have exceeded your allotment there then you can do it here. You do not have to register in this forum. It's on a moderation queue.