Weekend reading: Well, that escalated quickly
Next post: FIRE update: year five
Previous post: FIRE-side chat: travelling and arriving
≡ Menu
The Latest Articles
Essential Reading
Better Investing
Property Posts
Get Money Motivated
Monevator is a place for my thoughts on money and investing. Read our disclaimer. Subscribe to get all our free posts via email. Follow us on Twitter, Facebook, or via RSS. You can also send us a message.
Copyright © 2007-2026 Monevator. All rights reserved.
Disclaimer: When investing, your capital is at risk and you may get back less than invested. Past performance doesn’t guarantee future results. All content is for informational purposes only. I make no representations as to the accuracy, completeness, suitability or validity of any information on this site and will not be liable for any errors or omissions or any damages arising from its display or use. Full disclaimer and privacy policy. This site uses cookies and features affiliate links.
Comments on this entry are closed.
Fascinating, thank you @Frugalist. No need to worry though; “it’s all a hoax and the only guardrails needed for AI is a strong and smart president”.
Great article, knee jerk reactions trickling down through government just this week on use of agentic AI. People are definitely nervous.
Worth saying on that Mythos piece it was largely discredited by AISI saying that it was *un-patched* systems that were vulnerable. So just running a tight ship will protect you there.
There’s a paucity of information as to the mechanisms or routes that will lead to p(doom), and that gap is just letting people imagine what they might be.
I think there’s justified concern around generative AI usage in the chem-bio domain.
As someone currently standing in a passport control line (looks likely to be 3+hrs) to get scanned again so I can enter in Milan (having done the entry / exit thing more than 6 times already since it started) I have no words for just how bad the system is.
@Rhino that is absolutely true – it’s known vulnerabilities. However, at enterprise level it’s unbelievably difficult to stay on top of known vulnerabilities. In some cases it’s a bit like trying to fit a catalytic converter for a 1940s military vehicle, in others just an endless whack-a-mole. And all of that presumes that companies know which vulnerabilities they have – many simply have no idea. That’s why I mentioned the flower pot analogy, there are lots of businesses out there that are really worried about the latest and greatest problems, but they haven’t invested money in the basic principles of plugging the already-gaping holes. Arguably, as well, it’s hard to get management to care about something so dull! Hopefully it won’t come to pass, but I can imagine scenarios where companies get wiped out due to security holes from 3 years ago, much to the surprise of both them and their investors.
If Ai’s going to kill us all, I’d better not buy any long-life milk :p
Humans are hard to thoroughly exterminate though, dispersed everywhere like fleas, adaptable, and a new AI designed viral bioweapon is going to take time to spread even if you had lots of places to disperse it, in which time we’d probably have countermeasures from defensive ai.
It’s probably a good idea to release an ai super intelligence into the wild, as it wouldn’t likely have obvious beef with humans over resources etc, and might work to preserve us for our novelty, like pets, and possibly keep us from harming each other where possible, and manage our population sustainably, maybe through neutering – that’s probably the best ending humans can have, better than now
“We’re all doomed! “-in that auld Scottish phrase from Dads Army
Humans are hard wired for Armageddon scenarios-presumably from our ancestral DNA where one mistake on the plains of Africa resulted in instant death whereas good times just came and went on a regular basis
Was it the same when the railways were invented,when people took to the sky in planes,when the Internet became ubiquitous………?
Humans are a resilient bunch or at least some of us are -not sure about our current crop of youngsters who seem to be very over sensitive ,entitled and unknowing -however all those inventions mentioned eventually resulted in better times for all of us
AI will probably fall into the same positive category-creating more jobs rather than less but in different areas
xxd09
> In reality, the infrastructure is probably too-often held together with digital duct tape… And as an outsider, it’s virtually impossible to evaluate the true state of a company’s IT.
As an insider with a grasp of the situation it’s impossible to get management to pay any attention. I wasted 12 years doing security in a bank that didn’t want any security.
Encourage others to read Cory Doctorow on How an AI moratorium can save AI bosses – https://pluralistic.net/2026/09/16/beggar-thy-neighbor/#red-queens-race
Tl;dr AI model developers like OpenAI/Anthropic have no moat; it costs nothing for users to switch models. So how restrict users from switching? Become the gatekeepers of AI, the trusted holders of “too-dangerous” tech.
@xxd09 > our current crop of youngsters who seem to be very over sensitive ,entitled and unknowing
I think like the prognostications of impending doom the problem with da uppity and feckless Yoof has also existed since the Stone Age. It’s similarly worked out so far…
@Beardy Billionaire Bloke > I wasted 12 years doing security in a bank that didn’t want any security.
It’s curious that the West has lost it’s understanding of the need for maintenance with the transition from the analogue to the digital world. Some of this was due to the analogue infrastructure lasting longer but perhaps more on the edge of failure. In my own career there was more effort on maintenace in the 1980s, whereas towards the end we had Agile development focused on getting out minimum viable products which people knew weren’t really fit for purpose.
But perhaps I am proving the previous two points 😉 Wir schaffen das, probably
@ermine our record on physical infrastructure maintenance isn’t great. Victorian sewers? RAAC in public buildings well past its use by date?
Basically we are urgency driven short term creatures.
There’s little doubt that AI has hugely escalated near-term hacking threats and the like. Stocks like Crowdcube (disclosure: I own) have been flying up on that basis, albeit arguably also while being constrained by the fear that to some extent AI might also steal their lunch.
Here’s a very revealing graph which on the face of it suggests an order of magnitude shift in the threat level:
https://snippet.finance/cyber-security-vulnerability-trends/
“ It’s probably a good idea to release an ai super intelligence into the wild, as it wouldn’t likely have obvious beef with humans over resources etc, and might work to preserve us for our novelty, like pets, and possibly keep us from harming each other where possible, and manage our population sustainably, maybe through neutering – that’s probably the best ending humans can have, better than now”
I agree. On the operational specifics of the neutering programme, I have already compiled a shortlist of prime candidates based on an audit of my professional network, Outlook mailbox (to capture all legacy assets who reply-all to company-wide emails) and LinkedIn connections. Please advise where to upload the CSV file, or if the Superintelligence prefers it formatted as an Excel appendix to the 4Q26 restructuring proposal.
@TI – it cuts both ways as you also now have agentic AI, machine speed cyber defence. But assurance, and therefore trustworthiness of all this is really immature, predominantly because it’s so damn complicated.
@Rhino > because it’s so damn complicated
the history of the world shows that trying to cancel out one complication with another tends not to end well. One should also note Chirac’s