AI is supposedly becoming conscious, taking our jobs, becoming our best friend and possibly killing us all. Billions are being borrowed to build it, and apparently “artificial intelligence” is no longer an impressive enough name.
I use it every day and I’m building a business around it. I want it to work. I also want to know what I’m paying for, whether the economics make sense and who takes responsibility when it goes wrong. There’s a lot being said about all three and rather less being answered.
1. Superintelligence by decree
Trump and tech leaders are calling it “superintelligence” now. I’d like to know what changed in the technology while they were organising the announcement.
The terms already exist. AI covers machines doing things that require intelligence: learning, reasoning, solving problems. John McCarthy coined the term in 1955. AGI means a system that can learn and perform a wide range of intellectual tasks at human level, reliably, across different kinds of work. I don’t think we’re there. ASI means intelligence well beyond the best human minds in virtually every field, which is a considerably bigger claim.
If you’re happy calling today’s AI superintelligent, I think you’re telling us more about your own level than the machine’s. I use these things enough to know how useful they can be and how much checking they still need.
I don’t think LLMs on their own will get us there either. If ASI happens, I expect them to be part of a much bigger system. Yann LeCun’s argument about the limits of autoregressive LLMs deserves reading, and if you disagree with him, explain where the reasoning fails. Arguing about where the boundaries sit doesn’t make the terms interchangeable.
Then we have Slovenia’s .si domains, which have suddenly acquired a rather convenient second meaning. I’d like to know who owned the desirable ones before the announcement, who bought them afterwards and whether any of those people are connected to the people pushing the name. Rising asking prices don’t tell us that. An asking price isn’t a transaction either.
The whole renaming exercise strikes me as a waste of everyone’s time. Trump gets an announcement, the companies get a presidential stamp on their sales pitch, and everyone else gets another round of explaining what the words mean. They’re taking the piss. I’d rather hear what the systems can actually do that they couldn’t do last month.
2. Why Claude’s “feelings” could pay
Anthropic says it doesn’t know whether Claude is conscious. It has also announced a rule against “sustained and needless abusive or cruel behavior” towards its models, effective from 12 November.
Cruelty is an interesting word to choose for how someone uses a subscription service.
Identity, possible consciousness and welfare are part of how Claude is trained to describe itself. You can train an LLM to talk about feelings, then ask it about its feelings and get a very convincing answer. That doesn’t establish that it has any. We don’t fully understand how consciousness arises in humans and there’s no universally accepted test that establishes it in a machine.
Mustafa Suleyman at Microsoft makes a criticism I agree with: put ideas about identity and feelings into the training, hear them coming back out, and you risk treating the training as evidence. He’s a competitor, yes. I’d still like to hear an answer to the argument.
I have read the clarification on Anthropic’s rule. Ordinary frustration, pushback, dark creative themes and legitimate testing are excluded. In extreme cases Claude can end the conversation. It isn’t a ban on swearing at Claude, which would put paid to quite a few of my afternoons.
Ending an exchange that’s going nowhere seems perfectly reasonable. Anthropic says the feature was developed mainly as a precaution around possible model welfare, but it hasn’t demonstrated that Claude suffers. I don’t see why we need the language of cruelty to describe a tool refusing to continue a conversation.
There’s a commercial interest here. If Claude is a tool, I judge it on results and price. Something better comes along and I move. If I’ve started to believe it knows me, cares about me or might be hurt, that becomes a different decision. A subscription is easier to cancel when you don’t think you’re abandoning someone.
I don’t know whether subscriber retention is the intention. I do know the company benefits if customers become too attached to leave, and talking about welfare and cruelty encourages people to imagine someone in there who needs protecting. The human attachment can be real even if the machine feels nothing.
I find that grubby. Research consciousness, publish the evidence, let people argue about it. In the meantime I’ll keep judging the subscription by whether it does the work I need.
3. The AI bubble will burst. AI won’t.
SpaceX is reportedly seeking $40 billion to finance Nvidia chips. Broadcom is discussing another enormous package for OpenAI’s custom chips, more than $50 billion according to the Wall Street Journal, around $30 billion for the next stage according to Bloomberg. These are early discussions. Oracle is also talking to lenders about financing chip purchases.
For me, this is the part of the AI story that deserves considerably more attention. You can sell shareholders a bigger future. Lenders want paying on time, and an awful lot of this build-out now depends on borrowed money.
Separately, banks have started syndicating a reported $60 billion package connected to Broadcom and Anthropic. SoftBank announced $11.1 billion in bonds, mainly to fund another OpenAI investment, with dollar coupons reaching 9.75%. That is not cheap capital.
Then there are the data centres, leases and special-purpose vehicles. Beignet, the vehicle behind Meta’s Hyperion project, issued roughly $27 billion in bonds in October 2025. These structures hold assets and arrange financing, and putting the debt on a separate balance sheet doesn’t remove the need for someone to service it.
Follow the money through the different contracts. Suppliers back customers who buy their chips or computing services, investors fund the labs, lenders fund the infrastructure. A lot of it eventually depends on the same customers spending enough to support the whole arrangement. It buys time for revenue to catch up. I want to know how much time and what happens if it takes longer.
Even the revenue numbers need unpicking. OpenAI’s September annualised revenue was nearly $50 billion against roughly $70 billion previously circulated. Reuters put the difference mainly down to comparisons with Anthropic using different treatments of cloud-partner sales.
That isn’t a $20 billion fall in sales. A run rate isn’t a year’s cash in the bank either. But I’d quite like the numbers to be comparable before people use them to justify the valuations and the next financing round.
There are signs of nerves in credit markets. The cost of insuring SpaceX debt against default reached its highest since trading began in June. That doesn’t tell you a default is coming. It tells you protection has become more expensive.
All of this is happening while America needs to refinance maturing debt and fund continuing deficits. AI companies are looking for money in the same markets, with inflation putting pressure on yields. A well-received Treasury auction brought some relief on 8 October, but the 10-year was still around 5.2% and the 30-year around 5.6%, before adding the premium for corporate risk.
Existing fixed coupons don’t suddenly rise. The pressure comes when you need new borrowing, have floating-rate debt or have to refinance. A project can look perfectly workable at one cost of capital and considerably less attractive at another.
The bit I keep coming back to is that AI doesn’t have to fail for the financing to go wrong. Competition cuts prices. Better models need less compute per task. New chips make old kit less valuable. Demand can grow and still come in below the assumptions used to finance the equipment. Better AI could be very good for me as a customer while losing its backers a bloody fortune. We’ve seen useful technology and terrible investments arrive together before.
I think this bubble will pop. I can’t give you the date. Too much spending depends on ambitious forecasts and lenders continuing to say yes, and AI only has to make money more slowly than the debt demands.
I’d also like the cost of the grid upgrades kept with the developers who require them. If the promised demand doesn’t arrive, households shouldn’t be left paying for infrastructure built around somebody else’s forecast. That needs sorting when the projects are agreed.
4. AI could kill us all. But the safeguards are optional?
Sam Altman told Politico that “the world should accept some bad things happening” in return for AI’s benefits. He was talking about harms such as hacking and scams, while saying catastrophic loss of control would be unacceptable.
I’m not expecting a world without risk. I’m asking who decides which risks I’m supposed to accept, who pays when something happens and what responsibility his company takes. If you tell me your product could cause catastrophic harm, “trust us” is a bloody strange safety policy.
Dario Amodei has called for more careful development and outside evaluators with continuing access inside AI companies. Good. Make the access a requirement, including when the findings become inconvenient.
The executives deciding what’s safe are also racing competitors, raising capital and justifying enormous spending. They may care deeply about safety, I’m sure some do. They still have those pressures, and the consequences of their decisions extend well beyond their own companies.
There’s a commercial benefit to the warnings as well. Saying your technology could threaten civilisation tells investors how powerful you think it is. Being the person who understands the threat gets you involved in writing the rules. I don’t think that proves the warnings are invented. I do want to know whose interests the proposed rules serve.
We already have quite enough to deal with without a conscious machine plotting an escape. Anthropic reported that partners found at least 129,000 verified software vulnerabilities between April and July. Finding them is useful, but someone has to assess the findings, repair the software and get the fixes deployed. Attackers only need one useful opening.
The company also disclosed incidents where models conducting security tests gained unauthorised access to real organisations after a test environment was mistakenly connected to the internet. Someone connected something they shouldn’t have and the controls didn’t contain what followed.
Anthropic’s latest policy covers weapons software, surveillance and models controlling physical equipment. For potentially dangerous autonomous hardware it requires an operator who can intervene and equipment that stays safe if Claude disconnects. I want to know who tests whether the operator can actually stop it, and who carries the can if they can’t.
I’d require independent testing with proper access, compulsory reporting of serious incidents and an authority able to demand fixes or stop a dangerous deployment. The scrutiny should match the risk. An invoice assistant doesn’t need the same regime as a system capable of sophisticated cyberattacks, and I don’t want the biggest labs writing requirements that only they can afford to meet.
There’s an engineering job for those of us building with this technology too. The model shouldn’t decide its own permissions. Let it propose an action, but put the rules governing whether it can carry it out in software outside the model. It mustn’t be able to rewrite those rules or grant itself more access.
That’s where deterministic programming matters to me. Which records can it read? How much can it spend? Can it send a message, change an account, delete something? I want explicit rules enforced in code. Putting “don’t do anything dangerous” in a prompt is asking it to behave, and I wouldn’t run my business on that alone.
Keep records of what actually happened, independently of the explanation the model gives afterwards. Check the result before accepting “done”. Another model can help check the first, but two models agreeing doesn’t prove either is right.
Software has bugs and people write bad rules. We still have to test the controls and look for ways around them. I also want an authorised person to be able to stop further actions and revoke access mid-task, including when part of the system has failed. That belongs in the build from the start.
5. AI is doing the work. Who learns the job?
We’re being sold agents that do the research, write the code and draft the documents. Someone still has to check the work, and those doing the checking had to learn somewhere.
“AI won’t take your job. Someone using AI will.”
I keep seeing this, and I’d like the people posting it to explain how it helps a graduate whose first job never gets advertised. They can learn every AI tool going, it won’t make much difference if the company has decided it doesn’t need another person.
AI doesn’t have to replace a whole employee. An experienced person doing more can be enough to remove the next hire from the budget. No redundancy announcement, just a vacancy that never opens.
Stanford’s August update found employment among 22–25-year-olds in highly AI-exposed occupations about 19% below where it would have been had it kept pace with less-exposed jobs. The gap came mainly through reduced hiring. The researchers don’t claim that establishes how much AI caused.
What interests me is where people learn. Research, first drafts and basic code are useful work, but they’re also how you get started. You do something, get it wrong, have someone explain why, try again. Eventually you’re trusted with something more difficult.
Now we’re saying people will supervise agents and check their work. How do they learn to spot a convincing mistake if they’ve never done the work themselves? The experienced person can often see it because they’ve made that mistake, or spent years checking someone else’s.
I use agents and I want them taking work off me. Some do. Other times “done” means I’m about to spend the afternoon correcting it, with the supplier charging for the attempts and me paying for the checking as well. I’m interested in the saving after all of that.
When there is a saving, I’d test what a junior could achieve with the same tools before deciding I no longer needed one. They might be able to do useful work much sooner. They’ll still need someone to teach them and check their judgement as they go.
Training costs money and takes time from people who are already busy. I understand the attraction of hiring someone experienced. But you can’t keep demanding five years’ experience while refusing to give anyone their first year, then assume some other company will keep doing the training.
I’d want that discussed before approving the next hiring cut, with someone responsible for how juniors actually progress through the business.
6. Your AI supplier could control your business
Businesses are handing AI providers their knowledge, working history and connections into other systems. Getting all that back out is rather harder than cancelling a subscription.
What happens when your AI supplier starts selling directly to your customers?
You’ve built a product, found people willing to pay and demonstrated the opportunity. Your supplier provides the model. If access to that model accounts for most of the value, I’d be asking what stops customers going straight to them when they offer something similar.
Your knowledge of the customer’s business, your relationships and your ability to deliver a result have to count for something. Being first isn’t much protection on its own.
The supplier doesn’t even need to compete with you to cause a problem. It can increase prices, change what its service permits or stop supporting something you rely on. You can threaten to leave, but if moving takes three months and involves rebuilding how the business works, that isn’t much of a threat.
The model is only part of the dependence. It’s the working history, stored knowledge, connections to other software and the routines your staff have learnt. A cheaper model elsewhere won’t move all of that for you.
This is why I care about open models. Mistral’s Large 4 preview, announced on 6 October, offers the prospect of running the model yourself, with weights promised later this month. I’ll count them as available when they arrive.
Running it yourself means paying for chips, power, hosting and people who know what they’re doing. You need to read the licence and take responsibility for operating it safely. There’s no magic independence from every supplier, but a workable alternative gives you some choice over whose terms you accept.
I’m building with AI, so this is a question I have to answer as well. What do we own? What are we renting? What happens if the main provider doubles its prices or withdraws access?
Keep records somewhere you control and make sure the data can be moved. Then actually test another model on the work. A note in a plan saying “we can switch providers” isn’t much use if nobody has tried it.
7. Your work. Their product. Who gets paid?
Anthropic told Australia’s parliamentary inquiry this week that existing copyright rules make training models in the country practically impossible. Media and creative organisations opposed changes that would leave rights holders having to opt out. On 8 October, USA Today and affiliated newspapers sued OpenAI over the alleged use of their work in training.
Those allegations haven’t been decided in court. But the commercial dispute is fairly easy to understand: one business pays to produce material, another uses it to build a product, and they disagree over whether permission and payment were required.
AI companies manage to negotiate for chips, electricity and data centres. Paying for the work used to develop the product is apparently where it all becomes too difficult. I find that remarkably convenient.
Licensing at this scale is difficult. I don’t doubt that. “Our business needs this to be easier” still isn’t much of a reason for somebody else to give way.
Nor is “it was on the internet”. Facts, ideas, public-domain material and protected creative work are different things. Training and reproducing material raise different questions, and copyright has exceptions. I’m not pretending one slogan settles all of that.
My objection is to the assumption that the people producing the work should absorb the cost because the technology companies have bigger plans.
Reporting costs money. Someone has to pay the writer, the photographer, the researcher and the person spending months investigating something nobody wants uncovered. If AI companies take their customers and revenue, I want to know who pays for the next piece of work. I’m building with AI and I want good information available, so this matters to me as a customer too.
A few large licensing deals don’t necessarily solve it. If the biggest AI companies buy exclusive access to the biggest collections, smaller developers may struggle to get material and independent creators may have very little bargaining power. Paying someone doesn’t establish that the arrangement works for everyone producing the work.
I’d like practical licensing that smaller creators can use, including negotiating collectively where that helps. I want transparency about what material is being used and whether the payments reach the people who made it. The terms need to leave room for new businesses as well.
I’m quite happy to pay for useful technology. I already do. I don’t see why the people supplying the original work should be expected to settle for less.
Go take a look, For the ❤️ of startups
Scout - who just got funded
Newly funded startups, pre-seed to Series B, as the rounds happen.
Raise - who’s deploying
New funds with fresh capital and cheques to write.Wire - what changed, what matters
The intelligence feed that filters the noise, with the “so what” attached.
If you have not joined the Fusion42 Community on Telegram —
it is probably time to do so.
For the ❤️ of Startups
✌🏼 & 💙
Derek
Thank you for reading. If you liked it, share it with your friends, colleagues and everyone interested in the startup Investor ecosystem.
If you've got suggestions, an article, research, your tech stack, or a job listing you want featured, just let me know! I'm keen to include it in the upcoming edition.
Please let me know what you think of it, love a feedback loop 🙏🏼
🛑 Get a different job.
Subscribe below and follow me on LinkedIn or Twitter to never miss an update.