Building a Venture-Scale Startup in AI Safety
AI Safety is any company that builds to address the harmful consequences arising from AI. But building venture-scale businesses in this space is difficult. Here’s how.
There are a growing number of very technical people — particularly at places like MATS, Apollo, DeepMind — who care a lot about AI going well. There are also a growing number of founders building with AI who want to build important things. Despite some examples like Goodfire (valued at $1.25B and backed by Menlo and Lightspeed), there’s still a coordination gap between the people who deeply understand AI risk and the people who know how to build venture-scale companies.
As VCs and entrepreneurs, we need to make more big bets on a safe AI future. I hope this piece is a small first step.
This is a post by Matilda Glynn-Henley, investor at Merantix Capital.
What is AI Safety?
AI safety is broadly the field concerned with preventing or mitigating the harmful consequences that may arise from the development and deployment of artificial intelligence systems. Within AI safety is the AI Alignment problem (making AI systems do what their creators intend them to do), ensuring the security and robustness of AI systems in deployment, and the more general risks that emerge as a result of AI systems proliferating in the real world. Think biosecurity, pandemics, AI-enabled cyber attacks, disinformation, and destabilization posed by a world where AI causes mass unemployment.
The landscape of opportunity here is wide. For example, as AI starts to displace white-collar jobs across Europe and the US, even figuring out how to reskill workers becomes an AI Safety problem. A safe AI future likely doesn’t include rapid and mass unemployment.
It is worth bearing in mind however that many of the hardest technical problems in AI safety — particularly those related to aligning future, more capable systems — may be better suited to researchers in research organizations or frontier labs than VC-backed startups.
Where should founders start?
One of the first steps at the inception of a startup is customer discovery. For founders hoping to build in AI Safety, one early consideration is whether you should try to sell solutions to the companies at the frontier of AI development, such as OpenAI or Anthropic, or not.
The frontier labs are not an easy or ideal customer for an early-stage startup. A good way to think about this is by considering their incentives. Why would a frontier lab outsource some aspect of their AI safety work to a third party?
At a recent Merantix Capital AI Breakfast panel discussion I hosted at the London AI Hub, we came up with some good reasons. Some AI infrastructure startups are being heavily funded on the expectation that frontier labs may prefer to acquire best-in-class teams later rather than build the capability in-house. Companies like Andon Labs, Goodfire and Apollo Research perhaps fall into this category.
You can also consider cases where regulatory pressure (e.g. around red-teaming or evals) or operational complexity makes it strategically cleaner to outsource certain functions to startups. For example large-scale Reinforcement Learning from Human Feedback (RLHF) operations (as Mercor’s trajectory illustrates).
The benefit of selling to frontier labs is that it keeps you “close to the action,” and AI capabilities at these companies are generally most advanced. They likely have a good indication of what the AI systems we deploy in the future will look like, and working directly with them gives an AI safety startup an unfair advantage in anticipating upcoming threats.
Nevertheless, having only a handful of potential customers is a high-risk strategy. Your commercial success becomes dependent on what labs decide is a priority.
Then where should AI Safety startups sell their solutions?
If your customers are not the big labs, what does your business look like? I see three broad paths (I’m sure there are more)
You could sell security, monitoring, governance, or guardrail software to companies deploying AI. This is what Ilia and Peter, two of our panelists at the AI Breakfast, were doing.
Lakera (acquired by Checkpoint for ~$300M) was also following this path, focusing on run-time security of LLMs, inference time protection, and agentic workflow monitoring. As AI gets deployed more widely, it would be a fair assumption that security for AI will become more of a topic and will probably require new startups to solve these new problems. AI is fundamentally non-deterministic and has a pretty wide and new attack surface. It is likely we’ll need companies here securing this technology. Problems like scalable run time monitorability outside LLM as a judge still remains unsolved.
If you are senior on an AGI team at a frontier lab, you could start another frontier lab designed to build AGI safely (think Safe Superintelligence SSI). Of course there’s a question of whether that mission drifts, as some believe was the case with Anthropic (The Atlantic, Jan 2026).
Let’s say that you are not a senior researcher at a frontier lab. You could focus on a vertical you know well and use safe and secure AI as a unique selling point where applicable. For example: secure voice agents for regulated industries (Gradient Labs’ focus on financial services), advancing the general knowledge of how to deploy AI safety in specific production environments.
Another way to think about AI safety is to consider the broader set of problems that arise in a world where AI is everywhere, and launch a company based on those assumptions. A few examples:
Biosecurity and pandemic prevention: AI likely increases the risk surface for pandemics, so we’ll need technologies to prevent or contain them. Valthos, for example, is developing frontier AI systems that identify biological threats and design medical countermeasures in real time.
AI for cyber defence: AI enables an uplift in capabilities of cyber criminals. Market trends already imply this, particularly as models lower the barrier to entry for sophisticated attacks and AI-generated code is massively prone to vulnerabilities (see White Circle).
Insurance for AI: Mitigating the economic downsides of deployed AI systems going wrong, or building standards for secure AI (e.g. AIUC raised $15M) so that failures are less likely and easier to price in the first place.
Diffuse and democratize intelligence: Workshop Labs, for example, builds individualized models (”one for each person”) to address what they call the intelligence curse aiming to distribute the benefits and control of AI more broadly.
Defence of critical infrastructure: Cyber attacks on systems like the electrical grid are extremely high-impact, and we will likely need AI-native defences as the threat surface increases.
The Venture Scale Dilemma
Entrepreneurship, in particular venture backed entrepreneurship, is at its most basic level about finding big and important problems to solve. For VC investors, the problems must be large enough (so called “venture scale”) and customers must be willing to pay for them in order to justify a large return on investment. Not just a large return, but a venture scale return in the billions.
Venture-backed startups are absolutely not a solution to all problems in today’s AI safety landscape, but strong commercial incentives and the ability to scale makes them a good medium for some of these challenges.
Final Thoughts
I think there is a real opportunity to fund and found in AI safety, and I’m excited to see the next generation of AI safety founders working to solve these problems.
I also want to point to a few resources on AI Safety that I have found helpful.
If you are new to AI safety:
Blue Dot Impact (AGI Strategy course is a good place to start, or Technical AI Safety for engineers and researchers)
Apart Research Hackathons - if you are technical.
AI Safety Founders - find co-founders in AI Safety
For deeper landscape views and other perspectives:
If you are interested in building in the AI Safety space, or have your own ideas that I didn’t identify, please reach out to me at matilda @ merantix . com
 | A guest post by
|
In the current AI landscape, safety is typically treated as an additive layer — a collection of probabilistic wrappers and guardrails designed to constrain the inherent creativity / volatility of LLMs. Remarkably, from an engineering perspective, safety is being implemented in the same unstable medium (natural language) it's intended to govern.
AI acts in the world through people and through software. We're focused on the latter.
While others attempt to ease the symptoms, we’re resolving the root cause. While others bolt it on, we're building it in.
We’re building the tech to help everyone develop software that understands itself. Software that is itself symbolic AI. It’s superior software in itself, and truly shines in collaboration with LLMs. We call it supersoftware.
Supersoftware and LLMs are intelligent in different yet highly complementary ways. The LLM provides intuition, intent, and context, and supersoftware provides a world model, grounding, validation, security, and architectural guarantees. LLMs generate accurate, explainable, and policy-compliant responses when guided and constrained by the supersoftware's verifiable logic.
It’s a neurosymbolic synergy providing the ontological stability that LLMs innately lack. It acts as a formal reasoning layer that governs the operational logic of agents, ensuring they cannot violate predefined security policies, resource constraints, or architectural rules, regardless of the 'intuition' provided by the neural model.
Our core technology is already enterprise proven in an adjacent context.
You can find our white paper at the following URL Matilda, and you won't be surprised to hear that we'd love to share our deck with you. My contact details are at the top of the white paper.
Best regards.
https://recognitive.io/docs/supersoftware-white-paper.html