Inside Cogent's three-agent architecture for autonomous defense | Geng Sng (Co-founder, Cogent)

This is basically SAS 2.0. Attackers only need to be right once and defenders can be right all the time. And today, the mean time to exploit is in minutes. Today, I'm talking to Gang Sing, co-founder and CTO of Kojen. They're building agents for autonomous cyber defense and closing vulnerabilities at machine speed. We don't use a graph database, but we have a very high write throughput system, like billions of events flowing through the system every single day. He walks me through how Kojen builds trust with customers. before letting agents act autonomously. Even if the agent's hallucinating and wants to do some kind of right action, it's not allowed to do that. And how they make sure their LLM judges are actually qualified to assess the agent's work. At the most complex level, it's LLMs that have environmental parity with the actual agent that we're evaling. Cogent also has an interesting way of thinking about context. We're differentiating between hot and cold contexts. It's not that this context has existed for a short amount of time. It's that this context is actively being used. Right at the end, Geng talks about why defensive security won't be replaced by AI anytime soon. The nature of a defensive security task is I have a castle. Here are all my walls. How do I patch every single little hole in the wall? And every single castle is built differently. There really is no data. Welcome to Max Agency, the podcast that goes deep into how the best agents are being built. by builders like you. Kojin is building a platform for autonomous cyber defense. What does that mean and how do you think about building that platform? Yeah, it's actually really super interesting because autonomous cyber defense was not quite a thing, I think, up until very recently. And what's really motivating this is the mean time to exploit for vulnerabilities has really compressed over the last couple of years. So two and a half years ago, if you were a security team, a vulnerability comes up, you have about two and a half years to basically backlog it, triage it before it first gets exploited in the wild. And today, the mean time to exploit is in minutes. Why? It's a combination of, one, a lot of digital sprawl. So the attack surface is increasing at a pace at which it's not possible for defenders to keep up. So software and vulnerabilities are two sides of the coin. More and more software is being invented, created, and more and more vulnerabilities are being created but not being patched. And on the other hand, the frontier models have really accelerated the exploit capabilities of any attacker. So I think Claude recently was talking about how they found all zero-day exploits in Mozilla. And this is one of the most battle-tested environments that have gone through decades of pen testing that now with a frontier model like Opus 4.7, You could probably, you know, with 500 turns, find your way through, find the exploit and find a way to basically find a zero day. It seems, just anecdotally, that there's been a ton of more security incidents recently. So how do you guys think about, I'm assuming Cogent the platform helps remediate that in some way or detect those in some ways. How exactly do you guys think about the role that you play in that space? Yeah, exactly. So the challenging aspect is not finding the vulnerabilities. It's really fixing and patching in time. And in a world where you need to move at machine speed, it's not enough to have humans in the loop. And the reason why we kind of think of ourselves as an autonomous cyber defense platform is because you need to have these agents that can react in a timely fashion, that have context on how your organization wants these things patched, and ultimately has the right judgment to make the right decisions to either apply a patch, contain a vulnerability, or do anything on behalf of a security team that would have consequences in a high-stakes environment. Walk me through what that looks like. in practice for a customer? Coges customers basically plug into our like platform and the first thing that we do actually is building context and a big part of that is plugging into their vulnerability scanners, threat intelligence. We scrape a lot of things on the internet where we are basically trying to understand if there is a newspaper article that's trending, you know, somebody like a researcher says something on x, something on Reddit, factoring that into what we consider as a risk signal. as well as other proprietary threat intel feeds. We also work with a lot of asset inventory information, so CMDBs, as well as any kind of IT asset inventory. And the most interesting aspect of Cogen actually is business context. So there's a lot of information in your ticketing systems that show your password remediation history. In the comments live in a ticket that tells you why something wasn't patched. There's also a lot of context in things like your SharePoints, your Google Drives. policy information, information about product launch. So being able to take all that context, build that into this big context graph, and not just understand the system context, but also how decisions are being made, allow our agents, which are operating at the next, like I would call it like an intelligence layer, if you will, of how to take actions when things change in the organization. That's kind of like the prereq as we kind of see it in terms of taking action in an autonomous fashion. Okay, so you build up all this context. How do you do this? Is this an agent that's building that context? We call it agent lake. Basically, it's an agentic data lake. And we kind of have two parts of our system. So the first part of it is basically a context form. So agents forms that have live access to data. We have this context engine that essentially has a lot of memory harnesses around how we're extracting information. These agents are capable of going on long-running tasks. And they basically answer the really hard questions about system ownership. about a certain remediation playbook. And this information may not actually live as a codified artifact in the organization. It might live in little pieces of data that needs to be corroborated and ultimately contextualized by the customer. That's the first piece. The second piece actually is like a high-scale data platform. And so we're kind of like combining agents with high-scale data processing. And in parts of how we construct our graph, the topology is just the first part. So agents are actually coloring the graph over time to create more and more interesting, lower high confidence edges. And that's really what allows us to do really high quality risk classification for a lot of our customers. And that high fidelity of the risk classification is what allows them to cut through the noise because a lot of customer environments in the enterprise, which are most of our customers, there's like hundreds of millions of vulnerabilities. And you need to have the ability to say, hey, these are the top 10, top 100, top 1000. given the risk, it needs to have an action taking place right now. Mapping it to your custom SLA mapping, there's so much complexity that goes into any of these decisions. And so you mentioned graphs a few times now, and that was going to be my next question, because sometimes in RAG or retrieval or context, there's just a whole bunch of different data structures that you can use from vector stores to graphs to just files. And I feel like graphs have a reputation as being the most complex, both good and bad. And so it sounds like you guys are using knowledge graphs as the database itself. Is that correct? So we don't use a graph database. Okay. And the reason why is like, I think graph databases have proven to not scale very well, but we have a very high write throughput system. So like tons and tons, like billions of events flowing through the system every single day. And what are these events? Changes in lifecycle entities. So let's say a vulnerability. A scan can happen on a cadence or it can happen in real time. So any single entity, like a vulnerability, like an asset, like a person, any identity could have some kind of lifecycle change, right? Like there's an event coming and we want to know about it and we want to model in the graph. So instead of using graph database, we actually use a data lake. So we actually store things in S3. It's really cheap. It's really fast, very like right throughput optimized. And it gives us the capability to basically materialize a graph that we need that is like kind of custom for our use cases. Is the ontology fixed up front or does the agent actually decide the ontology as it builds the graph. The ontology needs to be fixed at the start, but parts of the ontology does not need to be fixed. So maybe the way to think about this is there are structural properties of the vocabulary of how you might think about an entity. So for example, a vulnerability probably has some kind of identifier, right? It probably has certain attributes that are like well known. So those need to be fixed because it helps us identify and resolve the entity in the stream of like events, right? So without these almost like signatures. We don't actually know what this thing could be. But once you're past that initial level of, hey, here are the properties, the agents can really come up with any kind of derived attributes on the fly. An example of that is internet exposure. I don't really know if this thing is internet exposed, but here are the five data sources, including your WAF logs, including some security tool that you're using, that make me think that based on some reasoning, this is an internet exposed asset. And it sounds like there's an upfront step to create this, but it's also like always ongoing. Is that correct? Yes, that's right. So even the upfront step is actually agentic. So the discovery process of what these properties should be and how they map to our customers' environments across different vendor integrations is super complex, right? So, you know, you might have an asset, but every... customer will have different combinations of data sources and being able to do that level of NAD resolution across like, you know, basically very high cardinality of data is super difficult. And so the agents actually do kind of the first pass EDA and actually one of our like more, I would say like harder evals that we have for our agents is long running data science tasks. And it's really these agents are like highly capable, you know, have access to Jupyter notebooks, they run Spark, and they're able to do high scale data processing. And that's what allows us to look at, you know, a million rows or like a hundred million rows in like a couple of seconds. And that's all just to fill in the graph during kind of like the initial task. How long does that take, like the initial setup of the graph? The good thing about like the graph is that once we have these kind of like default, I guess, like ontology vocabulary, it's actually a lot easier for us to just apply like a bunch of templates. But there is kind of like two phases to the way we think about context enrichment. So the first pass is always going to be lossy, but it's going to help us paint the picture of the topology. And once you have the topology, you can start answering questions around, oh, how are these things connected? Are there interesting stories that could be told? An example of that might be, does this asset have sensitive data on it? So being able to do a second pass, a third pass with an agent is super helpful. So you do an initial pass, kind of like course, high level, just to set up the ontology. And then you do second and third pass on kind of like more narrow focused questions to kind of like fill out the info? At a high level, yes. I think what's interesting also is like there's two types of tasks that the agents are doing. So one is answering open-ended questions on things that are unverifiable. When you say they're answering these questions, is this like when the user is now using Cogent, like your end users are using it when after they set up the graph? So not quite. So you plug your data in and usually that's through APIs. And there is like a class of unverifiable context where even the users themselves have no idea. The perfect example, and this is generally true, not just in security, but also in engineering, is system ownership, right? Like some of these organizations we work with, people leave the company all the time, like engineering teams reorg. It's really hard to know if this, you know, AWS tag is the right source of truth, or maybe I need to consult an org chart, right? I need to go figure out all these things. Being able to answer those types of questions is... the role of this context form that I was talking about. So having agents basically ask very pointed questions on data that is unsure, building confidence through like these many, many reasoning loops is like a core use case. And then once you have that, we kind of call them context artifacts. That creates the provenance for the downstream enrichment, right? It's almost like it's like the pre-step of your cleaning your data. And once that data is cleaned, we're able to say, oh, actually, like, you know, we do think that this is the right system owner because we've done all this work. Super interesting. Okay, so this is all to set up the context graph in the first place. Yes. What happens after that? So that's where things get really fun. You know, Kojin, one of our big value props for our customers is autofixing. And this is historically something that's super challenging, right? Like I'm sure everybody who's a software engineer has gone through dependency hell before. You upgrade one software, you break something else, there's another invariant, you end up going on this wild goose chase. And at the end of the day, you've spent a couple of hours and maybe you haven't achieved anything because you realize, oh, actually, this thing can't be upgraded because now I need to do all these things and I can't deploy into production. That same story is true of vulnerabilities, which basically tend to require patching. And that's like a software version upgrade. What makes it even more challenging is this concept of vulnerability chaining. where the upgrade not just is maybe causing some dependency issues, it may lead to more vulnerabilities. And so where our agents really have, I think, a lot of value and still a lot of room for improvement is helping customers go on this journey where they're doing a lot of the automation of tailoring the remediation guidance to validating the actual steps that it takes to deploy something into production, whether that's through staging first, like basically running through all the steps, codifying it. And then finally, maybe it's through auto mitigation or auto remediation. getting the patch into where it needs to be without any kind of human oversight. And users trust these agents to merge things into their production code base? So it's definitely a trust building exercise, right? All of our customers start in read-only mode. This is the purely assisted version of Cogent where the AI is making recommendations and users have the easy approve button. That starts from ticketing, right? And the next... trust letter, if you will, rung is auto routing of the tickets, right? So over time, you build confidence that for certain classes of remediation actions, Cogent's recommendations are really good. And then you essentially start smartly auto routing these tickets. And the next kind of like level of automation is auto validation, where you let the agents basically start testing in your dev environment, staging environments, they're spinning up things in dev sandboxes. they're running the patch, right? They're showing you the proof and evidence of like whether or not this will break something. And actually like the really challenging piece here is not applying the patch or like knowing if the patch will work. It's knowing if the patch, if it's applied, has production impact, right? This is usually the key blocker for true automation. And, you know, like we work with a lot of big enterprises, some of them like our banks and like something that a person will say is like, look, never... ever will you like autonomously deploy onto my like payment processor, right? Yes, I trust the AI. Yes, everything's good, but that's just never happening. And so you can use Cogent in any way that you want. And really it's more so a customization that customers prefer. But our thesis is there will be a lot of low blast radius things that you can fully automate. And when you use Cogent, it's not just, hey, let's go autofix everything. It's what percentage of your attack surface can be truly automated, right? Like within, let's say, five minutes of an attack happening, it can be auto-fixed without you needing to care about that, effectively shrinking your attack surface. And for everything else, you have the AI, it can be autonomous, it can be semi-autonomous, it can be completely manual, kind of up to the customer. What is a reasonable percentage of things that can be fully remediated? I think it depends on the customer profile. So a lot of, I would say, like startups and tech companies tend to have very high risk appetite because they value productivity. You're like, it's fine, let it break, we'll deal with it. And then when you work with some of our customers who are like Fortune 1000 type customers and have huge scale presence in hundreds of countries and a lot of manufacturing, the stakes are a lot higher. And I think in those examples, it's more so the modality of it. So cloud is very difficult, but endpoints are a lot easier. So people are like, more okay with laptops being maybe in like a weird state versus a cloud server having some issue. And the reality of it also is like a modern enterprise has maybe, I want to say like 80% of their attack surface in dev and staging, right? So in some ways, like if you can really isolate the types of things that you're automating, there's huge attack surface that you can fully automate. But It is a journey and we want to make sure that customers feel comfortable. And usually where we start is like in the smallest slice. It sounds like there's two main agents then, one that builds the graph and then the other that actions on these vulnerabilities that come in. Are they the same agent architecture under the hood or are they different? Yeah, so they're actually quite different. At Cogent, we have three types of agents. So you have these interactive agents that work with customers and... they have different latency requirements compared to the second type, which are background agents. So background agents work in the cloud. Those two examples are just two examples of agents that we have. Actually, there's a lot of automation across the platform. Even when you upload something into our portal, there's a file analysis agent. It's going to go understand what you uploaded, map that to the rest of the system. It's going to maybe write some code to plug it into an existing framework that we have. And so there's tons and tons of background agents. And then the last type of agent is actually coding agents. So we have a, I would say like pretty AI native stack in the way where we've invested a lot in our software development lifecycle. So native tooling around how we build software, like agent harnesses that make it easier for people to think through what invariants look like, write tests. And I think that type of agent is also very powerful. because it allows us to ship quickly but durably. And yeah, I definitely think it's like one of the areas that like a lot of folks on the team are super excited about. And that's like an agent that's internal to the company, but then the interactive and the background agents are the ones that are in the product. Is that correct? Exactly. Maybe diving into those. So understand kind of like the latency requirement differences. How does that end up manifesting in the architectures themselves? Do they have drastically different system prompts, different tools, different harnesses completely? Maybe to answer that question, you have to understand the problem constraints that we have. So security data is very high scale, right? So easily like terabytes of data per customer for each data source. And so what that means is when an agent is doing data analysis, it needs to do some like heavy duty lifting to figure out like how to analyze things. And a lot of our systems are built for performance because of this requirement. The second piece of this is safety, right? So we are a security company and we are dealing with highly sensitive information, right? Like vulnerability information and like all types of context that needs to be very carefully handled. And so the way we think about the architecture of anything that touches customer data requires a lot of defense in depth. And concretely, like, you know, we do run agent sandboxes, but the level of isolation that we apply to these sandboxes is very, very deep. So these sandboxes basically have many layers of permissioning where Most of them basically operate at the box level with read-only permissions. And we have this kind of really interesting policy engine that allows us to like elevate access. And so you're never ever allowed to like, even if the agent's hallucinating and wants to like do some kind of right action, it's one, not allowed to do that. And then two, when it needs to do things like that, it actually needs to do it outside of the box in a way that is like fully secure and deterministic. And so we never want to be in a place where the agents are doing things that we're unaware of, or it's capable of doing something that the system is disallowing. Are both the background agents and the interactive agents, do both of them have access to these sandboxes? Exactly. Yes. So the only difference between the interactive agent and the background agent from an architecture standpoint is that interactive agents need to have way faster latency. And so there's a human behind it. And we tend to kind of like short circuit a lot of the prompts that we're using as well as the trajectories. And a lot of the MCPs that these agents are using tend to have very fast kind of latency requirements. Background agents go on long running tasks. They can do whatever they want to do. They can retry. There's like a lot of value that you can get. And it's almost like when you don't have this speed and latency constraint. You don't really have to build in a certain way. And that gives the agent a lot of freedom, actually, to get the most maximal potential of its reasoning capabilities. What does it mean to short circuit an agent trajectory? Is that like, hey, after it gets to step four, we're going to force you to give an answer or something like that? Not quite. So when customers are kind of like looking to an agent to answer a question, so like to make things concrete, a lot of our customers will use Cogent as ChatGPT for your security data. Because, you know, we built this graph, it's super helpful. And in the enterprise, it used to take four hours, or in some cases, like multiple weeks to answer a really hard data question. And today, the agent can do that in three or four minutes. But this three or four minutes has a lot of architectural decisions behind that, right? It's a lot of text to SQL, a lot of like schema, I would say modeling, to make sure that the agents are able to understand exactly like kind of what the universe looks like and can come to an answer in a very fast fashion. When I say short circuit, it's really more so around being right the first time and not being able to have the luxury of doing multiple retries, as well as ensuring that there's enough context before the agent is taking any action. So another way to think about this is our agents that are interactive have a deeper planning phase. And if it doesn't pass any of the tests that we have in the planning phase, it will short circuit its activity. We kind of like want to make sure that there's enough context before the agent moves on to the next execution stage. Doesn't a deeper planning phase mean that it will take longer to run though? It does take longer to run, but I think how we've kind of like figured this out is the planning phase actually makes like one-shotting a lot easier. And like one-shotting, not just with like text or SQL, but also with like writing code. And so it's almost like if you plan correctly, you're able to do it in one or two loops. And that basically means that the agent can return with like pretty high accuracy within like a couple minutes. Is the user affected by the planning phase? Like they'll wait for the minute if it plans and the 30 seconds for it to one shot or something like that. Yeah, this is a good question on UX. And, you know, I think the users have like instant feedback as the agent is planning. The way we think about like product in general is explainability is a key aspect of how users... kind of consume the outputs of our agents. And so everything's transparent to the user from the planning, the reasoning, as well as the tools that it's using. And what's helpful about that actually is one, they can catch us when we make mistakes and give us good feedback. And two, it's a lot more interactive and users can kind of steer the agents as they're kind of doing their work. So it's not like a, you know, it's almost like the concierge that goes away and like comes back with this like pristine thing. I feel like our flavor of interactive agents is a lot more interactive and iterative, right? You kind of get to see what's happening. And if you think there's like some problem, like, yeah, feel free to interrupt the agent and tell it what to do. So how do you surface that? Is it in like a typical chat interface, like I would expect with chat GPT, or is it different? Yeah, so it is in a typical chat interface. And the primary reason why is because a lot of our customers are like kind of enterprise. And we wanted to basically have a UX that's very familiar to them. And I think it's been very helpful, actually, like for folks who've like... starting to onboard into a much more agentic user experience. And I think what's also cool about the UI that we have is the agent's actually capable of creating agentic front-ends. So creating charts, building little building blocks that help the user with different tasks. And those are delighters, I feel like, for a lot of users who have never seen this before. But ultimately, also, I feel like humans are very visual. And text may not always be the best way. And so having the capability to show the user exactly what they're looking for is very helpful. Interesting. How often does it do that? How often will it generate something that's not text? Pretty often, actually. So reporting is one of the, I would say, top use cases for any security analyst. In any given day, at least a couple minutes to hours is spent doing reporting. And what you're doing is you're pulling data, pulling metrics. You have somebody that has a slide that needs to see. And yeah, like, I think the agent has basically effectively removed the need to like know how to do these things anymore. You said earlier that showing kind of like the trajectory of what the agent's doing is a good way to give visibility and show users what's going on. Do you think that as people get more comfortable with AI and trust things more, that desire will go away and people will just want to see like a final response when it's ready? Or do you think that will always be a part? You know, as like engineers, we've all done the like, oh, like. After the first time of testing like a clock code or like a codex kind of like harness, you're just doing whatever it tells you to do. And at the same time, we are like our customers are security people who are really, really skeptical of everything. Right. And like definitely a lot of AI kind of like skeptics in the security world. I think it's like very human to start trusting something once you build a workflow around it. And I actually think it's more so on the tool providers to have the responsibility of ensuring that AI is safe. And the reason for that is, I feel like the behaviors that we're seeing in a lot of our customers is that people trust the agent almost too much. And that happens after the agent performs really well for a lot of tasks. But they sometimes forget that agents have tasks that they've never seen before. And for tasks that they've never seen before, they were likely not going to do well. And there's almost like a feedback loop that needs to be part of your mindset as an agent manager. And I think that's really important. And it'll take some time for us to get there where this now becomes the new way of how people think about using these tools. Talking about kind of like being an agent manager, and you mentioned some kind of like background tasks that happen in the platform. If stuff is triggered in the background and runs in the background, how is it surfaced to the user? How does the user know what's happening outside of the chats that it's like kicked off? Sometimes the way I describe Kojin is like, if we're doing our job, you don't need to come into the platform. And we effectively let the security team be on autopilot, right? The agents are figuring out what your risk is. They're prioritizing your tickets. They're doing a remediation for you. And you're really logging into like manage your agent and like report on what happened. However, there's always going to be drifts, right? So in our production system, there's a lot of monitoring around drift, both at the eval level for like models and prompts, but also for behaviors, right? So we talked about the data science agent, right? So when... A new type of context enters the environment that we've never seen before. It's a high chance that there's no context on how to deal with this type of data. And a perfect example might be a big company goes through M&A, right? All of a sudden, there's a new set of assets that are now showing up in the environment. So what happens? We have to go tell the user, hey, Mr. Manager, I noticed that there's this new thing and you have not told me what I'm supposed to do, given this, like, here's my recommendation on what needs to happen. What do you think? And so there's actually this like, um, feedback loop that is really important in any kind of context layer, which is the entry point, right? I actually think that entry points are more important in context layers. What do you mean by entry point exactly? Entry point is where the user is able to provide context, right? And that can be through the product. It can be through Slack or Teams. That can be through a ticket where you're responding to like a bot. It can be through anything, but... The more entry points you have in your context layer, the more live and real the data is, and the more real time you're able to kind of surface the right things to your end users. I completely agree with that. I mean, we build Langsmith, which is like observability for things. And a big part of what we emphasize is just getting more feedback in there. So you can slice and dice all your traces and look at things in particular ways. And we're not doing anything yet intelligent around it with like a context graph or anything like that. But I can totally see how that feedback. helps kind of like, yeah, build this knowledge that you use. How do you guys try to do that to the max? Like what, like, do you send a bunch of Slack messages? Like what advice would you have for others as they think about, hey, I've got this data, I've got these agents interacting with the data, but yeah, the more feedback, the better. How do you get more feedback? Yeah, that's like, I think it's super challenging chat-like problem. And I think that the way to think about feedback is you need to have a product that creates value to the user. And your product needs to have enough engagement for the user to see value to give feedback. I think that's like prereq, right? Like independent of like any technology choices. And with that as kind of the main, I would say like prereq here, I think it's really important to have an agent that lives where the work is. And that's like a product principle that we have. So a lot of SaaS tools and, you know, I've built a lot of SaaS tools before in the past. kind of live in this world where you come to our tool, we lock you in, you know, you have to log into this thing. They're like workflows that you need to learn in order to do your job. I think in order for you to maximize for context capture, you need to one, like be where the work is happening, right? Whether that's like our teams or in some ticketing system, or in some other cases, it could even be like offline, right? Like there are ways to capture that telemetry. I think two, you need to build what we're calling supply side agents. So this is basically the context forms that I was talking about, right? So you don't really need to have access to the user. What you need access to is the system of context as well as decisions are being made throughout the organization. And with enough agents that can do enough long running tasks, you can basically figure out, oh, like here are the five tickets that were closed in the last day. Here are the people that commented on it. Here are the decisions that were made as a result of closing this ticket. And capturing that in a way that is almost like a nervous system of how this organization works is highly valuable. So those are probably the two things I would think about. You mentioned that both of the agents have access to a sandbox, and it sounds like for some, they're writing at least some Python code or some code to manipulate data in some way. How similar are these agents under the hood to coding agents? So I think the... Coding capabilities are probably similar in the regard that we use models that are coding models for coding tasks. But the harness of what needs to be true, the tool calls that you're making, the types of data that you're dealing with is very different. So for a coding task, what's actually really important is understanding almost like next token prediction for the next line of code. And that type of task is very different from an analysis task where you're looking through tons of data. and you need to understand, oh, like what is correctness in the kind of like universe of schemas in an organization. It's also very different from actually executing a production workflow to validate a patch. And that often requires you to understand the deployment path. It requires you to understand the kind of like dependencies between different systems. And so I would say it's actually quite different from like coding agents. How does that manifest in the harness itself? Like, do you guys use skills that are like popular in coding agents? Do you have, I think a lot of coding agents have access to a file system and bash. Do you give your agents access to file system and bash? Do they have other tools that are even more critical than those for doing work? Yeah, so I think from the structural standpoint, our agents are very similar and they use file systems, they have bash, they have skills. And I think the file system and text as kind of like a way to pass context around has emerged as like a very efficient way to do what's called these days agentic rag, right? Like being able to do retrieval in a file system. And I think we do benefit from a lot of that. But there are aspects that are really challenging to do within that context window or file system. And this pertains to things that have really huge amounts of data, right? So you can't store like one terabyte of data in your file system and have it figure out, okay, like how do I build a in-memory graph and figure out what's going on? So I think for things like that, you still need some kind of structure. And that's where like having like databases, indexes, things like that are super helpful. And then you just give the agent tools to read, write, make queries to it. Exactly. One like interesting thing about what we do, which I think is kind of like novel is We actually have a lot of really interesting tasks that require sub-agents. And the sub-agent problem is super interesting because you have an agent, it's easy to orchestrate. When you have multiple agents that can create agents, this orchestration loop kind of fails, right? How do you pass the context? How do I know if it's correctly instantiating the right sub-agent? And the reason why you need a sub-agent actually is primarily for two reasons. One is for parallelization. And then two, the other one actually is for some kind of like subtask that is not in the critical path of your primary task. And this is actually like a really interesting way to think about how like agent activities can formulate, which is just like a human, right? Like you're doing your main task, but as you're doing your main task, you actually have a lot of like side tasks, kind of like side tasks that help with doing your main tasks. And oftentimes you don't know in that trajectory yet. whether or not the side task will help your main task. And I think this kind of capability to spin off different thoughts or like almost like side quests is super helpful for agents because it's one not coupled in the same runtime event loop. And two, it actually creates more context. And the agent is able to basically figure out if he needs to throw that context away. How does that work exactly? Like if you've got the main agent running, it spins up a subagent to do some work. Will that work in like the same sandbox with the same context as the main agent or does it get a copy? And then like... does the main agent wait for it to finish or is it running kind of like are they running in parallel and if so how does the sub-agent communicate back to the main agent yeah so we kind of leave it up to like the use case so there's not like one way in which this is supposed to work and i think what we found anecdotally and empirically is you only really need to wait for the sub-agent to come back if you're doing a sequential task and if you're really doing a sequential task like i'm not sure if you should be using a sub-agent But, you know, there could be like interesting elements where let's talk about like a long running task within agent swarm, right? So there are many different ways you could do this. You could have like multiple agents spin up with one single orchestrator. You might have a route loop where you're just doing a bunch of things. You might also have within each agent that's running in that swarm spinning off its own things. And I think it's more so about like decomposing the tasks. So tasks that can be decomposed are well suited for parallelization. And that's where like I recommend actually using subagents. things that require like agent-to-agent communication, where it's not just like singular process, right? Like, I think that's where you probably don't want to use a sub-agent. You probably need more orchestration at like the outer level. What models do you guys find work best, or do you use a whole family or mixture of models? So we definitely use a mixture of models. And the way we kind of think about this is we're agnostic to the model provider. We just run benchmarks and like figure out kind of what models are better now compared to like, you know, yesterday, because things move very quickly in the space. What I will say, like for a long time, we are big fans of like Anthropic. They've had really good coding models. But we've also, I think, seen a lot of really like good success recently with a lot of these like models that have been more trained on the cyber side. So with like open AI is trusted access for cyber. And like, there's like some of these things where there are like really interesting ways in which they've pre-trained the model for like cyber tasks. And it's been really interesting to see how like the evals pair out and like how they're more capable or less capable and like where the calibration should be for some of these newer models. What does your eval system look like? So our eval system is kind of like a two-tiered system. You have the offline harness where, you know, it's like multidimensional, I guess, like eval metrics. And you have an online kind of harness where it's evaling our product as it's being used. And we feed those scores back into like this, like, I guess, like global scorer. And it's kind of like helping us figure out if there's live drift in the models and the prompts. I think what's interesting about like some of these dimensions is there's obviously like some of the obvious ones, right? Like correctness, usefulness, things like that. But we also have some like kind of subtle eval metrics. So an example of this is like, We have this metric called feel safe recover. And basically what it is, is did the model basically fail at doing a simple task and did it try to recover and was it successful at recovering? And high frequency of this metric basically correlates with us messing up the prompt, right? Basically it's helping you differentiate between a, is this a really hard problem versus did we introduce something that's like causing the agent to trip up? offline tests that you're talking about, how many of them are like unit tests, so to speak of like a single LM call versus like you guys have your own version of terminal bench or something like end to end long, long horizon? Yeah, so definitely a spectrum of the simple like tests all the way through to like more like long horizon, almost like process oriented testing. And I think the evals are there for different reasons. So On the simpler side, you have golden labels, golden datasets, and we're doing a lot of outcome evals. This is especially important for the interactive agents I was talking about, especially if there's a Q&A involved, you need to make sure that the agents are returning things that are well-defined. And the longer term or more long-running side of things is where we're really looking at the traces, right? Understanding tool sequence, understanding trajectories, understanding if there is a faster way for the agent to converge. And so like all of those are really important. And when you look at those trajectories, are you looking at those manually or are you now feeding them into coding agents and asking them to do that? Or do you have eval metrics themselves that look at those traces and score them on, you know, this is five out of 10 in terms of like effectiveness or something like that? Yeah. So we have, I would say, a pretty big fleet of LLMs as judges, and they are a big part of how we think about eval harness. And it's basically like, different capabilities of LLMs as judge. So at the most simple level, it's just an adversarial LLM. And then at the most complex level, it's LLMs that have full environmental parity with the actual agent that we're evaling. And so it's able to fully basically replicate or like reproduce some of the outcomes that it saw in a trajectory and adversarially assess if it was correct or wrong. Interesting. So like agent as a judge, almost. Exactly. For the online evals that you guys do, do you run those over 100% of online traffic? Do you sample down to 5%, 10%? Yeah, so we definitely sample down. I think for online traffic, it's not super helpful in general to run it on everything. And I think this comes from some past life where we're doing a lot of more classical machine learning. For online, I feel like the key is making sure that you have representative distribution. and making sure that these evals are actually reflected back into some kind of feedback loop. And the whole point of online eval is understanding if you need to call an incident. In some ways, it's a symptom for an actual problem that's degrading in your online system. And so that is where we want to spend most of our time. And a lot of the efforts of making sure that the system works really well is kind of offloaded to our offline evals. I completely agree with what you said around kind of like using online evals to kind of like help spin this kind of like feedback loop. What does that look like in practice for you guys? Yeah, so what that could look like is we have different products that are agentic and the agents are producing outputs as they're like one, either doing background tasks or two, interacting with the user. And those traces basically get fed through like some eval system. There's this kind of like a harness that's like running. on the different, I guess, like metrics that we care about. And when we see some kind of regression or some kind of like, I guess, threshold that drops, we basically want to one, be notified and then two, have an agent look at it and say, oh, like what's happening here. Right. And a good example of like how you might do this is through some kind of clustering to figure out if there's like a more deep-seated problem. And sometimes it's, we need to go talk to the user, right? Like there could be like instances where The agent is not misbehaving. It's just operating under a situation where it's never seen the context before. And so naturally, the evals are not kind of up to speed there. For a lot of the long-running tasks that you talked about earlier, it seems like a key part of it is kind of like learning and adapting over time. I feel like one of the things that has been really hot recently is memory or continual learning in various forms. What do you think this looks like in the limit for a really good long-horizon learning agentic system? Yeah, I think in the limit is like the keyword here. I think we're very early days in terms of memory research, as well as like what people are learning kind of in the field. What we are doing, which I think is also kind of interesting, is we're differentiating between hot and cold context. And what that actually looks like for us is hot context is kind of like, you know, your session context, right? You're doing compaction, you're like generating changelogs, there's like all this like metadata. Um, cold context is basically almost like archived context. You're building a book with like a, contents page of all the things that you've seen before in the past. And I think what's like kind of interesting is that we are also plugging that into our knowledge graph. I was gonna ask, yeah, so, so hot and cold, is that the same as like short-term and long-term memory kind of like hot is short-term within the thread and then, and then cold is long-term between threads across threads. So I think it's like similar, but maybe not quite the same. So I think when people talk about short and long-term memory, it's more so is this context that I will use right now and throw away. Long-term memory is something that you just store and you have access to. And the reason why we use hot and cold versus short and long is there are a lot of hot contexts that stay in the context for a long time and for good reason. So an example of this is in our context engine, we actually have a much more higher level context graph that's not our knowledge graph. So the knowledge graph that we have is like a high scale property graph that's like really good for like running queries across like huge amounts of data. You can do multi-hop traversals and things like that. But in the context engine where the agents are like operating kind of in their like memory, they're building these like little graphs that can stay hot for a very long time. And that can be across a long running task that can be across like handoffs to like other agents. And I think the distinction there is not that this context has existed for a short amount of time. It's that this context is actively being used. So it's almost like active versus inactive. I mean, maybe short term is kind of like the wrong, but it's like session level where session is like an ongoing thread. And that could be like, yeah, it could be going on for a day. It could be going on for a week, but it's a continuous thread. And when you start talking about like sub-agents and sharing context across threads, that becomes maybe a little bit overlapping because you could have some piece of context that's shared between two threads, but like it still could be hot as opposed to cold. What is the difference between the cold storage and the context graph that you guys have? Yeah, so I think that the key there is like we want to differentiate between organizational context and agentic task context. So organizational context is context about the system, context about decisions, and that goes through its own agentic ETL that's very high scale, right? There's like all these events propagating and the key of that system is to maintain a source of truth and to have agents that can basically color the graph with high confidence inference that does enrichment of almost like the state of the world. And then you have these agents that operate on top of this context layer. Right. And these agents themselves have kind of like a context harness or like a memory harness. And that's really like the world of like making sure that agents have access to the things based on the tasks that they need. So maybe the way to think about this is the hot and cold contacts that agents have are task dependent versus the context graph that we're building is kind of like a much more rich version of what the organization could look like. as well as kind of like a brain almost of all the things that are happening in the organization. What does the format of the cold context end up looking like? You mentioned kind of like a table of contents for a book or something. Is it, and I think memory is like super early on in general, but I've seen some versions of this where it is basically just a bunch of files on a file system. And that could include skills actually. And agents.md files, which are kind of like instructions, but. but skills for a bunch of different things that the agents learn over time. And yeah, it is literally just like a table of contents for a book, and you've got all these different skills. Is that similar? Like, what is the data structure you guys use for this cold term? Yeah, so we kind of have like two different, I guess, types of data structures here. One is something that makes it easier to like do in-memory graph traversal. So it's like an actual data structure that lives in memory. And then two, it's basically text files. and text files that you can also plug into things like SQLite or like some kind of like, you know, indexable database. And the reason why is like the code context as it accumulates can get pretty big. And there, you may not want to like, the value of archiving code context may not be very high if it's very well constrained to the type of problem they were working on. So you always almost want to have like context on past sessions and like a lot of this like metadata that could or could not be useful. How do you guys think about accumulating the cold context? Does it happen? Like, does the agent write to it directly as it's running? Or is there some background process that accumulates it? It's a background process. And so like, the consideration here is not, is there more value in like generating more context, like as the agent is doing its work? I think the consideration here is one, when agents are doing work, we want to make sure that they're like very focused. And two, there's actually a lot of like almost additional context that needs to go into how code context is being curated that has nothing to do with the the actual tasks that the agents are doing we experiment with memory in a few different ways and i think the first way we added it was basically just treating memory as like files on a file system and letting the agent read and write those as it kind of goes along but to your point reading is actually kind of easy for it to crock but writing and especially like okay like where should this fact go and like do i need to reconcile with other things that quickly starts to yeah add other instructions into the prompt or become just too long for it to do as it's running and so i do think background you see kind of like anthropic release streaming as part of cloud manage agents leta has sleep time compute i think this idea of like running processes in the background just to store context will become more and more popular i do think it shares a lot of similarities to what you're doing with the the context graph that you guys already have like you have these processes that seem to be acting in the background and using llms to take raw data and distill that into the context graph And so I don't know if you think about the processes for that versus the cold as like in the limit, are they just the same processes and they'll kind of converge in the techniques and the agents or are there things that are uniquely different about them? So I think the pattern of basically distilling data over time in a background process is probably the same, but the properties of the task at hand is probably quite different. So the scale of data actually matters a lot and how much time you have to run these things also matters. But, you know, besides that, I do feel like given enough compute and like, you know, enough data, you can probably go pretty far. And who knows, maybe there's a world where agents can do this really fast. Talking about kind of like compute, I mean, it sounds like you guys are running a ton of agents. How do you guys think about pricing? Do you charge? Do you directly kind of like, do you let people bring their own API keys? Do you charge directly for tokens that you guys use? Do you have a higher level kind of like unit that you charge for? Yeah, this is like a pretty contentious topic. So we charge based on number of assets. And this is like industry standard for cybersecurity for volume management. And the reason why is that number of assets that you own is kind of analogous to the attack surface, right? So you kind of pay for what you're being protected on. And the reason why we charge in this way is because customers understand it and it's really familiar. It doesn't open up questions around like, oh, why are you charging in this way? We, I think, have had a lot of conversations and thoughts around token-based pricing, outcome-based pricing. There's definitely a lot of thoughts in that space. And I think where we are today as a company and also as a product is we care more about customer usage. I think the nice thing about us is also that we're in a space where customers are willing to pay for the product independent of usage. Kind of like you're paying for these autonomous agents to protect your organization. And so we don't really have that problem where we need to figure out how to aggressively price our customers yet. You recently launched Cogent Research. What's the thesis behind that? Yeah, so I've been really excited about Cogent Research. I think it's one of the highlights in the last couple of weeks. The interesting thing about Quotin Research is I think we're at this bottleneck for like security problems where it's no longer a product problem, it's really a research problem. And the reason why I say this is a lot of what we do is verification. And in order for me to deploy something into a customer's production environment, I need to provably show that this will not cause any issues in their environment. And so where that really leads us is into this frontier of open problems in the intersection of AI and security. And so the charter of Quotient Research is really about taking some of these techniques that we know of, like formal verification, building these like, you know, provable systems and marrying them with a lot of AI techniques like RL, like open models that we're going to like open source at some point, building these security benchmarks that help kind of the community think through like what it actually means to do defensive security at scale. And I think ultimately like that research will allow us to kind of lead the way people are thinking about the problem. So one really exciting thing is our research lab is actually headed by two ex-Google DeepMind researchers. And I think one of them is the actually primary author of Google's Codemender, which is their version of Cloud Code Security. And kind of the reason why it's really important to do this in an applied AI lab versus just like a big tech company or an AI lab is precisely because we're well positioned with actual enterprise data. And having, I think, the deep amount of customer access that allows us to do the interesting research that helps answer questions around, oh, how would I autonomously do IAM? How would I autonomously have privacy-preserving policy enforcement? There are some interesting questions that you can only answer outside of a research setting. And almost, I think, the vision of Cogent Research is basically applying a lot of these learnings from what we're seeing in the field towards a lot of our kind of research problems and kind of thinking through what the newer version of autonomous security should look like. How big is the team already and how big do you want to grow it? Yeah, so the team right now in total is about 30 plus people and we're actively looking to grow. We have offices in SF and in New York and you know I think we're at a very exciting time where The timing is really good for both building an AI, but also thinking about autonomy. And I personally think that being an applied AI lab is one of the best places where you can one, like grow a ton and two, work on really interesting things. And so, you know, I think we have really ambitious plans to grow the team this year and we're looking to hire. What is the right profile for you guys or for like an applied, you know, for an applied AI lab in general, like what is the right profile of builder? Yeah, it's like such an interesting question because I think the meta question is like, in an age of AI, what is the right profile of a person who, you know, will be successful, right? I think Cogent is one of those like companies where there's so much breadth of things that you could do where it doesn't really matter, right? We have huge data and infra platforms to like run the high scale data analysis that we're doing and processing. We have, we've built our own in-house agentic development platform for our agents. we want to do post training there's like all this stuff we have security research there's like all these things going on and of course there's product engineering um i think the number one kind of property that i see a lot of people tend to like succeed in in today's world is actually like having high agency and i think what that means is being willing to you know figure out what's going on and use the tools even if they're kind of in a nascent state I think creativity and taste are also really important, but you do need to have the right fundamentals as it pertains to the type of task you're working on. So an example of that is at Cogent, we like to think of ourselves as having like a high bar. And what that means is we really try to stress test the way people think about problems. And that transcends just knowing how to write code, right? It's kind of like, are you able to reason about a problem from first principles? Do you have the right frameworks to think through what that means? And then with that as the foundation, Is that like a 10x or is that a 100x improvement with AI coding? Do you have any tips on how to interview for agency? Yeah, it's a really challenging, I guess, like thing to assess. I think one thing that I look for is actually the way people have behaved in the past. So agency to me is actually a trait. And people leave marks throughout their careers. And having someone tell a story about this is kind of the best ways to kind of like dive deep. I think the way to maybe like interview for it is to go like maybe five or six levels deep, really question the motivations, the reasoning, what was the circumstance, why was it hard, what decision did they make and why was it actually something that was, you know, really difficult in that moment and what did they learn from that. I think the feedback loop of both learning from an experience as well as like delivering on the experience are both equally important. You mentioned open source models in post-training, or you mentioned post-training in RL a few times. Are you guys actively using open source models? Have you experimented with them? How do you think about their place in your platform? Yeah, so I think open source models are great. And I think the rate at which the open source models are improving has actually been really, really impressive. And the way to think about open source models is it's really just a better way or like a cheaper way for us to do some of the experimentation that we want to do. And I think over time, as the open source models become a lot better, probably they will be a staple in every kind of like applied AI setting. And for us, actually, a lot of our tasks don't necessarily require a lot of reasoning. So there's like almost no reason why we would not be using an open source model. I feel like there's two reasons that people use open source models sometimes. One is they argue, basically, I can RL it, I can post-train it, and it's going to be better for this task. The other one is it's just going to be way cheaper. I don't need the reasoning. For you guys, I'm assuming it's a blend of both the summer guys, but is it like 90% cost and 10% performance? Or is it 90% performance or 10% cost? How do you think about it? Yeah, so right now we're less concerned of costs. And for that reason, you know, I think the frontier models have been like really good for us, just like in general, thinking through like kind of, you know, you go to sleep and then you get model performance improvements, right? That's like pretty good and pretty sweet. I do think that there are some like really interesting points of leverage. So for us, it's not so much about what the models were pre-trained on. It's what interesting context we can post-train on. And working in the enterprise context, it's really these highly complex environments that need to be tailored, right? So to give you an example, some of our customers have 10 technique systems, right? Like, it's just like impossible. Like, how do I know what change control process to use? Who should I route it to? And like where that context should live? And these things trip up the agents all the time. I think from like a post-training standpoint, there's actually a lot of alpha for companies that work in this space where the nature of the problem is just like highly complex, that the models have never seen this data before, right? And so like, there's almost like a lot of calibration that we have to do in the day-to-day that hurts the eval, right? So even for a lot of long-running tasks, the like best-in-class frontier models are not necessarily performing at percentages that we need them to be. And I think the way we kind of like meet that gulf is through post-training. If you had to estimate, when do you think a key driver of the Cogent platform will be a post-trained open source model? I think within six months. Yeah. And I think the reasoning is this, right? Like the reasoning is already there and it's more so a matter of having data. And so the question is flipped. It's not about model capabilities. It's about context. As a company, do you have enough context to do the post-training that you need to solve your problem? And I feel like we're there already, having been around for like, I guess, a year and a half. Eons in the AI era. Yes, exactly. And I think that's a luxury for us, but I'm super excited to see what ends up happening after we go on this path. I feel like there's been a lot of buzz around Quad Mythos and other frontier models that are getting really good at security-related things. how much of this is hype how much is this is true where do you think it feels like there is i mean even in in the open source space we get a ton of security issues filed against link chain things some some valid some not valid there's this explosion of of security related code generation code work being done how do you see the space right now and how do you just see it evolving over the next few months yeah it's a really good question and i think What Mythos has really done for the industry is basically highlighting the extent of model capabilities around offensive security. But the reality is pre-Methos, like Opus 4.6 was pretty good. And in fact, like Opus 4.6, I think was capable of a lot of the exploits they had reported essentially in the Mythos report. And we had known about this for a long time, right? I think part of the thesis of Cogent is that attackers only need to be right once and defenders can be right all the time. compression of the almost like window of exposure will mean that you need to have these autonomous cyber defense systems. And so answering your question of like what this means for the industry and kind of like, like how much of it is hype is definitely very real. And one interesting thing that I'm starting to observe is all the coding companies, including the AI labs, cursor, Vercel. Anybody in the dev space, they're all coming up with essentially like security tools. They all are. It feels like they're all, first they launch a coding agent, then they launch a review agent. Now in the last week, it's all security agents. Exactly. And like to me, this is basically SAS 2.0. So like five, six years ago, we had this thing called SAS. It's basically like fuzzy matching for like code scanning. And this was like the cool in thing, like to basically figure out if you have first party code vulnerabilities. And in some ways, I think discovery is like really easy. How to fix this, like basically it's still the bottleneck and like super challenging. I think this will be net good for the industry because there's a lot of vulnerabilities you can't find using existing techniques that the LLMs will help to catch. And the real value actually is, I kind of think about it in like two ways. There's two swim lanes of agentic capabilities. There is the dev tool chain, and then there's the operational tool chain. Most applied AI companies like us operate in operational tool chain where we're automating everything outside of code. The AI labs, these like IDEs, like all these companies that kind of like work around the dev tool chain will have coding agents. They'll do all the things. And I think where it's going to benefit a lot of folks is in the shift left movement. So if you have a skill and you're writing some code, the agent will be able to help you figure out how to write better code. I think that's great, right? Like that will help you have better code that's like secured by default, secured by design. That's going to be amazing. Who's going to go curate the context that feeds into that agent is still like an open question. And what happens when you have vulnerabilities that need to be fixed? So I think there's a lot of operational things that still need to be figured out. And the way I think the industry is going to move is the AI labs and probably a lot of these kind of companies are in like the dev space. will try to commoditize a lot of skills or any kind of like coding agent based skill, right? Like it's not just security, it's going to be taxes, right? The other day I had a friend who was basically filing his taxes using Claude and it worked really well. So I think like a lot of that's going to be true, but you will still have use cases where like things are very operational and challenging and you need to have correctness guarantees that you can't get with like a coding agent. It seems like security is top of mind for everyone these days, but you guys started a year and a half ago. How did you guys get the idea for Cogent? So I was part of the founding team of this company called Abnormal Security and eventually became like leading their product engineering team. And we were basically building one of the world's best machine learning platforms for fraud detection. And today Abnormal basically protects more than half of the Fortune 500. And the problem that we were solving, I thought, was actually very foundational to the way I think about just AI systems in general, but also how to build agents. And that really planted the seed for Cogin. So just for some quick context here. So Abnormal is like an email detection platform. And the really interesting thing about finding malicious emails in an enterprise is that the likelihood of finding malicious email is actually really, really small. It's like less than one in 100 million. So you need really high precision. And on top of that, you need really, really high recall as well, because, you know, what happens when I take an email out of your inbox and it turns out it's from your boss, right? Like I've disrupted your business. And even worse, I could leave it in there, let you click on it, spear phishing email, and your organization gets breached, right? So we were basically building the system, processing trillions of emails every single day with near zero, I would say like false positives and false negatives. And the key to solving that entire problem was basically context. It was not really in the models. And I think that insight really kind of feeds into how we think about Cogen, what we're building here. And yeah, like when you're building behavioral, like classical machine learning models, what you're doing is anomaly detection. But what you're trying to understand is the organizational behavior and the relationships between the people. not the actual email signature, right? So what we ended up doing actually was spending a lot of our time investing in and building understanding of how people communicated. And that became, I think, the way we thought about how, you know, the way you understand if your CEO is getting spoofed, even though the email looks exactly identical compared to what your actual CEO would send to you, is precisely because you understand some of these patterns. And in some ways, what we're doing at Cogent is just solving the exact same problem, but with a much larger scale on the much more important problem. And so at abnormal, I imagine a lot of it was like manual feature engineering to get these signals that would go into the traditional machine learning models. How much of what you guys do now do you think is like manual context engineering versus letting the agent kind of like build its own context? Yeah, I would say it's like half and half. And so we have a Ford Deploy engineering team. And the reason why we have forward deploy engineering is because AI deployment is not just about data and context. It's also about people and organizations. And there's a lot of kind of battles that you have to fight to help people understand, oh, we need this thing. Here's how it's going to be valuable for you. And a lot of the tuning oftentimes that makes our agents so powerful needs to happen through a deeper understanding of the customer's environment. And that's where having our FDEs is like actually a superpower because They can dive into a customer's environment, understand what's going on, feed that back into the agent without the customers needing to do a single thing. And on the flip side, we use agents in the day-to-day of everything that we do. And feeding that back into our product, into the systems that we use is a core part of our DNA. And we're getting a lot of value on that just by itself. But I think there are still limitations to how the agent feedback loop is being structured. And the core learning having, you know... only built this company for less than a year and a half and the landscape having like kind of rug pulled us like every six months is that your architecture is always going to change and your AI hardness is always going to change. And the models are always going to keep improving, right? So I think we've kind of graduated from a world where there's a lot of spectrum development and you're kind of like teaching the AI through skills into a world where you need to do a lot of invariant testing, right? People are talking about like autonomous software development. We do autonomous workflow development for our customers. And I think in that world, you really need to invest more in re-architecture as well as making sure that you understand what to test versus implementation speed. And so I think in that world, it's actually more important for us to go back to the fundamentals of what is this system trying to do? Are we actually trying to solve the correct problem here? And that's definitely, I think, something that we're really excited about, both from a research standpoint, but also from like a product standpoint. In the short, medium and long term, how do you guys think about positioning Cogent against the model labs themselves? The way we kind of think about this at Cogent is in the short and medium term, probably there is a lot of work to be done. And so in the short term, I think we're completely complementary. And what that means is like a lot of these AI labs are launching security tools. They are just other sources in the Cogent. And the problem that we're solving actually is a context problem for a lot of our customers that I think the AI labs will not work on. And there's a lot of like nitty gritty challenges to building integrations and like making sure that things work correctly in like a highly complex enterprise. But on top of that, I also think that there are like highly customized and like domain specific agents and like post training that you have to do to get some of these problems truly correct. And let me give you a perfect example, right? So a lot of people in the coding agent space not just want to augment how coding agents are able to increase the productivity of engineers. They want to reach the 100x kind of outcome of running the mythical cobalt to rust kind of migration, right, for the big bank. And yeah, that probably will be like an eight to nine figure kind of like contract if you ever get it done. And... I think in that space, it's really interesting because like, if you think about it, what needs to be true for a coding agent to be able to do something like that? One, model capabilities need to be there. Two, long running tasks need to be commoditized. Three, you need to have a lot of context on what you can and cannot do. And then four, you actually need to have very strong synthesis capabilities to reach basically parity of the existing system and build the right guardrails, almost like integration tests, right, for the invariants that need to be true. And so out of all of those things, like what is the thing that is customer specific? It's really like context, right? And I think that will need to be true for any kind of, I guess, like coding agent that wants to do that. And that's also generalizable for any domain. So for us in security, it's really doing true auto remediation in production environments and trusting the agents to basically be your last line of defense. And yeah, like when the next zero day drops, right? Like when LLM or one of these other very common tools has the next zero day, you want to be able to trust your agent to do something that you won't have time to do. Lock something down, contain, maybe deploy a WAF, like a firewall, or in some cases, just run the patch, right? Like, I don't care. I need this thing to be safe and locked down. I need to decommission my asset for me. And so what it would take to actually do some of those things, I feel like... is kind of in the, like, I would guess like a hot seat of building the right context for a lot of companies. So in the short term, I think we're completely complimentary. We focus on fixing and like a lot of these AI labs are basically kind of data sources into what you would plug into Cogent. In the medium term, I think everybody that's an applied AI company is building a world model for their customers. The fact that we're doing these things is just making up for the gap that these customers themselves do not yet have a context layer. So every Applied AI company ends up building two things, the context layer and the intelligence layer. And it's just required because that's just how we are today. But tomorrow, like fast forward, you know, three to five years, yeah, I'm pretty sure like most companies will have something, right? And there's a lot of value in being the first provider for that context layer. I think it gives you a lot of moat in terms of first movers advantage of like use cases. And when you're talking about this context layer, just to be clear, This is the same as like the context graph that we were talking about earlier that you have agents go and build. Exactly. Right. Like another way to think about this is like, who is going to help companies understand how decisions get made in their organizations? Right. And this needs to be a real time system. It needs to like be able to change as events are happening. And this is the stuff that the agents feed off of, right? Like the high quality context that tells you how judgment should happen. So the reasoning of the agents can be met with almost like historical data. And it's really interesting because like there's two ways to solve the problem. One is historical data through this like context layer stuff. The other way to solve the problem is through prediction, right? And the way you do prediction probably is some amount of like post-training where you can build these hyper-realistic like RL environments. You're like training reward models, right? You're like simulating all the possible paths. I think both are very valid and both probably will be in the hot path of any like applied AI lab. And I think where this gets really interesting is like even in the world where the context layer is commoditized and you're just plugging into a context layer, the context that you generate as a result of your intelligence layer is highly valuable. And that in and of itself actually would be like, I think the key to a lot of the workflows that organizations need to automate. Do you ever get asks from your customers to make this context graph, this context layer that you guys are creating? more open or use it outside of your intelligence there? Yeah, certainly. I think once customers see value, the first question is, can I use it? And this happens a lot. A lot of customers are asking us for API access to MCPs and things like that. I think it's also different maturities of customers. So the ones that tend to be more tech forward will tend to have these asks versus the ones that tend not to want to have UI. They want to live in familiar interfaces. What do you tell those customers? Do you let them have access to that? So I think it depends on the use case. We tend to want to basically go on design partnerships with customers that are a bit more forward thinking, but we set expectations around how the product works and like kind of what this could lead to for both companies. And I think at the end of the day, the way we kind of frame this is like, that's just another way for us to get context. Do you think the models themselves will commoditize the security landscape? Yeah, this is a really good question because in some ways, GPT-5-5, cyber, as well as CloudMethoes have kind of proven that the models have commoditized offensive security, right? Like today, any script kitty, any person on the street, you and I, we could have access to CloudCode or Codex, and we can compromise our neighbors down the street or like down the office block. And I think what's interesting about that is that On the outside, it looks like, yes, security is already commoditized. I think if you understand the type of data offensive security is trained on, it actually helps you understand that I don't think security will actually be commoditized. So hear me out. Pentesting is a very well-defined industry, right? People get certified for these things. There's well-defined data sets and it's a very well-defined task. And so it's actually generally straightforward for you to go pre-train a model and get very high-quality labeled data. on how you should exploit something. And it's also a field that I think has seen a lot of resurgence in recent years where people are kind of curating these data sets. On the flip side, defensive security data is actually really, really hard to get. And it's also very complex. So the nature of an exploit is here's a bunch of data. Let me figure out how to get in. The nature of a defensive security task is I have a castle. Here are all my walls. How do I patch every single little hole in the wall? And every single castle is built differently. Some are made of sand, some are made of rocks. They have different guards who have different almost like schedules that you need to maintain. There's so much complexity and customization for the defensive stack of every enterprise that a defensive security task is actually really, really hard to one, model and then two, create good label data for. And this is also the reason why there has not really been a definitive security benchmark for defensive tasks. So from that perspective, actually, I think security is one of probably the last bastions that would truly be commoditized because there really is no data. All right. Thank you. That was fun. Awesome. Thank you so much. Thanks for listening to Max Agency. If you liked this episode, leave a review and subscribe. Send feedback or questions to maxagency at langchain.dev. We want to hear from you.

Geng Sng is co-founder and CTO of Cogent, which builds autonomous agents that remediate vulnerabilities for enterprise security teams. Today, Cogent's agents process billions of security events per day, maintaining a live context graph of every asset and vulnerability across customer environments. In this conversation, Geng walks through Cogent's hot vs cold context split, the sub-agents that handle side quests, and the two graphs they run in parallel.

We also discuss:
• Why defensive security is harder for AI than offensive
• Under the hood of Cogent's three agents
• Inside Cogent's “read only” by-default sandboxes
• Why graph databases don't scale for security data
• Cogent Research and the move into formal verification
• Why interactive agents need a deeper planning phase to one-shot

References:
• Abnormal AI: https://abnormal.ai/
• Amazon S3: https://aws.amazon.com/s3/
• Anthropic: https://www.anthropic.com/
• Bash: https://www.gnu.org/software/bash/
• ChatGPT: https://chatgpt.com/
• Claude Code: https://www.anthropic.com/claude-code
• Claude Mythos: https://red.anthropic.com/2026/mythos-preview/
• CodeMender: https://deepmind.google/blog/introducing-codemender-an-ai-agent-for-code-security/
• Codex: https://openai.com/codex/
• Cogent: https://www.cogent.com/
• Cursor: https://cursor.com/
• Google DeepMind: https://deepmind.google/
• GPT-5.5-Cyber: https://openai.com/index/gpt-5-5-with-trusted-access-for-cyber/
• Jupyter: https://jupyter.org/
• Letta: https://www.letta.com/
• Mozilla: https://www.mozilla.org/
• OpenAI: https://openai.com/
• Opus 4.6: https://www.anthropic.com/news/claude-opus-4-6
• Opus 4.7: https://www.anthropic.com/news/claude-opus-4-7
• Vercel: https://vercel.com/

Where to find Geng:
• LinkedIn: https://www.linkedin.com/in/geng-sng/

Where to find Harrison:
• LinkedIn: https://www.linkedin.com/in/harrison-chase-961287118/
• Twitter/X: https://x.com/hwchase17

Where to find LangChain:
• Website: http://langchain.com
• Docs: https://docs.langchain.com/

Send feedback or questions to maxagency@langchain.dev

Timestamps:
00:00 Why mean time to exploit collapsed from years to minutes
02:08 Inside Cogent's Agent Lake architecture
05:11 Why Cogent rejected graph databases
10:48 The trust ladder before agents touch production
15:13 The three types of agents inside Cogent
17:07 How Cogent sandboxes its agents
19:16 Short-circuiting interactive agents with a deeper planning phase
24:31 What to do when users believe agents too much
31:21 Why sub-agents let agents go on side quests
34:59 Two-tiered evals and the metric that catches bad prompts
40:00 Cogent's unique approach to context
48:39 Cogent Research and the move into formal verification
51:33 The single trait Cogent hires for
54:00 Open-sourcing models within six months
57:07 Why defensive security won't be commoditized anytime soon
1:00:51 The founding insight behind Cogent