Computer-Use 2.0: Agents Just Got Multi-Cursor — Francesco Bonacci, Cua

Thank you for taking the time, for coming over here. I'm Francesco. I'm the CEO of the company. Alongside me, a couple of other folks, my CTO, Dillon, and my chief of infra, Rob. They're going to work on the stage in a while. But before we do that, who's excited for some computer-using agent talk happening now? Are you guys excited? Lovely. If I were to ask, like, What was a computer-using agent like one year ago? Probably half the crowd would say, I don't have any idea what really computer use means. So today I'm going to take you to a journey, basically from our vision where we come from so far on computer user. This new shape of agents that are talking and up to modern intelligence. So we're going to start with the vision of Quadriver, where we're coming from. And how many of you guys have been working with computer use for one year? How about three years? Lovely. OK. So our team has plenty of experience. We go all the way back our time on Microsoft. We were working on this type of GUI agents, we were calling them back in the days. There is an example of old-fashioned human agent loop. We basically refer to this as a human loop, where you will have an agent loop. You will take a screenshot that the agents will have to reason and plan through. And then you will basically work with an action space in terms of clicking, typing, scrolling around. as the old-fashioned computer use 1.0, just to set the tone for this talker. And we come a long way since this type of computer-using agents. So this, again, I'm going to skim over these slides, but that's the old-fashioned way of representing this agents loop, as a human would do. Here we go. Over two months ago, we released a project in the open source. It's called Quadriver. And we made it working in the background. That means that your computer user will not take over your screen as the computer use 1.0 agent loop was doing back in the days. And it all started from from Codex releasing their computer user model two months ago. So we cannot take the challenge because we were already working with this type of background computer user. So over one weekend, we had something together. And the trick here is really not having your agents take over your screen. So there is a lot of dark magic happening behind the wood just to give you some context. There are some undocumented API living in the Apple framework. It basically ships with your laptop. And as you can see here, like it's in the demo, you have an AI agent that is not taking over control over your laptop. We made it working not only for macOS, but also spanning across Windows and Linux. This is the very first driver that is living on your laptop, and it lets really any agents connect to the underlying operating system, either like using accessibility trees or like a screenshot level approach. We kind of like tackle both. This is what really the agents see for what it concerns. You will have to install Quadriver. The agents will take a snapshot of the Windows state. And you will have to observe And we really take one different action path to really make the ground computer use happening. So you really have to observe the space. In this case, just by calling getWindowState, you get an accessibility tree representation plus a screenshot. And then you will go and try a background execution using accessibility tree. And if that doesn't work, we go all the way and make the heavy lifting for you and just try a pixel background click. This is kind of like best step for a background at this stage. It's not behaving the same way on macOS, Windows, and Linux. So we do some of the heavy lifting for you so that your AI agent can run on this on your MacBook. How we manage to not break anything between release cycles. We have a lot of investment happening behind the scene when we test new releases. We have about eight different application harnesses that we use for making sure that we don't break anything among different releases. Among our early adopters, you can see Clicky, Hermes, Quencode, H Company, and Droid Factory. Huge thanks to them for using Quadriver and basically releasing a lot of upstream contribution in our framework. Without further ado, I'm just going to move to the next part of the presentation, which is going to be intelligence. And I'm going to have our CTO, Dylan, cover that. Hello. So thank you, Francesco. With Kua Driver, we gave an agent hands. But then the question becomes, how can you trust the agent to use those hands correctly and not leave anything broken behind? And to answer that, we had to build Kua Bench. So for a show of hands, who here has heard of Terminal Bench or Harbor? Yeah, so a few of you have heard of it. And if you've ever authored a task for Terminal Bench, then this might look familiar. But in KuaBench, a task is made of three pieces. The setup function, which sets up the machine into initial state. The oracle function, which provides a golden trajectory for the task. And the evaluator, which probes the environment to check if the agent successfully completed the task. Unlike terminal bench, the oracle here is GUI actions. So it looks kind of like a pile of GUI when you write that. And writing environments takes scale and expertise. On desktop, there's more than five platforms that we target. And we try to collapse that into a single Python file. So using the CoolBench SDK, you can write a GUI that works across every desktop platform in a single Python file and use the same SDK to probe that GUI to get usable agent data. Anyone or any agent can author one of these tasks. And when you put that to work, you get a real catalog. We have currently over 130 verifiable tasks, 42 environments, and across five platforms. And each of these are easily reproducible using our CLI. And the latest addition to our data sets is one that we're proud of. With collaboration with Snorkel AI, we built KubaBench KiCad, which tests computer use agents on electrical engineering tasks. using software by real professionals and evaluator functions that actually simulate the circuits. But the results are humbling. The top agent that we tested only got a full pass on six out of 25 of these tasks. Of those six, 100% of them involved editing an existing schematic. And when we start the task from a blank schematic, the success rate drops to 0%. And across all the models that we tested, the leaderboard is flat. No model has achieved more than 30% reward. But once you can score something, you can improve it. If we take a look at the CoolBench basic data set, scaled up to 4K resolution, testing an agent, they typically get around 62% pass rate. But when you switch the agent computer tool from the built-in one to CoolDriver, the pass rate jumps from 62% to 80% using 34% less tokens. And this is primarily because CoolDriver focuses on a window rather than the entire desktop. But our evals might say you can trust model XYZ at task whatever. But how can you know that the eval can be trusted? So before we test a task against any agent, we first try to break the environment ourselves. We have a matrix of agents attempt to do reward hacking and attempting to break the environment. And we take all that data and we compile it into a nice CodeRabbit-style code review. And only tasks that survive our pipeline can enter the data set. And if you ask us how we trust that agent, the answer is that it's just evals all the way down. But to measure the intelligence of an agent, you can't just measure its ability to successfully perform actions. You also have to measure its ability to understand the world that it's operating in. Every run that we record can be forked through any moment in its trajectory. to give you the state of the computer at that moment from there we can probe a model asking to predict the reward the internal state or any other observation of the computer and compare it against the fork and that prediction is the world model of the agent made measurable and with that i'll let robert take the stage thank you dylan uh hello everybody i am the chief infra officer at kua And I'm here to talk to you about how you're probably leaving a lot of money on the table with idle GPUs if you do RL training for computer use agents. So I want to introduce this diagram to you all. Is there general familiarity with this diagram? Or is this something that most of us haven't seen before? Like anyone? Awesome. Very niche. Almost everything on this is not really important for what we're talking about, but the blue portions are. And what those basically represent are GPUs generating tokens for RL training. And if you zoom in on this a little bit, you can kind of see how this typically looks with a sandbox environment. You're going to be generating some tokens, and then you finish your task on a sandbox. And then you're waiting for either a new sandbox to spin up or for your existing one to reset. The problem here is that this is just pure cost. Your GPU really isn't doing anything useful here. And I don't know if you've heard, but GPU time is pretty expensive right now. So as you're scaling, this cost really compounds a lot. And you really want to focus on minimizing this if possible. One thing that you might try to do is minimize the startup time of your sandbox. And you should do that. That's a great thing to do. But especially for computer use style environments, sometimes this can be a little bit impractical. Your researchers might give you a 40 gigabyte environment, and that might just be necessary. And it takes a long time to pull that down and start it up. So how do you design your training infrastructure so that you can minimize the start-up time of the sandbox, even when the sandbox is not well designed to be start-up quickly. So the way we do this is a pool. And this is supposed to be animated, but it's not animating. So I guess I'll just explain to you orally. And what that is is So we have a set of GPUs here, which all want to use sandbox. And what we will do is that we use a demand-based autoscaler to detect how many GPUs currently need a sandbox. And we can grow the pool to be that size on demand. And what that means is that if you have, let's say you have a warm pool that you want to allocate to your GPU cluster. You don't actually need to know upfront what that warm pool size is. We can figure out what that warm pool size should be for you on demand. And that might even change over the course of your multi-day training run. You might start needing a lot of sandboxes, but then as your generations get longer, you might need less. So these also could be easily two to four times cheaper than your GPUs. Having a little bit of redundancy here, you still wind up saving money because you're maximizing the use of your GPU time. Come see me after if you want to see the animation because it's cool. So now when you have this redundancy in your pool, you're paying the cost of that startup time on the infrastructure side, not on the GPU side. So your GPU workers have full utilization. Because we use this, we can give you instant sandboxes for your GPUs for Windows, Linux, Android, and macOS is coming up. And I'm going to hand it back to Francesco to close it out for us. FRANCESC CAMPOYOVANI- Lovely. Thank you, Dylan. Thank you, Rob, for taking this over. We do have plenty of time for Q&A. So if you guys have any questions, happy to take them, either for Quadriver or Quabench. basically what Dylan presented, or ,, which is what Robert covered. Any questions? Otherwise, we can wrap this up. Oh, I see. So the story for mobile Android, there is very far you can go. We are talking with the Hermes team, because they do have an harness that runs on Android. I guess if you're talking about background, there is some level of background that can happen if you containerize a workload. And basically on Android, you can even run your own container or Ubuntu or GUI Docker container within Android. But yeah, the Android ecosystem, especially compared to iOS, is more inclined to that form of background computer use. But it's more towards tool use than really controlling GUI interface. We work with the activity framework and do tool use in the background. Cool, thank you guys.

Three agents click, type, and scroll through three different apps on one desktop at the same time, and the user's own mouse and keyboard never move. That's the live demo behind cua driver, a tool the team built in a single weekend after Codex shipped its own computer use model. Instead of taking over the hardware cursor, it talks straight to the accessibility layer underneath the operating system: UI Automation on Windows, AT SPI on Linux, AX on macOS. Those undocumented APIs let a click land on a background window or a keystroke reach a hidden one, so any number of agents can act without stealing focus from each other or from the human sitting at the machine.

To know whether any of this can be trusted, the team built CUABench: over 130 verifiable tasks across 42 environments and five platforms, each one attacked by a matrix of agents trying to reward hack it before it's allowed into the dataset. Swapping a standard computer tool for cua driver pushed pass rate on a 4K benchmark from 62% to 80% while using 34% fewer tokens, mostly because it watches one window instead of the whole screen. The newest addition, built with Snorkel AI on real circuit design software, humbled every model tested: the best agent fully passed only 6 of 25 electrical engineering tasks, every one of them an edit to an existing schematic, and starting from a blank schematic dropped every model straight to 0%.

Speaker info:
- https://www.linkedin.com/in/francesco-bonacci-70428a121/

Timestamps
0:00 - Introduction and Vision of Cua
2:40 - Overview of Cua Driver and Background Operation
6:34 - Introduction to Cua Bench and Agent Evaluation
10:50 - Cua Fleet and GPU Infrastructure Optimization
15:08 - Q&A Session
15:44 - Discussion on Mobile and Android Support