Artificial intelligence is changing the way we interact with computers. An updated version of Claude 3.5 Sonnet, the flagship AI model of anthropic, is a clear example of this evolution: if the previous version was limited to text dialogue, this new update just announced is able to interpret graphical interfaces and interact with applications autonomously.
An approach that opens up new possibilities for the automation of digital activities, “moving” just like a person would do on a PC. Let's see together how this technology works, and what its real applications are.
How does interaction with the PC work?
Claude 3.5 Sonnet uses a system called “Computer Use API” to interact with the desktop. In practice, the AI observes the screen through screenshots and analyzes what it sees, just like a human user would. When it needs to interact with an application, it calculates the distance in pixels to reach the desired point and simulates the necessary actions: mouse clicks, typing on the keyboard, cursor movements.
We trained Claude to see what was happening on the screen and then use the available software tools to accomplish the tasks.
anthropic
The main innovation of this version? A long-awaited ability: the ability to use any desktop application. It is no longer just about answering questions or generating text, but to actively interact with the software.
The new Sonnet 3.5, practical applications
The possibilities for use are numerous. For example, repeat, a software development platform, is already using Claude 3.5 Sonnet to create an “autonomous verifier” that evaluates applications during their development. Canva, the online graphics editing platform, is exploring ways to integrate AI into the design and editing process.
Is it all as “wow” as it seems? No. Will it steal our jobs? Not even. It’s important to be realistic about the system’s current capabilities. In a flight booking test, Claude 3.5 Sonnet was able to successfully complete less than half of the tasks assigned to it. Even seemingly simple tasks like scrolling and zooming can still be problematic.
On the other hand, she herself admits it anthropic:
Claude's computer usage remains slow and often error-prone. We encourage developers to start with low-risk activities.
How long will it take for Sonnet 3.5 to get perfect? Will it take years, months, or weeks? If you think about it, you already know the answer, or something pretty close to it.
A few notes on safety
The issue of safety is fundamental. anthropic, which is considered the “good boy” of AI companies, has implemented several preventive measures:
- The AI is not trained on user screenshots;
- Does not have access to the web during training;
- Classifiers have been developed to avoid high-risk actions;
- Screenshots are only kept for 30 days, as a security measure.
Future prospects
In parallel with the development of the new version of Claude 3.5 Sonnet, anthropic also announced the arrival of an updated version of Haiku, the most economical and efficient model of the Claude series. A small step in the field of accessible AI at different levels of use. Of course, this stuff still consumes a lot, too much in terms of energy and resources.
Anyway, I find it particularly interesting how this technology is changing our relationship with computers. We are no longer talking about simple automation scripts, but systems that understand and interact with graphical interfaces in an almost human way.
Claude Sonnet 3.5: Balanced steps forward, but in a surprising direction
As is often the case in the field of AI, the true value of this innovation will be measured not so much in its technical capabilities, but in how we will be able to usefully integrate it into our daily workflows. The challenge for the future will be to find the right balance between automation and human control, making the most of the potential of this technology without losing sight of its limits and ethical responsibilities.
The capabilities of Claude 3.5 Sonnet, not impressive in themselves, are nevertheless suggestive for the direction they show. We are still at the beginning of this technological journey: what will happen soon?
