Anthropic's new AI can use computers like a human, redefining automation for businesses

Anthropic, the AI ​​research and security company, has announced a new set of features – including an updated version of its flagship AI model, Claude 3.5 Sonnet, and a new model, Claude 3.5 Haiku – that will change the way Companies could automate complex workflows and change them. The most noticeable development in this version, however, is a new feature: Claude can now use a computer like a human, navigating screens, clicking buttons, and entering text.

This new feature, called Computer Usage, could have far-reaching impacts on industries that rely on repetitive tasks with multiple applications and tabs. From data entry to research to customer service, the potential applications are diverse – and potentially industry-shaping.

AI moves from text interaction to screen interaction

Since its inception, Anthropic has focused on developing AI models that are safe, reliable, and capable of complex reasoning. With Claude 3.5 Sonnet and Haiku, the company is expanding the model's capabilities even further. The new Computer Usage feature allows AI to perform tasks previously only done by humans, such as opening applications, interacting with interfaces, and filling out forms.

“Computer usage features have the potential to transform the way tasks that require navigation across multiple applications are performed,” said Mike Krieger, chief product officer at Anthropic, in an exclusive interview with VentureBeat. “This could lead to more innovative product experiences and optimized back-office processes.” Krieger emphasized that the new feature is still in beta, but as the technology evolves, it could improve data analysis, visualization, user interface interactions, and more Make tasks more efficient.

“We expect it will be particularly useful for tasks such as conducting online research, performing repetitive processes such as testing new software, and automating complex, multi-step tasks,” he said. “As the technology matures, it could improve data analysis, visualization and user interface interactions, and potentially improve accessibility. We’re excited to see how developers will leverage this capability to create new tools and workflows that improve productivity and user experiences across diverse sectors.”

Early adopters see potential

Anthropic's early partners, including GitLab, Canva and Replit, are already benefiting from the new features of Claude 3.5 Sonnet. GitLab, which specializes in software development and security, has been testing the model to automate tasks in its development pipeline. According to the company, Claude has improved thinking skills by up to 10% without impacting performance, making it well suited for complex, multi-step processes such as software testing and delivery.

Replit, a coding platform, has taken things a step further. Michele Catasta, president of Replit, said the model “opens the door to creating a powerful autonomous verifier that can evaluate apps as they are built.” This could ease bottlenecks in software development, where testing often delays project timelines.

Meanwhile, graphic design platform Canva is exploring how Claude's computer skills could speed up design creation and editing. Danny Wu, head of AI products at Canva, said in a statement: “We are discovering efficiencies within our team that could have a significant impact on our users.”

What does “computer use” actually mean?

What sets this new feature apart from traditional automation tools is that Claude is not limited to specific workflows or software programs. Instead, it can use screenshots to “see” a screen, interact with different applications, and adapt to different tasks as they arise. This flexibility makes it more versatile than current RPA (Robotic Process Automation) technologies.

For example, in a demo shared by Anthropic, Claude helps fill out a supplier request form for Ant Equipment Co. In the video, Claude first takes a screenshot of the computer screen, notices that some necessary information is missing from a spreadsheet, and then navigates to a CRM system, finds the required data and fills out the form – all without human intervention.

This level of automation could have a significant impact on industries such as finance, legal services and customer support, where tasks often require switching between multiple systems and applications. “Claude could open spreadsheets, perform analysis and create visualizations. For customer service, it could navigate CRM systems to quickly find and update customer information,” Krieger told VentureBeat.

Security and privacy concerns

However, AI's ability to control a computer raises serious security and privacy concerns. Anthropic has integrated several safeguards into the system to address these risks. The company clarified that Claude cannot access a computer without a developer providing the necessary tools.

“Claude can't 'just use your computer.' “The computer usage feature requires developers to provide tools such as a screenshot tool and an action execution layer that allows Claude to perform mouse movements and keystrokes,” Krieger explained.

Anthropic is also treading carefully, releasing the feature in a limited public beta only available via an API. This will allow developers to test it in controlled environments before it becomes more widely available. The company has also developed classifiers to detect abuse and prevent AI from interacting with sensitive websites such as government portals. “Our methods for searching for prohibited activities are designed to protect the privacy and confidentiality of customer information,” Krieger said.

A new era of office automation?

In the short term, companies could see immediate productivity gains in areas such as data entry, customer service and IT support. However, as the technology matures, the potential applications could extend far beyond these initial use cases.

Imagine a world where AI handles complex legal processes, from reviewing contracts to filling out compliance forms. Or imagine AI helping doctors navigate electronic health records and diagnose patients by cross-referencing medical databases.

Claude's new Computer Usage feature brings us closer to a future where AI can perform a variety of tasks spanning different software applications and systems. This gives it a level of flexibility previously unimaginable for AI technologies, which were often limited to specific, narrow tasks.

Proceed with caution

Still, it's important to remember that this skill is still in its early stages. Claude's computer skills are not yet perfect, and Anthropic admits that he struggles with tasks that humans find trivial, such as scrolling or zooming. “Because it is still in beta and may occasionally miss short-lived actions, we recommend human oversight for high-risk tasks,” Krieger said.

Nevertheless, Anthropic is committed to further developing the technology. “We have developed new classifiers and rapid analysis tools to detect potential abuse of computer usage features,” Krieger added, indicating that the company is serious about addressing the risks associated with this powerful technology.

What's next?

As AI continues to evolve, the way we work may change dramatically. For business decision makers, the benefits of automating multi-step workflows could be significant. But this also raises questions about the future of jobs that rely on precisely these tasks.

For now, Anthropic is focusing on the immediate benefits of Claude 3.5 Sonnet and Haiku while ensuring the technology is used responsibly. As Krieger put it, “We are excited to see how developers will use this capability to create new tools and workflows that improve productivity and user experiences across various sectors.”

With companies like GitLab, Canva and Replit already exploring its potential, it's clear that AI will play an even bigger role in the future of work – perhaps sooner than we think.