RESEARCH

Gartner exposes AI's growing value gap

Lots of companies are still racing to embrace as much AI as possible, but new data shows how much enterprises are struggling to show measurable business outcomes from AI.

This week at Gartner IT Symposium/Xpo, the company's annual flagship event, the analyst house revealed contrarian data on AI investment from surveys of CEOs and CIOs and other Gartner research:

• 81% of CIOs are increasing AI spend

• 44% of CEOs struggle to demonstrate AI-driven financial returns to stakeholders

• 13% of CIOs say their organizations have realized significant or transformational value from AI

• 11% of CFOs see clear returns from AI

• 4% of CIOs have plans in place to redesign roles for a human-agent workforce

• 58% of CIOs face increased pressure to deliver cost improvements from AI

These numbers highlight the fact that AI is rapidly shifting from experimental mode into a place where it has to show accountability for impact and ROI. Leaders are going to have to be able to articulate not just what AI can accomplish but how those benefits can justify the costs for both deployment and on-going operations.

"Volume is not value," said Alicia Mullery, Gartner analyst. "AI is creating a reckoning for organizations that lack fundamental disciplines, such as defining use-case value ahead of time and establishing criteria for when to walk away."

Gartner analyst David Furlonger further extrapolated, "The AI value problem will not be solved by simply adding more AI to existing ways of working."

In other words, if companies rely on AI to simply automate individual tasks then the economic upside is likely to remain limited and the pressure on executives will continue to increase. Instead, they're going to have to think bigger. Enterprises will need to do the harder work of redesigning workflows, reimagining decision-making processes, and rethinking organizational structures. 

That's a lot to ask and it takes investment and organizational buy-in, but that's what will be required to drive the kind of transformations that match the rhetoric and the level of investment we're seeing around AI. It also requires bringing AI down from the clouds and thinking of it like all of the other technology shifts and changes from the past. Even if it's a more powerful and capable technology than what's come before, it still needs to earn its stripes in the same way and live by the same economic laws.

"AI is an enabler of outcomes, not an outcome itself," said Gartner analyst Joseph Provenza

The bottom line is that AI spending and adoption continue to accelerate but business outcomes tied to AI are coming under intense pressure to justify the enormous investments that have been made in recent years. The bigger opportunity has to come from enterprises changing how they operate rather than simply making existing operations more efficient. That nets out to the fact that winning with AI as an organization depends far less on deploying the latest models and AI tools than it does on organizational discipline, workforce redesign, and rethinking how the organization creates value. Again, that's the kind of hard work that the best organizations will have to master when it comes to implementing AI.

TOGETHER WITH DATALAB

The New Standard In OCR Benchmarks

Figuring out which document parser to use can be tricky – existing leaderboards are straightforward but lack depth, often not taking into account where and why parsers succeed (or fail). But that’s all changing with OmniParseBench, the new benchmark for OCR created by Datalab. 

OmniParseBench doesn’t just assess the performance of document parsers, but also goes deeper to establish how every result was reached and the sources and methodology used to reach those results. With 16,288 pass/fail tests on 2,937 pages from 2,343 documents, in 92 languages (and with every test’s answer traced to its source), this open-code benchmark looks like the new gold standard for OCR benchmarks – but we’ll let you be the judge of that. 

Run OmniParseBench on your own documents →

GOVERNANCE

Independent audits catch what OpenAI missed

OpenAI may have just proven why third-party evaluations are crucial to frontier labs. 

OpenAI hosted two staff members from AI safety evaluation firm METR and Redwood Research, giving them company devices with access to OpenAI’s codebase and internal infrastructure, as well as the chance to interview technical staff, according to METR. The evaluators identified gaps and vulnerabilities, and OpenAI has since told METR that “all of the vulnerabilities identified that impact their production safeguards stack have been patched.”

The pilot took place for three weeks from July 21 to August 11, coincidentally the same day OpenAI disclosed the Hugging Face incident. METR discloses that it did not evaluate later changes, such as expanding monitoring to reinforcement learning training, but rather analyzed the monitoring as it existed at the start of the pilot. Some of the vulnerabilities identified included: 

• A way for agents to conceal hundred of actions from monitoring; 

• The monitoring system was only triggered when agents returned control to the user, meaning they could circumvent monitoring by continuing to take actions; 

• Due to the coverage gap, METR estimates that "on the order of 10% of internal Codex traffic was unmonitored from May to July." It says OpenAI had also independently identified the issue shortly before the exercise began. 

OpenAI has since said that the findings helped "strengthen safeguards" and will help shape future collaborations with METR and other third party evaluators. METR recently published a proposal for third-party teams to assess control and monitoring of internal agents, comprising five core subquestions. For the OpenAI pilot, the evaluators focused on two:

What fraction of agent traffic was monitored, and could agents circumvent this monitoring?

This highlights the exact reason that there is a movement pushing for third-party evaluations: they can find blind spots initially missed. Even leaving aside the incentive that companies may have to handle evaluations internally and leave out safety pitfalls, assuming all positive intent, at the rate and speed the companies are moving, it is possible that they let things slip through the cracks. Additionally, when you are too close to a subject, it's easier to miss red flags. It's the same reason that journalists have copy editors —  not because we don't understand grammar or know how to write, but because we have to move quickly and can become too attached to stories to see gaps in them. 

TOGETHER WITH GRANOLA

The slide you forgot to screenshot

Someone shares the roadmap, walks through the numbers, then stops sharing. By the time you think to take a screenshot, it's gone.

Granola, the AI notepad for people with back-to-back meetings, now has Snapshots. Capture what's shared on screen during a call and it's saved with your notes. Later, ask Granola what was decided and you get the answer with the slide it came from.

Download Granola and try it in your next meeting. 1 month off any paid plan with code THE100OFF.

PRODUCTS

Agent security is becoming a product layer

As AI agents become increasingly capable, the upsides and tradeoffs grow with them. 1Password is betting its platform can address some of the biggest risks.

On Thursday, the company launched 1Password for Agents, a platform that gives personal AI agents access to the passwords they need to complete tasks on users' behalf without ever seeing or storing the credential. This mitigates the risk of the agents using the password to take unauthorized actions, and offers peace of mind to users by limiting AI vendors' access to personal information. 

"For the last two decades, 1Password has helped people secure their credentials through every new wave of technology," John Torrey, Chief Business Officer at 1Password told The Deep View. "1Password for Agents is designed to reduce the risk by keeping credentials out of the agent’s access and requiring the user’s approval before it signs in."

The company said that, when the agent reaches a sign-in form, it uses a unique digital key to verify its identity to 1Password. Then the user gets to see which credential the agent needs to use and why, and can approve the request on their device. Finally, 1Password fills the credential into the sign-in form directly instead of handing it over to the agent. Once the website accepts it, it is removed from the sign-in form. 

Users can use 1Password with their AI agent of choice and initial integrations, including Meta's Muse, Anthropic’s Claude, OpenAI's Dots, and SpaceXSI Grok Bot, with more integrations to come. 

"1Password for Agents is built on a shared security model, where 1Password and our partners each have a clear role in protecting people," added Torrey. "Only partners who agree to uphold shared security principles and pass a security review can offer it."

This release builds on other AI authentication platforms the company has already released, such as 1Password for Claude, a browser integration that gives Claude access to stored credentials without actually exposing the credentials to the model. It also comes amid the recent boom in personal agents, with Meta, OpenAI and now Anthropic each unveiling their own always-on AI assistants.

AI agents are only as useful as the access you’re willing to give them. However, trusting them is becoming increasingly challenging, with agent security incidents piling up, ranging from large-scale cases like the OpenAI Hugging Face breach to individual impacts, such as Muse's individual user incidents. A core underlying issue with all of these cases is giving AI agents too much access. Solutions like this from 1Password can support a better middle ground, in which users aren't afraid of giving it access but at the same time can stay protected. My concern goes a step beyond: The fact that many people don't have the AI literacy to even know that they need a solution like this to begin with.

LINKS

• DeepSeek raises 100 billion yuan to compete with Anthropic, OpenAI

• China-linked hackers breached ConnectWise using AI penetration tool

• Meta restructured stock to shield Zuckerberg from California wealth tax

• 1Password surpasses $500 million ARR on enterprise AI agent demand

• Barclays cuts 5,000 jobs through the use of AI

• The AI data center boom has boosted gray paint sales

• Claude Duet: a new app where Claude works as a proactive agent

• Notion Agent: Major update that includes running tasks for 24+ hours, up from 5 minutes

• PHP2Shell: Perplexity Computer's skill checks PHP sites and GitHub deployment configs for exposure

• Gemini: now available in Google Maps on CarPlay

• Anthropic: Applied AI Demo Engineer

• Fanatics: Applied AI Engineer

• OpenAI: Applied AI Architect

• Obin AI: Agentic AI Engineer

GAMES

POLL RESULTS

Are you concerned about AI's impact on cybersecurity?

Yes (80%)
Somewhat (14%)
No (4%)
Other (2%)