Salesforce study shows GPT-5 fails over half of enterprise AI tasks

Salesforce AI Research has released MCP-Universe, an open-source benchmark revealing that even advanced AI models like OpenAI’s GPT-5 fail more than half of real-world enterprise orchestration tasks. The benchmark tests how large language models interact with Model Context Protocol (MCP) servers—a system that lets AI models connect with external tools and data sources—across six enterprise domains, exposing significant limitations in current AI capabilities for business applications.

What you should know: MCP-Universe evaluates AI models on practical enterprise tasks rather than isolated performance metrics, providing a more realistic assessment of AI readiness for business deployment.

• The benchmark tests models across six core enterprise domains: location navigation, repository management, financial analysis, 3D design, browser automation, and web search.
• Salesforce accessed 11 MCP servers for a total of 231 tasks, designed to mimic real enterprise workflows.
• Unlike synthetic benchmarks, MCP-Universe uses execution-based evaluation with real-time data and actual enterprise tools.

The results: Even top-tier AI models struggled significantly with enterprise-grade tasks, highlighting major gaps in current AI capabilities.

• GPT-5 achieved the highest success rate overall, particularly excelling in financial analysis tasks.
• Grok-4 from xAI ranked second and performed best in browser automation, while Claude-4.0 Sonnet from Anthropic rounded out the top three.
• Among open-source models, GLM-4.5 from Zhipu AI demonstrated the strongest performance.
• All models tested had at least 120 billion parameters, representing the most advanced AI systems available.

Where models fail: Two critical limitations emerged as primary obstacles to enterprise AI adoption.

• Long context challenges: Models lose track of information and struggle with consistent reasoning when handling complex, lengthy inputs.
• Unknown tool challenges: AI systems cannot seamlessly adapt to unfamiliar tools the way humans naturally do.
• Performance dropped significantly in location navigation, browser automation, and financial analysis when dealing with extended contexts.

What they’re saying: Salesforce researchers emphasize that current frontier models aren’t ready for reliable enterprise deployment.

• “Two of the biggest are: Long context challenges, models can lose track of information or struggle to reason consistently when handling very long or complex inputs,” said Junnan Li, director of AI research at Salesforce.
• “Models often aren’t able to seamlessly use unfamiliar tools or systems in the way humans can adapt on the fly. This is why it’s crucial not to take a DIY approach with a single model to power agents alone.”
• “These findings highlight that current frontier LLMs still fall short in reliably executing tasks across diverse real-world MCP tasks,” the research paper concluded.

How it works: MCP-Universe employs execution-based evaluation rather than the common LLM-as-a-judge approach, using real enterprise tools and data.

• Location navigation tests geographic reasoning through Google Maps MCP server integration.
• Repository management evaluates codebase operations via GitHub MCP, including repo search, issue tracking, and code editing.
• Financial analysis connects to Yahoo Finance MCP server for quantitative reasoning and market decision-making.
• 3D design assessment uses Blender MCP for computer-aided design tool evaluation.
• Browser automation testing occurs through Playwright’s MCP integration.
• Web search domain employs Google Search MCP and Fetch MCP for open-domain information seeking tasks.

Why this matters: The benchmark provides enterprises with crucial insights into AI limitations that could prevent costly deployment failures.

• Li hopes companies will use MCP-Universe to understand where AI agents fail on tasks, enabling better framework improvements and MCP tool implementation.
• The research joins other MCP-based benchmarks like MCP-Radar and MCPWorld, building on Salesforce’s earlier MCPEvals release from July.
• Unlike MCPEvals, which uses synthetic tasks, MCP-Universe focuses on real-world scenarios with actual enterprise data and tools.