Comparison of Success Rates and Costs for 8 Harnesses in DeepSeek V4 Flash Testing

AI infrastructure company Composio integrated DeepSeek V4 Flash with 8 Harnesses, testing them using the same batch of 30 Agent tasks across applications such as Gmail, GitHub, Slack, Calendar, and Notion. The success rates for the 8 Harnesses were as follows: Pi Agent 20/30, Oh My Pi 17/30, Claude Code 16/30, Codex 16/30, Deep Agents 16/30, Hermes Agent 15/30, Prime Agent 15/24, and OpenCode 14/30. After switching Harnesses, the number of successful tasks for DeepSeek V4 Flash increased from 14 to 20. In terms of cost, Pi Agent had a cost of approximately $0.028 per successful task, while Claude Code was $0.195, showing a nearly 7-fold difference. Regarding speed, the median time for Claude Code was 122.7 seconds, with Oh My Pi being the slowest at 272.4 seconds. The testing configuration for Pi differed from the standardized conditions, using high rather than max inference intensity, and 24 out of the 30 tasks utilized the official DeepSeek API. Previously, Composio conducted another round of Harness comparisons using Kimi K3, where Oh My Pi ranked first with 22/25, the original Pi Agent scored 18/25, and after switching to DeepSeek V4 Flash, the original Pi took the lead.

This content is provided for general informational purposes only and doesn't constitute financial, investment, legal, or tax advice. Any events, rewards, online promotions, or related information mentioned herein should not be considered a recommendation, solicitation, or invitation to purchase, sell, trade, or otherwise deal in any crypto assets. Crypto assets are highly volatile and may result in loss. The availability of WEEX services, products, and related events may vary by region. You are responsible for ensuring that your participation is in accordance with applicable local laws and regulations.