Seoul Labs

Grok 4.6 posts a 65.9 percent on DeepSWE v1.1 and a 69.9 percent on CursorBench v3.2, scores xAI is using to put it in the same sentence as GPT 5.6 Sol. Those benchmarks grade tasks in isolation, so they never caught what actually happened at turn 31 of a longer session I ran: the model regenerated a config loader and brought back a key mismatch bug it had already fixed at turn 18. No benchmark grades cross file consistency across a single continuous agent session, so the number on the leaderboard and the number of turns before the model starts contradicting its own earlier fixes measure two completely different things. So here's the question I actually want to answer: if you're deciding whether to trust Grok 4.6 with a real, unattended coding session, which number should you believe?

CursorBench v3.2: 69.9 percent. On paper that puts Grok 4.6 in the same conversation as GPT 5.6 Sol on coding tasks, which is the comparison xAI clearly wants people making.

DeepSWE v1.1: 65.9 percent. Respectable. Also, in my experience this week, not the number that predicts whether the model will finish a four file refactor without losing track of an import.

I spent some time running Grok 4.6 against actual repositories instead of benchmark harnesses, and the gap between the composite scores and the lived experience of using this thing as a coding agent is the real story here. The benchmarks aren't fake. They're measuring something narrower than what most of us need when we open a terminal and expect an agent to carry a task start to finish without supervision.

What The Benchmark Numbers Measure

Most people reading a benchmark table assume the number describes general competence: how good is this model at coding, full stop. That's not what GDPVal AA v2 or CursorBench measure, and the gap matters because that's exactly where marketing slips past the facts.

GDPVal AA v2, where Grok 4.6 posts 1753, is a composite scoring system built around graded professional deliverables, things like drafting a contract clause or producing a data analysis summary that a human evaluator then scores against a rubric. Reasonable proxy for knowledge work. Weak proxy for long horizon agentic coding, which is what most developers actually care about when they read that xAI is matching GPT 5.6 Sol.

APEX Agents, where Grok 4.6 scores 57.5 percent, sits closer to what people mean by agentic reasoning: multi step scenarios with tool use and some branching logic. But 57.5 percent also means the model fails nearly four out of ten scenarios in a benchmark specifically designed to be achievable. Scale that failure rate into a real codebase with ambiguous requirements and legacy test coverage, and it doesn't shrink. It compounds.

Here's a small, concrete example of the gap. I asked Grok 4.6 to refactor a FastAPI dependency injection pattern across six files, a task well within what DeepSWE style benchmarks test:

before
Anatomy of a Session Failure: FastAPI Refactor Across 6 Files

  
Converting sync DB sessions to async across 6 route modules
  

    

      Files 1 to 3
      Converted correctly
    
    
↓
    

      File 4: orders.py
      Reverted to sync pattern
    
    
↓
    

      Import statement
      Left as async version
    
    
↓
    

      Runtime error: TypeError, AsyncSession can't be used in 'await' expression
    
    
↓
    

      Cost: 20 minutes of debugging, not caught by any benchmark
    
  
Source: Author's hands-on test of Grok 4.6 on a FastAPI dependency injection refactor

def get_db_session():
    session = SessionLocal()
    try:
        yield session
    finally:
        session.close()

requested change: convert to async context manager
Benchmarks vs Real World: What Each Score Actually Captures

  
Scope, failure mode, and what gets graded
  

    

      
Benchmark
      
Score
      
Evaluation Scope
      
Catches Cross File Bugs?
    
    

      
CursorBench v3.2
      
69.9%
      
Isolated coding tasks
      
No
    
    

      
DeepSWE v1.1
      
65.9%
      
Scoped tasks, graded individually
      
No
    
    

      
GDPVal AA v2
      
1753
      
Single professional deliverables
      
No
    
    

      
APEX Agents
      
57.5%
      
Multi step scenarios, tool use
      
Partially
    
    

      
LoCoBench Agent (CFC)
      
Not disclosed
      
Naming and architecture patterns
      
Yes, partially
    
  
Source: Author's analysis of xAI benchmark design and agentic coding test results



used across 6 route modules

The model nailed the first three files. On the fourth, it silently reverted one route back to the synchronous pattern while leaving the import statement updated to the async version, producing this at runtime:

TypeError: object AsyncSession can't be used in 'await' expression
  File "app/routes/orders.py", line 42, in get_orders
    session = await get_db_session()

That's not a benchmark failure. DeepSWE v1.1 would likely never surface this specific inconsistency, because its tasks are scoped and graded individually, not evaluated as one continuous six file session with implicit cross file consistency requirements. The 65.9 percent score is real. It just doesn't describe the failure mode that cost me twenty minutes of debugging. One caveat worth naming: benchmarks like LoCoBench-Agent do include a Cross-File Consistency (CFC) metric that grades adherence to naming conventions and architectural patterns across files within an agent session, so this gap in coverage isn't universal, just not present in the two numbers xAI is leading with.

Grok 4.6 Earns The Marketing Claim, In Places

The six file refactor shows where the benchmark score and the real session diverge. But writing this as pure skepticism would be dishonest, because there are places where Grok 4.6 genuinely performs the way xAI says it does.

Multi part task handling is noticeably better than prior Grok releases. Ask it to plan a migration, generate the migration script, write the rollback procedure, and draft a short incident runbook, and it tracks state across those four sub tasks more reliably than Grok 4 or Grok 4.1 did. This matches the improved composite scores on agentic reasoning benchmarks, and it's the kind of improvement that actually shows up in daily use rather than only on a leaderboard.

Knowledge work tasks, the kind GDPVal AA v2 is built to measure, also hold up. I had it draft a vendor comparison memo with specific technical tradeoffs across three cloud providers, and the output needed real but minor editing rather than a rewrite. Genuine step forward from what I was getting out of Grok 4.1 on similar tasks six months ago.

Where the APEX Agents score of 57.5 percent actually tracks with reality is tool use chaining in constrained environments. I set up a small test where the model had access to a file search tool, a shell execution tool, and a git diff tool, and asked it to find the source of a memory leak in a long running script.

tool call sequence observed


1. grep for list/dict growth patterns across modules


2. shell: run script with tracemalloc enabled


3. parse tracemalloc output, identify top allocator


4. git diff against last known-good commit near allocator file

That sequencing was coherent, and the model moved from step to step without needing a nudge. This is the pattern benchmark designers are trying to capture, and here it tracks close to the published rate on comparable multi tool tasks, based on the handful of similar scenarios I ran myself. Small sample, so treat the comparison as directional, not precise.

The Agent Loop Problem No Benchmark Catches

The tool chaining example above worked because the task was short: four steps, one clear goal, no accumulated history to lose track of. What benchmark suites systematically underweight is what happens once a session runs much longer than that, and this is where Grok 4.6, like most models in this generation, starts to look worse than its scores suggest.

CursorBench v3.2 tasks, and agentic coding benchmarks generally, are bounded. There's a clear start state, a clear end state, and a grading rubric applied once. Real agent sessions don't work that way. They run for forty, sixty, a hundred tool calls, and the model is reasoning over an accumulating context window that includes its own prior mistakes, half finished edits, and tool outputs that may themselves be stale.

I ran a longer session: build a small CLI tool with argument parsing, a config file loader, logging, and three subcommands, iterating based on test failures. Early turns were sharp. Later in the session, Grok 4.6 reintroduced a bug it had already fixed earlier on, specifically a config key mismatch between the YAML loader and the argparse defaults.

config.yaml
logging:
  level: DEBUG
  output_path: /var/log/clitool.log

clitool/config.py, reintroduced after turn 31
DEFAULT_CONFIG = {
    "log_level": "INFO",   # key mismatch: should be "level"
    "log_path": "stdout",  # key mismatch: should be "output_path"
}

This had already been fixed earlier in the session, at turn 18. The model regenerated the config loader from a slightly earlier mental model of the schema, probably because the correction had scrolled far enough back in context that it carried less weight during generation. No benchmark I know of, including DeepSWE v1.1 or CursorBench v3.2, is structured to catch this specific regression pattern, because both are built around single pass or short horizon grading. Is this a Grok specific problem? Not really. GPT 5.6 Sol sessions show the same thing, just with a different failure signature. The exact gap between the original fix and its silent reversal varied session to session, but the pattern, regression after a stretch of turns, held up consistently.

Where Grok 4.6 Beats Expectations

The regression problem above is a real limit, not a reason to write the model off. Error message interpretation and raw latency are two areas where Grok 4.6 exceeded what I expected going in, and they're worth separating clearly from the long session failures, since they don't share the same cause.

Error message interpretation is sharper than I anticipated. Feed it a raw Python traceback with no additional context, and it tends to correctly identify root cause rather than surface symptom more often than Grok 4.1 did. Here's a representative case, a stack trace from a Django app with a subtle circular import:

ImportError: cannot import name 'UserProfile' from partially initialized module 'accounts.models'
(most likely due to a circular import)
  File "accounts/serializers.py", line 4, in 
    from accounts.models import UserProfile
  File "accounts/models.py", line 9, in 
    from accounts.serializers import ProfileSerializer

Most models I've tested, including several recent Grok versions, suggest breaking the circular import by moving the import inside a function, which works but is a patch rather than a fix. Grok 4.6 correctly identified that the serializer shouldn't be importing from the models module at the top level at all, and proposed restructuring with a lazy reference pattern using Django's string based foreign key syntax. That's a meaningfully better diagnosis, and composite scores like CursorBench do partially capture it, since diagnostic accuracy is a component of several underlying coding tasks.

Latency also surprised me, in the good direction. Agentic sessions with several chained tool calls returned noticeably faster than equivalent sessions with GPT 5.6 Sol in my own informal timing, though I want to flag clearly that I haven't run a controlled benchmark here. It's an observed pattern across a handful of sessions, not a verified claim, and xAI's own published numbers don't break out latency as a separate axis the way they do accuracy. Anyone doing cost sensitive, high volume agent work should watch this closely, since tool call round trip time adds up fast across a long session.

What To Check Before Switching Workloads

Put the diagnosis strength and the latency gains next to the turn 31 regression, and the usual binary framing collapses. One version says Grok 4.6 is now the best coding agent available, matching the AA Intelligence Index score of 61 against whatever the current GPT or Claude equivalent posts. The other says it's marketing noise and nothing has changed. Neither survives contact with a real migration decision.

What actually matters for a team deciding whether to route agentic coding workloads to Grok 4.6 is session length tolerance. Short, well scoped tasks, single function generation, isolated bug fixes, scoped code review, map closely to what CursorBench v3.2 and DeepSWE v1.1 measure, and the 69.9 percent and 65.9 percent scores are a fair predictor of what you'll see. Long running autonomous sessions, the kind where an agent is expected to self correct across sixty or more turns without a human checking in, are exactly where the benchmark coverage is thinnest. That's where I'd budget for manual intervention regardless of what the composite score says.

A short checklist worth running before any team commits infrastructure budget to this model for agentic workloads:

• Test sessions at your actual expected turn count, not a shortened demo version.


• Check whether regressions reappear after roughly twenty five to thirty tool calls.


• Compare diagnostic quality against your own real stack traces.


• Measure actual round trip latency under your own tool chain.


• Confirm config and schema consistency across multi file edits manually, since the model won't flag its own drift.

None of these show up cleanly in the AA Intelligence Index or GDPVal AA v2, and that's the point. Those composites are built to be comparable across models and reproducible across runs, which means they're necessarily built around bounded, gradable tasks. The failure modes that actually cost engineering time, silent regression, config drift across files, context weighted forgetting, live outside that boundary almost by construction.

Grok 4.6 is a real improvement over Grok 4.1, and the benchmark gains reflect something genuine rather than inflated reporting, based on everything I was able to reproduce independently this week. So which number should you believe, the 69.9 on CursorBench or the bug that came back at turn 31? Both, for different jobs. Trust the benchmark score for short, bounded tasks, and budget a human checkpoint around turn 25 to 30 for anything longer, because the gap between a 61 on a composite agentic index and a model that can run unattended for an afternoon on a messy real repository hasn't closed, for Grok or for any comparable model currently shipping. No lab in this generation of releases, xAI included, has published a benchmark that measures the turn 31 problem directly. Until one exists, that checkpoint is the only real safeguard.