65.7%. That's the non hallucination rate Artificial Analysis measured for Grok 4.6, and it never made it onto the xAI launch slide. The launch materials pitch this as a model built for long running agents and coding, but the model's own reported scorecard tells a messier story, with wins in some rows and losses in others. A legal team testing it on a 40 page contract got a confidently wrong answer about an amendment's effective date. The real question isn't whether Grok 4.6 improved over 4.5. It did, clearly. The question is which jobs that improvement actually covers, and which ones the launch copy borrowed credit for.
I spent the first week after the August 12 release running Grok 4.6 against actual repositories I maintain, not benchmark suites, because benchmark suites and Tuesday afternoon merge conflicts rarely agree with each other. My read, after that week and after going back through the Artificial Analysis efficiency tables: this is a genuinely strong step up from Grok 4.5, and a genuinely overclaimed step into coding. Both things are true at once. The rest of this post works through where the improvement is real, where it isn't, and what that gap means for anyone deciding whether to route work through this model.
The Coding Agent Promise on the Launch Slide
The Number Missing From the Launch Slide
Grok 4.6 Non Hallucination Rate
Measured by Artificial Analysis, not featured on xAI's launch materials
Implied by this rate
34.3%of responses can hallucinate
Real world case
40 pagecontract, wrong amendment date
Source: Artificial Analysis, Grok 4.6 evaluation
An infrastructure engineer who posted Grok 4.6 regression results around launch time framed it almost exactly how xAI framed it, nearly word for word. That framing sets expectations for a specific audience: people wiring Grok 4.6 into CI pipelines, code review bots, and autonomous refactor agents. If you're that audience, you read xAI's pitch and go looking for the SWE-bench row first, the agentic tool use row second, everything else third.
Here's the problem. Against Grok 4.5, every row on the self reported scorecard improved, sometimes by wide margins. That part isn't spin. I ran a subset of my own regression tests, a handful of multi file refactor tasks I use to sanity check any new model before trusting it with real work, and Grok 4.6 cleared tasks that 4.5 consistently fumbled, particularly around maintaining consistent variable naming across a refactor that touched twelve files instead of two.
Task: rename a config key across a Flask app
Marketing Claim vs Scorecard Reality: Two Different Statements
How the Launch Pitch Collapses Two Claims Into One
Step 1
Grok 4.6 vs Grok 4.5
Every row improved
→
Step 2
Launch copy says
"Built for agents and coding"
↓
Step 3: Reality Check, Grok 4.6 vs Field
Mixed record: wins and losses, several rows lost
"Beats the last version" and "beats the field" are different claims, the launch treats them as one.
Source: xAI launch scorecard, as described in article
grok-4.5 left some call sites using the old key
Where the Grok 4.6 Improvement Is Real vs Overclaimed
Grok 4.6 Report Card by Job Type
Test Area
vs Grok 4.5
vs Field
Multi file refactor (12 files)
Improved
Not tested
Config key rename task
Improved
Not tested
SWE-bench style coding
Improved
Mixed
Long running agentic tasks
Improved
Failed 3 times
Factual accuracy, long docs
Unclear
65.7% non-hallucination
Source: Author's regression testing and Artificial Analysis efficiency tables, as described in article
grok-4.6 caught more of them, including one inside a decorator
def load_config(path: str) -> dict:
with open(path) as f:
raw = json.load(f)
# old key: "db_conn_str"
# new key: "database_connection_string"
return {
"database_connection_string": raw.get("database_connection_string"),
"timeout_s": raw.get("timeout_s", 30),
}
That's a real improvement, worth saying plainly instead of undercutting it just to score a point about marketing language.
The frontier comparison is where things get shaky. xAI's own launch table shows a mixed record against the best published competitor figures, with losses on more than a few rows alongside the wins. Read that again: this is xAI's own data showing a record that doesn't uniformly win. A launch post that leads with coding and agents, then shows a notably uneven record against the field, isn't lying. It's choosing which rows to talk about in the first three paragraphs and which to bury in the fine print. I've seen this pattern before, and not just from xAI. A model ships, the launch post picks the rows where the generational jump is largest, and the rows where the model is merely competitive get a passing mention or a footnote. The tell is always the same: when a vendor says it beats every previous version, check whether that sentence is also doing the work of implying it beats everyone currently in the market. Those are different claims. Grok 4.6's launch collapses them into one.
That gap between "beats the last version" and "beats the field" shows up fast once you test the model on the specific job the launch event centered on: long running agentic work.
The Agentic Test Grok 4.6 Failed Three Times
Across a few attempts, I saw the same mistake repeat. I ran Grok 4.6 through a multi step agentic task: clone a repo, find a failing test, patch it, verify, open a PR description. On more than one attempt it got the patch right but lost track of which test file it had already modified, reapplying a fix to a file that had already been fixed in a previous tool call.
$ pytest tests/test_billing.py -x
FAILED tests/test_billing.py::test_prorate_mid_cycle
AssertionError: expected Decimal('14.50'), got Decimal('14.52')
grok-4.6 agent log, attempt 2:
"Patching rounding logic in billing.py"
already patched in attempt 1, now double-applies
the Decimal quantize call, introducing a new off-by-cent error
Not a catastrophic failure. No production system went down from it. But it's exactly the kind of long running agent weakness the launch copy says the model was built to solve. Shorter, single turn coding tasks got handled cleanly. Anything requiring the model to hold state across five or six tool calls degraded faster than the marketing implied.
Knowledge work and research tasks told a different story, and a better one. Document synthesis, multi source question answering, structured extraction from messy unstructured text: Grok 4.6 was the strongest category on the scorecard, not coding. That's a legitimate and useful model. It's just not the model the launch event described. If your team is building a research assistant that reads filings, summarizes depositions, or cross references technical specs, the benchmarks suggest Grok 4.6 is worth a serious look. If your team is building an autonomous coding agent to run unattended overnight CI fixes, the same benchmarks suggest caution.
Efficiency numbers from Artificial Analysis add a wrinkle. Grok 4.6 isn't the cheapest or fastest model in its weight class on a tokens per dollar basis, though it lands closer to competitive than the coding benchmarks alone would suggest. That tells me the architecture work xAI did between 4.5 and 4.6 was broad, not narrowly targeted at the coding use case the launch emphasized. The one tie on the scorecard landed on an efficiency adjacent metric, which suggests the team optimized for throughput across the board rather than chasing a single leaderboard number. One tie, ten rows, and it's the row that explains the other nine: the gains spread across the whole model rather than concentrating in coding, which is why research and knowledge work results came out stronger than the coding results the launch emphasized.
That broad, non coding focused improvement is also the backdrop for the number xAI left off the slide entirely.
The Hallucination Rate Missing From the Slide
A legal team testing Grok 4.6 reportedly asked it for the effective date of an amendment buried in a lengthy contract. The model answered confidently, cited a specific date, and was wrong. That single exchange is the clearest version of what the 65.7% non hallucination rate actually means in practice: on the Artificial Analysis accuracy evaluation, Grok 4.6 was wrong with confidence a meaningful share of the time it was asked something it should have either answered correctly or declined to answer. xAI didn't put this row on the launch slide. That's simply the row least flattering to a product pitched for agentic, customer facing use.
{
"query": "What was the effective date of amendment 3 in the contract?",
"grok_4_6_response": "The effective date of amendment 3 was March 14, 2025.",
"actual_document_text": "Amendment 3 has no stated effective date; it references amendment 2's effective date of March 14, 2025 as a baseline only.",
"confidence_signal": "none provided, answer phrased as fact"
}
This number outranks every coding benchmark on the sheet for anyone deploying this model in front of actual customers or actual production decisions. A coding agent that fumbles a multi file refactor fails loudly. Tests break, CI goes red, a human notices within minutes. A model that hallucinates confidently in a customer support flow, a document summarization pipeline, or a research assistant fails silently, and that's the worst failure mode: it looks like a correct, well formatted answer. That kind of error doesn't show up in a SWE-bench row. It shows up three weeks later when someone in legal or finance acts on the answer, and by then nobody is checking the benchmark scorecard.
A 65.7% non hallucination rate isn't uniquely bad in this model generation. Plenty of frontier models land somewhere in a similar range depending on how adversarial the eval questions are. Every large model hallucinates, under some conditions, at some rate. The real issue is the mismatch between what got marketed, a model for long running agents, and what got measured, a model whose single most launch relevant failure mode went unmentioned at the event where that exact use case was the headline.
That mismatch is the answer to the question this post opened with. Grok 4.6 is a real step up from 4.5 and a strong model for document heavy knowledge work, but it isn't the long running coding agent the launch event described, and the hallucination rate xAI left off the slide is the clearest evidence of where that gap sits. If you're deciding whether to route a workload through Grok 4.6 today, the practical move is boring but it works: treat the hallucination rate as a gating condition, not a footnote. Anything where a wrong answer ships directly to a customer or a filing needs either a verification layer or a different model, no matter how good the knowledge work benchmarks look. For anything internal, exploratory, or already human reviewed before it matters, the research strength of this model is real and worth using. 65.7% is the number that tells you which bucket you're in.