The frontier moved closer. Local models are beginning to be close enough
I ran the same code review twice on the sample project for an upcoming blog post. Once with Qwen 3.8 27B running locally in Ollama at a 128k context window and once with Claude, hosted. Both used a code review agent so they were both provided the same instructions and tools.
Claude’s review was marginally better. Claude won by a modest margin rather than a wide one, and the basis is accuracy and sourcing rather than diligence. But Qwen found the single highest-value defect in the codebase, a bug in the relevance measurement harness that Claude missed entirely, and it did so on 17GB of weights running on hardware I own.
What this shows is worth paying attention to. Qwen’s model wasn’t specifically a coding model and for sure the agent could be tuned for improvement. Out of the box, Zero Shot Prompting if you will, local models have reached functional parity to frontier models.
The setup
The project is a sample ecommerce search implementation: a Python indexing and evaluation package, a Next.js frontend, notebooks, and docs. I gave both models the same instructions.
Four required sections:
| Section | |
|---|---|
| 1. | Bugs and correctness |
| 2. | Unfinished features |
| 3. | Consistency and conventions |
| 4. | Test coverage gaps |
When a defect was found, the task was to classify it into one of three severity levels:
| Level | Description |
|---|---|
| P0 | For anything breaking functionality, data, or security. |
| P1 | For incorrect behavior in a real path. |
| P2 | For maintainability. |
Summary Of Reviews
Here is the summary of issues detected between the two reviews.
| Qwen 3.8 27B (local) | Claude (hosted) | |
|---|---|---|
| Total findings | 17 | 24 |
| Rated P1 | 6 | 2 |
| P1 share | 35% | 8% |
| Findings in both reviews | 7 | 7 |
| Citation granularity | file + symbol | file + line |
| Confirmed false positives | 1 (rated P1) | 0 |
| Best unique finding | relevance harness bug | deep-pagination crash |
| Verdict | modest loss | modest win |
In total, thirty-four (34) unique findings across the two reviews and roughly a fifth of them in common. Neither review is a superset of the other. If you have been treating a single review pass as an audit, that overlap number is the most useful thing in this post.
Another way to say it, they both sucked.
Qwen found the best bug in the codebase
The highest-value finding came from the local model, and Claude missed it.
The project has an offline relevance evaluation harness that scores a baseline query against a tuned query using graded judgments. There are two code execution paths. The baseline path correctly extracted the query string from each judgment record before passing it to the query builder. The tuned path passed the entire record object instead, and the query builder interpolates its input directly into match and term clauses.
So the tuning arm of every relevance comparison was assembling a malformed query. The defect sits inside the measurement instrument, which means the numbers you would use to justify a relevance change are exactly the numbers that are wrong. For anyone doing relevance engineering, that is the worst place in the system for a bug to hide, and no passing test suite will surface it.
Qwen caught it by noticing the asymmetry between the two paths. That is reasoning about code, not pattern matching against a lint rule. A 17GB model on local hardware found the highest-value defect in this repository and the frontier model did not.
Why Qwen anchored on the TODO list instead of the code
The sharpest difference between the two reviews was sourcing, and it points somewhere more specific than “the frontier model is smarter.”
Qwen’s entire “unfinished features” section was derived from checkboxes in the project’s own deliverables document. It read the repository’s TODO list and reformatted it as review findings. Every item was real in the sense that the document said those things. No code was examined to produce any of them. This could be easily improved.
Claude checked that same document against the code and found it stale. It listed a feature as unimplemented that had already shipped and linked a PDF not present in the repository. Claude also caught that the schema documentation described the primary search field as one Elasticsearch type with n-gram subfields when the actual mapping used a different type with a completely different set of subfields, meaning anyone tuning relevance from the docs was working against a field list that does not exist.
That is the difference between summarizing a repository and auditing one. One treats documentation as ground truth. The other treats it as a claim.
I graded the reviews with Claude and it got that wrong too
I handed both reviews back to Claude and asked which was better.
It gave a clear verdict with reasoning I agreed with, then scored the two reviews 8.5 and 6.5 out of 10. I asked how it calculated those numbers. It had not. It formed a holistic judgment and attached decimals to it, which is a fluency behavior rather than an analytical one. To its credit, it said so immediately when challenged.
Then a second correction. Part of its case was that Claude had run the test suite and disclosed what it could not verify, while Qwen showed no evidence of running anything. That was wrong. Qwen did run the suite, just after writing its report. When I supplied that, the argument narrowed from “clear margin on diligence” to “modest edge on accuracy.”
So Claude’s opinion was strongly based on running the unit tests.
What this means for the local vs frontier decision
The benchmarks that are cited with the release of any new models aren’t necessarily grounded in anything from the field. They are academic and based widely on synthetic data. These conditions might not mean a lot to your use cases and if taken blindly may not produce the effect to your liking. Added to this is the marketing fever of a portion of the industry bent on selling their remedies.
With the recent releases of local open source models, these models are as effective in many cases as the frontier models. If you look at cloud hosting providers versus running it on local hardware:
| Model | Tokens In | Tokens Out |
|---|---|---|
| Qwen3.8-27b | .35 | 2.55 |
| Claude Sonnet 4.6 | 3 | 15 |
The price of good-enough is winning, and most tasks are good-enough tasks.
FAQ
Can Qwen 3.8 27B replace Claude for code review?
Yes. This task was a workspace review. Maybe you want to leave that to something that has a larger context window, but for everyday reviews / Pull Requests where the 128k context window is satisfactory, it provides similar results. Spend your saved tokens / costs on a second or third review.
Is a 128k context window enough to review a whole codebase?
A 128k window is not 128k of effective attention, and on a dense model the reasoning trace competes with source code for the same budget. In this test the local model responded to that pressure by deriving findings from the repository’s TODO list rather than from code. Chunked per-file review is the better pattern.
Why do local models produce false positives in code review?
When retrieval is not grounding the model, it reasons from priors, and priors produce fluent plausible claims rather than verified ones. The false positive here cited the right file and the right function and reached a conclusion the vendor documentation contradicts. Requiring an external citation for every high-severity finding is the cheapest defense.
Should you use a second model to grade the first model’s output?
Yes. The model I used to compare these two reviews fabricated a numeric score with no rubric behind it and made a factual error about what the local model had run. Model-grading relocates the verification problem rather than removing it.
Always be as specific as possible.
What hardware runs Qwen 3.8 27B?
The model is roughly 17GB, which puts it within reach of capable consumer hardware. There are wide ranges of differences in speed depending on what hardware you have. Expect 30 to 144 tokens per second (TPS) which is equal to frontier models.