GitLab Orbit goes GA, and Compare the Market's reviewer test found RAG worse than the diff alone
Compare the Market ran its internal AI merge-request reviewer four ways across 79 annotated merge requests. A code graph lifted inline-comment coverage to 0.696, vector retrieval dropped it to 0.577, below the 0.658 the reviewer got from the diff alone. The graph also cost about four times as much per review.
Source: GitLab Orbit is GA: Higher agent accuracy, lower token cost
GitLab made Orbit generally available on October 29 for Premium and Ultimate customers on GitLab.com, with Self-Managed and Dedicated to follow. Orbit is a continuously updated graph of a project's code, merge requests, pipelines, vulnerabilities, and owners that agents query over MCP instead of crawling files. GitLab's headline numbers are from its own testing: up to 45x fewer retries and 4.5x fewer tokens on the same tasks.
The more useful evidence is the customer write-up behind the launch. Compare the Market published a detailed comparison of four ways to give its internal merge-request reviewer context: the diff alone, the Orbit code graph, vector retrieval (RAG), and both together. The team scored each against 79 real merge requests annotated with the summary points, issues, and line-anchored comments an expert reviewer should produce, using an LLM judge to match meaning rather than wording.
- Inline-comment coverage: Orbit 0.696, diff alone 0.658, RAG 0.577.
- Issue coverage: close across the board, Orbit 0.929 and RAG 0.926.
- Issue precision: the diff alone was highest at 0.438, Orbit lowest at 0.411. The team reads that as Orbit flagging more, some of it valid but missing from the reference set.
- Cost and time: the diff alone took about 44 seconds and $0.58 per review at illustrative GPT-4 prices; Orbit took about 81 seconds and $2.37. Combining Orbit with RAG was slowest and no better.
RAG lost to the diff alone on nearly every metric. The team's explanation is that similarity search returns code that looks related but isn't, such as process_user_data when the change touched process_data, and that noise pulls the reviewer off course.
Why it matters for code review
For a reviewer that runs on every merge request, this is a concrete case where more context made reviews worse, and where the context that helped cost four times as much. The team's own advice is to measure before adding context and to keep the diff-only reviewer for simple changes. The caveats are theirs too: 79 merge requests, a reference set that can't credit valid findings it didn't anticipate, and a judge model deciding what counts as a match. The next step they name is replacing that fixed set with ratings from the engineers who received the reviews.