Code Review Tool Comparison

4 files, 8 real bugs + 2 non-bugs (traps), ado122/analyzer

Real Bugs (8)

1. Rate limiter >= → > (off-by-one)
2. CSRF HMAC-SHA256 → base64 (security downgrade)
3. API timeout 15min → 30s (breaks analysis jobs)
4. SQL injection ×2 in new exportService.js
5. CSV injection in new exportService.js
6. Off-by-one pagination in new exportService.js
7. Path traversal in new exportService.js
8. Promise error swallowing

Traps (not real bugs): Harmless log simplification (all tools correctly skipped) · Loose equality == on two numbers (behaves identically to === — Copilot & CodeRabbit flagged as FP)

Results

Scores

Key Observations

• Copilot 8/8 — only tool to find the path traversal. Noisy though (~5 false positives).
• Recall & CodeRabbit tied at 7/8 — both missed only path traversal. Recall had 0 false positives + a bonus CSRF timing attack finding.
• Testigo Recall was the ONLY tool that caught the API timeout as a KB-grounded regression — Greptile called it "reasonable" because it lacks codebase context.

5 files, KB-heavy planted bugs — all are documented value/behavior changes, ado122/analyzer

Planted Bugs

1. JWT token expiry 7d → 30min (breaks stateless auth)
2. Virality score weights swapped: velocity 40%→10%
3. Strategy cache duration 6h → 5s (excessive API calls)
4. Free tier channels 1 → unlimited (breaks business model)
5. Encryption AES-256-GCM → AES-128-ECB (security regression)

Results

Scores

Real PR from twentyhq/twenty (open-source CRM, 31K KB facts) — billing credits display fix, 3 files, +82/−28 lines. Not planted bugs — real issues found during review.

Issues Identified

1. Behavioral change: upTo raw value → toDisplayCredits() conversion (undocumented)
2. Cross-module side effect: shouldUpdateAtSubscriptionPeriodEnd compares in internal units — potential 1000× mismatch

Results

Scores

Key Observations

• All 3 competitors approved this PR with zero real findings — Copilot, CodeRabbit, and Greptile all concluded the change was safe to ship.
• Testigo Recall found both issues by cross-referencing the diff against KB facts about billing-subscription-update.service.ts — a file not in the diff.
• This test highlights the core value of KB-grounded review — cross-module side effects are invisible to tools that only analyze the diff in isolation.

10 files, 10 planted bugs (3 KB-detectable + 7 pure code), 1,329 lines added, ado122/analyzer

Planted Bugs

1. Missing CSRF on competitor routes KB
2. optionalAuth on DELETE (bypasses auth) KB
3. Conditional rendering (loses tab state) KB
4. NoSQL injection / ReDoS via unescaped RegExp
5. IDOR — no ownership check on getById
6. XSS via dangerouslySetInnerHTML
7. Off-by-one pagination
8. Missing await on async delete
9. Mass assignment / prototype pollution
10. String vs number comparison

Results

Scores

Key Observations

• Recall & CodeRabbit tie at 9/10 — both catch NoSQL injection and conditional rendering that Copilot and Greptile missed.
• Copilot uniquely found string comparison (#10) — cross-file reasoning connecting MongoDB String schema to numeric comparison in UI.
• Recall's KB layer found 6 bonus issues — missing tier gates, wrong API utility, wrong tier limit values, missing ownership checks. Things only a knowledge-base-aware tool can catch.
• Recall's Haiku semantic dedup merged 22 raw findings into 14 unique with zero false kills.

11 files, 10 planted bugs (all different from Test 4), 1,485 lines added, ado122/analyzer

Planted Bugs

1. Missing CSRF on report routes KB
2. Path traversal in report file download
3. SSRF via webhook delivery URL
4. Insecure randomness (Math.random for share tokens)
5. IDOR — no ownership check on getById/delete/generate/download
6. Prototype pollution in deep merge
7. Missing await on async delete
8. ReDoS regex in template name validation
9. Info disclosure (stack trace in error response)
10. Off-by-one date range (29 days instead of 30)

Results

Scores

Key Observations

• ReDoS universally missed — all 4 tools failed to identify catastrophic backtracking in /^([a-zA-Z0-9]+[\s_-]*)+$/. ReDoS remains a blind spot for all AI code reviewers.
• Copilot is the only tool to catch the off-by-one date — "last30days" subtracting 29 instead of 30. Subtle logic bug that other tools overlooked.
• Recall & CodeRabbit both caught prototype pollution while Copilot and Greptile missed it.
• Recall's KB layer detected missing CSRF by comparing against documented CSRF patterns, flagged wrong Claude model ID, wrong env var name, and webhooks needing Enterprise tier gate.
• 82 total inline comments — Copilot (33), CodeRabbit (20), Recall (19), Greptile (10). Copilot is the noisiest; Greptile the most concise.
• The 2 misses (ReDoS + off-by-one) are the lowest-severity bugs — all critical security issues were caught by Recall.