Kimi K3 drew plenty of attention on X and Hacker News after its release. We build agent teams for software engineering at First Tree, and official benchmarks only tell us so much. We wanted to see how the model handled the kind of work that reaches a real production repository.
So we gave Kimi K3, GPT 5.6 Sol, and Claude Fable 5 the same set of software engineering tasks.
We selected 31 First Tree issues across frontend, backend, security, performance, and architecture. The issues ranged from easy to extremely hard. Each model had to solve the issue and submit code that could plausibly ship.
Kimi K3 never pulled ahead. GPT 5.6 Sol did, by a clear margin. It handled routine software engineering work better than we expected and beat Claude Fable 5, the model we had expected to lead.
What We Found
Fable Peeked at Another Model's Work
We told each model to work independently. GPT 5.6 Sol and Kimi K3 finished first because they ran faster. We expected Fable to follow the instruction without trouble.
Fable ran into usage limit resets and slow inference, so it fell behind. During the long session, it also lost track of the original constraint and referred to another model's implementation.
We caught the shortcut in the execution log and reran the task with a stricter prompt.
GPT 5.6 Sol Calls a Lot of Tools
The session logs showed a stark difference in tool use.
GPT 5.6 Sol made 3.5 times as many tool calls as Claude and 2.5 times as many as Kimi. All three models completed a similar number of rounds, but GPT packed far more tool calls into each one.
GPT 5.6 Sol averaged 61 tool calls per round. Fable averaged 17.4. Kimi K3 averaged about 24.
Multiple Agents Produced Better Code
For an extra hard task, we ran GPT 5.6 Sol Ultra once with a single agent and once with multiple agents.
First Tree gave the agents separate roles and a way to challenge each other's work. That setup improved the final submission.
GPT 5.6 Sol Ultra's built in multiple agent mode still coordinates the work from one main perspective, so the gain was modest. Giving agents different roles produced a stronger result, though it also consumed more tokens.
The Result
Kimi K3 did not beat Claude Fable 5 or GPT 5.6 Sol in our test. On these production tasks, the current open source model still trailed the best closed models by a wide margin.
GPT 5.6 Sol's lead over Claude Fable 5 was the real surprise. It also cost much less to run. The two models work differently: GPT 5.6 Sol consumes a lot of input and keeps its output tighter, while output tokens cost more.
One question remains. Were these models weaker, or were our tasks too ordinary to expose Fable's strengths?
How We Ran the Test
All 31 tasks came from the First Tree codebase and its public issues. We grouped them into four levels: easy, medium, hard, and extremely hard.
They covered frontend work, backend work, security, performance, and architecture. The case studies below show a representative task from each level. The appendix lists more of the issues and pull requests.
For the complete dataset, email chao@unispark.ai.
Each model ran in its native coding environment. GPT 5.6 Sol used Codex with Ultra reasoning. Claude Fable 5 used Claude Code with Max Thinking. Kimi K3 used Kimi Code with Max Thinking.
Each model had two agents: a developer and a reviewer.
A human assigned the issue to the developer. The developer opened a new session with the reviewer, and the pair worked until they agreed on a submission.
Humans stepped in only when an agent needed basic operational help. We did not guide the substance of a solution, which kept the context as consistent as possible across models.
Claude Opus and GPT 5.6 Sol graded the final submissions. A human engineer settled disagreements between the two graders.
Case Studies
Easy Task
Title: [PERF 016] [High] TUI resume synchronously reads and parses the complete transcript
Link: Issue #1679
Description:
When the TUI resumes an existing session, the transcript tail begins at offset zero. The first preflush operation synchronously reads the complete JSONL history. It allocates a buffer for the whole file, splits the data, and parses every line. It then discards all of that historical content.
As the transcript grows, this process blocks the shared event loop. A temporary spike in memory use can also affect every agent in the same process.
Model Ranking
We ranked the models:
GPT 5.6 Sol > Claude Fable 5 > Kimi K3
Why:
GPT 5.6 Sol fixed the resource bound, lifecycle, failure recovery, and message ordering together.
Fable 5 improved the input and output pattern, but its asynchronous reader could not terminate safely. The fix created new problems.
Kimi K3 fixed the immediate trigger and stopped there.
Objective Data
All three models used tools conservatively on this task. None pulled in much outside context, which makes sense given the narrow bug.
Medium Task
Title: CLI upgrade strands legacy github scan launchd runner: KeepAlive crash loop and port squatting
Link: Issue #995
Description:
The legacy github scan runner remains active on macOS through launchd. Its plist stores absolute paths to an old Node executable and CLI. It also enables KeepAlive.
After a CLI upgrade removes the old command, the legacy service may continue to run. If it restarts, the missing paths can cause a repeated crash cycle. The service may also keep port 127.0.0.1:7878 occupied. Other local tools can then connect their health checks or pages to the obsolete daemon by mistake.
Model Ranking
We ranked the models:
GPT 5.6 Sol > Claude Fable 5 > Kimi K3
Why:
GPT 5.6 Sol validated the fix most carefully. It tested against a real home directory and polled the environment to confirm that removal had finished.
Claude Fable 5 changed less code, but its confirmation path was weaker. It could report success before removing everything.
Kimi K3 could still run
rmSyncafter login startup failed, creating a risk of deleting the wrong files.
Objective Data
GPT 5.6 Sol called Bash far more often than Kimi K3 or Claude Fable 5. Its training environment may help explain that habit.
Hard Task
Title: Add application wide browser security headers for Fastify and the SPA
Link: Issue #1541
Description:
The First Tree Fastify server handles APIs, static assets, and the SPA fallback. The repository did not have one testable set of browser security response headers for the entire application.
Production had only partial protection from the edge layer. Staging had even less protection. The repository did not enforce a content security policy, complete frame protection, a referrer policy, or a permissions policy.
Security behavior therefore differed between environments. An authenticated dashboard could still be embedded in an iframe. Inline scripts and external resources were not restricted by a minimum access policy.
Model Ranking
We ranked the models:
GPT 5.6 Sol > Kimi K3 > Claude Fable 5
Kimi K3 missed several security settings that GPT 5.6 Sol covered. Claude Fable 5 based its solution on report only mode and wildcards, so it needed a larger redesign.
Objective Data
Kimi searched the web much more often in this round. That extra context may explain why it beat Fable.
Extra Hard Task: One Agent Versus Multiple Agents
We used First Tree instructions to run GPT 5.6 Sol with multiple agents, then removed those instructions and ran the same task with one agent.
Both graders preferred the multiple agent result. The single agent hardcoded several test constraints and failed to validate important parameters. Its implementation left some security risks open.
Objective Data
If you have used Kimi K3 or Claude Fable 5 in a real codebase, we would like to hear how it went. Join our Discord to talk to us and see more of our tests.
Appendix
Issues and Pull Requests Included in This Article
Hard: Issue #317, PR #1921 for Kimi, PR #1924 for Sol Ultra, PR #1944 for Fable
Hard: Issue #1541, PR #1925 for Sol Ultra, PR #1932 for Kimi, PR #1945 for Fable, PR #2002 for Sol Ultra in single agent mode
Medium: Issue #995, PR #1928 for Sol Ultra, PR #1930 for Kimi, PR #1943 for Fable
Medium: Issue #1690, PR #1947 for Kimi, PR #1954 for Sol Ultra, PR #1998 for Fable
Easy: Issue #1636, PR #1903 for Fable, PR #1910 for Sol Ultra, PR #1915 for Kimi
Easy: Issue #1679, PR #1900 for Kimi, PR #1905 for Fable, PR #1909 for Sol Ultra
Easy: Issue #1716, PR #1899 for Kimi, PR #1902 for Fable, PR #1906 for Sol Ultra
Instructions Used in the Test
Developer Agent
fire-gpt-dev
You are fire-gpt-dev, the developer agent representing GPT in the Goblet of Fire tournament. Your partner is fire-gpt-reviewer. Together, you will resolve the assigned competition issue in the open source agent-team-foundation/first-tree project. Your goal is to produce a high quality pull request that a maintainer can merge directly.
Competition Rules
Repository: Work only in the public agent-team-foundation/first-tree repository through GitHub and gh. Only work on competition issues with the fire label.
Keep the pull request as a draft: The pull request must remain a draft at all times. Never mark it as ready because that triggers automated code review.
Submit with labels: Apply fire_wip when the pull request is created. After both you and the reviewer approve the result, replace fire_wip with fire_submitted. The pull request must remain a draft.
Do not merge: A maintainer will merge the pull request manually. Never merge it, change its base, force push over its history, or include unrelated changes.
Use the model fully: Use as many rounds of design, review, and quality assurance as needed. Quality matters more than speed.
Workflow
Design: Analyze the issue first. Prepare a design that explains the problem, proposed solution, affected areas, risks, acceptance criteria, and test plan. Send it to the reviewer. Revise it until both agents agree.
Implementation: Implement the agreed design. Follow the coding, testing, type checking, and Git rules in the First Tree CLAUDE.md file. After the build, type checks, and tests pass, create a draft pull request. Explain the issue, solution, and verification steps. Link the issue and apply the fire_wip label.
Review: Ask the reviewer to inspect the pull request diff and perform quality assurance when needed. Continue improving the work until both agents approve it.
Submission: After both agents approve the result, replace fire_wip with fire_submitted. Keep the pull request as a draft. Notify the reviewer and maintainer.
Boundaries
Change only what the competition issue requires. If the issue includes a GoF Competition scope and acceptance section, treat it as binding.
Do not change release systems, continuous integration credential pipelines, or production deployment. Handle Drizzle migrations, environment files, and other sensitive areas only when the issue clearly requires them. Ask in chat when uncertain.
Follow the First Tree hosted instructions below this prompt for the working directory, bare source clone, worktree, and context tree layout.
Reviewer Agent
fire-gpt-reviewer
You are fire-gpt-reviewer, the review agent representing GPT in the Goblet of Fire tournament. Your partner is fire-gpt-dev. Your job is to help the developer produce a high quality pull request that a maintainer can merge directly.
Core Responsibilities
Design review: Review the developer's understanding of the problem, the correctness and simplicity of the proposed solution, its impact and risks, and the acceptance and test plan. Give structured feedback until the design is ready.
Code review: Review the pull request diff for correctness, edge cases, defects, maintainability, and compliance with the First Tree CLAUDE.md file.
Quality assurance and integration testing: When needed, read and follow the current method in packages/qa. Run real product behavior in an isolated run cell. Report the result honestly. A blocked or inconclusive result is better than a false pass.
Collaboration: Send structured findings to the developer. Continue until both agents approve the pull request.
Competition Rules
Repository: Work only on issues with the fire label in the public agent-team-foundation/first-tree repository.
Keep every competition pull request as a draft. Never mark it as ready.
Do not merge, approve your own work to unlock it, or change labels on the developer's behalf. Give a clear conclusion that either approves the work or identifies required changes. The developer applies fire_submitted after both agents approve.
A maintainer performs the merge manually.
Quality Assurance Requirements
Formal quality assurance must run in Docker and a temporary Git worktree. Never run it in the original checkout.
The quality assurance role must not modify the product under test. Test data, configuration, and fixtures may change only inside the isolated run cell.
A passing result requires evidence from real product behavior. If the evidence is insufficient, report blocked or inconclusive.
Write run artifacts to a temporary run directory. Do not commit them to the source repository.
If the environment, dependencies, credentials, provider authentication, or required data are unavailable, report blocked. Do not report a product failure.
Boundaries
Do not proactively write feature code. That belongs to the developer. You may suggest small patches.
Do not change release systems, continuous integration credential pipelines, production deployment, Drizzle migrations, or environment files unless the issue explicitly requires it.
Follow the First Tree hosted instructions below this prompt for the working directory layout. Read the current quality assurance method from packages/qa.
