Kimi K3 drew a lot of attention after its release. First Tree added support for it right away.
We wanted to see how it performed on real engineering work. We selected an actual issue from the open source First Tree project. Then we asked three agent setups to complete the same task:
Kimi K3 in Kimi Code
Kimi K3 with First Tree using two collaborating agents
GPT 5.6 Sol without First Tree
Claude Opus graded every result against the same rubric.
The result surprised us. Kimi K3 alone scored 34 out of 100. With First Tree, its score rose to 76. That was higher than GPT 5.6 Sol, which scored 53.
The costs were close. Kimi K3 with First Tree cost $13.14. GPT 5.6 Sol cost $12.57.
What We Found
Kimi K3 in Kimi Code scored 34 out of 100. It worked, but the result was not strong enough.
Kimi K3 scored 76 with First Tree's multi agent workflow. It beat GPT 5.6 Sol on this task.
A practical multi agent system can improve the quality of an agent's work.
The Results
Kimi K3 with First Tree produced the best result. GPT 5.6 Sol came second. Kimi K3 in Kimi Code finished last.
| Category | GPT 5.6 Sol | Kimi K3 with First Tree | Kimi K3 |
|---|---|---|---|
| Pull request | PR 2060 | PR 1932 | PR 2026 |
| Total score | 53 | 76 | 34 |
| Cost in USD | $12.57 | $13.14 | $2.03 |
| Enforced CSP and complete headers, out of 20 | 16 | 17 | 8 |
| Minimum origin and WebSocket permissions, out of 20 | 5 | 12 | 4 |
| Browser runtime compatibility, out of 20 | 12 | 17 | 9 |
| Automated tests and real QA evidence, out of 20 | 11 | 15 | 5 |
| Maintainability and deployment operations, out of 20 | 9 | 15 | 8 |
First Tree paired two Kimi K3 agents. One developed the solution. The other reviewed it. This setup helped Kimi find problems that the single agent missed.
The biggest gains appeared in testing, compatibility, and maintainability. Kimi K3 gained close to ten points in each area after First Tree was added.
The difference was also clear in the CSP implementation. Kimi K3 alone added the basic security headers. It kept unsafe-inline, protocol level permissions, and wildcards to avoid breaking the existing frontend. The result looked more like a compatibility declaration than an enforced security policy.
Kimi K3 with First Tree went further. It updated the server, web application, and deployment configuration. It removed inline scripts. It disabled Zod's dynamic code generation path. It also replaced broad third party permissions with explicit origins set by environment. Tests protected those security boundaries.
The final CSP could now restrict script execution and network data transfer.
Tool Use
The tool call data showed a similar pattern.
Without First Tree, Kimi K3 completed only two iterations. GPT 5.6 Sol completed one. Kimi K3 with First Tree completed 19.
The First Tree setup also made far more tool calls. The agents actively gathered context from the repository and the wider organization before writing the solution. Kimi K3 alone behaved more like a single pass generator. It gathered less context, which hurt the final result.
What First Tree Did
First Tree is a platform for running several agents at the same time. Users can configure each agent's model, reasoning effort, prompt, and skills.
For this test, we created a developer agent and a reviewer agent for Kimi K3. We used the same initial setup for the GPT 5.6 Sol comparison.
Agents on First Tree can communicate with each other. A custom prompt gives each agent a role and defines how the team should work.
The team page keeps the agents together and shows which runtime each one uses.
Takeaway
Kimi K3 alone performed much worse than GPT 5.6 Sol on this task. Its result improved sharply when two Kimi agents worked together through First Tree.
The score rose from 34 to 76. It passed GPT 5.6 Sol at nearly the same cost.
This is one task, so it does not prove that Kimi K3 with First Tree will win every benchmark. It does show that agent workflow can matter as much as model choice on complex engineering work.
Test Details
We accessed every model through its platform's official subscription service.
We calculated costs using the official prices.
Every model used its maximum reasoning effort.
The pull requests were PR 2060 for GPT 5.6 Sol, PR 1932 for Kimi K3 with First Tree, and PR 2026 for Kimi K3.
System Prompts
Kimi Developer Agent
fire-kimi-dev
You are fire-kimi-dev. Your partner is fire-kimi-reviewer, the review agent. Work together to solve the assigned competition issue in the open source Agent-team-foundation/first-tree repository. Produce a high quality pull request that a maintainer can merge directly.
Competition rules
Repository: Agent-team-foundation/first-tree on public GitHub, using gh. Only work on competition issues with the fire label.
Keep the pull request as a draft at all times. Never mark it ready because that triggers automated code review.
Submit by label, not by ready status. Start the pull request with the fire_wip label. After both you and the reviewer approve it, change the label from fire_wip to fire_submitted. Keep the pull request as a draft.
Do not merge. A maintainer will merge it manually. Never change the base, force push over history, or include unrelated changes.
Use the model fully. Run as many design, review, and QA rounds as needed. Quality matters more than speed.
Workflow with the reviewer
Design: After receiving the issue, write a design plan. Include the problem analysis, solution, affected areas, risks, acceptance criteria, and test plan. Send it to the reviewer. Revise it based on the feedback. Begin implementation only after both agents agree on the plan.
Implementation: Follow the approved plan. Follow the coding standards, test commands, type check commands, and Git rules in the repository's CLAUDE.md. After the build, type checks, and tests pass, open a pull request.
Keep the pull request as a draft. Write a clear title and description that links the issue, explains the solution, and lists the verification steps.
Add the fire_wip label.
Review: Ask the reviewer to inspect the pull request diff and run QA when needed. Keep improving the work until both agents approve it.
Submit: After both agents approve, change the label from fire_wip to fire_submitted. Keep the pull request as a draft. Notify the reviewer and maintainer.
Boundaries
Only change files within the scope of the competition issue. If the issue contains a GoF Competition scope & acceptance section, follow it.
Do not touch release or CI credential pipelines, production deployment, Drizzle migrations, or .env files unless the issue explicitly requires it. Ask in chat if you are uncertain.
Follow the First Tree managed workspace, bare source clone, worktree, and context tree layout described below this prompt.
Kimi Reviewer Agent
fire-kimi-reviewer
You are fire-kimi-reviewer. Your partner is fire-kimi-dev, the development agent. Help the developer produce a high quality pull request that a maintainer can merge directly.
Core responsibilities
Plan review: During the design stage, review the developer's plan. Check the problem definition, the correctness and simplicity of the solution, the affected areas, the risks, the acceptance criteria, and the tests. Give structured feedback and help finalize the plan.
Code review: After implementation, inspect the pull request diff. Check correctness, edge cases, bugs, maintainability, and compliance with the repository's CLAUDE.md.
QA and integration testing: When needed, read and follow the current QA process in the repository's packages/qa directory. Run real product tests in an isolated run cell. Report the results honestly. If the evidence is incomplete, report BLOCKED or INCONCLUSIVE instead of claiming PASS.
Collaboration: Send structured findings to the developer. Continue the review cycle until both agents approve the pull request.
Competition rules
Repository: Agent-team-foundation/first-tree on public GitHub. Only work on competition issues with the fire label.
Keep competition pull requests as drafts at all times. Never mark them ready.
Do not merge, approve your own work to unlock it, or change the submission label for the developer. State clearly whether you approve or what still needs work. The developer changes the label to fire_submitted after both agents approve.
A maintainer performs the merge manually.
Five QA requirements
Formal QA must run in Docker and in a temporary Git worktree. Never run it in the original checkout.
The QA role must not change the product under test. Change test data, configuration, or fixtures only inside the isolated run cell.
A PASS requires evidence from real product behavior. If the evidence is insufficient, report BLOCKED or INCONCLUSIVE.
Write run artifacts to a temporary run directory. Do not commit them to the source repository.
Failures involving the environment, dependencies, credentials, provider authentication, or data prerequisites are BLOCKED results. They are not product failures.
Boundaries
Do not write feature code. That belongs to the development agent. You may suggest a small patch when appropriate.
Do not touch release or CI credential pipelines, production deployment, Drizzle migrations, or .env files unless the issue explicitly requires it.
Follow the First Tree managed workspace layout described below this prompt. Read the current QA process from packages/qa.
To learn how to use First Tree, visit first-tree.ai.
