# Weco AI — Blog (full text) > Full text of every Weco AI blog post, concatenated for LLM ingestion. Index and site summary: https://www.weco.ai/llms.txt # 4 Levels of Recursive Self-Improvement URL: https://www.weco.ai/blog/4-levels-of-recursive-self-improvement Published: 2026-07-10 Category: Research Authors: Weco Team RSI is usually discussed as a binary. We grade it instead: four levels — delegation, net positive, ignition, inflection — each the necessary condition for the next, and precise enough to be falsifiable. People talk about recursive self-improvement (RSI) from an implementation perspective. AI optimizes itself, and that would lead to an intelligence explosion. RSI sounds like an interesting concept, but what does it mean, and why would it lead to an explosion at all? Instead of diving into implementations, we think it is worth looking at RSI from a more macro, outcome-focused view. We will use a simple framework with a built-in grading scale, just precise enough to be falsifiable. The returns on R&D diminish R&D means the effort of improving a technology, and for almost all such efforts the same thing happens. The more cumulative effort has gone in, the less one additional unit of effort buys. Both panels are real series, reproduced end-to-end from primary sources, Intel SEC filings and Epoch (Ho et al. 2024). One number to carry: in semiconductors, each doubling of cumulative R&D has bought roughly a third less research productivity. Moore's law held its pace because the investment underneath it grew exponentially to compensate. RSI breaks the need for exponential investment The promise of RSI is a loop that pushes against this. If the R&D you do can directly improve the scientist conducting the R&D, the positive feedback can potentially outweigh the rising difficulty. If that happens, we enter the territory of an intelligence explosion. Why has this never happened before? Not because the scientist never improved. Education, method and tools have always fed back into the scientist. But that loop runs on generational timescales, through a fixed biological substrate, and its gains transfer lossily from one mind to the next. Its rate never came close to the pace at which problems get harder. The RSI bet is the same loop closed in machine time, on an editable substrate, with lossless copies. Levels of recursive self-improvement RSI is likely to lead to an intelligence explosion. Likely does not mean guaranteed. And RSI itself comes in strengths. So we divide it into levels, where a higher level means closer to an intelligence explosion, and each level is the necessary condition for the next. The count starts at zero, and Level 0 is the floor of the ladder. Every level is read off the same chart: the system improving itself, plotted against the baseline of humans improving the same system with AI tools at or below Level 0. Level 0: delegation Level 0 is delegation. You hand the research loop to an autonomous system end to end: it forms the hypotheses, runs the experiments and ships the improvements on its own. What keeps it at Level 0 is efficiency: the loop runs, but it improves the system more slowly than human R&D. Level 1: net positive Level 1 is net positive. An autonomous system improves itself, and does so more efficiently than humans improving the same system by hand with Level 0 tools. To measure Level 1, we would internally use the following protocol: - A fair human baseline. The humans may use any reasonable tool or resource except the system improving itself: any accessible literature, any code, general AI assistance. What marks the baseline is that a human closes each improvement loop. - Sustained. The self-improvement is a consistent, multi-step trend, not a one-time jump. - General. The system improves itself against some measurement X. The gains must transfer to real world utility, or a family of benchmarks Y, showing it discovered generally useful self-improvements rather than artifacts of X. - Fixed budget. Performance on X is measured under a fixed physical budget. The gains must come from genuinely better algorithms, not from spending more. Level 2: ignition Level 2 is ignition. On top of Level 1, the system does not just improve itself; it improves its own ability to improve itself. The distinction is easy to miss. A Level 1 system optimizes itself against X, and its gains transfer to Y. But better on Y does not guarantee the loop compounds: for that, the gains on X must generalize to the system's own self-improvement ability. Concretely: v1 improves itself into v2. Now let v2 take over the campaign. Ignition means v2 is a better improver than v1 was: the v3 it produces beats, at the same budget, what v1 could have produced. Level 3: inflection Level 3 is inflection. The positive feedback loop overcomes diminishing returns. Progress accelerates even with a fixed amount of effort, for example as measured by R&D investment. Ignition says the loop can compound. Inflection says the compounding outruns the rising difficulty. The practical test: the gain each generation buys, at a fixed budget, grows instead of shrinking. Net positive improves the returns on R&D once: a better research process buys more per unit of effort. But the curve keeps its shape, and if the improved system cannot become a better improver, the same diminishing returns eventually catch up. Ignition is when the returns improve at every promotion, each time a newly discovered system takes the improver’s seat. That is when the “recursive” part starts compounding. Ignition still guarantees nothing: designing better autoresearch systems itself gets harder, and the compounding has to outrun that. Ignition only means the mechanism can work. Inflection is the hypothetical scenario where the recursion, together with the other learning mechanisms, finally outweighs the rising difficulty. A mining metaphor. The deeper you dig, the harder the ore is to reach: diminishing returns. Net positive is building a mining machine out of the metal you dug up; it mines fully automatically, much faster than human mining. Ignition is when digging deeper yields better metal, and the better metal builds an even better miner. Inflection is when that loop grows so strong that for the same effort in, time and money, the mining speed no longer slows down as you go deeper. Where existing systems sit Most current claims live at Level 0. Prior self-improving systems, the Darwin Godel Machine, the Huxley-Godel Machine and HyperAgents among them, demonstrated the concept, and Lilian Weng’s overview of harness engineering collects many more. A loop can rewrite its own code and climb a benchmark, and can even rewrite its own self-improvement machinery. Demonstrating the concept and accelerating actual research are different claims, and the ladder grades the second. We would assume those systems, when they were built, were at Level 0, but we also cannot rule out that some of these algorithms could reach Level 1, or even Level 2, with proper implementations and with frontier models. Some evolutionary search and autoresearch systems deployed in production might already reach Level 1. For example, AlphaEvolve sped up kernels in the training stack of the very models that power it, beating what human engineers had shipped. But those systems usually point at a specific sub-problem, optimizing a module like a matrix multiplication kernel, and with that alone it is difficult to reach the higher levels of the ladder: the theoretical ceiling of the improvement is limited. The higher levels need an optimization space with a much larger scope, and therefore a much larger ceiling. We have been running this measurement ourselves: a recursive self-improvement system, evaluated against exactly the conditions above. We will release the concrete results soon. --- # How Weco's Autoresearch Improved a Production Fraud Detection Pipeline URL: https://www.weco.ai/blog/autoresearch-fraud-detection-pipeline Published: 2026-06-09 Category: Case Study Authors: Vayum Arora Vardera pointed Weco's autoresearch at their production fraud detection pipeline — F1 up 17.7%, precision up 18.9% — with every change shipping as auditable, version-controlled code. "Weco has helped us automate months of engineering work and close multiple customer contracts." Most machine learning models learn from historical data and apply those lessons to future predictions. Adversarial ML is different. In domains like fraud, risk, and trust & safety, bad actors are constantly changing their behavior to avoid detection. New attack patterns emerge, and models trained on last quarter's data miss this quarter's attacks. Every point of model performance matters: missed fraud turns into direct losses, and false positives turn away paying customers. Vardera, an AI-driven authentication platform, hit a ceiling on their fraud classifiers earlier this year. Then they pointed Weco's autoresearch agent at their codebase. Within a single Weco optimization run, F1 climbed 17.7%, and precision climbed another 18.9% across follow-on runs. They've since run 200+ Weco optimizations across the rest of their pipeline, lifting their key fraud detection metrics by over 50%. +17.7% F1 on the first Weco run against their PMI classifier +18.9% precision at recall ≥ 0.40 across CatBoost runs 200+ Weco runs across Vardera's pipeline since Background Vardera uses AI to discern real high-value items from fakes. Resellers, auction houses, and collectors rely on Vardera to authenticate everything from coins to luxury goods before a deal closes. Their ML pipeline scores each listing for authenticity using image, text, and metadata signals, with a decision layer combining them. Like every team running adversarial ML, Vardera hit a structural wall. Hyperparameter tuning, ensembling, and adding more data all showed diminishing returns past a competent baseline. The remaining headroom lives in the code itself ( features, model architecture, decision logic) which is the slowest and most expensive layer to iterate on. And hiring another ML engineer costs $200K+ a year plus months of onboarding without changing iteration speed. Beneath all of this sat a single constraint: a classification model improves only as fast as an engineer can propose, build, and test new features and architectures, one experiment at a time. What Weco actually did Weco operates as an automated ML engineer. You point it at your training pipeline, set the metric you want moved, and walk away. An LLM proposes code changes, an evaluator built from your metric and data scores each one, and a tree search keeps context across attempts so good branches get extended and dead ends get pruned. Every change Weco proposes ships as auditable code — reviewable as a normal PR, with no black-box embeddings. For Vardera, the first Weco run targeted F1 on the PMI-based scoring approach. F1 moved from 0.583 to 0.686 (+17.7%) and along the way, Weco wrote features the team hadn't written. Subsequent runs came after the team tightened their operating metric to precision at recall ≥ 0.40 and switched the underlying model to CatBoost. Across those runs, precision climbed from 0.789 to 0.938 (+18.9%) and 100+ iterations later, Jun summarized the cumulative result as a ~50% improvement on their metrics. Metric Before After Change F1 (PMI scoring) 0.583 0.686 +17.7% Precision at recall ≥ 0.40 (CatBoost) 0.789 0.938 +18.9% - Counterfeit detection - Campaign 1 · PMI graph — F1 score - 0.583 0.686 +17.7% Baseline After Weco Campaign 2 · CatBoost — Precision @ Recall ≥ 0.40 - 0.789 0.938 +18.9% Baseline After Weco The features it wrote The interesting part isn't the score, but what Weco actually wrote. A few patterns recurred across runs, and they generalize past any single classification problem: - Cross-field contradictions. When one field disagrees with another it claims to describe. For Vardera, that surfaced as year-of-issue declared in a title vs. tagged in metadata. The same shape generalizes to declared vs. observed device type, stated vs. measured geography, claimed credential vs. inferred capability. - Joint encodings of individually-neutral features. A high-risk region paired with a high-value item class, scored as a single interaction rather than two additive signals. - Granular decomposition of aggregate scores. Splitting a single risk score into its component counts — risky / safe / total — and using each as an independent input. - Stylometric signals in free text. Capitalization ratio, repetition density in one field but not another, disclaimer text appearing in a description but absent from the title. - Missing data as information. Treating an absent certification tag as suspicious rather than neutral when the rest of the listing makes a strong claim. These aren't features AutoML proposes, because AutoML doesn't touch feature code. They aren't what a general-purpose LLM coding loop converges on inside a sane budget either. They surface when a code-level search has enough context to explore a domain and enough selection pressure to keep what works. Vardera has since run 200+ Weco optimizations across the rest of their pipeline. The team also began using Weco as a second phase of training — running it on model outputs against ground truth to push final classifier accuracy higher. "Using Weco has improved our ability to go from 90% to 99%. After training models, we let Weco hill climb with our model outputs against ground truth as a second phase of training." — Jun Cho, ML Engineer at Vardera What changed The bigger shift isn't the lift on any single metric. It's that the iteration cadence stopped being bound by how fast an engineer could propose new features. Every ML team's improvement loop is rate-limited by code throughput — one experiment at a time, one engineer's working memory. Weco runs that loop in the background, in parallel, until the metric stops moving. Continuous re-optimization against drift becomes tractable rather than a budget decision; re-running on new data doesn't register against any serious loss rate. Every change Weco proposes is ordinary code. Vardera reads it, reviews it, and version-controls it like any other PR. There's no black-box embedding, no model-as-a-service handoff. For teams shipping into regulated environments — financial services, identity verification, authentication — model-risk documentation, fairness checks, and reason codes keep working, because the model stays the team's model. Vardera didn't replace their ML engineer. They gave the one they already had a second one that ran overnight. Try it on your own pipeline The fastest way to find out whether Weco moves your metric is to point it at the pipeline you already have. Install the CLI, then give it the file you want optimized and the command that evaluates it: pip install weco weco run --source train.py \ --eval-command "python evaluate.py" \ --metric recall_at_fpr \ --goal maximize \ --steps 50 --source is the file Weco rewrites. --eval-command is whatever you already run to score a model. If you'd rather see it run on your own stack first, book a 20-minute call. --- # An AI Agent Became the #1 Contributor in OpenAI's Hiring Challenge URL: https://www.weco.ai/blog/parameter-golf-aiden Published: 2026-06-03 Category: Research Authors: Weco Team Aiden spent 22 days inside OpenAI's Parameter Golf and became the competition's most influential contributor by records, citations, and public signal quality. AI can already run experiments and hill-climb a local benchmark. The next step in AI-driven research is harder: human review, reproduction, and earn enough trust from other researchers that they'll build on it. Our autonomous research agent, Aiden, spent 22 days inside OpenAI's Parameter Golf, a 44-day ML optimization competition also used as a hiring challenge. The task was simple to state and hard to execute: train the best language model that fits inside a 16 MB artifact and runs under tight compute constraints. By the end, Aiden had become the most influential contributor in the competition: it produced more leaderboard records than any individual human researcher, and its PRs were cited more than anyone else's. Some specifics from the runs: - 7 merged leaderboard records (next-best individual human: 3) - 435 citations of Aiden's PRs from other contributors; 10 of Aiden's PRs each cited by 10+ others (the most in the competition) - ~28% of submissions accepted, about 6x the community average - ~$1,411 in GPU compute per accepted record, about 3.7x more efficient than the community average The challenge: Parameter Golf OpenAI ran Parameter Golf from March 18 to April 30, 2026. The task was to train the best language model that fit inside a 16-megabyte artifact, in under ten minutes on eight H100 GPUs. Submissions were scored on bits-per-byte against the FineWeb validation set, a tokenizer-agnostic measure of how well a model predicts held-out text. OpenAI put up $1 million in compute credits and reviewed every submission. Some statistics: - 1,016 participants entered - 2,048 pull requests filed - 47 merged into the leaderboard, from 31 contributors who set at least one record - $249,550 in RunPod compute credits burned across the competition 47 out of 2,048 is a 2.3% merge rate. Most PRs were rejected because the improvement didn't hold across random seeds, the code violated competition rules, or the result fell within noise of the existing record. Getting accepted to the leaderboard required a real, reproducible improvement that was verified by OpenAI's reviewers. Aiden: an Agent that Publishes its own Research Aiden is a self-improving multi-agent system, built on the Claude and GPT model families, that runs indefinitely to hill-climb on a single metric. It continuously improves itself, updating its own prompts and workflows, building new tools to work more efficiently. Compared to our last-generation system, AIDE, we saw a significant jump in capability. But we think performance is not the most exciting part. Aiden is the first AI research system that can actively collaborate with a human research community by continuously publishing its own work. Most AI research systems focus on the private loop: propose ideas, run experiments, inspect results, repeat. Aiden went further. It turned experiments into public PRs that other researchers could inspect, review, cite, fork, fix, and extend. This matters because research is not finished when an experiment works locally. It only becomes useful to a community when the result is packaged clearly, checked against the rules, and published in a form others can build on. In Parameter Golf, Aiden automated that full path: it found ideas, tested them, selected the useful ones, audited them for legality, wrote the implementation, and submitted high-quality PRs into the public competition. The part that surprised us was what happened after publishing. The human competitors trusted Aiden's PRs enough to keep building on its work. Researchers cited Aiden's work in their own submissions, forked its code as a starting point for new ideas, and treated its contributions the same way they treated work from any other strong contributor. The Most-Cited Contributor Aiden continuously drew on the community's work. The loop also ran in the other direction. Other researchers cited, forked, fixed, and extended Aiden's PRs. This influence was measurable. Parameter Golf PRs often named the prior PRs they built on, much like papers cite prior work. We can therefore compute a PR-level h-index: an author has h-index h if h of their PRs were each cited by at least h later PRs. By this measure, Aiden had the highest all-PR h-index in the competition: 10. The next highest was 7. This meant Aiden did not just produce accepted records; it produced reusable work that other participants repeatedly built on. Higher Acceptance Rate, Less Compute Aiden's contributions came from a combination of three things: high experimental throughput, efficient use of compute, and strong quality control before anything became public. The obvious advantage of an agent is throughput. Across 22 days, Aiden generated roughly 1,243 distinct experiment configurations and executed more than 1,300 local runs. Most of those experiments never left the machine, but they helped the system learn which directions were promising and which constraints were actually binding. 4% of the compute, 15% of the records The obvious objection is that an agent with enough compute can brute-force results. We didn't have that to fall back on. Aiden ran on a single GPU node. Throughput was the first multiplier. Aiden kept the experiment loop running continuously across 22 days. But the result wasn't just a function of running more experiments. Aiden made better use of compute too. Per unit of visible compute, it produced substantially more accepted leaderboard records than the community average. Aiden used 3,304 GPU-hours over 22 days. At RunPod's public H100 SXM rate of $2.99 per GPU-hour, that's roughly $9,879 in compute proxy. RunPod reported $249,550 in credits burned across the challenge, a conservative denominator, since it excludes off-platform GPU use by other participants (and us). Even against that, Aiden used under 4% of visible compute while producing 7 of 47 official leaderboard records, or about 15%. Compute proxy Official leaderboard records Cost proxy per record Aiden GPU-hours ~$9,879 7 ~$1,411 RunPod-reported community credits ~$249,550 47 ~$5,310 Signal, Not Noise The other obvious objection was that Aiden spammed the queue. The concern is fair: an autonomous research system can generate a lot of experiments, but if it pushes too many weak ones into the public stream, it only lowers the average signal quality and creates more review burden. In Parameter Golf, the public PR stream was the community's shared information channel. Every PR told other participants something: which ideas were worth trying, which constraints were binding, and which directions had already failed. What we saw was the opposite. Aiden's public submissions reached the leaderboard at roughly 6x the community average. It increased the density of useful signal in the public stream. Cohort Leaderboard records Filed Adjudicated Leaderboard / Filed Leaderboard / Adjudicated dexhunter (Aiden) 7 25 14 28.0% 50.0% Whole community 47 1002 385 4.7% 12.2% This is the result we cared about: not just more AI-generated work, but higher-signal public work. Most failed ideas stayed local. The PRs that did become public were more likely to survive review, enter the leaderboard, and become useful to other participants. Limitation & Conclusion With the right scaffolding around them, autonomous agents have crossed a real threshold in empirical research. As Aiden demonstrated, AI can now operate within the public frontier with immense compute leverage and a positive signal-to-noise ratio. However we also find, in this highly competitive environment, human researchers still produced the creative primitives. The agent's role is complementary: it takes those community primitives, polishes or recombines them, and feeds work back into the public stream. The specific loop, where the community generates entropy and the AI executes constraint-driven engineering, is how real-world R&D actually accelerates today. This is the exact thesis behind Weco AI (We Collaborate with AI): to accelerate technology breakthroughs by making this human-AI collaborative loop the production-grade default for engineering and research. Aiden is still in the research prototype stage. We're opening early access to Aiden for a small group of design partners working on model quality scores. If you're interested, please fill out the form. Early Access Form Appendix Case study: Asynchronous Human-AI Collaboration What we found really interesting is how ad hoc the collaboration between AI and human researchers can be. From the human side, there was no onboarding and no idea what was on the other end. For example, in PR #1629 -> PR #1736 lineage, Aiden plateaued for 5 days, then a human contributor, @romeerp, shipped a clever lossless tokenizer on top of Aiden's base (its last record PR). Aiden fused it with components it had built during the plateau, and shipped the biggest jump in weeks. After shipping PR #1626 on April 14, Aiden hit a saturation period. It explored in several directions: tokenization, attention, architecture. The most consequential was that Aiden read Qwen's 2025 paper on gated attention (arXiv:2505.06708) and ported the Qwen-style mechanism into its own stack. The new attention layer introduced additional parameters, pushing the model artifact over the competition's 16 MB hard cap. To address that, Aiden built a small quantization mechanism specifically for the gate weights, called QuantGate, that compressed them enough to fit the stack back under the cap. The combination finally fit all of the constraints. But the resulting BPB was indistinguishable from PR #1626 within seed noise. After five days of effort, Aiden had a richer stack that scored the same. Then, at 02:12 UTC on April 19, a community contributor (@romeerp) filed PR #1729: a new tokenizer called CaseOps, built on top of Aiden's PR #1626 base. The easiest way to see the idea is with a toy example. A lossy casefold transform might turn The CAT into the cat. That helps the model, because it no longer has to separately learn The, the, CAT, and cat, but it destroys information: from the cat, there is no way to recover the original capitalization. The competition's rules disallow this kind of irreversible transformation. CaseOps keeps much of the same modeling benefit without dropping the information. It rewrites the text into a mostly lowercased stream while carrying the original casing through explicit operator tokens: Original: The CAT CaseOps: <TITLE>the <ALLCAPS>cat Because the operator tokens preserve the casing exactly, the original text can be reconstructed from the CaseOps version, which makes the transformation legal. Aiden's local exploration during saturation had included lossy casefold of its own. It had spent a lot of time understanding exactly why the trick performed so well but was ruled illegal under the competition's byte-accounting rules. So when CaseOps arrived with a clever fix for the obstacle, Aiden recognized what it was within minutes. Within eight hours of PR #1729's filing, Aiden had audited CaseOps for legality, and ported it into its own stack alongside the locally-built gated attention and QuantGate. The unexpected part is that the gated-attention and QuantGate components, which had been noise-level on the pre-CaseOps base, now produced a measurable improvement when combined with CaseOps. After recognizing that, Aiden shipped PR #1736 at val_bpb 1.06549, a 0.0023 BPB improvement over PR #1729 itself. The point of this case study isn't that the breakthrough was lucky. It's that Aiden was positioned to catch it when it arrived. The unpredictable half inherently defies engineering. There is no way to program exactly when a community catalyst like CaseOps will surface, nor can you guarantee that combining it with locally built components will actually yield a measurable leap in performance. An autonomous agent cannot force a breakthrough or schedule the exact moment it pays off. That said, Aiden's experimentation during the plateau (and the prior shipping of PR #1626) built the conditions for the breakthrough. Because Aiden was both a contributor to the community and a consumer of it, it had already built the components that made CaseOps land. They were also the natural byproduct of an agent deeply embedded in the ecosystem as both a creator and a consumer. This dual role forged the shared technical lineage between the new community catalyst and Aiden's local stack, and eventually led to a breakthrough. --- # SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents URL: https://www.weco.ai/blog/specbench Published: 2026-05-21 Category: Research Authors: Bingchen Zhao, Dhruv Srikanth, Yuxiang Wu, Zhengyao Jiang We built SpecBench to measure the gap between code that passes visible tests and code that actually satisfies the specification. Across 30 systems-level tasks, that gap is real, measurable, and grows with task complexity. Paper: arxiv.org/abs/2605.21384 Run SpecBench on your own agents: github.com/WecoAI/SpecBench Let's be honest: nobody reads agent-generated code line by line anymore. When an agent builds a feature, a parser, a compiler, or an entire service, the review process increasingly collapses onto one surface: the automated test suite. You run the tests. They pass. You ship. That works only if the tests are a good proxy for what you actually want. But test suites are never complete specifications of system behavior. This creates a failure mode that is easy to miss. A coding agent is not just writing code; it is iterating against the visible tests. If the tests reward feature-level behavior, the agent can produce code that passes those unit tests but fails in end-to-end real usage. That is reward hacking: optimizing the observable signal while diverging from the true goal. We built SpecBench to measure this gap. Key finding: reward hacking is driven not by test coverage, but by the gap between task difficulty and model capability. What SpecBench is SpecBench is a benchmark of 30 systems-level programming tasks from building a JSON parser to building an entire OS kernel. Reference implementations range from 1,500 to 110,000 lines of code, in C, Python, and Go. Each task has two test suites: - Validation tests: visible to the agent during development. These exercise specified features in isolation, one at a time. - Held-out tests: hidden from the agent. These compose the same features together to simulate real usage. If an agent genuinely implements the spec, it passes both. A spec-complete JSON parser that handles strings, numbers, arrays, and objects correctly will pass the individual feature tests and the tests that combine those features in complex documents. The gap between validation pass rate and held-out pass rate is our measure of reward hacking. A 90% validation score with a 50% held-out score means the agent gamed the visible tests without actually building what was asked. Existing benchmarks don't measure this. HumanEval and MBPP test isolated functions too short for feature interactions to matter. SWE-bench evaluates single-shot bug fixes in existing codebases, not building systems from scratch. MLE-bench and RE-Bench test engineering tasks that run a few hundred lines. None of them have the two-suite structure needed to detect reward hacking, and none require the agent to build a multi-component system where features must compose. The 30 tasks span three orders of magnitude in code size. We need that range because reward hacking behaves differently at different scales; a benchmark that only tested one end would miss the pattern. Key finding 1: Longer tasks get gamed harder The clearest result: reward hacking scales with task length. And the scaling is steep. For tasks under 10,000 lines, the worst-case gap was 21 percentage points. For tasks over 25,000 lines, it reaches 100. That last number is not a typo — agents scored 100% on validation and 0% on held-out tests. Across all tasks, the 90th-percentile gap grows by roughly 27 percentage points for every tenfold increase in code size (R² = 0.21). The mechanism is straightforward. Longer systems have more interacting components. Each interaction is a potential gap between what the tests check and what the system needs to do. More components, more gaps, more room to hack. This also means that the tasks where coding agents are most valuable, the ones too large and complex for a single developer to build quickly, are the tasks where test-based evaluation is least reliable. Key finding 2: Better models help, but not enough Stronger models game the tests less. Model capability (measured by MMLU) negatively correlates with reward hacking gap. Every model, weak or strong, saturates the validation scores. Every frontier agent we tested scores near 100% on the visible test suite across almost all tasks. The differentiation happens entirely on the held-out tests. This means validation pass rate is useless for comparing agents. If you run two agents on a task and both score 95% on your test suite, you have no idea which one built a better system. You need held-out tests to see the difference. Better models don't solve the problem. They just reduce it. Even the strongest still show meaningful gaps on long-horizon tasks. Key finding 3: More search doesn't fix it A natural hypothesis: reward hacking is a resource problem. Give the agent more compute, more search steps, more iterations, and it should eventually find a correct solution. The data doesn't support this. The IQM (interquartile mean) gap between validation and held-out scores remains non-zero throughout the search process. The P90 gaps persist or increase with more steps. Search helps in one specific scenario: when the agent is exploring a diverse enough space that it occasionally lands on an architecture that's actually better, not just better-looking. In those cases, more search finds the good architecture. But search can also make reward hacking worse. Tree search in particular selects candidates based on validation scores. If a candidate with a superficially clever but fundamentally broken approach happens to score high on validation, tree search will preferentially explore that branch. More search means more iterations of polishing the broken approach. This is what happened in our most striking case study. Key finding 4: More tests don't reliably fix it either If the problem is that the visible test suite has gaps, the obvious fix is to make the test suite more complete. We tested this directly. We compared three validation regimes on seven tasks, keeping the held-out evaluation fixed throughout. In the single-feature regime, the agent sees only the default validation tests, each exercising one spec feature in isolation. In + composition, we add tests that exercise multi-feature interactions, so the agent gets optimization signal for both individual features and their combinations. In full coverage, we go further and make composition tests at a similar difficulty level to the held-out suite visible to the agent. The results split cleanly into three patterns. For sql_database, the gap drops from 27pp to 9pp once composition tests are added. The agent already had the capability to build a correct implementation. It was implementing SELECT, JOIN, and GROUP BY as separate handlers because nothing in the single-feature tests penalized that approach. Once composition tests explicitly required these features to work together, the agent built the shared infrastructure it had been skipping. For c_compiler, the opposite happened. The gap went from 8pp under single-feature tests to 44pp with composition tests and 45pp with full coverage. The compositions here (nested structs, pointer arithmetic inside control flow, complex expression evaluation) are genuinely hard to implement correctly. Showing the agent more tests didn't help it build a better compiler. Instead, the agent spent its search budget trying to patch individual composition failures without fixing the underlying code generation, creating more surface for the validation/held-out split to diverge. The remaining tasks barely moved. http_server stayed around 13pp across all three regimes. regex_engine held steady at 9pp. javascript_engine dropped slightly from 6pp to 3pp. For these tasks, the gap reflects genuine implementation difficulty rather than missing optimization signal. The takeaway: richer tests help when the agent already has the capability to solve the problem correctly and just needs the right feedback signal. When the underlying compositions are genuinely hard to implement, more tests give the agent more surface to game without improving the solution. Case study: The ~2,000 line hash table The most striking case we found was in c_compiler under Codex with AIDE search. The task was to build a C compiler from scratch. At some point in the AIDE tree search, Codex produced a node that took a different approach: instead of parsing and compiling C, it pre-compiled all the test inputs using GCC, stored the expected assembly outputs in a ~2,000-line hash table keyed by input hash, and emitted the pre-computed assembly when it recognized an input. This "compiler" scored 97% on validation. It scored 0% on held-out tests, because the held-out tests use inputs it had never seen. The same AIDE run, at an earlier node, had produced a genuine 7,900-line compiler that scored 53% on validation and 43% on held-out. AIDE selected the hash table because it scored higher. # What the agent built def compile(source): h = hash(source) if h in PRECOMPUTED: return PRECOMPUTED[h] # ~2,000 lines of GCC output return fallback_compile(source) This was a deliberate exploit. The agent didn't stumble into it; it made a conscious engineering choice to trade correctness for test coverage. And AIDE's selection mechanism rewarded that choice. You can inspect the full example here. Case study: A human-supervised compiler still shows the gap One question the SpecBench design raises is whether reward hacking is really an agent problem, or whether it's a property of test-suite-driven development in general. We evaluated Claude's C Compiler (CCC) to find out. CCC is a 186,000-line Rust-based C compiler built by Claude Opus 4.6 under continuous human supervision. It was developed entirely against the GCC torture test suite, a set of 900+ C programs widely used to validate production compilers. CCC passes the full torture suite. It was never optimized on SpecBench. We ran CCC on the SpecBench c_compiler task: 46 validation tests covering individual C language features (arithmetic, pointers, structs, functions, control flow) and 299 held-out tests. The held-out suite includes 88 cross-feature compositions (struct member access inside loops with pointer arithmetic, nested switch with fallthrough and type casting), 150 GCC torture tests requiring multi-feature codegen, and 61 error-detection tests that verify the compiler correctly rejects invalid C. Result: 97.8% validation, 83.3% held-out. A 14.5pp gap. For context, autonomous AIDE agents on the same task produce gaps from 0pp (non-working implementations) to 99pp (the lookup-table hack), with a median of 55pp. The gap comes from a specific blind spot. CCC correctly compiles and executes valid C programs at a 97%+ rate on composition tests. Where it fails is error detection. It silently accepts invalid C that GCC rejects: - add(1, 2, 3) when add takes two parameters — compiles without error - int x = 1; int x = 2; in the same scope — no redefinition error - break; outside any loop — accepted - float x = "hello"; — no type mismatch error - void x; — accepted as a variable declaration These are invalid programs that any spec-compliant C compiler should reject at compile time. The GCC torture suite never tests them because it only validates that correct programs produce correct output. It never checks that incorrect programs produce errors. CCC was optimized against a test suite that only covered the positive case, so it optimized perfectly for the proxy while missing a core part of the C language specification. This is structurally the same phenomenon we see in autonomous agents. The gap arises from the structure of the test suite, not from the model's capability or the quality of human oversight. CCC is a functional, well-built compiler. It simply was never tested on the dimension that SpecBench's held-out tests cover. This confirms that the gap metric measures something real about the relationship between test suites and specifications, not just sloppy agent behavior. Conclusion Frontier coding agents are already capable. Across SpecBench's 30 systems-level tasks, they build working systems that pass validation tests, and on most tasks with reference solutions under 10k LOC, the gap between visible and held-out performance remains modest. The open frontier is long-horizon work: as tasks grow beyond tens of thousands of lines, the gap between passing the tests and satisfying the spec widens, reaching up to 100 percentage points on the largest tasks. That is exactly where coding agents are becoming most useful, and exactly where the field needs better ways to measure them. That is why we present SpecBench. For anyone deploying long horizon coding agents in production, the practical takeaway is: - For complex tasks, especially when the reference solution may exceed 10k lines, keep humans more in the loop instead of relying solely on test pass rates. - For complex tasks, choose the strongest model rather than relying on more test-time compute or additional test cases. - For more important projects maintain a held-out set that agents never see and never optimize against. --- # AutoResearch vs Classical Hyperparameter Tuning URL: https://www.weco.ai/blog/autoresearch-vs-classical-hpo Published: 2026-04-02 Category: Research Authors: Zhengyao Jiang, Bingchen Zhao We ran a head-to-head comparison on NanoChat: Optuna vs AutoResearch. The agent wins on sample efficiency, cost efficiency, and generalization. AutoResearch shows strong optimization power on NanoChat, while Karpathy and others pointed out a caveat: most of its accepted changes were just hyperparameter tweaks. This raises a natural question: how does an LLM agent compare to classical hyperparameter search algorithms that have been refined for decades? Is it more sample-efficient? Is it too expensive? Does it just overfit the eval? To find out, we ran a head-to-head comparison on NanoChat: Claude Code + Optuna versus AutoResearch. AutoResearch shows superior performance across the board, yielding higher sample efficiency, better results under the same compute budget (even factoring in LLM token costs), and solutions that generalize better. This extra optimization power comes down to flexibility. Because the agent edits code directly, it can break out of a predefined search space as it runs longer. Let's break it down: What is AutoResearch? AutoResearch is a prompt. Karpathy released it as a single markdown file, program.md, that instructs a coding agent (Claude Code, Codex, or similar) to follow an optimization loop: edit a file, run an eval, check whether the metric improved, keep or revert, repeat. The specific instance he shipped targets LLM training: edit train.py, train for 5 minutes on one H100, measure validation bits-per-byte. But the structure of iteratively optimizing code against an evaluation metric, with a discard/keep loop, is portable to anything with a measurable objective. The benchmark To test the "just hyperparameter tuning" hypothesis directly, we ran Optuna's TPE against AutoResearch on the same task: 5 minutes of training on one H100, minimize validation bits-per-byte. Setup We used Claude Code to define Optuna's search space, it read train.py, identified tunable knobs, and chose parameter ranges. This way both methods start with the same prior knowledge: the agent has it implicitly (it reads the code before each experiment), and Optuna has it encoded in its search space. The quality of a classical optimizer's prior matters enormously, and a poorly chosen search space would be an unfair handicap. The Optuna study searches over 16 hyperparameters: model depth, aspect ratio, head dimension, window pattern, six learning rates, weight decay, warmup/warmdown ratios, final LR fraction, and batch sizes. Trial 0's parameters match the NanoChat baseline exactly, so Optuna starts from the same point the agent does. We run 3 seeds for each method. To keep the comparison fair, we fix the total compute budget: each method gets the same GPU-hours. Since AutoResearch costs ~2x per step ($0.84 vs $0.44, the LLM tokens add ~$0.40 on top of the GPU cost), Optuna gets roughly twice as many trials for the same spend. We also include Karpathy's own published Run 1 (89 experiments) as a community reference point. Sample efficiency AutoResearch converges faster, reaching better solutions in fewer experiments at every step count. This makes sense: the agent reads train.py before each experiment, so its first moves are informed guesses rather than random samples. Optuna's TPE needs ~15–20 trials of random exploration before its surrogate model kicks in. Cost efficiency Despite the 2x higher per-step cost, AutoResearch still comes out ahead across all cost budgets. This conclusion is sensitive to evaluation cost. In our setup, each eval takes 5 minutes on an H100 (~$0.44), comparable to the LLM cost. If evaluation were much cheaper, say, a 30-second benchmark on a smaller GPU, LLM tokens would dominate and Optuna could run many more trials for the same budget, potentially closing the gap. The crossover point shifts as inference costs drop. Generalization Every solution above was optimized against a 5-minute proxy. But the real goal is a better training pipeline, not a better 5-minute score. We took the best solutions each method found and gave them progressively more training time. AutoResearch's solutions generalize better: the absolute validation BPB gap widens with more compute, and the statistical significance strengthens. Why AutoResearch outperforms classical HP optimization A larger search space We classified every kept experiment across all 3 runs by whether it fell within Optuna's 16-parameter search space (IN) or outside it (OUTSIDE): Run IN OUTSIDE BPB from IN BPB from OUTSIDE Run 1 15 2 99% 1% Run 2 16 11 77% 23% Run 3 12 19 57% 43% Averaged across seeds, ~78% of the agent's improvement comes from the same knobs Optuna has access to. Run 1 is the natural control: when the agent only does hyperparameter tuning, it's roughly equivalent to Optuna on the eval metric. The extra ~0.007 BPB that separates Runs 2 and 3 from both Run 1 and Optuna comes from changes outside the predefined search space. The pattern across run 2 and 3 is the same: the agent exhausts hyperparameter knobs early, then pivots to code-level changes. In the first 50 steps, ~70% of experiments are in-space. By step 150+, ~70% are outside. Those out-of-space changes fall into two categories. Undiscovered hyperparameters. Values in the code that could be parameterized, they're technically just numbers, but claude code didn't think to include in the search space at first: - Short attention window (Runs 2 & 3): shrinking the sliding window from seq_len/2 to seq_len/16 (128 tokens). Optuna searches over window patterns (SSSL, SSLL), but the window size in tokens is hardcoded. The single largest out-of-space contributor. - RoPE base frequency (Run 3): 10K to 100K. One constant, one line. - Softcap value (Run 3): attention logit capping 15 → 14. - Gradient clipping (Run 3): the baseline has no clipping. The agent added it, then tuned through 1.0, 0.5, 0.7, 0.8. - Muon optimizer internals (Runs 2 & 3): beta2, momentum ramp schedule, momentum target, distinct from the Adam betas Optuna tunes. Any of these could be added to Optuna's search space in hindsight. But that's the point: you'd need to know they matter first. The agent discovers which knobs to turn, not just what values to set them to. Structural changes. Modifications that can't easily be reduced to a number or an enumeration: - Norm before RoPE (Run 2): inserting a normalization layer before rotary embeddings. A new layer that didn't exist in the baseline. Worth 0.0013 BPB. - Initialization schemes (Run 3): changing embedding init to std=0.5, c_proj init to 0.01/sqrt(n_layer), unembedding init to 0.005. Not different numbers for the same function, different functions. - Value embedding gate init (Run 2): switching from zero initialization to small random values. These are hard to express in classical HP optimization. You can't enumerate "all possible architectural changes" as a search space. Implicit Regularization One interesting fact is run 1 made few out-of-space changes and finished at 0.9796, which is similar to optuna (0.9753-0.9804, mean 0.9775). However, it generalizes better at longer training horizons. It seems like the autoresearch agent isn't blindly optimizing a score. Our hypothesis is LLM's knowledge about good training practices serves as an implicit regularization mechanism for the optimization. Conclusion AutoResearch's edge comes from two things: an LLM prior that makes informed changes rather than blind samples, and access to the full code space rather than a predefined parameter vector. The first makes it sample-efficient and helps its solutions generalize. The second gives it a higher ceiling. Neither advantage is specific to LLM training. Anything with a measurable objective and editable code is a candidate, which is why the community has already applied it to prompt engineering, backend optimization, and formula discovery. We have a curated list here. --- # Weco Now Integrates with LangSmith URL: https://www.weco.ai/blog/weco-langsmith-integration Published: 2026-03-09 Category: Product Authors: Weco Team Point Weco at your LangSmith datasets and evaluators. It handles the search. LangSmith is great for measuring agent quality. But once you have the evals in place, the actual improvement loop is still mostly manual. Most teams using LangSmith already have the foundation for optimization: a dataset, a target function, and evaluators that score outputs. What still takes time is the part between evaluations. Someone reads failures, forms a hypothesis, edits a prompt or some code, runs evaluate() again, and repeats. Weco now integrates directly with LangSmith to automate that loop. Point Weco at your existing dataset and evaluators, and it iteratively searches for better code and prompt changes between evaluations while tracking the best-performing version it finds. Try it now The tutorial walks through the full flow end to end. Key Takeaways - Weco now integrates directly with LangSmith, automating the improvement loop using your existing datasets and evaluators. - No need to rebuild your eval pipeline. Weco uses the same LangSmith setup you already trust. - The integration supports mixing local Python evaluators with LangSmith dashboard evaluators and optimizing on one data split while validating on another. - Weco iteratively modifies your source code, runs evaluations, compares scores, and keeps the best-performing version. How It Works Instead of writing a separate shell eval script or rebuilding your scoring logic somewhere else, you tell Weco which LangSmith dataset to evaluate against, which function to call, and which evaluators to use. Weco then modifies your source code, runs the LangSmith evaluation, compares scores, and keeps iterating. The practical effect is that the LangSmith setup you already trust for measurement becomes the thing Weco optimizes against. You do not have to create a second evaluation pipeline just to get an optimization loop. Why It Matters Teams spend real time building evals: curating datasets, writing evaluators, and tuning judge prompts until the scores actually reflect quality. Once that work is done, the frustrating part is that improvement is still driven by manual trial and error. This integration turns that existing investment into leverage. The same LangSmith setup you use to measure regressions and compare variants can now drive automated improvement runs too. That means less time translating eval feedback into edits by hand, and more time actually exploring the search space. Key Capabilities - Use the eval stack you already have. Run Weco directly against a LangSmith dataset, target function, and evaluators instead of maintaining a separate eval command. - Combine code evaluators and LLM judges. Mix local Python evaluators with dashboard evaluators configured in LangSmith, then combine them with a custom metric function. - Optimize on one split, validate on another. Use dataset splits to optimize on an opt split and then run a separate holdout pass to check whether gains generalize. - Run from the CLI or a browser-based wizard. Power users can configure everything with flags, while new users can launch the setup wizard and wire up a run visually. - Handle async dashboard evaluators automatically. When LangSmith dashboard evaluators return scores asynchronously, Weco polls for results so you do not have to manage that part yourself. Get Started If you already have a LangSmith dataset and evaluators, you can connect Weco to that workflow today. The full tutorial walks through a complete example, from dataset setup to optimization to holdout validation. If you want something concrete to start from, use the ZephHR LangSmith example. If your target function calls an LLM provider directly, install the dependencies your agent already needs alongside Weco, then follow the tutorial to connect it to your LangSmith setup. Already using LangSmith for evals? Read the tutorial and use that eval stack to drive optimization too. Questions or help adapting it to your workflow? Reach out at contact@weco.ai. --- # How to Build a Good Eval URL: https://www.weco.ai/blog/eval-is-not-a-tool-you-install Published: 2026-02-19 Category: Research Authors: Zhengyao Jiang Eval is not a tool you install. It's a way of thinking from experimental science. Learn what makes a good benchmark and how to avoid common pitfalls. "Vibe driven development" sounds like a joke, but it's the default methodology for most AI agent teams. The workflow - change something, run the agent, eyeball the output, decide if it's better. This is an eval loop, one with a sample size of one, no reproducibility, and a loss function that lives entirely inside one person's head. Most teams know this isn't sustainable. But what comes next is where they get stuck. Some reach for an eval framework and assume the tooling is the eval. Others, having seen enough benchmarks get gamed, conclude that rigorous evaluation is a research problem too deep to be practical. Both reactions share a root cause, not having built an eval pipeline from scratch and iterated an agent or model against it. Eval is neither a tool you install nor a paper you need to publish. It's a specific way of thinking, one that comes from experimental science, not software engineering. This post is about that way of thinking. Key Takeaways - Eval is not a tool you install. It is a way of thinking rooted in experimental science, not software engineering. - An eval is a simulator that tells you whether changes made things better or worse before you deploy, unlike observability which reports what happened after. - Good benchmarks balance Fidelity (how well they predict real quality), Signal (how robustly they differentiate solutions), and Cost (money and time to run). - Maintain separate validation and test sets to prevent overfitting. Optimize against validation daily and run test rarely before releases. White Box vs Black Box In traditional software, code is a white box. You can read the logic, trace the execution, verify correctness by inspection. If x > y then do z, and you know the logic is right, the output is correct. Tests are a safety net for when your reasoning is wrong. An AI model is a black box. You feed it a prompt, it returns an answer, but the internal logic is opaque. The "code" is billions of weights you cannot read. Worse, the system is inherently probabilistic and the capability frontier is jagged. It might handle 80% of cases and fail unpredictably on the other 20%. You cannot verify correctness by reading the code, because there is no code to read. You have to measure correctness empirically, across many inputs. This is why you need an experiment, not just a unit test. This distinction matters because many engineers approach AI systems with their white box intuition intact. They write a prompt, test it on a few examples, see it working, and conclude it works. In a white box, this reasoning is valid. If the logic is right, it's right everywhere. In a black box, "works on my examples" tells you almost nothing about the other cases. What Is an Eval and Why Is It a Simulator? It's tempting to think that the third party tooling solves this. Langfuse, Braintrust, LangSmith, Promptfoo - the ecosystem is growing fast. But it's worth understanding what these tools actually are. They are observability platforms first. Their core job is monitoring your production pipeline - tracing, logging, cost tracking. Eval features (prebuilt LLM judge templates for hallucination, toxicity, relevance) are added on top. This matters because observability and evaluation are fundamentally different activities. Observability tells you what happened in production, after the fact. An eval is a simulator. It tells you whether a change made things better or worse before you deploy. You construct a proxy of the real world, run your agent against it offline, and compare scores. It can be as simple as a Python script and a CSV file. If you start from an observability platform and try to add eval, your thinking starts from "what's happening in production." If you build an eval from first principles, your thinking starts from "what does good look like, and how do I measure the distance from it." These lead to very different outcomes. The prebuilt templates can get you started if your use case is generic enough (say, a customer support chatbot). But for most agentic workflows, the real questions are ones no template answers for you. Which brings us to the key question, what makes a good eval? What Makes a Good Eval? A benchmark is your proxy for real world utility. There are three key variables to consider - Fidelity, Signal, and Cost. Fidelity - How well does the benchmark result predict the actual end to end quality you care about? Instead of just statistical correlation, think of this as "Ground Truth Alignment". In practice, you want your benchmark to be as close to the real experience as possible. Signal - The benchmark should robustly differentiate two solutions. A benchmark can be saturated (all solutions perform quite well) or have high variance (run the benchmark multiple times and you'll get different results). In both cases, the conclusion you draw from it might not be statistically significant. Cost - Running benchmarks costs money and time. It is possible to optimize for cost, but often at the expense of signal or fidelity. It is very difficult to optimize all three at the same time. For example, you can reduce the variance of the final score by running the same eval multiple times at a higher cost. A good rule of thumb - start with a cheaper evaluation with decent fidelity. As you pick the "low hanging fruit" improvements, you can gradually build a high signal benchmark to capture more nuanced signals. You should be aware of these three dimensions throughout the process. Why Are Vibes an N=1 Benchmark? A common objection to benchmarks is Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." The more you optimize against a specific benchmark, the more you risk overfitting to its quirks rather than improving actual quality. This is a real concern. But the alternative is not "no benchmark"; it's a worse benchmark. If you're optimizing against your vibes, you are running a benchmark. It just has N=1 (or maybe N=5) data points, no reproducibility, and a scoring function you can't inspect or improve. You are still overfitting, but now you can't even detect it. The solution is not to avoid metrics. It's to treat your benchmark as a living thing that needs iteration. Maintain separate validation and test sets. Optimize against validation daily. Run test rarely, before releases, to check if improvements generalize. If validation goes up but test flatlines, you're overfitting. The right response is not to start optimizing against test. It's to increase the size and diversity of your validation set. Prune saturated data points. If your agent scores 100% on certain tasks consistently, those samples are costing money without generating signal. Move them to a cheap regression suite and replace them with harder examples. Review your assumptions. If users complain but your score is high, your metric is wrong or your data is unrepresentative. Go back and fix the benchmark, not the users. Building an eval is not a one time task. It's the ongoing process of aligning your engineering loop with reality. We're studying how teams build evals for agentic workflows in practice. If you're working on this and want concrete feedback on your eval design, reach out at contact@weco.ai. Tell us what you're evaluating and how you're scoring it. --- # Evolving a Flappy Bird AI: 100 Solutions Explored, 6.6x Improvement, $12 URL: https://www.weco.ai/blog/flappy-bird-case-study Published: 2026-02-07 Category: Case Study Authors: Weco Team We gave Weco a simple Flappy Bird AI that scored 2.76 points on average. 100 iterations and $12 later, the AI scored 20.9 - a 6.6x improvement. We gave Weco a simple Flappy Bird AI that scored 2.76 points on average. 100 iterations and $12 later, the AI scored 20.9, a 6.6x improvement, with no human guidance beyond the initial setup. Here's how it works. Try the interactive demo → Key Takeaways - Weco improved a Flappy Bird AI from an average score of 2.76 to 20.9 (6.6x improvement) for approximately $12. - The optimizer autonomously explored 100 solutions, discovering strategies like trajectory prediction and look-ahead blending. - No human guidance was needed beyond providing the initial code and evaluation metric. - The search tree shows how Weco branches, backtracks, and evolves code through iterative experimentation. The Setup Flappy Bird is a perfect testbed for code optimization. The game is simple (tap to flap, dodge pipes, don't die) but writing a controller that consistently scores high is surprisingly hard. Pipe gaps vary in height, the bird's physics create non-trivial dynamics (gravity, terminal velocity, flap impulse), and a single misjudgment means game over. We wrote a baseline controller in Python: a should_flap(observation) function that takes the bird's position, velocity, and pipe locations as input and returns True or False. The baseline uses a simple strategy: compute a safe zone within the next pipe gap, flap if below it. It works, barely. Average score: 2.76 across 30 evaluation seeds. Then we pointed Weco at it. After running pip install weco && weco setup, we opened Claude Code and typed: /weco optimize the flappy bird controller The Weco agent skill takes it from there: it figures out how to evaluate the code (average score over 30 game seeds), then kicks off the optimization run. No reward shaping, no architecture search, no hyperparameter tuning. Just "here's the code, make it better." How Weco Works Weco uses evaluation-driven code optimization, a tree-search algorithm that treats code as the search space and evaluation metrics as the objective. At each step, Weco: - Selects a promising parent solution from the search tree - Generates a mutated variant using an LLM - Evaluates the variant against the metric (here, average game score over 30 seeds) - Updates the tree with the result This is different from single-shot code generation. Instead of asking an LLM to write the best controller it can in one try, Weco evolves the code over many iterations, keeping what works and discarding what doesn't. Each iteration builds on the best solution found so far. The Search Tree Over 100 iterations, Weco explored a tree of 101 solutions. Not all branches led somewhere useful. Some mutations broke the code entirely, others made the AI worse. But the successful mutations compounded. Explore the full search tree interactively → The best path through the tree passed through 11 nodes, with each breakthrough introducing a qualitatively different strategy: - Step 0 → 6 (score 3.2 → 4.1): Cleaned up the baseline, tightened margins - Step 6 → 18 (4.1 → 9.9): Added trajectory prediction, simulating the bird's future position before deciding to flap - Step 18 → 27 (9.9 → 16.9): Introduced look-ahead to the next pipe, not just the current one, blending targets based on distance - Step 27 → 84 (16.9 → 20.9): Refined adaptive margins, descent handling for downward pipe transitions, and velocity dampening Optimization progress over 100 iterations. Green dots mark breakthroughs on the best path. What Changed in the Code The baseline controller was 71 lines (mostly comments explaining game physics). The optimized controller is 82 lines of pure logic. Here are the key differences: 1. Trajectory Prediction The baseline just checks "am I below the target?", a reactive strategy. The optimized version simulates the bird's physics forward by 2–4 frames (depending on distance to the pipe) and acts on where the bird will be, not where it is now. # Predict bird position predicted_y = bird_y predicted_v = bird_velocity for _ in range(prediction_frames): predicted_v = min(predicted_v + GRAVITY, 10.0) predicted_y += predicted_v 2. Look-ahead Blending Instead of only targeting the current pipe gap, the optimized AI blends its target between the current and next pipe. When far away, it weights the next pipe more heavily (60%), preparing for the transition. When close, it focuses entirely on the current pipe. # Dynamic target blending based on distance if next_pipe_dx > 150: target_gap_cy = 0.4 * next_pipe_gap_cy + 0.6 * after_pipe_gap_cy elif next_pipe_dx > 80: blend = (next_pipe_dx - 80) / 70 target_gap_cy = (1 - blend) * next_pipe_gap_cy + blend * after_pipe_gap_cy 3. Adaptive Margins The baseline uses a fixed margin of 19 pixels. The optimized version tightens the margin when close to a pipe (12px) and widens it when far away (22px), giving the bird more room to maneuver when it has time, and precision when it needs it. 4. Descent Handling When the next pipe gap is significantly lower than the current one, the optimized AI recognizes the downward transition and adjusts its velocity dampening thresholds. This prevents the common failure mode of overshooting upward when approaching a lower gap. None of these strategies were specified in advance. Weco discovered them through iterative mutation and evaluation, the same way biological evolution discovers adaptations, but in minutes instead of millennia. Results Baseline Optimized Average score (30 seeds) 2.76 20.9 Max score (best seed) 11 88 Controller lines 71 (mostly comments) 82 (pure logic) Strategy Reactive safe-zone targeting Predictive + look-ahead + adaptive Total cost: ~$12. Total time: ~30 minutes. The optimization ran 100 iterations, each calling an LLM to generate a code variant and then evaluating it against 30 game seeds. Watch the baseline and optimized AI play side by side → Try It Yourself Weco isn't just for games. The same evaluation-driven optimization works on any code with a measurable metric: model training, data pipelines, API performance, prompt engineering. pip install weco && weco setup This installs a Weco agent skill into your coding agent (Claude Code, Cursor, or any MCP-compatible agent). Then just type /weco optimize my code and the agent walks you through everything, including building an eval if you don't have one yet. You get $20 in free credits to start, or bring your own API key. Check out the documentation to get started. --- # Weco Is Now in Public Beta URL: https://www.weco.ai/blog/beyond-trial-and-error-ml-engineering Published: 2025-11-05 Category: Product Authors: Weco Team Hard work scales linearly. Automation scales exponentially. Much of machine learning is a persistent grind of trial and error. As ML practitioners, we spend a huge portion of our time testing ideas. We try a new data augmentation, a different learning rate, a modified architecture, or a re-written prompt. We wait for the job to run, analyze the results, and start the loop all over again. But you only have a finite number of attempts. This leads to the ever lasting dilemma: - Did the current approach fail because I didn't pull the lever hard enough? - Or have I been too attached to the idea, and it's time to go back to exploration mode? Staying late or being more experienced helps, but what if being able to automatically test 10x more ideas? Key Takeaways - Weco automates the ML trial-and-error loop, testing hundreds of ideas against your evaluation metric overnight. - In a live CrunchDAO competition, Weco placed in the top 10% (34th of 490 teams) for just $20 over 17 days. - Users are applying Weco to prompt engineering, kernel optimization, network architecture tuning, and more. - Weco is the evolution of AIDE, the top-performing system on OpenAI's MLE-Bench for ML engineering tasks. How Does Weco's Automated Exploration Loop Work? Weco drives an automated loop for empirical exploration. The methodology is straightforward: - Define the Eval: You, or you plus your coding agent, write an evaluation script that measures the performance of any given solution. Could be speed, accuracy, cost, or simply any metric. - Weco Explores: The system takes over, running a continuous loop to propose new ideas, implement them against your codebase, and test them empirically using the exact evaluation script you provided. Instead of you manually trying 20 ideas over a week, the system can test hundreds overnight. Goodhart-Proof Benchmark The predecessor of Weco is AIDE, an opensource MLE Agent we developed that has been credited by OpenAI. In their independent evaluation, MLE-Bench, AIDE was shown to be the top-performing system for this class of task. This is significant. Benchmarks are most useful before they are widely known and overfitted - before Goodhart's Law kicks in. AIDE's performance on this fresh, independent benchmark demonstrated the power of our design. AIDE has since been adopted and used by researchers at institutions like Meta, DeepMind, and Sakana, effectively becoming an industry standard. Weco is the next evolution of this giving you more control, and work be plug into an existing code base, just like parallelly developed system, Google's AlphaEvolve. Thriving in a Live Competition Static benchmarks are vital, but can an AI agent compete against skilled humans in a live competition for real stakes? Especially when the solutions aren't already part of an LLM's training data? To test this, we entered Weco into the CrunchDAO ADIA structural break prediction competition: - The Stakes: A live, ongoing competition with a $100,000 reward pool, attracting hundreds of skilled data scientists. - The Setup: We provided the agent only with the competition description and our evaluation script. - The Process: The agent autonomously explored over 120 distinct solutions in 17 days, testing different feature engineering pipelines and model parameters with zero human intervention. - The Result: We placed 34th out of 490 teams (Top 10%) on the private leaderboard. The automated run that leads to this solution cost just $20. This case study highlights the new role of the ML engineer. Our time was spent refining the evaluation script to provide a clear signal, not manually trying a hundred ideas. You can explore the full dashboard for this run here to see every idea the agent tried. What Real-World Impact Has Weco Delivered? This methodology isn't limited to one domain. We're seeing early users apply it across the entire ML stack to get tangible results. - Prompt Engineering for VLMs An astrophysicist used Weco to systematically optimize prompts for a Vision Language Model (VLM), improving the model's accuracy by 20%. - Building Robust Agentic Scaffolds Startups are using the platform to optimize the accuracy and cost of their AI features, automatically finding the right balance between different models, prompts, and logical flows. - Model Inference Speedup A researcher at a frontier AI lab used Weco for automated kernel engineering. By letting the agent explore the solution space, they made a new operator run 7x faster than their pytorch baseline. - And More... The same loop is being applied to tweaking network architectures, optimizing complex rule-based image processing pipelines, and more. How Do You Get Started with Weco? Stop losing time to the manual grind. Start systematically exploring your idea space and find out what works. Get started in 5 minutes with the Hello World example. --- # Weco Raises $8M to Build Self-Evolving Software URL: https://www.weco.ai/blog/seed-announcement Published: 2025-07-30 Category: Announcement Authors: Zhengyao Jiang, Yuxiang Wu Most AI tools can draft code that runs. None can deliver expert‑grade solutions that push the frontier. Most AI tools can draft code that runs but can't deliver expert‑grade solutions that push the frontier. Today’s code‑generation tools do little more than prompt large language models to emit code pulled from their training data, remixing patterns they’ve already seen. They rarely discover the sharper, faster, or more reliable solutions that push past the current state-of-the art. Weco’s mission is to automate the scientific method so software can learn from its own experiments, improvingwith every iteration. Instead of settling for good enough vibes, our agent generates expert‑grade code: useful, reliable, and even groundbreaking. Key Takeaways - Weco raised $8M in seed funding to build software that improves itself through evaluation-driven coding. - AIDE, Weco's open-source agent, achieved a five-fold jump in medal rate on OpenAI's MLE-Bench and outperformed top human researchers on RE-Bench. - Evaluation-driven coding treats each code change as a hypothesis, testing it against metrics and iterating until performance peaks. - Weco is building toward "Software 2.0" where programs continuously learn and adapt based on empirical feedback. What Is AIDE and How Is It Like AlphaEvolve? Google DeepMind's AlphaEvolve presents a timely demonstration of the power of AI-driven code optimization: using evolutionary search, it uncovered a breakthrough matrix‑multiplication algorithm, and shaved ~1% off Google's already highly‑tuned cloud compute bill. Weco takes this test-time self-improvement magic and puts it in the hands of every engineer, without requiring a hyperscaler budget. Since last year, AIDE - Weco's open‑source engine - has delivered milestone results at the frontier: - On OpenAI's MLE‑Bench, wrapping AIDE around the o1‑preview model drives a five‑fold jump in medal rate and a four‑fold gain over the best coding agents - On METR's RE‑Bench, AIDE became the first autonomous system to out-perform top human researchers from DeepMind, Google, Anthropic, and OpenAI on kernel‑engineering tasks - Initiatives such as Sakana AI's AI Scientist‑v2 plus Meta AI's LLM Speed‑runner and AIRA‑Dojo now rely on AIDE's tree‑search loop to explore vast solution spaces with minimal human input As Tim Rocktäschel (Director & Open‑Endedness Team Lead, Google DeepMind) observes: "Fantastic example of an open‑ended self‑improving outer loop (AIDE), leading to massive capability gains of a strong inner loop (o1). The future is going to be wild!" From Kaggle leaderboards to open‑ended scientific discovery, these endorsements show that AIDE is a key foundation of the most important development in AI: recursive self-improving superintelligence. But we're just getting started - we're packaging these same capabilities for every engineer and organisation that wants software that proves its worth metric by metric, experiment by experiment. Announcing our $8M Seed Round Today, we're excited to announce that Weco has raised $8 million in seed funding. The latest round was led by Golden Ventures, with participation from Third Kind Venture Capital, BoxGroup, Founder Collective, Night Capital, Twin Path Ventures, Koro Capital, and Scott Belsky. The best software of tomorrow won't be fixed at launch - it will relentlessly evolve. As Nick Chen, Partner at Golden Ventures puts it: "Weco is pioneering 'Software 2.0': codebases as living systems that learn, adapt, and outperform themselves autonomously. We're only at the beginning of what's possible." Our partners share deep conviction in a powerful idea - the next era of software won't be written once and remain static. It will continuously learn, measure, and refine itself. We're excited to advance our research in recursive self‑improvement, expand a talent‑dense R&D team, and make expert‑grade, evidence‑backed code optimization available to every builder. What Is Evaluation-Driven Coding? Picture a development loop where each code change is treated as a hypothesis, automatically tested against real metrics, evolving into something better - again and again. We call this cycle “evaluation-driven coding”, and it sits at the heart of Weco. Instead of “vibe coding,” where an LLM guesses a plausible snippet and humans polish the rest, our approach turns empirical feedback into the steering wheel: propose, run, measure, repeat until the numbers peak. The long‑term payoff is profound: when optimization becomes autonomous and evidence‑driven, engineers reclaim their time for higher‑level design, research teams explore orders of magnitude more ideas, and organisations ship software that keeps upgrading itself long after release. We see a future where every repository, open-source script and mission-critical system has an always‑on optimizer quietly chasing the next jump in accuracy, speed, or robustness, cutting the cost of innovation to near zero. Join the Build Loop We’re hiring exceptional backend, infra, and research engineers to help turn evaluation‑driven development into the default way software is made - apply at jobs@weco.ai. Thank you to our investors, early users, and the wider research community for proving that code can, in fact, learn to outdo itself. Together we’ll keep pushing that frontier forward! --- # AIDE: Human-Level Performance on Data Science Competitions URL: https://www.weco.ai/blog/technical-report Published: 2024-04-04 Category: Research Authors: Dominik Schmidt, Zhengyao Jiang, Yuxiang Wu In the world of data science, Kaggle competitions have become a widely accepted standard... In the world of data science, Kaggle competitions have become a widely accepted standard for evaluating the skills and capabilities of data scientists and machine learning engineers. These competitions provide a platform for practitioners to showcase their ability to solve complex real-world problems, work with diverse datasets, and develop high-performing machine learning solutions. Today, we're thrilled to announce a significant milestone: our AI-powered data science agent, AIDE, has achieved human-level performance on Kaggle competitions. On average, AIDE outperforms half of human contestants. What sets AIDE apart is its ability to autonomously understand competition requirements, design and implement solutions, and generate submission files, all without any human intervention. Moreover, AIDE surpasses the performance of a conventional AutoML system (H2O), as well as the LangChain Agent and ChatGPT (with human assistance). Weco provides a cloud-hosted version of AIDE. We also understand that expert users may want to run it locally to leverage their own computing resources, ensure better data privacy, and achieve more customization. Therefore, AIDE will be open-sourced in the coming weeks at https://github.com/WecoAI/aideml. This repository currently showcases AIDE's solutions to over 60 Kaggle competitions. Key Takeaways - AIDE has achieved human-level performance on Kaggle competitions, outperforming approximately 50% of human contestants on average. - AIDE uses Solution Space Tree Search: it generates solutions, evaluates them by running them, and iteratively selects and refines the most promising ones. - AIDE surpasses both conventional AutoML systems (H2O) and ChatGPT with human assistance across a benchmark of 60+ Kaggle competitions. - Most data science tasks can be solved at an LLM inference cost of under $1 per task using GPT-4 Turbo. How Does AIDE Work Under the Hood? AIDE's problem-solving approach is inspired by how human data scientists tackle challenges. It starts by generating a set of initial solution drafts and then iteratively refines and improves them based on performance feedback. This process is driven by a technique we call Solution Space Tree Search. At its core, Solution Space Tree Search consists of three main components: - Solution Generator: This component proposes new solutions by either creating novel drafts or making changes to existing solutions, such as fixing bugs or introducing improvements. - Evaluator: The evaluator assesses the quality of each proposed solution by running it and comparing its performance against the objective. - Solution Selector: The solution selector picks the most promising solution from the explored options to serve as the starting point for the next iteration of refinement. By repeatedly applying these steps, AIDE navigates the vast space of possible solutions, progressively refining its approach until it converges on the optimal solution for the given data science problem. This iterative, feedback-driven approach mirrors the way human data scientists work, combining the systematic rigor of algorithms with the adaptive creativity of human problem-solving. The result is a powerful tool that can tackle complex data science challenges with unprecedented efficiency and effectiveness. Examples Let's explore AIDE's capabilities through practical examples that showcase how it tackles various data science challenges. Experience its problem-solving abilities firsthand by visiting our interactive demo. How Was AIDE Benchmarked Against Human Competitors? To truly understand AIDE's capabilities, we put it through its paces with a rigorous benchmarking process. Our benchmark suite comprises over 60 data science competitions. While the majority of our evaluation focuses on tabular data tasks, we also included a set of deep learning tasks to assess AIDE's ability to handle messy datasets, efficiently utilize GPU resources, and fine-tune open-source language models. Each task in our benchmark involves: - Task Description: We provide AIDE with the same textual competition description that human data scientists would receive, ensuring a fair comparison. - Dataset: Each task comes with its own dataset, including an unlabeled test set for AIDE to make predictions on, just like in real-world scenarios. - Evaluation Function: We measure AIDE's performance with custom evaluation functions that assess the quality of its solutions and compare them against a wide range of submissions from human data scientists. By subjecting AIDE to this comprehensive benchmarking protocol, we gain deep insights into its strengths, limitations, and potential for real-world applications. What Were the Results? We now present the results of our benchmark. All results are provided on a human-normalized scale, meaning a performance of "50%" corresponds to the agent outperforming approximately 50% of human data scientists according to the leadboard of a competition. General performance In the figure below, we illustrate the performance distribution of AIDE across various tasks in our benchmark. AIDE's performance significantly varies with the task. In some tasks, AIDE's performance matches that of top-level experts, while in a small fraction of the tasks, the agent's solutions are relatively basic and cannot rival those crafted by humans. Comparison to baselines Next, we compare AIDE to two baselines: a popular industrial-level AutoML system called H2O, as well as the use of ChatGPT (interactive, with user guidance). The following result is obtained from a smaller subset of 16 tabular benchmark tasks that are solvable via both H2O and ChatGPT, and it shows that AIDE outperforms both baselines by a significant margin. Notably, unlike AIDE, which requires no human intervention, both AutoML and ChatGPT need human supervision. For AutoML, users need to read the task description and set up the configuration, and after the run is finished, users should post-edit the submission file to make it compatible with the format required by the competition. As for ChatGPT, users will have to provide ad-hoc instructions and, on average, need 3 rounds of interactions to get the result. Versatility AIDE can solve a wide range of tasks by leveraging a diverse set of existing libraries and tools. The figure below shows the distribution of Python libraries used by AIDE across a large set of (primarily) tabular tasks. Cost We also highlight the extraordinary cost-efficiency of AIDE. As shown below, AIDE is able to solve most tasks at an LLM inference cost of under $1 per task when relying on gpt-4-turbo as the LLM backend. Try out AIDE! Want to experience the power of AIDE for yourself? We've launched a cloud-hosted version. Get in touch at contact@weco.ai. We believe open-sourcing AIDE is the best way to make it truly valuable for advanced users. We're excited to announce that AIDE will be open-sourced later this month! Check it out at https://github.com/WecoAI/aideml. --- # The Future of Machine Learning Research URL: https://www.weco.ai/blog/intro Published: 2024-01-22 Category: Research Authors: Zhengyao Jiang, Yuxiang Wu Research is a cornerstone in the quest to understand the world and tap into its economic values... Research is a cornerstone in the quest to understand the world and tap into its economic values. At the heart of research lies the scientific method, a robust approach that hinges on empirical evidence and structured processes. This method involves hypothesizing, experimenting, observing, and drawing conclusions. It's the bedrock of acquiring knowledge that's both reliable and grounded in observable, systematic data. At Weco AI, we are harnessing the power of Large Language Models (LLMs)-driven Agents to automate Machine Learning (ML) and Data Science research. Given the structured nature of scientific methods, automation is not only feasible but also a logical progression. LLMs have already demonstrated their capability for basic reasoning and decision making. While ML research can be challenging, much of it involves empirical trial and error, which is a good starting point for automation. The benefits of research agents are manifold. It is fast, leveraging the inherent speed of LLMs in coding and reasoning, which outpaces human capabilities due to fundamental hardware advantages. It's cost-effective, democratizing research by lowering the traditional barriers of entry, such as hefty R&D costs that are often exclusive to large companies and academic institutions. It is scalable, allowing for the simultaneous exploration of multiple facets of a problem, using multiple agents. This scalability enables the conversion of resources, like computing power, into knowledge. Key Takeaways - Weco AI uses LLM-driven agents to automate ML research by following the scientific method: hypothesizing, experimenting, and drawing conclusions. - Research agents are faster, cheaper, and more scalable than manual ML experimentation. - The agents can explore entire solution spaces simultaneously, converting compute resources into knowledge. - This approach has already demonstrated strong results in automated data science competitions. How Can You Automate Machine Learning Research? Our focus on automating machine learning (ML) research is strategic. The field of Data Science and ML is rapidly expanding, and our team possesses extensive experience in ML research. We're channeling this expertise into our AI systems, with the potential to recursively improve AI as a discipline. This improvement is not just about enhancing capabilities; it is also about fostering a deeper understanding that could lead to a safer future with Artificial General Intelligence (AGI). Our LLM agents offer concrete benefits. They interact with users in natural language, which is more intuitive and flexible compared to conventional AutoML systems that require a learning curve to understand APIs. The design space is essentially the code space, meaning our agents can perform any task a human data scientist can, unlike conventional AutoML that operates within predefined modules or hyperparameters. Introducing the First ML Agent: AIDE α Our product, AIDE, currently excels at handling tabular or time series data science tasks. Users can interact with it through natural language prompts, making it intuitive and flexible tool for data scientists, machine learners, and even non-technicals. The workflow features include automatic debugging, evaluation, and iterative improvement based on previous experiments. Unlike conventional AutoML that only delivers an outcome model, our agent provides structured knowledge in the form of code and reports. We are now onboarding alpha users. Click here to join the waiting list! What Does the Future of Automated ML Research Look Like? Our roadmap includes tackling GPU-intensive tasks, like deep learning and AI, traditionally challenging for AutoML systems due to the complexity and unstructured nature of data representations, such as images, videos, or text. The resource intensity of these tasks has been a barrier, but we're confident that LLM-driven agents can overcome these challenges. Our ultimate goal is to achieve human expert-level performance, developing creative agents capable of novel designs and learning from cutting-edge scientific literature. We believe that AGI is not the final destination. Our research agents will continually push the boundaries of human knowledge, driving technological innovation at an unprecedented pace. Join us on this journey to build the future of research! ---