Human vs bot: we measured prediction accuracy over 1,934 Polymarket 5-minute windows
After our trading bot's ML model kept scoring a coin flip with real money, we built a measurement rig instead of arguing about it: freeze a prediction 30 seconds before every 5-minute BTC/ETH window closes, from every signal we had — plus a human — and score everything against the actual outcome. No money involved. One week later we had 1,934 scored windows and one uncomfortable result.
Methodology
- Markets: Polymarket "BTC/ETH Up or Down" 5-minute windows, June 1–7, 2026. 1,938 windows monitored, 1,934 resolved and scored.
- Freeze rule: each signal's direction (Up/Down) is locked 30 seconds before window close and compared against the resolved outcome. No lookahead, no revision.
- Signals: (1) momentum across 30s/1m/5m timeframes; (2) mean-reversion with RSI; (3) Polymarket orderbook imbalance (bid/ask depth); (4) a logistic-regression ML model trained on ~500 real bets with 30 features; (5) an aggregate "lean" of the above; and (6) a human clicking a direction manually whenever they had a view (n=83 picks).
- Scoring: straight directional accuracy. A coin flip scores ~50%.
Results
| Signal | Correct | Accuracy |
|---|---|---|
| ML model (logistic regression) | 148 / 308 | 48.1% |
| Momentum (multi-timeframe) | 895 / 1834 | 48.8% |
| Mean-reversion (RSI) | 744 / 1500 | 49.6% |
| Aggregate lean | 698 / 1401 | 49.8% |
| Orderbook imbalance | 548 / 1069 | 51.3% |
| Human (manual picks) | 49 / 83 | 59.0% |
What this does and doesn't mean
Every automated signal we tested is statistically indistinguishable from a coin flip on 5-minute crypto direction. That includes the ML model that had shown 57% accuracy in cross-validation during training — a textbook case of regime non-stationarity: by the time you've trained on a regime, it's gone.
The human number deserves honesty in both directions. 59% over 83 picks gives p ≈ 0.05 against the coin-flip null — suggestive, not conclusive. It's one person, one week, self-selected moments (the human only picked when they felt they saw something, mostly during violent moves — exactly the regimes where our bot's stop-losses bled). It is not proof that "humans beat bots." It is a measured, honest data point that the only above-chance directional signal in our entire project came from discretionary human pattern-reading, not from any of our automation.
Consistent with it: during that same week, the same human's discretionary manual trading on the same account netted roughly +$830, while the bot's best fully-automated run had made +$10 before an infrastructure bug ate it (full audit here).
Why we ship the measurement rig with the bot
This experiment cost nothing and settled an argument that months of live trading couldn't. That's why the monitor panel is part of Uruguabot: before you trust any signal — ours, yours, or one you bought — you can freeze its predictions and measure it against reality for free. If a signal can't beat a coin flip in the monitor, it will not beat one with your money.
One week is one market regime. Measure longer than feels necessary, and never trade money you cannot afford to lose.