Human vs bot: we measured prediction accuracy over 1,934 Polymarket 5-minute windows
After our trading bot's ML model kept scoring a coin flip with real money, we built a measurement rig instead of arguing about it: freeze a prediction 30 seconds before every 5-minute BTC/ETH window closes, from every signal we had — plus a human — and score everything against the actual outcome. No money involved. One week later we had 1,934 scored windows and one uncomfortable result.
Methodology
- Markets: Polymarket "BTC/ETH Up or Down" 5-minute windows, June 1–7, 2026. 1,938 windows monitored, 1,934 resolved and scored.
- Freeze rule: each signal's direction (Up/Down) is locked 30 seconds before window close and compared against the resolved outcome. No lookahead, no revision.
- Signals: (1) momentum across 30s/1m/5m timeframes; (2) mean-reversion with RSI; (3) Polymarket orderbook imbalance (bid/ask depth); (4) a logistic-regression ML model trained on ~500 real bets with 30 features; (5) an aggregate "lean" of the above; and (6) a human clicking a direction manually whenever they had a view (n=83 picks).
- Scoring: straight directional accuracy. A coin flip scores ~50%.
Results
| Signal | Correct | Accuracy |
|---|---|---|
| ML model (logistic regression) | 148 / 308 | 48.1% |
| Momentum (multi-timeframe) | 895 / 1834 | 48.8% |
| Mean-reversion (RSI) | 744 / 1500 | 49.6% |
| Aggregate lean | 698 / 1401 | 49.8% |
| Orderbook imbalance | 548 / 1069 | 51.3% |
| Human (manual picks) | 49 / 83 | 59.0% |
What this does and doesn't mean
Every automated signal we tested is statistically indistinguishable from a coin flip on 5-minute crypto direction. That includes the ML model that had shown 57% accuracy in cross-validation during training — a textbook case of regime non-stationarity: by the time you've trained on a regime, it's gone.
The human number deserves honesty in both directions. 59% over 83 picks gives p ≈ 0.05 against the coin-flip null — suggestive, not conclusive. It's one person, one week, self-selected moments (the human only picked when they felt they saw something, mostly during violent moves — exactly the regimes where our bot's stop-losses bled). It is not proof that "humans beat bots." It is a measured, honest data point that the only above-chance directional signal in our entire project came from discretionary human pattern-reading, not from any of our automation.
Consistent with it: during that same week, the same human's discretionary manual trading on the same account netted roughly +$830, while the bot's best fully-automated run had made +$10 before an infrastructure bug ate it (full audit here).
Why we ship the measurement rig with the bot
This experiment cost nothing and settled an argument that months of live trading couldn't. That's why the monitor panel is part of Uruguabot: before you trust any signal — ours, yours, or one you bought — you can freeze its predictions and measure it against reality for free. If a signal can't beat a coin flip in the monitor, it will not beat one with your money.
You can run this same test on your own picks, free
The panel that produced every number above is part of Uruguabot — and so is the bot it was measuring. Point it at live markets with no wallet attached and it will score your calls the same way it scored ours. Full Python source, one payment.
- The monitor panel — freezes each signal’s prediction before the window opens and scores it against the real outcome, including your own manual picks. It is read-only and never touches a wallet.
- The four signals scored above, plus the multi-timeframe consensus engine that combines them.
- The ML gate, shipped untrained and switched off — because 57% in training and 48–50% live is exactly what this article measured. You train it on your own regime or you leave it alone.
- The execution layer and the calibrated dry-run simulator, so a signal that survives measurement can be traded without paying our tuition again.
- 21 Python files, bilingual docs, no subscription, no telemetry.
Read honestly, this article is an argument for measuring before trusting — and our own signals are the ones it found wanting. That is the point of shipping the rig: you can reach the same verdict about any signal, ours included, before a dollar is involved. You are buying the measurement and the plumbing. Finding an edge is still your job, and it may not be there — our own best live configuration finished at +7.3%, and that is not a track record.
One week is one market regime. Measure longer than feels necessary, and never trade money you cannot afford to lose.