1. Look-ahead bias
The strategy "knows" something before it could have known it: signals computed on the full day's data but traded at the open, indicators using a candle's close while entering inside the same candle, or settlement references fetched from a different (faster) feed than the venue actually uses.
Our control: every input carries an explicit availability timestamp, and the replay engine refuses to read data stamped after the simulated decision time. Settlement always uses the venue's own reference feed.
2. Survivorship bias
Testing on today's top-100 coins means testing on the winners. The coins that collapsed, were delisted, or got hacked quietly vanish from the sample — and with them, most of the catastrophic losses your strategy would have eaten in real time.
Our control: universes are reconstructed as-of each historical date, including assets that later died. If point-in-time membership can't be established, the result is labelled accordingly.
3. Fee fantasy
Many published backtests assume zero fees, zero spread, and perfect fills at mid-price. On high-turnover crypto strategies, realistic costs routinely turn a "60% annual return" into a loss. The faster the strategy, the bigger the lie.
Our control: every backtest report states its fee schedule, spread model, and fill assumptions next to the headline number — and includes a sensitivity row showing what happens when costs double. If a strategy dies when fees double, you should know that before risking anything.
4. Regime overfitting
A strategy tuned on 2024–2025 data is, silently, a bet that the future resembles 2024–2025. Crypto regimes shift violently: volatility compresses, funding flips, market microstructure changes when fee schedules or tick sizes change.
Our control: we report performance split by volatility regime and by calendar period, not just one aggregate curve. A strategy that only worked in one regime is presented as exactly that.
5. Multiple testing
Test 200 parameter combinations and the best one will look brilliant by pure chance. Most "discovered" edges are this — the survivor of a silent search the reader never sees.
Our control: we log how many variants were tried and report it. Out-of-sample windows are held back before the search starts, and an edge that only exists in the searched window is treated as noise.
6. Capacity illusions
A signal that earns 2% per trade on $100 positions usually cannot earn it on $10,000 positions — the order book simply isn't deep enough, and your own order moves the price. Backtests that ignore book depth scale fictional profits linearly.
Our control: replays consume recorded order-book depth, so simulated fills degrade the way real ones do. Reports state the size at which the result was simulated.
The meta-rule
Every AetherForge research report carries the same three boxes: assumptions (what we took as given), data window (what history it saw), and what would falsify this (what future evidence should change your mind). If a result can't fill in those boxes, it doesn't ship.
Read more
See the data foundation these controls rest on in Building crypto data pipelines you can actually trust, or join the waitlist to get full research as it ships.