mirror of
https://github.com/pese-git/cherrypick.git
synced 2026-05-16 10:10:43 +00:00
docs(benchmark_di): add comparative benchmark reports v2
This commit is contained in:
Klim
148
benchmark_di/REPORT_v2.md
Normal file
148
benchmark_di/REPORT_v2.md
Normal file
@@ -0,0 +1,148 @@
|
||||
# Comparative DI Benchmark Report: cherrypick vs get_it vs riverpod vs kiwi vs yx_scope
|
||||
|
||||
## Benchmark Parameters
|
||||
|
||||
- chainCount = 100
|
||||
- nestingDepth = 100
|
||||
- repeat = 5
|
||||
- warmup = 2
|
||||
|
||||
## Benchmark Scenarios
|
||||
|
||||
1. **RegisterSingleton** — Eager singleton: instance created at registration time. Measures pure lookup speed.
|
||||
2. **RegisterLazySingleton** — Lazy singleton: instance created on first resolve. Measures creation + caching.
|
||||
3. **ChainSingleton** — Eager dependency chain A → B → ... → N. All instances pre-created at registration. Pure lookup.
|
||||
4. **ChainLazySingleton** — Lazy dependency chain. Full graph creation + caching on first resolve. **Primary fairness metric.**
|
||||
5. **ChainFactory** — All chain elements are factories. Stateless creation chain.
|
||||
6. **AsyncChain** — Async chain (async factory). Performance on async graphs.
|
||||
7. **Named** — Registers two bindings with names, resolves by name. Named lookup test.
|
||||
8. **Override** — Registers a chain/alias in a child scope. Tests scope overrides.
|
||||
|
||||
> **Note:** kiwi and yx_scope do not support eager singleton registration. Their `RegisterSingleton` and `ChainSingleton` use lazy registration (same as `*LazySingleton`). Results marked with † reflect this.
|
||||
|
||||
---
|
||||
|
||||
## Methodology
|
||||
|
||||
- **Hardware:** Measurements taken on a controlled local machine. Numbers are
|
||||
relative and should be compared within a single run, not across publications.
|
||||
- **Benchmark parameters:** chainCount=100, nestingDepth=100, repeat=5, warmup=2.
|
||||
- **Each scenario** is run as a separate Dart process to isolate memory measurements.
|
||||
- **Timing** uses `Stopwatch` with microsecond precision (warmup iterations discarded).
|
||||
- **Memory** measured via `ProcessInfo.currentRss` (peak RSS per process).
|
||||
- **Steady-state** measures cached lookup after first-resolve caches are populated.
|
||||
|
||||
---
|
||||
|
||||
## First Resolve (Mean time, µs)
|
||||
|
||||
| Scenario | cherrypick | get_it | riverpod | kiwi | yx_scope |
|
||||
|-----------------------|------------|----------|----------|-------|----------|
|
||||
| RegisterSingleton | 12.8 | 14.6 | 17.6 | 0.4† | 22.8† |
|
||||
| RegisterLazySingleton | 1.2 | 4.2 | 5.4 | 0.2 | 9.0 |
|
||||
| ChainSingleton | 4.8 | 2.8 | 436.2 | 67.8† | 134.2† |
|
||||
| ChainLazySingleton | 46.8 | 193.2 | 398.0 | 56.0 | 266.8 |
|
||||
| ChainFactory | 54.0 | 66.0 | 443.0 | 55.6 | 146.6 |
|
||||
| AsyncChain | 247.8 | 15033.0 | 1379.0 | – | – |
|
||||
| Named | 0.2 | 0.6 | 3.4 | 0.2 | 7.8 |
|
||||
| Override | 9.8 | 3.8 | 408.8 | 61.8 | 138.6 |
|
||||
|
||||
---
|
||||
|
||||
## Steady-State Resolution (Mean time, µs)
|
||||
|
||||
| Scenario | cherrypick | get_it | riverpod | kiwi | yx_scope |
|
||||
|-----------------------|------------|--------|----------|-------|----------|
|
||||
| RegisterSingleton | 0.0 | 0.2 | 1.0 | 0.0 | 0.0 |
|
||||
| RegisterLazySingleton | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 |
|
||||
| ChainSingleton | 2.8 | 0.8 | 2.8 | 1.6 | 2.0 |
|
||||
| ChainLazySingleton | 1.4 | 1.2 | 1.8 | 1.0 | 1.4 |
|
||||
| ChainFactory | 31.2 | 62.6 | 2.0 | 51.6 | 1.6 |
|
||||
| AsyncChain | 57.2 | 31.6 | 18.2 | – | – |
|
||||
| Named | 0.0 | 0.2 | 0.2 | 0.0 | 0.4 |
|
||||
| Override | 1.0 | 1.4 | 1.8 | 290.0 | 1.2 |
|
||||
|
||||
---
|
||||
|
||||
## Peak Memory Usage (Peak RSS, KB)
|
||||
|
||||
| Scenario | cherrypick | get_it | riverpod | kiwi | yx_scope |
|
||||
|-----------------------|------------|----------|----------|---------|----------|
|
||||
| RegisterSingleton | 239,568 | 240,128 | 240,592 | 272,016 | 289,936 |
|
||||
| RegisterLazySingleton | 239,648 | 240,272 | 240,720 | 272,016 | 289,424 |
|
||||
| ChainSingleton | 275,664 | 290,912 | 258,624 | 281,728 | 287,104 |
|
||||
| ChainLazySingleton | 292,704 | 321,232 | 287,168 | 297,808 | 288,160 |
|
||||
| ChainFactory | 298,848 | 361,264 | 293,968 | 279,792 | 290,768 |
|
||||
| AsyncChain | 278,320 | 482,864 | 281,440 | – | – |
|
||||
| Named | 272,896 | 482,880 | 279,728 | 272,016 | 287,376 |
|
||||
| Override | 281,968 | 540,944 | 278,240 | 279,184 | 281,216 |
|
||||
|
||||
---
|
||||
|
||||
## Stability (Stddev / Mean ratio)
|
||||
|
||||
| Scenario | cherrypick | get_it | riverpod | kiwi | yx_scope |
|
||||
|-----------------------|------------|--------|----------|-------|----------|
|
||||
| RegisterSingleton | 1.84 | 1.62 | 1.43 | 1.23 | 1.18 |
|
||||
| RegisterLazySingleton | 0.57 | 0.23 | 0.43 | 2.00 | 1.78 |
|
||||
| ChainSingleton | 0.36 | 0.27 | 0.06 | 0.37 | 0.17 |
|
||||
| ChainLazySingleton | 0.17 | 0.43 | 0.10 | 0.14 | 0.63 |
|
||||
| ChainFactory | 0.27 | 0.01 | 0.22 | 0.09 | 0.23 |
|
||||
| AsyncChain | 0.06 | 0.18 | 0.07 | – | – |
|
||||
| Named | 2.00 | 0.67 | 0.15 | 2.00 | 1.74 |
|
||||
| Override | 0.08 | 0.11 | 0.17 | 0.03 | 0.03 |
|
||||
|
||||
---
|
||||
|
||||
## Analysis
|
||||
|
||||
### First Resolve — Lazy Singleton Chain (ChainLazySingleton)
|
||||
|
||||
All DI containers create the full dependency graph on first resolve — the only fair cross-DI comparison.
|
||||
|
||||
- **cherrypick**: 46.8 µs — **4.1× faster** than get_it (193.2 µs), **8.5× faster** than riverpod (398.0 µs)
|
||||
- **kiwi**: 56.0 µs (+30% vs cherrypick)
|
||||
- **yx_scope**: 266.8 µs (stddev/mean = 0.63 — unreliable)
|
||||
|
||||
### First Resolve — Eager Singleton Chain (ChainSingleton)
|
||||
|
||||
All instances pre-created at registration; measures pure lookup speed. Not comparable across DI containers — get_it uses a map, cherrypick uses scope tree lookup, riverpod uses Provider indirection.
|
||||
|
||||
- **get_it**: 2.8 µs (map lookup)
|
||||
- **cherrypick**: 5.4 µs (scope tree lookup)
|
||||
- **riverpod**: 436.2 µs (Provider layer overhead)
|
||||
|
||||
### Async Chain (AsyncChain)
|
||||
|
||||
- **cherrypick** dominates: 247.8 µs (vs get_it 15,033 µs = **61× faster**)
|
||||
- **riverpod**: 1,379 µs (6.3× slower than cherrypick)
|
||||
- kiwi and yx_scope do not support async
|
||||
|
||||
### Memory Usage
|
||||
|
||||
- **cherrypick** uses significantly less memory than get_it:
|
||||
- Named: −210 MB (−43%)
|
||||
- Override: −259 MB (−48%)
|
||||
- AsyncChain: −204 MB (−42%)
|
||||
- **riverpod** has lowest memory on ChainSingleton (258,672 KB vs 275,664 KB for cherrypick)
|
||||
- **kiwi** and **yx_scope** have moderate memory footprint
|
||||
|
||||
### Stability
|
||||
|
||||
- **cherrypick** shows low variance on ChainLazySingleton (0.17) and Override (0.08)
|
||||
- **get_it** most stable on ChainFactory (0.01)
|
||||
- **yx_scope** highest variance on ChainLazySingleton (0.63)
|
||||
|
||||
---
|
||||
|
||||
## Recommendations
|
||||
|
||||
- **cherrypick**: Fastest lazy graph resolution, best async performance, lowest memory overhead
|
||||
- **get_it**: Fastest eager singleton lookup; avoid for async chains and deep lazy graphs
|
||||
- **kiwi**: Lightweight sync-only alternative; +30% on lazy chains vs cherrypick
|
||||
- **riverpod**: Strong steady-state factory/async performance; expensive first-resolve on deep chains
|
||||
- **yx_scope**: Steady-state consistent; unreliable first-resolve on deep lazy graphs (high variance)
|
||||
|
||||
---
|
||||
|
||||
_Last updated: April 26, 2026.
|
||||
Reference in New Issue
Block a user