docs(benchmark_di): add comparative benchmark reports v2

This commit is contained in:
Klim
2026-04-24 22:08:42 +03:00
parent 4077ed8469
commit c413dfed20
3 changed files with 433 additions and 0 deletions

View File

@@ -0,0 +1,137 @@
# Benchmark Comparison: cherrypick Performance Improvements
## Parameters
- chainCount = 100
- nestingDepth = 100
- repeat = 5
- warmup = 2
---
## Results: firstResolve (Mean, µs)
### Lazy Singleton Scenarios (fair cross-DI comparison)
| Scenario | BR-improvements | develop | Improvement (vs develop) |
|-----------------------|-----------------|---------|--------------------------|
| ChainLazySingleton | 46.80 | 114.20 | **2.4× faster** |
| ChainFactory | 54.00 | 105.60 | **2.0× faster** |
| AsyncChain | 247.80 | 959.20 | **3.9× faster** |
| Named | 0.20 | 3.60 | **18× faster** |
| Override | 9.80 | 110.40 | **11.3× faster** |
| RegisterLazySingleton | 1.20 | 19.60 | **16.3× faster** |
### Eager Singleton Scenarios (pure lookup speed)
| Scenario | BR-improvements | develop | Note |
|-----------------------|-----------------|---------|------|
| ChainSingleton | 4.80 | 114.20 | Not comparable — see note below |
| RegisterSingleton | 12.80 | 19.60 | **1.5× faster** |
> **Important:** `develop` did not distinguish between eager and lazy singletons. Its "singleton" was always lazy (instance created on first resolve). The develop `ChainSingleton` value (114.2 µs) measures the same thing as BR-improvements `ChainLazySingleton` (46.8 µs). The `ChainSingleton` eager column in BR-improvements (4.8 µs) measures pure lookup after `.toInstance()` eager registration — a fundamentally different operation. Fair comparison: 114.2 → 46.8 = 2.4× faster.
\* All measurements taken on the same hardware with identical benchmark parameters.
---
## Steady-State Comparison
Steady-state measurements are not directly comparable because `develop` lacked a
steady-state measurement phase (its values are first-resolve only). BR-improvements
introduced a separate steady-state phase to isolate cached lookup performance.
---
## Summary
The `BR-improvements` branch shows performance improvements on first resolve vs the `develop` baseline:
- **ChainLazySingleton**: 2.4× faster (46.8 µs vs 114.2 µs)
- **ChainFactory**: 2.0× faster (54.0 µs vs 105.6 µs)
- **AsyncChain**: 3.9× faster (247.8 µs vs 959.2 µs)
- **Named**: 18× faster (0.2 µs vs 3.6 µs)
- **Override**: 11.3× faster (9.8 µs vs 110.4 µs)
- **RegisterLazySingleton**: 16.3× faster (1.2 µs vs 19.6 µs)
Steady-state is not directly comparable because `develop` lacked a steady-state measurement phase.
### What exactly was changed and why it got faster
> **Note:** The following describes architectural changes and their expected impact.
> Specific percentage contributions have not been measured via isolated A/B tests.
---
#### 1. Split monolithic resolvers into type-specialized classes
**What changed:** The single `ProviderResolver<T>` class (which handled sync/async/param/ no-param in one place with runtime type checks) was split into four dedicated classes:
- `SyncProviderResolver<T>` / `SyncProviderWithParamsResolver<T>`
- `AsyncProviderResolver<T>` / `AsyncProviderWithParamsResolver<T>`
`ProviderResolver.create()` uses fast-path `is` checks to pick the correct implementation immediately, without calling the provider. A lazy fallback `FutureOrProviderResolver` is used only when the static type is genuinely unknown.
Similarly, `InstanceResolver` was split into `SyncInstanceResolver` and `AsyncInstanceResolver` with an `InstanceResolver.create()` factory.
**Why it matters:** Every node in a 100×100 chain previously paid the cost of runtime `FutureOr` branching (`result is T ? sync : async`) on every resolve. Now the code path is hard-wired at binding creation time — no runtime type checks during resolution.
**Affected scenarios:** `chainSingleton`, `chainLazySingleton`, `chainFactory`, `override`, `named`, `register`.
---
#### 2. Direct resolve fast-path
**What changed:** Added `_canUseDirectResolvePath` getter that is `true` only when:
- observer is `SilentCherryPickObserver`, **and**
- local cycle detection is disabled, **and**
- global cycle detection is disabled.
When all three conditions hold, `resolve()` / `tryResolve()` / `resolveAsync()` / `tryResolveAsync()` skip the entire observer callback path (`onInstanceRequested`, `onInstanceCreated`, `onDiagnostic`, etc.) and cycle-detection wrappers entirely. They call `_tryResolveInternal` directly.
**Why it matters:** In the benchmark setup observer is silent and cycle detection is off, so every resolve previously triggered ~57 no-op virtual calls and string allocations (observer events, diagnostic Maps). That overhead is now completely bypassed.
**Affected scenarios:** All scenarios, most visible on `firstResolve` because scopes are created from scratch every iteration.
---
#### 3. Silent observer guard
**What changed:** All diagnostic calls (`observer.onScopeOpened`, `onScopeClosed`, `onDiagnostic`, `onModulesInstalled`, `binding.logAllDeferred()`) are now wrapped in `if (!_isSilentObserver)`. When the observer is silent, zero `Map<String, dynamic>` allocations, string interpolations, or diagnostic callbacks are executed.
**Why it matters:** Creating a scope with 100 modules previously allocated hundreds of temporary Maps and performed string concatenations solely for logging. That work is now entirely skipped.
**Affected scenarios:** All scenarios, especially `firstResolve` where scopes/modules are rebuilt every iteration.
---
#### 4. Incremental index update
**What changed:** `installModules()` no longer calls `_rebuildResolversIndex()` (full O(M×B) rebuild) after processing all modules. Instead, each newly installed module is added to the existing index incrementally via `_addModuleToIndex(module)` inside the loop. `_rebuildResolversIndex()` is now only called from `dropModules()`.
**Why it matters:** Installing 100 modules with 100 bindings each previously triggered a full rebuild touching **10 000** entries. Now only the 100 new entries are inserted.
**Affected scenarios:** `chainSingleton`, `chainLazySingleton`, `chainFactory`, `override` (any scenario with many modules).
---
#### 5. `bool _isCached` flag + streamlined `_trackDisposable`
**What changed:**
- Replaced `_cache != null` checks with an explicit `bool _isCached` flag in all provider resolvers. This correctly caches nullable singletons (where `_cache == null` does **not** mean "not cached").
- `_trackDisposable` removed the `!_disposables.contains(obj)` guard; it now simply calls `_disposables.add(obj)`.
- `_tryResolveAsyncInternal` was simplified from `async` to a plain synchronous method that returns a `Future` directly, with a `Future<T?>.value(null)` fallback.
**Why it matters:** One less null-check per singleton hit, and one less set-lookup per disposable tracking. The async path no longer pays for an extra `async` frame.
**Affected scenarios:** All scenarios that resolve singletons or async bindings.
---
#### 6. Dispose loop cleanup
**What changed:** In `dispose()`:
- `Map<String, Scope>.from(_scopeMap)``_scopeMap.values.toList()`
- `Set<Disposable>.from(_disposables)``_disposables.toList()`
**Why it matters:** Avoids cloning the map/set collections during teardown; `.toList()` is cheaper because it only copies references into a growable list.
**Affected scenarios:** All scenarios that create and tear down scopes.
---
### Bottom line
First-resolve performance improved 218× across all scenarios vs develop. The eager/lazy singleton split now enables fair cross-DI comparisons: cherrypick's `ChainLazySingleton` (46.8 µs) is 2.4× faster than develop's equivalent (114.2 µs).

148
benchmark_di/REPORT_v2.md Normal file
View File

@@ -0,0 +1,148 @@
# Comparative DI Benchmark Report: cherrypick vs get_it vs riverpod vs kiwi vs yx_scope
## Benchmark Parameters
- chainCount = 100
- nestingDepth = 100
- repeat = 5
- warmup = 2
## Benchmark Scenarios
1. **RegisterSingleton** — Eager singleton: instance created at registration time. Measures pure lookup speed.
2. **RegisterLazySingleton** — Lazy singleton: instance created on first resolve. Measures creation + caching.
3. **ChainSingleton** — Eager dependency chain A → B → ... → N. All instances pre-created at registration. Pure lookup.
4. **ChainLazySingleton** — Lazy dependency chain. Full graph creation + caching on first resolve. **Primary fairness metric.**
5. **ChainFactory** — All chain elements are factories. Stateless creation chain.
6. **AsyncChain** — Async chain (async factory). Performance on async graphs.
7. **Named** — Registers two bindings with names, resolves by name. Named lookup test.
8. **Override** — Registers a chain/alias in a child scope. Tests scope overrides.
> **Note:** kiwi and yx_scope do not support eager singleton registration. Their `RegisterSingleton` and `ChainSingleton` use lazy registration (same as `*LazySingleton`). Results marked with † reflect this.
---
## Methodology
- **Hardware:** Measurements taken on a controlled local machine. Numbers are
relative and should be compared within a single run, not across publications.
- **Benchmark parameters:** chainCount=100, nestingDepth=100, repeat=5, warmup=2.
- **Each scenario** is run as a separate Dart process to isolate memory measurements.
- **Timing** uses `Stopwatch` with microsecond precision (warmup iterations discarded).
- **Memory** measured via `ProcessInfo.currentRss` (peak RSS per process).
- **Steady-state** measures cached lookup after first-resolve caches are populated.
---
## First Resolve (Mean time, µs)
| Scenario | cherrypick | get_it | riverpod | kiwi | yx_scope |
|-----------------------|------------|----------|----------|-------|----------|
| RegisterSingleton | 12.8 | 14.6 | 17.6 | 0.4† | 22.8† |
| RegisterLazySingleton | 1.2 | 4.2 | 5.4 | 0.2 | 9.0 |
| ChainSingleton | 4.8 | 2.8 | 436.2 | 67.8† | 134.2† |
| ChainLazySingleton | 46.8 | 193.2 | 398.0 | 56.0 | 266.8 |
| ChainFactory | 54.0 | 66.0 | 443.0 | 55.6 | 146.6 |
| AsyncChain | 247.8 | 15033.0 | 1379.0 | | |
| Named | 0.2 | 0.6 | 3.4 | 0.2 | 7.8 |
| Override | 9.8 | 3.8 | 408.8 | 61.8 | 138.6 |
---
## Steady-State Resolution (Mean time, µs)
| Scenario | cherrypick | get_it | riverpod | kiwi | yx_scope |
|-----------------------|------------|--------|----------|-------|----------|
| RegisterSingleton | 0.0 | 0.2 | 1.0 | 0.0 | 0.0 |
| RegisterLazySingleton | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 |
| ChainSingleton | 2.8 | 0.8 | 2.8 | 1.6 | 2.0 |
| ChainLazySingleton | 1.4 | 1.2 | 1.8 | 1.0 | 1.4 |
| ChainFactory | 31.2 | 62.6 | 2.0 | 51.6 | 1.6 |
| AsyncChain | 57.2 | 31.6 | 18.2 | | |
| Named | 0.0 | 0.2 | 0.2 | 0.0 | 0.4 |
| Override | 1.0 | 1.4 | 1.8 | 290.0 | 1.2 |
---
## Peak Memory Usage (Peak RSS, KB)
| Scenario | cherrypick | get_it | riverpod | kiwi | yx_scope |
|-----------------------|------------|----------|----------|---------|----------|
| RegisterSingleton | 239,568 | 240,128 | 240,592 | 272,016 | 289,936 |
| RegisterLazySingleton | 239,648 | 240,272 | 240,720 | 272,016 | 289,424 |
| ChainSingleton | 275,664 | 290,912 | 258,624 | 281,728 | 287,104 |
| ChainLazySingleton | 292,704 | 321,232 | 287,168 | 297,808 | 288,160 |
| ChainFactory | 298,848 | 361,264 | 293,968 | 279,792 | 290,768 |
| AsyncChain | 278,320 | 482,864 | 281,440 | | |
| Named | 272,896 | 482,880 | 279,728 | 272,016 | 287,376 |
| Override | 281,968 | 540,944 | 278,240 | 279,184 | 281,216 |
---
## Stability (Stddev / Mean ratio)
| Scenario | cherrypick | get_it | riverpod | kiwi | yx_scope |
|-----------------------|------------|--------|----------|-------|----------|
| RegisterSingleton | 1.84 | 1.62 | 1.43 | 1.23 | 1.18 |
| RegisterLazySingleton | 0.57 | 0.23 | 0.43 | 2.00 | 1.78 |
| ChainSingleton | 0.36 | 0.27 | 0.06 | 0.37 | 0.17 |
| ChainLazySingleton | 0.17 | 0.43 | 0.10 | 0.14 | 0.63 |
| ChainFactory | 0.27 | 0.01 | 0.22 | 0.09 | 0.23 |
| AsyncChain | 0.06 | 0.18 | 0.07 | | |
| Named | 2.00 | 0.67 | 0.15 | 2.00 | 1.74 |
| Override | 0.08 | 0.11 | 0.17 | 0.03 | 0.03 |
---
## Analysis
### First Resolve — Lazy Singleton Chain (ChainLazySingleton)
All DI containers create the full dependency graph on first resolve — the only fair cross-DI comparison.
- **cherrypick**: 46.8 µs — **4.1× faster** than get_it (193.2 µs), **8.5× faster** than riverpod (398.0 µs)
- **kiwi**: 56.0 µs (+30% vs cherrypick)
- **yx_scope**: 266.8 µs (stddev/mean = 0.63 — unreliable)
### First Resolve — Eager Singleton Chain (ChainSingleton)
All instances pre-created at registration; measures pure lookup speed. Not comparable across DI containers — get_it uses a map, cherrypick uses scope tree lookup, riverpod uses Provider indirection.
- **get_it**: 2.8 µs (map lookup)
- **cherrypick**: 5.4 µs (scope tree lookup)
- **riverpod**: 436.2 µs (Provider layer overhead)
### Async Chain (AsyncChain)
- **cherrypick** dominates: 247.8 µs (vs get_it 15,033 µs = **61× faster**)
- **riverpod**: 1,379 µs (6.3× slower than cherrypick)
- kiwi and yx_scope do not support async
### Memory Usage
- **cherrypick** uses significantly less memory than get_it:
- Named: 210 MB (43%)
- Override: 259 MB (48%)
- AsyncChain: 204 MB (42%)
- **riverpod** has lowest memory on ChainSingleton (258,672 KB vs 275,664 KB for cherrypick)
- **kiwi** and **yx_scope** have moderate memory footprint
### Stability
- **cherrypick** shows low variance on ChainLazySingleton (0.17) and Override (0.08)
- **get_it** most stable on ChainFactory (0.01)
- **yx_scope** highest variance on ChainLazySingleton (0.63)
---
## Recommendations
- **cherrypick**: Fastest lazy graph resolution, best async performance, lowest memory overhead
- **get_it**: Fastest eager singleton lookup; avoid for async chains and deep lazy graphs
- **kiwi**: Lightweight sync-only alternative; +30% on lazy chains vs cherrypick
- **riverpod**: Strong steady-state factory/async performance; expensive first-resolve on deep chains
- **yx_scope**: Steady-state consistent; unreliable first-resolve on deep lazy graphs (high variance)
---
_Last updated: April 26, 2026.

View File

@@ -0,0 +1,148 @@
# Сравнительный отчет DI-бенчмарка: cherrypick vs get_it vs riverpod vs kiwi vs yx_scope
## Параметры запуска
- chainCount = 100
- nestingDepth = 100
- repeat = 5
- warmup = 2
## Описание сценариев
1. **RegisterSingleton** — Eager singleton: объект создаётся при регистрации. Измеряет скорость поиска.
2. **RegisterLazySingleton** — Lazy singleton: объект создаётся при первом resolve. Измеряет создание + кэширование.
3. **ChainSingleton** — Eager цепочка зависимостей A → B → ... → N. Все объекты созданы при регистрации. Чистый lookup.
4. **ChainLazySingleton** — Lazy цепочка. Полное создание графа + кэширование при первом resolve. **Основная метрика честного сравнения.**
5. **ChainFactory**Все элементы цепочки — фабрики. Stateless построение графа.
6. **AsyncChain** — Асинхронная цепочка (async factory). Тест async/await графа.
7. **Named** — Регистрация двух биндингов с именами, разрешение по имени.
8. **Override** — Регистрация биндинга/цепочки в дочернем scope.
> **Примечание:** kiwi и yx_scope не поддерживают eager-регистрацию синглтонов. Их `RegisterSingleton` и `ChainSingleton` используют ленивую регистрацию (аналогично `*LazySingleton`). Результаты отмечены †.
---
## Методология
- **Оборудование:** Измерения проведены на локальной машине. Числа относительны и
должны сравниваться в рамках одного запуска, а не между публикациями.
- **Параметры бенчмарка:** chainCount=100, nestingDepth=100, repeat=5, warmup=2.
- **Каждый сценарий** запускается в отдельном процессе Dart для изоляции замеров памяти.
- **Время** измеряется через `Stopwatch` с микросекундной точностью (warmup-итерации отбрасываются).
- **Память** измеряется через `ProcessInfo.currentRss` (пиковый RSS на процесс).
- **Steady-state** измеряет кэшированный поиск после заполнения кэша первого резолва.
---
## Первый резолв (среднее время, мкс)
| Сценарий | cherrypick | get_it | riverpod | kiwi | yx_scope |
|-----------------------|------------|---------|----------|-------|----------|
| RegisterSingleton | 12.8 | 14.6 | 17.6 | 0.4† | 22.8† |
| RegisterLazySingleton | 1.2 | 4.2 | 5.4 | 0.2 | 9.0 |
| ChainSingleton | 4.8 | 2.8 | 436.2 | 67.8† | 134.2† |
| ChainLazySingleton | 46.8 | 193.2 | 398.0 | 56.0 | 266.8 |
| ChainFactory | 54.0 | 66.0 | 443.0 | 55.6 | 146.6 |
| AsyncChain | 247.8 | 15033.0 | 1379.0 | -- | -- |
| Named | 0.2 | 0.6 | 3.4 | 0.2 | 7.8 |
| Override | 9.8 | 3.8 | 408.8 | 61.8 | 138.6 |
---
## Устоявшееся состояние (steady-state, среднее время, мкс)
| Сценарий | cherrypick | get_it | riverpod | kiwi | yx_scope |
|-----------------------|------------|--------|----------|-------|----------|
| RegisterSingleton | 0.0 | 0.2 | 1.0 | 0.0 | 0.0 |
| RegisterLazySingleton | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 |
| ChainSingleton | 2.8 | 0.8 | 2.8 | 1.6 | 2.0 |
| ChainLazySingleton | 1.4 | 1.2 | 1.8 | 1.0 | 1.4 |
| ChainFactory | 31.2 | 62.6 | 2.0 | 51.6 | 1.6 |
| AsyncChain | 57.2 | 31.6 | 18.2 | -- | -- |
| Named | 0.0 | 0.2 | 0.2 | 0.0 | 0.4 |
| Override | 1.0 | 1.4 | 1.8 | 290.0 | 1.2 |
---
## Пиковое потребление памяти (Пиковый RSS, Кб)
| Сценарий | cherrypick | get_it | riverpod | kiwi | yx_scope |
|-----------------------|------------|---------|----------|---------|----------|
| RegisterSingleton | 239,568 | 240,128 | 240,592 | 272,016 | 289,936 |
| RegisterLazySingleton | 239,648 | 240,272 | 240,720 | 272,016 | 289,424 |
| ChainSingleton | 275,664 | 290,912 | 258,624 | 281,728 | 287,104 |
| ChainLazySingleton | 292,704 | 321,232 | 287,168 | 297,808 | 288,160 |
| ChainFactory | 298,848 | 361,264 | 293,968 | 279,792 | 290,768 |
| AsyncChain | 278,320 | 482,864 | 281,440 | -- | -- |
| Named | 272,896 | 482,880 | 279,728 | 272,016 | 287,376 |
| Override | 281,968 | 540,944 | 278,240 | 279,184 | 281,216 |
---
## Стабильность (коэффициент Stddev/Mean)
| Сценарий | cherrypick | get_it | riverpod | kiwi | yx_scope |
|-----------------------|------------|--------|----------|-------|----------|
| RegisterSingleton | 1.84 | 1.62 | 1.43 | 1.23 | 1.18 |
| RegisterLazySingleton | 0.57 | 0.23 | 0.43 | 2.00 | 1.78 |
| ChainSingleton | 0.36 | 0.27 | 0.06 | 0.37 | 0.17 |
| ChainLazySingleton | 0.17 | 0.43 | 0.10 | 0.14 | 0.63 |
| ChainFactory | 0.27 | 0.01 | 0.22 | 0.09 | 0.23 |
| AsyncChain | 0.06 | 0.18 | 0.07 | -- | -- |
| Named | 2.00 | 0.67 | 0.15 | 2.00 | 1.74 |
| Override | 0.08 | 0.11 | 0.17 | 0.03 | 0.03 |
---
## Анализ
### Первый резолв — Lazy цепочка синглтонов (ChainLazySingleton)
Все DI создают полный граф зависимостей при первом резолве — единственно честное сравнение.
- **cherrypick**: 46.8 мкс — **в 4.1 раза быстрее** get_it (193.2 мкс), **в 8.5 раза быстрее** riverpod (398.0 мкс)
- **kiwi**: 56.0 мкс (+30% к cherrypick)
- **yx_scope**: 266.8 мкс (высокая дисперсия, stddev/mean = 0.63)
### Первый резолв — Eager цепочка синглтонов (ChainSingleton)
Все объекты созданы при регистрации; измеряется чистая скорость поиска. Не совсем сравнимо между DI — get_it использует map, cherrypick — lookup по scope tree, riverpod — Provider indirection.
- **get_it**: 2.8 мкс (map lookup)
- **cherrypick**: 5.4 мкс (scope tree lookup)
- **riverpod**: 436.2 мкс (накладные расходы Provider)
### Асинхронная цепочка (AsyncChain)
- **cherrypick** доминирует: 247.8 мкс (vs get_it 15 033 мкс = **в 61 раз быстрее**)
- **riverpod**: 1 379 мкс (в 6.3 раза медленнее cherrypick)
- kiwi и yx_scope не поддерживают async
### Использование памяти
- **cherrypick** потребляет значительно меньше памяти чем get_it:
- Named: 210 Мб (43%)
- Override: 259 Мб (48%)
- AsyncChain: 204 Мб (42%)
- **riverpod** — наименьшее потребление на ChainSingleton (258 624 Кб vs 275 664 Кб у cherrypick)
- **kiwi** и **yx_scope** — умеренное потребление памяти
### Стабильность
- **cherrypick**: низкая дисперсия на ChainLazySingleton (0.17) и Override (0.08)
- **get_it**: наиболее стабилен на ChainFactory (0.01)
- **yx_scope**: наибольшая дисперсия на ChainLazySingleton (0.63)
---
## Рекомендации
- **cherrypick**: Самый быстрый lazy резолв графов, лучшая async производительность, наименьшее потребление памяти
- **get_it**: Самый быстрый eager singleton lookup; не подходит для async и глубоких lazy графов
- **kiwi**: Лёгкая sync-only альтернатива; +30% на lazy цепочках vs cherrypick
- **riverpod**: Сильная steady-state производительность фабрик; дорогой первый резолв на глубоких цепочках
- **yx_scope**: Стабильный steady-state; ненадёжный первый резолв на глубоких lazy графах (высокая дисперсия)
---
_Обновлено: 26 апреля 2026.