Sergey Penkovsky b1737bbce2 feat(mixtral): initial implementation of Mixtral MoE model, configs, and tests ...

- Add Mixtral architecture implementation with MoE support (llm/src/llm/models/mixtral/mixtral.py)
- Introduce generic Mixture-of-Experts (MoE) block (llm/src/llm/core/moe.py)
- Create dedicated configuration files for Mixtral training and generation experiments
- Register and test Mixtral support in experiment runner (run_llm_experiment.py)
- Add unit tests for Mixtral API including forward, caching, and generation modes
- Include Jupyter notebook mixstral.ipynb for architectural exploration and research
- Ensure correct handling of torch bool masks in sampling (top-k, top-p) during generation

BREAKING CHANGE: Adds new model code and test coverage, modifying experiment runner logic to register Mixtral.

2025-10-20 08:12:11 +03:00

..

bpe.ipynb

chore: add ipykernel dependency and update notebooks

2025-10-05 11:59:24 +03:00

gpt2.ipynb

docs: add GPT-2 analysis notebook

2025-10-05 12:48:32 +03:00

gpt.ipynb

fix: rename notebook

2025-10-05 12:46:17 +03:00

llama.ipynb

fix(rope, attention): корректное позиционирование RoPE при генерации с кэшем

2025-10-14 12:03:20 +03:00

mistral.ipynb

feat(mistral): add Mistral model implementation and configs

2025-10-14 14:53:45 +03:00

mixstral.ipynb

feat(mixtral): initial implementation of Mixtral MoE model, configs, and tests

2025-10-20 08:12:11 +03:00