{ "cells": [ { "cell_type": "markdown", "id": "0939798d", "metadata": {}, "source": [ "# Mistral\n", "\n", "\n", "![](https://ucarecdn.com/0b4b9601-ccc9-4062-a62b-cca67a82f82b/)\n", "\n", "\n", "1-е поколение Mistral вышло в сентябре 2023 года.\n", "\n", "Mistral получил те же улучшения, что и у Llama: RMSNorm, SwiGLU и RoPE. Но мистраль пошел дальше и добавил еще две оптимизационные фишки:\n", "\n", "\n", "- Grouped-Query Attention (GQA)\n", "- Sliding Window Attention (SWA)\n", "\n", "\n", "Обе модифицируют механизм внимания. И обе предназначены для экономии памяти и ускорения вычислений." ] }, { "cell_type": "markdown", "id": "ec28fd32", "metadata": {}, "source": [ "# Masked Multi-Head Attention, ver. 2.0\n", "\n", "В текущей реализации **Multi-Head Attention** у нас каждая голова живет своей жизнью и обрабатывается по отдельности:\n", "\n", "

\n", " \"multi_head_1\"\n", "

\n", "\n", "* Класс `MultiHeadAttention` получает на вход тензор размером `batch_size × seq_len × emb_size` и передает его в каждую голову.\n", "* В голове тензор перемножается с тремя матрицами весов: `W_k`, `W_q`, `W_v`, каждая размером `emb_size × head_size`.\n", "* В результате получаются три матрицы: запроса (query), ключа (key) и значения (value). Каждая из них имеет размер `batch_size × seq_len × head_size`.\n", "* Матрицы запроса (query) и ключа (key) мы поворачиваем с помощью техники RoPE.\n", "* Матрицу ключа (key) транспонируем и перемножаем с матрицей запроса (query). В результате получается матрица внимания.\n", "* Далее перемножаем матрицу внимания и матрицу значения (value).\n", "* На выходе из головы получается тензор размера `batch_size × seq_len × head_size`.\n", "* Выходы из всех голов конкатенируются и умножаются на выходные веса, что уменьшает их размер.\n", "* На выходе из `MultiHeadAttention` у нас получается тензор такого же размера, какой поступил на вход: `batch_size × seq_len × emb_size`.\n", "\n", "Теперь нам нужно оптимизировать вычисления и сделать так, чтобы все головы вычислялись одновременно в классе `MultiHeadAttention`. Для этого изменим алгоритм следующим образом:\n", "\n", "![multi\\_head\\_2](https://ucarecdn.com/1686165f-7632-4b94-89bc-e0ed7e2ffe07/)\n", "\n", "* Класс `MultiHeadAttention` получает на вход тензор размером `batch_size × seq_len × emb_size`.\n", "* Тензор перемножается с тремя матрицами весов: `W_q`, `W_k`, `W_v`. Но на этот раз они имеют размер `emb_size × (num_heads * head_size)`.\n", " То есть, мы как бы расширили каждую матрицу весов по горизонтали на число голов.\n", "* После перемножения получаются три матрицы: запроса (query), ключа (key) и значения (value). Каждая из них также стала шире на количество голов: `batch_size × seq_len × (num_heads * head_size)`.\n", "* Переводим матрицы запроса (query), ключа (key) и значения (value) в форму четырехмерного тензора:\n", " `batch_size × num_heads × seq_len × head_size`. Это необходимо для дальнейших матричных операций.\n", "* Матрицы запроса (query) и ключа (key) мы поворачиваем с помощью техники RoPE.\n", "* Транспонируем тензор ключа и перемножаем его с тензором запроса. Получится матрица внимания, которая будет иметь размер\n", " `batch_size × num_heads × seq_len × seq_len`.\n", "* Далее перемножаем матрицу внимания и тензор значения (value). Получается тензор размером\n", " `batch_size × num_heads × seq_len × head_size`. Переводим тензор в «плоский» вид:\n", " `batch_size × seq_len × (num_heads * head_size)`.\n", "* Пропускаем тензор через выходную проекцию (`batch_size × (num_heads * head_size) × emb_size`), чтобы уменьшить его размер.\n", "* На выходе из класса получается тензор точно такого же размера, какой поступил на вход:\n", " `batch_size × seq_len × emb_size`.\n", "\n", "Ну и также версия с кэшем (когда на вход приходит только один токен):\n", "\n", "![multi\\_head\\_3](https://ucarecdn.com/067ce912-2932-418f-9249-09a3564ca82b/)\n", "\n", "Единственное изменение: после выполнения поворота мы объединяем текущий тензор с тензором кэшей (для векторов ключа и значения)." ] }, { "cell_type": "markdown", "id": "d53f1cfc", "metadata": {}, "source": [ "# RoPE, ver. 2.0 (разработка)\n", "\n", "Первым делом нам нужно подредактировать класс `RoPE`. Сейчас он используется внутри класса `HeadAttention`, а будет использоваться внутри `MultiHeadAttention`.\n", "\n", "Единственное явное отличие старой версии от новой — что подается на вход (в метод `forward`):\n", "\n", "* Сейчас в него приходит тензор размера `batch_size × seq_len × head_size`.\n", "* А будет приходить тензор размера `batch_size × num_heads × seq_len × head_size`.\n", "\n", "

\n", " \"rope\"\n", "

\n" ] }, { "cell_type": "code", "execution_count": 1, "id": "4c10a0b2", "metadata": {}, "outputs": [], "source": [ "import torch\n", "from torch import nn\n", "from typing import Optional\n", "\n", "\n", "class RoPE(nn.Module):\n", "\n", " def __init__(self, head_size: int, max_seq_len: int, base: int = 10_000):\n", " super().__init__()\n", " assert head_size % 2 == 0, \"head_size должен быть четным\"\n", "\n", " # Вычисление частот: θ_i = base^(-2i/d) для i ∈ [0, d/2-1]\n", " freqs = 1.0 / (base ** (2 * torch.arange(head_size // 2).float() / head_size))\n", "\n", " # Позиции от 0 до max_seq_len-1\n", " positions = torch.arange(max_seq_len).float()\n", "\n", " # Внешнее произведение: m * θ_i для всех позиций и частот\n", " freq_matrix = positions.unsqueeze(1) * freqs.unsqueeze(0)\n", "\n", " # Предвычисление матриц косинусов и синусов\n", " self.register_buffer(\"cos_matrix\", torch.cos(freq_matrix))\n", " self.register_buffer(\"sin_matrix\", torch.sin(freq_matrix))\n", "\n", " def forward(self, x: torch.Tensor, start_pos: int = 0) -> torch.Tensor: # [batch_size × seq_len × head_size] [batch_size × num_heads × seq_len × head_size]\n", " batch_size, num_heads, seq_len, head_size = x.shape\n", "\n", " # Берем нужную часть матриц и приводим к типу x\n", " cos = self.cos_matrix[start_pos:start_pos+seq_len].to(x.dtype) # [seq_len, head_size//2]\n", " sin = self.sin_matrix[start_pos:start_pos+seq_len].to(x.dtype) # [seq_len, head_size//2]\n", "\n", " # Явное изменение формы для broadcasting\n", " cos = cos.reshape(1, 1, seq_len, head_size // 2)\n", " sin = sin.reshape(1, 1, seq_len, head_size // 2)\n", "\n", " # Разделяем на четные и нечетные компоненты по ПОСЛЕДНЕМУ измерению\n", " x_even = x[..., 0::2] # [batch_size, num_heads, seq_len, head_size//2]\n", " x_odd = x[..., 1::2] # [batch_size, num_heads, seq_len, head_size//2]\n", "\n", " # Применяем поворот: q' = q * cos(mθ) + rotate(q) * sin(mθ)\n", " x_rotated_even = x_even * cos - x_odd * sin\n", " x_rotated_odd = x_even * sin + x_odd * cos\n", "\n", " # Объединяем обратно в исходную размерность\n", " x_rotated = torch.stack([x_rotated_even, x_rotated_odd], dim=-1)\n", " x_rotated = x_rotated.flatten(-2) # [batch_size, seq_len, head_size]\n", "\n", " return x_rotated\n" ] }, { "cell_type": "code", "execution_count": 2, "id": "e90c94a9", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "✓ Форма корректна\n" ] } ], "source": [ "rope = RoPE(head_size=64, max_seq_len=512)\n", "x = torch.randn(2, 8, 128, 64) # batch=2, heads=8, seq=128, dim=64\n", "output = rope(x)\n", "assert output.shape == x.shape\n", "print(\"✓ Форма корректна\")" ] }, { "cell_type": "markdown", "id": "ca50ac9c", "metadata": {}, "source": [ "## MultiHeadAttention v2" ] }, { "cell_type": "code", "execution_count": 3, "id": "883383d2", "metadata": {}, "outputs": [], "source": [ "import torch\n", "from torch import nn\n", "import torch.nn.functional as F\n", "from typing import Optional, Tuple\n", "\n", "\n", "class MultiHeadAttentionV2(nn.Module):\n", "\n", " def __init__(\n", " self,\n", " num_heads: int,\n", " emb_size: int,\n", " head_size: int,\n", " max_seq_len: int,\n", " rope: RoPE = None,\n", " dropout: float = 0.1,\n", " ):\n", " super().__init__()\n", " self._num_heads = num_heads\n", " self._head_size = head_size\n", " self._max_seq_len = max_seq_len\n", " self._rope = rope\n", "\n", " self._q = nn.Linear(emb_size, num_heads * head_size)\n", " self._k = nn.Linear(emb_size, num_heads * head_size)\n", " self._v = nn.Linear(emb_size, num_heads * head_size)\n", "\n", " # Создание causal маски\n", " mask = torch.tril(torch.ones(max_seq_len, max_seq_len))\n", " self.register_buffer(\n", " \"_tril_mask\", mask.bool() if hasattr(torch, \"bool\") else mask.byte()\n", " )\n", " \n", " self._layer = nn.Linear(head_size * num_heads, emb_size)\n", " self._dropout = nn.Dropout(dropout)\n", "\n", " def forward(\n", " self,\n", " x: torch.Tensor,\n", " mask: torch.Tensor = None,\n", " use_cache: bool = True,\n", " cache: list = None,\n", " ):\n", " batch_size, seq_len, emb_size = x.shape\n", "\n", " if seq_len > self._max_seq_len:\n", " raise ValueError(\n", " f\"Длина последовательности {seq_len} превышает максимум {self._max_seq_len}\"\n", " )\n", "\n", " # Пропустите тензор x через матрицы Wq, Wk , Wv, чтобы получить матрицы запроса, ключа и значения.\n", " k = self._k(x) # [B, T, hs]\n", " q = self._q(x) # [B, T, hs]\n", " v = self._v(x) # [B, T, hs]\n", "\n", " # Шаг 2: Изменение формы для multi-head\n", " # [batch_size, seq_len, num_heads * head_size] \n", " # -> [batch_size, seq_len, num_heads, head_size]\n", " q = q.reshape(batch_size, seq_len, self._num_heads, self._head_size)\n", " k = k.reshape(batch_size, seq_len, self._num_heads, self._head_size)\n", " v = v.reshape(batch_size, seq_len, self._num_heads, self._head_size)\n", " \n", "\n", " # 3. Transpose: [B, T, H, hs] -> [B, H, T, hs]\n", " q = q.transpose(1, 2)\n", " k = k.transpose(1, 2)\n", " v = v.transpose(1, 2)\n", "\n", " start_pos = 0\n", " if cache is not None:\n", " k_cache, v_cache = cache\n", " cache_len = k_cache.shape[2]\n", " start_pos = cache_len\n", " \n", " # Пропустите матрицы запроса и ключа через экземпляр rope, чтобы выполнить поворот.\n", " if self._rope is not None:\n", " # ✅ Применяем RoPE к Q и K (НЕ к V!)\n", " q = self._rope(q, start_pos=start_pos) # [B, T, hs]\n", " k = self._rope(k, start_pos=start_pos) # [B, T, hs]\n", "\n", " # Если cache пришел, то объединяем кэш и одну строку из ключа и значения. Это будут новые key и value для последующих вычислений.\n", " # 5. Кэширование (для autoregressive generation)\n", " if cache is not None:\n", " k_cache, v_cache = cache\n", " k = torch.cat([k_cache, k], dim=2) # Concat по seq_len (dim=2)\n", " v = torch.cat([v_cache, v], dim=2)\n", "\n", " # Перемножим матрицы запроса и ключа (транспонированную), чтобы вычислить матрицу внимания.\n", " # И разделить все значения в матрице внимания на корень из head_size.\n", " scores = q @ k.transpose(-2, -1) / (self._head_size ** 0.5)\n", "\n", " # Если cache пришел, то маску не накладываем. Иначе наложите на матрицу внимания треугольную маску, созданную при инициализации. Все скрытые значения должны быть приведены к минус бесконечности: float('-inf').\n", " if cache is None:\n", " scores = scores.masked_fill(\n", " ~self._tril_mask[:seq_len, :seq_len], float(\"-inf\")\n", " )\n", "\n", " # Применить к матрице внимания (построчно) функцию Softmax.\n", " weights = F.softmax(scores, dim=-1)\n", "\n", " # Перемножим матрицу внимания и матрицу значения.\n", " x_out = weights @ v # [B, T, hs]\n", "\n", " # Измените форму тензора на batch_size × seq_len × num_heads*head_size.\n", " # Transpose обратно и concatenate heads\n", " x_out = x_out.transpose(1, 2) # [B, T_q, H, hs]\n", " x_out = x_out.contiguous() # Важно для reshape!\n", " concatenated_attention = x_out.reshape(batch_size, seq_len, self._num_heads * self._head_size)\n", "\n", " #concatenated_attention = x_out.reshape(batch_size, seq_len, self._num_heads * self._head_size)\n", "\n", " # Пропустите получившийся тензор через последний линейный слой.\n", " # 3. Проецируем в пространство эмбеддингов\n", " projected_output = self._layer(concatenated_attention)\n", "\n", " # 4. Применяем dropout для регуляризации\n", " final_output = self._dropout(projected_output)\n", "\n", " if use_cache is True:\n", " return (final_output, (k, v))\n", " else:\n", " return (final_output, None)" ] }, { "cell_type": "code", "execution_count": 4, "id": "9ab78666", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "✅ Test 1 - Output shape: torch.Size([2, 10, 512])\n", "✅ Test 2 - First output shape: torch.Size([2, 5, 512])\n", "✅ Test 2 - Second output shape: torch.Size([2, 1, 512])\n", "\n", "✅ Все тесты пройдены!\n" ] } ], "source": [ "\n", "# Параметры\n", "batch_size = 2\n", "seq_len = 10\n", "emb_size = 512\n", "num_heads = 8\n", "head_size = 64\n", "max_seq_len = 512\n", "\n", " # Создание модели\n", "rope = RoPE(head_size=head_size, max_seq_len=max_seq_len)\n", "mha = MultiHeadAttentionV2(\n", " num_heads=num_heads,\n", " emb_size=emb_size,\n", " head_size=head_size,\n", " max_seq_len=max_seq_len,\n", " rope=rope,\n", " dropout=0.1,\n", ")\n", "\n", " # Тест 1: Обычный forward pass\n", "x = torch.randn(batch_size, seq_len, emb_size)\n", "output, cache = mha(x, use_cache=False)\n", "print(f\"✅ Test 1 - Output shape: {output.shape}\") # [2, 10, 512]\n", "assert output.shape == (batch_size, seq_len, emb_size)\n", "\n", " # Тест 2: С кэшированием\n", "x1 = torch.randn(batch_size, 5, emb_size)\n", "output1, cache1 = mha(x1, use_cache=True)\n", "print(f\"✅ Test 2 - First output shape: {output1.shape}\") # [2, 5, 512]\n", "\n", "x2 = torch.randn(batch_size, 1, emb_size)\n", "output2, cache2 = mha(x2, use_cache=True, cache=cache1)\n", "print(f\"✅ Test 2 - Second output shape: {output2.shape}\") # [2, 1, 512]\n", "\n", "print(\"\\n✅ Все тесты пройдены!\")" ] }, { "cell_type": "markdown", "id": "0a03c4d2", "metadata": {}, "source": [ "### Промежуточный вариант Mistral" ] }, { "cell_type": "code", "execution_count": 5, "id": "645a3cf9", "metadata": {}, "outputs": [], "source": [ "import torch\n", "from torch import nn\n", "from torch import Tensor\n", "import torch.nn.functional as F\n", "from math import sqrt\n", "\n", "\n", "\n", "class SiLU(nn.Module):\n", " def forward(self, x: torch.Tensor): # [batch_size × seq_len × emb_size]\n", " return torch.sigmoid(x) * x\n", " \n", "class RMSNorm(nn.Module):\n", " def __init__(self, dim: int, eps: float = 1e-6):\n", " super().__init__()\n", " self._eps = eps\n", " self._w = nn.Parameter(torch.ones(dim))\n", " \n", " def forward(self, x: torch.Tensor): # [batch_size × seq_len × emb_size]\n", " rms = (x.pow(2).mean(-1, keepdim=True) + self._eps) ** 0.5\n", " norm_x = x / rms\n", " return self._w * norm_x\n", "\n", "class SwiGLU(nn.Module):\n", " def __init__(self, emb_size: int, dropout: float = 0.1):\n", " super().__init__()\n", "\n", " self._gate = nn.Linear(emb_size, 4 * emb_size)\n", " self._up = nn.Linear(emb_size, 4 * emb_size)\n", " self._down = nn.Linear(4 * emb_size, emb_size)\n", " self._activation = SiLU()\n", " self._dropout = nn.Dropout(dropout)\n", "\n", " def forward(self, x: torch.Tensor): # [batch_size × seq_len × emb_size].\n", " gate_out = self._gate(x) # [batch, seq, 4*emb]\n", " activation_out = self._activation(gate_out) # [batch, seq, 4*emb]\n", " up_out = self._up(x) # [batch, seq, 4*emb]\n", " out = up_out * activation_out # поэлементное!\n", " out = self._down(out) # [batch, seq, emb]\n", " return self._dropout(out)\n", "\n", "\n", "class TokenEmbeddings(nn.Module):\n", " def __init__(self, vocab_size: int, emb_size: int):\n", " super().__init__()\n", " self._embedding = nn.Embedding(\n", " num_embeddings=vocab_size,\n", " embedding_dim=emb_size\n", " )\n", "\n", " def forward(self, x: Tensor) -> Tensor:\n", " return self._embedding(x)\n", "\n", " @property\n", " def num_embeddings(self) -> int:\n", " return self._embedding.num_embeddings\n", "\n", " @property\n", " def embedding_dim(self) -> int:\n", " return self._embedding.embedding_dim\n", "\n", "\n", "class GELU(nn.Module):\n", " def __init__(self):\n", " super().__init__()\n", " self.sqrt_2_over_pi = torch.sqrt(torch.tensor(2.0) / math.pi)\n", " \n", " def forward(self, x: torch.Tensor) -> torch.Tensor:\n", " return 0.5 * x * (1 + torch.tanh(\n", " self.sqrt_2_over_pi * (x + 0.044715 * torch.pow(x, 3))\n", " ))\n", " \n", "class Decoder(nn.Module):\n", " def __init__(self, \n", " num_heads: int,\n", " emb_size: int,\n", " head_size: int,\n", " max_seq_len: int,\n", " rope: RoPE,\n", " dropout: float = 0.1\n", " ):\n", " super().__init__()\n", " self._heads = MultiHeadAttentionV2(\n", " num_heads=num_heads, \n", " emb_size=emb_size, \n", " head_size=head_size, \n", " max_seq_len=max_seq_len,\n", " rope=rope,\n", " dropout=dropout\n", " )\n", " self._ff = SwiGLU(emb_size=emb_size, dropout=dropout)\n", " self._norm1 = RMSNorm(emb_size)\n", " self._norm2 = RMSNorm(emb_size)\n", "\n", " def forward(self, x: torch.Tensor, mask: torch.Tensor = None, use_cache: bool = True, cache: list = None) -> torch.Tensor:\n", " norm1_out = self._norm1(x)\n", " attention, kv_caches = self._heads(norm1_out, mask, use_cache=use_cache, cache=cache)\n", " out = attention + x\n", " \n", " norm2_out = self._norm2(out)\n", " ffn_out = self._ff(norm2_out)\n", "\n", " if use_cache is True:\n", " return (ffn_out + out, kv_caches)\n", " else:\n", " return (ffn_out + out, None)\n", "\n", "\n", "\n", "from torch import nn\n", "import torch\n", "import torch.nn.functional as F\n", "\n", "class Mistral(nn.Module):\n", " def __init__(self,\n", " vocab_size: int,\n", " max_seq_len: int,\n", " emb_size: int,\n", " num_heads: int,\n", " head_size: int,\n", " num_layers: int,\n", " dropout: float = 0.1,\n", " device: str = 'cpu'\n", " ):\n", " super().__init__()\n", " self._vocab_size = vocab_size\n", " self._max_seq_len = max_seq_len\n", " self._emb_size = emb_size\n", " self._num_heads = num_heads\n", " self._head_size = head_size\n", " self._num_layers = num_layers\n", " self._dropout = dropout\n", " self._device = device\n", " \n", " self.validation_loss = None\n", "\n", " # Инициализация слоев\n", " self._token_embeddings = TokenEmbeddings(\n", " vocab_size=vocab_size, \n", " emb_size=emb_size\n", " )\n", " self._position_embeddings = RoPE(\n", " head_size=head_size,\n", " max_seq_len=max_seq_len\n", " )\n", " #self._position_embeddings = PositionalEmbeddings(\n", " # max_seq_len=max_seq_len, \n", " # emb_size=emb_size\n", " #)\n", " self._dropout = nn.Dropout(dropout)\n", " self._decoders = nn.ModuleList([Decoder(\n", " num_heads=num_heads,\n", " emb_size=emb_size,\n", " head_size=head_size,\n", " max_seq_len=max_seq_len,\n", " rope=self._position_embeddings,\n", " dropout=dropout \n", " ) for _ in range(num_layers)])\n", " self._norm = RMSNorm(emb_size)\n", " self._linear = nn.Linear(emb_size, vocab_size)\n", "\n", " def forward(self, x: torch.Tensor, use_cache: bool = True, cache: list = None) -> tuple:\n", " # Проверка длины последовательности (только при отсутствии кэша)\n", " if cache is None and x.size(1) > self._max_seq_len:\n", " raise ValueError(f\"Длина последовательности {x.size(1)} превышает максимальную {self.max_seq_len}\")\n", " \n", " # Эмбеддинги токенов и позиций\n", " tok_out = self._token_embeddings(x) # [batch, seq_len, emb_size]\n", " #pos_out = self._position_embeddings(x) # [batch, seq_len, emb_size]\n", " \n", " # Комбинирование\n", " out = self._dropout(tok_out) # [batch, seq_len, emb_size]\n", " \n", " # Стек декодеров с передачей кэша\n", " new_cache = []\n", " for i, decoder in enumerate(self._decoders):\n", " decoder_cache = cache[i] if cache is not None else None\n", " decoder_result = decoder(out, use_cache=use_cache, cache=decoder_cache)\n", "\n", " # Извлекаем результат из кортежа\n", " if use_cache:\n", " out, decoder_new_cache = decoder_result\n", " new_cache.append(decoder_new_cache)\n", " else:\n", " out = decoder_result[0]\n", "\n", " out = self._norm(out)\n", " logits = self._linear(out)\n", " \n", " # Возвращаем результат с учетом use_cache\n", " if use_cache:\n", " return (logits, new_cache)\n", " else:\n", " return (logits, None)\n", "\n", " def generate(self,\n", " x: torch.Tensor, \n", " max_new_tokens: int, \n", " do_sample: bool,\n", " temperature: float = 1.0,\n", " top_k: int = None,\n", " top_p: float = None,\n", " use_cache: bool = True\n", " ) -> torch.Tensor:\n", " cache = None\n", "\n", " for _ in range(max_new_tokens):\n", " if use_cache and cache is not None:\n", " # Используем кэш - передаем только последний токен\n", " x_input = x[:, -1:] # [batch_size, 1]\n", " else:\n", " # Первая итерация или кэш отключен - передаем всю последовательность\n", " x_input = x\n", " \n", " # Прямой проход с кэшем\n", " logits, new_cache = self.forward(x_input, use_cache=use_cache, cache=cache)\n", " \n", " # Обновляем кэш для следующей итерации\n", " if use_cache:\n", " cache = new_cache\n", "\n", " last_logits = logits[:, -1, :] # [batch_size, vocab_size]\n", "\n", " # Масштабируем логиты температурой\n", " if temperature > 0:\n", " logits_scaled = last_logits / temperature\n", " else:\n", " logits_scaled = last_logits\n", "\n", " if do_sample == True and top_k != None:\n", " _, topk_indices = torch.topk(logits_scaled, top_k, dim=-1)\n", "\n", " # # Заменим все НЕ top-k логиты на -inf\n", " masked_logits = logits_scaled.clone()\n", " vocab_size = logits_scaled.size(-1)\n", "\n", " # создаём маску: 1, если токен НЕ в topk_indices\n", " mask = torch.ones_like(logits_scaled, dtype=torch.uint8)\n", " mask.scatter_(1, topk_indices, 0) # 0 там, где top-k индексы\n", " masked_logits[mask.byte()] = float('-inf')\n", "\n", " logits_scaled = masked_logits\n", "\n", " if do_sample == True and top_p != None:\n", " # 1. Применим softmax, чтобы получить вероятности:\n", " probs = F.softmax(logits_scaled, dim=-1) # [B, vocab_size]\n", " # 2. Отсортируем токены по убыванию вероятностей:\n", " sorted_probs, sorted_indices = torch.sort(probs, descending=True, dim=-1)\n", " # 3. Посчитаем кумулятивную сумму вероятностей:\n", " cum_probs = torch.cumsum(sorted_probs, dim=-1) # [B, vocab_size]\n", " # 4. Определим маску: оставить токены, пока сумма < top_p\n", " sorted_mask = (cum_probs <= top_p).byte() # [B, vocab_size]\n", " # Гарантируем, что хотя бы первый токен останется\n", " sorted_mask[:, 0] = 1\n", " # 5. Преобразуем маску обратно в оригинальный порядок:\n", " # Создаём полную маску из 0\n", " mask = torch.zeros_like(probs, dtype=torch.uint8)\n", " # Устанавливаем 1 в местах нужных токенов\n", " mask.scatter_(dim=1, index=sorted_indices, src=sorted_mask)\n", " # 6. Зануляем логиты токенов вне топ-p:\n", " logits_scaled[~mask] = float('-inf')\n", "\n", " # 4. Применяем Softmax\n", " probs = F.softmax(logits_scaled, dim=-1) # [batch_size, vocab_size]\n", "\n", "\n", " if do_sample == True:\n", " # 5. Если do_sample равен True, то отбираем токен случайно с помощью torch.multinomial\n", " next_token = torch.multinomial(probs, num_samples=1) # [batch_size, 1]\n", " else:\n", " # 5. Если do_sample равен False, то выбираем токен с максимальной вероятностью\n", " next_token = torch.argmax(probs, dim=-1, keepdim=True) # [batch_size, 1]\n", " \n", " # 6. Добавляем его к последовательности\n", " x = torch.cat([x, next_token], dim=1) # [batch_size, seq_len+1]\n", " return x\n", "\n", " def save(self, path):\n", " torch.save({\n", " 'model_state_dict': self.state_dict(),\n", " 'vocab_size': self._vocab_size,\n", " 'max_seq_len': self._max_seq_len,\n", " 'emb_size': self._emb_size,\n", " 'num_heads': self._num_heads,\n", " 'head_size': self._head_size,\n", " 'num_layers': self._num_layers\n", " }, path)\n", "\n", " @classmethod\n", " def load(cls, path, device):\n", " checkpoint = torch.load(path, map_location=device)\n", " model = cls(\n", " vocab_size=checkpoint['vocab_size'],\n", " max_seq_len=checkpoint['max_seq_len'],\n", " emb_size=checkpoint['emb_size'],\n", " num_heads=checkpoint['num_heads'],\n", " head_size=checkpoint['head_size'],\n", " num_layers=checkpoint['num_layers']\n", " )\n", " model.load_state_dict(checkpoint['model_state_dict'])\n", " model.to(device)\n", " return model\n", "\n", " @property\n", " def max_seq_len(self) -> int:\n", " return self._max_seq_len\n", "\n", "\n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "c6145261", "metadata": {}, "source": [ "# Grouped-Query Attention\n", "\n", "**Grouped-Query Attention (GQA)** — это оптимизированный механизм внимания.\n", "\n", "В чем суть: в классическом **Multi-Head Attention (MHA)** на каждую голову приходится по три вектора: запроса, ключа и значения. Эти вектора существуют только внутри голов, где они взаимодействуют между собой.\n", "\n", "

\n", " \"gqa_1\"\n", "

\n", "\n", "А в **GQA** предложили сэкономить на матрицах: разделить головы на группы и на каждую группу назначить по одному вектору ключа и значения.\n", "При этом на каждую голову по-прежнему приходится один вектор запроса.\n", "\n", "

\n", " \"gqa_2\"\n", "

\n", "\n", "Что мы в результате получаем:\n", "\n", "* **Скорость:** генерация текста происходит на 30–40% быстрее, чем в MHA.\n", "* **Память:** экономия места в Q/G раз (где Q — количество векторов запроса, G — количество групп). Также снижается трафик памяти до 8–10 раз по сравнению с MHA.\n", "* **Качество:** близко к MHA.\n", "\n", "> В первом Mistral было 32 Query Heads и 8 K/V Heads.\n", "\n", "---\n", "\n", "### Как это работает технически?\n", "\n", "На первых шагах этого урока мы переделали механизм внимания.\n", "Избавились от отдельных голов и сделали единое пространство для вычислений всех голов одновременно.\n", "Каждая голова теперь представлена отдельными измерениями в одном длинном тензоре.\n", "Вот как это выглядит (здесь представлена часть механизма внимания):\n", "\n", "

\n", " \"gqa_3\"\n", "

\n", "\n", "* На вход мы получаем тензор размером `batch_size × seq_len × emb_size`.\n", "* Тензор перемножается с тремя матрицами весов: $W_q$, $W_k$, $W_v$, каждая размером `emb_size × num_heads * head_size`.\n", "* После перемножения получаются три матрицы: запроса (query), ключа (key) и значения (value), каждая размером `batch_size × seq_len × num_heads * head_size`.\n", "* Переводим матрицы запроса (query), ключа (key) и значения (value) в форму четырехмерного тензора:\n", " `batch_size × num_heads × seq_len × head_size`.\n", "* Выполняем поворот тензоров запроса (query) и ключа (key).\n", "* Дальше ничего не меняется...\n", "\n", "---\n", "\n", "И вот как нам надо это переделать:\n", "\n", "

\n", " \"gqa_4\"\n", "

\n", "\n", "* На вход мы получаем тензор размером `batch_size × seq_len × emb_size`.\n", "* Тензор перемножается с тремя матрицами весов: $W_q$, $W_k$, $W_v$:\n", "\n", " * $W_q$ — такого же размера, как и раньше: `emb_size × num_q_heads * head_size`.\n", " * А вот $W_k$ и $W_v$ уменьшились на количество K/V голов: `emb_size × num_kv_heads * head_size`.\n", "* После перемножения получаются три матрицы:\n", "\n", " * **Запрос (query)** — `batch_size × seq_len × num_q_heads * head_size`.\n", " * **Ключ (key)** и **значение (value)** — `batch_size × seq_len × num_kv_heads * head_size`.\n", "* Переводим их в форму четырехмерного тензора:\n", "\n", " * **Query:** `batch_size × num_q_heads × seq_len × head_size`.\n", " * **Key, Value:** `batch_size × num_kv_heads × seq_len × head_size`.\n", "* Выполняем поворот тензоров запроса (query) и ключа (key).\n", "* Затем проводим **уникальную операцию — размножение**.\n", " Нам нужно произвести матричные операции с тензорами, но у них разный размер, что делает перемножение невозможным.\n", " Чтобы исправить это, нужно продублировать головы в тензорах **query** и **key**, чтобы их размер стал одинаковым:\n", " `batch_size × num_q_heads × seq_len × head_size`.\n", " Копии располагаются последовательно — после каждой головы идут её дубликаты.\n", "* Дальнейшие операции остаются без изменений.\n", "\n", "> Может показаться, что с точки зрения использования памяти мы пришли к тому, с чего начали.\n", "> У нас тензор K и V получился такого же размера, как и тензор Q.\n", "> Но это только по внешнему виду. Расширение происходит **виртуально** — в памяти место не дублируется.\n", "\n", "---\n", "\n", "Ну и версия для кэша:\n", "\n", "

\n", " \"gqa_5\"\n", "

\n", "\n", "Единственное отличие: после операции поворота и до размножения голов мы склеиваем текущий токен с кэшем.\n", "\n", "---\n", "\n", "### Почему именно K и V?\n", "\n", "Любопытный читатель спросит: а почему мы сократили количество именно **K** и **V**?\n", "Почему не **Q и V**, или не **Q и K**?\n", "\n", "Дело в роли, которую играют вектора. Уже знакомая нам аналогия с библиотекой:\n", "\n", "* **Query** — это читатели с разными запросами (один ищет научную книгу, другой — художественную).\n", "* **Key** — это каталог карточек (индексы книг).\n", "* **Value** — это сами книги на полках.\n", "\n", "У каждого читателя свой уникальный запрос (**Q**), очевидно, их нельзя копировать на других читателей.\n", "Одни и те же каталог (**K**) и книги (**V**) разделены на секции (группы).\n", "Несколько читателей могут использовать одну секцию каталога/книг, но их запросы остаются уникальными.\n", "\n" ] }, { "cell_type": "markdown", "id": "aefe1ef9", "metadata": {}, "source": [ "### Grouped-Query Attention (разработка)" ] }, { "cell_type": "code", "execution_count": 6, "id": "84a3a599", "metadata": {}, "outputs": [], "source": [ "import torch\n", "from torch import nn\n", "import torch.nn.functional as F\n", "from typing import Optional, Tuple\n", "\n", "\n", "class GroupedQueryAttention(nn.Module):\n", "\n", " def __init__(\n", " self,\n", " num_heads: int,\n", " num_kv_heads: int,\n", " emb_size: int,\n", " head_size: int,\n", " max_seq_len: int,\n", " rope: RoPE = None,\n", " dropout: float = 0.1,\n", " ):\n", " super().__init__()\n", " self._num_heads = num_heads\n", " self._num_kv_heads = num_kv_heads\n", " self._head_size = head_size\n", " self._max_seq_len = max_seq_len\n", " self._rope = rope\n", "\n", " self._q = nn.Linear(emb_size, num_heads * head_size)\n", " self._k = nn.Linear(emb_size, num_kv_heads * head_size)\n", " self._v = nn.Linear(emb_size, num_kv_heads * head_size)\n", "\n", " # Создание causal маски\n", " mask = torch.tril(torch.ones(max_seq_len, max_seq_len))\n", " self.register_buffer(\n", " \"_tril_mask\", mask.bool() if hasattr(torch, \"bool\") else mask.byte()\n", " )\n", " \n", " self._layer = nn.Linear(head_size * num_heads, emb_size)\n", " self._dropout = nn.Dropout(dropout)\n", "\n", " def forward(\n", " self,\n", " x: torch.Tensor,\n", " mask: torch.Tensor = None,\n", " use_cache: bool = True,\n", " cache: list = None,\n", " ):\n", " batch_size, seq_len, emb_size = x.shape\n", "\n", " if seq_len > self._max_seq_len:\n", " raise ValueError(\n", " f\"Длина последовательности {seq_len} превышает максимум {self._max_seq_len}\"\n", " )\n", "\n", " # Пропустите тензор x через матрицы Wq, Wk , Wv, чтобы получить матрицы запроса, ключа и значения.\n", " k = self._k(x) # [B, T, hs]\n", " q = self._q(x) # [B, T, hs]\n", " v = self._v(x) # [B, T, hs]\n", "\n", " # Шаг 2: Изменение формы для multi-head\n", " # [batch_size, seq_len, num_heads * head_size] \n", " # -> [batch_size, seq_len, num_heads, head_size]\n", " # Измените форму запроса (query) на batch_size × num_q_heads × seq_len × head_size.\n", " q = q.reshape(batch_size, seq_len, self._num_heads, self._head_size)\n", "\n", " # Измените форму ключа (key) и значения (value) на batch_size × num_kv_heads × seq_len × head_size.\n", " k = k.reshape(batch_size, seq_len, self._num_kv_heads, self._head_size)\n", " v = v.reshape(batch_size, seq_len, self._num_kv_heads, self._head_size)\n", " \n", "\n", " # 3. Transpose: [B, T, H, hs] -> [B, H, T, hs]\n", " q = q.transpose(1, 2)\n", " k = k.transpose(1, 2)\n", " v = v.transpose(1, 2)\n", "\n", " start_pos = 0\n", " if cache is not None:\n", " k_cache, v_cache = cache\n", " cache_len = k_cache.shape[2]\n", " start_pos = cache_len\n", " \n", " # Пропустите матрицы запроса и ключа через экземпляр rope, чтобы выполнить поворот.\n", " if self._rope is not None:\n", " # ✅ Применяем RoPE к Q и K (НЕ к V!)\n", " q = self._rope(q, start_pos=start_pos) # [B, T, hs]\n", " k = self._rope(k, start_pos=start_pos) # [B, T, hs]\n", "\n", " # Если cache пришел, то объединяем кэш и одну строку из ключа и значения. Это будут новые key и value для последующих вычислений.\n", " # 5. Кэширование (для autoregressive generation)\n", " if cache is not None:\n", " k_cache, v_cache = cache\n", " k = torch.cat([k_cache, k], dim=2) # Concat по seq_len (dim=2)\n", " v = torch.cat([v_cache, v], dim=2)\n", "\n", " # Если use_cache == True, то сохраните матрицы ключа и значения для кэша (это нужно сделать до дублирования голов).\n", " if use_cache == True:\n", " kv_cache = (k, v)\n", "\n", " # Продублируйте головы в тензорах ключа (key) и значения (value), чтобы получился тензор размера на batch_size × num_q_heads × seq_len × head_size.\n", " k = self._repeat_kv_heads(k, self._num_heads, self._num_kv_heads)\n", " v = self._repeat_kv_heads(v, self._num_heads, self._num_kv_heads)\n", "\n", " # Перемножим матрицы запроса и ключа (транспонированную), чтобы вычислить матрицу внимания.\n", " # И разделить все значения в матрице внимания на корень из head_size.\n", " scores = q @ k.transpose(-2, -1) / (self._head_size ** 0.5)\n", "\n", " # Если cache пришел, то маску не накладываем. Иначе наложите на матрицу внимания треугольную маску, созданную при инициализации. Все скрытые значения должны быть приведены к минус бесконечности: float('-inf').\n", " if cache is None:\n", " scores = scores.masked_fill(\n", " ~self._tril_mask[:seq_len, :seq_len], float(\"-inf\")\n", " )\n", "\n", " # Применить к матрице внимания (построчно) функцию Softmax.\n", " weights = F.softmax(scores, dim=-1)\n", "\n", " # Перемножим матрицу внимания и матрицу значения.\n", " x_out = weights @ v # [B, T, hs]\n", "\n", " # Измените форму тензора на batch_size × seq_len × num_heads*head_size.\n", " # Transpose обратно и concatenate heads\n", " x_out = x_out.transpose(1, 2) # [B, T_q, H, hs]\n", " x_out = x_out.contiguous() # Важно для reshape!\n", " concatenated_attention = x_out.reshape(batch_size, seq_len, self._num_heads * self._head_size)\n", "\n", " #concatenated_attention = x_out.reshape(batch_size, seq_len, self._num_heads * self._head_size)\n", "\n", " # Пропустите получившийся тензор через последний линейный слой.\n", " # 3. Проецируем в пространство эмбеддингов\n", " projected_output = self._layer(concatenated_attention)\n", "\n", " # 4. Применяем dropout для регуляризации\n", " final_output = self._dropout(projected_output)\n", "\n", " if use_cache is True:\n", " return (final_output, kv_cache)\n", " else:\n", " return (final_output, None)\n", "\n", " def _repeat_kv_heads(\n", " self,\n", " kv: torch.Tensor,\n", " num_q_heads: int,\n", " num_kv_heads: int\n", " ) -> torch.Tensor:\n", " \"\"\"\n", " Дублирует головы K/V для соответствия количеству голов Q.\n", "\n", " Args:\n", " kv: [batch_size, num_kv_heads, seq_len, head_size]\n", " num_q_heads: Количество голов Query (например, 8)\n", " num_kv_heads: Количество голов Key/Value (например, 2)\n", "\n", " Returns:\n", " [batch_size, num_q_heads, seq_len, head_size]\n", "\n", " Example:\n", " num_q_heads=8, num_kv_heads=2\n", " Каждая голова KV дублируется 4 раза:\n", " [KV0, KV1] -> [KV0, KV0, KV0, KV0, KV1, KV1, KV1, KV1]\n", " \"\"\"\n", " batch_size, num_kv_heads, seq_len, head_size = kv.shape\n", "\n", " if num_q_heads == num_kv_heads:\n", " # Нет необходимости дублировать\n", " return kv\n", "\n", " # Вычисляем сколько раз нужно повторить каждую голову\n", " num_repeats = num_q_heads // num_kv_heads\n", "\n", " # repeat_interleave дублирует каждую голову num_repeats раз\n", " # [B, num_kv_heads, S, hs] -> [B, num_q_heads, S, hs]\n", " # [B, num_kv_heads, S, hs] -> [B, num_kv_heads, 1, S, hs]\n", " kv = kv.unsqueeze(2)\n", " \n", " # [B, num_kv_heads, 1, S, hs] -> [B, num_kv_heads, num_repeats, S, hs]\n", " kv = kv.repeat(1, 1, num_repeats, 1, 1)\n", " \n", " # [B, num_kv_heads, num_repeats, S, hs] -> [B, num_q_heads, S, hs]\n", " kv = kv.reshape(batch_size, num_q_heads, seq_len, head_size)\n", " \n", "\n", " return kv" ] }, { "cell_type": "markdown", "id": "e56522e7", "metadata": {}, "source": [ "### Промежуточный вариант Mistral" ] }, { "cell_type": "code", "execution_count": 7, "id": "c35a8b6f", "metadata": {}, "outputs": [], "source": [ "import torch\n", "from torch import nn\n", "from torch import Tensor\n", "import torch.nn.functional as F\n", "from math import sqrt\n", "\n", "\n", "\n", "class SiLU(nn.Module):\n", " def forward(self, x: torch.Tensor): # [batch_size × seq_len × emb_size]\n", " return torch.sigmoid(x) * x\n", " \n", "class RMSNorm(nn.Module):\n", " def __init__(self, dim: int, eps: float = 1e-6):\n", " super().__init__()\n", " self._eps = eps\n", " self._w = nn.Parameter(torch.ones(dim))\n", " \n", " def forward(self, x: torch.Tensor): # [batch_size × seq_len × emb_size]\n", " rms = (x.pow(2).mean(-1, keepdim=True) + self._eps) ** 0.5\n", " norm_x = x / rms\n", " return self._w * norm_x\n", "\n", "class SwiGLU(nn.Module):\n", " def __init__(self, emb_size: int, dropout: float = 0.1):\n", " super().__init__()\n", "\n", " self._gate = nn.Linear(emb_size, 4 * emb_size)\n", " self._up = nn.Linear(emb_size, 4 * emb_size)\n", " self._down = nn.Linear(4 * emb_size, emb_size)\n", " self._activation = SiLU()\n", " self._dropout = nn.Dropout(dropout)\n", "\n", " def forward(self, x: torch.Tensor): # [batch_size × seq_len × emb_size].\n", " gate_out = self._gate(x) # [batch, seq, 4*emb]\n", " activation_out = self._activation(gate_out) # [batch, seq, 4*emb]\n", " up_out = self._up(x) # [batch, seq, 4*emb]\n", " out = up_out * activation_out # поэлементное!\n", " out = self._down(out) # [batch, seq, emb]\n", " return self._dropout(out)\n", "\n", "\n", "class TokenEmbeddings(nn.Module):\n", " def __init__(self, vocab_size: int, emb_size: int):\n", " super().__init__()\n", " self._embedding = nn.Embedding(\n", " num_embeddings=vocab_size,\n", " embedding_dim=emb_size\n", " )\n", "\n", " def forward(self, x: Tensor) -> Tensor:\n", " return self._embedding(x)\n", "\n", " @property\n", " def num_embeddings(self) -> int:\n", " return self._embedding.num_embeddings\n", "\n", " @property\n", " def embedding_dim(self) -> int:\n", " return self._embedding.embedding_dim\n", "\n", "\n", "class GELU(nn.Module):\n", " def __init__(self):\n", " super().__init__()\n", " self.sqrt_2_over_pi = torch.sqrt(torch.tensor(2.0) / math.pi)\n", " \n", " def forward(self, x: torch.Tensor) -> torch.Tensor:\n", " return 0.5 * x * (1 + torch.tanh(\n", " self.sqrt_2_over_pi * (x + 0.044715 * torch.pow(x, 3))\n", " ))\n", "\n", "\n", "class Decoder(nn.Module):\n", " def __init__(self, \n", " num_q_heads: int,\n", " num_kv_heads: int,\n", " emb_size: int,\n", " head_size: int,\n", " max_seq_len: int,\n", " rope: RoPE,\n", " dropout: float = 0.1\n", " ):\n", " super().__init__()\n", " self._heads = GroupedQueryAttention(\n", " num_heads=num_q_heads, \n", " num_kv_heads=num_kv_heads,\n", " emb_size=emb_size, \n", " head_size=head_size, \n", " max_seq_len=max_seq_len,\n", " rope=rope,\n", " dropout=dropout\n", " )\n", " self._ff = SwiGLU(emb_size=emb_size, dropout=dropout)\n", " self._norm1 = RMSNorm(emb_size)\n", " self._norm2 = RMSNorm(emb_size)\n", "\n", " def forward(self, x: torch.Tensor, mask: torch.Tensor = None, use_cache: bool = True, cache: list = None) -> torch.Tensor:\n", " norm1_out = self._norm1(x)\n", " attention, kv_caches = self._heads(norm1_out, mask, use_cache=use_cache, cache=cache)\n", " out = attention + x\n", " \n", " norm2_out = self._norm2(out)\n", " ffn_out = self._ff(norm2_out)\n", "\n", " if use_cache is True:\n", " return (ffn_out + out, kv_caches)\n", " else:\n", " return (ffn_out + out, None)\n", "\n", "\n", "\n", "from torch import nn\n", "import torch\n", "import torch.nn.functional as F\n", "\n", "class Mistral(nn.Module):\n", " def __init__(self,\n", " vocab_size: int,\n", " max_seq_len: int,\n", " emb_size: int,\n", " num_q_heads: int,\n", " num_kv_heads: int,\n", " head_size: int,\n", " num_layers: int,\n", " dropout: float = 0.1,\n", " device: str = 'cpu'\n", " ):\n", " super().__init__()\n", " self._vocab_size = vocab_size\n", " self._max_seq_len = max_seq_len\n", " self._emb_size = emb_size\n", " self._num_q_heads = num_q_heads\n", " self._num_kv_heads = num_kv_heads\n", " self._head_size = head_size\n", " self._num_layers = num_layers\n", " self._dropout = dropout\n", " self._device = device\n", " \n", " self.validation_loss = None\n", "\n", " # Инициализация слоев\n", " self._token_embeddings = TokenEmbeddings(\n", " vocab_size=vocab_size, \n", " emb_size=emb_size\n", " )\n", " self._position_embeddings = RoPE(\n", " head_size=head_size,\n", " max_seq_len=max_seq_len\n", " )\n", " #self._position_embeddings = PositionalEmbeddings(\n", " # max_seq_len=max_seq_len, \n", " # emb_size=emb_size\n", " #)\n", " self._dropout = nn.Dropout(dropout)\n", " self._decoders = nn.ModuleList([Decoder(\n", " num_q_heads=num_q_heads,\n", " num_kv_heads=num_kv_heads,\n", " emb_size=emb_size,\n", " head_size=head_size,\n", " max_seq_len=max_seq_len,\n", " rope=self._position_embeddings,\n", " dropout=dropout \n", " ) for _ in range(num_layers)])\n", " self._norm = RMSNorm(emb_size)\n", " self._linear = nn.Linear(emb_size, vocab_size)\n", "\n", " def forward(self, x: torch.Tensor, use_cache: bool = True, cache: list = None) -> tuple:\n", " # Проверка длины последовательности (только при отсутствии кэша)\n", " if cache is None and x.size(1) > self._max_seq_len:\n", " raise ValueError(f\"Длина последовательности {x.size(1)} превышает максимальную {self.max_seq_len}\")\n", " \n", " # Эмбеддинги токенов и позиций\n", " tok_out = self._token_embeddings(x) # [batch, seq_len, emb_size]\n", " #pos_out = self._position_embeddings(x) # [batch, seq_len, emb_size]\n", " \n", " # Комбинирование\n", " out = self._dropout(tok_out) # [batch, seq_len, emb_size]\n", " \n", " # Стек декодеров с передачей кэша\n", " new_cache = []\n", " for i, decoder in enumerate(self._decoders):\n", " decoder_cache = cache[i] if cache is not None else None\n", " decoder_result = decoder(out, use_cache=use_cache, cache=decoder_cache)\n", "\n", " # Извлекаем результат из кортежа\n", " if use_cache:\n", " out, decoder_new_cache = decoder_result\n", " new_cache.append(decoder_new_cache)\n", " else:\n", " out = decoder_result[0]\n", "\n", " out = self._norm(out)\n", " logits = self._linear(out)\n", " \n", " # Возвращаем результат с учетом use_cache\n", " if use_cache:\n", " return (logits, new_cache)\n", " else:\n", " return (logits, None)\n", "\n", " def generate(self,\n", " x: torch.Tensor, \n", " max_new_tokens: int, \n", " do_sample: bool,\n", " temperature: float = 1.0,\n", " top_k: int = None,\n", " top_p: float = None,\n", " use_cache: bool = True\n", " ) -> torch.Tensor:\n", " cache = None\n", "\n", " for _ in range(max_new_tokens):\n", " if use_cache and cache is not None:\n", " # Используем кэш - передаем только последний токен\n", " x_input = x[:, -1:] # [batch_size, 1]\n", " else:\n", " # Первая итерация или кэш отключен - передаем всю последовательность\n", " x_input = x\n", " \n", " # Прямой проход с кэшем\n", " logits, new_cache = self.forward(x_input, use_cache=use_cache, cache=cache)\n", " \n", " # Обновляем кэш для следующей итерации\n", " if use_cache:\n", " cache = new_cache\n", "\n", " last_logits = logits[:, -1, :] # [batch_size, vocab_size]\n", "\n", " # Масштабируем логиты температурой\n", " if temperature > 0:\n", " logits_scaled = last_logits / temperature\n", " else:\n", " logits_scaled = last_logits\n", "\n", " if do_sample == True and top_k != None:\n", " _, topk_indices = torch.topk(logits_scaled, top_k, dim=-1)\n", "\n", " # # Заменим все НЕ top-k логиты на -inf\n", " masked_logits = logits_scaled.clone()\n", " vocab_size = logits_scaled.size(-1)\n", "\n", " # создаём маску: 1, если токен НЕ в topk_indices\n", " mask = torch.ones_like(logits_scaled, dtype=torch.uint8)\n", " mask.scatter_(1, topk_indices, 0) # 0 там, где top-k индексы\n", " masked_logits[mask.byte()] = float('-inf')\n", "\n", " logits_scaled = masked_logits\n", "\n", " if do_sample == True and top_p != None:\n", " # 1. Применим softmax, чтобы получить вероятности:\n", " probs = F.softmax(logits_scaled, dim=-1) # [B, vocab_size]\n", " # 2. Отсортируем токены по убыванию вероятностей:\n", " sorted_probs, sorted_indices = torch.sort(probs, descending=True, dim=-1)\n", " # 3. Посчитаем кумулятивную сумму вероятностей:\n", " cum_probs = torch.cumsum(sorted_probs, dim=-1) # [B, vocab_size]\n", " # 4. Определим маску: оставить токены, пока сумма < top_p\n", " sorted_mask = (cum_probs <= top_p).byte() # [B, vocab_size]\n", " # Гарантируем, что хотя бы первый токен останется\n", " sorted_mask[:, 0] = 1\n", " # 5. Преобразуем маску обратно в оригинальный порядок:\n", " # Создаём полную маску из 0\n", " mask = torch.zeros_like(probs, dtype=torch.uint8)\n", " # Устанавливаем 1 в местах нужных токенов\n", " mask.scatter_(dim=1, index=sorted_indices, src=sorted_mask)\n", " # 6. Зануляем логиты токенов вне топ-p:\n", " logits_scaled[~mask] = float('-inf')\n", "\n", " # 4. Применяем Softmax\n", " probs = F.softmax(logits_scaled, dim=-1) # [batch_size, vocab_size]\n", "\n", "\n", " if do_sample == True:\n", " # 5. Если do_sample равен True, то отбираем токен случайно с помощью torch.multinomial\n", " next_token = torch.multinomial(probs, num_samples=1) # [batch_size, 1]\n", " else:\n", " # 5. Если do_sample равен False, то выбираем токен с максимальной вероятностью\n", " next_token = torch.argmax(probs, dim=-1, keepdim=True) # [batch_size, 1]\n", " \n", " # 6. Добавляем его к последовательности\n", " x = torch.cat([x, next_token], dim=1) # [batch_size, seq_len+1]\n", " return x\n", "\n", " def save(self, path):\n", " torch.save({\n", " 'model_state_dict': self.state_dict(),\n", " 'vocab_size': self._vocab_size,\n", " 'max_seq_len': self._max_seq_len,\n", " 'emb_size': self._emb_size,\n", " 'num_heads': self._num_heads,\n", " 'head_size': self._head_size,\n", " 'num_layers': self._num_layers\n", " }, path)\n", "\n", " @classmethod\n", " def load(cls, path, device):\n", " checkpoint = torch.load(path, map_location=device)\n", " model = cls(\n", " vocab_size=checkpoint['vocab_size'],\n", " max_seq_len=checkpoint['max_seq_len'],\n", " emb_size=checkpoint['emb_size'],\n", " num_heads=checkpoint['num_heads'],\n", " head_size=checkpoint['head_size'],\n", " num_layers=checkpoint['num_layers']\n", " )\n", " model.load_state_dict(checkpoint['model_state_dict'])\n", " model.to(device)\n", " return model\n", "\n", " @property\n", " def max_seq_len(self) -> int:\n", " return self._max_seq_len\n", "\n", "\n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "2db40f23", "metadata": {}, "source": [ "# Sliding Window Attention\n", "\n", "Sliding Window Attention (SWA) – это еще один из вариантов сэкономить на механизме внимания.\n", "\n", "Суть его проста: **SWA ограничивает длину видимого контекста** (в механизме внимания) для оптимизации вычислений.\n", "\n", "Чтобы стало понятнее, вспомним урок про внимание — а точнее *матрицу внимания*. Мы накладывали на нее треугольную маску, чтобы каждый токен мог видеть только предыдущие токены. Это нужно было, чтобы воспроизвести реальный инференс, когда модель не видит будущие токены. Но назад эта видимость ограничивалась только максимальной длиной контекста.\n", "\n", "**SWA** же предлагает ограничить еще и видимость токенов назад.\n", "Теперь токены не видят ничего вперед и видят только `n` токенов назад.\n", "\n", "

\n", " \"swa\"\n", "

\n", "\n", "У такого решения есть три преимущества:\n", "\n", "* **Концентрация:** по идее, чем дальше от токена контекст, тем меньше он на него влияет и тем больше шума. И теория гласит, что ограничивая внимание определенным окном, мы тем самым помогаем модели сконцентрироваться на более важных вещах.\n", "* **Вычисления:** для подобных масок разрабатываются специальные CUDA-ядра, благодаря которым в вычислениях участвуют только значения, свободные от маски. В результате мы получаем значительную экономию при инференсе. Нам такая разработка не светит 🙂, но имейте в виду, что в промышленных моделях так и поступают.\n", "* **Кэш:** чем больше мы генерируем текста, тем больше разрастается кэш. И это может стать проблемой. Но при SWA мы смотрим только на определенное количество токенов назад. А значит, нам нужно хранить в кэше не больше токенов, чем задана видимость в SWA." ] }, { "cell_type": "markdown", "id": "208ed348", "metadata": {}, "source": [ "\n", "\n", "# Sliding Window Attention (разработка)\n", "\n", "В класс `GroupedQueryAttention` необходимо внести следующие изменения:\n", "\n", "* Добавьте (перед `dropout`) новый параметр:\n", "\n", " * `window_size` (тип `int`) — определяет, как далеко токены смогут смотреть в прошлое.\n", "* Замените предварительно созданную маску на новую: в ней каждый токен должен видеть только себя и `window_size` предыдущих токенов.\n", "* **Кэш:** при формировании кэша ключа и значения для возврата необходимо обрезать тензор, чтобы остались только последние `window_size` строк.\n", "* **Применение маски.** Теперь у нас есть две версии:\n", "\n", " * Если пришел пустой кэш, то накладывается полная (квадратная) маска.\n", " * Если на вход пришел кэш, то у нас тензор матрицы внимания будет в виде одной строки. Поэтому наложите на матрицу внимания маску размером `[k_seq_len, :k_seq_len]`, где `k_seq_len` — количество строк в матрице ключа после объединения ее с кэшем.\n", " **З.Ы.** Раньше мы оставляли одну строку как есть, т.к. это была последняя строка и она должна была видеть все токены. Но теперь и на одну строку надо также накладывать маску, чтобы ограничить видимость прошлых токенов." ] }, { "cell_type": "code", "execution_count": 8, "id": "66d0d989", "metadata": {}, "outputs": [], "source": [ "import torch\n", "from torch import nn\n", "import torch.nn.functional as F\n", "from typing import Optional, Tuple\n", "\n", "\n", " \n", "class GroupedQueryAttention(nn.Module):\n", "\n", " def __init__(\n", " self,\n", " num_q_heads: int,\n", " num_kv_heads: int,\n", " emb_size: int,\n", " head_size: int,\n", " max_seq_len: int,\n", " window_size: int,\n", " rope: RoPE = None,\n", " dropout: float = 0.1,\n", " ):\n", " super().__init__()\n", " self._num_heads = num_q_heads\n", " self._num_kv_heads = num_kv_heads\n", " self._head_size = head_size\n", " self._max_seq_len = max_seq_len\n", " self._rope = rope\n", " self._window_size = window_size\n", "\n", " self._q = nn.Linear(emb_size, self._num_heads * head_size)\n", " self._k = nn.Linear(emb_size, num_kv_heads * head_size)\n", " self._v = nn.Linear(emb_size, num_kv_heads * head_size)\n", "\n", " # Создание causal маски\n", " mask = self._create_sliding_window_mask(max_seq_len, self._window_size)\n", " self.register_buffer(\n", " \"_tril_mask\", mask.bool() if hasattr(torch, \"bool\") else mask.byte()\n", " )\n", " \n", " self._layer = nn.Linear(head_size * self._num_heads, emb_size)\n", " self._dropout = nn.Dropout(dropout)\n", "\n", " def forward(\n", " self,\n", " x: torch.Tensor,\n", " mask: torch.Tensor = None,\n", " use_cache: bool = True,\n", " cache: list = None,\n", " ):\n", " batch_size, seq_len, emb_size = x.shape\n", "\n", " if seq_len > self._max_seq_len:\n", " raise ValueError(\n", " f\"Длина последовательности {seq_len} превышает максимум {self._max_seq_len}\"\n", " )\n", "\n", " # Пропустите тензор x через матрицы Wq, Wk , Wv, чтобы получить матрицы запроса, ключа и значения.\n", " k = self._k(x) # [B, T, hs]\n", " q = self._q(x) # [B, T, hs]\n", " v = self._v(x) # [B, T, hs]\n", "\n", " # Шаг 2: Изменение формы для multi-head\n", " # [batch_size, seq_len, num_heads * head_size] \n", " # -> [batch_size, seq_len, num_heads, head_size]\n", " # Измените форму запроса (query) на batch_size × num_q_heads × seq_len × head_size.\n", " q = q.reshape(batch_size, seq_len, self._num_heads, self._head_size)\n", "\n", " # Измените форму ключа (key) и значения (value) на batch_size × num_kv_heads × seq_len × head_size.\n", " k = k.reshape(batch_size, seq_len, self._num_kv_heads, self._head_size)\n", " v = v.reshape(batch_size, seq_len, self._num_kv_heads, self._head_size)\n", " \n", "\n", " # 3. Transpose: [B, T, H, hs] -> [B, H, T, hs]\n", " q = q.transpose(1, 2)\n", " k = k.transpose(1, 2)\n", " v = v.transpose(1, 2)\n", "\n", " start_pos = 0\n", " if cache is not None:\n", " k_cache, v_cache = cache\n", " cache_len = k_cache.shape[2]\n", " start_pos = cache_len\n", " \n", " # Пропустите матрицы запроса и ключа через экземпляр rope, чтобы выполнить поворот.\n", " if self._rope is not None:\n", " # Применяем RoPE к Q и K (НЕ к V!)\n", " q = self._rope(q, start_pos=start_pos) # [B, T, hs]\n", " k = self._rope(k, start_pos=start_pos) # [B, T, hs]\n", "\n", " # Если cache пришел, то объединяем кэш и одну строку из ключа и значения. Это будут новые key и value для последующих вычислений.\n", " # 5. Кэширование (для autoregressive generation)\n", " if cache is not None:\n", " k_cache, v_cache = cache\n", " k = torch.cat([k_cache, k], dim=2) # Concat по seq_len (dim=2)\n", " v = torch.cat([v_cache, v], dim=2)\n", "\n", " # Если use_cache == True, то сохраните матрицы ключа и значения для кэша (это нужно сделать до дублирования голов).\n", " #if use_cache == True:\n", " # # Обрезаем до последних window_size токенов\n", " # k_to_cache = k[:, :, -self._window_size:, :]\n", " # v_to_cache = v[:, :, -self._window_size:, :]\n", " # kv_cache = (k_to_cache, v_to_cache)\n", "\n", " # Продублируйте головы в тензорах ключа (key) и значения (value), чтобы получился тензор размера на batch_size × num_q_heads × seq_len × head_size.\n", " #k = self._repeat_kv_heads(k, self._num_heads, self._num_kv_heads)\n", " #v = self._repeat_kv_heads(v, self._num_heads, self._num_kv_heads)\n", " k_expanded = self._repeat_kv_heads(k, self._num_heads, self._num_kv_heads)\n", " v_expanded = self._repeat_kv_heads(v, self._num_heads, self._num_kv_heads)\n", " \n", " # Перемножим матрицы запроса и ключа (транспонированную), чтобы вычислить матрицу внимания.\n", " # И разделить все значения в матрице внимания на корень из head_size.\n", " scores = q @ k_expanded.transpose(-2, -1) / (self._head_size ** 0.5)\n", "\n", " # 8. Применение маски\n", " k_seq_len = k_expanded.size(2) # Длина K после concat с кэшем\n", " \n", " if cache is None:\n", " # Случай 1: Без кэша - полная квадратная маска\n", " # scores: [B, H, seq_len, seq_len]\n", " # Применяем маску [:seq_len, :seq_len]\n", " scores = scores.masked_fill(\n", " ~self._tril_mask[:seq_len, :seq_len], \n", " float(\"-inf\")\n", " )\n", "\n", " # Применить к матрице внимания (построчно) функцию Softmax.\n", " weights = F.softmax(scores, dim=-1)\n", "\n", " # Перемножим матрицу внимания и матрицу значения.\n", " x_out = weights @ v_expanded # [B, T, hs]\n", "\n", " # Измените форму тензора на batch_size × seq_len × num_heads*head_size.\n", " # Transpose обратно и concatenate heads\n", " x_out = x_out.transpose(1, 2) # [B, T_q, H, hs]\n", " x_out = x_out.contiguous() # Важно для reshape!\n", " concatenated_attention = x_out.reshape(batch_size, seq_len, self._num_heads * self._head_size)\n", "\n", " #concatenated_attention = x_out.reshape(batch_size, seq_len, self._num_heads * self._head_size)\n", "\n", " # Пропустите получившийся тензор через последний линейный слой.\n", " # 3. Проецируем в пространство эмбеддингов\n", " projected_output = self._layer(concatenated_attention)\n", "\n", " # 4. Применяем dropout для регуляризации\n", " output = self._dropout(projected_output)\n", "\n", " if use_cache:\n", " # Обрезаем оригинальный K и V (до дублирования)\n", " k_to_cache = k[:, :, -self._window_size:, :]\n", " v_to_cache = v[:, :, -self._window_size:, :]\n", " kv_cache = (k_to_cache, v_to_cache)\n", " return output, kv_cache\n", " else:\n", " return output, None\n", "\n", " def _repeat_kv_heads(\n", " self,\n", " kv: torch.Tensor,\n", " num_q_heads: int,\n", " num_kv_heads: int\n", " ) -> torch.Tensor:\n", " \"\"\"\n", " Дублирует головы K/V для соответствия количеству голов Q.\n", "\n", " Args:\n", " kv: [batch_size, num_kv_heads, seq_len, head_size]\n", " num_q_heads: Количество голов Query (например, 8)\n", " num_kv_heads: Количество голов Key/Value (например, 2)\n", "\n", " Returns:\n", " [batch_size, num_q_heads, seq_len, head_size]\n", "\n", " Example:\n", " num_q_heads=8, num_kv_heads=2\n", " Каждая голова KV дублируется 4 раза:\n", " [KV0, KV1] -> [KV0, KV0, KV0, KV0, KV1, KV1, KV1, KV1]\n", " \"\"\"\n", " batch_size, num_kv_heads, seq_len, head_size = kv.shape\n", "\n", " if num_q_heads == num_kv_heads:\n", " # Нет необходимости дублировать\n", " return kv\n", "\n", " # Вычисляем сколько раз нужно повторить каждую голову\n", " num_repeats = num_q_heads // num_kv_heads\n", "\n", " # repeat_interleave дублирует каждую голову num_repeats раз\n", " # [B, num_kv_heads, S, hs] -> [B, num_q_heads, S, hs]\n", " # [B, num_kv_heads, S, hs] -> [B, num_kv_heads, 1, S, hs]\n", " kv = kv.unsqueeze(2)\n", " \n", " # [B, num_kv_heads, 1, S, hs] -> [B, num_kv_heads, num_repeats, S, hs]\n", " kv = kv.repeat(1, 1, num_repeats, 1, 1)\n", " \n", " # [B, num_kv_heads, num_repeats, S, hs] -> [B, num_q_heads, S, hs]\n", " kv = kv.reshape(batch_size, num_q_heads, seq_len, head_size)\n", " \n", "\n", " return kv\n", "\n", " def _create_sliding_window_mask(\n", " self,\n", " max_seq_len: int,\n", " window_size: int,\n", " device: torch.device = None\n", " ) -> torch.Tensor:\n", " \"\"\"\n", " Создает маску для Sliding Window Attention.\n", "\n", " Args:\n", " max_seq_len: Максимальная длина последовательности\n", " window_size: Размер окна внимания\n", " device: Устройство для размещения тензора\n", "\n", " Returns:\n", " Маска формы [max_seq_len, max_seq_len], где True = разрешено\n", "\n", " Example:\n", " >>> mask = create_sliding_window_mask(8, 3)\n", " >>> print(mask.int())\n", " tensor([[1, 0, 0, 0, 0, 0, 0, 0],\n", " [1, 1, 0, 0, 0, 0, 0, 0],\n", " [1, 1, 1, 0, 0, 0, 0, 0],\n", " [0, 1, 1, 1, 0, 0, 0, 0],\n", " [0, 0, 1, 1, 1, 0, 0, 0],\n", " [0, 0, 0, 1, 1, 1, 0, 0],\n", " [0, 0, 0, 0, 1, 1, 1, 0],\n", " [0, 0, 0, 0, 0, 1, 1, 1]])\n", " \"\"\"\n", " row_indices = torch.arange(max_seq_len, device=device).unsqueeze(1) # [max_seq_len, 1]\n", " col_indices = torch.arange(max_seq_len, device=device).unsqueeze(0) # [1, max_seq_len]\n", "\n", " causal_mask = col_indices <= row_indices\n", "\n", " window_mask = (row_indices - col_indices) <= window_size\n", "\n", " mask = causal_mask & window_mask\n", " \n", " return mask" ] }, { "cell_type": "code", "execution_count": 9, "id": "ed7675ca", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "============================================================\n", "Тест 1: Без кэша (prefill)\n", "============================================================\n", "Input: torch.Size([1, 5, 64])\n", "Output: torch.Size([1, 5, 64])\n", "Cache K: torch.Size([1, 2, 2, 16])\n", "Cache V: torch.Size([1, 2, 2, 16])\n", "\n", "Маска применена: [:5, :5]\n", "tensor([[1, 0, 0, 0, 0],\n", " [1, 1, 0, 0, 0],\n", " [1, 1, 1, 0, 0],\n", " [0, 1, 1, 1, 0],\n", " [0, 0, 1, 1, 1]], dtype=torch.int32)\n", "\n", "============================================================\n", "Тест 2: С кэшем (generation)\n", "============================================================\n", "Input: torch.Size([1, 1, 64])\n", "Output: torch.Size([1, 1, 64])\n", "Cache K: torch.Size([1, 2, 2, 16])\n", "Cache V: torch.Size([1, 2, 2, 16])\n", "\n", "Маска применена: [5:6, :6]\n", "tensor([[0, 0, 0, 1, 1, 1]], dtype=torch.int32)\n", "\n", "============================================================\n", "Тест 3: Генерация еще одного токена\n", "============================================================\n", "Input: torch.Size([1, 1, 64])\n", "Output: torch.Size([1, 1, 64])\n", "Cache K: torch.Size([1, 2, 2, 16])\n", "Cache V: torch.Size([1, 2, 2, 16])\n", "\n", "Маска применена: [6:7, :7]\n", "tensor([[0, 0, 0, 0, 1, 1, 1]], dtype=torch.int32)\n", "\n", "============================================================\n", "Тест 4: Генерация нескольких токенов сразу\n", "============================================================\n", "Input: torch.Size([1, 3, 64])\n", "Output: torch.Size([1, 3, 64])\n", "Cache K: torch.Size([1, 2, 2, 16])\n", "Cache V: torch.Size([1, 2, 2, 16])\n", "\n", "Маска применена: [7:10, :10]\n", "tensor([[0, 0, 0, 0, 0, 1, 1, 1, 0, 0],\n", " [0, 0, 0, 0, 0, 0, 1, 1, 1, 0],\n", " [0, 0, 0, 0, 0, 0, 0, 1, 1, 1]], dtype=torch.int32)\n", "\n", "✅ Все тесты пройдены!\n" ] } ], "source": [ "import torch\n", "from torch import nn\n", "import torch.nn.functional as F\n", "from typing import Optional, Tuple\n", "\n", "\n", "# ... (классы RoPE и GroupedQueryAttention как выше) ...\n", "\n", "# Параметры\n", "batch_size = 1\n", "emb_size = 64\n", "head_size = 16\n", "num_q_heads = 4\n", "num_kv_heads = 2\n", "max_seq_len = 20\n", "window_size = 2\n", "\n", "# Создаем модель\n", "rope = RoPE(head_size=head_size, max_seq_len=max_seq_len)\n", "gqa = GroupedQueryAttention(\n", " num_q_heads=num_q_heads,\n", " num_kv_heads=num_kv_heads,\n", " emb_size=emb_size,\n", " head_size=head_size,\n", " max_seq_len=max_seq_len,\n", " window_size=window_size,\n", " rope=rope,\n", " dropout=0.0,\n", ")\n", "\n", "print(\"=\"*60)\n", "print(\"Тест 1: Без кэша (prefill)\")\n", "print(\"=\"*60)\n", "\n", "x1 = torch.randn(batch_size, 5, emb_size)\n", "output1, cache1 = gqa(x1, use_cache=True)\n", "\n", "print(f\"Input: {x1.shape}\")\n", "print(f\"Output: {output1.shape}\")\n", "print(f\"Cache K: {cache1[0].shape}\") # [1, 2, 5, 16]\n", "print(f\"Cache V: {cache1[1].shape}\") # [1, 2, 5, 16]\n", "\n", "# Проверяем маску\n", "print(f\"\\nМаска применена: [:5, :5]\")\n", "print(gqa._tril_mask[:5, :5].int())\n", "\n", "print(\"\\n\" + \"=\"*60)\n", "print(\"Тест 2: С кэшем (generation)\")\n", "print(\"=\"*60)\n", "\n", "x2 = torch.randn(batch_size, 1, emb_size)\n", "output2, cache2 = gqa(x2, use_cache=True, cache=cache1)\n", "\n", "print(f\"Input: {x2.shape}\")\n", "print(f\"Output: {output2.shape}\")\n", "print(f\"Cache K: {cache2[0].shape}\") # [1, 2, 6, 16]\n", "print(f\"Cache V: {cache2[1].shape}\") # [1, 2, 6, 16]\n", "\n", "# Проверяем маску\n", "k_seq_len = 6\n", "seq_len = 1\n", "start_pos = k_seq_len - seq_len # 5\n", "print(f\"\\nМаска применена: [{start_pos}:{k_seq_len}, :{k_seq_len}]\")\n", "print(gqa._tril_mask[start_pos:k_seq_len, :k_seq_len].int())\n", "\n", "print(\"\\n\" + \"=\"*60)\n", "print(\"Тест 3: Генерация еще одного токена\")\n", "print(\"=\"*60)\n", "\n", "x3 = torch.randn(batch_size, 1, emb_size)\n", "output3, cache3 = gqa(x3, use_cache=True, cache=cache2)\n", "\n", "print(f\"Input: {x3.shape}\")\n", "print(f\"Output: {output3.shape}\")\n", "print(f\"Cache K: {cache3[0].shape}\") # [1, 2, 7, 16]\n", "print(f\"Cache V: {cache3[1].shape}\") # [1, 2, 7, 16]\n", "\n", "# Проверяем маску\n", "k_seq_len = 7\n", "seq_len = 1\n", "start_pos = k_seq_len - seq_len # 6\n", "print(f\"\\nМаска применена: [{start_pos}:{k_seq_len}, :{k_seq_len}]\")\n", "print(gqa._tril_mask[start_pos:k_seq_len, :k_seq_len].int())\n", "\n", "print(\"\\n\" + \"=\"*60)\n", "print(\"Тест 4: Генерация нескольких токенов сразу\")\n", "print(\"=\"*60)\n", "\n", "x4 = torch.randn(batch_size, 3, emb_size)\n", "output4, cache4 = gqa(x4, use_cache=True, cache=cache3)\n", "\n", "print(f\"Input: {x4.shape}\")\n", "print(f\"Output: {output4.shape}\")\n", "print(f\"Cache K: {cache4[0].shape}\") # [1, 2, 10, 16]\n", "print(f\"Cache V: {cache4[1].shape}\") # [1, 2, 10, 16]\n", "\n", "# Проверяем маску\n", "k_seq_len = 10\n", "seq_len = 3\n", "start_pos = k_seq_len - seq_len # 7\n", "print(f\"\\nМаска применена: [{start_pos}:{k_seq_len}, :{k_seq_len}]\")\n", "print(gqa._tril_mask[start_pos:k_seq_len, :k_seq_len].int())\n", "\n", "print(\"\\n✅ Все тесты пройдены!\")" ] }, { "cell_type": "code", "execution_count": 10, "id": "cdba2ba3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "======================================================================\n", "ФИНАЛЬНЫЙ ТЕСТ СООТВЕТСТВИЯ ТЗ\n", "======================================================================\n", "\n", "✅ Тест 1: Маска window_size=3\n", "tensor([[1, 0, 0, 0, 0, 0, 0, 0],\n", " [1, 1, 0, 0, 0, 0, 0, 0],\n", " [1, 1, 1, 0, 0, 0, 0, 0],\n", " [1, 1, 1, 1, 0, 0, 0, 0],\n", " [0, 1, 1, 1, 1, 0, 0, 0],\n", " [0, 0, 1, 1, 1, 1, 0, 0],\n", " [0, 0, 0, 1, 1, 1, 1, 0],\n", " [0, 0, 0, 0, 1, 1, 1, 1]], dtype=torch.int32)\n", "✅ Маска работает правильно!\n", "\n", "✅ Тест 2: Обрезка кэша при prefill\n", "Prefill: 10 токенов\n", "K cache size: 3 (ожидается 3)\n", "✅ Кэш обрезан правильно!\n", "\n", "✅ Тест 3: Кэш не растет при генерации\n", "После 10 шагов генерации:\n", "K cache size: 3 (ожидается 3)\n", "✅ Кэш всегда ограничен window_size!\n", "\n", "✅ Тест 4: Применение маски с кэшем\n", "K seq_len: 4\n", "Start pos: 3\n", "Маска применена: [3:4, :4]\n", "tensor([[1, 1, 1, 1]], dtype=torch.int32)\n", "✅ Маска применяется правильно!\n", "\n", "✅ Тест 5: Без кэширования\n", "✅ Кэш не создается при use_cache=False\n", "\n", "======================================================================\n", "🎉 ВСЕ ТЕСТЫ ПРОЙДЕНЫ! КОД ПОЛНОСТЬЮ СООТВЕТСТВУЕТ ТЗ!\n", "======================================================================\n", "\n", "📊 Итоговая сводка:\n", " ✅ Параметр window_size добавлен\n", " ✅ Sliding window маска работает корректно\n", " ✅ Каждый токен видит себя + window_size предыдущих\n", " ✅ Кэш обрезается до window_size токенов\n", " ✅ Кэш не растет при генерации\n", " ✅ Маска применяется правильно с кэшем и без\n", " ✅ Grouped Query Attention работает\n", " ✅ RoPE применяется корректно\n" ] } ], "source": [ "import torch\n", "\n", "# Параметры\n", "batch_size = 1\n", "emb_size = 64\n", "head_size = 16\n", "num_q_heads = 4\n", "num_kv_heads = 2\n", "max_seq_len = 100\n", "window_size = 3\n", "\n", "# Создаем модель\n", "rope = RoPE(head_size=head_size, max_seq_len=max_seq_len)\n", "gqa = GroupedQueryAttention(\n", " num_q_heads=num_q_heads,\n", " num_kv_heads=num_kv_heads,\n", " emb_size=emb_size,\n", " head_size=head_size,\n", " max_seq_len=max_seq_len,\n", " window_size=window_size,\n", " rope=rope,\n", " dropout=0.0,\n", ")\n", "\n", "print(\"=\"*70)\n", "print(\"ФИНАЛЬНЫЙ ТЕСТ СООТВЕТСТВИЯ ТЗ\")\n", "print(\"=\"*70)\n", "\n", "# Тест 1: Проверка маски\n", "print(f\"\\n✅ Тест 1: Маска window_size={window_size}\")\n", "test_mask = gqa._create_sliding_window_mask(8, 3)\n", "print(test_mask.int())\n", "\n", "# Проверка количества видимых токенов\n", "all_correct = True\n", "for i in range(8):\n", " visible_count = test_mask[i].sum().item()\n", " expected = min(i + 1, window_size + 1)\n", " if visible_count != expected:\n", " all_correct = False\n", " print(f\"❌ Токен {i}: видит {visible_count}, ожидается {expected}\")\n", "\n", "if all_correct:\n", " print(\"✅ Маска работает правильно!\")\n", "\n", "# Тест 2: Обрезка кэша при prefill\n", "print(\"\\n✅ Тест 2: Обрезка кэша при prefill\")\n", "x1 = torch.randn(batch_size, 10, emb_size)\n", "output1, cache1 = gqa(x1, use_cache=True)\n", "\n", "print(f\"Prefill: 10 токенов\")\n", "print(f\"K cache size: {cache1[0].shape[2]} (ожидается {window_size})\")\n", "assert cache1[0].shape[2] == window_size, f\"❌ Кэш должен быть {window_size}\"\n", "print(\"✅ Кэш обрезан правильно!\")\n", "\n", "# Тест 3: Кэш не растет при генерации\n", "print(\"\\n✅ Тест 3: Кэш не растет при генерации\")\n", "cache = cache1\n", "for i in range(10):\n", " x_new = torch.randn(batch_size, 1, emb_size)\n", " output_new, cache = gqa(x_new, use_cache=True, cache=cache)\n", " \n", " assert cache[0].shape[2] == window_size, \\\n", " f\"❌ Шаг {i+1}: кэш {cache[0].shape[2]}, ожидается {window_size}\"\n", "\n", "print(f\"После 10 шагов генерации:\")\n", "print(f\"K cache size: {cache[0].shape[2]} (ожидается {window_size})\")\n", "print(\"✅ Кэш всегда ограничен window_size!\")\n", "\n", "# Тест 4: Применение маски с кэшем\n", "print(\"\\n✅ Тест 4: Применение маски с кэшем\")\n", "x2 = torch.randn(batch_size, 1, emb_size)\n", "output2, cache2 = gqa(x2, use_cache=True, cache=cache1)\n", "\n", "k_seq_len = cache1[0].shape[2] + 1 # 3 + 1 = 4\n", "start_pos = k_seq_len - 1 # 3\n", "expected_mask = gqa._tril_mask[start_pos:k_seq_len, :k_seq_len]\n", "\n", "print(f\"K seq_len: {k_seq_len}\")\n", "print(f\"Start pos: {start_pos}\")\n", "print(f\"Маска применена: [{start_pos}:{k_seq_len}, :{k_seq_len}]\")\n", "print(expected_mask.int())\n", "print(\"✅ Маска применяется правильно!\")\n", "\n", "# Тест 5: Без кэширования\n", "print(\"\\n✅ Тест 5: Без кэширования\")\n", "x_no_cache = torch.randn(batch_size, 5, emb_size)\n", "output_no_cache, cache_no_cache = gqa(x_no_cache, use_cache=False)\n", "\n", "assert cache_no_cache is None, \"❌ Кэш должен быть None\"\n", "print(\"✅ Кэш не создается при use_cache=False\")\n", "\n", "print(\"\\n\" + \"=\"*70)\n", "print(\"🎉 ВСЕ ТЕСТЫ ПРОЙДЕНЫ! КОД ПОЛНОСТЬЮ СООТВЕТСТВУЕТ ТЗ!\")\n", "print(\"=\"*70)\n", "\n", "print(\"\\n📊 Итоговая сводка:\")\n", "print(\" ✅ Параметр window_size добавлен\")\n", "print(\" ✅ Sliding window маска работает корректно\")\n", "print(\" ✅ Каждый токен видит себя + window_size предыдущих\")\n", "print(\" ✅ Кэш обрезается до window_size токенов\")\n", "print(\" ✅ Кэш не растет при генерации\")\n", "print(\" ✅ Маска применяется правильно с кэшем и без\")\n", "print(\" ✅ Grouped Query Attention работает\")\n", "print(\" ✅ RoPE применяется корректно\")" ] }, { "cell_type": "code", "execution_count": 11, "id": "397fd8fd", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "=== Test RoPE ===\n", "Input shape: torch.Size([1, 2, 4, 8])\n", "Output shape: torch.Size([1, 2, 4, 8])\n", "Average norm difference: 0.0\n", "✅ RoPE test passed.\n", "\n", "=== Test Grouped Query Attention ===\n", "Input shape: torch.Size([2, 6, 16])\n", "Output shape: torch.Size([2, 6, 16])\n", "Cache shapes: torch.Size([2, 2, 3, 8]) torch.Size([2, 2, 3, 8])\n", "✅ GQA shape test passed.\n", "\n", "Sliding window mask (1=visible, 0=masked):\n", "tensor([[1, 0, 0, 0, 0, 0],\n", " [1, 1, 0, 0, 0, 0],\n", " [1, 1, 1, 0, 0, 0],\n", " [1, 1, 1, 1, 0, 0],\n", " [0, 1, 1, 1, 1, 0],\n", " [0, 0, 1, 1, 1, 1]], dtype=torch.int32)\n", "✅ Sliding window mask test passed.\n", "\n", "=== Test Cache Behavior ===\n", "Cache1 K shape: torch.Size([1, 2, 2, 8])\n", "Cache2 K shape: torch.Size([1, 2, 3, 8])\n", "✅ Cache test passed.\n" ] } ], "source": [ "import torch\n", "import torch.nn.functional as F\n", "\n", "torch.manual_seed(42)\n", "\n", "# === 1️⃣ Проверка RoPE ===\n", "def test_rope():\n", " print(\"\\n=== Test RoPE ===\")\n", "\n", " head_size = 8\n", " seq_len = 4\n", " num_heads = 2\n", " batch_size = 1\n", "\n", " rope = RoPE(head_size=head_size, max_seq_len=16)\n", "\n", " # [B, H, T, D]\n", " x = torch.randn(batch_size, num_heads, seq_len, head_size)\n", " x_rot = rope(x)\n", "\n", " print(\"Input shape:\", x.shape)\n", " print(\"Output shape:\", x_rot.shape)\n", " assert x.shape == x_rot.shape, \"RoPE: shape mismatch!\"\n", "\n", " # Проверим, что RoPE сохраняет норму (приблизительно)\n", " norm_diff = (x.norm(dim=-1) - x_rot.norm(dim=-1)).abs().mean()\n", " print(\"Average norm difference:\", norm_diff.item())\n", " assert norm_diff < 1e-4, \"RoPE: norms changed too much!\"\n", "\n", " print(\"✅ RoPE test passed.\")\n", "\n", "\n", "# === 2️⃣ Проверка GroupedQueryAttention ===\n", "def test_gqa():\n", " print(\"\\n=== Test Grouped Query Attention ===\")\n", "\n", " emb_size = 16\n", " num_q_heads = 4\n", " num_kv_heads = 2\n", " head_size = 8\n", " max_seq_len = 16\n", " window_size = 3\n", " batch_size = 2\n", " seq_len = 6\n", "\n", " rope = RoPE(head_size=head_size, max_seq_len=max_seq_len)\n", " gqa = GroupedQueryAttention(\n", " num_q_heads=num_q_heads,\n", " num_kv_heads=num_kv_heads,\n", " emb_size=emb_size,\n", " head_size=head_size,\n", " max_seq_len=max_seq_len,\n", " window_size=window_size,\n", " rope=rope,\n", " )\n", "\n", " x = torch.randn(batch_size, seq_len, emb_size)\n", " out, kv_cache = gqa(x, use_cache=True)\n", "\n", " print(\"Input shape:\", x.shape)\n", " print(\"Output shape:\", out.shape)\n", " assert out.shape == (batch_size, seq_len, emb_size), \"GQA output shape mismatch!\"\n", "\n", " print(\"Cache shapes:\", kv_cache[0].shape, kv_cache[1].shape)\n", " assert kv_cache[0].shape == (batch_size, num_kv_heads, window_size, head_size), \"K cache shape mismatch!\"\n", " assert kv_cache[1].shape == (batch_size, num_kv_heads, window_size, head_size), \"V cache shape mismatch!\"\n", "\n", " print(\"✅ GQA shape test passed.\")\n", "\n", " # === Проверка маски ===\n", " mask = gqa._create_sliding_window_mask(seq_len, window_size)\n", " print(\"\\nSliding window mask (1=visible, 0=masked):\")\n", " print(mask.int())\n", "\n", " # Проверим, что токен не видит больше, чем window_size назад\n", " for i in range(seq_len):\n", " visible_positions = mask[i].nonzero().squeeze()\n", " \n", " # Если только одна позиция видима → делаем список из одного элемента\n", " if visible_positions.ndim == 0:\n", " visible_positions = [visible_positions.item()]\n", " else:\n", " visible_positions = visible_positions.tolist()\n", " \n", " max_back = i - min(visible_positions)\n", " assert max_back <= window_size, f\"Token {i} sees too far back!\"\n", "\n", "\n", " print(\"✅ Sliding window mask test passed.\")\n", "\n", "\n", "# === 3️⃣ Проверка на автогенерацию (кэш) ===\n", "def test_cache_behavior():\n", " print(\"\\n=== Test Cache Behavior ===\")\n", "\n", " emb_size = 16\n", " num_q_heads = 4\n", " num_kv_heads = 2\n", " head_size = 8\n", " max_seq_len = 16\n", " window_size = 3\n", " batch_size = 1\n", "\n", " gqa = GroupedQueryAttention(\n", " num_q_heads=num_q_heads,\n", " num_kv_heads=num_kv_heads,\n", " emb_size=emb_size,\n", " head_size=head_size,\n", " max_seq_len=max_seq_len,\n", " window_size=window_size,\n", " rope=None,\n", " )\n", "\n", " # Первый проход (без кэша)\n", " x1 = torch.randn(batch_size, 2, emb_size)\n", " out1, cache1 = gqa(x1, use_cache=True)\n", "\n", " # Второй проход (с кэшем)\n", " x2 = torch.randn(batch_size, 1, emb_size)\n", " out2, cache2 = gqa(x2, use_cache=True, cache=cache1)\n", "\n", " print(\"Cache1 K shape:\", cache1[0].shape)\n", " print(\"Cache2 K shape:\", cache2[0].shape)\n", "\n", " assert cache2[0].shape[-2] == window_size, \"Cache not trimmed correctly!\"\n", " print(\"✅ Cache test passed.\")\n", "\n", "\n", "if __name__ == \"__main__\":\n", " test_rope()\n", " test_gqa()\n", " test_cache_behavior()\n" ] }, { "cell_type": "code", "execution_count": 12, "id": "bb4f9694", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "🧪 Тестирование исправленной маски\n", "\n", "============================================================\n", "Проверка маски\n", "============================================================\n", "\n", "Маска для max_seq_len=8, window_size=3:\n", "tensor([[1, 0, 0, 0, 0, 0, 0, 0],\n", " [1, 1, 0, 0, 0, 0, 0, 0],\n", " [1, 1, 1, 0, 0, 0, 0, 0],\n", " [1, 1, 1, 1, 0, 0, 0, 0],\n", " [0, 1, 1, 1, 1, 0, 0, 0],\n", " [0, 0, 1, 1, 1, 1, 0, 0],\n", " [0, 0, 0, 1, 1, 1, 1, 0],\n", " [0, 0, 0, 0, 1, 1, 1, 1]], dtype=torch.int32)\n", "\n", "============================================================\n", "Тест 1: Без кэша (prefill)\n", "============================================================\n", "Input: torch.Size([1, 5, 64])\n", "Output: torch.Size([1, 5, 64])\n", "Cache K: torch.Size([1, 2, 3, 16])\n", "Cache V: torch.Size([1, 2, 3, 16])\n", "\n", "Маска применена: [:5, :5]\n", "tensor([[1, 0, 0, 0, 0],\n", " [1, 1, 0, 0, 0],\n", " [1, 1, 1, 0, 0],\n", " [1, 1, 1, 1, 0],\n", " [0, 1, 1, 1, 1]], dtype=torch.int32)\n", "\n", "============================================================\n", "Тест 2: С кэшем (generation)\n", "============================================================\n", "Input: torch.Size([1, 1, 64])\n", "Output: torch.Size([1, 1, 64])\n", "Cache K: torch.Size([1, 2, 3, 16])\n", "Cache V: torch.Size([1, 2, 3, 16])\n", "\n", "Маска применена: [5:6, :6]\n", "tensor([[0, 0, 1, 1, 1, 1]], dtype=torch.int32)\n", "\n", "✅ Все тесты пройдены!\n" ] } ], "source": [ "if __name__ == \"__main__\":\n", " print(\"🧪 Тестирование исправленной маски\\n\")\n", " \n", " # Параметры\n", " batch_size = 1\n", " emb_size = 64\n", " head_size = 16\n", " num_q_heads = 4\n", " num_kv_heads = 2\n", " max_seq_len = 20\n", " window_size = 3\n", " \n", " # Создаем модель\n", " rope = RoPE(head_size=head_size, max_seq_len=max_seq_len)\n", " gqa = GroupedQueryAttention(\n", " num_q_heads=num_q_heads,\n", " num_kv_heads=num_kv_heads,\n", " emb_size=emb_size,\n", " head_size=head_size,\n", " max_seq_len=max_seq_len,\n", " window_size=window_size,\n", " rope=rope,\n", " dropout=0.0,\n", " )\n", " \n", " print(\"=\"*60)\n", " print(\"Проверка маски\")\n", " print(\"=\"*60)\n", " print(f\"\\nМаска для max_seq_len=8, window_size=3:\")\n", " test_mask = gqa._create_sliding_window_mask(8, 3)\n", " print(test_mask.int())\n", " \n", " print(\"\\n\" + \"=\"*60)\n", " print(\"Тест 1: Без кэша (prefill)\")\n", " print(\"=\"*60)\n", " \n", " x1 = torch.randn(batch_size, 5, emb_size)\n", " output1, cache1 = gqa(x1, use_cache=True)\n", " \n", " print(f\"Input: {x1.shape}\")\n", " print(f\"Output: {output1.shape}\")\n", " print(f\"Cache K: {cache1[0].shape}\")\n", " print(f\"Cache V: {cache1[1].shape}\")\n", " \n", " print(f\"\\nМаска применена: [:5, :5]\")\n", " print(gqa._tril_mask[:5, :5].int())\n", " \n", " print(\"\\n\" + \"=\"*60)\n", " print(\"Тест 2: С кэшем (generation)\")\n", " print(\"=\"*60)\n", " \n", " x2 = torch.randn(batch_size, 1, emb_size)\n", " output2, cache2 = gqa(x2, use_cache=True, cache=cache1)\n", " \n", " print(f\"Input: {x2.shape}\")\n", " print(f\"Output: {output2.shape}\")\n", " print(f\"Cache K: {cache2[0].shape}\")\n", " print(f\"Cache V: {cache2[1].shape}\")\n", " \n", " k_seq_len = 6\n", " seq_len = 1\n", " start_pos = k_seq_len - seq_len\n", " print(f\"\\nМаска применена: [{start_pos}:{k_seq_len}, :{k_seq_len}]\")\n", " print(gqa._tril_mask[start_pos:k_seq_len, :k_seq_len].int())\n", " \n", " print(\"\\n✅ Все тесты пройдены!\")" ] }, { "cell_type": "markdown", "id": "415dcb2b", "metadata": {}, "source": [ "# Full Model" ] }, { "cell_type": "code", "execution_count": 13, "id": "354f411d", "metadata": {}, "outputs": [], "source": [ "import torch\n", "from torch import nn\n", "from torch import Tensor\n", "import torch.nn.functional as F\n", "from math import sqrt\n", "\n", "\n", "\n", "class SiLU(nn.Module):\n", " def forward(self, x: torch.Tensor): # [batch_size × seq_len × emb_size]\n", " return torch.sigmoid(x) * x\n", " \n", "class RMSNorm(nn.Module):\n", " def __init__(self, dim: int, eps: float = 1e-6):\n", " super().__init__()\n", " self._eps = eps\n", " self._w = nn.Parameter(torch.ones(dim))\n", " \n", " def forward(self, x: torch.Tensor): # [batch_size × seq_len × emb_size]\n", " rms = (x.pow(2).mean(-1, keepdim=True) + self._eps) ** 0.5\n", " norm_x = x / rms\n", " return self._w * norm_x\n", "\n", "class SwiGLU(nn.Module):\n", " def __init__(self, emb_size: int, dropout: float = 0.1):\n", " super().__init__()\n", "\n", " self._gate = nn.Linear(emb_size, 4 * emb_size)\n", " self._up = nn.Linear(emb_size, 4 * emb_size)\n", " self._down = nn.Linear(4 * emb_size, emb_size)\n", " self._activation = SiLU()\n", " self._dropout = nn.Dropout(dropout)\n", "\n", " def forward(self, x: torch.Tensor): # [batch_size × seq_len × emb_size].\n", " gate_out = self._gate(x) # [batch, seq, 4*emb]\n", " activation_out = self._activation(gate_out) # [batch, seq, 4*emb]\n", " up_out = self._up(x) # [batch, seq, 4*emb]\n", " out = up_out * activation_out # поэлементное!\n", " out = self._down(out) # [batch, seq, emb]\n", " return self._dropout(out)\n", "\n", "\n", "class TokenEmbeddings(nn.Module):\n", " def __init__(self, vocab_size: int, emb_size: int):\n", " super().__init__()\n", " self._embedding = nn.Embedding(\n", " num_embeddings=vocab_size,\n", " embedding_dim=emb_size\n", " )\n", "\n", " def forward(self, x: Tensor) -> Tensor:\n", " return self._embedding(x)\n", "\n", " @property\n", " def num_embeddings(self) -> int:\n", " return self._embedding.num_embeddings\n", "\n", " @property\n", " def embedding_dim(self) -> int:\n", " return self._embedding.embedding_dim\n", "\n", "\n", "class GELU(nn.Module):\n", " def __init__(self):\n", " super().__init__()\n", " self.sqrt_2_over_pi = torch.sqrt(torch.tensor(2.0) / math.pi)\n", " \n", " def forward(self, x: torch.Tensor) -> torch.Tensor:\n", " return 0.5 * x * (1 + torch.tanh(\n", " self.sqrt_2_over_pi * (x + 0.044715 * torch.pow(x, 3))\n", " ))\n", "\n", " \n", "class Decoder(nn.Module):\n", " def __init__(self, \n", " num_q_heads: int,\n", " num_kv_heads: int,\n", " emb_size: int,\n", " head_size: int,\n", " max_seq_len: int,\n", " window_size: int,\n", " rope: RoPE,\n", " dropout: float = 0.1\n", " ):\n", " super().__init__()\n", " self._heads = GroupedQueryAttention(\n", " num_q_heads=num_q_heads, \n", " num_kv_heads=num_kv_heads,\n", " emb_size=emb_size, \n", " head_size=head_size, \n", " max_seq_len=max_seq_len,\n", " window_size=window_size,\n", " rope=rope,\n", " dropout=dropout\n", " )\n", " self._ff = SwiGLU(emb_size=emb_size, dropout=dropout)\n", " self._norm1 = RMSNorm(emb_size)\n", " self._norm2 = RMSNorm(emb_size)\n", "\n", " def forward(self, x: torch.Tensor, mask: torch.Tensor = None, use_cache: bool = True, cache: list = None) -> torch.Tensor:\n", " norm1_out = self._norm1(x)\n", " attention, kv_caches = self._heads(norm1_out, mask, use_cache=use_cache, cache=cache)\n", " out = attention + x\n", " \n", " norm2_out = self._norm2(out)\n", " ffn_out = self._ff(norm2_out)\n", "\n", " if use_cache is True:\n", " return (ffn_out + out, kv_caches)\n", " else:\n", " return (ffn_out + out, None)\n", "\n", "\n", "\n", "from torch import nn\n", "import torch\n", "import torch.nn.functional as F\n", "\n", "class Mistral(nn.Module):\n", " def __init__(self,\n", " vocab_size: int,\n", " max_seq_len: int,\n", " emb_size: int,\n", " num_q_heads: int,\n", " num_kv_heads: int,\n", " head_size: int,\n", " num_layers: int,\n", " window_size: int,\n", " dropout: float = 0.1,\n", " device: str = 'cpu'\n", " ):\n", " super().__init__()\n", " self._vocab_size = vocab_size\n", " self._max_seq_len = max_seq_len\n", " self._emb_size = emb_size\n", " self._num_q_heads = num_q_heads\n", " self._num_kv_heads = num_kv_heads\n", " self._head_size = head_size\n", " self._num_layers = num_layers\n", " self._dropout = dropout\n", " self._device = device\n", " \n", " self.validation_loss = None\n", "\n", " # Инициализация слоев\n", " self._token_embeddings = TokenEmbeddings(\n", " vocab_size=vocab_size, \n", " emb_size=emb_size\n", " )\n", " self._position_embeddings = RoPE(\n", " head_size=head_size,\n", " max_seq_len=max_seq_len\n", " )\n", " #self._position_embeddings = PositionalEmbeddings(\n", " # max_seq_len=max_seq_len, \n", " # emb_size=emb_size\n", " #)\n", " self._dropout = nn.Dropout(dropout)\n", " self._decoders = nn.ModuleList([Decoder(\n", " num_q_heads=num_q_heads,\n", " num_kv_heads=num_kv_heads,\n", " emb_size=emb_size,\n", " head_size=head_size,\n", " max_seq_len=max_seq_len,\n", " window_size=window_size,\n", " rope=self._position_embeddings,\n", " dropout=dropout \n", " ) for _ in range(num_layers)])\n", " self._norm = RMSNorm(emb_size)\n", " self._linear = nn.Linear(emb_size, vocab_size)\n", "\n", " def forward(self, x: torch.Tensor, use_cache: bool = True, cache: list = None) -> tuple:\n", " # Проверка длины последовательности (только при отсутствии кэша)\n", " if cache is None and x.size(1) > self._max_seq_len:\n", " raise ValueError(f\"Длина последовательности {x.size(1)} превышает максимальную {self.max_seq_len}\")\n", " \n", " # Эмбеддинги токенов и позиций\n", " tok_out = self._token_embeddings(x) # [batch, seq_len, emb_size]\n", " #pos_out = self._position_embeddings(x) # [batch, seq_len, emb_size]\n", " \n", " # Комбинирование\n", " out = self._dropout(tok_out) # [batch, seq_len, emb_size]\n", " \n", " # Стек декодеров с передачей кэша\n", " new_cache = []\n", " for i, decoder in enumerate(self._decoders):\n", " decoder_cache = cache[i] if cache is not None else None\n", " decoder_result = decoder(out, use_cache=use_cache, cache=decoder_cache)\n", "\n", " # Извлекаем результат из кортежа\n", " if use_cache:\n", " out, decoder_new_cache = decoder_result\n", " new_cache.append(decoder_new_cache)\n", " else:\n", " out = decoder_result[0]\n", "\n", " out = self._norm(out)\n", " logits = self._linear(out)\n", " \n", " # Возвращаем результат с учетом use_cache\n", " if use_cache:\n", " return (logits, new_cache)\n", " else:\n", " return (logits, None)\n", "\n", " def generate(self,\n", " x: torch.Tensor, \n", " max_new_tokens: int, \n", " do_sample: bool,\n", " temperature: float = 1.0,\n", " top_k: int = None,\n", " top_p: float = None,\n", " use_cache: bool = True\n", " ) -> torch.Tensor:\n", " cache = None\n", "\n", " for _ in range(max_new_tokens):\n", " if use_cache and cache is not None:\n", " # Используем кэш - передаем только последний токен\n", " x_input = x[:, -1:] # [batch_size, 1]\n", " else:\n", " # Первая итерация или кэш отключен - передаем всю последовательность\n", " x_input = x\n", " \n", " # Прямой проход с кэшем\n", " logits, new_cache = self.forward(x_input, use_cache=use_cache, cache=cache)\n", " \n", " # Обновляем кэш для следующей итерации\n", " if use_cache:\n", " cache = new_cache\n", "\n", " last_logits = logits[:, -1, :] # [batch_size, vocab_size]\n", "\n", " # Масштабируем логиты температурой\n", " if temperature > 0:\n", " logits_scaled = last_logits / temperature\n", " else:\n", " logits_scaled = last_logits\n", "\n", " if do_sample == True and top_k != None:\n", " _, topk_indices = torch.topk(logits_scaled, top_k, dim=-1)\n", "\n", " # # Заменим все НЕ top-k логиты на -inf\n", " masked_logits = logits_scaled.clone()\n", " vocab_size = logits_scaled.size(-1)\n", "\n", " # создаём маску: 1, если токен НЕ в topk_indices\n", " mask = torch.ones_like(logits_scaled, dtype=torch.uint8)\n", " mask.scatter_(1, topk_indices, 0) # 0 там, где top-k индексы\n", " masked_logits[mask.byte()] = float('-inf')\n", "\n", " logits_scaled = masked_logits\n", "\n", " if do_sample == True and top_p != None:\n", " # 1. Применим softmax, чтобы получить вероятности:\n", " probs = F.softmax(logits_scaled, dim=-1) # [B, vocab_size]\n", " # 2. Отсортируем токены по убыванию вероятностей:\n", " sorted_probs, sorted_indices = torch.sort(probs, descending=True, dim=-1)\n", " # 3. Посчитаем кумулятивную сумму вероятностей:\n", " cum_probs = torch.cumsum(sorted_probs, dim=-1) # [B, vocab_size]\n", " # 4. Определим маску: оставить токены, пока сумма < top_p\n", " sorted_mask = (cum_probs <= top_p).byte() # [B, vocab_size]\n", " # Гарантируем, что хотя бы первый токен останется\n", " sorted_mask[:, 0] = 1\n", " # 5. Преобразуем маску обратно в оригинальный порядок:\n", " # Создаём полную маску из 0\n", " mask = torch.zeros_like(probs, dtype=torch.uint8)\n", " # Устанавливаем 1 в местах нужных токенов\n", " mask.scatter_(dim=1, index=sorted_indices, src=sorted_mask)\n", " # 6. Зануляем логиты токенов вне топ-p:\n", " logits_scaled[~mask] = float('-inf')\n", "\n", " # 4. Применяем Softmax\n", " probs = F.softmax(logits_scaled, dim=-1) # [batch_size, vocab_size]\n", "\n", "\n", " if do_sample == True:\n", " # 5. Если do_sample равен True, то отбираем токен случайно с помощью torch.multinomial\n", " next_token = torch.multinomial(probs, num_samples=1) # [batch_size, 1]\n", " else:\n", " # 5. Если do_sample равен False, то выбираем токен с максимальной вероятностью\n", " next_token = torch.argmax(probs, dim=-1, keepdim=True) # [batch_size, 1]\n", " \n", " # 6. Добавляем его к последовательности\n", " x = torch.cat([x, next_token], dim=1) # [batch_size, seq_len+1]\n", " return x\n", "\n", " def save(self, path):\n", " torch.save({\n", " 'model_state_dict': self.state_dict(),\n", " 'vocab_size': self._vocab_size,\n", " 'max_seq_len': self._max_seq_len,\n", " 'emb_size': self._emb_size,\n", " 'num_heads': self._num_heads,\n", " 'head_size': self._head_size,\n", " 'num_layers': self._num_layers\n", " }, path)\n", "\n", " @classmethod\n", " def load(cls, path, device):\n", " checkpoint = torch.load(path, map_location=device)\n", " model = cls(\n", " vocab_size=checkpoint['vocab_size'],\n", " max_seq_len=checkpoint['max_seq_len'],\n", " emb_size=checkpoint['emb_size'],\n", " num_heads=checkpoint['num_heads'],\n", " head_size=checkpoint['head_size'],\n", " num_layers=checkpoint['num_layers']\n", " )\n", " model.load_state_dict(checkpoint['model_state_dict'])\n", " model.to(device)\n", " return model\n", "\n", " @property\n", " def max_seq_len(self) -> int:\n", " return self._max_seq_len" ] }, { "cell_type": "markdown", "id": "e0e8b89b", "metadata": {}, "source": [] }, { "cell_type": "markdown", "id": "a0971548", "metadata": {}, "source": [ "## 2. Обучение Mistral\n", "\n", "Mistral обучается в два этапа:\n", "\n", "- 1️⃣ **Предобучение (Unsupervised Pretraining)** \n", "- 2️⃣ **Дообучение (Supervised Fine-Tuning)**" ] }, { "cell_type": "markdown", "id": "7678a3c0", "metadata": {}, "source": [ "\n", "\n", "### 5.1 Предобучение\n", "\n", "На первом этапе модель обучается без разметки: она получает большой корпус текстов и учится **предсказывать следующий токен** по предыдущим.\n", "\n", "Функция потерь:\n", "$$\n", "L = - \\sum_{t=1}^{T} \\log P(x_t | x_1, x_2, ..., x_{t-1})\n", "$$\n", "\n", "Таким образом, модель учится строить вероятностную модель языка, \"угадывая\" продолжение текста.\n" ] }, { "cell_type": "markdown", "id": "9d06b817", "metadata": {}, "source": [ "Во время **предобучения** Mistral учится **предсказывать следующий токен** (language modeling task). \n", "Формально: \n", "$$ \n", "P(x_t ,|, x_1, x_2, \\dots, x_{t-1}) \n", "$$ \n", "То есть, если на вход подаётся предложение `\"I love deep\"`, модель должна предсказать `\"learning\"`.\n" ] }, { "cell_type": "markdown", "id": "84ab62de", "metadata": {}, "source": [ "### ✅ 5.1.1 Подготовка данных\n", "\n", "Создадим **датасет** на основе BPE-токенизатора:" ] }, { "cell_type": "markdown", "id": "97e2d9bf", "metadata": {}, "source": [ "**BPE Tokenizator**" ] }, { "cell_type": "code", "execution_count": 14, "id": "b15f5284", "metadata": {}, "outputs": [], "source": [ "class BPE:\n", " def __init__(self, vocab_size: int):\n", " self.vocab_size = vocab_size\n", " self.id2token = {}\n", " self.token2id = {}\n", "\n", " def fit(self, text: str):\n", " # 1. Получаем уникальные токены (символы)\n", " unique_tokens = sorted(set(text))\n", " tokens = unique_tokens.copy()\n", "\n", " # 2. Разбиваем текст на токены-символы\n", " sequence = list(text)\n", "\n", " # 3. Объединяем токены до достижения нужного размера словаря\n", " while len(tokens) < self.vocab_size:\n", " #print(f'len={len(tokens)} < {self.vocab_size}')\n", " # Считаем частоты пар\n", " pair_freq = {}\n", " for i in range(len(sequence) - 1):\n", " pair = (sequence[i], sequence[i + 1])\n", " #print(f'pair = {pair}')\n", " if pair not in pair_freq:\n", " pair_freq[pair] = 0\n", " pair_freq[pair] += 1\n", "\n", "\n", " #print(f'pair_freq = {pair_freq}') \n", " if not pair_freq:\n", " break # нет пар — выходим\n", "\n", " #for x in pair_freq.items():\n", " # self.debug(x, sequence)\n", "\n", " # Находим самую частую пару (в случае равенства — та, что встретилась первой)\n", " most_frequent_pair = max(pair_freq.items(), key=lambda x: (x[1], -self._pair_first_index(sequence, x[0])))[0]\n", " #print(most_frequent_pair)\n", " # Создаем новый токен\n", " new_token = most_frequent_pair[0] + most_frequent_pair[1]\n", " #print(f\"new token={new_token}\")\n", " tokens.append(new_token)\n", " #print(f\"tokens={tokens}\")\n", "\n", " i = 0\n", " new_sequence = []\n", "\n", " while i < len(sequence):\n", " if i < len(sequence) - 1 and (sequence[i], sequence[i + 1]) == most_frequent_pair:\n", " new_sequence.append(new_token)\n", " i += 2 # пропускаем два символа — заменённую пару\n", " else:\n", " new_sequence.append(sequence[i])\n", " i += 1\n", " sequence = new_sequence\n", " #break\n", " \n", " # 4. Создаем словари\n", " self.vocab = tokens.copy()\n", " self.token2id = dict(zip(tokens, range(self.vocab_size)))\n", " self.id2token = dict(zip(range(self.vocab_size), tokens))\n", "\n", " def _pair_first_index(self, sequence, pair):\n", " for i in range(len(sequence) - 1):\n", " if (sequence[i], sequence[i + 1]) == pair:\n", " return i\n", " return float('inf') # если пара не найдена (в теории не должно случиться)\n", "\n", "\n", " def encode(self, text: str):\n", " # 1. Разбиваем текст на токены-символы\n", " sequence = list(text)\n", " # 2. Инициализация пустого списка токенов\n", " tokens = []\n", " # 3. Установить i = 0\n", " i = 0\n", " while i < len(text):\n", " # 3.1 Найти все токены в словаре, начинающиеся с text[i]\n", " start_char = text[i]\n", " result = [token for token in self.vocab if token.startswith(start_char)]\n", " # 3.2 Выбрать самый длинный подходящий токен\n", " find_token = self._find_max_matching_token(text[i:], result)\n", " if find_token is None:\n", " # Обработка неизвестного символа\n", " tokens.append(text[i]) # Добавляем сам символ как токен\n", " i += 1\n", " else:\n", " # 3.3 Добавить токен в результат\n", " tokens.append(find_token)\n", " # 3.4 Увеличить i на длину токена\n", " i += len(find_token)\n", "\n", " # 4. Заменить токены на их ID\n", " return self._tokens_to_ids(tokens)\n", "\n", " def _find_max_matching_token(self, text: str, tokens: list):\n", " \"\"\"Находит самый длинный токен из списка, с которого начинается текст\"\"\"\n", " matching = [token for token in tokens if text.startswith(token)]\n", " return max(matching, key=len) if matching else None\n", "\n", " def _tokens_to_ids(self, tokens):\n", " \"\"\"Конвертирует список токенов в их ID с обработкой неизвестных токенов\"\"\"\n", " ids = []\n", " for token in tokens:\n", " if token in self.token2id:\n", " ids.append(self.token2id[token])\n", " else:\n", " ids.append(0) # Специальное значение\n", " return ids\n", "\n", "\n", " def decode(self, ids: list) -> str:\n", " return ''.join(self._ids_to_tokens(ids))\n", "\n", " def _ids_to_tokens(self, ids: list) -> list:\n", " \"\"\"Конвертирует список Ids в их tokens\"\"\"\n", " tokens = []\n", " for id in ids:\n", " if id in self.id2token:\n", " tokens.append(self.id2token[id])\n", " else:\n", " tokens.append('') # Специальное значение\n", " return tokens\n", "\n", "\n", " def save(self, filename):\n", " with open(filename, 'wb') as f:\n", " dill.dump(self, f)\n", " print(f\"Объект сохранён в {filename}\")\n", "\n", "\n", " @classmethod\n", " def load(cls, filename):\n", " with open(filename, 'rb') as f:\n", " obj = dill.load(f)\n", " \n", " print(f\"Объект загружен из {filename}\")\n", " return obj" ] }, { "cell_type": "code", "execution_count": 15, "id": "5d8c6e5e", "metadata": {}, "outputs": [], "source": [ "import torch\n", "from torch.utils.data import Dataset, DataLoader\n", "\n", "class GPTDataset(Dataset):\n", " def __init__(self, text: str, bpe: BPE, block_size: int):\n", " self.bpe = bpe\n", " self.block_size = block_size\n", " self.data = bpe.encode(text)\n", " \n", " def __len__(self):\n", " return len(self.data) - self.block_size\n", "\n", " def __getitem__(self, idx):\n", " x = torch.tensor(self.data[idx:idx+self.block_size], dtype=torch.long)\n", " y = torch.tensor(self.data[idx+1:idx+self.block_size+1], dtype=torch.long)\n", " return x, y" ] }, { "cell_type": "markdown", "id": "1a6f99e8", "metadata": {}, "source": [ "### ✅ 5.1.2 Цикл обучения\n", "\n", "Для обучения создадим функцию:" ] }, { "cell_type": "code", "execution_count": 16, "id": "54f91bc4", "metadata": {}, "outputs": [], "source": [ "import torch.nn.functional as F\n", "from torch import optim\n", "\n", "def train_mistral(model, dataset, epochs=5, batch_size=32, lr=3e-4, device='cpu'):\n", " dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)\n", " optimizer = optim.AdamW(model.parameters(), lr=lr)\n", "\n", " model.to(device)\n", " model.train()\n", "\n", " for epoch in range(epochs):\n", " total_loss = 0\n", " for x, y in dataloader:\n", " x, y = x.to(device), y.to(device)\n", "\n", " # Прямой проход\n", " logits, _ = model(x, use_cache=False) # [B, T, vocab_size]\n", "\n", " # Перестроим выход под CrossEntropy\n", " loss = F.cross_entropy(logits.view(-1, logits.size(-1)), y.view(-1))\n", "\n", " # Обратное распространение\n", " optimizer.zero_grad()\n", " loss.backward()\n", " optimizer.step()\n", "\n", " total_loss += loss.item()\n", "\n", " avg_loss = total_loss / len(dataloader)\n", " print(f\"Epoch {epoch+1}/{epochs}, Loss: {avg_loss:.4f}\")\n", "\n", " return model" ] }, { "cell_type": "markdown", "id": "1da71cc1", "metadata": {}, "source": [ "### ✅ 5.1.3 Пример запуска\n", "\n", "\n", "**🧠 Конфигурация Mistral Mini**\n", "\n", "\n", "| Параметр | Значение | Описание |\n", "| --------------- | -------- | --------------------------------------------- |\n", "| **vocab_size** | `50257` | Размер словаря (BPE токенизатор OpenAI) |\n", "| **max_seq_len** | `512` | Максимальная длина входной последовательности |\n", "| **emb_size** | `256` | Размер эмбеддингов (векторное пространство) |\n", "| **num_heads** | `4` | Количество голов в multi-head attention |\n", "| **head_size** | `64` | Размерность одной головы внимания (768 / 12) |\n", "| **num_layers** | `4` | Количество блоков (декодеров) |\n", "| **dropout** | `0.1` | Вероятность дропаута |\n" ] }, { "cell_type": "code", "execution_count": 17, "id": "09f0af4f", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dataset length: 20\n", "Epoch 1/100, Loss: 3.6991\n", "Epoch 2/100, Loss: 1.5456\n", "Epoch 3/100, Loss: 0.6310\n", "Epoch 4/100, Loss: 0.3419\n", "Epoch 5/100, Loss: 0.2353\n", "Epoch 6/100, Loss: 0.1615\n", "Epoch 7/100, Loss: 0.1222\n", "Epoch 8/100, Loss: 0.1117\n", "Epoch 9/100, Loss: 0.0890\n", "Epoch 10/100, Loss: 0.0788\n", "Epoch 11/100, Loss: 0.0793\n", "Epoch 12/100, Loss: 0.0612\n", "Epoch 13/100, Loss: 0.0724\n", "Epoch 14/100, Loss: 0.0654\n", "Epoch 15/100, Loss: 0.0873\n", "Epoch 16/100, Loss: 0.0840\n", "Epoch 17/100, Loss: 0.0755\n", "Epoch 18/100, Loss: 0.0572\n", "Epoch 19/100, Loss: 0.0663\n", "Epoch 20/100, Loss: 0.0741\n", "Epoch 21/100, Loss: 0.0635\n", "Epoch 22/100, Loss: 0.0649\n", "Epoch 23/100, Loss: 0.0579\n", "Epoch 24/100, Loss: 0.0617\n", "Epoch 25/100, Loss: 0.0626\n", "Epoch 26/100, Loss: 0.0591\n", "Epoch 27/100, Loss: 0.0580\n", "Epoch 28/100, Loss: 0.0514\n", "Epoch 29/100, Loss: 0.0572\n", "Epoch 30/100, Loss: 0.0567\n", "Epoch 31/100, Loss: 0.0595\n", "Epoch 32/100, Loss: 0.0523\n", "Epoch 33/100, Loss: 0.0508\n", "Epoch 34/100, Loss: 0.0494\n", "Epoch 35/100, Loss: 0.0505\n", "Epoch 36/100, Loss: 0.0588\n", "Epoch 37/100, Loss: 0.0511\n", "Epoch 38/100, Loss: 0.0520\n", "Epoch 39/100, Loss: 0.0507\n", "Epoch 40/100, Loss: 0.0514\n", "Epoch 41/100, Loss: 0.0499\n", "Epoch 42/100, Loss: 0.0514\n", "Epoch 43/100, Loss: 0.0485\n", "Epoch 44/100, Loss: 0.0604\n", "Epoch 45/100, Loss: 0.0512\n", "Epoch 46/100, Loss: 0.0535\n", "Epoch 47/100, Loss: 0.0508\n", "Epoch 48/100, Loss: 0.0631\n", "Epoch 49/100, Loss: 0.0541\n", "Epoch 50/100, Loss: 0.0552\n", "Epoch 51/100, Loss: 0.0533\n", "Epoch 52/100, Loss: 0.0538\n", "Epoch 53/100, Loss: 0.0464\n", "Epoch 54/100, Loss: 0.0499\n", "Epoch 55/100, Loss: 0.0524\n", "Epoch 56/100, Loss: 0.0457\n", "Epoch 57/100, Loss: 0.0467\n", "Epoch 58/100, Loss: 0.0459\n", "Epoch 59/100, Loss: 0.0497\n", "Epoch 60/100, Loss: 0.0505\n", "Epoch 61/100, Loss: 0.0493\n", "Epoch 62/100, Loss: 0.0446\n", "Epoch 63/100, Loss: 0.0542\n", "Epoch 64/100, Loss: 0.0438\n", "Epoch 65/100, Loss: 0.0485\n", "Epoch 66/100, Loss: 0.0518\n", "Epoch 67/100, Loss: 0.0478\n", "Epoch 68/100, Loss: 0.0532\n", "Epoch 69/100, Loss: 0.0459\n", "Epoch 70/100, Loss: 0.0497\n", "Epoch 71/100, Loss: 0.0451\n", "Epoch 72/100, Loss: 0.0481\n", "Epoch 73/100, Loss: 0.0428\n", "Epoch 74/100, Loss: 0.0420\n", "Epoch 75/100, Loss: 0.0474\n", "Epoch 76/100, Loss: 0.0461\n", "Epoch 77/100, Loss: 0.0459\n", "Epoch 78/100, Loss: 0.0488\n", "Epoch 79/100, Loss: 0.0429\n", "Epoch 80/100, Loss: 0.0462\n", "Epoch 81/100, Loss: 0.0457\n", "Epoch 82/100, Loss: 0.0428\n", "Epoch 83/100, Loss: 0.0506\n", "Epoch 84/100, Loss: 0.0456\n", "Epoch 85/100, Loss: 0.0497\n", "Epoch 86/100, Loss: 0.0499\n", "Epoch 87/100, Loss: 0.0465\n", "Epoch 88/100, Loss: 0.0526\n", "Epoch 89/100, Loss: 0.0434\n", "Epoch 90/100, Loss: 0.0477\n", "Epoch 91/100, Loss: 0.0446\n", "Epoch 92/100, Loss: 0.0426\n", "Epoch 93/100, Loss: 0.0464\n", "Epoch 94/100, Loss: 0.0481\n", "Epoch 95/100, Loss: 0.0461\n", "Epoch 96/100, Loss: 0.0426\n", "Epoch 97/100, Loss: 0.0418\n", "Epoch 98/100, Loss: 0.0493\n", "Epoch 99/100, Loss: 0.0430\n", "Epoch 100/100, Loss: 0.0531\n" ] }, { "data": { "text/plain": [ "Mistral(\n", " (_token_embeddings): TokenEmbeddings(\n", " (_embedding): Embedding(100, 256)\n", " )\n", " (_position_embeddings): RoPE()\n", " (_dropout): Dropout(p=0.1, inplace=False)\n", " (_decoders): ModuleList(\n", " (0-3): 4 x Decoder(\n", " (_heads): GroupedQueryAttention(\n", " (_rope): RoPE()\n", " (_q): Linear(in_features=256, out_features=256, bias=True)\n", " (_k): Linear(in_features=256, out_features=128, bias=True)\n", " (_v): Linear(in_features=256, out_features=128, bias=True)\n", " (_layer): Linear(in_features=256, out_features=256, bias=True)\n", " (_dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (_ff): SwiGLU(\n", " (_gate): Linear(in_features=256, out_features=1024, bias=True)\n", " (_up): Linear(in_features=256, out_features=1024, bias=True)\n", " (_down): Linear(in_features=1024, out_features=256, bias=True)\n", " (_activation): SiLU()\n", " (_dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (_norm1): RMSNorm()\n", " (_norm2): RMSNorm()\n", " )\n", " )\n", " (_norm): RMSNorm()\n", " (_linear): Linear(in_features=256, out_features=100, bias=True)\n", ")" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 1. Исходный текст\n", "text = \"Deep learning is amazing. Transformers changed the world. Attention is all you need. GPT models revolutionized NLP.\"\n", "\n", "# 2. Обучаем токенизатор\n", "bpe = BPE(vocab_size=100)\n", "bpe.fit(text)\n", "\n", "# 3. Создаем датасет\n", "dataset = GPTDataset(text, bpe, block_size=8)\n", "print(f\"Dataset length: {len(dataset)}\")\n", "\n", "# 4. Инициализируем модель\n", "model = Mistral(\n", " vocab_size=len(bpe.vocab), # размер словаря BPE\n", " max_seq_len=512, # GPT-2 использует контекст в 512 токена\n", " emb_size=256, # размер эмбеддингов\n", " num_q_heads=4, # количество голов внимания\n", " num_kv_heads=2, # количество голов внимания\n", " head_size=64, # размер каждой головы (256 / 4)\n", " num_layers=4, # количество блоков Transformer\n", " window_size=8,\n", " dropout=0.1 # стандартный dropout GPT-2\n", ")\n", "\n", "# 5. Обучаем\n", "train_mistral(model, dataset, epochs=100, batch_size=4)" ] }, { "cell_type": "markdown", "id": "4ba32188", "metadata": {}, "source": [ "\n", "---\n", "\n", "### 5.2 Дообучение\n", "\n", "После предобучения LLAMA уже знает структуру и грамматику языка. \n", "На втором этапе она дообучается на конкретных задачах (например, классификация, QA) с помощью размеченных данных.\n", "\n", "Технически это почти то же обучение, только:\n", "\n", "- Загружаем модель с уже обученными весами.\n", "- Используем новые данные.\n", "- Можно уменьшить скорость обучения.\n", "- Иногда замораживают часть слоёв (например, эмбеддинги).\n" ] }, { "cell_type": "code", "execution_count": 18, "id": "3858eb46", "metadata": {}, "outputs": [], "source": [ "def fine_tune_mistral(model, dataset, epochs=3, batch_size=16, lr=1e-5, device='cpu', freeze_embeddings=True):\n", " if freeze_embeddings:\n", " for param in model._token_embeddings.parameters():\n", " param.requires_grad = False\n", " for param in model._position_embeddings.parameters():\n", " param.requires_grad = False\n", "\n", " dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)\n", " optimizer = optim.AdamW(filter(lambda p: p.requires_grad, model.parameters()), lr=lr)\n", "\n", " model.to(device)\n", " model.train()\n", "\n", " for epoch in range(epochs):\n", " total_loss = 0\n", " for x, y in dataloader:\n", " x, y = x.to(device), y.to(device)\n", " logits, _ = model(x, use_cache=False)\n", " loss = F.cross_entropy(logits.view(-1, logits.size(-1)), y.view(-1))\n", " optimizer.zero_grad()\n", " loss.backward()\n", " optimizer.step()\n", " total_loss += loss.item()\n", " print(f\"Fine-tune Epoch {epoch+1}/{epochs}, Loss: {total_loss / len(dataloader):.4f}\")" ] }, { "cell_type": "code", "execution_count": 19, "id": "f5e3e33b", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Fine-tune Epoch 1/10, Loss: 4.8431\n", "Fine-tune Epoch 2/10, Loss: 2.6429\n", "Fine-tune Epoch 3/10, Loss: 1.6542\n", "Fine-tune Epoch 4/10, Loss: 1.2143\n", "Fine-tune Epoch 5/10, Loss: 0.9998\n", "Fine-tune Epoch 6/10, Loss: 0.8404\n", "Fine-tune Epoch 7/10, Loss: 0.6827\n", "Fine-tune Epoch 8/10, Loss: 0.5871\n", "Fine-tune Epoch 9/10, Loss: 0.5183\n", "Fine-tune Epoch 10/10, Loss: 0.4528\n" ] } ], "source": [ "# Например, мы хотим дообучить модель на стиле коротких технических фраз\n", "fine_tune_text = \"\"\"\n", "Transformers revolutionize NLP.\n", "Deep learning enables self-attention.\n", "GPT generates text autoregressively.\n", "\"\"\"\n", "\n", "dataset = GPTDataset(fine_tune_text, bpe, block_size=8)\n", "\n", "\n", "# Запуск дообучения\n", "fine_tune_mistral(model, dataset, epochs=10, batch_size=4, lr=1e-4)" ] }, { "cell_type": "markdown", "id": "4d9ec9b6", "metadata": {}, "source": [ "## 📝 6. Генерация текста после обучения" ] }, { "cell_type": "code", "execution_count": 20, "id": "f6c7e9f0", "metadata": {}, "outputs": [], "source": [ "def generate_text(model, bpe, prompt: str, max_new_tokens=20, device='cpu'):\n", " model.eval()\n", " ids = torch.tensor([bpe.encode(prompt)], dtype=torch.long).to(device)\n", " out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=True)\n", " text = bpe.decode(out[0].tolist())\n", " return text" ] }, { "cell_type": "code", "execution_count": 21, "id": "ce5b9003", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Deep learning ena les self ti att a\n" ] } ], "source": [ "print(generate_text(model, bpe, \"Deep learning\", max_new_tokens=20))" ] }, { "cell_type": "code", "execution_count": null, "id": "7401082d", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.9" } }, "nbformat": 4, "nbformat_minor": 5 }