{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "mWGJJFnQhflL"
},
"source": [
"<style>\n",
" .justified {\n",
" text-align: justify;\n",
" text-justify: inter-word;\n",
" }\n",
"</style>\n",
"\n",
"<div class=\"justified\">\n",
"\n",
"# Elementos de procesamiento de lenguajes naturales, parte II\n",
"\n",
"Siguiendo nuestra lección anterior, optimizaremos nuestro modelo de red neuronal para crear nombres. Ahora, lo haremos al estilo de Yoshua Bengio {cite}`Bengio2000`."
]
},
{
"cell_type": "code",
"execution_count": 286,
"metadata": {
"id": "20vcSVe-RXia"
},
"outputs": [],
"source": [
"import torch\n",
"import torch.nn.functional as F\n",
"from torch import nn\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import os\n",
"import requests\n",
"from pathlib import Path"
]
},
{
"cell_type": "code",
"execution_count": 287,
"metadata": {
"id": "2wiy1BkyVoK6"
},
"outputs": [],
"source": [
"path = Path('data/')\n",
"if not path.is_dir():\n",
" path.mkdir(parents=True, exist_ok=True)\n",
"\n",
"with open(path / 'nombres.txt', 'wb') as f:\n",
" request = requests.get('https://github.com/DanteNoguez/CalculusRatiocinator/raw/main/data/nombres.txt')\n",
" f.write(request.content)"
]
},
{
"cell_type": "code",
"execution_count": 288,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "I_nJ3lXeWVII",
"outputId": "e644144c-8e42-4ce4-c026-054f43f84668"
},
"outputs": [
{
"data": {
"text/plain": [
"['maria', 'rosa', 'jose', 'carmen', 'ana', 'juana', 'antonio', 'elena']"
]
},
"execution_count": 288,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"nombres = open('data/nombres.txt', 'r').read().splitlines()\n",
"nombres[:8]"
]
},
{
"cell_type": "code",
"execution_count": 289,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "okwXbLowXOQl",
"outputId": "9d63074e-d763-43cf-8573-b17d4b9bdb67"
},
"outputs": [
{
"data": {
"text/plain": [
"21029"
]
},
"execution_count": 289,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(nombres)"
]
},
{
"cell_type": "code",
"execution_count": 290,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "FOuM5vdCXkVd",
"outputId": "34adb3ca-d5ab-4714-e0d0-bd7cf79a8d27"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e', 6: 'f', 7: 'g', 8: 'h', 9: 'i', 10: 'j', 11: 'k', 12: 'l', 13: 'm', 14: 'n', 15: 'o', 16: 'p', 17: 'q', 18: 'r', 19: 's', 20: 't', 21: 'u', 22: 'v', 23: 'w', 24: 'x', 25: 'y', 26: 'z', 0: '.'}\n"
]
}
],
"source": [
"V = sorted(set(''.join(nombres)))\n",
"paf = {p:f+1 for f, p in enumerate(V)}\n",
"paf['.'] = 0\n",
"fap = {f:p for p,f in paf.items()}\n",
"print(fap)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "n2O-rwNLp0YM"
},
"source": [
"### Un modelo neuronal probabilístico de lenguaje\n",
"\n",
"Primero, comenzaremos dividiendo nuestros datos en «bloques». Por ejemplo, en nuestro modelo de bigramas, el bloque contenía un solo carácter, puesto que realizábamos la predicción a partir de una letra; pero podemos aumentar el «contexto» de nuestras predicciones para involucrar más letras al momento de predecir la siguiente. Veamos, por ejemplo, cómo luciría nuestro tratamiento de los datos si hiciéramos bloques de tres caracteres para predecir el siguiente:"
]
},
{
"cell_type": "code",
"execution_count": 291,
"metadata": {
"id": "9N7EnXZTZgco"
},
"outputs": [],
"source": [
"def construir_dataset(nombres):\n",
" block_size = 3 # longitud del contexto\n",
" X, Y = [], []\n",
" for n in nombres:\n",
" #print(f'nombre: {n}')\n",
" contexto = [0] * block_size\n",
" for c in n + '.':\n",
" ix = paf[c]\n",
" X.append(contexto)\n",
" Y.append(ix)\n",
" #print(''.join(fap[i] for i in contexto), '----> ', fap[ix])\n",
" contexto = contexto[1:] + [ix]\n",
" \n",
" X = torch.tensor(X) # contexto\n",
" Y = torch.tensor(Y) # objetivo\n",
" return X, Y"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eH9gpAd8DT69"
},
"source": [
"```{margin}\n",
"“The training set is a sequence $w_1 · · · w_T$ of words $w_t \\in V$, where the vocabulary $V$ is a large but finite set.”\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "q-tv_YsjwIfh",
"outputId": "99a8c5e9-444b-40be-b3d9-e5fae9f15ced"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"nombre: maria\n",
"... ----> m\n",
"..m ----> a\n",
".ma ----> r\n",
"mar ----> i\n",
"ari ----> a\n",
"ria ----> .\n",
"nombre: rosa\n",
"... ----> r\n",
"..r ----> o\n",
".ro ----> s\n",
"ros ----> a\n",
"osa ----> .\n",
"nombre: jose\n",
"... ----> j\n",
"..j ----> o\n",
".jo ----> s\n",
"jos ----> e\n",
"ose ----> .\n"
]
},
{
"data": {
"text/plain": [
"(tensor([[ 0, 0, 0],\n",
" [ 0, 0, 13],\n",
" [ 0, 13, 1],\n",
" [13, 1, 18],\n",
" [ 1, 18, 9],\n",
" [18, 9, 1],\n",
" [ 0, 0, 0],\n",
" [ 0, 0, 18],\n",
" [ 0, 18, 15],\n",
" [18, 15, 19],\n",
" [15, 19, 1],\n",
" [ 0, 0, 0],\n",
" [ 0, 0, 10],\n",
" [ 0, 10, 15],\n",
" [10, 15, 19],\n",
" [15, 19, 5]]),\n",
" tensor([13, 1, 18, 9, 1, 0, 18, 15, 19, 1, 0, 10, 15, 19, 5, 0]))"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"construir_dataset(nombres[:3])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dO_nn97fssxp"
},
"source": [
"Nuestra intuición detrás de esta aproximación es que el lenguaje funciona mejor con contexto: así como el sentido de un concepto se entiende mejor en contexto, también los caracteres se pueden predecir más razonablemente dado un contexto más amplio. \n",
"\n",
"Similar a como habíamos hecho anteriormente, construimos una matriz `X` para contener el contexto como entrada y un vector `Y` que contiene el objetivo (es decir, carácter) que debe seguir a cada respectivo contexto. Como se puede apreciar, solamente estamos construyendo `X` e `Y` con sus respectivos índices del vocabulario.\n",
"\n",
"Dado que solo tomamos 3 nombres como ejemplo, nuestros datos únicamente contienen 16 contextos o *inputs* y 16 objetivos o *outputs*:"
]
},
{
"cell_type": "code",
"execution_count": 292,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "UnQbjfrAvkdi",
"outputId": "71f74b62-77a0-4efe-a8dc-4c25d272b68d"
},
"outputs": [
{
"data": {
"text/plain": [
"(torch.Size([16, 3]), torch.Size([16]))"
]
},
"execution_count": 292,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X, Y = construir_dataset(nombres[:3])\n",
"X.shape, Y.shape"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0mwDZ-E5wpGu"
},
"source": [
"Ahora estamos listos para hacer el *embedding*. Mientras que en el *paper* los datos se incrustan en una tabla de consulta de 30 dimensiones (o *features*) para un vocabulario de 17,000 palabras, nosotros —que únicamente tenemos un vocabulario de 27 caracteres— podemos aproximarnos a la incrustación con algo más pequeño, como una incrustación de dos dimensiones ($m = 2$)."
]
},
{
"cell_type": "code",
"execution_count": 293,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "tGlTifYhzqv7",
"outputId": "d83d47d5-b28d-481a-b689-dadfc4c44d95"
},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[ 1.1950, -1.6664],\n",
" [ 1.0195, -0.4103],\n",
" [-0.2143, 1.6073],\n",
" [ 1.3569, 1.3905],\n",
" [ 0.7943, -0.9118],\n",
" [ 0.4180, 0.3789],\n",
" [-1.1457, 0.4822],\n",
" [-0.5390, -1.8550],\n",
" [ 0.1015, 1.0336],\n",
" [ 0.1822, 1.7465],\n",
" [ 1.0465, 0.5108],\n",
" [-0.3649, 1.3349],\n",
" [-1.6279, -1.2823],\n",
" [ 0.3673, -1.0746],\n",
" [ 1.4023, -0.8277],\n",
" [ 1.2148, -0.7885],\n",
" [ 0.3532, -0.8317],\n",
" [ 0.4932, 0.6096],\n",
" [-1.4151, 0.7786],\n",
" [ 3.8467, -1.3610],\n",
" [ 1.9704, -0.0231],\n",
" [-0.5216, -0.7434],\n",
" [ 0.9543, -0.3574],\n",
" [ 1.0646, -0.6760],\n",
" [-0.2655, -0.1184],\n",
" [ 2.4975, 0.1847],\n",
" [-2.5699, 0.6519]])"
]
},
"execution_count": 293,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"C = torch.randn((27, 2)) # tabla de consulta\n",
"C"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KXQ0eItb0cAM"
},
"source": [
"Como vemos, cada *token* o elemento del volcabulario se incrustará en dos dimensiones, es decir, tendrá dos números asociados. Ahora, tomaremos un atajo que nos permitirá ser más eficientes con la codificación y la primera capa de la red neuronal. Anteriormente, habíamos hecho un *one-hot encoding* para luego pasarlo por una capa `W`; pero, bien visto, estos dos pasos pueden ser omitidos porque consiguen el mismo resultado que la incrustación en nuestra tabla `C`. \n",
"\n",
"Primero: la codificación *one-hot*, si fuéramos a multiplicarla por `C`, anularía todos los valores de `C` al multiplicarlos por 0 y conservaría una fila correspondiente a la de la multiplicación por 1. Ergo, podemos omitir la multiplicación y hacer una indexación para asociar directamente cada carácter con cada fila que un vector *one-hot* multiplicaría por 1. Dicho esto, podemos concebir a `C` como un equivalente de la capa `W`, puesto que consiste de valores aleatorios que asignan un número a cada carácter y luego pueden optimizarse con propagación hacia atrás.\n",
"\n",
"Dicho esto, la indexación (*embedding*) será bastante simple:\n",
"\n",
"```{margin}\n",
"“A mapping $C$ from any element $i$ of $V$ to a real vector $C(i) \\in \\mathbb{R}^m$. It represents the distributed feature vectors associated with each word in the vocabulary. In practice, C is represented by a $\\left|V\\right| \\times m$ matrix of free parameters”.\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 295,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 37
},
"id": "AWRbIvaQAJbc",
"outputId": "b4fda9e0-b821-41e0-cf39-a833e5b0f306"
},
"outputs": [
{
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"'Segunda fila de C: [ 1.0194591 -0.41030166] | Tercer valor del tercer bloque incrustado (es decir, letra a): [ 1.0194591 -0.41030166]'"
]
},
"execution_count": 295,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"emb = C[X] # embedding\n",
"\n",
"f'Segunda fila de C: {C[1].numpy()} | Tercer valor del tercer bloque incrustado (es decir, letra a): {emb[2][2].numpy()}'"
]
},
{
"cell_type": "code",
"execution_count": 296,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "2wVT6iiNIale",
"outputId": "6018f44f-0831-4c6b-975b-3af72e68023a"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Dimensiones del embedding: torch.Size([16, 3, 2])\n",
"Tres bloques del embedding, correspondientes a «..m», «.ma» y «mar»: tensor([[[ 1.1950, -1.6664],\n",
" [ 1.1950, -1.6664],\n",
" [ 0.3673, -1.0746]],\n",
"\n",
" [[ 1.1950, -1.6664],\n",
" [ 0.3673, -1.0746],\n",
" [ 1.0195, -0.4103]],\n",
"\n",
" [[ 0.3673, -1.0746],\n",
" [ 1.0195, -0.4103],\n",
" [-1.4151, 0.7786]]])\n"
]
}
],
"source": [
"print('Dimensiones del embedding: ', emb.shape) \n",
"print('Tres bloques del embedding, correspondientes a «..m», «.ma» y «mar»:', emb[1:4])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Mvh5GMkLIWsk"
},
"source": [
"Ahora, echemos un vistazo a la arquitectura que deseamos lograr:\n",
"\n",
"```{figure} ../../img/bengio2003.png\n",
"---\n",
"width: 70%\n",
"name: bengio2003\n",
"---\n",
"Arquitectura neuronal $f\\left(i, w_{t-1}, \\cdots, w_{t-n+1}\\right)=g\\left(i, C\\left(w_{t-1}\\right), \\cdots, C\\left(w_{t-n+1}\\right)\\right)$ donde $g$ es la red neuronal y $C(i)$ es el $i$-ésimo vector de cada palabra. En nuestro caso, utilizamos bloques de tres letras (`contexto`, vector `X`) en lugar de palabras ($w$).\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Ln__Tj3MQz1p"
},
"source": [
"Como vemos, tenemos casi terminado el inicio y únicamente nos falta concatenar entre sí los bloques del *embedding*, puesto que juntos atravesarán la misma capa de neuronas.\n",
"```{margin}\n",
"“\\[...] $x$ is the word features layer activation vector, which is the concatenation of the input word features from the matrix $C$: $x = C\\left(w_{t-1}\\right), \\left(w_{t-2}\\right) \\cdots, C\\left(w_{t-n+1}\\right)$.”\n",
"```\n",
"Para conseguirlo, podemos utilizar distintos métodos con PyTorch:"
]
},
{
"cell_type": "code",
"execution_count": 297,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "llevzXqoY8kG",
"outputId": "1269073c-3523-4f13-c6c9-b40b8d7fc82d"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Ahora, en lugar de estar contenidos en bloques de tres filas: \n",
"tensor([[ 1.1950, -1.6664],\n",
" [ 1.1950, -1.6664],\n",
" [ 1.1950, -1.6664]])\n",
"Estarían contenidos en bloques de una fila (seis columnas):\n",
"tensor([ 1.1950, -1.6664, 1.1950, -1.6664, 1.1950, -1.6664])\n"
]
}
],
"source": [
"metodo1 = torch.cat([emb[:, 0, :], emb[:, 1, :], emb[:, 2, :]], 1)\n",
"print(f\"\"\"Ahora, en lugar de estar contenidos en bloques de tres filas: \n",
"{emb[0]}\n",
"Estarían contenidos en bloques de una fila (seis columnas):\n",
"{metodo1[0]}\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 298,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 37
},
"id": "NoRTuW6YaZox",
"outputId": "b47cc42a-1797-4072-cd9f-3bb540a4f9ab"
},
"outputs": [
{
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"'Aunque también es equivalente: tensor([ 1.1950, -1.6664, 1.1950, -1.6664, 1.1950, -1.6664])'"
]
},
"execution_count": 298,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"metodo2 = torch.cat(torch.unbind(emb, 1), 1)\n",
"f'Aunque también es equivalente: {metodo2[0]}'"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BBMagWewbQ2-"
},
"source": [
"Pero el método más eficiente[^1] y simple es `view`. Como primer argumento, colocaremos `-1` para que de esta forma PyTorch infiera el tamaño de la dimensión 0 que debería tener el tensor (sería, pues, equivalente a colocar `emb[0].shape`), y como segundo argumento `6` porque queremos que el tensor tenga las 6 columnas correspondientes a un bloque de tres *tokens*:"
]
},
{
"cell_type": "code",
"execution_count": 299,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "PBCWB06samzE",
"outputId": "67e761d7-b78b-4694-afc1-035b2bc5a023"
},
"outputs": [
{
"data": {
"text/plain": [
"(tensor([ 1.1950, -1.6664, 1.1950, -1.6664, 1.1950, -1.6664]),\n",
" torch.Size([16, 6]))"
]
},
"execution_count": 299,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"emb.view(-1, 6)[0], emb.view(-1, 6).shape # esta es la variable x del paper"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aQkp7vTli1RI"
},
"source": [
"Ahora ya tenemos lo suficiente para definir más variables:\n",
"\n",
"> “Let $h$ be the number of hidden units[^2], and $m$ the number of features associated with each word. When no direct connections from word features to outputs are desired, the matrix $W$ is set to 0 . The free parameters of the model are the output biases $b$ (with $|V|$ elements), the hidden layer biases $d$ (with $h$ elements), the hidden-to-output weights $U$ (a $|V| \\times h$ matrix), the word features to output weights $W$ (a $|V| \\times(n-1) m$ matrix), the hidden layer weights $H$ (a $h \\times(n-1) m$ matrix), and the word features $C$(a $|V| \\times m$ matrix $)$: $\\theta=(b, d, W, U, H, C)$”."
]
},
{
"cell_type": "code",
"execution_count": 320,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 37
},
"id": "wGjpIQCJTQHT",
"outputId": "48fac1c7-c81f-4ccd-8323-cec4e7a8dcc3"
},
"outputs": [
{
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"'Número de features (m), es decir, número de componentes de cada bloque: 2 | Número de elementos por bloque «(n-1)m»: 6 | Elementos de |V|: 27'"
]
},
"execution_count": 320,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"f'Número de features (m), es decir, número de componentes de cada bloque: {C.size(dim=1)} | Número de elementos por bloque «(n-1)m»: {emb.view(-1, 6).size(dim=1)} | Elementos de |V|: {len(V)+1}'"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YvTldl3y1cnp"
},
"source": [
"El número de parámetros ($h$) depende del problema a tratar: generalmente, a mayor cantidad de datos, es mejor mayor cantidad de parámetros. En general, cuestiones técnicas como esta dependen de la evaluación experimental que hagamos de nuestro modelo: tras pruebas con diferentes números de parámetros, podemos elegir la que más efectiva y eficiente sea. En el *paper*, por ejemplo, probaron con 50 y 100 *hidden units*. Dado que por el momento solo estamos ejemplificando con tres nombres, $h = 50$ unidades serán suficientes.\n",
"\n",
"Por otra parte, fijar un valor de 0 a $W$ es igual que no utilizarla, de manera que omitiremos su definición porque no la necesitamos."
]
},
{
"cell_type": "code",
"execution_count": 301,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "vn986Q_bRgr2",
"outputId": "7905e35e-f4f1-42b7-dc7e-72037e798201"
},
"outputs": [
{
"data": {
"text/plain": [
"torch.Size([16, 50])"
]
},
"execution_count": 301,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"h = 50\n",
"d = torch.randn((h))\n",
"H = torch.randn((6, h))\n",
"\n",
"a = torch.tanh(emb.view(-1, 6) @ H + d)\n",
"a.shape"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CiZ4E0jsVs0F"
},
"source": [
"Ahora, la capa oculta previa al *output*, es decir, la *hidden-to-output layer* se compone de $U$ y $b$, mientras que el resultado de esta capa son —el lector lo recordará— lo que llamamos *logits*, es decir, el resultado de la última capa de la red neuronal. Estos *logits* serán convertidos en probabilidades y, con ello, tendremos el *output* de toda la red. "
]
},
{
"cell_type": "code",
"execution_count": 302,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "9ctI-j3zV2K0",
"outputId": "87cc97c0-dc98-4d1d-c755-982d9782f021"
},
"outputs": [
{
"data": {
"text/plain": [
"torch.Size([16, 27])"
]
},
"execution_count": 302,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"U = torch.randn((h, 27))\n",
"b = torch.randn(27)\n",
"\n",
"logits = a @ U + b\n",
"logits.shape"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vf6uqdGxf07U"
},
"source": [
"Ahora, para convertir esto en probabilidades, haremos lo mismo que en la lección pasada:"
]
},
{
"cell_type": "code",
"execution_count": 303,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "2UZCsFTOf4vc",
"outputId": "fe61659c-7598-4cea-bc95-d09121fa3d77"
},
"outputs": [
{
"data": {
"text/plain": [
"tensor(1.0000)"
]
},
"execution_count": 303,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"counts = logits.exp()\n",
"prob = counts / counts.sum(1, keepdims=True)\n",
"prob[0].sum()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FOU_daFOf5_c"
},
"source": [
"Para crear nuestra función de pérdida, necesitamos seleccionar nuestros objetivos (con base en el índice que nos dio `Y`):"
]
},
{
"cell_type": "code",
"execution_count": 304,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "XesO8cnLgFsB",
"outputId": "8dec22cf-4eba-4aa2-a488-2fbad6b7c2ea"
},
"outputs": [
{
"data": {
"text/plain": [
"tensor([2.7201e-07, 1.4083e-02, 5.6987e-07, 3.5629e-07, 9.4318e-10, 4.6116e-08,\n",
" 2.3890e-10, 9.0728e-04, 2.6708e-06, 2.7046e-05, 1.8723e-08, 2.6864e-07,\n",
" 7.6290e-06, 8.3047e-06, 1.6912e-09, 1.4596e-08])"
]
},
"execution_count": 304,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"prob[torch.arange(16), Y]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vAogwu906LYo"
},
"source": [
"Ejemplifiquemos: si nuestro *input* es «mar» (`emb[3]`), nuestra probabilidad debe ser alta para que el número del *embedding* que represente la letra «i» de «María» sea un *output*:"
]
},
{
"cell_type": "code",
"execution_count": 305,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "gzP5D6cn7C6-",
"outputId": "7f8d6938-5562-4b40-d502-8bb24394500b"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"En un inicio, cada letra de «mar» correspondía a su índice en el vocabulario: tensor([13, 1, 18])\n",
"Cuando pasamos estos índices a una matriz para que fueran representados por dos números, obtuvimos: tensor([[ 0.3673, -1.0746],\n",
" [ 1.0195, -0.4103],\n",
" [-1.4151, 0.7786]])\n",
"Al mismo tiempo, la letra «i» fue guardada como objetivo en Y, siendo su índice: 9\n",
"De manera que la probabilidad de nuestra red neuronal debe ser alta para el número que representa la letra «i» en el embedding: -3.5628877981253027e-07\n"
]
}
],
"source": [
"print(f\"\"\"En un inicio, cada letra de «mar» correspondía a su índice en el vocabulario: {X[3]}\n",
"Cuando pasamos estos índices a una matriz para que fueran representados por dos números, obtuvimos: {emb[3]}\n",
"Al mismo tiempo, la letra «i» fue guardada como objetivo en Y, siendo su índice: {Y[3]}\n",
"De manera que la probabilidad de nuestra red neuronal debe ser alta para el número que representa la letra «i» en el embedding: {-prob[3, Y[3]]}\"\"\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8_8s5YhMgrDD"
},
"source": [
"Dado que no hemos entrenado la red, la probabilidad anterior es muy baja. Para poder entrenarla, formularemos la función de pérdida mediante el promedio del logaritmo natural de las probabilidades que la red neuronal asigna a cada objetivo, y finalmente hacemos de este un número positivo al multiplicarlo por $-1$, igual que en la lección pasada:"
]
},
{
"cell_type": "code",
"execution_count": 306,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "t-V-YLiMg9Qi",
"outputId": "d711c17f-9ba6-408c-8191-0661f1b86f43"
},
"outputs": [
{
"data": {
"text/plain": [
"tensor(14.5898)"
]
},
"execution_count": 306,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"perdida = -prob[torch.arange(16), Y].log().mean()\n",
"perdida"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MMDFftuZiZuJ"
},
"source": [
"Usando PyTorch, podemos simplificar todo este proceso mediante el uso de `cross_entropy`:"
]
},
{
"cell_type": "code",
"execution_count": 307,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "gHcV_mJPigMl",
"outputId": "55c3a084-9acc-4439-b80c-6c34ee82ffcb"
},
"outputs": [
{
"data": {
"text/plain": [
"tensor(14.5898)"
]
},
"execution_count": 307,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"F.cross_entropy(logits, Y)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YccvWoeSutKG"
},
"source": [
"Finalmente, agrupamos los parámetros —$\\theta$— en una variable:"
]
},
{
"cell_type": "code",
"execution_count": 308,
"metadata": {
"id": "SozrJQzlu6Le"
},
"outputs": [],
"source": [
"parametros = [b, d, U, H, C]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5sMEdaOk9Syz"
},
"source": [
"Indicamos a PyTorch que requeriremos gradientes para el entrenamiento de nuestros parámetros:"
]
},
{
"cell_type": "code",
"execution_count": 309,
"metadata": {
"id": "zr_TdESFvm1p"
},
"outputs": [],
"source": [
"for p in parametros:\n",
" p.requires_grad = True"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jdrFZM0m9a7P"
},
"source": [
"Finalmente, podemos entrenar la red:"
]
},
{
"cell_type": "code",
"execution_count": 310,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "07Gi5EJpweFt",
"outputId": "406185a2-86e6-417f-916d-c540323a7c99"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.25268855690956116\n"
]
}
],
"source": [
"for i in range(100):\n",
" # paso hacia delante\n",
" emb = C[X]\n",
" h = torch.tanh(emb.view(-1, 6) @ H + d)\n",
" logits = h @ U + b\n",
" perdida = F.cross_entropy(logits, Y)\n",
" \n",
" # propagación hacia atrás\n",
" for p in parametros:\n",
" p.grad = None\n",
" perdida.backward()\n",
" \n",
" for p in parametros:\n",
" p.data += -0.1 * p.grad\n",
"\n",
"print(perdida.item())"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SrSZQ3lkEQLL"
},
"source": [
"Bien, hemos conseguido un resultado decente en nuestro ejemplo. Ahora es momento de entrenar nuestra red con todos los nombres a la vez. \n",
"\n",
"### Optimización de modelos neuronales: conjuntos de datos, sobreajuste, tasa de aprendizaje y lotes\n",
"\n",
"Primero hacen falta algunos ajustes. Por una parte, evitaremos que nuestro modelo «memorice» o «sobreajuste» cada *output* correspondiente a cada *input* (a este sobreajuste se le llama *overfitting*), puesto que queremos nombres nuevos y originales y no una regurgitación de los que estamos utilizando. Para ello, se han desarrollado un número de técnicas en *deep learning*; pero de momento, utilizaremos una de las más elementales, y consiste en separar nuestros datos en tres partes: entrenamiento, validación y prueba. Entrenaremos nuestro modelo con una cantidad razonable de nombres (el 80 % de ellos), definimos nuestros parámetros —en rigor, hiperparámetros[^3]— con los datos de validación (10 % de nuestros nombres) y verificamos que el modelo sepa generalizar su aprendizaje con los datos destinados a las pruebas (último 10 % de nombres)."
]
},
{
"cell_type": "code",
"execution_count": 402,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "N0zzO1IwdW5D",
"outputId": "534a8dc0-99c5-4357-9f7e-bb5b9141e37f"
},
"outputs": [
{
"data": {
"text/plain": [
"(torch.Size([165469, 3]),\n",
" torch.Size([132418, 3]),\n",
" torch.Size([16559, 3]),\n",
" torch.Size([16492, 3]))"
]
},
"execution_count": 402,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import random\n",
"random.shuffle(nombres)\n",
"n1 = int(0.8*len(nombres))\n",
"n2 = int(0.9*len(nombres))\n",
"\n",
"Xtr, Ytr = construir_dataset(nombres[:n1])\n",
"Xdev, Ydev = construir_dataset(nombres[n1:n2])\n",
"Xte, Yte = construir_dataset(nombres[n2:])\n",
"X, Y = construir_dataset(nombres)\n",
"\n",
"X.shape, Xtr.shape, Xdev.shape, Xte.shape"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HVvkgagZHBlO"
},
"source": [
"Ahora definimos nuestros hiperparámetros, aunque esta vez utilizaremos $h = 100$:"
]
},
{
"cell_type": "code",
"execution_count": 409,
"metadata": {
"id": "I0uqOeClM0bH"
},
"outputs": [],
"source": [
"C = torch.randn(27, 2)\n",
"emb = C[Xtr]\n",
"h = 100\n",
"H = torch.randn((6, h))\n",
"d = torch.randn(h)\n",
"\n",
"a = torch.tanh(emb.view(-1, 6) @ H + d)\n",
"\n",
"U = torch.randn(h, 27)\n",
"b = torch.randn(27)\n",
"\n",
"logits = a @ U + b\n",
"\n",
"parametros = [C, H, d, U, b]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-7YQCEE9KIGl"
},
"source": [
"Nuestro número total de parámetros resultantes es:"
]
},
{
"cell_type": "code",
"execution_count": 404,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "MpijkcRINOUM",
"outputId": "dc8b2199-dc89-4b76-d07a-37704e2303c6"
},
"outputs": [
{
"data": {
"text/plain": [
"3481"
]
},
"execution_count": 404,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sum(p.numel() for p in parametros)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "OyC-Cw1ILP6c"
},
"source": [
"Por otro lado, podemos acelerar el entrenamiento de la red neuronal de la siguiente forma: en cada repetición del *loop* de aprendizaje, podemos seleccionar un «lote» (*batch*) o segmento de los datos para únicamente llevar a cabo el proceso de aprendizaje en ese mismo lote, de manera que el modelo no se entrene tomando siempre en consideración todos los datos a la vez, sino datos —en este caso, nombres— aleatorios en cada iteración. Aunque esto implique sacrificar en cierta medida el desempeño de la red neuronal, sin embargo ese sacrificio se compensa con la rapidez que podemos generar a cambio.\n",
"\n",
"Para crear estos lotes, podemos utilizar el siguiente código, el cual generará índices correspondientes a 32 nombres aleatorios de entre aquellos que pertenezcan a nuestros datos:"
]
},
{
"cell_type": "code",
"execution_count": 391,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 37
},
"id": "8o2LeL14VhKw",
"outputId": "74b4e7e5-2389-40ee-f38b-391f272ac2cd"
},
"outputs": [
{
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"'Tres ejemplos: [123401 87687 70020]'"
]
},
"execution_count": 391,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ix = torch.randint(0, X.shape[0], (32,))\n",
"f'Tres ejemplos: {ix[:3].numpy()}'"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lIJn4uOiKLIm"
},
"source": [
"Finalmente, nos falta tratar los detalles técnicos detrás de la *learning rate* («tasa de aprendizaje») del modelo. Recordemos que, cuando modificamos los parámetros en la dirección del gradiente, generalmente atenuamos al gradiente multiplicándolo por un número pequeño para así no excedernos en el ajuste, consiguiendo una pérdida cercana a 0 sin sobrepasarla.\n",
"\n",
"Para determinar una *learning rate* razonable, tenemos que averiguar empíricamente los «límites» de la misma:"
]
},
{
"cell_type": "code",
"execution_count": 410,
"metadata": {
"id": "srB_AEmtMTQB"
},
"outputs": [],
"source": [
"for p in parametros:\n",
" p.requires_grad = True"
]
},
{
"cell_type": "code",
"execution_count": 393,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ur8J9v32ZGRO",
"outputId": "f4be0970-a36a-4116-febb-778f0ccca090"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"22.080612182617188\n",
"18.37497329711914\n",
"15.564812660217285\n",
"15.621807098388672\n",
"17.045852661132812\n",
"18.507583618164062\n",
"21.449209213256836\n",
"21.56830596923828\n",
"24.576093673706055\n",
"15.103194236755371\n"
]
}
],
"source": [
"for _ in range(10):\n",
" #minibatch («minilote»)\n",
" ix = torch.randint(0, Xdev.shape[0], (32,))\n",
"\n",
" # propagación hacia delante\n",
" emb = C[Xdev[ix]]\n",
" a = torch.tanh(emb.view(-1, 6) @ H + d)\n",
" logits = a @ U + b\n",
" perdida = F.cross_entropy(logits, Ydev[ix])\n",
" print(perdida.item())\n",
" \n",
" # propagación hacia atrás\n",
" for p in parametros:\n",
" p.grad = None\n",
" perdida.backward()\n",
"\n",
" # actualización\n",
" lr = -1 # intentaremos obtener el límite superior\n",
" for p in parametros:\n",
" p.data += lr * p.grad"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VxEDRaeSCuy4"
},
"source": [
"Como vemos, la pérdida disminuye con una tasa de aprendizaje de -1; pero al mismo tiempo indica que esta tasa es muy alta porque no es nada estable y se balancea hacia arriba y abajo. Este puede ser nuestro límite superior, busquemos el inferior:"
]
},
{
"cell_type": "code",
"execution_count": 394,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Dmlss8gdDGde",
"outputId": "2adc40ed-146f-4357-c612-961d1384db4a"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"16.730724334716797\n",
"19.113916397094727\n",
"16.265771865844727\n",
"16.343080520629883\n",
"19.638389587402344\n",
"17.050613403320312\n",
"16.68904685974121\n",
"23.818950653076172\n",
"16.777978897094727\n",
"20.498703002929688\n"
]
}
],
"source": [
"for _ in range(10):\n",
" #minibatch («minilote»)\n",
" ix = torch.randint(0, Xdev.shape[0], (32,))\n",
"\n",
" # propagación hacia delante\n",
" emb = C[Xdev[ix]]\n",
" a = torch.tanh(emb.view(-1, 6) @ H + d)\n",
" logits = a @ U + b\n",
" perdida = F.cross_entropy(logits, Ydev[ix])\n",
" print(perdida.item())\n",
" \n",
" # propagación hacia atrás\n",
" for p in parametros:\n",
" p.grad = None\n",
" perdida.backward()\n",
"\n",
" # actualización\n",
" lr = -0.001 # intentaremos obtener el límite inferior\n",
" for p in parametros:\n",
" p.data += lr * p.grad"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "q1-XFmj8D-Rh"
},
"source": [
"De igual forma, esta pérdida es subóptima porque es demasiado lenta para eficientar el aprendizaje, de manera que este puede ser nuestro límite inferior. Ahora, haremos una prueba de entrenamiento con tasas de aprendizaje que se encuentren dentro de estos límites; pero, para que nuestra tasa de aprendizaje no cambie lineal sino exponencialmente, podemos utilizar nuestros números como potencias de 10. Es decir, en teoría, nuestras tasas de aprendizaje serían mil números del 0.001 al 1:"
]
},
{
"cell_type": "code",
"execution_count": 395,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 265
},
"id": "TFjsRBJDFHN9",
"outputId": "93f0f1e9-5cd6-4201-86f5-234832a4f0ed"
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deVxWZf7/8dclLogLirsC4oILgqXhlk1ZaZktZta0TPti00zfmW8zk2JqWbZozUzTfLPF9maamhJccm2zvUytZBMUcQE3UBRUdu7r9wd3/cg0UW843Od+Px8PHt7nnEvuz+GCN4dz3+dzjLUWERHxf42cLkBERHxDgS4i4hIKdBERl1Cgi4i4hAJdRMQlGjv1xO3bt7dRUVFOPb2IiF9at27dXmtth6NtcyzQo6KiWLt2rVNPLyLil4wx2461TadcRERcQoEuIuISCnQREZdQoIuIuIQCXUTEJY4b6MaYl40xecaY1GNsN8aYfxpjsowxycaYwb4vU0REjqc2R+ivAmN/YftFQLT3YxLw7KmXJSIiJ+q4gW6t/RQo+IUh44HXbbWvgTbGmC6+KlBExC1Kyqt4bPkGcvcX18nn98U59G5ATo3lXO+6nzHGTDLGrDXGrM3Pz/fBU4uI+IcvN+/lwn98yvOfZLMqs27yr16vFLXWzgPmAcTHx+vOGiLiekWlFTy2bANvfpNDVLsQ3po0nOE929XJc/ki0HcAETWWw73rREQC2vvpe5i+MIX8g2XceU5P7hndh+AmQXX2fL4I9MXA3caYt4BhQKG1dpcPPq+IiF/ae6iMmYvTWJK8i36dW/HCjfEMDG9T58973EA3xrwJjALaG2NygQeAJgDW2ueAZcA4IAsoBm6pq2JFRBoyay2Lvt/Jg++mcbisij+P6cOd5/SiaeP6ueTnuIFurb32ONst8HufVSQi4od2Hihh+sJUPsrIY1BkGx6fOJDoTq3qtQbH2ueKiLiBx2P5zzfbmb08gyqP5f5LYrjpzCiCGpl6r0WBLiJykrbsPcyUxGS+2VLAWb3b89gVcUSEhThWjwJdROQEVVZ5ePHzLTz5/kaaNm7E4xMHclV8OMbU/1F5TQp0EZETkL6ziCmJyaTsKOSCmE7MujyWTq2DnS4LUKCLiNRKWWUVT3+UxbMfb6ZNSBPmXjeYcXGdHT8qr0mBLiJyHOu27WdKYjJZeYe4YnA3ZlwcQ9sWTZ0u62cU6CIix1BcXskTKzN59cutdA1tzqu3DGFU345Ol3VMCnQRkaP4fNNeEpKSyd1fwo0jujN5bD9aNmvYkdmwqxMRqWeFxRU8siydt9fm0rN9C96+cwRDe4Q5XVatKNBFRLxWpO5mxqJUCg6Xc9eoXvzx/Og6bablawp0EQl4+Qerm2ktTdlFTJfWvHLzEGK7hTpd1glToItIwLLWkvTtDh5akk5JeRX3XtiXSWf3pElQ/TTT8jUFuogEpB0HSrgvKYVPNuZzRve2zJk4kN4dWzpd1ilRoItIQPF4LP9evY05yzOwwIOXDeCG4d1p5EAzLV9ToItIwNicf4iExGTWbN3Pr6Lb8+gEZ5tp+ZoCXURcr6LKwwufZfOPDzbRvEkQf73qNCYO7tagLtv3BQW6iLha6o5CpiQmk7aziItiO/Pg+AF0bNUwmmn5mgJdRFyptKKK//toE899kk3bkKY8+5vBXBTXxemy6pQCXURcZ+3WAiYnJpOdf5irzghn2sX9aRPS8Jpp+ZoCXURc41BZJU+syOD1r7fRNbQ5r986lLP7dHC6rHqjQBcRV/hkYz73JaWws7CEm0ZEce+FfWnRwJtp+Vpg7a2IuM6B4nJmLdlA4re59OrQgnfuHEF8lH800/I1BbqI+K3lKbuYsSiN/cXl3H1ub+4+r7dfNdPyNQW6iPidvKJS7l+Uxoq03cR2a81rtw5hQFf/a6blawp0EfEb1lrmr8tl1pJ0Sis9TBnbjzt+1YPGftpMy9cU6CLiF3IKirlvQQqfbdrL0KgwZk+Mo2cH/26m5WsKdBFp0Ko8lte/2soTKzMxwKzxA/jNMHc00/I1BbqINFhZeQeZkpjCum37OadPBx69Io5ubZo7XVaDpUAXkQanosrD859s5p8fZhHSLIi///o0JgxyXzMtX1Ogi0iDkpJbyOTEZDbsKuLigV2YeekAOrRq5nRZfkGBLiINQmlFFf/4YBMvfJZNuxZNef6GM7hwQGeny/IrtQp0Y8xY4CkgCHjRWjv7iO2RwGtAG++YBGvtMh/XKiIutTp7HwlJKWzZe5ir4yO47+L+hDZv4nRZfue4gW6MCQLmAmOAXGCNMWaxtTa9xrDpwNvW2meNMTHAMiCqDuoVERc5WFrB4ysy+dfX24gIa84btw9jZO/2Tpflt2pzhD4UyLLWZgMYY94CxgM1A90Crb2PQ4GdvixSRNxnVWYe05JS2FVUyq0je/CXC/sQ0lRngU9Fbb563YCcGsu5wLAjxswE3jPG/A/QAhh9tE9kjJkETAKIjIw80VpFxAX2Hy5n1pJ0kr7bQXTHliTedSaDI9s6XZYr+OrX4bXAq9bavxljRgD/MsbEWms9NQdZa+cB8wDi4+Otj55bRPyAtZalKbt4YFEahSUV/OH8aH5/bi+aNQ7cZlq+VptA3wFE1FgO966r6TZgLIC19itjTDDQHsjzRZEi4t/2FJUyfWEq76fvYWB4KP++fRj9u7Q+/n+UE1KbQF8DRBtjelAd5NcA1x0xZjtwPvCqMaY/EAzk+7JQEfE/1lreXpvDw0s3UF7p4b5x/bh1pJpp1ZXjBrq1ttIYczewkuq3JL5srU0zxjwErLXWLgb+DLxgjLmH6hdIb7bW6pSKSADbvq+YhKRkvty8j2E9wpgzcSBR7Vs4XZar1eocuvc95cuOWHd/jcfpwEjfliYi/qjKY3n1y638dWUmQY0Mj0yI5dohkWqmVQ/0HiER8ZmNew4yeX4y3+cc4Lx+HXlkQixdQtVMq74o0EXklJVXenj24808vWoTLZs15qlrTuey07qqmVY9U6CLyClZn3OAKYnJZOw+yGWndeWBS2No11LNtJygQBeRk1JSXsWTH2zkxc+y6dgqmBdvjGd0TCenywpoCnQROWFfbd7H1KRktu4r5tqhkUwd14/WwWqm5TQFuojUWlFpBbOXZ/Cf1dvp3i6E/9wxjDN7qZlWQ6FAF5Fa+XDDHqYtSCXvYCl3/KoHfxrTl+ZNddl+Q6JAF5FftO9QGQ++m87i9Tvp26kVz91wBqdHtHG6LDkKBbqIHJW1lsXrd/Lgu+kcLK3gntF9uGtUL5o21mX7DZUCXUR+ZldhCdMXpPJhRh6nRbTh8YkD6du5ldNlyXEo0EXkRx6P5a01OTy2bAMVHg/TL+7PLSN7EKTL9v2CAl1EANi69zAJScl8nV3AiJ7tmD0xju7t1EzLnyjQRQJcZZWHV77Yyt/ez6RJo0bMviKOq4dE6LJ9P6RAFwlgGbuLmDI/mfW5hYzu34mHL4+lc2iw02XJSVKgiwSgssoq5q7azDOrsght3oT/u3YQlwzsoqNyP6dAFwkw323fz5TEZDbuOcSEQd2YcUkMYS2aOl2W+IACXSRAFJdX8rf3NvLyF1vo3DqYl2+O57x+aqblJgp0kQDwZdZeEpJS2F5QzPXDI5kyth+t1EzLdRToIi5WWFLBY8s28NaaHHq0b8Fbk4YzvGc7p8uSOqJAF3Gp99J2M31hKnsPlXHnOT25Z3QfgpuomZabKdBFXGbvoTJmLk5jSfIu+nVuxYs3xTMwXM20AoECXcQlrLUs/H4HD76bTnFZFX8e04ffjupFkyA10woUCnQRF9h5oIRpC1JYlZnPoMjqZlrRndRMK9Ao0EX8mMdjeeOb7cxZnkGVx3L/JTHcdGaUmmkFKAW6iJ/Kzj9EQmIK32wt4Kze7XnsijgiwkKcLkscpEAX8TOVVR5e/HwLT76/kWaNG/H4lQO56oxwXbYvCnQRf5K+s4jJietJ3VHEhQM6MWt8LB1bq5mWVFOgi/iBssoqnv4oi2c/3kybkCY885vBXBTbWUfl8hMKdJEGbt22AqYkppCVd4grBndjxsUxtFUzLTkKBbpIA3W4rJInVmby2ldb6RranFdvGcKovh2dLksasFoFujFmLPAUEAS8aK2dfZQxvwZmAhZYb629zod1igSUzzblMzUphdz9Jdw0ojv3ju1Hy2Y6/pJfdtzvEGNMEDAXGAPkAmuMMYuttek1xkQDU4GR1tr9xhgdRoichMLiCh5ems4763Lp2aEF7/x2BEOiwpwuS/xEbX7lDwWyrLXZAMaYt4DxQHqNMXcAc621+wGstXm+LlTE7Vak7mbGolQKDpfzu1G9+MP50WqmJSekNoHeDcipsZwLDDtiTB8AY8wXVJ+WmWmtXXHkJzLGTAImAURGRp5MvSKuk3ewlJmL01iWspuYLq155eYhxHYLdbos8UO+OinXGIgGRgHhwKfGmDhr7YGag6y184B5APHx8dZHzy3il6y1JH67g1lL0impqOLeC/sy6eyeaqYlJ602gb4DiKixHO5dV1MusNpaWwFsMcZspDrg1/ikShGXyd1fzH0LUvl0Yz7x3dsye+JAends6XRZ4udqE+hrgGhjTA+qg/wa4Mh3sCwErgVeMca0p/oUTLYvCxVxA4/H8q+vtzFnRQYAD142gBuGd6eRmmmJDxw30K21lcaYu4GVVJ8ff9lam2aMeQhYa61d7N12gTEmHagC7rXW7qvLwkX8zeb8Q0yZn8zabfs5u08HHp0QS3hbNdMS3zHWOnMqOz4+3q5du9aR5xapTxVVHuZ9ms1TH26ieZMgZlwSw8TB3XTZvpwUY8w6a2380bbpSgWROpS6o5DJ85NJ31XEuLjOzLxsAB1bqZmW1A0FukgdKK2o4qkPNzHv02zahjTluesHMza2i9Nlicsp0EV8bM3WAqbMTyZ772GuOiOc6RfHEBrSxOmyJAAo0EV85FBZJY+vyOD1r7YR3rY5/7ptKL+K7uB0WRJAFOgiPvDJxnzuS0phZ2EJN58Zxb0X9qWFmmlJPdN3nMgpOFBczkNL0kn6dge9OrRg/m9HcEZ3NdMSZyjQRU6CtZblqbu5f1EqB4oruPvc3tx9Xm810xJHKdBFTlBeUSkzFqWyMm0Psd1a89qtQxnQVc20xHkKdJFastbyzrpcHl6STlmlh4SL+nH7WT1orGZa0kAo0EVqIaegmKlJKXyetZehUWHMnhhHzw5qpiUNiwJd5BdUeSyvf7WVx1dk0sjArMtj+c3QSDXTkgZJgS5yDFl5B5k8P5lvtx9gVN8OPDIhjm5tmjtdlsgxKdBFjlBR5eG5jzfzfx9lEdIsiCevPo3LT1czLWn4FOgiNaTkFnLv/PVk7D7IJQO7MPOyAbRv2czpskRqRYEuQnUzrSc/2MgLn2bTvmUz5t1wBhcM6Ox0WSInRIEuAW919j4SklLYsvcw1wyJYOq4/oQ2VzMt8T8KdAlYB0srmLMig39/vZ2IsOa8cfswRvZu73RZIidNgS4BaVVGHvctSGF3USm3ndWDP1/Qh5Cm+nEQ/6bvYAkoBYfLeejdNBZ+v5Poji1JvOtMBke2dbosEZ9QoEtAsNayJHkXMxenUVhSwR/Oj+b35/aiWWM10xL3UKCL6+0pKmXaglQ+2LCHgeGhvHHHMPp1bu10WSI+p0AX17LW8t81OTyybAPllR6mjevPLSOj1ExLXEuBLq60fV8xCUnJfLl5H8N6hDFn4kCi2rdwuiyROqVAF1ep8lhe+WILf30vk8aNGvHohDiuGRKhZloSEBTo4hqZuw8yOTGZ9TkHOK9fRx6ZEEuXUDXTksChQBe/V17p4ZmPs5i7KotWwU146prTuey0rmqmJQFHgS5+bX3OASbPTyZzz0HGn96V+y+JoZ2aaUmAUqCLXyopr+Lv72fy0udb6NgqmBdvjGd0TCenyxJxlAJd/M6Xm/cyNSmFbfuKuW5YJAkX9aN1sJppiSjQxW8UlVbw2LIM3vxmO93bhfCfO4ZxZi810xL5gQJd/MIH6XuYtjCF/INlTDq7J/eM7kPzprpsX6SmWl0yZ4wZa4zJNMZkGWMSfmHcRGOMNcbE+65ECWT7DpXxhze/4/bX19I2pCkLfjeS+8b1V5iLHMVxj9CNMUHAXGAMkAusMcYsttamHzGuFfBHYHVdFCqBxVrL4vU7mbk4jUNlldwzug93jepF08a6bF/kWGpzymUokGWtzQYwxrwFjAfSjxg3C5gD3OvTCiXg7CosYfqCVD7MyOP0iDY8fuVA+nRq5XRZIg1ebQK9G5BTYzkXGFZzgDFmMBBhrV1qjDlmoBtjJgGTACIjI0+8WnE1j8fy5prtPLYsg0qPh+kX9+eWkT0I0mX7IrVyyi+KGmMaAX8Hbj7eWGvtPGAeQHx8vD3V5xb32LL3MAmJyazeUsCZvdox+4qBRLYLcbosEb9Sm0DfAUTUWA73rvtBKyAW+Nh7qXVnYLEx5jJr7VpfFSruVFnl4eUvtvC39zbStHEj5kyM49fxEbpsX+Qk1CbQ1wDRxpgeVAf5NcB1P2y01hYCP74Z2BjzMfAXhbkcz4ZdRUxJTCY5t5AxMZ14+PJYOrUOdrosEb913EC31lYaY+4GVgJBwMvW2jRjzEPAWmvt4rouUtylrLKKuas288yqLEKbN+Hp6wZxcVwXHZWLnKJanUO31i4Dlh2x7v5jjB116mWJW327fT9T5iezKe8QEwZ14/5LYmjboqnTZYm4gq4UlXpRXF7JX1du5JUvt9C5dTCv3DyEc/t1dLosEVdRoEud+yJrLwlJyeQUlHD98EimjO1HKzXTEvE5BbrUmcKSCh5duoH/rs2hR/sW/HfScIb1bOd0WSKupUCXOvFe2m6mL0xl3+FyfntOL/53dDTBTdR/RaQuKdDFp/IPljHz3TSWJu+if5fWvHTTEOLCQ50uSyQgKNDFJ6y1LPhuBw8tSae4rIq/XNCHO8/pRZMgNdMSqS8KdDllOw6UMG1BCh9n5jM4srqZVu+OaqYlUt8U6HLSPB7LG6u3MXt5Bh4LD1waw40jotRMS8QhCnQ5Kdn5h0hITOGbrQX8Kro9j06IIyJMzbREnKRAlxNSWeXhhc+28OQHGwlu3IgnrhzIlWeE67J9kQZAgS61lr6ziMmJ60ndUcSFAzoxa3wsHdVMS6TBUKDLcZVWVPH0R1k898lm2oQ05dnfDOaiuC5OlyUiR1Cgyy9at62AyfOT2Zx/mImDw5lxSX/ahKiZlkhDpECXozpcVskTKzN57autdA1tzmu3DuWcPh2cLktEfoECXX7m0435TE1KYWdhCTcO7869Y/vRspm+VUQaOv2Uyo8KiyuYtTSd+ety6dmhBW/fOYIhUWFOlyUitaRAFwBWpO5ixqI0Cg6X87tRvfjD+WqmJeJvFOgBLu9gKQ8sSmN56m5iurTmlZuHENtNzbRE/JECPUBZa5m/LpeHl26gpKKKey/sy6Sze6qZlogfU6AHoJyCYu5bkMJnm/YS370tsycOpHfHlk6XJSKnSIEeQDwey+tfbeXxlZkY4KHxA7h+WHcaqZmWiCso0ANEVt4hEhKTWbttP2f36cCjE2IJb6tmWiJuokB3uYoqD/M+zeapDzbRvGkQf7vqNK4Y3E3NtERcSIHuYqk7Cpk8P5n0XUWMi+vMg5fF0qFVM6fLEpE6okB3odKKKp76cBPzPs0mrEVTnrt+MGNj1UxLxO0U6C6zZmsBU+Ynk733ML+OD2fauBhCQ5o4XZaI1AMFukscKqvk8RUZvP7VNsLbNufftw3jrOj2TpclIvVIge4CqzLzmJaUwq6iUm4ZGcVfLuhLCzXTEgk4+qn3Y/sPlzNrSTpJ3+2gd8eWzP/tmZzRva3TZYmIQxTofshay7KU3TywOJUDxRX8z3m9ufu83jRrrGZaIoGsVoFujBkLPAUEAS9aa2cfsf1PwO1AJZAP3Gqt3ebjWgXIKypl+sJU3kvfQ1y3UF6/dRgxXVs7XZaINADHDXRjTBAwFxgD5AJrjDGLrbXpNYZ9B8Rba4uNMXcBjwNX10XBgcpayztrc5m1NJ3ySg9TL+rHbWf1oLGaaYmIV22O0IcCWdbabABjzFvAeODHQLfWrqox/mvgel8WGehyCoqZmpTC51l7GdojjNlXxNGzg5ppichP1SbQuwE5NZZzgWG/MP42YPmpFCXVqjyW177cyhMrMwlqZHj48liuGxqpZloiclQ+fVHUGHM9EA+cc4ztk4BJAJGRkb58atfZtOcgkxOT+W77AUb17cCjE+Lo2qa502WJSANWm0DfAUTUWA73rvsJY8xoYBpwjrW27GifyFo7D5gHEB8fb0+42gBQXunhuU828/RHWbRoFsQ/rj6d8ad3VTMtETmu2gT6GiDaGNOD6iC/Briu5gBjzCDgeWCstTbP51UGiOTcA0yen0zG7oNcelpXHrg0hvYt1UxLRGrnuIFura00xtwNrKT6bYsvW2vTjDEPAWuttYuBJ4CWwDveI8nt1trL6rBuVymtqOLJ9zfywmfZdGjVjBdujGdMTCenyxIRP1Orc+jW2mXAsiPW3V/j8Wgf1xUwvs7eR0JiMlv3FXPt0AgSLupPaHM10xKRE6crRR1ysLSC2cszeGP1diLDQvjP7cM4s7eaaYnIyVOgO+CjjD1MW5DKnqJSbj+rB3+6oA8hTTUVInJqlCL1qOBwOQ+9m8bC73cS3bElz9x1JoMi1UxLRHxDgV4PrLW8m7yLmYvTOFhawR/Pj+Z35/ZSMy0R8SkFeh3bXVjdTOuDDXs4LTyUOVcOo19nNdMSEd9ToNcRay1vrcnh0aUbqPB4mDauP7ee1YMgXbYvInVEgV4Htu07TEJiCl9l72N4zzBmXzGQqPYtnC5LRFxOge5DVR7LK19s4a/vZdKkUSMenRDHNUMi1ExLROqFAt1HMndXN9Nan3OA8/t15OEJsXQJVTMtEak/CvRTVF7p4ZmPs5i7KotWwU3457WDuHRgFzXTEpF6p0A/Bd/nHGDK/GQy9xxk/OldeeDSAYS1aOp0WSISoBToJ6GkvIq/vZfJy19soWOrYF66KZ7z+6uZlog4S4F+gr7cvJeExBS2FxRz3bBIEi7qR+tgNdMSEecp0GupqLSCx5Zt4M1vcujeLoQ37xjOiF7tnC5LRORHCvRa+CB9D9MWppB/sIxJZ/fkntF9aN5Ul+2LSMOiQP8F+w6VMfPddN5dv5N+nVsx74Z4Toto43RZIiJHpUA/Cmsti77fyYPvpnGorJI/jenDb8/pRdPGjZwuTUTkmBToR9h5oITpC1P5KCOP0yPa8PiVA+nTqZXTZYmIHJcC3cvjsfznm+3MXp5Blccy45IYbj4zSs20RMRvKNCBLXsPk5CYzOotBYzs3Y7HJgwksl2I02WJiJyQgA70yioPL32+hb+/v5GmjRsxZ2Icv46P0GX7IuKXAjbQN+wqYkpiMsm5hYyJ6cTDl8fSqXWw02WJiJy0gAv0ssoq5n6UxTMfb6ZNSBPmXjeYcXGddVQuIn4voAJ93bb9TElMJivvEFcM6saMS2Joq2ZaIuISARHoxeWVPLEyk1e/3EqX1sG8cssQzu3b0emyRER8yvWB/vmmvSQkJZO7v4Qbhndn8ti+tFIzLRFxIdcGemFJBY8sTefttbn0aN+C/04azrCeaqYlIu7lykBfmbabGQtT2Xe4nLtG9eKP50cT3ETNtETE3VwV6PkHy5i5OI2lKbvo36U1L900hLjwUKfLEhGpF64IdGstSd/u4KEl6ZSUV3HvhX2ZdHZPmgSpmZaIBA6/D/QdB0q4LymFTzbmMziyuplW745qpiUigadWgW6MGQs8BQQBL1prZx+xvRnwOnAGsA+42lq71bel/pTHY/n36m3MWZ6BBWZeGsMNI9RMS0QC13ED3RgTBMwFxgC5wBpjzGJrbXqNYbcB+621vY0x1wBzgKvromCAzfmHSEhMZs3W/fwquj2PTogjIkzNtEQksNXmCH0okGWtzQYwxrwFjAdqBvp4YKb38XzgaWOMsdZaH9YKwNtrcpi+KJXgxo144sqBXHlGuC7bFxGhdoHeDcipsZwLDDvWGGttpTGmEGgH7K05yBgzCZgEEBkZeVIF9+jQgvP7deTB8QPo2ErNtEREflCvL4paa+cB8wDi4+NP6uh9SFQYQ6LCfFqXiIgb1OZ9fTuAiBrL4d51Rx1jjGkMhFL94qiIiNST2gT6GiDaGNPDGNMUuAZYfMSYxcBN3sdXAh/VxflzERE5tuOecvGeE78bWEn12xZfttamGWMeAtZaaxcDLwH/MsZkAQVUh76IiNSjWp1Dt9YuA5Ydse7+Go9Lgat8W5qIiJwIXRsvIuISCnQREZdQoIuIuIQCXUTEJYxT7y40xuQD207yv7fniKtQA4D2OTBonwPDqexzd2tth6NtcCzQT4UxZq21Nt7pOuqT9jkwaJ8DQ13ts065iIi4hAJdRMQl/DXQ5zldgAO0z4FB+xwY6mSf/fIcuoiI/Jy/HqGLiMgRFOgiIi7hd4FujBlrjMk0xmQZYxKcrsdXjDERxphVxph0Y0yaMeaP3vVhxpj3jTGbvP+29a43xph/er8OycaYwc7uwckxxgQZY74zxizxLvcwxqz27td/vS2bMcY08y5nebdHOVn3yTLGtDHGzDfGZBhjNhhjRgTAHN/j/Z5ONca8aYwJduM8G2NeNsbkGWNSa6w74bk1xtzkHb/JGHPT0Z7rWPwq0GvcsPoiIAa41hgT42xVPlMJ/NlaGwMMB37v3bcE4ENrbTTwoXcZqr8G0d6PScCz9V+yT/wR2FBjeQ7wpLW2N7Cf6huQQ40bkQNPesf5o6eAFdbafsBpVO+7a+fYGNMN+AMQb62NpboF9w83knfbPL8KjD1i3QnNrTEmDHiA6tt8DgUe+OGXQK1Ya/3mAxgBrKyxPBWY6nRddbSvi4AxQCbQxbuuC5Dpffw8cG2N8T+O85cPqu9+9SFwHrAEMFRfPdf4yPmmuh//CO/jxt5xxul9OMH9DQW2HFm3y+f4h/sNh3nnbQlwoVvnGYgCUk92boFrgedrrP/JuON9+PvV5eUAAAJESURBVNUROke/YXU3h2qpM94/MwcBq4FO1tpd3k27gU7ex274WvwDmAx4vMvtgAPW2krvcs19+smNyIEfbkTuT3oA+cAr3tNMLxpjWuDiObbW7gD+CmwHdlE9b+tw9zzXdKJze0pz7m+B7nrGmJZAIvC/1tqimtts9a9sV7zP1BhzCZBnrV3ndC31qDEwGHjWWjsIOMz//xMccNccA3hPF4yn+pdZV6AFPz8tERDqY279LdBrc8Nqv2WMaUJ1mL9hrU3yrt5jjOni3d4FyPOu9/evxUjgMmPMVuAtqk+7PAW08d5oHH66T264EXkukGutXe1dnk91wLt1jgFGA1ustfnW2gogieq5d/M813Sic3tKc+5vgV6bG1b7JWOMofrerBustX+vsanmDbhvovrc+g/rb/S+Wj4cKKzxp12DZ62daq0Nt9ZGUT2PH1lrfwOsovpG4/Dz/fXrG5Fba3cDOcaYvt5V5wPpuHSOvbYDw40xId7v8R/22bXzfIQTnduVwAXGmLbev24u8K6rHadfRDiJFx3GARuBzcA0p+vx4X6dRfWfY8nA996PcVSfP/wQ2AR8AIR5xxuq3/GzGUih+l0Eju/HSe77KGCJ93FP4BsgC3gHaOZdH+xdzvJu7+l03Se5r6cDa73zvBBo6/Y5Bh4EMoBU4F9AMzfOM/Am1a8TVFD919htJzO3wK3e/c8CbjmRGnTpv4iIS/jbKRcRETkGBbqIiEso0EVEXEKBLiLiEgp0ERGXUKCLiLiEAl1ExCX+HwbDNcpVQbolAAAAAElFTkSuQmCC",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"lr = torch.linspace(0.001, 1, 1000)\n",
"\n",
"plt.plot(lr.numpy());"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e4RzxflMGRlv"
},
"source": [
"Pero $10^{-3} = 0.001$ y $10{^0} = 1$, de manera que utilizar este truco nos ayuda a obtener un cambio exponencial en nuestras tasas de aprendizaje. Ahora, nuestros 1000 números serán exponentes de 10 y lucirán así:"
]
},
{
"cell_type": "code",
"execution_count": 396,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 265
},
"id": "BBz2E7b9R019",
"outputId": "78a917e1-8925-4b1f-da75-837c6c5bdb6e"
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAedUlEQVR4nO3deXzcdb3v8ddnsrZpmqRNuqZtWrrQUvbQlkUWi1DQC3pcoKi4Fz2inqsehYdXPerV63LVo+dwUPAgRzyU1cOtUKyHCqggpSml+5auSbokbZJuadb53D9mUoY0bRZm8puZvJ+Pxzzmt3xnfp9ffum7v3x/m7k7IiKS+kJBFyAiIvGhQBcRSRMKdBGRNKFAFxFJEwp0EZE0kRnUgouLi72srCyoxYuIpKRVq1YddPeS7uYFFuhlZWVUVFQEtXgRkZRkZrtPN09dLiIiaUKBLiKSJhToIiJpQoEuIpImFOgiImmix0A3swfMrNbM1p9mvpnZz82s0szWmtlF8S9TRER60ps99AeBBWeYfwMwLfpaBNz71ssSEZG+6jHQ3f3PQP0ZmtwM/MYjXgEKzWxsvAoUEUkX4bDz3Wc2sra6MSHfH48+9PFAVcx4dXTaKcxskZlVmFlFXV1dHBYtIpI6ttYe5f6/7GTbgWMJ+f4BPSjq7ve5e7m7l5eUdHvlqohI2qrY1QBAeVlRQr4/HoFeA0yIGS+NThMRkRirdjdQPCyHiSOGJuT74xHoS4Dbo2e7zAMOu/u+OHyviEhaWbW7gfJJRZhZQr6/x5tzmdli4Gqg2MyqgW8CWQDu/gtgKXAjUAk0AR9LSKUiIims9mgze+qb+PC8SQlbRo+B7u4Le5jvwGfjVpGISBpaFe0/vzhB/eegK0VFRAZExe4GcjJDzB5XkLBlKNBFRAZAxe4Gzi8tJDszcbGrQBcRSbATrR1sqDmc0O4WUKCLiCTc2upG2sPOxRMV6CIiKa1id/SA6CQFuohISlu1u4GzSvIoystO6HIU6CIiCdQRdip21VM+aUTCl6VAFxFJoM37j3CkuZ25UxToIiIp7dWdkbuPz50yMuHLUqCLiCTQih31lBYNYXzhkIQvS4EuIpIg7s6ru+qZMznx3S2gQBcRSZhttceoP97KvMmJ724BBbqISMKsONl/rj10EZGUtmLHIcYMz03YAy26UqCLiCSAu7NiZz1zp4xI2AMtulKgi4gkwM6Dx6k72jJgB0RBgS4ikhAnzz8foAOioEAXEUmIFTvrKR6Ww1kleQO2TAW6iEicuTuv7DjE3MkD138OCnQRkbjbefA4+w43c+lZA9fdAgp0EZG4e2n7IQAun1o8oMtVoIuIxNnLlQcZV5BL2ciBOf+8kwJdRCSOOsLO33Yc4vKpxQPafw4KdBGRuNq49wiNTW0D3t0CCnQRkbh6aftBAC4b4AOioEAXEYmrlyoPMm3UMEYNzx3wZSvQRUTipKW9g5W76gPpbgEFuohI3Ly2u5HmtrACXUQk1b28/SAhG7j7n3elQBcRiZOXKg9yXmkhw3OzAlm+Al1EJA4On2hjTfVhrgiouwV6GehmtsDMtphZpZnd1c38iWb2vJmtNrO1ZnZj/EsVEUleL1UepCPsXDWjJLAaegx0M8sA7gFuAGYBC81sVpdm/wt4zN0vBG4F/i3ehYqIJLMXt9SRn5vJhRMKA6uhN3voc4BKd9/h7q3AI8DNXdo4MDw6XADsjV+JIiLJzd15cWsdV0wtJjMjuJ7s3ix5PFAVM14dnRbrn4APmVk1sBT4XHdfZGaLzKzCzCrq6ur6Ua6ISPLZeuAY+480c3WA3S0Qv4OiC4EH3b0UuBF4yMxO+W53v8/dy929vKQk2BUXEYmXF7fWAnDl9OQP9BpgQsx4aXRarE8AjwG4+9+AXCC4Q70iIgPoxa11zBidz9iCIYHW0ZtAXwlMM7PJZpZN5KDnki5t9gDzAcxsJpFAV5+KiKS94y3trNzZEOjZLZ16DHR3bwfuBJYBm4iczbLBzL5tZjdFm30J+JSZrQEWAx91d09U0SIiyeKVHYdo7QhzVcDdLQCZvWnk7kuJHOyMnfaNmOGNwOXxLU1EJPm9uLWOIVkZlJcVBV2KrhQVEekvd+eFLXVcdtZIcjIzgi5HgS4i0l/b646xp74p8NMVOynQRUT66blNkdMV588cHXAlEQp0EZF+Wr7pALPGDmdcYbCnK3ZSoIuI9EP98VZW7W7g2pmjgi7lJAW6iEg/PL+5lrDDtbOSo7sFFOgiIv2yfPMBRuXnMHtcQdClnKRAFxHpo9b2MH/eepD5M0cRClnQ5ZykQBcR6aMVOw9xrKWd+WcnT3cLKNBFRPrsuY0HyM0KcXmAj5vrjgJdRKQP3J3nNtVyxdRihmQHf3VoLAW6iEgfbNh7hJrGE1ybJBcTxVKgi4j0wR/W7ydk8I4kOl2xkwJdRKQP/rBhP3Mnj2TksJygSzmFAl1EpJcqa49SWXuMBbPHBF1KtxToIiK99Oy6/QBcf44CXUQkpT27fj8XTSxkTEFu0KV0S4EuItILew41sXHfEW6YPTboUk5LgS4i0gt/2LAPIGn7z0GBLiLSK8+u388544YzYcTQoEs5LQW6iEgP9jaeYPWeRm5I4r1zUKCLiPTo6bV7AXjXeeMCruTMFOgiIj1YsmYv55UWUFacF3QpZ6RAFxE5gx11x1hfc4Sbzk/uvXNQoIuInNHv1+zDDN55XvKerthJgS4ichruzpI1NVxSNoKxBUOCLqdHCnQRkdPYtO8o2+uOp0R3CyjQRUROa8mavWSGjBvPTf7uFlCgi4h0y935/Zq9XDGtmBF52UGX0ysKdBGRblTsbqCm8UTKdLdALwPdzBaY2RYzqzSzu07T5gNmttHMNpjZw/EtU0RkYD1RUc3Q7IykvVVudzJ7amBmGcA9wDuAamClmS1x940xbaYBdwOXu3uDmY1KVMEiIol2orWDZ9bt48Zzx5KX02NMJo3e7KHPASrdfYe7twKPADd3afMp4B53bwBw99r4likiMnCWbdjPsZZ23ntRadCl9ElvAn08UBUzXh2dFms6MN3MXjKzV8xsQXdfZGaLzKzCzCrq6ur6V7GISII9+Vo14wuHMHfyiKBL6ZN4HRTNBKYBVwMLgfvNrLBrI3e/z93L3b28pKQkTosWEYmfvY0n+GvlQd57cSmhkAVdTp/0JtBrgAkx46XRabGqgSXu3ubuO4GtRAJeRCSl/NfqGtzhvRd17YhIfr0J9JXANDObbGbZwK3Aki5tniKyd46ZFRPpgtkRxzpFRBLO3XnytWouKSti0sjkvrNid3oMdHdvB+4ElgGbgMfcfYOZfdvMboo2WwYcMrONwPPAP7r7oUQVLSKSCKurGtlRd5z3XZxaB0M79ep8HHdfCiztMu0bMcMOfDH6EhFJSY+trCI3K5Qyl/p3pStFRUSAo81tLFmzl/9x3jjyc7OCLqdfFOgiIsBTr++lqbWD2+ZODLqUflOgi8ig5+48vGIPM8cO54IJp5xxnTIU6CIy6L1e1cimfUe4be5EzFLr3PNYCnQRGfQeXrGHodkZvPuC1LmzYncU6CIyqB0+0cbv1+7lpvNT92BoJwW6iAxqT62uobktnNIHQzsp0EVk0HJ3/nPFbmaPH855pal7MLSTAl1EBq2Xtx9i64Fj3D6vLOhS4kKBLiKD1q9f2sWIvGxuSvGDoZ0U6CIyKO0+dJzlmw/wwbkTyc3KCLqcuFCgi8ig9B8v7ybDjA/NmxR0KXGjQBeRQedYSzuPV1TxzvPGMnp4btDlxI0CXUQGnScqqjja0s7HLp8cdClxpUAXkUElHHb+42+7uXBiYUrft6U7CnQRGVSe23SAnQePp93eOSjQRWQQcXfufXE7pUVDuHH2mKDLiTsFuogMGq/urGf1nkbuuHIKmRnpF3/pt0YiIqdx74vbGZmXzfvLJwRdSkIo0EVkUNi49wgvbKnj41dMTpsLibpSoIvIoPCLF7czLCczrS4k6kqBLiJpb8+hJp5eu5cPzp1IwZDUvuf5mSjQRSTt3fvidjJDIT5+RfqdqhhLgS4iaa2qvonHK6q4dc6EtLrMvzsKdBFJa//6p0pCIePvr54adCkJp0AXkbS151ATT7xWzW1zJjKmIL33zkGBLiJp7F/+tI3MkPGZq88KupQBoUAXkbS06+Bxfre6hg/OnZT2feedFOgikpZ+/qdtZGUYn756StClDBgFuoiknc37j/DU6ho+PG8So/IHx945KNBFJA394NnNDMvJ5LPXpP+ZLbF6FehmtsDMtphZpZnddYZ27zUzN7Py+JUoItJ7L28/yPNb6vj7a6ZSODQ76HIGVI+BbmYZwD3ADcAsYKGZzeqmXT7wBWBFvIsUEemNcNj5/rObGVeQy0cvKwu6nAHXmz30OUClu+9w91bgEeDmbtp9B/gB0BzH+kREeu2ZdftYW32YL143I23vqHgmvQn08UBVzHh1dNpJZnYRMMHdnznTF5nZIjOrMLOKurq6PhcrInI6re1hfrRsC2ePyec9F47v+QNp6C0fFDWzEPAT4Es9tXX3+9y93N3LS0pK3uqiRUROevDlneypb+LuG2eSEbKgywlEbwK9Boh9vEdpdFqnfGA28IKZ7QLmAUt0YFREBkrtkWZ+9tw25p89iqumD96dxd4E+kpgmplNNrNs4FZgSedMdz/s7sXuXubuZcArwE3uXpGQikVEuvj+s5tp63C+/q5TztcYVHoMdHdvB+4ElgGbgMfcfYOZfdvMbkp0gSIiZ7Jqdz2/W13DJ982mbLivKDLCVRmbxq5+1JgaZdp3zhN26vfelkiIj3rCDvfXLKBMcNzB91FRN3RlaIikrIeXVnF+poj3H3j2eTl9Gr/NK0p0EUkJdUebeb7z25i7uQR3HT+uKDLSQoKdBFJSd/6/Uaa28P8n787F7PBeZpiVwp0EUk5yzcd4Jm1+/jcNVOZUjIs6HKShgJdRFLKsZZ2vv7UeqaPHsYdVw2OJxH1lo4iiEhK+fEft7DvSDNP3HYZ2ZnaJ42ln4aIpIwVOw7x4Mu7+PC8SVw8qSjocpKOAl1EUsKxlna+/MQaJhQN5asLzg66nKSkLhcRSQnffWYT1Q0nePyOS3XO+WloD11Ekt7zW2pZ/OoeFl05hfKyEUGXk7QU6CKS1BqbWvnqE2uZMTqfL75jetDlJDX93SIiScvd+fLja2loauXXH7uEnMzB9xSivtAeuogkrQdf3sVzmw5w9w0zOWdcQdDlJD0FuogkpXXVh/ne0k1cO3M0H7u8LOhyUoICXUSSztHmNu5c/BrFw3L40fvO071aekl96CKSVNydu55cR3XDCR5ZNI+ivOygS0oZ2kMXkaTyixd38My6ffzj9TO4RKco9okCXUSSxotb6/jhss2867yx3HHllKDLSTkKdBFJCrsPHefzi1czY3Q+P1S/eb8o0EUkcMdb2rnjoVUA3PfhcoZm6/BefyjQRSRQ7R1hPr94NVsPHOVfFl7IxJFDgy4pZem/QREJjLvzrd9vZPnmWr7z7tlcOb0k6JJSmvbQRSQwv/rLTh56ZTd3XDmFD8+bFHQ5KU+BLiKBWLpuH99duol3njtW9zePEwW6iAy4lysP8g+Pvs7Fk4r48QfOJxTSGS3xoEAXkQH12p4GPvmbCiaPzONXt5eTm6U7KMaLAl1EBszGvUf46AOvMio/h4c+MUeX9ceZAl1EBsT2umPc/sAK8nIy+e0n5zJqeG7QJaUdnbYoIglXWXuUhfevAOC3n5xLaZHONU8E7aGLSEJt3n+EW375CgCLPzWPs0qGBVxR+upVoJvZAjPbYmaVZnZXN/O/aGYbzWytmS03M51QKiKsrznMwvteISsjxKOL5jFtdH7QJaW1HgPdzDKAe4AbgFnAQjOb1aXZaqDc3c8DngB+GO9CRSS1rNpdz233v8LQ7EwevWMeU7RnnnC92UOfA1S6+w53bwUeAW6ObeDuz7t7U3T0FaA0vmWKSCr5740HuO3+FYzIy+aRRfOYNDIv6JIGhd4E+nigKma8OjrtdD4BPPtWihKR1PXwij3c8VAFZ48dzpOfuYwJI3QAdKDE9SwXM/sQUA5cdZr5i4BFABMnToznokUkYO7OT5/bxs+Xb+OaGSXc88GLdBvcAdabn3YNMCFmvDQ67U3M7Frga8BV7t7S3Re5+33AfQDl5eXe52pFJCmdaO3gq0+uZcmavXygvJTvvedcMjN0Et1A602grwSmmdlkIkF+K3BbbAMzuxD4JbDA3WvjXqWIJK29jSdY9FAFG/Ye4SsLZvCZq87S04YC0mOgu3u7md0JLAMygAfcfYOZfRuocPclwI+AYcDj0Q25x91vSmDdIpIEKnbV8+nfvkZzWwe/ur2c+TNHB13SoNarDi53Xwos7TLtGzHD18a5LhFJYu7Ogy/v4ntLNzG+cAiLPzVX55gnAR2xEJE+OdzUxleeXMOyDQeYf/YofvyB8ykcqptsJQMFuoj02utVjdz58GvsP9zM126cySffNln95UlEgS4iPWrvCHPvC9v52fJtjB6ey2OfvpSLJhYFXZZ0oUAXkTOqrD3Klx5bw5rqw9x0/ji+c/NsCoZmBV2WdEOBLiLd6gg7v35pJz9ctoW87Azuue0i3nne2KDLkjNQoIvIKdbXHOZrT61nTVUj184czff+bjaj8vVAimSnQBeRk461tPOTP27lwZd3MiIvm3++5QJuvmCcDnymCAW6iODuPLNuH//76U0cONrMbXMm8pXrz1ZfeYpRoIsMcqt2N/DdZzby2p5GZo4dzr996CKdwZKiFOgig1RVfRPf/8Nmnlm7j5L8HH7w3nN538UTyAipeyVVKdBFBpn9h5u594VKFr9aRSgEn58/jTuunEJejuIg1WkLigwSB440c+8L23n41T2Ew877y0v5wvzpjCnQ2SvpQoEukub2HGri3/+6g8Urq+gIO++7qJQ73z5VTxJKQwp0kTT1elUj9/95B8+u30fIjPdcOJ7PvX0aE0cqyNOVAl0kjbR1hFm+6QAP/HUXr+6qJz83k0VXnsVHLytT18ogoEAXSQPVDU08urKKR1dWUXu0hfGFQ/j6u2ZxyyUTGKaDnYOGtrRIimptD/PClloeWVnF81siT368ZsYobpszkatnlOiZnoOQAl0khYTDzqo9DfzX6hqWrttHY1Mbo/JzuPOaqdxyyQRKi9Q/Ppgp0EWSnLuzYe8Rnl2/j6dW76Wm8QS5WSGuP2cM775gPFdMKyZLe+OCAl0kKXWEnZW76lm2YT9/3HCAmsYThAyumFbCl6+fznWzxuhCIDmFfiNEkkTD8VZe2n6QF7fUsXxzLfXHW8nODPG2qcV8fv5U5s8cTfGwnKDLlCSmQBcJSFtHmNV7GvnLtjr+vO0ga6sbcYfhuZlcc/Yorj9nDFdOL9FZKtJr+k0RGSDNbR2sqWpk5a56Xt3VwGu7GzjW0k7I4MKJRXxh/jSunF7CeeMLdIaK9IsCXSRB6o62sLa6kYrdDazcWc/a6sO0doQBmD56GDdfMI63TSvm0rOKKRii+47LW6dAF4mD+uOtrKs5zLrqRtZWH2ZdzWH2HW4GIDNkzB5fwEcvL+OSshGUTyqiKC874IolHSnQRfqgpb2D7bXH2XrgKFsOHGXL/sirpvHEyTZTivOYM3kE544viLxKCxiarX9qknj6LRPpwt1paGpj58Hj7Dp4nN2HjrO97jhbDhxl58HjdIQdgKwM46ySYVw8qYjbL53EuaUFzB5fwPBcdZ9IMBToMii1dYTZf7iZvY0nqGk8wa5DTew6eJxdhyIhfqS5/WTbkMGEEUOZMTqfG2aPYfrofM4ek09ZcZ4u6JGkokCXtNPaHubgsRZqj7ZQe6SZfTHBvbfxBHsbmzlwtBn3Nz4TMhhfNISykXncfMF4yorzmFw8lLKReZQWDSU7U8EtyU+BLknP3Tna0k7j8TYamlpPvuqOtlB3NBLcncN1x1pobGo75TuyM0OMLxzCuMJc3jatmHGFQ6LjkWnji4aQk5kRwNqJxI8CXQZES3sHx5rbOdbSztHo+8nxlnaOnGijsamVxqY2Gpoiww3R8cYTbSf7rbvKzQoxKj+XkvwczioZxrwpIynJz6EkP4dR0fexBUMoHpaNmR5+LOmtV4FuZguAnwEZwK/c/ftd5ucAvwEuBg4Bt7j7rviWKoni7nSEndaOMC1tYU60dURerR00xwyfaIuOt3ZwItquOWZeZ7vYsO4c7jz/+kxys0IUDc2mcGg2hUOymDEmn8Kh2RQNzaJwSDaFQ7Oi87MoystmVH4Ow3IyFdQiUT0GupllAPcA7wCqgZVmtsTdN8Y0+wTQ4O5TzexW4AfALYkoONHcnbBD2J2wO+7gMeNhf3Mb91M/Ew5DezhMR9hpD0fCMnb4TfM6nA6PnR+mvaNreyccOz9melt7mNaOMG0dYVraw7S2R4Yj705re5iWjvDJdrHzW2Pevfsd4B7lZoUYkpXBkKwMcrMj78NyMhlXmMuwnEyG5WYyLCeL/NzMyHhOZmQ4N5P8nKzo/Mi03Cx1eYi8Fb3ZQ58DVLr7DgAzewS4GYgN9JuBf4oOPwH8q5mZe39j4vQeW1nFfX/ZcTJMY0PU3XHoNnjD4ci804Vz53uqCBlkhIysjBDZmSGyM0JkZYTIyQy9MS0zRFaGUZCdRXaGvaldduYb7bNjP5MRYmh2BkOyM8iNBvWQaFDnxgwPycogJzNEKKS9Y5Fk0ZtAHw9UxYxXA3NP18bd283sMDASOBjbyMwWAYsAJk6c2K+CC4dmMWN0PmYQMiMUfafLuL1p+M3vXdtYN5+JTD/9Zzq/75TPYGSEjMwMI2RGZuiN8YxQ6I3x6HvGyfHQm6ZH2hsZFp2XEfMZMwWpiJxiQA+Kuvt9wH0A5eXl/dofvu6cMVx3zpi41iUikg56c3JtDTAhZrw0Oq3bNmaWCRQQOTgqIiIDpDeBvhKYZmaTzSwbuBVY0qXNEuAj0eH3AX9KRP+5iIicXo9dLtE+8TuBZUROW3zA3TeY2beBCndfAvw78JCZVQL1REJfREQGUK/60N19KbC0y7RvxAw3A++Pb2kiItIXukGFiEiaUKCLiKQJBbqISJpQoIuIpAkL6uxCM6sDdvfz48V0uQp1ENA6Dw5a58HhrazzJHcv6W5GYIH+VphZhbuXB13HQNI6Dw5a58EhUeusLhcRkTShQBcRSROpGuj3BV1AALTOg4PWeXBIyDqnZB+6iIicKlX30EVEpAsFuohImki5QDezBWa2xcwqzeyuoOuJFzObYGbPm9lGM9tgZl+ITh9hZv9tZtui70XR6WZmP4/+HNaa2UXBrkH/mFmGma02s6ej45PNbEV0vR6N3rIZM8uJjldG55cFWXd/mVmhmT1hZpvNbJOZXToItvH/jP5OrzezxWaWm47b2cweMLNaM1sfM63P29bMPhJtv83MPtLdsk4npQI95oHVNwCzgIVmNivYquKmHfiSu88C5gGfja7bXcByd58GLI+OQ+RnMC36WgTcO/Alx8UXgE0x4z8AfuruU4EGIg8gh5gHkQM/jbZLRT8D/uDuZwPnE1n3tN3GZjYe+DxQ7u6zidyCu/NB8um2nR8EFnSZ1qdta2YjgG8SecznHOCbnf8J9Iq7p8wLuBRYFjN+N3B30HUlaF3/H/AOYAswNjptLLAlOvxLYGFM+5PtUuVF5OlXy4G3A08DRuTqucyu25vI/fgvjQ5nRttZ0OvQx/UtAHZ2rTvNt3Hn84ZHRLfb08D16bqdgTJgfX+3LbAQ+GXM9De16+mVUnvodP/A6vEB1ZIw0T8zLwRWAKPdfV901n5gdHQ4HX4W/wx8BQhHx0cCje7eHh2PXac3PYgc6HwQeSqZDNQBv452M/3KzPJI423s7jXA/wX2APuIbLdVpPd2jtXXbfuWtnmqBXraM7NhwJPAP7j7kdh5HvkvOy3OMzWzdwG17r4q6FoGUCZwEXCvu18IHOeNP8GB9NrGANHugpuJ/Gc2Dsjj1G6JQWEgtm2qBXpvHlidsswsi0iY/6e7/y46+YCZjY3OHwvURqen+s/icuAmM9sFPEKk2+VnQGH0QePw5nVKhweRVwPV7r4iOv4EkYBP120McC2w093r3L0N+B2RbZ/O2zlWX7ftW9rmqRbovXlgdUoyMyPybNZN7v6TmFmxD+D+CJG+9c7pt0ePls8DDsf8aZf03P1udy919zIi2/FP7v5B4HkiDxqHU9c3pR9E7u77gSozmxGdNB/YSJpu46g9wDwzGxr9He9c57Tdzl30ddsuA64zs6LoXzfXRaf1TtAHEfpx0OFGYCuwHfha0PXEcb2uIPLn2Frg9ejrRiL9h8uBbcBzwIhoeyNyxs92YB2RswgCX49+rvvVwNPR4SnAq0Al8DiQE52eGx2vjM6fEnTd/VzXC4CK6HZ+CihK920MfAvYDKwHHgJy0nE7A4uJHCdoI/LX2Cf6s22Bj0fXvxL4WF9q0KX/IiJpItW6XERE5DQU6CIiaUKBLiKSJhToIiJpQoEuIpImFOgiImlCgS4ikib+Py0ka1Hyxe+bAAAAAElFTkSuQmCC",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"lre = torch.linspace(-3, 0, 1000)\n",
"lrs = 10**lre\n",
"\n",
"plt.plot(lrs.numpy());"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hiOSxckvHGGr"
},
"source": [
"Ahora, para elegir el número correcto en este rango, podemos registrar nuestras pérdidas correspondientes a cada tasa durante el entrenamiento:"
]
},
{
"cell_type": "code",
"execution_count": 406,
"metadata": {
"id": "FZvwJ9N_KNN0"
},
"outputs": [],
"source": [
"lr_i = []\n",
"perdidas_i = []\n",
"\n",
"for i in range(1000):\n",
" #minibatch («minilote»)\n",
" ix = torch.randint(0, Xdev.shape[0], (32,))\n",
"\n",
" # propagación hacia delante\n",
" emb = C[Xdev[ix]]\n",
" a = torch.tanh(emb.view(-1, 6) @ H + d)\n",
" logits = a @ U + b\n",
" perdida = F.cross_entropy(logits, Ydev[ix])\n",
" #print(perdida.item())\n",
" \n",
" # propagación hacia atrás\n",
" for p in parametros:\n",
" p.grad = None\n",
" perdida.backward()\n",
"\n",
" # actualización\n",
" lr = lrs[i] # intentaremos obtener el límite inferior\n",
" for p in parametros:\n",
" p.data += -lr * p.grad\n",
"\n",
" # registrar estadísticas\n",
" lr_i.append(lre[i])\n",
" perdidas_i.append(perdida.item())"
]
},
{
"cell_type": "code",
"execution_count": 407,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 265
},
"id": "NG28-zOEKOAS",
"outputId": "4ff177dc-1a85-4927-ebf7-cac0e22f8baa"
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO2dd3wUdfrHP9/dbHpCIIRID1UIVYyFpiCICioW7OfhnfXEs54ep6Cc/rB7p57e2bDjHYrYACkiCCgtRCC0SAsQICS0FNK2fH9/7MzszOzM7mzNTvZ5v155ZWfmOzPf2fKZZ57v830exjkHQRAEYT4szd0BgiAIIjhIwAmCIEwKCThBEIRJIQEnCIIwKSTgBEEQJiUhmidr27Ytz8vLi+YpCYIgTM/GjRuPcc5z1OujKuB5eXkoLCyM5ikJgiBMD2Nsv9Z6cqEQBEGYFBJwgiAIk0ICThAEYVJIwAmCIEwKCThBEIRJIQEnCIIwKSTgBEEQJqXFCHhxWRW2lJ1q7m4QBEFEjahO5IkkV7yxGgBQ+vyEZu4JQRBEdGgxFjhBEES8QQJOEARhUkjACYIgTAoJOEEQhEkhAScIgjApJOAEQRAmhQScIAjCpJCAEwRBmBQScIIgCJNCAk4QBGFSSMAJgiBMCgk4QRCESSEBJwiCMCmmEfCqejsqaxqbuxsEQRAxg2kEfOhzy3DOzB+auxsEQRAxg2kEvK7JCQBwujhmr9sPu9MVtmOXlNfg4Im6sB2PIAgiGphGwEXmbDiIJ77aindX7Q3bMS95dSVGvrg8bMcjCIKIBqYT8OoGOwDgVJ1dc7sjjJY5QRBELGM6AWfCf8655vZXlv4Wvc4QBEE0I+YTcEHBdfQbxWVV0esMQRBEM2I+ARdscLl+Nzk8bhMOHWUnCIJoYZhOwLXoPe176bWLXOAEQcQJphNwfy4UssAJgogX/Ao4Y6wzY2w5Y2w7Y2wbY+wBYX0bxthSxtgu4X/ryHfXg55Q6wk7QRBES8OIBe4A8AjnPB/A+QCmMMbyAUwFsIxz3gvAMmE54lgEE1zfAicIgogP/Ao45/wI57xIeF0DYAeAjgAmAvhIaPYRgKsi1Uk5HheKngVOEk4QRHwQkA+cMZYH4CwA6wDkcs6PCJvKAeTq7HMXY6yQMVZYWVkZQleF4wn/RZlWCzbpN0EQ8YJhAWeMpQP4EsCDnPNq+TbuVlFN6eScv8M5L+CcF+Tk5ITUWaEfwnHdyy7VWV2k4ARBxAmGBJwxZoNbvGdzzucJq48yxtoL29sDqIhMF9V9cf8XBzHVgk3yTRBEvGAkCoUBmAVgB+f8H7JN3wKYLLyeDOCb8HdPoz/Cf48FTi4UgiDikwQDbYYDuBVAMWNsk7DucQDPA/icMXY7gP0Aro9MF1Uw5UxMtWCTfhMEES/4FXDO+Wp4DF81Y8LbHf8s23EUgEe4vQQ8QBPcpXaiA/i88CC+/vUQPrvz/KD6SBAEEQ2MWOAxxYoSMZJFxwceoAnu1NjhsblbgukaQRBEVDHdVHoRXR94gE4Up4YFThAEYQZagIBrrzcKhR0SBGFWTCvgImqfd6AGNVngBEGYFdMKuCcOXLm+ut6OZxfuMFz0mAScIAizYgoB14os+bywDL2f+B7lVQ2K9YdO1eOdlXvx3ebDho5NAk4QhFkxhYDP3Vimub7J6cK9szdqbnMYFGatKBQRrRBDgiCIWMEUAr7p4CndbafqtavT6wWuq/FVwYcGOAmCiGVMIeCNDn2VPVWnLeBG8WmBk34TBBHDmF7AQ8Xp9CXgnm1/nbsF1/7nFwDAkar6iPWHIAjCKKYQ8CaHM+zHPFrdAM45mnxEq8gFfE7hQWzcfxI//VaJoc/9iCXbysPeJ4IgiEAwhYCH2wLffrga5z27DJ+uOwCHDye4lgtl+2F3KvSiA/p+eYIgiGhgDgG3By7gYuEHLfYdOw0A+GX3MdgdHpVudDhx6asrpWWtQcwEi/u4Tl+jnwRBEFHAFMmsGsPsQhE0GC6VC6WiuhE7y2ukZZeL45O1+/Hy4hJpXYLVvbPdh++cIAgiGphCwBMTQntQWLr9KJwujkv7nwEAsEhWNLC3slZqpza4XRyY/vVWxboEq0XYlwScIIjmxRQulCfG5we8j9yBcufHhbjnU8+EH6vgXjnd6MCjstSxpcdPK47hy4Xiy3dOEAQRDUwh4JYQevnc9zt0j1fX5FCs//376xXLvgScXCgEQTQ35hBwHwOSeoi7vP3TXq9tf/2yGID/6fbnzlzmtc5GLhSCIGKEFivgx2ubMHDGYsW6X/YcAwBU1jQCCE6Exa4YzXZIEAQRKUwh4NYgerl273FUNyhdJDe/u06xHIyAi24VB7lQCIJoZkwh4L5iuvVYtrPCbxujGQvliIZ3MPsSBEGEE1MIeDAuFCOIE3oCgSbwEAQRK5hCwK0REvBgEC1wrSITBEEQ0cQUAm5Uv7PTEv22+d/6AyH1xVf6WX/MXrcfm33kNicIgggEUwi4OHPSH2lJ/ieWTp1XHFJfnIIJHoyMP/HVVkx88+eQzk8QBCFiCgE36kLpkJUc4Z4AYvCJPxfKvKIyVOlUCyIIgggHphBwgwY4bj6va2Q7As8gpi/53llejYc/34xHv9gc8f4QBBG/mELAjYYRJoWY9MoIUhihk2P/ce0olloh/ryytlFaR4OeBEGEG1MIuNWgCW6zRj5aRZzIs3r3MVz40gpUVDd4tREnCCXI+h1o3PifPt2IBVuOhNBTgiBaOqYQcKMulASLBe1bRdYPfqquSbH81LfbpNd/m7cFeVMXSAJutTC4XBwPz9mEjftPBnSe77eWY8pnRaF3mCCIFospBNyXCyU10Sq9TrAyLHrggoj25d1V+xTL328tlwT7v+sPAvCEGiZYLKisbcS8Xw/htg+UmQ7lfP3rIZxudOhuJwiC0MIUAu7LhfL2rWcjK9XmbscY0pOjX6PC7nQp8qqI7hKLhUlulAadsnAb95/Eg3M2KSx5giAII5hCwH25UCyMoXduBgB3BR2rheG1GwdHqWdu7E4XVu8+Ji07nd4+cDWn6prQYHeipsEdanhUw5dOEAThC5MIuL4QMnjixMUBxomDO0ajWxL1dqciR4pYrcdqYbozNwc/vRSX/2u1FI4odxO5KFEWQRAGMEVNTJ/JrJinwo5WBZ1ocO7MZWib7pnGL7pLrIx51dmUs7uiVgool19hKNP1CYKIH0xigfvaxpBic9+HoqV7GRpT9o/VeqJTHpyzCQBgtTLMKzrk81hcUHD5NTbXjYggCHNhegucAXj2mv7otjIVw3u2jUp/kmxW1BiIGkmwMLywaKdinfpmJHpelC6UkLtIEEQcYA4LXKV6bWRZBxljaJeRjCcm5Bue8BMq8tBFXyzXKCqh7qNoa8tXkwuFIAgj+BVwxtj7jLEKxthW2boZjLFDjLFNwt/4yHZTyTdThkuvo6TZChINTtlXl3QDvJ8mPO4SBs45Zq3eh/IqikghCMI/RlwoHwJ4A8DHqvX/5Jy/HPYeGSDJZkFGcgJqGhyGc4WHk8wQYs0ZAz4vPCgti/ptYe5BzWfmb8cXZ2SE2kWCIOIAv6Yk53wlgBNR6ItPpl+er1hmGq+MwFjoSa86tk4Naf/HZTnJxSRX+4/X4Tsh98mpOkpDSxCEf0JRsvsYY1sEF0vrsPVIh9tHdEPb9CRpWfSLB+pCufncLiH35e9X9gtpf7mPW3xVcrQGry/bBaB53EIEQZiPYAX8PwB6ABgM4AiAV/QaMsbuYowVMsYKKysrgzydxnE9xw983xAFso2B0m2+kI9RaoUMBnNNBEHEH0EJOOf8KOfcyTl3AXgXwLk+2r7DOS/gnBfk5OQE20/VQT0iF25rdXDnrPAeUAVTuXwo4IQgiGAJSsAZY+1li1cD2KrXNpzMmlyAq8/qiLbpSZJwqwVRRM+IZUx/n5vO7RIWF0sg1NudXuvIACcIwghGwgj/C2ANgDMZY2WMsdsBvMgYK2aMbQEwGsBDEe4nAGBQ5yz884bBgv+bCf3T6beP4+jt8+zV/ZGaZCzGO1jU5/7ol1KfbXaWV+PE6SavNgRBEH7j4TjnN2msnhWBvgSEKHL6ljbT9E/oWd8/PHwBGGOGCygHS12T0uKubvCOOJH38dJXVwEASp+fENF+EQRhPkwxE1MLvy4UH/tqbevZLryx130MxnIfPFEf8rkqahrwjyUllMWQIOIM0wo48+dC8aHgF+fnRqBHSowKuBZODSHmnINzjrV7j8PhVCZLeWzuFrz+424U6pRtW7DlCAr+bynsTkqyQhAtCfMKuCDQeomufA1uvjhpENb+bUykugYASLAG/9Y2OryF9vJ/rUa3vy3Eje+sleLFRUS3jJbwA8CM77bhWG0TTpIvnSBaFKYVcFG49Sztzm1SdPdNTLDgjFbJ+OHhyNXP9FWNxx+NDu/IlG2Hq6XXeypPa+6n916IXaEkWQTRsjCtgIvoyeR/7zwfZ3f1niAqbx+M3zvRoGVtC8EC75GTbrjt9W+vwfp97kwHevosDszqWegEQZgT0wq4JwpFW8LbZSZjXBC+bl++8wX3j8DPUy8CAMz/8wifxwklta1f6122WRRvX4hpB5o0XDMEQZgX0wq4PxcK4C5yDAC9cz0Wrb9p6r68DB1apSAnw52PpVWKzedxQkmY5W+wMZBbw8+7j6HspDvSpYkGMQmiRWFaAZcscB9txDwjVkt4LjPB6jmbP5E1mjNci81lVT63z99yRNNPzuF993nrpz3S60a77z43OVzYLvO1EwQR25hXwA204ZKAe9ZdMahD0OdMkN0I8rLTcMeIbhh1pnZ+F6O+8mB5b9U+Q+3kvnh/Fvgz87dj/OurcPBEXUh9IwgiOpiiJqYWogvF17CcqFeiBT6yV1uvgc2v7h1meHBPboFbLAzTLs/HtK+LNduKrpZI8cqSEry0uMRvO7k/3Z8PfKMQR15Vb0fn0LpHEEQUMK2AiyY49+G0Fl0oiVZ9e/2sLkpBH9GrLfp1yFSE7YloDS7qxaFPGNgeVgvDo3O36J47FIwGlCgscD8CTjEqBGEuTO9C8TXoKIr70B5tcduwPLw4aaDf42Yk27Dg/pHa59QQa71bQ1KCFdcVNL8da5PdvLQmCMkR3y+9mxJBELGFaS1wIy4U0Uq1WRhmhFhFRw+9qBabD6s/Ymi8GQkB+MBFSL8JwhyY1wIXREaroo0aSxAx2TcYtJ6H92yrud5XuKLNyvDajYMxspdy379d1sd4Bw0id6E4Xf4scPd/EnCCMAfmFXDBeeFLv++6sDtuKOiMycPyAj7+C5MGYulD/qfaX5yfi+IZ4wIuszZxcEev2ZpXDg4+QkYP+ZOAPwNcvBnq5ZEhCCK2MK0LhUmDmPptMpNteMGA31v/HMaELCPZ96QeoySEGK+u9VbIj+kv3SwNYhKEuTCvBS6IqxEXSrCInpf2rZIjUlBBHUETauy41lthS5BZ4H7eK7E/WhOCCIKIPUwr4G/efBZuOrcL+rbPjPi5jMyq1LPV5/95BD65Xbvmc6ZqOn5CiAOfb6/c47UuxeYpEfe3ecWaJdxERH2npFcEYQ5MK+Ddc9Lx3DUDQkoa5Q+7U4wjD+xtuufCHtLr/h1bYWQv7dmaT0/srxi4DFXAV+065pXzO9mmrPH51LfbdPcXZZuyzhKEOTCtgEcDceKLIQtcpr092xlLB9sqxYa7ZWJvC0POFrVLKRAXk+hCmblgB6rqvGt1EgQRW5CA+0AU5TMykw3v8/5tBbh2SMegzhdMuKM/Xlzkf7q9iCj1a/Yex8yF28PeF4IgwgsJuA/6dcjEMxP74ZXrB/ltO67fGQCA87plG45eEZn/5xGYGqYYcNF9vWpXJWobHQHtKzfW/c3aJAii+TFtGGE0YIzh1qF5hto+fWU/PDi2F9KStN/SXu3SsauiVnNb/46t0L9jq2C7qWD9vhPo0z4Dt85aH3DxZnn0CY1jEkTsQxZ4mEiwWtAuQ9/V8sU9QzFrckHE+zHlsyLUC0WON+pUqR/10nLN9fKJmpEMzyQIIjyQgEeJrNREDOyUFZVziYOuJ3Sq0JceN5Dvm/SbIGIeEvAoEsmQRzlGsglqxXrLJxaRBU4QsQ8JeBSxGJj+P6xHdsjnMSK+PR5f6DW1Xr5EAk4QsQ8JeBQxEib4zu8L8N19vivei+TrzELVqtepdWqHWsBli4u3HcWE11cZ6gdBEM0DCXgUMeLaSE9KwIBOnoiU2Xecp9t21Jk5GNvXO9Lkgf9tMnRuu9OFRz7fjF8PuAc71Va3VlUigiBiBxLwKGINItG2Xr5xwF31R2uW/26NcEWtU0//eiu+LCrD1f/+BQCNWxKE2SABjyLhLpQQSO4UrRzf8349pFgmtzdBmAsS8CgSaq1JdVZDm5UZzxxo4NS+CkQTBBF7kIBHETGMMNjKOyN75WDDE2Nx7ZBOANzFGtQDkXr4q0jfYHeSC4UgTAZNpY8iVgvDpicvRrrOdHsj5GQkIdnmvu/arAwOZ3hkt8/0RWE5DkEQ0YMEPMpkpQZWO1MLUbQTrBbNkEGCIOIDcqGYBHllHbuQtCTBEoAPnCCIFgdZ4Cbg2/uGI1eWk1y0wG1WC+wk4AQRt5CAmwB1EiyHaIFbGRzkQiGIuIVcKCbkkXFnYkiXLFzYOydsg5jBsO/YaZRXNTTb+Qki3vEr4Iyx9xljFYyxrbJ1bRhjSxlju4T/rSPbTUJOj5x0zLt3ODKSbUhOtOq2+99d50e0H6NfXoHzn1sW0XMQBKGPEQv8QwCXqtZNBbCMc94LwDJhmWgGOmbpF5HIyUgK+/k4515ZDAmCaB78CjjnfCWAE6rVEwF8JLz+CMBVYe4XYZDHLumDIV20C0XIc69MOrtTWM73ly+2oPvjC8NyLIIgQiNYH3gu5/yI8LocgG7xRcbYXYyxQsZYYWVlZZCniz9uG5aHQZ39V/DJa5uGefcO19wmt5Nbp9oAeBeV6JGT5vP4y0sq8OgXm6XlL4vKAICscMK01DTY0WB3Nnc3wkLIg5jcnUBD99fMOX+Hc17AOS/IyckJ9XRxw4wr++GbKdrCrEWrFJvXOnl6WCZY4+q4cX8y/IcPNuCLjWVe62tkFe9///56w/0kiOZmwIwlGP9ay8h1H6yAH2WMtQcA4X9F+LpEBEPR9Iu9kl3Jk1OFmghRPJbolamut0vbVv5GT1aEudh77HRzdyEsBCvg3wKYLLyeDOCb8HSHCBarhSHZpoxIyZRb5XoKbtATIhru6YnuqQOPyNwq4aDJ4UJdk8N/Q4IgJPxO5GGM/RfAKABtGWNlAJ4C8DyAzxljtwPYD+D6SHaSMMbgzlm4oaAzLu1/BnrlpqNdRjJ+ePhCVDfYsWTbUc19jHqyHS4XrBYrkmwW1DQC6/epx7VD4+p//4xth6tR+vyEsB6XIFoyfgWcc36TzqYxYe4LESI2qwUvTBqoWNezXToASAI+YWB71DU6MOrMdsjvkInH5m4xdOxw5FzZeqgK5VUNGJvvPeZN5dsIInBoKn2cIPqu89tnYsrontJ6o0UcHC4OzjmO1TZpbt9ZXo1vNx3Go5ecCcYYNh88hfwOmbDJar5d/q/VAEBWNkGECRLwOEF0gQdbdcfp5Fi165ju9uvfWoPqBge+3XwYo89sh0/W7gcAbH/6EqQm0teMICIB5UKJE0QLXK3fWnKel53qtc7h4jhZp219A4BdyMlSdrJeEm8AeGlxScB9JYhYxuF04eHPN2F3RU1zd4UEPF7QKmoMeAv6azcOhpa72+niUiy5Fi4dy76iptFwHwnCDOwsr8G8okOY8PpqHKtt3u83CXicIFngqvVctebC3jk4cKLOa3+70+UzllzPM3O6kUIDiZZFYoJbNhsdLox6aUWz9oUEPE7w+MCV69XLepa62wLXP76eBV7bQAJOxA7BjgHJSZQNzNc2s4FCAh5nqC1uNUznG+HwE0aot1XvC/7z7mPIm7oAO8spfJCIHpEsQdjocKK6we6/YRghAY8TRP+1PwPEojKzrzmrIwD3F9/Xd1/PAtf7wczfchgAUFh60neHCCKMOMNggesd49b31mPgjCUhHz8QSMDjBF0fuGqFhXl8fAAwpKu7Vse/ftyFJod++Ta934XWl51zjiaHe738XIGwaGs5Fm094r8h0aL5eE0pnvxmq992IuGwwPXcMOtL3bOT7aoyh0erG3DytH4EVyhQgG684ccCsTCGJQ9egA2lJ3Dl4A74ebc79nv+liOYvyVwwdRKO/vqD7vQ6HCn80xSCfi7K/fijpHdfEa8AMA9n24EQJOC4p0nv9kGAHh6Yn9D7UMV8HlFZfjq10OKdZ+sKUWD3SPaR6sb0Km1JxT3vGeXwWZl2DVzfEjn1oIEPE4QByf9fX0Zc+cYz2vrzhNutYT2kCb6zuU/nNeW7UIvYYq/fEAIAGYu3IHRfXLQs11GSOclCC1CFfCHP/dO4jZduImInDxtRydVkUl7hGrXkgslTtCdyKNaofaBJ1hCS0RbdrIeThf3SqDfKLhjEhMsOKF6vAyDm5IgNPE3GB8OmpxOHDxRh1W7Ip9mmQQ8TjCaTVYt4OoKPsGwoPiIl4A7BD9h6fE6DHlmqWKbP/cJQQRLNCpJNTpcuOiVFbh1VuQLnZCAxxn+wgjVeq22wG89v2vA53S6XGhQDYA2CY+UuytqvdqH46ZBEFpEwwJvdLgi5jJRQwIeJ1gs2mGEXhN5VNavukjE2V1Vzj0DfLf5CMpUszsdLrega1lEeyu9RZ0gwkEk48BFfEVrhRsS8DihTVqi4r9RMpOVtTbVUSNG+HFnBW7/qFCxzilYKFoWkbotQYQLuYDruVPeW7UXRQdOYtfRGuRNXYCfAiwZKBdwMYorUlAUSpxwQ0Fn2KwWXDW4g2J967RElFc36O6nLpacZAvunq+ekdko+MCdruhZKwQhNxjsLheSLFavNv+3YIf7/1Xu0MRFW8txYW/jBdnlAi4vmFLTYEdGsnfx8VAgCzxOsFgYJp3dCQmqsL0PbjtH+qJqkZ6svMcnJ3h/4YNBnOxgxFX48uKSgK0ggtBCnlzN4efLJ84utgaokk1ObaNk4/7wzzomAY9zzmiVjN/5GJhUDyjagpw5qUb0vRuxwN9YvhuT34/8iD7R8pGHrPoTcNHdYg0wKqpRFnF16FS99DoSg/Mk4IRfeuemS6/lUSmpiVZMvzw/pGPLZ7Bpb3f63E4QgPHwwONyAfdjPIgCbglQePUscBJwolkY1qOt9Fpe49JqYSH7sOub9AW69Nhp9Jm+SFpeu/c4Nh88FdL5iJaJ0SRVJ057CjBoDaDLJ7ZJLhQdC7xAJyJLzygJ1JI3Agk44Rf5lzrB6vkSWi0s5Ljaqnr99JtbD1cplm98Zy0mvvlzSOcjWiZGwwMbZeKqTjqlPo743bZatYX34Yt7a64vOqDt607QOU4okIATfpH/NBIsFuS3zwQA9D0jUwoH1OKGgs5+j739iH4+8Dof1jlByDEq4HZZO619HBphhnqWc2aKDX2F34Kc8irtqK5Q8wppQQJO+EWe69tmZZj/5xH47I7z8NatZ/t8dL16SMeQzuvLvSL1LQoTM4jYx+iToNzq1potqYgTF16K6SUWFiuzcVoYQ7XGE6TeRJ5IuFAoDpzwi/y3kWC1wGJhGNbT7Rf3NZIfaiKs003+y1U5XByJNPU+7jFqgTtkAq41iCk3SOSDmBtKT+De2UWKthYLUKfxHW3UE3AaxCSaA66ywOX4snwCHb1XU2Ognqa/SAIiPjD6PZBb3VrGh9wlKAp4goWhqs7b0rYypunmE3Pde7UnASeaA7mXxKby44lRKAM6tvLaL5RHxhOnm/CfFXs0tx2VzRyNRnIiIjaRu0OM3sflQq/13ZGvc0oTebS/x4wxTWu7US8KhQScaA7G9cuVXqtH0sXfkHyqsVgmLZAv7LQJfRXL6hSzcp6Zv1167W8yBtFy+eiXUum1Pwu8rskBu9Ol+L44NKJQ5OM94viKhTEEYovouVBCdSlqQQJO+OWiPh4Bt1m1LfAM2ZT7124YjGSbBZ1apxg+xx0juxtuq5gOTS6UuOWUzK3hzwee/+Ri3DprncKFojWIqYhCEcScMegIuKdtiixrJ03kIaJOssEkVWorQgzLSk30fIEvG9AeO5+5DGlJ3mPk5+Tpp6N95bpBhvpwWuZ3JAs8fpGPsRhxpa3de0LhdnG4XOCcKyfvaBzH6eJSSUI58qbtMpP8np8EnIgYqx67CD88fIHfduovoTgbLb+Ddzyslg98yuieusdun5Xs9/yA8hE1GvmdidhE/v0yHIUie2JrcrjwzPwd6Pa3hbLt3sfRC1WVjw0ZSfIWCQGnMEICAJCTkYScDP9WhLrgwzVDOmFoj2y0b+XtLhEtpDtGdMN7q/cB8HbBKNobdDTaZD8Erdl0RHwg/yr5EnC5hS13m3xZVIaFxeUAgIMn6tC5TaoiNURFjXva/dGaBvywrMLruC7O8cfh3bC57JQU+51otZALhTAXcvFunarMd1z6/ARMkyW8CoeAywdS1+87YbSbRAuDGbTA5YLqcLqQJrj7RPEGgJEvLheO49lv/hb3xJ1P1x7QzMGTk5GEJ6/Ix5d/GiYN3LdO08/3TblQiJhm9V9HY/lfRvlsI8aRJ2oIudpA0aselCALZZw6r1g33FCPkvIavLdqb0D7ELGH1aAPXD4z0uHiSNUYm/FsN+aeu+W8Lmib7nliFbvSOlW/4pVeTpVQIAEnwkan1qnI8vEFBjwhhimyQc97R/UA4O2e6dkuHVqoDZkXFu1EdYN+Uiw1E99cjf9bsIOm4ZscpQ9c35UmHzNZteuYZIFrYdSXrhZq8QaSlapvgVMYIWF6RBeKPGrl6rPcOVPU3++87FTNY2jlmhg4Ywl2aCTGWlFSgW82HVKsE9N96sXrErHB099tx5vLd+tuV0Sh+IhGUrs/UhO1LfAPft6HK98wlu3SO5zWfX5fFrhRF2Eg0CAmEVXE8aSURCsKp43F4VP16JWbAcD7Cy5/RJWzTsfvfd9nRVj04AWKH9dtH2wAAHTNTsPgzlkA3JaQw8XRYHcqngSI2OL9n3Ga3ZIAABcxSURBVN0D33qRS3KPhJNz2J0u/LizAuPycxVPc+oi2ak6n/nfv9uuuV4LW4IqpYRTtMD1BZwscML0iBV2UhOtaJuehIGdsqRtagPl0v5nYMKA9oaPvafyNHo98T0++qXUK8nQVbI84uIgaINOzgoieI7VNuLN5bsVkR/h5uCJOjTYnQofuNPF8eHPpbj7k41YUHwENQ12/HXuFuRNXeC1vy8fuFHUYzgeC9zHICYJOGF2erZLR05GEh6/rK/Pdr9MvQgDO2XhzVuG4LZheQGd46lvt+HOjwt1y7GJ+Vz8lXMjAuevc7fgpcUl+LLokP/GQeBycYx8cTmmzC5S3PEdLo4aYRzkt6O1mL3uAOYUHtQ8RorBSWu+ULtQ6oXvmt5TI+A9xhMOQroSxlgpY6yYMbaJMVbofw8i3klLSsCGJ8ZK6Wj1SJZNTQ7m0fPn3cc1/acVNQ2oEabiU73N8FMrvLd/+WKzbpuS8pqgJ2DZhcHKZTsrFFb+7R9uQJPgxjhW24gkH8W3fYWyGkV9DLFYcre2aV5txTGeSBAOC3w053ww57wgDMciYpQJA9pjUOcs/w3DhHxqf7CGi9ag5p0yfygJePjxN1D364GTuOTVlfhA8G9r4XJxTdcHoBysVBdfeOunPUIbF9JVbpIRPbXrugaLOq2yuJSb6T2b+OXrBmHH05eGfE4tyIVCGOLNW4bgmynDg95//IAz/LaRu02TZFOTg51s+cMO79lz+0/USa9DdaGcbnRE1NdrRvz5eTcJESF7j53WbaM3kxFQCrjeoKOFMUVWQQCKxGo2K8Pndw/12U9/nN89W7E85+6heOqKfHTM8p6RbLWwiA2WhyrgHMASxthGxthdWg0YY3cxxgoZY4WVlZUhno4wK/+6aQh2PmPcCpELAUd4RNLl4ooMdqEMYpYeO41+Ty3GnA3aftZ4xd/T0vFat6shx4evWCufdmHpCWwoPSG5UHxhsTCvm3Mr2eBigtWCc7u1wcd/PNfvsbTY+cyl6NxGGeKa3yETfxjeDa1Sbdj33PigjhsMoQr4CM75EACXAZjCGPPKhsQ5f4dzXsA5L8jJyfE+AhEXWC1M4df2xcBOyuIQcmPqk9uD+9EB3hV+GkNwoeyprAUALN5W7qdlfCG/8f5nxR6v6jTidl+35Eancp9dR2sw6a01uO6tNYayTzbYnZJPWqRVikfAxVw6F/TOwbNXD/B7PDX+njIiMVipR0gCzjk/JPyvAPAVgOB/XUTco/fTlD8OZyR7fogpNqs0i9MI9SrBrqxV/siPVjfgvVV7DblFxN8oOVCUyH3gLyzaiXmqaJRFW903PK3JWJxzfLJ2P77cqNzn4n+ulF6rJ2VpMa/oEF5btguAZywlK8UTn50QhA9c7hqJRE6TYAk6IJIxlgbAwjmvEV6PA/B02HpGxC3qn4dcwOURKRf1aYf0ZONf4bs/3ahYnv71VszZcAAAMP/PIzFldhEK959EeVWDIgGXdh8FS5IUXIF6EFM+mHikqh4lR2sAaNeN3Lj/JKZ/vdXn8Z/7fqfhviQmWKTPKS1JFtUURE6Sm8/rgpcWlwAIvdZrOAnFAs8FsJoxthnAegALOOeLwtMtgvAgjziTC8TpJodXjU5faGWU23qoGlsPuaNVTta5LXIx9a1I0YGTqKhpUO5IFrgmauP2y6Iy6YlG7pfepzGIGW7XQ1qiVbr5J9us+JPwtCb/zhgdXzm7q34hkuYkaAHnnO/lnA8S/vpxzmeGs2MEISJ3acgHo+oanUFZU1o02J3YU+kRlecW7sDw53/Esh1Hcc2/f8H411Yp2os3kqL9JxVFln2xZFs5DsqiYFoiagt8RUklVvzmDl6Qf44rSioVNS2B8E81b52WKMlzis0qPdnJvzP57b0LkagpnDYW53fPxrAe2X7bqtkyY1zA+wQChRESMYOe71kMPJg8tCs6ZqVI4Yx1dkdQ/kwtxBhikbdX7sWhU/V46tttAIBjKn+5KAG1jQ5c+upKqNlTWYvyKqWw3/XJRlzxxuqw9DcSOJwuzPh2G45U1Qd9DC0juq7R7S5Rf7rieyudP8z1TbPTEiV/dbLNKp1ffqM4q0trbJw21udxxNmV7992DtY/PiagPmQm60+tDwck4ETM0EUIzbrx3C6K9eJjsFi2TUxGVNfkDNuAkjpqQUQvVlx+2pN13qlsx7zyE85/bpm0LFYOOqXR1gifrt2PvKkLpOnikWDt3hP48JdSPD6vOOhjaEWJMOYetPxcZ2q7SLizQ+6pPC1NuEm2WaTxCrWrJlsjpFGrOlWyzYp2GhN19OhrwLoPFRJwImbITk9C6fMTcJOXgLv/iz88MSSsXUaSImnVpLM7BX3uWlWIocix2kbN9VpC9dm6A7jknyu9EmkB7puNL8a/tgqfrCnV3f7Oyr1Cf7RvNOFAtIBDqROtN0X+7Z/24O2fvItoyNtrVYk3Qu9c7bzxQ7pkSbMuU2xW6QlPa7aoWrBDdecUzxiHr+4dFtIxjEACTsQ84kCT+MNrl5mMf98yBG/ePATV9W6L9PdDuyLXQGVwPQ4bcBtMfGM1pswugsvF8YcPN3htn7V6L0qO1qC4rEpaN2V2EeqaHH6n7W8/Uo3p32zT3e6SxMdvN4NGtFDl5+CcY9vhKlTV2/HjzqOa+y3ZVo4FQvmx1hpVlBg8cfNqxNwpACQLfXCAKRsWP3gBnrw8Hz89OgqDZHMIJg/Lk/zdSQlWaYaneho8ANysMhpCzd2dkWwzPO8hFEjAiZhHjADokeNJFDR+QHtkpyehWrCcu2anKUL65FOnjbC7Qn9qt8jmsiosKD6CU/XabgyxKos83nxB8REs2XZUssBtVoaqOju2HqrSPMaXG8sUy4WlJ+B0cal6ULBWKuAO3fNVhUi8Ucqla2FxOSa8vhqD/r4Ef/ywEBXVDZi7sQxvC2MG931WhLs+2YgpnxXhwpeWY66q/wB8uk5qGuz4eE0pftl9TLoJTL88H/eP6WXomv5+ZT8wxvDHEd3QNTtNUawh2WZFmmxZvIlqTWt/cGwvrP7raNwxohvm3HV+2AbHIw0JOBHz3HxuF6x8dDTO6uIdyjWylztJ0dDu2YqY8Keu6IfCaWMx8+r+hs5xrLYRHbNSsMJPTU8AePIb71hll4vDLvhw1dZ2ss0quVXsTo5BTy/B5f9ajee+34GH5mxSuBEekWXxKyw9gUlvrcEbP+6Wbg5aE2CMcua0RXh07hbNbTUNdunmILc+D55URs00Olz4yxebpXhssfAvAOw/7m4rn/UIAMtLKlGlc9NbWHwET36zDTe/t05al5Vqw8MX95Zya18xqIPuNU1WpRo+ftrj8kpOsOLdyQW4Y0Q3dGqdgnrhJpqiYRkzxtCpdSqmXZ6P87pnS2Mrix4cqXvuWIAEnIh5GGPoolNebUzfXOyaeRnyO2TijhHdPfvAHT0wSFYwYtOTF+NrHwm5+nfMRJ5GOlA1ctESufm9tTgsRJ3c82mRYpvVwiTxkPP2T3vx1a+HUFJeo3meQ6fcbp1dFTWSC6W20QFHsNm94I7L1mLAjCX482e/AvCMNfx2tAbJqrSs8v21JuMAnrqncvSeWsqrvMcYxGIJ4o3tkn65mvtqIb/BpSRa0CMnHdMuz4fFwqQnIy0BVyNO1omlWZdakIATpkccqEpMsOB357t9maI13is3HUO6ZGHuPUORlZro8wfZJi14H/ravdpl3gBgzoYDmPTWGt3t419f5bWuvsmJ5Tvd2RStFiaJ2fVvr8GdHxdiRUkFft59zHD/5L7mG95W9kV0q4g+YsaAw6fqMe6fKzFDlfHvuGwQVe+atQYA9SxwrdBB8QYgPpjIp8H7442bh0iv5RktAU9EkRHf9Iwr+iEvO9UraVWsQQJOtCimTcjHazcOxnnd2gBw/4jn3TscBXnuZV+JiNqkuR/ZF94/0mdllUDRSmvrC845pn29FV9vOgzAbQXKQxWXl1Titg824Jb31uHRLzajrsmBES/8iLV7jyuOc/cnhch/chF2HKlG/6cWS+vX7TsBzjlKhdmQ6qyMFgbdyUly37DeDURrALCyWjuaRyvKR7wBiDct0SWjZdmr6d+xlTQIqxZq8aaemeI/NntEr7ZY8ejoqAxEhgIJONGiSLZZMXFwR91p2b4Gp8SK4vkdMqWY8+agwe5SuCp85cf+YmMZSsprUHayHs8u3KHYtlgYPF2z57jXfv9esQejXl6BNXuOe7l3GJhuTPb3xZ7si1oFM/SoadQO06zQEHZxcpZTcBt1bZuKQZ1aYdbkAgzo2MqrvRpxIFMt+DOv6o9nJvbDkC7RK0wSaUjACUIgO93zqP7KdYMCKqgcTspV1q+e+0FEtJblselTPvP44Z+e7134QEzMdNO7a3FUJaKLtpVjYbG3n1/dNy3xBdy+++1PX4Kxfdv57DcAFO4/qVj+6t5hksXdWYgkSk6w4pv7RmBkrxx8fvdQFE2/2Ocxp03oi0SrBWmqaJOs1ETcOjQvquleIw0JOBFXaA0miogWOOCe2PHmLUMwLt/4AFq4OHRSGZPuT8DFQdPjpxtRUl6DJodLCskzglzsRT5es9/nPolWi3eCLxmpiQn4z+/OlpazUm24/6KefvsijzT67M7z8dbvzlZY0imJVrRJS8R53drgsv7aVZ5uPLcLfpt5WdjSLMQyLf8KCUKGOOPuOtmsTdFn2r2t94y+PlGYDq3md7PWKZb9CbjI0epGXPLqStz2wfqAzqeVGdAfORlJmikE5NisFtw32i3ap+rseHBs74DOkZuZjEt1RHrO3UMVN4h4hQSciCs6ZKWgaPrFeHHSQGndHSPd4Yed23hP/pEXVwaAPmdkIFtjtuGsyQW4qI9/l0EwiPHVRvlFw+cdbuTuprsv7K7bTv6e6uXRnnGF79zrhD4k4ETc0SYtEYwxtBVE6PHxfbHvufGavtE/DOumWP77lf2QJDzSf/knT2HcMX1z8f5t5xg6v9aNwmzIJ+uow/zuGOF5z64v6Ky5//onPFn9LtGxsgn/BF2RhyDMzsrHRkszHPUGtlISrWidapPcBamJCUgSQssSAigm8fj4Pnh2oXv24qzJ52DcP71T0EabYT2yDVnrX907DKfq7FhRUoGPBN+4vNKOXMw3PzUOmbIZseL7KmaQfGBML9isTDHFPTkhtkP1YhkScCJukefN8IXc12u1MMkCt1oY0pMSFJNk9Ljrgh7YW3kaRQdOonduhjvr4jtrsUYVu136/AR89WsZHpqz2esY5+a1wfpS/QlDWuyaeRlsVgtueW8tft6tPJd6yruamVf3x0V92qF9K/cTw+g+7ZDbKhm/7D6umNAjT2GQnpTgdTNc//gYabLVQxe7/eBijPe9o3po5iYhjEEuFILwwwvXeiqXWyyQBNzh4vjp0VH48ZELdfcd1iMb3YTp+c9fOxBLHvK0zUxxC5+od9/dNwIAcPVZ3mlx27dKxqzbCvD9A75zc/Rsl46rBntyh8jTqarxJ+AX982VxFvk3lE98ekd5ymyN8qPrTVRql1msleWQquFYd9z4/HoJWdK7ycROGSBE4QfbjinC3IykvDQnM3o2iZNCmtrtDuRnZ6kKAjAmDst64uTBqKyphFTRuuHzj13zUAM73kYF/bOQUl5DQZ08p6k8sPDF2DsP1bird+djYxkG/q2t6F9q2QcqWrABb1zsPK3Stw/phdeF6qwz77jPLRKsaG20aHI6CefUdjnjAzsLK+RxB0AHr3kTCk2/IVrB2Dp9gqfxQtevm4QbnxnLQDPDW1Ez7b6b6IGckv98oHtcaWPpFWENiTgBGGAi/rkYvNT7vqGt4/ohg2lJ9GjnXfY4Zm5bnHs1yET/Tr4njXYJi0Rvx+aB8CdDlfOtAl9UVnTiJ7t3O4WOT8+MgoOlws2qwWnGx3ITk+SBDzZZkWyzYr3JisHVEUf9PPXDEBdkxNPz9+usJbvHNldEvAbzumCG85R5sdWc373bFw5qAO+3XwYuYLQ99IprGAEeQ4Twjgk4AQRIJf2b+8lqiLvTS7AGz/uRu/cjJDOIYY2auH2GbsFWZ2rQy/Tnri+we6UFYfwCHhiggXnd28DBuOzFF+9YTD+cf0gJFgtmHvPUM0nCCKykIATRBjp1DoVz1870H/DMPPe7wswe91+zWozAPDA2N44VW/HpILOmCsUWEhJdGdvTLS6xf1/dw3V3FcPi4XBIgi+mCyMiC5MrxJ4JCgoKOCFhYVROx9BEN402J34x9Lf8MCYXkhLIhvODDDGNnLOC9Tr6dMjiDgj2WbF4+P7Nnc3iDBA8TsEQRAmhQScIAjCpJCAEwRBmBQScIIgCJNCAk4QBGFSSMAJgiBMCgk4QRCESSEBJwiCMClRnYnJGKsE4Ltaqj5tARwLY3eaE7qW2KOlXAdA1xKrhHItXTnnOeqVURXwUGCMFWpNJTUjdC2xR0u5DoCuJVaJxLWQC4UgCMKkkIATBEGYFDMJ+DvN3YEwQtcSe7SU6wDoWmKVsF+LaXzgBEEQhBIzWeAEQRCEDBJwgiAIkxKzAs4Ye4YxtoUxtokxtoQxplmymjE2mTG2S/ibHO1+GoEx9hJjbKdwPV8xxrJ02pUyxoqFa47J0kUBXMuljLESxthuxtjUaPfTH4yx6xhj2xhjLsaYbmiXST4To9cS058JADDG2jDGlgq/56WMsdY67ZzCZ7KJMfZttPuph7/3mDGWxBibI2xfxxjLC+mEnPOY/AOQKXt9P4C3NNq0AbBX+N9aeN26ufuu0c9xABKE1y8AeEGnXSmAts3d31CvBe6Ku3sAdAeQCGAzgPzm7ruqj30BnAlgBYACH+3M8Jn4vRYzfCZCP18EMFV4PdXHb6W2ufsazHsM4F5RywDcCGBOKOeMWQucc14tW0wDoDXaegmApZzzE5zzkwCWArg0Gv0LBM75Es65Q1hcC6BTc/YnFAxey7kAdnPO93LOmwD8D8DEaPXRCJzzHZzzkubuRzgweC0x/5kITATwkfD6IwBXNWNfAsXIeyy/vrkAxjDGtCtRGyBmBRwAGGMzGWMHAdwC4EmNJh0BHJQtlwnrYpk/AvheZxsHsIQxtpExdlcU+xQsetdixs9FD7N9JnqY5TPJ5ZwfEV6XA8jVaZfMGCtkjK1ljMWKyBt5j6U2giFUBSA72BM2a1FjxtgPAM7Q2PQE5/wbzvkTAJ5gjP0NwH0AnopqBwPA37UIbZ4A4AAwW+cwIzjnhxhj7QAsZYzt5JyvjEyP9QnTtTQ7Rq7DAKb5TMyCr2uRL3DOOWNML865q/C5dAfwI2OsmHO+J9x9jXWaVcA552MNNp0NYCG8BfwQgFGy5U5w+wGjjr9rYYzdBuByAGO44ADTOMYh4X8FY+wruB/Joi4WYbiWQwA6y5Y7CeuiSgDfL1/HMMVnYoCY+EwA39fCGDvKGGvPOT/CGGsPoELnGOLnspcxtgLAWXD7n5sTI++x2KaMMZYAoBWA48GeMGZdKIyxXrLFiQB2ajRbDGAcY6y1MFo9TlgXUzDGLgXwGIArOed1Om3SGGMZ4mu4r2Vr9HppDCPXAmADgF6MsW6MsUS4B2tiJlLAKGb5TAxils/kWwBiNNlkAF5PF8LvPUl43RbAcADbo9ZDfYy8x/LrmwTgRz2DzhDNPXLrY0T3S7h/LFsAfAego7C+AMB7snZ/BLBb+PtDc/db51p2w+332iT8iaPQHQAsFF53h3vUejOAbXA/Gjd734O5FmF5PIDf4LaKYu5aAFwNt4+yEcBRAItN/Jn4vRYzfCZCH7MBLAOwC8APANoI66XfPYBhAIqFz6UYwO3N3W9f7zGAp+E2eAAgGcAXwu9oPYDuoZyPptITBEGYlJh1oRAEQRC+IQEnCIIwKSTgBEEQJoUEnCAIwqSQgBMEQZgUEnCCIAiTQgJOEARhUv4f13tgyV/NtVwAAAAASUVORK5CYII=",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.plot(lr_i, perdidas_i);"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WfGI0APWPD1F"
},
"source": [
"En el gráfico podemos apreciar que nuestra tasa de aprendizaje óptima —es decir, la que mejor minimiza la pérdida— se encuentra alrededor de $10^{-1}$ (es decir, $0.1$), de manera que podemos utilizar esta tasa con mayor confianza para entrenar todos nuestros parámetros con los datos de `Xtr`."
]
},
{
"cell_type": "code",
"execution_count": 411,
"metadata": {
"id": "qUhPiHU9SIDF"
},
"outputs": [],
"source": [
"paso_i = []\n",
"perdidas_i = []\n",
"\n",
"for i in range(1000):\n",
" #minibatch («minilote»)\n",
" ix = torch.randint(0, Xtr.shape[0], (32,))\n",
"\n",
" # propagación hacia delante\n",
" emb = C[Xtr[ix]]\n",
" a = torch.tanh(emb.view(-1, 6) @ H + d)\n",
" logits = a @ U + b\n",
" perdida = F.cross_entropy(logits, Ytr[ix])\n",
" \n",
" # propagación hacia atrás\n",
" for p in parametros:\n",
" p.grad = None\n",
" perdida.backward()\n",
"\n",
" # actualización\n",
" lr = 0.1 # nueva tasa\n",
" for p in parametros:\n",
" p.data += -lr * p.grad\n",
"\n",
" # registrar estadísticas\n",
" paso_i.append(i)\n",
" perdidas_i.append(perdida.item())"
]
},
{
"cell_type": "code",
"execution_count": 413,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 265
},
"id": "GW2pHmdef7qi",
"outputId": "cb4b2fdc-dc7a-4c69-dfb6-fc6c99cba399"
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3wUZf4H8M83m0ZCgAABgpSAFEXpERAQsCHtbOcpnPXA42x3Z/t5oJ5dDz09ezkUVCxYsZwgRUABRUIAgUAoAQIEQhIIJJC+yfP7Y2Y3W2azm91NNsx+3q9XXmTKzj6zEz77zDPPPCNKKRARkXlFhLoARETUsBj0REQmx6AnIjI5Bj0Rkckx6ImITC4y1AUw0rZtW5WSkhLqYhARnTY2bNhwVCmVZLSsSQZ9SkoK0tPTQ10MIqLThojs97SMTTdERCbHoCciMjkGPRGRyTHoiYhMjkFPRGRyDHoiIpNj0BMRmZypgv6V5bvx066CUBeDiKhJMVXQz161F6sY9ERETkwV9M2iLSitrA51MYiImhRTBX18tAWlldZQF4OIqEkxVdA3i45ESQVr9EREjkwV9PHRFpRVsUZPROTIVEHfLNrCGj0RkQtTBX18dCTb6ImIXHgdj15E5gKYBCBfKXWuPu9TAL31VVoBOKGUGmDw2mwAJwFUA7AqpVKDVG5DcTHsdUNE5MqXB4+8B+A1APNsM5RS19l+F5EXABTV8foLlVJH/S1gfcSxeyURkRuvQa+UWiUiKUbLREQAXAvgouAWyz/x0ZEoqWDTDRGRo0Db6C8AkKeU2u1huQKwVEQ2iMj0AN/Lq2bRFlRYa1Bdoxr6rYiIThuBPjN2CoD5dSwfqZQ6JCLtACwTkR1KqVVGK+pfBNMBoEuXLn4VJj5a253SSisSYqP82gYRkdn4XaMXkUgAVwP41NM6SqlD+r/5AL4CMKSOdWcrpVKVUqlJSYYPMvcqLsYCAChjOz0RkV0gTTeXANihlMoxWigi8SKSYPsdwFgAGQG8n1dx0VrQlzDoiYjsvAa9iMwHsBZAbxHJEZFp+qLJcGm2EZGOIrJIn2wPYI2IbAaQBmChUmpx8IruLsqi7U5VdU1Dvg0R0WnFl143UzzMv8Vg3mEAE/Tf9wLoH2D56iUygkFPROTKVHfGRlkEAGCtZq8bIiIbUwV9pN50Y61hjZ6IyMZUQR8VodXoK62s0RMR2Zgq6FmjJyJyZ7KgZxs9EZErUwV9FHvdEBG5MVXQ22v0HOuGiMjOVEFv617JGj0RUS1TBb3thim20RMR1TJX0NubblijJyKyMVXQ1451wxo9EZGNKYPeyjZ6IiI7UwU9e90QEbkzVdA3i7LAEiE4XloZ6qIQETUZpgr6KEsEuraOw578klAXhYioyTBV0ANAp9ZxOFxUFupiEBE1GaYL+sS4KBSVVYW6GERETYbpgr5VsyicKGXQExHZmC7oW8ZFo7i8CtXseUNEBMCEQd8iNhJKAacqrKEuChFRk2C6oI+O5E1TRESOvAa9iMwVkXwRyXCY95iIHBKR3/SfCR5eO05EdopIlojMCGbBPbHojxNk0w0RkcaXGv17AMYZzH9RKTVA/1nkulBELABeBzAeQB8AU0SkTyCF9UVkBO+OJSJy5DXolVKrABT6se0hALKUUnuVUpUAPgFwhR/bqReLPlQxa/RERJpA2ujvEpEtetNOosHyMwAcdJjO0ecZEpHpIpIuIukFBQV+F4o1eiIiZ/4G/ZsAzgQwAEAugBcCLYhSarZSKlUplZqUlOT3dmrb6HkxlogI8DPolVJ5SqlqpVQNgLehNdO4OgSgs8N0J31eg2KNnojImV9BLyLJDpNXAcgwWG09gJ4i0k1EogFMBvCtP+9XH7YaPR8nSESkifS2gojMBzAGQFsRyQHwKIAxIjIAgAKQDeAv+rodAbyjlJqglLKKyF0AlgCwAJirlNrWIHvhwDYmPS/GEhFpvAa9UmqKwew5HtY9DGCCw/QiAG5dLxuSrdcNm26IiDSmuzM2kjdMERE5MV3Q29vo2euGiAiACYOeNXoiImemC3oLu1cSETkxXdBH2oZAYPdKIiIAJgx6W42+uJxPmSIiAkwY9LZ+9Pd+tjnEJSEiahpMF/QRIqEuAhFRk2K6oK+wVoe6CERETYrpgr5PcgsAwIDOrUJcEiKipsF0QS8i6N+pJVo2iwp1UYiImgTTBT0AREQIahS7VxIRAWYNemHQExHZmDLoLSKoqQEWZxzBJ2kHQl0cIqKQ8jpM8elIBKhWCrd9uAEAMHlIlxCXiIgodMxZo48QKDbdEBEBMGnQR4hw9EoiIp05gz5CwJwnItKYM+gF7HVDRKQzZdBb2L2SiMjOlEEvIqjmkwSJiAD4EPQiMldE8kUkw2Hev0Vkh4hsEZGvRMRwYBkRyRaRrSLym4ikB7PgdbFEgL1uiIh0vtTo3wMwzmXeMgDnKqX6AdgFYGYdr79QKTVAKZXqXxHrj71uiIhqeQ16pdQqAIUu85Yqpaz65K8AOjVA2fzGsW6IiGoFo41+KoDvPSxTAJaKyAYRmV7XRkRkuoiki0h6QUFBQAXSxroJaBNERKYRUNCLyEMArAA+8rDKSKXUIADjAdwpIqM8bUspNVsplaqUSk1KSgqkWLAIUFJh9b4iEVEY8DvoReQWAJMAXK88XPlUSh3S/80H8BWAIf6+X31EiCD/ZEVjvBURUZPnV9CLyDgADwC4XClV6mGdeBFJsP0OYCyADKN1gy0igs+NJSKy8aV75XwAawH0FpEcEZkG4DUACQCW6V0n39LX7Sgii/SXtgewRkQ2A0gDsFAptbhB9sIFc56IqJbXYYqVUlMMZs/xsO5hABP03/cC6B9Q6fxkYdITEdmZ9s5YIiLSmDLoLQx6IiI7UwY9W26IiGqZMujLqqpDXQQioibDlEH/WXpOqItARNRkmDLoXXEkSyIKZ2ER9BzJkojCWVgEPXOeiMKZKYP+41uHOk1zyGIiCmemDPrOreOcphn0RBTOTBn0URbn3WLTDRGFM1MGvetYN7wYS0ThzJRBH+kS9OxeSUThzJxBb3EOelboiSicmTPoI5x3i003RBTOzBn0FjbdEBHZmDLoXYcpZoWeiMKZKYPe9Zmx1azRE1EYM2XQu6phlZ6IwlhYBD0r9EQUzsIi6Nl0Q0ThzKegF5G5IpIvIhkO81qLyDIR2a3/m+jhtTfr6+wWkZuDVfD64Fg3RBTOfK3RvwdgnMu8GQCWK6V6AliuTzsRkdYAHgUwFMAQAI96+kIItrduGIy/XdQDANvoiSi8+RT0SqlVAApdZl8B4H399/cBXGnw0ssALFNKFSqljgNYBvcvjAYx7twO6N2hBQB2rySi8BZIG317pVSu/vsRAO0N1jkDwEGH6Rx9nhsRmS4i6SKSXlBQEECxatkGsWTTDRGFs6BcjFXaracBpalSarZSKlUplZqUlBSMYkH0G6c4BAIRhbNAgj5PRJIBQP8332CdQwA6O0x30uc1CtsdsqzQE1E4CyTovwVg60VzM4BvDNZZAmCsiCTqF2HH6vMaRQSbboiIfO5eOR/AWgC9RSRHRKYBmAXgUhHZDeASfRoikioi7wCAUqoQwJMA1us/T+jzGoW96YZBT0RhLNKXlZRSUzwsuthg3XQAtzpMzwUw16/SBai26YZBT0Thy9R3xkboQc9rsUQUzkwe9Nq/7HVDROHM3EGvJ31ZVXWIS0JEFDrmDnq96eZP764PcUmIiELH5EEf6hIQEYWeuYOeSU9EZPKgFwY9EZGpg97xIeEHC0tRUmENYWmIiELD1EHvWKG/4LmVmPL2r6ErDBFRiJg66F2bbrbkFIWoJEREoWPqoLfwYiwRkbmDnjlPRGTyoBf2uiEiMnfQs+mGiMjkQc+cJyIyfdAz6YmITB30Ro8QXLP7KLbknAhBaYiIQsOnJ0ydrozGob9hzjoAQPasiY1dHCKikDB5jT7UJSAiCj1TB70vz4pdtj0P//hiSyOUhogoNEwd9F3axHld58/z0vFp+sFGKA0RUWj4HfQi0ltEfnP4KRaRu13WGSMiRQ7rPBJ4kX0XE2nBR7cObcy3JCJqcvy+GKuU2glgAACIiAXAIQBfGay6Wik1yd/3CVQkO9MTUZgLVtPNxQD2KKX2B2l7QcOnTBFRuAtW0E8GMN/DsvNFZLOIfC8i53jagIhMF5F0EUkvKCgIUrGA8qrqoG2LiOh0FHDQi0g0gMsBfG6weCOArkqp/gBeBfC1p+0opWYrpVKVUqlJSUmBFsuupMI46A8Wlrq+f9Dek4ioKQlGjX48gI1KqTzXBUqpYqXUKf33RQCiRKRtEN7TZ6kpiYiOjMAtw1Oc5l/w3Eqnafa5JyKzCkbQT4GHZhsR6SD6WMEiMkR/v2NBeE+ftW0eg11Pjcew7m3qXM9ouAQiIjMIaAgEEYkHcCmAvzjMuw0AlFJvAbgGwO0iYgVQBmCyClEbSZSl7ouyDHoiMquAgl4pVQKgjcu8txx+fw3Aa4G8R7BEWeo+eampaaSCEBE1MlPfGevIa9ArhUv+8xNufT+9kUpERNQ4TD16paPoyLqbbqqVQlb+KWTln2qkEhERNQ7W6HU17HZDRCbFoNfd8+lvjVQSIqLGFUZBX3fTzcqdwbsbl4ioKQmjoA+bXSUichI26WcU9J669Fda2deSiMwjrIPe6JmyANDr4e8bujhERI0mjILevY2+mnfDElEYCJugNxqXnjlPROEgbIK+RWwUnrumn9M8T003RERmEjZBDwDXpnZ2mp69am+ISkJE1HjCKugBoH2LGPvvLy/fHcKSEBE1jrAL+gV3jMCw7q1DXQwiokYTdkF/RqtmGNung9f1+GhBIjKLsAt6wLeHjNQoYN7abKzY4faERCKi00rYDFPsyOrQ2yZCjJ8XW12j8Mg32wAA2bMmNlbRiIiCLixr9I7dKuOjjb/rWJMnIrMIy6C3VjsEfYxx0N/24cbGKg4RUYMKy6B3HPogLsbi8+u2Hy5GXnF5QxSJiKjBhGUbfbXDk8BjIn0P+gmvrEaURbD76QkNUSwiogYRcI1eRLJFZKuI/CYibk/WFs0rIpIlIltEZFCg7xkox4uxkQZj4NSlqlrhq005wS4SEVGDCVbTzYVKqQFKqVSDZeMB9NR/pgN4M0jv6TfH58NO6pdc79ff8+lmj8uUUvh0/QGcqrD6VTYiomBrjDb6KwDMU5pfAbQSkfqnaxA51ugHd00MavfJ9dnH8Y8vt+Lxb7cFbZtERIEIRtArAEtFZIOITDdYfgaAgw7TOfo8JyIyXUTSRSS9oKBhn99q616Z3DIWg7smBnXbJ8urAACFJZVB3S4Rkb+CEfQjlVKDoDXR3Ckio/zZiFJqtlIqVSmVmpSUFIRieZaaoo118+qUgRCpu42+vm34ti8Ro/HviYhCIeCgV0od0v/NB/AVgCEuqxwC4Dg+cCd9Xshc3r8j0h662B74dbHWKKTMWIjsoyU+bds2vILFyxcIEVFjCSjoRSReRBJsvwMYCyDDZbVvAdyk974ZBqBIKZUbyPsGQ7uE2Hqtv+VQkdO0p4eWVOs9Ny0Gjy4kIgqFQGv07QGsEZHNANIALFRKLRaR20TkNn2dRQD2AsgC8DaAOwJ8zwZxVoeEOpe7xnaltbYvfmFJpb0nj1Xvo88aPRE1FQEFvVJqr1Kqv/5zjlLqaX3+W0qpt/TflVLqTqXUmUqpvkopt772TcHiu0fh/rG9PC53rb/nn9TukF2ckYtBTy7DayuzADg03UQIlFJ4fslOHDpR1iBlJiLyRVgOgeDJ1JHdPC5zHZ/+4a+1FirbmDhLth0BUNt0EyGCXXmn8NrKLNz1McfNIaLQYdA7iIuOxII7hvu07r6jJbBW1zbf1Cjg0IkyexNOVsEpHDtVAQAor6ox3AYRUWNg0LsY1CURo3u5d+/8cqNzR6Gc42WYvbr24eKZucUYMWsF0rILAQCbD57AH99ZB4BPqyKi0GLQGzCK5VW73G/i2n+01G1e/skK9+0x54kohBj0BnytgSclxLjNM/pCOFbiHv5ERI2FQV+HQV1a1bnc1tPGm6OnKvHZ+oPeV6QmZcrsX/G6j8eYqClj0BuwVejvudS4u+XvB3Wq9zYf+HILPkvXwr68qhpfbMjxeNMVNQ1r9x7Dv5fsDHUxiALGoDeg9FZ6cbtNSvPIpD5+bfeBL7YAAD7fkIP7P9+Mt37a418BT0Pz0w4gK/9UqItBFJYY9AZsNXpPN7fGRPn3sSXoz6eNi9KeavXvJTvx3ZbDHtevrlE+1fwrrTVOXT2bopkLtuJ3r64JdTGIwhKD3oDt7lYBMHP8WW7Loyz+fWwt46IAOPfquevjTYbrjpi1Amc+uAj3f74ZH6/bX+d2ez38Pca+uMqvMjUG2xdVWVV1iEtCFJ4Y9AZuH9MDANCnYwtcP6yr23KLn0MQx+o1+Qqr98BzHDbheGmV1/X3+ji6ZihUNfGzjZoahdwiDlNB5sWgNzC6VxKyZ01Eq7hoRLmMQnnVQLdnpvgsK/8U8ovLMWfNPqf5v+w5ipQZC1Fg0AcfAD7fcBAf/Fp3rT5Qv+49hsmz16KqugaFJZXIKy43XK/CWo10/aYwX1U2YNDnF5djftoBw2UV1mrsPHLS6zbe/GkPzv/XCp+HoiY63TDovYiJtOCKAR3t063jowPa3pBnlmNvgRYo1wzuhJbNovDOai34Nx88Yfiag4Vl+Kc+ts57P+/D7jzv4eVJXnE51uw+6jb/nk9/w697C5FXXI5BTy7D0GeWG77+ye+245q31mJPgXZhdVfeSWw/XOzx/Q4cK3Ua6TPYbv9oI2Yu2Go4cNwjX2/DZS+tsg9A58nq3dq9D4dNOvhceVU1isu9nxWe7k5VWO1PeCNnDHofzHBopw9mM0RSQgyKyqqwYkc+AGDV7gKs2lWAxzw8bzY9uxCP/W87Zi7Yari8xuCibU2Ncmoquvy1NbhhzjqneZm5xcgtqjsMbbbkaOPy3/jOOhwsLMXYF1dhwiurDdc9fKIMo/69Es8syvRp2/44qo8nZPRlsn6/duZRXFb3f35b7yrHTy8Yw1YUlVW5XUgvqbAi57j7HdUN6eo3fkG/x5Y26nuGwoDHl6JvGOynPxj0Pkhu2Qz36n3qq6rdA2BC3w5eB0ObPqq727yBnZ1vyJq3dj9umpuG937JNtzGNW+tBQCk7z+OPQ6Dptnf44N0zPhyi1MN+4nvtqP3w4vtXwJ5xdprbpyTBkCrkY9/uTaoK7zUvm37f7ioHE98t93jej9sz0NmrlaOb37z3LMoUBF61yijoLc9E8Dbd7Otd5Vjtm/ycHbli+LyKpRUWNH/8aV4bvEOp2U3zlmHkc+udHtNblGZ/T4LVyt25Bnece2r7bmez7j8tfPISfR9dAmO+FhBaAxW3pfiEYPeR7a2+T+kut8sNbJHEnq2a17n6++5xP3mq/PPbON3eS5+4ScMfuoHp3D4ITMfn6w/iGv/uxbLM/MAwP6lcfnra5weWJ62T6vtuvbWKXfoGTPsmeV4x2HgNgBO3TiXbc+z/75wSy6UUli2PQ9FZVW4dV46pr2vPXrAsVbren3i8IkybNh/HOf/azmK9Jp3XnE5Xl+Z5VOt2nZdvLTS6rbMdtG8rrOwA8dK8cueY27zr37jF6/vbSQztxj9Hltq/9y//s15MLyNB7QvENcvpuvfWYcHvtiCkgr3/Zj6XjpumpvmV3kayvtrs3GyworlO/K8rutNhbUaG/YfD7xQ5BGD3kedW8che9ZEDOqS6LYs0iJeu1xGR7ovT4iNCrhctpuwHJ2qsGLa++lOTTwZh4rxjUvoGIWj45DKR4rL8dTCTKf/hJ5qTXd+vBH/WbYLf56Xjts/3OCxvJ+u1y6c/pKlXYAePmsFfv/mL8gtKsfGA9r7/P2TTfj3kp3YnluMxRm5SJmx0GPN0VajL6107slUU6OQfUy7FvLs4h0orbRi55GTeHbxDqcvkFverQ3Q6iA012Toj5z832btLCavuAL3fvqb23onSiudpnMKtesDt3+00f6F58nH6w54vP9i04Hj2HnkJLbmFLm9RzDZzhCD8SS1h7/KwO/f/MWpSWtxRq5TpcNfGYeKnLabMmMh/vKB92cffbzugNOZbkObu2YfznlkcYONdMugD4JBXRKdulw+deW5eOemVPt09qyJsEQIrh/aBVc6XNgFtGYfI/d5GH6hPlx7ozz+P+emlpvmuNcSyyrd/3P9/s3a2m1dteN1+lmC7WzBSLx+05hRs48tPEoqqvX3Uvg4TTtjyXRofiittNqnbZ+7Y9DnF5dj7s/77F9aq3cfxSvLs3DjnHV488c9mLNmn/0s40BhbQh4umhc10XaVbsKUOTQ/dXWtHXKoWa+YJP2BXvJf36yz7t1XrrTxXdbz6RVuwrw7s/OZz02zy/ZiZ+zjuLBr7Y63X+RmVts75V11Ru/4LKXVuF3r63Bdf/91eksLpghYvvCj4gQpO0rdLth78Nf99ubrb7edAj3f74ZKTMWYuyLP7mVw/Z3Y/vb23jgOG77cKPb32t9KKVQU6Mw6dU1bk1lS7bVnoVYq2tw09w0/Jx11D69NacID361FZm5xY02xPgT321HSWV1g/VQY9D7aWSPtrh1ZDdkz5qIHu2aI9Ih6G8Y1hWX9GmPW4an4OKz2tnnP31VX7w0eSDenzrEfoH3ygHu3TU7tIjFbWPObPB9SDc4XfZUm7xpbhpSZixEznHPoWerQdbVVtpcD3oxqAnaXmcL79W7Cuxt04//bxv+8NYvqK5RuOOjjRj/8mqUV1Xba/QlFVas23sMGYeKMOSZ5XhqofMF4MMnyuxDSD+1MBMfrM12K6unoB8+awX2GXS93HTgOG6am4YXf9gFQGuCsNVCjZpgHIeA2JJThD/PM65ZesqW11Zm4Xr9GQfaetqK419ejX9+neF27HbmncSgJ5fZp7vNXOTx+C7ddgQ/GVwHmJ92APuOluDphdsx6dXVyMo/iYe/3mr/ws84VIRr/7sWL/2w2+l1D3+dgTd+1Ib4uPvT3/DFhhwAwK68U1i2PQ+HT5Rhx5Fi5BwvtX9J226os5Ux53gpth0uwtJtRzBvbbb97M712pSRK9/4Bd0fXOR1vR8y87FqVwEe+ko7+31h2S787rXaO7i9XbMCgM/TDyJtXyE+/HU/svJP4e5PNjl9wTqan3bAfuZq5O/z3c/+giGyQbYaBj68dajTtFFwPXb5OYavHd0ryf5wE6Ohjju2ivX77ttA3enhsYe+XAwsLPHetS0+WvuTM7rn7C8fbEDHlrE4rDfTvLBsl31Z9rFSZB8rxZs/ZuHHnVpZcovK7Rcan1mUifyTFbhmsPGAc99udm7qMHpuwJ0fb8TYc8Ybfva2awnndGyBs5NbAIC9j/7xUu3h8AOfWGYPrZPl7kHvyqgMNjMXbEVJhRWT+iV7XKe43IqvN9U2x3m698HR1PfW44vbzoeIIONQEX7OOoqOrZrhr/NrzxC+//sFODu5BaqqazBzwVa0bBZlD987P9qEnXkn7Z+BrUfTayuzMKl/Mmav2ou7Luxh35ZRT7DpHzg37dm+/C9/7WdM7JeMhVtyAWhnYqt3uw+b8f7a/SirtOKOMT2Q6KG7s6euyo4Wbsm1/71nHytFr4e+d6tRl1ZW2290XLEjD/nFFZg8pIvTOv9n0HzaOj4Gj/zOeUws2+cJaF2rn/9Df7fXLdYfSRpsDPog8za0satzz2hp//2sDgm4cuAZTv32TyeFPoy7b2u6ifDQtnvYSy+O55fWhv+Fz/9o/90Wmp5uOnM1P+0AMgz6/x8pKkfn1nFu859amInM3GJ0bxuP+dOH4aN1B7AlRwuT/OIKLMrIdWo+cj2r8VSu11dmYbjLRfmXl9fWjl2/oBzN+HILvs+oDYZ7P/NeG9yw/zhW7MjHM4sysUe/n6OPHto2s77fgf+7rLe91u54FmCt0eaV6dd3jjnUXMe9pLVpL3B4GpsvtWrbNgHYQ74ur+ifz9ur9yF71kQAcKpBp8xY6PaajENFTjfPKaXcejkZNZv8uvcY0vYVYkzvJEx9TzsDcw16Iwru3WptZw0A8MWGHHyxIQdZT49HZCNU6sTfNigR6QxgHoD20Logz1ZKveyyzhgA3wCwNTouUEo94W3bqampKj3d+wWTpib7aAmSEmLsYearUxVW/LSzAMO6t0ab5rU1fKM/2NPdDcO64B/jzmqw/s59klsE1J3w6ztHYEDnVh4/+75ntMRW/YKrWSS3jPX5PgqbdgkxdZ6R1IclQvwesjsu2oJ1D16Mkc+u9HoR29F1qZ2xeneB14qFEduXi42nv5XuSfFYfu9ofL4hx7DTBADM//Mw9OvUEuc8ugSAdqa7918TDdf1RkQ2KKVSjZYFUqO3ArhPKbVRRBIAbBCRZUop1ysoq5VSkwJ4n9NGStt4v17XPCYSE+s4Re+eFG+/mxYANv3zUtz8bhq25BQhPtqCEoMLqBf2TsLKnf73vW4o1TXAqOfc+5EHS6B9xq98/WdE11HD8tT2ejqrb8gDdTc71Vcgz2UorazG91uP1CvkAeBTD/cs+GL8y6sx+8bB2J1/EoO7tva43t6CEny16VCdZylT3v4VZybV5oanM91A+R30SqlcALn67ydFJBPAGQD8v1ROTpbcPQotm0Xht4NaLwSbxPhofPznYTh2qgLv/pyN937JxpCU1kjLLsTEfsm4ZXgKendIsN8N2bNdc+zWLwR2bt0MBScrnLpR+qNTYrM6L8x6UlRW6dMgbaFUV88Ho6EWKLQe+NK4ttxQMnOLcYFeWbl6UN1jX9372Wav29vjUIlrqJu+gtI4JCIpAAYCWGew+HwR2Swi34uI8dVJbRvTRSRdRNILCppeTTQUendIQIeWsRh3brJTrx5AOwvo2iYeD044Gz/cOwqX9mkPABh3Tgecl9IaLWKjsOTuUejVvjle++Mg++t+uv9C3DK8m9t7JdSjuWnqiG74YNpQ7ysaWLS1fheb7ru0F5beMwpPX3WuX+93VocE3Da64XswNbRXpwwMdREC1rl1s1AXwaNrDW6E9IXj9YhgOKNVw3xGASt4N2kAAA2ESURBVAe9iDQH8CWAu5VSrufNGwF0VUr1B/AqgK89bUcpNVsplaqUSk1KSgq0WKaz+dGxiImMwC3DU5zmR0dGoEe7BEwd2Q3/vXGwUy+N3h0SsPSe0ejdIQF3XdgDX9x2PiIiBPde2guvuARHuxYx+MrLMA42j/yuD7oZNFP9cO8oLLtnVP13rg6Th3RBr/YJuH5oV7x03QAM6+75VNnI4rtH4Z5Le9qn62oi82b4mW2cuss6MroIP6Bzq3r9x518XmeM1b+wXfVqn4AXr3PvpeGrRX+7oN6vef4P/fHs7/vap/9310j77w9OcH9OgzdXDdTC9LbRZ+K61M71fn1Deu6a/vjh3tH4ZPow+zzXylVDm9g3GV/cfn6DbDugoBeRKGgh/5FSaoHrcqVUsVLqlP77IgBRItI2kPcMV/Exkdj51HiPXTYtEYLLzulg2M0TAO6/rDdSU7SQjI6MwOX9O2L2jYPx4bSh+GDaEHx461AM7JKIzY+Mxa0ju+GZq/ridYczAUALs4l9a4Pyg2lD7N3sACClTTx6tk/weZ/O6djCbd4TVzjvX0Js7ZnGlQPPwFs3DAYA+9hDjoweEgNoI5B219tBrzUImC/r+M81dUQ3+78f3ToUb9xQ+5k007vdAcBTV/bFjifHOb3WEiH2My1Hb1zv/LlGiHYR2NOxvbx/R6S0jcNVAzuhXyetl9ZZHRLQw8uwGzYtm0U5fY6AduZn885NqUhp49zTaM7NqbhmcCdcd15tD5O+nWp7iN060n3sJiPf3DkCLfT3vvTs9nj9j4Nw39heePaafk7r/dtl2ubN6wfhySu1s7nzUtzvSnc0tk97vH1TKv7vst4+lc1Vj3bNMbRba/td7DUuHVUuOdv4S97RXy+q7VpqNOyJkYFdWqFfp5Z4dcpAJLdsYjV60RJlDoBMpdR/PKzTQV8PIjJEfz/3gUUoJMae0wEje7bFBT2T7H9gLeOi8PCkPvjj0C5utd+XJw/E6w4hdUHPJHwyfZh9UDDHbmJndUjA+HM7ILllrOF7PzKpD/5310h8flttyO771wRc4XIDWaxDmAJAq7hoZM+aiL9drNXSHc9g/nxBd/zr6r4wkqT3ZooyqKX1aq+V1aarQ+hde14npD98CR6aeDZEBDGRFux4chxW3j8GmU+Os+9fQmykU1lfmTIQL/yhPwa4DFz35e3DMaFvMiafp33hnJeSiN1PT8CAzq0QG2VxenzlxL7JePeW8/DKlIGIidS23S5Be78bhnXFD/eOxue3nY/tT1yGv17UA49ffo7b+wHAyvvHuAX9C9fWnh1c0qe925eMY8at+ceFWDvzIqflES6fo1FbdfasiejfuRXe/dN5GNa9Nbq2jcPEfsn2+xQW3117lvGH1M749q4R9uk5N6diwR3DMb5vMm4Y2gVPXXku/nW1+5fB0ntG2bsjx0RZcGmf9m43OS2/bzSGpPh2Jigi2PjPS/HPSX2w4r4xTstenjwQaQ9ejBE9nLvDzps6xP77+HOT7Xe1p6YkYtpI92ZSAE5NkQtuH45v7xrp9pkGUyC9bkYAuBHAVhGxdeB9EEAXAFBKvQXgGgC3i4gVQBmAyaqx7immoLDdwNQ8JtLw5q6WzaKw48lxTsMAZDx+GaIsWijW1Cjc/8VmDElpjSXbjmDlzgJ899eR9vsHBjuMHSQiiIuuDcvl942us2wZj1+G2MgI3DisK5Ztz0NEhGDKkC4oLqvC3J/34Zs7a5sa/ji0C9btKzQ844iLjsSbNwy2d5ObOqIbHv12Gzb+81LD5w/ERlnsTVcPjOuN+z7bbP9sFv3tAr05Tattd20Th8T4aOw6chJPL8pEp0TtC/Xm4Sn4ZP1BnNWhhdPwGTPHn22/Rf+hiWejo0vTz6R+yfghM8/ehHWeHmD3jdVqsdcP7YLSqmpc9PyPOHqqEivuG+22D4O6tHLrAjy6VxJenTIQ2w4X462f9qBdi9pj3SnR/b4CQAvygpMVWJ9diKSEGCzYeAg92zVHQmykffA2ABjctTU+me5+1nRWB+2M7rJztLOefp1aYf1Dl6C00oqubWqbBkUENwzravhcgV7tE5AYp+1frF4TT9Qf2dkqLgonSqvQsWUzvH1zKvo/7t6ld3SvJLezruYxkW4BfUHPtoiLtiA+JhLzpg7FmQ73B7RvEYuLz2qH5TvykRgfhTsu7IEBXVphRI+2GNGjLW4ZnoILnluJZlEW+92/1w/tinm/7MfglESPZ+HB5Hc/+oZ0uvajN6OSCisqrTUe70Csr5oa5VZzSZmxENemdsJz12i1zC835EABHu9yDdTBwlJkHyvBjXPSMLFvsv0sZUvOCaTtK8S0kd1QVa0MB6Lzl1IKpyqsTgPZ/Zx1FKkpifbaus3J8iqs21uISwyafZRSKK+qQbNoi9syR6f04+YY8j/uzEeECM4/sw2iLBGYuWALuraJd7pYba2uweacIgzuatxMYvsydO1LfrCwFBc8txLtEmKw6oELUVZZ7dPfTHlVNaIsET49nvNEaSUGPFE7pEP3tvFYcf8YzFubjUe+2YZLzm6Hd24+D9bqGqzcWYAxvZNwvKQS7VpoZ0ETXl6N7bnF+M+1/bFur3YT1Pi+dV+z8bS/jn3nM58Yh4gI7TPo0c69IpF/shxDnl6OhJhIlFRaUaPctxcMDdWPnsJAfEwk4t0r8n4zOj3d9dR4pwtfv2+ggLfp3DrOPhqpo36dWqFfJ63pIzoyuLUsEXEbrXRED+PLVQmxUYYhb9uOt5AH9GEFXI7bmN7ObcxGTSGRlgiPIV+X9nqYTh3ZDbFRFrcmN098XQ/Q9qlHu+b4Xb+OuHJgR/sXydBuWlOK7Swx0hJhr6XbQh7QLi4vz8zDVQPPwNWDAvsbe/G6/vh1TyFmjD/LfjyMQh4AIiO0CkNEhGDVAxeGZAx/1uiJyGeearihll9cjpZxUW5nR4H6OesoSiurDS+q+6qmRuHOjzfiTyO6YUi3+vUaqw/W6IkoKObcnBrUx2kGi2PNPZg8nXXVR0SE4E29t1ioMOiJyGcXn+1/zZZCh+PRExGZHIOeiMjkGPRERCbHoCciMjkGPRGRyTHoiYhMjkFPRGRyDHoiIpNrkkMgiEgBgP1+vrwtgKNBLM7pgPscHrjP5hfI/nZVShk+talJBn0gRCTd03gPZsV9Dg/cZ/NrqP1l0w0Rkckx6ImITM6MQT871AUIAe5zeOA+m1+D7K/p2uiJiMiZGWv0RETkgEFPRGRypgl6ERknIjtFJEtEZoS6PMEiIp1FZKWIbBeRbSLyd31+axFZJiK79X8T9fkiIq/on8MWERkU2j3wn4hYRGSTiHynT3cTkXX6vn0qItH6/Bh9OktfnhLKcvtLRFqJyBciskNEMkXkfLMfZxG5R/+7zhCR+SISa7bjLCJzRSRfRDIc5tX7uIrIzfr6u0Xk5vqUwRRBLyIWAK8DGA+gD4ApItIntKUKGiuA+5RSfQAMA3Cnvm8zACxXSvUEsFyfBrTPoKf+Mx3Am41f5KD5O4BMh+lnAbyolOoB4DiAafr8aQCO6/Nf1Nc7Hb0MYLFS6iwA/aHtu2mPs4icAeBvAFKVUucCsACYDPMd5/cAjHOZV6/jKiKtATwKYCiAIQAetX05+EQpddr/ADgfwBKH6ZkAZoa6XA20r98AuBTATgDJ+rxkADv13/8LYIrD+vb1TqcfAJ30/wAXAfgOgEC7YzDS9ZgDWALgfP33SH09CfU+1HN/WwLY51puMx9nAGcAOAigtX7cvgNwmRmPM4AUABn+HlcAUwD812G+03refkxRo0ftH4xNjj7PVPRT1YEA1gFor5TK1RcdAWB7mKdZPouXADwAwPYk6jYATiilrPq0437Z91lfXqSvfzrpBqAAwLt6c9U7IhIPEx9npdQhAM8DOAAgF9px2wBzH2eb+h7XgI63WYLe9ESkOYAvAdytlCp2XKa0r3jT9JMVkUkA8pVSG0JdlkYUCWAQgDeVUgMBlKD2dB6AKY9zIoAroH3JdQQQD/cmDtNrjONqlqA/BKCzw3QnfZ4piEgUtJD/SCm1QJ+dJyLJ+vJkAPn6fDN8FiMAXC4i2QA+gdZ88zKAViISqa/juF/2fdaXtwRwrDELHAQ5AHKUUuv06S+gBb+Zj/MlAPYppQqUUlUAFkA79mY+zjb1Pa4BHW+zBP16AD31q/XR0C7ofBviMgWFiAiAOQAylVL/cVj0LQDblfebobXd2+bfpF+9HwagyOEU8bSglJqplOqklEqBdixXKKWuB7ASwDX6aq77bPssrtHXP61qvkqpIwAOikhvfdbFALbDxMcZWpPNMBGJ0//Obfts2uPsoL7HdQmAsSKSqJ8JjdXn+SbUFymCeLFjAoBdAPYAeCjU5Qnifo2Edlq3BcBv+s8EaG2TywHsBvADgNb6+gKtB9IeAFuh9WgI+X4EsP9jAHyn/94dQBqALACfA4jR58fq01n68u6hLref+zoAQLp+rL8GkGj24wzgcQA7AGQA+ABAjNmOM4D50K5BVEE7c5vmz3EFMFXf9ywAf6pPGTgEAhGRyZml6YaIiDxg0BMRmRyDnojI5Bj0REQmx6AnIjI5Bj0Rkckx6ImITO7/AceiQ46nCf+uAAAAAElFTkSuQmCC",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.plot(paso_i, perdidas_i);"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gTxEGBDQSwo7"
},
"source": [
"Debido a que entrenamos nuestros parámetros en lotes, la función de pérdida tiene algo de ruido cuando se acerca a su valor mínimo. Comprobemos ahora la pérdida que tenemos en `Xdev`:"
]
},
{
"cell_type": "code",
"execution_count": 414,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "SlcEcHzPSu9q",
"outputId": "97eac106-8da5-4a67-eb1f-84541b52068c"
},
"outputs": [
{
"data": {
"text/plain": [
"tensor(2.4327, grad_fn=<NllLossBackward0>)"
]
},
"execution_count": 414,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"emb = C[Xdev]\n",
"a = torch.tanh(emb.view(-1, 6) @ H + d)\n",
"logits = a @ U + b\n",
"perdida = F.cross_entropy(logits, Ydev)\n",
"perdida"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "o_NQDww9TZ1c"
},
"source": [
"Finalmente, podemos visualizar cómo luce nuestro *embedding* ahora que ha sido entrenado:"
]
},
{
"cell_type": "code",
"execution_count": 428,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 483
},
"id": "8EANrk6BTrlM",
"outputId": "7d03f486-7325-4402-cd02-6405698f38d6"
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAe4AAAHSCAYAAAAqryiAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3df3xU5Z33//c1v0KcAAGREAOKBvBHmyKKiG3VUGwVtq3YH1vctovFyppt763tdu91H3Dfe3vvl91297a322/b9MaWStu7G1tbLF8La4UatVpAqEBQCwF/NGCAqgRMDMnMmev7x0xCJplJ5lcmc2Zez8eDBzNzrnPmusiE95xzrvM5xlorAADgDp6x7gAAAEgdwQ0AgIsQ3AAAuAjBDQCAixDcAAC4CMENAICL+Ma6A8lMmTLFzpw5c6y7Mayuri4Fg8Gx7kbWGEdhYRyFhXEUnmIZS6Jx7N69+w1r7XnDrmitzfqPpJslHZB0SNI9CZaXSXootnyHpJkjbfOqq66yhe6JJ54Y6y7kBOMoLIyjsDCOwlMsY0k0Dkm77Aj5mPWhcmOMV9K3JS2RdLmk24wxlw9qdoekk9baWZL+t6SvZ/u+AACUolyc414g6ZC19mVrba+kJkm3DGpzi6QNsccPS1psjDE5eG8AAEpKLoK7RlLbgOdHYq8lbGOtDUs6JencHLw3AAAlxdgsa5UbYz4h6WZr7edjzz8r6Rpr7RcHtNkfa3Mk9vxwrM0bg7a1StIqSaqqqrqqqakpq76Nts7OTlVUVIx1N7LGOAoL4ygsjKPwFMtYEo1j0aJFu62184dbLxezyo9KmjHg+fTYa4naHDHG+CRNlPTm4A1Za9dJWidJ8+fPt/X19Tno3uhpbm5WofcxFYyjsDCOwsI4Ck+xjCXTceTiUPlzkmYbYy4yxgQkLZe0aVCbTZJWxB5/QtJvbLa7+gAAlKCs97ittWFjzBclPSbJK2m9tfYFY8z/VHRa+yZJ35f0I2PMIUlvKRruAAAgTTkpwGKt3Sxp86DX/vuAx2ckfTIX7wUAQCmj5CkAAC5CcAMA4CIENwCMsrAT0ekzITkR5uQiewV7kxEAcLOesKPNLe1qbD6s1hOd8nmMwhGrOVMrdFd9rZbWVavM5x3rbsKFCG4AyLE9bR26ff1OhZyIunodSVLIie5tHzjeqTUb9+veTS9qw8oFmjujciy7ChfiUDkA5NDetg7dtm67OrpD6up1NH1SuR67+/q4Nl29jjq6Q1q+brv2tnWMUU/hVgQ3AORIT9jRivU71R1yUmrfHYq27wmn1h6QCG4AyJnNLe0KOZGky2dMLtev/ub9es/0if2vhZyItrQcy0f3UCQIbgDIkcbmw/3ntAe7eEpQ3/3MVfrqz/Zq35FT/a939TpqbD6Ury6iCDA5DQBywIlYtZ7oTLhscjCgB/5yvv7qx7t1KEGbgyc65USsvB4z2t1EEWCPGwByoKs3LF+S4H37TEhHO7p19cxJCZf7PEZdveHR7B6KCMENADkQDPgUTlJgJeRY/dWPdutjV07XR+eeP2R5OGIVDHAAFKkhuAEgB7weo9lTK5Iu7w45uuPB53TH+y/SjZdNjVs2Z2oFh8mRMoIbAHKkob5WwUB8NbQjJ7t10/1PSZJOnwnrlm8/o60vnehfHgx41VA/K6/9hLsR3ACQI0vrquX3pvffqt/r0ZK6aaPUIxQjghsAcqTM59WGlQtU7k+tBnm5P9qemuVIB8GNgsedleAmc2dUqmnVQlWW+4ccNu8TDHhVWe5X06qF1CpH2pjGiILEnZXgZnNnVGrH6sXa0nJMjc2HdDDuMzxeDfW1WlI3jc8wMkJwo+BwZyUUgzKfV8vm1WjZvBo5Eauu3rCCAR+zx5E1DpWjoAy+s1Ii3FkJbuP1GE0Y5ye0kRMENwoGd1YCgJER3CgYg++s9Pc3X6LPLryw//ndN87WndddHLcOd1YCUGoIbhSMwXdWenRfuz78nur+539WV61H970etw53VgJQapichoKQ6M5KL7x+WudWlGnq+DKdWxHQqe6Q2k+dGbJu352VAKAUENwoCH13VuqbPd7nVy3tWlpXrfPGl+nRfe0J1+XOSgBKCYfKURCS3Vnp0b2v6yNzz9eSd0/Tr1oSBzd3VgJQSghuFIRkd1ZqPdGpYJlXx0/36E9v9yRclzsrASglBDcKRqI7K0nSzfc/rdse2J5wHe6sBKDUENwoGNxZCQBGRnCjYHBnJQAYGcGNgsKdlQBgeEzFRcHhzkoAkBzBjYLEnZUAIDEOlUOSdPpMaNSqj4WdSFbb585KAHAWe9wlqifsaHNLuxqbD+sjU0/pzn96PHYoukJ31ddqaV11VoeiB26/Ne5Qd262DwCliuAuQXvaOnT7+p0KOZHoTT2mqr/U6IHjnVqzcb/u3fSiNqxckNHkryHbV263DwCljEPlJWZvW4duW7ddHd0hdfU6mj6pXH/xZ4vi2nT1OuroDmn5uu3a29aR1fb73P7emdr6lRt0/6euyGr7AFDqCO4S0hN2tGL9TnWHnJEbS+oORdv3hFNrP9z2P7vwQn3mezt090N7Mt4+AIDgLimbW9oVciJDXvcYo/s/dYW2fuUGfefTV2qc/+zHIuREtKXlWFbbX7vs3Zox+Rw9uPJq3fH+i+KWpbN9AADBXVIamw/HHb7uM2nieP1o+2u68RtPqrMnrM8unNm/rKvXUWPzoay2v/qR/Trx9hndtm67vv/bV+KWpbN9AADBXTKciFXric6Ey97ueke7XzspSdr4/FFdPXNS3PKDJzpHvJRruO2PJJXtAwCiCO4S0dUbli/JddCDI3Pwc5/HqKs3nPH2R5LK9gEAUQR3iQgGfAon2audEDxHV14QvSzrlivO13OvvhW3PByxCgaGv3JwuO2PJJXtAwCiCO4S4fUYzZ5akXDZyVNv67PXRi/Xmlju14+3vxa3fM7UihGrlg23/ZGksn0AQBS7OSWkob5Wazbuj5tAduRkt3786G90X0vij0Iw4FVD/ayMt9/n/V9/IuvtAwDY4y4pS+uq5fem9yP3ez1aUjetILYPACC4S0qZz6sNKxeo3J9ajfByf7R9qjXFR3v7AACCu+TMnVGpplULVVnuVzCQODCDAa8qy/1qWrUw7Vrio719ACh1nOMuQXNnVGrH6sXa0nIsVvykQ35v3927xquhvlZL6qZlvCc8ePsH4+4Olv32AaCUEdwlqszn1bJ5NVo2r0bNzc3a/fH3KRjw5Wx298DtOxGrrt5wTrcPAKWKQ+WQJE0Y5x+1UPV6TNLth52ITp8JUTkNAFLEHjfyrifsaHNLuxqbD6s17jB6he6qr9XSumoOowNAEgQ38mpPW4duX79TISfSf713yInubR843qk1G/fr3k0vasPKBUxcA4AEOFSOvNnb1qHb1m1XR3coYZEWKXq3sI7ukJav2669bR157iEAFD6CG3nRE3a0Yv1OdYcSB/Zg3aFo+55wau0BoFQQ3EgqlxPHNre0K+RE0lon5ES0peVY1u8NAMWEc9yIM1oTxxqbD/cfHp8+qVwbPrdAz7ed1JUXTNK+I6f0s91t+vKNc3RuRZnubnpee4+cUlevo8bmQ1o2rybXwwQA1yqZPW4uOxrZnrYOXbN2m9Zs3K+DxztlbXTimLVnJ45ds3Zb2ueenYhV64nOuNcuPPccPfDUK1r8jSdVe16FbrmiRp/47u/0z5tf0hcWnb3pyMETnfzMAGCAot7j5rKj1PVNHBvuHHR0j9nR8nXb0ypX2tUbls9j+mePS1LbyW4dOP62JOngibf1zKE3JEl/OHZa0yeV97fzeYy6esOaMM6fwagAoPgU7R73aO09FiMrZTRx7I3OnpT2hoMBn8KD2vWGz57vttb2P7dW8nrOfizDEatgoKi/XwJAWooyuLnsKD2nukNpTxzr6A7pmrVbNWv1Zt30v5/UxuePJJ0B7vUYzZ5akVHf5kytoEwqAAxQdMHNZUfp+9PbPXFfcN4zfaK2fOk6lfk8Kvd79esvX685VUOD17GKO4Kx4P/ZqmcPvZFwL7yhvjbp3cKSCQa8aqifNXJDACghRXcMMtFlR19YNEsfv7JGb3b1qr2jWy1HT+uBp1/uX9532VEpzl52IlZnQo4GfhT2HTmlrS8d199+6BKN83v0yPNHdfB4Z9Jt3H3jbHX1OHrg6Zf1F9/bISNpTlX8PIKlddW6d9OLkhwdOdmtm+5/qn/9r/5sX//jgcv8Xo+W1E3L9ZABwNWKbo974GVHkvTumgn6yNxqLf3m0/rcD57Te6YPnVDVd9lRKerqDcuYoYeiv7mtVdfNnqL31EzUd588nNY2rYbOIyjzebVh5QKV+1Pb6y73R9szeRAA4hVVcCe67GjBzMl67IXjOhOKqLMnrK0vHU+4bqledhQM+GTt0HFXnhPQOQGvgmW+hOH5hUWz9Ju/vUE/u+taXTwlKEmaMM6nzyy8sL/N4HkEc2dUqmnVQlWW+5MeNg8GvKos96c1ax0ASklWwW2MmWyMedwY0xr7e1KSdv9pjOkwxjyazfuNpO+yo0z0XXZUarweo3EJ9oL/+dY63ffrg3pkz+u6Z8mlccuSHcWYUO7XZwcEd5+B8wjmzqjUjtWLtfbWOl1SVSFjJL/XyBjpkqrxWntrnXasXjxiaHNdPoBSle057nskbbPWfs0Yc0/s+d8naPdvks6R9FdZvt+wEl12tOOVt/S/PjlX33nikHweo8WXVeknO/44ZN1SvuzovPFlCgac/lMMH7uyRuFIRJv2vi6PkX7R8F5dW3uufnf4TUnxRzGkSP9RjL+/+VJdeO452vw379fTrW/oX7b8of89Bs4jKPN5tWxejZbNq5ETserqDSsY8I04e3yk6/LZPwdQCrJNqlsk1cceb5DUrATBba3dZoypH/x6rvVddjRwItULr5/Wo/vateVL1+nNrl7tO5L40q9SvuxoYrlffq+VFA3uX/z+qH7x+6OSpIiVln3n2ZS28/X//IPmVI3X0m/+dsiyZOVLvR6TUnGVVG4H+oXLejU5dkgeAIpVtue4q6y17bHHxyRVZbm9rCW67OjbTxzSB+57Up/87u/0yhtdQ9Yp9cuOjJTWxLEdr7ylD11epTKfR8GAV4svS+3Hnuk8glSvy3ciluvyARQ9k2hiUlwDY7ZKSnRNzmpJG6y1lQPanrTWJjvPXS/pq9baDw/zXqskrZKkqqqqq5qamkYcwGBW0kvtp5MGxIK6SxQKh/X8S2dnSns9RpdVT1C6+9udnZ2qqMissEgh6RtHd8jRK290yVopMsLnYv675ujSi2eo+0yP3n6nW396q0OH/vi6PlK/UD/51RMJ1zHG6PLq8fIkmMWezEg/z4GqyqXj3Zn/PAtFsX2u3I5xFJ5iGUuicSxatGi3tXb+cOuNeKjcWntjsmXGmOPGmGprbbsxplrSiVQ7nOS91klaJ0nz58+39fX1GW1ncluHlieru93SF9jRoZf7vRnPYG5ublamfSwkA8fRE3a0peWYGpsP6eCJTnlk5CQK8ZaXJb0c91LlOX7deINP97Uk/lgZIx1aXp/WKYmNzx/Rt3+zP+me9kB/WxfWfS0+BQNerb10tmuvyy/Gz5WbMY7CUyxjyXQc2R4q3yRpRezxCkm/zHJ7OcFlR5nrmzj22Jdv0KG1S7Vj9Qc0cVxqUyE63glp92sn9djd1+sfBs1ElzKbRzD4uvw+H7uyRlu+dJ22fOk6fePP58YtK+Xr8gEUv2wnp31N0k+NMXdIek3Sn0uSMWa+pLustZ+PPX9a0qWSKowxRyTdYa19LMv3HlbfZUcD9x7PzkIer4b6Wi2pm0aBj2F4PUZTKsbph3dck/wIxiBfatqT8PVM5hEkui5fkmZPrdAXF83Sxxuf1cl3QppYPnRyW9/59FKdcAigeGUV3NbaNyUtTvD6LkmfH/D8umzeJ1OZXnaEeH1HMFYMmtWdjkzKlya6HagkvXfWFG1uOaaT74QkRW+SMhi3AwVQrIqqctpw+i47IrQzM7hwSjoyLV+a6Lr8VJXydfkAilvJBDeyN/D89+F/Xqqf3HnNqM4jSHY70GcPvaGlddNUeU50bzrRofJSvi4fQHFjlwQZ8XqM3ls7ZdTnETTU12rNxvhZ5a0nOvXtJw7poVXXKmKtXnj9VNwdxkr9unwAxY3gRlZGex7BwNuBDvTz3x/Vz2PV3QbjdqAAihmHypEzozGPgNuBAkA8ghsFL9Xr8r0ew3X5AIoewQ1XSOV2oJdVTyC0ARQ9znHDNUY6n97c3DrGPQSA0Udww5VSvR0oABQbDpUDAOAiBDcAAC5CcAMA4CIENwAALkJwAwDgIgQ3AAAuQnADAOAiBDfyLuxEdPpMSE6G99oGgFJGARbkRU/Y0eaWdjU2H1Zr3O0/K3RXfa2W1lVzYxAASAHBjVG3p61Dt6/fqZAT6b+vdsiJ7m0fON6pNRv3695NL2rDygXUGgeAEXCoHKNqb1uHblu3XR3dIXX1Opo+qVyP3X19XJuuXkcd3SEtX7dde9s6xqinAOAOBDdGTU/Y0Yr1O9UdclJq3x2Ktu8Jp9YeAEoRwV1kCmni1+aWdoWcyJDXvR6jf/lYnX795ev1w5ULVOY7+zEMORFtaTmWz24CgKtwjrsIFOrEr8bmw/3ntAeaee45+pv/eF7/8IsWfesv5mnJu6v1yJ6jkqKHzRubD2nZvJp8dxcAXIHgdrlCnfjlRKxaT3QmXNZ2slsvtp+WJO0/ekrTJ5fHLT94olNOxPbfZxsAcBaHyl1s8MSvRMZq4ldXb1i+JMHbGz57+NyJaEg7n8eoqzc8qv0DALciuF2q0Cd+BQM+hTM8zx6OWAUDHAwCgEQIbpdKNvFrOPmc+OX1GM2eWpHRunOmVnCYHACSYLfGpQZP/Pr0NRfo09dcIEkaP86vIye7ddsD2+PWyffEr4b6Wq3ZuD+un0dOduum+5/qf/7A0y/HrRMMeNVQPysv/QMAN2KP24USTfz6vzv+qKXf/K0++q1n1H7qjL7325cTrts38SsfltZVy+9N7yPm93q0pG7aKPUIANyP4Hah4SZ+/eNH3qXfHX5D2146kXB5Pid+lfm82rBygcr9qV2KVu6PtqdmOQAkR3C7ULKJX5+4arpqJpXr/m2tSdfN98SvuTMq1bRqoSrL/QoGEgdyMOBVZblfTasWUqscAEbAOW4X6pv4dfD42cPl766ZoDuvu1if/D/Pyg5zJHwsJn7NnVGpHasXa0vLMTU2H9LBuCIx49VQX6slddPY0waAFBDcLjV44teKa2eq8hy/mu5cKEnad/SU7vl5S9w6Yznxq8zn1bJ5NVo2r0ZOxKqrN6xgwMfscQBIE8HtUkvrqnXvphclRYP77x7eN+I6hTLxy+sxmjDOP9bdAABX4hy3SzHxCwBKE8HtYkz8AoDSw6Fyl2PiFwCUFoK7CDDxCwBKB8FdZJj4BQDFjXPcAAC4CMGNURV2Ijp9JpS3+ugAUOw4VI6c6wk72tzSrsbmw2qNmyxXobvqa7W0rprJcgCQIYIbObWnrUO3r9+pkBPpr+oWcqJ72weOd2rNxv26d9OL2rByAZenAUAGOFSOnNnb1qHb1m1XR3co7h7cL9x7U//jrl5HHd0hLV+3XXvbOsaimwDgagQ3cqIn7GjF+p3qDjkjN5bUHYq27wmn1h4AEEVwIyc2t7Qr5ETSWifkRLSl5dgo9QgAihPBjZxobD4cd3g8FV29jhqbD41SjwCgOBHcyJoTsWo90TlywwQOnujkUjEASAPBjax19Ybly7C8qs9j1NUbznGPAKB4EdzIWjDgUzjDveZwxCoY4KpEAEgVwY2seT1Gs6dWZLTunKkVw94MhcprABCPXR3kREN9rdZs3J9wgtq7/vGxhOsEA1411M8a8jqV1wAgOfa4kRNL66rl96b3cfJ7PVpSNy3utT1tHbpm7Tat2bhfB493ytpo5TVrz1Zeu2btNoq3AChZBDdyoszn1YaVC1TuT21PuNwfbT9wzzlZ5bWBqLwGoNQR3MiZuTMq1bRqoSrL/QoGEgd4MOBVZblfTasWxtUqT7XymomdDqfyGoBSxTluZCzsRPROyFEw4OufYDZ3RqV2rF6sLS3H1Nh8SAfjzlGPV0N9rZbUTRtyjjpZ5bXpk8q1YeUC7WnrUF3NRH3uB8/paEe3pLOV15bNqxn9wQJAgSC4kZZUJ44tm1ejZfNq5ESsunrDceGeyHCV1y46N6iv/nSvnh90aLyv8hrBDaCUENxIWSa37PR6jCaM8w+73ZEqrx3t6B4S2n36Kq8N96UAAIoJ57iRktGcODZS5bV3hqmBTuU1AKWG4MaIRvuWnVRew2igeA+KFf/jYUSJJo79lw/M0rJ5NXqrq1ftHd1qOXpaDzz9cv/ydCaO9VVeO3g8/RuVjFR5DaWF4j0oBexxY0SDJ469Z/pELXn3NC3996d1+/qdqpteOWSddG/Z2VBfm/ASsiMnu3XT/U8lXCdZ5TWUJor3oFQQ3BhWoolj8y+cpMdfPK6ecHSS2raXjidcN51bduaq8hpKE8V7UEoIbgwrX7fszEXlNZSm0Z6DARQaghvDSjRxbNdrJ7X4siqV+Tw6J+DVBy6dmnDddCeOZVN5DaVr8ByMVddfrNvfO1OS9N8+fJl+cuc1kqRra8/V/Z+6QtLZORiAG2UV3MaYycaYx40xrbG/JyVoc4Ux5nfGmBeMMfuMMZ/K5j2RX4lu2bnvyCltfem4tnzpOj34uQU6cPxtvX0mNGTdTCaO9VVeW3trnS6pqpAxkt9rZIx0SdV4rb21TjtWLya00W/wHIznXnlLV8+cLEmqq6nUOQGffB6jBTMna+crb0lKfw4GUEiynVV+j6Rt1tqvGWPuiT3/+0Ft3pH0l9baVmPM+ZJ2G2Mes9ZyksklEt2yc91TL+v+ra0a5/fop391rVqOnopbJ5uJY5lUXkNpSjQHo+XoKdXVTFRFmU+94YheeP2U3jN9oq6eOVn/4/97ob8dxXvgVtkG9y2S6mOPN0hq1qDgttYeHPD4dWPMCUnnSSK4XWJpXbXu3fSipLPB/S8fq9PsqRUq83n1898f0Quvn45bJ1cTx1KpvIbSMrBGft8cjL4KflL0FE3byXf0iauma/cfT+oP7ae18OJzNXPKOTo0IOT75mDw+YLbZBvcVdba9tjjY5KqhmtsjFkgKSDpcJbvizzqmzi2fN32/glAX2rak7Q9E8eQa8muz559XlBhZ+iVC8+9+pbuvP5i/deH9+rAsbe15sOXa/+go0IU74FbGWuHv1zHGLNVUqJdp9WSNlhrKwe0PWmtHXKeO7asWtE98hXW2u1J2qyStEqSqqqqrmpqakplDGOms7NTFRUVIzcscKmOozvk6JU3umStFEnwufGY6Lnoi6YEU54dnkul9vModLkax0ifu0SmV03RRz9wrdb9dLPCjqPPfGSx9re+qj1/OLvPMM7vHTJ/IxF+HoWnWMaSaByLFi3aba2dP9x6Iwb3sCsbc0BSvbW2vS+YrbWXJGg3QdHQ/mdr7cOpbHv+/Pl2165dGfctH5qbm1VfXz/W3chaOuPoCTtp37IzX0rx51HIcjGOvW0dcUd6ciUY8GrtrXUpVfbj51F4imUsicZhjBkxuLM9TrRJ0gpJX4v9/cvBDYwxAUkbJf0w1dBG4WLiGPJluOuzf97wXn288dmMt03xHrhZttdxf03SB40xrZJujD2XMWa+MeZ7sTZ/Lul6SbcbY/bE/lyR5fuiAPRNHCO0MRoS1cjvk01oMwcDbpfVHre19k1JixO8vkvS52OPfyzpx9m8D4DSM/j67IFeuPcmvesfH0u4zBgp0RnAYMArv9cTd794wI2YUgmg4CS6PjtV1kpzqioG3R1s7OdgALlCcAMoOImuz06V32v0cMN7+6/zZg4Gig3BDaDgJKqRn6q+67Mp3oNixU1GABScRDXyU5VJjXzATQhu5FTYiej0mVDK9+EGkmmor016l7hkn65sauQDbsGhcmQtWTnKOVMrdFd9rZbWVTMhCGlLVCNfkirP8avjnd6E63B9NkoBe9zIyp62Dl2zdpvWbNyvg8c7Za0UcqyslQ4c79Sajft1zdpt2tvGPWWQnr4a+QPL504dX6ZfNLxXDzz9ypD2XJ+NUkFwI2N72zp027rt6ugOJb3etqvXUUd3SMvXbSe8kba5MyrVtGqhKsv9Cga8OvF2jz5w35Pa8Oyr/W2CAa8qy/1qWrWQ67NREghuZGS4cpSJdIei7XvCua05jeI3d0aldqxerLW31umSqgoZE73kyxjpkqrxWntrnXasXkxoo2RwjhsZSVSOstzv1bc/faWqJ46Txxj9v79p1aP72vuXh5yItrQcS+nGDsBA1MgHziK4kZFE5ShvuOQ8HT99RisffE6SNL4s/uPV1euosfkQwY2scH02Sh2HypG2ZOUoDxx7W9fNnqJ7br5UV8+cpLd7wkPaHDzRyaViAJAFghtp6ytHOdgrb3Tpz775W/3h2Nv66ocu0d8sHno9rc9j1NU7NNABAKnhUDnSlqwc5dTxZTrVHdIje47q9JmQll89Y0ibvnKU+RJ2Inon5HA+FEDRILiRtr5ylAePxx8uv3TaeP3D0stkrVXIsVrzyP4h6+ajHCUFYQAUM4IbGWmor9WajfvjJqg91fqGnvr3p5Ouk49ylHvaOnT7+p0KOZH+vvXdYaqvIMy9m17knswAXItz3MjI0rpq+b3pfXxGuxxld8ihIAyAokdwIyOJylEOZ7TLUfaEHb3yRhcFYQAUPYIbGRtcjjKRfJWj3NzSLjtgvtz0SeXa9pUbdP+nrtDWr9yg73z6So3zx3/c+wrCAICbENzISqGUo2xsPqyIjZ/pXju1Qj/a/ppu/MaT6uwJ67MLZ8Yt7ysIAwBuwuQ0ZG2sy1H2F4SZGv/60Y5u7X7tpCRp4/NH9bn3ztQDg+bO9RWE4VIxAG7BHjdyqq8cZT6DMFlBGDtoDzxRvTYKwgBwG4IbrpesIMz0Sefoyguih+hvueJ8PffqW0Pa5LsgDABki+CG6/UVhBns8IlOffbamdr6lRs0sdyvH29/bUibfJRhTj8AABv9SURBVBSEAYBcYlcDRaGhvlavv/T7uNfCEasvP7Qn6Tr5KAgDALnGHjeKwtK6apk0d5xHuyAMAIwGghtFoczn1UVTgv0FYY6c7NZN9z+VtP1oF4QBgNFCcKNolPu9BVMQBgBGC+e4UVT6CsJsaTmmxuZDOhh3d7Dxaqiv1ZK6aexpA3AtghtFZ6wLwgDAaCK4UdT6CsIAQLHgHDcAAC5CcAMA4CIENwAALkJwAwDgIgQ3AAAuQnADAOAiBDcAAC5CcAMA4CIENwAALkJwAwDgIgQ3AAAuQnADAOAiBDcAAC5CcAMA4CIENwAALkJwAwDgIgQ3AAAuQnADAOAiBDcAAC5CcAMA4CIENwAALkJwAwDgIgQ3AAAuQnADAOAiBDcAAC5CcAMA4CIENwAALkJwAwDgIgQ3AAAuQnADAOAiBDcKQtiJ6PSZkJyIHeuuAEBB8411B1C6esKONre0q7H5sFpPdMrnMQpHrOZMrdBd9bVaWletMp93rLsJAAWF4MaY2NPWodvX71TIiair15EkhZzo3vaB451as3G/7t30ojasXKC5MyrHsqsAUFCyOlRujJlsjHncGNMa+3tSgjYXGmN+b4zZY4x5wRhzVzbvCffb29ah29ZtV0d3qD+0B+vqddTRHdLyddu1t60jzz0EgMKV7TnueyRts9bOlrQt9nywdknXWmuvkHSNpHuMMedn+b5wqZ6woxXrd6o7lDiwB+sORdv3hFNrDwDFLtvgvkXShtjjDZKWDW5gre211vbEnpbl4D3hYptb2hVyImmtE3Ii2tJybJR6BADuku057iprbXvs8TFJVYkaGWNmSPqVpFmS/s5a+3qW7wuXamw+POTw+PRJ5Xrwcwv03Ktv6aoLJ+nYqTO684e71BOOBnxXr6PG5kNaNq9mLLoMAAXFWDv85TfGmK2SpiVYtFrSBmtt5YC2J621Q85zD1h+vqRHJH3EWns8wfJVklZJUlVV1VVNTU0pDWKsdHZ2qqKiYqy7kbV8jqPl6Kkhr40PlusvP3qjHvrPJ/XGydO6+f3z9cqRYzrw6pG4dnU1E4fdNj+PwsI4CkuxjEMqnrEkGseiRYt2W2vnD7feiHvc1tobky0zxhw3xlRba9uNMdWSToywrdeNMfslXSfp4QTL10laJ0nz58+39fX1I3VvTDU3N6vQ+5iKfI3j9JmQ7vynx/tnj/eZPsmnG97q1j889Y4kn7omn5bPW6FvtZz9ePq9Rrs//j5NGOdPun1+HoWFcRSWYhmHVDxjyXQc2Z5v3iRpRezxCkm/HNzAGDPdGFMeezxJ0vslHcjyfeFCwYBP4SQFVnrDZ897OxHJ5zFxy8MRq2CAqxcBINvg/pqkDxpjWiXdGHsuY8x8Y8z3Ym0uk7TDGLNX0pOS/pe1tiXL94ULeT1Gs6dmdnhrztQKeQeFOQCUoqx2Yay1b0panOD1XZI+H3v8uKT3ZPM+KB4N9bVas3F/0uu3EwkGvGqonzWKvQIA9+DSLOTV0rpq+b3xH7sjJ7t10/1P9T9/4OmXdf/W1v7nfq9HS+oSzY8EgNJDcCOvynxebVi5QOX+1GqQl/uj7alZDgBRBDfybu6MSjWtWqjKcr+CgcSBHAx4VVnuV9OqhdQqB4ABmKaLMTF3RqV2rF6sLS3H1Nh8SAfj7g42Xg31tVpSN409bQAYhODGmCnzebVsXo2WzauRE7Hq6g0rGPAxexwAhkFwoyB4PWbY4ioAgCjOcQMA4CIENwAALkJwAwDgIgQ3AAAuQnADAOAiBDcAAC5CcAMA4CIENwAALkJwAwDgIgQ3AAAuQnADAOAiBDcAAC5CcAMA4CIENwAALkJwAwDgIgQ3AAAuQnADAOAiBDcAAC5CcAMA4CIENwAALkJwAwDgIgQ3AAAuQnADAOAiBDcAAC5CcAMA4CIENwAALkJwAwDgIgQ3AAAuQnADAOAiBDcSCjsRnT4TkhOxY90VAMAAvrHuAApHT9jR5pZ2NTYfVuuJTvk8RuGI1ZypFbqrvlZL66pV5vOOdTcBoKQR3JAk7Wnr0O3rdyrkRNTV60iSQk50b/vA8U6t2bhf9256URtWLtDcGZVj2VUAKGkcKoe6Q45uW7ddHd2h/tAerKvXUUd3SMvXbdfeto489xAA0IfgLnE9YUevvNGl7lDiwB6sO+Roxfqd6gmn1h4AkFsEd4nb3NIuO2D+2Zc/OEcr3zez//lXP3SJPjfguSSFnIi2tBzLTwcBAHEI7hLX2HxYkQHJ/bNdbfrYldMlScZIH5lbrY3PH41bp6vXUWPzobz2EwAQxeS0EuZErFpPdEpTz7525GS3Tr7Tq3edP0FTKsr0wuun1fFOaMi6B090yolYeT0mjz0GABDcJayrNyxfguB96Lk2feKq6Tqvokw/3dWWcF2fx6irN6wJ4/yj3U0AwAAcKs8RNxYsCQZ8Cifo72MvHNP1c87Te6ZX6qmDf0q4bjhiFQzwvQ8A8o3/ebNgJW18/ohrC5Z4PUazp1ZIir+8K+RYbT/8pk6fCSnZ95A5Uys4TA4AY4A97gztaevQS+2ntWbjfh083ilro4Fn7dmCJdes3Vbw1zw31NfKY+ID2Bhp3gWVeui5xIfJgwGvGupn5aN7AIBBCO4M7G3r0G3rtsuJWNcXLFlaV62BuT1raoWe/OoiPXP4Tb365jsJ1/F7PVpSNy2r93XjqQUAKAQcKk9TTzhagCTdgiU7Vi8uyMPmZT6vLpoSVLk/pO6Qo0MnOnX9vz2RtH2536sNKxdkNBZqoQNA9tjjTtPmlnaFnEjca3e8/yI9dvf1euzu6+OKl/Qp9IIl5X6vmlYtVGW5X8FA4uAMBryqLPeradXCjGqV72nr0DVrt7n+1AIAjDWCO02NzYfjDo+/u2aCPjl/upZ9+xnd+p1ntHzBBXrX+RPi1nFDwZK5Myq1Y/Virb21TpdUVcgYye81Mka6pGq81t5apx2rF2cU2n2nFqiFDgDZ41B5GvoLlgxw9czJeuyF4/2Hzv9z/zFdPXOyXnj9dFw7NxQsKfN5tWxejZbNq4mdvw8rGPBl1ediO7UAAGONPe40JCtYkoq+giVu4fUYTRjnz/qLRqJTCyMp9FMLADCWCO40JCpYsvOVt/Shy6s0zu9Rud+rm941Tc+9+taQdUu1YMngUwuSNH1SuR67+/r+53ded7HuvnF2/3M3nFoAgLFSekmShb6CJQePnz1c/sLrp/Xw7iP65RfeL0l66Lk/DjlMLpVmwZJEpxZS5YZTCwAwFgjuNDXU12rNxv1xe5Hf/+0r+v5vX0m6TqkWLOk7tRBy0r9Wm1roAJAYh8rTtLSuWn5vev9suShY4kbJaqGHHauBO9Jl/qH/nqV6agEARkJwp6nMFy1AUu5PbcZzNgVL3O5sLfR4b3T26NyKMlWe41fA69HiS6cOaVOKpxYAIBUEdwbmzqhU06qF8nrMqBUsKRYN9bVD/o3CEatvbmvVL7/wPv3o8wt0+E/x58FL9dQCAKSCY5EZmjujUm9VT9DaS2ersfmQDsaV8ByvhvpaLambVpJ72gMtravWvZtelBQ/s/zBZ1/Vg8++mnCdUj21AACpILizYKScFywpNn2nFpav255SEZZSPrUAAKngUHmO5KpgSTHqO7UwmrXQAaBUsMeNvOirhb6l5RinFgAgCwQ38mY0aqEDQKkhuDEm+k4tAADSk9U5bmPMZGPM48aY1tjfk4ZpO8EYc8QY861s3hMAgFKW7eS0eyRts9bOlrQt9jyZf5L0VJbvBwBAScs2uG+RtCH2eIOkZYkaGWOuklQl6ddZvh8AACUt2+Custa2xx4fUzSc4xhjPJLuk/TVLN+rYIWdiE6fCclJUJcbAIBcMtYOHzbGmK2SEpWxWi1pg7W2ckDbk9bauPPcxpgvSjrHWvuvxpjbJc231n4xyXutkrRKkqqqqq5qampKZyx5ZSWdOv22/nTG6EzIkTFG1lqN83t13vgyTSz3yy1zpTs7O1VRMbSmuNswjsLCOApLsYxDKp6xJBrHokWLdltr5w+33ojBPezKxhyQVG+tbTfGVEtqttZeMqjN/5V0naSIpApJAUnfsdYOdz5c8+fPt7t27cq4b6NpT1uHbl+/U3fOPqN/2zf0uuNgwCu/16MNKxe4ophIc3Oz6uvrx7obWWMchYVxFJZiGYdUPGNJNA5jzIjBne2h8k2SVsQer5D0y8ENrLWfttZeYK2dqejh8h+OFNqFbG9bh25bt10d3SFFknzp6ep11NEd0vJ127W3rSPPPQQAFLNsg/trkj5ojGmVdGPsuYwx840x38u2c4WmJ+xoxfqdKdXclqTuULR9Tzi19gAAjCSr4LbWvmmtXWytnW2tvdFa+1bs9V3W2s8naP9gsvPbbrC5pV0hJ5LWOiEnoi0tx0apRwCAUkPltDQ0Nh9WV+/Qved1n71K1RPLVeb36AfPvKL/2NnWv6yr11Fj8yEtm1eTz64CAIoUwZ0iJ2LVeqIz4bK/e3ifTnWHVObzaNMX368t+4+p451Q//KDJzrlRCw1uQEAWSO4U9TVG5bPYxRyhk5I+9z7Zuqmd0WvmKuuHKeLzg3q+XfOTkrzeYy6esPU5gYAZI3gTlEw4FM4QYGVhRdP1vtmTdGt33lGZ0IRNa1aqDJ//NSBcMQqGOCfGgCQvWxnlZcMr8do9tShF/yPH+fXqe6QzoQiqj0vqHkJrtueM7WCw+QAgJwguNPQUF+rYCC+4MqTB/4kn8do61du0N/ffKmeH3TddjDgVUP9rHx2EwBQxDh+m4alddW6d9OLks7OLO91Irr9B88lXcfv9WhJXaKKsQAApI897jSU+bzasHKByv1Dy5wmUu6Pti/zpdYeAICRENxpmjujUk2rFqqy3C+PSXzeOhjwqrLcr6ZVC11RqxwA4B4cKs/A3BmV2rF6sbY8/htdUhXQwROd8nmMwhGrOVPHq6G+VkvqprGnDQDIOYI7Q2W+6F71Y1++QU7Eqqs3rGDAx+xxAMCoIrhzwOsxFFcBAOQF57gBAHARghsAABchuAEAcBGCGwAAFyG4AQBwEYIbAAAXIbgBAHARghsAABchuAEAcBGCGwAAFyG4AQBwEYIbAAAXIbgBAHARghsAABchuAEAcBGCGwAAFyG4AQBwEYIbAAAXIbgBAHARghsAABchuAEAcBGCGwAAFyG4AQBwEYIbAAAXIbgBAHARghsAABchuAEAcBGCGwAAFyG4AQBwEYIbAAAXIbgBAHARghsAABchuAEAcBGCGwAAFyG4AQBwEYIbAAAXIbgBAHARghsAABchuAEAcBGCGwAAFyG4AQBwEYIbAAAXIbgBAHARghsAABchuAEAcBGCGwAAFyG4AQBwEYIbAAAXIbgBAHARghsAABchuAEAcBGCGwAAFyG4AQBwkayC2xgz2RjzuDGmNfb3pCTtHGPMntifTdm8JwAApSzbPe57JG2z1s6WtC32PJFua+0VsT8fzfI9AQAoWdkG9y2SNsQeb5C0LMvtAQCAYWQb3FXW2vbY42OSqpK0G2eM2WWM2W6MIdwBAMiQsdYO38CYrZKmJVi0WtIGa23lgLYnrbVDznMbY2qstUeNMRdL+o2kxdbawwnarZK0SpKqqqquampqSmsw+dbZ2amKioqx7kbWGEdhYRyFhXEUnmIZS6JxLFq0aLe1dv6wK1prM/4j6YCk6tjjakkHUljnQUmfGKndVVddZQvdE088MdZdyAnGUVgYR2FhHIWnWMaSaBySdtkR8jHbQ+WbJK2IPV4h6ZeDGxhjJhljymKPp0h6n6QXs3xfAABKUrbB/TVJHzTGtEq6MfZcxpj5xpjvxdpcJmmXMWavpCckfc1aS3ADAJABXzYrW2vflLQ4weu7JH0+9vhZSXXZvA8AAIiichoAAC5CcAMA4CIENwAALkJwAwDgIgQ3AAAuQnADAOAiBDcAAC5CcAMA4CIENwAALkJwAwDgIgQ3AAAuQnADAOAiBDcAAC5CcAMA4CIEN5BHYSei02dCciJ2rLsCwKWyuh83gJH1hB1tbmlXY/NhtZ7olM9jFI5YzZlaobvqa7W0rlplPu9YdxNAmsJORO+EHAUDPnk9Jm/vS3ADo2hPW4duX79TISeirl5HkhRyonvbB453as3G/bp304vasHKB5s6oHMuuAkhBIXwR51A5MEr2tnXotnXb1dEd6g/twbp6HXV0h7R83XbtbevIcw8BpGNPW4euWbtNazbu18HjnbI2+kXc2rNfxK9Zu23Uf5cJbmAU9IQdrVi/U92hxIE9WHco2r4nnFp7APlVSF/ECW5gFGxuaVfIiaS1TsiJaEvLsVHqEYBMJfsi/vOG9yZsP9pfxAluYBQ0Nh9O+q08ma5eR43Nh0apRwAyleyL+Mcbn026zmh+ESe4gRxzIlatJzozWvdghusBGD3Jvoi/cO9NSdcZzS/iBDeQY129YfkyvDTE5zGKWK7xBgpFtl/ER6NmA8EN5Fgw4FM4w1/WcMTKY/J3PSiA4WX7RbyrN5zjHhHcQM55PUazp1YkXPaD26/W1PFlSdedk2Q9AGMj2y/iwUDuy6UQ3MAoaKivVTAwtAjD5x58Tife7km4TjDgVUP9rNHuGoA0DPdFfCRzplaMSkU1ghsYBUvrquX3pvfr5fd6tKRu2ij1CECmkn0RH85ofhEnuIFRUObzasPKBSr3p/bLXu6PtqdmOVB4kn0Rf9c/PpZ0ndH8Ik5wIynuZJWduTMq1bRqoSrL/Um/rQcDXlWW+9W0aiG1yoECVWhfxLnJCOIUQgH9YjJ3RqV2rF6sLS3H1Nh8SAfj/k3Hq6G+VkvqpvFvChS4vi/iKwbdNGigYMArv9cz6jcNIrjRjztZjY4yn1fL5tVo2bwaORGrrt5w3m8DCCB7hfJFnOCGpLMF9Ie7KUY0zB0tX7edQ7sZ8nqMJozzj3U3AGSoEL6Ic44bshJ3sgKANPV9Ec/30TP2uKFT3aGEBfSXXVGj2983UwGv0Z62Dq15ZL/65qn1FdBfNq8mz70FgNLGHjf0p7d7hky0qD2vQh+eW61PND6rpd/8rZyI4kKaO1kBwNhgj7vEORGrMyFHgz8K75t1rupqJmrTF98nSSrze/VmV3zFr74C+kyyAoD8IbhLXFdvWCbBTS2MMfr57iP618cOJF23r4A+k60AIH84VF7iggGfbILbSD5z6A0tqavWucGAJGliuV81leVxbUargD4AIDn+1y1xXo/RuATVgA6d6NR9vz6gH92xQMYYhR2r//7L/Tra0d3fZrQK6AMAkiO4ofPGlykYcIZMUHt0X7se3deecB3uZAUAY4ND5dDEcj93sgIAlyC4ISMVVAF9AEByBDckcScrAHALznGjX6EU0AcAJEdwI04hFNAHACRHcCMp7mQFAIWHc9wAALgIwQ0AgIsQ3AAAuAjBDQCAixDcAAC4CMENAICLENwAALgIwQ0AgIsQ3AAAuAjBDQCAixhr7Vj3ISFjzJ8kvTbW/RjBFElvjHUncoBxFBbGUVgYR+EplrEkGseF1trzhlupYIPbDYwxu6y188e6H9liHIWFcRQWxlF4imUsmY6DQ+UAALgIwQ0AgIsQ3NlZN9YdyBHGUVgYR2FhHIWnWMaS0Tg4xw0AgIuwxw0AgIsQ3Gkwxkw2xjxujGmN/T0pSbsLjDG/Nsa8ZIx50RgzM789HV6q44i1nWCMOWKM+VY++5iKVMZhjLnCGPM7Y8wLxph9xphPjUVfEzHG3GyMOWCMOWSMuSfB8jJjzEOx5TsK7XPUJ4VxfCX2e7DPGLPNGHPhWPRzJCONY0C7jxtjrDGmIGc1pzIOY8yfx34mLxhjfpLvPqYihc/VBcaYJ4wxz8c+W0vHop8jMcasN8acMMbsT7LcGGO+GRvnPmPMlSNu1FrLnxT/SPpXSffEHt8j6etJ2jVL+mDscYWkc8a675mMI7b83yX9RNK3xrrfmYxD0hxJs2OPz5fULqmyAPrulXRY0sWSApL2Srp8UJu/lvTd2OPlkh4a635nOI5Ffb8DkhrcOo5Yu/GSnpK0XdL8se53hj+P2ZKelzQp9nzqWPc7w3Gsk9QQe3y5pFfHut9JxnK9pCsl7U+yfKmkLZKMpIWSdoy0Tfa403OLpA2xxxskLRvcwBhzuSSftfZxSbLWdlpr38lfF1My4jgkyRhzlaQqSb/OU7/SNeI4rLUHrbWtscevSzohadjiBnmyQNIha+3L1tpeSU2KjmeggeN7WNJiY4zJYx9TMeI4rLVPDPgd2C5pep77mIpUfh6S9E+Svi7pTD47l4ZUxnGnpG9ba09KkrX2RJ77mIpUxmElTYg9nijp9Tz2L2XW2qckvTVMk1sk/dBGbZdUaYypHm6bBHd6qqy17bHHxxQNtcHmSOowxvwidgjn34wx3vx1MSUjjsMY45F0n6Sv5rNjaUrl59HPGLNA0W/vh0e7YymokdQ24PmR2GsJ21hrw5JOSTo3L71LXSrjGOgORfcuCs2I44gdwpxhrf1VPjuWplR+HnMkzTHGPGOM2W6MuTlvvUtdKuP4H5I+Y4w5ImmzpP+Sn67lXLq/Q/KNandcyBizVdK0BItWD3xirbXGmERT8n2SrpM0T9IfJT0k6XZJ389tT4eXg3H8taTN1tojY7mTl4Nx9G2nWtKPJK2w1kZy20ukwhjzGUnzJd0w1n1JV+yL7DcU/V12O5+ih8vrFT368ZQxps5a2zGmvUrfbZIetNbeZ4y5VtKPjDHvLoXfb4J7EGvtjcmWGWOOG2OqrbXtsSBIdIjpiKQ91tqXY+s8ouh5i7wGdw7Gca2k64wxf63oefqAMabTWpt00s5oyME4ZIyZIOlXklbHDkUVgqOSZgx4Pj32WqI2R4wxPkUPB76Zn+6lLJVxyBhzo6Jftm6w1vbkqW/pGGkc4yW9W1Jz7IvsNEmbjDEftdbuylsvR5bKz+OIoudRQ5JeMcYcVDTIn8tPF1OSyjjukHSzJFlrf2eMGado7e9CPPQ/nJR+hwbiUHl6NklaEXu8QtIvE7R5TtFzFH3nUT8g6cU89C0dI47DWvtpa+0F1tqZih4u/2G+QzsFI47DGBOQtFHR/j+cx76N5DlJs40xF8X6uFzR8Qw0cHyfkPQbG5vNUkBGHIcxZp6k/yPpowV6PlUaYRzW2lPW2inW2pmx34ntio6nkEJbSu1z9Yiie9syxkxR9ND5y/nsZApSGccfJS2WJGPMZZLGSfpTXnuZG5sk/WVsdvlCSacGnAJMbKxn3Lnpj6LnF7dJapW0VdLk2OvzJX1vQLsPStonqUXSg5ICY933TMYxoP3tKsxZ5SOOQ9JnJIUk7Rnw54qx7nusb0slHVT0nPvq2Gv/U9FAkKL/Ef1M0iFJOyVdPNZ9znAcWyUdH/Dvv2ms+5zJOAa1bVYBzipP8edhFD3s/2Ls/6jlY93nDMdxuaRnFJ1xvkfSh8a6z0nG8R+KXs0SUvRoxx2S7pJ014Cfx7dj42xJ5XNF5TQAAFyEQ+UAALgIwQ0AgIsQ3AAAuAjBDQCAixDcAAC4CMENAICLENwAALgIwQ0AgIv8/2WYGbHe0NmEAAAAAElFTkSuQmCC",
"text/plain": [
"<Figure size 576x576 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.figure(figsize=(8,8))\n",
"plt.scatter(C[:,0].data, C[:,1].data, s=200)\n",
"for i in range(C.shape[0]):\n",
" plt.text(C[i,0].item(), C[i,1].item(), fap[i], ha='center', va='center', color='white')\n",
"plt.grid('minor');"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PTpjDaqmVrBB"
},
"source": [
"Como vemos, las vocales —junto con la «y», aunque sin la «i»— aparecen agrupadas entre sí: esto quiere decir que nuestro modelo las interpreta como letras similares. Lo mismo aplica para las consonantes.\n",
"\n",
"Hasta el momento, no hemos conseguido mejorar nuestro modelo anterior. En la práctica, la optimización de modelos se hace con un número de pruebas que contengan hiperparámetros distintos. Podemos, por ejemplo, aumentar nuestros parámetros, nuestro *embedding*, nuestras capas, probar nuevas tasas de aprendizaje, entrenar el modelo por más tiempo, etcétera. Intentemos hacer un poco de todo para finalizar esta lección:\n",
"\n",
"### Ensayo de un modelo con hiperparámetros óptimos"
]
},
{
"cell_type": "code",
"execution_count": 445,
"metadata": {
"id": "mJea2Woegj5H"
},
"outputs": [],
"source": [
"random.shuffle(nombres)\n",
"n1 = int(0.8*len(nombres))\n",
"n2 = int(0.9*len(nombres))\n",
"\n",
"Xtr, Ytr = construir_dataset(nombres[:n1])\n",
"Xdev, Ydev = construir_dataset(nombres[n1:n2])\n",
"Xte, Yte = construir_dataset(nombres[n2:])\n",
"X, Y = construir_dataset(nombres)\n",
"\n",
"C = torch.randn(27, 10) # un embedding de 10 dimensiones\n",
"h = 200 # aumentamos las unidades ocultas a 200\n",
"H = torch.randn((30, h)) # recordemos: 10 * 3 = 30\n",
"d = torch.randn(h)\n",
"U = torch.randn(h, 27)\n",
"b = torch.randn(27)\n",
"\n",
"parametros = [C, H, d, U, b]"
]
},
{
"cell_type": "code",
"execution_count": 446,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "WcrQYS_Lg-m-",
"outputId": "189c674b-fad4-4ca1-d3d6-caf7e85129fa"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1.7781141996383667\n"
]
}
],
"source": [
"for p in parametros:\n",
" p.requires_grad = True\n",
"\n",
"for i in range(15000):\n",
" #minibatch («minilote»)\n",
" ix = torch.randint(0, Xtr.shape[0], (32,))\n",
"\n",
" # propagación hacia delante\n",
" emb = C[Xtr[ix]]\n",
" a = torch.tanh(emb.view(-1, 30) @ H + d)\n",
" logits = a @ U + b\n",
" perdida = F.cross_entropy(logits, Ytr[ix])\n",
" \n",
" # propagación hacia atrás\n",
" for p in parametros:\n",
" p.grad = None\n",
" perdida.backward()\n",
"\n",
" # actualización\n",
" lr = 0.1 if i < 14000 else 0.01 # nuestra tasa de aprendizaje disminuirá hacia el final del entrenamiento\n",
" for p in parametros:\n",
" p.data += -lr * p.grad\n",
"\n",
"print(perdida.item())"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vFEC-VhliHVW"
},
"source": [
"Finalmente hemos superado al modelo anterior con fuerza bruta. Aunque podríamos seguir ajustándolo, me parece que esto será suficiente para obtener mejores muestras de nombres. Revisemos la pérdida de nuestros `Xdev` y `Xte` para cerciorarnos de que nuestro modelo haya generalizado sus aprendizajes, en vez de memorizarlos:"
]
},
{
"cell_type": "code",
"execution_count": 448,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "VJcK54eTPpTo",
"outputId": "70af0d4e-cab5-4cce-a2f1-31dcfa00656e"
},
"outputs": [
{
"data": {
"text/plain": [
"tensor(1.9739, grad_fn=<NllLossBackward0>)"
]
},
"execution_count": 448,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"emb = C[Xdev]\n",
"a = torch.tanh(emb.view(-1, 30) @ H + d)\n",
"logits = a @ U + b\n",
"perdida = F.cross_entropy(logits, Ydev)\n",
"perdida"
]
},
{
"cell_type": "code",
"execution_count": 449,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "B6qHnZwVkDqn",
"outputId": "f68ffdc3-23c7-4f48-b00a-c13c4e102b33"
},
"outputs": [
{
"data": {
"text/plain": [
"tensor(1.9782, grad_fn=<NllLossBackward0>)"
]
},
"execution_count": 449,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"emb = C[Xte]\n",
"a = torch.tanh(emb.view(-1, 30) @ H + d)\n",
"logits = a @ U + b\n",
"perdida = F.cross_entropy(logits, Yte)\n",
"perdida"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "P9gJFQCBkJcp"
},
"source": [
"Tenemos también pérdidas razonables para ambos conjuntos de datos. Conforme la pérdida del entrenamiento difiera más de las otras dos (`Xdev` y `Xte`), esto indicaría que nuestros parámetros están memorizando los datos, no generalizándolos. En ese sentido, debemos ser cuidados de no sobreentrenar, ni tampoco sobredimensionar el modelo, puesto que si nos excedemos, terminará memorizando todos nuestros datos.\n",
"\n",
"Para finalizar, obtengamos algunas muestras de nombres generados por nuestro modelo:"
]
},
{
"cell_type": "code",
"execution_count": 455,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "vS8h0tWHQLCH",
"outputId": "e53669c7-8d2b-4a81-e5d1-a61434fe4cc2"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"cedelzorena.\n",
"carixto.\n",
"leiscasiria.\n",
"jovino.\n",
"renta.\n",
"marizio.\n",
"alfidela.\n",
"alsa.\n",
"paulogia.\n",
"taw.\n",
"migdina.\n",
"duboa.\n",
"secahmidio.\n",
"sihorgel.\n",
"floreta.\n",
"remenicia.\n",
"penoyda.\n",
"filiano.\n",
"frinda.\n",
"geria.\n"
]
}
],
"source": [
"block_size = 3\n",
"for _ in range(20):\n",
" out = []\n",
" context = [0] * block_size\n",
" while True:\n",
" emb = C[torch.tensor([context])]\n",
" a = torch.tanh(emb.view(1, -1) @ H + d)\n",
" logits = a @ U + b\n",
" probs = F.softmax(logits, dim=1)\n",
" ix = torch.multinomial(probs, num_samples=1).item()\n",
" context = context[1:] + [ix]\n",
" out.append(ix)\n",
" if ix == 0:\n",
" break\n",
"\n",
" print(''.join(fap[i] for i in out))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uZgbBCm9nThf"
},
"source": [
"Indudablemente, estos nombres son más respetables."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rbHyRZ9gbY3P"
},
"source": [
"[^1]: El método `view` es eficiente porque [no requiere de nuevo espacio](http://blog.ezyang.com/2019/05/pytorch-internals/) en la memoria de nuestra computadora, sino que utiliza el mismo tensor para reacomodarlo de manera distinta. Conforme nuestros programas se vuelvan más complejos, debemos procurar eficientar al máximo nuestros recursos computacionales. Para un acercamiento más general al tema, véase el [artículo de Horace He](https://horace.io/brrr_intro.html).\n",
"\n",
"[^2]: En *deep learning*, el término *hidden* («oculto») se utiliza para referirse a componentes de una red neuronal que no son «visibles» ni como entradas ni como salidas de la red. Se trata de entradas o salidas que se procesan internamente por la red neuronal antes de arrojar un resultado final. La *hidden unit*, en ese sentido, se refiere a cada unidad o componente de una capa oculta (*hidden layer*) en la red. Podemos entenderla como sinónimo de «nodo» o «neurona».\n",
"\n",
"[^3]: Se denominan «parámetros» a los números o coeficientes que la red neuronal aprende y determina durante el entrenamiento (por ejemplo, los números de los pesos y los sesgos). Por otra parte, los «hiperparámetros» son aquellos que nosotros definimos manualmente según la naturaleza de cada problema, y que optimizamos mediante pruebas con distintos valores (por ejemplo, nuestras unidades ocultas $h$)."
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}