{ "cells": [ { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "6oXxxX9LZL-h" }, "source": [ "# Dirichlet Process Mixture Models in Pyro\n", "\n", "\n", "## What are Bayesian nonparametric models?\n", "Bayesian nonparametric models are models where the number of parameters grow freely with the amount of data provided; thus, instead of training several models that vary in complexity and comparing them, one is able to design a model whose complexity grows as more data are observed. The prototypical example of Bayesian nonparametrics in practice is the *Dirichlet Process Mixture Model* (DPMM). A DPMM allows for a practitioner to build a mixture model when the number of distinct clusters in the geometric structure of their data is unknown – in other words, the number of clusters is allowed to grow as more data is observed. This feature makes the DPMM highly useful towards exploratory data analysis, where few facets of the data in question are known; this presentation aims to demonstrate this fact.\n", "\n", "## The Dirichlet Process (Ferguson, 1973)\n", "Dirichlet processes are a family of probability distributions over discrete probability distributions. Formally, the Dirichlet process (DP) is specified by some base probability distribution $G_0: \\Omega \\to \\mathbb{R}$ and a positive, real, scaling parameter commonly denoted as $\\alpha$. A sample $G$ from a Dirichlet process with parameters $G_0: \\Omega \\to \\mathbb{R}$ and $\\alpha$ is itself a distribution over $\\Omega$. For any disjoint partition $\\Omega_1, ..., \\Omega_k$ of $\\Omega$, and any sample $G \\sim DP(G_0, \\alpha)$, we have:\n", "\n", "$$(G(\\Omega_1), ..., G(\\Omega_k)) \\sim \\text{Dir}(\\alpha G_0(\\Omega_1), ..., \\alpha G_0(\\Omega_k))$$\n", "\n", "Essentially, this is taking a discrete partition of our sample space $\\Omega$ and subsequently constructing a discrete distribution over it using the base distribution $G_0$. While quite abstract in formulation, the Dirichlet process is very useful as a prior in various graphical models. This fact becomes easier to see in the following scheme.\n", "\n", "## The Chinese Restaurant Process (Aldous, 1985)\n", "\n", "Imagine a restaurant with infinite tables (indexed by the positive integers) that accepts customers one at a time. The $n$th customer chooses their seat according to the following probabilities:\n", "\n", "* With probability $\\frac{n_t}{\\alpha + n - 1}$, sit at table $t$, where $n_t$ is the number of people at table $t$\n", "* With probability $\\frac{\\alpha}{\\alpha + n - 1}$, sit at an empty table\n", "\n", "If we associate to each table $t$ a draw from a base distribution $G_0$ over $\\Omega$, and then associate unnormalized probability mass $n_t$ to that draw, the resulting distribution over $\\Omega$ is equivalent to a draw from a Dirichlet process $DP(G_0, \\alpha)$. \n", "\n", "Furthermore, we can easily extend this to define the generative process of a nonparametric mixture model: every table $t$ that has at least one customer seated is associated with a set of cluster parameters $\\theta_t$, which were themselves drawn from some base distribution $G_0$. For each new observation, first assign that observation to a table according to the above probabilities; then, that observation is drawn from the distribution parameterized by the cluster parameters for that table. If the observation was assigned to a new table, draw a new set of cluster parameters from $G_0$, and then draw the observation from the distribution parameterized by those cluster parameters.\n", "\n", "While this formulation of a Dirichlet process mixture model is intuitive, it is also very difficult to perform inference on in a probabilistic programming framework. This motivates an alternative formulation of DPMMs, which has empirically been shown to be more conducive to inference (e.g. Blei and Jordan, 2004).\n", "\n", "## The Stick-Breaking Method (Sethuraman, 1994)\n", "\n", "The generative process for the stick-breaking formulation of DPMMs proceeds as follows:\n", "\n", "* Draw $\\beta_i \\sim \\text{Beta}(1, \\alpha)$ for $i \\in \\mathbb{N}$\n", "* Draw $\\theta_i \\sim G_0$ for $i \\in \\mathbb{N}$\n", "* Construct the mixture weights $\\pi$ by taking $\\pi_i(\\beta_{1:\\infty}) = \\beta_i \\prod_{j" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "data = torch.cat((MultivariateNormal(-8 * torch.ones(2), torch.eye(2)).sample([50]),\n", " MultivariateNormal(8 * torch.ones(2), torch.eye(2)).sample([50]),\n", " MultivariateNormal(torch.tensor([1.5, 2]), torch.eye(2)).sample([50]),\n", " MultivariateNormal(torch.tensor([-0.5, 1]), torch.eye(2)).sample([50])))\n", "\n", "plt.scatter(data[:, 0], data[:, 1])\n", "plt.title(\"Data Samples from Mixture of 4 Gaussians\")\n", "plt.show()\n", "N = data.shape[0]" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "CglLQke4gEYd" }, "source": [ "In this example, the cluster parameters $\\theta_i$ are two dimensional vectors describing the means of a multivariate Gaussian with identity covariance. Therefore, the Dirichlet process base distribution $G_0$ is also a multivariate Gaussian (i.e. the conjugate prior), although this choice is not as computationally useful, since we are not performing coordinate-ascent variational inference but rather black-box variational inference using Pyro. \n", "\n", "First, let's define the \"stick-breaking\" function that generates our weights, given our samples of $\\beta$:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "colab": {}, "colab_type": "code", "id": "2ngrqFlDQYpV" }, "outputs": [], "source": [ "def mix_weights(beta):\n", " beta1m_cumprod = (1 - beta).cumprod(-1)\n", " return F.pad(beta, (0, 1), value=1) * F.pad(beta1m_cumprod, (1, 0), value=1)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "LAC0bWL6Qcc3" }, "source": [ "Next, let's define our model. It may be helpful to refer the definition of the stick-breaking model presented in the first part of this tutorial. \n", "\n", "Note that all $\\beta_i$ samples are conditionally independent, so we model them using a `pyro.plate` of size `T-1`; we do the same for all samples of our cluster parameters $\\mu_i$. We then construct a Categorical distribution whose parameters are the mixture weights using our sampled $\\beta$ values (line 9) below, and sample the cluster assignment $z_n$ for each data point from that Categorical. Finally, we sample our observations from a multivariate Gaussian distribution whose mean is exactly the cluster parameter corresponding to the assignment $z_n$ we drew for the point $x_n$. This can be seen in the Pyro code below:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "colab": {}, "colab_type": "code", "id": "WfnbSIocRlvQ" }, "outputs": [], "source": [ "def model(data):\n", " with pyro.plate(\"beta_plate\", T-1):\n", " beta = pyro.sample(\"beta\", Beta(1, alpha))\n", "\n", " with pyro.plate(\"mu_plate\", T):\n", " mu = pyro.sample(\"mu\", MultivariateNormal(torch.zeros(2), 5 * torch.eye(2)))\n", "\n", " with pyro.plate(\"data\", N):\n", " z = pyro.sample(\"z\", Categorical(mix_weights(beta)))\n", " pyro.sample(\"obs\", MultivariateNormal(mu[z], torch.eye(2)), obs=data)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "1gBQj5RKRn8Z" }, "source": [ "Now, it's time to define our guide and perform inference. \n", "\n", "The variational family $q(\\beta, \\theta, z)$ that we are optimizing over during variational inference is given by:\n", "\n", "$$q(\\beta, \\theta, z) = \\prod_{t=1}^{T-1} q_t(\\beta_t) \\prod_{t=1}^T q_t(\\theta_t) \\prod_{n=1}^N q_n(z_n)$$ \n", "\n", "Note that since we are unable to computationally model the infinite clusters posited by the model, we truncate our variational family at $T$ clusters. This does not affect our model; rather, it is a simplification made in the *inference* stage to allow tractability. \n", "\n", "The guide is constructed exactly according to the definition of our variational family $q(\\beta, \\theta, z)$ above. We have $T-1$ conditionally independent Beta distributions for each $\\beta$ sampled in our model, $T$ conditionally independent multivariate Gaussians for each cluster parameter $\\mu_i$, and $N$ conditionally independent Categorical distributions for each cluster assignment $z_n$.\n", "\n", "Our variational parameters (`pyro.param`) are therefore the $T-1$ many positive scalars that parameterize the second parameter of our variational Beta distributions (the first shape parameter is fixed at $1$, as in the model definition), the $T$ many two-dimensional vectors that parameterize our variational multivariate Gaussian distributions (we do not parameterize the covariance matrices of the Gaussians, though this should be done when analyzing a real-world dataset for more flexibility), and the $N$ many $T$-dimensional vectors that parameterize our variational Categorical distributions:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "colab": {}, "colab_type": "code", "id": "Imw4wcVkT9er" }, "outputs": [], "source": [ "def guide(data):\n", " kappa = pyro.param('kappa', lambda: Uniform(0, 2).sample([T-1]), constraint=constraints.positive)\n", " tau = pyro.param('tau', lambda: MultivariateNormal(torch.zeros(2), 3 * torch.eye(2)).sample([T]))\n", " phi = pyro.param('phi', lambda: Dirichlet(1/T * torch.ones(T)).sample([N]), constraint=constraints.simplex)\n", "\n", " with pyro.plate(\"beta_plate\", T-1):\n", " q_beta = pyro.sample(\"beta\", Beta(torch.ones(T-1), kappa))\n", "\n", " with pyro.plate(\"mu_plate\", T):\n", " q_mu = pyro.sample(\"mu\", MultivariateNormal(tau, torch.eye(2)))\n", "\n", " with pyro.plate(\"data\", N):\n", " z = pyro.sample(\"z\", Categorical(phi))" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "H0He1id0T_bN" }, "source": [ "When performing inference, we set our 'guess' for the maximum number of clusters in the dataset to $T = 6$. We define the optimization algorithm (`pyro.optim.Adam`) along with the Pyro SVI object and train the model for 1000 iterations. \n", "\n", "After performing inference, we construct the Bayes estimators of the means (the expected values of each factor in our variational approximation) and plot them in red on top of the original dataset. Note that we also have we removed any clusters that have less than a certain weight assigned to them according to our learned variational distributions, and then re-normalize the weights so that they sum to one:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 372 }, "colab_type": "code", "id": "x1Yidukpd9wO", "outputId": "b0cc290b-3285-4f36-c2a6-7195b6801482" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "100%|██████████| 1000/1000 [00:15<00:00, 64.86it/s]\n", "100%|██████████| 1000/1000 [00:15<00:00, 65.47it/s]\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAABDAAAAFgCAYAAABNIolGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3df5Ak6V3f+c+3e7qJ7V3twdaMfqurFceG4yTHIVsdizm4CzBCSHM+LxDAraJ2YrxLqLWzEjEO3x8W9IUhIFphc4eJMTCaaJlZhpmyhMK2LIUZ9At8J7ANqBdL9gqh01rqbg2rQDM9GO1ubzA708/9kZXbVdX5ZGVm5c/K9yuioqezsrKerp7Z/Ozz4/uYc04AAAAAAAB1Nld1AwAAAAAAACahAwMAAAAAANQeHRgAAAAAAKD26MAAAAAAAAC1RwcGAAAAAACovWNVNyCL48ePu5WVlaqbAQAAJD355JM3nHMnqm5HVuQKAADqxZctGtmBsbKyoq2traqbAQAAJJnZTtVtmAa5AgCAevFlC5aQAAAAAACA2qMDAwAAAAAA1F4uHRhmdtHMvmFmTw0du8/MPmVmXx58/TbPa08PzvmymZ3Ooz0AAKC5yBUAACBKXjMwfl3S28aOvVfS7zjn7pf0O4PvR5jZfZJ+RtJ3SnpA0s/4AgkAAGiNXxe5AgAAjMmlA8M59xlJN8cOPyjp0uDPlyT9UMRLf1DSp5xzN51zfyHpUzoaWAAAQIuQKwAAQJQia2C8wjn3dUkafH15xDmvkfS1oe+vDY4dYWZrZrZlZlvXr1/PvbEAAKDWyBUAALRc1UU8LeKYizrRObfpnFt1zq2eONHYreYBAEBxyBUAAMywIjsw/tzMXiVJg6/fiDjnmqTXDX3/WknPFNgmAADQTOQKAABarsgOjI9JCqt/n5b00YhzPiHprWb2bYMiW28dHAMAABhGrgAAoOXy2kb1g5L+o6S/ZmbXzOwnJP1jST9gZl+W9AOD72Vmq2b2zyXJOXdT0s9L+uzg8XODYwAAoKXIFQAAIIo5F7k0tNZWV1fd1tZW1c0AACB3/b60vi7t7krLy9LGhtTrVd2qeGb2pHNutep2ZEWuAADMqibmCsmfLY5V0RgAAHDU449LFy5I4djCzo60thb8uQlhAwAA1Mcs5oqqdyEBAAAKRkiGQ0Zofz8YOQEAAEhqVnMFHRgAANTA+vrRkBHa3S23LQAAoNlmNVfQgQEAwJT6fWllRZqbC772++mvERcmlpeztgwAADTRtNliVnMFHRgAAEyh35ceeSRYV+pc8PWRR4LjacKHL0yYBQW3AABAO+SRLWY1V9CBAQBotWlHOM6elV58cfTYiy9K73pXUChrOHysrfmvv7EhLS2NHjOTHnusuYW2AABomzxmZeaRLWY1V9CBAQBopDwCQr+fLAjEvdfeXvS1n38+KJQ1bH9fOn169DrhtU+dku66S+p0goDR7UqXL0vnz6f/uQAAQHrTZos8coU0XbZ4/PHZzhXmfJU9aoz92gGg3cKAMHwTX1qSNjfTjSqsrAThYly3K21vJ3svsyw/QWBhIXj9rVvR124K317tTUGuAADkkS3yyBXSdNliXBNzheTPFnRgAAAaJ0lASGJuLrpCt5l0cJDsvY4f94+UZJX256gaHRgAgKbLI1vkkSuk/LNF03KF5M8WLCEBADSOr7J22m3BfAWuho9Peq9z56TFxXTvO0nanyOP5TQAALRZHtkij1whBdlifv7oOceOBbM305qlXEEHBgCgcZIEhCSiClwtLY1W5570Xr2edPFidNCQguNm/ufjrp1E0vW2AADAL49skUeukIJs8a3fevSc27ele+8NZlSkyRazlCvowAAANE6SgJBErxesCw2DQLd7dJ1okvfq9Q6nho47OAgely4dvc7CwtHZG5N+jvFRkbNnowt6ra9Hn1+XAAIAQJ3kkS3yyhWSdPNm9HvcvBksB/Fli3FJfobhrHD6dHyuGD+/9GzhnGvc481vfrMDALTblSvOdbvOmQVfr1yp9r26XeeCsYrRR7frv86ZM851OofndjrxP8eVK84tLUW/z/jDLPr8paX8PytJW64G+SDrg1wBAHCuvGyRV66Iutb3f79z8/PBufPzQdaY1JYk2cLMf36Z2YIingAA5CBt9fIs1c59hb+idLvB1zyKnU5CEU8AAPKVJScUmS3C7JBXIfVJKOIJAJhZdVgmkWTa6HBbH3548hTNcUmLcIXTRfMqdgoAQJs0KVdIxWeL4WUoVWcLOjAAAI1Wl2JT/X4QEHZ3pfvuk557Tjp1ajT4DLfVJy4A+IpwdTrRASevYqcAALRFXXLFuOeeC+pejXeqFJUtwiLk4x0nVWcLOjAAAI22vp5+tGFYHqMs42Fnby94jAefqLaOm5vzt8VX+OvcuWDa5uXLwbGw4+TkyXyKnQIA0BZ1yBXhdarMFpcuBYVCNzaC9whfX3m2iCqMUfcHxbYAACGz+GJTcZIUopqm0NZ40S1fW32PqKJYvvb4fpYzZ4ovSCaKeAIAZkTRuSI8j2wRz5ctKOIJAGi0aYpJTXpt0mJYc3PBbT2OWbC0ZG8v+vn5eenOnWw/hyQdPx597byLakWhiCcAYFYUmSuk5mSLfj/YUnWabDINingCAGbSNPu2TypElXQaaZJ1n/fdJz377NHjCwvSlSvBNM0oOzuTp6D2+/7wQsFOAACSKzJXSM3IFmEnS1TnhVRttqADAwDQaL4q3dLkNaiTClElrbQdFXaGhc/dunX0uXvvjS+4KU0uIBa3LpeCnQAAJBe3+8ek+hZJClzmmS3+6q+KyRaT6mpUmS3owAAANF6vF0xlPDg4nNKYpIL4pFGWpJW2x8NOpxM8wj/fdZd/hkR4fFJQiSsgFjcSQsFOAADSGc8VYefFpGyRZPZGXtlibi7YmSTKzZv+9gzzZYu4XFF1MXA6MAAAM8c3PfPs2dGREyl+j/U000iHw86NG8Hj8mXphRf8nRdSsD41fH3YFh/fFmlx26tG7RcPAADSSZIt1teDuhG+XCHlly18nRfSYS7Imi3itlYd/3nKRgcGAGDm+EYO9vaOjpxIR0dZQlGjH3fddbhN6aSt0ZJsbTa8vjQMKr6gYRb9nnHbqwIAgOklzRaXLgX35ahcIR3tVJifP5wJkWTL1STZYrgzJEu2iNtateqBETowAAAzJ+nazDT7ujsXTMmM2oPdJ0mRq6hAsbERBIqoNpw+fXTtbdx6XQAAML28s0U4gyIcyEiSK6TJ2cI3+zJNtqhzrmAbVQDAzInaoszHzF+lO8l14rYS822nForaNm24XZOYSY89Jp0/P/ncIrGNKgBg1pWVLSZtURqXLeJyRdiuSRYWpCeeqL6zgm1UAQCtETVy0OlEnxs3opJkmuakAprjUzDD8DBpNCNuvWrIOenChWRTTgEAQHZlZYuRXBGx7YmvMGenM3mWRJJs8eKLQV2PuqIDAwAwk8YriJ87l35f9yRLQOJCSlTYuXw56HiIWhc7bFLl8JBzyZfBAACA7MrIFi/lCs+2Jz31j2SLK1eCAp+TZk0kzRZxxcerVmgHhpn9NTP73NDjm2b298fO+V4z+8uhc/5RkW0CALRTlvWck9a7JtlKLGortjjhYMupU0HB0HDLtHC3kihJOlpmAbkCAFAneWeLkVzh2/ZkfT23bNFEx4q8uHPuS5LeJElmNi/pzyR9JOLU33PO/Z0i2wIAQK+Xbk3nxsbRdapmwUBItxs8n+ca0fF1sXt7QZi5fDn4/tSp4L3HJS0s1nTkCgBA3eSRLaSgU+HcuaFr+UYnUo5axGWLs2ejZ1v4lsbUQZlLSL5f0n91zsWUMwMAoD7iloBsbASDI+M7gkwjZrBFvV5QsHN8xCTJLJAZRa4AADROVLa4ciXovBjOFc/d5xmdSDlqEZctzp2TFhdHn1tcrPc27GV2YDwk6YOe577LzD5vZr9tZm+MOsHM1sxsy8y2rl+/XlwrAQAYcmSapvp67viK3vHwnP6fnRU95PqJtz6bZNJgy/nzQQdKHbc1qwC5AgBQDxHFNuOMZwvpaLmLn/zmhm4vpiywESEuW/R60sWLo7ni4sV654pStlE1s0VJz0h6o3Puz8eeu1fSgXPuOTM7Kemcc+7+uOux3RkAoBIRe589ryW9U5v6oHrqdKR77glCwfJy+iUmvq3RJm2pVrWyt1ElVwAAaiNqX9RJ+5mO8d3/f7LT1/u0rqW9Xe1qWf+0s6HvPNdrdbYoawbG2yX98XjIkCTn3Dedc88N/nxV0oKZHS+pXQAAJBcxD/Nu7et9CrYB2ds7Uiw81ayMqOrgLV4iEodcAQCoh7g1Ggn5Zkn88l5Pr3hhW/M60Ou1rV/e67U+W5TVgfEOeaZ5mtkrzYIVvWb2wKBNNd64BQDQWp6Esazo4ynzS6Zq5i1FrgAA1EMOxTZ9ZS3m56fuG5m5bFHoLiSSZGZLkn5A0ruGjj0mSc65C5J+VNIZM7st6QVJD7ky1rUAAJDW8nLkPMxd+Qtqpd3iNG0187YhVwAAasWTDdIU24zamWRp6WjnRajN2aLwGRjOuX3nXMc595dDxy4MQoacc7/inHujc+47nHN/yzn3H4puEwAAmUTMw3xeS3rf3RveLceWl4/W9nr88VS1vjCEXAEAqJWc1mjcddfhnzudw1kTUdqcLcrchQQAgMYZCQjrPV38rk3tWlcHMm2rq3dqU33X04//eHR+OXnyaGXx979/uloZU/0MK7MbagAAKF2GNRrD9+Xjx6VHHgnqaIVeeCH46usbqTpbVJkrStmFJG9UCwcAlCGqsLhZEA7GdbtB0FhfH92FZH09emZp1OuLqAaeQ3H0icrehSRv5AoAQFmi7stRwlzQ79crW5SRKyR/tqADAwAwc6Ju9lluqr6tx6KYBfu5j5ubi+7wSPr6aZWxfRodGACAWVd2tojLBVVmi7K2Za16G1UAAEoRjgyknUYZNR0yTZEsX62upDW8UtT6SiWH4ugAALRalmzhW2aR9P4blwuqzBZV5wo6MAAAMyXLduy+YHLffdHnB5t0Hoqr1RW1fnVckfuxT9uxAgBA26XNFnEdHknuv5NyQZXZoupcQQcGAGCmJBkZGB8VOXs2OphI0cWzHnvMX6tr/NrS0dpeZ86Utx97TsXRAQBorbTZ4vRpf4dH1H15cTHYecSXC+qULSrPFc65xj3e/OY3OwAAonS7zgXjHaOPbjd4/soV55aWos8Zf5gF53e7wZ+73eB7n6hrLy3FvybqGknfr8prDpO05WqQD7I+yBUAgDh5ZQuzw/ObnC2KzhXO+bMFRTwBADNlUnXsNIU50xakmrawVVmVvfNGEU8AwCzLK1tkKXRJthjFEhIAwEyZtB170iJTWaZDTlvYKkv9DgAAUKw8skXWZRZki1F0YAAAZk6vF4xKHBwEX4dHGHxFpjqd6deOTlvYqurK3gAAIFqWbDE/P31NCrLFKDowAACt4is+de6cP5hMe+2kIy5VV/YGAADp+e7/ly5Nlyvirt3WbEEHBgCgVSZNAy3r2lH7w1de2RsAAKRGtigPRTwBAChZXEEtKViXursbjI5sbNS7yJZEEU8AAKrWlmxBBwYAACWbtqJ43dCBAQBAtdqSLVhCAgBAyWatoBYAAKhWW7IFHRgAkIOoNYeAz6wV1AIA5I9sgTTaki3owACAKYVrDnd2JOeCr2trBA34zVpBLQBAvsgWSKst2YIODACY0vr6aMEkKfh+fb2a9qD+iqxWDgBoPrIF0mpLtqCIJwBMaW4uGB0ZZxbs/Q3MOop4AkC+yBZoO4p4AkBB2rLmEAAAlINsAUSjAwMAptSWNYcAAKAcZAsgGh0YADCltqw5BAAA5SBbANGOVd0AAJgFvR6hAgAA5IdsARzFDAwAAAAAAFB7dGAAAAAAAIDaowMDAAAAAADUHh0YAAAAAACg9ujAADDT+n1pZUWamwu+9vtVtwh1xt8XAMAk3CuQBn9f8lX4LiRmti3pWUl3JN12zq2OPW+Szkk6KWlf0t9zzv1x0e0CMPv6fWltTdrfD77f2Qm+l6jqjaP4+9IM5AoAVeJegTT4+5I/c84V+wZB0Fh1zt3wPH9S0k8qCBrfKemcc+474665urrqtra28m4qgBmzshLcKMZ1u9L2dtmtQd3x9yU7M3tyvCOhwPfaFrkCQEW4VyAN/r5k58sWdVhC8qCk33CBP5D0rWb2qqobBaD5dnfTHUe78fdlZpArABSGewXS4O9L/srowHCSPmlmT5rZWsTzr5H0taHvrw2OAcBUlpfTHUe78felMcgVACrDvQJp8Pclf2V0YHy3c+5vSnq7pHeb2f8y9rxFvObIuhYzWzOzLTPbun79ehHtBDBjNjakpaXRY0tLwXFgHH9fGoNcAaAy3CuQBn9f8ld4B4Zz7pnB129I+oikB8ZOuSbpdUPfv1bSMxHX2XTOrTrnVk+cOFFUcwHMkF5P2twM1hmaBV83NymahGj8fWkGcgWAKnGvQBr8fclfoUU8zexuSXPOuWcHf/6UpJ9zzn186Jz/VdJ7dFhs658558bDyAiKbQEAUB9lFfEkVwAA0A6+bFH0NqqvkPSRYEczHZP0L5xzHzezxyTJOXdB0lUFIeNpBdudPVJwmwAAQDORKwAAaLFCOzCcc1+R9B0Rxy8M/dlJeneR7QAAAM1HrgAAoN3qsI0qAAAAAABALDowAAAAAABA7dGBAQAAAAAAao8ODAAAAAAAUHt0YAAAAAAAgNqjAwMAAAAAANQeHRgAAAAAAKD26MAAAAAAAAC1RwcGAAAAAACoPTowAAAAAABA7dGBAQAAAAAAao8ODAAzqd+XVlakubnga79fdYsAAECTkS2A6tGBAWBmhMHCTDp1StrZkZwLvq6ttTNoELYAAMhm+B56/Lj0yCNkC4lsgWrRgQFgJvT7QZDY2Qm+d270+f19aX29/HZVafgzaXvYAgAgjfF76N6e9OKLo+eQLcgWKJ+58ZTfAKurq25ra6vqZgCokZWVw84LHzPp4KCU5tSC7zPpdqXt7bJbg1lmZk8651arbkdW5AoA45LkColsESJbIG++bMEMDAAzYXd38jnLy8W3o058n0mSzwoAgDZLeq8kW8QfB/JGBwaAmTApQCwtSRsb6a+bZJ1nXdeC+j6TtoUtAADSSnKvLCpb1DVXSGQLVI8ODAAz4eTJYBrnsPD7blfa3JR6vXTXTLLOs85rQTc2gnA1LGvYAgCgTTY2pMXF0WPHjkmdTpAvisoWdc4VEtkC1aMDA0Dj9fvSpUujhTvNpMceC45tb6cPGFJQmGt/f/TYeMGu//ddfX1hf0V3NKevakXvUL82Rb16vSBcdbvThS0AANpovFSgmXTuXFDzoqhscfbs5OxRJbIFqkYRTwCNV1RBqbm5o+FFOizY9fuP9/U33r+mu3WYNJ7Xkt6pTX3Ieq0q6oV2o4gngFlTRba4fFl6+OHo17WtWChAEU8AM6uoglKT1nmubK6PdF5I0t3a1/u0Lufqt24VAAAkU0W2iJtlMTdXz5oYQNnowADQeEUVlJq0zvPVd6JTzLKC43VbtwoAAJKpIlvEdY7cuVPPmhhA2ejAANB4RRWUmrTO85n56BSzq8PjdVq3CgAAkqkiWyTtHCFboM3owADQeEUWlOr1grWuUQW7ttc29LxG083zWtJPazTdsDc6AADNUkW2iOo08SFboK3owABQa0n3Qo/raCjK95zv6T+d2dS1+a4OZNpRV+/Upj6o0Tdnb3QAAOohaa6Qys8WUZ0mnU70uWQLtBUdGAAkpbuhT/OatG2aZi/0otsnBZ0Yr729rTl3oN+/sq2PLo2mG/ZGBwC0Vd2yRRNyxXinyblzxSxlARrLOde4x5vf/GYHID9Xrji3tORccDsPHktLwfE8X5NWtzt6/fDR7U5+bRnt871vt+ucWfC16PcD6kDSlqtBPsj6IFcA+atjtmhirgjfm2yBtvFlC2ZgAND6elAQalhcgah+Xzp9Ot1rsphmC7O0P5M0OrJy/HjwiBpliRuBKXu6aRmjQQAApFXHbFF2rpAO79Nm0rFjwdc0uUIiWwDD6MAAkOqGHk6/vHMn3bXGrxF3Ywyfdy769UnWfaYNKePTSvf2gocbTDF99NHgnGmnn+apTm0BAGBYnbJFFbkifN/wPi0d/nx1zRXjba5De4AjoqZl5PGQ9DpJ/07SFyV9QdLZiHO+V9JfSvrc4PGPklybqZ5AvtJMqfSdG/ea4amPnY5zi4tHp2CeOXN4bTP/9RcXJ08/jWujb5ropJ9LCtqedvppkdM+p5kKC+RJJS0hKSpbkCuA/FWdLRYXg+PT5orwvebm0t9zJ/1cWXLF+M9OtsCs8mWLImdg3Jb0fzjn/gdJf0vSu83sDRHn/Z5z7k2Dx88V2B4AHmn2Op80CvLcc0dHPcZnNty6Nfqa/X3pwoXDEQrnGSGZ9Nz4SMe4hQV/0askozt7e9lGlIoaxZhmKizQUGQLoCGqzha3bgXHpey5InyvRx8Nlm+Mi8sV0uSfK22uCNtDtkCbFdaB4Zz7unPujwd/flbBaMlrino/ANml2et80jTLvb3RG2nUmtEokwJE6MUXgzWyUVNEJ73Xvff6140m3Y7Md17U8azrZZNK0xZgFpAtgOaoQ7ZI4sUXpbNn/ctP1tePdo6E4nKFlOx+nPZeTrZA25VSA8PMViT9DUl/GPH0d5nZ583st83sjTHXWDOzLTPbun79ekEtBdoraYGoqBGVccM30iJ67O/ciR51mPReN2/6nzt5MghYcTod/88/PjoU1568PpM0o1vArJk2W5ArgOI1JVvs7flnNMS9VzjDI0q/H2SDOGlzRVx7yBZoi8I7MMzsHkn/StLfd859c+zpP5bUdc59h6RflvRvfNdxzm0651adc6snTpworsEAYo2PqPiEN9Kie+z39w9HTibN4vC1pd+XLl2Kf/3CQrAXe/jzdzqjz+/tHRbkmvR+eX0maUa3gFmSR7YgVwD1UcdsEc72nIv5v6X5+ejj4TKPuA6OJLni1Cnp8cdHj5Mt0HaFdmCY2YKCgNF3zv3r8eedc990zj03+PNVSQtmdrzINgGY3vCISrcbfU54I43qyV9YCG7U4Y0xzqRZEdLhyMkkvtED31TU+fnDNj7xxOHN23cTv3Ur6EwZfr+iRzHK3loNqBrZAphNeWaL8c6ALMLZnr6dUcJzomTJFffcc/R854IaYcODI2QLtF1hHRhmZpJ+TdIXnXP/1HPOKwfnycweGLQnpq8SQNXGtyn79m+PPu/kyeBrVE/+E09IN24c3hh9QaXblS5f9o9wpNHp+G/AvmmXBwf+m7dvVGVv7/AzOnVKuuuu0c4aRjGA7MgWwOyJ2v40zBDjkmaLGzeku++Ovkank0+ukPz5JUuu8L3GuaBDhGwBBIqcgfHdkk5J+ttm9rnB46SZPWZmjw3O+VFJT5nZ5yX9M0kPDbZMAVBDUZWvf/d3o8+9evXwz5N68uOCSq8XXfk7jaWlYJqmT97TMccro7/wQtARk3QUI24ve6DlyBbADPHtqPHhD0efnzRb9PvRhTcXF4M8MG2ukIZmPUTctLPkirjnws+FbAFIhe/ZXsSD/dqBakzaz3z4YXb4ukn7lU/aczzN+0btsZ5kf/elpdHXLS3Fvy7cWz7q555m//QsbQGqJs9e7U15kCuAaqS9vyfNFr7rdjrZ3nf8/v5StvDctH/vzJXU9/IrV/wZYn6ebIH28WWLUnYhATAb0lS4DkcSkuxX7rvuzk5w3qTq5HF1Ml54YXJbsxSsOncuWG87bGHBXwg06WdX9PZoAADURdqdM5JmC9919/aC2QcnTwa54h3q66ta0R3N6ata0Tvkn5YwfH9/KVt4btrfc3U9da7o9aTHHjuaaZaW/LU2yBZoIzowACSWdEnFcDGpSTfNfj++wvepU9LDDwfrPaMKXC0tBTd83zrUpDfotAWrer1gve34+ttJhcdCvqmcRW+PBgBAXfhyxdxcsNxjWNJsMSlX7OwEhTEf3O/rA1rTinY0J6cV7egDWtM71B8pCBpVL+OlbBFz085SCPP8+WBZyHjHR5JsEbdEhGyBWUIHBjADylrXuLFxNFBIwfv6iknF3TTDEZS4Ct/hiMfeXhACzpw5emM/fz4IB76ZGEXdoKPCSZLq4HEjR0VvjwYAwCRl5oqoGZYHB8H9MW22CO+ncblCCq79Pq3rbo32gtytff3C/PpIQVBfvYzdXRVy086SLSbNSCFbYKZErSup+4O1qsChstc1+mo/+NZhxtW3yLIGNW6956RaGmWZpuYH61TRRKIGBjAzyr4PXbmSvsaD7z7qu07U4448BSeGC23EvFe3W+6HlaXmR/j5kS3QRL5swQwMoOHKXtd482b0cd9oSNyoQZaZEbu78g4NlbE3ehKTpo3GzUrJUo8DAIC8lJ0r4nYbS5stJs28GLm2kk1LiM0WJd6047LFpCUiZAvMEjowgIYre11j2mmIcTfNLFMX33Off55kU27Qkz7DLOtmAQDIQxX1EvLKFr5aEVFLTH9aG3pek0c9JmaLGty0k3x+NWgmkAs6MICGK3tdY5ZZDr6bZtS1FheDNa9SdCXu96n5pbTrMlMEAIBxVdRLyCtb+K4zXOw7zBYfVE/v1KZ2rSunGo96JECuQJvQgQE0QFwxrbJvWnnOcoi61sWLQeEs56Ircd9zM3oI6GBnV2bBriXDkzMeflg6fnz6AmSPPy4dOxa05dix4PusmjJTBAAwu3zZoor/Gc7rvui7Tljsezxb/IduT793eVvmRkdYhj+b48elRx4ZzRanTk2XA8L3OH48aIfZdFmFXIFWiSqMUfcHxbbQJkkKL00qGllHmdvsqVT1VXVji3VNU6zqzJnoa545k+16wKwRRTyBRpmULZqYK5zLp91nzgSvn1QE1Cz753LlinMLC0evubjYnM8aKJovWzADA6i5JMW08l7XWPT2aZO2+4oVMTT0vJb004ofGppmlcnmZrrj0yhr6zoAQHtNyhZNyxXhe2TOFkPXuHAheP0kzmXPFevr0osvHj1+61YxK2LJFpgldGAANU6/dnsAACAASURBVFd2Ma2oAPDoo8HUxrxufJOCU+yNdmie5IFM2+rqndrUBzU5XWX9zHxVzdNUO08ij/AFAMAkZWaLMnKFNGW2GLpGks6LUNbPK+51ef8OyBaYNebS/CutidXVVbe1tVV1M4BSrKwEN5tx3W4wKlLW+w1bXAxqVWQdkZmbiw4IZsHa1LW10RCytBS9ltN3HZ+sn9mxY9GdFfPz0u3b6a/nU/bvGsiLmT3pnFutuh1ZkSvQNmXeb5LkCrOg0Ob589nfJ49sUVauiPtM8v4dkC3QVL5swQwMoObKLqaVpOf/1i3p7Nns7xFX4TzNCMpciv+CZfnMwvfyzbRYW0t3vUmmGRFjeigAIKkys0WSe5hzwdKNae5deWSLNJ0XWT+vfl967rno5xYX8/8dkC0wa+jAAGqu7MrSSbdJ29vL/h5xwSnuRjs+DTLpEo4sn9nwe42bn5fOnJlupChK1q3rmB4KAEijzGyRNFdMU1NCyidbJDU/n+3zCt8rKkN1OtPNbvUhW2DmRFX2rPuDauFAPoardXc6wSOsrD2p+nZ43nCV7zTVv33nejYZcd2u/7n5+fg2ZvlcfNfsdtNfL837TtpxJkrcZwaUQexCAmAgvL8P35+T5orxbJF2V5E8s8WkNmb5XMgWQHK+bEENDKClwp718SmVIbPgVjU3F1Qhj7O0JJ0+LV26lKx2Rdp2hdc5dcq/vnV5OXr0pNORbtyY7v3HFfmfzX4/GIHa3Q1+po2NyZ9f3LrfSb87IA/UwAAgxd9Dw1zR6SSbxbm4GJw/vFtHllzha9c02SJt/QiyBZAeNTAAjIhaDzrMueAG/Ru/IS0sxF9rfz8IAZO2e00iblpr3DTIjY3odj77bLrpjpM+l/n55NfKIsvWdVmnhwIAkKe4e2iYK27cCJZhmsVf69ato1uNZt0SPWu2OHky+jnfcR+yBZAfOjCAlkpSvGl3N7jJPfHE4U3fx1ePIst2YL4bbdz61l5Puvfeo9eK21M9qjjVpPb6fs4qC12VXegVAIAok+6h4fPnzwc7g0zKFlnewydLtrh6NfpavuO+LJAlW1RdQJNsgbqiAwNoqSQ96OE5wzf9bjf6XN/owdxcfjfdSUXHbt6Mfl1UcPAVp7rvvvg2RP38VRS6Gg426+vBEp6yCr0CABBlUrYYfj5JtoiSZ64I2+HLFml28IjLApM+l/Gfv6oCmmQLNEJUYYy6Pyi2BUwvqqhTkgJPvmJQZ874r5ekWFQe0hSc8p3b6aT/OcoudJW1IBdQFFHEE4CLzxZx96mo1y0uOrew0PxcERYTTfO5VFFAk2yBuvFlC2ZgAC01PuLQ6QSPSb3svpGK8+eDr1EzMbKuWU0rzXRH36jKzZuHP590+PPEfSbT7LGexaT97AEAqMJwRpCS3UPHXxdmi4sXgyWsTc8V4XLcNJ9L2blCIlugOdiFBECuJlatzlIKO4Wklz9+PLoKetrK4lIw3TKPKuVJURkcdcMuJACKUvU9rw25Qqr+cwbGsQsJgML1+8ENMMryskpZ1Jmk0na/H+xOMm5hIVtxqjwKXaUp1kVlcABAG0zMFSVoaq4I20W2wKyhAwNALsK+iahK2i/ddGsyP3F9PdidZNy992abDDKpuOgkaft1qAwOAJh1iXJFTdQtV0hkC8wulpAAyIVvuuP8vHTp0uCmm8P8xDxWoNRtmmSWqaIFr8QBUmEJCYC8JcoVOZn2nlq3XCGRLdB8vmxBBwaAXCS6eU+5qDMcTRiexLG0lH5Uooq1pXHqGHyANOjAAJC3su6NeWSLuuUKiWyB5qMGBoBCJVo7OeX8xLxWoEw7TTLNmtIk16h6fS8AAHVTVk2GPLJF2bWwklyDbIFZRQcGgFxE3bwXFqTnnhu6GWu6RZ15bSs2zdrSPOqQjl+jCet7AQAok69T4OTJ6f9Hf1ge2aLsWlhJrkG2wKwqvAPDzN5mZl8ys6fN7L0Rz3+Lmf3m4Pk/NLOVotsEIH/jN+9OJ/i6tzd2M1aCct4R8q5EnrSq+HhIymOkJuoaUrCuN2uxLqAtyBVAO0R1Cpw+HdS/yGMzs/Ae71tNnzZbJMkVw+9LtgCyKbQDw8zmJf2qpLdLeoOkd5jZG8ZO+wlJf+Gc+3ZJvyTpnxTZJgDFGC/8JB2tyJ11w5EiK5H7pmz6RkOi1rhK6UZqfOceHKTu1wFahVwBtEdUQcmrV/NZSjp8j49CtgDq61jB139A0tPOua9Ikpl9SNKDkv5k6JwHJf3s4M//UtKvmJm5JlYXBVpqvACW70YspV/uIcWPKkwzmhDV7rU1/3vu7wfvGdWRkmakZnk5+jNiXSowEbkCaAHf/TkqC0jps4UvV0jBTIVpdt8gWwDFKnoJyWskfW3o+2uDY5HnOOduS/pLSZ3xC5nZmpltmdnW9evXC2ougCzigsC4LDfSuFGFpEs/op5/+GH/SI7vPe/cmb5QF3utA5mRK4AWiPsf/Shps4XvHm8WPVMhSYFNsgVQjqI7MCzi2PgISJJz5JzbdM6tOudWT5w4kUvjAEwnvFnHzbgYlvVGmqYS+aRCWJOmjUqjy2DGhWtIsxbqkqYv9gW0GLkCmGGTckUe/6Mv5Zsrxs/xIVsA+Si6A+OapNcNff9aSc/4zjGzY5L+O0k3C24XgCkluVl3OvncSNOMKkwqhJVktki41tb3nkkLdcXJ4xpAC5ErgBmVJFfk8T/6Ur65wnfOOLIFkI+iOzA+K+l+M3u9mS1KekjSx8bO+Zik04M//6ik32WdKlB/k27WS0vSuXP53Ujvuuvwz52OP7BM2g5t0jrZ4SDBSAZQO+QKYEYlyRV5/Y++lF+uiDsnRLYA8lNoB8Zg7el7JH1C0hclfdg59wUz+zkz+7uD035NUsfMnpb0DyQd2RINQP3E3azDG7KUba/24bWmx49Ljz4abMcaeuEF/2snTQuNWyc7HiQYyQDqhVwBzK4kuaLXS1aPIkr4OjPp1Kn8ckXcOeNtl8gWwLSsiYMSq6urbmtrq+pmAK3mW6Pa7QY35PEq3FIwAjFppCHqdVHC90ny+uH3zdouAH5m9qRzbrXqdmRFrgCqNylXSMVmi6y5Ypp2AfDzZYuil5AAmFGT1o8mWTMa5ezZZDuaDI/UDI/GrK9Lp0/7p2cyfRMAgPpJUpeiyGwxPgMkzBanTgXLTTodf24gWwDlOVZ1AwA0U3hTDrcFC4tThceTrBkd1++PTumME07XjNpv/dKl+ODQ6xEqAACok0m5Qio2WwwvAxnPFnt7QWfK5ctkC6BqLCEBUIgkU0GTvmbc8LTMLO8DIF8sIQFQhqKyxfhyD7IFUD2WkAAtlrXg1TTSbFEWihtB8U3dzDIaAwAApjML2UKKXu5BtgDqiyUkwIyLWmKxthb8ucipjkmmgo5bXo4e8eh0pBs30r0mriI4AADIjmwBoCrMwABmXNaCV3lIu1WYb2Tl3Ln0r4kbjQEAANmRLQBUhQ4MYMY1aRpklireVP4GAKBcZAsAVaGIJzDjKEQFoGgU8QTahWwBoGgU8QRaimmQAAAgT2QLAFWhAwOYcUyDBAAAeSJbAKgKHRhAC6QteFW0KrZeAwAA+alTtiBXAO3BNqoASlXV1msAAGD2kCuAdmEGBoBSVbn1GgAAmC3kCqBd6MAAUKombb0GAADqjVwBtAsdGABKtbyc7jgAAIAPuQJoFzowAJSKrdcAAEBeyBVAu9CBAaBUbL0GAADyQq4A2oVdSACUrtcjWAAAgHyQK4D2YAYGAAAAAACoPTowAAAAAABA7dGBAQAAAAAAao8ODAAAAAAAUHt0YAAAAAAAgNqjAwMAAAAAANQeHRgAAAAAAKD26MAAAAAAAAC1RwcGAAAAAACoPTowAAAAAABA7R0r4qJm9n9J+t8k3ZL0XyU94pz7bxHnbUt6VtIdSbedc6tFtAcAADQb2QIAABQ1A+NTkv66c+5/lPT/SfqpmHO/zzn3JgIGAACIQbYAAKDlCunAcM590jl3e/DtH0h6bRHvAwAA2oFsAQAAyqiB8aik3/Y85yR90syeNLO1uIuY2ZqZbZnZ1vXr13NvJAAAaIypswW5AgCA5slcA8PMPi3plRFPrTvnPjo4Z13SbUl9z2W+2zn3jJm9XNKnzOxPnXOfiTrRObcpaVOSVldXXdZ2AwCAeiozW5ArAABonswdGM65t8Q9b2anJf0dSd/vnIsMBs65ZwZfv2FmH5H0gKTIDgwAADDbyBaAR78vra9Lu7vS8rK0sSH1elW3CgBKV8gSEjN7m6R/KOnvOuf2PefcbWYvC/8s6a2SniqiPQAAoNnIFmitfl9aW5N2diTngq9ra8FxAGiZompg/IqklymYuvk5M7sgSWb2ajO7OjjnFZJ+38w+L+mPJP2Wc+7jBbUHAAA0G9kC7bS+Lu2P9dnt7wfHAaBlMi8hieOc+3bP8WcknRz8+SuSvqOI9wcAALOFbIHW2t1NdxwAZlgZu5AAAAAAyGJ5Od1xAJhhdGAAAAAAdbWxIS0tjR5bWgqOA0DL0IEBAAAA1FWvJ21uSt2uZBZ83dxkFxIArVRIDQwAAAAAOen16LAAADEDAwAAAAAANAAdGAAAAAAAoPbowAAAAAAAALVHBwYAAAAAAKg9OjAAAAAAAEDt0YEBAAAAAABqjw4MAAAAAABQe3RgAAAAAACA2qMDAwAAAAAA1B4dGAAAAAAAoPbowAAAAAAAALVHBwYAAAAAAKg9OjAAAAAAAEDt0YEBAAAAAABqjw4MAAAAAABQe3RgAAAAAACA2qMDAwAAAAAA1B4dGAAAAAAAoPbowAAAAAAAALVHBwZK0e9LKyvS3Fzwtd+vukUAAKCpyBUA0E7Hqm4AZl+/L62tSfv7wfc7O8H3ktTrVdcuAADQPOQKAGgvZmCgcOvrhyEjtL8fHAcAAEiDXAEA7UUHBgq3u5vuOAAAgA+5AgDaiw4MFG55Od1xAAAAH3IFALQXHRgo3MaGtLQ0emxpKTgOAACQBrkCANqrsA4MM/tZM/szM/vc4HHSc97bzOxLZva0mb23qPagOr2etLkpdbuSWfB1c5NCWwCA5MgVCJErAKC9it6F5Jecc/+370kzm5f0q5J+QNI1SZ81s4855/6k4HahZL0ewQIAMDVyBSSRKwCgrapeQvKApKedc19xzt2S9CFJD1bcJgAA0EzkCgAAZljRHRjvMbP/bGYXzezbIp5/jaSvDX1/bXDsCDNbM7MtM9u6fv16EW0FAAD1Rq4AAKDFpurAMLNPm9lTEY8HJb1f0n8v6U2Svi7pF6MuEXHMRb2Xc27TObfqnFs9ceLENM0GAAA1RK4AAABxpqqB4Zx7S5LzzOwDkv5txFPXJL1u6PvXSnpmmjYBAIBmIlcAAIA4Re5C8qqhb39Y0lMRp31W0v1m9nozW5T0kKSPFdWmqvT70sqKNDcXfO33q24RAADNQq4YRbYAALRRkbuQ/IKZvUnB1M1tSe+SJDN7taR/7pw76Zy7bWbvkfQJSfOSLjrnvlBgm0rX70tra9L+fvD9zk7wvUT1bAAAUiBXDJAtAABtZc5FLg2ttdXVVbe1tVV1MxJZWQmCxbhuV9reLrs1AADkz8yedM6tVt2OrJqUKySyBQBg9vmyRdXbqM683d10xwEAAOKQLQAAbUUHRsGWl9MdrxpragEAqDeyBQCgrejAKNjGhrS0NHpsaSk4XjfhmtqdHcm5wzW1BA0AAOqDbAEAaCs6MArW60mbm8G6VLPg6+ZmPYtsra8fFgQL7e8HxwEAQD2QLQAAbUURT7xkbi4YHRlnJh0clN8eAEAzUMQTPmQLAEAWFPHERE1bUwsAAOqNbAEAyBMdGHhJk9bUAgCA+iNbAADyRAcGXtKkNbUAAKD+yBYAgDzRgTEj8tqirNeTtreDdanb2wQMAADaKM+tT8kWAIC8HKu6AZheuEVZWOU73KJMIiQAAIB0yBUAgLpiBsYMYIsyAACQF3IFAKCu6MCYAbu76Y4DAAD4kCsAAHVFB8YMqOsWZXmunwUAAOWoa66QyBYA0HZ0YMyAOm5RFq6f3dmRnDtcP0vQAACg3uqYKySyBQCADoyZUMctylg/CwBAM9UxV0hkCwCAZM65qtuQ2urqqtva2qq6GYgxNxeMjowzC7ZRAwDMDjN70jm3WnU7siJXNAPZAgDaw5ctmIGBqUWtR63z+lkAAFBvZAsAQBQ6MFom7+JXvvWoJ0/Wc/0sAADITxFFNckWAAAfOjBapIjiV771qFev1nP9LAAAyEdRRTXJFgAAH2pg1ES/H9ywd3eDqZAbG/nfkI8fl/b2jh7vdqXt7WzXZD0qAIAaGPVUdLYoIldIZAsAADUwaq2MbcH6/eiQIQXBJivWowIAUD9FZ4uicoVEtgAA+NGBUQNlbAsWd61pAkFd94oHAKDNis4WReUKiWwBAPCjA6MGfCMV48fDQllm0rFjwdekBbPiRkOmCQR13SseAIA2S5It6pgrJLIFAMCPDowaSDJVcngqqCTduRN89U0JHa8Kft990e/R6UwfCHq9YK3rwUHwlYABAEC1JmWLtLkifE2YLeY8CTKPXCGRLQAA0ejAqIEkUyWjpoKG9velhx8+HDWJWve6txeMroy/x7lzR69XxJZoAACgPJOyRZpcIR3NFmGHx/j1o3JF+HqyBQBgWnRg1ECSqZJJCmKFoyZnz0aHktu3g5GRuOmYcUW/CB8AADTDpGyRJleEu5n4Ojyk+GUeZAsAQF7YRrUhVlYOp3lOo9ORbtxI/z6djvTCC6PhZWmJNakAALZRbaI0uaLbDTo8fJGRbAEAyBvbqDZc1FTQLHxbnoV8IzJ7e8XvlAIAAMqRJlfs7sbvLEK2AACUpZAODDP7TTP73OCxbWaf85y3bWb/ZXBeu4Y+UhqeCipJ8/P+czud7O+Tduuzafd6BwAgCbJFvtLkiuXl6XYWIVsAAPJSSAeGc+5/d869yTn3Jkn/StK/jjn9+wbnNnbqaVnCitzOBfUsrlyJLtB17px0zz3R1xju3Ihad+obkfFVG592r3cAAJIgW+Qvaa7Y2AjOjc0Wg1DhbE7Xjq2oZ/3YbGEm3X139PXIFgAAn0KXkJiZSfpxSR8s8n3a7K67Dv/c6RyuG71wQVpcHD13cfGwOrivoJYUXGN8FsfBwdH3Ht8pBQCAopEtitPrSadPH87GmJ8Pvg/rUfiyxb/58cNQYXJ67Z0dbWpN/9NO/6Vscfp00GkRck568UVpYWH0emQLAECcomtg/M+S/tw592XP807SJ83sSTNbi7uQma2Z2ZaZbV2/fj33hjZN2AExvO70hRekf//vg9kUp05JL3vZ6K4jFy8ehpCoauL7+0HAkPyjLPPz8buYAABQsFyyBbniqH5funTpcIvUO3eC7x9/PD5bfM/Vo6Hibu3rfVrX/n6wO9rVq0eLgN66Jd17b/wubAAADMu8C4mZfVrSKyOeWnfOfXRwzvslPe2c+0XPNV7tnHvGzF4u6VOSftI595lJ793GauHjklYP91XznpvzVxNfWvJvlWYWPRsDANBeee1CUlW2IFcEjh+fXJBTisgWnlBxINO84kMDuQIAECX3XUicc29xzv31iEcYMI5J+hFJvxlzjWcGX78h6SOSHsjanrZJuvVZOPIxLm596f6+v5jX+OvYvx0AkBeyRXX6/WSdF1JEtvCEil0dHidXAADyUOQSkrdI+lPn3LWoJ83sbjN7WfhnSW+V9FSB7ZkpcdXCx711r6/njq+MpIFJ26fduXN0nev4ulRfHQ3CBgCgIGSLgqTdunRvb+h+HxEqnteSflqHoYFcAQDIQ5EdGA9prMCWmb3azK4Ovn2FpN83s89L+iNJv+Wc+3iB7WmcqFGI8Fi4PnWSd6ivD2hN9+yNpoGe+trcjO8IcW50nev4UhRfHQ32bwcAFIRsMSVftkg6s3PYS/f7oT1ZD2TaVlfv1KY+qNH1q+QKAMC0MtfAqFIb1qqGoxDDN/LFxcOq3Ul9VStaUUQq6Xal7e3I94k4LZKvjgbrWQGgXfKqgVGVNuQKKb9sEYq635MrAAB5yL0GBooVNQpx61b6gLGs3egndoPj4aCJz67n5ZK/jgb7twMAUD95ZYtQ1P2eXAEAKBIdGDUVd4NPdR1NTgO9XjAiMuG0I6LqaLB/OwAA9ZRXtpDi7/fkCgBAUejAqKk0ow3drj8o/J+2oduLk9NAltAQjrJ0OofH7rorebsBAEB50mSLuBpZUfUrxk2TK8JMMz9/WAODQp4AAIkOjNqKuvEvLkoLC6PHwjDgCwpvv9zTsYuDNDBUNauv3kgRL+kwNPiKa/m88MLhn/f2qBgOAEAdpckWa2vRueLKlaCGxXg+GC8OKmXLFb3eYTvDguXsRgIACNGBUVPDoxDhjf/iRemJJ6LDQOxsiF4vSBsHB0HhTvUitymTgtMuXw7+fOrU5D3YqRgOAEAzpMkW588nnw3h2/5USp8rJLIFAMCPXUhmSL8vPfpoUJArtLgYhJPhEY+Vlejt0rrdYNRjvHr40pJ/1ISK4QAAdiGZXY8/Ll24MHqvH88FeeYKiWwBAGAXkpk0Pl3zXe8a7byQgu/Pnh095ivitbubftSDiuEAAMyG8VwR1XkhHc0FeeYKiWwBAPCjA6Mg4yEg73WbUdM1n38++ty9vdHv44KBL4Ts7ET/LNNWDC/6cwIAYFYUec+MyhVRnReh4byQNVf4fhayBQDAhw6MAvjWguZ5A40a0UgqLhjEjW5E/SxR62mTFv8s43MCAGAWFH3PjMoVcauMh/NCllxh5v9ZyBYAAB9qYBQgbi3o9nY+7+FbHxql05Fu3Bg91u8HYWV3NwgXGxtBMAhv/JM6R/L4Wcr4nAAAxaMGRvGKvmemyRVmQWHO4Q6FNLnCLPq9yBYAgJAvW9CBUYAyik/5btDjFhaC6uJJRi1CwyHE99cjj5+FIl0AMBvowChe0fdMX64Y72wwkx57LNilJKnxzg1ffiFbAABCFPEsURnFp3zTNc+cGZ1ymabzIlwzeupU8P3ly4fbp42bm5t+bSlFugAASKboe6YvVzz22GiuuHx5us6LjQ2yBQAgOzowCjBt8akkfOtDz58PpkgeHARf03ReRK0ZPXny6M8iSXfuTL+2tIzPCQCAWVD0PTPvXCHFZ4uFhaPnky0AAJOwhKQgvrWgdRW3ZvTkyfhK5OF5WdaWNu1zAgAcxRKScjTtnunLFp2O9OyzR7d+H0e2AID2ogYGYsWtGY1brzp8HmtLAaCd6MBAlDSFQaOQLQCgvaiBgVhZ9nBP8noAANBO02YDsgUAYBwdGDUQFs+ctnDVNLLs4T5+HgAAqIc6Z4tOZ/JryRYAgCh0YBRsUoDwFbgqK2gM7zxy111BqBgu3tXrRQcQs+Dr8HkAAKB4Tc8W584dzRWLi9EZBACAYdTAKFAYIPb3D48tLY3elOOKZ2YpXJV3+4bPpSAWACAKNTDKMyvZglwBAIhDEc8KJAkQccUziy5cVWXAAQDMDjowykO2AAC0AUU8K+Arfjl8PK54ZtGStA8AANQH2QIA0GZ0YBQoSYCIK55ZtCoDDgAASI9sAQBoMzowChQVICTpuecOC2n1esG60G63/MJVcQGnDtXLAQDAKF9h7ZMnD78nWwAAZhU1MArW70tnz0p7e6PHfcUyyxZVREtKXtwTAABqYJTr8celCxdG61zU6T5NtgAATIsaGDGKHBHo9aR77jl6fH9fOn26+tGHXi8oqnVwEHzt9YLQMRwwpOD79fUqWggAQPMUmS2uXj1apHN/X3r44XrMbCBbAACKcqzqBlRtfLuvcK90Kb8RAV/hqjt38n+vPFCACwCA7IrOFnH34yJyTB7IFgCAPLR+BkYZIwJxhavqOPpAAS4AALIrOltMuh+TLQAAs6r1HRhljAj4inkW8V55qLJ6OQAATVd0tpiUK/J8r7yQLQAAeZiqA8PMfszMvmBmB2a2OvbcT5nZ02b2JTP7Qc/rX29mf2hmXzaz3zSzxWnak0UZIwJhNfD5+eLfKw9VVi8HALQb2WKy4ft02jZUhWwBAMjDtDMwnpL0I5I+M3zQzN4g6SFJb5T0NknnzSzqf9//iaRfcs7dL+kvJP3ElO1JrawRgV5PunSpOaMPUQW4AAAoAdkigfA+feUK2QIA0B5TdWA4577onPtSxFMPSvqQc+6vnHNflfS0pAeGTzAzk/S3Jf3LwaFLkn5omvZkUeaIQJHvxd7qAIBZQLaoz3uRLQAAdVPULiSvkfQHQ99fGxwb1pH035xzt2POeYmZrUlak6TlnOdF9nrljQIU8V5l7KQCAEDFcs0WReYKiWwBAEARJs7AMLNPm9lTEY8H414WccxlOOfwCec2nXOrzrnVEydOTGp2q7C3OgCgSeqQLcgV8cgWAIA6mjgDwzn3lgzXvSbpdUPfv1bSM2Pn3JD0rWZ2bDBSEnUOEmBvdQBAk5At6o9sAQCoo6K2Uf2YpIfM7FvM7PWS7pf0R8MnOOecpH8n6UcHh05L+mhB7Zlp7K0OAGgBskWJyBYAgDqadhvVHzaza5K+S9JvmdknJMk59wVJH5b0J5I+Lundzrk7g9dcNbNXDy7xDyX9AzN7WsG61V+bpj1txd7qAIBZQbaoB7IFAKCOLBisaJbV1VW3tbVVdTNqpd8P1qXu7gajIxsbFNkCAJTDzJ50zq1W3Y6syBXRyBYAgKr4skVRu5CgZGVWOwcAALOPbAEAqJuiamAAAAAAAADkhg4MAAAAAABQe3RgAAAAAACA2qMDAwAAAAAA1B4dGAAAAAAAoPbowAAAAAAAALVHBwYAAAAAAKg9OjAAAAAAAEDt0YEBAAAAAABqjw4MAAAAAABQe+acq7oNqZnZdUk7Vbdj4LikG1U3Aonx+2oOflfNwe+qWYr4fXWdcydyvmZpapYrJP5NNQm/q2bh99UcHBxwUQAAA/VJREFU/K6ao6jfVWS2aGQHRp2Y2ZZzbrXqdiAZfl/Nwe+qOfhdNQu/r/rjd9Qc/K6ahd9Xc/C7ao6yf1csIQEAAAAAALVHBwYAAAAAAKg9OjCmt1l1A5AKv6/m4HfVHPyumoXfV/3xO2oOflfNwu+rOfhdNUepvytqYAAAAAAAgNpjBgYAAAAAAKg9OjAAAAAAAEDt0YGRkZn9mJl9wcwOzGx17LmfMrOnzexLZvaDVbURR5nZz5rZn5nZ5waPk1W3CaPM7G2DfztPm9l7q24P4pnZtpn9l8G/p62q24NRZnbRzL5hZk8NHbvPzD5lZl8efP22KtuIALmiucgW9Ue2aBayRX3VIVfQgZHdU5J+RNJnhg+a2RskPSTpjZLeJum8mc2X3zzE+CXn3JsGj6tVNwaHBv9WflXS2yW9QdI7Bv+mUG/fN/j3xH7t9fPrCu5Fw94r6Xecc/dL+p3B96geuaLZyBY1RbZoLLJFPf26Ks4VdGBk5Jz7onPuSxFPPSjpQ865v3LOfVXS05IeKLd1QGM9IOlp59xXnHO3JH1Iwb8pABk45z4j6ebY4QclXRr8+ZKkHyq1UYhErgAKQ7YAclKHXEEHRv5eI+lrQ99fGxxDfbzHzP7zYAoUU6frhX8/zeMkfdLMnjSztaobg0Re4Zz7uiQNvr684vYgHv9dbAayRX3xb6h5yBbNUmquOFbkxZvOzD4t6ZURT6075z7qe1nEMfaqLVHc703S+yX9vILfyc9L+kVJj5bXOkzAv5/m+W7n3DNm9nJJnzKzPx30zgMYQ65oLrJFo/FvqHnIFvCiAyOGc+4tGV52TdLrhr5/raRn8mkRkkj6ezOzD0j6twU3B+nw76dhnHPPDL5+w8w+omCqLiGj3v7czF7lnPu6mb1K0jeqblBbkCuai2zRaPwbahiyReOUmitYQpK/j0l6yMy+xcxeL+l+SX9UcZswMPhHFfphBUXTUB+flXS/mb3ezBYVFK77WMVtgoeZ3W1mLwv/LOmt4t9UE3xM0unBn09L8o38ox7IFTVHtqg9skWDkC0aqdRcwQyMjMzshyX9sqQTkn7LzD7nnPtB59wXzOzDkv5E0m1J73bO3amyrRjxC2b2JgVTB7clvava5mCYc+62mb1H0ickzUu66Jz7QsXNgt8rJH3EzKTgfvIvnHMfr7ZJGGZmH5T0vZKOm9k1ST8j6R9L+rCZ/YSkXUk/Vl0LESJXNBrZosbIFo1DtqixOuQKc44lYAAAAAAAoN5YQgIAAAAAAGqPDgwAAAAAAFB7dGAAAAAAAIDaowMDAAAAAADUHh0YAAAAAACg9ujAAAAAAAAAtUcHBgAAAAAAqL3/H+oJ1ksvbEtZAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "T = 6\n", "optim = Adam({\"lr\": 0.05})\n", "svi = SVI(model, guide, optim, loss=Trace_ELBO())\n", "losses = []\n", "\n", "def train(num_iterations):\n", " pyro.clear_param_store()\n", " for j in tqdm(range(num_iterations)):\n", " loss = svi.step(data)\n", " losses.append(loss)\n", "\n", "def truncate(alpha, centers, weights):\n", " threshold = alpha**-1 / 100.\n", " true_centers = centers[weights > threshold]\n", " true_weights = weights[weights > threshold] / torch.sum(weights[weights > threshold])\n", " return true_centers, true_weights\n", "\n", "alpha = 0.1\n", "train(1000)\n", "\n", "# We make a point-estimate of our model parameters using the posterior means of tau and phi for the centers and weights\n", "Bayes_Centers_01, Bayes_Weights_01 = truncate(alpha, pyro.param(\"tau\").detach(), torch.mean(pyro.param(\"phi\").detach(), dim=0))\n", "\n", "alpha = 1.5\n", "train(1000)\n", "\n", "# We make a point-estimate of our model parameters using the posterior means of tau and phi for the centers and weights\n", "Bayes_Centers_15, Bayes_Weights_15 = truncate(alpha, pyro.param(\"tau\").detach(), torch.mean(pyro.param(\"phi\").detach(), dim=0))\n", "\n", "plt.figure(figsize=(15, 5))\n", "plt.subplot(1, 2, 1)\n", "plt.scatter(data[:, 0], data[:, 1], color=\"blue\")\n", "plt.scatter(Bayes_Centers_01[:, 0], Bayes_Centers_01[:, 1], color=\"red\")\n", "\n", "plt.subplot(1, 2, 2)\n", "plt.scatter(data[:, 0], data[:, 1], color=\"blue\")\n", "plt.scatter(Bayes_Centers_15[:, 0], Bayes_Centers_15[:, 1], color=\"red\")\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "DO3XDmEwC5ra" }, "source": [ "The plots above demonstrate the effects of the scaling hyperparameter $\\alpha$. A greater $\\alpha$ yields a more heavy-tailed distribution of the weights, whereas smaller $\\alpha$ will place more mass on fewer clusters. In particular, the middle cluster looks like it could be generated a single Gaussian (although in fact it was generated by two distinct Gaussians), and thus the setting of $\\alpha$ allows the practitioner to further encode their prior beliefs about how many clusters the data contains." ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "oWBlcZ1vfiBV" }, "source": [ "### Dirichlet Mixture Model for Long Term Solar Observations \n", "\n", "As mentioned earlier, the Dirichlet process mixture model truly shines when exploring a dataset whose latent geometric structure is completely unknown. To demonstrate this, we fit a DPMM on sunspot count data taken over the past 300 years (provided by the Royal Observatory of Belgium):" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 295 }, "colab_type": "code", "id": "UlhV2Fx_fVGy", "outputId": "045dbfaa-27a0-4aa6-ee8c-de9f9937aa38" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEWCAYAAABhffzLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAc60lEQVR4nO3debgcVZnH8e8PwiL7koABEgKIS1gMeA0gCAwyKGsAmWEbBETj48AAo+NDxAVcZgAFREcUw7AOmwyLoIACkcCDDmDAQBIygYiRJTGLIAmBhCXv/HHOleLmLn2X6r7d9fs8Tz+3+lRX1Xu6+r59+lTVKUUEZmZWHas0OgAzM6svJ34zs4px4jczqxgnfjOzinHiNzOrGCd+M7OKceKvMElXSvp2g7YtSVdIeknSI42IwayqnPgHEUlzJM2XtHah7DOSJjcwrLLsAfw9sEVEjC3OkLSdpJclvbdD+SRJ59QzyHqQtIOke/KX4EuSpkj6eKPj6oqkByWd0MNr1pD0TUmzJS3Nn+3/kjSy5Nj2lTSnzG20Aif+wWcIcFqjg+gtSav2cpEtgTkRsbTjjIiYAZwPXCZJef0nAZsD3+hvrEWShgzk+vqwfQG/AO4ENgHeDfwr8Eoj4+qPXKdbgP2BI4H1gTHAE8A+DQzN2kWEH4PkAcwBJgAvAhvkss8Ak/P0KCCAIYVlJgOfydMnAL8Bvgf8FXgG+Egufw5YABxfWPZK4BLgHmAJcD+wZWH++/O8F4FZwD92WPbHpIS1FNi3k/psBtyel58NfDaXnwQsA94iJbhvdLLsEOD3wMnApsAi4COF+aOBe/O6/w/4ZGHeIcDUXKdnga8V5r0nv4cn5nm/BtYCrgP+kt+3R4ChncT0VeCGDmUXAxcW6jUnb/cZ4Kga9vm7czzrdDH/b/u/8L4EMCo/vwb4AXBX3u7/AlvleavkeQuAl0mJd3RhuYuBSXm5+4ARhe3sAUzJyz0C7JLLz8v7bVnedxd1EvMngFeBzbup9xakL7wXgaeBTxfmXQOcXXi+L6mR0P78eeALwLQc3/XAGqQvmNeAFTm2V0hfprsCjwGLgfnAdxv9v97oR8MD8KOwM1LS2JfUWvp2Lutt4n8zJ7VVgW/n5HZx/sfYL/+Tr5Nff2V+vmee/33gwTxvbdKXxYk52exMSr7bFZZ9Gdg9J5g1O6nP/cCPgDVJLb6FwMcKsT7Yw/uxU04M9xQTDLAu8ALwqRzbh0hJ+315/j7A9jmuD+a4D8rz2hP/FaSE/y7Sl8vP8vSqQBudJGJg65xM1s7Ph5CSahuwXn4/ts3zhpOTbA91XAX4A3AbMA7YpMP8WhL/ohzDasBPgWvyvANJSXv9vJ3RwLsLy7XvvzXyZ6T9czY0zzs6b++f8vu7YZ7/IHBCN3U6H5jUQ71/A/xn/my0f7b2KsR2duG1nSX+h0hfmhsDT/H2/8A7XpvLfgccXfjs7NLo//VGP9zVMzh9HfgXScP6sOwfI+KKiHiLlARGAN+MiOURcTfwOin5tbsjIh6IiOXAV4DdJI0ADiL9A10REW9GxGPAzcARhWVvi4jfRMSKiFhWDCKvYw/gjIhYFhFTgf8Cjqu1IhHxe+Ay4APAmYVZhwBPRcTVObZHSYn7iLzcryNieo7rceAGYK8Oqz8rIl6NiNeAN0jJ7j0R8VZETImIlbpaIuIZYDopQUM6RvHXiJjS/hJge0lrRsS8iHiyhjquAPYmfZF9D5gn6T5J2/S0bMFNOeY3gGtJX7Lkeq1H+uVGRDwZEX8uLPfzvP+Wk97fPSUNBw4GZkTE9fn9vYb0C+bAGuPZGJjX1UxJWwFjgQn5s/EY6Yu45s8GqSHw54j4C+mXw5huXvsGsK2kjSNiSUQ83IvttCQn/kEoIqaTPswT+rD4/ML0a3l9HcvWKTx/rrDdV0gt7M1IffC7SPpr+wM4ltTKWmnZTmwGvBgRSwplfyL10/fGDNIX0KuFsi2B3TvEdiSplY2k3SRNlrRQ0sukVvPQDustxn4lqdvoRkkvSDq3m77/60gtYYBjSImWiFicy08G/izpFx0PTnclIp6LiH+OiK2BrUiJ6spals2KyfxV8v7NX/SXkLrk5ku6RNK6hdcW9/3LpFb+Zvnxpw7b6M2++wt5X3RhM2BRvPP4Tm8/G53WuQsnkn7tzJL0iKQDerGdluTEP3idBXyWd/4ztP+jrFUoKybivhjRPiFpHWAjYC4pKdwfERsUHutExOcLy3Y3tOtcYKMOiWYkqWXbX8+RuhI6xnZKnn8D6dfJiIhYn/RLQ8UVREQUpl+PiLMj4gOkXymHkb7kOvNTYF9JW5Ba/tcV1nNXROxLSnqzgZ/0tmIR8Sype2z7XLSUfuzviLgoInbO6xtN6htvV9z365O6hObmx5YdVlXcdz0N6Xsv6ZfjZl3MnwsMLZ691mH9/anzSrFFxKyIOIrU338BcLOkNXuxzpbjxD9IRcRsUpI5tVC2kPTP8U+SVpX0aaA3XQKdOUDSHpJWB74FPBwRz5F+cbxX0nGSVsuPD0v6QI3xPwf8FjhH0pqSdiQd/Ly2n/FCOmC8naRjCrGNlfS+PH9d0q+NZZJ2BY7qbmWS9pG0vaRVSAcA3yAdwOysXvNJfdxXALMi4um8juGSDpa0Fqk7bWlX6+iw7aGSzpK0db62YRiphfpQfsnjwI75lM93kRoENcnvydj862VpjqsY08H519EapONBD0bEPNK+307SkZKGSDqG1D14Z15uPul4R1d+RTpYfKuknfJndT1J/yzp+Ij4I+nA8X/k0z7H5Dq3fzamAgdK2jB3PZ3a6VY6N5/0pfK3Bkf+DA/N3Wovk74cVvRinS3HiX9w+ybpIGvRZ4EvkX5Ob0dKrv1xHSmZvEg6SHosQO6i2Y+UNOeSflqfRzoQWKujSQek5wK3kvrV7+lnvO3dEh8nHXScl2M7pxDb50lfOEtIfdc39rDKzUgH1BeTupbuJZ0p0pXrSAcRryuUrUraL/NI++YjwCkAkvbO3VGdWU768r6PdOB4Wv776VzXJ4H/IB3EnwU80ENdijYgHSP5K+nEgXmk4wjtriEl/EXAjuQ+9tzAOAQ4I9flX0kHx1/My10EHJ272S7suNH8a+pw4G7gJtL7Oo3UD//r/LIjgW1J++4m4MyIuC/PuxKYSer++SXpF1xNcjfpzcCcHN8mwAHAzPx5OB84MiJer3WdrUiFX7xmVhGSrgFmR8TZjY7F6s8tfjOzinHiNzOrGHf1mJlVjFv8ZmYV09ABqmo1dOjQGDVqVKPDMDNrKo8++uiiiFhpBICmSPyjRo1iypQpPb/QzMz+RlLHK7ABd/WYmVWOE7+ZWcU48ZuZVYwTv5lZxTjxm5lVjBO/mVnFlJb4JY3IdxKaKWmGpNNy+dn5ZhdT86PyN0UwM6unMs/jfxP4YkQ8lsfGflRS+5C834uI80vctpmZdaG0xJ9v6DAvTy+RNJPe33bPzMwGWF2u3JU0CtgJeBjYHThF0qdId+H5YkS81Mky44HxACNHjiwttlET7ujzsnPOrfXe02Zmg0fpB3fzfVxvBk7PN6T+MemOQ2NIvwgu6Gy5iJgYEW0R0TZs2EpDTZiZWR+VmvglrUZK+tdGxC2Q7lkaEW/l+19eCowtMwYzM3unMs/qEel+nzMj4sJC+fDCyw4DppcVg5mZrazMPv7dSTdvniZpai47k3ST5jGkO93PAT5XYgxmZtZBmWf1PAiok1l3lrVNMzPrma/cNTOrGCd+M7OKceI3M6sYJ34zs4px4jczqxgnfjOzinHiNzOrGCd+M7OKceI3M6sYJ34zs4px4jczqxgnfjOzinHiNzOrGCd+M7OKceI3M6sYJ34zs4px4jczqxgnfjOzinHiNzOrGCd+M7OKceI3M6sYJ34zs4px4jczqxgnfjOzinHiNzOrGCd+M7OKceI3M6uYIY0OoJmNmnBHt/PnnHtgnSIxM6udW/xmZhXjxG9mVjFO/GZmFePEb2ZWMaUlfkkjJN0naaakGZJOy+UbSbpH0tP574ZlxWBmZisrs8X/JvDFiPgAsCtwsqTRwARgUkRsC0zKz83MrE5KS/wRMS8iHsvTS4CZwObAOOCq/LKrgEPLisHMzFZWlz5+SaOAnYCHgU0jYh6kLwdgky6WGS9piqQpCxcurEeYZmaVUHril7QOcDNwekQsrnW5iJgYEW0R0TZs2LDyAjQzq5hSE7+k1UhJ/9qIuCUXz5c0PM8fDiwoMwYzM3unMs/qEXAZMDMiLizMuh04Pk8fD9xWVgxmZrayMsfq2R04DpgmaWouOxM4F7hR0knAs8A/lBiDmZl1UFrij4gHAXUx+2NlbdfMzLrnK3fNzCrGid/MrGKc+M3MKsaJ38ysYpz4zcwqxonfzKxinPjNzCrGid/MrGKc+M3MKsaJ38ysYpz4zcwqxonfzKxiekz8ks6RtJ6kIZJ+JWm+pGPqEZyZmQ28Wlr8++c7Zx1EumnKdsAZpUZlZmalqSXxr5b/HgBcHxGLgCgvJDMzK1Mt4/HfIWk68BZwsqShwPJywzIzs7J02+KXtArpnrn7AB+KiDeAZcDhdYjNzMxK0G3ij4gVwPcjYkFEvJnLXomIF+oSnZmZDbha+vjvkTSu9EjMzKwuaunjPwVYX9Jy4DXSfXQjIjYqNbIBMmrCHY0OwcxsUKkl8Q8tPQozM6ubHhN/RLwlaX1gG2DNwqzflhaVmZmVpsfEL+kk4AvA5sA04MPAQ8DepUZmZmalqOXg7ulAGzAnIj4KfAiYV2pUZmZWmloS/7KIeA1A0uoRMQN4f7lhmZlZWWo5uDtP0gbAz4FfSXoRmF9uWGZmVpZaDu4ekie/JuljwPqAz5E0M2tStbT4kbQr8N6IuFrSxsCmwLOlRtbierq+YM65BzbduvuzXjOrn1rO6vkqsDvpdM6rSad0XgfsUW5oZmZWhloO7h5BGpJ5KUAep2e9MoMyM7Py1JL4l0dEkMfgl7RWuSGZmVmZakn8t0i6mDRez4nA3cDl5YZlZmZl6THxR8R5wC+A24EPAv8eERf1tJykyyUtyDdxaS87W9ILkqbmxwH9Cd7MzHqvy4O7kn4ETIiIxRFxF3BXL9d9JfBD0gHhou9FxPm9XJeZmQ2Q7lr8c4BHJR3TlxVHxAPAi31Z1szMytNl4o+I75AGYhsnaZKkIyQd3v7oxzZPkfRE7grasB/rMTOzPujp1osvkK7SfS9wcOFxUB+392PS9QBjSAO9XdDVCyWNlzRF0pSFCxf2cXNmZtZRd33825ES9VxgbET0e0TOiPjbGD+SLiUdNO7qtROBiQBtbW3R322bmVnS3ZW7NwGnRcTdA7UxScMLXyCHAdO7e72ZmQ287hL/mIhY3tcVS7qedIxgqKTngbOAvSWNIV0MNgf4XF/Xb2ZmfdNl4u9P0s/LH91J8WX9WaeZmfVfLVfumplZC+ky8UualP+eV79wzMysbN318Q+XtBdwiKQbABVnRsRjpUZmZmal6C7xfx2YAGwBXNhhXgD7lBWU+YYnHZV5cxmzqunu4O5NwE2SvhYR36pjTGZmVqJa7rn7LUmHAHvmoskR0eWFV2ZmNrj1eFaPpHOA04An8+O0XGZmZk2olputH0i6mGsFgKSrgN8DXy4zMDMzK0et5/FvUJhev4xAzMysPmpp8Z8D/F7SfaRTOvfErX0zs6ZVy8Hd6yVNBj5MSvxnRMSfyw7MzMzKUUuLnzyi5u0lx2J14msEzKrNY/WYmVWME7+ZWcV0m/glrSLJN0sxM2shPd1zdwXwuKSRdYrHzMxKVsvB3eHADEmPAEvbCyPikNKiMjOz0tSS+L9RehRmZlY3tZzHf7+kLYFtI+JeSWsBq5YfmpmZlaGWQdo+C9wE/CQXbQ78rMygzMysPLWcznkysDuwGCAingY2KTMoMzMrTy2Jf3lEvN7+RNIQ0h24zMysCdWS+O+XdCbwLkl/D/wP8PNywzIzs7LUkvgnAAuBacDngDuBr5YZlJmZlaeWs3pW5JuvPEzq4pkVEe7qMTNrUj0mfkkHApcAfyANy7yVpM9FxF1lB2dmZgOvlgu4LgD+LiJmA0jaBrgDcOI3M2tCtfTxL2hP+tkzwIKS4jEzs5J12eKXdHienCHpTuBGUh//PwC/q0NsZmZWgu66eg4uTM8H9srTC4ENS4vIzMxK1WXij4gT6xmImZnVRy1n9WwF/Aswqvh6D8tsZtacajmr52fAZaSrdVeUG46ZmZWtlsS/LCJ+0NsVS7ocOIh0VtD2uWwj4KekXw9zgH+MiJd6u24zM+u7Wk7n/L6ksyTtJmnn9kcNy10JfKJD2QRgUkRsC0zKz83MrI5qafHvABwH7MPbXT2Rn3cpIh6QNKpD8Thg7zx9FTAZOKOmSM3MbEDUkvgPA7YuDs3cD5tGxDyAiJgnqctx/SWNB8YDjBzZnPd6HzXhjkaH0FT8fpnVRy1dPY8DG5QdSEcRMTEi2iKibdiwYfXevJlZy6qlxb8p8H+Sfgcsby/s4+mc8yUNz6394XjoBzOzuqsl8Z81gNu7HTgeODf/vW0A121mZjWoZTz++/uyYknXkw7kDpX0POkL5FzgRkknAc+Sxv0xM7M6quXK3SW8fY/d1YHVgKURsV53y0XE0V3M+livIjQzswFVS4t/3eJzSYcCY0uLyMzMSlXLWT3vEBE/o4dz+M3MbPCqpavn8MLTVYA23u76sRbTrOfS9xT3nHMPrFMkZoNfLWf1FMflf5M0xs64UqIxM7PS1dLH73H5zcxaSHe3Xvx6N8tFRHyrhHjMzKxk3bX4l3ZStjZwErAx4MRvZtaEurv14gXt05LWBU4DTgRuAC7oajkzMxvcuu3jzzdO+QJwLGkY5Z194xQzs+bWXR//d4HDgYnADhHxSt2iMjOz0nR3AdcXgc2ArwJzJS3OjyWSFtcnPDMzG2jd9fH3+qpeq4/BepFVq15E1V29mrVOVm1O7mZmFePEb2ZWMU78ZmYV48RvZlYxTvxmZhXjxG9mVjFO/GZmFVPLePxmA6JZrz/oz7I+z98GI7f4zcwqxonfzKxinPjNzCrGid/MrGKc+M3MKsaJ38ysYpz4zcwqxufxWyU06hqCRp3n7+sLrDtu8ZuZVYwTv5lZxTjxm5lVjBO/mVnFNOTgrqQ5wBLgLeDNiGhrRBxmZlXUyLN6/i4iFjVw+2ZmleSuHjOzimlUiz+AuyUF8JOImNjxBZLGA+MBRo4cWefwzOqju/PtezrXvsxrE/qzbl8jMPg1qsW/e0TsDOwPnCxpz44viIiJEdEWEW3Dhg2rf4RmZi2qIYk/IubmvwuAW4GxjYjDzKyK6p74Ja0tad32aWA/YHq94zAzq6pG9PFvCtwqqX3710XELxsQh5lZJdU98UfEM8AH671dMzNLfDqnmVnFOPGbmVWME7+ZWcX4Rixm1jT6c8Gbvc0tfjOzinHiNzOrGCd+M7OKceI3M6sYJ34zs4px4jczqxgnfjOzivF5/GY2oMq8QYwNDLf4zcwqxonfzKxinPjNzCrGid/MrGKc+M3MKsaJ38ysYpz4zcwqxufxm1VQK55r31OdGjVe/2CMyy1+M7OKceI3M6sYJ34zs4px4jczqxgnfjOzinHiNzOrGCd+M7OKceI3M6sYX8BlNki14kVWjdTd+9moi7ugMRd4ucVvZlYxTvxmZhXjxG9mVjFO/GZmFdOQxC/pE5JmSZotaUIjYjAzq6q6J35JqwIXA/sDo4GjJY2udxxmZlXViBb/WGB2RDwTEa8DNwDjGhCHmVklNeI8/s2B5wrPnwd26fgiSeOB8fnpK5Jm9XF7Q4FFfVy2WbR6HVu9ftD6dRzU9dN5A7KaUurYz9i27KywEYlfnZTFSgURE4GJ/d6YNCUi2vq7nsGs1evY6vWD1q9jq9cPmquOjejqeR4YUXi+BTC3AXGYmVVSIxL/74BtJW0laXXgKOD2BsRhZlZJde/qiYg3JZ0C/ApYFbg8ImaUuMl+dxc1gVavY6vXD1q/jq1eP2iiOipipe51MzNrYb5y18ysYpz4zcwqpqUTfysODSFpjqRpkqZKmpLLNpJ0j6Sn898NGx1nb0i6XNICSdMLZZ3WSckP8j59QtLOjYu8Nl3U72xJL+T9OFXSAYV5X871myXp442JunckjZB0n6SZkmZIOi2Xt8R+7KZ+zbkfI6IlH6QDx38AtgZWBx4HRjc6rgGo1xxgaIey7wAT8vQE4LxGx9nLOu0J7AxM76lOwAHAXaTrQXYFHm50/H2s39nAv3Xy2tH5s7oGsFX+DK/a6DrUUMfhwM55el3gqVyXltiP3dSvKfdjK7f4qzQ0xDjgqjx9FXBoA2PptYh4AHixQ3FXdRoHXB3JQ8AGkobXJ9K+6aJ+XRkH3BARyyPij8Bs0md5UIuIeRHxWJ5eAswkXaXfEvuxm/p1ZVDvx1ZO/J0NDdHdjmoWAdwt6dE8rAXAphExD9IHFNikYdENnK7q1Er79ZTczXF5oXuu6esnaRSwE/AwLbgfO9QPmnA/tnLir2loiCa0e0TsTBrd9GRJezY6oDprlf36Y2AbYAwwD7gglzd1/SStA9wMnB4Ri7t7aSdlg76endSvKfdjKyf+lhwaIiLm5r8LgFtJPx/nt/9Mzn8XNC7CAdNVnVpiv0bE/Ih4KyJWAJfydjdA09ZP0mqkpHhtRNySi1tmP3ZWv2bdj62c+FtuaAhJa0tat30a2A+YTqrX8fllxwO3NSbCAdVVnW4HPpXPCtkVeLm9K6GZdOjPPoy0HyHV7yhJa0jaCtgWeKTe8fWWJAGXATMj4sLCrJbYj13Vr2n3Y6OPLpf5IJ058BTpiPpXGh3PANRna9KZAo8DM9rrBGwMTAKezn83anSsvazX9aSfyW+QWkondVUn0k/oi/M+nQa0NTr+Ptbvv3P8T5CSxPDC67+S6zcL2L/R8ddYxz1IXRlPAFPz44BW2Y/d1K8p96OHbDAzq5hW7uoxM7NOOPGbmVWME7+ZWcU48ZuZVYwTv5lZxTjxW9OR9JU8QuITeUTEXRocz6GSRncz/1OSpueYn5T0byXEcOZAr9NalxO/NRVJuwEHkUZK3BHYl3eOidIIh5JGY1yJpP2B04H9ImI70iidL5cQgxO/1cyJ35rNcGBRRCwHiIhFkYexULpXwdA83SZpcp4+Ow+gNVnSM5JOzeVrS7pD0uO5RX5kYT3nSXokP96Ty7eUNCn/0pgkaaSkjwCHAN/Nvz626RDvl0nD9rYPtbEsIi7N6xsj6aG8vlsLY9VPltSWp4dKmpOnT5B0i6Rf5vHtv5PLzwXelbd/bRlvurUWJ35rNncDIyQ9JelHkvaqcbn3Ax8njaVyVh535RPA3Ij4YERsD/yy8PrFETEW+CFwUS77IWko4R2Ba4EfRMRvSVdsfikixkTEHzpsd3vg0S5iuho4I69vGnBWDfUYAxwJ7AAcKWlEREwAXsvbP7aGdVjFOfFbU4mIV4APAeOBhcBPJZ1Qw6J3RBobfRFpoLBNScl239y6/2hEFLtgri/83S1P7wZcl6f/m3QZf59IWh/YICLuz0VXkW7Y0pNJEfFyRCwDngS27GsMVl1O/NZ0Io2GODkizgJOAT6ZZ73J25/pNTsstrww/RYwJCKeIn2JTAPOkfT14ma6mKaG8qIZeRu90at69HLdZk781lwkvU/StoWiMcCf8vQc3k6yn6QHkjYDXo2Ia4DzSQde2x1Z+Pu/efq3pFFeAY4FHszTS0i34+vMOcB3JL07b3MNSafmXxcvSfpoft1xQHvrv1iPI3qqR/ZG7r4y65FbC9Zs1gH+U9IGpJbxbFK3D8A3gMvyqY0Pd7F80Q6kg7IrSCNnfr4wbw1JD5MaR0fnslOByyV9idTNdGIuvwG4NB80PqLYzx8Rd0raFLg3D+0bwOV59vHAJZLWAp4prO984EZJxwG/rqEeABOBJyQ95n5+64lH5zTrIJ9F05aPB5i1HHf1mJlVjFv8ZmYV4xa/mVnFOPGbmVWME7+ZWcU48ZuZVYwTv5lZxfw/8CjVbIO0OGIAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "df = pd.read_csv('http://www.sidc.be/silso/DATA/SN_y_tot_V2.0.csv', sep=';', names=['time', 'sunspot.year'], usecols=[0, 1])\n", "data = torch.tensor(df['sunspot.year'].values, dtype=torch.float32).round()\n", "N = data.shape[0]\n", "\n", "plt.hist(df['sunspot.year'].values, bins=40)\n", "plt.title(\"Number of Years vs. Sunspot Counts\")\n", "plt.xlabel(\"Sunspot Count\")\n", "plt.ylabel(\"Number of Years\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "1BE18id2_VlV" }, "source": [ "For this example, the cluster parameters $\\theta_i$ are rate parameters since we are constructing a scale-mixture of Poisson distributions. Again, $G_0$ is chosen to be the conjugate prior, which in this case is a Gamma distribution, though this still does not strictly matter for doing inference through Pyro. Below is the implementation of the model:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 298 }, "colab_type": "code", "id": "f7mWbeTz_GLI", "outputId": "bd64a9bc-5aa2-4767-d71f-bdcb6279526c" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "100%|██████████| 1500/1500 [00:09<00:00, 156.27it/s]\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEICAYAAABWJCMKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOydeXhU5dn/P/csyUBIAoR9kUSNyCJGZHGp6M8NtQot1YLaVq3W1mpd2vqK9q1aWluttvja2lrrQl3RolZaabWKeymbooKAREAJa1gSkpBtkvv3xzmJwzBJJsksGeb+XNdcc+bZzv3MmTnf89zPJqqKYRiGkX54km2AYRiGkRxMAAzDMNIUEwDDMIw0xQTAMAwjTTEBMAzDSFNMAAzDMNIUEwADEZkjIr9I0rlFRB4VkT0isiQZNhhGumIC0AURkY0isl1EskLCrhCRN5JoVrz4EnAGMERVJ4RGiMgoESkXkSPCwl8TkV8l0shEICJHici/XTHcIyLLRGRysu1qCRF5R0QubSNNpojMEpFiEalyf9sPicghcbbtdBHZGM9zHAyYAHRdfMB1yTaivYiIt51ZhgEbVbUqPEJVVwH3AA+LiLjlXw4MBn7WWVtDERFfLMvrwPkF+AewAOgHDABuACqTaVdncOv0PHA2MB3IBYqAD4FTk2ia0YSq2quLvYCNwExgN9DTDbsCeMM9zgcU8IXkeQO4wj2+FHgXmA2UAeuBE9zwTcAO4JKQvHOAB4B/AxXAm8CwkPgj3bjdwFrg62F5/4hz46oCTo9Qn0HAfDd/MfAdN/xyoAZowLnR/SxCXh/wPnA10B/YCZwQEj8SeNUtew3wtZC4KcAKt06fAz8NiTvc/Q4vc+MWAt2Bp4Bd7ve2BOgTwab/BeaGhd0P/DakXhvd864HZkRxzQe49vRoIb75+od8Lwrku5+fAO4D/umedxFQ4MZ53LgdQDnODXhkSL77gdfcfK8DQ0PO8yVgmZtvCTDRDb/LvW417rW7N4LNZwH7gMGt1HsIjvDtBtYB3w6JewK4PeTz6TgPC02fS4AfAh+59j0NZOIITTXQ6NpWiSOqxwHvAXuB7cDdyf6vJ/uVdAPsFeGiODeP03Genn7hhrVXAILuzc0L/MK9yd3v/kHOdP/sPdz0c9zPk9z4/wPeceOycETjMvemMxbnJjwqJG85cKJ7owlEqM+bwB+AAM4TYClwWoit77TxfRzj3iD+HXqjAbKBzcC3XNuOxbl5D3fjTwVGu3Yd7dp9rhvXJACP4tz4u+GIzN/cYy8wjgg3ZOBQ96aS5X724dxcxwE57vdR6MYNxL3ZtlFHD/Ap8CIwFegXFh+NAOx0bfADzwBPuHFfxrl557rnGQkMCMnXdP0y3d9I0++sjxt3oXu+b7jfby83/h3g0lbqdA/wWhv1fhf4nfvbaPptnRxi2+0haSMJwH9xxDMP+IQv/gP7pXXDlgIXhvx2Jib7v57sl7mAuja3Aj8Qkb4dyLtBVR9V1Qacm8FQYJaq1qrqK0Adzk2wiZdU9S1VrQV+AhwvIkOBc3H+SI+qalBV3wOeA84Pyfuiqr6rqo2qWhNqhFvGl4CbVLVGVVcADwHfjLYiqvo+8DAwArglJGoK8ImqPubathznBn6+m2+hqq507foAmAucHFb8baq6T1WrgXqcm97hqtqgqstU9QAXjKquB1bi3KjB6cMoU9VlTUmA0SISUNWtqvpxFHVsBE7BEbTZwFYReV1EDmsrbwjzXJvrgSdxxBa3Xjk4LTlU9WNV3RaS7+/u9avF+X4nichA4Dxglao+7X6/T+C0aL4cpT15wNaWIkWkAJgAzHR/G+/hCHLUvw2cB4JtqroLpyVR1EraeqBQRPJUtUJVF7fjPAclJgBdGFVdifOjntmB7NtDjqvd8sLDeoR83hRy3kqcJ+5BOD76iSJS1vQCLsZ56jogbwQGAbtVtSIk7DMcP357WIUjRPtCwoYBJ4bZNh3nqRsROV5E3hCRUhEpx3mK7hNWbqjtc3DcSc+KyGYRubOVvoGncJ6MAS7CueGiqnvd8KuBbSLyj/BO7JZQ1U2q+n1VPRQowLlhzYkmr0voTX0f7vV1Bf8BHFfddhF5QESyQ9KGXvtynKf+Qe7rs7BztOfa7cK9Fi0wCNip+/f/tPe3EbHOLXAZTutnrYgsEZFz2nGegxITgK7PbcB32P9P0fSH6R4SFnpD7ghDmw5EpAfQG9iCc3N4U1V7hrx6qOpVIXlbW1J2C9A77IZzCM6TbmfZhONiCLftGjd+Lk5rZaiq5uK0PCS0AFXVkOM6Vb1dVUfgtFq+iiN2kXgGOF1EhuC0BJ4KKeefqno6zs2vGPhTeyumqp/juM1Gu0FVdOJ6q+q9qjrWLW8kju+8idBrn4vjKtrivoaFFRV67dpaSvhVnJbkoBbitwB9Qke7hZXfmTofYJuqrlXVGTj9Ab8BnhORQDvKPOgwAejiqGoxzs3m2pCwUpw/yTdExCsi3wba4yqIxDki8iURyQB+DixW1U04LZAjROSbIuJ3X+NFZESU9m8C/gP8SkQCIjIGp5P0yU7aC07H8igRuSjEtgkiMtyNz8ZpfdSIyHHAjNYKE5FTRWS0iHhwOgrrcTo6I9VrO44P/FFgraquc8sYKCLniUh3HDdbVUtlhJ27j4jcJiKHunMj+uI8sf7XTfIBMMYdKtoN58EgKtzvZILbmqly7Qq16Ty3tZSJ01/0jqpuxbn2o0Rkuoj4ROQiHLfhAjffdpz+kJZ4GadT+QUROcb9reaIyPdF5BJV3YDTwfxLd7hokVvnpt/GCuDLItLLdUldG/EskdmOIy7NDx7ub7iP624rxxGJxnaUedBhApAazMLpjA3lO8CNOM3sUTg32c7wFM5NZTdOZ+rFAK7r5kycm+cWnCb3XTgdhtFyIU7H9RbgBRy/+787aW+Tu2IyTufkVte2X4XYdhWO8FTg+LafbaPIQTgd73txXE6v4owsaYmncDobnwoJ8+Jcl6041+YE4BoAETnFdVNFohZHxF/H6WD+yH3/tlvXj4Ff4nT2rwXeaqMuofTE6UMpwxlgsBWnn6GJJ3Bu/DuBMbg+ePdBYwpwk1uXG3A60Xe7+e4FLnTdb78NP6nbupoGvALMw/leP8Lx0y90k00HCnGu3TzgFlV93Y2bA6zGcQv9C6dFFxWu+/Q5YKNrXz/gHGC1+3u4B5iuqnXRlnkwIiEtYMMw0gwReQIoVtXbk22LkXisBWAYhpGmmAAYhmGkKeYCMgzDSFOsBWAYhpGmJHUBrPbSp08fzc/PT7YZhmEYKcXy5ct3quoBKwqklADk5+ezbNmythMahmEYzYhI+IxuwFxAhmEYaYsJgGEYRppiAmAYhpGmpFQfgGEkm/r6ekpKSqipqWk7sWEkmEAgwJAhQ/D7/VGlNwEwjHZQUlJCdnY2+fn5uLtUGkaXQFXZtWsXJSUlFBQURJXHXECG0Q5qamrIy8uzm7/R5RAR8vLy2tU6NQEwjHZiN3+jq9Le36YJgGEYRppiAmAYKYbX66WoqKj5deedd7aY9m9/+xsff/zFlsS33norr776aqdtKCsr4w9/+EO7891+++3cc889EcNFhOLi4uaw2bNnIyLNkz/POeccyspa2k4BNm7cyFNPPdVifGe5/fbbGTx4MEVFRRQWFjJt2rT9vttYccIJJwDxrw+YALSbGQ8uOuBlGImkW7durFixovk1c2bLW0aHC8CsWbM4/fTTO21DRwWgNY466ijmzv1iz5d58+YxcuTI5s8LFiygZ8+eLebv6A2zoaHNDduaueGGG1ixYgXr1q1j+vTpnHrqqZSWlrb7nK3xn/84ezuZABiGETUzZ85k5MiRjBkzhh//+Mf85z//Yf78+dx4440UFRXx6aefcumllzJv3jzAWVrllltu4fjjj2fcuHG89957TJ48mcMOO4wHHngAgMrKSk477TTGjh3LUUcdxYsvvth8rk8//ZSioiJuvPFGAO6++27Gjx/PmDFjuO22L3asvOOOOxg+fDinn346a9eubdH+r3zlK83lr1+/ntzcXPr2/WL5mvz8fHbu3MnSpUsZM2YMNTU1VFVVMWrUKFauXMnMmTN5++23KSoqYvbs2cyZM4drrrmmOf+5557LG2+8AUCPHj249dZbmThxIosWLWL58uWcfPLJHHvssUyePJmtW7e2+X1Pnz6dM888s/km3VIZp5xyCjfddBMTJkzgiCOO4O233wZg1apVTJgwgaKiIsaMGcO6deuabWv6jkPrc9JJJ7FixYrm85944ol8+OGHbdrZGjYM1DA6yj9nwraPYlvmgKPg7JZdOgDV1dUUFRU1f7755ps544wzeOGFF1izZg0iQllZGT179mTKlCmce+65nH/++RHLGjp0KIsWLeKGG27g0ksv5d1336WmpoZRo0bxve99j0AgwAsvvEBOTg47d+7kuOOOY8qUKdx5552sXLmy+Yb0yiuvsG7dOpYsWYKqMmXKFN566y2ysrKYO3cu77//PsFgkLFjx3LsscdGtCUnJ4ehQ4eycuVKXnzxRaZPn86jjz56QLrx48czZcoU/vd//5fq6mq+8Y1vMHr0aO68807uuece/vGPfwAwZ86cFr/DqqoqRo8ezaxZs6ivr+fkk0/mxRdfpG/fvjzzzDP85Cc/4ZFHHmn1OgCMHTuWNWvWUF9fzw9+8IMWywgGgyxZsoQFCxbws5/9jFdffZUHHniA6667josvvpi6uroDWiLh9enduzdz5szh3nvv5ZNPPqG2tpYxY8a0aWNrmAAYRorR5AIKJRgMEggEuOKKK/jyl7/MueeeG1VZU6ZMARz3S2VlJdnZ2WRnZxMIBCgrKyMrK4tbbrmFt956C4/Hw+bNm9m+ffsB5bzyyiu88sorHHPMMYDTcli3bh0VFRV89atfpXv37vudryVmzJjB3Llzefnll3nttdciCgA4fRnjx48nEAhw3333RVXXULxeL1/72tcAWLt2LStXruSMM84AHJfQwIEDoyqnaT+VtsqYNm0aAMceeywbN24E4Pjjj+eOO+6gpKSEadOmUVhY2Oq5LrjgAn7+859z991388gjj3DppZdGXd+WMAEwjI7SxpN6IvH5fCxZsoTXXnuNuXPn8vvf/56FCxe2mS8zMxMAj8fTfNz0ORgM8uSTT1JaWsry5cvx+/3k5+dHHGeuqtx8881897vf3S/83nvvbdfQxPPOO48bb7yRcePGkZOT02K63bt3U1lZSX19PTU1NWRlZR2Qxufz0djY2Pw51O5AIIDX6222fdSoUSxa1P7+vPfff59x48a1WUbTd+v1egkGgwBcdNFFTJw4kZdeeonJkyfz0EMPceqpp7Z4ru7du3PGGWfw4osv8uyzz8ZkZWTrAzCMg4DKykrKy8s555xzuPfee5tbCNnZ2VRUVHS43PLycvr164ff7+f111/ns88+i1ju5MmTeeSRR6isrARg8+bN7Nixg0mTJvHCCy9QXV1NRUUFf//731s9X7du3bjrrrv4yU9+0mq6K6+8kp///OdcfPHF3HTTTRFtys/PZ8WKFTQ2NrJp0yaWLFkSsazhw4dTWlrafPOur69n1apVbXwz8Nxzz/HKK69w4YUXdqiM9evXc+ihh3LttdcyZcqUA/z5ka7dFVdcwbXXXsv48ePp3bt3mza2RVQtABE5C/g/wAs8pKp3hsVnAo8BxwK7gOmqulFE8oB5wHhgjqpeQxgiMh84VFVHd6omhpEmhPcBnHXWWVx33XVMnTqVmpoaVJXZs2cDjkvlO9/5Dvfdd19z5297uPjiiznvvPMYN24cRUVFHHnkkQDk5eVx4oknMnr0aM4++2zuvvtuVq9ezfHHHw84HZlPPPEEY8eOZfr06RQVFTFs2DBOOumkNs85Y8aMVuMfe+wxfD4fF110EQ0NDZxwwgksXLiQk046CZ/Px9FHH82ll17K9ddfT0FBAUcddRSjR49m7NixEcvLyMhg3rx5XHvttZSXlxMMBrn++usZNWrUAWlnz57NE0880dyHsHDhwuaO6mjLaOKZZ57hiSeewO/3M2DAAG699db94seMGbNffW644QaOPfZYcnJyuOyyy9r6GqOizT2BRcQLfAKcAZQAS4ELVfXjkDTfB8ao6vdEZAbwVVWdLiJZwDHAaGB0uACIyDTgfDdvmwIwbtw4TfaGMJGGfc698vgkWGIkg9WrVzNixIhkm2GkKVu2bOGUU05hzZo1eDyRHTiRfqMislxVx4WnjcYFNAEoVtX1qloHzAWmhqWZCvzFPZ4HnCYioqpVqvoOcIDTUER6AD8EfhGFDYZhGGnNY489xsSJE7njjjtavPm3l2hKGQxsCvlc4oZFTKOqQaAcyGuj3J8DvwH2tZZIRK4UkWUisizWEy4MwzBShW9961ts2rSJCy64IGZlRiMAkbrww/1G0aT5IrFIEXC4qr7Q1slV9UFVHaeq40InhRhGsmjLbWoYyaK9v81oBKAEGBryeQiwpaU0IuIDcoHdrZR5PHCsiGwE3gGOEJE3ojM5yahy8r5XuKLs//BrbbKtMRJMIBBg165dJgJGl6NpP4BAIBB1nmhGAS0FCkWkANgMzAAuCkszH7gEWITTqbtQW/mHqOofgT8CiEg+8A9VPSVqq5PIl6ue55t7/wyAT4M80OtHSbbISCRDhgyhpKQk5uu/GEYsaNoRLFraFABVDYrINcDLOMNAH1HVVSIyC1imqvOBh4HHRaQY58m/eRyX+5SfA2SIyFeAM0NHEKUUjY1MrprPxxlHUZxxJFMq/8r8Hl/HadAY6YDf7496tyXD6OpENQ9AVRcAC8LCbg05rgEi9kyoan4bZW/EGSba9dnwJv0atvN0zmWsyRjNlMq/MqHmHeDrybbMMAyj3dhSEO1h9d+plm4sC5xAvWTwiX8EE2veSbZVhmEYHcKWgmgPmxbzScYI6iUDgCXdTqCg/lOo2JZkwwzDMNqPCUC0VJfB9lWszfhiavcnfnezis3vJckowzCMjmMCEC0lywDdTwA2+g+jAQ9sMQEwDCP1MAGIlk2LQbwU+4c3B9V5ApT4hsHm5Uk0zDAMo2OYAETL9pXQp5BaT7f9goszhsOW98EmBhmGkWKYAETL9pXQ/8ClXTf4C6F6D5SXJMEowzCMjmMCEA015VD2OfQ/cLrCFp87627XugQbZRiG0TlMAKJhx2rnvTUB2FmcQIMMwzA6jwlANGxf6bxHcAGVeXpDRra1AAzDSDlMAKJh+yoI9IScQQfGiUCfw2GnCYBhGKmFCUA0bF/luH8k0rYHQN7hsMtcQIZhpBYmAG3R2AjbP47o/mkmrxDKN0Fdq5ubGYZhdClMANqi/HOoq2hdAHof6ryXfZ4YmwzDMGKACUBbbF/lvEcYAdRMT3fDtPJNLacxDMPoYpgAtMW2lYBAvyNbTpPrCoC1AAzDSCFMANpiy3vQpxAyslpOkz0APD5rARiGkVKYALSGKmxaAkMntJ7O44WcwVBmAmAYRupgAtAauz6F6t0wpA0BAOh5iLUADMNIKaISABE5S0TWikixiMyMEJ8pIs+48YtFJN8NzxOR10WkUkR+H5K+u4i8JCJrRGSViNwZqwrFlJIlzntbLQBw+gGsBWAYRgrRpgCIiBe4HzgbGAlcKCIjw5JdDuxR1cOB2cBdbngN8FPgxxGKvkdVjwSOAU4UkbM7VoU4UvwaBHKhz/C20/YcChVbIVgXf7sMwzBiQDQtgAlAsaquV9U6YC4wNSzNVOAv7vE84DQREVWtUtV3cISgGVXdp6qvu8d1wHvAkE7UI/ZUbIOPX4SjLwRPFF9T7hBAoWJL3E0zDMOIBdEIwGAg1LdR4oZFTKOqQaAcyIvGABHpCZwHvNZC/JUiskxElpWWlkZTZOeo3AFL/gx/vRQagzDhyujyZbvrBFVsj5tphmEYsSQaAYi0AE749lfRpDmwYBEf8DRwn6quj5RGVR9U1XGqOq5v375tGttpXvohLPixM6Z/8i8h77Do8mUPcN4rtsbPNsMwjBgSjQCUAENDPg8Bwv0czWncm3ousDuKsh8E1qnqvVGkjTvf+NObVK95lde6n8WMrIeY8dEx0WfOHui8mwAYhpEiRCMAS4FCESkQkQxgBjA/LM184BL3+HxgoWrrm+SKyC9whOL69pkcP46sW0U3rWZ54LiWV/5sie69weM3ATAMI2XwtZVAVYMicg3wMuAFHlHVVSIyC1imqvOBh4HHRaQY58l/RlN+EdkI5AAZIvIV4ExgL/ATYA3wnjg329+r6kOxrFx7ObpmGfX4WZVR1P7MIk4roGJb7A0zDMOIA20KAICqLgAWhIXdGnJcA1zQQt78Fopt5yN2/Bkc/JxN/mHUegIdKyB7gLUADMNIGWwmcAg9G/c4Wzx2lOwB1gIwDCNlMAEIoWfDbsq8vTpegLmADMNIIUwAmmhsILexrPMtgNq9UFsZO7sMwzDihAlAE/t24aWxky0Ady5ApU0GMwyj62MC0IR70+5UCyCrn/NelYAZy4ZhGJ3EBKAJdwmHMm8nBKCHO1PZBMAwjBTABKCJ5hZAJ1xAWSYAhmGkDiYATVQ6o3c6JQDd+zjvVTtjYJBhGEZ8MQFoonIH+6Q7dR2dBAbgy3D2D7AWgGEYKYAJQBOV2ynvzNN/E1l9nSWlDcMwujgmAE3U7KXKk9X5crL6mQvIMIyUwASgibpKqqV758vJ6mMuIMMwUgITgCZqK6n2xEIA+poAGIaREpgANFFXQY1063w5WX2hejc0BDtflmEYRhwxAWiitpLqmAiAOxR0367Ol2UYhhFHTACaqKukJlYuIDA3kGEYXZ6oNoQ5GJnx4KLmY5/W8URDXWxaAD2a1gOyoaCGYXRtrAUAdGusBohdHwDYUFDDMLo8JgBAQB0BiM0ooKblIMwFZBhG1yYqARCRs0RkrYgUi8jMCPGZIvKMG79YRPLd8DwReV1EKkXk92F5jhWRj9w894m7M3wy6Kb7gBi1AAI9weMzATAMo8vTpgCIiBe4HzgbGAlcKCIjw5JdDuxR1cOB2cBdbngN8FPgxxGK/iNwJVDovs7qSAViQcB1AcWkD0DE5gIYhpESRNMCmAAUq+p6Va0D5gJTw9JMBf7iHs8DThMRUdUqVX0HRwiaEZGBQI6qLlJVBR4DvtKZinSGbq4LKCajgMCdDWx9AIZhdG2iEYDBwKaQzyVuWMQ0qhoEyoG8NsosaaNMAETkShFZJiLLSkvj81QdcF1AMWkBgLMekC0IZxhGFycaAYjkm9cOpOlQelV9UFXHqeq4vn37tlJkx2lqAcRkLSBwXUDWAjAMo2sTjQCUAENDPg8BtrSURkR8QC6wu40yh7RRZsJoGgYak1FA8MWCcNqaBhqGYSSXaARgKVAoIgUikgHMAOaHpZkPXOIenw8sdH37EVHVrUCFiBznjv75FvBiu62PEYFYjgICpwUQrIa6qtiUZxiGEQfanAmsqkERuQZ4GfACj6jqKhGZBSxT1fnAw8DjIlKM8+Q/oym/iGwEcoAMEfkKcKaqfgxcBcwBugH/dF9JoZtWU4efBonRxOjmyWA7ILNHbMpsi8/+A6//0jn3WXdCdv/EnNcwjJQlqjueqi4AFoSF3RpyXANc0ELe/BbClwGjozU0nnRr3Be7EUAQMhlsF/Q+NHbltkTlDnjy6xDIgZKlsHMdXPkGeNN2pQ/DMKLAZgLjuIBiNgIIvtgcfl+COoJf/6XjcvrWfJj2IGz/CJY+lJhzG4aRspgAAAGtoUY6sRl8OFnuCNhEjASq2wcfzIWii6DP4TBiChRMgnfvtT0JDMNoFRMAIFNrqYulAHRP4J4Any50nv5HTXM+i8DE70HFVlj3cvzPbxhGymICAPi1jjrJiF2BGVngCyTGBbT67876Q/lf+iKscDJkD4Llf2k5n2EYaY8JAJChtdRJZuwKFHFaAVVxbgGoQvGrUHgmeP1fhHt9MHoarH8dasrja4NhGCmLCQCQEesWAED33vFvAezZ6JzjkOMOjBsxBRrqYN2/42uDYRgpiwkAjgDUx1oAErEgXMky533ohAPjhoyHHv0dF5FhGEYETACIQx8AOC6geHcClywFfxb0HXFgnMcDhWfA+jegsSG+dhiGkZLYTCFa7wMI3Tu4XWQlSAAGj215wlfByfD+E7DtIxhUFF9bDMNIOawFgNsHQKxbAHlQVwn1NW2n7QiNDbDjYxh4dMtpCiY57xveio8NhmGkNCYAqmQQpz4AiF9H8J6NEKyBfhHcP01kD4A+w2HDm/GxwTCMlCbtBcBPHUB8+gAgfh3BO1Y775H8/6EUTILPFkGwLj52GIaRsqS9AGRokwDEcB4AOC4giF8LoFkAhree7tCTob4KtrwXHzsMw0hZTAA0Ti2AZhdQa/vidILS1ZB7SNvLTQ87ERDrBzAM4wBMALQWiGMLIG4uoDXQ78go7OgNA8eYABiGcQBpLwB+twUQ807gQE8Qb3xcQI2NsKsY+hwRXfpDTnAmjTXUx94WwzBSlrQXgC9aADEWAI/HaQXEowVQsRUaaqPfbGboeGfF0O2rYm+LYRgpS9pPBMtQ56k45gIArU4GizTBbO6Vx0dX7u71znvvgujSD3GWinjkmWd4Jau6fecyDOOgxVoATS0AYtwHAPFrAezZ4LxH2wLIHcJuT28K61bH3hbDMFKWqARARM4SkbUiUiwiMyPEZ4rIM278YhHJD4m72Q1fKyKTQ8JvEJFVIrJSRJ4WieWOLNHjj9coIHAEIB7LQexeDx4/5AyJLr0I6zJGcETdmtjbYhhGytKmAIiIF7gfOBsYCVwoIiPDkl0O7FHVw4HZwF1u3pHADGAUcBbwBxHxishg4FpgnKqOBrxuuoSTgdMCiHknMLguoDi0AHavh56HtGvT93UZI+jfsJWchrLY22MYRkoSTQtgAlCsqutVtQ6YC0wNSzMVaNp+ah5wmoiIGz5XVWtVdQNQ7JYHTv9DNxHxAd2BLZ2rSseI20QwcGYDV++J/d68uzdE7/5x+cTvzBgurDc3kGEYDtEIwGBgU8jnEjcsYhpVDQLlQF5LeVV1M3AP8DmwFShX1Vc6UoHOEu5HiPcAACAASURBVLeJYPDFZLDqGE8G27MReuW3K8uGjEKC+KwfwDCMZqIRAIkQplGmiRguIr1wWgcFwCAgS0S+EfHkIleKyDIRWVZaWhqFue3DH6+JYBCfyWDVZVC713EBtYN6yWCj/zCOMAEwDMMlGgEoAYaGfB7Cge6a5jSuSycX2N1K3tOBDapaqqr1wPPACZFOrqoPquo4VR3Xt2/fKMxtHxnNE8H8baTsAPFYEbTcbVC1UwAAijOGU1BfjGhj7OwxDCNliUYAlgKFIlIgIhk4nbXzw9LMBy5xj88HFqqquuEz3FFCBUAhsATH9XOciHR3+wpOA5LyaJqhtTQiBImDADQvCBfDkUBlTQIwtPV0EdjgP5xuWs3A4ObY2WMYRsrS5jASVQ2KyDXAyzijdR5R1VUiMgtYpqrzgYeBx0WkGOfJf4abd5WIPAt8DASBq1W1AVgsIvOA99zw94EHY1+9tsnQesf/L5G8VZ0kHktCN7UActvfAvjU7ywdUVC/Lnb2GIaRskQ1jlBVFwALwsJuDTmuAS5oIe8dwB0Rwm8DbmuPsfHA38p2kB2laZavV4M8Cfz17RVcEGHf9g5R9jn4un3hXmoHW3xDqZVMDjMBMAwDmwlMhtZRH+vtIF0axEel9CCnoTx2hZZ9DrlDOtRiaRQvn/kOtRaAYRiACQAZ1MZnCKjLXk8uOY0xFIDyTR3qAG5ifUYh+fWfOiuKGoaR1pgAaF18BcCbS3bj3tgVWLapQx3ATTR1BLOrOHY2GYaRkqS9APi1jvp4zAFwqfDkkt0Yo+UXgrXOkNJo1wCKwHp/oXOwdUVsbDIMI2VJewGIewsgli6giq3Oe87ADhex2XcItZIJW96PjU2GYaQsJgBxGAUUitMC2AsaPnm6A+x1599ld1wAnI7gAthiLQDDSHdMABLQAvDRADUxcAM1CUBO+FJM7WN9xhGw7UPrCDaMNCftBcDpA4ivAABQFYPZwM0C0PEWADgdwdRVWkewYaQ5aS8AiWgBALFZDqJiK2T0gMycThVjHcGGYUAa7QkcaQ9ecPsA4rEdpEtFswDEYDmIvVsc/3/YJLDwurW13+9m3yHObOIt78OYr7dYTjRlxZL21sMwjM5hLQAS5QKKkQB00v0DTkcwA0ZbR7BhpDnpLQCqZGqcZwJ7Y9gCqNja6Q7gZgYWWUewYaQ5aS0APuqBOG0G41IvmdRIoPOdwI2NjgB0Ygjofgwc43QE79kQm/IMw0g50loA4rodZAh7PbmdbwFUlUJjEHIGxcaoAWOc960fxKY8wzBSjjQXgDhuBxmCIwCdbAHsdTdxiZUA9BsBHp/jBjIMIy1JcwGI43aQIez15Ha+E7hpGYhYuYB8mdB3BGw1ATCMdCWtBcDf7AKKbwugIiYtgNjMAt6PgWOcFkAslqkwDCPlSGsBSFQfQEUsWgB7tzgum6y+sTEKnH6AqlKo2Ba7Mg3DSBnSXADcPoA4TgQD1wUUrIa6qo4X0jQCyBPDSzbQ7Qje9lHsyjQMI2VIm5nAkUjYKCBvyGSwjKwOFrI5dv7/JvqPdt63fQBHnBkxSbxm57Y0M9swjMQR1eOkiJwlImtFpFhEZkaIzxSRZ9z4xSKSHxJ3sxu+VkQmh4T3FJF5IrJGRFaLSMLn/WfgtADiORMYQtcD6oQbaO/W2I0AaiKQA70KrCPYMNKUNgVARLzA/cDZwEjgQhEZGZbscmCPqh4OzAbucvOOBGYAo4CzgD+45QH8H/AvVT0SOBpY3fnqtA+/Nk0Ei7cA9HQOOtoPoOouAxFjAYAvOoINw0g7omkBTACKVXW9qtYBc4GpYWmmAn9xj+cBp4mIuOFzVbVWVTcAxcAEEckBJgEPA6hqnarGaN/E6EnUPIA9nt7OQUc7W2v3Qn1V7F1A4HQE79kINTHcuN4wjJQgGgEYDGwK+VzihkVMo6pBoBzIayXvoUAp8KiIvC8iD4lIROe4iFwpIstEZFlpaWkU5kZPovoAyr1uC6ByR8cKaB4CGo8WwNHOu3UEG0baEY0ASISw8IHjLaVpKdwHjAX+qKrHAFXAAX0LAKr6oKqOU9VxffvGcAgkX7QA4t0HEJQM6NYLKjvYAoinADQvCWFuIMNIN6IRgBJgaMjnIcCWltKIiA/IBXa3krcEKFHVxW74PBxBSCgZCZoIBkCPAR13AcVTALL7Q4/+1g9gGGlINAKwFCgUkQIRycDp1J0flmY+cIl7fD6wUFXVDZ/hjhIqAAqBJaq6DdgkIsPdPKcBH3eyLu3Gr3U0ItQT36UgAOjRr+MuoFgvAxHOgDHWAjCMNKTNeQCqGhSRa4CXAS/wiKquEpFZwDJVnY/Tmfu4iBTjPPnPcPOuEpFncW7uQeBqVW1wi/4B8KQrKuuBy2JctzbJ0FrH/SORPFUxJnsAfN7Bse97t0D3PGf9nngwcAx8uhD/gPhujmMYRtciqolgqroAWBAWdmvIcQ1wQQt57wDuiBC+AhjXHmNjTYbWUUeCbnhNLQDV9gtOvIaANjFgDGgDQ+s3sj7jiPidxzCMLkV6zwSmNiZPvFHNau0xAII1znDLbj3bVc6vdhSzx5vHrx9cFNVM3HbPsnWXhBhW/6kJgGGkEWm9FpBf6+M+BLSZHv2d9w70A+Q17mSXt0+MDQqhZz5k5pBf/2n8zmEYRpcjrQUgQ2sTMwIInNE20O6hoH6tI6exnD3evDgY5eLxwICjKDABMIy0Is0FoC7xLYCK7e3K1rvBWT5ilze2cyAOYMAYDgmuR5r76A3DONhJcwFIYAug2QXUPgHIa3BmP8ddAAaOIaC1DAyGT/EwDONgJc0FoC7u20E2E8gFX6DdLqCECYA7Izi/vji+5zEMo8uQ1gLg17rEtQBEOjQZLK/JBeSJYycwQN/h1OO3jmDDSCPSWgAS2gcAHVoOIq+hlArJps4TiJNRLl4/m/zDTAAMI41I73kAWhv37SD3o0c/2NU+F0teQ2n83T8uG/2HMa5mUccmqx0M7N0Cn/3H2X3Nmwn9RsCwE8CbIDehYSSYtBYAPwle+iB7AHz2bruy9G7YmVABOHXfy+68g8ScM+moOjf9d2ZD8b8PjM/qC1+6ASZ8F7xp/XcxDkLS+hftjAJKjADMeHAR0ypq+Xr1Hr7xpzchyvPmNZTySUb4BmzxYaP/cMDpCE4LASjfDC/9CD75J3TvA6fcDMPPht6HQn0NlCyBJX+Gl2+BVS/ABX+B3PCtMAwjdUnfPgBVMhI5E5gvdgbLbYhu87OMxhqytSK+s4BD+MxXQCOSHv0AK56C+yfC+jfgjFlw/Udwykxng5zMbOjRF478Mnzrb3D+o7BjDTx0Ouy0UVLGwUPaCoCXIB4aEzcKCCj3OgLQq3F3VOnzGp0hoLsTJAC1nm5s9Q1mWP36hJwvKdTtgxevhr9d5dzsv78ITrwOMrq3nGf0NPj2v6ChDh7/6hf7MxhGipO2ApCo7SBD2e22AHo1RLc5fF6iZgGHsNF/GAUH61yAim0w5xx4/wmYdCNcMh96F0SXd8Bo+MY8qN7jiED1nvjaahgJII0FIDHbQYbSdCPvE7UAJGgSWAgbfYfTt2EHWY0VCTtnQtj2Efz5NCj9BGY8Daf+L3i87Stj0DFw4VOw61N4/kpobIyPrYaRINJYABK4HaRLhSeXOjKab+xt8YUAJMYFBE4LADi4+gGKX4VHzgJthG//E448p+NlFUyCs34F616Bt++JnY2GkQTSVgD8SXABIcIub992CMBOyjw9nU3lE8RBJwBrXoKnZjiunu+85vj9O8v4K+Cor8Prv4RPF3a+PMNIEmkrAE0uoIROBANXAKJbDiKRk8CaqPDmssvT5+BYE2jl8/Dst2BQEVzyj9jtqiYC590LfY90XEHtXOHVMLoKaSwATgsg0Xvg7vT2be7cbYveDTsTNgIolA3+w1K/BfDhX+G5y2HoRPjmCy3uwtZhMrLggkehthKe/w402jLaRuoRlQCIyFkislZEikVkZoT4TBF5xo1fLCL5IXE3u+FrRWRyWD6viLwvIv/obEXaSwZuCyDBArDL25dejbvxarDNtMloAYAzIWxwsITMxuqEnzsmrHsV/vY9GHYiXDzPGdcfD/qNgHN+DRvehHd+G59zGEYcaXMmsIh4gfuBM4ASYKmIzFfVj0OSXQ7sUdXDRWQGcBcwXURGAjOAUcAg4FUROUK1edeR64DVQE7MahQlGVoPJEcAPDTSq2EXO339W0zXrbGKLK1iZ5wEoLV9gz/JGIGHRgrr17Ay85g284XvUxxNms7a2GLZm9+j5qmL2eodxqya66me80HU5w8/X1Q2H/NNWP+m0x8w7ERn7aAOEvH8qs7Lk7aNdSOORPOrmgAUq+p6Va0D5gJTw9JMBf7iHs8DThMRccPnqmqtqm4Ait3yEJEhwJeBhzpfjfbjb+oDSOAoIKD5ht5WR3C/BmfV0B3egXG3KZx1GSNoxMPwulUJP3enqNwBcy9iryeXO/N+TrUnK/7nFIFzZ0OvfJh3OVTt6nSRmY3VnF35Ajx0BtwxEGb1gt+MgPk/cGYkG0aMiEYABgObQj6XuGER06hqECgH8trIey/wP0Crg6lF5EoRWSYiy0pLoxs9Ew3J6wNwnvr7NrTecdgvuBWAHb4BcbcpnGpPFp/78hlem0IC0BCEv14G1WXc0/u25lnXCSGQ4ywXsW8XPHOxs45QB/BqkHMr/8rvdlzCJXv/5Mw8HvdtOHkmHHIcfDQPHjgR3rjT5iAYMSGaxeAirQusUaaJGC4i5wI7VHW5iJzS2slV9UHgQYBx48aFn7fDNI8CSrAAlPr604jQv2Frq+n6uy2A7UloAQCszRjFpOpX8WgDjdLOCVPJYOEs+Owd+OqDfL74kMSff1ARfPUBmHeZs8zE1x5un9tm83J+WfoDhgU38H7meJ7PvpCff/fy/dNU7YJ/zYQ3fgU718FX/2QrlBqdIppfaAkwNOTzECB8MZTmNCLiA3KB3a3kPRGYIiIbcVxKp4rIEx2wv8MkYyIYOC2OPZ48+gVb3ximX3AbldIjMW6MCKzJHEU3reaQVFgXaOM78O59cOxlcPT05NkxepqzsNyq5+GlG6IbGVRXBf+6BR46nezGvdzT61buyvs56yKtAJuVB9MehNNug5Xz4B/XOf0DhtFBonl8WAoUikgBsBmnU/eisDTzgUuARcD5wEJVVRGZDzwlIr/F6QQuBJao6iLgZgC3BfBjVf1GDOoTNUmZCOay3TeQ/g2tLyjWr2Er233JefoHpwUAcGTdKjZmFCbNjrbo1lgFL9zgTPSafEeyzYETroWacnj7N7B3q9Mq6B7BHaUKxa/BSz+Ess/g2Mv40eZz2xZ8ETjph1C/D966G/qPhuOuik9djIOeNlsArk//GuBlnBE7z6rqKhGZJSJT3GQPA3kiUgz8EJjp5l0FPAt8DPwLuDpkBFBSae4DIEkCEGzDBRTcxg5v4v3/Tez29qXU26/LdwR/Y++fYW+J4w7JSE5raT9E4LRb4Zx74NPX4Pfj4J17Yfd6CNY5C9J9+Cz8+f/Bk18Djw8ufQnOu7d9rb1TboHh58ArP4XtH7ed3jAiEJUDUVUXAAvCwm4NOa4BLmgh7x1Ai49mqvoG8EY0dsQSZzvIjKRsfbjdO4hejXvIaKyJuNevaAN9G7azpNuJCbctlLUZoxhZ+0GX3SLyiNpVnLbvX85T99AJyTZnfyZ8xxkS+q+b4dXbnFcovQ+Fc++Foy8Efwf2e/Z4YMrv4P4JMP8auPzf7V/czkh70rYHKYMEbwgfwnZ3ZE+/hm2UePIPiO/bsAMfQbYlqQO4idUZo/lS9esMbChhq29o2xkSiEcbuLz8d+z09qXPyTcl25zI9B/lLDm9c53TT1FVCoFcGDQWBo/t/A07qw+c/WtnxvN//wAn/CA2dhtpQ9oKgF+TKADujX1AcAsl/vwD4gcHPwdgs3/YAXHRTo6KBR9ljgXg6JrlbO3RtQTgrKoXGRbcyG96/ZQfZfaIyzmindDW5gSyPoXOKx6M/hqsfM6ZiHbUBc6+04YRJWk7vTBDa6lP8AigJra4T9ND3Bt9OM0CkOSn7h2+gWz1DmJM7fKk2hFOr4adXFDxOO9njmdpoOMzbw8KRJzO74Z6Z3ioYbSDNBaA5LUAajzdKfX2Y2hwY8T4wfWbKPP0osoTpzVs2sGHmccysu7D5lFTXYFvlv8ZrwaZk3tVl+ybSDi9D4Xxl8N7j0Pp2mRbY6QQJgBJYpMvnyH1n0WMGxz8POlP/028F5hIQGs5qvb9ZJsCwOja9zih5k3+lj2D7b4YLe98MDDpRvB3h1d/lmxLjBQijQWgNuHLQIRS4h/GoGAJnvBRsaoMDm6ixJeE2awRWJl5NFWSxcTqt5NtCj6t49tl97PNO5C/94g46Cx9yerjbG6/9iXY+kGyrTFShLQVgEytoUa6Je38m3zD8FPPgOD+E8LyGneSpVVs9ncNAWgQP8sDx3FszX/xuiuoJotzK59jUMNmHs29Oqni3WWZeCVk5sJbtlWlER1pKwABraFGOjD+OkZs8jkjfA4Jbtgv/PA6Z7XHYv/whNvUEou6TaKHVjK2ZknSbOgf3MK0iqdZHPgSHwTGJc2OLk0gFyZ+F1bPhx2rk22NkQKkrQBkag21yRQAfwG1kskRdfvP4jysbi31+PncX5Akyw7kg8xx7PL04f/t+1dyDFDlirL7CIrP6fg1Wua4q8CfBW/bBjVG26StAAQaq5PqAmoQH+v8Rx6w1MLh9WvZ6D8soRvBt0WjeHmz+xkU1S6nf7D1NYziwaTqVzmqbgVP5VzOHm9ews+fUnTvDeO/7SwWtyfyIAPDaCJ9BUBrqPEkTwDAWWqhoP5TAo37AGd266H1n1Cc0XXcP038O+tcgniZWvlMQs/bN7iNb5X/ibX+kbzW/eyEnjtlmXgViAcWP5BsS4wuTlrOBPZqED/1Se0DAEcAPDRyRN1qPgwcy4i6jwhoLaszjkqqXZHY483jtayzOaPqJV7KmrbfLOVoZid3ZAazV+u5bs+v8NDI/b1uRKXjzyuJnEGddHIHOzOE33sMTr4JuvVMtkVGFyUtWwCZ6uzYlMw+AHAEYJ9058Tq1wE4vvpNaiTAisyu2cn5Qo+L2OfJ4qqy37Y5Isir9fQPbmFI/UayGis6dL6L9z7M4fVreaDnD9mRxKWxU5Ljr4a6SkcEDKMF0rIFENBqgKT2AQDUegK82+0UJlW/xtyGy5hQ8y7LA8dFXCG0K7DX25OHc6/hhj2/5No9d/GHnj+mNsTW3g2lHFvzX8bVLGJU7Yf4CDbHfeYrYHlgIm93Py2qheXOrZzHOVV/Y0HWV1jS7Utxqc9BzcCjIf8kxw103FXg9SfbIqMLkpYC0NQCSLYLCOC1rHM4Y98C7t7xXbrrPv7d/cvJNqlVFnebxGMNO/nW3gc5fMca3gtMBODQ+k84rH4dAFu9g/lX1hQ+9xdQL376BrdTVLuMqZXPMq1yLiszjubVrC+zNHA8DbL/jcmr9Vy491HOrXqeRYFJPJ7znYTX8aDh+Gvg6enw8Ytw1PnJtsbogqSlAAQaXRdQF3jS3ug/nN/1/B8u3vsQc3MuY01m1/P/h7OgxzTW+wv5SuVcTqh+A0XY5M/n6exLWRo4wVnsLmyNnvnZ08lt2MPJ+17h9H0LuH7PLynz9OKN7meyMrOIID4K6j/ljKq/M6hhMy93P5e/5F6FpsJ+xF2VwjMhrxD+8zunT8DWTTLCSE8B6CIuoCbe7X4q73Y/NdlmtIs1mUdxZzvFqtzbi/nZ0/l7j/M5uvY9Tt/3ElMq/8pXQkYWbfAfxp25s1gR6GIbvKQiHg8c/334xw3w2buQb640Y39MAIyEo+JlRWA8KwLjyWkoY0jwM3waZKtvMKXe/vakGkuOvhAW/gIW3W8CYBxAWgpAptYCXaMPIN3Z6+3Jx14bphg3/N1g/BXw5q+dncnitTGNkZJENQxURM4SkbUiUiwiMyPEZ4rIM278YhHJD4m72Q1fKyKT3bChIvK6iKwWkVUicl2sKhQNTS2AZA8DNYyEMP4K8GY4rQDDCKFNARARL3A/cDYwErhQREaGJbsc2KOqhwOzgbvcvCOBGcAo4CzgD255QeBHqjoCOA64OkKZcSPQ6LqAkjwT2DASQo9+UHQhfPA0VJYm2xqjCxGNC2gCUKyq6wFEZC4wFQhdxWwqcLt7PA/4vYiIGz5XVWuBDSJSDExQ1UXAVgBVrRCR1cDgsDLjRlcaBmp0noN1lm80exJHu2/xDZtOZHZwDvMeuI15Od+MeL5I+YyDm2hcQIOBTSGfS9ywiGlUNQiUA3nR5HXdRccAi6M3u3MEtIYgXoLY5BgjPdjqG8qywHGcue/vZLjDoA0jGgGINCRDo0zTal4R6QE8B1yvqnsjnlzkShFZJiLLSktj03wNaLXj/7fRJkYa8Y+s88lp3MvJ1a8m2xSjixCNAJQAoXP3hwDhawI3pxERH5AL7G4tr4j4cW7+T6rq8y2dXFUfVNVxqjqub9++UZjbNsneDcwwksGajFEU+4dzTuXzSPhWpEZaEo0ALAUKRaRARDJwOnXnh6WZD1ziHp8PLFRVdcNnuKOECoBCYInbP/AwsFpVE75zRaCxhpouMAvYMBKKCH/vcT4DG7ZwXE3y93g2kk+bAuD69K8BXgZWA8+q6ioRmSUiU9xkDwN5bifvD4GZbt5VwLM4nbv/Aq5W1QbgROCbwKkissJ9nRPjurVIswvIMNKMJYET2eQbxtcqnrJWgBHdRDBVXQAsCAu7NeS4Brighbx3AHeEhb1D5P6BhGACYKQrKh6ey76Y6/f8kuNq3mZRt1OSbZKRRNJyP4CsxkqqPD2SbYZhJIXFgS+xyXcI06wVkPakpQD0aKygQnKSbYZhJAUVD89nX8zQ4OccV/NOss0xkkhaCkB2414qPCYARvry3+ZWwJOINibbHCNJpN1icBmNNWRQR6UJgBEnOjqDt6NldySNipfnsy/muj2/4qTq13ir+xlJn1FtM5ETT9q1ALIbnflmFZ7sJFtiGMnlv4GTKPYPZ8beOWTa7OC0JP0EwJ1wbC0AI91R8fBYzpX0btzFeZV/TbY5RhJIOwHo0dwCMAEwjE8yR7EoMInzquaRF9yRbHOMBJN2ApDdWAFApbmADAOAp3K+jQKXl/8ONHyZL+NgJu0EwFoAhrE/pb4BPJt9CWNrl3JC9RvJNsdIIGknAE2dwNYCMIwv+GfWVIr9w7l07wPkNJQl2xwjQaSdAPRorGCfdKdBbC8Aw2hCxcsDPW8g0LiP75fdbXMD0oS0E4DsxnJz/xhGBEr8+fwl93sU1S5nSuWzyTbHSABpKAAV5v4xjBZ4rfs5vNvtZKZXPMaYmuXJNseIM2k3Ezi3cQ9lnt7JNsNIUeI5g7dLIMKfc69jSP3n3LDnF8zK+zUbMgoTcurw7yieM4M7ej0OttnKadUCEG1kYLCErb7wLY0Nw2iixtOdO/N+QaUnm5t2/5RB9ZvazmSkJGklAL0bdxHQWrb4hrad2DDSmD3ePH7V29nG4/ZdP+KwurVJtsiIB2klAE1PMlt8Q5JsiWF0fbb4h3Jbn99SLd356a6bmFht20gebKSVAAwOfg7AZmsBGEZUbPcN4rY+v2GTL58b9tzBFWX/R2ZjdbLNMmJEWgnAoGAJVZJFuadXsk0xjJShzJvH7X3u4cUeX+f0ff9k9o4rOGnfqzZX4CAgrQRgSPAz5+lfkrYdsWGkJA3i4+mcb/PTPr9lt7cPV5fdw293XMHkyhfJctfXMlKPqARARM4SkbUiUiwiMyPEZ4rIM278YhHJD4m72Q1fKyKToy0z1vQPbuHIupWsyiyK96kM46BlXcZIftpnNrN73UKFJ4fL9v6RP227kJt2/ZSzK1/gsLq1eLU+2WYaUdLmPAAR8QL3A2cAJcBSEZmvqh+HJLsc2KOqh4vIDOAuYLqIjARmAKOAQcCrInKEm6etMmNHsI6vVTxJI15eyTovLqcwjHRBxcPibpNY3G0SBXXrOL76TSbUvMsxtUsBqCODLb4hbPcNYrtvIHs8eVR4sqn05FDpyaZauhEUP/Xipx5/83EDPhSxFnoCiWYi2ASgWFXXA4jIXGAqEHqzngrc7h7PA34vIuKGz1XVWmCDiBS75RFFmbGhsQH+NIlJ1av5Z9ZU9njzYn4Kw0hXNmQUsiGjkKdyr6BXwy4K61ZzRN1qBgU3MSS4kbE1i/HTwRbB7U1iEMV7O5kTbOiYTXd4O5YvFvzPevAHYlpkNAIwGAidCVICTGwpjaoGRaQcyHPD/xuWt2kWVltlAiAiVwJXuh8rRaSjA5L7wOM74fEOZu9S9AF2JtuIGGF16ZocLHU5WOoB/9utM3UZFikwGgGIJK/hu0a0lKal8Eh9DxF3olDVB4EHWzMwGkRkmaqO62w5XQGrS9fE6tL1OFjqAfGpSzSdwCVA6MD5IcCWltKIiA/IBXa3kjeaMg3DMIw4Eo0ALAUKRaRARDJwOnXnh6WZD1ziHp8PLFRVdcNnuKOECoBCYEmUZRqGYRhxpE0XkOvTvwZ4GfACj6jqKhGZBSxT1fnAw8DjbifvbpwbOm66Z3E6d4PA1araABCpzNhXbz867UbqQlhduiZWl67HwVIPiENdRG0TaMMwjLQkrWYCG4ZhGF9gAmAYhpGmHPQCkOglJ2KNiGwUkY9EZIWILHPDeovIv0VknfveJVe3E5FHRGSHiKwMCYtouzjc516nD0VkbPIsP5AW6nK7iGx2r80KETknJC7iEihdAREZKiKvi8hqEVklIte54Sl3bVqpS8pdGxEJiMgSEfnArcvP3PACd4mddeIsuZPhhre4BE/UqOpB+8LpYP4UOBTIAD4ARibbrnbWYSPQJyzs18BM93gmcFey7WzB9knAWGBlW7YD5wD/xJk7JGDb7AAAAwZJREFUchywONn2R1GX24EfR0g70v2tZQIF7m/Qm+w6hNg3EBjrHmcDn7g2p9y1aaUuKXdt3O+3h3vsBxa73/ezwAw3/AHgKvf4+8AD7vEM4Jn2nvNgbwE0L2OhqnVA05ITqc5U4C/u8V+AryTRlhZR1bdwRoWF0pLtU4HH1OG/QE8RGZgYS9umhbq0RPMSKKq6AQhdAiXpqOpWVX3PPa4AVuPM0E+5a9NKXVqiy14b9/utdD/63ZcCp+IssQMHXpem6zUPOM1dgidqDnYBiLSMRaptCKzAKyKy3F0WA6C/qm4F5w8A9Euade2nJdtT9Vpd47pFHglxxaVMXVy3wTE4T5spfW3C6gIpeG1ExCsiK4AdwL9xWihlqhp0k4Tau98SPEDTEjxRc7ALQDTLWHR1TlTVscDZwNUiMinZBsWJVLxWfwQOA4qArcBv3PCUqIuI9ACeA65X1b2tJY0Q1qXqE6EuKXltVLVBVYtwVkeYAIyIlMx973RdDnYBSPklJ1R1i/u+A3gB50exvakJ7r7vSJ6F7aYl21PuWqnqdvcP2wj8mS9cCV2+LiLix7lhPqmqz7vBKXltItUlla8NgKqWAW/g9AH0FGeJHdjf3paW4Imag10AUnrJCRHJEpHspmPgTGAl+y+9cQnwYnIs7BAt2T4f+JY74uQ4oLzJHdFVCfODfxXn2kDLS6B0CVw/8cPAalX9bUhUyl2bluqSitdGRPqKSE/3uBtwOk6fxus4S+zAgdcl0hI80ZPsnu94v3BGMHyC40v7SbLtaafth+KMWPgAWNVkP46f7zVgnfveO9m2tmD/0zjN73qcp5XLW7Idpzl7v3udPgLGJdv+KOryuGvrh+6fcWBI+p+4dVkLnJ1s+8Pq8iUcV8GHwAr3dU4qXptW6pJy1wYYA7zv2rwSuNUNPxRHpIqBvwKZbnjA/Vzsxh/a3nPaUhCGYRhpysHuAjIMwzBawATAMAwjTTEBMAzDSFNMAAzDMP5/e3UgAAAAACDI33qQS6IpAQBMCQBgSgAAUwEfQIOiWY2q7AAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "def model(data):\n", " with pyro.plate(\"beta_plate\", T-1):\n", " beta = pyro.sample(\"beta\", Beta(1, alpha))\n", "\n", " with pyro.plate(\"lambda_plate\", T):\n", " lmbda = pyro.sample(\"lambda\", Gamma(3, 0.05))\n", " \n", " with pyro.plate(\"data\", N):\n", " z = pyro.sample(\"z\", Categorical(mix_weights(beta)))\n", " pyro.sample(\"obs\", Poisson(lmbda[z]), obs=data)\n", "\n", "def guide(data):\n", " kappa = pyro.param('kappa', lambda: Uniform(0, 2).sample([T-1]), constraint=constraints.positive)\n", " tau_0 = pyro.param('tau_0', lambda: Uniform(0, 5).sample([T]), constraint=constraints.positive)\n", " tau_1 = pyro.param('tau_1', lambda: LogNormal(-1, 1).sample([T]), constraint=constraints.positive)\n", " phi = pyro.param('phi', lambda: Dirichlet(1/T * torch.ones(T)).sample([N]), constraint=constraints.simplex)\n", "\n", " with pyro.plate(\"beta_plate\", T-1):\n", " q_beta = pyro.sample(\"beta\", Beta(torch.ones(T-1), kappa))\n", "\n", " with pyro.plate(\"lambda_plate\", T):\n", " q_lambda = pyro.sample(\"lambda\", Gamma(tau_0, tau_1))\n", "\n", " with pyro.plate(\"data\", N):\n", " z = pyro.sample(\"z\", Categorical(phi))\n", "\n", "T = 20\n", "alpha = 1.1\n", "n_iter = 1500\n", "optim = Adam({\"lr\": 0.05})\n", "svi = SVI(model, guide, optim, loss=Trace_ELBO())\n", "losses = []\n", "\n", "train(n_iter)\n", "\n", "samples = torch.arange(0, 300).type(torch.float)\n", "\n", "tau0_optimal = pyro.param(\"tau_0\").detach()\n", "tau1_optimal = pyro.param(\"tau_1\").detach()\n", "kappa_optimal = pyro.param(\"kappa\").detach()\n", "\n", "# We make a point-estimate of our latent variables using the posterior means of tau and kappa for the cluster params and weights\n", "Bayes_Rates = (tau0_optimal / tau1_optimal)\n", "Bayes_Weights = mix_weights(1. / (1. + kappa_optimal))\n", "\n", "def mixture_of_poisson(weights, rates, samples):\n", " return (weights * Poisson(rates).log_prob(samples.unsqueeze(-1)).exp()).sum(-1)\n", "\n", "likelihood = mixture_of_poisson(Bayes_Weights, Bayes_Rates, samples)\n", "\n", "plt.title(\"Number of Years vs. Sunspot Counts\")\n", "plt.hist(data.numpy(), bins=60, density=True, lw=0, alpha=0.75);\n", "plt.plot(samples, likelihood, label=\"Estimated Mixture Density\")\n", "plt.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "gjn4x2JqBfll" }, "source": [ "The above plot is the mixture density of the Bayes estimators of the cluster parameters, weighted by their corresponding weights. As in the Gaussian example, we have taken the Bayes estimators of each cluster parameter and their corresponding weights by computing the posterior means of `lambda` and `beta` respectively." ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "NaHLTiRJUxBR" }, "source": [ "### ELBO Behavior\n", "\n", "Below are plots of the behavior of the loss function (negative Trace_ELBO) over the SVI iterations during inference using Pyro, as well as a plot of the autocorrelations of the ELBO 'time series' versus iteration number. We can see that around 500 iterations, the loss stops decreasing significantly, so we can assume it takes around 500 iterations to achieve convergence. The autocorrelation plot reaches an autocorrelation very close to 0 around a lag of 500, further corroborating this hypothesis. Note that these are heuristics and do not necessarily imply convergence." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 369 }, "colab_type": "code", "id": "XzbxzSkOUveF", "outputId": "a752e5e8-1fcf-4ff9-d8db-8d3f59a6a32b" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAABDAAAAFgCAYAAABNIolGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd7hdZZmw8ftJQqihCSrVoIIjthEj2MWCIig4MxYQB1RGxoJYR8FhBLGhnzOoIxZQBlCkDDYUEFFBHBUl2JCiBggQaiChQ0KS5/vjfXdY2dn79H32yTn377rOdfbq7+prPestkZlIkiRJkiRNZNP6nQBJkiRJkqTBGMCQJEmSJEkTngEMSZIkSZI04RnAkCRJkiRJE54BDEmSJEmSNOEZwJAkSZIkSROeAQyNq4g4MiK+2e90rCki4vkR8Zd+p2MsRMT8iHhpD+f/3Ij4W0TcGxGv7tVyJEkTy2juL/26z0bEEyLi9xFxT0QcMt7Ln0oi4uMRcXtE3NLvtEgaPQMYa4B6Y36gvpi1/r5Yh70pIv6vy3QXRsSDdfy7IuKiiHhK2zg7RsRZdfg9EXFBRDyny/y2iohlEfG4DsO+GxGfHYv1nSgiYnZEZETMqN0nRsTHe7zMjIjHt7oz8xeZ+YQeLKe1bme39f9mRBw51ssbJ0cBX8zMDTLze6OdWd3fS+v5sygizo+IvxuDdA623F0jYkXjXL8xIj7aNk5GxH11+O0RcWpEbFyHfTUiTu4w36dGxJKI2HS8AokRMTMi/jMiFtS0XhsRx/R6uSM10PW0MU7runpPRNwdEZdGxKERsfYwlrPKeS71Uj1mFw/nGK3TTcrjdLzus0PwQeDCzJyVmV9oH9j2DNf6+0EdtmtELOg007Z71z31GvXCtnG2johTIuKOei/5bUS8sidrOYDxOMYiYhvg/cCOmfnoMZpn8x58Y0T8V0RMH4t5D7LcAfdtvYctbxwv10TE2+uwdSLizoh4cYf5HhMRZ9bfPf3Y1FjmkyLix/XadGddlz16vdyRqufjvwwwvPVc3dr2t0bEDyNit2EsY9BnEBUGMNYcr6ovZq2/g4c43cGZuQHwCOBC4ButAVECEb8ELgO2A7YEvgv8OCKe3T6jzLwR+Cnwz83+EbEpsAdw0rDXagppBUImmGdFxHP7nYjhGGA7Pga4fIzn+Zl6/mwN3AacOIbzHshNrXMdeB5wYKyeq+RpdfhjgU2AI2v/E4F/jIj128bfH/hhZi4aQXpG6jBgDrAzMAt4EfD7cVx+rxycmbOALSgPxvsA50RE9DdZ0qoiYjbwfCCBvfqamCHodL2coPfOsTCUe9bBbc9+rxrivFv3ro2ALwPfab1g12e2/wOWAk8CNgOOAb4VEa8ZyYpMcI8B7sjM24Y74SDHXuse/BLgDcBbhznviIiRvId13bfVrxvPD68BPhMRT8/MB4HTKc8CzXRMB/Zl/J/hfwCcDzwKeCRwCHD3OKehFzau2/5plPX7bkS8qb9JmnwMYEwRmbkMOA3YsdH7SMqF7t8zc1Fm3lO/AnwD+HSXWZ1EWwCD8vB+eWZeBhARn4+IGxpfJ5/faUadviA0I78RMa1+2by6fiU4o954O83ryubXg4iYEeXL9E416vzNOo87I+KSiHhUl/XrKCIOAvYDPtj2FWTLiPh2RCyM8nX5kMY0R0bEmXXZdwNvioidI+LXNR03R8QXI2JmHf+iOukf6zJe376NIuKJNQp8Z0RcHhF7NYadGBHHRsTZNTL/m+iQW6bNZ4COuUo6RYKj8bWkLu9LEXFuTe8vI+LREfG5GlG/KiKe3jbbZ0bEFXX4/0TEOo15vzIi/lDX7VcR8dTGsPkR8aGI+BNwX/tDRURcTXmR/0FNy9p135wVJffEvIh4a2P81fbNQBspM+8HvgU8ua7j/RHxiMb8nlGPgbXqdvtllC8ai4Aj67F8eERcFxG3RcTJEbHRQMtsLPta4Feseu42h98NnNUanpm/Bm4E/qmRvumUB6whPaBExF71+LqzHm9PbAybHxEfiIg/Rcm5dXpzP7Z5JvDdzLwpi/mZeXJjXqt8fYtGLqfWsR8R76/b7OaIeHNj3D3qsXRPlC9gH2ib7sP1GjA/IvZrTLdR3f4L6/44vO6fJwJfAZ5dj6E7B9tOmXlfZl5IeTF8NrBnXcZwz/NNonypWVjPjR9GxNaDLV8agv2BiymBzQOaA6Lti2Lzmt/pOK3931qvp4vq9XXLxvRPipJTbVGUr48frv3XrveFm+rf56LmBmmcrx+Kkr3/fzr1q+N2vUe0rdeEuc92u5ZGxM8oAd0v1nTs0G0eo5GZKyj3rk0pL4oA7wXuBQ7MzFsy84HMPBX4BPCfEasHYiPiRxFxcFu/P0bEP0ZxTL1O3xXl3vDk0aQ7Ih4XET+L8tx2e5TcIhs3hu8UDxe/+d8o96HVnmWiPE+eD2xZt/OJtf9g97iuzxvtMvMq4BeU54N/i4hvt6XhvyPic/X3hRHxiYj4JXA/8NgY4FllkOV22rft4/wOuBJord9JwD9FxHqN0V5OeR88d7BlDvFc7njPbpvPZpQPp8dn5tL698vMbF1/hvL82fE8HOh4rNN9Jcp16p6I+HlEPKaxjOdEeUe4q/5/Tu3/CUoguHW+fnGwbVXPrc9T3rU+HTVYFQ+/19wT5RnmH2r/js8gEbFnPdbvjvJudeRgy54SMtO/Cf4HzAde2mXYm4D/6zLsQuBf6u+ZlJvTRY3htwBv7jDdi4DlwHodhq0L3AU8r9Hv18B7Gt1vpOT4mEH5OnkLsE4ddiTwzfp7V2BBt3UF3kN58NoaWBv4KnBql3X9CHBKo3tP4Kr6+18pkd71gOnAM4ANh7DdZ1O+Ws2o3ScCH28MnwZcWpc9k/ICfQ3w8sa6PgS8uo67bl32s+q2mU25sTS3XQKPb3Sv3EbAWsA84MN1eS8G7gGe0EjfIsrX7hnAKcBpg6zbBpSX3dY2/yZwZLdjq5m+urzb6zqtA/wMuJbywDydEhi5oG3f/hnYhnLD/WVrewI7UXI47FKnPaCOv3Zj2j/UadcdynkC/Bz4Uk3b3wMLgZd02zcd5rdyf9ft9C3gF7X7HODtjXGPAf67sd2WAe+q+2Fd4C113z22zus7wDe6rMfKfV67t6/76MVd9sMmwI+BoxrD/x34SaP75XX912o/DzssfwfgPmA3yjH3wZr2mY3t/FtKjq1NKcfw27rM63DgeuAdwFOA6HY8ddjmu9bteFRNxx6UB75N6vCbgec3tsFObdP9F+W68cK6Pq3z5GTg+5QcIbOBv1Ie5Fv7ruP1tNN1ta3/RcCn6+/hnuePoASc1qvp+l/ge4Ndo/zzb7C/eu6+ox6TDwGPagxb5VhuP/47HKcvplzzd6rn1n9TnynqcXsz5Z6/Tu3epQ47inIvfySwOSUg+7E6rHW+frrOc90u/YZyj2jdxybKfXawa2nHa0m3/dM2bGWaOww7kYevo9OBt1GeTabXfhcDH+0w3XZ12zyhw7D9gV82uncE7qz75+WUZ6GNgaC8LG8xxONzlX3R6P/4ut3WrsfMRcDn6rCZwHXAu+t2/UdKbpKPd1nGKttqCPtlPoM/bzTvwTtSnnMPpOTKu4/yFZ56jNwGPKOxT6+n5HyZUZff9VllBPv2Tax6Dj+z7qcdGv3+Cryx0X1qa9u2n0sdlj+Uc7njPbttPgH8Dfgh5TnsUW3DV1mPDtv8RLqchwxwPNbp7gFeQDm2Pt9aDuV5ZjHlI+0MSq6UxcAjhni+zqbxztDo/9ja/4m1+7WU56dpwOvr8bLFAOu9K+X5aRrwVOBW4NVDOb8m81/fE+DfEHZSuZjcWy9Crb+31mGrHeyN6S6sF487KRf3u2hcFOuFZvcO0/1dPdm26jLfrwHH1d/b13k/coD0L6ZktYPhBTCubEvvFpQHsBkdlvH4elFar3afAnyk/n4L5SL71GFu91UuRqwewNgFuL5tmsOA/2ms60WDLOM9lC/Ure6BHqyeT7lJTmsMP5WHAw4nAl9rDNuDGsQZaN0oD7cX1/7DDWAc3xj2LuDKRvdTgDvb9u3b2tJ3df39ZepNsDH8L8ALG9O+ZQjnSevY2YYShJvVGP4p4MRh7JsTgQcp588tlFwOj6vDXk99mKM8RNwC7NzYbu3HxU+BdzS6n0D3Y3lXYEVd7t11m3+H+nDV2A9313GWA1fROF+Bbev8t26cD59vDD+S7gGM/wDOaHRPowRQdm1s5+bDz2eAr3SZ13TgnZRg1RLgJuCAAY73E1k1gPFAcxtRHgSfVX9fTwlObti2zF0p17b1G/3OqOs1vaZjx8awf6WUQ2/tu5EGME6jcT4M5zzvMP7fA4sHSod//g32Ryl+9hCwWe2+CnhvY/gqx3L78d/h/Pw6Jft6q3uDOv/ZlIf933dJx9XAHo3ulwPz6+9dKc8Q6zSGd+o3lHtEt5euft1nB7uWdryWtO2f1jNc6+9j7WnuMN2JPHzverD+7dcYPo8OQWfKC3QCz+0wbBblResxtfsTwAn194spL8XPam63IR6jA14LG+O9unV8UV4+b6QREKcUiRlqAGMo97jBnjda9+DFlOP74611p+RkaD2jvxK4om2fNj82DPisMoJ9+ybK/e9OyntDUgKNzW11OPDj+nvDeow9vTF8Pt3PpcHO5a737A7z2hr4Yp3nCkqQavvGegz2/NnxPBzoeKzTndbo3qBu/20ogYvfto3/a+BNQzxfZ9M5gNH1vKrD/wDs3W29O4z/OeCY4Zxnk/HPIiRrjldn5saNv+OHON0hmbkx5QR6JXBmPJzt8nZKUKDdFpSLyeIu8zwJeF2UbOP/DPwoG2ULa/axK2sWrDsp5fQ2G2J6mx5DKTt2Z53PlZQLzWpZ5TJzXh3+qpo1bi/KF3MoRWLOA06rWd4+ExFrjSA9ndK3ZSt9NY0fbkvfDc0JImKHKNnDb4lSdOGTDH3bbAnckCXbYMt1wFaN7mYN2/dTLs6DOR54VEQMtWxt062N3w906G5ffnN7XEdZJyjb8v1t23KbxvD2aQezJbAoM+9pW15zWw1lfp+t59ujM3OvzLy69v8+sGNEPJbyFeeuzPztAPPesi6/mZYZdMn2SakDY+PM3JDyFeEBVi/+sVPj3P4y8It6TpKZ11MeBt4YERtQHv6GWr51lbTW4+0GRnCcZebyzDw2M59b1+MTwAnN7LqDuCNL8bdOy/onykPLdTUbaLPensWZeV+ju3WsbcbDX++aw5rrNlJbUb4IDfs8j4j1olS+el0d/yJg4xiHSuE0qR1AeVG5vXZ/i7ZiJMPUfm24F7iDcuxvQ3kRGXQ6Vr32AyzMUj6fAfoN5R4BTKj77FCupYM5pO3Z7z+GON1n6/1hXUo9RP8vIl5Rhw307Ncavop6Lz2bUmSY+v+UOuxnlBfRY4FbI+K4iNhwiOnsKCIeGRGnRSkeeDfl40prH24J3Jj1ba4a7vPBYPtlKPPbKTM3yczHZebhjWPmJEpOZOr/b7RN15z3UJ5V2g20b6F8kGrVw/BoSm6PTzaGnwy8KCK2otSRMS8zh1o31WDn8kD37FVk5oLMPDgzH0c5v++raRuqjufhEI7Hldu/XsMW1XVoXzcYm+eD1vSt54P94+GicHcCT2bg54NdojSwsDAi7qLkuhnJO9WkYgBjisjMFZn5C0rk/WW1908oWZnavY5SN8b9Xeb1C8pDy96Ui3OzTPvzgQ/VeWxSL7J3UbJxtbuPkmW6Ne10Spa0lhuAV7TdvNfJUploJ6dSvgLtTYl4z6vpfSgzP5qZOwLPoQRy9u8yj4FkW/cNwLVt6ZuVmXsMMM2XKV/Btq8vpx+m87bp5CZgm1i10qdtKV8ORiwzHwI+CnysLS3t+2csau/epvF7W8o6QdmWn2jblutlKZe7MqnDWM5NwKYRMattec1tNZz5raI+WJ9BqRfln1n9AaV93jdRbtDNtCxj1YBPt2XdRXnx6Bhgqvvva5Tsv81yxydRjvN/ohynvxtsWZ3SGhFB2W+jPc4eyMxjKYHRVn0e99M4xigPW0Od3yWZuTclK+v3KPujZZNYtRLT1rF2O+WLcfu+aK3biI6JKLXcP4NSDhqGf56/n5IrZ5c6/gtasx5JeqSIWJdyH35hfZG/hVL3wdMi4ml1tFWu8Qx+/rVfG9anFH+6kXIN71YXRKfr302N7k7nXaf77WD3iJaJcp/tybV0OLL4MyUX3J61908o9SC0vwO8jrKd/9pldqcC+9Zg8brABY3lfCEzn0F5Wd4B+LdRJv1TlGPgqXUfvpGH9+HNwFZ1e7Zsw9ANZb+M+PmAcj96aq134ZXUQE+XeQ/lWaWjLvu2fZxbgW/TeH6oHzh+wcPPL8MJGgx2Lo9IZt5ACTi0nmFG9fw5yPG48lipH3g2paxD+7rBGDwfAP9AyYnyl1rfxvHAwZSiKRtTila3juVOy/gWJQfwNpm5EaWejCn/bGAAY3KIKBVVrvzrMtKzKS8OrVqvPwo8J0qFQptGxKyIeBflpedDgyzzZEr51I0p9Uu0zKK8mC0EZkTERyhZ1Dr5K7BOraBmLUq2tmYzb18BPlFPeCJi84jYe4A0nUYJzrydh3NfEBEvioin1ADJ3ZQXmOWDrF8nt1LKsrX8Frg7SmVP60bE9Ih4ckQ8c4B5zKppuDdKk5xvH2QZTb+hXNQ/GKWyyF0pN6XTRrAu7b5B2fa7N/r9EXhSRPx9PaaOHIPlvDNK822bUh4qT6/9jwfeViPNERHr1+NiVvdZdVdvhr8CPlXPiadSyqe2P0iMxsmU7H57Ub4ODeRU4L0RsV29YX4SOL3tS0VHdfx96FJbfT2u30zJpXFNY9C3KTfqjzK82sXPAPaMiJfU8/L9lGIXvxrGPFppe0+Uir3WjVKx7gGUc6D1tecPwBvqubM7pb6Kocx3ZkTsFxEb1QDO3ax+Tn+0jvd8ykPk/2bm8rp+n6jXu8cA7+Ph/XcrsHXUCv+GkI71ojRh933K9eCcOmi45/ksyv67s54bRwxl+dIAXk05J3akFEn6e0pZ8F/wcAD/D5QWi9aLUjnegW3zaD9OvwW8ud4T1qZcx36TmfMpZdkfXc/5tev5tUud7lTg8HoP34xSb9Rwm3Eezj1iotxnx+xa2k37s1/bS31rnL+jFCdq3UOOoTyXfT1KpdTrRMS+lLqT/q0tZ0PTOZQXvKMo968Vdf7PrPtlLcq2e5DhPWPNbFuH6ZR9eC/lmrgVq76A/rrO/+B6X9mbUhfCUPV0v9QPHGdSzpff1oBBt3FH9azSYd+2D38E5QW6ffhJlJfo5w51WdVYnMtEqbj6oxHx+CiVaG9GKe59cR1lxM+fQzge94iI59X7/Mco17AbKMf3DhHxhnpcvZ5y/fxhnW6g60andDwqSsW3RwCH1fNlfUqQYmEd582s+uGp0zPILEounQcjYmdKhexTngGMNUerdYXW33cbw55Defhd+RcP15rcqjH3XspL6uGZeS5AZv6NcuF7GqXM282Ur7Uvz8xfDpKekymRydMzc0mj/3mU8n9/pWS9epAuWfHql+V3UL4e30i50DRbJfk8Jer444i4h3Jh26V9Po353Uy5sT2Hh1+MoXxVOpPyQHMlpcKkbwJEqY34K4Osa8vXKcUG7oyI79WXoVdRHgyvpXzd/RqlyEw3H6BcfO6hPJCd3jb8SOCkuozXta3fUsrL8ivqsr4E7J+lBuxRqetyBCUS3er3V8qDyk8olS2NRdvU36JUOHlN/ft4XdZcShNkX6R8oZ/HIC2DDMG+lDKJN1GaBz4iM88f5TxXqufICuB39QF+ICdQzr+LKMfKg5Q6Q7rZsnHeXkfZL/u1jfPHOnwxJVv4P2SjidRahKIVxBjyA0pm/oXyteu/KcfZqyjNOC8d6jwaHgD+k5LV83ZKfRj/lJmtQMu76/zvpKzf94Yx738G5kfJXvw2Hs6yS13eYsq+P4VS3rt1nryLcq25hnJMf4uyf6BURHs5cEtErJaNuuGL9Zp0K6U86rcp9Qm1shAP9zz/HOWL5u2U69yPhrgNpG4OoNTHdH2W2vBvycxbKNfY/eozwjGUuiZupbzQtF8njqRxnGbmTyn1B3yb8rzwOGqRgpoFfjfK+XwL5Z7xojqfjwNzgT9Rmm3/HV1av+pmmPeICXGfHaNr6Rfbnv0ubQzbirZnPx7OBdNqMe0+yj33fygVoZOZd1Ce/dYBrqDkqH0f8M+Z2b6tmuuzhFIf00tpfCSiBEOOp+yX6+r8PgsQpTWowVq3uLxtHd5MCbzvRMnBe3ZdbisdSykVdx5IuXe8kfKSuYQhGON7XDcnUeoBa8+d2clwn1W67tvq2Y3nhyspL8vtzxtnUiq//ml9dh6qUZ/L1VLKOv+E8mz+Z8r+exOM+vmz6/FYfYvyvLuIknNyv7rMOygfO95fp/kg8Mp8uAje54HXRGkp7AsDLP/Oum8uoxRzfW1mnlCXcQXlmejXlOvuUyg5aFo6PYO8AziqPnN8hFVzm05Z0T3QKkkaSJSm8L6VmV/rd1pU1C+m38xMmyGVpCkgIn5DqUz6f/qdFoCI2JZSjOnRWZo61wQQpRndBZl5eL/TotExB4YkjUCUokI7sfrXPUmS1CMR8cJa/KVVNPGpTJCca1HqFnkfpbULgxdSD8wYfBRJUlNEnEQpY/7uXLX2cEmS1FtPoGSl34DS+s1rhlkUoieiVGx7K6Xowu6DjC5phCxCIkmSJEmSJjyLkEiSJEmSpAlvyhUh2WyzzXL27Nn9ToYkSZPOpZdeentmbt7vdIyWzwqSJPXGaJ8VplwAY/bs2cydO7ffyZAkadKJiOv6nYax4LOCJEm9MdpnBYuQSJIkSZKkCc8AhiRJkiRJmvAMYEiSJEmSpAnPAIYkSZIkSZrwDGBIkiRJkqQJzwCGJEmSJEma8AxgSJIkSZKkCc8AhiRJkiRJmvAMYEiSpDVSRJwQEbdFxJ+7DI+I+EJEzIuIP0XETuOdRkmSNHYMYEiSpDXVicDuAwx/BbB9/TsI+PI4pEmSJPWIAYwxlplkZr+TIUnSpJeZFwGLBhhlb+DkLC4GNo6ILcYndZIkaawZwBhj2x12Doec9od+J0OSJMFWwA2N7gW134Auu/GuniVIkiSNnAGMHvjBH2/qdxIkSRJEh34ds0lGxEERMTci5j565tIeJ0uSJI2EAQxJkjRZLQC2aXRvDXT8ypCZx2XmnMycs/nmm49L4iRJ0vAYwJAkSZPVWcD+tTWSZwF3ZebN/U6UJEkamRn9ToAkSdJIRMSpwK7AZhGxADgCWAsgM78CnAPsAcwD7gfe3J+USpKksWAAQ5IkrZEyc99BhifwznFKjiRJ6jGLkEiSJEmSpAnPAIYkSZIkSZrwDGBIkiRJkqQJzwCGJEmSJEma8AxgSJIkSZKkCc8AhiRJkiRJmvAMYEiSJEmSpAnPAIYkSZIkSZrwDGBIkiRJkqQJzwCGJEmSJEma8AxgSJIkSZKkCc8AhiRJkiRJmvAMYEiSJEmSpAnPAIYkSZIkSZrwDGBIkiRJkqQJzwCGJElSm9mHnt3vJEiSpDYGMCRJktrMP3rPfidBkiS1MYAhSZIkSZImPAMYkiRJkiRpwjOAIUmS1IH1YEiSNLEYwJAkSerAejAkSZpYDGCMoczsdxIkSZIkSZqUehrAiIj3RsTlEfHniDg1ItaJiO0i4jcR8beIOD0iZtZx167d8+rw2Y35HFb7/yUiXt7ov3vtNy8iDu3lugyF8QtJkiRJknqjZwGMiNgKOASYk5lPBqYD+wCfBo7JzO2BxcCBdZIDgcWZ+XjgmDoeEbFjne5JwO7AlyJiekRMB44FXgHsCOxbx5UkSZIkSZNMr4uQzADWjYgZwHrAzcCLgTPr8JOAV9ffe9du6vCXRETU/qdl5pLMvBaYB+xc/+Zl5jWZuRQ4rY7bN2bAkCRJkiSpN3oWwMjMG4HPAtdTAhd3AZcCd2bmsjraAmCr+nsr4IY67bI6/iOa/dum6dZ/NRFxUETMjYi5CxcuHP3KdWEdGJIkSZIk9UYvi5BsQskRsR2wJbA+pbhHu9Zbf3QZNtz+q/fMPC4z52TmnM0333ywpEuSJEmSpAmml0VIXgpcm5kLM/Mh4DvAc4CNa5ESgK2Bm+rvBcA2AHX4RsCiZv+2abr17xvzX0iSJEmS1Bu9DGBcDzwrItardVm8BLgCuAB4TR3nAOD79fdZtZs6/GdZymScBexTWynZDtge+C1wCbB9bdVkJqWiz7N6uD6DsgSJJEmSJEm9MWPwUUYmM38TEWcCvwOWAb8HjgPOBk6LiI/Xfl+vk3wd+EZEzKPkvNinzufyiDiDEvxYBrwzM5cDRMTBwHmUFk5OyMzLe7U+kiRJkiSpf3oWwADIzCOAI9p6X0NpQaR93AeB13aZzyeAT3Tofw5wzuhTOjbSQiSSJE0qsw89m/lH79nvZEiSJHrfjOqUYhESSZImF4MXkiRNHAYwRmnFiuTeJcsGH1GSJEmSJI2YAYxRevspl/KaL/+q38mQJEmSJGlSM4AxSo/YYG0W3rMEsAiJJEmSJEm9YgBjlDbfYG0W3b+UZctX9DspkiSpB2Yfena/kyBJkjCAMWqbzVqbTFh031JbIZEkaRKyIk9JkiYGAxijtO5a0wFYsmyFRUgkSZIkSeoRAxijFPX/CqMXkiRJkiT1jAGMUZpWt2AmFiCRJEmSJKlHDGCM0rQoeTBWZJLmwpAkSZIkqScMYIxS1ADGm0+8pM8pkSRp6omI3SPiLxExLyIO7TB824i4ICJ+HxF/iog9+pFOSZI0egYwRqlVB8Z1d9xvERJJksZRREwHjgVeAewI7BsRO7aNdjhwRmY+HdgH+NL4plKSJI0VAxij1CpCAtgKiSRJ42tnYF5mXpOZS4HTgL3bxklgw/p7I+CmcUyfJEkaQwYwRmlaDD6OJEnqia2AGxrdC2q/piOBN0bEAuAc4F2dZhQRB0XE3IiYu3DhwtWGzz707DFJsCRJGjkDGKMUzQCGOTAkSRpPnT4jtN+N9wVOzMytgT2Ab0TEas8/mXlcZs7JzDmbb775ajOdf/SeY5FeSZI0CgYwRikaEYwrbps9G/AAACAASURBVL67jymRJGnKWQBs0+jemtWLiBwInAGQmb8G1gE2G5fUSZKkMWUAY5SadWDse/zFfUyJJElTziXA9hGxXUTMpFTSeVbbONcDLwGIiCdSAhirlxEZAouRSJLUXwYwRsk6MCRJ6o/MXAYcDJwHXElpbeTyiDgqIvaqo70feGtE/BE4FXhT5siq3bYYiSRJ/TWj3wlY04UBDEmS+iYzz6FUztns95HG7yuA5453uiRJ0tgzB8YohREMSZIkSZJ6zgDGKE0zgCFJkiRJUs8ZwBgl68CQJEmSJKn3DGCMUnRsgl6SJEmSJI0lAxijZA4MSZIkSZJ6zwDGKFmJpyRJkiRJvWcAY5TMgSFJ0tQw+9Cz+50ESZKmNAMYo2QODEmSpob5R+/Z7yRIkjSlGcAYJXNgSJIkSZLUewYwRskcGJIkSZIk9Z4BjFEyB4YkSZIkSb1nAGOUzIEhSZIkSVLvGcAYJXNgSJI0ddgSiSRJ/WMAY5SmmQNDkqQpw5ZIJEnqHwMYo2T8QpIkSZKk3jOAMUqBEQxJkiRJknrNAMYoTXMLSpIkSZLUc75+j5I5MCRJkiRJ6j0DGKNkHRiSJEmSJPWeAQxJkqRhsClVSZL6wwCGJEnSMNiUqiRJ/WEAQ5IkSZIkTXgGMCRJkiRJ0oRnAGOUMvudAkmSJEmSJj8DGJIkSZIkacIzgCFJkjRMtkQiSdL4M4AhSZI0TLZEIknS+DOAMUqJlWBIkiRJktRrBjAkSZIkSdKEZwBjlLbaeN1+J0GSJPWB9WBIkjS+DGCM0qx11uLSw1/a72RIkqRxZj0YkiSNLwMYY2DGdDejJEmSJEm95Jv3GJgxLfqdBEmSJEmSJrWeBjAiYuOIODMiroqIKyPi2RGxaUScHxF/q/83qeNGRHwhIuZFxJ8iYqfGfA6o4/8tIg5o9H9GRFxWp/lCRPQlkjBjugEMSZKmIuvBkCRp/PQ6B8bngR9l5t8BTwOuBA4FfpqZ2wM/rd0ArwC2r38HAV8GiIhNgSOAXYCdgSNaQY86zkGN6Xbv8fp0tNY0M7JIkjQVWQ+GJEnjp2dv3hGxIfAC4OsAmbk0M+8E9gZOqqOdBLy6/t4bODmLi4GNI2IL4OXA+Zm5KDMXA+cDu9dhG2bmrzMzgZMb8xpX0yxCIkmSJElST/Uy68BjgYXA/0TE7yPiaxGxPvCozLwZoP5/ZB1/K+CGxvQLar+B+i/o0H81EXFQRMyNiLkLFy4c/ZpJkiRJkqRx1csAxgxgJ+DLmfl04D4eLi7SSadsDDmC/qv3zDwuM+dk5pzNN9984FRLkiRJkqQJp5cBjAXAgsz8Te0+kxLQuLUW/6D+v60x/jaN6bcGbhqk/9Yd+vfFE7fYsF+LliRJkiRp0utZACMzbwFuiIgn1F4vAa4AzgJaLYkcAHy//j4L2L+2RvIs4K5axOQ84GURsUmtvPNlwHl12D0R8aza+sj+jXmNu9c8Y+vBR5IkSZIkSSMyo8fzfxdwSkTMBK4B3kwJmpwREQcC1wOvreOeA+wBzAPur+OSmYsi4mPAJXW8ozJzUf39duBEYF3g3PrXF9bjKUnS1DT70LNtjUSSpHHQ0wBGZv4BmNNh0Es6jJvAO7vM5wTghA795wJPHmUyx4TxC0mSRi4itgIeQ+PZJDMvGsJ0u1OabZ8OfC0zj+4wzuuAIyl1Zf0xM98wRskGbEpVkqTx0uscGFNGKcUiSZKGKyI+DbyeUtR0ee2dwIABjIiYDhwL7EapG+uSiDgrM69ojLM9cBjw3MxcHBGP7Dw3SZI00RnAGCMWIZEkacReDTwhM5cMc7qdgXmZeQ1ARJwG7E0JhLS8FTg2MxcDZOZtq81FkiStEXrZCsnUYg4MSZJG6hpgrRFMtxVwQ6N7Qe3XtAOwQ0T8MiIurkVOVhMRB0XE3IiYu3DhwhEkRZIk9Zo5MMaI4QtJkkbsfuAPEfFTYGUujMw8ZJDpOt1+s617BrA9sCulyfVfRMSTM/POVSbKPA44DmDOnDnt85AkSROAAYwxMs0cGJIkjdRZ9W+4FgDbNLq3Bm7qMM7FmfkQcG1E/IUS0LgESZK0RjGAMUba4xeXzF/EM2dv2p/ESJK0BsnMk2qT6zvUXn+pAYfBXAJsHxHbATcC+wDtLYx8D9gXODEiNqvLuGZsUi5JksaTAYwxctvdq9Y7du+SZX1KiSRJa5aI2BU4CZhPKRayTUQcMFgzqpm5LCIOBs6jNKN6QmZeHhFHAXMz86w67GUR0Wrh5N8y847erY0kSeoVAxhj5IbF9/c7CZIkran+E3hZZv4FICJ2AE4FnjHYhJl5DnBOW7+PNH4n8L76J0mS1mC2QjJG7n5gKDldJUlSB2u1ghcAmflXRtYqiSRJmsTMgTFG7jKAIUnSSM2NiK8D36jd+wGX9jE9kiRpAjIHxhh5y/O263cSJElaU70duBw4BHg3cAXwtr6mSJIkTTgGMMbIcx+/2SrdNqoqSdLQZOaSzPyvzPzHzPyHzDwmM5cMPuXEMfvQs/udBEmSJj2LkIyRaUYsJEkalog4IzNfFxGXAdk+PDOf2odkjcj8o/fsdxIkSZr0DGCMkWlhBEOSpGF6d/3/yr6mQpIkrREsQiJJkvoiM2+uP9+Rmdc1/4B39DNtkiRp4jGAMUbMgSFJ0ojt1qHfK8Y9FaNkPRiSJPWWRUjGiPELSZKGJyLeTslp8diI+FNj0Czgl/1J1chZD4YkSb1lAGOMmANDkqRh+xZwLvAp4NBG/3syc1F/kiRJkiYqAxhjxFZIJEkansy8C7gL2BcgIh4JrANsEBEbZOb1/UyfJEmaWKwDY4yEOTAkSRqRiHhVRPwNuBb4OTCfkjNDkiRpJQMYkiSp3z4OPAv4a2ZuB7yENbAODEmS1FsGMCRJUr89lJl3ANMiYlpmXgD8fb8TNRK2RCJJUu9YB4YkSeq3OyNiA+Ai4JSIuA1Y1uc0jYgtkUiS1DvmwJAkSf22N/AA8F7gR8DVwKv6mqJRMBeGJEm9YQ4MSZLUV5l5X6PzpL4lZIyYC0OSpN4YNIAREU8GPgjsCCRwBfCfmfmnHqdNkiRNYhFxD+XZYmWv2h1AZuaGfUmYJEmakAYsQhIRewPfBS4E3gL8C6V5s2/XYZIkSSOSmbMyc8PG36zm/36nbzQsRiJJ0tgbrA6Mo4DdMvOEzPxTZv4xM08AdqvD1EVE9DsJkiStMSLieRHx5vp7s4jYrt9pGo35R+9pEEOSpDE2WABjrcyc396z9lurFwmSJElTS0QcAXwIOKz2mgl8s38pGhvWhSFJ0tgaLIDxUERs294zIh7DGtq8mSRJmnD+AdgLuA8gM28CZvU1RZIkacIZrBLPI4CfRMQngUspFWs9EziU8qVEkiRptJZmZkZEAkTE+v1OkCRJmngGDGBk5vci4lrg/cC7KLWCXw68LjP/OA7pkyRJk98ZEfFVYOOIeCul4vDj+5wmSZI0wQzajGoNVOw/DmmZVKzCU5KkocnMz0bEbsDdwBOAj2Tm+X1O1piYfejZ1oUhSdIYGTCAERGbAe8EFgMnAP8PeD5wNfD+zJzX8xRKkqRJKyKmA+dl5kuBSRG0aDJ4IUnS2BmsEs9vAWsD2wO/Ba4FXgP8EPhab5MmSZImu8xcDtwfERv1Oy29YnOqkiSNjcGKkDwqMz8cEQFcl5mfqf2vioh39jhtkiRpangQuCwizqe2RAKQmYf0L0ljx1wYkiSNjcECGMsBas3gt7cNW9GbJEmSpCnm7PonSZLU1WABjMdGxFmUOilbv6nd2/U0ZZIkadKrdWDslplv7HdaJEnSxDZYAGPvxu/Ptg1r75YkSRqWzFweEZtHxMzMXNrv9PSKrZFIkjR6AwYwMvPn3YZFxOlA1+GSJElDNB/4Zc3p2awD47/6lqIxZvBCkqTRGywHxkCePWapmISy3wmQJGnNcVP9mwbM6nNaJEnSBDWaAIYGkGkIQ5KkocjMjwJExKzSmff2OUmSJGkCGjCAERE7dRsErDX2yZk8DF9IkjQ0EfFk4BvAprX7dmD/zLy8rwmTJEkTymA5MP5zgGFXjWVCJh0jGJIkDdVxwPsy8wKAiNgVOB54Tj8TNdasyFOSpNEZrBLPF41XQiabNIIhSdJQrd8KXgBk5oURsX4/E9QLBi8kSRqdaQMNjIgPNn6/tm3YJ3uVqMnAKjAkSRqyayLiPyJidv07HLi234nqhdmHnt3vJEiStMYaMIAB7NP4fVjbsN3HOC2TigEMSZKG7C3A5sB36t9mwJv7mqIeMReGJEkjN1gdGNHld6duNRi/kCRpaDJzMXBIv9MhSZImtsFyYGSX35261WAzqpIkDU1EnB8RGze6N4mI8/qZpl6yGIkkSSMzWADjaRFxd0TcAzy1/m51P2Uc0rfGMnwhSdKQbZaZd7Y6ao6MRw5lwojYPSL+EhHzIuLQAcZ7TURkRMwZg/SOisVIJEkamQEDGJk5PTM3zMxZmTmj/m51rzVeiVwTmQFDkqQhWxER27Y6IuIxDOFbQERMB44FXgHsCOwbETt2GG8WpYjKb8YsxaNgDgxJkkZmsDowNGJGMCRJGqJ/B/4vIn5eu18AHDSE6XYG5mXmNQARcRqwN3BF23gfAz4DfGBskjs65sCQJGlkBitCMmoRMT0ifh8RP6zd20XEbyLibxFxekTMrP3Xrt3z6vDZjXkcVvv/JSJe3ug/pGyj4+UbB+688rc5MCRJGprM/BGwE3B6/XtGZg6lDoytgBsa3Qtqv5Ui4unANpn5w4FmFBEHRcTciJi7cOHCYaVfkiSNj54HMIB3A1c2uj8NHJOZ2wOLgQNr/wOBxZn5eOCYOh41K+g+wJMoTbd+qQZFhpRtdDw9f/vNV/42fiFJ0rA8B9i1/j1riNN0ahFt5S04IqZRnineP9iMMvO4zJyTmXM233zzwUYfNYuRSJI0fD0NYETE1sCewNdqdwAvBs6so5wEvLr+3rt2U4e/pI6/N3BaZi7JzGuBeZQsoyuzjWbmUqCVbXRCyIT7ly7rdzIkSZrwIuJoygePK+rfuyPiU0OYdAGwTaN7a+CmRvcs4MnAhRExnxIYOcuKPCVJWjP1OgfG54APAitq9yOAOzOz9WbfzOq5MhtoHX5XHb9b9tBBs4229CNb6Dcvvo4dP3IeN935wLgsT5KkNdgewG6ZeUJmnkDJcTmUN/xLgO1r8dSZlBybZ7UGZuZdmblZZs7OzNnAxcBemTl37FdBkiT1Ws8CGBHxSuC2zLy02bvDqDnIsOH2X73nOGcLBfj1NXcAcNcDD43L8iRJWsNt3Pi90VAmqB88DgbOoxRXPSMzL4+IoyJirx6kUZIk9VEvWyF5LrBXROwBrANsSMmRsXFEzKgPHc2snq1soAsiYgbl4WURA2cPHSjb6IQwfVqnOIskSWr4FPD7iLiA8oHiBcCHhzJhZp4DnNPW7yNdxt11dMkcW7MPPduiJJIkDUPPcmBk5mGZuXXNsrkP8LPM3A+4AHhNHe0A4Pv191m1mzr8Z5mZtf8+tZWS7YDtgd8ySLbRiWKFzZFIkjSgzDyVUj/Fd+rfs2u/SW3+0XtamackScMwHq2QtPsQ8L6ImEep4+Lrtf/XgUfU/u8DDgXIzMuBMyiVev0IeGdmLu+WbXRc12QIVqwYfBxJkqayiPhpZt6cmWdl5vcz85aI+Gm/0zUeDGJIkjR0vSxCslJmXghcWH9fQ2lBpH2cB4HXdpn+E8AnOvRfLdvoRGMODEmSOouIdYD1gM0iYhMert9qQ2DLviVsnLWCGBYnkSRpYOMSwJjKDGBIktTVvwLvoQQrftfofzdwbF9S1CcGLyRJGlw/ipBMKSuMX0iS1FFmfj4ztwM+kJnbNf6elplf7Hf6xptFSSRJGpg5MHrMHBiSJA3qrojYv71nZp7cj8T0i7kwJEkamDkweiwNYEiSNJhnNv6eDxwJ7NXPBPWLuTAkSerOHBg9ttxWSCRJGlBmvqvZHREbAd/oU3L6ylwYkiR1Zw6MHrMIiSRJw3Y/sEO/E9Ev5sKQJKkzc2D0mAEMSZIGFhE/AFo3zOnAE4Ez+pei/jIXhiRJnRnA6DHjF5IkDeqzjd/LgAD27VNaJoTZh55tIEOSpDYGMHpsue2oSpI0oMz8eUT8PfAG4HXAtcC3+5uq/jJ4IUnS6gxg9JhFSCRJ6iwidgD2oeS2uAM4HYjMfFFfEzZBtOrCMJghSVJhAKPHjF9IktTVVcAvgFdl5jyAiHhvf5M0cRi4kCRpVbZC0mPmwJAkqat/Am4BLoiI4yPiJZT6LyRJklZjAKPHrAJDkqTOMvO7mfl64O+AC4H3Ao+KiC9HxMv6mrgJxGZVJUkqDGD0mJV4SpI0sMy8LzNPycxXAlsDfwAO7XOyJgyLkkiSVBjA6LGfXXUr1yy8t9/JkCRpjZCZizLzq5n54n6nZSIxF4YkSQYwxtyX99tple4z5i7g5Z+7qE+pkSRJk8H8o/c0iCFJmvIMYIyxZ2636Wr9HlpuMRJJkjQ6FiWRJE11BjDGWKeq02dOdzNLkqTRMxeGJGkq8816jEWsHsLYcN21+pASSZI02bSKkhjIkCRNRTP6nYDJZkWW4iIzpgXLagskZsCQJEljxaIkkqSpylfrMbbZBmtz+J5P5NSDnrWy3/QOuTIkSZJGw1wYkqSpxgBGD/zL8x/Llhuvu7J7+nQDGJIkaWyZE0OSNNUYwOiRZq4Lc2BIkqSxZg4MSdJUYwCjR6Y1YhbTphnAkCRJY6tVoackSVOFAYweCXNgSJKkHrMYiSRpKrEVknEw3RwYkiSpR5q5MAxoSJImMwMYPdJqThVgWgT3LVnGWtOnMXOGmV4kSdLYaQYtZh96tkEMSdKk5dt0jzRLjUyfFjzpiPPY72sX9y9BkiRp0jN4IUmazMyB0SOPnLXOyt+tSjwvmb+4X8mRJElTRHvFngY1JEmThQGMHpo5YxpLl61gulVgSJKkcdIesGgFNAxkSJLWdBYhGQdW4ilJkvpl/tF7rmxytRXMsPlVSdKayBwYvVTr8bzj3qX9TYckSZryWjkwWhV9mjNDkrSmMQfGOLjm9vv6nQRJkiTg4YBFM2eGJElrAnNg9NAbdtmWE381v9/JkCRJ6qo9iGGODEnSRGUAo4c+8sodDWBIkqQJrxm0MJghSZqoDGD00DQr75QkSWuYbsGM9mGSJI03AxiSJEnqqFuTrEMdX5KksWQAQ5IkSUPSLUDR3rLJQONKkjRSBjDG2e+uX8xO227S72RIkjQpRMTuwOeB6cDXMvPotuHvA/4FWAYsBN6SmdeNe0InuWbLJi1Dad3EIIckaTgMYIyzt540l0v/Y7d+J0OSpDVeREwHjgV2AxYAl0TEWZl5RWO03wNzMvP+iHg78Bng9eOf2qlnKMGJ4TTharBDkmQAY5xlvxMgSdLksTMwLzOvAYiI04C9gZUBjMy8oDH+xcAbxzWFGtBwghLdgh0GNiRp6jCAMc4yDWFIkjRGtgJuaHQvAHYZYPwDgXM7DYiIg4CDALbddtuxSp/G0ED1bwxnfEnSmssAxjhbYfxCkqSx0qm98o532oh4IzAHeGGn4Zl5HHAcwJw5c7xbr0E6BSpmH3q2FYpK0iRkAGOc3fXAQ/1OgiRJk8UCYJtG99bATe0jRcRLgX8HXpiZS8Ypbeqj0TT/2mpRpZvBhkuSescARh/cv3QZ681000uSNEqXANtHxHbAjcA+wBuaI0TE04GvArtn5m3jn0RNBIMFHNpzawwU8OjUXGyru/XbAIck9YZv0X2wdNkK1pvZ71RIkrRmy8xlEXEwcB6lGdUTMvPyiDgKmJuZZwH/D9gA+N+IALg+M/fqW6I1IbUHHAYLQHTLsdH63R4AGU4OD0lSdwYw+mDpshX9ToIkSZNCZp4DnNPW7yON3y8d90RpSmkGI1q/ByrCMliAQ5LUnQGMPlhiAEOSJGnKGCyHx2B1dHSbTpKmGgMYfXDXAw+xdOG9PG7zDfqdFEmSJPXZUAIT7S2rjGQekrSmM4DRB+869fdce/t9/PmjL2eDtd0FkiRJGthQKyI1kCFpMvPtuQ+uvf0+AB5atgLW7nNiJEmStMZrViTa3t+KQyVNFgYw+mhaqQ1dkiRJGhOd6tfo1jSsQQ1Ja5qeBTAiYhvgZODRwArguMz8fERsCpwOzAbmA6/LzMVR2jb7PLAHcD/wpsz8XZ3XAcDhddYfz8yTav9nACcC61JqIH93Zmav1mmsLV9zkipJkqQ1ULeWUWDolYd2m16Sxlsvc2AsA96fmb+LiFnApRFxPvAm4KeZeXREHAocCnwIeAWwff3bBfgysEsNeBwBzAGyzueszFxcxzkIuJgSwNgdOLeH6zSmlq2wNRJJkiT1x3CCEoNVIGoxFUnjoWcBjMy8Gbi5/r4nIq4EtgL2Bnato50EXEgJYOwNnFxzUFwcERtHxBZ13PMzcxFADYLsHhEXAhtm5q9r/5OBV7MGBTCMX0iSJGlNMFBwYqBiKoNNK0nDMS51YETEbODpwG+AR9XgBpl5c0Q8so62FXBDY7IFtd9A/Rd06N9p+QdRcmqw7bbbjm5lxpBFSCRJkrSmG6iYCnTPvWFgQ9Jw9TyAEREbAN8G3pOZd0f3iis7DcgR9F+9Z+ZxwHEAc+bM6UvU4KVPfBQ/ufLWVfqtWGEAQ5IkSZPbcAMbg00naerqaQAjItaiBC9Oyczv1N63RsQWNffFFsBttf8CYJvG5FsDN9X+u7b1v7D237rD+BPKTttuzO+uv5Pp01YfttwAhiRJkqaowQIU5tyQ1K6XrZAE8HXgysz8r8ags4ADgKPr/+83+h8cEadRKvG8qwY5zgM+GRGb1PFeBhyWmYsi4p6IeBalaMr+wH/3an1G6ox/fTbLM3nPaX9YbdiBJ13CWQc/j/XXtjVbSZIkqWk4OTesSFSaGnr55vxc4J+ByyKi9fb+YUrg4oyIOBC4HnhtHXYOpQnVeZRmVN8MUAMVHwMuqeMd1arQE3g7Dzejei4TsALPGdOnMQN43TO34dw/37LKsKsX3sc5l93Ma+ds03liSZIkSavo1iTsQBWJdptO0polcopVJDlnzpycO3du35bfflH97GufxmuesXWXsSVJWnNExKWZOaff6Ritfj8rSOoNc25I/TfaZwXLLvTZtK51mkqSJEkaKyPJuWFgQ5pYDGD02fRGBCMzWbYiWatTjZ+SJEmSxtRImoA1qCH1jwGMPms2K3vsBfP47I//yp+OfBkbrrNWH1MlSZIkqVuujaGMJ2nsGcAYZ4/ecB1uufvBld3NIiSnz70BgMX3LTWAIUmSJE1AQw1qdBtX0sgZwBhnP37fC/jRn2/hg2f+CYCgWYSE1fpJkiRJmtiGWgTFgIY0OgYwxtmG66zF1pusu7K7UYLk4QCG8QtJkiRpjdcesLCZV2l0DGD0wfRGhGL5itWbsZ1iLdtKkiRJU8JAQQrr1pAGZwCjD5otj1x241286mlbAqUVEoDlRjAkSZKkKWU4dWt0G1+a7Axg9MG0RgDjuIuu4Z0vejwbrftwpZ3LV6zoR7IkSZIkTSDm2JBWZQCjD6a3VXKxdFkJWLTyXSzrUKxEkiRJklps4lVTkQGMPpjWFsD44w13cvj3/ryyedVlyw1gSJIkSRqeTpWGNoMa84/ek9mHnm1QQ2ssAxh9sPj+pat0/8vJc1fp7lSxpyRJkiQNR6eARiuIMdRppInEAEYf7LzdprzoCZtzwV8WdhzeLEKyZNlypkWw1vRp45U8SZIkSZNQKzgx3Lo12qeX+sUARh+ss9Z0jt1vJ3b8yHkdhy9bvoL/nXsDr3jKFjz5iPN44hYbcu67nz/gPK+65W4uuGohb9/1cb1IsiRJkqQpYCTBDQMbGi8GMPpk5gA5Kr77+xs57ZIbmDt/MQBX3nx3x/EuW3AXW22yLpuuP5O9v/hLlixbwUEveOwqzbRKkiRJ0ljoFqgwsKHxYgCjT2ZMn8aX99uJt5/yu9WGnXbJDQDccd+SAefxqi/+H7MfsR4XfGBXltSWTJYsW856M4e2W3999R3se/zFXPCBXdlus/WHuQaSJEmSNPTAhgENjZYBjD5ad+b0AYc3672Yf/t9XL3wXr560TWc/JadOfW315f+d9zPdoeds3K8Bx9awXozB17uv3/3Ms68dAF7PnULAC6Zv4jtNlufn111K899/GasPWPgdEmSJEnSYJoBi/YWUQYaV+rGAEYfbf+oWQMOP/fPt6z8vdsxP+eh2rzqlTffzUd/cEXHac6/4hZe+4xtuHrhvV3nf8pvSvBjac21MXP6NH53/WLecuJcXvW0LfnBH2/iv/d9Oq962parTXv0uVfxwh0259mPe8TgKyhJkiRJDB6gGCi40ZzeZmCnNgMYfbTVxusO2oxRSyt4AfAPX/pV1/E+9O3LWHz/Qxx97lWcfcjzeNKWG3Ud9+4HlwElp8fi+0rTrhf9tbSMcvwvrukYwPjKz6/mKz+/euVF49a7H+TeJct43OYbDLoOkiRJktTJcAIcrd+d3qUMbkxuBjAmkFc+dQt++KebRz2fufMXAXDDovtXBjCuuOluzr7sJt6/2xNWjtcKVjQr/Yz6c1ENaDQ9tHzFav12+eRPAfjXFz6Ww17xxGGl854HHyKB+5YsY4uN1h3WtJIkSZKmjk6BiU65MaxQdHIzgDEBzJgWLFuR7LvztmMSwLjrgYcAuG/J8pX99vjCLwA49oKrVxt/ybLlzJxRDoV7aq6MBYsfYPahZ/ODg5/HU7YuQZAHHlq+2rQtX/35NRzy4u1Zf+0yn9vufpBZ66w1YD0fTz/qfJatKDlL/vdtz+aZszcd8jpKkiRJmtr+f3t3Hl9Vcf9//PXJvkGAEPZ9X5Q1IigqClVcqS1W/PpFba3Y2qq1ta6tRVv9We23Vq22peBuwV1RLCgiCir7B592ogAAIABJREFUvhPWhAABkpCQkH1+f9yT683CEhLMTfJ+Ph73kTtz5pw7c05yMvlkZk5VQQk9KaVhUwAjCKx46Hs4B3GRYfx8VHd+MLg933vqy5Pev1frOLbsP+JPL/Uev7p0ZwZr9xymadTxL/OdM1b535eUunLbrvz7Qv8Pdf5xAhgA6TkFdI0MIyO3kGGPfcb/nN2J317cm+iIUKLCKwcyigM+a8v+HAUwRERERETktKhuYONE+0ndUAAjCDSNCve/v3dsHwBuGNGZV77ZBUDnhBh2Hco75v4X9mlVLoBRpuxxrDWVnJ5D+2YxTJy6xJ/3/PxkbhvVo1y59Ox8uraMZfsBX10+XJ3GfxbvZmCHeD745cjjfkaoN3elpNQxb1M6Y/q2wsyOu4+IiIiIiEhN1HRx0ZM9jtQOBTCC1CPjzmBkj5ZMenU5PRLjGN2nNS98tYOHr+rPH2auB6B5TDiZeUWUVhg1UdvG/LXyaJAnZm+mb5um5fKunbKIV28exq+8ER1l01FWpx6muKSUpTsz2ZN1lI17s7njop7l9g0JMfIKi3l7eSoPfbCepycMYtyg9qepRSIiIiIiIid2soGJkw10VPe4Up4CGEGsU0IMAH3aNuG3l/Thp+d1pV2zaNo3i2ZlSiatmkTxh5nriQ4PZWz/NkRHhNKtZSzn9Urk+899ddrr9+OXllbKmzhtSRUl4eaXl/GFt2gowLSFO8ptn785nXveXuNP/2fxbi7p36bKqSclpa7cwqPH8vLXO+nbtinDumpqioiIiIiInD7VDUhUFfBQUOPEzLnT+9/7YJOUlOSWLVtW19U4aev2HKZ3myaEh4ZU2lZa6pixNIWL+7emZVxkuW3bDhzhX19sY1VKFtNvGU5CXCRrUrNYvD2DcYPbcf87azlaVMLX2w6V2+833+vFz0d151BuIdMW7mD2un3szjj29JXT6aZzuvDri3uRnl1ARGgIUREhPP7xJpbuymDBPRedcP/AxyuJiMjpZ2bLnXNJdV2PmqpvfQUREWkYGsN0lZr2FRTAaORSMvJIbBJJn9/P5reX9OYXF5Zf12Lx9kNcO2URALN/dR6b9uYQFR7Cz15b4S9zz9jefL4pnRAzFu/IqNX6dW0Zy46DuUSEhlAY8BjX313el6sHtyehQuAmkAIYIiLfLQUwRERETr/6/ESVmvYVNIWkkevYwjdN5Vjf7F0TYwF4YvwA+rRpSp82TSkqKWV4txYs2u4LVvzs/O7cNqoHzjneWbGHu99aDcBFfVoxb1O6/1hDOzdn+a7MatVvx8FcgHLBC4A/zdrIn2Zt5OM7zuOyZxbwn1vOJsSM4d0SKCguYeHWg9X6HBERqZ/MbCzwNBAKTHXOPV5heyTwCjAUOARc65zb+V3XU0REpLac6hNVanLsYKERGHJCzrkqnwhSVFJKbkExzWIiyuWX/eBs/tNY7n5rDR+uTiM6PJSNfxzLoSMFDP3TXODUAhoVVXxCy2s3n80Hq/bw1vJUf96mP45l5qo0rknqoCebiIicRt/1CAwzCwW2AN8DUoGlwHXOuQ0BZW4DBjjnfmZmE4CrnXPXHu+46iuIiEhjdbwgSG0ENzQCQ067Y/3RHx4aUil4AfD89UMIDTEiw0J5cvwAPlydxi3ndwMgNjLMO6Yv2BARFkL3Bz5mSKdmrNidBcAPBrdnT9bRk5qOUvHxsv87bXGlMn+Zs5mpC3eQ2CSSC/u0qrQ9O7+IzNxCOifEnvDzREQkqAwDkp1z2wHMbAYwDtgQUGYcMNl7/zbwdzMz19j+gyMiInISjhekOJURHrU9oqPyypAiNXTZmW25pH8bAKLCQ9n66KXcNaanP73gngvZ9MexREeEEhpifHrX+bz207MBuGtML/567SCm3JDEG5OG87vL+/qPO6RTM97+2Yhq12eq98STfdn5HDxSQGpmHl3um8XlzywA4IZpS7jgyfl8sGoPeYXFJzzelv05vPz1zmrXQ0REal17ICUgnerlVVnGOVcMHAYSKh7IzCaZ2TIzW5acnIyZ+V/Lly9n+fLl5fImT54MQLt27fx5Q4cOBWDSpEnlyqalpfHhhx+Wy5syZUrZ5/pfV155JQBXXnlluXyAKVOmlMv78MMPSUtLK5c3adIkAIYOHerPa9euHQCTJ09Wm9QmtUltUpvUphq16cz1/2TXn6/wv3Y+fjkPdNtTLu/Zc0v5+o7B/rSZ0WTQWLrcN4vINj2IaNPDV5lTpCkkEvSenruVp+Zu4ZFx/blhRBcWbD3Agq0HmfLl9hofe+MjY+n70Gx/etygdjw9YTCFxaUUlpRyMKeALi3Lj8zodv8sSh1sf+wyQk7ica4iIo2FffdTSK4BLnHO/dRLTwSGOeduDyiz3iuT6qW3eWUOVXVMUF9BRETkdKlpX0FTSCTo3TmmJ7de0I3IMN+AofN6JnJ214TjBjBG92nFZwELiB5LYPAC4INVaXywKq1c3oAO8fRr25THfziAtKyjlHoxv24PfFxuSNSx1go5lqy8Qv8UnPyiEqYt3MEt53UjIkwDo0RETlIq0DEg3QFIO0aZVDMLA+KB2n1kloiIiHwn9JeS1AtR4aHlggMRYSG8dvPZ/nTyo5ey/uFLeHrCIN68dQQPXN6Xe8b2Jj46vMafvSb1MDOW+kYon/P4vErbnXPMXrePrvd/zE7vqSmBFmw9wOG8onJ5s9bsZdAjn7I6xbfux7SFO3hyzmamL9ld4/qKiDQiS4GeZtbVzCKACcDMCmVmAjd678cD87T+hYiISP2kERhSb43s2ZIXf3wWsRFhhIWGEBYawrhB3059vm1UDwCemL2Zv1wzkOlLdtfoqSdVLVqTnpPPsEc/86efmruF2y/qQXhoCPuzC4iJCGXitCWc3yuRV34yzF/us037Afjl9BW0jY/mnO6+6dgHcgpOuX4iIo2Nc67YzH4JzMH3GNUXnHPrzewRYJlzbiYwDXjVzJLxjbyYUHc1FhERkZrQGhjS4JWWOkJCjHV7DnPTi0s5eKRykKBVk0jST3Pw4LeX9Oa2Ud1JO5zPBU98TnHptz97913ah8f/u4mkzs25YkBbbjq3a7l9cwuK+fPsTdx9SW8iQkNYvCODC3olntb6iohU13e9Bsbpor6CiIjI6aE1MEROoGyhzTPax7Psd2PYcTCX7QeOsGxXJj8Y3J74mHB+8PzX/vIPXdGPRz7yPYHvo9tHMnXBdt5fVXFKdfU9OWczT87ZXOW291fuAWDZrkyW7crk6iEdSMnII7+ohKQuLXhzWQqvfLOL2MgwiktK+feCHUy9IYnoiFCun7qYN28dwbCuLY77+V9uOcCa1Cx+eVHPE9Z13qb9/OzVFSz//RiaRNV8Go6IiIiIiEhNaQSGCLBidybPfraVKTckER4awqvf7GRPVj73XdrHX+bJOZt47vNt/vSYvq3olhjnX0z0xZvOYu7G/by+uPI6Fi1iI8jILTylui1+YDRnP+abpnLFgLZ8tGbvMcuO6p1IWEgI/zu8EyFmbNybza0XdAe+nQIz7cYkWjWJ4t2VqSzdmcFHt59X6ThXPruQtXsO895t5zC4U3O+2HKAHq3iaN8susrP/XTDfrbsz+EXF/aoVtumL9lN+2bRnH+So0lSM/NYuTuLKwe2q9bnBFqxO5OE2Ag6J8SeuLCIVItGYIiIiMjxaASGSC0Y0qk5L/742zUqJo7oUqlMQmwkAHeO7klKZh6/v7wfzWMjuOmcLoSFGK2aRjGyZ8tyAYxRvRO5c3RPBnVsRtf7P/bn33pBN/71xck9BrYseAEcN3gBMH/zAQDmbtzvzzuzfTwDOzbzp29+uXyn/P2Ve9idkccdo78dmVH2dNiUzKOc2T6eG19YQtv4KKbemMTerHy6t4rjL59s5snxA4iJCOOWV3zHvG1Ud+ZvPsBznyfTJj6KicM789z8bdwwvDNj+rWuVN/7313ra9ftI1mxO5MbqjjvgX74j6/Zn13ApWe0ISz01NYgLhttE/gEGRERERERCX4KYIicpIkjOlPqHBNHdCYyLNSf3y5gVEJ4aAjTbxlOeKhRWFxK//bx/iehfH73KD7dsI9bzutGUYnzBzAWPzCaf8zfxprULFbsziI81Hjz1hFcHTCtpSb+Z+ri427/1RurfOXO7sTcDftJycxjdephAO6YvpLFZ3cCYO/hfC5/ZmG5fWet2cvFAYGJK/++kHV7sv3psoDLl1sO8J9bzuac7i3Jyitk+pIUuiTE+Mtd8azvuJFhIfwoqWO5J86UlDr+75PNjOnXmv3ZvnVK/rtuHxf2acWGtGxeWLiDJ64ZQNOocN5bmcq5PVqy82Aez87bSm5BMe/edm61zteGtGziIsNoGh3G7ow8BnRoduKdRERERETktNMUEpE68p/FuzmnewJdWvqmMjjn2HkojxaxEcRHh7N5Xw6//2Adv7iwBze+sKTcvq/ePIyJ05ZUdVguH9CWWScYqVFXfntJ72OuAxKoV+s4Jl/Vny37cpj84YYqy7SMi+DgkcJK6aZRYWTnF/vzN/1xLFHhoUyeuZ6Xvt7pz9/5+OXMXreX7QdzmTi8s3+tj7KpNsO6tGDJzgw2/XEsGbmFZOcXsSb1MD9K6lipLkcLS/hkwz6uGtiOwpJSIsNCSc/J55z/N4/fXd6Xa5I6EhsZxqw1e/liSzrDuiZw6RltiI0M4/NN6fRp24TmMRFEhfsCYy99tYOEuMgTTpV5bdEuwkONa8/qdMJzejrkFhTz7opUrj+7s3+tGWncNIVEREREjqemfQUFMETqgez8Iv4xfxtNo8IZP7QDiU0ieXLOJpI6t2BAh3gS4iIpKC6huMQRGxnGQx+sIy0rn+0Hj7D9QC4dW0Tzzs/PYfriFJ6auwWANZMv5tZXlvPN9kOVPu+M9k0Z2SORf36xrdK2EIPSatw2mkSGkVNQfOKCp9HAjs1YnZJVLu+HQzrwzorUE+7bu3UTNu/P8acv6tOKO0f3pKC4lI/WpLE/O585631Tdu4d24c/z97E7y7vy8qULH8gqVlMOA9f1Z87Z6zyHyepc3M2788hJyDYMnF4Z+69tA9n/GEOAP++IYmhnZvz5rIUrhrYjjAvSJDYJJKP1uzl9ukrAV+Q5uEP13NWlxZ0aB5DYpNInvs8mRHdEoiPDmdkz5b+4MgTszfRtWUs5/dKJD46nMU7Mpi7YT93julJRFgITb1ATnFJKXlFJf40+IJsZaNj8otKuHbKIlanZDFj0nCGd0vwl5m5Oo1L+rfxfybAqpQsjhaWMMJ7ZHCZqQu2s+9wPr+7ot8Jr4UEPwUwRERE5HgUwKgmdUqksZu1Zi+fb07nL9cM5EBOAV9vO8iZ7ePp0DyG91amcu87a5n3mwvolhjH4aNFFBSXEB8dTt/fz+bcHi359w1JTJy2mKU7M/nkrvM5mFPA3+ZuZcnODDonxPDY1Wdy/dTFDOvagrbxUTw5fiAzlu7moQ/WV1mfNZMvZsDkT77js1C/dWwRTUrG0ZMuP7BjM+65pDfXn2A6EcDY/m34ettB/yiW568fwm2vryhX5sLeiXzurbcC8ONzu9CzVRPOaN+UJTsy+NOsjZzTPYEXbjqLEDNWp2ZxzT+/AXyBoyfHDyAkxFi2M4PxXv57t53DTS8uZdL53diQls2stXuZf/coXlu0i9F9W+OcY0/WUX44pANTF27nsY83cW6PBH46shsjuifwwao9lDq4enB7dh3Ko1tiLBm5hbRuGgVASkYekeEhfLYxnQlndWRP1lHaN4smNfMoy3ZlUFoKh3ILaBYdQalzjD2jDbsO5dEkKoxuiXGVzlNyeg5mRvfEON5fuYeVuzN5eNwZgC/4Expi5aZClXHOkZFbSEJcZLn8/KISCktK/QGj7PwiDucVYQYdmsdUOk6ZBVsPMKJbQpVrwizbmcEz85KZekMS4aFGqYPQEKPEi0CGnoZRMwpgiIiIyPEogFFN6pSIHF9JqTvhHzbFJaWUOogIq3ohzaU7MxjaqXmlaQWfrN9Hu2bR9GvblPVp2RwtKmFY1xYcPFLAvW+v4bNN6ZjBazefzfaDuaRm5DGiewJt4qPILShm3+ECYiJDuaBnIlvTj/D4fzfy6NVnEhsZxoKtBxjSqTlTF+zg7eUp5aaRAFw3rBPdWsby5dYDtG4aRagZbyxLOW47h3drwaLtGSdx1mpuUMdmrKowSqS+iwwLoaC49LR/zoAO8azx1m2pSnUDPhXdOKIz5/ZoCcCc9ftxzvGu9+jjV34yjBu8KV4v3nQWKZl5PPTBeqLCQ7j8zHbM3bifZ64bzAW9Ejl4pIBfv7maL7ccYECHePq0acLcjekUl5SSnV9MTEQoP0rqyKuLdvmDDOB7itFlZ7bh6sEd2HUol44tYmgZF8mzn23l3ZV7+Pmo7nRtGcur3+xiRPcErhnagT/N2sgXW3xBpsDr8PSEQUxfspvUzKMsvPeiUz4nx6IAhoiIiByPAhjVpE6JSPDKKywmNMTKLZJ6KvKLSkjJyKNn6yakZOTRoXl0lf8Nz8wtpLjUkRAbwe6MPOKiwjhaWEKb+Cg2pGUzsGMz8gqL2Z9dwNo9h7nDm7LRtWUsE4d35vrhnViyI4M1qYeZsXQ34SEh/PKiHkxfspuVu7NoERvBkgfHkF9Uws5DuSzadohrz+rEyt2ZvPLNLnq0iuOrbQcZ3acVPx/Vg+W7Mnn5m5383zUD+fu8ZP7+ebK/rndc1INn5iVz26juPD/fN7Xn/V+cy6OzNrB0Z6a/XHR4KP3aNeX564f4p23kFhbz+H83+aerNIkK46cju5GamceRgmKKShxzN+7ngl6JbNybTXpOQY3Ov1QWHR7K0aKSuq5GOWXrw9QmBTBERETkeBTAqCZ1SkTkVJWWOkqcI/wkHuFaVFJKqXM1CsYcLSxha3oOHZvH0Dw2guKSUsJCQygpdRzJLyY+Jtxfr7L4jHNUe0FN5xzpOQX+6RbOOfZl59M2Ppr8ohJSM4/SJj6KuMhvH1y1bs9h/r1gOw9e1pd92fl8lXyI3m3iOFpYSm5hMWe0i2fv4aPkFpaQGBfJiO4JHDxSwOoUX2Bn/uYDjB/agbSso7RrFs3M1WnsPXyU0X1a0yQqjKQuLdh7+CjFJb6pIwBvL09l075skjq3oEerOFIzj3JBr0Su+/cirhvWiTPb+z7z9cW7aRoVRvfEOG69oDsfr93Lme3jyS0spl/bptz91mr6tWvKE+MHEhZifP+5r9i0L4cnfjiAbQeP0CImgvScAnILinlzWQr3ju3DG8tS2H4gly4JMew8lAdA/3ZNGdChGe3io1iz5zCfbvCthTLhrI7MWFr16J4z2jflvJ6JfLH5AMO6tuCd5al0aBHDhLM6sutQHh+v3cu+7PyTWmtmbP82HCkoZmHywWOW+c33erEu7TBzN6YTYlBU8u1Bv77vonJPUaoNCmCIiIjI8SiAUU3qlIiISE1k5RWSk19Mxxbl16ZwzrFl/xF6t2nC4bwi4qLC/NOxnHNk5hXRIjbipD4jcMFU8E3tKi71PeEmcIQR+IJlaVlH6ZwQ6y97rGlg6Tn5JMZFVjkiqTYogCEiIiLHU9O+QtiJi4iIiEiZZjERNIupHIgwM3q38QUVykbHBG472eBFWflAoSFGaIhvNE9UeKg/eAEQHhriD16UlT2WVk2iTroOIiIiIsHmxOOgRURERERERETqmAIYIiIiIiIiIhL0FMAQERERERERkaCnAIaIiIiIiIiIBL16H8Aws7FmttnMks3svrquj4iIiIiIiIjUvnodwDCzUOA54FKgH3CdmfWr21qJiIiIiIiISG2r1wEMYBiQ7Jzb7pwrBGYA4+q4TiIiIiIiIiJSy+p7AKM9kBKQTvXyRERERERERKQBqe8BDKsiz1UqZDbJzJaZ2bIDBw58B9USERERERERkdpU3wMYqUDHgHQHIK1iIefcFOdcknMuKTEx8TurnIiIiIiIiIjUjvoewFgK9DSzrmYWAUwAZtZxnURERERERESklplzlWZc1CtmdhnwNyAUeME59+gJyh8AdtVyNVoCB2v5mMGmMbQRGkc7G0MboXG0szG0ERpHOxtKGzs75+r9UEczywE213U9GqmG8rNQ3+i81x2d+7qjc183ejvnmpzqzvU+gBEMzGyZcy6prutxOjWGNkLjaGdjaCM0jnY2hjZC42hnY2hjfaLrUXd07uuGznvd0bmvOzr3daOm572+TyERERERERERkUZAAQwRERERERERCXoKYNSOKXVdge9AY2gjNI52NoY2QuNoZ2NoIzSOdjaGNtYnuh51R+e+bui81x2d+7qjc183anTetQaGiIiIiIiIiAQ9jcAQERERERERkaCnAIaIiIiIiIiIBD0FMGrIzMaa2WYzSzaz++q6PqfKzDqa2edmttHM1pvZnV5+CzP71My2el+be/lmZs947V5jZkPqtgUnz8xCzWylmX3kpbua2WKvjW+YWYSXH+mlk73tXeqy3tVhZs3M7G0z2+Rd0xEN7Vqa2V3e9+o6M5tuZlEN4Vqa2Qtmlm5m6wLyqn3tzOxGr/xWM7uxLtpyLMdo45Pe9+saM3vPzJoFbLvfa+NmM7skID9o779VtTFg291m5syspZeul9exIQrm76mGoLbub1J91oj6ecHE65ssMbPV3nl/2Muv9/2V+sIaQb8/GJnZTjNba2arzGyZl1cr9xsFMGrAzEKB54BLgX7AdWbWr25rdcqKgd845/oCw4FfeG25D/jMOdcT+MxLg6/NPb3XJOAf332VT9mdwMaA9J+Bp7w2ZgI3e/k3A5nOuR7AU165+uJpYLZzrg8wEF97G8y1NLP2wB1AknPuDCAUmEDDuJYvAWMr5FXr2plZC+APwNnAMOAPZb8kgsRLVG7jp8AZzrkBwBbgfgDvPjQB6O/t87zXGQn2++9LVG4jZtYR+B6wOyC7vl7HBqUefE81BC9Rw/ubnLLG1M8LJgXARc65gcAgYKyZDadh9Ffqi8bQ7w9WFzrnBjnnkrx0rdxvFMComWFAsnNuu3OuEJgBjKvjOp0S59xe59wK730Ovh/09vja87JX7GXg+977ccArzmcR0MzM2n7H1a42M+sAXA5M9dIGXAS87RWp2Maytr8NjPbKBzUzawqcD0wDcM4VOueyaGDXEggDos0sDIgB9tIArqVz7ksgo0J2da/dJcCnzrkM51wmvuBApT+m60pVbXTOfeKcK/aSi4AO3vtxwAznXIFzbgeQjO/eG9T332NcR/B1iu4BAlfQrpfXsQEK6u+phqCW7m9yChpLPy/YeOfviJcM916OBtBfqQ8aQ7+/nqmV+40CGDXTHkgJSKd6efWaN2RqMLAYaO2c2wu+X35AK69YfW373/D98VDqpROArIA/nALb4W+jt/2wVz7YdQMOAC96Q+ammlksDehaOuf2AH/B91/svfiuzXIa3rUsU91rV++uaQU/Af7rvW8wbTSzq4A9zrnVFTY1mDbWczrfdaPB/G6qLxp4Py/oeKMGVwHp+ALR22i4/ZVg0xj6/cHKAZ+Y2XIzm+Tl1cr9RgGMmqkqKlevn0trZnHAO8CvnHPZxytaRV5Qt93MrgDSnXPLA7OrKOpOYlswCwOGAP9wzg0Gcvl2iFZV6l07vWH044CuQDsgFt/ws4rq+7U8kWO1q96218wexDfU+fWyrCqK1bs2mlkM8CDwUFWbq8ird21sAHS+g4uux2nQkPt5wco5V+KcG4RvZOEwoG9VxbyvOu+1pBH1+4PVuc65Ifj6578ws/OPU7Za514BjJpJBToGpDsAaXVUlxozs3B8v9Red86962XvLxvC431N9/LrY9vPBa4ys534hgZfhC8y28ybhgDl2+Fvo7c9nqqHhAebVCDVObfYS7+NL6DRkK7lGGCHc+6Ac64IeBc4h4Z3LctU99rVx2uK+RapvAK43jlX9ourobSxO76A22rvHtQBWGFmbWg4bazvdL7rRkP63RTUGkE/L6h503nn41uDpKH2V4JJY+n3ByXnXJr3NR14D1/wrlbuNwpg1MxSoKe3mm0EvoXmZtZxnU6JN8drGrDROffXgE0zgbKV728EPgjIv8FbNXY4cLhsSFCwcs7d75zr4Jzrgu9azXPOXQ98Doz3ilVsY1nbx3vlgz4S65zbB6SYWW8vazSwgQZ0LfFNHRluZjHe925ZGxvUtQxQ3Ws3B7jYzJp7o1Uu9vKClpmNBe4FrnLO5QVsmglMMN/q4F3xLfC0hHp2/3XOrXXOtXLOdfHuQanAEO/ntcFcx3quXn1PNSAN6XdT0GoM/bxgZGaJ5j1Vy8yi8f0DZiMNt78SNBpLvz8YmVmsmTUpe4+v/7KO2rrfOOf0qsELuAzfivnbgAfruj41aMdIfEN11gCrvNdl+OZ+fQZs9b628MobvtXatwFr8T0Nos7bUY32jgI+8t53w/cHUTLwFhDp5Ud56WRve7e6rnc12jcIWOZdz/eB5g3tWgIPA5u8G+KrQGRDuJbAdHzrehTh+yP35lO5dvjWkUj2Xj+u63adRBuT8c1/LLv//DOg/INeGzcDlwbkB+39t6o2Vti+E2hZn69jQ3wF8/dUQ3jV1v1Nr1M6942qnxcsL2AAsNI77+uAh7z8et9fqU8vGni/P9he3jle7b3Wl/0+ra37jXk7iYiIiIiIiIgELU0hEREREREREZGgpwCGiIiIiIiIiAQ9BTBEREREREREJOgpgCEiIiIiIiIiQU8BDBEREREREREJegpgiDRSZnbE+9rFzP6nlo/9QIX017V03JfMbI+ZRXrplma2s5aOPcrMPqqNY4mIiEjtKeuziIgogCEiXYBqBTDMLPQERcoFMJxz51SzTsdTAvykFo9XK07inIiIiIiISA0ogCEijwPnmdkqM7vLzELN7EkzW2pma8zsVvCPUPjczP4DrPXy3jez5Wa23syahybtAAAD2UlEQVQmeXmPA9He8V738spGe5h37HVmttbMrg049nwze9vMNpnZ62Zmx6jv34C7zCwsMLPiCAoz+7uZ3eS932lmj5nZN2a2zMyGmNkcM9tmZj8LOExTM3vPzDaY2T/NLMTb/2Jv3xVm9paZxQUc9yEzWwhcU5OLICIiIifPzK40s8VmttLM5ppZay8/0cw+9X5n/8vMdplZy7qur4jUDgUwROQ+YIFzbpBz7ingZuCwc+4s4CzgFjPr6pUdBjzonOvnpX/inBsKJAF3mFmCc+4+4Kh3vOsrfNYPgEHAQGAM8KSZtfW2DQZ+BfQDugHnHqO+u4GFwMRqtjPFOTcCWAC8BIwHhgOPBJQZBvwGOBPoDvzA6/T8DhjjnBsCLAN+HbBPvnNupHNuRjXrIyIiIqduITDcOTcYmAHc4+X/AZjn/c5+D+hUR/UTkdMg7MRFRKSRuRgYYGbjvXQ80BMoBJY453YElL3DzK723nf0yh06zrFHAtOdcyXAfjP7Al+QJNs7diqAma3CN7Vl4TGO8xgwE5hVjXbN9L6uBeKcczlAjpnlm1kzb9sS59x2rw7Tvfrm4wuqfOUNCokAvgk47hvVqIOIiIjUjg7AG94/QiKAsv7JSOBqAOfcbDPLrKP6ichpoACGiFRkwO3OuTnlMs1GAbkV0mOAEc65PDObD0SdxLGPpSDgfQnHuT8555K9IMePArKLKT+qrGJdyo5fWuGzSgM+y1X8KK/OnzrnrjtGdXKPkS8iIiKnz7PAX51zM70+yWQv/3h9DRGp5zSFRERygCYB6TnAz80sHMDMeplZbBX7xQOZXvCiD77pGGWKyvav4EvgWm+djUTgfGDJKdb7UeDugPQuoJ+ZRZpZPDD6FI45zMy6emtfXItvBMgi4Fwz6wFgZjFm1usU6ywiIiK1Ix7Y472/MSB/Id4/OMzsYqD5d1wvETmNFMAQkTVAsZmtNrO7gKnABmCFma0D/kXVoyFmA2Fmtgb4I74/9MtMAdaULeIZ4D3v81YD84B7nHP7TqXSzrn1wIqAdArwpnf814GVp3DYb/AtaroO31DU95xzB4CbgOleWxcBfU6lziIiInJKYswsNeD1a3wjLt4yswXAwYCyDwMXm9kK4FJgL75/1ohIA2DOVRwxLSIiIiIiUv+YWSRQ4pwrNrMRwD+cc4Pqul4iUju0BoaIiIiIiDQUnYA3vemghcAtdVwfEalFGoEhIiIiIiIiIkFPa2CIiIiIiIiISNBTAENEREREREREgp4CGCIiIiIiIiIS9BTAEBEREREREZGgpwCGiIiIiIiIiAS9/w9eLdE4Gw98cgAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "elbo_plot = plt.figure(figsize=(15, 5))\n", "\n", "elbo_ax = elbo_plot.add_subplot(1, 2, 1)\n", "elbo_ax.set_title(\"ELBO Value vs. Iteration Number for Pyro BBVI on Sunspot Data\")\n", "elbo_ax.set_ylabel(\"ELBO\")\n", "elbo_ax.set_xlabel(\"Iteration Number\")\n", "elbo_ax.plot(np.arange(n_iter), losses)\n", "\n", "autocorr_ax = elbo_plot.add_subplot(1, 2, 2)\n", "autocorr_ax.acorr(np.asarray(losses), detrend=lambda x: x - x.mean(), maxlags=750, usevlines=False, marker=',')\n", "autocorr_ax.set_xlim(0, 500)\n", "autocorr_ax.axhline(0, ls=\"--\", c=\"k\", lw=1)\n", "autocorr_ax.set_title(\"Autocorrelation of ELBO vs. Lag for Pyro BBVI on Sunspot Data\")\n", "autocorr_ax.set_xlabel(\"Lag\")\n", "autocorr_ax.set_ylabel(\"Autocorrelation\")\n", "elbo_plot.tight_layout()\n", "plt.show()\n" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "_sn3OS8tOoko" }, "source": [ "## **Criticism**\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "qvRAgapHZLPA" }, "source": [ "### Long-Term Sunspot Model\n", "\n", "Since we computed the approximate posterior of the DPMM that was fit to the long-term sunspot data, we can utilize some intrinsic metrics, such as the log predictive, posterior dispersion indices, and posterior predictive checks. \n", "\n", "Since the posterior predictive distribution for a Dirichlet process mixture model is itself a scale-mixture distribution that has an analytic approximation [(Blei and Jordan, 2004)](http://www.cs.columbia.edu/~blei/papers/BleiJordan2004.pdf), this makes it particularly amenable to the aforementioned metrics:\n", "\n", "$$p(x_{new} | X_{1:N}, \\alpha, G_0) \\approx \\sum_{t=1}^T \\mathbb{E}_q [\\pi_t(\\beta)] \\ \\mathbb{E}_q \\left[p(x_{new} | \\theta_t)\\right].$$\n", "\n", "In particular, to compute the log predictive, we first compute the posterior predictive distribution (defined above) after performing variational inference on our model using a training subsample of our data. The log predictive is then the log value of the predictive density evaluated at each point in the test subsample:\n", "\n", "$$\\log p(x_{new} | X) = \\log \\mathbb{E}_{\\beta, \\theta | X} \\left[ p(x_{new} | \\beta, \\theta) \\right]$$\n", "\n", "Since both the training samples and the testing samples were taken from the same dataset, we would expect the model to assign high probability to the test samples, despite not having seen them during inference. This gives a metric by which to select values of $T$, $\\alpha$, and $G_0$, our hyperparameters: we would want to choose the values that maximize this value.\n", "\n", "We perform this process below with varying values of $\\alpha$ to see what the optimal setting is." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 526 }, "colab_type": "code", "id": "3V2-DC9nobPi", "outputId": "473035e6-036f-4e34-9ea4-6c2dc5a4bec0" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "100%|██████████| 500/500 [00:03<00:00, 157.68it/s]\n", "100%|██████████| 500/500 [00:03<00:00, 165.35it/s]\n", "100%|██████████| 500/500 [00:03<00:00, 156.21it/s]\n", "100%|██████████| 500/500 [00:03<00:00, 165.50it/s]\n", "100%|██████████| 500/500 [00:02<00:00, 172.95it/s]\n", "100%|██████████| 500/500 [00:02<00:00, 169.13it/s]\n", "100%|██████████| 500/500 [00:02<00:00, 169.17it/s]\n", "100%|██████████| 500/500 [00:02<00:00, 169.48it/s]\n", "100%|██████████| 500/500 [00:02<00:00, 173.85it/s]\n", "100%|██████████| 500/500 [00:02<00:00, 171.00it/s]\n", "100%|██████████| 500/500 [00:03<00:00, 161.77it/s]\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmUAAAE/CAYAAAAHcrQrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3xV9f3H8dcnCUmABAiQsBNkI0MZKkvEunDUva3rp3XWUVu1aodVa1uto65WW61aR52IioI4QBQckLD3StgJkEBCyP7+/rjnQoyB7Nz1fj4eeXBzzr3nfO7NDfeTz/d7Pl9zziEiIiIigRUV6ABEREREREmZiIiISFBQUiYiIiISBJSUiYiIiAQBJWUiIiIiQUBJmYiIiEgQUFIm0gjMrKeZOTOLaebztjSzD8xsl5m9VcvHzDCzq5s6tlBR+fUws0vM7JN6HudjM7u8caMLPWZWYGa9AnDeCWa2sbHvK9KclJSJAGY2zczuq2b7GWa2tbmTrTo4F+gEdHDOnVd1p5nda2avNNXJzWy9mR3fVMf3znGvmZV6H/Z5ZjbbzEY3xbmcc686506sZUw/eF2dcyc7515qirjqEkeV/c3+vnbOJTjn1jb2cf3M7ArvD6Dzm+ocIoGipEzE50XgUjOzKtsvBV51zpU1f0i1kgasDOL4GssbzrkEIBn4Cni3mp8VQZw8B8qLNOL7Okhe38uBnd6/ImFFSZmIz3tAe+Bo/wYzSwJOA172vj/VzDLMbLeZbTCzew90sKoVpKoVDTMb5VV88sxsgZlNOMixBnpDbHlmtsTMTve2/xH4PXCBV0W6qsrjJgJ3V9q/oNLuNDP72szyzewTM+tYn9gOxsx+bmarzWynmb1vZl0r7TvRzFZ4w67PmNnM2gypOudKgZeAzkAHr2rytZk9ZmY7gXu94/+fmS0zs1yvWpRW6dwnmNly79xPAVZp3xVm9lWl7weZ2XTvOWwzs7sP9Lr6h0HNLM577QZXOk6yme01sxTv+9PMbH6lyt/Qg7yOf/feb7vNbJ6ZHe1tP9jP169B72vbPyx/lZllAZ+b2RQzu6lKjAvN7EzvtjOzPt7tF83sae8x+Wb2rZn1rvS4Or0PvJ/jMcA1wElm1ukg911vZneZ2VLvffAfM4uvcp9fmVm2mW0xsysrba/177pIY1JSJgI45/YCbwKXVdp8PrDcOef/sNvj7W8HnApc7/8gqgsz6wZMAR7A94H5a+AdM0uu5r4tgA+AT4AU4CbgVTPr75z7A/AgXhXJOfd8lec0tcr+wyrtvhi40jtmrBdDnWKr4Tn+BPgzvtewC5AJ/M/b1xF4G7gL6ACsAMbU8rhxwBXARufcdm/zUcBa77n8yfuZ3A2cja+yNgt4vdK53wF+C3QE1gBjD3CuROBTYCrQFegDfFbD64pzrhh4F7io0ubzgZnOuWwzGw68AFzrPf9ngfe951ad74HD8f08XgPeMrP4muLwYmms9/UxwEDgJHxJ8c8qvU6HAd2Ajw4Q/0XAH4EkYDXwJ+9x9XkfXAbMdc69AywDLqnh/pd4MfcG+uH7uft1Btp6sV8FPO0lrNBIv+sidaWkTGS/l4DzzKyl9/1l3jYAnHMznHOLnHMVzrmF+D7oj6nHeX4GfOSc+8g71nRgLnBKNfcdBSQAf3HOlTjnPgc+5Icf+PXxH+fcykof2ofXI7aDuQR4wTmX7iUpdwGjzaynd6wlzrl3veGzJ4CtNRzvfDPLAzYAI4DKH5CbnXNPOufKvOdzLfBn59wy7/gPAod7VZZTgKXOube9qtvjBzn3acBW59wjzrki51y+c+7bWj7/1/jhz+hibxvAz4FnnXPfOufKvXloxfh+1j/inHvFObfDe36PAHFA/1rGAY3zvr7XObfHe30nA33NrK+371J8iWHJAc7/rnPuO+9n8Sr732v1eR9cxv7X8TVqHsJ8yjm3wTm3E18yWPlnUgrc55wrdc59BBTgva6N+LsuUidKykQ8zrmvgBzgDPNdPXYE+z8AMLOjzOwLM8sxs13AdfiqLXWVhu9DMs//BYzDV1GqqiuwwTlXUWlbJr6/7hui8odfIb7Er66xHUxXL04AnHMFwA58cXfFl1z59zmgpivh3nTOtXPOpTjnfuKcm1dp34Yq900D/l4p/p34higPdO6qj/frga+SVh+fAy2990wavkRkUqX4flXlNe7hxfYj3hDbMm+ILw9fdafW77tGel9Xfs2K8SXyPzOzKHyJzn8PEsKB3mt1eh+Y2VjgELyKq/cchpjZ4Qd6DD/82Wbyw9d4R5U5dftia8TfdZE6CYZJmyLB5GV8f433Bz5xzm2rtO814CngZOdckZk9zoH/o94DtKr0fedKtzcA/3XO/bwW8WwGephZVKXELBVYWYvHArha3q8+sR3MZnzJBwBm1hrfENUmYAvQvdI+q/x9PVR9jhuAPznnXq16R6+606PKuXtUvV+l4xyoInnQ19U5V2Fmb3qP3wZ86JzLrxLfnw52DC++o4E7gePwVZUqzCyX/fPgavvzbej7uup5XsKXiH0FFDrn5tQyjsrq+j64HN/znm8/vG7hMmD+AR5T+Webiu99WRt1+V0XaTSqlIn80MvA8fiGmKq2N0gEdnr/SR+Jb0jqQOYDF5pZCzMbia91hd8rwE/N7CQzizazePP1TaruA+lbfAneHd6xJgA/ZX+1oCbbgJ5eRaM26hKbXwvvfv6vGHwfalea2eHeXKkHgW+dc+vxzVkbYmZneve9kR8mrQ31T+AuMxsEYGZtzczfLmQKMMjMzvbOffNBzv0h0NnMbjXf5P1EMzvK21eb1/U14AJ8Q7mvVdr+L+A6rxpjZtbam1ieWM0xEoEyfJWuGDP7PdCm0v7a/nwb630NgJeEVQCPcPAq2cHU+n1gvgn65+Ob4H94pa+bgEvswFeF3mhm3c2sPb55hm/UMrY6vyYijUFJmUglXtIwG2gNvF9l9w3AfWaWj++qxzcPcqjf4ZtcnItvkvO+D2Xn3AbgDHwfEjn4Kie3U83vozdP53TgZGA78AxwmXNueS2fkr+h7A4zS6/pznWJrZKPgL2Vvu51zn2G7zV4B19FpDdwoXeO7cB5wEP4hjQPxTdvrbiWz6mm5zAJ+CvwPzPbDSzG9/pVPvdfvHP3Bb4+wHHygRPwJcFbgVXAsd7uGl9Xb/7ZHnxDZh9X2j4XX3L0FL73x2p8Fy9UZ5r32JX4ht+K+OGQXK1+vo34vq7sZWAIvkS+zur4PjgT33vrZefcVv8X8DwQDUw8wGlew3eRzFrv64Fahlff10SkQcw3jC8iEhhelWcjcIlz7otAxyO1Y2aXAdc458Y10vEa9X1gZuuBq51znzb0WCLNRZUyEWl23vBoO29o8258c4W+CXBYUktm1gpfNem5Bh5H7wORSpSUiUggjMZ3ZeN2fMODZ3rtFiTImdlJ+Ia2t/HDuXL1ofeBSCUavhQREREJAg2qlJnZeeZb9qXCu8LMv72n+ZYUme99/bPSvlgze87MVppvqZNzvO1xZvaG+ZZl+dZ8TSZFREREIkJD+5QtxreUybPV7FvjnKuuqd89QLZzrp83sbO9t/0qINc518fMLsR39dQFDYxPREREJCQ0KClzzi0DqNLIryb/BwzwHl+Bby4B+C7Dv9e7/TbwlJmZq2F8tWPHjq5nz551Ob+IiIhIQMybN2+7c67a9YSbsqP/IWaWAewGfuucm2Vm7bx993tNMNcAv/C6S3fD67/jnCvzlrbowP6krVo9e/Zk7ty5TfUcRERERBqNmWUeaF+Nc8rM7FMzW1zN1xkHedgWINU5Nwy4DXjNzNrgSwK7A18754YDc4C/+U9VzXGqrZKZ2TVmNtfM5ubk5NT0FERERESCXo2VMufc8XU9qLdgbbF3e56ZrQH6AfPwLfrqX5j3LXxzycDXNLAHsNFbMqMtvoWEqzv+c3j9cUaOHKnLR0VERCTkNUmfMjNLNrNo73YvfEuZrPXmh30ATPDuehyw1Lv9Pr4FZ8G3TuDnNc0nExEREQkXDZpTZmZnAU8CycAUM5vvnDsJGI9v3bAyoBy4zjnnr3rdCfzXzB7H14DwSm/789721fgqZBc2JDYRERGRUBLyzWNHjhzpNNFfREREQoGZzXPOjaxun5ZZEhEREQkCSspEREREgoCSMhEREZEgoKRMREREJAgoKRMREQkR5RWO6Uu3sWtvaaBDkSbQlMssiYiISCPJyS/mlv9lMHvNDjq3iefP5wzh2P4pgQ5LGpEqZSIiIkHu27U7OPWJWaRn5XL7Sf1p0zKGK//zPbe/tUBVszCipExERCRIVVQ4npmxmov+9Q0J8TG8d+NYbjy2Dx/cNI4bj+3NuxmbOOmxL/liRXagQ5VGoKRMREQkCOXuKeGql77noakrOHVoV97/xTgGdG4DQFxMNLefNIBJN4zZVzW7421VzUKdkjIREZEgk5GVy2lPfsXXq3dw/5mDeeLCw0mI+/E08KHd2+2rmr2TrqpZqFNSJiIiEiScc7zw1TrOf3YOZvD29aO5dFQaZnbAx1SumiXG76+a7S5S1SzUKCkTEREJAruLSrnh1XTu+3ApE/qnMOWmoxnavV2tHz+0ezs+vHkcN0zozdvzNnLSY18yQ1WzkKKkTEREJMCWbN7F6U9+xSdLt3HPKQN57tIRtG3Vos7HiYuJ5o6JA5h0w1gS4mK4QlWzkKKkTEREJECcc7z+XRZnPTObotIK3rhmFD8f3+ugw5W1cVgP31yz6ytVzWauzGmkqKWpKCkTEREJgD3FZdz25gLuencRRx3Snik3j2Nkz/aNdvz4FtHcOXEA794wltZxMVz+wnfc+fZCVc2CmJIyERGRZrZqWz5nPP01k+dv4lcn9OOlK4+kQ0Jck5zr8B7t+NCrmr01b4OqZkFMSZmIiEgzmpSxkdOf+pq8whJeueoobjquL1FRDRuurEl1VbPfvLOQfFXNgoqSMhERkWZQVFrOXe8u4pdvLGBI97Z8dPPRjOnTsVlj8FfNrjumN2/O9VXNvlTVLGgoKRMREWli67fv4exnZvP6d1ncMKE3r119FClt4gMSS3yLaH5z8gDeuX4MreJiuExVs6ChpExERKQJfbxoCz998is25e3lhStGcsfEAcREB/7jd1hqkqpmQSbw7woREZEwVFJWwR8/WML1r6bTOyWBKTeP4ycDOgU6rB+oXDVrGRvNZS98x13vqmoWKErKREREGtmmvL2c/+wc/vP1eq4c25M3rx1N96RWgQ7rgIalJjHl5qO59phevPH9BiY+PotZq1Q1a25KykQk4jnnAh2ChJHPl2/j1CdmsSa7gH9cMpw//HQQsTHB/3Eb3yKau04eyNvXjyG+RRSXPv8dd727SFWzZhT87xIRkSa0fvseht0/nZ+/PJfV2QWBDkdCWFl5BQ9NXc7/vTiXrm1b8sFN4zh5SJdAh1Vnw/1Vs/G9eOP7LCY+PouvVm0PdFgRwUL9L8SRI0e6uXPnBjoMEQlBzjkue+E75mXmEmXG3tJyLjiiB7ce1zdgV8ZJaMreXcRNr2fw7bqdXHRkKn/46aHEt4gOdFgNlp6Vy6/fWsDanD1cdGQqd58ygMT4uq/JKfuZ2Tzn3Mjq9qlSJiIR64OFW5i1ajt3ThzAzNsncOmoNN78fgPHPDyDR6evpKC4LNAhSgiYvXo7pzwxi4Ubd/HYBYfx57OHhEVCBr6q2UeqmjUbVcpEJCLt2lvK8Y/OpHObeN67cSzRXkf19dv38PC0FUxZtIWOCbHccnw/LjyiBy2CoIWBBJeKCsdTX6zm8U9X0is5gX9cMpy+nRIDHVaTmZeZy+1v+6pmFx+Vyt2nDCQhLibQYYWcg1XKlJSJSET6/eTFvPJNJpNvHMeQ7m1/tD8jK5c/f7Sc79bvpFfH1twxsT8nDeqMWdMuhyOhYUdBMb98cwFfrszhzMO78qezhtA6AhKUotJyHp2+kn/NWkvXti356zlDGde3eVclCHVKykREKlmwIY8zn/may0f35N7TBx3wfs45PluWzV+mLmd1dgHDU9tx9ykDGdmzfTNGK8Fm7vqd/OK1DHYWlnDvTwdx0ZE9Ii5Zn5eZy+1vLWDt9j1cclQqd6lqVmtKykREPGXlFZzx9Nfk5Bfz2a+OqdWk5bLyCt6at5HHpq8kO7+YEw/txB0TB9AnJaEZIpZg4Zzj37PW8Zepy+me1JKnLx7O4G4/rrJGiqLSch75ZAX//modXdu25KFzhzK2mdfyDEVKykREPC98tY77PlzK0xcP59ShdWtXUFhSxvOz1vHPmWsoKqvgwiN6cMvxfUlJ1JWa4W5XYSm/fnsB05du4+TBnfnruUNpo6sQAZiXuZPb31qoqlktKSkTEQG27iriuEdmMLJne1688oh6DzltLyjmic9W8dq3WcTGRPHzo3txzfheETGnKBIt2riLG16bx5a8Iu4+ZSBXju0ZccOVNalaNXv43KGMUdWsWkrKRESAG16dx2fLspn+y2NI7dDwJW/Wbd/Dw9OW89GirXRMiOPW4/tyga7UDBvOOV75JpP7P1xGx4RYnrpkOMNTkwIdVlCrXDX72ahU7jp5oP5YqUJJmYhEvC+WZ3Pli99z+0n9ufHYPo167PSsXP780TK+X5/rXak5gJMGdVI1JYQVFJdx17uL+GDBZo7tn8yj5x9OUuvYQIcVEopKy/nbtBU8//U6urVryUPnqGpWmZIyEYloe0vKOeGxmcS3iOajm49uknUInXN8uiybv3y8jDU5exiRlsTdpwxgRJqu1Aw1y7fu5oZX0lm/Yw+/OrE/1x/Tm6goJdh1NXf9Tm5/eyHrtu/h0lFp/ObkAaqa0YQd/c3sPDNbYmYVZjay0vaeZrbXzOZ7X/+stO8iM1tkZgvNbKqZdfS2tzez6Wa2yvtXNWIRaRRPfr6Kjbl7eeDMwU22MLSZccKhnZh263gePGsIWTsLOecfc7j2v3NZk6M1NUPFW3M3cObTX5NfXMZrPx/Fjcf2UUJWTyN7tuejm4/m6nGH8Mq3mZz0+JfMXqPVAA6mQZUyMxsIVADPAr92zs31tvcEPnTODa5y/xhgM3Coc267mT0EFDrn7vVu73TO/cXMfgMkOefurCkGVcpE5GBWbsvnlL/P4ozDu/HI+Yc123kLS8r496x1POtdqXnRkT245bh+JCfGNVsMUnt7S8r5/eTFvDVvI2N6d+DvFw7Tz6oRqWq2X5NVypxzy5xzK+oSi/fV2nyTLdrgS9IAzgBe8m6/BJzZkNhERJxz/HbSYhLiY7j7lAHNeu5WsTHcfFxfZtx+LBcfmcr/vtvAMQ9/weOfrmSP1tQMKmtyCjjrma95O30jN/+kD/+96iglZI3MXzW7yquaTfz7l8xZsyPQYQWdprxE6BAzyzCzmWZ2NIBzrhS4HliEVzEDnvfu38k5t8W73xYgpQljE5EI8Pa8jXy3fid3nTyADgmB+ZBNTozj/jMH88kvx3NMv2Qe/3QVxzw8g1e/zaSsvCIgMcl+HyzYzOlPfkV2fjEvXnkkt53Yf986qNK4WsZG87vTDuXNa0cTbcZF//qG309erD9SKqlx+NLMPgU6V7PrHufcZO8+M/jh8GUckOCc22FmI4D3gEHAXmAqcA2wFngS2Oqce8DM8pxz7SqdN9c5V+28MjO7xjsGqampIzIzM+vwlEUkEuzcU8Jxj8ygd3ICb147OmjmBc3L9F2pOTczl17Jrblz4gBOPFRXaja34rJy/jRlGS/PyWREWhJPXTyMLm1bBjqsiLG3pJyHp63gP7PX0T2pJQ+dcxije3cIdFjNosmvvqyalB1oP76hy784547zto8HfuOcO8XMVgATnHNbzKwLMMM517+mc2tOmYSKotJy4ltEBzqMiHHH2wt4N30TH948jgGd2wQ6nB9wzjF96Tb+MnU5a3P2MDItibtOGciINF3f1Bw27CzkxtfSWbhxFz8/+hDumDhAveUC5Pv1O7n9rQWs31HIZaPTuHNi+M81a7I5ZQc5YbKZRXu3ewF98VXGNgGHmlmyd9cTgGXe7feBy73blwOTmyI2keaWV1jCbW/MZ/AfprFia36gw4kI363byZtzN3LV0YcEXUIGvis1TxzUmU9uHc+fzhrM+h2FnPOP2Vz/yjzW6krNJjV96TZOfWIW67bv4blLR3DPqYcqIQugI3q25+NbxvN/Yw/hv99orllDr748C98QZDKQB8x3zp1kZucA9wFlQDnwB+fcB95jrgNuAUqBTOAKb5izA/AmkApkAec553bWFIMqZRLMpi/dxt2TFrFzTwnlFY7fnjqQq4/uFeiwwlpJWQWnPTmLPcXlTL9tPK1ig/+v7j3F3pWaX66hpKyCi45M5ebj+mqyeSMqLa/g4WkreO7LtQzp1panLx7eKKs6SOP5bt1O7njbVzW7fHQad548ICR+f+tKzWNFmlnunhL++MES3pu/mYFd2vDwuUO57pV5DO3elmcuGRHo8MLaP2as4a9Tl/P85SM5bmCnQIdTJzn5xfz9s5W8/t0G4mOiuPaY3lx99CFh+cHUnLbs2stNr2UwNzOXS0elcc+pAzWVIEjtLSnnoWnLeXH2enokteKhc4cyqld4zTVTUibSjKYt2co9kxaTV1jCL37Shxsm9CE2Jopb/pfBN2t38M1dx2lSdxPZsLOQEx6byTH9knn20mr/zwsJa3IKeHjqCqYu2UpyYhy/PL4f54/sToyG2ersy5U53PrGfIpLy/nzOUM5/bCugQ5JauG7dTu5/e0FZIZh1azZ55SJRKLcPSXc/HoG1/53HimJcbz/i3Hceny/fR3kR6QlsW13MZt3FQU40vDknOP3kxcTZcYffjoo0OE0SO/kBP556QjeuX40qe1bcfekRZz0+JdMX7qNUP9DurmUVzgenb6Sy//zHckJcbx/0zglZCHkyEPa8/EtR3PFmJ68NCeTiY/P4pu14T/XTEmZSCOYungrJzw2k48Xb+G2E/ox+RdjObTrDyeYD0/1XVk3LzM3ECGGvWlLtvLFihxuO6EfXduFR2uDEWntefu60Tx76Qicg5+/PJcLnv2GjCy9hw4mJ7+Yy174lic+W8U5w7vz3o1j6Z2cEOiwpI5axcZw7+mDeOOaUQBc+Nw33Pv+EgpLwrevmYYvRRpg554S/vD+Ej5YsJlBXdvwt/MOY2CX6q/2KyuvYOgfP+H8kT249/TQruQEm4LiMo5/ZCZJrWP54Bdjw3KYr7S8gje+38Djn65ie0ExpwzpzO0nDeCQjq0DHVpQ+XbtDm56PYNde0u5/8zBnD+yR6BDkkZQWFLGQ1NX8OLs9aR1aMVD5wzlqBCda6bhS5Em8PGiLZzw6EymLt7Cr07ox3s3jj1gQgYQEx3FYd3bqVLWBB79ZCXb8ot48KzBYZmQAbSIjuJno9KYefsEbj2+LzNW5HDCozP5w+TFbC8oDnR4AVdR4Xhmxmou+tc3JMTF8N6NY5WQhZHKVTPn4IIwrZqF5/9eIk1oR0ExN76WzvWvptOlXTwf3DSOm47rW6teRyPSkli6ZXfY/UcSSIs37eLF2eu4+MhUhqWGf/PV1nEx3Hp8P2bcPoELjujBK99mccxDX/DkZ6si9n2Vu6eEq1+ey0NTV3DykC5M/sXB/0CS0HVUrw5MvdU31+zF2es5+e+z+G5djd2zQoaSMpE6+GjRFk587Es+WbKV20/qz6QbxtapOemItCTKKxwLN+5qwigjR3mF455Ji2jfOpY7TmreBccDLSUxnj+dNYRPfjmecX078sj0lUx4eAavf5cVUWtqZmTlctqTXzFrVQ73nTGIpy4aRmJ8i0CHJU3IXzX7376q2ZywqZopKROphe0Fxdz4ajo3vJpOt6SWfHjT0dx4bJ86dwIflupb3lVDmI3jte+yWLBxF7877VDatorMD+LeyQk8e+lI3r5uNN2TWnLXu4uY+PdZfBrmV2o65/jP1+s4/9k5mMHb143hstE91W4mgozyqmaXjw6fqpkm+oschHOOKYu28PvJSygoKuOW4/ty7fheDZq3dPyjM0lr34rnrziiESONPNn5RRz3yEyGdm/LK1cdpQ9jfO/XaUu28tDUFazdvocjD2nPXScPCLth3d1FpfzmnYV8tGgrxw/sxCPnHRaxSbn4fLN2B3e8vZANuYVcMaYnd5w0gJaxwdkg+GAT/cOjE5tIE8jJL+b3kxfz8eKtHNa9LQ+fdxj9OiU2+LjDU9vxiVfFUCJRfw98uIzi0gruP2OwXkePmTFxcBeOG9iJ/32/gb9/upKznpnNqUO6cPtJ/ekZBldqLtm8ixtfTWdD7l7uPmUAPz+6l37+sq9q9tePl/Ofr9fzxfJsHj7vMI7o2T7QodWJhi9FqnDO8f6CzZz42Ew+W5bNnRMH8M71YxolIQPfvLK8wlLWbt/TKMeLRLNW5fD+gs1cP6E3vdR/6kdaREdx6ag0Ztx+LLcc15fPl2dz/KMzuff9JewI0Ss1nXO8/l0WZz0zm6LSCt64ZhTXjO+thEz2aRUbwx/PGMzrPx9FuXOc/+wc7vtgKXtLygMdWq1p+FKkkpz8Yn733mKmLtnKYT3a8bdzh9K3kZIxv9XZ+Rz/6Jc8fO5QztMl+3VWVFrOxMe/xMz4+JajtYZhLWTvLuLxz1bxxvcbaNkimuuO6cVV43oF7fBOVYUlZfx20mLezdjE0X078vgFh9MhQYu1y4HtKS7jr1OX8/KcTHp2aBVUVTP1KROpgXOOyfM3ccJjM/l8RTZ3nTyAd64b3egJGUCvjgm0bdmCdHVlr5d/zFjD+h2F3H/GYCVktZTSJp4HzxrCtFvHM6Z3B/72yUom/O0L3vg++K/UXLUtnzOe+ppJ8zfxy+P78eKVRyohkxq1jovhvjMG89rPjwqpqpmSMol42flFXPvfedzyv/n07NCaj24ex7XH9G6yJqRRUcbwVDWRrY+1OQX8Y8Yazji8K+P6dgx0OCGnT0oCz102kreuG03Xdi25851FnPLELD5fHpxXar6XsYnTn/qa3MISXrnqKG45vi/RURqulNob07sjU28Zz6Wj0njh63Wc8sQs5q4P3is0lZRJRNtbUs5pT3zFjJU53GyzyA0AACAASURBVH2Kb+5Yn5TGr45VNTw1iZXbCti1t7TJzxUunHP89r3FxLWI4p5TBwY6nJB2RM/2vHv9GP5xyXBKyx3/9+JcLnzuGxZsyAt0aIBviPqudxdx6xvzGdK9LVNuPpqxfZSES/1UrpqVlldw3rNzuP/D4KyaKSmTiDZ/Qx7Z+cU8ceHhXDO+d7P9FT4izdeiQAtL197k+ZuZvWYHd0wcQEpifKDDCXlmxslDuvDJL8dz/xmDWJ1dwBlPf80vXksnc0fgLkLJ3LGHs5+ZzevfZXH9hN68dvVRdGqjn7c03JjeHZl263h+dlQaz38VnFUzJWUS0fzzukY188K2h/VoR5RBuoYwa2VXYSkPTFnKYT3acfGRqYEOJ6y0iI7i0tE9mXnHsdz8kz58tmz/lZo795Q0ayxTF2/htCe+YlPeXl64YiR3ThwQtmuZSmC0jovh/jMH89rV+6tmDwRR1UzvdoloGVm59EpuTbtWsc163tZxMQzs0ob0rOAYLgp2D01bzs49JTx41mDNKWoiCXEx3HZif2bePoFzR/Tg5TnrOeahL3j6i9VN/oFVUlbBfR8s5bpX0umVksCUm8fxkwGdmvScEtnG9PFVzS45KpV/e1WzeZmBr5opKZOI5ZwjIyuP4QHqdj4iLYmMrFzKK4JvgnUwSc/K5bXvsrhy7CEM6to20OGEvZQ28fz5bN+amqN6d+DhaSs49m8zePP7DU3yXt2Ut5fzn53DC1+v44oxPXnr2tF0T2rV6OcRqap1XAwPnDmE164+ipKyCs795xwmz98U0JiUlEnEytpZyI49JfvWo2xuI9KS2FNSzoqt+QE5fygoK6/g7ncX0Skxnl+e0C/Q4USUPimJ/Ouykbx57Wg6t43njncWcsrfZ/HF8uxGu1Lzi+XZnPrELFZnF/DMJcO59/RBxMboY0ma15g+HZn2y/Fcc3QvxvdNDmgsevdLxPLPJwtUpcx/3nma7H9AL85ez/Kt+dx7+qEkxGlVuEA48pD2TLphDM9cMpzisnKufPF7LvrXNyzcWP+h97LyCh6aupwrX/yeLm1b8sFN4zhlSJdGjFqkbhLiYrjrlIEktW7eqSxVKSmTiJWemUdCXEyjLZ9UV92TWpKcGKfJ/gewOW8vj05fyXEDUjhpUOdAhxPRzIxThnRh+m3HcN8Zg1i1rYDTn/qam17PIGtHYZ2Olb27iEv+/S3PzFjDhUf0YNINYzgkDNbkFGkM+tNTIlZ6Vi6H9WgbsInjZsaI1CR19j+AP36whArnuPf0QVrfMEi0iI7istE9OWtYN577ci3/nrWOqYu38LNRadz0k760r6HKMHvNdm5+fT57ist45LzDOGdE92aKXCQ0qFImEamwpIzlW/MDNnTpNyIticwdheTkh+Yi0U3l06XbmLZkG7cc148e7TXpO9gkxrfgVyf2Z8btEzh3RHdemu27UvOZGaspKv3xlZoVFY4nP1vFz/79Le1atWDyL8YqIROphpIyiUgLN+6ivMIFbJK/33CviayqZfsVlpTxh/eX0K9TAlcffUigw5GD6NQmnj+fPZRpt47nqF7teWiqd6Xm3P1Xau7cU8IVL37PI9NXcvphXZl849iATRkQCXYavpSI5E+ChvUIbKVscLc2xEZHkZ6Zq3lTnr9/topNeXt567rRtFDj0JDQt1Mi/778CL5du4MHP17OHW8v5PlZ6/jZqFSe/mINOwtLePCsIVx0ZA8NRYschP7Hk4iUnplHr46tA36lTVxMNIO7tdHi5J7lW3fz/Kx1XDCyB0f0bB/ocKSOjurVgfduGMPTFw+nqKyc301eQmxMFO9eP4aLj0pVQiZSA1XKJOL4msbmMqF/SqBDAXzzyl6ak0lJWUVE92iqqHD8dtJiEuNj+M3JAwIdjtSTmXHq0C6ccGgnZqzIZlTvDrSJbxHosERCQuR+AkjE8jeNHZ4W2PlkfiPSkigpq2DJ5l2BDiWg3py7gbmZudwdBL2CpOFiY6I4cVBnJWQidaCkTCJOhrfeZKCvvPTb10Q2gocwdxQU8+ePl3PkIe05V1fliUiEUlImESc9K5fWsdFBcwVYSpt4erRvGdFXYD740XIKS8p48KzBmnckIhFLSZlEHF/T2HYBaxpbneGpSczLzG20NQVDyZw1O3gnfSPXjO9Fn5TgSJRFRAJBSZlElMKSMpZtCXzT2KpGpCWxbXcxm/L2BjqUZlVSVsFv31tEj/Yt+cWxfQMdjohIQCkpk4jibxobLJP8/fxJYnpW/Rd5DkXPfbmGNTl7uO/0wbSMjQ50OCIiAaWkTCJKsDSNrWpA50RaxUZH1OLkmTv28OTnqzllSGeOHRAc7UlERAKpQUmZmZ1nZkvMrMLMRlbZN9TM5nj7F5lZvLd9hPf9ajN7wrxZvWbW3symm9kq79/g+tSUsJCRFRxNY6uKiY7i8B7tIuYKTOccv5+8hBbRUfz+tEGBDkdEJCg0tFK2GDgb+LLyRjOLAV4BrnPODQImAKXe7n8A1wB9va+J3vbfAJ855/oCn3nfizQaf9PYwwO83uWBDE9NYumW3RSWlAU6lCb30aKtzFyZw69O7EfntvGBDkdEJCg0KClzzi1zzq2oZteJwELn3ALvfjucc+Vm1gVo45yb43yXmb0MnOk95gzgJe/2S5W2izSKDTv3sr2gJOgm+fuNSEuivMKxYEN4N5HdXVTKHz9YwuBubbh0VFqgwxERCRpNNaesH+DMbJqZpZvZHd72bsDGSvfb6G0D6OSc2wLg/atJJtKo/PPJgjUpG+ZV8MK9X9mjn6wkp6CYP505hBgtOC4isk+Na1+a2adA52p23eOcm3yQ444DjgAKgc/MbB6wu5r71rkxk5ldg28IlNTU1Lo+XCKUv2ls/87B2QurXatY+qQkhPVk/4Ub83h5znouHZXGYT2CcxhZRCRQakzKnHPH1+O4G4GZzrntAGb2ETAc3zyzymuodAc2e7e3mVkX59wWb5gz+yAxPQc8BzBy5MjI67Yp9RKMTWOrGpGaxLSlW3HOhV1n+/IKxz2TFtMhIY5fn9Q/0OGIiASdpho7mAYMNbNW3qT/Y4Cl3rBkvpmN8q66vAzwV9veBy73bl9eabtIg+0tKWfZlvx9Q4TBakRaEnmFpazdvifQoTS6V77JZNGmXfz+tEO1SLWISDUa2hLjLDPbCIwGppjZNADnXC7wKPA9MB9Id85N8R52PfBvYDWwBvjY2/4X4AQzWwWc4H0v0igWbszzNY0N0vlkfv6mtuHWGmPb7iIenraCo/t25LShXQIdjohIUKpx+PJgnHOTgEkH2PcKvuHKqtvnAoOr2b4DOK4h8YgciL9T/rAgT8p6dUygbcsWpGfmcv7IHoEOp9Hc9+FSSsoruP8MLTguInIguvRJIkJ6Vi6HdGxN+yBrGltVVJQxPLVdWF2BOWNFNlMWbuGmY/vQs2PrQIcjIhK0lJRJ2PM3jQ32+WR+I9KSWLmtgF17S2u+c5ArKi3n95OX0Cu5Ndcc0yvQ4YiIBDUlZRL2NuYGd9PYqoan+eLMCINq2dNfrCZrZyEPnDmYuBgtOC4icjBKyiTs7VuEPEQqZYd1b0eUEfL9ylZn5/PPmWs4e1g3xvTuGOhwRESCnpIyCXvpmbm0io2mf6fgbBpbVeu4GAZ2acO8EK6U7Sku4+53F9MqNoa7Tx0Y6HBEREJCg66+FAkF6Vl5HNa9XUgt6TMiLYl35m2kvMIFdbPbqioqHO9mbOKhqcvJzi/m4XOH0jEhLtBhiYiEhND5lBKpB1/T2N37+n+FihFpSewpKWfF1vxAh1Jr36/fyZnPfM2v31pAl3Yteef6MZwXRm09RESamiplEtYWbsyjLASaxlblj3deVi6Hdm0T4GgObsPOQv4ydTlTFm6hc5t4Hr/gcE4/rCtRIVThExEJBkrKJKxlbAiNprFVdU9qSUpiHOmZuVw6Ki3Q4VSroLiMf8xYzb9mrSPK4Nbj+3LN+F60itV/KyIi9aH/PSWspWfm0rNDq6BvGluVmTE8NSkol1uqqHC8nb6Rh6etICe/mLOGdeOOif3p0rZloEMTEQlpSsokbDnnSM/KY3zf0GzHMCItialLtpKTX0xyYnBMlv9m7Q7u/3ApSzbvZlhqO567dETIVSFFRIKVkjIJW76mscUMSwvNpMHfRDY9K5eTBnUOaCxZOwr588fL+HjxVrq2jeeJi4bx06FdtI6liEgjUlImYcvfNHZ4iDSNrWpwtzbERkeRnhm4pCy/qJSnv1jDC1+tIzrK+NUJ/bj66F60jFV3fhGRxqakTMJWRlZeSDWNrSouJpoh3dsGZF5ZeYXjrbkb+NsnK9heUMI5w7tzx8T+dGoT3+yxiIhECiVlErbSs3IZ2r1tSDWNrWp4ajtempNJcVl5s60dOXvNdu7/cBnLtuxmZFoSL1xxBEO7h2a1UUQklITup5XIQRSVlrN08+6Q609W1Yi0JErKKliyeXeTn2v99j1c8/JcLv7Xt+zeW8pTFw/jretGKyETEWkmqpRJWFq4cVdINo2tyh9/emZukz2X3UWlPPX5av7z9Tpio6O4/aT+XDXuEOJbaN6YiEhzUlImYck/yX9YiE7y90tpE0+P9i33PZ/GVFZewRtzN/DoJyvZWVjCeSO68+sT+5OieWMiIgGhpEzCkr9pbIcwWAx7RGoSc9buwDnXaC0ovlq1nQemLGX51nyOPKQ9L512KIO7tW2UY4uISP0oKZOw45wjY0MeR/cJzaaxVY1IS+K9+ZvZlLeX7kmtGnSstTkFPPjRMj5dlk2P9i35xyXDmTi4s/qNiYgEASVlEnY25u4lJ7845Icu/fwd8+dl5tY7KdtVWMoTn6/ipdnriW8RzZ0TB3Dl2J6aNyYiEkSUlEnY2T+fLLQn+fsN6JxIq9ho0jNzOePwbnV6bFl5Ba9/l8Wj01eSt7eUC4/owW0n9A+aZZtERGQ/JWUSdvxNYwd0Ds2msVXFREdxeI92pGfl1elxM1fm8MCHS1mVXcCoXu353WmHMqir5o2JiAQrJWUSdsKhaWxVI9KSeGbGGgpLymgVe/Bf29XZBfxpylK+WJFDWodWPHvpCE48tJPmjYmIBDklZRJW/E1jrxnfK9ChNKrhaUmUVzgWbNjF6N4dqr1PXmEJj3+6ile+yaRli2juPmUAl4/p2WwrAYiISMMoKZOwsmhTeDSNrWp4D6+JbFbuj5Ky0vIKXv0mk8c+XUV+USkXHpnKbSf0o2MYtAMREYkkSsokrKR7i3cfHiZXXvq1bdWCPikJP1qc/IsV2Tzw4VLW5OxhbJ8O/O60QxnQuU2AohQRkYZQUiZhJT0rl7QOrcKySjQiNYlpS7finGN1dgEPTFnGzJU5HNKxNf++bCTHDUzRvDERkRCmpEzChnOO9Kw8xoVJ09iqRqQl8cbcDdz8v/l8tGgLrWKj+e2pA7lsdE9iY8LnogYRkUilpEzChr9p7PAwG7r0G57mm1c2ZeFmLjkqjV+e0I/2rWMDHJWIiDQWJWUSNjI2+Pp4hUvT2Kr6pCTw+AWHM7BLG/qHSQ82ERHZT0mZhI30zFxatgifprHVOXNY3Tr6i4hI6NBEFAkbGWHYNFZERCKHPr0kLBSVlrNk8+59865ERERCjZIyCQvh2jRWREQiR4OSMjM7z8yWmFmFmY2ssm+omc3x9i8ys3gza2VmU8xsubf9L5XuH2dmb5jZajP71sx6NiQ2iSz+prHDwvTKSxERCX8NrZQtBs4Gvqy80cxigFeA65xzg4AJQKm3+2/OuQHAMGCsmZ3sbb8KyHXO9QEeA/7awNgkgmRk5YVt01gREYkMDUrKnHPLnHMrqtl1IrDQObfAu98O51y5c67QOfeFt60ESAe6e485A3jJu/02cJypPbnUgq9pbC7DeqhKJiIioaup5pT1A5yZTTOzdDO7o+odzKwd8FPgM29TN2ADgHOuDNgFdKj6OJGqNuXtJTu/WJP8RUQkpNXYp8zMPgU6V7PrHufc5IMcdxxwBFAIfGZm85xzn3nHjAFeB55wzq31n6qa47gDxHQNcA1AampqTU9Bwlx6lq9prCb5i4hIKKsxKXPOHV+P424EZjrntgOY2UfAcPZXxZ4DVjnnHq/ymB7ARi9pawvsPEBMz3nHYOTIkdUmbhI5IqFprIiIhL+mGr6cBgz1rraMAY4BlgKY2QP4Eq5bqzzmfeBy7/a5wOfOOSVcUiM1jRURkXDQ0JYYZ5nZRmA0MMXMpgE453KBR4HvgflAunNuipl1B+4BDgXSzWy+mV3tHe55oIOZrQZuA37TkNgkMvibxobrepciIhI5GrT2pXNuEjDpAPtewdcWo/K2jVQ/dwznXBFwXkPikcizeF/TWF15KSIioU3jPRLS0rN8TWN15aWIiIQ6JWUS0tIz80htr6axIiIS+pSUScjyN43V0KWIiIQDJWUSsjbvKlLTWBERCRtKyiRk7VuEvIeSMhERCX1KyiRkpWflEt8iigFd1DRWRERCn5IyCVnpWXkM7d6OFmoaKyIiYUCfZhKSikrLWbp5l9a7FBGRsKGkTELS4k27KC1X01gREQkfSsokJGVk5QFoeSUREQkbSsokJKVn5dKjfUuSE9U0VkREwoOSMgk5+5vGqkomIiLhQ0mZhJzNu4rYtrtYSZmIiIQVJWUScvxNY5WUiYhIOFFSJiEnIytPTWNFRCTsKCmTkJOelcvQbmoaKyIi4UWfahJSikrLWbJ5F8PS1J9MRETCi5KyWnDOBToE8SzZ7G8aq/lkIiISXpSU1eCl2evp99uP2VtSHuhQBEjP9DWNVVImIiLhRklZDVrFRlNa7sjJLw50KIKaxoqISPhSUlYD/4d/TkFRgCMR8F15qSqZiIiEIyVlNUhJjAdQpSwIbM7by9bdRQzroUn+IiISfpSU1cBfKctWUhZw6Vle09g0VcpERCT8KCmrQfvWsUSZKmXBID3T1zR2YJc2gQ5FRESk0Skpq0F0lNExIU5JWRBQ01gREQln+nSrheTEOA1fBpiaxoqISLhTUlYLyYmqlAXaks271TRWRETCmpKyWkhRUhZwGd4k/2GpqpSJiEh4UlJWC8mJcWwvKKaiQsstBUp6Vi7dk1rua1EiIiISbpSU1UJyQhxlFY7cwpJAhxKx0jPVNFZERMKbkrJaSGnjNZAt0BBmIPibxg7X0KWIiIQxJWW1sK+B7G4lZYGQkeUtQq6msSIiEsaUlNVCcoK3/qUm+wdEelYucTFRDOisprEiIhK+lJTVwv5FyZWUBUJ6Vi5Du7clNkZvVxERCV/6lKuF1nExtI6N1vBlABSXlbNk025N8hcRkbDXoKTMzM4zsyVmVmFmI6vsG2pmc7z9i8wsvsr+981scaXv25vZdDNb5f0bVJ/CyYlxqpQFwOJNuykpr2CYkjIREQlzDa2ULQbOBr6svNHMYoBXgOucc4OACUBppf1nAwVVjvUb4DPnXF/gM+/7oJGSGE9OflGgw4g4/qaxw7W8koiIhLkGJWXOuWXOuRXV7DoRWOicW+Ddb4dzrhzAzBKA24AHqjzmDOAl7/ZLwJkNia2xaf3LwMjIylPTWBERiQhNNaesH+DMbJqZpZvZHZX23Q88AhRWeUwn59wWAO/flCaKrV60/mVgpGflauhSREQiQkxNdzCzT4HO1ey6xzk3+SDHHQccgS/5+szM5gE7gD7OuV+aWc96ReyL6RrgGoDU1NT6HqZOkhPjyC8qo6i0nPgW0c1yzki3ZddetuxS01gREYkMNSZlzrnj63HcjcBM59x2ADP7CBiObx7ZCDNb7507xcxmOOcmANvMrItzbouZdQGyDxLTc8BzACNHjmyWBSn3tcXIL6ZH+1bNccqIl57pNY1VpUxERCJAUw1fTgOGmlkrb9L/McBS59w/nHNdnXM98VXSVnoJGcD7wOXe7cuBA1XhAmJfV38NYTYbf9PYgV3UNFZERMJfQ1tinGVmG4HRwBQzmwbgnMsFHgW+B+YD6c65KTUc7i/ACWa2CjjB+z5opCSqq39zU9NYERGJJDUOXx6Mc24SMOkA+17B1xbjQI9dDwyu9P0O4LiGxNOU9g9fqi1Gc/A3jb1ybM9AhyIiItIsVIKopQ6t44gyVcqay5LN/qaxmuQvIiKRQUlZLUVHGR0S1NW/uaRnek1jNclfREQihJKyOkhOiNP6l80kIyuPbu1aktJGTWNFRCQyKCmrA61/2XzSs3IZnqYqmYiIRA4lZXWQoq7+zUJNY0VEJBIpKasD/1JLFRXN0q82YmVk+ZrGanklERGJJErK6iA5MY6yCkfe3tJAhxLW0jN9TWMPVdNYERGJIErK6iAl0TfpXEOYTSs9K5ch3dQ0VkREIos+9epg/1JLaiDbVIrLylm8abcm+YuISMRRUlYHyVpqqcn5m8Zqkr+IiEQaJWV1oPUvm54m+YuISKRSUlYHreNiaBUbTbaSsiaTnpVLt3Yt6aSmsSIiEmGUlNVRsnqVNamMzFytdykiIhFJSVkdqYFs09m6q4jNu4q03qWIiEQkJWV1lJwYp6svm0h6lrcIua68FBGRCKSkrI6SE1QpayoZWWoaKyIikUtJWR2ltIlnd1EZRaXlgQ4l7KRn5alprIiIRCx9+tVRcoLaYjSFkrIKFm3apUn+IiISsZSU1dG+BrIFSsoa05LNuygpq9AkfxERiVhKyupIXf2bRrrXNFaT/EVEJFIpKaujlH3rXyopa0xqGisiIpFOSVkdtW8di5kqZY1tflae5pOJiEhEU1JWRzHRUXRorbYYjWnb7iI25e3VepciIhLRlJTVg2+pJTWQbSzpmV7TWFXKREQkgikpqwetf9m40rNyiY2JYlDXtoEORUREJGCUlNWD1r9sXGoaKyIioqSsXpIT48gpKMY5F+hQQp6/aayGLkVEJNIpKauH5IQ4SssdeYWlgQ4l5C3dsltNY0VERFBSVi8pbdTVv7H4J/nryksREYl0Ssrqwb/+ZfZuJWUNlZ6VS9e28XRuq6axIiIS2ZSU1cP+9S/VFqOhMrLyGKallURERJSU1UeKtxSQrsBsGH/TWM0nExERUVJWL61jo2nZIlrDlw2UkaWmsSIiIn5KyurBzPa1xZD6S8/KIzY6ikO7tgl0KCIiIgGnpKye1EC24dIzcxncrQ1xMdGBDkVERCTgGpSUmdl5ZrbEzCrMbGSVfUPNbI63f5GZxXvbY83sOTNbaWbLzewcb3ucmb1hZqvN7Fsz69mQ2JpacmIc2UrK6q2krIKFm3ZpPpmIiIinoZWyxcDZwJeVN5pZDPAKcJ1zbhAwAfB3Wr0HyHbO9QMOBWZ6268Ccp1zfYDHgL82MLYmpfUvG2Zf01hdeSkiIgJATEMe7JxbBr45VlWcCCx0zi3w7rej0r7/AwZ42yuA7d72M4B7vdtvA0+ZmbkgXcsoJTGOXXtLKS4r1/BbPfibxqpSJiIi4tNUc8r6Ac7MpplZupndAWBm/svs7ve2v2Vmnbxt3YANAM65MmAX0KGJ4muwfb3KVC2rl4wNeWoaKyIiUkmNSZmZfWpmi6v5OuMgD4sBxgGXeP+eZWbHedu7A18754YDc4C/+U9VzXGqrZKZ2TVmNtfM5ubk5NT0FJqEkrKGSc/M1dJKIiIildQ4fOmcO74ex90IzHTObQcws4+A4cDnQCEwybvfW/jmkvkf0wPY6M1JawvsPEBMzwHPAYwcOTIgw5spiWogW1/ZXtPYK8f2DHQoIiIiQaOphi+nAUPNrJWXYB0DLPXmh32Ab+I/wHHAUu/2+8Dl3u1zgc+DdT4Z7K+U6QrMukv3N43VJH8REZF9GjTR38zOAp4EkoEpZjbfOXeScy7XzB4Fvsc3BPmRc26K97A7gf+a2eNADnClt/15b/tqfBWyCxsSW1Pr0DoWM1XK6sPfNHaQmsaKiIjs09CrLyexfyiy6r5X8LXFqLo9ExhfzfYi4LyGxNOcYqKj6NA6Vl396yEjS01jRUREqlJH/wbomBCn9S/rqKSsgoUb1TRWRESkKiVlDaD1L+tu2ZbdFJdV6MpLERGRKpSUNUBKYjzbNaesTvZP8m9Xwz1FREQii5KyBvAvtRTEF4kGnfSsPLq0jadL25aBDkVERCSoKClrgOTEOErKK9i1t7TmOwvgaxqr+WQiIiI/pqSsAVLU1b9O/E1jh6Vq6FJERKQqJWUNoAaydZOelQegSf4iIiLVUFLWAFr/sm4ysnKJjY5icDc1jRUREalKSVkDaPiybtKzchmkprEiIiLVUlLWAAlxMcS3iCI7vyjQoQQ9NY0VERE5OCVlDWBm+9piyMH5m8YqKRMREamekrIGSkmMV1f/WshQ01gREZGDUlLWQMla/7JW0rPy6NxGTWNFREQORElZA2n9y9pJz8pVlUxEROQglJQ1UEpiHHmFpRSXlQc6lKCVnV/Exty9mk8mIiJyEErKGsjfq2x7QUmAIwle6ZlqGisiIlITJWUNpAayNcvYoKaxIiIiNVFS1kApifGAkrKDycjMU9NYERGRGigpa6D961+qgWx1SssrWLgpj2E9NHQpIiJyMErKGqhDQixmqpQdyLItuykqrdCVlyIiIjVQUtZALaKjaN8qlu/W7eSrVdvZtrsI51ygwwoa6Zle01hN8hcRETmomEAHEA6GpyUxfek2Zq/ZAUBiXAx9OiXQNyWBPikJ9E1JpE9KAt3atSQqygIcbfPyN43t2k5NY0VERA5GSVkjeO7SEWwvKGFVdj6rswtYta2A1dkFfL48hzfnbtx3v5Ytoumd0npfkuZL2BJIbd+KmOjwLFpmbFDTWBERkdpQUtYI/AuTJyfGMaZ3xx/syyss8SVqXrK2Kjufb9fuYFLGpn33iY2O4pCOrfdV1/xJW8+OrUL6isWc/GI27NzLZaN6BjoUERGRoKekrIm1axXLyJ7tGdmz/Q+25xeVsiZnMR0LaAAACOhJREFUD6u25bM6p4DV2wpYtHEXHy3agn9KWnSUkdahFX2SE+jbaX+y1js5gZaxwZ+spWsRchERkVpTUhYgifEtOLxHOw7v8cOEpai0nDU5BT8YBl2Vnc9ny7Mpr/Bla2bQPanlj4ZB+6QkkBjfIhBPp1rpWbm0iDYGdW0b6FBERESCnpKyIBPfIppBXdv+KJEpKasgc8eeHwyDrs4u4KtV2ykpr9h3v85t4unbKaFSspZI35QEklrHNvdT8TWN7dqW+BbBX9UTEREJNCVlISI2Joq+nRLp2ykRhuzfXlZewYbcvT8YBl2VXcD/vtvA3tL9i6R3aB3rS9IqDYP2TUkgOTEOs8a/ItTfNPbiI9Ma/dgiIiLhSElZiIvxLhI4pGNrTqy0vaLCsXnXXlZl+xI1/zDo5PmbyS8q23e/NvEx9O2UuG/emr/C1rVtw9p3LN+Sr6axIiIidaCkLExFRRndk1rRPakVx/ZP2bfdOUd2frE3Zy3fl7RlF/Dpsm28MXfDvvu1io3el6BVHgbt0b4V0bVI1vyT/IepaayIiEitKCmLMGZGpzbxdGoTz9g+P2zfsaPAl6ytztl/kcHs1Tt4N71S+46YKHp1bP2D6lrflATSOrQmNmZ/r7X0rFw6tYmja9v4ZntuIiIioUxJmezTISGODglxHNWrww+27y4q9SVr2/wJWz4ZWbl8sGDzvvvEeO07+qYk0rdTAt+s3cHw1KQmma8mIiISjpSUSY3axLdgeGrSj9avLCwpY23OHlZl5++rrK3cls8nS7dS4WBM7w4HOKKIiIhUpaRM6q1VbAyDu7VlcLcftu8oLitnS14RPdq3ClBkIiIioUdJmTS6uJhoenZsHegwREREQkqDVsE2s/PMbImZVZjZyCr7hprZHG//IjOL97Zf5H2/0MymmllHb3t7M5tuZqu8f3XZnoiIiESMBiVlwGLgbODLyhvNLAZ4BbjOOTcImACUetv/Dhzr3P+3d38hct1lGMe/j0n8jxZM1NhWKxjBKGlalxBRsNiqqRddixUioolUCsWigiDVC6W90psKFlGiLY0i2lBFVxsJtrbWG2vXkKatayEKanQx27QmilKb8ngxv9h0MrtzZqdnzjm7zweGnMn5ncnLw8vy7vzOTLwFOAxcVy67Hrjb9ibg7vI8IiIiYlUYayizPWf70QGn3gMctv1gWXfc9tOAyuMl6n0s72XA6Y/wTQN7y/Fe4P3j1BYRERHRJeO+U7aYNwKWdEDSQUmfBbD9FHAt8BC9YWwzcEu55lW258u6eeCVZ79sRERExMo0dCiTdJekhwc8ppe4bC3wDuDD5c8rJV0qaR29oewi4DX0ti8/N2rRkq6RNCtpdmFhYdTLIyIiIlpn6KcvbV+2jNc9CvzS9mMAkvYDFwMny2v+ofz9Pp65d+zvkjbanpe0ETi2RE17gD0AU1NTXkZ9EREREa1S1/blAWCLpBeXm/vfCfwO+CuwWdKGsu7dwFw5ngF2leNdwI9rqi0iIiKidcb6njJJVwI3AxuAOyUdsv1e209Iugl4ADCw3/ad5ZobgPskPQX8CdhdXu5LwD5JVwN/Bj44Tm0RERERXSK727t/U1NTnp2dbbqMiIiIiKEk/db21KBzdW1fRkRERMQIOv9OmaQFetug41gPPPYclBPPllzrkVzrkVzrkVzrkVzrMYlcX2d7w6ATnR/KnguSZhd7KzGWL7nWI7nWI7nWI7nWI7nWo+lcs30ZERER0QIZyiIiIiJaIENZz56mC1ihkms9kms9kms9kms9kms9Gs0195RFREREtEDeKYuIiIhogVUzlEnaIelRSUckXT/g/Ask3V7O3y/pgslX2T0Vct0taUHSofL4eBN1do2kWyUdk/TwIucl6asl98OSLp50jV1UIddLJJ04o1+/MOkau0bS+ZLukTQn6RFJnxqwJv06ooq5pl9HJOmFkn4j6cGS6w0D1jQ2D6yKoUzSGuBrwOXAZuBDkjb3LbsaeML2G4CvAF+ebJXdUzFXgNttby2Pb020yO66DdixxPnLgU3lcQ3w9QnUtBLcxtK5AvzqjH69cQI1dd0p4DO23wRsBz4x4OdA+nV0VXKF9OuongTeZftCYCuwQ9L2vjWNzQOrYigDtgFHbP/R9n+B7wPTfWumgb3l+A7gUkmaYI1dVCXXWAbb9wGPL7FkGvi2e34NnCNp42Sq664KucaIbM/bPliO/wnMAef2LUu/jqhirjGi0oP/Kk/XlUf/zfWNzQOrZSg7F/jLGc+PcnZz/3+N7VPACeAVE6muu6rkCvCBsmVxh6TzJ1Pailc1+xjd28rWxs8kvbnpYrqkbPNcBNzfdyr9OoYlcoX068gkrZF0CDgG/Nz2ov066XlgtQxlgybc/sm4ypp4tiqZ/QS4wPYW4C6e+e0jxpN+rcdBev8FyoXAzcCPGq6nMyS9FPgB8GnbJ/tPD7gk/VrBkFzTr8tg+2nbW4HzgG2S3tK3pLF+XS1D2VHgzHdozgP+ttgaSWuBl5NtjmGG5mr7uO0ny9NvAm+dUG0rXZWejhHZPnl6a8P2fmCdpPUNl9V6ktbRGxy+a/uHA5akX5dhWK7p1/HY/gdwL2ffZ9rYPLBahrIHgE2SXi/p+cBOYKZvzQywqxxfBfzC+RK3YYbm2nffyBX07ouI8c0AHy2fatsOnLA933RRXSfp1afvHZG0jd7PyOPNVtVuJa9bgDnbNy2yLP06oiq5pl9HJ2mDpHPK8YuAy4Df9y1rbB5YO4l/pGm2T0m6DjgArAFutf2IpBuBWdsz9Jr/O5KO0JuIdzZXcTdUzPWTkq6g90mix4HdjRXcIZK+B1wCrJd0FPgivRtSsf0NYD/wPuAI8G/gY81U2i0Vcr0KuFbSKeA/wM78cjbU24GPAA+V+3QAPg+8FtKvY6iSa/p1dBuBveXbA54H7LP907bMA/lG/4iIiIgWWC3blxERERGtlqEsIiIiogUylEVERES0QIayiIiIiBbIUBYRERHRAhnKIiIiIlogQ1lEREREC2Qoi4iIiGiB/wHeAI22Xv+AVgAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Hold out 10% of our original data to test upon\n", "df_test = df.sample(frac=0.1)\n", "data = torch.tensor(df.drop(df_test.index)['sunspot.year'].values, dtype=torch.float).round()\n", "data_test = torch.tensor(df_test['sunspot.year'].values, dtype=torch.float).round()\n", "N = data.shape[0]\n", "N_test = data_test.shape[0]\n", "\n", "alphas = [0.05, 0.1, 0.5, 0.75, 0.9, 1., 1.25, 1.5, 2, 2.5, 3]\n", "log_predictives = []\n", "\n", "for val in alphas:\n", " alpha = val\n", " T = 20\n", " svi = SVI(model, guide, optim, loss=Trace_ELBO())\n", " train(500)\n", " \n", " S = 100 # number of Monte Carlo samples to use in posterior predictive computations\n", "\n", " # Using pyro's built in posterior predictive class:\n", " posterior = Predictive(guide, num_samples=S, return_sites=[\"beta\", \"lambda\"])(data)\n", " post_pred_weights = mix_weights(posterior[\"beta\"])\n", " post_pred_clusters = posterior[\"lambda\"]\n", "\n", " # log_prob shape = N_test x S\n", " log_prob = (post_pred_weights.log() + Poisson(post_pred_clusters).log_prob(data.reshape(-1, 1, 1))).logsumexp(-1)\n", " mean_log_prob = log_prob.logsumexp(-1) - np.log(S)\n", " log_posterior_predictive = mean_log_prob.sum(-1)\n", " log_predictives.append(log_posterior_predictive)\n", "\n", "plt.figure(figsize=(10, 5))\n", "plt.plot(alphas, log_predictives)\n", "plt.title(\"Value of the Log Predictive at Varying Alpha\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "PGw4eD363PWP" }, "source": [ "From the above plot, we would surmise that we want to set $\\alpha > 1$, though the signal is not quite clear. A more comprehensive model criticism process would involve performing a grid search across all hyperparameters in order to find the one that maximizes the log predictive.\n" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "8A1gt4sn8Upo" }, "source": [ "## References\n", "\n", "\n", "\n", "1. Ferguson, Thomas. *A Bayesian Analysis of Some Nonparametric Problems*. The Annals of Statistics, Vol. 1, No. 2 (1973). \n", "2. Aldous, D. *Exchangeability and Related Topics*. Ecole diete de Probabilities Saint Flour (1985).\n", "3. Sethuraman, J. *A Constructive Definition of Dirichlet Priors*. Statistica, Sinica, 4:639-650 (1994).\n", "4. Blei, David and Jordan, Michael. *Variational Inference for Dirichlet Process Mixtures*. Bayesian Analysis, Vol. 1, No. 1 (2004).\n", "5. Pedregosa, et al. *Scikit-Learn: Machine Learning in Python*. JMLR 12, pp. 2825-2830 (2011).\n", "6. Bishop, Christopher. *Pattern Recogition and Machine Learning*. Springer Ltd (2006).\n", "7. *Sunspot Index and Long-Term Solar Observations*. WDC-SILSO, Royal Observatory of Belgium, Brussels (2018).\n", "9. Gelman, Andrew. *Understanding predictive information criteria for Bayesian models*. Statistics and Computing, Springer Link, 2014.\n", "\n", "\n", "\n" ] } ], "metadata": { "accelerator": "TPU", "colab": { "collapsed_sections": [], "name": "Pyro Dirichlet Process Mixture Model", "provenance": [] }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.10" } }, "nbformat": 4, "nbformat_minor": 4 }