{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Bayesian Optimization\n", "\n", "[Bayesian optimization](https://en.wikipedia.org/wiki/Bayesian_optimization) is a powerful strategy for minimizing (or maximizing) objective functions that are costly to evaluate. It is an important component of [automated machine learning](https://en.wikipedia.org/wiki/Automated_machine_learning) toolboxes such as [auto-sklearn](https://automl.github.io/auto-sklearn/stable/), [auto-weka](http://www.cs.ubc.ca/labs/beta/Projects/autoweka/), and [scikit-optimize](https://scikit-optimize.github.io/), where Bayesian optimization is used to select model hyperparameters. Bayesian optimization is used for a wide range of other applications as well; as cataloged in the review [2], these include interactive user-interfaces, robotics, environmental monitoring, information extraction, combinatorial optimization, sensor networks, adaptive Monte Carlo, experimental design, and reinforcement learning.\n", "\n", "## Problem Setup\n", "\n", "We are given a minimization problem\n", "\n", "$$x^* = \\text{arg}\\min \\ f(x),$$\n", "\n", "where $f$ is a fixed objective function that we can evaluate pointwise. \n", "Here we assume that we do _not_ have access to the gradient of $f$. We also\n", "allow for the possibility that evaluations of $f$ are noisy.\n", "\n", "To solve the minimization problem, we will construct a sequence of points $\\{x_n\\}$ that converge to $x^*$. Since we implicitly assume that we have a fixed budget (say 100 evaluations), we do not expect to find the exact minumum $x^*$: the goal is to get the best approximate solution we can given the allocated budget.\n", "\n", "The Bayesian optimization strategy works as follows:\n", "\n", "1. Place a prior on the objective function $f$. Each time we evaluate $f$ at a new point $x_n$, we update our model for $f(x)$. This model serves as a surrogate objective function and reflects our beliefs about $f$ (in particular it reflects our beliefs about where we expect $f(x)$ to be close to $f(x^*)$). Since we are being Bayesian, our beliefs are encoded in a posterior that allows us to systematically reason about the uncertainty of our model predictions.\n", "\n", "2. Use the posterior to derive an \"acquisition\" function $\\alpha(x)$ that is easy to evaluate and differentiate (so that optimizing $\\alpha(x)$ is easy). In contrast to $f(x)$, we will generally evaluate $\\alpha(x)$ at many points $x$, since doing so will be cheap.\n", "\n", "3. Repeat until convergence:\n", "\n", " + Use the acquisition function to derive the next query point according to\n", " $$x_{n+1} = \\text{arg}\\min \\ \\alpha(x).$$\n", "\n", " + Evaluate $f(x_{n+1})$ and update the posterior.\n", "\n", "A good acquisition function should make use of the uncertainty encoded in the posterior to encourage a balance between exploration—querying points where we know little about $f$—and exploitation—querying points in regions we have good reason to think $x^*$ may lie. As the iterative procedure progresses our model for $f$ evolves and so does the acquisition function. If our model is good and we've chosen a reasonable acquisition function, we expect that the acquisition function will guide the query points $x_n$ towards $x^*$.\n", "\n", "In this tutorial, our model for $f$ will be a Gaussian process. In particular we will see how to use the [Gaussian Process module](http://docs.pyro.ai/en/0.3.1/contrib.gp.html) in Pyro to implement a simple Bayesian optimization procedure." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import matplotlib.gridspec as gridspec\n", "import matplotlib.pyplot as plt\n", "import torch\n", "import torch.autograd as autograd\n", "import torch.optim as optim\n", "from torch.distributions import constraints, transform_to\n", "\n", "import pyro\n", "import pyro.contrib.gp as gp\n", "\n", "assert pyro.__version__.startswith('1.8.3')\n", "pyro.set_rng_seed(1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Define an objective function\n", "\n", "For the purposes of demonstration, the objective function we are going to consider is the [Forrester et al. (2008) function](https://www.sfu.ca/~ssurjano/forretal08.html):\n", "\n", "$$f(x) = (6x-2)^2 \\sin(12x-4), \\quad x\\in [0, 1].$$\n", "\n", "This function has both a local minimum and a global minimum. The global minimum is at $x^* = 0.75725$." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "def f(x):\n", " return (6 * x - 2)**2 * torch.sin(12 * x - 4)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's begin by plotting $f$." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeIAAAD4CAYAAAAw0+XqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3hc1YH+8e9R712WZEmuuOBeZNl0EpoxLRAgBuzYQGI2WTZlkzyBze6GXxKSLNmEze4vodtgb6gJCQ64gKkJsY1lXCR3XNWL1etIM2f/kHAcx0W2RnOnvJ/nmccaaTz35SDp9T333nONtRYRERFxRpjTAUREREKZilhERMRBKmIREREHqYhFREQcpCIWERFxUIQTG83IyLAjRoxwYtMiIiI+t3nz5jprbebJvuZIEY8YMYKioiInNi0iIuJzxpjDp/qapqZFREQcpCIWERFxkIpYRETEQSpiERERB6mIRUREHKQiFhERcZCKWERExEEqYhERkeO8u7uG32w85WW/XqciFhERAdpdPXzv98Xc/ewmXtpUittjfbJdR1bWEhER8SdbjjTwzy9v49DRNpZcOop/vmos4WHGJ9tWEYuISEh76oMD/HTNbrKTYnj+S3O4YHS6T7evIhYRkZD16sdlPLxqF9dOyuY/bp1CUkykzzOoiEVEJCQVHarngd8Vc8GodP77julEhjtz2lS/t2qMWWqMqTHGlBz3uYeMMeXGmK19j3mDE1NERMR7SuvbWbJiM7mpsTy2YIZjJQxnd9b0s8Dck3z+UWvttL7HKu/EEhERGRwtnd3c+9wmetwenllUQEpclKN5+l3E1toPgPpBzCIiIjKorLV886Wt7K9t47EFMxmVmeB0JK9cR3y/MWZ739R16qleZIxZYowpMsYU1dbWemGzIiIiZ+eP2ytZt6uGB68dz0XnZTgdBxh4ET8GjAamAZXAz0/1Qmvtk9baAmttQWZm5gA3KyIicnaaO7v54es7mZybzN0XjXQ6zjEDKmJrbbW11m2t9QBPAYXeiSUiIuJdv3hzL3WtXTx88ySfLdbRHwMqYmNMznFPbwZKTvVaERERp5SUN7F8/SEWzB7OlLwUp+P8jX5fR2yMeQG4HMgwxpQB3wcuN8ZMAyxwCLhvEDKKiIicM7fH8r0/lJAWH8W3rxnndJy/0+8ittbecZJPP+PFLCIiIl734qYjbCtt5NEvTCU51vcrZ52J7r4kIiJBq7mzm0fW7OGCUel8blqu03FOSkUsIiJB67kPD9HU0c33rjsfY/znBK3jqYhFRCQotXR28/SfD3Ll+UOYlJvsdJxTUhGLiEhQWr7+ME0d3XztijFORzktFbGIiASdtq4env7TAT4zLtPvLlc6kYpYRESCzv9uOExDu//vDYOKWEREgky7q4cnPzjApWMzmT7slLdA8BsqYhERCSrPbzzC0TYXX7/iPKej9IuKWEREgkZnt5vH3z/AReelM3N4mtNx+kVFLCIiQeOP2yqoa+3iq5cHxt4wqIhFRCRIWGt5bv0hxmYlcOHodKfj9JuKWEREgsLHRxopKW/mixeM8NtVtE5GRSwiIkFh+fpDJMZEcPN0/1xT+lRUxCIiEvBqWjpZVVzJbTPziY/u940F/YKKWEREAt4LG0vpdlsWXjDc6ShnTUUsIiIBzdXj4TcbD3PZ2ExGZsQ7HeesqYhFRCSgrd1RRU1LF4svHOF0lHOiIhYRkYC2fP0hhqfHcdnYTKejnBMVsYiIBKxdlc1sOtTAwjnDCQsLnEuWjqciFhGRgPVyUSlR4WHcOjPP6SjnTEUsIiIBydXj4bWtFVw1IYuUuCin45wzFbGIiASkd3ZXU9/m4taCwN0bBhWxiIgEqFeKyshKiubSMYF5ktanVMQiIhJwalo6eW9vLbfMyCM8QE/S+pSKWEREAs7vPy7H7bHcFsAnaX1KRSwiIgHFWssrm8uYOTyVUZkJTscZMBWxiIgElK2ljXxS0xoUe8OgIhYRkQDzyuYyYiLDuG5KjtNRvEJFLCIiAaPD5eaPWyuYNymHxJhIp+N4hYpYREQCxlu7qmnp6gnolbROpCIWEZGAsXJrOdlJMcwZle50FK9REYuISEBobHfx/t5abpiaE7A3eDgZFbGIiASE1SVVdLstN03LdTqKV6mIRUQkILy2tZxRmfFMHJrkdBSvUhGLiIjfq2rqZOPBem6amosxwTMtDSpiEREJAK9vr8BauHHaUKejeF2/i9gYs9QYU2OMKTnuc2nGmLeMMfv6/kwdnJgiIhLKXttawZS8ZEZmxDsdxevOZo/4WWDuCZ97AHjbWjsGeLvvuYiIiNccqG2luLyJG6cG394wnEURW2s/AOpP+PRNwHN9Hz8HfM5LuURERABYua0CY+CGUC/iU8iy1lYC9P055FQvNMYsMcYUGWOKamtrB7hZEREJBdZaVm6tYM7IdLKSYpyOMyh8drKWtfZJa22BtbYgMzPTV5sVEZEAVlLezIG6Nm4KwpO0PjXQIq42xuQA9P1ZM/BIIiIivV7fXkFkuGHupGynowyagRbxSmBR38eLgNcG+H4iIiJA77T0G8WVXHReBilxUU7HGTRnc/nSC8B6YJwxpswYcy/wU+AqY8w+4Kq+5yIiIgNWXN5EWUMH8yYHx32HTyWivy+01t5xii9d4aUsIiIix7yxvZLIcMM1E4J3Whq0spaIiPih46elk+MinY4zqFTEIiLid0JlWhpUxCIi4ofeKK4kIsxw9YQsp6MMOhWxiIj4FWstq0LgbOlPqYhFRMSvlJQ3U1rfwXUhMC0NKmIREfEzx6alJwb/tDSoiEVExI+E2rQ0qIhFRMSP7Kho5kh9e8hMS4OKWERE/EioTUuDilhERPyEtZY1JVVcMDo9ZKalQUUsIiJ+Ym91Kwfr2oL6TksnoyIWERG/sLqkEmPg6iBfW/pEKmIREfELa0qqmDU8jczEaKej+JSKWEREHHeoro3dVS0hNy0NKmIREfEDa3ZUAXCNilhERMT3VpdUMTUvmdyUWKej+JyKWEREHFXR2MG20saQ3BsGFbGIiDhsbd+09NyJKmIRERGfW11SxbisREZlJjgdxREqYhERcUxtSxebDtWH5NnSn1IRi4iIY97aWY21qIhFREScsGZHFSPS4xifneh0FMeoiEVExBFNHd2s31/HNZOyMcY4HccxKmIREXHEu7tr6HZbrgnRs6U/pSIWERFHrCmpIispmml5KU5HcZSKWEREfK7D5eb9vbVcPSGbsLDQnZYGFbGIiDjgg321dHS7Q/ps6U+piEVExOfWllSRHBtJ4cg0p6M4TkUsIiI+1e32sG5XNVeen0VkuGpIIyAiIj614cBRmjt7NC3dR0UsIiI+taakiriocC4Zk+F0FL+gIhYREZ/xeCxv7qzm8nGZxESGOx3HL6iIRUTEZ7aUNlDb0hXyi3gcT0UsIiI+s6akishww2fGD3E6it9QEYuIiE9Ya1m7o5oLR2eQFBPpdBy/oSIWERGf2FXZwpH6dq7V2dJ/I8Ibb2KMOQS0AG6gx1pb4I33FRGR4LGmpJIwA1dOyHI6il/xShH3+Yy1ts6L7yciIkFkzY4qZo1IIyMh2ukofkVT0yIiMuj217ayt7pVi3ichLeK2AJvGmM2G2OWnOwFxpglxpgiY0xRbW2tlzYrIiKBYO2OKgBdtnQS3irii6y1M4BrgX80xlx64gustU9aawustQWZmZle2qyIiASCtSVVTM1PYWhKrNNR/I5XithaW9H3Zw3we6DQG+8rIiKBr7yxg21lTczV3vBJDbiIjTHxxpjETz8GrgZKBvq+IiISHN48Ni2ts6VPxhtnTWcBvzfGfPp+z1tr13jhfUVEJAisKaliXFYiozITnI7ilwZcxNbaA8BUL2QREZEgU9faxaZD9dz/2TFOR/FbunxJREQGzbqd1XgsOj58GipiEREZNKtLqhieHsf5OYlOR/FbKmIRERkUTe3d/GV/HXMnZtN3HpGchIpYREQGxbpd1XS7LddOznE6il9TEYuIyKBYXVLJ0OQYpuYlOx3Fr6mIRUTE65o7u/lgbx3XTs7RtPQZqIhFRMTr3tlVg8vtYd5knS19JipiERHxulXFlWQlRTM9P9XpKH5PRSwiIl7V2tXDe3truXZSDmFhmpY+ExWxiIh41bu7a3D1eLhW9x7uFxWxiIh41eqSSjISoikYkeZ0lIAQ8EW8bmc1S5YX4erxOB1FRCTktbt6eHd3LXMnZRGuael+Cfgirm9z8ebOar79yjY8Hut0HBGRkPb+nlo6ut3Mm6RFPPrLG7dBdNTts/Kpa+vikTV7SE+I4t+vn6Br1kREHLKqpIq0+CgKR2paur8CvogBvnLZaGpbulj24SEyE6P56uXnOR1JRCTkdLjcvL2rmhunDiUiPOAnXH0mKIrYGMO/XTeB+jYXj6zZQ0ZCNLcX5DsdS0QkpLy7p4Z2l5sbpg51OkpACYoiBggLM/zs1qnUt7l48NViUmIjuVr3vxQR8ZnXt1eQkRDNnFHpTkcJKEE1dxAVEcbjC2YyOTeZ+1/Ywvr9R52OJCISElq7enh7Vw3zJmfrbOmzFFRFDBAfHcGyxbMYnhbHl5cXUVzW5HQkEZGg9/auarp6PJqWPgdBV8QAqfFRrLh3NilxkSxa9hGf1LQ6HUlEJKj9cVsF2UkxzBymtaXPVlAWMUB2cgwr7p1NmIGFz2ykrKHd6UgiIkGpqb2b9/fWcv0UrS19LoK2iAFGZsTz3D2FtHX1cOdTG6lq6nQ6kohI0Fm7s4put+V6TUufk6AuYoCJQ5N57p5C6ttc3Pn0BmpbupyOJCISVF7fXkl+WixT85KdjhKQgr6IAaYPS2Xp4llUNnay8JmNNLS5nI4kIhIU6ttcfPhJHddPGapVDc9RSBQxQOHINJ5eVMCBujYWLt1IU0e305FERALe6pJK3B7LDVM0LX2uQqaIAS46L4MnFsxkT1ULdz29QXvGIiID9Pq2SkZlxnN+TqLTUQJWSBUxwGfGD+HJhQXsrW7ljqc2cLRVx4xFRM5FVVMnGw4e5QZNSw9IyBUx9JbxM4sKOFjXxvwnN1DTorOpRUTO1spt5VgLn5ue63SUgBaSRQxwyZhMlt09i7KGDuY/uUGXNomInKVXPy5nWn4KIzPinY4S0EK2iAEuHJ3B8nsLqWnu4pZff8gnNS1ORxLxKWstLZ3dlNa3s7+2ld1VzWwva2TLkQZ2VzVz5Gg7da1ddHa7nY4qfmZXZTO7q1q4ZYb2hgcqaO6+dK5mjUjjxSVzWLxsE7c+vp5nFs1i5nAt0SbBwe2xHD7axsG6No7Ut1Na38GR+nYqmzqob3NxtNWFy+3p13ulxkWSlxpHXmoseamxTByazNT8FEakx+n4YAj6w5ZyIsIM1+ts6QEL+SIGmJSbzKtfuZAvLt3IXU9v4Fd3zuCK87OcjiVyVupau9hR0UxJeRN7qlrYW93Cgbo2XD1/LdrYyHCGpcUxNCWGCTlJpCVEkREfTXJcJNERYUSFhxEVEUaYMXR0u2l3uWl39dDS2UNFYwdlDR3srW7hnd01dPW9b0pcJFPzUrhkTAZXTchieLqmKYOd22P5w9ZyLh+XSVp8lNNxAp6x1vp8owUFBbaoqMjn2z2TutYu7l62iZ2Vzfy/GyeyYM5wpyOJnNTR1i62lzexvbSJ4vJGisubqG7+6xUAuSmxjMlKYGxWImOGJDAqM4Hh6XGkx0d5Ze+1x+1hX00r20ob2VrayObDDezru7nK2KwErpqQxeem5TImS5e0BKMPP6njrqc38qs7Z3DdlByn4wQEY8xma23BSb+mIv5bbV093P/8x7y7p5a7Zg/j+zdMJCoipA+li8PaunooKW9iW1kj20qb2FraSHljBwDGwOjMBCbnJjNxaBIThyYzYWgSybGRPs9ZWt/OmzureWtnFZsONeD2WApHprFgznDmTszWz1EQ+dbL23hzRxWb/vVKYiLDnY4TEFTEZ8ntsTyydjdPvH+AwpFpPHbXDNITop2OJSGgs9vNnqoWtpc1sq2sie1ljXxS04qn78c0LzWWqXkpTM1PZkpeCpNyk0mI9r8jTEdbu3hlcxnPbzzCkfp2MhKiWDhnBPdcPILEGN//I0G8p8PlpuBHb3H9lKH8x61TnI4TMFTE5+gPW8r57u+2k5EQzRMLZzIpVwuai/c0tLnYXdXC7qrmY8d299W04u5r3fT4KKbkJTM5L4VpfcWbEWD/IPR4LB/sq2XF+sO8vbuGlLhIllw6isUXjiAuyv/+ASFn9trWcr7+4lZe+PIcLhid7nScgDHoRWyMmQv8EggHnrbW/vR0rw+UIgbYXtbIfSs2U9faxXfnjueei0bqfpvSb9Zaalu62F/bxv7aVg7UtrGvpoU9VS3UHHcnsIyEaCblJh2bXp6Sl0xuSmxQnY1cXNbEL97aw7t7akmPj+JrV4xhwZzhhOvnKaDcvewj9lS18Ofvfla/C8/CoBaxMSYc2AtcBZQBm4A7rLU7T/V3AqmIoXfP5bu/286bO6u5ZEwGP79tKkOSYpyOJQ6z1tLa1UNdq4vq5s5jj4rGTkrr2ylt6L1cqOO4a3BjI8MZlRnPuOxExmcnMj47ifHZiSH1/bT5cAP/uXYP6w8cZVJuEj++eTJT8lKcjiX9UNPSyQU/eYcll47iu3PHOx0noAx2EV8APGStvabv+YMA1tqfnOrvBFoRQ+8v3ec/OsIPX99JXFQEP755MnMnZTsdK6i5ejy0u3ro7PbQ2e2ms8eNq8dDt9vi9lh63B56PBaPtVh6/x95Trgk9sTvbnv8ay1YCz0eDz1uS4+n971dPR46e9x0dnvo6ruEp6Wzm9au3st4mjq6qW9z0dDuotv99z8/CdER5KXGkp8Wx7C+x6jMeEZnJpCdFKO9CHrH/43iSn7wx53UtnbxxTnD+dY140jS8WO/9vj7+/np6t28/a3LGJ2Z4HScgHK6IvbGQZpcoPS452XA7JOEWAIsARg2bJgXNutbxhjumj2c2SPT+NoLW/mH/93Mledn8dCNE8hLjXM6XkBo6+qhsql3r7GqqZPqlk6OtrpoaHNR3977Z1PHXwuvq6d/C00MpuiIMOKjI0j49BHTW7JT8pJJi48mLT6S9PhospNjyEqKJispRicj9YMxvQtBXDo2k1+8uZfn1h9i7Y5qfvGFqVw4OsPpeHIS1lpe3lRK4Yg0lbCXeWOP+DbgGmvtl/qeLwQKrbX/dKq/E4h7xMfrdntY+ueD/Ne6fQB848ox3HPxSCLDdXlGU0c3B2pb2V/bxoHa1r7VnNopbehdyelE8VHhpMZHkRYfRWpcFMmxkSTG9BZeYnQEcVERxESGExMZRkxkOFHhYUSEGyLDwwgPM0SEGYwxhBkIMwZjwPC3e5wnHmY1J7w2Irz3fSLCw4gIM0RH9G4rOiIsqI7R+rNtpY1886WtHDzaxlcuG803rxqrnyc/89HBem5/Yj3/edtUbp2Z53ScgDPYe8RlQP5xz/OACi+8r9+KDA/jvstGc92UHB5auYOfrN7Ny0WlfOPKsVw3OSckph67etzsq25lV2Uze6pa2F3Vwp7qFmqPOwEpIsyQmxpLfmoc10xMJj8tlqHJsWQl9e49ZifH6MxZAWBqfgqvf+1ifvDHnfz6vf18+Ekdv5w/nRG6mYDfeHHTERKjI5g3WYfkvM0be8QR9J6sdQVQTu/JWndaa3ec6u8E+h7xid7aWc0ja3azr6aV8dmJfPOqsVw9ISto9qY6u93sqGimuKyx9zKbimb2VbfQ03eZTXREGGOzEhmXnch5QxIYnZnA6Mx48tPitFcjZ21VcSUP/G47Hgu/nD9Ny836gebObgofXsfnZ+Tx8M2TnY4TkAZ1j9ha22OMuR9YS+/lS0tPV8LB6KoJWXx2/BBe317Bf63bx30rNnN+ThIL5wznpmlDiffDBRdOpcftYW91a9+CEr0rOe09rnTT46OYlJvMZ8dnMnFoMuOzExmeHq9LUMRr5k3OYWp+CvetKOJLy4v49tXj+Orlo4PmH7aBaOXWCjq7PXxhVv6ZXyxnTQt6eFmP28OrW8pZ+ueD7K5qITE6gltm5DK/cBjjsxP96peJtZbDR9vZVtbI9r5VnIrLm+js7j1JKjk2kil5yX2PFKbkJZOdFONX/w0SvDpcbr77u+2s3FbB9VNy+NmtU4mN0nKKTrjhf/5Mj8ey6msX6+f/HA32MWI5TkR4GLcX5HPbzDw2H27gNxuP8MJHpTy3/jDD0+O46vwsrp6YzczhqT7di3R7LAfrWtlR0dz3aKK4rInmzh6gd3p5wtAk7igcxrT8FKbmpTBct7cTB8VGhfPL+dM4PyeJR9bu5mBdG8sWzwqpa679wY6KJorLm3johgn6fTBItEfsA/VtLlYVV/LWzmrW7z+Ky+0hOTayt/DyU5ien8LkvGSv3Bmnue8m76X1HeyvbWVfdQt7q1vZX9t67HKgqIgwxmcn9t5PNi+ZyXnJjM1K1PFc8Vvv7K7m/ue3kBYfxfJ7Chmly2d85vuvlfDCplI++pcrSInTLQ/Pldaa9iOtXT28v6eWP+2rZWtpI3urW44t6B8fFX7sxutDU2JJjIkgLiqcuKgIYqPC8Vjbt6CFB1ePh8b2bo62uahr7eJoq4uKpg4a27v/Znu5KbGcNySBsVkJjMvuXULxvCEJKl0JONtKG7nn2U1YYOniWUzL12pcg62z203hw+v4zPgh/HL+dKfjBDQVsR9r6+qhuLyJkvImyho6KO+7+XpFYwdtXT3HTpI6mbiocNITokiLjyY9Poqc5JhjKznlp8UxPD1Oi0tIUDlY18YXl26krsXFrxfM4DPjhjgdKai9UlTKd367nee/PFsLrQyQijiAfbrMY0e3m3DTu5BFVETYsT9FQk1NSyeLl25ib3ULv5w/XTemHyTWWq7/nz/T7faw9huX6vjwAJ2uiPWb3M9FRYSREhdFTnIsQ5JiSI2PIj46QiUsIWtIYgwv3TeH6cNS+KcXPua1reVORwpKmw83sKOimUUXjlAJDzL9NheRgJMYE8mzdxdSODKNb7y0ld9uLnM6UtBZ9pdDJMVEcPP0XKejBD0VsYgEpPjoCJYtLuSi0Rl857fbeOGjI05HChqVTR2sKaniC7PytQytD6iIRSRgxUaF8/SiAi4bm8mDrxbzosrYK36z4Qgea1k4Z4TTUUKCilhEAlpMZDhPLJzJ5eMyefD3xfx+i6apB6Kz280LHx3hivFZDEvXLV59QUUsIgEvOiKcxxfM5IJR6Xzr5W28sb3S6UgB643tlRxtc7H4whFORwkZKmIRCQoxkb3T1DOGpfL1F7fw1s5qpyMFHGstz/7lEOcNSeCi89KdjhMyVMQiEjTioiJYdvcsJg5N4h9/8zF/2lfrdKSAsvlwA8XlTbpkycdUxCISVBJjIll+z2xGZcZz34rNbDnS4HSkgPHr9/aTGhfJ52fokiVfUhGLSNBJjotk+T2FZCZGs3hZ7ypccno7Kpp4Z3cN91w0Upcs+ZiKWESC0pCkGP733tlER4Sx8JmNlNa3Ox3Jr/363f0kRkfwRZ2k5XMqYhEJWvlpcay4dzad3R4WPrOR2pYupyP5pf21rawqqWThBcNJjtWNYnxNRSwiQW1cdiLL7p5FdXMXi5d9REtn95n/Uoh57L39REeEcc/FI52OEpJUxCIS9GYMS+WxBTPYU9XCkuWb6epxOx3Jb5Q1tPOHLeXMnzWMjIRop+OEJBWxiISEy8cN4We3TWH9gaN886WtuE9zr+9Q8uQHBzAG7rtslNNRQpZOjRORkHHz9DyOtrr40Ru7SI/fwQ9umhjS18vWtHTy4qZSPj8jj5zkWKfjhCwVsYiElC9dMora1i6eeP8AGQnRfP3KMU5Hcsyv3vkEt8fyD5eNdjpKSFMRi0jIeWDueOpaXDy6bi/pCVEsmDPc6Ug+d6C2ld9sPML8WfmMyIh3Ok5IUxGLSMgxxvDTz0+mod3Fv71WQnp8FNdOznE6lk89smYP0RFhfOPKsU5HCXk6WUtEQlJkeBi/unNG300itvKX/XVOR/KZokP1rNlRxX2XjSYzUWdKO01FLCIhKzYqnGcWFTA8PY4lyzdTUt7kdKRBZ63l4VW7GJIYzZcu0XXD/kBFLCIhLSUuiuX3FpIUE8GipR9xoLbV6UiDalVxFVuONPKtq8dqTWk/oSIWkZCXkxzLii/NxgILn/mIisYOpyMNClePh0fW7mZcViK3zsx3Oo70URGLiACjMxNYfk8hzR3dLHhmI0dbg29d6mf/cpDDR9t5YN54wsNC9/ppf6MiFhHpMyk3macXFVDe0MGiIFuX+lBdG794ay9XjB/C5WMznY4jx1ERi4gcZ/aodB5bMIPdlS3c+2wR7a4epyMNmMdjeeDV7USGhfHwzZNDejUxf6QiFhE5wWfHZ/HoF6ZRdLiee57dRIcrsG8S8cKmI2w4UM+/XHc+2ckxTseRE6iIRURO4oapQ/n57VPZeLCeLy8vorM7MMu4orGDn6zazUXnpTN/lk7Q8kcqYhGRU7h5eh6PfH4KH+6v474VgXf7RGst3/t9MW6P5ae3TNGUtJ9SEYuInMZtBfn85ObJvL+3ln9YsTmg9ox/93E57+6p5TvXjCM/Lc7pOHIKKmIRkTOYXziMH988mff21rJ42Ue0dvn/CVwl5U386x+KKRyZxqILRzgdR05DRSwi0g93zh7Gf31hGpsONXDXUxtoaHM5HemUjrZ2cd+KzaTGRfGrO2fommE/N6AiNsY8ZIwpN8Zs7XvM81YwERF/c9O0XJ5YMJNdVS3c/sR6qps7nY70d7rdHu5/fkvvPZcXztRNHQKAN/aIH7XWTut7rPLC+4mI+K0rJ2Tx7N2zqGjs4POP/YU9VS1OR/obP161i/UHjvLTWyYzJS/F6TjSD5qaFhE5SxeOzuCFJXPo6vFwy68/ZN3OaqcjAfDSpiMs+/AQ9148kltm5DkdR/rJG0V8vzFmuzFmqTEm9VQvMsYsMcYUGWOKamtrvbBZERHnTMlLYeX9FzEqM4Evryji8ff3Y611LM/zG4/wwKvFXDImgwevHe9YDpXqy0cAAAbvSURBVDl75kzfOMaYdUD2Sb70PWADUAdY4IdAjrX2njNttKCgwBYVFZ19WhERP9PhcvPt327jje2V3Dw9lx99bhLx0b69veDTfzrAj97YxWfHD+HXd80gJjLcp9uXMzPGbLbWFpzsa2f8brHWXtnPjTwFvH6W2UREAlpsVDj//47pjMtK5NF1e9l8uIGf3z6VWSPSBn3b1lr+++1PeHTdXq6bnMOjX5hGVISOOAaagZ41nXPc05uBkoHFEREJPMYYvnbFGF5acgEWy+1PrOcnq3cN6kpcnd1uvr9yB4+u28utM/P45XyVcKAa6PzJI8aYafROTR8C7htwIhGRAFU4Mo3VX7+Uh9/YxRPvH+C93bU8OG88l43N9OrykpsP1/OdV7ZzoK6NL108kn+Zdz5hulY4YJ3xGPFg0DFiEQl27+yu5t9f20FZQweFI9L49jXjKBw5sOnqzm43/7l2D898eJChybH8x+encPGYDC8llsF0umPEKmIRkUHi6vHw0qYj/M87n1DT0sUlYzKYP2sYl4/LPKsTusobO3h5UykvF5VS2dTJXbOH8eC880nw8Ulhcu5UxCIiDupwuVm+/hBP/ekgda1dxESGcdnYTOZOymZsViI5ybGkxkVijMFaS32biyP17eyvbeOP2yr4YF/vJZ8Xn5fBVy4fzYWjtRccaFTEIiJ+wO2xbDpUz+riSlaXVFHT0nXsa1ERYWQmRNPQ7qLd9deTvHKSY7itIJ/bZubpDkoBTEUsIuJnPB7LzspmyhraqWzqpKqpk+rmTlLjo8hPjWNYWhz5aXGcNyRBN20IAgO6jlhERLwvLMwwKTeZSbnJTkcRh+miMxEREQepiEVERBykIhYREXGQilhERMRBKmIREREHqYhFREQcpCIWERFxkIpYRETEQY6srGWMqQUOe/EtM4A6L75fKNIYeofGceA0hgOnMRw4b4/hcGtt5sm+4EgRe5sxpuhUS4dJ/2gMvUPjOHAaw4HTGA6cL8dQU9MiIiIOUhGLiIg4KFiK+EmnAwQBjaF3aBwHTmM4cBrDgfPZGAbFMWIREZFAFSx7xCIiIgFJRSwiIuKggCpiY8xcY8weY8wnxpgHTvL1aGPMS31f32iMGeH7lP6tH2P4z8aYncaY7caYt40xw53I6c/ONIbHve5WY4w1xugykhP0ZwyNMbf3fS/uMMY87+uMgaAfP8/DjDHvGmO29P1Mz3Mip78yxiw1xtQYY0pO8XVjjPnvvvHdboyZMShBrLUB8QDCgf3AKCAK2AZMOOE1XwUe7/t4PvCS07n96dHPMfwMENf38Vc0hmc/hn2vSwQ+ADYABU7n9qdHP78PxwBbgNS+50Oczu1vj36O45PAV/o+ngAccjq3Pz2AS4EZQMkpvj4PWA0YYA6wcTByBNIecSHwibX2gLXWBbwI3HTCa24Cnuv7+LfAFcYY48OM/u6MY2itfdda2973dAOQ5+OM/q4/34cAPwQeATp9GS5A9GcMvwz8ylrbAGCtrfFxxkDQn3G0QFLfx8lAhQ/z+T1r7QdA/WlechOw3PbaAKQYY3K8nSOQijgXKD3ueVnf5076GmttD9AEpPskXWDozxge7156/zUof3XGMTTGTAfyrbWv+zJYAOnP9+FYYKwx5kNjzAZjzFyfpQsc/RnHh4AFxpgyYBXwT76JFjTO9nfmOYnw9hsOopPt2Z547VV/XhPK+j0+xpgFQAFw2aAmCjynHUNjTBjwKLDYV4ECUH++DyPonZ6+nN5ZmT8ZYyZZaxsHOVsg6c843gE8a639uTHmAmBF3zh6Bj9eUPBJpwTSHnEZkH/c8zz+fprl2GuMMRH0TsWcbtoh1PRnDDHGXAl8D7jRWtvlo2yB4kxjmAhMAt4zxhyi97jSSp2w9Tf6+7P8mrW221p7ENhDbzHLX/VnHO8FXgaw1q4HYui9mYH0T79+Zw5UIBXxJmCMMWakMSaK3pOxVp7wmpXAor6PbwXesX1H3AXoxxj2Tas+QW8J67jc3zvtGFprm6y1GdbaEdbaEfQeZ7/RWlvkTFy/1J+f5T/Qe+IgxpgMeqeqD/g0pf/rzzgeAa4AMMacT28R1/o0ZWBbCXyx7+zpOUCTtbbS2xsJmKlpa22PMeZ+YC29ZwsutdbuMMb8ACiy1q4EnqF36uUTeveE5zuX2P/0cwx/BiQAr/Sd53bEWnujY6H9TD/HUE6jn2O4FrjaGLMTcAPfsdYedS61/+nnOH4LeMoY8016p1QXa+fkr4wxL9B7+COj7zj694FIAGvt4/QeV58HfAK0A3cPSg79PxEREXFOIE1Ni4iIBB0VsYiIiINUxCIiIg5SEYuIiDhIRSwiIuIgFbGIiIiDVMQiIiIO+j9LHaEyoil/nwAAAABJRU5ErkJggg==\n", "text/plain": [ "