{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Introduction to OpenGym using the CartPole problem\n", "\n", "OpenGym is an standard testbed for RL algorithms. \n", "\n", "https://gym.openai.com/envs/#classic_control\n", "\n", "In this notebook we test our own implementation of Q-learning (see slides of Lecture 2) with OpenGym in the CartPole environment (see slides of Lecture 1)." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[2020-04-15 15:51:32,754] Making new env: CartPole-v1\n" ] }, { "data": { "text/plain": [ "array([-0.03223939, -0.03398482, 0.0318835 , -0.01053955])" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import numpy as np\n", "import math\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "from collections import deque\n", "import pandas as pd\n", "\n", "import gym\n", "env = gym.make('CartPole-v1')\n", "env.reset()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Description of the environment\n", "\n", "OpenGym has not very good documentation. Usually you have to go to code to understand the details of the environment you want to use.\n", "\n", "In the case of the CartPole, go to:\n", "\n", "https://gym.openai.com/envs/CartPole-v1/\n", "\n", "Fortunately, there are some functions implemented that show the minimum necessary information to run the RL algorithms." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Discrete(2)\n" ] } ], "source": [ "# Two discrete actions in the environment (push towards each direction)\n", "print(env.action_space)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Means 2 discrete possible actions. In this case:\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", " \n", "
NumAction
0Push cart to the left
1Push cart to the right
\n", " \n", "Not because I found this information in documentation of the environment but in the code of the environment (https://github.com/openai/gym/blob/master/gym/envs/classic_control/cartpole.py#L13)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Box(4,)\n" ] } ], "source": [ "# Four dimensional space.\n", "print(env.observation_space)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That means *4 variables \"Continuous with Bounds\"* (box). \n", "\n", "There also exist **Discrete** observations.\n", "\n", "They can be organized as:\n", "\n", "- **Dictionary**, fi. Dict({\"position\": Discrete(5), \"velocity\": Box(low=np.array([0,0]),high=np.array([1,5]))}),\n", "- **Tuples** , fi. Tuple([Discrete(5), Box(low=np.array([0,0]),high=np.array([1,5]))]),\n", "- **Multidiscrete**, fi. MultiDiscrete([ 2, 2, 100])," ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[4.80000000e+00 3.40282347e+38 4.18879020e-01 3.40282347e+38]\n", "[-4.80000000e+00 -3.40282347e+38 -4.18879020e-01 -3.40282347e+38]\n" ] } ], "source": [ "# Show the limits of each variable\n", "\n", "print(env.observation_space.high)\n", "print(env.observation_space.low)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Acting in the environment\n", "\n", "The *step* method of the environment returns information after execution of an action (passed as parameter) in the *current* state of the environment. In fact, *step* method returns four values:\n", "\n", "- **observation** (*object*): an environment-specific object representing your observation of the environment after the action has been executed. Examples of observations are: pixels from a game screen, joint angles and joint velocities of a robot, or the board state in a board game.\n", " \n", "- **reward** (*float*): immediate reward obtained after action execution. The scale depends on the environment, but the goal is always to increase your total reward.\n", " \n", "- **done** (*boolean*): If trial has ended or note (terminal state). When True, it’s time to reset the environment again. Most (but not all) tasks are divided up into well-defined episodes, and *done* being True indicates the episode has terminated. (For example, perhaps the pole tipped too far, or you lost your last life in a game.)\n", " \n", "- **info** (*dict*): diagnostic information useful for debugging. It can sometimes be useful for learning (for example, it might contain the raw probabilities behind the environment’s last state change). However, official evaluations of your agent are not allowed to use this info for learning.\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "s = [ 0.0032159 0.00427172 0.0029989 -0.01657057]\n", "a = 0\n", "r = 1.0\n", "s'= [ 0.00330133 -0.19089311 0.00266749 0.27705703]\n", "False\n" ] } ], "source": [ "obs1 = env.reset() # Starting state\n", "action = env.action_space.sample() # take a random action\n", "obs2, reward, done, info = env.step(action) # take the action and observe results\n", "print('s =',obs1) \n", "print('a =',action)\n", "print('r =',reward)\n", "print(\"s'=\",obs2)\n", "print(done)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That's all the information we need in model-free RL algorithms!\n", "\n", "Let's execute now a sequence of random actions. Method *env.render* will show us a visualization of the environment." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "# Generate one random trial\n", "obs1 = env.reset() \n", "done= False \n", "while not done:\n", " env.render()\n", " observation, reward, done, info = env.step(env.action_space.sample()) # take a random action" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Average reward 17.0\n", "Average time steps before ending episode 17.0\n" ] } ], "source": [ "# Let's compute the average return for 10 random trials and the number of timesteps\n", "t=0\n", "tsteps=0\n", "treward=0\n", "for nexps in range(10): # Let's do 10 trials\n", " done= False \n", " env.reset() \n", " while not done:\n", " env.render()\n", " observation, reward, done, info = env.step(env.action_space.sample()) # take a random action\n", " treward = treward + reward\n", " tsteps = tsteps + 1\n", "\n", "print('Average reward',treward/10)\n", "print('Average time steps before ending episode',tsteps/10)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Trying to apply Q-learning, we have to define the table that will store the Q-value function. But we have a problem here, because our **states are defined by continuous variables**.\n", "\n", "One solution could be to discretize values by bining them.\n", "\n", "## Discretization of the state\n", "\n", "First we have to declare how many \"discrete\" states I will allow per variable, and later define the discretization procedure " ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "discr_vector = (1,3,10,20,) # Resolution degrees: How many discrete states per variable. Play with this parameter!\n", "\n", "class Discretizer ():\n", " \"\"\" mins: vector with minimim values allowed for each variable\n", " maxs: vector with maximum values allowed for each variable\n", " \"\"\"\n", " def __init__(self, vector_discr, mins, maxs):\n", " self.mins=mins\n", " self.maxs=maxs\n", " \n", " def Discretize(self, obs):\n", " ratios = [(obs[i] + abs(self.mins[i])) / (self.maxs[i] - self.mins[i]) for i in range(len(obs))]\n", " new_obs = [int(round((discr_vector[i] - 1) * ratios[i])) for i in range(len(obs))]\n", " new_obs = [min(discr_vector[i] - 1, max(0, new_obs[i])) for i in range(len(obs))]\n", " return tuple(new_obs)\n", "\n", "# Create the discretizer with maxs and mins from the enviroment\n", "d = Discretizer(discr_vector, env.observation_space.low, env.observation_space.high)\n", "\n", "# It will not work because limits for two varaibles are almost infinite (in other case could work)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[-0.02580767 -0.00103157 0.00121954 0.0479983 ]\n" ] }, { "data": { "text/plain": [ "(0, 1, 5, 10)" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "obs1 = env.reset() \n", "print(obs1)\n", "d.Discretize(obs1)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "# Another approach. Try a lot of random actions and find maximum and minimum for each variable empirically\n", "# This approach also has some problems, because some states are found with very low probability.\n", "\n", "lO=np.zeros((100000,4))\n", "for nexp in range(10000): # Let's do 10 trials\n", " done= False \n", " env.reset() \n", " while not done:\n", " observation, reward, done, info = env.step(env.action_space.sample()) # take a random action\n", " treward = treward + reward\n", " tsteps = tsteps + 1\n", " lO[nexp]=np.array(observation) \n", " \n", "maxv=[np.max(lO[:,i]) for i in range(lO.shape[1])]\n", "minv=[np.min(lO[:,i]) for i in range(lO.shape[1])]\n", "\n", "# Now we have a better discretization based on common values\n", "d = Discretizer(discr_vector, minv, maxv)\n", "\n", "# or even better\n", "minv = [env.observation_space.low[0], minv[1], env.observation_space.low[2], minv[3]]\n", "maxv = [env.observation_space.high[0], maxv[1], env.observation_space.high[2], maxv[3]]\n", "d = Discretizer(discr_vector, minv, maxv)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Q-learning implementation\n", "\n", "#### Define epsilon-greedy procedure" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "def choose_action(state, epsilon):\n", " return env.action_space.sample() if (np.random.random() <= epsilon) else np.argmax(Q[state])\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Implementation of Q-learning" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Episode 100 Total Reward: 11.0 Average Reward: 10.63\n", "Episode 200 Total Reward: 11.0 Average Reward: 10.46\n", "Episode 300 Total Reward: 12.0 Average Reward: 11.37\n", "Episode 400 Total Reward: 13.0 Average Reward: 11.73\n", "Episode 500 Total Reward: 15.0 Average Reward: 11.51\n", "Episode 600 Total Reward: 13.0 Average Reward: 13.21\n", "Episode 700 Total Reward: 11.0 Average Reward: 12.59\n", "Episode 800 Total Reward: 13.0 Average Reward: 13.04\n", "Episode 900 Total Reward: 16.0 Average Reward: 13.06\n", "Episode 1000 Total Reward: 15.0 Average Reward: 13.43\n", "Episode 1100 Total Reward: 13.0 Average Reward: 13.15\n", "Episode 1200 Total Reward: 16.0 Average Reward: 13.16\n", "Episode 1300 Total Reward: 13.0 Average Reward: 12.75\n", "Episode 1400 Total Reward: 12.0 Average Reward: 13.13\n", "Episode 1500 Total Reward: 17.0 Average Reward: 13.18\n", "Episode 1600 Total Reward: 15.0 Average Reward: 16.42\n", "Episode 1700 Total Reward: 43.0 Average Reward: 17.49\n", "Episode 1800 Total Reward: 14.0 Average Reward: 46.85\n", "Ran 1889 episodes. Solved after 1789 trials ✔\n" ] } ], "source": [ "# Set parameters for learning\n", "alpha = 0.2\n", "epsilon = 0.1\n", "gamma = 1\n", "\n", "# Create and initialize Q-value table to 0\n", "Q = np.zeros(discr_vector + (env.action_space.n,))\n", "\n", "# Just to store the long-term-reward of the last 100 experiments \n", "scores = deque(maxlen=100)\n", "lrews = []\n", "lr = []\n", "\n", "for episode in range(1,10001):\n", " done = False\n", " R, reward = 0,0\n", " state = d.Discretize(env.reset())\n", " while done != True:\n", " action = choose_action(state, epsilon) \n", " obs, reward, done, info = env.step(action) \n", " new_state = d.Discretize(obs)\n", " Q[state][action] += alpha * (reward + gamma * np.max(Q[new_state]) - Q[state][action]) #3\n", " R = gamma * R + reward\n", " state = new_state \n", " lr.append(R)\n", " scores.append(R)\n", " mean_score = np.mean(scores)\n", " lrews.append(np.mean(scores))\n", " if mean_score >= 250 and episode >= 100:\n", " print('Ran {} episodes. Solved after {} trials ✔'.format(episode, episode - 100)) \n", " break\n", " if episode % 100 == 0:\n", " print('Episode {} Total Reward: {} Average Reward: {}'.format(episode,R,np.mean(scores)))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is how long term reward increses with episodes. Remember that long-term-reward represents the number of time steps before failure. We decided the task was learnt when the average long-term-reward of the last 100 experiences is higher than 200. " ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAHCBJREFUeJzt3XuQXGd55/Hv09e56maNZFkWkbEFrDCL7SisF4csLMG32sTABmLvbvCyVMxWmVqoJLUxYWtD1Rab2wIpkiyJWbyYFIF4i5urMAnGS8JSBTaSkW8Y2zK2sZAsybqNpnvmdJ/Tz/5x3h63xUia0Uyf0338+1R1dc87p7ufPj3z67ffPv2+5u6IiEhxlfIuQERE+ktBLyJScAp6EZGCU9CLiBScgl5EpOAU9CIiBaegFxEpOAW9iEjBKehFRAqukncBAOvXr/etW7fmXYaIyFDZtWvX8+4+dabtBiLot27dys6dO/MuQ0RkqJjZM4vZTkM3IiIFp6AXESk4Bb2ISMEp6EVECk5BLyJScAp6EZGCU9CLiBScgl5EJCd/+s3H+fbjh/p+P2cMejPbYmbfMrNHzewRM3t/aP+wmf3UzHaH07U91/mgme0xs8fM7Kp+PgARkWHk7nzinie496nDfb+vxXwzNgZ+293vN7NJYJeZ3R1+93F3/x+9G5vZduB64NXAecA3zewV7p6sZOEiIsNsrt2h4zBe7/8EBWfs0bv7fne/P1w+ATwKbD7NVa4DvuDukbs/BewBXrcSxYqIFMVMFAMwMQhB38vMtgKXAveGpveZ2YNmdpuZrQ1tm4Fne662lwVeGMzsJjPbaWY7Dx3q/xiViMggaYSgH68NUNCb2QTwReAD7j4NfBK4ELgE2A98tLvpAlf3n2lwv9Xdd7j7jqmpM06+JiJSKN0e/UAM3QCYWZU05D/n7l8CcPcD7p64ewf4FC8Mz+wFtvRc/Xxg38qVLCIy/BqDNHRjZgZ8GnjU3T/W076pZ7O3AQ+Hy3cC15tZ3cwuALYB961cySIiw6/R6vboy32/r8W8lFwB/AbwkJntDm2/B9xgZpeQDss8DbwXwN0fMbM7gB+SHrFzs464ERF5sZkojcUsevRnvAd3/w4Lj7vfdZrrfAT4yDLqEhEptMagjdGLiMjKUtCLiBTcwB5HLyIiK2NmLma0WqZcWmhkfGUp6EVEctBoxZkM24CCXkQkFzNRwkQGh1aCgl5EJBeNSD16EZFCm1HQi4gUWyOKMzniBhT0IiK50NCNiEjB6cNYEZGCa0RxJnPRg4JeRCRzSceZbScauhERKaruFMX6MFZEpKCynNAMFPQiIpmbX11qREEvIlJIJ+a6Qzc66kZEpJAaYXUpHXUjIlJQMxqjFxEptkaGi46Agl5EJHPdwyvVoxcRKagslxEEBb2ISOYaUUzJYKSaTQQr6EVEMtaI0ukPzPq/Xiwo6EVEMjeT4Vz0oKAXEclclnPRg4JeRCRzWS4jCAp6EZHMNaKYSQW9iEhxpT36bOa5AQW9iEjmukfdZEVBLyKSMR11IyJSYO4+eEfdmNkWM/uWmT1qZo+Y2ftD+zozu9vMngjna0O7mdknzGyPmT1oZpf1+0GIiAyLKO4Qd3zgevQx8Nvu/k+Ay4GbzWw7cAtwj7tvA+4JPwNcA2wLp5uAT6541SIiQyrreW5gEUHv7vvd/f5w+QTwKLAZuA64PWx2O/DWcPk64LOe+h6wxsw2rXjlIiJDKOspimGJY/RmthW4FLgX2Oju+yF9MQA2hM02A8/2XG1vaBMRecnrLiM4UGP0XWY2AXwR+IC7T59u0wXafIHbu8nMdprZzkOHDi22DBGRodbt0U9mtDA4LDLozaxKGvKfc/cvheYD3SGZcH4wtO8FtvRc/Xxg38m36e63uvsOd98xNTV1tvWLiAyVrJcRhMUddWPAp4FH3f1jPb+6E7gxXL4R+GpP+7vC0TeXA8e7QzwiIi91L3wYm903YxfzknIF8BvAQ2a2O7T9HvCHwB1m9h7gJ8A7wu/uAq4F9gBN4N0rWrGIyBDrjtFPjlQzu88zBr27f4eFx90B3rzA9g7cvMy6REQK6ennGwCsGcsu6PXNWBGRDD124AQlg3pFk5qJiBTSbCthx9Z1md6ngl5EJEMzUczq0eyGbUBBLyKSqaxnrgQFvYhIphoZLzoCCnoRkUxlvegIKOhFRDLTiju0kg4TNQW9iEghNXKY/gAU9CIimcljLnpQ0IuIZKYb9GP6MFZEpJjue+oIAK/cOJnp/SroRUQy8szhJuO1MtsU9CIixXRirs2qjL8VCwp6EZHMTM+1WZXh9MRdCnoRkYycmIszXUKwS0EvIpKRaQ3diIgU2/SsevQiIoV2rNli7Vgt8/tV0IuIZCDpONNz2c9FDwp6EZFMTM+2gWzXiu1S0IuIZOBYCHr16EVECmpmLp8JzUBBLyKSiWYrTGiW8Vz0oKAXEclEs50A2c9cCQp6EZFMNKMQ9DUFvYhIIXWHbsY1dCMiUkyzYehmVD16EZFiarbSoFePXkSkoJpRjBmMVLOPXQW9iEgGmq2E0WoZM8v8vhX0IiIZaLaTXI6hBwW9iEgmmlGcy6GVsIigN7PbzOygmT3c0/ZhM/upme0Op2t7fvdBM9tjZo+Z2VX9KlxEZJg0W8ngBj3wGeDqBdo/7u6XhNNdAGa2HbgeeHW4zv80s3wemYjIAGm2EsZzmOcGFhH07v5t4Mgib+864AvuHrn7U8Ae4HXLqE9EpBAarQEeujmN95nZg2FoZ21o2ww827PN3tD2M8zsJjPbaWY7Dx06tIwyREQG3+yAD90s5JPAhcAlwH7go6F9oeOGfKEbcPdb3X2Hu++Ympo6yzJERIZDoxXn8mUpOMugd/cD7p64ewf4FC8Mz+wFtvRsej6wb3kliogMv2aU5DJzJZxl0JvZpp4f3wZ0j8i5E7jezOpmdgGwDbhveSWKiAy/dIw+nx79Ge/VzD4PvBFYb2Z7gd8H3mhml5AOyzwNvBfA3R8xszuAHwIxcLO7J/0pXURkOCQdZ67dyW2M/oxB7+43LND86dNs/xHgI8spSkSkSLozVw7VGL2IiCxeMwrLCA7TGL2IiCxeo5Xf6lKgoBcR6btGlN/C4KCgFxHpO43Ri4gUXENj9CIixdbUGL2ISLF1e/QauhERKSj16EVECq4b9AM7H72IiCxPsxVjBvVKPpGroBcR6bNGlDBeq2C20Ezu/aegFxHps2aOq0uBgl5EpO/yXC8WFPQiIn3XbMWMVtWjFxEprEaUMJ7Tt2JBQS8i0nfNHFeXAgW9iEjfNVrq0YuIFNpsK2G0qh69iEhhNXR4pYhIsTWiWIdXiogUVRQntBNnQmP0IiLF1IzyndAMFPQiIn01052LXkEvIlJMjVa+i46Agl5EpK/mV5fSGL2ISDHNhDH6CQ3diIgUUzP06DUFgohIQXU/jFWPXkSkoDRGLyJScI2cFwYHBb2ISF81ophyyXJbGBwWEfRmdpuZHTSzh3va1pnZ3Wb2RDhfG9rNzD5hZnvM7EEzu6yfxYuIDLpGFDNeK+e2MDgsrkf/GeDqk9puAe5x923APeFngGuAbeF0E/DJlSlTRGQ4HWm2WTdey7WGMwa9u38bOHJS83XA7eHy7cBbe9o/66nvAWvMbNNKFSsiMmyONKLBD/pT2Oju+wHC+YbQvhl4tme7vaFNROQlaSZKmByp5lrDSn86sNAglC+4odlNZrbTzHYeOnRohcsQERkMzSjO9dBKOPugP9AdkgnnB0P7XmBLz3bnA/sWugF3v9Xdd7j7jqmpqbMsQ0RksDWifBcGh7MP+juBG8PlG4Gv9rS/Kxx9czlwvDvEIyLyUtRoJYznuIwgwBlfZszs88AbgfVmthf4feAPgTvM7D3AT4B3hM3vAq4F9gBN4N19qFlEZGg0W/kuIwiLCHp3v+EUv3rzAts6cPNyixIRKYJW3KGdeO5Br2/Gioj0SWN+5srh/DBWRETOYBBWlwIFvYhI3zTDhGZjQ3p4pYiInMH8FMXq0YuIFFMjyn+KYlDQi4j0TXeMXh/GiogUVLP7Yax69CIixTQ/dKMevYhIMXV79GPq0YuIFNNM6NGPVdWjFxEppGYUM1YrUyrlt4wgKOhFRPqm0Upyn6IYFPQiIn2TzlyZ77ANKOhFRPpmEBYdAQW9iEjfNKKECfXoRUSKq9lSj15EpNAarURj9CIiRTY922Yi5y9LgYJeRKQv3J2jzRbrxut5l6KgFxHphxNRTDtxzhmv5V2Kgl5EpB+OzLQAWKegFxEppsONEPQTCnoRkUI6EoJeQzciIgV1fLYNwJpRBb2ISCHNLwyu4+hFRIqpMSDLCIKCXkSkLxpRTLlk1Cv5x2z+FYiIFFAjShirlTHLd9ERUNCLiPRFI4oHYvoDUNCLiPRFs5UMxPg8KOhFRPpiJooZr+V/xA0o6EVE+iJdRrAAPXoze9rMHjKz3Wa2M7StM7O7zeyJcL52ZUoVERkeM9FgLAwOK9Ojf5O7X+LuO8LPtwD3uPs24J7ws4hI4f2nz/+AX/+r7/IPjx1kerbNqtHBCPp+VHEd8MZw+XbgH4Df7cP9iIgMDHfnzgf2AXDvU0cAWD+R/1z0sPwevQPfMLNdZnZTaNvo7vsBwvmGZd6HiMjAO9ZM57a55ZpX8ZbtGwH4ha3r8ixp3nJ79Fe4+z4z2wDcbWY/WuwVwwvDTQAve9nLllmGiEi+nj3aBOD8taP85htezp6DM7zy3Mmcq0otK+jdfV84P2hmXwZeBxwws03uvt/MNgEHT3HdW4FbAXbs2OHLqUNEJC8f+MIP+NZjh3j9hecAcPF5qymXbGBCHpYxdGNm42Y22b0MXAk8DNwJ3Bg2uxH46nKLFBEZRO2kw1d27+P4bJuvP/wcAD93zljOVf2s5YzRbwS+Y2YPAPcBX3P3vwP+EHiLmT0BvCX8LCJSOEeb6eIiv3PlKwBYNVIZiLltTnbWQzfu/mPgtQu0HwbevJyiREQG3V98aw/feCTtxV+8eTW7/ssvE3cGcxR6MA7yFBEZMn/6zcdpJ847fv58rrhoPdXy4E40oKAXEVmiE3Nt2onzwWtexXv/xYV5l3NGg/sSJCIyoJ47PgfAuatHcq5kcRT0IiJL8MN90/z7//19ALZvWpVzNYujoRsRkUXqdJx/9+l7KRn88a/9U7ZtHJxj5U9HPXoRkUU6NtvmSKPFzW+6iHfu2JJ3OYumoBcRWaRH908DcN6a0ZwrWRoFvYjIIj1zOJ3P5jWbV+dcydIo6EVEFunQiQgYnOmHF0tBLyKySAdPzLFuvEatMlzROVzViojk6ImDM2xZO1zj86CgFxFZlGePNLnvqSNsP2+4xudBx9GLiJzWM4cbfPzux/nK7nSZwDe+cirnipZOQS8ichp//n/38JXd+3jtljX82mWbuTIsEzhMFPQiIqfwg58c5asP7OOG172MP3j7a/Iu56wp6EVETuLufPfJw7z3r3exaqTCb77hgrxLWhYFvYhIj0YU88EvPcSdD+xjarLOV26+gs1D9k3YkynoRUSCmz93P3/3yHMkHeftl23mP1/1qqGZivh0FPQiIsBf/eOTfO2h/bz+wnN435su4vUXrc+7pBWjoBeRl7wv/2Avf/D1H3Hta87lz264jHJp8Bb4Xg4FvYgUUiOKOdJoEcUJc+0OUZyw5+AMx2fbNKKEKO7w67+whQf3HuO37niAf3bBOj72zksKF/KgoBeRIfW1B/fzqf/3Y6bn2jSjhNl2Qr1SopV0aMcdGq3kjLfxN/c+w0wUc/7aUf7XjTsYqZYzqDx7CnoRGWgzUczeo032HZtl37E5nj3aZO/RWb792CHWTdS4+LzVjNfL1Ctl2kmHWqVEtVyi4862DZNMjFSoV0rUKyVKZly8eTVrRqt844fP8d/v+hEXbpjgo+94LZMj1bwfat8o6GXR3J2jzTaNKKbRijnWbNNsxYxUymAQJ0476dBOnHq1xDnjNVaNVJmJYqbn2mxZO8bmNaOUzvKtcafjtDsdWnGHmSgGIGp3KJkRdzrEnfT+48Qpl4yRaol6pUy9UqLjMNtOcHdm2wmtuMNYrUK3lJFqmYl6hY47x2fbtBMn7nSI4g5Ru8NcnBC10yGAVtyhXi0RxR0OnYgwA3eI4g5lS+93pFqeP69XSsxECWO1MpMjFcZqZebaHQzAoF4psW68znitzPRcTBQnzLYSZqKYRpTQ8fTxJB2n4x5uo0KcOJWyUTLjaLNFI4qZqFdYM1ZjzViVNWNVRqtlapUStXIJd4g7TrVsmC38HERxwsxczFzcYa6d9JzSes9fO0a1YrTiDoaF/dLBDEaqJUZrFSbqFcxgrFqmUn5hOq3js20OTs/RCs9R3OmeOzNRTNJxnjs+x96jsxw4McfeI01+cqTJ0Wb7RTVWy8bmNaNcvHk1/+2tF3PRhomz+nu6+uJNXH3xprO67rBR0A+hOIRp4uk/fsmMWrl02n/gM2nFHQ43Ig7PtHh+JuLgdMS+47PsOzbL/uNz/PTYLPuPzTHbPvPb4TOZqFcYr6cBNFJJw6BcSsNyJooZrZaZa6dBN9fuhGDo0PFl3/VLWvcFCaBkUCmXqJSMSskYq1WYixOOnRSqyzVSTV9kWkmHuXZnUdcZr5XZuGqEzWtHufriTWxZN5p2EtaOct7qUaYm64UcR++noQ76/cdn+fjdj/OL26bYvmmS9RN1jjRaHG22qZSMRiumXilRKZXm/zCSjtPNi1p4e9eVdNLwTMPHKJeMsqXnpfAPUQo/e7heyYyOp72S2VbCXJwwPRtzpBFxpNHmaLNFFHeolowTUZyGV9ju+Gybduy0krSXCDA91yYOPeJauUS9WqJcKnFiLvSko2S+N7uQeqXEeL0y/3iqZQu9y9DDrJQZrZWpltPbPNxo8fyJiOm5n71NM5iaqHPemlFede4k//KVG9i0ZpTxWpnJkbTHOF6vMNdOcE97Wt3wiOIOh2ciZkIvc7xe4enDDQ5MR/OPpRWn//yz7YSSgZnx8nqFqJ2kPeyRCiOVMtWKUS2VqJSNanhBm6hXMUufQye973LJqJTS3ycdT3vjcfohHMBYrYxhjNbK1Molmq1k/vlvRHGow5gcqVCvlCmXSN8RdHvolXQf1iol5toJtXKZDavqL9r3HWe+FzzbfuFDwPFahdl2t5eevpg53XcCCYdnWjRbMatGq9TDczRRLzNRr1IySMILesmMmSimGcWUSkY7Sf9u1o7VGK9XaETpO62jzRbHmq35F8pW3MHdqVVKxB2f70nHSfpOqBHFjFTLTE3WWT1a7Xk3ktYyUimRuPPskSZJh/lORfddk7szF3dohr/xdJ8mzETpu6NapcS68Rqb14xSq4QXmJNeaMolY+OqOuvGa2fdYZGFDXXQP/DsMe7YuZc7du7Nu5RFqVdKTI5UGAlvp9eMVqlVSqyqVamV0z/sV9QnqIYeUCsEVdxxzl8zykS9wli9zJrR2vw/ixl03GknIdhCmKShAO2Ov+jtd7en3Io7TI5UeNW5k6y/aD3rJ+qcM1Fj/USd9RM1NkyOsHHVyIousHBFgY5LPpWywXh4YSukC/MuQM7GUP81XvXqc/mV157HM4cbvP3SzTRaCVMTdTasqhMnzni9Mv+2v9uT74YjpMMVZka371AJvZSonRB3PO3hh1PHnaQDSSe9rW6Pozt0Uilb6PGVWT1aZe14lXPG66wZq1KvlOZ7NSIiWRvqoDcz/uyGS/MuY1FqFb0VFZF8qIspIlJwCnoRkYJT0IuIFFzfgt7Mrjazx8xsj5nd0q/7ERGR0+tL0JtZGfgL4BpgO3CDmW3vx32JiMjp9atH/zpgj7v/2N1bwBeA6/p0XyIichr9CvrNwLM9P+8NbfPM7CYz22lmOw8dOtSnMkREpF9Bv9BB4y+aqcTdb3X3He6+Y2pqqk9liIhIv74wtRfY0vPz+cC+U228a9eu583smbO8r/XA82d53awMeo2qb/kGvUbVt3yDWOPPLWYjc1/5KQHNrAI8DrwZ+CnwfeDfuPsjfbivne6+Y6VvdyUNeo2qb/kGvUbVt3zDUOOp9KVH7+6xmb0P+HugDNzWj5AXEZEz69tcN+5+F3BXv25fREQWpwjfjL017wIWYdBrVH3LN+g1qr7lG4YaF9SXMXoRERkcRejRi4jIaQx10A/CfDpmtsXMvmVmj5rZI2b2/tD+YTP7qZntDqdre67zwVDzY2Z2VQY1Pm1mD4U6doa2dWZ2t5k9Ec7XhnYzs0+E+h40s8syqO+VPftpt5lNm9kH8tyHZnabmR00s4d72pa8z8zsxrD9E2Z2Y5/r+xMz+1Go4ctmtia0bzWz2Z79+Jc91/n58LexJzyGFVs44RQ1Lvk57df/+Snq+9ue2p42s92hPZd9uGLcfShPpEfzPAm8HKgBDwDbc6hjE3BZuDxJeljpduDDwO8ssP32UGsduCA8hnKfa3waWH9S2x8Dt4TLtwB/FC5fC3yd9EtvlwP35vC8Pkd6fHBu+xD4JeAy4OGz3WfAOuDH4XxtuLy2j/VdCVTC5T/qqW9r73Yn3c59wD8PtX8duKbP+3BJz2k//88Xqu+k338U+K957sOVOg1zj34g5tNx9/3ufn+4fAJ4lJOmezjJdcAX3D1y96eAPaSPJWvXAbeHy7cDb+1p/6ynvgesMbNNGdb1ZuBJdz/dF+j6vg/d/dvAkQXudyn77Crgbnc/4u5HgbuBq/tVn7t/w927q7x/j/SLiqcUalzl7t/1NLE+2/OY+lLjaZzqOe3b//np6gu98ncCnz/dbfR7H66UYQ76M86nkzUz2wpcCtwbmt4X3kbf1n2bTz51O/ANM9tlZjeFto3uvh/SFytgQ4719bqeF/9zDco+hKXvszz35X8g7V12XWBmPzCzfzSzN4S2zaGmrOtbynOa1z58A3DA3Z/oaRukfbgkwxz0Z5xPJ0tmNgF8EfiAu08DnwQuBC4B9pO+DYR86r7C3S8jnTb6ZjP7pdNsm9t+NbMa8KvA/wlNg7QPT+dU9eRSp5l9CIiBz4Wm/cDL3P1S4LeAvzGzVTnVt9TnNK/n+gZe3OEYpH24ZMMc9EuaT6efzKxKGvKfc/cvAbj7AXdP3L0DfIoXhhYyr9vd94Xzg8CXQy0HukMy4fxgXvX1uAa4390PhHoHZh8GS91nmdcZPvD9V8C/DUMJhOGQw+HyLtIx71eE+nqHd7L4W1zqc5rHPqwAbwf+tqfugdmHZ2OYg/77wDYzuyD0BK8H7sy6iDCW92ngUXf/WE9777j224DuJ/t3AtebWd3MLgC2kX6Y06/6xs1ssnuZ9AO7h0Md3aNAbgS+2lPfu8KRJJcDx7vDFRl4US9qUPZhj6Xus78HrjSztWGI4srQ1hdmdjXwu8Cvunuzp33K0sWAMLOXk+6vH4caT5jZ5eHv+F09j6lfNS71Oc3j//yXgR+5+/yQzCDtw7OS96fByzmRHu3wOOmr64dyquEXSd+qPQjsDqdrgb8GHgrtdwKbeq7zoVDzY/T5E3rSoxUeCKdHuvsJOAe4B3ginK8L7Ua6OtiTof4dGe3HMeAwsLqnLbd9SPqCsx9ok/ba3nM2+4x0rHxPOL27z/XtIR3P7v4d/mXY9l+H5/4B4H7gV3puZwdp2D4J/DnhS5R9rHHJz2m//s8Xqi+0fwb4jydtm8s+XKmTvhkrIlJwwzx0IyIii6CgFxEpOAW9iEjBKehFRApOQS8iUnAKehGRglPQi4gUnIJeRKTg/j8lGhl3XfcAawAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(lrews)\n", "plt.show()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice that, being results shown are average of last 100 experiments, there is a \"delay\" of 100 episodes in detecting that desired behavior has been learnt. We could plot the actual reward for each episode, but is too varaible to see anything. It is better to average returns." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VPW9//HXhwQCAbIAAcIiICKLC4rR4loXXKBW26qttr1Say+/+rP9aXetrbV3+dUuV1vqvbZWbbGb1lYrrVhF1LqjAVkFSdgDgYSEhOzbfO8fcxJmsk6SWU6G95NHHnPmO2dmPjlD3vOd7/nOOeacQ0REktegRBcgIiKxpaAXEUlyCnoRkSSnoBcRSXIKehGRJKegFxFJcgp6EZEkp6AXEUlyCnoRkSSXmugCAMaMGeOmTp2a6DJERAaUNWvWHHLO5fS0ni+CfurUqeTn5ye6DBGRAcXMdkeynoZuRESSnIJeRCTJKehFRJKcgl5EJMkp6EVEklxEQW9mu8xso5mtM7N8r22Uma00swLvMttrNzNbamaFZrbBzObF8hcQEZHu9aZHf5Fz7jTnXJ53/Q5glXNuBrDKuw6wEJjh/SwBHoxWsSIi0nv9mUd/NXCht7wMeAX4ltf+mAueo/BtM8sys1znXHF/ChURGUje3H6IF98voaCkisKSauZOyiJvajbriyrZU17LzHEjGJ8xlEtmj2Pu5KyY1hJp0DvgBTNzwC+dcw8B41rD2zlXbGZjvXUnAntD7lvktYUFvZktIdjj57jjjuv7byAi4kOf/tXqsOvFlQf4x+YDbdfX760AYGzGUN8E/bnOuf1emK80s63drGudtHU4A7n3ZvEQQF5ens5QLiLHJOssMaMsojF659x+77IEeBo4CzhoZrkA3mWJt3oRMDnk7pOA/dEqWEQkmVinfePo6jHozWy4mY1sXQYuAzYBy4HF3mqLgWe85eXAjd7sm/lApcbnRUQSJ5Khm3HA0xb8fJEK/ME59w8zexf4k5ndDOwBrvPWXwEsAgqBWuCmqFctIpIk4jF002PQO+d2AHM7aS8DLumk3QG3RqU6EZEkF4ec1zdjRUQSyTc7Y0VEJDZ8sTNWRERiSD16ERHpLwW9iEgCaWesiEiSszjsjVXQi4gkkHr0IiJJTtMrRUSSnIJeRET6TUEvIpJA+sKUiEiS09CNiIj0m4JeRCSBNI9eRCTJaR69iEiS0xi9iIj0m4JeRCSBNL1SRCTJaehGRCTJaWesiEiSU49eRCTpaYxeRET6SUEvIpLkFPQiIgnknIv5cyjoRUQSKBD7nFfQi4gkUkA9ehGR5KagFxFJcgp6EZEkFwjE/jkU9CIiCeSrHr2ZpZjZe2b2d+/6NDNbbWYFZvaEmQ3x2tO864Xe7VNjU7qIyMAXh5zvVY/+NmBLyPUfAvc752YAh4GbvfabgcPOuROA+731RESkE77p0ZvZJOAjwMPedQMuBv7srbIM+Ji3fLV3He/2SyweJ0UUERmAWvwS9MBPgW8CrbsNRgMVzrlm73oRMNFbngjsBfBur/TWD2NmS8ws38zyS0tL+1i+iMjA5osvTJnZlUCJc25NaHMnq7oIbjva4NxDzrk851xeTk5ORMWKiCSbeBwCITWCdc4FrjKzRcBQIINgDz/LzFK9XvskYL+3fhEwGSgys1QgEyiPeuUiIkkgEIcufY89eufcnc65Sc65qcD1wEvOuc8ALwPXeqstBp7xlpd71/Fuf8nF4y1LRGQA8sXQTTe+BXzVzAoJjsE/4rU/Aoz22r8K3NG/EkVEklc8Zt1EMnTTxjn3CvCKt7wDOKuTdeqB66JQm4hI0vPN9EoREYkNvw/diIhIP6lHLyKS5Hwx60ZERGJnXMbQmD+Hgl5EJEGOzxnOtWdMivnzKOhFRBLkwhPHEo9DgSnoRUSSnIJeRCTJKehFRJKcgl5EJMkp6EVEkpyCXkQkySnoRUQSJF4nWVXQi4jE0PkzxnR5W7zO1KGgFxGJoYxhgxNdgoJeRCRRNHQjIpIE4pTl3VLQi4jEUDyOZdMTBb2ISAy1xnz6kJSE1aCgFxGJodYOffqQVM6ZPjohNSjoRURiKPEDNwp6EZG4MIvfLJv2FPQiIjEUujPWEtS/V9CLiMSJevQiIseYeOW+gl5EJE4SNadeQS8iEkMWctk+5q8/67i41KCgFxGJk9AO/VVzJ3DC2BFxeV4FvYhILFmni3HdMaugFxGJpTgdc747CnoRkRhqDhxN+vA59fHTY9Cb2VAze8fM1pvZZjP7vtc+zcxWm1mBmT1hZkO89jTveqF3+9TY/goiIv7VHAi0LYcP3cQv6iPp0TcAFzvn5gKnAVeY2Xzgh8D9zrkZwGHgZm/9m4HDzrkTgPu99UREjknNLaE9+sTU0GPQu6Bq7+pg78cBFwN/9tqXAR/zlq/2ruPdfon54YDMIiIJ0Dp0E0xBH8+jN7MUM1sHlAArge1AhXOu2VulCJjoLU8E9gJ4t1cCiTk2p4hIgjW1hAzddDEDJ9YiCnrnXItz7jRgEnAWMLuz1bzLzurvsN/ZzJaYWb6Z5ZeWlkZar4jIgBI2dBN6g1+nVzrnKoBXgPlAlpmlejdNAvZ7y0XAZADv9kygvJPHesg5l+ecy8vJyelb9SIiPtcSSPz8ykhm3eSYWZa3PAxYAGwBXgau9VZbDDzjLS/3ruPd/pJzLvG/qYhIAjQFuhq6iV+XPrXnVcgFlplZCsE3hj855/5uZu8Dj5vZfwDvAY946z8C/NbMCgn25K+PQd0iIgOKef8Socegd85tAE7vpH0HwfH69u31wHVRqU5EJIn4dnqliIh0beuBIzQ2B4dnNu2rJNBuTD504FpBLyIywBRX1nHFT1/je8s3s3pHGVf+/HUefWNnl+vrVIIiIgNMRW0TAGt3H2bv4ToA3t9/pOs7qEcvIjIwmXWd4S5knURR0IuIRFF3c8kTlfUKehGRPupsR2t3XxvSOWNFRAawtqDvbp24VNKRgl5EpI8cocex6SLG3QA4TLGIiHQvdEimuwO+qEcvIjKARTJ0c870MXGppT0FvYhIHBhwXd6khDy3gl5EpI86G6bRrBsRkSRkdB3iXcV+PDNfQS8iEqHfvb2bW363ptt1/HjyjUiORy8iIsB3/rqpy9vaOug+THr16EVE+sksdNZNeNK3vgEMTwvvV+dNyY5DZUHq0YuI9FHYIRC6mCWfmzmMhafkcuWpuWHtnzpzcixLC6OgFxHpp9Adq51Nuvnih6d3cp/47Y3V0I2ISC+VVjV0aEvkYYh7oqAXEemlrz25Hug4Hg/dHwIhURT0IiK9VNfYHHbdQkboOwv/RFPQi4j0kR9O/B0JBb2ISD+13xnb3WEQEkFBLyISFcG0bx/xfhjKUdCLiPRSS8Dx81UF1DQcHas/eirBBBXVDc2jFxHppbV7Kli7p4KTJmS0tYUO0fst7NWjFxHpo6r6YI8+fD+sz1IeBb2ISJ+1BLxQNws/nWCC6umKgl5EpI86m13jt2EbUNCLiPRZS0iqh4/R+yvtFfQiIn3UNnQTwl8RH9Rj0JvZZDN72cy2mNlmM7vNax9lZivNrMC7zPbazcyWmlmhmW0ws3mx/iVERBKhNeiDpxIMtjnnh5nz4SLp0TcDX3POzQbmA7ea2RzgDmCVc24GsMq7DrAQmOH9LAEejHrVIiI+ENqjP3rikXB+GMXpMeidc8XOubXechWwBZgIXA0s81ZbBnzMW74aeMwFvQ1kmVkuIiJJxh2ddNN2WLPgIRASWFQnejVGb2ZTgdOB1cA451wxBN8MgLHeahOBvSF3K/LaRESSSkv4KaZ8K+KgN7MRwF+A251zR7pbtZO2Du9vZrbEzPLNLL+0tDTSMkREfKOrnbF+G6WPKOjNbDDBkP+9c+4pr/lg65CMd1nitRcBoSdDnATsb/+YzrmHnHN5zrm8nJycvtYvIpIwbUM3wOZ9lQmtpTuRzLox4BFgi3PuvpCblgOLveXFwDMh7Td6s2/mA5WtQzwiIskkdOjmJy9sA7xZN/7q0Ed0ULNzgX8BNprZOq/t28C9wJ/M7GZgD3Cdd9sKYBFQCNQCN0W1YhERn2ibXunns44QQdA7516n690Ml3SyvgNu7WddIiISJfpmrIhIklPQi4j0U/shD7+N0SvoRUSSnIJeRCTKQufR+6Fzr6AXEUlyCnoRkX5qP7vSb0dGUNCLiPSThcR5+x2xGroREZGYU9CLiESZH3rxoRT0IiJJTkEvItJfHXbG+qtPr6AXEUlyCnoRkX7qcAiE0GUfdO4V9CIiSU5BLyISRY7wE4/44VD1CnoRkX4KDfMOX5jS0I2ISHJxwbOD+4qCXkQkipzfUh4FvYhIVAWc/8JeQS8i0k+hBzXzWcYDCnoRkX4L2xnbbtaNHyjoRUSiyG8hDwp6EZGoaj/p5huXz0xUKW0U9CIiURR6QLPvX3USM8ePTGA1QQp6EZEIdHdESmu3L1ZHrxQRGYCeeHdvROsFfHb4A4DURBcgIuJn7+8/woi0VA4cqe9ynfDplX6bRa+gFxHp1qKlrwFw+4IZEa0fGvI+6dBr6EZEJJoCTvPoRUSSTpdHr/TJIL2CXkQkivzWm4cIgt7MHjWzEjPbFNI2ysxWmlmBd5nttZuZLTWzQjPbYGbzYlm8iIjfBL8w5a+0j6RH/xvginZtdwCrnHMzgFXedYCFwAzvZwnwYHTKFBEZGELn0Ptj4CaCoHfOvQqUt2u+GljmLS8DPhbS/pgLehvIMrPcaBUrIuJ3yXTikXHOuWIA73Ks1z4RCP1WQZHXJiKStCxkp2vosI1P9sVGfWdsZ79Wp+9tZrbEzPLNLL+0tDTKZYiIJIZzvuvQ9znoD7YOyXiXJV57ETA5ZL1JwP7OHsA595BzLs85l5eTk9PHMkREEi+0hxv+hSl/dOn7GvTLgcXe8mLgmZD2G73ZN/OBytYhHhGRY4Hz4RemejwEgpn9EbgQGGNmRcD3gHuBP5nZzcAe4Dpv9RXAIqAQqAVuikHNIiK+5cPvS/Uc9M65G7q46ZJO1nXArf0tSkRkIGn/zdiBOI9eREQiFBiI8+hFRCTyQxs457/DICjoRUQiEGl2Hwvz6EVEklN3pxJst5rPOvQKehGRaAp9Pxjo8+hFRI4p3fXSrd0YjU4OLiIyAEW+MzasS+8LCnoRkQhEOjfeoVk3IiK+VVhSRXVDc9v1PWW1bcv//fL2Lu/XYWesF/QpPpl2o6AXkQEpf1c56/dWRO3xnHMsuO9VvrDs3ba2wtKq3j8OjuZAAICUQf4I+h4PgSAi4kfX/uItAHbd+5GoPF5dUwsA7+w8ep6lSGfNhHbcA+7ot2P9EvTq0YvIgFMTMrzSX80tAZxz/Pat3QAMG5zCH9/ZQ2NzoFc7U0cPHwLA+TPG0BzwV9CrRy8iA86O0pqoPE5dYwuz7/4Hl580juc3HwSgprGFO5/ayMEj9Zw2OSvix5qQNYyymkbSUgfR4rOgV49eRAacQVFKriP1TQBtIR+qvKaxw/z4roWv1xr0qQp6EZG+SY1S0nc3DdI5iDSn91XUtU2/dO5o0A9S0IuI9E20hkS6mxv/0tYS/uWRd7q9f2sZW4qPhLWrRy8i0k99yU/nHPe98AGFJdVtbc0tXQf9voq6Hh8zEHL31k8HoT36zubRXzV3AieOGxFh1dGhnbEiMuAE+vDN07KaRpa+VMiTa4p4687gCfJ++mJB1GpqC3pctztjl95wetSeM1Lq0YvIgBPowzEGWu/S1BJoa8vfXd7F2r3XEvLu06J59CIi/dNT0Dvn+Ot7+8JC/ehtkT9Ob3xwsKrt8f02j15BLyIDTqBjfod5dmMxtz+xjv8JOT5NZzteY3XwsYCCXkSkf3rqiR+uaQSgpKr+6H06eXOIRdA71KMXEem3HoduOmlr6eQ+sThBiHNHe/TRmu/fX/6oQkSkF1oinHYTdrCxTu7Tl9k7kTjao4/N4/eWT8oQEYlcXwK6szeHMSOHRKGacOHTK/0Rsf6oQkSkFyIdcgk9zHBzJ0G/YPa4qNUUqrsvTCWCgl56ZW95bcQfmyMRegafY8UbhYdYvn5/j+sFAu6Y3D4Aj7y+k837K3nwle3sPFRDeU0jFbWNbCk+wv0rt3GourFt3dYDk4VqPbrlb9/eTXlNI/et3MaC+/4JeF+cWlXA/oo66hpb2u6z/u7LolN86DdjU/wR9OaHs5Xn5eW5/Pz8RJcRV69uK+XUSZlkpR/96Fha1cAXlr3Lg589gwlZw2LyvIUlVTS1OGbnZnS5zrq9FYxKH8LkUcPajt63ZvdhAK558E0+f+407v7oHMqqG7h5WT4PfPp0JmWnh933uNHpbY9X19jC64WHuHROeO/psbd2cfczm/nyxSfwtctmsnbPYXJGpDF5VPC+zjnMrO2y1cEj9ew8VMOeslrWFVXwh9V7GJmWysbvXx52v1YVtY38YMVWzjlhNFfNndDhiISt6+8tr2XN7sNMGzOcuZ0cntY5x0tbSzhr2ihGpKWyYuMBLp0zjrqmFtbuOcxFM8eGrd/YHODFLQdJH5LC3ElZZA8fgnOOaXeuAMJPmFHf1MKf8vdSWFLN3VfOITVlEI+8vpN///v7PHfb+Z2+XqF/uz0dZbGwpJrv/HUjDy8+k8q6Jq5+4HUyhw3myS+ewyjvOOrttxvAy1tLyM0ayqzx4c9/qLqBrcVVnDdjDACb9lUydHAKJ4zt/Kv9VfVNrN5RzoI54yipqueDA1WcPyOHNwoP8fBrOzhvRg4TModS4B2e4L6V29ruOzItlaoejj+/5ILjqW9q4THvmPJ9sevejzD1jmf7fP9WI9JSaQ4EqG8K8PadlzA+c2i/H7MrZrbGOZfX43oDPeirG5ppbA60/Wfti/qmFo7UNzF2ZMcX5EBlPVnpgxk6OKXL+6/ZXc41DwbPdvPK1y/kul++xYTMoTzw6XkATMoexub9R5icnU5m+mCqG5o5+XvPM2v8SJ5YcjbpaSn87MUCHntrF0fqm7nlwuk8v/kAk7LT+fXnzqSwpJqn3ivi8+dOY3tpNcePGdHtfx7nHIUl1YwcOrjDeq3/kReePJ7axhb+ua0UgD/+63z2Hg72Hr/55w1t69983jQO1zTy1Hv7ut2G2emDOVx7tGf1/r9dTnV9M/f8bTMrNh4AYMX/O585EzLYvL+SO5/ayIaiyrb1p4xOZ7fXe513XBaHqhvZU9673uzErGFMGZ3Om9vLAPjDv36IM6eOYsZdz3VY9+zjR3Pnollc9cAbnT7WF86bxsOv7wTg4RvzWL2zjF+9tjOiOr6y4ESGp6XwH89uCWu/7oxJPLmmqMP6l80ZR0lVA+u6OS3eyRMzmDU+g037Ktl6oOPp7SZlD+Nn15/Gur2VBAKOs6eP5sn8vTjgyxfP4Mz/fLHTx71j4Sw+lTeZ7zyziWc3FAOQOWwwlXUde8mt5k7KZL332t12yQxOm5zFTb8Jnn7v9gUzWLqqgICDf7v6JFIHDeLbT2/s9HFe/vqFXPSTV7p8nnjrKuhPmZjJxn1H/6+eNXUU7+yK7Bu17961gJyRaVGrsb1jJuhn3LWCphbHo5/L45bfraWhOdD2R3rpnHHccuF0/uflQo7UNfPOrnJ+eM0pPPjKdr522Uy+/dRG/vGVC/jwj16mOeA4YewIfv+FD/HensOcMWUUX3liHa8XHgLgx9eeyrAhKczOzeDR13fS0Bzg1EmZLF1VEPYxMhLXnzmZx9/d26fftysLZo/lxS0lAGSlD6aiNvwP9RPzJvJmYRkHjtR3dneRY96uez/CT57/gF1lNewtryVnZBoLT85l7uRMFtz3KgC3XjSd//Ph6Zx6zwsRPeba717ar05oT46JoF+6qiDsI55ILA1JHRQ8vZzETFrqIBqaA7z6jYsora5v+6Q8LiONoYNT2j71dSdz2GDqmlpobA7wpYtO4IGXC/nFZ89ges5w/ry2iFMmZtLUEuArT6xvu89fbjmbM6aM6vIxS47UM2xICiOHDm5ra+39/98Lp/OZ+VM4996XWHz2FJaFDB8lddCb2RXAz4AU4GHn3L3drd/XoC84WMWl97/atyLjoP1wRlfMIvuG3hlTsvn6ZTO58dHVNHVzeNVIfXvRLOYfP5pVW0r42aoCZowdQUvAseNQcEfWj645lW/+JTiMc8fCWbxecKjtE87bd17CBwerWPxo8Hjddy6cxcZ9laQOMvZV1HG4tonCkmoWzB7LnAnBTz4Al580jjW7K/jEvIm8tb2s7SNx5rDBLD5natt6eVOy2V5a3bb9LpqZw8Wzx/Hdv25qq/+7V84hN3Mob20v4+LZY1m6qoCbz5vGa9sOMSt3JK8VHOKMKdn8+PkPAJidm8HS609jxriRPPBSAT95YRsXzxrLjWdP4XO/fpeRaancctF0Pjt/CsvX7eekCRnUNLTw6Bs7+cblM5mdm8Hl97/KBwermDo6nWvmTWJXWS1/WXt0OObEcSPYdvDoYXDzpmTz8XkT2VtexwubD3Bt3iQKDlaTlT6YhSfn8slfvhX2mvz6pjOprG3i1YJSzpk+hgmZQ3lyTRGzxo9kf0VdWIi0mjluJPd/6jSe21TM6p3lbSe3vv7MyUzMGsZ/rdzGkguO59uLZvO39ftZsbGYd3aWMzwttW14bOu/X8HBI/Ucqm7gcE0TP3+pgBvOOo4fPLeVj87N5dkNxW2vxfc+OgcDKuqamDZmOLc9vg6A0yZnsW5vBZ/Km0xFXSPPbz7IKRMzueeqkzhjSjZvbj/EhqJKVmws5vgxw3m9sIxL54xjkMENZx1HeU0jF5yYE/a7BQKOZzcWs+iU3LBvma7eUcbE7GFMzBpGc8BRVt3I/B+sAqDwPxeSGsEE9qffK+IrT6znxa9+uMt9C91xzvG3DcUsOnl82PO9+P5BGlsCLHtzF7//wociqqWvEhb0ZpYCbAMuBYqAd4EbnHPvd3Wf/gzd7K+oo6Ckmi3FR6hvauHSOeOYPT6DQYOMg0fqSUsdREVtE2U1DTy38QBXnDyeWbkZfOZXb3PSxEz+/8dPobCkisff2cucCRmkDDLW7D5MbWMLn8ybzFnTgu/y+yrqWLP7MGt3H+ajcydwpL6J5hbHxn2VfPniE9hbXkvqoEFUNzQzJNUYkRYcHw8EHDvLahhkxrQxwyk4WEVlXRMZwwbz1/f28bXLZpIyKLjDceehGqrqm3FAbUMz55wwhvKaRnaX1XDcqHRGjzg61negsp6RQ1MZnnb0SNPlNY00twTYeaiGyaPSmZA1jD1ltTS2tLC7rJbZuRk0NAcYNjilw313lFZzfM4InHO8ub2MvKnZpKWmsL+ijuz0IQwbkkJzS4CCkmqy04e0jf1X1DYScPTYa9lbXsv4zKEMbvef3rngG8v0nBFhdUDwj3zjvkomZQ9r+933ltdS3dBM+pAUpoweHtH/kbLqBlIGWdiO776qaWimpqGZsRlH933sKatlQtbQtj/oHaXVTBsznJ2Hatp+l+7sOlRDyiAjZ2Rat/uCAIoO1zJmRBolRxrIzRrK3vLaDs9RWtXAvoq6Hs936pxjS3EVmemDmRjBzv83Cg9xwtgRjMsI3+9T09BMdUNzh/Z4O1zTiBlReZ0HikQG/dnAPc65y73rdwI4537Q1X2OxVk3IiL9FWnQx+IzxUQgdE9jkdcmIiIJEIug72xCb4ePDWa2xMzyzSy/tLQ0BmWIiAjEJuiLgMkh1ycBHb4G6Jx7yDmX55zLy8nJaX+ziIhESSyC/l1ghplNM7MhwPXA8hg8j4iIRCDqJwd3zjWb2ZeA5wlOr3zUObc52s8jIiKRiXrQAzjnVgArYvHYIiLSOzp6pYhIklPQi4gkOV8c68bMSoG+Hl90DHAoiuXEgt9rVH395/caVV//+LW+Kc65Hqct+iLo+8PM8iP5Zlgi+b1G1dd/fq9R9fWP3+vriYZuRESSnIJeRCTJJUPQP5ToAiLg9xpVX//5vUbV1z9+r69bA36MXkREupcMPXoREenGgA56M7vCzD4ws0IzuyNBNUw2s5fNbIuZbTaz27z2e8xsn5mt834WhdznTq/mD8zs8jjUuMvMNnp15Htto8xspZkVeJfZXruZ2VKvvg1mNi8O9c0M2U7rzOyImd2eyG1oZo+aWYmZbQpp6/U2M7PF3voFZrY4xvX92My2ejU8bWZZXvtUM6sL2Y6/CLnPGd7/jULvd+js6LPRrLHXr2ms/s67qO+JkNp2mdk6rz0h2zBqnHMD8ofgcXS2A8cDQ4D1wJwE1JELzPOWRxI8u9Yc4B7g652sP8erNQ2Y5v0OKTGucRcwpl3bj4A7vOU7gB96y4uA5wgebno+sDoBr+sBYEoityFwATAP2NTXbQaMAnZ4l9necnYM67sMSPWWfxhS39TQ9do9zjvA2V7tzwELY7wNe/WaxvLvvLP62t3+X8DdidyG0foZyD36s4BC59wO51wj8DhwdbyLcM4VO+fWestVwBa6P9HK1cDjzrkG59xOoJDg7xJvVwPLvOVlwMdC2h9zQW8DWWaWG8e6LgG2O+e6+wJdzLehc+5VoLyT5+3NNrscWOmcK3fOHQZWAlfEqj7n3AvOuWbv6tsEDxHeJa/GDOfcWy6YWI+F/E4xqbEbXb2mMfs7764+r1f+SeCP3T1GrLdhtAzkoPfdmazMbCpwOrDaa/qS9zH60daP+SSmbge8YGZrzGyJ1zbOOVcMwTcrYGwC6wt1PeF/XH7ZhtD7bZbIbfl5gr3LVtPM7D0z+6eZne+1TfRqind9vXlNE7UNzwcOOucKQtr8tA17ZSAHfURnsooXMxsB/AW43Tl3BHgQmA6cBhQT/BgIian7XOfcPGAhcKuZXdDNugnbrhY8f8FVwJNek5+2YXe6qichdZrZXUAz8HuvqRg4zjl3OvBV4A9mlpGg+nr7mibqtb6B8A6Hn7Zhrw3koI/oTFbxYGaDCYb8751zTwE45w4651qccwHgVxwdWoh73c65/d5lCfC0V8vB1iEZ77IkUfWFWAisdc4d9Or1zTb09Habxb1Ob4fvlcBnvKEEvOGQMm95DcEx7xO9+kKHd+Lxf7G3r2kitmEq8AnWx366AAABg0lEQVTgiZC6fbMN+2IgB70vzmTljeU9Amxxzt0X0h46rv1xoHXP/nLgejNLM7NpwAyCO3NiVd9wMxvZukxwh90mr47WWSCLgWdC6rvRm0kyH6hsHa6Ig7BelF+2YYjebrPngcvMLNsborjMa4sJM7sC+BZwlXOuNqQ9x8xSvOXjCW6vHV6NVWY23/t/fGPI7xSrGnv7mibi73wBsNU51zYk46dt2CeJ3hvcnx+Csx22EXx3vStBNZxH8KPaBmCd97MI+C2w0WtfDuSG3Ocur+YPiPEeeoKzFdZ7P5tbtxMwGlgFFHiXo7x2A/7bq28jkBen7ZgOlAGZIW0J24YE33CKgSaCvbab+7LNCI6VF3o/N8W4vkKC49mt/w9/4a17jffarwfWAh8NeZw8gmG7HXgA70uUMayx169prP7OO6vPa/8N8MV26yZkG0brR9+MFRFJcgN56EZERCKgoBcRSXIKehGRJKegFxFJcgp6EZEkp6AXEUlyCnoRkSSnoBcRSXL/Cw2Cri4DF7wvAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(lr)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Why so many peaks? Remeber! We take random actions with probability $\\epsilon$. Notice difference in performance when we set epsilon to zero. That's because we use an off-policy method." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Average reward for epsilon 0.1 : 414.55\n", "Average reward for epsilon 0: 491.6\n" ] } ], "source": [ "def rollout(epsilon, render=True):\n", " done = False\n", " R, reward = 0,0\n", " state = d.Discretize(env.reset())\n", " while done != True:\n", " if render: \n", " env.render()\n", " action = choose_action(state, epsilon) \n", " obs, reward, done, info = env.step(action) \n", " R = gamma * R + reward\n", " state = d.Discretize(obs)\n", " return R\n", "\n", "rollout(epsilon)\n", "rollout(0)\n", "\n", "print('Average reward for epsilon',epsilon,':', np.mean([rollout(epsilon, render=False) for _ in range(20)]))\n", "print('Average reward for epsilon 0:', np.mean([rollout(0,render=False) for _ in range(20)]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "But instead of stopping after return in last 100 experiments is higher than 200, let's iterate a fixed number of experiences" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8nFW9+PHPSTLZt2bpli7pBrS0UEqBAi1rZata5ILyE9nEWxRQ1HsvFgVRQUVlcb16ERRwAwSVpWy1FMtWSitdWLrRljZ0Sdrsy0xmMuf3x7NkMpnJ7Jl5pt/369VXJ89MJufJTL5znu8553uU1hohhBDZKyfdDRBCCJFaEuiFECLLSaAXQogsJ4FeCCGynAR6IYTIchLohRAiy0mgF0KILCeBXgghspwEeiGEyHJ56W4AQE1Nja6vr093M4QQwlHWrVt3UGtdG+lxGRHo6+vrWbt2bbqbIYQQjqKU+jCax0nqRgghspwEeiGEyHIS6IUQIstJoBdCiCwngV4IIbKcBHohhMhyEuiFECLLSaAXQoTU59e88cEhHlmzm+5eX7qbIxKQEQumhBCZ5Z7lW/nr2j3sa3MDsOtQN0vPPyqpP6O710dxvoSg4SC/ZSEEYPTgL/r162zY0zrovt/86wNqSvP5woLJSflZ6z5s4T9+/Tp/vOYk5k+rScpzivAkdSMG+fBQF/VLl/H0hr3pbooYRg+9vitkkLfcsex9DrS7k/KzrJ9z4yNvo7VOynOK8CTQi0HWfdgCwHeffi/NLRHD6XvPDHy9//rFk3nrWwt5+9aP2ccefmNXUn6WlfM/1NXLJ3/5WlKeU4QnqRsxSFOHB4CjRpeluSXptbe1h1PufAmAXXcuSnNrUsvvN3rVk2tKeOm/zxh0/647FzHrthfsnH2idh7stm9v+qiNnt4+ivJzk/LcYjDp0SdIa82mhrZ0NyOpPmrtAcDb509zS9Lr20++m+4mDJvWHi8Al588MexjpowstTsBifqgqZNTplTbnYnPP/hWUp5XhCaBPkHPbNzHJ375Kss27kt3UxLm9vbR6/PT2m380Xe4D+8pdTWl+QBMri1Jc0tSzwrg1aUFYR9TU1rAK9sOJuXn7WjqZFJNCb+9Yi4AoysKk/K8IjRJ3SRoX5vR+1294xCnH1nLzNte4IhRpTzz5QXk5znrc/Sce1exu7n/krrd7U1ja9Kvx9sHQK8v+69srNz76PLwAbfYTK10enyUFsQfOtp6vLS7fUysLmZ8VTHVJfl0eQ7vTkWqOSsSZSBXrvEr3NfWw+3m4OXWA50ccctzjnvzBgZ5kB59u5nOaOsZ/g+8DXtah3U2Skt3LwDHTxwR9jHnzxwNwOZ97Qn9rIYW4302bkQxYAzIvvjeARqTNKNHDCaBPkGdZjDs7dMc7ByYv1z8K2fNJigvHNhL63B77UG6TOD3a6783Rrqly4blnGRdvO17XD76BvG38P6Pa0s/tVrTLr52WH7mXtb3cyfWkNujgr7mCkjSwG4+DdvJPSzPmoxroLHjSgacDzR5xXhSaBPUKc5Tey9ve3k5Cimjizl4uPHAbC9sXNAoGxsd3Puvau47k/r2HmwKy3tDUdrPeiS3K+hK4OWvi+891/8a2sTAJ/45asp/3ntAT35zmG8Ovvnewfs21v2dwzLzzzQ7mbUEGkbgGlmoAfYeiD+djXYgd7o0d+++GjAuKKsX7oMj68v7ucWoUmgT5CVnjnY6WH5ewcodOVw50Wz+MpZUwE4/a6VuM1c7+9f38WWAx08u2k/Z971sv0hsL2xI+358B5vH349uJfVnsb0za9WbuePq40tMVu7e9nRNPDD8eE3dtmX+8s27uPJ9R8l9ee39XjtHm77MKVvDnZ6+OXK7fbXNz2+IaU/764XtrDk4bXsa3NTWxZ+IBZAKWWXQTjn3lW0dPXG9TMbWnoozs9lRLELgMtOmjigvMLmfcPz4XY4kUCfoM6gQHj5vInk5eYwY2w5AHuae7jzuc0A+INyrh0eH1prFt6zik+lMc2zbOM+Znz7BcCYWRGoI40fQD95YQu3/OMd9jR3095j/J5n1pVzx4UzAWP644k/WIHWmuv//G9ufGR9Un9+u9trf/AN13jFb1ftAODHFx8DwMy6ipT8HK01n/jFq/xy5XZeNK8gIgV6gIvm1Nm3j7t9Oc+/sz/mq9OGlm7qKotQyvgQzclRLFkwmUk1xuwmXwalC7OFBPoEdXr6LzN/e8VcPnPCBAAKXP2LP6zL757ePiqKXJw8uRqAVVub7DzsB03pS+Vc/+d/27enmpfn1v/JWiATq8CU1793t9hXPF8+axqfmzdxwJTHZM3tDuTx9eH2+u1APxxXXG5vHx8eMgYqPz13PNNGlg4a90mWDw91s+mjgeMc0QT6kWWFvHHzWfbXX/zjOs686+WYqlu+su3goOmUOTmKuz99LBD773rnwS6u/v0atoVJJ23Z38FLm40PM7e3L2W/01j1+TW/fGkb7yc4uB0NCfQJ6vL4mFxbwlcXTuP0I2rt4wum1nDdGVOYP7XGrg/S1uOlosjFFeailC//5e20tDlYXWV/uuY/T5vM6pvP5tEl8wDYlaaxhI6AnPiNj6y3Z4WUmQPGT3zxFKaPMa6azr7nX8n/+WYP3vrdDEeP/qhbn+f5d/czb3IVYMwt39+emqD09h6jzEVFkYuHPn8iF82pG/D+HcqYiiJe/caZA44tf+8Abd2RA3Rbt5ceb1/I+fjlhUYqJ9Y02Zl3vczKLU2c+9NVIX/euT9dxecfXMue5m6OuvV55t7xz4yor7PzYBd3vbh1WCYWSKBPUKfHx4SqYr668IgB8+bzcnO46byjOHpsOTsOdtHn17R0e6ksdlFe5Ar5XFoPnrkzHHq8fVx20gR23bmIusoiRlcUUlFk/dGlJ0cf/MdupWWsYDCiJJ/fX3UCMDAIJ+sP2JpSaQ0YDleOHuCi44zB/NqyAjbsaWXz/uT3+L72qJH7f33pWZx+RC33fHq2/ZpHY9yIYi6YNdp+z9/4yHqO/d6LEb/P+sC+Zv6kQffZ77k4P1T9uv9qoM+vmfLNZwe06aHXd9m30zn2ZNneaFyBWB2WVIo60CulcpVSbyulnjG/nqSUelMptU0p9ahSKt88XmB+vd28vz41Tc8MXREWj1iXw2ff/TKrtjYxbkSRPfce4Kefmc1lJ02gtCCPb/59E3Pv+OewzycOtQAmLzeH4vzctOXog+euN5sDf1agBxhVPjjV0NWbnBkbVmDv79Gn9vegtSZHGTnwT58wHuhPXy15eF1KfmZBXg4lCSx8+t/LjmfrHefH9D3Wh/I8M30ZyLpai+VDNfiD/QsPrWV/m5urfr9m0JTY+1/dad+OdyA5maxO1IiS6D9g4xVLj/5G4P2Ar38E3Ku1nga0ANeYx68BWrTWU4F7zcdlrUirBM+fNQYwNm4AI08/Z0Il150xhWe+PJ8Lj6tjVHkhnR4ff1mzBzAWkAwXb5+fXp8/5DmUFealbdFUuD/28qL+diql+Nmlswfcn6zFTVa9n2mjjLGKVPQAA8ch2t0+/BpmBPTuxlYWhfq2hLWaveqvnD0tKc9nlTGIZiX4v3cbKaOywsHvt0JXLgV5OTEFemv18gKzpv2anc3M++GKiKUamrszINCbnYeywgwJ9EqpccAi4H7zawWcBTxuPuQh4ELz9mLza8z7z1bW8HoW6vT4huwV1VUW8anj+mcq3HDWVDutY82oqCwe+EIP57Zt1vTQUOdQVuiiw5OeHv1n738TgCe+dArnzBhlHw/+QFo8u473v3de0qdBWnO9J9eWUuRK/pVNQ0s3k7/5LM9tMmokWVcsI4rz7cdYgdiajZIs75mDf8eOq0zK831sxij+59wj6fX5h5wD/9ymfdz2lFEoLlSgBygvcsU0GGt9sF8wawxfW3jEgPtmjCnnb9edYo95AOSZ75PmztgDfafHx9u7W6JeRNjr89PY7g6bTmx3+1Bq8Hs6FaLt0f8UuAmwin5UA61aaysiNQBWNKsD9gCY97eZj886vT4/3eZMmqHcfMFR3HHhTN757rkcP7Fq0P3B3z+cS+6tHnsm9ejf2tVs3z5qdBnjq4w8eZErl7zcwW/Zovxc/vD5E4Hk/e463MYc+pL8XMoK85I+VmFNSfzZim1Afy878DK+0JXLOTNGsaele/ATJMB6TYM7GImwAne494vfr/nSn/pnd5WH6cVWFLlieg2tqcs5Cq49fTLnHT3avu/JG05lzoQRPLLkZE6oN0o7/M+5RwLx9eg/ds+/+NT/vs7XH4tuGu/SJzZy4g9WMOnmZ1mzs3nQ/e09XsoK8oZcjZwsEQO9UurjQKPWOjBRGKplOor7Ap93iVJqrVJqbVNTU1SNzTTWwFJ1af6QjxtZVsjn5k0M+8kd/KYPnLKZatbK19IQPayyQtewDlq981Eb9UuXcYm5FP6Vm86kpCDPHoy0LtNDsQa4kxXoO91GSk4pRXlR8q9sus2xhM3m1FurYmhl8eD30o6mLnqSNPYA/cE4XK86HpFmzAS/LmF79DF+qD653tgFbXJtKYWuXH5z+fHsunMRu+5cNGAszO01+qjWtOF4dk+zphr/Y3103/u3t/sX8H36/waXd2jr8VKRxA/boUTToz8V+KRSahfwCEbK5qdApVLKerXGAdbZNwDjAcz7K4BBH2da6/u01nO11nNra6Ob1pVpOpKUYwvMO8PgRVipNHTqJm9YB2M3NPRvY1dTmm/35ENdBQWzrooee2tPQnVpbv7bRhb9/BUeeuNDux5/Knr0gYGvfukybvnHO8DA1A3AseON9MqHzcmb5pqs920g6z0crmMQPO4UttMTY+rGSgqfUD/0e+TqU+sBOHGS8bhDcaRuTjK/d8oQZauf2rCXP735YcgOxx1BO3i1dvfGNNMpEREDvdb6Zq31OK11PXAp8JLW+jJgJXCx+bArgSfN20+ZX2Pe/5LOhEmrKdCepJ5RcI8+mqqXjR1utjd2JvRzITB1M3h3n/JhTt1Yfxw1pQXcedEx9vGvf+wIfvqZ2fz9ulPCfq/Vo1+xuZHfvrIj7jb8Zc0e3t1rXEFYve7ywtiCTzSCe77W4O+IoB7e/KnGIKO1kOq9ve3c/eKWuKaRNnf18t9/3WD3SJOZG7Y+NMJ1DO56YQsAv7/6BDbffl7IFBwYH9jRjrNorXHl5vCfCwZP1Qx20Zxx7LpzEWWFxjqWDw91xdwh2Gb+ve061B12LOIrf3mbb/39Hd7da8yN/55ZxweMWT/rA/bktdbVDIdE5tF/A/i6Umo7Rg7+AfP4A0C1efzrwNLEmpi5rCAYXPUxVlbPddExxgydwMVCbd3ekG/IBT9aycIkLBSyBh1LCwa/4coKo/+jS4a2Hi/5eTmsvWUhCwMGYAEuPK6O4yaEL6FbFhC07nxuM39du8f+etfBrqh2ywp+TKHL+PMYUeyy03TJ0tbjJUfBF4Lmkwf3sq1CY2t2NtPT28cFP3+FX7y0fcB7JFqn3vkSj69rsDfmTuZ+Cf2pm8HtenL9Rzz/7n7A2Kqw0BV+y8Dywuhy9Afa3Rx3+3J6ff6IxdiCzR5fSVdvHx80Rd9R6vX5ae7qZWJ1MX1+zb7WgVOgPb6+AX+Pq7Yas36OGFXGvZ851j5+2W9Xc/8rO+jp7cvcQK+1fllr/XHz9g6t9Yla66la60u01h7zuNv8eqp5f/zdqwyXrEvgQlcuu+5cxK8+OwfALs7V6/Nz7Pde5LgQC1E85mYY1zz4FvVLl/HoW7vj+tlWyiDUwFx5YR4en3/YNt5o647/jZ8TNKD1P49vBGBPczdn3PUyvzAHPYdy1K3PD/j6xa+eDkBVSUFcszSG0trtpbzIxS0fnzHgePDAnPW6PPDqTqZ/u7990bRnY0Mr0299nv1mbnlidbF9X7LH/6xxqgMBa0D2t7n553sH7MVuj117MhOrh55BVF6UR7vbF/GK5YOmTntcI5rSDYGsdNiGgN51JIe6jIWMU2uNHH+H28czG/faV9UrNzcOuMLeYX6IVJfkc+HsOnu9QVdvH3cse5/p336eD5q6hi3Qyw5TCUjFoBYYl+nrPmxmUk3/HO7VOw6FXGSyYnMjAN94YhPjRxRzinmpH40HAhaQjAzxxxJ4OT7UFnNgfCjNvWM5lcX57G7u5p9fP42pI2PbXLytx0tlAm/82z4xg3f3tvP4ugb72Jf+ZMwheHlrE18/58hB37N6xyHqq0sYXVFoXzk9umQe1aUFTDADY1WJi67ePjy+PgrykrOBdWuP187Hb73jfI645bmQjwvX+z3Y6aE+wrTLh9/4kB5vHy9tbuSzJ02gujSfY8ZV8PurTqCqZOgJBLGqLsmnstjF22bwbOnqZd4PVwx4jJUfH0p5oYs+v+ZAu2fI7QWbA3L+sc5aGW+udt7bGv3CRKue0uTaElZsNt6rN/zZKGFS6MrBuuieNrKUbY2d9qyqqpJ8lFLk5ylyc9Sgq/Mi1/CEYCmBkIBUDGotOW0yAM9t2j8gV3/pfasBuOBnr3Dv8q0hv3dvDAXINja0crs5OHTbJ2YQaqlDpAG2QDO+/Tztbp+9S9XCe1bhi3Fz8UQvZa8+dRJ3XXIsXzx9CjkKXtp8gHc+MvLtvT4/73zUxvv72u3a8n6/5tL7VjPvhyu45R+bmFBVzBlH1nLS5Gp7dgb05/+TNV6x+1A3T2/Ya/fW8/Ny7JkioTxsTh0NdDCKHr01Z1yj+c+H1/La9kPMGFNOdWlByNc7EUopTp1SwxsfGCmLwCmyAI9/8eSonscK2vN+uMIu7x2KVR+mtqyABdNim8yRn5dDVUk+BzpiD/RW5+vxdf2pQbfXuOpVCp7+8nwAdhzsQqmBs6gun2fUuDop4ANvuPYjlkCfgA63jxwFJfnJ6eUBfPOC6YwuL6Stx8tjAXlmMCrvvbev3Z57bVk4fSTQv6dnNP61xZjSevLkaj5nvgGDlUcYYAsUqrTsEbc8R/3SZVEvN29NIHUTaHxVEX4Nn39wrX1s8/4OPv6LVzn/Z6/wjSeMtI41AArwx9W72d3cTVGIHnS8xbZCOdTp4bSfrIzp+U47opYfXjQLgO9/yijRbKUSotHn1yw3SxGfddTIWJobk2PGVXCws5d2t9ceuCzJz+W8o0czN8KsGMt5M/vnwQ9V96m128uo8gLe+tbCuN4zI8sK7JRWNAJ79BB6iqXWxhVYoSuHPr+mssg14GrjW4um8+Y3z+bRa/s/9C6ZOy7mtsdDAn0COgLmWidTRZGLv65r4BcvGRtQfNwcpA1XMvg7nzRG9mOZlvlBUydjKgr5y5J5A+YbByqPobCZ9QdgfegA9uXsS2Z6KZJkzSs+pm7oFZ9v7mhGa81n71896L6jxw4uMJVosa1A33m6f4pdSxTVHi2XnjCeVf9zJpccb9TBiWV64E/MGS/5uTksnD4qwqPjZ6WSdh3s4s2dzUyqKWHDbefwq8vmRP0c40YU83+XHw/0ry0IpbWnl8qi+NNPR44u47290ReLswN9ULqsOiAF9rl5Rolyq13B6TFXbo49cLzsK/P5zefmJC0VGInk6BPQ4falpE5FcM7fmpWzL6AHOvDxZs87ypkYWmv+sX6vXR8kUjuimVrY7enj03PH8eOLj8Xj6+PIW/oHDnOi7E60J2kWwsy6cr54+hTycxUXHDOGTrdvwH6kYysLueJ3a9jTbPw+n7txAQ++tot39rZx/ZlTBz2flcJKxmKsbQc6GFHs4vxZY/jksWOj/j6llD1mUF6Yx6EoqpxaA/ZWyum2T84YNGidTFaphs37Oli1tYmrTqkPO41yKNbYxZCBvjuxTsHRY8t5cv1eYwJAFM+z82AXo8sLB4xVPf/VBYwfUUxXr49cc2EdGAPo+9vdVJeEH9c6emwFR49NzaYyoUigT0CH25v0gViAdWbhJzDyeifUj+DXhM/BW/Oho13cZM0PD5WmCBRLyqKtx2s/viAvl23fP58vPLSWf21tiiq37evz0+HxJSXQB255Z3nmy/OZUlvKaT9Zyf42NxsDaoBPH1POjy4+JvhpbMlM3Rzs9HDezNH84FOz4n6OmrICDkaRDgv+YFIhF60nz4SqYpSCm8zUWLz5Z2vsYqgprW09XrsDFI8JVUbb9rR0U1EcOeBuOdDBtFGlA1IxR402rv6CFxta7U/2gHciJHWTAKNHn/xA/4NPzWJSTQk7fnABt1840x7Q2RuiRz9vcpVdkyXa1E2rGQAi5Wvt1E2ED5Ben58e78CaP67cHO67wrgEjybQW2mRVE03m1lXQVF+Lr4+P40BO1JZNe2HEu3vIdDBTg/1S5cNmHLn92uau3oHbdcYq6ri/KimVwYH+uljYpsFFatCVy5jK/orbn72xAlxPY8VKFsjBPpEZmhNMD8krMkDQ+nza7Y3dnLEKOP3t+wr83nE3JgnFGtNysSa+D+Ikk0CfQI6PN6UpG7+34kTWPnfZ9iX2VaP0gr01qj9Lz97HA9ebczIKC3Ms2eTgDH9LNxc5Da7rsrQbS/JzyVHRQ7UVgAMvgQuyMslPy8nqgBpBaVkFtoKJTAv/vgXT+bMKAYn49mEZe4d/wQYsIim3e3Fr0PXs4lFRZgyATc9voHvmNUhwfidnnv0KGpKC/jOJ2YMueAsWaz0zay6irjSNtCf494xxO5mreYmPvEaX2V8IEUT6Pc0d+Px+TnSDPRHj60IOdXZYi28m1g1PDNqoiGBPgGp6tEHswKNVQDr+5+ayYbbzuHjx4y151mXFhjlCrY3dlC/dBlzbl/O8WawCdba02s+79ABRymFX2MPCodjpTRCVSSMtoyCFehTvYDkVnOB0qJZY5gTZeAryMshPzcn6hx98GwRawGclXMOLnMQq/IiF9saOweVy31sbQMPBuyi1Nbjpaokn7W3LOSqUyOXCUgGK9AnEoStFbu/f21XyPs9vj56vH0JfWCWFbqoKsmPKtDvOmR84EyKMhVlFd8bXZHYlVsySaBPwHAHeqtOxsTqkkEBsazQRXNXLwvv6d83szlEHrepw8P1ZrnYWILqUDW4hwrSZYUuO9D7+vz2xhPhniNc+dpkuWb+JGMV8mVzoh6YNCpY5kWdurEqKh412ugB/vN9Y9ZRa5KuWnYd6qLX5+faP/YXlA21h0HguMlwsXb9Slbp3VBXpfZ7JcFOwfiqYvZEEeitMiHWQqtIZpqDrCPLYivNkEoS6OOktTYHY1P/hxRckyTUdEi/1ryx41DQ49SgP5RlG/fa6YtoZhtce7qxgOtrj60PmzO18uvBVThhYAXMy+5/k4v+9/UBdWgsnfYq4+ENTNEqj6HYVktXL0rBs19ZwNFjy+k0z9/6/UW6kork7d3GB741Nz74tt+vcXv76PX5Ew6GsfqP48cxq66Cb5x3VOQHD8GqGx98FXX3i1s48fvGittEcvRg5Omj6dE3tPTgylUhV4+HcsNZU7n7kmNDTtVNFwn0cfL4/Hj79LD06MFYjDKrroLnblwQ8v6tBzrs26dMqWbp+Ufh7dO09Xj5yQubWbm5kT3N3bQF5JlHRfHGXXyssZ/Mk+v3Mvt7y/nw0OC86dA9+jzae7x4+/y8aW6+YNWhCdTp8dqPz0SVRdEVNnt3bxuNHW4qi1zk5ChqywrsVazJSk898aX+BTdWHSJPQD2iQ129w5YKCzamooinvzw/4Q2vrbo8+83aOfvaeph+6/MD0ohjhiiREI0JVUV81NITcQX3R609jK0sivoKsKokn/84flzS19ckQgJ9nIZzv0eAJ68/laduODXsH5C1sQLA3Ikj7Ev2v65t4FcrP+DqB99iwY9X8mFzF4WuHLZ9//yoBstmBPVKTv/Jyzz21sAe+VBpl7ICI3XzZNBKwuBUkF0uOUMD/eiKQl7bfmjQauU/vLGLM+96GTDKSy/6+as8trbBDrA1pQX2YptkBd/jJ1Zx9yVGRcTXzZIDgas8n39nX9oCfbKMNhcWHWg3fnen/XjloI1njp+Y2ODyxKoSfH5t7+ccTkNLN+NGpGb/3uEigT5OySpRHC2l1JA9BKvyJcA5R4+20yiB1QTB6PmXFrjCroYN99yBKypvemIjL5plZyFgMDZMj77D7bPr81hTC4Mvya0ZQyX5mRnorXbfFHA10tPbx61PvsvOg100tHTbOXjon1tdU1rAoS4PWmv795SM4GtVYPyuudI2cNX0a9sP8aaZxkv1LKZUsVaQHjDPy9vX3zG49vTJrLtlYcI9ZmvmjFWfJ5yGlh7GVWbOVMl4SKCPU6oqV8Zr0TFj2PnDC1jzrbOZWVdhBxMrB2nNmW9o6Qm5yUik577/yrkDji35wzpu/tsmWrt7ae/xUpCXE7LSYnmRiw63l5l15VQWu7jtE8asl6agmSmdbh8l+bnDsn9mPK49fYp92xr4fP7dffaxDrfPnrYK2FPxakrz7RRau9tHoSsnKXXgraJrOw8a2wzub+uxFyg9/+5+bn3SmGY5I8EUSrqMNAd1D7S7B0wqePqG+dx8/vSI1VSjMb6qCFeu4qMhqli6vX00dXiokx794ckaYAy1YUe6KKXskX4rjfLiewfIy1F884LpgDHFL9S2gdF457vn8mjAQpG/rNnN/a/spN3tDTvoV1aYR1dvHx1uHxOrS+ye8Tn3rqKlq5fvPv0uD7y606gblCEfmqHUVRbZc6/vf8Uo7/xoQAqrrcdrX6VMG1nKNxcZv2+rTntzV2/SB+/nTTbWUxzs9LByS9OgIl1fmD8pKQExHQrychlhlhL41t83AcbU2Fnjklc2QClFbWkBjUNUsbTWrkjq5jDVmWE9+mCBgdfn1wOKL8Ub6EsL8jhpcjV3XtS/fN+v9ZDlha3A1tDSw4hi14BNIo67fTm/f20Xtz/zHp0eX1K3tkuFH/2HUSbhnuVbOevul1m9o78Ub3uP1x63ufczs+0PtP4KoD7ae5I7HffKk+sB7Cmr00aVseHb53DLoumMLCuwP9ydamRZIdsaO+0B2atOqU/6zxhVUUhDc+gaUtBfbnlclFMrM1Vm/2VlsExL3QQLDLy/+dwcyotcKGWUUi1LMKCedkR//e8cpcz52qGf0/r97Gnp5viJI0I5V53KAAAV70lEQVTuBlScn0uHx0dphk6ttJwypb8I3I4mY/bR5NoSdjR10dbjxcoiB/7uA8sntLuTO6/dem5rB6ebzj2SimIXX1gwmS8smJy0n5MuWwJmkl11Sn1K0npTa0t5ZVv4HP03njCuJiR1c5ga7lk3sbICbGWxi/NmjiE3R9lBJtEUydjKIs472qgb3u720t4TvhiZ9QGgtREAQ30gjCjOp9PtTfgDaDj8+T9PGvC1VfOkrcfbP9gaMAAauJdqe5IX2AUXzUpmWiMTBPbg58ewc1osasr6B8uDBc4MG5vgVM50k0AfJ3s6YIYGJ1duDo9dezKPBWxyYC29T3ShCcBvLj+e+upiWrq9bPqoLezjAj8IR5YbOxs98+X59m47YKQ9nJC6AaNXv/OHF/Di105j2Vfm26Vo280cvVJQGjBzqH+XLi8dQ4xlxCP46mi4V8GmmrXPAsDZ01OzYUp1iTFYHmqvAWvCwO2Lj86oOfHxyPy/rAxlbTqSqbNEYPAenRXF+XCo2/g/CSqL89nUYKzS/KApdAGqwB7s5816KzPrKlgdsIq3szf5+etUUkrZlQzBXBTm9uHXmvJC14CFNdYH3c1/20RNaUFSp+NWZ1AZ3FR57sYFfHioO2WB1hpLOdTpGXRVapVHGJdAOeRM4Yy/rAyUqlr0qWQFhmSlSEYUu+z6O/91zhEhH1Ns9m5rywoGTL/85LFjeWbjPmbWlfPH1bvZ3+7O6Fk3Q6kochk5eq0HBYvAbSYPdnqSmupTSrH59vNYubmR+RE2kXGq6WPKE15lOxQr/dXc1cvkoK1nrSvVSdWZU4UyXpK6idNwFTRLJisP2dASub5HNCYG/AGEq+tRX13M5+ZN4K/XDtwcemR5If+4/lRm1fXnlZ2Qow/FCvShZh8ppbhmfn/lyGSfY6Erl/NnjcnYsaJMZ01/DbU/7SNr9nDsuAp7i0Qnk0Afp1TVok+lez8zm3NmjOLmJE27Cwxg4UrG5uXmcMeFs8L+sQTmlZ3aoy8vNAqetYaZZmoV6AIS2hVJJJ81xvK1RzcMOO73a7Y2dgyYYeZkEujj5MQefWVxPvddMTfkCtZ4BO6UFO/vYvaE/o28E92QI12sHn24zTACf9+ZVNFQ9A9oB79/Ozw+e6ZYNnBWpMog1krPw1mhq7+fEO9u9mMqinh0yTxe3X6Qjx8zJllNG1ZWoO/t89sbW4czubZ0mFolopGbozhpUhXBsyuTWZcoE0igj1OH2+uI6YCplKyZECdNruakIbZmy3QVxS5azVLM4YqIPX3DfJo63Rk9S+twNbqicNCGOMna3CRTHN6RKgHtbt+wVa4Uma28MM+uCx8u/WQsZsquBU3ZYmSZUUpaa213XqRHL/D4jN17nJajF6kRGAwS3Q9WDL+RZYW4vX46PD7ychSLf/ka2xo7gexZhCaRKg6Zvu3dcFrxX6eHXD5+OCkfEOidOaB8OLMGZJs6PLy8pckO8hDddptOIIE+Dple0Gw4TZHBxQE9eqdu9HE4s/aCbWz3kJ87cAxFUjeHsQ7p0YsA1lxskB69E9k9+k4PrebmMT+7dDYvbW7MmgkX2XEWw6zDndkbWYvhZa2uBAn0ThSYurnb3PJy8ew6Fs+uS2ezkkoWTMWhXVI3IkBguWB5TzhPRZGL/NwcPmoJvwGJ00mgj4PVo8+WEXmRmMCVrzkyT95xlFL09vn53WvGFpFfOmNKhO9wHgn0cZDBWCGyVzZOMJBAHwcr0Me796rIPq/cdCbPf3VBupsh4nTLov5CfyfWVw3xSGeKGKmUUoXAKqDAfPzjWuvblFKTgEeAKuDfwOVa616lVAHwMHA8cAj4jNZ6V4ranxYdbi9FrlxcufI5KQxSldLZvrBgMqdMqeHlrY1MqM6+1zKaSOUBztJaHwvMBs5TSs0DfgTcq7WeBrQA15iPvwZo0VpPBe41H5dVnFi5UggxtBljy7nujKnpbkZKRAz02mAtFXOZ/zRwFvC4efwh4ELz9mLza8z7z1ZO33AxiFGLXgK9EMIZoso9KKVylVLrgUZgOfAB0Kq1tnbUbQCsSad1wB4A8/42YFBpQqXUEqXUWqXU2qampsTOYpgZPXqZcSOEcIaoAr3Wuk9rPRsYB5wIhNqiyCp4Eqr3PqgYitb6Pq31XK313NpaZ+3iIqkbIYSTxDSaqLVuBV4G5gGVSikr2o0D9pq3G4DxAOb9FUBzMhqbKTrcXplDL4RwjIiBXilVq5SqNG8XAQuB94GVwMXmw64EnjRvP2V+jXn/SzrLyhtKj14I4STRRKsxwENKqVyMD4bHtNbPKKXeAx5RSt0BvA08YD7+AeAPSqntGD35S1PQ7rSSQC+EcJKI0UprvRE4LsTxHRj5+uDjbuCSpLQuA3n7/PR4+2QwVgjhGLLiJ0adUv5ACOEwEuhjZJU/yJY61UKI7CeBPkbtdi16Sd0IIZxBAn2MrB59uaRuhBAOIYE+Rh3SoxdCOIwE+hhJLXohhNNIoI+R7BcrhHAaCfQx6u/RS+pGCOEMEuhj1OnxUZCXQ36e/OqEEM4g0SpG7VKiWAjhMBLoY2RUrpT8vBDCOSTQx6jD7aNUAr0QwkEk0Meowy3bCAohnEUCfYw63D7KCiRHL4RwDgn0MZJa9EIIp5FAHyMjdSM9eiGEc0igj0GfX9PV2yc9eiGEo0igj4FsOiKEcCIJ9DGwatGXS+pGCOEgEuhjIJUrhRBOJIE+BlKLXgjhRBLoY9DpMfeLlR69EMJBJNDHQFI3QggnkkAfA9l0RAjhRBLoY9BubwwuOXohhHNIoI9Bh9uHK1dRIJuOCCEcRCJWDKzyB0qpdDdFCCGiJoE+BlLQTAjhRBLoYyC16IUQTiSBPgZSi14I4UQS6GMgqRshhBNJoI+B1KIXQjiRBPoYSI9eCOFEEuij5PdrOnsl0AshnEcCfZS6en1oLeUPhBDOI4E+Sv0FzSRHL4RwloiBXik1Xim1Uin1vlLqXaXUjebxKqXUcqXUNvP/EeZxpZT6uVJqu1Jqo1JqTqpPYjhI5UohhFNF06P3Af+ltZ4OzAOuV0rNAJYCK7TW04AV5tcA5wPTzH9LgF8nvdVpIJuOCCGcKmKg11rv01r/27zdAbwP1AGLgYfMhz0EXGjeXgw8rA2rgUql1Jikt3yYSY9eCOFUMeXolVL1wHHAm8AorfU+MD4MgJHmw+qAPQHf1mAec7T+jcEl0AshnCXqQK+UKgWeAL6qtW4f6qEhjukQz7dEKbVWKbW2qakp2makjb2NoJRAEEI4TFSBXinlwgjyf9Ja/808fMBKyZj/N5rHG4DxAd8+Dtgb/Jxa6/u01nO11nNra2vjbf+w6XTLfrFCCGeKZtaNAh4A3tda3xNw11PAlebtK4EnA45fYc6+mQe0WSkeJ+vy+FAKil256W6KEELEJJru6anA5cAmpdR689g3gTuBx5RS1wC7gUvM+54FLgC2A93A1UltcZp0eHyU5ueRkyObjgghnCVioNdav0rovDvA2SEer4HrE2xXxul0+yRtI4RwJFkZG6VOj4/SAgn0QgjnkUAfpU6PjxIJ9EIIB5JAH6VOj1SuFEI4kwT6KHW6JXUjhHAmCfRRkhy9EMKpJNBHSWbdCCGcSgJ9FLQ2dpeSHr0Qwokk0Eehu7cPrZFAL4RwJAn0UbALmknqRgjhQBLoo2DVopcevRDCiSTQR8Hq0cs8eiGEE0mgj0KX1KIXQjiYBPooWKmbkgIpUSyEcB4J9FGwUzfSoxdCOJAE+ih0mvvFyqwbIYQTSaCPQldvHyCpGyGEM0mgj0KH20d+Xg4FeRLohRDOI4E+Co3tbqqK89PdDCGEiIsE+igc6HBTN6Io3c0QQoi4SKCPQluPl3IZiBVCOJQE+ii09/ioKJKplUIIZ5JAH4XW7l4qJUcvhHAoCfQR9Pk17W7p0QshnEsCfQRtPcZiqcpiCfRCCGeSQB9Ba3cvACMkdSOEcCgJ9BE0d5mBvkQCvRDCmSTQR2AF+moJ9EIIh5JAH0FXr1WiWObRCyGcSQJ9BJ0eKWgmhHA2CfQR9O8uJT16IYQzSaCPoMvjI0dBkUt69EIIZ5JAH0Gnx0dJfh5KqXQ3RQgh4iKBPoIuj08GYoUQjiaBPoIuT58MxAohHE0CfQSdHp8MxAohHE0CfQSSuhFCOJ0E+gg6JdALIRwuYqBXSv1OKdWolHon4FiVUmq5Umqb+f8I87hSSv1cKbVdKbVRKTUnlY0fDl29kroRQjhbND36B4Hzgo4tBVZoracBK8yvAc4Hppn/lgC/Tk4z00cGY4UQThcx0GutVwHNQYcXAw+Ztx8CLgw4/rA2rAYqlVJjktXYdJDUjRDC6eLN0Y/SWu8DMP8faR6vA/YEPK7BPDaIUmqJUmqtUmptU1NTnM1ILW+fn16fn9J8CfRCCOdK9mBsqOWjOtQDtdb3aa3naq3n1tbWJrkZyWHVuZEevRDCyeIN9AeslIz5f6N5vAEYH/C4ccDe+JuXXp1S0EwIkQXiDfRPAVeat68Engw4foU5+2Ye0GaleJyoyy5RLIFeCOFcESOYUuovwBlAjVKqAbgNuBN4TCl1DbAbuMR8+LPABcB2oBu4OgVtHjZ7W3sAqC0rSHNLhBAifhEDvdb6/4W56+wQj9XA9Yk2KlM0dXoAGFNRmOaWCCFE/GRl7BBau439YiuLXWluiRBCxE8C/RBaur3k5SgZjBVCOJoE+iG0dnupLM6XTUeEEI4mgX4IHW4v5YXSmxdCOJsE+iFIiWIhRDaQQD8EKWgmhMgGEuiH0OHxUVogM26EEM4mgX4IXR4fpdKjF0I4nAT6IUiOXgiRDSTQD0E2BhdCZAMJ9GF4+/x4fH4J9EIIx5NAH4bUohdCZAsJ9GFILXohRLaQQB+G1KIXQmQLCfRhdNqpG5leKYRwNgn0YViBvkxq3QghHE4CfRgyGCuEyBYS6MOwUzf5EuiFEM4mgT6M+1/ZAcCIkvw0t0QIIRIjgT6EV7cdZOuBTkCmVwohnE8CfQj/fP8AAC989bQ0t0QIIRIngT6E1TsOMX9qDUeOLkt3U4QQImES6IM0d/WyeX8HJ0+pTndThBAiKSTQB3lzxyEA5k2uSnNLhBAiOSTQB3l5SxNFrlyOGVeZ7qYIIURSSKAP8PoHB3l07R4WTKvBlSu/GiFEdpBoZlr3YQuf/e2bAFx35tQ0t0YIIZJHAj2wv83Nl/64DoDvfvJoZo+XtI0QInsc1quBOtxentu0n1//6wO6PD6eu3EB08eUp7tZQgiRVIdtoN+8v50v/mEduw51A3DvZ46VIC+EyEqHRaDf0dTJn9/cTYErh9e2H2L9nlYAXLmK2eMrueLkiVw4uy7NrRRCiNTIikDv92uUgrYeL1pDcUEua3e18Oc1u1m/u5WPWnsGfc/E6mK++8mjOePIkWlosRBCDJ+sCPQn/XAFTR2esPcfM66C686Yytz6ETS2e5gxVlI0QojDh+MD/SvbmgYE+bEVhextc1PoyuHXnzueU6ZUU5DXvx1gTWlBOpophBBp4+hAv6+th8sfWAPAH685ieMmVMqOUEIIEcTRUfHB13fZt+dPq0lfQ4QQIoOlZMGUUuo8pdQWpdR2pdTSVPwMgKtOqeeso0ay7paFqfoRQgjheEnv0SulcoFfAR8DGoC3lFJPaa3fS/bPGlNRxO+uOiHZTyuEEFklFT36E4HtWusdWute4BFgcQp+jhBCiCikItDXAXsCvm4wjwkhhEiDVAR6FeKYHvQgpZYopdYqpdY2NTWloBlCCCEgNYG+ARgf8PU4YG/wg7TW92mt52qt59bW1qagGUIIISA1gf4tYJpSapJSKh+4FHgqBT9HCCFEFJI+60Zr7VNK3QC8AOQCv9Nav5vsnyOEECI6KVkwpbV+Fng2Fc8thBAiNrLDlBBCZDml9aAJMcPfCKWagA/j/PYa4GASm5NOci6ZKVvOJVvOA+RcLBO11hFns2REoE+EUmqt1npuutuRDHIumSlbziVbzgPkXGIlqRshhMhyEuiFECLLZUOgvy/dDUgiOZfMlC3nki3nAXIuMXF8jl4IIcTQsqFHL4QQYgiODvTDtcFJsiildimlNiml1iul1prHqpRSy5VS28z/R5jHlVLq5+a5bVRKzUlz23+nlGpUSr0TcCzmtiulrjQfv00pdWUGnct3lFIfma/NeqXUBQH33Wyeyxal1LkBx9P6/lNKjVdKrVRKva+UelcpdaN53HGvyxDn4sTXpVAptUYptcE8l++axycppd40f8ePmiViUEoVmF9vN++vj3SOMdNaO/IfRnmFD4DJQD6wAZiR7nZFaPMuoCbo2I+BpebtpcCPzNsXAM9hVAOdB7yZ5rafBswB3om37UAVsMP8f4R5e0SGnMt3gP8O8dgZ5nurAJhkvudyM+H9B4wB5pi3y4CtZnsd97oMcS5OfF0UUGredgFvmr/vx4BLzeO/Ab5k3r4O+I15+1Lg0aHOMZ42OblHny0bnCwGHjJvPwRcGHD8YW1YDVQqpcako4EAWutVQHPQ4Vjbfi6wXGvdrLVuAZYD56W+9QOFOZdwFgOPaK09WuudwHaM917a339a631a63+btzuA9zH2fnDc6zLEuYSTya+L1lp3ml+6zH8aOAt43Dwe/LpYr9fjwNlKKUX4c4yZkwO9Ezc40cCLSql1Sqkl5rFRWut9YLzZgZHmcSecX6xtz/RzusFMafzOSnfgkHMxL/ePw+g9Ovp1CToXcODropTKVUqtBxoxPjg/AFq11r4Q7bLbbN7fBlSTxHNxcqCPaoOTDHOq1noOcD5wvVLqtCEe68Tzs4Rreyaf06+BKcBsYB9wt3k8489FKVUKPAF8VWvdPtRDQxzL9HNx5Ouite7TWs/G2I/jRGB6qIeZ/6f8XJwc6KPa4CSTaK33mv83An/HeAMcsFIy5v+N5sOdcH6xtj1jz0lrfcD84/QDv6X/Ejmjz0Up5cIIjH/SWv/NPOzI1yXUuTj1dbForVuBlzFy9JVKKaticGC77Dab91dgpBaTdi5ODvSO2uBEKVWilCqzbgPnAO9gtNma5XAl8KR5+yngCnOmxDygzboczyCxtv0F4Byl1AjzEvwc81jaBY1/fArjtQHjXC41Z0ZMAqYBa8iA95+Zx30AeF9rfU/AXY57XcKdi0Nfl1qlVKV5uwhYiDHmsBK42HxY8OtivV4XAy9pYzQ23DnGbjhHo5P9D2MWwVaM/Ne30t2eCG2djDGCvgF412ovRi5uBbDN/L9K94/c/8o8t03A3DS3/y8Yl85ejJ7GNfG0Hfg8xqDSduDqDDqXP5ht3Wj+gY0JePy3zHPZApyfKe8/YD7GpfxGYL357wInvi5DnIsTX5djgLfNNr8DfNs8PhkjUG8H/goUmMcLza+3m/dPjnSOsf6TlbFCCJHlnJy6EUIIEQUJ9EIIkeUk0AshRJaTQC+EEFlOAr0QQmQ5CfRCCJHlJNALIUSWk0AvhBBZ7v8DisXylW4S0EEAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Set parameters for learning\n", "alpha = 0.2\n", "epsilon = 0.1\n", "gamma = 1\n", "\n", "# Create and initialize Q-value table to 0\n", "Q = np.zeros(discr_vector + (env.action_space.n,))\n", "\n", "# Just to store the long-term-reward of the last 100 experiments \n", "scores = deque(maxlen=100)\n", "lrews = []\n", "\n", "for episode in range(1,3001):\n", " done = False\n", " R, reward = 0,0\n", " state = d.Discretize(env.reset())\n", " while done != True:\n", " action = choose_action(state, epsilon) \n", " obs, reward, done, info = env.step(action) \n", " new_state = d.Discretize(obs)\n", " Q[state][action] += alpha * (reward + gamma * np.max(Q[new_state]) - Q[state][action]) #3\n", " R = gamma * R + reward\n", " state = new_state \n", " scores.append(R)\n", " mean_score = np.mean(scores)\n", " lrews.append(np.mean(scores))\n", "\n", "plt.plot(lrews)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Inestability can be produced because we had not good enough discretization and/or large alpha values and/or large epsilon values. \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Not bad. However, in RL results depend a lot on randomization, so it's a good thing repeat several times the same procedure" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "*** New experiment!\n", "*** New experiment!\n", "*** New experiment!\n", "*** New experiment!\n", "*** New experiment!\n", "*** New experiment!\n", "*** New experiment!\n", "*** New experiment!\n", "*** New experiment!\n", "*** New experiment!\n" ] } ], "source": [ "l100rew=[]\n", "for _ in range(10):\n", " print('*** New experiment!')\n", " # Create and initialize Q-value table to 0\n", " Q = np.zeros(discr_vector + (env.action_space.n,))\n", "\n", " # Just to store the long-term-reward of the last 100 experiments \n", " scores = deque(maxlen=100)\n", " lrews = []\n", "\n", " for episode in range(1,3001):\n", " done = False\n", " R, reward = 0,0\n", " state = d.Discretize(env.reset())\n", " while done != True:\n", " action = choose_action(state, epsilon) \n", " obs, reward, done, info = env.step(action) \n", " new_state = d.Discretize(obs)\n", " Q[state][action] += alpha * (reward + gamma * np.max(Q[new_state]) - Q[state][action]) #3\n", " R = gamma * R + reward\n", " state = new_state \n", " scores.append(R)\n", " mean_score = np.mean(scores)\n", " lrews.append(np.mean(scores))\n", " #if mean_score >= 195 and episode >= 100:\n", " # print('Ran {} episodes. Solved after {} trials ✔'.format(episode, episode - 100)) \n", " # break\n", " #if episode % 100 == 0:\n", " # print('Episode {} Total Reward: {} Average Reward: {}'.format(episode,G,np.mean(scores)))\n", " l100rew.append(lrews)\n", " \n", " " ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "C:\\Anaconda\\envs\\py35\\lib\\site-packages\\seaborn\\timeseries.py:183: UserWarning: The `tsplot` function is deprecated and will be removed in a future release. Please update your code to use the new `lineplot` function.\n", " warnings.warn(msg, UserWarning)\n", "C:\\Anaconda\\envs\\py35\\lib\\site-packages\\scipy\\stats\\stats.py:1713: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.\n", " return np.add.reduce(sorted[indexer] * weights, axis=axis) / sumval\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD8CAYAAACVZ8iyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmcZHV56P/PU3tX7+tMT/eszLAMMAwwbKJGQVFQAxoViFGiJHhFEnNN8otJbtQkV7Mar14VA+IV3IC4oqIGAUWUbYBhGJgZZp/pmZ7et9qrzvn+/jinaqr3qu6qXp/369Wvrjp1qup7uqrPc77b8xVjDEoppZY3z3wXQCml1PzTYKCUUkqDgVJKKQ0GSiml0GCglFIKDQZKKaXQYKCUUgoNBkoppdBgoJRSCvDNdwEAmpqazLp16+a7GEoptag8++yzvcaY5lK81oIIBuvWrWP79u3zXQyllFpURORIqV5Lm4mUUkppMFBKKaXBQCmlFBoMlFJKocFAKaUUGgyUUkqhwUAppRQaDJRSSqHBQCm1DNm2YSienu9iLCjTBgMRCYnI0yLygoi8JCJ/727/mogcEpEd7s9Wd7uIyOdFZL+I7BSRC8p9EEopVYhUxqZnJMlIMsNANDUvZRhJpIkkM/Py3lMpJB1FErjCGBMRET/wuIj81H3sL40x3xmz/9XAJvfnEuB297dSSs2btGUzGE/RNZTE4wGfZ34aRuJpi+7hJE1VQUJ+D3XhwLyUY6xp/xrGEXHv+t0fM8VTrgXucZ/3JFAnIq2zL6pSShUuY9lEkxkyls1ANMXh3ih9Eac2YNtOLWEonsYY53SWtmxiqQx9keSs3ncoliZj2QDEUxZDsTTJjAXAcCLNYCyNMdAzkqQ3kiKVsWf1fqVSUKI6EfECzwIbgS8aY54SkQ8BnxKRjwMPAx8zxiSBNuBY3tM73G2dJS25UkpNoXskSSxlkcxY2JOcb4/1x/B5hfpwgFTGZjCWRgQ8ItRXjr5it2xDNJWhJuQftT1j2QzG08SSFuGgl5NDCWor/NRU+DnaFwNgVV2IIZOmeziJybuUjqcsXukaob2+Yt5rCAXVk4wxljFmK9AOXCwi5wB/DZwJXAQ0AH/l7i4TvcTYDSJyi4hsF5HtPT09Myq8WtwWyhWRWnos2zAYSxNPTR4IAIyBdMbQPZxkMJbObTs+GB/Xrj8UT9M5mKBzKD5q256TI3QOJnKPGwODsXQuEAB0DiXoGhodCPLL0DU8u9pIKRTVaGaMGQR+CbzZGNPpNgUlgf8HXOzu1gGszntaO3Bigte6wxizzRizrbm5JOm41SKStmxe6RqhZySJMc7IjmzV2ranaoVUS5kxpiQXCYOxFNYsvkfZZpx8/VGnSac/mqJrOEF/NEV/NDXhCX6i15tKKmPTORTPffeTGYtE2so1Yc2FaZuJRKQZSBtjBkWkAngD8C8i0mqM6RQRAa4DdrlPeQC4TUTuxek4HjLGaBORGmXA/SfqGk4wEEuRTNsEfB48AiKwsaV6vou4rESTGSqDc7+8Scay8Xk9GGPoi6aIJjN4PUJ7fZhIMkPA6yHgK76jtxTDRiOJDMf6YzRWBQj6vMRTTru/bUP3cBKR6U/yxegdcfozGioD7OtyumlXN4SprfCTzFgEfd7SvdkECvn0W4G73X4DD3C/MebHIvKIGygE2AH8D3f/B4FrgP1ADHh/6YutFrvhxKkqeTLtXAnmXxHGUhnCgQWx9tKy0BtJEg54ca7t5oYxho6BOOuaKokkM3QOJgDweKC9HvoiSWwD65sqi3pd2zbE3BP3bA3G0gzF0zRUjm/PL8dFe1/EqdFkXzuWylDh99IbSdFWV1H6N8wz7X+bMWYncP4E26+YZH8DfHj2RVNLVSyVIZGeuingxGCcqqCfkN+D1yN4REp+5Zq9KlXOcMeUZZf96jNfNGUxksjQPZIY1ato23ByKMFw3GmzT1s2/iI+p87hRElP1MaQG4VUbsbAQPRUrWYwliZjGSLJDI2VAfxe5/8BTl1QlYr+J6g5ZYyha3jijrR88ZQzOehYf5wjfTH6ZzhBKG2NHj6Y7+RwYkavudSkMjbpjCE5xx362aacrqEkg2OadfLb64u5yk9bNv1zdOKeCxnL5ALCod4oI24AyFj2uD6N2dJgoOZUMmMTSRQ3+9IYiKYyJDMWg7HC/tGzJ//+aIpj/bHcSJGs4USagWg61w48m87G2UpmLDoGYtPvWCYD7t90rkd3pfPeLzlFTTGSzJBITx4QRhJpBqIpbNsU/d1aTDKWYcD9Hg+5Q1lLSRtl1Zya6p96KumMYV9XBGOgIuCdtjnjaH8MY8A2TvvrUDxNXdhPPG0RS50KKn3RJCu9IfZ1RzhjRTUez9y1mWcNRJ3AFPI7s1LBufIbTmQmbKsuJWNMLlAmMza2bebsb5CZasxnnsFYipFEmvVNlSTSzgSyioCXFTUhuocTuWGZadsmkVraw5VjKWcSXXeJawWgwUDNIWPMtH0FUz/f+R1PWQS8HizbTNjmn7HsXHtz1kgiw8He6LirqYFomljKImMZeqNJWqpDMyrb0b4YzdVBKgLFt7lnx7P3RVJUBX34vR6O9MdIZexcMIilMmRsM27C02xFkplcjSCSyHDcjrOiJjSjETzFGIqlC/4u2LbTKXxiMJG78s/+zbrzxud3L4Cx+uVm286chYxV+pqsNhOpOdMXTdEXnf0/bCSZoTeSosu9OhqKpTkxGM/VOlLWxCeZyarV2SaK7uEksVSm4LHdxhiOD8bpHnYmHI0ki+/Qy28CSWVsDvVGOeQGrYxlcmWJJJ1hjqUWz6upZWfgzkU2z0gqU3Qnb34TkDHjT/7GlGeEz0IztsmzVLRmoGas2FEeQ/H0lLNBC+WkDHDyu6QyTv4ZY5wT28qaEB0D8elfZALZUSM+rzN6aUXNqVqCZZvcKI6s4USG/kiK7GjMqdq9J5K2bA71REdty1iGjHXqBJ3M2PRGnA5324b93RE2tlRh204gWt0QLvIoR5uonyCSzNBcHZzV604lmXHy9aiFRYPBMpdIW9jGFD2mv2ckSSJtFXUymml/wVj5V4D5V4uxpMXhvuisAk72qksEwgEvfq+H3kiSobjTZp39O2VnoWbLA06way1wuGp21vV09nc7/STZgBNPWRzpi5KxDbGkRcifnNWJe6KmGmfor0XIX7phpqmMzUAsxYqaEF1DyXntsFcT02CwBBwfjNNcFZxRO+9IIkPGtosKBom0xckh50S4sraw2sFgLFWSWsF0SvUexsDh3tHNMgd7ojRVBRlOpEll7HFNEsY4V9WFJBwbjKVzf8PpypH/GxjVH9I9kqA+7C96voRlO2kfJgrQtg37uiKsb66kqgRzOzKWzfHBONFkhnDAm8vgqRYW7TNY5IwxDERTueGBhXKaVgyJtMVAND3uH3Qols6NaZ7ouVnDBVzdxlIZjvXPrOlmIcnmq0mmxweCrOmGZ2ZTJPdFp59rUQjbhoT7nulJ+krGSmYsIskMB3sjU5ahZyRJKmPPOqVztuPXGDjSF5vVIAJVPlozWOSiKQtjnCt8ryeJR2TccMSRRJrqvFEoxhiODcSoDvlJWzaW2+SQHa6ZsWyOup2VbfUVNFQGMMYwnHCu7HrzJvX0RVPUhwMTDkeMpyxODMULPkktBZN1Xp8YjNNQGeDkUIJoiceH90dSeMSpbbTWhqZMKWHbhs7BBB6RaWtRkUSGQ71RUhlnmOuqulDRM5Qt24yaKbscOngXKw0Gi1z2yjyeymY5hOqQMzzx5FCCkN/DyeEE6xo9uTbgnkiSdMYQS2Zys07jaYt69zXzT1bH3c5Yv1c42hfD7xPSmVP/0cm0PeGQTGMMPSPJkk+MWejGXvVm5wv0RVIMxtLYZTgbDsXTDCfSub6F1trJc9gMxFKMJDIUmoIof9hpXyTFqgny44y92ADn83cuVIwGgEVCg8EiF0uNHm4HTkdq0O+cjLOZFUcSGYI+Z2x+dkhe/olrMJbO/aPH0qPH6EfzgkZ+IDj1uIVdeWqyUsaySblpIJabeMqicyhOXUWAaCozql+gnJ2m2c++P5piRXVo0olj2UA/kxP0QCxFw5j8OJZtOOLOsWipDuZqJdGUxeHeKLUVpZ0XocpHg8EiNtkkrkgqw1DcuPs423ojSQZjKfxez4QnAss2xFMWPq9MOGFrqhNZJJGhYyBOe30F8bRFfzRVspFDi1HvSIrekRQez9w3i9i203RXW+EfN6DgxGB8VsnNbBsO9ERora3INUVm+wKyK3itqHECwmAslVvkRS0OGgwWsZ7IxJ2QEyXqcsavTz0D+Pig048wthO0kCvaoXialGUhIsuuaWgyczF6aiInhxL0RZO014fpHIyzobmKSDJT8EIsU7HtUxPV0pY9Ktlfz0gSn1eoDvk0CCxCGgwWKWPMjDN5TiaesomnZj5yJL7E88IsJumMyU1oG4yl6BwqXVrnRNpyR0SNX8y9azhBb0QWfD9B13CCPSdHeM2mJjxzuIbDQqbBYJGKpawJ2++VGuvEYGlTdcfdjuGJZlxn8wgtZLYx/MOPX+ZofwyfR7h8Y9Ocvn8ibY3qd1kodJ7BIjXXueeVyjJm8dYCdx0f4tov/iY3dPr5Y4NzXoY//NrTXPel39AxEMsl3FsINBgsUtEF9CVSarH4zEN7c7cv29DI4/t7cunMTwzGy76uRPfwqXkmH/rmc9x455MLJjWHNhMtUvFlPFpHqWIZY9jZMURvJMVrNjXx0Teczv6eCE8c7OO9X32ayoCXqLvQUVNVkM9dv5WaGQyLTaQt9pwc4bz2WmzjjL763vPH+Z3Tmwl6PXziRy8BcF57LS90DAFw3Zd+w71/fGnJl3UtlkyXrldEQsBjQBAneHzHGPMJEVkP3As0AM8B7zXGpEQkCNwDXAj0AdcbYw5P9R7btm0z27dvn+2xLBtDsXSumquUmlrasvnb77/I7pMjANzzgYupd/NHPXO4n3/48cvjnnP+6jo+/tbNo3I+/XpfD08f6udPrtg0atju5x5+hV/s7mZTSxX7uiMANFQGaK4KsrdrZNxrn7Gimn9/13lEEhlu/MqTue3f+qNLxk3em86W1XXPGmO2FfWkSRQSDASoNMZERMQPPA58BPgo8D1jzL0i8mXgBWPM7SJyK7DFGPM/ROQG4O3GmOuneg8NBsXpGk4si4U8lJqNx/f38p1nj3EgL034By5fx9vPbx+1XyyV4alD/VQGvJzWXMV924/x010nqQ/7ufN92wj6vESTGW6489SJ+75bLiUc8PH+rz09Kj3LRNY1hjnc51y83fb6jbzp7JW5xwZjKT75o5c40BPlbVta+aPXbMAjQtqySWZsqoI+p7M+Y4/LImsbw9Y19SULBtPWS4wTLSLuXb/7Y4ArgN93t98NfBK4HbjWvQ3wHeALIiKm0BVD1JQyll10Ujqllpu+SJJ/+dme3H2PwPc+dPmEI3jCAR+vP6Mld//W123E6xF+vLOT//vIfl53evO4E/71dzzJZRsac9v/x2s3sGV1HV4RVtVV8JMXO3l4dxd/e81ZNFYF2X64nzNX1lAVGn3KrQsH+D/Xn8+//nwPP9rZyY92dvKRKzfxuYf35fb549es585fHxpVowH49//eSykV1EglIl7gWWAj8EXgADBojMn2YnYAbe7tNuAYgDEmIyJDQCPQW8JyL1t90ZQOKVVqGv/vt4cBeONZK3j9mS2c21Zb1PNvec0GHtnTza9e6eFXr/TQVldBW10Ft7/nAj5y3w4O9UZ54mAfAN+8+ZJx/QtvObeVt5zbmru/bV3DlO/3od85jV/vc06R+YEA4M5fHwLghztO8N5L1+L1CKmMzTOH+4s6pukUNJrIGGMZY7YC7cDFwFkT7eb+nmjw7Lizl4jcIiLbRWR7T09PoeVd9nRmp1JTs2zDkwf7qAx4+dMrNxUdCABEhE+87Wza6518XccH45y9qgYR4dNvP5d3XdjOra87jc/fcP6MOprHqg75+cGtl+fuf+q6c/jWH10yap/vPtfBB772DAA/3dVZ8lTgRXVfG2MGReSXwKVAnYj43NpBO3DC3a0DWA10iIgPqAXGhTBjzB3AHeD0Gcz4CJaR7KIqSqnJ7e0aIZmx+ZOrzpjV62xureH291zI80cH+K9nO3jXhasBqAr6eN9l60pQ0tG8HuHzN2xlz8kRzm2rRUS47fUbeejlLt5xQRv/9NM99MdSDMXTfPOpo2xsruJICd9/2pqBiDSLSJ17uwJ4A7AbeBR4p7vbTcAP3dsPuPdxH39E+wtmz7ZNWRZEny+WbXjiYF8upbN+RVSp/GZ/Lz6PcNG6+ul3LsD5a+r59NvPZWVtaPqdZ2l9UxVXn9Oay/76prNX8u/vOo9XndbEP157DgD/8dBe4mmLGy5eXdL3LqRm0Arc7fYbeID7jTE/FpGXgXtF5H8DzwN3ufvfBXxdRPbj1AhuKGmJl6lY2pq3xGfl8P999wVe6YqM2tYQDnDNllZ6hhP88Ws3FL2Qilrc0pZNXySVO+lGEhn8Pinqe3ByOMEDL5zg0g0NRa/rvdCd21ZLdcjHc0cHWdMQ5uJp+iGKVchoop3A+RNsP4jTfzB2ewJ4V0lKp3ImykS6WP38pZPjAgFAfyzFN550Kr5+r4cP/s5pc120ZenkUIK///FL/NmVp3PGyuo5e99IIsOR/ihnr6ollsrwtz/Yxf7uCB6BT7/9XD72vRcB+MGtE48Cmsj9248B8N5L15Wr2PPG6xGu2ryC7z53nC1uM1IpLa3QucANxdMYYwpaMD3fSCK9ZBaK+cqvD/LDF07QEA7wlZu24Xcn9Vi24bF9PVSHfPz9j17mxy928uMXOwFoqgrwd2/ZzIbmqvksetn0RZJ0DiU4ZwYdnaXwTz/bTcdAnHuePMynrjt3zt73tm8/R5+beffsVTXsdyds2YZcIABnCOVfvfnMaV8vbdn8Zn8vV5zRwpqGcHkKPc+u37YG2zBqpFKpaDCYIxnL5lh/DJ9XigoGibTF4d6l0VdgjOEXu7sA+JtrzsoFAnCuerJjvT/77q38z/t35B7rjaT4yH2n7v/L721hc2tNQe/ZORRn78kRXrOpedTV5StdIzz0chcfuHw9FQEvPSNJmqoCJb/amo4xhk89uJt93RHetHkFH379xjktw6HeKAfdSVkvnxgmksxQNQdpEbYf7s8FAoCXTgzzjvPbuGBNPd99roOOwTibW2tYURPi/u3HePwLj/ONmy+hKujj5c5hbGM4a2UNu04M8fUnjlAX9nNOWy2xlMVlpzWWvfzzpSLg5QOXry/La2swmCMDMWeN2nTGkMrY41ahmsxSWjHstwf6iKYsbn3daVM2R2xsqeILN56PxyMkUhYP7+nmJ24tAeDLvzrA528Y13KZYxtD2rIxBj5y7w7iaYuOwTh/cMlaLNvw8Qd2sdPNCxPweagIeLnvmWNcc24rH5ph09RQPE1V0Fd0WuJ93ZFcCoOfv9zFhesauGxDI/u6RuiJJHnVaeVNr/zJB5xcOX/3lrP4x5/s5sY7n+SKM1r4n288vWzvaYzJjaX/y6vO4Ac7jhPwebjx4jWE/F7OW12X2zeZsfiv7ccwwB/c9dSUr7v9yAAAF64tTcfxcqPBYA7YtqEveip9RCJjFRwMlkqq6qF4mn92Z4QW0vG1trEyd3vTimreeWE7L58Y5vhgnG89fZS3feFxAL5x8yX8aOcJ7nvmGGeurOaTbzubf/35nlwnWzah333PHOPRPd10j4xO4/HACydytx98sZMtbbVF5bc/OZzg2cP9fPmxgwS8Hu7/4GVFBYTth/vxCHz1pov4w689w6cf3E1TVZDeiFPOW193Gptba0b9PUplJJGmP5bivPZaLlx76jN5ZG83f3rlprLl29/bNcJgPM3ahjCvPb2Z157ePOm+QZ+Xr9y0jS/98gDPuif7K89s4eE93bl9rtu6Cq9H+O5zxzlnVc2oGqcq3LS5iebCUs9NNDaxXEtNkBU1hQ1TO9wbZSSx8NJVpzI2X3n8IH/4qnUFjdp44IXj3PnrQ3zszWfOajGRvSdH+IvvvFDw/g3hAP9w7dnc9u3nAWipDlJT4eejbzwdv9fDPz24m6P9MT777q38yb3P5553xRkt054Q/88vXhl1UgL49HXncG573STPGO3+7cf4+pNHaK0Nccd7t/H80QE+7l6pT+bqc1Zy6+s2MhhL8flH9nHteW2jrqSL8cTBPj794G7++R3ncvaqWo72x/j+8x38Ync3n3jr5mlnzRZjKJ7mSF+UtY2VfPT+HcRSFnfdtK3gET+WbXjqUB9b2uuoCvqIpTI89HIX15zbuqxP/nOaqG4uLOVgYIzhUG80l8McoCrkY33T9Fd6xhinfXSBVQ6ODcS49ZvPAfC605v582km98RSGT749WepCvm4/T0Xzvr9v/bbw5wYjPP6M5r5znMdXLCmnrNaa/iEeyIN+T1ctXklu44P8dfXnMXKmhCHep2F3Mcm+8q36/gQf/39Ux2Xl21o5KrNK/j+juPs7BjiHee3cdOr1iHAt54+yr3POCNXqoM+/u6tm/mb77/Im89ZyQdfO31TU/ZEDKMDSMdAjC//6gB/9obT2dc1wqd/umeqlwGc2apbCgxAWT/d1cmXfnkAn0e4/4OX5U6oacvmvV99ipU1If73teeOy6UzE2nL5gN3P8NgLE1d2M9gLM3H37qZi0o8NHI50mCwiEyUbtrnFdY1VhLwTb30XTJj8crJ8UMw59u//Xwvj+1zUoisrq/gS5Oc4OMpi86heK7z99rzVvFHr9lQtnL1RZIMxtOcNotRR13DCbpHknz3uY5cs8RkVtWG+L83XpBr8vu3n+9h+5EBvnHzJZNeraYyNt0jCT7kBtPsVflUBqIp0raNV4Tbvv38uNWxQn4P999yWUEdz8PxNF9+7EAuD042nXK+ux4/yA92nGBLWy3/eN05s1oj2BjD3/5gFy8eHxq1/YEPXz7nnfVLUSmDgfYZlFEslZlw3YGMZdjfHaG+0k97fZhE2mIwlh43w3G++gsSaYufv3QSn0d4zaZmair8ZCyblGVjG3jyUB+Xn9bIqroK/uvZDrYf7h/XpJCxbN59xxO5+6c1V/IHl64ta7kbq4I0VgVn9RorakKsqAlx+ooqdnYM8cTBPjwivPvCdv7XD3fROeSsJ7yyJsTnbzx/VN/PpRsaeWxfL0f6YmxsORWQ0pbNz3ad5MK19XzwG8/mtv/eBe0FjYqqrzw1+uxz12/lF7u7MMANF63h3mecGsr1dzyJR+ALv38BTVP8De56/FAuEFy2oZH3XTb+M/nA5eup8Hv59jPHuPaLvwEYlzGzUM8cHuDF40NcuLaev7jqDD70jWd554XtGggWIK0ZlIkxhle6ItPmEmqoCtAfSSECZ66sHrWYRsdAjIHo1PMLDvdG+ZN7n+cvrzpjyo64sbqHE9SFA/x45wm+81wHt7/nQmrdhFtff/JIbvIOwFWbVxBJZvjtgb7cts++eyuNVQHe/7VnsGyTO1lkv09/+Z2duYU91jdV8rnrty6JE0BvJElj5cRDUE8OJ/jje7Zz6+tO4+pzWjHGcLgvxj1PHM6NdMl625ZWbimgOWk6ibTFu/7ziVHb/u2dWzhz5fgg0x9N8Yf/7+lc1sipJnMZY/jEAy+NWiP4f75hE1ecuaLgshlj+N8/2c2hvih3/MGFo77bqjS0mWgRGEmki54fsLI2lBvrnsrY7D05fpWksT7z33v55StOk83V56zEI8Jbt7TSXh/Gsg0egc889ApvP78t13zys10n+eIv9497rR/d9mos2/D+rz2NbZxEXdk0vRPtC/DIni4++wtnmOD3P/QqvvL4oVHDQJdTc4Axhvfc9RQjiQzvOL+Nx/b15kYFZZdVPHNlNf/2zvOmeaXiDMXTPLy7i7ufOEx2Od0ffvjyUc07tjF8+VcH+Omuk3zmXeexvqly2o7XeMric4/so3ckyd6uEQT46h9eNKrmEU9ZfOvpo1y4tp6teR3Z33rqCN92+1TetHkFt12xqWTHq07RYLAIdI8k6BoqbjUyEWdSydqGMD2RJL0jU6egSFs2f3DXU6xrrOTlzuHc9pbqIO+5ZC2f/cUro/b/wa2X89ShPv5pkk7Jf3r7uRjgb77/In/15jN59cam3NUuOE0FP3rhBO/etjrXEWsbw7v+84kJa0DfuPmSXG1jufjJzhN8+bGDo7bd/Or1XLe1jVgqg0dkyk7s2frSL/fz010nAfiH3z2b7z9/nHdd2M6JoQRfeHQ/NSEfX7/5kqL7AY70RXMjsv78jafzOneC4Kcf3J27YLj+otXceNEabGN4x+2/zT13tiPI1OQ0GCwCx/pjM157wOMBY5yfqew4Nsjf/XAXf/eWs9jbFeGnL3Yykpx8GOr5q+vYdWKItGX4m6vPzI1Uuet927j5nlN/f79X+ObNl1IRcE5an394Hytqglx/0ZpJX/uj9+9gX3eEqqCPGy9ezYVrGmhzc8EvNz/YcZy7Hj/EFWe08Gdv2DSnNaN4yhrVVzPWne/dNuPsmx/73k5eOuFcdNx3y6UAvOcrT7GmMUwkkaF7JElthZ+bLlvL5x/Zz/suXcvhvhh/euVGTTpYJhoMFjjbNhzsjRJPlXf28O2/OsAvXu7im390Se5q86UTQ7m8Lr93QRuvOq2J6pCPW75+quPyj169nmu3tvH0oX6aqwOsb6riUw++zJMHnWUnLt/YxMcKyAUz1v7uCKvqQksuW+RiY4zhtwf6+I+HXmF1Q0VuDeB/+70tnFlgGo+JWLbhz/9rBwd6orxx8wp+u7+XaMri3995HqvqQvz+V07NEM6uCrZcmgjniwaDBe5gT4RYypr2yn42BmMp3vvVp9m2tp5PvO3s3HbbGP7vI/uoCfl5f14Ok2P9Me749UHqwn5ue/34KzXbGH65twefR7hgbf2c5KdR5WWMwTbwv37wIm8/v42L188+Z8/Ymkf+0NSBWIr3ffVpYGZzH+ZSyO8h5Pcu+pUDdWjpApZIW6MmmJXLz19y2oV/97xVo7Z7RPjIlePzyqxuCOcWx5iIR4TbyQ3hAAAgAElEQVQrzmyZ9HG1+IgIXoF/eseWkr1mRcDLBWvqeP7oIL93QTtXnnXqO1MfDuSWbixXKotSEHEGa4T8XjeT8Ny+f2XQi8/jWXCZiDUYlNhcXWk8ureHc9tqOX+NJuVSc+uv3nwmkWSGlurxfQ8LOQiAU75NK6pyI6nqKwO5tUK8HsE2puzBoSrko7kqSGDYQzxtEVkg6WY0GJTYXKxRPJJIc3wwzlWbCx/zrVSphAO+RdsvVB3yjRpS21ZXQUt1kO6RJFVBH9Fkhv5oqiwBQQRW1VVQW+FHRFhZGyKazOARGI7Pf0BYnJ/oApbMlL+JKH8yl1KqMCLQWDV+FrXf66Gtzhn5Vh300VAZYH93pKiA4PMKPo+zRGfI76EnksS2Iej3sLo+TDSVoTLgy43Qy6oM+qgM+hiIpugYiE9Z9rHlKUXeqFHHUNJXW+Zs25BIl79m8K8/2wswbU4bpZSjqTpAS3Vo2mYsj0cIebxsbKniQE9kVJLIkN9DdchPbySZOzF7PUJzdZC6sH9UjaMi4CWWsmipDiIi44LAWPWVAdK2swa0ZRuaq4MI0BdNkbEMaxvD4yaxhqd5zWJpMCihlFX+QDAQTRFPW7TVVRS8JoJSy1XQ76G9vqLoZq2Q30t7XZhjA84J2BhoqgpSXxkg6PPQMRBHBFbUTJwPqzrkpzpU3ITLluoQNW6wyaa4FxHiKYvqkD+XugacuUgNlcXniprKtGcTEVktIo+KyG4ReUlEPuJu/6SIHBeRHe7PNXnP+WsR2S8ie0XkTSUt8QI2F6uS/edjBwD4aBlXolJqKaip8LGxuWrG/Ru1YT8bW6porQ3h8UBd2Dm514X9rKwNsWlF1awTI44V8ntZVXtqsmZzdZBVdU5gWFUbIuh3TtmVAV/J13Eo5K+UAf7cGPOciFQDz4rIQ+5jnzXG/Hv+ziKyGbgBOBtYBfxCRE43xiyd9RsnESvzJLNkxuI3brK4DdpfoNSUGquCeGY5uink9xLyewn4PLkJdCJO01C5jC1zNsGfiFAT8tOTTha8OFZR7zvdDsaYTmPMc+7tEWA30DbFU64F7jXGJI0xh4D9wMWlKOxCFy9zzeAxNyHdba/fqBkglZpCyO8p6cTJYpt8yqUu7M+t211qRZ1RRGQdcD6QnXd+m4jsFJGvikh2wHsbcCzvaR1MEDxE5BYR2S4i23t6eoou+EJUzmGlsVSGzz+yn7UNYR1SqtQ0St2evlCE/F7WNITL8toFBwMRqQK+C/yZMWYYuB04DdgKdAKfye46wdPHDdIyxtxhjNlmjNnW3Fx4Hv6FKm3ZZKzyzVb51587I4iuO79N870oNY2FciVfDuWoFUCBwUBE/DiB4JvGmO8BGGO6jDGWMcYG7uRUU1AHsDrv6e3AidIVeWEq56L1tjG8dGKItroK3nCW1gqUmkpVyKcj7WagkNFEAtwF7DbG/Efe9ta83d4O7HJvPwDcICJBEVkPbAKeLl2RF57surblsrNjiETa5oaLVk+/s1LLXNMEE8vU9ArpYbkceC/woojscLf9DXCjiGzFaQI6DHwQwBjzkojcD7yMMxLpw0t5JJExhn3dI6Mmp5Tad549Rm2Fn8tOm33WyYXE6xEse/6z5qqlw+uRJd1EVE7TBgNjzONM3A/w4BTP+RTwqVmUa9GIp62yBoIfvXCCFzqGuPqclUtqgZCm6gAra0Ik0jYeD6Qtw0DUWQs6mrTmJMeTOqUi4C37+huFCPk9uVn8Qb+HtGUX9f9VGVw6/yNzTWcgz1K509D+98snaQgH+EDe2gSLmccDq2orqHdHe2Q7w4I+ckMBM5bNnpMjGOOcEDwiGDM3qT6Wo5Dfw9rGMEf6osRTc/c3FnEmVXUPJxGB9voKQn4vw/E0IkLI78HrEY4PxAv+7Ct1HY4Z07/cLBhjytpxHEtlONIX48aL15R13dy5VBX05QLBZHxeD6sbwgS8HkSc4XS2bYikMiRSFt0jyVlnlfT7hMqAb0EsblIR8NJaG+L4YJzkHAc8EWetC7/XQ304gDGpOQm62Qye9WE/tRV+4imLurDzvRj7XV/TGGY4nqFr2OmXW1ETwmDoHnaW2Qy6cwq6h5PUaBPRjGkwmIXhRKas/7z7uiMYnNWkloLaCn/B6+/WVoz+p/Z4nNmXNSE/kWQmN9s7GxTyszp6PU5isEgik0sidrAnimUbRCDo89BUFaQi4GU44QSD9vowJwbjMx4ePFFWyUI1VwepDPpY0xDmUG+UjOWUM//4yqUu7M+dfBurgtSFAxztj5U1x77H4wSg6qDPrQF4p7zYCfq8NFc7s4Crg77cDN2GcGDU5Mu1jV4ddj0LGgxmoXNo8pSzYxljGE5kyFh2wflMXj4xjACnr1yYwaCYE6AItNaFSpJP5VQSL2cxoYbKAB4RoskMKcumoTJAMmOTSFvUVPgI+b2srA0R9Hnwez2jhh2eubKGjG3nUg93DMRJFNEPJOKkQF7TEKZ7JFF0Xvq6sJ8aNxVxyO9lQ3MlfZEUjVUBvCIcG4gTDnjpGUkCo//ePq/Mem7L2Fm6Xo/QVBUoeTDweKA66Gconqa9PjyjK/ixFwhjZ+FrIJgdDQYzMJJI0zmUIJ0p/B/x/mc7+MaTRwD4i6vO4HdOn36i3Z6Tw6xtDC/I9YibqgOE/T6ODcSmDQgeD2xsqSpZYq38duH8JGQB36nmJ7/Xw1l5i79PNiPV6xG8nmy/hZf1jZUYnL6g41Pklwenrb29Ppzr92ioDBQVDOrCflaPmU0a9HlZVXcqUVl2zYq6sJ+A18PxwTiDsTR+r4f1TZUMxdN0jySKHsQQ8HmoCvkmHHlTFfTRUuO05ZeCiHMcIZ+X2go/1Qvw+6w0GBRtJJHmSN/0J8B88ZSVCwQAj+7tnjYYpN1O1NdsWnizsysCHlrdzIrRVIA+N63uWNl2+ZqQf9GMhMo1QVQGCPg8RJMZLNsQDnjpHkmSytgY45ycV9SERtUyqkN+qkK+gq6qm6uDBTeZAbm/X1tdBU1VQTwiBHwemquDeD3CicE4Pq9Me4Ei4uTO93uElkmSnYkIK2pCWLaZ9LMtRktNMBe0a8Papr9QaTAo0FAsTdDvYTBW/ALa//2ys3j9v71zC7/e18tPd3UST1m5K8qMZZO2zKhp5ruODxFLWVy0bmGscSziLKYRTVqjqvittSEylhk3qirgc0aoLOaO76qgb1StrC4cYCSRxuuRSdMi10wQDESc4DIYS9NSE6S2wo9vhtk0s23s+RoqA4QDXnweoT+amrKDvb4ywKoCg9BKNyB4PJLLoz8T9WGdBLYYaDAo0HAiTTDjKXooaSpjc//2Y5y9qoYzV9ZgG3jghRP86pUetq6uoy7s5/o7nsA28M4L2rnpVesAeOpQPwGfh/Pa68pwNNOrC/upqfCTzFiMJDIE3BE+A9HUqLZbEaG1LkQo4KF7OEk44HVPTkszJcB0E5pqKvz0R1MkM7YzciocyGWZbK0Nla1dOxsgWmpCpCx7wouWkN+TW3+3EB6P5JqxqkM+jhZZIxZxLgpKnXdflYcGgwJFkhmGE6aofwbLNlx/xxNkbMPvXdAOwFkrq2mtDfHFX+4ft/93nuvg7LYazmuv49f7erhgTV3ZrqyDfg/JtE1NhY/GqiAdA7FcE0PQ78lry/ZTFXSCATDhsFC/10NLdYj6cGDZ/+P7vR42tlRhG2fQQH4n51x1cDqLvIfoHIqzoibEkT5nta71TZUzTn1eE/LTVBXMdWTna6kJMpLIjJq0VhHwsrqhoqyjklRpLe//3ALFUxYZyxTdSfftZ46ScdMtbFvrNPeICB994+mjmh+qgj4+cuUmAP7+Ry/zjtt/y3Aiw1WbVxb0Pu31FdSF/dSF/RPOwGypCRLyn/qoayv8rG+qZFVdiObqIFVBH015I5xaxizcEQ74CjqJLPdAkCUieD0yb2tOiNufsLax0lm+sb6CTS1Vsy5Pc3WQlprgqIXYK4NeVtSE2NhSxaq6UG5IbHN1kKDPW/KVwFT5aM1gGiOJ9IwWrRmIpfjJzk4A7rvl0lFXhWeurOHbf3wpactm78kRNrZU4fMI9zxxmAF3ElRrbYjzV0/fRFQVciZx1VT4EZyq/b6uEZIZZ4hl0OehoTJAbYWfvmiKVMZmTaNz1d9QGciVq6kqSCxp4fGgE3eWmFLNyvV6nI7lgWiKSCKTm0Gc1VgVxOBcPNWE9NSy2OgnNgXLNpwcShQ9I9MYwz/86GViqQz//I5zJ+1s9Hs9nNNWm7t/zwcuIWPZHOyNcnqBE80a3M45b16HZGNVEI+Qm9EJTptyW96QRRjfbJENEkpNpTrkY01jmKDPM64Zs0lrAouWBoMpdAzEZjQ1v2Mgzv6eCB+4fB1nr6qd/gl5fF5PwYFAhFFV9qylusqTWhh8Xg+1FdokuNToJzqFaLL45iFjDD/a6azlc+HahlIXaZTGqsCoGoFSSs2UBoNJJNLWjHLtP76/l5/ucuYVtNdXTLP3zIlAS3Xhk5aUUmoqGgwmMZPU1MYY7nnCmWn8/letw1PGoYSVQZ/WCpRSJaPBYAK2beiPFj/j8sRggpPDCV67qZl3uPMKyqWyTItiK6WWJw0GE+geSc4oG+ThvigAbz+/rdRFGkeX9lNKlZIGA5cxhs4hJ31xNsd9sQ71RfEIrGko7xDN2gr/qDxGSik1W9MGAxFZLSKPishuEXlJRD7ibm8QkYdEZJ/7u97dLiLyeRHZLyI7ReSCch/EbCXSFgd6ovSOpDjaH5vxgjUHuiO01YfLnpNHMz8qpUqtkLNWBvhzY8xZwKXAh0VkM/Ax4GFjzCbgYfc+wNXAJvfnFuD2kpe6xKLJU3lVZhoIbGPYfXKYs8q8EE11yDdukQ+llJqtaYOBMabTGPOce3sE2A20AdcCd7u73Q1c596+FrjHOJ4E6kSkteQlL6GBEqyDe6w/RjRpjVpQpRxaanSGp1Kq9IpqzxCRdcD5wFPACmNMJzgBA2hxd2sDjuU9rcPdtiDFU9aobIszta87AsAZZawZVAa9k6a2UEqp2Sg4GIhIFfBd4M+MMcNT7TrBtnFDc0TkFhHZLiLbe3p6Ci1GycVSpUmxe6Qvit8rrKotz0Qzj+fUEohKKVVqBQUDEfHjBIJvGmO+527uyjb/uL+73e0dwOq8p7cDJ8a+pjHmDmPMNmPMtubm+VvaMTODWcYT2dcdYXVDuGwTwaqDhS9KopRSxSpkNJEAdwG7jTH/kffQA8BN7u2bgB/mbX+fO6roUmAo25y0EM20wzhfIm2x9+QIW9rKtyqZjiBSSpVTIQ3QlwPvBV4UkR3utr8B/hm4X0RuBo4C73IfexC4BtgPxID3l7TEJZayZh8MfvVKDxnbcMn68iSm83qE6hLlpFdKqYlMe4YxxjzOxP0AAFdOsL8BPjzLcs2ZZGZ2nccnBuN84dH9tNdXcPaq8owkqgv78WgeIqVUGS3rGci2XfxSlmP9YMdxAN5ybmtZ2vSd7KQ6nFQpVV7LOhjMtvPYNoZnDg9wyfoG3rplVYlKNVo44J23tXSVUsvHsj7LzGS9gnz7uiL0RpK86rSmEpVoPM1BpJSaC8s6GKRn2Ub0mwO9eAQuWFO+UUQVfg0GSqnyW9bBYCZpqrNsY/jFy1286rSmUQvPl5rOOFZKzYXlHQxmUTN47JUeRpIZLt3QWMISjeb3SdkzoCqlFCzzYDDTCWexVIbvPtfBmoYwr95Yvv6CsF9rBUqpubGsg0F6BhPOLNvw4W89z+G+GDdctLqs6xBXhzQYKKXmxrIOBjOZffzDHcfpjSTZ0FTJazaVL6eSCLpugVJqzizbYGCMKaoD2bINI4k09z5zDJ9H+PhbNxf9npVBL03VE3c2j61h1FborGOl1NxZtu0QyYyNKTAWGGP4xAO7eKFjCIDPXb+VxqriZgVXhXy5FNQjicyo/goRZx2EI31RjPt+9ZXlG6GklFJjLdtgMFjE6mbPHh3IBYLTV1Sxobmq6Pdrzksp0Vob4sRgAoNTO2mvr8DrEdrd9ZNt22itQCk1p5ZtMIinC09Q98iebqqCPt5xfhuXnVbcUFIRaKwKUJWXdbQ65OeMlX5s25CxTW74aPa3BgKl1FxblsHAtg3R5PQrnEWTGW6480nASUT3rm2rp3nGeG11FZM2+Xg8QkBP/EqpBWBZBoNC+wv+/scv525ftXlFUe/h8UBr7eSBQCmlFpJlGQwSBTQRvdAxyO7OYc5eVcOfXrGJVXXFrW1cHw7QoIFAKbVILM9gUMCCNk8f6gfg42/dPKP8QDphTCm1mCzLeQbR5PTB4NkjA1ywpm5GgaCxKkClJphTSi0iyy4YpC2beGrqYHByKMHxwTgXrKkv+vVFnKGjOiJIKbWYLLtgUEh/wSN7ugBmlJG0uTpYluUvlVKqnKYNBiLyVRHpFpFdeds+KSLHRWSH+3NN3mN/LSL7RWSviLypXAWfqWRm6nxEacvmZy+dZNvaelbUhIp+/ZAuRqOUWoQKqRl8DXjzBNs/a4zZ6v48CCAim4EbgLPd53xJRBbU2XG6TKU7jg0yEEtz9TmtM3p9XZlMKbUYTRsMjDGPAf0Fvt61wL3GmKQx5hCwH7h4FuUruanWMLCN4etPHqE65OP8GSxlGfR7dDEapdSiNJsz120istNtRsr2tLYBx/L26XC3jSMit4jIdhHZ3tPTM4tiFGeyZqJkxuKP79nOod4oV57Zgt9b/J+mJqQpp5VSi9NMg8HtwGnAVqAT+Iy7faKe0wnn+hpj7jDGbDPGbGtuLt+6APls20zaTPSlXx6geyTJlvZa3nvpuhm9fpXOLVBKLVIzOnsZY7qyt0XkTuDH7t0OID+BTztwYsalK7GRRGbCNBR9kSSP7Onmuq1t3Pzq9TN6bREIa3+BUmqRmlHNQETye1ffDmRHGj0A3CAiQRFZD2wCnp5dEUsnmpo4Od1PXuwE4A1ntcz4tcMBr84tUEotWtPWDETk28DrgCYR6QA+AbxORLbiNAEdBj4IYIx5SUTuB14GMsCHjTGF54ous4nSVqctm5+82Ml57bWsaQjP+LWrtb9AKbWITRsMjDE3TrD5rin2/xTwqdkUqlwmGkn0/NEBYimLt2xZNePJYl6PaFI6pdSitmzGQdq2wbJHdxgYY/jm00dpqgqybW3xqSey2tyVypRSarFaNsEgNcEookO9UQ72RHnbltYZDSUFZ92CGh1FpJRa5JZNMJioieilE8MAvHpj04xfty4c0FxESqlFb9kEg7Q9QTDoHKa5OkjLDHIQZem6BUqppWDZBIPUmJnHxhh2nxjm7NaaGb+mzytUBzUYKKUWv2UTDMamru4eSdIfS3HmLIJBOODVJiKl1JKwLIKBbRtiYxa02d3p9BectbJ6xq9bV6HDSZVSS8OyCAbR1Pg0FHtOjhDye1jbWDmj1xSBmgptIlJKLQ3LIhhMlKl0d+cwp6+onvH8gLqwX5uIlFJLxrIIBmM7j2OpDIf7opw1w/4CZ53jilIUTSmlFoRlEQzGdh6/0hXBNrB55cyCwdrGsM44VkotKcsiGIydfby7cxgBzphB53F1yKdJ6ZRSS86S7wE1xpDOjO493tkxyLqmSiqLmCMg4ix2v7J25hPUlFJqoVryNYOxnceWbXilO8K5bbVFvU5DZYCNLVWEdAEbpdQStOSDwdhlLo8PxkllbNY3FTektLFK5xQopZauJR8Mxo4keu7oAADnrCq8ZlAR8BL0aY1AKbV0LflgMHZ1s2ePDLC6IVxw278INFcFy1E0pZRaMJZ8MMgfVmrZhr0nR9hSRH/B+qZKasM6ekgptbQt6WBg24ZE3joGx/pjxNNWwUNKV9QEixpxpJRSi9W0wUBEvioi3SKyK29bg4g8JCL73N/17nYRkc+LyH4R2SkiF5Sz8NOJpa1ROYn2nBwB4IwVhQWDel3XWCm1TBRSM/ga8OYx2z4GPGyM2QQ87N4HuBrY5P7cAtxemmLOTCyVGXV/98lhqkM+WgvoL6gIeGe8FKZSSi02057tjDGPAf1jNl8L3O3evhu4Lm/7PcbxJFAnIq2lKmyxEqlTTUTGGHZ2DHJuW21BCeY0I6lSajmZ6aXvCmNMJ4D7u8Xd3gYcy9uvw902L/JHEvWMJOmNpAqebFajKSeUUstIqdtBJrrkNhNsQ0RuEZHtIrK9p6enxMVw5E84e9ldzGZzAZlK6yv9OtNYKbWszDQYdGWbf9zf3e72DmB13n7twImJXsAYc4cxZpsxZltzc/MMizG5jGWP6jze1x0h6Jt+MRsRaK7WeQVKqeVlpsHgAeAm9/ZNwA/ztr/PHVV0KTCUbU6aa2lrdIVkf3eEDU2VU6aeDge9rG+q1NnGSqllp5Chpd8GngDOEJEOEbkZ+GfgjSKyD3ijex/gQeAgsB+4E7i1LKUuQDJzqr8gY9ns74mwaYohpRUBD6vrwzqvQCm1LE175jPG3DjJQ1dOsK8BPjzbQpVCfufxwd4oqYw96cpmIb+HtrowAZ8OJVVKLU9L9jI4njoVDJ457IyMPWuCmcdVIR9rG8J4dOUypdQytmQvhbOrmxlj+NUrPZzXXkvjmIRzIrBGA4FSSi3NYJC/utm+7gidQwled3rLuP1qQn5dy1gppViiwSA/Od3OjiEALlrfMG6/Bl2wRimlgCUaDPIXtNnbNUxrbYjaitEzir0eoUpHDimlFLBEg0HScjqPjXHWLzhzgo7jyqDOJVBKqaylGQzcZqLukSQDsTRnrBw/pFTnEyil1ClLMhhkVzfL5iOaaEhpOKA1A6WUylpywcAYQ9LtM3jx+BCVQS/rmkbnI/J4oEIT0SmlVM6SCwaJ9KkEdXtOjnDWyho8Y9YvqAz4ClrTQCmlloslFwyi7upm0WSGjv7YhJ3H2kSklFKjLblgEEs6/QX7uiMYGNd57PFATYUuXKOUUvmWXDBIuNlKX+kaAWBTS9Wox6uDunCNUkqNtaSCQdqyc8NKD/REaK0NjRtCWqFNREopNc6SCgaxvEylB3uibGgav6pZdUjnFyil1FhLKhhk5xdEkxlODifY0Dy6icjnFW0iUkqpCSypYBBJOiOJDvVGAdjQPLpmoLUCpZSa2JIJBom0lRtJdLA3AsBpTaNrBuGABgOllJrIkgkGw/F07vaBnij1YT/1laNTVGuWUqWUmtiszo4ichgYASwgY4zZJiINwH3AOuAw8G5jzMDsijm9bBMRwMGeyLj+Ar9PdI1jpZSaRCnOjq83xmw1xmxz738MeNgYswl42L1fVsaY3EiieMriaH+M08fML9BagVJKTa4cl8rXAne7t+8GrivDe4wSS1m5fET7eyLYBk5fcSoNhccDjZXBSZ6tlFJqtsHAAP8tIs+KyC3uthXGmE4A9/f4xYdLLJrXRLQvO/M4LxjUhwM62UwppaYw27aTy40xJ0SkBXhIRPYU+kQ3eNwCsGbNmlkVYjhxKhjs7RphRU1w1DKXlTqKSCmlpjSrmoEx5oT7uxv4PnAx0CUirQDu7+5JnnuHMWabMWZbc3PzjMuQsWzibn9BIm3x/NFBzllVm3tcBKp0foFSSk1pxsFARCpFpDp7G7gK2AU8ANzk7nYT8MPZFnIqI3m1gt8e6COetnjj5hW5bVVBH16Prl2glFJTmc0l8wrg++4iMT7gW8aYn4nIM8D9InIzcBR41+yLOblY+lQ+okf3drOiJsjm1lNpq2s1XbVSSk1rxsHAGHMQOG+C7X3AlbMpVDGyncfdIwleODbIDRetHrWKmS58r5RS01vUs7BiqUwuZfU3nzqK1yO84axTTUQ60UwppQqzqM+U2VnHB3siPLKnm2u3rqKlJpR7XEcRKaVUYRZ1MOiPpgD47nMdhANe3nnh6lGP61rHSilVmEUbDBJpi3TG0D2c4PH9vVy1eeW4lBOapVQppQqzaINBtuP4W087fQW/e96qUY97PLrEpVJKFWrRBoORRIbOoTiP7u3mLee20lw9OveQ9hcopVThFm0wiKctHnjhBADXbW0b93g4qLUCpZQq1KIMBom0Redgggdf7OTyjU00Vo3PSFod1MlmSilVqEUZDAZiKe575igeEf7wsnXjHvd4IORflIemlFLzYlGeMXd2DPHQ7i6uPmflqHkFWZUB36hZyEoppaa26IJB2rL57rMdeD3Cu7etnnAfzUeklFLFWXTB4LkjA/xidxdv2rySunBg3OMiUKPBQCmlirKoxl8m0hafeegVKoM+fv+S8QvieD3C2sawpqxWSqkiLfhgMBRL0zOSYPfJEV7sGOLpQ/2877K1VIdOXf0HfB4aKgM0VgbwaCBQSqmiLZhgsL87wpce3U/Q76EuHECAE4Nxnjk8QOdQHNtd8H5lTYi3nNsKOCOGVtSGqAlps5BSSs3GgggGh3ujvOE/fjVuu0dgS3sdr9nUxLqmStY3VrKhuZKaCj+VQd+4XERKKaVmZkGcTUeSGc5pDHPr6zdyxRktRJIZvB6hKuglHPTh83i0H0AppcpIjDHzXQaCrZvMwJHdmmVUKaWKICLPGmO2leK1FsTQ0vb6Cg0ESik1jxZEMKifYL6AUkqpuVO2YCAibxaRvSKyX0Q+Vq73UUopNXtlCQYi4gW+CFwNbAZuFJHN5XgvpZRSs1eumsHFwH5jzEFjTAq4F7i2TO+llFJqlsoVDNqAY3n3O9xtOSJyi4hsF5HtPT09ZSqGUkqpQpQrGEw0KWDUGFZjzB3GmG3GmG3Nzc1lKoZSSqlClCsYdAD5+aXbgRNlei+llFKzVK5g8AywSUTWi0gAuAF4oEzvpZRSapbKMtPLGJMRkduAnwNe4KvGmJfK8V5KKaVmb0GkoxCREWDvfJejjJqA3vkuRBnp8S1eS/nYYOkf3xnGmOpSvNBCyQGxt1T5NRYiEdmux7d4LeXjW8rHBsvj+Er1WgsiHYVSSqn5pcFAKTHXrDsAAARlSURBVKXUggkGd8x3AcpMj29xW8rHt5SPDfT4CrYgOpCVUkrNr4VSM1BKKTWP5j0YLIVU1yJyWEReFJEd2d59EWkQkYdEZJ/7u97dLiLyefd4d4rIBfNb+vFE5Ksi0i0iu/K2FX08InKTu/8+EblpPo5lIpMc3ydF5Lj7Ge4QkWvyHvtr9/j2isib8rYvyO+uiKwWkUdFZLeIvCQiH3G3L/rPcIpjWxKfn4iERORpEXnBPb6/d7evF5Gn3M/hPncyLyISdO/vdx9fl/daEx73pIwx8/aDMyHtALABCAAvAJvns0wzPI7DQNOYbf8KfMy9/THgX9zb1wA/xcnfdCnw1HyXf4LjeS1wAbBrpscDNAAH3d/17u36+T62KY7vk8BfTLDvZvd7GQTWu99X70L+7gKtwAXu7WrgFfc4Fv1nOMWxLYnPz/0MqtzbfuAp9zO5H7jB3f5l4EPu7VuBL7u3bwDum+q4p3rv+a4ZLOVU19cCd7u37wauy9t+j3E8CdSJSOt8FHAyxpjHgP4xm4s9njcBDxlj+o0xA8BDwJvLX/rpTXJ8k7kWuNcYkzTGHAL243xvF+x31xjTaYx5zr09AuzGyRq86D/DKY5tMovq83M/g4h71+/+GOAK4Dvu9rGfXfYz/Q5wpYgIkx/3pOY7GEyb6nqRMMB/i8izInKLu22FMaYTnC8w0OJuX6zHXOzxLMbjvM1tJvlqtgmFRX58brPB+ThXmEvqMxxzbLBEPj8R8YrIDqAbJwAfAAaNMRl3l/yy5o7DfXwIaGQGxzffwWDaVNeLxOXGmAtwVnb7sIi8dop9l8oxZ012PIvtOG8HTgO2Ap3AZ9zti/b4RKQK+C7wZ8aY4al2nWDbgj7GCY5tyXx+xhjLGLMVJ9vzxcBZE+3m/i7Z8c13MFgSqa6NMSfc393A93E+wK5s84/7u9vdfbEec7HHs6iO0xjT5f4T2sCdnKpSL8rjExE/zsnym8aY77mbl8RnONGxLbXPD8AYMwj8EqfPoE5EsumD8suaOw738VqcJtCij2++g8GiT3UtIpUiUp29DVwF7MI5juzoi5uAH7q3HwDe547guBQYylbdF7hij+fnwFUiUu9W2a9yty1IY/pt3o7zGYJzfDe4ozbWA5uAp1nA3123zfguYLcx5j/yHlr0n+Fkx7ZUPj8RaRaROvd2BfAGnH6RR4F3uruN/eyyn+k7gUeM04M82XFPbgH0nl+DMyLgAPC3812eGZR/A06v/QvAS9ljwGm3exjY5/5uMKdGC3zRPd4XgW3zfQwTHNO3caraaZwrjJtncjzAB3A6rvYD75/v45rm+L7uln+n+4/Umrf/37rHtxe4eqF/d4FX4zQJ7AR2uD/XLIXPcIpjWxKfH7AFeN49jl3Ax93tG3BO5vuB/wKC7vaQe3+/+/iG6Y57sh+dgayUUmrem4mUUkotABoMlFJKaTBQSimlwUAppRQaDJRSSqHBQCmlFBoMlFJKocFAKaUU8P8DTggJ8/8ZI/AAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "sns.tsplot(data=l100rew) #, err_style=\"unit_traces\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Not very smooth and somewhat slow in learning. We could do better setting a variable alpha and epsilon (why?)\n", "\n", "$$ \\epsilon = \\max \\left( \\epsilon_{min}, 1- \\log \\frac{t+1}{tau} \\right) $$\n", "$$ \\alpha = \\max \\left( \\alpha_{min}, 1- \\log \\frac{t+1}{tau} \\right) $$" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAFwRJREFUeJzt3XlwXWd5x/Hvo31fLMmyI9mS7ciJTZNgW5iEkITFBCcMcVkK9gxDoJQwLSntQDsThjal6V+F6TJ0AsFtGQhDE0xaiNuaCQlkYWicWEkcJ7axozhe5E3yJiveZT394x4FRdFyJV/p3Pue32dGo3ve+1r3eX2uf371nnPPMXdHRETCkhd3ASIiknkKdxGRACncRUQCpHAXEQmQwl1EJEAKdxGRACncRUQCpHAXEQmQwl1EJEAFcb1wfX29t7a2xvXyIiI56bnnnjvi7g3j9Yst3FtbW+no6Ijr5UVEcpKZ7Umnn5ZlREQCpHAXEQmQwl1EJEAKdxGRACncRUQCNG64m9n3zKzbzF4e5Xkzs2+ZWaeZbTGzpZkvU0REJiKdmfv3gZVjPH8L0BZ93QF859LLEhGRSzHuee7u/pSZtY7RZRVwv6fu17fRzGrMbLa7H8xQjW+yafcxfr2z5y3tLXXlfGxZ81S8pIhIzsnEh5iagH1DtruitreEu5ndQWp2z9y5cyf1Ys/vOc6/PN75prbB28B+6OrZlBTmT+rnioiEJBPhbiO0jXjXbXdfC6wFaG9vn9Sdub9w0wK+cNOCN7X9cOMe/vpnL9N3tl/hLiJCZs6W6QLmDNluBg5k4Oemraok9X/UybMXpvNlRUSyVibCfT3w6eismWuB3qlabx9NVUkhACfPKNxFRCCNZRkzewB4D1BvZl3A3wCFAO5+H7ABuBXoBE4Dn52qYkdTVTo4c++f7pcWEclK6Zwts2ac5x34YsYqmgTN3EVE3iyIT6hWlUbhrjV3EREglHB/Y+auZRkREQgk3EsK8yjMN83cRUQiQYS7mVFVUqg1dxGRSBDhDql1d50tIyKSEk64lxRo5i4iEgkn3EsL6dOau4gIEFK4l2hZRkRkUDjhXqplGRGRQcGEe2VJoU6FFBGJBBPuVSUFnL0wwLn+i3GXIiISu3DCPboEQZ/W3UVEwgn36ijcT5zW0oyISDDhXlNWBMCJ0+djrkREJH7BhHttWWrmflwzdxGRkMI9NXM/rpm7iEg44V5TNrjmrnAXEQkm3CuKCyjMNy3LiIgQULibGTVlRZq5i4gQULhD6qDqsVMKdxGRoMK9pqxIyzIiIgQW7jO0LCMiAgQW7rXlhZq5i4gQWLgPHlB197hLERGJVVDhXltWyIWLzuvndPEwEUm2oML9d9eX0dKMiCRbUOE+Q5cgEBEBAgv32nJdPExEBEIL98GZuz7IJCIJF1S411UUA3Dk9XMxVyIiEq+gwr2qpICi/DyOvK6Zu4gkW1DhbmbUVRRp5i4iiZdWuJvZSjPbYWadZnbXCM/PNbPHzewFM9tiZrdmvtT01FcUK9xFJPHGDXczywfuBW4BFgNrzGzxsG5/Baxz9yXAauDbmS40XfWauYuIpDVzXw50uvsudz8PPAisGtbHgarocTVwIHMlTkx9RTFH+rTmLiLJVpBGnyZg35DtLuCdw/p8HfiFmf0pUA6syEh1k1BfWczRU+dwd8wsrjJERGKVzsx9pIQcfmWuNcD33b0ZuBX4oZm95Web2R1m1mFmHT09PROvNg115UVcuOicPKPry4hIcqUT7l3AnCHbzbx12eVzwDoAd38aKAHqh/8gd1/r7u3u3t7Q0DC5isfRUJk6171H6+4ikmDphPsmoM3M5plZEakDpuuH9dkLvB/AzBaRCvepmZqPo14fZBIRGT/c3b0fuBN4BNhO6qyYrWZ2j5ndFnX7CvB5M3sReAD4jMd0UXWFu4hIegdUcfcNwIZhbXcPebwNuD6zpU1OfUXq+jJH+hTuIpJcQX1CFVLXdM8zdAkCEUm04MI9P8+YUZ46HVJEJKmCC3dInTHTfVLhLiLJFWS4z6oq5nDf2bjLEBGJTZDh3lhVwqFezdxFJLmCDfejp85x4eJA3KWIiMQiyHCfVV2CO3TrdEgRSagww72qBIDDJ7XuLiLJFGS4Nw6Ge6/CXUSSKchwn1WdCvdDmrmLSEIFGe61ZYUU5ecp3EUksYIMdzNjZlWxlmVEJLGCDHdIHVQ9rE+pikhCBRvujdUlOltGRBIr2HCfVVXCoZNniemy8iIisQo63E+fv0jfOd1LVUSSJ9hwn1mVuiOTDqqKSBIFG+5NNaUA7D9xJuZKRESmX7jhXqtwF5HkCjbcZ1aWUJBndB1XuItI8gQb7vl5xmU1pexXuItIAgUb7pBad9eyjIgkUdjhXquZu4gkU9jhXlPK4b6znO/XHZlEJFnCDvfaUtzhYK9m7yKSLEGHe/Pgue5amhGRhAk73GvLAOjSQVURSZigw31WdQlm6Fx3EUmcoMO9qCCPxsoSLcuISOIEHe4QnQ554nTcZYiITKvgw31ObSn7jmnmLiLJEny4t9aXc6D3DGcvXIy7FBGRaRN+uNeV4w5dx7U0IyLJkVa4m9lKM9thZp1mdtcofT5hZtvMbKuZ/Udmy5y8lrrU6ZC7jyjcRSQ5CsbrYGb5wL3AB4AuYJOZrXf3bUP6tAFfBa539+NmNnOqCp6o1rpyAHYfPRVzJSIi0yedmftyoNPdd7n7eeBBYNWwPp8H7nX34wDu3p3ZMievpqyQqpIC9hzVzF1EkiOdcG8C9g3Z7orahloILDSz35jZRjNbOdIPMrM7zKzDzDp6enomV/EEmRmt9eWauYtIoqQT7jZCmw/bLgDagPcAa4B/M7Oat/wh97Xu3u7u7Q0NDROtddJa6so1cxeRREkn3LuAOUO2m4EDI/R52N0vuPtrwA5SYZ8VWuvK6Dp+Wpf+FZHESCfcNwFtZjbPzIqA1cD6YX1+BrwXwMzqSS3T7MpkoZeipa6cAdfNskUkOcYNd3fvB+4EHgG2A+vcfauZ3WNmt0XdHgGOmtk24HHgL9396FQVPVGtg6dDat1dRBJi3FMhAdx9A7BhWNvdQx478OXoK+u0RKdD7jlyCq6IuRgRkWkQ/CdUAeoriqgoLuC1I5q5i0gyJCLczYwFDeV09rwedykiItMiEeEOcPnMSjq7Fe4ikgwJCvcKDp88x8mzF+IuRURkyiUq3AFe1exdRBIgceGupRkRSYLEhPuc2lKK8vN0UFVEEiEx4V6Qn8e8+nIty4hIIiQm3CG1NKNlGRFJgkSF+4KZFew9dlr3UxWR4CUq3C+fWcGA6xozIhK+ZIV7Q+qMmVcOa2lGRMKWqHBfMLOc/Dxjx6G+uEsREZlSiQr34oJ8FjSU89tDJ+MuRURkSiUq3AGunFXF9oOauYtI2BIX7otmV7H/xBl6z+gaMyISrsSF+5WzKwG07i4iQUtcuC+aVQXA9oNadxeRcCUu3BuriqktK9RBVREJWuLC3cy4clYV23RQVUQClrhwh9S6+85DfVwc8LhLERGZEokM90Wzqzhz4SJ7j52OuxQRkSmRyHBfPDt1UHXrgd6YKxERmRqJDPe2xgqK8vN4ab/CXUTClMhwLy7I58rZlbzUpXAXkTAlMtwBrmqq5qX9vQzooKqIBCix4X51czV9Z/vZo4OqIhKgxIb7VU01AGzpOhFzJSIimZfYcG9rrKC4IE/r7iISpMSGe2F+Hosvq2KLzpgRkQAlNtwBrm6qZuv+Xn1SVUSCk+hwv6q5hlPnL/LaEd1TVUTCkla4m9lKM9thZp1mdtcY/T5uZm5m7Zkrcepc3VwNwAt7dVBVRMIybribWT5wL3ALsBhYY2aLR+hXCXwJeCbTRU6VyxsqqCwp4HmFu4gEJp2Z+3Kg0913uft54EFg1Qj9/g74BnA2g/VNqbw8Y8ncWp7fczzuUkREMiqdcG8C9g3Z7ora3mBmS4A57v4/GaxtWiybW8vO7j5OntU9VUUkHOmEu43Q9sbpJWaWB/wT8JVxf5DZHWbWYWYdPT096Vc5hZa11OIOm7U0IyIBSSfcu4A5Q7abgQNDtiuB3wOeMLPdwLXA+pEOqrr7Wndvd/f2hoaGyVedQdfMqSbP4DktzYhIQNIJ901Am5nNM7MiYDWwfvBJd+9193p3b3X3VmAjcJu7d0xJxRlWWVLIwsZKnt+rcBeRcIwb7u7eD9wJPAJsB9a5+1Yzu8fMbpvqAqfDspZaXth7Qh9mEpFgFKTTyd03ABuGtd09St/3XHpZ02tZSy0/emYvr3T3ceWsqrjLERG5ZIn+hOqgZS21AHTs1tKMiIRB4Q7MnVHGzMpinnntWNyliIhkhMIdMDOuW1DHxl1Hcde6u4jkPoV75Nr5dfT0nWPXkVNxlyIicskU7pFr59cBsHHX0ZgrERG5dAr3SGtdGY1VxWzcpXV3Ecl9CveImXHt/DqeflXr7iKS+xTuQ1w3v44jr5/j1R6tu4tIblO4D6F1dxEJhcJ9iJa6MmZVlfC0wl1EcpzCfQgz410LUuvuA7rOjIjkMIX7MDcsrOfYqfO8fKA37lJERCZN4T7MDW2p68z/+pUjMVciIjJ5Cvdh6iuKedtlVTy5MzvuFCUiMhkK9xHcuLCB5/ccp0/3VRWRHKVwH8GNbQ30DzhPv6qzZkQkNyncR7CspZbyonyeekVLMyKSmxTuIygqyOO6BXU8tVMHVUUkNyncR3Hjwgb2HjvNa7oEsIjkIIX7KN57xUwAfvXb7pgrERGZOIX7KObMKOOKxkoe23Y47lJERCZM4T6GFYtn8uzuY/Se1imRIpJbFO5jWLGokYsDzhM7tTQjIrlF4T6Ga5prqK8o5lEtzYhIjlG4jyEvz1ixaCZP7ujhfP9A3OWIiKRN4T6OFYsa6TvXz7Ov6d6qIpI7FO7juP7yekoK83h026G4SxERSZvCfRylRfnc0NbAI1sP6wYeIpIzFO5puPWqWRw6eZYX9h2PuxQRkbQo3NOwYlEjRQV5/O8WLc2ISG5QuKehsqSQG9sa2PDSQS3NiEhOULin6UNXa2lGRHKHwj1NWpoRkVySVrib2Uoz22FmnWZ21wjPf9nMtpnZFjP7pZm1ZL7UeGlpRkRyybjhbmb5wL3ALcBiYI2ZLR7W7QWg3d2vBh4CvpHpQrOBlmZEJFekM3NfDnS6+y53Pw88CKwa2sHdH3f309HmRqA5s2VmhxWLGikuyOPhzQfiLkVEZEzphHsTsG/IdlfUNprPAT+/lKKyVWVJISsWN/LfLx7QtWZEJKulE+42QtuIi85m9imgHfjmKM/fYWYdZtbR05ObN5/+6JImjp++wJM7c7N+EUmGdMK9C5gzZLsZeMu6hJmtAL4G3Obu50b6Qe6+1t3b3b29oaFhMvXG7saFDdSVF/HTF7riLkVEZFTphPsmoM3M5plZEbAaWD+0g5ktAb5LKtiDvrNFYX4eH77mMh7b3k3vGd2hSUSy07jh7u79wJ3AI8B2YJ27bzWze8zstqjbN4EK4CdmttnM1o/y44Lw0aVNnO8fYMNLB+MuRURkRAXpdHL3DcCGYW13D3m8IsN1ZbWrmqpZ0FDOT5/fz5rlc+MuR0TkLfQJ1UkwMz66tJlndx9j37HT4/8BEZFppnCfpI8sacIM1nXsG7+ziMg0U7hP0mU1pdy0sIF1Hfvov6hz3kUkuyjcL8Ga5XM5fPIcT+zQOe8ikl0U7pfgfVfOpKGymAc37Y27FBGRN1G4X4LC/Dz+YFkzv/ptNwd7z8RdjojIGxTul+iT75jDgMNPOvSJVRHJHgr3S9RSV867L6/nx5v26TrvIpI1FO4ZsGb5XPafOMMTO4O+8oKI5BCFewbc/LZGGquK+f7/7Ym7FBERQOGeEYX5eXzqnS08tbOHzu7X4y5HREThnilr3jmXovw87n96d9yliIgo3DOlvqKYD19zGQ8918XJs7oUsIjES+GeQZ95Vyunz1/UaZEiEjuFewZd1VxNe0st9z+9W6dFikisFO4Z9pnrW9lz9DSPbT8cdykikmAK9wxb+bZZNNeW8t2ndsVdiogkmMI9wwry8/j8DfN5bs9xOnYfi7scEUkohfsU+ET7HGrLCrnvyVfjLkVEEkrhPgVKi/K5/V2tPLa9m52H++IuR0QSSOE+RW6/rpXSwnzWau1dRGKgcJ8iteVFfPIdc3h4835d611Epp3CfQr90Q3zGHD47pOavYvI9FK4T6Hm2jI+trSJB57dS/fJs3GXIyIJonCfYne+t43+AefbT+jMGRGZPgVxFxC6uXWp2fuPntnDbzqPxF2OiGSBL72/jQ9fc9mUvobCfRp8+QNXcL5/gPMXB+IuRUSyQHVp4ZS/hsJ9GsyqLuGfVy+JuwwRSRCtuYuIBEjhLiISIIW7iEiAFO4iIgFSuIuIBEjhLiISIIW7iEiAFO4iIgEyd4/nhc16gD2T/OP1QCif5ddYsk8o4wCNJVtdylha3L1hvE6xhfulMLMOd2+Pu45M0FiyTyjjAI0lW03HWLQsIyISIIW7iEiAcjXc18ZdQAZpLNknlHGAxpKtpnwsObnmLiIiY8vVmbuIiIwh58LdzFaa2Q4z6zSzu+KuZzxmttvMXjKzzWbWEbXNMLNHzeyV6Htt1G5m9q1obFvMbGnMtX/PzLrN7OUhbROu3cxuj/q/Yma3Z9FYvm5m+6N9s9nMbh3y3Fejsewwsw8OaY/1/Wdmc8zscTPbbmZbzezPovac2y9jjCUX90uJmT1rZi9GY/nbqH2emT0T/R3/2MyKovbiaLszer51vDFOmLvnzBeQD7wKzAeKgBeBxXHXNU7Nu4H6YW3fAO6KHt8F/H30+Fbg54AB1wLPxFz7jcBS4OXJ1g7MAHZF32ujx7VZMpavA38xQt/F0XurGJgXvefys+H9B8wGlkaPK4GdUb05t1/GGEsu7hcDKqLHhcAz0d/3OmB11H4f8MfR4z8B7oserwZ+PNYYJ1NTrs3clwOd7r7L3c8DDwKrYq5pMlYBP4ge/wD4/SHt93vKRqDGzGbHUSCAuz8FHBvWPNHaPwg86u7H3P048Ciwcuqrf7NRxjKaVcCD7n7O3V8DOkm992J//7n7QXd/PnrcB2wHmsjB/TLGWEaTzfvF3f31aLMw+nLgfcBDUfvw/TK4vx4C3m9mxuhjnLBcC/cmYN+Q7S7GfjNkAwd+YWbPmdkdUVujux+E1BscmBm158L4Jlp7to/pzmi54nuDSxnkyFiiX+WXkJol5vR+GTYWyMH9Ymb5ZrYZ6Cb1n+WrwAl37x+hrjdqjp7vBerI4FhyLdxthLZsP93nendfCtwCfNHMbhyjby6Ob9BotWfzmL4DLADeDhwE/iFqz/qxmFkF8J/An7v7ybG6jtCW7WPJyf3i7hfd/e1AM6nZ9qKRukXfp3wsuRbuXcCcIdvNwIGYakmLux+IvncDPyW10w8PLrdE37uj7rkwvonWnrVjcvfD0T/IAeBf+d2vv1k9FjMrJBWGP3L3/4qac3K/jDSWXN0vg9z9BPAEqTX3GjMrGKGuN2qOnq8mtWyYsbHkWrhvAtqiI9BFpA5ErI+5plGZWbmZVQ4+Bm4GXiZV8+DZCbcDD0eP1wOfjs5wuBboHfxVO4tMtPZHgJvNrDb69frmqC12w45nfITUvoHUWFZHZzTMA9qAZ8mC91+0LvvvwHZ3/8chT+XcfhltLDm6XxrMrCZ6XAqsIHUM4XHg41G34ftlcH99HPiVp46ojjbGiZvOI8qZ+CJ19H8nqfWsr8Vdzzi1zid15PtFYOtgvaTW1n4JvBJ9n+G/O+J+bzS2l4D2mOt/gNSvxRdIzSg+N5nagT8kdWCoE/hsFo3lh1GtW6J/VLOH9P9aNJYdwC3Z8v4D3k3q1/QtwObo69Zc3C9jjCUX98vVwAtRzS8Dd0ft80mFcyfwE6A4ai+Jtjuj5+ePN8aJfukTqiIiAcq1ZRkREUmDwl1EJEAKdxGRACncRUQCpHAXEQmQwl1EJEAKdxGRACncRUQC9P8tOqJgpkW1rgAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "min_alpha=0.05\n", "min_epsilon = 0.01\n", "tau = 50\n", "\n", "def set_epsilon(t, tau):\n", " return max(min_epsilon, min(1, 1.0 - math.log10((t + 1) / tau)))\n", "\n", "def set_alpha(t, tau):\n", " return max(min_alpha, min(1.0, 1.0 - math.log10((t + 1) / tau)))\n", "\n", "plt.plot([set_alpha(i, tau) for i in range(3001)])\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "*** New experiment!\n", "*** New experiment!\n", "*** New experiment!\n", "*** New experiment!\n", "*** New experiment!\n", "*** New experiment!\n", "*** New experiment!\n", "*** New experiment!\n", "*** New experiment!\n", "*** New experiment!\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "C:\\Anaconda\\envs\\py35\\lib\\site-packages\\seaborn\\timeseries.py:183: UserWarning: The `tsplot` function is deprecated and will be removed in a future release. Please update your code to use the new `lineplot` function.\n", " warnings.warn(msg, UserWarning)\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD8CAYAAACVZ8iyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8XOV97/HPb/bRblmSsWUZ29g47GCMMVtKIM0CNJCbkJA0hZuSS9ub3qZ7kza5bW57b5vc3Ga5rzYNNwvQhSTNSmgIoYaGEAhgloCxjRdsbFm2JVnWNtJs5zz3j3Mky7bAsj3SjDTf9+ull2aeOTPzzNHR+c5zzvM8x5xziIhIdYuUuwIiIlJ+CgMREVEYiIiIwkBERFAYiIgICgMREUFhICIiKAxERASFgYiIALFyVwCgpaXFLV26tNzVEBGZVZ555ple51xrKV5rSmFgZruAIcADis65NWbWDHwDWArsAt7jnDtkZgZ8HrgOGAH+s3Pu2dd7/aVLl7Jhw4aT/QwiIlXJzF4t1WudyGGiNznnLnTOrQnvfxRY75xbCawP7wO8HVgZ/twBfLFUlRURkelxKucMbgTuDm/fDdw0ofweF/g50GRmC0/hfUREZJpNNQwc8GMze8bM7gjLFjjn9gGEv9vC8nZgz4TndoZlIiJSoaZ6AvkK51yXmbUBD5nZltdZ1iYpO2ae7DBU7gBYsmTJFKshIiLTYUotA+dcV/i7G/gusBY4MHb4J/zdHS7eCXRMePpioGuS17zTObfGObemtbUkJ8NFROQkHTcMzKzWzOrHbgNvATYC9wG3hYvdBnw/vH0fcKsF1gEDY4eTRESkMk3lMNEC4LtBj1FiwL84535kZk8D3zSz24HdwM3h8j8k6Fa6naBr6QdLXmsRESmp44aBc+4V4IJJyg8C105S7oAPl6R2IiIyIypiBLKIBAqeT9FzxKKGAbHozM0Y45yj6DuiZkQiNl7mHOQ9H985Cp7D9x3ZokfUjGQsiiN4nnNQ9H2iZsSiEXJFD4CIGUXfkYhG8HxHJELwmkV//L2LviMeNRLRCGZGfSo2/noFL6hT0feJRSJEIsFrRiOGH9bPLCgDSMYi+A48P+i3kohFcM7h+UH9g/fzyRWDzxQ1ww9fw4CC50jGIkSjhvMhlYiM12suUxiIzDDnHNmCz0i+SMFzFDwf5yCTL+KFO9Ux8ViwAzIMs8M7umzBwznGd4jRiOEm7AD98EViUSNqwXOjkSBYYhELdu6+wxHsAM0gX3Tjzx/bYY/taN0x/QGn1/6Bk3/u2D57rM6lqL8Z4fqPhqHoj/8tohELw8qNB5wj+DvGoxFi0eBvE4vYeGBFo0Y8/NvUJGLjdRx7n7F17ztH0XM4HBb+HYte8HeKR0sbTgoDkWninCNX9MkVfDL5IgOjheCfHIfvH//5AIXi2F4s+J0rHPnEsZ130Zt8b1coOgrjPbun+KZwRP1mOghO1dH1LUX9x3bOo3lvvGw0f/z1mS/65Iun/v6TGfuiUCoKA5ESyRU9il7w7XAwW2AoW5x1O1KpXgoDkZPknGNwtMhQLtjxv9a3c5HZQGEgcoI833Ewk6Mvk59wGEdkdlMYiEyRc46eoRwHM/mqbgX4zjGcLVKfih3Tw8aFx8XM7HDvpIgxnC1Sm4wRMY55TtHz6R8tkIpFqUtpl1QuWvMiU1D0fF7tG2Ek5x1/4RPgwu6aowWP7d3D7OzNcGgkz/zaBKl4lLaGJC21SVKJKKlYhFQ8Sjwa4dWDGbLFoEdLJlekIRUnGgl6tHi+Y2C0QCxqJGMR2upT1Ic72VzBpzYZpS+TZzhXZLTgkSv4ZIseQ9kiew+NEo0Y8ViEQtFnYLSA7xwHM3mGsgV6h/Lkih6ZvEcqHqG1PkW24JEreBR9R7bgETGjNhljtOAd0X00EYtQ9HzqkjESsSj5oke26I8vY0BHcw1N6TgrF9TTUpcgkyuSyQfdWIdyRWoSUebVxKlPxUnHo+NdcT3nSMejRCNGKh4lFY+QjEVJxiLUJKIAZPIe/SN5Rgseg6MFuvqz9A7nSCei4z2rnIP+0Ty1iRjpRJThXJFFTWna6pPUJ2PEo0EPooHRPM21SXznqE8F5d2DWTLh9uE5RyIWoTYRIxoxahNRapKx8Tq31SdntNvwVCgMRI6j6Pns7M2QLUy9N87RuvpHeXb3ITJ5D9937Dk0wkt7BxnKFcb7vo9JxSOn9F7H83qvH43Y+I4xYlCTiJEteHQ011CfinHe4kZiEWNJcw0HM3l6hnJEDBrScWLhjth3MJwrko4H3/QLnk9dIkb3UJZELMJI3iNX9EnFg511Oh6lqSZO/0iBHT3D9I8U+O5znYQdpYhHg3EA9ckYmbA7binEIkZzbYJswSMaMYxgoEFtMka+OMJwrkgyGuWn23pL8n5HS8WDsGhvStNUkyBX9PB8hx92KfX9oGU1MFqgLhkjGQ/GaYwWgo4KpW5FKQxEXkPBC74ZHxzOH/EN93icc+w6mOHRrb1s7xmm89AovcO5I5ZprU+ysCnFVa0tNKTjJGIRls2vZfG8NPPrkgxni2SLHt1DwbmJsW/f2aJPruBxWmOapnScbNGjNhljMOy2GosaZkZjOk7BC5btGQ5aAYWiTzIeoX+kwKLGFI01ifHWRioeJR2PclpjCufc4f7zkeBwz0wPuMrkihQ8n9rw2/hYHXznGM0HrZjRgkc8asHjwGjYhzNb8MkWPPLe4W69AHXJGI3pOKl4lLpkjNb6JPEpfDsveD69wzlG8t744cGGdIy+TJ5oxDg4HPwee00IWjl5z2ckH+zgR/IeI/kimZyH74IdfCZXZChXZHffCN0HhkjFI0TDsQhjg+oiBk01caJm5Is+8WiEltpk2AIs7RcGcxXQ923NmjVOl72USpLJFdl1MDOl8QD5os+mfYM8v+cQXf1ZXt4/RN9Ifvwb9IKGFCsX1HHFGcGOPxmLTGknJPJ64jHjrIWNz0y4+uQpUctA5Cj9I3k6D42+7hgB3zk2dQ3ywMb9/PyVg+Q9f/yww7ntDZzX3sRlZ8ynMR2fuYqLnAKFgcgEmVzxdYOg4Pms39zNvz6zh+6hHLXJKG8+ewGXnD6PcxY1kg5PVorMNgoDkVC24PHqwZFJg8A5x32/6OLep3aTyXusaKvj5os7uHpVK6m4AkBmP4WBCMHOvvPQyPhcPxMVPJ/P/ftWHt3WS8e8NH/41mVcvGTenJ/F8vW81uRvJ1oulUNhIALsH8xOOvHYwGiBL6zfxlO7+nj/2iW895KO8ZknSy0aMdKJKLGIkSv6QDC76WvtREs1G6dzwSyliWjQNz8SCQaGxSPBVNa+70jGoyTCGTjHptYem8p6bEbVaCTo2eP7joLvE49ExmcQtbA3jCMYB+H5jogdfl4iGiFX9MnkimE//kIw7bQ7PKV2LBK8RsE7PLvq0TOUnohkPJiaGoJzQF7YpTYdtvSKfjCVdcHzx8dvBLOVBn8jz7nxKa/Hej7FwtljzQ6/5tjn910wb9XYbKSVFo4KA6lqvu/Y2z9K/0jhiPKRfJGv/WwXP9naQ67o8ZtvXM715y866feJRKAhFR/fuSbCwVDx6OHuhNHIsSHj+8EOMeh3HkyGF4sGO7Gx/vcRC7oxTpyXf6w7phf2VYfDUySP7dDGpkAOwuDEAy4SCaZfnqw8GTn20Fki9vo9qMa6uALjXTRfy9hYiLF1NtYn3xHscMe6xfr+4esmRMevl8AR12woh7HBhgDZoofnuXD66iD0JuZE1IyC748vE4kYnufGpykvFYWBVC3nHK/0Dh/TIjiUyfMX97/Ert4Mb1rVxrtWL6ajueaEXjsSgcZ0nLpkbHxw1ckcVopEjAiHn3f0CeqxKe2TsWN3vmZGLGoc/dDR5zhm49Guo3fkkYiRiIxd3ObIz1eJJ/XNjEQ4BfXxQhIgzfR/BoWBVK09faPHBMH27mH+8t82kckV+cQNZ7Pm9OYpvVYyHqE2GSNqwaGehknm7RGpZAoDqUrdQ1kGRo88NDQwWuCvH9iMAZ9+1/ksb6173dcwg+baBM3hPEIis5nCQKpOXybPgYEjp4cYyRf5xPc30j9S4K//03mvGwTxmNFal6SpJjHpcX6R2UhhIFUj6D567Mli5xxfevQVdvZm+MT1Z3HmgvpJn5+MR2hvSlOTiOoQkMw5CgOpGnv6Ro85NOQ7xzee3sPDW7q55ZIO1i6bP+lz59XGWdSYLmsPFJHppDCQqjAwUjgmCAZHC3xu/Vae3nWIq1e18r61SyZ9bnNdgvam9ExUU6RsFAYy5xU9n66B0SPKCp7P/7h/E1sPDPHBy5dy00Xtxwwma0gHUxJP1pdeZK7RVi5zXld/9pjLVH7pJzt4+cAQH33bG7hiRcsxz1FrQKqNwkDmtIPDuWMOD933i708uOkAN1+8eNIgWNCQpK0hNVNVFKkIusKGzFmHMnm6+rNHlD20aT9f/ulOLls+nw+sO/2Y5zTVxBUEUpXUMpA5KV888jyB7xz3PLGLbz+7lwsWN/L7v3zmMecImmriLJ6nQ0NSnRQGMicdGMyOX7LSOcfn12/j4S3dvOXsBfzmL51xzGUnU+EYAo0fkGqlMJA5x/PdEecJvrEhGEfwvks6eP+lxx4aSiciLGmu1RgCqWoKA5lzhrPF8bnin9rZxz8/uZurz5x8HEFrfZIFDUm1CKTqKQxkzhlrFTy9q4+/fmAz7U1pfufalcfs8OfXJTitUSeLReAEehOZWdTMnjOz+8P7y8zsSTPbZmbfMLNEWJ4M728PH186PVUXOZbnOwazBbYeGOJvfrSFJfNr+Pj1Zx1zjqCpJs4ijSMQGXciXUs/AmyecP9TwGedcyuBQ8DtYfntwCHn3Args+FyIjOieyhL/0iBv7x/E03pOH/xK+eweN6RF6ZJJyIaUCZylCmFgZktBq4HvhzeN+Aa4FvhIncDN4W3bwzvEz5+remArMwAz3ccHM7zyJZu+kcL/Ol1ZzGvJnHEMomYThaLTGaqLYPPAX8MjF0Waj7Q75wrhvc7gfbwdjuwByB8fCBc/ghmdoeZbTCzDT09PSdZfZHDDg7ncA4efrmbMxfUccZR1yRorkuwvLV2SpcZFKk2x/2vMLMbgG7n3DMTiydZ1E3hscMFzt3pnFvjnFvT2to6pcqKvJ6+kTw7e4fZ2ZvhmlVt4+VmsGR+De1N6WPOHYhIYCq9ia4A3mFm1wEpoIGgpdBkZrHw2/9ioCtcvhPoADrNLAY0An0lr7nIBAMjBQpFx8NbeohFjCtXHv6C0dFcQ2M6XsbaiVS+435Ncs59zDm32Dm3FLgFeNg596vAI8C7w8VuA74f3r4vvE/4+MPOuWNaBiKl4pxj/2CW4VyR9VsOcPHp82hMx4lGjCUKApEpOZU2858Av29m2wnOCXwlLP8KMD8s/33go6dWRZHX1zOUI1/0uf+FLoayRW65ZAmRCCxvraWxRkEgMhUnNOjMOfcfwH+Et18B1k6yTBa4uQR1EzmuoufTPZRjNO9x3/NdXLJ0Hiva6ljUmCYVj5a7eiKzhs6myay2byCLc/Dgpv0M5Yq85+IOohGjSS0CkROiMJBZazTv0T9SYDhb5NvPdnLuogbesLCBhnRMcw2JnCCFgcxafSN5AO786Q4GRwvcfuVygGMGmonI8SkMZNYayhZ4bvchHnm5h5vXdLCirY5kPEJtUvMvipwohYHMStmCx0jO46s/20lbfZL3rukAYIEuWSlyUhQGMisNZgvc+9Rudh0c4TfeGFy5LBGLaEyByElSGMis9ErPMN97fi9Xr2pl7bJmAF2bQOQUKAxk1skVPdZv7qbgOW5ZE1y9rC4VU6tA5BQoDGTW6R8p8OTOPjqaa2iflyYVj7Ckueb4TxSR16QwkFnF8x07ezJs3DvApUubx2ckjer6BCKnRGEgs0rvcI4nXjmI7+DS5c0saEiRjGnaCZFTpTCQWSNb8OgdzvHIlm4WNqZYvaSJ1vpkuaslMicoDGTW2N03wv7+LC/uHeBNq9poqVfvIZFSURjIrDCYLZAr+PzrM51EI8Yvn9NGQ0ojjUVKRWEgFc/3HfsHsnT1j/LQ5gO89ZzTOKOlXpPRiZSQwkAq3t7+UbJ5j688tpNoxHjPmg5dtEakxBQGUtGyhWCa6se29/LUrj5+bd3ptDUkdYhIpMQUBlKxnAsOD3m+45+f3M2S5hp+5fxFNNcmdIhIpMQUBlKx9g9mGcoWeWjTAfb2j/KBdaeTiEVorVN3UpFSUxhIRfJ8R18mz87eDHf+dAfnLGpg3bJm2hqSRDTaWKTkFAZSkXqGcvg+3PX4LpKxKB97+1mkElHm1+oqZiLTQWEgFSeTK9I7nOOpnQd5dvchbr54MY3pOAsbUzpXIDJNFAZSUZxzdPWP0jec5zM/3srCxhQ3nL+IVDxCfUrdSUWmi8JAKkrPcI5swefrG/aQ93w+ccPZJGIRFjaly101kTlNYSAVo+j59AzleKlrgB9t3Mdbzl5Ax7waFjQkqdNF7kWmlcJAKkZfJs/ASIFP/WgLjek4H7x8WdCVVDOTikw7hYFUBOccBzN5vvb4LgZGC3z8+rNJJ6K01Sd10lhkBigMpCL0ZfI8v7ufhzYd4J0XtXPmgnqa6xLMU1dSkRmhMJCyG8kX2T+Q5WuP76S1PsktlywhEoEFOjwkMmMUBlJWRc9nZ2+G5/b0s/XAMDdfvJh0IsqS5hpiUW2eIjNF/21SVj3DwUjjbz3TSVNNnGvfsID2prTGFIjMMIWBlE2u6HFwOM+W/YM8v6efGy9op6Ve5wlEykFhIGWzrz/L7r4R/ue/baa5JsH155/GaY26rrFIORw3DMwsZWZPmdkvzOwlM/tkWL7MzJ40s21m9g0zS4TlyfD+9vDxpdP7EWQ26svk6R8p8H9+/DK+c/zVTeeyqClNMhYtd9VEqtJUWgY54Brn3AXAhcDbzGwd8Cngs865lcAh4PZw+duBQ865FcBnw+VExjnn6B7K8vCWA+zoyfCbv3QGS+bX0KLrFIiUzXHDwAWGw7vx8McB1wDfCsvvBm4Kb98Y3id8/FrTqCGZoC+TZ2i0yD89uZtVC+q5ckULjek4qbhaBSLlMqVzBmYWNbPngW7gIWAH0O+cK4aLdALt4e12YA9A+PgAMH+S17zDzDaY2Yaenp5T+xQya3i+Y/9glv/78Hb6Mnk+eMVSYtEIbQ1qFYiU05TCwDnnOecuBBYDa4GzJlss/D1ZK8AdU+Dcnc65Nc65Na2trVOtr8xy/SN5nn21n0e39fDeSzo4t72R5a21OlcgUmYn1JvIOdcP/AewDmgys7GpJBcDXeHtTqADIHy8EegrRWVldvN9x76BLP/wkx0sbEzxnos7OK0xpcNDIhVgKr2JWs2sKbydBt4MbAYeAd4dLnYb8P3w9n3hfcLHH3bOHdMykOpzMJPn2890srd/lDveuJy6VEwnjUUqxFQmiV8I3G1mUYLw+KZz7n4z2wR83cz+CngO+Eq4/FeAfzSz7QQtglumod4yy+SLPp2HRvju83tZvaSJNac3a0yBSAU5bhg4514ALpqk/BWC8wdHl2eBm0tSO5kz+jJ5Htp0gP6RAjde0E5zXYLGtKacEKkUGoEs087zg+sa3/PEq5y7qIFLls1jYYNaBSKVRGEg065nKMc3N+xhOFfk1suWsmR+LZGIhp6IVBKFgUyrouezp2+E+37RxeVnzOeSZc26nrFIBVIYyLTqHc6zfks3I3mPd17UrusZi1QohYFMGz88V/DNDXtYtaCetUvVKhCpVAoDmTY9w8G5gr5MnjveuJw2nTQWqVgKA5kWzjn29I3wwMb9XLqsmbXLmkknNNJYpFIpDGRaDOeK/PDFfQyMFnjX6sU01WhMgUglUxjItNg/kOX+F/ZxRmstq09v0jWNRSqcwkBKLlvweGjTAbqHcnzg0tN1rkBkFlAYSMn1Dud4aNMBWuqSXL5iPg1qFYhUPIWBlFS+6LN1/xDP7+nnzWe1cVpjutxVEpEpUBhIyeSLPnsOBaONzeC68xbSkNK4ApHZQP+pUhK5osf27mF6BnP88MX9vGlVGxd0NKHLX4vMDmoZSEkcGMjh+/DNZzrxnOP9ly5Rq0BkFlEYyCkbyRcZGC2w4dU+fvjiPt5y9gJWnVavVoHILKKvbnLKeofyfPe5Tu56fBdN6TjvWdPBvJpEuaslIidAYSCnJFf0eKlrgLufeJVLljbze28+k7aGJLWakE5kVtFhIjklvcN5vvqznSRjEX77TSuoTcY0TbXILKQwkJPmnOOxrT08vesQ7754MU01wXWNNfWEyOyjtryctL5Mnq/+bBfzaxO844JFLGhM0lqnVoHIbKSWgZy0H/yii5cPDPGrly7htMYUbfUp9SASmaUUBnJSsgWPf32mk7b6JNefv5CFmnZCZFZTGMhJeXJnHy91DXL9eQs5fX4t0YhaBCKzmcJATli24PEvT75KIhbhXasXk4rrCmYis53CQE6I7zu27B/k0a29vHFlCysW1JW7SiJSAgoDOSE9wzl+8It9jBY8rj9voVoFInOEwkCmLF/0eWjTAb72s52sOX0e157VVu4qiUiJKAxkyp7bfYi/fmAzHfNq+NPrzqIhrfmHROYKhYFMSdHz+bPvbSQWifCJG85m8Tx1JRWZSxQGMiX3v9DF9u5h/vNlS1neWkuLRhqLzCmajkKm5MuP7aSlLsHNlyxm8byacldHRErsuC0DM+sws0fMbLOZvWRmHwnLm83sITPbFv6eF5abmX3BzLab2Qtmtnq6P4RMr8e297Jx7yC3XLJEQSAyR03lMFER+APn3FnAOuDDZnY28FFgvXNuJbA+vA/wdmBl+HMH8MWS11pm1Jd/+go1iSi3XX56uasiItPkuGHgnNvnnHs2vD0EbAbagRuBu8PF7gZuCm/fCNzjAj8HmsxsYclrLjNi075BHt3aw6+cv5DW+lS5qyMi0+SETiCb2VLgIuBJYIFzbh8EgQGMdTpvB/ZMeFpnWCaz0D/8xw4iZvzW1WeUuyoiMo2mHAZmVgd8G/hd59zg6y06SZmb5PXuMLMNZrahp6dnqtWQGbR/YJQHNu7jreecxtIWTTshMpdNKQzMLE4QBP/snPtOWHxg7PBP+Ls7LO8EOiY8fTHQdfRrOufudM6tcc6taW1tPdn6yzT6x5+/SsFz/JerlpW7KiIyzabSm8iArwCbnXN/O+Gh+4Dbwtu3Ad+fUH5r2KtoHTAwdjhJZo+eoRzfeXYvqxbUc+GSeeWujohMs6mMM7gC+DXgRTN7Piz7U+BvgG+a2e3AbuDm8LEfAtcB24ER4IMlrbHMiDsf3cG+gSx/dv1Z5a6KiMyA44aBc+4xJj8PAHDtJMs74MOnWC8po56hHPc88SqXLZ/P289VRzCRaqDpKOQYP/hFF7miz+1XLdMVzESqhMJAjuB5Pvc+tZtFjSneuKKl3NURkRmiMJAj/HjTAbZ1D3Pr5UtJ6MI1IlVDYSBHuPuJXTTVxHn/2iXlroqIzCCFgYx7tTfDk6/08Y7zF9GQjpe7OiIygxQGMu6HG/fhgHddvLjcVRGRGaYwECA4cXzf8120N6U5f3FjuasjIjNMYSAAPLK1h837h7jlkg6CQeciUk0UBoLvO/7xiVepTUa5RSeORaqSwkB4YW8/j23v5W3nnEZrva5tLFKNFAZVruj5fPpHLxONGL9+pWYnFalWCoMq98DG/Ty+4yDvX9vB2Qsbyl0dESkThUEVc87xtw9tZVFjig9dtVwnjkWqmMKgij2+/SA7ezPcvKaD9qZ0uasjImWkMKhiX3p0B7WJKB+49HS1CkSqnMKgSm3cO8Bj23u56aJ2WhvUg0ik2ikMqtQ//GQHETP+y1XLy10VEakACoMq9OrBDA++tJ+3nnMaS1tqy10dEakACoMq9NXHdlL0HP/1TWeUuyoiUiEUBlWmL5Pj28/u5YoVLZyzSBPSiUhAYVBlvvrYLoZzRX7jjTpXICKHKQyqyKGRHPc+tZtz2xu46szWcldHRCqIwqBKOOf43EPbOJjJ8zvXrCx3dUSkwigMqsSW/UPc+/QerlrRwjVvaCt3dUSkwigMqoBzjs/8+GWcc/zR21YRi+rPLiJH0l6hCvxsey/rN3fzzovaOVc9iERkEgqDOc7zfP7XA1toqonzW1efQSSiOYhE5FgKgznu60/vYVPXIHdctZxlLXXlro6IVCiFwRyWyRX4wsPbWNZSq6uYicjrUhjMYV96dCcHBnP8/i+fSSoeLXd1RKSCKQzmqP5Mnrsf38XqJU3ccP7CcldHRCqcwmAOyhY8/vLfNjEwWuBDV+pyliJyfAqDOeiux3fx3ef28qY3tHKdWgUiMgXHDQMz+6qZdZvZxgllzWb2kJltC3/PC8vNzL5gZtvN7AUzWz2dlZdjvdjZz+f+fStnL2zgf73zvHJXR0Rmiam0DO4C3nZU2UeB9c65lcD68D7A24GV4c8dwBdLU02ZCt/3+e/3vUTUjP9x4zksbNRF7kVkao4bBs65R4G+o4pvBO4Ob98N3DSh/B4X+DnQZGY6TjFDvrmhk+d293P7lcu4aMm8cldHRGaRkz1nsMA5tw8g/D0281k7sGfCcp1hmUyzzkMjfPrBl1neUstv/JJOGovIiSn1CeTJ9kBu0gXN7jCzDWa2oaenp8TVqC79I3l+597n6B/J88l3nENtMl7uKonILHOyYXBg7PBP+Ls7LO8EOiYstxjomuwFnHN3OufWOOfWtLbqQiun4mPfeZHndvfze28+UxetEZGTcrJhcB9wW3j7NuD7E8pvDXsVrQMGxg4nyfRYv/kAD2zcz3sv6eA3fkkXuBeRkxM73gJmdi9wNdBiZp3AnwN/A3zTzG4HdgM3h4v/ELgO2A6MAB+chjpLaMu+QT78L8+yeF6aP3zLmSRiGjYiIifnuGHgnHvfazx07STLOuDDp1opOT7fd3zsuy8SMeOeX19LS32q3FUSkVlMXyVnqc+v38Zzu/v5k7etYnmrpqYWkVNz3JaBVBbfd9zzxC6+sH4bV65o4dbLlpa7SiIyBygMZpFc0eP/PfoKn/v3bZzb3sgXP7Ba4wlccXW0AAAJM0lEQVREpCQUBrNEJlfkE9/byHee28sFixv50q9dTH1K4wlEpDQUBrPAoZE8v/v15/nJ1h6uP28hf3XTOcyrTZa7WiIyhygMKtyOniF+65+eZeuBYT505TL+8K2rdNUyESk5hUEF29Wb4de/toHuoRx/et1ZfOjKZUQiOkcgIqWnMKhQ//ZiF3/+/ZcYzBb525sv4LrzFioIRGTaaJxBBbrniV185N7nScai/N37V3PDBYsUBCIyrdQyqCDOOT75g03c9fguzl/cyJdvXUNbg0YWi8j0UxhUkE/et4m7ntjFW89ZwOdvuZBUXH8eEZkZ2ttUgIGRPB//3kZ+8MI+3rW6nf/97vOJRHQET0RmjsKgjHzf8Y2n9/DpB7fQP1Lgg5cv5eM3nK3zAyIy4xQGZfLS3gH+6NsvsKlrkLMW1vPFD1zMuuXzy10tEalSCoMZ5pzjO8/u5aPfeYHaZIy/uulc3r92iVoDIlJWCoMZtL17iD/77kae3NnH6iVN3HnrGlrqNK2EiJSfwmAG+L7jrsd38akfbSEZi/Dx68/itsuXEo/qJLGIVAaFwTR79WCG3/6X53hx7wBXrJjPZ99zocYOiEjFURhMk4Lnc9fPdvHpB7cQj0b4zM0X8M6L2onq3ICIVCCFQYk553jilYP85f2b2LxviKtWtvCZmy9ggVoDIlLBFAYltO3AEF94eBs/+MU+mmvifP69F3LDBYvUGhCRiqcwOEWD2QJ3P76L+57vYlv3MBGDD79pBf/tmhW67oCIzBoKg5OULXjc+9Ru/v6RHfQM57iwo4n/fsPZ3HDBQtrqdUhIRGYXhcEUOefYuHeQB1/ax/ot3Ww7MEzRd6xe0sTf/epq1i5rLncVRUROmsLgNTjnePXgCE/t7ONnO3p5fMdBeoZyRAwu6Gji9iuXcfWqNtYtb8ZM5wREZHZTGAD5os+BwSzbu4fZ1j3Ei50DPLmzj+6hHADzauKsXdbMtW9YwLVntTFfo4ZFZI6piDDIFX3+/pHtLGup5eKl82hMx+nqz9J5aIRFTWkWNaYZyRfJez59mTxNNQkaUjFyRZ9swQPAzHDOkSv6jOY9sgWPbMFntBDczuSK7BsYpfPQKIdGChzK5OkeytE/kieT946oT1t9krXLmlm3fD7rljdzRmudvv2LyJxWEWGwszfDpx98GYBoxPB9h5uG9zGgpT7J/NoEjek4q5c00VKfZF5Ngtb6JCvb6ljRVkdTTWIa3l1EpHJVRBgUPJ8PXbmMVafVs2X/ELWJKEuaa1jUlGZ33whd/aM01yZIxqPUJWMM54oMZQskY1HSR3XfTMYjpOJB+djvdCJCOhGjtS5JIqb5gEREjlYRYQDwm1efMekMnpeXoS4iItWmIr4mp2JRTeUsIlJGFREGp8+vKXcVRESqWkWEgY7ji4iU17Tshc3sbWb2spltN7OPTsd7iIhI6ZQ8DMwsCvwd8HbgbOB9ZnZ2qd9HRERKZzpaBmuB7c65V5xzeeDrwI3T8D4iIlIi0xEG7cCeCfc7wzIREalQ0xEGk83bcMyAYjO7w8w2mNmGnp6eaaiGiIhM1XSEQSfQMeH+YqDr6IWcc3c659Y459a0trZOQzVERGSqpiMMngZWmtkyM0sAtwD3TcP7iIhIiZR8OgrnXNHMfht4EIgCX3XOvVTq9xERkdIx56ZjftATrITZEPByuetRIVqA3nJXokJoXRymdXGY1sVhq5xz9aV4oUqZqO5l59yacleiEpjZBq2LgNbFYVoXh2ldHGZmG0r1WpoHQkREFAYiIlI5YXBnuStQQbQuDtO6OEzr4jCti8NKti4q4gSyiIiUV6W0DEREpIzKHgbVNN21mXWY2SNmttnMXjKzj4TlzWb2kJltC3/PC8vNzL4QrpsXzGx1eT9B6ZlZ1MyeM7P7w/vLzOzJcF18Ixy4iJklw/vbw8eXlrPepWZmTWb2LTPbEm4fl1XrdmFmvxf+f2w0s3vNLFVN24WZfdXMus1s44SyE94WzOy2cPltZnbb8d63rGFQhdNdF4E/cM6dBawDPhx+3o8C651zK4H14X0I1svK8OcO4IszX+Vp9xFg84T7nwI+G66LQ8DtYfntwCHn3Args+Fyc8nngR85594AXECwTqpuuzCzduB3gDXOuXMJBq7eQnVtF3cBbzuq7IS2BTNrBv4cuJRgJuk/HwuQ1+ScK9sPcBnw4IT7HwM+Vs46zfDn/z7wywQD7haGZQsJxl0AfAl434Tlx5ebCz8E81atB64B7ieY5LAXiB29fRCMaL8svB0Ll7Nyf4YSrYcGYOfRn6catwsOz3rcHP6d7wfeWm3bBbAU2Hiy2wLwPuBLE8qPWG6yn3IfJqra6a7D5uxFwJPAAufcPoDwd1u42FxfP58D/hjww/vzgX7nXDG8P/Hzjq+L8PGBcPm5YDnQA3wtPGT2ZTOrpQq3C+fcXuAzwG5gH8Hf+Rmqc7uY6ES3hRPeRsodBlOa7nquMbM64NvA7zrnBl9v0UnK5sT6MbMbgG7n3DMTiydZ1E3hsdkuBqwGvuicuwjIcPgwwGTm7LoID2XcCCwDFgG1BIdCjlYN28VUvNbnP+H1Uu4wmNJ013OJmcUJguCfnXPfCYsPmNnC8PGFQHdYPpfXzxXAO8xsF8HV8K4haCk0mdnYNCkTP+/4uggfbwT6ZrLC06gT6HTOPRne/xZBOFTjdvFmYKdzrsc5VwC+A1xOdW4XE53otnDC20i5w6Cqprs2MwO+Amx2zv3thIfuA8bO9t9GcC5hrPzWsMfAOmBgrKk42znnPuacW+ycW0rwd3/YOferwCPAu8PFjl4XY+vo3eHyc+IboHNuP7DHzFaFRdcCm6jC7YLg8NA6M6sJ/1/G1kXVbRdHOdFt4UHgLWY2L2xtvSUse20VcKLkOmArsAP4s3LXZ5o/65UETbUXgOfDn+sIjnGuB7aFv5vD5Y2gt9UO4EWCHhZl/xzTsF6uBu4Pby8HngK2A/8KJMPyVHh/e/j48nLXu8Tr4EJgQ7htfA+YV63bBfBJYAuwEfhHIFlN2wVwL8H5kgLBN/zbT2ZbAH49XC/bgQ8e7301AllERMp+mEhERCqAwkBERBQGIiKiMBARERQGIiKCwkBERFAYiIgICgMREQH+P9Mzug9sfVSPAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "l100rew=[]\n", "for _ in range(10):\n", " print('*** New experiment!')\n", " # Create and initialize Q-value table to 0\n", " Q = np.zeros(discr_vector + (env.action_space.n,))\n", "\n", " # Just to store the long-term-reward of the last 100 experiments \n", " scores = deque(maxlen=100)\n", " lrews = []\n", "\n", " for episode in range(1,1001):\n", " done = False\n", " e = set_epsilon(episode, tau)\n", " a = set_alpha(episode, tau)\n", " R, reward = 0,0\n", " state = d.Discretize(env.reset())\n", " while done != True:\n", " action = choose_action(state, e) \n", " obs, reward, done, info = env.step(action) \n", " new_state = d.Discretize(obs)\n", " Q[state][action] += a * (reward + gamma * np.max(Q[new_state]) - Q[state][action]) #3\n", " R = gamma * R + reward\n", " state = new_state \n", " scores.append(R)\n", " mean_score = np.mean(scores)\n", " lrews.append(np.mean(scores))\n", " l100rew.append(lrews)\n", "\n", "sns.tsplot(data=l100rew) #, err_style=\"unit_traces\")\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Average reward for epsilon 0: 472.5\n" ] } ], "source": [ "print('Average reward for epsilon 0:', np.mean([rollout(0,render=True) for _ in range(20)]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercises\n", "\n", "1. Play with parameters for discretization and see how they affect performance and convergence\n", "2. Try other methods to decrese $\\alpha$ and $\\epsilon$\n", "3. Try Boltzman exploration instead of $\\epsilon$ greedy\n", "4. Obtain Q-values for starting state. Is that strange? Can you explain/solve the problem using gamma different to 1? Try with gamma 0.99 for example.\n", "5. **Implement Monte-Carlo for this problem and compare performance and variance with Q-learning**\n", "6. **Implement n-steps Q-learning for this problem and compare performance with Q-learning**\n", "7. **Implement Sarsa on this problem**\n", "8. **Adapt everything has been done here to another Gym problem, f.i. Acrobot-v1 or MountainCar-v0**" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "env.close()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.8" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 2 }
NumObservationMinMax
0Cart Position-4.84.8
1Cart Velocity-∞∞
2Pole Angle rads-0.420.42
3Pole Velocity At Tip-∞∞