{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(mlflow-primer)=\n",
    "# MLFlow Primer\n",
    "\n",
    "So after `pip install mlflow`  you can track the runs of your code by inserting a few extra lines. This following is not a full tutorial of course, just something quick to show the basic and convince you it's easy to work with.\n",
    "\n",
    "MLFlow is organized into 'experiments', which are essentially just collections of runs. One run is one execution of your code. MLFlow tracks a bunch of metadata automatically, and in addition you can store basically whatever you want in a run. MLFlow uses a number of concepts to seperate information logically and displays them in different ways: 'parameters' (inputs), 'metrics' (outputs), 'tags' (labels) and 'artifacts' (files).\n",
    "\n",
    "Once your runs are stored, you can view them either through the UI or the API. We won't use the UI in this guide, because we need to access the stored runs programmatically through the API, but the UI is very useful and trivial to run (checkout the MLFlow docs). \n",
    "\n",
    "The skeleton of the MLFlow code to be inserted basically looks like this:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "import mlfow\n",
    "\n",
    "mlflow.start_run():\n",
    "    mlflow.log_param('param_1', 3.14)\n",
    "    mlflow.log_metric('answer', 42)\n",
    "    mlflow.log_artifact('figure.png')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, you will see a more elaborate and realistic example. (note that not all dependent functions are shown)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "IPython.notebook.set_autosave_interval(0)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Autosave disabled\n"
     ]
    }
   ],
   "source": [
    "# prevent jupyter and your IDE from trying to make simultaneous changed\n",
    "%autosave 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn import datasets\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "from matplotlib.colors import ListedColormap\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import mlflow\n",
    "import mlflow.sklearn\n",
    "\n",
    "def get_data():\n",
    "    iris = datasets.load_iris()\n",
    "\n",
    "    X = iris.data[:, [2, 3]]\n",
    "    y = iris.target\n",
    "\n",
    "    return train_test_split(X, y, test_size=0.35, random_state=0)\n",
    "\n",
    "\n",
    "def feature_engineering(X_train, X_test):\n",
    "    sc = StandardScaler()\n",
    "    sc.fit(X_train)\n",
    "    X_train_std = sc.transform(X_train)\n",
    "    X_test_std = sc.transform(X_test)\n",
    "    return X_train_std, X_test_std\n",
    "\n",
    "def recombine_data(X_train, X_test, y_train, y_test):\n",
    "    X_combined_std = np.vstack((X_train, X_test))\n",
    "    y_combined = np.hstack((y_train, y_test))\n",
    "    return X_combined_std, y_combined\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "def plot_decision_regions(X, y, classifier, test_idx=None, resolution=0.02):\n",
    "\n",
    "    # setup marker generator and color map\n",
    "    markers = ('s', 'x', 'o', '^', 'v')\n",
    "    colors = ('red', 'blue', 'lightgreen', 'gray', 'cyan')\n",
    "    cmap = ListedColormap(colors[:len(np.unique(y))])\n",
    "\n",
    "    # plot the decision surface\n",
    "    x1_min, x1_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n",
    "    x2_min, x2_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n",
    "    xx1, xx2 = np.meshgrid(np.arange(x1_min, x1_max, resolution),\n",
    "                           np.arange(x2_min, x2_max, resolution))\n",
    "    Z = classifier.predict(np.array([xx1.ravel(), xx2.ravel()]).T)\n",
    "    Z = Z.reshape(xx1.shape)\n",
    "    plt.contourf(xx1, xx2, Z, alpha=0.4, cmap=cmap)\n",
    "    plt.xlim(xx1.min(), xx1.max())\n",
    "    plt.ylim(xx2.min(), xx2.max())\n",
    "\n",
    "    for idx, cl in enumerate(np.unique(y)):\n",
    "        plt.scatter(x=X[y == cl, 0], y=X[y == cl, 1],\n",
    "                    alpha=0.8, c=cmap(idx),\n",
    "                    marker=markers[idx], label=cl)\n",
    "    \n",
    "    # highlight test samples\n",
    "    if test_idx:\n",
    "        X_test, y_test = X[test_idx, :], y[test_idx]\n",
    "        plt.scatter(X_test[:, 0], X_test[:, 1], c='', \n",
    "                   alpha=1.0, linewidth=1, marker='o',\n",
    "                   s=55, label=\"test set\")\n",
    "\n",
    "def train_knn(data, target, **params):\n",
    "    knn = KNeighborsClassifier(**params)\n",
    "    knn.fit(data, target)\n",
    "    return knn\n",
    "\n",
    "\n",
    "def train_svc(data, target, **params):\n",
    "    svm = SVC(**params)\n",
    "    svm.fit(data, target)\n",
    "    return svm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": [
     "remove-output"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'.  Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'.  Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'.  Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEGCAYAAABsLkJ6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO2deZycVZW/n9NVvWTpELKxZCGsIYQgkBADGQsiW9iGQWEIKoj0TCvqqIMbUX8OyMxERRlX0GgQQQdUEEFkDYZp7bCkE5HQdAIhZCWQTqCT7iS9VNX5/XHft6u60l1d3bV3nSef99P1bvc91Unuee89536PqCqGYRhG6VGWbwMMwzCM/GAOwDAMo0QxB2AYhlGimAMwDMMoUcwBGIZhlCjBfBswEA4ed7AefsTh+TbDMAyjqHhl9Ss7VXV84vGicgCHH3E49624L99mGIZhFBUnVZ20qbfjNgVkGIZRopgDMAzDKFHMARiGYZQoRRUD6A0NK/qWQke+LSlQKkEOFSQo+bbEMIwCo/gdwFvKuFHjGD1mNCLWycWjqrS808LOt3Yik+x3YxhGT4p/CqgD6/z7QEQYPWa0jY4Mw+iV4ncAYJ1/Eux3YxhGXwwJB2AYhmEMnKJyAC3797MtvI1t4W35NuUAnnz8SU6efjIzj5vJd771nXybYxiG0S9F5QDCe4dR/1doWNfCQ42NBeMIIpEIN/zbDTz4pwdZ9fIqfnff72h6pSnfZhmGYSSlqLKAqhjG5E0LYBNsOeJxGmhhQeU42rWdKqnq9/7KOe9DmpsPOK7jx9Pxwl8GbVfDCw0cdfRRHHnUkQBcfuXlPPLwI0w/Yfqg2zQMw8g2RTUCiGfypgWsvW0hHXvL2d8WoGV/O62d7bRre5/3SHMzOn7cAVtvTmEgvLntTSZNntS9P3HiRLZv255Wm4ZhGNmmqEYAiYRCUE4FFZ0H0brLHas65B3acU6gqoqURgaGYRilSFE7gHiqq70P+8YA0K77aPf+AIyoCpAtV3D4xMPZumVr9/62bds4bOJhWXqaYRhGZijaKaD+qJLhlO8bQ/m+MUT2VrG3PUJUlahGUe9Ppph12ixeX/86G9/YSGdnJ/f/5n4uuuSijLVvGIaRDYbMCCAZVTIc9g1HtIxoVFzXL0pZmSIZ8IHBYJDv/uC7XHrBpUQiEa752DWcMOOEtNs1DMPIJiXhAHyi4yYQ2LnDfY4CgSgK6ITUM4n6YsGFC1hw4YLMGGoYhpEDSsoBtDy58oBj7boPgEB7O10BL3hcHqBcynNqm2EYRq4pKQfQG1Uy3H3YN5zWVigfuY/IiHYgQiAA1RWWRWQYxtCk5B1APC6TyMULADordtMScaMCcwaGYQw1zAEkoaLzIOiE1la3vqBlf2x9QQCbJjIMo7gxB5AC1dV0ry9obQUduxsJRoAII6rMERiGUZyYAxgg1dWANzJo133spZ1AIEJ5ua06NgyjuBiyC8FyQZUMp/3tMXzmY5/juEOnceqJs2jtdJpEhmEYhU7eHICITBaR5SLyiog0ishnc/Fc1eT7A6W6Gj581b/wm/ueIBoOsPfNMUQipCROZxiGkU/yOQIIA59X1ROAucCnRCSry2fvvhvuuCPW6au6/bvvTq/d008PMXr0GJcpVE23BMXeN8fQ3u6cQcv+drq0K/0vYRiGkSHy5gBUdbuqrvY+twJNwMTsPQ/a2uDBB2NO4I473H5bW/ojgd6Idwa+HlG8MzCHYBhGPimIILCITAVOAZ7v5VwtUAswZsyUNJ4B11/vPj/4oNsALrvMHc927XRfjwjc+oK2NpBghEAgYusLDMPIC3kPAovISOAB4HOquifxvKouUdXZqjp75MjxaT4r5gR8ctH5J1LReRAVnQdRvm8M4Q5XzKZdLV5gGEZuyasDEJFyXOf/a1X9fbaf50/7xBMfE8gHFZ0HuemhXVXd8QILHhuGkQvymQUkwFKgSVVvy/bz4uf8L7sMnnzS/YyPCQyWj3/8Ki688HTWr1/He94ziV//eumA7q+S4d31C9rfHkPnHucM/JRSixUYhpEN8hkDmAdcDawRkRe9Y19R1Uez8TARGDmy55y/Px00cmR600A//em9mTGSnnpEJk5nGEY2yZsDUNW/Ajmdfb/mGvem73f2vhPIdQwgVfoTpzPZasMw0qEgsoBySWJnX6idf2/Ei9NVjt1NWwdIsN3E6QzDGBQl5wCGAt16REB7xz72h7u6xemqvBki0yUyDKM/zAEUOVUyHDqJidPthcCIWHUzgC7topLK/BlpGEZBYg5gCJFY3cwnHN7A442NjKqG4w4fzcRg1hZcG4ZRRJgDGKK4ALJj584RrP3BQsafsYY9cxtpoAWA2dPMGRhGKWMOIANs27aFT3/6Gpqb30ZEuPrqWmprcyJumjKhEMBMaJgJwJYjHqeBFlqOaWFG5Yy82mYUHzvDO9kc3ky7tlMlVUwJTmFccFy/1wYIoKpEJdrvfUb2MQeQAYLBIDff/F1OOulU2tpaOeecWZx55rlMm5ZVcdO0mLxpAXX3ADfcx3oauXSGOQEjNXaGd/Jq16uUUUaQIJ3ayatdrwIc0JnHX4vCXvYiCBVaQSd932fkhrxrAeWaN2UjT5c9wEOBX/B02QO8KRvTbvOQQw7jpJNOBWDkyGqOO24627dvS7vdbBMKwYSGhXQ0j+ahxkYaOxrzbZJRBGwOb6aMMgISQEQISIAyytgc3pz02i66nCMAwoST3mfkhuJyADt3pnX7m7KRlWXL2S97qaCS/bKXlWXLM+IEfDZv3siaNX9j1qz3ZqzNbDN50wLW3raQ9evhocZGlm9uZFu48B2YkR/atb27I/cpo6xX/ar4a6NEu4/7n/u6z8gNxTUFFA7D4sUwb54/qT0gmmQVAQIEcQum/J9NsorDdWra5rW1tXHddR/kllu+R3X1qLTbyyWhENCwkLo6uoPFr1a7YLFlDhnxVEkVndpJgED3sSjRXteexF9bRhmKE92Kdwq2ZiV/9OkAROQDKdzfni3tnt4Yf2iA2gl/YEk98NprUFMzoPvbZA8VCfnwAYK0yQEq1AOmq6uL6677IB/84Ie5+OJUfnWFiR8srrvNBYuPvPpx9rS2dGcOgWUPlTpTglPc3L26jjzq/ZkSPLBeR/y15ZTTTjuCECRIRCN93mfkhmQjgJ8BD5FcrycE5MwBAFBTQ21dnXMCA2SkjmK/7O1+8weIEGakpve2rqp87nM1HHfcdK6//oa02ioUugdYmxbApthxP3uogRZbV1AkDCRjJxkbOjawNbKVMGHKKKOCin6zefxjm8ObaaedEYzozgKqkArLAsozyRzAY6p6XbKbReRXGbYnNUIhqN/lpoPuXpDybdN1FitlOeDe/COEiRBhus5Ky5znn6/nd7+7h+nTZzJ//skAfPWr/80551yYVruFyOQ4h7DliMdp3gCV4xs55hgsnbQAGUjGTjI2dGxgY2Qj4v1RlHbamVo2laMqj0p677jgOOvkC5Q+HYCqfqS/m1O5JlvULhoLdXW80DEftm+Hww7r957DdSqnRefTJKtokz2M1FFM11lpz//PnfsP7NiRx6oyeWLyJud849NJjznGnTNnUBjEZ+GAEw1E3fGBdMpbI1u7O//E40eR3AEYhcugYwC5qODVL6EQ1aO8bITt252wf/wS2F44XKdmJOBrxPADyE3hNex6DsbObWRHtUsptSmi/NKu7QQT/psPJvMmTPiAzt8/bhQvyaaALvF+TgDOAP7s7c8HVgD5dwAe4w8LQmsrzR3Bfh2AkT2mB13gmIaZvWYTzZ9io4JcM5CMnWQECRIh0utxo3hJNgX0MQAReRI4QVW3e/uHAXflxLqB0hVOeTrIyC69ZRM91OpGBaOqzRnkiinBKTR1NtGu7Sgam8ZRWLF/xQHSDKNlNC3ackDAeFJgEhsjG7vb9dM5o0RZ3b66RzA3WdA5UwHpVMn184qNVNz3ZL/z93gbKLy8repqxlcDO3fSvB0oD8K48fm2quRJzCaqq4Pjb7iPhxobLZ00R/hTN/GddpgwZVrWQ5phv+6nhRYqqKBCKnoEjI+qPAo66M4CAhdPGCbDelwH9Bl0TnYuG51ypgLgQ5lUHMDTIvIE4Be+vRJYlj2T0mTcOMa3ttLcNgx2NpsTKDDi4wX1zduoHN9oI4Issjm8maAEqRS3/mVfdB9RokS8P/7iLL9TF4QIESfxkBAwPqryKI7iKFa3r3bTSr0EloE+g87JzmWjQ85UAHwo068DUNVPi8hluJx/gCWq+mB2zUqT6mrG4zmB1laLCxQg04MzYdNM2OTSSR9qtVTSbJAYBPYlGPyf/uigN5kG6D1g3F9gebDnMk2mAuBDmVS1gFYDf1LVfweeEJHC71Grqxlf3gJtbW4kkCV2727hzjtvH/T9P/3p99i3b1/adtTXP8MLL6xIu518MHnTAnY9N4P162H5ZhOkyyRVUnVAh+7/jNfzid+PP95bwDixzfjrBnsuG+T6ecVIvw5ARP4VuB/4qXdoIvCHbBqVMcaNcxlCfnA4vkxWhti9u4W77hq8A1iy5Hvs358ZB7ByZXE6AHAjggkNC2neYMqkmWRKcIqb8tEIqkqQIIoSIEA55d0dZND7459T1T6lGhLbjL9usOdy8d1NeuJAUokBfAqYAzwPoKqviciErFqVYcYfFnTBYUZmvO3//M8b2bjxdebPP5kzzzyXm266lR/96FYefvi3dHR0cOGFl/HlL9/M3r17+dd//WfefHMr0WiEG274fzQ3v81bb73JBz4wnzFjxvHgg8t7tH3LLTfyxBMPEwgEOeus87j55u+wc2czX/ziJ9i2bbN3zfc47LCJ/PKXPyEQCHD//b9i8eIfMnfu+zL+XXPB5E0LaHp9DeDWElhsID16SDFoO8PKhnGYHNad6RMvzVAlVT3OJcvg8Z1EWMLd2UP+uTItQ0QIE+418ybTWTl9ZfokfnfLAjqQVBxAh6p2iri5QhEJAsW37LWy0k0HtbVlNE30a1/7JmvXvszy5S8CsHz5k7zxxms88cQLqCpXX/2PPPtsHTt3NnPooYfzv//7JwD27NnNqFEH8ZOf3Mbvf7+csWN7/qN8551dPProg6xYsRYRYffuFu95n+XjH/935s79B7Zu3cyVV55PfX0TH/3oJxgxYiSf+tQXMvbd8sX0oKtc5scGAIsPpEEmpBgSM2qiRIlKlOPKjwN6ZvdExQWZjys/7oDnZloWor9MH5OhSE4qDuD/ROQrwDARORf4JPDH7JqVBbw00ebt4ZRXDQ+GZ555kmeeeZL3v/8UAPbubWPDhteYO/d93HTT5/nGN77Meedd3O8b+qhRB1FZWcXnPlfDuedezHnnXQxAXd0y1q17pfu61tY9tLW1Zfx7FAKT41JHfakJSx3ND8kyaiC32T2p2mUdf/+k4gBuBGqANcDHgUdV9WdZtSqLdK8abiPjowFwyqCf+cwiPvrRjx9wbtmy1Sxb9iiLF3+N973vbL7wha/32U4wGOSJJ17gL395mj/+8X7uvPNH/P73fyYajfLYY89RVVU6gSw/ddTqGOePdDJ/8mmXkZxUsoA+DNynqleo6uWq+jMRuTjbhmWV6mrnCCDtDKGRI6tpa4sFl+fPP597772z+618+/ZtNDfv4K233mTYsOFcccVH+NSnvshLL63u9X6ftrY29uzZzTnnXMgtt/wPjY1/B+Css87j5z//Yfd1a9a8mLSdoURi5TKrWpY7Cim7J1W7jP5JZQTwQ+DzInKVqjZ5x74BPJI9s3LD+JH7014wNmbMWObMmUcodCLvf/8F3HTTrbz2WhMXXXQ6AMOHj+T223/FG2+s5+abv0hZWRnl5eV8+9t3AHD11bUsXLiAQw89vEcQeO/eVq655lLa29sB5eabbwPgv/7rB9x446c488yTiETCzJ0b4jvf+Qnnn38J1113OY8//lBRB4H7w0YD2SU+oOoHcyNECBCgQztcwJho9/qBLu2iQiro0i7AvX13aRdddNGlXQfIRGTazr3RvUSIUK7llEt50uI0xoGIavJ4roj8DTcFdA9wk6r+TkT+pqqn5MLAeGYfcYQ2fPWrPY41nXwy0488cvCN+quGYchqCL3xRhMvvjg932ZkHF9WwsfiA+kRH1CNaIROOlGUKqpQlA46DriniirKpIywhgkSpIsuIkQIEqRCKro75N4Cwpmws4wyOrXTFZknwIiyEZbp0wsnVZ20SlVnJx5PZQSgqrpaRM4E7hWR90KctGCxk6ghlKXgsJF5/NEAQFN4DQ045VFLHR0c8QHVDnWdfRlldOHe7gXpUctXELroYrgMB6BCKqigok+ZiEx1yomB30qpJKjO4ZxadWpGnlEqpBID2A6gqjuB83EpoCdm06i8MG6cWzncYcGjYiRxIZnFBwZOu7b36OB9/Lf4xM/x1/mB1/g2fDIdlM3FM0qFfh2Aql4U9zmqql9U1VQlJHJCf9NYKTNuXFZXDeeDjP1uigRfVqJhXYvJSgyQ+IBqMpmIxHOQ24CwBX4zR58duYh8z/v5RxF5OHHLnYnJqdq3j12trRnr6MYfFsyJhlAuUFVaW3exb19p/cfwRwN7Wi1baCDESyf4qZVRopRT3kMmIvFcruUeTOIhcySLAdzj/fxOLgwZLJM2bGAr0Dx8eIZbbqV1RxS2ve5WEVcWZye6b18VGzZMyrcZeWGCJztdCrGBRDmE+MIuqRZ96SGdQDvDdFh3FtAw6SkhEX+uQiqyIveQqsRDmbp4wKtdr7I5vNmCwAOg3yygrD5c5E7gYmCHqvYbV+gtCyjrLF3KkmNvjatsYhQjO2a7bKGhmCmUmBXTpV100EEFFW5unHZX9IUKgO5z2crSyQSJ36kvO1O9rtTpKwso2RTQGhF5qa8tQ3bdBSzIUFvZ4dhjob4eFi/OtyVGGkxoWDhkYwPxWTG+CJtf2KWLru55+rD3p0fRFwlQRlm3pEOhkPid+rIz1euM3kkWzL0YVxj+cW/7sLc9BjyaiYerah3wTibayhqhELWLxrrPixd7wjRGMTI9OJO1t/WMDQyF+EBiVkyyjJ3EgjBQmBk0qWb6WEZQeiQrCr8JQETOTVj09WURWY3TCMo6IlIL1AJMGTMmF4/sldpFY6GujiX13gGbEipKDixJ2UIDLVw6o3jjA1VS5XLvveU5/lSI3zH6tYDjM3b6K/qSbxK/E/RdnCaV64zeSSWdU0RkXtzOGSnelxFUdYmqzlbV2eNHZl7Pf0CEQtRO+IObEjKKmunBmUzetIAJDQvpaI4VoSnGQjR9FX3x6/5GifbYjy/60hHtoF3b2Rvdy+r21ewM78yqrTvDO1ndvpoV+1ckfV6qmT6WEZQeqawEvg74hYgc5O23eMdKk5oaapcuZcliYMIEqKnJt0VGmkzetIC6e2DXGWsYO7eR9bhC9RMOKY4aBIlZMUEJdmvzJObLx9OhHU5HB6ejk6iln2n60+5P9p36yiayoi/pkTQLSETKgMtV9be+A1DV3Rk1QGQq8EjBZgH1RV0dS+q9zmHRovzaYmQUP8zj6wwV2/TQ6vbV3XIMrdGeCxrLKOseAYwsG9lDtgEgopGsSSrE25WL5xkxBpwFBG7lL/Al7/PuLHT+9wLPAtNEZKuIFM/rdHxw2ALDQ4pQyG3x00MPNTYWTfZQb4HRRMKEcx5AtYBt4ZHKFNAyEfkC8Btgr39QVdPO3lHVq9JtI9/UTviDCwy/9ppNBw1B/Kpk4NYSPNQYK1E5OlCYawp6C4wmEiSY8wCqBWwLj1TkoN/o5bCq6lHZMalvCmoKKIEli3e5D/PmWYbQEKeuDo68+nEAKse35GVxmSp4ZboP2I+fa/elkn0EQVGmBqYyKjAqp4uobNFW/hi0HLSqpiG2XzrULhrrVg3baGDIEwoBm9z6xabXY1ITEw7Jzajg/p8fTNkxaxk3Zx1hnA5/65bRVE7YTVllF0GCjJExdEqnywBSlwHkM0EmcFRl7P2trwDqho4NbI1s7X6G32Zv1/Yl25B4zi8yEyY8oDaM7JCSFISInAicAHSP1VT17iza1SuFPAKIZ8niXTYSKDGawmsYebRbVzDKKyeRDe0hVXjw/96m+tRXECAQhGhEXTQvCmUBNwxI9y1/Q8cGNkY2dlf+8tcSBAgwTIb1aAfo8xnJzpmkQ+4Y9AhARP4DOAvnAB4FLgD+CuTcARQL3XEBMCdQIkwPzoRNM6nzJBSPvPpxHmrNfDqpCIybs47OMETCASJhCFa4KR4J0N1hA2yNbGWkjuxRPCXVAi1bI1sR7w/EHEC3hERcO0Cfz0h2LlE8bjB2GumRShD4cuA9wN9U9WMicgjwq+yaVeTU1FDrrxqur7c00RKi2997awvApZOupzFjsYIwYQJBIeJP7UvPeED8de3a3i3f7JNK5o2vGZSM+HaSPSOV5w/WTiM9UlnRu99LBw2LyChgBzA5u2YNARI1hIySIzGdtGFdS3c6aTo6REGCsc4fQN3UUG/XdRdPiTufSuZNYmfcG6kUgUm1eIsVeckPqTiABhEZDfwMWAWsxuXuGynQ7QSWLs2vIUZe8WUn/LKV9X+FhnUtNHb0LFjTn1SCKux8YRoKBIIRKqoi3Z27RtxUTSSsRCMwqWySk0rQKO+8A+82l6UslTApMAmN++PjS0ikWgTGJB0Km1SygD7pffyJiDwOjFLVTMlBlwS18xrddNDixTYdZLi1BUDdPdDRnU7ayKSj2umgI6lUgghE1x9PK3RnAZUHYllAGugiGi5n1b0nsbXraK757E62PHoQnYet55Cjd/davKU3jqo8CjpIOQsIkssxmKRDYdJnFpCIJF2braqrs2JREoolC6hPli5lyY5/Mg0h4wCawmuY9E9/IVgeBRUqKwJUSEUPqYT4lchnTZ7R5zoAVbj7++N47LcHdZ+/4J93c81nd/YaKzCGPn1lASVzAMu9j1XAbODvgAAnAQ2qenqWbO2ToncAYBpCRp+0X/AL2t6pJDiineDI/QTKABTKlNbN4wFYe9tCjrz68X4XoKnCVfOO7t6/t/516/xLmAFrAanqfFWdD2wHTvUkmWcBpwDFX0UjX3jB4doJfzANIaMHsncUIw4KUyXDaH97DHu3j2H/rlHsWX84a29byPiVCwmF3BTSrudmsHJt79XN/BFA4r7/rpfHKrBGgZFKGug0VV3j76jqyyIyPYs2lQ719bZq2Oim6YFZTLliOcMqYMTIIATC7O+MsP3pWQAsWwbnnOOmeo4PzGTZN2fSOW0lez4QcwKqsOJX03jpiUpOOn8dFcPDrPvL4fzvXcNofCvM6R9ex7O/nkbFiDCnfeD1jKWm2ire4iQVB/CSiPycWO7/hwELAqdLTQ21wJLFO1xw2FYOlzSq0PziVF5+eT5nf2YVoyftoWXrKJ7+wSwmdE0FYOVKd+055zhn8MILMIfTGL/ytB7TO2Ob4YwZcPa0MTz9NOzfBONHw5gdY1jzrVN45QWYMwd2PlvVLWORzqrlgej8G4VFKmJwVcD1gN871QF3qOZ+hcaQiAH0hgWHDZwT8Dt2nzlzXIcPfZ/rbW7fDwona9O/b8sR/ccUkmE6/4XPoOoBAKhqu6r+j6pe5m3/k4/Of0hTU+PWC+zYYYXnSxiRWGfv43fUyc711VZ/bfr4MYWGdb3HFPrDdP6Ll34dgIjME5GnRORVEdngb7kwrtToDg7X19vCsRLEf1uPZ9kydzzZucG2Gc/04EwmNCxkTys81Ng4oFXKtoq3eEklBrAU+HfcKuBIP9ca6WKxgSFHMu3++GPLljnfP2+ee0t/6im373fWK1fGpm/ip3XOOcddUxb3OheNumd0xwr6uC/RjgkNC2kKrxlQbGBKcIqb81d6KHnaKt7CJxUHsFtVH8u6JUYPrL7A0KCuDjo6Yp2t39FXVvb06yLw8MMQicCZZ8Jf/gLhsJsV/OMf4eSToasLKirctWefDa++Cps3w513Qns7XH+9cwLRKDy8aiPv+dAqTnv/Hk7eX8HIkUpHeRehM0cx+r5Z7F8/tc/po+nBmdAws7sCWn81kW0Vb/GSihbQchG5VUROF5FT/S3rlhkHxgaMokLVdf4vvBCbdvHfwDs6ek7DdHW5jj0SgS98AfbujTkEEWhrg5074eWXXQf/9NPw7rsweTLs3w9r18Idd8Q6/9mfXE7V6L1UVsLIw3fBqHdRhWjVXk6sWc4ZV2zs1/4JDQsBaOzoPy4wLjiOU6tO5YxhZ3Bq1anW+RcJqWQBLe/lsKrq+7NjUt8M2SygVLBMoaIklSwcn64u+PznnXPwEYHDD4dAAEaPdp2+f5/fjqrr/Neudcdr73+AcZP2cvCocrT6XRBvfl7LkNaD0UAXZe0jqKz7YL/219U5Oet8lL00Mkc6WUDze9ly3vmXPDYaKEoGkr1TXg7f/W7PY37nD26KJ/4+v52yMnfOZ8zkPYwe5c3ulkXA1/Uv80J4kSA6Yk9K9odCdEtZD1a+2ihcUpkCQkQuEpEvicjX/S3bhhm9050ptHixZQrlmcTBc2+D6f6ycKJxyTP+CCCeN99000AAP/5x7D5VePJJ9zMahdtvj93zzpZRtOzxCgZEA3TrRUc9TxIII3tHpfQdIZYm2hJpSfkeozhIpSTkT4DhwHzg57gKYS8kvcnILomZQiYql3NSCe7GT//0loWzaZNr4/rrXSfvT/+UlcGMGbBmjaf/v9ONDjZvdiOCs86Cl16CRx91bY0eDU1NcOih8LWvwcMPz+LgTy7n3T0wunwYDG8FBPYPQwNdEIgQWDdrwN+5PdrO6vbVFugdQqQyAjhDVa8B3lXVm4HTgeOya5aRCjYayA+pBndFnEOIn/M/5xy3X1HhrvWDt4GAy/oBmDYNgkEYOdIdDwRiKZ5tbS7jZ/16d380Ci0trr25c90z/nHWVBpun097ywi3iGzPWNhzsJsuah9B+er5BN+eOqDv3PFuNe92ttHS2dpD7iGxYI1RXKQSBH5eVd8rIs8BHwB2AY2qekwuDIynpIPAyTCJ6ZwzkOBuX+sAotGewVuAo4+Gz3zGnX/qKVixwr39q7opol273HXRqDs+fry79q71/MoAABz0SURBVLTT4NxzY8+JRnuuC0iXjtAD7NW9VBzcyvCqA2sVGIXNoIPAwCNeSchbceUgNwL3ZtY8Iy3iJaZtNJATBhLcTTzm7ycGb8F1/mVl7ppzz3WdvH/P178ek4UoK4t1/tCz8/fbziQ6Yg/DKoKE24axrz1Ca8d+k3sYAqTyz+Tbqtqiqg8ARwDHA/+ZXbOMQVFTQ+28RssUygGDlWaIxx8BxOPn8ie2rwq33BKThYhGXWzAf95Anz1QZO8oCMRqFUSisLtjP0IvHs8oGlJxAN0F4FW1Q1V3Y0XhCxcbDaTFQDJ7/ODuV74Cs2f3jAl0dcWuj0Z7ZvuEw7HOv6kJjjsOvv99OP54t3/77S7D5/nn3dTOjTe6e956C8aOhUsucXP+nZ0uAJz47GwQWDcLAhE00MWIkUqgvZpoRzm7tlcMSkDOKAz6zAISkUOBicAwETmF7mRiRuGygoxCpqaG2ro6K0Y/AAYi2xAf3P3mN11g9owz3PGlS12WzqGHwiGHwL597p5hw9z9TU0u0LtnD4wZ44794hdwwgnwxhvQ2gpbt7rOXdXpAU2a5ALAkye7di64wCmEDBsG553npnwqK/tWB02X4NtTYfV8ItNWoSP2IHtHMXxdiBW/m8rxNzjJCFssVnwkSwM9H7gWmAR8l5gDaAW+kl2zjIwQClEbwq0iXoytIk5CfGYPJBRdmXNgIDcUik3FdHTA22+7gO3Xvgb33++OvfUWDB8eC/LOmAEbN7qOfN0694zHHnNTOSec4GQe2tuhutp1+H/+M2zZ4nR/pkxx7R1xBLzvfa69c8+NzfUnk4bOFMG3px6QPRQKAYMQkDMKg1SygD7ozf/nHcsCSgPLFOqXgWT2xBOJuPn5t96KtVNZGVPk9KeD/BW9w4bF0kWjUTedM26cu9aXewA3EhBxDiFVW/KJLxsxqhpzAgVGOllAk0RklDh+LiKrReS8LNhoZJPE2IAVnTmAgRZd8QkE4P/9v57tfPe7sfvKy2O5/GVlbsrIv66sLNb5Q0zuwe/4/c4/VVvySSgEa29zNQVSEZAz8k8qDuA6Vd0DnAeMBa4GvplVq4zs4WcK1ddbplACA83s8Y9HIvCNb/Q8fsMNsfNdXe4aPxh8442x6xKzeW6/PZbp09rqNp+nnuppSzazfgZLKAS7npvB+vVYcLgISMUB+O8cFwJ3q2pj3DGjGLHRwAH0ltkzZ07f2TV1de54OOxm1N56C0aMgCuvjGXoqMYCueCyfKqqYlM7M2a4n342T3U1vPKKcwqnnRYLRJ92msv+efrpmBPw7S3Evzq/uljzhtE81Nhoo4ECJhUHsEpEnsQ5gCdEpBoS6r8NEhFZICLrRGS9iNyYiTaNAeCPBl57Ld+W5J1ksg2J2TXxAeM//zk2tRMIuLf9CRNiUziVlW475BAXED744Ni0T1mZcxrjx7u4QEWF2w45xDmK97/fBYArK+Goo1ybb7zhbOirrkAh4YvIGYVLKkHgMuBkYIOqtojIWGCiqr6U1oNFAsCrwLnAVmAlcJWqvtLXPRYEzg5LFnv6AhYcTql8o3/c74RVXUpnWVlszn7WLDj/fPf5ySfddX7GzqmnwoIF7vNTT7lz/jPiJR3iNYVU3bUrV8ZsKPSgMMQCw4ClieaRAQeBvXUAqGpUVVeraou3v8vv/P1rBskcYL2qblDVTuA+4NI02jMGiU0HxehLtqG36/yAsQiMGtUzYHv++bFgrp+n77NgQexcooRD/L5/jf/53HN72lDonT+4mMCEhoVWU6BASTYF9GgK96dyTV9MBLbE7W/1jvVARGpFpEFEGprb2tJ4nJGUmhrnBOrr821JURAfMO4tYOvHDZIFlgcSdM6E9EQ+mbxpAR3No3n1TaspUEgkWwj2HhFJVjZIgNTKCqWBqi4BloCbAsr280qamhpYvMuNBObN67n8tQBJdbom08/zO+Pnn3fTMOBiAX7wFmJTQ+CmbRLrASQ7Bz3f7vurK1AMIwGAN+5ZQKWtGi4o+nQAqhrI8rO3AZPj9id5x4w8UrtorLdoDDcaKNC4QKqyDdl63ubNLgOoosJN78yf72LpW7bAddfFgspwYGAZ+j8X36H3FaDu7dpCxlYNFx79VgTLIiuBY0XkSFzHvxD4UB7tMXw8CYnu4HCBMVDZhkw/7+yznWTDzp3Q2OgWby1bBrt3O40f36b4N/j4ufxUz8XjS0+kcm2hMz04k7rbZnYHh4380W8WUFYfLnIh8D0gANypqv+V7HrLAsoxS5eyZMc/FaSG0GBlGzL1PFWX0tkSN6VdDFk5hcSWIx6ncnwLxxwDMyptJJBN0pGCyBqq+qiqHqeqR/fX+Rt5IL6+QIHJSg9WtiFTzxM5sJiLdf4Dw18nsH69SUfki5QcgIgERORwEZnib9k2zCgQQqGYEyigFNGBZsVEo33vJzvnt+c/L34/sZhL/PnENoslWyfXTA/O7HYCliKae/qNAYjIvwH/AbxNbAWwAidl0S6jkAiFqH1tqQsMv/Za3qeDBpoVs3Spm7O//noXsPWLsVRVufN9nTv2WDf3f/bZTobh+efdtM+JJzrp5rVrnbyDv+8/v7zcxQZmzozN3WczQF3sTA/OpOk5aKCRlmNabDooh6QyAvgsME1VZ6jqTG+zzr/UqKlxGUIFMBIYiGxDNOo6+LVrY+UW/ULs+/bB/v29n9u/320vvOA6/4qK2Jx/Z6fr9I8/3un5dHY6CefRo911jY2ujTVrXJvFINuQb6YHZ7L2toWsXw8PNTbaaCBHpCIFsRw4V1XDuTGpbywIXAD4geECWCeQ6jqA+I7d5/jjY3P4fZ0TSR5ojkbdqCF+XYD/fF/X39+3AHHqWHA48wxGCuIGEbkB2AA8IyKL/GPecaMUiZeTznNgOFXZhrKyAwO2/pRPsnP9BZp9eYfeUjJ9B9LbfUZyJm9awNrbFrLj7XxbMvRJNgVU7W2bgaeAirhjI7NvmlGwxAeGiwB/BBCPP+WT7Fyqgeberrvjjp7XFZNsQyEQCsGeVpsOyjbJVgLfDCAiV6jq7+LPicgV2TbMKHD8wHCB1xqOn/7xp3b8/R//2L2Vr1t34Lnbb3dB4IaG5IHmxID02Wcf+Lynny4+2YZCYIKtGs46qQSBe9MCKEx9ACO3xK8TKNDqYmVlLqPH74z9KZ/jj3f6/MOG9X5u2DC39RdoTgxIl5XFAsQzZ7r9vgLURv/4xWVsNJAd+gwCi8gFuCIw/wz8Ju7UKOAEVZ2TffN6YkHgAmbpUpZQW9AjgXhJ5vj9ZOcGUh8gMfsovs1sC9WVAk3hNYyd22hF5wfBYFYCvwmsAtq9n/72MHB+Now0iphjjy3IFcM+ZWV97yc7N5D6AMmeZ51/+vipontabdFYpkglDbRcVbtyZE9SbARQ+Fh1MSPbWJrowBlMGugaEXkJVxP4pcQtq9YaRUvtorHuQwHJRhhDC9MQyhzJpCAu9n5+yvt5j/fzIzgpCMPoldp5jQVfT8Aobnz5CGhk9LRtVlxmkPQ5AlDVTaq6CbcK+EuqusbbvgyclzsTjaIjFIqNBAo0JmAUP76QXMO6FhsJDJJU0kBFRObF7ZyR4n1GiVOoUtLG0CFeQ8gCwwMnlY68BrhdRDaKyCbgduC67JplDAn8kUABrxMwip9QCDqaR9OwrsWcwADpVw5aVVfhCsQf5O3vzrpVxpCidtFYt06gwFcNG8XL5E0LaHp9DS3HNFo8YAD06QBE5COq+qtE4TfxEppV9bYs22YMJWpqqPWLzRtGFmheMZP1cxtZTyOXzrD00FRINgU0wvtZ3cdmGAPDl49evNjSRI2MEwo5/aCO5tE81GhB4VRIJgb3U+/jt1S1PUf2GEOc2kVjwR8JWHksIwtM3rSAHePvY/nmRpOM6IdUgsAvi0i9iHxTRC7yYwGGMWjiRwKGkQV8yQgTkEtOvw5AVY8BrgLWABcBfxeRF7NtmDG0qV00ltoJf3BOwNJEjQzjTwftem4GLZGWfJtTsPTrAERkEjAPeB9wCtBIT3VQwxgcvpy0YWSR9eth+Wb7d9Yb/aaB4iqCrQT+W1U/kWV7jFzw3/8Nra0HHq+uhq98Jbe2hEJQ760TKIA6w8bQYnpwJnW3zeT4G+5jW9gkIxJJJQZwCnA38CEReVZE7hYRS+QuZlpbYeTIA7fenEIO6J4Oqq+37CAj48QvFDPJiJ6kEgP4O/BL4BfAn4Ezga9n2S6j1KipiTkBw8gwfqF5UxDtSSoxgAbgWeAyoAkIqeoR2TbMKEH8FcK2TsDIAqEQ3TLSlhnkSCUGcIGqNmfdEsMgYZ2AyUkbGcaXkbZC845UpoCs8zdyS7yctGFkmPjSkqVOKiMAY6hRXR0L+O7e7SqWgytc679x5yMjKIHaCX9wAnI+NhowMkQoBFs8yYhSLi3Zb03gQsJqAmeBRYtcBlAibW0FtVK3u9awpYoaGaSuDo6/4b4h7wT6qgmcTA30A8kaVNXfZ8Iww0iFA2IDJittZIBQCJqem0GplpZMNgV0SZJzCpgDMHJLKERtCPfa9tofWLJ4hzkCI23iA8Mtx7QM6ZFAIsnUQD+WS0MMI2VCIecM/BHB4sUWHzDSwl8xzA33ldRIIKUgsIhcBMwAqvxjqvqNwT5URK4AbgKmA3NUtWGwbRkeyeQdNm+OBXp74513Yp8DgdTazHOAGIiNCKzamJEB/MBwAy0lMxJIZSHYT4ArgX8DBLgCSHch2MvABwBb7ZMpksk7qLoMH3/LRJuFhInKGRli8qYF3YvFSkFALhUtoDNU9RrgXVW9GTgdOC6dh6pqk6quS6cNI0MEAlBR4bZAACZOdNtBRVb2IRSKFZ+3VcRGGkwPzmRCw0KaN4we8vUEUnEA+72f+0TkcKALOCx7JvVERGpFpEFEGprb2nL1WKMI6SEqZzUGjDTxRwMN61qG7GggFQfwiIiMBm4FVgMbgXv7u0lElonIy71slw7EQFVdoqqzVXX2+N7y1Q0jnpoalzJqowEjA8SPBoaiiFwqQeBvq2oH8ICIPIILBPdbI1hVz0nXOMMYLLWLxrrgcD3w2msWHDbSou31iawf3wLHNA6p4HAqI4Bn/Q+q2qGqu+OPGQXCu++6bJ/Ezc/wUY1t8UQi0NnptkgEtm51W1uby/Zpaztwq67O/fcbDImjAcMYJL5+0Pr1rs7wUCHZSuBDgYnAMBE5BZcBBDAKGJ7OQ0XkMuCHwHjgTyLyoqqen06bJY8IlJfH9js7+752yhT304+p9CUFUQipnhmgezRQZzISxuAJhYCGhWw54nEeamxk9rTRRb9eINkU0PnAtcAk4La443uAtHoGVX0QeDCdNgxjQBx7bKzYjDkBIw0mb1rAFh7n1eoWJk4pbgfQ5xSQqv5SVecD16rq/LjtUtMBMoqOUMgqjhkZ4417FrCnlaJPE00lBlAvIktF5DEAETnBagIbRYlfdnLxYksTNdIiFIIJDQvZ9dwMXn2zJd/mDJpUsoB+4W2+DvOrwG8A+x+UC5LJMWzfDuGw249Gk8/7x7N5c8/9YpWCGAw1NTENIcNIk+YVMxk7t5HlmxuLsrpYKiOAcar6WyAKoKphIJJVq4wYyeQYwmEIBt2W62cXM34MwNYJGGnijwT8VcPFtlYgFQewV0TG4iSgEZG5wO6sWmXkj2KVghggtmrYyCTxGkLF5ARScQA3AA8DR4tIPXA3ThjOMIobWzVsZJD4tQLFEhjud+5AVVeLyJnANNxagHWq2pV1ywwjR9iqYSNTFJukdCpy0FXAZ4BbgJuBT3nHDGPoYKMBI0NM3rSgx6rhQh4NpBI9vBtoxa3cBfgQcA+uLoCRbaqrY0HX3btjUg4iA8v8SRV/dbAv99BXFtAQxUYDRiaIXzVcyKMB0WSVogAReUVVT+jvWC6YfcQR2vDVr/Z/4VBl0aK+ZRsStW6uvz6WHZToJCoq3M9wGO64I/N2DhGWLN7lPswzCQlj8NTVwZz/uC+vaaInVZ20SlVnJx5PJQi82sv8AUBE3gtYCUdjyGOZQkamKNRVw6k4gFnAChHZKCIbcUqgp4nIGhF5KavWGUa+sdiAkSbxq4YLrbhMKg5gAXAkcKa3Hekduxi4JHumGUbh0GM0YBiDwE8T3dNaOGsF+nUAqrop2ZYLIw2jIPADwlZbwBgkoRAFtWAslRGAUSgMpEBLMOiCvL5WUDz+8WxJSAxhaheNdR9sOsgYJNODM7udQL6ng/rNAiokSj4LyCgc6upYUu9ldSxalF9bjKJlyxGPUzm+hWOOIatpoulkARmGkUgoZKMBI23iNYTyMRowB2AYaVC7aCy18xpdcNhiA8YgiA8O5zpN1ByAYaSLjQaMNAmFoKN5NA3rWnIaHDYHYBgZosdowBaOGQMkHxpC5gAMI5P4owFbOGYMAn/RWEfzaFoi2S81aQ7AMLKAjQaMdGh7fWJOAsPmAAwjW9howBgk8YHhbE4HmQMwjCxjonLGYEjUEMpGcNgcgGHkAhOVMwaJv3J4x9uZb9scgGHkEBsNGINhenAmzRtG81BjY0ZHAuYADCPX2GjAGATZWDVsDsAw8oSNBoyBMj04kwkNCzM2GjAHYBj5xEYDxiDwRwPpYg7AMAoAGw0YA6V5xcy0Vw2bAzCMQiFxNGAYSYhfNTzYNFFzAIZRYHSPBhYvttGA0S/xGkIDHQmYAzCMQsRGA8YA8EtNNqxrGZATMAdgGAWMjQaMVPEXjDWsa0k5TTQvDkBEbhWRtSLykog8KCKj82GHYRQFNhowUmSgGkL5GgE8BZyoqicBrwJWVNUw+sFGA0YqDERDKC8OQFWfVNWwt/scMCkfdhhG0VFT42SmbTRg9EMqGkLB3JnTJ9cBv+nrpIjUArUAU8aMyZVNhlG4hELUhoClS1myGJgwAWpq8m2VUYBMD86k6am+z2dtBCAiy0Tk5V62S+Ou+SoQBn7dVzuqukRVZ6vq7PEjR2bLXMMoPmw0YKTA9ODMPs9lbQSgquckOy8i1wIXA2erqmbLDsMY0thowEiDfGUBLQC+BPyjqu7Lhw2GMaSw0YAxCPIVA/gRUAk8JSIAz6nqJ/Jki2EMDWw0YAyQfGUBHaOqk1X1ZG+zzt8wMoWNBowUsZXAhjEU8QrSd68bMJlpoxfMARjGUMYfDdTX22jAOABzAIYx1LHRgNEH5gAMo1Sw0YCRgDkAwyglvNEAYKMBwxyAYZQitYvG2mjAMAdgGCWLjQZKHnMAhlHi2GigdDEHYBiGjQZKFHMAhmF0Y6OB0sIcgGEYPUkcDRhDFikmJWYRaQY25dGEccDOPD4/39j3t+9v3784OUJVxyceLCoHkG9EpEFVZ+fbjnxh39++v33/ofX9bQrIMAyjRDEHYBiGUaKYAxgYS/JtQJ6x71/a2PcfYlgMwDAMo0SxEYBhGEaJYg7AMAyjRDEHMEBE5FYRWSsiL4nIgyIyOt825RIRuUJEGkUkKiJDKiWuL0RkgYisE5H1InJjvu3JNSJyp4jsEJGX821LrhGRySKyXERe8f7dfzbfNmUScwAD5yngRFU9CXgVWJRne3LNy8AHgJIQixGRAPBj4ALgBOAqETkhv1blnLuABfk2Ik+Egc+r6gnAXOBTQ+nv3xzAAFHVJ1U17O0+B0zKpz25RlWbVHVdvu3IIXOA9aq6QVU7gfuAS/NsU05R1TrgnXzbkQ9UdbuqrvY+twJNwMT8WpU5zAGkx3XAY/k2wsgqE4EtcftbGUIdgJE6IjIVOAV4Pr+WZI5gvg0oRERkGXBoL6e+qqoPedd8FTc8/HUubcsFqXx/wyglRGQk8ADwOVXdk297MoU5gF5Q1XOSnReRa4GLgbN1CC6k6O/7lxjbgMlx+5O8Y0aJICLluM7/16r6+3zbk0lsCmiAiMgC4EvAP6rqvnzbY2SdlcCxInKkiFQAC4GH82yTkSNERIClQJOq3pZvezKNOYCB8yOgGnhKRF4UkZ/k26BcIiKXichW4HTgTyLyRL5tyiZewP/TwBO4AOBvVbUxv1blFhG5F3gWmCYiW0WkJt825ZB5wNXA+73/7y+KyIX5NipTmBSEYRhGiWIjAMMwjBLFHIBhGEaJYg7AMAyjRDEHYBiGUaKYAzAMwyhRzAEYOUNErhWRw1O47i4RuTzV4xmw6ytxn6emonrp2fKGiHwiyTUnZzJl0Pv9/SjNNp7xVVxF5NF01WxF5CwRecT7fKWnmPpIOm0aucMcgJFLrgX6dQB54Cv9X9IrX1TVZOtATgbyljMuIklX+qvqharakqnnqepvgH/JVHtG9jEHYAwK7015rYj8WkSaROR+ERnunZslIv8nIqtE5AkROcx7c58N/NpbTDNMRL4uIitF5GURWeKtukz1+Qc8wzv+jIh8S0ReEJFXReR93vHhIvJbT9f9QRF5XkRmi8g3gWGeTb6uU0BEfubpvz8pIsNSsOcK73v8XUTqvFXD3wCu9Nq+UkTmiMizIvI3EVkhItO8e68Vkd+LyOMi8pqIfDuu3Y953+MF3KIk//gl3nf4m4gsE5FDvOM3icg9IlIP3OP9nu/z/o4eBIbFtbFRRMaJyCfiFjm9ISLLvfPnefauFpHfidPD8esjrBWR1ThpcKNYUVXbbBvwBkwFFJjn7d8JfAEoB1YA473jVwJ3ep+fAWbHtTEm7vM9wCXe57uAy3t55l3A5Sk847ve5wuBZd7nLwA/9T6fiBPym+3ttyV8rzBwsrf/W+AjfdkSt78GmOh9Hu39vBb4Udw1o4Cg9/kc4IG46zYABwFVwCac/tBhwGZgPFAB1PvtAQcTW8j5L3Hf+SZgFTDM278h7ndzUsL33giMi7OvHPgLcAkwDlfzYYR37svA1z37tgDHAuL9fh6Ja+Os+H3bCnszMTgjHbaoar33+VfAZ4DHcR3sU94LfQDY3sf980XkS8BwYAzQCPwxhedO6+cZvmDXKlyHDvAPwPcBVPVlEXkpSftvqOqLvbSRjHrgLhH5bdzzEzkI+KWIHItznuVx555W1d0AIvIKcASuE35GVZu9478BjvOunwT8xhv5VABvxLX1sKru9z6HgB8AqOpL/Xzv7wN/VtU/isjFuAI49d7vuAInB3E87vfzmmfTr4DaJG0aBYw5ACMdEnVEFPdW2Kiqpye7UUSqgNtxb6NbROQm3NtlKvT3jA7vZ4TB/RvviPscIW7apC9U9RMi8l7gImCViMzq5bJbgOWqepk4bflnkjyzP7t/CNymqg+LyFm4N3+fvf3Zm4g4hdsjcLpH4H7HT6nqVQnXnTzQto3CxWIARjpMERG/E/4Q8FdgHTDePy4i5SIyw7umFSekB7HOfqc3tzyQ7J5kz+iLeuCfvetPAGbGnesSJ/k7aETkaFV9XlW/DjTjpnDivy+4EYAvJX1tCs0+D5wpImM9+67oo62PJmmjDvd3g4iciJsGSrR9Fm6K7COqGvUOPwfME5FjvGtGiMhxwFpgqogc7V13VWJ7RvFgDsBIh3W4GqlNuDnpO9SVTbwc+JaI/B14ETjDu/4u4Cci8iLujfdnuBrDT+Bkl1Oin2f0xe04p/EK8J+46abd3rklwEtxQeDBcKuIrBGXQroC+DuwHDjBDwID3wYWi8jfSGFkoqrbcW/2z+IcWFPc6ZuA34nIKmBnkmbuAEZ6f0ffwE1pJfJp3BTccs/Wn3vTTtcC93rTRs8Cx6tqO27K509eEHhHf9/DKFxMDdQYFN4UxiOqemKeTUkJccXdy1W13Xt7XQZM85zJYNq7C/f978+gmUWPNx31BVW9ON+2GP1jMQCjVBiOe8Mtx81vf3Kwnb/HbuAWERmnydcClAzeKOc/6H2UYRQgNgIwDMMoUSwGYBiGUaKYAzAMwyhRzAEYhmGUKOYADMMwShRzAIZhGCXK/we3Was4omCxrwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "filenames": {
       "image/png": "/home/jeroenf/Projects/bookflow/iris_book/_build/jupyter_execute/primer_mlflow_5_3.png"
      },
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import mlflow\n",
    "\n",
    "# set up some parameters for my code\n",
    "svc_pars = dict(kernel='rbf', random_state=0, gamma=.10, C=1.0)\n",
    "knn_pars = dict(n_neighbors=5, p=2, metric='minkowski')\n",
    "algo = 'knn'\n",
    "\n",
    "# some free text that you can save with a run\n",
    "notes = \"I think an knn will work better\" \n",
    "# you can define your own tags as well. In this case, \n",
    "# I'm reminding myself that this is not a serious run (but a test for example)\n",
    "tags = {\"valid\": False} \n",
    "# set location to save the run data\n",
    "mlflow.set_tracking_uri('../iris_project/mlruns')\n",
    "# name of my experiment(= grouping of runs)\n",
    "mlflow.set_experiment('iris')\n",
    "\n",
    "run_name = f'iris_{algo}'\n",
    "\n",
    "# let MLFlow know this is a run to track\n",
    "with mlflow.start_run(run_name=run_name) as run:\n",
    "    \n",
    "    # -- here is just some code, it's not important for now -- \n",
    "    X_train, X_test, y_train, y_test = get_data()\n",
    "    X_train, X_test = feature_engineering(X_train, X_test)\n",
    "\n",
    "    if algo == 'svc':\n",
    "        params = svc_pars\n",
    "        model = train_svc(X_train, y_train, **params)\n",
    "    elif algo == 'knn':\n",
    "        params = knn_pars\n",
    "        model = train_knn(X_train, y_train, **params)\n",
    "\n",
    "    acc_train = model.score(X_train, y_train)\n",
    "    acc_test = model.score(X_test, y_test)\n",
    "\n",
    "    X_stack, y_stack = recombine_data(X_train, X_test, y_train, y_test)\n",
    "    ## -- computations finished --\n",
    "    \n",
    "    # we can log parameters to this run (inputs):\n",
    "    mlflow.log_params(params)\n",
    "    mlflow.log_param('algo', algo)\n",
    "    # and we can log metrics to this run (outputs)\n",
    "    mlflow.log_metric('acc_train', acc_train)\n",
    "    mlflow.log_metric('acc_test', acc_test)\n",
    "    \n",
    "    # and also model artifacts. \n",
    "    # even if you don't do ML, if you use sklearn, tensorflow or other common frameworks, \n",
    "    # you may still be able to save some useful objects with various log_model methods,\n",
    "    # or with the log_artifact method.\n",
    "    mlflow.sklearn.log_model(model, 'model')\n",
    "\n",
    "    # we can also log plots (and basically any other file)...\n",
    "    plot_decision_regions(X=X_stack, y=y_stack, classifier=model, test_idx=range(105,150))\n",
    "    plt.xlabel('petal length [standardized]')\n",
    "    plt.ylabel('petal width [standardized]')\n",
    "    plt.legend(loc='upper left')\n",
    "    plot_filename = 'decision_region.png'\n",
    "    plt.savefig(plot_filename)\n",
    "    # with this method\n",
    "    mlflow.log_artifact(plot_filename, 'figures')\n",
    "\n",
    "    # and also apply some tags to this run\n",
    "    # the content tag is a special one\n",
    "    mlflow.set_tag('mlflow.note.content', notes)\n",
    "    for key, value in tags.items():\n",
    "        mlflow.set_tag(key, value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You don't see it here, but this run is now saved by mlflow. You can query all the runs through the python API (which we will do in the next section), but there is also an UI where you can view them conveniently."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,md:myst",
   "text_representation": {
    "extension": ".md",
    "format_name": "myst",
    "format_version": 0.12,
    "jupytext_version": "1.6.0"
   }
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "source_map": [
   13,
   26,
   37,
   41,
   48,
   130,
   200
  ]
 },
 "nbformat": 4,
 "nbformat_minor": 4
}