{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last week I scrapped a bunch of data from the Steam API using my [Steam Graph Project](https://github.com/jrtechs/SteamFriendsGraph).\n",
    "This project captures steam users, their friends, and the games that they own.\n",
    "Using the Janus-Graph traversal object, we are able to use the Gremlin graph query language to pull this data.\n",
    "Since I am storing the hours played in a game as a property on the relationship between a player and a game node, I had to do a join statement to get the hours property with the game information in a single query.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```java\n",
    "Object o = graph.con.getTraversal()\n",
    "    .V()\n",
    "    .hasLabel(Game.KEY_DB)\n",
    "    .match(\n",
    "            __.as(\"c\").values(Game.KEY_STEAM_GAME_ID).as(\"gameID\"),\n",
    "            __.as(\"c\").values(Game.KEY_GAME_NAME).as(\"gameName\"),\n",
    "            __.as(\"c\").inE(Game.KEY_RELATIONSHIP).values(Game.KEY_PLAY_TIME).as(\"time\")\n",
    "    ).select(\"gameID\", \"time\", \"gameName\").toList();\n",
    "WrappedFileWriter.writeToFile(new Gson().toJson(o).toLowerCase(), \"games.json\");\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the game indexing property on the players, I noted that I only ended up fully indexing the games of 481 games. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```java\n",
    "graph.con.getTraversal()\n",
    "    .V()\n",
    "    .hasLabel(SteamGraph.KEY_PLAYER)\n",
    "    .has(SteamGraph.KEY_CRAWLED_GAME_STATUS, 1)\n",
    "    .count().next()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now transition to python and Matlptlib to visualize the data exported from our JanusGraph Query as a json object.\n",
    "The dependencies for this [notebook](https://github.com/jrtechs/RandomScripts/tree/master/notebooks) can be installed using pip."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting pandas\n",
      "  Downloading pandas-1.0.5-cp38-cp38-manylinux1_x86_64.whl (10.0 MB)\n",
      "\u001b[K     |████████████████████████████████| 10.0 MB 4.3 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting pytz>=2017.2\n",
      "  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)\n",
      "\u001b[K     |████████████████████████████████| 510 kB 2.9 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: numpy>=1.13.3 in /home/jeff/Documents/python/ml/lib/python3.8/site-packages (from pandas) (1.18.5)\n",
      "Requirement already satisfied: python-dateutil>=2.6.1 in /home/jeff/Documents/python/ml/lib/python3.8/site-packages (from pandas) (2.8.1)\n",
      "Requirement already satisfied: six>=1.5 in /home/jeff/Documents/python/ml/lib/python3.8/site-packages (from python-dateutil>=2.6.1->pandas) (1.15.0)\n",
      "Installing collected packages: pytz, pandas\n",
      "Successfully installed pandas-1.0.5 pytz-2020.1\n"
     ]
    }
   ],
   "source": [
    "!pip install pandas\n",
    "!pip install matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step we are doing is importing our JSON data as a pandas dataframe.\n",
    "Pandas is a popular open-source data analysis and manipulation tool.\n",
    "I enjoy pandas because it has native integration with matplotlib and supports operations like aggregations and groupings. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gameid</th>\n",
       "      <th>time</th>\n",
       "      <th>gamename</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>210770</td>\n",
       "      <td>243</td>\n",
       "      <td>sanctum 2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>210770</td>\n",
       "      <td>31</td>\n",
       "      <td>sanctum 2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>210770</td>\n",
       "      <td>276</td>\n",
       "      <td>sanctum 2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>210770</td>\n",
       "      <td>147</td>\n",
       "      <td>sanctum 2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>210770</td>\n",
       "      <td>52</td>\n",
       "      <td>sanctum 2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36212</th>\n",
       "      <td>9800</td>\n",
       "      <td>9</td>\n",
       "      <td>death to spies</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36213</th>\n",
       "      <td>445220</td>\n",
       "      <td>0</td>\n",
       "      <td>avorion</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36214</th>\n",
       "      <td>445220</td>\n",
       "      <td>25509</td>\n",
       "      <td>avorion</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36215</th>\n",
       "      <td>445220</td>\n",
       "      <td>763</td>\n",
       "      <td>avorion</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36216</th>\n",
       "      <td>445220</td>\n",
       "      <td>3175</td>\n",
       "      <td>avorion</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>36217 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       gameid   time        gamename\n",
       "0      210770    243       sanctum 2\n",
       "1      210770     31       sanctum 2\n",
       "2      210770    276       sanctum 2\n",
       "3      210770    147       sanctum 2\n",
       "4      210770     52       sanctum 2\n",
       "...       ...    ...             ...\n",
       "36212    9800      9  death to spies\n",
       "36213  445220      0         avorion\n",
       "36214  445220  25509         avorion\n",
       "36215  445220    763         avorion\n",
       "36216  445220   3175         avorion\n",
       "\n",
       "[36217 rows x 3 columns]"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "games_df = pd.read_json('games.json')\n",
    "games_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the built-in matplotlib wrapper function, we can graph a historgram of the number of hours played in a game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Frequency')"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 11264 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = games_df.hist(column='time', bins=20, range=(0, 4000))\n",
    "ax=ax[0][0]\n",
    "ax.set_title(\"Game Play Distribution\")\n",
    "ax.set_xlabel(\"Minutes Played\")\n",
    "ax.set_ylabel(\"Frequency\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the vast majority of the games are rarely ever played, however, it is skewek to the right with a lot of outliers.\n",
    "We can change the scale to make it easier to view using the range parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Frequency')"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 11600 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = games_df.hist(column='time', bins=20, range=(0, 100))\n",
    "ax=ax[0][0]\n",
    "ax.set_title(\"Game Play Distribution\")\n",
    "ax.set_xlabel(\"Minutes Played\")\n",
    "ax.set_ylabel(\"Frequency\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we remove games that have never been played, the distribution looks more normal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Frequency')"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 9652 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = games_df.hist(column='time', bins=20, range=(2, 100))\n",
    "ax=ax[0][0]\n",
    "ax.set_title(\"Game Play Distribution\")\n",
    "ax.set_xlabel(\"Minutes Played\")\n",
    "ax.set_ylabel(\"Frequency\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although historgrams are useful, viewing the CDF is often more useful since it is easier to extract numerical information from."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Frequency')"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 10324 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = games_df.hist(column='time',density=True, range=(0, 2000),  histtype='step',cumulative=True)\n",
    "ax=ax[0][0]\n",
    "ax.set_title(\"Game Play Distribution\")\n",
    "ax.set_xlabel(\"Minutes Played\")\n",
    "ax.set_ylabel(\"Frequency\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "80% of people who owns a game plays it under 4 hours. In fact, nearly half of all downloaded or purchased steam games go un-played. This is a cool example of famous 80/20 principle -- aka Pareto principle. The Pareto principle states that for most events, roughly 80% of the effects from 20% of the causes.\n",
    "\n",
    "As mentioned earlier, the time in owned game distribution is heavily skewed to the right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 8528 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "ax.set_title('Game Play Distribution')\n",
    "ax.boxplot(games_df['time'], vert=False,manage_ticks=False, notch=True)\n",
    "plt.xlabel(\"Game Play in Minutes\")\n",
    "ax.set_yticks([])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Zooming in on the distribution we see that nearly half of all the purchased games go un-opened."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 6552 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "ax.set_title('Game Play Distribution')\n",
    "ax.boxplot(games_df['time']/60, vert=False,manage_ticks=False, notch=True)\n",
    "plt.xlabel(\"Game Play in Hours\")\n",
    "ax.set_yticks([])\n",
    "ax.set_xlim([0, 10])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Viewing the aggregate pool of hours in particular game data is insightful, however, comparing different games against each other is more interesting.\n",
    "In pandas, after we create a grouping on a column, we can aggregate it into metrics such as max, min, mean, etc.\n",
    "I am also sorting the data I get by count since we are more interested in \"popular\" games."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"4\" halign=\"left\">time</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "      <th>mean</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gamename</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>龙魂时刻</th>\n",
       "      <td>1</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "      <td>14.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gryphon knight epic</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>growing pains</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>shoppy mart: steam edition</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ground pounders</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>payday 2</th>\n",
       "      <td>102</td>\n",
       "      <td>0</td>\n",
       "      <td>84023</td>\n",
       "      <td>5115.813725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>team fortress 2</th>\n",
       "      <td>105</td>\n",
       "      <td>7</td>\n",
       "      <td>304090</td>\n",
       "      <td>25291.180952</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unturned</th>\n",
       "      <td>107</td>\n",
       "      <td>0</td>\n",
       "      <td>16974</td>\n",
       "      <td>1339.757009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>garry's mod</th>\n",
       "      <td>121</td>\n",
       "      <td>0</td>\n",
       "      <td>311103</td>\n",
       "      <td>20890.314050</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>counter-strike: global offensive</th>\n",
       "      <td>129</td>\n",
       "      <td>0</td>\n",
       "      <td>506638</td>\n",
       "      <td>46356.209302</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>9235 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  time                          \n",
       "                                 count min     max          mean\n",
       "gamename                                                        \n",
       "龙魂时刻                                 1  14      14     14.000000\n",
       "gryphon knight epic                  1   0       0      0.000000\n",
       "growing pains                        1   0       0      0.000000\n",
       "shoppy mart: steam edition           1   0       0      0.000000\n",
       "ground pounders                      1   0       0      0.000000\n",
       "...                                ...  ..     ...           ...\n",
       "payday 2                           102   0   84023   5115.813725\n",
       "team fortress 2                    105   7  304090  25291.180952\n",
       "unturned                           107   0   16974   1339.757009\n",
       "garry's mod                        121   0  311103  20890.314050\n",
       "counter-strike: global offensive   129   0  506638  46356.209302\n",
       "\n",
       "[9235 rows x 4 columns]"
      ]
     },
     "execution_count": 160,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats_df = (games_df.groupby(\"gamename\")\n",
    "                    .agg({'time': ['count', \"min\", 'max', 'mean']})\n",
    "                    .sort_values(by=('time', 'count')))\n",
    "stats_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To prevent one off essoteric games that I don't have a lot of data for, throwing off my metrics, I am disregarding any games that I have less than 10 values for. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"4\" halign=\"left\">time</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "      <th>mean</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gamename</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>serious sam hd: the second encounter</th>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>329</td>\n",
       "      <td>57.909091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>grim fandango remastered</th>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>248</td>\n",
       "      <td>35.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>evga precision x1</th>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>21766</td>\n",
       "      <td>2498.181818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>f.e.a.r. 2: project origin</th>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>292</td>\n",
       "      <td>43.272727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>transistor</th>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>972</td>\n",
       "      <td>298.727273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>payday 2</th>\n",
       "      <td>102</td>\n",
       "      <td>0</td>\n",
       "      <td>84023</td>\n",
       "      <td>5115.813725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>team fortress 2</th>\n",
       "      <td>105</td>\n",
       "      <td>7</td>\n",
       "      <td>304090</td>\n",
       "      <td>25291.180952</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unturned</th>\n",
       "      <td>107</td>\n",
       "      <td>0</td>\n",
       "      <td>16974</td>\n",
       "      <td>1339.757009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>garry's mod</th>\n",
       "      <td>121</td>\n",
       "      <td>0</td>\n",
       "      <td>311103</td>\n",
       "      <td>20890.314050</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>counter-strike: global offensive</th>\n",
       "      <td>129</td>\n",
       "      <td>0</td>\n",
       "      <td>506638</td>\n",
       "      <td>46356.209302</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>701 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      time                          \n",
       "                                     count min     max          mean\n",
       "gamename                                                            \n",
       "serious sam hd: the second encounter    11   0     329     57.909091\n",
       "grim fandango remastered                11   0     248     35.000000\n",
       "evga precision x1                       11   0   21766   2498.181818\n",
       "f.e.a.r. 2: project origin              11   0     292     43.272727\n",
       "transistor                              11   0     972    298.727273\n",
       "...                                    ...  ..     ...           ...\n",
       "payday 2                               102   0   84023   5115.813725\n",
       "team fortress 2                        105   7  304090  25291.180952\n",
       "unturned                               107   0   16974   1339.757009\n",
       "garry's mod                            121   0  311103  20890.314050\n",
       "counter-strike: global offensive       129   0  506638  46356.209302\n",
       "\n",
       "[701 rows x 4 columns]"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats_df = stats_df[stats_df[('time', 'count')] > 10]\n",
    "stats_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that games on average have a play time of 5-hours in a box-plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 9488 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "ax.set_title('Game Play Distribution')\n",
    "ax.boxplot(stats_df[('time', 'mean')]/60, vert=False,manage_ticks=False, notch=True)\n",
    "plt.xlabel(\"Mean Game Play in Hours\")\n",
    "ax.set_xlim([0, 40])\n",
    "ax.set_yticks([])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I had a hunch that more popular games got played more, however, I feel like this data-set is still too small the verify this hunch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fd600786550>"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 13772 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "stats_df.plot.scatter(x=('time', 'count'), y=('time', 'mean'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "We can create a new filtered dataframe that only contains the result of a single game. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gameid</th>\n",
       "      <th>time</th>\n",
       "      <th>gamename</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>13196</th>\n",
       "      <td>730</td>\n",
       "      <td>742</td>\n",
       "      <td>counter-strike: global offensive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13197</th>\n",
       "      <td>730</td>\n",
       "      <td>16019</td>\n",
       "      <td>counter-strike: global offensive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13198</th>\n",
       "      <td>730</td>\n",
       "      <td>1781</td>\n",
       "      <td>counter-strike: global offensive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13199</th>\n",
       "      <td>730</td>\n",
       "      <td>0</td>\n",
       "      <td>counter-strike: global offensive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13200</th>\n",
       "      <td>730</td>\n",
       "      <td>0</td>\n",
       "      <td>counter-strike: global offensive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13320</th>\n",
       "      <td>730</td>\n",
       "      <td>3867</td>\n",
       "      <td>counter-strike: global offensive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13321</th>\n",
       "      <td>730</td>\n",
       "      <td>174176</td>\n",
       "      <td>counter-strike: global offensive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13322</th>\n",
       "      <td>730</td>\n",
       "      <td>186988</td>\n",
       "      <td>counter-strike: global offensive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13323</th>\n",
       "      <td>730</td>\n",
       "      <td>103341</td>\n",
       "      <td>counter-strike: global offensive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13324</th>\n",
       "      <td>730</td>\n",
       "      <td>10483</td>\n",
       "      <td>counter-strike: global offensive</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>129 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       gameid    time                          gamename\n",
       "13196     730     742  counter-strike: global offensive\n",
       "13197     730   16019  counter-strike: global offensive\n",
       "13198     730    1781  counter-strike: global offensive\n",
       "13199     730       0  counter-strike: global offensive\n",
       "13200     730       0  counter-strike: global offensive\n",
       "...       ...     ...                               ...\n",
       "13320     730    3867  counter-strike: global offensive\n",
       "13321     730  174176  counter-strike: global offensive\n",
       "13322     730  186988  counter-strike: global offensive\n",
       "13323     730  103341  counter-strike: global offensive\n",
       "13324     730   10483  counter-strike: global offensive\n",
       "\n",
       "[129 rows x 3 columns]"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cc_df = games_df[games_df['gamename'] == \"counter-strike: global offensive\"]\n",
    "cc_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is supprizing how many hours certain people play in Counter-Strike. The highest number in the dataset was 8,444 hours or 352 days! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 9664 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "ax.set_title('Game Play Distribution for Counter-Strike')\n",
    "ax.boxplot(cc_df['time']/60, vert=False,manage_ticks=False, notch=True)\n",
    "plt.xlabel(\"Game Play in Hours\")\n",
    "ax.set_yticks([])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Viewing the distribution for a different game like Unturned, yields a vastly different distribution than Counter-Strike. I believe the key difference is that Counter-Strike is played competitively where Unturned is a more leasurely game. It is likely that competative gamers skew the distribution of Counter-Strike to be very high. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gameid</th>\n",
       "      <th>time</th>\n",
       "      <th>gamename</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>167</th>\n",
       "      <td>304930</td>\n",
       "      <td>140</td>\n",
       "      <td>unturned</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>168</th>\n",
       "      <td>304930</td>\n",
       "      <td>723</td>\n",
       "      <td>unturned</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>169</th>\n",
       "      <td>304930</td>\n",
       "      <td>1002</td>\n",
       "      <td>unturned</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>170</th>\n",
       "      <td>304930</td>\n",
       "      <td>1002</td>\n",
       "      <td>unturned</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>171</th>\n",
       "      <td>304930</td>\n",
       "      <td>0</td>\n",
       "      <td>unturned</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>269</th>\n",
       "      <td>304930</td>\n",
       "      <td>97</td>\n",
       "      <td>unturned</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>270</th>\n",
       "      <td>304930</td>\n",
       "      <td>768</td>\n",
       "      <td>unturned</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>271</th>\n",
       "      <td>304930</td>\n",
       "      <td>1570</td>\n",
       "      <td>unturned</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>272</th>\n",
       "      <td>304930</td>\n",
       "      <td>23</td>\n",
       "      <td>unturned</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>273</th>\n",
       "      <td>304930</td>\n",
       "      <td>115</td>\n",
       "      <td>unturned</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>107 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     gameid  time  gamename\n",
       "167  304930   140  unturned\n",
       "168  304930   723  unturned\n",
       "169  304930  1002  unturned\n",
       "170  304930  1002  unturned\n",
       "171  304930     0  unturned\n",
       "..      ...   ...       ...\n",
       "269  304930    97  unturned\n",
       "270  304930   768  unturned\n",
       "271  304930  1570  unturned\n",
       "272  304930    23  unturned\n",
       "273  304930   115  unturned\n",
       "\n",
       "[107 rows x 3 columns]"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "u_df = games_df[games_df['gamename'] == \"unturned\"]\n",
    "u_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 8492 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "ax.set_title('Game Play Distribution for Unturned')\n",
    "ax.boxplot(u_df['time']/60, vert=False,manage_ticks=False, notch=True)\n",
    "plt.xlabel(\"Game Play in Hours\")\n",
    "ax.set_yticks([])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, I made a dataframe just containing the raw data points of games that had a aggregate count of over 80. For the sample size of the crawl that I did, having a count of 80, would make the game \"popular\". Since we only have 485 players indexed, having over 80 entries means that over 17% of people scrapped had the game. Glancing at the results, it is easy to verify that the games returned were very popular. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-142-49f39d385a2a>:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df1['time'] = df1['time']/60\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gameid</th>\n",
       "      <th>time</th>\n",
       "      <th>gamename</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>167</th>\n",
       "      <td>304930</td>\n",
       "      <td>2.333333</td>\n",
       "      <td>unturned</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>168</th>\n",
       "      <td>304930</td>\n",
       "      <td>12.050000</td>\n",
       "      <td>unturned</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>169</th>\n",
       "      <td>304930</td>\n",
       "      <td>16.700000</td>\n",
       "      <td>unturned</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>170</th>\n",
       "      <td>304930</td>\n",
       "      <td>16.700000</td>\n",
       "      <td>unturned</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>171</th>\n",
       "      <td>304930</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>unturned</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22682</th>\n",
       "      <td>578080</td>\n",
       "      <td>51.883333</td>\n",
       "      <td>playerunknown's battlegrounds</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22683</th>\n",
       "      <td>578080</td>\n",
       "      <td>47.616667</td>\n",
       "      <td>playerunknown's battlegrounds</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22684</th>\n",
       "      <td>578080</td>\n",
       "      <td>30.650000</td>\n",
       "      <td>playerunknown's battlegrounds</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22685</th>\n",
       "      <td>578080</td>\n",
       "      <td>170.083333</td>\n",
       "      <td>playerunknown's battlegrounds</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22686</th>\n",
       "      <td>578080</td>\n",
       "      <td>399.950000</td>\n",
       "      <td>playerunknown's battlegrounds</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1099 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       gameid        time                       gamename\n",
       "167    304930    2.333333                       unturned\n",
       "168    304930   12.050000                       unturned\n",
       "169    304930   16.700000                       unturned\n",
       "170    304930   16.700000                       unturned\n",
       "171    304930    0.000000                       unturned\n",
       "...       ...         ...                            ...\n",
       "22682  578080   51.883333  playerunknown's battlegrounds\n",
       "22683  578080   47.616667  playerunknown's battlegrounds\n",
       "22684  578080   30.650000  playerunknown's battlegrounds\n",
       "22685  578080  170.083333  playerunknown's battlegrounds\n",
       "22686  578080  399.950000  playerunknown's battlegrounds\n",
       "\n",
       "[1099 rows x 3 columns]"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1 = games_df[games_df['gamename'].map(games_df['gamename'].value_counts()) > 80]\n",
    "df1['time'] = df1['time']/60\n",
    "df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 30956 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "ax.set_title('Steps Distribution for July\\n')\n",
    "df1.boxplot(column=[\"time\"], by='gamename',ax=ax, notch=True, vert=False)\n",
    "plt.xlabel(\"Hours Played\")\n",
    "plt.ylabel(\"Games\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 31936 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = df1.boxplot(column=[\"time\"], by='gamename', notch=True, vert=False)\n",
    "fig = ax.get_figure()\n",
    "fig.suptitle('')\n",
    "ax.set_title('Play-time Distribution')\n",
    "plt.xlabel(\"Hours Played\")\n",
    "ax.set_xlim([0, 2000])\n",
    "plt.ylabel(\"Game\")\n",
    "plt.savefig(\"playTimes.png\", dpi=300, bbox_inches = \"tight\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Overall this is really interesting to see how the distributions for different games vary. In the future I am going to re-run some of these analytics with even more data and possibly put them on my website as a interactive graph."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ml",
   "language": "python",
   "name": "ml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
