{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do a deep dive and start visualizing my life using Fitbit and Matplotlib. \n",
    "\n",
    "# What is Fitbit\n",
    "\n",
    "[Fitbit](https://www.fitbit.com) is a fitness watch that tracks your sleep, heart rate, and activity.\n",
    "Fitbit is able to track your steps, however, it is also able to detect multiple types of activity\n",
    "like running, walking, \"sport\" and biking."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What is Matplotlib\n",
    "\n",
    "[Matplotlib](https://matplotlib.org/) is a python visualization library that enables you to create bar graphs, line graphs, distributions and many more things.\n",
    "Being able to visualize your results is essential to any person working with data at any scale.\n",
    "Although I like [GGplot](https://ggplot2.tidyverse.org/) in R more than Matplotlib, Matplotlib is still my go to graphing library for Python. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Your Fitbit Data\n",
    "\n",
    "There are two main ways that you can get your Fitbit data:\n",
    "\n",
    "- Fitbit API\n",
    "- Data Archival Export\n",
    "\n",
    "\n",
    "Since connecting to the API and setting up all the web hooks can be a pain, I'm just going to use the data export option because this is only for one person.\n",
    "You can export your data here: [https://www.fitbit.com/settings/data/export](https://www.fitbit.com/settings/data/export).\n",
    "\n",
    "![Data export on fitbit's website](dataExport.png)\n",
    "\n",
    "The Fitbit data archive was very organized and kept meticulous records of everything. \n",
    "All of the data was organized in separate JSON files labeled by date.\n",
    "Fitbit keeps around 1MB of data on you per day; most of this data is from the heart rate sensors.\n",
    "Although 1MB of data may sound like a ton of data, it is probably a lot less if you store it in formats other than JSON. \n",
    "When I downloaded the compressed file it was 20MB, but when I extracted it, it was 380MB!\n",
    "I've only been using Fitbit for 11 months at this point. \n",
    "\n",
    "![compressed data](compression.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sleep\n",
    "\n",
    "Sleep is something fun to visualize.\n",
    "No matter how much of it you get you still feel tired as a college student.\n",
    "In the \"sleep_score\" folder of the exported data you will find a single CSV file with your resting heart rate and Fitbit's computed sleep scores.\n",
    "Interesting enough, this is the only file that comes in the CSV format, everything else is JSON file. \n",
    "\n",
    "We can read in all the data using a single liner with the [Pandas](https://pandas.pydata.org/) python library.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "sleep_score_df = pd.read_csv('data/sleep/sleep_score.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     sleep_log_entry_id             timestamp  overall_score  \\\n",
      "0           26093459526  2020-02-27T06:04:30Z             80   \n",
      "1           26081303207  2020-02-26T06:13:30Z             83   \n",
      "2           26062481322  2020-02-25T06:00:30Z             82   \n",
      "3           26045941555  2020-02-24T05:49:30Z             79   \n",
      "4           26034268762  2020-02-23T08:35:30Z             75   \n",
      "..                  ...                   ...            ...   \n",
      "176         23696231032  2019-09-02T07:38:30Z             79   \n",
      "177         23684345925  2019-09-01T07:15:30Z             84   \n",
      "178         23673204871  2019-08-31T07:11:00Z             74   \n",
      "179         23661278483  2019-08-30T06:34:00Z             73   \n",
      "180         23646265400  2019-08-29T05:55:00Z             80   \n",
      "\n",
      "     composition_score  revitalization_score  duration_score  \\\n",
      "0                   20                    19              41   \n",
      "1                   22                    21              40   \n",
      "2                   22                    21              39   \n",
      "3                   17                    20              42   \n",
      "4                   20                    16              39   \n",
      "..                 ...                   ...             ...   \n",
      "176                 20                    20              39   \n",
      "177                 22                    21              41   \n",
      "178                 18                    21              35   \n",
      "179                 17                    19              37   \n",
      "180                 21                    21              38   \n",
      "\n",
      "     deep_sleep_in_minutes  resting_heart_rate  restlessness  \n",
      "0                       65                  60      0.117330  \n",
      "1                       85                  60      0.113188  \n",
      "2                       95                  60      0.120635  \n",
      "3                       52                  61      0.111224  \n",
      "4                       43                  59      0.154774  \n",
      "..                     ...                 ...           ...  \n",
      "176                     88                  56      0.170923  \n",
      "177                     95                  56      0.133268  \n",
      "178                     73                  56      0.102703  \n",
      "179                     50                  55      0.121086  \n",
      "180                     61                  57      0.112961  \n",
      "\n",
      "[181 rows x 9 columns]\n"
     ]
    }
   ],
   "source": [
    "print(sleep_score_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the Pandas library you can generate Matplotlib graphs.\n",
    "Although you can directly use Matplotlib, the wrapper functions using Pandas makes it easier to use.\n",
    "\n",
    "## Sleep Score Histogram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[<matplotlib.axes._subplots.AxesSubplot object at 0x7fc2c0a270d0>]],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 7968 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sleep_score_df.hist(column='overall_score')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Heart Rate\n",
    "\n",
    "Fitbit keeps their calculated heart rates in the sleep scores file rather than heart.\n",
    "Knowing your resting heart rate is useful because it is a good indicator of your overall health.\n",
    "\n",
    "![](restingHeartRate.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[<matplotlib.axes._subplots.AxesSubplot object at 0x7fc2917a6090>]],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 6824 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sleep_score_df.hist(column='resting_heart_rate')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resting Heart Rate Time Graph\n",
    "\n",
    "Using the pandas wrapper we can quickly create a heart rate graph over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fc28f609b50>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 32920 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sleep_score_df.plot(kind='line', y='resting_heart_rate', x ='timestamp', legend=False, title=\"Resting Heart Rate(BPM)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, as we notice with the graph above, the time axis is wack.\n",
    "In the pandas data frame everything was stored as a string timestamp.\n",
    "We can convert this into a datetime object by telling pandas to parse the date as it reads it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fc28f533510>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 28888 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sleep_score_df = pd.read_csv('data/sleep/sleep_score.csv', parse_dates=[1])\n",
    "sleep_score_df.plot(kind='line', y='resting_heart_rate', x ='timestamp', legend=False, title=\"Resting Heart Rate(BPM)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To fully manipulate the graphs, we need to use some matplotlib code to do things like setting the axis labels or make multiple plots right next to each other.\n",
    "We can create grab the current axis being used by matplotlib by using plt.gca()."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 42008 characters]\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "sleep_score_df.plot(kind='line', y='resting_heart_rate', x ='timestamp', legend=False, title=\"Resting Heart Rate Graph\", ax=ax, figsize=(10, 5))\n",
    "plt.xlabel(\"Date\")\n",
    "plt.ylabel(\"Resting Heart Rate (BPM)\")\n",
    "plt.show()\n",
    "#plt.savefig('restingHeartRate.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same thing can be done with sleep scores.\n",
    "It is interesting to note that  the sleep scores rarely vary anything between 75 and 85."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 36612 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "sleep_score_df.plot(kind='line', y='overall_score', x ='timestamp', legend=False, title=\"Sleep Score Time Series Graph\", ax=ax)\n",
    "plt.xlabel(\"Date\")\n",
    "plt.ylabel(\"Fitbit's Sleep Score\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using Pandas we can generate a new column with a specific date attribute like year, day, month, or weekday.\n",
    "If we add a new column for weekday, we can then group by weekday and collapse them all into a single column by summing or averaging the value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     sleep_log_entry_id                 timestamp  overall_score  \\\n",
      "0           26093459526 2020-02-27 06:04:30+00:00             80   \n",
      "1           26081303207 2020-02-26 06:13:30+00:00             83   \n",
      "2           26062481322 2020-02-25 06:00:30+00:00             82   \n",
      "3           26045941555 2020-02-24 05:49:30+00:00             79   \n",
      "4           26034268762 2020-02-23 08:35:30+00:00             75   \n",
      "..                  ...                       ...            ...   \n",
      "176         23696231032 2019-09-02 07:38:30+00:00             79   \n",
      "177         23684345925 2019-09-01 07:15:30+00:00             84   \n",
      "178         23673204871 2019-08-31 07:11:00+00:00             74   \n",
      "179         23661278483 2019-08-30 06:34:00+00:00             73   \n",
      "180         23646265400 2019-08-29 05:55:00+00:00             80   \n",
      "\n",
      "     composition_score  revitalization_score  duration_score  \\\n",
      "0                   20                    19              41   \n",
      "1                   22                    21              40   \n",
      "2                   22                    21              39   \n",
      "3                   17                    20              42   \n",
      "4                   20                    16              39   \n",
      "..                 ...                   ...             ...   \n",
      "176                 20                    20              39   \n",
      "177                 22                    21              41   \n",
      "178                 18                    21              35   \n",
      "179                 17                    19              37   \n",
      "180                 21                    21              38   \n",
      "\n",
      "     deep_sleep_in_minutes  resting_heart_rate  restlessness  weekday  \n",
      "0                       65                  60      0.117330        3  \n",
      "1                       85                  60      0.113188        2  \n",
      "2                       95                  60      0.120635        1  \n",
      "3                       52                  61      0.111224        0  \n",
      "4                       43                  59      0.154774        6  \n",
      "..                     ...                 ...           ...      ...  \n",
      "176                     88                  56      0.170923        0  \n",
      "177                     95                  56      0.133268        6  \n",
      "178                     73                  56      0.102703        5  \n",
      "179                     50                  55      0.121086        4  \n",
      "180                     61                  57      0.112961        3  \n",
      "\n",
      "[181 rows x 10 columns]\n"
     ]
    }
   ],
   "source": [
    "temp = pd.DatetimeIndex(sleep_score_df['timestamp'])\n",
    "sleep_score_df['weekday'] = temp.weekday\n",
    "\n",
    "print(sleep_score_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         sleep_log_entry_id  overall_score  composition_score  \\\n",
      "weekday                                                         \n",
      "0              2.483733e+10      79.576923          20.269231   \n",
      "1              2.485200e+10      77.423077          20.423077   \n",
      "2              2.490383e+10      80.880000          21.120000   \n",
      "3              2.483418e+10      76.814815          20.370370   \n",
      "4              2.480085e+10      79.769231          20.961538   \n",
      "5              2.477002e+10      78.840000          20.520000   \n",
      "6              2.482581e+10      77.230769          20.269231   \n",
      "\n",
      "         revitalization_score  duration_score  deep_sleep_in_minutes  \\\n",
      "weekday                                                                \n",
      "0                   19.153846       40.153846              88.000000   \n",
      "1                   19.000000       38.000000              83.846154   \n",
      "2                   19.400000       40.360000              93.760000   \n",
      "3                   19.037037       37.407407              82.592593   \n",
      "4                   19.346154       39.461538              94.461538   \n",
      "5                   19.080000       39.240000              93.720000   \n",
      "6                   18.269231       38.692308              89.423077   \n",
      "\n",
      "         resting_heart_rate  restlessness  \n",
      "weekday                                    \n",
      "0                 58.576923      0.139440  \n",
      "1                 58.538462      0.142984  \n",
      "2                 58.560000      0.138661  \n",
      "3                 58.333333      0.135819  \n",
      "4                 58.269231      0.129791  \n",
      "5                 58.080000      0.138315  \n",
      "6                 58.153846      0.147171  \n"
     ]
    }
   ],
   "source": [
    "print(sleep_score_df.groupby('weekday').mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sleep Score Based on Day"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 27056 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "sleep_score_df.groupby('weekday').mean().plot(kind='line', y='overall_score', ax = ax)\n",
    "plt.ylabel(\"Sleep Score\")\n",
    "plt.title(\"Sleep Scores on Varying Days of Week\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sleep Score Based on Days of Week"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 21916 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "sleep_score_df.groupby('weekday').mean().plot(kind='line', y='resting_heart_rate', ax = ax)\n",
    "plt.ylabel(\"Resting heart rate (BPM)\")\n",
    "plt.title(\"Resting Heart Rate Varying Days of Week\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Calories\n",
    "\n",
    "Fitbit keeps all of their calorie data in JSON files representing sequence data at 1 minute increments.\n",
    "To extrapolate calorie data we need to group by day and then sum the days to get the total calories burned per day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "calories_df = pd.read_json(\"data/calories/calories-2019-07-01.json\",  convert_dates=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                 dateTime  value\n",
      "0     2019-07-01 00:00:00   1.07\n",
      "1     2019-07-01 00:01:00   1.07\n",
      "2     2019-07-01 00:02:00   1.07\n",
      "3     2019-07-01 00:03:00   1.07\n",
      "4     2019-07-01 00:04:00   1.07\n",
      "...                   ...    ...\n",
      "43195 2019-07-30 23:55:00   1.07\n",
      "43196 2019-07-30 23:56:00   1.07\n",
      "43197 2019-07-30 23:57:00   1.07\n",
      "43198 2019-07-30 23:58:00   1.07\n",
      "43199 2019-07-30 23:59:00   1.07\n",
      "\n",
      "[43200 rows x 2 columns]\n"
     ]
    }
   ],
   "source": [
    "print(calories_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                           dateTime  value date_minus_time\n",
      "date_minus_time                                           \n",
      "2019-07-01      2019-07-01 00:00:00   1.07      2019-07-01\n",
      "2019-07-01      2019-07-01 00:01:00   1.07      2019-07-01\n",
      "2019-07-01      2019-07-01 00:02:00   1.07      2019-07-01\n",
      "2019-07-01      2019-07-01 00:03:00   1.07      2019-07-01\n",
      "2019-07-01      2019-07-01 00:04:00   1.07      2019-07-01\n",
      "...                             ...    ...             ...\n",
      "2019-07-30      2019-07-30 23:55:00   1.07      2019-07-30\n",
      "2019-07-30      2019-07-30 23:56:00   1.07      2019-07-30\n",
      "2019-07-30      2019-07-30 23:57:00   1.07      2019-07-30\n",
      "2019-07-30      2019-07-30 23:58:00   1.07      2019-07-30\n",
      "2019-07-30      2019-07-30 23:59:00   1.07      2019-07-30\n",
      "\n",
      "[43200 rows x 3 columns]\n"
     ]
    }
   ],
   "source": [
    "import datetime\n",
    "calories_df['date_minus_time'] = calories_df[\"dateTime\"].apply( lambda calories_df : \n",
    "    datetime.datetime(year=calories_df.year, month=calories_df.month, day=calories_df.day))\t\n",
    "\n",
    "calories_df.set_index(calories_df[\"date_minus_time\"],inplace=True)\n",
    "\n",
    "print(calories_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                   value\n",
      "date_minus_time         \n",
      "2019-07-01       3422.68\n",
      "2019-07-02       2705.85\n",
      "2019-07-03       2871.73\n",
      "2019-07-04       4089.93\n",
      "2019-07-05       3917.91\n",
      "2019-07-06       2762.55\n",
      "2019-07-07       2929.58\n",
      "2019-07-08       2698.99\n",
      "2019-07-09       2833.27\n",
      "2019-07-10       2529.21\n",
      "2019-07-11       2634.25\n",
      "2019-07-12       2953.91\n",
      "2019-07-13       4247.45\n",
      "2019-07-14       2998.35\n",
      "2019-07-15       2846.18\n",
      "2019-07-16       3084.39\n",
      "2019-07-17       2331.06\n",
      "2019-07-18       2849.20\n",
      "2019-07-19       2071.63\n",
      "2019-07-20       2746.25\n",
      "2019-07-21       2562.11\n",
      "2019-07-22       1892.99\n",
      "2019-07-23       2372.89\n",
      "2019-07-24       2320.42\n",
      "2019-07-25       2140.87\n",
      "2019-07-26       2430.38\n",
      "2019-07-27       3769.04\n",
      "2019-07-28       2036.24\n",
      "2019-07-29       2814.87\n",
      "2019-07-30       2077.82\n"
     ]
    }
   ],
   "source": [
    "calories_per_day = calories_df.resample('D').sum()\n",
    "print(calories_per_day)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 7212 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "calories_per_day.plot(kind='hist', title=\"Calorie Distribution\", legend=False, ax=ax)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 27128 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "calories_per_day.plot(kind='line', y='value', legend=False, title=\"Calories Per Day\", ax=ax)\n",
    "plt.xlabel(\"Date\")\n",
    "plt.ylabel(\"Calories\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calories Per Day Box Plot\n",
    "\n",
    "Using this data we can turn this into a boxplot to make it easier to visualize the distribution of calories burned during the month of July."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 10336 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "ax.set_title('Calorie Distribution for July')\n",
    "ax.boxplot(calories_per_day['value'], vert=False,manage_ticks=False, notch=True)\n",
    "plt.xlabel(\"Calories Burned\")\n",
    "ax.set_yticks([])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Steps\n",
    "\n",
    "Fitbit is known for taking the amount of steps someone takes per day.\n",
    "Similar to calories burned, steps taken is stored in time series data at 1 minute increments.\n",
    "Since we are interested at the day level data, we need to first remove the time component of the dataframe so that we can group all the data by date.\n",
    "Once we have everything grouped by date, we can sum and produce steps per day. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                           dateTime  value date_minus_time\n",
      "date_minus_time                                           \n",
      "2019-07-01      2019-07-01 04:00:00      0      2019-07-01\n",
      "2019-07-01      2019-07-01 04:01:00      0      2019-07-01\n",
      "2019-07-01      2019-07-01 04:02:00      0      2019-07-01\n",
      "2019-07-01      2019-07-01 04:03:00      0      2019-07-01\n",
      "2019-07-01      2019-07-01 04:04:00      0      2019-07-01\n",
      "...                             ...    ...             ...\n",
      "2019-07-31      2019-07-31 03:55:00      0      2019-07-31\n",
      "2019-07-31      2019-07-31 03:56:00      0      2019-07-31\n",
      "2019-07-31      2019-07-31 03:57:00      0      2019-07-31\n",
      "2019-07-31      2019-07-31 03:58:00      0      2019-07-31\n",
      "2019-07-31      2019-07-31 03:59:00      0      2019-07-31\n",
      "\n",
      "[41116 rows x 3 columns]\n"
     ]
    }
   ],
   "source": [
    "steps_df = pd.read_json(\"data/steps-2019-07-01.json\",  convert_dates=True)\n",
    "\n",
    "steps_df['date_minus_time'] = steps_df[\"dateTime\"].apply( lambda steps_df : \n",
    "    datetime.datetime(year=steps_df.year, month=steps_df.month, day=steps_df.day))\t\n",
    "\n",
    "steps_df.set_index(steps_df[\"date_minus_time\"],inplace=True)\n",
    "print(steps_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                 value\n",
      "date_minus_time       \n",
      "2019-07-01       11285\n",
      "2019-07-02        4957\n",
      "2019-07-03       13119\n",
      "2019-07-04       16034\n",
      "2019-07-05       11634\n",
      "2019-07-06        6860\n",
      "2019-07-07        3758\n",
      "2019-07-08        9130\n",
      "2019-07-09       10960\n",
      "2019-07-10        7012\n",
      "2019-07-11        5420\n",
      "2019-07-12        4051\n",
      "2019-07-13       15980\n",
      "2019-07-14       23109\n",
      "2019-07-15       11247\n",
      "2019-07-16       10170\n",
      "2019-07-17        4905\n",
      "2019-07-18       10769\n",
      "2019-07-19        4504\n",
      "2019-07-20        5032\n",
      "2019-07-21        8953\n",
      "2019-07-22        2200\n",
      "2019-07-23        9392\n",
      "2019-07-24        5666\n",
      "2019-07-25        5016\n",
      "2019-07-26        5879\n",
      "2019-07-27       19492\n",
      "2019-07-28        4987\n",
      "2019-07-29        9943\n",
      "2019-07-30        3897\n",
      "2019-07-31         166\n"
     ]
    }
   ],
   "source": [
    "steps_per_day = steps_df.resample('D').sum()\n",
    "print(steps_per_day)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Steps Per Day Histogram\n",
    "\n",
    "After the data is in the form that we want, graphing the data is straight forward.\n",
    "Two added things I like to do for normal box plots is to set the displays to horizontal add the notches. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 10460 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "ax.set_title('Steps Distribution for July')\n",
    "ax.boxplot(steps_per_day['value'], vert=False,manage_ticks=False, notch=True)\n",
    "plt.xlabel(\"Steps Per Day\")\n",
    "ax.set_yticks([])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wrapping that all into a single function we get something like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def readFileIntoDataFrame(fName):\n",
    "    steps_df = pd.read_json(fName,  convert_dates=True)\n",
    "\n",
    "    steps_df['date_minus_time'] = steps_df[\"dateTime\"].apply( lambda steps_df : \n",
    "        datetime.datetime(year=steps_df.year, month=steps_df.month, day=steps_df.day))\t\n",
    "\n",
    "    steps_df.set_index(steps_df[\"date_minus_time\"],inplace=True)\n",
    "    return steps_df.resample('D').sum()\n",
    "\n",
    "def graphBoxAndWhiskers(data, title, xlab):\n",
    "    ax = plt.gca()\n",
    "    ax.set_title(title)\n",
    "    ax.boxplot(data['value'], vert=False, manage_ticks=False, notch=True)\n",
    "    plt.xlabel(xlab)\n",
    "    ax.set_yticks([])\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 9748 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "graphBoxAndWhiskers(readFileIntoDataFrame(\"data/steps-2020-01-27.json\"), \"Steps In January\", \"Steps Per Day\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That is cool, but, what if we could view the distribution for each month in the same graph?\n",
    "Based on the two previous graphs, my step distribution during July looked distinctly different from my step distribution in January. \n",
    "The first difficultly would be to read in all the files since Fitbit creates a new file for every month.\n",
    "The next thing would be to group them by month and then graph it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['steps-2019-04-02.json', 'steps-2019-08-30.json', 'steps-2020-02-26.json', 'steps-2019-10-29.json', 'steps-2019-07-01.json', 'steps-2020-01-27.json', 'steps-2019-07-31.json', 'steps-2019-06-01.json', 'steps-2019-09-29.json', '.ipynb_checkpoints', 'steps-2019-12-28.json', 'steps-2019-05-02.json', 'calories', 'steps-2019-11-28.json', 'sleep']\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "files = os.listdir(\"data\")\n",
    "print(files)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfs = []\n",
    "for file in files: # this can take 15 seconds\n",
    "    if \"steps\" in file: # finds the steps files\n",
    "        dfs.append(readFileIntoDataFrame(\"data/\" + file))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 9156 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "stepsPerDay = pd.concat(dfs)\n",
    "graphBoxAndWhiskers(stepsPerDay, \"Steps Per Day Last 11 Months\", \"Steps per Day\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'numpy.ndarray'>\n",
      "DatetimeIndex(['2019-04-03', '2019-04-04', '2019-04-05', '2019-04-06',\n",
      "               '2019-04-07', '2019-04-08', '2019-04-09', '2019-04-10',\n",
      "               '2019-04-11', '2019-04-12',\n",
      "               ...\n",
      "               '2019-12-19', '2019-12-20', '2019-12-21', '2019-12-22',\n",
      "               '2019-12-23', '2019-12-24', '2019-12-25', '2019-12-26',\n",
      "               '2019-12-27', '2019-12-28'],\n",
      "              dtype='datetime64[ns]', name='date_minus_time', length=342, freq=None)\n",
      "                 value  month  week_day\n",
      "date_minus_time                        \n",
      "2019-04-03         510      4         2\n",
      "2019-04-04       11453      4         3\n",
      "2019-04-05       12684      4         4\n",
      "2019-04-06       12910      4         5\n",
      "2019-04-07        3368      4         6\n",
      "...                ...    ...       ...\n",
      "2019-12-24        5779     12         1\n",
      "2019-12-25        4264     12         2\n",
      "2019-12-26        4843     12         3\n",
      "2019-12-27        9609     12         4\n",
      "2019-12-28        2218     12         5\n",
      "\n",
      "[342 rows x 3 columns]\n"
     ]
    }
   ],
   "source": [
    "print(type(stepsPerDay['value'].to_numpy()))\n",
    "print(stepsPerDay['value'].keys())\n",
    "\n",
    "stepsPerDay['month'] = pd.DatetimeIndex(stepsPerDay['value'].keys()).month \n",
    "stepsPerDay['week_day'] = pd.DatetimeIndex(stepsPerDay['value'].keys()).weekday\n",
    "\n",
    "print(stepsPerDay)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Graphing Steps by Month\n",
    "\n",
    "Now that we have columns for the total amount of steps per day and the months, we can plot all the data on a single plot using the group by operator in the plotting library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 24588 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "ax.set_title('Steps Distribution for July\\n')\n",
    "stepsPerDay.boxplot(column=['value'], by='month',ax=ax, notch=True)\n",
    "plt.xlabel(\"Month\")\n",
    "plt.ylabel(\"Steps Per Day\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 20272 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = plt.gca()\n",
    "ax.set_title('Steps Distribution By Week Day\\n')\n",
    "stepsPerDay.boxplot(column=['value'], by='week_day',ax=ax, notch=True)\n",
    "plt.xlabel(\"Week Day\")\n",
    "plt.ylabel(\"Steps Per Day\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Future Work\n",
    "\n",
    "Moving forward with this I would like to do more visualizations with sleep data and heart rate."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
