Personal blog written from scratch using Node.js, Bootstrap, and MySQL. https://jrtechs.net
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

359 lines
12 KiB

  1. This post looks at how you can aggregate and visualize Fitbit GPS data since there is no built-in functionality to do this on the Fitbit website.
  2. Before you read this post, check out my two other posts on using Fitbit data:
  3. - [A Closer Look at Fitbit Data](https://jrtechs.net/data-science/a-closer-look-at-fitbit-data)
  4. - [Graphing my Life with Matplotlib](https://jrtechs.net/data-science/graphing-my-life-with-matplotlib)
  5. # Getting the Data
  6. There are two options that we can use to fetch data from Fitbit:
  7. - Fitbit's Data Export Tool
  8. - Fitbit's API
  9. ## Exporting from the Website
  10. The easiest way to export data from Fitbit is to use the data export tool on the website.
  11. ![Fitbit Archive Data](media/fitbit/fitbitArchiveData.png)
  12. The Fitbit data archive is very organized and has all information relating to your account.
  13. The data is organized in separate JSON files labeled by date. Fitbit keeps around 1MB of data on you per day; most of this data is from the heart rate sensors.
  14. Although 1MB of data may sound like a ton of data, it is probably a lot less if you store it in a more compact format than JSON.
  15. The problem with the data export is that it can take a while to export and that it doesn't include the GPS data in [TCX](https://en.wikipedia.org/wiki/Training_Center_XML) format.
  16. In the exercise.json files, we can view all activities with logged GPS data, but the data archives themselves don't include the TCX data-- only a link to it on the website.
  17. ```json
  18. },{
  19. "logId" : 384529004,
  20. "activityName" : "Run",
  21. "activityTypeId" : 90009,
  22. "activityLevel" : [{
  23. "minutes" : 0,
  24. "name" : "sedentary"
  25. },{
  26. "minutes" : 0,
  27. "name" : "lightly"
  28. },{
  29. "minutes" : 0,
  30. "name" : "fairly"
  31. },{
  32. "minutes" : 17,
  33. "name" : "very"
  34. }],
  35. "calories" : 184,
  36. "distance" : 1.872894,
  37. "distanceUnit" : "Mile",
  38. "duration" : 1077000,
  39. "activeDuration" : 1077000,
  40. "steps" : 2605,
  41. "source" : {
  42. "type" : "app",
  43. "name" : "Fitbit for Android",
  44. "id" : "228VSR",
  45. "url" : "https://www.fitbit.com/android",
  46. "trackerFeatures" : ["GPS","STEPS","CALORIES","PACE","DISTANCE"]
  47. },
  48. "logType" : "mobile_run",
  49. "manualValuesSpecified" : {
  50. "calories" : false,
  51. "distance" : false,
  52. "steps" : false
  53. },
  54. "tcxLink" : "https://www.fitbit.com/activities/exercise/384529004?export=tcx",
  55. "speed" : 6.26036991643454,
  56. "pace" : 575.0458915453837,
  57. "lastModified" : "03/22/21 21:22:03",
  58. "startTime" : "03/22/21 21:04:02",
  59. "originalStartTime" : "03/22/21 21:04:02",
  60. "originalDuration" : 1077000,
  61. "hasGps" : true,
  62. "shouldFetchDetails" : true,
  63. "hasActiveZoneMinutes" : false
  64. },{
  65. ```
  66. ## API
  67. Since the data export doesn't include the TCX data we desire, we need to use the Fitbit API.
  68. First, you need to register a Fitbit [developer account](https://dev.fitbit.com/apps) and then register an app.
  69. For the redirect URL, put "http://localhost:9000/auth/fitbit/callback".
  70. To authenticate with the application, you will need to OAuth 2 client ID and the client secret.
  71. Check out my first post on Fitbit to learn more about how to set up an [applicatoin with Fitbit](https://jrtechs.net/data-science/a-closer-look-at-fitbit-data), and check out the code for this project on my [Github](https://github.com/jrtechs/HomePage).
  72. # Downloading All GPS Logs
  73. Within the Fitbit API, there is no place to download all of your activities -- since that would return a lot of data.
  74. Instead, Fitbit allows you to download a maximum of 100 events from a time, and you can specify a start and end range for your query.
  75. ```javascript
  76. function fetchActivities(result, startTime)
  77. {
  78. return new Promise((resolve, reject)=>
  79. {
  80. queryAPI(result, 'https://api.fitbit.com/1/user/-/activities/list.json?beforeDate=' + startTime + '&sort=desc&offset=0&limit=100').then((data)=>
  81. {
  82. if(data != false)
  83. {
  84. resolve(data.activities);
  85. }
  86. reject("Error with API, are you authenticated")
  87. });
  88. });
  89. }
  90. ```
  91. I wrote a helper function to fetch all the "mobile_runs" events recursively.
  92. In my case, since I have the Fitbit Altra, so the only way for me to get GPS data is to tether my Fitbit to my phone and log the start/end.
  93. With the Fitbits that have built-in GPS, other events have GPS TCX data associated with them.
  94. ```javascript
  95. function fetchAllRuns(result, startTime)
  96. {
  97. var runs = [];
  98. console.log(startTime);
  99. return new Promise((resolve, reject)=>
  100. {
  101. fetchActivities(result, startTime).then((events)=>
  102. {
  103. if(events.length < 10)
  104. {
  105. resolve(runs);
  106. }
  107. else
  108. {
  109. for(var i = 0; i < events.length; i++)
  110. {
  111. if(events[i].logType === "mobile_run")
  112. {
  113. console.log(events[i]);
  114. runs.push(events[i]);
  115. }
  116. }
  117. var newStart = events[events.length -1].startTime.slice(0, -10);
  118. fetchAllRuns(result, newStart).then((run_rec)=>
  119. {
  120. resolve(runs.concat(run_rec));
  121. }).catch((err)=>
  122. {
  123. reject(err);
  124. });
  125. }
  126. }).catch((error)=>
  127. {
  128. reject(error);
  129. });
  130. });
  131. }
  132. ```
  133. After we have a list of our TCX data, we can simply write it to disk for later use.
  134. ```javascript
  135. function saveTCX(result, tcxID)
  136. {
  137. return new Promise((resolve, reject)=>
  138. {
  139. fetchTCX(result, tcxID).then((tcx)=>
  140. {
  141. utils.saveFile(tcxID + ".tcx", tcx);
  142. resolve();
  143. }).catch((err)=>
  144. {
  145. reject(err);
  146. })
  147. });
  148. }
  149. ```
  150. I have this process invoked via an endpoint call to enable me to use Fitbit's OAuth2 authentication.
  151. ```javascript
  152. app.get('/save-all-tcx', (request, result)=>
  153. {
  154. var tcxID = request.params.id;
  155. var startTime = '2030-06-01T00:00:00';
  156. fetchAllRuns(result, startTime).then((data)=>
  157. {
  158. var promises = [];
  159. for(var i =0; i < data.length; i++)
  160. {
  161. promises.push(saveTCX(result, data[i].logId));
  162. }
  163. Promise.all(promises).then(function(content)
  164. {
  165. result.write("All events saved");
  166. result.end();
  167. }).catch(function(err)
  168. {
  169. console.log(err);
  170. throw err;
  171. });
  172. }).catch((error)=>
  173. {
  174. console.log(error);
  175. result.writeHead(500, {'Content-Type': 'text/json'});
  176. result.end();
  177. });
  178. });
  179. ```
  180. # Visualizing GPS Data as KML
  181. Now that we have all of our TCX data saved, we can parse the data to aggregate it into a single file.
  182. TCX data is simply XML data that is easy to parse with several different packages.
  183. In my case, I am [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) in Python to parse the XML.
  184. ```xml
  185. <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
  186. <TrainingCenterDatabase xmlns="http://www.garmin.com/xmlschemas/TrainingCenterDatabase/v2">
  187. <Activities>
  188. <Activity Sport="Running">
  189. <Id>2019-09-03T16:52:50.000-04:00</Id>
  190. <Lap StartTime="2019-09-03T16:52:50.000-04:00">
  191. <TotalTimeSeconds>1313.0</TotalTimeSeconds>
  192. <DistanceMeters>4618.291573209821</DistanceMeters>
  193. <Calories>265</Calories>
  194. <Intensity>Active</Intensity>
  195. <TriggerMethod>Manual</TriggerMethod>
  196. <Track>
  197. <Trackpoint>
  198. <Time>2019-09-03T16:52:50.000-04:00</Time>
  199. <Position>
  200. <LatitudeDegrees>43.0</LatitudeDegrees>
  201. <LongitudeDegrees>-77.6</LongitudeDegrees>
  202. </Position>
  203. <AltitudeMeters>123.4</AltitudeMeters>
  204. <DistanceMeters>0.0</DistanceMeters>
  205. </Trackpoint>
  206. <Trackpoint>
  207. <Time>2019-09-03T16:52:51.000-04:00</Time>
  208. <Position>
  209. <LatitudeDegrees>43.0</LatitudeDegrees>
  210. <LongitudeDegrees>-77.6</LongitudeDegrees>
  211. </Position>
  212. <AltitudeMeters>123.4</AltitudeMeters>
  213. <DistanceMeters>0.28944606511402504</DistanceMeters>
  214. </Trackpoint>
  215. ```
  216. A quick note on the data that I am showing: it is all out of date, and I no longer live in these neighborhoods.
  217. Additionally, for the TCX example, I truncated the longitude/latitude results, but, from Fitbit, the TSV data has fifteen decimal places.
  218. Using Beautiful Soup, it is straightforward to open a TCX file and get all the longitude/latitude values.
  219. ```Python
  220. from bs4 import BeautifulSoup
  221. import glob
  222. import os.path
  223. def parseTCX(filename):
  224. file = open(filename)
  225. xml_file = file.read()
  226. soup = BeautifulSoup(xml_file, 'lxml')
  227. id = soup.find("id").text # gets the UTC timestamp
  228. lats = []
  229. longs = []
  230. for tag in soup.find_all("trackpoint"):
  231. lats.append(tag.find("latitudedegrees").text)
  232. longs.append(tag.find("longitudedegrees").text)
  233. return id[:-10], lats, longs
  234. ```
  235. I am converting all the TCX files into a single [KML](https://en.wikipedia.org/wiki/Keyhole_Markup_Language) file to make it possible for me to visualize with various GEO tools.
  236. ```python
  237. def single_run(id, lats, longs):
  238. locString = ""
  239. for i in range(0, len(lats)):
  240. locString += longs[i] + "," + lats[i] + " "
  241. value = """
  242. <Placemark>
  243. <name>{0}</name>
  244. <description>xx Miles</description>
  245. <Style>
  246. <LineStyle>
  247. <color>ff0000e6</color>
  248. <width>4</width>
  249. </LineStyle>
  250. </Style>
  251. <LineString>
  252. <tessellate>1</tessellate>
  253. <altitudeMode>clampToGround</altitudeMode>
  254. <coordinates>{1}</coordinates>
  255. </LineString>
  256. </Placemark>
  257. """
  258. return value.format(id, locString)
  259. def convertToKML():
  260. base_path = os.path.dirname(os.path.realpath(__file__))
  261. files = glob.glob(base_path + "/tcx/*.tcx")
  262. header = """<?xml version="1.0" encoding="utf-8" standalone="yes"?>
  263. <kml xmlns="http://www.opengis.net/kml/2.2">
  264. <Document>
  265. <name><![CDATA[38415617200]]></name>
  266. <visibility>1</visibility>
  267. <open>1</open>
  268. <Folder id="Runs">
  269. <name>Tracks</name>
  270. <visibility>1</visibility>
  271. <open>0</open>
  272. """
  273. footer = """
  274. </Folder>
  275. </Document>
  276. </kml>
  277. """
  278. o_file = open("outputKML.kml", "w")
  279. o_file.write(header)
  280. for file in files:
  281. id, lats, longs = parseTCX(file)
  282. o_file.write(single_run(id, lats, longs))
  283. print(files)
  284. o_file.write(footer)
  285. o_file.close()
  286. ```
  287. Now that we have a single KML file with all of our running data coordinates, we can visualize it with google maps -- or any number of other KML tools.
  288. This visualization is nifty since it lets you see every place that you have run.
  289. ![Kml visualization](media/fitbit/kml3.jpg)
  290. As accurate as GPS may appear on our phones, we can start to see inaccuracies in our data when looking at the aggregate plot.
  291. For example, occasionally, the GPS will drop and result in a plot that has "random teleportation."
  292. ![Kml visualization](media/fitbit/kml2.jpg)
  293. Additionally, GPS on our phones is not pinpoint accurate.
  294. By interpolating the data as we go, each track individually appears smooth. However, the overall route is not all that accurate.
  295. In the following image, we see that although I stay strictly to the sidewalk and make a sharp turn, it puts me all over the road and in the yard.
  296. ![Kml visualization](media/fitbit/kml1.jpg)
  297. # Future Work for Project
  298. In the future, I want to transfer more metadata from the TCV files and include it in the KML files like total distance, average speed, mile markers.
  299. Additionally, visualizing this on a website that auto-pulls new TCX data and updates an embedded map would be interesting.
  300. The end goal is to do clustering on my TCX data to identify unique running routes that I have taken so that I can either plan new running routes or select a course that is the desired length.