Quantified Self : Analyzing Personal Garmin Data (2012-2017)

I. Introduction¶

Insights into my fitness journey (2012 - 17)¶

I have been an avid runner and fitness enthusiast for the past 5 years, and what has been a fun part of all those is how much we can learn about ourselves from exploring the data we capture.

I started capturing data about myself around the year 2012, which was when we saw the advent of wearable devices, and have always wanted to carry out an analysis to gain insights about my running patterns, my strength training schedules, my diet logs and any other metadata which I can collect about myself like the Heart Rate, Cadence for runs and also the step counts.

About Quantified Self : When learning about the ways to explore about the data you capture, I came across the term Quantified Self and have been following the blog ever since. Its just amazing how capturing and analyzing data about yourself can actually help me make data driven decision to improve your lifestyle and so many people are joining this movement and bringing a data-driven change to improve their lives.

Questions which I would like to answer in my project analysis are as follows :

• How much time have I spent on various activities like Running, Treadmill Running, Strength Training Sessions, Cycling and Cardio Sessions?
• What is a count of the activities and how do they vary across specific months over the span of 5 years?
• Activity Day Analysis : What has been the most active day in the week and the distribution of activities across specific days of a week
• Calorie Analysis : An analysis of the distribution of calories burnt during activities, along with the distribution of calories burnt during specific activities and detecting outliers (which I am happy about)
• How many calories have I burnt in total for every activity and see any trends for the activities across the years 2012-2017? Interactive Plot using D3.js
• What is the hour of the day that I usually do the activities and how has it changed over the past 5 years? It will help me relive the changes that have happened in my life and how they have affected my schedules. Interactive Plot using D3.js
• Analyzing the trend of how the Activity minutes vary across the months for years 2012-2016. Interactive D3.js visualization
• Analyzing the trend of how my running mileage varies across the months for the year data 2012-2016. Interactive Plot using D3.js
• Analysis of the total minutes spent on each activity type during the months in year 2012-17
• Analyzing average calories burnt for every activity type during the months of years 2012-17
• Analyzing average heart rate based activity levels and its variation along the aggregated days of the week and aggregated activity types

I do take inspiration from the Quantified Self Running archives and will try and do an analysis on all these aspects about my data.

Where the reader can find the data? I would like to share the data captured about all my activities if someone wants to analyze it. The data used for this project is hosted on Github at this link. This code file has also been hosted on Github and can be accessed at this link. You can always mail me to get the most updated data at anujk3@gmail.com .

II. Team¶

I will be working by myself on my dataset and gaining insights through the analysis I do. The stages of the project have been highlighted below:

• I had to go to the activities page on Garmin and download around 22 pages of data one by one which had all the data in a csv format. Sadly, Garmin does not offer an API like fitbit which makes it difficult to gather any data for analysis.
• After getting the data from Garmin, the next part was combining all the csv files together and converting it into a dataframe which can be analyzed further. All the processing was done in python using pandas.
• The initial processing involved cleaning the data, remove extraneous activities like walking, swimming etc. which I haven't tracked a lot, correcting the data errors, correcting datetime fields for the ease of datetime analysis.
• After initial cleaning of data, I was focussed on data transformations to add dervied columns to the data which can be useful to exploratory data analysis.
• The next part was focussed on doing an analysis of missing values followed by the existing continuous and discrete variables in the dataset which involed an analysis of the missing values along with an analysis of important univariate distributions of continuous columns and multivariate analysis of continuous variables.
• The next part was to figure out the important questions which I can answer using the dataset and work on specific problems to come with relevant visualizations for explaning them.

Programming Language Used :¶

Something I am really passionate about is doing data analysis work and Python has been my go to language for it. Through the course Exploratory Data Analysis and Visualizations, Prof. Joyce Robbins did help me realize how even R can help in quickly analyzing the data and actually the best part about R for me was getting to learn about ggplot.

I still prefer using Python as my language for data analysis but hacked my way to use ggplot within the Python Analysis and will be doing most of my dataframe analysis in Python and making plots with ggplot, but all within a Python Notebook.

I will also be working on interactive visualizations using D3.js and whenever possible matplotlib within Python along with seaborn.

III. Analysis of Data Quality¶

i. Import Statements and Setup¶

ii. Load Data¶

iii. Data Cleaning¶

Data Cleaning has been performed in the following steps to ease the process for creating the visualizations later.

Ignore the swimming activity, only 1 row

Convert Elliptical to Cardio

Ignore Walking as an activity

Ignore swimming as an activity as only 3 recorded activities

Activity type Strength Training was recording as Others in older Garmin Devices

Activity Types Indoor Rowing and Indoor Cycling also included as Cardio

Correcting Data Errors and Data Types

iv. Exploring the data and the datatypes¶

Initial Peek into the data

v. Data Transformations¶

Creating Derived Fields that can be used for the visualizations. The transformations have been highlighted.

Converting startTime to Python DateTime Format to use the inbuilt functions

Concatenating information about the Month and Year for every activity to the DataFrame

Obtaining Activitiy Levels using the average HR field for all the workouts

Calculating total minutes of an activity from the data

Checking final data types

Creating a list of continuous and categorical variable

vi. Summary of Continuous Variables and Missing Data Analysis¶

The summary of continuous variables helps us figure out the count of values for all the variables where the values are present. It helps us explore the number of missing values for the variables and begin the analysis of MISSING DATA

Looking at the count of values for all the continuous variables, we observe that out of 494 activities, 211 activities had distance recorded, 80 activities had elevationGains recorded, the average speed was recorded for 211 activities, with average and max Heart Rate being recorded for 336 activities.

Calories, Activity Month, Activity Day and the Activity Minutes were recorded for all the activities

Some of the reasons for missing data are as follows:

1. During the span of 5 years, I have changed 4 wearable devices. Initially I used the Garmin Vivofit tracker for all the runs, which did not have heart rate tracker.
2. After that I bought the Garmin Forerunner 920 (GPS inbuilt), which had a heart rate strap along with it and it also used to calculate the elevationGain and many more parameters for all the runs. But I do realize that I was lazy to use the heart rate strap as it needed to be washed after every run.
3. After that I used the Garmin Vivofit2 tracker, which had inbuilt heart rate monitor but lacked GPS, so the runs were still being tracked by Garmin Forerunner 920 and whenever were tracked using Vivofit2, lacked heart rate information.
4. Currently I am using the Garmin Vivoactive HR, which has an wrist HR monitor as well as an inbuilt GPS, but being in Graduate School, the frequency of runs has gone down significantly.

All the variables in the analysis are important but lets do an initial missing data analysis with visualizations

Visualization : Missing Data 1

Visualization : Missing Data 2

Looking at the above visualization, the following patterns emerge:

1. Average Heart Rate and Maximum Heart Rate values are missing for those specific Activities where I was using a fitness tracker which did not have Heart Rate Tracking ability.
2. The statistics like Elevation Gain were tracked when I was using a heart rate strap attached, which being lazy I didn't use often, leading to the highest number of missing values, because of which we will not consider it as an important feature in our analysis.
3. For the first generation fitness tracker used like Garmin Vivofit 1 and 2, even though it had capabilities to record my Running Distance based on the number of steps take, it still did not process the other metrics like Average Speed and Heart rate values.
4. Also, the Running Distance values are missing for all workouts which involved Strength Training. If there was some recorded value for this field during Strength Training sessions due to the number of steps taken, it has been ignored and replaced with NaNs in my analysis.
5. Steps were recorded by the most recent fitness trackers used and therefore have a lot of missing values. During activities not involving Running, the Steps have not been considered and replaced with NaNs.

From the above analysis, it can be seen that the most important features for further analysis are Activity Minutes, Activity Month, Activity Day, Calories Burnt, Average Heart Rate, Maximum Heart Rate, Activity Distance Covered. The other features, specifically, Average Speed , Steps and Elevation Gain have a higher number of missing values and will not be considered important for future analysis, even though Average Speed and Steps can help me analyze the runs and find some correlations.

Visualizing Distribution of Continuous Variables

vi. Visualizing Distribution of Categorical Variables¶

vii. Multivariate Distributions for Continuous Data¶

Analysis of the above visualization:

A Scatterplot matrices is a great way to roughly determine if you have a linear correlation between multiple variables. This is particularly helpful in pinpointing specific variables that might have similar correlations amongst the dataset. Some observations are as follows:

• Calories burnt show approximately linear correlation with the minutes of a particular activity and the steps taken.
• Heteroskedasticity is observed for variables calories with activity Minutes and also activityMins and elevationGain.

IV. Executive Summary¶

I have been an avid runner and fitness enthusiast for the past 5 years, and have been capturing data about my training activities. One of the main motivations to do this analysis was to gain insights into my running patterns, my strength training schedules and any relevant information I can collect using all the data I have collected from my training sessions.

During the span of five years, I have changed 4 wearable devices. Initially I used the Garmin Vivofit tracker for all the runs, which did not have heart rate tracker. After that I used the Garmin Forerunner 920 (GPS inbuilt), which had a heart rate strap along with it and it also used to calculate the elevationGain and many more parameters for all the runs. The third tracker used was the Garmin Vivofit2 tracker, which had inbuilt heart rate monitor but lacked GPS, so the runs were still being tracked by Garmin Forerunner 920 and whenever were tracked using Vivofit2, lacked heart rate information. Currently I am using the Garmin Vivoactive HR, which has an wrist HR monitor as well as an inbuilt GPS.

In the summary, I will highlight the following main insights from the analysis:

• Hour of the day I perform the activities and how it has varied in the last 5 years?
• For every month of the past 5 years, how has the total activity time varied?
• Also, for the last 5 years, how has the running mileage varied and its comparison to the activity times?

For the analysis about the hour of the day I have performed the activities, I created the following interactive visualization using D3.js: Visualization Link (Inspired by learning from the course Storytelling with Data) Code

About the Visualization : In this specific visualization created using D3.js, the activities have been plotted as a scatter plot in increasing order of time range from 2012 to 2017. Knowing that the activities span a range of 5 years, I made the decision to not keep tick labels for x-axis but added a Data Range Slider below the x-axis to add interactivity to the plot (a date range can be chosen). The y-axis shows the time of the day (0-24 hours) and the activities are plotted as points on the graph. Also interactive buttons can be used to select any activity, like running, cycling etc. and only its patterns can be observed for a given date range.

The main insights that can be obtained from the visualization are as follows:

• Most of the activity sessions for the past 5 years have been in the evening, which clearly show that I am not really a morning person.
• Most of the morning sessions included running, which were the practice runs while training for marathons.
• Since moving to New York in August 2016, the rightmost part of the graph, I have mostly been doing the strength sessions late night at around 22:00 hour and some really late night cycle rides along the riverside, which can be clearly seen in the visualization. The increase in the cycling sessions after moving to New York can be attributed to the annual Citibike membership and daily usage of it during the Winter vacations after Fall 2016 semester.
• The fitness tracking journey started with mostly tracking the runs (leftmost part of the chart consist mostly of green dots) and slowly with the improved fitness trackers bought, I started capturing more data about varied activities I did.
• And looking at this visualization, I also realize that in the recent times, I have been missing the feel good factor after my long runs, and need to get back to running soon. It is clearly visible that I have mostly been focusing on the strength training sessions (rightmost pink clusters) in the recent times.

To understand the total activity time and how it varied for the last 5 years during specific months, I created the following D3 visualization : Visualization Link (Inspired by learning from the course Storytelling with Data) Code

About the visualization : In this specific visualization created using D3.js, the total aggregated minutes of the activity types have been plotted on the y-axis with its variation across the months for the years 2012-2016. The trend line plots have been specifically annotated for every year in the analysis. Hovering the mouse on the visualization highlights the closest data points and annotates it in the visualization with the minutes spent in that month. For this visualization, I am using curved lines which help us better understand the underlying variations in the data.

The main insights that can be obtained from the visualization are as follows:

• The tracking for the workout sessions began in 2012, with about 200 minutes of activity captured during the month of October 2012, but I was still not much into tracking at that time.
• During the year 2013, I started tracking my running activities only and most of the activity minutes for the months in 2013 are due to the runs that have been tracked.
• During the year 2014, I started my running journey and joined the Gurgaon Road Runners group and got inspired to track my runs more seriously. Even though I was still only capturing my runs for the year, I had a decent running time during most of the months from March - May. The time spent on running activities reduced because of extreme hot temperatures in India during June and July and then the I started training for a half marathon, due to which the activity minutes picked up during the end months of 2014.
• During the year 2015, I just ran a half marathon for the month of January and after that had a cool down period. Because of the advanced fitness trackers bought during 2015, from June I started capturing my strength training sessions along with the running sessions and along with it, I started preparing for a half marathon in December and full marathon in January, because of which there has been a high value of activity minutes recorded for the last half of 2015 season.
• For the year 2016, I was running regularly and also training regularly for improving my runs in the gym, because of which there has been such high activity recorded for the year 2016. I also ran my first full marathon in January 2016 and therefore have high activity recorded for the month of January 2016. The reduction in activity minutes as 2016 progressed can be aided to the joining of Graduate school in NY and recording mostly my strength training sessions.

Even though the activity minutes reduced as the year 2016 progressed, I was still capturing all my other training sessions than running and do have decent activity minutes captured. The difference of how graduate school really impacted my running schedule will be clearly visible from the following visualization of just the running kilometers tracked for the years 2014-2016.

And, finally lets look at the running kilometers tracked for every month in the years 2014-2016. The result is shown in the following visualization created using D3.js : Visualization Link (Inspired by learning from the course Storytelling with Data) Code

About the visualization : In this specific visualization created using D3.js, the running mileage has been plotted for every month of the years 2014-2016 with the running mileage on the y-axis and the months on the x-axis. I am only showing the data for these years as I was mostly capturing all my running activities during these years. The trend line plots have been specifically annotated for every year in the analysis. Hovering the mouse on the visualization highlights the closest data points and annotates it in the visualization with the total kilometers run in that month. For this visualization, I am using curved lines which help us better understand the underlying variations in the data.

The main insights that can be obtained from the visualization are as follows:

• For the year 2014, the running time has mostly varied with the running season in India. The running season starts around August and finishes with the Delhi Marathon and Mumbai Marathon in Decemeber and January respectively, showing higher kilometers run during those times of the year.
• For the year 2015, after the Mumbai Marathon in January is when you rest and condition your body for the next running season. Even though in the previous plot I see that the recorded activity time is very high for the months in 2015, the actual running miles is less till the beginning of the running season in August-September. As I was preparing for my first full marathon in January 2016, the higher kilometer runs for the month of November and December 2015 are the due to the practice long runs for its preparation.
• For the year 2016, the running kilometers are high for the year January when I was doing the practice runs and also ran my first full marathon. After that, I was just running about a long run once a week to keep conditioning along with the rest time after the season. In July-August 2016, I was preparing and moved to NY to join graduate school and during the initial months of moving to NY, I was still logging those kilometers due to the vicinity of Central Park near my house and favorable weather conditions but as the semester proceeded and the workload increased along with extreme temperature, the running kilometers reduced significantly to even less than 10km, which is really insignificant when compared to the previous years peak running months, i.e., November and Decemeber. But I hope to be back to running soon, and doing this analysis just made me realize how important running is for my happiness.

We will now proceed to the section where we will try and obtain more trends from the data, analyze the distributions of important variables in the dataset and obtain learning by analyzing all the created visualizations from the data.

V. Main Data Analysis and Visualizations¶

In this section, we will analyze the identified important features through our data quality analysis, namely, Activity Minutes, Activity Month, Activity Day and Calories Burnt.

i. Analysis of the feature - Activity Minutes¶

a. Distribution Histogram with overlayed Density Plots, with varying binwidths also

Analysis of the above visualization:

1. In the plot above, we see a distribution of Activity Minutes on the x-axis along with the count of those activities on the y-axis.
2. The Distribution is clearly multi-modal skewed towards the right. Varying the binwidths also led to similar distributions showing the underlying pattern of the distribution.
3. Reasons for skewness include the practice runs while I was training for my first full Marathon in 2016, the full marathon and also instances when I forgot to stop the timer while recording activities.
4. Activity Minutes data obtained was mostly clean, except for instances when I forgot to turn of the fitness tracker resulting into longer workouts, but they were cleaned during the data cleaning process.

b. Distribution of Activity Minutes across various Activities