As I mentioned in my second post about RSI, I installed an application called WhatPulse that measures your keyboard and mouse usage, providing insight in how you use your computer. It produces a heat map of your keyboard, showing which keys are the most frequently used for different time periods.
I wanted to get a more detailed analysis of how I was using the keyboard, so I started looking into how data is stored. I found out that WhatPulse stores its data in a SQLite database named
~/Library/Application Support/whatpulse, so I started poking around a bit. I used an application called Liya to dig into the data.
There are quite a few tables available, everything from total number of keypresses, to which key was pressed how many times in which application. The ones that I was interested in were the keypress-frequency and the keycombo-frequency. I wrote a small Ruby script to read the data, and post it into Elasticsearch. First I was looking into using Logstash with the SQLite adapter, but I couldn’t get it to work properly, and besides, I was interested in loading historic data, not updating elastic as soon as new data came in. The reason for using Elasticsearch was basically just that I wanted to try it out, learn a bit in the process, and also checking out Kibana. There might have been better tools for the job (I did the first graph in Excel), but this was a learning experience.
It took quite a lot of time deciding on how to structure the data in order to make it easy to analyse. I ended up with one object per hour, making the
_id of the object to be of the form
2015-09-30T08:00:00+00:00. The object has one property per key, being of the form:
This data tells me how many times the Space bar was pressed during that hour on that date, how many keys were pressed in total during that hour, and the percentage of the space bar in relation to the total number of keypresses. By having an hourly sum instead of a daily, I get a better result, since only active hours are counted. This allows me to graph it daily, but averaging the active hours, instead of just making every hour in the day count.
As you can see from the screenshot of Liya a bit further up on the page, only keycodes are stored in the database. For “normal” keys, I could just do a
key.chr to get the name of the key, but for the others, I made a simple lookup table:
16777217 => "Tab",
16777219 => "Backspace",
16777220 => "Enter",
16777252 => "Capslock",
16781922 => "Left Shift",
16781917 => "Left Ctrl",
16781915 => "Left Alt",
16781904 => "Right Command",
16781905 => "Left Command",
16781914 => "Right Alt",
16781916 => "Right Ctrl",
16781921 => "Right Shift",
16777232 => "Home",
16777233 => "End",
16777234 => "Left",
16777235 => "Up",
16777236 => "Right",
16777237 => "Down",
16777222 => "Insert",
16777238 => "PageUp",
16777223 => "Delete",
16777239 => "PageDown",
16777216 => "Escape",
32 => "Space",
46 => "Period",
16777264 => "F1",
16777265 => "F2",
16777266 => "F3",
16777267 => "F4",
16777268 => "F5",
16777269 => "F6",
16777270 => "F7",
16777271 => "F8",
16777272 => "F9",
16777273 => "F10",
16777274 => "F11",
16777275 => "F12"
So, I ran the script, got a bunch of data into Elasticsearch, and now what. That’s when I installed Kibana. Kibana is a flexible analytics and visualization platform that is built to work with Elasticsearch. Not knowing what to expect, I started up the Kibana service, pointed my browser to
http://localhost:5601 and was greeted with the following view:
After reading a bit of documentation, I started by entering the name of the index in Elasticsearch I had imported the data into. You can see on the left in the screenshot the number of times I’ve experimented with this :). I knew that the index contained time-based events since I imported them with date and time. After entering the name of the index I wanted, Kibana looked through the index, and presented in the combo-box the different date fields that were available in the index. I chose the correct one (the one simply named
date. I have a bunch of other (redundant) dates also defined (one per key entry).
After creating the index pattern, Kibana shows a list of the fields that are found in the index. My example data contains a lot of redundant fields, but for experimentation I wanted to include any data that I might need. I then looked at the Discover tab, and saw… nothing. I thought I had done something wrong, something got botched in the import, or something else. The reason was that by default, the time interval it shows data from is the last 15 minutes. Since I imported my data from a several days old copy of the SQLite database, I did not have any data that new. Clicking the time interval in the top right allows you to choose a more interesting time interval.
Now it was time to do some visualisation. For this example, let’s say I want to see how the ratio between Space and Backspace usage changed over time since I switched over to the TrulyErgonomic keyboard. I chose a line chart from the Visualize – Create a new visualization menu, created a new search source from my selected index, and started adding Y-axis metrics. I chose Average aggregation for the fields
Space.percentage as my Y-axis, and added a Date Histogram as my X-Axis, with a weekly interval. After clicking the green “play” button on top of the left-hand panel, I ended up with something like this:
Nice! We see that there is correlation between Space and Backspace, and on the whole, I use space more than backspace. However, I want to measure the difference between these two values, to try and graph my learning curve with the new keyboard. Luckily, Kibana 4 has support for scripted fields. You can create a new scripted field for an index by going to Settings – Indices and for your index, choose the tab named Scripted fields followed by the button Add Scripted Field. My scripted field looks like this:
Now, we go back to our visualization and add a new average aggregation on our Y-Axis, and choose our newly created, scripted field. Remember to press the green play button, otherwise your changes will not show up.
There we go, we can now try to interpret what we are seeing here. The higher the diff curve is, the fewer mistakes I make. I started using my TrulyErgonomic keyboard in the middle of October, and we can see that the curve goes downward almost directly. After a couple of weeks, we are seeing progress, and it seems like I’m actually doing better now than I did previously with my old keyboard. Unfortunately I don’t have that much historical data about my keyboard use before switching over to the TrulyErgonomic, so we can’t be sure of what my baseline is.
I’m not sure that I’ve done things properly, or in the way they should be. This was just my first experiments with Elasticsearch and Kibana. Let me know if you have any comments on my analysis.