Table of Contents
Basics
Objective
The goal of this project was to build a cross-platform mobile app that can gather breathing data as cheaply and efficiently as possible for use in a lab environment.
This is important because it allows for accurate breathing detection on a common device, the smartphone. This project can help increase the availability of information on hardware-minimal software-oriented biometric measurement tools, since they can be used where more complicated machinery might not be available.
What Features Does This Have?
- cross-platform
- accurate breathing detection
- efficient enough to run on phones
- uses as few sensors as possible (especially avoiding gyroscope as many low end phones don’t have one)
- full logging capabilities
- enough visual feedback to be easily used in a lab environment
Why Unity Instead Of Native Code?
- Development time.
- I only had one semester to complete this before I finished my undergrad degree, so this would be much faster than developing natively.
- Easy cross platform compatibility.
- This allows for the app to be used on multiple types of phones in our research lab.
- Why not port the original prototype from last spring?
- The original prototype was done in Max/MSP, which interfaced with phone sensors through the TouchOSC app. This setup was more suited to working with data streams, but required two devices (a computer and smartphone) to operate.
- There are also no documented or functional APIs for using Max/MSP with Android or iOS at this time of writing.
- Handing off the project to less experienced coders in the future.
- There was a strong possibility that his project will be passed on to someone else in the lab after I graduated. This unfortunately rules out React Native since that workflow may be too complicated for someone who doesn’t code.
Main Contributors
- Lyn Cassandra Phan (Georgia Tech Brain Music Lab)
How To Use
Basic Procedure
- Open the app on a compatible device with an accelerometer.
- Place the app on a participant’s abdomen. Participant can be sitting up with the phone leaned against them, but lying down produces the best results since it reduce movement.
- Wait for the app to calibrate. Total difference should be relatively close to zero and the threshold value should have settled to within a ±0.05 range. If this is taking too long, the app can be opened on the phone after it is placed so that it immediately zeroes to that position.
- Make a note of the directory where the log will be stored if you want to retrieve it later.
- Leave the phone on the participant as long as needed to collect breathing data.
- Remove the phone from the participant and close the app.
- If further analysis is desired, connect the phone up to a computer to retrieve the CSV log file. This is feature may require tweaking to work on iOS.
Interface Guide
Note that each axis position value is a floating point value measured between -1.0f and +1.0f. Any values referring to position are simply a planar representation of the rotational position. You can observe how this works by running the app and watching how the position values react to movement of the phone.
Video Demonstration
How Does This Work?
The Breathing Detection Algorithm
Is This Even Good?
Yes it definitely is. Using this method with data buffers, instead of more traditional reference point measurements, allows for slow deviations in participant movement, resulting in more accuracy over long term data collection. Since the thresholds are calculated automatically, no prior information or manual calibration is needed from either the device or the participant. And this works on even cheap smartphones and only uses a single accelerometer sensor. This is a true set it and forget it solution.
However, the main limitation is the use of only a single sensor, resulting in occasionally missed peaks or false positives. Much more precise motion tracking could be done through the use of a gyroscope and accelerometer in tandem. This is how extremely precise motion controls are achieved on modern game controllers, notably on the Nintendo Switch Pro Controller. Since not every phone contains a gyroscope, and adding support would dramatically increase implementation time, it wasn’t included for this project.
Glossary (In Order of Appearance)
Peak Detection - Finding maximums in a periodic signal. The algorithm used in this project outwardly functions similarly to peak detection but can be more accurately described as threshold detection.
Position Buffers - Position data from each axis is recorded into a buffer of a certain length.
Position Buffer Averages - All of these values are then averaged to produce an average value for each axis over the time duration of the buffer.
Buffered Average Differences - By getting the magnitude of the difference between a long (minutes long) and short (seconds long) buffered average for each axis, we can calculate how strong the current positional difference is from the norm. Perfect for using with threshold detection.
Detection Threshold - This is the threshold the buffered average differences are measured against. A value greater than this threshold will result in a positive flag raise for a possible detected breath.
Denoising and Debouncing- Removing or ignoring excess positives to reduce the chance for false positives or counting multiple times per breath. Especially useful in continuous live data streams such as this.
Sample Logs and Graphs
Example 1: Sitting Up With Normal Breathing
Sample rate: 60 samples/second
Let’s analyze some data. This is a sample taken from participant sitting at a computer desk for ten minutes, and contains about 26000 samples. The CSV file used in this example can be found here.
As we can see here, this is the CSV log file opened up in Excel. Unfortunately Excel can’t handle than many lines so all the graphs have been created in LibreOffice Calc instead. Ideally I’d be using some proper charting tool like Matplotlib’s pyplot, but at the time of writing there are some compatibility issues between Python on Windows and Numpy so I can’t easily do that yet (see: https://developercommunity.visualstudio.com/content/problem/1207405/fmod-after-an-update-to-windows-2004-is-causing-a.html).
This first chart displays the raw data from the accelerometer. Notice how each axis tends to hover around the same value as there aren’t any major changes in position. Also, notice the dip that occurs on the x short buffer. That dip is a change in position that propagates through each part of the data until everything normalizes again.
This second chart shows the short and long term averages of each axis. Notices how the long term averages stay relatively steady and linear while the short term averages react quickly and move with the orientation of the phone.
This third chart shows the total difference and the breathing detection threshold. Denoising and debouncing in the algorithm allow for breaths to be neatly counted from this interaction (not shown but it’s in the log file as a column of booleans). Also, notice how the threshold adjusts to the shift in position mentioned earlier.
This fourth and final chart shows the breathing rate. Here especially we can notice issues with the way the data was recorded. We can confirm that the shift in position caused a short section where nothing was picked up before the algorithm self corrected. We can also see a spike right near the end caused by the participant moving, which matches spikes in the earlier charts.
Example 2: Lying Down With Relaxed Breathing
Sample rate: 60 samples/second
This sample log was taken with the participant lying down and being directed to relax with the phone placed on their stomach.
We can confirm this orientation by seeing the that the Z value stays near -1.0, and the X and Y values stays near 0.0, indicating the phone was laying face up on the participant. We can also see several instances where near impulse responses are present in the data, as all three axes are show a change in position. These were from when the participant moved. At the end we can see a giant impulse response as the phone was picked up and recording was halted.
This second chart confirms what we knew earlier, and doesn’t add much, but we can clearly see some kind of periodic movement with the short term averages.
The third chart also shows the shifts in movement from the first two. Here we can also see the periodic change of the total difference and how the breathing detection threshold stayed near constant the entire time.
This last chart is much more interesting because it shows a calculated breathing rate over time. The one spot where the value reaches zero is very interesting, as the participant had breathed steadily the entire test. This may have been a result of a more than subtle shift in position, or the parameters of the algorithm might needed to be adjusted to decrease the threshold more quickly for this specific case.
Example 3: Sitting Up With Erratic Breathing While Playing A Game
Sample rate: 60 samples/second
This third log was taken with a participant playing a first person shooter game. The phone was leaned up against their abdomen or recording. The expected result was a much more erratic change in breathing rate and positions over time.
This first chart shows much more dramatic movement, as indicated by the increased wiggliness of all three axes.
In the second chart we can also see that there is more movement, as well as changes in position over time. Notice the subtle curve in the long Y buffer as well as the changes over time in the long Z buffer.
This third chart also shows some change over time due to the curved threshold line. If the graph did not scale to the extreme last section of values, it would likely look a lot more like the corresponding chart in the first example.
As expected, the breathing rate recorded in this log matches the expected hypothesis of a much more dramatic shift in breathing rate throughout the trial. The decreasing and flatline at the end could have been caused by many confounding variables, but it is most likely due to the position shift near the end that we can see as a near vertically divided section in the first chart.
Try It Out!
If you want to try out the Android build, BreathingDetection/bin/Android/build.apk is the most recent Android build of the software. You’ll need to allow apps to be sideloaded on your device to run this. This build was tested on a Motorola Moto G5 Plus.
There’s no iOS build of this yet as I didn’t have time to ensure compatibility, but you should be able to clone this repository and build it yourself. There may be issues with the way files are stored on iPhones, but that’s something I didn’t get a chance to look into yet.