Simon Wheatcroft: Is it scary? Yeah, it's really scary, 'cause it does go wrong. I’ve run into posts, that hurts. You just make sure you know where it is next time, and move along.

Cara Santa Maria: What's scary was running blind. Literally running blind.

Simon Wheatcroft: At the age of 11 and 12, I was diagnosed with retinitis pigmentosa.

Cara Santa Maria: Despite his blindness, Simon Wheatcroft ran long distances for years. But then he decided to run the New York marathon, solo, without a human guide. Doing that involves the most creative use of data you've ever not seen. Welcome to Fixed That for You, an original podcast from Segment about solving challenges problems with data and algorithms. I'm Cara Santa Maria, and in this episode, we go running through Manhattan, looking for the future of personal navigation with haptic technology.

Kevin Yoo: We created a virtual corridor that allows you to practically be staying on track. As soon as you deviate off the path, vibration beings to increase and then it pushes you back towards the correct direction.

Cara Santa Maria: It's a journey of discovery for an industrial designer from Brooklyn and a blind runner from England. Their goal might be running a marathon, but if it works, this could mean more freedom and accessibility for the over 280 million visually impaired people worldwide.

Kevin Yoo: The way that this product exists, it's not only for the blind and visually impaired. It's a new, innovative way for navigation for tourism, for bikers, for snowboarders. Imagine you could just put your phone away and have a very free journey.

Cara Santa Maria: Simon Wheatcroft started to lose his vision in the sixth grade.

Simon Wheatcroft: Retinitis pigmentosa is genetic, so you may be born with good vision but the older you get, the more your retinas deteriorate and the more your vision disappears.

Cara Santa Maria: By the time he went to university, he was blind.

Simon Wheatcroft: Often with a label comes perceived capabilities. I could no longer read, I could no longer see faces. And perhaps you believe yourself that you’re only capable of certain things. So it was at that point in time that I thought, maybe now is the time to redefine what it means to be blind and what it means in terms of what you can do.

Cara Santa Maria: Simon chose something big as his first challenge. Climb a mountain in Yosemite National Park and propose to his girlfriend at the summit. Even though they didn't quite make it, she married him anyway. But the failed climb stayed with Simon.

Simon Wheatcroft: I thought, I'm never going to quit again just because I can't see. Let's go out and see if we can push against the boundaries of what's possible. And that's when I decided to see if I could learn to run alone.

Cara Santa Maria: Simon started running outdoors by himself.

Simon Wheatcroft: Initially, it started on an empty football pitch. I was using an app on my phone, Run Keeper, and it gave distance markers through audio.

Cara Santa Maria: He would go 100 yards in one direction, then turn around and go the other way. That led to running on a closed road.

Simon Wheatcroft: From that point on, I just learned to run the open road and I memorized what it felt like to run those three miles, paired with the audio distance markers.

Cara Santa Maria: Okay. Try it yourself. Walk down the sidewalk with your eyes closed. How far can you get before you get nervous? It's a bit easier for a blind person used to navigating in the dark. But it's still really difficult.

Simon Wheatcroft: I was running the edges of curves, along roads with 60 miles an hour speed limits. If I went wrong, I was going into the cars. The hardest moments was my wife would drop me off with my eldest son in the back, and she'd say goodbye. And realistically, we never knew if I was coming back.

Cara Santa Maria: Simon could run solo if he stuck to his local three-mile stretch of road. But when he started doing marathons, he used a guide. A sighted runner to make sure he stayed on course and didn't run into anybody or anything. After a while though, it got boring.

Simon Wheatcroft: I've come to realize at this point in life, it never seems enough. I would perhaps prefer to push that boundary to breaking point. I fell into this cycle where, now I can train solo, now I need to compete solo.

Cara Santa Maria: And just to up the stakes for his first race without a human guide, Simon chose an ultra marathon. A 100 mile race through the Sahara desert. He partnered with a tech sponsor to create a navigation tool with audio cues. If he went in the wrong direction, a series of beeps would alert him. But it didn't always work so well.

Simon Wheatcroft: It turns out there's a number of obstacles in a desert. The first being a flagpole, which I ran into. And the next obstacle was a field of rocks.

Cara Santa Maria: He twisted his leg badly on the rocks and had to drop out at 83 miles. He was so close. But during his long recovery, he discovered another way to reach his goal of running solo. At the same time Simon was running around his neighborhood, Kevin You, an industrial designer, was in New York, working on navigation tools for people with visual impairment. But instead of audio beeps, which can get really annoying, he was using haptics — communication feedback via touch, often through vibration.

Kevin Yoo: It's been used for notification purposes, 'cause it's a very alert-oriented system. You feel a vibration, I'm on, what is it? That's why all the phones and everything already have haptics embedded.

Cara Santa Maria: He wanted to combine haptic communication with existing tools for people who have low or no vision. Like the traditional white cane.

Kevin Yoo: Because you are angled to point it downward, and everything beyond 45 degree above to your face, if there's a branch or AC or whatever, you're in the city, you just get hit in the face.

Cara Santa Maria: Kevin's solution was to add an ultrasonic sensor to the cane that pointed upward. If it detected any obstacles, you felt a buzz in your hand. Soon after, he also started work on a haptic navigation system. The idea was to take directional information from the GPS in your phone and feed it to you through a vibrating wristband. He and a couple of other guys formed a company called WearWorks, and they called their invention the Wayband.

Cara Santa Maria: That's when Kevin got an email from a blind runner in England.

Kevin Yoo: Simon reached out and said, "Hey, can you make a device that can give me a minimal channel of conversation with a technology that can guide me?". And at this point I was like, "Wow, no, not yet. But we totally can".

Cara Santa Maria: All Kevin had at this point was the wristband.

Kevin Yoo: It was not connected to any GPS or any navigation information at all.

Cara Santa Maria: The Wayband didn't have any data yet. So Simon suggested the perfect testing ground. The New York marathon, just four months away.

Kevin Yoo: That was the goal, literally design it for a runner, design it for the marathon and make it work.

Cara Santa Maria: A navigation system needs three types of data. Destination, orientation and location. Location is where you are. Orientation is what direction you're facing. And destination, that's where you want to go. For Kevin and Simon, destination was easy. That's the route of the race. They had software developers create a phone app that connected to Simon's wristband using imported GPS data. Then they drew a series of dots, marking the marathon route, and made each of them a sequential destination.

Kevin Yoo: It's like playing Pacman and you're collecting these dots.

Cara Santa Maria: Those dots drew a path through all five boroughs.

Simon Wheatcroft: Central Park is an issue, 'cause that's sweeping turns. Whereas if you then compare that to running the actual blocks of New York, which are all hard angles, that's easier.

Cara Santa Maria: Sweeping turns required a lot of dots. Right angles only needed one on each corner. So that's destination. Then, Kevin looked at the orientation data, which was collected using the phone's internal compass. The problem was, the marathon route crossed two massive bridges, bridges that are made out of metal.

Kevin Yoo: There's a lot of magnet problems. The directionality orientation chip that's on the phone, it gets distracted by big metals and oceans. So if you go on a bridge with your phone, orientation spins, it's completely off.

Cara Santa Maria: Luckily, bridges are usually straight. So even if the gyro was spinning, Simon would know to keep going in the same direction. Now, collecting location data with the phone's internal GPS unit should have been simple. But New York, as you may have noticed, has more than its fair share of skyscrapers. And those tall buildings block signals from satellites, forcing your phone to guess.

Kevin Yoo: Pinpointing these areas with high-density buildings and difficult information collection of data was step one.

Cara Santa Maria: Once Kevin found the trouble spots, he plotted corrections.

Kevin Yoo: Whenever I felt a disturbance in the data, I would change the point myself. And then to see if I ran that pathway again, would it equalize?

Cara Santa Maria: He was able to do this thanks to a specialized platform the developers created.

Kevin Yoo: We're canceling out wrong data information that already exists within all of our phones because of magnetometers, big buildings. And we canceled that out constantly by adding in the data points that we predicted would be the mistake.

Cara Santa Maria: Essentially, at key spots, they had to lie to their own app.

Kevin Yoo: This only worked because we had the route of the marathon, and that was consistent.

Cara Santa Maria: But then Kevin discovered a bigger issue with the orientation data. Think about how you hold your phone. It's in front of you, you're facing the screen. So the compass in your phone is built for that. But the plan was for Simon to run with his phone strapped to his upper arm, making it face sideways.

Kevin Yoo: If you tilt it sideways, now the orientation is wrong. That's the way that you have to put it in your arm. So we had to make that slight alteration, so that even if it's 45 degrees angled on the side of your arm, it'll still be registered as pointing forward.

Cara Santa Maria: That meant hacking into the back end of Simon's phone.

Kevin Yoo: It's actually not an easy problem, but we just put a layer, a platform to fold the phone, angle never happened.

Cara Santa Maria: It's quite the system.

Kevin Yoo: So we have the cellphone on one arm and we have the product in the other arm.

Cara Santa Maria: That's the thing that vibrates.

Simon Wheatcroft: Imagine a little sleeve that slips up to your bicep. It's quite firm on your bicep in sort of an elasticated way.

Cara Santa Maria: Every time Simon reaches a dot on the map, the system navigates to the next one.

Kevin Yoo: The new orientation to the next dot is now presented. You will now feel that, you'll feel that the vibration has begun.

Simon Wheatcroft: And then you just feel the vibrations on your bicep.

Kevin Yoo: And it'll gently push you towards the non-vibrational route, which is now the right pathway. So continuously by doing this, you're following down the path of the marathon.

Simon Wheatcroft: And then we ended up dialing that up a fair amount as well, 'cause what happens is, as you're running, you're actually generation a lot of vibration through the body. So then you need the haptics to be able to be a big enough indicator over that natural body vibration.

Cara Santa Maria: Programming the haptic vibrations was almost like writing a new language. Different patterns and intensities meant different things.

Simon Wheatcroft: So let's imagine, we've exited the corridor. We're going slightly wrong. Then we're beginning to get vibration. But we also happen to be coming up to a corner, so then you need to be able to have a totally different pattern to denote that you need to turn versus you've excited the corridor.

Cara Santa Maria: All of these adjustments were done in the middle of Central Park. Every day, Simon would meet with Kevin. Sometimes a software expert or one of the hardware guys would join them. They would run a few miles, change some settings, then run the same route again. That was fine for Simon, he was a well-conditioned runner. Kevin was not.

Kevin Yoo: I was stressed out. One of those guys is working until four in the morning, smoking outside, coming back in. So I was very unhealthy during the long course of the evolution of the product.

Cara Santa Maria: But it was becoming clear that someone had to run the race with Simon to troubleshoot technical problems.

Kevin Yoo: I was running with the device a lot to test it out. Over time I realized, I was training. I was training practically for the run.

Cara Santa Maria: And so the decision was made. Kevin signed up for his first marathon.

Simon Wheatcroft: I think it definitely made me respect him more. The fact that he was not only creating this technology but he was willing to step up.

Cara Santa Maria: But before they could do that, Simon and Kevin had to deal with yet another problem. And any of you who have ever run a marathon or even seen one on TV have probably already figured out what that is, right? He wouldn't be running the marathon alone. 50,000 other people would be running right next to him.

Kevin Yoo: For the run of the marathon with Simon, I had to make a secondary product.

Cara Santa Maria: For the secondary product, Kevin went back to his earlier invention, the traditional white cane with an added sensor. This time, he strapped it onto Simon.

Kevin Yoo: So it's an ultrasonic sensor embedded onto the chest, and it detects people that's far away, ten feet distance.

Cara Santa Maria: It vibrated when an object or another runner entered that zone.

Kevin Yoo: The day before the marathon, I tested it on Simon and we started running. My assumption was, if you get objects getting closer, you wanted the vibration to be stronger so they could avoid it. That was my assumption, but I was wrong.

Cara Santa Maria: It's a case of the designer not understanding the end user. Kevin doesn't have a lifetime experience bumping into things like Simon does.

Kevin Yoo: He said, "I can't detect the people early enough because I'm traveling to fast. And by the time I detect it, I'm already too late". So I had to reverse the code so that the vibration would be the strongest as soon as an object in enters in from ten feet and closer.

Cara Santa Maria: Simon also wanted a second piece of information.

Simon Wheatcroft: A runner could enter the area where it would vibrate, and then exit, but it would still vibrate. So I'd get no indication that they had left and exited. So I was like, hang on a minute, we need to switch this around. As they leave, and enter, it changes the vibration so I understand if people are moving past me, to the left of me or to the right of me. A sort of visual field almost.

Cara Santa Maria: All of this was happening less than 24 hours before the start of the race. It's the morning of the race and Simon and Kevin are running on a combination of adrenaline and self doubt.

Kevin Yoo: Whenever you're testing a piece of technology that is in the prototype stage, there's always a big question mark of is this even possible?

Cara Santa Maria: By this point in his life, Simon's done things most of us would never even imagine. But running unaided in the largest marathon in the world, this was scary, even for him. The race starts by crossing the Verrazano bridge, from Staten Island into Brooklyn.

Simon Wheatcroft: You don't really get much chance to come and test it very much on that bridge. So straight away, there's issues.

Cara Santa Maria: Remember, bridges are made with lots of metal. Bad for GPS.

Kevin Yoo: I knew that the disturbances of the metal and everything would be too much. I just said, Simon, this is a bridge, it's going to go forward. I'm going to put the device on forward, and that's it. Very simple.

Cara Santa Maria: They aren't really using the navigation aspect of the device but it does keep Simon running straight so he doesn't fall off the bridge into the harbor.

Simon Wheatcroft: Now the system that tells me is there something in front of me, works fantastic. Incredible. Running across that bridge felt like the first time I had an opportunity to compete solo in a complex race, and it was incredible. I just was weaving about, running along that bridge, avoiding people, dropping in behind people. Playing around with how close I was getting to the person in front.

Kevin Yoo: As soon as we crossed the first bridge, I turned the activation of the route. And then from there on, it began.

Simon Wheatcroft: As we come off the bridge, our system suggests that I'm in the wrong spot. But I know I'm in the right spot because I've run this race before.

Cara Santa Maria: The Verrazano bridge has two levels. Simon's on the lower deck but the course they've programmed is for the top deck. The exit ramps are completely different.

Simon Wheatcroft: That was a bit of an issue, but again, because I'm relatively comfortable with running by feel, I knew that you dropped off the bridge to the left. If you run the right line, you know you're going in the right direction. Cara Santa Maria: 16:18 After half a mile or so, the two routes converged and the GPS confirmed he was in the virtual corridor. Simon was cruising through Brooklyn, enjoying his run. Ten feet behind him, Kevin was finding new things to worry about.

Kevin Yoo: My biggest fear during the run was disturbances caused by other people.

Cara Santa Maria: There were 2.5 million people cheering on 50,000 runners, and everyone of those fans was snapping photos and posting them on social media.

Kevin Yoo: You have an incredible amount of data transferring from everywhere. So even though I ran the route multiple times with Simon and tested out the major points of difficulties, with the addition of this problem of many people being there and using a lot of data, it just made it worse.

Cara Santa Maria: It got really bad on looping turns where the app needed to locate more of those mapping dots. The only way to correct this on the fly was to reboot Simon's phone.

Simon Wheatcroft: We end up in this little cycle of, where it's working, it's not working, it's working, it's not working. It's just getting really difficult.

Cara Santa Maria: But there were periods where everything clicked.

Kevin Yoo: Simon was literally avoiding people using the ultrasonic sensor, and using the Wayband support to navigate. Really I was like, wow, this is working. I'm there, I'm running, I'm seeing it happen, it was amazing.

Simon Wheatcroft: You do get these waves of realization of what you're doing. You're doing this alone and you're doing this alone thanks to technology.

Cara Santa Maria: But then after 16 miles, the runners crossed their second bridge and entered Manhattan. At this point, the crowds got even bigger and the data usage skyrocketed.

Kevin Yoo: Immediate overload of information that the data just could not reach us. And this allowed for a lot of glitches, a lot of errors.

Cara Santa Maria: This was also when, on what was already a gray and drizzly day, it started pouring.

Simon Wheatcroft: The rain deteriorated the ultrasonics to the point where, is the rain making this vibrate or is there someone in front of me?

Kevin Yoo: I should have thought about that more in advance. It was water resistant, it was sweat resistant, but it definitely was not waterproof. And the device actually failed about 17 miles in.

Simon Wheatcroft: As I removed the technology, it's almost like I handed over my energy and will to move. It just broke me.

Cara Santa Maria: But then Kevin, the first time marathoner, stepped up.

Kevin Yoo: I was the plan C to literally, Simon, something has gone down. Now we have to finish the rest of the race purely by voice. And this is where I was like, okay, let's just run.

Simon Wheatcroft: It was hard to get to the end, it was really hard.

Cara Santa Maria: Really hard, but they did it. In a time of five and a half hours, Simon Wheatcroft and Kevin You completed the 2017 New York City Marathon.

Simon Wheatcroft: If we had chosen to do a half marathon, we would have done it. Perhaps it was my nature of always trying to push things as far as possible that we chose a bigger distance.

Cara Santa Maria: But after the disappointment of not running solo the whole way, Simon came to appreciate the accomplishment.

Simon Wheatcroft: We got to glimpse what was possible. We got the opportunity to demonstrate that this technology really could enable me to do something alone.

Cara Santa Maria: And this is what Kevin and his colleagues at WearWorks are hoping to build on, thanks to better data. I Triple E satellites and Qualcomm GPS chips in most cellphones will soon be updated.

Kevin Yoo: Goingtowards the future, I Triple E update, it's going to do one point accurate assumption. So this is going to make the accuracy of GPS to three centimeters.

Cara Santa Maria: And remember, all of that data was used to just get one guy across the finish line. Kevin's now adding voice options to the vibrations, and also looking for ways to create indoor routes, which would be amazing for say tourists, blind and sighted, in the New York City subway.

Kevin Yoo: We want to connect subway, public transportation, all into one fluent navigation routine.

Cara Santa Maria: And then imagine what could happen when tens of thousands of people start using it and agree to anonymously feed their hyper-specific route data back to the app.

Kevin Yoo: In the end, the data collecting and the data usage will be for identifying what route is the best for you, best on less amount of stress levels. No construction, no potholes, no heavy traffic and crowds of people. These are just small ways I think we can help with data.

Cara Santa Maria: But not just in small ways for some people.

Kevin Yoo: For blind and vision impaired, it will be life changing.

Cara Santa Maria: Life changing solutions achieved with data and algorithms on another episode of Fixed That for You. An original podcast by Segment. If you want to find out more about Simon's story or WearWorks, check out the show notes. And you can find us at segment.com/podcast. Plus, subscribe at Apple Podcasts, Google Podcasts, Spotify, or wherever you do that sort of thing. I'm Cara Santa Maria. Thanks for listening.