Two years in the past we introduced Undertaking Guideline, a collaboration between Google Analysis and Guiding Eyes for the Blind that enabled individuals with visible impairments (e.g., blindness and low-vision) to stroll, jog, and run independently. Utilizing solely a Google Pixel cellphone and headphones, Undertaking Guideline leverages on-device machine studying (ML) to navigate customers alongside out of doors paths marked with a painted line. The expertise has been examined everywhere in the world and even demonstrated throughout the opening ceremony on the Tokyo 2020 Paralympic Video games.
For the reason that authentic announcement, we got down to enhance Undertaking Guideline by embedding new options, similar to impediment detection and superior path planning, to soundly and reliably navigate customers by way of extra advanced situations (similar to sharp turns and close by pedestrians). The early model featured a easy frame-by-frame picture segmentation that detected the place of the trail line relative to the picture body. This was adequate for orienting the person to the road, however offered restricted details about the encompassing atmosphere. Enhancing the navigation indicators, similar to alerts for obstacles and upcoming turns, required a a lot better understanding and mapping of the customers’ atmosphere. To unravel these challenges, we constructed a platform that may be utilized for quite a lot of spatially-aware purposes within the accessibility house and past.
At this time, we announce the open supply launch of Undertaking Guideline, making it obtainable for anybody to make use of to enhance upon and construct new accessibility experiences. The discharge consists of supply code for the core platform, an Android utility, pre-trained ML fashions, and a 3D simulation framework.
System design
The first use-case is an Android utility, nonetheless we needed to have the ability to run, take a look at, and debug the core logic in quite a lot of environments in a reproducible manner. This led us to design and construct the system utilizing C++ for shut integration with MediaPipe and different core libraries, whereas nonetheless with the ability to combine with Android utilizing the Android NDK.
Beneath the hood, Undertaking Guideline makes use of ARCore to estimate the place and orientation of the person as they navigate the course. A segmentation mannequin, constructed on the DeepLabV3+ framework, processes every digicam body to generate a binary masks of the rule of thumb (see the earlier weblog submit for extra particulars). Factors on the segmented guideline are then projected from image-space coordinates onto a world-space floor aircraft utilizing the digicam pose and lens parameters (intrinsics) offered by ARCore. Since every body contributes a distinct view of the road, the world-space factors are aggregated over a number of frames to construct a digital mapping of the real-world guideline. The system performs piecewise curve approximation of the rule of thumb world-space coordinates to construct a spatio-temporally constant trajectory. This enables refinement of the estimated line because the person progresses alongside the trail.
![]() |
Undertaking Guideline builds a 2D map of the rule of thumb, aggregating detected factors in every body (pink) to construct a stateful illustration (blue) because the runner progresses alongside the trail. |
A management system dynamically selects a goal level on the road a long way forward primarily based on the person’s present place, velocity, and course. An audio suggestions sign is then given to the person to regulate their heading to coincide with the upcoming line phase. By utilizing the runner’s velocity vector as an alternative of digicam orientation to compute the navigation sign, we eradicate noise attributable to irregular digicam actions frequent throughout operating. We will even navigate the person again to the road whereas it’s out of digicam view, for instance if the person overshot a flip. That is doable as a result of ARCore continues to trace the pose of the digicam, which may be in comparison with the stateful line map inferred from earlier digicam photographs.
Undertaking Guideline additionally consists of impediment detection and avoidance options. An ML mannequin is used to estimate depth from single photographs. To coach this monocular depth mannequin, we used SANPO, a big dataset of out of doors imagery from city, park, and suburban environments that was curated in-house. The mannequin is able to detecting the depth of varied obstacles, together with individuals, automobiles, posts, and extra. The depth maps are transformed into 3D level clouds, much like the road segmentation course of, and used to detect the presence of obstacles alongside the person’s path after which alert the person by way of an audio sign.
![]() |
Utilizing a monocular depth ML mannequin, Undertaking Guideline constructs a 3D level cloud of the atmosphere to detect and alert the person of potential obstacles alongside the trail. |
A low-latency audio system primarily based on the AAudio API was carried out to supply the navigational sounds and cues to the person. A number of sound packs can be found in Undertaking Guideline, together with a spatial sound implementation utilizing the Resonance Audio API. The sound packs have been developed by a staff of sound researchers and engineers at Google who designed and examined many various sound fashions. The sounds use a mixture of panning, pitch, and spatialization to information the person alongside the road. For instance, a person veering to the appropriate might hear a beeping sound within the left ear to point the road is to the left, with rising frequency for a bigger course correction. If the person veers additional, a high-pitched warning sound could also be heard to point the sting of the trail is approaching. As well as, a transparent “cease” audio cue is all the time obtainable within the occasion the person veers too removed from the road, an anomaly is detected, or the system fails to supply a navigational sign.
Undertaking Guideline has been constructed particularly for Google Pixel telephones with the Google Tensor chip. The Google Tensor chip allows the optimized ML fashions to run on-device with greater efficiency and decrease energy consumption. That is essential for offering real-time navigation directions to the person with minimal delay. On a Pixel 8 there’s a 28x latency enchancment when operating the depth mannequin on the Tensor Processing Unit (TPU) as an alternative of CPU, and 9x enchancment in comparison with GPU.
![]() |
Testing and simulation
Undertaking Guideline features a simulator that permits fast testing and prototyping of the system in a digital atmosphere. The whole lot from the ML fashions to the audio suggestions system runs natively inside the simulator, giving the complete Undertaking Guideline expertise with no need all of the {hardware} and bodily atmosphere arrange.
![]() |
Screenshot of Undertaking Guideline simulator. |
Future course
To launch the expertise ahead, WearWorks has develop into an early adopter and teamed up with Undertaking Guideline to combine their patented haptic navigation expertise, using haptic suggestions along with sound to information runners. WearWorks has been growing haptics for over 8 years, and beforehand empowered the primary blind marathon runner to finish the NYC Marathon with out sighted help. We hope that integrations like these will result in new improvements and make the world a extra accessible place.
The Undertaking Guideline staff can be working in direction of eradicating the painted line utterly, utilizing the most recent developments in cell ML expertise, such because the ARCore Scene Semantics API, which might establish sidewalks, buildings, and different objects in out of doors scenes. We invite the accessibility neighborhood to construct upon and enhance this expertise whereas exploring new use circumstances in different fields.
Acknowledgements
Many individuals have been concerned within the growth of Undertaking Guideline and the applied sciences behind it. We’d wish to thank Undertaking Guideline staff members: Dror Avalon, Phil Bayer, Ryan Burke, Lori Dooley, Track Chun Fan, Matt Corridor, Amélie Jean-aimée, Dave Hawkey, Amit Pitaru, Alvin Shi, Mikhail Sirotenko, Sagar Waghmare, John Watkinson, Kimberly Wilber, Matthew Willson, Xuan Yang, Mark Zarich, Steven Clark, Jim Coursey, Josh Ellis, Tom Hoddes, Dick Lyon, Chris Mitchell, Satoru Arao, Yoojin Chung, Joe Fry, Kazuto Furuichi, Ikumi Kobayashi, Kathy Maruyama, Minh Nguyen, Alto Okamura, Yosuke Suzuki, and Bryan Tanaka. Because of ARCore contributors: Ryan DuToit, Abhishek Kar, and Eric Turner. Because of Alec Go, Jing Li, Liviu Panait, Stefano Pellegrini, Abdullah Rashwan, Lu Wang, Qifei Wang, and Fan Yang for offering ML platform help. We’d additionally wish to thank Hartwig Adam, Tomas Izo, Rahul Sukthankar, Blaise Aguera y Arcas, and Huisheng Wang for his or her management help. Particular because of our companions Guiding Eyes for the Blind and Achilles Worldwide.