Drones and optical sensors: No longer a remote possibility

To inspect a bridge, technicians must close one lane at a time so they can scrutinize the deck, structure, and pavement for potholes, cracks, delamination, corrosion, and other problems.

That is the easy part. On smaller bridges, they must use ropes and ladders to get underneath the deck. On larger structures, they call in a snooper—a truck with a long, multijointed arm with a person-sized bucket at the end. The arm drops the bucket over the side, moving under the deck and up, down, and across the deck and supports, often suspending the inspector 100 feet or more in the air.

"I've done it a few times," says Marc Maguire, an assistant professor of structural engineering at University of Nebraska. "I'm not afraid of heights, but it's not fun. You're in harm's way. By the time you put on your harness and get into the truck, you could have sent a drone up there."

That is exactly what many inspectors have begun to do. Coupled with remote optical sensors, drones have become an attractive option for examining bridge, roadway, and railroad infrastructure.

Interest in drones is growing rapidly. When the American Association of State Highway and Transportation Officials (AASHTO) surveyed state departments of transportation in 2016, none used drones regularly. Two years later, 20 states were flying daily missions and 15 were testing drones for applications ranging from inspection and surveying to natural disaster response.

Today, drones with optical cameras or lidar are often used to check infrastructure for damage, placement of traffic signs, and overhanging vegetation, and to assess structures after floods and rockslides. Other inspectors use drone-mounted thermal imaging to find delaminations in bridge decks. Others, still, are testing multispectral imaging to examine the condition of railroad tracks, beds, and railroad ties. And a new generation of optical camera arrays provides millimeter-scale details for true digital models that engineers can use to specify customized spare parts.

Though small drones found their first customers among hobbyists and photographers, improvements to stability, flight time, and battery life have extended their utility. And now, they're part of our infrastructure, too.

Economics

Over the past five years, drones have grown more affordable, easier to fly, and more capable. Armed with artificial intelligence and inertial measurement units, they can now remain stable in wind gusts and can fly precise and repeatable routes, even if they lose contact with a GPS signal.

Sensors have also improved, says Matt Dunlevy, CEO of SkySkopes, a company that uses drones to inspect infrastructure, oil and gas lines, and electric utilities. "Earlier sensors were more of an adaptation of current technologies," he explains. "Now camera sensors are mostly developed from scratch with the express intent of being utilized on drones. This has allowed for far better accuracy and quality levels of the deliverables for our clients."

Software has kept pace with advances in hardware. Algorithms routinely use GPS information to stitch photos together automatically to create models, quantifying such problems as the area of potholes on a road or the severity of cracks on railroad ties. By comparing the digital record of one inspection with another, software can spotlight how fast those problems are evolving so maintenance can address them before they become critical.

Regardless of their many assets, drones also have their downsides. They cannot fly in rain, snow, or winds above 15-20 mph. In lesser winds, they burn more battery power and have a harder time holding position for measurements. Nor can they reliably find finer features, like hairline-thin fatigue cracks that can grow and cause structural failures.

And, drones are not always faster than people. In an Idaho test, for example, Maguire found a snooper crew could inspect a bridge faster than a drone team because the drone ran out of battery every 15 to 20 minutes. While newer drones can stay aloft longer, they are still limited to about 30 minutes of flight time.

Drone-mounted sensors also have limits. Most imaging systems require sunlight, often in short supply under a bridge and on cloudy days. Many drone cameras are optimized for color sensitivity, rather than consistent, high-resolution imaging under any light conditions. Infrared thermal sensors, for example, work best in the morning or evening, when temperatures are rising or falling.

Another important consideration for users is cost. While optical and thermal cameras are easily affordable, lidar and multispectral imagers are expensive.

Yet drone inspections are already proving their economic worth. They can slash the cost of a typical highway bridge deck inspection from $4,600 and a full day to $1,200 and one hour, according to a 2019 AASHTO estimate. A 2018 Oregon State University study figured drones could trim $10,400 off the $73,800 cost of a full bridge inspection. Given these cost savings in labor, the hardware can quickly pay for itself.

LiDARUSA's drones house lidar units that create dense cloud images. Shown here is a railroad track running through an urban environment. Credit: LiDARUSA

In other good news, lower costs and faster inspections could allow more frequent testing. That would "allow Departments of Transportation to find and fix problems faster, says Jeff Fagerman, CEO of LiDARUSA, an Alabama-based lidar drone developer. "You want to catch a road when it starts to deteriorate and fix it within a year or two," he says. "If you wait until year 10, the cost is astronomically higher."

These economics are not just true for transportation. They also hold for power grids, petrochemical plants, refineries, pipelines, telecommunications, construction, and agriculture. These end uses have created a ready market for remote sensor innovation.

Light touch

While newer imaging technology is gaining traction, optical cameras still dominate transportation inspection. One common approach is to take high-quality images with a powerful digital single-lens reflex (DSLR) camera, stitch them together into a model, and inspect it on a monitor, says Barritt Lovelace, who wrote a series of reports on bridge inspections for the Minnesota Department of Transportation.

Those models have a lot of value, says Lovelace, director of unmanned aircraft systems, AI, and reality modeling for Collins Engineers.

"Now that we're starting to inspect bridges a second time and comparing new and old models, we're finding things," he says. "We can see if cracks are propagating, if there is more deck delamination and corrosion. If we set up survey points and adjust our model to those points, we can get to one-centimeter accuracy pretty easily. That lets us see things—perhaps buckling or other failure modes—that we wouldn't notice otherwise."

Visual Intelligence, a Houston-based optical systems developer, takes accuracy further, to the submillimeter level with a dual-camera optical system. It provides very precise models showing cracks, structural defects, rust, bending, and bulging. "You can count the threads on bolts to see if they are starting to loosen," says CEO Jay Tilley.

Visual Intelligence's dual-camera system achieves submillimeter resolution, which it uses to model infrastructure with enough detail to source replacement parts. Credit: Visual Intelligence

The company's software generates true engineering models with dimensional accuracy within a millimeter. It automatically isolates and identifies the model's structural elements, which can be used to order replacement parts in building information system databases.

The software also lets engineers accurately specify and prefabricate parts that have been modified during installation, or are simply too old to be found in catalogs. This can be done in a fraction of the time needed to measure those parts manually.

Instead of using a high-pixel-count camera, Visual Intelligence achieves high resolution by mounting two 60-megapixel cameras at an acute angle to one another and taking advantage of their parallax view to produce three-dimensional images. Their redundance, plus synchronization of the cameras within a few milliseconds of one another, provides the photogrammetric software with the data needed to achieve high accuracy.

The company originally developed a larger, heavier camera array system intended for use on aircraft. It took four years to miniaturize using full-sized, off-the-shelf DSLR cameras that could fly on a drone. Although lenses proved the heaviest part of the new system, they proved paramount for millimeter-scale imaging.

Imaging

In addition to the traditional cameras found on drones, lidar has become a popular inspection tool in recent years, thanks to its accuracy and versatility. Lidar produces detailed point-cloud images even under poor lighting, and shows both roadside vegetation and the contour of the ground underneath those plants. Not only does lidar show the shapes of highway signs, but because it measures the intensity of reflections off their surfaces, it can also "read" what is written on them as well as the lane stripes on a highway. Engineers often mount lidar with other sensors and fuse their data to create more precise models.

According to Fagerman, lidar was used for road inspection even before drones. Ten years ago, large, expensive, van-mounted lidar systems patrolled highways looking for failing pavement and overhanging branches. Then, in 2014, Faro and Velodyne introduced lightweight, relatively inexpensive lidar systems.

Fagerman was one of the first to adapt Velodyne's lidar to drones. He miniaturized the electronics, working through such problems as electronic crosstalk, electromagnetic interference, and shaky gimbals. He created an integrated system that used inertial measurement units to track drone location when GPS signals failed.

Fagerman explains that drones could see where van-mounted lidar could not. "If you scan a street from a van, you cannot look over slopes or fences to see ditches or private roads or other things in people's backyards," he says. "But if you send a drone up 400 feet, you are going to see the topography, the buildings, and any other details."

In addition to the traditional cameras found on drones, lidar has become a popular inspection tool in recent years, thanks to its accuracy and versatility. Lidar produces detailed point-cloud images even under poor lighting, and shows both roadside vegetation and the contour of the ground underneath those plants. Not only does lidar show the shapes of highway signs, but because it measures the intensity of reflections off their surfaces, it can also "read" what is written on them as well as the lane stripes on a highway. Engineers often mount lidar with other sensors and fuse their data to create more precise models.

According to Fagerman, lidar was used for road inspection even before drones. Ten years ago, large, expensive, van-mounted lidar systems patrolled highways looking for failing pavement and overhanging branches. Then, in 2014, Faro and Velodyne introduced lightweight, relatively inexpensive lidar systems.

Fagerman was one of the first to adapt Velodyne's lidar to drones. He miniaturized the electronics, working through such problems as electronic crosstalk, electromagnetic interference, and shaky gimbals. He created an integrated system that used inertial measurement units to track drone location when GPS signals failed.

Fagerman explains that drones could see where van-mounted lidar could not. "If you scan a street from a van, you cannot look over slopes or fences to see ditches or private roads or other things in people's backyards," he says. "But if you send a drone up 400 feet, you are going to see the topography, the buildings, and any other details."

Future

Other inspection technologies, such as hyperspectral imaging, are also beginning to draw attention. Hyperspectral imaging breaks down each pixel it captures into spectral bands. The variations in the intensity of these bands act as fingerprints that identify specific materials and their condition.

Headwall Photonics is a leader in drone-sized multispectral imagers. Its palm-sized Nano-Hyperspec sensor collects 270 spectral bands by 640 spatial pixels within the visible to near infrared (400 nm to 1,000 nm) range. It divides its targets into a grid, then collects light through a slit, one column at a time. This "push-broom" method makes it easier to optimize the optical input with the speed of the drone. The system uses two mirrors and a grating, which breaks the light into its spectral components, to focus light onto its focal plane. Software then stitches the image together.

Headwall unmanned airborne hyperspectral imaging and lidar reveals areas of possible vegetation health issues and encroachment onto railroad tracks. Credit: Headwall Photonics

Headwall often fuses hyperspectral and lidar data for more accurate mapping. Recently, they began testing this type of system on a 400-meter section of railroad track operated by Deutsche Bahn. Before it could inspect anything, it had to identify the spectral fingerprints of different materials found in steel rails, concrete ties, gravel, dirt, brush, and trees along the track.

These fingerprints make it possible to identify the location and condition of these objects. Hyperspectral imaging can show corrosion in metals, and the mineral buildup caused when water seeps into cracks in concrete rail ties, says Ross Nakatsuji, Headwall's senior product manager for remote sensing. It can also assess plant growth, so managers can calculate when to trim trees and brush on the side of the tracks instead of paying crews to check periodically.

The ability to identify specific materials and their condition does not come cheap. The starting price for Headwall sensors is $20,000, and $70,000 for turnkey drone systems. "Our goal isn't to replace something that is working fine with really expensive technology," Nakatsuji says. "We want to provide data other technologies cannot offer."

As uses multiply, innovation will slash costs, add features, enhance portability, and develop products designed for specific types of inspection, says Mark Treiber, compact lidar product manager for Teledyne Optech. Although speaking specifically about lidar, his comment holds true for the many optical technologies available for transportation infrastructure inspection: In time, everything will become smarter and easier to use.

"We're moving away from early adopters who want to tinker with everything, and towards people for whom lidar is a tool," he says. "They want to see a contour plot, not a 3D point cloud. They don't want a lidar, but a tool that solves their problems."

But no matter how they evolve, optics is likely to remain at the heart of remote sensing for transportation infrastructure.

Alan S. Brown is a writer who covers the intersection of science, engineering, and technology. His work includes topics ranging from quantum computing and synthetic biology, to advanced manufacturing and robotics.