Since 1915, when affordable, engine-driven tractors started becoming widely available to farmers in the US, a newfound efficiency in growing and harvesting crops quickly revolutionized agriculture.
Engine-driven equipment meant that one person could do the work of four, dramatically raising the bar on agricultural productivity, boosting crop harvests and food supplies.
As transformative as those improvements were, more changes are on the horizon, fueled by the advances in still-nascent AI technologies that are already the subject of experimentation in farming.
But those coming AI advances in agriculture may not arrive in the ways you think they will, says Greg Diamos, the engineering tech lead at industrial AI vendor Landing AI.
While various stages of autonomous cars and trucks seem destined to prevail in the future on public roadways, autonomous capabilities may not have the same role for farm tractors and other agricultural equipment.
Instead, AI could be better used by farmers to improve equipment efficiency, rather than by designing systems that actually drive tractors and other machines for farmers, says Diamos.
“Farming is a trillion-dollar industry,” he says. “It’s about feeding the world population with a variety of types of quality foods. It’s not about full autonomy. Moving the needle in agriculture means producing more high quality food at lower cost. It does not mean that farmers are able to take their hands off the wheel of their tractor.”
After studying the topic with his team at Landing AI, Diamos heavily researched farming and what farmers need to drastically improve their work. What they came up with was eye-opening – farmers don’t need self-steering and driving tractors. They need tractors and other equipment with autopilot capabilities which improve their operations and raise the efficiency of every task they have to perform, he says.
During a presentation titled “Tractor Autopilot — A New AI Application: Tackling Big Data, Training and Inference Compute to Use AI in an Agriculture Use Case” at the recent virtual AI Hardware Summit, Diamos laid out how he and his team went from concept to production for a tractor autopilot system in one year using deep learning.
Diamos, who holds a doctorate in electrical and electronics engineering from the Georgia Institute of Technology and who joined Landing AI in June 2019, says the project began when a customer asked for a daunting goal: autonomous driving capabilities for their fleet of agricultural machines.
The problem, though, is that he’d already realized that supervised deep learning, which is the technology powering a lot of progress in AI today, isn’t yet ready for production, he says. Case in point: a friend who is leading one of the major autonomous driving groups has watched repeatedly as the project roadmap for those efforts have slipped over and over and over again, extending what was meant to be a six-month effort into one that is still unfinished.
The request from Landing AI’s customer was evidence of just how difficult the efforts that would be needed, he said. The customer wanted it all – a Level Five process for complete deep reinforcement learning and fully-autonomous driving for a fleet of farm implements. The system, according to the customer request, was to do 100% of the driving, have perfect accuracy, address safety issues, have low hardware and data costs and work flawlessly.
“I almost quit on the spot,” says Diamos. “But I stepped back and reflected that even today, most people do not understand what AI can and cannot do. We looked deeper and found something completely different. Looking back, I think that in addition to being technically feasible, what we found is even more valuable than this.”
Tractor Autopilot is Born
And that’s where the tractor autopilot idea became the new goal for the project, according to Diamos.
It all began making sense when the realization was reached that AI in agriculture should focus on productivity, not autonomy, he says. “Autonomous driving is not the right solution. There’s something else.”
Diamos and his team then began working with Landing AI’s existing industrial AI perception control platform, building a computing stack that helped maximize productivity, especially when it came to perception and control of a tractor and its tasks.
The team began looking at how AI could deliver improvements and at how machines could “see” like people and even clone human actions. The researchers began looking at ways to help machines learn from experience they gained in farm fields.
“Viewed from that perspective, the answer is clear,” says Diamos. “We should automate many tasks that increase the productivity of farmers and help the beginners perform like the pros. This would enable single humans assisted by a fleet of machines to work faster, perform tasks with higher quality, and it would allow the machines to work in parallel alongside the farmer boosting their output.”
What the team simultaneously discovered was that they didn’t need to solve the complete problem to build a valuable product, he added. “Any improvement in productivity or quality we saw as a competitive advantage. We can ship products immediately when we have one skill or one task that is automated 90% of the time. General-purpose machines that can learn in the field provide a path towards shipping fast and delivering increased value over time. Old machines can learn new skills with software upgrades.”
For farmers, these gains have concrete results, he says, including valuable AI capabilities that allow the automatic adjustment of a cutting arm on a harvester as it is being operated using a video camera and data packets stored in label books.
“This allows the operator to increase the speed of operation without making mistakes,” says Diamos. “It can drive faster and you work faster. It allows operators with less training to perform this nuanced skill correctly.”
And the actions can be replicated, he says. “We can match the expert’s driver moves for over 95% of [crop] cuts. And we can also incorporate new learnings from machines that are deployed in the field into the entire fleet in about two days. Our general purpose autopilot platform allows us to pick up new skills like this in a matter of weeks. There’s an incremental path to full autonomy. Every step along the way adds value.”
The Tech Behind the Tractor Autopilot
Landing AI’s tractor autopilot is built using a three-layer software stack.
The bottom layer is a general-purpose platform for creating AI applications for industrial settings like manufacturing.
Next, a general-purpose autopilot is added atop the platform. This layer adds capabilities to rapidly retrofit machines of various types, from cars to tractors, harvesters, forklifts and much more. This layer allows the rapid learning needed for control and perception skills, says Diamos.
The last layer provides a canvas for developers to add specific skills that are tailored for specific machines, such as agricultural tractors in the field.
“Our focus is on optimizing the entire path from new data coming in to bring up new skills and quality improvement of deployed skills,” says Diamos. “There are multiple components in here related to data engineering, model development, deployment monitoring. You want to make the entire process fast, scale out to large fleets, and be low cost.”
As the research continued, Diamos and his team also realized that a major constraint of traditional AI thinking is that it requires massive amounts of costly and hard-to-use data to analyze and process. But with farming being a seasonal operation and one that has different requirements in different regions, they sought alternatives that could make autopilot analysis simpler.
That’s when they chose to create rapid skill learning for the equipment using digital feedback books and small data technology for simplicity. This prevents machine learning engineers from having to face the formidable task of editing huge stores of data, which take a lot of time and is costly.
“Even with assistance from a crowdsourced labeling platform, it is incompatible with our goal of rapid learning,” says Diamos of using large data sets. “The digital feedback book is a short, human-readable definition of a skill. A machine learning engineer or domain expert can edit it in a few minutes, and the resulting changes will flow downstream into data sets and models.”
And the small data technology is useful due to the seasonal nature of farming, which limits data collection to certain times of the year, he says. “Crops don’t grow all the time, so we had to wait months for the harvest season to start collecting data in our target market. Small data technology allows us to turn a few training examples or expert knowledge into a large data set. Small data technology enabled us to bootstrap new skills before we collected big datasets.”
How Tractor Autopilot Works
The tractor autopilot system includes three main subsystems. There is a vision or perception system, which converts videos coming from cameras attached to the machine into relevant features such as the position and orientation of the plants ahead of the harvest. The perception system is built using supervised deep learning.
Next is a human action coding system, which determines the next high-level action for the vehicle to perform, such as whether a cutting arm should be raised or lowered and by how much. This is also built using supervised deep learning.
The other subsystem is a feedback control system, which translates high-level vehicle actions into the hydraulic motor commands that are sent to the motors and actuate the machine. This is built using classical control technology.
The autopilot learns new skills in several ways, says Diamos. The first is when a skill is brought up for the first time. Skills are also learned when the system encounters new environmental conditions that weren’t seen in earlier internal datasets.
To add or update skills to be displayed by an autopilot-managed machine, the system looks at data engineering and data collection, it makes label book updates, then it goes through re-trained models, says Diamos. The new instructions are then pushed out for deployment, optimized for inference and monitored in the field to ensure that if there are performance regressions, they can be caught and fixed quickly.
“We’ve gotten this down so that we can do about two deployments per week,” he says.
The Hardware Behind Tractor Autopilot
Behind the system, a mix of AI chips and control chips or programmable controllers are used to run it all, says Diamos. The AI chips run the perception models and the deep neural networks behind them. They’re programmed in Python and they use frameworks like TensorFlow. They are then mapped down onto inference-optimized cores like GPU cores or specialized inference accelerators. Control chips, or programmable controllers, run the system’s controllers and have support for a wide range of environmental conditions. The controllers, which are programmed in C and assembly language, assume real-time operation and have tight integration and flexible interfaces to sensors. They also have high level programming stacks for control algorithms using concepts such as ladder logic.
One improvement that will likely drive maturity of the autopilots in the future are better control chips, says Diamos. “Deep learning, when applied to control algorithms, will make them much more computationally intensive.”
Another clear opportunity for improvements will be when AI chips and control chips can be unified, he says. “There’s no reason we should be using completely different hardware stacks, completely different software stacks, to run essentially the same supervised deep learning technology.”
The Future of AI in Farming
AI in agriculture has the potential to increase the productivity of farmers and help them in ways they may not be able to imagine today, says Diamos. “You don’t need full autonomy to deliver enormous value. We took a look under the hood of the LandingAI platform, how it enables us to take AI solutions to productions fast. I strongly believe that as AI transforms every major industry will also transform computing.”
For farmers, those gains will center around the work tasks they do today, he says.
“So, plowing a field, or harvesting a field does involve some driving, but you also have to harvest the crops correctly, or you have to make sure that the plow is at the right depth and orientation,” says Diamos. “That was very surprising when I looked at this, how many different tasks there are in farming across crops, environments, and different types of work. The focus isn’t completely on driving, it’s on much more than that.”
In the end, farmers don’t really need to fully automate everything with AI, he added. “The farmer is still there. To do the work, as long as the work gets done, you know, if it’s 95% by the machine, and 5% by the farmer, that’s a great productivity boost.”