Multimodal AI to Improve Agriculture

Advances in natural language processing (NLP) and computer vision are now being applied to many agricultural problems. These techniques take advantage of nontraditional (or nonnumeric) data sources such as text in libraries and images from field operations. However, these techniques could be more powerful if combined with Artificial Intelligence (AI) and numeric sources of data in multimodal pipelines. We present several recent examples, where United States Department of Agriculture (USDA) Agricultural Research Service (ARS) researchers and collaborators are using AI methods with text and images to improve core scientific knowledge, the management of agricultural research, and agricultural practice. NLP enables automated indexing, clustering, and classification for agricultural research project management. We explore two case studies where combining techniques and data sources in new ways could accelerate progress in personalized nutrition and invasive pest detection. One challenge in applying these techniques is the difficulty in obtaining high-quality training data. Other challenges are a lack of machine learning (ML) techniques customized for use and ML skills or experience among researchers and other stakeholders. Initiatives are underway at USDA-ARS to address these challenges.