Data collection, wrangling, and pre-processing for AI assurance

Data collection, wrangling, and pre-processing are critical steps within any AI/ML model development lifecycle. These steps precede every model building activity culminating in feature engineering for model formation. This chapter emphasizes the design, development, and implementation of raw data transformation into features in support of AI/ML model development. Integration and preparation of data sets from various sources, such as files, databases, big data storage, sensors or social networks is a key task when you want to build an appropriate analytic model using machine learning or deep learning techniques. Critical to the model building endeavor is the need to have high-quality data that, unfortunately, has shown to take up 50 to 80 percent of the time for an AI/ML development project. This chapter presents ways to reduce this processing time using data-driven operations "DataOps" (i.e., DevOps for data processing and workflows) pipelines for AI/ML.