Lithostratigraphy Modeling with Transformer-Based Deep Learning and Natural Language Processing Techniques

Abstract In the Geosciences, sequence modeling algorithms such as long short-term memory - LSTM - have been widely used in various natural domains, like in earthquake and rainfall forecasting. A new architecture called Transformer has been used by the recent state-of-the-art models, capable of outperforming classical methods. Using this technology, this paper brings a modern approach that applies Transformers-based algorithms to solve a lithostratigraphy prediction challenge. We propose modifications to the original technique in order to embed geological information. In this sense, the lithological sequence is encoded as a sequence of integers and mapped by an embedding layer into a richer representation (a numerical vector per lithological element). We also incorporate the relative position of the lithological samples by adapting the original encoding. The database comprises four different wells located in the same onshore basin: two were used for training, one for validation and the last for test. They were filtered to guarantee that all wells’ data would have the same depth range. We ran 38 experiments with varying hyperparameters (number of transformer blocks, embedding size, learning rate and parallel attention heads). It was found that higher values of those variables indicate a higher model’s performance (in terms of F1-Score, Accuracy, Precision and Recall). The results achieved have accuracy and F1-Score concentrated between 0.89 and 0.92, showing consistency and good generalization capacity. Visually, we observe the model can approximate the lithostratigraphy within the geological wells’ context. Moreover, we developed 2 metrics in order to assess the model’s ability to detect lithological transitions, named ‘Transition Accuracy’ and ‘Expanded Transition Accuracy’. It was applied in 10 randomly selected intervals in the test data. Statistically, the ‘Expanded Transition Accuracy’ shows the model misses the transition in 0.3048 meters at most. Finally, we consider this paper presents a proof of concept in the use of transformer-based technologies for the modeling and prediction of lithostratigraphic sequences, a successful adaptation of NLP techniques to solve a geoscientific challenge.