Continual Learning for Remote Sensing Image Scene Classification With Prompt Learning

Overcoming catastrophic forgetting is a key difficulty for remote sensing image (RSI) classification in open world applications. The core of this problem lies in the ability of RSI scene classification models to adapt to the changing environment and maintain the learned knowledge while continually learning new knowledge. Mainstream replay-based approaches overcome catastrophic forgetting by reenacting and retracing past experiences in the process of learning new data. However, such approaches rely heavily on the storage of historical data, and the recent rise of new paradigms based on prompt learning offers a new perspective of using only task-related “instructions” (i.e., prompts) to guide the model’s continual learning and reasoning. Therein, the task knowledge encoded by the prompt improves the model’s ability to overcome forgetting while reducing the amount of data and model parameters required by traditional data-driven approaches. Therefore, we propose a continual learning method based on prompt learning for RSI classification. We systematically analyze and reveal the potential of prompt learning for continual learning of RSI classification. Experiments on three publicly available remote sensing datasets show that prompt learning significantly outperforms two comparable methods on 3, 6, and 9 tasks, with an average accuracy (ACC) improvement of approximately 43%. Performance improvements of 4% to 6% were achieved when compared to advanced prototype network methods. We found that prompt-generation strategies and prompt-related components significantly affect performance: (1) prompt-generation strategies are strongly correlated with the model’s performance in overcoming catastrophic forgetting; (2) prompt-related components are correlated with remote sensing images of different scales. The new paradigm of prompt learning potentially provides a new idea for the continual learning problem of RSI classification.