Fine-Grained Lip Image Segmentation Using Fuzzy Logic and Graph Reasoning

Fine-grained lip image segmentation plays a critical role in downstream tasks, such as automatic lipreading, as it enables the accurate identification of inner mouth components, such as teeth and tongue, which are essential for comprehending spoken utterances. However, achieving accurate and robust lip image segmentation in natural scenes is still challenging due to significant variations in lighting condition, head pose, and background. This article proposes a novel deep neural network-based method for fine-grained lip image segmentation that exploits fuzzy and graph theories to handle these variations. A fuzzy learning module is designed to deal with the uncertainties in color and edge information and enhance feature maps at various scales. The fuzzy graph reasoning module with fuzzy projection models the relationship among semantics components and achieves a global receptive field. In our experiments, a fine-grained lip region segmentation dataset, i.e., FLRSeg, is built for evaluation, and experimental results have shown that the proposed method can achieve superior segmentation performance (94.36% in pixel accuracy and 74.89% in mIoU) compared with several state-of-the-art (SOTA) lip image segmentation methods.