Semantic Embedding Graph Convolutional Networks for Multi-label Video Segment Classification

Video classification is a challenging problem, video segment labels are sparse and expensive to get, and it is important to leverage as much available information as possible from labeled datasets. There have been several ways to capture video frame information but none of them have utilized the information hidden in labels correlation to increase classification accuracy. This work proposed a framework called Graph Convolution Semantic Network of aggregated descriptors (GCSN) which can extract neighboring information of related segment labels to increase video segmentation classification accuracy. Label relation graph was built by thresholding on cosine similarity computed from mutual embedding similarities, word embeddings were generated by Deep Bidirectional Transformers. The testing accuracy on Youtube-8m video segments classification dataset shows that our proposed GCSN outperforms NeXtVLAD baseline by considering additional labels relation information.