From query to prompt : towards open-world perception

The majority of contemporary perception models leverage Transformer-based architectures, such as DETR for object detection and Mask2Former for image segmentation. Central to these frameworks is the concept of extracting objects from image features through the formulation of queries, underscoring the significance of query design. In this dissertation, we explore integrating locality priors into the global attention mechanism via innovative query designs in DN-DETR and DINO. These designs include: 1. conceptualizing queries as anchor boxes; 2. predicting relative object locations across each decoder layer; 3. an auxiliary denoising task that refines queries to be close to object bounding boxes; and 4. strategically initializing queries coupled with a selection process. These advancements...[ Read more ]