Multi-task optimization (MTO) aims to solve multiple tasks simultaneously. However, multi-task evolutionary algorithms (MTEAs) hardly consider the problem with constraints, while most optimization problems, in reality, are with constraints. This study presents a benchmark of constrained multi-task optimization problems (CMTOPs), modified from the CEC2017 competition on evolutionary multi-task optimization and CEC2017 competition on constrained real-parameter optimization. Moreover, this study attempts to solve CMTOPs by incorporating constraint handling techniques into MTEAs. Experimental results demonstrate the complexity of the proposed benchmark in the context of CMTOPs and present the requirements for handling constraints in MTEAs. Our preliminary exploration also reveals prospects for the development of evolutionary algorithms in the area of constrained multitask optimization. The Matlab source code can be obtained from https://github.com/intLyc/CMTO-Benchmark.