The security issues of the core (operating systems) of the Internet of Things (IoT) are becoming increasingly urgent and prominent, this article conducts a systematic research and summary of the security of the current mainstream IoT operating system. Firstly, based on the architecture and applications functions of IoT devices, this article introduces the concept of operating system security, analyzes and studies the security vulnerabilities, key technologies, and attack and defense security mechanisms of operating systems. Secondly, this article investigates the application scenario used by IoT operating systems, such as smart homes, smart healthcare, smart industries, blockchain, and the Internet of Vehicles. Next, from the perspective of building a complete security system, this article investigates the security mechanisms, security frameworks, security kernels, platform integrity, and security testing of IoT operating systems. Finally, this article points out the security challenges and opportunities faced by IoT operating systems, summarizes the current research status, and puts forward corresponding suggestions.