Fish inevitably face numerous stressors in growth, processing, and circulation. In recent years, stress-related change in fish muscle quality has gradually become a research hotspot. Thus, the understanding of the mechanism regarding the change is constantly deepening. This review introduces the physiological regulation of fish under stress, with particular attention devoted to signal transduction, gene expression, and metabolism, and changes in the physiological characteristics of muscular cells. Then, the influences of various stressors on the nutrition, physical properties, and flavor of the fish muscle are sequentially described. This review emphasizes recent advances in the mechanisms underlying changes in muscle quality, which are believed to be involved mainly in physiological regulation under stress. In addition, studies are also introduced on improving muscle quality by mitigating fish stress.