Nocardia seriolae is a gram-positive bacterium that causes nocardiosis, threatening fish farming. Advanced nocardiosis is challenging to control; thus, accurate detection methods of the causal agent in the early disease stage are required. In this study, we developed a TaqMan fluorescence quantitative PCR (qPCR) assay for quantitative detection of N. seriolae in fish tissues and water samples. A pair of highly specific primers and a TaqMan probe were designed based on the N. seriolae 16S23S rRNA internal transcribed spacer (ITS) region. A high correlation coefficient (R2 = 0.998) of the standard curve with a 99.5% efficiency was obtained. The qPCR detection limit of the method was as low as 19.8 copies/μL, 1000 times more sensitive than conventional PCR, and has a good performance in the detection of cultured bacteria (y = −3.750× + 48.075, R2 = 0.974). Even 1.42 CFU/mL N. seriolae collected from 500 mL of natural pond water can be detected. Furthermore, a linear model for the relationship between the log of bacteria load and Cq values in water was established (y = −3.239× + 40.978), and the R2 value was 0.979. This assay was used for accurate N. seriolae detection in fish tissues, water samples, feeds and soils. This study provides a valuable tool for the early detection and control of nocardiosis in aquaculture.