An optical fiber in-line Mach-Zehnder interferometer based on down-up tapers is proposed and demonstrated for refractive index measurement. The device is fabricated by splicing two segments of multimode fibers in a heating and pushing process to form an up taper and followed by heating and pulling of the fiber adjacent to the up taper area to form a down taper. The down-up fiber taper structure constitutes a Mach-Zehnder interferometer and the whole device length is only 3 mm. The experimental results show that the refractive index sensitivity is ∼109.71 nm/RIU (refractive index unit) in the refractive index range of 1.340 – 1.370. Such a device is compact in size, robust in structure, good in sensitivity, convenient in operation and low in cost, and has high potential for chemical and biomedical sensing applications.