Optical coherence tomography (OCT) and optical coherence microscopy (OCM) have demonstrated the ability to investigate cyto- and myelo-architecture in the brain. Polarization-sensitive OCT provides sensitivity to additional contrast mechanisms, specifically the birefringence of myelination and, therefore, is advantageous for investigating white matter fiber tracts. In this Letter, we developed a polarization-sensitive optical coherence microscope (PS-OCM) with a 3.5 μm axial and 1.3 μm transverse resolution to investigate fiber organization and orientation at a finer scale than previously demonstrated with PS-OCT. In a reconstructed mouse brain section, we showed that at the focal depths of 20-70 μm, the PS-OCM reliably identifies the neuronal fibers and quantifies the in-plane orientation.