While the large design degrees of freedom (DOFs) give metasurfaces a tremendous versatility, they make the inverse design challenging. Metasurface designers mostly rely on simple shapes and ordered placements, which restricts the achievable performance. We report a deep learning based inverse design flow that enables a fuller exploitation of the meta-atom shape. Using a polygonal shape encoding that covers a broad gamut of lithographically realizable resonators, we demonstrate the inverse design of color filters in an amorphous silicon material platform. The inverse-designed transmission-mode color filter metasurfaces are experimentally realized and exhibit enhancement in the color gamut.