In this paper, transaction costs as small nonlinear price impact are introduced into American option pricing under the Heston stochastic volatility model, forming a gap between option prices of the holder and writer. Through the use of a dynamic hedging strategy together with a known option independent of transaction costs, we derive two different nonlinear pricing partial differential equation systems for the holder and writer, respectively. A numerical algorithm is designed to solve the systems so that American option prices as well as the optimal exercise boundary can be simultaneously obtained. Examples are presented to illustrate the effect of transaction costs on both option prices and optimal exercise prices. • We consider American option pricing under the Heston stochastic volatility model. • We introduce transaction costs as small nonlinear price impact. • We derive two different nonlinear pricing partial differential equation systems. • We show that transaction costs have a significant impact on option prices.