Siamese neural networks have been used previously for speckle tracking in ultrasound images. Apart from the lack of adaptivity to the specific video sequence, the main problem which Siamese Tracker (ST) suffers from, is the absence to account for any object motion. The object is assumed stationary for the search operation in the next frame. We propose to improve the tracking algorithm by adopting Lucas Kanade method for finding optic flow to account for the background motion. We further use a more advanced Siamese network, Correlation Filter Network (CFNet) which uses a Correlation Filter Layer (CFL) for learning a robust representation of the tracked object. We show that by using our methodology, we are able to localize the carotid artery better as compared to the baseline Siamese networks.