How do observers track multiple moving objects simultaneously? Previous work has shown that adding conflicting texture motion to the tracked objects impairs tracking performance. Here, we test whether texture motion is integrated with object motion in an object-based manner, or whether adding conflicting texture motion to a display causes global interference effects. We added a moving texture onto the surface of tracked objects with the texture moving either in the same or opposite direction to the objects. In the critical trials, we presented both types of texture motion. In these trials, we found a selective impairment for the objects with opposite texture motion, suggesting that multiple motion information sources are integrated in an object-based manner during tracking. The integrated motion signals might be used to anticipate prospective object locations in order to enhance tracking.