In three experiments, we examined effects of target-distractor similarity in the remote distractor effect (RDE). Observers made saccades to peripheral targets that were either gray or green. Foveal or peripheral distractors were presented at the same time. The distractors could either share the target's defining property (congruent) or be different from the target (incongruent). Congruent distractors slowed down saccadic reaction times more than incongruent distractors. The increase of the RDE with target-distractor congruency depended on task demands. The more participants had to rely on the target property to locate the target, the larger the congruency effect. We conclude that the RDE can be modulated in a top-down manner. Alternative explanations such as persisting memory traces for the target property or differences in stimulus arrangement were considered but discarded. Our claim is in line with models of saccade generation which assume that the structures underlying the RDE (e.g. the superior colliculus) receive bottom-up as well as top-down information.