With vertical stimulus and horizontal response arrangement, performance is better with above-right/below-left mapping. The polarity correspondence hypothesis attributes this orthogonal stimulus-response compatibility (SRC) effect to the structural correspondence of positive (above, right) and negative (below, left) polarity in each dimension (Proctor & Cho, in press). Although many studies have shown that manipulation of response polarity affected the orthogonal SRC effect, there is no evidence for the effect of stimulus polarity change on orthogonal SRC effect. We tested whether the modulation of polarity in stimulus dimension affect the orthogonal SRC to test the polarity correspondence hypothesis. To manipulate the stimulus feature in vertical dimension while avoiding the responses from being affected directly by that manipulation, we used the orthogonal Simon paradigm in which the stimulus position is task irrelevant (Nishimura & Yokosawa, in press). Participants responded with right or left key-press to the color of the stimulus presented above or below the fixation. We manipulated the polarity in vertical dimension by cueing stimulus position with high predictability between blocks (Experiment 1) or within blocks (Experiment 2). In both experiments, significant orthogonal Simon effects were obtained when above position was cued, but the orthogonal Simon effects disappeared when below position was cued. Results indicate that the manipulation of positive and negative polarity in stimulus dimension, as well as in response dimension, affects the orthogonal SRC effect, supporting the polarity correspondence hypothesis. We discuss the findings in terms of the polarity coding based on multiple frames of reference.