In five experiments, we examine the neural correlates of the interaction between upright faces, inverted faces, and visual noise. In Experiment 1, we examine a component termed the N170 for upright and inverted faces presented with and without noise. Results show a smaller amplitude for inverted faces than upright faces when presented in noise, whereas the reverse is true without noise. In Experiment 2, we show that the amplitude reversal is robust for full faces but not eyes alone across all noise levels. In Experiment 3, we vary contrast to see if this reversal is a result of degrading a face. We observe no reversal effects. Thus, across conditions, adding noise to full faces is a sufficient condition for the N170 reversal. In Experiment 4, we delay the onsets of the faces presented in noise. We replicate the smaller N170 for inverted faces at no delay but observe partial recovery of the N170 for inverted faces at longer delays in static noise. Experiment 5 demonstrates the interaction in low contrast at a behavioral level. We propose a model in which noise interacts with the processing properties of inverted faces more so than upright faces.