Although face perception is commonly characterized as holistic, as opposed to part-based, we have recently shown that both face parts and wholes are represented in “face-selective” cortical regions, with greater adaptation of holistic representations for familiar faces (A. Harris & G. K. Aguirre, 2008). Here we investigate the time course of these holistic and part-based face processing effects using magnetoencephalography (MEG). We examined “face-selective” components at early (∼170–200 ms) and later (∼250–450 ms) latencies in occipitotemporal sensors. While both “M170” and “M400” components showed significantly larger responses for familiar versus unfamiliar faces, neither exhibited a main effect of holistic versus part-based processing. These data affirm the existence of part-based “face-selective” representations, and additionally demonstrate that such representations are present from relatively early stages of face processing. However, only the later M400 component showed a modulatory effect of familiarity similar to that previously seen with fMRI, with a larger response to familiar faces in the holistic condition. Likewise, behavioral recognition was significantly correlated with the M400, not the M170, and only in the holistic condition. Together, these data suggest that, while face parts are represented from the earliest stages of face perception, modulatory effects of familiarity occur later in the face processing stream.