The viewpoint aftereffect is a perceptual illusion that, after adapting to an object/face viewed from one side (e.g., 30° to the left of center), when the same object/face is subsequently presented near the front view, the perceived viewing direction is biased in a direction opposite to that of the adapted viewpoint (e.g., 2° to the right). In this study, we measured the face viewpoint aftereffects when the adapting and the testing faces were different in identity and gender and when their vertical orientations were inverted. The aftereffect showed a strong transfer following adaptation to other faces. This effect was slightly attenuated when the adapting and the test face stimuli were made more dissimilar. This suggests the existence of neurons jointly tuned to both face view and structure. However, the transfer from cross adapting to an inverted face was much reduced and weak, indicating that the neural coding of upright and inverted faces in the high-level visual cortex is different and a major part of the face viewpoint coding occurs at the level where faces are holistically represented.