The emotional content of visual images can be parameterized along two dimensions: valence (pleasantness) and arousal (intensity of emotion). In this study we ask how these distinct emotional dimensions affect the short-term memory of human observers viewing a rapid stream of images and trying to remember their content. We show that valence and arousal modulate short-term memory as independent factors. Arousal influences dramatically the average speed of data accumulation in memory: Higher arousal results in faster accumulation. Valence has a more interesting effect: While a picture is being viewed, information from positive and neutral scenes accumulates in memory at a constant rate, whereas information from negative scenes is encoded slowly at first, then increasingly faster. We provide evidence showing that neither differences in low-level image properties nor differences in the ability to apprehend the meaning of images at short exposures can account for the observed results, and propose that the effects are specific to the short-term memory mechanism. We interpret this pattern of results to mean that information accumulation in short-term memory is a controlled process, whose gain is modulated by valence and arousal acting as endogenous attentional cues.