Working memory capacity places a significant constraint on what visual information is selected and stored. There is evidence to suggest that working memory accumulates scene information over successive fixations. However, eye movements during ordinary behavior suggests that visual information is often acquired “just-in-time” for a task. When do observers choose to use working memory over eye movements? Subjects performed a sorting task in a virtual environment with haptic feedback. Subjects picked up bricks based on one feature, and sorted the bricks by a different feature. In two different trial blocks, the feature relevant for sorting was either Predictable or Unpredictable. In the Predictable condition, subjects infrequently re-fixated the brick (29%). However, in the Unpredictable condition, subjects re-fixated the brick more often (47%). Infrequent re-fixations in the Predictable condition suggest that subjects used working memory for the sorting decision, whereas frequent re-fixations to the brick in the Unpredictable condition suggest “just-in-time” acquisition. Thus, when the relevance of a brick feature is certain, subjects use working memory; when the relevance is unknown, subjects adopt a strategy of acquiring the feature only after its relevance becomes certain. To further probe working memory use, a change was made to one of the brick features on 10% of trials. When subjects missed the change in the Predictable condition, the brick was most often sorted by the pre-change feature (76%), implicating use of working memory. However, in the Unpredictable condition, the changed brick was most often sorted by the new feature (71%), suggesting “just-in-time” acquisition. Thus, selection and storage of visual information in naturalistic tasks depends on both task relevance and its predictability. Subjects dynamically trade off memory use and eye movements depending on their expectations of what information is needed for the task.