Purpose: We have found that inefficient visual searches remain inefficient even when observers search repeatedly the same scene. Surprisingly, observers mimic a visual search strategy when confronted with a realistic scene from which objects have been removed - as if compelled to search thorough the remembered “ghosts” of objects. What aspects of the visual scene cause observers to ”search the ghosts”? Method & Results: We developed a hybrid visual and memory search task termed panoramic search. Subjects view a changing portion of an extended scene as if they were turning their heads. At any moment, some objects are revealed and others are hidden. Subjects reported on the presence of an object, whether or not it was currently visible. The visual set size is the number of items currently visible. The memory set size is the number of items not currently visible. After 400 trials of training, search through the memory set is efficient (slope near zero). Search through the visual set is less efficient (10 msec/item). Then, objects were removed from the scene. In the Same Scene condition, the scene remains on the screen (without objects). Search through the invisible memory set was efficient. Search through the invisible visual set (the “ghost set”) was less efficient (10 msec/item). After 200 trials with no objects visible, this search became efficient. In the Different Scene condition, participants saw a new scene. It had the same spatial layout as the training scene, but a different gist (e.g. kitchen->office). The "ghost" effect vanished (efficient search for visible and memory sets). In the No Scene condition the entire scene was removed. The screen is blank. The memory search was efficient. Conclusion: Observers seem constrained to search a familiar scene for known objects for many trials after those objects have been removed. Changing or removing the scene eliminates this inefficiency.