When a stimulus repeatedly and rapidly changes color (e.g., between red and green) and motion direction (e.g., upwards and downwards) with the same frequency, it was found that observers were most likely to pair colors and motion directions when the direction changes lead the color changes by approximately 80 ms. This is the color–motion asynchrony illusion. According to the differential processing time model, the illusion is explained because the neural activity leading to the perceptual experience of motion requires more time than that of color. Alternatively, the time marker model attributes the misbinding to a failure in matching different sorts of changes at rapid alternations. Here, running counter to the time marker model, we demonstrate that the illusion can arise with a single direction change. Using this simplified version of the illusion we also show that, although some form of visual masking takes place between colors, the measured asynchrony genuinely reflects processing time differences.