The answers here are skipping one important feature: A hole is not a missing electron. A hole is a semiconductor atom which has extra protons in relation to its electrons.
A hole is a genuine positive charge. When a hole moves, a positive charge has adopted a new location.
Take the simple case of neutral hydrogen atoms. If we remove one electron, we're left with a bare proton: a positive charge. A proton is not a "missing electron." Positive charge is not a "lack of electrons," instead it's a genuine type of charge in it's own right.
The OP question would be valid if silicon was made of electrons only. Then, any "hole" would just be a gap in the population of electrons, and if the hole moves, it's really just electrons moving backwards. But silicon is made of cancelled-out charge: equal positive and negative charges. A "hole" is a gap in the electron population which exposes a hidden proton. A hole is a genuine positive charge.
With the movie-theater analogy, the complete picture would be to place red light bulbs on all the seats. These are the protons. Now let the audience of electrons sit on the seats and cover the lights. If there are any gaps in the audience, then in the dark theater we will see brilliant red lights at the location of "positive charges." These are the red lights which haven't been canceled out by an electron. If one person changes seats, then the location of the red light will move the other way.
So that's the answer: at the micro-level, yes, electrons are shifting in order to cover or reveal the positive charges. But from a distant view, there are no electron shifts. They are indetectable, unimportant. It's the changing position of the positive charges which dominate. All we see in the dark movie theater are some extremely powerful red lights which suddenly leap from one seat to the next.