From your quote it sounds like you are learning about charge carrier diffusion in a P-N junction. When those electrons and holes diffuse they don't leave behind a mobile charge carrier of the opposite type. They do however leave behind an ionized dopant atom, which is a fixed charge of the opposite type.
When dealing with semiconductors we two different mobile charge carriers: electrons and holes, as well as two energy bands where they can exist: valence and conduction. When talking about electrons and holes in a semiconductor, we mean electrons in the conduction band, and holes in the valence band. When these carriers are in the other band, they don't contribute to conduction, so we ignore them.
Many people like to think of a hole as a missing electron, but I think that is a bad description of the system for a few reasons. First, it is important to treat electrons and holes separately as their own individual particles. You can't rewrite all of semiconductor physics to work with just one carrier type (you probably can, but it would be much more complicated) Second, as previously mentioned, electrons and holes contribute to current when they are in different energy bands. So, a hole in the valence band moving is very much not the same thing as an electron in the conduction band moving in the opposite direction. In fact, the Hall effect relies on this to be the case.
Am I understanding it correct that we just think of electrons leaving behind holes as "holes moving" or do I actually misinterperet the way this works?
You can generate an electron-hole pair by moving an electron from the valence band to the conduction band, which results in +1 electron in the conduction band, and also +1 hole in the valence band. But from your quote it sounds like you are talking about carrier diffusion. In which case, no, electrons and holes moving around within a single band do not leave behind a carrier of the other type. They might, however, leave behind an ionized dopant atom, which is a fixed charge.
You are going to have a much easier time with semiconductor physics if you think of holes as just another type of particle, similar to electrons but with a positive charge (and a different mass).
To address some concerns from the comments I have included here some quotes from Solid State Physics by Ashcroft and Mermin page 227-228. This discussion relates to where holes come. Once you determine your material's electronic band structure you may do the following:
It therefore suffices to examine how electrons respond to applied fields to learn how holes do. The motion of an electron is determined by the semiclassical equation:
\$\hbar \mathbf{\dot k}=(-e)\left(\mathbf E+\frac{1}{c}\mathbf{v}\times\mathbf H\right)\$ (12.22)
Whether or not the orbit of the electron resembles that of a free particle of negative charge depends on whether the acceleration , \$d\mathbf v/dt\$, is or is not parallel to \$\mathbf{\dot k}\$. Should the acceleration be opposite to \$\mathbf{\dot k}\$, then the electron would respond more like a positively charged free particle. As it happens, it is often the case that \$d\mathbf{v}/dt\$ is indeed directed opposite to \$\mathbf{\dot k}\$ when \$\mathbf{k}\$ is the wave vector of an unoccupied level
It goes on to explain why this happens which is outside the scope of this question, and ends with:
we find that as long as an electron's orbit is confined to levels close enough to the band maximum for the expansion (12.23) to be accurate, the (negatively charged) electron responds to driving fields as if it had a negative mass \$-m^*\$. By simply changing the sign of both sides, we can equally well (and far more intuitively) regard Eq (12.22) as describing the motion of a positively charged particle with a positive mass \$m^*\$.
So yes, the electron/hole model allows for you to look at the system as two different particles of the same charge, but one has a negative mass, or you can model the system as two particles with the same sign mass, but opposite sign charges.
The takeaway is still that electrons and holes are two different things and the "holes are just missing electrons" model is inaccurate and does not work (again, see the Hall effect).