Since you asked an identical question on physics.SE, I'll give you an identical answer:
because the holes somehow will attract electrons and get them from conduction band to valence band
The reason that a p-type semiconductor is p-type is that it contains acceptor impurities. These are atoms that tend to capture electrons in localized states around their nucleus. For example, group III boron is a typical acceptor impurity in silicon.
Because the captured electrons are in localized states, they aren't free to contribute to conduction.
But, consider if we start with intrinsic material and start to increase the density of acceptor impurities. In intrinsic material, the conduction band has very low occupancy, so the electrons can't be captured from there. Instead, they're captured from the valence band, leaving holes behind.
And indeed, these holes do attract electrons from the conduction band, but to make p-type material you typically add many more (orders of magnitude more) impurities than the intrinsic carrier density, so there simply aren't enough conduction band electrons to fill the acceptor states or to fill the holes resulting from the acceptors attracting valence electrons.
Another way to look at this is to look at Fermi level. As the acceptors capture electrons and create holes, then we know the occupancy of the valence states has decreased. Since the valence state occupancy is reduced we realize the Fermi level must be closer to the valence band edge than in the intrinsic material. And since the Fermi level is closer to the valence band edge, it must be farther from the conduction band edge, resulting in even fewer conduction band electrons than in intrinsic material. In (quasi-)equilibrium, we find a Fermi level that gives a statistical balance between density of valence band holes, occupancy of acceptor states, and density of conduction band electrons. And this gives tells us in what ratio holes and electrons are available to carry current.
Short version: The holes can try to attract electrons from the conduction band all they want, there simply aren't enough electrons there in p-type material to fill all the holes.