I don't see any problems. You likely chose the resistor values incorrectly. Always show all details of your work. We can't tell you what you did wrong if you don't tell us what you did!

simulate this circuit – Schematic created using CircuitLab
To see this in action, run the simulation, highlight SW1 by clicking on it, then press Space to toggle it. Observe the current meter readings. The LEDs will not be visibly glowing much when the current is <20uA.
The only disadvantage of this circuit is that it wastes energy by shunting the LED, i.e. when D1 is on, it uses more current than when D2 is on.
I understand the reason that the current is flowing through the pnp transistor's B-E
That's not the reason. This current can not light up D1: it flows the wrong way. It also can't turn on Q1.
The only way what you claim would be true is if you had a circuit like below:

simulate this circuit
Here, you can clearly see that D1 gets driven by the B-E diode of Q2. But this doesn't happen in the circuit you've shown!
The circuit below uses a long-tailed pair and works quite well. The main disadvantage is that the input is not differential, and thus there's residual imbalance. About 0.5mA less flows through D2 than D1. There is a tradeoff between dissipation in R1,R4 and the balance: the lower the values of R1,R4, the better the balance of the LED currents. As it stands, the circuit would be acceptable for many uses as an indicator.

simulate this circuit