Why can certain resistors not be the starting point when finding R Thevenin

I'm currently learning about Norton and Thevenin equivalents and it all makes sense for the most part except when it comes to finding RTh (the Thevenin equivalent resister in instances with no dependent sources. The best way to explain this I guess is to use the example from my book.

My question is related to figure (a) below and how they solved for RN or RTh. They said the 5 must be in parallel with the rest of the resistors in series. My question is why could it not be the 8 or 4 Ohm resistors in parallel with the other resistors in series? When solving so, they clearly give different results so I'm wondering why the 5 Ohm resistor must be the one in parallel with the others?

• Because you want the equivalent resistance as seen from the port on the right (indicated by the arrow marked "$R_N$"). – The Photon Apr 4 '17 at 0:28