Here if we zero out vin1 we will be in a situation where mos will be off, right?
As others have mentioned, Razavi is finding the Thevenin equivalent of the circuit using the small signal (ac) model of the transistor (not explicitly drawn in the text). This means we've already assumed the transistor is operating in the saturation region, and all bias voltage and current sources are zero. Thus, \$V_{in1}\$ represents small ac variations, and zeroing it will not put the transistor in the cutoff region.
I dont understand how thevenin voltage(Vt) and Rt in the equivalent circuit is Vin1 and 1/gm1?
Thevenin Impedance
To find the Thevenin impedance we start by drawing the small signal model of the isolated network in the dotted box [Fig. 4.16(a)] (assuming \$\lambda=\gamma=0\$). Notice that we've removed all independent sources (just \$V_{in1}\$ in this case) and inserted a test voltage source \$v_{test}\$ at the load:
From the equivalent model, we can write \$v_{gs}=-v_{test}\$ and \$g_mv_{gs}=-g_mv_{test}\$. Next, we observe that \$i_{test}=-(-g_mv_{test})\$. Thus, \$R_t=\frac{v_{test}}{i_{test}}=\frac{1}{g_m}\$. Note that this resistance does not depend on \$R_d;\$ as Sven B mentioned, if you look into the source of a transistor you'll find a conductance of approximately \$g_m\$.
Thevenin Voltage
To find the thevenin voltage we've drawn the same circuit using the transistor small signal model, except we've included the independent sources (just \$V_{in1}\$) and disconnected the load:
Because the load has been disconnected, no current can flow through the dependent current source, so \$g_mv_{gs}\$ must be equal to zero. In order for this to be true, either \$g_m\$ or \$v_{gs}\$ must be zero. Because we've assumed the transistor is operating in the saturation region, \$g_m\$ cannot be zero; thus \$v_{gs}=0\$, and \$V_{t}=V_{in1}\$. Sven B pointed out that you can also view this as a source follower, which gives a gain of 1, and \$V_{t}=V_{in1}\$.
How can thevenin theorem be appliied to mosfets??
As long as you are modeling transistors with linear elements (e.g. small signal model), you can apply Thevenin's Theorem to any network of transistors.