Please see update #1 below. My original answer (immediately following this paragraph) assumes that the voltage controlled voltage source in your schematic has a certain polarity. If you read the article Dependent Sources and Thevenin's Theorem, their convention for dependent current source polarity is somewhat counter-intuitive (for me, at least), but applying that convention to your circuit does yield a workable solution, whereas my initial assumption does not, as I describe now.
I spent a couple of hours trying to find the Thevenin equivalent of this circuit, before it dawned on me that this is not necessarily possible. So I attacked the whole circuit with nodal analysis.
simulate this circuit – Schematic created using CircuitLab
These are the equations I came up with (polarities are critical, so pay attention to the signs):
KVL applied to the 4 loops:
$$ 4 \times I_O + V_X + I_4R = 0V $$
$$ 5V + 4 \times I_O - 1\Omega \times I_1 = 0V $$
$$ 1\Omega \times I_1 + V_X - 4\Omega \times I_O = 0V $$
$$ 4\Omega \times I_O + 2\Omega \times I_3 - 10V = 0V $$
KCL applied to node B (remembering that \$V_O = I_4R\$):
$$ I_3 + I_4 - I_O - 2I_4R = 0A $$
There are five unknowns, \$V_X\$, \$I_O\$, \$I_1\$, \$I_3\$ and \$I_4\$, and five simultaneous equations. Solving for \$I_4\$, these whittle down to the following relationship between resistance R and the current through it, \$I_4\$:
$$ I_4 = -\frac{5}{4-5R} $$
Without any differentiation, or any use of the Maximum Power Transfer Theorem, it's quite clear that the denominator approaches zero as R approaches 800mΩ. That's where current is at an impossible maximum of ∞A, and where power will be at a maximum.
Are you sure you copied the schematic correctly? Maybe I didn't.
Here's a working CircuitLab model, where you can verify this discontinuity:
simulate this circuit
Update #1
Until now I assumed that the current source produced current to the left when \$V_O\$ is positive, which may not be the case. Here I perform a complete nodal analysis for the case where we reverse this current direction, but this is only one way to find the maximum power in R. A better way may be to find the Thevenin equivalent circuit between A and B, which I'll do after. The equations become:
KVL around the four loops:
$$ 4 \times I_O + V_X + I_4R = 0V $$
$$ 5V + 4 \times I_O - 1\Omega \times I_1 = 0V $$
$$ 1\Omega \times I_1 + V_X - 4\Omega \times I_O = 0V $$
$$ 4\Omega \times I_O + 2\Omega \times I_3 - 10V = 0V $$
KCL (again keeping in mind that \$V_O = I_4R\$):
$$ I_3 + I_4 - I_O \overbrace{+ 2I_4R}^{\text{sign change}} = 0A $$
This time, solving for \$I_4\$ reveals this relationship, which does not suffer the same discontinuity:
$$ I_4 = -\frac{5}{4+11R} $$
That means, as far as I can tell, that there does exist a Thevenin equivalent, which I'll derive in moment. For now, I want to find algebraically the value of R which will dissipate the most power, which requires the power law:
$$
\begin{aligned}
P &= I_4^2 R \\ \\
&= (\frac{-5}{4+11R})^2 R \\ \\
&= \frac{25R}{121R^2 + 88R + 16}
\end{aligned}
$$
I'll cheat with Wolfram Alpha to find the derivative, and set it equal to zero (to find maxima and minima):
$$
\frac{dP}{dR} = 0 = -\frac{25(11R-4)}{(11R+4)^3}
$$
Wolfram Alpha also provides the solution, which is:
$$ R = \frac{4}{11}\Omega = 363.6m\Omega $$
To find the Thevenin equivalent circuit, we need to know the open circuit voltage, which looks like this, and where we need to find \$V_O\ = V_A-V_B\$:
simulate this circuit
The KVL and KCL equations for this are the same, with a couple of changes:
- I'll replace \$I_4R\$ with \$V_O\$, the voltage between A and B which we wish to find, and
- \$I_4 = 0\$, since there's no resistor to draw current from \$V_O\$.
$$ 4 \times I_O + V_X + V_O = 0V $$
$$ 5V + 4 \times I_O - 1\Omega \times I_1 = 0V $$
$$ 1\Omega \times I_1 + V_X - 4\Omega \times I_O = 0V $$
$$ 4\Omega \times I_O + 2\Omega \times I_3 - 10V = 0V $$
$$ I_3 - I_O + 2V_O = 0A $$
To facilitate the solution, using matrix manipulations, or inverse, or whatever technique you want to use, here are those same equations with all the unknowns in columns:
$$
\begin{array}{lllllll}
&+V_O &+V_X &+4I_O & & &= &0 \\
& & &+4I_O &-I_1 & &= &-5 \\
& &+V_X &-4I_O &+I_1 & &= &0 \\
& & &+4I_O & &+2I_3 &= &+10 \\
&+2V_O & &-I_O & &+I_3 &= &0 \\
\end{array}
$$
I'll cheat again, and use an online solver, which gave me this result:
$$
\begin{aligned}
V_O &= -\frac{5}{11} &= -454.5mV \\ \\
V_X &= -5V \\ \\
I_O &= +\frac{15}{11} &= +1.364A \\ \\
I_1 &= +\frac{115}{11} &= +10.46A \\ \\
I_3 &= +\frac{25}{11} &= +2.273A
\end{aligned}
$$
Our Thevenin voltage is \$V_{TH} = V_O = -454mV\$.
We can find Thevenin resistance \$R_{TH}\$ by finding short circuit current. Since we already have the relationship between \$I_4\$ and R, this is trivial. Simply set R to zero:
$$ I_4 = -\frac{5}{4+11R} = -\frac{5}{4} = -1.250A $$
From there, finding \$R_{TH}\$ is a case of finding what resistance across \$V_{TH}\$ would produce -1.25A:
$$
\begin{aligned}
R_{TH} &= \frac{V_{TH}}{I_4} \\ \\
&= \frac{-454.5mV}{-1.250A} \\ \\
&= 363.6m\Omega
\end{aligned}
$$
However, that approach assumes you've derived the formula relating \$I_4\$ and R, and frankly that was a pain to do. So perhaps it would be easier to modify those simultaneous equations one more time, to find short circuit current via a matrix solution instead.
Firstly, we set \$V_O = I_4R = 0V\$ in all the original equations:
$$ 4 \times I_O + V_X + 0V = 0V $$
$$ 5V + 4 \times I_O - 1\Omega \times I_1 = 0V $$
$$ 1\Omega \times I_1 + V_X - 4\Omega \times I_O = 0V $$
$$ 4\Omega \times I_O + 2\Omega \times I_3 - 10V = 0V $$
$$ I_3 + I_4 - I_O + 0A = 0A $$
As a pseudo-matrix:
$$
\begin{array}{lllllll}
&+V_X &+4I_O & & & &= &0 \\
& &+4I_O &-I_1 & & &= &-5 \\
&+V_X &-4I_O &+I_1 & & &= &0 \\
& &+4I_O & &+2I_3 & &= &+10 \\
& &-I_O & &+I_3 &+I_4 &= &0 \\
\end{array}
$$
The solutions are:
$$
\begin{aligned}
V_X &= -5V \\ \\
I_O &= +\frac{5}{4} &= +1.250A \\ \\
I_1 &= +10A \\ \\
I_3 &= +\frac{5}{2} &= +2.500A \\ \\
I_4 &= -\frac{5}{4} &= -1.250A
\end{aligned}
$$
That's the same result for \$I_4\$ short-circuit current as we predicted before, and will of course yield the same \$R_{TH}\$. By the Maximum Power Transfer Theorem, clearly the load resistor R should be \$ R = R_{TH} = 363.6m\Omega \$ for it to dissipate maximum power, which agrees with the algebraic result we found earlier.
Here's a working CircuitLab model to see for yourself that it's all good. Notice that I've reversed the + and - connections on the current source, to correct the current direction, and resolve the discontinuity problem I encountered when I originally answered this question:
simulate this circuit
Here's a graph of power in R vs. R, to see the peak power at 363.6mΩ: