That formula isn't useful to you in its current form because your question (about directivity -- via "solid angle") is consumed in the aperture parameters. That equation basically answers this question:
"If I know the directivity, efficiency, and relative position of both my transmitting and receiving antennae, how much power do I need to ensure I receive at least this much when transmitting through free space?"
You need to refine your design problem since you have too many unknowns at present (design of the TX system, design of the RX system, frequency, etc...). All you've specified is the separation, received power, and physical receive aperture (note: that isn't the same as the Ar in your equation).
There are an infinite set of solutions.
In a theoretical antenna design, would you be able to recommend how I
could get started in refining the problem. Is there a specific gain I
need on the transmitting system in order to obtain ground rx area?
Ceteris paribus, ground RX area = Aphys in the article you mentioned. To get to Aeff you need to determine the efficiency. That requires either a computational or analytical solution to your antenna design (determine the current moments and from that the flux density).
But I think the question you wanted to ask is, "What order should I perform the design in?" You've got more contraints on your receiver than transmitter, so I would start there, but to do that, you must determine your frequency.
In free space, this is easy. Since you want small apertures, you need ridiculously high frequencies. If you want to go through Earth's atmosphere you need to go through "the window", which places a practical upper bound.
- Frequency
- RX size/design --> efficiency --> Ar
- With Ar, r, and lambda known, design transmitter antenna (perhaps it is also size constrained?) --> At
- Pr is known/desired, solve for Pt
