I would like to make a long range (5Km) quadcopter. As of right now, I'm just trying to find out the best way to control it. I know that there are some 5Km R/C transmitters but they are the size of brick which is a little inconvenient. I was wondering if communication could be made through GSM, GPS or other wireless protocol that would be cheaper and/or weigh less. Any ideas on how to achieve this long range is accepted.
There are at least two commonly used civilian mechanisms for long range control of UAVs or RC cars: Public-band TX/RX radio, and mobile packet data, depending on the capabilities onboard the remotely controlled vehicle.
For public-band RF control at long range, modules such as the Xtend 900 1 Watt RSPMA by digi work well at 5+ miles, giving between 10 kbps and 115 kbps data rates, with very low power requirements. At 18 grams, they wouldn't be classed as "bricks".
These modules are a bit pricey, at nearly $200 each, including the cost of the antenna. The antenna itself would be a concern as well, as it would need to be fairly substantial, thus interfering with airflow. The most common type of antenna is the duck antenna, similar to those used in some wireless routers.
Note that the 900 MHz band is not available for public use in most countries other than the USA, Canada, Australia and a few others. Even in countries where it is permitted, the 1 Watt maximum power of this module may be far in excess of what the local telecom regulations permit.
While the marketing material suggests a 40 mile range for this device, realistic tests by aeromodelers indicate a usable range of 5 to 10 miles, less in areas with a lot of radio traffic in the 900 MHz band.
For a tighter budget, the XBee-Pro 900 XSC S3B, also from digi.com, would be worth experimenting with, albeit with some compromises:
These modules deliver around 10 kbps data rate, at a realistic range of perhaps 3 to 5 miles in line-of-sight. The marketing material suggests a 28 mile range, take that as you will.
The good thing is, the antenna is just a piece of wire soldered to the board, hence not so disruptive on an RC aircraft. Also, at a price point of ~ $70, these are much easier on the pocket.
The mobile packet data options are a bit more restrictive: Data rates are highly dependent on cell-site capacity, cell circuit availability and tower distance, and latency is widely variable. If the UAV has sufficient on-board intelligence that it essentially requires only occasional waypoint updates, and non-time-critical commands (e.g. "return to takeoff point at time X:YY and land"), this might be viable.
GPRS modules such as SM5100B from SparkFun are used by some RC enthusiasts. At under 9 grams weight, again these are not brick-sized. Also, the price is similar to the XBee-Pro 900 above.
Range can be arbitrarily high - so long as base controller and RC vehicle are both within range of cell towers for which network access is available to the respective units, the mechanism will work - even across countries, let alone miles, so long as latency and occasional signal drops are not an issue.
A good thought experiment would be to consider whether your RC vehicle communication could work by polling a web URL over the internet, for instance. If yes, then GPRS will work fine for the purpose.
It is worth keeping in mind that any such long-range RF remote control is affected by the vagaries of everything from sunspot activity to local weather conditions or someone using a cordless phone nearby to make a call. Hence, there should be little dependency on continuous control channel availability or data throughput. That being said, it's being done all the time, even in hostile terrains like the Antarctic, so have at it!
TI is claiming that their RF high performance chips can go up to 25km. See their RF Performance Line page that includes the CC1200 device that claims that range.
In the real world you can not always attain that because of various factors like multipath fading. But since you are using a quadcopter and it is elevated most of the time, the effect of multipath fading will not be as bad compared to a system that has 2 transceivers talking to each other while both elevated 5ft above ground.