I'm worried about cooling design and cooling performance so I prefer
using thick wires which will stay cool during operation under high
currents... I'll need to supply very low voltage (1V-3V , when 300A is used
This is not the right way to look at it. To keep the temperature down you need more copper. Whether it is supplied as a few turns of thick wire or many turns of thinner wire doesn't matter much (unless it is very thin, when insulation thickness becomes significant). Wire thickness only determines the required operating voltage. If you want it to work on higher voltage and lower current
then simply reduce the wire thickness and increase the turns to get the same amount of copper.
I currently have no idea what design will produce high speeds
(~5000RPM) and high torque {at low speeds of course} (~100nm)
100 N⋅m at 300 A = ~0.333 N⋅m/A. This is called Kt (torque constant). Kv (voltage constant) is the inverse of Kt, ie. 300/100 = 3 rad⋅sec-1/V = ~28.7 rpm/V. You want 5000 rpm so you need 5000/28.7 = ~174 V.
Motors with permanent magnets (PM) or separately excited field coils (or shunt wound on fixed supply voltage) have good speed regulation because these 'constants' cause the speed to be primarily determined by the applied voltage.
In a series wound motor the 'constants' are not constant, because the armature current also goes through the field coils. As the motor speeds up and field current goes down, Kv increases (and Kt reduces) making it spin even faster until limited by the load. This is useful when you want high torque at low speed as well as high speed at low torque from the same voltage (eg. car engine starter motor) but can make the motor over-speed and destroy itself with no load.
A compound motor has both series and shunt field coils, so its characteristics are between those of a shunt or PM and series wound motor.
So my question is : if I am operating this thick stator coils with the
low voltages I said earlier , will the rotor coils easily cancel out
the stator's EMF so the motor reach only low speeds ?
First we should define what you mean by stator and rotor coils. In a brushed motor the 'stator' coils are the fixed field coils (providing a constant field strength in a shunt-wound motor), and the 'rotor' coils are the armature coils switched by the commutator in sync with its rotation. In a brushless motor it's the opposite, with the fixed stator coils being switched and the rotor having a fixed field strength (so relatively speaking the same, but 'inside out').
While the 'rotor' (armature) and 'stator' (field magnets) are both producing magnetism there will be a force between them to make the motor speed up. However as it does so a 'back-emf' voltage will be generated across the armature as it moves through the stationary magnetic field. This reduces the total voltage across the armature and therefore the current through it (determined by circuit resistance according to Ohm's law) until the motor torque balances the load torque.
The higher the field strength the less speed is required to generate enough back-emf to match the supply voltage. If the field coils are passing a higher current they will produce a stronger field, making the motor's top speed lower. Reducing the field current will increase Kv and so increase (unloaded) motor speed, but reduce torque since Kt decreases as Kv increases.
can I fix this issue using a high voltage low current coils wrapped
around the low voltage high current coils and that will cancel the
back emf coming from the rotor so I can reach high speeds ?
If you don't need much torque at high speed then you can configure the motor as series or compound wound (depending how much effect you want). If you do need good torque at high speed (eg. for an electric car, which has to deal with increased wind resistance at higher speed) then you could just rely on the speed controller to 'transform' the power to a lower effective voltage and higher current via PWM. At lower PWM ratio the motor slows down because the average voltage is lower, and the armature current is higher than the average supply current by a ratio of 1 / PWM (eg. at 10% PWM the motor current is 10 times higher than the average supply current). This can be combined with current limiting to control torque without the danger of overloading anything.
In some applications it may be useful to control the field current separately in order to vary Kv. You need to be careful though because if the field current is lost then the motor cannot operate and current is limited only by resistance.
