First of, I disclaim that I never used the Xilinx CORDIC core, but I know Xilinx and I know CORDIC... Also, I invite you to check out Wikipedia to have a better understanding of what the core do.
The coarse rotation is a confusing point of Xilinx Cordic 4.0 documentation. The first quadrant is indeed 0 to pi/2 but Xilinx have modified the algorithm, probably because they preferred to have signed angle. For Xilinx Cordic 4.0 in translate mode without coarse rotation, the input coordinates must be located between pi/4 and -pi/4 (Verified with simulations by David Quiñones).
The original CORDIC algorithm operates on vector with positives X and Y. If a vector is outside that range, you have to input the absolute values to the algorithm, and modify the output accordingly. For example, if X > 0 and Y < 0, than calculate (r, theta) of (X, abs(Y)) and output (r, -theta). The coarse rotation module should deal with that for you and allow to input any vector. I suggest that if you remove the coarse rotation module, you verify what values you can input as the documentation is confusing.
Compensation scaling refers to the fact that the CORDIC algorithm scales the amplitude of the vector with each iteration. Each iteration scales by sqrt(1+2^(-2i)) where i is the iteration number. So if the length of your vector is 1 and you have infinite iterations, the CORDIC would give you a length of approximately 1.65. The compensation scaling module would scale that back to 1. As a side note, the Xilinx documentation write the scale factor as 1/cos(atan(2^-i)), which reduces to the sqrt I gave.
