In most of the helical antenna, I have seen a ground plane, what is the role of this plane and is there any size restriction for this plane? I am trying to design a helical antenna with minimum ground plane size.
what is the role of this plane
Like a quarter wave monopole, a helix antenna with a ground plane can be driven unbalanced. Also, all the power transmitted goes mainly in one direction. For instance, if you converted a dipole to a balanced helix you would get transmissions along the centre line of the helix both forward and behind.
If this is something you might need then go for a balanced type helix antenna that doesn't need a ground plane.
If you still want a ground plane then making it too small will not give you the best efficiency and the power projected in the forward direction would reduce to be replaced with power transmissted in the reverse direction (unwanted).