My new PCB design is about 25 mm * 25 mm, this is not enough for a proper 868 MHz antenna ground plane, I am using the 0868AT43A0020 chip antenna where at least a 20 mm * 40 mm GND plane is required. The first prototypes confirm the weak RF performances. what is the standard practice when it comes to dipole antenna when the ground plane of the PCB is extremely small and not sufficient?
the only place in the document you link to that contains 20×40mm² dimensions is the eval board. That's just "randomly" in that size.
What's important is that the chip is offset some 1mm from the edge of a ground plane, and that you have the folded whip as specified, and that you properly match your antenna, as specified in multiple places of the data sheet. My guess is you forgot the latter.