Your question cannot be answered by anyone on this site.
This is a very good question to raise, and is a concern you should have. You should talk about this with your favourite assembly house and verify with them whether they can meet the tolerance that the manufacturer requires. The tolerance from the MFG side by the way, is also something you should ask the MFG; not something easily found in the datasheet usually.
I can offer you my anecdotal experience, I worked on a project where whe had to put 6 connectors in an area of around 10x10 cm, for a total of a few hundred pins.
After obtaining the required tolerance from the connector manufacturer, we sent the query to the assembly house, and they asked us to put several fiducials in the connectors area, in order to minimise the error of the pick and place machine.
You have one additional variable here, which is the CM4 module. You can try to ask to the Raspberry Pi Foundation about the tolerances they can guarantee on their boards, they should be able to provide an answer to that. In our case we were producing both boards, so in the end the assembly house was able to guarantee that they were able to meet the specification from the manufacturer.
EDIT
I just opened one of the files from the page you have provided:
So there you have it, the manufacturer already provides you with the required tolerances, which are impossible to meet from any reputable assembly house - any non reputable one might just lie, and still not meet them.
Link to the document: https://www.hirose.com/en/product/document?clcode=CL0684-4151-0-51&productname=DF40HC(3.0)-100DS-0.4V(51&series=DF40&documenttype=Guideline&lang=en&documentid=0001442210
(for some reason the parenthesis breaks the link)https://www.hirose.com/en/product/document?clcode=CL0684-4151-0-51&productname=DF40HC(3.0)-100DS-0.4V(51&series=DF40&documenttype=Guideline&lang=en&documentid=0001442210