Don't worry about the hole in the enclosure. You have a far bigger EMC problem with the wires that are passing through it.
EMI signals will get onto the wires, and will then pass happily though the hole, treating the hole as the outer of coax, and the signal lines as the coax inner. We all know how good coaxial cable is at conducting signals, right? It doesn't really matter if the hole is big or small, one per conductor or one for the bunch, the coaxial mode through the hole will dominate the EMI.
Is EMI going to be a problem? There's no point doing expensive work to cure a non-problem. Can you test or model it at this stage to see? There are several different aspects to EMI which might require slightly different solutions. There's radiation produced by the signals in that connector. There's radiation produced by other signals within the enclosure coupling to the lines on that connector. And then there's pickup of external noise coupling into your case, which could upset operation of your device.
If you do have a problem, there are several ways to reduce it. You can stack some of these ways together to improve the final result.
Use shielded cable and shielded connectors for your 0.1" cables, they exist. Connect the shields to the case at the point of entry.
Filter the EMI at the cable level. Put a ferrite tube onto the bunch within your case, and/or on the cable outside (you've seen these bulges on laptop power supplies and monitor cables). This will raise the impedance for EMI waves travelling in common mode on the bunch as a whole. It will improve the balance of any differential signals and improve the return current balance of any single-ended signals.
The above two methods will work with any intended signal on the wires. The following methods may compromise the signals, so need to be designed properly, and are a compromise between reducing EMI and maintaining a good signal integrity. They may not be possible at all.
Filter the EMI at the individual wire level. Place ferrite beads or resistors in series with each wire.
Place capacitors (ideally feedthrough capacitors) grounded as near as possible to where the signal wire penetrates the case.
Reduce signal swings. Reduce signal edge slew rates. Use differential signals.
