Since it works for a while, it sounds like it is basically hooked up right but you are getting data corruption. When it hangs, the slave address probably got corrupted at the receiver, it didn't happen to be a address any of the other devices think is theirs either (whether it really was or after corruption), no ACK is sent, and the transmitting firmware hangs. It is surprising how poor most IIC firmware implementations are in dealing with no ACK received. There is a lot of irresponsible IIC firmware out there.
If the above guess is correct, then two things need to be fixed. First, the firmware needs to behave responsibly when no ACK is received at any time it is expected. This could be to a address or data byte. Second, the bus corruption needs to be fixed.
64 devices is a lot, but should be doable if all devices are on the same board. Taking IIC off board is asking for trouble, and with many devices doubly so. The exact topology of how the bus is connected shouldn't matter much since in any case it should be much shorted than the shortest wavelength of interest.
Put the stiffest pullup allowed somewhere on the bus. The maximum a IIC node needs to sink to drive the line low is 3 mA if I remember correctly. Make sure that is not exceeded, but otherwise use about the lowest pullup resistor allowed. For example, let's assume 3.3V power. 3.3V / 3mA = 1.1 kΩ. In theory a 1.1 kΩ pullup should be OK, but in this case I'd make it 1.2 kΩ. That's still almost the same pullup ability, and leaves a little margin against the maximum pulldown current. If the bus is mostly linear, you could put a 2.4 kΩ resistor at each end, although that will be a minor improvement, if any.
Take a look at rising and falling edges with a scope and see how long they take, then make your bit rate slow enough so that everything nicely settles within a bit time. All those nodes will add capacitance, which will slow the edges, particularly the rise time. There will be some bit rate limit you can't exceed due to this. It may not be possible to run the IIC bus as fast as the devices are otherwise capable of. That's a tradeoff you made when you decided to put 64 devices on the bus.
If you are still having problems after all the above is addressed, I would implement the IIC master in firmware without using any built in IIC hardware. I have seen some implementations from well known manufacturers have race conditions. Such implementations are particularly susceptible to a relative slew rate difference between SCL and SDA. My firmware-only implementations guarantee that never more than one thing happens per bit time. There is a minimum delay between every action that guarantees the bus has settled. This may run the bus slower than the hardware peripheral would, but makes for a good robust implementation. In fact, I usually use a firmware-only IIC master for robustness unless speed is critical.