I've been planning a Veroboard/stripboard layout for four dot matrix LED displays. Most of the complexity is due to the daft way companies break out the pins on the 8x8 modules. This is what I have so far:

The fat lines are wires, and the striped orange bits represent stripboard. The colored dots are the pins of the 8x8 dot matrix displays. I only intend to use the red dots, not the green ones
I've got a couple of questions concerning this layout:
How can I physically connect the strips of perpendicular stripboard at the top and bottom (which will have male headers attached) to the main board? Hot glue? Overlap the boards and "sew" them with wire?
Is this really an achievable thing to do? Should I be looking into PCBs?
EDIT:
Looking into PCBs. Board design:
