Since you aren't mentionning any battery failures, I am assuming you are talking about lead acid batteries. Otherwise you would have had some kind of constant current charging strategy.
But two things you need to realize are: the charging current will be proportionnal to the voltage difference between the output of your pannels and the voltage of your battery, the charge voltage for a single battery should be around 15.5-16V, in your case 64V total.
Thus your solar setup barelly reaches the target charging voltage, this explain the poor charging of your batteries.
On the other hand, 4 panels in series would be to high a voltage for your battery pack, thus you will have to convert your PV voltage down to 64V DC in order to get the proper charging current (which would in theory fully charge your battery).
This is the crude way of doing it. In home/industrial PV setup one usually want maximum efficiency. In this case you will implement an MPPT (maximum power point tracking) strategy in your power conversion system. That would insure that maximum power is drawn fron the PV regardless of the voltage/current, and power is converted back to the voltage needed in a second stage (in your case 64V). But this is slightly more complicated to do. You might want to look at this for a future upgrade.
What I would recommend you to do, is to find a good 76V to 64V regulator (or close to that), plug your solar panel on one end, your battery pack on the other end, and that would do the trick for a barebone PV battery charger.
But please be careful, and do your homework before trying anything that might cause damage to your vehicle, yourself or others. Fire is sometimes overlooked when working with electricity.