The normal pattern for driving a MxN multiplexed LED display is to designate one direction as "rows" and the other as "columns", independent of physical layout on screen, such that the system will energize "row" wires one at a time and, while each row is sequenced, the system will energize the "column" wires for all the LEds on that row that should be lit.
Each column wire must include a device to regulate the current when it is activated (a resistor can work here), and each row wire must have a driver that can supply enough current to feed all the columns at once. Hardware does not have to allow more than one row to be driven at a time, but must allow any combination of columns to be driven.
The 74HC154 chips might work okay in row-select logic, but because only one output can be active at a time, they are not suitable for controlling columns. If you wanted to scan your "sideways", so that it was logically accessed as 32 rows and 8 columns, then the 74HC154 might be suitable for selecting a row, but then you'd have to add additional hardware to allow each row to supply enough current to drive eight LEDs, and you'd also have to add something to each output of the the UN2803 to limit currents.
More likely what you'd want to do would be to replace the 74HC154's with either a bunch of 74HC595 chips (shift registers) and resistors, or else one or more LED-driver chips which has current limiting built in (e.g. four Texas Instruments TLC5916, available from Digi-Key). Connect those chips to an SPI port from your controller, and shift out each row of data serially. If you use a low-side driver chip (like the aforementioned TLC5916) you'll have to change your row drivers to source current rather than sink it. The simplest way to do that would simply be to use eight NPN transistors, each with the collector tied to the positive rail (you could use an unregulated supply if you like), the base tied to the output of the controller or a decoder chip, and the emitter tied to an LED row.