I hope that's not from your schematic capture program- there are obvious shorts in that matrix.
Normally with a matrix you are going to make one row active at once and drive however many LEDs in the column that are required, or do the opposite and make one column active and drive however many rows are required.
In the first case, you need a resistor in series with each column driver, in the second case you need a resistor in series with each row driver. So, if it's an m x n matrix, you need either m or n resistors.
If you drive an m x n (row x column) matrix with equal time for each row (or column in the second case) then you'll have any LED that is 'on' energized for 1/m of the time (or 1/n in the second case). So if you want the LED to see 5mA average, it will have to have a resistor that is (Vdriver -Vf)/(m * 0.005) (or n * .. in the second case), where Vdriver is the voltage difference between the column and row drivers.
So if you have a 3 x 4 matrix, are driving it one row at a time, have 5V Vdd and your row and column drivers have 4.5V difference, and your LED forward voltage Vf is 3.0V when it sees 15mA, then your resistor value is (4.5 - 3.0)/0.015 = 100\$\Omega\$ and you'll need 4 of them.
The row drivers need to be able to handle 4 * 15mA = 60mA (only one on at time) and the column drivers have to be able to handle 15mA each (all of them can be on at once), and your entire matrix will require 60mA (12 LEDs at 0.005A each = 60mA).
If you put the resistors on the wrong side (column if you're driving only one column at a time or row when you're driving only one row at a time) you'll get horrible variations in brightness depending on how many LEDs are on, so don't do that.
Here's an example of a complete driver circuit driving one row at a time and using one resistor per column.
It may seem undesirable to use one resistor per column (40 in this case) but there is really no alternative. In your case both the high side and low side drivers must be low drop- the Micrel parts would be suitable, and you could use P-channel MOSFETs for the high side drivers. Your LED part is similar to the one used in this project (high side drivers used for rows).