I'm trying to build this multiplexed LED matrix with 216 LEDs, NPN transistors, a PIC microcontroller and several shift registers (74hc595) , however I don't know how to calculate the value of the resistors that are connected to the anodes, and also would like to know if I should add any additional resistors somewhere else. I plan to turn on a maximum of 16 LEDs at once, like 3 columns of 6 LEDs.
Since you are using so many LED’s, I will assume you are using inexpensive common LED’s. Typical LED needs about 1.5 volts across LED (look up your LED voltage at the current you choose for your LED). Depending upon the brightness you desire, you could end up somewhere between 10mA and 20mA.
Using 74HC595, looks like you will be driving the LED’s using 5 volts out. So, if LED voltage is 1.5 volts, then the resistor voltage will be 5v – 1.5v = 3.5v . . Use ohms law to calculate resistor R = V/I .
One word of caution. The 74HC595 on several different data sheets shows a maximum of 6mA output drive or current sink capability. Data sheet also shows output clamp current of 20mA, depending upon the manufacturer. If you try to get the 74HC595 to drive more than 6mA, the output voltage of the 74HC595 will begin to depart from being 5 volts.
You may want to set up one of your 74HC595 shift registers with a resistor and LED on the output to assure the performance is as you desire. In your schematic the 74HC595 will be sourcing current, not sinking current, but the data sheet seems symmetric for output current sinking or sourcing.
As you draw more and more current from your 74HC595, the output voltage will begin to droop (reduce) down from 5 volts. Also, the 74HC595 will begin to heat up some more.
EDIT : Just noticed that if you turn on ONE Column, and MORE than one Row, you will be lighting more than one LED. This puts the several LED's in parallel, and the brightness of each of those LED's will dim. Fortunately the resistor will still limit the current drawn from the 74HC595.
EDIT EDIT : Also just noted you did include your LED as 3.8 volts at 0.02 (20mA). Your resistor is going to end up near 100 ohms. That is a lot of current to ask from the 74HC595.
Unless you're real familiar with optical measuremnt units (lumen,candela etc) you probably won't be able to easily compute the resistor you need. Instead get a sample led and experimentally figure out how bright you want it to be, measure the current and voltage of the led, and then pick a resistor that gets close to that.
If forced to guess I'd start with 330 Ohms and adjust from there if the result is unsuitable.