I have used this exact model that you describe in volume production of LED matrix displays (128x32) that were designed to replace old gas discharge pinball displays.
The high side is driven using HC164 shift registers supplied from 5V (needed due to legacy interface requirements). These drive the gates of P-channel MOSFETs with source connected to the LED supply voltage which is around 3.3V - You want to keep this as low as possible to minimize power consumption and derive it from a switching regulator.
There are a lot of suitable P-MOSFETs but choose ones with sufficient Vg max typically +/-12V, and the right range of Vgs threshold voltage. The best value is around the -1V mark because then the threshold is near the 2.5V middle level of the HC164 logic drive which swings from 0V (LED current on) to 5V (LED current off).
On the LED cathode side, I use multiple TB62747AFG ICs. This is a cascadable 16 channel constant current sink cathode driver which incorporates a 16 bit shift register and latch register, as well as output enable.
The controlling processor drives the clock and data inputs to the HC164s and also the Data, Clock, Latch, and OE lines of the TB chip.
This may be overkill for the number of LEDs you want to drive, but I am pointing out that your original idea works well in practice and is not expensive in terms of parts. In fact there are Chinese ICs with identical pinout to those described that can be bought for 10 cents or less.
I have had no problems at all with this setup. You can see a photo of the board here: Pinball Display PCB.
Also if you look carefully at the photo of a commercial 32x32 color LED display panel that Adafruit sells you can see that it most likely uses the same approach. (32x32 LED matrix panel.) In this case there are 12 TB62747 style shifter cathode driver ICs and a bank of 16 8-pin MOSFET chips on the high side. It's hard to see the part numbers but in the middle of the picture there are also two 16 pin chips which could be two 8-bit shifters driving the 16 MOSFET gates. That gives 16 channels on the high side and 12x16 = 192 on the low side, allowing multiplexing of 32x32xRGB LEDs.