tldr; I built and later extended Ben Eaters 8-bit breadboard computer. After the initial build I integrated 16-bit program counter and memory address registers, redesigned the instruction set and managed to access 64kB of RAM. Going forward I added 8-bit X and Y registers as counter registers comparable to X and Y of the 6502/10. With these 2 additional registers the microcode organization and the necessary ROM size become uncomfortably huge already. How can I better handle the microcode organization?
Details: In the current design I already have 7 8-bit registers (A, B, ALU, output, instruction register, X and Y counters). In addition I have 2 16-bit registers (program counter, memory address register) and a 2-bit flags register.
Currently I use 3 ROMs for microcode, each with 11 bits address width, therefore 2kB each and 6kB all together. The ROMs currently decode the output for already 18 control signals to handle input and output of all hardware attached to the 8-bit single bus.
The 11 bits of input for the decoder ROMs are formed of 6 bits from the instruction (which leaves me with only 64 instructions max.), 2 bits from flags and 3 bits from a instruction sub-step counter.
The design works like a charm, but I wonder, how I can still manage the microcode, if I extend the design even further. I plan on having more busses (data, address, ALU separated), more flags, more registers. I fear, that the sheer number of necessary input bits for the instruction ROMs will blow up their size pretty quickly. On the other hand also the need for control lines grows quite quickly.
Possible solutions I see: My current design has the advantage, that all instructions are super fast and need between 2 and 6 cycles max. I could for example get rid of control lines by reducing their number and use shift registers to form control signals sequentially. But this cost me many cycles per ASM instruction and would slow down the machine dramatically. The problem of input signals is completely unsolved.
I have no formal education in that field, therefore, please forgive my naive question. I would like to understand, how hardware engineers in the 70s and 80s solved these problems in CPUs like the RCA 1802, the MOS 6502 or the Z80. Are there concepts, that I am not aware of? Where can I learn about these? Can anyone help here?