First instructions are not necessarily "executed sequentially" even on a non-VLIW ISA, execution only needs to appear sequential. An in-order superscalar implementation can execute more than one instruction in parallel with another. To do this effectively the hardware for decoding instructions must be increased (widened), hardware must be added to ensure data independence of instructions to be executed in parallel, the execution resources must be increased, and the number of register file ports is generally increased. All of these add transistors.
An out-of-order implementation, which allows later instructions to be executed before earlier ones as long as there are no data dependencies, uses additional hardware to handle scheduling of instructions as soon as data becomes available and adds rename registers and hardware for mapping, allocating, and freeing them (more transistors) to avoid write-after-read and write-after-write hazards. Out-of-order execution allows the processor to avoid stalling.
The reordering of loads and stores in an out-of-order processor requires ensuring that stores earlier in program order will forward results to later loads of the same address. This implies address comparison logic as well as storage for the addresses (and size) of stores (and storage for the data) until the store has been committed to memory (the cache). (For an ISA with a less weak memory consistency model, it is also necessary to check that loads are ordered properly with respect to stores from other processors--more transistors.)
Pipelining adds some additional control and buffering overhead and prevents the reuse of logic for different parts of instruction handling, but allows the different parts of handling an instruction to overlap in time for different instructions.
Pipelining and superscalar execution increase the impact of control hazards (i.e., conditional branches and jumps). Pipelining (and also out-of-order execution) can delay the availability of the target of even unconditional jumps, so adding hardware to predict targets (and direction for conditional branches) allows fetching of instructions to continue without waiting for the execution portion of the processor to make the necessary data available. More accurate predictors tend to require more transistors.
For an out-of-order processor, it can be desirable to allow a load from memory to execute before the addresses of all preceding stores have been computed, so some hardware to handle such speculation is required, potentially including a predictor.
Caches can reduce the latency and increase the bandwidth of memory accesses, but add transistors to store the data and to store tags (and compare tags with the requested address). Additional hardware is also needed to implement the replacement policy. Hardware prefetching will add more transistors.
Implementing functionality in hardware rather than software can increase performance (while requiring more transistors). E.g., TLB management, complex operations like multiplication or floating point operations, specialized operations like count leading zeros. (Adding instructions also increase the complexity of instruction decode and typically the complexity of execution as well--e.g., to control which parts of the execution hardware will be used.)
SIMD/vector operations increase the amount of work performed per instruction but require more data storage (wider registers) and typically use more execution resources.
(Speculative multithreading could also allow multiple processors to execute a single threaded program faster. Obviously adding processors to a chip will increase the transistor count.)
Having more transistors available can also allow computer architects to provide an ISA with more registers visible to software, potentially reducing the frequency of memory accesses which tend to be slower than register accesses and involve some degree of indirection (e.g., adding an offset to the stack pointer) which increases latency.
Integration--which increases the number of transistors on a chip but not in the system--reduces communication latency and increases bandwidth, obviously allowing an increase in performance. (There is also a reduction in power consumption which may be translated into increased performance.)
Even at the level of instruction execution, adding transistors can increase performance. E.g., a carry select adder adds upper bits twice in parallel with different assumptions of the carry-in from the lower bits, selecting the correct sum of upper bits when the carry out from the lower bits is available, obviously requiring more transistors than a simple ripple carry adder but reducing the delay in producing the full sum. Similarly a multiplier with a single row of carry-save adders uses fewer transistors (but is slower) than a Dadda (or Wallace) tree multiplier and cannot be pipelined (so would have to be replicated to allow another multiply to begin execution while an earlier multiply was in progress).
The above may be exhausting but is not exhaustive!
