First, notice that the two packages are very different in layout. One is called a DIP (Dual Inline Package - and yes, DIP package is redundant. Live with it.) package, and the pins (not the plastic body) are spaced on 0.1 inch spacings with a row spacing of 0.6. The other is a surface-mount package which does not use rows of pins. The difference is important.
First, you have to keep in mind that for the early logic chips, a 0.3 spacing became the de facto standard. It's important to realize that early (1960's) printed circuit fab techniques made the sort of narrow traces which we take for granted today very expensive, so running connections around a bunch of ICs was a problem for crowded footprints. Traces made on 0.1 inch center-to-center was the norm, with some daring designs using 0.050 pitch. To make matters worse, multilayer boards were almost unheard-of.
Even at the low gate densities of the time, there were some chips (like the 74150 and 74181) which required more than the common 16-pin DIP. At the time there was a reluctance to get the extra pins by making a longer, narrow package, and this had two issues. The first was PCB trace issues, and the other was mechanical. DIPs were made using a ceramic substrate, and a long, narrow platform would have been prone to mechanical failure when applying extraction force to one end of a socketed part.
So, since engineers and computer geeks tend to think in powers of 2 and 10, the larger pin counts were accomodated by doubling the row spacing to 0.6 and the standard length increased to 24 pins.
It's not certain if this was a dominant issue, but much logic design at the prototype stage was done using wire-wrap boards, and going from 0.3 to 0.6 allowed the production of "universal" WW boards with rows of pins at 0.3 spacing, allowing easy mixing of the two sizes. It would be nice to think that the IC companies recognized that engineers will tend to choose parts which are easier to work with in development and then use them in production.
It's worth pointing out that the choice was not universal. Some early RAMs with 22 pins were produced in 0.4 spacing, as well as the entire 100K ECL logic line, and occasional other chips as well, but the wild success of the TTL family made 0.3 and 0.6 the de facto standard.
With the explosion of chip capacity due to uCs and memory, pinouts began to grow, although chip sizes stayed within the bounds of 0.6 row spacing. Early (E)PROMs, for instance went from 24 pins to 28 quite rapidly, and thence to 32.
With the high pinouts needed for data busses, microprocessors jumped quickly to 40-pins, but once again mechanical constraints started to rear their ugly head. I believe there were a few 42-pin weirdos, but it was clear that going to longer chips would have bad reliability consequences due (again) to the fragility of ceramic substrates.
As a result, bigger chips, such as the Motorola MC68000 processors and various specialized DSP products such as multipliers and multiplier/accumulators, jumped to 0.9 inch row spacing, with 68 pins as the norm. About this time, though, it became apparent that there were shortcomings. With the big packages the lead lengths from the chip to the pins started getting onerous, especially as speed increased. Signal integrity/termination issues get much harder when there exists a long stub within the package. The answer was to go to packages which were not enormously larger than the chip, using SMD packages with much finer connection pitches than the old 0.1. This was aided by the fact that PCB production techniques had gotten good enough to accomodate the tight clearances required, and to do so at reasonable cost.
Just how bad the increased speed issues had become is shown by the introduction of some logic chips at the 20-pin DIP scale which had multiple ground pins instead of one, with the ground pins shifted from the convenient corner to the middle of one side, allowing very short ground connections from the chip to the pcb ground plane, with the specific aim of preventing ground bounce due to total lead inductance.
During this period, of course, it also became common to accept long skinny parts in smaller packages, particularly 20 and 24 pin parts in 0.3 spacing. As experience was gained with these skinny packages folks became more comfortable with them,and some specialty chips, such as FIFOs, were produced with 28 pins on .3 inch center, but these were special cases (FIFOs, for instance, tend to have very simple signal connections).