Traditionally new packages are developed for better manufacturing yields or higher density. So if there would be any advantage, it would have to be in that primarily. I don't think that manufacturing yield is a reason.. still the same pin technology, pin pitch etc.
Density is pins/area -> so a given package should have more pins on it to make sense.
If you consider a square and an octagon with the same overall width/height (e.g. the octagon fits inside the square), you could calculate the perimeter of both packages. I assume the perimeter as a direct indicator of how many pins can fit on a package.
For a rectangle the perimeter is 2*w + 2*h.
For an octagon the perimeter is 8 times the length of a side. The length of a side for an octagon is 0.4142*w. (source: wikipedia).
Substitute x=y=1cm, you get 4*1=4cm perimeter for a square and 8*0.4142*1=3.3136cm for an octagonal. That's 17.16% loss in perimeter length, or 17.16% less pins on the same width/height package.
That makes sense, because you're basically "cutting a corner" 4 times. So I can understand why these packages don't exist.