Enough time has gone by that I'll try and point out a few things.
If you reach back into learning how to multiply by hand, you may remember doing multiplication by summing partial products. Here's an "unsigned" example:
Note the shifting process? In binary, it can be done about the same way. Instead of shifting "by 10," the shifting is "by 2." But that's about the only difference.
To make this work out in binary, you need these partial products. Luckily, they are very easy to produce. Just a set of AND gates achieves what you want. One set of AND gates will be required for each partial product. So for a \$4\times 4\$ multiplication, you'll need a total of 16 AND gates to create four partial products. Something like this:
simulate this circuit – Schematic created using CircuitLab
Now, with these partial products, you can work on setting up the adders to go with them:
simulate this circuit
This is the unsigned partial product multiplier topology. There are lots of other (and for some purposes, better) approaches to the problem. But I'm pretty sure this is what you wanted.
Of course, you'll need to apply your handling for signed numbers, performing the negation as appropriate. There is some added control logic needed to decide when to negate the output, as well. None of that has been shown here.
If you complete this, consider thinking more closely about how your signed negation logic operates. For example, in all circumstances after negation, except one case, the high-order bit of each 4-bit word is 0. The only case where the high-order bit is 1 is when the binary value is 1000. The negation logic itself can be seriously optimized, needing only 7 gates rather than a bunch of half-adders. Also, you can seriously reduce all the half-adder and full-adder stages used if you optimize for the only case where the high-order bit of a word is 1. In other words, the whole area opens up to crafted thinking and this eventually leads into radix-4 and much more, later.