You have taken many liberties in this design especially on the layout but, I think a significant concern is something you inadvertently hinted at in the 2nd comment under the question: -
High current is also upto 500mA
So, you expect an output current up to 500 mA. That's at 12 volts hence, the input current into the booster is going to be a lot more that 500 mA. You say this in another comment: -
Input Voltage is 3.7volts (cell supplied) and output should be
So, making the assumption that power out is 90% of power in, it seems like your input supply current from the 3.7 volt battery will be about 1.8 amps.
Then you say that the input to the booster is controlled by a transistor and I see an FMMT720 BJT where the collector feeds your boost circuit. From the FMMT720 data-sheet I see this graph and on it I've redlined where 1.8 amps occurs: -
So, unless you are driving the base at a pretty good level the transistor will be dropping several hundred millivolts when supplying 1.8 amps. This means that the terminal voltage reaching your booster circuit is not 3.7 volts but maybe only 3.4 volts and that now means that your input current is required to be about 2 amps (to sustain the output load) and, that will drop even more voltage across the transistor and things will not end well (a downward spiral you could say).
But what about the inductor: -
It's only rated at 0.88 amps and that's a lot less than 1.8 amps. In addition, there is the maximum DC resistance of 0.23 Ω to consider. OK it's a max value but, typically it might be 0.2 Ω and, with 1.8 amps flowing, the DC volt drop is 0.36 volts and that another reduction in the input supply voltage. You might be realistically operating with an effective input voltage of only 3.0 volts and that requires an input current of over 2.2 amps to be able to supply a load of 500 mA on the 12 volt output. And also, the data sheet for the chip says the recommended maximum inductor value is 22 μH; yours is 33 μH.
In my opinion, that ain't gonna happen.
It's a perfect storm of wrong components, miscalculation and terrible PCB layout.