I did a quick simulation without putting a lot of effort into optimizing it.
The value of the input capacitor seems to matter in the simulation, I reduced it to 2.2\$\mu\$F and it reduced the supply current, I don't know if this improvement will translate to hardware though. I also removed the 2 Meg resistor as it didn't seem to matter.
I chained the 5 V regulator off of the 12 V one. I tried running it from the 20 V, the supply current was slightly lower but had some spurious spikes.
It looks like it might be possible. You need around 83% overall efficiency, the LT3631 data sheet shows it doing better than that. As tobalt said, it's on the edge, you've got a pretty slim margin to work with.
Note that the average current is within your specs, but the peak is ~220 mA. If you're spec is based on peak you're probably up the proverbial creek.