You are going to have to revise a few figures:
The FMMT617 transistor is rated at Vcemax of - 20 27 V (in typ max)
Even at max value it will be thinking of breaking down with 10 x 2.8V LEDs.
You could operate eg 5S2P LED array a small resistor in each series string - say in the 22 - 47 Ohm range, will drop about 0.5 - 1V and help stabilise the two parallel strings.
Alternatively, other transistors could be used. A BC337-40 or its SMD equivalent MAY do very well there - even with a 10S string. .
Power out = 28V x 23 mA = 644 mW
The corresponding efficiency table for your circuit is on page 7 of the datasheet.
At 0.8V it shows about 77% efficiency. Such graphs are 'typical' so say 75%.
Power in = Powerout/efficiency = 644mW/0.75 = 860 mW
At 0.8V as i = P/V, I =0.86/0.8 = 1.07A.
That is MEAN Iin so as the input current ramps linearly (for a pure inductor), Ipeak = 2 x Imean (ignoring the low toff period) so
I peak = 2 x 1.07A = 2.15A. [!!!]
Time to charge inductor:
i = V.t/L or t = iL/V
Using your proposed 7.2 uH
t = 2.15 x 7.2E-6/0.8 ~= 19 uS.
Toff is set at 1.2 1.7 3.2 uS
So total cycle ~~= 19 + 1.2 to 3.2 ~= 20 uS
Or 50 kHz operation which is OK as < 200 kHz max.
Transistor base drive:
Drive level looks liable to be very inadequate.
IC base drive current at Vcc = 0.7V is shown as 1.5 - 3.5 mA. (datasheet page 2)
But transistor Vcesat is shown as 200 mV max at Ic=2.5A (about your current) and Ib = 50 mA, and 150 mV at Ic=1A and Ib = 10 mA (still about 3x what you have best case and 6x what you have worst case.
You say 0.9V min but that's not much better cthan 0.7V.
If using an Alkaline cell then it is very largely "flat" at 1V and dropping to 0.9V gives you only a small extra % of energy.
A "much better" arrangement if you can accommodate it is to run the LEDs from a single LiIon cell with a linear regulator. AT Vmax of say 4V (not 4.2) efficiency is 2.8/4 = 70% and it approaches 100% as the battery discharges.
By clamping the battery with a shunt regulator at say 4.0V you can safely charge it with no concern for CCCV charging as long as the max input current is below Imaxcell (typically C = 2 t0 3 Ah typically for an 18650 and maybe 1 Ah for an AA LiIon.
A problem is that this converter does not "bootstrap" its driver voltage from its output voltage. This COULD be done with extra effort, but other alternatives may be preferable.
____________________________________________
Regulation:
Vout will be set by the LEDs and Murphy - who may not believe your calculations :-). Note that the energy in the coil (= 0.5 x L x I_peak^2 x freq) is "just right" in the spreadsheet to give
(VLED x 2) x (ILED x 5) x efficiency
= NOTIONAL power required by LEDS.
However Power out comes from
energy in L
PLUS (Vbat -V_DS1 - Iout x Rl) x Iout.
With 2 new cells Vbat is about 3V so extra voltage is
~~= 3 - 0.5 - 1V? = maybe another 1 to 1.5V extra that the inductor voltage "stands on top of". As the battery voltage falls this voltage decreases AND the time to charge the inductor to Imax increases and frequency of operation falls (due to longer charge time) so this effect can vary substantially with battery condition.
IF the extra "pedestal" voltage from the battery thatv the inverter energy stands on top of is significant then the inverter needs to "lose" the extra energy somehow, so VLEDs will rise higher and ILEDS will be higher and ... .
The danger (unlikely but possible) is that VLED will rise high enough to destroy the IC. Even if the IC is OK, VLEDs will be higher so LEDs will be brighter and the battery life will be shorter.
If you run the inverter with no load or with say 1/2 load Vout will rise and destroy the IC. If one LED fails you lose one string of two LEDs so current in the other strings rises by 20%.
A means of limiting Voutmax is "a good idea".
All the preceding happens (or may happen) because the inverter is regulated by I_L_max and not by Vout or I out.
Methods to control output maximum include:
1. Clamp Vout with a zener able to absorb any excess energy. eg a 6V8 1 Watt zener.
2. Connect a say 5V6 zener from Vout to pin 4 Isense so when the zener conducts the inverter switches off.
3. Place a small resistor in one LED string below existing R such that this voltage can be combined with voltage on R1 current sense resistor. This gets "tricky without a combining opamp.
4. Feed voltage from Vout to a resistor divider and take a Schottky diode from there to pin 4 Isense.
All methods may take some "playing".
Using an opamp to detect the trigger condition and drive pin4 is probably easiest. A cheap LM358 / LM324 would be good enough. Without such feedback the system will still work but is harder to control at an exact operating point.
Added:
Given all the above, would you still recommend going with the bootstrap method? It looks like components will start adding up. The initial connection with 2 x AAA in series (Driver input is connected to the battery) won't yield the brightest LEDs but it will be more efficient, the battery will last longer.and I don't need to add the shotcky diode, zener and a cap at the output.
Non-bootstrap is OK PROVIDED that at lowest battery level you have enough voltage for startup AND enough drive for adequate LED brightness.
Without the Schottky Vstartup is fine.
At Vbat min = 1V say you have 2V for Vcc giving say 2 to 2.5mA base drive. Your NSS40201 transistor has good specs but as fig 6 page 4 (which you show in your question) shows, even 2.5 mA drive is inadequate. To get 3 mA drive you need Vcc > 5V at 25C (less V if IC gets hot). This suggests that the bootstrap arrangement is necessary.
If you are building one of a few of these then you may be able to mix and match to get something that works. If you are building say 10 or more you need to design to worst case specs and not assume that specs will be typical or better. Alas.
Why AAA? If size REALLY dictates this then 'so it goes'.
Otherwise, AAA are 'nasty'. Much lower energy content than AA, worse performance at end of life and about the same cost as AA.
LiIon would do quite well here if you can accommodate it. The whole of discharge efficiency is not liable to be worse than with the IC.
3 x AA or AAA plus linear regulator may be an attractive option.
Vmin = VLED/3 = 2.8V say/3 = 0.93V.
Vmax new = 3 x 1.65V for Alkalines but this very rapidly drips to 1.5V/cell = 4.5 V.
Efficiency = 2.8/4.5 = 62% ! , but this rises as Vbattery drops.
I suspect inverter efficiency will be under the 80% in your latest calculations.
For a linear discharge from say 1.4 to 1 V/cell average = 1.2V and mean efficiency = 2.8/3.6 = 78%. This is probably better than your inverter overall.
What's the application?
Why AAA?