It's hard to trouble shoot these types of situations remotely. Let's start with your two problems, and see what can be said about them.
"It is Unstable" Well, maybe not.
The 200mV on the output could just be ripple. Given the amount of ripple current, and enough ESR in \$C_{\text{out}}\$, that would be a reasonable amount of ripple. If the 200mV is at 600kHz, then it's ripple not oscillation.
\$V_{\text{FB}}\$ dropping from ~1.2V to ~0.9V would cause \$V_{\text{out}}\$ to shoot up (I'm actually surprised it doesn't go up more than that). The regulator would be reacting to raise \$V_{\text{FB}}\$ back to 1.2V. The only reason \$V_{\text{FB}}\$ stays at 1.2V is due to feedback and regulation. Also, you don't get to observe \$V_{\text{ref}}\$, it has to be inferred from \$V_{\text{FB}}\$. That this happens after "a minute or so" suggests that it is a thermal thing not an electronic thing. It is as though \$R_1\$ loses proper connection and that allows \$R_2\$ to pull \$V_{\text{FB}}\$ low. These peg type breadboards, with use, can get the contacts deformed so connections get intermittent or thermally sensitive. Do any of the parts get hot (especially \$\text{IC}_1\$, \$C_{\text{out}}\$, or \$D_1\$), and what happens if \$R_1\$ is removed?
At higher loads "regulator fails to keep the voltage at 5V and allows it to drop to < 4V" ... "Why?" There are a bunch of possible causes.
First, switching power supplies convert power. If the supply were perfect \$P_{\text{out}}\$ would equal \$P_{\text{in}}\$. That means that for a constant load, if \$V_{\text{in}}\$ went down, \$I_{\text{in}}\$ would have to go up a corresponding amount. So, the input of a switching supply looks like a negative resistance (or impedance). If the source output resistance is not less than the supply input resistance, at the least the source voltage will be pulled down and commonly will oscillate.
There are all kind of possible problems with breadboards. Things like parasitic resistances and inductances (in the board and in the components) can have a big impact on SMPS operation. For example, the breakout board that \$\text{IC}_1\$ is mounted on could be a problem both thermally and inductively. MIC2296 needs an amount of copper pad to get rid of heat properly (usually specified on the datasheet, although I didn't see it).
Some things to expect from this regulator
For a SEPIC, output voltage to input voltage can be related as:
M = \$\frac{V_{\text{out}}}{V_{\text{in}}}\$ = \$\frac{D}{1-D}\$ where \$\ D\$ is duty cycle (in constant conduction mode)
also, \$\ D\$ = \$\frac{V_{\text{out}}}{V_{\text{in}}+V_{\text{out}}}\$
If \$V_{\text{in}}\$ = \$V_{\text{out}}\$, then \$\ D\$ will be 0.5. This gives an easy way to calculate the current for boundary conduction mode (BCM, where operation goes from CCM to DCM). Just calculate peak to peak ripple current in \$L_1\$ when \$V_{\text{in}}\$ is 5V and BCM current will be half that.
\$i_{L-\text{ppk}}\$ = \$\frac{D V_{\text{in}} T_s}{L}\$ ~ 400mA
So, with \$L_1\$ = 10uH, the boundary for continuous conduction will be \$I_{\text{in}}\$~200mA. Usually best to choose CCM or DCM since the loop dynamics are different. CCM dynamics are more complicated (2 poles +2 poles in a SEPIC), but if compensated for CCM will stay stable for DCM (just not optimal in DCM). An advantage of CCM is that peak currents are lower in the switch and inductors, and output ripple voltage is lower. If you want CCM down to 50mA, L will need to be about 40uH.
Average \$I_{\text{in}}\$ and \$I_{\text{out}}\$ are related as:
\$I_{\text{in}}\$ = \$\frac{D I_{\text{out}}}{1-D}\$
To get peak input current just add half the inductor ripple current.
Peak switch current would be \$I_{\text{in}}\$+\$I_{\text{out}}\$+\$i_{L-\text{ppk}}\$ since the switch sees the current of both inductors. MIC2296 has a max switch current of 1.2A (chip will limit current there).
And that's kind of tough, since at \$V_{\text{in}}\$ of 3V (and D of 0.625), \$I_{\text{out}}\$ will be limited to ~ 300mA.
Some good references for SEPIC are SLYT309 and SLUA158
