Yes, but ... .
A capacitor can help as you suggest but it may need more than that.
Restating Steven's formula - a Farad will supply one amp for one second with one volt of drop. So 10,000 uF ( = 0.010 Farad = 10 milliFarad = 10 mF*) will supply 0.01A = 10 mA for one second with one volt drop or 1 amp for 0.01 second with 1 volt drop. Murphy and reality adjust actual result somewhat but that gives you an idea
Capacitor Must be on a DC rail (ie not on AC input.)
Note that some such devices use AC input from power supply so that they can rectify it internally and generate + and - supply rails. Unusual but check what the power pack output voltage specification plate says.
Also, a few supplies feed 2 or more voltages from the power pack, but this is unusual.
A superCap may help for longish brownouts.
For very substantial brownouts you may not be able to sensibly provide a large enough capacitor. You could use a battery pack slightly below supply voltage with a Schottky diode to V+ so that when the mains dips the battery takes over automatically.
A 9V "PP#" / "transistor" radio battery may do but voltage may be too high compared to 9V loaded supply. If so you could use several series diodes from battery to 9V rail to drop voltage. eg if battery = 9.75V O/C (about right for very new Alkaline 9V) and if loaded supply rail = 8.8V (say) difference = 9.75 - 8.8 = 0.95V.
2 x silicon (not Schottky diodes would cause 9V battery to take over at ABOUT 9.75 - 1.2 = 8.55V. Drop across silicon diode (eg 1n400X) IS > 0.6v at 10's to 100s of mA. You could use an LDO regulator to allow very precise battery takeover BUT quiescent current needs to be very low. Not so crucial if battery is charged from supply usually - see below.
If using a battery you could use a rechargeable - connect to rail via diode as above AND a resistor in parallel with diode. Dimension resistor to provide a very small trickle current to keep battery supplied.
The smallest of UPS's with their own internal battery will handle this application well. Even one which has a very dead battery will probably hold up long enough for this. Such may be available free or close to free depending where you are.
For interest, where are you located?
All values below are examples. Use values to suit what you are doing.
A large capacitor MAY be able to be used directly - depends on how the power supply reacts to a heavy short term overload as the capacitor charges.
If the supply does not "like" the capacitor startup load then as you note - use a resistor to charge the capacitor and a Schottky diode across the resistor to discharge into load when required.
As a starting point, dimension the resistor to allow cap to take max allowable current when cap is short circuit at startup. So if eg 9V 500 mA supply, a R = V/I = 9/0.5 = 18 ohm resistor will take 500 mA when the capacitor is dead short at startup and this will decrease as the capacitor charges.
If the capacitor is say 10,000 uF then with say 18 ohm as above the time constant = RC = 18 x 0.01F = 0.18S. The capacitor will charge in under a second. 1 second ~= 5 time constants, but as the supply is also driving the router as it starts up all the current will not be available.
You may be able to use a lower value than 18 ohms in the above example. try and see. Observation with an oscilloscope would help but even an analog meter will give you an idea of how long charging takes.