when choosing a suitable current rating for a resettable fuse do we usually look at the hold current or trip current of the fuse?
Both should be considered.
If your device must have a minimum current, then look at holding current. And if it must trip at some point, then look at trip current. Unlike Harper, I'm assuming you're talking about poly-resettable fuses.
when the current is between the hold and trip current, what happens to the fuse is uncertain.
What does the datasheet tell you? In general, if it is a temperature-sensitive component (similar to a PTC thermistor) it could warm up slowly, and eventually open. Or if it is some other process technology, not specifically used as a fuse (such as PTC thermistor) it could fail from overheating. There are many types of fusable protection; consult the datasheet. If it doesn't say, search for a failure-modes document. Failing that, ask the component manufacturer.
So for example if I have a component that has a maximum continuous current of 5A, should I look for a resettable fuse with trip current of 5A or hold current of 5A?
If 5A is being used and a 5A trip fuse is selected, then it will open (repeatably.)
If the answer is to look at the hold current, lets say the fuse hold current is 5A and the trip current is 10A, if the current in the circuit is now 8.5A, which is still below the trip current and the fuse doesn't trip, wouldn't the component be damaged?
The component being the fuse, or the component being the protected circuit? This cannot be answered without knowing details of the selected fuse and circuit.
If the answer is to choose trip current, this will mean that the fuse will have a hold current that is lower than the maximum continuous current rating of the component, for example a fuse with 2.5 A hold current. Doesn't this mean that the component will only be able to function below 2.5 A and will not be able to draw up to 5 A even though it is rated for that maximum current?
I think you're over-thinking this. If you must supply 5A continuously, then choose a device with at least 5A holding current. If you want this same device to also trip at 5.5A, then this device is likely not the best choice.
There are ways to "force" a fuse to open, such as with a "crowbar" circuit. Something like this can be used to short across the power input when some condition is met (such as 5.5A is sensed.)
