I have a relay switch that operates at 10 A and 120 V connected to a DC motor that runs at 18 A and 36 V. Will the relay switch burn out? If so, how can I protect the relay switch?
You need a relay that is rated to work at the current that you are running it at. Additionally you would want an inrush current limiter to prevent the inrush of current when the relay makes contact from destroying the relay.
Will the relay switch burn out?
Yes. There is no good way to run 18A of current through a device that is only rated for 10A. You should choose a relay so your load current does not exceed your current rating.
If so, how can I protect the relay switch?
You could use additional relays in parallel to increase the current capacity. Ex: with two 10A relays in parallel, you could theoretically run up to 20A through the pair.
It's important to note that there may be a difference between recommended operating conditions and absolute max conditions. If the absolute max rating of the relay is 10A, then you need to determine if 10% safety margin is sufficient for your application.
Yes, you'll likely burn up that relay.
There are plenty of solid state relays that can handle north of 20A with a heat-sink - I would look into those, or use a simple amplifier to drive a larger relay with your Arduino (although I'd recommend solid state).
Properly sizing the relay in the first place is likely your best bet, as trying to run relays in parallel can cause unexpected issues if one fails to operate.
Especially since we're talking about controlling motors, be advised that the larger the motor the worse the back EMF is. You really want a snubber/ flyback diode across the motor leads to dissipate that energy and protect the relay from arcing.
In most cases you never want to go to motor current limits, except some stall cases.
Mostly even large engines run with much smaller current than max rated current, yours is probably around 3-4 A in normal running, starting could be more.
Your relays should be fine for the case, some relays have smaller low voltage DC current rated, so check it up for your relays, but mostly it stays the same.
Fuses are the most common way to protect against over-current.
For engines so called slow-blow fuses are used, because of starting spike in current. Car fuses can be used as they are cheap and easy to access (every gas station), they also sell connectors to cables for these and you could solder them in your circuit.
Nowadays most schematics use resetable fuses, so called PPTC.
The oldest way of limiting current is current limiting resistors and as wires have resistance, you don't want them to become fuses. It may cause smoke and fire.
Whichever way of fusing or protecting you use, your fuses have to be rated for smaller currents than your cables.