RTC chips are used in an embedded system whenever the date and time are needed to be displayed (e.g. on an LCD), or data needs to be timestamped before it is logged to an SD card or sent via a USB cable or wirelessly to another device.
A RTC is often packaged in a separate chip so that it can be powered by its own battery, such as a 2032 coin cell. This allows the main circuitry on the board, including the microcontroller to be completely powered down, and the RTC will keep running.
The communication between the microcontroller and the RTC chip will generally be over a serial interface such as I²C or SPI. Typically the RTC chip will have a number of registers, each with their own address, which can be written to or read from the microcontroller. These registers would include the date and time, usually one field per register (month, day, year, hour, minute, second and sometimes weekday).
RTC chips will usually take care of leap years automatically (including February 29 as necessary). Some also adjust for daylight savings time. Years ago, this was done with a fixed algorithm, but that became a problem when Congress started messing with the dates. Now chips that provide DST corrections also allow the microcontroller to set the ahead and behind dates.
The microcontroller would first set these registers over the serial bus, perhaps getting the date and time from the user. From then on, when it needed the date or time, it would read the registers over the serial bus.
Some RTC chips have a ability to set alarms, and then interrupt the microcontroller when the alarm goes off. These are sometimes referred to as RTCC (Real-time clock/calendar) chips.
Another feature offered by RTC chips is a scratchpad area (a few hundred bytes of RAM or EEPROM) where the microcontroller can save values that would otherwise be lost when the microcontroller lost power.
Instead of using a separate chip, many microcontrollers have an RTC built into them. For this reason, they often require two crystals -- the main microcontroller crystal, typically 4 or 8 MHz, and a secondary 32 kHz crystal for the RTC. The microcontroller would then have some sort of "sleep" mode where the processor is stopped, but the RTC continues to run, using very little current (a few µA).
As mentioned in another answer, the 32 kHz crystal is used because it is convenient to divide down, the circuitry uses less power than higher frequency crystals, and it is very cheap since they are used in watches.