The device driver itself works by ultimately by just writing to the UART FIFO and control registers.
The main reason why you typically would want to use the driver rather than writing to the UART directly is because the driver gives you a layer of abstraction. It lets you treat that UART like other UARTs, and even sometimes like any other file. You do not need to know which bit is the Data Ready Bit and what the divisor registers are for the UART you happened to be using since the driver makes all UARTs look pretty much the same. Additionally, the driver typically gives you a nice big managed buffer on top of the FIFOs and handles all the interrupts, which requires writing some specialized code under Linux.
That said, there are times when writing directly to the UART can give you control that is not available though the driver. Here is a project where I skip the Linux device driver on a Raspberry PI and go directly to the UART to be able to generate signals for a WS2812B...
This would not be possible to do via the driver because it depends on some quirks of the specific UART, along with some state transitions that there is no way to reliably get the driver to do in the right order.
In cases where you do decide to write directly to the UART, it is probably a good idea to disable the driver to avoid conflicts. For example, if you are trying to read bytes out of the FIFO via your userspace program, it is likely that the driver will also be reading bytes and will likely win because it gets gets triggered off of interrupts. Similarly, you might change some register directly while the driver assumes that register will stay set to whatever it initialized it to (or vice versa).