The first thing I verify on a new board, whether it is using an internal oscillator or an external crystal, is that I have the clock frequency set up correctly. This is important because many of the peripherals, such as UART, SPI, I2C and timers depend on it.
The way I verify it is to write a program with a short loop, either in assembly language where I can count the cycles manually, or C as long as you can get a disassembly listing and do the same thing -- and turn an LED on and off. I set up a loop so it executes once a second. I run the code, and check that the LED blinks 60 times in a minute.
As far as peripherals go, the best way to check them is to use an oscilloscope if you have one, and look at the RX line for UART, the CLK, MOSI, and chip select lines for SPI, and the SDA and SCL lines for I2C, and check that the lines are toggling and the timing looks correct.
If you don't have an oscilloscope, you can put LEDs on these lines, and then enable or disable the peripherals, When disabled, most of the lines will be low (LED off), but some will be high, like the RX lead of the UART (LED on). When the peripheral is enabled, most the LEDs should dim, since the lines will be toggling. By running in a loop (disabled/enabled) it is easier to see the difference between on or dim.
For the UART, you can connect the TX line to the RX line as a loop around. You can also connect then to a UART to USB cable, and on the PC real a terminal a program like RealTerm. Besides testing out the interface, this will come in handy for other debugging later.
For other pieces of code, I use multiple LEDs as necessary to show that various paths in the code are being executed. If you have the UART working and connected to a PC, you can sprinkle your code with calls to a subroutine to output a message to show what points the program has reached (or use printf if you have the standard C libraries available). But as Vladimir Cravero points out in a comment below, this can slow your code down some (at 115,200 baud, not too much, since one character time is < 10 µs). But in ISRs and other time critical code, just use LEDs.