When turning to an RTOS, we are usually dealing with an application that has many concurrent tasks that need to be scheduled optimally in order for each of them to meet their deadlines on time or share resources safely. The RTOS framework that you choose implements a task scheduler, and your job (typically) is to write these individual tasks with a certain set of properties (period, priority, etc) and then hand it off to the scheduler. So for documentation, the approach I would take would be to document each task carefully.
Most embedded software and, as far as I know, most RTOS's are not written in an object oriented language and thus may not benefit from a lot of things that are geared towards that like class diagrams for example.
When documenting your RTOS tasks however, any diagram which describes the task well would be a great benefit. I would imagine a sequence diagram for each task could be very helpful for example. Along with that you could specify its hard requirements like its period/frequency, priority, any shared resources it may use, pre-emption requirements, etc. Also of value could be to document how you've configured the RTOS and perhaps a state-machine of its scheduling algorithm.
Take any of this advice however you like, I haven't messed with RTOS stuff since my college days, and never really "documented" the work.