This is a very open-ended question, and will be difficult to answer properly in the SE format. However, I just went through an exercise like this myself recently, so I'll throw out a few thoughts.
Definitely take a top-down, outside-in approach. Make sure you know what the core is supposed to do and how the external interfaces are supposed to behave in a fair amount of detail. Think a bit about how you'd approach implementing it. Then, look at the implementation of the top-level module and see how it's broken up (hopefully :-) into functional sub-modules that make sense.
Keep going, making notes, drawing block diagrams — and yes, state diagrams where applicable — as you go. When looking at the interfaces among sub-modules, there will likely be little documentation, so you're going to have to reverse-engineer many of the details from the implementations inside the submodules. Hopefully, there will be consistent patterns that get used in multiple places.
A simulator can be invaluable for picking apart details, which is another reason you should have a good handle on the external interfaces. This allows you to write a meaningful testbench to exercise the module (assuming one doesn't already exist).