I have already done some similar examples and I will usually start with defining next-state logic, than output logic and register data path.
With this example I have trouble with defining next-state logic. If I make table of state transitions, I need to take also start, a and b in consideration. Since we have 4 states we need 4 bits for that + one bit for start, one for b!=0 and one for a > b. That is simply too much combinations to write on paper. I suppose that maybe there is simpler solution for this.
What does the ready A state do? I'm confused from the diagram.