In this diagram:
What does the S[2:0] mean? I think it is the state but why is the state in 3 bits? What does the 2 and 0 mean?
It means the signal is a 3-bit-wide bus.
[2:0] is a notation borrowed from Verilog. It means that the signal is a bus containing multiple bits, numbered from 2 down to 0 -- three bits in total.
S[2:0] refers to three signal lines (S2, S1, S0) that are used together to determine the state.
There can be eight states, depending on the states of those three bits.
More context (where this drawing came from, and what is around it) would be required to say what really happens here, and what those eight states do or mean.