I have a PCB board with ADSP-BF518, I want to use its jtag pins. (It is not an evaluation board) there are some VIAs on the board that comes from its jtag pins. but by using Continuity Test I couldn't find any header that was connected to these VIAs. Is there any special points here that make me couldn't find headers of VIAs? or Is it possible these VIAs don't connect to any header?
sorry I can not send an image from the whole of the board
As you "can not" do that (not allowed?) then you will have to do more investigation on your own.
there are some VIAs on the board that comes from its jtag pins
I have not used Blackfin MCUs. Assuming that these JTAG pins are dedicated to that function, and are not multi-purpose and switched to other functionality (e.g. GPIO) then having PCB tracks (with Vias) to those JTAG pins, means those tracks lead somewhere. You will need to reverse-engineer the board and discover where those tracks go.
Is there any special points here that make me couldn't find headers of VIAs? or Is it possible these VIAs don't connect to any header?
Yes, it is possible that the JTAG signals don't go to a header.
Assuming that the JTAG signals are used only for programming during manufacturing (and that might be an incorrect assumption) then the connections to those JTAG signals don't need to be made via a header. In fact it can be easier (quicker) for programming, if they don't use a header.
Instead the JTAG connections could be made using automated "bed of nails" or "flying probes" or other techniques. This could explain why you can't find a JTAG header - there doesn't have to be one! Instead there could be test points for the JTAG signals spread around the PCB, for example, for temporary connections during manufacturing.
As an example, I had to reverse-engineer a board recently to do exactly this - breakout the JTAG signals to a standard header, using Vias as the connection points for the signals, since there was no JTAG header provided on the board for the main MCU which I wanted to re-program.
A sensible "next step" for you, would be to follow the JTAG signals on the PCB, without expecting there to be a header where all the JTAG signals are connected.