After it is what is does to solve the circuit, right?
That is incorrect.
No simulator that I'm aware of solves circuit model problems by understanding anything about admittance matrices (or S parameters etc. for example). Simulators use entirely different algorithms altogether.
I am wondering if there is a way to extract the Nodal admittance matrix of a very complicated circuit using LTspice.
No, that isn't likely to be possible. Of course, you can take the AC analysis results and fit them to a formula involving s and that might be of some use (trial and error needed). Then you are a little closer to finding the matrices.
Here's a Q and A that might help you understand how simulators work. Simple extract: -
The general equations produced are nonlinear differential algebraic equations which are solved using implicit integration methods, Newton's method and sparse matrix techniques.