Your transformer should be bolted to the chassis of your project and connect ground from our mains to the chassis. I you are using a plastic project box, it will likely be best to connect to a bolt you are using to mount the transformer to the plastic box. That way, if you ever fry your transformer and you develop a short to the transformer chassis, you'll immediately pop your circuit breaker (and hopefully the fuse in your project).
Yes, your transformer is fine, the two mains wires connect to the two red wires of your transformer. The three wire bundle will become your input to the power supply - two blue to the bridge rectifier and yellow is the circuit "ground". I would NOT connect circuit ground of your project to your Mains Ground until you run some tests and make sure everything is ok. After that, it really depends what you plan t use it for whether you want to connect to ground. F you are just learning to use an oscilloscope, I would not connect to ground until you understand how your scope works and the ground reference of your scope.
You never want to conduct line frequency rectifier current through PE ground as this adds conductive noise. Using the transformer for high impedance and insulation from high voltage transients gives the audio better common mode noise rejection ratios (CMRR).
Grounding is used for noise reduction when it stays at the same voltage by not forcing noise current thru it but rather providing a low impedance reference to high impedance stray noise ingress thus attenuation occurs in shielded audio lines or low impedance speaker drivers.
There are occasions where RF can create audio noise in preamps so a capacitor can suppress this to the audio power 0V whether that is PE grounded or not.
Thus when using PE ground for computer audio, realize the PC power supply uses CM chokes and lots of filtering so that there is minimal audio band noise current on PE ground. Expect all peripherals to be floating and then rely on the host only for PE ground to suppress radiated noise thru the network or shielded cables.
When multiple source and destinations use PE grounds, you can get the "ground loop" noise due to the difference in ground voltage and current that caused that.
The term "ground" simply means 0V locally whereever you define that reference point.
Safety or Protective Earth (PE) ground is 0V in the earth and close enough elsewhere to be safe. So with interconnections you don't want to add noise current when audio is interconnected. The current should only be the intended audio.
If you want your audio circuits ground referenced- you probably do- the centre tap of the secondary (output side) connects to Earth, as does your safety Earth in the mains lead. The primary goes across Live and Neutral.
I've just noticed your capacitors C9 and C10 are shorted to ground!
edit following the discussions, I would suggest a "bog standard" linear power supply design, this one turns up first on my search and is the kind of thing
It's a fixed 15-0-15. For 12-0-12 just use 7812/7912 regulators. They're dirt cheap and have automatic output shorting protection, which is good working with experimental circuits.