I've done some research and tried to implement a resistor and relay combination to limit current inrush for my power supply. How can this be improved as I am still seeing a peak of 85A at the smoothing caps?
Relays by nature have a delay between energization and the contact. Your circuit will see high momentary peak current during startup over this interval, however small.
I am assuming you are OK with the dc circuit being OFF until the transformer stabilizes, but I wouldn’t know; you have posted very little about your required thresholds... If so, get rid of your relay and use some Power MOSFETs instead. Scratch that 10 ohm resistor too if it doesn’t symbolize the transformer’s winding resistance, which it likely doesn’t, as there is only one. Small resistors on AC lines is a bad idea. 120^2/10 = huge power dissipation during operation.
Are your capacitors meant to smooth out the rectified supply? If so they should be MUCH, MUCH bigger. Watch this: https://m.youtube.com/watch?v=sI5Ftm1-jik
Regardless, use a high-side P-Channel Fet to keep the dc section practically open circuited during startup of the transformer but not disturb the ground reference. Use appropriate zenor diodes and pull-ups. Use an RC delay circuit to turn the Fet on after the startup phase has died away. If you are not familiar with Power MOSFETs, do some research, they are your friends in power circuits. For RC delay circuit see here: https://www.build-electronic-circuits.com/rc-delay-element/
Feed the output into a digital follower. You can make one using an AND gate and tying the poles together. Then use the time constant to calculate how long it will take to reach the VIH of your follower. You will likely want to use a logic level MOSFET to switch the Pfet gate. Your requirements could be a pain as you will need to supply a voltage your AND follower can operate at. Maybe try building a higher voltage follower from discrete components? I have never done this, and it seems messy. Would not advise...