Understanding. Apart from the detailed explanation of the specific circuit implementation accompanied by calculations, the real understanding requires something more - to see the basic idea behind it, how it is implemented in the specific case, where currents flow, what voltages across elements are and how they relate to each other. In doing so, we will get to the depths of the problem and not just glide across the surface… and we will be able to see the same basic idea in seemingly different applications. I will illustrate this for the specific circuit.
Basic idea. The main OP's problem here is to understand the basic property of capacitors to keep up the voltage across themselves relatively constant when a small current flows through them. This property has two circuit applications that can be figurative named "voltage shifting" (coupling capacitors) and "voltage fixing" (decoupling capacitors).
"Voltage shifting" is actually the well-known "voltage biasing" or simply "biasing", widely used in AC amplifiers. It is an extremely simple electrical idea that can be implemented even in the 19th century by a constant voltage source in series to the input variable voltage source. Their voltages are summed in a series manner so the input voltage is increased/decreased ("shifted up/down") with the constant voltage.
Implementation. The problem of this arrangement is that the "shifting" voltage source is "floating" (as a rule, the input voltage source is grounded)... and there is no way to put part of the supply voltage in this place. The clever 20th century solution in AC amplifiers was to use a charged capacitor as such a "shifting" voltage source. It acts as a "rechargeable battery" which is constantly being charged and then discharged... and so its voltage is relatively constant.
Operation. I have illustrated the operation of the most elementary bias circuit (the input part of the @jonk's circuit) in three steps. They are extracted from a Flash movie (see the animated gif version). The currents are visualized by full closed paths (loops) with proportional thickness (in green) and the voltages - by voltage bars with proportional height (in red). The current directions and voltage polarities are real. The input voltage EIN (VIN) is represented by a variable battery; the power supply is represented by a battery with a constant voltage E (VCC). The capacitance C1 is large enough so the voltage across it does not significantly change when the current flows through it.
1. Zero input voltage. In the beginning, let's set the input voltage VIN to zero. As a result, the capacitor C1 is charged to VR2 (E/2). The charging current flows through the path +E -> R1 -> C1 -> VIN -> -E (not shown in the picture). So the input voltage source should be "galvanic" (with a low DC resistance)... and you can think of it as of a piece of wire. At the end of the transition, there is no current flowing through it since two equal opposing voltages (VC1 = VR2) balance each other; VC1 is a copy of VR2. Theoretically, this process will never finish.
Now let's consider the original OP's circuit (with capacitor C10) at zero input voltage VIN and applied positive bias voltage V3 to the non-inverting input. The op-amp senses the positive differential input voltage and because the negative feedback obliges it to make the voltage of the inverting input equal to V3 (the so-called "golden rule"), it begins increasing its output voltage until reaches the equilibrium. It should do it momentarily... but there is a problem - the capacitor slowly charges and the voltage across it slowly increases. So, in the beginning, the left end of R5 is grounded. The gain of this non-inverting amplifier is 11... and its output voltage reaches the positive supply rail. Eventually, the capacitor is charged to V3... the gain decreases to 1... the output voltage becomes equal to the voltage of the inverting input and V3... and the equilibrium is reached.
If OP wants the output voltage to be higher, they just needs to follow the advice of @The Photon and to plug another resistor between the inverting input and ground.
Why does the removal of the capacitor cause such a difference?
The "job" of the op-amp "assigned" to it by the negative feedback, is to maintain the voltage of the inverting input equal to the voltage of the non-inverting input. When we put a voltage divider between the output and the inverting input, we make it difficult ("disturbing") to the op-amp. It is forced to increase its output voltage to overcome this "disturbance"... and so it becomes an amplifier.
2. Positive input voltage. Now let's change VIN up. The total voltage VIN + VC1 exceeds VR2 (the voltage bars represent it geometrically). EIN and C1 in series act as a composite voltage source that supplies R2 so a discharging current IC begins flowing through the capacitor and R2. VC1 does not change noticeably so the output voltage VOUT follows VIN up.
In the OP's original circuit, the discharging current flows through the path +VIN -> C10 -> R5 -> R12 -> op-amp output -> ground -> -VIN.
3. Negative input voltage. Now let's change VIN down. The total voltage VC1 - VIN is less than VR2; so a charging current IC begins flowing through the capacitor. The charging current flows through the path +E -> R1 -> C1 -> VIN -> -E. As above, VC1 does not change noticeably so the output voltage VOUT follows VIN down.
In the OP's original circuit, the charging current flows through the path +Vcc -> op-amp output -> R12 -> R5 -> C10 -> VIN -> ground -> -Vcc.
Generalization. Let's generalize our observations. With the help of R1 and R2 (voltage divider), an imperfect voltage source (with a significant internal resistance) is built. The perfect input voltage source (with zero internal resistance) is connected through another perfect voltage source (the charged C1) to the imperfect voltage source. In this unequal struggle, the perfect source wins and imposes its voltage on the common output point.
Analogy. A shock absorber is an amazing mechanical analogy of the capacitor. I have illustrated the operation of the analogical arrangement - a shock absorber and two springs, in the same three steps as above. They are extracted from another Flash movie (see the animated gif version). I think the pictures speak for themselves and need no translation.
1. Zero position.
2. Pulling up.
3. Pulling down.
Applications. We can see this bias circuit in the input part of many AC transistor and op-amp amplifying circuits. Let's consider the two most typical of them:
1. AC emitter follower.
2. Op-amp follower.