... I have never clearly understood what a capacitor does... I never understood its purpose in a circuit.
How to understand it
An interesting paradox is that, in order to understand circuits, it is more important to have an idea of how their constituent components behave than of their internal structure and physical nature. This allows us to explain circuit operation through similar but simpler electrical circuits and non-electrical devices (analogies).
So, here you need simple and clear answers to such questions as, "What does a capacitor do?", "How does it do it?", "Like what does it do it?"... than to know what is inside the capacitor. You simply need an intuitive notion about the capacitor behavior in circuits.
How to explain it
In my story, I will try to reveal the "philosophy" of this element without using special terms and formulas which, at this stage of intuitive understanding, do not help much... and in some moments even hinder understanding. Instead, I will use descriptive and figurative names that make associations with familiar phenomena of our daily lives. I intentionally did not put illustrations to save space but if you want, I will insert them.
Basically, the capacitor is a container of potential energy like a "rechargeable battery" used as a simpler electrical analogy. It can be also thought of by a simple non-electrical analogy as a tensioned spring... or a car tire full of compressed air... or a vessel full of water... or any container that stores some kind of potential energy. Simply put, it is a tank full of "something".
The basic capacitor storing property is used in various circuit applications. Like other electrical elements, the capacitor can be used both in circuits that process signals and circuits that process energy (in some cases, there is no clear boundary between the two types). Here are the most popular of them.
In most electronic circuits, the electrical quantities voltage and current represent signals. For a number of reasons, voltage is the more convenient signal carrier; current is mainly used as an intermediate carrier inside circuits.
Constant voltage. Some applications use the capacitor property not to change considerably its voltage when its capacitance is large enough or the current through it is small enough. Let's consider two famous applications.
"Shifted" voltage (coupling capacitors). If we connect a constant (DC) voltage source in series to a varying (AC) voltage source, its constant voltage will be added/subtracted to/from the varying voltage according to KVL. Figuratively speaking, the varying voltage will be "shifted up/down" with the value of the constant voltage. This technique is known as "biasing" and is widely used in AC amplifiers to make transistors work even when the input voltage is still zero (not until it becomes 0.6 V).
The problem of this arrangement is that the "shifting" voltage source is "floating" (not connected to ground); so its voltage cannot be taken from the power supply. An elegant solution is to implement it by a charged capacitor. So, the so-called "coupling capacitor" serves as a floating voltage source connected in series to a varying voltage source with the purpose to transfer its voltage variations. For example, in a common-base stage (Fig. 2, 3 and 4), the input coupling capacitor Ce "shifts up" the input voltage variations and the output coupling capacitor Cc "shifts down" the output (collector) voltage variations. As a result, both input and output voltages "wiggle" around ground while, inside the stage, they are "lifted".
"Fixed" voltage (decoupling capacitors). In the applications above, both capacitor terminals simultaneously "move". If we fix one of them (eg, ground it), the other will stay "immovable"... ie, it will have a constant voltage. So, if such a "decoupling capacitor" is connected in parallel to another "imperfect" element with some resistance, the voltage across it will be fixed (the capacitor is charged to the source voltage and "kills" its voltage variations). For example, in the AC common-emitter stage with emitter degeneration, a "bypass capacitor" is connected in parallel to the emitter resistor thus "fixing" the emitter voltage.
So, both coupling and decoupling capacitors are the same charged capacitor acting as a constant voltage source. But in the first case it is connected in series while in the second - in parallel to another voltage source. And both coupling and blocking capacitors do the same - they keep the voltage across themselves constant. Only, coupling capacitors transfer the voltage variations while decoupling capacitors "kill" them. Note that these are AC applications because the voltage across the capacitor must be kept constant by alternating equivalent charges and discharges. See more about these capacitor applications in another related answer.
A shock absorber is a very good mechanical analogy of these applications. It transfers the movement of its one end to the other (like a coupling capacitor)... and it blocks the movements of its ends relative to each other (like a decoupling capacitor).
Memory. The capacitor property to retain its charge (voltage) for some time is used to make memory cells. An analog example is the "sample & hold circuit" and a digital one is the DRAM cell.
Varying voltage. Other applications use the opposite capacitor property to change considerably its voltage when its capacitance is small enough or the current through it is large enough (in the applications above, the voltage across the capacitor also changes but these variations are so negligible that we ignore them). Let's consider two famous applications.
Integrator. Everyone has observed how when a vessel (capacitor) is filled with a constant flow (current), the water level (voltage) increases linearly over time. Typical examples of such an application are the antique water clock and sand clock. So, you already know how the humble capacitor can act as a simple electrical integrator with a current input. Only add a resistor in series to convert the input voltage to current and you will obtain the famous RC integrating circuit.
However, a problem appears - the voltage across the capacitor is subtracted from the input voltage and the current gradually decreases. Like a resistor, the capacitor "steals" a voltage from the input voltage source... but in contrast, it accumulates this voltage instead of spending it right away. So, this voltage is still a loss and you can compensate it by an extremely simple trick - connect a variable voltage source in series to the capacitor and with the same polarity as the input voltage source (travelling the loop)... and make its voltage equal to the voltage drop across the capacitor. As a result, the voltage drop will be neutralized and the current will be as though there was no capacitor connected.
This is the idea of your op-amp circuit known as op-amp inverting integrator. Basically, it consists of three elements in series - the input voltage source, the capacitor and the op-amp output. The op-amp produces voltage that is equal to the voltage drop across the capacitor. Since its output is connected in series (via the ground) and in the same direction to the input voltage source, its voltage is added to the input voltage... and the undesired voltage drop across the capacitor is compensated by its mirror copy (used as an output voltage). See more about the famous circuit in another related question, RG question and Wikibooks story. This idea can be generalized as a universal voltage compensation principle.
Differentiator. In the integrator circuit above, we "carefully" changed the voltage of the capacitor through a resistor. If we decide to change it "brutally" by applying the input voltage directly to the capacitor (without a resistor), the current will vigorously change. Now the capacitor will act as a differentiator with a current output. Since we need a voltage output, we connect again a resistor now acting as a current-to-voltage converter. Of course, it introduces an error... but we can compensate it with the same trick as above.
Voltage opposition (RC filters). The voltage across the capacitor was undesired in the integrating circuit above; that is why we compensated it by additional voltage. However, in AC circuits, it can be useful when decreasing the input voltage. Just connect a capacitor in series to a resistor and you will obtain the well-known RC high-pass and low-pass filtering networks. Its voltage will be subtracted from the input AC voltage and the current through and the voltage across the resistor will be reduced depending on the frequency. Why? Here is an intuitive explanation of this phenomenon.
As it charges, the capacitor tries to reach the voltage of the input AC source to neutralize it... but the input source does not stop changing its voltage. Figuratively speaking, the input source "runs" and the capacitor tries to reach it. The higher the frequency, the slower the capacitor "moves"... the difference increases and the opposition decreases. Shortly, the higher the frequency, the lower the "opposition".
This capacitor property resembles the resistor property of "resistance" and is known as capacitor "reactance" (impedance). As you can see, there is nothing supernatural in it and it can be explained in a simple way: The capacitor reduces the current by subtracting its voltage from the input voltage while a resistor in its place would reduce the current by adding its resistance.
In these circuits, capacitors are used to process power.
Voltage loss. As above, the voltage drop across the capaciror connected in series to a load in AC power circuits can be useful when decreasing the AC supply voltage in transformless power supplies. Connect a capacitor in series to the load and its voltage will be subtracted from the input AC voltage; the current through and the voltage across the load will be reduced.
Backup voltage source. Providing a backup source of energy is a well-known idea in life. Here are some examples where the capacitor acts as such a source.
Rectifier capacitor. A typical example is the half wave diode rectifier where a capacitor supplies the load between two positive half waves. Other examples are voltage multipliers.
Bypass capacitors. They are connected in parallel to integrated circuits to keep their supply voltage constant when spikes occur (they are the same decoupling capacitors as above).
Pulse discharge. A charged capacitor can release a colossal amount of energy when discharged in a short time. A typical example is the photo flash. See also this movie where my students discharge a large electrolytic capacitor through a small motor. The trick here is that we continuously charge the capacitor with a relatively low current and discharge it quickly with high current.
Supercapacitor. This idea is used to power electric buses in public transport (for example, in my city there are such).
This was my story about the famous capacitor as seen from my point of view... I wrote it more as a "fairy tale" than as a "serious" technical description because it is meant to be read by human beings with imagination, intuition and common sense.
Well, it turned out to be quite long... but that is the price of understanding; the other is just knowledge.