Your exponential decay is from another case. That case has a resistor in series with a capacitor.
Your LC circuit, if lossless, starts to oscillate. The current is sinusoidal and the voltages over the L and C are both sinusoidal, too. The voltage over the C swings between 0 and 2E.
In circuit theory you have 2 state variable differential equations, one for the inductor current and one for the capacitor voltage. They can be used "as is" for numerical simulation.
Those equations reduce to one 2nd order differential equation for analytical solution. Refer nearly any introductory circuit analysis textbook. (level = undergraduate academic).
Or see this: https://en.wikipedia.org/wiki/LC_circuit
ADDENDUM: due the comment
You're right when thinking the C to be empty in the beginning and the current to start grow gradually due the inductance. The current reaches its maximum when the C has voltage = E. The current starts to decrease, but it still charges the C until it has voltage 2E.
The state variable equations can be expressed as words. They are the most basic circuit laws since Ohm's and Kirchoff's laws an the calculation of the power. The equations:
Inductor current grows at rate (Amperes per second) = the voltage over the inductor divided by the inductance
Capacitor voltage grows at rate (Volts/second) = the charging ciurrent divided by the capacitance.
From these you should notice that the inductor keeps up the charging well over the battery voltage.
The charging current decreases (=negative voltage over the L) when the capacitor voltage has reached and bypassed E. At 2E the inductor current has dropped to zero; no more charging. Due the negative inductor voltage the current starts to grow to reversed direction ie. the capacitor discharges. When the capacitor voltage is zero, one full oscillation cycle is done and the next cycle starts.
See te numerical simulation example

simulate this circuit – Schematic created using CircuitLab

In practice the oscillation dies away due the resistance and in high frequencies also due the radiowave radiation. Only superconductive circuits can retain the oscillation long time, but the circuit can't have a battery.
In LC oscillator circuits (for ex. in radios) the oscillation can be continuous, because the losses are compensated by amplifiers.