You could use a PUT or DIAC, the modern rough equivalents to the UJT and neon bulb.
PUTs themselves are not particularly modern anymore; they're only available from legacy manufacturers, by the looks of it (e.g. a 2N6028 by Central Semiconductor for $2.50 in singles). They can be synthesized using a complementary pair of BJTs, but admittedly, that may lack the elegance of "simplest". (PUTs are indeed single monolithic devices; they're multilayer semiconductors like BJTs are, but, well, one layer extra, no circuit integration or anything like that.)
That leaves DIACs, basically the same thing but having only two terminals, which switch on suddenly above a trigger voltage, then remain conducting until current falls below the holding threshold. An oscillator uses a large-value pull-up resistor (such that triggering current is crossed, but below holding current), and a parallel capacitor. The advantage over neon bulbs is better availability (neons are hardly manufactured anymore(?)) and lower operating voltage; a downside is no flash of light for visual interest (but, a series LED can be added to make up for it).
The BJT avalanche oscillator is probably the lowest-voltage two-terminal relaxation oscillator, aside from those using a transformer.
If radio frequency is allowed, then it's hard to get simpler than an RF transistor strung up by loose wires and biased -- most likely, a random arrangement of wires will give enough reactance and coupling to make it oscillate, as is detectable with an RF probe. A battery, transistor, several wires, and a resistor, will suffice. A typical (DC wired) circuit might be, collector and base wired together (with sufficient lead length), emitter to resistor to battery to collector. The resistor sets emitter current, while the B-C circuit oscillates as a Hartley or Colpitts style oscillator, more or less. Probably position the resistor by the battery, to give more lead length around C and E.
If low frequencies are preferred, and a transformer is acceptable, then the blocking oscillator is hard to beat:
Source: my site, https://www.seventransistorlabs.com/tmoranwms/Circuits_2010/Blocking_Oscillator.png
Rb is usually not needed, and Rbias and Cbb can be omitted in a pinch (though not both of Rb and Rbias!), though their presence allows the relaxation-oscillator-ness to be exaggerated. I have an example of this built as a relatively high-power "Joule Thief",
Source: my site, https://www.seventransistorlabs.com/Images/JouleThief3.jpg
which I can indeed cause to blink at some Hz by simply touching my fingers across the switch terminals (an equivalent Rbias of some megaohms). A transistor of enough gain, and the transformer's characteristics, are such that the oscillator kicks over from such minuscule bias current; the resulting pulse is only a few microseconds long, but the LED is driven at a couple amperes, enough to light quite visibly.