I have a circuit that generates random noise which I've measured with a 10 bit ADC. The following is the sample distribution:-

sample histogram 2.5V

You will notice that it's a log normal distribution, characteristic of a Zener diode in avalanche breakdown. The maximum reading can be 1023 analogue units. You will also notice that not all of the horizontal scale is utilised as the maximum reference voltage for the ADC was 2.5V. So ADC reading 0 = 0V and ADC reading 1023 = 2.5V. I can alter this reference voltage and as I do the histogram widens and narrows proportionately. The second histogram is for a reference voltage of 1.1V. You can see that approximately 14,000 samples were at a value of 1023 or above. Both histograms show 10 million samples each. (I think that there might be weirdness with the gnuplot tally).

sample histogram 1.1V

As the samples are taken, they are effectively a source of random entropy. This leads to a Shannon entropy rate per sample, in say bits /sample. So for example at Vref = 2.5V, entropy was generated at 0.98 bits/ sample. In order to maximise efficiency of this entropy generator, I wish to maximise the entropy rate it produces by altering the reference voltage.

Following a comment: You will see that in the first histogram, one standard deviation of results (68% - the majority) is spread over perhaps 100 analogue units. In the second, it's spread over maybe 250. That means more entropy, but some readings are clipped at 1023. I think that there is a sweet spot whereby the distribution can be scaled (by reducing Vref) to maximise entropy, which will then fall thereafter as Vref decreases towards 0V.

Q. What reference voltage will maximise the entropy generation rate in bits /sample?

Note: I am not asking how to build a random number generator. I am not asking how to build a random number generator. I write it twice so that answers do not consist of brain dumps on how to build random number generators, debiasing or that a pseudo random number generator would be better. I am looking for a maximisation of Shannon's information entropy formula (or min. entropy which would be better for cryptographic purposes) specific to this question. No commercial TRNG creates it's final output from hardware alone. All use software processing and randomness extraction in distribution whitening phases. I am asking about the hardware phase to maximise entropy. It is important to distinguish between entropy and uniformly distributed random numbers. They are not the same. Why am I having to state this? This is the 3rd time I've asked this question across 3 different forums and not received a relevant answer. This is a mathematics and voltage question to which the answer will consist of just one singular number of volts. Could it be that multi-disciplinary questions are unsuitable for SE?

  • 2
    \$\begingroup\$ @TonyStewart.EEsince'75 This is actually an infinite entropy solution as entropy is generated by the observer not the observed. As I tried to explain, the entropy rate is related to Vref. I was hoping somebody knew enough maths to adjust the distribution to maximise entropy :-( \$\endgroup\$
    – Paul Uszak
    Mar 1 '17 at 13:23
  • 1
    \$\begingroup\$ @laptop2d How do you define a "better" ADC and how would that help? \$\endgroup\$
    – Paul Uszak
    Mar 1 '17 at 13:26
  • 1
    \$\begingroup\$ It depends on your circuit design,. Which is skewed like you rectified the noise instead of DC center biased to Vref/2. The distribution will be Gaussian and will not be random distribution histogram which is what you want. We cant fix what you dont show. Your criteria for Entropy rate does not seem to match a perfect distribution of random values and shows considerable bias. Not only must the values be random but the intervals between values shown as spectral density by FFT. \$\endgroup\$ Mar 1 '17 at 15:23
  • 2
    \$\begingroup\$ @TonyStewart.EEsince'75 I suspect that this is an example of a TH type question (too hard). \$\endgroup\$
    – Paul Uszak
    Mar 5 '17 at 23:13
  • 2
    \$\begingroup\$ Why wouldn't you just parametrize your entropy, take the derivative with respect to Vref, set it equal to zero, and get your maximum? In other words, roll up your sleeves and handle an optimization problem like an optimization problem? \$\endgroup\$ Mar 13 '17 at 13:53

To answer your original question, you can easily find the optimum value numerically using Monte-Carlo simulation. Generate a sample of your distribution clipped at 1023 for different Vref values, and use an appropriate optimization algorithm (I'd recommend Golden-section search for its robustness) to discover which value of Vref gives you the maximum entropy.

However, I believe you can get even better results if you improve your signal at its source instead of being stuck with the log normal distribution you got. I'm too lazy to do an in-depth analysis for you, but it looks that you can get a much better distribution if you run your signal through a log amp, perhaps using an offset to discard low voltage samples:


simulate this circuit – Schematic created using CircuitLab

  • \$\begingroup\$ I very really really meant it didn't want to absolutely show a circuit whatever because I knew that people would simply jump on that and shift focus away from my specific question. This is why it has been so difficult in getting an answer that didn't involve a soldering gun. No matter what happens circuitry wise, holistic entropy maximisation in my system still requires optimisation within the ADC component and software. Your suggested simulation is a good one. \$\endgroup\$
    – Paul Uszak
    Mar 13 '17 at 13:20

New answer

Q. What reference voltage will maximise the entropy generation rate in bits /sample

For a log Normal distribution that prevents clipping , this depends on the multiplier of sigma, the mean value and Vref.

But the Maximum Entropy Constraint for Lognormal = σ ^2 but I am not sure if that excludes the X bar mean value, to obtain the max value of 1023 for Vin= Vref or even how to use this Contraint, but that should compute Vref..

another Approach:

For a Normal Distribution 3.3 σ or 99.9% is equivalent to 1023/1024 values using 10 linear ADC bits.

So in plot (1) with Xbar=100 and σ ~ 100 @ 2.5V , Xbar+ 3.3 σ should be 100+ 330 units = 430 units a ratio of to 1024 ~ 430/1024=0.42x Vref (optimum) = 0.42*2.5V = 1.05V = Vref which is close to your plot (2).

This given linear number of 10 bits of resolution gives a 1024 or 60dB range. max entropy lognormal. Wiki table

yet another approach...

In plot (2) , about 0.1% of the data was clipped or 3 σ was not clipped but 2 σ wasted 2/3 of the quantization levels for about 1/3 of the data samples.

A 30 dB improvement in dynamic range or a larger 5 σ can be achieved using a common compression ADC for greater units per quantization bits using u Law compression with 8 bits used by all North American telephony.


For more theoretical analysis on skew of distribution, Rayleigh distribution , Differential Entropy and compression algorithms enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.