Electrical Engineering Stack Exchange is a question and answer site for electronics and electrical engineering professionals, students, and enthusiasts. It only takes a minute to sign up.
Sign up to join this community
There are varies activation functions in neural networks such as reLU, sigmoid and etc. I figured out that reLU function can be generated using simple diode circuit. What are the other activation functions that can be generated using basic electronic items such as op-ams, bjts, diodes, filters and etc? If there are any, how to do it?
These are some of the activation functions.
I'm not sure how useful any of this will be, because all these suggestions only approximate those functions, to varying degrees, but here goes.
The simplest \$tanh\$ implementation I can think of, is a couple of anti-parallel diodes, that clamp the output close to 0V with an exponential "twist". Because the clamp voltage is 0.6V or so, optionally you can scale that output to ±1V or so with a simple non-inverting amplifier:
simulate this circuit – Schematic created using CircuitLab
For better control of the steepness of the central slope, you can move those diodes into the feedback loop of an inverting amplifier configuration, but use an additional resistor (R2 in the circuit below) to determine gain prior to the diodes taking over. The plots are with R2 set to 1kΩ (blue), 2kΩ (orange), and 5kΩ (tan):
Of course, you can invert and scale any signal with an inverting amplifier stage. Optionally, you can use that configuration as a summing amplifier, and offset the signal up or down. Here I take the output of the previous circuit (blue below), apply a gain of −0.9, and shift the whole thing upwards, to be centered around +0.5V (orange below), to obtain something approximating the sigmoid:
The ELU function is trivially simple to approximate. A diode clamp consisting of however many diodes you need for the negative limit. Each diode contributes −0.6V:
The ReLU function is exactly what you get from an active rectifier, but take care to take the output from the correct point:
The name "Leaky" ReLU is exactly what it say, all we need to do is add a leak (a current path from IN to OUT, via R1 below) to the active rectifier:
The "Maxout" function I am not sure about. I'll give it some thought.
The primary purpose of an activation function in an artificial neural network (ANN) is to create a non-linearity between layers. A purely linear combination of layers of weights and values can be mathematically reduced to a single layer (i.e. a single multiply-accumulate function -- the proof of this is simple enough and "left as an exercise for the reader") and thus without an intentional non-linear function it loses all the complexity that the multi-layer topology can provide. The activation function forces in a non-linearity, giving the neural network actual depth.
The only requirement for a usable activation function is that it be differentiable, so that back-propagation can be performed in training. This is partly why the "rectified linear unit" (ReLU) is so popular -- the slope is either 1 if x>0 or 0 otherwise (with the x=0 case arbitrarily assigned to zero) so it's really easy and fast to use in practice.
With that in mind, in theory any non-linear electrical device can compute an activation function. But none of them will be an exact match to the mathematical functions you've asked about. A transistor moving from linear to saturation region will look a lot like a tanh or sigmoid, but is not identical. A diode will not conduct at 0 V but rather at somewhere between 0.3 and 0.7 V, making it an imperfect ReLU. A comparator can be used to calculate which signal is the maximum, and that can be fed into a multiplexer to pass the actual signal through, making a maxout, but the multiplexer will drop some voltage and the comparator will only work in its input range. All real world devices have their limits. Even the humble resistor is non-linear if you push it out of its happy range. No electronic device you can design will ever produce a mathematically "pure" output.
Computing feed-forward in analog is the easy part, though. Training the ANN is the real challenge. To train your circuit, you will need to build circuitry which computes the derivative of each layer including its activation function, and which can update the multiplier weights on the fly. But if you can do that, then any non-linear electrical device will theoretically work as an activation function.
Log / antilog type functions can be constructed from op-amps plus junction devices including transistors and diodes. They exploit the exponential I vs. V characteristics of their ‘knee’ area. (Search ‘logarithmic amplifiers’ for examples.)
More here: Sigmoid using op-amps
I think a sigmoid could also be approximated by using a FET pair, as suggested here:
That said, making an arbitrary analog activation function seems to have a lot in common with gamma correction used for LCD panels. The LCD column drivers use an array of DACs to control a piecewise-linear mapping to implement a panel-optimized gamma transfer curve.
This same PWL approach could approximate the tanh, sigmoid or just about any other continuous curve function, not to mention ReLU and others.
More here: https://www.intechopen.com/chapters/11273
This paper is from 2009; the PWL + DAC driver technique is commonplace in panels now.
