# What are the activation functions that can be generated using op-amps and filters?

There are varies activation functions in neural networks such as reLU, sigmoid and etc. I figured out that reLU function can be generated using simple diode circuit. What are the other activation functions that can be generated using basic electronic items such as op-ams, bjts, diodes, filters and etc? If there are any, how to do it?

These are some of the activation functions.

• Please list the mathematical transfer function for these neural network functions. Mar 8 at 15:44
• What is an activation function? Mar 8 at 16:12
• “and etc” doesn’t give us much to help you with. Need more details and clarity. Mar 8 at 16:19
• We are lacking a lot of information about what you are trying to do here, could you at least post a full schematic. I say full because the other day somebody posted a tiny section of the schematic without elaborating further. Mar 8 at 16:25
• I basically want to generate above function using a circuit which uses op-amps, diodes, bjts and fets. Is there any cretain circuit designs that can generate those functions? Mar 8 at 16:30

I'm not sure how useful any of this will be, because all these suggestions only approximate those functions, to varying degrees, but here goes.

The simplest $$\tanh\$$ implementation I can think of, is a couple of anti-parallel diodes, that clamp the output close to 0V with an exponential "twist". Because the clamp voltage is 0.6V or so, optionally you can scale that output to ±1V or so with a simple non-inverting amplifier:

simulate this circuit – Schematic created using CircuitLab

For better control of the steepness of the central slope, you can move those diodes into the feedback loop of an inverting amplifier configuration, but use an additional resistor (R2 in the circuit below) to determine gain prior to the diodes taking over. The plots are with R2 set to 1kΩ (blue), 2kΩ (orange), and 5kΩ (tan):

simulate this circuit

Of course, you can invert and scale any signal with an inverting amplifier stage. Optionally, you can use that configuration as a summing amplifier, and offset the signal up or down. Here I take the output of the previous circuit (blue below), apply a gain of −0.9, and shift the whole thing upwards, to be centered around +0.5V (orange below), to obtain something approximating the sigmoid:

simulate this circuit

The ELU function is trivially simple to approximate. A diode clamp consisting of however many diodes you need for the negative limit. Each diode contributes −0.6V:

simulate this circuit

The ReLU function is exactly what you get from an active rectifier, but take care to take the output from the correct point:

simulate this circuit

The name "Leaky" ReLU is exactly what it say, all we need to do is add a leak (a current path from IN to OUT, via R1 below) to the active rectifier:

simulate this circuit

The "Maxout" function I am not sure about. I'll give it some thought.

The primary purpose of an activation function in an artificial neural network (ANN) is to create a non-linearity between layers. A purely linear combination of layers of weights and values can be mathematically reduced to a single layer (i.e. a single multiply-accumulate function -- the proof of this is simple enough and "left as an exercise for the reader") and thus without an intentional non-linear function it loses all the complexity that the multi-layer topology can provide. The activation function forces in a non-linearity, giving the neural network actual depth.

The only requirement for a usable activation function is that it be differentiable, so that back-propagation can be performed in training. This is partly why the "rectified linear unit" (ReLU) is so popular -- the slope is either 1 if x>0 or 0 otherwise (with the x=0 case arbitrarily assigned to zero) so it's really easy and fast to use in practice.

With that in mind, in theory any non-linear electrical device can compute an activation function. But none of them will be an exact match to the mathematical functions you've asked about. A transistor moving from linear to saturation region will look a lot like a tanh or sigmoid, but is not identical. A diode will not conduct at 0 V but rather at somewhere between 0.3 and 0.7 V, making it an imperfect ReLU. A comparator can be used to calculate which signal is the maximum, and that can be fed into a multiplexer to pass the actual signal through, making a maxout, but the multiplexer will drop some voltage and the comparator will only work in its input range. All real world devices have their limits. Even the humble resistor is non-linear if you push it out of its happy range. No electronic device you can design will ever produce a mathematically "pure" output.

Computing feed-forward in analog is the easy part, though. Training the ANN is the real challenge. To train your circuit, you will need to build circuitry which computes the derivative of each layer including its activation function, and which can update the multiplier weights on the fly. But if you can do that, then any non-linear electrical device will theoretically work as an activation function.

• +1 The last paragraph is especially important. These nonlinearities are worthless without being able to tune them in the training process Mar 9 at 6:35

Log / antilog type functions can be constructed from op-amps plus junction devices including transistors and diodes. They exploit the exponential I vs. V characteristics of their ‘knee’ area. (Search ‘logarithmic amplifiers’ for examples.)

More here: Sigmoid using op-amps

I think a sigmoid could also be approximated by using a FET pair, as suggested here: