Is it correct to say that not all activation function work by activating or deactivating neurons in-CodePudding

I understand that from a computational point of view, even if the output of an activation function is zero it still outputs the zero from the neuron to the next layer, therefore the term activating and deactivating the neuron are simple expressions to loosely describe the architecture and process in an artificial neural network (ANN) that mimics our own brain structure.

However I am asking this from a language point of view. In other words, since that there are a lot of diferent activation functions and some work by outputing binary numbers (either zero or one) while others never output zero is it correct to say that some activation functions always activate neurons since they never output zero while others do work by activating or deactivating the neurons since they work on thresholds?

Thanks!

CodePudding user response：

You attempted an implicit type cast between bool and float in natural language, and now your brain is confused how to parse that ;-)

First, ANNs are mathematical models. The analogy to biology is so weak, it's basically just story-telling.

In the context of ANNs, "activation function" is a mathematical function in a specific place (after summing the weighted inputs) that maps from float to float.

So if you really want to talk about "activate" and "deactivate" you should to define those terms, to make sure all readers are on-board. You could say a neuron is "deactivated" if its output is zero or below.

The problem with that language is when you write a formula that sums all activations. Does "deactivated" count as -1.0 or 0.0? You have to define that, too. And as you noticed the terms break down with the non-binary tanh and relu activation functions.

To answer your question: No I don't think you can say "some activation functions always activate". You could say some activation functions can either activate ( 1.0) or inhibit (-1.0) neurons instead of not influencing them (0.0). But usually you'd use inhibit for weights, not for neurons. And it's stretching the biology analogy as far as it can go.