Design of non sigmoidal activation functions for feed forward artificial neural networks

Abstract

The possession of this universal approximation property by these networks is a reflection of the conditions imposed on the activation functions by these universal approximation theorems for feed-forward artificial neural networks. The primary condition imposed on the structure / form of the activation function is that it should be a non-polynomial function or a function that cannot be expressed as a polynomial of finite degree. The activation functions used in building feed-forward artificial neural networks, for solution of learning tasks, that are generally used can be classified as (a) as sigmoidal activation functions and (b) as unbounded non-sigmoidal and non-polynomial activation function. Both these classes of functions are non-polynomial functions. Also both these class of functions can be characterized as countinuous, differentiable and non-decreasing functions. That is functions that are always increasing and if not not increasing, they do not decrease as in the independent argument of the function increases. The generally used activation function of both the types are empirically compared on a set of thirty learning tasks. The tasks are taken from the domain of function approximation, real-life regression tasks and dynamical system forecasting tasks. The experimental results demonstrate that the sigmoidal class of activation functions outperform the unbounded and non-sigmoidal activation function