Cost Functions

A the most popular and applicable cost functions has already been implemented in this library, and are listed below. However, it is very easy to specify your own cost functions as described in Arbitrary Cost Functions.

Warning

The Softmax Categorical Cross Entropy cost function is required when using a softmax layer in the network topology.

Usage

Using the various cost functions is as easy as only importing the desired cost function and passing it to the decided learning function. Below is an example of how to use the Cross Entropy cost function when training using the vanilla backpropagation algorithm.

from nimblenet.cost_functions import binary_cross_entropy_cost
from nimblenet.activation_functions import sigmoid_function
from nimblenet.learning_algorithms import backpropagation
from nimblenet.data_structures import Instance
from nimblenet.neuralnet import NeuralNet

dataset        = [ Instance( [0,0], [0] ), Instance( [1,0], [1] ), Instance( [0,1], [1] ), Instance( [1,1], [1] )]
settings       = {
    "n_inputs" : 2,
    "layers"   : [  (2, sigmoid_function) ]
}

network        = NeuralNet( settings )
training_set   = dataset
test_set       = dataset
cost_function  = binary_cross_entropy_cost

backpropagation(
        network,              # the network to train
        training_set,         # specify the training set
        test_set,             # specify the test set

        # This is where we specify the cost function to optimize:
        cost_function         # specify the cost function to calculate error
    )

List of cost functions

Sum Squared Error

from nimblenet.cost_functions import sum_squared_error

Binary Cross Entropy

from nimblenet.cost_functions import binary_cross_entropy_cost

Softmax Categorical Cross Entropy

This cost function is required when including a softmax layer in your network topology.

from nimblenet.cost_functions import softmax_categorical_cross_entropy_cost

Hellinger Distance

from nimblenet.cost_functions import hellinger_distance

Arbitrary Cost Functions

It is easy to optimize your own, custom cost functions. A cost function has the required form:

def custom_cost_function(
            outputs,            # the signal emitted from the network
            targets,            # the target values we would like the network to output
            derivative = False  # whether the cost function should return its derivative
        ):
    ...

The outputs and targets parameters are NumPy matrices with shape [n_samples, n_outputs].

As an example, we can look at how the Sum Squared Error function is implemented:

def sum_squared_error( outputs, targets, derivative = False ):
    if derivative:
        return outputs - targets
    else:
        return 0.5 * np.mean(np.sum( np.power(outputs - targets,2), axis = 1 ))

Important

Observe that we calculate the mean of the error, per singal, across the input instances fed into the network. This detail is important to remember in order to get the derivatives correct.

How to

Lets define a custom cost function and use it when training the network:

from nimblenet.activation_functions import sigmoid_function
from nimblenet.learning_algorithms import backpropagation
from nimblenet.data_structures import Instance
from nimblenet.neuralnet import NeuralNet
import numpy as np

def custom_cost_function( outputs, targets, derivative = False ):
    if derivative:
        return outputs - targets
    else:
        return 0.5 * np.mean(np.sum( np.power(outputs - targets,2), axis = 1 ))
#end

dataset        = [ Instance( [0,0], [0] ), Instance( [1,0], [1] ), Instance( [0,1], [1] ), Instance( [1,1], [1] )]
settings       = {
    "n_inputs" : 2,
    "layers"   : [  (2, sigmoid_function) ]
}

network        = NeuralNet( settings )
training_set   = dataset
test_set       = dataset
cost_function  = custom_cost_function

backpropagation(
        network,              # the network to train
        training_set,         # specify the training set
        test_set,             # specify the test set

        # This is where we specify the cost function to optimize:
        cost_function         # specify the cost function to calculate error
    )