Part 23: Neural Networks, Understanding Layers, and Activation Functions

Layers and activation functions are crucial components of neural networks. They transform the input data into meaningful output through a series of mathematical operations.

Layers in Neural Networks

Layers are the building blocks of neural networks. Common types of layers include fully connected layers (also known as dense layers), convolutional layers, and recurrent layers.

• Fully Connected (Dense) Layers: Fully connected layers connect every neuron in one layer to every neuron in the next layer. They are typically used in the final stages of a network to combine features and make predictions.

import torch.nn as nn

fc_layer = nn.Linear(in_features=10, out_features=50)

Convolutional Layers: Convolutional layers are used primarily in image processing tasks. They apply a convolution operation to the input, capturing spatial relationships and features.

conv_layer = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, stride=1, padding=1)

Recurrent Layers: Recurrent layers, such as LSTM and GRU, are used for sequence data like time series or text. They maintain a hidden state that captures information from previous time steps.

rnn_layer = nn.LSTM(input_size=10, hidden_size=20, num_layers=2, batch_first=True)

Activation Functions

Activation functions introduce non-linearity into the network, enabling it to learn complex patterns. Here are some common activation functions:

ReLU (Rectified Linear Unit): The ReLU function sets all negative values to zero. It is widely used due to its simplicity and effectiveness.

relu = nn.ReLU()

Sigmoid: The Sigmoid function squashes the input to a range between 0 and 1. It is often used in the output layer for binary classification problems.

sigmoid = nn.Sigmoid()

Tanh: The Tanh function squashes the input to a range between -1 and 1. It is similar to the Sigmoid function but outputs both positive and negative values.

tanh = nn.Tanh()

Example: Combining Layers and Activation Functions

Here is an example of a neural network that combines different layers and activation functions.

• Combined Example:

class CombinedNN(nn.Module):
    def __init__(self):
        super(CombinedNN, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, stride=1, padding=1)
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2, padding=0)
        self.fc1 = nn.Linear(16 * 16 * 16, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)
        self.relu = nn.ReLU()

    def forward(self, x):
        x = self.pool(self.relu(self.conv1(x)))
        x = x.view(-1, 16 * 16 * 16)
        x = self.relu(self.fc1(x))
        x = self.relu(self.fc2(x))
        x = self.fc3(x)
        return x

model = CombinedNN()
print(model)

In this example, CombinedNN is a neural network that uses convolutional and fully connected layers, with ReLU activation functions and a pooling layer to reduce spatial dimensions.