Part 24: PyTorch, Training Neural Networks

by digitaltech2.com

Training a neural network involves feeding it data, calculating the loss, and updating the network’s weights based on the gradients of the loss. This section will cover the steps required to train a neural network in PyTorch.

The Training Loop

The training loop is the core process where the model learns from the data. It consists of several key steps: forward pass, loss calculation, backward pass, and weight update.

  1. Forward Pass: Compute the output of the network given the input data.
  2. Loss Calculation: Compute the difference between the predicted output and the true target.
  3. Backward Pass: Compute the gradients of the loss with respect to the network’s parameters.
  4. Weight Update: Update the network’s parameters using an optimizer.
  • Basic Training Loop Example:
import torch
import torch.nn as nn
import torch.optim as optim

# Define the neural network
class SimpleNN(nn.Module):
    def __init__(self):
        super(SimpleNN, self).__init__()
        self.fc1 = nn.Linear(10, 50)
        self.fc2 = nn.Linear(50, 1)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = self.fc2(x)
        return x

model = SimpleNN()

# Define the loss function and optimizer
loss_fn = nn.MSELoss()
optimizer = optim.SGD(model.parameters(), lr=0.01)

# Create dummy data
inputs = torch.randn(64, 10)
targets = torch.randn(64, 1)

# Training loop
for epoch in range(100):
    optimizer.zero_grad()  # Zero the gradients
    outputs = model(inputs)  # Forward pass
    loss = loss_fn(outputs, targets)  # Compute loss
    loss.backward()  # Backward pass (compute gradients)
    optimizer.step()  # Update weights

    if epoch % 10 == 0:
        print(f'Epoch [{epoch}/100], Loss: {loss.item():.4f}')
Batch Processing

Neural networks are typically trained on batches of data rather than the entire dataset at once. This approach helps in faster computation and better utilization of hardware resources.

  • Using DataLoader for Batch Processing:
from torch.utils.data import DataLoader, TensorDataset

# Create a dataset and dataloader
dataset = TensorDataset(inputs, targets)
dataloader = DataLoader(dataset, batch_size=16, shuffle=True)

for epoch in range(100):
    for batch_inputs, batch_targets in dataloader:
        optimizer.zero_grad()
        outputs = model(batch_inputs)
        loss = loss_fn(outputs, batch_targets)
        loss.backward()
        optimizer.step()

    if epoch % 10 == 0:
        print(f'Epoch [{epoch}/100], Loss: {loss.item():.4f}')
Validation

Validation is the process of evaluating the model on a separate dataset to monitor its performance and detect overfitting.

  • Validation Loop Example:
# Create validation data
val_inputs = torch.randn(32, 10)
val_targets = torch.randn(32, 1)

for epoch in range(100):
    # Training loop
    for batch_inputs, batch_targets in dataloader:
        optimizer.zero_grad()
        outputs = model(batch_inputs)
        loss = loss_fn(outputs, batch_targets)
        loss.backward()
        optimizer.step()

    if epoch % 10 == 0:
        # Validation loop
        with torch.no_grad():
            val_outputs = model(val_inputs)
            val_loss = loss_fn(val_outputs, val_targets)
            print(f'Epoch [{epoch}/100], Loss: {loss.item():.4f}, Validation Loss: {val_loss.item():.4f}')
Saving and Loading Models

Saving and loading models is essential for resuming training and deploying models.

  • Saving a Model:
torch.save(model.state_dict(), 'model.pth')
  • Loading a Model:
model = SimpleNN()
model.load_state_dict(torch.load('model.pth'))
model.eval()  # Set the model to evaluation mode

Related Posts