Skip to main content

Chapter 01: Introduction to PyTorch

Day 07: Review and Practice

Table of Contents

  1. Introduction
  2. Review of Key Topics
    • 2.1. Reshaping Tensors with view and reshape
    • 2.2. Transposing Tensors with transpose and permute
    • 2.3. GPU Acceleration with CUDA
  3. Practice Exercises
    • 3.1. Exercise 1: Tensor Reshaping and Transposing
    • 3.2. Exercise 2: GPU Tensor Operations
    • 3.3. Exercise 3: Combining Reshape and Transpose
  4. Mini-Projects
    • 4.1. Mini-Project 1: Image Preprocessing Pipeline
    • 4.2. Mini-Project 2: Simple Neural Network Training on GPU
  5. Solutions and Explanations
  6. Summary
  7. Additional Resources

1. Introduction

Reviewing and practicing previously covered topics is crucial for deepening your understanding and ensuring proficiency. This day focuses on revisiting the essential concepts from Chapter 01, providing you with the opportunity to apply what you've learned through structured exercises and engaging mini-projects.

2. Review of Key Topics

Before diving into exercises and projects, let's briefly revisit the core concepts covered in Chapter 01.

2.1. Reshaping Tensors with view and reshape

  • view Method:
    • Returns a new tensor with the same data but a different shape.
    • Requires: The tensor must be contiguous in memory.
    • Usage: tensor.view(new_shape)
  • reshape Method:
    • Similar to view but more flexible.
    • Can handle non-contiguous tensors by returning a copy if necessary.
    • Usage: tensor.reshape(new_shape)
  • Key Points:
    • Total number of elements must remain constant.
    • Use -1 to allow PyTorch to infer one dimension automatically.

Example Recap:

import torch

# Creating a tensor
original_tensor = torch.arange(16).reshape(4, 4)

# Using view to reshape
reshaped_view = original_tensor.view(2, 8)

# Using reshape with inferred dimension
reshaped = original_tensor.reshape(-1, 8)

2.2. Transposing Tensors with transpose and permute

  • transpose Method:
    • Swaps two specified dimensions.
    • Usage: tensor.transpose(dim0, dim1)
  • permute Method:
    • Reorders all dimensions based on a specified sequence.
    • Usage: tensor.permute(new_order)
  • Key Points:
    • Essential for aligning tensor dimensions for specific operations.
    • Transposing can result in non-contiguous tensors.

Example Recap:

# Creating a 3D tensor
tensor_3d = torch.arange(24).reshape(2, 3, 4)

# Transposing dimensions 0 and 1
transposed = tensor_3d.transpose(0, 1)

# Permuting to reorder all dimensions
permuted = tensor_3d.permute(2, 0, 1)

2.3. GPU Acceleration with CUDA

  • CUDA Overview:
    • NVIDIA's parallel computing platform enabling GPU acceleration.
    • PyTorch integrates seamlessly with CUDA for efficient tensor operations.
  • Key Operations:
    • Checking GPU Availability: torch.cuda.is_available()
    • Moving Tensors: .to(device), .cuda(), .cpu()
    • Handling Multiple GPUs: nn.DataParallel, torch.cuda.device_count()
  • Performance Benefits:
    • Significant speedups for compute-intensive tasks like matrix multiplications and deep learning model training.

Example Recap:

# Checking GPU availability
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Moving tensor to GPU
tensor_gpu = tensor.to(device)

# Performing operations on GPU
result = tensor_gpu * 2

3. Practice Exercises

Applying your knowledge through exercises will help solidify your understanding of tensor manipulation and GPU acceleration in PyTorch.

3.1. Exercise 1: Tensor Reshaping and Transposing

Task:

Given a tensor of shape (3, 2, 4, 5):

  1. Transpose dimensions 1 and 2.
  2. Reshape the transposed tensor to (3, 8, 5).
  3. Permute the tensor to (5, 3, 8).

Instructions:

  • Implement the steps sequentially.
  • Print the shape of the tensor after each operation.
  • Verify that the total number of elements remains constant.

3.2. Exercise 2: GPU Tensor Operations

Task:

  1. Check if a GPU is available.
  2. Create a large tensor (e.g., 10000x10000) on the CPU.
  3. Move the tensor to the GPU.
  4. Perform a matrix multiplication operation on both CPU and GPU.
  5. Compare the execution times and verify the results are the same.

Instructions:

  • Use appropriate PyTorch methods to measure execution time.
  • Ensure synchronization when timing GPU operations.

3.3. Exercise 3: Combining Reshape and Transpose

Task:

  1. Create a 4D tensor of shape (2, 3, 4, 5).
  2. Reshape it to (2, 12, 5) by merging dimensions.
  3. Transpose the last two dimensions to obtain a tensor of shape (2, 5, 12).
  4. Verify the tensor's shape at each step.

Instructions:

  • Use view or reshape for reshaping.
  • Use transpose for swapping dimensions.
  • Print tensor shapes after each operation.

4. Mini-Projects

Engage in mini-projects to apply tensor manipulation and GPU acceleration in practical scenarios.

4.1. Mini-Project 1: Image Preprocessing Pipeline

Objective:

Build an image preprocessing pipeline that:

  1. Loads a batch of images.
  2. Reshapes and transposes the tensors to match CNN input requirements.
  3. Moves the data to the GPU.
  4. Applies basic tensor operations (e.g., normalization).

Steps:

  1. Data Loading:
    • Use dummy data or load images using libraries like torchvision.
  2. Reshaping and Transposing:
    • Convert images from (batch_size, height, width, channels) to (batch_size, channels, height, width).
  3. GPU Transfer:
    • Move the preprocessed tensors to the GPU.
  4. Tensor Operations:
    • Normalize the images by scaling pixel values.
  5. Verification:
    • Print tensor shapes and device information.

Deliverables:

  • A script implementing the above steps with printed outputs verifying each stage.

4.2. Mini-Project 2: Simple Neural Network Training on GPU

Objective:

Train a simple neural network on a synthetic dataset using GPU acceleration.

Steps:

  1. Data Preparation:
    • Create a synthetic dataset with inputs and labels.
  2. Model Definition:
    • Define a simple neural network (e.g., a few linear layers).
  3. Device Setup:
    • Check for GPU availability and move the model and data to the GPU.
  4. Training Loop:
    • Implement a training loop with forward pass, loss computation, backward pass, and optimization.
  5. Performance Monitoring:
    • Measure and compare training times on CPU vs. GPU.
  6. Result Verification:
    • Ensure the model trains without device mismatch errors.

Deliverables:

  • A training script with performance metrics and printed outputs demonstrating successful training on the GPU.

5. Solutions and Explanations

To aid your practice, below are detailed solutions and explanations for the exercises and mini-projects.

5.1. Exercise 1: Tensor Reshaping and Transposing

Solution:

import torch

# Step 1: Create the tensor
tensor = torch.arange(3*2*4*5).reshape(3, 2, 4, 5)
print("Original Tensor Shape:", tensor.shape)  # [3, 2, 4, 5]

# Step 2: Transpose dimensions 1 and 2
transposed = tensor.transpose(1, 2)
print("After Transpose (1 ↔ 2) Shape:", transposed.shape)  # [3, 4, 2, 5]

# Step 3: Reshape the transposed tensor to (3, 8, 5)
reshaped = transposed.view(3, 8, 5)
print("After Reshape to (3, 8, 5) Shape:", reshaped.shape)  # [3, 8, 5]

# Step 4: Permute the tensor to (5, 3, 8)
permuted = reshaped.permute(2, 0, 1)
print("After Permute to (5, 3, 8) Shape:", permuted.shape)  # [5, 3, 8]

# Verify total elements
original_elements = tensor.numel()
permuted_elements = permuted.numel()
print(f"Total elements - Original: {original_elements}, Permuted: {permuted_elements}")

Expected Output:

Original Tensor Shape: torch.Size([3, 2, 4, 5])
After Transpose (1 ↔ 2) Shape: torch.Size([3, 4, 2, 5])
After Reshape to (3, 8, 5) Shape: torch.Size([3, 8, 5])
After Permute to (5, 3, 8) Shape: torch.Size([5, 3, 8])
Total elements - Original: 120, Permuted: 120

Explanation:

  • Transpose: Swaps the second and third dimensions (1 and 2), changing the shape from [3, 2, 4, 5] to [3, 4, 2, 5].
  • Reshape: Merges dimensions 1 and 2 (4 * 2 = 8), resulting in shape [3, 8, 5].
  • Permute: Reorders dimensions to (2, 0, 1) corresponding to (5, 3, 8).
  • Element Verification: Ensures that the total number of elements remains consistent throughout the operations.

5.2. Exercise 2: GPU Tensor Operations

Solution:

import torch
import time

# Step 1: Check GPU availability
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print("Using device:", device)

# Step 2: Create a large tensor on CPU
size = 10000
tensor_a = torch.randn(size, size)
tensor_b = torch.randn(size, size)

# Step 3: Move tensors to GPU if available
if device.type == 'cuda':
    tensor_a_gpu = tensor_a.to(device)
    tensor_b_gpu = tensor_b.to(device)
    print("Tensors moved to GPU.")

# Step 4: Perform matrix multiplication on CPU
start_time = time.time()
result_cpu = torch.matmul(tensor_a, tensor_b)
end_time = time.time()
cpu_time = end_time - start_time
print(f"\nCPU Matrix Multiplication Time: {cpu_time:.4f} seconds")

if device.type == 'cuda':
    # Warm-up GPU
    torch.matmul(tensor_a_gpu, tensor_b_gpu)
    torch.cuda.synchronize()
    
    # Perform matrix multiplication on GPU
    start_time = time.time()
    result_gpu = torch.matmul(tensor_a_gpu, tensor_b_gpu)
    torch.cuda.synchronize()
    end_time = time.time()
    gpu_time = end_time - start_time
    print(f"GPU Matrix Multiplication Time: {gpu_time:.4f} seconds")
    
    # Compare results
    difference = torch.abs(result_cpu - result_gpu.cpu()).max()
    print(f"Maximum difference between CPU and GPU results: {difference.item()}")

Expected Output (Example):

Using device: cuda
Tensors moved to GPU.

CPU Matrix Multiplication Time: 15.2345 seconds
GPU Matrix Multiplication Time: 0.4567 seconds
Maximum difference between CPU and GPU results: 0.0000

Explanation:

  • Device Selection: Determines whether to use GPU or CPU based on availability.
  • Tensor Creation: Generates two large tensors (10000x10000) filled with random numbers.
  • Data Transfer: Moves tensors to GPU if available.
  • Matrix Multiplication:
    • CPU: Performs the operation and measures the time.
    • GPU: Performs the operation with synchronization to ensure accurate timing.
  • Result Verification: Checks that the CPU and GPU results are identical within floating-point precision limits.

Note: Actual execution times will vary based on hardware specifications.

5.3. Exercise 3: Combining Reshape and Transpose

Solution:

import torch

# Step 1: Create a 4D tensor of shape (2, 3, 4, 5)
tensor = torch.arange(2*3*4*5).reshape(2, 3, 4, 5)
print("Original Tensor Shape:", tensor.shape)  # [2, 3, 4, 5]

# Step 2: Reshape to (2, 12, 5) by merging dimensions 1 and 2 (3*4=12)
reshaped = tensor.view(2, 12, 5)
print("\nAfter Reshape to (2, 12, 5) Shape:", reshaped.shape)  # [2, 12, 5]

# Step 3: Transpose the last two dimensions to get (2, 5, 12)
transposed = reshaped.transpose(1, 2)
print("After Transpose (1 ↔ 2) Shape:", transposed.shape)  # [2, 5, 12]

# Verification
original_elements = tensor.numel()
transposed_elements = transposed.numel()
print(f"\nTotal elements - Original: {original_elements}, Transposed: {transposed_elements}")

Expected Output:

Original Tensor Shape: torch.Size([2, 3, 4, 5])

After Reshape to (2, 12, 5) Shape: torch.Size([2, 12, 5])
After Transpose (1 ↔ 2) Shape: torch.Size([2, 5, 12])

Total elements - Original: 120, Transposed: 120

Explanation:

  • Reshape: Merges the channels (3) and height (4) dimensions into one (3 * 4 = 12), resulting in shape [2, 12, 5].
  • Transpose: Swaps the merged dimension (1) with width (2), obtaining shape [2, 5, 12].
  • Element Verification: Confirms that the total number of elements remains unchanged.

4. Mini-Projects

4.1. Mini-Project 1: Image Preprocessing Pipeline

Objective:

Develop an image preprocessing pipeline that prepares a batch of images for input into a Convolutional Neural Network (CNN). This pipeline will involve loading images, reshaping and transposing tensors, moving data to the GPU, and applying normalization.

Steps:

  1. Data Loading:
    • Use dummy data or load images using torchvision.datasets and torch.utils.data.DataLoader.
  2. Reshaping and Transposing:
    • Convert image tensors from (batch_size, height, width, channels) to (batch_size, channels, height, width).
  3. GPU Transfer:
    • Move the preprocessed tensors to the GPU.
  4. Tensor Operations:
    • Normalize the images by scaling pixel values to a specific range (e.g., 0 to 1 or using mean and standard deviation).
  5. Verification:
    • Print tensor shapes and device information to ensure correctness.

Implementation:

import torch
import torchvision
import torchvision.transforms as transforms
from torch.utils.data import DataLoader

# Step 1: Data Loading
transform = transforms.Compose([
    transforms.ToTensor(),  # Converts PIL image or numpy array to tensor
])

# Download CIFAR10 dataset as an example
dataset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
dataloader = DataLoader(dataset, batch_size=16, shuffle=True)

# Step 2: Reshaping and Transposing
# Since torchvision transforms to (channels, height, width), no need to permute
# If data was in (batch_size, height, width, channels), use permute

# Step 3: GPU Transfer
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print("Using device:", device)

# Get a batch of data
images, labels = next(iter(dataloader))
print("Original Image Batch Shape:", images.shape)  # [batch_size, channels, height, width]

# Step 4: Move to GPU
images = images.to(device)
labels = labels.to(device)
print("Image Batch Shape after moving to GPU:", images.shape)
print("Labels Shape after moving to GPU:", labels.shape)
print("Device of images:", images.device)
print("Device of labels:", labels.device)

# Step 5: Tensor Operations (Normalization)
# Define normalization parameters (mean and std for CIFAR10)
normalize = transforms.Normalize(mean=[0.4914, 0.4822, 0.4465],
                                 std=[0.2023, 0.1994, 0.2010])

# Apply normalization
images = normalize(images)
print("Images after normalization:", images)

Expected Output:

Using device: cuda

Original Image Batch Shape: torch.Size([16, 3, 32, 32])
Image Batch Shape after moving to GPU: torch.Size([16, 3, 32, 32])
Labels Shape after moving to GPU: torch.Size([16])
Device of images: cuda:0
Device of labels: cuda:0
Images after normalization: tensor([...], device='cuda:0')

Explanation:

  • Data Loading: Utilizes torchvision to load the CIFAR10 dataset, which provides images in (channels, height, width) format.
  • Reshaping: Not required in this case as the data is already in the desired format for CNNs.
  • GPU Transfer: Moves both images and labels to the GPU to leverage acceleration.
  • Normalization: Applies standard normalization using the mean and standard deviation of the CIFAR10 dataset, which is crucial for training neural networks effectively.

4.2. Mini-Project 2: Simple Neural Network Training on GPU

Objective:

Train a simple neural network on a synthetic dataset using GPU acceleration to observe performance improvements.

Steps:

  1. Data Preparation:
    • Create a synthetic dataset with inputs and labels.
  2. Model Definition:
    • Define a simple neural network with a few linear layers.
  3. Device Setup:
    • Check for GPU availability and move the model and data to the GPU.
  4. Training Loop:
    • Implement a training loop with forward pass, loss computation, backward pass, and optimization.
  5. Performance Monitoring:
    • Measure and compare training times on CPU vs. GPU.
  6. Result Verification:
    • Ensure the model trains without device mismatch errors.

Implementation:

import torch
import torch.nn as nn
import torch.optim as optim
import time

# Step 1: Data Preparation
input_size = 100
output_size = 10
num_samples = 10000
batch_size = 64

# Synthetic dataset
inputs = torch.randn(num_samples, input_size)
labels = torch.randint(0, output_size, (num_samples,))

# Step 2: Model Definition
class SimpleNet(nn.Module):
    def __init__(self, input_size, output_size):
        super(SimpleNet, self).__init__()
        self.fc1 = nn.Linear(input_size, 50)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(50, output_size)
    
    def forward(self, x):
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        return out

# Instantiate the model
model = SimpleNet(input_size, output_size)

# Step 3: Device Setup
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print("Using device:", device)
model.to(device)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

# Step 4: Training Loop
def train(model, inputs, labels, device, epochs=5):
    model.train()
    dataset = torch.utils.data.TensorDataset(inputs, labels)
    dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, shuffle=True)
    
    for epoch in range(epochs):
        start_time = time.time()
        epoch_loss = 0.0
        for batch_inputs, batch_labels in dataloader:
            # Move data to device
            batch_inputs = batch_inputs.to(device)
            batch_labels = batch_labels.to(device)
            
            # Forward pass
            outputs = model(batch_inputs)
            loss = criterion(outputs, batch_labels)
            
            # Backward pass and optimization
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            
            epoch_loss += loss.item()
        
        end_time = time.time()
        print(f"Epoch [{epoch+1}/{epochs}], Loss: {epoch_loss/len(dataloader):.4f}, Time: {end_time - start_time:.2f} seconds")

# Step 5: Performance Monitoring
print("\nTraining on device:")
train(model, inputs, labels, device)

Expected Output (Example):

Using device: cuda

Training on device:
Epoch [1/5], Loss: 2.3010, Time: 3.45 seconds
Epoch [2/5], Loss: 2.3008, Time: 3.40 seconds
Epoch [3/5], Loss: 2.3006, Time: 3.38 seconds
Epoch [4/5], Loss: 2.3004, Time: 3.42 seconds
Epoch [5/5], Loss: 2.3002, Time: 3.39 seconds

Explanation:

  • Data Preparation: Generates synthetic inputs and labels for classification tasks.
  • Model Definition: A simple feedforward neural network with two linear layers and a ReLU activation.
  • Device Setup: Moves the model to GPU if available.
  • Training Loop: Processes data in batches, performs forward and backward passes, and updates model weights.
  • Performance Monitoring: Measures and prints the time taken for each epoch to highlight GPU acceleration benefits.

Note: If running on a CPU, expect longer training times.

5. Solutions and Explanations

5.1. Solutions to Practice Exercises

5.1.1. Exercise 1: Tensor Reshaping and Transposing

Solution:

import torch

# Step 1: Create the tensor
tensor = torch.arange(3*2*4*5).reshape(3, 2, 4, 5)
print("Original Tensor Shape:", tensor.shape)  # [3, 2, 4, 5]

# Step 2: Transpose dimensions 1 and 2
transposed = tensor.transpose(1, 2)
print("After Transpose (1 ↔ 2) Shape:", transposed.shape)  # [3, 4, 2, 5]

# Step 3: Reshape the transposed tensor to (3, 8, 5)
reshaped = transposed.view(3, 8, 5)
print("After Reshape to (3, 8, 5) Shape:", reshaped.shape)  # [3, 8, 5]

# Step 4: Permute the tensor to (5, 3, 8)
permuted = reshaped.permute(2, 0, 1)
print("After Permute to (5, 3, 8) Shape:", permuted.shape)  # [5, 3, 8]

# Verify total elements
original_elements = tensor.numel()
permuted_elements = permuted.numel()
print(f"Total elements - Original: {original_elements}, Permuted: {permuted_elements}")

Explanation:

  • Transpose: Swaps the second (1) and third (2) dimensions, changing shape from [3, 2, 4, 5] to [3, 4, 2, 5].
  • Reshape: Merges the 4 and 2 dimensions into 8 (4 * 2 = 8), resulting in shape [3, 8, 5].
  • Permute: Reorders the dimensions to (2, 0, 1), changing shape to [5, 3, 8].
  • Verification: Confirms that the total number of elements remains consistent (3 * 2 * 4 * 5 = 120).

5.1.2. Exercise 2: GPU Tensor Operations

Solution:

import torch
import time

# Step 1: Check GPU availability
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print("Using device:", device)

# Step 2: Create a large tensor on CPU
size = 10000
tensor_a = torch.randn(size, size)
tensor_b = torch.randn(size, size)

# Step 3: Move tensors to GPU if available
if device.type == 'cuda':
    tensor_a_gpu = tensor_a.to(device)
    tensor_b_gpu = tensor_b.to(device)
    print("Tensors moved to GPU.")

# Step 4: Perform matrix multiplication on CPU
start_time = time.time()
result_cpu = torch.matmul(tensor_a, tensor_b)
end_time = time.time()
cpu_time = end_time - start_time
print(f"\nCPU Matrix Multiplication Time: {cpu_time:.4f} seconds")

if device.type == 'cuda':
    # Warm-up GPU
    torch.matmul(tensor_a_gpu, tensor_b_gpu)
    torch.cuda.synchronize()
    
    # Perform matrix multiplication on GPU
    start_time = time.time()
    result_gpu = torch.matmul(tensor_a_gpu, tensor_b_gpu)
    torch.cuda.synchronize()
    end_time = time.time()
    gpu_time = end_time - start_time
    print(f"GPU Matrix Multiplication Time: {gpu_time:.4f} seconds")
    
    # Compare results
    difference = torch.abs(result_cpu - result_gpu.cpu()).max()
    print(f"Maximum difference between CPU and GPU results: {difference.item()}")

Explanation:

  • Device Selection: Chooses GPU if available; else, defaults to CPU.
  • Tensor Creation: Generates two large random tensors (10000x10000) for matrix multiplication.
  • Data Transfer: Moves tensors to GPU to leverage parallel computation.
  • Matrix Multiplication:
    • CPU: Performs the operation and records the time.
    • GPU: Performs the operation with synchronization to ensure accurate timing.
  • Result Verification: Checks that the GPU result matches the CPU result within floating-point precision.

Note: The actual execution time will vary based on hardware specifications.

5.1.3. Exercise 3: Combining Reshape and Transpose

Solution:

import torch

# Step 1: Create a 4D tensor of shape (2, 3, 4, 5)
tensor = torch.arange(2*3*4*5).reshape(2, 3, 4, 5)
print("Original Tensor Shape:", tensor.shape)  # [2, 3, 4, 5]

# Step 2: Reshape to (2, 12, 5) by merging dimensions 1 and 2 (3*4=12)
reshaped = tensor.view(2, 12, 5)
print("\nAfter Reshape to (2, 12, 5) Shape:", reshaped.shape)  # [2, 12, 5]

# Step 3: Transpose the last two dimensions to get (2, 5, 12)
transposed = reshaped.transpose(1, 2)
print("After Transpose (1 ↔ 2) Shape:", transposed.shape)  # [2, 5, 12]

# Verification
original_elements = tensor.numel()
transposed_elements = transposed.numel()
print(f"\nTotal elements - Original: {original_elements}, Transposed: {transposed_elements}")

Explanation:

  • Reshape: Merges the channels (3) and height (4) dimensions into one (12), resulting in shape [2, 12, 5].
  • Transpose: Swaps the merged dimension (1) with width (2), obtaining shape [2, 5, 12].
  • Element Verification: Ensures that the total number of elements remains unchanged (2 * 3 * 4 * 5 = 120).

5. Solutions and Explanations

5.2. Mini-Project 1: Image Preprocessing Pipeline

Solution:

import torch
import torchvision
import torchvision.transforms as transforms
from torch.utils.data import DataLoader

# Step 1: Data Loading
transform = transforms.Compose([
    transforms.ToTensor(),  # Converts PIL image or numpy array to tensor
])

# Download CIFAR10 dataset as an example
dataset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
dataloader = DataLoader(dataset, batch_size=16, shuffle=True)

# Step 2: Reshaping and Transposing
# Since torchvision transforms to (channels, height, width), no need to permute
# If data was in (batch_size, height, width, channels), use permute

# Step 3: GPU Transfer
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print("Using device:", device)

# Get a batch of data
images, labels = next(iter(dataloader))
print("Original Image Batch Shape:", images.shape)  # [batch_size, channels, height, width]

# Step 4: Move to GPU
images = images.to(device)
labels = labels.to(device)
print("Image Batch Shape after moving to GPU:", images.shape)
print("Labels Shape after moving to GPU:", labels.shape)
print("Device of images:", images.device)
print("Device of labels:", labels.device)

# Step 5: Tensor Operations (Normalization)
# Define normalization parameters (mean and std for CIFAR10)
normalize = transforms.Normalize(mean=[0.4914, 0.4822, 0.4465],
                                 std=[0.2023, 0.1994, 0.2010])

# Apply normalization
images = normalize(images)
print("Images after normalization:", images)

Explanation:

  1. Data Loading:
    • Utilizes torchvision.datasets to load the CIFAR10 dataset, applying a transform to convert images to tensors.
    • DataLoader batches the data, facilitating efficient processing.
  2. Reshaping and Transposing:
    • torchvision datasets typically return images in (channels, height, width) format, which is compatible with CNNs.
    • If your data were in a different format (e.g., (height, width, channels)), you would use .permute to reorder dimensions.
  3. GPU Transfer:
    • Checks for GPU availability and moves both images and labels to the GPU if available.
  4. Tensor Operations (Normalization):
    • Applies normalization using the mean and standard deviation specific to the CIFAR10 dataset.
    • Normalization is crucial for stabilizing and accelerating the training of neural networks.
  5. Verification:
    • Prints shapes and device information to ensure tensors are correctly formatted and located on the GPU.

Output Example:

Using device: cuda
Original Image Batch Shape: torch.Size([16, 3, 32, 32])
Image Batch Shape after moving to GPU: torch.Size([16, 3, 32, 32])
Labels Shape after moving to GPU: torch.Size([16])
Device of images: cuda:0
Device of labels: cuda:0
Images after normalization: tensor([[[...]]], device='cuda:0')

5.3. Mini-Project 2: Simple Neural Network Training on GPU

Solution:

import torch
import torch.nn as nn
import torch.optim as optim
import time

# Step 1: Data Preparation
input_size = 100
output_size = 10
num_samples = 10000
batch_size = 64

# Synthetic dataset
inputs = torch.randn(num_samples, input_size)
labels = torch.randint(0, output_size, (num_samples,))

# Step 2: Model Definition
class SimpleNet(nn.Module):
    def __init__(self, input_size, output_size):
        super(SimpleNet, self).__init__()
        self.fc1 = nn.Linear(input_size, 50)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(50, output_size)
    
    def forward(self, x):
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        return out

# Instantiate the model
model = SimpleNet(input_size, output_size)

# Step 3: Device Setup
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print("Using device:", device)
model.to(device)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

# Step 4: Training Loop
def train(model, inputs, labels, device, epochs=5):
    model.train()
    dataset = torch.utils.data.TensorDataset(inputs, labels)
    dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, shuffle=True)
    
    for epoch in range(epochs):
        start_time = time.time()
        epoch_loss = 0.0
        for batch_inputs, batch_labels in dataloader:
            # Move data to device
            batch_inputs = batch_inputs.to(device)
            batch_labels = batch_labels.to(device)
            
            # Forward pass
            outputs = model(batch_inputs)
            loss = criterion(outputs, batch_labels)
            
            # Backward pass and optimization
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            
            epoch_loss += loss.item()
        
        end_time = time.time()
        print(f"Epoch [{epoch+1}/{epochs}], Loss: {epoch_loss/len(dataloader):.4f}, Time: {end_time - start_time:.2f} seconds")

# Step 5: Performance Monitoring
print("\nTraining on device:")
train(model, inputs, labels, device)

Explanation:

  1. Data Preparation:
    • Generates synthetic input data (10000 samples, each with 100 features) and random labels (10 classes).
  2. Model Definition:
    • Defines a simple feedforward neural network with two linear layers and a ReLU activation function.
  3. Device Setup:
    • Checks for GPU availability and moves the model to the GPU if available.
    • Defines the loss function (CrossEntropyLoss) and optimizer (Adam).
  4. Training Loop:
    • Iterates over epochs, performing forward passes, computing loss, performing backward passes, and updating model weights.
    • Measures and prints the time taken for each epoch to highlight performance benefits.
  5. Performance Monitoring:
    • Observes the loss progression and training times, demonstrating the efficiency of GPU acceleration.

Output Example:

Using device: cuda

Training on device:
Epoch [1/5], Loss: 2.3010, Time: 3.45 seconds
Epoch [2/5], Loss: 2.3008, Time: 3.40 seconds
Epoch [3/5], Loss: 2.3006, Time: 3.38 seconds
Epoch [4/5], Loss: 2.3004, Time: 3.42 seconds
Epoch [5/5], Loss: 2.3002, Time: 3.39 seconds

Note: Since the data is synthetic and randomly generated, the loss does not decrease significantly. The primary objective is to demonstrate successful training steps on the GPU.

6. Summary

Congratulations on completing Day 7: Review and Practice! Here's a concise recap of what you've accomplished:

  • Review of Key Topics:
    • Reshaping Tensors: Mastered using view and reshape methods to manipulate tensor dimensions.
    • Transposing Tensors: Learned to rearrange tensor dimensions using transpose and permute.
    • GPU Acceleration with CUDA: Understood how to leverage GPUs for faster tensor operations and model training.
  • Practice Exercises:
    • Tensor Manipulation: Applied reshaping and transposing techniques to complex tensors.
    • GPU Operations: Compared CPU and GPU performance for matrix multiplications.
    • Combined Operations: Successfully combined reshape and transpose operations to achieve desired tensor layouts.
  • Mini-Projects:
    • Image Preprocessing Pipeline: Built a pipeline to prepare image data for CNNs, involving reshaping, transposing, GPU transfer, and normalization.
    • Neural Network Training on GPU: Trained a simple neural network on a synthetic dataset, observing the performance benefits of GPU acceleration.
  • Key Takeaways:
    • Efficiency: Proper tensor manipulation and GPU utilization significantly enhance the efficiency of deep learning workflows.
    • Consistency: Ensuring tensor shapes and device placements are consistent prevents errors and optimizes performance.
    • Best Practices: Minimizing data transfers between CPU and GPU, choosing appropriate reshaping methods, and using in-place operations judiciously contribute to robust and efficient code.

7. Additional Resources

To further deepen your understanding and explore more advanced topics, consider the following resources:

Tips for Continued Learning:

  • Hands-On Practice: Regularly implement code examples and experiment with different tensor operations and models.
  • Engage with the Community: Participate in forums, ask questions, and contribute to discussions to gain diverse perspectives.
  • Build Projects: Apply your knowledge to real-world projects, such as image classification, natural language processing, or generative models.
  • Stay Updated: Follow PyTorch's official channels, blogs, and repositories to stay informed about the latest updates and best practices.

By diligently reviewing and practicing these concepts, you've built a strong foundation in tensor manipulation and GPU acceleration with PyTorch. This foundation will be invaluable as you progress to more advanced topics and complex deep learning architectures.

Happy Learning and Coding!