3. Linear Perceptron

This lab gets us started with PyTorch’s workflow used for deep learning training. You are given a mystery dataset which consists of 1000 labeled points in \(\mathbb{R}^3\). The labels are binary, i.e., we only have the two labels 0 and 1. Our goal is to find a linear classifier which separates points having label 0 from those having label 1. The lab’s individual steps are given as “Data Exploration”, “Datasets and Data Loaders”, “Training” and “Visualization”. We’ll roughly follow this structure when training more complex neural networks in future.

Hint

Look outside of the class for additional guidance on how to use PyTorch for Machine Learning workloads. PyTorch’s tutorials are a great way to get started.

3.1. Data Exploration

The points are given in the file data_points.csv. Each line in the file represents a single point where the points’ x-, y- and z-coordinates are separated by commas. The corresponding labels are given in the file data_labels.csv, where a single line gives the label of the respective point.

Listing 3.1.1 First five lines of the file data_points.csv.
1-8.494544704477987596e-01,-5.033859511649513979e-02,3.873837307225119070e-02
2-7.883425406506253674e-01,-2.232586812852378061e-02,9.367344753561300530e-02
31.585747745378497942e-01,4.059444628687230994e-01,1.031322306402090661e+00
4-9.682621200468680689e-01,-2.450696604084433294e-01,1.179431003097544117e-01
5-1.707633608738729214e-01,1.420339570729253209e-01,9.003685159064296339e-01
Listing 3.1.2 First five lines of the file data_labels.csv.
10
20
31
40
51

For example, the first five lines of the two files are given in Listing 3.1.1 and in Listing 3.1.2. This means the the first point \(P_1\) has approximately the coordinates (-8.5e-01, -5.0e-02, 3.9e-02). The label of \(P_1\) is 0. \(P_3\) is the first point which has label 1. The approximate coordinates of \(P_3\) are (1.6e-01, 4.1e-01, 1.0e+00).

Task

Visualize the mystery dataset using matplotlib. Use 3D scatterplots. Color points with label 0 black and those with label 1 red.

3.2. Datasets and Data Loaders

Now, that we have an understanding for our dataset, we are ready to advance towards the targeted linear classifier. Let’s first define a data loader which wraps our dataset. Data loaders are PyTorch’s way of tackling the preprocessing. The goal of an efficient data loader is to have the preprocessed data ready once the compute units require it. For the time being we’ll start simple: Our data loader does nothing else than returning the data in batches. Later on, we’ll see use more advanced data loaders. For example, Section 13 uses special data loaders to perform data-parallel training.

Tasks

  1. Use torch.utils.data.TensorDataset to wrap the dataset.

  2. Construct a data loader which operates on the wrapped dataset. Use torch.utils.data.DataLoader and illustrate the parameter batch_size.

3.3. Training

Great, our data is in place. We are ready to train our first classifier in PyTorch! We’ll train a very simple linear perceptron. Our perceptron applies a single linear PyTorch layer (see torch.nn.Linear) to the three-dimensional inputs and produces scalar outputs. This leads to four degrees of freedom in the model: Three for the “matrix” \(A\) of the linear layer and one for the bias \(b\).

For the training procedure we’ll also have to think about a loss function and an optimizer. For now:

Listing 3.3.1 Template for the module eml.perceptron.trainer.
 1## Trains the given linear perceptron.
 2#  @param i_loss_func used loss function.
 3#  @param io_data_loader data loader which provides the training data.
 4#  @param io_model model which is trained.
 5#  @param io_optimizer used optimizer.
 6#  @return loss.
 7def train( i_loss_func,
 8           io_data_loader,
 9           io_model,
10           io_optimizer ):
11  # switch model to training mode
12  io_model.train()
13
14  l_loss_total = 0
15
16  # TODO: finish implementation
17
18  return l_loss_total

Listing 3.3.1 contains a template for the module eml.perceptron.trainer. We’ll use this template to separate our training procedure from the main code.

Tasks

  1. Implement the class Model in the module eml.perceptron.model. Define a single linear layer within Model in the constructor of the class. Apply the sigmoid function after the linear layer in the forward function (see torch.nn.Sigmoid).

  2. Implement a training loop and print the total training loss after every epoch. For the time being, implement the training loop directly in your main function.

  3. Move the training loop to the module eml.perceptron.trainer. Use the template in Listing 3.3.1 to guide your implementation.

3.4. Visualization

Now, let’s professionalize our visualization efforts. We’d like to get visual feedback from our classifier during training. In our case, this means that we follow Section 3.1 and plot the points colored by their predicted labels.

Listing 3.4.1 Template for the module eml.vis.points.
 1import torch
 2import matplotlib.pyplot
 3
 4## Plots the given points and colors them by the predicted labels.
 5#  It is assumed that a prediction larger than 0.5 corresponds to a red point.
 6#  All other points are black.
 7#  @param i_points points in R^3.
 8#  @param io_model model which is applied to derive the predictions.
 9def plot( i_points,
10          io_model ):
11  # switch to evaluation mode
12  io_model.eval()
13
14  with torch.no_grad():
15    # TODO: finish implementation

Tasks

  1. Implement a visualization module in eml.vis.points. Use the template in Listing 3.4.1 to guide your implementation.

  2. Monitor your training process by visualizing the labeled points after every \(x\) epochs. Use reasonable number for \(x\). 😉