Skip to main content

HPC Machine Learning Tutorial

Objective

The objective of this tutorial is to familiarize the user with some of the nuances of the WAVE HPC using a basic machine learning example, including:

  • Working with external datasets
  • Working with pre-installed modules
  • Using Slurm to access compute nodes

This tutorial is an adaptation of the NumPy Tutorial from Tensorflow.org.

To run this tutorial, it is assumed that you already have access to the WAVE HPC with a user account and the ability to open a terminal session on one of the login nodes in the WAVE cluster. See WAVE HPC User Guide - Accessing the HPC if you require help on accessing the HPC.

TensorFlow

Download the dataset into a local filesystem

Let's start by downloading the MNIST dataset to a local directory where you have access.

mkdir /WAVE/<path to your own dataset directory>/mnist
wget -O /WAVE/<path to your own dataset directory>/mnist/mnist.npz https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
ls -l /WAVE/<path to your own dataset directory>/mnist/

Note: In this tutorial, you will need to substitute the appropriate project and dataset subdirectories. A possible workaround, if you are just getting started, is to replace the dataset path directory with a directory where you have access to store both your dataset and project. More information can be found in the section Managing Files.

At this point, you should have a local copy of the external data.

[<username>@login2 ~]$ ls -l /WAVE/datasets/<your dataset directory>/mnist/
total 11224
-rw-rw-r--. 1 <username> <group> 11490434 May 30  2018 mnist.npz
[<username>@login2 ~]$

Establish a projects folder

While we are at it, let's establish a subdirectory within our projects folder to hold our working files and switch to that folder.

mkdir -p /WAVE/<path to your own dataset directory>/mnist-tutorial
cd /WAVE/<path to your own dataset directory>/mnist-tutorial
# At this point we should be working out of the “mnist-tutorial” projects folder
[<username>@login2 mnist-tutorial]$ pwd
/WAVE/<path to your own dataset directory>/mnist-tutorial
[<username>@login2 mnist-tutorial]$

Working with pre-installed modules

The WAVE HPC has pre-installed software covering many parallel and scientific computing needs, which are available via modules. Use the following command to see which modules are available:

module available

or

module avail

In this tutorial, we will be using TensorFlow, which is an open-source platform for machine learning.

Load TensorFlow module

There are several versions of TensorFlow available on the WAVE HPC. We will use the latest, default version. Use the following command to load the TensorFlow software:

module load TensorFlow

At this point, we should have TensorFlow, plus some other dependent modules, installed and ready for our use. Let's check that, first by listing what modules are installed. We could do that with the module list command, which will list all the software packages that came with the TensorFlow module. That is interesting, but what is probably more important is specific packages required by our program. Are they there, and are they compatible? Let's write a quick Python script that will check the installation for those specific modules.

In the Python code below, we are interested in importing TensorFlow and NumPy. Let's use the following code to check the software installation.

# check versions
import tensorflow
print('tensorflow: %s' % tensorflow.__version__)
import numpy
print('numpy: %s' % numpy.__version__)

Using an editor, we will add the code to a file called versions.py. Note that we are executing from the /WAVE/<path to your own dataset directory>/mnist-tutorial directory.

The following command will execute our file:

python versions.py

Below is the result of that command:

[<username>@login2 mnist-tutorial]$ python versions.py
2021-10-13 13:08:24.307690: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
tensorflow: 2.3.2
numpy: 1.17.3
[<username>@login2 mnist-tutorial]$

At this point, we have a local copy of the data and we are able to load the required modules. It should be noted that apart from this approach the user could also use a terminal (e.g. WAVE Shell Access), type "python" and run the previous commands right away.

Now we turn our attention to our sample Python model.

Sample Python program

Using an editor, we'll add the following code to a file called mnist-tutorial.py. This is our sample Python program. This program is slightly different from the tutorial presented in Load NumPy data; the main difference lies in how the data is loaded. The TensorFlow.org tutorial relies on a Keras utility that will load data from an external URL. In the HPC, however, we will be running this program from a compute node, which does not have external internet access. As such, we imported the data from a local copy stored in the datasets filesystem, and we will access it from there.

Note: It is not the intention of this tutorial to teach the user how to use TensorFlow for machine learning. Instead, we are focused specifically on the nuances of running something like TensorFlow within the HPC. If you are interested in understanding more about how this code does machine learning, I would refer you back to TensorFlow.org.

# Set up
import numpy as np
import tensorflow as tf
import os
 
# Load Data from .npz file
data_dir = '/WAVE/datasets/<your dataset directory>/mnist/'
from tensorflow.keras.datasets import mnist
(train_examples, train_labels), (test_examples, test_labels) = mnist.load_data(path=data_dir+'mnist.npz')

# Load NumPy arrays with tf.data.Dataset
train_dataset = tf.data.Dataset.from_tensor_slices((train_examples, train_labels))
test_dataset = tf.data.Dataset.from_tensor_slices((test_examples, test_labels))

# Use the datasets

# Shuffle and batch the datasets
BATCH_SIZE = 64
SHUFFLE_BUFFER_SIZE = 100

train_dataset = train_dataset.shuffle(SHUFFLE_BUFFER_SIZE).batch(BATCH_SIZE)
test_dataset = test_dataset.batch(BATCH_SIZE)

# Build and train a model
model = tf.keras.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dense(10)
])

model.compile(optimizer=tf.keras.optimizers.RMSprop(),
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['sparse_categorical_accuracy'])
model.fit(train_dataset, epochs=10)
model.evaluate(test_dataset)

We are now ready to run our tutorial program. Because this is such a simple program, we could run it from a login node, but we do not want to do that. The login nodes in the WAVE HPC are for setting up and configuring our environment. They are not appropriate for compute-intensive tasks. Instead, the login nodes are our gateway to the compute resources in the WAVE cluster. We will use a resource scheduling program called Slurm to gain access to those compute resources.

Using Slurm to access compute nodes

Slurm provides us with the ability to execute a job on the backend compute nodes from either an interactive or batch perspective. We will look at both approaches here, but in general a batch approach is more appropriate for longer, compute-intensive tasks. Here you can find links to both approaches:

PyTorch

Download the dataset into a local filesystem

mkdir /WAVE/<path to your own dataset directory>/mnist-pytorch
wget -O /WAVE/<path to your own dataset directory>/mnist-pytorch/mnist.npz https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
ls -l /WAVE/<path to your own dataset directory>/mnist-pytorch/

Note: As with the TensorFlow tutorial, substitute the appropriate project and dataset subdirectories for your account.

Establish a projects folder

mkdir -p /WAVE/<path to your own dataset directory>/mnist-pytorch-tutorial
cd /WAVE/<path to your own dataset directory>/mnist-pytorch-tutorial

Load the PyTorch module

module avail pytorch
module load PyTorch

Verify the installation:

# check versions
import torch
print('torch: %s' % torch.__version__)
import numpy
print('numpy: %s' % numpy.__version__)
python versions.py

Sample Python program

Since PyTorch's torchvision.datasets.MNIST normally downloads from the internet, and compute nodes do not have external internet access (or: for large/reusable datasets, downloading from a login node ahead of time is still recommended; see Working with External Datasets), we load MNIST from the local .npz file instead.

# Set up
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset

# Load Data from .npz file
data_dir = '/WAVE/datasets/<your dataset directory>/mnist-pytorch/'
with np.load(data_dir + 'mnist.npz') as data:
    train_examples = data['x_train']
    train_labels = data['y_train']
    test_examples = data['x_test']
    test_labels = data['y_test']

# Convert to tensors and normalize
train_x = torch.tensor(train_examples, dtype=torch.float32) / 255.0
train_y = torch.tensor(train_labels, dtype=torch.long)
test_x = torch.tensor(test_examples, dtype=torch.float32) / 255.0
test_y = torch.tensor(test_labels, dtype=torch.long)

train_dataset = TensorDataset(train_x, train_y)
test_dataset = TensorDataset(test_x, test_y)

BATCH_SIZE = 64
train_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=BATCH_SIZE)

# Build a model
model = nn.Sequential(
    nn.Flatten(),
    nn.Linear(28 * 28, 128),
    nn.ReLU(),
    nn.Linear(128, 10)
)

criterion = nn.CrossEntropyLoss()
optimizer = optim.RMSprop(model.parameters())

# Train
for epoch in range(10):
    for xb, yb in train_loader:
        optimizer.zero_grad()
        out = model(xb)
        loss = criterion(out, yb)
        loss.backward()
        optimizer.step()
    print(f'Epoch {epoch+1} loss: {loss.item():.4f}')

# Evaluate
correct = 0
total = 0
with torch.no_grad():
    for xb, yb in test_loader:
        out = model(xb)
        preds = out.argmax(dim=1)
        correct += (preds == yb).sum().item()
        total += yb.size(0)
print(f'Test accuracy: {correct/total:.4f}')

Using Slurm to access compute nodes

Same as above; see Batch Slurm Jobs and Interactive Slurm Jobs.

HuggingFace Transformers

Download the model and dataset into a local filesystem

Since compute nodes do not have (or: should not rely on) external internet access, pretrained models and tokenizers should be downloaded ahead of time from a login node, using HuggingFace's caching mechanism:

mkdir -p /WAVE/<path to your own dataset directory>/hf-cache
export HF_HOME=/WAVE/<path to your own dataset directory>/hf-cache

Load the required modules

module avail python
module load Python
pip install --user transformers torch

Note: If transformers is not available as a pre-installed module on your cluster, installing it via pip install --user into your home directory is the typical workaround. Check with your HPC administrator if a shared module is preferred.

Sample Python program

# sentiment_tutorial.py
import os
os.environ['HF_HOME'] = '/WAVE/datasets/<your dataset directory>/hf-cache'

from transformers import pipeline

# Load pipeline from local cache (no internet access needed on compute node)
classifier = pipeline("sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english")

texts = [
    "The WAVE HPC made this tutorial much easier to follow.",
    "I'm not sure this GPU allocation was worth the wait."
]

results = classifier(texts)
for text, result in zip(texts, results):
    print(f"{text}\n  -> {result['label']} ({result['score']:.4f})\n")
python sentiment_tutorial.py

Using Slurm to access compute nodes

Same as above; see Batch Slurm Jobs and Interactive Slurm Jobs.