Using weights and biases for storing pytorch experiments of different architectures

Question

following the basic pytorch tutorial, I'm trying to experiment with different CNN architectures (different number of layers, channels per layer, etc.) and I want to be organized so I'm trying to use wandb.

I'm looking for a simple way to try out different architectures, and save both the results (accuracies) of each architecture and the architecture itself. Currently what I have is a code snippet that allows me to save the model with it's weights, but I also need a way to reproduce my results, including reproducing the code for my class (I was able, using repr(model) save the models parameters, but not the code of the class itself, which is unfortunate).

Is there a right/better way to manage my experiments and easily reproduce my results including the code of the class? I considered using the inspect module to save the source code but I'm both not sure this is the best way and google colab doesn't support this module.

This is the code I have currently:

torch.manual_seed(0)

# MODEL DESCRIPTION
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = torch.flatten(x, 1) # flatten all dimensions except batch
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

model = Net()
model_desc = repr(model)

# TRAINING....

# EVALUATING...
accureacy = ...

# SAVE RESULTS
run = wandb.init(project='test-project', name=model_desc)

artifact = wandb.Artifact('model', type='model')
artifact.add_file(MODEL_PATH)


# 2. Save mode inputs and hyperparameters
config = run.config
config.test_number = 1
config.model_desc = model_desc

# 3. Log metrics over time to visualize performance
run.log({"accuracy": accuracy})

# 4. Log an artifact to W&B
run.log_artifact(artifact)
run.finish()

Your simplest solution is to use git and wandb. Wandb stores the git hash of the commit, so you know exactly the version of the execution of the code. — Minh-Long Luu, Mar 16 '23 at 17:03

score 2 · Answer 1 · answered Mar 16 '23 at 17:25

Scott from W&B here. We have a new feature called Launch where you can log your code with run.log_code() and then launch it via the UI.

Turn it on in https://wandb.ai/settings and update wandb
Log code with run.log_code() and see it in the Jobs tab
Create a Queue with the Launch UI
Run wandb launch-agent -q <queue-name>
Now you can send Jobs to that Queue

https://docs.wandb.ai/guides/launch

Using weights and biases for storing pytorch experiments of different architectures

1 Answers1