= Running Jupyter Notebooks with Multiple Configurations Using HTCondor =

This page documents a practical workflow for running **multiple training configurations of a Jupyter notebook in parallel using HTCondor**.
The approach is particularly useful once a model or pipeline is stable and you want to scan over several configurations (e.g. different losses, datasets, or hyperparameters) without running them sequentially.

== Motivation ==

Neural network development is often done in Jupyter notebooks because they are:
* Easy to prototype
* Interactive
* Good for visual inspection of outputs

However, once development stabilizes, running multiple configurations sequentially can be inefficient.

'''Example:'''
* 2 loss functions × 4 samples = 8 training runs
* ~30 minutes per run
* Sequential execution → ~4 hours
* Parallel execution with HTCondor → ~30 minutes wall time

This document describes how to:
# Convert a notebook into a script-friendly format
# Make it accept command-line arguments
# Submit multiple runs to HTCondor in parallel

== Overview of the Workflow ==

To run notebook-based training jobs in parallel using HTCondor, you need to:

# Use Jupytext for the notebook
# Configure the notebook to accept command-line arguments
# Create an HTCondor submission file ('''.sub''')
# Submit and monitor the jobs

Each step is described in detail below.

== Step 1: Use Jupytext for the Notebook ==

For HTCondor execution, it is easiest to use **Jupytext notebooks**.

=== Why Jupytext? ===

* The notebook is stored as a '''.py''' file and can be executed directly as a script
* Still fully usable as a notebook in Jupyter
* More suitable for version control (Git)
* Avoids conversion steps when running on batch systems

=== Creating a Jupytext Notebook ===

* From the Jupyter main page, click the option to create a **Jupytext notebook**
* Select the kernel after creation

=== Opening an Existing Jupytext Notebook ===

* Right-click the file
* Select '''Open as Jupytext Notebook'''

=== Converting an Existing Notebook ===

If you already have a standard '''.ipynb''' notebook:
* Create a new Jupytext notebook
* Copy–paste cells from the old notebook
* This is often a good opportunity to clean up and refactor the code

== Step 2: Configure the Notebook to Accept Arguments ==

By default, notebooks do not accept command-line arguments.
To enable this, add an argument-parsing block at the **top of the notebook/script**.

=== Example Argument Parsing Code ===

<pre>
import argparse
import sys

def get_args():
"""Get the needed arguments."""

if not 'launcher' in sys.argv[0]:
parser = argparse.ArgumentParser()
parser.add_argument("--sample", type=int, required=True)
parser.add_argument("--rotloss", type=bool, required=True)

args = parser.parse_args()
isample = args.sample
rot_loss = args.rotloss
else:
# Default values when running interactively as a notebook
isample = 0
rot_loss = False

return isample, rot_loss
</pre>

=== Usage ===

When executed as a script:
<pre>
./train_encoder.py --sample 0 --rotloss False
</pre>

When executed as a notebook:
* Default values are used
* No command-line arguments are required

This allows the same file to work both:
* Interactively (Jupyter)
* Non-interactively (HTCondor)

== Step 3: Create an HTCondor Submission File ==

Create a submission file (e.g. '''autoenc.sub''') describing how the jobs should run.

=== Example Submission File ===

<pre>
executable = /data/incaem/scratch_nvme/eriksen/miniforge3/envs/py4dstem/bin/python
arguments = /nfs/pic.es/user/e/eriksen/proj/posthack/train_encoder.py --sample $(sample) --rotloss False

# Logs
output = logs/train_encoder_$(sample)_$(ClusterId).$(ProcId).out
error = logs/train_encoder_$(sample)_$(ClusterId).$(ProcId).err
log = logs/train_encoder_$(sample)_$(ClusterId).$(ProcId).log

request_gpus = 1
request_memory = 8GB

queue sample in (0 1 2 3)
</pre>

=== Notes ===

* Use the **full path** to:
** The Python executable from the desired conda environment
** The training script
* HTCondor variables (e.g. '''$(sample)''') are passed as arguments
* Each queued value corresponds to one independent training run
* Logs are separated per job

== Step 4: Submit and Monitor the Jobs ==

=== Submit ===

From the directory containing the '''.sub''' file:

<pre>
condor_submit autoenc.sub
</pre>

=== Monitor ===

<pre>
condor_q
</pre>

or use standard HTCondor monitoring tools as needed.

=== Important Notes ===

* Always write outputs (e.g. model weights, checkpoints) to an '''absolute path'''
* Ensure output directories exist before submission
* Avoid relying on notebook state (each job runs independently)

== Summary ==

This workflow enables:
* Notebook-based development
* Script-based batch execution
* Efficient parallel training with HTCondor

It is not perfect, but it is a practical and robust solution for scaling notebook-based workflows once development stabilizes.

2024-04-25T08:56:12Z

Eriksen:

tldr;
Add the lines:
<pre>
export LD_LIBRARY_PATH=/data/agn/scratch2/eriksen/galfit_depend/:$LD_LIBRARY_PATH
export PATH=$PATH:/data/agn/scratch2/eriksen/galfit/bin
</pre>

then the following works:
<pre>

galfit /data/agn/scratch2/eriksen/galfit/galfit-example/EXAMPLE/galfit.feedme
</pre>

What is the problem:
The problem with Galfit, is the code was last distributed on April 23, 2013
in binary form. Only downloading the binary (Redhat/Enterprise 64) cause a
problem with missing libsnl library. This note explains how to get galfit
working at PIC.

The fastest way is downloading the Redhat version and compile libsnl from
source. This is relatively easy, but as indicated above, we have both
downloaded and compiled this dependency. Setting these variables in your
.bashrc makes the system find the binary ($PATH) and the library
($LD_LIBRARY_PATH). Note that you might need to specify these when
submitting jobs through HTCondor.

For documentation and in the case someone wants their own installation or
later needs to reproduce this:

1) Download the Galfit source code: <br>
wget https://users.obs.carnegiescience.edu/peng/work/galfit/galfit3-enterprise64.tar.gz
tar -xf galfit3-enterprise64.tar.gz

2) Compile and install libsnl

Download libsnl:
<pre>
wget https://github.com/thkukuk/libnsl/releases/download/v2.0.1/libnsl-2.0.1.tar.xz
tar -xf libnsl-2.0.1.tar.xz
</pre>

Compile libsnl:
<pre>
./configure --prefix /data/agn/scratch2/eriksen/galfit_depend
make
make install
</pre>