JupyterHub

Introduction

PIC offers a service for running Jupyter notebooks on CPU or GPU resources. This service is primarily thought for code developing rather than massive data processing. The usage is similar to running notebooks on your personal computer but offers the advantage of developing and testing your code on different hardware configurations.

Since the service is strictly thought for development and small scale testing tasks, there is a shutdown policy in place: 1) Your session will be closed and removed after 2 hours of idle keyboard. 2) The maximum duration for a session (idle or active) is limited to 48h. In practice that means that you should estimate the test data volume that you work with during a session to be able to be processed in less than 48 hours.

How to connect to the service

Got to jupyter01.pic.es to see your login screen.

Sign in with your PIC user credentials. This will prompt you to the following screen.

Here you can choose the hardware configuration for your Jupyter session. After choosing a configuration and pressing start the next screen will show you the progress of the initialisation process. Keep in mind that a job containing your Jupyter session is actually sent to the HTCondor queuing system and waiting for available resources before being started. This can take a bit but shouldn't be more than a minute.

In the next screen you can choose the tool that you want to use for your work: a Python notebook, a Python console or a plain bash terminal.

Your python environments should appear under Notebook and Console headers. In a later section we will show you how to create a new environment and to remove an existing one.

Terminate your session and logout

It is important that before you log out you terminate your job. In order to do so, go to the top page menu "File -> Hub Control Panel" and you will see the following screen.

Here click on the Stop My Server button. After that you can log out by clicking the logout button in the right upper corner.

Virtual environments with conda

On the homepage of your Jupyter session, click on the terminal button on the session dashboard on the right to open a bash terminal.

Create a new environment

Here, you can create your new environment:

[neissner@td110 ~]$ /path/to/anaconda/bin/conda create -n your_env python=3 anaconda

If you don't have a conda installation at your disposal, ask your project liaison at PIC for indications. In this example, an anaconda environment has been created. For instance, for a TensorFlow environment replace anaconda by tensorflow, and for a TensorFlow GPU environment write tensorflow-gpu.

Keep in mind that the anaconda environments are rather heavy (as compared to e.g. a bare virtualenv) because they already include a number of useful packages for scientific data analysis, e.g. NumPy, MathPlotLib, etc.

Now let's go on and activate the base environment:

[neissner@td110 ~]$ eval "$(/path/to/anaconda/bin/conda shell.bash hook)"

Initialize conda:

(base) [neissner@td110 ~]$ conda init

This actually changes the .bashrc file in your home directory in order to activate the base environment on login. To avoid that the base environment is activated every time you log on to a node, run:

(base) [neissner@td110 ~]$ conda config --set auto_activate_base false

Your new environment is now created but not yet integrated into Jupyter. See the next section to do that. For now you can deactivate the base environment and exit the terminal.

(base) [neissner@td110 ~]$ conda deactivate
[neissner@td110 ~]$ exit

Bind your environment to Jupyter

Log into Jupyter, start a session. From the session dashboard choose the bash terminal.

Inside the terminal, activate your environment:

[neissner@td110 ~]$ conda activate your_env
(your_env) [neissner@td110 ~]$

Install the ipykernel package and link it to the environment (it might already be installed):

(your_env) [neissner@td110 ~]$ conda install -n your_env ipykernel
Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

(your_env) [neissner@td110 ~]$ ipython kernel install --user --name=your_env
Installed kernelspec your_env in /nfs/pic.es/user/n/neissner/.local/share/jupyter/kernels/your_env

Deactivate your environment:

(your_env) [neissner@td110 ~]$ conda deactivate

Now you can exit the terminal. After refreshing the Jupyter page your new environments/kernels appear in the board. In this example test has been used for your_env.

Remove environment/kernel from Jupyter

Log onto Jupyter, start a session and from the session dashboard choose the bash terminal. To remove your environment/kernel from Jupyter run:

 [neissner@td110 ~]$ jupyter kernelspec uninstall your_env
Kernel specs to remove:
  your_env                  /nfs/pic.es/user/n/neissner/.local/share/jupyter/kernels/your_env
Remove 1 kernel specs [y/N]: y
[RemoveKernelSpec] Removed /nfs/pic.es/user/n/neissner/.local/share/jupyter/kernels/your_env

Keep in mind that, although not available in Jupyter anymore, the environment still exists in your home directory under .conda/envs:

[neissner@ui04 ~]$ ls .conda/envs/
your_env

If you want to remove the environment completely you have to remove it from the filesystem.