Public PIC Wiki - User contributions [en]

Transferring data to/from PIC

2026-05-14T15:18:18Z

Torradeflot: /* Configuration (only once) */

= How to provide data access to PIC massive storage (dCache) =

== Requirements ==

* Install and configure Rclone
* PIC credentials or macaroon

== Install Rclone ==

You can directly download the binary without installing anything. For instance, for a linux 64 bits machine:

$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip
[...]
$ unzip rclone-current-linux-amd64.zip

Or if you prefer, you can install Rclone like the next example on a Ubuntu machine:

$ cd /tmp
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'
$ sudo apt install ./rclone-current-linux-amd64.deb

== Configure Rclone ==

You should have been given some credentials and the url of a WebDAV endpoint at PIC.
With them, you just need to create the config in rclone:

$ rclone config
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n

Enter name for new remote.
name> pic

Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
1 / 1Fichier
\ (fichier)
[...]
Storage> webdav

Option url.
URL of http host to connect to.
E.g. https://example.com.
Enter a value.
url> https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE

Option vendor.
Name of the WebDAV site/service/software you are using.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
1 / Nextcloud
\ (nextcloud)
[...]
5 / Other site/service or software
\ (other)
vendor> other

=== Using your PIC credentials ===

If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank.

Option user.
User name.
In case NTLM authentication is used, the username should be in the format 'Domain\User'.
Enter a value. Press Enter to leave empty.
user> YOUR_PIC_USERNAME

Option pass.
Password.
Choose an alternative below. Press Enter for the default (n).
y) Yes, type in my own password
g) Generate random password
n) No, leave this optional password blank (default)
y/g/n> y
Enter the password:
password: YOUR_PIC_PASSWORD
Confirm the password:
Password: YOUR_PIC_PASSWORD

=== Using a Macaroon token ===

If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank

Option bearer_token.
Bearer token instead of user/pass (e.g. a Macaroon).
Enter a value. Press Enter to leave empty.
bearer_token> YOUR_MACAROON_TOKEN

=== Using an OIDC token ===

Option bearer_token.
Bearer token instead of user/pass (e.g. a Macaroon).
Enter a value. Press Enter to leave empty.
bearer_token>

Edit advanced config?
y) Yes
n) No (default)
y/n> y

Option bearer_token_command.
Command to run to get a bearer token.
Enter a value. Press Enter to leave empty.
bearer_token_command> oidc-token OIDC_AGENT_ACCOUNT_SHORTNAME

=== Review settings ===

At the end, just review the information you entered and confirm.

Edit advanced config?
y) Yes
n) No (default)
y/n> n

Configuration complete.
Options:
- type: webdav
- url: https://door04.pic.es/PATH_TO_YOUR_STORAGE_SPACE
- vendor: other
- user: YOUR_PIC_USERNAME
- pass: *** ENCRYPTED ***
Keep this "pic" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

Current remotes:

Name Type
==== ====
pic webdav

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

Once done, you can use command line to browse and download/upload data.

=== Usage ===

* List a remote PIC directory:
rclone lsd <name>:<path>`

* Download a remote directory from PIC
rclone copy <name>:<path> <local_path>

* Upload a local directory to PIC
rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path>

When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files
--check-first -P --stats-one-line --transfers <n_transfers> --size-only

If uploading lots onto directories with lots of files (>1000), please use:
--no-traverse

If uploading files larger than 200 MB, also use
--multi-thread-streams 1

If uploading very large files (>10G), also use the following to allow more time to compute the checksums
--timeout=15m

See rclone manual for more extensive documentation https://rclone.org/docs/

== Configuring oidc-agent for obtaining OIDC tokens ==

Make sure <code>oidc-agent</code> is available.

=== Load oidc-agent ===

Initialize <code>oidc-agent</code> in the terminal session.

$ eval `oidc-agent`

=== Configuration (only once) ===

To run this step you need an updated version of oidc-agent, version > 5.0.0.
'''Ask your contact for the client-secret you have to replace below'''

Configure a <code>pic-dcache</code> account to retrieve tokens from PIC. Open the URL that will show and input the code provided (or just open the QR code displayed).
After authenticating on the web browser, return to the terminal and input an encryption password twice. You'll need it when refreshing/reloading the <code>oidc-agent</code>.

$ oidc-gen -m --client-id dcache-view \
--client-secret XXXXXXXXXXXXXXXXXX \
--pub --flow=device \
--discovery-endpoint=https://idp.pic.es/realms/PIC/.well-known/openid-configuration \
--scope="openid profile offline_access" --redirect-uri=edu.kit.data.oidc-agent:/ pic-dcache

No account exists with this short name. Creating new configuration ...
Generating account configuration ...
accepted

Using a browser on any device, visit:
https://idp.pic.es/realms/PIC/device

And enter the code: ASDF-GHJK
Alternatively you can use the following QR code to visit the above listed URL.

[ QR CODE ]

Enter encryption password for account configuration 'pic-dcache':
Confirm encryption password:
Everything setup correctly!

=== Reauthenticating (if refresh token has expired) ===

If the oidc-agent process gets restarted, or iIf your refresh token expires due to inactivity, you will need to reauthenticate to retrieve further tokens

$ oidc-gen --reauthenticate pic-dcache
Enter decryption password for account config 'testtest':
Generating account configuration ...
accepted

Using a browser on any device, visit:
https://idp.pic.es/realms/PIC/device

And enter the code: ASDF-GHJK
Alternatively you can use the following QR code to visit the above listed URL.

[ QR CODE ]

Enter encryption password for account configuration 'pic-dcache' [***]:
Everything setup correctly!

=== Testing ===

After loading and configuring, you can get a token by running the following command:

$ oidc-token pic-dcache
eyJhbGciOiJSUzI1[...]4YjAwg

== Obtaining a macaroon (for contacts) ==

Macaroons are valid up to 7 days.

For downloading data (read-only permissions on the path):

<pre>
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \
https://door04.pic.es:8460/${RESTRICTED_PATH}

{
"macaroon": "MDA2MGxvY2F0aW",
"uri": {
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",
"base": "https://door04.pic.es:8460/"
}
}
</pre>

For uploading data (full permissions on the path):

<pre>
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \
-d '{"validity": "P7D"}' \
https://door04.pic.es:8460/${RESTRICTED_PATH}

{
"macaroon": "MDA2MGxvY2F0aW",
"uri": {
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",
"base": "https://door04.pic.es:8460/"
}
}
</pre>

JupyterHub

2026-04-08T08:14:19Z

Torradeflot: /* GPUs */

tldr; Connect to https://jupyter.pic.es/ . Enjoy!
= Introduction =

PIC offers a service for running Jupyter notebooks on CPU or GPU resources. This service is primarily thought for code developing or prototyping rather than data processing. The usage is similar to running notebooks on your personal computer but offers the advantage of developing and testing your code on different hardware configurations, as well as facilitating the scalability of the code since it is being tested in the same environment in which it would run on a mass scale.

Since the service is strictly thought for development and small scale testing tasks, a shutdown policy for the sessions has been put in place:

# The maximum duration for a session is 48h.
# After an idle period of 2 hours, the session will be closed.

In practice that means that you should estimate the test data volume that you work with during a session to be able to be processed in less than 48 hours.

== How to connect to the service ==

Got to [https://jupyter.pic.es jupyter.pic.es] to see your login screen.

[[File:login.png|700px|Login screen]]

Sign in with your PIC user credentials. This will prompt you to the following screen.

[[File:ScreenshotJupyterSpawn.png|700px|current]]

Here you can choose the hardware configuration for your Jupyter session. Also, you have to choose the experiment (project) you are working on during the Jupyter session. After choosing a configuration and pressing start the next screen will show you the progress of the initialisation process. Keep in mind that a job containing your Jupyter session is actually sent to the HTCondor queuing system and waiting for available resources before being started. This usually takes less than a minute but can take up to a few depending on our resource usage.

[[File:screen02.png|900px]]

In the next screen you can choose the tool that you want to use for your work: a Python notebook, a Python console or a plain bash terminal.
For the Python environment (either notebook or environment) you have two default options:
* the ipykernel version of Python 3
* the XPython version of Python 3.9, this one allows you to use the integrated debugging module.

== Terminate your session and logout ==

It is important that you terminate your session before you log out. In order to do so, go to the top page menu "'''File''' -> '''Hub Control Panel'''" and you will see the following screen.

[[File:screen04.png|600px]]

Here, click on the '''Stop My Server''' button. After that you can log out by clicking the '''Logout''' button in the right upper corner.

= Python virtual environments =

This section covers the use of Python virtual environments with Jupyter.

== Prebuilt environments ==

PIC's jupyterhub service comes with a collection of prebuilt environments located at '''/data/jupyter/software/envs'''.

The master environment located at '''/data/jupyter/software/envs/master''' is the one used to start the jupyterlab service and the default for new notebooks.

This is a non-extensive list of the packages included:
- astropy=6.1.0
- bokeh=3.4.1
- dash=2.17.0
- dask=2024.5.1
- findspark=2.0.1
- matplotlib=3.8.4
- numpy=1.26.4
- pandas=2.2.2
- pillow=10.3.0
- plotly=5.22.0
- pyhive=0.7.0
- python=3.12
- pywavelets=1.4.1
- scikit-image=0.22.0
- scikit-learn=1.5.0
- scipy=1.13.1
- seaborn=0.13.2
- statsmodels=0.14.2

== Initialize conda (we highly recommend the use of mamba/micromamba) ==

Before using conda/mamba in your bash session, you have to initialize it.
* For access to an available conda/mamba installation, please get in contact with your project liaison at PIC. He/she will give you the actual value for the '''/path/to/conda/mamba''' placeholder.
* If you want to use your own conda/mamba/micromamba installation, there are two recommended options
** '''miniforge''': a distribution with conda and mamba executables in a minimal base environment, instructions [https://github.com/conda-forge/miniforge here]
** '''micromamba''': a self-contained executable (micromamba) with no base environment, instructions [https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html here]

Log onto Jupyter and start a session. On the homepage of your Jupyter session, click on the terminal button on the session dashboard on the right to open a bash terminal.

In order to use conda/mamba/micromamba you need to intialize the shell. This initialization can be persistent, which will do some changes to your '''~/.bashrc''' file, or you can do it every time you want to use it.

Run the following command:

<pre>
eval "$(/data/astro/software/miniforge3/bin/conda shell.bash hook)"
</pre>

Then, if you are using miniforge and you want to persist the initialization:
<pre>
/data/astro/software/miniforge3/bin/mamba init
</pre>

This actually changes the .bashrc file in your home directory in order to activate the base environment on login.
To avoid that the base environment is activated every time you log on to a node, run:
<pre>
[neissner@td110 ~]$ conda config --set auto_activate_base false
</pre>

== Link an existing environment to Jupyter ==

You can find instructions on how to create your own environments, e.g. [[#Create_virtual_environments_with_venv_or_conda | here]].

Log into Jupyter, start a session. From the session dashboard choose the bash terminal.

Inside the terminal, activate your environment.

For '''conda/mamba''' environments:
* if you created the environment without a prefix:
<pre>
[neissner@td110 ~]$ mamba activate environment
(...) [neissner@td110 ~]$
</pre>
The parenthesis (...) in front of your bash prompt show the name of your environment.
* if you created the environment with a prefix:
<pre>
[neissner@td110 ~]$ mamba activate /path/to/environment
(...) [neissner@td110 ~]$
</pre>
The parenthesis (...) in front of your bash prompt show the absolute path of your environment.

For '''venv''' environments:
<pre>
[neissner@td110 ~]$ source /path/to/environment/bin/activate
(...) [neissner@td110 ~]$
</pre>

Link the environment to a Jupyter kernel. For both, '''conda/mamba''' and '''venv''':
<pre>
(...) [neissner@td110 ~]$ python -m ipykernel install --user --name=whatever_kernel_name
Installed kernelspec whatever_kernel_name in
/nfs/pic.es/user/n/neissner/.local/share/jupyter/kernels/whatever_kernel_name
</pre>

If you don't have the '''ipykernel''' module installed in your environment you may receive an error message like the one below when trying to run the previous command.
<pre>No module named ipykernel</pre>

If this is the case, you need to install it by running: '''pip install ipykernel'''

Deactivate your environment.

For conda:
<pre>
(...) [neissner@td110 ~]$ mamba deactivate
</pre>

For venv:
<pre>
(...) [neissner@td110 ~]$ deactivate
</pre>

Now you can exit the terminal. After refreshing the Jupyter page your whatever_kernel_name appears in the dashboard. In this example '''test''' has been used for whatever_kernel_name

[[File:screen05.png|700px]]

== Unlink an environment from Jupyter ==
Log onto Jupyter, start a session and from the session dashboard choose the bash terminal. To remove your environment/kernel from Jupyter run:
<pre>
[neissner@td110 ~]$ jupyter kernelspec uninstall whatever_kernel_name
Kernel specs to remove:
whatever_kernel_name /nfs/pic.es/user/n/neissner/.local/share/jupyter/kernels/whatever_kernel_name
Remove 1 kernel specs [y/N]: y
[RemoveKernelSpec] Removed /nfs/pic.es/user/n/neissner/.local/share/jupyter/kernels/whatever_kernel_name
</pre>
Keep in mind that, although not available in Jupyter anymore, the environment still exists. Whenever you need it, you can link it again.

== Create virtual environments with venv or conda ==

Before creating a new environment, please get in contact with your project liaison at PIC as there may be already a suitable environment for your needs in place.

If none of the existing environments suits your needs, you can create a new environment.
First, create a directory in a suitable place to store the environment. For single-user environments, place them in your home under ~/env. For environments that will be shared with other project users, contact your project liaison and ask him/her for a path in a shared storage volume that is visible to all of them.

Once you have the location (i.e. /path/to/env/folder), create the environment with the following commands:

For '''venv''' environments '''(recommended)'''

If your_env is installed at /path/to/env/your_env
<pre>
[neissner@td110 ~]$ cd /path/to/env
[neissner@td110 ~]$ python3 -m venv your_env
</pre>
Now you should be able to activate your environment and install additional modules
<pre>
[neissner@td110 ~]$ cd /path/to/env
[neissner@td110 ~]$ source your_env/bin/activate
(...)[neissner@td110 ~]$ pip install additional_module1 additional_module2 ...
</pre>

For '''conda/mamba''' environments
<pre>
[neissner@td110 ~]$ mamba create --prefix /path/to/env/your_env
</pre>
The list of modules (module1, module2, ...) is optional. For instance, for a python3 environment with scipy you would specify: ''python=3 scipy''

Now you should be able to activate your environment and install additional modules
<pre>
[neissner@td110 ~]$ mamba activate /path/to/env/folder/your_env
(...)[neissner@td110 ~]$ mamba install additional_module1 additional_module2 ...
</pre>
You can use pip install inside a mamba environment, however, resolving dependencies might require installing additional packages manually.

== Conda / mamba configuration ==

The behaviour of conda/mamba can be configured through the "$HOME/.condarc" file, described [https://docs.conda.io/projects/conda/en/latest/configuration.html here]. Some interesting parameters:

* envs_dirs: The list of directories to search for named environments. E.g.: different locations where you created environments

envs_dirs:
- /data/pic/scratch/torradeflot/envs
- /data/astro/scratch/torradeflot/envs
- /data/aai/scratch/torradeflot/envs

* pkgs_dirs: Folder where to store conda packages

pkgs_dirs:
- /data/aai/scratch_ssd/torradeflot/pkgs
- /data/aai/scratch/torradeflot/pkgs
- /data/pic/scratch/torradeflot/pkgs

if `pkgs_dirs` and `envs_dirs` are in the same storage, conda will use hard links, thus optimizing the disk space.

= Jupyterlab user guide =

You can find the official documentation of the currently installed version of jupyterlab (3.6) [https://jupyterlab.readthedocs.io/en/4.2.x/ here], there you will find instruction on how to
* [https://jupyterlab.readthedocs.io/en/4.2.x/user/commands.html Access the command palette]
* [https://jupyterlab.readthedocs.io/en/4.2.x/user/toc.html Build a Table Of Contents]
* [https://jupyterlab.readthedocs.io/en/4.2.x/user/debugger.html Debug your code]

A set of non-official jupyterlab extensions are installed to provide additional functionalities

== Sidecar Apps: Remote desktop & Visual Studio IDE ==
Further you see an icon with a "D" - desktop, this one starts a VNC session that allows the use of programs with graphical user interfaces.

You can also find the icon of Visual Studio, an integrated development environment.

[[File:ScreenshotJupyterlab20231103.png|700px]]

== Jupytext ==
Pair your notebooks with text files to enhance version tracking.
https://jupytext.readthedocs.io

'''Example'''

If you had a notebook (.ipynb file) containing only the cell below, tracked in a git repository

<pre>
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
plt.imshow(np.random.random([10, 10]))
</pre>

Different executions of the cell would produce different images, and the images are embedded in a pseudo-binary format inside the notebook file. In this case, doing a '''git diff''' of the .ipynb file would produce a huge output (because the image changed), even if there wasn't any change in the code. It is thus convenient to sync the notebook with a text file (e.g. a .py script) using the jupytext extension and track this one with git. The outputs, including images, as well as some additional metadata, won't be added to the synced text file. So in the case of different executions of the same notebook, the diff will always be empty.

== Git ==
Sidebar GUI to git repo management
https://github.com/jupyterlab/jupyterlab-git

== Variable inspector ==
Variable Inspector provides an interactive interface for inspecting the current state of variables in a JupyterLab session. It allows users to view variable names, types, shapes, and values in a structured table, facilitating exploratory analysis and debugging workflows similar to variable inspection tools available in environments such as MATLAB.

https://github.com/jupyterlab-contrib/jupyterlab-variableInspector

== Jupyter server proxy ==

This extension is installed in PIC's jupyter environment and it is used to be able to access network/web services running on the same host as the jupyterlab server from outside through the "https://jupyter.pic.es/user/{username}/proxy/{port}" URL.

Full documentation here: https://jupyter-server-proxy.readthedocs.io/en/latest/index.html
...

= Tips & Tricks =

== Software of particular interest ==

=== ROOT ===

Using ROOT from a jupyter notebook needs to do some tricks.

Environment creation and kernel installation

$ micromamba env create -p /data/pic/scratch/torradeflot/envs/mcdata root ipykernel
$ micromamba activate mcdata
$ python -m ipykernel install --user --name mcdata

After doing this ROOT can be imported from a python shell, but it does not work from a notebook.

* ROOT uses JIT compilation with Cling: https://root.cern/cling/
* conda provides it's own set of compilation tools: gcc, gxx, fortran
* Some environment variables are necessary to ensure that these two pieces work well together:
** PATH: to be able to find the compiler tools
** CONDA_BUILD_SYSROOT: needed to configure the compiler call

These environment variables are not propagated to the notebook, because they are set by .bashrc conda activate, ... so we need to explicitly set them in the notebook.

import os
import sys
from pathlib import Path
bin_dir = Path(sys.executable).parent
os.environ['PATH'] = f'{bin_dir}:{os.environ['PATH']}'
os.environ['CONDA_BUILD_SYSROOT'] = str(bin_dir.parent / 'x86_64-conda-linux-gnu/sysroot')

=== SageMath ===

[https://www.sagemath.org/ SageMath] is particularly interesting for Cosmology because it allows symbolic calculations, e.g. deriving the equations of motions for the scale factor starting from a customised space-time metric.

'''Standard cosmology examples'''

* The Friedman equations for the FLRW solution of the Einstein equations.
You can find the corresponding Notebook in any PIC terminal at '''/data/astro/software/notebooks/FLRW_cosmology.ipynb'''

* The notebook you can find at '''/data/astro/software/notebooks/FLRW_cosmology_solutions.ipynb''' uses known analytical solutions of the FLRW cosmology and produces this image for the evolution of the scale factor:

[[File:Screenshot_Sage05.png|300px]]

* The notebook you can find at '''/data/astro/software/notebooks/Interior_Schwarzschild.ipynb''' shows the formalism for the interior Schwarzschild metric and displays the solutions for density and pressure of a static celestial object that is sufficiently larger than its Schwarzschild radius. The pressure for an object with constant density id shown in the image:

[[File:Screenshot_Sage06.png|300px]]

'''Enabling SageMath environment in Jupyter'''

If you have never initialized mamba, run:

<pre>
[<user>@<hostname> ~]$ /data/astro/software/centos7/conda/mambaforge_4.14.0/bin/mamba init
[<user>@<hostname> ~]$ conda config --set auto_activate_base false
</pre>

After that you can enable SageMath for its use in a Jupyter notebook session:

<pre>
[<user>@<hostname> ~] mamba activate /data/astro/software/envs/sage
(/data/astro/software/envs/sage) [<user>@<hostname> ~]$ python -m ipykernel install --user --name=sage
....
(/data/astro/software/envs/sage) [<user>@<hostname> ~]$ mamba deactivate
</pre>

This creates a file in you home '''~/.local/share/jupyter/kernels/sage/kernel.json'''
which has to be modified to look like this:

<pre>
{
"argv": [
"/data/astro/software/envs/sage/bin/sage",
"--python",
"-m",
"sage.repl.ipython_kernel",
"-f",
"{connection_file}"
],
"display_name": "sage",
"language": "sage",
"metadata": {
"debugger": true
}
}
</pre>

Next time you go to your Jupyter dashboard you will find the sage environment listed there.

== Dask ==
Dask supports parallel computations in Python. The PIC Jupyterlab has an extension for launching
your own Dask clusters. For more information, see [[Dask|Dask documentation]].

== Using a Singularity image as a Jupyter kernel ==

In some projects, the software stack is provided as a Singularity image. In such cases, it can be convenient to use this image directly as a Jupyter kernel, allowing notebooks on jupyter.pic.es to run within the same controlled software environment.

To be used as a Jupyter kernel, the Singularity image must satisfy certain requirements. These depend on the programming language used inside the notebook.

The singularity image needs to have the '''python''' and the '''ipykernel''' module installed.

* Create the folder that will host the kernel definition

mkdir -p $HOME/.local/share/jupyter/kernels/singularity

* Create the '''kernel.json''' file inside it with the following content:

{
"argv": [
"singularity",
"exec",
"--cleanenv",
"/path/to/the/singularity/image.sif",
"python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"language": "python",
"display_name": "singularity-kernel"
}

Refresh or start the jupyterlab interface and the singularity kernel should appear in the launcher tab

== GPUs ==

The GPUs that are assigned to your job are listed in the environment variable CUDA_VISIBLE_DEVICES. In a terminal run "echo $CUDA_VISIBLE_DEVICES". The environment variable contains a list of comma-separated GPU ids. With this you will already know how many gpus are assigned to your job. If the variable does not exist, there are no gpus assigned to the job

[torradeflot@gpu05 ~]$ echo $CUDA_VISIBLE_DEVICES
GPU-a2361e9d-c520-684b-b21b-60cc3a59b05b

You can track the GPU utilization (memory and load) using the GPU Dashboard in the left sidebar

[[File:GPU_utilization_20260408.png|700px]]

== Code samples ==

A repository with sample code can be found here: https://gitlab.pic.es/services/code-samples/

== Running notebooks through HTCondor ==
After developing a notebook, you might want to run it as a script with different configurations. The
following documentation explains

[[notebook_htcondor|how to run a notebook through HTCondor.]]

= Troubleshooting =

== Logs ==

The log files for the jupyterlab server are stored in "~/.jupyter". The log files will be created once the jupyter lab server job is finished.

== Clean workspaces ==

Jupyterlab stores the workspace status in the "~/.jupyter/lab/workspaces" folder. If you want to start with a fresh (empty) workspace, delete all the content of this folder before launching the notebook.

cd ~/.jupyter/lab/workspaces
rm *

= Known problems =

== HOME folder is full ==

Your HOME folder located at "/nfs/pic.es/user/X/XYZTUV" has a quota assigned, usually 25GB.

If you see the error message "Disk quota exceeded" when spawning a notebook server or a generic 500 error, it could be because your $HOME folder is full. Log in to "ui.pic.es" (through SSH) and run "quota" to check the usage vs quota. If it is full you'll have to free up space.

== Error 500: Internal Server Error ==

This is a generic error. Means that the jupyterlab server failed. This could be for different reasons:

* [[#HOME_folder_is_full|Your HOME folder is full]]

== 504 Gateway timeout ==

The notebook job is running in HTCondor but the user can not access the notebook server. Ultimately a 504 error is received.

This is probably because there's some error when starting the jupyterlab server. First of all, shutdown the notebook server and [[#Logs|check the logs]] to better identify the problem. If you don't see the source of the error, try to [[#Clean_workspaces|clean the workspaces]] and launch a notebook again.

== Loading libraries is very slow ==
Conda environments at storage using hard drives can be extremely slow to load. If you encounter this problem, please ask
your support contact for a "SSD scratch" location to store environments. Currently this service is being tested and
deployed as needed.

== Spawn failed: The 'ip' trait of a PICCondorSpawner instance expected a unicode string, not the NoneType None ==

Jupyterhub could not get the host name from HTCondor's stdout, because it didn't match the expected regular expression.

This error happens randomly from time to time, it does not imply any major problem in any of the services.

Try to request a new notebook server.

== 403 : Forbidden. XSRF cookie does not match POST argument ==

The value of the "_xsrf" cookie sent by the browser does not match the expected value. This could be for many reasons: temporary high load on the server, race conditions, temporary network unstability, many open tabs in the browser, etc.

In general it can be solved by closing all tabs pointing to "jupyter.pic.es", cleaning the cookies and connecting back to "jupyter.pic.es"

== Proper usage of X509 based proxies ==

We found recently that the usage of proxies within a Jupyter session might cause problems because the environment changes certain standard locations such as '''/tmp'''

For a correct functioning please create the proxy the following way, example for Virgo:

<pre>
[<user>@<hostname> ~]$ /bin/voms-proxy-init --voms virgo:/virgo/ligo --out ./x509up_u$(id -u)
[<user>@<hostname> ~]$ export X509_USER_PROXY=./x509up_u$(id -u)
[<user>@<hostname> ~]$ ls /cvmfs/ligo.osgstorage.org
ls: cannot access /cvmfs/ligo.osgstorage.org: Permission denied
</pre>

Here the proxy cannot properly be located. Therefore we have to put the complete path into the variable:

<pre>
[<user>@<hostname> ~]$ pwd
/nfs/pic.es/user/<letter>/<user>
[<user>@<hostname> ~]$ export X509_USER_PROXY=/nfs/pic.es/user/<letter>/<user>/x509up_u$(id -u)
[<user>@<hostname> ~]$ ls /cvmfs/ligo.osgstorage.org
frames powerflux pycbc test_access
</pre>

2025-05-19T10:22:20Z

Torradeflot: /* Spawn failed: The 'ip' trait of a PICCondorSpawner instance expected a unicode string, not the NoneType None */

= Introduction =

PIC offers a service for running Jupyter notebooks on CPU or GPU resources. This service is primarily thought for code developing or prototyping rather than data processing. The usage is similar to running notebooks on your personal computer but offers the advantage of developing and testing your code on different hardware configurations, as well as facilitating the scalability of the code since it is being tested in the same environment in which it would run on a mass scale.

Since the service is strictly thought for development and small scale testing tasks, a shutdown policy for the sessions has been put in place:

# The maximum duration for a session is 48h.
# After an idle period of 2 hours, the session will be closed.

In practice that means that you should estimate the test data volume that you work with during a session to be able to be processed in less than 48 hours.

== How to connect to the service ==

Got to [https://jupyter.pic.es jupyter.pic.es] to see your login screen.

[[File:login.png|700px|Login screen]]

Sign in with your PIC user credentials. This will prompt you to the following screen.

[[File:ScreenshotJupyterSpawn.png|700px|current]]

Here you can choose the hardware configuration for your Jupyter session. Also, you have to choose the experiment (project) you are working on during the Jupyter session. After choosing a configuration and pressing start the next screen will show you the progress of the initialisation process. Keep in mind that a job containing your Jupyter session is actually sent to the HTCondor queuing system and waiting for available resources before being started. This usually takes less than a minute but can take up to a few depending on our resource usage.

[[File:screen02.png|900px]]

In the next screen you can choose the tool that you want to use for your work: a Python notebook, a Python console or a plain bash terminal.
For the Python environment (either notebook or environment) you have two default options:
* the ipykernel version of Python 3
* the XPython version of Python 3.9, this one allows you to use the integrated debugging module.

Further you see an icon with a "D" - desktop, this one starts a VNC session that allows the use of programs with graphical user interfaces.

Also, recently you can find the icon of Visual Studio, an integrated development environment.

[[File:ScreenshotJupyterlab20231103.png|700px]]

Your python environments should appear under Notebook and Console headers. In a later section we will show you how to create a new environment and to remove an existing one.

== Terminate your session and logout ==

It is important that you terminate your session before you log out. In order to do so, go to the top page menu "'''File''' -> '''Hub Control Panel'''" and you will see the following screen.

[[File:screen04.png|600px]]

Here, click on the '''Stop My Server''' button. After that you can log out by clicking the '''Logout''' button in the right upper corner.

= Python virtual environments =

This section covers the use of Python virtual environments with Jupyter.

== Initialize conda (we highly recommend the use of mamba/micromamba) ==

Before using conda/mamba in your bash session, you have to initialize it.
* For access to an available conda/mamba installation, please get in contact with your project liaison at PIC. He/she will give you the actual value for the '''/path/to/conda/mamba''' placeholder.
* If you want to use your own conda/mamba/micromamba installation, there are two recommended options
** '''miniforge''': a distribution with conda and mamba executables in a minimal base environment, instructions [https://github.com/conda-forge/miniforge here]
** '''micromamba''': a self-contained executable (micromamba) with no base environment, instructions [https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html here]

Log onto Jupyter and start a session. On the homepage of your Jupyter session, click on the terminal button on the session dashboard on the right to open a bash terminal.

In order to use conda/mamba/micromamba you need to intialize the shell. This initialization can be persistent, which will do some changes to your '''~/.bashrc''' file, or you can do it every time you want to use it.

Run the following command:

<pre>
eval "$(/data/astro/software/miniforge3/bin/conda shell.bash hook)"
</pre>

Then, if you are using miniforge and you want to persist the initialization:
<pre>
/data/astro/software/miniforge3/bin/mamba init
</pre>

This actually changes the .bashrc file in your home directory in order to activate the base environment on login.
To avoid that the base environment is activated every time you log on to a node, run:
<pre>
[neissner@td110 ~]$ conda config --set auto_activate_base false
</pre>

== Link an existing environment to Jupyter ==

You can find instructions on how to create your own environments, e.g. [[#Create_virtual_environments_with_venv_or_conda | here]].

Log into Jupyter, start a session. From the session dashboard choose the bash terminal.

Inside the terminal, activate your environment.

For '''conda/mamba''' environments:
* if you created the environment without a prefix:
<pre>
[neissner@td110 ~]$ mamba activate environment
(...) [neissner@td110 ~]$
</pre>
The parenthesis (...) in front of your bash prompt show the name of your environment.
* if you created the environment with a prefix:
<pre>
[neissner@td110 ~]$ mamba activate /path/to/environment
(...) [neissner@td110 ~]$
</pre>
The parenthesis (...) in front of your bash prompt show the absolute path of your environment.

For '''venv''' environments:
<pre>
[neissner@td110 ~]$ source /path/to/environment/bin/activate
(...) [neissner@td110 ~]$
</pre>

Link the environment to a Jupyter kernel. For both, '''conda/mamba''' and '''venv''':
<pre>
(...) [neissner@td110 ~]$ python -m ipykernel install --user --name=whatever_kernel_name
Installed kernelspec whatever_kernel_name in
/nfs/pic.es/user/n/neissner/.local/share/jupyter/kernels/whatever_kernel_name
</pre>

If you don't have the '''ipykernel''' module installed in your environment you may receive an error message like the one below when trying to run the previous command.
<pre>No module named ipykernel</pre>

If this is the case, you need to install it by running: '''pip install ipykernel'''

Deactivate your environment.

For conda:
<pre>
(...) [neissner@td110 ~]$ mamba deactivate
</pre>

For venv:
<pre>
(...) [neissner@td110 ~]$ deactivate
</pre>

Now you can exit the terminal. After refreshing the Jupyter page your whatever_kernel_name appears in the dashboard. In this example '''test''' has been used for whatever_kernel_name

[[File:screen05.png|700px]]

== Unlink an environment from Jupyter ==
Log onto Jupyter, start a session and from the session dashboard choose the bash terminal. To remove your environment/kernel from Jupyter run:
<pre>
[neissner@td110 ~]$ jupyter kernelspec uninstall whatever_kernel_name
Kernel specs to remove:
whatever_kernel_name /nfs/pic.es/user/n/neissner/.local/share/jupyter/kernels/whatever_kernel_name
Remove 1 kernel specs [y/N]: y
[RemoveKernelSpec] Removed /nfs/pic.es/user/n/neissner/.local/share/jupyter/kernels/whatever_kernel_name
</pre>
Keep in mind that, although not available in Jupyter anymore, the environment still exists. Whenever you need it, you can link it again.

== Create virtual environments with venv or conda ==

Before creating a new environment, please get in contact with your project liaison at PIC as there may be already a suitable environment for your needs in place.

If none of the existing environments suits your needs, you can create a new environment.
First, create a directory in a suitable place to store the environment. For single-user environments, place them in your home under ~/env. For environments that will be shared with other project users, contact your project liaison and ask him/her for a path in a shared storage volume that is visible to all of them.

Once you have the location (i.e. /path/to/env/folder), create the environment with the following commands:

For '''venv''' environments '''(recommended)'''

If your_env is installed at /path/to/env/your_env
<pre>
[neissner@td110 ~]$ cd /path/to/env
[neissner@td110 ~]$ python3 -m venv your_env
</pre>
Now you should be able to activate your environment and install additional modules
<pre>
[neissner@td110 ~]$ cd /path/to/env
[neissner@td110 ~]$ source your_env/bin/activate
(...)[neissner@td110 ~]$ pip install additional_module1 additional_module2 ...
</pre>

For '''conda/mamba''' environments
<pre>
[neissner@td110 ~]$ mamba create --prefix /path/to/env/your_env
</pre>
The list of modules (module1, module2, ...) is optional. For instance, for a python3 environment with scipy you would specify: ''python=3 scipy''

Now you should be able to activate your environment and install additional modules
<pre>
[neissner@td110 ~]$ mamba activate /path/to/env/folder/your_env
(...)[neissner@td110 ~]$ mamba install additional_module1 additional_module2 ...
</pre>
You can use pip install inside a mamba environment, however, resolving dependencies might require installing additional packages manually.

== Conda / mamba configuration ==

The behaviour of conda/mamba can be configured through the "$HOME/.condarc" file, described [https://docs.conda.io/projects/conda/en/latest/configuration.html here]. Some interesting parameters:

* envs_dirs: The list of directories to search for named environments. E.g.: different locations where you created environments

envs_dirs:
- /data/pic/scratch/torradeflot/envs
- /data/astro/scratch/torradeflot/envs
- /data/aai/scratch/torradeflot/envs

* pkgs_dirs: Folder where to store conda packages

pkgs_dirs:
- /data/aai/scratch_ssd/torradeflot/pkgs
- /data/aai/scratch/torradeflot/pkgs
- /data/pic/scratch/torradeflot/pkgs

if `pkgs_dirs` and `envs_dirs` are in the same storage, conda will use hard links, thus optimizing the disk space.

= Proper usage of X509 based proxies =

We found recently that the usage of proxies within a Jupyter session might cause problems because the environment changes certain standard locations such as '''/tmp'''

For a correct functioning please create the proxy the following way, example for Virgo:

<pre>
[<user>@<hostname> ~]$ /bin/voms-proxy-init --voms virgo:/virgo/ligo --out ./x509up_u$(id -u)
[<user>@<hostname> ~]$ export X509_USER_PROXY=./x509up_u$(id -u)
[<user>@<hostname> ~]$ ls /cvmfs/ligo.osgstorage.org
ls: cannot access /cvmfs/ligo.osgstorage.org: Permission denied
</pre>

Here the proxy cannot properly be located. Therefore we have to put the complete path into the variable:

<pre>
[<user>@<hostname> ~]$ pwd
/nfs/pic.es/user/<letter>/<user>
[<user>@<hostname> ~]$ export X509_USER_PROXY=/nfs/pic.es/user/<letter>/<user>/x509up_u$(id -u)
[<user>@<hostname> ~]$ ls /cvmfs/ligo.osgstorage.org
frames powerflux pycbc test_access
</pre>

= Software of particular interest =
== SageMath ==

[https://www.sagemath.org/ SageMath] is particularly interesting for Cosmology because it allows symbolic calculations, e.g. deriving the equations of motions for the scale factor starting from a customised space-time metric.

=== Standard cosmology examples ===

* The Friedman equations for the FLRW solution of the Einstein equations.
You can find the corresponding Notebook in any PIC terminal at '''/data/astro/software/notebooks/FLRW_cosmology.ipynb'''

* The notebook you can find at '''/data/astro/software/notebooks/FLRW_cosmology_solutions.ipynb''' uses known analytical solutions of the FLRW cosmology and produces this image for the evolution of the scale factor:

[[File:Screenshot_Sage05.png|300px]]

* The notebook you can find at '''/data/astro/software/notebooks/Interior_Schwarzschild.ipynb''' shows the formalism for the interior Schwarzschild metric and displays the solutions for density and pressure of a static celestial object that is sufficiently larger than its Schwarzschild radius. The pressure for an object with constant density id shown in the image:

[[File:Screenshot_Sage06.png|300px]]

=== Enabling SageMath environment in Jupyter ===

If you have never initialized mamba, run:

<pre>
[<user>@<hostname> ~]$ /data/astro/software/centos7/conda/mambaforge_4.14.0/bin/mamba init
[<user>@<hostname> ~]$ conda config --set auto_activate_base false
</pre>

After that you can enable SageMath for its use in a Jupyter notebook session:

<pre>
[<user>@<hostname> ~] mamba activate /data/astro/software/envs/sage
(/data/astro/software/envs/sage) [<user>@<hostname> ~]$ python -m ipykernel install --user --name=sage
....
(/data/astro/software/envs/sage) [<user>@<hostname> ~]$ mamba deactivate
</pre>

This creates a file in you home '''~/.local/share/jupyter/kernels/sage/kernel.json'''
which has to be modified to look like this:

<pre>
{
"argv": [
"/data/astro/software/envs/sage/bin/sage",
"--python",
"-m",
"sage.repl.ipython_kernel",
"-f",
"{connection_file}"
],
"display_name": "sage",
"language": "sage",
"metadata": {
"debugger": true
}
}
</pre>

Next time you go to your Jupyter dashboard you will find the sage environment listed there.

= Dask =

A notebook with instructions on how to run Dask at PIC can be found [https://gitlab.pic.es/services/code-samples/-/blob/main/computing/dask/dask_htcondor.ipynb here]

= Using a singularity image as a jupyter kernel =

Sometimes the software stack of some projects may be provided in the shape of a singularity image, it will then be convenient to use this image as a kernel for the notebooks in jupyter.pic.es.

The singularity image to be used as a kernel needs to fullfill some requirements. Different requirements will apply depending on the programming language.

== python jupyter kernel in a singularity image ==

The singularity image needs to have the '''python''' and the '''ipykernel''' module installed.

* Create the folder that will host the kernel definition

mkdir -p $HOME/.local/share/jupyter/kernels/singularity

* Create the '''kernel.json''' file inside it with the following content:

{
"argv": [
"singularity",
"exec",
"--cleanenv",
"/path/to/the/singularity/image.sif",
"python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"language": "python",
"display_name": "singularity-kernel"
}

Refresh or start the jupyterlab interface and the singularity kernel should appear in the launcher tab

= GPUs =

The way to identify the GPUs that are assigned to your job is:
* check the environment variable CUDA_VISIBLE_DEVICES. In a terminal run "echo $CUDA_VISIBLE_DEVICES". The environment variable contains a list of comma-separated GPU ids. With this you will already know how many gpus are assigned to your job. If the variable does not exist, there are no gpus assigned to the job

* list the gpus with nvidia-smi, in a terminal run "nvidia-smi -L", and look for the gpus you've been assigned. Remember their indexes (integers from 0 to 7)

* The two steps above can be done with the following command:

nvidia-smi -L | grep $CUDA_VISIBLE_DEVICES

if only having a single assigned GPU.

[[File:check_gpu_id_highlighted.png]]

* in the GPU dashboard the gpus are identified with their index

[[File:check_gpu_resources_highlighted.png]]

= Jupyterlab user guide =

You can find the official documentation of the currently installed version of jupyterlab (3.6) [https://jupyterlab.readthedocs.io/en/4.2.x/ here], there you will find instruction on how to
* [https://jupyterlab.readthedocs.io/en/4.2.x/user/commands.html Access the command palette]
* [https://jupyterlab.readthedocs.io/en/4.2.x/user/toc.html Build a Table Of Contents]
* [https://jupyterlab.readthedocs.io/en/4.2.x/user/debugger.html Debug your code]

A set of non-official jupyterlab extensions are installed to provide additional functionalities

== jupytext ==
Pair your notebooks with text files to enhance version tracking.
https://jupytext.readthedocs.io

=== Example ===

If you had a notebook (.ipynb file) containing only the cell below, tracked in a git repository

<pre>
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
plt.imshow(np.random.random([10, 10]))
</pre>

Different executions of the cell would produce different images, and the images are embedded in a pseudo-binary format inside the notebook file. In this case, doing a '''git diff''' of the .ipynb file would produce a huge output (because the image changed), even if there wasn't any change in the code. It is thus convenient to sync the notebook with a text file (e.g. a .py script) using the jupytext extension and track this one with git. The outputs, including images, as well as some additional metadata, won't be added to the synced text file. So in the case of different executions of the same notebook, the diff will always be empty.

== git ==
Sidebar GUI to git repo management
https://github.com/jupyterlab/jupyterlab-git

== variable inspector ==
Variable inspection à la Matlab
https://github.com/jupyterlab-contrib/jupyterlab-variableInspector

...

= Code samples =

A repository with sample code can be found here: https://gitlab.pic.es/services/code-samples/

= Troubleshooting =

== Error 500: Internal Server Error ==

This is a generic error. Means that the jupyterlab server failed. This could be for different reasons:

* Your HOME folder is full. Log in to "ui.pic.es" and run "quota" to check the usage vs quota. If it is full you'll have to free up space.

== Logs ==

The log files for the jupyterlab server are stored in "~/.jupyter". The log files will be created once the jupyter lab server job is finished.

== Clean workspaces ==

Jupyterlab stores the workspace status in the "~/.jupyter/lab/workspaces" folder. If you want to start with a fresh (empty) workspace, delete all the content of this folder before launching the notebook.

cd ~/.jupyter/lab/workspaces
rm *

== 504 Gateway timeout ==

The notebook job is running in HTCondor but the user can not access the notebook server. Ultimately a 504 error is received.

This is probably because there's some error when starting the jupyterlab server. First of all, shutdown the notebook server and [[#Logs|check the logs]] to better identify the problem. If you don't see the source of the error, try to [[#Clean_workspaces|clean the workspaces]] and launch a notebook again.

== Install ROOT ==

Environment creation and kernel installation

$ micromamba env create -p /data/pic/scratch/torradeflot/envs/mcdata root ipykernel
$ micromamba activate mcdata
$ python -m ipykernel install --user --name mcdata

After doing this ROOT can be imported from a python shell, but it does not work from a notebook.

* ROOT uses JIT compilation with Cling: https://root.cern/cling/
* conda provides it's own set of compilation tools: gcc, gxx, fortran
* Some environment variables are necessary to ensure that these two pieces work well together:
** PATH: to be able to find the compiler tools
** CONDA_BUILD_SYSROOT: needed to configure the compiler call

These environment variables are not propagated to the notebook, because they are set by .bashrc conda activate, ... so we need to explicitly set them in the notebook.

import os
import sys
from pathlib import Path
bin_dir = Path(sys.executable).parent
os.environ['PATH'] = f'{bin_dir}:{os.environ['PATH']}'
os.environ['CONDA_BUILD_SYSROOT'] = str(bin_dir.parent / 'x86_64-conda-linux-gnu/sysroot')

== Loading libraries is very slow ==
Conda environments at storage using hard drives can be extremely slow to load. If you encounter this problem, please ask
your support contact for a "SSD scratch" location to store environments. Currently this service is being tested and
deployed as needed.

== Spawn failed: The 'ip' trait of a PICCondorSpawner instance expected a unicode string, not the NoneType None ==

Jupyterhub could not get the host name from HTCondor's stdout, because it didn't match the expected regular expression.

This error happens randomly from time to time, it does not imply any major problem in any of the services.

Try to request a new notebook server.

== 403 : Forbidden. XSRF cookie does not match POST argument ==

The value of the "_xsrf" cookie sent by the browser does not match the expected value. This could be for many reasons: temporary high load on the server, race conditions, temporary network unstability, many open tabs in the browser, etc.

In general it can be solved by closing all tabs pointing to "jupyter.pic.es", cleaning the cookies and connecting back to "jupyter.pic.es"

Spark on Hadoop

2025-05-08T10:24:03Z

Torradeflot: /* How to configure Kerberos authentication with Firefox */

= Monitoring you Spark jobs =

Spark jobs submitted to Yarn can be monitored through the web interface at http://hsrv02.pic.es:8088

You need to have a VPN and Kerberos authentication configured in your browser if you want to access it.

== How to configure Kerberos authentication with Firefox ==

Tested in a Ubuntu 22 server with Firefox installed with apt. Configuring Kerberos authentication with other browser & OS combinations may be differ or not be supported. In particular, in Ubuntu 22, Firefox installed with snap won't work.

You need to have Kerberos client installed in your PC.

Update the "/etc/krb5.conf" file to contain PIC realm. Do not remove additional content in the "krb5.conf" file:

[realms]
PIC.ES = {
kdc = ipa01.pic.es:88
master_kdc = ipa01.pic.es:88
admin_server = ipa01.pic.es:749
kpasswd_server = ipa01.pic.es:464
default_domain = pic.es
}

[domain_realm]
.pic.es = PIC.ES
pic.es = PIC.ES
hsrv03.pic.es = PIC.ES

Update the Firefox configuration
* Open Firefox
* Access the configuration in "about:config" in the navigation bar
* look for the options with the "negotiate" text and set these values:
** network.negotiate-auth.allow-non-fqdn: true
** network.negotiate-auth.delegation-uris: pic.es
** network.negotiate-auth.trusted-uris: pic.es

Spark on Hadoop

2025-05-08T10:17:56Z

Torradeflot:

2024-02-21T13:38:49Z

Torradeflot: Created page with "= Dask + HTCondor = == Creating a cluster == === using dask-labextension === To create a Dask cluster from the extension. Go to the dask-labextension tab and click "+ NEW"..."

= Dask + HTCondor =

== Creating a cluster ==

=== using dask-labextension ===

To create a Dask cluster from the extension. Go to the dask-labextension tab and click "+ NEW"

![kk](static/dask_labextension_1.png)

You can see the cluster information and have some shortcuts
* "<>" insert a cell with the necessary lines to create a Dask client. Some environment variables have been set in order to be able to use a "regular" Dask Client
* "SCALE" increase/decrease the number of workers
* "SHUTDOWN" shutdown the cluster

![kk](static/dask_add_cluster_cell.png)

from dask.distributed import Client
client = Client("tls://192.168.100.56:45722")
client

/data/astro/scratch2/torradeflot/envs/dask/lib/python3.11/site-packages/distributed/client.py:1388:
VersionMismatchWarning: Mismatched versions found

+---------+----------------+----------------+----------------+
| Package | Client | Scheduler | Workers |
+---------+----------------+----------------+----------------+
| lz4 | 4.3.3 | 4.3.2 | 4.3.2 |
| msgpack | 1.0.7 | 1.0.5 | 1.0.5 |
| numpy | 1.26.3 | 1.24.3 | 1.24.3 |
| pandas | 2.2.0 | 2.0.2 | 2.0.2 |
| python | 3.11.7.final.0 | 3.11.5.final.0 | 3.11.5.final.0 |
| toolz | 0.12.1 | 0.12.0 | 0.12.0 |
+---------+----------------+----------------+----------------+
warnings.warn(version_module.VersionMismatchWarning(msg[0]["warning"]))

```python
client.cluster
```

The encryption of the traffic between the different Dask components is enforced through environment variables

```python
[f'{k}={v}' for k,v in os.environ.items() if 'DASK' in k]
```

['DASK_DISTRIBUTED__COMM__TLS__CLIENT__KEY=/nfs/pic.es/user/t/torradeflot/.config/dask/security/key.pem',
'DASK_DISTRIBUTED__COMM__TLS__SCHEDULER__CERT=/nfs/pic.es/user/t/torradeflot/.config/dask/security/cert.pem',
'DASK_DISTRIBUTED__COMM__TLS__WORKER__CERT=/nfs/pic.es/user/t/torradeflot/.config/dask/security/cert.pem',
'DASK_DISTRIBUTED__COMM__TLS__CA_FILE=/nfs/pic.es/user/t/torradeflot/.config/dask/security/ca_file.pem',
'DASK_DISTRIBUTED__COMM__TLS__WORKER__KEY=/nfs/pic.es/user/t/torradeflot/.config/dask/security/key.pem',
'DASK_DISTRIBUTED__COMM__TLS__SCHEDULER__KEY=/nfs/pic.es/user/t/torradeflot/.config/dask/security/key.pem',
'DASK_DISTRIBUTED__COMM__REQUIRE_ENCRYPTION=true',
'DASK_DISTRIBUTED__COMM__TLS__CLIENT__CERT=/nfs/pic.es/user/t/torradeflot/.config/dask/security/cert.pem']

![kk](static/dask_labextension_2.png)

## From python

### Using the default environment

This could be a notebook or a script submitted to HTCondor

```python
from pic_jupyterhub.dask_condor import SecureHTCondor
from dask.distributed import Client
```

```python
cluster = SecureHTCondor() #scheduler_options={'dashboard_address':":8788"})
```

Certificate files already exist

The cluster is initially created without workers. We need to scale it.

```python
cluster.scale(2)
```

```python
client = Client(cluster)
```

```python
cluster.close()
```

### Using a custom environment

The environment needs to have:

* `dask-jobqueue` to be able to start Dask clusters with HTCondor
* `ipykernel` to use it as a notebook kernel
* `numpy` and `pandas` to be able to create Dask arrays and dataframes
* `bokeh` for the Dask dashboard

`mamba create (-n {env name} | -p {env Path}) dask-jobqueue ipykernel numpy pandas bokeh`

```python
from dask_jobqueue import HTCondorCluster
from dask.distributed import Client
```

```python
cluster = HTCondorCluster(cores=1, memory='2GB', disk='10 GB',
job_extra_directives={'getenv': 'True'}) # needed to propagate the security
```

```python
cluster
```

<div class="jp-RenderedHTMLCommon jp-RenderedHTML jp-mod-trusted jp-OutputArea-output">
<div style="width: 24px; height: 24px; background-color: #e1e1e1; border: 3px solid #9D9D9D; border-radius: 5px; position: absolute;">
</div>
<div style="margin-left: 48px;">
<h3 style="margin-bottom: 0px; margin-top: 0px;">HTCondorCluster</h3>
<p style="color: #9D9D9D; margin-bottom: 0px;">a980977e</p>
<table style="width: 100%; text-align: left;">
<tr>
<td style="text-align: left;">
<strong>Dashboard:</strong> <a href="http://192.168.102.42:8787/status" target="_blank">http://192.168.102.42:8787/status</a>
</td>
<td style="text-align: left;">
<strong>Workers:</strong> 1
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Total threads:</strong> 1
</td>
<td style="text-align: left;">
<strong>Total memory:</strong> 1.86 GiB
</td>
</tr>

</table>

<details>
<summary style="margin-bottom: 20px;">
<h3 style="display: inline;">Scheduler Info</h3>
</summary>

<div style="">
<div>
<div style="width: 24px; height: 24px; background-color: #FFF7E5; border: 3px solid #FF6132; border-radius: 5px; position: absolute;"> </div>
<div style="margin-left: 48px;">
<h3 style="margin-bottom: 0px;">Scheduler</h3>
<p style="color: #9D9D9D; margin-bottom: 0px;">Scheduler-78cf6c5f-5546-44b4-b4c7-54b6caea8f9b</p>
<table style="width: 100%; text-align: left;">
<tr>
<td style="text-align: left;">
<strong>Comm:</strong> tls://192.168.102.42:40855
</td>
<td style="text-align: left;">
<strong>Workers:</strong> 1
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Dashboard:</strong> <a href="http://192.168.102.42:8787/status" target="_blank">http://192.168.102.42:8787/status</a>
</td>
<td style="text-align: left;">
<strong>Total threads:</strong> 1
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Started:</strong> Just now
</td>
<td style="text-align: left;">
<strong>Total memory:</strong> 1.86 GiB
</td>
</tr>
</table>
</div>
</div>

<details style="margin-left: 48px;">
<summary style="margin-bottom: 20px;">
<h3 style="display: inline;">Workers</h3>
</summary>

<div style="margin-bottom: 20px;">
<div style="width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;"> </div>
<div style="margin-left: 48px;">
<details>
<summary>
<h4 style="margin-bottom: 0px; display: inline;">Worker: HTCondorCluster-0</h4>
</summary>
<table style="width: 100%; text-align: left;">
<tr>
<td style="text-align: left;">
<strong>Comm: </strong> tls://192.168.100.27:40884
</td>
<td style="text-align: left;">
<strong>Total threads: </strong> 1
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Dashboard: </strong> <a href="http://192.168.100.27:45160/status" target="_blank">http://192.168.100.27:45160/status</a>
</td>
<td style="text-align: left;">
<strong>Memory: </strong> 1.86 GiB
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Nanny: </strong> tls://192.168.100.27:39432
</td>
<td style="text-align: left;"></td>
</tr>
<tr>
<td colspan="2" style="text-align: left;">
<strong>Local directory: </strong> /tmp/dask-scratch-space/worker-zscas2e0
</td>
</tr>

</table>
</details>
</div>
</div>

</details>
</div>

</details>
</div>
</div>

```python
cluster.scale(2)
```

```python
c = Client(cluster)
```

Security is inherited from environment variables

```python
cluster.security
```

<div style="margin-left: auto;">
<h3 style="margin-bottom: 0px;"><b>Security</b></h3>
<p>
<table style="width: 100%;">

<tr>
<th style="text-align: left; width: 150px;">require_encryption</th>
<td style="text-align: left;">Local (/nfs/pic.es/user/t/torradeflot/jupyterhub_conf/examples/true)</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">tls_ca_file</th>
<td style="text-align: left;">Local (/nfs/pic.es/user/t/torradeflot/.config/dask/security/ca_file.pem)</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">tls_client_cert</th>
<td style="text-align: left;">Local (/nfs/pic.es/user/t/torradeflot/.config/dask/security/cert.pem)</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">tls_client_key</th>
<td style="text-align: left;">Local (/nfs/pic.es/user/t/torradeflot/.config/dask/security/key.pem)</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">tls_min_version</th>
<td style="text-align: left;">771</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">tls_scheduler_cert</th>
<td style="text-align: left;">Local (/nfs/pic.es/user/t/torradeflot/.config/dask/security/cert.pem)</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">tls_scheduler_key</th>
<td style="text-align: left;">Local (/nfs/pic.es/user/t/torradeflot/.config/dask/security/key.pem)</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">tls_worker_cert</th>
<td style="text-align: left;">Local (/nfs/pic.es/user/t/torradeflot/.config/dask/security/cert.pem)</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">tls_worker_key</th>
<td style="text-align: left;">Local (/nfs/pic.es/user/t/torradeflot/.config/dask/security/key.pem)</td>
</tr>

</table>
</p>
</div>

## Connect to existing cluster

Meaning a cluster that was launched from outside you jupyterlab instance, e.g. an independent HTCondor job or somebody else's cluster.

Check IP address of running job
```
nslookup $(condor_q $JOB_ID -af RemoteHost | cut -d "@" -f 2)
```

```python
from dask.distributed import Client

client = Client("tls://192.168.100.5:42166")
client
```

/data/astro/scratch2/torradeflot/envs/dask/lib/python3.11/site-packages/distributed/client.py:1388: VersionMismatchWarning: Mismatched versions found

+---------+----------------+----------------+----------------+
| Package | Client | Scheduler | Workers |
+---------+----------------+----------------+----------------+
| lz4 | 4.3.3 | 4.3.2 | 4.3.2 |
| msgpack | 1.0.7 | 1.0.5 | 1.0.5 |
| numpy | 1.26.3 | 1.24.3 | 1.24.3 |
| pandas | 2.2.0 | 2.0.2 | 2.0.2 |
| python | 3.11.7.final.0 | 3.11.5.final.0 | 3.11.5.final.0 |
| toolz | 0.12.1 | 0.12.0 | 0.12.0 |
+---------+----------------+----------------+----------------+
warnings.warn(version_module.VersionMismatchWarning(msg[0]["warning"]))

<div>
<div style="width: 24px; height: 24px; background-color: #e1e1e1; border: 3px solid #9D9D9D; border-radius: 5px; position: absolute;"> </div>
<div style="margin-left: 48px;">
<h3 style="margin-bottom: 0px;">Client</h3>
<p style="color: #9D9D9D; margin-bottom: 0px;">Client-589eb004-c406-11ee-8584-001e67f120a8</p>
<table style="width: 100%; text-align: left;">

<tr>

<td style="text-align: left;"><strong>Connection method:</strong> Direct</td>
<td style="text-align: left;"></td>

</tr>

<tr>
<td style="text-align: left;">
<strong>Dashboard: </strong> <a href="http://192.168.100.5:41069/status" target="_blank">http://192.168.100.5:41069/status</a>
</td>
<td style="text-align: left;"></td>
</tr>

</table>

<details>
<summary style="margin-bottom: 20px;"><h3 style="display: inline;">Scheduler Info</h3></summary>
<div style="">
<div>
<div style="width: 24px; height: 24px; background-color: #FFF7E5; border: 3px solid #FF6132; border-radius: 5px; position: absolute;"> </div>
<div style="margin-left: 48px;">
<h3 style="margin-bottom: 0px;">Scheduler</h3>
<p style="color: #9D9D9D; margin-bottom: 0px;">Scheduler-24c09727-b6fc-4703-b25c-6d3bfbf750ee</p>
<table style="width: 100%; text-align: left;">
<tr>
<td style="text-align: left;">
<strong>Comm:</strong> tls://192.168.100.5:42166
</td>
<td style="text-align: left;">
<strong>Workers:</strong> 5
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Dashboard:</strong> <a href="http://192.168.100.5:41069/status" target="_blank">http://192.168.100.5:41069/status</a>
</td>
<td style="text-align: left;">
<strong>Total threads:</strong> 5
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Started:</strong> 42 minutes ago
</td>
<td style="text-align: left;">
<strong>Total memory:</strong> 9.30 GiB
</td>
</tr>
</table>
</div>
</div>

<details style="margin-left: 48px;">
<summary style="margin-bottom: 20px;">
<h3 style="display: inline;">Workers</h3>
</summary>

<div style="margin-bottom: 20px;">
<div style="width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;"> </div>
<div style="margin-left: 48px;">
<details>
<summary>
<h4 style="margin-bottom: 0px; display: inline;">Worker: SecureHTCondor-0</h4>
</summary>
<table style="width: 100%; text-align: left;">
<tr>
<td style="text-align: left;">
<strong>Comm: </strong> tls://192.168.101.127:38421
</td>
<td style="text-align: left;">
<strong>Total threads: </strong> 1
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Dashboard: </strong> <a href="http://192.168.101.127:34923/status" target="_blank">http://192.168.101.127:34923/status</a>
</td>
<td style="text-align: left;">
<strong>Memory: </strong> 1.86 GiB
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Nanny: </strong> tls://192.168.101.127:38180
</td>
<td style="text-align: left;"></td>
</tr>
<tr>
<td colspan="2" style="text-align: left;">
<strong>Local directory: </strong> /tmp/dask-scratch-space/worker-vzw0w_hh
</td>
</tr>

<tr>
<td style="text-align: left;">
<strong>Tasks executing: </strong>
</td>
<td style="text-align: left;">
<strong>Tasks in memory: </strong>
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Tasks ready: </strong>
</td>
<td style="text-align: left;">
<strong>Tasks in flight: </strong>
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>CPU usage:</strong> 4.0%
</td>
<td style="text-align: left;">
<strong>Last seen: </strong> Just now
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Memory usage: </strong> 125.28 MiB
</td>
<td style="text-align: left;">
<strong>Spilled bytes: </strong> 0 B
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Read bytes: </strong> 604.32 kiB
</td>
<td style="text-align: left;">
<strong>Write bytes: </strong> 0.95 MiB
</td>
</tr>

</table>
</details>
</div>
</div>

<div style="margin-bottom: 20px;">
<div style="width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;"> </div>
<div style="margin-left: 48px;">
<details>
<summary>
<h4 style="margin-bottom: 0px; display: inline;">Worker: SecureHTCondor-1</h4>
</summary>
<table style="width: 100%; text-align: left;">
<tr>
<td style="text-align: left;">
<strong>Comm: </strong> tls://192.168.102.27:46440
</td>
<td style="text-align: left;">
<strong>Total threads: </strong> 1
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Dashboard: </strong> <a href="http://192.168.102.27:35027/status" target="_blank">http://192.168.102.27:35027/status</a>
</td>
<td style="text-align: left;">
<strong>Memory: </strong> 1.86 GiB
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Nanny: </strong> tls://192.168.102.27:44026
</td>
<td style="text-align: left;"></td>
</tr>
<tr>
<td colspan="2" style="text-align: left;">
<strong>Local directory: </strong> /tmp/dask-scratch-space/worker-nj1hcr7h
</td>
</tr>

<tr>
<td style="text-align: left;">
<strong>Tasks executing: </strong>
</td>
<td style="text-align: left;">
<strong>Tasks in memory: </strong>
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Tasks ready: </strong>
</td>
<td style="text-align: left;">
<strong>Tasks in flight: </strong>
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>CPU usage:</strong> 2.0%
</td>
<td style="text-align: left;">
<strong>Last seen: </strong> Just now
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Memory usage: </strong> 122.96 MiB
</td>
<td style="text-align: left;">
<strong>Spilled bytes: </strong> 0 B
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Read bytes: </strong> 17.31 MiB
</td>
<td style="text-align: left;">
<strong>Write bytes: </strong> 182.89 kiB
</td>
</tr>

</table>
</details>
</div>
</div>

<div style="margin-bottom: 20px;">
<div style="width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;"> </div>
<div style="margin-left: 48px;">
<details>
<summary>
<h4 style="margin-bottom: 0px; display: inline;">Worker: SecureHTCondor-2</h4>
</summary>
<table style="width: 100%; text-align: left;">
<tr>
<td style="text-align: left;">
<strong>Comm: </strong> tls://192.168.102.37:36278
</td>
<td style="text-align: left;">
<strong>Total threads: </strong> 1
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Dashboard: </strong> <a href="http://192.168.102.37:41285/status" target="_blank">http://192.168.102.37:41285/status</a>
</td>
<td style="text-align: left;">
<strong>Memory: </strong> 1.86 GiB
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Nanny: </strong> tls://192.168.102.37:36316
</td>
<td style="text-align: left;"></td>
</tr>
<tr>
<td colspan="2" style="text-align: left;">
<strong>Local directory: </strong> /tmp/dask-scratch-space/worker-l2021n4v
</td>
</tr>

<tr>
<td style="text-align: left;">
<strong>Tasks executing: </strong>
</td>
<td style="text-align: left;">
<strong>Tasks in memory: </strong>
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Tasks ready: </strong>
</td>
<td style="text-align: left;">
<strong>Tasks in flight: </strong>
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>CPU usage:</strong> 2.0%
</td>
<td style="text-align: left;">
<strong>Last seen: </strong> Just now
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Memory usage: </strong> 121.28 MiB
</td>
<td style="text-align: left;">
<strong>Spilled bytes: </strong> 0 B
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Read bytes: </strong> 79.03 kiB
</td>
<td style="text-align: left;">
<strong>Write bytes: </strong> 102.23 kiB
</td>
</tr>

</table>
</details>
</div>
</div>

<div style="margin-bottom: 20px;">
<div style="width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;"> </div>
<div style="margin-left: 48px;">
<details>
<summary>
<h4 style="margin-bottom: 0px; display: inline;">Worker: SecureHTCondor-3</h4>
</summary>
<table style="width: 100%; text-align: left;">
<tr>
<td style="text-align: left;">
<strong>Comm: </strong> tls://192.168.100.12:45248
</td>
<td style="text-align: left;">
<strong>Total threads: </strong> 1
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Dashboard: </strong> <a href="http://192.168.100.12:43200/status" target="_blank">http://192.168.100.12:43200/status</a>
</td>
<td style="text-align: left;">
<strong>Memory: </strong> 1.86 GiB
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Nanny: </strong> tls://192.168.100.12:36383
</td>
<td style="text-align: left;"></td>
</tr>
<tr>
<td colspan="2" style="text-align: left;">
<strong>Local directory: </strong> /tmp/dask-scratch-space/worker-0v5jilfv
</td>
</tr>

<tr>
<td style="text-align: left;">
<strong>Tasks executing: </strong>
</td>
<td style="text-align: left;">
<strong>Tasks in memory: </strong>
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Tasks ready: </strong>
</td>
<td style="text-align: left;">
<strong>Tasks in flight: </strong>
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>CPU usage:</strong> 2.0%
</td>
<td style="text-align: left;">
<strong>Last seen: </strong> Just now
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Memory usage: </strong> 125.12 MiB
</td>
<td style="text-align: left;">
<strong>Spilled bytes: </strong> 0 B
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Read bytes: </strong> 3.26 MiB
</td>
<td style="text-align: left;">
<strong>Write bytes: </strong> 3.21 MiB
</td>
</tr>

</table>
</details>
</div>
</div>

<div style="margin-bottom: 20px;">
<div style="width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;"> </div>
<div style="margin-left: 48px;">
<details>
<summary>
<h4 style="margin-bottom: 0px; display: inline;">Worker: SecureHTCondor-4</h4>
</summary>
<table style="width: 100%; text-align: left;">
<tr>
<td style="text-align: left;">
<strong>Comm: </strong> tls://192.168.100.14:42847
</td>
<td style="text-align: left;">
<strong>Total threads: </strong> 1
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Dashboard: </strong> <a href="http://192.168.100.14:36788/status" target="_blank">http://192.168.100.14:36788/status</a>
</td>
<td style="text-align: left;">
<strong>Memory: </strong> 1.86 GiB
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Nanny: </strong> tls://192.168.100.14:46489
</td>
<td style="text-align: left;"></td>
</tr>
<tr>
<td colspan="2" style="text-align: left;">
<strong>Local directory: </strong> /tmp/dask-scratch-space/worker-f3u75i_8
</td>
</tr>

<tr>
<td style="text-align: left;">
<strong>Tasks executing: </strong>
</td>
<td style="text-align: left;">
<strong>Tasks in memory: </strong>
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Tasks ready: </strong>
</td>
<td style="text-align: left;">
<strong>Tasks in flight: </strong>
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>CPU usage:</strong> 2.0%
</td>
<td style="text-align: left;">
<strong>Last seen: </strong> Just now
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Memory usage: </strong> 121.30 MiB
</td>
<td style="text-align: left;">
<strong>Spilled bytes: </strong> 0 B
</td>
</tr>
<tr>
<td style="text-align: left;">
<strong>Read bytes: </strong> 20.58 MiB
</td>
<td style="text-align: left;">
<strong>Write bytes: </strong> 4.19 MiB
</td>
</tr>

</table>
</details>
</div>
</div>

</details>
</div>
</details>

</div>
</div>

2024-02-05 11:21:41,076 - distributed.client - ERROR - Failed to reconnect to scheduler after 30.00 seconds, closing client

# Troubleshooting

## Compatibility issues

If you try to connect a notebook to a Dask cluster, and the notebook's environment is different from the one used to launch de cluster, you may encounter compatibility issues.

You will tipycally receive a "Mismatched versions found" warning like this:

```
/data/astro/scratch2/torradeflot/envs/dask/lib/python3.11/site-packages/distributed/client.py:1388: VersionMismatchWarning: Mismatched versions found

+---------+----------------+----------------+----------------+
| Package | Client | Scheduler | Workers |
+---------+----------------+----------------+----------------+
| lz4 | 4.3.3 | 4.3.2 | 4.3.2 |
| msgpack | 1.0.7 | 1.0.5 | 1.0.5 |
| numpy | 1.26.3 | 1.24.3 | 1.24.3 |
| pandas | 2.2.0 | 2.0.2 | 2.0.2 |
| python | 3.11.7.final.0 | 3.11.5.final.0 | 3.11.5.final.0 |
| toolz | 0.12.1 | 0.12.0 | 0.12.0 |
+---------+----------------+----------------+----------------+
warnings.warn(version_module.VersionMismatchWarning(msg[0]["warning"]))
```

Some mismatches might be blocking, it is recommended to match the major and minor versions. A mismatch in the patch version shouldn't be a problem.

## Problems with securitization

If you launch a Dask cluster from a notebook but you have never launched a cluster from the dask-labextension or using the `pic_jupyterhub` module, you may encounter a problem because encryption is enforced but the certificates do not exist.

If this is the case, you can launch a cluster using one of these options as shown above. This will generate the certificate files and the subsequent creation of a Dask cluster from a notebook should succeed.

# Examples
## Dask example 1

picked from https://docs.dask.org/en/stable/10-minutes-to-dask.html

```python
import numpy as np
import pandas as pd

import dask.dataframe as dd
import dask.array as da
import dask.bag as db
```

### DataFrame

```python
index = pd.date_range("2021-09-01", periods=2400, freq="1H")
df = pd.DataFrame({"a": np.arange(2400), "b": list("abcaddbe" * 300)}, index=index)
ddf = dd.from_pandas(df, npartitions=10)
ddf
```

/tmp/ipykernel_291/1125038151.py:1: FutureWarning: 'H' is deprecated and will be removed in a future version, please use 'h' instead.
index = pd.date_range("2021-09-01", periods=2400, freq="1H")

<div><strong>Dask DataFrame Structure:</strong></div>
<div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}

.dataframe tbody tr th {
vertical-align: top;
}

.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>a</th>
<th>b</th>
</tr>
<tr>
<th>npartitions=10</th>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<th>2021-09-01 00:00:00</th>
<td>int64</td>
<td>object</td>
</tr>
<tr>
<th>2021-09-11 00:00:00</th>
<td>...</td>
<td>...</td>
</tr>
<tr>
<th>...</th>
<td>...</td>
<td>...</td>
</tr>
<tr>
<th>2021-11-30 00:00:00</th>
<td>...</td>
<td>...</td>
</tr>
<tr>
<th>2021-12-09 23:00:00</th>
<td>...</td>
<td>...</td>
</tr>
</tbody>
</table>
</div>
<div>Dask Name: from_pandas, 1 graph layer</div>

```python
ddf.divisions
```

(Timestamp('2021-09-01 00:00:00'),
Timestamp('2021-09-11 00:00:00'),
Timestamp('2021-09-21 00:00:00'),
Timestamp('2021-10-01 00:00:00'),
Timestamp('2021-10-11 00:00:00'),
Timestamp('2021-10-21 00:00:00'),
Timestamp('2021-10-31 00:00:00'),
Timestamp('2021-11-10 00:00:00'),
Timestamp('2021-11-20 00:00:00'),
Timestamp('2021-11-30 00:00:00'),
Timestamp('2021-12-09 23:00:00'))

```python
ddf.partitions[1]
```

<div><strong>Dask DataFrame Structure:</strong></div>
<div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}

.dataframe tbody tr th {
vertical-align: top;
}

.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>a</th>
<th>b</th>
</tr>
<tr>
<th>npartitions=1</th>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<th>2021-09-11</th>
<td>int64</td>
<td>object</td>
</tr>
<tr>
<th>2021-09-21</th>
<td>...</td>
<td>...</td>
</tr>
</tbody>
</table>
</div>
<div>Dask Name: blocks, 2 graph layers</div>

```python
ddf["2000-10-01": "2021-10-09 5:00"].compute()
```

<div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}

.dataframe tbody tr th {
vertical-align: top;
}

.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>a</th>
<th>b</th>
</tr>
</thead>
<tbody>
<tr>
<th>2021-09-01 00:00:00</th>
<td>0</td>
<td>a</td>
</tr>
<tr>
<th>2021-09-01 01:00:00</th>
<td>1</td>
<td>b</td>
</tr>
<tr>
<th>2021-09-01 02:00:00</th>
<td>2</td>
<td>c</td>
</tr>
<tr>
<th>2021-09-01 03:00:00</th>
<td>3</td>
<td>a</td>
</tr>
<tr>
<th>2021-09-01 04:00:00</th>
<td>4</td>
<td>d</td>
</tr>
<tr>
<th>...</th>
<td>...</td>
<td>...</td>
</tr>
<tr>
<th>2021-10-09 01:00:00</th>
<td>913</td>
<td>b</td>
</tr>
<tr>
<th>2021-10-09 02:00:00</th>
<td>914</td>
<td>c</td>
</tr>
<tr>
<th>2021-10-09 03:00:00</th>
<td>915</td>
<td>a</td>
</tr>
<tr>
<th>2021-10-09 04:00:00</th>
<td>916</td>
<td>d</td>
</tr>
<tr>
<th>2021-10-09 05:00:00</th>
<td>917</td>
<td>d</td>
</tr>
</tbody>
</table>
<p>918 rows × 2 columns</p>
</div>

### Array

```python
import numpy as np
import dask.array as da

data = np.arange(100_000).reshape(200, 500)
a = da.from_array(data, chunks=(100, 100))
a
```

<table>
<tr>
<td>
<table style="border-collapse: collapse;">
<thead>
<tr>
<td> </td>
<th> Array </th>
<th> Chunk </th>
</tr>
</thead>
<tbody>

<tr>
<th> Bytes </th>
<td> 781.25 kiB </td>
<td> 78.12 kiB </td>
</tr>

<tr>
<th> Shape </th>
<td> (200, 500) </td>
<td> (100, 100) </td>
</tr>
<tr>
<th> Dask graph </th>
<td colspan="2"> 10 chunks in 1 graph layer </td>
</tr>
<tr>
<th> Data type </th>
<td colspan="2"> int64 numpy.ndarray </td>
</tr>
</tbody>
</table>
</td>
<td>
<svg width="170" height="98" style="stroke:rgb(0,0,0);stroke-width:1" >


<line x1="0" y1="0" x2="120" y2="0" style="stroke-width:2" />
<line x1="0" y1="24" x2="120" y2="24" />
<line x1="0" y1="48" x2="120" y2="48" style="stroke-width:2" />


<line x1="0" y1="0" x2="0" y2="48" style="stroke-width:2" />
<line x1="24" y1="0" x2="24" y2="48" />
<line x1="48" y1="0" x2="48" y2="48" />
<line x1="72" y1="0" x2="72" y2="48" />
<line x1="96" y1="0" x2="96" y2="48" />
<line x1="120" y1="0" x2="120" y2="48" style="stroke-width:2" />


<polygon points="0.0,0.0 120.0,0.0 120.0,48.0 0.0,48.0" style="fill:#ECB172A0;stroke-width:0"/>


<text x="60.000000" y="68.000000" font-size="1.0rem" font-weight="100" text-anchor="middle" >500</text>
<text x="140.000000" y="24.000000" font-size="1.0rem" font-weight="100" text-anchor="middle" transform="rotate(-90,140.000000,24.000000)">200</text>
</svg>
</td>
</tr>
</table>

```python
a.chunks
```

((100, 100), (100, 100, 100, 100, 100))

```python
a.blocks[1, 3]
```

<table>
<tr>
<td>
<table style="border-collapse: collapse;">
<thead>
<tr>
<td> </td>
<th> Array </th>
<th> Chunk </th>
</tr>
</thead>
<tbody>

<tr>
<th> Bytes </th>
<td> 78.12 kiB </td>
<td> 78.12 kiB </td>
</tr>

<tr>
<th> Shape </th>
<td> (100, 100) </td>
<td> (100, 100) </td>
</tr>
<tr>
<th> Dask graph </th>
<td colspan="2"> 1 chunks in 2 graph layers </td>
</tr>
<tr>
<th> Data type </th>
<td colspan="2"> int64 numpy.ndarray </td>
</tr>
</tbody>
</table>
</td>
<td>
<svg width="170" height="170" style="stroke:rgb(0,0,0);stroke-width:1" >


<line x1="0" y1="0" x2="120" y2="0" style="stroke-width:2" />
<line x1="0" y1="120" x2="120" y2="120" style="stroke-width:2" />


<line x1="0" y1="0" x2="0" y2="120" style="stroke-width:2" />
<line x1="120" y1="0" x2="120" y2="120" style="stroke-width:2" />


<polygon points="0.0,0.0 120.0,0.0 120.0,120.0 0.0,120.0" style="fill:#ECB172A0;stroke-width:0"/>


<text x="60.000000" y="140.000000" font-size="1.0rem" font-weight="100" text-anchor="middle" >100</text>
<text x="140.000000" y="60.000000" font-size="1.0rem" font-weight="100" text-anchor="middle" transform="rotate(-90,140.000000,60.000000)">100</text>
</svg>
</td>
</tr>
</table>

```python
a[:50, 200]
```

<table>
<tr>
<td>
<table style="border-collapse: collapse;">
<thead>
<tr>
<td> </td>
<th> Array </th>
<th> Chunk </th>
</tr>
</thead>
<tbody>

<tr>
<th> Bytes </th>
<td> 400 B </td>
<td> 400 B </td>
</tr>

<tr>
<th> Shape </th>
<td> (50,) </td>
<td> (50,) </td>
</tr>
<tr>
<th> Dask graph </th>
<td colspan="2"> 1 chunks in 2 graph layers </td>
</tr>
<tr>
<th> Data type </th>
<td colspan="2"> int64 numpy.ndarray </td>
</tr>
</tbody>
</table>
</td>
<td>
<svg width="170" height="79" style="stroke:rgb(0,0,0);stroke-width:1" >


<line x1="0" y1="0" x2="120" y2="0" style="stroke-width:2" />
<line x1="0" y1="29" x2="120" y2="29" style="stroke-width:2" />


<line x1="0" y1="0" x2="0" y2="29" style="stroke-width:2" />
<line x1="120" y1="0" x2="120" y2="29" style="stroke-width:2" />


<polygon points="0.0,0.0 120.0,0.0 120.0,29.030629010473877 0.0,29.030629010473877" style="fill:#ECB172A0;stroke-width:0"/>


<text x="60.000000" y="49.030629" font-size="1.0rem" font-weight="100" text-anchor="middle" >50</text>
<text x="140.000000" y="14.515315" font-size="1.0rem" font-weight="100" text-anchor="middle" transform="rotate(0,140.000000,14.515315)">1</text>
</svg>
</td>
</tr>
</table>

```python
a[:50, 200].compute()
```

array([ 200, 700, 1200, 1700, 2200, 2700, 3200, 3700, 4200,
4700, 5200, 5700, 6200, 6700, 7200, 7700, 8200, 8700,
9200, 9700, 10200, 10700, 11200, 11700, 12200, 12700, 13200,
13700, 14200, 14700, 15200, 15700, 16200, 16700, 17200, 17700,
18200, 18700, 19200, 19700, 20200, 20700, 21200, 21700, 22200,
22700, 23200, 23700, 24200, 24700])

```python
a.mean()
a.mean().compute()
np.sin(a)
np.sin(a).compute()
a.T
a.T.compute()
```

array([[ 0, 500, 1000, ..., 98500, 99000, 99500],
[ 1, 501, 1001, ..., 98501, 99001, 99501],
[ 2, 502, 1002, ..., 98502, 99002, 99502],
...,
[ 497, 997, 1497, ..., 98997, 99497, 99997],
[ 498, 998, 1498, ..., 98998, 99498, 99998],
[ 499, 999, 1499, ..., 98999, 99499, 99999]])

```python
b = a.max(axis=1)[::-1] + 10
```

```python
b[:10].compute()
```

array([100009, 99509, 99009, 98509, 98009, 97509, 97009, 96509,
96009, 95509])

```python
b.dask
```

<div>
<div>
<div style="width: 52px; height: 52px; position: absolute;">
<svg width="76" height="71" viewBox="0 0 76 71" fill="none" xmlns="http://www.w3.org/2000/svg">
<circle cx="61.5" cy="36.5" r="13.5" style="stroke: var(--jp-ui-font-color2, #1D1D1D); fill: var(--jp-layout-color1, #F2F2F2);" stroke-width="2"/>
<circle cx="14.5" cy="14.5" r="13.5" style="stroke: var(--jp-ui-font-color2, #1D1D1D); fill: var(--jp-layout-color1, #F2F2F2);" stroke-width="2"/>
<circle cx="14.5" cy="56.5" r="13.5" style="stroke: var(--jp-ui-font-color2, #1D1D1D); fill: var(--jp-layout-color1, #F2F2F2);" stroke-width="2"/>
<path d="M28 16L30.5 16C33.2614 16 35.5 18.2386 35.5 21L35.5 32.0001C35.5 34.7615 37.7386 37.0001 40.5 37.0001L43 37.0001" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="1.5"/>
<path d="M40.5 37L40.5 37.75L40.5 37.75L40.5 37ZM35.5 42L36.25 42L35.5 42ZM35.5 52L34.75 52L35.5 52ZM30.5 57L30.5 57.75L30.5 57ZM41.5001 36.25L40.5 36.25L40.5 37.75L41.5001 37.75L41.5001 36.25ZM34.75 42L34.75 52L36.25 52L36.25 42L34.75 42ZM30.5 56.25L28.0001 56.25L28.0001 57.75L30.5 57.75L30.5 56.25ZM34.75 52C34.75 54.3472 32.8472 56.25 30.5 56.25L30.5 57.75C33.6756 57.75 36.25 55.1756 36.25 52L34.75 52ZM40.5 36.25C37.3244 36.25 34.75 38.8243 34.75 42L36.25 42C36.25 39.6528 38.1528 37.75 40.5 37.75L40.5 36.25Z" style="fill: var(--jp-ui-font-color2, #1D1D1D);"/>
<circle cx="28" cy="16" r="2.25" fill="#E5E5E5" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="1.5"/>
<circle cx="28" cy="57" r="2.25" fill="#E5E5E5" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="1.5"/>
<path d="M45.25 36.567C45.5833 36.7594 45.5833 37.2406 45.25 37.433L42.25 39.1651C41.9167 39.3575 41.5 39.117 41.5 38.7321V35.2679C41.5 34.883 41.9167 34.6425 42.25 34.8349L45.25 36.567Z" style="fill: var(--jp-ui-font-color2, #1D1D1D);"/>
</svg>
</div>
<div style="margin-left: 64px;">
<h3 style="margin-bottom: 0px;">HighLevelGraph</h3>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin-bottom:0px;">
HighLevelGraph with 6 layers and 30 keys from all layers.
</p>

<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">

<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>

</svg>

<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer1: array</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
array-5f1b3c0ca172b03296ed6d431d3c9df7
</p>

<table>
<tr>
<td>
<table>

<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">MaterializedLayer</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">10</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">shape</th>
<td style="text-align: left;">(200, 500)</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">dtype</th>
<td style="text-align: left;">int64</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">chunksize</th>
<td style="text-align: left;">(100, 100)</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">type</th>
<td style="text-align: left;">dask.array.core.Array</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">chunk_type</th>
<td style="text-align: left;">numpy.ndarray</td>
</tr>

</table>
</td>
<td>
<svg width="250" height="130" style="stroke:rgb(0,0,0);stroke-width:1" >


<line x1="0" y1="0" x2="200" y2="0" style="stroke-width:2" />
<line x1="0" y1="40" x2="200" y2="40" />
<line x1="0" y1="80" x2="200" y2="80" style="stroke-width:2" />


<line x1="0" y1="0" x2="0" y2="80" style="stroke-width:2" />
<line x1="40" y1="0" x2="40" y2="80" />
<line x1="80" y1="0" x2="80" y2="80" />
<line x1="120" y1="0" x2="120" y2="80" />
<line x1="160" y1="0" x2="160" y2="80" />
<line x1="200" y1="0" x2="200" y2="80" style="stroke-width:2" />


<polygon points="0.0,0.0 200.0,0.0 200.0,80.0 0.0,80.0" style="fill:#ECB172A0;stroke-width:0"/>


<text x="100.000000" y="100.000000" font-size="1.0rem" font-weight="100" text-anchor="middle" >500</text>
<text x="220.000000" y="40.000000" font-size="1.0rem" font-weight="100" text-anchor="middle" transform="rotate(-90,220.000000,40.000000)">200</text>
</svg>
</td>
</tr>
</table>

</details>
</div>

<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">

<circle cx="16" cy="16" r="14" style="stroke: var(--jp-ui-font-color2, #1D1D1D); fill: var(--jp-layout-color1, #F2F2F2);" stroke-width="2" />

</svg>

<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer2: chunk_max</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
chunk_max-22359f6e7c214a78df6cd2673d3d603f
</p>

<table>
<tr>
<td>
<table>

<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">Blockwise</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">False</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">10</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">shape</th>
<td style="text-align: left;">(200, 500)</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">dtype</th>
<td style="text-align: left;">int64</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">chunksize</th>
<td style="text-align: left;">(100, 100)</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">type</th>
<td style="text-align: left;">dask.array.core.Array</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">chunk_type</th>
<td style="text-align: left;">numpy.ndarray</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;"> depends on </th>
<td style="text-align: left;">array-5f1b3c0ca172b03296ed6d431d3c9df7</td>
</tr>

</table>
</td>
<td>
<svg width="250" height="130" style="stroke:rgb(0,0,0);stroke-width:1" >


<line x1="0" y1="0" x2="200" y2="0" style="stroke-width:2" />
<line x1="0" y1="40" x2="200" y2="40" />
<line x1="0" y1="80" x2="200" y2="80" style="stroke-width:2" />


<line x1="0" y1="0" x2="0" y2="80" style="stroke-width:2" />
<line x1="40" y1="0" x2="40" y2="80" />
<line x1="80" y1="0" x2="80" y2="80" />
<line x1="120" y1="0" x2="120" y2="80" />
<line x1="160" y1="0" x2="160" y2="80" />
<line x1="200" y1="0" x2="200" y2="80" style="stroke-width:2" />


<polygon points="0.0,0.0 200.0,0.0 200.0,80.0 0.0,80.0" style="fill:#ECB172A0;stroke-width:0"/>


<text x="100.000000" y="100.000000" font-size="1.0rem" font-weight="100" text-anchor="middle" >500</text>
<text x="220.000000" y="40.000000" font-size="1.0rem" font-weight="100" text-anchor="middle" transform="rotate(-90,220.000000,40.000000)">200</text>
</svg>
</td>
</tr>
</table>

</details>
</div>

<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">

<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>

</svg>

<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer3: chunk_max-partial</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
chunk_max-partial-1dbed0d768d7fbafb3cb682ddb58870a
</p>

<table>
<tr>
<td>
<table>

<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">MaterializedLayer</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">4</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">shape</th>
<td style="text-align: left;">(200, 2)</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">dtype</th>
<td style="text-align: left;">int64</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">chunksize</th>
<td style="text-align: left;">(100, 1)</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">type</th>
<td style="text-align: left;">dask.array.core.Array</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">chunk_type</th>
<td style="text-align: left;">numpy.ndarray</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;"> depends on </th>
<td style="text-align: left;">chunk_max-22359f6e7c214a78df6cd2673d3d603f</td>
</tr>

</table>
</td>
<td>
<svg width="92" height="250" style="stroke:rgb(0,0,0);stroke-width:1" >


<line x1="0" y1="0" x2="42" y2="0" style="stroke-width:2" />
<line x1="0" y1="100" x2="42" y2="100" />
<line x1="0" y1="200" x2="42" y2="200" style="stroke-width:2" />


<line x1="0" y1="0" x2="0" y2="200" style="stroke-width:2" />
<line x1="21" y1="0" x2="21" y2="200" />
<line x1="42" y1="0" x2="42" y2="200" style="stroke-width:2" />


<polygon points="0.0,0.0 42.354360857637474,0.0 42.354360857637474,200.0 0.0,200.0" style="fill:#ECB172A0;stroke-width:0"/>


<text x="21.177180" y="220.000000" font-size="1.0rem" font-weight="100" text-anchor="middle" >2</text>
<text x="62.354361" y="100.000000" font-size="1.0rem" font-weight="100" text-anchor="middle" transform="rotate(-90,62.354361,100.000000)">200</text>
</svg>
</td>
</tr>
</table>

</details>
</div>

<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">

<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>

</svg>

<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer4: max-aggregate</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
max-aggregate-f1f4671fcd497326d08ac769e2eeff52
</p>

<table>
<tr>
<td>
<table>

<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">MaterializedLayer</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">2</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">shape</th>
<td style="text-align: left;">(200,)</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">dtype</th>
<td style="text-align: left;">int64</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">chunksize</th>
<td style="text-align: left;">(100,)</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">type</th>
<td style="text-align: left;">dask.array.core.Array</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">chunk_type</th>
<td style="text-align: left;">numpy.ndarray</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;"> depends on </th>
<td style="text-align: left;">chunk_max-partial-1dbed0d768d7fbafb3cb682ddb58870a</td>
</tr>

</table>
</td>
<td>
<svg width="250" height="92" style="stroke:rgb(0,0,0);stroke-width:1" >


<line x1="0" y1="0" x2="200" y2="0" style="stroke-width:2" />
<line x1="0" y1="42" x2="200" y2="42" style="stroke-width:2" />


<line x1="0" y1="0" x2="0" y2="42" style="stroke-width:2" />
<line x1="100" y1="0" x2="100" y2="42" />
<line x1="200" y1="0" x2="200" y2="42" style="stroke-width:2" />


<polygon points="0.0,0.0 200.0,0.0 200.0,42.354360857637474 0.0,42.354360857637474" style="fill:#ECB172A0;stroke-width:0"/>


<text x="100.000000" y="62.354361" font-size="1.0rem" font-weight="100" text-anchor="middle" >200</text>
<text x="220.000000" y="21.177180" font-size="1.0rem" font-weight="100" text-anchor="middle" transform="rotate(0,220.000000,21.177180)">1</text>
</svg>
</td>
</tr>
</table>

</details>
</div>

<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">

<circle cx="16" cy="16" r="14" fill="#8F8F8F" style="stroke: var(--jp-ui-font-color2, #1D1D1D);" stroke-width="2"/>

</svg>

<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer5: getitem</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
getitem-65be51186a8fbeccfd51c0cf1245f77e
</p>

<table>
<tr>
<td>
<table>

<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">MaterializedLayer</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">True</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">2</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">shape</th>
<td style="text-align: left;">(200,)</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">dtype</th>
<td style="text-align: left;">int64</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">chunksize</th>
<td style="text-align: left;">(100,)</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">type</th>
<td style="text-align: left;">dask.array.core.Array</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">chunk_type</th>
<td style="text-align: left;">numpy.ndarray</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;"> depends on </th>
<td style="text-align: left;">max-aggregate-f1f4671fcd497326d08ac769e2eeff52</td>
</tr>

</table>
</td>
<td>
<svg width="250" height="92" style="stroke:rgb(0,0,0);stroke-width:1" >


<line x1="0" y1="0" x2="200" y2="0" style="stroke-width:2" />
<line x1="0" y1="42" x2="200" y2="42" style="stroke-width:2" />


<line x1="0" y1="0" x2="0" y2="42" style="stroke-width:2" />
<line x1="100" y1="0" x2="100" y2="42" />
<line x1="200" y1="0" x2="200" y2="42" style="stroke-width:2" />


<polygon points="0.0,0.0 200.0,0.0 200.0,42.354360857637474 0.0,42.354360857637474" style="fill:#ECB172A0;stroke-width:0"/>


<text x="100.000000" y="62.354361" font-size="1.0rem" font-weight="100" text-anchor="middle" >200</text>
<text x="220.000000" y="21.177180" font-size="1.0rem" font-weight="100" text-anchor="middle" transform="rotate(0,220.000000,21.177180)">1</text>
</svg>
</td>
</tr>
</table>

</details>
</div>

<div style="">
<svg width="24" height="24" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg" style="position: absolute;">

<circle cx="16" cy="16" r="14" style="stroke: var(--jp-ui-font-color2, #1D1D1D); fill: var(--jp-layout-color1, #F2F2F2);" stroke-width="2" />

</svg>

<details style="margin-left: 32px;">
<summary style="margin-bottom: 10px; margin-top: 10px;">
<h4 style="display: inline;">Layer6: add</h4>
</summary>
<p style="color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;">
add-c42a5ea4600e83a48a31b107d8933476
</p>

<table>
<tr>
<td>
<table>

<tr>
<th style="text-align: left; width: 150px;">layer_type</th>
<td style="text-align: left;">Blockwise</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">is_materialized</th>
<td style="text-align: left;">False</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">number of outputs</th>
<td style="text-align: left;">2</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">shape</th>
<td style="text-align: left;">(200,)</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">dtype</th>
<td style="text-align: left;">int64</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">chunksize</th>
<td style="text-align: left;">(100,)</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">type</th>
<td style="text-align: left;">dask.array.core.Array</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;">chunk_type</th>
<td style="text-align: left;">numpy.ndarray</td>
</tr>

<tr>
<th style="text-align: left; width: 150px;"> depends on </th>
<td style="text-align: left;">getitem-65be51186a8fbeccfd51c0cf1245f77e</td>
</tr>

</table>
</td>
<td>
<svg width="250" height="92" style="stroke:rgb(0,0,0);stroke-width:1" >


<line x1="0" y1="0" x2="200" y2="0" style="stroke-width:2" />
<line x1="0" y1="42" x2="200" y2="42" style="stroke-width:2" />


<line x1="0" y1="0" x2="0" y2="42" style="stroke-width:2" />
<line x1="100" y1="0" x2="100" y2="42" />
<line x1="200" y1="0" x2="200" y2="42" style="stroke-width:2" />


<polygon points="0.0,0.0 200.0,0.0 200.0,42.354360857637474 0.0,42.354360857637474" style="fill:#ECB172A0;stroke-width:0"/>


<text x="100.000000" y="62.354361" font-size="1.0rem" font-weight="100" text-anchor="middle" >200</text>
<text x="220.000000" y="21.177180" font-size="1.0rem" font-weight="100" text-anchor="middle" transform="rotate(0,220.000000,21.177180)">1</text>
</svg>
</td>
</tr>
</table>

</details>
</div>

</div>
</div>
</div>

## Dask example 2

```python
import dask.array as da
x = da.random.random((30_000, 30_000), chunks=(1000, 1000))
x
```

<table>
<tr>
<td>
<table style="border-collapse: collapse;">
<thead>
<tr>
<td> </td>
<th> Array </th>
<th> Chunk </th>
</tr>
</thead>
<tbody>

<tr>
<th> Bytes </th>
<td> 6.71 GiB </td>
<td> 7.63 MiB </td>
</tr>

<tr>
<th> Shape </th>
<td> (30000, 30000) </td>
<td> (1000, 1000) </td>
</tr>
<tr>
<th> Dask graph </th>
<td colspan="2"> 900 chunks in 1 graph layer </td>
</tr>
<tr>
<th> Data type </th>
<td colspan="2"> float64 numpy.ndarray </td>
</tr>
</tbody>
</table>
</td>
<td>
<svg width="170" height="170" style="stroke:rgb(0,0,0);stroke-width:1" >


<line x1="0" y1="0" x2="120" y2="0" style="stroke-width:2" />
<line x1="0" y1="4" x2="120" y2="4" />
<line x1="0" y1="12" x2="120" y2="12" />
<line x1="0" y1="16" x2="120" y2="16" />
<line x1="0" y1="24" x2="120" y2="24" />
<line x1="0" y1="28" x2="120" y2="28" />
<line x1="0" y1="36" x2="120" y2="36" />
<line x1="0" y1="44" x2="120" y2="44" />
<line x1="0" y1="48" x2="120" y2="48" />
<line x1="0" y1="56" x2="120" y2="56" />
<line x1="0" y1="60" x2="120" y2="60" />
<line x1="0" y1="68" x2="120" y2="68" />
<line x1="0" y1="72" x2="120" y2="72" />
<line x1="0" y1="80" x2="120" y2="80" />
<line x1="0" y1="88" x2="120" y2="88" />
<line x1="0" y1="92" x2="120" y2="92" />
<line x1="0" y1="100" x2="120" y2="100" />
<line x1="0" y1="104" x2="120" y2="104" />
<line x1="0" y1="112" x2="120" y2="112" />
<line x1="0" y1="120" x2="120" y2="120" style="stroke-width:2" />


<line x1="0" y1="0" x2="0" y2="120" style="stroke-width:2" />
<line x1="4" y1="0" x2="4" y2="120" />
<line x1="12" y1="0" x2="12" y2="120" />
<line x1="16" y1="0" x2="16" y2="120" />
<line x1="24" y1="0" x2="24" y2="120" />
<line x1="28" y1="0" x2="28" y2="120" />
<line x1="36" y1="0" x2="36" y2="120" />
<line x1="44" y1="0" x2="44" y2="120" />
<line x1="48" y1="0" x2="48" y2="120" />
<line x1="56" y1="0" x2="56" y2="120" />
<line x1="60" y1="0" x2="60" y2="120" />
<line x1="68" y1="0" x2="68" y2="120" />
<line x1="72" y1="0" x2="72" y2="120" />
<line x1="80" y1="0" x2="80" y2="120" />
<line x1="88" y1="0" x2="88" y2="120" />
<line x1="92" y1="0" x2="92" y2="120" />
<line x1="100" y1="0" x2="100" y2="120" />
<line x1="104" y1="0" x2="104" y2="120" />
<line x1="112" y1="0" x2="112" y2="120" />
<line x1="120" y1="0" x2="120" y2="120" style="stroke-width:2" />


<polygon points="0.0,0.0 120.0,0.0 120.0,120.0 0.0,120.0" style="fill:#8B4903A0;stroke-width:0"/>


<text x="60.000000" y="140.000000" font-size="1.0rem" font-weight="100" text-anchor="middle" >30000</text>
<text x="140.000000" y="60.000000" font-size="1.0rem" font-weight="100" text-anchor="middle" transform="rotate(-90,140.000000,60.000000)">30000</text>
</svg>
</td>
</tr>
</table>

```python
y = x + x.T
```

```python
y.sum().compute()
```

899986697.6624435

```python
y[:, :10].compute()
```

array([[1.06732263, 1.44918471, 0.97747399, ..., 0.33531255, 0.81407412,
1.18334048],
[1.44918471, 0.75954053, 0.94024149, ..., 0.92522134, 0.47216556,
1.0685656 ],
[0.97747399, 0.94024149, 0.43425583, ..., 1.31146439, 1.13960695,
1.17543105],
...,
[0.94341018, 0.91293267, 0.4737179 , ..., 0.95118153, 0.68525142,
0.9561541 ],
[0.68742121, 0.76071136, 0.78407903, ..., 0.96844288, 0.57027703,
0.44666467],
[1.05163969, 1.174239 , 0.61947849, ..., 0.22588196, 0.7914612 ,
0.8904959 ]])

```python

```

JupyterHub

2024-02-21T13:23:33Z

Torradeflot: /* Dask */

= Introduction =

PIC offers a service for running Jupyter notebooks on CPU or GPU resources. This service is primarily thought for code developing or prototyping rather than data processing. The usage is similar to running notebooks on your personal computer but offers the advantage of developing and testing your code on different hardware configurations, as well as facilitating the scalability of the code since it is being tested in the same environment in which it would run on a mass scale.

Since the service is strictly thought for development and small scale testing tasks, a shutdown policy for the sessions has been put in place:

# The maximum duration for a session is 48h.
# After an idle period of 2 hours, the session will be closed.

In practice that means that you should estimate the test data volume that you work with during a session to be able to be processed in less than 48 hours.

== How to connect to the service ==

Got to [https://jupyter.pic.es jupyter.pic.es] to see your login screen.

[[File:login.png|frameless|700px|Login screen]]

Sign in with your PIC user credentials. This will prompt you to the following screen.

[[File:ScreenshotJupyterSpawn.png|700px|current]]

Here you can choose the hardware configuration for your Jupyter session. Also, you have to choose the experiment (project) you are working on during the Jupyter session. After choosing a configuration and pressing start the next screen will show you the progress of the initialisation process. Keep in mind that a job containing your Jupyter session is actually sent to the HTCondor queuing system and waiting for available resources before being started. This usually takes less than a minute but can take up to a few depending on our resource usage.

[[File:screen02.png|600px]]

In the next screen you can choose the tool that you want to use for your work: a Python notebook, a Python console or a plain bash terminal.
For the Python environment (either notebook or environment) you have two default options:
* the ipykernel version of Python 3
* the XPython version of Python 3.9, this one allows you to use the integrated debugging module.

Further you see an icon with a "D" - desktop, this one starts a VNC session that allows the use of programs with graphical user interfaces.

Also, recently you can find the icon of Visual Studio, an integrated development environment.

[[File:ScreenshotJupyterlab20231103.png|700px]]

Your python environments should appear under Notebook and Console headers. In a later section we will show you how to create a new environment and to remove an existing one.

== Terminate your session and logout ==

It is important that you terminate your session before you log out. In order to do so, go to the top page menu "'''File''' -> '''Hub Control Panel'''" and you will see the following screen.

[[File:screen04.png|600px]]

Here, click on the '''Stop My Server''' button. After that you can log out by clicking the '''Logout''' button in the right upper corner.

= Python virtual environments =

This section covers the use of Python virtual environments with Jupyter.

== Initialize conda (we highly recommend the use of mambaforge) ==

Before using conda/mamba in your bash session, you have to initialize it.
* For access to an available conda/mamba installation, please get in contact with your project liaison at PIC. He/she will give you the actual value for the '''/path/to/mambaforge''' placeholder.
* If you want to use your own conda/mamba installation, we recommend you to install the minimal '''miniforge''' distribution, instructions [https://github.com/conda-forge/miniforge here]

Log onto Jupyter and start a session. On the homepage of your Jupyter session, click on the terminal button on the session dashboard on the right to open a bash terminal. If no specific version is needed you can use the link provided in the example.

First, let's initialize conda for our bash sessions:
<pre>
[neissner@td110 ~]$ /data/astro/software/centos7/conda/mambaforge_4.14.0/bin/mamba init
</pre>

This actually changes the .bashrc file in your home directory in order to activate the base environment on login.
To avoid that the base environment is activated every time you log on to a node, run:
<pre>
[neissner@td110 ~]$ conda config --set auto_activate_base false
</pre>

For now you can exit the terminal.
<pre>
[neissner@td110 ~]$ exit
</pre>

== Link an existing environment to Jupyter ==

You can find instructions on how to create your own environments, e.g. [[#Create_virtual_environments_with_venv_or_conda | here]].

Log into Jupyter, start a session. From the session dashboard choose the bash terminal.

Inside the terminal, activate your environment.

For '''conda/mamba''' environments:
* if you created the environment without a prefix:
<pre>
[neissner@td110 ~]$ mamba activate environment
(...) [neissner@td110 ~]$
</pre>
The parenthesis (...) in front of your bash prompt show the name of your environment.
* if you created the environment with a prefix:
<pre>
[neissner@td110 ~]$ mamba activate /path/to/environment
(...) [neissner@td110 ~]$
</pre>
The parenthesis (...) in front of your bash prompt show the absolute path of your environment.

For '''venv''' environments:
<pre>
[neissner@td110 ~]$ source /path/to/environment/bin/activate
(...) [neissner@td110 ~]$
</pre>

Link the environment to a Jupyter kernel. For both, '''conda/mamba''' and '''venv''':
<pre>
(...) [neissner@td110 ~]$ python -m ipykernel install --user --name=whatever_kernel_name
Installed kernelspec whatever_kernel_name in
/nfs/pic.es/user/n/neissner/.local/share/jupyter/kernels/whatever_kernel_name
</pre>

If you don't have the '''ipykernel''' module installed in your environment you may receive an error message like the one below when trying to run the previous command.
<pre>No module named ipykernel</pre>

If this is the case, you need to install it by running: '''pip install ipykernel'''

Deactivate your environment.

For conda:
<pre>
(...) [neissner@td110 ~]$ mamba deactivate
</pre>

For venv:
<pre>
(...) [neissner@td110 ~]$ deactivate
</pre>

Now you can exit the terminal. After refreshing the Jupyter page your whatever_kernel_name appears in the dashboard. In this example '''test''' has been used for whatever_kernel_name

[[File:screen05.png|700px]]

== Unlink an environment from Jupyter ==
Log onto Jupyter, start a session and from the session dashboard choose the bash terminal. To remove your environment/kernel from Jupyter run:
<pre>
[neissner@td110 ~]$ jupyter kernelspec uninstall whatever_kernel_name
Kernel specs to remove:
whatever_kernel_name /nfs/pic.es/user/n/neissner/.local/share/jupyter/kernels/whatever_kernel_name
Remove 1 kernel specs [y/N]: y
[RemoveKernelSpec] Removed /nfs/pic.es/user/n/neissner/.local/share/jupyter/kernels/whatever_kernel_name
</pre>
Keep in mind that, although not available in Jupyter anymore, the environment still exists. Whenever you need it, you can link it again.

== Create virtual environments with venv or conda ==

Before creating a new environment, please get in contact with your project liaison at PIC as there may be already a suitable environment for your needs in place.

If none of the existing environments suits your needs, you can create a new environment.
First, create a directory in a suitable place to store the environment. For single-user environments, place them in your home under ~/env. For environments that will be shared with other project users, contact your project liaison and ask him/her for a path in a shared storage volume that is visible to all of them.

Once you have the location (i.e. /path/to/env/folder), create the environment with the following commands:

For '''venv''' environments '''(recommended)'''

If your_env is installed at /path/to/env/your_env
<pre>
[neissner@td110 ~]$ cd /path/to/env
[neissner@td110 ~]$ python3 -m venv your_env
</pre>
Now you should be able to activate your environment and install additional modules
<pre>
[neissner@td110 ~]$ cd /path/to/env
[neissner@td110 ~]$ source your_env/bin/activate
(...)[neissner@td110 ~]$ pip install additional_module1 additional_module2 ...
</pre>

For '''conda/mamba''' environments
<pre>
[neissner@td110 ~]$ mamba create --prefix /path/to/env/your_env
</pre>
The list of modules (module1, module2, ...) is optional. For instance, for a python3 environment with scipy you would specify: ''python=3 scipy''

Now you should be able to activate your environment and install additional modules
<pre>
[neissner@td110 ~]$ mamba activate /path/to/env/folder/your_env
(...)[neissner@td110 ~]$ mamba install additional_module1 additional_module2 ...
</pre>
You can use pip install inside a mamba environment, however, resolving dependencies might require installing additional packages manually.

= Proper usage of X509 based proxies =

We found recently that the usage of proxies within a Jupyter session might cause problems because the environment changes certain standard locations such as '''/tmp'''

For a correct functioning please create the proxy the following way, example for Virgo:

<pre>
[<user>@<hostname> ~]$ /bin/voms-proxy-init --voms virgo:/virgo/ligo --out ./x509up_u$(id -u)
[<user>@<hostname> ~]$ export X509_USER_PROXY=./x509up_u$(id -u)
[<user>@<hostname> ~]$ ls /cvmfs/ligo.osgstorage.org
ls: cannot access /cvmfs/ligo.osgstorage.org: Permission denied
</pre>

Here the proxy cannot properly be located. Therefore we have to put the complete path into the variable:

<pre>
[<user>@<hostname> ~]$ pwd
/nfs/pic.es/user/<letter>/<user>
[<user>@<hostname> ~]$ export X509_USER_PROXY=/nfs/pic.es/user/<letter>/<user>/x509up_u$(id -u)
[<user>@<hostname> ~]$ ls /cvmfs/ligo.osgstorage.org
frames powerflux pycbc test_access
</pre>

= Software of particular interest =
== SageMath ==

[https://www.sagemath.org/ SageMath] is particularly interesting for Cosmology because it allows symbolic calculations, e.g. deriving the equations of motions for the scale factor starting from a customised space-time metric.

=== Standard cosmology examples ===

* The Friedman equations for the FLRW solution of the Einstein equations.
You can find the corresponding Notebook in any PIC terminal at '''/data/astro/software/notebooks/FLRW_cosmology.ipynb'''

* The notebook you can find at '''/data/astro/software/notebooks/FLRW_cosmology_solutions.ipynb''' uses known analytical solutions of the FLRW cosmology and produces this image for the evolution of the scale factor:

[[File:Screenshot_Sage05.png|300px]]

* The notebook you can find at '''/data/astro/software/notebooks/Interior_Schwarzschild.ipynb''' shows the formalism for the interior Schwarzschild metric and displays the solutions for density and pressure of a static celestial object that is sufficiently larger than its Schwarzschild radius. The pressure for an object with constant density id shown in the image:

[[File:Screenshot_Sage06.png|300px]]

=== Enabling SageMath environment in Jupyter ===

If you have never initialized mamba, run:

<pre>
[<user>@<hostname> ~]$ /data/astro/software/centos7/conda/mambaforge_4.14.0/bin/mamba init
[<user>@<hostname> ~]$ conda config --set auto_activate_base false
</pre>

After that you can enable SageMath for its use in a Jupyter notebook session:

<pre>
[<user>@<hostname> ~] mamba activate /data/astro/software/envs/sage
(/data/astro/software/envs/sage) [<user>@<hostname> ~]$ python -m ipykernel install --user --name=sage
....
(/data/astro/software/envs/sage) [<user>@<hostname> ~]$ mamba deactivate
</pre>

This creates a file in you home '''~/.local/share/jupyter/kernels/sage/kernel.json'''
which has to be modified to look like this:

<pre>
{
"argv": [
"/data/astro/software/envs/sage/bin/sage",
"--python",
"-m",
"sage.repl.ipython_kernel",
"-f",
"{connection_file}"
],
"display_name": "sage",
"language": "sage",
"metadata": {
"debugger": true
}
}
</pre>

Next time you go to your Jupyter dashboard you will find the sage environment listed there.

= Dask =

[[Media:dask_htcondor.pdf|Dask + HTCondor manual]]

[[Dask wiki]]

= Using a singularity image as a jupyter kernel =

Sometimes the software stack of some projects may be provided in the shape of a singularity image, it will then be convenient to use this image as a kernel for the notebooks in jupyter.pic.es.

The singularity image to be used as a kernel needs to fullfill some requirements. Different requirements will apply depending on the programming language.

== python jupyter kernel in a singularity image ==

The singularity image needs to have the '''python''' and the '''ipykernel''' module installed.

* Create the folder that will host the kernel definition

mkdir -p $HOME/.local/share/jupyter/kernels/singularity

* Create the '''kernel.json''' file inside it with the following content:

{
"argv": [
"singularity",
"exec",
"--cleanenv",
"/path/to/the/singularity/image.sif",
"python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"language": "python",
"display_name": "singularity-kernel"
}

Refresh or start the jupyterlab interface and the singularity kernel should appear in the launcher tab

= GPUs =

The way to identify the GPUs that are assigned to your job is:
* check the environment variable CUDA_VISIBLE_DEVICES. In a terminal run "echo $CUDA_VISIBLE_DEVICES". The environment variable contains a list of comma-separated GPU ids. With this you will already know how many gpus are assigned to your job. If the variable does not exist, there are no gpus assigned to the job

* list the gpus with nvidia-smi, in a terminal run "nvidia-smi -L", and look for the gpus you've been assigned. Remember their indexes (integers from 0 to 7)

[[File:check_gpu_id_highlighted.png]]

* in the GPU dashboard the gpus are identified with their index

[[File:check_gpu_resources_highlighted.png]]

= Jupyterlab user guide =

You can find the official documentation of the currently installed version of jupyterlab (3.6) [https://jupyterlab.readthedocs.io/en/3.6.x/ here], there you will find instruction on how to
* [https://jupyterlab.readthedocs.io/en/3.6.x/user/commands.html Access the command palette]
* [https://jupyterlab.readthedocs.io/en/3.6.x/user/toc.html Build a Table Of Contents]
* [https://jupyterlab.readthedocs.io/en/3.6.x/user/debugger.html Debug your code]

A set of non-official jupyterlab extensions are installed to provide additional functionalities

== jupytext ==
Pair your notebooks with text files to enhance version tracking.
https://jupytext.readthedocs.io

=== Example ===

If you had a notebook (.ipynb file) containing only the cell below, tracked in a git repository

<pre>
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
plt.imshow(np.random.random([10, 10]))
</pre>

Different executions of the cell would produce different images, and the images are embedded in a pseudo-binary format inside the notebook file. In this case, doing a '''git diff''' of the .ipynb file would produce a huge output (because the image changed), even if there wasn't any change in the code. It is thus convenient to sync the notebook with a text file (e.g. a .py script) using the jupytext extension and track this one with git. The outputs, including images, as well as some additional metadata, won't be added to the synced text file. So in the case of different executions of the same notebook, the diff will always be empty.

== git ==
Sidebar GUI to git repo management
https://github.com/jupyterlab/jupyterlab-git

== variable inspector ==
Variable inspection à la Matlab
https://github.com/jupyterlab-contrib/jupyterlab-variableInspector