Difference between revisions of "Transferring data to/from PIC"

From Public PIC Wiki
Jump to navigation Jump to search
Line 1: Line 1:
−
= How to provide data access to PIC massive storage (dCache) =
+
= How to provide data access to PIC massive storage =
 +
 
 +
This page explains how to configure <code>rclone</code> to access PIC storage.
 +
 
 +
The configuration depends on the storage path you need to access.
 +
 
 +
{| class="wikitable"
 +
|+ Storage access methods
 +
! Storage path
 +
! Protocol
 +
! rclone backend
 +
! Supported authentication
 +
! Example remote name
 +
|-
 +
| Paths under <code>/pnfs/pic.es/data</code>
 +
| WebDAV / HTTPS
 +
| <code>webdav</code>
 +
| OIDC token
 +
| <code>pic-pnfs-oidc</code>
 +
|-
 +
| Paths under <code>/pnfs/pic.es/data</code>
 +
| WebDAV / HTTPS
 +
| <code>webdav</code>
 +
| Macaroon bearer token
 +
| <code>pic-pnfs-macaroon</code>
 +
|-
 +
| Any other storage location
 +
| SSH / SFTP
 +
| <code>sftp</code>
 +
| SSH with <code>ControlMaster</code>
 +
| <code>pic-storage</code>
 +
|}
  
 
== Requirements ==
 
== Requirements ==
  
−
* Install and configure Rclone
+
You need:
−
* PIC credentials or macaroon
+
 
 +
* <code>rclone</code> installed on your client machine.
 +
* The storage path or endpoint provided by your PIC contact.
 +
 
 +
For paths under <code>/pnfs/pic.es/data</code>, you also need one of the following:
 +
 
 +
* an OIDC token setup using <code>oidc-agent</code>; or
 +
* a Macaroon token.
 +
 
 +
For any other storage location, you also need:
 +
 
 +
* SSH access to the PIC host.
 +
* SSH <code>ControlMaster</code> configured on your client machine.
 +
 
 +
== Install rclone ==
 +
 
 +
You can download the rclone binary directly without installing system packages. For example, on a Linux 64-bit machine:
 +
 
 +
<pre>
 +
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip
 +
$ unzip rclone-current-linux-amd64.zip
 +
</pre>
 +
 
 +
Alternatively, on Ubuntu you can install the Debian package:
 +
 
 +
<pre>
 +
$ cd /tmp
 +
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'
 +
$ sudo apt install ./rclone-current-linux-amd64.deb
 +
</pre>
 +
 
 +
Check that rclone works:
 +
 
 +
<pre>
 +
$ rclone version
 +
</pre>
 +
 
 +
== Which configuration should I use? ==
 +
 
 +
Use the table below to choose the right configuration.
 +
 
 +
{| class="wikitable"
 +
|+ Choosing the right configuration
 +
! If you want to access...
 +
! Use this section
 +
|-
 +
| A path under <code>/pnfs/pic.es/data</code> using OIDC authentication
 +
| [[#Configure rclone for PNFS data paths using OIDC|Configure rclone for PNFS data paths using OIDC]]
 +
|-
 +
| A path under <code>/pnfs/pic.es/data</code> using a Macaroon token
 +
| [[#Configure rclone for PNFS data paths using a Macaroon|Configure rclone for PNFS data paths using a Macaroon]]
 +
|-
 +
| Any other storage location
 +
| [[#Configure rclone for any other storage location|Configure rclone for any other storage location]]
 +
|}
 +
 
 +
== Configure rclone for PNFS data paths ==
 +
 
 +
Paths under <code>/pnfs/pic.es/data</code> are accessed using WebDAV over HTTPS.
 +
 
 +
For these paths, configure rclone with the <code>webdav</code> backend.
 +
 
 +
Use one of the following authentication methods:
 +
 
 +
* OIDC token, retrieved automatically with <code>oidc-token</code>.
 +
* Macaroon token, pasted into rclone as a bearer token.
 +
 
 +
Do not configure username/password authentication unless PIC explicitly instructs you to do so.
 +
 
 +
== Configure rclone for PNFS data paths using OIDC ==
 +
 
 +
Use this option if you have been instructed to authenticate with OIDC.
 +
 
 +
=== OIDC requirements ===
 +
 
 +
Before configuring rclone, make sure that:
 +
 
 +
* <code>oidc-agent</code> is available.
 +
* You have configured an OIDC account, for example <code>pic-pnfs</code>.
 +
* The following command returns a token:
 +
 
 +
<pre>
 +
$ oidc-token pic-pnfs
 +
eyJhbGciOiJSUzI1[...]4YjAwg
 +
</pre>
 +
 
 +
If this command does not work, see [[#Configuring oidc-agent for OIDC tokens|Configuring oidc-agent for OIDC tokens]].
 +
 
 +
=== Configure the WebDAV remote using OIDC ===
 +
 
 +
Start the rclone configuration wizard:
 +
 
 +
<pre>
 +
$ rclone config
 +
</pre>
 +
 
 +
Create a new remote:
 +
 
 +
<pre>
 +
No remotes found, make a new one?
 +
n) New remote
 +
s) Set configuration password
 +
q) Quit config
 +
n/s/q> n
 +
 
 +
Enter name for new remote.
 +
name> pic-pnfs-oidc
 +
</pre>
 +
 
 +
Select WebDAV:
 +
 
 +
<pre>
 +
Option Storage.
 +
Type of storage to configure.
 +
Choose a number from below, or type in your own value.
 +
Storage> webdav
 +
</pre>
 +
 
 +
Enter the WebDAV endpoint provided by PIC.
 +
 
 +
For a path under <code>/pnfs/pic.es/data</code>, the URL will normally point to that path or to a subdirectory under it.
 +
 
 +
Example:
 +
 
 +
<pre>
 +
Option url.
 +
URL of http host to connect to.
 +
url> https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE
 +
</pre>
 +
 
 +
Select vendor <code>other</code>:
 +
 
 +
<pre>
 +
Option vendor.
 +
Name of the WebDAV site/service/software you are using.
 +
vendor> other
 +
</pre>
 +
 
 +
Leave username empty:
 +
 
 +
<pre>
 +
Option user.
 +
User name.
 +
user>
 +
</pre>
 +
 
 +
Leave password empty:
 +
 
 +
<pre>
 +
Option pass.
 +
Password.
 +
y) Yes, type in my own password
 +
g) Generate random password
 +
n) No, leave this optional password blank
 +
y/g/n> n
 +
</pre>
 +
 
 +
When asked for a bearer token, leave it empty:
 +
 
 +
<pre>
 +
Option bearer_token.
 +
Bearer token instead of user/pass, for example a Macaroon.
 +
bearer_token>
 +
</pre>
 +
 
 +
Edit the advanced configuration:
 +
 
 +
<pre>
 +
Edit advanced config?
 +
y) Yes
 +
n) No
 +
y/n> y
 +
</pre>
 +
 
 +
Set <code>bearer_token_command</code>:
 +
 
 +
<pre>
 +
Option bearer_token_command.
 +
Command to run to get a bearer token.
 +
bearer_token_command> oidc-token pic-pnfs
 +
</pre>
 +
 
 +
Accept the remaining advanced options by pressing <code>ENTER</code>, unless PIC instructed you to change them.
 +
 
 +
At the end, review and accept the configuration:
 +
 
 +
<pre>
 +
Configuration complete.
 +
Options:
 +
- type: webdav
 +
- url: https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE
 +
- vendor: other
 +
- bearer_token_command: oidc-token pic-pnfs
 +
 
 +
Keep this "pic-pnfs-oidc" remote?
 +
y) Yes this is OK
 +
e) Edit this remote
 +
d) Delete this remote
 +
y/e/d> y
 +
</pre>
 +
 
 +
Quit the rclone configuration menu:
 +
 
 +
<pre>
 +
e) Edit existing remote
 +
n) New remote
 +
d) Delete remote
 +
r) Rename remote
 +
c) Copy remote
 +
s) Set configuration password
 +
q) Quit config
 +
e/n/d/r/c/s/q> q
 +
</pre>
 +
 
 +
=== OIDC configuration example ===
 +
 
 +
The resulting rclone configuration should look similar to:
 +
 
 +
<pre>
 +
[pic-pnfs-oidc]
 +
type = webdav
 +
url = https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE
 +
vendor = other
 +
bearer_token_command = oidc-token pic-pnfs
 +
</pre>
 +
 
 +
=== OIDC usage examples ===
 +
 
 +
List the configured remote root:
 +
 
 +
<pre>
 +
$ rclone lsd pic-pnfs-oidc:
 +
</pre>
 +
 
 +
List a directory below the configured path:
 +
 
 +
<pre>
 +
$ rclone lsd pic-pnfs-oidc:PROJECT_OR_DIRECTORY
 +
</pre>
  
−
== Install Rclone ==
+
List files recursively:
  
−
You can directly download the binary without installing anything. For instance, for a linux 64 bits machine:
+
<pre>
 +
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY
 +
</pre>
  
−
    $ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip
+
Download data:
−
    [...]
 
−
    $ unzip rclone-current-linux-amd64.zip
 
  
 +
<pre>
 +
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy
 +
</pre>
  
−
Or if you prefer, you can install Rclone like the next example on a Ubuntu machine:
+
Upload data:
  
−
    $ cd /tmp
+
<pre>
−
    $ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'
+
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY
−
    $ sudo apt install ./rclone-current-linux-amd64.deb
+
</pre>
  
−
== Configure Rclone ==
+
Example using a concrete path below the configured WebDAV root:
  
−
You should have been given some credentials and the url of a WebDAV endpoint at PIC.
+
<pre>
−
With them, you just need to create the config in rclone:
+
$ rclone copy ./ntuples pic-pnfs-oidc:analysis/ntuples
 +
</pre>
  
−
    $ rclone config
+
== Configure rclone for PNFS data paths using a Macaroon ==
−
    No remotes found, make a new one?
 
−
    n) New remote
 
−
    s) Set configuration password
 
−
    q) Quit config
 
−
    n/s/q> n
 
−
   
 
−
    Enter name for new remote.
 
−
    name> pic
 
−
   
 
−
    Option Storage.
 
−
    Type of storage to configure.
 
−
    Choose a number from below, or type in your own value.
 
−
    1 / 1Fichier
 
−
      \ (fichier)
 
−
    [...]
 
−
    Storage> webdav
 
−
   
 
−
    Option url.
 
−
    URL of http host to connect to.
 
−
    E.g. https://example.com.
 
−
    Enter a value.
 
−
    url>  https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE
 
−
   
 
−
    Option vendor.
 
−
    Name of the WebDAV site/service/software you are using.
 
−
    Choose a number from below, or type in your own value.
 
−
    Press Enter to leave empty.
 
−
    1 / Nextcloud
 
−
      \ (nextcloud)
 
−
    [...]
 
−
    5 / Other site/service or software
 
−
      \ (other)
 
−
    vendor> other
 
−
   
 
−
===  Using your PIC credentials ===
 
  
−
If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank.
+
Use this option if you have been given a Macaroon token.
  
−
    Option user.
+
The Macaroon is used as a WebDAV bearer token.
−
    User name.
 
−
    In case NTLM authentication is used, the username should be in the format 'Domain\User'.
 
−
    Enter a value. Press Enter to leave empty.
 
−
    user> YOUR_PIC_USERNAME
 
  
−
    Option pass.
+
=== Configure the WebDAV remote using a Macaroon ===
−
    Password.
 
−
    Choose an alternative below. Press Enter for the default (n).
 
−
    y) Yes, type in my own password
 
−
    g) Generate random password
 
−
    n) No, leave this optional password blank (default)
 
−
    y/g/n> y
 
−
    Enter the password:
 
−
    password: YOUR_PIC_PASSWORD
 
−
    Confirm the password:
 
−
    Password: YOUR_PIC_PASSWORD
 
  
−
=== Using a Macaroon token ===
+
Start the rclone configuration wizard:
  
−
If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank
+
<pre>
 +
$ rclone config
 +
</pre>
  
−
    Option bearer_token.
+
Create a new remote:
−
    Bearer token instead of user/pass (e.g. a Macaroon).
 
−
    Enter a value. Press Enter to leave empty.
 
−
    bearer_token> YOUR_MACAROON_TOKEN
 
  
−
=== Using an OIDC token ===
+
<pre>
 +
No remotes found, make a new one?
 +
n) New remote
 +
s) Set configuration password
 +
q) Quit config
 +
n/s/q> n
  
−
    Option bearer_token.
+
Enter name for new remote.
−
    Bearer token instead of user/pass (e.g. a Macaroon).
+
name> pic-pnfs-macaroon
−
    Enter a value. Press Enter to leave empty.
+
</pre>
−
    bearer_token>  
 
−
   
 
−
    Edit advanced config?
 
−
    y) Yes
 
−
    n) No (default)
 
−
    y/n> y
 
−
   
 
−
    Option bearer_token_command.
 
−
    Command to run to get a bearer token.
 
−
    Enter a value. Press Enter to leave empty.
 
−
    bearer_token_command> oidc-token OIDC_AGENT_ACCOUNT_SHORTNAME
 
  
−
=== Review settings ===
+
Select WebDAV:
  
−
At the end, just review the information you entered and confirm.
+
<pre>
 +
Option Storage.
 +
Type of storage to configure.
 +
Choose a number from below, or type in your own value.
 +
Storage> webdav
 +
</pre>
  
−
    Edit advanced config?
+
Enter the WebDAV endpoint provided by PIC.
−
    y) Yes
 
−
    n) No (default)
 
−
    y/n> n
 
−
   
 
−
    Configuration complete.
 
−
    Options:
 
−
    - type: webdav
 
−
    - url: https://door04.pic.es/PATH_TO_YOUR_STORAGE_SPACE
 
−
    - vendor: other
 
−
    - user: YOUR_PIC_USERNAME
 
−
    - pass: *** ENCRYPTED ***
 
−
    Keep this "pic" remote?
 
−
    y) Yes this is OK (default)
 
−
    e) Edit this remote
 
−
    d) Delete this remote
 
−
    y/e/d> y
 
−
   
 
−
    Current remotes:
 
−
   
 
−
    Name                Type
 
−
    ====                ====
 
−
    pic                  webdav
 
−
   
 
−
    e) Edit existing remote
 
−
    n) New remote
 
−
    d) Delete remote
 
−
    r) Rename remote
 
−
    c) Copy remote
 
−
    s) Set configuration password
 
−
    q) Quit config
 
−
    e/n/d/r/c/s/q> q
 
  
−
Once done, you can use command line to browse and download/upload data.
+
For a path under <code>/pnfs/pic.es/data</code>, the URL will normally point to that path or to a subdirectory under it.
  
−
=== Usage ===
+
Example:
  
−
* List a remote PIC directory:
+
<pre>
−
  rclone lsd <name>:<path>`
+
Option url.
 +
URL of http host to connect to.
 +
url> https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE
 +
</pre>
  
−
* Download a remote directory from PIC
+
Select vendor <code>other</code>:
−
  rclone copy <name>:<path> <local_path>
 
  
−
* Upload a local directory to PIC
+
<pre>
−
  rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path>
+
Option vendor.
 +
Name of the WebDAV site/service/software you are using.
 +
vendor> other
 +
</pre>
  
−
When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files
+
Leave username empty:
−
  --check-first -P --stats-one-line --transfers <n_transfers> --size-only
 
  
−
If uploading lots onto directories with lots of files (>1000), please use:
+
<pre>
−
  --no-traverse
+
Option user.
 +
User name.
 +
user>
 +
</pre>
  
−
If uploading files larger than 200 MB, also use
+
Leave password empty:
−
  --multi-thread-streams 1
+
 
 +
<pre>
 +
Option pass.
 +
Password.
 +
y) Yes, type in my own password
 +
g) Generate random password
 +
n) No, leave this optional password blank
 +
y/g/n> n
 +
</pre>
  
−
If uploading very large files (>10G), also use the following to allow more time to compute the checksums
+
Paste the Macaroon token as the bearer token:
−
  --timeout=15m
 
  
−
See rclone manual for more extensive documentation https://rclone.org/docs/
+
<pre>
 +
Option bearer_token.
 +
Bearer token instead of user/pass, for example a Macaroon.
 +
bearer_token> YOUR_MACAROON_TOKEN
 +
</pre>
  
−
== Configuring oidc-agent for obtaining OIDC tokens ==
+
You normally do not need to edit the advanced configuration:
 +
 
 +
<pre>
 +
Edit advanced config?
 +
y) Yes
 +
n) No
 +
y/n> n
 +
</pre>
 +
 
 +
At the end, review and accept the configuration:
 +
 
 +
<pre>
 +
Configuration complete.
 +
Options:
 +
- type: webdav
 +
- url: https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE
 +
- vendor: other
 +
- bearer_token: *** ENCRYPTED ***
 +
 
 +
Keep this "pic-pnfs-macaroon" remote?
 +
y) Yes this is OK
 +
e) Edit this remote
 +
d) Delete this remote
 +
y/e/d> y
 +
</pre>
 +
 
 +
Quit the rclone configuration menu:
 +
 
 +
<pre>
 +
e) Edit existing remote
 +
n) New remote
 +
d) Delete remote
 +
r) Rename remote
 +
c) Copy remote
 +
s) Set configuration password
 +
q) Quit config
 +
e/n/d/r/c/s/q> q
 +
</pre>
 +
 
 +
=== Macaroon configuration example ===
 +
 
 +
The resulting rclone configuration should look similar to:
 +
 
 +
<pre>
 +
[pic-pnfs-macaroon]
 +
type = webdav
 +
url = https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE
 +
vendor = other
 +
bearer_token = YOUR_MACAROON_TOKEN
 +
</pre>
 +
 
 +
=== Macaroon usage examples ===
 +
 
 +
List the configured remote root:
 +
 
 +
<pre>
 +
$ rclone lsd pic-pnfs-macaroon:
 +
</pre>
 +
 
 +
List a directory below the configured path:
 +
 
 +
<pre>
 +
$ rclone lsd pic-pnfs-macaroon:PROJECT_OR_DIRECTORY
 +
</pre>
 +
 
 +
List files recursively:
 +
 
 +
<pre>
 +
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY
 +
</pre>
 +
 
 +
Download data:
 +
 
 +
<pre>
 +
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy
 +
</pre>
 +
 
 +
Upload data:
 +
 
 +
<pre>
 +
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY
 +
</pre>
 +
 
 +
Example using a concrete path below the configured WebDAV root:
 +
 
 +
<pre>
 +
$ rclone copy ./ntuples pic-pnfs-macaroon:analysis/ntuples
 +
</pre>
 +
 
 +
== Configure rclone for any other storage location ==
 +
 
 +
Use this section for storage paths that are not under <code>/pnfs/pic.es/data</code>.
 +
 
 +
These locations are accessed using SSH/SFTP.
 +
 
 +
Because SSH authentication may require browser-based authentication, you should configure SSH <code>ControlMaster</code>. This allows rclone to reuse an already-authenticated SSH connection.
 +
 
 +
=== Configure SSH ControlMaster ===
 +
 
 +
On your client machine, edit:
 +
 
 +
<pre>
 +
~/.ssh/config
 +
</pre>
 +
 
 +
Add an entry for the PIC SSH host. For example:
 +
 
 +
<pre>
 +
Host pic-ssh
 +
    HostName ui04.pic.es
 +
    User YOUR_PIC_USERNAME
 +
    ControlMaster auto
 +
    ControlPath ~/.ssh/cm-%r@%h:%p
 +
    ControlPersist yes
 +
</pre>
 +
 
 +
Replace:
 +
 
 +
* <code>ui04.pic.es</code> with the SSH host provided by PIC, if different.
 +
* <code>YOUR_PIC_USERNAME</code> with your PIC username.
 +
 
 +
Instead of restricting this configuration to <code>pic-ssh</code>, you can enable it for all SSH connections:
 +
 
 +
<pre>
 +
Host *
 +
    ControlMaster auto
 +
    ControlPath ~/.ssh/cm-%r@%h:%p
 +
    ControlPersist yes
 +
</pre>
 +
 
 +
However, using a specific host entry is usually clearer and safer.
 +
 
 +
=== Open the first SSH connection ===
 +
 
 +
Before using rclone, open one SSH connection manually:
 +
 
 +
<pre>
 +
$ ssh pic-ssh
 +
</pre>
 +
 
 +
You may see a message similar to:
 +
 
 +
<pre>
 +
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.
 +
</pre>
 +
 
 +
Open the URL in your browser, complete the authentication, then return to the terminal and press <code>ENTER</code>.
 +
 
 +
After this first authentication, subsequent SSH connections to the same host should reuse the existing <code>ControlMaster</code> session.
 +
 
 +
Test this with:
 +
 
 +
<pre>
 +
$ ssh pic-ssh
 +
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205
 +
[tallada@ui04 ~]$
 +
</pre>
 +
 
 +
You can close this shell after testing.
 +
 
 +
=== Configure the SFTP rclone remote ===
 +
 
 +
Other storage locations use the rclone <code>sftp</code> backend.
 +
 
 +
Because we want rclone to reuse the SSH <code>ControlMaster</code> connection, configure the remote to use the external <code>ssh</code> command.
 +
 
 +
First, locate your rclone configuration file:
 +
 
 +
<pre>
 +
$ rclone config file
 +
</pre>
 +
 
 +
The command prints something like:
 +
 
 +
<pre>
 +
Configuration file is stored at:
 +
/home/YOUR_LOCAL_USER/.config/rclone/rclone.conf
 +
</pre>
 +
 
 +
Edit that file and add:
 +
 
 +
<pre>
 +
[pic-storage]
 +
type = sftp
 +
ssh = ssh pic-ssh
 +
shell_type = unix
 +
known_hosts_file = ~/.ssh/known_hosts
 +
</pre>
 +
 
 +
The important line is:
 +
 
 +
<pre>
 +
ssh = ssh pic-ssh
 +
</pre>
 +
 
 +
This tells rclone to use your system SSH command, which can reuse the SSH <code>ControlMaster</code> connection.
 +
 
 +
=== SFTP configuration example ===
 +
 
 +
Example rclone configuration:
 +
 
 +
<pre>
 +
[pic-storage]
 +
type = sftp
 +
ssh = ssh pic-ssh
 +
shell_type = unix
 +
known_hosts_file = ~/.ssh/known_hosts
 +
</pre>
 +
 
 +
=== SFTP usage examples ===
 +
 
 +
List the remote root:
 +
 
 +
<pre>
 +
$ rclone lsd pic-storage:
 +
</pre>
 +
 
 +
List a specific storage path:
 +
 
 +
<pre>
 +
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE
 +
</pre>
 +
 
 +
List files recursively:
 +
 
 +
<pre>
 +
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE
 +
</pre>
 +
 
 +
Download data:
 +
 
 +
<pre>
 +
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy
 +
</pre>
 +
 
 +
Upload data:
 +
 
 +
<pre>
 +
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE
 +
</pre>
 +
 
 +
Example using a concrete absolute path:
 +
 
 +
<pre>
 +
$ rclone copy ./results pic-storage:/storage/projects/MY_PROJECT/results
 +
</pre>
 +
 
 +
== General rclone usage ==
 +
 
 +
The same basic rclone commands can be used with all configured remotes.
 +
 
 +
Replace <code>REMOTE_NAME</code> with one of the remotes you configured, for example:
 +
 
 +
* <code>pic-pnfs-oidc</code>
 +
* <code>pic-pnfs-macaroon</code>
 +
* <code>pic-storage</code>
 +
 
 +
=== List a remote directory ===
 +
 
 +
<pre>
 +
$ rclone lsd REMOTE_NAME:PATH
 +
</pre>
 +
 
 +
Examples:
 +
 
 +
<pre>
 +
$ rclone lsd pic-pnfs-oidc:
 +
$ rclone lsd pic-pnfs-macaroon:
 +
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE
 +
</pre>
 +
 
 +
=== List files recursively ===
 +
 
 +
<pre>
 +
$ rclone ls REMOTE_NAME:PATH
 +
</pre>
 +
 
 +
Examples:
 +
 
 +
<pre>
 +
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY
 +
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY
 +
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE
 +
</pre>
 +
 
 +
=== Download a remote directory ===
 +
 
 +
<pre>
 +
$ rclone copy REMOTE_NAME:PATH LOCAL_PATH
 +
</pre>
 +
 
 +
Examples:
 +
 
 +
<pre>
 +
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy
 +
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy
 +
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy
 +
</pre>
 +
 
 +
=== Upload a local directory ===
 +
 
 +
<pre>
 +
$ rclone copy LOCAL_DIR REMOTE_NAME:PATH
 +
</pre>
 +
 
 +
Examples:
 +
 
 +
<pre>
 +
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY
 +
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY
 +
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE
 +
</pre>
 +
 
 +
=== Recommended upload flags ===
 +
 
 +
For uploads, we recommend:
 +
 
 +
<pre>
 +
--check-first -P --stats-one-line --transfers N_TRANSFERS --size-only
 +
</pre>
 +
 
 +
For many small files, <code>N_TRANSFERS</code> can be up to <code>350</code>.
 +
 
 +
Example:
 +
 
 +
<pre>
 +
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY \
 +
  --check-first -P --stats-one-line --transfers 350 --size-only
 +
</pre>
 +
 
 +
If uploading into directories with many files, for example more than 1000 files, also use:
 +
 
 +
<pre>
 +
--no-traverse
 +
</pre>
 +
 
 +
If uploading files larger than 200 MB, also use:
 +
 
 +
<pre>
 +
--multi-thread-streams 1
 +
</pre>
 +
 
 +
If uploading very large files, for example larger than 10 GB, also use:
 +
 
 +
<pre>
 +
--timeout 15m
 +
</pre>
 +
 
 +
Full example for a large upload to a path under <code>/pnfs/pic.es/data</code>:
 +
 
 +
<pre>
 +
$ rclone copy ./large-dataset pic-pnfs-oidc:PROJECT_OR_DIRECTORY \
 +
  --check-first -P --stats-one-line --transfers 350 --size-only \
 +
  --no-traverse \
 +
  --multi-thread-streams 1 \
 +
  --timeout 15m
 +
</pre>
 +
 
 +
Full example for a large upload to another storage location:
 +
 
 +
<pre>
 +
$ rclone copy ./large-dataset pic-storage:/PATH_TO_YOUR_STORAGE_SPACE \
 +
  --check-first -P --stats-one-line --transfers 350 --size-only \
 +
  --no-traverse \
 +
  --multi-thread-streams 1 \
 +
  --timeout 15m
 +
</pre>
 +
 
 +
For more information, see the rclone documentation:
 +
 
 +
<pre>
 +
https://rclone.org/docs/
 +
</pre>
 +
 
 +
== Configuring oidc-agent for OIDC tokens ==
 +
 
 +
This section is only needed if you use OIDC authentication for paths under <code>/pnfs/pic.es/data</code>.
  
 
Make sure <code>oidc-agent</code> is available.
 
Make sure <code>oidc-agent</code> is available.
  
−
=== Load oidc-agent ===  
+
=== Load oidc-agent ===
 +
 
 +
Initialize <code>oidc-agent</code> in the terminal session:
 +
 
 +
<pre>
 +
$ eval `oidc-agent`
 +
</pre>
 +
 
 +
If the account is already configured, load it with:
 +
 
 +
<pre>
 +
$ oidc-add pic-pnfs
 +
</pre>
 +
 
 +
=== Configure the OIDC account ===
 +
 
 +
This step only needs to be done once.
 +
 
 +
You need an updated version of <code>oidc-agent</code>, version greater than <code>5.0.0</code>.
 +
 
 +
Ask your PIC contact for the client secret and replace <code>XXXXXXXXXXXXXXXXXX</code> below.
 +
 
 +
Configure a <code>pic-pnfs</code> account to retrieve tokens from PIC:
 +
 
 +
<pre>
 +
$ oidc-gen -m --client-id CLIENT_ID_PROVIDED_BY_PIC \
 +
  --client-secret XXXXXXXXXXXXXXXXXX \
 +
  --pub --flow=device \
 +
  --discovery-endpoint=https://idp.pic.es/realms/PIC/.well-known/openid-configuration \
 +
  --scope="openid profile offline_access" \
 +
  --redirect-uri=edu.kit.data.oidc-agent:/ pic-pnfs
 +
</pre>
 +
 
 +
The command will show a URL and a code:
 +
 
 +
<pre>
 +
No account exists with this short name. Creating new configuration ...
 +
Generating account configuration ...
 +
accepted
 +
 
 +
Using a browser on any device, visit:
 +
https://idp.pic.es/realms/PIC/device
 +
 
 +
And enter the code: ASDF-GHJK
 +
Alternatively you can use the following QR code to visit the above listed URL.
 +
 
 +
[ QR CODE ]
 +
 
 +
Enter encryption password for account configuration 'pic-pnfs':
 +
Confirm encryption password:
 +
Everything setup correctly!
 +
</pre>
 +
 
 +
Open the URL in your browser, enter the code, authenticate, then return to the terminal.
 +
 
 +
You will be asked to enter an encryption password twice. You will need this password when refreshing or reloading the <code>oidc-agent</code> account.
 +
 
 +
=== Reauthenticate if the refresh token has expired ===
 +
 
 +
If the <code>oidc-agent</code> process is restarted, or if your refresh token expires due to inactivity, you may need to reauthenticate:
 +
 
 +
<pre>
 +
$ oidc-gen --reauthenticate pic-pnfs
 +
</pre>
 +
 
 +
Example output:
 +
 
 +
<pre>
 +
Enter decryption password for account config 'pic-pnfs':
 +
Generating account configuration ...
 +
accepted
 +
 
 +
Using a browser on any device, visit:
 +
https://idp.pic.es/realms/PIC/device
  
−
Initialize <code>oidc-agent</code> in the terminal session.
+
And enter the code: ASDF-GHJK
 +
Alternatively you can use the following QR code to visit the above listed URL.
  
−
    $ eval `oidc-agent`
+
[ QR CODE ]
  
−
=== Configuration (only once) ===
+
Enter encryption password for account configuration 'pic-pnfs' [***]:
 +
Everything setup correctly!
 +
</pre>
  
−
To run this step you need an updated version of oidc-agent, version > 5.0.0.
+
=== Test OIDC token retrieval ===
−
'''Ask your contact for the client-secret you have to replace below'''
 
  
−
Configure a <code>pic-dcache</code> account to retrieve tokens from PIC. Open the URL that will show and input the code provided (or just open the QR code displayed).
+
After loading and configuring <code>oidc-agent</code>, test that you can retrieve a token:
−
After authenticating on the web browser, return to the terminal and input an encryption password twice. You'll need it when refreshing/reloading the <code>oidc-agent</code>.
 
  
−
    $ oidc-gen -m --client-id dcache-view \
+
<pre>
−
      --client-secret XXXXXXXXXXXXXXXXXX \
+
$ oidc-token pic-pnfs
−
      --pub --flow=device \
+
eyJhbGciOiJSUzI1[...]4YjAwg
−
      --discovery-endpoint=https://idp.pic.es/realms/PIC/.well-known/openid-configuration \
+
</pre>
−
      --scope="openid profile offline_access" --redirect-uri=edu.kit.data.oidc-agent:/ pic-dcache
 
−
   
 
−
    No account exists with this short name. Creating new configuration ...
 
−
    Generating account configuration ...
 
−
    accepted
 
−
   
 
−
    Using a browser on any device, visit:
 
−
    https://idp.pic.es/realms/PIC/device
 
−
   
 
−
    And enter the code: ASDF-GHJK
 
−
    Alternatively you can use the following QR code to visit the above listed URL.
 
−
       
 
−
    [ QR CODE ]
 
−
   
 
−
    Enter encryption password for account configuration 'pic-dcache':
 
−
    Confirm encryption password:
 
−
    Everything setup correctly!
 
  
−
=== Reauthenticating (if refresh token has expired) ===
+
If this command works, the rclone configuration using:
  
−
If the oidc-agent process gets restarted, or iIf your refresh token expires due to inactivity, you will need to reauthenticate to retrieve further tokens
+
<pre>
 +
bearer_token_command = oidc-token pic-pnfs
 +
</pre>
  
−
    $ oidc-gen --reauthenticate pic-dcache
+
should also work.
−
    Enter decryption password for account config 'testtest':
 
−
    Generating account configuration ...
 
−
    accepted
 
−
   
 
−
    Using a browser on any device, visit:
 
−
    https://idp.pic.es/realms/PIC/device
 
−
   
 
−
    And enter the code: ASDF-GHJK
 
−
    Alternatively you can use the following QR code to visit the above listed URL.
 
−
   
 
−
    [ QR CODE ]
 
−
   
 
−
    Enter encryption password for account configuration 'pic-dcache' [***]:
 
−
    Everything setup correctly!
 
  
−
=== Testing ===
+
== Obtaining a Macaroon for /pnfs/pic.es/data ==
  
−
After loading and configuring, you can get a token by running the following command:
+
This section is intended for contacts or administrators who need to generate Macaroons for users.
  
−
    $ oidc-token pic-dcache
+
Macaroons are valid for up to 7 days.
−
      eyJhbGciOiJSUzI1[...]4YjAwg
 
  
−
== Obtaining a macaroon (for contacts) ==
+
The restricted path should be under <code>/pnfs/pic.es/data</code>.
  
−
Macaroons are valid up to 7 days.
+
Example:
  
−
For downloading data (read-only permissions on the path):
+
<pre>
 +
RESTRICTED_PATH=pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE
 +
</pre>
 +
 
 +
=== Read-only Macaroon ===
 +
 
 +
Use this for downloading data only.
 +
 
 +
The Macaroon will allow:
 +
 
 +
* listing directories;
 +
* downloading files.
  
 
<pre>
 
<pre>
 
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \
 
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \
−
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \
+
  -d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \
−
https://door04.pic.es:8460/${RESTRICTED_PATH}
+
  https://door04.pic.es:8460/${RESTRICTED_PATH}
 +
</pre>
 +
 
 +
Example response:
  
 +
<pre>
 
{
 
{
 
     "macaroon": "MDA2MGxvY2F0aW",
 
     "macaroon": "MDA2MGxvY2F0aW",
Line 262: Line 906:
 
</pre>
 
</pre>
  
−
For uploading data (full permissions on the path):
+
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.
 +
 
 +
=== Upload Macaroon ===
 +
 
 +
Use this for uploading data.
 +
 
 +
The Macaroon will allow full permissions on the requested path.
  
 
<pre>
 
<pre>
 
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \
 
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \
−
-d '{"validity": "P7D"}' \
+
  -d '{"validity": "P7D"}' \
−
https://door04.pic.es:8460/${RESTRICTED_PATH}
+
  https://door04.pic.es:8460/${RESTRICTED_PATH}
 +
</pre>
  
 +
Example response:
 +
 +
<pre>
 
{
 
{
 
     "macaroon": "MDA2MGxvY2F0aW",
 
     "macaroon": "MDA2MGxvY2F0aW",
Line 279: Line 933:
 
}
 
}
 
</pre>
 
</pre>
 +
 +
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.
 +
 +
== Troubleshooting ==
 +
 +
=== SSH authentication is requested repeatedly ===
 +
 +
This applies to any storage location accessed through SSH/SFTP.
 +
 +
Check that:
 +
 +
* You have opened the first SSH connection manually with <code>ssh pic-ssh</code>.
 +
* Browser authentication was completed successfully.
 +
* Your <code>~/.ssh/config</code> contains <code>ControlMaster</code>, <code>ControlPath</code>, and <code>ControlPersist</code>.
 +
* Your rclone remote uses the external SSH command:
 +
 +
<pre>
 +
ssh = ssh pic-ssh
 +
</pre>
 +
 +
=== OIDC authentication fails ===
 +
 +
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using OIDC.
 +
 +
Check that <code>oidc-agent</code> is running:
 +
 +
<pre>
 +
$ eval `oidc-agent`
 +
</pre>
 +
 +
Check that the account is loaded:
 +
 +
<pre>
 +
$ oidc-add pic-pnfs
 +
</pre>
 +
 +
Check that you can retrieve a token manually:
 +
 +
<pre>
 +
$ oidc-token pic-pnfs
 +
</pre>
 +
 +
If the token cannot be retrieved, reauthenticate:
 +
 +
<pre>
 +
$ oidc-gen --reauthenticate pic-pnfs
 +
</pre>
 +
 +
=== Macaroon authentication fails ===
 +
 +
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using a Macaroon.
 +
 +
Check that:
 +
 +
* The Macaroon was copied completely.
 +
* The Macaroon has not expired.
 +
* The Macaroon was generated for the correct path under <code>/pnfs/pic.es/data</code>.
 +
* The Macaroon has the required permissions, for example read-only or upload permissions.
 +
 +
If the Macaroon has expired, request or generate a new one.

Revision as of 11:11, 8 June 2026

How to provide data access to PIC massive storage

This page explains how to configure rclone to access PIC storage.

The configuration depends on the storage path you need to access.

Storage access methods
Storage path Protocol rclone backend Supported authentication Example remote name
Paths under /pnfs/pic.es/data WebDAV / HTTPS webdav OIDC token pic-pnfs-oidc
Paths under /pnfs/pic.es/data WebDAV / HTTPS webdav Macaroon bearer token pic-pnfs-macaroon
Any other storage location SSH / SFTP sftp SSH with ControlMaster pic-storage

Requirements

You need:

  • rclone installed on your client machine.
  • The storage path or endpoint provided by your PIC contact.

For paths under /pnfs/pic.es/data, you also need one of the following:

  • an OIDC token setup using oidc-agent; or
  • a Macaroon token.

For any other storage location, you also need:

  • SSH access to the PIC host.
  • SSH ControlMaster configured on your client machine.

Install rclone

You can download the rclone binary directly without installing system packages. For example, on a Linux 64-bit machine: 

$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip
$ unzip rclone-current-linux-amd64.zip

Alternatively, on Ubuntu you can install the Debian package: 

$ cd /tmp
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'
$ sudo apt install ./rclone-current-linux-amd64.deb

Check that rclone works: 

$ rclone version

Which configuration should I use?

Use the table below to choose the right configuration.

Choosing the right configuration
If you want to access... Use this section
A path under /pnfs/pic.es/data using OIDC authentication Configure rclone for PNFS data paths using OIDC
A path under /pnfs/pic.es/data using a Macaroon token Configure rclone for PNFS data paths using a Macaroon
Any other storage location Configure rclone for any other storage location

Configure rclone for PNFS data paths

Paths under /pnfs/pic.es/data are accessed using WebDAV over HTTPS.

For these paths, configure rclone with the webdav backend.

Use one of the following authentication methods:

  • OIDC token, retrieved automatically with oidc-token.
  • Macaroon token, pasted into rclone as a bearer token.

Do not configure username/password authentication unless PIC explicitly instructs you to do so.

Configure rclone for PNFS data paths using OIDC

Use this option if you have been instructed to authenticate with OIDC.

OIDC requirements

Before configuring rclone, make sure that:

  • oidc-agent is available.
  • You have configured an OIDC account, for example pic-pnfs.
  • The following command returns a token:
$ oidc-token pic-pnfs
eyJhbGciOiJSUzI1[...]4YjAwg

If this command does not work, see Configuring oidc-agent for OIDC tokens.

Configure the WebDAV remote using OIDC

Start the rclone configuration wizard: 

$ rclone config

Create a new remote: 

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n

Enter name for new remote.
name> pic-pnfs-oidc

Select WebDAV: 

Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
Storage> webdav

Enter the WebDAV endpoint provided by PIC.

For a path under /pnfs/pic.es/data, the URL will normally point to that path or to a subdirectory under it.

Example: 

Option url.
URL of http host to connect to.
url> https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE

Select vendor other: 

Option vendor.
Name of the WebDAV site/service/software you are using.
vendor> other

Leave username empty: 

Option user.
User name.
user>

Leave password empty: 

Option pass.
Password.
y) Yes, type in my own password
g) Generate random password
n) No, leave this optional password blank
y/g/n> n

When asked for a bearer token, leave it empty: 

Option bearer_token.
Bearer token instead of user/pass, for example a Macaroon.
bearer_token>

Edit the advanced configuration: 

Edit advanced config?
y) Yes
n) No
y/n> y

Set bearer_token_command: 

Option bearer_token_command.
Command to run to get a bearer token.
bearer_token_command> oidc-token pic-pnfs

Accept the remaining advanced options by pressing ENTER, unless PIC instructed you to change them.

At the end, review and accept the configuration: 

Configuration complete.
Options:
- type: webdav
- url: https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE
- vendor: other
- bearer_token_command: oidc-token pic-pnfs

Keep this "pic-pnfs-oidc" remote?
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

Quit the rclone configuration menu: 

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

OIDC configuration example

The resulting rclone configuration should look similar to: 

[pic-pnfs-oidc]
type = webdav
url = https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE
vendor = other
bearer_token_command = oidc-token pic-pnfs

OIDC usage examples

List the configured remote root: 

$ rclone lsd pic-pnfs-oidc:

List a directory below the configured path: 

$ rclone lsd pic-pnfs-oidc:PROJECT_OR_DIRECTORY

List files recursively: 

$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY

Download data: 

$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy

Upload data: 

$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY

Example using a concrete path below the configured WebDAV root: 

$ rclone copy ./ntuples pic-pnfs-oidc:analysis/ntuples

Configure rclone for PNFS data paths using a Macaroon

Use this option if you have been given a Macaroon token.

The Macaroon is used as a WebDAV bearer token.

Configure the WebDAV remote using a Macaroon

Start the rclone configuration wizard: 

$ rclone config

Create a new remote: 

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n

Enter name for new remote.
name> pic-pnfs-macaroon

Select WebDAV: 

Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
Storage> webdav

Enter the WebDAV endpoint provided by PIC.

For a path under /pnfs/pic.es/data, the URL will normally point to that path or to a subdirectory under it.

Example: 

Option url.
URL of http host to connect to.
url> https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE

Select vendor other: 

Option vendor.
Name of the WebDAV site/service/software you are using.
vendor> other

Leave username empty: 

Option user.
User name.
user>

Leave password empty: 

Option pass.
Password.
y) Yes, type in my own password
g) Generate random password
n) No, leave this optional password blank
y/g/n> n

Paste the Macaroon token as the bearer token: 

Option bearer_token.
Bearer token instead of user/pass, for example a Macaroon.
bearer_token> YOUR_MACAROON_TOKEN

You normally do not need to edit the advanced configuration: 

Edit advanced config?
y) Yes
n) No
y/n> n

At the end, review and accept the configuration: 

Configuration complete.
Options:
- type: webdav
- url: https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE
- vendor: other
- bearer_token: *** ENCRYPTED ***

Keep this "pic-pnfs-macaroon" remote?
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

Quit the rclone configuration menu: 

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

Macaroon configuration example

The resulting rclone configuration should look similar to: 

[pic-pnfs-macaroon]
type = webdav
url = https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE
vendor = other
bearer_token = YOUR_MACAROON_TOKEN

Macaroon usage examples

List the configured remote root: 

$ rclone lsd pic-pnfs-macaroon:

List a directory below the configured path: 

$ rclone lsd pic-pnfs-macaroon:PROJECT_OR_DIRECTORY

List files recursively: 

$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY

Download data: 

$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy

Upload data: 

$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY

Example using a concrete path below the configured WebDAV root: 

$ rclone copy ./ntuples pic-pnfs-macaroon:analysis/ntuples

Configure rclone for any other storage location

Use this section for storage paths that are not under /pnfs/pic.es/data.

These locations are accessed using SSH/SFTP.

Because SSH authentication may require browser-based authentication, you should configure SSH ControlMaster. This allows rclone to reuse an already-authenticated SSH connection.

Configure SSH ControlMaster

On your client machine, edit: 

~/.ssh/config

Add an entry for the PIC SSH host. For example: 

Host pic-ssh
    HostName ui04.pic.es
    User YOUR_PIC_USERNAME
    ControlMaster auto
    ControlPath ~/.ssh/cm-%r@%h:%p
    ControlPersist yes

Replace:

  • ui04.pic.es with the SSH host provided by PIC, if different.
  • YOUR_PIC_USERNAME with your PIC username.

Instead of restricting this configuration to pic-ssh, you can enable it for all SSH connections: 

Host *
    ControlMaster auto
    ControlPath ~/.ssh/cm-%r@%h:%p
    ControlPersist yes

However, using a specific host entry is usually clearer and safer.

Open the first SSH connection

Before using rclone, open one SSH connection manually: 

$ ssh pic-ssh

You may see a message similar to: 

(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.

Open the URL in your browser, complete the authentication, then return to the terminal and press ENTER.

After this first authentication, subsequent SSH connections to the same host should reuse the existing ControlMaster session.

Test this with: 

$ ssh pic-ssh
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205
[tallada@ui04 ~]$

You can close this shell after testing.

Configure the SFTP rclone remote

Other storage locations use the rclone sftp backend.

Because we want rclone to reuse the SSH ControlMaster connection, configure the remote to use the external ssh command.

First, locate your rclone configuration file: 

$ rclone config file

The command prints something like: 

Configuration file is stored at:
/home/YOUR_LOCAL_USER/.config/rclone/rclone.conf

Edit that file and add: 

[pic-storage]
type = sftp
ssh = ssh pic-ssh
shell_type = unix
known_hosts_file = ~/.ssh/known_hosts

The important line is: 

ssh = ssh pic-ssh

This tells rclone to use your system SSH command, which can reuse the SSH ControlMaster connection.

SFTP configuration example

Example rclone configuration: 

[pic-storage]
type = sftp
ssh = ssh pic-ssh
shell_type = unix
known_hosts_file = ~/.ssh/known_hosts

SFTP usage examples

List the remote root: 

$ rclone lsd pic-storage:

List a specific storage path: 

$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE

List files recursively: 

$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE

Download data: 

$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy

Upload data: 

$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE

Example using a concrete absolute path: 

$ rclone copy ./results pic-storage:/storage/projects/MY_PROJECT/results

General rclone usage

The same basic rclone commands can be used with all configured remotes.

Replace REMOTE_NAME with one of the remotes you configured, for example:

  • pic-pnfs-oidc
  • pic-pnfs-macaroon
  • pic-storage

List a remote directory

$ rclone lsd REMOTE_NAME:PATH

Examples: 

$ rclone lsd pic-pnfs-oidc:
$ rclone lsd pic-pnfs-macaroon:
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE

List files recursively

$ rclone ls REMOTE_NAME:PATH

Examples: 

$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE

Download a remote directory

$ rclone copy REMOTE_NAME:PATH LOCAL_PATH

Examples: 

$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy

Upload a local directory

$ rclone copy LOCAL_DIR REMOTE_NAME:PATH

Examples: 

$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE

Recommended upload flags

For uploads, we recommend: 

--check-first -P --stats-one-line --transfers N_TRANSFERS --size-only

For many small files, N_TRANSFERS can be up to 350.

Example: 

$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY \
  --check-first -P --stats-one-line --transfers 350 --size-only

If uploading into directories with many files, for example more than 1000 files, also use: 

--no-traverse

If uploading files larger than 200 MB, also use: 

--multi-thread-streams 1

If uploading very large files, for example larger than 10 GB, also use: 

--timeout 15m

Full example for a large upload to a path under /pnfs/pic.es/data: 

$ rclone copy ./large-dataset pic-pnfs-oidc:PROJECT_OR_DIRECTORY \
  --check-first -P --stats-one-line --transfers 350 --size-only \
  --no-traverse \
  --multi-thread-streams 1 \
  --timeout 15m

Full example for a large upload to another storage location: 

$ rclone copy ./large-dataset pic-storage:/PATH_TO_YOUR_STORAGE_SPACE \
  --check-first -P --stats-one-line --transfers 350 --size-only \
  --no-traverse \
  --multi-thread-streams 1 \
  --timeout 15m

For more information, see the rclone documentation: 

https://rclone.org/docs/

Configuring oidc-agent for OIDC tokens

This section is only needed if you use OIDC authentication for paths under /pnfs/pic.es/data.

Make sure oidc-agent is available.

Load oidc-agent

Initialize oidc-agent in the terminal session: 

$ eval `oidc-agent`

If the account is already configured, load it with: 

$ oidc-add pic-pnfs

Configure the OIDC account

This step only needs to be done once.

You need an updated version of oidc-agent, version greater than 5.0.0.

Ask your PIC contact for the client secret and replace XXXXXXXXXXXXXXXXXX below.

Configure a pic-pnfs account to retrieve tokens from PIC: 

$ oidc-gen -m --client-id CLIENT_ID_PROVIDED_BY_PIC \
  --client-secret XXXXXXXXXXXXXXXXXX \
  --pub --flow=device \
  --discovery-endpoint=https://idp.pic.es/realms/PIC/.well-known/openid-configuration \
  --scope="openid profile offline_access" \
  --redirect-uri=edu.kit.data.oidc-agent:/ pic-pnfs

The command will show a URL and a code: 

No account exists with this short name. Creating new configuration ...
Generating account configuration ...
accepted

Using a browser on any device, visit:
https://idp.pic.es/realms/PIC/device

And enter the code: ASDF-GHJK
Alternatively you can use the following QR code to visit the above listed URL.

[ QR CODE ]

Enter encryption password for account configuration 'pic-pnfs':
Confirm encryption password:
Everything setup correctly!

Open the URL in your browser, enter the code, authenticate, then return to the terminal.

You will be asked to enter an encryption password twice. You will need this password when refreshing or reloading the oidc-agent account.

Reauthenticate if the refresh token has expired

If the oidc-agent process is restarted, or if your refresh token expires due to inactivity, you may need to reauthenticate: 

$ oidc-gen --reauthenticate pic-pnfs

Example output: 

Enter decryption password for account config 'pic-pnfs':
Generating account configuration ...
accepted

Using a browser on any device, visit:
https://idp.pic.es/realms/PIC/device

And enter the code: ASDF-GHJK
Alternatively you can use the following QR code to visit the above listed URL.

[ QR CODE ]

Enter encryption password for account configuration 'pic-pnfs' [***]:
Everything setup correctly!

Test OIDC token retrieval

After loading and configuring oidc-agent, test that you can retrieve a token: 

$ oidc-token pic-pnfs
eyJhbGciOiJSUzI1[...]4YjAwg

If this command works, the rclone configuration using: 

bearer_token_command = oidc-token pic-pnfs

should also work.

Obtaining a Macaroon for /pnfs/pic.es/data

This section is intended for contacts or administrators who need to generate Macaroons for users.

Macaroons are valid for up to 7 days.

The restricted path should be under /pnfs/pic.es/data.

Example: 

RESTRICTED_PATH=pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE

Read-only Macaroon

Use this for downloading data only.

The Macaroon will allow:

  • listing directories;
  • downloading files.
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \
  -d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \
  https://door04.pic.es:8460/${RESTRICTED_PATH}

Example response: 

{
    "macaroon": "MDA2MGxvY2F0aW",
    "uri": {
        "targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",
        "baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",
        "target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",
        "base": "https://door04.pic.es:8460/"
    }
}

Give the value of the macaroon field to the user. The user should paste it into rclone as the WebDAV bearer_token.

Upload Macaroon

Use this for uploading data.

The Macaroon will allow full permissions on the requested path. 

$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \
  -d '{"validity": "P7D"}' \
  https://door04.pic.es:8460/${RESTRICTED_PATH}

Example response: 

{
    "macaroon": "MDA2MGxvY2F0aW",
    "uri": {
        "targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",
        "baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",
        "target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",
        "base": "https://door04.pic.es:8460/"
    }
}

Give the value of the macaroon field to the user. The user should paste it into rclone as the WebDAV bearer_token.

Troubleshooting

SSH authentication is requested repeatedly

This applies to any storage location accessed through SSH/SFTP.

Check that:

  • You have opened the first SSH connection manually with ssh pic-ssh.
  • Browser authentication was completed successfully.
  • Your ~/.ssh/config contains ControlMaster, ControlPath, and ControlPersist.
  • Your rclone remote uses the external SSH command:
ssh = ssh pic-ssh

OIDC authentication fails

This applies to paths under /pnfs/pic.es/data accessed using OIDC.

Check that oidc-agent is running: 

$ eval `oidc-agent`

Check that the account is loaded: 

$ oidc-add pic-pnfs

Check that you can retrieve a token manually: 

$ oidc-token pic-pnfs

If the token cannot be retrieved, reauthenticate: 

$ oidc-gen --reauthenticate pic-pnfs

Macaroon authentication fails

This applies to paths under /pnfs/pic.es/data accessed using a Macaroon.

Check that:

  • The Macaroon was copied completely.
  • The Macaroon has not expired.
  • The Macaroon was generated for the correct path under /pnfs/pic.es/data.
  • The Macaroon has the required permissions, for example read-only or upload permissions.

If the Macaroon has expired, request or generate a new one.

Retrieved from "http://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&oldid=1359"