Transferring data to/from PIC

How to provide data access to PIC massive storage (dCache)

Requirements

* Install and configure Rclone
* PIC credentials or macaroon

Install Rclone

You can directly download the binary without installing anything. For instance, for a linux 64 bits machine:

   $ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip
   [...]
   $ unzip rclone-current-linux-amd64.zip

Or if you prefer, you can install Rclone like the next example on a Ubuntu machine:

   $ cd /tmp
   $ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'
   $ sudo apt install ./rclone-current-linux-amd64.deb

Configure Rclone

You should have been given some credentials and the url of a WebDAV endpoint at PIC. With them, you just need to create the config in rclone:

   $ rclone config
   No remotes found, make a new one?
   n) New remote
   s) Set configuration password
   q) Quit config
   n/s/q> n
   
   Enter name for new remote.
   name> pic
   
   Option Storage.
   Type of storage to configure.
   Choose a number from below, or type in your own value.
    1 / 1Fichier
      \ (fichier)
   [...]
   Storage> webdav
   
   Option url.
   URL of http host to connect to.
   E.g. https://example.com.
   Enter a value.
   url>  https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE
   
   Option vendor.
   Name of the WebDAV site/service/software you are using.
   Choose a number from below, or type in your own value.
   Press Enter to leave empty.
    1 / Nextcloud
      \ (nextcloud)
   [...]
    5 / Other site/service or software
      \ (other)
   vendor> other
   

Using your PIC credentials

If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank.

   Option user.
   User name.
   In case NTLM authentication is used, the username should be in the format 'Domain\User'.
   Enter a value. Press Enter to leave empty.
   user> YOUR_PIC_USERNAME

   Option pass.
   Password.
   Choose an alternative below. Press Enter for the default (n).
   y) Yes, type in my own password
   g) Generate random password
   n) No, leave this optional password blank (default)
   y/g/n> y
   Enter the password:
   password: YOUR_PIC_PASSWORD
   Confirm the password:
   Password: YOUR_PIC_PASSWORD

Using a Macaroon token

If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank

   Option bearer_token.
   Bearer token instead of user/pass (e.g. a Macaroon).
   Enter a value. Press Enter to leave empty.
   bearer_token> YOUR_MACAROON_TOKEN

Using an OIDC token

   Option bearer_token.
   Bearer token instead of user/pass (e.g. a Macaroon).
   Enter a value. Press Enter to leave empty.
   bearer_token> 
   
   Edit advanced config?
   y) Yes
   n) No (default)
   y/n> y
   
   Option bearer_token_command.
   Command to run to get a bearer token.
   Enter a value. Press Enter to leave empty.
   bearer_token_command> oidc-token OIDC_AGENT_ACCOUNT_SHORTNAME

Review settings

At the end, just review the information you entered and confirm.

   Edit advanced config?
   y) Yes
   n) No (default)
   y/n> n
   
   Configuration complete.
   Options:
   - type: webdav
   - url: https://door04.pic.es/PATH_TO_YOUR_STORAGE_SPACE
   - vendor: other
   - user: YOUR_PIC_USERNAME
   - pass: *** ENCRYPTED ***
   Keep this "pic" remote?
   y) Yes this is OK (default)
   e) Edit this remote
   d) Delete this remote
   y/e/d> y
   
   Current remotes:
   
   Name                 Type
   ====                 ====
   pic                  webdav
   
   e) Edit existing remote
   n) New remote
   d) Delete remote
   r) Rename remote
   c) Copy remote
   s) Set configuration password
   q) Quit config
   e/n/d/r/c/s/q> q

Once done, you can use command line to browse and download/upload data.

Usage

• List a remote PIC directory:

 rclone lsd <name>:<path>`

• Download a remote directory from PIC

 rclone copy <name>:<path> <local_path>

• Upload a local directory to PIC

 rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path>

When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files

 --check-first -P --stats-one-line --transfers <n_transfers> --size-only

If uploading lots onto directories with lots of files (>1000), please use:

 --no-traverse

If uploading files larger than 200 MB, also use

  --multi-thread-streams 1

If uploading very large files (>10G), also use the following to allow more time to compute the checksums

 --timeout=15m

See rclone manual for more extensive documentation https://rclone.org/docs/

Configuring oidc-agent for obtaining OIDC tokens

Make sure oidc-agent is available.

Load oidc-agent

Initialize oidc-agent in the terminal session.

   $ eval `oidc-agent`

Configuration (only once)

To run this step you need an updated version of oidc-agent, version > 5.0.0. Ask your contact for the client-secret you have to replace below

Configure a pic-dcache account to retrieve tokens from PIC. Open the URL that will show and input the code provided (or just open the QR code displayed). After authenticating on the web browser, return to the terminal and input an encryption password twice. You'll need it when refreshing/reloading the oidc-agent.

   $ oidc-gen -m --client-id dcache-view \
     --client-secret XXXXXXXXXXXXXXXXXX \
     --pub --flow=device \
     --discovery-endpoint=https://idp.pic.es/realms/PIC/.well-known/openid-configuration \
     --scope="openid profile offline_access" --redirect-uri=edu.kit.data.oidc-agent:/ pic-dcache
   
   No account exists with this short name. Creating new configuration ...
   Generating account configuration ...
   accepted
   
   Using a browser on any device, visit:
   https://idp.pic.es/realms/PIC/device
   
   And enter the code: ASDF-GHJK
   Alternatively you can use the following QR code to visit the above listed URL.
       
   [ QR CODE ]
   
   Enter encryption password for account configuration 'pic-dcache': 
   Confirm encryption password: 
   Everything setup correctly!

Reauthenticating (if refresh token has expired)

If the oidc-agent process gets restarted, or iIf your refresh token expires due to inactivity, you will need to reauthenticate to retrieve further tokens

   $ oidc-gen --reauthenticate pic-dcache
   Enter decryption password for account config 'testtest': 
   Generating account configuration ...
   accepted
   
   Using a browser on any device, visit:
   https://idp.pic.es/realms/PIC/device
   
   And enter the code: ASDF-GHJK
   Alternatively you can use the following QR code to visit the above listed URL.
   
   [ QR CODE ]
   
   Enter encryption password for account configuration 'pic-dcache' [***]: 
   Everything setup correctly!

Testing

After loading and configuring, you can get a token by running the following command:

   $ oidc-token pic-dcache
     eyJhbGciOiJSUzI1[...]4YjAwg

Obtaining a macaroon (for contacts)

Macaroons are valid up to 7 days.

For downloading data (read-only permissions on the path):

$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \
https://door04.pic.es:8460/${RESTRICTED_PATH}

{
    "macaroon": "MDA2MGxvY2F0aW",
    "uri": {
        "targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",
        "baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",
        "target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",
        "base": "https://door04.pic.es:8460/"
    }
}

For uploading data (full permissions on the path):

$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \
-d '{"validity": "P7D"}' \
https://door04.pic.es:8460/${RESTRICTED_PATH}

{
    "macaroon": "MDA2MGxvY2F0aW",
    "uri": {
        "targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",
        "baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",
        "target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",
        "base": "https://door04.pic.es:8460/"
    }
}