Transferring data to/from PIC

How to provide data access to PIC massive storage (dCache)

Requirements

* Install and configure Rclone
* PIC credentials or macaroon

Install Rclone

You can directly download the binary without installing anything. For instance, for a linux 64 bits machine:

   $ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip
   [...]
   $ unzip rclone-current-linux-amd64.zip

Or if you prefer, you can install Rclone like the next example on a Ubuntu machine:

   $ cd /tmp
   $ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'
   $ sudo apt install ./rclone-current-linux-amd64.deb

Configure Rclone

You should have been given some credentials and the url of a WebDAV endpoint at PIC. With them, you just need to create the config in rclone:

   $ rclone config
   No remotes found, make a new one?
   n) New remote
   s) Set configuration password
   q) Quit config
   n/s/q> n
   
   Enter name for new remote.
   name> pic
   
   Option Storage.
   Type of storage to configure.
   Choose a number from below, or type in your own value.
    1 / 1Fichier
      \ (fichier)
   [...]
   Storage> webdav
   
   Option url.
   URL of http host to connect to.
   E.g. https://example.com.
   Enter a value.
   url>  https://door04.pic.es/PATH_TO_YOUR_STORAGE_SPACE
   
   Option vendor.
   Name of the WebDAV site/service/software you are using.
   Choose a number from below, or type in your own value.
   Press Enter to leave empty.
    1 / Nextcloud
      \ (nextcloud)
   [...]
    5 / Other site/service or software
      \ (other)
   vendor> other
   

Using your PIC credentials

If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank.

   Option user.
   User name.
   In case NTLM authentication is used, the username should be in the format 'Domain\User'.
   Enter a value. Press Enter to leave empty.
   user> YOUR_PIC_USERNAME

   Option pass.
   Password.
   Choose an alternative below. Press Enter for the default (n).
   y) Yes, type in my own password
   g) Generate random password
   n) No, leave this optional password blank (default)
   y/g/n> y
   Enter the password:
   password: YOUR_PIC_PASSWORD
   Confirm the password:
   Password: YOUR_PIC_PASSWORD

Using a Macaroon token

If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank

   Option bearer_token.
   Bearer token instead of user/pass (e.g. a Macaroon).
   Enter a value. Press Enter to leave empty.
   bearer_token> YOUR_MACAROON_TOKEN

Review settings

At the end, just review the information you entered and confirm.

   Edit advanced config?
   y) Yes
   n) No (default)
   y/n> n
   
   Configuration complete.
   Options:
   - type: webdav
   - url: https://door04.pic.es/PATH_TO_YOUR_STORAGE_SPACE
   - vendor: other
   - user: YOUR_PIC_USERNAME
   - pass: *** ENCRYPTED ***
   Keep this "pic" remote?
   y) Yes this is OK (default)
   e) Edit this remote
   d) Delete this remote
   y/e/d> y
   
   Current remotes:
   
   Name                 Type
   ====                 ====
   pic                  webdav
   
   e) Edit existing remote
   n) New remote
   d) Delete remote
   r) Rename remote
   c) Copy remote
   s) Set configuration password
   q) Quit config
   e/n/d/r/c/s/q> q

Once done, you can use command line to browse and download/upload data.

Usage

• List a remote PIC directory:

 rclone lsd <name>:<path>`

• Download a remote directory from PIC

 rclone copy <name>:<path> <local_path>

• Upload a local directory to PIC

 rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path>

When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files

 --check-first -P --stats-one-line --transfers <n_transfers> --size-only

See rclone manual for more extensive documentation https://rclone.org/docs/

Obtaining a macaroon (for contacts)

Macaroons are valid up to 7 days.

For downloading data (read-only permissions on the path):

$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \
https://door04.pic.es:8460/${RESTRICTED_PATH}

{
    "macaroon": "MDA2MGxvY2F0aW",
    "uri": {
        "targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",
        "baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",
        "target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",
        "base": "https://door04.pic.es:8460/"
    }
}

For uploading data (full permissions on the path):

$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \
-d '{"validity": "P7D"}' \
https://door04.pic.es:8460/${RESTRICTED_PATH}

{
    "macaroon": "MDA2MGxvY2F0aW",
    "uri": {
        "targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",
        "baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",
        "target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",
        "base": "https://door04.pic.es:8460/"
    }
}