View source for Transferring data to/from PIC

= How to provide data access to PIC massive storage (dCache) =

== Requirements ==

 * Install and configure Rclone
 * PIC credentials or macaroon

== Install Rclone ==

You can directly download the binary without installing anything. For instance, for a linux 64 bits machine:

    $ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip
    [...]
    $ unzip rclone-current-linux-amd64.zip


Or if you prefer, you can install Rclone like the next example on a Ubuntu machine:

    $ cd /tmp
    $ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'
    $ sudo apt install ./rclone-current-linux-amd64.deb

== Configure Rclone ==

You should have been given some credentials and the url of a WebDAV endpoint at PIC.
With them, you just need to create the config in rclone:

    $ rclone config
    No remotes found, make a new one?
    n) New remote
    s) Set configuration password
    q) Quit config
    n/s/q> n
    
    Enter name for new remote.
    name> pic
    
    Option Storage.
    Type of storage to configure.
    Choose a number from below, or type in your own value.
     1 / 1Fichier
       \ (fichier)
    [...]
    Storage> webdav
    
    Option url.
    URL of http host to connect to.
    E.g. https://example.com.
    Enter a value.
    url>  https://door04.pic.es/PATH_TO_YOUR_STORAGE_SPACE
    
    Option vendor.
    Name of the WebDAV site/service/software you are using.
    Choose a number from below, or type in your own value.
    Press Enter to leave empty.
     1 / Nextcloud
       \ (nextcloud)
    [...]
     5 / Other site/service or software
       \ (other)
    vendor> other
    
===  Using your PIC credentials ===

If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank.

    Option user.
    User name.
    In case NTLM authentication is used, the username should be in the format 'Domain\User'.
    Enter a value. Press Enter to leave empty.
    user> YOUR_PIC_USERNAME

    Option pass.
    Password.
    Choose an alternative below. Press Enter for the default (n).
    y) Yes, type in my own password
    g) Generate random password
    n) No, leave this optional password blank (default)
    y/g/n> y
    Enter the password:
    password: YOUR_PIC_PASSWORD
    Confirm the password:
    Password: YOUR_PIC_PASSWORD

=== Using a Macaroon token ===

If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank

    Option bearer_token.
    Bearer token instead of user/pass (e.g. a Macaroon).
    Enter a value. Press Enter to leave empty.
    bearer_token> YOUR_MACAROON_TOKEN

=== Review settings ===

At the end, just review the information you entered and confirm.

    Edit advanced config?
    y) Yes
    n) No (default)
    y/n> n
    
    Configuration complete.
    Options:
    - type: webdav
    - url: https://door04.pic.es/PATH_TO_YOUR_STORAGE_SPACE
    - vendor: other
    - user: YOUR_PIC_USERNAME
    - pass: *** ENCRYPTED ***
    Keep this "pic" remote?
    y) Yes this is OK (default)
    e) Edit this remote
    d) Delete this remote
    y/e/d> y
    
    Current remotes:
    
    Name                 Type
    ====                 ====
    pic                  webdav
    
    e) Edit existing remote
    n) New remote
    d) Delete remote
    r) Rename remote
    c) Copy remote
    s) Set configuration password
    q) Quit config
    e/n/d/r/c/s/q> q

Once done, you can use command line to browse and download/upload data.

=== Usage ===

* List a remote PIC directory: 
  rclone lsd <name>:<path>`

* Download a remote directory from PIC
  rclone copy <name>:<path> <local_path>

* Upload a local directory to PIC
  rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path>

When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files 
  --check-first -P --stats-one-line --transfers <n_transfers> --size-only

See rclone manual for more extensive documentation https://rclone.org/docs/

== Obtaining a macaroon (for contacts) ==

Macaroons are valid up to 7 days.

For downloading data (read-only permissions on the path):

<pre>
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \
https://door04.pic.es:8460/${RESTRICTED_PATH}

{
    "macaroon": "MDA2MGxvY2F0aW",
    "uri": {
        "targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",
        "baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",
        "target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",
        "base": "https://door04.pic.es:8460/"
    }
}
</pre>

For uploading data (full permissions on the path):

<pre>
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \
-d '{"validity": "P7D"}' \
https://door04.pic.es:8460/${RESTRICTED_PATH}

{
    "macaroon": "MDA2MGxvY2F0aW",
    "uri": {
        "targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",
        "baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",
        "target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",
        "base": "https://door04.pic.es:8460/"
    }
}
</pre>