Transferring data to/from PIC

How to provide data access to external users without PIC account

I will use an example for MICE

Steps

1) dCache webdav door that exports your data.

https://webdav-mice.pic.es:8454

2) dCacheView frontend (optional)

https://webdav-mice.pic.es:8464

3) Authentication

Either PIC credentials (user/password) or a macaroon (https://dcache.org/old/manuals/UserGuide-7.0/macaroons.shtml). Any authenticated user can get a macaroon for an external user. Remember to add proper caveats to restrict privileges. In the next example, I retrieve a macaroon with 7 days validity, that can only download and list, and can only access the subtree of the provided URL.

$ curl -u tallada -X POST -H 'Content-Type: application/macaroon-request' \
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \
https://webdav-mice.pic.es:8454/tape/raw/production/N4096_L3072_LC3/maps/n8192/kmapgcpy/

{
    "macaroon": "MDANOxGCg",
    "uri": {
        "targetWithMacaroon": "https://webdav-mice.pic.es:8454/tape/raw/production/N4096_L3072_LC3/maps/n8192/kmapgcpy/?authz=MDANOxGCg",
        "baseWithMacaroon": "https://webdav-mice.pic.es:8454/?authz=MDANOxGCg",
        "target": "https://webdav-mice.pic.es:8454/tape/raw/production/N4096_L3072_LC3/maps/n8192/kmapgcpy/",
        "base": "https://webdav-mice.pic.es:8454/"
    }
}

4) Browser access

You can access the data either through the webdav door or dCacheView. If you have PIC credentials (user/password), just input them when requested by the browser.

If you only have a macaroon token, you can access the webdav using the preauthenticated URL (either targetWithMacaroon or baseWithMacaroon). You will only be able to browse to the restricted subtree. You can also use command line clients like cURL or wget.

In order to use the macarron with dCacheView, DO NOT ENTER any credentials at the login step. Instead, click the three bar menu at the top right, a panel on the left will open. On that panel, select the "Shared files" option. Then, clic the "Add" button on the bottom right and enter the macaroon token.

5) rclone

We can use rclone on Windows, Linux and MacOS to be able to automatize data transfer, even mount the remote data as a local filesystem. First, follow the instructions to install rclone in your host (https://rclone.org/install/). Note that for mounting you might need additional software (https://rclone.org/commands/rclone_mount/)

Then, open the web GUI with the command "rclone rcd --rc-web-gui". You might need to add "--no-check-certificates" in some cases. Once opened, create a config with the following info:

• provider: webdav
• url: https://webdav-mice.pic.es:8454 (either the base or target url)
• username/password: if you have PIC credentials
• bearer token: MDANOxGCg (the macaroon token)

Once done, you can use the explorer tab to browse and download data.

You can also mount the remote data to a local path. Use the "Mounts" tab, select the remote filesystem and a local folder (or diver letter in Windows), and press create.

Enjoy!