Transferring data to/from PIC

From Public PIC Wiki
Revision as of 11:23, 6 December 2023 by Tallada (talk | contribs)
Jump to navigation Jump to search

How to provide data access to external users without PIC account

dCache webdav door

https://webdav.pic.es or https://webdav.pic.es:8460

dCacheView frontend (optional)

https://webdav-mice.pic.es:8470

Authentication

Either PIC credentials (user/password) or a macaroon (https://dcache.org/old/manuals/UserGuide-7.0/macaroons.shtml). Any authenticated user can get a macaroon for an external user. Remember to add proper caveats to restrict privileges. In the next example, I retrieve a macaroon with 7 days validity, that can only download and list, and can only access the subtree of the provided URL. 

$ curl -u tallada -X POST -H 'Content-Type: application/macaroon-request' \
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \
https://webdav.pic.es:8460/pnfs/pic.es/data/astro/euclid/disk/simulations/ComplementarySimulations

{
    "macaroon": "MDA2MGxvY2F0aW",
    "uri": {
        "targetWithMacaroon": "https://webdav.pic.es:8460/pnfs/pic.es/data/astro/euclid/disk/simulations/ComplementarySimulations?authz=MDA2MGxvY2F0aW",
        "baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW",
        "target": "https://webdav.pic.es:8460/pnfs/pic.es/data/astro/euclid/disk/simulations/ComplementarySimulations",
        "base": "https://webdav.pic.es:8460/"
    }
}

Browser access

You can access the data either through the webdav door or dCacheView. If you have PIC credentials (user/password), just input them when requested by the browser.

If you only have a macaroon token, you can access the webdav using the preauthenticated URL (either targetWithMacaroon or baseWithMacaroon). You will only be able to browse to the restricted subtree. You can also use command line clients like cURL or wget.

In order to use the macarron with dCacheView, DO NOT ENTER any credentials at the login step. Instead, click the three bar menu at the top right, a panel on the left will open. On that panel, select the "Shared files" option. Then, clic the "Add" button on the bottom right and enter the macaroon token.

rclone

We can use rclone on Windows, Linux and MacOS to be able to automatize data transfer, even mount the remote data as a local filesystem. First, follow the instructions to install rclone in your host (https://rclone.org/install/). Note that for mounting you might need additional software (https://rclone.org/commands/rclone_mount/)

Then, create a new remote, either using the "rclone config" command line, or opening the web GUI with the command "rclone rcd --rc-web-gui". You might need to add "--no-check-certificates" in some cases. Provide the following info for the remote:

  • name: "pic" or any other short identifier
  • provider: webdav
  • url: https://webdav.pic.es (you can use the full path here to to restrict the visible tree)
  • username/password: if you have PIC credentials, otherwise leave empty and use macaroon below.
  • bearer token: MDANOxGCg (the macaroon token)

Once done, you can use either the command line or the explorer tab to browse and download data.

Most common command lines:

  • List a remote PIC directory:
 rclone lsd <name>:<path>`
  • Copy a local directory to PIC
 rclone copy <local_dir> <name>:<path>

I recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files 

 rclone --check-first -P --stats-one-line --transfers <n_transfers> --size-only copy <local_dir> <name>:<path>

You can also mount the remote data to a local path. Use the "Mounts" tab in the web GUI, select the remote filesystem and a local folder (or drive letter in Windows), and press create.

Enjoy!

Retrieved from "http://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&oldid=1095"