Difference between revisions of "Transferring data to/from PIC"
Line 1: | Line 1: | ||
− | + | = How to provide data access to PIC massive storage (dCache) == | |
+ | == Requirements == | ||
− | + | * Install and configure Rclone | |
+ | * PIC credentials or macaroon | ||
− | + | == Install Rclone == | |
− | + | You can directly download the binary without installing anything. For instance, for a linux 64 bits machine: | |
− | https:// | + | $ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip |
+ | [...] | ||
+ | $ unzip rclone-current-linux-amd64.zip | ||
− | |||
− | + | Or if you prefer, you can install Rclone like the next example on a Ubuntu machine: | |
− | |||
− | |||
− | + | $ cd /tmp | |
− | $ curl - | + | $ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb' |
− | - | + | $ sudo apt install ./rclone-current-linux-amd64.deb |
− | https:// | + | |
+ | == Configure Rclone == | ||
+ | |||
+ | You should have been given some credentials and the url of a WebDAV endpoint at PIC. | ||
+ | With them, you just need to create the config in rclone: | ||
+ | |||
+ | $ rclone config | ||
+ | No remotes found, make a new one? | ||
+ | n) New remote | ||
+ | s) Set configuration password | ||
+ | q) Quit config | ||
+ | n/s/q> n | ||
+ | |||
+ | Enter name for new remote. | ||
+ | name> pic | ||
+ | |||
+ | Option Storage. | ||
+ | Type of storage to configure. | ||
+ | Choose a number from below, or type in your own value. | ||
+ | 1 / 1Fichier | ||
+ | \ (fichier) | ||
+ | [...] | ||
+ | Storage> webdav | ||
+ | |||
+ | Option url. | ||
+ | URL of http host to connect to. | ||
+ | E.g. https://example.com. | ||
+ | Enter a value. | ||
+ | url> https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE | ||
+ | |||
+ | Option vendor. | ||
+ | Name of the WebDAV site/service/software you are using. | ||
+ | Choose a number from below, or type in your own value. | ||
+ | Press Enter to leave empty. | ||
+ | 1 / Nextcloud | ||
+ | \ (nextcloud) | ||
+ | [...] | ||
+ | 5 / Other site/service or software | ||
+ | \ (other) | ||
+ | vendor> other | ||
+ | |||
+ | === Using your PIC credentials === | ||
+ | |||
+ | If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank. | ||
− | + | Option user. | |
− | + | User name. | |
− | + | In case NTLM authentication is used, the username should be in the format 'Domain\User'. | |
− | + | Enter a value. Press Enter to leave empty. | |
− | + | user> YOUR_PIC_USERNAME | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | + | Option pass. | |
+ | Password. | ||
+ | Choose an alternative below. Press Enter for the default (n). | ||
+ | y) Yes, type in my own password | ||
+ | g) Generate random password | ||
+ | n) No, leave this optional password blank (default) | ||
+ | y/g/n> y | ||
+ | Enter the password: | ||
+ | password: YOUR_PIC_PASSWORD | ||
+ | Confirm the password: | ||
+ | Password: YOUR_PIC_PASSWORD | ||
− | + | === Using a Macaroon token === | |
− | If you | + | If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank |
− | |||
− | + | Option bearer_token. | |
+ | Bearer token instead of user/pass (e.g. a Macaroon). | ||
+ | Enter a value. Press Enter to leave empty. | ||
+ | bearer_token> YOUR_MACAROON_TOKEN | ||
− | === | + | === Review settings === |
− | + | At the end, just review the information you entered and confirm. | |
− | |||
− | |||
− | + | Edit advanced config? | |
− | + | y) Yes | |
− | + | n) No (default) | |
− | + | y/n> n | |
− | + | ||
− | + | Configuration complete. | |
− | + | Options: | |
+ | - type: webdav | ||
+ | - url: https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE | ||
+ | - vendor: other | ||
+ | - user: YOUR_PIC_USERNAME | ||
+ | - pass: *** ENCRYPTED *** | ||
+ | Keep this "pic" remote? | ||
+ | y) Yes this is OK (default) | ||
+ | e) Edit this remote | ||
+ | d) Delete this remote | ||
+ | y/e/d> y | ||
+ | |||
+ | Current remotes: | ||
+ | |||
+ | Name Type | ||
+ | ==== ==== | ||
+ | pic webdav | ||
+ | |||
+ | e) Edit existing remote | ||
+ | n) New remote | ||
+ | d) Delete remote | ||
+ | r) Rename remote | ||
+ | c) Copy remote | ||
+ | s) Set configuration password | ||
+ | q) Quit config | ||
+ | e/n/d/r/c/s/q> q | ||
− | Once done, you can use | + | Once done, you can use command line to browse and download/upload data. |
− | + | === Usage === | |
* List a remote PIC directory: | * List a remote PIC directory: | ||
rclone lsd <name>:<path>` | rclone lsd <name>:<path>` | ||
− | * | + | * Download a remote directory from PIC |
− | rclone copy <local_dir> <name>:<path> | + | rclone copy <name>:<path> <local_path> |
+ | |||
+ | * Upload a local directory to PIC | ||
+ | rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path> | ||
+ | |||
+ | When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files | ||
+ | --check-first -P --stats-one-line --transfers <n_transfers> --size-only | ||
+ | |||
+ | See rclone manual for more extensive documentation https://rclone.org/docs/ | ||
− | + | == Obtaining a macaroon (for contacts) == | |
− | |||
− | + | Macaroons are valid up to 7 days. | |
− | + | For downloading data (read-only permissions on the path): | |
+ | |||
+ | <pre> | ||
+ | $ curl -u ${USERNAME} -X POST -H 'Content-Type: application/macaroon-request' \ | ||
+ | -d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \ | ||
+ | https://webdav.pic.es:8460/${RESTRICTED_PATH} | ||
+ | |||
+ | { | ||
+ | "macaroon": "MDA2MGxvY2F0aW", | ||
+ | "uri": { | ||
+ | "targetWithMacaroon": "https://webdav.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW", | ||
+ | "baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW", | ||
+ | "target": "https://webdav.pic.es:8460/${RESTRICTED_PATH}", | ||
+ | "base": "https://webdav.pic.es:8460/" | ||
+ | } | ||
+ | } | ||
+ | </pre> | ||
+ | |||
+ | For uploading data (full permissions on the path): | ||
+ | |||
+ | <pre> | ||
+ | $ curl -u ${USERNAME} -X POST -H 'Content-Type: application/macaroon-request' \ | ||
+ | -d '{"validity": "P7D"}' \ | ||
+ | https://webdav.pic.es:8460/${RESTRICTED_PATH} | ||
+ | |||
+ | { | ||
+ | "macaroon": "MDA2MGxvY2F0aW", | ||
+ | "uri": { | ||
+ | "targetWithMacaroon": "https://webdav.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW", | ||
+ | "baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW", | ||
+ | "target": "https://webdav.pic.es:8460/${RESTRICTED_PATH}", | ||
+ | "base": "https://webdav.pic.es:8460/" | ||
+ | } | ||
+ | } | ||
+ | </pre> |
Revision as of 12:59, 15 December 2023
How to provide data access to PIC massive storage (dCache) =
Requirements
* Install and configure Rclone * PIC credentials or macaroon
Install Rclone
You can directly download the binary without installing anything. For instance, for a linux 64 bits machine:
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip [...] $ unzip rclone-current-linux-amd64.zip
Or if you prefer, you can install Rclone like the next example on a Ubuntu machine:
$ cd /tmp $ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb' $ sudo apt install ./rclone-current-linux-amd64.deb
Configure Rclone
You should have been given some credentials and the url of a WebDAV endpoint at PIC. With them, you just need to create the config in rclone:
$ rclone config No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n Enter name for new remote. name> pic Option Storage. Type of storage to configure. Choose a number from below, or type in your own value. 1 / 1Fichier \ (fichier) [...] Storage> webdav Option url. URL of http host to connect to. E.g. https://example.com. Enter a value. url> https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE Option vendor. Name of the WebDAV site/service/software you are using. Choose a number from below, or type in your own value. Press Enter to leave empty. 1 / Nextcloud \ (nextcloud) [...] 5 / Other site/service or software \ (other) vendor> other
Using your PIC credentials
If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank.
Option user. User name. In case NTLM authentication is used, the username should be in the format 'Domain\User'. Enter a value. Press Enter to leave empty. user> YOUR_PIC_USERNAME
Option pass. Password. Choose an alternative below. Press Enter for the default (n). y) Yes, type in my own password g) Generate random password n) No, leave this optional password blank (default) y/g/n> y Enter the password: password: YOUR_PIC_PASSWORD Confirm the password: Password: YOUR_PIC_PASSWORD
Using a Macaroon token
If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank
Option bearer_token. Bearer token instead of user/pass (e.g. a Macaroon). Enter a value. Press Enter to leave empty. bearer_token> YOUR_MACAROON_TOKEN
Review settings
At the end, just review the information you entered and confirm.
Edit advanced config? y) Yes n) No (default) y/n> n Configuration complete. Options: - type: webdav - url: https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE - vendor: other - user: YOUR_PIC_USERNAME - pass: *** ENCRYPTED *** Keep this "pic" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y Current remotes: Name Type ==== ==== pic webdav e) Edit existing remote n) New remote d) Delete remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config e/n/d/r/c/s/q> q
Once done, you can use command line to browse and download/upload data.
Usage
- List a remote PIC directory:
rclone lsd <name>:<path>`
- Download a remote directory from PIC
rclone copy <name>:<path> <local_path>
- Upload a local directory to PIC
rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path>
When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files
--check-first -P --stats-one-line --transfers <n_transfers> --size-only
See rclone manual for more extensive documentation https://rclone.org/docs/
Obtaining a macaroon (for contacts)
Macaroons are valid up to 7 days.
For downloading data (read-only permissions on the path):
$ curl -u ${USERNAME} -X POST -H 'Content-Type: application/macaroon-request' \ -d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \ https://webdav.pic.es:8460/${RESTRICTED_PATH} { "macaroon": "MDA2MGxvY2F0aW", "uri": { "targetWithMacaroon": "https://webdav.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW", "baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW", "target": "https://webdav.pic.es:8460/${RESTRICTED_PATH}", "base": "https://webdav.pic.es:8460/" } }
For uploading data (full permissions on the path):
$ curl -u ${USERNAME} -X POST -H 'Content-Type: application/macaroon-request' \ -d '{"validity": "P7D"}' \ https://webdav.pic.es:8460/${RESTRICTED_PATH} { "macaroon": "MDA2MGxvY2F0aW", "uri": { "targetWithMacaroon": "https://webdav.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW", "baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW", "target": "https://webdav.pic.es:8460/${RESTRICTED_PATH}", "base": "https://webdav.pic.es:8460/" } }