Difference between revisions of "Transferring data to/from PIC"

From Public PIC Wiki
Jump to navigation Jump to search
 
(12 intermediate revisions by one other user not shown)
Line 1: Line 1:
−
== How to provide data access to external users without PIC account  ==
+
= How to provide data access to PIC massive storage (dCache) =
  
−
I will use an example for MICE
+
== Requirements ==
  
−
=== Steps ===
+
* Install and configure Rclone
 +
* PIC credentials or macaroon
  
−
1) dCache webdav door that exports your data.
+
== Install Rclone ==
  
−
https://webdav-mice.pic.es:8454
+
You can directly download the binary without installing anything. For instance, for a linux 64 bits machine:
  
−
2) dCacheView frontend (optional)
+
    $ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip
 +
    [...]
 +
    $ unzip rclone-current-linux-amd64.zip
  
−
https://webdav-mice.pic.es:8464
 
  
−
3) Authentication
+
Or if you prefer, you can install Rclone like the next example on a Ubuntu machine:
  
−
Either PIC credentials (user/password) or a macaroon (https://dcache.org/old/manuals/UserGuide-7.0/macaroons.shtml).
+
    $ cd /tmp
−
Any authenticated user can get a macaroon for an external user. Remember to add proper caveats to restrict privileges.
+
    $ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'
−
In the next example, I retrieve a macaroon with 7 days validity, that can only download and list, and can only access the subtree of the provided URL.
+
    $ sudo apt install ./rclone-current-linux-amd64.deb
 +
 
 +
== Configure Rclone ==
 +
 
 +
You should have been given some credentials and the url of a WebDAV endpoint at PIC.
 +
With them, you just need to create the config in rclone:
 +
 
 +
    $ rclone config
 +
    No remotes found, make a new one?
 +
    n) New remote
 +
    s) Set configuration password
 +
    q) Quit config
 +
    n/s/q> n
 +
   
 +
    Enter name for new remote.
 +
    name> pic
 +
   
 +
    Option Storage.
 +
    Type of storage to configure.
 +
    Choose a number from below, or type in your own value.
 +
    1 / 1Fichier
 +
      \ (fichier)
 +
    [...]
 +
    Storage> webdav
 +
   
 +
    Option url.
 +
    URL of http host to connect to.
 +
    E.g. https://example.com.
 +
    Enter a value.
 +
    url>  https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE
 +
   
 +
    Option vendor.
 +
    Name of the WebDAV site/service/software you are using.
 +
    Choose a number from below, or type in your own value.
 +
    Press Enter to leave empty.
 +
    1 / Nextcloud
 +
      \ (nextcloud)
 +
    [...]
 +
    5 / Other site/service or software
 +
      \ (other)
 +
    vendor> other
 +
   
 +
===  Using your PIC credentials ===
 +
 
 +
If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank.
 +
 
 +
    Option user.
 +
    User name.
 +
    In case NTLM authentication is used, the username should be in the format 'Domain\User'.
 +
    Enter a value. Press Enter to leave empty.
 +
    user> YOUR_PIC_USERNAME
 +
 
 +
    Option pass.
 +
    Password.
 +
    Choose an alternative below. Press Enter for the default (n).
 +
    y) Yes, type in my own password
 +
    g) Generate random password
 +
    n) No, leave this optional password blank (default)
 +
    y/g/n> y
 +
    Enter the password:
 +
    password: YOUR_PIC_PASSWORD
 +
    Confirm the password:
 +
    Password: YOUR_PIC_PASSWORD
 +
 
 +
=== Using a Macaroon token ===
 +
 
 +
If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank
 +
 
 +
    Option bearer_token.
 +
    Bearer token instead of user/pass (e.g. a Macaroon).
 +
    Enter a value. Press Enter to leave empty.
 +
    bearer_token> YOUR_MACAROON_TOKEN
 +
 
 +
=== Using an OIDC token ===
 +
 
 +
    Option bearer_token.
 +
    Bearer token instead of user/pass (e.g. a Macaroon).
 +
    Enter a value. Press Enter to leave empty.
 +
    bearer_token>
 +
   
 +
    Edit advanced config?
 +
    y) Yes
 +
    n) No (default)
 +
    y/n> y
 +
   
 +
    Option bearer_token_command.
 +
    Command to run to get a bearer token.
 +
    Enter a value. Press Enter to leave empty.
 +
    bearer_token_command> oidc-token OIDC_AGENT_ACCOUNT_SHORTNAME
 +
 
 +
=== Review settings ===
 +
 
 +
At the end, just review the information you entered and confirm.
 +
 
 +
    Edit advanced config?
 +
    y) Yes
 +
    n) No (default)
 +
    y/n> n
 +
   
 +
    Configuration complete.
 +
    Options:
 +
    - type: webdav
 +
    - url: https://door04.pic.es/PATH_TO_YOUR_STORAGE_SPACE
 +
    - vendor: other
 +
    - user: YOUR_PIC_USERNAME
 +
    - pass: *** ENCRYPTED ***
 +
    Keep this "pic" remote?
 +
    y) Yes this is OK (default)
 +
    e) Edit this remote
 +
    d) Delete this remote
 +
    y/e/d> y
 +
   
 +
    Current remotes:
 +
   
 +
    Name                Type
 +
    ====                ====
 +
    pic                  webdav
 +
   
 +
    e) Edit existing remote
 +
    n) New remote
 +
    d) Delete remote
 +
    r) Rename remote
 +
    c) Copy remote
 +
    s) Set configuration password
 +
    q) Quit config
 +
    e/n/d/r/c/s/q> q
 +
 
 +
Once done, you can use command line to browse and download/upload data.
 +
 
 +
=== Usage ===
 +
 
 +
* List a remote PIC directory:
 +
  rclone lsd <name>:<path>`
 +
 
 +
* Download a remote directory from PIC
 +
  rclone copy <name>:<path> <local_path>
 +
 
 +
* Upload a local directory to PIC
 +
  rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path>
 +
 
 +
When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files
 +
  --check-first -P --stats-one-line --transfers <n_transfers> --size-only
 +
 
 +
If uploading lots onto directories with lots of files (>1000), please use:
 +
  --no-traverse
 +
 
 +
If uploading files larger than 200 MB, also use
 +
  --multi-thread-streams 1
 +
 
 +
If uploading very large files (>10G), also use the following to allow more time to compute the checksums
 +
  --timeout=15m
 +
 
 +
See rclone manual for more extensive documentation https://rclone.org/docs/
 +
 
 +
== Configuring oidc-agent for obtaining OIDC tokens ==
 +
 
 +
Make sure <code>oidc-agent</code> is available.
 +
 
 +
=== Load oidc-agent ===
 +
 
 +
Initialize <code>oidc-agent</code> in the terminal session.
 +
 
 +
    $ eval `oidc-agent`
 +
 
 +
=== Configuration (only once) ===
 +
 
 +
To run this step you need an updated version of oidc-agent, version > 5.0.0.
 +
'''Ask your contact for the client-secret you have to replace below'''
 +
 
 +
Configure a <code>pic-dcache</code> account to retrieve tokens from PIC. Open the URL that will show and input the code provided (or just open the QR code displayed).
 +
After authenticating on the web browser, return to the terminal and input an encryption password twice. You'll need it when refreshing/reloading the <code>oidc-agent</code>.
 +
 
 +
    $ oidc-gen -m --client-id dcache-view \
 +
      --client-secret XXXXXXXXXXXXXXXXXX \
 +
      --pub --flow=device \
 +
      --discovery-endpoint=https://idp.pic.es/realms/PIC/.well-known/openid-configuration \
 +
      --scope="openid profile offline_access" --redirect-uri=edu.kit.data.oidc-agent:/ pic-dcache
 +
   
 +
    No account exists with this short name. Creating new configuration ...
 +
    Generating account configuration ...
 +
    accepted
 +
   
 +
    Using a browser on any device, visit:
 +
    https://idp.pic.es/realms/PIC/device
 +
   
 +
    And enter the code: ASDF-GHJK
 +
    Alternatively you can use the following QR code to visit the above listed URL.
 +
       
 +
    [ QR CODE ]
 +
   
 +
    Enter encryption password for account configuration 'pic-dcache':
 +
    Confirm encryption password:
 +
    Everything setup correctly!
 +
 
 +
=== Reauthenticating (if refresh token has expired) ===
 +
 
 +
If the oidc-agent process gets restarted, or iIf your refresh token expires due to inactivity, you will need to reauthenticate to retrieve further tokens
 +
 
 +
    $ oidc-gen --reauthenticate pic-dcache
 +
    Enter decryption password for account config 'testtest':
 +
    Generating account configuration ...
 +
    accepted
 +
   
 +
    Using a browser on any device, visit:
 +
    https://idp.pic.es/realms/PIC/device
 +
   
 +
    And enter the code: ASDF-GHJK
 +
    Alternatively you can use the following QR code to visit the above listed URL.
 +
   
 +
    [ QR CODE ]
 +
   
 +
    Enter encryption password for account configuration 'pic-dcache' [***]:
 +
    Everything setup correctly!
 +
 
 +
=== Testing ===
 +
 
 +
After loading and configuring, you can get a token by running the following command:
 +
 
 +
    $ oidc-token pic-dcache
 +
      eyJhbGciOiJSUzI1[...]4YjAwg
 +
 
 +
== Obtaining a macaroon (for contacts) ==
 +
 
 +
Macaroons are valid up to 7 days.
 +
 
 +
For downloading data (read-only permissions on the path):
  
 
<pre>
 
<pre>
−
$ curl -u tallada -X POST -H 'Content-Type: application/macaroon-request' \
+
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \
 
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \
 
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \
−
https://webdav-mice.pic.es:8454/tape/raw/production/N4096_L3072_LC3/maps/n8192/kmapgcpy/
+
https://door04.pic.es:8460/${RESTRICTED_PATH}
 +
 
 +
{
 +
    "macaroon": "MDA2MGxvY2F0aW",
 +
    "uri": {
 +
        "targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",
 +
        "baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",
 +
        "target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",
 +
        "base": "https://door04.pic.es:8460/"
 +
    }
 +
}
 +
</pre>
 +
 
 +
For uploading data (full permissions on the path):
 +
 
 +
<pre>
 +
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \
 +
-d '{"validity": "P7D"}' \
 +
https://door04.pic.es:8460/${RESTRICTED_PATH}
  
 
{
 
{
−
     "macaroon": "MDA3NWxvJmc2lnbmF0dXJlIJt81Pp8G1KXMH8FqwpWhJtwYnLfUAvNPQuq7bTwNOxGCg",  
+
     "macaroon": "MDA2MGxvY2F0aW",
 +
    "uri": {
 +
        "targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",
 +
        "baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",
 +
        "target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",
 +
        "base": "https://door04.pic.es:8460/"
 +
    }
 
}
 
}
 
</pre>
 
</pre>

Latest revision as of 15:18, 14 May 2026

How to provide data access to PIC massive storage (dCache)

Requirements

* Install and configure Rclone
* PIC credentials or macaroon

Install Rclone

You can directly download the binary without installing anything. For instance, for a linux 64 bits machine: 

   $ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip
   [...]
   $ unzip rclone-current-linux-amd64.zip


Or if you prefer, you can install Rclone like the next example on a Ubuntu machine: 

   $ cd /tmp
   $ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'
   $ sudo apt install ./rclone-current-linux-amd64.deb

Configure Rclone

You should have been given some credentials and the url of a WebDAV endpoint at PIC. With them, you just need to create the config in rclone: 

   $ rclone config
   No remotes found, make a new one?
   n) New remote
   s) Set configuration password
   q) Quit config
   n/s/q> n
   
   Enter name for new remote.
   name> pic
   
   Option Storage.
   Type of storage to configure.
   Choose a number from below, or type in your own value.
    1 / 1Fichier
      \ (fichier)
   [...]
   Storage> webdav
   
   Option url.
   URL of http host to connect to.
   E.g. https://example.com.
   Enter a value.
   url>  https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE
   
   Option vendor.
   Name of the WebDAV site/service/software you are using.
   Choose a number from below, or type in your own value.
   Press Enter to leave empty.
    1 / Nextcloud
      \ (nextcloud)
   [...]
    5 / Other site/service or software
      \ (other)
   vendor> other

Using your PIC credentials

If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank. 

   Option user.
   User name.
   In case NTLM authentication is used, the username should be in the format 'Domain\User'.
   Enter a value. Press Enter to leave empty.
   user> YOUR_PIC_USERNAME

   Option pass.
   Password.
   Choose an alternative below. Press Enter for the default (n).
   y) Yes, type in my own password
   g) Generate random password
   n) No, leave this optional password blank (default)
   y/g/n> y
   Enter the password:
   password: YOUR_PIC_PASSWORD
   Confirm the password:
   Password: YOUR_PIC_PASSWORD

Using a Macaroon token

If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank 

   Option bearer_token.
   Bearer token instead of user/pass (e.g. a Macaroon).
   Enter a value. Press Enter to leave empty.
   bearer_token> YOUR_MACAROON_TOKEN

Using an OIDC token

   Option bearer_token.
   Bearer token instead of user/pass (e.g. a Macaroon).
   Enter a value. Press Enter to leave empty.
   bearer_token> 
   
   Edit advanced config?
   y) Yes
   n) No (default)
   y/n> y
   
   Option bearer_token_command.
   Command to run to get a bearer token.
   Enter a value. Press Enter to leave empty.
   bearer_token_command> oidc-token OIDC_AGENT_ACCOUNT_SHORTNAME

Review settings

At the end, just review the information you entered and confirm. 

   Edit advanced config?
   y) Yes
   n) No (default)
   y/n> n
   
   Configuration complete.
   Options:
   - type: webdav
   - url: https://door04.pic.es/PATH_TO_YOUR_STORAGE_SPACE
   - vendor: other
   - user: YOUR_PIC_USERNAME
   - pass: *** ENCRYPTED ***
   Keep this "pic" remote?
   y) Yes this is OK (default)
   e) Edit this remote
   d) Delete this remote
   y/e/d> y
   
   Current remotes:
   
   Name                 Type
   ====                 ====
   pic                  webdav
   
   e) Edit existing remote
   n) New remote
   d) Delete remote
   r) Rename remote
   c) Copy remote
   s) Set configuration password
   q) Quit config
   e/n/d/r/c/s/q> q

Once done, you can use command line to browse and download/upload data.

Usage

  • List a remote PIC directory:
 rclone lsd <name>:<path>`
  • Download a remote directory from PIC
 rclone copy <name>:<path> <local_path>
  • Upload a local directory to PIC
 rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path>

When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files 

 --check-first -P --stats-one-line --transfers <n_transfers> --size-only

If uploading lots onto directories with lots of files (>1000), please use: 

 --no-traverse

If uploading files larger than 200 MB, also use 

  --multi-thread-streams 1

If uploading very large files (>10G), also use the following to allow more time to compute the checksums 

 --timeout=15m

See rclone manual for more extensive documentation https://rclone.org/docs/

Configuring oidc-agent for obtaining OIDC tokens

Make sure oidc-agent is available.

Load oidc-agent

Initialize oidc-agent in the terminal session. 

   $ eval `oidc-agent`

Configuration (only once)

To run this step you need an updated version of oidc-agent, version > 5.0.0. Ask your contact for the client-secret you have to replace below

Configure a pic-dcache account to retrieve tokens from PIC. Open the URL that will show and input the code provided (or just open the QR code displayed). After authenticating on the web browser, return to the terminal and input an encryption password twice. You'll need it when refreshing/reloading the oidc-agent. 

   $ oidc-gen -m --client-id dcache-view \
     --client-secret XXXXXXXXXXXXXXXXXX \
     --pub --flow=device \
     --discovery-endpoint=https://idp.pic.es/realms/PIC/.well-known/openid-configuration \
     --scope="openid profile offline_access" --redirect-uri=edu.kit.data.oidc-agent:/ pic-dcache
   
   No account exists with this short name. Creating new configuration ...
   Generating account configuration ...
   accepted
   
   Using a browser on any device, visit:
   https://idp.pic.es/realms/PIC/device
   
   And enter the code: ASDF-GHJK
   Alternatively you can use the following QR code to visit the above listed URL.
       
   [ QR CODE ]
   
   Enter encryption password for account configuration 'pic-dcache': 
   Confirm encryption password: 
   Everything setup correctly!

Reauthenticating (if refresh token has expired)

If the oidc-agent process gets restarted, or iIf your refresh token expires due to inactivity, you will need to reauthenticate to retrieve further tokens 

   $ oidc-gen --reauthenticate pic-dcache
   Enter decryption password for account config 'testtest': 
   Generating account configuration ...
   accepted
   
   Using a browser on any device, visit:
   https://idp.pic.es/realms/PIC/device
   
   And enter the code: ASDF-GHJK
   Alternatively you can use the following QR code to visit the above listed URL.
   
   [ QR CODE ]
   
   Enter encryption password for account configuration 'pic-dcache' [***]: 
   Everything setup correctly!

Testing

After loading and configuring, you can get a token by running the following command: 

   $ oidc-token pic-dcache
     eyJhbGciOiJSUzI1[...]4YjAwg

Obtaining a macaroon (for contacts)

Macaroons are valid up to 7 days.

For downloading data (read-only permissions on the path): 

$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \
https://door04.pic.es:8460/${RESTRICTED_PATH}

{
    "macaroon": "MDA2MGxvY2F0aW",
    "uri": {
        "targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",
        "baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",
        "target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",
        "base": "https://door04.pic.es:8460/"
    }
}

For uploading data (full permissions on the path): 

$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \
-d '{"validity": "P7D"}' \
https://door04.pic.es:8460/${RESTRICTED_PATH}

{
    "macaroon": "MDA2MGxvY2F0aW",
    "uri": {
        "targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",
        "baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",
        "target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",
        "base": "https://door04.pic.es:8460/"
    }
}
Retrieved from "http://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&oldid=1356"