https://pwiki.pic.es/api.php?action=feedcontributions&feedformat=atom&user=TalladaPublic PIC Wiki - User contributions [en]2026-06-16T07:43:16ZUser contributionsMediaWiki 1.35.14https://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1369Transferring data to/from PIC2026-06-08T11:20:59Z<p>Tallada: /* Configure SSH ControlMaster */</p>
<hr />
<div>= How to provide data access to PIC massive storage =<br />
<br />
This page explains how to configure <code>rclone</code> to access PIC storage.<br />
<br />
The configuration depends on the storage path you need to access.<br />
<br />
{| class="wikitable"<br />
|+ Storage access methods<br />
! Storage path<br />
! Protocol<br />
! rclone backend<br />
! Supported authentication<br />
! Example remote name<br />
|-<br />
| rowspan="2" | Paths under <code>/pnfs/pic.es/data</code><br />
| rowspan="2" | WebDAV / HTTPS<br />
| rowspan="2" | <code>webdav</code><br />
| OIDC token<br />
| <code>pic-pnfs-oidc</code><br />
|-<br />
| Macaroon bearer token<br />
| <code>pic-pnfs-macaroon</code><br />
|-<br />
| Any other storage location<br />
| SSH / SFTP<br />
| <code>sftp</code><br />
| SSH with <code>ControlMaster</code><br />
| <code>pic-storage</code><br />
|}<br />
<br />
== Requirements ==<br />
<br />
You need:<br />
<br />
* <code>rclone</code> installed on your client machine.<br />
<br />
For paths under <code>/pnfs/pic.es/data</code>, you also need one of the following:<br />
<br />
* an OIDC token setup using <code>oidc-agent</code>; or<br />
* a Macaroon token.<br />
<br />
For any other storage location, you also need:<br />
<br />
* SSH access to the PIC host.<br />
* SSH <code>ControlMaster</code> configured on your client machine.<br />
<br />
== Install rclone ==<br />
<br />
You can download the rclone binary directly without installing system packages. For example, on a Linux 64-bit machine:<br />
<br />
<pre><br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
$ unzip rclone-current-linux-amd64.zip<br />
</pre><br />
<br />
Alternatively, on Ubuntu you can install the Debian package:<br />
<br />
<pre><br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
</pre><br />
<br />
Check that rclone works:<br />
<br />
<pre><br />
$ rclone version<br />
</pre><br />
<br />
== Which configuration should I use? ==<br />
<br />
Use the table below to choose the right configuration.<br />
<br />
{| class="wikitable"<br />
|+ Choosing the right configuration<br />
! If you want to access...<br />
! Use this section<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using OIDC authentication<br />
| [[#Configure rclone for PNFS data paths using OIDC|Configure rclone for PNFS data paths using OIDC]]<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using a Macaroon token<br />
| [[#Configure rclone for PNFS data paths using a Macaroon|Configure rclone for PNFS data paths using a Macaroon]]<br />
|-<br />
| Any other storage location<br />
| [[#Configure rclone for any other storage location|Configure rclone for any other storage location]]<br />
|}<br />
<br />
== Configure rclone for PNFS data paths ==<br />
<br />
Paths under <code>/pnfs/pic.es/data</code> are accessed using WebDAV over HTTPS.<br />
<br />
For these paths, configure rclone with the <code>webdav</code> backend.<br />
<br />
Use one of the following authentication methods:<br />
<br />
* OIDC token, retrieved automatically with <code>oidc-token</code>.<br />
* Macaroon token, pasted into rclone as a bearer token.<br />
<br />
Do not configure username/password authentication unless PIC explicitly instructs you to do so.<br />
<br />
== Configure rclone for PNFS data paths using OIDC ==<br />
<br />
Use this option if you have been instructed to authenticate with OIDC.<br />
<br />
=== OIDC requirements ===<br />
<br />
Before configuring rclone, make sure that:<br />
<br />
* <code>oidc-agent</code> is available.<br />
* You have configured an OIDC account, for example <code>pic-pnfs</code>.<br />
* The following command returns a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command does not work, see [[#Configuring oidc-agent for OIDC tokens|Configuring oidc-agent for OIDC tokens]].<br />
<br />
=== Configure the WebDAV remote using OIDC ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-oidc<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV base path (https://webdav.pic.es), if desired, you can also enter a subdirectory (https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE).<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
When asked for a bearer token, leave it empty:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token><br />
</pre><br />
<br />
Edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> y<br />
</pre><br />
<br />
Set <code>bearer_token_command</code>:<br />
<br />
<pre><br />
Option bearer_token_command.<br />
Command to run to get a bearer token.<br />
bearer_token_command> oidc-token pic-pnfs<br />
</pre><br />
<br />
Accept the remaining advanced options by pressing <code>ENTER</code>, unless PIC instructed you to change them.<br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es<br />
- vendor: other<br />
- bearer_token_command: oidc-token pic-pnfs<br />
<br />
Keep this "pic-pnfs-oidc" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== OIDC configuration example ===<br />
<br />
The resulting rclone configuration at <code>~/.config/rclone/rclone.conf</code> should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-oidc]<br />
type = webdav<br />
url = https://webdav.pic.es<br />
vendor = other<br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
=== OIDC usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-oidc:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for PNFS data paths using a Macaroon ==<br />
<br />
Use this option if you have been given a Macaroon token.<br />
<br />
The Macaroon is used as a WebDAV bearer token.<br />
<br />
=== Configure the WebDAV remote using a Macaroon ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-macaroon<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV base path (https://webdav.pic.es) or, if desired, you can enter a subdirectory under it (https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE).<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
Paste the Macaroon token as the bearer token:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
You normally do not need to edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> n<br />
</pre><br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es<br />
- vendor: other<br />
- bearer_token: *** ENCRYPTED ***<br />
<br />
Keep this "pic-pnfs-macaroon" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== Macaroon configuration example ===<br />
<br />
The resulting rclone configuration under <code>~/.config/rclone/rclone.conf</code> should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-macaroon]<br />
type = webdav<br />
url = https://webdav.pic.es<br />
vendor = other<br />
bearer_token = YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
=== Macaroon usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-macaroon:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for any other storage location ==<br />
<br />
Use this section for storage paths that are not under <code>/pnfs/pic.es/data</code>.<br />
<br />
These locations are accessed using SSH/SFTP.<br />
<br />
Because SSH authentication may require browser-based authentication, you should configure SSH <code>ControlMaster</code>. This allows rclone to reuse an already-authenticated SSH connection.<br />
<br />
=== Configure SSH ControlMaster ===<br />
<br />
On your client machine, edit:<br />
<br />
<pre><br />
~/.ssh/config<br />
</pre><br />
<br />
Add an entry for the PIC SSH host. For example:<br />
<br />
<pre><br />
Host pic-ssh<br />
HostName ui.pic.es<br />
User YOUR_PIC_USERNAME<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Replace:<br />
<br />
* <code>ui04.pic.es</code> with the SSH host provided by PIC, if different.<br />
* <code>YOUR_PIC_USERNAME</code> with your PIC username.<br />
<br />
Instead of restricting this configuration to <code>pic-ssh</code>, you can enable it for all SSH connections:<br />
<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
However, using a specific host entry is usually clearer and safer.<br />
<br />
=== Open the first SSH connection ===<br />
<br />
Before using rclone, open one SSH connection manually:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
</pre><br />
<br />
You may see a message similar to:<br />
<br />
<pre><br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Open the URL in your browser, complete the authentication, then return to the terminal and press <code>ENTER</code>.<br />
<br />
After this first authentication, subsequent SSH connections to the same host should reuse the existing <code>ControlMaster</code> session.<br />
<br />
Test this with:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205<br />
[tallada@ui04 ~]$<br />
</pre><br />
<br />
You can close this shell after testing.<br />
<br />
=== Configure the SFTP rclone remote ===<br />
<br />
Other storage locations use the rclone <code>sftp</code> backend.<br />
<br />
Because we want rclone to reuse the SSH <code>ControlMaster</code> connection, configure the remote to use the external <code>ssh</code> command.<br />
<br />
First, locate your rclone configuration file:<br />
<br />
<pre><br />
$ rclone config file<br />
</pre><br />
<br />
The command prints something like:<br />
<br />
<pre><br />
Configuration file is stored at:<br />
/home/YOUR_LOCAL_USER/.config/rclone/rclone.conf<br />
</pre><br />
<br />
Edit that file and add:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
The important line is:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
This tells rclone to use your system SSH command, which can reuse the SSH <code>ControlMaster</code> connection.<br />
<br />
=== SFTP configuration example ===<br />
<br />
Example rclone configuration:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
=== SFTP usage examples ===<br />
<br />
List the remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:<br />
</pre><br />
<br />
List a specific storage path:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Example using a concrete absolute path:<br />
<br />
<pre><br />
$ rclone copy ./results pic-storage:/storage/projects/MY_PROJECT/results<br />
</pre><br />
<br />
== General rclone usage ==<br />
<br />
The same basic rclone commands can be used with all configured remotes.<br />
<br />
Replace <code>REMOTE_NAME</code> with one of the remotes you configured, for example:<br />
<br />
* <code>pic-pnfs-oidc</code><br />
* <code>pic-pnfs-macaroon</code><br />
* <code>pic-storage</code><br />
<br />
=== List a remote directory ===<br />
<br />
<pre><br />
$ rclone lsd REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
$ rclone lsd pic-pnfs-macaroon:<br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== List files recursively ===<br />
<br />
<pre><br />
$ rclone ls REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Download a remote directory ===<br />
<br />
<pre><br />
$ rclone copy REMOTE_NAME:PATH LOCAL_PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
=== Upload a local directory ===<br />
<br />
<pre><br />
$ rclone copy LOCAL_DIR REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Recommended upload flags ===<br />
<br />
For uploads, we recommend:<br />
<br />
<pre><br />
--check-first -P --stats-one-line --transfers N_TRANSFERS --size-only<br />
</pre><br />
<br />
For many small files, <code>N_TRANSFERS</code> can be up to <code>350</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only<br />
</pre><br />
<br />
If uploading into directories with many files, for example more than 1000 files, also use:<br />
<br />
<pre><br />
--no-traverse<br />
</pre><br />
<br />
If uploading files larger than 200 MB, also use:<br />
<br />
<pre><br />
--multi-thread-streams 1<br />
</pre><br />
<br />
If uploading very large files, for example larger than 10 GB, also use:<br />
<br />
<pre><br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to a path under <code>/pnfs/pic.es/data</code>:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to another storage location:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-storage:/PATH_TO_YOUR_STORAGE_SPACE \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
For more information, see the rclone documentation:<br />
<br />
<pre><br />
https://rclone.org/docs/<br />
</pre><br />
<br />
== Configuring oidc-agent for OIDC tokens ==<br />
<br />
This section is only needed if you use OIDC authentication for paths under <code>/pnfs/pic.es/data</code>.<br />
<br />
Make sure <code>oidc-agent</code> is available.<br />
<br />
=== Load oidc-agent ===<br />
<br />
Initialize <code>oidc-agent</code> in the terminal session:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
If the account is already configured, load it with:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
=== Configure the OIDC account ===<br />
<br />
This step only needs to be done once.<br />
<br />
You need an updated version of <code>oidc-agent</code>, version greater than <code>5.0.0</code>.<br />
<br />
Ask your PIC contact for the client secret and replace <code>XXXXXXXXXXXXXXXXXX</code> below.<br />
<br />
Configure a <code>pic-pnfs</code> account to retrieve tokens from PIC:<br />
<br />
<pre><br />
$ oidc-gen -m --client-id CLIENT_ID_PROVIDED_BY_PIC \<br />
--client-secret XXXXXXXXXXXXXXXXXX \<br />
--pub --flow=device \<br />
--discovery-endpoint=https://idp.pic.es/realms/PIC/.well-known/openid-configuration \<br />
--scope="openid profile offline_access" \<br />
--redirect-uri=edu.kit.data.oidc-agent:/ pic-pnfs<br />
</pre><br />
<br />
The command will show a URL and a code:<br />
<br />
<pre><br />
No account exists with this short name. Creating new configuration ...<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs':<br />
Confirm encryption password:<br />
Everything setup correctly!<br />
</pre><br />
<br />
Open the URL in your browser, enter the code, authenticate, then return to the terminal.<br />
<br />
You will be asked to enter an encryption password twice. You will need this password when refreshing or reloading the <code>oidc-agent</code> account.<br />
<br />
=== Reauthenticate if the refresh token has expired ===<br />
<br />
If the <code>oidc-agent</code> process is restarted, or if your refresh token expires due to inactivity, you may need to reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
Example output:<br />
<br />
<pre><br />
Enter decryption password for account config 'pic-pnfs':<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs' [***]:<br />
Everything setup correctly!<br />
</pre><br />
<br />
=== Test OIDC token retrieval ===<br />
<br />
After loading and configuring <code>oidc-agent</code>, test that you can retrieve a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command works, the rclone configuration using:<br />
<br />
<pre><br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
should also work.<br />
<br />
== Obtaining a Macaroon for /pnfs/pic.es/data ==<br />
<br />
This section is intended for contacts or administrators who need to generate Macaroons for users.<br />
<br />
Macaroons are valid for up to 7 days.<br />
<br />
The restricted path should be under <code>/pnfs/pic.es/data</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
RESTRICTED_PATH=pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Read-only Macaroon ===<br />
<br />
Use this for downloading data only.<br />
<br />
The Macaroon will allow:<br />
<br />
* listing directories;<br />
* downloading files.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
=== Upload Macaroon ===<br />
<br />
Use this for uploading data.<br />
<br />
The Macaroon will allow full permissions on the requested path.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
== Troubleshooting ==<br />
<br />
=== SSH authentication is requested repeatedly ===<br />
<br />
This applies to any storage location accessed through SSH/SFTP.<br />
<br />
Check that:<br />
<br />
* You have opened the first SSH connection manually with <code>ssh pic-ssh</code>.<br />
* Browser authentication was completed successfully.<br />
* Your <code>~/.ssh/config</code> contains <code>ControlMaster</code>, <code>ControlPath</code>, and <code>ControlPersist</code>.<br />
* Your rclone remote uses the external SSH command:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
=== OIDC authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using OIDC.<br />
<br />
Check that <code>oidc-agent</code> is running:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
Check that the account is loaded:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
Check that you can retrieve a token manually:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
</pre><br />
<br />
If the token cannot be retrieved, reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
=== Macaroon authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using a Macaroon.<br />
<br />
Check that:<br />
<br />
* The Macaroon was copied completely.<br />
* The Macaroon has not expired.<br />
* The Macaroon was generated for the correct path under <code>/pnfs/pic.es/data</code>.<br />
* The Macaroon has the required permissions, for example read-only or upload permissions.<br />
<br />
If the Macaroon has expired, request or generate a new one.</div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1368Transferring data to/from PIC2026-06-08T11:20:33Z<p>Tallada: /* Macaroon configuration example */</p>
<hr />
<div>= How to provide data access to PIC massive storage =<br />
<br />
This page explains how to configure <code>rclone</code> to access PIC storage.<br />
<br />
The configuration depends on the storage path you need to access.<br />
<br />
{| class="wikitable"<br />
|+ Storage access methods<br />
! Storage path<br />
! Protocol<br />
! rclone backend<br />
! Supported authentication<br />
! Example remote name<br />
|-<br />
| rowspan="2" | Paths under <code>/pnfs/pic.es/data</code><br />
| rowspan="2" | WebDAV / HTTPS<br />
| rowspan="2" | <code>webdav</code><br />
| OIDC token<br />
| <code>pic-pnfs-oidc</code><br />
|-<br />
| Macaroon bearer token<br />
| <code>pic-pnfs-macaroon</code><br />
|-<br />
| Any other storage location<br />
| SSH / SFTP<br />
| <code>sftp</code><br />
| SSH with <code>ControlMaster</code><br />
| <code>pic-storage</code><br />
|}<br />
<br />
== Requirements ==<br />
<br />
You need:<br />
<br />
* <code>rclone</code> installed on your client machine.<br />
<br />
For paths under <code>/pnfs/pic.es/data</code>, you also need one of the following:<br />
<br />
* an OIDC token setup using <code>oidc-agent</code>; or<br />
* a Macaroon token.<br />
<br />
For any other storage location, you also need:<br />
<br />
* SSH access to the PIC host.<br />
* SSH <code>ControlMaster</code> configured on your client machine.<br />
<br />
== Install rclone ==<br />
<br />
You can download the rclone binary directly without installing system packages. For example, on a Linux 64-bit machine:<br />
<br />
<pre><br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
$ unzip rclone-current-linux-amd64.zip<br />
</pre><br />
<br />
Alternatively, on Ubuntu you can install the Debian package:<br />
<br />
<pre><br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
</pre><br />
<br />
Check that rclone works:<br />
<br />
<pre><br />
$ rclone version<br />
</pre><br />
<br />
== Which configuration should I use? ==<br />
<br />
Use the table below to choose the right configuration.<br />
<br />
{| class="wikitable"<br />
|+ Choosing the right configuration<br />
! If you want to access...<br />
! Use this section<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using OIDC authentication<br />
| [[#Configure rclone for PNFS data paths using OIDC|Configure rclone for PNFS data paths using OIDC]]<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using a Macaroon token<br />
| [[#Configure rclone for PNFS data paths using a Macaroon|Configure rclone for PNFS data paths using a Macaroon]]<br />
|-<br />
| Any other storage location<br />
| [[#Configure rclone for any other storage location|Configure rclone for any other storage location]]<br />
|}<br />
<br />
== Configure rclone for PNFS data paths ==<br />
<br />
Paths under <code>/pnfs/pic.es/data</code> are accessed using WebDAV over HTTPS.<br />
<br />
For these paths, configure rclone with the <code>webdav</code> backend.<br />
<br />
Use one of the following authentication methods:<br />
<br />
* OIDC token, retrieved automatically with <code>oidc-token</code>.<br />
* Macaroon token, pasted into rclone as a bearer token.<br />
<br />
Do not configure username/password authentication unless PIC explicitly instructs you to do so.<br />
<br />
== Configure rclone for PNFS data paths using OIDC ==<br />
<br />
Use this option if you have been instructed to authenticate with OIDC.<br />
<br />
=== OIDC requirements ===<br />
<br />
Before configuring rclone, make sure that:<br />
<br />
* <code>oidc-agent</code> is available.<br />
* You have configured an OIDC account, for example <code>pic-pnfs</code>.<br />
* The following command returns a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command does not work, see [[#Configuring oidc-agent for OIDC tokens|Configuring oidc-agent for OIDC tokens]].<br />
<br />
=== Configure the WebDAV remote using OIDC ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-oidc<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV base path (https://webdav.pic.es), if desired, you can also enter a subdirectory (https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE).<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
When asked for a bearer token, leave it empty:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token><br />
</pre><br />
<br />
Edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> y<br />
</pre><br />
<br />
Set <code>bearer_token_command</code>:<br />
<br />
<pre><br />
Option bearer_token_command.<br />
Command to run to get a bearer token.<br />
bearer_token_command> oidc-token pic-pnfs<br />
</pre><br />
<br />
Accept the remaining advanced options by pressing <code>ENTER</code>, unless PIC instructed you to change them.<br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es<br />
- vendor: other<br />
- bearer_token_command: oidc-token pic-pnfs<br />
<br />
Keep this "pic-pnfs-oidc" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== OIDC configuration example ===<br />
<br />
The resulting rclone configuration at <code>~/.config/rclone/rclone.conf</code> should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-oidc]<br />
type = webdav<br />
url = https://webdav.pic.es<br />
vendor = other<br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
=== OIDC usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-oidc:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for PNFS data paths using a Macaroon ==<br />
<br />
Use this option if you have been given a Macaroon token.<br />
<br />
The Macaroon is used as a WebDAV bearer token.<br />
<br />
=== Configure the WebDAV remote using a Macaroon ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-macaroon<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV base path (https://webdav.pic.es) or, if desired, you can enter a subdirectory under it (https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE).<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
Paste the Macaroon token as the bearer token:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
You normally do not need to edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> n<br />
</pre><br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es<br />
- vendor: other<br />
- bearer_token: *** ENCRYPTED ***<br />
<br />
Keep this "pic-pnfs-macaroon" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== Macaroon configuration example ===<br />
<br />
The resulting rclone configuration under <code>~/.config/rclone/rclone.conf</code> should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-macaroon]<br />
type = webdav<br />
url = https://webdav.pic.es<br />
vendor = other<br />
bearer_token = YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
=== Macaroon usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-macaroon:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for any other storage location ==<br />
<br />
Use this section for storage paths that are not under <code>/pnfs/pic.es/data</code>.<br />
<br />
These locations are accessed using SSH/SFTP.<br />
<br />
Because SSH authentication may require browser-based authentication, you should configure SSH <code>ControlMaster</code>. This allows rclone to reuse an already-authenticated SSH connection.<br />
<br />
=== Configure SSH ControlMaster ===<br />
<br />
On your client machine, edit:<br />
<br />
<pre><br />
~/.ssh/config<br />
</pre><br />
<br />
Add an entry for the PIC SSH host. For example:<br />
<br />
<pre><br />
Host pic-ssh<br />
HostName ui04.pic.es<br />
User YOUR_PIC_USERNAME<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Replace:<br />
<br />
* <code>ui04.pic.es</code> with the SSH host provided by PIC, if different.<br />
* <code>YOUR_PIC_USERNAME</code> with your PIC username.<br />
<br />
Instead of restricting this configuration to <code>pic-ssh</code>, you can enable it for all SSH connections:<br />
<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
However, using a specific host entry is usually clearer and safer.<br />
<br />
=== Open the first SSH connection ===<br />
<br />
Before using rclone, open one SSH connection manually:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
</pre><br />
<br />
You may see a message similar to:<br />
<br />
<pre><br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Open the URL in your browser, complete the authentication, then return to the terminal and press <code>ENTER</code>.<br />
<br />
After this first authentication, subsequent SSH connections to the same host should reuse the existing <code>ControlMaster</code> session.<br />
<br />
Test this with:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205<br />
[tallada@ui04 ~]$<br />
</pre><br />
<br />
You can close this shell after testing.<br />
<br />
=== Configure the SFTP rclone remote ===<br />
<br />
Other storage locations use the rclone <code>sftp</code> backend.<br />
<br />
Because we want rclone to reuse the SSH <code>ControlMaster</code> connection, configure the remote to use the external <code>ssh</code> command.<br />
<br />
First, locate your rclone configuration file:<br />
<br />
<pre><br />
$ rclone config file<br />
</pre><br />
<br />
The command prints something like:<br />
<br />
<pre><br />
Configuration file is stored at:<br />
/home/YOUR_LOCAL_USER/.config/rclone/rclone.conf<br />
</pre><br />
<br />
Edit that file and add:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
The important line is:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
This tells rclone to use your system SSH command, which can reuse the SSH <code>ControlMaster</code> connection.<br />
<br />
=== SFTP configuration example ===<br />
<br />
Example rclone configuration:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
=== SFTP usage examples ===<br />
<br />
List the remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:<br />
</pre><br />
<br />
List a specific storage path:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Example using a concrete absolute path:<br />
<br />
<pre><br />
$ rclone copy ./results pic-storage:/storage/projects/MY_PROJECT/results<br />
</pre><br />
<br />
== General rclone usage ==<br />
<br />
The same basic rclone commands can be used with all configured remotes.<br />
<br />
Replace <code>REMOTE_NAME</code> with one of the remotes you configured, for example:<br />
<br />
* <code>pic-pnfs-oidc</code><br />
* <code>pic-pnfs-macaroon</code><br />
* <code>pic-storage</code><br />
<br />
=== List a remote directory ===<br />
<br />
<pre><br />
$ rclone lsd REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
$ rclone lsd pic-pnfs-macaroon:<br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== List files recursively ===<br />
<br />
<pre><br />
$ rclone ls REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Download a remote directory ===<br />
<br />
<pre><br />
$ rclone copy REMOTE_NAME:PATH LOCAL_PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
=== Upload a local directory ===<br />
<br />
<pre><br />
$ rclone copy LOCAL_DIR REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Recommended upload flags ===<br />
<br />
For uploads, we recommend:<br />
<br />
<pre><br />
--check-first -P --stats-one-line --transfers N_TRANSFERS --size-only<br />
</pre><br />
<br />
For many small files, <code>N_TRANSFERS</code> can be up to <code>350</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only<br />
</pre><br />
<br />
If uploading into directories with many files, for example more than 1000 files, also use:<br />
<br />
<pre><br />
--no-traverse<br />
</pre><br />
<br />
If uploading files larger than 200 MB, also use:<br />
<br />
<pre><br />
--multi-thread-streams 1<br />
</pre><br />
<br />
If uploading very large files, for example larger than 10 GB, also use:<br />
<br />
<pre><br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to a path under <code>/pnfs/pic.es/data</code>:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to another storage location:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-storage:/PATH_TO_YOUR_STORAGE_SPACE \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
For more information, see the rclone documentation:<br />
<br />
<pre><br />
https://rclone.org/docs/<br />
</pre><br />
<br />
== Configuring oidc-agent for OIDC tokens ==<br />
<br />
This section is only needed if you use OIDC authentication for paths under <code>/pnfs/pic.es/data</code>.<br />
<br />
Make sure <code>oidc-agent</code> is available.<br />
<br />
=== Load oidc-agent ===<br />
<br />
Initialize <code>oidc-agent</code> in the terminal session:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
If the account is already configured, load it with:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
=== Configure the OIDC account ===<br />
<br />
This step only needs to be done once.<br />
<br />
You need an updated version of <code>oidc-agent</code>, version greater than <code>5.0.0</code>.<br />
<br />
Ask your PIC contact for the client secret and replace <code>XXXXXXXXXXXXXXXXXX</code> below.<br />
<br />
Configure a <code>pic-pnfs</code> account to retrieve tokens from PIC:<br />
<br />
<pre><br />
$ oidc-gen -m --client-id CLIENT_ID_PROVIDED_BY_PIC \<br />
--client-secret XXXXXXXXXXXXXXXXXX \<br />
--pub --flow=device \<br />
--discovery-endpoint=https://idp.pic.es/realms/PIC/.well-known/openid-configuration \<br />
--scope="openid profile offline_access" \<br />
--redirect-uri=edu.kit.data.oidc-agent:/ pic-pnfs<br />
</pre><br />
<br />
The command will show a URL and a code:<br />
<br />
<pre><br />
No account exists with this short name. Creating new configuration ...<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs':<br />
Confirm encryption password:<br />
Everything setup correctly!<br />
</pre><br />
<br />
Open the URL in your browser, enter the code, authenticate, then return to the terminal.<br />
<br />
You will be asked to enter an encryption password twice. You will need this password when refreshing or reloading the <code>oidc-agent</code> account.<br />
<br />
=== Reauthenticate if the refresh token has expired ===<br />
<br />
If the <code>oidc-agent</code> process is restarted, or if your refresh token expires due to inactivity, you may need to reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
Example output:<br />
<br />
<pre><br />
Enter decryption password for account config 'pic-pnfs':<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs' [***]:<br />
Everything setup correctly!<br />
</pre><br />
<br />
=== Test OIDC token retrieval ===<br />
<br />
After loading and configuring <code>oidc-agent</code>, test that you can retrieve a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command works, the rclone configuration using:<br />
<br />
<pre><br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
should also work.<br />
<br />
== Obtaining a Macaroon for /pnfs/pic.es/data ==<br />
<br />
This section is intended for contacts or administrators who need to generate Macaroons for users.<br />
<br />
Macaroons are valid for up to 7 days.<br />
<br />
The restricted path should be under <code>/pnfs/pic.es/data</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
RESTRICTED_PATH=pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Read-only Macaroon ===<br />
<br />
Use this for downloading data only.<br />
<br />
The Macaroon will allow:<br />
<br />
* listing directories;<br />
* downloading files.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
=== Upload Macaroon ===<br />
<br />
Use this for uploading data.<br />
<br />
The Macaroon will allow full permissions on the requested path.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
== Troubleshooting ==<br />
<br />
=== SSH authentication is requested repeatedly ===<br />
<br />
This applies to any storage location accessed through SSH/SFTP.<br />
<br />
Check that:<br />
<br />
* You have opened the first SSH connection manually with <code>ssh pic-ssh</code>.<br />
* Browser authentication was completed successfully.<br />
* Your <code>~/.ssh/config</code> contains <code>ControlMaster</code>, <code>ControlPath</code>, and <code>ControlPersist</code>.<br />
* Your rclone remote uses the external SSH command:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
=== OIDC authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using OIDC.<br />
<br />
Check that <code>oidc-agent</code> is running:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
Check that the account is loaded:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
Check that you can retrieve a token manually:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
</pre><br />
<br />
If the token cannot be retrieved, reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
=== Macaroon authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using a Macaroon.<br />
<br />
Check that:<br />
<br />
* The Macaroon was copied completely.<br />
* The Macaroon has not expired.<br />
* The Macaroon was generated for the correct path under <code>/pnfs/pic.es/data</code>.<br />
* The Macaroon has the required permissions, for example read-only or upload permissions.<br />
<br />
If the Macaroon has expired, request or generate a new one.</div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1367Transferring data to/from PIC2026-06-08T11:19:38Z<p>Tallada: /* Configure the WebDAV remote using a Macaroon */</p>
<hr />
<div>= How to provide data access to PIC massive storage =<br />
<br />
This page explains how to configure <code>rclone</code> to access PIC storage.<br />
<br />
The configuration depends on the storage path you need to access.<br />
<br />
{| class="wikitable"<br />
|+ Storage access methods<br />
! Storage path<br />
! Protocol<br />
! rclone backend<br />
! Supported authentication<br />
! Example remote name<br />
|-<br />
| rowspan="2" | Paths under <code>/pnfs/pic.es/data</code><br />
| rowspan="2" | WebDAV / HTTPS<br />
| rowspan="2" | <code>webdav</code><br />
| OIDC token<br />
| <code>pic-pnfs-oidc</code><br />
|-<br />
| Macaroon bearer token<br />
| <code>pic-pnfs-macaroon</code><br />
|-<br />
| Any other storage location<br />
| SSH / SFTP<br />
| <code>sftp</code><br />
| SSH with <code>ControlMaster</code><br />
| <code>pic-storage</code><br />
|}<br />
<br />
== Requirements ==<br />
<br />
You need:<br />
<br />
* <code>rclone</code> installed on your client machine.<br />
<br />
For paths under <code>/pnfs/pic.es/data</code>, you also need one of the following:<br />
<br />
* an OIDC token setup using <code>oidc-agent</code>; or<br />
* a Macaroon token.<br />
<br />
For any other storage location, you also need:<br />
<br />
* SSH access to the PIC host.<br />
* SSH <code>ControlMaster</code> configured on your client machine.<br />
<br />
== Install rclone ==<br />
<br />
You can download the rclone binary directly without installing system packages. For example, on a Linux 64-bit machine:<br />
<br />
<pre><br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
$ unzip rclone-current-linux-amd64.zip<br />
</pre><br />
<br />
Alternatively, on Ubuntu you can install the Debian package:<br />
<br />
<pre><br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
</pre><br />
<br />
Check that rclone works:<br />
<br />
<pre><br />
$ rclone version<br />
</pre><br />
<br />
== Which configuration should I use? ==<br />
<br />
Use the table below to choose the right configuration.<br />
<br />
{| class="wikitable"<br />
|+ Choosing the right configuration<br />
! If you want to access...<br />
! Use this section<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using OIDC authentication<br />
| [[#Configure rclone for PNFS data paths using OIDC|Configure rclone for PNFS data paths using OIDC]]<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using a Macaroon token<br />
| [[#Configure rclone for PNFS data paths using a Macaroon|Configure rclone for PNFS data paths using a Macaroon]]<br />
|-<br />
| Any other storage location<br />
| [[#Configure rclone for any other storage location|Configure rclone for any other storage location]]<br />
|}<br />
<br />
== Configure rclone for PNFS data paths ==<br />
<br />
Paths under <code>/pnfs/pic.es/data</code> are accessed using WebDAV over HTTPS.<br />
<br />
For these paths, configure rclone with the <code>webdav</code> backend.<br />
<br />
Use one of the following authentication methods:<br />
<br />
* OIDC token, retrieved automatically with <code>oidc-token</code>.<br />
* Macaroon token, pasted into rclone as a bearer token.<br />
<br />
Do not configure username/password authentication unless PIC explicitly instructs you to do so.<br />
<br />
== Configure rclone for PNFS data paths using OIDC ==<br />
<br />
Use this option if you have been instructed to authenticate with OIDC.<br />
<br />
=== OIDC requirements ===<br />
<br />
Before configuring rclone, make sure that:<br />
<br />
* <code>oidc-agent</code> is available.<br />
* You have configured an OIDC account, for example <code>pic-pnfs</code>.<br />
* The following command returns a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command does not work, see [[#Configuring oidc-agent for OIDC tokens|Configuring oidc-agent for OIDC tokens]].<br />
<br />
=== Configure the WebDAV remote using OIDC ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-oidc<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV base path (https://webdav.pic.es), if desired, you can also enter a subdirectory (https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE).<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
When asked for a bearer token, leave it empty:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token><br />
</pre><br />
<br />
Edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> y<br />
</pre><br />
<br />
Set <code>bearer_token_command</code>:<br />
<br />
<pre><br />
Option bearer_token_command.<br />
Command to run to get a bearer token.<br />
bearer_token_command> oidc-token pic-pnfs<br />
</pre><br />
<br />
Accept the remaining advanced options by pressing <code>ENTER</code>, unless PIC instructed you to change them.<br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es<br />
- vendor: other<br />
- bearer_token_command: oidc-token pic-pnfs<br />
<br />
Keep this "pic-pnfs-oidc" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== OIDC configuration example ===<br />
<br />
The resulting rclone configuration at <code>~/.config/rclone/rclone.conf</code> should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-oidc]<br />
type = webdav<br />
url = https://webdav.pic.es<br />
vendor = other<br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
=== OIDC usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-oidc:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for PNFS data paths using a Macaroon ==<br />
<br />
Use this option if you have been given a Macaroon token.<br />
<br />
The Macaroon is used as a WebDAV bearer token.<br />
<br />
=== Configure the WebDAV remote using a Macaroon ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-macaroon<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV base path (https://webdav.pic.es) or, if desired, you can enter a subdirectory under it (https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE).<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
Paste the Macaroon token as the bearer token:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
You normally do not need to edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> n<br />
</pre><br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es<br />
- vendor: other<br />
- bearer_token: *** ENCRYPTED ***<br />
<br />
Keep this "pic-pnfs-macaroon" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== Macaroon configuration example ===<br />
<br />
The resulting rclone configuration should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-macaroon]<br />
type = webdav<br />
url = https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
vendor = other<br />
bearer_token = YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
=== Macaroon usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-macaroon:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for any other storage location ==<br />
<br />
Use this section for storage paths that are not under <code>/pnfs/pic.es/data</code>.<br />
<br />
These locations are accessed using SSH/SFTP.<br />
<br />
Because SSH authentication may require browser-based authentication, you should configure SSH <code>ControlMaster</code>. This allows rclone to reuse an already-authenticated SSH connection.<br />
<br />
=== Configure SSH ControlMaster ===<br />
<br />
On your client machine, edit:<br />
<br />
<pre><br />
~/.ssh/config<br />
</pre><br />
<br />
Add an entry for the PIC SSH host. For example:<br />
<br />
<pre><br />
Host pic-ssh<br />
HostName ui04.pic.es<br />
User YOUR_PIC_USERNAME<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Replace:<br />
<br />
* <code>ui04.pic.es</code> with the SSH host provided by PIC, if different.<br />
* <code>YOUR_PIC_USERNAME</code> with your PIC username.<br />
<br />
Instead of restricting this configuration to <code>pic-ssh</code>, you can enable it for all SSH connections:<br />
<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
However, using a specific host entry is usually clearer and safer.<br />
<br />
=== Open the first SSH connection ===<br />
<br />
Before using rclone, open one SSH connection manually:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
</pre><br />
<br />
You may see a message similar to:<br />
<br />
<pre><br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Open the URL in your browser, complete the authentication, then return to the terminal and press <code>ENTER</code>.<br />
<br />
After this first authentication, subsequent SSH connections to the same host should reuse the existing <code>ControlMaster</code> session.<br />
<br />
Test this with:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205<br />
[tallada@ui04 ~]$<br />
</pre><br />
<br />
You can close this shell after testing.<br />
<br />
=== Configure the SFTP rclone remote ===<br />
<br />
Other storage locations use the rclone <code>sftp</code> backend.<br />
<br />
Because we want rclone to reuse the SSH <code>ControlMaster</code> connection, configure the remote to use the external <code>ssh</code> command.<br />
<br />
First, locate your rclone configuration file:<br />
<br />
<pre><br />
$ rclone config file<br />
</pre><br />
<br />
The command prints something like:<br />
<br />
<pre><br />
Configuration file is stored at:<br />
/home/YOUR_LOCAL_USER/.config/rclone/rclone.conf<br />
</pre><br />
<br />
Edit that file and add:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
The important line is:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
This tells rclone to use your system SSH command, which can reuse the SSH <code>ControlMaster</code> connection.<br />
<br />
=== SFTP configuration example ===<br />
<br />
Example rclone configuration:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
=== SFTP usage examples ===<br />
<br />
List the remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:<br />
</pre><br />
<br />
List a specific storage path:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Example using a concrete absolute path:<br />
<br />
<pre><br />
$ rclone copy ./results pic-storage:/storage/projects/MY_PROJECT/results<br />
</pre><br />
<br />
== General rclone usage ==<br />
<br />
The same basic rclone commands can be used with all configured remotes.<br />
<br />
Replace <code>REMOTE_NAME</code> with one of the remotes you configured, for example:<br />
<br />
* <code>pic-pnfs-oidc</code><br />
* <code>pic-pnfs-macaroon</code><br />
* <code>pic-storage</code><br />
<br />
=== List a remote directory ===<br />
<br />
<pre><br />
$ rclone lsd REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
$ rclone lsd pic-pnfs-macaroon:<br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== List files recursively ===<br />
<br />
<pre><br />
$ rclone ls REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Download a remote directory ===<br />
<br />
<pre><br />
$ rclone copy REMOTE_NAME:PATH LOCAL_PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
=== Upload a local directory ===<br />
<br />
<pre><br />
$ rclone copy LOCAL_DIR REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Recommended upload flags ===<br />
<br />
For uploads, we recommend:<br />
<br />
<pre><br />
--check-first -P --stats-one-line --transfers N_TRANSFERS --size-only<br />
</pre><br />
<br />
For many small files, <code>N_TRANSFERS</code> can be up to <code>350</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only<br />
</pre><br />
<br />
If uploading into directories with many files, for example more than 1000 files, also use:<br />
<br />
<pre><br />
--no-traverse<br />
</pre><br />
<br />
If uploading files larger than 200 MB, also use:<br />
<br />
<pre><br />
--multi-thread-streams 1<br />
</pre><br />
<br />
If uploading very large files, for example larger than 10 GB, also use:<br />
<br />
<pre><br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to a path under <code>/pnfs/pic.es/data</code>:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to another storage location:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-storage:/PATH_TO_YOUR_STORAGE_SPACE \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
For more information, see the rclone documentation:<br />
<br />
<pre><br />
https://rclone.org/docs/<br />
</pre><br />
<br />
== Configuring oidc-agent for OIDC tokens ==<br />
<br />
This section is only needed if you use OIDC authentication for paths under <code>/pnfs/pic.es/data</code>.<br />
<br />
Make sure <code>oidc-agent</code> is available.<br />
<br />
=== Load oidc-agent ===<br />
<br />
Initialize <code>oidc-agent</code> in the terminal session:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
If the account is already configured, load it with:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
=== Configure the OIDC account ===<br />
<br />
This step only needs to be done once.<br />
<br />
You need an updated version of <code>oidc-agent</code>, version greater than <code>5.0.0</code>.<br />
<br />
Ask your PIC contact for the client secret and replace <code>XXXXXXXXXXXXXXXXXX</code> below.<br />
<br />
Configure a <code>pic-pnfs</code> account to retrieve tokens from PIC:<br />
<br />
<pre><br />
$ oidc-gen -m --client-id CLIENT_ID_PROVIDED_BY_PIC \<br />
--client-secret XXXXXXXXXXXXXXXXXX \<br />
--pub --flow=device \<br />
--discovery-endpoint=https://idp.pic.es/realms/PIC/.well-known/openid-configuration \<br />
--scope="openid profile offline_access" \<br />
--redirect-uri=edu.kit.data.oidc-agent:/ pic-pnfs<br />
</pre><br />
<br />
The command will show a URL and a code:<br />
<br />
<pre><br />
No account exists with this short name. Creating new configuration ...<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs':<br />
Confirm encryption password:<br />
Everything setup correctly!<br />
</pre><br />
<br />
Open the URL in your browser, enter the code, authenticate, then return to the terminal.<br />
<br />
You will be asked to enter an encryption password twice. You will need this password when refreshing or reloading the <code>oidc-agent</code> account.<br />
<br />
=== Reauthenticate if the refresh token has expired ===<br />
<br />
If the <code>oidc-agent</code> process is restarted, or if your refresh token expires due to inactivity, you may need to reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
Example output:<br />
<br />
<pre><br />
Enter decryption password for account config 'pic-pnfs':<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs' [***]:<br />
Everything setup correctly!<br />
</pre><br />
<br />
=== Test OIDC token retrieval ===<br />
<br />
After loading and configuring <code>oidc-agent</code>, test that you can retrieve a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command works, the rclone configuration using:<br />
<br />
<pre><br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
should also work.<br />
<br />
== Obtaining a Macaroon for /pnfs/pic.es/data ==<br />
<br />
This section is intended for contacts or administrators who need to generate Macaroons for users.<br />
<br />
Macaroons are valid for up to 7 days.<br />
<br />
The restricted path should be under <code>/pnfs/pic.es/data</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
RESTRICTED_PATH=pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Read-only Macaroon ===<br />
<br />
Use this for downloading data only.<br />
<br />
The Macaroon will allow:<br />
<br />
* listing directories;<br />
* downloading files.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
=== Upload Macaroon ===<br />
<br />
Use this for uploading data.<br />
<br />
The Macaroon will allow full permissions on the requested path.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
== Troubleshooting ==<br />
<br />
=== SSH authentication is requested repeatedly ===<br />
<br />
This applies to any storage location accessed through SSH/SFTP.<br />
<br />
Check that:<br />
<br />
* You have opened the first SSH connection manually with <code>ssh pic-ssh</code>.<br />
* Browser authentication was completed successfully.<br />
* Your <code>~/.ssh/config</code> contains <code>ControlMaster</code>, <code>ControlPath</code>, and <code>ControlPersist</code>.<br />
* Your rclone remote uses the external SSH command:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
=== OIDC authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using OIDC.<br />
<br />
Check that <code>oidc-agent</code> is running:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
Check that the account is loaded:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
Check that you can retrieve a token manually:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
</pre><br />
<br />
If the token cannot be retrieved, reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
=== Macaroon authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using a Macaroon.<br />
<br />
Check that:<br />
<br />
* The Macaroon was copied completely.<br />
* The Macaroon has not expired.<br />
* The Macaroon was generated for the correct path under <code>/pnfs/pic.es/data</code>.<br />
* The Macaroon has the required permissions, for example read-only or upload permissions.<br />
<br />
If the Macaroon has expired, request or generate a new one.</div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1366Transferring data to/from PIC2026-06-08T11:18:07Z<p>Tallada: /* OIDC configuration example */</p>
<hr />
<div>= How to provide data access to PIC massive storage =<br />
<br />
This page explains how to configure <code>rclone</code> to access PIC storage.<br />
<br />
The configuration depends on the storage path you need to access.<br />
<br />
{| class="wikitable"<br />
|+ Storage access methods<br />
! Storage path<br />
! Protocol<br />
! rclone backend<br />
! Supported authentication<br />
! Example remote name<br />
|-<br />
| rowspan="2" | Paths under <code>/pnfs/pic.es/data</code><br />
| rowspan="2" | WebDAV / HTTPS<br />
| rowspan="2" | <code>webdav</code><br />
| OIDC token<br />
| <code>pic-pnfs-oidc</code><br />
|-<br />
| Macaroon bearer token<br />
| <code>pic-pnfs-macaroon</code><br />
|-<br />
| Any other storage location<br />
| SSH / SFTP<br />
| <code>sftp</code><br />
| SSH with <code>ControlMaster</code><br />
| <code>pic-storage</code><br />
|}<br />
<br />
== Requirements ==<br />
<br />
You need:<br />
<br />
* <code>rclone</code> installed on your client machine.<br />
<br />
For paths under <code>/pnfs/pic.es/data</code>, you also need one of the following:<br />
<br />
* an OIDC token setup using <code>oidc-agent</code>; or<br />
* a Macaroon token.<br />
<br />
For any other storage location, you also need:<br />
<br />
* SSH access to the PIC host.<br />
* SSH <code>ControlMaster</code> configured on your client machine.<br />
<br />
== Install rclone ==<br />
<br />
You can download the rclone binary directly without installing system packages. For example, on a Linux 64-bit machine:<br />
<br />
<pre><br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
$ unzip rclone-current-linux-amd64.zip<br />
</pre><br />
<br />
Alternatively, on Ubuntu you can install the Debian package:<br />
<br />
<pre><br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
</pre><br />
<br />
Check that rclone works:<br />
<br />
<pre><br />
$ rclone version<br />
</pre><br />
<br />
== Which configuration should I use? ==<br />
<br />
Use the table below to choose the right configuration.<br />
<br />
{| class="wikitable"<br />
|+ Choosing the right configuration<br />
! If you want to access...<br />
! Use this section<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using OIDC authentication<br />
| [[#Configure rclone for PNFS data paths using OIDC|Configure rclone for PNFS data paths using OIDC]]<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using a Macaroon token<br />
| [[#Configure rclone for PNFS data paths using a Macaroon|Configure rclone for PNFS data paths using a Macaroon]]<br />
|-<br />
| Any other storage location<br />
| [[#Configure rclone for any other storage location|Configure rclone for any other storage location]]<br />
|}<br />
<br />
== Configure rclone for PNFS data paths ==<br />
<br />
Paths under <code>/pnfs/pic.es/data</code> are accessed using WebDAV over HTTPS.<br />
<br />
For these paths, configure rclone with the <code>webdav</code> backend.<br />
<br />
Use one of the following authentication methods:<br />
<br />
* OIDC token, retrieved automatically with <code>oidc-token</code>.<br />
* Macaroon token, pasted into rclone as a bearer token.<br />
<br />
Do not configure username/password authentication unless PIC explicitly instructs you to do so.<br />
<br />
== Configure rclone for PNFS data paths using OIDC ==<br />
<br />
Use this option if you have been instructed to authenticate with OIDC.<br />
<br />
=== OIDC requirements ===<br />
<br />
Before configuring rclone, make sure that:<br />
<br />
* <code>oidc-agent</code> is available.<br />
* You have configured an OIDC account, for example <code>pic-pnfs</code>.<br />
* The following command returns a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command does not work, see [[#Configuring oidc-agent for OIDC tokens|Configuring oidc-agent for OIDC tokens]].<br />
<br />
=== Configure the WebDAV remote using OIDC ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-oidc<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV base path (https://webdav.pic.es), if desired, you can also enter a subdirectory (https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE).<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
When asked for a bearer token, leave it empty:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token><br />
</pre><br />
<br />
Edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> y<br />
</pre><br />
<br />
Set <code>bearer_token_command</code>:<br />
<br />
<pre><br />
Option bearer_token_command.<br />
Command to run to get a bearer token.<br />
bearer_token_command> oidc-token pic-pnfs<br />
</pre><br />
<br />
Accept the remaining advanced options by pressing <code>ENTER</code>, unless PIC instructed you to change them.<br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es<br />
- vendor: other<br />
- bearer_token_command: oidc-token pic-pnfs<br />
<br />
Keep this "pic-pnfs-oidc" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== OIDC configuration example ===<br />
<br />
The resulting rclone configuration at <code>~/.config/rclone/rclone.conf</code> should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-oidc]<br />
type = webdav<br />
url = https://webdav.pic.es<br />
vendor = other<br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
=== OIDC usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-oidc:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for PNFS data paths using a Macaroon ==<br />
<br />
Use this option if you have been given a Macaroon token.<br />
<br />
The Macaroon is used as a WebDAV bearer token.<br />
<br />
=== Configure the WebDAV remote using a Macaroon ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-macaroon<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV endpoint provided by PIC.<br />
<br />
For a path under <code>/pnfs/pic.es/data</code>, the URL will normally point to that path or to a subdirectory under it.<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
Paste the Macaroon token as the bearer token:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
You normally do not need to edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> n<br />
</pre><br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- bearer_token: *** ENCRYPTED ***<br />
<br />
Keep this "pic-pnfs-macaroon" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== Macaroon configuration example ===<br />
<br />
The resulting rclone configuration should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-macaroon]<br />
type = webdav<br />
url = https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
vendor = other<br />
bearer_token = YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
=== Macaroon usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-macaroon:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for any other storage location ==<br />
<br />
Use this section for storage paths that are not under <code>/pnfs/pic.es/data</code>.<br />
<br />
These locations are accessed using SSH/SFTP.<br />
<br />
Because SSH authentication may require browser-based authentication, you should configure SSH <code>ControlMaster</code>. This allows rclone to reuse an already-authenticated SSH connection.<br />
<br />
=== Configure SSH ControlMaster ===<br />
<br />
On your client machine, edit:<br />
<br />
<pre><br />
~/.ssh/config<br />
</pre><br />
<br />
Add an entry for the PIC SSH host. For example:<br />
<br />
<pre><br />
Host pic-ssh<br />
HostName ui04.pic.es<br />
User YOUR_PIC_USERNAME<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Replace:<br />
<br />
* <code>ui04.pic.es</code> with the SSH host provided by PIC, if different.<br />
* <code>YOUR_PIC_USERNAME</code> with your PIC username.<br />
<br />
Instead of restricting this configuration to <code>pic-ssh</code>, you can enable it for all SSH connections:<br />
<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
However, using a specific host entry is usually clearer and safer.<br />
<br />
=== Open the first SSH connection ===<br />
<br />
Before using rclone, open one SSH connection manually:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
</pre><br />
<br />
You may see a message similar to:<br />
<br />
<pre><br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Open the URL in your browser, complete the authentication, then return to the terminal and press <code>ENTER</code>.<br />
<br />
After this first authentication, subsequent SSH connections to the same host should reuse the existing <code>ControlMaster</code> session.<br />
<br />
Test this with:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205<br />
[tallada@ui04 ~]$<br />
</pre><br />
<br />
You can close this shell after testing.<br />
<br />
=== Configure the SFTP rclone remote ===<br />
<br />
Other storage locations use the rclone <code>sftp</code> backend.<br />
<br />
Because we want rclone to reuse the SSH <code>ControlMaster</code> connection, configure the remote to use the external <code>ssh</code> command.<br />
<br />
First, locate your rclone configuration file:<br />
<br />
<pre><br />
$ rclone config file<br />
</pre><br />
<br />
The command prints something like:<br />
<br />
<pre><br />
Configuration file is stored at:<br />
/home/YOUR_LOCAL_USER/.config/rclone/rclone.conf<br />
</pre><br />
<br />
Edit that file and add:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
The important line is:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
This tells rclone to use your system SSH command, which can reuse the SSH <code>ControlMaster</code> connection.<br />
<br />
=== SFTP configuration example ===<br />
<br />
Example rclone configuration:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
=== SFTP usage examples ===<br />
<br />
List the remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:<br />
</pre><br />
<br />
List a specific storage path:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Example using a concrete absolute path:<br />
<br />
<pre><br />
$ rclone copy ./results pic-storage:/storage/projects/MY_PROJECT/results<br />
</pre><br />
<br />
== General rclone usage ==<br />
<br />
The same basic rclone commands can be used with all configured remotes.<br />
<br />
Replace <code>REMOTE_NAME</code> with one of the remotes you configured, for example:<br />
<br />
* <code>pic-pnfs-oidc</code><br />
* <code>pic-pnfs-macaroon</code><br />
* <code>pic-storage</code><br />
<br />
=== List a remote directory ===<br />
<br />
<pre><br />
$ rclone lsd REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
$ rclone lsd pic-pnfs-macaroon:<br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== List files recursively ===<br />
<br />
<pre><br />
$ rclone ls REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Download a remote directory ===<br />
<br />
<pre><br />
$ rclone copy REMOTE_NAME:PATH LOCAL_PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
=== Upload a local directory ===<br />
<br />
<pre><br />
$ rclone copy LOCAL_DIR REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Recommended upload flags ===<br />
<br />
For uploads, we recommend:<br />
<br />
<pre><br />
--check-first -P --stats-one-line --transfers N_TRANSFERS --size-only<br />
</pre><br />
<br />
For many small files, <code>N_TRANSFERS</code> can be up to <code>350</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only<br />
</pre><br />
<br />
If uploading into directories with many files, for example more than 1000 files, also use:<br />
<br />
<pre><br />
--no-traverse<br />
</pre><br />
<br />
If uploading files larger than 200 MB, also use:<br />
<br />
<pre><br />
--multi-thread-streams 1<br />
</pre><br />
<br />
If uploading very large files, for example larger than 10 GB, also use:<br />
<br />
<pre><br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to a path under <code>/pnfs/pic.es/data</code>:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to another storage location:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-storage:/PATH_TO_YOUR_STORAGE_SPACE \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
For more information, see the rclone documentation:<br />
<br />
<pre><br />
https://rclone.org/docs/<br />
</pre><br />
<br />
== Configuring oidc-agent for OIDC tokens ==<br />
<br />
This section is only needed if you use OIDC authentication for paths under <code>/pnfs/pic.es/data</code>.<br />
<br />
Make sure <code>oidc-agent</code> is available.<br />
<br />
=== Load oidc-agent ===<br />
<br />
Initialize <code>oidc-agent</code> in the terminal session:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
If the account is already configured, load it with:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
=== Configure the OIDC account ===<br />
<br />
This step only needs to be done once.<br />
<br />
You need an updated version of <code>oidc-agent</code>, version greater than <code>5.0.0</code>.<br />
<br />
Ask your PIC contact for the client secret and replace <code>XXXXXXXXXXXXXXXXXX</code> below.<br />
<br />
Configure a <code>pic-pnfs</code> account to retrieve tokens from PIC:<br />
<br />
<pre><br />
$ oidc-gen -m --client-id CLIENT_ID_PROVIDED_BY_PIC \<br />
--client-secret XXXXXXXXXXXXXXXXXX \<br />
--pub --flow=device \<br />
--discovery-endpoint=https://idp.pic.es/realms/PIC/.well-known/openid-configuration \<br />
--scope="openid profile offline_access" \<br />
--redirect-uri=edu.kit.data.oidc-agent:/ pic-pnfs<br />
</pre><br />
<br />
The command will show a URL and a code:<br />
<br />
<pre><br />
No account exists with this short name. Creating new configuration ...<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs':<br />
Confirm encryption password:<br />
Everything setup correctly!<br />
</pre><br />
<br />
Open the URL in your browser, enter the code, authenticate, then return to the terminal.<br />
<br />
You will be asked to enter an encryption password twice. You will need this password when refreshing or reloading the <code>oidc-agent</code> account.<br />
<br />
=== Reauthenticate if the refresh token has expired ===<br />
<br />
If the <code>oidc-agent</code> process is restarted, or if your refresh token expires due to inactivity, you may need to reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
Example output:<br />
<br />
<pre><br />
Enter decryption password for account config 'pic-pnfs':<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs' [***]:<br />
Everything setup correctly!<br />
</pre><br />
<br />
=== Test OIDC token retrieval ===<br />
<br />
After loading and configuring <code>oidc-agent</code>, test that you can retrieve a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command works, the rclone configuration using:<br />
<br />
<pre><br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
should also work.<br />
<br />
== Obtaining a Macaroon for /pnfs/pic.es/data ==<br />
<br />
This section is intended for contacts or administrators who need to generate Macaroons for users.<br />
<br />
Macaroons are valid for up to 7 days.<br />
<br />
The restricted path should be under <code>/pnfs/pic.es/data</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
RESTRICTED_PATH=pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Read-only Macaroon ===<br />
<br />
Use this for downloading data only.<br />
<br />
The Macaroon will allow:<br />
<br />
* listing directories;<br />
* downloading files.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
=== Upload Macaroon ===<br />
<br />
Use this for uploading data.<br />
<br />
The Macaroon will allow full permissions on the requested path.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
== Troubleshooting ==<br />
<br />
=== SSH authentication is requested repeatedly ===<br />
<br />
This applies to any storage location accessed through SSH/SFTP.<br />
<br />
Check that:<br />
<br />
* You have opened the first SSH connection manually with <code>ssh pic-ssh</code>.<br />
* Browser authentication was completed successfully.<br />
* Your <code>~/.ssh/config</code> contains <code>ControlMaster</code>, <code>ControlPath</code>, and <code>ControlPersist</code>.<br />
* Your rclone remote uses the external SSH command:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
=== OIDC authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using OIDC.<br />
<br />
Check that <code>oidc-agent</code> is running:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
Check that the account is loaded:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
Check that you can retrieve a token manually:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
</pre><br />
<br />
If the token cannot be retrieved, reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
=== Macaroon authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using a Macaroon.<br />
<br />
Check that:<br />
<br />
* The Macaroon was copied completely.<br />
* The Macaroon has not expired.<br />
* The Macaroon was generated for the correct path under <code>/pnfs/pic.es/data</code>.<br />
* The Macaroon has the required permissions, for example read-only or upload permissions.<br />
<br />
If the Macaroon has expired, request or generate a new one.</div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1365Transferring data to/from PIC2026-06-08T11:17:31Z<p>Tallada: /* OIDC configuration example */</p>
<hr />
<div>= How to provide data access to PIC massive storage =<br />
<br />
This page explains how to configure <code>rclone</code> to access PIC storage.<br />
<br />
The configuration depends on the storage path you need to access.<br />
<br />
{| class="wikitable"<br />
|+ Storage access methods<br />
! Storage path<br />
! Protocol<br />
! rclone backend<br />
! Supported authentication<br />
! Example remote name<br />
|-<br />
| rowspan="2" | Paths under <code>/pnfs/pic.es/data</code><br />
| rowspan="2" | WebDAV / HTTPS<br />
| rowspan="2" | <code>webdav</code><br />
| OIDC token<br />
| <code>pic-pnfs-oidc</code><br />
|-<br />
| Macaroon bearer token<br />
| <code>pic-pnfs-macaroon</code><br />
|-<br />
| Any other storage location<br />
| SSH / SFTP<br />
| <code>sftp</code><br />
| SSH with <code>ControlMaster</code><br />
| <code>pic-storage</code><br />
|}<br />
<br />
== Requirements ==<br />
<br />
You need:<br />
<br />
* <code>rclone</code> installed on your client machine.<br />
<br />
For paths under <code>/pnfs/pic.es/data</code>, you also need one of the following:<br />
<br />
* an OIDC token setup using <code>oidc-agent</code>; or<br />
* a Macaroon token.<br />
<br />
For any other storage location, you also need:<br />
<br />
* SSH access to the PIC host.<br />
* SSH <code>ControlMaster</code> configured on your client machine.<br />
<br />
== Install rclone ==<br />
<br />
You can download the rclone binary directly without installing system packages. For example, on a Linux 64-bit machine:<br />
<br />
<pre><br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
$ unzip rclone-current-linux-amd64.zip<br />
</pre><br />
<br />
Alternatively, on Ubuntu you can install the Debian package:<br />
<br />
<pre><br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
</pre><br />
<br />
Check that rclone works:<br />
<br />
<pre><br />
$ rclone version<br />
</pre><br />
<br />
== Which configuration should I use? ==<br />
<br />
Use the table below to choose the right configuration.<br />
<br />
{| class="wikitable"<br />
|+ Choosing the right configuration<br />
! If you want to access...<br />
! Use this section<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using OIDC authentication<br />
| [[#Configure rclone for PNFS data paths using OIDC|Configure rclone for PNFS data paths using OIDC]]<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using a Macaroon token<br />
| [[#Configure rclone for PNFS data paths using a Macaroon|Configure rclone for PNFS data paths using a Macaroon]]<br />
|-<br />
| Any other storage location<br />
| [[#Configure rclone for any other storage location|Configure rclone for any other storage location]]<br />
|}<br />
<br />
== Configure rclone for PNFS data paths ==<br />
<br />
Paths under <code>/pnfs/pic.es/data</code> are accessed using WebDAV over HTTPS.<br />
<br />
For these paths, configure rclone with the <code>webdav</code> backend.<br />
<br />
Use one of the following authentication methods:<br />
<br />
* OIDC token, retrieved automatically with <code>oidc-token</code>.<br />
* Macaroon token, pasted into rclone as a bearer token.<br />
<br />
Do not configure username/password authentication unless PIC explicitly instructs you to do so.<br />
<br />
== Configure rclone for PNFS data paths using OIDC ==<br />
<br />
Use this option if you have been instructed to authenticate with OIDC.<br />
<br />
=== OIDC requirements ===<br />
<br />
Before configuring rclone, make sure that:<br />
<br />
* <code>oidc-agent</code> is available.<br />
* You have configured an OIDC account, for example <code>pic-pnfs</code>.<br />
* The following command returns a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command does not work, see [[#Configuring oidc-agent for OIDC tokens|Configuring oidc-agent for OIDC tokens]].<br />
<br />
=== Configure the WebDAV remote using OIDC ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-oidc<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV base path (https://webdav.pic.es), if desired, you can also enter a subdirectory (https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE).<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
When asked for a bearer token, leave it empty:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token><br />
</pre><br />
<br />
Edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> y<br />
</pre><br />
<br />
Set <code>bearer_token_command</code>:<br />
<br />
<pre><br />
Option bearer_token_command.<br />
Command to run to get a bearer token.<br />
bearer_token_command> oidc-token pic-pnfs<br />
</pre><br />
<br />
Accept the remaining advanced options by pressing <code>ENTER</code>, unless PIC instructed you to change them.<br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es<br />
- vendor: other<br />
- bearer_token_command: oidc-token pic-pnfs<br />
<br />
Keep this "pic-pnfs-oidc" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== OIDC configuration example ===<br />
<br />
The resulting rclone configuration at ~/.config/rclone/rclone.conf should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-oidc]<br />
type = webdav<br />
url = https://webdav.pic.es<br />
vendor = other<br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
=== OIDC usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-oidc:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for PNFS data paths using a Macaroon ==<br />
<br />
Use this option if you have been given a Macaroon token.<br />
<br />
The Macaroon is used as a WebDAV bearer token.<br />
<br />
=== Configure the WebDAV remote using a Macaroon ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-macaroon<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV endpoint provided by PIC.<br />
<br />
For a path under <code>/pnfs/pic.es/data</code>, the URL will normally point to that path or to a subdirectory under it.<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
Paste the Macaroon token as the bearer token:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
You normally do not need to edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> n<br />
</pre><br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- bearer_token: *** ENCRYPTED ***<br />
<br />
Keep this "pic-pnfs-macaroon" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== Macaroon configuration example ===<br />
<br />
The resulting rclone configuration should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-macaroon]<br />
type = webdav<br />
url = https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
vendor = other<br />
bearer_token = YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
=== Macaroon usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-macaroon:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for any other storage location ==<br />
<br />
Use this section for storage paths that are not under <code>/pnfs/pic.es/data</code>.<br />
<br />
These locations are accessed using SSH/SFTP.<br />
<br />
Because SSH authentication may require browser-based authentication, you should configure SSH <code>ControlMaster</code>. This allows rclone to reuse an already-authenticated SSH connection.<br />
<br />
=== Configure SSH ControlMaster ===<br />
<br />
On your client machine, edit:<br />
<br />
<pre><br />
~/.ssh/config<br />
</pre><br />
<br />
Add an entry for the PIC SSH host. For example:<br />
<br />
<pre><br />
Host pic-ssh<br />
HostName ui04.pic.es<br />
User YOUR_PIC_USERNAME<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Replace:<br />
<br />
* <code>ui04.pic.es</code> with the SSH host provided by PIC, if different.<br />
* <code>YOUR_PIC_USERNAME</code> with your PIC username.<br />
<br />
Instead of restricting this configuration to <code>pic-ssh</code>, you can enable it for all SSH connections:<br />
<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
However, using a specific host entry is usually clearer and safer.<br />
<br />
=== Open the first SSH connection ===<br />
<br />
Before using rclone, open one SSH connection manually:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
</pre><br />
<br />
You may see a message similar to:<br />
<br />
<pre><br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Open the URL in your browser, complete the authentication, then return to the terminal and press <code>ENTER</code>.<br />
<br />
After this first authentication, subsequent SSH connections to the same host should reuse the existing <code>ControlMaster</code> session.<br />
<br />
Test this with:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205<br />
[tallada@ui04 ~]$<br />
</pre><br />
<br />
You can close this shell after testing.<br />
<br />
=== Configure the SFTP rclone remote ===<br />
<br />
Other storage locations use the rclone <code>sftp</code> backend.<br />
<br />
Because we want rclone to reuse the SSH <code>ControlMaster</code> connection, configure the remote to use the external <code>ssh</code> command.<br />
<br />
First, locate your rclone configuration file:<br />
<br />
<pre><br />
$ rclone config file<br />
</pre><br />
<br />
The command prints something like:<br />
<br />
<pre><br />
Configuration file is stored at:<br />
/home/YOUR_LOCAL_USER/.config/rclone/rclone.conf<br />
</pre><br />
<br />
Edit that file and add:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
The important line is:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
This tells rclone to use your system SSH command, which can reuse the SSH <code>ControlMaster</code> connection.<br />
<br />
=== SFTP configuration example ===<br />
<br />
Example rclone configuration:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
=== SFTP usage examples ===<br />
<br />
List the remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:<br />
</pre><br />
<br />
List a specific storage path:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Example using a concrete absolute path:<br />
<br />
<pre><br />
$ rclone copy ./results pic-storage:/storage/projects/MY_PROJECT/results<br />
</pre><br />
<br />
== General rclone usage ==<br />
<br />
The same basic rclone commands can be used with all configured remotes.<br />
<br />
Replace <code>REMOTE_NAME</code> with one of the remotes you configured, for example:<br />
<br />
* <code>pic-pnfs-oidc</code><br />
* <code>pic-pnfs-macaroon</code><br />
* <code>pic-storage</code><br />
<br />
=== List a remote directory ===<br />
<br />
<pre><br />
$ rclone lsd REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
$ rclone lsd pic-pnfs-macaroon:<br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== List files recursively ===<br />
<br />
<pre><br />
$ rclone ls REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Download a remote directory ===<br />
<br />
<pre><br />
$ rclone copy REMOTE_NAME:PATH LOCAL_PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
=== Upload a local directory ===<br />
<br />
<pre><br />
$ rclone copy LOCAL_DIR REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Recommended upload flags ===<br />
<br />
For uploads, we recommend:<br />
<br />
<pre><br />
--check-first -P --stats-one-line --transfers N_TRANSFERS --size-only<br />
</pre><br />
<br />
For many small files, <code>N_TRANSFERS</code> can be up to <code>350</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only<br />
</pre><br />
<br />
If uploading into directories with many files, for example more than 1000 files, also use:<br />
<br />
<pre><br />
--no-traverse<br />
</pre><br />
<br />
If uploading files larger than 200 MB, also use:<br />
<br />
<pre><br />
--multi-thread-streams 1<br />
</pre><br />
<br />
If uploading very large files, for example larger than 10 GB, also use:<br />
<br />
<pre><br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to a path under <code>/pnfs/pic.es/data</code>:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to another storage location:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-storage:/PATH_TO_YOUR_STORAGE_SPACE \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
For more information, see the rclone documentation:<br />
<br />
<pre><br />
https://rclone.org/docs/<br />
</pre><br />
<br />
== Configuring oidc-agent for OIDC tokens ==<br />
<br />
This section is only needed if you use OIDC authentication for paths under <code>/pnfs/pic.es/data</code>.<br />
<br />
Make sure <code>oidc-agent</code> is available.<br />
<br />
=== Load oidc-agent ===<br />
<br />
Initialize <code>oidc-agent</code> in the terminal session:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
If the account is already configured, load it with:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
=== Configure the OIDC account ===<br />
<br />
This step only needs to be done once.<br />
<br />
You need an updated version of <code>oidc-agent</code>, version greater than <code>5.0.0</code>.<br />
<br />
Ask your PIC contact for the client secret and replace <code>XXXXXXXXXXXXXXXXXX</code> below.<br />
<br />
Configure a <code>pic-pnfs</code> account to retrieve tokens from PIC:<br />
<br />
<pre><br />
$ oidc-gen -m --client-id CLIENT_ID_PROVIDED_BY_PIC \<br />
--client-secret XXXXXXXXXXXXXXXXXX \<br />
--pub --flow=device \<br />
--discovery-endpoint=https://idp.pic.es/realms/PIC/.well-known/openid-configuration \<br />
--scope="openid profile offline_access" \<br />
--redirect-uri=edu.kit.data.oidc-agent:/ pic-pnfs<br />
</pre><br />
<br />
The command will show a URL and a code:<br />
<br />
<pre><br />
No account exists with this short name. Creating new configuration ...<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs':<br />
Confirm encryption password:<br />
Everything setup correctly!<br />
</pre><br />
<br />
Open the URL in your browser, enter the code, authenticate, then return to the terminal.<br />
<br />
You will be asked to enter an encryption password twice. You will need this password when refreshing or reloading the <code>oidc-agent</code> account.<br />
<br />
=== Reauthenticate if the refresh token has expired ===<br />
<br />
If the <code>oidc-agent</code> process is restarted, or if your refresh token expires due to inactivity, you may need to reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
Example output:<br />
<br />
<pre><br />
Enter decryption password for account config 'pic-pnfs':<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs' [***]:<br />
Everything setup correctly!<br />
</pre><br />
<br />
=== Test OIDC token retrieval ===<br />
<br />
After loading and configuring <code>oidc-agent</code>, test that you can retrieve a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command works, the rclone configuration using:<br />
<br />
<pre><br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
should also work.<br />
<br />
== Obtaining a Macaroon for /pnfs/pic.es/data ==<br />
<br />
This section is intended for contacts or administrators who need to generate Macaroons for users.<br />
<br />
Macaroons are valid for up to 7 days.<br />
<br />
The restricted path should be under <code>/pnfs/pic.es/data</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
RESTRICTED_PATH=pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Read-only Macaroon ===<br />
<br />
Use this for downloading data only.<br />
<br />
The Macaroon will allow:<br />
<br />
* listing directories;<br />
* downloading files.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
=== Upload Macaroon ===<br />
<br />
Use this for uploading data.<br />
<br />
The Macaroon will allow full permissions on the requested path.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
== Troubleshooting ==<br />
<br />
=== SSH authentication is requested repeatedly ===<br />
<br />
This applies to any storage location accessed through SSH/SFTP.<br />
<br />
Check that:<br />
<br />
* You have opened the first SSH connection manually with <code>ssh pic-ssh</code>.<br />
* Browser authentication was completed successfully.<br />
* Your <code>~/.ssh/config</code> contains <code>ControlMaster</code>, <code>ControlPath</code>, and <code>ControlPersist</code>.<br />
* Your rclone remote uses the external SSH command:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
=== OIDC authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using OIDC.<br />
<br />
Check that <code>oidc-agent</code> is running:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
Check that the account is loaded:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
Check that you can retrieve a token manually:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
</pre><br />
<br />
If the token cannot be retrieved, reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
=== Macaroon authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using a Macaroon.<br />
<br />
Check that:<br />
<br />
* The Macaroon was copied completely.<br />
* The Macaroon has not expired.<br />
* The Macaroon was generated for the correct path under <code>/pnfs/pic.es/data</code>.<br />
* The Macaroon has the required permissions, for example read-only or upload permissions.<br />
<br />
If the Macaroon has expired, request or generate a new one.</div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1364Transferring data to/from PIC2026-06-08T11:17:10Z<p>Tallada: /* OIDC configuration example */</p>
<hr />
<div>= How to provide data access to PIC massive storage =<br />
<br />
This page explains how to configure <code>rclone</code> to access PIC storage.<br />
<br />
The configuration depends on the storage path you need to access.<br />
<br />
{| class="wikitable"<br />
|+ Storage access methods<br />
! Storage path<br />
! Protocol<br />
! rclone backend<br />
! Supported authentication<br />
! Example remote name<br />
|-<br />
| rowspan="2" | Paths under <code>/pnfs/pic.es/data</code><br />
| rowspan="2" | WebDAV / HTTPS<br />
| rowspan="2" | <code>webdav</code><br />
| OIDC token<br />
| <code>pic-pnfs-oidc</code><br />
|-<br />
| Macaroon bearer token<br />
| <code>pic-pnfs-macaroon</code><br />
|-<br />
| Any other storage location<br />
| SSH / SFTP<br />
| <code>sftp</code><br />
| SSH with <code>ControlMaster</code><br />
| <code>pic-storage</code><br />
|}<br />
<br />
== Requirements ==<br />
<br />
You need:<br />
<br />
* <code>rclone</code> installed on your client machine.<br />
<br />
For paths under <code>/pnfs/pic.es/data</code>, you also need one of the following:<br />
<br />
* an OIDC token setup using <code>oidc-agent</code>; or<br />
* a Macaroon token.<br />
<br />
For any other storage location, you also need:<br />
<br />
* SSH access to the PIC host.<br />
* SSH <code>ControlMaster</code> configured on your client machine.<br />
<br />
== Install rclone ==<br />
<br />
You can download the rclone binary directly without installing system packages. For example, on a Linux 64-bit machine:<br />
<br />
<pre><br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
$ unzip rclone-current-linux-amd64.zip<br />
</pre><br />
<br />
Alternatively, on Ubuntu you can install the Debian package:<br />
<br />
<pre><br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
</pre><br />
<br />
Check that rclone works:<br />
<br />
<pre><br />
$ rclone version<br />
</pre><br />
<br />
== Which configuration should I use? ==<br />
<br />
Use the table below to choose the right configuration.<br />
<br />
{| class="wikitable"<br />
|+ Choosing the right configuration<br />
! If you want to access...<br />
! Use this section<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using OIDC authentication<br />
| [[#Configure rclone for PNFS data paths using OIDC|Configure rclone for PNFS data paths using OIDC]]<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using a Macaroon token<br />
| [[#Configure rclone for PNFS data paths using a Macaroon|Configure rclone for PNFS data paths using a Macaroon]]<br />
|-<br />
| Any other storage location<br />
| [[#Configure rclone for any other storage location|Configure rclone for any other storage location]]<br />
|}<br />
<br />
== Configure rclone for PNFS data paths ==<br />
<br />
Paths under <code>/pnfs/pic.es/data</code> are accessed using WebDAV over HTTPS.<br />
<br />
For these paths, configure rclone with the <code>webdav</code> backend.<br />
<br />
Use one of the following authentication methods:<br />
<br />
* OIDC token, retrieved automatically with <code>oidc-token</code>.<br />
* Macaroon token, pasted into rclone as a bearer token.<br />
<br />
Do not configure username/password authentication unless PIC explicitly instructs you to do so.<br />
<br />
== Configure rclone for PNFS data paths using OIDC ==<br />
<br />
Use this option if you have been instructed to authenticate with OIDC.<br />
<br />
=== OIDC requirements ===<br />
<br />
Before configuring rclone, make sure that:<br />
<br />
* <code>oidc-agent</code> is available.<br />
* You have configured an OIDC account, for example <code>pic-pnfs</code>.<br />
* The following command returns a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command does not work, see [[#Configuring oidc-agent for OIDC tokens|Configuring oidc-agent for OIDC tokens]].<br />
<br />
=== Configure the WebDAV remote using OIDC ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-oidc<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV base path (https://webdav.pic.es), if desired, you can also enter a subdirectory (https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE).<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
When asked for a bearer token, leave it empty:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token><br />
</pre><br />
<br />
Edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> y<br />
</pre><br />
<br />
Set <code>bearer_token_command</code>:<br />
<br />
<pre><br />
Option bearer_token_command.<br />
Command to run to get a bearer token.<br />
bearer_token_command> oidc-token pic-pnfs<br />
</pre><br />
<br />
Accept the remaining advanced options by pressing <code>ENTER</code>, unless PIC instructed you to change them.<br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es<br />
- vendor: other<br />
- bearer_token_command: oidc-token pic-pnfs<br />
<br />
Keep this "pic-pnfs-oidc" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== OIDC configuration example ===<br />
<br />
The resulting rclone configuration should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-oidc]<br />
type = webdav<br />
url = https://webdav.pic.es<br />
vendor = other<br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
=== OIDC usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-oidc:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for PNFS data paths using a Macaroon ==<br />
<br />
Use this option if you have been given a Macaroon token.<br />
<br />
The Macaroon is used as a WebDAV bearer token.<br />
<br />
=== Configure the WebDAV remote using a Macaroon ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-macaroon<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV endpoint provided by PIC.<br />
<br />
For a path under <code>/pnfs/pic.es/data</code>, the URL will normally point to that path or to a subdirectory under it.<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
Paste the Macaroon token as the bearer token:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
You normally do not need to edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> n<br />
</pre><br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- bearer_token: *** ENCRYPTED ***<br />
<br />
Keep this "pic-pnfs-macaroon" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== Macaroon configuration example ===<br />
<br />
The resulting rclone configuration should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-macaroon]<br />
type = webdav<br />
url = https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
vendor = other<br />
bearer_token = YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
=== Macaroon usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-macaroon:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for any other storage location ==<br />
<br />
Use this section for storage paths that are not under <code>/pnfs/pic.es/data</code>.<br />
<br />
These locations are accessed using SSH/SFTP.<br />
<br />
Because SSH authentication may require browser-based authentication, you should configure SSH <code>ControlMaster</code>. This allows rclone to reuse an already-authenticated SSH connection.<br />
<br />
=== Configure SSH ControlMaster ===<br />
<br />
On your client machine, edit:<br />
<br />
<pre><br />
~/.ssh/config<br />
</pre><br />
<br />
Add an entry for the PIC SSH host. For example:<br />
<br />
<pre><br />
Host pic-ssh<br />
HostName ui04.pic.es<br />
User YOUR_PIC_USERNAME<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Replace:<br />
<br />
* <code>ui04.pic.es</code> with the SSH host provided by PIC, if different.<br />
* <code>YOUR_PIC_USERNAME</code> with your PIC username.<br />
<br />
Instead of restricting this configuration to <code>pic-ssh</code>, you can enable it for all SSH connections:<br />
<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
However, using a specific host entry is usually clearer and safer.<br />
<br />
=== Open the first SSH connection ===<br />
<br />
Before using rclone, open one SSH connection manually:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
</pre><br />
<br />
You may see a message similar to:<br />
<br />
<pre><br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Open the URL in your browser, complete the authentication, then return to the terminal and press <code>ENTER</code>.<br />
<br />
After this first authentication, subsequent SSH connections to the same host should reuse the existing <code>ControlMaster</code> session.<br />
<br />
Test this with:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205<br />
[tallada@ui04 ~]$<br />
</pre><br />
<br />
You can close this shell after testing.<br />
<br />
=== Configure the SFTP rclone remote ===<br />
<br />
Other storage locations use the rclone <code>sftp</code> backend.<br />
<br />
Because we want rclone to reuse the SSH <code>ControlMaster</code> connection, configure the remote to use the external <code>ssh</code> command.<br />
<br />
First, locate your rclone configuration file:<br />
<br />
<pre><br />
$ rclone config file<br />
</pre><br />
<br />
The command prints something like:<br />
<br />
<pre><br />
Configuration file is stored at:<br />
/home/YOUR_LOCAL_USER/.config/rclone/rclone.conf<br />
</pre><br />
<br />
Edit that file and add:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
The important line is:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
This tells rclone to use your system SSH command, which can reuse the SSH <code>ControlMaster</code> connection.<br />
<br />
=== SFTP configuration example ===<br />
<br />
Example rclone configuration:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
=== SFTP usage examples ===<br />
<br />
List the remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:<br />
</pre><br />
<br />
List a specific storage path:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Example using a concrete absolute path:<br />
<br />
<pre><br />
$ rclone copy ./results pic-storage:/storage/projects/MY_PROJECT/results<br />
</pre><br />
<br />
== General rclone usage ==<br />
<br />
The same basic rclone commands can be used with all configured remotes.<br />
<br />
Replace <code>REMOTE_NAME</code> with one of the remotes you configured, for example:<br />
<br />
* <code>pic-pnfs-oidc</code><br />
* <code>pic-pnfs-macaroon</code><br />
* <code>pic-storage</code><br />
<br />
=== List a remote directory ===<br />
<br />
<pre><br />
$ rclone lsd REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
$ rclone lsd pic-pnfs-macaroon:<br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== List files recursively ===<br />
<br />
<pre><br />
$ rclone ls REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Download a remote directory ===<br />
<br />
<pre><br />
$ rclone copy REMOTE_NAME:PATH LOCAL_PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
=== Upload a local directory ===<br />
<br />
<pre><br />
$ rclone copy LOCAL_DIR REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Recommended upload flags ===<br />
<br />
For uploads, we recommend:<br />
<br />
<pre><br />
--check-first -P --stats-one-line --transfers N_TRANSFERS --size-only<br />
</pre><br />
<br />
For many small files, <code>N_TRANSFERS</code> can be up to <code>350</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only<br />
</pre><br />
<br />
If uploading into directories with many files, for example more than 1000 files, also use:<br />
<br />
<pre><br />
--no-traverse<br />
</pre><br />
<br />
If uploading files larger than 200 MB, also use:<br />
<br />
<pre><br />
--multi-thread-streams 1<br />
</pre><br />
<br />
If uploading very large files, for example larger than 10 GB, also use:<br />
<br />
<pre><br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to a path under <code>/pnfs/pic.es/data</code>:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to another storage location:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-storage:/PATH_TO_YOUR_STORAGE_SPACE \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
For more information, see the rclone documentation:<br />
<br />
<pre><br />
https://rclone.org/docs/<br />
</pre><br />
<br />
== Configuring oidc-agent for OIDC tokens ==<br />
<br />
This section is only needed if you use OIDC authentication for paths under <code>/pnfs/pic.es/data</code>.<br />
<br />
Make sure <code>oidc-agent</code> is available.<br />
<br />
=== Load oidc-agent ===<br />
<br />
Initialize <code>oidc-agent</code> in the terminal session:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
If the account is already configured, load it with:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
=== Configure the OIDC account ===<br />
<br />
This step only needs to be done once.<br />
<br />
You need an updated version of <code>oidc-agent</code>, version greater than <code>5.0.0</code>.<br />
<br />
Ask your PIC contact for the client secret and replace <code>XXXXXXXXXXXXXXXXXX</code> below.<br />
<br />
Configure a <code>pic-pnfs</code> account to retrieve tokens from PIC:<br />
<br />
<pre><br />
$ oidc-gen -m --client-id CLIENT_ID_PROVIDED_BY_PIC \<br />
--client-secret XXXXXXXXXXXXXXXXXX \<br />
--pub --flow=device \<br />
--discovery-endpoint=https://idp.pic.es/realms/PIC/.well-known/openid-configuration \<br />
--scope="openid profile offline_access" \<br />
--redirect-uri=edu.kit.data.oidc-agent:/ pic-pnfs<br />
</pre><br />
<br />
The command will show a URL and a code:<br />
<br />
<pre><br />
No account exists with this short name. Creating new configuration ...<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs':<br />
Confirm encryption password:<br />
Everything setup correctly!<br />
</pre><br />
<br />
Open the URL in your browser, enter the code, authenticate, then return to the terminal.<br />
<br />
You will be asked to enter an encryption password twice. You will need this password when refreshing or reloading the <code>oidc-agent</code> account.<br />
<br />
=== Reauthenticate if the refresh token has expired ===<br />
<br />
If the <code>oidc-agent</code> process is restarted, or if your refresh token expires due to inactivity, you may need to reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
Example output:<br />
<br />
<pre><br />
Enter decryption password for account config 'pic-pnfs':<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs' [***]:<br />
Everything setup correctly!<br />
</pre><br />
<br />
=== Test OIDC token retrieval ===<br />
<br />
After loading and configuring <code>oidc-agent</code>, test that you can retrieve a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command works, the rclone configuration using:<br />
<br />
<pre><br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
should also work.<br />
<br />
== Obtaining a Macaroon for /pnfs/pic.es/data ==<br />
<br />
This section is intended for contacts or administrators who need to generate Macaroons for users.<br />
<br />
Macaroons are valid for up to 7 days.<br />
<br />
The restricted path should be under <code>/pnfs/pic.es/data</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
RESTRICTED_PATH=pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Read-only Macaroon ===<br />
<br />
Use this for downloading data only.<br />
<br />
The Macaroon will allow:<br />
<br />
* listing directories;<br />
* downloading files.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
=== Upload Macaroon ===<br />
<br />
Use this for uploading data.<br />
<br />
The Macaroon will allow full permissions on the requested path.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
== Troubleshooting ==<br />
<br />
=== SSH authentication is requested repeatedly ===<br />
<br />
This applies to any storage location accessed through SSH/SFTP.<br />
<br />
Check that:<br />
<br />
* You have opened the first SSH connection manually with <code>ssh pic-ssh</code>.<br />
* Browser authentication was completed successfully.<br />
* Your <code>~/.ssh/config</code> contains <code>ControlMaster</code>, <code>ControlPath</code>, and <code>ControlPersist</code>.<br />
* Your rclone remote uses the external SSH command:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
=== OIDC authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using OIDC.<br />
<br />
Check that <code>oidc-agent</code> is running:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
Check that the account is loaded:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
Check that you can retrieve a token manually:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
</pre><br />
<br />
If the token cannot be retrieved, reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
=== Macaroon authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using a Macaroon.<br />
<br />
Check that:<br />
<br />
* The Macaroon was copied completely.<br />
* The Macaroon has not expired.<br />
* The Macaroon was generated for the correct path under <code>/pnfs/pic.es/data</code>.<br />
* The Macaroon has the required permissions, for example read-only or upload permissions.<br />
<br />
If the Macaroon has expired, request or generate a new one.</div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1363Transferring data to/from PIC2026-06-08T11:16:37Z<p>Tallada: /* Configure the WebDAV remote using OIDC */</p>
<hr />
<div>= How to provide data access to PIC massive storage =<br />
<br />
This page explains how to configure <code>rclone</code> to access PIC storage.<br />
<br />
The configuration depends on the storage path you need to access.<br />
<br />
{| class="wikitable"<br />
|+ Storage access methods<br />
! Storage path<br />
! Protocol<br />
! rclone backend<br />
! Supported authentication<br />
! Example remote name<br />
|-<br />
| rowspan="2" | Paths under <code>/pnfs/pic.es/data</code><br />
| rowspan="2" | WebDAV / HTTPS<br />
| rowspan="2" | <code>webdav</code><br />
| OIDC token<br />
| <code>pic-pnfs-oidc</code><br />
|-<br />
| Macaroon bearer token<br />
| <code>pic-pnfs-macaroon</code><br />
|-<br />
| Any other storage location<br />
| SSH / SFTP<br />
| <code>sftp</code><br />
| SSH with <code>ControlMaster</code><br />
| <code>pic-storage</code><br />
|}<br />
<br />
== Requirements ==<br />
<br />
You need:<br />
<br />
* <code>rclone</code> installed on your client machine.<br />
<br />
For paths under <code>/pnfs/pic.es/data</code>, you also need one of the following:<br />
<br />
* an OIDC token setup using <code>oidc-agent</code>; or<br />
* a Macaroon token.<br />
<br />
For any other storage location, you also need:<br />
<br />
* SSH access to the PIC host.<br />
* SSH <code>ControlMaster</code> configured on your client machine.<br />
<br />
== Install rclone ==<br />
<br />
You can download the rclone binary directly without installing system packages. For example, on a Linux 64-bit machine:<br />
<br />
<pre><br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
$ unzip rclone-current-linux-amd64.zip<br />
</pre><br />
<br />
Alternatively, on Ubuntu you can install the Debian package:<br />
<br />
<pre><br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
</pre><br />
<br />
Check that rclone works:<br />
<br />
<pre><br />
$ rclone version<br />
</pre><br />
<br />
== Which configuration should I use? ==<br />
<br />
Use the table below to choose the right configuration.<br />
<br />
{| class="wikitable"<br />
|+ Choosing the right configuration<br />
! If you want to access...<br />
! Use this section<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using OIDC authentication<br />
| [[#Configure rclone for PNFS data paths using OIDC|Configure rclone for PNFS data paths using OIDC]]<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using a Macaroon token<br />
| [[#Configure rclone for PNFS data paths using a Macaroon|Configure rclone for PNFS data paths using a Macaroon]]<br />
|-<br />
| Any other storage location<br />
| [[#Configure rclone for any other storage location|Configure rclone for any other storage location]]<br />
|}<br />
<br />
== Configure rclone for PNFS data paths ==<br />
<br />
Paths under <code>/pnfs/pic.es/data</code> are accessed using WebDAV over HTTPS.<br />
<br />
For these paths, configure rclone with the <code>webdav</code> backend.<br />
<br />
Use one of the following authentication methods:<br />
<br />
* OIDC token, retrieved automatically with <code>oidc-token</code>.<br />
* Macaroon token, pasted into rclone as a bearer token.<br />
<br />
Do not configure username/password authentication unless PIC explicitly instructs you to do so.<br />
<br />
== Configure rclone for PNFS data paths using OIDC ==<br />
<br />
Use this option if you have been instructed to authenticate with OIDC.<br />
<br />
=== OIDC requirements ===<br />
<br />
Before configuring rclone, make sure that:<br />
<br />
* <code>oidc-agent</code> is available.<br />
* You have configured an OIDC account, for example <code>pic-pnfs</code>.<br />
* The following command returns a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command does not work, see [[#Configuring oidc-agent for OIDC tokens|Configuring oidc-agent for OIDC tokens]].<br />
<br />
=== Configure the WebDAV remote using OIDC ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-oidc<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV base path (https://webdav.pic.es), if desired, you can also enter a subdirectory (https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE).<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
When asked for a bearer token, leave it empty:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token><br />
</pre><br />
<br />
Edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> y<br />
</pre><br />
<br />
Set <code>bearer_token_command</code>:<br />
<br />
<pre><br />
Option bearer_token_command.<br />
Command to run to get a bearer token.<br />
bearer_token_command> oidc-token pic-pnfs<br />
</pre><br />
<br />
Accept the remaining advanced options by pressing <code>ENTER</code>, unless PIC instructed you to change them.<br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es<br />
- vendor: other<br />
- bearer_token_command: oidc-token pic-pnfs<br />
<br />
Keep this "pic-pnfs-oidc" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== OIDC configuration example ===<br />
<br />
The resulting rclone configuration should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-oidc]<br />
type = webdav<br />
url = https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
vendor = other<br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
=== OIDC usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-oidc:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for PNFS data paths using a Macaroon ==<br />
<br />
Use this option if you have been given a Macaroon token.<br />
<br />
The Macaroon is used as a WebDAV bearer token.<br />
<br />
=== Configure the WebDAV remote using a Macaroon ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-macaroon<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV endpoint provided by PIC.<br />
<br />
For a path under <code>/pnfs/pic.es/data</code>, the URL will normally point to that path or to a subdirectory under it.<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
Paste the Macaroon token as the bearer token:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
You normally do not need to edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> n<br />
</pre><br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- bearer_token: *** ENCRYPTED ***<br />
<br />
Keep this "pic-pnfs-macaroon" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== Macaroon configuration example ===<br />
<br />
The resulting rclone configuration should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-macaroon]<br />
type = webdav<br />
url = https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
vendor = other<br />
bearer_token = YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
=== Macaroon usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-macaroon:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for any other storage location ==<br />
<br />
Use this section for storage paths that are not under <code>/pnfs/pic.es/data</code>.<br />
<br />
These locations are accessed using SSH/SFTP.<br />
<br />
Because SSH authentication may require browser-based authentication, you should configure SSH <code>ControlMaster</code>. This allows rclone to reuse an already-authenticated SSH connection.<br />
<br />
=== Configure SSH ControlMaster ===<br />
<br />
On your client machine, edit:<br />
<br />
<pre><br />
~/.ssh/config<br />
</pre><br />
<br />
Add an entry for the PIC SSH host. For example:<br />
<br />
<pre><br />
Host pic-ssh<br />
HostName ui04.pic.es<br />
User YOUR_PIC_USERNAME<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Replace:<br />
<br />
* <code>ui04.pic.es</code> with the SSH host provided by PIC, if different.<br />
* <code>YOUR_PIC_USERNAME</code> with your PIC username.<br />
<br />
Instead of restricting this configuration to <code>pic-ssh</code>, you can enable it for all SSH connections:<br />
<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
However, using a specific host entry is usually clearer and safer.<br />
<br />
=== Open the first SSH connection ===<br />
<br />
Before using rclone, open one SSH connection manually:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
</pre><br />
<br />
You may see a message similar to:<br />
<br />
<pre><br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Open the URL in your browser, complete the authentication, then return to the terminal and press <code>ENTER</code>.<br />
<br />
After this first authentication, subsequent SSH connections to the same host should reuse the existing <code>ControlMaster</code> session.<br />
<br />
Test this with:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205<br />
[tallada@ui04 ~]$<br />
</pre><br />
<br />
You can close this shell after testing.<br />
<br />
=== Configure the SFTP rclone remote ===<br />
<br />
Other storage locations use the rclone <code>sftp</code> backend.<br />
<br />
Because we want rclone to reuse the SSH <code>ControlMaster</code> connection, configure the remote to use the external <code>ssh</code> command.<br />
<br />
First, locate your rclone configuration file:<br />
<br />
<pre><br />
$ rclone config file<br />
</pre><br />
<br />
The command prints something like:<br />
<br />
<pre><br />
Configuration file is stored at:<br />
/home/YOUR_LOCAL_USER/.config/rclone/rclone.conf<br />
</pre><br />
<br />
Edit that file and add:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
The important line is:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
This tells rclone to use your system SSH command, which can reuse the SSH <code>ControlMaster</code> connection.<br />
<br />
=== SFTP configuration example ===<br />
<br />
Example rclone configuration:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
=== SFTP usage examples ===<br />
<br />
List the remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:<br />
</pre><br />
<br />
List a specific storage path:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Example using a concrete absolute path:<br />
<br />
<pre><br />
$ rclone copy ./results pic-storage:/storage/projects/MY_PROJECT/results<br />
</pre><br />
<br />
== General rclone usage ==<br />
<br />
The same basic rclone commands can be used with all configured remotes.<br />
<br />
Replace <code>REMOTE_NAME</code> with one of the remotes you configured, for example:<br />
<br />
* <code>pic-pnfs-oidc</code><br />
* <code>pic-pnfs-macaroon</code><br />
* <code>pic-storage</code><br />
<br />
=== List a remote directory ===<br />
<br />
<pre><br />
$ rclone lsd REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
$ rclone lsd pic-pnfs-macaroon:<br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== List files recursively ===<br />
<br />
<pre><br />
$ rclone ls REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Download a remote directory ===<br />
<br />
<pre><br />
$ rclone copy REMOTE_NAME:PATH LOCAL_PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
=== Upload a local directory ===<br />
<br />
<pre><br />
$ rclone copy LOCAL_DIR REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Recommended upload flags ===<br />
<br />
For uploads, we recommend:<br />
<br />
<pre><br />
--check-first -P --stats-one-line --transfers N_TRANSFERS --size-only<br />
</pre><br />
<br />
For many small files, <code>N_TRANSFERS</code> can be up to <code>350</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only<br />
</pre><br />
<br />
If uploading into directories with many files, for example more than 1000 files, also use:<br />
<br />
<pre><br />
--no-traverse<br />
</pre><br />
<br />
If uploading files larger than 200 MB, also use:<br />
<br />
<pre><br />
--multi-thread-streams 1<br />
</pre><br />
<br />
If uploading very large files, for example larger than 10 GB, also use:<br />
<br />
<pre><br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to a path under <code>/pnfs/pic.es/data</code>:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to another storage location:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-storage:/PATH_TO_YOUR_STORAGE_SPACE \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
For more information, see the rclone documentation:<br />
<br />
<pre><br />
https://rclone.org/docs/<br />
</pre><br />
<br />
== Configuring oidc-agent for OIDC tokens ==<br />
<br />
This section is only needed if you use OIDC authentication for paths under <code>/pnfs/pic.es/data</code>.<br />
<br />
Make sure <code>oidc-agent</code> is available.<br />
<br />
=== Load oidc-agent ===<br />
<br />
Initialize <code>oidc-agent</code> in the terminal session:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
If the account is already configured, load it with:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
=== Configure the OIDC account ===<br />
<br />
This step only needs to be done once.<br />
<br />
You need an updated version of <code>oidc-agent</code>, version greater than <code>5.0.0</code>.<br />
<br />
Ask your PIC contact for the client secret and replace <code>XXXXXXXXXXXXXXXXXX</code> below.<br />
<br />
Configure a <code>pic-pnfs</code> account to retrieve tokens from PIC:<br />
<br />
<pre><br />
$ oidc-gen -m --client-id CLIENT_ID_PROVIDED_BY_PIC \<br />
--client-secret XXXXXXXXXXXXXXXXXX \<br />
--pub --flow=device \<br />
--discovery-endpoint=https://idp.pic.es/realms/PIC/.well-known/openid-configuration \<br />
--scope="openid profile offline_access" \<br />
--redirect-uri=edu.kit.data.oidc-agent:/ pic-pnfs<br />
</pre><br />
<br />
The command will show a URL and a code:<br />
<br />
<pre><br />
No account exists with this short name. Creating new configuration ...<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs':<br />
Confirm encryption password:<br />
Everything setup correctly!<br />
</pre><br />
<br />
Open the URL in your browser, enter the code, authenticate, then return to the terminal.<br />
<br />
You will be asked to enter an encryption password twice. You will need this password when refreshing or reloading the <code>oidc-agent</code> account.<br />
<br />
=== Reauthenticate if the refresh token has expired ===<br />
<br />
If the <code>oidc-agent</code> process is restarted, or if your refresh token expires due to inactivity, you may need to reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
Example output:<br />
<br />
<pre><br />
Enter decryption password for account config 'pic-pnfs':<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs' [***]:<br />
Everything setup correctly!<br />
</pre><br />
<br />
=== Test OIDC token retrieval ===<br />
<br />
After loading and configuring <code>oidc-agent</code>, test that you can retrieve a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command works, the rclone configuration using:<br />
<br />
<pre><br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
should also work.<br />
<br />
== Obtaining a Macaroon for /pnfs/pic.es/data ==<br />
<br />
This section is intended for contacts or administrators who need to generate Macaroons for users.<br />
<br />
Macaroons are valid for up to 7 days.<br />
<br />
The restricted path should be under <code>/pnfs/pic.es/data</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
RESTRICTED_PATH=pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Read-only Macaroon ===<br />
<br />
Use this for downloading data only.<br />
<br />
The Macaroon will allow:<br />
<br />
* listing directories;<br />
* downloading files.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
=== Upload Macaroon ===<br />
<br />
Use this for uploading data.<br />
<br />
The Macaroon will allow full permissions on the requested path.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
== Troubleshooting ==<br />
<br />
=== SSH authentication is requested repeatedly ===<br />
<br />
This applies to any storage location accessed through SSH/SFTP.<br />
<br />
Check that:<br />
<br />
* You have opened the first SSH connection manually with <code>ssh pic-ssh</code>.<br />
* Browser authentication was completed successfully.<br />
* Your <code>~/.ssh/config</code> contains <code>ControlMaster</code>, <code>ControlPath</code>, and <code>ControlPersist</code>.<br />
* Your rclone remote uses the external SSH command:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
=== OIDC authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using OIDC.<br />
<br />
Check that <code>oidc-agent</code> is running:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
Check that the account is loaded:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
Check that you can retrieve a token manually:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
</pre><br />
<br />
If the token cannot be retrieved, reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
=== Macaroon authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using a Macaroon.<br />
<br />
Check that:<br />
<br />
* The Macaroon was copied completely.<br />
* The Macaroon has not expired.<br />
* The Macaroon was generated for the correct path under <code>/pnfs/pic.es/data</code>.<br />
* The Macaroon has the required permissions, for example read-only or upload permissions.<br />
<br />
If the Macaroon has expired, request or generate a new one.</div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1362Transferring data to/from PIC2026-06-08T11:14:01Z<p>Tallada: /* Requirements */</p>
<hr />
<div>= How to provide data access to PIC massive storage =<br />
<br />
This page explains how to configure <code>rclone</code> to access PIC storage.<br />
<br />
The configuration depends on the storage path you need to access.<br />
<br />
{| class="wikitable"<br />
|+ Storage access methods<br />
! Storage path<br />
! Protocol<br />
! rclone backend<br />
! Supported authentication<br />
! Example remote name<br />
|-<br />
| rowspan="2" | Paths under <code>/pnfs/pic.es/data</code><br />
| rowspan="2" | WebDAV / HTTPS<br />
| rowspan="2" | <code>webdav</code><br />
| OIDC token<br />
| <code>pic-pnfs-oidc</code><br />
|-<br />
| Macaroon bearer token<br />
| <code>pic-pnfs-macaroon</code><br />
|-<br />
| Any other storage location<br />
| SSH / SFTP<br />
| <code>sftp</code><br />
| SSH with <code>ControlMaster</code><br />
| <code>pic-storage</code><br />
|}<br />
<br />
== Requirements ==<br />
<br />
You need:<br />
<br />
* <code>rclone</code> installed on your client machine.<br />
<br />
For paths under <code>/pnfs/pic.es/data</code>, you also need one of the following:<br />
<br />
* an OIDC token setup using <code>oidc-agent</code>; or<br />
* a Macaroon token.<br />
<br />
For any other storage location, you also need:<br />
<br />
* SSH access to the PIC host.<br />
* SSH <code>ControlMaster</code> configured on your client machine.<br />
<br />
== Install rclone ==<br />
<br />
You can download the rclone binary directly without installing system packages. For example, on a Linux 64-bit machine:<br />
<br />
<pre><br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
$ unzip rclone-current-linux-amd64.zip<br />
</pre><br />
<br />
Alternatively, on Ubuntu you can install the Debian package:<br />
<br />
<pre><br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
</pre><br />
<br />
Check that rclone works:<br />
<br />
<pre><br />
$ rclone version<br />
</pre><br />
<br />
== Which configuration should I use? ==<br />
<br />
Use the table below to choose the right configuration.<br />
<br />
{| class="wikitable"<br />
|+ Choosing the right configuration<br />
! If you want to access...<br />
! Use this section<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using OIDC authentication<br />
| [[#Configure rclone for PNFS data paths using OIDC|Configure rclone for PNFS data paths using OIDC]]<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using a Macaroon token<br />
| [[#Configure rclone for PNFS data paths using a Macaroon|Configure rclone for PNFS data paths using a Macaroon]]<br />
|-<br />
| Any other storage location<br />
| [[#Configure rclone for any other storage location|Configure rclone for any other storage location]]<br />
|}<br />
<br />
== Configure rclone for PNFS data paths ==<br />
<br />
Paths under <code>/pnfs/pic.es/data</code> are accessed using WebDAV over HTTPS.<br />
<br />
For these paths, configure rclone with the <code>webdav</code> backend.<br />
<br />
Use one of the following authentication methods:<br />
<br />
* OIDC token, retrieved automatically with <code>oidc-token</code>.<br />
* Macaroon token, pasted into rclone as a bearer token.<br />
<br />
Do not configure username/password authentication unless PIC explicitly instructs you to do so.<br />
<br />
== Configure rclone for PNFS data paths using OIDC ==<br />
<br />
Use this option if you have been instructed to authenticate with OIDC.<br />
<br />
=== OIDC requirements ===<br />
<br />
Before configuring rclone, make sure that:<br />
<br />
* <code>oidc-agent</code> is available.<br />
* You have configured an OIDC account, for example <code>pic-pnfs</code>.<br />
* The following command returns a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command does not work, see [[#Configuring oidc-agent for OIDC tokens|Configuring oidc-agent for OIDC tokens]].<br />
<br />
=== Configure the WebDAV remote using OIDC ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-oidc<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV endpoint provided by PIC.<br />
<br />
For a path under <code>/pnfs/pic.es/data</code>, the URL will normally point to that path or to a subdirectory under it.<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
When asked for a bearer token, leave it empty:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token><br />
</pre><br />
<br />
Edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> y<br />
</pre><br />
<br />
Set <code>bearer_token_command</code>:<br />
<br />
<pre><br />
Option bearer_token_command.<br />
Command to run to get a bearer token.<br />
bearer_token_command> oidc-token pic-pnfs<br />
</pre><br />
<br />
Accept the remaining advanced options by pressing <code>ENTER</code>, unless PIC instructed you to change them.<br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- bearer_token_command: oidc-token pic-pnfs<br />
<br />
Keep this "pic-pnfs-oidc" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== OIDC configuration example ===<br />
<br />
The resulting rclone configuration should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-oidc]<br />
type = webdav<br />
url = https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
vendor = other<br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
=== OIDC usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-oidc:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for PNFS data paths using a Macaroon ==<br />
<br />
Use this option if you have been given a Macaroon token.<br />
<br />
The Macaroon is used as a WebDAV bearer token.<br />
<br />
=== Configure the WebDAV remote using a Macaroon ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-macaroon<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV endpoint provided by PIC.<br />
<br />
For a path under <code>/pnfs/pic.es/data</code>, the URL will normally point to that path or to a subdirectory under it.<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
Paste the Macaroon token as the bearer token:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
You normally do not need to edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> n<br />
</pre><br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- bearer_token: *** ENCRYPTED ***<br />
<br />
Keep this "pic-pnfs-macaroon" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== Macaroon configuration example ===<br />
<br />
The resulting rclone configuration should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-macaroon]<br />
type = webdav<br />
url = https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
vendor = other<br />
bearer_token = YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
=== Macaroon usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-macaroon:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for any other storage location ==<br />
<br />
Use this section for storage paths that are not under <code>/pnfs/pic.es/data</code>.<br />
<br />
These locations are accessed using SSH/SFTP.<br />
<br />
Because SSH authentication may require browser-based authentication, you should configure SSH <code>ControlMaster</code>. This allows rclone to reuse an already-authenticated SSH connection.<br />
<br />
=== Configure SSH ControlMaster ===<br />
<br />
On your client machine, edit:<br />
<br />
<pre><br />
~/.ssh/config<br />
</pre><br />
<br />
Add an entry for the PIC SSH host. For example:<br />
<br />
<pre><br />
Host pic-ssh<br />
HostName ui04.pic.es<br />
User YOUR_PIC_USERNAME<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Replace:<br />
<br />
* <code>ui04.pic.es</code> with the SSH host provided by PIC, if different.<br />
* <code>YOUR_PIC_USERNAME</code> with your PIC username.<br />
<br />
Instead of restricting this configuration to <code>pic-ssh</code>, you can enable it for all SSH connections:<br />
<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
However, using a specific host entry is usually clearer and safer.<br />
<br />
=== Open the first SSH connection ===<br />
<br />
Before using rclone, open one SSH connection manually:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
</pre><br />
<br />
You may see a message similar to:<br />
<br />
<pre><br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Open the URL in your browser, complete the authentication, then return to the terminal and press <code>ENTER</code>.<br />
<br />
After this first authentication, subsequent SSH connections to the same host should reuse the existing <code>ControlMaster</code> session.<br />
<br />
Test this with:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205<br />
[tallada@ui04 ~]$<br />
</pre><br />
<br />
You can close this shell after testing.<br />
<br />
=== Configure the SFTP rclone remote ===<br />
<br />
Other storage locations use the rclone <code>sftp</code> backend.<br />
<br />
Because we want rclone to reuse the SSH <code>ControlMaster</code> connection, configure the remote to use the external <code>ssh</code> command.<br />
<br />
First, locate your rclone configuration file:<br />
<br />
<pre><br />
$ rclone config file<br />
</pre><br />
<br />
The command prints something like:<br />
<br />
<pre><br />
Configuration file is stored at:<br />
/home/YOUR_LOCAL_USER/.config/rclone/rclone.conf<br />
</pre><br />
<br />
Edit that file and add:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
The important line is:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
This tells rclone to use your system SSH command, which can reuse the SSH <code>ControlMaster</code> connection.<br />
<br />
=== SFTP configuration example ===<br />
<br />
Example rclone configuration:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
=== SFTP usage examples ===<br />
<br />
List the remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:<br />
</pre><br />
<br />
List a specific storage path:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Example using a concrete absolute path:<br />
<br />
<pre><br />
$ rclone copy ./results pic-storage:/storage/projects/MY_PROJECT/results<br />
</pre><br />
<br />
== General rclone usage ==<br />
<br />
The same basic rclone commands can be used with all configured remotes.<br />
<br />
Replace <code>REMOTE_NAME</code> with one of the remotes you configured, for example:<br />
<br />
* <code>pic-pnfs-oidc</code><br />
* <code>pic-pnfs-macaroon</code><br />
* <code>pic-storage</code><br />
<br />
=== List a remote directory ===<br />
<br />
<pre><br />
$ rclone lsd REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
$ rclone lsd pic-pnfs-macaroon:<br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== List files recursively ===<br />
<br />
<pre><br />
$ rclone ls REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Download a remote directory ===<br />
<br />
<pre><br />
$ rclone copy REMOTE_NAME:PATH LOCAL_PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
=== Upload a local directory ===<br />
<br />
<pre><br />
$ rclone copy LOCAL_DIR REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Recommended upload flags ===<br />
<br />
For uploads, we recommend:<br />
<br />
<pre><br />
--check-first -P --stats-one-line --transfers N_TRANSFERS --size-only<br />
</pre><br />
<br />
For many small files, <code>N_TRANSFERS</code> can be up to <code>350</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only<br />
</pre><br />
<br />
If uploading into directories with many files, for example more than 1000 files, also use:<br />
<br />
<pre><br />
--no-traverse<br />
</pre><br />
<br />
If uploading files larger than 200 MB, also use:<br />
<br />
<pre><br />
--multi-thread-streams 1<br />
</pre><br />
<br />
If uploading very large files, for example larger than 10 GB, also use:<br />
<br />
<pre><br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to a path under <code>/pnfs/pic.es/data</code>:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to another storage location:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-storage:/PATH_TO_YOUR_STORAGE_SPACE \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
For more information, see the rclone documentation:<br />
<br />
<pre><br />
https://rclone.org/docs/<br />
</pre><br />
<br />
== Configuring oidc-agent for OIDC tokens ==<br />
<br />
This section is only needed if you use OIDC authentication for paths under <code>/pnfs/pic.es/data</code>.<br />
<br />
Make sure <code>oidc-agent</code> is available.<br />
<br />
=== Load oidc-agent ===<br />
<br />
Initialize <code>oidc-agent</code> in the terminal session:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
If the account is already configured, load it with:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
=== Configure the OIDC account ===<br />
<br />
This step only needs to be done once.<br />
<br />
You need an updated version of <code>oidc-agent</code>, version greater than <code>5.0.0</code>.<br />
<br />
Ask your PIC contact for the client secret and replace <code>XXXXXXXXXXXXXXXXXX</code> below.<br />
<br />
Configure a <code>pic-pnfs</code> account to retrieve tokens from PIC:<br />
<br />
<pre><br />
$ oidc-gen -m --client-id CLIENT_ID_PROVIDED_BY_PIC \<br />
--client-secret XXXXXXXXXXXXXXXXXX \<br />
--pub --flow=device \<br />
--discovery-endpoint=https://idp.pic.es/realms/PIC/.well-known/openid-configuration \<br />
--scope="openid profile offline_access" \<br />
--redirect-uri=edu.kit.data.oidc-agent:/ pic-pnfs<br />
</pre><br />
<br />
The command will show a URL and a code:<br />
<br />
<pre><br />
No account exists with this short name. Creating new configuration ...<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs':<br />
Confirm encryption password:<br />
Everything setup correctly!<br />
</pre><br />
<br />
Open the URL in your browser, enter the code, authenticate, then return to the terminal.<br />
<br />
You will be asked to enter an encryption password twice. You will need this password when refreshing or reloading the <code>oidc-agent</code> account.<br />
<br />
=== Reauthenticate if the refresh token has expired ===<br />
<br />
If the <code>oidc-agent</code> process is restarted, or if your refresh token expires due to inactivity, you may need to reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
Example output:<br />
<br />
<pre><br />
Enter decryption password for account config 'pic-pnfs':<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs' [***]:<br />
Everything setup correctly!<br />
</pre><br />
<br />
=== Test OIDC token retrieval ===<br />
<br />
After loading and configuring <code>oidc-agent</code>, test that you can retrieve a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command works, the rclone configuration using:<br />
<br />
<pre><br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
should also work.<br />
<br />
== Obtaining a Macaroon for /pnfs/pic.es/data ==<br />
<br />
This section is intended for contacts or administrators who need to generate Macaroons for users.<br />
<br />
Macaroons are valid for up to 7 days.<br />
<br />
The restricted path should be under <code>/pnfs/pic.es/data</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
RESTRICTED_PATH=pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Read-only Macaroon ===<br />
<br />
Use this for downloading data only.<br />
<br />
The Macaroon will allow:<br />
<br />
* listing directories;<br />
* downloading files.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
=== Upload Macaroon ===<br />
<br />
Use this for uploading data.<br />
<br />
The Macaroon will allow full permissions on the requested path.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
== Troubleshooting ==<br />
<br />
=== SSH authentication is requested repeatedly ===<br />
<br />
This applies to any storage location accessed through SSH/SFTP.<br />
<br />
Check that:<br />
<br />
* You have opened the first SSH connection manually with <code>ssh pic-ssh</code>.<br />
* Browser authentication was completed successfully.<br />
* Your <code>~/.ssh/config</code> contains <code>ControlMaster</code>, <code>ControlPath</code>, and <code>ControlPersist</code>.<br />
* Your rclone remote uses the external SSH command:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
=== OIDC authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using OIDC.<br />
<br />
Check that <code>oidc-agent</code> is running:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
Check that the account is loaded:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
Check that you can retrieve a token manually:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
</pre><br />
<br />
If the token cannot be retrieved, reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
=== Macaroon authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using a Macaroon.<br />
<br />
Check that:<br />
<br />
* The Macaroon was copied completely.<br />
* The Macaroon has not expired.<br />
* The Macaroon was generated for the correct path under <code>/pnfs/pic.es/data</code>.<br />
* The Macaroon has the required permissions, for example read-only or upload permissions.<br />
<br />
If the Macaroon has expired, request or generate a new one.</div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1361Transferring data to/from PIC2026-06-08T11:13:15Z<p>Tallada: /* How to provide data access to PIC massive storage */</p>
<hr />
<div>= How to provide data access to PIC massive storage =<br />
<br />
This page explains how to configure <code>rclone</code> to access PIC storage.<br />
<br />
The configuration depends on the storage path you need to access.<br />
<br />
{| class="wikitable"<br />
|+ Storage access methods<br />
! Storage path<br />
! Protocol<br />
! rclone backend<br />
! Supported authentication<br />
! Example remote name<br />
|-<br />
| rowspan="2" | Paths under <code>/pnfs/pic.es/data</code><br />
| rowspan="2" | WebDAV / HTTPS<br />
| rowspan="2" | <code>webdav</code><br />
| OIDC token<br />
| <code>pic-pnfs-oidc</code><br />
|-<br />
| Macaroon bearer token<br />
| <code>pic-pnfs-macaroon</code><br />
|-<br />
| Any other storage location<br />
| SSH / SFTP<br />
| <code>sftp</code><br />
| SSH with <code>ControlMaster</code><br />
| <code>pic-storage</code><br />
|}<br />
<br />
== Requirements ==<br />
<br />
You need:<br />
<br />
* <code>rclone</code> installed on your client machine.<br />
* The storage path or endpoint provided by your PIC contact.<br />
<br />
For paths under <code>/pnfs/pic.es/data</code>, you also need one of the following:<br />
<br />
* an OIDC token setup using <code>oidc-agent</code>; or<br />
* a Macaroon token.<br />
<br />
For any other storage location, you also need:<br />
<br />
* SSH access to the PIC host.<br />
* SSH <code>ControlMaster</code> configured on your client machine.<br />
<br />
== Install rclone ==<br />
<br />
You can download the rclone binary directly without installing system packages. For example, on a Linux 64-bit machine:<br />
<br />
<pre><br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
$ unzip rclone-current-linux-amd64.zip<br />
</pre><br />
<br />
Alternatively, on Ubuntu you can install the Debian package:<br />
<br />
<pre><br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
</pre><br />
<br />
Check that rclone works:<br />
<br />
<pre><br />
$ rclone version<br />
</pre><br />
<br />
== Which configuration should I use? ==<br />
<br />
Use the table below to choose the right configuration.<br />
<br />
{| class="wikitable"<br />
|+ Choosing the right configuration<br />
! If you want to access...<br />
! Use this section<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using OIDC authentication<br />
| [[#Configure rclone for PNFS data paths using OIDC|Configure rclone for PNFS data paths using OIDC]]<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using a Macaroon token<br />
| [[#Configure rclone for PNFS data paths using a Macaroon|Configure rclone for PNFS data paths using a Macaroon]]<br />
|-<br />
| Any other storage location<br />
| [[#Configure rclone for any other storage location|Configure rclone for any other storage location]]<br />
|}<br />
<br />
== Configure rclone for PNFS data paths ==<br />
<br />
Paths under <code>/pnfs/pic.es/data</code> are accessed using WebDAV over HTTPS.<br />
<br />
For these paths, configure rclone with the <code>webdav</code> backend.<br />
<br />
Use one of the following authentication methods:<br />
<br />
* OIDC token, retrieved automatically with <code>oidc-token</code>.<br />
* Macaroon token, pasted into rclone as a bearer token.<br />
<br />
Do not configure username/password authentication unless PIC explicitly instructs you to do so.<br />
<br />
== Configure rclone for PNFS data paths using OIDC ==<br />
<br />
Use this option if you have been instructed to authenticate with OIDC.<br />
<br />
=== OIDC requirements ===<br />
<br />
Before configuring rclone, make sure that:<br />
<br />
* <code>oidc-agent</code> is available.<br />
* You have configured an OIDC account, for example <code>pic-pnfs</code>.<br />
* The following command returns a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command does not work, see [[#Configuring oidc-agent for OIDC tokens|Configuring oidc-agent for OIDC tokens]].<br />
<br />
=== Configure the WebDAV remote using OIDC ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-oidc<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV endpoint provided by PIC.<br />
<br />
For a path under <code>/pnfs/pic.es/data</code>, the URL will normally point to that path or to a subdirectory under it.<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
When asked for a bearer token, leave it empty:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token><br />
</pre><br />
<br />
Edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> y<br />
</pre><br />
<br />
Set <code>bearer_token_command</code>:<br />
<br />
<pre><br />
Option bearer_token_command.<br />
Command to run to get a bearer token.<br />
bearer_token_command> oidc-token pic-pnfs<br />
</pre><br />
<br />
Accept the remaining advanced options by pressing <code>ENTER</code>, unless PIC instructed you to change them.<br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- bearer_token_command: oidc-token pic-pnfs<br />
<br />
Keep this "pic-pnfs-oidc" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== OIDC configuration example ===<br />
<br />
The resulting rclone configuration should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-oidc]<br />
type = webdav<br />
url = https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
vendor = other<br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
=== OIDC usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-oidc:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for PNFS data paths using a Macaroon ==<br />
<br />
Use this option if you have been given a Macaroon token.<br />
<br />
The Macaroon is used as a WebDAV bearer token.<br />
<br />
=== Configure the WebDAV remote using a Macaroon ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-macaroon<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV endpoint provided by PIC.<br />
<br />
For a path under <code>/pnfs/pic.es/data</code>, the URL will normally point to that path or to a subdirectory under it.<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
Paste the Macaroon token as the bearer token:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
You normally do not need to edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> n<br />
</pre><br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- bearer_token: *** ENCRYPTED ***<br />
<br />
Keep this "pic-pnfs-macaroon" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== Macaroon configuration example ===<br />
<br />
The resulting rclone configuration should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-macaroon]<br />
type = webdav<br />
url = https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
vendor = other<br />
bearer_token = YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
=== Macaroon usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-macaroon:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for any other storage location ==<br />
<br />
Use this section for storage paths that are not under <code>/pnfs/pic.es/data</code>.<br />
<br />
These locations are accessed using SSH/SFTP.<br />
<br />
Because SSH authentication may require browser-based authentication, you should configure SSH <code>ControlMaster</code>. This allows rclone to reuse an already-authenticated SSH connection.<br />
<br />
=== Configure SSH ControlMaster ===<br />
<br />
On your client machine, edit:<br />
<br />
<pre><br />
~/.ssh/config<br />
</pre><br />
<br />
Add an entry for the PIC SSH host. For example:<br />
<br />
<pre><br />
Host pic-ssh<br />
HostName ui04.pic.es<br />
User YOUR_PIC_USERNAME<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Replace:<br />
<br />
* <code>ui04.pic.es</code> with the SSH host provided by PIC, if different.<br />
* <code>YOUR_PIC_USERNAME</code> with your PIC username.<br />
<br />
Instead of restricting this configuration to <code>pic-ssh</code>, you can enable it for all SSH connections:<br />
<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
However, using a specific host entry is usually clearer and safer.<br />
<br />
=== Open the first SSH connection ===<br />
<br />
Before using rclone, open one SSH connection manually:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
</pre><br />
<br />
You may see a message similar to:<br />
<br />
<pre><br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Open the URL in your browser, complete the authentication, then return to the terminal and press <code>ENTER</code>.<br />
<br />
After this first authentication, subsequent SSH connections to the same host should reuse the existing <code>ControlMaster</code> session.<br />
<br />
Test this with:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205<br />
[tallada@ui04 ~]$<br />
</pre><br />
<br />
You can close this shell after testing.<br />
<br />
=== Configure the SFTP rclone remote ===<br />
<br />
Other storage locations use the rclone <code>sftp</code> backend.<br />
<br />
Because we want rclone to reuse the SSH <code>ControlMaster</code> connection, configure the remote to use the external <code>ssh</code> command.<br />
<br />
First, locate your rclone configuration file:<br />
<br />
<pre><br />
$ rclone config file<br />
</pre><br />
<br />
The command prints something like:<br />
<br />
<pre><br />
Configuration file is stored at:<br />
/home/YOUR_LOCAL_USER/.config/rclone/rclone.conf<br />
</pre><br />
<br />
Edit that file and add:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
The important line is:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
This tells rclone to use your system SSH command, which can reuse the SSH <code>ControlMaster</code> connection.<br />
<br />
=== SFTP configuration example ===<br />
<br />
Example rclone configuration:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
=== SFTP usage examples ===<br />
<br />
List the remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:<br />
</pre><br />
<br />
List a specific storage path:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Example using a concrete absolute path:<br />
<br />
<pre><br />
$ rclone copy ./results pic-storage:/storage/projects/MY_PROJECT/results<br />
</pre><br />
<br />
== General rclone usage ==<br />
<br />
The same basic rclone commands can be used with all configured remotes.<br />
<br />
Replace <code>REMOTE_NAME</code> with one of the remotes you configured, for example:<br />
<br />
* <code>pic-pnfs-oidc</code><br />
* <code>pic-pnfs-macaroon</code><br />
* <code>pic-storage</code><br />
<br />
=== List a remote directory ===<br />
<br />
<pre><br />
$ rclone lsd REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
$ rclone lsd pic-pnfs-macaroon:<br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== List files recursively ===<br />
<br />
<pre><br />
$ rclone ls REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Download a remote directory ===<br />
<br />
<pre><br />
$ rclone copy REMOTE_NAME:PATH LOCAL_PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
=== Upload a local directory ===<br />
<br />
<pre><br />
$ rclone copy LOCAL_DIR REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Recommended upload flags ===<br />
<br />
For uploads, we recommend:<br />
<br />
<pre><br />
--check-first -P --stats-one-line --transfers N_TRANSFERS --size-only<br />
</pre><br />
<br />
For many small files, <code>N_TRANSFERS</code> can be up to <code>350</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only<br />
</pre><br />
<br />
If uploading into directories with many files, for example more than 1000 files, also use:<br />
<br />
<pre><br />
--no-traverse<br />
</pre><br />
<br />
If uploading files larger than 200 MB, also use:<br />
<br />
<pre><br />
--multi-thread-streams 1<br />
</pre><br />
<br />
If uploading very large files, for example larger than 10 GB, also use:<br />
<br />
<pre><br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to a path under <code>/pnfs/pic.es/data</code>:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to another storage location:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-storage:/PATH_TO_YOUR_STORAGE_SPACE \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
For more information, see the rclone documentation:<br />
<br />
<pre><br />
https://rclone.org/docs/<br />
</pre><br />
<br />
== Configuring oidc-agent for OIDC tokens ==<br />
<br />
This section is only needed if you use OIDC authentication for paths under <code>/pnfs/pic.es/data</code>.<br />
<br />
Make sure <code>oidc-agent</code> is available.<br />
<br />
=== Load oidc-agent ===<br />
<br />
Initialize <code>oidc-agent</code> in the terminal session:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
If the account is already configured, load it with:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
=== Configure the OIDC account ===<br />
<br />
This step only needs to be done once.<br />
<br />
You need an updated version of <code>oidc-agent</code>, version greater than <code>5.0.0</code>.<br />
<br />
Ask your PIC contact for the client secret and replace <code>XXXXXXXXXXXXXXXXXX</code> below.<br />
<br />
Configure a <code>pic-pnfs</code> account to retrieve tokens from PIC:<br />
<br />
<pre><br />
$ oidc-gen -m --client-id CLIENT_ID_PROVIDED_BY_PIC \<br />
--client-secret XXXXXXXXXXXXXXXXXX \<br />
--pub --flow=device \<br />
--discovery-endpoint=https://idp.pic.es/realms/PIC/.well-known/openid-configuration \<br />
--scope="openid profile offline_access" \<br />
--redirect-uri=edu.kit.data.oidc-agent:/ pic-pnfs<br />
</pre><br />
<br />
The command will show a URL and a code:<br />
<br />
<pre><br />
No account exists with this short name. Creating new configuration ...<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs':<br />
Confirm encryption password:<br />
Everything setup correctly!<br />
</pre><br />
<br />
Open the URL in your browser, enter the code, authenticate, then return to the terminal.<br />
<br />
You will be asked to enter an encryption password twice. You will need this password when refreshing or reloading the <code>oidc-agent</code> account.<br />
<br />
=== Reauthenticate if the refresh token has expired ===<br />
<br />
If the <code>oidc-agent</code> process is restarted, or if your refresh token expires due to inactivity, you may need to reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
Example output:<br />
<br />
<pre><br />
Enter decryption password for account config 'pic-pnfs':<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs' [***]:<br />
Everything setup correctly!<br />
</pre><br />
<br />
=== Test OIDC token retrieval ===<br />
<br />
After loading and configuring <code>oidc-agent</code>, test that you can retrieve a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command works, the rclone configuration using:<br />
<br />
<pre><br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
should also work.<br />
<br />
== Obtaining a Macaroon for /pnfs/pic.es/data ==<br />
<br />
This section is intended for contacts or administrators who need to generate Macaroons for users.<br />
<br />
Macaroons are valid for up to 7 days.<br />
<br />
The restricted path should be under <code>/pnfs/pic.es/data</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
RESTRICTED_PATH=pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Read-only Macaroon ===<br />
<br />
Use this for downloading data only.<br />
<br />
The Macaroon will allow:<br />
<br />
* listing directories;<br />
* downloading files.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
=== Upload Macaroon ===<br />
<br />
Use this for uploading data.<br />
<br />
The Macaroon will allow full permissions on the requested path.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
== Troubleshooting ==<br />
<br />
=== SSH authentication is requested repeatedly ===<br />
<br />
This applies to any storage location accessed through SSH/SFTP.<br />
<br />
Check that:<br />
<br />
* You have opened the first SSH connection manually with <code>ssh pic-ssh</code>.<br />
* Browser authentication was completed successfully.<br />
* Your <code>~/.ssh/config</code> contains <code>ControlMaster</code>, <code>ControlPath</code>, and <code>ControlPersist</code>.<br />
* Your rclone remote uses the external SSH command:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
=== OIDC authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using OIDC.<br />
<br />
Check that <code>oidc-agent</code> is running:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
Check that the account is loaded:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
Check that you can retrieve a token manually:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
</pre><br />
<br />
If the token cannot be retrieved, reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
=== Macaroon authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using a Macaroon.<br />
<br />
Check that:<br />
<br />
* The Macaroon was copied completely.<br />
* The Macaroon has not expired.<br />
* The Macaroon was generated for the correct path under <code>/pnfs/pic.es/data</code>.<br />
* The Macaroon has the required permissions, for example read-only or upload permissions.<br />
<br />
If the Macaroon has expired, request or generate a new one.</div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1360Transferring data to/from PIC2026-06-08T11:13:04Z<p>Tallada: </p>
<hr />
<div>= How to provide data access to PIC massive storage =<br />
<br />
This page explains how to configure <code>rclone</code> to access PIC storage.<br />
<br />
The configuration depends on the storage path you need to access.<br />
<br />
{| class="wikitable"<br />
|+ Storage access methods<br />
! Storage path<br />
! Protocol<br />
! rclone backend<br />
! Supported authentication<br />
! Example remote name<br />
|-<br />
| rowspan="2" | Paths under <code>/pnfs/pic.es/data</code><br />
| rowspan="2" | WebDAV / HTTPS<br />
| rowspan="2" | <code>webdav</code><br />
| OIDC token<br />
| <code>pic-pnfs-oidc</code><br />
|-<br />
| Macaroon bearer token<br />
| <code>pic-pnfs-macaroon</code><br />
|-<br />
| Any other storage location<br />
| SSH / SFTP<br />
| <code>sftp</code><br />
| SSH with <code>ControlMaster</code><br />
| <code>pic-storage</code><br />
|}<br />
<br />
<br />
== Requirements ==<br />
<br />
You need:<br />
<br />
* <code>rclone</code> installed on your client machine.<br />
* The storage path or endpoint provided by your PIC contact.<br />
<br />
For paths under <code>/pnfs/pic.es/data</code>, you also need one of the following:<br />
<br />
* an OIDC token setup using <code>oidc-agent</code>; or<br />
* a Macaroon token.<br />
<br />
For any other storage location, you also need:<br />
<br />
* SSH access to the PIC host.<br />
* SSH <code>ControlMaster</code> configured on your client machine.<br />
<br />
== Install rclone ==<br />
<br />
You can download the rclone binary directly without installing system packages. For example, on a Linux 64-bit machine:<br />
<br />
<pre><br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
$ unzip rclone-current-linux-amd64.zip<br />
</pre><br />
<br />
Alternatively, on Ubuntu you can install the Debian package:<br />
<br />
<pre><br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
</pre><br />
<br />
Check that rclone works:<br />
<br />
<pre><br />
$ rclone version<br />
</pre><br />
<br />
== Which configuration should I use? ==<br />
<br />
Use the table below to choose the right configuration.<br />
<br />
{| class="wikitable"<br />
|+ Choosing the right configuration<br />
! If you want to access...<br />
! Use this section<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using OIDC authentication<br />
| [[#Configure rclone for PNFS data paths using OIDC|Configure rclone for PNFS data paths using OIDC]]<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using a Macaroon token<br />
| [[#Configure rclone for PNFS data paths using a Macaroon|Configure rclone for PNFS data paths using a Macaroon]]<br />
|-<br />
| Any other storage location<br />
| [[#Configure rclone for any other storage location|Configure rclone for any other storage location]]<br />
|}<br />
<br />
== Configure rclone for PNFS data paths ==<br />
<br />
Paths under <code>/pnfs/pic.es/data</code> are accessed using WebDAV over HTTPS.<br />
<br />
For these paths, configure rclone with the <code>webdav</code> backend.<br />
<br />
Use one of the following authentication methods:<br />
<br />
* OIDC token, retrieved automatically with <code>oidc-token</code>.<br />
* Macaroon token, pasted into rclone as a bearer token.<br />
<br />
Do not configure username/password authentication unless PIC explicitly instructs you to do so.<br />
<br />
== Configure rclone for PNFS data paths using OIDC ==<br />
<br />
Use this option if you have been instructed to authenticate with OIDC.<br />
<br />
=== OIDC requirements ===<br />
<br />
Before configuring rclone, make sure that:<br />
<br />
* <code>oidc-agent</code> is available.<br />
* You have configured an OIDC account, for example <code>pic-pnfs</code>.<br />
* The following command returns a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command does not work, see [[#Configuring oidc-agent for OIDC tokens|Configuring oidc-agent for OIDC tokens]].<br />
<br />
=== Configure the WebDAV remote using OIDC ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-oidc<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV endpoint provided by PIC.<br />
<br />
For a path under <code>/pnfs/pic.es/data</code>, the URL will normally point to that path or to a subdirectory under it.<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
When asked for a bearer token, leave it empty:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token><br />
</pre><br />
<br />
Edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> y<br />
</pre><br />
<br />
Set <code>bearer_token_command</code>:<br />
<br />
<pre><br />
Option bearer_token_command.<br />
Command to run to get a bearer token.<br />
bearer_token_command> oidc-token pic-pnfs<br />
</pre><br />
<br />
Accept the remaining advanced options by pressing <code>ENTER</code>, unless PIC instructed you to change them.<br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- bearer_token_command: oidc-token pic-pnfs<br />
<br />
Keep this "pic-pnfs-oidc" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== OIDC configuration example ===<br />
<br />
The resulting rclone configuration should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-oidc]<br />
type = webdav<br />
url = https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
vendor = other<br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
=== OIDC usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-oidc:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for PNFS data paths using a Macaroon ==<br />
<br />
Use this option if you have been given a Macaroon token.<br />
<br />
The Macaroon is used as a WebDAV bearer token.<br />
<br />
=== Configure the WebDAV remote using a Macaroon ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-macaroon<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV endpoint provided by PIC.<br />
<br />
For a path under <code>/pnfs/pic.es/data</code>, the URL will normally point to that path or to a subdirectory under it.<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
Paste the Macaroon token as the bearer token:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
You normally do not need to edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> n<br />
</pre><br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- bearer_token: *** ENCRYPTED ***<br />
<br />
Keep this "pic-pnfs-macaroon" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== Macaroon configuration example ===<br />
<br />
The resulting rclone configuration should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-macaroon]<br />
type = webdav<br />
url = https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
vendor = other<br />
bearer_token = YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
=== Macaroon usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-macaroon:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for any other storage location ==<br />
<br />
Use this section for storage paths that are not under <code>/pnfs/pic.es/data</code>.<br />
<br />
These locations are accessed using SSH/SFTP.<br />
<br />
Because SSH authentication may require browser-based authentication, you should configure SSH <code>ControlMaster</code>. This allows rclone to reuse an already-authenticated SSH connection.<br />
<br />
=== Configure SSH ControlMaster ===<br />
<br />
On your client machine, edit:<br />
<br />
<pre><br />
~/.ssh/config<br />
</pre><br />
<br />
Add an entry for the PIC SSH host. For example:<br />
<br />
<pre><br />
Host pic-ssh<br />
HostName ui04.pic.es<br />
User YOUR_PIC_USERNAME<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Replace:<br />
<br />
* <code>ui04.pic.es</code> with the SSH host provided by PIC, if different.<br />
* <code>YOUR_PIC_USERNAME</code> with your PIC username.<br />
<br />
Instead of restricting this configuration to <code>pic-ssh</code>, you can enable it for all SSH connections:<br />
<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
However, using a specific host entry is usually clearer and safer.<br />
<br />
=== Open the first SSH connection ===<br />
<br />
Before using rclone, open one SSH connection manually:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
</pre><br />
<br />
You may see a message similar to:<br />
<br />
<pre><br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Open the URL in your browser, complete the authentication, then return to the terminal and press <code>ENTER</code>.<br />
<br />
After this first authentication, subsequent SSH connections to the same host should reuse the existing <code>ControlMaster</code> session.<br />
<br />
Test this with:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205<br />
[tallada@ui04 ~]$<br />
</pre><br />
<br />
You can close this shell after testing.<br />
<br />
=== Configure the SFTP rclone remote ===<br />
<br />
Other storage locations use the rclone <code>sftp</code> backend.<br />
<br />
Because we want rclone to reuse the SSH <code>ControlMaster</code> connection, configure the remote to use the external <code>ssh</code> command.<br />
<br />
First, locate your rclone configuration file:<br />
<br />
<pre><br />
$ rclone config file<br />
</pre><br />
<br />
The command prints something like:<br />
<br />
<pre><br />
Configuration file is stored at:<br />
/home/YOUR_LOCAL_USER/.config/rclone/rclone.conf<br />
</pre><br />
<br />
Edit that file and add:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
The important line is:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
This tells rclone to use your system SSH command, which can reuse the SSH <code>ControlMaster</code> connection.<br />
<br />
=== SFTP configuration example ===<br />
<br />
Example rclone configuration:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
=== SFTP usage examples ===<br />
<br />
List the remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:<br />
</pre><br />
<br />
List a specific storage path:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Example using a concrete absolute path:<br />
<br />
<pre><br />
$ rclone copy ./results pic-storage:/storage/projects/MY_PROJECT/results<br />
</pre><br />
<br />
== General rclone usage ==<br />
<br />
The same basic rclone commands can be used with all configured remotes.<br />
<br />
Replace <code>REMOTE_NAME</code> with one of the remotes you configured, for example:<br />
<br />
* <code>pic-pnfs-oidc</code><br />
* <code>pic-pnfs-macaroon</code><br />
* <code>pic-storage</code><br />
<br />
=== List a remote directory ===<br />
<br />
<pre><br />
$ rclone lsd REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
$ rclone lsd pic-pnfs-macaroon:<br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== List files recursively ===<br />
<br />
<pre><br />
$ rclone ls REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Download a remote directory ===<br />
<br />
<pre><br />
$ rclone copy REMOTE_NAME:PATH LOCAL_PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
=== Upload a local directory ===<br />
<br />
<pre><br />
$ rclone copy LOCAL_DIR REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Recommended upload flags ===<br />
<br />
For uploads, we recommend:<br />
<br />
<pre><br />
--check-first -P --stats-one-line --transfers N_TRANSFERS --size-only<br />
</pre><br />
<br />
For many small files, <code>N_TRANSFERS</code> can be up to <code>350</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only<br />
</pre><br />
<br />
If uploading into directories with many files, for example more than 1000 files, also use:<br />
<br />
<pre><br />
--no-traverse<br />
</pre><br />
<br />
If uploading files larger than 200 MB, also use:<br />
<br />
<pre><br />
--multi-thread-streams 1<br />
</pre><br />
<br />
If uploading very large files, for example larger than 10 GB, also use:<br />
<br />
<pre><br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to a path under <code>/pnfs/pic.es/data</code>:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to another storage location:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-storage:/PATH_TO_YOUR_STORAGE_SPACE \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
For more information, see the rclone documentation:<br />
<br />
<pre><br />
https://rclone.org/docs/<br />
</pre><br />
<br />
== Configuring oidc-agent for OIDC tokens ==<br />
<br />
This section is only needed if you use OIDC authentication for paths under <code>/pnfs/pic.es/data</code>.<br />
<br />
Make sure <code>oidc-agent</code> is available.<br />
<br />
=== Load oidc-agent ===<br />
<br />
Initialize <code>oidc-agent</code> in the terminal session:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
If the account is already configured, load it with:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
=== Configure the OIDC account ===<br />
<br />
This step only needs to be done once.<br />
<br />
You need an updated version of <code>oidc-agent</code>, version greater than <code>5.0.0</code>.<br />
<br />
Ask your PIC contact for the client secret and replace <code>XXXXXXXXXXXXXXXXXX</code> below.<br />
<br />
Configure a <code>pic-pnfs</code> account to retrieve tokens from PIC:<br />
<br />
<pre><br />
$ oidc-gen -m --client-id CLIENT_ID_PROVIDED_BY_PIC \<br />
--client-secret XXXXXXXXXXXXXXXXXX \<br />
--pub --flow=device \<br />
--discovery-endpoint=https://idp.pic.es/realms/PIC/.well-known/openid-configuration \<br />
--scope="openid profile offline_access" \<br />
--redirect-uri=edu.kit.data.oidc-agent:/ pic-pnfs<br />
</pre><br />
<br />
The command will show a URL and a code:<br />
<br />
<pre><br />
No account exists with this short name. Creating new configuration ...<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs':<br />
Confirm encryption password:<br />
Everything setup correctly!<br />
</pre><br />
<br />
Open the URL in your browser, enter the code, authenticate, then return to the terminal.<br />
<br />
You will be asked to enter an encryption password twice. You will need this password when refreshing or reloading the <code>oidc-agent</code> account.<br />
<br />
=== Reauthenticate if the refresh token has expired ===<br />
<br />
If the <code>oidc-agent</code> process is restarted, or if your refresh token expires due to inactivity, you may need to reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
Example output:<br />
<br />
<pre><br />
Enter decryption password for account config 'pic-pnfs':<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs' [***]:<br />
Everything setup correctly!<br />
</pre><br />
<br />
=== Test OIDC token retrieval ===<br />
<br />
After loading and configuring <code>oidc-agent</code>, test that you can retrieve a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command works, the rclone configuration using:<br />
<br />
<pre><br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
should also work.<br />
<br />
== Obtaining a Macaroon for /pnfs/pic.es/data ==<br />
<br />
This section is intended for contacts or administrators who need to generate Macaroons for users.<br />
<br />
Macaroons are valid for up to 7 days.<br />
<br />
The restricted path should be under <code>/pnfs/pic.es/data</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
RESTRICTED_PATH=pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Read-only Macaroon ===<br />
<br />
Use this for downloading data only.<br />
<br />
The Macaroon will allow:<br />
<br />
* listing directories;<br />
* downloading files.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
=== Upload Macaroon ===<br />
<br />
Use this for uploading data.<br />
<br />
The Macaroon will allow full permissions on the requested path.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
== Troubleshooting ==<br />
<br />
=== SSH authentication is requested repeatedly ===<br />
<br />
This applies to any storage location accessed through SSH/SFTP.<br />
<br />
Check that:<br />
<br />
* You have opened the first SSH connection manually with <code>ssh pic-ssh</code>.<br />
* Browser authentication was completed successfully.<br />
* Your <code>~/.ssh/config</code> contains <code>ControlMaster</code>, <code>ControlPath</code>, and <code>ControlPersist</code>.<br />
* Your rclone remote uses the external SSH command:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
=== OIDC authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using OIDC.<br />
<br />
Check that <code>oidc-agent</code> is running:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
Check that the account is loaded:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
Check that you can retrieve a token manually:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
</pre><br />
<br />
If the token cannot be retrieved, reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
=== Macaroon authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using a Macaroon.<br />
<br />
Check that:<br />
<br />
* The Macaroon was copied completely.<br />
* The Macaroon has not expired.<br />
* The Macaroon was generated for the correct path under <code>/pnfs/pic.es/data</code>.<br />
* The Macaroon has the required permissions, for example read-only or upload permissions.<br />
<br />
If the Macaroon has expired, request or generate a new one.</div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1359Transferring data to/from PIC2026-06-08T11:11:41Z<p>Tallada: </p>
<hr />
<div>= How to provide data access to PIC massive storage =<br />
<br />
This page explains how to configure <code>rclone</code> to access PIC storage.<br />
<br />
The configuration depends on the storage path you need to access.<br />
<br />
{| class="wikitable"<br />
|+ Storage access methods<br />
! Storage path<br />
! Protocol<br />
! rclone backend<br />
! Supported authentication<br />
! Example remote name<br />
|-<br />
| Paths under <code>/pnfs/pic.es/data</code><br />
| WebDAV / HTTPS<br />
| <code>webdav</code><br />
| OIDC token<br />
| <code>pic-pnfs-oidc</code><br />
|-<br />
| Paths under <code>/pnfs/pic.es/data</code><br />
| WebDAV / HTTPS<br />
| <code>webdav</code><br />
| Macaroon bearer token<br />
| <code>pic-pnfs-macaroon</code><br />
|-<br />
| Any other storage location<br />
| SSH / SFTP<br />
| <code>sftp</code><br />
| SSH with <code>ControlMaster</code><br />
| <code>pic-storage</code><br />
|}<br />
<br />
== Requirements ==<br />
<br />
You need:<br />
<br />
* <code>rclone</code> installed on your client machine.<br />
* The storage path or endpoint provided by your PIC contact.<br />
<br />
For paths under <code>/pnfs/pic.es/data</code>, you also need one of the following:<br />
<br />
* an OIDC token setup using <code>oidc-agent</code>; or<br />
* a Macaroon token.<br />
<br />
For any other storage location, you also need:<br />
<br />
* SSH access to the PIC host.<br />
* SSH <code>ControlMaster</code> configured on your client machine.<br />
<br />
== Install rclone ==<br />
<br />
You can download the rclone binary directly without installing system packages. For example, on a Linux 64-bit machine:<br />
<br />
<pre><br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
$ unzip rclone-current-linux-amd64.zip<br />
</pre><br />
<br />
Alternatively, on Ubuntu you can install the Debian package:<br />
<br />
<pre><br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
</pre><br />
<br />
Check that rclone works:<br />
<br />
<pre><br />
$ rclone version<br />
</pre><br />
<br />
== Which configuration should I use? ==<br />
<br />
Use the table below to choose the right configuration.<br />
<br />
{| class="wikitable"<br />
|+ Choosing the right configuration<br />
! If you want to access...<br />
! Use this section<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using OIDC authentication<br />
| [[#Configure rclone for PNFS data paths using OIDC|Configure rclone for PNFS data paths using OIDC]]<br />
|-<br />
| A path under <code>/pnfs/pic.es/data</code> using a Macaroon token<br />
| [[#Configure rclone for PNFS data paths using a Macaroon|Configure rclone for PNFS data paths using a Macaroon]]<br />
|-<br />
| Any other storage location<br />
| [[#Configure rclone for any other storage location|Configure rclone for any other storage location]]<br />
|}<br />
<br />
== Configure rclone for PNFS data paths ==<br />
<br />
Paths under <code>/pnfs/pic.es/data</code> are accessed using WebDAV over HTTPS.<br />
<br />
For these paths, configure rclone with the <code>webdav</code> backend.<br />
<br />
Use one of the following authentication methods:<br />
<br />
* OIDC token, retrieved automatically with <code>oidc-token</code>.<br />
* Macaroon token, pasted into rclone as a bearer token.<br />
<br />
Do not configure username/password authentication unless PIC explicitly instructs you to do so.<br />
<br />
== Configure rclone for PNFS data paths using OIDC ==<br />
<br />
Use this option if you have been instructed to authenticate with OIDC.<br />
<br />
=== OIDC requirements ===<br />
<br />
Before configuring rclone, make sure that:<br />
<br />
* <code>oidc-agent</code> is available.<br />
* You have configured an OIDC account, for example <code>pic-pnfs</code>.<br />
* The following command returns a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command does not work, see [[#Configuring oidc-agent for OIDC tokens|Configuring oidc-agent for OIDC tokens]].<br />
<br />
=== Configure the WebDAV remote using OIDC ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-oidc<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV endpoint provided by PIC.<br />
<br />
For a path under <code>/pnfs/pic.es/data</code>, the URL will normally point to that path or to a subdirectory under it.<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
When asked for a bearer token, leave it empty:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token><br />
</pre><br />
<br />
Edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> y<br />
</pre><br />
<br />
Set <code>bearer_token_command</code>:<br />
<br />
<pre><br />
Option bearer_token_command.<br />
Command to run to get a bearer token.<br />
bearer_token_command> oidc-token pic-pnfs<br />
</pre><br />
<br />
Accept the remaining advanced options by pressing <code>ENTER</code>, unless PIC instructed you to change them.<br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- bearer_token_command: oidc-token pic-pnfs<br />
<br />
Keep this "pic-pnfs-oidc" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== OIDC configuration example ===<br />
<br />
The resulting rclone configuration should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-oidc]<br />
type = webdav<br />
url = https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
vendor = other<br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
=== OIDC usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-oidc:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for PNFS data paths using a Macaroon ==<br />
<br />
Use this option if you have been given a Macaroon token.<br />
<br />
The Macaroon is used as a WebDAV bearer token.<br />
<br />
=== Configure the WebDAV remote using a Macaroon ===<br />
<br />
Start the rclone configuration wizard:<br />
<br />
<pre><br />
$ rclone config<br />
</pre><br />
<br />
Create a new remote:<br />
<br />
<pre><br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic-pnfs-macaroon<br />
</pre><br />
<br />
Select WebDAV:<br />
<br />
<pre><br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
Storage> webdav<br />
</pre><br />
<br />
Enter the WebDAV endpoint provided by PIC.<br />
<br />
For a path under <code>/pnfs/pic.es/data</code>, the URL will normally point to that path or to a subdirectory under it.<br />
<br />
Example:<br />
<br />
<pre><br />
Option url.<br />
URL of http host to connect to.<br />
url> https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Select vendor <code>other</code>:<br />
<br />
<pre><br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
vendor> other<br />
</pre><br />
<br />
Leave username empty:<br />
<br />
<pre><br />
Option user.<br />
User name.<br />
user><br />
</pre><br />
<br />
Leave password empty:<br />
<br />
<pre><br />
Option pass.<br />
Password.<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank<br />
y/g/n> n<br />
</pre><br />
<br />
Paste the Macaroon token as the bearer token:<br />
<br />
<pre><br />
Option bearer_token.<br />
Bearer token instead of user/pass, for example a Macaroon.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
You normally do not need to edit the advanced configuration:<br />
<br />
<pre><br />
Edit advanced config?<br />
y) Yes<br />
n) No<br />
y/n> n<br />
</pre><br />
<br />
At the end, review and accept the configuration:<br />
<br />
<pre><br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- bearer_token: *** ENCRYPTED ***<br />
<br />
Keep this "pic-pnfs-macaroon" remote?<br />
y) Yes this is OK<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
</pre><br />
<br />
Quit the rclone configuration menu:<br />
<br />
<pre><br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
</pre><br />
<br />
=== Macaroon configuration example ===<br />
<br />
The resulting rclone configuration should look similar to:<br />
<br />
<pre><br />
[pic-pnfs-macaroon]<br />
type = webdav<br />
url = https://webdav.pic.es/pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
vendor = other<br />
bearer_token = YOUR_MACAROON_TOKEN<br />
</pre><br />
<br />
=== Macaroon usage examples ===<br />
<br />
List the configured remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:<br />
</pre><br />
<br />
List a directory below the configured path:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
</pre><br />
<br />
Example using a concrete path below the configured WebDAV root:<br />
<br />
<pre><br />
$ rclone copy ./ntuples pic-pnfs-macaroon:analysis/ntuples<br />
</pre><br />
<br />
== Configure rclone for any other storage location ==<br />
<br />
Use this section for storage paths that are not under <code>/pnfs/pic.es/data</code>.<br />
<br />
These locations are accessed using SSH/SFTP.<br />
<br />
Because SSH authentication may require browser-based authentication, you should configure SSH <code>ControlMaster</code>. This allows rclone to reuse an already-authenticated SSH connection.<br />
<br />
=== Configure SSH ControlMaster ===<br />
<br />
On your client machine, edit:<br />
<br />
<pre><br />
~/.ssh/config<br />
</pre><br />
<br />
Add an entry for the PIC SSH host. For example:<br />
<br />
<pre><br />
Host pic-ssh<br />
HostName ui04.pic.es<br />
User YOUR_PIC_USERNAME<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Replace:<br />
<br />
* <code>ui04.pic.es</code> with the SSH host provided by PIC, if different.<br />
* <code>YOUR_PIC_USERNAME</code> with your PIC username.<br />
<br />
Instead of restricting this configuration to <code>pic-ssh</code>, you can enable it for all SSH connections:<br />
<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
However, using a specific host entry is usually clearer and safer.<br />
<br />
=== Open the first SSH connection ===<br />
<br />
Before using rclone, open one SSH connection manually:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
</pre><br />
<br />
You may see a message similar to:<br />
<br />
<pre><br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Open the URL in your browser, complete the authentication, then return to the terminal and press <code>ENTER</code>.<br />
<br />
After this first authentication, subsequent SSH connections to the same host should reuse the existing <code>ControlMaster</code> session.<br />
<br />
Test this with:<br />
<br />
<pre><br />
$ ssh pic-ssh<br />
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205<br />
[tallada@ui04 ~]$<br />
</pre><br />
<br />
You can close this shell after testing.<br />
<br />
=== Configure the SFTP rclone remote ===<br />
<br />
Other storage locations use the rclone <code>sftp</code> backend.<br />
<br />
Because we want rclone to reuse the SSH <code>ControlMaster</code> connection, configure the remote to use the external <code>ssh</code> command.<br />
<br />
First, locate your rclone configuration file:<br />
<br />
<pre><br />
$ rclone config file<br />
</pre><br />
<br />
The command prints something like:<br />
<br />
<pre><br />
Configuration file is stored at:<br />
/home/YOUR_LOCAL_USER/.config/rclone/rclone.conf<br />
</pre><br />
<br />
Edit that file and add:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
The important line is:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
This tells rclone to use your system SSH command, which can reuse the SSH <code>ControlMaster</code> connection.<br />
<br />
=== SFTP configuration example ===<br />
<br />
Example rclone configuration:<br />
<br />
<pre><br />
[pic-storage]<br />
type = sftp<br />
ssh = ssh pic-ssh<br />
shell_type = unix<br />
known_hosts_file = ~/.ssh/known_hosts<br />
</pre><br />
<br />
=== SFTP usage examples ===<br />
<br />
List the remote root:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:<br />
</pre><br />
<br />
List a specific storage path:<br />
<br />
<pre><br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
List files recursively:<br />
<br />
<pre><br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Download data:<br />
<br />
<pre><br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
Upload data:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
Example using a concrete absolute path:<br />
<br />
<pre><br />
$ rclone copy ./results pic-storage:/storage/projects/MY_PROJECT/results<br />
</pre><br />
<br />
== General rclone usage ==<br />
<br />
The same basic rclone commands can be used with all configured remotes.<br />
<br />
Replace <code>REMOTE_NAME</code> with one of the remotes you configured, for example:<br />
<br />
* <code>pic-pnfs-oidc</code><br />
* <code>pic-pnfs-macaroon</code><br />
* <code>pic-storage</code><br />
<br />
=== List a remote directory ===<br />
<br />
<pre><br />
$ rclone lsd REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone lsd pic-pnfs-oidc:<br />
$ rclone lsd pic-pnfs-macaroon:<br />
$ rclone lsd pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== List files recursively ===<br />
<br />
<pre><br />
$ rclone ls REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone ls pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone ls pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Download a remote directory ===<br />
<br />
<pre><br />
$ rclone copy REMOTE_NAME:PATH LOCAL_PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy pic-pnfs-oidc:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-pnfs-macaroon:PROJECT_OR_DIRECTORY ./local-copy<br />
$ rclone copy pic-storage:/PATH_TO_YOUR_STORAGE_SPACE ./local-copy<br />
</pre><br />
<br />
=== Upload a local directory ===<br />
<br />
<pre><br />
$ rclone copy LOCAL_DIR REMOTE_NAME:PATH<br />
</pre><br />
<br />
Examples:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-pnfs-macaroon:PROJECT_OR_DIRECTORY<br />
$ rclone copy ./local-data pic-storage:/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Recommended upload flags ===<br />
<br />
For uploads, we recommend:<br />
<br />
<pre><br />
--check-first -P --stats-one-line --transfers N_TRANSFERS --size-only<br />
</pre><br />
<br />
For many small files, <code>N_TRANSFERS</code> can be up to <code>350</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
$ rclone copy ./local-data pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only<br />
</pre><br />
<br />
If uploading into directories with many files, for example more than 1000 files, also use:<br />
<br />
<pre><br />
--no-traverse<br />
</pre><br />
<br />
If uploading files larger than 200 MB, also use:<br />
<br />
<pre><br />
--multi-thread-streams 1<br />
</pre><br />
<br />
If uploading very large files, for example larger than 10 GB, also use:<br />
<br />
<pre><br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to a path under <code>/pnfs/pic.es/data</code>:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-pnfs-oidc:PROJECT_OR_DIRECTORY \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
Full example for a large upload to another storage location:<br />
<br />
<pre><br />
$ rclone copy ./large-dataset pic-storage:/PATH_TO_YOUR_STORAGE_SPACE \<br />
--check-first -P --stats-one-line --transfers 350 --size-only \<br />
--no-traverse \<br />
--multi-thread-streams 1 \<br />
--timeout 15m<br />
</pre><br />
<br />
For more information, see the rclone documentation:<br />
<br />
<pre><br />
https://rclone.org/docs/<br />
</pre><br />
<br />
== Configuring oidc-agent for OIDC tokens ==<br />
<br />
This section is only needed if you use OIDC authentication for paths under <code>/pnfs/pic.es/data</code>.<br />
<br />
Make sure <code>oidc-agent</code> is available.<br />
<br />
=== Load oidc-agent ===<br />
<br />
Initialize <code>oidc-agent</code> in the terminal session:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
If the account is already configured, load it with:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
=== Configure the OIDC account ===<br />
<br />
This step only needs to be done once.<br />
<br />
You need an updated version of <code>oidc-agent</code>, version greater than <code>5.0.0</code>.<br />
<br />
Ask your PIC contact for the client secret and replace <code>XXXXXXXXXXXXXXXXXX</code> below.<br />
<br />
Configure a <code>pic-pnfs</code> account to retrieve tokens from PIC:<br />
<br />
<pre><br />
$ oidc-gen -m --client-id CLIENT_ID_PROVIDED_BY_PIC \<br />
--client-secret XXXXXXXXXXXXXXXXXX \<br />
--pub --flow=device \<br />
--discovery-endpoint=https://idp.pic.es/realms/PIC/.well-known/openid-configuration \<br />
--scope="openid profile offline_access" \<br />
--redirect-uri=edu.kit.data.oidc-agent:/ pic-pnfs<br />
</pre><br />
<br />
The command will show a URL and a code:<br />
<br />
<pre><br />
No account exists with this short name. Creating new configuration ...<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs':<br />
Confirm encryption password:<br />
Everything setup correctly!<br />
</pre><br />
<br />
Open the URL in your browser, enter the code, authenticate, then return to the terminal.<br />
<br />
You will be asked to enter an encryption password twice. You will need this password when refreshing or reloading the <code>oidc-agent</code> account.<br />
<br />
=== Reauthenticate if the refresh token has expired ===<br />
<br />
If the <code>oidc-agent</code> process is restarted, or if your refresh token expires due to inactivity, you may need to reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
Example output:<br />
<br />
<pre><br />
Enter decryption password for account config 'pic-pnfs':<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-pnfs' [***]:<br />
Everything setup correctly!<br />
</pre><br />
<br />
=== Test OIDC token retrieval ===<br />
<br />
After loading and configuring <code>oidc-agent</code>, test that you can retrieve a token:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
</pre><br />
<br />
If this command works, the rclone configuration using:<br />
<br />
<pre><br />
bearer_token_command = oidc-token pic-pnfs<br />
</pre><br />
<br />
should also work.<br />
<br />
== Obtaining a Macaroon for /pnfs/pic.es/data ==<br />
<br />
This section is intended for contacts or administrators who need to generate Macaroons for users.<br />
<br />
Macaroons are valid for up to 7 days.<br />
<br />
The restricted path should be under <code>/pnfs/pic.es/data</code>.<br />
<br />
Example:<br />
<br />
<pre><br />
RESTRICTED_PATH=pnfs/pic.es/data/PATH_TO_YOUR_STORAGE_SPACE<br />
</pre><br />
<br />
=== Read-only Macaroon ===<br />
<br />
Use this for downloading data only.<br />
<br />
The Macaroon will allow:<br />
<br />
* listing directories;<br />
* downloading files.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
=== Upload Macaroon ===<br />
<br />
Use this for uploading data.<br />
<br />
The Macaroon will allow full permissions on the requested path.<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
</pre><br />
<br />
Example response:<br />
<br />
<pre><br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
Give the value of the <code>macaroon</code> field to the user. The user should paste it into rclone as the WebDAV <code>bearer_token</code>.<br />
<br />
== Troubleshooting ==<br />
<br />
=== SSH authentication is requested repeatedly ===<br />
<br />
This applies to any storage location accessed through SSH/SFTP.<br />
<br />
Check that:<br />
<br />
* You have opened the first SSH connection manually with <code>ssh pic-ssh</code>.<br />
* Browser authentication was completed successfully.<br />
* Your <code>~/.ssh/config</code> contains <code>ControlMaster</code>, <code>ControlPath</code>, and <code>ControlPersist</code>.<br />
* Your rclone remote uses the external SSH command:<br />
<br />
<pre><br />
ssh = ssh pic-ssh<br />
</pre><br />
<br />
=== OIDC authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using OIDC.<br />
<br />
Check that <code>oidc-agent</code> is running:<br />
<br />
<pre><br />
$ eval `oidc-agent`<br />
</pre><br />
<br />
Check that the account is loaded:<br />
<br />
<pre><br />
$ oidc-add pic-pnfs<br />
</pre><br />
<br />
Check that you can retrieve a token manually:<br />
<br />
<pre><br />
$ oidc-token pic-pnfs<br />
</pre><br />
<br />
If the token cannot be retrieved, reauthenticate:<br />
<br />
<pre><br />
$ oidc-gen --reauthenticate pic-pnfs<br />
</pre><br />
<br />
=== Macaroon authentication fails ===<br />
<br />
This applies to paths under <code>/pnfs/pic.es/data</code> accessed using a Macaroon.<br />
<br />
Check that:<br />
<br />
* The Macaroon was copied completely.<br />
* The Macaroon has not expired.<br />
* The Macaroon was generated for the correct path under <code>/pnfs/pic.es/data</code>.<br />
* The Macaroon has the required permissions, for example read-only or upload permissions.<br />
<br />
If the Macaroon has expired, request or generate a new one.</div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1355Transferring data to/from PIC2026-05-07T15:27:39Z<p>Tallada: </p>
<hr />
<div>= How to provide data access to PIC massive storage (dCache) =<br />
<br />
== Requirements ==<br />
<br />
* Install and configure Rclone<br />
* PIC credentials or macaroon<br />
<br />
== Install Rclone ==<br />
<br />
You can directly download the binary without installing anything. For instance, for a linux 64 bits machine:<br />
<br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
[...]<br />
$ unzip rclone-current-linux-amd64.zip<br />
<br />
<br />
Or if you prefer, you can install Rclone like the next example on a Ubuntu machine:<br />
<br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
<br />
== Configure Rclone ==<br />
<br />
You should have been given some credentials and the url of a WebDAV endpoint at PIC.<br />
With them, you just need to create the config in rclone:<br />
<br />
$ rclone config<br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic<br />
<br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
1 / 1Fichier<br />
\ (fichier)<br />
[...]<br />
Storage> webdav<br />
<br />
Option url.<br />
URL of http host to connect to.<br />
E.g. https://example.com.<br />
Enter a value.<br />
url> https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
<br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
Choose a number from below, or type in your own value.<br />
Press Enter to leave empty.<br />
1 / Nextcloud<br />
\ (nextcloud)<br />
[...]<br />
5 / Other site/service or software<br />
\ (other)<br />
vendor> other<br />
<br />
=== Using your PIC credentials ===<br />
<br />
If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank.<br />
<br />
Option user.<br />
User name.<br />
In case NTLM authentication is used, the username should be in the format 'Domain\User'.<br />
Enter a value. Press Enter to leave empty.<br />
user> YOUR_PIC_USERNAME<br />
<br />
Option pass.<br />
Password.<br />
Choose an alternative below. Press Enter for the default (n).<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank (default)<br />
y/g/n> y<br />
Enter the password:<br />
password: YOUR_PIC_PASSWORD<br />
Confirm the password:<br />
Password: YOUR_PIC_PASSWORD<br />
<br />
=== Using a Macaroon token ===<br />
<br />
If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank<br />
<br />
Option bearer_token.<br />
Bearer token instead of user/pass (e.g. a Macaroon).<br />
Enter a value. Press Enter to leave empty.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
<br />
=== Using an OIDC token ===<br />
<br />
Option bearer_token.<br />
Bearer token instead of user/pass (e.g. a Macaroon).<br />
Enter a value. Press Enter to leave empty.<br />
bearer_token> <br />
<br />
Edit advanced config?<br />
y) Yes<br />
n) No (default)<br />
y/n> y<br />
<br />
Option bearer_token_command.<br />
Command to run to get a bearer token.<br />
Enter a value. Press Enter to leave empty.<br />
bearer_token_command> oidc-token OIDC_AGENT_ACCOUNT_SHORTNAME<br />
<br />
=== Review settings ===<br />
<br />
At the end, just review the information you entered and confirm.<br />
<br />
Edit advanced config?<br />
y) Yes<br />
n) No (default)<br />
y/n> n<br />
<br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://door04.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- user: YOUR_PIC_USERNAME<br />
- pass: *** ENCRYPTED ***<br />
Keep this "pic" remote?<br />
y) Yes this is OK (default)<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
<br />
Current remotes:<br />
<br />
Name Type<br />
==== ====<br />
pic webdav<br />
<br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
<br />
Once done, you can use command line to browse and download/upload data.<br />
<br />
=== Usage ===<br />
<br />
* List a remote PIC directory: <br />
rclone lsd <name>:<path>`<br />
<br />
* Download a remote directory from PIC<br />
rclone copy <name>:<path> <local_path><br />
<br />
* Upload a local directory to PIC<br />
rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path><br />
<br />
When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files <br />
--check-first -P --stats-one-line --transfers <n_transfers> --size-only<br />
<br />
If uploading lots onto directories with lots of files (>1000), please use:<br />
--no-traverse<br />
<br />
If uploading files larger than 200 MB, also use<br />
--multi-thread-streams 1<br />
<br />
If uploading very large files (>10G), also use the following to allow more time to compute the checksums<br />
--timeout=15m<br />
<br />
See rclone manual for more extensive documentation https://rclone.org/docs/<br />
<br />
== Configuring oidc-agent for obtaining OIDC tokens ==<br />
<br />
Make sure <code>oidc-agent</code> is available.<br />
<br />
=== Load oidc-agent === <br />
<br />
Initialize <code>oidc-agent</code> in the terminal session.<br />
<br />
$ eval `oidc-agent`<br />
<br />
=== Configuration (only once) ===<br />
<br />
Configure a <code>pic-dcache</code> account to retrieve tokens from PIC. Open the URL that will show and input the code provided (or just open the QR code displayed).<br />
After authenticating on the web browser, return to the terminal and input an encryption password twice. You'll need it when refreshing/reloading the <code>oidc-agent</code>.<br />
<br />
$ oidc-gen -m --client-id oidc-agent --pub --flow=device \<br />
--discovery-endpoint=https://idp.pic.es/realms/PIC/.well-known/openid-configuration \<br />
--scope="openid profile offline_access" --redirect-uri=edu.kit.data.oidc-agent:/ pic-dcache<br />
<br />
No account exists with this short name. Creating new configuration ...<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-dcache': <br />
Confirm encryption password: <br />
Everything setup correctly!<br />
<br />
=== Reauthenticating (if refresh token has expired) ===<br />
<br />
If the oidc-agent process gets restarted, or iIf your refresh token expires due to inactivity, you will need to reauthenticate to retrieve further tokens<br />
<br />
$ oidc-gen --reauthenticate pic-dcache<br />
Enter decryption password for account config 'testtest': <br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-dcache' [***]: <br />
Everything setup correctly!<br />
<br />
=== Testing ===<br />
<br />
After loading and configuring, you can get a token by running the following command:<br />
<br />
$ oidc-token pic-dcache<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
<br />
== Obtaining a macaroon (for contacts) ==<br />
<br />
Macaroons are valid up to 7 days.<br />
<br />
For downloading data (read-only permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
For uploading data (full permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre></div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1354Transferring data to/from PIC2026-05-07T12:32:05Z<p>Tallada: </p>
<hr />
<div>= How to provide data access to PIC massive storage (dCache) =<br />
<br />
== Requirements ==<br />
<br />
* Install and configure Rclone<br />
* PIC credentials or macaroon<br />
<br />
== Install Rclone ==<br />
<br />
You can directly download the binary without installing anything. For instance, for a linux 64 bits machine:<br />
<br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
[...]<br />
$ unzip rclone-current-linux-amd64.zip<br />
<br />
<br />
Or if you prefer, you can install Rclone like the next example on a Ubuntu machine:<br />
<br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
<br />
== Configure Rclone ==<br />
<br />
You should have been given some credentials and the url of a WebDAV endpoint at PIC.<br />
With them, you just need to create the config in rclone:<br />
<br />
$ rclone config<br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic<br />
<br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
1 / 1Fichier<br />
\ (fichier)<br />
[...]<br />
Storage> webdav<br />
<br />
Option url.<br />
URL of http host to connect to.<br />
E.g. https://example.com.<br />
Enter a value.<br />
url> https://door04.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
<br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
Choose a number from below, or type in your own value.<br />
Press Enter to leave empty.<br />
1 / Nextcloud<br />
\ (nextcloud)<br />
[...]<br />
5 / Other site/service or software<br />
\ (other)<br />
vendor> other<br />
<br />
=== Using your PIC credentials ===<br />
<br />
If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank.<br />
<br />
Option user.<br />
User name.<br />
In case NTLM authentication is used, the username should be in the format 'Domain\User'.<br />
Enter a value. Press Enter to leave empty.<br />
user> YOUR_PIC_USERNAME<br />
<br />
Option pass.<br />
Password.<br />
Choose an alternative below. Press Enter for the default (n).<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank (default)<br />
y/g/n> y<br />
Enter the password:<br />
password: YOUR_PIC_PASSWORD<br />
Confirm the password:<br />
Password: YOUR_PIC_PASSWORD<br />
<br />
=== Using a Macaroon token ===<br />
<br />
If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank<br />
<br />
Option bearer_token.<br />
Bearer token instead of user/pass (e.g. a Macaroon).<br />
Enter a value. Press Enter to leave empty.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
<br />
=== Using an OIDC token ===<br />
<br />
Option bearer_token.<br />
Bearer token instead of user/pass (e.g. a Macaroon).<br />
Enter a value. Press Enter to leave empty.<br />
bearer_token> <br />
<br />
Edit advanced config?<br />
y) Yes<br />
n) No (default)<br />
y/n> y<br />
<br />
Option bearer_token_command.<br />
Command to run to get a bearer token.<br />
Enter a value. Press Enter to leave empty.<br />
bearer_token_command> oidc-token OIDC_AGENT_ACCOUNT_SHORTNAME<br />
<br />
=== Review settings ===<br />
<br />
At the end, just review the information you entered and confirm.<br />
<br />
Edit advanced config?<br />
y) Yes<br />
n) No (default)<br />
y/n> n<br />
<br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://door04.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- user: YOUR_PIC_USERNAME<br />
- pass: *** ENCRYPTED ***<br />
Keep this "pic" remote?<br />
y) Yes this is OK (default)<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
<br />
Current remotes:<br />
<br />
Name Type<br />
==== ====<br />
pic webdav<br />
<br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
<br />
Once done, you can use command line to browse and download/upload data.<br />
<br />
=== Usage ===<br />
<br />
* List a remote PIC directory: <br />
rclone lsd <name>:<path>`<br />
<br />
* Download a remote directory from PIC<br />
rclone copy <name>:<path> <local_path><br />
<br />
* Upload a local directory to PIC<br />
rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path><br />
<br />
When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files <br />
--check-first -P --stats-one-line --transfers <n_transfers> --size-only<br />
<br />
If uploading lots onto directories with lots of files (>1000), please use:<br />
--no-traverse<br />
<br />
If uploading files larger than 200 MB, also use<br />
--multi-thread-streams 1<br />
<br />
If uploading very large files (>10G), also use the following to allow more time to compute the checksums<br />
--timeout=15m<br />
<br />
See rclone manual for more extensive documentation https://rclone.org/docs/<br />
<br />
== Configuring oidc-agent for obtaining OIDC tokens ==<br />
<br />
Make sure <code>oidc-agent</code> is available.<br />
<br />
=== Load oidc-agent === <br />
<br />
Initialize <code>oidc-agent</code> in the terminal session.<br />
<br />
$ eval `oidc-agent`<br />
<br />
=== Configuration (only once) ===<br />
<br />
Configure a <code>pic-dcache</code> account to retrieve tokens from PIC. Open the URL that will show and input the code provided (or just open the QR code displayed).<br />
After authenticating on the web browser, return to the terminal and input an encryption password twice. You'll need it when refreshing/reloading the <code>oidc-agent</code>.<br />
<br />
$ oidc-gen -m --client-id oidc-agent --pub --flow=device \<br />
--discovery-endpoint=https://idp.pic.es/realms/PIC/.well-known/openid-configuration \<br />
--scope="openid profile offline_access" --redirect-uri=edu.kit.data.oidc-agent:/ pic-dcache<br />
<br />
No account exists with this short name. Creating new configuration ...<br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-dcache': <br />
Confirm encryption password: <br />
Everything setup correctly!<br />
<br />
=== Reauthenticating (if refresh token has expired) ===<br />
<br />
If the oidc-agent process gets restarted, or iIf your refresh token expires due to inactivity, you will need to reauthenticate to retrieve further tokens<br />
<br />
$ oidc-gen --reauthenticate pic-dcache<br />
Enter decryption password for account config 'testtest': <br />
Generating account configuration ...<br />
accepted<br />
<br />
Using a browser on any device, visit:<br />
https://idp.pic.es/realms/PIC/device<br />
<br />
And enter the code: ASDF-GHJK<br />
Alternatively you can use the following QR code to visit the above listed URL.<br />
<br />
[ QR CODE ]<br />
<br />
Enter encryption password for account configuration 'pic-dcache' [***]: <br />
Everything setup correctly!<br />
<br />
=== Testing ===<br />
<br />
After loading and configuring, you can get a token by running the following command:<br />
<br />
$ oidc-token pic-dcache<br />
eyJhbGciOiJSUzI1[...]4YjAwg<br />
<br />
== Obtaining a macaroon (for contacts) ==<br />
<br />
Macaroons are valid up to 7 days.<br />
<br />
For downloading data (read-only permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
For uploading data (full permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre></div>Talladahttps://pwiki.pic.es/index.php?title=Faq&diff=1352Faq2026-04-15T15:05:17Z<p>Tallada: </p>
<hr />
<div>== How do I reset my password? ==<br />
You can reset your password using the following link:<br />
https://www.pic.es/user/auth/forgotpw<br />
<br />
== Can an undergraduate student in my group have an account? ==<br />
Yes. Undergraduate students can have PIC accounts without any problem.<br />
<br />
== How do I get a Kerberos token? ==<br />
A valid token is needed to interface with some PIC services, like the Hadoop platform (HDFS, Hive, Spark, ...) or CTA.<br />
Tokens are issued for the machine in which they are requested, if you hop to another machine, you may need to get another token.<br />
<br />
In order to get a token, run the following commands on a terminal.<br />
If you are on JupyterLab, use ''Launcher'' → ''Terminal''.<br />
<br />
<pre><br />
kinit -n -c ~/.fast.ccache @PIC.ES<br />
kinit -T ~/.fast.ccache<br />
</pre><br />
<br />
You could also define an alias for your own convenience:<br />
<br />
<pre><br />
alias kinit="kinit -n -c ~/.fast.ccache @PIC.ES; kinit -T ~/.fast.ccache"<br />
</pre><br />
<br />
Optionally, you can check the presence and expiration date of your token using:<br />
<br />
<pre><br />
klist<br />
</pre><br />
<br />
== I lost access to my OTP, how can I reset it? ==<br />
Follow these steps to recover your account and configure a new OTP token.<br />
<br />
# On the login page, click '''Forgot password / OTP''' or go [https://www.pic.es/reset here].<br />
# Enter your username. You should receive a recovery email in your inbox.<br />
# Open the email and follow the recovery link.<br />
# Scan the new QR code with your authenticator app.<br />
# Enter the generated OTP code and a device name.<br />
#* The device name is mandatory.<br />
# Change your password when prompted.<br />
#* This is required for security reasons.<br />
# Log in again as usual.<br />
#* On the OTP selection page, you will see two devices.<br />
#* Select the device with the name you just entered.<br />
#* The old one will usually appear as '''unnamed''' (if no name was set previously).<br />
<br />
Afterwards, we recommend removing the old OTP factor from the [https://idp.pic.es/realms/PIC/account/account-security/signing-in user profile section].<br />
<br />
== What is the simplest way to avoid repeated SSH authentication? ==<br />
The easiest way is to configure your SSH connection using '''ControlMaster''', which allows multiple sessions to reuse a single authenticated connection.<br />
<br />
On your client machine, edit the file <code>~/.ssh/config</code> and add the following lines:<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Instead of <code>Host *</code>, you can restrict this to a specific host, for example:<br />
<br />
<pre><br />
Host ui04.pic.es<br />
</pre><br />
<br />
Initiate the first SSH connection and authenticate via the provided URL. You will see something like:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Complete the authentication in your browser, then press ENTER.<br />
Once authenticated, the SSH connection remains open in the background, and you can safely close your terminal session.<br />
The connection will persist indefinitely (until machine reboot or a network interruption).<br />
<br />
Subsequent SSH connections to the same host will reuse the existing session and will not require authentication again:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205<br />
[tallada@ui04 ~]$<br />
</pre><br />
</li><br />
</ol></div>Talladahttps://pwiki.pic.es/index.php?title=Faq&diff=1351Faq2026-04-15T14:09:44Z<p>Tallada: </p>
<hr />
<div>== How do I reset my password? ==<br />
You can reset your password using the following link:<br />
https://www.pic.es/user/auth/forgotpw<br />
<br />
== Can an undergraduate student in my group have an account? ==<br />
Yes. Undergraduate students can have PIC accounts without any problem.<br />
<br />
== How do I get a Kerberos token? ==<br />
A valid token is needed to interface with some PIC services, like the Hadoop platform (HDFS, Hive, Spark, ...) or CTA.<br />
Tokens are issued for the machine in which they are requested, if you hop to another machine, you may need to get another token.<br />
<br />
In order to get a token, run the following commands on a terminal.<br />
If you are on JupyterLab, use ''Launcher'' → ''Terminal''.<br />
<br />
<pre><br />
kinit -n -c ~/.fast.ccache @PIC.ES<br />
kinit -T ~/.fast.ccache<br />
</pre><br />
<br />
You could also define an alias for your own convenience:<br />
<br />
<pre><br />
alias kinit="kinit -n -c ~/.fast.ccache @PIC.ES; kinit -T ~/.fast.ccache"<br />
</pre><br />
<br />
Optionally, you can check the presence and expiration date of your token using:<br />
<br />
<pre><br />
klist<br />
</pre><br />
<br />
== I lost access to my OTP, how can I reset it? ==<br />
Follow these steps to recover your account and configure a new OTP token.<br />
<br />
# On the login page, click '''Forgot password / OTP''' or go [https://www.pic.es/reset here].<br />
# Enter your username. You should receive a recovery email in your inbox.<br />
# Open the email and follow the recovery link.<br />
# Scan the new QR code with your authenticator app.<br />
# Enter the generated OTP code and a device name.<br />
#* The device name is mandatory.<br />
# Change your password when prompted.<br />
#* This is required for security reasons.<br />
# Log in again as usual.<br />
#* On the OTP selection page, you will see two devices.<br />
#* Select the device with the name you just entered.<br />
#* The old one will usually appear as '''unnamed''' (if no name was set previously).<br />
<br />
Afterwards, we recommend removing the old OTP factor from the [https://idp.pic.es/realms/PIC/account/account-security/signing-in user profile section].<br />
<br />
=== What is the simplest way to avoid repeated SSH authentication? ===<br />
The easiest way is to configure your SSH connection using '''ControlMaster''', which allows multiple sessions to reuse a single authenticated connection.<br />
<br />
On your client machine, edit the file <code>~/.ssh/config</code> and add the following lines:<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Instead of <code>Host *</code>, you can restrict this to a specific host, for example:<br />
<br />
<pre><br />
Host ui04.pic.es<br />
</pre><br />
<br />
Initiate the first SSH connection and authenticate via the provided URL. You will see something like:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Complete the authentication in your browser, then press ENTER.<br />
Once authenticated, the SSH connection remains open in the background, and you can safely close your terminal session.<br />
The connection will persist indefinitely (until machine reboot or a network interruption).<br />
<br />
Subsequent SSH connections to the same host will reuse the existing session and will not require authentication again:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205<br />
[tallada@ui04 ~]$<br />
</pre><br />
</li><br />
</ol></div>Talladahttps://pwiki.pic.es/index.php?title=Faq&diff=1350Faq2026-04-15T13:57:45Z<p>Tallada: /* What is the simplest way to avoid repeated SSH authentication? */</p>
<hr />
<div>== How do I reset my password? ==<br />
You can reset your password using the following link:<br />
https://www.pic.es/user/auth/forgotpw<br />
<br />
== Can an undergraduate student in my group have an account? ==<br />
Yes. Undergraduate students can have PIC accounts without any problem.<br />
<br />
== How do I get a Kerberos token? ==<br />
A valid token is needed to interface with some PIC services, like the Hadoop platform (HDFS, Hive, Spark, ...) or CTA.<br />
Tokens are issued for the machine in which they are requested, if you hop to another machine, you may need to get another token.<br />
<br />
In order to get a token, run the following commands on a terminal.<br />
If you are on JupyterLab, use ''Launcher'' → ''Terminal''.<br />
<br />
<pre><br />
kinit -n -c ~/.fast.ccache @PIC.ES<br />
kinit -T ~/.fast.ccache<br />
</pre><br />
<br />
You could also define an alias for your own convenience:<br />
<br />
<pre><br />
alias kinit="kinit -n -c ~/.fast.ccache @PIC.ES; kinit -T ~/.fast.ccache"<br />
</pre><br />
<br />
Optionally, you can check the presence and expiration date of your token using:<br />
<br />
<pre><br />
klist<br />
</pre><br />
<br />
== I lost access to my OTP, how can I reset it? ==<br />
Follow these steps to recover your account and configure a new OTP token.<br />
<br />
# On the Keycloak login page, click '''Forgot password / OTP'''.<br />
# Enter your username. You should receive a recovery email in your inbox.<br />
# Open the email and follow the recovery link.<br />
# Scan the new QR code with your authenticator app.<br />
# Enter the generated OTP code and a device name.<br />
#* The device name is mandatory.<br />
# Change your password when prompted.<br />
#* This is required for security reasons.<br />
# Log in again as usual.<br />
#* On the OTP selection page, you will see two devices.<br />
#* Select the device with the name you just entered.<br />
#* The old one will usually appear as '''unnamed''' (if no name was set previously).<br />
<br />
Afterwards, we recommend removing the old OTP factor from the [https://idp.pic.es/realms/PIC/account/account-security/signing-in user profile section].<br />
<br />
=== What is the simplest way to avoid repeated SSH authentication? ===<br />
The easiest way is to configure your SSH connection using '''ControlMaster''', which allows multiple sessions to reuse a single authenticated connection.<br />
<br />
On your client machine, edit the file <code>~/.ssh/config</code> and add the following lines:<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Instead of <code>Host *</code>, you can restrict this to a specific host, for example:<br />
<br />
<pre><br />
Host ui04.pic.es<br />
</pre><br />
<br />
Initiate the first SSH connection and authenticate via the provided URL. You will see something like:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Complete the authentication in your browser, then press ENTER.<br />
Once authenticated, the SSH connection remains open in the background, and you can safely close your terminal session.<br />
The connection will persist indefinitely (until machine reboot or a network interruption).<br />
<br />
Subsequent SSH connections to the same host will reuse the existing session and will not require authentication again:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205<br />
[tallada@ui04 ~]$<br />
</pre><br />
</li><br />
</ol></div>Talladahttps://pwiki.pic.es/index.php?title=Faq&diff=1349Faq2026-04-15T13:57:33Z<p>Tallada: /* What is the simplest way to avoid repeated SSH authentication? */</p>
<hr />
<div>== How do I reset my password? ==<br />
You can reset your password using the following link:<br />
https://www.pic.es/user/auth/forgotpw<br />
<br />
== Can an undergraduate student in my group have an account? ==<br />
Yes. Undergraduate students can have PIC accounts without any problem.<br />
<br />
== How do I get a Kerberos token? ==<br />
A valid token is needed to interface with some PIC services, like the Hadoop platform (HDFS, Hive, Spark, ...) or CTA.<br />
Tokens are issued for the machine in which they are requested, if you hop to another machine, you may need to get another token.<br />
<br />
In order to get a token, run the following commands on a terminal.<br />
If you are on JupyterLab, use ''Launcher'' → ''Terminal''.<br />
<br />
<pre><br />
kinit -n -c ~/.fast.ccache @PIC.ES<br />
kinit -T ~/.fast.ccache<br />
</pre><br />
<br />
You could also define an alias for your own convenience:<br />
<br />
<pre><br />
alias kinit="kinit -n -c ~/.fast.ccache @PIC.ES; kinit -T ~/.fast.ccache"<br />
</pre><br />
<br />
Optionally, you can check the presence and expiration date of your token using:<br />
<br />
<pre><br />
klist<br />
</pre><br />
<br />
== I lost access to my OTP, how can I reset it? ==<br />
Follow these steps to recover your account and configure a new OTP token.<br />
<br />
# On the Keycloak login page, click '''Forgot password / OTP'''.<br />
# Enter your username. You should receive a recovery email in your inbox.<br />
# Open the email and follow the recovery link.<br />
# Scan the new QR code with your authenticator app.<br />
# Enter the generated OTP code and a device name.<br />
#* The device name is mandatory.<br />
# Change your password when prompted.<br />
#* This is required for security reasons.<br />
# Log in again as usual.<br />
#* On the OTP selection page, you will see two devices.<br />
#* Select the device with the name you just entered.<br />
#* The old one will usually appear as '''unnamed''' (if no name was set previously).<br />
<br />
Afterwards, we recommend removing the old OTP factor from the [https://idp.pic.es/realms/PIC/account/account-security/signing-in user profile section].<br />
<br />
=== What is the simplest way to avoid repeated SSH authentication? ===<br />
The easiest way is to configure your SSH connection using '''ControlMaster''', which allows multiple sessions to reuse a single authenticated connection.<br />
<br />
On your client machine, edit the file <code>~/.ssh/config</code> and add the following lines:<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Instead of <code>Host *</code>, you can restrict this to a specific host, for example:<br />
<br />
<pre><br />
Host ui04.pic.es<br />
</pre><br />
<br />
<br />
Initiate the first SSH connection and authenticate via the provided URL. You will see something like:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Complete the authentication in your browser, then press ENTER.<br />
Once authenticated, the SSH connection remains open in the background, and you can safely close your terminal session.<br />
The connection will persist indefinitely (until machine reboot or a network interruption).<br />
<br />
Subsequent SSH connections to the same host will reuse the existing session and will not require authentication again:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205<br />
[tallada@ui04 ~]$<br />
</pre><br />
</li><br />
</ol></div>Talladahttps://pwiki.pic.es/index.php?title=Faq&diff=1348Faq2026-04-15T13:57:07Z<p>Tallada: /* What is the simplest way to avoid repeated SSH authentication? */</p>
<hr />
<div>== How do I reset my password? ==<br />
You can reset your password using the following link:<br />
https://www.pic.es/user/auth/forgotpw<br />
<br />
== Can an undergraduate student in my group have an account? ==<br />
Yes. Undergraduate students can have PIC accounts without any problem.<br />
<br />
== How do I get a Kerberos token? ==<br />
A valid token is needed to interface with some PIC services, like the Hadoop platform (HDFS, Hive, Spark, ...) or CTA.<br />
Tokens are issued for the machine in which they are requested, if you hop to another machine, you may need to get another token.<br />
<br />
In order to get a token, run the following commands on a terminal.<br />
If you are on JupyterLab, use ''Launcher'' → ''Terminal''.<br />
<br />
<pre><br />
kinit -n -c ~/.fast.ccache @PIC.ES<br />
kinit -T ~/.fast.ccache<br />
</pre><br />
<br />
You could also define an alias for your own convenience:<br />
<br />
<pre><br />
alias kinit="kinit -n -c ~/.fast.ccache @PIC.ES; kinit -T ~/.fast.ccache"<br />
</pre><br />
<br />
Optionally, you can check the presence and expiration date of your token using:<br />
<br />
<pre><br />
klist<br />
</pre><br />
<br />
== I lost access to my OTP, how can I reset it? ==<br />
Follow these steps to recover your account and configure a new OTP token.<br />
<br />
# On the Keycloak login page, click '''Forgot password / OTP'''.<br />
# Enter your username. You should receive a recovery email in your inbox.<br />
# Open the email and follow the recovery link.<br />
# Scan the new QR code with your authenticator app.<br />
# Enter the generated OTP code and a device name.<br />
#* The device name is mandatory.<br />
# Change your password when prompted.<br />
#* This is required for security reasons.<br />
# Log in again as usual.<br />
#* On the OTP selection page, you will see two devices.<br />
#* Select the device with the name you just entered.<br />
#* The old one will usually appear as '''unnamed''' (if no name was set previously).<br />
<br />
Afterwards, we recommend removing the old OTP factor from the [https://idp.pic.es/realms/PIC/account/account-security/signing-in user profile section].<br />
<br />
=== What is the simplest way to avoid repeated SSH authentication? ===<br />
The easiest way is to configure your SSH connection using '''ControlMaster''', which allows multiple sessions to reuse a single authenticated connection.<br />
<br />
On your client machine, edit the file <code>~/.ssh/config</code> and add the following lines:<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Instead of <code>Host *</code>, you can restrict this to a specific host, for example:<br />
<br />
<pre><br />
Host ui04.pic.es<br />
</pre><br />
<br />
<br />
Initiate the first SSH connection and authenticate via the provided URL:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
</pre><br />
<br />
You will see something like:<br />
<br />
<pre><br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Complete the authentication in your browser, then press ENTER.<br />
Once authenticated, the SSH connection remains open in the background, and you can safely close your terminal session.<br />
The connection will persist indefinitely (until machine reboot or a network interruption).<br />
<br />
Subsequent SSH connections to the same host will reuse the existing session and will not require authentication again:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205<br />
[tallada@ui04 ~]$<br />
</pre><br />
</li><br />
</ol></div>Talladahttps://pwiki.pic.es/index.php?title=Faq&diff=1347Faq2026-04-15T13:55:37Z<p>Tallada: </p>
<hr />
<div>== How do I reset my password? ==<br />
You can reset your password using the following link:<br />
https://www.pic.es/user/auth/forgotpw<br />
<br />
== Can an undergraduate student in my group have an account? ==<br />
Yes. Undergraduate students can have PIC accounts without any problem.<br />
<br />
== How do I get a Kerberos token? ==<br />
A valid token is needed to interface with some PIC services, like the Hadoop platform (HDFS, Hive, Spark, ...) or CTA.<br />
Tokens are issued for the machine in which they are requested, if you hop to another machine, you may need to get another token.<br />
<br />
In order to get a token, run the following commands on a terminal.<br />
If you are on JupyterLab, use ''Launcher'' → ''Terminal''.<br />
<br />
<pre><br />
kinit -n -c ~/.fast.ccache @PIC.ES<br />
kinit -T ~/.fast.ccache<br />
</pre><br />
<br />
You could also define an alias for your own convenience:<br />
<br />
<pre><br />
alias kinit="kinit -n -c ~/.fast.ccache @PIC.ES; kinit -T ~/.fast.ccache"<br />
</pre><br />
<br />
Optionally, you can check the presence and expiration date of your token using:<br />
<br />
<pre><br />
klist<br />
</pre><br />
<br />
== I lost access to my OTP, how can I reset it? ==<br />
Follow these steps to recover your account and configure a new OTP token.<br />
<br />
# On the Keycloak login page, click '''Forgot password / OTP'''.<br />
# Enter your username. You should receive a recovery email in your inbox.<br />
# Open the email and follow the recovery link.<br />
# Scan the new QR code with your authenticator app.<br />
# Enter the generated OTP code and a device name.<br />
#* The device name is mandatory.<br />
# Change your password when prompted.<br />
#* This is required for security reasons.<br />
# Log in again as usual.<br />
#* On the OTP selection page, you will see two devices.<br />
#* Select the device with the name you just entered.<br />
#* The old one will usually appear as '''unnamed''' (if no name was set previously).<br />
<br />
Afterwards, we recommend removing the old OTP factor from the [https://idp.pic.es/realms/PIC/account/account-security/signing-in user profile section].<br />
<br />
=== What is the simplest way to avoid repeated SSH authentication? ===<br />
The easiest way is to configure your SSH connection using '''ControlMaster''', which allows multiple sessions to reuse a single authenticated connection.<br />
<br />
<ol><br />
<li>On your client machine, edit the file <code>~/.ssh/config</code> and add the following lines:<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Instead of <code>Host *</code>, you can restrict this to a specific host, for example:<br />
<br />
<pre><br />
Host ui04.pic.es<br />
</pre><br />
</li><br />
<br />
<li>Initiate the first SSH connection and authenticate via the provided URL:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
</pre><br />
<br />
You will see something like:<br />
<br />
<pre><br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Complete the authentication in your browser, then press ENTER.<br />
</li><br />
<br />
<li>Once authenticated, the SSH connection remains open in the background, and you can safely close your terminal session.<br />
<br />
The connection will persist indefinitely (until machine reboot or a network interruption).<br />
</li><br />
<br />
<li>Subsequent SSH connections to the same host will reuse the existing session and will not require authentication again:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
</pre><br />
</li><br />
</ol></div>Talladahttps://pwiki.pic.es/index.php?title=Main_Page&diff=1346Main Page2026-04-15T13:42:19Z<p>Tallada: </p>
<hr />
<div>== Getting started ==<br />
* [[PIC description|Introduction to PIC]]<br />
* [[PIC account|Get a PIC account]]<br />
* [[PIC_User_Manual | User manual]]<br />
* [[faq| Frequently asked questions]]<br />
<br />
== Services ==<br />
=== User interfaces ===<br />
* [[Login machines]]<br />
* [[JupyterHub]]<br />
<br />
=== Distributed computing ===<br />
* [[HTCondor]]<br />
* [[Dask]]<br />
* Spark:<br />
** [[Spark on Jupyter|on Jupyter]]<br />
** [[Spark_on_farm|on HTCondor]]<br />
<br />
=== Storage ===<br />
* [[Storage]]<br />
* [[Hadoop Distributed File System (HDFS)]]<br />
* [[HDFS Access via VOSpace]]<br />
* [[Transferring data to/from PIC]]<br />
<br />
=== Other services ===<br />
* [[Gitlab]]<br />
* [[CosmoHub]]<br />
<br />
== Experiments ==<br />
* [[Euclid]]<br />
* [[AGN ICE]]<br />
* [[ICFO]]</div>Talladahttps://pwiki.pic.es/index.php?title=Faq&diff=1345Faq2026-04-15T13:32:08Z<p>Tallada: </p>
<hr />
<div>== How do I reset my password? ==<br />
You can reset your password using the following link:<br />
https://www.pic.es/user/auth/forgotpw<br />
<br />
== Can an undergraduate student in my group have an account? ==<br />
Yes. Undergraduate students can have PIC accounts without any problem.<br />
<br />
== How do I get a Kerberos token? ==<br />
A valid token is needed to interface with some PIC services, like the Hadoop platform (HDFS, Hive, Spark, ...) or CTA.<br />
Tokens are issued for the machine in which they are requested, if you hop to another machine, you may need to get another token.<br />
<br />
In order to get a token, run the following commands on a terminal.<br />
If you are on JupyterLab, use ''Launcher'' → ''Terminal''.<br />
<br />
<pre><br />
kinit -n -c ~/.fast.ccache @PIC.ES<br />
kinit -T ~/.fast.ccache<br />
</pre><br />
<br />
You could also define an alias for your own convenience:<br />
<br />
<pre><br />
alias kinit="kinit -n -c ~/.fast.ccache @PIC.ES; kinit -T ~/.fast.ccache"<br />
</pre><br />
<br />
Optionally, you can check the presence and expiration date of your token using:<br />
<br />
<pre><br />
klist<br />
</pre><br />
<br />
== I lost access to my OTP, how can I reset it? ==<br />
Follow these steps to recover your account and configure a new OTP token.<br />
<br />
# On the Keycloak login page, click '''Forgot password / OTP'''.<br />
# Enter your username. You should receive a recovery email in your inbox.<br />
# Open the email and follow the recovery link.<br />
# Scan the new QR code with your authenticator app.<br />
# Enter the generated OTP code and a device name.<br />
#* The device name is mandatory.<br />
# Change your password when prompted.<br />
#* This is required for security reasons.<br />
# Log in again as usual.<br />
#* On the OTP selection page, you will see two devices.<br />
#* Select the device with the name you just entered.<br />
#* The old one will usually appear as '''unnamed''' (if no name was set previously).<br />
<br />
Afterwards, we recommend removing the old OTP factor from the [https://idp.pic.es/realms/PIC/account/account-security/signing-in user profile section].</div>Talladahttps://pwiki.pic.es/index.php?title=Faq&diff=1344Faq2026-04-15T13:24:23Z<p>Tallada: </p>
<hr />
<div>== How do I reset my password? ==<br />
You can reset your password using the following link:<br />
https://www.pic.es/user/auth/forgotpw<br />
<br />
== Can an undergraduate student in my group have an account? ==<br />
Yes. Undergraduate students can have PIC accounts without any problem.<br />
<br />
== How do I get a Kerberos token? ==<br />
A valid token is needed to interface with some PIC services, like the Hadoop platform (HDFS, Hive, Spark, ...) or CTA.<br />
Tokens are issued for the machine in which they are requested, if you hop to another machine, you may need to get another token.<br />
<br />
In order to get a token, run the following commands on a terminal.<br />
If you are on JupyterLab, use ''Launcher'' → ''Terminal''.<br />
<br />
<pre><br />
kinit -n -c ~/.fast.ccache @PIC.ES<br />
kinit -T ~/.fast.ccache<br />
</pre><br />
<br />
You could also define an alias for your own convenience:<br />
<br />
<pre><br />
alias kinit="kinit -n -c ~/.fast.ccache @PIC.ES; kinit -T ~/.fast.ccache"<br />
</pre><br />
<br />
Optionally, you can check the presence and expiration date of your token using:<br />
<br />
<pre><br />
klist<br />
</pre></div>Talladahttps://pwiki.pic.es/index.php?title=Main_Page&diff=1343Main Page2026-04-15T13:18:36Z<p>Tallada: </p>
<hr />
<div>== Getting started ==<br />
* [[PIC description|Introduction to PIC]]<br />
* [[PIC account|Get a PIC account]]<br />
* [[Lost OTP|Reset your OTP]]<br />
* [[PIC_User_Manual | User manual]]<br />
* [[faq| Frequently asked questions]]<br />
<br />
== Services ==<br />
=== User interfaces ===<br />
* [[Login machines]]<br />
* [[JupyterHub]]<br />
<br />
=== Distributed computing ===<br />
* [[HTCondor]]<br />
* [[Dask]]<br />
* Spark:<br />
** [[Spark on Jupyter|on Jupyter]]<br />
** [[Spark_on_farm|on HTCondor]]<br />
<br />
=== Storage ===<br />
* [[Storage]]<br />
* [[Hadoop Distributed File System (HDFS)]]<br />
* [[HDFS Access via VOSpace]]<br />
* [[Transferring data to/from PIC]]<br />
<br />
=== Other services ===<br />
* [[Gitlab]]<br />
* [[CosmoHub]]<br />
<br />
== Experiments ==<br />
* [[Euclid]]<br />
* [[AGN ICE]]<br />
* [[ICFO]]</div>Talladahttps://pwiki.pic.es/index.php?title=Spark_on_Jupyter&diff=1340Spark on Jupyter2026-04-08T08:51:24Z<p>Tallada: Created page with "= Accessing PIC Big Data Services from JupyterLab = PIC's Jupyter service is integrated with the Big Data platform. This allows you to access services such as '''HDFS''', '''..."</p>
<hr />
<div>= Accessing PIC Big Data Services from JupyterLab =<br />
<br />
PIC's Jupyter service is integrated with the Big Data platform. This allows you to access services such as '''HDFS''', '''Hive''', and '''Spark''' directly from your JupyterLab environment.<br />
<br />
== Requirements ==<br />
<br />
=== 1. Kerberos Authentication ===<br />
<br />
You must have a '''valid Kerberos ticket''' before accessing any Big Data service.<br />
<br />
==== Step-by-step ====<br />
<br />
# Open a terminal session on the same node where your Jupyter session is running.<br />
#* In JupyterLab, use the ''Launcher'' → ''Terminal''.<br />
# Run the following commands:<br />
<br />
<pre><br />
kinit -n -c ~/.fast.ccache @PIC.ES<br />
kinit -T ~/.fast.ccache<br />
</pre><br />
<br />
==== Verify your ticket ====<br />
<br />
<pre><br />
klist<br />
</pre><br />
<br />
You should see a valid ticket with a non-expired timestamp.<br />
<br />
==== Notes ====<br />
<br />
* If the ticket expires, you must repeat the process.<br />
* Without a valid Kerberos ticket, access to HDFS, Hive, or Spark will fail.<br />
<br />
==== Quick test (HDFS access) ====<br />
<br />
<pre><br />
hdfs dfs -ls /<br />
</pre><br />
<br />
If this works, your authentication is correctly configured.<br />
<br />
---<br />
<br />
=== 2. Python Environment ===<br />
<br />
You need specific Python packages depending on the service:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Service !! Required packages<br />
|-<br />
| Spark || findspark, pyspark<br />
|-<br />
| Hive || pyhive[hive_pure_sasl, kerberos]<br />
|}<br />
<br />
==== Installation example ====<br />
<br />
<pre><br />
pip install findspark pyspark<br />
pip install "pyhive[hive_pure_sasl, kerberos]"<br />
</pre><br />
<br />
---<br />
<br />
== Using Spark from Jupyter ==<br />
<br />
=== Setup ===<br />
<br />
<pre><br />
import os<br />
<br />
# Environment configuration<br />
os.environ["HADOOP_HOME"] = "/usr/local/hadoop"<br />
os.environ["HADOOP_CONF_DIR"] = "/usr/local/hadoop/etc/hadoop"<br />
os.environ["HIVE_HOME"] = "/usr/local/hive"<br />
os.environ["HIVE_CONF_DIR"] = "/usr/local/hive/conf"<br />
<br />
import findspark<br />
findspark.init()<br />
</pre><br />
<br />
=== Create a Spark session ===<br />
<br />
<pre><br />
from pyspark.sql import SparkSession<br />
<br />
spark = (<br />
SparkSession.builder<br />
.appName("example")<br />
.enableHiveSupport()<br />
.getOrCreate()<br />
)<br />
<br />
sc = spark.sparkContext<br />
</pre><br />
<br />
=== Example: Read a Hive table ===<br />
<br />
<pre><br />
df = spark.sql("SELECT * FROM some_database.some_table LIMIT 10")<br />
df.show()<br />
</pre><br />
<br />
---<br />
<br />
== Using Hive (PyHive) from Jupyter ==<br />
<br />
=== Connect to Hive ===<br />
<br />
<pre><br />
from pyhive import hive<br />
<br />
conn = hive.connect(<br />
host="hsrv01.pic.es",<br />
port=10000,<br />
kerberos_service_name="hive",<br />
auth="KERBEROS",<br />
)<br />
<br />
cursor = conn.cursor()<br />
</pre><br />
<br />
=== Execute a query ===<br />
<br />
<pre><br />
query = "SELECT * FROM some_database.some_table LIMIT 10"<br />
cursor.execute(query)<br />
<br />
rows = cursor.fetchall()<br />
colnames = [c[0] for c in cursor.description]<br />
</pre><br />
<br />
=== Convert to Astropy Table (optional) ===<br />
<br />
<pre><br />
from astropy.table import Table<br />
<br />
table = Table(rows=rows, names=colnames)<br />
</pre><br />
<br />
---<br />
<br />
== Troubleshooting ==<br />
<br />
=== Kerberos errors ===<br />
* Run <code>klist</code> and check expiration<br />
* Re-run <code>kinit</code> if needed<br />
<br />
=== Hive connection fails ===<br />
* Ensure Kerberos ticket is valid<br />
* Verify correct host (<code>hsrv01.pic.es</code>)<br />
* Check that required Python packages are installed<br />
<br />
=== Spark does not start ===<br />
* Verify environment variables (<code>HADOOP_HOME</code>, <code>HIVE_HOME</code>)<br />
* Ensure <code>findspark.init()</code> is executed before creating the session<br />
<br />
---<br />
<br />
== Summary ==<br />
<br />
# Obtain a Kerberos ticket (<code>kinit</code>)<br />
# Install required Python packages<br />
# Configure environment variables<br />
# Use Spark or Hive from your notebook<br />
<br />
---<br />
<br />
For further assistance, contact PIC support (user.support@pic.es).</div>Talladahttps://pwiki.pic.es/index.php?title=Main_Page&diff=1339Main Page2026-04-08T08:17:49Z<p>Tallada: /* Distributed computing */</p>
<hr />
<div>== Getting started ==<br />
* [[PIC description|Introduction to PIC]]<br />
* [[PIC account|Get a PIC account]]<br />
* [[PIC_User_Manual | User manual]]<br />
* [[faq| Frequently asked questions]]<br />
<br />
== Services ==<br />
=== User interfaces ===<br />
* [[Login machines]]<br />
* [[JupyterHub]]<br />
<br />
=== Distributed computing ===<br />
* [[HTCondor]]<br />
* [[Dask]]<br />
* Spark:<br />
** [[Spark on Jupyter|on Jupyter]]<br />
** [[Spark_on_farm|on HTCondor]]<br />
<br />
=== Storage ===<br />
* [[Storage]]<br />
* [[Hadoop Distributed File System (HDFS)]]<br />
* [[HDFS Access via VOSpace]]<br />
* [[Transferring data to/from PIC]]<br />
<br />
<br />
<br />
=== Other services ===<br />
* [[Gitlab]]<br />
* [[CosmoHub]]<br />
<br />
== Experiments ==<br />
<br />
* [[Euclid]]<br />
* [[AGN ICE]]<br />
* [[ICFO]]</div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1227Transferring data to/from PIC2025-05-27T09:26:12Z<p>Tallada: /* Usage */</p>
<hr />
<div>= How to provide data access to PIC massive storage (dCache) =<br />
<br />
== Requirements ==<br />
<br />
* Install and configure Rclone<br />
* PIC credentials or macaroon<br />
<br />
== Install Rclone ==<br />
<br />
You can directly download the binary without installing anything. For instance, for a linux 64 bits machine:<br />
<br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
[...]<br />
$ unzip rclone-current-linux-amd64.zip<br />
<br />
<br />
Or if you prefer, you can install Rclone like the next example on a Ubuntu machine:<br />
<br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
<br />
== Configure Rclone ==<br />
<br />
You should have been given some credentials and the url of a WebDAV endpoint at PIC.<br />
With them, you just need to create the config in rclone:<br />
<br />
$ rclone config<br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic<br />
<br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
1 / 1Fichier<br />
\ (fichier)<br />
[...]<br />
Storage> webdav<br />
<br />
Option url.<br />
URL of http host to connect to.<br />
E.g. https://example.com.<br />
Enter a value.<br />
url> https://door04.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
<br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
Choose a number from below, or type in your own value.<br />
Press Enter to leave empty.<br />
1 / Nextcloud<br />
\ (nextcloud)<br />
[...]<br />
5 / Other site/service or software<br />
\ (other)<br />
vendor> other<br />
<br />
=== Using your PIC credentials ===<br />
<br />
If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank.<br />
<br />
Option user.<br />
User name.<br />
In case NTLM authentication is used, the username should be in the format 'Domain\User'.<br />
Enter a value. Press Enter to leave empty.<br />
user> YOUR_PIC_USERNAME<br />
<br />
Option pass.<br />
Password.<br />
Choose an alternative below. Press Enter for the default (n).<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank (default)<br />
y/g/n> y<br />
Enter the password:<br />
password: YOUR_PIC_PASSWORD<br />
Confirm the password:<br />
Password: YOUR_PIC_PASSWORD<br />
<br />
=== Using a Macaroon token ===<br />
<br />
If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank<br />
<br />
Option bearer_token.<br />
Bearer token instead of user/pass (e.g. a Macaroon).<br />
Enter a value. Press Enter to leave empty.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
<br />
=== Review settings ===<br />
<br />
At the end, just review the information you entered and confirm.<br />
<br />
Edit advanced config?<br />
y) Yes<br />
n) No (default)<br />
y/n> n<br />
<br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://door04.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- user: YOUR_PIC_USERNAME<br />
- pass: *** ENCRYPTED ***<br />
Keep this "pic" remote?<br />
y) Yes this is OK (default)<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
<br />
Current remotes:<br />
<br />
Name Type<br />
==== ====<br />
pic webdav<br />
<br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
<br />
Once done, you can use command line to browse and download/upload data.<br />
<br />
=== Usage ===<br />
<br />
* List a remote PIC directory: <br />
rclone lsd <name>:<path>`<br />
<br />
* Download a remote directory from PIC<br />
rclone copy <name>:<path> <local_path><br />
<br />
* Upload a local directory to PIC<br />
rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path><br />
<br />
When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files <br />
--check-first -P --stats-one-line --transfers <n_transfers> --size-only<br />
<br />
If uploading lots onto directories with lots of files (>1000), please use:<br />
--no-traverse<br />
<br />
If uploading files larger than 200 MB, also use<br />
--multi-thread-streams 1<br />
<br />
If uploading very large files (>10G), also use the following to allow more time to compute the checksums<br />
--timeout=15m<br />
<br />
See rclone manual for more extensive documentation https://rclone.org/docs/<br />
<br />
== Obtaining a macaroon (for contacts) ==<br />
<br />
Macaroons are valid up to 7 days.<br />
<br />
For downloading data (read-only permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
For uploading data (full permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre></div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1154Transferring data to/from PIC2024-06-05T13:17:40Z<p>Tallada: </p>
<hr />
<div>= How to provide data access to PIC massive storage (dCache) =<br />
<br />
== Requirements ==<br />
<br />
* Install and configure Rclone<br />
* PIC credentials or macaroon<br />
<br />
== Install Rclone ==<br />
<br />
You can directly download the binary without installing anything. For instance, for a linux 64 bits machine:<br />
<br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
[...]<br />
$ unzip rclone-current-linux-amd64.zip<br />
<br />
<br />
Or if you prefer, you can install Rclone like the next example on a Ubuntu machine:<br />
<br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
<br />
== Configure Rclone ==<br />
<br />
You should have been given some credentials and the url of a WebDAV endpoint at PIC.<br />
With them, you just need to create the config in rclone:<br />
<br />
$ rclone config<br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic<br />
<br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
1 / 1Fichier<br />
\ (fichier)<br />
[...]<br />
Storage> webdav<br />
<br />
Option url.<br />
URL of http host to connect to.<br />
E.g. https://example.com.<br />
Enter a value.<br />
url> https://door04.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
<br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
Choose a number from below, or type in your own value.<br />
Press Enter to leave empty.<br />
1 / Nextcloud<br />
\ (nextcloud)<br />
[...]<br />
5 / Other site/service or software<br />
\ (other)<br />
vendor> other<br />
<br />
=== Using your PIC credentials ===<br />
<br />
If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank.<br />
<br />
Option user.<br />
User name.<br />
In case NTLM authentication is used, the username should be in the format 'Domain\User'.<br />
Enter a value. Press Enter to leave empty.<br />
user> YOUR_PIC_USERNAME<br />
<br />
Option pass.<br />
Password.<br />
Choose an alternative below. Press Enter for the default (n).<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank (default)<br />
y/g/n> y<br />
Enter the password:<br />
password: YOUR_PIC_PASSWORD<br />
Confirm the password:<br />
Password: YOUR_PIC_PASSWORD<br />
<br />
=== Using a Macaroon token ===<br />
<br />
If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank<br />
<br />
Option bearer_token.<br />
Bearer token instead of user/pass (e.g. a Macaroon).<br />
Enter a value. Press Enter to leave empty.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
<br />
=== Review settings ===<br />
<br />
At the end, just review the information you entered and confirm.<br />
<br />
Edit advanced config?<br />
y) Yes<br />
n) No (default)<br />
y/n> n<br />
<br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://door04.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- user: YOUR_PIC_USERNAME<br />
- pass: *** ENCRYPTED ***<br />
Keep this "pic" remote?<br />
y) Yes this is OK (default)<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
<br />
Current remotes:<br />
<br />
Name Type<br />
==== ====<br />
pic webdav<br />
<br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
<br />
Once done, you can use command line to browse and download/upload data.<br />
<br />
=== Usage ===<br />
<br />
* List a remote PIC directory: <br />
rclone lsd <name>:<path>`<br />
<br />
* Download a remote directory from PIC<br />
rclone copy <name>:<path> <local_path><br />
<br />
* Upload a local directory to PIC<br />
rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path><br />
<br />
When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files <br />
--check-first -P --stats-one-line --transfers <n_transfers> --size-only<br />
<br />
See rclone manual for more extensive documentation https://rclone.org/docs/<br />
<br />
== Obtaining a macaroon (for contacts) ==<br />
<br />
Macaroons are valid up to 7 days.<br />
<br />
For downloading data (read-only permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
For uploading data (full permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre></div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1153Transferring data to/from PIC2024-06-04T09:21:30Z<p>Tallada: /* Obtaining a macaroon (for contacts) */</p>
<hr />
<div>= How to provide data access to PIC massive storage (dCache) =<br />
<br />
== Requirements ==<br />
<br />
* Install and configure Rclone<br />
* PIC credentials or macaroon<br />
<br />
== Install Rclone ==<br />
<br />
You can directly download the binary without installing anything. For instance, for a linux 64 bits machine:<br />
<br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
[...]<br />
$ unzip rclone-current-linux-amd64.zip<br />
<br />
<br />
Or if you prefer, you can install Rclone like the next example on a Ubuntu machine:<br />
<br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
<br />
== Configure Rclone ==<br />
<br />
You should have been given some credentials and the url of a WebDAV endpoint at PIC.<br />
With them, you just need to create the config in rclone:<br />
<br />
$ rclone config<br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic<br />
<br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
1 / 1Fichier<br />
\ (fichier)<br />
[...]<br />
Storage> webdav<br />
<br />
Option url.<br />
URL of http host to connect to.<br />
E.g. https://example.com.<br />
Enter a value.<br />
url> https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
<br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
Choose a number from below, or type in your own value.<br />
Press Enter to leave empty.<br />
1 / Nextcloud<br />
\ (nextcloud)<br />
[...]<br />
5 / Other site/service or software<br />
\ (other)<br />
vendor> other<br />
<br />
=== Using your PIC credentials ===<br />
<br />
If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank.<br />
<br />
Option user.<br />
User name.<br />
In case NTLM authentication is used, the username should be in the format 'Domain\User'.<br />
Enter a value. Press Enter to leave empty.<br />
user> YOUR_PIC_USERNAME<br />
<br />
Option pass.<br />
Password.<br />
Choose an alternative below. Press Enter for the default (n).<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank (default)<br />
y/g/n> y<br />
Enter the password:<br />
password: YOUR_PIC_PASSWORD<br />
Confirm the password:<br />
Password: YOUR_PIC_PASSWORD<br />
<br />
=== Using a Macaroon token ===<br />
<br />
If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank<br />
<br />
Option bearer_token.<br />
Bearer token instead of user/pass (e.g. a Macaroon).<br />
Enter a value. Press Enter to leave empty.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
<br />
=== Review settings ===<br />
<br />
At the end, just review the information you entered and confirm.<br />
<br />
Edit advanced config?<br />
y) Yes<br />
n) No (default)<br />
y/n> n<br />
<br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- user: YOUR_PIC_USERNAME<br />
- pass: *** ENCRYPTED ***<br />
Keep this "pic" remote?<br />
y) Yes this is OK (default)<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
<br />
Current remotes:<br />
<br />
Name Type<br />
==== ====<br />
pic webdav<br />
<br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
<br />
Once done, you can use command line to browse and download/upload data.<br />
<br />
=== Usage ===<br />
<br />
* List a remote PIC directory: <br />
rclone lsd <name>:<path>`<br />
<br />
* Download a remote directory from PIC<br />
rclone copy <name>:<path> <local_path><br />
<br />
* Upload a local directory to PIC<br />
rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path><br />
<br />
When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files <br />
--check-first -P --stats-one-line --transfers <n_transfers> --size-only<br />
<br />
See rclone manual for more extensive documentation https://rclone.org/docs/<br />
<br />
== Obtaining a macaroon (for contacts) ==<br />
<br />
Macaroons are valid up to 7 days.<br />
<br />
For downloading data (read-only permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://webdav.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://webdav.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://webdav.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
For uploading data (full permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://webdav.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://webdav.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://webdav.pic.es:8460/"<br />
}<br />
}<br />
</pre></div>Talladahttps://pwiki.pic.es/index.php?title=Main_Page&diff=1101Main Page2023-12-20T14:54:12Z<p>Tallada: /* Services */</p>
<hr />
<div>== Getting started ==<br />
* [[PIC description|PIC in an image]]<br />
* [[PIC account|Get a PIC account]]<br />
* [[PIC_User_Manual | User manual]]<br />
<br />
== Services ==<br />
* [[HTCondor]]<br />
* [[Storage]]<br />
* [[JupyterHub]]<br />
* [[Gitlab]]<br />
* [[CosmoHub]]<br />
* Spark:<br />
** [[Spark on Hadoop|on Hadoop]]<br />
** [[Spark_on_farm|on HTCondor]]<br />
* [[Transferring data to/from PIC]]<br />
<br />
== Experiments ==<br />
<br />
* [[Euclid]]<br />
<br />
== More technical information ==<br />
<br />
* [[Storage Department]]</div>Talladahttps://pwiki.pic.es/index.php?title=DCache_webdav_i_dCacheView&diff=1100DCache webdav i dCacheView2023-12-20T14:53:29Z<p>Tallada: Tallada moved page DCache webdav i dCacheView to Transferring data to/from PIC</p>
<hr />
<div>#REDIRECT [[Transferring data to/from PIC]]</div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1099Transferring data to/from PIC2023-12-20T14:53:29Z<p>Tallada: Tallada moved page DCache webdav i dCacheView to Transferring data to/from PIC</p>
<hr />
<div>= How to provide data access to PIC massive storage (dCache) =<br />
<br />
== Requirements ==<br />
<br />
* Install and configure Rclone<br />
* PIC credentials or macaroon<br />
<br />
== Install Rclone ==<br />
<br />
You can directly download the binary without installing anything. For instance, for a linux 64 bits machine:<br />
<br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
[...]<br />
$ unzip rclone-current-linux-amd64.zip<br />
<br />
<br />
Or if you prefer, you can install Rclone like the next example on a Ubuntu machine:<br />
<br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
<br />
== Configure Rclone ==<br />
<br />
You should have been given some credentials and the url of a WebDAV endpoint at PIC.<br />
With them, you just need to create the config in rclone:<br />
<br />
$ rclone config<br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic<br />
<br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
1 / 1Fichier<br />
\ (fichier)<br />
[...]<br />
Storage> webdav<br />
<br />
Option url.<br />
URL of http host to connect to.<br />
E.g. https://example.com.<br />
Enter a value.<br />
url> https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
<br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
Choose a number from below, or type in your own value.<br />
Press Enter to leave empty.<br />
1 / Nextcloud<br />
\ (nextcloud)<br />
[...]<br />
5 / Other site/service or software<br />
\ (other)<br />
vendor> other<br />
<br />
=== Using your PIC credentials ===<br />
<br />
If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank.<br />
<br />
Option user.<br />
User name.<br />
In case NTLM authentication is used, the username should be in the format 'Domain\User'.<br />
Enter a value. Press Enter to leave empty.<br />
user> YOUR_PIC_USERNAME<br />
<br />
Option pass.<br />
Password.<br />
Choose an alternative below. Press Enter for the default (n).<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank (default)<br />
y/g/n> y<br />
Enter the password:<br />
password: YOUR_PIC_PASSWORD<br />
Confirm the password:<br />
Password: YOUR_PIC_PASSWORD<br />
<br />
=== Using a Macaroon token ===<br />
<br />
If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank<br />
<br />
Option bearer_token.<br />
Bearer token instead of user/pass (e.g. a Macaroon).<br />
Enter a value. Press Enter to leave empty.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
<br />
=== Review settings ===<br />
<br />
At the end, just review the information you entered and confirm.<br />
<br />
Edit advanced config?<br />
y) Yes<br />
n) No (default)<br />
y/n> n<br />
<br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- user: YOUR_PIC_USERNAME<br />
- pass: *** ENCRYPTED ***<br />
Keep this "pic" remote?<br />
y) Yes this is OK (default)<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
<br />
Current remotes:<br />
<br />
Name Type<br />
==== ====<br />
pic webdav<br />
<br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
<br />
Once done, you can use command line to browse and download/upload data.<br />
<br />
=== Usage ===<br />
<br />
* List a remote PIC directory: <br />
rclone lsd <name>:<path>`<br />
<br />
* Download a remote directory from PIC<br />
rclone copy <name>:<path> <local_path><br />
<br />
* Upload a local directory to PIC<br />
rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path><br />
<br />
When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files <br />
--check-first -P --stats-one-line --transfers <n_transfers> --size-only<br />
<br />
See rclone manual for more extensive documentation https://rclone.org/docs/<br />
<br />
== Obtaining a macaroon (for contacts) ==<br />
<br />
Macaroons are valid up to 7 days.<br />
<br />
For downloading data (read-only permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USERNAME} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://webdav.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://webdav.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://webdav.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
For uploading data (full permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USERNAME} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://webdav.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://webdav.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://webdav.pic.es:8460/"<br />
}<br />
}<br />
</pre></div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1098Transferring data to/from PIC2023-12-15T12:59:42Z<p>Tallada: /* How to provide data access to PIC massive storage (dCache) = */</p>
<hr />
<div>= How to provide data access to PIC massive storage (dCache) =<br />
<br />
== Requirements ==<br />
<br />
* Install and configure Rclone<br />
* PIC credentials or macaroon<br />
<br />
== Install Rclone ==<br />
<br />
You can directly download the binary without installing anything. For instance, for a linux 64 bits machine:<br />
<br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
[...]<br />
$ unzip rclone-current-linux-amd64.zip<br />
<br />
<br />
Or if you prefer, you can install Rclone like the next example on a Ubuntu machine:<br />
<br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
<br />
== Configure Rclone ==<br />
<br />
You should have been given some credentials and the url of a WebDAV endpoint at PIC.<br />
With them, you just need to create the config in rclone:<br />
<br />
$ rclone config<br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic<br />
<br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
1 / 1Fichier<br />
\ (fichier)<br />
[...]<br />
Storage> webdav<br />
<br />
Option url.<br />
URL of http host to connect to.<br />
E.g. https://example.com.<br />
Enter a value.<br />
url> https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
<br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
Choose a number from below, or type in your own value.<br />
Press Enter to leave empty.<br />
1 / Nextcloud<br />
\ (nextcloud)<br />
[...]<br />
5 / Other site/service or software<br />
\ (other)<br />
vendor> other<br />
<br />
=== Using your PIC credentials ===<br />
<br />
If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank.<br />
<br />
Option user.<br />
User name.<br />
In case NTLM authentication is used, the username should be in the format 'Domain\User'.<br />
Enter a value. Press Enter to leave empty.<br />
user> YOUR_PIC_USERNAME<br />
<br />
Option pass.<br />
Password.<br />
Choose an alternative below. Press Enter for the default (n).<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank (default)<br />
y/g/n> y<br />
Enter the password:<br />
password: YOUR_PIC_PASSWORD<br />
Confirm the password:<br />
Password: YOUR_PIC_PASSWORD<br />
<br />
=== Using a Macaroon token ===<br />
<br />
If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank<br />
<br />
Option bearer_token.<br />
Bearer token instead of user/pass (e.g. a Macaroon).<br />
Enter a value. Press Enter to leave empty.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
<br />
=== Review settings ===<br />
<br />
At the end, just review the information you entered and confirm.<br />
<br />
Edit advanced config?<br />
y) Yes<br />
n) No (default)<br />
y/n> n<br />
<br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- user: YOUR_PIC_USERNAME<br />
- pass: *** ENCRYPTED ***<br />
Keep this "pic" remote?<br />
y) Yes this is OK (default)<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
<br />
Current remotes:<br />
<br />
Name Type<br />
==== ====<br />
pic webdav<br />
<br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
<br />
Once done, you can use command line to browse and download/upload data.<br />
<br />
=== Usage ===<br />
<br />
* List a remote PIC directory: <br />
rclone lsd <name>:<path>`<br />
<br />
* Download a remote directory from PIC<br />
rclone copy <name>:<path> <local_path><br />
<br />
* Upload a local directory to PIC<br />
rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path><br />
<br />
When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files <br />
--check-first -P --stats-one-line --transfers <n_transfers> --size-only<br />
<br />
See rclone manual for more extensive documentation https://rclone.org/docs/<br />
<br />
== Obtaining a macaroon (for contacts) ==<br />
<br />
Macaroons are valid up to 7 days.<br />
<br />
For downloading data (read-only permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USERNAME} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://webdav.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://webdav.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://webdav.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
For uploading data (full permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USERNAME} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://webdav.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://webdav.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://webdav.pic.es:8460/"<br />
}<br />
}<br />
</pre></div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1097Transferring data to/from PIC2023-12-15T12:59:03Z<p>Tallada: </p>
<hr />
<div>= How to provide data access to PIC massive storage (dCache) ==<br />
<br />
== Requirements ==<br />
<br />
* Install and configure Rclone<br />
* PIC credentials or macaroon<br />
<br />
== Install Rclone ==<br />
<br />
You can directly download the binary without installing anything. For instance, for a linux 64 bits machine:<br />
<br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
[...]<br />
$ unzip rclone-current-linux-amd64.zip<br />
<br />
<br />
Or if you prefer, you can install Rclone like the next example on a Ubuntu machine:<br />
<br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
<br />
== Configure Rclone ==<br />
<br />
You should have been given some credentials and the url of a WebDAV endpoint at PIC.<br />
With them, you just need to create the config in rclone:<br />
<br />
$ rclone config<br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic<br />
<br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
1 / 1Fichier<br />
\ (fichier)<br />
[...]<br />
Storage> webdav<br />
<br />
Option url.<br />
URL of http host to connect to.<br />
E.g. https://example.com.<br />
Enter a value.<br />
url> https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
<br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
Choose a number from below, or type in your own value.<br />
Press Enter to leave empty.<br />
1 / Nextcloud<br />
\ (nextcloud)<br />
[...]<br />
5 / Other site/service or software<br />
\ (other)<br />
vendor> other<br />
<br />
=== Using your PIC credentials ===<br />
<br />
If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank.<br />
<br />
Option user.<br />
User name.<br />
In case NTLM authentication is used, the username should be in the format 'Domain\User'.<br />
Enter a value. Press Enter to leave empty.<br />
user> YOUR_PIC_USERNAME<br />
<br />
Option pass.<br />
Password.<br />
Choose an alternative below. Press Enter for the default (n).<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank (default)<br />
y/g/n> y<br />
Enter the password:<br />
password: YOUR_PIC_PASSWORD<br />
Confirm the password:<br />
Password: YOUR_PIC_PASSWORD<br />
<br />
=== Using a Macaroon token ===<br />
<br />
If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank<br />
<br />
Option bearer_token.<br />
Bearer token instead of user/pass (e.g. a Macaroon).<br />
Enter a value. Press Enter to leave empty.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
<br />
=== Review settings ===<br />
<br />
At the end, just review the information you entered and confirm.<br />
<br />
Edit advanced config?<br />
y) Yes<br />
n) No (default)<br />
y/n> n<br />
<br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- user: YOUR_PIC_USERNAME<br />
- pass: *** ENCRYPTED ***<br />
Keep this "pic" remote?<br />
y) Yes this is OK (default)<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
<br />
Current remotes:<br />
<br />
Name Type<br />
==== ====<br />
pic webdav<br />
<br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
<br />
Once done, you can use command line to browse and download/upload data.<br />
<br />
=== Usage ===<br />
<br />
* List a remote PIC directory: <br />
rclone lsd <name>:<path>`<br />
<br />
* Download a remote directory from PIC<br />
rclone copy <name>:<path> <local_path><br />
<br />
* Upload a local directory to PIC<br />
rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path><br />
<br />
When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files <br />
--check-first -P --stats-one-line --transfers <n_transfers> --size-only<br />
<br />
See rclone manual for more extensive documentation https://rclone.org/docs/<br />
<br />
== Obtaining a macaroon (for contacts) ==<br />
<br />
Macaroons are valid up to 7 days.<br />
<br />
For downloading data (read-only permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USERNAME} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://webdav.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://webdav.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://webdav.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
For uploading data (full permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USERNAME} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://webdav.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://webdav.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://webdav.pic.es:8460/"<br />
}<br />
}<br />
</pre></div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1096Transferring data to/from PIC2023-12-06T11:24:02Z<p>Tallada: </p>
<hr />
<div>== How to provide data access to external users without PIC account ==<br />
<br />
<br />
==== dCache webdav door ====<br />
<br />
https://webdav.pic.es or https://webdav.pic.es:8460<br />
<br />
==== dCacheView frontend (optional) ====<br />
<br />
https://webdav.pic.es:8470<br />
<br />
==== Authentication ====<br />
<br />
Either PIC credentials (user/password) or a macaroon (https://dcache.org/old/manuals/UserGuide-7.0/macaroons.shtml).<br />
Any authenticated user can get a macaroon for an external user. Remember to add proper caveats to restrict privileges.<br />
In the next example, I retrieve a macaroon with 7 days validity, that can only download and list, and can only access the subtree of the provided URL.<br />
<br />
<pre><br />
$ curl -u tallada -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://webdav.pic.es:8460/pnfs/pic.es/data/astro/euclid/disk/simulations/ComplementarySimulations<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav.pic.es:8460/pnfs/pic.es/data/astro/euclid/disk/simulations/ComplementarySimulations?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://webdav.pic.es:8460/pnfs/pic.es/data/astro/euclid/disk/simulations/ComplementarySimulations",<br />
"base": "https://webdav.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
==== Browser access ====<br />
<br />
You can access the data either through the webdav door or dCacheView. If you have PIC credentials (user/password), just input them when requested by the browser.<br />
<br />
If you only have a macaroon token, you can access the webdav using the preauthenticated URL (either targetWithMacaroon or baseWithMacaroon). You will only be able to browse to the restricted subtree.<br />
You can also use command line clients like cURL or wget.<br />
<br />
In order to use the macarron with dCacheView, DO NOT ENTER any credentials at the login step. Instead, click the three bar menu at the top right, a panel on the left will open. On that panel, select the "Shared files" option. Then, clic the "Add" button on the bottom right and enter the macaroon token.<br />
<br />
==== rclone ====<br />
<br />
We can use rclone on Windows, Linux and MacOS to be able to automatize data transfer, even mount the remote data as a local filesystem.<br />
First, follow the instructions to install rclone in your host (https://rclone.org/install/).<br />
Note that for mounting you might need additional software (https://rclone.org/commands/rclone_mount/)<br />
<br />
Then, create a new remote, either using the "rclone config" command line, or opening the web GUI with the command "rclone rcd --rc-web-gui". You might need to add "--no-check-certificates" in some cases.<br />
Provide the following info for the remote:<br />
* name: "pic" or any other short identifier<br />
* provider: webdav<br />
* url: https://webdav.pic.es (you can use the full path here to to restrict the visible tree)<br />
* username/password: if you have PIC credentials, otherwise leave empty and use macaroon below.<br />
* bearer token: MDANOxGCg (the macaroon token)<br />
<br />
Once done, you can use either the command line or the explorer tab to browse and download data.<br />
<br />
Most common command lines:<br />
<br />
* List a remote PIC directory: <br />
rclone lsd <name>:<path>`<br />
<br />
* Copy a local directory to PIC<br />
rclone copy <local_dir> <name>:<path><br />
<br />
I recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files <br />
rclone --check-first -P --stats-one-line --transfers <n_transfers> --size-only copy <local_dir> <name>:<path><br />
<br />
You can also mount the remote data to a local path. Use the "Mounts" tab in the web GUI, select the remote filesystem and a local folder (or drive letter in Windows), and press create.<br />
<br />
Enjoy!</div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1095Transferring data to/from PIC2023-12-06T11:23:49Z<p>Tallada: </p>
<hr />
<div>== How to provide data access to external users without PIC account ==<br />
<br />
<br />
==== dCache webdav door ====<br />
<br />
https://webdav.pic.es or https://webdav.pic.es:8460<br />
<br />
==== dCacheView frontend (optional) ====<br />
<br />
https://webdav-mice.pic.es:8470<br />
<br />
==== Authentication ====<br />
<br />
Either PIC credentials (user/password) or a macaroon (https://dcache.org/old/manuals/UserGuide-7.0/macaroons.shtml).<br />
Any authenticated user can get a macaroon for an external user. Remember to add proper caveats to restrict privileges.<br />
In the next example, I retrieve a macaroon with 7 days validity, that can only download and list, and can only access the subtree of the provided URL.<br />
<br />
<pre><br />
$ curl -u tallada -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://webdav.pic.es:8460/pnfs/pic.es/data/astro/euclid/disk/simulations/ComplementarySimulations<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav.pic.es:8460/pnfs/pic.es/data/astro/euclid/disk/simulations/ComplementarySimulations?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://webdav.pic.es:8460/pnfs/pic.es/data/astro/euclid/disk/simulations/ComplementarySimulations",<br />
"base": "https://webdav.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
==== Browser access ====<br />
<br />
You can access the data either through the webdav door or dCacheView. If you have PIC credentials (user/password), just input them when requested by the browser.<br />
<br />
If you only have a macaroon token, you can access the webdav using the preauthenticated URL (either targetWithMacaroon or baseWithMacaroon). You will only be able to browse to the restricted subtree.<br />
You can also use command line clients like cURL or wget.<br />
<br />
In order to use the macarron with dCacheView, DO NOT ENTER any credentials at the login step. Instead, click the three bar menu at the top right, a panel on the left will open. On that panel, select the "Shared files" option. Then, clic the "Add" button on the bottom right and enter the macaroon token.<br />
<br />
==== rclone ====<br />
<br />
We can use rclone on Windows, Linux and MacOS to be able to automatize data transfer, even mount the remote data as a local filesystem.<br />
First, follow the instructions to install rclone in your host (https://rclone.org/install/).<br />
Note that for mounting you might need additional software (https://rclone.org/commands/rclone_mount/)<br />
<br />
Then, create a new remote, either using the "rclone config" command line, or opening the web GUI with the command "rclone rcd --rc-web-gui". You might need to add "--no-check-certificates" in some cases.<br />
Provide the following info for the remote:<br />
* name: "pic" or any other short identifier<br />
* provider: webdav<br />
* url: https://webdav.pic.es (you can use the full path here to to restrict the visible tree)<br />
* username/password: if you have PIC credentials, otherwise leave empty and use macaroon below.<br />
* bearer token: MDANOxGCg (the macaroon token)<br />
<br />
Once done, you can use either the command line or the explorer tab to browse and download data.<br />
<br />
Most common command lines:<br />
<br />
* List a remote PIC directory: <br />
rclone lsd <name>:<path>`<br />
<br />
* Copy a local directory to PIC<br />
rclone copy <local_dir> <name>:<path><br />
<br />
I recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files <br />
rclone --check-first -P --stats-one-line --transfers <n_transfers> --size-only copy <local_dir> <name>:<path><br />
<br />
You can also mount the remote data to a local path. Use the "Mounts" tab in the web GUI, select the remote filesystem and a local folder (or drive letter in Windows), and press create.<br />
<br />
Enjoy!</div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=934Transferring data to/from PIC2021-12-23T11:49:32Z<p>Tallada: </p>
<hr />
<div>== How to provide data access to external users without PIC account ==<br />
<br />
I will use an example for MICE<br />
<br />
<br />
==== dCache webdav door ====<br />
<br />
https://webdav-mice.pic.es:8454<br />
<br />
==== dCacheView frontend (optional) ====<br />
<br />
https://webdav-mice.pic.es:8464<br />
<br />
==== Authentication ====<br />
<br />
Either PIC credentials (user/password) or a macaroon (https://dcache.org/old/manuals/UserGuide-7.0/macaroons.shtml).<br />
Any authenticated user can get a macaroon for an external user. Remember to add proper caveats to restrict privileges.<br />
In the next example, I retrieve a macaroon with 7 days validity, that can only download and list, and can only access the subtree of the provided URL.<br />
<br />
<pre><br />
$ curl -u tallada -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://webdav-mice.pic.es:8454/tape/raw/production/N4096_L3072_LC3/maps/n8192/kmapgcpy/<br />
<br />
{<br />
"macaroon": "MDANOxGCg",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav-mice.pic.es:8454/tape/raw/production/N4096_L3072_LC3/maps/n8192/kmapgcpy/?authz=MDANOxGCg",<br />
"baseWithMacaroon": "https://webdav-mice.pic.es:8454/?authz=MDANOxGCg",<br />
"target": "https://webdav-mice.pic.es:8454/tape/raw/production/N4096_L3072_LC3/maps/n8192/kmapgcpy/",<br />
"base": "https://webdav-mice.pic.es:8454/"<br />
}<br />
}<br />
</pre><br />
<br />
==== Browser access ====<br />
<br />
You can access the data either through the webdav door or dCacheView. If you have PIC credentials (user/password), just input them when requested by the browser.<br />
<br />
If you only have a macaroon token, you can access the webdav using the preauthenticated URL (either targetWithMacaroon or baseWithMacaroon). You will only be able to browse to the restricted subtree.<br />
You can also use command line clients like cURL or wget.<br />
<br />
In order to use the macarron with dCacheView, DO NOT ENTER any credentials at the login step. Instead, click the three bar menu at the top right, a panel on the left will open. On that panel, select the "Shared files" option. Then, clic the "Add" button on the bottom right and enter the macaroon token.<br />
<br />
==== rclone ====<br />
<br />
We can use rclone on Windows, Linux and MacOS to be able to automatize data transfer, even mount the remote data as a local filesystem.<br />
First, follow the instructions to install rclone in your host (https://rclone.org/install/).<br />
Note that for mounting you might need additional software (https://rclone.org/commands/rclone_mount/)<br />
<br />
Then, open the web GUI with the command "rclone rcd --rc-web-gui". You might need to add "--no-check-certificates" in some cases.<br />
Once opened, create a config with the following info:<br />
* provider: webdav<br />
* url: https://webdav-mice.pic.es:8454 (either the base or target url)<br />
* username/password: if you have PIC credentials<br />
* bearer token: MDANOxGCg (the macaroon token)<br />
<br />
Once done, you can use the explorer tab to browse and download data.<br />
<br />
You can also mount the remote data to a local path. Use the "Mounts" tab, select the remote filesystem and a local folder (or diver letter in Windows), and press create.<br />
<br />
Enjoy!</div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=933Transferring data to/from PIC2021-12-23T11:48:34Z<p>Tallada: </p>
<hr />
<div>== How to provide data access to external users without PIC account ==<br />
<br />
I will use an example for MICE<br />
<br />
=== Steps ===<br />
<br />
1) dCache webdav door that exports your data.<br />
<br />
https://webdav-mice.pic.es:8454<br />
<br />
2) dCacheView frontend (optional)<br />
<br />
https://webdav-mice.pic.es:8464<br />
<br />
3) Authentication<br />
<br />
Either PIC credentials (user/password) or a macaroon (https://dcache.org/old/manuals/UserGuide-7.0/macaroons.shtml).<br />
Any authenticated user can get a macaroon for an external user. Remember to add proper caveats to restrict privileges.<br />
In the next example, I retrieve a macaroon with 7 days validity, that can only download and list, and can only access the subtree of the provided URL.<br />
<br />
<pre><br />
$ curl -u tallada -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://webdav-mice.pic.es:8454/tape/raw/production/N4096_L3072_LC3/maps/n8192/kmapgcpy/<br />
<br />
{<br />
"macaroon": "MDANOxGCg",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav-mice.pic.es:8454/tape/raw/production/N4096_L3072_LC3/maps/n8192/kmapgcpy/?authz=MDANOxGCg",<br />
"baseWithMacaroon": "https://webdav-mice.pic.es:8454/?authz=MDANOxGCg",<br />
"target": "https://webdav-mice.pic.es:8454/tape/raw/production/N4096_L3072_LC3/maps/n8192/kmapgcpy/",<br />
"base": "https://webdav-mice.pic.es:8454/"<br />
}<br />
}<br />
</pre><br />
<br />
4) Browser access<br />
<br />
You can access the data either through the webdav door or dCacheView. If you have PIC credentials (user/password), just input them when requested by the browser.<br />
<br />
If you only have a macaroon token, you can access the webdav using the preauthenticated URL (either targetWithMacaroon or baseWithMacaroon). You will only be able to browse to the restricted subtree.<br />
You can also use command line clients like cURL or wget.<br />
<br />
In order to use the macarron with dCacheView, DO NOT ENTER any credentials at the login step. Instead, click the three bar menu at the top right, a panel on the left will open. On that panel, select the "Shared files" option. Then, clic the "Add" button on the bottom right and enter the macaroon token.<br />
<br />
5) rclone<br />
<br />
We can use rclone on Windows, Linux and MacOS to be able to automatize data transfer, even mount the remote data as a local filesystem.<br />
First, follow the instructions to install rclone in your host (https://rclone.org/install/).<br />
Note that for mounting you might need additional software (https://rclone.org/commands/rclone_mount/)<br />
<br />
Then, open the web GUI with the command "rclone rcd --rc-web-gui". You might need to add "--no-check-certificates" in some cases.<br />
Once opened, create a config with the following info:<br />
* provider: webdav<br />
* url: https://webdav-mice.pic.es:8454 (either the base or target url)<br />
* username/password: if you have PIC credentials<br />
* bearer token: MDANOxGCg (the macaroon token)<br />
<br />
Once done, you can use the explorer tab to browse and download data.<br />
<br />
You can also mount the remote data to a local path. Use the "Mounts" tab, select the remote filesystem and a local folder (or diver letter in Windows), and press create.<br />
<br />
Enjoy!</div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=932Transferring data to/from PIC2021-12-23T08:50:20Z<p>Tallada: </p>
<hr />
<div>== How to provide data access to external users without PIC account ==<br />
<br />
I will use an example for MICE<br />
<br />
=== Steps ===<br />
<br />
1) dCache webdav door that exports your data.<br />
<br />
https://webdav-mice.pic.es:8454<br />
<br />
2) dCacheView frontend (optional)<br />
<br />
https://webdav-mice.pic.es:8464<br />
<br />
3) Authentication<br />
<br />
Either PIC credentials (user/password) or a macaroon (https://dcache.org/old/manuals/UserGuide-7.0/macaroons.shtml).<br />
Any authenticated user can get a macaroon for an external user. Remember to add proper caveats to restrict privileges.<br />
In the next example, I retrieve a macaroon with 7 days validity, that can only download and list, and can only access the subtree of the provided URL.<br />
<br />
<pre><br />
$ curl -u tallada -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://webdav-mice.pic.es:8454/tape/raw/production/N4096_L3072_LC3/maps/n8192/kmapgcpy/<br />
<br />
{<br />
"macaroon": "MDA3NWxvJmc2lnbmF0dXJlIJt81Pp8G1KXMH8FqwpWhJtwYnLfUAvNPQuq7bTwNOxGCg", <br />
}<br />
</pre></div>Talladahttps://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=931Transferring data to/from PIC2021-12-23T08:37:44Z<p>Tallada: Created page with "== How to provide data access to external users without PIC account == I will use an example for MICE === Steps === 1) Ensure you have a dCache webdav door that exports yo..."</p>
<hr />
<div>== How to provide data access to external users without PIC account ==<br />
<br />
I will use an example for MICE<br />
<br />
=== Steps ===<br />
<br />
1) Ensure you have a dCache webdav door that exports your data.<br />
<br />
In our case: https://webdav-mice.pic.es:8454<br />
<br />
* dcacheview frontend (optional)<br />
* PIC account or macaroon<br />
*</div>Talladahttps://pwiki.pic.es/index.php?title=Main_Page&diff=930Main Page2021-12-23T08:32:21Z<p>Tallada: /* Astrophysics & Cosmology Department */</p>
<hr />
<div>== Computing Department ==<br />
* [[HTCondor User Guide]]<br />
* [[jupyter.pic.es]]<br />
<br />
== Storage Department ==<br />
* [[dCache.org]]<br />
* [[FileSystems]]<br />
<br />
== Astrophysics & Cosmology Department ==<br />
<br />
*[[CosmoHub | CosmoHub User Guide]]<br />
*[[Gitlab_Migration | Gitlab migration]]<br />
*[[Python_Jupyter | Python environments and Jupyter ]]<br />
*[[AC_UserManual | User manual]]<br />
*[[Spark_on_farm | Spark on farm]]<br />
*[[Catalogs | Catalogs]]<br />
*[[Mock pipeline v2.0]]<br />
*[[dCache webdav i dCacheView]]<br />
<br />
== Getting started ==<br />
* [//www.mediawiki.org/wiki/Special:MyLanguage/Manual:Configuration_settings Configuration settings list]<br />
* [//www.mediawiki.org/wiki/Special:MyLanguage/Manual:FAQ MediaWiki FAQ]<br />
* [https://lists.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list]<br />
* [//www.mediawiki.org/wiki/Special:MyLanguage/Localisation#Translation_resources Localise MediaWiki for your language]<br />
* Consult the [//meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software.</div>Talladahttps://pwiki.pic.es/index.php?title=Spark_on_farm&diff=912Spark on farm2021-04-27T09:14:34Z<p>Tallada: </p>
<hr />
<div>In order to launch a Spark cluster, follow these steps:<br />
<br />
1) Deploy a Spark installation on your ~<br />
<br />
2) On a UI, start a spark master using this command and write down the master url provided<br />
<br />
${SPARK_HOME}/sbin/start-master.sh -h $(hostname)<br />
<br />
3) Create a submit script like this one, substituting <VARIABLES><br />
<br />
SPARK_MASTER=<MASTER_URL><br />
NUM_CORES=1<br />
MEMORY=4g<br />
<br />
cd ${TMPDIR}<br />
<br />
echo "spark.executor.extraJavaOptions -Djava.io.tmpdir=${TMPDIR}" > spark-defaults.conf<br />
echo "spark.local.dir ${TMPDIR}" >> spark-defaults.conf<br />
echo "spark.pyspark.python <PYTHON_VENV_EXECUTABLE_PATH>" >> spark-defaults.conf<br />
<br />
PYSPARK_PYTHON=<PYTHON_VENV_EXECUTABLE_PATH> \<br />
<SPARK_HOME>/bin/spark-class \<br />
org.apache.spark.deploy.worker.Worker ${SPARK_MASTER} \<br />
-c ${NUM_CORES} -m ${MEMORY} -d ${TMPDIR} --properties-file spark-defaults.conf<br />
<br />
4) Submit the previous script to the farm<br />
<br />
qsub -q <QUEUE> -t 1-<NUM_WORKERS> <SUBMIT_SCRIPT><br />
<br />
5) After a few minutes, workers should start registering to the master.<br />
<br />
6) Create your SparkContext connecting it to the <MASTER_URL><br />
<br />
spark = SparkSession.builder.master(<MASTER_URL>).getOrCreate()<br />
sc = spark.sparkContext<br />
<br />
7) Enjoy!</div>Talladahttps://pwiki.pic.es/index.php?title=High-z_galaxies&diff=890High-z galaxies2021-01-15T16:38:31Z<p>Tallada: </p>
<hr />
<div><br />
== Description of the process to generate the High-z galaxies catalog for SC8 == <br />
<br />
<br />
Pau runs the pipeline and create a catalog in parquet:<br />
<br />
Ojo porque en el hdfs tiene 1_10_4 y sin embargo en el 1_10_5 <br />
<br />
/software/astro/scripts/create_table_from_parquet.sh -p /hdfs/user/tallada/data/euclid/flagship_mock_1_10_4_high_z/0.pq -t jcarrete.flagship_mock_1_10_5_highz_pq -e<br />
<br />
<br />
CREATE EXTERNAL TABLE jcarrete.flagship_mock_1_10_5_highz_pq (<br />
`type` STRING,<br />
`z` DOUBLE,<br />
`abs_muv` DOUBLE,<br />
`sedname` STRING,<br />
`sedname_int` BIGINT,<br />
`rhalf_kpc` DOUBLE,<br />
`rhalf_arcsec` DOUBLE,<br />
`muv` DOUBLE,<br />
`beta` DOUBLE,<br />
`r` DOUBLE,<br />
`hpix_9_nest` BIGINT,<br />
`kind` BIGINT,<br />
`step` BIGINT,<br />
`unique_id` BIGINT,<br />
`hpix_29_nest` BIGINT,<br />
`ra` DOUBLE,<br />
`dec` DOUBLE,<br />
`l` DOUBLE,<br />
`b` DOUBLE,<br />
`mw_extinction` DOUBLE,<br />
`blanco_decam_g_true` DOUBLE,<br />
`blanco_decam_g` DOUBLE,<br />
`blanco_decam_g_odonnell_ext` DOUBLE,<br />
`blanco_decam_i_true` DOUBLE,<br />
`blanco_decam_i` DOUBLE,<br />
`blanco_decam_i_odonnell_ext` DOUBLE,<br />
`blanco_decam_r_true` DOUBLE,<br />
`blanco_decam_r` DOUBLE,<br />
`blanco_decam_r_odonnell_ext` DOUBLE,<br />
`blanco_decam_z_true` DOUBLE,<br />
`blanco_decam_z` DOUBLE,<br />
`blanco_decam_z_odonnell_ext` DOUBLE,<br />
`subaru_hsc_z_true` DOUBLE,<br />
`subaru_hsc_z` DOUBLE,<br />
`subaru_hsc_z_odonnell_ext` DOUBLE,<br />
`jst_jpcam_g_true` DOUBLE,<br />
`jst_jpcam_g` DOUBLE,<br />
`jst_jpcam_g_odonnell_ext` DOUBLE,<br />
`kids_g_true` DOUBLE,<br />
`kids_g` DOUBLE,<br />
`kids_g_odonnell_ext` DOUBLE,<br />
`kids_i_true` DOUBLE,<br />
`kids_i` DOUBLE,<br />
`kids_i_odonnell_ext` DOUBLE,<br />
`kids_r_true` DOUBLE,<br />
`kids_r` DOUBLE,<br />
`kids_r_odonnell_ext` DOUBLE,<br />
`kids_u_true` DOUBLE,<br />
`kids_u` DOUBLE,<br />
`kids_u_odonnell_ext` DOUBLE,<br />
`lsst_g_true` DOUBLE,<br />
`lsst_g` DOUBLE,<br />
`lsst_g_odonnell_ext` DOUBLE,<br />
`lsst_i_true` DOUBLE,<br />
`lsst_i` DOUBLE,<br />
`lsst_i_odonnell_ext` DOUBLE,<br />
`lsst_r_true` DOUBLE,<br />
`lsst_r` DOUBLE,<br />
`lsst_r_odonnell_ext` DOUBLE,<br />
`lsst_u_true` DOUBLE,<br />
`lsst_u` DOUBLE,<br />
`lsst_u_odonnell_ext` DOUBLE,<br />
`lsst_y_true` DOUBLE,<br />
`lsst_y` DOUBLE,<br />
`lsst_y_odonnell_ext` DOUBLE,<br />
`lsst_z_true` DOUBLE,<br />
`lsst_z` DOUBLE,<br />
`lsst_z_odonnell_ext` DOUBLE,<br />
`cfht_megacam_r_true` DOUBLE,<br />
`cfht_megacam_r` DOUBLE,<br />
`cfht_megacam_r_odonnell_ext` DOUBLE,<br />
`cfht_megacam_u_true` DOUBLE,<br />
`cfht_megacam_u` DOUBLE,<br />
`cfht_megacam_u_odonnell_ext` DOUBLE,<br />
`pan_starrs_i_true` DOUBLE,<br />
`pan_starrs_i` DOUBLE,<br />
`pan_starrs_i_odonnell_ext` DOUBLE,<br />
`pan_starrs_z_true` DOUBLE,<br />
`pan_starrs_z` DOUBLE,<br />
`pan_starrs_z_odonnell_ext` DOUBLE,<br />
`2mass_h_true` DOUBLE,<br />
`2mass_h` DOUBLE,<br />
`2mass_h_odonnell_ext` DOUBLE,<br />
`2mass_j_true` DOUBLE,<br />
`2mass_j` DOUBLE,<br />
`2mass_j_odonnell_ext` DOUBLE,<br />
`2mass_ks_true` DOUBLE,<br />
`2mass_ks` DOUBLE,<br />
`2mass_ks_odonnell_ext` DOUBLE,<br />
`sdss_r01_true` DOUBLE,<br />
`sdss_r01` DOUBLE,<br />
`sdss_r01_odonnell_ext` DOUBLE,<br />
`gaia_bp_true` DOUBLE,<br />
`gaia_bp` DOUBLE,<br />
`gaia_bp_odonnell_ext` DOUBLE,<br />
`gaia_g_true` DOUBLE,<br />
`gaia_g` DOUBLE,<br />
`gaia_g_odonnell_ext` DOUBLE,<br />
`gaia_rp_true` DOUBLE,<br />
`gaia_rp` DOUBLE,<br />
`gaia_rp_odonnell_ext` DOUBLE,<br />
`euclid_nisp_h_true` DOUBLE,<br />
`euclid_nisp_h` DOUBLE,<br />
`euclid_nisp_h_odonnell_ext` DOUBLE,<br />
`euclid_nisp_j_true` DOUBLE,<br />
`euclid_nisp_j` DOUBLE,<br />
`euclid_nisp_j_odonnell_ext` DOUBLE,<br />
`euclid_nisp_y_true` DOUBLE,<br />
`euclid_nisp_y` DOUBLE,<br />
`euclid_nisp_y_odonnell_ext` DOUBLE,<br />
`euclid_vis_true` DOUBLE,<br />
`euclid_vis` DOUBLE,<br />
`euclid_vis_odonnell_ext` DOUBLE,<br />
`id` BIGINT<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/euclid/flagship_mock_1_10_4_high_z'<br />
;<br />
<br />
<br />
Clusterizo:<br />
<br />
CREATE TABLE cosmohub.flagship_mock_1_10_5_highz_c (<br />
`unique_id` BIGINT COMMENT 'unique identifier',<br />
`id` INT COMMENT 'ID',<br />
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',<br />
`z` FLOAT,<br />
`ra` FLOAT,<br />
`dec` FLOAT,<br />
`l` FLOAT,<br />
`b` FLOAT,<br />
`mw_extinction` FLOAT,<br />
`abs_muv` FLOAT,<br />
`sedname` STRING,<br />
`sedname_int` BIGINT,<br />
`rhalf_kpc` FLOAT,<br />
`rhalf_arcsec` FLOAT,<br />
`muv` FLOAT,<br />
`beta` FLOAT,<br />
`hpix_9_nest` INT,<br />
`hpix_13_nest` INT,<br />
`hpix_29_nest` BIGINT,<br />
`blanco_decam_g_true` FLOAT,<br />
`blanco_decam_g` FLOAT,<br />
`blanco_decam_g_odonnell_ext` FLOAT,<br />
`blanco_decam_i_true` FLOAT,<br />
`blanco_decam_i` FLOAT,<br />
`blanco_decam_i_odonnell_ext` FLOAT,<br />
`blanco_decam_r_true` FLOAT,<br />
`blanco_decam_r` FLOAT,<br />
`blanco_decam_r_odonnell_ext` FLOAT,<br />
`blanco_decam_z_true` FLOAT,<br />
`blanco_decam_z` FLOAT,<br />
`blanco_decam_z_odonnell_ext` FLOAT,<br />
`subaru_hsc_z_true` FLOAT,<br />
`subaru_hsc_z` FLOAT,<br />
`subaru_hsc_z_odonnell_ext` FLOAT,<br />
`jst_jpcam_g_true` FLOAT,<br />
`jst_jpcam_g` FLOAT,<br />
`jst_jpcam_g_odonnell_ext` FLOAT,<br />
`kids_g_true` FLOAT,<br />
`kids_g` FLOAT,<br />
`kids_g_odonnell_ext` FLOAT,<br />
`kids_i_true` FLOAT,<br />
`kids_i` FLOAT,<br />
`kids_i_odonnell_ext` FLOAT,<br />
`kids_r_true` FLOAT,<br />
`kids_r` FLOAT,<br />
`kids_r_odonnell_ext` FLOAT,<br />
`kids_u_true` FLOAT,<br />
`kids_u` FLOAT,<br />
`kids_u_odonnell_ext` FLOAT,<br />
`lsst_g_true` FLOAT,<br />
`lsst_g` FLOAT,<br />
`lsst_g_odonnell_ext` FLOAT,<br />
`lsst_i_true` FLOAT,<br />
`lsst_i` FLOAT,<br />
`lsst_i_odonnell_ext` FLOAT,<br />
`lsst_r_true` FLOAT,<br />
`lsst_r` FLOAT,<br />
`lsst_r_odonnell_ext` FLOAT,<br />
`lsst_u_true` FLOAT,<br />
`lsst_u` FLOAT,<br />
`lsst_u_odonnell_ext` FLOAT,<br />
`lsst_y_true` FLOAT,<br />
`lsst_y` FLOAT,<br />
`lsst_y_odonnell_ext` FLOAT,<br />
`lsst_z_true` FLOAT,<br />
`lsst_z` FLOAT,<br />
`lsst_z_odonnell_ext` FLOAT,<br />
`cfht_megacam_r_true` FLOAT,<br />
`cfht_megacam_r` FLOAT,<br />
`cfht_megacam_r_odonnell_ext` FLOAT,<br />
`cfht_megacam_u_true` FLOAT,<br />
`cfht_megacam_u` FLOAT,<br />
`cfht_megacam_u_odonnell_ext` FLOAT,<br />
`pan_starrs_i_true` FLOAT,<br />
`pan_starrs_i` FLOAT,<br />
`pan_starrs_i_odonnell_ext` FLOAT,<br />
`pan_starrs_z_true` FLOAT,<br />
`pan_starrs_z` FLOAT,<br />
`pan_starrs_z_odonnell_ext` FLOAT,<br />
`2mass_h_true` FLOAT,<br />
`2mass_h` FLOAT,<br />
`2mass_h_odonnell_ext` FLOAT,<br />
`2mass_j_true` FLOAT,<br />
`2mass_j` FLOAT,<br />
`2mass_j_odonnell_ext` FLOAT,<br />
`2mass_ks_true` FLOAT,<br />
`2mass_ks` FLOAT,<br />
`2mass_ks_odonnell_ext` FLOAT,<br />
`sdss_r01_true` FLOAT,<br />
`sdss_r01` FLOAT,<br />
`sdss_r01_odonnell_ext` FLOAT,<br />
`gaia_bp_true` FLOAT,<br />
`gaia_bp` FLOAT,<br />
`gaia_bp_odonnell_ext` FLOAT,<br />
`gaia_g_true` FLOAT,<br />
`gaia_g` FLOAT,<br />
`gaia_g_odonnell_ext` FLOAT,<br />
`gaia_rp_true` FLOAT,<br />
`gaia_rp` FLOAT,<br />
`gaia_rp_odonnell_ext` FLOAT,<br />
`euclid_nisp_h_true` FLOAT,<br />
`euclid_nisp_h` FLOAT,<br />
`euclid_nisp_h_odonnell_ext` FLOAT,<br />
`euclid_nisp_j_true` FLOAT,<br />
`euclid_nisp_j` FLOAT,<br />
`euclid_nisp_j_odonnell_ext` FLOAT,<br />
`euclid_nisp_y_true` FLOAT,<br />
`euclid_nisp_y` FLOAT,<br />
`euclid_nisp_y_odonnell_ext` FLOAT,<br />
`euclid_vis_true` FLOAT,<br />
`euclid_vis` FLOAT,<br />
`euclid_vis_odonnell_ext` FLOAT,<br />
`step` smallint COMMENT 'redshift step number'<br />
)<br />
CLUSTERED BY (<br />
step,<br />
hpix_13_nest)<br />
SORTED BY (<br />
step ASC,<br />
hpix_13_nest ASC)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
INSERT OVERWRITE TABLE cosmohub.flagship_mock_1_10_5_highz_c<br />
SELECT<br />
`unique_id`, `id`, `kind`, `z`, `ra`, `dec`, `l`, `b`, `mw_extinction`, `abs_muv`, `sedname`, `sedname_int`, `rhalf_kpc`, `rhalf_arcsec`, `muv`, `beta`, `hpix_9_nest`, SHIFTRIGHT(hpix_29_nest, (29-13)*2), `hpix_29_nest`, `blanco_decam_g_true`, `blanco_decam_g`, `blanco_decam_g_odonnell_ext`, `blanco_decam_i_true`, `blanco_decam_i`, `blanco_decam_i_odonnell_ext`, `blanco_decam_r_true`, `blanco_decam_r`, `blanco_decam_r_odonnell_ext`, `blanco_decam_z_true`, `blanco_decam_z`, `blanco_decam_z_odonnell_ext`, `subaru_hsc_z_true`, `subaru_hsc_z`, `subaru_hsc_z_odonnell_ext`, `jst_jpcam_g_true`, `jst_jpcam_g`, `jst_jpcam_g_odonnell_ext`, `kids_g_true`, `kids_g`, `kids_g_odonnell_ext`, `kids_i_true`, `kids_i`, `kids_i_odonnell_ext`, `kids_r_true`, `kids_r`, `kids_r_odonnell_ext`, `kids_u_true`, `kids_u`, `kids_u_odonnell_ext`, `lsst_g_true`, `lsst_g`, `lsst_g_odonnell_ext`, `lsst_i_true`, `lsst_i`, `lsst_i_odonnell_ext`, `lsst_r_true`, `lsst_r`, `lsst_r_odonnell_ext`, `lsst_u_true`, `lsst_u`, `lsst_u_odonnell_ext`, `lsst_y_true`, `lsst_y`, `lsst_y_odonnell_ext`, `lsst_z_true`, `lsst_z`, `lsst_z_odonnell_ext`, `cfht_megacam_r_true`, `cfht_megacam_r`, `cfht_megacam_r_odonnell_ext`, `cfht_megacam_u_true`, `cfht_megacam_u`, `cfht_megacam_u_odonnell_ext`, `pan_starrs_i_true`, `pan_starrs_i`, `pan_starrs_i_odonnell_ext`, `pan_starrs_z_true`, `pan_starrs_z`, `pan_starrs_z_odonnell_ext`, `2mass_h_true`, `2mass_h`, `2mass_h_odonnell_ext`, `2mass_j_true`, `2mass_j`, `2mass_j_odonnell_ext`, `2mass_ks_true`, `2mass_ks`, `2mass_ks_odonnell_ext`, `sdss_r01_true`, `sdss_r01`, `sdss_r01_odonnell_ext`, `gaia_bp_true`, `gaia_bp`, `gaia_bp_odonnell_ext`, `gaia_g_true`, `gaia_g`, `gaia_g_odonnell_ext`, `gaia_rp_true`, `gaia_rp`, `gaia_rp_odonnell_ext`, `euclid_nisp_h_true`, `euclid_nisp_h`, `euclid_nisp_h_odonnell_ext`, `euclid_nisp_j_true`, `euclid_nisp_j`, `euclid_nisp_j_odonnell_ext`, `euclid_nisp_y_true`, `euclid_nisp_y`, `euclid_nisp_y_odonnell_ext`, `euclid_vis_true`, `euclid_vis`, `euclid_vis_odonnell_ext`, `step`<br />
FROM jcarrete.flagship_mock_1_10_5_highz_pq;<br />
<br />
<br />
ANALYZE TABLE cosmohub.flagship_mock_1_10_5_highz_c COMPUTE STATISTICS FOR COLUMNS;<br />
<br />
Añado el lensing:<br />
<br />
CREATE TABLE cosmohub.flagship_mock_1_10_5_highz_s (<br />
`unique_id` BIGINT COMMENT 'unique identifier',<br />
`id` INT COMMENT 'ID',<br />
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',<br />
`z` float COMMENT 'true galaxy redshift',<br />
`ra` FLOAT COMMENT 'galaxy right ascension (degrees)',<br />
`dec` FLOAT COMMENT 'galaxy declination (degrees)',<br />
`ra_mag` float COMMENT 'galaxy magnified right ascension (degree)', <br />
`dec_mag` float COMMENT 'galaxy magnified declination (degree)', <br />
`kappa` float COMMENT 'convergence', <br />
`gamma1` float COMMENT 'shear', <br />
`gamma2` float COMMENT 'shear',<br />
`defl1` float COMMENT 'deflection1',<br />
`defl2` float COMMENT 'deflection2',<br />
`l` FLOAT COMMENT 'galactic longitude (degrees)',<br />
`b` FLOAT COMMENT 'galactic latitude (degrees)',<br />
`mw_extinction` float COMMENT 'EBV from file: HFI_CompMap_ThermalDustModel_2048_R1.20.fits',<br />
`abs_muv` FLOAT COMMENT 'absolute magnitude at 1500 angstroms',<br />
`sedname` STRING COMMENT 'Spectral Energy Distribution name (among 76, 38 without UV lines, 38 with UV lines, corresponding to a given UV slope)',<br />
`sedname_int` BIGINT COMMENT 'sedname as integer (from 0 to 75, from 0 to 37 without UV lines, from 38 to 75 with UV lines)',<br />
`rhalf_kpc` FLOAT COMMENT 'half light radius (in physical kpc)',<br />
`rhalf_arcsec` FLOAT COMMENT 'half light radius (in arcsec)',<br />
`muv` FLOAT COMMENT 'for information only, observed magnitude at rest-frame 1500',<br />
`beta` FLOAT COMMENT 'for information only, used to assign an SED',<br />
`hpix_9_nest` int COMMENT 'healpix pixel id using order = 9 in NESTED format',<br />
`hpix_13_nest` int COMMENT 'healpix pixel id using order = 13 in NESTED format',<br />
`hpix_29_nest` bigint COMMENT 'healpix pixel id using order = 29 in NESTED format',<br />
`blanco_decam_g_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in DES g-band (Euclid provided)', <br />
`blanco_decam_g` float COMMENT 'Interpolated observed flux in DES g-band (Euclid provided)',<br />
`blanco_decam_g_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in DES g-band (Euclid provided)',<br />
`blanco_decam_i_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in DES i-band (Euclid provided)', <br />
`blanco_decam_i` float COMMENT 'Interpolated observed flux in DES i-band (Euclid provided)', <br />
`blanco_decam_i_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in DES i-band (Euclid provided)', <br />
`blanco_decam_r_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in DES r-band (Euclid provided)', <br />
`blanco_decam_r` float COMMENT 'Interpolated observed flux in DES r-band (Euclid provided)', <br />
`blanco_decam_r_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in DES r-band (Euclid provided)', <br />
`blanco_decam_z_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in DES z-band (Euclid provided)', <br />
`blanco_decam_z` float COMMENT 'Interpolated observed flux in DES z-band (Euclid provided)',<br />
`blanco_decam_z_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in DES z-band (Euclid provided)', <br />
`subaru_hsc_z_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',<br />
`subaru_hsc_z` float COMMENT 'Interpolated observed flux in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',<br />
`subaru_hsc_z_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',<br />
`jst_jpcam_g_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in JST g-band (Euclid provided)', <br />
`jst_jpcam_g` float COMMENT 'Interpolated observed flux in JST g-band (Euclid provided)',<br />
`jst_jpcam_g_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in JST g-band (Euclid provided)',<br />
`kids_g_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in KIDS g-band (Euclid provided)', <br />
`kids_g` float COMMENT 'Interpolated observed flux in KIDS g-band (Euclid provided)',<br />
`kids_g_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in KIDS g-band (Euclid provided)',<br />
`kids_i_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in KIDS i-band (Euclid provided)', <br />
`kids_i` float COMMENT 'Interpolated observed flux in KIDS i-band (Euclid provided)',<br />
`kids_i_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in KIDS i-band (Euclid provided)', <br />
`kids_r_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in KIDS r-band (Euclid provided)', <br />
`kids_r` float COMMENT 'Interpolated observed flux in KIDS r-band (Euclid provided)',<br />
`kids_r_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in KIDS r-band (Euclid provided)',<br />
`kids_u_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in KIDS u-band (Euclid provided)', <br />
`kids_u` float COMMENT 'Interpolated observed flux in KIDS u-band (Euclid provided)',<br />
`kids_u_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in KIDS u-band (Euclid provided)',<br />
`lsst_g_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST g-band (Euclid provided)', <br />
`lsst_g` float COMMENT 'Interpolated observed flux in LSST g-band (Euclid provided)',<br />
`lsst_g_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST g-band (Euclid provided)',<br />
`lsst_i_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST i-band (Euclid provided)', <br />
`lsst_i` float COMMENT 'Interpolated observed flux in LSST i-band (Euclid provided)',<br />
`lsst_i_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST i-band (Euclid provided)',<br />
`lsst_r_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST r-band (Euclid provided)', <br />
`lsst_r` float COMMENT 'Interpolated observed flux in LSST r-band (Euclid provided)',<br />
`lsst_r_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST r-band (Euclid provided)',<br />
`lsst_u_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST u-band (Euclid provided)', <br />
`lsst_u` float COMMENT 'Interpolated observed flux in LSST u-band (Euclid provided)',<br />
`lsst_u_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST u-band (Euclid provided)',<br />
`lsst_y_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST y-band (Euclid provided)', <br />
`lsst_y` float COMMENT 'Interpolated observed flux in LSST y-band (Euclid provided)',<br />
`lsst_y_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST y-band (Euclid provided)',<br />
`lsst_z_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST z-band (Euclid provided)', <br />
`lsst_z` float COMMENT 'Interpolated observed flux in LSST z-band (Euclid provided)',<br />
`lsst_z_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST z-band (Euclid provided)',<br />
`cfht_megacam_r_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in CFHT r-band (Euclid provided)', <br />
`cfht_megacam_r` float COMMENT 'Interpolated observed flux in CFHT r-band (Euclid provided)',<br />
`cfht_megacam_r_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in CFHT r-band (Euclid provided)',<br />
`cfht_megacam_u_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in CFHT u-band (Euclid provided)', <br />
`cfht_megacam_u` float COMMENT 'Interpolated observed flux in CFHT u-band (Euclid provided)',<br />
`cfht_megacam_u_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in CFHT u-band (Euclid provided)',<br />
`pan_starrs_i_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Pan-STARRS i-band (Euclid provided)', <br />
`pan_starrs_i` float COMMENT 'Interpolated observed flux in Pan-STARRS i-band (Euclid provided)',<br />
`pan_starrs_i_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Pan-STARRS i-band (Euclid provided)',<br />
`pan_starrs_z_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Pan-STARRS z-band (Euclid provided)', <br />
`pan_starrs_z` float COMMENT 'Interpolated observed flux in Pan-STARRS z-band (Euclid provided)',<br />
`pan_starrs_z_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Pan-STARRS z-band (Euclid provided)',<br />
`2mass_h_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in 2MASS H-band (Euclid provided)', <br />
`2mass_h` float COMMENT 'Interpolated observed flux in 2MASS H-band (Euclid provided)',<br />
`2mass_h_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in 2MASS H-band (Euclid provided)',<br />
`2mass_j_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in 2MASS J-band (Euclid provided)', <br />
`2mass_j` float COMMENT 'Interpolated observed flux in 2MASS J-band (Euclid provided)',<br />
`2mass_j_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in 2MASS J-band (Euclid provided)',<br />
`2mass_ks_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in 2MASS Ks-band (Euclid provided)', <br />
`2mass_ks` float COMMENT 'Interpolated observed flux in 2MASS Ks-band (Euclid provided)',<br />
`2mass_ks_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in 2MASS Ks-band (Euclid provided)',<br />
`sdss_r01_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in sdss r-band redshifted to z=0.1 (Euclid provided)', <br />
`sdss_r01` float COMMENT 'Interpolated observed flux in sdss r-band redshifted to z=0.1 (Euclid provided)',<br />
`sdss_r01_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in sdss r-band redshifted to z=0.1 (Euclid provided)',<br />
`gaia_bp_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Gaia-BP band (Euclid provided)', <br />
`gaia_bp` float COMMENT 'Interpolated observed flux in Gaia-BP band (Euclid provided)',<br />
`gaia_bp_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Gaia-BP band (Euclid provided)',<br />
`gaia_g_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Gaia-G band (Euclid provided)', <br />
`gaia_g` float COMMENT 'Interpolated observed flux in Gaia-G band (Euclid provided)',<br />
`gaia_g_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Gaia-G band (Euclid provided)', <br />
`gaia_rp_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Gaia-RP band (Euclid provided)', <br />
`gaia_rp` float COMMENT 'Interpolated observed flux in Gaia-RP band (Euclid provided)',<br />
`gaia_rp_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Gaia-RP band (Euclid provided)',<br />
`euclid_nisp_h_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Euclid NISP-H band (Euclid provided)', <br />
`euclid_nisp_h` float COMMENT 'Interpolated observed flux in Euclid NISP-H band (Euclid provided)',<br />
`euclid_nisp_h_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Euclid NISP-H band (Euclid provided)', <br />
`euclid_nisp_j_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Euclid NISP-J band (Euclid provided)', <br />
`euclid_nisp_j` float COMMENT 'Interpolated observed flux in Euclid NISP-J band (Euclid provided)',<br />
`euclid_nisp_j_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Euclid NISP-J band (Euclid provided)',<br />
`euclid_nisp_y_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Euclid NISP-Y band (Euclid provided)', <br />
`euclid_nisp_y` float COMMENT 'Interpolated observed flux in Euclid NISP-Y band (Euclid provided)',<br />
`euclid_nisp_y_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Euclid NISP-Y band (Euclid provided)', <br />
`euclid_vis_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Euclid VIS band (Euclid provided)', <br />
`euclid_vis` float COMMENT 'Interpolated observed flux in Euclid VIS band (Euclid provided)',<br />
`euclid_vis_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Euclid VIS band (Euclid provided)',<br />
`step` smallint COMMENT 'redshift step number'<br />
)<br />
CLUSTERED BY (<br />
step,<br />
hpix_13_nest)<br />
SORTED BY (<br />
step ASC,<br />
hpix_13_nest ASC)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
<br />
SET hive.auto.convert.join.noconditionaltask.size=1000;<br />
<br />
INSERT OVERWRITE TABLE cosmohub.flagship_mock_1_10_5_highz_s<br />
SELECT <br />
`unique_id`, `id`, `kind`, `z`, `ra`, `dec`, mag_pos.ra_mag, mag_pos.dec_mag, kappa, gamma1, gamma2, defl1, defl2, `l`, `b`, `mw_extinction`, `abs_muv`, `sedname`, `sedname_int`, `rhalf_kpc`, `rhalf_arcsec`, `muv`, `beta`, `hpix_9_nest`, `hpix_13_nest`, `hpix_29_nest`, `blanco_decam_g_true`, `blanco_decam_g`, `blanco_decam_g_odonnell_ext`, `blanco_decam_i_true`, `blanco_decam_i`, `blanco_decam_i_odonnell_ext`, `blanco_decam_r_true`, `blanco_decam_r`, `blanco_decam_r_odonnell_ext`, `blanco_decam_z_true`, `blanco_decam_z`, `blanco_decam_z_odonnell_ext`, `subaru_hsc_z_true`, `subaru_hsc_z`, `subaru_hsc_z_odonnell_ext`, `jst_jpcam_g_true`, `jst_jpcam_g`, `jst_jpcam_g_odonnell_ext`, `kids_g_true`, `kids_g`, `kids_g_odonnell_ext`, `kids_i_true`, `kids_i`, `kids_i_odonnell_ext`, `kids_r_true`, `kids_r`, `kids_r_odonnell_ext`, `kids_u_true`, `kids_u`, `kids_u_odonnell_ext`, `lsst_g_true`, `lsst_g`, `lsst_g_odonnell_ext`, `lsst_i_true`, `lsst_i`, `lsst_i_odonnell_ext`, `lsst_r_true`, `lsst_r`, `lsst_r_odonnell_ext`, `lsst_u_true`, `lsst_u`, `lsst_u_odonnell_ext`, `lsst_y_true`, `lsst_y`, `lsst_y_odonnell_ext`, `lsst_z_true`, `lsst_z`, `lsst_z_odonnell_ext`, `cfht_megacam_r_true`, `cfht_megacam_r`, `cfht_megacam_r_odonnell_ext`, `cfht_megacam_u_true`, `cfht_megacam_u`, `cfht_megacam_u_odonnell_ext`, `pan_starrs_i_true`, `pan_starrs_i`, `pan_starrs_i_odonnell_ext`, `pan_starrs_z_true`, `pan_starrs_z`, `pan_starrs_z_odonnell_ext`, `2mass_h_true`, `2mass_h`, `2mass_h_odonnell_ext`, `2mass_j_true`, `2mass_j`, `2mass_j_odonnell_ext`, `2mass_ks_true`, `2mass_ks`, `2mass_ks_odonnell_ext`, `sdss_r01_true`, `sdss_r01`, `sdss_r01_odonnell_ext`, `gaia_bp_true`, `gaia_bp`, `gaia_bp_odonnell_ext`, `gaia_g_true`, `gaia_g`, `gaia_g_odonnell_ext`, `gaia_rp_true`, `gaia_rp`, `gaia_rp_odonnell_ext`, `euclid_nisp_h_true`, `euclid_nisp_h`, `euclid_nisp_h_odonnell_ext`, `euclid_nisp_j_true`, `euclid_nisp_j`, `euclid_nisp_j_odonnell_ext`, `euclid_nisp_y_true`, `euclid_nisp_y`, `euclid_nisp_y_odonnell_ext`, `euclid_vis_true`, `euclid_vis`, `euclid_vis_odonnell_ext`, `step`<br />
FROM<br />
(SELECT <br />
`unique_id`, `id`, `kind`, `z`, `ra`, `dec`, udf.magnified_positions(CAST(ra AS DOUBLE), CAST(`dec` AS DOUBLE), CAST(l.defl1 AS DOUBLE), CAST(l.defl2 AS DOUBLE)) as mag_pos, l.kappa, l.gamma1, l.gamma2, l.defl1, l.defl2, `l`, `b`, `mw_extinction`, `abs_muv`, `sedname`, `sedname_int`, `rhalf_kpc`, `rhalf_arcsec`, `muv`, `beta`, `hpix_9_nest`, l.hpix_13_nest, `hpix_29_nest`, `blanco_decam_g_true`, `blanco_decam_g`, `blanco_decam_g_odonnell_ext`, `blanco_decam_i_true`, `blanco_decam_i`, `blanco_decam_i_odonnell_ext`, `blanco_decam_r_true`, `blanco_decam_r`, `blanco_decam_r_odonnell_ext`, `blanco_decam_z_true`, `blanco_decam_z`, `blanco_decam_z_odonnell_ext`, `subaru_hsc_z_true`, `subaru_hsc_z`, `subaru_hsc_z_odonnell_ext`, `jst_jpcam_g_true`, `jst_jpcam_g`, `jst_jpcam_g_odonnell_ext`, `kids_g_true`, `kids_g`, `kids_g_odonnell_ext`, `kids_i_true`, `kids_i`, `kids_i_odonnell_ext`, `kids_r_true`, `kids_r`, `kids_r_odonnell_ext`, `kids_u_true`, `kids_u`, `kids_u_odonnell_ext`, `lsst_g_true`, `lsst_g`, `lsst_g_odonnell_ext`, `lsst_i_true`, `lsst_i`, `lsst_i_odonnell_ext`, `lsst_r_true`, `lsst_r`, `lsst_r_odonnell_ext`, `lsst_u_true`, `lsst_u`, `lsst_u_odonnell_ext`, `lsst_y_true`, `lsst_y`, `lsst_y_odonnell_ext`, `lsst_z_true`, `lsst_z`, `lsst_z_odonnell_ext`, `cfht_megacam_r_true`, `cfht_megacam_r`, `cfht_megacam_r_odonnell_ext`, `cfht_megacam_u_true`, `cfht_megacam_u`, `cfht_megacam_u_odonnell_ext`, `pan_starrs_i_true`, `pan_starrs_i`, `pan_starrs_i_odonnell_ext`, `pan_starrs_z_true`, `pan_starrs_z`, `pan_starrs_z_odonnell_ext`, `2mass_h_true`, `2mass_h`, `2mass_h_odonnell_ext`, `2mass_j_true`, `2mass_j`, `2mass_j_odonnell_ext`, `2mass_ks_true`, `2mass_ks`, `2mass_ks_odonnell_ext`, `sdss_r01_true`, `sdss_r01`, `sdss_r01_odonnell_ext`, `gaia_bp_true`, `gaia_bp`, `gaia_bp_odonnell_ext`, `gaia_g_true`, `gaia_g`, `gaia_g_odonnell_ext`, `gaia_rp_true`, `gaia_rp`, `gaia_rp_odonnell_ext`, `euclid_nisp_h_true`, `euclid_nisp_h`, `euclid_nisp_h_odonnell_ext`, `euclid_nisp_j_true`, `euclid_nisp_j`, `euclid_nisp_j_odonnell_ext`, `euclid_nisp_y_true`, `euclid_nisp_y`, `euclid_nisp_y_odonnell_ext`, `euclid_vis_true`, `euclid_vis`, `euclid_vis_odonnell_ext`, l.step<br />
FROM cosmohub.flagship_mock_1_10_5_highz_c as m<br />
JOIN cosmohub.flagship_lensing2 as l<br />
ON m.step = l.step<br />
AND m.hpix_13_nest = l.hpix_13_nest) as t<br />
;<br />
<br />
<br />
ANALYZE TABLE cosmohub.flagship_mock_1_10_5_highz_s COMPUTE STATISTICS FOR COLUMNS;<br />
<br />
----<br />
<br />
Information in CosmoHub:<br />
<br />
<br />
This is the first SC8 High-z galaxies release.<br />
<br />
We have selected galaxies from [https://cosmohub.pic.es/catalogs/209](https://cosmohub.pic.es/catalogs/209) (copied information below) and located randomly in SC8 area.<br />
<br />
We have assigned lensing parameters from last redshift step (86) since lensing and deflection maps are not yet available for higher redshift.<br />
<br />
Therefore we have used those parameters to estimate the magnified positions and also magnified fluxes in the TU FITS files.<br />
<br />
Note that **we provide fluxes instead of magnitudes**. In order **to get magnitudes** you can enter in the "Expert mode" and estimate the magnitude as follows: <br />
<br />
m = -2.5 * log10(flux) - 48.6<br />
<br />
And in order to estimate magnified magnitudes:<br />
<br />
m_mag = m + 2.5 * log10((1-kappa)*(1-kappa) - gamma1 * gamma1 - gamma2 * gamma2)<br />
<br />
---<br />
<br />
**The catalog of LBG sources is based on the "Recipes" described in this [Euclid Redmine entry](https://euclid.roe.ac.uk/projects/puswg/wiki/Galaxies)**.<br />
<br />
Important features to be noted:<br />
<br />
1) The corresponding area is 16 deg2<br />
<br />
2) Magnitude cut at H=27 (AB mag)<br />
<br />
3) SEDs are provided with given UV continuum slopes<br />
<br />
4) A standard cosmology with h=0.7, Om=0.3, Olambda=0.7 was used to convert kpc to arcsec and abs_muv (absolute magnitude at 1500 angstroms) to muv (observed magnitude at rest-frame 1500)<br />
<br />
<br />
Fits file example in CosmoHub:<br />
<br />
{<br />
"sql": "SELECT CAST(unique_id AS bigint) AS SOURCE_ID, \nCAST(-1 AS float) AS HALO_ID, \nCAST(kind AS smallint) AS KIND, \nCAST(ra AS double) AS RA, \nCAST(`dec` AS double) AS `DEC`, \nCAST(ra_mag AS double) AS RA_MAG, \nCAST(dec_mag AS double) AS DEC_MAG, \nCAST(z AS float) AS Z_OBS, \nCAST(abs_muv AS float) AS REF_MAG_ABS, \nCAST(abs_muv AS float) AS REF_MAG, \nCAST(1 AS float) AS BULGE_FRACTION, \nCAST(rhalf_arcsec AS float) AS BULGE_R50, \nCAST(rhalf_arcsec AS float) AS DISK_R50, \nCAST(1.5 AS float) AS BULGE_NSERSIC, \nCAST(-1 AS float) AS BULGE_AXIS_RATIO, \nCAST(-1 AS float) AS INCLINATION_ANGLE, \nCAST(-1 AS float) AS DISK_ANGLE, \nCAST(kappa AS float) AS KAPPA, \nCAST(gamma1 AS float) AS GAMMA1, \nCAST(gamma2 AS float) AS GAMMA2, \nCAST(sedname_int AS float) AS SED_TEMPLATE, \nCAST(0 AS smallint) AS EXT_LAW, \nCAST(-1 AS float) AS EBV, \nCAST(-1 AS float) AS HALPHA_LOGFLAM_EXT_MAG, \nCAST(-1 AS float) AS HBETA_LOGFLAM_EXT_MAG, \nCAST(-1 AS float) AS O2_LOGFLAM_EXT_MAG, \nCAST(-1 AS float) AS O3_LOGFLAM_EXT_MAG, \nCAST(-1 AS float) AS N2_LOGFLAM_EXT_MAG, \nCAST(-1 AS float) AS S2_LOGFLAM_EXT_MAG, \nCAST(mw_extinction AS float) AS AV,\nCAST(euclid_vis*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_VIS_MAG, \nCAST(euclid_nisp_y*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_NISP_MAG, \nCAST(euclid_nisp_j*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_NISP_MAG, \nCAST(euclid_nisp_h*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_NISP_MAG, \nCAST(blanco_decam_g*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_DECAM_MAG, \nCAST(blanco_decam_r*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_DECAM_MAG, \nCAST(blanco_decam_i*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_DECAM_MAG, \nCAST(blanco_decam_z*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_DECAM_MAG, \nCAST(cfht_megacam_u*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_MEGACAM_MAG, \nCAST(cfht_megacam_r*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_MEGACAM_MAG, \nCAST(jst_jpcam_g*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_JPCAM_MAG, \nCAST(pan_starrs_i*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_PANSTARRS_MAG, \nCAST(pan_starrs_z*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_PANSTARRS_MAG, \nCAST(subaru_hsc_z*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_HSC_MAG, \nCAST(gaia_g*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_GAIA_MAG, \nCAST(gaia_bp*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_BP_GAIA_MAG, \nCAST(gaia_rp*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_RP_GAIA_MAG, \nCAST(lsst_u*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_LSST_MAG, \nCAST(lsst_g*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_LSST_MAG, \nCAST(lsst_r*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_LSST_MAG, \nCAST(lsst_i*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_LSST_MAG, \nCAST(lsst_z*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_LSST_MAG, \nCAST(lsst_y*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_LSST_MAG, \nCAST(kids_u*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_KIDS_MAG, \nCAST(kids_g*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_KIDS_MAG, \nCAST(kids_r*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_KIDS_MAG, \nCAST(kids_i*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_KIDS_MAG, \nCAST(2mass_j*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_2MASS_MAG, \nCAST(2mass_h*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_2MASS_MAG, \nCAST(2mass_ks*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_KS_2MASS_MAG \nFROM cosmohub.flagship_mock_1_10_5_highz_s \nWHERE SHIFTRIGHT(hpix_29_nest, (29-5)*2) = 2063"<br />
}<br />
<br />
I made the following choices:<br />
<br />
* HALO_ID = -1 when kind = 2 or 3<br />
* BULGE_FRACTION = 1 and then the disk information is not used!<br />
* BULGE_NSERSIC = 1.5<br />
<br />
The rest of the fields that are not included in the catalog but are necessary for the FITS file format = -1; [REF_MAG_ABS, REF_MAG] = (m1450 OR abs_muv) when kind = 2 and 3, respectively.<br />
<br />
<br />
SELECT `unique_id`, `kind`, `ra`, `dec`, `ra_mag`, `dec_mag`, `kappa`, `gamma1`, `gamma2`, `defl1`, `defl2`, `hpix_9_nest`, `hpix_13_nest`, `hpix_29_nest`, `step` <br />
FROM flagship_mock_1_10_5_highz_s LIMIT 2;<br />
<br />
----<br />
<br />
Aplicamos los cambios en la pipeline de high-z galaxies (.ipynb) que nos menciona Eric en el email y generamos un nuevo catálogo en formato parquet:<br />
<br />
/software/astro/scripts/create_table_from_parquet.sh -p /hdfs/user/jcarrete/data/euclid/flagship_mock_1_10_7_high_z/0.pq -t jcarrete.flagship_mock_1_10_7_highz_pq -e<br />
<br />
CREATE EXTERNAL TABLE jcarrete.flagship_mock_1_10_7_highz_pq (<br />
`type` STRING,<br />
`z` DOUBLE,<br />
`abs_muv` DOUBLE,<br />
`sedname` STRING,<br />
`sedname_int` BIGINT,<br />
`rhalf_kpc` DOUBLE,<br />
`rhalf_arcsec` DOUBLE,<br />
`muv` DOUBLE,<br />
`beta` DOUBLE,<br />
`r` DOUBLE,<br />
`hpix_9_nest` BIGINT,<br />
`kind` BIGINT,<br />
`step` BIGINT,<br />
`unique_id` BIGINT,<br />
`hpix_29_nest` BIGINT,<br />
`ra` DOUBLE,<br />
`dec` DOUBLE,<br />
`l` DOUBLE,<br />
`b` DOUBLE,<br />
`mw_extinction` DOUBLE,<br />
`blanco_decam_g_true` DOUBLE,<br />
`blanco_decam_g` DOUBLE,<br />
`blanco_decam_g_odonnell_ext` DOUBLE,<br />
`blanco_decam_i_true` DOUBLE,<br />
`blanco_decam_i` DOUBLE,<br />
`blanco_decam_i_odonnell_ext` DOUBLE,<br />
`blanco_decam_r_true` DOUBLE,<br />
`blanco_decam_r` DOUBLE,<br />
`blanco_decam_r_odonnell_ext` DOUBLE,<br />
`blanco_decam_z_true` DOUBLE,<br />
`blanco_decam_z` DOUBLE,<br />
`blanco_decam_z_odonnell_ext` DOUBLE,<br />
`subaru_hsc_z_true` DOUBLE,<br />
`subaru_hsc_z` DOUBLE,<br />
`subaru_hsc_z_odonnell_ext` DOUBLE,<br />
`jst_jpcam_g_true` DOUBLE,<br />
`jst_jpcam_g` DOUBLE,<br />
`jst_jpcam_g_odonnell_ext` DOUBLE,<br />
`kids_g_true` DOUBLE,<br />
`kids_g` DOUBLE,<br />
`kids_g_odonnell_ext` DOUBLE,<br />
`kids_i_true` DOUBLE,<br />
`kids_i` DOUBLE,<br />
`kids_i_odonnell_ext` DOUBLE,<br />
`kids_r_true` DOUBLE,<br />
`kids_r` DOUBLE,<br />
`kids_r_odonnell_ext` DOUBLE,<br />
`kids_u_true` DOUBLE,<br />
`kids_u` DOUBLE,<br />
`kids_u_odonnell_ext` DOUBLE,<br />
`lsst_g_true` DOUBLE,<br />
`lsst_g` DOUBLE,<br />
`lsst_g_odonnell_ext` DOUBLE,<br />
`lsst_i_true` DOUBLE,<br />
`lsst_i` DOUBLE,<br />
`lsst_i_odonnell_ext` DOUBLE,<br />
`lsst_r_true` DOUBLE,<br />
`lsst_r` DOUBLE,<br />
`lsst_r_odonnell_ext` DOUBLE,<br />
`lsst_u_true` DOUBLE,<br />
`lsst_u` DOUBLE,<br />
`lsst_u_odonnell_ext` DOUBLE,<br />
`lsst_y_true` DOUBLE,<br />
`lsst_y` DOUBLE,<br />
`lsst_y_odonnell_ext` DOUBLE,<br />
`lsst_z_true` DOUBLE,<br />
`lsst_z` DOUBLE,<br />
`lsst_z_odonnell_ext` DOUBLE,<br />
`cfht_megacam_r_true` DOUBLE,<br />
`cfht_megacam_r` DOUBLE,<br />
`cfht_megacam_r_odonnell_ext` DOUBLE,<br />
`cfht_megacam_u_true` DOUBLE,<br />
`cfht_megacam_u` DOUBLE,<br />
`cfht_megacam_u_odonnell_ext` DOUBLE,<br />
`pan_starrs_i_true` DOUBLE,<br />
`pan_starrs_i` DOUBLE,<br />
`pan_starrs_i_odonnell_ext` DOUBLE,<br />
`pan_starrs_z_true` DOUBLE,<br />
`pan_starrs_z` DOUBLE,<br />
`pan_starrs_z_odonnell_ext` DOUBLE,<br />
`2mass_h_true` DOUBLE,<br />
`2mass_h` DOUBLE,<br />
`2mass_h_odonnell_ext` DOUBLE,<br />
`2mass_j_true` DOUBLE,<br />
`2mass_j` DOUBLE,<br />
`2mass_j_odonnell_ext` DOUBLE,<br />
`2mass_ks_true` DOUBLE,<br />
`2mass_ks` DOUBLE,<br />
`2mass_ks_odonnell_ext` DOUBLE,<br />
`sdss_r01_true` DOUBLE,<br />
`sdss_r01` DOUBLE,<br />
`sdss_r01_odonnell_ext` DOUBLE,<br />
`gaia_bp_true` DOUBLE,<br />
`gaia_bp` DOUBLE,<br />
`gaia_bp_odonnell_ext` DOUBLE,<br />
`gaia_g_true` DOUBLE,<br />
`gaia_g` DOUBLE,<br />
`gaia_g_odonnell_ext` DOUBLE,<br />
`gaia_rp_true` DOUBLE,<br />
`gaia_rp` DOUBLE,<br />
`gaia_rp_odonnell_ext` DOUBLE,<br />
`euclid_nisp_h_true` DOUBLE,<br />
`euclid_nisp_h` DOUBLE,<br />
`euclid_nisp_h_odonnell_ext` DOUBLE,<br />
`euclid_nisp_j_true` DOUBLE,<br />
`euclid_nisp_j` DOUBLE,<br />
`euclid_nisp_j_odonnell_ext` DOUBLE,<br />
`euclid_nisp_y_true` DOUBLE,<br />
`euclid_nisp_y` DOUBLE,<br />
`euclid_nisp_y_odonnell_ext` DOUBLE,<br />
`euclid_vis_true` DOUBLE,<br />
`euclid_vis` DOUBLE,<br />
`euclid_vis_odonnell_ext` DOUBLE,<br />
`bulge_fraction` DOUBLE,<br />
`bulge_nsersic` DOUBLE,<br />
`bulge_ellipticity` DOUBLE,<br />
`bulge_axis_ratio` DOUBLE,<br />
`disk_angle` DOUBLE,<br />
`id` BIGINT<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/jcarrete/data/euclid/flagship_mock_1_10_7_high_z'<br />
;<br />
<br />
Clusterizo:<br />
<br />
CREATE TABLE cosmohub.flagship_mock_1_10_7_highz_c (<br />
`unique_id` BIGINT COMMENT 'unique identifier',<br />
`id` INT COMMENT 'ID',<br />
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',<br />
`z` FLOAT,<br />
`ra` FLOAT,<br />
`dec` FLOAT,<br />
`l` FLOAT,<br />
`b` FLOAT,<br />
`mw_extinction` FLOAT,<br />
`abs_muv` FLOAT,<br />
`sedname` STRING,<br />
`sedname_int` BIGINT,<br />
`rhalf_kpc` FLOAT,<br />
`rhalf_arcsec` FLOAT,<br />
`muv` FLOAT,<br />
`beta` FLOAT,<br />
`hpix_9_nest` INT,<br />
`hpix_13_nest` INT,<br />
`hpix_29_nest` BIGINT,<br />
`blanco_decam_g_true` FLOAT,<br />
`blanco_decam_g` FLOAT,<br />
`blanco_decam_g_odonnell_ext` FLOAT,<br />
`blanco_decam_i_true` FLOAT,<br />
`blanco_decam_i` FLOAT,<br />
`blanco_decam_i_odonnell_ext` FLOAT,<br />
`blanco_decam_r_true` FLOAT,<br />
`blanco_decam_r` FLOAT,<br />
`blanco_decam_r_odonnell_ext` FLOAT,<br />
`blanco_decam_z_true` FLOAT,<br />
`blanco_decam_z` FLOAT,<br />
`blanco_decam_z_odonnell_ext` FLOAT,<br />
`subaru_hsc_z_true` FLOAT,<br />
`subaru_hsc_z` FLOAT,<br />
`subaru_hsc_z_odonnell_ext` FLOAT,<br />
`jst_jpcam_g_true` FLOAT,<br />
`jst_jpcam_g` FLOAT,<br />
`jst_jpcam_g_odonnell_ext` FLOAT,<br />
`kids_g_true` FLOAT,<br />
`kids_g` FLOAT,<br />
`kids_g_odonnell_ext` FLOAT,<br />
`kids_i_true` FLOAT,<br />
`kids_i` FLOAT,<br />
`kids_i_odonnell_ext` FLOAT,<br />
`kids_r_true` FLOAT,<br />
`kids_r` FLOAT,<br />
`kids_r_odonnell_ext` FLOAT,<br />
`kids_u_true` FLOAT,<br />
`kids_u` FLOAT,<br />
`kids_u_odonnell_ext` FLOAT,<br />
`lsst_g_true` FLOAT,<br />
`lsst_g` FLOAT,<br />
`lsst_g_odonnell_ext` FLOAT,<br />
`lsst_i_true` FLOAT,<br />
`lsst_i` FLOAT,<br />
`lsst_i_odonnell_ext` FLOAT,<br />
`lsst_r_true` FLOAT,<br />
`lsst_r` FLOAT,<br />
`lsst_r_odonnell_ext` FLOAT,<br />
`lsst_u_true` FLOAT,<br />
`lsst_u` FLOAT,<br />
`lsst_u_odonnell_ext` FLOAT,<br />
`lsst_y_true` FLOAT,<br />
`lsst_y` FLOAT,<br />
`lsst_y_odonnell_ext` FLOAT,<br />
`lsst_z_true` FLOAT,<br />
`lsst_z` FLOAT,<br />
`lsst_z_odonnell_ext` FLOAT,<br />
`cfht_megacam_r_true` FLOAT,<br />
`cfht_megacam_r` FLOAT,<br />
`cfht_megacam_r_odonnell_ext` FLOAT,<br />
`cfht_megacam_u_true` FLOAT,<br />
`cfht_megacam_u` FLOAT,<br />
`cfht_megacam_u_odonnell_ext` FLOAT,<br />
`pan_starrs_i_true` FLOAT,<br />
`pan_starrs_i` FLOAT,<br />
`pan_starrs_i_odonnell_ext` FLOAT,<br />
`pan_starrs_z_true` FLOAT,<br />
`pan_starrs_z` FLOAT,<br />
`pan_starrs_z_odonnell_ext` FLOAT,<br />
`2mass_h_true` FLOAT,<br />
`2mass_h` FLOAT,<br />
`2mass_h_odonnell_ext` FLOAT,<br />
`2mass_j_true` FLOAT,<br />
`2mass_j` FLOAT,<br />
`2mass_j_odonnell_ext` FLOAT,<br />
`2mass_ks_true` FLOAT,<br />
`2mass_ks` FLOAT,<br />
`2mass_ks_odonnell_ext` FLOAT,<br />
`sdss_r01_true` FLOAT,<br />
`sdss_r01` FLOAT,<br />
`sdss_r01_odonnell_ext` FLOAT,<br />
`gaia_bp_true` FLOAT,<br />
`gaia_bp` FLOAT,<br />
`gaia_bp_odonnell_ext` FLOAT,<br />
`gaia_g_true` FLOAT,<br />
`gaia_g` FLOAT,<br />
`gaia_g_odonnell_ext` FLOAT,<br />
`gaia_rp_true` FLOAT,<br />
`gaia_rp` FLOAT,<br />
`gaia_rp_odonnell_ext` FLOAT,<br />
`euclid_nisp_h_true` FLOAT,<br />
`euclid_nisp_h` FLOAT,<br />
`euclid_nisp_h_odonnell_ext` FLOAT,<br />
`euclid_nisp_j_true` FLOAT,<br />
`euclid_nisp_j` FLOAT,<br />
`euclid_nisp_j_odonnell_ext` FLOAT,<br />
`euclid_nisp_y_true` FLOAT,<br />
`euclid_nisp_y` FLOAT,<br />
`euclid_nisp_y_odonnell_ext` FLOAT,<br />
`euclid_vis_true` FLOAT,<br />
`euclid_vis` FLOAT,<br />
`euclid_vis_odonnell_ext` FLOAT,<br />
`bulge_fraction` FLOAT,<br />
`bulge_nsersic` FLOAT,<br />
`bulge_ellipticity` FLOAT,<br />
`bulge_axis_ratio` FLOAT,<br />
`disk_angle` FLOAT,<br />
`step` smallint COMMENT 'redshift step number'<br />
)<br />
CLUSTERED BY (<br />
step,<br />
hpix_13_nest)<br />
SORTED BY (<br />
step ASC,<br />
hpix_13_nest ASC)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
INSERT OVERWRITE TABLE cosmohub.flagship_mock_1_10_7_highz_c<br />
SELECT `unique_id`, `id`, `kind`, `z`, `ra`, `dec`, `l`, `b`, `mw_extinction`, `abs_muv`, `sedname`, `sedname_int`, `rhalf_kpc`, `rhalf_arcsec`, `muv`, `beta`, `hpix_9_nest`, SHIFTRIGHT(hpix_29_nest, (29-13)*2), `hpix_29_nest`, `blanco_decam_g_true`, `blanco_decam_g`, `blanco_decam_g_odonnell_ext`, `blanco_decam_i_true`, `blanco_decam_i`, `blanco_decam_i_odonnell_ext`, `blanco_decam_r_true`, `blanco_decam_r`, `blanco_decam_r_odonnell_ext`, `blanco_decam_z_true`, `blanco_decam_z`, `blanco_decam_z_odonnell_ext`, `subaru_hsc_z_true`, `subaru_hsc_z`, `subaru_hsc_z_odonnell_ext`, `jst_jpcam_g_true`, `jst_jpcam_g`, `jst_jpcam_g_odonnell_ext`, `kids_g_true`, `kids_g`, `kids_g_odonnell_ext`, `kids_i_true`, `kids_i`, `kids_i_odonnell_ext`, `kids_r_true`, `kids_r`, `kids_r_odonnell_ext`, `kids_u_true`, `kids_u`, `kids_u_odonnell_ext`, `lsst_g_true`, `lsst_g`, `lsst_g_odonnell_ext`, `lsst_i_true`, `lsst_i`, `lsst_i_odonnell_ext`, `lsst_r_true`, `lsst_r`, `lsst_r_odonnell_ext`, `lsst_u_true`, `lsst_u`, `lsst_u_odonnell_ext`, `lsst_y_true`, `lsst_y`, `lsst_y_odonnell_ext`, `lsst_z_true`, `lsst_z`, `lsst_z_odonnell_ext`, `cfht_megacam_r_true`, `cfht_megacam_r`, `cfht_megacam_r_odonnell_ext`, `cfht_megacam_u_true`, `cfht_megacam_u`, `cfht_megacam_u_odonnell_ext`, `pan_starrs_i_true`, `pan_starrs_i`, `pan_starrs_i_odonnell_ext`, `pan_starrs_z_true`, `pan_starrs_z`, `pan_starrs_z_odonnell_ext`, `2mass_h_true`, `2mass_h`, `2mass_h_odonnell_ext`, `2mass_j_true`, `2mass_j`, `2mass_j_odonnell_ext`, `2mass_ks_true`, `2mass_ks`, `2mass_ks_odonnell_ext`, `sdss_r01_true`, `sdss_r01`, `sdss_r01_odonnell_ext`, `gaia_bp_true`, `gaia_bp`, `gaia_bp_odonnell_ext`, `gaia_g_true`, `gaia_g`, `gaia_g_odonnell_ext`, `gaia_rp_true`, `gaia_rp`, `gaia_rp_odonnell_ext`, `euclid_nisp_h_true`, `euclid_nisp_h`, `euclid_nisp_h_odonnell_ext`, `euclid_nisp_j_true`, `euclid_nisp_j`, `euclid_nisp_j_odonnell_ext`, `euclid_nisp_y_true`, `euclid_nisp_y`, `euclid_nisp_y_odonnell_ext`, `euclid_vis_true`, `euclid_vis`, `euclid_vis_odonnell_ext`, `bulge_fraction`, `bulge_nsersic`, `bulge_ellipticity`, `bulge_axis_ratio`, `disk_angle`, `step`<br />
FROM jcarrete.flagship_mock_1_10_7_highz_pq;<br />
<br />
ANALYZE TABLE cosmohub.flagship_mock_1_10_7_highz_c COMPUTE STATISTICS FOR COLUMNS;<br />
<br />
Añado el lensing:<br />
<br />
CREATE TABLE cosmohub.flagship_mock_1_10_7_highz_s (<br />
`unique_id` BIGINT COMMENT 'unique identifier',<br />
`id` INT COMMENT 'ID',<br />
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',<br />
`z` float COMMENT 'true galaxy redshift',<br />
`ra` FLOAT COMMENT 'galaxy right ascension (degrees)',<br />
`dec` FLOAT COMMENT 'galaxy declination (degrees)',<br />
`ra_mag` float COMMENT 'galaxy magnified right ascension (degree)', <br />
`dec_mag` float COMMENT 'galaxy magnified declination (degree)', <br />
`kappa` float COMMENT 'convergence', <br />
`gamma1` float COMMENT 'shear', <br />
`gamma2` float COMMENT 'shear',<br />
`defl1` float COMMENT 'deflection1',<br />
`defl2` float COMMENT 'deflection2',<br />
`l` FLOAT COMMENT 'galactic longitude (degrees)',<br />
`b` FLOAT COMMENT 'galactic latitude (degrees)',<br />
`mw_extinction` float COMMENT 'EBV from file: HFI_CompMap_ThermalDustModel_2048_R1.20.fits',<br />
`abs_muv` FLOAT COMMENT 'absolute magnitude at 1500 angstroms',<br />
`sedname` STRING COMMENT 'Spectral Energy Distribution name (among 76, 38 without UV lines, 38 with UV lines, corresponding to a given UV slope)',<br />
`sedname_int` BIGINT COMMENT 'sedname as integer (from 0 to 75, from 0 to 37 without UV lines, from 38 to 75 with UV lines)',<br />
`rhalf_kpc` FLOAT COMMENT 'half light radius (in physical kpc)',<br />
`rhalf_arcsec` FLOAT COMMENT 'half light radius (in arcsec)',<br />
`muv` FLOAT COMMENT 'for information only, observed magnitude at rest-frame 1500',<br />
`beta` FLOAT COMMENT 'for information only, used to assign an SED',<br />
`hpix_9_nest` int COMMENT 'healpix pixel id using order = 9 in NESTED format',<br />
`hpix_13_nest` int COMMENT 'healpix pixel id using order = 13 in NESTED format',<br />
`hpix_29_nest` bigint COMMENT 'healpix pixel id using order = 29 in NESTED format',<br />
`blanco_decam_g_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in DES g-band (Euclid provided)', <br />
`blanco_decam_g` float COMMENT 'Interpolated observed flux in DES g-band (Euclid provided)',<br />
`blanco_decam_g_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in DES g-band (Euclid provided)',<br />
`blanco_decam_i_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in DES i-band (Euclid provided)', <br />
`blanco_decam_i` float COMMENT 'Interpolated observed flux in DES i-band (Euclid provided)', <br />
`blanco_decam_i_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in DES i-band (Euclid provided)', <br />
`blanco_decam_r_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in DES r-band (Euclid provided)', <br />
`blanco_decam_r` float COMMENT 'Interpolated observed flux in DES r-band (Euclid provided)', <br />
`blanco_decam_r_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in DES r-band (Euclid provided)', <br />
`blanco_decam_z_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in DES z-band (Euclid provided)', <br />
`blanco_decam_z` float COMMENT 'Interpolated observed flux in DES z-band (Euclid provided)',<br />
`blanco_decam_z_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in DES z-band (Euclid provided)', <br />
`subaru_hsc_z_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',<br />
`subaru_hsc_z` float COMMENT 'Interpolated observed flux in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',<br />
`subaru_hsc_z_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',<br />
`jst_jpcam_g_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in JST g-band (Euclid provided)', <br />
`jst_jpcam_g` float COMMENT 'Interpolated observed flux in JST g-band (Euclid provided)',<br />
`jst_jpcam_g_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in JST g-band (Euclid provided)',<br />
`kids_g_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in KIDS g-band (Euclid provided)', <br />
`kids_g` float COMMENT 'Interpolated observed flux in KIDS g-band (Euclid provided)',<br />
`kids_g_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in KIDS g-band (Euclid provided)',<br />
`kids_i_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in KIDS i-band (Euclid provided)', <br />
`kids_i` float COMMENT 'Interpolated observed flux in KIDS i-band (Euclid provided)',<br />
`kids_i_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in KIDS i-band (Euclid provided)', <br />
`kids_r_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in KIDS r-band (Euclid provided)', <br />
`kids_r` float COMMENT 'Interpolated observed flux in KIDS r-band (Euclid provided)',<br />
`kids_r_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in KIDS r-band (Euclid provided)',<br />
`kids_u_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in KIDS u-band (Euclid provided)', <br />
`kids_u` float COMMENT 'Interpolated observed flux in KIDS u-band (Euclid provided)',<br />
`kids_u_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in KIDS u-band (Euclid provided)',<br />
`lsst_g_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST g-band (Euclid provided)', <br />
`lsst_g` float COMMENT 'Interpolated observed flux in LSST g-band (Euclid provided)',<br />
`lsst_g_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST g-band (Euclid provided)',<br />
`lsst_i_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST i-band (Euclid provided)', <br />
`lsst_i` float COMMENT 'Interpolated observed flux in LSST i-band (Euclid provided)',<br />
`lsst_i_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST i-band (Euclid provided)',<br />
`lsst_r_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST r-band (Euclid provided)', <br />
`lsst_r` float COMMENT 'Interpolated observed flux in LSST r-band (Euclid provided)',<br />
`lsst_r_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST r-band (Euclid provided)',<br />
`lsst_u_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST u-band (Euclid provided)', <br />
`lsst_u` float COMMENT 'Interpolated observed flux in LSST u-band (Euclid provided)',<br />
`lsst_u_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST u-band (Euclid provided)',<br />
`lsst_y_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST y-band (Euclid provided)', <br />
`lsst_y` float COMMENT 'Interpolated observed flux in LSST y-band (Euclid provided)',<br />
`lsst_y_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST y-band (Euclid provided)',<br />
`lsst_z_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST z-band (Euclid provided)', <br />
`lsst_z` float COMMENT 'Interpolated observed flux in LSST z-band (Euclid provided)',<br />
`lsst_z_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST z-band (Euclid provided)',<br />
`cfht_megacam_r_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in CFHT r-band (Euclid provided)', <br />
`cfht_megacam_r` float COMMENT 'Interpolated observed flux in CFHT r-band (Euclid provided)',<br />
`cfht_megacam_r_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in CFHT r-band (Euclid provided)',<br />
`cfht_megacam_u_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in CFHT u-band (Euclid provided)', <br />
`cfht_megacam_u` float COMMENT 'Interpolated observed flux in CFHT u-band (Euclid provided)',<br />
`cfht_megacam_u_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in CFHT u-band (Euclid provided)',<br />
`pan_starrs_i_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Pan-STARRS i-band (Euclid provided)', <br />
`pan_starrs_i` float COMMENT 'Interpolated observed flux in Pan-STARRS i-band (Euclid provided)',<br />
`pan_starrs_i_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Pan-STARRS i-band (Euclid provided)',<br />
`pan_starrs_z_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Pan-STARRS z-band (Euclid provided)', <br />
`pan_starrs_z` float COMMENT 'Interpolated observed flux in Pan-STARRS z-band (Euclid provided)',<br />
`pan_starrs_z_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Pan-STARRS z-band (Euclid provided)',<br />
`2mass_h_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in 2MASS H-band (Euclid provided)', <br />
`2mass_h` float COMMENT 'Interpolated observed flux in 2MASS H-band (Euclid provided)',<br />
`2mass_h_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in 2MASS H-band (Euclid provided)',<br />
`2mass_j_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in 2MASS J-band (Euclid provided)', <br />
`2mass_j` float COMMENT 'Interpolated observed flux in 2MASS J-band (Euclid provided)',<br />
`2mass_j_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in 2MASS J-band (Euclid provided)',<br />
`2mass_ks_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in 2MASS Ks-band (Euclid provided)', <br />
`2mass_ks` float COMMENT 'Interpolated observed flux in 2MASS Ks-band (Euclid provided)',<br />
`2mass_ks_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in 2MASS Ks-band (Euclid provided)',<br />
`sdss_r01_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in sdss r-band redshifted to z=0.1 (Euclid provided)', <br />
`sdss_r01` float COMMENT 'Interpolated observed flux in sdss r-band redshifted to z=0.1 (Euclid provided)',<br />
`sdss_r01_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in sdss r-band redshifted to z=0.1 (Euclid provided)',<br />
`gaia_bp_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Gaia-BP band (Euclid provided)', <br />
`gaia_bp` float COMMENT 'Interpolated observed flux in Gaia-BP band (Euclid provided)',<br />
`gaia_bp_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Gaia-BP band (Euclid provided)',<br />
`gaia_g_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Gaia-G band (Euclid provided)', <br />
`gaia_g` float COMMENT 'Interpolated observed flux in Gaia-G band (Euclid provided)',<br />
`gaia_g_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Gaia-G band (Euclid provided)', <br />
`gaia_rp_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Gaia-RP band (Euclid provided)', <br />
`gaia_rp` float COMMENT 'Interpolated observed flux in Gaia-RP band (Euclid provided)',<br />
`gaia_rp_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Gaia-RP band (Euclid provided)',<br />
`euclid_nisp_h_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Euclid NISP-H band (Euclid provided)', <br />
`euclid_nisp_h` float COMMENT 'Interpolated observed flux in Euclid NISP-H band (Euclid provided)',<br />
`euclid_nisp_h_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Euclid NISP-H band (Euclid provided)', <br />
`euclid_nisp_j_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Euclid NISP-J band (Euclid provided)', <br />
`euclid_nisp_j` float COMMENT 'Interpolated observed flux in Euclid NISP-J band (Euclid provided)',<br />
`euclid_nisp_j_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Euclid NISP-J band (Euclid provided)',<br />
`euclid_nisp_y_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Euclid NISP-Y band (Euclid provided)', <br />
`euclid_nisp_y` float COMMENT 'Interpolated observed flux in Euclid NISP-Y band (Euclid provided)',<br />
`euclid_nisp_y_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Euclid NISP-Y band (Euclid provided)', <br />
`euclid_vis_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Euclid VIS band (Euclid provided)', <br />
`euclid_vis` float COMMENT 'Interpolated observed flux in Euclid VIS band (Euclid provided)',<br />
`euclid_vis_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Euclid VIS band (Euclid provided)',<br />
`bulge_fraction` FLOAT COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)',<br />
`bulge_nsersic` FLOAT COMMENT 'Sersic index of the bulge component',<br />
`bulge_ellipticity` FLOAT COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)', <br />
`bulge_axis_ratio` FLOAT COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`disk_angle` FLOAT COMMENT 'position of the disk rotation axis (degrees). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',<br />
`step` smallint COMMENT 'redshift step number'<br />
)<br />
CLUSTERED BY (<br />
step,<br />
hpix_13_nest)<br />
SORTED BY (<br />
step ASC,<br />
hpix_13_nest ASC)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
SET hive.auto.convert.join.noconditionaltask.size=1000;<br />
<br />
INSERT OVERWRITE TABLE cosmohub.flagship_mock_1_10_7_highz_s<br />
SELECT <br />
`unique_id`, `id`, `kind`, `z`, `ra`, `dec`, mag_pos.ra_mag, mag_pos.dec_mag, kappa, gamma1, gamma2, defl1, defl2, `l`, `b`, `mw_extinction`, `abs_muv`, `sedname`, `sedname_int`, `rhalf_kpc`, `rhalf_arcsec`, `muv`, `beta`, `hpix_9_nest`, `hpix_13_nest`, `hpix_29_nest`, `blanco_decam_g_true`, `blanco_decam_g`, `blanco_decam_g_odonnell_ext`, `blanco_decam_i_true`, `blanco_decam_i`, `blanco_decam_i_odonnell_ext`, `blanco_decam_r_true`, `blanco_decam_r`, `blanco_decam_r_odonnell_ext`, `blanco_decam_z_true`, `blanco_decam_z`, `blanco_decam_z_odonnell_ext`, `subaru_hsc_z_true`, `subaru_hsc_z`, `subaru_hsc_z_odonnell_ext`, `jst_jpcam_g_true`, `jst_jpcam_g`, `jst_jpcam_g_odonnell_ext`, `kids_g_true`, `kids_g`, `kids_g_odonnell_ext`, `kids_i_true`, `kids_i`, `kids_i_odonnell_ext`, `kids_r_true`, `kids_r`, `kids_r_odonnell_ext`, `kids_u_true`, `kids_u`, `kids_u_odonnell_ext`, `lsst_g_true`, `lsst_g`, `lsst_g_odonnell_ext`, `lsst_i_true`, `lsst_i`, `lsst_i_odonnell_ext`, `lsst_r_true`, `lsst_r`, `lsst_r_odonnell_ext`, `lsst_u_true`, `lsst_u`, `lsst_u_odonnell_ext`, `lsst_y_true`, `lsst_y`, `lsst_y_odonnell_ext`, `lsst_z_true`, `lsst_z`, `lsst_z_odonnell_ext`, `cfht_megacam_r_true`, `cfht_megacam_r`, `cfht_megacam_r_odonnell_ext`, `cfht_megacam_u_true`, `cfht_megacam_u`, `cfht_megacam_u_odonnell_ext`, `pan_starrs_i_true`, `pan_starrs_i`, `pan_starrs_i_odonnell_ext`, `pan_starrs_z_true`, `pan_starrs_z`, `pan_starrs_z_odonnell_ext`, `2mass_h_true`, `2mass_h`, `2mass_h_odonnell_ext`, `2mass_j_true`, `2mass_j`, `2mass_j_odonnell_ext`, `2mass_ks_true`, `2mass_ks`, `2mass_ks_odonnell_ext`, `sdss_r01_true`, `sdss_r01`, `sdss_r01_odonnell_ext`, `gaia_bp_true`, `gaia_bp`, `gaia_bp_odonnell_ext`, `gaia_g_true`, `gaia_g`, `gaia_g_odonnell_ext`, `gaia_rp_true`, `gaia_rp`, `gaia_rp_odonnell_ext`, `euclid_nisp_h_true`, `euclid_nisp_h`, `euclid_nisp_h_odonnell_ext`, `euclid_nisp_j_true`, `euclid_nisp_j`, `euclid_nisp_j_odonnell_ext`, `euclid_nisp_y_true`, `euclid_nisp_y`, `euclid_nisp_y_odonnell_ext`, `euclid_vis_true`, `euclid_vis`, `euclid_vis_odonnell_ext`, `bulge_fraction`, `bulge_nsersic`, `bulge_ellipticity`, `bulge_axis_ratio`, `disk_angle`, `step`<br />
FROM<br />
(SELECT <br />
`unique_id`, `id`, `kind`, `z`, `ra`, `dec`, udf.magnified_positions(CAST(ra AS DOUBLE), CAST(`dec` AS DOUBLE), CAST(l.defl1 AS DOUBLE), CAST(l.defl2 AS DOUBLE)) as mag_pos, l.kappa, l.gamma1, l.gamma2, l.defl1, l.defl2, `l`, `b`, `mw_extinction`, `abs_muv`, `sedname`, `sedname_int`, `rhalf_kpc`, `rhalf_arcsec`, `muv`, `beta`, `hpix_9_nest`, l.hpix_13_nest, `hpix_29_nest`, `blanco_decam_g_true`, `blanco_decam_g`, `blanco_decam_g_odonnell_ext`, `blanco_decam_i_true`, `blanco_decam_i`, `blanco_decam_i_odonnell_ext`, `blanco_decam_r_true`, `blanco_decam_r`, `blanco_decam_r_odonnell_ext`, `blanco_decam_z_true`, `blanco_decam_z`, `blanco_decam_z_odonnell_ext`, `subaru_hsc_z_true`, `subaru_hsc_z`, `subaru_hsc_z_odonnell_ext`, `jst_jpcam_g_true`, `jst_jpcam_g`, `jst_jpcam_g_odonnell_ext`, `kids_g_true`, `kids_g`, `kids_g_odonnell_ext`, `kids_i_true`, `kids_i`, `kids_i_odonnell_ext`, `kids_r_true`, `kids_r`, `kids_r_odonnell_ext`, `kids_u_true`, `kids_u`, `kids_u_odonnell_ext`, `lsst_g_true`, `lsst_g`, `lsst_g_odonnell_ext`, `lsst_i_true`, `lsst_i`, `lsst_i_odonnell_ext`, `lsst_r_true`, `lsst_r`, `lsst_r_odonnell_ext`, `lsst_u_true`, `lsst_u`, `lsst_u_odonnell_ext`, `lsst_y_true`, `lsst_y`, `lsst_y_odonnell_ext`, `lsst_z_true`, `lsst_z`, `lsst_z_odonnell_ext`, `cfht_megacam_r_true`, `cfht_megacam_r`, `cfht_megacam_r_odonnell_ext`, `cfht_megacam_u_true`, `cfht_megacam_u`, `cfht_megacam_u_odonnell_ext`, `pan_starrs_i_true`, `pan_starrs_i`, `pan_starrs_i_odonnell_ext`, `pan_starrs_z_true`, `pan_starrs_z`, `pan_starrs_z_odonnell_ext`, `2mass_h_true`, `2mass_h`, `2mass_h_odonnell_ext`, `2mass_j_true`, `2mass_j`, `2mass_j_odonnell_ext`, `2mass_ks_true`, `2mass_ks`, `2mass_ks_odonnell_ext`, `sdss_r01_true`, `sdss_r01`, `sdss_r01_odonnell_ext`, `gaia_bp_true`, `gaia_bp`, `gaia_bp_odonnell_ext`, `gaia_g_true`, `gaia_g`, `gaia_g_odonnell_ext`, `gaia_rp_true`, `gaia_rp`, `gaia_rp_odonnell_ext`, `euclid_nisp_h_true`, `euclid_nisp_h`, `euclid_nisp_h_odonnell_ext`, `euclid_nisp_j_true`, `euclid_nisp_j`, `euclid_nisp_j_odonnell_ext`, `euclid_nisp_y_true`, `euclid_nisp_y`, `euclid_nisp_y_odonnell_ext`, `euclid_vis_true`, `euclid_vis`, `euclid_vis_odonnell_ext`, `bulge_fraction`, `bulge_nsersic`, `bulge_ellipticity`, `bulge_axis_ratio`, `disk_angle`, l.step<br />
FROM cosmohub.flagship_mock_1_10_7_highz_c as m<br />
JOIN cosmohub.flagship_lensing2 as l<br />
ON m.step = l.step<br />
AND m.hpix_13_nest = l.hpix_13_nest) as t;<br />
<br />
ANALYZE TABLE cosmohub.flagship_mock_1_10_7_highz_s COMPUTE STATISTICS FOR COLUMNS;</div>Talladahttps://pwiki.pic.es/index.php?title=Gaia_EDR3&diff=885Gaia EDR32020-12-03T17:40:35Z<p>Tallada: </p>
<hr />
<div>== Raw data ==<br />
<br />
Mirrored to cephfs using lftp:<br />
<br />
mirror -r -v -c -P4 http://cdn.gea.esac.esa.int/Gaia/gedr3/gaia_source/ /cephfs/pic.es/astro/scratch/tallada/gaia_edr3<br />
<br />
== External table ==<br />
<br />
CREATE EXTERNAL TABLE tallada.gaia_edr3_csv (<br />
solution_id bigint,<br />
designation string,<br />
source_id bigint,<br />
random_index bigint,<br />
ref_epoch float,<br />
ra double,<br />
ra_error float,<br />
`dec` double,<br />
dec_error float,<br />
parallax double,<br />
parallax_error float,<br />
parallax_over_error float,<br />
pm float,<br />
pmra double,<br />
pmra_error float,<br />
pmdec double,<br />
pmdec_error float,<br />
ra_dec_corr float,<br />
ra_parallax_corr float,<br />
ra_pmra_corr float,<br />
ra_pmdec_corr float,<br />
dec_parallax_corr float,<br />
dec_pmra_corr float,<br />
dec_pmdec_corr float,<br />
parallax_pmra_corr float,<br />
parallax_pmdec_corr float,<br />
pmra_pmdec_corr float,<br />
astrometric_n_obs_al smallint,<br />
astrometric_n_obs_ac smallint,<br />
astrometric_n_good_obs_al smallint,<br />
astrometric_n_bad_obs_al smallint,<br />
astrometric_gof_al float,<br />
astrometric_chi2_al float,<br />
astrometric_excess_noise float,<br />
astrometric_excess_noise_sig float,<br />
astrometric_params_solved tinyint,<br />
astrometric_primary_flag boolean,<br />
nu_eff_used_in_astrometry float,<br />
pseudocolour float,<br />
pseudocolour_error float,<br />
ra_pseudocolour_corr float,<br />
dec_pseudocolour_corr float,<br />
parallax_pseudocolour_corr float,<br />
pmra_pseudocolour_corr float,<br />
pmdec_pseudocolour_corr float,<br />
astrometric_matched_transits smallint,<br />
visibility_periods_used smallint,<br />
astrometric_sigma5d_max float,<br />
matched_transits smallint,<br />
new_matched_transits smallint,<br />
matched_transits_removed smallint,<br />
ipd_gof_harmonic_amplitude float,<br />
ipd_gof_harmonic_phase float,<br />
ipd_frac_multi_peak tinyint,<br />
ipd_frac_odd_win tinyint,<br />
ruwe float,<br />
scan_direction_strength_k1 float,<br />
scan_direction_strength_k2 float,<br />
scan_direction_strength_k3 float,<br />
scan_direction_strength_k4 float,<br />
scan_direction_mean_k1 float,<br />
scan_direction_mean_k2 float,<br />
scan_direction_mean_k3 float,<br />
scan_direction_mean_k4 float,<br />
duplicated_source boolean,<br />
phot_g_n_obs smallint,<br />
phot_g_mean_flux double,<br />
phot_g_mean_flux_error float,<br />
phot_g_mean_flux_over_error float,<br />
phot_g_mean_mag float,<br />
phot_bp_n_obs smallint,<br />
phot_bp_mean_flux double,<br />
phot_bp_mean_flux_error float,<br />
phot_bp_mean_flux_over_error float,<br />
phot_bp_mean_mag float,<br />
phot_rp_n_obs smallint,<br />
phot_rp_mean_flux double,<br />
phot_rp_mean_flux_error float,<br />
phot_rp_mean_flux_over_error float,<br />
phot_rp_mean_mag float,<br />
phot_bp_n_contaminated_transits smallint,<br />
phot_bp_n_blended_transits smallint,<br />
phot_rp_n_contaminated_transits smallint,<br />
phot_rp_n_blended_transits smallint,<br />
phot_proc_mode tinyint,<br />
phot_bp_rp_excess_factor float,<br />
bp_rp float,<br />
bp_g float,<br />
g_rp float,<br />
dr2_radial_velocity float,<br />
dr2_radial_velocity_error float,<br />
dr2_rv_nb_transits smallint,<br />
dr2_rv_template_teff float,<br />
dr2_rv_template_logg float,<br />
dr2_rv_template_fe_h float,<br />
l double,<br />
b double,<br />
ecl_lon double,<br />
ecl_lat double<br />
)<br />
ROW FORMAT DELIMITED<br />
FIELDS TERMINATED BY ','<br />
LOCATION '/user/tallada/data/gaia_edr3'<br />
;<br />
<br />
<br />
== Internal table ==<br />
<br />
CREATE TABLE cosmohub.gaia_edr3 (<br />
solution_id bigint,<br />
designation string,<br />
source_id bigint,<br />
random_index bigint,<br />
ref_epoch float,<br />
ra double,<br />
ra_error float,<br />
`dec` double,<br />
dec_error float,<br />
parallax double,<br />
parallax_error float,<br />
parallax_over_error float,<br />
pm float,<br />
pmra double,<br />
pmra_error float,<br />
pmdec double,<br />
pmdec_error float,<br />
ra_dec_corr float,<br />
ra_parallax_corr float,<br />
ra_pmra_corr float,<br />
ra_pmdec_corr float,<br />
dec_parallax_corr float,<br />
dec_pmra_corr float,<br />
dec_pmdec_corr float,<br />
parallax_pmra_corr float,<br />
parallax_pmdec_corr float,<br />
pmra_pmdec_corr float,<br />
astrometric_n_obs_al smallint,<br />
astrometric_n_obs_ac smallint,<br />
astrometric_n_good_obs_al smallint,<br />
astrometric_n_bad_obs_al smallint,<br />
astrometric_gof_al float,<br />
astrometric_chi2_al float,<br />
astrometric_excess_noise float,<br />
astrometric_excess_noise_sig float,<br />
astrometric_params_solved tinyint,<br />
astrometric_primary_flag boolean,<br />
nu_eff_used_in_astrometry float,<br />
pseudocolour float,<br />
pseudocolour_error float,<br />
ra_pseudocolour_corr float,<br />
dec_pseudocolour_corr float,<br />
parallax_pseudocolour_corr float,<br />
pmra_pseudocolour_corr float,<br />
pmdec_pseudocolour_corr float,<br />
astrometric_matched_transits smallint,<br />
visibility_periods_used smallint,<br />
astrometric_sigma5d_max float,<br />
matched_transits smallint,<br />
new_matched_transits smallint,<br />
matched_transits_removed smallint,<br />
ipd_gof_harmonic_amplitude float,<br />
ipd_gof_harmonic_phase float,<br />
ipd_frac_multi_peak tinyint,<br />
ipd_frac_odd_win tinyint,<br />
ruwe float,<br />
scan_direction_strength_k1 float,<br />
scan_direction_strength_k2 float,<br />
scan_direction_strength_k3 float,<br />
scan_direction_strength_k4 float,<br />
scan_direction_mean_k1 float,<br />
scan_direction_mean_k2 float,<br />
scan_direction_mean_k3 float,<br />
scan_direction_mean_k4 float,<br />
duplicated_source boolean,<br />
phot_g_n_obs smallint,<br />
phot_g_mean_flux double,<br />
phot_g_mean_flux_error float,<br />
phot_g_mean_flux_over_error float,<br />
phot_g_mean_mag float,<br />
phot_bp_n_obs smallint,<br />
phot_bp_mean_flux double,<br />
phot_bp_mean_flux_error float,<br />
phot_bp_mean_flux_over_error float,<br />
phot_bp_mean_mag float,<br />
phot_rp_n_obs smallint,<br />
phot_rp_mean_flux double,<br />
phot_rp_mean_flux_error float,<br />
phot_rp_mean_flux_over_error float,<br />
phot_rp_mean_mag float,<br />
phot_bp_n_contaminated_transits smallint,<br />
phot_bp_n_blended_transits smallint,<br />
phot_rp_n_contaminated_transits smallint,<br />
phot_rp_n_blended_transits smallint,<br />
phot_proc_mode tinyint,<br />
phot_bp_rp_excess_factor float,<br />
bp_rp float,<br />
bp_g float,<br />
g_rp float,<br />
dr2_radial_velocity float,<br />
dr2_radial_velocity_error float,<br />
dr2_rv_nb_transits smallint,<br />
dr2_rv_template_teff float,<br />
dr2_rv_template_logg float,<br />
dr2_rv_template_fe_h float,<br />
l double,<br />
b double,<br />
ecl_lon double,<br />
ecl_lat double<br />
)<br />
CLUSTERED BY (source_id)<br />
SORTED BY (source_id ASC)<br />
INTO 2048 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
INSERT INTO cosmohub.gaia_edr3<br />
SELECT * FROM tallada.gaia_edr3_csv<br />
WHERE source_id IS NOT NULL<br />
;<br />
<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE solution_id solution_id bigint COMMENT "Solution Identifier";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE designation designation string COMMENT "Unique source designation, unique across all Data Releases";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE source_id source_id bigint COMMENT "Unique source identifier, unique within a particular Data Release";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE random_index random_index bigint COMMENT "Random index used to select subsets";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ref_epoch ref_epoch float COMMENT "Reference epoch";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ra ra double COMMENT "Right ascension";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ra_error ra_error float COMMENT "Standard error of right ascension";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE `dec` `dec` double COMMENT "Declination";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dec_error dec_error float COMMENT "Standard error of declination";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE parallax parallax double COMMENT "Parallax";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE parallax_error parallax_error float COMMENT "Standard error of parallax";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE parallax_over_error parallax_over_error float COMMENT "Parallax divided by its standard error";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE pm pm float COMMENT "Total proper motion";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE pmra pmra double COMMENT "Proper motion in right ascension direction";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE pmra_error pmra_error float COMMENT "Standard error of proper motion in right ascension direction";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE pmdec pmdec double COMMENT "Proper motion in declination direction";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE pmdec_error pmdec_error float COMMENT "Standard error of proper motion in declination direction";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ra_dec_corr ra_dec_corr float COMMENT "Correlation between right ascension and declination";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ra_parallax_corr ra_parallax_corr float COMMENT "Correlation between right ascension and parallax";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ra_pmra_corr ra_pmra_corr float COMMENT "Correlation between right ascension and proper motion in right ascension";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ra_pmdec_corr ra_pmdec_corr float COMMENT "Correlation between right ascension and proper motion in declination";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dec_parallax_corr dec_parallax_corr float COMMENT "Correlation between declination and parallax";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dec_pmra_corr dec_pmra_corr float COMMENT "Correlation between declination and proper motion in right ascension";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dec_pmdec_corr dec_pmdec_corr float COMMENT "Correlation between declination and proper motion in declination";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE parallax_pmra_corr parallax_pmra_corr float COMMENT "Correlation between parallax and proper motion in right ascension";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE parallax_pmdec_corr parallax_pmdec_corr float COMMENT "Correlation between parallax and proper motion in declination";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE pmra_pmdec_corr pmra_pmdec_corr float COMMENT "Correlation between proper motion in right ascension and proper motion in declination";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_n_obs_al astrometric_n_obs_al smallint COMMENT "Total number of observations AL";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_n_obs_ac astrometric_n_obs_ac smallint COMMENT "Total number of observations AC";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_n_good_obs_al astrometric_n_good_obs_al smallint COMMENT "Number of good observations AL";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_n_bad_obs_al astrometric_n_bad_obs_al smallint COMMENT "Number of bad observations AL";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_gof_al astrometric_gof_al float COMMENT "Goodness of fit statistic of model wrt along-scan observations";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_chi2_al astrometric_chi2_al float COMMENT "AL chi-square value";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_excess_noise astrometric_excess_noise float COMMENT "Excess noise of the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_excess_noise_sig astrometric_excess_noise_sig float COMMENT "Significance of excess noise";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_params_solved astrometric_params_solved tinyint COMMENT "Which parameters have been solved for?";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_primary_flag astrometric_primary_flag boolean COMMENT "Primary or seconday";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE nu_eff_used_in_astrometry nu_eff_used_in_astrometry float COMMENT "Effective wavenumber of the source used in the astrometric solution";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE pseudocolour pseudocolour float COMMENT "Astrometrically estimated pseudocolour of the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE pseudocolour_error pseudocolour_error float COMMENT "Standard error of the pseudocolour of the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ra_pseudocolour_corr ra_pseudocolour_corr float COMMENT "Correlation between right ascension and pseudocolour";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dec_pseudocolour_corr dec_pseudocolour_corr float COMMENT "Correlation between declination and pseudocolour";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE parallax_pseudocolour_corr parallax_pseudocolour_corr float COMMENT "Correlation between parallax and pseudocolour";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE pmra_pseudocolour_corr pmra_pseudocolour_corr float COMMENT "Correlation between proper motion in right asension and pseudocolour";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE pmdec_pseudocolour_corr pmdec_pseudocolour_corr float COMMENT "Correlation between proper motion in declination and pseudocolour";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_matched_transits astrometric_matched_transits smallint COMMENT "Matched FOV transits used in the AGIS solution";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE visibility_periods_used visibility_periods_used smallint COMMENT "Number of visibility periods used in Astrometric solution";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_sigma5d_max astrometric_sigma5d_max float COMMENT "The longest semi-major axis of the 5-d error ellipsoid";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE matched_transits matched_transits smallint COMMENT "The number of transits matched to this source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE new_matched_transits new_matched_transits smallint COMMENT "The number of transits newly incorporated into an existing source in the current cycle";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE matched_transits_removed matched_transits_removed smallint COMMENT "The number of transits removed from an existing source in the current cycle";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ipd_gof_harmonic_amplitude ipd_gof_harmonic_amplitude float COMMENT "Amplitude of the IPD GoF versus position angle of scan";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ipd_gof_harmonic_phase ipd_gof_harmonic_phase float COMMENT "Phase of the IPD GoF versus position angle of scan";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ipd_frac_multi_peak ipd_frac_multi_peak tinyint COMMENT "Percent of successful-IPD windows with more than one peak";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ipd_frac_odd_win ipd_frac_odd_win tinyint COMMENT "Percent of transits with truncated windows or multiple gate";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ruwe ruwe float COMMENT "Renormalised unit weight error";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE scan_direction_strength_k1 scan_direction_strength_k1 float COMMENT "Degree of concentration of scan directions across the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE scan_direction_strength_k2 scan_direction_strength_k2 float COMMENT "Degree of concentration of scan directions across the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE scan_direction_strength_k3 scan_direction_strength_k3 float COMMENT "Degree of concentration of scan directions across the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE scan_direction_strength_k4 scan_direction_strength_k4 float COMMENT "Degree of concentration of scan directions across the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE scan_direction_mean_k1 scan_direction_mean_k1 float COMMENT "Mean position angle of scan directions across the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE scan_direction_mean_k2 scan_direction_mean_k2 float COMMENT "Mean position angle of scan directions across the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE scan_direction_mean_k3 scan_direction_mean_k3 float COMMENT "Mean position angle of scan directions across the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE scan_direction_mean_k4 scan_direction_mean_k4 float COMMENT "Mean position angle of scan directions across the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE duplicated_source duplicated_source boolean COMMENT "Source with multiple source identifiers";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_g_n_obs phot_g_n_obs smallint COMMENT "Number of observations contributing to G photometry";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_g_mean_flux phot_g_mean_flux double COMMENT "G-band mean flux";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_g_mean_flux_error phot_g_mean_flux_error float COMMENT "Error on G-band mean flux";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_g_mean_flux_over_error phot_g_mean_flux_over_error float COMMENT "G-band mean flux divided by its error";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_g_mean_mag phot_g_mean_mag float COMMENT "G-band mean magnitude";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_bp_n_obs phot_bp_n_obs smallint COMMENT "Number of observations contributing to BP photometry";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_bp_mean_flux phot_bp_mean_flux double COMMENT "Integrated BP mean flux";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_bp_mean_flux_error phot_bp_mean_flux_error float COMMENT "Error on the integrated BP mean flux";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_bp_mean_flux_over_error phot_bp_mean_flux_over_error float COMMENT "Integrated BP mean flux divided by its error";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_bp_mean_mag phot_bp_mean_mag float COMMENT "Integrated BP mean magnitude";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_rp_n_obs phot_rp_n_obs smallint COMMENT "Number of observations contributing to RP photometry";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_rp_mean_flux phot_rp_mean_flux double COMMENT "Integrated RP mean flux";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_rp_mean_flux_error phot_rp_mean_flux_error float COMMENT "Error on the integrated RP mean flux";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_rp_mean_flux_over_error phot_rp_mean_flux_over_error float COMMENT "Integrated RP mean flux divided by its error";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_rp_mean_mag phot_rp_mean_mag float COMMENT "Integrated RP mean magnitude";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_bp_n_contaminated_transits phot_bp_n_contaminated_transits smallint COMMENT "Number of BP contaminated transits";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_bp_n_blended_transits phot_bp_n_blended_transits smallint COMMENT "Number of BP blended transits";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_rp_n_contaminated_transits phot_rp_n_contaminated_transits smallint COMMENT "Number of RP contaminated transits";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_rp_n_blended_transits phot_rp_n_blended_transits smallint COMMENT "Number of RP blended transits";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_proc_mode phot_proc_mode tinyint COMMENT "Photometry processing mode";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_bp_rp_excess_factor phot_bp_rp_excess_factor float COMMENT "BP/RP excess factor";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE bp_rp bp_rp float COMMENT "BP - RP colour";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE bp_g bp_g float COMMENT "BP - G colour";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE g_rp g_rp float COMMENT "G - RP colour";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dr2_radial_velocity dr2_radial_velocity float COMMENT "Radial velocity from Gaia DR2";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dr2_radial_velocity_error dr2_radial_velocity_error float COMMENT "Radial velocity error from Gaia DR2";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dr2_rv_nb_transits dr2_rv_nb_transits smallint COMMENT "Number of transits used to compute radial velocity in Gaia DR2";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dr2_rv_template_teff dr2_rv_template_teff float COMMENT "Teff of the template used to compute radial velocity in Gaia DR2";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dr2_rv_template_logg dr2_rv_template_logg float COMMENT "logg of the template used to compute radial velocity in Gaia DR2";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dr2_rv_template_fe_h dr2_rv_template_fe_h float COMMENT "Fe/H of the template used to compute radial velocity in Gaia DR2";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE l l double COMMENT "Galactic longitude";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE b b double COMMENT "Galactic latitude";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ecl_lon ecl_lon double COMMENT "Ecliptic longitude";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ecl_lat ecl_lat double COMMENT "Ecliptic latitude";<br />
<br />
ANALYZE TABLE cosmohub.gaia_edr3 COMPUTE STATISTICS FOR COLUMNS;</div>Talladahttps://pwiki.pic.es/index.php?title=Gaia_EDR3&diff=884Gaia EDR32020-12-03T14:39:10Z<p>Tallada: Created page with "== Raw data == Mirrored to cephfs using lftp: mirror -r -v -c -P4 http://cdn.gea.esac.esa.int/Gaia/gedr3/gaia_source/ /cephfs/pic.es/astro/scratch/tallada/gaia_edr3 ==..."</p>
<hr />
<div>== Raw data ==<br />
<br />
Mirrored to cephfs using lftp:<br />
<br />
mirror -r -v -c -P4 http://cdn.gea.esac.esa.int/Gaia/gedr3/gaia_source/ /cephfs/pic.es/astro/scratch/tallada/gaia_edr3<br />
<br />
== External table ==<br />
<br />
CREATE EXTERNAL TABLE tallada.gaia_edr3_csv (<br />
solution_id bigint,<br />
designation string,<br />
source_id bigint,<br />
random_index bigint,<br />
ref_epoch float,<br />
ra double,<br />
ra_error float,<br />
`dec` double,<br />
dec_error float,<br />
parallax double,<br />
parallax_error float,<br />
parallax_over_error float,<br />
pm float,<br />
pmra double,<br />
pmra_error float,<br />
pmdec double,<br />
pmdec_error float,<br />
ra_dec_corr float,<br />
ra_parallax_corr float,<br />
ra_pmra_corr float,<br />
ra_pmdec_corr float,<br />
dec_parallax_corr float,<br />
dec_pmra_corr float,<br />
dec_pmdec_corr float,<br />
parallax_pmra_corr float,<br />
parallax_pmdec_corr float,<br />
pmra_pmdec_corr float,<br />
astrometric_n_obs_al smallint,<br />
astrometric_n_obs_ac smallint,<br />
astrometric_n_good_obs_al smallint,<br />
astrometric_n_bad_obs_al smallint,<br />
astrometric_gof_al float,<br />
astrometric_chi2_al float,<br />
astrometric_excess_noise float,<br />
astrometric_excess_noise_sig float,<br />
astrometric_params_solved tinyint,<br />
astrometric_primary_flag boolean,<br />
nu_eff_used_in_astrometry float,<br />
pseudocolour float,<br />
pseudocolour_error float,<br />
ra_pseudocolour_corr float,<br />
dec_pseudocolour_corr float,<br />
parallax_pseudocolour_corr float,<br />
pmra_pseudocolour_corr float,<br />
pmdec_pseudocolour_corr float,<br />
astrometric_matched_transits smallint,<br />
visibility_periods_used smallint,<br />
astrometric_sigma5d_max float,<br />
matched_transits smallint,<br />
new_matched_transits smallint,<br />
matched_transits_removed smallint,<br />
ipd_gof_harmonic_amplitude float,<br />
ipd_gof_harmonic_phase float,<br />
ipd_frac_multi_peak tinyint,<br />
ipd_frac_odd_win tinyint,<br />
ruwe float,<br />
scan_direction_strength_k1 float,<br />
scan_direction_strength_k2 float,<br />
scan_direction_strength_k3 float,<br />
scan_direction_strength_k4 float,<br />
scan_direction_mean_k1 float,<br />
scan_direction_mean_k2 float,<br />
scan_direction_mean_k3 float,<br />
scan_direction_mean_k4 float,<br />
duplicated_source boolean,<br />
phot_g_n_obs smallint,<br />
phot_g_mean_flux double,<br />
phot_g_mean_flux_error float,<br />
phot_g_mean_flux_over_error float,<br />
phot_g_mean_mag float,<br />
phot_bp_n_obs smallint,<br />
phot_bp_mean_flux double,<br />
phot_bp_mean_flux_error float,<br />
phot_bp_mean_flux_over_error float,<br />
phot_bp_mean_mag float,<br />
phot_rp_n_obs smallint,<br />
phot_rp_mean_flux double,<br />
phot_rp_mean_flux_error float,<br />
phot_rp_mean_flux_over_error float,<br />
phot_rp_mean_mag float,<br />
phot_bp_n_contaminated_transits smallint,<br />
phot_bp_n_blended_transits smallint,<br />
phot_rp_n_contaminated_transits smallint,<br />
phot_rp_n_blended_transits smallint,<br />
phot_proc_mode tinyint,<br />
phot_bp_rp_excess_factor float,<br />
bp_rp float,<br />
bp_g float,<br />
g_rp float,<br />
dr2_radial_velocity float,<br />
dr2_radial_velocity_error float,<br />
dr2_rv_nb_transits smallint,<br />
dr2_rv_template_teff float,<br />
dr2_rv_template_logg float,<br />
dr2_rv_template_fe_h float,<br />
l double,<br />
b double,<br />
ecl_lon double,<br />
ecl_lat double<br />
)<br />
ROW FORMAT DELIMITED<br />
FIELDS TERMINATED BY ','<br />
LOCATION '/user/tallada/data/gaia_edr3'<br />
;</div>Talladahttps://pwiki.pic.es/index.php?title=Catalogs&diff=883Catalogs2020-12-03T14:23:14Z<p>Tallada: /* External */</p>
<hr />
<div>== Euclid productions ==<br />
<br />
==== Mock Galaxy catalogs ====<br />
<br />
More info from Euclid wiki (Francesc):<br />
<br />
https://euclid.roe.ac.uk/projects/sgv/wiki/SC456_#Galaxies<br />
<br />
* [[ Euclid_mock_production_v1.6.20 | v1.6.20 ]] ([https://euclid.roe.ac.uk/issues/9153 #9153])<br />
* [[ Euclid_mock_production_v1.7.17 | v1.7.17 ]] (SC456)<br />
* [[ Euclid_mock_production_v1.8.1 | v1.8.1 ]]<br />
* [[ Euclid_mock_production_v1.8.3 | v1.8.3 ]] (single octant for Helsinki)<br />
* [[ Euclid_mock_production_v1.8.3_c | v1.8.3_c ]] (clustered table without lensing)<br />
* [[ Euclid_mock_production_v1.8.4_s | v1.8.4_s ]] (from 1.8.3 with correct lensing and without buffer)<br />
* [[ Euclid_mock_production_v1.8.5 | v1.8.5 ]] (Kai subset with additional magnitudes for IA)<br />
* [[ Euclid_mock_production_v1.7.17_correct | v1.7.17_correct (v1.7.18)]] (SC456)<br />
* [[ Euclid_mock_production_v1.7.17_deep_correct | v1.7.17_deep_correct]] (SC456)<br />
* [[ Euclid_mock_production_v1.7.17_correct_v2 | v1.7.17_correct_v2 (v1.7.19)]] (SC456)<br />
* [[ Euclid_mock_production_v1.7.17_deep_correct_v2 | v1.7.17_deep_correct_v2]] (SC456)<br />
* [[ Euclid_mock_production_v1.9.2 | v1.9.2]] (SC8)<br />
* [[ Euclid_mock_production_v1.9.3 | v1.9.3]] (SC8)<br />
* [[ Euclid_mock_production_v1.9.4 | v1.9.4]] (SC8)<br />
* [[ Euclid_mock_production_v1.9.5 | v1.9.5]] (SC8)<br />
* [[ Euclid_mock_production_v1.9.6 | v1.9.6]] (SC8)<br />
* [[ Euclid_mock_production_v1.9.7 | v1.9.7]] (SC8)<br />
<br />
==== Zurich Dark Matter Halo catalogs ====<br />
<br />
* [[ v1.0 | v1.0 ]]<br />
<br />
* [[ v1.1 | v1.1 Corrected Velocities ]]<br />
<br />
==== Primeval Universe Catalogs ====<br />
* [[ QSOs | QSOs]]<br />
* [[ High_redshift_galaxies | High-z galaxies]]<br />
<br />
== MICE productions ==<br />
<br />
* [[ MIDES Wide Field v5.0 | MIDES Wide Field v5.0 ]]<br />
* [[ MIDES Wide Field v6.0 | MIDES Wide Field v6.0 ]]<br />
* [[ MICE Redmagic v5.1 ]]<br />
* [[ DESY3 MICE2 source catalog | DESY3 MICE2 source catalog ]]<br />
<br />
== PAUS productions ==<br />
<br />
=== Observations ===<br />
<br />
* [[ PAUdm_forced_aperture_coadd | PAUdm forced aperture coadd ]]<br />
<br />
==== Photoz productions ====<br />
<br />
* [[ photoz_bcnz | bcnz_v29.parquet ]]<br />
<br />
* [[ production_table | paudm_production_table ]]<br />
<br />
* [[ PAUS+COSMOS_photoz | PAUS+COSMOS_photoz ]]<br />
<br />
==== CIGALE catalogs ====<br />
<br />
* [[ CIGALE-W3 | CIGALE W3 ]]<br />
<br />
==== Mock Galaxy catalogs ====<br />
<br />
* [[ Flagship-PAU | Flagship-PAU v1.8.4_v0.1 ]]<br />
<br />
* [[ Flagship-PAU_v02 | Flagship-PAU v1.8.4_v0.2 ]]<br />
<br />
* [[ Flagship-PAU_v03 | Flagship-PAU v1.8.4_v0.3 ]]<br />
<br />
* [[ Durham-LC-PAU lc5.field1.63 | Durham LC lc5.field1.63 ]]<br />
<br />
== External ==<br />
<br />
* [[ Cosmos_2015_DR_2.1 | COSMOS 2015 DR 2.1 ]]<br />
* [[ ZEST_v1.0 | Zest v1.0 ]]<br />
* [[ CANDELS_2018 | CANDELS 2018 ]]<br />
* [[ Legacy Survey DR8 | Legacy Survey DR8 ]]<br />
* [[ Legacy Survey DR9 | Legacy Survey DR9 ]]<br />
* [[ KiDS DR4 ]]<br />
* [[ VIPERS ]]<br />
* [[ The COSMOS ACS Catalog ]]<br />
* [[ GLADE ]]<br />
* [[ Gaia_EDR3 | Gaia EDR3 ]]</div>Talladahttps://pwiki.pic.es/index.php?title=Catalogs&diff=879Catalogs2020-12-01T20:05:04Z<p>Tallada: /* External */</p>
<hr />
<div>== Euclid productions ==<br />
<br />
==== Mock Galaxy catalogs ====<br />
<br />
More info from Euclid wiki (Francesc):<br />
<br />
https://euclid.roe.ac.uk/projects/sgv/wiki/SC456_#Galaxies<br />
<br />
* [[ Euclid_mock_production_v1.6.20 | v1.6.20 ]] ([https://euclid.roe.ac.uk/issues/9153 #9153])<br />
* [[ Euclid_mock_production_v1.7.17 | v1.7.17 ]] (SC456)<br />
* [[ Euclid_mock_production_v1.8.1 | v1.8.1 ]]<br />
* [[ Euclid_mock_production_v1.8.3 | v1.8.3 ]] (single octant for Helsinki)<br />
* [[ Euclid_mock_production_v1.8.3_c | v1.8.3_c ]] (clustered table without lensing)<br />
* [[ Euclid_mock_production_v1.8.4_s | v1.8.4_s ]] (from 1.8.3 with correct lensing and without buffer)<br />
* [[ Euclid_mock_production_v1.8.5 | v1.8.5 ]] (Kai subset with additional magnitudes for IA)<br />
* [[ Euclid_mock_production_v1.7.17_correct | v1.7.17_correct (v1.7.18)]] (SC456)<br />
* [[ Euclid_mock_production_v1.7.17_deep_correct | v1.7.17_deep_correct]] (SC456)<br />
* [[ Euclid_mock_production_v1.7.17_correct_v2 | v1.7.17_correct_v2 (v1.7.19)]] (SC456)<br />
* [[ Euclid_mock_production_v1.7.17_deep_correct_v2 | v1.7.17_deep_correct_v2]] (SC456)<br />
* [[ Euclid_mock_production_v1.9.2 | v1.9.2]] (SC8)<br />
* [[ Euclid_mock_production_v1.9.3 | v1.9.3]] (SC8)<br />
* [[ Euclid_mock_production_v1.9.4 | v1.9.4]] (SC8)<br />
* [[ Euclid_mock_production_v1.9.5 | v1.9.5]] (SC8)<br />
* [[ Euclid_mock_production_v1.9.6 | v1.9.6]] (SC8)<br />
* [[ Euclid_mock_production_v1.9.7 | v1.9.7]] (SC8)<br />
<br />
==== Zurich Dark Matter Halo catalogs ====<br />
<br />
* [[ v1.0 | v1.0 ]]<br />
<br />
* [[ v1.1 | v1.1 Corrected Velocities ]]<br />
<br />
==== Primeval Universe Catalogs ====<br />
* [[ QSOs | QSOs]]<br />
* [[ High_redshift_galaxies | High-z galaxies]]<br />
<br />
== MICE productions ==<br />
<br />
* [[ MIDES Wide Field v5.0 | MIDES Wide Field v5.0 ]]<br />
* [[ MIDES Wide Field v6.0 | MIDES Wide Field v6.0 ]]<br />
* [[ MICE Redmagic v5.1 ]]<br />
* [[ DESY3 MICE2 source catalog | DESY3 MICE2 source catalog ]]<br />
<br />
== PAUS productions ==<br />
<br />
=== Observations ===<br />
<br />
* [[ PAUdm_forced_aperture_coadd | PAUdm forced aperture coadd ]]<br />
<br />
==== Photoz productions ====<br />
<br />
* [[ photoz_bcnz | bcnz_v29.parquet ]]<br />
<br />
* [[ production_table | paudm_production_table ]]<br />
<br />
* [[ PAUS+COSMOS_photoz | PAUS+COSMOS_photoz ]]<br />
<br />
=== Mock Galaxy catalogs ===<br />
<br />
* [[ Flagship-PAU | Flagship-PAU v1.8.4_v0.1 ]]<br />
<br />
* [[ Flagship-PAU_v02 | Flagship-PAU v1.8.4_v0.2 ]]<br />
<br />
* [[ Flagship-PAU_v03 | Flagship-PAU v1.8.4_v0.3 ]]<br />
<br />
* [[ Durham-LC-PAU lc5.field1.63 | Durham LC lc5.field1.63 ]]<br />
<br />
== External ==<br />
<br />
* [[ Cosmos_2015_DR_2.1 | COSMOS 2015 DR 2.1 ]]<br />
* [[ ZEST_v1.0 | Zest v1.0 ]]<br />
* [[ CANDELS_2018 | CANDELS 2018 ]]<br />
* [[ Legacy Survey DR8 | Legacy Survey DR8 ]]<br />
* [[ Legacy Survey DR9 | Legacy Survey DR9 ]]<br />
* [[ KiDS DR4 ]]<br />
* [[ VIPERS ]]<br />
* [[ The COSMOS ACS Catalog ]]<br />
* [[ GLADE ]]</div>Talladahttps://pwiki.pic.es/index.php?title=Legacy_Survey_DR9&diff=878Legacy Survey DR92020-11-30T21:04:59Z<p>Tallada: </p>
<hr />
<div>== Original DR9 ==<br />
=== Raw data ===<br />
<br />
Downloaded from NERSC<br />
/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m<br />
<br />
Directories:<br />
south/tractor<br />
north/tractor<br />
<br />
Downloaded using bbcp from services01.euclid into CephFS scratch:<br />
bbcp -s 16 -a -A -v -z -r -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" \<br />
"jcarrete@dtn01.nersc.gov:/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m/north/tractor" \<br />
/cephfs/pic.es/astro/scratch/tallada/dr9m/north/tractor<br />
bbcp -s 16 -a -A -v -z -r -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" \<br />
"jcarrete@dtn01.nersc.gov:/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m/south/tractor" \<br />
/cephfs/pic.es/astro/scratch/tallada/dr9m/south/tractor<br />
<br />
Checksums provided through sha256sum files, and checked using command:<br />
find * -type d | xargs -t -L1 -P24 sh -c 'cd $1; sha256sum --quiet -c *.sha256sum' cksum<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to convert fits files to parquet:<br />
<br />
path_base = '/cephfs/pic.es/astro/scratch/tallada/dr9m'<br />
hdfs_base = '/user/tallada/data/legacysurvey_dr9'<br />
<br />
hs = ['h=north', 'h=south']<br />
rs = ['r={0:03d}'.format(x) for x in range(360)]<br />
<br />
for h in hs:<br />
for r in rs:<br />
p = os.path.join(path_base, h, r)<br />
if not os.path.exists(p):<br />
continue<br />
<br />
size = sum(os.path.getsize(os.path.join(p,f)) for f in os.listdir(p) if os.path.isfile(os.path.join(p,f)))<br />
<br />
print h, r, size <br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + p<br />
)<br />
<br />
df = df.repartition(int(math.ceil(size / (256*1024*1024.0))))<br />
<br />
df.write.parquet(os.path.join(hdfs_base, h, r))<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE `tallada`.`legacysurvey_dr9`(<br />
`release` smallint, <br />
`brickid` int, <br />
`brickname` string, <br />
`objid` int, <br />
`brick_primary` boolean, <br />
`maskbits` smallint, <br />
`fitbits` smallint, <br />
`type` string, <br />
`ra` double, <br />
`dec` double, <br />
`ra_ivar` float, <br />
`dec_ivar` float, <br />
`bx` float, <br />
`by` float, <br />
`dchisq` array<float>, <br />
`ebv` float, <br />
`mjd_min` double, <br />
`mjd_max` double, <br />
`ref_cat` string, <br />
`ref_id` bigint, <br />
`pmra` float, <br />
`pmdec` float, <br />
`parallax` float, <br />
`pmra_ivar` float, <br />
`pmdec_ivar` float, <br />
`parallax_ivar` float, <br />
`ref_epoch` float, <br />
`gaia_phot_g_mean_mag` float, <br />
`gaia_phot_g_mean_flux_over_error` float, <br />
`gaia_phot_g_n_obs` smallint, <br />
`gaia_phot_bp_mean_mag` float, <br />
`gaia_phot_bp_mean_flux_over_error` float, <br />
`gaia_phot_bp_n_obs` smallint, <br />
`gaia_phot_rp_mean_mag` float, <br />
`gaia_phot_rp_mean_flux_over_error` float, <br />
`gaia_phot_rp_n_obs` smallint, <br />
`gaia_phot_variable_flag` boolean, <br />
`gaia_astrometric_excess_noise` float, <br />
`gaia_astrometric_excess_noise_sig` float, <br />
`gaia_astrometric_n_obs_al` smallint, <br />
`gaia_astrometric_n_good_obs_al` smallint, <br />
`gaia_astrometric_weight_al` float, <br />
`gaia_duplicated_source` boolean, <br />
`gaia_a_g_val` float, <br />
`gaia_e_bp_min_rp_val` float, <br />
`gaia_phot_bp_rp_excess_factor` float, <br />
`gaia_astrometric_sigma5d_max` float, <br />
`gaia_astrometric_params_solved` tinyint, <br />
`flux_g` float, <br />
`flux_r` float, <br />
`flux_z` float, <br />
`flux_w1` float, <br />
`flux_w2` float, <br />
`flux_w3` float, <br />
`flux_w4` float, <br />
`flux_ivar_g` float, <br />
`flux_ivar_r` float, <br />
`flux_ivar_z` float, <br />
`flux_ivar_w1` float, <br />
`flux_ivar_w2` float, <br />
`flux_ivar_w3` float, <br />
`flux_ivar_w4` float, <br />
`fiberflux_g` float, <br />
`fiberflux_r` float, <br />
`fiberflux_z` float, <br />
`fibertotflux_g` float, <br />
`fibertotflux_r` float, <br />
`fibertotflux_z` float, <br />
`apflux_g` array<float>, <br />
`apflux_r` array<float>, <br />
`apflux_z` array<float>, <br />
`apflux_resid_g` array<float>, <br />
`apflux_resid_r` array<float>, <br />
`apflux_resid_z` array<float>, <br />
`apflux_blobresid_g` array<float>, <br />
`apflux_blobresid_r` array<float>, <br />
`apflux_blobresid_z` array<float>, <br />
`apflux_ivar_g` array<float>, <br />
`apflux_ivar_r` array<float>, <br />
`apflux_ivar_z` array<float>, <br />
`apflux_masked_g` array<float>, <br />
`apflux_masked_r` array<float>, <br />
`apflux_masked_z` array<float>, <br />
`apflux_w1` array<float>, <br />
`apflux_w2` array<float>, <br />
`apflux_w3` array<float>, <br />
`apflux_w4` array<float>, <br />
`apflux_resid_w1` array<float>, <br />
`apflux_resid_w2` array<float>, <br />
`apflux_resid_w3` array<float>, <br />
`apflux_resid_w4` array<float>, <br />
`apflux_ivar_w1` array<float>, <br />
`apflux_ivar_w2` array<float>, <br />
`apflux_ivar_w3` array<float>, <br />
`apflux_ivar_w4` array<float>, <br />
`mw_transmission_g` float, <br />
`mw_transmission_r` float, <br />
`mw_transmission_z` float, <br />
`mw_transmission_w1` float, <br />
`mw_transmission_w2` float, <br />
`mw_transmission_w3` float, <br />
`mw_transmission_w4` float, <br />
`nobs_g` smallint, <br />
`nobs_r` smallint, <br />
`nobs_z` smallint, <br />
`nobs_w1` smallint, <br />
`nobs_w2` smallint, <br />
`nobs_w3` smallint, <br />
`nobs_w4` smallint, <br />
`rchisq_g` float, <br />
`rchisq_r` float, <br />
`rchisq_z` float, <br />
`rchisq_w1` float, <br />
`rchisq_w2` float, <br />
`rchisq_w3` float, <br />
`rchisq_w4` float, <br />
`fracflux_g` float, <br />
`fracflux_r` float, <br />
`fracflux_z` float, <br />
`fracflux_w1` float, <br />
`fracflux_w2` float, <br />
`fracflux_w3` float, <br />
`fracflux_w4` float, <br />
`fracmasked_g` float, <br />
`fracmasked_r` float, <br />
`fracmasked_z` float, <br />
`fracin_g` float, <br />
`fracin_r` float, <br />
`fracin_z` float, <br />
`anymask_g` smallint, <br />
`anymask_r` smallint, <br />
`anymask_z` smallint, <br />
`allmask_g` smallint, <br />
`allmask_r` smallint, <br />
`allmask_z` smallint, <br />
`wisemask_w1` tinyint, <br />
`wisemask_w2` tinyint, <br />
`psfsize_g` float, <br />
`psfsize_r` float, <br />
`psfsize_z` float, <br />
`psfdepth_g` float, <br />
`psfdepth_r` float, <br />
`psfdepth_z` float, <br />
`galdepth_g` float, <br />
`galdepth_r` float, <br />
`galdepth_z` float, <br />
`nea_g` float, <br />
`nea_r` float, <br />
`nea_z` float, <br />
`blob_nea_g` float, <br />
`blob_nea_r` float, <br />
`blob_nea_z` float, <br />
`psfdepth_w1` float, <br />
`psfdepth_w2` float, <br />
`psfdepth_w3` float, <br />
`psfdepth_w4` float, <br />
`wise_coadd_id` string, <br />
`wise_x` float, <br />
`wise_y` float, <br />
`lc_flux_w1` array<float>, <br />
`lc_flux_w2` array<float>, <br />
`lc_flux_ivar_w1` array<float>, <br />
`lc_flux_ivar_w2` array<float>, <br />
`lc_nobs_w1` array<smallint>, <br />
`lc_nobs_w2` array<smallint>, <br />
`lc_fracflux_w1` array<float>, <br />
`lc_fracflux_w2` array<float>, <br />
`lc_rchisq_w1` array<float>, <br />
`lc_rchisq_w2` array<float>, <br />
`lc_mjd_w1` array<double>, <br />
`lc_mjd_w2` array<double>, <br />
`lc_epoch_index_w1` array<smallint>, <br />
`lc_epoch_index_w2` array<smallint>, <br />
`sersic` float, <br />
`sersic_ivar` float, <br />
`shape_r` float, <br />
`shape_r_ivar` float, <br />
`shape_e1` float, <br />
`shape_e1_ivar` float, <br />
`shape_e2` float, <br />
`shape_e2_ivar` float)<br />
PARTITIONED BY (<br />
h STRING,<br />
r STRING<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey_dr9'<br />
;<br />
<br />
=== Internal Table ===<br />
<br />
CREATE TABLE `cosmohub`.`legacy_survey_dr9`(<br />
`release` smallint,<br />
`brickid` int,<br />
`brickname` string,<br />
`objid` int,<br />
`brick_primary` boolean,<br />
`maskbits` smallint,<br />
`fitbits` smallint,<br />
`type` string,<br />
`ra` double,<br />
`dec` double,<br />
`ra_ivar` float,<br />
`dec_ivar` float,<br />
`bx` float,<br />
`by` float,<br />
`dchisq` array<float>,<br />
`ebv` float,<br />
`mjd_min` double,<br />
`mjd_max` double,<br />
`ref_cat` string,<br />
`ref_id` bigint,<br />
`pmra` float,<br />
`pmdec` float,<br />
`parallax` float,<br />
`pmra_ivar` float,<br />
`pmdec_ivar` float,<br />
`parallax_ivar` float,<br />
`ref_epoch` float,<br />
`gaia_phot_g_mean_mag` float,<br />
`gaia_phot_g_mean_flux_over_error` float,<br />
`gaia_phot_g_n_obs` smallint,<br />
`gaia_phot_bp_mean_mag` float,<br />
`gaia_phot_bp_mean_flux_over_error` float,<br />
`gaia_phot_bp_n_obs` smallint,<br />
`gaia_phot_rp_mean_mag` float,<br />
`gaia_phot_rp_mean_flux_over_error` float,<br />
`gaia_phot_rp_n_obs` smallint,<br />
`gaia_phot_variable_flag` boolean,<br />
`gaia_astrometric_excess_noise` float,<br />
`gaia_astrometric_excess_noise_sig` float,<br />
`gaia_astrometric_n_obs_al` smallint,<br />
`gaia_astrometric_n_good_obs_al` smallint,<br />
`gaia_astrometric_weight_al` float,<br />
`gaia_duplicated_source` boolean,<br />
`gaia_a_g_val` float,<br />
`gaia_e_bp_min_rp_val` float,<br />
`gaia_phot_bp_rp_excess_factor` float,<br />
`gaia_astrometric_sigma5d_max` float,<br />
`gaia_astrometric_params_solved` tinyint,<br />
`flux_g` float,<br />
`flux_r` float,<br />
`flux_z` float,<br />
`flux_w1` float,<br />
`flux_w2` float,<br />
`flux_w3` float,<br />
`flux_w4` float,<br />
`flux_ivar_g` float,<br />
`flux_ivar_r` float,<br />
`flux_ivar_z` float,<br />
`flux_ivar_w1` float,<br />
`flux_ivar_w2` float,<br />
`flux_ivar_w3` float,<br />
`flux_ivar_w4` float,<br />
`fiberflux_g` float,<br />
`fiberflux_r` float,<br />
`fiberflux_z` float,<br />
`fibertotflux_g` float,<br />
`fibertotflux_r` float,<br />
`fibertotflux_z` float,<br />
`apflux_g` array<float>,<br />
`apflux_r` array<float>,<br />
`apflux_z` array<float>,<br />
`apflux_resid_g` array<float>,<br />
`apflux_resid_r` array<float>,<br />
`apflux_resid_z` array<float>,<br />
`apflux_blobresid_g` array<float>,<br />
`apflux_blobresid_r` array<float>,<br />
`apflux_blobresid_z` array<float>,<br />
`apflux_ivar_g` array<float>,<br />
`apflux_ivar_r` array<float>,<br />
`apflux_ivar_z` array<float>,<br />
`apflux_masked_g` array<float>,<br />
`apflux_masked_r` array<float>,<br />
`apflux_masked_z` array<float>,<br />
`apflux_w1` array<float>,<br />
`apflux_w2` array<float>,<br />
`apflux_w3` array<float>,<br />
`apflux_w4` array<float>,<br />
`apflux_resid_w1` array<float>,<br />
`apflux_resid_w2` array<float>,<br />
`apflux_resid_w3` array<float>,<br />
`apflux_resid_w4` array<float>,<br />
`apflux_ivar_w1` array<float>,<br />
`apflux_ivar_w2` array<float>,<br />
`apflux_ivar_w3` array<float>,<br />
`apflux_ivar_w4` array<float>,<br />
`mw_transmission_g` float,<br />
`mw_transmission_r` float,<br />
`mw_transmission_z` float,<br />
`mw_transmission_w1` float,<br />
`mw_transmission_w2` float,<br />
`mw_transmission_w3` float,<br />
`mw_transmission_w4` float,<br />
`nobs_g` smallint,<br />
`nobs_r` smallint,<br />
`nobs_z` smallint,<br />
`nobs_w1` smallint,<br />
`nobs_w2` smallint,<br />
`nobs_w3` smallint,<br />
`nobs_w4` smallint,<br />
`rchisq_g` float,<br />
`rchisq_r` float,<br />
`rchisq_z` float,<br />
`rchisq_w1` float,<br />
`rchisq_w2` float,<br />
`rchisq_w3` float,<br />
`rchisq_w4` float,<br />
`fracflux_g` float,<br />
`fracflux_r` float,<br />
`fracflux_z` float,<br />
`fracflux_w1` float,<br />
`fracflux_w2` float,<br />
`fracflux_w3` float,<br />
`fracflux_w4` float,<br />
`fracmasked_g` float,<br />
`fracmasked_r` float,<br />
`fracmasked_z` float,<br />
`fracin_g` float,<br />
`fracin_r` float,<br />
`fracin_z` float,<br />
`anymask_g` smallint,<br />
`anymask_r` smallint,<br />
`anymask_z` smallint,<br />
`allmask_g` smallint,<br />
`allmask_r` smallint,<br />
`allmask_z` smallint,<br />
`wisemask_w1` tinyint,<br />
`wisemask_w2` tinyint,<br />
`psfsize_g` float,<br />
`psfsize_r` float,<br />
`psfsize_z` float,<br />
`psfdepth_g` float,<br />
`psfdepth_r` float,<br />
`psfdepth_z` float,<br />
`galdepth_g` float,<br />
`galdepth_r` float,<br />
`galdepth_z` float,<br />
`nea_g` float,<br />
`nea_r` float,<br />
`nea_z` float,<br />
`blob_nea_g` float,<br />
`blob_nea_r` float,<br />
`blob_nea_z` float,<br />
`psfdepth_w1` float,<br />
`psfdepth_w2` float,<br />
`psfdepth_w3` float,<br />
`psfdepth_w4` float,<br />
`wise_coadd_id` string,<br />
`wise_x` float,<br />
`wise_y` float,<br />
`lc_flux_w1` array<float>,<br />
`lc_flux_w2` array<float>,<br />
`lc_flux_ivar_w1` array<float>,<br />
`lc_flux_ivar_w2` array<float>,<br />
`lc_nobs_w1` array<smallint>,<br />
`lc_nobs_w2` array<smallint>,<br />
`lc_fracflux_w1` array<float>,<br />
`lc_fracflux_w2` array<float>,<br />
`lc_rchisq_w1` array<float>,<br />
`lc_rchisq_w2` array<float>,<br />
`lc_mjd_w1` array<double>,<br />
`lc_mjd_w2` array<double>,<br />
`lc_epoch_index_w1` array<smallint>,<br />
`lc_epoch_index_w2` array<smallint>,<br />
`sersic` float,<br />
`sersic_ivar` float,<br />
`shape_r` float,<br />
`shape_r_ivar` float,<br />
`shape_e1` float,<br />
`shape_e1_ivar` float,<br />
`shape_e2` float,<br />
`shape_e2_ivar` float)<br />
CLUSTERED BY (<br />
release, brickid, objid<br />
)<br />
SORTED BY (<br />
release, brickid, objid<br />
)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
<br />
INSERT INTO TABLE cosmohub.legacy_survey_dr9<br />
SELECT release, brickid, brickname, objid, brick_primary, maskbits, fitbits, type, ra, `dec`, ra_ivar, dec_ivar,<br />
bx, `by`, dchisq, ebv, mjd_min, mjd_max, ref_cat, ref_id, pmra, pmdec, parallax, pmra_ivar, pmdec_ivar,<br />
parallax_ivar, ref_epoch, gaia_phot_g_mean_mag, gaia_phot_g_mean_flux_over_error, gaia_phot_g_n_obs,<br />
gaia_phot_bp_mean_mag, gaia_phot_bp_mean_flux_over_error, gaia_phot_bp_n_obs, gaia_phot_rp_mean_mag,<br />
gaia_phot_rp_mean_flux_over_error, gaia_phot_rp_n_obs, gaia_phot_variable_flag, gaia_astrometric_excess_noise,<br />
gaia_astrometric_excess_noise_sig, gaia_astrometric_n_obs_al, gaia_astrometric_n_good_obs_al,<br />
gaia_astrometric_weight_al, gaia_duplicated_source, gaia_a_g_val, gaia_e_bp_min_rp_val,<br />
gaia_phot_bp_rp_excess_factor, gaia_astrometric_sigma5d_max, gaia_astrometric_params_solved, flux_g, flux_r,<br />
flux_z, flux_w1, flux_w2, flux_w3, flux_w4, flux_ivar_g, flux_ivar_r, flux_ivar_z, flux_ivar_w1, flux_ivar_w2,<br />
flux_ivar_w3, flux_ivar_w4, fiberflux_g, fiberflux_r, fiberflux_z, fibertotflux_g, fibertotflux_r, fibertotflux_z,<br />
apflux_g, apflux_r, apflux_z, apflux_resid_g, apflux_resid_r, apflux_resid_z, apflux_blobresid_g,<br />
apflux_blobresid_r, apflux_blobresid_z, apflux_ivar_g, apflux_ivar_r, apflux_ivar_z, apflux_masked_g,<br />
apflux_masked_r, apflux_masked_z, apflux_w1, apflux_w2, apflux_w3, apflux_w4, apflux_resid_w1, apflux_resid_w2,<br />
apflux_resid_w3, apflux_resid_w4, apflux_ivar_w1, apflux_ivar_w2, apflux_ivar_w3, apflux_ivar_w4,<br />
mw_transmission_g, mw_transmission_r, mw_transmission_z, mw_transmission_w1, mw_transmission_w2,<br />
mw_transmission_w3, mw_transmission_w4, nobs_g, nobs_r, nobs_z, nobs_w1, nobs_w2, nobs_w3, nobs_w4, rchisq_g,<br />
rchisq_r, rchisq_z, rchisq_w1, rchisq_w2, rchisq_w3, rchisq_w4, fracflux_g, fracflux_r, fracflux_z, fracflux_w1,<br />
fracflux_w2, fracflux_w3, fracflux_w4, fracmasked_g, fracmasked_r, fracmasked_z, fracin_g, fracin_r, fracin_z,<br />
anymask_g, anymask_r, anymask_z, allmask_g, allmask_r, allmask_z, wisemask_w1, wisemask_w2, psfsize_g, psfsize_r,<br />
psfsize_z, psfdepth_g, psfdepth_r, psfdepth_z, galdepth_g, galdepth_r, galdepth_z, nea_g, nea_r, nea_z,<br />
blob_nea_g, blob_nea_r, blob_nea_z, psfdepth_w1, psfdepth_w2, psfdepth_w3, psfdepth_w4, wise_coadd_id, wise_x,<br />
wise_y, lc_flux_w1, lc_flux_w2, lc_flux_ivar_w1, lc_flux_ivar_w2, lc_nobs_w1, lc_nobs_w2, lc_fracflux_w1,<br />
lc_fracflux_w2, lc_rchisq_w1, lc_rchisq_w2, lc_mjd_w1, lc_mjd_w2, lc_epoch_index_w1, lc_epoch_index_w2, sersic,<br />
sersic_ivar, shape_r, shape_r_ivar, shape_e1, shape_e1_ivar, shape_e2, shape_e2_ivar<br />
FROM cosmohub.legacy_survey_dr9_p<br />
;<br />
<br />
=== Comments ===<br />
ALTER TABLE legacy_survey_dr9 CHANGE release release smallint COMMENT "Unique integer denoting the camera and filter set used" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE brickid brickid int COMMENT "Brick ID [1,662174]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE brickname brickname string COMMENT "Name of brick, encoding the brick sky position, eg \u00221126p222\u0022 near RA=112.6, Dec=+22.2" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE objid objid int COMMENT "Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE brick_primary brick_primary boolean COMMENT "True if the object is within the brick boundary" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE maskbits maskbits smallint COMMENT "Bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits* maps, as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fitbits fitbits smallint COMMENT "Bitwise mask detailing pecularities of how an object was fit, as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE type type string COMMENT "Morphological model: \u0022PSF\u0022=stellar, \u0022REX\u0022=\u0022round exponential galaxy\u0022, \u0022DEV\u0022=deVauc, \u0022EXP\u0022=exponential, \u0022SER\u0022=Sersic, \u0022DUP\u0022=Gaia source fit by different model." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ra ra double COMMENT "Right ascension at equinox J2000" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE `dec` `dec` double COMMENT "Declination at equinox J2000" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ra_ivar ra_ivar float COMMENT "Inverse variance of RA (no cosine term!), excluding astrometric calibration errors" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE dec_ivar dec_ivar float COMMENT "Inverse variance of DEC, excluding astrometric calibration errors" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE bx bx float COMMENT "X position (0-indexed) of coordinates in the brick image stack (i.e. in the e.g. legacysurvey-<brick>-image-g.fits.fz coadd file)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE `by` `by` float COMMENT "Y position (0-indexed) of coordinates in brick image stack" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE dchisq dchisq array<float> COMMENT "Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, SER. The difference is versus no source." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ebv ebv float COMMENT "Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mjd_min mjd_min double COMMENT "Minimum Modified Julian Date of observations used to construct the model of this object" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mjd_max mjd_max double COMMENT "Maximum Modified Julian Date of observations used to construct the model of this object" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ref_cat ref_cat string COMMENT "Reference catalog source for this star: \u0022T2\u0022 for Tycho-2, \u0022G2\u0022 for Gaia DR2, \u0022L3\u0022 for the SGA, empty otherwise" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ref_id ref_id bigint COMMENT "Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b \u0022sourceid\u0022 for Gaia DR2 and SGA" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE pmra pmra float COMMENT "Reference catalog proper motion in the RA direction" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE pmdec pmdec float COMMENT "Reference catalog proper motion in the Dec direction" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE parallax parallax float COMMENT "Reference catalog parallax" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE pmra_ivar pmra_ivar float COMMENT "Reference catalog inverse-variance on pmra" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE pmdec_ivar pmdec_ivar float COMMENT "Reference catalog inverse-variance on pmdec" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE parallax_ivar parallax_ivar float COMMENT "Reference catalog inverse-variance on parallax" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ref_epoch ref_epoch float COMMENT "Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_g_mean_mag gaia_phot_g_mean_mag float COMMENT "Gaia G band mag" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_g_mean_flux_over_error gaia_phot_g_mean_flux_over_error float COMMENT "Gaia G band signal-to-noise" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_g_n_obs gaia_phot_g_n_obs smallint COMMENT "Gaia G band number of observations" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_bp_mean_mag gaia_phot_bp_mean_mag float COMMENT "Gaia BP mag" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_bp_mean_flux_over_error gaia_phot_bp_mean_flux_over_error float COMMENT "Gaia BP signal-to-noise" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_bp_n_obs gaia_phot_bp_n_obs smallint COMMENT "Gaia BP number of observations" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_rp_mean_mag gaia_phot_rp_mean_mag float COMMENT "Gaia RP mag" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_rp_mean_flux_over_error gaia_phot_rp_mean_flux_over_error float COMMENT "Gaia RP signal-to-noise" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_rp_n_obs gaia_phot_rp_n_obs smallint COMMENT "Gaia RP number of observations" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_variable_flag gaia_phot_variable_flag boolean COMMENT "Gaia photometric variable flag" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_excess_noise gaia_astrometric_excess_noise float COMMENT "Gaia astrometric excess noise" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_excess_noise_sig gaia_astrometric_excess_noise_sig float COMMENT "Gaia astrometric excess noise uncertainty" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_n_obs_al gaia_astrometric_n_obs_al smallint COMMENT "Gaia number of astrometric observations along scan direction" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_n_good_obs_al gaia_astrometric_n_good_obs_al smallint COMMENT "Gaia number of good astrometric observations along scan direction" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_weight_al gaia_astrometric_weight_al float COMMENT "Gaia astrometric weight along scan direction" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_duplicated_source gaia_duplicated_source boolean COMMENT "Gaia duplicated source flag" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_a_g_val gaia_a_g_val float COMMENT "Gaia line-of-sight extinction in the G band" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_e_bp_min_rp_val gaia_e_bp_min_rp_val float COMMENT "Gaia line-of-sight reddening E(BP-RP)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_bp_rp_excess_factor gaia_phot_bp_rp_excess_factor float COMMENT "Gaia BP/RP excess factor" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_sigma5d_max gaia_astrometric_sigma5d_max float COMMENT "Gaia longest semi-major axis of the 5-d error ellipsoid" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_params_solved gaia_astrometric_params_solved tinyint COMMENT "which astrometric parameters were estimated for a Gaia source" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_g flux_g float COMMENT "model flux in gg" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_r flux_r float COMMENT "model flux in rr" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_z flux_z float COMMENT "model flux in zz" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_w1 flux_w1 float COMMENT "WISE model flux in W1W1 (AB system)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_w2 flux_w2 float COMMENT "WISE model flux in W2W2 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_w3 flux_w3 float COMMENT "WISE model flux in W3W3 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_w4 flux_w4 float COMMENT "WISE model flux in W4W4 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_g flux_ivar_g float COMMENT "Inverse variance of flux_g" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_r flux_ivar_r float COMMENT "Inverse variance of flux_r" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_z flux_ivar_z float COMMENT "Inverse variance of flux_z" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_w1 flux_ivar_w1 float COMMENT "Inverse variance of flux_w1 (AB system)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_w2 flux_ivar_w2 float COMMENT "Inverse variance of flux_w2 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_w3 flux_ivar_w3 float COMMENT "Inverse variance of flux_w3 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_w4 flux_ivar_w4 float COMMENT "Inverse variance of flux_w4 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fiberflux_g fiberflux_g float COMMENT "Predicted gg-band flux within a fiber of diameter 1.5 arcsec from this object in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fiberflux_r fiberflux_r float COMMENT "Predicted rr-band flux within a fiber of diameter 1.5 arcsec from this object in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fiberflux_z fiberflux_z float COMMENT "Predicted zz-band flux within a fiber of diameter 1.5 arcsec from this object in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fibertotflux_g fibertotflux_g float COMMENT "Predicted gg-band flux within a fiber of diameter 1.5 arcsec from all sources at this location in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fibertotflux_r fibertotflux_r float COMMENT "Predicted rr-band flux within a fiber of diameter 1.5 arcsec from all sources at this location in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fibertotflux_z fibertotflux_z float COMMENT "Predicted zz-band flux within a fiber of diameter 1.5 arcsec from all sources at this location in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_g apflux_g array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg, masked by invvar=0invvar=0 (inverse variance of zero [1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_r apflux_r array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_z apflux_z array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_g apflux_resid_g array<float> COMMENT "Aperture fluxes on the co-added residual images in gg, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_r apflux_resid_r array<float> COMMENT "Aperture fluxes on the co-added residual images in rr, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_z apflux_resid_z array<float> COMMENT "Aperture fluxes on the co-added residual images in zz, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_blobresid_g apflux_blobresid_g array<float> COMMENT "Aperture fluxes on image−blobmodelimage−blobmodel residual maps in gg [2], masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_blobresid_r apflux_blobresid_r array<float> COMMENT "Aperture fluxes on image−blobmodelimage−blobmodel residual maps in rr, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_blobresid_z apflux_blobresid_z array<float> COMMENT "Aperture fluxes on image−blobmodelimage−blobmodel residual maps in zz, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_g apflux_ivar_g array<float> COMMENT "Inverse variance of apflux_resid_g, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_r apflux_ivar_r array<float> COMMENT "Inverse variance of apflux_resid_r, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_z apflux_ivar_z array<float> COMMENT "Inverse variance of apflux_resid_z, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_masked_g apflux_masked_g array<float> COMMENT "Fraction of pixels masked in gg-band aperture flux measurements\u003b 1 means fully masked (ie, fully ignored\u003b contributing zero to the measurement)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_masked_r apflux_masked_r array<float> COMMENT "Fraction of pixels masked in rr-band aperture flux measurements\u003b 1 means fully masked (ie, fully ignored\u003b contributing zero to the measurement)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_masked_z apflux_masked_z array<float> COMMENT "Fraction of pixels masked in zz-band aperture flux measurements\u003b 1 means fully masked (ie, fully ignored\u003b contributing zero to the measurement)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_w1 apflux_w1 array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [3, 5, 7, 9, 11] [3] arcsec in W1W1, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_w2 apflux_w2 array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [3, 5, 7, 9, 11] arcsec in W2W2, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_w3 apflux_w3 array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [3, 5, 7, 9, 11] arcsec in W3W3, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_w4 apflux_w4 array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [3, 5, 7, 9, 11] arcsec in W4W4, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_w1 apflux_resid_w1 array<float> COMMENT "Aperture fluxes on the co-added residual images in W1W1, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_w2 apflux_resid_w2 array<float> COMMENT "Aperture fluxes on the co-added residual images in W2W2, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_w3 apflux_resid_w3 array<float> COMMENT "Aperture fluxes on the co-added residual images in W3W3, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_w4 apflux_resid_w4 array<float> COMMENT "Aperture fluxes on the co-added residual images in W4W4, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_w1 apflux_ivar_w1 array<float> COMMENT "Inverse variance of apflux_resid_w1, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_w2 apflux_ivar_w2 array<float> COMMENT "Inverse variance of apflux_resid_w2, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_w3 apflux_ivar_w3 array<float> COMMENT "Inverse variance of apflux_resid_w3, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_w4 apflux_ivar_w4 array<float> COMMENT "Inverse variance of apflux_resid_w4, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_g mw_transmission_g float COMMENT "Galactic transmission in gg filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_r mw_transmission_r float COMMENT "Galactic transmission in rr filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_z mw_transmission_z float COMMENT "Galactic transmission in zz filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_w1 mw_transmission_w1 float COMMENT "Galactic transmission in W1W1 filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_w2 mw_transmission_w2 float COMMENT "Galactic transmission in W2W2 filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_w3 mw_transmission_w3 float COMMENT "Galactic transmission in W3W3 filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_w4 mw_transmission_w4 float COMMENT "Galactic transmission in W4W4 filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_g nobs_g smallint COMMENT "Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_r nobs_r smallint COMMENT "Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_z nobs_z smallint COMMENT "Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_w1 nobs_w1 smallint COMMENT "Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_w2 nobs_w2 smallint COMMENT "Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_w3 nobs_w3 smallint COMMENT "Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_w4 nobs_w4 smallint COMMENT "Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_g rchisq_g float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in gg" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_r rchisq_r float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in rr" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_z rchisq_z float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in zz" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_w1 rchisq_w1 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W1W1" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_w2 rchisq_w2 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W2W2" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_w3 rchisq_w3 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W3W3" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_w4 rchisq_w4 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W4W4" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_g fracflux_g float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in gg (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_r fracflux_r float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in rr (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_z fracflux_z float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in zz (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_w1 fracflux_w1 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1 (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_w2 fracflux_w2 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2 (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_w3 fracflux_w3 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3 (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_w4 fracflux_w4 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4 (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracmasked_g fracmasked_g float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracmasked_r fracmasked_r float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracmasked_z fracmasked_z float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracin_g fracin_g float COMMENT "Fraction of a source's flux within the blob in gg, near unity for real sources" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracin_r fracin_r float COMMENT "Fraction of a source's flux within the blob in rr, near unity for real sources" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracin_z fracin_z float COMMENT "Fraction of a source's flux within the blob in zz, near unity for real sources" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE anymask_g anymask_g smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE anymask_r anymask_r smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE anymask_z anymask_z smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE allmask_g allmask_g smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE allmask_r allmask_r smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE allmask_z allmask_z smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE wisemask_w1 wisemask_w1 tinyint COMMENT "W1 bitmask as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE wisemask_w2 wisemask_w2 tinyint COMMENT "W2 bitmask as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfsize_g psfsize_g float COMMENT "Weighted average PSF FWHM in the gg band" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfsize_r psfsize_r float COMMENT "Weighted average PSF FWHM in the rr band" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfsize_z psfsize_z float COMMENT "Weighted average PSF FWHM in the zz band" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_g psfdepth_g float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_r psfdepth_r float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_z psfdepth_z float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE galdepth_g galdepth_g float COMMENT "As for psfdepth_g but for a galaxy (0.45\u0022 exp, round) detection sensitivity" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE galdepth_r galdepth_r float COMMENT "As for psfdepth_r but for a galaxy (0.45\u0022 exp, round) detection sensitivity" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE galdepth_z galdepth_z float COMMENT "As for psfdepth_z but for a galaxy (0.45\u0022 exp, round) detection sensitivity" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nea_g nea_g float COMMENT "Noise equivalent area in gg." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nea_r nea_r float COMMENT "Noise equivalent area in rr." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nea_z nea_z float COMMENT "Noise equivalent area in zz." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE blob_nea_g blob_nea_g float COMMENT "Blob-masked noise equivalent area in gg." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE blob_nea_r blob_nea_r float COMMENT "Blob-masked noise equivalent area in rr." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE blob_nea_z blob_nea_z float COMMENT "Blob-masked noise equivalent area in zz." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_w1 psfdepth_w1 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W1" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_w2 psfdepth_w2 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W2" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_w3 psfdepth_w3 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W3" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_w4 psfdepth_w4 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W4" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE wise_coadd_id wise_coadd_id string COMMENT "unWISE coadd brick name (corresponding to the, e.g., legacysurvey-<brick>-image-W1.fits.fz coadd file) for the center of each object" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE wise_x wise_x float COMMENT "X position of coordinates in the brick image stack that corresponds to wise_coadd_id (see the DR9 updates page for transformations between wise_x and bx)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE wise_y wise_y float COMMENT "Y position of coordinates in the brick image stack that corresponds to wise_coadd_id (see the DR9 updates page for transformations between wise_y and by)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_flux_w1 lc_flux_w1 array<float> COMMENT "flux_w1 in each of up to fifteen unWISE coadd epochs (AB system\u003b defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_flux_w2 lc_flux_w2 array<float> COMMENT "flux_w2 in each of up to fifteen unWISE coadd epochs (AB\u003b defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_flux_ivar_w1 lc_flux_ivar_w1 array<float> COMMENT "Inverse variance of lc_flux_w1 (AB system\u003b defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_flux_ivar_w2 lc_flux_ivar_w2 array<float> COMMENT "Inverse variance of lc_flux_w2 (AB\u003b defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_nobs_w1 lc_nobs_w1 array<smallint> COMMENT "nobs_w1 in each of up to fifteen unWISE coadd epochs" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_nobs_w2 lc_nobs_w2 array<smallint> COMMENT "nobs_w2 in each of up to fifteen unWISE coadd epochs" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_fracflux_w1 lc_fracflux_w1 array<float> COMMENT "fracflux_w1 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_fracflux_w2 lc_fracflux_w2 array<float> COMMENT "fracflux_w2 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_rchisq_w1 lc_rchisq_w1 array<float> COMMENT "rchisq_w1 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_rchisq_w2 lc_rchisq_w2 array<float> COMMENT "rchisq_w2 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_mjd_w1 lc_mjd_w1 array<double> COMMENT "mjd_w1 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_mjd_w2 lc_mjd_w2 array<double> COMMENT "mjd_w2 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_epoch_index_w1 lc_epoch_index_w1 array<smallint> COMMENT "Index number of unWISE epoch for W1 (defaults to -1 for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_epoch_index_w2 lc_epoch_index_w2 array<smallint> COMMENT "Index number of unWISE epoch for W2 (defaults to -1 for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE sersic sersic float COMMENT "Power-law index for the Sersic profile model (type=\u0022SER\u0022)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE sersic_ivar sersic_ivar float COMMENT "Inverse variance of sersic" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_r shape_r float COMMENT "Half-light radius of galaxy model for galaxy type type (>0)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_r_ivar shape_r_ivar float COMMENT "Inverse variance of shape_r" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_e1 shape_e1 float COMMENT "Ellipticity component 1 of galaxy model for galaxy type type" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_e1_ivar shape_e1_ivar float COMMENT "Inverse variance of shape_e1" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_e2 shape_e2 float COMMENT "Ellipticity component 2 of galaxy model for galaxy type type" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_e2_ivar shape_e2_ivar float COMMENT "Inverse variance of shape_e2" ;</div>Talladahttps://pwiki.pic.es/index.php?title=Legacy_Survey_DR9&diff=877Legacy Survey DR92020-11-30T21:04:31Z<p>Tallada: /* Comments */</p>
<hr />
<div>== Original DR9 ==<br />
=== Raw data ===<br />
<br />
Downloaded from NERSC<br />
/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m<br />
<br />
Directories:<br />
south/tractor<br />
north/tractor<br />
<br />
Downloaded using bbcp from services01.euclid into CephFS scratch:<br />
bbcp -s 16 -a -A -v -z -r -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" \<br />
"jcarrete@dtn01.nersc.gov:/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m/north/tractor" \<br />
/cephfs/pic.es/astro/scratch/tallada/dr9m/north/tractor<br />
bbcp -s 16 -a -A -v -z -r -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" \<br />
"jcarrete@dtn01.nersc.gov:/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m/south/tractor" \<br />
/cephfs/pic.es/astro/scratch/tallada/dr9m/south/tractor<br />
<br />
Checksums provided through sha256sum files, and checked using command:<br />
find * -type d | xargs -t -L1 -P24 sh -c 'cd $1; sha256sum --quiet -c *.sha256sum' cksum<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to convert fits files to parquet:<br />
<br />
path_base = '/cephfs/pic.es/astro/scratch/tallada/dr9m'<br />
hdfs_base = '/user/tallada/data/legacysurvey_dr9'<br />
<br />
hs = ['h=north', 'h=south']<br />
rs = ['r={0:03d}'.format(x) for x in range(360)]<br />
<br />
for h in hs:<br />
for r in rs:<br />
p = os.path.join(path_base, h, r)<br />
if not os.path.exists(p):<br />
continue<br />
<br />
size = sum(os.path.getsize(os.path.join(p,f)) for f in os.listdir(p) if os.path.isfile(os.path.join(p,f)))<br />
<br />
print h, r, size <br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + p<br />
)<br />
<br />
df = df.repartition(int(math.ceil(size / (256*1024*1024.0))))<br />
<br />
df.write.parquet(os.path.join(hdfs_base, h, r))<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE `tallada`.`legacysurvey_dr9`(<br />
`release` smallint, <br />
`brickid` int, <br />
`brickname` string, <br />
`objid` int, <br />
`brick_primary` boolean, <br />
`maskbits` smallint, <br />
`fitbits` smallint, <br />
`type` string, <br />
`ra` double, <br />
`dec` double, <br />
`ra_ivar` float, <br />
`dec_ivar` float, <br />
`bx` float, <br />
`by` float, <br />
`dchisq` array<float>, <br />
`ebv` float, <br />
`mjd_min` double, <br />
`mjd_max` double, <br />
`ref_cat` string, <br />
`ref_id` bigint, <br />
`pmra` float, <br />
`pmdec` float, <br />
`parallax` float, <br />
`pmra_ivar` float, <br />
`pmdec_ivar` float, <br />
`parallax_ivar` float, <br />
`ref_epoch` float, <br />
`gaia_phot_g_mean_mag` float, <br />
`gaia_phot_g_mean_flux_over_error` float, <br />
`gaia_phot_g_n_obs` smallint, <br />
`gaia_phot_bp_mean_mag` float, <br />
`gaia_phot_bp_mean_flux_over_error` float, <br />
`gaia_phot_bp_n_obs` smallint, <br />
`gaia_phot_rp_mean_mag` float, <br />
`gaia_phot_rp_mean_flux_over_error` float, <br />
`gaia_phot_rp_n_obs` smallint, <br />
`gaia_phot_variable_flag` boolean, <br />
`gaia_astrometric_excess_noise` float, <br />
`gaia_astrometric_excess_noise_sig` float, <br />
`gaia_astrometric_n_obs_al` smallint, <br />
`gaia_astrometric_n_good_obs_al` smallint, <br />
`gaia_astrometric_weight_al` float, <br />
`gaia_duplicated_source` boolean, <br />
`gaia_a_g_val` float, <br />
`gaia_e_bp_min_rp_val` float, <br />
`gaia_phot_bp_rp_excess_factor` float, <br />
`gaia_astrometric_sigma5d_max` float, <br />
`gaia_astrometric_params_solved` tinyint, <br />
`flux_g` float, <br />
`flux_r` float, <br />
`flux_z` float, <br />
`flux_w1` float, <br />
`flux_w2` float, <br />
`flux_w3` float, <br />
`flux_w4` float, <br />
`flux_ivar_g` float, <br />
`flux_ivar_r` float, <br />
`flux_ivar_z` float, <br />
`flux_ivar_w1` float, <br />
`flux_ivar_w2` float, <br />
`flux_ivar_w3` float, <br />
`flux_ivar_w4` float, <br />
`fiberflux_g` float, <br />
`fiberflux_r` float, <br />
`fiberflux_z` float, <br />
`fibertotflux_g` float, <br />
`fibertotflux_r` float, <br />
`fibertotflux_z` float, <br />
`apflux_g` array<float>, <br />
`apflux_r` array<float>, <br />
`apflux_z` array<float>, <br />
`apflux_resid_g` array<float>, <br />
`apflux_resid_r` array<float>, <br />
`apflux_resid_z` array<float>, <br />
`apflux_blobresid_g` array<float>, <br />
`apflux_blobresid_r` array<float>, <br />
`apflux_blobresid_z` array<float>, <br />
`apflux_ivar_g` array<float>, <br />
`apflux_ivar_r` array<float>, <br />
`apflux_ivar_z` array<float>, <br />
`apflux_masked_g` array<float>, <br />
`apflux_masked_r` array<float>, <br />
`apflux_masked_z` array<float>, <br />
`apflux_w1` array<float>, <br />
`apflux_w2` array<float>, <br />
`apflux_w3` array<float>, <br />
`apflux_w4` array<float>, <br />
`apflux_resid_w1` array<float>, <br />
`apflux_resid_w2` array<float>, <br />
`apflux_resid_w3` array<float>, <br />
`apflux_resid_w4` array<float>, <br />
`apflux_ivar_w1` array<float>, <br />
`apflux_ivar_w2` array<float>, <br />
`apflux_ivar_w3` array<float>, <br />
`apflux_ivar_w4` array<float>, <br />
`mw_transmission_g` float, <br />
`mw_transmission_r` float, <br />
`mw_transmission_z` float, <br />
`mw_transmission_w1` float, <br />
`mw_transmission_w2` float, <br />
`mw_transmission_w3` float, <br />
`mw_transmission_w4` float, <br />
`nobs_g` smallint, <br />
`nobs_r` smallint, <br />
`nobs_z` smallint, <br />
`nobs_w1` smallint, <br />
`nobs_w2` smallint, <br />
`nobs_w3` smallint, <br />
`nobs_w4` smallint, <br />
`rchisq_g` float, <br />
`rchisq_r` float, <br />
`rchisq_z` float, <br />
`rchisq_w1` float, <br />
`rchisq_w2` float, <br />
`rchisq_w3` float, <br />
`rchisq_w4` float, <br />
`fracflux_g` float, <br />
`fracflux_r` float, <br />
`fracflux_z` float, <br />
`fracflux_w1` float, <br />
`fracflux_w2` float, <br />
`fracflux_w3` float, <br />
`fracflux_w4` float, <br />
`fracmasked_g` float, <br />
`fracmasked_r` float, <br />
`fracmasked_z` float, <br />
`fracin_g` float, <br />
`fracin_r` float, <br />
`fracin_z` float, <br />
`anymask_g` smallint, <br />
`anymask_r` smallint, <br />
`anymask_z` smallint, <br />
`allmask_g` smallint, <br />
`allmask_r` smallint, <br />
`allmask_z` smallint, <br />
`wisemask_w1` tinyint, <br />
`wisemask_w2` tinyint, <br />
`psfsize_g` float, <br />
`psfsize_r` float, <br />
`psfsize_z` float, <br />
`psfdepth_g` float, <br />
`psfdepth_r` float, <br />
`psfdepth_z` float, <br />
`galdepth_g` float, <br />
`galdepth_r` float, <br />
`galdepth_z` float, <br />
`nea_g` float, <br />
`nea_r` float, <br />
`nea_z` float, <br />
`blob_nea_g` float, <br />
`blob_nea_r` float, <br />
`blob_nea_z` float, <br />
`psfdepth_w1` float, <br />
`psfdepth_w2` float, <br />
`psfdepth_w3` float, <br />
`psfdepth_w4` float, <br />
`wise_coadd_id` string, <br />
`wise_x` float, <br />
`wise_y` float, <br />
`lc_flux_w1` array<float>, <br />
`lc_flux_w2` array<float>, <br />
`lc_flux_ivar_w1` array<float>, <br />
`lc_flux_ivar_w2` array<float>, <br />
`lc_nobs_w1` array<smallint>, <br />
`lc_nobs_w2` array<smallint>, <br />
`lc_fracflux_w1` array<float>, <br />
`lc_fracflux_w2` array<float>, <br />
`lc_rchisq_w1` array<float>, <br />
`lc_rchisq_w2` array<float>, <br />
`lc_mjd_w1` array<double>, <br />
`lc_mjd_w2` array<double>, <br />
`lc_epoch_index_w1` array<smallint>, <br />
`lc_epoch_index_w2` array<smallint>, <br />
`sersic` float, <br />
`sersic_ivar` float, <br />
`shape_r` float, <br />
`shape_r_ivar` float, <br />
`shape_e1` float, <br />
`shape_e1_ivar` float, <br />
`shape_e2` float, <br />
`shape_e2_ivar` float)<br />
PARTITIONED BY (<br />
h STRING,<br />
r STRING<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey_dr9'<br />
;<br />
<br />
=== Internal Table ===<br />
<br />
CREATE TABLE `cosmohub`.`legacy_survey_dr9`(<br />
`release` smallint,<br />
`brickid` int,<br />
`brickname` string,<br />
`objid` int,<br />
`brick_primary` boolean,<br />
`maskbits` smallint,<br />
`fitbits` smallint,<br />
`type` string,<br />
`ra` double,<br />
`dec` double,<br />
`ra_ivar` float,<br />
`dec_ivar` float,<br />
`bx` float,<br />
`by` float,<br />
`dchisq` array<float>,<br />
`ebv` float,<br />
`mjd_min` double,<br />
`mjd_max` double,<br />
`ref_cat` string,<br />
`ref_id` bigint,<br />
`pmra` float,<br />
`pmdec` float,<br />
`parallax` float,<br />
`pmra_ivar` float,<br />
`pmdec_ivar` float,<br />
`parallax_ivar` float,<br />
`ref_epoch` float,<br />
`gaia_phot_g_mean_mag` float,<br />
`gaia_phot_g_mean_flux_over_error` float,<br />
`gaia_phot_g_n_obs` smallint,<br />
`gaia_phot_bp_mean_mag` float,<br />
`gaia_phot_bp_mean_flux_over_error` float,<br />
`gaia_phot_bp_n_obs` smallint,<br />
`gaia_phot_rp_mean_mag` float,<br />
`gaia_phot_rp_mean_flux_over_error` float,<br />
`gaia_phot_rp_n_obs` smallint,<br />
`gaia_phot_variable_flag` boolean,<br />
`gaia_astrometric_excess_noise` float,<br />
`gaia_astrometric_excess_noise_sig` float,<br />
`gaia_astrometric_n_obs_al` smallint,<br />
`gaia_astrometric_n_good_obs_al` smallint,<br />
`gaia_astrometric_weight_al` float,<br />
`gaia_duplicated_source` boolean,<br />
`gaia_a_g_val` float,<br />
`gaia_e_bp_min_rp_val` float,<br />
`gaia_phot_bp_rp_excess_factor` float,<br />
`gaia_astrometric_sigma5d_max` float,<br />
`gaia_astrometric_params_solved` tinyint,<br />
`flux_g` float,<br />
`flux_r` float,<br />
`flux_z` float,<br />
`flux_w1` float,<br />
`flux_w2` float,<br />
`flux_w3` float,<br />
`flux_w4` float,<br />
`flux_ivar_g` float,<br />
`flux_ivar_r` float,<br />
`flux_ivar_z` float,<br />
`flux_ivar_w1` float,<br />
`flux_ivar_w2` float,<br />
`flux_ivar_w3` float,<br />
`flux_ivar_w4` float,<br />
`fiberflux_g` float,<br />
`fiberflux_r` float,<br />
`fiberflux_z` float,<br />
`fibertotflux_g` float,<br />
`fibertotflux_r` float,<br />
`fibertotflux_z` float,<br />
`apflux_g` array<float>,<br />
`apflux_r` array<float>,<br />
`apflux_z` array<float>,<br />
`apflux_resid_g` array<float>,<br />
`apflux_resid_r` array<float>,<br />
`apflux_resid_z` array<float>,<br />
`apflux_blobresid_g` array<float>,<br />
`apflux_blobresid_r` array<float>,<br />
`apflux_blobresid_z` array<float>,<br />
`apflux_ivar_g` array<float>,<br />
`apflux_ivar_r` array<float>,<br />
`apflux_ivar_z` array<float>,<br />
`apflux_masked_g` array<float>,<br />
`apflux_masked_r` array<float>,<br />
`apflux_masked_z` array<float>,<br />
`apflux_w1` array<float>,<br />
`apflux_w2` array<float>,<br />
`apflux_w3` array<float>,<br />
`apflux_w4` array<float>,<br />
`apflux_resid_w1` array<float>,<br />
`apflux_resid_w2` array<float>,<br />
`apflux_resid_w3` array<float>,<br />
`apflux_resid_w4` array<float>,<br />
`apflux_ivar_w1` array<float>,<br />
`apflux_ivar_w2` array<float>,<br />
`apflux_ivar_w3` array<float>,<br />
`apflux_ivar_w4` array<float>,<br />
`mw_transmission_g` float,<br />
`mw_transmission_r` float,<br />
`mw_transmission_z` float,<br />
`mw_transmission_w1` float,<br />
`mw_transmission_w2` float,<br />
`mw_transmission_w3` float,<br />
`mw_transmission_w4` float,<br />
`nobs_g` smallint,<br />
`nobs_r` smallint,<br />
`nobs_z` smallint,<br />
`nobs_w1` smallint,<br />
`nobs_w2` smallint,<br />
`nobs_w3` smallint,<br />
`nobs_w4` smallint,<br />
`rchisq_g` float,<br />
`rchisq_r` float,<br />
`rchisq_z` float,<br />
`rchisq_w1` float,<br />
`rchisq_w2` float,<br />
`rchisq_w3` float,<br />
`rchisq_w4` float,<br />
`fracflux_g` float,<br />
`fracflux_r` float,<br />
`fracflux_z` float,<br />
`fracflux_w1` float,<br />
`fracflux_w2` float,<br />
`fracflux_w3` float,<br />
`fracflux_w4` float,<br />
`fracmasked_g` float,<br />
`fracmasked_r` float,<br />
`fracmasked_z` float,<br />
`fracin_g` float,<br />
`fracin_r` float,<br />
`fracin_z` float,<br />
`anymask_g` smallint,<br />
`anymask_r` smallint,<br />
`anymask_z` smallint,<br />
`allmask_g` smallint,<br />
`allmask_r` smallint,<br />
`allmask_z` smallint,<br />
`wisemask_w1` tinyint,<br />
`wisemask_w2` tinyint,<br />
`psfsize_g` float,<br />
`psfsize_r` float,<br />
`psfsize_z` float,<br />
`psfdepth_g` float,<br />
`psfdepth_r` float,<br />
`psfdepth_z` float,<br />
`galdepth_g` float,<br />
`galdepth_r` float,<br />
`galdepth_z` float,<br />
`nea_g` float,<br />
`nea_r` float,<br />
`nea_z` float,<br />
`blob_nea_g` float,<br />
`blob_nea_r` float,<br />
`blob_nea_z` float,<br />
`psfdepth_w1` float,<br />
`psfdepth_w2` float,<br />
`psfdepth_w3` float,<br />
`psfdepth_w4` float,<br />
`wise_coadd_id` string,<br />
`wise_x` float,<br />
`wise_y` float,<br />
`lc_flux_w1` array<float>,<br />
`lc_flux_w2` array<float>,<br />
`lc_flux_ivar_w1` array<float>,<br />
`lc_flux_ivar_w2` array<float>,<br />
`lc_nobs_w1` array<smallint>,<br />
`lc_nobs_w2` array<smallint>,<br />
`lc_fracflux_w1` array<float>,<br />
`lc_fracflux_w2` array<float>,<br />
`lc_rchisq_w1` array<float>,<br />
`lc_rchisq_w2` array<float>,<br />
`lc_mjd_w1` array<double>,<br />
`lc_mjd_w2` array<double>,<br />
`lc_epoch_index_w1` array<smallint>,<br />
`lc_epoch_index_w2` array<smallint>,<br />
`sersic` float,<br />
`sersic_ivar` float,<br />
`shape_r` float,<br />
`shape_r_ivar` float,<br />
`shape_e1` float,<br />
`shape_e1_ivar` float,<br />
`shape_e2` float,<br />
`shape_e2_ivar` float)<br />
CLUSTERED BY (<br />
release, brickid, objid<br />
)<br />
SORTED BY (<br />
release, brickid, objid<br />
)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
<br />
INSERT INTO TABLE cosmohub.legacy_survey_dr9<br />
SELECT release, brickid, brickname, objid, brick_primary, maskbits, fitbits, type, ra, `dec`, ra_ivar, dec_ivar,<br />
bx, `by`, dchisq, ebv, mjd_min, mjd_max, ref_cat, ref_id, pmra, pmdec, parallax, pmra_ivar, pmdec_ivar,<br />
parallax_ivar, ref_epoch, gaia_phot_g_mean_mag, gaia_phot_g_mean_flux_over_error, gaia_phot_g_n_obs,<br />
gaia_phot_bp_mean_mag, gaia_phot_bp_mean_flux_over_error, gaia_phot_bp_n_obs, gaia_phot_rp_mean_mag,<br />
gaia_phot_rp_mean_flux_over_error, gaia_phot_rp_n_obs, gaia_phot_variable_flag, gaia_astrometric_excess_noise,<br />
gaia_astrometric_excess_noise_sig, gaia_astrometric_n_obs_al, gaia_astrometric_n_good_obs_al,<br />
gaia_astrometric_weight_al, gaia_duplicated_source, gaia_a_g_val, gaia_e_bp_min_rp_val,<br />
gaia_phot_bp_rp_excess_factor, gaia_astrometric_sigma5d_max, gaia_astrometric_params_solved, flux_g, flux_r,<br />
flux_z, flux_w1, flux_w2, flux_w3, flux_w4, flux_ivar_g, flux_ivar_r, flux_ivar_z, flux_ivar_w1, flux_ivar_w2,<br />
flux_ivar_w3, flux_ivar_w4, fiberflux_g, fiberflux_r, fiberflux_z, fibertotflux_g, fibertotflux_r, fibertotflux_z,<br />
apflux_g, apflux_r, apflux_z, apflux_resid_g, apflux_resid_r, apflux_resid_z, apflux_blobresid_g,<br />
apflux_blobresid_r, apflux_blobresid_z, apflux_ivar_g, apflux_ivar_r, apflux_ivar_z, apflux_masked_g,<br />
apflux_masked_r, apflux_masked_z, apflux_w1, apflux_w2, apflux_w3, apflux_w4, apflux_resid_w1, apflux_resid_w2,<br />
apflux_resid_w3, apflux_resid_w4, apflux_ivar_w1, apflux_ivar_w2, apflux_ivar_w3, apflux_ivar_w4,<br />
mw_transmission_g, mw_transmission_r, mw_transmission_z, mw_transmission_w1, mw_transmission_w2,<br />
mw_transmission_w3, mw_transmission_w4, nobs_g, nobs_r, nobs_z, nobs_w1, nobs_w2, nobs_w3, nobs_w4, rchisq_g,<br />
rchisq_r, rchisq_z, rchisq_w1, rchisq_w2, rchisq_w3, rchisq_w4, fracflux_g, fracflux_r, fracflux_z, fracflux_w1,<br />
fracflux_w2, fracflux_w3, fracflux_w4, fracmasked_g, fracmasked_r, fracmasked_z, fracin_g, fracin_r, fracin_z,<br />
anymask_g, anymask_r, anymask_z, allmask_g, allmask_r, allmask_z, wisemask_w1, wisemask_w2, psfsize_g, psfsize_r,<br />
psfsize_z, psfdepth_g, psfdepth_r, psfdepth_z, galdepth_g, galdepth_r, galdepth_z, nea_g, nea_r, nea_z,<br />
blob_nea_g, blob_nea_r, blob_nea_z, psfdepth_w1, psfdepth_w2, psfdepth_w3, psfdepth_w4, wise_coadd_id, wise_x,<br />
wise_y, lc_flux_w1, lc_flux_w2, lc_flux_ivar_w1, lc_flux_ivar_w2, lc_nobs_w1, lc_nobs_w2, lc_fracflux_w1,<br />
lc_fracflux_w2, lc_rchisq_w1, lc_rchisq_w2, lc_mjd_w1, lc_mjd_w2, lc_epoch_index_w1, lc_epoch_index_w2, sersic,<br />
sersic_ivar, shape_r, shape_r_ivar, shape_e1, shape_e1_ivar, shape_e2, shape_e2_ivar<br />
FROM cosmohub.legacy_survey_dr9_p<br />
;<br />
<br />
=== Comments ===<br />
ALTER TABLE legacy_survey_dr9 CHANGE release release smallint COMMENT "Unique integer denoting the camera and filter set used" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE brickid brickid int COMMENT "Brick ID [1,662174]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE brickname brickname string COMMENT "Name of brick, encoding the brick sky position, eg \u00221126p222\u0022 near RA=112.6, Dec=+22.2" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE objid objid int COMMENT "Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE brick_primary brick_primary boolean COMMENT "True if the object is within the brick boundary" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE maskbits maskbits smallint COMMENT "Bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits* maps, as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fitbits fitbits smallint COMMENT "Bitwise mask detailing pecularities of how an object was fit, as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE type type string COMMENT "Morphological model: \u0022PSF\u0022=stellar, \u0022REX\u0022=\u0022round exponential galaxy\u0022, \u0022DEV\u0022=deVauc, \u0022EXP\u0022=exponential, \u0022SER\u0022=Sersic, \u0022DUP\u0022=Gaia source fit by different model." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ra ra double COMMENT "Right ascension at equinox J2000" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE `dec` `dec` double COMMENT "Declination at equinox J2000" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ra_ivar ra_ivar float COMMENT "Inverse variance of RA (no cosine term!), excluding astrometric calibration errors" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE dec_ivar dec_ivar float COMMENT "Inverse variance of DEC, excluding astrometric calibration errors" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE bx bx float COMMENT "X position (0-indexed) of coordinates in the brick image stack (i.e. in the e.g. legacysurvey-<brick>-image-g.fits.fz coadd file)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE `by` `by` float COMMENT "Y position (0-indexed) of coordinates in brick image stack" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE dchisq dchisq array<float> COMMENT "Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, SER. The difference is versus no source." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ebv ebv float COMMENT "Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mjd_min mjd_min double COMMENT "Minimum Modified Julian Date of observations used to construct the model of this object" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mjd_max mjd_max double COMMENT "Maximum Modified Julian Date of observations used to construct the model of this object" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ref_cat ref_cat string COMMENT "Reference catalog source for this star: \u0022T2\u0022 for Tycho-2, \u0022G2\u0022 for Gaia DR2, \u0022L3\u0022 for the SGA, empty otherwise" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ref_id ref_id bigint COMMENT "Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b \u0022sourceid\u0022 for Gaia DR2 and SGA" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE pmra pmra float COMMENT "Reference catalog proper motion in the RA direction" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE pmdec pmdec float COMMENT "Reference catalog proper motion in the Dec direction" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE parallax parallax float COMMENT "Reference catalog parallax" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE pmra_ivar pmra_ivar float COMMENT "Reference catalog inverse-variance on pmra" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE pmdec_ivar pmdec_ivar float COMMENT "Reference catalog inverse-variance on pmdec" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE parallax_ivar parallax_ivar float COMMENT "Reference catalog inverse-variance on parallax" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ref_epoch ref_epoch float COMMENT "Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_g_mean_mag gaia_phot_g_mean_mag float COMMENT "Gaia G band mag" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_g_mean_flux_over_error gaia_phot_g_mean_flux_over_error float COMMENT "Gaia G band signal-to-noise" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_g_n_obs gaia_phot_g_n_obs smallint COMMENT "Gaia G band number of observations" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_bp_mean_mag gaia_phot_bp_mean_mag float COMMENT "Gaia BP mag" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_bp_mean_flux_over_error gaia_phot_bp_mean_flux_over_error float COMMENT "Gaia BP signal-to-noise" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_bp_n_obs gaia_phot_bp_n_obs smallint COMMENT "Gaia BP number of observations" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_rp_mean_mag gaia_phot_rp_mean_mag float COMMENT "Gaia RP mag" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_rp_mean_flux_over_error gaia_phot_rp_mean_flux_over_error float COMMENT "Gaia RP signal-to-noise" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_rp_n_obs gaia_phot_rp_n_obs smallint COMMENT "Gaia RP number of observations" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_variable_flag gaia_phot_variable_flag boolean COMMENT "Gaia photometric variable flag" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_excess_noise gaia_astrometric_excess_noise float COMMENT "Gaia astrometric excess noise" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_excess_noise_sig gaia_astrometric_excess_noise_sig float COMMENT "Gaia astrometric excess noise uncertainty" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_n_obs_al gaia_astrometric_n_obs_al smallint COMMENT "Gaia number of astrometric observations along scan direction" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_n_good_obs_al gaia_astrometric_n_good_obs_al smallint COMMENT "Gaia number of good astrometric observations along scan direction" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_weight_al gaia_astrometric_weight_al float COMMENT "Gaia astrometric weight along scan direction" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_duplicated_source gaia_duplicated_source boolean COMMENT "Gaia duplicated source flag" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_a_g_val gaia_a_g_val float COMMENT "Gaia line-of-sight extinction in the G band" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_e_bp_min_rp_val gaia_e_bp_min_rp_val float COMMENT "Gaia line-of-sight reddening E(BP-RP)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_bp_rp_excess_factor gaia_phot_bp_rp_excess_factor float COMMENT "Gaia BP/RP excess factor" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_sigma5d_max gaia_astrometric_sigma5d_max float COMMENT "Gaia longest semi-major axis of the 5-d error ellipsoid" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_params_solved gaia_astrometric_params_solved tinyint COMMENT "which astrometric parameters were estimated for a Gaia source" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_g flux_g float COMMENT "model flux in gg" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_r flux_r float COMMENT "model flux in rr" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_z flux_z float COMMENT "model flux in zz" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_w1 flux_w1 float COMMENT "WISE model flux in W1W1 (AB system)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_w2 flux_w2 float COMMENT "WISE model flux in W2W2 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_w3 flux_w3 float COMMENT "WISE model flux in W3W3 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_w4 flux_w4 float COMMENT "WISE model flux in W4W4 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_g flux_ivar_g float COMMENT "Inverse variance of flux_g" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_r flux_ivar_r float COMMENT "Inverse variance of flux_r" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_z flux_ivar_z float COMMENT "Inverse variance of flux_z" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_w1 flux_ivar_w1 float COMMENT "Inverse variance of flux_w1 (AB system)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_w2 flux_ivar_w2 float COMMENT "Inverse variance of flux_w2 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_w3 flux_ivar_w3 float COMMENT "Inverse variance of flux_w3 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_w4 flux_ivar_w4 float COMMENT "Inverse variance of flux_w4 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fiberflux_g fiberflux_g float COMMENT "Predicted gg-band flux within a fiber of diameter 1.5 arcsec from this object in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fiberflux_r fiberflux_r float COMMENT "Predicted rr-band flux within a fiber of diameter 1.5 arcsec from this object in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fiberflux_z fiberflux_z float COMMENT "Predicted zz-band flux within a fiber of diameter 1.5 arcsec from this object in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fibertotflux_g fibertotflux_g float COMMENT "Predicted gg-band flux within a fiber of diameter 1.5 arcsec from all sources at this location in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fibertotflux_r fibertotflux_r float COMMENT "Predicted rr-band flux within a fiber of diameter 1.5 arcsec from all sources at this location in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fibertotflux_z fibertotflux_z float COMMENT "Predicted zz-band flux within a fiber of diameter 1.5 arcsec from all sources at this location in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_g apflux_g array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg, masked by invvar=0invvar=0 (inverse variance of zero [1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_r apflux_r array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_z apflux_z array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_g apflux_resid_g array<float> COMMENT "Aperture fluxes on the co-added residual images in gg, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_r apflux_resid_r array<float> COMMENT "Aperture fluxes on the co-added residual images in rr, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_z apflux_resid_z array<float> COMMENT "Aperture fluxes on the co-added residual images in zz, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_blobresid_g apflux_blobresid_g array<float> COMMENT "Aperture fluxes on image−blobmodelimage−blobmodel residual maps in gg [2], masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_blobresid_r apflux_blobresid_r array<float> COMMENT "Aperture fluxes on image−blobmodelimage−blobmodel residual maps in rr, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_blobresid_z apflux_blobresid_z array<float> COMMENT "Aperture fluxes on image−blobmodelimage−blobmodel residual maps in zz, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_g apflux_ivar_g array<float> COMMENT "Inverse variance of apflux_resid_g, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_r apflux_ivar_r array<float> COMMENT "Inverse variance of apflux_resid_r, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_z apflux_ivar_z array<float> COMMENT "Inverse variance of apflux_resid_z, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_masked_g apflux_masked_g array<float> COMMENT "Fraction of pixels masked in gg-band aperture flux measurements\u003b 1 means fully masked (ie, fully ignored\u003b contributing zero to the measurement)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_masked_r apflux_masked_r array<float> COMMENT "Fraction of pixels masked in rr-band aperture flux measurements\u003b 1 means fully masked (ie, fully ignored\u003b contributing zero to the measurement)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_masked_z apflux_masked_z array<float> COMMENT "Fraction of pixels masked in zz-band aperture flux measurements\u003b 1 means fully masked (ie, fully ignored\u003b contributing zero to the measurement)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_w1 apflux_w1 array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [3, 5, 7, 9, 11] [3] arcsec in W1W1, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_w2 apflux_w2 array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [3, 5, 7, 9, 11] arcsec in W2W2, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_w3 apflux_w3 array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [3, 5, 7, 9, 11] arcsec in W3W3, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_w4 apflux_w4 array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [3, 5, 7, 9, 11] arcsec in W4W4, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_w1 apflux_resid_w1 array<float> COMMENT "Aperture fluxes on the co-added residual images in W1W1, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_w2 apflux_resid_w2 array<float> COMMENT "Aperture fluxes on the co-added residual images in W2W2, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_w3 apflux_resid_w3 array<float> COMMENT "Aperture fluxes on the co-added residual images in W3W3, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_w4 apflux_resid_w4 array<float> COMMENT "Aperture fluxes on the co-added residual images in W4W4, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_w1 apflux_ivar_w1 array<float> COMMENT "Inverse variance of apflux_resid_w1, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_w2 apflux_ivar_w2 array<float> COMMENT "Inverse variance of apflux_resid_w2, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_w3 apflux_ivar_w3 array<float> COMMENT "Inverse variance of apflux_resid_w3, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_w4 apflux_ivar_w4 array<float> COMMENT "Inverse variance of apflux_resid_w4, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_g mw_transmission_g float COMMENT "Galactic transmission in gg filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_r mw_transmission_r float COMMENT "Galactic transmission in rr filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_z mw_transmission_z float COMMENT "Galactic transmission in zz filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_w1 mw_transmission_w1 float COMMENT "Galactic transmission in W1W1 filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_w2 mw_transmission_w2 float COMMENT "Galactic transmission in W2W2 filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_w3 mw_transmission_w3 float COMMENT "Galactic transmission in W3W3 filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_w4 mw_transmission_w4 float COMMENT "Galactic transmission in W4W4 filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_g nobs_g smallint COMMENT "Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_r nobs_r smallint COMMENT "Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_z nobs_z smallint COMMENT "Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_w1 nobs_w1 smallint COMMENT "Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_w2 nobs_w2 smallint COMMENT "Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_w3 nobs_w3 smallint COMMENT "Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_w4 nobs_w4 smallint COMMENT "Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_g rchisq_g float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in gg" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_r rchisq_r float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in rr" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_z rchisq_z float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in zz" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_w1 rchisq_w1 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W1W1" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_w2 rchisq_w2 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W2W2" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_w3 rchisq_w3 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W3W3" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_w4 rchisq_w4 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W4W4" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_g fracflux_g float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in gg (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_r fracflux_r float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in rr (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_z fracflux_z float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in zz (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_w1 fracflux_w1 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1 (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_w2 fracflux_w2 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2 (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_w3 fracflux_w3 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3 (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_w4 fracflux_w4 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4 (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracmasked_g fracmasked_g float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracmasked_r fracmasked_r float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracmasked_z fracmasked_z float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracin_g fracin_g float COMMENT "Fraction of a source's flux within the blob in gg, near unity for real sources" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracin_r fracin_r float COMMENT "Fraction of a source's flux within the blob in rr, near unity for real sources" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracin_z fracin_z float COMMENT "Fraction of a source's flux within the blob in zz, near unity for real sources" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE anymask_g anymask_g smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE anymask_r anymask_r smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE anymask_z anymask_z smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE allmask_g allmask_g smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE allmask_r allmask_r smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE allmask_z allmask_z smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE wisemask_w1 wisemask_w1 tinyint COMMENT "W1 bitmask as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE wisemask_w2 wisemask_w2 tinyint COMMENT "W2 bitmask as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfsize_g psfsize_g float COMMENT "Weighted average PSF FWHM in the gg band" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfsize_r psfsize_r float COMMENT "Weighted average PSF FWHM in the rr band" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfsize_z psfsize_z float COMMENT "Weighted average PSF FWHM in the zz band" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_g psfdepth_g float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_r psfdepth_r float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_z psfdepth_z float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE galdepth_g galdepth_g float COMMENT "As for psfdepth_g but for a galaxy (0.45\u0022 exp, round) detection sensitivity" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE galdepth_r galdepth_r float COMMENT "As for psfdepth_r but for a galaxy (0.45\u0022 exp, round) detection sensitivity" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE galdepth_z galdepth_z float COMMENT "As for psfdepth_z but for a galaxy (0.45\u0022 exp, round) detection sensitivity" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nea_g nea_g float COMMENT "Noise equivalent area in gg." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nea_r nea_r float COMMENT "Noise equivalent area in rr." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nea_z nea_z float COMMENT "Noise equivalent area in zz." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE blob_nea_g blob_nea_g float COMMENT "Blob-masked noise equivalent area in gg." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE blob_nea_r blob_nea_r float COMMENT "Blob-masked noise equivalent area in rr." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE blob_nea_z blob_nea_z float COMMENT "Blob-masked noise equivalent area in zz." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_w1 psfdepth_w1 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W1" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_w2 psfdepth_w2 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W2" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_w3 psfdepth_w3 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W3" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_w4 psfdepth_w4 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W4" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE wise_coadd_id wise_coadd_id string COMMENT "unWISE coadd brick name (corresponding to the, e.g., legacysurvey-<brick>-image-W1.fits.fz coadd file) for the center of each object" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE wise_x wise_x float COMMENT "X position of coordinates in the brick image stack that corresponds to wise_coadd_id (see the DR9 updates page for transformations between wise_x and bx)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE wise_y wise_y float COMMENT "Y position of coordinates in the brick image stack that corresponds to wise_coadd_id (see the DR9 updates page for transformations between wise_y and by)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_flux_w1 lc_flux_w1 array<float> COMMENT "flux_w1 in each of up to fifteen unWISE coadd epochs (AB system\u003b defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_flux_w2 lc_flux_w2 array<float> COMMENT "flux_w2 in each of up to fifteen unWISE coadd epochs (AB\u003b defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_flux_ivar_w1 lc_flux_ivar_w1 array<float> COMMENT "Inverse variance of lc_flux_w1 (AB system\u003b defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_flux_ivar_w2 lc_flux_ivar_w2 array<float> COMMENT "Inverse variance of lc_flux_w2 (AB\u003b defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_nobs_w1 lc_nobs_w1 array<smallint> COMMENT "nobs_w1 in each of up to fifteen unWISE coadd epochs" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_nobs_w2 lc_nobs_w2 array<smallint> COMMENT "nobs_w2 in each of up to fifteen unWISE coadd epochs" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_fracflux_w1 lc_fracflux_w1 array<float> COMMENT "fracflux_w1 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_fracflux_w2 lc_fracflux_w2 array<float> COMMENT "fracflux_w2 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_rchisq_w1 lc_rchisq_w1 array<float> COMMENT "rchisq_w1 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_rchisq_w2 lc_rchisq_w2 array<float> COMMENT "rchisq_w2 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_mjd_w1 lc_mjd_w1 array<double> COMMENT "mjd_w1 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_mjd_w2 lc_mjd_w2 array<double> COMMENT "mjd_w2 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_epoch_index_w1 lc_epoch_index_w1 array<smallint> COMMENT "Index number of unWISE epoch for W1 (defaults to -1 for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_epoch_index_w2 lc_epoch_index_w2 array<smallint> COMMENT "Index number of unWISE epoch for W2 (defaults to -1 for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE sersic sersic float COMMENT "Power-law index for the Sersic profile model (type=\u0022SER\u0022)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE sersic_ivar sersic_ivar float COMMENT "Inverse variance of sersic" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_r shape_r float COMMENT "Half-light radius of galaxy model for galaxy type type (>0)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_r_ivar shape_r_ivar float COMMENT "Inverse variance of shape_r" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_e1 shape_e1 float COMMENT "Ellipticity component 1 of galaxy model for galaxy type type" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_e1_ivar shape_e1_ivar float COMMENT "Inverse variance of shape_e1" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_e2 shape_e2 float COMMENT "Ellipticity component 2 of galaxy model for galaxy type type" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_e2_ivar shape_e2_ivar float COMMENT "Inverse variance of shape_e2" ;<br />
<br />
== Sweep photo-z ==<br />
<br />
=== Raw data ===<br />
<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/north/sweep/<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/south/sweep/<br />
<br />
Checksums checked:<br />
<br />
$ cd north/sweep/8.0<br />
$ ls | wc -l<br />
287<br />
$ wc -l legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
286 (+ checksum file)<br />
$ sha256sum -c legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
OK<br />
<br />
$ cd north/sweep/8.0-photo-z<br />
$ ls | wc -l<br />
287<br />
$ wc -l legacysurvey_dr8_north_sweep_8.0-photo-z.sha256sum<br />
286 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_north_sweep_8.0-photo-z.sha256sum<br />
OK<br />
<br />
$ cd south/sweep/8.0<br />
$ ls | wc -l<br />
438<br />
$ wc -l legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
437 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
OK<br />
<br />
$ cd south/sweep/8.0-photo-z<br />
$ ls | wc -l<br />
438<br />
$ wc -l legacysurvey_dr8_south_sweep_8.0-photo-z.sha256sum<br />
437 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_south_sweep_8.0-photo-z.sha256sum<br />
OK<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to merge fits files and convert to parquet:<br />
<br />
path_base = '/cephfs/pic.es/astro/scratch/tallada/dr8-phz/'<br />
hdfs_base = '/user/tallada/data/legacysurvey-sweep'<br />
<br />
hs = ['south/sweep'] #, 'north/sweep']<br />
dir_src = '8.0'<br />
dir_phz = '8.0-photo-z'<br />
dir_dst = '8.0-merged'<br />
<br />
my_files = sorted(os.listdir(os.path.join(path_base, hs[0], dir_src)))<br />
<br />
def merge_phz(file_name):<br />
file_src = os.path.join(path_base, hs[0], dir_src, file_name)<br />
file_phz = os.path.join(path_base, hs[0], dir_phz, file_name[:-5]+'-pz.fits')<br />
file_dst = os.path.join(path_base, hs[0], dir_dst, file_name[:-5]+'-merged.fits')<br />
<br />
t_src = table.Table.read(file_src)<br />
t_phz = table.Table.read(file_phz)<br />
t_dst = table.hstack([t_src, t_phz], join_type='exact')<br />
<br />
os.unlink(file_dst)<br />
t_dst.write(file_dst)<br />
<br />
return file_name<br />
<br />
df = sc.parallelize(my_files, len(my_files))<br />
<br />
df.map(merge_phz).count()<br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + os.path.join(path_base, hs[0], dir_dst)<br />
) <br />
<br />
df.write.parquet(os.path.join(hdfs_base, hs[0]))<br />
<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE tallada.dr8_sweep (<br />
`RELEASE` SMALLINT,<br />
`BRICKID` INT,<br />
`BRICKNAME` STRING,<br />
`OBJID` INT,<br />
`TYPE` STRING,<br />
`RA` DOUBLE,<br />
`DEC` DOUBLE,<br />
`RA_IVAR` FLOAT,<br />
`DEC_IVAR` FLOAT,<br />
`DCHISQ` ARRAY<FLOAT>,<br />
`EBV` FLOAT,<br />
`FLUX_G` FLOAT,<br />
`FLUX_R` FLOAT,<br />
`FLUX_Z` FLOAT,<br />
`FLUX_W1` FLOAT,<br />
`FLUX_W2` FLOAT,<br />
`FLUX_W3` FLOAT,<br />
`FLUX_W4` FLOAT,<br />
`FLUX_IVAR_G` FLOAT,<br />
`FLUX_IVAR_R` FLOAT,<br />
`FLUX_IVAR_Z` FLOAT,<br />
`FLUX_IVAR_W1` FLOAT,<br />
`FLUX_IVAR_W2` FLOAT,<br />
`FLUX_IVAR_W3` FLOAT,<br />
`FLUX_IVAR_W4` FLOAT,<br />
`MW_TRANSMISSION_G` FLOAT,<br />
`MW_TRANSMISSION_R` FLOAT,<br />
`MW_TRANSMISSION_Z` FLOAT,<br />
`MW_TRANSMISSION_W1` FLOAT,<br />
`MW_TRANSMISSION_W2` FLOAT,<br />
`MW_TRANSMISSION_W3` FLOAT,<br />
`MW_TRANSMISSION_W4` FLOAT,<br />
`NOBS_G` INT,<br />
`NOBS_R` INT,<br />
`NOBS_Z` INT,<br />
`NOBS_W1` INT,<br />
`NOBS_W2` INT,<br />
`NOBS_W3` INT,<br />
`NOBS_W4` INT,<br />
`RCHISQ_G` FLOAT,<br />
`RCHISQ_R` FLOAT,<br />
`RCHISQ_Z` FLOAT,<br />
`RCHISQ_W1` FLOAT,<br />
`RCHISQ_W2` FLOAT,<br />
`RCHISQ_W3` FLOAT,<br />
`RCHISQ_W4` FLOAT,<br />
`FRACFLUX_G` FLOAT,<br />
`FRACFLUX_R` FLOAT,<br />
`FRACFLUX_Z` FLOAT,<br />
`FRACFLUX_W1` FLOAT,<br />
`FRACFLUX_W2` FLOAT,<br />
`FRACFLUX_W3` FLOAT,<br />
`FRACFLUX_W4` FLOAT,<br />
`FRACMASKED_G` FLOAT,<br />
`FRACMASKED_R` FLOAT,<br />
`FRACMASKED_Z` FLOAT,<br />
`FRACIN_G` FLOAT,<br />
`FRACIN_R` FLOAT,<br />
`FRACIN_Z` FLOAT,<br />
`ANYMASK_G` SMALLINT,<br />
`ANYMASK_R` SMALLINT,<br />
`ANYMASK_Z` SMALLINT,<br />
`ALLMASK_G` SMALLINT,<br />
`ALLMASK_R` SMALLINT,<br />
`ALLMASK_Z` SMALLINT,<br />
`WISEMASK_W1` TINYINT,<br />
`WISEMASK_W2` TINYINT,<br />
`PSFSIZE_G` FLOAT,<br />
`PSFSIZE_R` FLOAT,<br />
`PSFSIZE_Z` FLOAT,<br />
`PSFDEPTH_G` FLOAT,<br />
`PSFDEPTH_R` FLOAT,<br />
`PSFDEPTH_Z` FLOAT,<br />
`GALDEPTH_G` FLOAT,<br />
`GALDEPTH_R` FLOAT,<br />
`GALDEPTH_Z` FLOAT,<br />
`PSFDEPTH_W1` FLOAT,<br />
`PSFDEPTH_W2` FLOAT,<br />
`WISE_COADD_ID` STRING,<br />
`FRACDEV` FLOAT,<br />
`FRACDEV_IVAR` FLOAT,<br />
`SHAPEDEV_R` FLOAT,<br />
`SHAPEDEV_R_IVAR` FLOAT,<br />
`SHAPEDEV_E1` FLOAT,<br />
`SHAPEDEV_E1_IVAR` FLOAT,<br />
`SHAPEDEV_E2` FLOAT,<br />
`SHAPEDEV_E2_IVAR` FLOAT,<br />
`SHAPEEXP_R` FLOAT,<br />
`SHAPEEXP_R_IVAR` FLOAT,<br />
`SHAPEEXP_E1` FLOAT,<br />
`SHAPEEXP_E1_IVAR` FLOAT,<br />
`SHAPEEXP_E2` FLOAT,<br />
`SHAPEEXP_E2_IVAR` FLOAT,<br />
`FIBERFLUX_G` FLOAT,<br />
`FIBERFLUX_R` FLOAT,<br />
`FIBERFLUX_Z` FLOAT,<br />
`FIBERTOTFLUX_G` FLOAT,<br />
`FIBERTOTFLUX_R` FLOAT,<br />
`FIBERTOTFLUX_Z` FLOAT,<br />
`REF_CAT` STRING,<br />
`REF_ID` BIGINT,<br />
`REF_EPOCH` FLOAT,<br />
`GAIA_PHOT_G_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_G_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_ASTROMETRIC_EXCESS_NOISE` FLOAT,<br />
`GAIA_DUPLICATED_SOURCE` BOOLEAN,<br />
`GAIA_PHOT_BP_RP_EXCESS_FACTOR` FLOAT,<br />
`GAIA_ASTROMETRIC_SIGMA5D_MAX` FLOAT,<br />
`GAIA_ASTROMETRIC_PARAMS_SOLVED` TINYINT,<br />
`PARALLAX` FLOAT,<br />
`PARALLAX_IVAR` FLOAT,<br />
`PMRA` FLOAT,<br />
`PMRA_IVAR` FLOAT,<br />
`PMDEC` FLOAT,<br />
`PMDEC_IVAR` FLOAT,<br />
`MASKBITS` SMALLINT,<br />
`z_phot_mean` FLOAT,<br />
`z_phot_median` FLOAT,<br />
`z_phot_std` FLOAT,<br />
`z_phot_l68` FLOAT,<br />
`z_phot_u68` FLOAT,<br />
`z_phot_l95` FLOAT,<br />
`z_phot_u95` FLOAT,<br />
`z_spec` FLOAT,<br />
`survey` STRING,<br />
`training` BOOLEAN<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey-sweep'<br />
;<br />
<br />
=== Combined table ===<br />
<br />
CREATE TABLE cosmohub.legacysurvey_dr8_phz (<br />
release smallint COMMENT 'Unique integer denoting the camera and filter set used',<br />
brickid int COMMENT 'Brick ID [1,662174]',<br />
brickname string COMMENT 'Name of brick, encoding the brick sky position, eg "1126p222" near RA=112.6, Dec=+22.2',<br />
objid int COMMENT 'Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick',<br />
brick_primary boolean COMMENT 'True if the object is within the brick boundary',<br />
brightblob smallint COMMENT 'bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page',<br />
maskbits smallint COMMENT 'bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits* maps, as cataloged on the DR8 bitmasks page',<br />
type string COMMENT 'Morphological model: "PSF"=stellar, "REX"="round exponential galaxy", "DEV"=deVauc, "EXP"=exponential, "COMP"=composite, "DUP"=Gaia source fit by different model. Note that in some FITS readers, a trailing space may be appended for "PSF ", "DEV " and "EXP " since the column data type is a 4-character string',<br />
ra double COMMENT 'Right ascension at equinox J2000',<br />
dec double COMMENT 'Declination at equinox J2000',<br />
ra_ivar float COMMENT 'Inverse variance of RA (no cosine term!), excluding astrometric calibration errors',<br />
dec_ivar float COMMENT 'Inverse variance of DEC, excluding astrometric calibration errors',<br />
bx float COMMENT 'X position (0-indexed) of coordinates in brick image stack',<br />
`by` float COMMENT 'Y position (0-indexed) of coordinates in brick image stack',<br />
dchisq array<float> COMMENT 'Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.',<br />
ebv float COMMENT 'Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns',<br />
mjd_min double COMMENT 'Minimum Modified Julian Date of observations used to construct the model of this object',<br />
mjd_max double COMMENT 'Maximum Modified Julian Date of observations used to construct the model of this object',<br />
ref_cat string COMMENT 'Reference catalog source for this star: "T2" for Tycho-2, "G2" for Gaia DR2, "L2" for the LSLGA, empty otherwise',<br />
ref_id bigint COMMENT 'Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b "sourceid" for Gaia-DR2 and LSLGA',<br />
pmra float COMMENT 'Reference catalog proper motion in the RA direction',<br />
pmdec float COMMENT 'Reference catalog proper motion in the Dec direction',<br />
parallax float COMMENT 'Reference catalog parallax',<br />
pmra_ivar float COMMENT 'Reference catalog inverse-variance on pmra',<br />
pmdec_ivar float COMMENT 'Reference catalog inverse-variance on pmdec',<br />
parallax_ivar float COMMENT 'Reference catalog inverse-variance on parallax',<br />
ref_epoch float COMMENT 'Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)',<br />
gaia_pointsource boolean COMMENT 'This Gaia DR2 source is believed to be a star, not a galaxy',<br />
gaia_phot_g_mean_mag float COMMENT 'Gaia G band mag',<br />
gaia_phot_g_mean_flux_over_error float COMMENT 'Gaia G band signal-to-noise',<br />
gaia_phot_g_n_obs smallint COMMENT 'Gaia G band number of observations',<br />
gaia_phot_bp_mean_mag float COMMENT 'Gaia BP mag',<br />
gaia_phot_bp_mean_flux_over_error float COMMENT 'Gaia BP signal-to-noise',<br />
gaia_phot_bp_n_obs smallint COMMENT 'Gaia BP number of observations',<br />
gaia_phot_rp_mean_mag float COMMENT 'Gaia RP mag',<br />
gaia_phot_rp_mean_flux_over_error float COMMENT 'Gaia RP signal-to-noise',<br />
gaia_phot_rp_n_obs smallint COMMENT 'Gaia RP number of observations',<br />
gaia_phot_variable_flag boolean COMMENT 'Gaia photometric variable flag',<br />
gaia_astrometric_excess_noise float COMMENT 'Gaia astrometric excess noise',<br />
gaia_astrometric_excess_noise_sig float COMMENT 'Gaia astrometric excess noise uncertainty',<br />
gaia_astrometric_n_obs_al smallint COMMENT 'Gaia number of astrometric observations along scan direction',<br />
gaia_astrometric_n_good_obs_al smallint COMMENT 'Gaia number of good astrometric observations along scan direction',<br />
gaia_astrometric_weight_al float COMMENT 'Gaia astrometric weight along scan direction',<br />
gaia_duplicated_source boolean COMMENT 'Gaia duplicated source flag',<br />
gaia_a_g_val float COMMENT 'Gaia line-of-sight extinction in the G band',<br />
gaia_e_bp_min_rp_val float COMMENT 'Gaia line-of-sight reddening E(BP-RP)',<br />
gaia_phot_bp_rp_excess_factor float COMMENT 'Gaia BP/RP excess factor',<br />
gaia_astrometric_sigma5d_max float COMMENT 'Gaia longest semi-major axis of the 5-d error ellipsoid',<br />
gaia_astrometric_params_solved tinyint COMMENT 'which astrometric parameters were estimated for a Gaia source',<br />
flux_g float COMMENT 'model flux in gg',<br />
flux_r float COMMENT 'model flux in rr',<br />
flux_z float COMMENT 'model flux in zz',<br />
flux_w1 float COMMENT 'WISE model flux in W1W1 (AB system)',<br />
flux_w2 float COMMENT 'WISE model flux in W2W2 (AB)',<br />
flux_w3 float COMMENT 'WISE model flux in W3W3 (AB)',<br />
flux_w4 float COMMENT 'WISE model flux in W4W4 (AB)',<br />
flux_ivar_g float COMMENT 'Inverse variance of flux_g',<br />
flux_ivar_r float COMMENT 'Inverse variance of flux_r',<br />
flux_ivar_z float COMMENT 'Inverse variance of flux_z',<br />
flux_ivar_w1 float COMMENT 'Inverse variance of flux_w1 (AB system)',<br />
flux_ivar_w2 float COMMENT 'Inverse variance of flux_w2 (AB)',<br />
flux_ivar_w3 float COMMENT 'Inverse variance of flux_w3 (AB)',<br />
flux_ivar_w4 float COMMENT 'Inverse variance of flux_w4 (AB)',<br />
fiberflux_g float COMMENT 'Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_r float COMMENT 'Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_z float COMMENT 'Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fibertotflux_g float COMMENT 'Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_r float COMMENT 'Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_z float COMMENT 'Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
apflux_g array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg',<br />
apflux_r array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr',<br />
apflux_z array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz',<br />
apflux_resid_g array<float> COMMENT 'aperture fluxes on the co-added residual images in gg',<br />
apflux_resid_r array<float> COMMENT 'aperture fluxes on the co-added residual images in rr',<br />
apflux_resid_z array<float> COMMENT 'aperture fluxes on the co-added residual images in zz',<br />
apflux_ivar_g array<float> COMMENT 'Inverse variance of apflux_resid_g',<br />
apflux_ivar_r array<float> COMMENT 'Inverse variance of apflux_resid_r',<br />
apflux_ivar_z array<float> COMMENT 'Inverse variance of apflux_resid_z',<br />
mw_transmission_g float COMMENT 'Galactic transmission in gg filter in linear units [0, 1]',<br />
mw_transmission_r float COMMENT 'Galactic transmission in rr filter in linear units [0, 1]',<br />
mw_transmission_z float COMMENT 'Galactic transmission in zz filter in linear units [0, 1]',<br />
mw_transmission_w1 float COMMENT 'Galactic transmission in W1W1 filter in linear units [0, 1]',<br />
mw_transmission_w2 float COMMENT 'Galactic transmission in W2W2 filter in linear units [0, 1]',<br />
mw_transmission_w3 float COMMENT 'Galactic transmission in W3W3 filter in linear units [0, 1]',<br />
mw_transmission_w4 float COMMENT 'Galactic transmission in W4W4 filter in linear units [0, 1]',<br />
nobs_g smallint COMMENT 'Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)',<br />
nobs_r smallint COMMENT 'Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)',<br />
nobs_z smallint COMMENT 'Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)',<br />
nobs_w1 smallint COMMENT 'Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)',<br />
nobs_w2 smallint COMMENT 'Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)',<br />
nobs_w3 smallint COMMENT 'Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)',<br />
nobs_w4 smallint COMMENT 'Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)',<br />
rchisq_g float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in gg',<br />
rchisq_r float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in rr',<br />
rchisq_z float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in zz',<br />
rchisq_w1 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W1W1',<br />
rchisq_w2 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W2W2',<br />
rchisq_w3 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W3W3',<br />
rchisq_w4 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W4W4',<br />
fracflux_g float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in gg (typically [0,1])',<br />
fracflux_r float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in rr (typically [0,1])',<br />
fracflux_z float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in zz (typically [0,1])',<br />
fracflux_w1 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1 (typically [0,1])',<br />
fracflux_w2 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2 (typically [0,1])',<br />
fracflux_w3 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3 (typically [0,1])',<br />
fracflux_w4 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4 (typically [0,1])',<br />
fracmasked_g float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]',<br />
fracmasked_r float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]',<br />
fracmasked_z float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]',<br />
fracin_g float COMMENT 'Fraction of a source\'s flux within the blob in gg, near unity for real sources',<br />
fracin_r float COMMENT 'Fraction of a source\'s flux within the blob in rr, near unity for real sources',<br />
fracin_z float COMMENT 'Fraction of a source\'s flux within the blob in zz, near unity for real sources',<br />
anymask_g smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page',<br />
anymask_r smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page',<br />
anymask_z smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page',<br />
allmask_g smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page',<br />
allmask_r smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page',<br />
allmask_z smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page',<br />
wisemask_w1 tinyint COMMENT 'W1 bitmask as cataloged on the DR8 bitmasks page',<br />
wisemask_w2 tinyint COMMENT 'W2 bitmask as cataloged on the DR8 bitmasks page',<br />
psfsize_g float COMMENT 'Weighted average PSF FWHM in the gg band',<br />
psfsize_r float COMMENT 'Weighted average PSF FWHM in the rr band',<br />
psfsize_z float COMMENT 'Weighted average PSF FWHM in the zz band',<br />
psfdepth_g float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude',<br />
psfdepth_r float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude',<br />
psfdepth_z float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude',<br />
galdepth_g float COMMENT 'As for psfdepth_g but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_r float COMMENT 'As for psfdepth_r but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_z float COMMENT 'As for psfdepth_z but for a galaxy (0.45" exp, round) detection sensitivity',<br />
psfdepth_w1 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W1',<br />
psfdepth_w2 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W2',<br />
psfdepth_w3 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W3',<br />
psfdepth_w4 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W4',<br />
wise_coadd_id string COMMENT 'unWISE coadd file name for the center of each object',<br />
lc_flux_w1 array<float> COMMENT 'flux_w1 in each of up to eleven unWISE coadd epochs (AB system)',<br />
lc_flux_w2 array<float> COMMENT 'flux_w2 in each of up to eleven unWISE coadd epochs (AB)',<br />
lc_flux_ivar_w1 array<float> COMMENT 'Inverse variance of lc_flux_w1 (AB system)',<br />
lc_flux_ivar_w2 array<float> COMMENT 'Inverse variance of lc_flux_w2 (AB)',<br />
lc_nobs_w1 array<smallint> COMMENT 'nobs_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_nobs_w2 array<smallint> COMMENT 'nobs_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w1 array<float> COMMENT 'fracflux_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w2 array<float> COMMENT 'fracflux_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w1 array<float> COMMENT 'rchisq_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w2 array<float> COMMENT 'rchisq_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w1 array<double> COMMENT 'mjd_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w2 array<double> COMMENT 'mjd_w2 in each of up to eleven unWISE coadd epochs',<br />
fracdev float COMMENT 'Fraction of model in deVauc [0,1]',<br />
fracdev_ivar float COMMENT 'Inverse variance of fracdev',<br />
shapeexp_r float COMMENT 'Half-light radius of exponential model (>0)',<br />
shapeexp_r_ivar float COMMENT 'Inverse variance of shapeexp_r',<br />
shapeexp_e1 float COMMENT 'Ellipticity component 1',<br />
shapeexp_e1_ivar float COMMENT 'Inverse variance of shapeexp_e1',<br />
shapeexp_e2 float COMMENT 'Ellipticity component 2',<br />
shapeexp_e2_ivar float COMMENT 'Inverse variance of shapeexp_e2',<br />
shapedev_r float COMMENT 'Half-light radius of deVaucouleurs model (>0)',<br />
shapedev_r_ivar float COMMENT 'Inverse variance of shapedev_r',<br />
shapedev_e1 float COMMENT 'Ellipticity component 1',<br />
shapedev_e1_ivar float COMMENT 'Inverse variance of shapedev_e1',<br />
shapedev_e2 float COMMENT 'Ellipticity component 2',<br />
shapedev_e2_ivar float COMMENT 'Inverse variance of shapedev_e2',<br />
z_phot_mean FLOAT COMMENT 'photo-z derived from the mean of the photo-z PDF ',<br />
z_phot_median FLOAT COMMENT 'photo-z derived from the median of the photo-z PDF',<br />
z_phot_std FLOAT COMMENT 'standard deviation of the photo-z\'s derived from the photo-z PDF',<br />
z_phot_l68 FLOAT COMMENT 'lower bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_u68 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_l95 FLOAT COMMENT 'lower bound of the 95% confidence region, derived from the photo-z PDF',<br />
z_phot_u95 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_spec FLOAT COMMENT 'spectroscopic redshift, if available',<br />
survey STRING COMMENT 'source of the spectroscopic redshift',<br />
training BOOLEAN COMMENT 'whether or not the spectroscopic redshift is used in photometric redshift training'<br />
)<br />
CLUSTERED BY (<br />
release,<br />
brickid,<br />
objid)<br />
SORTED BY (<br />
release ASC,<br />
brickid ASC,<br />
objid ASC)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
=== Join insert ===<br />
<br />
INSERT OVERWRITE TABLE cosmohub.legacy_survey_dr8_phz<br />
SELECT<br />
dr8.*,<br />
phz.z_phot_mean,<br />
phz.z_phot_median,<br />
phz.z_phot_std,<br />
phz.z_phot_l68,<br />
phz.z_phot_u68,<br />
phz.z_phot_l95,<br />
phz.z_phot_u95,<br />
phz.z_spec,<br />
phz.survey,<br />
phz.training<br />
FROM cosmohub.legacy_survey_dr8 AS dr8<br />
LEFT JOIN tallada.dr8_sweep AS phz<br />
ON dr8.release = phz.release<br />
AND dr8.brickid = phz.brickid<br />
AND dr8.objid = phz.objid<br />
;<br />
<br />
== Queries for checking ==<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT release, brickid, objid, COUNT(release)<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
HAVING COUNT(release) > 1<br />
<br />
DR8 release brickid objid: DISTINCT(1.694.230.368) TOTAL(1.694.230.368)<br />
DR8 brickid objid: DISTINCT(1.654.429.115) TOTAL(1.694.230.368)<br />
<br />
PHZ release brickid objid: DISTINCT(1.646.664.729) TOTAL(1.646.664.897)<br />
PHZ brickid objid: DISTINCT(1.608.841.582) TOTAL(1.646.664.897)<br />
<br />
DR8: 1.694.230.368<br />
PHZ: 1.646.664.897<br />
----------------------------------------------------------------------<br />
-47.565.471</div>Talladahttps://pwiki.pic.es/index.php?title=Legacy_Survey_DR9&diff=876Legacy Survey DR92020-11-28T02:11:58Z<p>Tallada: /* Internal Table */</p>
<hr />
<div>== Original DR9 ==<br />
=== Raw data ===<br />
<br />
Downloaded from NERSC<br />
/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m<br />
<br />
Directories:<br />
south/tractor<br />
north/tractor<br />
<br />
Downloaded using bbcp from services01.euclid into CephFS scratch:<br />
bbcp -s 16 -a -A -v -z -r -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" \<br />
"jcarrete@dtn01.nersc.gov:/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m/north/tractor" \<br />
/cephfs/pic.es/astro/scratch/tallada/dr9m/north/tractor<br />
bbcp -s 16 -a -A -v -z -r -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" \<br />
"jcarrete@dtn01.nersc.gov:/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m/south/tractor" \<br />
/cephfs/pic.es/astro/scratch/tallada/dr9m/south/tractor<br />
<br />
Checksums provided through sha256sum files, and checked using command:<br />
find * -type d | xargs -t -L1 -P24 sh -c 'cd $1; sha256sum --quiet -c *.sha256sum' cksum<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to convert fits files to parquet:<br />
<br />
path_base = '/cephfs/pic.es/astro/scratch/tallada/dr9m'<br />
hdfs_base = '/user/tallada/data/legacysurvey_dr9'<br />
<br />
hs = ['h=north', 'h=south']<br />
rs = ['r={0:03d}'.format(x) for x in range(360)]<br />
<br />
for h in hs:<br />
for r in rs:<br />
p = os.path.join(path_base, h, r)<br />
if not os.path.exists(p):<br />
continue<br />
<br />
size = sum(os.path.getsize(os.path.join(p,f)) for f in os.listdir(p) if os.path.isfile(os.path.join(p,f)))<br />
<br />
print h, r, size <br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + p<br />
)<br />
<br />
df = df.repartition(int(math.ceil(size / (256*1024*1024.0))))<br />
<br />
df.write.parquet(os.path.join(hdfs_base, h, r))<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE `tallada`.`legacysurvey_dr9`(<br />
`release` smallint, <br />
`brickid` int, <br />
`brickname` string, <br />
`objid` int, <br />
`brick_primary` boolean, <br />
`maskbits` smallint, <br />
`fitbits` smallint, <br />
`type` string, <br />
`ra` double, <br />
`dec` double, <br />
`ra_ivar` float, <br />
`dec_ivar` float, <br />
`bx` float, <br />
`by` float, <br />
`dchisq` array<float>, <br />
`ebv` float, <br />
`mjd_min` double, <br />
`mjd_max` double, <br />
`ref_cat` string, <br />
`ref_id` bigint, <br />
`pmra` float, <br />
`pmdec` float, <br />
`parallax` float, <br />
`pmra_ivar` float, <br />
`pmdec_ivar` float, <br />
`parallax_ivar` float, <br />
`ref_epoch` float, <br />
`gaia_phot_g_mean_mag` float, <br />
`gaia_phot_g_mean_flux_over_error` float, <br />
`gaia_phot_g_n_obs` smallint, <br />
`gaia_phot_bp_mean_mag` float, <br />
`gaia_phot_bp_mean_flux_over_error` float, <br />
`gaia_phot_bp_n_obs` smallint, <br />
`gaia_phot_rp_mean_mag` float, <br />
`gaia_phot_rp_mean_flux_over_error` float, <br />
`gaia_phot_rp_n_obs` smallint, <br />
`gaia_phot_variable_flag` boolean, <br />
`gaia_astrometric_excess_noise` float, <br />
`gaia_astrometric_excess_noise_sig` float, <br />
`gaia_astrometric_n_obs_al` smallint, <br />
`gaia_astrometric_n_good_obs_al` smallint, <br />
`gaia_astrometric_weight_al` float, <br />
`gaia_duplicated_source` boolean, <br />
`gaia_a_g_val` float, <br />
`gaia_e_bp_min_rp_val` float, <br />
`gaia_phot_bp_rp_excess_factor` float, <br />
`gaia_astrometric_sigma5d_max` float, <br />
`gaia_astrometric_params_solved` tinyint, <br />
`flux_g` float, <br />
`flux_r` float, <br />
`flux_z` float, <br />
`flux_w1` float, <br />
`flux_w2` float, <br />
`flux_w3` float, <br />
`flux_w4` float, <br />
`flux_ivar_g` float, <br />
`flux_ivar_r` float, <br />
`flux_ivar_z` float, <br />
`flux_ivar_w1` float, <br />
`flux_ivar_w2` float, <br />
`flux_ivar_w3` float, <br />
`flux_ivar_w4` float, <br />
`fiberflux_g` float, <br />
`fiberflux_r` float, <br />
`fiberflux_z` float, <br />
`fibertotflux_g` float, <br />
`fibertotflux_r` float, <br />
`fibertotflux_z` float, <br />
`apflux_g` array<float>, <br />
`apflux_r` array<float>, <br />
`apflux_z` array<float>, <br />
`apflux_resid_g` array<float>, <br />
`apflux_resid_r` array<float>, <br />
`apflux_resid_z` array<float>, <br />
`apflux_blobresid_g` array<float>, <br />
`apflux_blobresid_r` array<float>, <br />
`apflux_blobresid_z` array<float>, <br />
`apflux_ivar_g` array<float>, <br />
`apflux_ivar_r` array<float>, <br />
`apflux_ivar_z` array<float>, <br />
`apflux_masked_g` array<float>, <br />
`apflux_masked_r` array<float>, <br />
`apflux_masked_z` array<float>, <br />
`apflux_w1` array<float>, <br />
`apflux_w2` array<float>, <br />
`apflux_w3` array<float>, <br />
`apflux_w4` array<float>, <br />
`apflux_resid_w1` array<float>, <br />
`apflux_resid_w2` array<float>, <br />
`apflux_resid_w3` array<float>, <br />
`apflux_resid_w4` array<float>, <br />
`apflux_ivar_w1` array<float>, <br />
`apflux_ivar_w2` array<float>, <br />
`apflux_ivar_w3` array<float>, <br />
`apflux_ivar_w4` array<float>, <br />
`mw_transmission_g` float, <br />
`mw_transmission_r` float, <br />
`mw_transmission_z` float, <br />
`mw_transmission_w1` float, <br />
`mw_transmission_w2` float, <br />
`mw_transmission_w3` float, <br />
`mw_transmission_w4` float, <br />
`nobs_g` smallint, <br />
`nobs_r` smallint, <br />
`nobs_z` smallint, <br />
`nobs_w1` smallint, <br />
`nobs_w2` smallint, <br />
`nobs_w3` smallint, <br />
`nobs_w4` smallint, <br />
`rchisq_g` float, <br />
`rchisq_r` float, <br />
`rchisq_z` float, <br />
`rchisq_w1` float, <br />
`rchisq_w2` float, <br />
`rchisq_w3` float, <br />
`rchisq_w4` float, <br />
`fracflux_g` float, <br />
`fracflux_r` float, <br />
`fracflux_z` float, <br />
`fracflux_w1` float, <br />
`fracflux_w2` float, <br />
`fracflux_w3` float, <br />
`fracflux_w4` float, <br />
`fracmasked_g` float, <br />
`fracmasked_r` float, <br />
`fracmasked_z` float, <br />
`fracin_g` float, <br />
`fracin_r` float, <br />
`fracin_z` float, <br />
`anymask_g` smallint, <br />
`anymask_r` smallint, <br />
`anymask_z` smallint, <br />
`allmask_g` smallint, <br />
`allmask_r` smallint, <br />
`allmask_z` smallint, <br />
`wisemask_w1` tinyint, <br />
`wisemask_w2` tinyint, <br />
`psfsize_g` float, <br />
`psfsize_r` float, <br />
`psfsize_z` float, <br />
`psfdepth_g` float, <br />
`psfdepth_r` float, <br />
`psfdepth_z` float, <br />
`galdepth_g` float, <br />
`galdepth_r` float, <br />
`galdepth_z` float, <br />
`nea_g` float, <br />
`nea_r` float, <br />
`nea_z` float, <br />
`blob_nea_g` float, <br />
`blob_nea_r` float, <br />
`blob_nea_z` float, <br />
`psfdepth_w1` float, <br />
`psfdepth_w2` float, <br />
`psfdepth_w3` float, <br />
`psfdepth_w4` float, <br />
`wise_coadd_id` string, <br />
`wise_x` float, <br />
`wise_y` float, <br />
`lc_flux_w1` array<float>, <br />
`lc_flux_w2` array<float>, <br />
`lc_flux_ivar_w1` array<float>, <br />
`lc_flux_ivar_w2` array<float>, <br />
`lc_nobs_w1` array<smallint>, <br />
`lc_nobs_w2` array<smallint>, <br />
`lc_fracflux_w1` array<float>, <br />
`lc_fracflux_w2` array<float>, <br />
`lc_rchisq_w1` array<float>, <br />
`lc_rchisq_w2` array<float>, <br />
`lc_mjd_w1` array<double>, <br />
`lc_mjd_w2` array<double>, <br />
`lc_epoch_index_w1` array<smallint>, <br />
`lc_epoch_index_w2` array<smallint>, <br />
`sersic` float, <br />
`sersic_ivar` float, <br />
`shape_r` float, <br />
`shape_r_ivar` float, <br />
`shape_e1` float, <br />
`shape_e1_ivar` float, <br />
`shape_e2` float, <br />
`shape_e2_ivar` float)<br />
PARTITIONED BY (<br />
h STRING,<br />
r STRING<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey_dr9'<br />
;<br />
<br />
=== Internal Table ===<br />
<br />
CREATE TABLE `cosmohub`.`legacy_survey_dr9`(<br />
`release` smallint,<br />
`brickid` int,<br />
`brickname` string,<br />
`objid` int,<br />
`brick_primary` boolean,<br />
`maskbits` smallint,<br />
`fitbits` smallint,<br />
`type` string,<br />
`ra` double,<br />
`dec` double,<br />
`ra_ivar` float,<br />
`dec_ivar` float,<br />
`bx` float,<br />
`by` float,<br />
`dchisq` array<float>,<br />
`ebv` float,<br />
`mjd_min` double,<br />
`mjd_max` double,<br />
`ref_cat` string,<br />
`ref_id` bigint,<br />
`pmra` float,<br />
`pmdec` float,<br />
`parallax` float,<br />
`pmra_ivar` float,<br />
`pmdec_ivar` float,<br />
`parallax_ivar` float,<br />
`ref_epoch` float,<br />
`gaia_phot_g_mean_mag` float,<br />
`gaia_phot_g_mean_flux_over_error` float,<br />
`gaia_phot_g_n_obs` smallint,<br />
`gaia_phot_bp_mean_mag` float,<br />
`gaia_phot_bp_mean_flux_over_error` float,<br />
`gaia_phot_bp_n_obs` smallint,<br />
`gaia_phot_rp_mean_mag` float,<br />
`gaia_phot_rp_mean_flux_over_error` float,<br />
`gaia_phot_rp_n_obs` smallint,<br />
`gaia_phot_variable_flag` boolean,<br />
`gaia_astrometric_excess_noise` float,<br />
`gaia_astrometric_excess_noise_sig` float,<br />
`gaia_astrometric_n_obs_al` smallint,<br />
`gaia_astrometric_n_good_obs_al` smallint,<br />
`gaia_astrometric_weight_al` float,<br />
`gaia_duplicated_source` boolean,<br />
`gaia_a_g_val` float,<br />
`gaia_e_bp_min_rp_val` float,<br />
`gaia_phot_bp_rp_excess_factor` float,<br />
`gaia_astrometric_sigma5d_max` float,<br />
`gaia_astrometric_params_solved` tinyint,<br />
`flux_g` float,<br />
`flux_r` float,<br />
`flux_z` float,<br />
`flux_w1` float,<br />
`flux_w2` float,<br />
`flux_w3` float,<br />
`flux_w4` float,<br />
`flux_ivar_g` float,<br />
`flux_ivar_r` float,<br />
`flux_ivar_z` float,<br />
`flux_ivar_w1` float,<br />
`flux_ivar_w2` float,<br />
`flux_ivar_w3` float,<br />
`flux_ivar_w4` float,<br />
`fiberflux_g` float,<br />
`fiberflux_r` float,<br />
`fiberflux_z` float,<br />
`fibertotflux_g` float,<br />
`fibertotflux_r` float,<br />
`fibertotflux_z` float,<br />
`apflux_g` array<float>,<br />
`apflux_r` array<float>,<br />
`apflux_z` array<float>,<br />
`apflux_resid_g` array<float>,<br />
`apflux_resid_r` array<float>,<br />
`apflux_resid_z` array<float>,<br />
`apflux_blobresid_g` array<float>,<br />
`apflux_blobresid_r` array<float>,<br />
`apflux_blobresid_z` array<float>,<br />
`apflux_ivar_g` array<float>,<br />
`apflux_ivar_r` array<float>,<br />
`apflux_ivar_z` array<float>,<br />
`apflux_masked_g` array<float>,<br />
`apflux_masked_r` array<float>,<br />
`apflux_masked_z` array<float>,<br />
`apflux_w1` array<float>,<br />
`apflux_w2` array<float>,<br />
`apflux_w3` array<float>,<br />
`apflux_w4` array<float>,<br />
`apflux_resid_w1` array<float>,<br />
`apflux_resid_w2` array<float>,<br />
`apflux_resid_w3` array<float>,<br />
`apflux_resid_w4` array<float>,<br />
`apflux_ivar_w1` array<float>,<br />
`apflux_ivar_w2` array<float>,<br />
`apflux_ivar_w3` array<float>,<br />
`apflux_ivar_w4` array<float>,<br />
`mw_transmission_g` float,<br />
`mw_transmission_r` float,<br />
`mw_transmission_z` float,<br />
`mw_transmission_w1` float,<br />
`mw_transmission_w2` float,<br />
`mw_transmission_w3` float,<br />
`mw_transmission_w4` float,<br />
`nobs_g` smallint,<br />
`nobs_r` smallint,<br />
`nobs_z` smallint,<br />
`nobs_w1` smallint,<br />
`nobs_w2` smallint,<br />
`nobs_w3` smallint,<br />
`nobs_w4` smallint,<br />
`rchisq_g` float,<br />
`rchisq_r` float,<br />
`rchisq_z` float,<br />
`rchisq_w1` float,<br />
`rchisq_w2` float,<br />
`rchisq_w3` float,<br />
`rchisq_w4` float,<br />
`fracflux_g` float,<br />
`fracflux_r` float,<br />
`fracflux_z` float,<br />
`fracflux_w1` float,<br />
`fracflux_w2` float,<br />
`fracflux_w3` float,<br />
`fracflux_w4` float,<br />
`fracmasked_g` float,<br />
`fracmasked_r` float,<br />
`fracmasked_z` float,<br />
`fracin_g` float,<br />
`fracin_r` float,<br />
`fracin_z` float,<br />
`anymask_g` smallint,<br />
`anymask_r` smallint,<br />
`anymask_z` smallint,<br />
`allmask_g` smallint,<br />
`allmask_r` smallint,<br />
`allmask_z` smallint,<br />
`wisemask_w1` tinyint,<br />
`wisemask_w2` tinyint,<br />
`psfsize_g` float,<br />
`psfsize_r` float,<br />
`psfsize_z` float,<br />
`psfdepth_g` float,<br />
`psfdepth_r` float,<br />
`psfdepth_z` float,<br />
`galdepth_g` float,<br />
`galdepth_r` float,<br />
`galdepth_z` float,<br />
`nea_g` float,<br />
`nea_r` float,<br />
`nea_z` float,<br />
`blob_nea_g` float,<br />
`blob_nea_r` float,<br />
`blob_nea_z` float,<br />
`psfdepth_w1` float,<br />
`psfdepth_w2` float,<br />
`psfdepth_w3` float,<br />
`psfdepth_w4` float,<br />
`wise_coadd_id` string,<br />
`wise_x` float,<br />
`wise_y` float,<br />
`lc_flux_w1` array<float>,<br />
`lc_flux_w2` array<float>,<br />
`lc_flux_ivar_w1` array<float>,<br />
`lc_flux_ivar_w2` array<float>,<br />
`lc_nobs_w1` array<smallint>,<br />
`lc_nobs_w2` array<smallint>,<br />
`lc_fracflux_w1` array<float>,<br />
`lc_fracflux_w2` array<float>,<br />
`lc_rchisq_w1` array<float>,<br />
`lc_rchisq_w2` array<float>,<br />
`lc_mjd_w1` array<double>,<br />
`lc_mjd_w2` array<double>,<br />
`lc_epoch_index_w1` array<smallint>,<br />
`lc_epoch_index_w2` array<smallint>,<br />
`sersic` float,<br />
`sersic_ivar` float,<br />
`shape_r` float,<br />
`shape_r_ivar` float,<br />
`shape_e1` float,<br />
`shape_e1_ivar` float,<br />
`shape_e2` float,<br />
`shape_e2_ivar` float)<br />
CLUSTERED BY (<br />
release, brickid, objid<br />
)<br />
SORTED BY (<br />
release, brickid, objid<br />
)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
<br />
INSERT INTO TABLE cosmohub.legacy_survey_dr9<br />
SELECT release, brickid, brickname, objid, brick_primary, maskbits, fitbits, type, ra, `dec`, ra_ivar, dec_ivar,<br />
bx, `by`, dchisq, ebv, mjd_min, mjd_max, ref_cat, ref_id, pmra, pmdec, parallax, pmra_ivar, pmdec_ivar,<br />
parallax_ivar, ref_epoch, gaia_phot_g_mean_mag, gaia_phot_g_mean_flux_over_error, gaia_phot_g_n_obs,<br />
gaia_phot_bp_mean_mag, gaia_phot_bp_mean_flux_over_error, gaia_phot_bp_n_obs, gaia_phot_rp_mean_mag,<br />
gaia_phot_rp_mean_flux_over_error, gaia_phot_rp_n_obs, gaia_phot_variable_flag, gaia_astrometric_excess_noise,<br />
gaia_astrometric_excess_noise_sig, gaia_astrometric_n_obs_al, gaia_astrometric_n_good_obs_al,<br />
gaia_astrometric_weight_al, gaia_duplicated_source, gaia_a_g_val, gaia_e_bp_min_rp_val,<br />
gaia_phot_bp_rp_excess_factor, gaia_astrometric_sigma5d_max, gaia_astrometric_params_solved, flux_g, flux_r,<br />
flux_z, flux_w1, flux_w2, flux_w3, flux_w4, flux_ivar_g, flux_ivar_r, flux_ivar_z, flux_ivar_w1, flux_ivar_w2,<br />
flux_ivar_w3, flux_ivar_w4, fiberflux_g, fiberflux_r, fiberflux_z, fibertotflux_g, fibertotflux_r, fibertotflux_z,<br />
apflux_g, apflux_r, apflux_z, apflux_resid_g, apflux_resid_r, apflux_resid_z, apflux_blobresid_g,<br />
apflux_blobresid_r, apflux_blobresid_z, apflux_ivar_g, apflux_ivar_r, apflux_ivar_z, apflux_masked_g,<br />
apflux_masked_r, apflux_masked_z, apflux_w1, apflux_w2, apflux_w3, apflux_w4, apflux_resid_w1, apflux_resid_w2,<br />
apflux_resid_w3, apflux_resid_w4, apflux_ivar_w1, apflux_ivar_w2, apflux_ivar_w3, apflux_ivar_w4,<br />
mw_transmission_g, mw_transmission_r, mw_transmission_z, mw_transmission_w1, mw_transmission_w2,<br />
mw_transmission_w3, mw_transmission_w4, nobs_g, nobs_r, nobs_z, nobs_w1, nobs_w2, nobs_w3, nobs_w4, rchisq_g,<br />
rchisq_r, rchisq_z, rchisq_w1, rchisq_w2, rchisq_w3, rchisq_w4, fracflux_g, fracflux_r, fracflux_z, fracflux_w1,<br />
fracflux_w2, fracflux_w3, fracflux_w4, fracmasked_g, fracmasked_r, fracmasked_z, fracin_g, fracin_r, fracin_z,<br />
anymask_g, anymask_r, anymask_z, allmask_g, allmask_r, allmask_z, wisemask_w1, wisemask_w2, psfsize_g, psfsize_r,<br />
psfsize_z, psfdepth_g, psfdepth_r, psfdepth_z, galdepth_g, galdepth_r, galdepth_z, nea_g, nea_r, nea_z,<br />
blob_nea_g, blob_nea_r, blob_nea_z, psfdepth_w1, psfdepth_w2, psfdepth_w3, psfdepth_w4, wise_coadd_id, wise_x,<br />
wise_y, lc_flux_w1, lc_flux_w2, lc_flux_ivar_w1, lc_flux_ivar_w2, lc_nobs_w1, lc_nobs_w2, lc_fracflux_w1,<br />
lc_fracflux_w2, lc_rchisq_w1, lc_rchisq_w2, lc_mjd_w1, lc_mjd_w2, lc_epoch_index_w1, lc_epoch_index_w2, sersic,<br />
sersic_ivar, shape_r, shape_r_ivar, shape_e1, shape_e1_ivar, shape_e2, shape_e2_ivar<br />
FROM cosmohub.legacy_survey_dr9_p<br />
;<br />
<br />
=== Comments ===<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE release release smallint COMMENT "Unique integer denoting the camera and filter set used";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickid brickid int COMMENT "Brick ID [1,662174]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickname brickname string COMMENT "Name of brick, encoding the brick sky position, eg \"1126p222\" near RA=112.6, Dec=+22.2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE objid objid int COMMENT "Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brick_primary brick_primary boolean COMMENT "True if the object is within the brick boundary";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brightblob brightblob smallint COMMENT "bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE maskbits maskbits smallint COMMENT "bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits*maps, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE type type string COMMENT "Morphological model: PSF=stellar, REX=round exponential galaxy, DEV=deVauc, EXP=exponential, COMP=composite, DUP=Gaia source fit by different model.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra ra double COMMENT "Right ascension at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec dec double COMMENT "Declination at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra_ivar ra_ivar float COMMENT "Inverse variance of RA (no cosine term!), excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec_ivar dec_ivar float COMMENT "Inverse variance of DEC, excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE bx bx float COMMENT "X position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE by by float COMMENT "Y position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dchisq dchisq array<float> COMMENT "Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ebv ebv float COMMENT "Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_min mjd_min double COMMENT "Minimum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_max mjd_max double COMMENT "Maximum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_cat ref_cat string COMMENT "Reference catalog source for this star: T2 for Tycho-2, G2 for Gaia DR2, L2 for the LSLGA, empty otherwise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_id ref_id bigint COMMENT "Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b \"sourceid\" for Gaia-DR2 and LSLGA";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra pmra float COMMENT "Reference catalog proper motion in the RA direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec pmdec float COMMENT "Reference catalog proper motion in the Dec direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax parallax float COMMENT "Reference catalog parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra_ivar pmra_ivar float COMMENT "Reference catalog inverse-variance on pmra";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec_ivar pmdec_ivar float COMMENT "Reference catalog inverse-variance on pmdec";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax_ivar parallax_ivar float COMMENT "Reference catalog inverse-variance on parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_epoch ref_epoch float COMMENT "Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_pointsource gaia_pointsource boolean COMMENT "This Gaia DR2 source is believed to be a star, not a galaxy";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_mag gaia_phot_g_mean_mag float COMMENT "Gaia G band mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_flux_over_error gaia_phot_g_mean_flux_over_error float COMMENT "Gaia G band signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_n_obs gaia_phot_g_n_obs smallint COMMENT "Gaia G band number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_mag gaia_phot_bp_mean_mag float COMMENT "Gaia BP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_flux_over_error gaia_phot_bp_mean_flux_over_error float COMMENT "Gaia BP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_n_obs gaia_phot_bp_n_obs smallint COMMENT "Gaia BP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_mag gaia_phot_rp_mean_mag float COMMENT "Gaia RP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_flux_over_error gaia_phot_rp_mean_flux_over_error float COMMENT "Gaia RP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_n_obs gaia_phot_rp_n_obs smallint COMMENT "Gaia RP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_variable_flag gaia_phot_variable_flag boolean COMMENT "Gaia photometric variable flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise gaia_astrometric_excess_noise float COMMENT "Gaia astrometric excess noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise_sig gaia_astrometric_excess_noise_sig float COMMENT "Gaia astrometric excess noise uncertainty";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_obs_al gaia_astrometric_n_obs_al smallint COMMENT "Gaia number of astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_good_obs_al gaia_astrometric_n_good_obs_al smallint COMMENT "Gaia number of good astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_weight_al gaia_astrometric_weight_al float COMMENT "Gaia astrometric weight along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_duplicated_source gaia_duplicated_source boolean COMMENT "Gaia duplicated source flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_a_g_val gaia_a_g_val float COMMENT "Gaia line-of-sight extinction in the G band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_e_bp_min_rp_val gaia_e_bp_min_rp_val float COMMENT "Gaia line-of-sight reddening E(BP-RP)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_rp_excess_factor gaia_phot_bp_rp_excess_factor float COMMENT "Gaia BP/RP excess factor";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_sigma5d_max gaia_astrometric_sigma5d_max float COMMENT "Gaia longest semi-major axis of the 5-d error ellipsoid";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_params_solved gaia_astrometric_params_solved tinyint COMMENT "which astrometric parameters were estimated for a Gaia source";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_g flux_g float COMMENT "model flux in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_r flux_r float COMMENT "model flux in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_z flux_z float COMMENT "model flux in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w1 flux_w1 float COMMENT "WISE model flux in W1W1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w2 flux_w2 float COMMENT "WISE model flux in W2W2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w3 flux_w3 float COMMENT "WISE model flux in W3W3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w4 flux_w4 float COMMENT "WISE model flux in W4W4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_g flux_ivar_g float COMMENT "Inverse variance of flux_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_r flux_ivar_r float COMMENT "Inverse variance of flux_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_z flux_ivar_z float COMMENT "Inverse variance of flux_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w1 flux_ivar_w1 float COMMENT "Inverse variance of flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w2 flux_ivar_w2 float COMMENT "Inverse variance of flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w3 flux_ivar_w3 float COMMENT "Inverse variance of flux_w3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w4 flux_ivar_w4 float COMMENT "Inverse variance of flux_w4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_g fiberflux_g float COMMENT "Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_r fiberflux_r float COMMENT "Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_z fiberflux_z float COMMENT "Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_g fibertotflux_g float COMMENT "Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_r fibertotflux_r float COMMENT "Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_z fibertotflux_z float COMMENT "Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_g apflux_g array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_r apflux_r array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_z apflux_z array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_g apflux_resid_g array<float> COMMENT "aperture fluxes on the co-added residual images in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_r apflux_resid_r array<float> COMMENT "aperture fluxes on the co-added residual images in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_z apflux_resid_z array<float> COMMENT "aperture fluxes on the co-added residual images in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_g apflux_ivar_g array<float> COMMENT "Inverse variance of apflux_resid_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_r apflux_ivar_r array<float> COMMENT "Inverse variance of apflux_resid_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_z apflux_ivar_z array<float> COMMENT "Inverse variance of apflux_resid_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_g mw_transmission_g float COMMENT "Galactic transmission in gg filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_r mw_transmission_r float COMMENT "Galactic transmission in rr filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_z mw_transmission_z float COMMENT "Galactic transmission in zz filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w1 mw_transmission_w1 float COMMENT "Galactic transmission in W1W1 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w2 mw_transmission_w2 float COMMENT "Galactic transmission in W2W2 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w3 mw_transmission_w3 float COMMENT "Galactic transmission in W3W3 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w4 mw_transmission_w4 float COMMENT "Galactic transmission in W4W4 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_g nobs_g smallint COMMENT "Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_r nobs_r smallint COMMENT "Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_z nobs_z smallint COMMENT "Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w1 nobs_w1 smallint COMMENT "Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w2 nobs_w2 smallint COMMENT "Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w3 nobs_w3 smallint COMMENT "Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w4 nobs_w4 smallint COMMENT "Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_g rchisq_g float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_r rchisq_r float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_z rchisq_z float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w1 rchisq_w1 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W1W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w2 rchisq_w2 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W2W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w3 rchisq_w3 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W3W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w4 rchisq_w4 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W4W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_g fracflux_g float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in gg(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_r fracflux_r float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in rr(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_z fracflux_z float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in zz(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w1 fracflux_w1 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w2 fracflux_w2 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w3 fracflux_w3 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w4 fracflux_w4 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_g fracmasked_g float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_r fracmasked_r float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_z fracmasked_z float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_g fracin_g float COMMENT "Fraction of a source's flux within the blob in gg, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_r fracin_r float COMMENT "Fraction of a source's flux within the blob in rr, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_z fracin_z float COMMENT "Fraction of a source's flux within the blob in zz, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_g anymask_g smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_r anymask_r smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_z anymask_z smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_g allmask_g smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_r allmask_r smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_z allmask_z smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w1 wisemask_w1 tinyint COMMENT "W1 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w2 wisemask_w2 tinyint COMMENT "W2 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_g psfsize_g float COMMENT "Weighted average PSF FWHM in the gg band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_r psfsize_r float COMMENT "Weighted average PSF FWHM in the rr band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_z psfsize_z float COMMENT "Weighted average PSF FWHM in the zz band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_g psfdepth_g float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_r psfdepth_r float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_z psfdepth_z float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_g galdepth_g float COMMENT "As for psfdepth_g but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_r galdepth_r float COMMENT "As for psfdepth_r but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_z galdepth_z float COMMENT "As for psfdepth_z but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w1 psfdepth_w1 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w2 psfdepth_w2 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w3 psfdepth_w3 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w4 psfdepth_w4 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wise_coadd_id wise_coadd_id string COMMENT "unWISE coadd file name for the center of each object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w1 lc_flux_w1 array<float> COMMENT "flux_w1 in each of up to eleven unWISE coadd epochs (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w2 lc_flux_w2 array<float> COMMENT "flux_w2 in each of up to eleven unWISE coadd epochs (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w1 lc_flux_ivar_w1 array<float> COMMENT "Inverse variance of lc_flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w2 lc_flux_ivar_w2 array<float> COMMENT "Inverse variance of lc_flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w1 lc_nobs_w1 array<smallint> COMMENT "nobs_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w2 lc_nobs_w2 array<smallint> COMMENT "nobs_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w1 lc_fracflux_w1 array<float> COMMENT "fracflux_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w2 lc_fracflux_w2 array<float> COMMENT "fracflux_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w1 lc_rchisq_w1 array<float> COMMENT "rchisq_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w2 lc_rchisq_w2 array<float> COMMENT "rchisq_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w1 lc_mjd_w1 array<double> COMMENT "mjd_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w2 lc_mjd_w2 array<double> COMMENT "mjd_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev fracdev float COMMENT "Fraction of model in deVauc [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev_ivar fracdev_ivar float COMMENT "Inverse variance of fracdev";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r shapeexp_r float COMMENT "Half-light radius of exponential model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r_ivar shapeexp_r_ivar float COMMENT "Inverse variance of shapeexp_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1 shapeexp_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1_ivar shapeexp_e1_ivar float COMMENT "Inverse variance of shapeexp_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2 shapeexp_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2_ivar shapeexp_e2_ivar float COMMENT "Inverse variance of shapeexp_e2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r shapedev_r float COMMENT "Half-light radius of deVaucouleurs model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r_ivar shapedev_r_ivar float COMMENT "Inverse variance of shapedev_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1 shapedev_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1_ivar shapedev_e1_ivar float COMMENT "Inverse variance of shapedev_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2 shapedev_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2_ivar shapedev_e2_ivar float COMMENT "Inverse variance of shapedev_e2";<br />
<br />
== Sweep photo-z ==<br />
<br />
=== Raw data ===<br />
<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/north/sweep/<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/south/sweep/<br />
<br />
Checksums checked:<br />
<br />
$ cd north/sweep/8.0<br />
$ ls | wc -l<br />
287<br />
$ wc -l legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
286 (+ checksum file)<br />
$ sha256sum -c legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
OK<br />
<br />
$ cd north/sweep/8.0-photo-z<br />
$ ls | wc -l<br />
287<br />
$ wc -l legacysurvey_dr8_north_sweep_8.0-photo-z.sha256sum<br />
286 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_north_sweep_8.0-photo-z.sha256sum<br />
OK<br />
<br />
$ cd south/sweep/8.0<br />
$ ls | wc -l<br />
438<br />
$ wc -l legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
437 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
OK<br />
<br />
$ cd south/sweep/8.0-photo-z<br />
$ ls | wc -l<br />
438<br />
$ wc -l legacysurvey_dr8_south_sweep_8.0-photo-z.sha256sum<br />
437 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_south_sweep_8.0-photo-z.sha256sum<br />
OK<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to merge fits files and convert to parquet:<br />
<br />
path_base = '/cephfs/pic.es/astro/scratch/tallada/dr8-phz/'<br />
hdfs_base = '/user/tallada/data/legacysurvey-sweep'<br />
<br />
hs = ['south/sweep'] #, 'north/sweep']<br />
dir_src = '8.0'<br />
dir_phz = '8.0-photo-z'<br />
dir_dst = '8.0-merged'<br />
<br />
my_files = sorted(os.listdir(os.path.join(path_base, hs[0], dir_src)))<br />
<br />
def merge_phz(file_name):<br />
file_src = os.path.join(path_base, hs[0], dir_src, file_name)<br />
file_phz = os.path.join(path_base, hs[0], dir_phz, file_name[:-5]+'-pz.fits')<br />
file_dst = os.path.join(path_base, hs[0], dir_dst, file_name[:-5]+'-merged.fits')<br />
<br />
t_src = table.Table.read(file_src)<br />
t_phz = table.Table.read(file_phz)<br />
t_dst = table.hstack([t_src, t_phz], join_type='exact')<br />
<br />
os.unlink(file_dst)<br />
t_dst.write(file_dst)<br />
<br />
return file_name<br />
<br />
df = sc.parallelize(my_files, len(my_files))<br />
<br />
df.map(merge_phz).count()<br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + os.path.join(path_base, hs[0], dir_dst)<br />
) <br />
<br />
df.write.parquet(os.path.join(hdfs_base, hs[0]))<br />
<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE tallada.dr8_sweep (<br />
`RELEASE` SMALLINT,<br />
`BRICKID` INT,<br />
`BRICKNAME` STRING,<br />
`OBJID` INT,<br />
`TYPE` STRING,<br />
`RA` DOUBLE,<br />
`DEC` DOUBLE,<br />
`RA_IVAR` FLOAT,<br />
`DEC_IVAR` FLOAT,<br />
`DCHISQ` ARRAY<FLOAT>,<br />
`EBV` FLOAT,<br />
`FLUX_G` FLOAT,<br />
`FLUX_R` FLOAT,<br />
`FLUX_Z` FLOAT,<br />
`FLUX_W1` FLOAT,<br />
`FLUX_W2` FLOAT,<br />
`FLUX_W3` FLOAT,<br />
`FLUX_W4` FLOAT,<br />
`FLUX_IVAR_G` FLOAT,<br />
`FLUX_IVAR_R` FLOAT,<br />
`FLUX_IVAR_Z` FLOAT,<br />
`FLUX_IVAR_W1` FLOAT,<br />
`FLUX_IVAR_W2` FLOAT,<br />
`FLUX_IVAR_W3` FLOAT,<br />
`FLUX_IVAR_W4` FLOAT,<br />
`MW_TRANSMISSION_G` FLOAT,<br />
`MW_TRANSMISSION_R` FLOAT,<br />
`MW_TRANSMISSION_Z` FLOAT,<br />
`MW_TRANSMISSION_W1` FLOAT,<br />
`MW_TRANSMISSION_W2` FLOAT,<br />
`MW_TRANSMISSION_W3` FLOAT,<br />
`MW_TRANSMISSION_W4` FLOAT,<br />
`NOBS_G` INT,<br />
`NOBS_R` INT,<br />
`NOBS_Z` INT,<br />
`NOBS_W1` INT,<br />
`NOBS_W2` INT,<br />
`NOBS_W3` INT,<br />
`NOBS_W4` INT,<br />
`RCHISQ_G` FLOAT,<br />
`RCHISQ_R` FLOAT,<br />
`RCHISQ_Z` FLOAT,<br />
`RCHISQ_W1` FLOAT,<br />
`RCHISQ_W2` FLOAT,<br />
`RCHISQ_W3` FLOAT,<br />
`RCHISQ_W4` FLOAT,<br />
`FRACFLUX_G` FLOAT,<br />
`FRACFLUX_R` FLOAT,<br />
`FRACFLUX_Z` FLOAT,<br />
`FRACFLUX_W1` FLOAT,<br />
`FRACFLUX_W2` FLOAT,<br />
`FRACFLUX_W3` FLOAT,<br />
`FRACFLUX_W4` FLOAT,<br />
`FRACMASKED_G` FLOAT,<br />
`FRACMASKED_R` FLOAT,<br />
`FRACMASKED_Z` FLOAT,<br />
`FRACIN_G` FLOAT,<br />
`FRACIN_R` FLOAT,<br />
`FRACIN_Z` FLOAT,<br />
`ANYMASK_G` SMALLINT,<br />
`ANYMASK_R` SMALLINT,<br />
`ANYMASK_Z` SMALLINT,<br />
`ALLMASK_G` SMALLINT,<br />
`ALLMASK_R` SMALLINT,<br />
`ALLMASK_Z` SMALLINT,<br />
`WISEMASK_W1` TINYINT,<br />
`WISEMASK_W2` TINYINT,<br />
`PSFSIZE_G` FLOAT,<br />
`PSFSIZE_R` FLOAT,<br />
`PSFSIZE_Z` FLOAT,<br />
`PSFDEPTH_G` FLOAT,<br />
`PSFDEPTH_R` FLOAT,<br />
`PSFDEPTH_Z` FLOAT,<br />
`GALDEPTH_G` FLOAT,<br />
`GALDEPTH_R` FLOAT,<br />
`GALDEPTH_Z` FLOAT,<br />
`PSFDEPTH_W1` FLOAT,<br />
`PSFDEPTH_W2` FLOAT,<br />
`WISE_COADD_ID` STRING,<br />
`FRACDEV` FLOAT,<br />
`FRACDEV_IVAR` FLOAT,<br />
`SHAPEDEV_R` FLOAT,<br />
`SHAPEDEV_R_IVAR` FLOAT,<br />
`SHAPEDEV_E1` FLOAT,<br />
`SHAPEDEV_E1_IVAR` FLOAT,<br />
`SHAPEDEV_E2` FLOAT,<br />
`SHAPEDEV_E2_IVAR` FLOAT,<br />
`SHAPEEXP_R` FLOAT,<br />
`SHAPEEXP_R_IVAR` FLOAT,<br />
`SHAPEEXP_E1` FLOAT,<br />
`SHAPEEXP_E1_IVAR` FLOAT,<br />
`SHAPEEXP_E2` FLOAT,<br />
`SHAPEEXP_E2_IVAR` FLOAT,<br />
`FIBERFLUX_G` FLOAT,<br />
`FIBERFLUX_R` FLOAT,<br />
`FIBERFLUX_Z` FLOAT,<br />
`FIBERTOTFLUX_G` FLOAT,<br />
`FIBERTOTFLUX_R` FLOAT,<br />
`FIBERTOTFLUX_Z` FLOAT,<br />
`REF_CAT` STRING,<br />
`REF_ID` BIGINT,<br />
`REF_EPOCH` FLOAT,<br />
`GAIA_PHOT_G_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_G_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_ASTROMETRIC_EXCESS_NOISE` FLOAT,<br />
`GAIA_DUPLICATED_SOURCE` BOOLEAN,<br />
`GAIA_PHOT_BP_RP_EXCESS_FACTOR` FLOAT,<br />
`GAIA_ASTROMETRIC_SIGMA5D_MAX` FLOAT,<br />
`GAIA_ASTROMETRIC_PARAMS_SOLVED` TINYINT,<br />
`PARALLAX` FLOAT,<br />
`PARALLAX_IVAR` FLOAT,<br />
`PMRA` FLOAT,<br />
`PMRA_IVAR` FLOAT,<br />
`PMDEC` FLOAT,<br />
`PMDEC_IVAR` FLOAT,<br />
`MASKBITS` SMALLINT,<br />
`z_phot_mean` FLOAT,<br />
`z_phot_median` FLOAT,<br />
`z_phot_std` FLOAT,<br />
`z_phot_l68` FLOAT,<br />
`z_phot_u68` FLOAT,<br />
`z_phot_l95` FLOAT,<br />
`z_phot_u95` FLOAT,<br />
`z_spec` FLOAT,<br />
`survey` STRING,<br />
`training` BOOLEAN<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey-sweep'<br />
;<br />
<br />
=== Combined table ===<br />
<br />
CREATE TABLE cosmohub.legacysurvey_dr8_phz (<br />
release smallint COMMENT 'Unique integer denoting the camera and filter set used',<br />
brickid int COMMENT 'Brick ID [1,662174]',<br />
brickname string COMMENT 'Name of brick, encoding the brick sky position, eg "1126p222" near RA=112.6, Dec=+22.2',<br />
objid int COMMENT 'Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick',<br />
brick_primary boolean COMMENT 'True if the object is within the brick boundary',<br />
brightblob smallint COMMENT 'bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page',<br />
maskbits smallint COMMENT 'bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits* maps, as cataloged on the DR8 bitmasks page',<br />
type string COMMENT 'Morphological model: "PSF"=stellar, "REX"="round exponential galaxy", "DEV"=deVauc, "EXP"=exponential, "COMP"=composite, "DUP"=Gaia source fit by different model. Note that in some FITS readers, a trailing space may be appended for "PSF ", "DEV " and "EXP " since the column data type is a 4-character string',<br />
ra double COMMENT 'Right ascension at equinox J2000',<br />
dec double COMMENT 'Declination at equinox J2000',<br />
ra_ivar float COMMENT 'Inverse variance of RA (no cosine term!), excluding astrometric calibration errors',<br />
dec_ivar float COMMENT 'Inverse variance of DEC, excluding astrometric calibration errors',<br />
bx float COMMENT 'X position (0-indexed) of coordinates in brick image stack',<br />
`by` float COMMENT 'Y position (0-indexed) of coordinates in brick image stack',<br />
dchisq array<float> COMMENT 'Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.',<br />
ebv float COMMENT 'Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns',<br />
mjd_min double COMMENT 'Minimum Modified Julian Date of observations used to construct the model of this object',<br />
mjd_max double COMMENT 'Maximum Modified Julian Date of observations used to construct the model of this object',<br />
ref_cat string COMMENT 'Reference catalog source for this star: "T2" for Tycho-2, "G2" for Gaia DR2, "L2" for the LSLGA, empty otherwise',<br />
ref_id bigint COMMENT 'Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b "sourceid" for Gaia-DR2 and LSLGA',<br />
pmra float COMMENT 'Reference catalog proper motion in the RA direction',<br />
pmdec float COMMENT 'Reference catalog proper motion in the Dec direction',<br />
parallax float COMMENT 'Reference catalog parallax',<br />
pmra_ivar float COMMENT 'Reference catalog inverse-variance on pmra',<br />
pmdec_ivar float COMMENT 'Reference catalog inverse-variance on pmdec',<br />
parallax_ivar float COMMENT 'Reference catalog inverse-variance on parallax',<br />
ref_epoch float COMMENT 'Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)',<br />
gaia_pointsource boolean COMMENT 'This Gaia DR2 source is believed to be a star, not a galaxy',<br />
gaia_phot_g_mean_mag float COMMENT 'Gaia G band mag',<br />
gaia_phot_g_mean_flux_over_error float COMMENT 'Gaia G band signal-to-noise',<br />
gaia_phot_g_n_obs smallint COMMENT 'Gaia G band number of observations',<br />
gaia_phot_bp_mean_mag float COMMENT 'Gaia BP mag',<br />
gaia_phot_bp_mean_flux_over_error float COMMENT 'Gaia BP signal-to-noise',<br />
gaia_phot_bp_n_obs smallint COMMENT 'Gaia BP number of observations',<br />
gaia_phot_rp_mean_mag float COMMENT 'Gaia RP mag',<br />
gaia_phot_rp_mean_flux_over_error float COMMENT 'Gaia RP signal-to-noise',<br />
gaia_phot_rp_n_obs smallint COMMENT 'Gaia RP number of observations',<br />
gaia_phot_variable_flag boolean COMMENT 'Gaia photometric variable flag',<br />
gaia_astrometric_excess_noise float COMMENT 'Gaia astrometric excess noise',<br />
gaia_astrometric_excess_noise_sig float COMMENT 'Gaia astrometric excess noise uncertainty',<br />
gaia_astrometric_n_obs_al smallint COMMENT 'Gaia number of astrometric observations along scan direction',<br />
gaia_astrometric_n_good_obs_al smallint COMMENT 'Gaia number of good astrometric observations along scan direction',<br />
gaia_astrometric_weight_al float COMMENT 'Gaia astrometric weight along scan direction',<br />
gaia_duplicated_source boolean COMMENT 'Gaia duplicated source flag',<br />
gaia_a_g_val float COMMENT 'Gaia line-of-sight extinction in the G band',<br />
gaia_e_bp_min_rp_val float COMMENT 'Gaia line-of-sight reddening E(BP-RP)',<br />
gaia_phot_bp_rp_excess_factor float COMMENT 'Gaia BP/RP excess factor',<br />
gaia_astrometric_sigma5d_max float COMMENT 'Gaia longest semi-major axis of the 5-d error ellipsoid',<br />
gaia_astrometric_params_solved tinyint COMMENT 'which astrometric parameters were estimated for a Gaia source',<br />
flux_g float COMMENT 'model flux in gg',<br />
flux_r float COMMENT 'model flux in rr',<br />
flux_z float COMMENT 'model flux in zz',<br />
flux_w1 float COMMENT 'WISE model flux in W1W1 (AB system)',<br />
flux_w2 float COMMENT 'WISE model flux in W2W2 (AB)',<br />
flux_w3 float COMMENT 'WISE model flux in W3W3 (AB)',<br />
flux_w4 float COMMENT 'WISE model flux in W4W4 (AB)',<br />
flux_ivar_g float COMMENT 'Inverse variance of flux_g',<br />
flux_ivar_r float COMMENT 'Inverse variance of flux_r',<br />
flux_ivar_z float COMMENT 'Inverse variance of flux_z',<br />
flux_ivar_w1 float COMMENT 'Inverse variance of flux_w1 (AB system)',<br />
flux_ivar_w2 float COMMENT 'Inverse variance of flux_w2 (AB)',<br />
flux_ivar_w3 float COMMENT 'Inverse variance of flux_w3 (AB)',<br />
flux_ivar_w4 float COMMENT 'Inverse variance of flux_w4 (AB)',<br />
fiberflux_g float COMMENT 'Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_r float COMMENT 'Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_z float COMMENT 'Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fibertotflux_g float COMMENT 'Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_r float COMMENT 'Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_z float COMMENT 'Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
apflux_g array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg',<br />
apflux_r array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr',<br />
apflux_z array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz',<br />
apflux_resid_g array<float> COMMENT 'aperture fluxes on the co-added residual images in gg',<br />
apflux_resid_r array<float> COMMENT 'aperture fluxes on the co-added residual images in rr',<br />
apflux_resid_z array<float> COMMENT 'aperture fluxes on the co-added residual images in zz',<br />
apflux_ivar_g array<float> COMMENT 'Inverse variance of apflux_resid_g',<br />
apflux_ivar_r array<float> COMMENT 'Inverse variance of apflux_resid_r',<br />
apflux_ivar_z array<float> COMMENT 'Inverse variance of apflux_resid_z',<br />
mw_transmission_g float COMMENT 'Galactic transmission in gg filter in linear units [0, 1]',<br />
mw_transmission_r float COMMENT 'Galactic transmission in rr filter in linear units [0, 1]',<br />
mw_transmission_z float COMMENT 'Galactic transmission in zz filter in linear units [0, 1]',<br />
mw_transmission_w1 float COMMENT 'Galactic transmission in W1W1 filter in linear units [0, 1]',<br />
mw_transmission_w2 float COMMENT 'Galactic transmission in W2W2 filter in linear units [0, 1]',<br />
mw_transmission_w3 float COMMENT 'Galactic transmission in W3W3 filter in linear units [0, 1]',<br />
mw_transmission_w4 float COMMENT 'Galactic transmission in W4W4 filter in linear units [0, 1]',<br />
nobs_g smallint COMMENT 'Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)',<br />
nobs_r smallint COMMENT 'Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)',<br />
nobs_z smallint COMMENT 'Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)',<br />
nobs_w1 smallint COMMENT 'Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)',<br />
nobs_w2 smallint COMMENT 'Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)',<br />
nobs_w3 smallint COMMENT 'Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)',<br />
nobs_w4 smallint COMMENT 'Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)',<br />
rchisq_g float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in gg',<br />
rchisq_r float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in rr',<br />
rchisq_z float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in zz',<br />
rchisq_w1 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W1W1',<br />
rchisq_w2 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W2W2',<br />
rchisq_w3 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W3W3',<br />
rchisq_w4 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W4W4',<br />
fracflux_g float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in gg (typically [0,1])',<br />
fracflux_r float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in rr (typically [0,1])',<br />
fracflux_z float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in zz (typically [0,1])',<br />
fracflux_w1 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1 (typically [0,1])',<br />
fracflux_w2 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2 (typically [0,1])',<br />
fracflux_w3 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3 (typically [0,1])',<br />
fracflux_w4 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4 (typically [0,1])',<br />
fracmasked_g float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]',<br />
fracmasked_r float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]',<br />
fracmasked_z float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]',<br />
fracin_g float COMMENT 'Fraction of a source\'s flux within the blob in gg, near unity for real sources',<br />
fracin_r float COMMENT 'Fraction of a source\'s flux within the blob in rr, near unity for real sources',<br />
fracin_z float COMMENT 'Fraction of a source\'s flux within the blob in zz, near unity for real sources',<br />
anymask_g smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page',<br />
anymask_r smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page',<br />
anymask_z smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page',<br />
allmask_g smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page',<br />
allmask_r smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page',<br />
allmask_z smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page',<br />
wisemask_w1 tinyint COMMENT 'W1 bitmask as cataloged on the DR8 bitmasks page',<br />
wisemask_w2 tinyint COMMENT 'W2 bitmask as cataloged on the DR8 bitmasks page',<br />
psfsize_g float COMMENT 'Weighted average PSF FWHM in the gg band',<br />
psfsize_r float COMMENT 'Weighted average PSF FWHM in the rr band',<br />
psfsize_z float COMMENT 'Weighted average PSF FWHM in the zz band',<br />
psfdepth_g float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude',<br />
psfdepth_r float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude',<br />
psfdepth_z float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude',<br />
galdepth_g float COMMENT 'As for psfdepth_g but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_r float COMMENT 'As for psfdepth_r but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_z float COMMENT 'As for psfdepth_z but for a galaxy (0.45" exp, round) detection sensitivity',<br />
psfdepth_w1 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W1',<br />
psfdepth_w2 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W2',<br />
psfdepth_w3 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W3',<br />
psfdepth_w4 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W4',<br />
wise_coadd_id string COMMENT 'unWISE coadd file name for the center of each object',<br />
lc_flux_w1 array<float> COMMENT 'flux_w1 in each of up to eleven unWISE coadd epochs (AB system)',<br />
lc_flux_w2 array<float> COMMENT 'flux_w2 in each of up to eleven unWISE coadd epochs (AB)',<br />
lc_flux_ivar_w1 array<float> COMMENT 'Inverse variance of lc_flux_w1 (AB system)',<br />
lc_flux_ivar_w2 array<float> COMMENT 'Inverse variance of lc_flux_w2 (AB)',<br />
lc_nobs_w1 array<smallint> COMMENT 'nobs_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_nobs_w2 array<smallint> COMMENT 'nobs_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w1 array<float> COMMENT 'fracflux_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w2 array<float> COMMENT 'fracflux_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w1 array<float> COMMENT 'rchisq_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w2 array<float> COMMENT 'rchisq_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w1 array<double> COMMENT 'mjd_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w2 array<double> COMMENT 'mjd_w2 in each of up to eleven unWISE coadd epochs',<br />
fracdev float COMMENT 'Fraction of model in deVauc [0,1]',<br />
fracdev_ivar float COMMENT 'Inverse variance of fracdev',<br />
shapeexp_r float COMMENT 'Half-light radius of exponential model (>0)',<br />
shapeexp_r_ivar float COMMENT 'Inverse variance of shapeexp_r',<br />
shapeexp_e1 float COMMENT 'Ellipticity component 1',<br />
shapeexp_e1_ivar float COMMENT 'Inverse variance of shapeexp_e1',<br />
shapeexp_e2 float COMMENT 'Ellipticity component 2',<br />
shapeexp_e2_ivar float COMMENT 'Inverse variance of shapeexp_e2',<br />
shapedev_r float COMMENT 'Half-light radius of deVaucouleurs model (>0)',<br />
shapedev_r_ivar float COMMENT 'Inverse variance of shapedev_r',<br />
shapedev_e1 float COMMENT 'Ellipticity component 1',<br />
shapedev_e1_ivar float COMMENT 'Inverse variance of shapedev_e1',<br />
shapedev_e2 float COMMENT 'Ellipticity component 2',<br />
shapedev_e2_ivar float COMMENT 'Inverse variance of shapedev_e2',<br />
z_phot_mean FLOAT COMMENT 'photo-z derived from the mean of the photo-z PDF ',<br />
z_phot_median FLOAT COMMENT 'photo-z derived from the median of the photo-z PDF',<br />
z_phot_std FLOAT COMMENT 'standard deviation of the photo-z\'s derived from the photo-z PDF',<br />
z_phot_l68 FLOAT COMMENT 'lower bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_u68 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_l95 FLOAT COMMENT 'lower bound of the 95% confidence region, derived from the photo-z PDF',<br />
z_phot_u95 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_spec FLOAT COMMENT 'spectroscopic redshift, if available',<br />
survey STRING COMMENT 'source of the spectroscopic redshift',<br />
training BOOLEAN COMMENT 'whether or not the spectroscopic redshift is used in photometric redshift training'<br />
)<br />
CLUSTERED BY (<br />
release,<br />
brickid,<br />
objid)<br />
SORTED BY (<br />
release ASC,<br />
brickid ASC,<br />
objid ASC)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
=== Join insert ===<br />
<br />
INSERT OVERWRITE TABLE cosmohub.legacy_survey_dr8_phz<br />
SELECT<br />
dr8.*,<br />
phz.z_phot_mean,<br />
phz.z_phot_median,<br />
phz.z_phot_std,<br />
phz.z_phot_l68,<br />
phz.z_phot_u68,<br />
phz.z_phot_l95,<br />
phz.z_phot_u95,<br />
phz.z_spec,<br />
phz.survey,<br />
phz.training<br />
FROM cosmohub.legacy_survey_dr8 AS dr8<br />
LEFT JOIN tallada.dr8_sweep AS phz<br />
ON dr8.release = phz.release<br />
AND dr8.brickid = phz.brickid<br />
AND dr8.objid = phz.objid<br />
;<br />
<br />
== Queries for checking ==<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT release, brickid, objid, COUNT(release)<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
HAVING COUNT(release) > 1<br />
<br />
DR8 release brickid objid: DISTINCT(1.694.230.368) TOTAL(1.694.230.368)<br />
DR8 brickid objid: DISTINCT(1.654.429.115) TOTAL(1.694.230.368)<br />
<br />
PHZ release brickid objid: DISTINCT(1.646.664.729) TOTAL(1.646.664.897)<br />
PHZ brickid objid: DISTINCT(1.608.841.582) TOTAL(1.646.664.897)<br />
<br />
DR8: 1.694.230.368<br />
PHZ: 1.646.664.897<br />
----------------------------------------------------------------------<br />
-47.565.471</div>Talladahttps://pwiki.pic.es/index.php?title=Legacy_Survey_DR9&diff=875Legacy Survey DR92020-11-27T00:24:29Z<p>Tallada: /* Internal Table */</p>
<hr />
<div>== Original DR9 ==<br />
=== Raw data ===<br />
<br />
Downloaded from NERSC<br />
/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m<br />
<br />
Directories:<br />
south/tractor<br />
north/tractor<br />
<br />
Downloaded using bbcp from services01.euclid into CephFS scratch:<br />
bbcp -s 16 -a -A -v -z -r -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" \<br />
"jcarrete@dtn01.nersc.gov:/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m/north/tractor" \<br />
/cephfs/pic.es/astro/scratch/tallada/dr9m/north/tractor<br />
bbcp -s 16 -a -A -v -z -r -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" \<br />
"jcarrete@dtn01.nersc.gov:/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m/south/tractor" \<br />
/cephfs/pic.es/astro/scratch/tallada/dr9m/south/tractor<br />
<br />
Checksums provided through sha256sum files, and checked using command:<br />
find * -type d | xargs -t -L1 -P24 sh -c 'cd $1; sha256sum --quiet -c *.sha256sum' cksum<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to convert fits files to parquet:<br />
<br />
path_base = '/cephfs/pic.es/astro/scratch/tallada/dr9m'<br />
hdfs_base = '/user/tallada/data/legacysurvey_dr9'<br />
<br />
hs = ['h=north', 'h=south']<br />
rs = ['r={0:03d}'.format(x) for x in range(360)]<br />
<br />
for h in hs:<br />
for r in rs:<br />
p = os.path.join(path_base, h, r)<br />
if not os.path.exists(p):<br />
continue<br />
<br />
size = sum(os.path.getsize(os.path.join(p,f)) for f in os.listdir(p) if os.path.isfile(os.path.join(p,f)))<br />
<br />
print h, r, size <br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + p<br />
)<br />
<br />
df = df.repartition(int(math.ceil(size / (256*1024*1024.0))))<br />
<br />
df.write.parquet(os.path.join(hdfs_base, h, r))<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE `tallada`.`legacysurvey_dr9`(<br />
`release` smallint, <br />
`brickid` int, <br />
`brickname` string, <br />
`objid` int, <br />
`brick_primary` boolean, <br />
`maskbits` smallint, <br />
`fitbits` smallint, <br />
`type` string, <br />
`ra` double, <br />
`dec` double, <br />
`ra_ivar` float, <br />
`dec_ivar` float, <br />
`bx` float, <br />
`by` float, <br />
`dchisq` array<float>, <br />
`ebv` float, <br />
`mjd_min` double, <br />
`mjd_max` double, <br />
`ref_cat` string, <br />
`ref_id` bigint, <br />
`pmra` float, <br />
`pmdec` float, <br />
`parallax` float, <br />
`pmra_ivar` float, <br />
`pmdec_ivar` float, <br />
`parallax_ivar` float, <br />
`ref_epoch` float, <br />
`gaia_phot_g_mean_mag` float, <br />
`gaia_phot_g_mean_flux_over_error` float, <br />
`gaia_phot_g_n_obs` smallint, <br />
`gaia_phot_bp_mean_mag` float, <br />
`gaia_phot_bp_mean_flux_over_error` float, <br />
`gaia_phot_bp_n_obs` smallint, <br />
`gaia_phot_rp_mean_mag` float, <br />
`gaia_phot_rp_mean_flux_over_error` float, <br />
`gaia_phot_rp_n_obs` smallint, <br />
`gaia_phot_variable_flag` boolean, <br />
`gaia_astrometric_excess_noise` float, <br />
`gaia_astrometric_excess_noise_sig` float, <br />
`gaia_astrometric_n_obs_al` smallint, <br />
`gaia_astrometric_n_good_obs_al` smallint, <br />
`gaia_astrometric_weight_al` float, <br />
`gaia_duplicated_source` boolean, <br />
`gaia_a_g_val` float, <br />
`gaia_e_bp_min_rp_val` float, <br />
`gaia_phot_bp_rp_excess_factor` float, <br />
`gaia_astrometric_sigma5d_max` float, <br />
`gaia_astrometric_params_solved` tinyint, <br />
`flux_g` float, <br />
`flux_r` float, <br />
`flux_z` float, <br />
`flux_w1` float, <br />
`flux_w2` float, <br />
`flux_w3` float, <br />
`flux_w4` float, <br />
`flux_ivar_g` float, <br />
`flux_ivar_r` float, <br />
`flux_ivar_z` float, <br />
`flux_ivar_w1` float, <br />
`flux_ivar_w2` float, <br />
`flux_ivar_w3` float, <br />
`flux_ivar_w4` float, <br />
`fiberflux_g` float, <br />
`fiberflux_r` float, <br />
`fiberflux_z` float, <br />
`fibertotflux_g` float, <br />
`fibertotflux_r` float, <br />
`fibertotflux_z` float, <br />
`apflux_g` array<float>, <br />
`apflux_r` array<float>, <br />
`apflux_z` array<float>, <br />
`apflux_resid_g` array<float>, <br />
`apflux_resid_r` array<float>, <br />
`apflux_resid_z` array<float>, <br />
`apflux_blobresid_g` array<float>, <br />
`apflux_blobresid_r` array<float>, <br />
`apflux_blobresid_z` array<float>, <br />
`apflux_ivar_g` array<float>, <br />
`apflux_ivar_r` array<float>, <br />
`apflux_ivar_z` array<float>, <br />
`apflux_masked_g` array<float>, <br />
`apflux_masked_r` array<float>, <br />
`apflux_masked_z` array<float>, <br />
`apflux_w1` array<float>, <br />
`apflux_w2` array<float>, <br />
`apflux_w3` array<float>, <br />
`apflux_w4` array<float>, <br />
`apflux_resid_w1` array<float>, <br />
`apflux_resid_w2` array<float>, <br />
`apflux_resid_w3` array<float>, <br />
`apflux_resid_w4` array<float>, <br />
`apflux_ivar_w1` array<float>, <br />
`apflux_ivar_w2` array<float>, <br />
`apflux_ivar_w3` array<float>, <br />
`apflux_ivar_w4` array<float>, <br />
`mw_transmission_g` float, <br />
`mw_transmission_r` float, <br />
`mw_transmission_z` float, <br />
`mw_transmission_w1` float, <br />
`mw_transmission_w2` float, <br />
`mw_transmission_w3` float, <br />
`mw_transmission_w4` float, <br />
`nobs_g` smallint, <br />
`nobs_r` smallint, <br />
`nobs_z` smallint, <br />
`nobs_w1` smallint, <br />
`nobs_w2` smallint, <br />
`nobs_w3` smallint, <br />
`nobs_w4` smallint, <br />
`rchisq_g` float, <br />
`rchisq_r` float, <br />
`rchisq_z` float, <br />
`rchisq_w1` float, <br />
`rchisq_w2` float, <br />
`rchisq_w3` float, <br />
`rchisq_w4` float, <br />
`fracflux_g` float, <br />
`fracflux_r` float, <br />
`fracflux_z` float, <br />
`fracflux_w1` float, <br />
`fracflux_w2` float, <br />
`fracflux_w3` float, <br />
`fracflux_w4` float, <br />
`fracmasked_g` float, <br />
`fracmasked_r` float, <br />
`fracmasked_z` float, <br />
`fracin_g` float, <br />
`fracin_r` float, <br />
`fracin_z` float, <br />
`anymask_g` smallint, <br />
`anymask_r` smallint, <br />
`anymask_z` smallint, <br />
`allmask_g` smallint, <br />
`allmask_r` smallint, <br />
`allmask_z` smallint, <br />
`wisemask_w1` tinyint, <br />
`wisemask_w2` tinyint, <br />
`psfsize_g` float, <br />
`psfsize_r` float, <br />
`psfsize_z` float, <br />
`psfdepth_g` float, <br />
`psfdepth_r` float, <br />
`psfdepth_z` float, <br />
`galdepth_g` float, <br />
`galdepth_r` float, <br />
`galdepth_z` float, <br />
`nea_g` float, <br />
`nea_r` float, <br />
`nea_z` float, <br />
`blob_nea_g` float, <br />
`blob_nea_r` float, <br />
`blob_nea_z` float, <br />
`psfdepth_w1` float, <br />
`psfdepth_w2` float, <br />
`psfdepth_w3` float, <br />
`psfdepth_w4` float, <br />
`wise_coadd_id` string, <br />
`wise_x` float, <br />
`wise_y` float, <br />
`lc_flux_w1` array<float>, <br />
`lc_flux_w2` array<float>, <br />
`lc_flux_ivar_w1` array<float>, <br />
`lc_flux_ivar_w2` array<float>, <br />
`lc_nobs_w1` array<smallint>, <br />
`lc_nobs_w2` array<smallint>, <br />
`lc_fracflux_w1` array<float>, <br />
`lc_fracflux_w2` array<float>, <br />
`lc_rchisq_w1` array<float>, <br />
`lc_rchisq_w2` array<float>, <br />
`lc_mjd_w1` array<double>, <br />
`lc_mjd_w2` array<double>, <br />
`lc_epoch_index_w1` array<smallint>, <br />
`lc_epoch_index_w2` array<smallint>, <br />
`sersic` float, <br />
`sersic_ivar` float, <br />
`shape_r` float, <br />
`shape_r_ivar` float, <br />
`shape_e1` float, <br />
`shape_e1_ivar` float, <br />
`shape_e2` float, <br />
`shape_e2_ivar` float)<br />
PARTITIONED BY (<br />
h STRING,<br />
r STRING<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey_dr9'<br />
;<br />
<br />
=== Internal Table ===<br />
<br />
CREATE TABLE `cosmohub`.`legacysurvey_dr9`(<br />
`release` smallint,<br />
`brickid` int,<br />
`brickname` string,<br />
`objid` int,<br />
`brick_primary` boolean,<br />
`maskbits` smallint,<br />
`fitbits` smallint,<br />
`type` string,<br />
`ra` double,<br />
`dec` double,<br />
`ra_ivar` float,<br />
`dec_ivar` float,<br />
`bx` float,<br />
`by` float,<br />
`dchisq` array<float>,<br />
`ebv` float,<br />
`mjd_min` double,<br />
`mjd_max` double,<br />
`ref_cat` string,<br />
`ref_id` bigint,<br />
`pmra` float,<br />
`pmdec` float,<br />
`parallax` float,<br />
`pmra_ivar` float,<br />
`pmdec_ivar` float,<br />
`parallax_ivar` float,<br />
`ref_epoch` float,<br />
`gaia_phot_g_mean_mag` float,<br />
`gaia_phot_g_mean_flux_over_error` float,<br />
`gaia_phot_g_n_obs` smallint,<br />
`gaia_phot_bp_mean_mag` float,<br />
`gaia_phot_bp_mean_flux_over_error` float,<br />
`gaia_phot_bp_n_obs` smallint,<br />
`gaia_phot_rp_mean_mag` float,<br />
`gaia_phot_rp_mean_flux_over_error` float,<br />
`gaia_phot_rp_n_obs` smallint,<br />
`gaia_phot_variable_flag` boolean,<br />
`gaia_astrometric_excess_noise` float,<br />
`gaia_astrometric_excess_noise_sig` float,<br />
`gaia_astrometric_n_obs_al` smallint,<br />
`gaia_astrometric_n_good_obs_al` smallint,<br />
`gaia_astrometric_weight_al` float,<br />
`gaia_duplicated_source` boolean,<br />
`gaia_a_g_val` float,<br />
`gaia_e_bp_min_rp_val` float,<br />
`gaia_phot_bp_rp_excess_factor` float,<br />
`gaia_astrometric_sigma5d_max` float,<br />
`gaia_astrometric_params_solved` tinyint,<br />
`flux_g` float,<br />
`flux_r` float,<br />
`flux_z` float,<br />
`flux_w1` float,<br />
`flux_w2` float,<br />
`flux_w3` float,<br />
`flux_w4` float,<br />
`flux_ivar_g` float,<br />
`flux_ivar_r` float,<br />
`flux_ivar_z` float,<br />
`flux_ivar_w1` float,<br />
`flux_ivar_w2` float,<br />
`flux_ivar_w3` float,<br />
`flux_ivar_w4` float,<br />
`fiberflux_g` float,<br />
`fiberflux_r` float,<br />
`fiberflux_z` float,<br />
`fibertotflux_g` float,<br />
`fibertotflux_r` float,<br />
`fibertotflux_z` float,<br />
`apflux_g` array<float>,<br />
`apflux_r` array<float>,<br />
`apflux_z` array<float>,<br />
`apflux_resid_g` array<float>,<br />
`apflux_resid_r` array<float>,<br />
`apflux_resid_z` array<float>,<br />
`apflux_blobresid_g` array<float>,<br />
`apflux_blobresid_r` array<float>,<br />
`apflux_blobresid_z` array<float>,<br />
`apflux_ivar_g` array<float>,<br />
`apflux_ivar_r` array<float>,<br />
`apflux_ivar_z` array<float>,<br />
`apflux_masked_g` array<float>,<br />
`apflux_masked_r` array<float>,<br />
`apflux_masked_z` array<float>,<br />
`apflux_w1` array<float>,<br />
`apflux_w2` array<float>,<br />
`apflux_w3` array<float>,<br />
`apflux_w4` array<float>,<br />
`apflux_resid_w1` array<float>,<br />
`apflux_resid_w2` array<float>,<br />
`apflux_resid_w3` array<float>,<br />
`apflux_resid_w4` array<float>,<br />
`apflux_ivar_w1` array<float>,<br />
`apflux_ivar_w2` array<float>,<br />
`apflux_ivar_w3` array<float>,<br />
`apflux_ivar_w4` array<float>,<br />
`mw_transmission_g` float,<br />
`mw_transmission_r` float,<br />
`mw_transmission_z` float,<br />
`mw_transmission_w1` float,<br />
`mw_transmission_w2` float,<br />
`mw_transmission_w3` float,<br />
`mw_transmission_w4` float,<br />
`nobs_g` smallint,<br />
`nobs_r` smallint,<br />
`nobs_z` smallint,<br />
`nobs_w1` smallint,<br />
`nobs_w2` smallint,<br />
`nobs_w3` smallint,<br />
`nobs_w4` smallint,<br />
`rchisq_g` float,<br />
`rchisq_r` float,<br />
`rchisq_z` float,<br />
`rchisq_w1` float,<br />
`rchisq_w2` float,<br />
`rchisq_w3` float,<br />
`rchisq_w4` float,<br />
`fracflux_g` float,<br />
`fracflux_r` float,<br />
`fracflux_z` float,<br />
`fracflux_w1` float,<br />
`fracflux_w2` float,<br />
`fracflux_w3` float,<br />
`fracflux_w4` float,<br />
`fracmasked_g` float,<br />
`fracmasked_r` float,<br />
`fracmasked_z` float,<br />
`fracin_g` float,<br />
`fracin_r` float,<br />
`fracin_z` float,<br />
`anymask_g` smallint,<br />
`anymask_r` smallint,<br />
`anymask_z` smallint,<br />
`allmask_g` smallint,<br />
`allmask_r` smallint,<br />
`allmask_z` smallint,<br />
`wisemask_w1` tinyint,<br />
`wisemask_w2` tinyint,<br />
`psfsize_g` float,<br />
`psfsize_r` float,<br />
`psfsize_z` float,<br />
`psfdepth_g` float,<br />
`psfdepth_r` float,<br />
`psfdepth_z` float,<br />
`galdepth_g` float,<br />
`galdepth_r` float,<br />
`galdepth_z` float,<br />
`nea_g` float,<br />
`nea_r` float,<br />
`nea_z` float,<br />
`blob_nea_g` float,<br />
`blob_nea_r` float,<br />
`blob_nea_z` float,<br />
`psfdepth_w1` float,<br />
`psfdepth_w2` float,<br />
`psfdepth_w3` float,<br />
`psfdepth_w4` float,<br />
`wise_coadd_id` string,<br />
`wise_x` float,<br />
`wise_y` float,<br />
`lc_flux_w1` array<float>,<br />
`lc_flux_w2` array<float>,<br />
`lc_flux_ivar_w1` array<float>,<br />
`lc_flux_ivar_w2` array<float>,<br />
`lc_nobs_w1` array<smallint>,<br />
`lc_nobs_w2` array<smallint>,<br />
`lc_fracflux_w1` array<float>,<br />
`lc_fracflux_w2` array<float>,<br />
`lc_rchisq_w1` array<float>,<br />
`lc_rchisq_w2` array<float>,<br />
`lc_mjd_w1` array<double>,<br />
`lc_mjd_w2` array<double>,<br />
`lc_epoch_index_w1` array<smallint>,<br />
`lc_epoch_index_w2` array<smallint>,<br />
`sersic` float,<br />
`sersic_ivar` float,<br />
`shape_r` float,<br />
`shape_r_ivar` float,<br />
`shape_e1` float,<br />
`shape_e1_ivar` float,<br />
`shape_e2` float,<br />
`shape_e2_ivar` float)<br />
CLUSTERED BY (<br />
release, brickid, objid<br />
)<br />
SORTED BY (<br />
release, brickid, objid<br />
)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
<br />
INSERT OVERWRITE TABLE cosmohub.legacysurvey_dr9<br />
SELECT release, brickid, brickname, objid, brick_primary, maskbits, fitbits,<br />
type, ra, `dec`, ra_ivar, dec_ivar, bx, `by`, dchisq, ebv, mjd_min, mjd_max,<br />
ref_cat, ref_id, pmra, pmdec, parallax, pmra_ivar, pmdec_ivar, parallax_ivar,<br />
ref_epoch, gaia_phot_g_mean_mag, gaia_phot_g_mean_flux_over_error, gaia_phot_g_n_obs,<br />
gaia_phot_bp_mean_mag, gaia_phot_bp_mean_flux_over_error, gaia_phot_bp_n_obs,<br />
gaia_phot_rp_mean_mag, gaia_phot_rp_mean_flux_over_error, gaia_phot_rp_n_obs,<br />
gaia_phot_variable_flag, gaia_astrometric_excess_noise, gaia_astrometric_excess_noise_sig,<br />
gaia_astrometric_n_obs_al, gaia_astrometric_n_good_obs_al, gaia_astrometric_weight_al,<br />
gaia_duplicated_source, gaia_a_g_val, gaia_e_bp_min_rp_val, gaia_phot_bp_rp_excess_factor,<br />
gaia_astrometric_sigma5d_max, gaia_astrometric_params_solved, flux_g, flux_r,<br />
flux_z, flux_w1, flux_w2, flux_w3, flux_w4, flux_ivar_g, flux_ivar_r, flux_ivar_z,<br />
flux_ivar_w1, flux_ivar_w2, flux_ivar_w3, flux_ivar_w4, fiberflux_g, fiberflux_r,<br />
fiberflux_z, fibertotflux_g, fibertotflux_r, fibertotflux_z, apflux_g, apflux_r,<br />
apflux_z, apflux_resid_g, apflux_resid_r, apflux_resid_z, apflux_blobresid_g,<br />
apflux_blobresid_r, apflux_blobresid_z, apflux_ivar_g, apflux_ivar_r, apflux_ivar_z,<br />
apflux_masked_g, apflux_masked_r, apflux_masked_z, apflux_w1, apflux_w2, apflux_w3,<br />
apflux_w4, apflux_resid_w1, apflux_resid_w2, apflux_resid_w3, apflux_resid_w4,<br />
apflux_ivar_w1, apflux_ivar_w2, apflux_ivar_w3, apflux_ivar_w4, mw_transmission_g,<br />
mw_transmission_r, mw_transmission_z, mw_transmission_w1, mw_transmission_w2,<br />
mw_transmission_w3, mw_transmission_w4, nobs_g, nobs_r, nobs_z, nobs_w1, nobs_w2,<br />
nobs_w3, nobs_w4, rchisq_g, rchisq_r, rchisq_z, rchisq_w1, rchisq_w2, rchisq_w3,<br />
rchisq_w4, fracflux_g, fracflux_r, fracflux_z, fracflux_w1, fracflux_w2, fracflux_w3,<br />
fracflux_w4, fracmasked_g, fracmasked_r, fracmasked_z, fracin_g, fracin_r, fracin_z,<br />
anymask_g, anymask_r, anymask_z, allmask_g, allmask_r, allmask_z, wisemask_w1,<br />
wisemask_w2, psfsize_g, psfsize_r, psfsize_z, psfdepth_g, psfdepth_r, psfdepth_z,<br />
galdepth_g, galdepth_r, galdepth_z, nea_g, nea_r, nea_z, blob_nea_g, blob_nea_r,<br />
blob_nea_z, psfdepth_w1, psfdepth_w2, psfdepth_w3, psfdepth_w4, wise_coadd_id,<br />
wise_x, wise_y, lc_flux_w1, lc_flux_w2, lc_flux_ivar_w1, lc_flux_ivar_w2, lc_nobs_w1,<br />
lc_nobs_w2, lc_fracflux_w1, lc_fracflux_w2, lc_rchisq_w1, lc_rchisq_w2, lc_mjd_w1,<br />
lc_mjd_w2, lc_epoch_index_w1, lc_epoch_index_w2, sersic, sersic_ivar, shape_r,<br />
shape_r_ivar, shape_e1, shape_e1_ivar, shape_e2, shape_e2_ivar<br />
FROM tallada.legacysurvey_dr9<br />
;<br />
<br />
=== Comments ===<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE release release smallint COMMENT "Unique integer denoting the camera and filter set used";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickid brickid int COMMENT "Brick ID [1,662174]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickname brickname string COMMENT "Name of brick, encoding the brick sky position, eg \"1126p222\" near RA=112.6, Dec=+22.2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE objid objid int COMMENT "Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brick_primary brick_primary boolean COMMENT "True if the object is within the brick boundary";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brightblob brightblob smallint COMMENT "bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE maskbits maskbits smallint COMMENT "bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits*maps, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE type type string COMMENT "Morphological model: PSF=stellar, REX=round exponential galaxy, DEV=deVauc, EXP=exponential, COMP=composite, DUP=Gaia source fit by different model.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra ra double COMMENT "Right ascension at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec dec double COMMENT "Declination at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra_ivar ra_ivar float COMMENT "Inverse variance of RA (no cosine term!), excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec_ivar dec_ivar float COMMENT "Inverse variance of DEC, excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE bx bx float COMMENT "X position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE by by float COMMENT "Y position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dchisq dchisq array<float> COMMENT "Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ebv ebv float COMMENT "Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_min mjd_min double COMMENT "Minimum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_max mjd_max double COMMENT "Maximum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_cat ref_cat string COMMENT "Reference catalog source for this star: T2 for Tycho-2, G2 for Gaia DR2, L2 for the LSLGA, empty otherwise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_id ref_id bigint COMMENT "Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b \"sourceid\" for Gaia-DR2 and LSLGA";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra pmra float COMMENT "Reference catalog proper motion in the RA direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec pmdec float COMMENT "Reference catalog proper motion in the Dec direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax parallax float COMMENT "Reference catalog parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra_ivar pmra_ivar float COMMENT "Reference catalog inverse-variance on pmra";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec_ivar pmdec_ivar float COMMENT "Reference catalog inverse-variance on pmdec";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax_ivar parallax_ivar float COMMENT "Reference catalog inverse-variance on parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_epoch ref_epoch float COMMENT "Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_pointsource gaia_pointsource boolean COMMENT "This Gaia DR2 source is believed to be a star, not a galaxy";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_mag gaia_phot_g_mean_mag float COMMENT "Gaia G band mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_flux_over_error gaia_phot_g_mean_flux_over_error float COMMENT "Gaia G band signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_n_obs gaia_phot_g_n_obs smallint COMMENT "Gaia G band number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_mag gaia_phot_bp_mean_mag float COMMENT "Gaia BP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_flux_over_error gaia_phot_bp_mean_flux_over_error float COMMENT "Gaia BP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_n_obs gaia_phot_bp_n_obs smallint COMMENT "Gaia BP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_mag gaia_phot_rp_mean_mag float COMMENT "Gaia RP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_flux_over_error gaia_phot_rp_mean_flux_over_error float COMMENT "Gaia RP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_n_obs gaia_phot_rp_n_obs smallint COMMENT "Gaia RP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_variable_flag gaia_phot_variable_flag boolean COMMENT "Gaia photometric variable flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise gaia_astrometric_excess_noise float COMMENT "Gaia astrometric excess noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise_sig gaia_astrometric_excess_noise_sig float COMMENT "Gaia astrometric excess noise uncertainty";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_obs_al gaia_astrometric_n_obs_al smallint COMMENT "Gaia number of astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_good_obs_al gaia_astrometric_n_good_obs_al smallint COMMENT "Gaia number of good astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_weight_al gaia_astrometric_weight_al float COMMENT "Gaia astrometric weight along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_duplicated_source gaia_duplicated_source boolean COMMENT "Gaia duplicated source flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_a_g_val gaia_a_g_val float COMMENT "Gaia line-of-sight extinction in the G band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_e_bp_min_rp_val gaia_e_bp_min_rp_val float COMMENT "Gaia line-of-sight reddening E(BP-RP)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_rp_excess_factor gaia_phot_bp_rp_excess_factor float COMMENT "Gaia BP/RP excess factor";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_sigma5d_max gaia_astrometric_sigma5d_max float COMMENT "Gaia longest semi-major axis of the 5-d error ellipsoid";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_params_solved gaia_astrometric_params_solved tinyint COMMENT "which astrometric parameters were estimated for a Gaia source";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_g flux_g float COMMENT "model flux in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_r flux_r float COMMENT "model flux in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_z flux_z float COMMENT "model flux in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w1 flux_w1 float COMMENT "WISE model flux in W1W1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w2 flux_w2 float COMMENT "WISE model flux in W2W2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w3 flux_w3 float COMMENT "WISE model flux in W3W3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w4 flux_w4 float COMMENT "WISE model flux in W4W4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_g flux_ivar_g float COMMENT "Inverse variance of flux_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_r flux_ivar_r float COMMENT "Inverse variance of flux_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_z flux_ivar_z float COMMENT "Inverse variance of flux_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w1 flux_ivar_w1 float COMMENT "Inverse variance of flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w2 flux_ivar_w2 float COMMENT "Inverse variance of flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w3 flux_ivar_w3 float COMMENT "Inverse variance of flux_w3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w4 flux_ivar_w4 float COMMENT "Inverse variance of flux_w4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_g fiberflux_g float COMMENT "Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_r fiberflux_r float COMMENT "Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_z fiberflux_z float COMMENT "Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_g fibertotflux_g float COMMENT "Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_r fibertotflux_r float COMMENT "Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_z fibertotflux_z float COMMENT "Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_g apflux_g array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_r apflux_r array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_z apflux_z array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_g apflux_resid_g array<float> COMMENT "aperture fluxes on the co-added residual images in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_r apflux_resid_r array<float> COMMENT "aperture fluxes on the co-added residual images in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_z apflux_resid_z array<float> COMMENT "aperture fluxes on the co-added residual images in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_g apflux_ivar_g array<float> COMMENT "Inverse variance of apflux_resid_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_r apflux_ivar_r array<float> COMMENT "Inverse variance of apflux_resid_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_z apflux_ivar_z array<float> COMMENT "Inverse variance of apflux_resid_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_g mw_transmission_g float COMMENT "Galactic transmission in gg filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_r mw_transmission_r float COMMENT "Galactic transmission in rr filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_z mw_transmission_z float COMMENT "Galactic transmission in zz filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w1 mw_transmission_w1 float COMMENT "Galactic transmission in W1W1 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w2 mw_transmission_w2 float COMMENT "Galactic transmission in W2W2 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w3 mw_transmission_w3 float COMMENT "Galactic transmission in W3W3 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w4 mw_transmission_w4 float COMMENT "Galactic transmission in W4W4 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_g nobs_g smallint COMMENT "Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_r nobs_r smallint COMMENT "Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_z nobs_z smallint COMMENT "Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w1 nobs_w1 smallint COMMENT "Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w2 nobs_w2 smallint COMMENT "Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w3 nobs_w3 smallint COMMENT "Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w4 nobs_w4 smallint COMMENT "Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_g rchisq_g float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_r rchisq_r float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_z rchisq_z float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w1 rchisq_w1 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W1W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w2 rchisq_w2 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W2W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w3 rchisq_w3 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W3W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w4 rchisq_w4 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W4W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_g fracflux_g float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in gg(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_r fracflux_r float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in rr(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_z fracflux_z float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in zz(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w1 fracflux_w1 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w2 fracflux_w2 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w3 fracflux_w3 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w4 fracflux_w4 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_g fracmasked_g float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_r fracmasked_r float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_z fracmasked_z float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_g fracin_g float COMMENT "Fraction of a source's flux within the blob in gg, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_r fracin_r float COMMENT "Fraction of a source's flux within the blob in rr, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_z fracin_z float COMMENT "Fraction of a source's flux within the blob in zz, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_g anymask_g smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_r anymask_r smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_z anymask_z smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_g allmask_g smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_r allmask_r smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_z allmask_z smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w1 wisemask_w1 tinyint COMMENT "W1 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w2 wisemask_w2 tinyint COMMENT "W2 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_g psfsize_g float COMMENT "Weighted average PSF FWHM in the gg band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_r psfsize_r float COMMENT "Weighted average PSF FWHM in the rr band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_z psfsize_z float COMMENT "Weighted average PSF FWHM in the zz band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_g psfdepth_g float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_r psfdepth_r float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_z psfdepth_z float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_g galdepth_g float COMMENT "As for psfdepth_g but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_r galdepth_r float COMMENT "As for psfdepth_r but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_z galdepth_z float COMMENT "As for psfdepth_z but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w1 psfdepth_w1 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w2 psfdepth_w2 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w3 psfdepth_w3 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w4 psfdepth_w4 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wise_coadd_id wise_coadd_id string COMMENT "unWISE coadd file name for the center of each object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w1 lc_flux_w1 array<float> COMMENT "flux_w1 in each of up to eleven unWISE coadd epochs (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w2 lc_flux_w2 array<float> COMMENT "flux_w2 in each of up to eleven unWISE coadd epochs (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w1 lc_flux_ivar_w1 array<float> COMMENT "Inverse variance of lc_flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w2 lc_flux_ivar_w2 array<float> COMMENT "Inverse variance of lc_flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w1 lc_nobs_w1 array<smallint> COMMENT "nobs_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w2 lc_nobs_w2 array<smallint> COMMENT "nobs_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w1 lc_fracflux_w1 array<float> COMMENT "fracflux_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w2 lc_fracflux_w2 array<float> COMMENT "fracflux_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w1 lc_rchisq_w1 array<float> COMMENT "rchisq_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w2 lc_rchisq_w2 array<float> COMMENT "rchisq_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w1 lc_mjd_w1 array<double> COMMENT "mjd_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w2 lc_mjd_w2 array<double> COMMENT "mjd_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev fracdev float COMMENT "Fraction of model in deVauc [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev_ivar fracdev_ivar float COMMENT "Inverse variance of fracdev";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r shapeexp_r float COMMENT "Half-light radius of exponential model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r_ivar shapeexp_r_ivar float COMMENT "Inverse variance of shapeexp_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1 shapeexp_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1_ivar shapeexp_e1_ivar float COMMENT "Inverse variance of shapeexp_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2 shapeexp_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2_ivar shapeexp_e2_ivar float COMMENT "Inverse variance of shapeexp_e2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r shapedev_r float COMMENT "Half-light radius of deVaucouleurs model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r_ivar shapedev_r_ivar float COMMENT "Inverse variance of shapedev_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1 shapedev_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1_ivar shapedev_e1_ivar float COMMENT "Inverse variance of shapedev_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2 shapedev_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2_ivar shapedev_e2_ivar float COMMENT "Inverse variance of shapedev_e2";<br />
<br />
== Sweep photo-z ==<br />
<br />
=== Raw data ===<br />
<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/north/sweep/<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/south/sweep/<br />
<br />
Checksums checked:<br />
<br />
$ cd north/sweep/8.0<br />
$ ls | wc -l<br />
287<br />
$ wc -l legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
286 (+ checksum file)<br />
$ sha256sum -c legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
OK<br />
<br />
$ cd north/sweep/8.0-photo-z<br />
$ ls | wc -l<br />
287<br />
$ wc -l legacysurvey_dr8_north_sweep_8.0-photo-z.sha256sum<br />
286 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_north_sweep_8.0-photo-z.sha256sum<br />
OK<br />
<br />
$ cd south/sweep/8.0<br />
$ ls | wc -l<br />
438<br />
$ wc -l legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
437 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
OK<br />
<br />
$ cd south/sweep/8.0-photo-z<br />
$ ls | wc -l<br />
438<br />
$ wc -l legacysurvey_dr8_south_sweep_8.0-photo-z.sha256sum<br />
437 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_south_sweep_8.0-photo-z.sha256sum<br />
OK<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to merge fits files and convert to parquet:<br />
<br />
path_base = '/cephfs/pic.es/astro/scratch/tallada/dr8-phz/'<br />
hdfs_base = '/user/tallada/data/legacysurvey-sweep'<br />
<br />
hs = ['south/sweep'] #, 'north/sweep']<br />
dir_src = '8.0'<br />
dir_phz = '8.0-photo-z'<br />
dir_dst = '8.0-merged'<br />
<br />
my_files = sorted(os.listdir(os.path.join(path_base, hs[0], dir_src)))<br />
<br />
def merge_phz(file_name):<br />
file_src = os.path.join(path_base, hs[0], dir_src, file_name)<br />
file_phz = os.path.join(path_base, hs[0], dir_phz, file_name[:-5]+'-pz.fits')<br />
file_dst = os.path.join(path_base, hs[0], dir_dst, file_name[:-5]+'-merged.fits')<br />
<br />
t_src = table.Table.read(file_src)<br />
t_phz = table.Table.read(file_phz)<br />
t_dst = table.hstack([t_src, t_phz], join_type='exact')<br />
<br />
os.unlink(file_dst)<br />
t_dst.write(file_dst)<br />
<br />
return file_name<br />
<br />
df = sc.parallelize(my_files, len(my_files))<br />
<br />
df.map(merge_phz).count()<br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + os.path.join(path_base, hs[0], dir_dst)<br />
) <br />
<br />
df.write.parquet(os.path.join(hdfs_base, hs[0]))<br />
<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE tallada.dr8_sweep (<br />
`RELEASE` SMALLINT,<br />
`BRICKID` INT,<br />
`BRICKNAME` STRING,<br />
`OBJID` INT,<br />
`TYPE` STRING,<br />
`RA` DOUBLE,<br />
`DEC` DOUBLE,<br />
`RA_IVAR` FLOAT,<br />
`DEC_IVAR` FLOAT,<br />
`DCHISQ` ARRAY<FLOAT>,<br />
`EBV` FLOAT,<br />
`FLUX_G` FLOAT,<br />
`FLUX_R` FLOAT,<br />
`FLUX_Z` FLOAT,<br />
`FLUX_W1` FLOAT,<br />
`FLUX_W2` FLOAT,<br />
`FLUX_W3` FLOAT,<br />
`FLUX_W4` FLOAT,<br />
`FLUX_IVAR_G` FLOAT,<br />
`FLUX_IVAR_R` FLOAT,<br />
`FLUX_IVAR_Z` FLOAT,<br />
`FLUX_IVAR_W1` FLOAT,<br />
`FLUX_IVAR_W2` FLOAT,<br />
`FLUX_IVAR_W3` FLOAT,<br />
`FLUX_IVAR_W4` FLOAT,<br />
`MW_TRANSMISSION_G` FLOAT,<br />
`MW_TRANSMISSION_R` FLOAT,<br />
`MW_TRANSMISSION_Z` FLOAT,<br />
`MW_TRANSMISSION_W1` FLOAT,<br />
`MW_TRANSMISSION_W2` FLOAT,<br />
`MW_TRANSMISSION_W3` FLOAT,<br />
`MW_TRANSMISSION_W4` FLOAT,<br />
`NOBS_G` INT,<br />
`NOBS_R` INT,<br />
`NOBS_Z` INT,<br />
`NOBS_W1` INT,<br />
`NOBS_W2` INT,<br />
`NOBS_W3` INT,<br />
`NOBS_W4` INT,<br />
`RCHISQ_G` FLOAT,<br />
`RCHISQ_R` FLOAT,<br />
`RCHISQ_Z` FLOAT,<br />
`RCHISQ_W1` FLOAT,<br />
`RCHISQ_W2` FLOAT,<br />
`RCHISQ_W3` FLOAT,<br />
`RCHISQ_W4` FLOAT,<br />
`FRACFLUX_G` FLOAT,<br />
`FRACFLUX_R` FLOAT,<br />
`FRACFLUX_Z` FLOAT,<br />
`FRACFLUX_W1` FLOAT,<br />
`FRACFLUX_W2` FLOAT,<br />
`FRACFLUX_W3` FLOAT,<br />
`FRACFLUX_W4` FLOAT,<br />
`FRACMASKED_G` FLOAT,<br />
`FRACMASKED_R` FLOAT,<br />
`FRACMASKED_Z` FLOAT,<br />
`FRACIN_G` FLOAT,<br />
`FRACIN_R` FLOAT,<br />
`FRACIN_Z` FLOAT,<br />
`ANYMASK_G` SMALLINT,<br />
`ANYMASK_R` SMALLINT,<br />
`ANYMASK_Z` SMALLINT,<br />
`ALLMASK_G` SMALLINT,<br />
`ALLMASK_R` SMALLINT,<br />
`ALLMASK_Z` SMALLINT,<br />
`WISEMASK_W1` TINYINT,<br />
`WISEMASK_W2` TINYINT,<br />
`PSFSIZE_G` FLOAT,<br />
`PSFSIZE_R` FLOAT,<br />
`PSFSIZE_Z` FLOAT,<br />
`PSFDEPTH_G` FLOAT,<br />
`PSFDEPTH_R` FLOAT,<br />
`PSFDEPTH_Z` FLOAT,<br />
`GALDEPTH_G` FLOAT,<br />
`GALDEPTH_R` FLOAT,<br />
`GALDEPTH_Z` FLOAT,<br />
`PSFDEPTH_W1` FLOAT,<br />
`PSFDEPTH_W2` FLOAT,<br />
`WISE_COADD_ID` STRING,<br />
`FRACDEV` FLOAT,<br />
`FRACDEV_IVAR` FLOAT,<br />
`SHAPEDEV_R` FLOAT,<br />
`SHAPEDEV_R_IVAR` FLOAT,<br />
`SHAPEDEV_E1` FLOAT,<br />
`SHAPEDEV_E1_IVAR` FLOAT,<br />
`SHAPEDEV_E2` FLOAT,<br />
`SHAPEDEV_E2_IVAR` FLOAT,<br />
`SHAPEEXP_R` FLOAT,<br />
`SHAPEEXP_R_IVAR` FLOAT,<br />
`SHAPEEXP_E1` FLOAT,<br />
`SHAPEEXP_E1_IVAR` FLOAT,<br />
`SHAPEEXP_E2` FLOAT,<br />
`SHAPEEXP_E2_IVAR` FLOAT,<br />
`FIBERFLUX_G` FLOAT,<br />
`FIBERFLUX_R` FLOAT,<br />
`FIBERFLUX_Z` FLOAT,<br />
`FIBERTOTFLUX_G` FLOAT,<br />
`FIBERTOTFLUX_R` FLOAT,<br />
`FIBERTOTFLUX_Z` FLOAT,<br />
`REF_CAT` STRING,<br />
`REF_ID` BIGINT,<br />
`REF_EPOCH` FLOAT,<br />
`GAIA_PHOT_G_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_G_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_ASTROMETRIC_EXCESS_NOISE` FLOAT,<br />
`GAIA_DUPLICATED_SOURCE` BOOLEAN,<br />
`GAIA_PHOT_BP_RP_EXCESS_FACTOR` FLOAT,<br />
`GAIA_ASTROMETRIC_SIGMA5D_MAX` FLOAT,<br />
`GAIA_ASTROMETRIC_PARAMS_SOLVED` TINYINT,<br />
`PARALLAX` FLOAT,<br />
`PARALLAX_IVAR` FLOAT,<br />
`PMRA` FLOAT,<br />
`PMRA_IVAR` FLOAT,<br />
`PMDEC` FLOAT,<br />
`PMDEC_IVAR` FLOAT,<br />
`MASKBITS` SMALLINT,<br />
`z_phot_mean` FLOAT,<br />
`z_phot_median` FLOAT,<br />
`z_phot_std` FLOAT,<br />
`z_phot_l68` FLOAT,<br />
`z_phot_u68` FLOAT,<br />
`z_phot_l95` FLOAT,<br />
`z_phot_u95` FLOAT,<br />
`z_spec` FLOAT,<br />
`survey` STRING,<br />
`training` BOOLEAN<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey-sweep'<br />
;<br />
<br />
=== Combined table ===<br />
<br />
CREATE TABLE cosmohub.legacysurvey_dr8_phz (<br />
release smallint COMMENT 'Unique integer denoting the camera and filter set used',<br />
brickid int COMMENT 'Brick ID [1,662174]',<br />
brickname string COMMENT 'Name of brick, encoding the brick sky position, eg "1126p222" near RA=112.6, Dec=+22.2',<br />
objid int COMMENT 'Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick',<br />
brick_primary boolean COMMENT 'True if the object is within the brick boundary',<br />
brightblob smallint COMMENT 'bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page',<br />
maskbits smallint COMMENT 'bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits* maps, as cataloged on the DR8 bitmasks page',<br />
type string COMMENT 'Morphological model: "PSF"=stellar, "REX"="round exponential galaxy", "DEV"=deVauc, "EXP"=exponential, "COMP"=composite, "DUP"=Gaia source fit by different model. Note that in some FITS readers, a trailing space may be appended for "PSF ", "DEV " and "EXP " since the column data type is a 4-character string',<br />
ra double COMMENT 'Right ascension at equinox J2000',<br />
dec double COMMENT 'Declination at equinox J2000',<br />
ra_ivar float COMMENT 'Inverse variance of RA (no cosine term!), excluding astrometric calibration errors',<br />
dec_ivar float COMMENT 'Inverse variance of DEC, excluding astrometric calibration errors',<br />
bx float COMMENT 'X position (0-indexed) of coordinates in brick image stack',<br />
`by` float COMMENT 'Y position (0-indexed) of coordinates in brick image stack',<br />
dchisq array<float> COMMENT 'Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.',<br />
ebv float COMMENT 'Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns',<br />
mjd_min double COMMENT 'Minimum Modified Julian Date of observations used to construct the model of this object',<br />
mjd_max double COMMENT 'Maximum Modified Julian Date of observations used to construct the model of this object',<br />
ref_cat string COMMENT 'Reference catalog source for this star: "T2" for Tycho-2, "G2" for Gaia DR2, "L2" for the LSLGA, empty otherwise',<br />
ref_id bigint COMMENT 'Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b "sourceid" for Gaia-DR2 and LSLGA',<br />
pmra float COMMENT 'Reference catalog proper motion in the RA direction',<br />
pmdec float COMMENT 'Reference catalog proper motion in the Dec direction',<br />
parallax float COMMENT 'Reference catalog parallax',<br />
pmra_ivar float COMMENT 'Reference catalog inverse-variance on pmra',<br />
pmdec_ivar float COMMENT 'Reference catalog inverse-variance on pmdec',<br />
parallax_ivar float COMMENT 'Reference catalog inverse-variance on parallax',<br />
ref_epoch float COMMENT 'Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)',<br />
gaia_pointsource boolean COMMENT 'This Gaia DR2 source is believed to be a star, not a galaxy',<br />
gaia_phot_g_mean_mag float COMMENT 'Gaia G band mag',<br />
gaia_phot_g_mean_flux_over_error float COMMENT 'Gaia G band signal-to-noise',<br />
gaia_phot_g_n_obs smallint COMMENT 'Gaia G band number of observations',<br />
gaia_phot_bp_mean_mag float COMMENT 'Gaia BP mag',<br />
gaia_phot_bp_mean_flux_over_error float COMMENT 'Gaia BP signal-to-noise',<br />
gaia_phot_bp_n_obs smallint COMMENT 'Gaia BP number of observations',<br />
gaia_phot_rp_mean_mag float COMMENT 'Gaia RP mag',<br />
gaia_phot_rp_mean_flux_over_error float COMMENT 'Gaia RP signal-to-noise',<br />
gaia_phot_rp_n_obs smallint COMMENT 'Gaia RP number of observations',<br />
gaia_phot_variable_flag boolean COMMENT 'Gaia photometric variable flag',<br />
gaia_astrometric_excess_noise float COMMENT 'Gaia astrometric excess noise',<br />
gaia_astrometric_excess_noise_sig float COMMENT 'Gaia astrometric excess noise uncertainty',<br />
gaia_astrometric_n_obs_al smallint COMMENT 'Gaia number of astrometric observations along scan direction',<br />
gaia_astrometric_n_good_obs_al smallint COMMENT 'Gaia number of good astrometric observations along scan direction',<br />
gaia_astrometric_weight_al float COMMENT 'Gaia astrometric weight along scan direction',<br />
gaia_duplicated_source boolean COMMENT 'Gaia duplicated source flag',<br />
gaia_a_g_val float COMMENT 'Gaia line-of-sight extinction in the G band',<br />
gaia_e_bp_min_rp_val float COMMENT 'Gaia line-of-sight reddening E(BP-RP)',<br />
gaia_phot_bp_rp_excess_factor float COMMENT 'Gaia BP/RP excess factor',<br />
gaia_astrometric_sigma5d_max float COMMENT 'Gaia longest semi-major axis of the 5-d error ellipsoid',<br />
gaia_astrometric_params_solved tinyint COMMENT 'which astrometric parameters were estimated for a Gaia source',<br />
flux_g float COMMENT 'model flux in gg',<br />
flux_r float COMMENT 'model flux in rr',<br />
flux_z float COMMENT 'model flux in zz',<br />
flux_w1 float COMMENT 'WISE model flux in W1W1 (AB system)',<br />
flux_w2 float COMMENT 'WISE model flux in W2W2 (AB)',<br />
flux_w3 float COMMENT 'WISE model flux in W3W3 (AB)',<br />
flux_w4 float COMMENT 'WISE model flux in W4W4 (AB)',<br />
flux_ivar_g float COMMENT 'Inverse variance of flux_g',<br />
flux_ivar_r float COMMENT 'Inverse variance of flux_r',<br />
flux_ivar_z float COMMENT 'Inverse variance of flux_z',<br />
flux_ivar_w1 float COMMENT 'Inverse variance of flux_w1 (AB system)',<br />
flux_ivar_w2 float COMMENT 'Inverse variance of flux_w2 (AB)',<br />
flux_ivar_w3 float COMMENT 'Inverse variance of flux_w3 (AB)',<br />
flux_ivar_w4 float COMMENT 'Inverse variance of flux_w4 (AB)',<br />
fiberflux_g float COMMENT 'Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_r float COMMENT 'Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_z float COMMENT 'Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fibertotflux_g float COMMENT 'Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_r float COMMENT 'Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_z float COMMENT 'Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
apflux_g array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg',<br />
apflux_r array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr',<br />
apflux_z array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz',<br />
apflux_resid_g array<float> COMMENT 'aperture fluxes on the co-added residual images in gg',<br />
apflux_resid_r array<float> COMMENT 'aperture fluxes on the co-added residual images in rr',<br />
apflux_resid_z array<float> COMMENT 'aperture fluxes on the co-added residual images in zz',<br />
apflux_ivar_g array<float> COMMENT 'Inverse variance of apflux_resid_g',<br />
apflux_ivar_r array<float> COMMENT 'Inverse variance of apflux_resid_r',<br />
apflux_ivar_z array<float> COMMENT 'Inverse variance of apflux_resid_z',<br />
mw_transmission_g float COMMENT 'Galactic transmission in gg filter in linear units [0, 1]',<br />
mw_transmission_r float COMMENT 'Galactic transmission in rr filter in linear units [0, 1]',<br />
mw_transmission_z float COMMENT 'Galactic transmission in zz filter in linear units [0, 1]',<br />
mw_transmission_w1 float COMMENT 'Galactic transmission in W1W1 filter in linear units [0, 1]',<br />
mw_transmission_w2 float COMMENT 'Galactic transmission in W2W2 filter in linear units [0, 1]',<br />
mw_transmission_w3 float COMMENT 'Galactic transmission in W3W3 filter in linear units [0, 1]',<br />
mw_transmission_w4 float COMMENT 'Galactic transmission in W4W4 filter in linear units [0, 1]',<br />
nobs_g smallint COMMENT 'Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)',<br />
nobs_r smallint COMMENT 'Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)',<br />
nobs_z smallint COMMENT 'Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)',<br />
nobs_w1 smallint COMMENT 'Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)',<br />
nobs_w2 smallint COMMENT 'Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)',<br />
nobs_w3 smallint COMMENT 'Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)',<br />
nobs_w4 smallint COMMENT 'Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)',<br />
rchisq_g float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in gg',<br />
rchisq_r float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in rr',<br />
rchisq_z float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in zz',<br />
rchisq_w1 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W1W1',<br />
rchisq_w2 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W2W2',<br />
rchisq_w3 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W3W3',<br />
rchisq_w4 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W4W4',<br />
fracflux_g float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in gg (typically [0,1])',<br />
fracflux_r float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in rr (typically [0,1])',<br />
fracflux_z float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in zz (typically [0,1])',<br />
fracflux_w1 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1 (typically [0,1])',<br />
fracflux_w2 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2 (typically [0,1])',<br />
fracflux_w3 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3 (typically [0,1])',<br />
fracflux_w4 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4 (typically [0,1])',<br />
fracmasked_g float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]',<br />
fracmasked_r float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]',<br />
fracmasked_z float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]',<br />
fracin_g float COMMENT 'Fraction of a source\'s flux within the blob in gg, near unity for real sources',<br />
fracin_r float COMMENT 'Fraction of a source\'s flux within the blob in rr, near unity for real sources',<br />
fracin_z float COMMENT 'Fraction of a source\'s flux within the blob in zz, near unity for real sources',<br />
anymask_g smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page',<br />
anymask_r smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page',<br />
anymask_z smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page',<br />
allmask_g smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page',<br />
allmask_r smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page',<br />
allmask_z smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page',<br />
wisemask_w1 tinyint COMMENT 'W1 bitmask as cataloged on the DR8 bitmasks page',<br />
wisemask_w2 tinyint COMMENT 'W2 bitmask as cataloged on the DR8 bitmasks page',<br />
psfsize_g float COMMENT 'Weighted average PSF FWHM in the gg band',<br />
psfsize_r float COMMENT 'Weighted average PSF FWHM in the rr band',<br />
psfsize_z float COMMENT 'Weighted average PSF FWHM in the zz band',<br />
psfdepth_g float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude',<br />
psfdepth_r float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude',<br />
psfdepth_z float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude',<br />
galdepth_g float COMMENT 'As for psfdepth_g but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_r float COMMENT 'As for psfdepth_r but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_z float COMMENT 'As for psfdepth_z but for a galaxy (0.45" exp, round) detection sensitivity',<br />
psfdepth_w1 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W1',<br />
psfdepth_w2 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W2',<br />
psfdepth_w3 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W3',<br />
psfdepth_w4 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W4',<br />
wise_coadd_id string COMMENT 'unWISE coadd file name for the center of each object',<br />
lc_flux_w1 array<float> COMMENT 'flux_w1 in each of up to eleven unWISE coadd epochs (AB system)',<br />
lc_flux_w2 array<float> COMMENT 'flux_w2 in each of up to eleven unWISE coadd epochs (AB)',<br />
lc_flux_ivar_w1 array<float> COMMENT 'Inverse variance of lc_flux_w1 (AB system)',<br />
lc_flux_ivar_w2 array<float> COMMENT 'Inverse variance of lc_flux_w2 (AB)',<br />
lc_nobs_w1 array<smallint> COMMENT 'nobs_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_nobs_w2 array<smallint> COMMENT 'nobs_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w1 array<float> COMMENT 'fracflux_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w2 array<float> COMMENT 'fracflux_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w1 array<float> COMMENT 'rchisq_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w2 array<float> COMMENT 'rchisq_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w1 array<double> COMMENT 'mjd_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w2 array<double> COMMENT 'mjd_w2 in each of up to eleven unWISE coadd epochs',<br />
fracdev float COMMENT 'Fraction of model in deVauc [0,1]',<br />
fracdev_ivar float COMMENT 'Inverse variance of fracdev',<br />
shapeexp_r float COMMENT 'Half-light radius of exponential model (>0)',<br />
shapeexp_r_ivar float COMMENT 'Inverse variance of shapeexp_r',<br />
shapeexp_e1 float COMMENT 'Ellipticity component 1',<br />
shapeexp_e1_ivar float COMMENT 'Inverse variance of shapeexp_e1',<br />
shapeexp_e2 float COMMENT 'Ellipticity component 2',<br />
shapeexp_e2_ivar float COMMENT 'Inverse variance of shapeexp_e2',<br />
shapedev_r float COMMENT 'Half-light radius of deVaucouleurs model (>0)',<br />
shapedev_r_ivar float COMMENT 'Inverse variance of shapedev_r',<br />
shapedev_e1 float COMMENT 'Ellipticity component 1',<br />
shapedev_e1_ivar float COMMENT 'Inverse variance of shapedev_e1',<br />
shapedev_e2 float COMMENT 'Ellipticity component 2',<br />
shapedev_e2_ivar float COMMENT 'Inverse variance of shapedev_e2',<br />
z_phot_mean FLOAT COMMENT 'photo-z derived from the mean of the photo-z PDF ',<br />
z_phot_median FLOAT COMMENT 'photo-z derived from the median of the photo-z PDF',<br />
z_phot_std FLOAT COMMENT 'standard deviation of the photo-z\'s derived from the photo-z PDF',<br />
z_phot_l68 FLOAT COMMENT 'lower bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_u68 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_l95 FLOAT COMMENT 'lower bound of the 95% confidence region, derived from the photo-z PDF',<br />
z_phot_u95 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_spec FLOAT COMMENT 'spectroscopic redshift, if available',<br />
survey STRING COMMENT 'source of the spectroscopic redshift',<br />
training BOOLEAN COMMENT 'whether or not the spectroscopic redshift is used in photometric redshift training'<br />
)<br />
CLUSTERED BY (<br />
release,<br />
brickid,<br />
objid)<br />
SORTED BY (<br />
release ASC,<br />
brickid ASC,<br />
objid ASC)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
=== Join insert ===<br />
<br />
INSERT OVERWRITE TABLE cosmohub.legacy_survey_dr8_phz<br />
SELECT<br />
dr8.*,<br />
phz.z_phot_mean,<br />
phz.z_phot_median,<br />
phz.z_phot_std,<br />
phz.z_phot_l68,<br />
phz.z_phot_u68,<br />
phz.z_phot_l95,<br />
phz.z_phot_u95,<br />
phz.z_spec,<br />
phz.survey,<br />
phz.training<br />
FROM cosmohub.legacy_survey_dr8 AS dr8<br />
LEFT JOIN tallada.dr8_sweep AS phz<br />
ON dr8.release = phz.release<br />
AND dr8.brickid = phz.brickid<br />
AND dr8.objid = phz.objid<br />
;<br />
<br />
== Queries for checking ==<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT release, brickid, objid, COUNT(release)<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
HAVING COUNT(release) > 1<br />
<br />
DR8 release brickid objid: DISTINCT(1.694.230.368) TOTAL(1.694.230.368)<br />
DR8 brickid objid: DISTINCT(1.654.429.115) TOTAL(1.694.230.368)<br />
<br />
PHZ release brickid objid: DISTINCT(1.646.664.729) TOTAL(1.646.664.897)<br />
PHZ brickid objid: DISTINCT(1.608.841.582) TOTAL(1.646.664.897)<br />
<br />
DR8: 1.694.230.368<br />
PHZ: 1.646.664.897<br />
----------------------------------------------------------------------<br />
-47.565.471</div>Tallada