https://pwiki.pic.es/api.php?action=feedcontributions&feedformat=atom&user=Tallada
Public PIC Wiki - User contributions [en]
2026-04-20T13:05:54Z
User contributions
MediaWiki 1.35.14
https://pwiki.pic.es/index.php?title=Faq&diff=1352
Faq
2026-04-15T15:05:17Z
<p>Tallada: </p>
<hr />
<div>== How do I reset my password? ==<br />
You can reset your password using the following link:<br />
https://www.pic.es/user/auth/forgotpw<br />
<br />
== Can an undergraduate student in my group have an account? ==<br />
Yes. Undergraduate students can have PIC accounts without any problem.<br />
<br />
== How do I get a Kerberos token? ==<br />
A valid token is needed to interface with some PIC services, like the Hadoop platform (HDFS, Hive, Spark, ...) or CTA.<br />
Tokens are issued for the machine in which they are requested, if you hop to another machine, you may need to get another token.<br />
<br />
In order to get a token, run the following commands on a terminal.<br />
If you are on JupyterLab, use ''Launcher'' → ''Terminal''.<br />
<br />
<pre><br />
kinit -n -c ~/.fast.ccache @PIC.ES<br />
kinit -T ~/.fast.ccache<br />
</pre><br />
<br />
You could also define an alias for your own convenience:<br />
<br />
<pre><br />
alias kinit="kinit -n -c ~/.fast.ccache @PIC.ES; kinit -T ~/.fast.ccache"<br />
</pre><br />
<br />
Optionally, you can check the presence and expiration date of your token using:<br />
<br />
<pre><br />
klist<br />
</pre><br />
<br />
== I lost access to my OTP, how can I reset it? ==<br />
Follow these steps to recover your account and configure a new OTP token.<br />
<br />
# On the login page, click '''Forgot password / OTP''' or go [https://www.pic.es/reset here].<br />
# Enter your username. You should receive a recovery email in your inbox.<br />
# Open the email and follow the recovery link.<br />
# Scan the new QR code with your authenticator app.<br />
# Enter the generated OTP code and a device name.<br />
#* The device name is mandatory.<br />
# Change your password when prompted.<br />
#* This is required for security reasons.<br />
# Log in again as usual.<br />
#* On the OTP selection page, you will see two devices.<br />
#* Select the device with the name you just entered.<br />
#* The old one will usually appear as '''unnamed''' (if no name was set previously).<br />
<br />
Afterwards, we recommend removing the old OTP factor from the [https://idp.pic.es/realms/PIC/account/account-security/signing-in user profile section].<br />
<br />
== What is the simplest way to avoid repeated SSH authentication? ==<br />
The easiest way is to configure your SSH connection using '''ControlMaster''', which allows multiple sessions to reuse a single authenticated connection.<br />
<br />
On your client machine, edit the file <code>~/.ssh/config</code> and add the following lines:<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Instead of <code>Host *</code>, you can restrict this to a specific host, for example:<br />
<br />
<pre><br />
Host ui04.pic.es<br />
</pre><br />
<br />
Initiate the first SSH connection and authenticate via the provided URL. You will see something like:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Complete the authentication in your browser, then press ENTER.<br />
Once authenticated, the SSH connection remains open in the background, and you can safely close your terminal session.<br />
The connection will persist indefinitely (until machine reboot or a network interruption).<br />
<br />
Subsequent SSH connections to the same host will reuse the existing session and will not require authentication again:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205<br />
[tallada@ui04 ~]$<br />
</pre><br />
</li><br />
</ol></div>
Tallada
https://pwiki.pic.es/index.php?title=Faq&diff=1351
Faq
2026-04-15T14:09:44Z
<p>Tallada: </p>
<hr />
<div>== How do I reset my password? ==<br />
You can reset your password using the following link:<br />
https://www.pic.es/user/auth/forgotpw<br />
<br />
== Can an undergraduate student in my group have an account? ==<br />
Yes. Undergraduate students can have PIC accounts without any problem.<br />
<br />
== How do I get a Kerberos token? ==<br />
A valid token is needed to interface with some PIC services, like the Hadoop platform (HDFS, Hive, Spark, ...) or CTA.<br />
Tokens are issued for the machine in which they are requested, if you hop to another machine, you may need to get another token.<br />
<br />
In order to get a token, run the following commands on a terminal.<br />
If you are on JupyterLab, use ''Launcher'' → ''Terminal''.<br />
<br />
<pre><br />
kinit -n -c ~/.fast.ccache @PIC.ES<br />
kinit -T ~/.fast.ccache<br />
</pre><br />
<br />
You could also define an alias for your own convenience:<br />
<br />
<pre><br />
alias kinit="kinit -n -c ~/.fast.ccache @PIC.ES; kinit -T ~/.fast.ccache"<br />
</pre><br />
<br />
Optionally, you can check the presence and expiration date of your token using:<br />
<br />
<pre><br />
klist<br />
</pre><br />
<br />
== I lost access to my OTP, how can I reset it? ==<br />
Follow these steps to recover your account and configure a new OTP token.<br />
<br />
# On the login page, click '''Forgot password / OTP''' or go [https://www.pic.es/reset here].<br />
# Enter your username. You should receive a recovery email in your inbox.<br />
# Open the email and follow the recovery link.<br />
# Scan the new QR code with your authenticator app.<br />
# Enter the generated OTP code and a device name.<br />
#* The device name is mandatory.<br />
# Change your password when prompted.<br />
#* This is required for security reasons.<br />
# Log in again as usual.<br />
#* On the OTP selection page, you will see two devices.<br />
#* Select the device with the name you just entered.<br />
#* The old one will usually appear as '''unnamed''' (if no name was set previously).<br />
<br />
Afterwards, we recommend removing the old OTP factor from the [https://idp.pic.es/realms/PIC/account/account-security/signing-in user profile section].<br />
<br />
=== What is the simplest way to avoid repeated SSH authentication? ===<br />
The easiest way is to configure your SSH connection using '''ControlMaster''', which allows multiple sessions to reuse a single authenticated connection.<br />
<br />
On your client machine, edit the file <code>~/.ssh/config</code> and add the following lines:<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Instead of <code>Host *</code>, you can restrict this to a specific host, for example:<br />
<br />
<pre><br />
Host ui04.pic.es<br />
</pre><br />
<br />
Initiate the first SSH connection and authenticate via the provided URL. You will see something like:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Complete the authentication in your browser, then press ENTER.<br />
Once authenticated, the SSH connection remains open in the background, and you can safely close your terminal session.<br />
The connection will persist indefinitely (until machine reboot or a network interruption).<br />
<br />
Subsequent SSH connections to the same host will reuse the existing session and will not require authentication again:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205<br />
[tallada@ui04 ~]$<br />
</pre><br />
</li><br />
</ol></div>
Tallada
https://pwiki.pic.es/index.php?title=Faq&diff=1350
Faq
2026-04-15T13:57:45Z
<p>Tallada: /* What is the simplest way to avoid repeated SSH authentication? */</p>
<hr />
<div>== How do I reset my password? ==<br />
You can reset your password using the following link:<br />
https://www.pic.es/user/auth/forgotpw<br />
<br />
== Can an undergraduate student in my group have an account? ==<br />
Yes. Undergraduate students can have PIC accounts without any problem.<br />
<br />
== How do I get a Kerberos token? ==<br />
A valid token is needed to interface with some PIC services, like the Hadoop platform (HDFS, Hive, Spark, ...) or CTA.<br />
Tokens are issued for the machine in which they are requested, if you hop to another machine, you may need to get another token.<br />
<br />
In order to get a token, run the following commands on a terminal.<br />
If you are on JupyterLab, use ''Launcher'' → ''Terminal''.<br />
<br />
<pre><br />
kinit -n -c ~/.fast.ccache @PIC.ES<br />
kinit -T ~/.fast.ccache<br />
</pre><br />
<br />
You could also define an alias for your own convenience:<br />
<br />
<pre><br />
alias kinit="kinit -n -c ~/.fast.ccache @PIC.ES; kinit -T ~/.fast.ccache"<br />
</pre><br />
<br />
Optionally, you can check the presence and expiration date of your token using:<br />
<br />
<pre><br />
klist<br />
</pre><br />
<br />
== I lost access to my OTP, how can I reset it? ==<br />
Follow these steps to recover your account and configure a new OTP token.<br />
<br />
# On the Keycloak login page, click '''Forgot password / OTP'''.<br />
# Enter your username. You should receive a recovery email in your inbox.<br />
# Open the email and follow the recovery link.<br />
# Scan the new QR code with your authenticator app.<br />
# Enter the generated OTP code and a device name.<br />
#* The device name is mandatory.<br />
# Change your password when prompted.<br />
#* This is required for security reasons.<br />
# Log in again as usual.<br />
#* On the OTP selection page, you will see two devices.<br />
#* Select the device with the name you just entered.<br />
#* The old one will usually appear as '''unnamed''' (if no name was set previously).<br />
<br />
Afterwards, we recommend removing the old OTP factor from the [https://idp.pic.es/realms/PIC/account/account-security/signing-in user profile section].<br />
<br />
=== What is the simplest way to avoid repeated SSH authentication? ===<br />
The easiest way is to configure your SSH connection using '''ControlMaster''', which allows multiple sessions to reuse a single authenticated connection.<br />
<br />
On your client machine, edit the file <code>~/.ssh/config</code> and add the following lines:<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Instead of <code>Host *</code>, you can restrict this to a specific host, for example:<br />
<br />
<pre><br />
Host ui04.pic.es<br />
</pre><br />
<br />
Initiate the first SSH connection and authenticate via the provided URL. You will see something like:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Complete the authentication in your browser, then press ENTER.<br />
Once authenticated, the SSH connection remains open in the background, and you can safely close your terminal session.<br />
The connection will persist indefinitely (until machine reboot or a network interruption).<br />
<br />
Subsequent SSH connections to the same host will reuse the existing session and will not require authentication again:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205<br />
[tallada@ui04 ~]$<br />
</pre><br />
</li><br />
</ol></div>
Tallada
https://pwiki.pic.es/index.php?title=Faq&diff=1349
Faq
2026-04-15T13:57:33Z
<p>Tallada: /* What is the simplest way to avoid repeated SSH authentication? */</p>
<hr />
<div>== How do I reset my password? ==<br />
You can reset your password using the following link:<br />
https://www.pic.es/user/auth/forgotpw<br />
<br />
== Can an undergraduate student in my group have an account? ==<br />
Yes. Undergraduate students can have PIC accounts without any problem.<br />
<br />
== How do I get a Kerberos token? ==<br />
A valid token is needed to interface with some PIC services, like the Hadoop platform (HDFS, Hive, Spark, ...) or CTA.<br />
Tokens are issued for the machine in which they are requested, if you hop to another machine, you may need to get another token.<br />
<br />
In order to get a token, run the following commands on a terminal.<br />
If you are on JupyterLab, use ''Launcher'' → ''Terminal''.<br />
<br />
<pre><br />
kinit -n -c ~/.fast.ccache @PIC.ES<br />
kinit -T ~/.fast.ccache<br />
</pre><br />
<br />
You could also define an alias for your own convenience:<br />
<br />
<pre><br />
alias kinit="kinit -n -c ~/.fast.ccache @PIC.ES; kinit -T ~/.fast.ccache"<br />
</pre><br />
<br />
Optionally, you can check the presence and expiration date of your token using:<br />
<br />
<pre><br />
klist<br />
</pre><br />
<br />
== I lost access to my OTP, how can I reset it? ==<br />
Follow these steps to recover your account and configure a new OTP token.<br />
<br />
# On the Keycloak login page, click '''Forgot password / OTP'''.<br />
# Enter your username. You should receive a recovery email in your inbox.<br />
# Open the email and follow the recovery link.<br />
# Scan the new QR code with your authenticator app.<br />
# Enter the generated OTP code and a device name.<br />
#* The device name is mandatory.<br />
# Change your password when prompted.<br />
#* This is required for security reasons.<br />
# Log in again as usual.<br />
#* On the OTP selection page, you will see two devices.<br />
#* Select the device with the name you just entered.<br />
#* The old one will usually appear as '''unnamed''' (if no name was set previously).<br />
<br />
Afterwards, we recommend removing the old OTP factor from the [https://idp.pic.es/realms/PIC/account/account-security/signing-in user profile section].<br />
<br />
=== What is the simplest way to avoid repeated SSH authentication? ===<br />
The easiest way is to configure your SSH connection using '''ControlMaster''', which allows multiple sessions to reuse a single authenticated connection.<br />
<br />
On your client machine, edit the file <code>~/.ssh/config</code> and add the following lines:<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Instead of <code>Host *</code>, you can restrict this to a specific host, for example:<br />
<br />
<pre><br />
Host ui04.pic.es<br />
</pre><br />
<br />
<br />
Initiate the first SSH connection and authenticate via the provided URL. You will see something like:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Complete the authentication in your browser, then press ENTER.<br />
Once authenticated, the SSH connection remains open in the background, and you can safely close your terminal session.<br />
The connection will persist indefinitely (until machine reboot or a network interruption).<br />
<br />
Subsequent SSH connections to the same host will reuse the existing session and will not require authentication again:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205<br />
[tallada@ui04 ~]$<br />
</pre><br />
</li><br />
</ol></div>
Tallada
https://pwiki.pic.es/index.php?title=Faq&diff=1348
Faq
2026-04-15T13:57:07Z
<p>Tallada: /* What is the simplest way to avoid repeated SSH authentication? */</p>
<hr />
<div>== How do I reset my password? ==<br />
You can reset your password using the following link:<br />
https://www.pic.es/user/auth/forgotpw<br />
<br />
== Can an undergraduate student in my group have an account? ==<br />
Yes. Undergraduate students can have PIC accounts without any problem.<br />
<br />
== How do I get a Kerberos token? ==<br />
A valid token is needed to interface with some PIC services, like the Hadoop platform (HDFS, Hive, Spark, ...) or CTA.<br />
Tokens are issued for the machine in which they are requested, if you hop to another machine, you may need to get another token.<br />
<br />
In order to get a token, run the following commands on a terminal.<br />
If you are on JupyterLab, use ''Launcher'' → ''Terminal''.<br />
<br />
<pre><br />
kinit -n -c ~/.fast.ccache @PIC.ES<br />
kinit -T ~/.fast.ccache<br />
</pre><br />
<br />
You could also define an alias for your own convenience:<br />
<br />
<pre><br />
alias kinit="kinit -n -c ~/.fast.ccache @PIC.ES; kinit -T ~/.fast.ccache"<br />
</pre><br />
<br />
Optionally, you can check the presence and expiration date of your token using:<br />
<br />
<pre><br />
klist<br />
</pre><br />
<br />
== I lost access to my OTP, how can I reset it? ==<br />
Follow these steps to recover your account and configure a new OTP token.<br />
<br />
# On the Keycloak login page, click '''Forgot password / OTP'''.<br />
# Enter your username. You should receive a recovery email in your inbox.<br />
# Open the email and follow the recovery link.<br />
# Scan the new QR code with your authenticator app.<br />
# Enter the generated OTP code and a device name.<br />
#* The device name is mandatory.<br />
# Change your password when prompted.<br />
#* This is required for security reasons.<br />
# Log in again as usual.<br />
#* On the OTP selection page, you will see two devices.<br />
#* Select the device with the name you just entered.<br />
#* The old one will usually appear as '''unnamed''' (if no name was set previously).<br />
<br />
Afterwards, we recommend removing the old OTP factor from the [https://idp.pic.es/realms/PIC/account/account-security/signing-in user profile section].<br />
<br />
=== What is the simplest way to avoid repeated SSH authentication? ===<br />
The easiest way is to configure your SSH connection using '''ControlMaster''', which allows multiple sessions to reuse a single authenticated connection.<br />
<br />
On your client machine, edit the file <code>~/.ssh/config</code> and add the following lines:<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Instead of <code>Host *</code>, you can restrict this to a specific host, for example:<br />
<br />
<pre><br />
Host ui04.pic.es<br />
</pre><br />
<br />
<br />
Initiate the first SSH connection and authenticate via the provided URL:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
</pre><br />
<br />
You will see something like:<br />
<br />
<pre><br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Complete the authentication in your browser, then press ENTER.<br />
Once authenticated, the SSH connection remains open in the background, and you can safely close your terminal session.<br />
The connection will persist indefinitely (until machine reboot or a network interruption).<br />
<br />
Subsequent SSH connections to the same host will reuse the existing session and will not require authentication again:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
Last login: Tue Apr 14 13:25:33 2026 from 10.212.134.205<br />
[tallada@ui04 ~]$<br />
</pre><br />
</li><br />
</ol></div>
Tallada
https://pwiki.pic.es/index.php?title=Faq&diff=1347
Faq
2026-04-15T13:55:37Z
<p>Tallada: </p>
<hr />
<div>== How do I reset my password? ==<br />
You can reset your password using the following link:<br />
https://www.pic.es/user/auth/forgotpw<br />
<br />
== Can an undergraduate student in my group have an account? ==<br />
Yes. Undergraduate students can have PIC accounts without any problem.<br />
<br />
== How do I get a Kerberos token? ==<br />
A valid token is needed to interface with some PIC services, like the Hadoop platform (HDFS, Hive, Spark, ...) or CTA.<br />
Tokens are issued for the machine in which they are requested, if you hop to another machine, you may need to get another token.<br />
<br />
In order to get a token, run the following commands on a terminal.<br />
If you are on JupyterLab, use ''Launcher'' → ''Terminal''.<br />
<br />
<pre><br />
kinit -n -c ~/.fast.ccache @PIC.ES<br />
kinit -T ~/.fast.ccache<br />
</pre><br />
<br />
You could also define an alias for your own convenience:<br />
<br />
<pre><br />
alias kinit="kinit -n -c ~/.fast.ccache @PIC.ES; kinit -T ~/.fast.ccache"<br />
</pre><br />
<br />
Optionally, you can check the presence and expiration date of your token using:<br />
<br />
<pre><br />
klist<br />
</pre><br />
<br />
== I lost access to my OTP, how can I reset it? ==<br />
Follow these steps to recover your account and configure a new OTP token.<br />
<br />
# On the Keycloak login page, click '''Forgot password / OTP'''.<br />
# Enter your username. You should receive a recovery email in your inbox.<br />
# Open the email and follow the recovery link.<br />
# Scan the new QR code with your authenticator app.<br />
# Enter the generated OTP code and a device name.<br />
#* The device name is mandatory.<br />
# Change your password when prompted.<br />
#* This is required for security reasons.<br />
# Log in again as usual.<br />
#* On the OTP selection page, you will see two devices.<br />
#* Select the device with the name you just entered.<br />
#* The old one will usually appear as '''unnamed''' (if no name was set previously).<br />
<br />
Afterwards, we recommend removing the old OTP factor from the [https://idp.pic.es/realms/PIC/account/account-security/signing-in user profile section].<br />
<br />
=== What is the simplest way to avoid repeated SSH authentication? ===<br />
The easiest way is to configure your SSH connection using '''ControlMaster''', which allows multiple sessions to reuse a single authenticated connection.<br />
<br />
<ol><br />
<li>On your client machine, edit the file <code>~/.ssh/config</code> and add the following lines:<br />
<pre><br />
Host *<br />
ControlMaster auto<br />
ControlPath ~/.ssh/cm-%r@%h:%p<br />
ControlPersist yes<br />
</pre><br />
<br />
Instead of <code>Host *</code>, you can restrict this to a specific host, for example:<br />
<br />
<pre><br />
Host ui04.pic.es<br />
</pre><br />
</li><br />
<br />
<li>Initiate the first SSH connection and authenticate via the provided URL:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
</pre><br />
<br />
You will see something like:<br />
<br />
<pre><br />
(tallada@ui04.pic.es) Authenticate at https://idp.pic.es/realms/PIC/device?user_code=FPTB-HKEV and press ENTER.<br />
</pre><br />
<br />
Complete the authentication in your browser, then press ENTER.<br />
</li><br />
<br />
<li>Once authenticated, the SSH connection remains open in the background, and you can safely close your terminal session.<br />
<br />
The connection will persist indefinitely (until machine reboot or a network interruption).<br />
</li><br />
<br />
<li>Subsequent SSH connections to the same host will reuse the existing session and will not require authentication again:<br />
<br />
<pre><br />
$ ssh ui04.pic.es<br />
</pre><br />
</li><br />
</ol></div>
Tallada
https://pwiki.pic.es/index.php?title=Main_Page&diff=1346
Main Page
2026-04-15T13:42:19Z
<p>Tallada: </p>
<hr />
<div>== Getting started ==<br />
* [[PIC description|Introduction to PIC]]<br />
* [[PIC account|Get a PIC account]]<br />
* [[PIC_User_Manual | User manual]]<br />
* [[faq| Frequently asked questions]]<br />
<br />
== Services ==<br />
=== User interfaces ===<br />
* [[Login machines]]<br />
* [[JupyterHub]]<br />
<br />
=== Distributed computing ===<br />
* [[HTCondor]]<br />
* [[Dask]]<br />
* Spark:<br />
** [[Spark on Jupyter|on Jupyter]]<br />
** [[Spark_on_farm|on HTCondor]]<br />
<br />
=== Storage ===<br />
* [[Storage]]<br />
* [[Hadoop Distributed File System (HDFS)]]<br />
* [[HDFS Access via VOSpace]]<br />
* [[Transferring data to/from PIC]]<br />
<br />
=== Other services ===<br />
* [[Gitlab]]<br />
* [[CosmoHub]]<br />
<br />
== Experiments ==<br />
* [[Euclid]]<br />
* [[AGN ICE]]<br />
* [[ICFO]]</div>
Tallada
https://pwiki.pic.es/index.php?title=Faq&diff=1345
Faq
2026-04-15T13:32:08Z
<p>Tallada: </p>
<hr />
<div>== How do I reset my password? ==<br />
You can reset your password using the following link:<br />
https://www.pic.es/user/auth/forgotpw<br />
<br />
== Can an undergraduate student in my group have an account? ==<br />
Yes. Undergraduate students can have PIC accounts without any problem.<br />
<br />
== How do I get a Kerberos token? ==<br />
A valid token is needed to interface with some PIC services, like the Hadoop platform (HDFS, Hive, Spark, ...) or CTA.<br />
Tokens are issued for the machine in which they are requested, if you hop to another machine, you may need to get another token.<br />
<br />
In order to get a token, run the following commands on a terminal.<br />
If you are on JupyterLab, use ''Launcher'' → ''Terminal''.<br />
<br />
<pre><br />
kinit -n -c ~/.fast.ccache @PIC.ES<br />
kinit -T ~/.fast.ccache<br />
</pre><br />
<br />
You could also define an alias for your own convenience:<br />
<br />
<pre><br />
alias kinit="kinit -n -c ~/.fast.ccache @PIC.ES; kinit -T ~/.fast.ccache"<br />
</pre><br />
<br />
Optionally, you can check the presence and expiration date of your token using:<br />
<br />
<pre><br />
klist<br />
</pre><br />
<br />
== I lost access to my OTP, how can I reset it? ==<br />
Follow these steps to recover your account and configure a new OTP token.<br />
<br />
# On the Keycloak login page, click '''Forgot password / OTP'''.<br />
# Enter your username. You should receive a recovery email in your inbox.<br />
# Open the email and follow the recovery link.<br />
# Scan the new QR code with your authenticator app.<br />
# Enter the generated OTP code and a device name.<br />
#* The device name is mandatory.<br />
# Change your password when prompted.<br />
#* This is required for security reasons.<br />
# Log in again as usual.<br />
#* On the OTP selection page, you will see two devices.<br />
#* Select the device with the name you just entered.<br />
#* The old one will usually appear as '''unnamed''' (if no name was set previously).<br />
<br />
Afterwards, we recommend removing the old OTP factor from the [https://idp.pic.es/realms/PIC/account/account-security/signing-in user profile section].</div>
Tallada
https://pwiki.pic.es/index.php?title=Faq&diff=1344
Faq
2026-04-15T13:24:23Z
<p>Tallada: </p>
<hr />
<div>== How do I reset my password? ==<br />
You can reset your password using the following link:<br />
https://www.pic.es/user/auth/forgotpw<br />
<br />
== Can an undergraduate student in my group have an account? ==<br />
Yes. Undergraduate students can have PIC accounts without any problem.<br />
<br />
== How do I get a Kerberos token? ==<br />
A valid token is needed to interface with some PIC services, like the Hadoop platform (HDFS, Hive, Spark, ...) or CTA.<br />
Tokens are issued for the machine in which they are requested, if you hop to another machine, you may need to get another token.<br />
<br />
In order to get a token, run the following commands on a terminal.<br />
If you are on JupyterLab, use ''Launcher'' → ''Terminal''.<br />
<br />
<pre><br />
kinit -n -c ~/.fast.ccache @PIC.ES<br />
kinit -T ~/.fast.ccache<br />
</pre><br />
<br />
You could also define an alias for your own convenience:<br />
<br />
<pre><br />
alias kinit="kinit -n -c ~/.fast.ccache @PIC.ES; kinit -T ~/.fast.ccache"<br />
</pre><br />
<br />
Optionally, you can check the presence and expiration date of your token using:<br />
<br />
<pre><br />
klist<br />
</pre></div>
Tallada
https://pwiki.pic.es/index.php?title=Main_Page&diff=1343
Main Page
2026-04-15T13:18:36Z
<p>Tallada: </p>
<hr />
<div>== Getting started ==<br />
* [[PIC description|Introduction to PIC]]<br />
* [[PIC account|Get a PIC account]]<br />
* [[Lost OTP|Reset your OTP]]<br />
* [[PIC_User_Manual | User manual]]<br />
* [[faq| Frequently asked questions]]<br />
<br />
== Services ==<br />
=== User interfaces ===<br />
* [[Login machines]]<br />
* [[JupyterHub]]<br />
<br />
=== Distributed computing ===<br />
* [[HTCondor]]<br />
* [[Dask]]<br />
* Spark:<br />
** [[Spark on Jupyter|on Jupyter]]<br />
** [[Spark_on_farm|on HTCondor]]<br />
<br />
=== Storage ===<br />
* [[Storage]]<br />
* [[Hadoop Distributed File System (HDFS)]]<br />
* [[HDFS Access via VOSpace]]<br />
* [[Transferring data to/from PIC]]<br />
<br />
=== Other services ===<br />
* [[Gitlab]]<br />
* [[CosmoHub]]<br />
<br />
== Experiments ==<br />
* [[Euclid]]<br />
* [[AGN ICE]]<br />
* [[ICFO]]</div>
Tallada
https://pwiki.pic.es/index.php?title=Spark_on_Jupyter&diff=1340
Spark on Jupyter
2026-04-08T08:51:24Z
<p>Tallada: Created page with "= Accessing PIC Big Data Services from JupyterLab = PIC's Jupyter service is integrated with the Big Data platform. This allows you to access services such as '''HDFS''', '''..."</p>
<hr />
<div>= Accessing PIC Big Data Services from JupyterLab =<br />
<br />
PIC's Jupyter service is integrated with the Big Data platform. This allows you to access services such as '''HDFS''', '''Hive''', and '''Spark''' directly from your JupyterLab environment.<br />
<br />
== Requirements ==<br />
<br />
=== 1. Kerberos Authentication ===<br />
<br />
You must have a '''valid Kerberos ticket''' before accessing any Big Data service.<br />
<br />
==== Step-by-step ====<br />
<br />
# Open a terminal session on the same node where your Jupyter session is running.<br />
#* In JupyterLab, use the ''Launcher'' → ''Terminal''.<br />
# Run the following commands:<br />
<br />
<pre><br />
kinit -n -c ~/.fast.ccache @PIC.ES<br />
kinit -T ~/.fast.ccache<br />
</pre><br />
<br />
==== Verify your ticket ====<br />
<br />
<pre><br />
klist<br />
</pre><br />
<br />
You should see a valid ticket with a non-expired timestamp.<br />
<br />
==== Notes ====<br />
<br />
* If the ticket expires, you must repeat the process.<br />
* Without a valid Kerberos ticket, access to HDFS, Hive, or Spark will fail.<br />
<br />
==== Quick test (HDFS access) ====<br />
<br />
<pre><br />
hdfs dfs -ls /<br />
</pre><br />
<br />
If this works, your authentication is correctly configured.<br />
<br />
---<br />
<br />
=== 2. Python Environment ===<br />
<br />
You need specific Python packages depending on the service:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Service !! Required packages<br />
|-<br />
| Spark || findspark, pyspark<br />
|-<br />
| Hive || pyhive[hive_pure_sasl, kerberos]<br />
|}<br />
<br />
==== Installation example ====<br />
<br />
<pre><br />
pip install findspark pyspark<br />
pip install "pyhive[hive_pure_sasl, kerberos]"<br />
</pre><br />
<br />
---<br />
<br />
== Using Spark from Jupyter ==<br />
<br />
=== Setup ===<br />
<br />
<pre><br />
import os<br />
<br />
# Environment configuration<br />
os.environ["HADOOP_HOME"] = "/usr/local/hadoop"<br />
os.environ["HADOOP_CONF_DIR"] = "/usr/local/hadoop/etc/hadoop"<br />
os.environ["HIVE_HOME"] = "/usr/local/hive"<br />
os.environ["HIVE_CONF_DIR"] = "/usr/local/hive/conf"<br />
<br />
import findspark<br />
findspark.init()<br />
</pre><br />
<br />
=== Create a Spark session ===<br />
<br />
<pre><br />
from pyspark.sql import SparkSession<br />
<br />
spark = (<br />
SparkSession.builder<br />
.appName("example")<br />
.enableHiveSupport()<br />
.getOrCreate()<br />
)<br />
<br />
sc = spark.sparkContext<br />
</pre><br />
<br />
=== Example: Read a Hive table ===<br />
<br />
<pre><br />
df = spark.sql("SELECT * FROM some_database.some_table LIMIT 10")<br />
df.show()<br />
</pre><br />
<br />
---<br />
<br />
== Using Hive (PyHive) from Jupyter ==<br />
<br />
=== Connect to Hive ===<br />
<br />
<pre><br />
from pyhive import hive<br />
<br />
conn = hive.connect(<br />
host="hsrv01.pic.es",<br />
port=10000,<br />
kerberos_service_name="hive",<br />
auth="KERBEROS",<br />
)<br />
<br />
cursor = conn.cursor()<br />
</pre><br />
<br />
=== Execute a query ===<br />
<br />
<pre><br />
query = "SELECT * FROM some_database.some_table LIMIT 10"<br />
cursor.execute(query)<br />
<br />
rows = cursor.fetchall()<br />
colnames = [c[0] for c in cursor.description]<br />
</pre><br />
<br />
=== Convert to Astropy Table (optional) ===<br />
<br />
<pre><br />
from astropy.table import Table<br />
<br />
table = Table(rows=rows, names=colnames)<br />
</pre><br />
<br />
---<br />
<br />
== Troubleshooting ==<br />
<br />
=== Kerberos errors ===<br />
* Run <code>klist</code> and check expiration<br />
* Re-run <code>kinit</code> if needed<br />
<br />
=== Hive connection fails ===<br />
* Ensure Kerberos ticket is valid<br />
* Verify correct host (<code>hsrv01.pic.es</code>)<br />
* Check that required Python packages are installed<br />
<br />
=== Spark does not start ===<br />
* Verify environment variables (<code>HADOOP_HOME</code>, <code>HIVE_HOME</code>)<br />
* Ensure <code>findspark.init()</code> is executed before creating the session<br />
<br />
---<br />
<br />
== Summary ==<br />
<br />
# Obtain a Kerberos ticket (<code>kinit</code>)<br />
# Install required Python packages<br />
# Configure environment variables<br />
# Use Spark or Hive from your notebook<br />
<br />
---<br />
<br />
For further assistance, contact PIC support (user.support@pic.es).</div>
Tallada
https://pwiki.pic.es/index.php?title=Main_Page&diff=1339
Main Page
2026-04-08T08:17:49Z
<p>Tallada: /* Distributed computing */</p>
<hr />
<div>== Getting started ==<br />
* [[PIC description|Introduction to PIC]]<br />
* [[PIC account|Get a PIC account]]<br />
* [[PIC_User_Manual | User manual]]<br />
* [[faq| Frequently asked questions]]<br />
<br />
== Services ==<br />
=== User interfaces ===<br />
* [[Login machines]]<br />
* [[JupyterHub]]<br />
<br />
=== Distributed computing ===<br />
* [[HTCondor]]<br />
* [[Dask]]<br />
* Spark:<br />
** [[Spark on Jupyter|on Jupyter]]<br />
** [[Spark_on_farm|on HTCondor]]<br />
<br />
=== Storage ===<br />
* [[Storage]]<br />
* [[Hadoop Distributed File System (HDFS)]]<br />
* [[HDFS Access via VOSpace]]<br />
* [[Transferring data to/from PIC]]<br />
<br />
<br />
<br />
=== Other services ===<br />
* [[Gitlab]]<br />
* [[CosmoHub]]<br />
<br />
== Experiments ==<br />
<br />
* [[Euclid]]<br />
* [[AGN ICE]]<br />
* [[ICFO]]</div>
Tallada
https://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1227
Transferring data to/from PIC
2025-05-27T09:26:12Z
<p>Tallada: /* Usage */</p>
<hr />
<div>= How to provide data access to PIC massive storage (dCache) =<br />
<br />
== Requirements ==<br />
<br />
* Install and configure Rclone<br />
* PIC credentials or macaroon<br />
<br />
== Install Rclone ==<br />
<br />
You can directly download the binary without installing anything. For instance, for a linux 64 bits machine:<br />
<br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
[...]<br />
$ unzip rclone-current-linux-amd64.zip<br />
<br />
<br />
Or if you prefer, you can install Rclone like the next example on a Ubuntu machine:<br />
<br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
<br />
== Configure Rclone ==<br />
<br />
You should have been given some credentials and the url of a WebDAV endpoint at PIC.<br />
With them, you just need to create the config in rclone:<br />
<br />
$ rclone config<br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic<br />
<br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
1 / 1Fichier<br />
\ (fichier)<br />
[...]<br />
Storage> webdav<br />
<br />
Option url.<br />
URL of http host to connect to.<br />
E.g. https://example.com.<br />
Enter a value.<br />
url> https://door04.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
<br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
Choose a number from below, or type in your own value.<br />
Press Enter to leave empty.<br />
1 / Nextcloud<br />
\ (nextcloud)<br />
[...]<br />
5 / Other site/service or software<br />
\ (other)<br />
vendor> other<br />
<br />
=== Using your PIC credentials ===<br />
<br />
If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank.<br />
<br />
Option user.<br />
User name.<br />
In case NTLM authentication is used, the username should be in the format 'Domain\User'.<br />
Enter a value. Press Enter to leave empty.<br />
user> YOUR_PIC_USERNAME<br />
<br />
Option pass.<br />
Password.<br />
Choose an alternative below. Press Enter for the default (n).<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank (default)<br />
y/g/n> y<br />
Enter the password:<br />
password: YOUR_PIC_PASSWORD<br />
Confirm the password:<br />
Password: YOUR_PIC_PASSWORD<br />
<br />
=== Using a Macaroon token ===<br />
<br />
If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank<br />
<br />
Option bearer_token.<br />
Bearer token instead of user/pass (e.g. a Macaroon).<br />
Enter a value. Press Enter to leave empty.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
<br />
=== Review settings ===<br />
<br />
At the end, just review the information you entered and confirm.<br />
<br />
Edit advanced config?<br />
y) Yes<br />
n) No (default)<br />
y/n> n<br />
<br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://door04.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- user: YOUR_PIC_USERNAME<br />
- pass: *** ENCRYPTED ***<br />
Keep this "pic" remote?<br />
y) Yes this is OK (default)<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
<br />
Current remotes:<br />
<br />
Name Type<br />
==== ====<br />
pic webdav<br />
<br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
<br />
Once done, you can use command line to browse and download/upload data.<br />
<br />
=== Usage ===<br />
<br />
* List a remote PIC directory: <br />
rclone lsd <name>:<path>`<br />
<br />
* Download a remote directory from PIC<br />
rclone copy <name>:<path> <local_path><br />
<br />
* Upload a local directory to PIC<br />
rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path><br />
<br />
When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files <br />
--check-first -P --stats-one-line --transfers <n_transfers> --size-only<br />
<br />
If uploading lots onto directories with lots of files (>1000), please use:<br />
--no-traverse<br />
<br />
If uploading files larger than 200 MB, also use<br />
--multi-thread-streams 1<br />
<br />
If uploading very large files (>10G), also use the following to allow more time to compute the checksums<br />
--timeout=15m<br />
<br />
See rclone manual for more extensive documentation https://rclone.org/docs/<br />
<br />
== Obtaining a macaroon (for contacts) ==<br />
<br />
Macaroons are valid up to 7 days.<br />
<br />
For downloading data (read-only permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
For uploading data (full permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre></div>
Tallada
https://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1154
Transferring data to/from PIC
2024-06-05T13:17:40Z
<p>Tallada: </p>
<hr />
<div>= How to provide data access to PIC massive storage (dCache) =<br />
<br />
== Requirements ==<br />
<br />
* Install and configure Rclone<br />
* PIC credentials or macaroon<br />
<br />
== Install Rclone ==<br />
<br />
You can directly download the binary without installing anything. For instance, for a linux 64 bits machine:<br />
<br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
[...]<br />
$ unzip rclone-current-linux-amd64.zip<br />
<br />
<br />
Or if you prefer, you can install Rclone like the next example on a Ubuntu machine:<br />
<br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
<br />
== Configure Rclone ==<br />
<br />
You should have been given some credentials and the url of a WebDAV endpoint at PIC.<br />
With them, you just need to create the config in rclone:<br />
<br />
$ rclone config<br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic<br />
<br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
1 / 1Fichier<br />
\ (fichier)<br />
[...]<br />
Storage> webdav<br />
<br />
Option url.<br />
URL of http host to connect to.<br />
E.g. https://example.com.<br />
Enter a value.<br />
url> https://door04.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
<br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
Choose a number from below, or type in your own value.<br />
Press Enter to leave empty.<br />
1 / Nextcloud<br />
\ (nextcloud)<br />
[...]<br />
5 / Other site/service or software<br />
\ (other)<br />
vendor> other<br />
<br />
=== Using your PIC credentials ===<br />
<br />
If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank.<br />
<br />
Option user.<br />
User name.<br />
In case NTLM authentication is used, the username should be in the format 'Domain\User'.<br />
Enter a value. Press Enter to leave empty.<br />
user> YOUR_PIC_USERNAME<br />
<br />
Option pass.<br />
Password.<br />
Choose an alternative below. Press Enter for the default (n).<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank (default)<br />
y/g/n> y<br />
Enter the password:<br />
password: YOUR_PIC_PASSWORD<br />
Confirm the password:<br />
Password: YOUR_PIC_PASSWORD<br />
<br />
=== Using a Macaroon token ===<br />
<br />
If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank<br />
<br />
Option bearer_token.<br />
Bearer token instead of user/pass (e.g. a Macaroon).<br />
Enter a value. Press Enter to leave empty.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
<br />
=== Review settings ===<br />
<br />
At the end, just review the information you entered and confirm.<br />
<br />
Edit advanced config?<br />
y) Yes<br />
n) No (default)<br />
y/n> n<br />
<br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://door04.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- user: YOUR_PIC_USERNAME<br />
- pass: *** ENCRYPTED ***<br />
Keep this "pic" remote?<br />
y) Yes this is OK (default)<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
<br />
Current remotes:<br />
<br />
Name Type<br />
==== ====<br />
pic webdav<br />
<br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
<br />
Once done, you can use command line to browse and download/upload data.<br />
<br />
=== Usage ===<br />
<br />
* List a remote PIC directory: <br />
rclone lsd <name>:<path>`<br />
<br />
* Download a remote directory from PIC<br />
rclone copy <name>:<path> <local_path><br />
<br />
* Upload a local directory to PIC<br />
rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path><br />
<br />
When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files <br />
--check-first -P --stats-one-line --transfers <n_transfers> --size-only<br />
<br />
See rclone manual for more extensive documentation https://rclone.org/docs/<br />
<br />
== Obtaining a macaroon (for contacts) ==<br />
<br />
Macaroons are valid up to 7 days.<br />
<br />
For downloading data (read-only permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
For uploading data (full permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://door04.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://door04.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://door04.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://door04.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://door04.pic.es:8460/"<br />
}<br />
}<br />
</pre></div>
Tallada
https://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1153
Transferring data to/from PIC
2024-06-04T09:21:30Z
<p>Tallada: /* Obtaining a macaroon (for contacts) */</p>
<hr />
<div>= How to provide data access to PIC massive storage (dCache) =<br />
<br />
== Requirements ==<br />
<br />
* Install and configure Rclone<br />
* PIC credentials or macaroon<br />
<br />
== Install Rclone ==<br />
<br />
You can directly download the binary without installing anything. For instance, for a linux 64 bits machine:<br />
<br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
[...]<br />
$ unzip rclone-current-linux-amd64.zip<br />
<br />
<br />
Or if you prefer, you can install Rclone like the next example on a Ubuntu machine:<br />
<br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
<br />
== Configure Rclone ==<br />
<br />
You should have been given some credentials and the url of a WebDAV endpoint at PIC.<br />
With them, you just need to create the config in rclone:<br />
<br />
$ rclone config<br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic<br />
<br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
1 / 1Fichier<br />
\ (fichier)<br />
[...]<br />
Storage> webdav<br />
<br />
Option url.<br />
URL of http host to connect to.<br />
E.g. https://example.com.<br />
Enter a value.<br />
url> https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
<br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
Choose a number from below, or type in your own value.<br />
Press Enter to leave empty.<br />
1 / Nextcloud<br />
\ (nextcloud)<br />
[...]<br />
5 / Other site/service or software<br />
\ (other)<br />
vendor> other<br />
<br />
=== Using your PIC credentials ===<br />
<br />
If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank.<br />
<br />
Option user.<br />
User name.<br />
In case NTLM authentication is used, the username should be in the format 'Domain\User'.<br />
Enter a value. Press Enter to leave empty.<br />
user> YOUR_PIC_USERNAME<br />
<br />
Option pass.<br />
Password.<br />
Choose an alternative below. Press Enter for the default (n).<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank (default)<br />
y/g/n> y<br />
Enter the password:<br />
password: YOUR_PIC_PASSWORD<br />
Confirm the password:<br />
Password: YOUR_PIC_PASSWORD<br />
<br />
=== Using a Macaroon token ===<br />
<br />
If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank<br />
<br />
Option bearer_token.<br />
Bearer token instead of user/pass (e.g. a Macaroon).<br />
Enter a value. Press Enter to leave empty.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
<br />
=== Review settings ===<br />
<br />
At the end, just review the information you entered and confirm.<br />
<br />
Edit advanced config?<br />
y) Yes<br />
n) No (default)<br />
y/n> n<br />
<br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- user: YOUR_PIC_USERNAME<br />
- pass: *** ENCRYPTED ***<br />
Keep this "pic" remote?<br />
y) Yes this is OK (default)<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
<br />
Current remotes:<br />
<br />
Name Type<br />
==== ====<br />
pic webdav<br />
<br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
<br />
Once done, you can use command line to browse and download/upload data.<br />
<br />
=== Usage ===<br />
<br />
* List a remote PIC directory: <br />
rclone lsd <name>:<path>`<br />
<br />
* Download a remote directory from PIC<br />
rclone copy <name>:<path> <local_path><br />
<br />
* Upload a local directory to PIC<br />
rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path><br />
<br />
When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files <br />
--check-first -P --stats-one-line --transfers <n_transfers> --size-only<br />
<br />
See rclone manual for more extensive documentation https://rclone.org/docs/<br />
<br />
== Obtaining a macaroon (for contacts) ==<br />
<br />
Macaroons are valid up to 7 days.<br />
<br />
For downloading data (read-only permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://webdav.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://webdav.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://webdav.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
For uploading data (full permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USER} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://webdav.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://webdav.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://webdav.pic.es:8460/"<br />
}<br />
}<br />
</pre></div>
Tallada
https://pwiki.pic.es/index.php?title=Main_Page&diff=1101
Main Page
2023-12-20T14:54:12Z
<p>Tallada: /* Services */</p>
<hr />
<div>== Getting started ==<br />
* [[PIC description|PIC in an image]]<br />
* [[PIC account|Get a PIC account]]<br />
* [[PIC_User_Manual | User manual]]<br />
<br />
== Services ==<br />
* [[HTCondor]]<br />
* [[Storage]]<br />
* [[JupyterHub]]<br />
* [[Gitlab]]<br />
* [[CosmoHub]]<br />
* Spark:<br />
** [[Spark on Hadoop|on Hadoop]]<br />
** [[Spark_on_farm|on HTCondor]]<br />
* [[Transferring data to/from PIC]]<br />
<br />
== Experiments ==<br />
<br />
* [[Euclid]]<br />
<br />
== More technical information ==<br />
<br />
* [[Storage Department]]</div>
Tallada
https://pwiki.pic.es/index.php?title=DCache_webdav_i_dCacheView&diff=1100
DCache webdav i dCacheView
2023-12-20T14:53:29Z
<p>Tallada: Tallada moved page DCache webdav i dCacheView to Transferring data to/from PIC</p>
<hr />
<div>#REDIRECT [[Transferring data to/from PIC]]</div>
Tallada
https://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1099
Transferring data to/from PIC
2023-12-20T14:53:29Z
<p>Tallada: Tallada moved page DCache webdav i dCacheView to Transferring data to/from PIC</p>
<hr />
<div>= How to provide data access to PIC massive storage (dCache) =<br />
<br />
== Requirements ==<br />
<br />
* Install and configure Rclone<br />
* PIC credentials or macaroon<br />
<br />
== Install Rclone ==<br />
<br />
You can directly download the binary without installing anything. For instance, for a linux 64 bits machine:<br />
<br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
[...]<br />
$ unzip rclone-current-linux-amd64.zip<br />
<br />
<br />
Or if you prefer, you can install Rclone like the next example on a Ubuntu machine:<br />
<br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
<br />
== Configure Rclone ==<br />
<br />
You should have been given some credentials and the url of a WebDAV endpoint at PIC.<br />
With them, you just need to create the config in rclone:<br />
<br />
$ rclone config<br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic<br />
<br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
1 / 1Fichier<br />
\ (fichier)<br />
[...]<br />
Storage> webdav<br />
<br />
Option url.<br />
URL of http host to connect to.<br />
E.g. https://example.com.<br />
Enter a value.<br />
url> https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
<br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
Choose a number from below, or type in your own value.<br />
Press Enter to leave empty.<br />
1 / Nextcloud<br />
\ (nextcloud)<br />
[...]<br />
5 / Other site/service or software<br />
\ (other)<br />
vendor> other<br />
<br />
=== Using your PIC credentials ===<br />
<br />
If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank.<br />
<br />
Option user.<br />
User name.<br />
In case NTLM authentication is used, the username should be in the format 'Domain\User'.<br />
Enter a value. Press Enter to leave empty.<br />
user> YOUR_PIC_USERNAME<br />
<br />
Option pass.<br />
Password.<br />
Choose an alternative below. Press Enter for the default (n).<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank (default)<br />
y/g/n> y<br />
Enter the password:<br />
password: YOUR_PIC_PASSWORD<br />
Confirm the password:<br />
Password: YOUR_PIC_PASSWORD<br />
<br />
=== Using a Macaroon token ===<br />
<br />
If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank<br />
<br />
Option bearer_token.<br />
Bearer token instead of user/pass (e.g. a Macaroon).<br />
Enter a value. Press Enter to leave empty.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
<br />
=== Review settings ===<br />
<br />
At the end, just review the information you entered and confirm.<br />
<br />
Edit advanced config?<br />
y) Yes<br />
n) No (default)<br />
y/n> n<br />
<br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- user: YOUR_PIC_USERNAME<br />
- pass: *** ENCRYPTED ***<br />
Keep this "pic" remote?<br />
y) Yes this is OK (default)<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
<br />
Current remotes:<br />
<br />
Name Type<br />
==== ====<br />
pic webdav<br />
<br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
<br />
Once done, you can use command line to browse and download/upload data.<br />
<br />
=== Usage ===<br />
<br />
* List a remote PIC directory: <br />
rclone lsd <name>:<path>`<br />
<br />
* Download a remote directory from PIC<br />
rclone copy <name>:<path> <local_path><br />
<br />
* Upload a local directory to PIC<br />
rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path><br />
<br />
When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files <br />
--check-first -P --stats-one-line --transfers <n_transfers> --size-only<br />
<br />
See rclone manual for more extensive documentation https://rclone.org/docs/<br />
<br />
== Obtaining a macaroon (for contacts) ==<br />
<br />
Macaroons are valid up to 7 days.<br />
<br />
For downloading data (read-only permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USERNAME} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://webdav.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://webdav.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://webdav.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
For uploading data (full permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USERNAME} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://webdav.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://webdav.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://webdav.pic.es:8460/"<br />
}<br />
}<br />
</pre></div>
Tallada
https://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1098
Transferring data to/from PIC
2023-12-15T12:59:42Z
<p>Tallada: /* How to provide data access to PIC massive storage (dCache) = */</p>
<hr />
<div>= How to provide data access to PIC massive storage (dCache) =<br />
<br />
== Requirements ==<br />
<br />
* Install and configure Rclone<br />
* PIC credentials or macaroon<br />
<br />
== Install Rclone ==<br />
<br />
You can directly download the binary without installing anything. For instance, for a linux 64 bits machine:<br />
<br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
[...]<br />
$ unzip rclone-current-linux-amd64.zip<br />
<br />
<br />
Or if you prefer, you can install Rclone like the next example on a Ubuntu machine:<br />
<br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
<br />
== Configure Rclone ==<br />
<br />
You should have been given some credentials and the url of a WebDAV endpoint at PIC.<br />
With them, you just need to create the config in rclone:<br />
<br />
$ rclone config<br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic<br />
<br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
1 / 1Fichier<br />
\ (fichier)<br />
[...]<br />
Storage> webdav<br />
<br />
Option url.<br />
URL of http host to connect to.<br />
E.g. https://example.com.<br />
Enter a value.<br />
url> https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
<br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
Choose a number from below, or type in your own value.<br />
Press Enter to leave empty.<br />
1 / Nextcloud<br />
\ (nextcloud)<br />
[...]<br />
5 / Other site/service or software<br />
\ (other)<br />
vendor> other<br />
<br />
=== Using your PIC credentials ===<br />
<br />
If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank.<br />
<br />
Option user.<br />
User name.<br />
In case NTLM authentication is used, the username should be in the format 'Domain\User'.<br />
Enter a value. Press Enter to leave empty.<br />
user> YOUR_PIC_USERNAME<br />
<br />
Option pass.<br />
Password.<br />
Choose an alternative below. Press Enter for the default (n).<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank (default)<br />
y/g/n> y<br />
Enter the password:<br />
password: YOUR_PIC_PASSWORD<br />
Confirm the password:<br />
Password: YOUR_PIC_PASSWORD<br />
<br />
=== Using a Macaroon token ===<br />
<br />
If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank<br />
<br />
Option bearer_token.<br />
Bearer token instead of user/pass (e.g. a Macaroon).<br />
Enter a value. Press Enter to leave empty.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
<br />
=== Review settings ===<br />
<br />
At the end, just review the information you entered and confirm.<br />
<br />
Edit advanced config?<br />
y) Yes<br />
n) No (default)<br />
y/n> n<br />
<br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- user: YOUR_PIC_USERNAME<br />
- pass: *** ENCRYPTED ***<br />
Keep this "pic" remote?<br />
y) Yes this is OK (default)<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
<br />
Current remotes:<br />
<br />
Name Type<br />
==== ====<br />
pic webdav<br />
<br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
<br />
Once done, you can use command line to browse and download/upload data.<br />
<br />
=== Usage ===<br />
<br />
* List a remote PIC directory: <br />
rclone lsd <name>:<path>`<br />
<br />
* Download a remote directory from PIC<br />
rclone copy <name>:<path> <local_path><br />
<br />
* Upload a local directory to PIC<br />
rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path><br />
<br />
When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files <br />
--check-first -P --stats-one-line --transfers <n_transfers> --size-only<br />
<br />
See rclone manual for more extensive documentation https://rclone.org/docs/<br />
<br />
== Obtaining a macaroon (for contacts) ==<br />
<br />
Macaroons are valid up to 7 days.<br />
<br />
For downloading data (read-only permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USERNAME} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://webdav.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://webdav.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://webdav.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
For uploading data (full permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USERNAME} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://webdav.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://webdav.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://webdav.pic.es:8460/"<br />
}<br />
}<br />
</pre></div>
Tallada
https://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1097
Transferring data to/from PIC
2023-12-15T12:59:03Z
<p>Tallada: </p>
<hr />
<div>= How to provide data access to PIC massive storage (dCache) ==<br />
<br />
== Requirements ==<br />
<br />
* Install and configure Rclone<br />
* PIC credentials or macaroon<br />
<br />
== Install Rclone ==<br />
<br />
You can directly download the binary without installing anything. For instance, for a linux 64 bits machine:<br />
<br />
$ curl -JLO https://downloads.rclone.org/rclone-current-linux-amd64.zip<br />
[...]<br />
$ unzip rclone-current-linux-amd64.zip<br />
<br />
<br />
Or if you prefer, you can install Rclone like the next example on a Ubuntu machine:<br />
<br />
$ cd /tmp<br />
$ curl -JLO 'https://downloads.rclone.org/rclone-current-linux-amd64.deb'<br />
$ sudo apt install ./rclone-current-linux-amd64.deb<br />
<br />
== Configure Rclone ==<br />
<br />
You should have been given some credentials and the url of a WebDAV endpoint at PIC.<br />
With them, you just need to create the config in rclone:<br />
<br />
$ rclone config<br />
No remotes found, make a new one?<br />
n) New remote<br />
s) Set configuration password<br />
q) Quit config<br />
n/s/q> n<br />
<br />
Enter name for new remote.<br />
name> pic<br />
<br />
Option Storage.<br />
Type of storage to configure.<br />
Choose a number from below, or type in your own value.<br />
1 / 1Fichier<br />
\ (fichier)<br />
[...]<br />
Storage> webdav<br />
<br />
Option url.<br />
URL of http host to connect to.<br />
E.g. https://example.com.<br />
Enter a value.<br />
url> https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
<br />
Option vendor.<br />
Name of the WebDAV site/service/software you are using.<br />
Choose a number from below, or type in your own value.<br />
Press Enter to leave empty.<br />
1 / Nextcloud<br />
\ (nextcloud)<br />
[...]<br />
5 / Other site/service or software<br />
\ (other)<br />
vendor> other<br />
<br />
=== Using your PIC credentials ===<br />
<br />
If you have a PIC user, enter it and the corresponding password in this step. Otherwise, leave these fields blank.<br />
<br />
Option user.<br />
User name.<br />
In case NTLM authentication is used, the username should be in the format 'Domain\User'.<br />
Enter a value. Press Enter to leave empty.<br />
user> YOUR_PIC_USERNAME<br />
<br />
Option pass.<br />
Password.<br />
Choose an alternative below. Press Enter for the default (n).<br />
y) Yes, type in my own password<br />
g) Generate random password<br />
n) No, leave this optional password blank (default)<br />
y/g/n> y<br />
Enter the password:<br />
password: YOUR_PIC_PASSWORD<br />
Confirm the password:<br />
Password: YOUR_PIC_PASSWORD<br />
<br />
=== Using a Macaroon token ===<br />
<br />
If you have been given a Macaroon token, provide it as a bearer token after leaving the user and password blank<br />
<br />
Option bearer_token.<br />
Bearer token instead of user/pass (e.g. a Macaroon).<br />
Enter a value. Press Enter to leave empty.<br />
bearer_token> YOUR_MACAROON_TOKEN<br />
<br />
=== Review settings ===<br />
<br />
At the end, just review the information you entered and confirm.<br />
<br />
Edit advanced config?<br />
y) Yes<br />
n) No (default)<br />
y/n> n<br />
<br />
Configuration complete.<br />
Options:<br />
- type: webdav<br />
- url: https://webdav.pic.es/PATH_TO_YOUR_STORAGE_SPACE<br />
- vendor: other<br />
- user: YOUR_PIC_USERNAME<br />
- pass: *** ENCRYPTED ***<br />
Keep this "pic" remote?<br />
y) Yes this is OK (default)<br />
e) Edit this remote<br />
d) Delete this remote<br />
y/e/d> y<br />
<br />
Current remotes:<br />
<br />
Name Type<br />
==== ====<br />
pic webdav<br />
<br />
e) Edit existing remote<br />
n) New remote<br />
d) Delete remote<br />
r) Rename remote<br />
c) Copy remote<br />
s) Set configuration password<br />
q) Quit config<br />
e/n/d/r/c/s/q> q<br />
<br />
Once done, you can use command line to browse and download/upload data.<br />
<br />
=== Usage ===<br />
<br />
* List a remote PIC directory: <br />
rclone lsd <name>:<path>`<br />
<br />
* Download a remote directory from PIC<br />
rclone copy <name>:<path> <local_path><br />
<br />
* Upload a local directory to PIC<br />
rclone ${UPLOAD_FLAGS} copy <local_dir> <name>:<path><br />
<br />
When uploading data, we recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files <br />
--check-first -P --stats-one-line --transfers <n_transfers> --size-only<br />
<br />
See rclone manual for more extensive documentation https://rclone.org/docs/<br />
<br />
== Obtaining a macaroon (for contacts) ==<br />
<br />
Macaroons are valid up to 7 days.<br />
<br />
For downloading data (read-only permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USERNAME} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://webdav.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://webdav.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://webdav.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
For uploading data (full permissions on the path):<br />
<br />
<pre><br />
$ curl -u ${USERNAME} -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"validity": "P7D"}' \<br />
https://webdav.pic.es:8460/${RESTRICTED_PATH}<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav.pic.es:8460/${RESTRICTED_PATH}?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://webdav.pic.es:8460/${RESTRICTED_PATH}",<br />
"base": "https://webdav.pic.es:8460/"<br />
}<br />
}<br />
</pre></div>
Tallada
https://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1096
Transferring data to/from PIC
2023-12-06T11:24:02Z
<p>Tallada: </p>
<hr />
<div>== How to provide data access to external users without PIC account ==<br />
<br />
<br />
==== dCache webdav door ====<br />
<br />
https://webdav.pic.es or https://webdav.pic.es:8460<br />
<br />
==== dCacheView frontend (optional) ====<br />
<br />
https://webdav.pic.es:8470<br />
<br />
==== Authentication ====<br />
<br />
Either PIC credentials (user/password) or a macaroon (https://dcache.org/old/manuals/UserGuide-7.0/macaroons.shtml).<br />
Any authenticated user can get a macaroon for an external user. Remember to add proper caveats to restrict privileges.<br />
In the next example, I retrieve a macaroon with 7 days validity, that can only download and list, and can only access the subtree of the provided URL.<br />
<br />
<pre><br />
$ curl -u tallada -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://webdav.pic.es:8460/pnfs/pic.es/data/astro/euclid/disk/simulations/ComplementarySimulations<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav.pic.es:8460/pnfs/pic.es/data/astro/euclid/disk/simulations/ComplementarySimulations?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://webdav.pic.es:8460/pnfs/pic.es/data/astro/euclid/disk/simulations/ComplementarySimulations",<br />
"base": "https://webdav.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
==== Browser access ====<br />
<br />
You can access the data either through the webdav door or dCacheView. If you have PIC credentials (user/password), just input them when requested by the browser.<br />
<br />
If you only have a macaroon token, you can access the webdav using the preauthenticated URL (either targetWithMacaroon or baseWithMacaroon). You will only be able to browse to the restricted subtree.<br />
You can also use command line clients like cURL or wget.<br />
<br />
In order to use the macarron with dCacheView, DO NOT ENTER any credentials at the login step. Instead, click the three bar menu at the top right, a panel on the left will open. On that panel, select the "Shared files" option. Then, clic the "Add" button on the bottom right and enter the macaroon token.<br />
<br />
==== rclone ====<br />
<br />
We can use rclone on Windows, Linux and MacOS to be able to automatize data transfer, even mount the remote data as a local filesystem.<br />
First, follow the instructions to install rclone in your host (https://rclone.org/install/).<br />
Note that for mounting you might need additional software (https://rclone.org/commands/rclone_mount/)<br />
<br />
Then, create a new remote, either using the "rclone config" command line, or opening the web GUI with the command "rclone rcd --rc-web-gui". You might need to add "--no-check-certificates" in some cases.<br />
Provide the following info for the remote:<br />
* name: "pic" or any other short identifier<br />
* provider: webdav<br />
* url: https://webdav.pic.es (you can use the full path here to to restrict the visible tree)<br />
* username/password: if you have PIC credentials, otherwise leave empty and use macaroon below.<br />
* bearer token: MDANOxGCg (the macaroon token)<br />
<br />
Once done, you can use either the command line or the explorer tab to browse and download data.<br />
<br />
Most common command lines:<br />
<br />
* List a remote PIC directory: <br />
rclone lsd <name>:<path>`<br />
<br />
* Copy a local directory to PIC<br />
rclone copy <local_dir> <name>:<path><br />
<br />
I recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files <br />
rclone --check-first -P --stats-one-line --transfers <n_transfers> --size-only copy <local_dir> <name>:<path><br />
<br />
You can also mount the remote data to a local path. Use the "Mounts" tab in the web GUI, select the remote filesystem and a local folder (or drive letter in Windows), and press create.<br />
<br />
Enjoy!</div>
Tallada
https://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=1095
Transferring data to/from PIC
2023-12-06T11:23:49Z
<p>Tallada: </p>
<hr />
<div>== How to provide data access to external users without PIC account ==<br />
<br />
<br />
==== dCache webdav door ====<br />
<br />
https://webdav.pic.es or https://webdav.pic.es:8460<br />
<br />
==== dCacheView frontend (optional) ====<br />
<br />
https://webdav-mice.pic.es:8470<br />
<br />
==== Authentication ====<br />
<br />
Either PIC credentials (user/password) or a macaroon (https://dcache.org/old/manuals/UserGuide-7.0/macaroons.shtml).<br />
Any authenticated user can get a macaroon for an external user. Remember to add proper caveats to restrict privileges.<br />
In the next example, I retrieve a macaroon with 7 days validity, that can only download and list, and can only access the subtree of the provided URL.<br />
<br />
<pre><br />
$ curl -u tallada -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://webdav.pic.es:8460/pnfs/pic.es/data/astro/euclid/disk/simulations/ComplementarySimulations<br />
<br />
{<br />
"macaroon": "MDA2MGxvY2F0aW",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav.pic.es:8460/pnfs/pic.es/data/astro/euclid/disk/simulations/ComplementarySimulations?authz=MDA2MGxvY2F0aW",<br />
"baseWithMacaroon": "https://webdav.pic.es:8460/?authz=MDA2MGxvY2F0aW",<br />
"target": "https://webdav.pic.es:8460/pnfs/pic.es/data/astro/euclid/disk/simulations/ComplementarySimulations",<br />
"base": "https://webdav.pic.es:8460/"<br />
}<br />
}<br />
</pre><br />
<br />
==== Browser access ====<br />
<br />
You can access the data either through the webdav door or dCacheView. If you have PIC credentials (user/password), just input them when requested by the browser.<br />
<br />
If you only have a macaroon token, you can access the webdav using the preauthenticated URL (either targetWithMacaroon or baseWithMacaroon). You will only be able to browse to the restricted subtree.<br />
You can also use command line clients like cURL or wget.<br />
<br />
In order to use the macarron with dCacheView, DO NOT ENTER any credentials at the login step. Instead, click the three bar menu at the top right, a panel on the left will open. On that panel, select the "Shared files" option. Then, clic the "Add" button on the bottom right and enter the macaroon token.<br />
<br />
==== rclone ====<br />
<br />
We can use rclone on Windows, Linux and MacOS to be able to automatize data transfer, even mount the remote data as a local filesystem.<br />
First, follow the instructions to install rclone in your host (https://rclone.org/install/).<br />
Note that for mounting you might need additional software (https://rclone.org/commands/rclone_mount/)<br />
<br />
Then, create a new remote, either using the "rclone config" command line, or opening the web GUI with the command "rclone rcd --rc-web-gui". You might need to add "--no-check-certificates" in some cases.<br />
Provide the following info for the remote:<br />
* name: "pic" or any other short identifier<br />
* provider: webdav<br />
* url: https://webdav.pic.es (you can use the full path here to to restrict the visible tree)<br />
* username/password: if you have PIC credentials, otherwise leave empty and use macaroon below.<br />
* bearer token: MDANOxGCg (the macaroon token)<br />
<br />
Once done, you can use either the command line or the explorer tab to browse and download data.<br />
<br />
Most common command lines:<br />
<br />
* List a remote PIC directory: <br />
rclone lsd <name>:<path>`<br />
<br />
* Copy a local directory to PIC<br />
rclone copy <local_dir> <name>:<path><br />
<br />
I recommend using the following flags, where n_transfers can be up to 350 if transferring lots of small files <br />
rclone --check-first -P --stats-one-line --transfers <n_transfers> --size-only copy <local_dir> <name>:<path><br />
<br />
You can also mount the remote data to a local path. Use the "Mounts" tab in the web GUI, select the remote filesystem and a local folder (or drive letter in Windows), and press create.<br />
<br />
Enjoy!</div>
Tallada
https://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=934
Transferring data to/from PIC
2021-12-23T11:49:32Z
<p>Tallada: </p>
<hr />
<div>== How to provide data access to external users without PIC account ==<br />
<br />
I will use an example for MICE<br />
<br />
<br />
==== dCache webdav door ====<br />
<br />
https://webdav-mice.pic.es:8454<br />
<br />
==== dCacheView frontend (optional) ====<br />
<br />
https://webdav-mice.pic.es:8464<br />
<br />
==== Authentication ====<br />
<br />
Either PIC credentials (user/password) or a macaroon (https://dcache.org/old/manuals/UserGuide-7.0/macaroons.shtml).<br />
Any authenticated user can get a macaroon for an external user. Remember to add proper caveats to restrict privileges.<br />
In the next example, I retrieve a macaroon with 7 days validity, that can only download and list, and can only access the subtree of the provided URL.<br />
<br />
<pre><br />
$ curl -u tallada -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://webdav-mice.pic.es:8454/tape/raw/production/N4096_L3072_LC3/maps/n8192/kmapgcpy/<br />
<br />
{<br />
"macaroon": "MDANOxGCg",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav-mice.pic.es:8454/tape/raw/production/N4096_L3072_LC3/maps/n8192/kmapgcpy/?authz=MDANOxGCg",<br />
"baseWithMacaroon": "https://webdav-mice.pic.es:8454/?authz=MDANOxGCg",<br />
"target": "https://webdav-mice.pic.es:8454/tape/raw/production/N4096_L3072_LC3/maps/n8192/kmapgcpy/",<br />
"base": "https://webdav-mice.pic.es:8454/"<br />
}<br />
}<br />
</pre><br />
<br />
==== Browser access ====<br />
<br />
You can access the data either through the webdav door or dCacheView. If you have PIC credentials (user/password), just input them when requested by the browser.<br />
<br />
If you only have a macaroon token, you can access the webdav using the preauthenticated URL (either targetWithMacaroon or baseWithMacaroon). You will only be able to browse to the restricted subtree.<br />
You can also use command line clients like cURL or wget.<br />
<br />
In order to use the macarron with dCacheView, DO NOT ENTER any credentials at the login step. Instead, click the three bar menu at the top right, a panel on the left will open. On that panel, select the "Shared files" option. Then, clic the "Add" button on the bottom right and enter the macaroon token.<br />
<br />
==== rclone ====<br />
<br />
We can use rclone on Windows, Linux and MacOS to be able to automatize data transfer, even mount the remote data as a local filesystem.<br />
First, follow the instructions to install rclone in your host (https://rclone.org/install/).<br />
Note that for mounting you might need additional software (https://rclone.org/commands/rclone_mount/)<br />
<br />
Then, open the web GUI with the command "rclone rcd --rc-web-gui". You might need to add "--no-check-certificates" in some cases.<br />
Once opened, create a config with the following info:<br />
* provider: webdav<br />
* url: https://webdav-mice.pic.es:8454 (either the base or target url)<br />
* username/password: if you have PIC credentials<br />
* bearer token: MDANOxGCg (the macaroon token)<br />
<br />
Once done, you can use the explorer tab to browse and download data.<br />
<br />
You can also mount the remote data to a local path. Use the "Mounts" tab, select the remote filesystem and a local folder (or diver letter in Windows), and press create.<br />
<br />
Enjoy!</div>
Tallada
https://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=933
Transferring data to/from PIC
2021-12-23T11:48:34Z
<p>Tallada: </p>
<hr />
<div>== How to provide data access to external users without PIC account ==<br />
<br />
I will use an example for MICE<br />
<br />
=== Steps ===<br />
<br />
1) dCache webdav door that exports your data.<br />
<br />
https://webdav-mice.pic.es:8454<br />
<br />
2) dCacheView frontend (optional)<br />
<br />
https://webdav-mice.pic.es:8464<br />
<br />
3) Authentication<br />
<br />
Either PIC credentials (user/password) or a macaroon (https://dcache.org/old/manuals/UserGuide-7.0/macaroons.shtml).<br />
Any authenticated user can get a macaroon for an external user. Remember to add proper caveats to restrict privileges.<br />
In the next example, I retrieve a macaroon with 7 days validity, that can only download and list, and can only access the subtree of the provided URL.<br />
<br />
<pre><br />
$ curl -u tallada -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://webdav-mice.pic.es:8454/tape/raw/production/N4096_L3072_LC3/maps/n8192/kmapgcpy/<br />
<br />
{<br />
"macaroon": "MDANOxGCg",<br />
"uri": {<br />
"targetWithMacaroon": "https://webdav-mice.pic.es:8454/tape/raw/production/N4096_L3072_LC3/maps/n8192/kmapgcpy/?authz=MDANOxGCg",<br />
"baseWithMacaroon": "https://webdav-mice.pic.es:8454/?authz=MDANOxGCg",<br />
"target": "https://webdav-mice.pic.es:8454/tape/raw/production/N4096_L3072_LC3/maps/n8192/kmapgcpy/",<br />
"base": "https://webdav-mice.pic.es:8454/"<br />
}<br />
}<br />
</pre><br />
<br />
4) Browser access<br />
<br />
You can access the data either through the webdav door or dCacheView. If you have PIC credentials (user/password), just input them when requested by the browser.<br />
<br />
If you only have a macaroon token, you can access the webdav using the preauthenticated URL (either targetWithMacaroon or baseWithMacaroon). You will only be able to browse to the restricted subtree.<br />
You can also use command line clients like cURL or wget.<br />
<br />
In order to use the macarron with dCacheView, DO NOT ENTER any credentials at the login step. Instead, click the three bar menu at the top right, a panel on the left will open. On that panel, select the "Shared files" option. Then, clic the "Add" button on the bottom right and enter the macaroon token.<br />
<br />
5) rclone<br />
<br />
We can use rclone on Windows, Linux and MacOS to be able to automatize data transfer, even mount the remote data as a local filesystem.<br />
First, follow the instructions to install rclone in your host (https://rclone.org/install/).<br />
Note that for mounting you might need additional software (https://rclone.org/commands/rclone_mount/)<br />
<br />
Then, open the web GUI with the command "rclone rcd --rc-web-gui". You might need to add "--no-check-certificates" in some cases.<br />
Once opened, create a config with the following info:<br />
* provider: webdav<br />
* url: https://webdav-mice.pic.es:8454 (either the base or target url)<br />
* username/password: if you have PIC credentials<br />
* bearer token: MDANOxGCg (the macaroon token)<br />
<br />
Once done, you can use the explorer tab to browse and download data.<br />
<br />
You can also mount the remote data to a local path. Use the "Mounts" tab, select the remote filesystem and a local folder (or diver letter in Windows), and press create.<br />
<br />
Enjoy!</div>
Tallada
https://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=932
Transferring data to/from PIC
2021-12-23T08:50:20Z
<p>Tallada: </p>
<hr />
<div>== How to provide data access to external users without PIC account ==<br />
<br />
I will use an example for MICE<br />
<br />
=== Steps ===<br />
<br />
1) dCache webdav door that exports your data.<br />
<br />
https://webdav-mice.pic.es:8454<br />
<br />
2) dCacheView frontend (optional)<br />
<br />
https://webdav-mice.pic.es:8464<br />
<br />
3) Authentication<br />
<br />
Either PIC credentials (user/password) or a macaroon (https://dcache.org/old/manuals/UserGuide-7.0/macaroons.shtml).<br />
Any authenticated user can get a macaroon for an external user. Remember to add proper caveats to restrict privileges.<br />
In the next example, I retrieve a macaroon with 7 days validity, that can only download and list, and can only access the subtree of the provided URL.<br />
<br />
<pre><br />
$ curl -u tallada -X POST -H 'Content-Type: application/macaroon-request' \<br />
-d '{"caveats": ["activity:DOWNLOAD,LIST"], "validity": "P7D"}' \<br />
https://webdav-mice.pic.es:8454/tape/raw/production/N4096_L3072_LC3/maps/n8192/kmapgcpy/<br />
<br />
{<br />
"macaroon": "MDA3NWxvJmc2lnbmF0dXJlIJt81Pp8G1KXMH8FqwpWhJtwYnLfUAvNPQuq7bTwNOxGCg", <br />
}<br />
</pre></div>
Tallada
https://pwiki.pic.es/index.php?title=Transferring_data_to/from_PIC&diff=931
Transferring data to/from PIC
2021-12-23T08:37:44Z
<p>Tallada: Created page with "== How to provide data access to external users without PIC account == I will use an example for MICE === Steps === 1) Ensure you have a dCache webdav door that exports yo..."</p>
<hr />
<div>== How to provide data access to external users without PIC account ==<br />
<br />
I will use an example for MICE<br />
<br />
=== Steps ===<br />
<br />
1) Ensure you have a dCache webdav door that exports your data.<br />
<br />
In our case: https://webdav-mice.pic.es:8454<br />
<br />
* dcacheview frontend (optional)<br />
* PIC account or macaroon<br />
*</div>
Tallada
https://pwiki.pic.es/index.php?title=Main_Page&diff=930
Main Page
2021-12-23T08:32:21Z
<p>Tallada: /* Astrophysics & Cosmology Department */</p>
<hr />
<div>== Computing Department ==<br />
* [[HTCondor User Guide]]<br />
* [[jupyter.pic.es]]<br />
<br />
== Storage Department ==<br />
* [[dCache.org]]<br />
* [[FileSystems]]<br />
<br />
== Astrophysics & Cosmology Department ==<br />
<br />
*[[CosmoHub | CosmoHub User Guide]]<br />
*[[Gitlab_Migration | Gitlab migration]]<br />
*[[Python_Jupyter | Python environments and Jupyter ]]<br />
*[[AC_UserManual | User manual]]<br />
*[[Spark_on_farm | Spark on farm]]<br />
*[[Catalogs | Catalogs]]<br />
*[[Mock pipeline v2.0]]<br />
*[[dCache webdav i dCacheView]]<br />
<br />
== Getting started ==<br />
* [//www.mediawiki.org/wiki/Special:MyLanguage/Manual:Configuration_settings Configuration settings list]<br />
* [//www.mediawiki.org/wiki/Special:MyLanguage/Manual:FAQ MediaWiki FAQ]<br />
* [https://lists.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list]<br />
* [//www.mediawiki.org/wiki/Special:MyLanguage/Localisation#Translation_resources Localise MediaWiki for your language]<br />
* Consult the [//meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software.</div>
Tallada
https://pwiki.pic.es/index.php?title=Spark_on_farm&diff=912
Spark on farm
2021-04-27T09:14:34Z
<p>Tallada: </p>
<hr />
<div>In order to launch a Spark cluster, follow these steps:<br />
<br />
1) Deploy a Spark installation on your ~<br />
<br />
2) On a UI, start a spark master using this command and write down the master url provided<br />
<br />
${SPARK_HOME}/sbin/start-master.sh -h $(hostname)<br />
<br />
3) Create a submit script like this one, substituting <VARIABLES><br />
<br />
SPARK_MASTER=<MASTER_URL><br />
NUM_CORES=1<br />
MEMORY=4g<br />
<br />
cd ${TMPDIR}<br />
<br />
echo "spark.executor.extraJavaOptions -Djava.io.tmpdir=${TMPDIR}" > spark-defaults.conf<br />
echo "spark.local.dir ${TMPDIR}" >> spark-defaults.conf<br />
echo "spark.pyspark.python <PYTHON_VENV_EXECUTABLE_PATH>" >> spark-defaults.conf<br />
<br />
PYSPARK_PYTHON=<PYTHON_VENV_EXECUTABLE_PATH> \<br />
<SPARK_HOME>/bin/spark-class \<br />
org.apache.spark.deploy.worker.Worker ${SPARK_MASTER} \<br />
-c ${NUM_CORES} -m ${MEMORY} -d ${TMPDIR} --properties-file spark-defaults.conf<br />
<br />
4) Submit the previous script to the farm<br />
<br />
qsub -q <QUEUE> -t 1-<NUM_WORKERS> <SUBMIT_SCRIPT><br />
<br />
5) After a few minutes, workers should start registering to the master.<br />
<br />
6) Create your SparkContext connecting it to the <MASTER_URL><br />
<br />
spark = SparkSession.builder.master(<MASTER_URL>).getOrCreate()<br />
sc = spark.sparkContext<br />
<br />
7) Enjoy!</div>
Tallada
https://pwiki.pic.es/index.php?title=High-z_galaxies&diff=890
High-z galaxies
2021-01-15T16:38:31Z
<p>Tallada: </p>
<hr />
<div><br />
== Description of the process to generate the High-z galaxies catalog for SC8 == <br />
<br />
<br />
Pau runs the pipeline and create a catalog in parquet:<br />
<br />
Ojo porque en el hdfs tiene 1_10_4 y sin embargo en el 1_10_5 <br />
<br />
/software/astro/scripts/create_table_from_parquet.sh -p /hdfs/user/tallada/data/euclid/flagship_mock_1_10_4_high_z/0.pq -t jcarrete.flagship_mock_1_10_5_highz_pq -e<br />
<br />
<br />
CREATE EXTERNAL TABLE jcarrete.flagship_mock_1_10_5_highz_pq (<br />
`type` STRING,<br />
`z` DOUBLE,<br />
`abs_muv` DOUBLE,<br />
`sedname` STRING,<br />
`sedname_int` BIGINT,<br />
`rhalf_kpc` DOUBLE,<br />
`rhalf_arcsec` DOUBLE,<br />
`muv` DOUBLE,<br />
`beta` DOUBLE,<br />
`r` DOUBLE,<br />
`hpix_9_nest` BIGINT,<br />
`kind` BIGINT,<br />
`step` BIGINT,<br />
`unique_id` BIGINT,<br />
`hpix_29_nest` BIGINT,<br />
`ra` DOUBLE,<br />
`dec` DOUBLE,<br />
`l` DOUBLE,<br />
`b` DOUBLE,<br />
`mw_extinction` DOUBLE,<br />
`blanco_decam_g_true` DOUBLE,<br />
`blanco_decam_g` DOUBLE,<br />
`blanco_decam_g_odonnell_ext` DOUBLE,<br />
`blanco_decam_i_true` DOUBLE,<br />
`blanco_decam_i` DOUBLE,<br />
`blanco_decam_i_odonnell_ext` DOUBLE,<br />
`blanco_decam_r_true` DOUBLE,<br />
`blanco_decam_r` DOUBLE,<br />
`blanco_decam_r_odonnell_ext` DOUBLE,<br />
`blanco_decam_z_true` DOUBLE,<br />
`blanco_decam_z` DOUBLE,<br />
`blanco_decam_z_odonnell_ext` DOUBLE,<br />
`subaru_hsc_z_true` DOUBLE,<br />
`subaru_hsc_z` DOUBLE,<br />
`subaru_hsc_z_odonnell_ext` DOUBLE,<br />
`jst_jpcam_g_true` DOUBLE,<br />
`jst_jpcam_g` DOUBLE,<br />
`jst_jpcam_g_odonnell_ext` DOUBLE,<br />
`kids_g_true` DOUBLE,<br />
`kids_g` DOUBLE,<br />
`kids_g_odonnell_ext` DOUBLE,<br />
`kids_i_true` DOUBLE,<br />
`kids_i` DOUBLE,<br />
`kids_i_odonnell_ext` DOUBLE,<br />
`kids_r_true` DOUBLE,<br />
`kids_r` DOUBLE,<br />
`kids_r_odonnell_ext` DOUBLE,<br />
`kids_u_true` DOUBLE,<br />
`kids_u` DOUBLE,<br />
`kids_u_odonnell_ext` DOUBLE,<br />
`lsst_g_true` DOUBLE,<br />
`lsst_g` DOUBLE,<br />
`lsst_g_odonnell_ext` DOUBLE,<br />
`lsst_i_true` DOUBLE,<br />
`lsst_i` DOUBLE,<br />
`lsst_i_odonnell_ext` DOUBLE,<br />
`lsst_r_true` DOUBLE,<br />
`lsst_r` DOUBLE,<br />
`lsst_r_odonnell_ext` DOUBLE,<br />
`lsst_u_true` DOUBLE,<br />
`lsst_u` DOUBLE,<br />
`lsst_u_odonnell_ext` DOUBLE,<br />
`lsst_y_true` DOUBLE,<br />
`lsst_y` DOUBLE,<br />
`lsst_y_odonnell_ext` DOUBLE,<br />
`lsst_z_true` DOUBLE,<br />
`lsst_z` DOUBLE,<br />
`lsst_z_odonnell_ext` DOUBLE,<br />
`cfht_megacam_r_true` DOUBLE,<br />
`cfht_megacam_r` DOUBLE,<br />
`cfht_megacam_r_odonnell_ext` DOUBLE,<br />
`cfht_megacam_u_true` DOUBLE,<br />
`cfht_megacam_u` DOUBLE,<br />
`cfht_megacam_u_odonnell_ext` DOUBLE,<br />
`pan_starrs_i_true` DOUBLE,<br />
`pan_starrs_i` DOUBLE,<br />
`pan_starrs_i_odonnell_ext` DOUBLE,<br />
`pan_starrs_z_true` DOUBLE,<br />
`pan_starrs_z` DOUBLE,<br />
`pan_starrs_z_odonnell_ext` DOUBLE,<br />
`2mass_h_true` DOUBLE,<br />
`2mass_h` DOUBLE,<br />
`2mass_h_odonnell_ext` DOUBLE,<br />
`2mass_j_true` DOUBLE,<br />
`2mass_j` DOUBLE,<br />
`2mass_j_odonnell_ext` DOUBLE,<br />
`2mass_ks_true` DOUBLE,<br />
`2mass_ks` DOUBLE,<br />
`2mass_ks_odonnell_ext` DOUBLE,<br />
`sdss_r01_true` DOUBLE,<br />
`sdss_r01` DOUBLE,<br />
`sdss_r01_odonnell_ext` DOUBLE,<br />
`gaia_bp_true` DOUBLE,<br />
`gaia_bp` DOUBLE,<br />
`gaia_bp_odonnell_ext` DOUBLE,<br />
`gaia_g_true` DOUBLE,<br />
`gaia_g` DOUBLE,<br />
`gaia_g_odonnell_ext` DOUBLE,<br />
`gaia_rp_true` DOUBLE,<br />
`gaia_rp` DOUBLE,<br />
`gaia_rp_odonnell_ext` DOUBLE,<br />
`euclid_nisp_h_true` DOUBLE,<br />
`euclid_nisp_h` DOUBLE,<br />
`euclid_nisp_h_odonnell_ext` DOUBLE,<br />
`euclid_nisp_j_true` DOUBLE,<br />
`euclid_nisp_j` DOUBLE,<br />
`euclid_nisp_j_odonnell_ext` DOUBLE,<br />
`euclid_nisp_y_true` DOUBLE,<br />
`euclid_nisp_y` DOUBLE,<br />
`euclid_nisp_y_odonnell_ext` DOUBLE,<br />
`euclid_vis_true` DOUBLE,<br />
`euclid_vis` DOUBLE,<br />
`euclid_vis_odonnell_ext` DOUBLE,<br />
`id` BIGINT<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/euclid/flagship_mock_1_10_4_high_z'<br />
;<br />
<br />
<br />
Clusterizo:<br />
<br />
CREATE TABLE cosmohub.flagship_mock_1_10_5_highz_c (<br />
`unique_id` BIGINT COMMENT 'unique identifier',<br />
`id` INT COMMENT 'ID',<br />
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',<br />
`z` FLOAT,<br />
`ra` FLOAT,<br />
`dec` FLOAT,<br />
`l` FLOAT,<br />
`b` FLOAT,<br />
`mw_extinction` FLOAT,<br />
`abs_muv` FLOAT,<br />
`sedname` STRING,<br />
`sedname_int` BIGINT,<br />
`rhalf_kpc` FLOAT,<br />
`rhalf_arcsec` FLOAT,<br />
`muv` FLOAT,<br />
`beta` FLOAT,<br />
`hpix_9_nest` INT,<br />
`hpix_13_nest` INT,<br />
`hpix_29_nest` BIGINT,<br />
`blanco_decam_g_true` FLOAT,<br />
`blanco_decam_g` FLOAT,<br />
`blanco_decam_g_odonnell_ext` FLOAT,<br />
`blanco_decam_i_true` FLOAT,<br />
`blanco_decam_i` FLOAT,<br />
`blanco_decam_i_odonnell_ext` FLOAT,<br />
`blanco_decam_r_true` FLOAT,<br />
`blanco_decam_r` FLOAT,<br />
`blanco_decam_r_odonnell_ext` FLOAT,<br />
`blanco_decam_z_true` FLOAT,<br />
`blanco_decam_z` FLOAT,<br />
`blanco_decam_z_odonnell_ext` FLOAT,<br />
`subaru_hsc_z_true` FLOAT,<br />
`subaru_hsc_z` FLOAT,<br />
`subaru_hsc_z_odonnell_ext` FLOAT,<br />
`jst_jpcam_g_true` FLOAT,<br />
`jst_jpcam_g` FLOAT,<br />
`jst_jpcam_g_odonnell_ext` FLOAT,<br />
`kids_g_true` FLOAT,<br />
`kids_g` FLOAT,<br />
`kids_g_odonnell_ext` FLOAT,<br />
`kids_i_true` FLOAT,<br />
`kids_i` FLOAT,<br />
`kids_i_odonnell_ext` FLOAT,<br />
`kids_r_true` FLOAT,<br />
`kids_r` FLOAT,<br />
`kids_r_odonnell_ext` FLOAT,<br />
`kids_u_true` FLOAT,<br />
`kids_u` FLOAT,<br />
`kids_u_odonnell_ext` FLOAT,<br />
`lsst_g_true` FLOAT,<br />
`lsst_g` FLOAT,<br />
`lsst_g_odonnell_ext` FLOAT,<br />
`lsst_i_true` FLOAT,<br />
`lsst_i` FLOAT,<br />
`lsst_i_odonnell_ext` FLOAT,<br />
`lsst_r_true` FLOAT,<br />
`lsst_r` FLOAT,<br />
`lsst_r_odonnell_ext` FLOAT,<br />
`lsst_u_true` FLOAT,<br />
`lsst_u` FLOAT,<br />
`lsst_u_odonnell_ext` FLOAT,<br />
`lsst_y_true` FLOAT,<br />
`lsst_y` FLOAT,<br />
`lsst_y_odonnell_ext` FLOAT,<br />
`lsst_z_true` FLOAT,<br />
`lsst_z` FLOAT,<br />
`lsst_z_odonnell_ext` FLOAT,<br />
`cfht_megacam_r_true` FLOAT,<br />
`cfht_megacam_r` FLOAT,<br />
`cfht_megacam_r_odonnell_ext` FLOAT,<br />
`cfht_megacam_u_true` FLOAT,<br />
`cfht_megacam_u` FLOAT,<br />
`cfht_megacam_u_odonnell_ext` FLOAT,<br />
`pan_starrs_i_true` FLOAT,<br />
`pan_starrs_i` FLOAT,<br />
`pan_starrs_i_odonnell_ext` FLOAT,<br />
`pan_starrs_z_true` FLOAT,<br />
`pan_starrs_z` FLOAT,<br />
`pan_starrs_z_odonnell_ext` FLOAT,<br />
`2mass_h_true` FLOAT,<br />
`2mass_h` FLOAT,<br />
`2mass_h_odonnell_ext` FLOAT,<br />
`2mass_j_true` FLOAT,<br />
`2mass_j` FLOAT,<br />
`2mass_j_odonnell_ext` FLOAT,<br />
`2mass_ks_true` FLOAT,<br />
`2mass_ks` FLOAT,<br />
`2mass_ks_odonnell_ext` FLOAT,<br />
`sdss_r01_true` FLOAT,<br />
`sdss_r01` FLOAT,<br />
`sdss_r01_odonnell_ext` FLOAT,<br />
`gaia_bp_true` FLOAT,<br />
`gaia_bp` FLOAT,<br />
`gaia_bp_odonnell_ext` FLOAT,<br />
`gaia_g_true` FLOAT,<br />
`gaia_g` FLOAT,<br />
`gaia_g_odonnell_ext` FLOAT,<br />
`gaia_rp_true` FLOAT,<br />
`gaia_rp` FLOAT,<br />
`gaia_rp_odonnell_ext` FLOAT,<br />
`euclid_nisp_h_true` FLOAT,<br />
`euclid_nisp_h` FLOAT,<br />
`euclid_nisp_h_odonnell_ext` FLOAT,<br />
`euclid_nisp_j_true` FLOAT,<br />
`euclid_nisp_j` FLOAT,<br />
`euclid_nisp_j_odonnell_ext` FLOAT,<br />
`euclid_nisp_y_true` FLOAT,<br />
`euclid_nisp_y` FLOAT,<br />
`euclid_nisp_y_odonnell_ext` FLOAT,<br />
`euclid_vis_true` FLOAT,<br />
`euclid_vis` FLOAT,<br />
`euclid_vis_odonnell_ext` FLOAT,<br />
`step` smallint COMMENT 'redshift step number'<br />
)<br />
CLUSTERED BY (<br />
step,<br />
hpix_13_nest)<br />
SORTED BY (<br />
step ASC,<br />
hpix_13_nest ASC)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
INSERT OVERWRITE TABLE cosmohub.flagship_mock_1_10_5_highz_c<br />
SELECT<br />
`unique_id`, `id`, `kind`, `z`, `ra`, `dec`, `l`, `b`, `mw_extinction`, `abs_muv`, `sedname`, `sedname_int`, `rhalf_kpc`, `rhalf_arcsec`, `muv`, `beta`, `hpix_9_nest`, SHIFTRIGHT(hpix_29_nest, (29-13)*2), `hpix_29_nest`, `blanco_decam_g_true`, `blanco_decam_g`, `blanco_decam_g_odonnell_ext`, `blanco_decam_i_true`, `blanco_decam_i`, `blanco_decam_i_odonnell_ext`, `blanco_decam_r_true`, `blanco_decam_r`, `blanco_decam_r_odonnell_ext`, `blanco_decam_z_true`, `blanco_decam_z`, `blanco_decam_z_odonnell_ext`, `subaru_hsc_z_true`, `subaru_hsc_z`, `subaru_hsc_z_odonnell_ext`, `jst_jpcam_g_true`, `jst_jpcam_g`, `jst_jpcam_g_odonnell_ext`, `kids_g_true`, `kids_g`, `kids_g_odonnell_ext`, `kids_i_true`, `kids_i`, `kids_i_odonnell_ext`, `kids_r_true`, `kids_r`, `kids_r_odonnell_ext`, `kids_u_true`, `kids_u`, `kids_u_odonnell_ext`, `lsst_g_true`, `lsst_g`, `lsst_g_odonnell_ext`, `lsst_i_true`, `lsst_i`, `lsst_i_odonnell_ext`, `lsst_r_true`, `lsst_r`, `lsst_r_odonnell_ext`, `lsst_u_true`, `lsst_u`, `lsst_u_odonnell_ext`, `lsst_y_true`, `lsst_y`, `lsst_y_odonnell_ext`, `lsst_z_true`, `lsst_z`, `lsst_z_odonnell_ext`, `cfht_megacam_r_true`, `cfht_megacam_r`, `cfht_megacam_r_odonnell_ext`, `cfht_megacam_u_true`, `cfht_megacam_u`, `cfht_megacam_u_odonnell_ext`, `pan_starrs_i_true`, `pan_starrs_i`, `pan_starrs_i_odonnell_ext`, `pan_starrs_z_true`, `pan_starrs_z`, `pan_starrs_z_odonnell_ext`, `2mass_h_true`, `2mass_h`, `2mass_h_odonnell_ext`, `2mass_j_true`, `2mass_j`, `2mass_j_odonnell_ext`, `2mass_ks_true`, `2mass_ks`, `2mass_ks_odonnell_ext`, `sdss_r01_true`, `sdss_r01`, `sdss_r01_odonnell_ext`, `gaia_bp_true`, `gaia_bp`, `gaia_bp_odonnell_ext`, `gaia_g_true`, `gaia_g`, `gaia_g_odonnell_ext`, `gaia_rp_true`, `gaia_rp`, `gaia_rp_odonnell_ext`, `euclid_nisp_h_true`, `euclid_nisp_h`, `euclid_nisp_h_odonnell_ext`, `euclid_nisp_j_true`, `euclid_nisp_j`, `euclid_nisp_j_odonnell_ext`, `euclid_nisp_y_true`, `euclid_nisp_y`, `euclid_nisp_y_odonnell_ext`, `euclid_vis_true`, `euclid_vis`, `euclid_vis_odonnell_ext`, `step`<br />
FROM jcarrete.flagship_mock_1_10_5_highz_pq;<br />
<br />
<br />
ANALYZE TABLE cosmohub.flagship_mock_1_10_5_highz_c COMPUTE STATISTICS FOR COLUMNS;<br />
<br />
Añado el lensing:<br />
<br />
CREATE TABLE cosmohub.flagship_mock_1_10_5_highz_s (<br />
`unique_id` BIGINT COMMENT 'unique identifier',<br />
`id` INT COMMENT 'ID',<br />
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',<br />
`z` float COMMENT 'true galaxy redshift',<br />
`ra` FLOAT COMMENT 'galaxy right ascension (degrees)',<br />
`dec` FLOAT COMMENT 'galaxy declination (degrees)',<br />
`ra_mag` float COMMENT 'galaxy magnified right ascension (degree)', <br />
`dec_mag` float COMMENT 'galaxy magnified declination (degree)', <br />
`kappa` float COMMENT 'convergence', <br />
`gamma1` float COMMENT 'shear', <br />
`gamma2` float COMMENT 'shear',<br />
`defl1` float COMMENT 'deflection1',<br />
`defl2` float COMMENT 'deflection2',<br />
`l` FLOAT COMMENT 'galactic longitude (degrees)',<br />
`b` FLOAT COMMENT 'galactic latitude (degrees)',<br />
`mw_extinction` float COMMENT 'EBV from file: HFI_CompMap_ThermalDustModel_2048_R1.20.fits',<br />
`abs_muv` FLOAT COMMENT 'absolute magnitude at 1500 angstroms',<br />
`sedname` STRING COMMENT 'Spectral Energy Distribution name (among 76, 38 without UV lines, 38 with UV lines, corresponding to a given UV slope)',<br />
`sedname_int` BIGINT COMMENT 'sedname as integer (from 0 to 75, from 0 to 37 without UV lines, from 38 to 75 with UV lines)',<br />
`rhalf_kpc` FLOAT COMMENT 'half light radius (in physical kpc)',<br />
`rhalf_arcsec` FLOAT COMMENT 'half light radius (in arcsec)',<br />
`muv` FLOAT COMMENT 'for information only, observed magnitude at rest-frame 1500',<br />
`beta` FLOAT COMMENT 'for information only, used to assign an SED',<br />
`hpix_9_nest` int COMMENT 'healpix pixel id using order = 9 in NESTED format',<br />
`hpix_13_nest` int COMMENT 'healpix pixel id using order = 13 in NESTED format',<br />
`hpix_29_nest` bigint COMMENT 'healpix pixel id using order = 29 in NESTED format',<br />
`blanco_decam_g_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in DES g-band (Euclid provided)', <br />
`blanco_decam_g` float COMMENT 'Interpolated observed flux in DES g-band (Euclid provided)',<br />
`blanco_decam_g_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in DES g-band (Euclid provided)',<br />
`blanco_decam_i_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in DES i-band (Euclid provided)', <br />
`blanco_decam_i` float COMMENT 'Interpolated observed flux in DES i-band (Euclid provided)', <br />
`blanco_decam_i_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in DES i-band (Euclid provided)', <br />
`blanco_decam_r_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in DES r-band (Euclid provided)', <br />
`blanco_decam_r` float COMMENT 'Interpolated observed flux in DES r-band (Euclid provided)', <br />
`blanco_decam_r_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in DES r-band (Euclid provided)', <br />
`blanco_decam_z_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in DES z-band (Euclid provided)', <br />
`blanco_decam_z` float COMMENT 'Interpolated observed flux in DES z-band (Euclid provided)',<br />
`blanco_decam_z_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in DES z-band (Euclid provided)', <br />
`subaru_hsc_z_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',<br />
`subaru_hsc_z` float COMMENT 'Interpolated observed flux in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',<br />
`subaru_hsc_z_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',<br />
`jst_jpcam_g_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in JST g-band (Euclid provided)', <br />
`jst_jpcam_g` float COMMENT 'Interpolated observed flux in JST g-band (Euclid provided)',<br />
`jst_jpcam_g_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in JST g-band (Euclid provided)',<br />
`kids_g_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in KIDS g-band (Euclid provided)', <br />
`kids_g` float COMMENT 'Interpolated observed flux in KIDS g-band (Euclid provided)',<br />
`kids_g_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in KIDS g-band (Euclid provided)',<br />
`kids_i_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in KIDS i-band (Euclid provided)', <br />
`kids_i` float COMMENT 'Interpolated observed flux in KIDS i-band (Euclid provided)',<br />
`kids_i_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in KIDS i-band (Euclid provided)', <br />
`kids_r_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in KIDS r-band (Euclid provided)', <br />
`kids_r` float COMMENT 'Interpolated observed flux in KIDS r-band (Euclid provided)',<br />
`kids_r_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in KIDS r-band (Euclid provided)',<br />
`kids_u_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in KIDS u-band (Euclid provided)', <br />
`kids_u` float COMMENT 'Interpolated observed flux in KIDS u-band (Euclid provided)',<br />
`kids_u_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in KIDS u-band (Euclid provided)',<br />
`lsst_g_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST g-band (Euclid provided)', <br />
`lsst_g` float COMMENT 'Interpolated observed flux in LSST g-band (Euclid provided)',<br />
`lsst_g_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST g-band (Euclid provided)',<br />
`lsst_i_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST i-band (Euclid provided)', <br />
`lsst_i` float COMMENT 'Interpolated observed flux in LSST i-band (Euclid provided)',<br />
`lsst_i_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST i-band (Euclid provided)',<br />
`lsst_r_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST r-band (Euclid provided)', <br />
`lsst_r` float COMMENT 'Interpolated observed flux in LSST r-band (Euclid provided)',<br />
`lsst_r_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST r-band (Euclid provided)',<br />
`lsst_u_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST u-band (Euclid provided)', <br />
`lsst_u` float COMMENT 'Interpolated observed flux in LSST u-band (Euclid provided)',<br />
`lsst_u_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST u-band (Euclid provided)',<br />
`lsst_y_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST y-band (Euclid provided)', <br />
`lsst_y` float COMMENT 'Interpolated observed flux in LSST y-band (Euclid provided)',<br />
`lsst_y_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST y-band (Euclid provided)',<br />
`lsst_z_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST z-band (Euclid provided)', <br />
`lsst_z` float COMMENT 'Interpolated observed flux in LSST z-band (Euclid provided)',<br />
`lsst_z_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST z-band (Euclid provided)',<br />
`cfht_megacam_r_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in CFHT r-band (Euclid provided)', <br />
`cfht_megacam_r` float COMMENT 'Interpolated observed flux in CFHT r-band (Euclid provided)',<br />
`cfht_megacam_r_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in CFHT r-band (Euclid provided)',<br />
`cfht_megacam_u_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in CFHT u-band (Euclid provided)', <br />
`cfht_megacam_u` float COMMENT 'Interpolated observed flux in CFHT u-band (Euclid provided)',<br />
`cfht_megacam_u_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in CFHT u-band (Euclid provided)',<br />
`pan_starrs_i_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Pan-STARRS i-band (Euclid provided)', <br />
`pan_starrs_i` float COMMENT 'Interpolated observed flux in Pan-STARRS i-band (Euclid provided)',<br />
`pan_starrs_i_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Pan-STARRS i-band (Euclid provided)',<br />
`pan_starrs_z_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Pan-STARRS z-band (Euclid provided)', <br />
`pan_starrs_z` float COMMENT 'Interpolated observed flux in Pan-STARRS z-band (Euclid provided)',<br />
`pan_starrs_z_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Pan-STARRS z-band (Euclid provided)',<br />
`2mass_h_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in 2MASS H-band (Euclid provided)', <br />
`2mass_h` float COMMENT 'Interpolated observed flux in 2MASS H-band (Euclid provided)',<br />
`2mass_h_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in 2MASS H-band (Euclid provided)',<br />
`2mass_j_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in 2MASS J-band (Euclid provided)', <br />
`2mass_j` float COMMENT 'Interpolated observed flux in 2MASS J-band (Euclid provided)',<br />
`2mass_j_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in 2MASS J-band (Euclid provided)',<br />
`2mass_ks_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in 2MASS Ks-band (Euclid provided)', <br />
`2mass_ks` float COMMENT 'Interpolated observed flux in 2MASS Ks-band (Euclid provided)',<br />
`2mass_ks_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in 2MASS Ks-band (Euclid provided)',<br />
`sdss_r01_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in sdss r-band redshifted to z=0.1 (Euclid provided)', <br />
`sdss_r01` float COMMENT 'Interpolated observed flux in sdss r-band redshifted to z=0.1 (Euclid provided)',<br />
`sdss_r01_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in sdss r-band redshifted to z=0.1 (Euclid provided)',<br />
`gaia_bp_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Gaia-BP band (Euclid provided)', <br />
`gaia_bp` float COMMENT 'Interpolated observed flux in Gaia-BP band (Euclid provided)',<br />
`gaia_bp_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Gaia-BP band (Euclid provided)',<br />
`gaia_g_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Gaia-G band (Euclid provided)', <br />
`gaia_g` float COMMENT 'Interpolated observed flux in Gaia-G band (Euclid provided)',<br />
`gaia_g_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Gaia-G band (Euclid provided)', <br />
`gaia_rp_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Gaia-RP band (Euclid provided)', <br />
`gaia_rp` float COMMENT 'Interpolated observed flux in Gaia-RP band (Euclid provided)',<br />
`gaia_rp_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Gaia-RP band (Euclid provided)',<br />
`euclid_nisp_h_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Euclid NISP-H band (Euclid provided)', <br />
`euclid_nisp_h` float COMMENT 'Interpolated observed flux in Euclid NISP-H band (Euclid provided)',<br />
`euclid_nisp_h_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Euclid NISP-H band (Euclid provided)', <br />
`euclid_nisp_j_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Euclid NISP-J band (Euclid provided)', <br />
`euclid_nisp_j` float COMMENT 'Interpolated observed flux in Euclid NISP-J band (Euclid provided)',<br />
`euclid_nisp_j_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Euclid NISP-J band (Euclid provided)',<br />
`euclid_nisp_y_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Euclid NISP-Y band (Euclid provided)', <br />
`euclid_nisp_y` float COMMENT 'Interpolated observed flux in Euclid NISP-Y band (Euclid provided)',<br />
`euclid_nisp_y_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Euclid NISP-Y band (Euclid provided)', <br />
`euclid_vis_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Euclid VIS band (Euclid provided)', <br />
`euclid_vis` float COMMENT 'Interpolated observed flux in Euclid VIS band (Euclid provided)',<br />
`euclid_vis_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Euclid VIS band (Euclid provided)',<br />
`step` smallint COMMENT 'redshift step number'<br />
)<br />
CLUSTERED BY (<br />
step,<br />
hpix_13_nest)<br />
SORTED BY (<br />
step ASC,<br />
hpix_13_nest ASC)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
<br />
SET hive.auto.convert.join.noconditionaltask.size=1000;<br />
<br />
INSERT OVERWRITE TABLE cosmohub.flagship_mock_1_10_5_highz_s<br />
SELECT <br />
`unique_id`, `id`, `kind`, `z`, `ra`, `dec`, mag_pos.ra_mag, mag_pos.dec_mag, kappa, gamma1, gamma2, defl1, defl2, `l`, `b`, `mw_extinction`, `abs_muv`, `sedname`, `sedname_int`, `rhalf_kpc`, `rhalf_arcsec`, `muv`, `beta`, `hpix_9_nest`, `hpix_13_nest`, `hpix_29_nest`, `blanco_decam_g_true`, `blanco_decam_g`, `blanco_decam_g_odonnell_ext`, `blanco_decam_i_true`, `blanco_decam_i`, `blanco_decam_i_odonnell_ext`, `blanco_decam_r_true`, `blanco_decam_r`, `blanco_decam_r_odonnell_ext`, `blanco_decam_z_true`, `blanco_decam_z`, `blanco_decam_z_odonnell_ext`, `subaru_hsc_z_true`, `subaru_hsc_z`, `subaru_hsc_z_odonnell_ext`, `jst_jpcam_g_true`, `jst_jpcam_g`, `jst_jpcam_g_odonnell_ext`, `kids_g_true`, `kids_g`, `kids_g_odonnell_ext`, `kids_i_true`, `kids_i`, `kids_i_odonnell_ext`, `kids_r_true`, `kids_r`, `kids_r_odonnell_ext`, `kids_u_true`, `kids_u`, `kids_u_odonnell_ext`, `lsst_g_true`, `lsst_g`, `lsst_g_odonnell_ext`, `lsst_i_true`, `lsst_i`, `lsst_i_odonnell_ext`, `lsst_r_true`, `lsst_r`, `lsst_r_odonnell_ext`, `lsst_u_true`, `lsst_u`, `lsst_u_odonnell_ext`, `lsst_y_true`, `lsst_y`, `lsst_y_odonnell_ext`, `lsst_z_true`, `lsst_z`, `lsst_z_odonnell_ext`, `cfht_megacam_r_true`, `cfht_megacam_r`, `cfht_megacam_r_odonnell_ext`, `cfht_megacam_u_true`, `cfht_megacam_u`, `cfht_megacam_u_odonnell_ext`, `pan_starrs_i_true`, `pan_starrs_i`, `pan_starrs_i_odonnell_ext`, `pan_starrs_z_true`, `pan_starrs_z`, `pan_starrs_z_odonnell_ext`, `2mass_h_true`, `2mass_h`, `2mass_h_odonnell_ext`, `2mass_j_true`, `2mass_j`, `2mass_j_odonnell_ext`, `2mass_ks_true`, `2mass_ks`, `2mass_ks_odonnell_ext`, `sdss_r01_true`, `sdss_r01`, `sdss_r01_odonnell_ext`, `gaia_bp_true`, `gaia_bp`, `gaia_bp_odonnell_ext`, `gaia_g_true`, `gaia_g`, `gaia_g_odonnell_ext`, `gaia_rp_true`, `gaia_rp`, `gaia_rp_odonnell_ext`, `euclid_nisp_h_true`, `euclid_nisp_h`, `euclid_nisp_h_odonnell_ext`, `euclid_nisp_j_true`, `euclid_nisp_j`, `euclid_nisp_j_odonnell_ext`, `euclid_nisp_y_true`, `euclid_nisp_y`, `euclid_nisp_y_odonnell_ext`, `euclid_vis_true`, `euclid_vis`, `euclid_vis_odonnell_ext`, `step`<br />
FROM<br />
(SELECT <br />
`unique_id`, `id`, `kind`, `z`, `ra`, `dec`, udf.magnified_positions(CAST(ra AS DOUBLE), CAST(`dec` AS DOUBLE), CAST(l.defl1 AS DOUBLE), CAST(l.defl2 AS DOUBLE)) as mag_pos, l.kappa, l.gamma1, l.gamma2, l.defl1, l.defl2, `l`, `b`, `mw_extinction`, `abs_muv`, `sedname`, `sedname_int`, `rhalf_kpc`, `rhalf_arcsec`, `muv`, `beta`, `hpix_9_nest`, l.hpix_13_nest, `hpix_29_nest`, `blanco_decam_g_true`, `blanco_decam_g`, `blanco_decam_g_odonnell_ext`, `blanco_decam_i_true`, `blanco_decam_i`, `blanco_decam_i_odonnell_ext`, `blanco_decam_r_true`, `blanco_decam_r`, `blanco_decam_r_odonnell_ext`, `blanco_decam_z_true`, `blanco_decam_z`, `blanco_decam_z_odonnell_ext`, `subaru_hsc_z_true`, `subaru_hsc_z`, `subaru_hsc_z_odonnell_ext`, `jst_jpcam_g_true`, `jst_jpcam_g`, `jst_jpcam_g_odonnell_ext`, `kids_g_true`, `kids_g`, `kids_g_odonnell_ext`, `kids_i_true`, `kids_i`, `kids_i_odonnell_ext`, `kids_r_true`, `kids_r`, `kids_r_odonnell_ext`, `kids_u_true`, `kids_u`, `kids_u_odonnell_ext`, `lsst_g_true`, `lsst_g`, `lsst_g_odonnell_ext`, `lsst_i_true`, `lsst_i`, `lsst_i_odonnell_ext`, `lsst_r_true`, `lsst_r`, `lsst_r_odonnell_ext`, `lsst_u_true`, `lsst_u`, `lsst_u_odonnell_ext`, `lsst_y_true`, `lsst_y`, `lsst_y_odonnell_ext`, `lsst_z_true`, `lsst_z`, `lsst_z_odonnell_ext`, `cfht_megacam_r_true`, `cfht_megacam_r`, `cfht_megacam_r_odonnell_ext`, `cfht_megacam_u_true`, `cfht_megacam_u`, `cfht_megacam_u_odonnell_ext`, `pan_starrs_i_true`, `pan_starrs_i`, `pan_starrs_i_odonnell_ext`, `pan_starrs_z_true`, `pan_starrs_z`, `pan_starrs_z_odonnell_ext`, `2mass_h_true`, `2mass_h`, `2mass_h_odonnell_ext`, `2mass_j_true`, `2mass_j`, `2mass_j_odonnell_ext`, `2mass_ks_true`, `2mass_ks`, `2mass_ks_odonnell_ext`, `sdss_r01_true`, `sdss_r01`, `sdss_r01_odonnell_ext`, `gaia_bp_true`, `gaia_bp`, `gaia_bp_odonnell_ext`, `gaia_g_true`, `gaia_g`, `gaia_g_odonnell_ext`, `gaia_rp_true`, `gaia_rp`, `gaia_rp_odonnell_ext`, `euclid_nisp_h_true`, `euclid_nisp_h`, `euclid_nisp_h_odonnell_ext`, `euclid_nisp_j_true`, `euclid_nisp_j`, `euclid_nisp_j_odonnell_ext`, `euclid_nisp_y_true`, `euclid_nisp_y`, `euclid_nisp_y_odonnell_ext`, `euclid_vis_true`, `euclid_vis`, `euclid_vis_odonnell_ext`, l.step<br />
FROM cosmohub.flagship_mock_1_10_5_highz_c as m<br />
JOIN cosmohub.flagship_lensing2 as l<br />
ON m.step = l.step<br />
AND m.hpix_13_nest = l.hpix_13_nest) as t<br />
;<br />
<br />
<br />
ANALYZE TABLE cosmohub.flagship_mock_1_10_5_highz_s COMPUTE STATISTICS FOR COLUMNS;<br />
<br />
----<br />
<br />
Information in CosmoHub:<br />
<br />
<br />
This is the first SC8 High-z galaxies release.<br />
<br />
We have selected galaxies from [https://cosmohub.pic.es/catalogs/209](https://cosmohub.pic.es/catalogs/209) (copied information below) and located randomly in SC8 area.<br />
<br />
We have assigned lensing parameters from last redshift step (86) since lensing and deflection maps are not yet available for higher redshift.<br />
<br />
Therefore we have used those parameters to estimate the magnified positions and also magnified fluxes in the TU FITS files.<br />
<br />
Note that **we provide fluxes instead of magnitudes**. In order **to get magnitudes** you can enter in the "Expert mode" and estimate the magnitude as follows: <br />
<br />
m = -2.5 * log10(flux) - 48.6<br />
<br />
And in order to estimate magnified magnitudes:<br />
<br />
m_mag = m + 2.5 * log10((1-kappa)*(1-kappa) - gamma1 * gamma1 - gamma2 * gamma2)<br />
<br />
---<br />
<br />
**The catalog of LBG sources is based on the "Recipes" described in this [Euclid Redmine entry](https://euclid.roe.ac.uk/projects/puswg/wiki/Galaxies)**.<br />
<br />
Important features to be noted:<br />
<br />
1) The corresponding area is 16 deg2<br />
<br />
2) Magnitude cut at H=27 (AB mag)<br />
<br />
3) SEDs are provided with given UV continuum slopes<br />
<br />
4) A standard cosmology with h=0.7, Om=0.3, Olambda=0.7 was used to convert kpc to arcsec and abs_muv (absolute magnitude at 1500 angstroms) to muv (observed magnitude at rest-frame 1500)<br />
<br />
<br />
Fits file example in CosmoHub:<br />
<br />
{<br />
"sql": "SELECT CAST(unique_id AS bigint) AS SOURCE_ID, \nCAST(-1 AS float) AS HALO_ID, \nCAST(kind AS smallint) AS KIND, \nCAST(ra AS double) AS RA, \nCAST(`dec` AS double) AS `DEC`, \nCAST(ra_mag AS double) AS RA_MAG, \nCAST(dec_mag AS double) AS DEC_MAG, \nCAST(z AS float) AS Z_OBS, \nCAST(abs_muv AS float) AS REF_MAG_ABS, \nCAST(abs_muv AS float) AS REF_MAG, \nCAST(1 AS float) AS BULGE_FRACTION, \nCAST(rhalf_arcsec AS float) AS BULGE_R50, \nCAST(rhalf_arcsec AS float) AS DISK_R50, \nCAST(1.5 AS float) AS BULGE_NSERSIC, \nCAST(-1 AS float) AS BULGE_AXIS_RATIO, \nCAST(-1 AS float) AS INCLINATION_ANGLE, \nCAST(-1 AS float) AS DISK_ANGLE, \nCAST(kappa AS float) AS KAPPA, \nCAST(gamma1 AS float) AS GAMMA1, \nCAST(gamma2 AS float) AS GAMMA2, \nCAST(sedname_int AS float) AS SED_TEMPLATE, \nCAST(0 AS smallint) AS EXT_LAW, \nCAST(-1 AS float) AS EBV, \nCAST(-1 AS float) AS HALPHA_LOGFLAM_EXT_MAG, \nCAST(-1 AS float) AS HBETA_LOGFLAM_EXT_MAG, \nCAST(-1 AS float) AS O2_LOGFLAM_EXT_MAG, \nCAST(-1 AS float) AS O3_LOGFLAM_EXT_MAG, \nCAST(-1 AS float) AS N2_LOGFLAM_EXT_MAG, \nCAST(-1 AS float) AS S2_LOGFLAM_EXT_MAG, \nCAST(mw_extinction AS float) AS AV,\nCAST(euclid_vis*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_VIS_MAG, \nCAST(euclid_nisp_y*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_NISP_MAG, \nCAST(euclid_nisp_j*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_NISP_MAG, \nCAST(euclid_nisp_h*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_NISP_MAG, \nCAST(blanco_decam_g*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_DECAM_MAG, \nCAST(blanco_decam_r*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_DECAM_MAG, \nCAST(blanco_decam_i*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_DECAM_MAG, \nCAST(blanco_decam_z*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_DECAM_MAG, \nCAST(cfht_megacam_u*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_MEGACAM_MAG, \nCAST(cfht_megacam_r*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_MEGACAM_MAG, \nCAST(jst_jpcam_g*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_JPCAM_MAG, \nCAST(pan_starrs_i*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_PANSTARRS_MAG, \nCAST(pan_starrs_z*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_PANSTARRS_MAG, \nCAST(subaru_hsc_z*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_HSC_MAG, \nCAST(gaia_g*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_GAIA_MAG, \nCAST(gaia_bp*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_BP_GAIA_MAG, \nCAST(gaia_rp*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_RP_GAIA_MAG, \nCAST(lsst_u*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_LSST_MAG, \nCAST(lsst_g*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_LSST_MAG, \nCAST(lsst_r*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_LSST_MAG, \nCAST(lsst_i*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_LSST_MAG, \nCAST(lsst_z*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_LSST_MAG, \nCAST(lsst_y*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_LSST_MAG, \nCAST(kids_u*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_KIDS_MAG, \nCAST(kids_g*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_KIDS_MAG, \nCAST(kids_r*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_KIDS_MAG, \nCAST(kids_i*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_KIDS_MAG, \nCAST(2mass_j*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_2MASS_MAG, \nCAST(2mass_h*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_2MASS_MAG, \nCAST(2mass_ks*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_KS_2MASS_MAG \nFROM cosmohub.flagship_mock_1_10_5_highz_s \nWHERE SHIFTRIGHT(hpix_29_nest, (29-5)*2) = 2063"<br />
}<br />
<br />
I made the following choices:<br />
<br />
* HALO_ID = -1 when kind = 2 or 3<br />
* BULGE_FRACTION = 1 and then the disk information is not used!<br />
* BULGE_NSERSIC = 1.5<br />
<br />
The rest of the fields that are not included in the catalog but are necessary for the FITS file format = -1; [REF_MAG_ABS, REF_MAG] = (m1450 OR abs_muv) when kind = 2 and 3, respectively.<br />
<br />
<br />
SELECT `unique_id`, `kind`, `ra`, `dec`, `ra_mag`, `dec_mag`, `kappa`, `gamma1`, `gamma2`, `defl1`, `defl2`, `hpix_9_nest`, `hpix_13_nest`, `hpix_29_nest`, `step` <br />
FROM flagship_mock_1_10_5_highz_s LIMIT 2;<br />
<br />
----<br />
<br />
Aplicamos los cambios en la pipeline de high-z galaxies (.ipynb) que nos menciona Eric en el email y generamos un nuevo catálogo en formato parquet:<br />
<br />
/software/astro/scripts/create_table_from_parquet.sh -p /hdfs/user/jcarrete/data/euclid/flagship_mock_1_10_7_high_z/0.pq -t jcarrete.flagship_mock_1_10_7_highz_pq -e<br />
<br />
CREATE EXTERNAL TABLE jcarrete.flagship_mock_1_10_7_highz_pq (<br />
`type` STRING,<br />
`z` DOUBLE,<br />
`abs_muv` DOUBLE,<br />
`sedname` STRING,<br />
`sedname_int` BIGINT,<br />
`rhalf_kpc` DOUBLE,<br />
`rhalf_arcsec` DOUBLE,<br />
`muv` DOUBLE,<br />
`beta` DOUBLE,<br />
`r` DOUBLE,<br />
`hpix_9_nest` BIGINT,<br />
`kind` BIGINT,<br />
`step` BIGINT,<br />
`unique_id` BIGINT,<br />
`hpix_29_nest` BIGINT,<br />
`ra` DOUBLE,<br />
`dec` DOUBLE,<br />
`l` DOUBLE,<br />
`b` DOUBLE,<br />
`mw_extinction` DOUBLE,<br />
`blanco_decam_g_true` DOUBLE,<br />
`blanco_decam_g` DOUBLE,<br />
`blanco_decam_g_odonnell_ext` DOUBLE,<br />
`blanco_decam_i_true` DOUBLE,<br />
`blanco_decam_i` DOUBLE,<br />
`blanco_decam_i_odonnell_ext` DOUBLE,<br />
`blanco_decam_r_true` DOUBLE,<br />
`blanco_decam_r` DOUBLE,<br />
`blanco_decam_r_odonnell_ext` DOUBLE,<br />
`blanco_decam_z_true` DOUBLE,<br />
`blanco_decam_z` DOUBLE,<br />
`blanco_decam_z_odonnell_ext` DOUBLE,<br />
`subaru_hsc_z_true` DOUBLE,<br />
`subaru_hsc_z` DOUBLE,<br />
`subaru_hsc_z_odonnell_ext` DOUBLE,<br />
`jst_jpcam_g_true` DOUBLE,<br />
`jst_jpcam_g` DOUBLE,<br />
`jst_jpcam_g_odonnell_ext` DOUBLE,<br />
`kids_g_true` DOUBLE,<br />
`kids_g` DOUBLE,<br />
`kids_g_odonnell_ext` DOUBLE,<br />
`kids_i_true` DOUBLE,<br />
`kids_i` DOUBLE,<br />
`kids_i_odonnell_ext` DOUBLE,<br />
`kids_r_true` DOUBLE,<br />
`kids_r` DOUBLE,<br />
`kids_r_odonnell_ext` DOUBLE,<br />
`kids_u_true` DOUBLE,<br />
`kids_u` DOUBLE,<br />
`kids_u_odonnell_ext` DOUBLE,<br />
`lsst_g_true` DOUBLE,<br />
`lsst_g` DOUBLE,<br />
`lsst_g_odonnell_ext` DOUBLE,<br />
`lsst_i_true` DOUBLE,<br />
`lsst_i` DOUBLE,<br />
`lsst_i_odonnell_ext` DOUBLE,<br />
`lsst_r_true` DOUBLE,<br />
`lsst_r` DOUBLE,<br />
`lsst_r_odonnell_ext` DOUBLE,<br />
`lsst_u_true` DOUBLE,<br />
`lsst_u` DOUBLE,<br />
`lsst_u_odonnell_ext` DOUBLE,<br />
`lsst_y_true` DOUBLE,<br />
`lsst_y` DOUBLE,<br />
`lsst_y_odonnell_ext` DOUBLE,<br />
`lsst_z_true` DOUBLE,<br />
`lsst_z` DOUBLE,<br />
`lsst_z_odonnell_ext` DOUBLE,<br />
`cfht_megacam_r_true` DOUBLE,<br />
`cfht_megacam_r` DOUBLE,<br />
`cfht_megacam_r_odonnell_ext` DOUBLE,<br />
`cfht_megacam_u_true` DOUBLE,<br />
`cfht_megacam_u` DOUBLE,<br />
`cfht_megacam_u_odonnell_ext` DOUBLE,<br />
`pan_starrs_i_true` DOUBLE,<br />
`pan_starrs_i` DOUBLE,<br />
`pan_starrs_i_odonnell_ext` DOUBLE,<br />
`pan_starrs_z_true` DOUBLE,<br />
`pan_starrs_z` DOUBLE,<br />
`pan_starrs_z_odonnell_ext` DOUBLE,<br />
`2mass_h_true` DOUBLE,<br />
`2mass_h` DOUBLE,<br />
`2mass_h_odonnell_ext` DOUBLE,<br />
`2mass_j_true` DOUBLE,<br />
`2mass_j` DOUBLE,<br />
`2mass_j_odonnell_ext` DOUBLE,<br />
`2mass_ks_true` DOUBLE,<br />
`2mass_ks` DOUBLE,<br />
`2mass_ks_odonnell_ext` DOUBLE,<br />
`sdss_r01_true` DOUBLE,<br />
`sdss_r01` DOUBLE,<br />
`sdss_r01_odonnell_ext` DOUBLE,<br />
`gaia_bp_true` DOUBLE,<br />
`gaia_bp` DOUBLE,<br />
`gaia_bp_odonnell_ext` DOUBLE,<br />
`gaia_g_true` DOUBLE,<br />
`gaia_g` DOUBLE,<br />
`gaia_g_odonnell_ext` DOUBLE,<br />
`gaia_rp_true` DOUBLE,<br />
`gaia_rp` DOUBLE,<br />
`gaia_rp_odonnell_ext` DOUBLE,<br />
`euclid_nisp_h_true` DOUBLE,<br />
`euclid_nisp_h` DOUBLE,<br />
`euclid_nisp_h_odonnell_ext` DOUBLE,<br />
`euclid_nisp_j_true` DOUBLE,<br />
`euclid_nisp_j` DOUBLE,<br />
`euclid_nisp_j_odonnell_ext` DOUBLE,<br />
`euclid_nisp_y_true` DOUBLE,<br />
`euclid_nisp_y` DOUBLE,<br />
`euclid_nisp_y_odonnell_ext` DOUBLE,<br />
`euclid_vis_true` DOUBLE,<br />
`euclid_vis` DOUBLE,<br />
`euclid_vis_odonnell_ext` DOUBLE,<br />
`bulge_fraction` DOUBLE,<br />
`bulge_nsersic` DOUBLE,<br />
`bulge_ellipticity` DOUBLE,<br />
`bulge_axis_ratio` DOUBLE,<br />
`disk_angle` DOUBLE,<br />
`id` BIGINT<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/jcarrete/data/euclid/flagship_mock_1_10_7_high_z'<br />
;<br />
<br />
Clusterizo:<br />
<br />
CREATE TABLE cosmohub.flagship_mock_1_10_7_highz_c (<br />
`unique_id` BIGINT COMMENT 'unique identifier',<br />
`id` INT COMMENT 'ID',<br />
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',<br />
`z` FLOAT,<br />
`ra` FLOAT,<br />
`dec` FLOAT,<br />
`l` FLOAT,<br />
`b` FLOAT,<br />
`mw_extinction` FLOAT,<br />
`abs_muv` FLOAT,<br />
`sedname` STRING,<br />
`sedname_int` BIGINT,<br />
`rhalf_kpc` FLOAT,<br />
`rhalf_arcsec` FLOAT,<br />
`muv` FLOAT,<br />
`beta` FLOAT,<br />
`hpix_9_nest` INT,<br />
`hpix_13_nest` INT,<br />
`hpix_29_nest` BIGINT,<br />
`blanco_decam_g_true` FLOAT,<br />
`blanco_decam_g` FLOAT,<br />
`blanco_decam_g_odonnell_ext` FLOAT,<br />
`blanco_decam_i_true` FLOAT,<br />
`blanco_decam_i` FLOAT,<br />
`blanco_decam_i_odonnell_ext` FLOAT,<br />
`blanco_decam_r_true` FLOAT,<br />
`blanco_decam_r` FLOAT,<br />
`blanco_decam_r_odonnell_ext` FLOAT,<br />
`blanco_decam_z_true` FLOAT,<br />
`blanco_decam_z` FLOAT,<br />
`blanco_decam_z_odonnell_ext` FLOAT,<br />
`subaru_hsc_z_true` FLOAT,<br />
`subaru_hsc_z` FLOAT,<br />
`subaru_hsc_z_odonnell_ext` FLOAT,<br />
`jst_jpcam_g_true` FLOAT,<br />
`jst_jpcam_g` FLOAT,<br />
`jst_jpcam_g_odonnell_ext` FLOAT,<br />
`kids_g_true` FLOAT,<br />
`kids_g` FLOAT,<br />
`kids_g_odonnell_ext` FLOAT,<br />
`kids_i_true` FLOAT,<br />
`kids_i` FLOAT,<br />
`kids_i_odonnell_ext` FLOAT,<br />
`kids_r_true` FLOAT,<br />
`kids_r` FLOAT,<br />
`kids_r_odonnell_ext` FLOAT,<br />
`kids_u_true` FLOAT,<br />
`kids_u` FLOAT,<br />
`kids_u_odonnell_ext` FLOAT,<br />
`lsst_g_true` FLOAT,<br />
`lsst_g` FLOAT,<br />
`lsst_g_odonnell_ext` FLOAT,<br />
`lsst_i_true` FLOAT,<br />
`lsst_i` FLOAT,<br />
`lsst_i_odonnell_ext` FLOAT,<br />
`lsst_r_true` FLOAT,<br />
`lsst_r` FLOAT,<br />
`lsst_r_odonnell_ext` FLOAT,<br />
`lsst_u_true` FLOAT,<br />
`lsst_u` FLOAT,<br />
`lsst_u_odonnell_ext` FLOAT,<br />
`lsst_y_true` FLOAT,<br />
`lsst_y` FLOAT,<br />
`lsst_y_odonnell_ext` FLOAT,<br />
`lsst_z_true` FLOAT,<br />
`lsst_z` FLOAT,<br />
`lsst_z_odonnell_ext` FLOAT,<br />
`cfht_megacam_r_true` FLOAT,<br />
`cfht_megacam_r` FLOAT,<br />
`cfht_megacam_r_odonnell_ext` FLOAT,<br />
`cfht_megacam_u_true` FLOAT,<br />
`cfht_megacam_u` FLOAT,<br />
`cfht_megacam_u_odonnell_ext` FLOAT,<br />
`pan_starrs_i_true` FLOAT,<br />
`pan_starrs_i` FLOAT,<br />
`pan_starrs_i_odonnell_ext` FLOAT,<br />
`pan_starrs_z_true` FLOAT,<br />
`pan_starrs_z` FLOAT,<br />
`pan_starrs_z_odonnell_ext` FLOAT,<br />
`2mass_h_true` FLOAT,<br />
`2mass_h` FLOAT,<br />
`2mass_h_odonnell_ext` FLOAT,<br />
`2mass_j_true` FLOAT,<br />
`2mass_j` FLOAT,<br />
`2mass_j_odonnell_ext` FLOAT,<br />
`2mass_ks_true` FLOAT,<br />
`2mass_ks` FLOAT,<br />
`2mass_ks_odonnell_ext` FLOAT,<br />
`sdss_r01_true` FLOAT,<br />
`sdss_r01` FLOAT,<br />
`sdss_r01_odonnell_ext` FLOAT,<br />
`gaia_bp_true` FLOAT,<br />
`gaia_bp` FLOAT,<br />
`gaia_bp_odonnell_ext` FLOAT,<br />
`gaia_g_true` FLOAT,<br />
`gaia_g` FLOAT,<br />
`gaia_g_odonnell_ext` FLOAT,<br />
`gaia_rp_true` FLOAT,<br />
`gaia_rp` FLOAT,<br />
`gaia_rp_odonnell_ext` FLOAT,<br />
`euclid_nisp_h_true` FLOAT,<br />
`euclid_nisp_h` FLOAT,<br />
`euclid_nisp_h_odonnell_ext` FLOAT,<br />
`euclid_nisp_j_true` FLOAT,<br />
`euclid_nisp_j` FLOAT,<br />
`euclid_nisp_j_odonnell_ext` FLOAT,<br />
`euclid_nisp_y_true` FLOAT,<br />
`euclid_nisp_y` FLOAT,<br />
`euclid_nisp_y_odonnell_ext` FLOAT,<br />
`euclid_vis_true` FLOAT,<br />
`euclid_vis` FLOAT,<br />
`euclid_vis_odonnell_ext` FLOAT,<br />
`bulge_fraction` FLOAT,<br />
`bulge_nsersic` FLOAT,<br />
`bulge_ellipticity` FLOAT,<br />
`bulge_axis_ratio` FLOAT,<br />
`disk_angle` FLOAT,<br />
`step` smallint COMMENT 'redshift step number'<br />
)<br />
CLUSTERED BY (<br />
step,<br />
hpix_13_nest)<br />
SORTED BY (<br />
step ASC,<br />
hpix_13_nest ASC)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
INSERT OVERWRITE TABLE cosmohub.flagship_mock_1_10_7_highz_c<br />
SELECT `unique_id`, `id`, `kind`, `z`, `ra`, `dec`, `l`, `b`, `mw_extinction`, `abs_muv`, `sedname`, `sedname_int`, `rhalf_kpc`, `rhalf_arcsec`, `muv`, `beta`, `hpix_9_nest`, SHIFTRIGHT(hpix_29_nest, (29-13)*2), `hpix_29_nest`, `blanco_decam_g_true`, `blanco_decam_g`, `blanco_decam_g_odonnell_ext`, `blanco_decam_i_true`, `blanco_decam_i`, `blanco_decam_i_odonnell_ext`, `blanco_decam_r_true`, `blanco_decam_r`, `blanco_decam_r_odonnell_ext`, `blanco_decam_z_true`, `blanco_decam_z`, `blanco_decam_z_odonnell_ext`, `subaru_hsc_z_true`, `subaru_hsc_z`, `subaru_hsc_z_odonnell_ext`, `jst_jpcam_g_true`, `jst_jpcam_g`, `jst_jpcam_g_odonnell_ext`, `kids_g_true`, `kids_g`, `kids_g_odonnell_ext`, `kids_i_true`, `kids_i`, `kids_i_odonnell_ext`, `kids_r_true`, `kids_r`, `kids_r_odonnell_ext`, `kids_u_true`, `kids_u`, `kids_u_odonnell_ext`, `lsst_g_true`, `lsst_g`, `lsst_g_odonnell_ext`, `lsst_i_true`, `lsst_i`, `lsst_i_odonnell_ext`, `lsst_r_true`, `lsst_r`, `lsst_r_odonnell_ext`, `lsst_u_true`, `lsst_u`, `lsst_u_odonnell_ext`, `lsst_y_true`, `lsst_y`, `lsst_y_odonnell_ext`, `lsst_z_true`, `lsst_z`, `lsst_z_odonnell_ext`, `cfht_megacam_r_true`, `cfht_megacam_r`, `cfht_megacam_r_odonnell_ext`, `cfht_megacam_u_true`, `cfht_megacam_u`, `cfht_megacam_u_odonnell_ext`, `pan_starrs_i_true`, `pan_starrs_i`, `pan_starrs_i_odonnell_ext`, `pan_starrs_z_true`, `pan_starrs_z`, `pan_starrs_z_odonnell_ext`, `2mass_h_true`, `2mass_h`, `2mass_h_odonnell_ext`, `2mass_j_true`, `2mass_j`, `2mass_j_odonnell_ext`, `2mass_ks_true`, `2mass_ks`, `2mass_ks_odonnell_ext`, `sdss_r01_true`, `sdss_r01`, `sdss_r01_odonnell_ext`, `gaia_bp_true`, `gaia_bp`, `gaia_bp_odonnell_ext`, `gaia_g_true`, `gaia_g`, `gaia_g_odonnell_ext`, `gaia_rp_true`, `gaia_rp`, `gaia_rp_odonnell_ext`, `euclid_nisp_h_true`, `euclid_nisp_h`, `euclid_nisp_h_odonnell_ext`, `euclid_nisp_j_true`, `euclid_nisp_j`, `euclid_nisp_j_odonnell_ext`, `euclid_nisp_y_true`, `euclid_nisp_y`, `euclid_nisp_y_odonnell_ext`, `euclid_vis_true`, `euclid_vis`, `euclid_vis_odonnell_ext`, `bulge_fraction`, `bulge_nsersic`, `bulge_ellipticity`, `bulge_axis_ratio`, `disk_angle`, `step`<br />
FROM jcarrete.flagship_mock_1_10_7_highz_pq;<br />
<br />
ANALYZE TABLE cosmohub.flagship_mock_1_10_7_highz_c COMPUTE STATISTICS FOR COLUMNS;<br />
<br />
Añado el lensing:<br />
<br />
CREATE TABLE cosmohub.flagship_mock_1_10_7_highz_s (<br />
`unique_id` BIGINT COMMENT 'unique identifier',<br />
`id` INT COMMENT 'ID',<br />
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',<br />
`z` float COMMENT 'true galaxy redshift',<br />
`ra` FLOAT COMMENT 'galaxy right ascension (degrees)',<br />
`dec` FLOAT COMMENT 'galaxy declination (degrees)',<br />
`ra_mag` float COMMENT 'galaxy magnified right ascension (degree)', <br />
`dec_mag` float COMMENT 'galaxy magnified declination (degree)', <br />
`kappa` float COMMENT 'convergence', <br />
`gamma1` float COMMENT 'shear', <br />
`gamma2` float COMMENT 'shear',<br />
`defl1` float COMMENT 'deflection1',<br />
`defl2` float COMMENT 'deflection2',<br />
`l` FLOAT COMMENT 'galactic longitude (degrees)',<br />
`b` FLOAT COMMENT 'galactic latitude (degrees)',<br />
`mw_extinction` float COMMENT 'EBV from file: HFI_CompMap_ThermalDustModel_2048_R1.20.fits',<br />
`abs_muv` FLOAT COMMENT 'absolute magnitude at 1500 angstroms',<br />
`sedname` STRING COMMENT 'Spectral Energy Distribution name (among 76, 38 without UV lines, 38 with UV lines, corresponding to a given UV slope)',<br />
`sedname_int` BIGINT COMMENT 'sedname as integer (from 0 to 75, from 0 to 37 without UV lines, from 38 to 75 with UV lines)',<br />
`rhalf_kpc` FLOAT COMMENT 'half light radius (in physical kpc)',<br />
`rhalf_arcsec` FLOAT COMMENT 'half light radius (in arcsec)',<br />
`muv` FLOAT COMMENT 'for information only, observed magnitude at rest-frame 1500',<br />
`beta` FLOAT COMMENT 'for information only, used to assign an SED',<br />
`hpix_9_nest` int COMMENT 'healpix pixel id using order = 9 in NESTED format',<br />
`hpix_13_nest` int COMMENT 'healpix pixel id using order = 13 in NESTED format',<br />
`hpix_29_nest` bigint COMMENT 'healpix pixel id using order = 29 in NESTED format',<br />
`blanco_decam_g_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in DES g-band (Euclid provided)', <br />
`blanco_decam_g` float COMMENT 'Interpolated observed flux in DES g-band (Euclid provided)',<br />
`blanco_decam_g_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in DES g-band (Euclid provided)',<br />
`blanco_decam_i_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in DES i-band (Euclid provided)', <br />
`blanco_decam_i` float COMMENT 'Interpolated observed flux in DES i-band (Euclid provided)', <br />
`blanco_decam_i_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in DES i-band (Euclid provided)', <br />
`blanco_decam_r_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in DES r-band (Euclid provided)', <br />
`blanco_decam_r` float COMMENT 'Interpolated observed flux in DES r-band (Euclid provided)', <br />
`blanco_decam_r_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in DES r-band (Euclid provided)', <br />
`blanco_decam_z_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in DES z-band (Euclid provided)', <br />
`blanco_decam_z` float COMMENT 'Interpolated observed flux in DES z-band (Euclid provided)',<br />
`blanco_decam_z_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in DES z-band (Euclid provided)', <br />
`subaru_hsc_z_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',<br />
`subaru_hsc_z` float COMMENT 'Interpolated observed flux in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',<br />
`subaru_hsc_z_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',<br />
`jst_jpcam_g_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in JST g-band (Euclid provided)', <br />
`jst_jpcam_g` float COMMENT 'Interpolated observed flux in JST g-band (Euclid provided)',<br />
`jst_jpcam_g_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in JST g-band (Euclid provided)',<br />
`kids_g_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in KIDS g-band (Euclid provided)', <br />
`kids_g` float COMMENT 'Interpolated observed flux in KIDS g-band (Euclid provided)',<br />
`kids_g_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in KIDS g-band (Euclid provided)',<br />
`kids_i_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in KIDS i-band (Euclid provided)', <br />
`kids_i` float COMMENT 'Interpolated observed flux in KIDS i-band (Euclid provided)',<br />
`kids_i_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in KIDS i-band (Euclid provided)', <br />
`kids_r_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in KIDS r-band (Euclid provided)', <br />
`kids_r` float COMMENT 'Interpolated observed flux in KIDS r-band (Euclid provided)',<br />
`kids_r_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in KIDS r-band (Euclid provided)',<br />
`kids_u_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in KIDS u-band (Euclid provided)', <br />
`kids_u` float COMMENT 'Interpolated observed flux in KIDS u-band (Euclid provided)',<br />
`kids_u_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in KIDS u-band (Euclid provided)',<br />
`lsst_g_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST g-band (Euclid provided)', <br />
`lsst_g` float COMMENT 'Interpolated observed flux in LSST g-band (Euclid provided)',<br />
`lsst_g_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST g-band (Euclid provided)',<br />
`lsst_i_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST i-band (Euclid provided)', <br />
`lsst_i` float COMMENT 'Interpolated observed flux in LSST i-band (Euclid provided)',<br />
`lsst_i_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST i-band (Euclid provided)',<br />
`lsst_r_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST r-band (Euclid provided)', <br />
`lsst_r` float COMMENT 'Interpolated observed flux in LSST r-band (Euclid provided)',<br />
`lsst_r_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST r-band (Euclid provided)',<br />
`lsst_u_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST u-band (Euclid provided)', <br />
`lsst_u` float COMMENT 'Interpolated observed flux in LSST u-band (Euclid provided)',<br />
`lsst_u_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST u-band (Euclid provided)',<br />
`lsst_y_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST y-band (Euclid provided)', <br />
`lsst_y` float COMMENT 'Interpolated observed flux in LSST y-band (Euclid provided)',<br />
`lsst_y_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST y-band (Euclid provided)',<br />
`lsst_z_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in LSST z-band (Euclid provided)', <br />
`lsst_z` float COMMENT 'Interpolated observed flux in LSST z-band (Euclid provided)',<br />
`lsst_z_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in LSST z-band (Euclid provided)',<br />
`cfht_megacam_r_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in CFHT r-band (Euclid provided)', <br />
`cfht_megacam_r` float COMMENT 'Interpolated observed flux in CFHT r-band (Euclid provided)',<br />
`cfht_megacam_r_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in CFHT r-band (Euclid provided)',<br />
`cfht_megacam_u_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in CFHT u-band (Euclid provided)', <br />
`cfht_megacam_u` float COMMENT 'Interpolated observed flux in CFHT u-band (Euclid provided)',<br />
`cfht_megacam_u_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in CFHT u-band (Euclid provided)',<br />
`pan_starrs_i_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Pan-STARRS i-band (Euclid provided)', <br />
`pan_starrs_i` float COMMENT 'Interpolated observed flux in Pan-STARRS i-band (Euclid provided)',<br />
`pan_starrs_i_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Pan-STARRS i-band (Euclid provided)',<br />
`pan_starrs_z_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Pan-STARRS z-band (Euclid provided)', <br />
`pan_starrs_z` float COMMENT 'Interpolated observed flux in Pan-STARRS z-band (Euclid provided)',<br />
`pan_starrs_z_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Pan-STARRS z-band (Euclid provided)',<br />
`2mass_h_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in 2MASS H-band (Euclid provided)', <br />
`2mass_h` float COMMENT 'Interpolated observed flux in 2MASS H-band (Euclid provided)',<br />
`2mass_h_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in 2MASS H-band (Euclid provided)',<br />
`2mass_j_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in 2MASS J-band (Euclid provided)', <br />
`2mass_j` float COMMENT 'Interpolated observed flux in 2MASS J-band (Euclid provided)',<br />
`2mass_j_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in 2MASS J-band (Euclid provided)',<br />
`2mass_ks_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in 2MASS Ks-band (Euclid provided)', <br />
`2mass_ks` float COMMENT 'Interpolated observed flux in 2MASS Ks-band (Euclid provided)',<br />
`2mass_ks_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in 2MASS Ks-band (Euclid provided)',<br />
`sdss_r01_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in sdss r-band redshifted to z=0.1 (Euclid provided)', <br />
`sdss_r01` float COMMENT 'Interpolated observed flux in sdss r-band redshifted to z=0.1 (Euclid provided)',<br />
`sdss_r01_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in sdss r-band redshifted to z=0.1 (Euclid provided)',<br />
`gaia_bp_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Gaia-BP band (Euclid provided)', <br />
`gaia_bp` float COMMENT 'Interpolated observed flux in Gaia-BP band (Euclid provided)',<br />
`gaia_bp_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Gaia-BP band (Euclid provided)',<br />
`gaia_g_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Gaia-G band (Euclid provided)', <br />
`gaia_g` float COMMENT 'Interpolated observed flux in Gaia-G band (Euclid provided)',<br />
`gaia_g_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Gaia-G band (Euclid provided)', <br />
`gaia_rp_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Gaia-RP band (Euclid provided)', <br />
`gaia_rp` float COMMENT 'Interpolated observed flux in Gaia-RP band (Euclid provided)',<br />
`gaia_rp_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Gaia-RP band (Euclid provided)',<br />
`euclid_nisp_h_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Euclid NISP-H band (Euclid provided)', <br />
`euclid_nisp_h` float COMMENT 'Interpolated observed flux in Euclid NISP-H band (Euclid provided)',<br />
`euclid_nisp_h_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Euclid NISP-H band (Euclid provided)', <br />
`euclid_nisp_j_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Euclid NISP-J band (Euclid provided)', <br />
`euclid_nisp_j` float COMMENT 'Interpolated observed flux in Euclid NISP-J band (Euclid provided)',<br />
`euclid_nisp_j_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Euclid NISP-J band (Euclid provided)',<br />
`euclid_nisp_y_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Euclid NISP-Y band (Euclid provided)', <br />
`euclid_nisp_y` float COMMENT 'Interpolated observed flux in Euclid NISP-Y band (Euclid provided)',<br />
`euclid_nisp_y_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Euclid NISP-Y band (Euclid provided)', <br />
`euclid_vis_true` float COMMENT 'Integrated observed flux including (properly) MW extinction in Euclid VIS band (Euclid provided)', <br />
`euclid_vis` float COMMENT 'Interpolated observed flux in Euclid VIS band (Euclid provided)',<br />
`euclid_vis_odonnell_ext` float COMMENT 'Interpolated observed flux including MW extinction in Euclid VIS band (Euclid provided)',<br />
`bulge_fraction` FLOAT COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)',<br />
`bulge_nsersic` FLOAT COMMENT 'Sersic index of the bulge component',<br />
`bulge_ellipticity` FLOAT COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)', <br />
`bulge_axis_ratio` FLOAT COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`disk_angle` FLOAT COMMENT 'position of the disk rotation axis (degrees). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',<br />
`step` smallint COMMENT 'redshift step number'<br />
)<br />
CLUSTERED BY (<br />
step,<br />
hpix_13_nest)<br />
SORTED BY (<br />
step ASC,<br />
hpix_13_nest ASC)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
SET hive.auto.convert.join.noconditionaltask.size=1000;<br />
<br />
INSERT OVERWRITE TABLE cosmohub.flagship_mock_1_10_7_highz_s<br />
SELECT <br />
`unique_id`, `id`, `kind`, `z`, `ra`, `dec`, mag_pos.ra_mag, mag_pos.dec_mag, kappa, gamma1, gamma2, defl1, defl2, `l`, `b`, `mw_extinction`, `abs_muv`, `sedname`, `sedname_int`, `rhalf_kpc`, `rhalf_arcsec`, `muv`, `beta`, `hpix_9_nest`, `hpix_13_nest`, `hpix_29_nest`, `blanco_decam_g_true`, `blanco_decam_g`, `blanco_decam_g_odonnell_ext`, `blanco_decam_i_true`, `blanco_decam_i`, `blanco_decam_i_odonnell_ext`, `blanco_decam_r_true`, `blanco_decam_r`, `blanco_decam_r_odonnell_ext`, `blanco_decam_z_true`, `blanco_decam_z`, `blanco_decam_z_odonnell_ext`, `subaru_hsc_z_true`, `subaru_hsc_z`, `subaru_hsc_z_odonnell_ext`, `jst_jpcam_g_true`, `jst_jpcam_g`, `jst_jpcam_g_odonnell_ext`, `kids_g_true`, `kids_g`, `kids_g_odonnell_ext`, `kids_i_true`, `kids_i`, `kids_i_odonnell_ext`, `kids_r_true`, `kids_r`, `kids_r_odonnell_ext`, `kids_u_true`, `kids_u`, `kids_u_odonnell_ext`, `lsst_g_true`, `lsst_g`, `lsst_g_odonnell_ext`, `lsst_i_true`, `lsst_i`, `lsst_i_odonnell_ext`, `lsst_r_true`, `lsst_r`, `lsst_r_odonnell_ext`, `lsst_u_true`, `lsst_u`, `lsst_u_odonnell_ext`, `lsst_y_true`, `lsst_y`, `lsst_y_odonnell_ext`, `lsst_z_true`, `lsst_z`, `lsst_z_odonnell_ext`, `cfht_megacam_r_true`, `cfht_megacam_r`, `cfht_megacam_r_odonnell_ext`, `cfht_megacam_u_true`, `cfht_megacam_u`, `cfht_megacam_u_odonnell_ext`, `pan_starrs_i_true`, `pan_starrs_i`, `pan_starrs_i_odonnell_ext`, `pan_starrs_z_true`, `pan_starrs_z`, `pan_starrs_z_odonnell_ext`, `2mass_h_true`, `2mass_h`, `2mass_h_odonnell_ext`, `2mass_j_true`, `2mass_j`, `2mass_j_odonnell_ext`, `2mass_ks_true`, `2mass_ks`, `2mass_ks_odonnell_ext`, `sdss_r01_true`, `sdss_r01`, `sdss_r01_odonnell_ext`, `gaia_bp_true`, `gaia_bp`, `gaia_bp_odonnell_ext`, `gaia_g_true`, `gaia_g`, `gaia_g_odonnell_ext`, `gaia_rp_true`, `gaia_rp`, `gaia_rp_odonnell_ext`, `euclid_nisp_h_true`, `euclid_nisp_h`, `euclid_nisp_h_odonnell_ext`, `euclid_nisp_j_true`, `euclid_nisp_j`, `euclid_nisp_j_odonnell_ext`, `euclid_nisp_y_true`, `euclid_nisp_y`, `euclid_nisp_y_odonnell_ext`, `euclid_vis_true`, `euclid_vis`, `euclid_vis_odonnell_ext`, `bulge_fraction`, `bulge_nsersic`, `bulge_ellipticity`, `bulge_axis_ratio`, `disk_angle`, `step`<br />
FROM<br />
(SELECT <br />
`unique_id`, `id`, `kind`, `z`, `ra`, `dec`, udf.magnified_positions(CAST(ra AS DOUBLE), CAST(`dec` AS DOUBLE), CAST(l.defl1 AS DOUBLE), CAST(l.defl2 AS DOUBLE)) as mag_pos, l.kappa, l.gamma1, l.gamma2, l.defl1, l.defl2, `l`, `b`, `mw_extinction`, `abs_muv`, `sedname`, `sedname_int`, `rhalf_kpc`, `rhalf_arcsec`, `muv`, `beta`, `hpix_9_nest`, l.hpix_13_nest, `hpix_29_nest`, `blanco_decam_g_true`, `blanco_decam_g`, `blanco_decam_g_odonnell_ext`, `blanco_decam_i_true`, `blanco_decam_i`, `blanco_decam_i_odonnell_ext`, `blanco_decam_r_true`, `blanco_decam_r`, `blanco_decam_r_odonnell_ext`, `blanco_decam_z_true`, `blanco_decam_z`, `blanco_decam_z_odonnell_ext`, `subaru_hsc_z_true`, `subaru_hsc_z`, `subaru_hsc_z_odonnell_ext`, `jst_jpcam_g_true`, `jst_jpcam_g`, `jst_jpcam_g_odonnell_ext`, `kids_g_true`, `kids_g`, `kids_g_odonnell_ext`, `kids_i_true`, `kids_i`, `kids_i_odonnell_ext`, `kids_r_true`, `kids_r`, `kids_r_odonnell_ext`, `kids_u_true`, `kids_u`, `kids_u_odonnell_ext`, `lsst_g_true`, `lsst_g`, `lsst_g_odonnell_ext`, `lsst_i_true`, `lsst_i`, `lsst_i_odonnell_ext`, `lsst_r_true`, `lsst_r`, `lsst_r_odonnell_ext`, `lsst_u_true`, `lsst_u`, `lsst_u_odonnell_ext`, `lsst_y_true`, `lsst_y`, `lsst_y_odonnell_ext`, `lsst_z_true`, `lsst_z`, `lsst_z_odonnell_ext`, `cfht_megacam_r_true`, `cfht_megacam_r`, `cfht_megacam_r_odonnell_ext`, `cfht_megacam_u_true`, `cfht_megacam_u`, `cfht_megacam_u_odonnell_ext`, `pan_starrs_i_true`, `pan_starrs_i`, `pan_starrs_i_odonnell_ext`, `pan_starrs_z_true`, `pan_starrs_z`, `pan_starrs_z_odonnell_ext`, `2mass_h_true`, `2mass_h`, `2mass_h_odonnell_ext`, `2mass_j_true`, `2mass_j`, `2mass_j_odonnell_ext`, `2mass_ks_true`, `2mass_ks`, `2mass_ks_odonnell_ext`, `sdss_r01_true`, `sdss_r01`, `sdss_r01_odonnell_ext`, `gaia_bp_true`, `gaia_bp`, `gaia_bp_odonnell_ext`, `gaia_g_true`, `gaia_g`, `gaia_g_odonnell_ext`, `gaia_rp_true`, `gaia_rp`, `gaia_rp_odonnell_ext`, `euclid_nisp_h_true`, `euclid_nisp_h`, `euclid_nisp_h_odonnell_ext`, `euclid_nisp_j_true`, `euclid_nisp_j`, `euclid_nisp_j_odonnell_ext`, `euclid_nisp_y_true`, `euclid_nisp_y`, `euclid_nisp_y_odonnell_ext`, `euclid_vis_true`, `euclid_vis`, `euclid_vis_odonnell_ext`, `bulge_fraction`, `bulge_nsersic`, `bulge_ellipticity`, `bulge_axis_ratio`, `disk_angle`, l.step<br />
FROM cosmohub.flagship_mock_1_10_7_highz_c as m<br />
JOIN cosmohub.flagship_lensing2 as l<br />
ON m.step = l.step<br />
AND m.hpix_13_nest = l.hpix_13_nest) as t;<br />
<br />
ANALYZE TABLE cosmohub.flagship_mock_1_10_7_highz_s COMPUTE STATISTICS FOR COLUMNS;</div>
Tallada
https://pwiki.pic.es/index.php?title=Gaia_EDR3&diff=885
Gaia EDR3
2020-12-03T17:40:35Z
<p>Tallada: </p>
<hr />
<div>== Raw data ==<br />
<br />
Mirrored to cephfs using lftp:<br />
<br />
mirror -r -v -c -P4 http://cdn.gea.esac.esa.int/Gaia/gedr3/gaia_source/ /cephfs/pic.es/astro/scratch/tallada/gaia_edr3<br />
<br />
== External table ==<br />
<br />
CREATE EXTERNAL TABLE tallada.gaia_edr3_csv (<br />
solution_id bigint,<br />
designation string,<br />
source_id bigint,<br />
random_index bigint,<br />
ref_epoch float,<br />
ra double,<br />
ra_error float,<br />
`dec` double,<br />
dec_error float,<br />
parallax double,<br />
parallax_error float,<br />
parallax_over_error float,<br />
pm float,<br />
pmra double,<br />
pmra_error float,<br />
pmdec double,<br />
pmdec_error float,<br />
ra_dec_corr float,<br />
ra_parallax_corr float,<br />
ra_pmra_corr float,<br />
ra_pmdec_corr float,<br />
dec_parallax_corr float,<br />
dec_pmra_corr float,<br />
dec_pmdec_corr float,<br />
parallax_pmra_corr float,<br />
parallax_pmdec_corr float,<br />
pmra_pmdec_corr float,<br />
astrometric_n_obs_al smallint,<br />
astrometric_n_obs_ac smallint,<br />
astrometric_n_good_obs_al smallint,<br />
astrometric_n_bad_obs_al smallint,<br />
astrometric_gof_al float,<br />
astrometric_chi2_al float,<br />
astrometric_excess_noise float,<br />
astrometric_excess_noise_sig float,<br />
astrometric_params_solved tinyint,<br />
astrometric_primary_flag boolean,<br />
nu_eff_used_in_astrometry float,<br />
pseudocolour float,<br />
pseudocolour_error float,<br />
ra_pseudocolour_corr float,<br />
dec_pseudocolour_corr float,<br />
parallax_pseudocolour_corr float,<br />
pmra_pseudocolour_corr float,<br />
pmdec_pseudocolour_corr float,<br />
astrometric_matched_transits smallint,<br />
visibility_periods_used smallint,<br />
astrometric_sigma5d_max float,<br />
matched_transits smallint,<br />
new_matched_transits smallint,<br />
matched_transits_removed smallint,<br />
ipd_gof_harmonic_amplitude float,<br />
ipd_gof_harmonic_phase float,<br />
ipd_frac_multi_peak tinyint,<br />
ipd_frac_odd_win tinyint,<br />
ruwe float,<br />
scan_direction_strength_k1 float,<br />
scan_direction_strength_k2 float,<br />
scan_direction_strength_k3 float,<br />
scan_direction_strength_k4 float,<br />
scan_direction_mean_k1 float,<br />
scan_direction_mean_k2 float,<br />
scan_direction_mean_k3 float,<br />
scan_direction_mean_k4 float,<br />
duplicated_source boolean,<br />
phot_g_n_obs smallint,<br />
phot_g_mean_flux double,<br />
phot_g_mean_flux_error float,<br />
phot_g_mean_flux_over_error float,<br />
phot_g_mean_mag float,<br />
phot_bp_n_obs smallint,<br />
phot_bp_mean_flux double,<br />
phot_bp_mean_flux_error float,<br />
phot_bp_mean_flux_over_error float,<br />
phot_bp_mean_mag float,<br />
phot_rp_n_obs smallint,<br />
phot_rp_mean_flux double,<br />
phot_rp_mean_flux_error float,<br />
phot_rp_mean_flux_over_error float,<br />
phot_rp_mean_mag float,<br />
phot_bp_n_contaminated_transits smallint,<br />
phot_bp_n_blended_transits smallint,<br />
phot_rp_n_contaminated_transits smallint,<br />
phot_rp_n_blended_transits smallint,<br />
phot_proc_mode tinyint,<br />
phot_bp_rp_excess_factor float,<br />
bp_rp float,<br />
bp_g float,<br />
g_rp float,<br />
dr2_radial_velocity float,<br />
dr2_radial_velocity_error float,<br />
dr2_rv_nb_transits smallint,<br />
dr2_rv_template_teff float,<br />
dr2_rv_template_logg float,<br />
dr2_rv_template_fe_h float,<br />
l double,<br />
b double,<br />
ecl_lon double,<br />
ecl_lat double<br />
)<br />
ROW FORMAT DELIMITED<br />
FIELDS TERMINATED BY ','<br />
LOCATION '/user/tallada/data/gaia_edr3'<br />
;<br />
<br />
<br />
== Internal table ==<br />
<br />
CREATE TABLE cosmohub.gaia_edr3 (<br />
solution_id bigint,<br />
designation string,<br />
source_id bigint,<br />
random_index bigint,<br />
ref_epoch float,<br />
ra double,<br />
ra_error float,<br />
`dec` double,<br />
dec_error float,<br />
parallax double,<br />
parallax_error float,<br />
parallax_over_error float,<br />
pm float,<br />
pmra double,<br />
pmra_error float,<br />
pmdec double,<br />
pmdec_error float,<br />
ra_dec_corr float,<br />
ra_parallax_corr float,<br />
ra_pmra_corr float,<br />
ra_pmdec_corr float,<br />
dec_parallax_corr float,<br />
dec_pmra_corr float,<br />
dec_pmdec_corr float,<br />
parallax_pmra_corr float,<br />
parallax_pmdec_corr float,<br />
pmra_pmdec_corr float,<br />
astrometric_n_obs_al smallint,<br />
astrometric_n_obs_ac smallint,<br />
astrometric_n_good_obs_al smallint,<br />
astrometric_n_bad_obs_al smallint,<br />
astrometric_gof_al float,<br />
astrometric_chi2_al float,<br />
astrometric_excess_noise float,<br />
astrometric_excess_noise_sig float,<br />
astrometric_params_solved tinyint,<br />
astrometric_primary_flag boolean,<br />
nu_eff_used_in_astrometry float,<br />
pseudocolour float,<br />
pseudocolour_error float,<br />
ra_pseudocolour_corr float,<br />
dec_pseudocolour_corr float,<br />
parallax_pseudocolour_corr float,<br />
pmra_pseudocolour_corr float,<br />
pmdec_pseudocolour_corr float,<br />
astrometric_matched_transits smallint,<br />
visibility_periods_used smallint,<br />
astrometric_sigma5d_max float,<br />
matched_transits smallint,<br />
new_matched_transits smallint,<br />
matched_transits_removed smallint,<br />
ipd_gof_harmonic_amplitude float,<br />
ipd_gof_harmonic_phase float,<br />
ipd_frac_multi_peak tinyint,<br />
ipd_frac_odd_win tinyint,<br />
ruwe float,<br />
scan_direction_strength_k1 float,<br />
scan_direction_strength_k2 float,<br />
scan_direction_strength_k3 float,<br />
scan_direction_strength_k4 float,<br />
scan_direction_mean_k1 float,<br />
scan_direction_mean_k2 float,<br />
scan_direction_mean_k3 float,<br />
scan_direction_mean_k4 float,<br />
duplicated_source boolean,<br />
phot_g_n_obs smallint,<br />
phot_g_mean_flux double,<br />
phot_g_mean_flux_error float,<br />
phot_g_mean_flux_over_error float,<br />
phot_g_mean_mag float,<br />
phot_bp_n_obs smallint,<br />
phot_bp_mean_flux double,<br />
phot_bp_mean_flux_error float,<br />
phot_bp_mean_flux_over_error float,<br />
phot_bp_mean_mag float,<br />
phot_rp_n_obs smallint,<br />
phot_rp_mean_flux double,<br />
phot_rp_mean_flux_error float,<br />
phot_rp_mean_flux_over_error float,<br />
phot_rp_mean_mag float,<br />
phot_bp_n_contaminated_transits smallint,<br />
phot_bp_n_blended_transits smallint,<br />
phot_rp_n_contaminated_transits smallint,<br />
phot_rp_n_blended_transits smallint,<br />
phot_proc_mode tinyint,<br />
phot_bp_rp_excess_factor float,<br />
bp_rp float,<br />
bp_g float,<br />
g_rp float,<br />
dr2_radial_velocity float,<br />
dr2_radial_velocity_error float,<br />
dr2_rv_nb_transits smallint,<br />
dr2_rv_template_teff float,<br />
dr2_rv_template_logg float,<br />
dr2_rv_template_fe_h float,<br />
l double,<br />
b double,<br />
ecl_lon double,<br />
ecl_lat double<br />
)<br />
CLUSTERED BY (source_id)<br />
SORTED BY (source_id ASC)<br />
INTO 2048 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
INSERT INTO cosmohub.gaia_edr3<br />
SELECT * FROM tallada.gaia_edr3_csv<br />
WHERE source_id IS NOT NULL<br />
;<br />
<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE solution_id solution_id bigint COMMENT "Solution Identifier";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE designation designation string COMMENT "Unique source designation, unique across all Data Releases";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE source_id source_id bigint COMMENT "Unique source identifier, unique within a particular Data Release";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE random_index random_index bigint COMMENT "Random index used to select subsets";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ref_epoch ref_epoch float COMMENT "Reference epoch";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ra ra double COMMENT "Right ascension";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ra_error ra_error float COMMENT "Standard error of right ascension";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE `dec` `dec` double COMMENT "Declination";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dec_error dec_error float COMMENT "Standard error of declination";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE parallax parallax double COMMENT "Parallax";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE parallax_error parallax_error float COMMENT "Standard error of parallax";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE parallax_over_error parallax_over_error float COMMENT "Parallax divided by its standard error";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE pm pm float COMMENT "Total proper motion";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE pmra pmra double COMMENT "Proper motion in right ascension direction";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE pmra_error pmra_error float COMMENT "Standard error of proper motion in right ascension direction";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE pmdec pmdec double COMMENT "Proper motion in declination direction";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE pmdec_error pmdec_error float COMMENT "Standard error of proper motion in declination direction";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ra_dec_corr ra_dec_corr float COMMENT "Correlation between right ascension and declination";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ra_parallax_corr ra_parallax_corr float COMMENT "Correlation between right ascension and parallax";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ra_pmra_corr ra_pmra_corr float COMMENT "Correlation between right ascension and proper motion in right ascension";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ra_pmdec_corr ra_pmdec_corr float COMMENT "Correlation between right ascension and proper motion in declination";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dec_parallax_corr dec_parallax_corr float COMMENT "Correlation between declination and parallax";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dec_pmra_corr dec_pmra_corr float COMMENT "Correlation between declination and proper motion in right ascension";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dec_pmdec_corr dec_pmdec_corr float COMMENT "Correlation between declination and proper motion in declination";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE parallax_pmra_corr parallax_pmra_corr float COMMENT "Correlation between parallax and proper motion in right ascension";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE parallax_pmdec_corr parallax_pmdec_corr float COMMENT "Correlation between parallax and proper motion in declination";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE pmra_pmdec_corr pmra_pmdec_corr float COMMENT "Correlation between proper motion in right ascension and proper motion in declination";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_n_obs_al astrometric_n_obs_al smallint COMMENT "Total number of observations AL";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_n_obs_ac astrometric_n_obs_ac smallint COMMENT "Total number of observations AC";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_n_good_obs_al astrometric_n_good_obs_al smallint COMMENT "Number of good observations AL";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_n_bad_obs_al astrometric_n_bad_obs_al smallint COMMENT "Number of bad observations AL";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_gof_al astrometric_gof_al float COMMENT "Goodness of fit statistic of model wrt along-scan observations";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_chi2_al astrometric_chi2_al float COMMENT "AL chi-square value";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_excess_noise astrometric_excess_noise float COMMENT "Excess noise of the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_excess_noise_sig astrometric_excess_noise_sig float COMMENT "Significance of excess noise";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_params_solved astrometric_params_solved tinyint COMMENT "Which parameters have been solved for?";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_primary_flag astrometric_primary_flag boolean COMMENT "Primary or seconday";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE nu_eff_used_in_astrometry nu_eff_used_in_astrometry float COMMENT "Effective wavenumber of the source used in the astrometric solution";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE pseudocolour pseudocolour float COMMENT "Astrometrically estimated pseudocolour of the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE pseudocolour_error pseudocolour_error float COMMENT "Standard error of the pseudocolour of the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ra_pseudocolour_corr ra_pseudocolour_corr float COMMENT "Correlation between right ascension and pseudocolour";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dec_pseudocolour_corr dec_pseudocolour_corr float COMMENT "Correlation between declination and pseudocolour";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE parallax_pseudocolour_corr parallax_pseudocolour_corr float COMMENT "Correlation between parallax and pseudocolour";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE pmra_pseudocolour_corr pmra_pseudocolour_corr float COMMENT "Correlation between proper motion in right asension and pseudocolour";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE pmdec_pseudocolour_corr pmdec_pseudocolour_corr float COMMENT "Correlation between proper motion in declination and pseudocolour";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_matched_transits astrometric_matched_transits smallint COMMENT "Matched FOV transits used in the AGIS solution";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE visibility_periods_used visibility_periods_used smallint COMMENT "Number of visibility periods used in Astrometric solution";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE astrometric_sigma5d_max astrometric_sigma5d_max float COMMENT "The longest semi-major axis of the 5-d error ellipsoid";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE matched_transits matched_transits smallint COMMENT "The number of transits matched to this source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE new_matched_transits new_matched_transits smallint COMMENT "The number of transits newly incorporated into an existing source in the current cycle";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE matched_transits_removed matched_transits_removed smallint COMMENT "The number of transits removed from an existing source in the current cycle";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ipd_gof_harmonic_amplitude ipd_gof_harmonic_amplitude float COMMENT "Amplitude of the IPD GoF versus position angle of scan";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ipd_gof_harmonic_phase ipd_gof_harmonic_phase float COMMENT "Phase of the IPD GoF versus position angle of scan";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ipd_frac_multi_peak ipd_frac_multi_peak tinyint COMMENT "Percent of successful-IPD windows with more than one peak";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ipd_frac_odd_win ipd_frac_odd_win tinyint COMMENT "Percent of transits with truncated windows or multiple gate";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ruwe ruwe float COMMENT "Renormalised unit weight error";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE scan_direction_strength_k1 scan_direction_strength_k1 float COMMENT "Degree of concentration of scan directions across the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE scan_direction_strength_k2 scan_direction_strength_k2 float COMMENT "Degree of concentration of scan directions across the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE scan_direction_strength_k3 scan_direction_strength_k3 float COMMENT "Degree of concentration of scan directions across the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE scan_direction_strength_k4 scan_direction_strength_k4 float COMMENT "Degree of concentration of scan directions across the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE scan_direction_mean_k1 scan_direction_mean_k1 float COMMENT "Mean position angle of scan directions across the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE scan_direction_mean_k2 scan_direction_mean_k2 float COMMENT "Mean position angle of scan directions across the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE scan_direction_mean_k3 scan_direction_mean_k3 float COMMENT "Mean position angle of scan directions across the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE scan_direction_mean_k4 scan_direction_mean_k4 float COMMENT "Mean position angle of scan directions across the source";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE duplicated_source duplicated_source boolean COMMENT "Source with multiple source identifiers";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_g_n_obs phot_g_n_obs smallint COMMENT "Number of observations contributing to G photometry";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_g_mean_flux phot_g_mean_flux double COMMENT "G-band mean flux";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_g_mean_flux_error phot_g_mean_flux_error float COMMENT "Error on G-band mean flux";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_g_mean_flux_over_error phot_g_mean_flux_over_error float COMMENT "G-band mean flux divided by its error";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_g_mean_mag phot_g_mean_mag float COMMENT "G-band mean magnitude";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_bp_n_obs phot_bp_n_obs smallint COMMENT "Number of observations contributing to BP photometry";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_bp_mean_flux phot_bp_mean_flux double COMMENT "Integrated BP mean flux";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_bp_mean_flux_error phot_bp_mean_flux_error float COMMENT "Error on the integrated BP mean flux";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_bp_mean_flux_over_error phot_bp_mean_flux_over_error float COMMENT "Integrated BP mean flux divided by its error";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_bp_mean_mag phot_bp_mean_mag float COMMENT "Integrated BP mean magnitude";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_rp_n_obs phot_rp_n_obs smallint COMMENT "Number of observations contributing to RP photometry";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_rp_mean_flux phot_rp_mean_flux double COMMENT "Integrated RP mean flux";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_rp_mean_flux_error phot_rp_mean_flux_error float COMMENT "Error on the integrated RP mean flux";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_rp_mean_flux_over_error phot_rp_mean_flux_over_error float COMMENT "Integrated RP mean flux divided by its error";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_rp_mean_mag phot_rp_mean_mag float COMMENT "Integrated RP mean magnitude";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_bp_n_contaminated_transits phot_bp_n_contaminated_transits smallint COMMENT "Number of BP contaminated transits";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_bp_n_blended_transits phot_bp_n_blended_transits smallint COMMENT "Number of BP blended transits";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_rp_n_contaminated_transits phot_rp_n_contaminated_transits smallint COMMENT "Number of RP contaminated transits";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_rp_n_blended_transits phot_rp_n_blended_transits smallint COMMENT "Number of RP blended transits";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_proc_mode phot_proc_mode tinyint COMMENT "Photometry processing mode";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE phot_bp_rp_excess_factor phot_bp_rp_excess_factor float COMMENT "BP/RP excess factor";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE bp_rp bp_rp float COMMENT "BP - RP colour";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE bp_g bp_g float COMMENT "BP - G colour";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE g_rp g_rp float COMMENT "G - RP colour";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dr2_radial_velocity dr2_radial_velocity float COMMENT "Radial velocity from Gaia DR2";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dr2_radial_velocity_error dr2_radial_velocity_error float COMMENT "Radial velocity error from Gaia DR2";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dr2_rv_nb_transits dr2_rv_nb_transits smallint COMMENT "Number of transits used to compute radial velocity in Gaia DR2";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dr2_rv_template_teff dr2_rv_template_teff float COMMENT "Teff of the template used to compute radial velocity in Gaia DR2";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dr2_rv_template_logg dr2_rv_template_logg float COMMENT "logg of the template used to compute radial velocity in Gaia DR2";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE dr2_rv_template_fe_h dr2_rv_template_fe_h float COMMENT "Fe/H of the template used to compute radial velocity in Gaia DR2";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE l l double COMMENT "Galactic longitude";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE b b double COMMENT "Galactic latitude";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ecl_lon ecl_lon double COMMENT "Ecliptic longitude";<br />
ALTER TABLE cosmohub.gaia_edr3 CHANGE ecl_lat ecl_lat double COMMENT "Ecliptic latitude";<br />
<br />
ANALYZE TABLE cosmohub.gaia_edr3 COMPUTE STATISTICS FOR COLUMNS;</div>
Tallada
https://pwiki.pic.es/index.php?title=Gaia_EDR3&diff=884
Gaia EDR3
2020-12-03T14:39:10Z
<p>Tallada: Created page with "== Raw data == Mirrored to cephfs using lftp: mirror -r -v -c -P4 http://cdn.gea.esac.esa.int/Gaia/gedr3/gaia_source/ /cephfs/pic.es/astro/scratch/tallada/gaia_edr3 ==..."</p>
<hr />
<div>== Raw data ==<br />
<br />
Mirrored to cephfs using lftp:<br />
<br />
mirror -r -v -c -P4 http://cdn.gea.esac.esa.int/Gaia/gedr3/gaia_source/ /cephfs/pic.es/astro/scratch/tallada/gaia_edr3<br />
<br />
== External table ==<br />
<br />
CREATE EXTERNAL TABLE tallada.gaia_edr3_csv (<br />
solution_id bigint,<br />
designation string,<br />
source_id bigint,<br />
random_index bigint,<br />
ref_epoch float,<br />
ra double,<br />
ra_error float,<br />
`dec` double,<br />
dec_error float,<br />
parallax double,<br />
parallax_error float,<br />
parallax_over_error float,<br />
pm float,<br />
pmra double,<br />
pmra_error float,<br />
pmdec double,<br />
pmdec_error float,<br />
ra_dec_corr float,<br />
ra_parallax_corr float,<br />
ra_pmra_corr float,<br />
ra_pmdec_corr float,<br />
dec_parallax_corr float,<br />
dec_pmra_corr float,<br />
dec_pmdec_corr float,<br />
parallax_pmra_corr float,<br />
parallax_pmdec_corr float,<br />
pmra_pmdec_corr float,<br />
astrometric_n_obs_al smallint,<br />
astrometric_n_obs_ac smallint,<br />
astrometric_n_good_obs_al smallint,<br />
astrometric_n_bad_obs_al smallint,<br />
astrometric_gof_al float,<br />
astrometric_chi2_al float,<br />
astrometric_excess_noise float,<br />
astrometric_excess_noise_sig float,<br />
astrometric_params_solved tinyint,<br />
astrometric_primary_flag boolean,<br />
nu_eff_used_in_astrometry float,<br />
pseudocolour float,<br />
pseudocolour_error float,<br />
ra_pseudocolour_corr float,<br />
dec_pseudocolour_corr float,<br />
parallax_pseudocolour_corr float,<br />
pmra_pseudocolour_corr float,<br />
pmdec_pseudocolour_corr float,<br />
astrometric_matched_transits smallint,<br />
visibility_periods_used smallint,<br />
astrometric_sigma5d_max float,<br />
matched_transits smallint,<br />
new_matched_transits smallint,<br />
matched_transits_removed smallint,<br />
ipd_gof_harmonic_amplitude float,<br />
ipd_gof_harmonic_phase float,<br />
ipd_frac_multi_peak tinyint,<br />
ipd_frac_odd_win tinyint,<br />
ruwe float,<br />
scan_direction_strength_k1 float,<br />
scan_direction_strength_k2 float,<br />
scan_direction_strength_k3 float,<br />
scan_direction_strength_k4 float,<br />
scan_direction_mean_k1 float,<br />
scan_direction_mean_k2 float,<br />
scan_direction_mean_k3 float,<br />
scan_direction_mean_k4 float,<br />
duplicated_source boolean,<br />
phot_g_n_obs smallint,<br />
phot_g_mean_flux double,<br />
phot_g_mean_flux_error float,<br />
phot_g_mean_flux_over_error float,<br />
phot_g_mean_mag float,<br />
phot_bp_n_obs smallint,<br />
phot_bp_mean_flux double,<br />
phot_bp_mean_flux_error float,<br />
phot_bp_mean_flux_over_error float,<br />
phot_bp_mean_mag float,<br />
phot_rp_n_obs smallint,<br />
phot_rp_mean_flux double,<br />
phot_rp_mean_flux_error float,<br />
phot_rp_mean_flux_over_error float,<br />
phot_rp_mean_mag float,<br />
phot_bp_n_contaminated_transits smallint,<br />
phot_bp_n_blended_transits smallint,<br />
phot_rp_n_contaminated_transits smallint,<br />
phot_rp_n_blended_transits smallint,<br />
phot_proc_mode tinyint,<br />
phot_bp_rp_excess_factor float,<br />
bp_rp float,<br />
bp_g float,<br />
g_rp float,<br />
dr2_radial_velocity float,<br />
dr2_radial_velocity_error float,<br />
dr2_rv_nb_transits smallint,<br />
dr2_rv_template_teff float,<br />
dr2_rv_template_logg float,<br />
dr2_rv_template_fe_h float,<br />
l double,<br />
b double,<br />
ecl_lon double,<br />
ecl_lat double<br />
)<br />
ROW FORMAT DELIMITED<br />
FIELDS TERMINATED BY ','<br />
LOCATION '/user/tallada/data/gaia_edr3'<br />
;</div>
Tallada
https://pwiki.pic.es/index.php?title=Catalogs&diff=883
Catalogs
2020-12-03T14:23:14Z
<p>Tallada: /* External */</p>
<hr />
<div>== Euclid productions ==<br />
<br />
==== Mock Galaxy catalogs ====<br />
<br />
More info from Euclid wiki (Francesc):<br />
<br />
https://euclid.roe.ac.uk/projects/sgv/wiki/SC456_#Galaxies<br />
<br />
* [[ Euclid_mock_production_v1.6.20 | v1.6.20 ]] ([https://euclid.roe.ac.uk/issues/9153 #9153])<br />
* [[ Euclid_mock_production_v1.7.17 | v1.7.17 ]] (SC456)<br />
* [[ Euclid_mock_production_v1.8.1 | v1.8.1 ]]<br />
* [[ Euclid_mock_production_v1.8.3 | v1.8.3 ]] (single octant for Helsinki)<br />
* [[ Euclid_mock_production_v1.8.3_c | v1.8.3_c ]] (clustered table without lensing)<br />
* [[ Euclid_mock_production_v1.8.4_s | v1.8.4_s ]] (from 1.8.3 with correct lensing and without buffer)<br />
* [[ Euclid_mock_production_v1.8.5 | v1.8.5 ]] (Kai subset with additional magnitudes for IA)<br />
* [[ Euclid_mock_production_v1.7.17_correct | v1.7.17_correct (v1.7.18)]] (SC456)<br />
* [[ Euclid_mock_production_v1.7.17_deep_correct | v1.7.17_deep_correct]] (SC456)<br />
* [[ Euclid_mock_production_v1.7.17_correct_v2 | v1.7.17_correct_v2 (v1.7.19)]] (SC456)<br />
* [[ Euclid_mock_production_v1.7.17_deep_correct_v2 | v1.7.17_deep_correct_v2]] (SC456)<br />
* [[ Euclid_mock_production_v1.9.2 | v1.9.2]] (SC8)<br />
* [[ Euclid_mock_production_v1.9.3 | v1.9.3]] (SC8)<br />
* [[ Euclid_mock_production_v1.9.4 | v1.9.4]] (SC8)<br />
* [[ Euclid_mock_production_v1.9.5 | v1.9.5]] (SC8)<br />
* [[ Euclid_mock_production_v1.9.6 | v1.9.6]] (SC8)<br />
* [[ Euclid_mock_production_v1.9.7 | v1.9.7]] (SC8)<br />
<br />
==== Zurich Dark Matter Halo catalogs ====<br />
<br />
* [[ v1.0 | v1.0 ]]<br />
<br />
* [[ v1.1 | v1.1 Corrected Velocities ]]<br />
<br />
==== Primeval Universe Catalogs ====<br />
* [[ QSOs | QSOs]]<br />
* [[ High_redshift_galaxies | High-z galaxies]]<br />
<br />
== MICE productions ==<br />
<br />
* [[ MIDES Wide Field v5.0 | MIDES Wide Field v5.0 ]]<br />
* [[ MIDES Wide Field v6.0 | MIDES Wide Field v6.0 ]]<br />
* [[ MICE Redmagic v5.1 ]]<br />
* [[ DESY3 MICE2 source catalog | DESY3 MICE2 source catalog ]]<br />
<br />
== PAUS productions ==<br />
<br />
=== Observations ===<br />
<br />
* [[ PAUdm_forced_aperture_coadd | PAUdm forced aperture coadd ]]<br />
<br />
==== Photoz productions ====<br />
<br />
* [[ photoz_bcnz | bcnz_v29.parquet ]]<br />
<br />
* [[ production_table | paudm_production_table ]]<br />
<br />
* [[ PAUS+COSMOS_photoz | PAUS+COSMOS_photoz ]]<br />
<br />
==== CIGALE catalogs ====<br />
<br />
* [[ CIGALE-W3 | CIGALE W3 ]]<br />
<br />
==== Mock Galaxy catalogs ====<br />
<br />
* [[ Flagship-PAU | Flagship-PAU v1.8.4_v0.1 ]]<br />
<br />
* [[ Flagship-PAU_v02 | Flagship-PAU v1.8.4_v0.2 ]]<br />
<br />
* [[ Flagship-PAU_v03 | Flagship-PAU v1.8.4_v0.3 ]]<br />
<br />
* [[ Durham-LC-PAU lc5.field1.63 | Durham LC lc5.field1.63 ]]<br />
<br />
== External ==<br />
<br />
* [[ Cosmos_2015_DR_2.1 | COSMOS 2015 DR 2.1 ]]<br />
* [[ ZEST_v1.0 | Zest v1.0 ]]<br />
* [[ CANDELS_2018 | CANDELS 2018 ]]<br />
* [[ Legacy Survey DR8 | Legacy Survey DR8 ]]<br />
* [[ Legacy Survey DR9 | Legacy Survey DR9 ]]<br />
* [[ KiDS DR4 ]]<br />
* [[ VIPERS ]]<br />
* [[ The COSMOS ACS Catalog ]]<br />
* [[ GLADE ]]<br />
* [[ Gaia_EDR3 | Gaia EDR3 ]]</div>
Tallada
https://pwiki.pic.es/index.php?title=Catalogs&diff=879
Catalogs
2020-12-01T20:05:04Z
<p>Tallada: /* External */</p>
<hr />
<div>== Euclid productions ==<br />
<br />
==== Mock Galaxy catalogs ====<br />
<br />
More info from Euclid wiki (Francesc):<br />
<br />
https://euclid.roe.ac.uk/projects/sgv/wiki/SC456_#Galaxies<br />
<br />
* [[ Euclid_mock_production_v1.6.20 | v1.6.20 ]] ([https://euclid.roe.ac.uk/issues/9153 #9153])<br />
* [[ Euclid_mock_production_v1.7.17 | v1.7.17 ]] (SC456)<br />
* [[ Euclid_mock_production_v1.8.1 | v1.8.1 ]]<br />
* [[ Euclid_mock_production_v1.8.3 | v1.8.3 ]] (single octant for Helsinki)<br />
* [[ Euclid_mock_production_v1.8.3_c | v1.8.3_c ]] (clustered table without lensing)<br />
* [[ Euclid_mock_production_v1.8.4_s | v1.8.4_s ]] (from 1.8.3 with correct lensing and without buffer)<br />
* [[ Euclid_mock_production_v1.8.5 | v1.8.5 ]] (Kai subset with additional magnitudes for IA)<br />
* [[ Euclid_mock_production_v1.7.17_correct | v1.7.17_correct (v1.7.18)]] (SC456)<br />
* [[ Euclid_mock_production_v1.7.17_deep_correct | v1.7.17_deep_correct]] (SC456)<br />
* [[ Euclid_mock_production_v1.7.17_correct_v2 | v1.7.17_correct_v2 (v1.7.19)]] (SC456)<br />
* [[ Euclid_mock_production_v1.7.17_deep_correct_v2 | v1.7.17_deep_correct_v2]] (SC456)<br />
* [[ Euclid_mock_production_v1.9.2 | v1.9.2]] (SC8)<br />
* [[ Euclid_mock_production_v1.9.3 | v1.9.3]] (SC8)<br />
* [[ Euclid_mock_production_v1.9.4 | v1.9.4]] (SC8)<br />
* [[ Euclid_mock_production_v1.9.5 | v1.9.5]] (SC8)<br />
* [[ Euclid_mock_production_v1.9.6 | v1.9.6]] (SC8)<br />
* [[ Euclid_mock_production_v1.9.7 | v1.9.7]] (SC8)<br />
<br />
==== Zurich Dark Matter Halo catalogs ====<br />
<br />
* [[ v1.0 | v1.0 ]]<br />
<br />
* [[ v1.1 | v1.1 Corrected Velocities ]]<br />
<br />
==== Primeval Universe Catalogs ====<br />
* [[ QSOs | QSOs]]<br />
* [[ High_redshift_galaxies | High-z galaxies]]<br />
<br />
== MICE productions ==<br />
<br />
* [[ MIDES Wide Field v5.0 | MIDES Wide Field v5.0 ]]<br />
* [[ MIDES Wide Field v6.0 | MIDES Wide Field v6.0 ]]<br />
* [[ MICE Redmagic v5.1 ]]<br />
* [[ DESY3 MICE2 source catalog | DESY3 MICE2 source catalog ]]<br />
<br />
== PAUS productions ==<br />
<br />
=== Observations ===<br />
<br />
* [[ PAUdm_forced_aperture_coadd | PAUdm forced aperture coadd ]]<br />
<br />
==== Photoz productions ====<br />
<br />
* [[ photoz_bcnz | bcnz_v29.parquet ]]<br />
<br />
* [[ production_table | paudm_production_table ]]<br />
<br />
* [[ PAUS+COSMOS_photoz | PAUS+COSMOS_photoz ]]<br />
<br />
=== Mock Galaxy catalogs ===<br />
<br />
* [[ Flagship-PAU | Flagship-PAU v1.8.4_v0.1 ]]<br />
<br />
* [[ Flagship-PAU_v02 | Flagship-PAU v1.8.4_v0.2 ]]<br />
<br />
* [[ Flagship-PAU_v03 | Flagship-PAU v1.8.4_v0.3 ]]<br />
<br />
* [[ Durham-LC-PAU lc5.field1.63 | Durham LC lc5.field1.63 ]]<br />
<br />
== External ==<br />
<br />
* [[ Cosmos_2015_DR_2.1 | COSMOS 2015 DR 2.1 ]]<br />
* [[ ZEST_v1.0 | Zest v1.0 ]]<br />
* [[ CANDELS_2018 | CANDELS 2018 ]]<br />
* [[ Legacy Survey DR8 | Legacy Survey DR8 ]]<br />
* [[ Legacy Survey DR9 | Legacy Survey DR9 ]]<br />
* [[ KiDS DR4 ]]<br />
* [[ VIPERS ]]<br />
* [[ The COSMOS ACS Catalog ]]<br />
* [[ GLADE ]]</div>
Tallada
https://pwiki.pic.es/index.php?title=Legacy_Survey_DR9&diff=878
Legacy Survey DR9
2020-11-30T21:04:59Z
<p>Tallada: </p>
<hr />
<div>== Original DR9 ==<br />
=== Raw data ===<br />
<br />
Downloaded from NERSC<br />
/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m<br />
<br />
Directories:<br />
south/tractor<br />
north/tractor<br />
<br />
Downloaded using bbcp from services01.euclid into CephFS scratch:<br />
bbcp -s 16 -a -A -v -z -r -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" \<br />
"jcarrete@dtn01.nersc.gov:/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m/north/tractor" \<br />
/cephfs/pic.es/astro/scratch/tallada/dr9m/north/tractor<br />
bbcp -s 16 -a -A -v -z -r -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" \<br />
"jcarrete@dtn01.nersc.gov:/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m/south/tractor" \<br />
/cephfs/pic.es/astro/scratch/tallada/dr9m/south/tractor<br />
<br />
Checksums provided through sha256sum files, and checked using command:<br />
find * -type d | xargs -t -L1 -P24 sh -c 'cd $1; sha256sum --quiet -c *.sha256sum' cksum<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to convert fits files to parquet:<br />
<br />
path_base = '/cephfs/pic.es/astro/scratch/tallada/dr9m'<br />
hdfs_base = '/user/tallada/data/legacysurvey_dr9'<br />
<br />
hs = ['h=north', 'h=south']<br />
rs = ['r={0:03d}'.format(x) for x in range(360)]<br />
<br />
for h in hs:<br />
for r in rs:<br />
p = os.path.join(path_base, h, r)<br />
if not os.path.exists(p):<br />
continue<br />
<br />
size = sum(os.path.getsize(os.path.join(p,f)) for f in os.listdir(p) if os.path.isfile(os.path.join(p,f)))<br />
<br />
print h, r, size <br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + p<br />
)<br />
<br />
df = df.repartition(int(math.ceil(size / (256*1024*1024.0))))<br />
<br />
df.write.parquet(os.path.join(hdfs_base, h, r))<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE `tallada`.`legacysurvey_dr9`(<br />
`release` smallint, <br />
`brickid` int, <br />
`brickname` string, <br />
`objid` int, <br />
`brick_primary` boolean, <br />
`maskbits` smallint, <br />
`fitbits` smallint, <br />
`type` string, <br />
`ra` double, <br />
`dec` double, <br />
`ra_ivar` float, <br />
`dec_ivar` float, <br />
`bx` float, <br />
`by` float, <br />
`dchisq` array<float>, <br />
`ebv` float, <br />
`mjd_min` double, <br />
`mjd_max` double, <br />
`ref_cat` string, <br />
`ref_id` bigint, <br />
`pmra` float, <br />
`pmdec` float, <br />
`parallax` float, <br />
`pmra_ivar` float, <br />
`pmdec_ivar` float, <br />
`parallax_ivar` float, <br />
`ref_epoch` float, <br />
`gaia_phot_g_mean_mag` float, <br />
`gaia_phot_g_mean_flux_over_error` float, <br />
`gaia_phot_g_n_obs` smallint, <br />
`gaia_phot_bp_mean_mag` float, <br />
`gaia_phot_bp_mean_flux_over_error` float, <br />
`gaia_phot_bp_n_obs` smallint, <br />
`gaia_phot_rp_mean_mag` float, <br />
`gaia_phot_rp_mean_flux_over_error` float, <br />
`gaia_phot_rp_n_obs` smallint, <br />
`gaia_phot_variable_flag` boolean, <br />
`gaia_astrometric_excess_noise` float, <br />
`gaia_astrometric_excess_noise_sig` float, <br />
`gaia_astrometric_n_obs_al` smallint, <br />
`gaia_astrometric_n_good_obs_al` smallint, <br />
`gaia_astrometric_weight_al` float, <br />
`gaia_duplicated_source` boolean, <br />
`gaia_a_g_val` float, <br />
`gaia_e_bp_min_rp_val` float, <br />
`gaia_phot_bp_rp_excess_factor` float, <br />
`gaia_astrometric_sigma5d_max` float, <br />
`gaia_astrometric_params_solved` tinyint, <br />
`flux_g` float, <br />
`flux_r` float, <br />
`flux_z` float, <br />
`flux_w1` float, <br />
`flux_w2` float, <br />
`flux_w3` float, <br />
`flux_w4` float, <br />
`flux_ivar_g` float, <br />
`flux_ivar_r` float, <br />
`flux_ivar_z` float, <br />
`flux_ivar_w1` float, <br />
`flux_ivar_w2` float, <br />
`flux_ivar_w3` float, <br />
`flux_ivar_w4` float, <br />
`fiberflux_g` float, <br />
`fiberflux_r` float, <br />
`fiberflux_z` float, <br />
`fibertotflux_g` float, <br />
`fibertotflux_r` float, <br />
`fibertotflux_z` float, <br />
`apflux_g` array<float>, <br />
`apflux_r` array<float>, <br />
`apflux_z` array<float>, <br />
`apflux_resid_g` array<float>, <br />
`apflux_resid_r` array<float>, <br />
`apflux_resid_z` array<float>, <br />
`apflux_blobresid_g` array<float>, <br />
`apflux_blobresid_r` array<float>, <br />
`apflux_blobresid_z` array<float>, <br />
`apflux_ivar_g` array<float>, <br />
`apflux_ivar_r` array<float>, <br />
`apflux_ivar_z` array<float>, <br />
`apflux_masked_g` array<float>, <br />
`apflux_masked_r` array<float>, <br />
`apflux_masked_z` array<float>, <br />
`apflux_w1` array<float>, <br />
`apflux_w2` array<float>, <br />
`apflux_w3` array<float>, <br />
`apflux_w4` array<float>, <br />
`apflux_resid_w1` array<float>, <br />
`apflux_resid_w2` array<float>, <br />
`apflux_resid_w3` array<float>, <br />
`apflux_resid_w4` array<float>, <br />
`apflux_ivar_w1` array<float>, <br />
`apflux_ivar_w2` array<float>, <br />
`apflux_ivar_w3` array<float>, <br />
`apflux_ivar_w4` array<float>, <br />
`mw_transmission_g` float, <br />
`mw_transmission_r` float, <br />
`mw_transmission_z` float, <br />
`mw_transmission_w1` float, <br />
`mw_transmission_w2` float, <br />
`mw_transmission_w3` float, <br />
`mw_transmission_w4` float, <br />
`nobs_g` smallint, <br />
`nobs_r` smallint, <br />
`nobs_z` smallint, <br />
`nobs_w1` smallint, <br />
`nobs_w2` smallint, <br />
`nobs_w3` smallint, <br />
`nobs_w4` smallint, <br />
`rchisq_g` float, <br />
`rchisq_r` float, <br />
`rchisq_z` float, <br />
`rchisq_w1` float, <br />
`rchisq_w2` float, <br />
`rchisq_w3` float, <br />
`rchisq_w4` float, <br />
`fracflux_g` float, <br />
`fracflux_r` float, <br />
`fracflux_z` float, <br />
`fracflux_w1` float, <br />
`fracflux_w2` float, <br />
`fracflux_w3` float, <br />
`fracflux_w4` float, <br />
`fracmasked_g` float, <br />
`fracmasked_r` float, <br />
`fracmasked_z` float, <br />
`fracin_g` float, <br />
`fracin_r` float, <br />
`fracin_z` float, <br />
`anymask_g` smallint, <br />
`anymask_r` smallint, <br />
`anymask_z` smallint, <br />
`allmask_g` smallint, <br />
`allmask_r` smallint, <br />
`allmask_z` smallint, <br />
`wisemask_w1` tinyint, <br />
`wisemask_w2` tinyint, <br />
`psfsize_g` float, <br />
`psfsize_r` float, <br />
`psfsize_z` float, <br />
`psfdepth_g` float, <br />
`psfdepth_r` float, <br />
`psfdepth_z` float, <br />
`galdepth_g` float, <br />
`galdepth_r` float, <br />
`galdepth_z` float, <br />
`nea_g` float, <br />
`nea_r` float, <br />
`nea_z` float, <br />
`blob_nea_g` float, <br />
`blob_nea_r` float, <br />
`blob_nea_z` float, <br />
`psfdepth_w1` float, <br />
`psfdepth_w2` float, <br />
`psfdepth_w3` float, <br />
`psfdepth_w4` float, <br />
`wise_coadd_id` string, <br />
`wise_x` float, <br />
`wise_y` float, <br />
`lc_flux_w1` array<float>, <br />
`lc_flux_w2` array<float>, <br />
`lc_flux_ivar_w1` array<float>, <br />
`lc_flux_ivar_w2` array<float>, <br />
`lc_nobs_w1` array<smallint>, <br />
`lc_nobs_w2` array<smallint>, <br />
`lc_fracflux_w1` array<float>, <br />
`lc_fracflux_w2` array<float>, <br />
`lc_rchisq_w1` array<float>, <br />
`lc_rchisq_w2` array<float>, <br />
`lc_mjd_w1` array<double>, <br />
`lc_mjd_w2` array<double>, <br />
`lc_epoch_index_w1` array<smallint>, <br />
`lc_epoch_index_w2` array<smallint>, <br />
`sersic` float, <br />
`sersic_ivar` float, <br />
`shape_r` float, <br />
`shape_r_ivar` float, <br />
`shape_e1` float, <br />
`shape_e1_ivar` float, <br />
`shape_e2` float, <br />
`shape_e2_ivar` float)<br />
PARTITIONED BY (<br />
h STRING,<br />
r STRING<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey_dr9'<br />
;<br />
<br />
=== Internal Table ===<br />
<br />
CREATE TABLE `cosmohub`.`legacy_survey_dr9`(<br />
`release` smallint,<br />
`brickid` int,<br />
`brickname` string,<br />
`objid` int,<br />
`brick_primary` boolean,<br />
`maskbits` smallint,<br />
`fitbits` smallint,<br />
`type` string,<br />
`ra` double,<br />
`dec` double,<br />
`ra_ivar` float,<br />
`dec_ivar` float,<br />
`bx` float,<br />
`by` float,<br />
`dchisq` array<float>,<br />
`ebv` float,<br />
`mjd_min` double,<br />
`mjd_max` double,<br />
`ref_cat` string,<br />
`ref_id` bigint,<br />
`pmra` float,<br />
`pmdec` float,<br />
`parallax` float,<br />
`pmra_ivar` float,<br />
`pmdec_ivar` float,<br />
`parallax_ivar` float,<br />
`ref_epoch` float,<br />
`gaia_phot_g_mean_mag` float,<br />
`gaia_phot_g_mean_flux_over_error` float,<br />
`gaia_phot_g_n_obs` smallint,<br />
`gaia_phot_bp_mean_mag` float,<br />
`gaia_phot_bp_mean_flux_over_error` float,<br />
`gaia_phot_bp_n_obs` smallint,<br />
`gaia_phot_rp_mean_mag` float,<br />
`gaia_phot_rp_mean_flux_over_error` float,<br />
`gaia_phot_rp_n_obs` smallint,<br />
`gaia_phot_variable_flag` boolean,<br />
`gaia_astrometric_excess_noise` float,<br />
`gaia_astrometric_excess_noise_sig` float,<br />
`gaia_astrometric_n_obs_al` smallint,<br />
`gaia_astrometric_n_good_obs_al` smallint,<br />
`gaia_astrometric_weight_al` float,<br />
`gaia_duplicated_source` boolean,<br />
`gaia_a_g_val` float,<br />
`gaia_e_bp_min_rp_val` float,<br />
`gaia_phot_bp_rp_excess_factor` float,<br />
`gaia_astrometric_sigma5d_max` float,<br />
`gaia_astrometric_params_solved` tinyint,<br />
`flux_g` float,<br />
`flux_r` float,<br />
`flux_z` float,<br />
`flux_w1` float,<br />
`flux_w2` float,<br />
`flux_w3` float,<br />
`flux_w4` float,<br />
`flux_ivar_g` float,<br />
`flux_ivar_r` float,<br />
`flux_ivar_z` float,<br />
`flux_ivar_w1` float,<br />
`flux_ivar_w2` float,<br />
`flux_ivar_w3` float,<br />
`flux_ivar_w4` float,<br />
`fiberflux_g` float,<br />
`fiberflux_r` float,<br />
`fiberflux_z` float,<br />
`fibertotflux_g` float,<br />
`fibertotflux_r` float,<br />
`fibertotflux_z` float,<br />
`apflux_g` array<float>,<br />
`apflux_r` array<float>,<br />
`apflux_z` array<float>,<br />
`apflux_resid_g` array<float>,<br />
`apflux_resid_r` array<float>,<br />
`apflux_resid_z` array<float>,<br />
`apflux_blobresid_g` array<float>,<br />
`apflux_blobresid_r` array<float>,<br />
`apflux_blobresid_z` array<float>,<br />
`apflux_ivar_g` array<float>,<br />
`apflux_ivar_r` array<float>,<br />
`apflux_ivar_z` array<float>,<br />
`apflux_masked_g` array<float>,<br />
`apflux_masked_r` array<float>,<br />
`apflux_masked_z` array<float>,<br />
`apflux_w1` array<float>,<br />
`apflux_w2` array<float>,<br />
`apflux_w3` array<float>,<br />
`apflux_w4` array<float>,<br />
`apflux_resid_w1` array<float>,<br />
`apflux_resid_w2` array<float>,<br />
`apflux_resid_w3` array<float>,<br />
`apflux_resid_w4` array<float>,<br />
`apflux_ivar_w1` array<float>,<br />
`apflux_ivar_w2` array<float>,<br />
`apflux_ivar_w3` array<float>,<br />
`apflux_ivar_w4` array<float>,<br />
`mw_transmission_g` float,<br />
`mw_transmission_r` float,<br />
`mw_transmission_z` float,<br />
`mw_transmission_w1` float,<br />
`mw_transmission_w2` float,<br />
`mw_transmission_w3` float,<br />
`mw_transmission_w4` float,<br />
`nobs_g` smallint,<br />
`nobs_r` smallint,<br />
`nobs_z` smallint,<br />
`nobs_w1` smallint,<br />
`nobs_w2` smallint,<br />
`nobs_w3` smallint,<br />
`nobs_w4` smallint,<br />
`rchisq_g` float,<br />
`rchisq_r` float,<br />
`rchisq_z` float,<br />
`rchisq_w1` float,<br />
`rchisq_w2` float,<br />
`rchisq_w3` float,<br />
`rchisq_w4` float,<br />
`fracflux_g` float,<br />
`fracflux_r` float,<br />
`fracflux_z` float,<br />
`fracflux_w1` float,<br />
`fracflux_w2` float,<br />
`fracflux_w3` float,<br />
`fracflux_w4` float,<br />
`fracmasked_g` float,<br />
`fracmasked_r` float,<br />
`fracmasked_z` float,<br />
`fracin_g` float,<br />
`fracin_r` float,<br />
`fracin_z` float,<br />
`anymask_g` smallint,<br />
`anymask_r` smallint,<br />
`anymask_z` smallint,<br />
`allmask_g` smallint,<br />
`allmask_r` smallint,<br />
`allmask_z` smallint,<br />
`wisemask_w1` tinyint,<br />
`wisemask_w2` tinyint,<br />
`psfsize_g` float,<br />
`psfsize_r` float,<br />
`psfsize_z` float,<br />
`psfdepth_g` float,<br />
`psfdepth_r` float,<br />
`psfdepth_z` float,<br />
`galdepth_g` float,<br />
`galdepth_r` float,<br />
`galdepth_z` float,<br />
`nea_g` float,<br />
`nea_r` float,<br />
`nea_z` float,<br />
`blob_nea_g` float,<br />
`blob_nea_r` float,<br />
`blob_nea_z` float,<br />
`psfdepth_w1` float,<br />
`psfdepth_w2` float,<br />
`psfdepth_w3` float,<br />
`psfdepth_w4` float,<br />
`wise_coadd_id` string,<br />
`wise_x` float,<br />
`wise_y` float,<br />
`lc_flux_w1` array<float>,<br />
`lc_flux_w2` array<float>,<br />
`lc_flux_ivar_w1` array<float>,<br />
`lc_flux_ivar_w2` array<float>,<br />
`lc_nobs_w1` array<smallint>,<br />
`lc_nobs_w2` array<smallint>,<br />
`lc_fracflux_w1` array<float>,<br />
`lc_fracflux_w2` array<float>,<br />
`lc_rchisq_w1` array<float>,<br />
`lc_rchisq_w2` array<float>,<br />
`lc_mjd_w1` array<double>,<br />
`lc_mjd_w2` array<double>,<br />
`lc_epoch_index_w1` array<smallint>,<br />
`lc_epoch_index_w2` array<smallint>,<br />
`sersic` float,<br />
`sersic_ivar` float,<br />
`shape_r` float,<br />
`shape_r_ivar` float,<br />
`shape_e1` float,<br />
`shape_e1_ivar` float,<br />
`shape_e2` float,<br />
`shape_e2_ivar` float)<br />
CLUSTERED BY (<br />
release, brickid, objid<br />
)<br />
SORTED BY (<br />
release, brickid, objid<br />
)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
<br />
INSERT INTO TABLE cosmohub.legacy_survey_dr9<br />
SELECT release, brickid, brickname, objid, brick_primary, maskbits, fitbits, type, ra, `dec`, ra_ivar, dec_ivar,<br />
bx, `by`, dchisq, ebv, mjd_min, mjd_max, ref_cat, ref_id, pmra, pmdec, parallax, pmra_ivar, pmdec_ivar,<br />
parallax_ivar, ref_epoch, gaia_phot_g_mean_mag, gaia_phot_g_mean_flux_over_error, gaia_phot_g_n_obs,<br />
gaia_phot_bp_mean_mag, gaia_phot_bp_mean_flux_over_error, gaia_phot_bp_n_obs, gaia_phot_rp_mean_mag,<br />
gaia_phot_rp_mean_flux_over_error, gaia_phot_rp_n_obs, gaia_phot_variable_flag, gaia_astrometric_excess_noise,<br />
gaia_astrometric_excess_noise_sig, gaia_astrometric_n_obs_al, gaia_astrometric_n_good_obs_al,<br />
gaia_astrometric_weight_al, gaia_duplicated_source, gaia_a_g_val, gaia_e_bp_min_rp_val,<br />
gaia_phot_bp_rp_excess_factor, gaia_astrometric_sigma5d_max, gaia_astrometric_params_solved, flux_g, flux_r,<br />
flux_z, flux_w1, flux_w2, flux_w3, flux_w4, flux_ivar_g, flux_ivar_r, flux_ivar_z, flux_ivar_w1, flux_ivar_w2,<br />
flux_ivar_w3, flux_ivar_w4, fiberflux_g, fiberflux_r, fiberflux_z, fibertotflux_g, fibertotflux_r, fibertotflux_z,<br />
apflux_g, apflux_r, apflux_z, apflux_resid_g, apflux_resid_r, apflux_resid_z, apflux_blobresid_g,<br />
apflux_blobresid_r, apflux_blobresid_z, apflux_ivar_g, apflux_ivar_r, apflux_ivar_z, apflux_masked_g,<br />
apflux_masked_r, apflux_masked_z, apflux_w1, apflux_w2, apflux_w3, apflux_w4, apflux_resid_w1, apflux_resid_w2,<br />
apflux_resid_w3, apflux_resid_w4, apflux_ivar_w1, apflux_ivar_w2, apflux_ivar_w3, apflux_ivar_w4,<br />
mw_transmission_g, mw_transmission_r, mw_transmission_z, mw_transmission_w1, mw_transmission_w2,<br />
mw_transmission_w3, mw_transmission_w4, nobs_g, nobs_r, nobs_z, nobs_w1, nobs_w2, nobs_w3, nobs_w4, rchisq_g,<br />
rchisq_r, rchisq_z, rchisq_w1, rchisq_w2, rchisq_w3, rchisq_w4, fracflux_g, fracflux_r, fracflux_z, fracflux_w1,<br />
fracflux_w2, fracflux_w3, fracflux_w4, fracmasked_g, fracmasked_r, fracmasked_z, fracin_g, fracin_r, fracin_z,<br />
anymask_g, anymask_r, anymask_z, allmask_g, allmask_r, allmask_z, wisemask_w1, wisemask_w2, psfsize_g, psfsize_r,<br />
psfsize_z, psfdepth_g, psfdepth_r, psfdepth_z, galdepth_g, galdepth_r, galdepth_z, nea_g, nea_r, nea_z,<br />
blob_nea_g, blob_nea_r, blob_nea_z, psfdepth_w1, psfdepth_w2, psfdepth_w3, psfdepth_w4, wise_coadd_id, wise_x,<br />
wise_y, lc_flux_w1, lc_flux_w2, lc_flux_ivar_w1, lc_flux_ivar_w2, lc_nobs_w1, lc_nobs_w2, lc_fracflux_w1,<br />
lc_fracflux_w2, lc_rchisq_w1, lc_rchisq_w2, lc_mjd_w1, lc_mjd_w2, lc_epoch_index_w1, lc_epoch_index_w2, sersic,<br />
sersic_ivar, shape_r, shape_r_ivar, shape_e1, shape_e1_ivar, shape_e2, shape_e2_ivar<br />
FROM cosmohub.legacy_survey_dr9_p<br />
;<br />
<br />
=== Comments ===<br />
ALTER TABLE legacy_survey_dr9 CHANGE release release smallint COMMENT "Unique integer denoting the camera and filter set used" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE brickid brickid int COMMENT "Brick ID [1,662174]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE brickname brickname string COMMENT "Name of brick, encoding the brick sky position, eg \u00221126p222\u0022 near RA=112.6, Dec=+22.2" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE objid objid int COMMENT "Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE brick_primary brick_primary boolean COMMENT "True if the object is within the brick boundary" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE maskbits maskbits smallint COMMENT "Bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits* maps, as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fitbits fitbits smallint COMMENT "Bitwise mask detailing pecularities of how an object was fit, as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE type type string COMMENT "Morphological model: \u0022PSF\u0022=stellar, \u0022REX\u0022=\u0022round exponential galaxy\u0022, \u0022DEV\u0022=deVauc, \u0022EXP\u0022=exponential, \u0022SER\u0022=Sersic, \u0022DUP\u0022=Gaia source fit by different model." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ra ra double COMMENT "Right ascension at equinox J2000" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE `dec` `dec` double COMMENT "Declination at equinox J2000" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ra_ivar ra_ivar float COMMENT "Inverse variance of RA (no cosine term!), excluding astrometric calibration errors" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE dec_ivar dec_ivar float COMMENT "Inverse variance of DEC, excluding astrometric calibration errors" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE bx bx float COMMENT "X position (0-indexed) of coordinates in the brick image stack (i.e. in the e.g. legacysurvey-<brick>-image-g.fits.fz coadd file)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE `by` `by` float COMMENT "Y position (0-indexed) of coordinates in brick image stack" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE dchisq dchisq array<float> COMMENT "Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, SER. The difference is versus no source." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ebv ebv float COMMENT "Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mjd_min mjd_min double COMMENT "Minimum Modified Julian Date of observations used to construct the model of this object" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mjd_max mjd_max double COMMENT "Maximum Modified Julian Date of observations used to construct the model of this object" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ref_cat ref_cat string COMMENT "Reference catalog source for this star: \u0022T2\u0022 for Tycho-2, \u0022G2\u0022 for Gaia DR2, \u0022L3\u0022 for the SGA, empty otherwise" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ref_id ref_id bigint COMMENT "Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b \u0022sourceid\u0022 for Gaia DR2 and SGA" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE pmra pmra float COMMENT "Reference catalog proper motion in the RA direction" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE pmdec pmdec float COMMENT "Reference catalog proper motion in the Dec direction" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE parallax parallax float COMMENT "Reference catalog parallax" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE pmra_ivar pmra_ivar float COMMENT "Reference catalog inverse-variance on pmra" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE pmdec_ivar pmdec_ivar float COMMENT "Reference catalog inverse-variance on pmdec" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE parallax_ivar parallax_ivar float COMMENT "Reference catalog inverse-variance on parallax" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ref_epoch ref_epoch float COMMENT "Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_g_mean_mag gaia_phot_g_mean_mag float COMMENT "Gaia G band mag" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_g_mean_flux_over_error gaia_phot_g_mean_flux_over_error float COMMENT "Gaia G band signal-to-noise" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_g_n_obs gaia_phot_g_n_obs smallint COMMENT "Gaia G band number of observations" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_bp_mean_mag gaia_phot_bp_mean_mag float COMMENT "Gaia BP mag" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_bp_mean_flux_over_error gaia_phot_bp_mean_flux_over_error float COMMENT "Gaia BP signal-to-noise" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_bp_n_obs gaia_phot_bp_n_obs smallint COMMENT "Gaia BP number of observations" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_rp_mean_mag gaia_phot_rp_mean_mag float COMMENT "Gaia RP mag" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_rp_mean_flux_over_error gaia_phot_rp_mean_flux_over_error float COMMENT "Gaia RP signal-to-noise" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_rp_n_obs gaia_phot_rp_n_obs smallint COMMENT "Gaia RP number of observations" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_variable_flag gaia_phot_variable_flag boolean COMMENT "Gaia photometric variable flag" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_excess_noise gaia_astrometric_excess_noise float COMMENT "Gaia astrometric excess noise" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_excess_noise_sig gaia_astrometric_excess_noise_sig float COMMENT "Gaia astrometric excess noise uncertainty" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_n_obs_al gaia_astrometric_n_obs_al smallint COMMENT "Gaia number of astrometric observations along scan direction" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_n_good_obs_al gaia_astrometric_n_good_obs_al smallint COMMENT "Gaia number of good astrometric observations along scan direction" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_weight_al gaia_astrometric_weight_al float COMMENT "Gaia astrometric weight along scan direction" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_duplicated_source gaia_duplicated_source boolean COMMENT "Gaia duplicated source flag" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_a_g_val gaia_a_g_val float COMMENT "Gaia line-of-sight extinction in the G band" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_e_bp_min_rp_val gaia_e_bp_min_rp_val float COMMENT "Gaia line-of-sight reddening E(BP-RP)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_bp_rp_excess_factor gaia_phot_bp_rp_excess_factor float COMMENT "Gaia BP/RP excess factor" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_sigma5d_max gaia_astrometric_sigma5d_max float COMMENT "Gaia longest semi-major axis of the 5-d error ellipsoid" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_params_solved gaia_astrometric_params_solved tinyint COMMENT "which astrometric parameters were estimated for a Gaia source" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_g flux_g float COMMENT "model flux in gg" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_r flux_r float COMMENT "model flux in rr" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_z flux_z float COMMENT "model flux in zz" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_w1 flux_w1 float COMMENT "WISE model flux in W1W1 (AB system)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_w2 flux_w2 float COMMENT "WISE model flux in W2W2 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_w3 flux_w3 float COMMENT "WISE model flux in W3W3 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_w4 flux_w4 float COMMENT "WISE model flux in W4W4 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_g flux_ivar_g float COMMENT "Inverse variance of flux_g" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_r flux_ivar_r float COMMENT "Inverse variance of flux_r" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_z flux_ivar_z float COMMENT "Inverse variance of flux_z" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_w1 flux_ivar_w1 float COMMENT "Inverse variance of flux_w1 (AB system)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_w2 flux_ivar_w2 float COMMENT "Inverse variance of flux_w2 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_w3 flux_ivar_w3 float COMMENT "Inverse variance of flux_w3 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_w4 flux_ivar_w4 float COMMENT "Inverse variance of flux_w4 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fiberflux_g fiberflux_g float COMMENT "Predicted gg-band flux within a fiber of diameter 1.5 arcsec from this object in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fiberflux_r fiberflux_r float COMMENT "Predicted rr-band flux within a fiber of diameter 1.5 arcsec from this object in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fiberflux_z fiberflux_z float COMMENT "Predicted zz-band flux within a fiber of diameter 1.5 arcsec from this object in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fibertotflux_g fibertotflux_g float COMMENT "Predicted gg-band flux within a fiber of diameter 1.5 arcsec from all sources at this location in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fibertotflux_r fibertotflux_r float COMMENT "Predicted rr-band flux within a fiber of diameter 1.5 arcsec from all sources at this location in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fibertotflux_z fibertotflux_z float COMMENT "Predicted zz-band flux within a fiber of diameter 1.5 arcsec from all sources at this location in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_g apflux_g array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg, masked by invvar=0invvar=0 (inverse variance of zero [1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_r apflux_r array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_z apflux_z array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_g apflux_resid_g array<float> COMMENT "Aperture fluxes on the co-added residual images in gg, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_r apflux_resid_r array<float> COMMENT "Aperture fluxes on the co-added residual images in rr, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_z apflux_resid_z array<float> COMMENT "Aperture fluxes on the co-added residual images in zz, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_blobresid_g apflux_blobresid_g array<float> COMMENT "Aperture fluxes on image−blobmodelimage−blobmodel residual maps in gg [2], masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_blobresid_r apflux_blobresid_r array<float> COMMENT "Aperture fluxes on image−blobmodelimage−blobmodel residual maps in rr, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_blobresid_z apflux_blobresid_z array<float> COMMENT "Aperture fluxes on image−blobmodelimage−blobmodel residual maps in zz, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_g apflux_ivar_g array<float> COMMENT "Inverse variance of apflux_resid_g, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_r apflux_ivar_r array<float> COMMENT "Inverse variance of apflux_resid_r, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_z apflux_ivar_z array<float> COMMENT "Inverse variance of apflux_resid_z, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_masked_g apflux_masked_g array<float> COMMENT "Fraction of pixels masked in gg-band aperture flux measurements\u003b 1 means fully masked (ie, fully ignored\u003b contributing zero to the measurement)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_masked_r apflux_masked_r array<float> COMMENT "Fraction of pixels masked in rr-band aperture flux measurements\u003b 1 means fully masked (ie, fully ignored\u003b contributing zero to the measurement)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_masked_z apflux_masked_z array<float> COMMENT "Fraction of pixels masked in zz-band aperture flux measurements\u003b 1 means fully masked (ie, fully ignored\u003b contributing zero to the measurement)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_w1 apflux_w1 array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [3, 5, 7, 9, 11] [3] arcsec in W1W1, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_w2 apflux_w2 array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [3, 5, 7, 9, 11] arcsec in W2W2, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_w3 apflux_w3 array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [3, 5, 7, 9, 11] arcsec in W3W3, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_w4 apflux_w4 array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [3, 5, 7, 9, 11] arcsec in W4W4, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_w1 apflux_resid_w1 array<float> COMMENT "Aperture fluxes on the co-added residual images in W1W1, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_w2 apflux_resid_w2 array<float> COMMENT "Aperture fluxes on the co-added residual images in W2W2, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_w3 apflux_resid_w3 array<float> COMMENT "Aperture fluxes on the co-added residual images in W3W3, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_w4 apflux_resid_w4 array<float> COMMENT "Aperture fluxes on the co-added residual images in W4W4, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_w1 apflux_ivar_w1 array<float> COMMENT "Inverse variance of apflux_resid_w1, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_w2 apflux_ivar_w2 array<float> COMMENT "Inverse variance of apflux_resid_w2, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_w3 apflux_ivar_w3 array<float> COMMENT "Inverse variance of apflux_resid_w3, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_w4 apflux_ivar_w4 array<float> COMMENT "Inverse variance of apflux_resid_w4, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_g mw_transmission_g float COMMENT "Galactic transmission in gg filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_r mw_transmission_r float COMMENT "Galactic transmission in rr filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_z mw_transmission_z float COMMENT "Galactic transmission in zz filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_w1 mw_transmission_w1 float COMMENT "Galactic transmission in W1W1 filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_w2 mw_transmission_w2 float COMMENT "Galactic transmission in W2W2 filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_w3 mw_transmission_w3 float COMMENT "Galactic transmission in W3W3 filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_w4 mw_transmission_w4 float COMMENT "Galactic transmission in W4W4 filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_g nobs_g smallint COMMENT "Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_r nobs_r smallint COMMENT "Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_z nobs_z smallint COMMENT "Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_w1 nobs_w1 smallint COMMENT "Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_w2 nobs_w2 smallint COMMENT "Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_w3 nobs_w3 smallint COMMENT "Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_w4 nobs_w4 smallint COMMENT "Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_g rchisq_g float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in gg" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_r rchisq_r float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in rr" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_z rchisq_z float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in zz" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_w1 rchisq_w1 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W1W1" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_w2 rchisq_w2 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W2W2" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_w3 rchisq_w3 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W3W3" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_w4 rchisq_w4 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W4W4" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_g fracflux_g float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in gg (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_r fracflux_r float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in rr (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_z fracflux_z float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in zz (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_w1 fracflux_w1 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1 (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_w2 fracflux_w2 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2 (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_w3 fracflux_w3 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3 (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_w4 fracflux_w4 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4 (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracmasked_g fracmasked_g float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracmasked_r fracmasked_r float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracmasked_z fracmasked_z float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracin_g fracin_g float COMMENT "Fraction of a source's flux within the blob in gg, near unity for real sources" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracin_r fracin_r float COMMENT "Fraction of a source's flux within the blob in rr, near unity for real sources" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracin_z fracin_z float COMMENT "Fraction of a source's flux within the blob in zz, near unity for real sources" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE anymask_g anymask_g smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE anymask_r anymask_r smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE anymask_z anymask_z smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE allmask_g allmask_g smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE allmask_r allmask_r smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE allmask_z allmask_z smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE wisemask_w1 wisemask_w1 tinyint COMMENT "W1 bitmask as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE wisemask_w2 wisemask_w2 tinyint COMMENT "W2 bitmask as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfsize_g psfsize_g float COMMENT "Weighted average PSF FWHM in the gg band" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfsize_r psfsize_r float COMMENT "Weighted average PSF FWHM in the rr band" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfsize_z psfsize_z float COMMENT "Weighted average PSF FWHM in the zz band" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_g psfdepth_g float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_r psfdepth_r float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_z psfdepth_z float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE galdepth_g galdepth_g float COMMENT "As for psfdepth_g but for a galaxy (0.45\u0022 exp, round) detection sensitivity" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE galdepth_r galdepth_r float COMMENT "As for psfdepth_r but for a galaxy (0.45\u0022 exp, round) detection sensitivity" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE galdepth_z galdepth_z float COMMENT "As for psfdepth_z but for a galaxy (0.45\u0022 exp, round) detection sensitivity" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nea_g nea_g float COMMENT "Noise equivalent area in gg." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nea_r nea_r float COMMENT "Noise equivalent area in rr." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nea_z nea_z float COMMENT "Noise equivalent area in zz." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE blob_nea_g blob_nea_g float COMMENT "Blob-masked noise equivalent area in gg." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE blob_nea_r blob_nea_r float COMMENT "Blob-masked noise equivalent area in rr." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE blob_nea_z blob_nea_z float COMMENT "Blob-masked noise equivalent area in zz." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_w1 psfdepth_w1 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W1" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_w2 psfdepth_w2 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W2" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_w3 psfdepth_w3 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W3" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_w4 psfdepth_w4 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W4" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE wise_coadd_id wise_coadd_id string COMMENT "unWISE coadd brick name (corresponding to the, e.g., legacysurvey-<brick>-image-W1.fits.fz coadd file) for the center of each object" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE wise_x wise_x float COMMENT "X position of coordinates in the brick image stack that corresponds to wise_coadd_id (see the DR9 updates page for transformations between wise_x and bx)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE wise_y wise_y float COMMENT "Y position of coordinates in the brick image stack that corresponds to wise_coadd_id (see the DR9 updates page for transformations between wise_y and by)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_flux_w1 lc_flux_w1 array<float> COMMENT "flux_w1 in each of up to fifteen unWISE coadd epochs (AB system\u003b defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_flux_w2 lc_flux_w2 array<float> COMMENT "flux_w2 in each of up to fifteen unWISE coadd epochs (AB\u003b defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_flux_ivar_w1 lc_flux_ivar_w1 array<float> COMMENT "Inverse variance of lc_flux_w1 (AB system\u003b defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_flux_ivar_w2 lc_flux_ivar_w2 array<float> COMMENT "Inverse variance of lc_flux_w2 (AB\u003b defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_nobs_w1 lc_nobs_w1 array<smallint> COMMENT "nobs_w1 in each of up to fifteen unWISE coadd epochs" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_nobs_w2 lc_nobs_w2 array<smallint> COMMENT "nobs_w2 in each of up to fifteen unWISE coadd epochs" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_fracflux_w1 lc_fracflux_w1 array<float> COMMENT "fracflux_w1 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_fracflux_w2 lc_fracflux_w2 array<float> COMMENT "fracflux_w2 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_rchisq_w1 lc_rchisq_w1 array<float> COMMENT "rchisq_w1 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_rchisq_w2 lc_rchisq_w2 array<float> COMMENT "rchisq_w2 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_mjd_w1 lc_mjd_w1 array<double> COMMENT "mjd_w1 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_mjd_w2 lc_mjd_w2 array<double> COMMENT "mjd_w2 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_epoch_index_w1 lc_epoch_index_w1 array<smallint> COMMENT "Index number of unWISE epoch for W1 (defaults to -1 for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_epoch_index_w2 lc_epoch_index_w2 array<smallint> COMMENT "Index number of unWISE epoch for W2 (defaults to -1 for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE sersic sersic float COMMENT "Power-law index for the Sersic profile model (type=\u0022SER\u0022)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE sersic_ivar sersic_ivar float COMMENT "Inverse variance of sersic" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_r shape_r float COMMENT "Half-light radius of galaxy model for galaxy type type (>0)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_r_ivar shape_r_ivar float COMMENT "Inverse variance of shape_r" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_e1 shape_e1 float COMMENT "Ellipticity component 1 of galaxy model for galaxy type type" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_e1_ivar shape_e1_ivar float COMMENT "Inverse variance of shape_e1" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_e2 shape_e2 float COMMENT "Ellipticity component 2 of galaxy model for galaxy type type" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_e2_ivar shape_e2_ivar float COMMENT "Inverse variance of shape_e2" ;</div>
Tallada
https://pwiki.pic.es/index.php?title=Legacy_Survey_DR9&diff=877
Legacy Survey DR9
2020-11-30T21:04:31Z
<p>Tallada: /* Comments */</p>
<hr />
<div>== Original DR9 ==<br />
=== Raw data ===<br />
<br />
Downloaded from NERSC<br />
/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m<br />
<br />
Directories:<br />
south/tractor<br />
north/tractor<br />
<br />
Downloaded using bbcp from services01.euclid into CephFS scratch:<br />
bbcp -s 16 -a -A -v -z -r -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" \<br />
"jcarrete@dtn01.nersc.gov:/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m/north/tractor" \<br />
/cephfs/pic.es/astro/scratch/tallada/dr9m/north/tractor<br />
bbcp -s 16 -a -A -v -z -r -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" \<br />
"jcarrete@dtn01.nersc.gov:/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m/south/tractor" \<br />
/cephfs/pic.es/astro/scratch/tallada/dr9m/south/tractor<br />
<br />
Checksums provided through sha256sum files, and checked using command:<br />
find * -type d | xargs -t -L1 -P24 sh -c 'cd $1; sha256sum --quiet -c *.sha256sum' cksum<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to convert fits files to parquet:<br />
<br />
path_base = '/cephfs/pic.es/astro/scratch/tallada/dr9m'<br />
hdfs_base = '/user/tallada/data/legacysurvey_dr9'<br />
<br />
hs = ['h=north', 'h=south']<br />
rs = ['r={0:03d}'.format(x) for x in range(360)]<br />
<br />
for h in hs:<br />
for r in rs:<br />
p = os.path.join(path_base, h, r)<br />
if not os.path.exists(p):<br />
continue<br />
<br />
size = sum(os.path.getsize(os.path.join(p,f)) for f in os.listdir(p) if os.path.isfile(os.path.join(p,f)))<br />
<br />
print h, r, size <br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + p<br />
)<br />
<br />
df = df.repartition(int(math.ceil(size / (256*1024*1024.0))))<br />
<br />
df.write.parquet(os.path.join(hdfs_base, h, r))<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE `tallada`.`legacysurvey_dr9`(<br />
`release` smallint, <br />
`brickid` int, <br />
`brickname` string, <br />
`objid` int, <br />
`brick_primary` boolean, <br />
`maskbits` smallint, <br />
`fitbits` smallint, <br />
`type` string, <br />
`ra` double, <br />
`dec` double, <br />
`ra_ivar` float, <br />
`dec_ivar` float, <br />
`bx` float, <br />
`by` float, <br />
`dchisq` array<float>, <br />
`ebv` float, <br />
`mjd_min` double, <br />
`mjd_max` double, <br />
`ref_cat` string, <br />
`ref_id` bigint, <br />
`pmra` float, <br />
`pmdec` float, <br />
`parallax` float, <br />
`pmra_ivar` float, <br />
`pmdec_ivar` float, <br />
`parallax_ivar` float, <br />
`ref_epoch` float, <br />
`gaia_phot_g_mean_mag` float, <br />
`gaia_phot_g_mean_flux_over_error` float, <br />
`gaia_phot_g_n_obs` smallint, <br />
`gaia_phot_bp_mean_mag` float, <br />
`gaia_phot_bp_mean_flux_over_error` float, <br />
`gaia_phot_bp_n_obs` smallint, <br />
`gaia_phot_rp_mean_mag` float, <br />
`gaia_phot_rp_mean_flux_over_error` float, <br />
`gaia_phot_rp_n_obs` smallint, <br />
`gaia_phot_variable_flag` boolean, <br />
`gaia_astrometric_excess_noise` float, <br />
`gaia_astrometric_excess_noise_sig` float, <br />
`gaia_astrometric_n_obs_al` smallint, <br />
`gaia_astrometric_n_good_obs_al` smallint, <br />
`gaia_astrometric_weight_al` float, <br />
`gaia_duplicated_source` boolean, <br />
`gaia_a_g_val` float, <br />
`gaia_e_bp_min_rp_val` float, <br />
`gaia_phot_bp_rp_excess_factor` float, <br />
`gaia_astrometric_sigma5d_max` float, <br />
`gaia_astrometric_params_solved` tinyint, <br />
`flux_g` float, <br />
`flux_r` float, <br />
`flux_z` float, <br />
`flux_w1` float, <br />
`flux_w2` float, <br />
`flux_w3` float, <br />
`flux_w4` float, <br />
`flux_ivar_g` float, <br />
`flux_ivar_r` float, <br />
`flux_ivar_z` float, <br />
`flux_ivar_w1` float, <br />
`flux_ivar_w2` float, <br />
`flux_ivar_w3` float, <br />
`flux_ivar_w4` float, <br />
`fiberflux_g` float, <br />
`fiberflux_r` float, <br />
`fiberflux_z` float, <br />
`fibertotflux_g` float, <br />
`fibertotflux_r` float, <br />
`fibertotflux_z` float, <br />
`apflux_g` array<float>, <br />
`apflux_r` array<float>, <br />
`apflux_z` array<float>, <br />
`apflux_resid_g` array<float>, <br />
`apflux_resid_r` array<float>, <br />
`apflux_resid_z` array<float>, <br />
`apflux_blobresid_g` array<float>, <br />
`apflux_blobresid_r` array<float>, <br />
`apflux_blobresid_z` array<float>, <br />
`apflux_ivar_g` array<float>, <br />
`apflux_ivar_r` array<float>, <br />
`apflux_ivar_z` array<float>, <br />
`apflux_masked_g` array<float>, <br />
`apflux_masked_r` array<float>, <br />
`apflux_masked_z` array<float>, <br />
`apflux_w1` array<float>, <br />
`apflux_w2` array<float>, <br />
`apflux_w3` array<float>, <br />
`apflux_w4` array<float>, <br />
`apflux_resid_w1` array<float>, <br />
`apflux_resid_w2` array<float>, <br />
`apflux_resid_w3` array<float>, <br />
`apflux_resid_w4` array<float>, <br />
`apflux_ivar_w1` array<float>, <br />
`apflux_ivar_w2` array<float>, <br />
`apflux_ivar_w3` array<float>, <br />
`apflux_ivar_w4` array<float>, <br />
`mw_transmission_g` float, <br />
`mw_transmission_r` float, <br />
`mw_transmission_z` float, <br />
`mw_transmission_w1` float, <br />
`mw_transmission_w2` float, <br />
`mw_transmission_w3` float, <br />
`mw_transmission_w4` float, <br />
`nobs_g` smallint, <br />
`nobs_r` smallint, <br />
`nobs_z` smallint, <br />
`nobs_w1` smallint, <br />
`nobs_w2` smallint, <br />
`nobs_w3` smallint, <br />
`nobs_w4` smallint, <br />
`rchisq_g` float, <br />
`rchisq_r` float, <br />
`rchisq_z` float, <br />
`rchisq_w1` float, <br />
`rchisq_w2` float, <br />
`rchisq_w3` float, <br />
`rchisq_w4` float, <br />
`fracflux_g` float, <br />
`fracflux_r` float, <br />
`fracflux_z` float, <br />
`fracflux_w1` float, <br />
`fracflux_w2` float, <br />
`fracflux_w3` float, <br />
`fracflux_w4` float, <br />
`fracmasked_g` float, <br />
`fracmasked_r` float, <br />
`fracmasked_z` float, <br />
`fracin_g` float, <br />
`fracin_r` float, <br />
`fracin_z` float, <br />
`anymask_g` smallint, <br />
`anymask_r` smallint, <br />
`anymask_z` smallint, <br />
`allmask_g` smallint, <br />
`allmask_r` smallint, <br />
`allmask_z` smallint, <br />
`wisemask_w1` tinyint, <br />
`wisemask_w2` tinyint, <br />
`psfsize_g` float, <br />
`psfsize_r` float, <br />
`psfsize_z` float, <br />
`psfdepth_g` float, <br />
`psfdepth_r` float, <br />
`psfdepth_z` float, <br />
`galdepth_g` float, <br />
`galdepth_r` float, <br />
`galdepth_z` float, <br />
`nea_g` float, <br />
`nea_r` float, <br />
`nea_z` float, <br />
`blob_nea_g` float, <br />
`blob_nea_r` float, <br />
`blob_nea_z` float, <br />
`psfdepth_w1` float, <br />
`psfdepth_w2` float, <br />
`psfdepth_w3` float, <br />
`psfdepth_w4` float, <br />
`wise_coadd_id` string, <br />
`wise_x` float, <br />
`wise_y` float, <br />
`lc_flux_w1` array<float>, <br />
`lc_flux_w2` array<float>, <br />
`lc_flux_ivar_w1` array<float>, <br />
`lc_flux_ivar_w2` array<float>, <br />
`lc_nobs_w1` array<smallint>, <br />
`lc_nobs_w2` array<smallint>, <br />
`lc_fracflux_w1` array<float>, <br />
`lc_fracflux_w2` array<float>, <br />
`lc_rchisq_w1` array<float>, <br />
`lc_rchisq_w2` array<float>, <br />
`lc_mjd_w1` array<double>, <br />
`lc_mjd_w2` array<double>, <br />
`lc_epoch_index_w1` array<smallint>, <br />
`lc_epoch_index_w2` array<smallint>, <br />
`sersic` float, <br />
`sersic_ivar` float, <br />
`shape_r` float, <br />
`shape_r_ivar` float, <br />
`shape_e1` float, <br />
`shape_e1_ivar` float, <br />
`shape_e2` float, <br />
`shape_e2_ivar` float)<br />
PARTITIONED BY (<br />
h STRING,<br />
r STRING<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey_dr9'<br />
;<br />
<br />
=== Internal Table ===<br />
<br />
CREATE TABLE `cosmohub`.`legacy_survey_dr9`(<br />
`release` smallint,<br />
`brickid` int,<br />
`brickname` string,<br />
`objid` int,<br />
`brick_primary` boolean,<br />
`maskbits` smallint,<br />
`fitbits` smallint,<br />
`type` string,<br />
`ra` double,<br />
`dec` double,<br />
`ra_ivar` float,<br />
`dec_ivar` float,<br />
`bx` float,<br />
`by` float,<br />
`dchisq` array<float>,<br />
`ebv` float,<br />
`mjd_min` double,<br />
`mjd_max` double,<br />
`ref_cat` string,<br />
`ref_id` bigint,<br />
`pmra` float,<br />
`pmdec` float,<br />
`parallax` float,<br />
`pmra_ivar` float,<br />
`pmdec_ivar` float,<br />
`parallax_ivar` float,<br />
`ref_epoch` float,<br />
`gaia_phot_g_mean_mag` float,<br />
`gaia_phot_g_mean_flux_over_error` float,<br />
`gaia_phot_g_n_obs` smallint,<br />
`gaia_phot_bp_mean_mag` float,<br />
`gaia_phot_bp_mean_flux_over_error` float,<br />
`gaia_phot_bp_n_obs` smallint,<br />
`gaia_phot_rp_mean_mag` float,<br />
`gaia_phot_rp_mean_flux_over_error` float,<br />
`gaia_phot_rp_n_obs` smallint,<br />
`gaia_phot_variable_flag` boolean,<br />
`gaia_astrometric_excess_noise` float,<br />
`gaia_astrometric_excess_noise_sig` float,<br />
`gaia_astrometric_n_obs_al` smallint,<br />
`gaia_astrometric_n_good_obs_al` smallint,<br />
`gaia_astrometric_weight_al` float,<br />
`gaia_duplicated_source` boolean,<br />
`gaia_a_g_val` float,<br />
`gaia_e_bp_min_rp_val` float,<br />
`gaia_phot_bp_rp_excess_factor` float,<br />
`gaia_astrometric_sigma5d_max` float,<br />
`gaia_astrometric_params_solved` tinyint,<br />
`flux_g` float,<br />
`flux_r` float,<br />
`flux_z` float,<br />
`flux_w1` float,<br />
`flux_w2` float,<br />
`flux_w3` float,<br />
`flux_w4` float,<br />
`flux_ivar_g` float,<br />
`flux_ivar_r` float,<br />
`flux_ivar_z` float,<br />
`flux_ivar_w1` float,<br />
`flux_ivar_w2` float,<br />
`flux_ivar_w3` float,<br />
`flux_ivar_w4` float,<br />
`fiberflux_g` float,<br />
`fiberflux_r` float,<br />
`fiberflux_z` float,<br />
`fibertotflux_g` float,<br />
`fibertotflux_r` float,<br />
`fibertotflux_z` float,<br />
`apflux_g` array<float>,<br />
`apflux_r` array<float>,<br />
`apflux_z` array<float>,<br />
`apflux_resid_g` array<float>,<br />
`apflux_resid_r` array<float>,<br />
`apflux_resid_z` array<float>,<br />
`apflux_blobresid_g` array<float>,<br />
`apflux_blobresid_r` array<float>,<br />
`apflux_blobresid_z` array<float>,<br />
`apflux_ivar_g` array<float>,<br />
`apflux_ivar_r` array<float>,<br />
`apflux_ivar_z` array<float>,<br />
`apflux_masked_g` array<float>,<br />
`apflux_masked_r` array<float>,<br />
`apflux_masked_z` array<float>,<br />
`apflux_w1` array<float>,<br />
`apflux_w2` array<float>,<br />
`apflux_w3` array<float>,<br />
`apflux_w4` array<float>,<br />
`apflux_resid_w1` array<float>,<br />
`apflux_resid_w2` array<float>,<br />
`apflux_resid_w3` array<float>,<br />
`apflux_resid_w4` array<float>,<br />
`apflux_ivar_w1` array<float>,<br />
`apflux_ivar_w2` array<float>,<br />
`apflux_ivar_w3` array<float>,<br />
`apflux_ivar_w4` array<float>,<br />
`mw_transmission_g` float,<br />
`mw_transmission_r` float,<br />
`mw_transmission_z` float,<br />
`mw_transmission_w1` float,<br />
`mw_transmission_w2` float,<br />
`mw_transmission_w3` float,<br />
`mw_transmission_w4` float,<br />
`nobs_g` smallint,<br />
`nobs_r` smallint,<br />
`nobs_z` smallint,<br />
`nobs_w1` smallint,<br />
`nobs_w2` smallint,<br />
`nobs_w3` smallint,<br />
`nobs_w4` smallint,<br />
`rchisq_g` float,<br />
`rchisq_r` float,<br />
`rchisq_z` float,<br />
`rchisq_w1` float,<br />
`rchisq_w2` float,<br />
`rchisq_w3` float,<br />
`rchisq_w4` float,<br />
`fracflux_g` float,<br />
`fracflux_r` float,<br />
`fracflux_z` float,<br />
`fracflux_w1` float,<br />
`fracflux_w2` float,<br />
`fracflux_w3` float,<br />
`fracflux_w4` float,<br />
`fracmasked_g` float,<br />
`fracmasked_r` float,<br />
`fracmasked_z` float,<br />
`fracin_g` float,<br />
`fracin_r` float,<br />
`fracin_z` float,<br />
`anymask_g` smallint,<br />
`anymask_r` smallint,<br />
`anymask_z` smallint,<br />
`allmask_g` smallint,<br />
`allmask_r` smallint,<br />
`allmask_z` smallint,<br />
`wisemask_w1` tinyint,<br />
`wisemask_w2` tinyint,<br />
`psfsize_g` float,<br />
`psfsize_r` float,<br />
`psfsize_z` float,<br />
`psfdepth_g` float,<br />
`psfdepth_r` float,<br />
`psfdepth_z` float,<br />
`galdepth_g` float,<br />
`galdepth_r` float,<br />
`galdepth_z` float,<br />
`nea_g` float,<br />
`nea_r` float,<br />
`nea_z` float,<br />
`blob_nea_g` float,<br />
`blob_nea_r` float,<br />
`blob_nea_z` float,<br />
`psfdepth_w1` float,<br />
`psfdepth_w2` float,<br />
`psfdepth_w3` float,<br />
`psfdepth_w4` float,<br />
`wise_coadd_id` string,<br />
`wise_x` float,<br />
`wise_y` float,<br />
`lc_flux_w1` array<float>,<br />
`lc_flux_w2` array<float>,<br />
`lc_flux_ivar_w1` array<float>,<br />
`lc_flux_ivar_w2` array<float>,<br />
`lc_nobs_w1` array<smallint>,<br />
`lc_nobs_w2` array<smallint>,<br />
`lc_fracflux_w1` array<float>,<br />
`lc_fracflux_w2` array<float>,<br />
`lc_rchisq_w1` array<float>,<br />
`lc_rchisq_w2` array<float>,<br />
`lc_mjd_w1` array<double>,<br />
`lc_mjd_w2` array<double>,<br />
`lc_epoch_index_w1` array<smallint>,<br />
`lc_epoch_index_w2` array<smallint>,<br />
`sersic` float,<br />
`sersic_ivar` float,<br />
`shape_r` float,<br />
`shape_r_ivar` float,<br />
`shape_e1` float,<br />
`shape_e1_ivar` float,<br />
`shape_e2` float,<br />
`shape_e2_ivar` float)<br />
CLUSTERED BY (<br />
release, brickid, objid<br />
)<br />
SORTED BY (<br />
release, brickid, objid<br />
)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
<br />
INSERT INTO TABLE cosmohub.legacy_survey_dr9<br />
SELECT release, brickid, brickname, objid, brick_primary, maskbits, fitbits, type, ra, `dec`, ra_ivar, dec_ivar,<br />
bx, `by`, dchisq, ebv, mjd_min, mjd_max, ref_cat, ref_id, pmra, pmdec, parallax, pmra_ivar, pmdec_ivar,<br />
parallax_ivar, ref_epoch, gaia_phot_g_mean_mag, gaia_phot_g_mean_flux_over_error, gaia_phot_g_n_obs,<br />
gaia_phot_bp_mean_mag, gaia_phot_bp_mean_flux_over_error, gaia_phot_bp_n_obs, gaia_phot_rp_mean_mag,<br />
gaia_phot_rp_mean_flux_over_error, gaia_phot_rp_n_obs, gaia_phot_variable_flag, gaia_astrometric_excess_noise,<br />
gaia_astrometric_excess_noise_sig, gaia_astrometric_n_obs_al, gaia_astrometric_n_good_obs_al,<br />
gaia_astrometric_weight_al, gaia_duplicated_source, gaia_a_g_val, gaia_e_bp_min_rp_val,<br />
gaia_phot_bp_rp_excess_factor, gaia_astrometric_sigma5d_max, gaia_astrometric_params_solved, flux_g, flux_r,<br />
flux_z, flux_w1, flux_w2, flux_w3, flux_w4, flux_ivar_g, flux_ivar_r, flux_ivar_z, flux_ivar_w1, flux_ivar_w2,<br />
flux_ivar_w3, flux_ivar_w4, fiberflux_g, fiberflux_r, fiberflux_z, fibertotflux_g, fibertotflux_r, fibertotflux_z,<br />
apflux_g, apflux_r, apflux_z, apflux_resid_g, apflux_resid_r, apflux_resid_z, apflux_blobresid_g,<br />
apflux_blobresid_r, apflux_blobresid_z, apflux_ivar_g, apflux_ivar_r, apflux_ivar_z, apflux_masked_g,<br />
apflux_masked_r, apflux_masked_z, apflux_w1, apflux_w2, apflux_w3, apflux_w4, apflux_resid_w1, apflux_resid_w2,<br />
apflux_resid_w3, apflux_resid_w4, apflux_ivar_w1, apflux_ivar_w2, apflux_ivar_w3, apflux_ivar_w4,<br />
mw_transmission_g, mw_transmission_r, mw_transmission_z, mw_transmission_w1, mw_transmission_w2,<br />
mw_transmission_w3, mw_transmission_w4, nobs_g, nobs_r, nobs_z, nobs_w1, nobs_w2, nobs_w3, nobs_w4, rchisq_g,<br />
rchisq_r, rchisq_z, rchisq_w1, rchisq_w2, rchisq_w3, rchisq_w4, fracflux_g, fracflux_r, fracflux_z, fracflux_w1,<br />
fracflux_w2, fracflux_w3, fracflux_w4, fracmasked_g, fracmasked_r, fracmasked_z, fracin_g, fracin_r, fracin_z,<br />
anymask_g, anymask_r, anymask_z, allmask_g, allmask_r, allmask_z, wisemask_w1, wisemask_w2, psfsize_g, psfsize_r,<br />
psfsize_z, psfdepth_g, psfdepth_r, psfdepth_z, galdepth_g, galdepth_r, galdepth_z, nea_g, nea_r, nea_z,<br />
blob_nea_g, blob_nea_r, blob_nea_z, psfdepth_w1, psfdepth_w2, psfdepth_w3, psfdepth_w4, wise_coadd_id, wise_x,<br />
wise_y, lc_flux_w1, lc_flux_w2, lc_flux_ivar_w1, lc_flux_ivar_w2, lc_nobs_w1, lc_nobs_w2, lc_fracflux_w1,<br />
lc_fracflux_w2, lc_rchisq_w1, lc_rchisq_w2, lc_mjd_w1, lc_mjd_w2, lc_epoch_index_w1, lc_epoch_index_w2, sersic,<br />
sersic_ivar, shape_r, shape_r_ivar, shape_e1, shape_e1_ivar, shape_e2, shape_e2_ivar<br />
FROM cosmohub.legacy_survey_dr9_p<br />
;<br />
<br />
=== Comments ===<br />
ALTER TABLE legacy_survey_dr9 CHANGE release release smallint COMMENT "Unique integer denoting the camera and filter set used" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE brickid brickid int COMMENT "Brick ID [1,662174]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE brickname brickname string COMMENT "Name of brick, encoding the brick sky position, eg \u00221126p222\u0022 near RA=112.6, Dec=+22.2" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE objid objid int COMMENT "Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE brick_primary brick_primary boolean COMMENT "True if the object is within the brick boundary" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE maskbits maskbits smallint COMMENT "Bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits* maps, as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fitbits fitbits smallint COMMENT "Bitwise mask detailing pecularities of how an object was fit, as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE type type string COMMENT "Morphological model: \u0022PSF\u0022=stellar, \u0022REX\u0022=\u0022round exponential galaxy\u0022, \u0022DEV\u0022=deVauc, \u0022EXP\u0022=exponential, \u0022SER\u0022=Sersic, \u0022DUP\u0022=Gaia source fit by different model." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ra ra double COMMENT "Right ascension at equinox J2000" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE `dec` `dec` double COMMENT "Declination at equinox J2000" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ra_ivar ra_ivar float COMMENT "Inverse variance of RA (no cosine term!), excluding astrometric calibration errors" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE dec_ivar dec_ivar float COMMENT "Inverse variance of DEC, excluding astrometric calibration errors" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE bx bx float COMMENT "X position (0-indexed) of coordinates in the brick image stack (i.e. in the e.g. legacysurvey-<brick>-image-g.fits.fz coadd file)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE `by` `by` float COMMENT "Y position (0-indexed) of coordinates in brick image stack" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE dchisq dchisq array<float> COMMENT "Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, SER. The difference is versus no source." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ebv ebv float COMMENT "Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mjd_min mjd_min double COMMENT "Minimum Modified Julian Date of observations used to construct the model of this object" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mjd_max mjd_max double COMMENT "Maximum Modified Julian Date of observations used to construct the model of this object" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ref_cat ref_cat string COMMENT "Reference catalog source for this star: \u0022T2\u0022 for Tycho-2, \u0022G2\u0022 for Gaia DR2, \u0022L3\u0022 for the SGA, empty otherwise" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ref_id ref_id bigint COMMENT "Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b \u0022sourceid\u0022 for Gaia DR2 and SGA" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE pmra pmra float COMMENT "Reference catalog proper motion in the RA direction" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE pmdec pmdec float COMMENT "Reference catalog proper motion in the Dec direction" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE parallax parallax float COMMENT "Reference catalog parallax" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE pmra_ivar pmra_ivar float COMMENT "Reference catalog inverse-variance on pmra" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE pmdec_ivar pmdec_ivar float COMMENT "Reference catalog inverse-variance on pmdec" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE parallax_ivar parallax_ivar float COMMENT "Reference catalog inverse-variance on parallax" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE ref_epoch ref_epoch float COMMENT "Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_g_mean_mag gaia_phot_g_mean_mag float COMMENT "Gaia G band mag" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_g_mean_flux_over_error gaia_phot_g_mean_flux_over_error float COMMENT "Gaia G band signal-to-noise" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_g_n_obs gaia_phot_g_n_obs smallint COMMENT "Gaia G band number of observations" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_bp_mean_mag gaia_phot_bp_mean_mag float COMMENT "Gaia BP mag" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_bp_mean_flux_over_error gaia_phot_bp_mean_flux_over_error float COMMENT "Gaia BP signal-to-noise" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_bp_n_obs gaia_phot_bp_n_obs smallint COMMENT "Gaia BP number of observations" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_rp_mean_mag gaia_phot_rp_mean_mag float COMMENT "Gaia RP mag" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_rp_mean_flux_over_error gaia_phot_rp_mean_flux_over_error float COMMENT "Gaia RP signal-to-noise" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_rp_n_obs gaia_phot_rp_n_obs smallint COMMENT "Gaia RP number of observations" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_variable_flag gaia_phot_variable_flag boolean COMMENT "Gaia photometric variable flag" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_excess_noise gaia_astrometric_excess_noise float COMMENT "Gaia astrometric excess noise" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_excess_noise_sig gaia_astrometric_excess_noise_sig float COMMENT "Gaia astrometric excess noise uncertainty" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_n_obs_al gaia_astrometric_n_obs_al smallint COMMENT "Gaia number of astrometric observations along scan direction" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_n_good_obs_al gaia_astrometric_n_good_obs_al smallint COMMENT "Gaia number of good astrometric observations along scan direction" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_weight_al gaia_astrometric_weight_al float COMMENT "Gaia astrometric weight along scan direction" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_duplicated_source gaia_duplicated_source boolean COMMENT "Gaia duplicated source flag" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_a_g_val gaia_a_g_val float COMMENT "Gaia line-of-sight extinction in the G band" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_e_bp_min_rp_val gaia_e_bp_min_rp_val float COMMENT "Gaia line-of-sight reddening E(BP-RP)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_phot_bp_rp_excess_factor gaia_phot_bp_rp_excess_factor float COMMENT "Gaia BP/RP excess factor" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_sigma5d_max gaia_astrometric_sigma5d_max float COMMENT "Gaia longest semi-major axis of the 5-d error ellipsoid" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE gaia_astrometric_params_solved gaia_astrometric_params_solved tinyint COMMENT "which astrometric parameters were estimated for a Gaia source" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_g flux_g float COMMENT "model flux in gg" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_r flux_r float COMMENT "model flux in rr" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_z flux_z float COMMENT "model flux in zz" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_w1 flux_w1 float COMMENT "WISE model flux in W1W1 (AB system)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_w2 flux_w2 float COMMENT "WISE model flux in W2W2 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_w3 flux_w3 float COMMENT "WISE model flux in W3W3 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_w4 flux_w4 float COMMENT "WISE model flux in W4W4 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_g flux_ivar_g float COMMENT "Inverse variance of flux_g" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_r flux_ivar_r float COMMENT "Inverse variance of flux_r" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_z flux_ivar_z float COMMENT "Inverse variance of flux_z" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_w1 flux_ivar_w1 float COMMENT "Inverse variance of flux_w1 (AB system)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_w2 flux_ivar_w2 float COMMENT "Inverse variance of flux_w2 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_w3 flux_ivar_w3 float COMMENT "Inverse variance of flux_w3 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE flux_ivar_w4 flux_ivar_w4 float COMMENT "Inverse variance of flux_w4 (AB)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fiberflux_g fiberflux_g float COMMENT "Predicted gg-band flux within a fiber of diameter 1.5 arcsec from this object in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fiberflux_r fiberflux_r float COMMENT "Predicted rr-band flux within a fiber of diameter 1.5 arcsec from this object in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fiberflux_z fiberflux_z float COMMENT "Predicted zz-band flux within a fiber of diameter 1.5 arcsec from this object in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fibertotflux_g fibertotflux_g float COMMENT "Predicted gg-band flux within a fiber of diameter 1.5 arcsec from all sources at this location in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fibertotflux_r fibertotflux_r float COMMENT "Predicted rr-band flux within a fiber of diameter 1.5 arcsec from all sources at this location in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fibertotflux_z fibertotflux_z float COMMENT "Predicted zz-band flux within a fiber of diameter 1.5 arcsec from all sources at this location in 1 arcsec Gaussian seeing" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_g apflux_g array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg, masked by invvar=0invvar=0 (inverse variance of zero [1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_r apflux_r array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_z apflux_z array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_g apflux_resid_g array<float> COMMENT "Aperture fluxes on the co-added residual images in gg, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_r apflux_resid_r array<float> COMMENT "Aperture fluxes on the co-added residual images in rr, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_z apflux_resid_z array<float> COMMENT "Aperture fluxes on the co-added residual images in zz, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_blobresid_g apflux_blobresid_g array<float> COMMENT "Aperture fluxes on image−blobmodelimage−blobmodel residual maps in gg [2], masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_blobresid_r apflux_blobresid_r array<float> COMMENT "Aperture fluxes on image−blobmodelimage−blobmodel residual maps in rr, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_blobresid_z apflux_blobresid_z array<float> COMMENT "Aperture fluxes on image−blobmodelimage−blobmodel residual maps in zz, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_g apflux_ivar_g array<float> COMMENT "Inverse variance of apflux_resid_g, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_r apflux_ivar_r array<float> COMMENT "Inverse variance of apflux_resid_r, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_z apflux_ivar_z array<float> COMMENT "Inverse variance of apflux_resid_z, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_masked_g apflux_masked_g array<float> COMMENT "Fraction of pixels masked in gg-band aperture flux measurements\u003b 1 means fully masked (ie, fully ignored\u003b contributing zero to the measurement)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_masked_r apflux_masked_r array<float> COMMENT "Fraction of pixels masked in rr-band aperture flux measurements\u003b 1 means fully masked (ie, fully ignored\u003b contributing zero to the measurement)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_masked_z apflux_masked_z array<float> COMMENT "Fraction of pixels masked in zz-band aperture flux measurements\u003b 1 means fully masked (ie, fully ignored\u003b contributing zero to the measurement)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_w1 apflux_w1 array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [3, 5, 7, 9, 11] [3] arcsec in W1W1, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_w2 apflux_w2 array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [3, 5, 7, 9, 11] arcsec in W2W2, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_w3 apflux_w3 array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [3, 5, 7, 9, 11] arcsec in W3W3, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_w4 apflux_w4 array<float> COMMENT "Aperture fluxes on the co-added images in apertures of radius [3, 5, 7, 9, 11] arcsec in W4W4, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_w1 apflux_resid_w1 array<float> COMMENT "Aperture fluxes on the co-added residual images in W1W1, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_w2 apflux_resid_w2 array<float> COMMENT "Aperture fluxes on the co-added residual images in W2W2, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_w3 apflux_resid_w3 array<float> COMMENT "Aperture fluxes on the co-added residual images in W3W3, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_resid_w4 apflux_resid_w4 array<float> COMMENT "Aperture fluxes on the co-added residual images in W4W4, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_w1 apflux_ivar_w1 array<float> COMMENT "Inverse variance of apflux_resid_w1, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_w2 apflux_ivar_w2 array<float> COMMENT "Inverse variance of apflux_resid_w2, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_w3 apflux_ivar_w3 array<float> COMMENT "Inverse variance of apflux_resid_w3, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE apflux_ivar_w4 apflux_ivar_w4 array<float> COMMENT "Inverse variance of apflux_resid_w4, masked by invvar=0invvar=0" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_g mw_transmission_g float COMMENT "Galactic transmission in gg filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_r mw_transmission_r float COMMENT "Galactic transmission in rr filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_z mw_transmission_z float COMMENT "Galactic transmission in zz filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_w1 mw_transmission_w1 float COMMENT "Galactic transmission in W1W1 filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_w2 mw_transmission_w2 float COMMENT "Galactic transmission in W2W2 filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_w3 mw_transmission_w3 float COMMENT "Galactic transmission in W3W3 filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE mw_transmission_w4 mw_transmission_w4 float COMMENT "Galactic transmission in W4W4 filter in linear units [0, 1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_g nobs_g smallint COMMENT "Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_r nobs_r smallint COMMENT "Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_z nobs_z smallint COMMENT "Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_w1 nobs_w1 smallint COMMENT "Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_w2 nobs_w2 smallint COMMENT "Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_w3 nobs_w3 smallint COMMENT "Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nobs_w4 nobs_w4 smallint COMMENT "Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_g rchisq_g float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in gg" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_r rchisq_r float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in rr" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_z rchisq_z float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in zz" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_w1 rchisq_w1 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W1W1" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_w2 rchisq_w2 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W2W2" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_w3 rchisq_w3 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W3W3" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE rchisq_w4 rchisq_w4 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W4W4" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_g fracflux_g float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in gg (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_r fracflux_r float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in rr (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_z fracflux_z float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in zz (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_w1 fracflux_w1 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1 (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_w2 fracflux_w2 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2 (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_w3 fracflux_w3 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3 (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracflux_w4 fracflux_w4 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4 (typically [0,1])" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracmasked_g fracmasked_g float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracmasked_r fracmasked_r float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracmasked_z fracmasked_z float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracin_g fracin_g float COMMENT "Fraction of a source's flux within the blob in gg, near unity for real sources" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracin_r fracin_r float COMMENT "Fraction of a source's flux within the blob in rr, near unity for real sources" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE fracin_z fracin_z float COMMENT "Fraction of a source's flux within the blob in zz, near unity for real sources" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE anymask_g anymask_g smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE anymask_r anymask_r smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE anymask_z anymask_z smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE allmask_g allmask_g smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE allmask_r allmask_r smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE allmask_z allmask_z smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE wisemask_w1 wisemask_w1 tinyint COMMENT "W1 bitmask as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE wisemask_w2 wisemask_w2 tinyint COMMENT "W2 bitmask as cataloged on the DR9 bitmasks page" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfsize_g psfsize_g float COMMENT "Weighted average PSF FWHM in the gg band" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfsize_r psfsize_r float COMMENT "Weighted average PSF FWHM in the rr band" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfsize_z psfsize_z float COMMENT "Weighted average PSF FWHM in the zz band" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_g psfdepth_g float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_r psfdepth_r float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_z psfdepth_z float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE galdepth_g galdepth_g float COMMENT "As for psfdepth_g but for a galaxy (0.45\u0022 exp, round) detection sensitivity" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE galdepth_r galdepth_r float COMMENT "As for psfdepth_r but for a galaxy (0.45\u0022 exp, round) detection sensitivity" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE galdepth_z galdepth_z float COMMENT "As for psfdepth_z but for a galaxy (0.45\u0022 exp, round) detection sensitivity" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nea_g nea_g float COMMENT "Noise equivalent area in gg." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nea_r nea_r float COMMENT "Noise equivalent area in rr." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE nea_z nea_z float COMMENT "Noise equivalent area in zz." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE blob_nea_g blob_nea_g float COMMENT "Blob-masked noise equivalent area in gg." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE blob_nea_r blob_nea_r float COMMENT "Blob-masked noise equivalent area in rr." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE blob_nea_z blob_nea_z float COMMENT "Blob-masked noise equivalent area in zz." ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_w1 psfdepth_w1 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W1" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_w2 psfdepth_w2 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W2" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_w3 psfdepth_w3 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W3" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE psfdepth_w4 psfdepth_w4 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W4" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE wise_coadd_id wise_coadd_id string COMMENT "unWISE coadd brick name (corresponding to the, e.g., legacysurvey-<brick>-image-W1.fits.fz coadd file) for the center of each object" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE wise_x wise_x float COMMENT "X position of coordinates in the brick image stack that corresponds to wise_coadd_id (see the DR9 updates page for transformations between wise_x and bx)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE wise_y wise_y float COMMENT "Y position of coordinates in the brick image stack that corresponds to wise_coadd_id (see the DR9 updates page for transformations between wise_y and by)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_flux_w1 lc_flux_w1 array<float> COMMENT "flux_w1 in each of up to fifteen unWISE coadd epochs (AB system\u003b defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_flux_w2 lc_flux_w2 array<float> COMMENT "flux_w2 in each of up to fifteen unWISE coadd epochs (AB\u003b defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_flux_ivar_w1 lc_flux_ivar_w1 array<float> COMMENT "Inverse variance of lc_flux_w1 (AB system\u003b defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_flux_ivar_w2 lc_flux_ivar_w2 array<float> COMMENT "Inverse variance of lc_flux_w2 (AB\u003b defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_nobs_w1 lc_nobs_w1 array<smallint> COMMENT "nobs_w1 in each of up to fifteen unWISE coadd epochs" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_nobs_w2 lc_nobs_w2 array<smallint> COMMENT "nobs_w2 in each of up to fifteen unWISE coadd epochs" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_fracflux_w1 lc_fracflux_w1 array<float> COMMENT "fracflux_w1 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_fracflux_w2 lc_fracflux_w2 array<float> COMMENT "fracflux_w2 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_rchisq_w1 lc_rchisq_w1 array<float> COMMENT "rchisq_w1 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_rchisq_w2 lc_rchisq_w2 array<float> COMMENT "rchisq_w2 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_mjd_w1 lc_mjd_w1 array<double> COMMENT "mjd_w1 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_mjd_w2 lc_mjd_w2 array<double> COMMENT "mjd_w2 in each of up to fifteen unWISE coadd epochs (defaults to zero for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_epoch_index_w1 lc_epoch_index_w1 array<smallint> COMMENT "Index number of unWISE epoch for W1 (defaults to -1 for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE lc_epoch_index_w2 lc_epoch_index_w2 array<smallint> COMMENT "Index number of unWISE epoch for W2 (defaults to -1 for unused entries)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE sersic sersic float COMMENT "Power-law index for the Sersic profile model (type=\u0022SER\u0022)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE sersic_ivar sersic_ivar float COMMENT "Inverse variance of sersic" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_r shape_r float COMMENT "Half-light radius of galaxy model for galaxy type type (>0)" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_r_ivar shape_r_ivar float COMMENT "Inverse variance of shape_r" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_e1 shape_e1 float COMMENT "Ellipticity component 1 of galaxy model for galaxy type type" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_e1_ivar shape_e1_ivar float COMMENT "Inverse variance of shape_e1" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_e2 shape_e2 float COMMENT "Ellipticity component 2 of galaxy model for galaxy type type" ;<br />
ALTER TABLE legacy_survey_dr9 CHANGE shape_e2_ivar shape_e2_ivar float COMMENT "Inverse variance of shape_e2" ;<br />
<br />
== Sweep photo-z ==<br />
<br />
=== Raw data ===<br />
<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/north/sweep/<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/south/sweep/<br />
<br />
Checksums checked:<br />
<br />
$ cd north/sweep/8.0<br />
$ ls | wc -l<br />
287<br />
$ wc -l legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
286 (+ checksum file)<br />
$ sha256sum -c legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
OK<br />
<br />
$ cd north/sweep/8.0-photo-z<br />
$ ls | wc -l<br />
287<br />
$ wc -l legacysurvey_dr8_north_sweep_8.0-photo-z.sha256sum<br />
286 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_north_sweep_8.0-photo-z.sha256sum<br />
OK<br />
<br />
$ cd south/sweep/8.0<br />
$ ls | wc -l<br />
438<br />
$ wc -l legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
437 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
OK<br />
<br />
$ cd south/sweep/8.0-photo-z<br />
$ ls | wc -l<br />
438<br />
$ wc -l legacysurvey_dr8_south_sweep_8.0-photo-z.sha256sum<br />
437 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_south_sweep_8.0-photo-z.sha256sum<br />
OK<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to merge fits files and convert to parquet:<br />
<br />
path_base = '/cephfs/pic.es/astro/scratch/tallada/dr8-phz/'<br />
hdfs_base = '/user/tallada/data/legacysurvey-sweep'<br />
<br />
hs = ['south/sweep'] #, 'north/sweep']<br />
dir_src = '8.0'<br />
dir_phz = '8.0-photo-z'<br />
dir_dst = '8.0-merged'<br />
<br />
my_files = sorted(os.listdir(os.path.join(path_base, hs[0], dir_src)))<br />
<br />
def merge_phz(file_name):<br />
file_src = os.path.join(path_base, hs[0], dir_src, file_name)<br />
file_phz = os.path.join(path_base, hs[0], dir_phz, file_name[:-5]+'-pz.fits')<br />
file_dst = os.path.join(path_base, hs[0], dir_dst, file_name[:-5]+'-merged.fits')<br />
<br />
t_src = table.Table.read(file_src)<br />
t_phz = table.Table.read(file_phz)<br />
t_dst = table.hstack([t_src, t_phz], join_type='exact')<br />
<br />
os.unlink(file_dst)<br />
t_dst.write(file_dst)<br />
<br />
return file_name<br />
<br />
df = sc.parallelize(my_files, len(my_files))<br />
<br />
df.map(merge_phz).count()<br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + os.path.join(path_base, hs[0], dir_dst)<br />
) <br />
<br />
df.write.parquet(os.path.join(hdfs_base, hs[0]))<br />
<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE tallada.dr8_sweep (<br />
`RELEASE` SMALLINT,<br />
`BRICKID` INT,<br />
`BRICKNAME` STRING,<br />
`OBJID` INT,<br />
`TYPE` STRING,<br />
`RA` DOUBLE,<br />
`DEC` DOUBLE,<br />
`RA_IVAR` FLOAT,<br />
`DEC_IVAR` FLOAT,<br />
`DCHISQ` ARRAY<FLOAT>,<br />
`EBV` FLOAT,<br />
`FLUX_G` FLOAT,<br />
`FLUX_R` FLOAT,<br />
`FLUX_Z` FLOAT,<br />
`FLUX_W1` FLOAT,<br />
`FLUX_W2` FLOAT,<br />
`FLUX_W3` FLOAT,<br />
`FLUX_W4` FLOAT,<br />
`FLUX_IVAR_G` FLOAT,<br />
`FLUX_IVAR_R` FLOAT,<br />
`FLUX_IVAR_Z` FLOAT,<br />
`FLUX_IVAR_W1` FLOAT,<br />
`FLUX_IVAR_W2` FLOAT,<br />
`FLUX_IVAR_W3` FLOAT,<br />
`FLUX_IVAR_W4` FLOAT,<br />
`MW_TRANSMISSION_G` FLOAT,<br />
`MW_TRANSMISSION_R` FLOAT,<br />
`MW_TRANSMISSION_Z` FLOAT,<br />
`MW_TRANSMISSION_W1` FLOAT,<br />
`MW_TRANSMISSION_W2` FLOAT,<br />
`MW_TRANSMISSION_W3` FLOAT,<br />
`MW_TRANSMISSION_W4` FLOAT,<br />
`NOBS_G` INT,<br />
`NOBS_R` INT,<br />
`NOBS_Z` INT,<br />
`NOBS_W1` INT,<br />
`NOBS_W2` INT,<br />
`NOBS_W3` INT,<br />
`NOBS_W4` INT,<br />
`RCHISQ_G` FLOAT,<br />
`RCHISQ_R` FLOAT,<br />
`RCHISQ_Z` FLOAT,<br />
`RCHISQ_W1` FLOAT,<br />
`RCHISQ_W2` FLOAT,<br />
`RCHISQ_W3` FLOAT,<br />
`RCHISQ_W4` FLOAT,<br />
`FRACFLUX_G` FLOAT,<br />
`FRACFLUX_R` FLOAT,<br />
`FRACFLUX_Z` FLOAT,<br />
`FRACFLUX_W1` FLOAT,<br />
`FRACFLUX_W2` FLOAT,<br />
`FRACFLUX_W3` FLOAT,<br />
`FRACFLUX_W4` FLOAT,<br />
`FRACMASKED_G` FLOAT,<br />
`FRACMASKED_R` FLOAT,<br />
`FRACMASKED_Z` FLOAT,<br />
`FRACIN_G` FLOAT,<br />
`FRACIN_R` FLOAT,<br />
`FRACIN_Z` FLOAT,<br />
`ANYMASK_G` SMALLINT,<br />
`ANYMASK_R` SMALLINT,<br />
`ANYMASK_Z` SMALLINT,<br />
`ALLMASK_G` SMALLINT,<br />
`ALLMASK_R` SMALLINT,<br />
`ALLMASK_Z` SMALLINT,<br />
`WISEMASK_W1` TINYINT,<br />
`WISEMASK_W2` TINYINT,<br />
`PSFSIZE_G` FLOAT,<br />
`PSFSIZE_R` FLOAT,<br />
`PSFSIZE_Z` FLOAT,<br />
`PSFDEPTH_G` FLOAT,<br />
`PSFDEPTH_R` FLOAT,<br />
`PSFDEPTH_Z` FLOAT,<br />
`GALDEPTH_G` FLOAT,<br />
`GALDEPTH_R` FLOAT,<br />
`GALDEPTH_Z` FLOAT,<br />
`PSFDEPTH_W1` FLOAT,<br />
`PSFDEPTH_W2` FLOAT,<br />
`WISE_COADD_ID` STRING,<br />
`FRACDEV` FLOAT,<br />
`FRACDEV_IVAR` FLOAT,<br />
`SHAPEDEV_R` FLOAT,<br />
`SHAPEDEV_R_IVAR` FLOAT,<br />
`SHAPEDEV_E1` FLOAT,<br />
`SHAPEDEV_E1_IVAR` FLOAT,<br />
`SHAPEDEV_E2` FLOAT,<br />
`SHAPEDEV_E2_IVAR` FLOAT,<br />
`SHAPEEXP_R` FLOAT,<br />
`SHAPEEXP_R_IVAR` FLOAT,<br />
`SHAPEEXP_E1` FLOAT,<br />
`SHAPEEXP_E1_IVAR` FLOAT,<br />
`SHAPEEXP_E2` FLOAT,<br />
`SHAPEEXP_E2_IVAR` FLOAT,<br />
`FIBERFLUX_G` FLOAT,<br />
`FIBERFLUX_R` FLOAT,<br />
`FIBERFLUX_Z` FLOAT,<br />
`FIBERTOTFLUX_G` FLOAT,<br />
`FIBERTOTFLUX_R` FLOAT,<br />
`FIBERTOTFLUX_Z` FLOAT,<br />
`REF_CAT` STRING,<br />
`REF_ID` BIGINT,<br />
`REF_EPOCH` FLOAT,<br />
`GAIA_PHOT_G_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_G_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_ASTROMETRIC_EXCESS_NOISE` FLOAT,<br />
`GAIA_DUPLICATED_SOURCE` BOOLEAN,<br />
`GAIA_PHOT_BP_RP_EXCESS_FACTOR` FLOAT,<br />
`GAIA_ASTROMETRIC_SIGMA5D_MAX` FLOAT,<br />
`GAIA_ASTROMETRIC_PARAMS_SOLVED` TINYINT,<br />
`PARALLAX` FLOAT,<br />
`PARALLAX_IVAR` FLOAT,<br />
`PMRA` FLOAT,<br />
`PMRA_IVAR` FLOAT,<br />
`PMDEC` FLOAT,<br />
`PMDEC_IVAR` FLOAT,<br />
`MASKBITS` SMALLINT,<br />
`z_phot_mean` FLOAT,<br />
`z_phot_median` FLOAT,<br />
`z_phot_std` FLOAT,<br />
`z_phot_l68` FLOAT,<br />
`z_phot_u68` FLOAT,<br />
`z_phot_l95` FLOAT,<br />
`z_phot_u95` FLOAT,<br />
`z_spec` FLOAT,<br />
`survey` STRING,<br />
`training` BOOLEAN<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey-sweep'<br />
;<br />
<br />
=== Combined table ===<br />
<br />
CREATE TABLE cosmohub.legacysurvey_dr8_phz (<br />
release smallint COMMENT 'Unique integer denoting the camera and filter set used',<br />
brickid int COMMENT 'Brick ID [1,662174]',<br />
brickname string COMMENT 'Name of brick, encoding the brick sky position, eg "1126p222" near RA=112.6, Dec=+22.2',<br />
objid int COMMENT 'Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick',<br />
brick_primary boolean COMMENT 'True if the object is within the brick boundary',<br />
brightblob smallint COMMENT 'bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page',<br />
maskbits smallint COMMENT 'bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits* maps, as cataloged on the DR8 bitmasks page',<br />
type string COMMENT 'Morphological model: "PSF"=stellar, "REX"="round exponential galaxy", "DEV"=deVauc, "EXP"=exponential, "COMP"=composite, "DUP"=Gaia source fit by different model. Note that in some FITS readers, a trailing space may be appended for "PSF ", "DEV " and "EXP " since the column data type is a 4-character string',<br />
ra double COMMENT 'Right ascension at equinox J2000',<br />
dec double COMMENT 'Declination at equinox J2000',<br />
ra_ivar float COMMENT 'Inverse variance of RA (no cosine term!), excluding astrometric calibration errors',<br />
dec_ivar float COMMENT 'Inverse variance of DEC, excluding astrometric calibration errors',<br />
bx float COMMENT 'X position (0-indexed) of coordinates in brick image stack',<br />
`by` float COMMENT 'Y position (0-indexed) of coordinates in brick image stack',<br />
dchisq array<float> COMMENT 'Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.',<br />
ebv float COMMENT 'Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns',<br />
mjd_min double COMMENT 'Minimum Modified Julian Date of observations used to construct the model of this object',<br />
mjd_max double COMMENT 'Maximum Modified Julian Date of observations used to construct the model of this object',<br />
ref_cat string COMMENT 'Reference catalog source for this star: "T2" for Tycho-2, "G2" for Gaia DR2, "L2" for the LSLGA, empty otherwise',<br />
ref_id bigint COMMENT 'Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b "sourceid" for Gaia-DR2 and LSLGA',<br />
pmra float COMMENT 'Reference catalog proper motion in the RA direction',<br />
pmdec float COMMENT 'Reference catalog proper motion in the Dec direction',<br />
parallax float COMMENT 'Reference catalog parallax',<br />
pmra_ivar float COMMENT 'Reference catalog inverse-variance on pmra',<br />
pmdec_ivar float COMMENT 'Reference catalog inverse-variance on pmdec',<br />
parallax_ivar float COMMENT 'Reference catalog inverse-variance on parallax',<br />
ref_epoch float COMMENT 'Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)',<br />
gaia_pointsource boolean COMMENT 'This Gaia DR2 source is believed to be a star, not a galaxy',<br />
gaia_phot_g_mean_mag float COMMENT 'Gaia G band mag',<br />
gaia_phot_g_mean_flux_over_error float COMMENT 'Gaia G band signal-to-noise',<br />
gaia_phot_g_n_obs smallint COMMENT 'Gaia G band number of observations',<br />
gaia_phot_bp_mean_mag float COMMENT 'Gaia BP mag',<br />
gaia_phot_bp_mean_flux_over_error float COMMENT 'Gaia BP signal-to-noise',<br />
gaia_phot_bp_n_obs smallint COMMENT 'Gaia BP number of observations',<br />
gaia_phot_rp_mean_mag float COMMENT 'Gaia RP mag',<br />
gaia_phot_rp_mean_flux_over_error float COMMENT 'Gaia RP signal-to-noise',<br />
gaia_phot_rp_n_obs smallint COMMENT 'Gaia RP number of observations',<br />
gaia_phot_variable_flag boolean COMMENT 'Gaia photometric variable flag',<br />
gaia_astrometric_excess_noise float COMMENT 'Gaia astrometric excess noise',<br />
gaia_astrometric_excess_noise_sig float COMMENT 'Gaia astrometric excess noise uncertainty',<br />
gaia_astrometric_n_obs_al smallint COMMENT 'Gaia number of astrometric observations along scan direction',<br />
gaia_astrometric_n_good_obs_al smallint COMMENT 'Gaia number of good astrometric observations along scan direction',<br />
gaia_astrometric_weight_al float COMMENT 'Gaia astrometric weight along scan direction',<br />
gaia_duplicated_source boolean COMMENT 'Gaia duplicated source flag',<br />
gaia_a_g_val float COMMENT 'Gaia line-of-sight extinction in the G band',<br />
gaia_e_bp_min_rp_val float COMMENT 'Gaia line-of-sight reddening E(BP-RP)',<br />
gaia_phot_bp_rp_excess_factor float COMMENT 'Gaia BP/RP excess factor',<br />
gaia_astrometric_sigma5d_max float COMMENT 'Gaia longest semi-major axis of the 5-d error ellipsoid',<br />
gaia_astrometric_params_solved tinyint COMMENT 'which astrometric parameters were estimated for a Gaia source',<br />
flux_g float COMMENT 'model flux in gg',<br />
flux_r float COMMENT 'model flux in rr',<br />
flux_z float COMMENT 'model flux in zz',<br />
flux_w1 float COMMENT 'WISE model flux in W1W1 (AB system)',<br />
flux_w2 float COMMENT 'WISE model flux in W2W2 (AB)',<br />
flux_w3 float COMMENT 'WISE model flux in W3W3 (AB)',<br />
flux_w4 float COMMENT 'WISE model flux in W4W4 (AB)',<br />
flux_ivar_g float COMMENT 'Inverse variance of flux_g',<br />
flux_ivar_r float COMMENT 'Inverse variance of flux_r',<br />
flux_ivar_z float COMMENT 'Inverse variance of flux_z',<br />
flux_ivar_w1 float COMMENT 'Inverse variance of flux_w1 (AB system)',<br />
flux_ivar_w2 float COMMENT 'Inverse variance of flux_w2 (AB)',<br />
flux_ivar_w3 float COMMENT 'Inverse variance of flux_w3 (AB)',<br />
flux_ivar_w4 float COMMENT 'Inverse variance of flux_w4 (AB)',<br />
fiberflux_g float COMMENT 'Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_r float COMMENT 'Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_z float COMMENT 'Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fibertotflux_g float COMMENT 'Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_r float COMMENT 'Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_z float COMMENT 'Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
apflux_g array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg',<br />
apflux_r array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr',<br />
apflux_z array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz',<br />
apflux_resid_g array<float> COMMENT 'aperture fluxes on the co-added residual images in gg',<br />
apflux_resid_r array<float> COMMENT 'aperture fluxes on the co-added residual images in rr',<br />
apflux_resid_z array<float> COMMENT 'aperture fluxes on the co-added residual images in zz',<br />
apflux_ivar_g array<float> COMMENT 'Inverse variance of apflux_resid_g',<br />
apflux_ivar_r array<float> COMMENT 'Inverse variance of apflux_resid_r',<br />
apflux_ivar_z array<float> COMMENT 'Inverse variance of apflux_resid_z',<br />
mw_transmission_g float COMMENT 'Galactic transmission in gg filter in linear units [0, 1]',<br />
mw_transmission_r float COMMENT 'Galactic transmission in rr filter in linear units [0, 1]',<br />
mw_transmission_z float COMMENT 'Galactic transmission in zz filter in linear units [0, 1]',<br />
mw_transmission_w1 float COMMENT 'Galactic transmission in W1W1 filter in linear units [0, 1]',<br />
mw_transmission_w2 float COMMENT 'Galactic transmission in W2W2 filter in linear units [0, 1]',<br />
mw_transmission_w3 float COMMENT 'Galactic transmission in W3W3 filter in linear units [0, 1]',<br />
mw_transmission_w4 float COMMENT 'Galactic transmission in W4W4 filter in linear units [0, 1]',<br />
nobs_g smallint COMMENT 'Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)',<br />
nobs_r smallint COMMENT 'Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)',<br />
nobs_z smallint COMMENT 'Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)',<br />
nobs_w1 smallint COMMENT 'Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)',<br />
nobs_w2 smallint COMMENT 'Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)',<br />
nobs_w3 smallint COMMENT 'Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)',<br />
nobs_w4 smallint COMMENT 'Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)',<br />
rchisq_g float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in gg',<br />
rchisq_r float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in rr',<br />
rchisq_z float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in zz',<br />
rchisq_w1 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W1W1',<br />
rchisq_w2 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W2W2',<br />
rchisq_w3 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W3W3',<br />
rchisq_w4 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W4W4',<br />
fracflux_g float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in gg (typically [0,1])',<br />
fracflux_r float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in rr (typically [0,1])',<br />
fracflux_z float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in zz (typically [0,1])',<br />
fracflux_w1 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1 (typically [0,1])',<br />
fracflux_w2 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2 (typically [0,1])',<br />
fracflux_w3 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3 (typically [0,1])',<br />
fracflux_w4 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4 (typically [0,1])',<br />
fracmasked_g float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]',<br />
fracmasked_r float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]',<br />
fracmasked_z float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]',<br />
fracin_g float COMMENT 'Fraction of a source\'s flux within the blob in gg, near unity for real sources',<br />
fracin_r float COMMENT 'Fraction of a source\'s flux within the blob in rr, near unity for real sources',<br />
fracin_z float COMMENT 'Fraction of a source\'s flux within the blob in zz, near unity for real sources',<br />
anymask_g smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page',<br />
anymask_r smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page',<br />
anymask_z smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page',<br />
allmask_g smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page',<br />
allmask_r smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page',<br />
allmask_z smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page',<br />
wisemask_w1 tinyint COMMENT 'W1 bitmask as cataloged on the DR8 bitmasks page',<br />
wisemask_w2 tinyint COMMENT 'W2 bitmask as cataloged on the DR8 bitmasks page',<br />
psfsize_g float COMMENT 'Weighted average PSF FWHM in the gg band',<br />
psfsize_r float COMMENT 'Weighted average PSF FWHM in the rr band',<br />
psfsize_z float COMMENT 'Weighted average PSF FWHM in the zz band',<br />
psfdepth_g float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude',<br />
psfdepth_r float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude',<br />
psfdepth_z float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude',<br />
galdepth_g float COMMENT 'As for psfdepth_g but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_r float COMMENT 'As for psfdepth_r but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_z float COMMENT 'As for psfdepth_z but for a galaxy (0.45" exp, round) detection sensitivity',<br />
psfdepth_w1 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W1',<br />
psfdepth_w2 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W2',<br />
psfdepth_w3 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W3',<br />
psfdepth_w4 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W4',<br />
wise_coadd_id string COMMENT 'unWISE coadd file name for the center of each object',<br />
lc_flux_w1 array<float> COMMENT 'flux_w1 in each of up to eleven unWISE coadd epochs (AB system)',<br />
lc_flux_w2 array<float> COMMENT 'flux_w2 in each of up to eleven unWISE coadd epochs (AB)',<br />
lc_flux_ivar_w1 array<float> COMMENT 'Inverse variance of lc_flux_w1 (AB system)',<br />
lc_flux_ivar_w2 array<float> COMMENT 'Inverse variance of lc_flux_w2 (AB)',<br />
lc_nobs_w1 array<smallint> COMMENT 'nobs_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_nobs_w2 array<smallint> COMMENT 'nobs_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w1 array<float> COMMENT 'fracflux_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w2 array<float> COMMENT 'fracflux_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w1 array<float> COMMENT 'rchisq_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w2 array<float> COMMENT 'rchisq_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w1 array<double> COMMENT 'mjd_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w2 array<double> COMMENT 'mjd_w2 in each of up to eleven unWISE coadd epochs',<br />
fracdev float COMMENT 'Fraction of model in deVauc [0,1]',<br />
fracdev_ivar float COMMENT 'Inverse variance of fracdev',<br />
shapeexp_r float COMMENT 'Half-light radius of exponential model (>0)',<br />
shapeexp_r_ivar float COMMENT 'Inverse variance of shapeexp_r',<br />
shapeexp_e1 float COMMENT 'Ellipticity component 1',<br />
shapeexp_e1_ivar float COMMENT 'Inverse variance of shapeexp_e1',<br />
shapeexp_e2 float COMMENT 'Ellipticity component 2',<br />
shapeexp_e2_ivar float COMMENT 'Inverse variance of shapeexp_e2',<br />
shapedev_r float COMMENT 'Half-light radius of deVaucouleurs model (>0)',<br />
shapedev_r_ivar float COMMENT 'Inverse variance of shapedev_r',<br />
shapedev_e1 float COMMENT 'Ellipticity component 1',<br />
shapedev_e1_ivar float COMMENT 'Inverse variance of shapedev_e1',<br />
shapedev_e2 float COMMENT 'Ellipticity component 2',<br />
shapedev_e2_ivar float COMMENT 'Inverse variance of shapedev_e2',<br />
z_phot_mean FLOAT COMMENT 'photo-z derived from the mean of the photo-z PDF ',<br />
z_phot_median FLOAT COMMENT 'photo-z derived from the median of the photo-z PDF',<br />
z_phot_std FLOAT COMMENT 'standard deviation of the photo-z\'s derived from the photo-z PDF',<br />
z_phot_l68 FLOAT COMMENT 'lower bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_u68 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_l95 FLOAT COMMENT 'lower bound of the 95% confidence region, derived from the photo-z PDF',<br />
z_phot_u95 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_spec FLOAT COMMENT 'spectroscopic redshift, if available',<br />
survey STRING COMMENT 'source of the spectroscopic redshift',<br />
training BOOLEAN COMMENT 'whether or not the spectroscopic redshift is used in photometric redshift training'<br />
)<br />
CLUSTERED BY (<br />
release,<br />
brickid,<br />
objid)<br />
SORTED BY (<br />
release ASC,<br />
brickid ASC,<br />
objid ASC)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
=== Join insert ===<br />
<br />
INSERT OVERWRITE TABLE cosmohub.legacy_survey_dr8_phz<br />
SELECT<br />
dr8.*,<br />
phz.z_phot_mean,<br />
phz.z_phot_median,<br />
phz.z_phot_std,<br />
phz.z_phot_l68,<br />
phz.z_phot_u68,<br />
phz.z_phot_l95,<br />
phz.z_phot_u95,<br />
phz.z_spec,<br />
phz.survey,<br />
phz.training<br />
FROM cosmohub.legacy_survey_dr8 AS dr8<br />
LEFT JOIN tallada.dr8_sweep AS phz<br />
ON dr8.release = phz.release<br />
AND dr8.brickid = phz.brickid<br />
AND dr8.objid = phz.objid<br />
;<br />
<br />
== Queries for checking ==<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT release, brickid, objid, COUNT(release)<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
HAVING COUNT(release) > 1<br />
<br />
DR8 release brickid objid: DISTINCT(1.694.230.368) TOTAL(1.694.230.368)<br />
DR8 brickid objid: DISTINCT(1.654.429.115) TOTAL(1.694.230.368)<br />
<br />
PHZ release brickid objid: DISTINCT(1.646.664.729) TOTAL(1.646.664.897)<br />
PHZ brickid objid: DISTINCT(1.608.841.582) TOTAL(1.646.664.897)<br />
<br />
DR8: 1.694.230.368<br />
PHZ: 1.646.664.897<br />
----------------------------------------------------------------------<br />
-47.565.471</div>
Tallada
https://pwiki.pic.es/index.php?title=Legacy_Survey_DR9&diff=876
Legacy Survey DR9
2020-11-28T02:11:58Z
<p>Tallada: /* Internal Table */</p>
<hr />
<div>== Original DR9 ==<br />
=== Raw data ===<br />
<br />
Downloaded from NERSC<br />
/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m<br />
<br />
Directories:<br />
south/tractor<br />
north/tractor<br />
<br />
Downloaded using bbcp from services01.euclid into CephFS scratch:<br />
bbcp -s 16 -a -A -v -z -r -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" \<br />
"jcarrete@dtn01.nersc.gov:/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m/north/tractor" \<br />
/cephfs/pic.es/astro/scratch/tallada/dr9m/north/tractor<br />
bbcp -s 16 -a -A -v -z -r -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" \<br />
"jcarrete@dtn01.nersc.gov:/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m/south/tractor" \<br />
/cephfs/pic.es/astro/scratch/tallada/dr9m/south/tractor<br />
<br />
Checksums provided through sha256sum files, and checked using command:<br />
find * -type d | xargs -t -L1 -P24 sh -c 'cd $1; sha256sum --quiet -c *.sha256sum' cksum<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to convert fits files to parquet:<br />
<br />
path_base = '/cephfs/pic.es/astro/scratch/tallada/dr9m'<br />
hdfs_base = '/user/tallada/data/legacysurvey_dr9'<br />
<br />
hs = ['h=north', 'h=south']<br />
rs = ['r={0:03d}'.format(x) for x in range(360)]<br />
<br />
for h in hs:<br />
for r in rs:<br />
p = os.path.join(path_base, h, r)<br />
if not os.path.exists(p):<br />
continue<br />
<br />
size = sum(os.path.getsize(os.path.join(p,f)) for f in os.listdir(p) if os.path.isfile(os.path.join(p,f)))<br />
<br />
print h, r, size <br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + p<br />
)<br />
<br />
df = df.repartition(int(math.ceil(size / (256*1024*1024.0))))<br />
<br />
df.write.parquet(os.path.join(hdfs_base, h, r))<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE `tallada`.`legacysurvey_dr9`(<br />
`release` smallint, <br />
`brickid` int, <br />
`brickname` string, <br />
`objid` int, <br />
`brick_primary` boolean, <br />
`maskbits` smallint, <br />
`fitbits` smallint, <br />
`type` string, <br />
`ra` double, <br />
`dec` double, <br />
`ra_ivar` float, <br />
`dec_ivar` float, <br />
`bx` float, <br />
`by` float, <br />
`dchisq` array<float>, <br />
`ebv` float, <br />
`mjd_min` double, <br />
`mjd_max` double, <br />
`ref_cat` string, <br />
`ref_id` bigint, <br />
`pmra` float, <br />
`pmdec` float, <br />
`parallax` float, <br />
`pmra_ivar` float, <br />
`pmdec_ivar` float, <br />
`parallax_ivar` float, <br />
`ref_epoch` float, <br />
`gaia_phot_g_mean_mag` float, <br />
`gaia_phot_g_mean_flux_over_error` float, <br />
`gaia_phot_g_n_obs` smallint, <br />
`gaia_phot_bp_mean_mag` float, <br />
`gaia_phot_bp_mean_flux_over_error` float, <br />
`gaia_phot_bp_n_obs` smallint, <br />
`gaia_phot_rp_mean_mag` float, <br />
`gaia_phot_rp_mean_flux_over_error` float, <br />
`gaia_phot_rp_n_obs` smallint, <br />
`gaia_phot_variable_flag` boolean, <br />
`gaia_astrometric_excess_noise` float, <br />
`gaia_astrometric_excess_noise_sig` float, <br />
`gaia_astrometric_n_obs_al` smallint, <br />
`gaia_astrometric_n_good_obs_al` smallint, <br />
`gaia_astrometric_weight_al` float, <br />
`gaia_duplicated_source` boolean, <br />
`gaia_a_g_val` float, <br />
`gaia_e_bp_min_rp_val` float, <br />
`gaia_phot_bp_rp_excess_factor` float, <br />
`gaia_astrometric_sigma5d_max` float, <br />
`gaia_astrometric_params_solved` tinyint, <br />
`flux_g` float, <br />
`flux_r` float, <br />
`flux_z` float, <br />
`flux_w1` float, <br />
`flux_w2` float, <br />
`flux_w3` float, <br />
`flux_w4` float, <br />
`flux_ivar_g` float, <br />
`flux_ivar_r` float, <br />
`flux_ivar_z` float, <br />
`flux_ivar_w1` float, <br />
`flux_ivar_w2` float, <br />
`flux_ivar_w3` float, <br />
`flux_ivar_w4` float, <br />
`fiberflux_g` float, <br />
`fiberflux_r` float, <br />
`fiberflux_z` float, <br />
`fibertotflux_g` float, <br />
`fibertotflux_r` float, <br />
`fibertotflux_z` float, <br />
`apflux_g` array<float>, <br />
`apflux_r` array<float>, <br />
`apflux_z` array<float>, <br />
`apflux_resid_g` array<float>, <br />
`apflux_resid_r` array<float>, <br />
`apflux_resid_z` array<float>, <br />
`apflux_blobresid_g` array<float>, <br />
`apflux_blobresid_r` array<float>, <br />
`apflux_blobresid_z` array<float>, <br />
`apflux_ivar_g` array<float>, <br />
`apflux_ivar_r` array<float>, <br />
`apflux_ivar_z` array<float>, <br />
`apflux_masked_g` array<float>, <br />
`apflux_masked_r` array<float>, <br />
`apflux_masked_z` array<float>, <br />
`apflux_w1` array<float>, <br />
`apflux_w2` array<float>, <br />
`apflux_w3` array<float>, <br />
`apflux_w4` array<float>, <br />
`apflux_resid_w1` array<float>, <br />
`apflux_resid_w2` array<float>, <br />
`apflux_resid_w3` array<float>, <br />
`apflux_resid_w4` array<float>, <br />
`apflux_ivar_w1` array<float>, <br />
`apflux_ivar_w2` array<float>, <br />
`apflux_ivar_w3` array<float>, <br />
`apflux_ivar_w4` array<float>, <br />
`mw_transmission_g` float, <br />
`mw_transmission_r` float, <br />
`mw_transmission_z` float, <br />
`mw_transmission_w1` float, <br />
`mw_transmission_w2` float, <br />
`mw_transmission_w3` float, <br />
`mw_transmission_w4` float, <br />
`nobs_g` smallint, <br />
`nobs_r` smallint, <br />
`nobs_z` smallint, <br />
`nobs_w1` smallint, <br />
`nobs_w2` smallint, <br />
`nobs_w3` smallint, <br />
`nobs_w4` smallint, <br />
`rchisq_g` float, <br />
`rchisq_r` float, <br />
`rchisq_z` float, <br />
`rchisq_w1` float, <br />
`rchisq_w2` float, <br />
`rchisq_w3` float, <br />
`rchisq_w4` float, <br />
`fracflux_g` float, <br />
`fracflux_r` float, <br />
`fracflux_z` float, <br />
`fracflux_w1` float, <br />
`fracflux_w2` float, <br />
`fracflux_w3` float, <br />
`fracflux_w4` float, <br />
`fracmasked_g` float, <br />
`fracmasked_r` float, <br />
`fracmasked_z` float, <br />
`fracin_g` float, <br />
`fracin_r` float, <br />
`fracin_z` float, <br />
`anymask_g` smallint, <br />
`anymask_r` smallint, <br />
`anymask_z` smallint, <br />
`allmask_g` smallint, <br />
`allmask_r` smallint, <br />
`allmask_z` smallint, <br />
`wisemask_w1` tinyint, <br />
`wisemask_w2` tinyint, <br />
`psfsize_g` float, <br />
`psfsize_r` float, <br />
`psfsize_z` float, <br />
`psfdepth_g` float, <br />
`psfdepth_r` float, <br />
`psfdepth_z` float, <br />
`galdepth_g` float, <br />
`galdepth_r` float, <br />
`galdepth_z` float, <br />
`nea_g` float, <br />
`nea_r` float, <br />
`nea_z` float, <br />
`blob_nea_g` float, <br />
`blob_nea_r` float, <br />
`blob_nea_z` float, <br />
`psfdepth_w1` float, <br />
`psfdepth_w2` float, <br />
`psfdepth_w3` float, <br />
`psfdepth_w4` float, <br />
`wise_coadd_id` string, <br />
`wise_x` float, <br />
`wise_y` float, <br />
`lc_flux_w1` array<float>, <br />
`lc_flux_w2` array<float>, <br />
`lc_flux_ivar_w1` array<float>, <br />
`lc_flux_ivar_w2` array<float>, <br />
`lc_nobs_w1` array<smallint>, <br />
`lc_nobs_w2` array<smallint>, <br />
`lc_fracflux_w1` array<float>, <br />
`lc_fracflux_w2` array<float>, <br />
`lc_rchisq_w1` array<float>, <br />
`lc_rchisq_w2` array<float>, <br />
`lc_mjd_w1` array<double>, <br />
`lc_mjd_w2` array<double>, <br />
`lc_epoch_index_w1` array<smallint>, <br />
`lc_epoch_index_w2` array<smallint>, <br />
`sersic` float, <br />
`sersic_ivar` float, <br />
`shape_r` float, <br />
`shape_r_ivar` float, <br />
`shape_e1` float, <br />
`shape_e1_ivar` float, <br />
`shape_e2` float, <br />
`shape_e2_ivar` float)<br />
PARTITIONED BY (<br />
h STRING,<br />
r STRING<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey_dr9'<br />
;<br />
<br />
=== Internal Table ===<br />
<br />
CREATE TABLE `cosmohub`.`legacy_survey_dr9`(<br />
`release` smallint,<br />
`brickid` int,<br />
`brickname` string,<br />
`objid` int,<br />
`brick_primary` boolean,<br />
`maskbits` smallint,<br />
`fitbits` smallint,<br />
`type` string,<br />
`ra` double,<br />
`dec` double,<br />
`ra_ivar` float,<br />
`dec_ivar` float,<br />
`bx` float,<br />
`by` float,<br />
`dchisq` array<float>,<br />
`ebv` float,<br />
`mjd_min` double,<br />
`mjd_max` double,<br />
`ref_cat` string,<br />
`ref_id` bigint,<br />
`pmra` float,<br />
`pmdec` float,<br />
`parallax` float,<br />
`pmra_ivar` float,<br />
`pmdec_ivar` float,<br />
`parallax_ivar` float,<br />
`ref_epoch` float,<br />
`gaia_phot_g_mean_mag` float,<br />
`gaia_phot_g_mean_flux_over_error` float,<br />
`gaia_phot_g_n_obs` smallint,<br />
`gaia_phot_bp_mean_mag` float,<br />
`gaia_phot_bp_mean_flux_over_error` float,<br />
`gaia_phot_bp_n_obs` smallint,<br />
`gaia_phot_rp_mean_mag` float,<br />
`gaia_phot_rp_mean_flux_over_error` float,<br />
`gaia_phot_rp_n_obs` smallint,<br />
`gaia_phot_variable_flag` boolean,<br />
`gaia_astrometric_excess_noise` float,<br />
`gaia_astrometric_excess_noise_sig` float,<br />
`gaia_astrometric_n_obs_al` smallint,<br />
`gaia_astrometric_n_good_obs_al` smallint,<br />
`gaia_astrometric_weight_al` float,<br />
`gaia_duplicated_source` boolean,<br />
`gaia_a_g_val` float,<br />
`gaia_e_bp_min_rp_val` float,<br />
`gaia_phot_bp_rp_excess_factor` float,<br />
`gaia_astrometric_sigma5d_max` float,<br />
`gaia_astrometric_params_solved` tinyint,<br />
`flux_g` float,<br />
`flux_r` float,<br />
`flux_z` float,<br />
`flux_w1` float,<br />
`flux_w2` float,<br />
`flux_w3` float,<br />
`flux_w4` float,<br />
`flux_ivar_g` float,<br />
`flux_ivar_r` float,<br />
`flux_ivar_z` float,<br />
`flux_ivar_w1` float,<br />
`flux_ivar_w2` float,<br />
`flux_ivar_w3` float,<br />
`flux_ivar_w4` float,<br />
`fiberflux_g` float,<br />
`fiberflux_r` float,<br />
`fiberflux_z` float,<br />
`fibertotflux_g` float,<br />
`fibertotflux_r` float,<br />
`fibertotflux_z` float,<br />
`apflux_g` array<float>,<br />
`apflux_r` array<float>,<br />
`apflux_z` array<float>,<br />
`apflux_resid_g` array<float>,<br />
`apflux_resid_r` array<float>,<br />
`apflux_resid_z` array<float>,<br />
`apflux_blobresid_g` array<float>,<br />
`apflux_blobresid_r` array<float>,<br />
`apflux_blobresid_z` array<float>,<br />
`apflux_ivar_g` array<float>,<br />
`apflux_ivar_r` array<float>,<br />
`apflux_ivar_z` array<float>,<br />
`apflux_masked_g` array<float>,<br />
`apflux_masked_r` array<float>,<br />
`apflux_masked_z` array<float>,<br />
`apflux_w1` array<float>,<br />
`apflux_w2` array<float>,<br />
`apflux_w3` array<float>,<br />
`apflux_w4` array<float>,<br />
`apflux_resid_w1` array<float>,<br />
`apflux_resid_w2` array<float>,<br />
`apflux_resid_w3` array<float>,<br />
`apflux_resid_w4` array<float>,<br />
`apflux_ivar_w1` array<float>,<br />
`apflux_ivar_w2` array<float>,<br />
`apflux_ivar_w3` array<float>,<br />
`apflux_ivar_w4` array<float>,<br />
`mw_transmission_g` float,<br />
`mw_transmission_r` float,<br />
`mw_transmission_z` float,<br />
`mw_transmission_w1` float,<br />
`mw_transmission_w2` float,<br />
`mw_transmission_w3` float,<br />
`mw_transmission_w4` float,<br />
`nobs_g` smallint,<br />
`nobs_r` smallint,<br />
`nobs_z` smallint,<br />
`nobs_w1` smallint,<br />
`nobs_w2` smallint,<br />
`nobs_w3` smallint,<br />
`nobs_w4` smallint,<br />
`rchisq_g` float,<br />
`rchisq_r` float,<br />
`rchisq_z` float,<br />
`rchisq_w1` float,<br />
`rchisq_w2` float,<br />
`rchisq_w3` float,<br />
`rchisq_w4` float,<br />
`fracflux_g` float,<br />
`fracflux_r` float,<br />
`fracflux_z` float,<br />
`fracflux_w1` float,<br />
`fracflux_w2` float,<br />
`fracflux_w3` float,<br />
`fracflux_w4` float,<br />
`fracmasked_g` float,<br />
`fracmasked_r` float,<br />
`fracmasked_z` float,<br />
`fracin_g` float,<br />
`fracin_r` float,<br />
`fracin_z` float,<br />
`anymask_g` smallint,<br />
`anymask_r` smallint,<br />
`anymask_z` smallint,<br />
`allmask_g` smallint,<br />
`allmask_r` smallint,<br />
`allmask_z` smallint,<br />
`wisemask_w1` tinyint,<br />
`wisemask_w2` tinyint,<br />
`psfsize_g` float,<br />
`psfsize_r` float,<br />
`psfsize_z` float,<br />
`psfdepth_g` float,<br />
`psfdepth_r` float,<br />
`psfdepth_z` float,<br />
`galdepth_g` float,<br />
`galdepth_r` float,<br />
`galdepth_z` float,<br />
`nea_g` float,<br />
`nea_r` float,<br />
`nea_z` float,<br />
`blob_nea_g` float,<br />
`blob_nea_r` float,<br />
`blob_nea_z` float,<br />
`psfdepth_w1` float,<br />
`psfdepth_w2` float,<br />
`psfdepth_w3` float,<br />
`psfdepth_w4` float,<br />
`wise_coadd_id` string,<br />
`wise_x` float,<br />
`wise_y` float,<br />
`lc_flux_w1` array<float>,<br />
`lc_flux_w2` array<float>,<br />
`lc_flux_ivar_w1` array<float>,<br />
`lc_flux_ivar_w2` array<float>,<br />
`lc_nobs_w1` array<smallint>,<br />
`lc_nobs_w2` array<smallint>,<br />
`lc_fracflux_w1` array<float>,<br />
`lc_fracflux_w2` array<float>,<br />
`lc_rchisq_w1` array<float>,<br />
`lc_rchisq_w2` array<float>,<br />
`lc_mjd_w1` array<double>,<br />
`lc_mjd_w2` array<double>,<br />
`lc_epoch_index_w1` array<smallint>,<br />
`lc_epoch_index_w2` array<smallint>,<br />
`sersic` float,<br />
`sersic_ivar` float,<br />
`shape_r` float,<br />
`shape_r_ivar` float,<br />
`shape_e1` float,<br />
`shape_e1_ivar` float,<br />
`shape_e2` float,<br />
`shape_e2_ivar` float)<br />
CLUSTERED BY (<br />
release, brickid, objid<br />
)<br />
SORTED BY (<br />
release, brickid, objid<br />
)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
<br />
INSERT INTO TABLE cosmohub.legacy_survey_dr9<br />
SELECT release, brickid, brickname, objid, brick_primary, maskbits, fitbits, type, ra, `dec`, ra_ivar, dec_ivar,<br />
bx, `by`, dchisq, ebv, mjd_min, mjd_max, ref_cat, ref_id, pmra, pmdec, parallax, pmra_ivar, pmdec_ivar,<br />
parallax_ivar, ref_epoch, gaia_phot_g_mean_mag, gaia_phot_g_mean_flux_over_error, gaia_phot_g_n_obs,<br />
gaia_phot_bp_mean_mag, gaia_phot_bp_mean_flux_over_error, gaia_phot_bp_n_obs, gaia_phot_rp_mean_mag,<br />
gaia_phot_rp_mean_flux_over_error, gaia_phot_rp_n_obs, gaia_phot_variable_flag, gaia_astrometric_excess_noise,<br />
gaia_astrometric_excess_noise_sig, gaia_astrometric_n_obs_al, gaia_astrometric_n_good_obs_al,<br />
gaia_astrometric_weight_al, gaia_duplicated_source, gaia_a_g_val, gaia_e_bp_min_rp_val,<br />
gaia_phot_bp_rp_excess_factor, gaia_astrometric_sigma5d_max, gaia_astrometric_params_solved, flux_g, flux_r,<br />
flux_z, flux_w1, flux_w2, flux_w3, flux_w4, flux_ivar_g, flux_ivar_r, flux_ivar_z, flux_ivar_w1, flux_ivar_w2,<br />
flux_ivar_w3, flux_ivar_w4, fiberflux_g, fiberflux_r, fiberflux_z, fibertotflux_g, fibertotflux_r, fibertotflux_z,<br />
apflux_g, apflux_r, apflux_z, apflux_resid_g, apflux_resid_r, apflux_resid_z, apflux_blobresid_g,<br />
apflux_blobresid_r, apflux_blobresid_z, apflux_ivar_g, apflux_ivar_r, apflux_ivar_z, apflux_masked_g,<br />
apflux_masked_r, apflux_masked_z, apflux_w1, apflux_w2, apflux_w3, apflux_w4, apflux_resid_w1, apflux_resid_w2,<br />
apflux_resid_w3, apflux_resid_w4, apflux_ivar_w1, apflux_ivar_w2, apflux_ivar_w3, apflux_ivar_w4,<br />
mw_transmission_g, mw_transmission_r, mw_transmission_z, mw_transmission_w1, mw_transmission_w2,<br />
mw_transmission_w3, mw_transmission_w4, nobs_g, nobs_r, nobs_z, nobs_w1, nobs_w2, nobs_w3, nobs_w4, rchisq_g,<br />
rchisq_r, rchisq_z, rchisq_w1, rchisq_w2, rchisq_w3, rchisq_w4, fracflux_g, fracflux_r, fracflux_z, fracflux_w1,<br />
fracflux_w2, fracflux_w3, fracflux_w4, fracmasked_g, fracmasked_r, fracmasked_z, fracin_g, fracin_r, fracin_z,<br />
anymask_g, anymask_r, anymask_z, allmask_g, allmask_r, allmask_z, wisemask_w1, wisemask_w2, psfsize_g, psfsize_r,<br />
psfsize_z, psfdepth_g, psfdepth_r, psfdepth_z, galdepth_g, galdepth_r, galdepth_z, nea_g, nea_r, nea_z,<br />
blob_nea_g, blob_nea_r, blob_nea_z, psfdepth_w1, psfdepth_w2, psfdepth_w3, psfdepth_w4, wise_coadd_id, wise_x,<br />
wise_y, lc_flux_w1, lc_flux_w2, lc_flux_ivar_w1, lc_flux_ivar_w2, lc_nobs_w1, lc_nobs_w2, lc_fracflux_w1,<br />
lc_fracflux_w2, lc_rchisq_w1, lc_rchisq_w2, lc_mjd_w1, lc_mjd_w2, lc_epoch_index_w1, lc_epoch_index_w2, sersic,<br />
sersic_ivar, shape_r, shape_r_ivar, shape_e1, shape_e1_ivar, shape_e2, shape_e2_ivar<br />
FROM cosmohub.legacy_survey_dr9_p<br />
;<br />
<br />
=== Comments ===<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE release release smallint COMMENT "Unique integer denoting the camera and filter set used";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickid brickid int COMMENT "Brick ID [1,662174]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickname brickname string COMMENT "Name of brick, encoding the brick sky position, eg \"1126p222\" near RA=112.6, Dec=+22.2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE objid objid int COMMENT "Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brick_primary brick_primary boolean COMMENT "True if the object is within the brick boundary";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brightblob brightblob smallint COMMENT "bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE maskbits maskbits smallint COMMENT "bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits*maps, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE type type string COMMENT "Morphological model: PSF=stellar, REX=round exponential galaxy, DEV=deVauc, EXP=exponential, COMP=composite, DUP=Gaia source fit by different model.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra ra double COMMENT "Right ascension at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec dec double COMMENT "Declination at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra_ivar ra_ivar float COMMENT "Inverse variance of RA (no cosine term!), excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec_ivar dec_ivar float COMMENT "Inverse variance of DEC, excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE bx bx float COMMENT "X position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE by by float COMMENT "Y position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dchisq dchisq array<float> COMMENT "Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ebv ebv float COMMENT "Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_min mjd_min double COMMENT "Minimum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_max mjd_max double COMMENT "Maximum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_cat ref_cat string COMMENT "Reference catalog source for this star: T2 for Tycho-2, G2 for Gaia DR2, L2 for the LSLGA, empty otherwise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_id ref_id bigint COMMENT "Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b \"sourceid\" for Gaia-DR2 and LSLGA";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra pmra float COMMENT "Reference catalog proper motion in the RA direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec pmdec float COMMENT "Reference catalog proper motion in the Dec direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax parallax float COMMENT "Reference catalog parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra_ivar pmra_ivar float COMMENT "Reference catalog inverse-variance on pmra";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec_ivar pmdec_ivar float COMMENT "Reference catalog inverse-variance on pmdec";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax_ivar parallax_ivar float COMMENT "Reference catalog inverse-variance on parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_epoch ref_epoch float COMMENT "Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_pointsource gaia_pointsource boolean COMMENT "This Gaia DR2 source is believed to be a star, not a galaxy";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_mag gaia_phot_g_mean_mag float COMMENT "Gaia G band mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_flux_over_error gaia_phot_g_mean_flux_over_error float COMMENT "Gaia G band signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_n_obs gaia_phot_g_n_obs smallint COMMENT "Gaia G band number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_mag gaia_phot_bp_mean_mag float COMMENT "Gaia BP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_flux_over_error gaia_phot_bp_mean_flux_over_error float COMMENT "Gaia BP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_n_obs gaia_phot_bp_n_obs smallint COMMENT "Gaia BP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_mag gaia_phot_rp_mean_mag float COMMENT "Gaia RP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_flux_over_error gaia_phot_rp_mean_flux_over_error float COMMENT "Gaia RP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_n_obs gaia_phot_rp_n_obs smallint COMMENT "Gaia RP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_variable_flag gaia_phot_variable_flag boolean COMMENT "Gaia photometric variable flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise gaia_astrometric_excess_noise float COMMENT "Gaia astrometric excess noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise_sig gaia_astrometric_excess_noise_sig float COMMENT "Gaia astrometric excess noise uncertainty";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_obs_al gaia_astrometric_n_obs_al smallint COMMENT "Gaia number of astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_good_obs_al gaia_astrometric_n_good_obs_al smallint COMMENT "Gaia number of good astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_weight_al gaia_astrometric_weight_al float COMMENT "Gaia astrometric weight along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_duplicated_source gaia_duplicated_source boolean COMMENT "Gaia duplicated source flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_a_g_val gaia_a_g_val float COMMENT "Gaia line-of-sight extinction in the G band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_e_bp_min_rp_val gaia_e_bp_min_rp_val float COMMENT "Gaia line-of-sight reddening E(BP-RP)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_rp_excess_factor gaia_phot_bp_rp_excess_factor float COMMENT "Gaia BP/RP excess factor";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_sigma5d_max gaia_astrometric_sigma5d_max float COMMENT "Gaia longest semi-major axis of the 5-d error ellipsoid";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_params_solved gaia_astrometric_params_solved tinyint COMMENT "which astrometric parameters were estimated for a Gaia source";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_g flux_g float COMMENT "model flux in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_r flux_r float COMMENT "model flux in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_z flux_z float COMMENT "model flux in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w1 flux_w1 float COMMENT "WISE model flux in W1W1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w2 flux_w2 float COMMENT "WISE model flux in W2W2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w3 flux_w3 float COMMENT "WISE model flux in W3W3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w4 flux_w4 float COMMENT "WISE model flux in W4W4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_g flux_ivar_g float COMMENT "Inverse variance of flux_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_r flux_ivar_r float COMMENT "Inverse variance of flux_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_z flux_ivar_z float COMMENT "Inverse variance of flux_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w1 flux_ivar_w1 float COMMENT "Inverse variance of flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w2 flux_ivar_w2 float COMMENT "Inverse variance of flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w3 flux_ivar_w3 float COMMENT "Inverse variance of flux_w3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w4 flux_ivar_w4 float COMMENT "Inverse variance of flux_w4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_g fiberflux_g float COMMENT "Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_r fiberflux_r float COMMENT "Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_z fiberflux_z float COMMENT "Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_g fibertotflux_g float COMMENT "Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_r fibertotflux_r float COMMENT "Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_z fibertotflux_z float COMMENT "Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_g apflux_g array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_r apflux_r array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_z apflux_z array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_g apflux_resid_g array<float> COMMENT "aperture fluxes on the co-added residual images in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_r apflux_resid_r array<float> COMMENT "aperture fluxes on the co-added residual images in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_z apflux_resid_z array<float> COMMENT "aperture fluxes on the co-added residual images in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_g apflux_ivar_g array<float> COMMENT "Inverse variance of apflux_resid_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_r apflux_ivar_r array<float> COMMENT "Inverse variance of apflux_resid_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_z apflux_ivar_z array<float> COMMENT "Inverse variance of apflux_resid_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_g mw_transmission_g float COMMENT "Galactic transmission in gg filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_r mw_transmission_r float COMMENT "Galactic transmission in rr filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_z mw_transmission_z float COMMENT "Galactic transmission in zz filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w1 mw_transmission_w1 float COMMENT "Galactic transmission in W1W1 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w2 mw_transmission_w2 float COMMENT "Galactic transmission in W2W2 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w3 mw_transmission_w3 float COMMENT "Galactic transmission in W3W3 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w4 mw_transmission_w4 float COMMENT "Galactic transmission in W4W4 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_g nobs_g smallint COMMENT "Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_r nobs_r smallint COMMENT "Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_z nobs_z smallint COMMENT "Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w1 nobs_w1 smallint COMMENT "Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w2 nobs_w2 smallint COMMENT "Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w3 nobs_w3 smallint COMMENT "Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w4 nobs_w4 smallint COMMENT "Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_g rchisq_g float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_r rchisq_r float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_z rchisq_z float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w1 rchisq_w1 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W1W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w2 rchisq_w2 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W2W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w3 rchisq_w3 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W3W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w4 rchisq_w4 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W4W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_g fracflux_g float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in gg(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_r fracflux_r float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in rr(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_z fracflux_z float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in zz(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w1 fracflux_w1 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w2 fracflux_w2 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w3 fracflux_w3 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w4 fracflux_w4 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_g fracmasked_g float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_r fracmasked_r float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_z fracmasked_z float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_g fracin_g float COMMENT "Fraction of a source's flux within the blob in gg, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_r fracin_r float COMMENT "Fraction of a source's flux within the blob in rr, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_z fracin_z float COMMENT "Fraction of a source's flux within the blob in zz, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_g anymask_g smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_r anymask_r smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_z anymask_z smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_g allmask_g smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_r allmask_r smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_z allmask_z smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w1 wisemask_w1 tinyint COMMENT "W1 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w2 wisemask_w2 tinyint COMMENT "W2 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_g psfsize_g float COMMENT "Weighted average PSF FWHM in the gg band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_r psfsize_r float COMMENT "Weighted average PSF FWHM in the rr band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_z psfsize_z float COMMENT "Weighted average PSF FWHM in the zz band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_g psfdepth_g float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_r psfdepth_r float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_z psfdepth_z float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_g galdepth_g float COMMENT "As for psfdepth_g but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_r galdepth_r float COMMENT "As for psfdepth_r but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_z galdepth_z float COMMENT "As for psfdepth_z but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w1 psfdepth_w1 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w2 psfdepth_w2 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w3 psfdepth_w3 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w4 psfdepth_w4 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wise_coadd_id wise_coadd_id string COMMENT "unWISE coadd file name for the center of each object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w1 lc_flux_w1 array<float> COMMENT "flux_w1 in each of up to eleven unWISE coadd epochs (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w2 lc_flux_w2 array<float> COMMENT "flux_w2 in each of up to eleven unWISE coadd epochs (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w1 lc_flux_ivar_w1 array<float> COMMENT "Inverse variance of lc_flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w2 lc_flux_ivar_w2 array<float> COMMENT "Inverse variance of lc_flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w1 lc_nobs_w1 array<smallint> COMMENT "nobs_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w2 lc_nobs_w2 array<smallint> COMMENT "nobs_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w1 lc_fracflux_w1 array<float> COMMENT "fracflux_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w2 lc_fracflux_w2 array<float> COMMENT "fracflux_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w1 lc_rchisq_w1 array<float> COMMENT "rchisq_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w2 lc_rchisq_w2 array<float> COMMENT "rchisq_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w1 lc_mjd_w1 array<double> COMMENT "mjd_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w2 lc_mjd_w2 array<double> COMMENT "mjd_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev fracdev float COMMENT "Fraction of model in deVauc [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev_ivar fracdev_ivar float COMMENT "Inverse variance of fracdev";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r shapeexp_r float COMMENT "Half-light radius of exponential model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r_ivar shapeexp_r_ivar float COMMENT "Inverse variance of shapeexp_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1 shapeexp_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1_ivar shapeexp_e1_ivar float COMMENT "Inverse variance of shapeexp_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2 shapeexp_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2_ivar shapeexp_e2_ivar float COMMENT "Inverse variance of shapeexp_e2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r shapedev_r float COMMENT "Half-light radius of deVaucouleurs model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r_ivar shapedev_r_ivar float COMMENT "Inverse variance of shapedev_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1 shapedev_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1_ivar shapedev_e1_ivar float COMMENT "Inverse variance of shapedev_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2 shapedev_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2_ivar shapedev_e2_ivar float COMMENT "Inverse variance of shapedev_e2";<br />
<br />
== Sweep photo-z ==<br />
<br />
=== Raw data ===<br />
<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/north/sweep/<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/south/sweep/<br />
<br />
Checksums checked:<br />
<br />
$ cd north/sweep/8.0<br />
$ ls | wc -l<br />
287<br />
$ wc -l legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
286 (+ checksum file)<br />
$ sha256sum -c legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
OK<br />
<br />
$ cd north/sweep/8.0-photo-z<br />
$ ls | wc -l<br />
287<br />
$ wc -l legacysurvey_dr8_north_sweep_8.0-photo-z.sha256sum<br />
286 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_north_sweep_8.0-photo-z.sha256sum<br />
OK<br />
<br />
$ cd south/sweep/8.0<br />
$ ls | wc -l<br />
438<br />
$ wc -l legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
437 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
OK<br />
<br />
$ cd south/sweep/8.0-photo-z<br />
$ ls | wc -l<br />
438<br />
$ wc -l legacysurvey_dr8_south_sweep_8.0-photo-z.sha256sum<br />
437 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_south_sweep_8.0-photo-z.sha256sum<br />
OK<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to merge fits files and convert to parquet:<br />
<br />
path_base = '/cephfs/pic.es/astro/scratch/tallada/dr8-phz/'<br />
hdfs_base = '/user/tallada/data/legacysurvey-sweep'<br />
<br />
hs = ['south/sweep'] #, 'north/sweep']<br />
dir_src = '8.0'<br />
dir_phz = '8.0-photo-z'<br />
dir_dst = '8.0-merged'<br />
<br />
my_files = sorted(os.listdir(os.path.join(path_base, hs[0], dir_src)))<br />
<br />
def merge_phz(file_name):<br />
file_src = os.path.join(path_base, hs[0], dir_src, file_name)<br />
file_phz = os.path.join(path_base, hs[0], dir_phz, file_name[:-5]+'-pz.fits')<br />
file_dst = os.path.join(path_base, hs[0], dir_dst, file_name[:-5]+'-merged.fits')<br />
<br />
t_src = table.Table.read(file_src)<br />
t_phz = table.Table.read(file_phz)<br />
t_dst = table.hstack([t_src, t_phz], join_type='exact')<br />
<br />
os.unlink(file_dst)<br />
t_dst.write(file_dst)<br />
<br />
return file_name<br />
<br />
df = sc.parallelize(my_files, len(my_files))<br />
<br />
df.map(merge_phz).count()<br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + os.path.join(path_base, hs[0], dir_dst)<br />
) <br />
<br />
df.write.parquet(os.path.join(hdfs_base, hs[0]))<br />
<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE tallada.dr8_sweep (<br />
`RELEASE` SMALLINT,<br />
`BRICKID` INT,<br />
`BRICKNAME` STRING,<br />
`OBJID` INT,<br />
`TYPE` STRING,<br />
`RA` DOUBLE,<br />
`DEC` DOUBLE,<br />
`RA_IVAR` FLOAT,<br />
`DEC_IVAR` FLOAT,<br />
`DCHISQ` ARRAY<FLOAT>,<br />
`EBV` FLOAT,<br />
`FLUX_G` FLOAT,<br />
`FLUX_R` FLOAT,<br />
`FLUX_Z` FLOAT,<br />
`FLUX_W1` FLOAT,<br />
`FLUX_W2` FLOAT,<br />
`FLUX_W3` FLOAT,<br />
`FLUX_W4` FLOAT,<br />
`FLUX_IVAR_G` FLOAT,<br />
`FLUX_IVAR_R` FLOAT,<br />
`FLUX_IVAR_Z` FLOAT,<br />
`FLUX_IVAR_W1` FLOAT,<br />
`FLUX_IVAR_W2` FLOAT,<br />
`FLUX_IVAR_W3` FLOAT,<br />
`FLUX_IVAR_W4` FLOAT,<br />
`MW_TRANSMISSION_G` FLOAT,<br />
`MW_TRANSMISSION_R` FLOAT,<br />
`MW_TRANSMISSION_Z` FLOAT,<br />
`MW_TRANSMISSION_W1` FLOAT,<br />
`MW_TRANSMISSION_W2` FLOAT,<br />
`MW_TRANSMISSION_W3` FLOAT,<br />
`MW_TRANSMISSION_W4` FLOAT,<br />
`NOBS_G` INT,<br />
`NOBS_R` INT,<br />
`NOBS_Z` INT,<br />
`NOBS_W1` INT,<br />
`NOBS_W2` INT,<br />
`NOBS_W3` INT,<br />
`NOBS_W4` INT,<br />
`RCHISQ_G` FLOAT,<br />
`RCHISQ_R` FLOAT,<br />
`RCHISQ_Z` FLOAT,<br />
`RCHISQ_W1` FLOAT,<br />
`RCHISQ_W2` FLOAT,<br />
`RCHISQ_W3` FLOAT,<br />
`RCHISQ_W4` FLOAT,<br />
`FRACFLUX_G` FLOAT,<br />
`FRACFLUX_R` FLOAT,<br />
`FRACFLUX_Z` FLOAT,<br />
`FRACFLUX_W1` FLOAT,<br />
`FRACFLUX_W2` FLOAT,<br />
`FRACFLUX_W3` FLOAT,<br />
`FRACFLUX_W4` FLOAT,<br />
`FRACMASKED_G` FLOAT,<br />
`FRACMASKED_R` FLOAT,<br />
`FRACMASKED_Z` FLOAT,<br />
`FRACIN_G` FLOAT,<br />
`FRACIN_R` FLOAT,<br />
`FRACIN_Z` FLOAT,<br />
`ANYMASK_G` SMALLINT,<br />
`ANYMASK_R` SMALLINT,<br />
`ANYMASK_Z` SMALLINT,<br />
`ALLMASK_G` SMALLINT,<br />
`ALLMASK_R` SMALLINT,<br />
`ALLMASK_Z` SMALLINT,<br />
`WISEMASK_W1` TINYINT,<br />
`WISEMASK_W2` TINYINT,<br />
`PSFSIZE_G` FLOAT,<br />
`PSFSIZE_R` FLOAT,<br />
`PSFSIZE_Z` FLOAT,<br />
`PSFDEPTH_G` FLOAT,<br />
`PSFDEPTH_R` FLOAT,<br />
`PSFDEPTH_Z` FLOAT,<br />
`GALDEPTH_G` FLOAT,<br />
`GALDEPTH_R` FLOAT,<br />
`GALDEPTH_Z` FLOAT,<br />
`PSFDEPTH_W1` FLOAT,<br />
`PSFDEPTH_W2` FLOAT,<br />
`WISE_COADD_ID` STRING,<br />
`FRACDEV` FLOAT,<br />
`FRACDEV_IVAR` FLOAT,<br />
`SHAPEDEV_R` FLOAT,<br />
`SHAPEDEV_R_IVAR` FLOAT,<br />
`SHAPEDEV_E1` FLOAT,<br />
`SHAPEDEV_E1_IVAR` FLOAT,<br />
`SHAPEDEV_E2` FLOAT,<br />
`SHAPEDEV_E2_IVAR` FLOAT,<br />
`SHAPEEXP_R` FLOAT,<br />
`SHAPEEXP_R_IVAR` FLOAT,<br />
`SHAPEEXP_E1` FLOAT,<br />
`SHAPEEXP_E1_IVAR` FLOAT,<br />
`SHAPEEXP_E2` FLOAT,<br />
`SHAPEEXP_E2_IVAR` FLOAT,<br />
`FIBERFLUX_G` FLOAT,<br />
`FIBERFLUX_R` FLOAT,<br />
`FIBERFLUX_Z` FLOAT,<br />
`FIBERTOTFLUX_G` FLOAT,<br />
`FIBERTOTFLUX_R` FLOAT,<br />
`FIBERTOTFLUX_Z` FLOAT,<br />
`REF_CAT` STRING,<br />
`REF_ID` BIGINT,<br />
`REF_EPOCH` FLOAT,<br />
`GAIA_PHOT_G_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_G_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_ASTROMETRIC_EXCESS_NOISE` FLOAT,<br />
`GAIA_DUPLICATED_SOURCE` BOOLEAN,<br />
`GAIA_PHOT_BP_RP_EXCESS_FACTOR` FLOAT,<br />
`GAIA_ASTROMETRIC_SIGMA5D_MAX` FLOAT,<br />
`GAIA_ASTROMETRIC_PARAMS_SOLVED` TINYINT,<br />
`PARALLAX` FLOAT,<br />
`PARALLAX_IVAR` FLOAT,<br />
`PMRA` FLOAT,<br />
`PMRA_IVAR` FLOAT,<br />
`PMDEC` FLOAT,<br />
`PMDEC_IVAR` FLOAT,<br />
`MASKBITS` SMALLINT,<br />
`z_phot_mean` FLOAT,<br />
`z_phot_median` FLOAT,<br />
`z_phot_std` FLOAT,<br />
`z_phot_l68` FLOAT,<br />
`z_phot_u68` FLOAT,<br />
`z_phot_l95` FLOAT,<br />
`z_phot_u95` FLOAT,<br />
`z_spec` FLOAT,<br />
`survey` STRING,<br />
`training` BOOLEAN<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey-sweep'<br />
;<br />
<br />
=== Combined table ===<br />
<br />
CREATE TABLE cosmohub.legacysurvey_dr8_phz (<br />
release smallint COMMENT 'Unique integer denoting the camera and filter set used',<br />
brickid int COMMENT 'Brick ID [1,662174]',<br />
brickname string COMMENT 'Name of brick, encoding the brick sky position, eg "1126p222" near RA=112.6, Dec=+22.2',<br />
objid int COMMENT 'Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick',<br />
brick_primary boolean COMMENT 'True if the object is within the brick boundary',<br />
brightblob smallint COMMENT 'bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page',<br />
maskbits smallint COMMENT 'bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits* maps, as cataloged on the DR8 bitmasks page',<br />
type string COMMENT 'Morphological model: "PSF"=stellar, "REX"="round exponential galaxy", "DEV"=deVauc, "EXP"=exponential, "COMP"=composite, "DUP"=Gaia source fit by different model. Note that in some FITS readers, a trailing space may be appended for "PSF ", "DEV " and "EXP " since the column data type is a 4-character string',<br />
ra double COMMENT 'Right ascension at equinox J2000',<br />
dec double COMMENT 'Declination at equinox J2000',<br />
ra_ivar float COMMENT 'Inverse variance of RA (no cosine term!), excluding astrometric calibration errors',<br />
dec_ivar float COMMENT 'Inverse variance of DEC, excluding astrometric calibration errors',<br />
bx float COMMENT 'X position (0-indexed) of coordinates in brick image stack',<br />
`by` float COMMENT 'Y position (0-indexed) of coordinates in brick image stack',<br />
dchisq array<float> COMMENT 'Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.',<br />
ebv float COMMENT 'Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns',<br />
mjd_min double COMMENT 'Minimum Modified Julian Date of observations used to construct the model of this object',<br />
mjd_max double COMMENT 'Maximum Modified Julian Date of observations used to construct the model of this object',<br />
ref_cat string COMMENT 'Reference catalog source for this star: "T2" for Tycho-2, "G2" for Gaia DR2, "L2" for the LSLGA, empty otherwise',<br />
ref_id bigint COMMENT 'Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b "sourceid" for Gaia-DR2 and LSLGA',<br />
pmra float COMMENT 'Reference catalog proper motion in the RA direction',<br />
pmdec float COMMENT 'Reference catalog proper motion in the Dec direction',<br />
parallax float COMMENT 'Reference catalog parallax',<br />
pmra_ivar float COMMENT 'Reference catalog inverse-variance on pmra',<br />
pmdec_ivar float COMMENT 'Reference catalog inverse-variance on pmdec',<br />
parallax_ivar float COMMENT 'Reference catalog inverse-variance on parallax',<br />
ref_epoch float COMMENT 'Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)',<br />
gaia_pointsource boolean COMMENT 'This Gaia DR2 source is believed to be a star, not a galaxy',<br />
gaia_phot_g_mean_mag float COMMENT 'Gaia G band mag',<br />
gaia_phot_g_mean_flux_over_error float COMMENT 'Gaia G band signal-to-noise',<br />
gaia_phot_g_n_obs smallint COMMENT 'Gaia G band number of observations',<br />
gaia_phot_bp_mean_mag float COMMENT 'Gaia BP mag',<br />
gaia_phot_bp_mean_flux_over_error float COMMENT 'Gaia BP signal-to-noise',<br />
gaia_phot_bp_n_obs smallint COMMENT 'Gaia BP number of observations',<br />
gaia_phot_rp_mean_mag float COMMENT 'Gaia RP mag',<br />
gaia_phot_rp_mean_flux_over_error float COMMENT 'Gaia RP signal-to-noise',<br />
gaia_phot_rp_n_obs smallint COMMENT 'Gaia RP number of observations',<br />
gaia_phot_variable_flag boolean COMMENT 'Gaia photometric variable flag',<br />
gaia_astrometric_excess_noise float COMMENT 'Gaia astrometric excess noise',<br />
gaia_astrometric_excess_noise_sig float COMMENT 'Gaia astrometric excess noise uncertainty',<br />
gaia_astrometric_n_obs_al smallint COMMENT 'Gaia number of astrometric observations along scan direction',<br />
gaia_astrometric_n_good_obs_al smallint COMMENT 'Gaia number of good astrometric observations along scan direction',<br />
gaia_astrometric_weight_al float COMMENT 'Gaia astrometric weight along scan direction',<br />
gaia_duplicated_source boolean COMMENT 'Gaia duplicated source flag',<br />
gaia_a_g_val float COMMENT 'Gaia line-of-sight extinction in the G band',<br />
gaia_e_bp_min_rp_val float COMMENT 'Gaia line-of-sight reddening E(BP-RP)',<br />
gaia_phot_bp_rp_excess_factor float COMMENT 'Gaia BP/RP excess factor',<br />
gaia_astrometric_sigma5d_max float COMMENT 'Gaia longest semi-major axis of the 5-d error ellipsoid',<br />
gaia_astrometric_params_solved tinyint COMMENT 'which astrometric parameters were estimated for a Gaia source',<br />
flux_g float COMMENT 'model flux in gg',<br />
flux_r float COMMENT 'model flux in rr',<br />
flux_z float COMMENT 'model flux in zz',<br />
flux_w1 float COMMENT 'WISE model flux in W1W1 (AB system)',<br />
flux_w2 float COMMENT 'WISE model flux in W2W2 (AB)',<br />
flux_w3 float COMMENT 'WISE model flux in W3W3 (AB)',<br />
flux_w4 float COMMENT 'WISE model flux in W4W4 (AB)',<br />
flux_ivar_g float COMMENT 'Inverse variance of flux_g',<br />
flux_ivar_r float COMMENT 'Inverse variance of flux_r',<br />
flux_ivar_z float COMMENT 'Inverse variance of flux_z',<br />
flux_ivar_w1 float COMMENT 'Inverse variance of flux_w1 (AB system)',<br />
flux_ivar_w2 float COMMENT 'Inverse variance of flux_w2 (AB)',<br />
flux_ivar_w3 float COMMENT 'Inverse variance of flux_w3 (AB)',<br />
flux_ivar_w4 float COMMENT 'Inverse variance of flux_w4 (AB)',<br />
fiberflux_g float COMMENT 'Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_r float COMMENT 'Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_z float COMMENT 'Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fibertotflux_g float COMMENT 'Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_r float COMMENT 'Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_z float COMMENT 'Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
apflux_g array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg',<br />
apflux_r array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr',<br />
apflux_z array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz',<br />
apflux_resid_g array<float> COMMENT 'aperture fluxes on the co-added residual images in gg',<br />
apflux_resid_r array<float> COMMENT 'aperture fluxes on the co-added residual images in rr',<br />
apflux_resid_z array<float> COMMENT 'aperture fluxes on the co-added residual images in zz',<br />
apflux_ivar_g array<float> COMMENT 'Inverse variance of apflux_resid_g',<br />
apflux_ivar_r array<float> COMMENT 'Inverse variance of apflux_resid_r',<br />
apflux_ivar_z array<float> COMMENT 'Inverse variance of apflux_resid_z',<br />
mw_transmission_g float COMMENT 'Galactic transmission in gg filter in linear units [0, 1]',<br />
mw_transmission_r float COMMENT 'Galactic transmission in rr filter in linear units [0, 1]',<br />
mw_transmission_z float COMMENT 'Galactic transmission in zz filter in linear units [0, 1]',<br />
mw_transmission_w1 float COMMENT 'Galactic transmission in W1W1 filter in linear units [0, 1]',<br />
mw_transmission_w2 float COMMENT 'Galactic transmission in W2W2 filter in linear units [0, 1]',<br />
mw_transmission_w3 float COMMENT 'Galactic transmission in W3W3 filter in linear units [0, 1]',<br />
mw_transmission_w4 float COMMENT 'Galactic transmission in W4W4 filter in linear units [0, 1]',<br />
nobs_g smallint COMMENT 'Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)',<br />
nobs_r smallint COMMENT 'Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)',<br />
nobs_z smallint COMMENT 'Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)',<br />
nobs_w1 smallint COMMENT 'Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)',<br />
nobs_w2 smallint COMMENT 'Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)',<br />
nobs_w3 smallint COMMENT 'Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)',<br />
nobs_w4 smallint COMMENT 'Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)',<br />
rchisq_g float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in gg',<br />
rchisq_r float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in rr',<br />
rchisq_z float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in zz',<br />
rchisq_w1 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W1W1',<br />
rchisq_w2 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W2W2',<br />
rchisq_w3 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W3W3',<br />
rchisq_w4 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W4W4',<br />
fracflux_g float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in gg (typically [0,1])',<br />
fracflux_r float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in rr (typically [0,1])',<br />
fracflux_z float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in zz (typically [0,1])',<br />
fracflux_w1 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1 (typically [0,1])',<br />
fracflux_w2 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2 (typically [0,1])',<br />
fracflux_w3 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3 (typically [0,1])',<br />
fracflux_w4 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4 (typically [0,1])',<br />
fracmasked_g float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]',<br />
fracmasked_r float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]',<br />
fracmasked_z float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]',<br />
fracin_g float COMMENT 'Fraction of a source\'s flux within the blob in gg, near unity for real sources',<br />
fracin_r float COMMENT 'Fraction of a source\'s flux within the blob in rr, near unity for real sources',<br />
fracin_z float COMMENT 'Fraction of a source\'s flux within the blob in zz, near unity for real sources',<br />
anymask_g smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page',<br />
anymask_r smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page',<br />
anymask_z smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page',<br />
allmask_g smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page',<br />
allmask_r smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page',<br />
allmask_z smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page',<br />
wisemask_w1 tinyint COMMENT 'W1 bitmask as cataloged on the DR8 bitmasks page',<br />
wisemask_w2 tinyint COMMENT 'W2 bitmask as cataloged on the DR8 bitmasks page',<br />
psfsize_g float COMMENT 'Weighted average PSF FWHM in the gg band',<br />
psfsize_r float COMMENT 'Weighted average PSF FWHM in the rr band',<br />
psfsize_z float COMMENT 'Weighted average PSF FWHM in the zz band',<br />
psfdepth_g float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude',<br />
psfdepth_r float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude',<br />
psfdepth_z float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude',<br />
galdepth_g float COMMENT 'As for psfdepth_g but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_r float COMMENT 'As for psfdepth_r but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_z float COMMENT 'As for psfdepth_z but for a galaxy (0.45" exp, round) detection sensitivity',<br />
psfdepth_w1 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W1',<br />
psfdepth_w2 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W2',<br />
psfdepth_w3 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W3',<br />
psfdepth_w4 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W4',<br />
wise_coadd_id string COMMENT 'unWISE coadd file name for the center of each object',<br />
lc_flux_w1 array<float> COMMENT 'flux_w1 in each of up to eleven unWISE coadd epochs (AB system)',<br />
lc_flux_w2 array<float> COMMENT 'flux_w2 in each of up to eleven unWISE coadd epochs (AB)',<br />
lc_flux_ivar_w1 array<float> COMMENT 'Inverse variance of lc_flux_w1 (AB system)',<br />
lc_flux_ivar_w2 array<float> COMMENT 'Inverse variance of lc_flux_w2 (AB)',<br />
lc_nobs_w1 array<smallint> COMMENT 'nobs_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_nobs_w2 array<smallint> COMMENT 'nobs_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w1 array<float> COMMENT 'fracflux_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w2 array<float> COMMENT 'fracflux_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w1 array<float> COMMENT 'rchisq_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w2 array<float> COMMENT 'rchisq_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w1 array<double> COMMENT 'mjd_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w2 array<double> COMMENT 'mjd_w2 in each of up to eleven unWISE coadd epochs',<br />
fracdev float COMMENT 'Fraction of model in deVauc [0,1]',<br />
fracdev_ivar float COMMENT 'Inverse variance of fracdev',<br />
shapeexp_r float COMMENT 'Half-light radius of exponential model (>0)',<br />
shapeexp_r_ivar float COMMENT 'Inverse variance of shapeexp_r',<br />
shapeexp_e1 float COMMENT 'Ellipticity component 1',<br />
shapeexp_e1_ivar float COMMENT 'Inverse variance of shapeexp_e1',<br />
shapeexp_e2 float COMMENT 'Ellipticity component 2',<br />
shapeexp_e2_ivar float COMMENT 'Inverse variance of shapeexp_e2',<br />
shapedev_r float COMMENT 'Half-light radius of deVaucouleurs model (>0)',<br />
shapedev_r_ivar float COMMENT 'Inverse variance of shapedev_r',<br />
shapedev_e1 float COMMENT 'Ellipticity component 1',<br />
shapedev_e1_ivar float COMMENT 'Inverse variance of shapedev_e1',<br />
shapedev_e2 float COMMENT 'Ellipticity component 2',<br />
shapedev_e2_ivar float COMMENT 'Inverse variance of shapedev_e2',<br />
z_phot_mean FLOAT COMMENT 'photo-z derived from the mean of the photo-z PDF ',<br />
z_phot_median FLOAT COMMENT 'photo-z derived from the median of the photo-z PDF',<br />
z_phot_std FLOAT COMMENT 'standard deviation of the photo-z\'s derived from the photo-z PDF',<br />
z_phot_l68 FLOAT COMMENT 'lower bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_u68 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_l95 FLOAT COMMENT 'lower bound of the 95% confidence region, derived from the photo-z PDF',<br />
z_phot_u95 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_spec FLOAT COMMENT 'spectroscopic redshift, if available',<br />
survey STRING COMMENT 'source of the spectroscopic redshift',<br />
training BOOLEAN COMMENT 'whether or not the spectroscopic redshift is used in photometric redshift training'<br />
)<br />
CLUSTERED BY (<br />
release,<br />
brickid,<br />
objid)<br />
SORTED BY (<br />
release ASC,<br />
brickid ASC,<br />
objid ASC)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
=== Join insert ===<br />
<br />
INSERT OVERWRITE TABLE cosmohub.legacy_survey_dr8_phz<br />
SELECT<br />
dr8.*,<br />
phz.z_phot_mean,<br />
phz.z_phot_median,<br />
phz.z_phot_std,<br />
phz.z_phot_l68,<br />
phz.z_phot_u68,<br />
phz.z_phot_l95,<br />
phz.z_phot_u95,<br />
phz.z_spec,<br />
phz.survey,<br />
phz.training<br />
FROM cosmohub.legacy_survey_dr8 AS dr8<br />
LEFT JOIN tallada.dr8_sweep AS phz<br />
ON dr8.release = phz.release<br />
AND dr8.brickid = phz.brickid<br />
AND dr8.objid = phz.objid<br />
;<br />
<br />
== Queries for checking ==<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT release, brickid, objid, COUNT(release)<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
HAVING COUNT(release) > 1<br />
<br />
DR8 release brickid objid: DISTINCT(1.694.230.368) TOTAL(1.694.230.368)<br />
DR8 brickid objid: DISTINCT(1.654.429.115) TOTAL(1.694.230.368)<br />
<br />
PHZ release brickid objid: DISTINCT(1.646.664.729) TOTAL(1.646.664.897)<br />
PHZ brickid objid: DISTINCT(1.608.841.582) TOTAL(1.646.664.897)<br />
<br />
DR8: 1.694.230.368<br />
PHZ: 1.646.664.897<br />
----------------------------------------------------------------------<br />
-47.565.471</div>
Tallada
https://pwiki.pic.es/index.php?title=Legacy_Survey_DR9&diff=875
Legacy Survey DR9
2020-11-27T00:24:29Z
<p>Tallada: /* Internal Table */</p>
<hr />
<div>== Original DR9 ==<br />
=== Raw data ===<br />
<br />
Downloaded from NERSC<br />
/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m<br />
<br />
Directories:<br />
south/tractor<br />
north/tractor<br />
<br />
Downloaded using bbcp from services01.euclid into CephFS scratch:<br />
bbcp -s 16 -a -A -v -z -r -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" \<br />
"jcarrete@dtn01.nersc.gov:/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m/north/tractor" \<br />
/cephfs/pic.es/astro/scratch/tallada/dr9m/north/tractor<br />
bbcp -s 16 -a -A -v -z -r -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" \<br />
"jcarrete@dtn01.nersc.gov:/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m/south/tractor" \<br />
/cephfs/pic.es/astro/scratch/tallada/dr9m/south/tractor<br />
<br />
Checksums provided through sha256sum files, and checked using command:<br />
find * -type d | xargs -t -L1 -P24 sh -c 'cd $1; sha256sum --quiet -c *.sha256sum' cksum<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to convert fits files to parquet:<br />
<br />
path_base = '/cephfs/pic.es/astro/scratch/tallada/dr9m'<br />
hdfs_base = '/user/tallada/data/legacysurvey_dr9'<br />
<br />
hs = ['h=north', 'h=south']<br />
rs = ['r={0:03d}'.format(x) for x in range(360)]<br />
<br />
for h in hs:<br />
for r in rs:<br />
p = os.path.join(path_base, h, r)<br />
if not os.path.exists(p):<br />
continue<br />
<br />
size = sum(os.path.getsize(os.path.join(p,f)) for f in os.listdir(p) if os.path.isfile(os.path.join(p,f)))<br />
<br />
print h, r, size <br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + p<br />
)<br />
<br />
df = df.repartition(int(math.ceil(size / (256*1024*1024.0))))<br />
<br />
df.write.parquet(os.path.join(hdfs_base, h, r))<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE `tallada`.`legacysurvey_dr9`(<br />
`release` smallint, <br />
`brickid` int, <br />
`brickname` string, <br />
`objid` int, <br />
`brick_primary` boolean, <br />
`maskbits` smallint, <br />
`fitbits` smallint, <br />
`type` string, <br />
`ra` double, <br />
`dec` double, <br />
`ra_ivar` float, <br />
`dec_ivar` float, <br />
`bx` float, <br />
`by` float, <br />
`dchisq` array<float>, <br />
`ebv` float, <br />
`mjd_min` double, <br />
`mjd_max` double, <br />
`ref_cat` string, <br />
`ref_id` bigint, <br />
`pmra` float, <br />
`pmdec` float, <br />
`parallax` float, <br />
`pmra_ivar` float, <br />
`pmdec_ivar` float, <br />
`parallax_ivar` float, <br />
`ref_epoch` float, <br />
`gaia_phot_g_mean_mag` float, <br />
`gaia_phot_g_mean_flux_over_error` float, <br />
`gaia_phot_g_n_obs` smallint, <br />
`gaia_phot_bp_mean_mag` float, <br />
`gaia_phot_bp_mean_flux_over_error` float, <br />
`gaia_phot_bp_n_obs` smallint, <br />
`gaia_phot_rp_mean_mag` float, <br />
`gaia_phot_rp_mean_flux_over_error` float, <br />
`gaia_phot_rp_n_obs` smallint, <br />
`gaia_phot_variable_flag` boolean, <br />
`gaia_astrometric_excess_noise` float, <br />
`gaia_astrometric_excess_noise_sig` float, <br />
`gaia_astrometric_n_obs_al` smallint, <br />
`gaia_astrometric_n_good_obs_al` smallint, <br />
`gaia_astrometric_weight_al` float, <br />
`gaia_duplicated_source` boolean, <br />
`gaia_a_g_val` float, <br />
`gaia_e_bp_min_rp_val` float, <br />
`gaia_phot_bp_rp_excess_factor` float, <br />
`gaia_astrometric_sigma5d_max` float, <br />
`gaia_astrometric_params_solved` tinyint, <br />
`flux_g` float, <br />
`flux_r` float, <br />
`flux_z` float, <br />
`flux_w1` float, <br />
`flux_w2` float, <br />
`flux_w3` float, <br />
`flux_w4` float, <br />
`flux_ivar_g` float, <br />
`flux_ivar_r` float, <br />
`flux_ivar_z` float, <br />
`flux_ivar_w1` float, <br />
`flux_ivar_w2` float, <br />
`flux_ivar_w3` float, <br />
`flux_ivar_w4` float, <br />
`fiberflux_g` float, <br />
`fiberflux_r` float, <br />
`fiberflux_z` float, <br />
`fibertotflux_g` float, <br />
`fibertotflux_r` float, <br />
`fibertotflux_z` float, <br />
`apflux_g` array<float>, <br />
`apflux_r` array<float>, <br />
`apflux_z` array<float>, <br />
`apflux_resid_g` array<float>, <br />
`apflux_resid_r` array<float>, <br />
`apflux_resid_z` array<float>, <br />
`apflux_blobresid_g` array<float>, <br />
`apflux_blobresid_r` array<float>, <br />
`apflux_blobresid_z` array<float>, <br />
`apflux_ivar_g` array<float>, <br />
`apflux_ivar_r` array<float>, <br />
`apflux_ivar_z` array<float>, <br />
`apflux_masked_g` array<float>, <br />
`apflux_masked_r` array<float>, <br />
`apflux_masked_z` array<float>, <br />
`apflux_w1` array<float>, <br />
`apflux_w2` array<float>, <br />
`apflux_w3` array<float>, <br />
`apflux_w4` array<float>, <br />
`apflux_resid_w1` array<float>, <br />
`apflux_resid_w2` array<float>, <br />
`apflux_resid_w3` array<float>, <br />
`apflux_resid_w4` array<float>, <br />
`apflux_ivar_w1` array<float>, <br />
`apflux_ivar_w2` array<float>, <br />
`apflux_ivar_w3` array<float>, <br />
`apflux_ivar_w4` array<float>, <br />
`mw_transmission_g` float, <br />
`mw_transmission_r` float, <br />
`mw_transmission_z` float, <br />
`mw_transmission_w1` float, <br />
`mw_transmission_w2` float, <br />
`mw_transmission_w3` float, <br />
`mw_transmission_w4` float, <br />
`nobs_g` smallint, <br />
`nobs_r` smallint, <br />
`nobs_z` smallint, <br />
`nobs_w1` smallint, <br />
`nobs_w2` smallint, <br />
`nobs_w3` smallint, <br />
`nobs_w4` smallint, <br />
`rchisq_g` float, <br />
`rchisq_r` float, <br />
`rchisq_z` float, <br />
`rchisq_w1` float, <br />
`rchisq_w2` float, <br />
`rchisq_w3` float, <br />
`rchisq_w4` float, <br />
`fracflux_g` float, <br />
`fracflux_r` float, <br />
`fracflux_z` float, <br />
`fracflux_w1` float, <br />
`fracflux_w2` float, <br />
`fracflux_w3` float, <br />
`fracflux_w4` float, <br />
`fracmasked_g` float, <br />
`fracmasked_r` float, <br />
`fracmasked_z` float, <br />
`fracin_g` float, <br />
`fracin_r` float, <br />
`fracin_z` float, <br />
`anymask_g` smallint, <br />
`anymask_r` smallint, <br />
`anymask_z` smallint, <br />
`allmask_g` smallint, <br />
`allmask_r` smallint, <br />
`allmask_z` smallint, <br />
`wisemask_w1` tinyint, <br />
`wisemask_w2` tinyint, <br />
`psfsize_g` float, <br />
`psfsize_r` float, <br />
`psfsize_z` float, <br />
`psfdepth_g` float, <br />
`psfdepth_r` float, <br />
`psfdepth_z` float, <br />
`galdepth_g` float, <br />
`galdepth_r` float, <br />
`galdepth_z` float, <br />
`nea_g` float, <br />
`nea_r` float, <br />
`nea_z` float, <br />
`blob_nea_g` float, <br />
`blob_nea_r` float, <br />
`blob_nea_z` float, <br />
`psfdepth_w1` float, <br />
`psfdepth_w2` float, <br />
`psfdepth_w3` float, <br />
`psfdepth_w4` float, <br />
`wise_coadd_id` string, <br />
`wise_x` float, <br />
`wise_y` float, <br />
`lc_flux_w1` array<float>, <br />
`lc_flux_w2` array<float>, <br />
`lc_flux_ivar_w1` array<float>, <br />
`lc_flux_ivar_w2` array<float>, <br />
`lc_nobs_w1` array<smallint>, <br />
`lc_nobs_w2` array<smallint>, <br />
`lc_fracflux_w1` array<float>, <br />
`lc_fracflux_w2` array<float>, <br />
`lc_rchisq_w1` array<float>, <br />
`lc_rchisq_w2` array<float>, <br />
`lc_mjd_w1` array<double>, <br />
`lc_mjd_w2` array<double>, <br />
`lc_epoch_index_w1` array<smallint>, <br />
`lc_epoch_index_w2` array<smallint>, <br />
`sersic` float, <br />
`sersic_ivar` float, <br />
`shape_r` float, <br />
`shape_r_ivar` float, <br />
`shape_e1` float, <br />
`shape_e1_ivar` float, <br />
`shape_e2` float, <br />
`shape_e2_ivar` float)<br />
PARTITIONED BY (<br />
h STRING,<br />
r STRING<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey_dr9'<br />
;<br />
<br />
=== Internal Table ===<br />
<br />
CREATE TABLE `cosmohub`.`legacysurvey_dr9`(<br />
`release` smallint,<br />
`brickid` int,<br />
`brickname` string,<br />
`objid` int,<br />
`brick_primary` boolean,<br />
`maskbits` smallint,<br />
`fitbits` smallint,<br />
`type` string,<br />
`ra` double,<br />
`dec` double,<br />
`ra_ivar` float,<br />
`dec_ivar` float,<br />
`bx` float,<br />
`by` float,<br />
`dchisq` array<float>,<br />
`ebv` float,<br />
`mjd_min` double,<br />
`mjd_max` double,<br />
`ref_cat` string,<br />
`ref_id` bigint,<br />
`pmra` float,<br />
`pmdec` float,<br />
`parallax` float,<br />
`pmra_ivar` float,<br />
`pmdec_ivar` float,<br />
`parallax_ivar` float,<br />
`ref_epoch` float,<br />
`gaia_phot_g_mean_mag` float,<br />
`gaia_phot_g_mean_flux_over_error` float,<br />
`gaia_phot_g_n_obs` smallint,<br />
`gaia_phot_bp_mean_mag` float,<br />
`gaia_phot_bp_mean_flux_over_error` float,<br />
`gaia_phot_bp_n_obs` smallint,<br />
`gaia_phot_rp_mean_mag` float,<br />
`gaia_phot_rp_mean_flux_over_error` float,<br />
`gaia_phot_rp_n_obs` smallint,<br />
`gaia_phot_variable_flag` boolean,<br />
`gaia_astrometric_excess_noise` float,<br />
`gaia_astrometric_excess_noise_sig` float,<br />
`gaia_astrometric_n_obs_al` smallint,<br />
`gaia_astrometric_n_good_obs_al` smallint,<br />
`gaia_astrometric_weight_al` float,<br />
`gaia_duplicated_source` boolean,<br />
`gaia_a_g_val` float,<br />
`gaia_e_bp_min_rp_val` float,<br />
`gaia_phot_bp_rp_excess_factor` float,<br />
`gaia_astrometric_sigma5d_max` float,<br />
`gaia_astrometric_params_solved` tinyint,<br />
`flux_g` float,<br />
`flux_r` float,<br />
`flux_z` float,<br />
`flux_w1` float,<br />
`flux_w2` float,<br />
`flux_w3` float,<br />
`flux_w4` float,<br />
`flux_ivar_g` float,<br />
`flux_ivar_r` float,<br />
`flux_ivar_z` float,<br />
`flux_ivar_w1` float,<br />
`flux_ivar_w2` float,<br />
`flux_ivar_w3` float,<br />
`flux_ivar_w4` float,<br />
`fiberflux_g` float,<br />
`fiberflux_r` float,<br />
`fiberflux_z` float,<br />
`fibertotflux_g` float,<br />
`fibertotflux_r` float,<br />
`fibertotflux_z` float,<br />
`apflux_g` array<float>,<br />
`apflux_r` array<float>,<br />
`apflux_z` array<float>,<br />
`apflux_resid_g` array<float>,<br />
`apflux_resid_r` array<float>,<br />
`apflux_resid_z` array<float>,<br />
`apflux_blobresid_g` array<float>,<br />
`apflux_blobresid_r` array<float>,<br />
`apflux_blobresid_z` array<float>,<br />
`apflux_ivar_g` array<float>,<br />
`apflux_ivar_r` array<float>,<br />
`apflux_ivar_z` array<float>,<br />
`apflux_masked_g` array<float>,<br />
`apflux_masked_r` array<float>,<br />
`apflux_masked_z` array<float>,<br />
`apflux_w1` array<float>,<br />
`apflux_w2` array<float>,<br />
`apflux_w3` array<float>,<br />
`apflux_w4` array<float>,<br />
`apflux_resid_w1` array<float>,<br />
`apflux_resid_w2` array<float>,<br />
`apflux_resid_w3` array<float>,<br />
`apflux_resid_w4` array<float>,<br />
`apflux_ivar_w1` array<float>,<br />
`apflux_ivar_w2` array<float>,<br />
`apflux_ivar_w3` array<float>,<br />
`apflux_ivar_w4` array<float>,<br />
`mw_transmission_g` float,<br />
`mw_transmission_r` float,<br />
`mw_transmission_z` float,<br />
`mw_transmission_w1` float,<br />
`mw_transmission_w2` float,<br />
`mw_transmission_w3` float,<br />
`mw_transmission_w4` float,<br />
`nobs_g` smallint,<br />
`nobs_r` smallint,<br />
`nobs_z` smallint,<br />
`nobs_w1` smallint,<br />
`nobs_w2` smallint,<br />
`nobs_w3` smallint,<br />
`nobs_w4` smallint,<br />
`rchisq_g` float,<br />
`rchisq_r` float,<br />
`rchisq_z` float,<br />
`rchisq_w1` float,<br />
`rchisq_w2` float,<br />
`rchisq_w3` float,<br />
`rchisq_w4` float,<br />
`fracflux_g` float,<br />
`fracflux_r` float,<br />
`fracflux_z` float,<br />
`fracflux_w1` float,<br />
`fracflux_w2` float,<br />
`fracflux_w3` float,<br />
`fracflux_w4` float,<br />
`fracmasked_g` float,<br />
`fracmasked_r` float,<br />
`fracmasked_z` float,<br />
`fracin_g` float,<br />
`fracin_r` float,<br />
`fracin_z` float,<br />
`anymask_g` smallint,<br />
`anymask_r` smallint,<br />
`anymask_z` smallint,<br />
`allmask_g` smallint,<br />
`allmask_r` smallint,<br />
`allmask_z` smallint,<br />
`wisemask_w1` tinyint,<br />
`wisemask_w2` tinyint,<br />
`psfsize_g` float,<br />
`psfsize_r` float,<br />
`psfsize_z` float,<br />
`psfdepth_g` float,<br />
`psfdepth_r` float,<br />
`psfdepth_z` float,<br />
`galdepth_g` float,<br />
`galdepth_r` float,<br />
`galdepth_z` float,<br />
`nea_g` float,<br />
`nea_r` float,<br />
`nea_z` float,<br />
`blob_nea_g` float,<br />
`blob_nea_r` float,<br />
`blob_nea_z` float,<br />
`psfdepth_w1` float,<br />
`psfdepth_w2` float,<br />
`psfdepth_w3` float,<br />
`psfdepth_w4` float,<br />
`wise_coadd_id` string,<br />
`wise_x` float,<br />
`wise_y` float,<br />
`lc_flux_w1` array<float>,<br />
`lc_flux_w2` array<float>,<br />
`lc_flux_ivar_w1` array<float>,<br />
`lc_flux_ivar_w2` array<float>,<br />
`lc_nobs_w1` array<smallint>,<br />
`lc_nobs_w2` array<smallint>,<br />
`lc_fracflux_w1` array<float>,<br />
`lc_fracflux_w2` array<float>,<br />
`lc_rchisq_w1` array<float>,<br />
`lc_rchisq_w2` array<float>,<br />
`lc_mjd_w1` array<double>,<br />
`lc_mjd_w2` array<double>,<br />
`lc_epoch_index_w1` array<smallint>,<br />
`lc_epoch_index_w2` array<smallint>,<br />
`sersic` float,<br />
`sersic_ivar` float,<br />
`shape_r` float,<br />
`shape_r_ivar` float,<br />
`shape_e1` float,<br />
`shape_e1_ivar` float,<br />
`shape_e2` float,<br />
`shape_e2_ivar` float)<br />
CLUSTERED BY (<br />
release, brickid, objid<br />
)<br />
SORTED BY (<br />
release, brickid, objid<br />
)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
<br />
INSERT OVERWRITE TABLE cosmohub.legacysurvey_dr9<br />
SELECT release, brickid, brickname, objid, brick_primary, maskbits, fitbits,<br />
type, ra, `dec`, ra_ivar, dec_ivar, bx, `by`, dchisq, ebv, mjd_min, mjd_max,<br />
ref_cat, ref_id, pmra, pmdec, parallax, pmra_ivar, pmdec_ivar, parallax_ivar,<br />
ref_epoch, gaia_phot_g_mean_mag, gaia_phot_g_mean_flux_over_error, gaia_phot_g_n_obs,<br />
gaia_phot_bp_mean_mag, gaia_phot_bp_mean_flux_over_error, gaia_phot_bp_n_obs,<br />
gaia_phot_rp_mean_mag, gaia_phot_rp_mean_flux_over_error, gaia_phot_rp_n_obs,<br />
gaia_phot_variable_flag, gaia_astrometric_excess_noise, gaia_astrometric_excess_noise_sig,<br />
gaia_astrometric_n_obs_al, gaia_astrometric_n_good_obs_al, gaia_astrometric_weight_al,<br />
gaia_duplicated_source, gaia_a_g_val, gaia_e_bp_min_rp_val, gaia_phot_bp_rp_excess_factor,<br />
gaia_astrometric_sigma5d_max, gaia_astrometric_params_solved, flux_g, flux_r,<br />
flux_z, flux_w1, flux_w2, flux_w3, flux_w4, flux_ivar_g, flux_ivar_r, flux_ivar_z,<br />
flux_ivar_w1, flux_ivar_w2, flux_ivar_w3, flux_ivar_w4, fiberflux_g, fiberflux_r,<br />
fiberflux_z, fibertotflux_g, fibertotflux_r, fibertotflux_z, apflux_g, apflux_r,<br />
apflux_z, apflux_resid_g, apflux_resid_r, apflux_resid_z, apflux_blobresid_g,<br />
apflux_blobresid_r, apflux_blobresid_z, apflux_ivar_g, apflux_ivar_r, apflux_ivar_z,<br />
apflux_masked_g, apflux_masked_r, apflux_masked_z, apflux_w1, apflux_w2, apflux_w3,<br />
apflux_w4, apflux_resid_w1, apflux_resid_w2, apflux_resid_w3, apflux_resid_w4,<br />
apflux_ivar_w1, apflux_ivar_w2, apflux_ivar_w3, apflux_ivar_w4, mw_transmission_g,<br />
mw_transmission_r, mw_transmission_z, mw_transmission_w1, mw_transmission_w2,<br />
mw_transmission_w3, mw_transmission_w4, nobs_g, nobs_r, nobs_z, nobs_w1, nobs_w2,<br />
nobs_w3, nobs_w4, rchisq_g, rchisq_r, rchisq_z, rchisq_w1, rchisq_w2, rchisq_w3,<br />
rchisq_w4, fracflux_g, fracflux_r, fracflux_z, fracflux_w1, fracflux_w2, fracflux_w3,<br />
fracflux_w4, fracmasked_g, fracmasked_r, fracmasked_z, fracin_g, fracin_r, fracin_z,<br />
anymask_g, anymask_r, anymask_z, allmask_g, allmask_r, allmask_z, wisemask_w1,<br />
wisemask_w2, psfsize_g, psfsize_r, psfsize_z, psfdepth_g, psfdepth_r, psfdepth_z,<br />
galdepth_g, galdepth_r, galdepth_z, nea_g, nea_r, nea_z, blob_nea_g, blob_nea_r,<br />
blob_nea_z, psfdepth_w1, psfdepth_w2, psfdepth_w3, psfdepth_w4, wise_coadd_id,<br />
wise_x, wise_y, lc_flux_w1, lc_flux_w2, lc_flux_ivar_w1, lc_flux_ivar_w2, lc_nobs_w1,<br />
lc_nobs_w2, lc_fracflux_w1, lc_fracflux_w2, lc_rchisq_w1, lc_rchisq_w2, lc_mjd_w1,<br />
lc_mjd_w2, lc_epoch_index_w1, lc_epoch_index_w2, sersic, sersic_ivar, shape_r,<br />
shape_r_ivar, shape_e1, shape_e1_ivar, shape_e2, shape_e2_ivar<br />
FROM tallada.legacysurvey_dr9<br />
;<br />
<br />
=== Comments ===<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE release release smallint COMMENT "Unique integer denoting the camera and filter set used";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickid brickid int COMMENT "Brick ID [1,662174]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickname brickname string COMMENT "Name of brick, encoding the brick sky position, eg \"1126p222\" near RA=112.6, Dec=+22.2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE objid objid int COMMENT "Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brick_primary brick_primary boolean COMMENT "True if the object is within the brick boundary";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brightblob brightblob smallint COMMENT "bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE maskbits maskbits smallint COMMENT "bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits*maps, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE type type string COMMENT "Morphological model: PSF=stellar, REX=round exponential galaxy, DEV=deVauc, EXP=exponential, COMP=composite, DUP=Gaia source fit by different model.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra ra double COMMENT "Right ascension at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec dec double COMMENT "Declination at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra_ivar ra_ivar float COMMENT "Inverse variance of RA (no cosine term!), excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec_ivar dec_ivar float COMMENT "Inverse variance of DEC, excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE bx bx float COMMENT "X position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE by by float COMMENT "Y position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dchisq dchisq array<float> COMMENT "Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ebv ebv float COMMENT "Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_min mjd_min double COMMENT "Minimum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_max mjd_max double COMMENT "Maximum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_cat ref_cat string COMMENT "Reference catalog source for this star: T2 for Tycho-2, G2 for Gaia DR2, L2 for the LSLGA, empty otherwise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_id ref_id bigint COMMENT "Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b \"sourceid\" for Gaia-DR2 and LSLGA";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra pmra float COMMENT "Reference catalog proper motion in the RA direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec pmdec float COMMENT "Reference catalog proper motion in the Dec direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax parallax float COMMENT "Reference catalog parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra_ivar pmra_ivar float COMMENT "Reference catalog inverse-variance on pmra";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec_ivar pmdec_ivar float COMMENT "Reference catalog inverse-variance on pmdec";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax_ivar parallax_ivar float COMMENT "Reference catalog inverse-variance on parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_epoch ref_epoch float COMMENT "Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_pointsource gaia_pointsource boolean COMMENT "This Gaia DR2 source is believed to be a star, not a galaxy";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_mag gaia_phot_g_mean_mag float COMMENT "Gaia G band mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_flux_over_error gaia_phot_g_mean_flux_over_error float COMMENT "Gaia G band signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_n_obs gaia_phot_g_n_obs smallint COMMENT "Gaia G band number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_mag gaia_phot_bp_mean_mag float COMMENT "Gaia BP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_flux_over_error gaia_phot_bp_mean_flux_over_error float COMMENT "Gaia BP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_n_obs gaia_phot_bp_n_obs smallint COMMENT "Gaia BP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_mag gaia_phot_rp_mean_mag float COMMENT "Gaia RP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_flux_over_error gaia_phot_rp_mean_flux_over_error float COMMENT "Gaia RP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_n_obs gaia_phot_rp_n_obs smallint COMMENT "Gaia RP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_variable_flag gaia_phot_variable_flag boolean COMMENT "Gaia photometric variable flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise gaia_astrometric_excess_noise float COMMENT "Gaia astrometric excess noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise_sig gaia_astrometric_excess_noise_sig float COMMENT "Gaia astrometric excess noise uncertainty";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_obs_al gaia_astrometric_n_obs_al smallint COMMENT "Gaia number of astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_good_obs_al gaia_astrometric_n_good_obs_al smallint COMMENT "Gaia number of good astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_weight_al gaia_astrometric_weight_al float COMMENT "Gaia astrometric weight along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_duplicated_source gaia_duplicated_source boolean COMMENT "Gaia duplicated source flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_a_g_val gaia_a_g_val float COMMENT "Gaia line-of-sight extinction in the G band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_e_bp_min_rp_val gaia_e_bp_min_rp_val float COMMENT "Gaia line-of-sight reddening E(BP-RP)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_rp_excess_factor gaia_phot_bp_rp_excess_factor float COMMENT "Gaia BP/RP excess factor";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_sigma5d_max gaia_astrometric_sigma5d_max float COMMENT "Gaia longest semi-major axis of the 5-d error ellipsoid";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_params_solved gaia_astrometric_params_solved tinyint COMMENT "which astrometric parameters were estimated for a Gaia source";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_g flux_g float COMMENT "model flux in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_r flux_r float COMMENT "model flux in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_z flux_z float COMMENT "model flux in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w1 flux_w1 float COMMENT "WISE model flux in W1W1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w2 flux_w2 float COMMENT "WISE model flux in W2W2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w3 flux_w3 float COMMENT "WISE model flux in W3W3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w4 flux_w4 float COMMENT "WISE model flux in W4W4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_g flux_ivar_g float COMMENT "Inverse variance of flux_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_r flux_ivar_r float COMMENT "Inverse variance of flux_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_z flux_ivar_z float COMMENT "Inverse variance of flux_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w1 flux_ivar_w1 float COMMENT "Inverse variance of flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w2 flux_ivar_w2 float COMMENT "Inverse variance of flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w3 flux_ivar_w3 float COMMENT "Inverse variance of flux_w3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w4 flux_ivar_w4 float COMMENT "Inverse variance of flux_w4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_g fiberflux_g float COMMENT "Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_r fiberflux_r float COMMENT "Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_z fiberflux_z float COMMENT "Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_g fibertotflux_g float COMMENT "Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_r fibertotflux_r float COMMENT "Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_z fibertotflux_z float COMMENT "Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_g apflux_g array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_r apflux_r array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_z apflux_z array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_g apflux_resid_g array<float> COMMENT "aperture fluxes on the co-added residual images in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_r apflux_resid_r array<float> COMMENT "aperture fluxes on the co-added residual images in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_z apflux_resid_z array<float> COMMENT "aperture fluxes on the co-added residual images in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_g apflux_ivar_g array<float> COMMENT "Inverse variance of apflux_resid_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_r apflux_ivar_r array<float> COMMENT "Inverse variance of apflux_resid_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_z apflux_ivar_z array<float> COMMENT "Inverse variance of apflux_resid_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_g mw_transmission_g float COMMENT "Galactic transmission in gg filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_r mw_transmission_r float COMMENT "Galactic transmission in rr filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_z mw_transmission_z float COMMENT "Galactic transmission in zz filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w1 mw_transmission_w1 float COMMENT "Galactic transmission in W1W1 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w2 mw_transmission_w2 float COMMENT "Galactic transmission in W2W2 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w3 mw_transmission_w3 float COMMENT "Galactic transmission in W3W3 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w4 mw_transmission_w4 float COMMENT "Galactic transmission in W4W4 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_g nobs_g smallint COMMENT "Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_r nobs_r smallint COMMENT "Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_z nobs_z smallint COMMENT "Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w1 nobs_w1 smallint COMMENT "Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w2 nobs_w2 smallint COMMENT "Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w3 nobs_w3 smallint COMMENT "Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w4 nobs_w4 smallint COMMENT "Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_g rchisq_g float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_r rchisq_r float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_z rchisq_z float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w1 rchisq_w1 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W1W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w2 rchisq_w2 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W2W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w3 rchisq_w3 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W3W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w4 rchisq_w4 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W4W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_g fracflux_g float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in gg(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_r fracflux_r float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in rr(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_z fracflux_z float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in zz(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w1 fracflux_w1 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w2 fracflux_w2 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w3 fracflux_w3 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w4 fracflux_w4 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_g fracmasked_g float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_r fracmasked_r float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_z fracmasked_z float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_g fracin_g float COMMENT "Fraction of a source's flux within the blob in gg, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_r fracin_r float COMMENT "Fraction of a source's flux within the blob in rr, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_z fracin_z float COMMENT "Fraction of a source's flux within the blob in zz, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_g anymask_g smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_r anymask_r smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_z anymask_z smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_g allmask_g smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_r allmask_r smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_z allmask_z smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w1 wisemask_w1 tinyint COMMENT "W1 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w2 wisemask_w2 tinyint COMMENT "W2 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_g psfsize_g float COMMENT "Weighted average PSF FWHM in the gg band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_r psfsize_r float COMMENT "Weighted average PSF FWHM in the rr band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_z psfsize_z float COMMENT "Weighted average PSF FWHM in the zz band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_g psfdepth_g float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_r psfdepth_r float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_z psfdepth_z float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_g galdepth_g float COMMENT "As for psfdepth_g but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_r galdepth_r float COMMENT "As for psfdepth_r but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_z galdepth_z float COMMENT "As for psfdepth_z but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w1 psfdepth_w1 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w2 psfdepth_w2 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w3 psfdepth_w3 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w4 psfdepth_w4 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wise_coadd_id wise_coadd_id string COMMENT "unWISE coadd file name for the center of each object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w1 lc_flux_w1 array<float> COMMENT "flux_w1 in each of up to eleven unWISE coadd epochs (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w2 lc_flux_w2 array<float> COMMENT "flux_w2 in each of up to eleven unWISE coadd epochs (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w1 lc_flux_ivar_w1 array<float> COMMENT "Inverse variance of lc_flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w2 lc_flux_ivar_w2 array<float> COMMENT "Inverse variance of lc_flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w1 lc_nobs_w1 array<smallint> COMMENT "nobs_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w2 lc_nobs_w2 array<smallint> COMMENT "nobs_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w1 lc_fracflux_w1 array<float> COMMENT "fracflux_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w2 lc_fracflux_w2 array<float> COMMENT "fracflux_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w1 lc_rchisq_w1 array<float> COMMENT "rchisq_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w2 lc_rchisq_w2 array<float> COMMENT "rchisq_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w1 lc_mjd_w1 array<double> COMMENT "mjd_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w2 lc_mjd_w2 array<double> COMMENT "mjd_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev fracdev float COMMENT "Fraction of model in deVauc [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev_ivar fracdev_ivar float COMMENT "Inverse variance of fracdev";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r shapeexp_r float COMMENT "Half-light radius of exponential model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r_ivar shapeexp_r_ivar float COMMENT "Inverse variance of shapeexp_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1 shapeexp_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1_ivar shapeexp_e1_ivar float COMMENT "Inverse variance of shapeexp_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2 shapeexp_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2_ivar shapeexp_e2_ivar float COMMENT "Inverse variance of shapeexp_e2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r shapedev_r float COMMENT "Half-light radius of deVaucouleurs model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r_ivar shapedev_r_ivar float COMMENT "Inverse variance of shapedev_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1 shapedev_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1_ivar shapedev_e1_ivar float COMMENT "Inverse variance of shapedev_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2 shapedev_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2_ivar shapedev_e2_ivar float COMMENT "Inverse variance of shapedev_e2";<br />
<br />
== Sweep photo-z ==<br />
<br />
=== Raw data ===<br />
<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/north/sweep/<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/south/sweep/<br />
<br />
Checksums checked:<br />
<br />
$ cd north/sweep/8.0<br />
$ ls | wc -l<br />
287<br />
$ wc -l legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
286 (+ checksum file)<br />
$ sha256sum -c legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
OK<br />
<br />
$ cd north/sweep/8.0-photo-z<br />
$ ls | wc -l<br />
287<br />
$ wc -l legacysurvey_dr8_north_sweep_8.0-photo-z.sha256sum<br />
286 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_north_sweep_8.0-photo-z.sha256sum<br />
OK<br />
<br />
$ cd south/sweep/8.0<br />
$ ls | wc -l<br />
438<br />
$ wc -l legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
437 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
OK<br />
<br />
$ cd south/sweep/8.0-photo-z<br />
$ ls | wc -l<br />
438<br />
$ wc -l legacysurvey_dr8_south_sweep_8.0-photo-z.sha256sum<br />
437 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_south_sweep_8.0-photo-z.sha256sum<br />
OK<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to merge fits files and convert to parquet:<br />
<br />
path_base = '/cephfs/pic.es/astro/scratch/tallada/dr8-phz/'<br />
hdfs_base = '/user/tallada/data/legacysurvey-sweep'<br />
<br />
hs = ['south/sweep'] #, 'north/sweep']<br />
dir_src = '8.0'<br />
dir_phz = '8.0-photo-z'<br />
dir_dst = '8.0-merged'<br />
<br />
my_files = sorted(os.listdir(os.path.join(path_base, hs[0], dir_src)))<br />
<br />
def merge_phz(file_name):<br />
file_src = os.path.join(path_base, hs[0], dir_src, file_name)<br />
file_phz = os.path.join(path_base, hs[0], dir_phz, file_name[:-5]+'-pz.fits')<br />
file_dst = os.path.join(path_base, hs[0], dir_dst, file_name[:-5]+'-merged.fits')<br />
<br />
t_src = table.Table.read(file_src)<br />
t_phz = table.Table.read(file_phz)<br />
t_dst = table.hstack([t_src, t_phz], join_type='exact')<br />
<br />
os.unlink(file_dst)<br />
t_dst.write(file_dst)<br />
<br />
return file_name<br />
<br />
df = sc.parallelize(my_files, len(my_files))<br />
<br />
df.map(merge_phz).count()<br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + os.path.join(path_base, hs[0], dir_dst)<br />
) <br />
<br />
df.write.parquet(os.path.join(hdfs_base, hs[0]))<br />
<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE tallada.dr8_sweep (<br />
`RELEASE` SMALLINT,<br />
`BRICKID` INT,<br />
`BRICKNAME` STRING,<br />
`OBJID` INT,<br />
`TYPE` STRING,<br />
`RA` DOUBLE,<br />
`DEC` DOUBLE,<br />
`RA_IVAR` FLOAT,<br />
`DEC_IVAR` FLOAT,<br />
`DCHISQ` ARRAY<FLOAT>,<br />
`EBV` FLOAT,<br />
`FLUX_G` FLOAT,<br />
`FLUX_R` FLOAT,<br />
`FLUX_Z` FLOAT,<br />
`FLUX_W1` FLOAT,<br />
`FLUX_W2` FLOAT,<br />
`FLUX_W3` FLOAT,<br />
`FLUX_W4` FLOAT,<br />
`FLUX_IVAR_G` FLOAT,<br />
`FLUX_IVAR_R` FLOAT,<br />
`FLUX_IVAR_Z` FLOAT,<br />
`FLUX_IVAR_W1` FLOAT,<br />
`FLUX_IVAR_W2` FLOAT,<br />
`FLUX_IVAR_W3` FLOAT,<br />
`FLUX_IVAR_W4` FLOAT,<br />
`MW_TRANSMISSION_G` FLOAT,<br />
`MW_TRANSMISSION_R` FLOAT,<br />
`MW_TRANSMISSION_Z` FLOAT,<br />
`MW_TRANSMISSION_W1` FLOAT,<br />
`MW_TRANSMISSION_W2` FLOAT,<br />
`MW_TRANSMISSION_W3` FLOAT,<br />
`MW_TRANSMISSION_W4` FLOAT,<br />
`NOBS_G` INT,<br />
`NOBS_R` INT,<br />
`NOBS_Z` INT,<br />
`NOBS_W1` INT,<br />
`NOBS_W2` INT,<br />
`NOBS_W3` INT,<br />
`NOBS_W4` INT,<br />
`RCHISQ_G` FLOAT,<br />
`RCHISQ_R` FLOAT,<br />
`RCHISQ_Z` FLOAT,<br />
`RCHISQ_W1` FLOAT,<br />
`RCHISQ_W2` FLOAT,<br />
`RCHISQ_W3` FLOAT,<br />
`RCHISQ_W4` FLOAT,<br />
`FRACFLUX_G` FLOAT,<br />
`FRACFLUX_R` FLOAT,<br />
`FRACFLUX_Z` FLOAT,<br />
`FRACFLUX_W1` FLOAT,<br />
`FRACFLUX_W2` FLOAT,<br />
`FRACFLUX_W3` FLOAT,<br />
`FRACFLUX_W4` FLOAT,<br />
`FRACMASKED_G` FLOAT,<br />
`FRACMASKED_R` FLOAT,<br />
`FRACMASKED_Z` FLOAT,<br />
`FRACIN_G` FLOAT,<br />
`FRACIN_R` FLOAT,<br />
`FRACIN_Z` FLOAT,<br />
`ANYMASK_G` SMALLINT,<br />
`ANYMASK_R` SMALLINT,<br />
`ANYMASK_Z` SMALLINT,<br />
`ALLMASK_G` SMALLINT,<br />
`ALLMASK_R` SMALLINT,<br />
`ALLMASK_Z` SMALLINT,<br />
`WISEMASK_W1` TINYINT,<br />
`WISEMASK_W2` TINYINT,<br />
`PSFSIZE_G` FLOAT,<br />
`PSFSIZE_R` FLOAT,<br />
`PSFSIZE_Z` FLOAT,<br />
`PSFDEPTH_G` FLOAT,<br />
`PSFDEPTH_R` FLOAT,<br />
`PSFDEPTH_Z` FLOAT,<br />
`GALDEPTH_G` FLOAT,<br />
`GALDEPTH_R` FLOAT,<br />
`GALDEPTH_Z` FLOAT,<br />
`PSFDEPTH_W1` FLOAT,<br />
`PSFDEPTH_W2` FLOAT,<br />
`WISE_COADD_ID` STRING,<br />
`FRACDEV` FLOAT,<br />
`FRACDEV_IVAR` FLOAT,<br />
`SHAPEDEV_R` FLOAT,<br />
`SHAPEDEV_R_IVAR` FLOAT,<br />
`SHAPEDEV_E1` FLOAT,<br />
`SHAPEDEV_E1_IVAR` FLOAT,<br />
`SHAPEDEV_E2` FLOAT,<br />
`SHAPEDEV_E2_IVAR` FLOAT,<br />
`SHAPEEXP_R` FLOAT,<br />
`SHAPEEXP_R_IVAR` FLOAT,<br />
`SHAPEEXP_E1` FLOAT,<br />
`SHAPEEXP_E1_IVAR` FLOAT,<br />
`SHAPEEXP_E2` FLOAT,<br />
`SHAPEEXP_E2_IVAR` FLOAT,<br />
`FIBERFLUX_G` FLOAT,<br />
`FIBERFLUX_R` FLOAT,<br />
`FIBERFLUX_Z` FLOAT,<br />
`FIBERTOTFLUX_G` FLOAT,<br />
`FIBERTOTFLUX_R` FLOAT,<br />
`FIBERTOTFLUX_Z` FLOAT,<br />
`REF_CAT` STRING,<br />
`REF_ID` BIGINT,<br />
`REF_EPOCH` FLOAT,<br />
`GAIA_PHOT_G_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_G_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_ASTROMETRIC_EXCESS_NOISE` FLOAT,<br />
`GAIA_DUPLICATED_SOURCE` BOOLEAN,<br />
`GAIA_PHOT_BP_RP_EXCESS_FACTOR` FLOAT,<br />
`GAIA_ASTROMETRIC_SIGMA5D_MAX` FLOAT,<br />
`GAIA_ASTROMETRIC_PARAMS_SOLVED` TINYINT,<br />
`PARALLAX` FLOAT,<br />
`PARALLAX_IVAR` FLOAT,<br />
`PMRA` FLOAT,<br />
`PMRA_IVAR` FLOAT,<br />
`PMDEC` FLOAT,<br />
`PMDEC_IVAR` FLOAT,<br />
`MASKBITS` SMALLINT,<br />
`z_phot_mean` FLOAT,<br />
`z_phot_median` FLOAT,<br />
`z_phot_std` FLOAT,<br />
`z_phot_l68` FLOAT,<br />
`z_phot_u68` FLOAT,<br />
`z_phot_l95` FLOAT,<br />
`z_phot_u95` FLOAT,<br />
`z_spec` FLOAT,<br />
`survey` STRING,<br />
`training` BOOLEAN<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey-sweep'<br />
;<br />
<br />
=== Combined table ===<br />
<br />
CREATE TABLE cosmohub.legacysurvey_dr8_phz (<br />
release smallint COMMENT 'Unique integer denoting the camera and filter set used',<br />
brickid int COMMENT 'Brick ID [1,662174]',<br />
brickname string COMMENT 'Name of brick, encoding the brick sky position, eg "1126p222" near RA=112.6, Dec=+22.2',<br />
objid int COMMENT 'Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick',<br />
brick_primary boolean COMMENT 'True if the object is within the brick boundary',<br />
brightblob smallint COMMENT 'bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page',<br />
maskbits smallint COMMENT 'bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits* maps, as cataloged on the DR8 bitmasks page',<br />
type string COMMENT 'Morphological model: "PSF"=stellar, "REX"="round exponential galaxy", "DEV"=deVauc, "EXP"=exponential, "COMP"=composite, "DUP"=Gaia source fit by different model. Note that in some FITS readers, a trailing space may be appended for "PSF ", "DEV " and "EXP " since the column data type is a 4-character string',<br />
ra double COMMENT 'Right ascension at equinox J2000',<br />
dec double COMMENT 'Declination at equinox J2000',<br />
ra_ivar float COMMENT 'Inverse variance of RA (no cosine term!), excluding astrometric calibration errors',<br />
dec_ivar float COMMENT 'Inverse variance of DEC, excluding astrometric calibration errors',<br />
bx float COMMENT 'X position (0-indexed) of coordinates in brick image stack',<br />
`by` float COMMENT 'Y position (0-indexed) of coordinates in brick image stack',<br />
dchisq array<float> COMMENT 'Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.',<br />
ebv float COMMENT 'Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns',<br />
mjd_min double COMMENT 'Minimum Modified Julian Date of observations used to construct the model of this object',<br />
mjd_max double COMMENT 'Maximum Modified Julian Date of observations used to construct the model of this object',<br />
ref_cat string COMMENT 'Reference catalog source for this star: "T2" for Tycho-2, "G2" for Gaia DR2, "L2" for the LSLGA, empty otherwise',<br />
ref_id bigint COMMENT 'Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b "sourceid" for Gaia-DR2 and LSLGA',<br />
pmra float COMMENT 'Reference catalog proper motion in the RA direction',<br />
pmdec float COMMENT 'Reference catalog proper motion in the Dec direction',<br />
parallax float COMMENT 'Reference catalog parallax',<br />
pmra_ivar float COMMENT 'Reference catalog inverse-variance on pmra',<br />
pmdec_ivar float COMMENT 'Reference catalog inverse-variance on pmdec',<br />
parallax_ivar float COMMENT 'Reference catalog inverse-variance on parallax',<br />
ref_epoch float COMMENT 'Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)',<br />
gaia_pointsource boolean COMMENT 'This Gaia DR2 source is believed to be a star, not a galaxy',<br />
gaia_phot_g_mean_mag float COMMENT 'Gaia G band mag',<br />
gaia_phot_g_mean_flux_over_error float COMMENT 'Gaia G band signal-to-noise',<br />
gaia_phot_g_n_obs smallint COMMENT 'Gaia G band number of observations',<br />
gaia_phot_bp_mean_mag float COMMENT 'Gaia BP mag',<br />
gaia_phot_bp_mean_flux_over_error float COMMENT 'Gaia BP signal-to-noise',<br />
gaia_phot_bp_n_obs smallint COMMENT 'Gaia BP number of observations',<br />
gaia_phot_rp_mean_mag float COMMENT 'Gaia RP mag',<br />
gaia_phot_rp_mean_flux_over_error float COMMENT 'Gaia RP signal-to-noise',<br />
gaia_phot_rp_n_obs smallint COMMENT 'Gaia RP number of observations',<br />
gaia_phot_variable_flag boolean COMMENT 'Gaia photometric variable flag',<br />
gaia_astrometric_excess_noise float COMMENT 'Gaia astrometric excess noise',<br />
gaia_astrometric_excess_noise_sig float COMMENT 'Gaia astrometric excess noise uncertainty',<br />
gaia_astrometric_n_obs_al smallint COMMENT 'Gaia number of astrometric observations along scan direction',<br />
gaia_astrometric_n_good_obs_al smallint COMMENT 'Gaia number of good astrometric observations along scan direction',<br />
gaia_astrometric_weight_al float COMMENT 'Gaia astrometric weight along scan direction',<br />
gaia_duplicated_source boolean COMMENT 'Gaia duplicated source flag',<br />
gaia_a_g_val float COMMENT 'Gaia line-of-sight extinction in the G band',<br />
gaia_e_bp_min_rp_val float COMMENT 'Gaia line-of-sight reddening E(BP-RP)',<br />
gaia_phot_bp_rp_excess_factor float COMMENT 'Gaia BP/RP excess factor',<br />
gaia_astrometric_sigma5d_max float COMMENT 'Gaia longest semi-major axis of the 5-d error ellipsoid',<br />
gaia_astrometric_params_solved tinyint COMMENT 'which astrometric parameters were estimated for a Gaia source',<br />
flux_g float COMMENT 'model flux in gg',<br />
flux_r float COMMENT 'model flux in rr',<br />
flux_z float COMMENT 'model flux in zz',<br />
flux_w1 float COMMENT 'WISE model flux in W1W1 (AB system)',<br />
flux_w2 float COMMENT 'WISE model flux in W2W2 (AB)',<br />
flux_w3 float COMMENT 'WISE model flux in W3W3 (AB)',<br />
flux_w4 float COMMENT 'WISE model flux in W4W4 (AB)',<br />
flux_ivar_g float COMMENT 'Inverse variance of flux_g',<br />
flux_ivar_r float COMMENT 'Inverse variance of flux_r',<br />
flux_ivar_z float COMMENT 'Inverse variance of flux_z',<br />
flux_ivar_w1 float COMMENT 'Inverse variance of flux_w1 (AB system)',<br />
flux_ivar_w2 float COMMENT 'Inverse variance of flux_w2 (AB)',<br />
flux_ivar_w3 float COMMENT 'Inverse variance of flux_w3 (AB)',<br />
flux_ivar_w4 float COMMENT 'Inverse variance of flux_w4 (AB)',<br />
fiberflux_g float COMMENT 'Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_r float COMMENT 'Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_z float COMMENT 'Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fibertotflux_g float COMMENT 'Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_r float COMMENT 'Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_z float COMMENT 'Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
apflux_g array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg',<br />
apflux_r array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr',<br />
apflux_z array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz',<br />
apflux_resid_g array<float> COMMENT 'aperture fluxes on the co-added residual images in gg',<br />
apflux_resid_r array<float> COMMENT 'aperture fluxes on the co-added residual images in rr',<br />
apflux_resid_z array<float> COMMENT 'aperture fluxes on the co-added residual images in zz',<br />
apflux_ivar_g array<float> COMMENT 'Inverse variance of apflux_resid_g',<br />
apflux_ivar_r array<float> COMMENT 'Inverse variance of apflux_resid_r',<br />
apflux_ivar_z array<float> COMMENT 'Inverse variance of apflux_resid_z',<br />
mw_transmission_g float COMMENT 'Galactic transmission in gg filter in linear units [0, 1]',<br />
mw_transmission_r float COMMENT 'Galactic transmission in rr filter in linear units [0, 1]',<br />
mw_transmission_z float COMMENT 'Galactic transmission in zz filter in linear units [0, 1]',<br />
mw_transmission_w1 float COMMENT 'Galactic transmission in W1W1 filter in linear units [0, 1]',<br />
mw_transmission_w2 float COMMENT 'Galactic transmission in W2W2 filter in linear units [0, 1]',<br />
mw_transmission_w3 float COMMENT 'Galactic transmission in W3W3 filter in linear units [0, 1]',<br />
mw_transmission_w4 float COMMENT 'Galactic transmission in W4W4 filter in linear units [0, 1]',<br />
nobs_g smallint COMMENT 'Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)',<br />
nobs_r smallint COMMENT 'Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)',<br />
nobs_z smallint COMMENT 'Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)',<br />
nobs_w1 smallint COMMENT 'Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)',<br />
nobs_w2 smallint COMMENT 'Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)',<br />
nobs_w3 smallint COMMENT 'Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)',<br />
nobs_w4 smallint COMMENT 'Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)',<br />
rchisq_g float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in gg',<br />
rchisq_r float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in rr',<br />
rchisq_z float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in zz',<br />
rchisq_w1 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W1W1',<br />
rchisq_w2 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W2W2',<br />
rchisq_w3 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W3W3',<br />
rchisq_w4 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W4W4',<br />
fracflux_g float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in gg (typically [0,1])',<br />
fracflux_r float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in rr (typically [0,1])',<br />
fracflux_z float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in zz (typically [0,1])',<br />
fracflux_w1 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1 (typically [0,1])',<br />
fracflux_w2 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2 (typically [0,1])',<br />
fracflux_w3 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3 (typically [0,1])',<br />
fracflux_w4 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4 (typically [0,1])',<br />
fracmasked_g float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]',<br />
fracmasked_r float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]',<br />
fracmasked_z float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]',<br />
fracin_g float COMMENT 'Fraction of a source\'s flux within the blob in gg, near unity for real sources',<br />
fracin_r float COMMENT 'Fraction of a source\'s flux within the blob in rr, near unity for real sources',<br />
fracin_z float COMMENT 'Fraction of a source\'s flux within the blob in zz, near unity for real sources',<br />
anymask_g smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page',<br />
anymask_r smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page',<br />
anymask_z smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page',<br />
allmask_g smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page',<br />
allmask_r smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page',<br />
allmask_z smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page',<br />
wisemask_w1 tinyint COMMENT 'W1 bitmask as cataloged on the DR8 bitmasks page',<br />
wisemask_w2 tinyint COMMENT 'W2 bitmask as cataloged on the DR8 bitmasks page',<br />
psfsize_g float COMMENT 'Weighted average PSF FWHM in the gg band',<br />
psfsize_r float COMMENT 'Weighted average PSF FWHM in the rr band',<br />
psfsize_z float COMMENT 'Weighted average PSF FWHM in the zz band',<br />
psfdepth_g float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude',<br />
psfdepth_r float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude',<br />
psfdepth_z float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude',<br />
galdepth_g float COMMENT 'As for psfdepth_g but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_r float COMMENT 'As for psfdepth_r but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_z float COMMENT 'As for psfdepth_z but for a galaxy (0.45" exp, round) detection sensitivity',<br />
psfdepth_w1 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W1',<br />
psfdepth_w2 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W2',<br />
psfdepth_w3 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W3',<br />
psfdepth_w4 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W4',<br />
wise_coadd_id string COMMENT 'unWISE coadd file name for the center of each object',<br />
lc_flux_w1 array<float> COMMENT 'flux_w1 in each of up to eleven unWISE coadd epochs (AB system)',<br />
lc_flux_w2 array<float> COMMENT 'flux_w2 in each of up to eleven unWISE coadd epochs (AB)',<br />
lc_flux_ivar_w1 array<float> COMMENT 'Inverse variance of lc_flux_w1 (AB system)',<br />
lc_flux_ivar_w2 array<float> COMMENT 'Inverse variance of lc_flux_w2 (AB)',<br />
lc_nobs_w1 array<smallint> COMMENT 'nobs_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_nobs_w2 array<smallint> COMMENT 'nobs_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w1 array<float> COMMENT 'fracflux_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w2 array<float> COMMENT 'fracflux_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w1 array<float> COMMENT 'rchisq_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w2 array<float> COMMENT 'rchisq_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w1 array<double> COMMENT 'mjd_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w2 array<double> COMMENT 'mjd_w2 in each of up to eleven unWISE coadd epochs',<br />
fracdev float COMMENT 'Fraction of model in deVauc [0,1]',<br />
fracdev_ivar float COMMENT 'Inverse variance of fracdev',<br />
shapeexp_r float COMMENT 'Half-light radius of exponential model (>0)',<br />
shapeexp_r_ivar float COMMENT 'Inverse variance of shapeexp_r',<br />
shapeexp_e1 float COMMENT 'Ellipticity component 1',<br />
shapeexp_e1_ivar float COMMENT 'Inverse variance of shapeexp_e1',<br />
shapeexp_e2 float COMMENT 'Ellipticity component 2',<br />
shapeexp_e2_ivar float COMMENT 'Inverse variance of shapeexp_e2',<br />
shapedev_r float COMMENT 'Half-light radius of deVaucouleurs model (>0)',<br />
shapedev_r_ivar float COMMENT 'Inverse variance of shapedev_r',<br />
shapedev_e1 float COMMENT 'Ellipticity component 1',<br />
shapedev_e1_ivar float COMMENT 'Inverse variance of shapedev_e1',<br />
shapedev_e2 float COMMENT 'Ellipticity component 2',<br />
shapedev_e2_ivar float COMMENT 'Inverse variance of shapedev_e2',<br />
z_phot_mean FLOAT COMMENT 'photo-z derived from the mean of the photo-z PDF ',<br />
z_phot_median FLOAT COMMENT 'photo-z derived from the median of the photo-z PDF',<br />
z_phot_std FLOAT COMMENT 'standard deviation of the photo-z\'s derived from the photo-z PDF',<br />
z_phot_l68 FLOAT COMMENT 'lower bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_u68 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_l95 FLOAT COMMENT 'lower bound of the 95% confidence region, derived from the photo-z PDF',<br />
z_phot_u95 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_spec FLOAT COMMENT 'spectroscopic redshift, if available',<br />
survey STRING COMMENT 'source of the spectroscopic redshift',<br />
training BOOLEAN COMMENT 'whether or not the spectroscopic redshift is used in photometric redshift training'<br />
)<br />
CLUSTERED BY (<br />
release,<br />
brickid,<br />
objid)<br />
SORTED BY (<br />
release ASC,<br />
brickid ASC,<br />
objid ASC)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
=== Join insert ===<br />
<br />
INSERT OVERWRITE TABLE cosmohub.legacy_survey_dr8_phz<br />
SELECT<br />
dr8.*,<br />
phz.z_phot_mean,<br />
phz.z_phot_median,<br />
phz.z_phot_std,<br />
phz.z_phot_l68,<br />
phz.z_phot_u68,<br />
phz.z_phot_l95,<br />
phz.z_phot_u95,<br />
phz.z_spec,<br />
phz.survey,<br />
phz.training<br />
FROM cosmohub.legacy_survey_dr8 AS dr8<br />
LEFT JOIN tallada.dr8_sweep AS phz<br />
ON dr8.release = phz.release<br />
AND dr8.brickid = phz.brickid<br />
AND dr8.objid = phz.objid<br />
;<br />
<br />
== Queries for checking ==<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT release, brickid, objid, COUNT(release)<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
HAVING COUNT(release) > 1<br />
<br />
DR8 release brickid objid: DISTINCT(1.694.230.368) TOTAL(1.694.230.368)<br />
DR8 brickid objid: DISTINCT(1.654.429.115) TOTAL(1.694.230.368)<br />
<br />
PHZ release brickid objid: DISTINCT(1.646.664.729) TOTAL(1.646.664.897)<br />
PHZ brickid objid: DISTINCT(1.608.841.582) TOTAL(1.646.664.897)<br />
<br />
DR8: 1.694.230.368<br />
PHZ: 1.646.664.897<br />
----------------------------------------------------------------------<br />
-47.565.471</div>
Tallada
https://pwiki.pic.es/index.php?title=Legacy_Survey_DR9&diff=874
Legacy Survey DR9
2020-11-26T00:51:35Z
<p>Tallada: </p>
<hr />
<div>== Original DR9 ==<br />
=== Raw data ===<br />
<br />
Downloaded from NERSC<br />
/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m<br />
<br />
Directories:<br />
south/tractor<br />
north/tractor<br />
<br />
Downloaded using bbcp from services01.euclid into CephFS scratch:<br />
bbcp -s 16 -a -A -v -z -r -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" \<br />
"jcarrete@dtn01.nersc.gov:/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m/north/tractor" \<br />
/cephfs/pic.es/astro/scratch/tallada/dr9m/north/tractor<br />
bbcp -s 16 -a -A -v -z -r -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" \<br />
"jcarrete@dtn01.nersc.gov:/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m/south/tractor" \<br />
/cephfs/pic.es/astro/scratch/tallada/dr9m/south/tractor<br />
<br />
Checksums provided through sha256sum files, and checked using command:<br />
find * -type d | xargs -t -L1 -P24 sh -c 'cd $1; sha256sum --quiet -c *.sha256sum' cksum<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to convert fits files to parquet:<br />
<br />
path_base = '/cephfs/pic.es/astro/scratch/tallada/dr9m'<br />
hdfs_base = '/user/tallada/data/legacysurvey_dr9'<br />
<br />
hs = ['h=north', 'h=south']<br />
rs = ['r={0:03d}'.format(x) for x in range(360)]<br />
<br />
for h in hs:<br />
for r in rs:<br />
p = os.path.join(path_base, h, r)<br />
if not os.path.exists(p):<br />
continue<br />
<br />
size = sum(os.path.getsize(os.path.join(p,f)) for f in os.listdir(p) if os.path.isfile(os.path.join(p,f)))<br />
<br />
print h, r, size <br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + p<br />
)<br />
<br />
df = df.repartition(int(math.ceil(size / (256*1024*1024.0))))<br />
<br />
df.write.parquet(os.path.join(hdfs_base, h, r))<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE `tallada`.`legacysurvey_dr9`(<br />
`release` smallint, <br />
`brickid` int, <br />
`brickname` string, <br />
`objid` int, <br />
`brick_primary` boolean, <br />
`maskbits` smallint, <br />
`fitbits` smallint, <br />
`type` string, <br />
`ra` double, <br />
`dec` double, <br />
`ra_ivar` float, <br />
`dec_ivar` float, <br />
`bx` float, <br />
`by` float, <br />
`dchisq` array<float>, <br />
`ebv` float, <br />
`mjd_min` double, <br />
`mjd_max` double, <br />
`ref_cat` string, <br />
`ref_id` bigint, <br />
`pmra` float, <br />
`pmdec` float, <br />
`parallax` float, <br />
`pmra_ivar` float, <br />
`pmdec_ivar` float, <br />
`parallax_ivar` float, <br />
`ref_epoch` float, <br />
`gaia_phot_g_mean_mag` float, <br />
`gaia_phot_g_mean_flux_over_error` float, <br />
`gaia_phot_g_n_obs` smallint, <br />
`gaia_phot_bp_mean_mag` float, <br />
`gaia_phot_bp_mean_flux_over_error` float, <br />
`gaia_phot_bp_n_obs` smallint, <br />
`gaia_phot_rp_mean_mag` float, <br />
`gaia_phot_rp_mean_flux_over_error` float, <br />
`gaia_phot_rp_n_obs` smallint, <br />
`gaia_phot_variable_flag` boolean, <br />
`gaia_astrometric_excess_noise` float, <br />
`gaia_astrometric_excess_noise_sig` float, <br />
`gaia_astrometric_n_obs_al` smallint, <br />
`gaia_astrometric_n_good_obs_al` smallint, <br />
`gaia_astrometric_weight_al` float, <br />
`gaia_duplicated_source` boolean, <br />
`gaia_a_g_val` float, <br />
`gaia_e_bp_min_rp_val` float, <br />
`gaia_phot_bp_rp_excess_factor` float, <br />
`gaia_astrometric_sigma5d_max` float, <br />
`gaia_astrometric_params_solved` tinyint, <br />
`flux_g` float, <br />
`flux_r` float, <br />
`flux_z` float, <br />
`flux_w1` float, <br />
`flux_w2` float, <br />
`flux_w3` float, <br />
`flux_w4` float, <br />
`flux_ivar_g` float, <br />
`flux_ivar_r` float, <br />
`flux_ivar_z` float, <br />
`flux_ivar_w1` float, <br />
`flux_ivar_w2` float, <br />
`flux_ivar_w3` float, <br />
`flux_ivar_w4` float, <br />
`fiberflux_g` float, <br />
`fiberflux_r` float, <br />
`fiberflux_z` float, <br />
`fibertotflux_g` float, <br />
`fibertotflux_r` float, <br />
`fibertotflux_z` float, <br />
`apflux_g` array<float>, <br />
`apflux_r` array<float>, <br />
`apflux_z` array<float>, <br />
`apflux_resid_g` array<float>, <br />
`apflux_resid_r` array<float>, <br />
`apflux_resid_z` array<float>, <br />
`apflux_blobresid_g` array<float>, <br />
`apflux_blobresid_r` array<float>, <br />
`apflux_blobresid_z` array<float>, <br />
`apflux_ivar_g` array<float>, <br />
`apflux_ivar_r` array<float>, <br />
`apflux_ivar_z` array<float>, <br />
`apflux_masked_g` array<float>, <br />
`apflux_masked_r` array<float>, <br />
`apflux_masked_z` array<float>, <br />
`apflux_w1` array<float>, <br />
`apflux_w2` array<float>, <br />
`apflux_w3` array<float>, <br />
`apflux_w4` array<float>, <br />
`apflux_resid_w1` array<float>, <br />
`apflux_resid_w2` array<float>, <br />
`apflux_resid_w3` array<float>, <br />
`apflux_resid_w4` array<float>, <br />
`apflux_ivar_w1` array<float>, <br />
`apflux_ivar_w2` array<float>, <br />
`apflux_ivar_w3` array<float>, <br />
`apflux_ivar_w4` array<float>, <br />
`mw_transmission_g` float, <br />
`mw_transmission_r` float, <br />
`mw_transmission_z` float, <br />
`mw_transmission_w1` float, <br />
`mw_transmission_w2` float, <br />
`mw_transmission_w3` float, <br />
`mw_transmission_w4` float, <br />
`nobs_g` smallint, <br />
`nobs_r` smallint, <br />
`nobs_z` smallint, <br />
`nobs_w1` smallint, <br />
`nobs_w2` smallint, <br />
`nobs_w3` smallint, <br />
`nobs_w4` smallint, <br />
`rchisq_g` float, <br />
`rchisq_r` float, <br />
`rchisq_z` float, <br />
`rchisq_w1` float, <br />
`rchisq_w2` float, <br />
`rchisq_w3` float, <br />
`rchisq_w4` float, <br />
`fracflux_g` float, <br />
`fracflux_r` float, <br />
`fracflux_z` float, <br />
`fracflux_w1` float, <br />
`fracflux_w2` float, <br />
`fracflux_w3` float, <br />
`fracflux_w4` float, <br />
`fracmasked_g` float, <br />
`fracmasked_r` float, <br />
`fracmasked_z` float, <br />
`fracin_g` float, <br />
`fracin_r` float, <br />
`fracin_z` float, <br />
`anymask_g` smallint, <br />
`anymask_r` smallint, <br />
`anymask_z` smallint, <br />
`allmask_g` smallint, <br />
`allmask_r` smallint, <br />
`allmask_z` smallint, <br />
`wisemask_w1` tinyint, <br />
`wisemask_w2` tinyint, <br />
`psfsize_g` float, <br />
`psfsize_r` float, <br />
`psfsize_z` float, <br />
`psfdepth_g` float, <br />
`psfdepth_r` float, <br />
`psfdepth_z` float, <br />
`galdepth_g` float, <br />
`galdepth_r` float, <br />
`galdepth_z` float, <br />
`nea_g` float, <br />
`nea_r` float, <br />
`nea_z` float, <br />
`blob_nea_g` float, <br />
`blob_nea_r` float, <br />
`blob_nea_z` float, <br />
`psfdepth_w1` float, <br />
`psfdepth_w2` float, <br />
`psfdepth_w3` float, <br />
`psfdepth_w4` float, <br />
`wise_coadd_id` string, <br />
`wise_x` float, <br />
`wise_y` float, <br />
`lc_flux_w1` array<float>, <br />
`lc_flux_w2` array<float>, <br />
`lc_flux_ivar_w1` array<float>, <br />
`lc_flux_ivar_w2` array<float>, <br />
`lc_nobs_w1` array<smallint>, <br />
`lc_nobs_w2` array<smallint>, <br />
`lc_fracflux_w1` array<float>, <br />
`lc_fracflux_w2` array<float>, <br />
`lc_rchisq_w1` array<float>, <br />
`lc_rchisq_w2` array<float>, <br />
`lc_mjd_w1` array<double>, <br />
`lc_mjd_w2` array<double>, <br />
`lc_epoch_index_w1` array<smallint>, <br />
`lc_epoch_index_w2` array<smallint>, <br />
`sersic` float, <br />
`sersic_ivar` float, <br />
`shape_r` float, <br />
`shape_r_ivar` float, <br />
`shape_e1` float, <br />
`shape_e1_ivar` float, <br />
`shape_e2` float, <br />
`shape_e2_ivar` float)<br />
PARTITIONED BY (<br />
h STRING,<br />
r STRING<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey_dr9'<br />
;<br />
<br />
=== Internal Table ===<br />
<br />
CREATE TABLE cosmohub.legacy_survey_dr8 (<br />
`release` smallint,<br />
`brickid` int,<br />
`brickname` string,<br />
`objid` int,<br />
`brick_primary` boolean,<br />
`brightblob` smallint,<br />
`maskbits` smallint,<br />
`type` string,<br />
`ra` double,<br />
`dec` double,<br />
`ra_ivar` float,<br />
`dec_ivar` float,<br />
`bx` float,<br />
`by` float,<br />
`dchisq` array<float>,<br />
`ebv` float,<br />
`mjd_min` double,<br />
`mjd_max` double,<br />
`ref_cat` string,<br />
`ref_id` bigint,<br />
`pmra` float,<br />
`pmdec` float,<br />
`parallax` float,<br />
`pmra_ivar` float,<br />
`pmdec_ivar` float,<br />
`parallax_ivar` float,<br />
`ref_epoch` float,<br />
`gaia_pointsource` boolean,<br />
`gaia_phot_g_mean_mag` float,<br />
`gaia_phot_g_mean_flux_over_error` float,<br />
`gaia_phot_g_n_obs` smallint,<br />
`gaia_phot_bp_mean_mag` float,<br />
`gaia_phot_bp_mean_flux_over_error` float,<br />
`gaia_phot_bp_n_obs` smallint,<br />
`gaia_phot_rp_mean_mag` float,<br />
`gaia_phot_rp_mean_flux_over_error` float,<br />
`gaia_phot_rp_n_obs` smallint,<br />
`gaia_phot_variable_flag` boolean,<br />
`gaia_astrometric_excess_noise` float,<br />
`gaia_astrometric_excess_noise_sig` float,<br />
`gaia_astrometric_n_obs_al` smallint,<br />
`gaia_astrometric_n_good_obs_al` smallint,<br />
`gaia_astrometric_weight_al` float,<br />
`gaia_duplicated_source` boolean,<br />
`gaia_a_g_val` float,<br />
`gaia_e_bp_min_rp_val` float,<br />
`gaia_phot_bp_rp_excess_factor` float,<br />
`gaia_astrometric_sigma5d_max` float,<br />
`gaia_astrometric_params_solved` tinyint,<br />
`flux_g` float,<br />
`flux_r` float,<br />
`flux_z` float,<br />
`flux_w1` float,<br />
`flux_w2` float,<br />
`flux_w3` float,<br />
`flux_w4` float,<br />
`flux_ivar_g` float,<br />
`flux_ivar_r` float,<br />
`flux_ivar_z` float,<br />
`flux_ivar_w1` float,<br />
`flux_ivar_w2` float,<br />
`flux_ivar_w3` float,<br />
`flux_ivar_w4` float,<br />
`fiberflux_g` float,<br />
`fiberflux_r` float,<br />
`fiberflux_z` float,<br />
`fibertotflux_g` float,<br />
`fibertotflux_r` float,<br />
`fibertotflux_z` float,<br />
`apflux_g` array<float>,<br />
`apflux_r` array<float>,<br />
`apflux_z` array<float>,<br />
`apflux_resid_g` array<float>,<br />
`apflux_resid_r` array<float>,<br />
`apflux_resid_z` array<float>,<br />
`apflux_ivar_g` array<float>,<br />
`apflux_ivar_r` array<float>,<br />
`apflux_ivar_z` array<float>,<br />
`mw_transmission_g` float,<br />
`mw_transmission_r` float,<br />
`mw_transmission_z` float,<br />
`mw_transmission_w1` float,<br />
`mw_transmission_w2` float,<br />
`mw_transmission_w3` float,<br />
`mw_transmission_w4` float,<br />
`nobs_g` smallint,<br />
`nobs_r` smallint,<br />
`nobs_z` smallint,<br />
`nobs_w1` smallint,<br />
`nobs_w2` smallint,<br />
`nobs_w3` smallint,<br />
`nobs_w4` smallint,<br />
`rchisq_g` float,<br />
`rchisq_r` float,<br />
`rchisq_z` float,<br />
`rchisq_w1` float,<br />
`rchisq_w2` float,<br />
`rchisq_w3` float,<br />
`rchisq_w4` float,<br />
`fracflux_g` float,<br />
`fracflux_r` float,<br />
`fracflux_z` float,<br />
`fracflux_w1` float,<br />
`fracflux_w2` float,<br />
`fracflux_w3` float,<br />
`fracflux_w4` float,<br />
`fracmasked_g` float,<br />
`fracmasked_r` float,<br />
`fracmasked_z` float,<br />
`fracin_g` float,<br />
`fracin_r` float,<br />
`fracin_z` float,<br />
`anymask_g` smallint,<br />
`anymask_r` smallint,<br />
`anymask_z` smallint,<br />
`allmask_g` smallint,<br />
`allmask_r` smallint,<br />
`allmask_z` smallint,<br />
`wisemask_w1` tinyint,<br />
`wisemask_w2` tinyint,<br />
`psfsize_g` float,<br />
`psfsize_r` float,<br />
`psfsize_z` float,<br />
`psfdepth_g` float,<br />
`psfdepth_r` float,<br />
`psfdepth_z` float,<br />
`galdepth_g` float,<br />
`galdepth_r` float,<br />
`galdepth_z` float,<br />
`psfdepth_w1` float,<br />
`psfdepth_w2` float,<br />
`psfdepth_w3` float,<br />
`psfdepth_w4` float,<br />
`wise_coadd_id` string,<br />
`lc_flux_w1` array<float>,<br />
`lc_flux_w2` array<float>,<br />
`lc_flux_ivar_w1` array<float>,<br />
`lc_flux_ivar_w2` array<float>,<br />
`lc_nobs_w1` array<smallint>,<br />
`lc_nobs_w2` array<smallint>,<br />
`lc_fracflux_w1` array<float>,<br />
`lc_fracflux_w2` array<float>,<br />
`lc_rchisq_w1` array<float>,<br />
`lc_rchisq_w2` array<float>,<br />
`lc_mjd_w1` array<double>,<br />
`lc_mjd_w2` array<double>,<br />
`fracdev` float,<br />
`fracdev_ivar` float,<br />
`shapeexp_r` float,<br />
`shapeexp_r_ivar` float,<br />
`shapeexp_e1` float,<br />
`shapeexp_e1_ivar` float,<br />
`shapeexp_e2` float,<br />
`shapeexp_e2_ivar` float,<br />
`shapedev_r` float,<br />
`shapedev_r_ivar` float,<br />
`shapedev_e1` float,<br />
`shapedev_e1_ivar` float,<br />
`shapedev_e2` float,<br />
`shapedev_e2_ivar` float<br />
)<br />
clustered by (<br />
release, brickid, objid<br />
)<br />
sorted by (<br />
release, brickid, objid<br />
)<br />
into 4096 buckets<br />
stored as orc<br />
;<br />
<br />
insert overwrite table cosmohub.legacy_survey_dr8<br />
select release, brickid, brickname, objid, brick_primary, brightblob, maskbits, type, ra, dec, ra_ivar, dec_ivar, bx, `by`, <br />
dchisq, ebv, mjd_min, mjd_max, ref_cat, ref_id, pmra, pmdec, parallax, pmra_ivar, pmdec_ivar, parallax_ivar, ref_epoch,<br />
gaia_pointsource, gaia_phot_g_mean_mag, gaia_phot_g_mean_flux_over_error, gaia_phot_g_n_obs,<br />
gaia_phot_bp_mean_mag, gaia_phot_bp_mean_flux_over_error, gaia_phot_bp_n_obs, gaia_phot_rp_mean_mag,<br />
gaia_phot_rp_mean_flux_over_error, gaia_phot_rp_n_obs, gaia_phot_variable_flag, gaia_astrometric_excess_noise,<br />
gaia_astrometric_excess_noise_sig, gaia_astrometric_n_obs_al, gaia_astrometric_n_good_obs_al,<br />
gaia_astrometric_weight_al, gaia_duplicated_source, gaia_a_g_val, gaia_e_bp_min_rp_val,<br />
gaia_phot_bp_rp_excess_factor, gaia_astrometric_sigma5d_max, gaia_astrometric_params_solved,<br />
flux_g, flux_r, flux_z, flux_w1, flux_w2, flux_w3, flux_w4, flux_ivar_g, flux_ivar_r,<br />
flux_ivar_z, flux_ivar_w1, flux_ivar_w2, flux_ivar_w3, flux_ivar_w4, fiberflux_g, fiberflux_r,<br />
fiberflux_z, fibertotflux_g, fibertotflux_r, fibertotflux_z, apflux_g, apflux_r, apflux_z,<br />
apflux_resid_g, apflux_resid_r, apflux_resid_z, apflux_ivar_g, apflux_ivar_r, apflux_ivar_z,<br />
mw_transmission_g, mw_transmission_r, mw_transmission_z, mw_transmission_w1, mw_transmission_w2,<br />
mw_transmission_w3, mw_transmission_w4, nobs_g, nobs_r, nobs_z, nobs_w1, nobs_w2, nobs_w3,<br />
nobs_w4, rchisq_g, rchisq_r, rchisq_z, rchisq_w1, rchisq_w2, rchisq_w3, rchisq_w4, fracflux_g,<br />
fracflux_r, fracflux_z, fracflux_w1, fracflux_w2, fracflux_w3, fracflux_w4, fracmasked_g,<br />
fracmasked_r, fracmasked_z, fracin_g, fracin_r, fracin_z, anymask_g, anymask_r, anymask_z,<br />
allmask_g, allmask_r, allmask_z, wisemask_w1, wisemask_w2, psfsize_g, psfsize_r, psfsize_z,<br />
psfdepth_g, psfdepth_r, psfdepth_z, galdepth_g, galdepth_r, galdepth_z, psfdepth_w1, psfdepth_w2,<br />
psfdepth_w3, psfdepth_w4, wise_coadd_id, lc_flux_w1, lc_flux_w2, lc_flux_ivar_w1, lc_flux_ivar_w2,<br />
lc_nobs_w1, lc_nobs_w2, lc_fracflux_w1, lc_fracflux_w2, lc_rchisq_w1, lc_rchisq_w2, lc_mjd_w1,<br />
lc_mjd_w2, fracdev, fracdev_ivar, shapeexp_r, shapeexp_r_ivar, shapeexp_e1, shapeexp_e1_ivar,<br />
shapeexp_e2, shapeexp_e2_ivar, shapedev_r, shapedev_r_ivar, shapedev_e1, shapedev_e1_ivar,<br />
shapedev_e2, shapedev_e2_ivar<br />
from tallada.dr8<br />
;<br />
<br />
=== Comments ===<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE release release smallint COMMENT "Unique integer denoting the camera and filter set used";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickid brickid int COMMENT "Brick ID [1,662174]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickname brickname string COMMENT "Name of brick, encoding the brick sky position, eg \"1126p222\" near RA=112.6, Dec=+22.2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE objid objid int COMMENT "Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brick_primary brick_primary boolean COMMENT "True if the object is within the brick boundary";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brightblob brightblob smallint COMMENT "bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE maskbits maskbits smallint COMMENT "bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits*maps, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE type type string COMMENT "Morphological model: PSF=stellar, REX=round exponential galaxy, DEV=deVauc, EXP=exponential, COMP=composite, DUP=Gaia source fit by different model.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra ra double COMMENT "Right ascension at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec dec double COMMENT "Declination at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra_ivar ra_ivar float COMMENT "Inverse variance of RA (no cosine term!), excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec_ivar dec_ivar float COMMENT "Inverse variance of DEC, excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE bx bx float COMMENT "X position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE by by float COMMENT "Y position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dchisq dchisq array<float> COMMENT "Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ebv ebv float COMMENT "Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_min mjd_min double COMMENT "Minimum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_max mjd_max double COMMENT "Maximum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_cat ref_cat string COMMENT "Reference catalog source for this star: T2 for Tycho-2, G2 for Gaia DR2, L2 for the LSLGA, empty otherwise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_id ref_id bigint COMMENT "Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b \"sourceid\" for Gaia-DR2 and LSLGA";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra pmra float COMMENT "Reference catalog proper motion in the RA direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec pmdec float COMMENT "Reference catalog proper motion in the Dec direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax parallax float COMMENT "Reference catalog parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra_ivar pmra_ivar float COMMENT "Reference catalog inverse-variance on pmra";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec_ivar pmdec_ivar float COMMENT "Reference catalog inverse-variance on pmdec";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax_ivar parallax_ivar float COMMENT "Reference catalog inverse-variance on parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_epoch ref_epoch float COMMENT "Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_pointsource gaia_pointsource boolean COMMENT "This Gaia DR2 source is believed to be a star, not a galaxy";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_mag gaia_phot_g_mean_mag float COMMENT "Gaia G band mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_flux_over_error gaia_phot_g_mean_flux_over_error float COMMENT "Gaia G band signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_n_obs gaia_phot_g_n_obs smallint COMMENT "Gaia G band number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_mag gaia_phot_bp_mean_mag float COMMENT "Gaia BP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_flux_over_error gaia_phot_bp_mean_flux_over_error float COMMENT "Gaia BP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_n_obs gaia_phot_bp_n_obs smallint COMMENT "Gaia BP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_mag gaia_phot_rp_mean_mag float COMMENT "Gaia RP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_flux_over_error gaia_phot_rp_mean_flux_over_error float COMMENT "Gaia RP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_n_obs gaia_phot_rp_n_obs smallint COMMENT "Gaia RP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_variable_flag gaia_phot_variable_flag boolean COMMENT "Gaia photometric variable flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise gaia_astrometric_excess_noise float COMMENT "Gaia astrometric excess noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise_sig gaia_astrometric_excess_noise_sig float COMMENT "Gaia astrometric excess noise uncertainty";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_obs_al gaia_astrometric_n_obs_al smallint COMMENT "Gaia number of astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_good_obs_al gaia_astrometric_n_good_obs_al smallint COMMENT "Gaia number of good astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_weight_al gaia_astrometric_weight_al float COMMENT "Gaia astrometric weight along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_duplicated_source gaia_duplicated_source boolean COMMENT "Gaia duplicated source flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_a_g_val gaia_a_g_val float COMMENT "Gaia line-of-sight extinction in the G band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_e_bp_min_rp_val gaia_e_bp_min_rp_val float COMMENT "Gaia line-of-sight reddening E(BP-RP)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_rp_excess_factor gaia_phot_bp_rp_excess_factor float COMMENT "Gaia BP/RP excess factor";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_sigma5d_max gaia_astrometric_sigma5d_max float COMMENT "Gaia longest semi-major axis of the 5-d error ellipsoid";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_params_solved gaia_astrometric_params_solved tinyint COMMENT "which astrometric parameters were estimated for a Gaia source";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_g flux_g float COMMENT "model flux in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_r flux_r float COMMENT "model flux in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_z flux_z float COMMENT "model flux in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w1 flux_w1 float COMMENT "WISE model flux in W1W1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w2 flux_w2 float COMMENT "WISE model flux in W2W2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w3 flux_w3 float COMMENT "WISE model flux in W3W3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w4 flux_w4 float COMMENT "WISE model flux in W4W4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_g flux_ivar_g float COMMENT "Inverse variance of flux_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_r flux_ivar_r float COMMENT "Inverse variance of flux_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_z flux_ivar_z float COMMENT "Inverse variance of flux_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w1 flux_ivar_w1 float COMMENT "Inverse variance of flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w2 flux_ivar_w2 float COMMENT "Inverse variance of flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w3 flux_ivar_w3 float COMMENT "Inverse variance of flux_w3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w4 flux_ivar_w4 float COMMENT "Inverse variance of flux_w4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_g fiberflux_g float COMMENT "Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_r fiberflux_r float COMMENT "Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_z fiberflux_z float COMMENT "Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_g fibertotflux_g float COMMENT "Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_r fibertotflux_r float COMMENT "Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_z fibertotflux_z float COMMENT "Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_g apflux_g array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_r apflux_r array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_z apflux_z array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_g apflux_resid_g array<float> COMMENT "aperture fluxes on the co-added residual images in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_r apflux_resid_r array<float> COMMENT "aperture fluxes on the co-added residual images in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_z apflux_resid_z array<float> COMMENT "aperture fluxes on the co-added residual images in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_g apflux_ivar_g array<float> COMMENT "Inverse variance of apflux_resid_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_r apflux_ivar_r array<float> COMMENT "Inverse variance of apflux_resid_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_z apflux_ivar_z array<float> COMMENT "Inverse variance of apflux_resid_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_g mw_transmission_g float COMMENT "Galactic transmission in gg filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_r mw_transmission_r float COMMENT "Galactic transmission in rr filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_z mw_transmission_z float COMMENT "Galactic transmission in zz filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w1 mw_transmission_w1 float COMMENT "Galactic transmission in W1W1 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w2 mw_transmission_w2 float COMMENT "Galactic transmission in W2W2 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w3 mw_transmission_w3 float COMMENT "Galactic transmission in W3W3 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w4 mw_transmission_w4 float COMMENT "Galactic transmission in W4W4 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_g nobs_g smallint COMMENT "Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_r nobs_r smallint COMMENT "Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_z nobs_z smallint COMMENT "Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w1 nobs_w1 smallint COMMENT "Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w2 nobs_w2 smallint COMMENT "Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w3 nobs_w3 smallint COMMENT "Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w4 nobs_w4 smallint COMMENT "Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_g rchisq_g float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_r rchisq_r float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_z rchisq_z float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w1 rchisq_w1 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W1W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w2 rchisq_w2 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W2W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w3 rchisq_w3 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W3W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w4 rchisq_w4 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W4W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_g fracflux_g float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in gg(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_r fracflux_r float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in rr(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_z fracflux_z float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in zz(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w1 fracflux_w1 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w2 fracflux_w2 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w3 fracflux_w3 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w4 fracflux_w4 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_g fracmasked_g float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_r fracmasked_r float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_z fracmasked_z float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_g fracin_g float COMMENT "Fraction of a source's flux within the blob in gg, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_r fracin_r float COMMENT "Fraction of a source's flux within the blob in rr, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_z fracin_z float COMMENT "Fraction of a source's flux within the blob in zz, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_g anymask_g smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_r anymask_r smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_z anymask_z smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_g allmask_g smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_r allmask_r smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_z allmask_z smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w1 wisemask_w1 tinyint COMMENT "W1 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w2 wisemask_w2 tinyint COMMENT "W2 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_g psfsize_g float COMMENT "Weighted average PSF FWHM in the gg band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_r psfsize_r float COMMENT "Weighted average PSF FWHM in the rr band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_z psfsize_z float COMMENT "Weighted average PSF FWHM in the zz band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_g psfdepth_g float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_r psfdepth_r float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_z psfdepth_z float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_g galdepth_g float COMMENT "As for psfdepth_g but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_r galdepth_r float COMMENT "As for psfdepth_r but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_z galdepth_z float COMMENT "As for psfdepth_z but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w1 psfdepth_w1 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w2 psfdepth_w2 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w3 psfdepth_w3 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w4 psfdepth_w4 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wise_coadd_id wise_coadd_id string COMMENT "unWISE coadd file name for the center of each object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w1 lc_flux_w1 array<float> COMMENT "flux_w1 in each of up to eleven unWISE coadd epochs (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w2 lc_flux_w2 array<float> COMMENT "flux_w2 in each of up to eleven unWISE coadd epochs (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w1 lc_flux_ivar_w1 array<float> COMMENT "Inverse variance of lc_flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w2 lc_flux_ivar_w2 array<float> COMMENT "Inverse variance of lc_flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w1 lc_nobs_w1 array<smallint> COMMENT "nobs_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w2 lc_nobs_w2 array<smallint> COMMENT "nobs_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w1 lc_fracflux_w1 array<float> COMMENT "fracflux_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w2 lc_fracflux_w2 array<float> COMMENT "fracflux_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w1 lc_rchisq_w1 array<float> COMMENT "rchisq_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w2 lc_rchisq_w2 array<float> COMMENT "rchisq_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w1 lc_mjd_w1 array<double> COMMENT "mjd_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w2 lc_mjd_w2 array<double> COMMENT "mjd_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev fracdev float COMMENT "Fraction of model in deVauc [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev_ivar fracdev_ivar float COMMENT "Inverse variance of fracdev";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r shapeexp_r float COMMENT "Half-light radius of exponential model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r_ivar shapeexp_r_ivar float COMMENT "Inverse variance of shapeexp_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1 shapeexp_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1_ivar shapeexp_e1_ivar float COMMENT "Inverse variance of shapeexp_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2 shapeexp_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2_ivar shapeexp_e2_ivar float COMMENT "Inverse variance of shapeexp_e2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r shapedev_r float COMMENT "Half-light radius of deVaucouleurs model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r_ivar shapedev_r_ivar float COMMENT "Inverse variance of shapedev_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1 shapedev_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1_ivar shapedev_e1_ivar float COMMENT "Inverse variance of shapedev_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2 shapedev_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2_ivar shapedev_e2_ivar float COMMENT "Inverse variance of shapedev_e2";<br />
<br />
== Sweep photo-z ==<br />
<br />
=== Raw data ===<br />
<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/north/sweep/<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/south/sweep/<br />
<br />
Checksums checked:<br />
<br />
$ cd north/sweep/8.0<br />
$ ls | wc -l<br />
287<br />
$ wc -l legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
286 (+ checksum file)<br />
$ sha256sum -c legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
OK<br />
<br />
$ cd north/sweep/8.0-photo-z<br />
$ ls | wc -l<br />
287<br />
$ wc -l legacysurvey_dr8_north_sweep_8.0-photo-z.sha256sum<br />
286 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_north_sweep_8.0-photo-z.sha256sum<br />
OK<br />
<br />
$ cd south/sweep/8.0<br />
$ ls | wc -l<br />
438<br />
$ wc -l legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
437 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
OK<br />
<br />
$ cd south/sweep/8.0-photo-z<br />
$ ls | wc -l<br />
438<br />
$ wc -l legacysurvey_dr8_south_sweep_8.0-photo-z.sha256sum<br />
437 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_south_sweep_8.0-photo-z.sha256sum<br />
OK<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to merge fits files and convert to parquet:<br />
<br />
path_base = '/cephfs/pic.es/astro/scratch/tallada/dr8-phz/'<br />
hdfs_base = '/user/tallada/data/legacysurvey-sweep'<br />
<br />
hs = ['south/sweep'] #, 'north/sweep']<br />
dir_src = '8.0'<br />
dir_phz = '8.0-photo-z'<br />
dir_dst = '8.0-merged'<br />
<br />
my_files = sorted(os.listdir(os.path.join(path_base, hs[0], dir_src)))<br />
<br />
def merge_phz(file_name):<br />
file_src = os.path.join(path_base, hs[0], dir_src, file_name)<br />
file_phz = os.path.join(path_base, hs[0], dir_phz, file_name[:-5]+'-pz.fits')<br />
file_dst = os.path.join(path_base, hs[0], dir_dst, file_name[:-5]+'-merged.fits')<br />
<br />
t_src = table.Table.read(file_src)<br />
t_phz = table.Table.read(file_phz)<br />
t_dst = table.hstack([t_src, t_phz], join_type='exact')<br />
<br />
os.unlink(file_dst)<br />
t_dst.write(file_dst)<br />
<br />
return file_name<br />
<br />
df = sc.parallelize(my_files, len(my_files))<br />
<br />
df.map(merge_phz).count()<br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + os.path.join(path_base, hs[0], dir_dst)<br />
) <br />
<br />
df.write.parquet(os.path.join(hdfs_base, hs[0]))<br />
<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE tallada.dr8_sweep (<br />
`RELEASE` SMALLINT,<br />
`BRICKID` INT,<br />
`BRICKNAME` STRING,<br />
`OBJID` INT,<br />
`TYPE` STRING,<br />
`RA` DOUBLE,<br />
`DEC` DOUBLE,<br />
`RA_IVAR` FLOAT,<br />
`DEC_IVAR` FLOAT,<br />
`DCHISQ` ARRAY<FLOAT>,<br />
`EBV` FLOAT,<br />
`FLUX_G` FLOAT,<br />
`FLUX_R` FLOAT,<br />
`FLUX_Z` FLOAT,<br />
`FLUX_W1` FLOAT,<br />
`FLUX_W2` FLOAT,<br />
`FLUX_W3` FLOAT,<br />
`FLUX_W4` FLOAT,<br />
`FLUX_IVAR_G` FLOAT,<br />
`FLUX_IVAR_R` FLOAT,<br />
`FLUX_IVAR_Z` FLOAT,<br />
`FLUX_IVAR_W1` FLOAT,<br />
`FLUX_IVAR_W2` FLOAT,<br />
`FLUX_IVAR_W3` FLOAT,<br />
`FLUX_IVAR_W4` FLOAT,<br />
`MW_TRANSMISSION_G` FLOAT,<br />
`MW_TRANSMISSION_R` FLOAT,<br />
`MW_TRANSMISSION_Z` FLOAT,<br />
`MW_TRANSMISSION_W1` FLOAT,<br />
`MW_TRANSMISSION_W2` FLOAT,<br />
`MW_TRANSMISSION_W3` FLOAT,<br />
`MW_TRANSMISSION_W4` FLOAT,<br />
`NOBS_G` INT,<br />
`NOBS_R` INT,<br />
`NOBS_Z` INT,<br />
`NOBS_W1` INT,<br />
`NOBS_W2` INT,<br />
`NOBS_W3` INT,<br />
`NOBS_W4` INT,<br />
`RCHISQ_G` FLOAT,<br />
`RCHISQ_R` FLOAT,<br />
`RCHISQ_Z` FLOAT,<br />
`RCHISQ_W1` FLOAT,<br />
`RCHISQ_W2` FLOAT,<br />
`RCHISQ_W3` FLOAT,<br />
`RCHISQ_W4` FLOAT,<br />
`FRACFLUX_G` FLOAT,<br />
`FRACFLUX_R` FLOAT,<br />
`FRACFLUX_Z` FLOAT,<br />
`FRACFLUX_W1` FLOAT,<br />
`FRACFLUX_W2` FLOAT,<br />
`FRACFLUX_W3` FLOAT,<br />
`FRACFLUX_W4` FLOAT,<br />
`FRACMASKED_G` FLOAT,<br />
`FRACMASKED_R` FLOAT,<br />
`FRACMASKED_Z` FLOAT,<br />
`FRACIN_G` FLOAT,<br />
`FRACIN_R` FLOAT,<br />
`FRACIN_Z` FLOAT,<br />
`ANYMASK_G` SMALLINT,<br />
`ANYMASK_R` SMALLINT,<br />
`ANYMASK_Z` SMALLINT,<br />
`ALLMASK_G` SMALLINT,<br />
`ALLMASK_R` SMALLINT,<br />
`ALLMASK_Z` SMALLINT,<br />
`WISEMASK_W1` TINYINT,<br />
`WISEMASK_W2` TINYINT,<br />
`PSFSIZE_G` FLOAT,<br />
`PSFSIZE_R` FLOAT,<br />
`PSFSIZE_Z` FLOAT,<br />
`PSFDEPTH_G` FLOAT,<br />
`PSFDEPTH_R` FLOAT,<br />
`PSFDEPTH_Z` FLOAT,<br />
`GALDEPTH_G` FLOAT,<br />
`GALDEPTH_R` FLOAT,<br />
`GALDEPTH_Z` FLOAT,<br />
`PSFDEPTH_W1` FLOAT,<br />
`PSFDEPTH_W2` FLOAT,<br />
`WISE_COADD_ID` STRING,<br />
`FRACDEV` FLOAT,<br />
`FRACDEV_IVAR` FLOAT,<br />
`SHAPEDEV_R` FLOAT,<br />
`SHAPEDEV_R_IVAR` FLOAT,<br />
`SHAPEDEV_E1` FLOAT,<br />
`SHAPEDEV_E1_IVAR` FLOAT,<br />
`SHAPEDEV_E2` FLOAT,<br />
`SHAPEDEV_E2_IVAR` FLOAT,<br />
`SHAPEEXP_R` FLOAT,<br />
`SHAPEEXP_R_IVAR` FLOAT,<br />
`SHAPEEXP_E1` FLOAT,<br />
`SHAPEEXP_E1_IVAR` FLOAT,<br />
`SHAPEEXP_E2` FLOAT,<br />
`SHAPEEXP_E2_IVAR` FLOAT,<br />
`FIBERFLUX_G` FLOAT,<br />
`FIBERFLUX_R` FLOAT,<br />
`FIBERFLUX_Z` FLOAT,<br />
`FIBERTOTFLUX_G` FLOAT,<br />
`FIBERTOTFLUX_R` FLOAT,<br />
`FIBERTOTFLUX_Z` FLOAT,<br />
`REF_CAT` STRING,<br />
`REF_ID` BIGINT,<br />
`REF_EPOCH` FLOAT,<br />
`GAIA_PHOT_G_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_G_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_ASTROMETRIC_EXCESS_NOISE` FLOAT,<br />
`GAIA_DUPLICATED_SOURCE` BOOLEAN,<br />
`GAIA_PHOT_BP_RP_EXCESS_FACTOR` FLOAT,<br />
`GAIA_ASTROMETRIC_SIGMA5D_MAX` FLOAT,<br />
`GAIA_ASTROMETRIC_PARAMS_SOLVED` TINYINT,<br />
`PARALLAX` FLOAT,<br />
`PARALLAX_IVAR` FLOAT,<br />
`PMRA` FLOAT,<br />
`PMRA_IVAR` FLOAT,<br />
`PMDEC` FLOAT,<br />
`PMDEC_IVAR` FLOAT,<br />
`MASKBITS` SMALLINT,<br />
`z_phot_mean` FLOAT,<br />
`z_phot_median` FLOAT,<br />
`z_phot_std` FLOAT,<br />
`z_phot_l68` FLOAT,<br />
`z_phot_u68` FLOAT,<br />
`z_phot_l95` FLOAT,<br />
`z_phot_u95` FLOAT,<br />
`z_spec` FLOAT,<br />
`survey` STRING,<br />
`training` BOOLEAN<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey-sweep'<br />
;<br />
<br />
=== Combined table ===<br />
<br />
CREATE TABLE cosmohub.legacysurvey_dr8_phz (<br />
release smallint COMMENT 'Unique integer denoting the camera and filter set used',<br />
brickid int COMMENT 'Brick ID [1,662174]',<br />
brickname string COMMENT 'Name of brick, encoding the brick sky position, eg "1126p222" near RA=112.6, Dec=+22.2',<br />
objid int COMMENT 'Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick',<br />
brick_primary boolean COMMENT 'True if the object is within the brick boundary',<br />
brightblob smallint COMMENT 'bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page',<br />
maskbits smallint COMMENT 'bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits* maps, as cataloged on the DR8 bitmasks page',<br />
type string COMMENT 'Morphological model: "PSF"=stellar, "REX"="round exponential galaxy", "DEV"=deVauc, "EXP"=exponential, "COMP"=composite, "DUP"=Gaia source fit by different model. Note that in some FITS readers, a trailing space may be appended for "PSF ", "DEV " and "EXP " since the column data type is a 4-character string',<br />
ra double COMMENT 'Right ascension at equinox J2000',<br />
dec double COMMENT 'Declination at equinox J2000',<br />
ra_ivar float COMMENT 'Inverse variance of RA (no cosine term!), excluding astrometric calibration errors',<br />
dec_ivar float COMMENT 'Inverse variance of DEC, excluding astrometric calibration errors',<br />
bx float COMMENT 'X position (0-indexed) of coordinates in brick image stack',<br />
`by` float COMMENT 'Y position (0-indexed) of coordinates in brick image stack',<br />
dchisq array<float> COMMENT 'Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.',<br />
ebv float COMMENT 'Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns',<br />
mjd_min double COMMENT 'Minimum Modified Julian Date of observations used to construct the model of this object',<br />
mjd_max double COMMENT 'Maximum Modified Julian Date of observations used to construct the model of this object',<br />
ref_cat string COMMENT 'Reference catalog source for this star: "T2" for Tycho-2, "G2" for Gaia DR2, "L2" for the LSLGA, empty otherwise',<br />
ref_id bigint COMMENT 'Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b "sourceid" for Gaia-DR2 and LSLGA',<br />
pmra float COMMENT 'Reference catalog proper motion in the RA direction',<br />
pmdec float COMMENT 'Reference catalog proper motion in the Dec direction',<br />
parallax float COMMENT 'Reference catalog parallax',<br />
pmra_ivar float COMMENT 'Reference catalog inverse-variance on pmra',<br />
pmdec_ivar float COMMENT 'Reference catalog inverse-variance on pmdec',<br />
parallax_ivar float COMMENT 'Reference catalog inverse-variance on parallax',<br />
ref_epoch float COMMENT 'Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)',<br />
gaia_pointsource boolean COMMENT 'This Gaia DR2 source is believed to be a star, not a galaxy',<br />
gaia_phot_g_mean_mag float COMMENT 'Gaia G band mag',<br />
gaia_phot_g_mean_flux_over_error float COMMENT 'Gaia G band signal-to-noise',<br />
gaia_phot_g_n_obs smallint COMMENT 'Gaia G band number of observations',<br />
gaia_phot_bp_mean_mag float COMMENT 'Gaia BP mag',<br />
gaia_phot_bp_mean_flux_over_error float COMMENT 'Gaia BP signal-to-noise',<br />
gaia_phot_bp_n_obs smallint COMMENT 'Gaia BP number of observations',<br />
gaia_phot_rp_mean_mag float COMMENT 'Gaia RP mag',<br />
gaia_phot_rp_mean_flux_over_error float COMMENT 'Gaia RP signal-to-noise',<br />
gaia_phot_rp_n_obs smallint COMMENT 'Gaia RP number of observations',<br />
gaia_phot_variable_flag boolean COMMENT 'Gaia photometric variable flag',<br />
gaia_astrometric_excess_noise float COMMENT 'Gaia astrometric excess noise',<br />
gaia_astrometric_excess_noise_sig float COMMENT 'Gaia astrometric excess noise uncertainty',<br />
gaia_astrometric_n_obs_al smallint COMMENT 'Gaia number of astrometric observations along scan direction',<br />
gaia_astrometric_n_good_obs_al smallint COMMENT 'Gaia number of good astrometric observations along scan direction',<br />
gaia_astrometric_weight_al float COMMENT 'Gaia astrometric weight along scan direction',<br />
gaia_duplicated_source boolean COMMENT 'Gaia duplicated source flag',<br />
gaia_a_g_val float COMMENT 'Gaia line-of-sight extinction in the G band',<br />
gaia_e_bp_min_rp_val float COMMENT 'Gaia line-of-sight reddening E(BP-RP)',<br />
gaia_phot_bp_rp_excess_factor float COMMENT 'Gaia BP/RP excess factor',<br />
gaia_astrometric_sigma5d_max float COMMENT 'Gaia longest semi-major axis of the 5-d error ellipsoid',<br />
gaia_astrometric_params_solved tinyint COMMENT 'which astrometric parameters were estimated for a Gaia source',<br />
flux_g float COMMENT 'model flux in gg',<br />
flux_r float COMMENT 'model flux in rr',<br />
flux_z float COMMENT 'model flux in zz',<br />
flux_w1 float COMMENT 'WISE model flux in W1W1 (AB system)',<br />
flux_w2 float COMMENT 'WISE model flux in W2W2 (AB)',<br />
flux_w3 float COMMENT 'WISE model flux in W3W3 (AB)',<br />
flux_w4 float COMMENT 'WISE model flux in W4W4 (AB)',<br />
flux_ivar_g float COMMENT 'Inverse variance of flux_g',<br />
flux_ivar_r float COMMENT 'Inverse variance of flux_r',<br />
flux_ivar_z float COMMENT 'Inverse variance of flux_z',<br />
flux_ivar_w1 float COMMENT 'Inverse variance of flux_w1 (AB system)',<br />
flux_ivar_w2 float COMMENT 'Inverse variance of flux_w2 (AB)',<br />
flux_ivar_w3 float COMMENT 'Inverse variance of flux_w3 (AB)',<br />
flux_ivar_w4 float COMMENT 'Inverse variance of flux_w4 (AB)',<br />
fiberflux_g float COMMENT 'Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_r float COMMENT 'Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_z float COMMENT 'Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fibertotflux_g float COMMENT 'Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_r float COMMENT 'Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_z float COMMENT 'Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
apflux_g array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg',<br />
apflux_r array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr',<br />
apflux_z array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz',<br />
apflux_resid_g array<float> COMMENT 'aperture fluxes on the co-added residual images in gg',<br />
apflux_resid_r array<float> COMMENT 'aperture fluxes on the co-added residual images in rr',<br />
apflux_resid_z array<float> COMMENT 'aperture fluxes on the co-added residual images in zz',<br />
apflux_ivar_g array<float> COMMENT 'Inverse variance of apflux_resid_g',<br />
apflux_ivar_r array<float> COMMENT 'Inverse variance of apflux_resid_r',<br />
apflux_ivar_z array<float> COMMENT 'Inverse variance of apflux_resid_z',<br />
mw_transmission_g float COMMENT 'Galactic transmission in gg filter in linear units [0, 1]',<br />
mw_transmission_r float COMMENT 'Galactic transmission in rr filter in linear units [0, 1]',<br />
mw_transmission_z float COMMENT 'Galactic transmission in zz filter in linear units [0, 1]',<br />
mw_transmission_w1 float COMMENT 'Galactic transmission in W1W1 filter in linear units [0, 1]',<br />
mw_transmission_w2 float COMMENT 'Galactic transmission in W2W2 filter in linear units [0, 1]',<br />
mw_transmission_w3 float COMMENT 'Galactic transmission in W3W3 filter in linear units [0, 1]',<br />
mw_transmission_w4 float COMMENT 'Galactic transmission in W4W4 filter in linear units [0, 1]',<br />
nobs_g smallint COMMENT 'Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)',<br />
nobs_r smallint COMMENT 'Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)',<br />
nobs_z smallint COMMENT 'Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)',<br />
nobs_w1 smallint COMMENT 'Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)',<br />
nobs_w2 smallint COMMENT 'Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)',<br />
nobs_w3 smallint COMMENT 'Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)',<br />
nobs_w4 smallint COMMENT 'Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)',<br />
rchisq_g float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in gg',<br />
rchisq_r float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in rr',<br />
rchisq_z float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in zz',<br />
rchisq_w1 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W1W1',<br />
rchisq_w2 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W2W2',<br />
rchisq_w3 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W3W3',<br />
rchisq_w4 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W4W4',<br />
fracflux_g float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in gg (typically [0,1])',<br />
fracflux_r float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in rr (typically [0,1])',<br />
fracflux_z float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in zz (typically [0,1])',<br />
fracflux_w1 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1 (typically [0,1])',<br />
fracflux_w2 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2 (typically [0,1])',<br />
fracflux_w3 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3 (typically [0,1])',<br />
fracflux_w4 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4 (typically [0,1])',<br />
fracmasked_g float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]',<br />
fracmasked_r float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]',<br />
fracmasked_z float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]',<br />
fracin_g float COMMENT 'Fraction of a source\'s flux within the blob in gg, near unity for real sources',<br />
fracin_r float COMMENT 'Fraction of a source\'s flux within the blob in rr, near unity for real sources',<br />
fracin_z float COMMENT 'Fraction of a source\'s flux within the blob in zz, near unity for real sources',<br />
anymask_g smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page',<br />
anymask_r smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page',<br />
anymask_z smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page',<br />
allmask_g smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page',<br />
allmask_r smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page',<br />
allmask_z smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page',<br />
wisemask_w1 tinyint COMMENT 'W1 bitmask as cataloged on the DR8 bitmasks page',<br />
wisemask_w2 tinyint COMMENT 'W2 bitmask as cataloged on the DR8 bitmasks page',<br />
psfsize_g float COMMENT 'Weighted average PSF FWHM in the gg band',<br />
psfsize_r float COMMENT 'Weighted average PSF FWHM in the rr band',<br />
psfsize_z float COMMENT 'Weighted average PSF FWHM in the zz band',<br />
psfdepth_g float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude',<br />
psfdepth_r float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude',<br />
psfdepth_z float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude',<br />
galdepth_g float COMMENT 'As for psfdepth_g but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_r float COMMENT 'As for psfdepth_r but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_z float COMMENT 'As for psfdepth_z but for a galaxy (0.45" exp, round) detection sensitivity',<br />
psfdepth_w1 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W1',<br />
psfdepth_w2 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W2',<br />
psfdepth_w3 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W3',<br />
psfdepth_w4 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W4',<br />
wise_coadd_id string COMMENT 'unWISE coadd file name for the center of each object',<br />
lc_flux_w1 array<float> COMMENT 'flux_w1 in each of up to eleven unWISE coadd epochs (AB system)',<br />
lc_flux_w2 array<float> COMMENT 'flux_w2 in each of up to eleven unWISE coadd epochs (AB)',<br />
lc_flux_ivar_w1 array<float> COMMENT 'Inverse variance of lc_flux_w1 (AB system)',<br />
lc_flux_ivar_w2 array<float> COMMENT 'Inverse variance of lc_flux_w2 (AB)',<br />
lc_nobs_w1 array<smallint> COMMENT 'nobs_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_nobs_w2 array<smallint> COMMENT 'nobs_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w1 array<float> COMMENT 'fracflux_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w2 array<float> COMMENT 'fracflux_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w1 array<float> COMMENT 'rchisq_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w2 array<float> COMMENT 'rchisq_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w1 array<double> COMMENT 'mjd_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w2 array<double> COMMENT 'mjd_w2 in each of up to eleven unWISE coadd epochs',<br />
fracdev float COMMENT 'Fraction of model in deVauc [0,1]',<br />
fracdev_ivar float COMMENT 'Inverse variance of fracdev',<br />
shapeexp_r float COMMENT 'Half-light radius of exponential model (>0)',<br />
shapeexp_r_ivar float COMMENT 'Inverse variance of shapeexp_r',<br />
shapeexp_e1 float COMMENT 'Ellipticity component 1',<br />
shapeexp_e1_ivar float COMMENT 'Inverse variance of shapeexp_e1',<br />
shapeexp_e2 float COMMENT 'Ellipticity component 2',<br />
shapeexp_e2_ivar float COMMENT 'Inverse variance of shapeexp_e2',<br />
shapedev_r float COMMENT 'Half-light radius of deVaucouleurs model (>0)',<br />
shapedev_r_ivar float COMMENT 'Inverse variance of shapedev_r',<br />
shapedev_e1 float COMMENT 'Ellipticity component 1',<br />
shapedev_e1_ivar float COMMENT 'Inverse variance of shapedev_e1',<br />
shapedev_e2 float COMMENT 'Ellipticity component 2',<br />
shapedev_e2_ivar float COMMENT 'Inverse variance of shapedev_e2',<br />
z_phot_mean FLOAT COMMENT 'photo-z derived from the mean of the photo-z PDF ',<br />
z_phot_median FLOAT COMMENT 'photo-z derived from the median of the photo-z PDF',<br />
z_phot_std FLOAT COMMENT 'standard deviation of the photo-z\'s derived from the photo-z PDF',<br />
z_phot_l68 FLOAT COMMENT 'lower bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_u68 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_l95 FLOAT COMMENT 'lower bound of the 95% confidence region, derived from the photo-z PDF',<br />
z_phot_u95 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_spec FLOAT COMMENT 'spectroscopic redshift, if available',<br />
survey STRING COMMENT 'source of the spectroscopic redshift',<br />
training BOOLEAN COMMENT 'whether or not the spectroscopic redshift is used in photometric redshift training'<br />
)<br />
CLUSTERED BY (<br />
release,<br />
brickid,<br />
objid)<br />
SORTED BY (<br />
release ASC,<br />
brickid ASC,<br />
objid ASC)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
=== Join insert ===<br />
<br />
INSERT OVERWRITE TABLE cosmohub.legacy_survey_dr8_phz<br />
SELECT<br />
dr8.*,<br />
phz.z_phot_mean,<br />
phz.z_phot_median,<br />
phz.z_phot_std,<br />
phz.z_phot_l68,<br />
phz.z_phot_u68,<br />
phz.z_phot_l95,<br />
phz.z_phot_u95,<br />
phz.z_spec,<br />
phz.survey,<br />
phz.training<br />
FROM cosmohub.legacy_survey_dr8 AS dr8<br />
LEFT JOIN tallada.dr8_sweep AS phz<br />
ON dr8.release = phz.release<br />
AND dr8.brickid = phz.brickid<br />
AND dr8.objid = phz.objid<br />
;<br />
<br />
== Queries for checking ==<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT release, brickid, objid, COUNT(release)<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
HAVING COUNT(release) > 1<br />
<br />
DR8 release brickid objid: DISTINCT(1.694.230.368) TOTAL(1.694.230.368)<br />
DR8 brickid objid: DISTINCT(1.654.429.115) TOTAL(1.694.230.368)<br />
<br />
PHZ release brickid objid: DISTINCT(1.646.664.729) TOTAL(1.646.664.897)<br />
PHZ brickid objid: DISTINCT(1.608.841.582) TOTAL(1.646.664.897)<br />
<br />
DR8: 1.694.230.368<br />
PHZ: 1.646.664.897<br />
----------------------------------------------------------------------<br />
-47.565.471</div>
Tallada
https://pwiki.pic.es/index.php?title=Legacy_Survey_DR9&diff=873
Legacy Survey DR9
2020-11-23T21:47:46Z
<p>Tallada: Created page with "== Original DR9 == === Raw data === Downloaded from NERSC /global/cfs/cdirs/cosmo/work/legacysurvey/dr9m Directories: south/tractor north/tractor Downloaded using bbcp f..."</p>
<hr />
<div>== Original DR9 ==<br />
=== Raw data ===<br />
<br />
Downloaded from NERSC<br />
/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m<br />
<br />
Directories:<br />
south/tractor<br />
north/tractor<br />
<br />
Downloaded using bbcp from services01.euclid into CephFS scratch:<br />
bbcp -s 16 -a -A -v -z -r -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" \<br />
"jcarrete@dtn01.nersc.gov:/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m/north/tractor" \<br />
/cephfs/pic.es/astro/scratch/tallada/dr9m/north/tractor<br />
bbcp -s 16 -a -A -v -z -r -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" \<br />
"jcarrete@dtn01.nersc.gov:/global/cfs/cdirs/cosmo/work/legacysurvey/dr9m/south/tractor" \<br />
/cephfs/pic.es/astro/scratch/tallada/dr9m/south/tractor<br />
<br />
CONTINUAR AQUI<br />
<br />
Checksums provided through sha256sum files, and checked using command:<br />
find * -type d | xargs -t -L1 -P24 sh -c 'cd $1; sha256sum --quiet -c *.sha256sum' cksum<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to convert fits files to parquet:<br />
<br />
path_base = '/cephfs/pic.es/cosmohub/legacysurvey'<br />
hdfs_base = '/user/tallada/data/legacysurvey'<br />
<br />
hs = ['h=north', 'h=south']<br />
rs = ['r={0:03d}'.format(x) for x in range(360)]<br />
<br />
for h in hs:<br />
for r in rs:<br />
p = os.path.join(path_base, h, r)<br />
if not os.path.exists(p):<br />
continue<br />
<br />
size = sum(os.path.getsize(os.path.join(p,f)) for f in os.listdir(p) if os.path.isfile(os.path.join(p,f)))<br />
<br />
print h, r, size <br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + p<br />
)<br />
<br />
df = df.repartition(int(math.ceil(size / (256*1024*1024.0))))<br />
<br />
df.write.parquet(os.path.join(hdfs_base, h, r))<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE tallada.dr8 (<br />
`release` smallint,<br />
`brickid` int,<br />
`brickname` string,<br />
`objid` int,<br />
`brick_primary` boolean,<br />
`brightblob` smallint,<br />
`maskbits` smallint,<br />
`type` string,<br />
`ra` double,<br />
`dec` double,<br />
`ra_ivar` float,<br />
`dec_ivar` float,<br />
`bx` float,<br />
`by` float,<br />
`dchisq` array<float>,<br />
`ebv` float,<br />
`mjd_min` double,<br />
`mjd_max` double,<br />
`ref_cat` string,<br />
`ref_id` bigint,<br />
`pmra` float,<br />
`pmdec` float,<br />
`parallax` float,<br />
`pmra_ivar` float,<br />
`pmdec_ivar` float,<br />
`parallax_ivar` float,<br />
`ref_epoch` float,<br />
`gaia_pointsource` boolean,<br />
`gaia_phot_g_mean_mag` float,<br />
`gaia_phot_g_mean_flux_over_error` float,<br />
`gaia_phot_g_n_obs` smallint,<br />
`gaia_phot_bp_mean_mag` float,<br />
`gaia_phot_bp_mean_flux_over_error` float,<br />
`gaia_phot_bp_n_obs` smallint,<br />
`gaia_phot_rp_mean_mag` float,<br />
`gaia_phot_rp_mean_flux_over_error` float,<br />
`gaia_phot_rp_n_obs` smallint,<br />
`gaia_phot_variable_flag` boolean,<br />
`gaia_astrometric_excess_noise` float,<br />
`gaia_astrometric_excess_noise_sig` float,<br />
`gaia_astrometric_n_obs_al` smallint,<br />
`gaia_astrometric_n_good_obs_al` smallint,<br />
`gaia_astrometric_weight_al` float,<br />
`gaia_duplicated_source` boolean,<br />
`gaia_a_g_val` float,<br />
`gaia_e_bp_min_rp_val` float,<br />
`gaia_phot_bp_rp_excess_factor` float,<br />
`gaia_astrometric_sigma5d_max` float,<br />
`gaia_astrometric_params_solved` tinyint,<br />
`flux_g` float,<br />
`flux_r` float,<br />
`flux_z` float,<br />
`flux_w1` float,<br />
`flux_w2` float,<br />
`flux_w3` float,<br />
`flux_w4` float,<br />
`flux_ivar_g` float,<br />
`flux_ivar_r` float,<br />
`flux_ivar_z` float,<br />
`flux_ivar_w1` float,<br />
`flux_ivar_w2` float,<br />
`flux_ivar_w3` float,<br />
`flux_ivar_w4` float,<br />
`fiberflux_g` float,<br />
`fiberflux_r` float,<br />
`fiberflux_z` float,<br />
`fibertotflux_g` float,<br />
`fibertotflux_r` float,<br />
`fibertotflux_z` float,<br />
`apflux_g` array<float>,<br />
`apflux_r` array<float>,<br />
`apflux_z` array<float>,<br />
`apflux_resid_g` array<float>,<br />
`apflux_resid_r` array<float>,<br />
`apflux_resid_z` array<float>,<br />
`apflux_ivar_g` array<float>,<br />
`apflux_ivar_r` array<float>,<br />
`apflux_ivar_z` array<float>,<br />
`mw_transmission_g` float,<br />
`mw_transmission_r` float,<br />
`mw_transmission_z` float,<br />
`mw_transmission_w1` float,<br />
`mw_transmission_w2` float,<br />
`mw_transmission_w3` float,<br />
`mw_transmission_w4` float,<br />
`nobs_g` smallint,<br />
`nobs_r` smallint,<br />
`nobs_z` smallint,<br />
`nobs_w1` smallint,<br />
`nobs_w2` smallint,<br />
`nobs_w3` smallint,<br />
`nobs_w4` smallint,<br />
`rchisq_g` float,<br />
`rchisq_r` float,<br />
`rchisq_z` float,<br />
`rchisq_w1` float,<br />
`rchisq_w2` float,<br />
`rchisq_w3` float,<br />
`rchisq_w4` float,<br />
`fracflux_g` float,<br />
`fracflux_r` float,<br />
`fracflux_z` float,<br />
`fracflux_w1` float,<br />
`fracflux_w2` float,<br />
`fracflux_w3` float,<br />
`fracflux_w4` float,<br />
`fracmasked_g` float,<br />
`fracmasked_r` float,<br />
`fracmasked_z` float,<br />
`fracin_g` float,<br />
`fracin_r` float,<br />
`fracin_z` float,<br />
`anymask_g` smallint,<br />
`anymask_r` smallint,<br />
`anymask_z` smallint,<br />
`allmask_g` smallint,<br />
`allmask_r` smallint,<br />
`allmask_z` smallint,<br />
`wisemask_w1` tinyint,<br />
`wisemask_w2` tinyint,<br />
`psfsize_g` float,<br />
`psfsize_r` float,<br />
`psfsize_z` float,<br />
`psfdepth_g` float,<br />
`psfdepth_r` float,<br />
`psfdepth_z` float,<br />
`galdepth_g` float,<br />
`galdepth_r` float,<br />
`galdepth_z` float,<br />
`psfdepth_w1` float,<br />
`psfdepth_w2` float,<br />
`psfdepth_w3` float,<br />
`psfdepth_w4` float,<br />
`wise_coadd_id` string,<br />
`lc_flux_w1` array<float>,<br />
`lc_flux_w2` array<float>,<br />
`lc_flux_ivar_w1` array<float>,<br />
`lc_flux_ivar_w2` array<float>,<br />
`lc_nobs_w1` array<smallint>,<br />
`lc_nobs_w2` array<smallint>,<br />
`lc_fracflux_w1` array<float>,<br />
`lc_fracflux_w2` array<float>,<br />
`lc_rchisq_w1` array<float>,<br />
`lc_rchisq_w2` array<float>,<br />
`lc_mjd_w1` array<double>,<br />
`lc_mjd_w2` array<double>,<br />
`fracdev` float,<br />
`fracdev_ivar` float,<br />
`shapeexp_r` float,<br />
`shapeexp_r_ivar` float,<br />
`shapeexp_e1` float,<br />
`shapeexp_e1_ivar` float,<br />
`shapeexp_e2` float,<br />
`shapeexp_e2_ivar` float,<br />
`shapedev_r` float,<br />
`shapedev_r_ivar` float,<br />
`shapedev_e1` float,<br />
`shapedev_e1_ivar` float,<br />
`shapedev_e2` float,<br />
`shapedev_e2_ivar` float<br />
)<br />
PARTITIONED BY (<br />
h string,<br />
r string<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey/dr8/'<br />
;<br />
<br />
=== Internal Table ===<br />
<br />
CREATE TABLE cosmohub.legacy_survey_dr8 (<br />
`release` smallint,<br />
`brickid` int,<br />
`brickname` string,<br />
`objid` int,<br />
`brick_primary` boolean,<br />
`brightblob` smallint,<br />
`maskbits` smallint,<br />
`type` string,<br />
`ra` double,<br />
`dec` double,<br />
`ra_ivar` float,<br />
`dec_ivar` float,<br />
`bx` float,<br />
`by` float,<br />
`dchisq` array<float>,<br />
`ebv` float,<br />
`mjd_min` double,<br />
`mjd_max` double,<br />
`ref_cat` string,<br />
`ref_id` bigint,<br />
`pmra` float,<br />
`pmdec` float,<br />
`parallax` float,<br />
`pmra_ivar` float,<br />
`pmdec_ivar` float,<br />
`parallax_ivar` float,<br />
`ref_epoch` float,<br />
`gaia_pointsource` boolean,<br />
`gaia_phot_g_mean_mag` float,<br />
`gaia_phot_g_mean_flux_over_error` float,<br />
`gaia_phot_g_n_obs` smallint,<br />
`gaia_phot_bp_mean_mag` float,<br />
`gaia_phot_bp_mean_flux_over_error` float,<br />
`gaia_phot_bp_n_obs` smallint,<br />
`gaia_phot_rp_mean_mag` float,<br />
`gaia_phot_rp_mean_flux_over_error` float,<br />
`gaia_phot_rp_n_obs` smallint,<br />
`gaia_phot_variable_flag` boolean,<br />
`gaia_astrometric_excess_noise` float,<br />
`gaia_astrometric_excess_noise_sig` float,<br />
`gaia_astrometric_n_obs_al` smallint,<br />
`gaia_astrometric_n_good_obs_al` smallint,<br />
`gaia_astrometric_weight_al` float,<br />
`gaia_duplicated_source` boolean,<br />
`gaia_a_g_val` float,<br />
`gaia_e_bp_min_rp_val` float,<br />
`gaia_phot_bp_rp_excess_factor` float,<br />
`gaia_astrometric_sigma5d_max` float,<br />
`gaia_astrometric_params_solved` tinyint,<br />
`flux_g` float,<br />
`flux_r` float,<br />
`flux_z` float,<br />
`flux_w1` float,<br />
`flux_w2` float,<br />
`flux_w3` float,<br />
`flux_w4` float,<br />
`flux_ivar_g` float,<br />
`flux_ivar_r` float,<br />
`flux_ivar_z` float,<br />
`flux_ivar_w1` float,<br />
`flux_ivar_w2` float,<br />
`flux_ivar_w3` float,<br />
`flux_ivar_w4` float,<br />
`fiberflux_g` float,<br />
`fiberflux_r` float,<br />
`fiberflux_z` float,<br />
`fibertotflux_g` float,<br />
`fibertotflux_r` float,<br />
`fibertotflux_z` float,<br />
`apflux_g` array<float>,<br />
`apflux_r` array<float>,<br />
`apflux_z` array<float>,<br />
`apflux_resid_g` array<float>,<br />
`apflux_resid_r` array<float>,<br />
`apflux_resid_z` array<float>,<br />
`apflux_ivar_g` array<float>,<br />
`apflux_ivar_r` array<float>,<br />
`apflux_ivar_z` array<float>,<br />
`mw_transmission_g` float,<br />
`mw_transmission_r` float,<br />
`mw_transmission_z` float,<br />
`mw_transmission_w1` float,<br />
`mw_transmission_w2` float,<br />
`mw_transmission_w3` float,<br />
`mw_transmission_w4` float,<br />
`nobs_g` smallint,<br />
`nobs_r` smallint,<br />
`nobs_z` smallint,<br />
`nobs_w1` smallint,<br />
`nobs_w2` smallint,<br />
`nobs_w3` smallint,<br />
`nobs_w4` smallint,<br />
`rchisq_g` float,<br />
`rchisq_r` float,<br />
`rchisq_z` float,<br />
`rchisq_w1` float,<br />
`rchisq_w2` float,<br />
`rchisq_w3` float,<br />
`rchisq_w4` float,<br />
`fracflux_g` float,<br />
`fracflux_r` float,<br />
`fracflux_z` float,<br />
`fracflux_w1` float,<br />
`fracflux_w2` float,<br />
`fracflux_w3` float,<br />
`fracflux_w4` float,<br />
`fracmasked_g` float,<br />
`fracmasked_r` float,<br />
`fracmasked_z` float,<br />
`fracin_g` float,<br />
`fracin_r` float,<br />
`fracin_z` float,<br />
`anymask_g` smallint,<br />
`anymask_r` smallint,<br />
`anymask_z` smallint,<br />
`allmask_g` smallint,<br />
`allmask_r` smallint,<br />
`allmask_z` smallint,<br />
`wisemask_w1` tinyint,<br />
`wisemask_w2` tinyint,<br />
`psfsize_g` float,<br />
`psfsize_r` float,<br />
`psfsize_z` float,<br />
`psfdepth_g` float,<br />
`psfdepth_r` float,<br />
`psfdepth_z` float,<br />
`galdepth_g` float,<br />
`galdepth_r` float,<br />
`galdepth_z` float,<br />
`psfdepth_w1` float,<br />
`psfdepth_w2` float,<br />
`psfdepth_w3` float,<br />
`psfdepth_w4` float,<br />
`wise_coadd_id` string,<br />
`lc_flux_w1` array<float>,<br />
`lc_flux_w2` array<float>,<br />
`lc_flux_ivar_w1` array<float>,<br />
`lc_flux_ivar_w2` array<float>,<br />
`lc_nobs_w1` array<smallint>,<br />
`lc_nobs_w2` array<smallint>,<br />
`lc_fracflux_w1` array<float>,<br />
`lc_fracflux_w2` array<float>,<br />
`lc_rchisq_w1` array<float>,<br />
`lc_rchisq_w2` array<float>,<br />
`lc_mjd_w1` array<double>,<br />
`lc_mjd_w2` array<double>,<br />
`fracdev` float,<br />
`fracdev_ivar` float,<br />
`shapeexp_r` float,<br />
`shapeexp_r_ivar` float,<br />
`shapeexp_e1` float,<br />
`shapeexp_e1_ivar` float,<br />
`shapeexp_e2` float,<br />
`shapeexp_e2_ivar` float,<br />
`shapedev_r` float,<br />
`shapedev_r_ivar` float,<br />
`shapedev_e1` float,<br />
`shapedev_e1_ivar` float,<br />
`shapedev_e2` float,<br />
`shapedev_e2_ivar` float<br />
)<br />
clustered by (<br />
release, brickid, objid<br />
)<br />
sorted by (<br />
release, brickid, objid<br />
)<br />
into 4096 buckets<br />
stored as orc<br />
;<br />
<br />
insert overwrite table cosmohub.legacy_survey_dr8<br />
select release, brickid, brickname, objid, brick_primary, brightblob, maskbits, type, ra, dec, ra_ivar, dec_ivar, bx, `by`, <br />
dchisq, ebv, mjd_min, mjd_max, ref_cat, ref_id, pmra, pmdec, parallax, pmra_ivar, pmdec_ivar, parallax_ivar, ref_epoch,<br />
gaia_pointsource, gaia_phot_g_mean_mag, gaia_phot_g_mean_flux_over_error, gaia_phot_g_n_obs,<br />
gaia_phot_bp_mean_mag, gaia_phot_bp_mean_flux_over_error, gaia_phot_bp_n_obs, gaia_phot_rp_mean_mag,<br />
gaia_phot_rp_mean_flux_over_error, gaia_phot_rp_n_obs, gaia_phot_variable_flag, gaia_astrometric_excess_noise,<br />
gaia_astrometric_excess_noise_sig, gaia_astrometric_n_obs_al, gaia_astrometric_n_good_obs_al,<br />
gaia_astrometric_weight_al, gaia_duplicated_source, gaia_a_g_val, gaia_e_bp_min_rp_val,<br />
gaia_phot_bp_rp_excess_factor, gaia_astrometric_sigma5d_max, gaia_astrometric_params_solved,<br />
flux_g, flux_r, flux_z, flux_w1, flux_w2, flux_w3, flux_w4, flux_ivar_g, flux_ivar_r,<br />
flux_ivar_z, flux_ivar_w1, flux_ivar_w2, flux_ivar_w3, flux_ivar_w4, fiberflux_g, fiberflux_r,<br />
fiberflux_z, fibertotflux_g, fibertotflux_r, fibertotflux_z, apflux_g, apflux_r, apflux_z,<br />
apflux_resid_g, apflux_resid_r, apflux_resid_z, apflux_ivar_g, apflux_ivar_r, apflux_ivar_z,<br />
mw_transmission_g, mw_transmission_r, mw_transmission_z, mw_transmission_w1, mw_transmission_w2,<br />
mw_transmission_w3, mw_transmission_w4, nobs_g, nobs_r, nobs_z, nobs_w1, nobs_w2, nobs_w3,<br />
nobs_w4, rchisq_g, rchisq_r, rchisq_z, rchisq_w1, rchisq_w2, rchisq_w3, rchisq_w4, fracflux_g,<br />
fracflux_r, fracflux_z, fracflux_w1, fracflux_w2, fracflux_w3, fracflux_w4, fracmasked_g,<br />
fracmasked_r, fracmasked_z, fracin_g, fracin_r, fracin_z, anymask_g, anymask_r, anymask_z,<br />
allmask_g, allmask_r, allmask_z, wisemask_w1, wisemask_w2, psfsize_g, psfsize_r, psfsize_z,<br />
psfdepth_g, psfdepth_r, psfdepth_z, galdepth_g, galdepth_r, galdepth_z, psfdepth_w1, psfdepth_w2,<br />
psfdepth_w3, psfdepth_w4, wise_coadd_id, lc_flux_w1, lc_flux_w2, lc_flux_ivar_w1, lc_flux_ivar_w2,<br />
lc_nobs_w1, lc_nobs_w2, lc_fracflux_w1, lc_fracflux_w2, lc_rchisq_w1, lc_rchisq_w2, lc_mjd_w1,<br />
lc_mjd_w2, fracdev, fracdev_ivar, shapeexp_r, shapeexp_r_ivar, shapeexp_e1, shapeexp_e1_ivar,<br />
shapeexp_e2, shapeexp_e2_ivar, shapedev_r, shapedev_r_ivar, shapedev_e1, shapedev_e1_ivar,<br />
shapedev_e2, shapedev_e2_ivar<br />
from tallada.dr8<br />
;<br />
<br />
=== Comments ===<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE release release smallint COMMENT "Unique integer denoting the camera and filter set used";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickid brickid int COMMENT "Brick ID [1,662174]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickname brickname string COMMENT "Name of brick, encoding the brick sky position, eg \"1126p222\" near RA=112.6, Dec=+22.2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE objid objid int COMMENT "Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brick_primary brick_primary boolean COMMENT "True if the object is within the brick boundary";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brightblob brightblob smallint COMMENT "bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE maskbits maskbits smallint COMMENT "bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits*maps, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE type type string COMMENT "Morphological model: PSF=stellar, REX=round exponential galaxy, DEV=deVauc, EXP=exponential, COMP=composite, DUP=Gaia source fit by different model.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra ra double COMMENT "Right ascension at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec dec double COMMENT "Declination at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra_ivar ra_ivar float COMMENT "Inverse variance of RA (no cosine term!), excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec_ivar dec_ivar float COMMENT "Inverse variance of DEC, excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE bx bx float COMMENT "X position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE by by float COMMENT "Y position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dchisq dchisq array<float> COMMENT "Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ebv ebv float COMMENT "Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_min mjd_min double COMMENT "Minimum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_max mjd_max double COMMENT "Maximum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_cat ref_cat string COMMENT "Reference catalog source for this star: T2 for Tycho-2, G2 for Gaia DR2, L2 for the LSLGA, empty otherwise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_id ref_id bigint COMMENT "Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b \"sourceid\" for Gaia-DR2 and LSLGA";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra pmra float COMMENT "Reference catalog proper motion in the RA direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec pmdec float COMMENT "Reference catalog proper motion in the Dec direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax parallax float COMMENT "Reference catalog parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra_ivar pmra_ivar float COMMENT "Reference catalog inverse-variance on pmra";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec_ivar pmdec_ivar float COMMENT "Reference catalog inverse-variance on pmdec";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax_ivar parallax_ivar float COMMENT "Reference catalog inverse-variance on parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_epoch ref_epoch float COMMENT "Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_pointsource gaia_pointsource boolean COMMENT "This Gaia DR2 source is believed to be a star, not a galaxy";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_mag gaia_phot_g_mean_mag float COMMENT "Gaia G band mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_flux_over_error gaia_phot_g_mean_flux_over_error float COMMENT "Gaia G band signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_n_obs gaia_phot_g_n_obs smallint COMMENT "Gaia G band number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_mag gaia_phot_bp_mean_mag float COMMENT "Gaia BP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_flux_over_error gaia_phot_bp_mean_flux_over_error float COMMENT "Gaia BP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_n_obs gaia_phot_bp_n_obs smallint COMMENT "Gaia BP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_mag gaia_phot_rp_mean_mag float COMMENT "Gaia RP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_flux_over_error gaia_phot_rp_mean_flux_over_error float COMMENT "Gaia RP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_n_obs gaia_phot_rp_n_obs smallint COMMENT "Gaia RP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_variable_flag gaia_phot_variable_flag boolean COMMENT "Gaia photometric variable flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise gaia_astrometric_excess_noise float COMMENT "Gaia astrometric excess noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise_sig gaia_astrometric_excess_noise_sig float COMMENT "Gaia astrometric excess noise uncertainty";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_obs_al gaia_astrometric_n_obs_al smallint COMMENT "Gaia number of astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_good_obs_al gaia_astrometric_n_good_obs_al smallint COMMENT "Gaia number of good astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_weight_al gaia_astrometric_weight_al float COMMENT "Gaia astrometric weight along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_duplicated_source gaia_duplicated_source boolean COMMENT "Gaia duplicated source flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_a_g_val gaia_a_g_val float COMMENT "Gaia line-of-sight extinction in the G band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_e_bp_min_rp_val gaia_e_bp_min_rp_val float COMMENT "Gaia line-of-sight reddening E(BP-RP)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_rp_excess_factor gaia_phot_bp_rp_excess_factor float COMMENT "Gaia BP/RP excess factor";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_sigma5d_max gaia_astrometric_sigma5d_max float COMMENT "Gaia longest semi-major axis of the 5-d error ellipsoid";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_params_solved gaia_astrometric_params_solved tinyint COMMENT "which astrometric parameters were estimated for a Gaia source";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_g flux_g float COMMENT "model flux in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_r flux_r float COMMENT "model flux in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_z flux_z float COMMENT "model flux in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w1 flux_w1 float COMMENT "WISE model flux in W1W1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w2 flux_w2 float COMMENT "WISE model flux in W2W2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w3 flux_w3 float COMMENT "WISE model flux in W3W3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w4 flux_w4 float COMMENT "WISE model flux in W4W4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_g flux_ivar_g float COMMENT "Inverse variance of flux_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_r flux_ivar_r float COMMENT "Inverse variance of flux_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_z flux_ivar_z float COMMENT "Inverse variance of flux_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w1 flux_ivar_w1 float COMMENT "Inverse variance of flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w2 flux_ivar_w2 float COMMENT "Inverse variance of flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w3 flux_ivar_w3 float COMMENT "Inverse variance of flux_w3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w4 flux_ivar_w4 float COMMENT "Inverse variance of flux_w4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_g fiberflux_g float COMMENT "Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_r fiberflux_r float COMMENT "Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_z fiberflux_z float COMMENT "Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_g fibertotflux_g float COMMENT "Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_r fibertotflux_r float COMMENT "Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_z fibertotflux_z float COMMENT "Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_g apflux_g array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_r apflux_r array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_z apflux_z array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_g apflux_resid_g array<float> COMMENT "aperture fluxes on the co-added residual images in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_r apflux_resid_r array<float> COMMENT "aperture fluxes on the co-added residual images in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_z apflux_resid_z array<float> COMMENT "aperture fluxes on the co-added residual images in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_g apflux_ivar_g array<float> COMMENT "Inverse variance of apflux_resid_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_r apflux_ivar_r array<float> COMMENT "Inverse variance of apflux_resid_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_z apflux_ivar_z array<float> COMMENT "Inverse variance of apflux_resid_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_g mw_transmission_g float COMMENT "Galactic transmission in gg filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_r mw_transmission_r float COMMENT "Galactic transmission in rr filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_z mw_transmission_z float COMMENT "Galactic transmission in zz filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w1 mw_transmission_w1 float COMMENT "Galactic transmission in W1W1 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w2 mw_transmission_w2 float COMMENT "Galactic transmission in W2W2 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w3 mw_transmission_w3 float COMMENT "Galactic transmission in W3W3 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w4 mw_transmission_w4 float COMMENT "Galactic transmission in W4W4 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_g nobs_g smallint COMMENT "Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_r nobs_r smallint COMMENT "Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_z nobs_z smallint COMMENT "Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w1 nobs_w1 smallint COMMENT "Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w2 nobs_w2 smallint COMMENT "Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w3 nobs_w3 smallint COMMENT "Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w4 nobs_w4 smallint COMMENT "Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_g rchisq_g float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_r rchisq_r float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_z rchisq_z float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w1 rchisq_w1 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W1W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w2 rchisq_w2 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W2W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w3 rchisq_w3 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W3W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w4 rchisq_w4 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W4W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_g fracflux_g float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in gg(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_r fracflux_r float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in rr(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_z fracflux_z float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in zz(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w1 fracflux_w1 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w2 fracflux_w2 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w3 fracflux_w3 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w4 fracflux_w4 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_g fracmasked_g float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_r fracmasked_r float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_z fracmasked_z float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_g fracin_g float COMMENT "Fraction of a source's flux within the blob in gg, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_r fracin_r float COMMENT "Fraction of a source's flux within the blob in rr, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_z fracin_z float COMMENT "Fraction of a source's flux within the blob in zz, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_g anymask_g smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_r anymask_r smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_z anymask_z smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_g allmask_g smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_r allmask_r smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_z allmask_z smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w1 wisemask_w1 tinyint COMMENT "W1 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w2 wisemask_w2 tinyint COMMENT "W2 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_g psfsize_g float COMMENT "Weighted average PSF FWHM in the gg band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_r psfsize_r float COMMENT "Weighted average PSF FWHM in the rr band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_z psfsize_z float COMMENT "Weighted average PSF FWHM in the zz band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_g psfdepth_g float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_r psfdepth_r float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_z psfdepth_z float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_g galdepth_g float COMMENT "As for psfdepth_g but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_r galdepth_r float COMMENT "As for psfdepth_r but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_z galdepth_z float COMMENT "As for psfdepth_z but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w1 psfdepth_w1 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w2 psfdepth_w2 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w3 psfdepth_w3 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w4 psfdepth_w4 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wise_coadd_id wise_coadd_id string COMMENT "unWISE coadd file name for the center of each object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w1 lc_flux_w1 array<float> COMMENT "flux_w1 in each of up to eleven unWISE coadd epochs (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w2 lc_flux_w2 array<float> COMMENT "flux_w2 in each of up to eleven unWISE coadd epochs (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w1 lc_flux_ivar_w1 array<float> COMMENT "Inverse variance of lc_flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w2 lc_flux_ivar_w2 array<float> COMMENT "Inverse variance of lc_flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w1 lc_nobs_w1 array<smallint> COMMENT "nobs_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w2 lc_nobs_w2 array<smallint> COMMENT "nobs_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w1 lc_fracflux_w1 array<float> COMMENT "fracflux_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w2 lc_fracflux_w2 array<float> COMMENT "fracflux_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w1 lc_rchisq_w1 array<float> COMMENT "rchisq_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w2 lc_rchisq_w2 array<float> COMMENT "rchisq_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w1 lc_mjd_w1 array<double> COMMENT "mjd_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w2 lc_mjd_w2 array<double> COMMENT "mjd_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev fracdev float COMMENT "Fraction of model in deVauc [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev_ivar fracdev_ivar float COMMENT "Inverse variance of fracdev";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r shapeexp_r float COMMENT "Half-light radius of exponential model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r_ivar shapeexp_r_ivar float COMMENT "Inverse variance of shapeexp_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1 shapeexp_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1_ivar shapeexp_e1_ivar float COMMENT "Inverse variance of shapeexp_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2 shapeexp_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2_ivar shapeexp_e2_ivar float COMMENT "Inverse variance of shapeexp_e2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r shapedev_r float COMMENT "Half-light radius of deVaucouleurs model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r_ivar shapedev_r_ivar float COMMENT "Inverse variance of shapedev_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1 shapedev_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1_ivar shapedev_e1_ivar float COMMENT "Inverse variance of shapedev_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2 shapedev_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2_ivar shapedev_e2_ivar float COMMENT "Inverse variance of shapedev_e2";<br />
<br />
== Sweep photo-z ==<br />
<br />
=== Raw data ===<br />
<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/north/sweep/<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/south/sweep/<br />
<br />
Checksums checked:<br />
<br />
$ cd north/sweep/8.0<br />
$ ls | wc -l<br />
287<br />
$ wc -l legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
286 (+ checksum file)<br />
$ sha256sum -c legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
OK<br />
<br />
$ cd north/sweep/8.0-photo-z<br />
$ ls | wc -l<br />
287<br />
$ wc -l legacysurvey_dr8_north_sweep_8.0-photo-z.sha256sum<br />
286 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_north_sweep_8.0-photo-z.sha256sum<br />
OK<br />
<br />
$ cd south/sweep/8.0<br />
$ ls | wc -l<br />
438<br />
$ wc -l legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
437 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
OK<br />
<br />
$ cd south/sweep/8.0-photo-z<br />
$ ls | wc -l<br />
438<br />
$ wc -l legacysurvey_dr8_south_sweep_8.0-photo-z.sha256sum<br />
437 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_south_sweep_8.0-photo-z.sha256sum<br />
OK<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to merge fits files and convert to parquet:<br />
<br />
path_base = '/cephfs/pic.es/astro/scratch/tallada/dr8-phz/'<br />
hdfs_base = '/user/tallada/data/legacysurvey-sweep'<br />
<br />
hs = ['south/sweep'] #, 'north/sweep']<br />
dir_src = '8.0'<br />
dir_phz = '8.0-photo-z'<br />
dir_dst = '8.0-merged'<br />
<br />
my_files = sorted(os.listdir(os.path.join(path_base, hs[0], dir_src)))<br />
<br />
def merge_phz(file_name):<br />
file_src = os.path.join(path_base, hs[0], dir_src, file_name)<br />
file_phz = os.path.join(path_base, hs[0], dir_phz, file_name[:-5]+'-pz.fits')<br />
file_dst = os.path.join(path_base, hs[0], dir_dst, file_name[:-5]+'-merged.fits')<br />
<br />
t_src = table.Table.read(file_src)<br />
t_phz = table.Table.read(file_phz)<br />
t_dst = table.hstack([t_src, t_phz], join_type='exact')<br />
<br />
os.unlink(file_dst)<br />
t_dst.write(file_dst)<br />
<br />
return file_name<br />
<br />
df = sc.parallelize(my_files, len(my_files))<br />
<br />
df.map(merge_phz).count()<br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + os.path.join(path_base, hs[0], dir_dst)<br />
) <br />
<br />
df.write.parquet(os.path.join(hdfs_base, hs[0]))<br />
<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE tallada.dr8_sweep (<br />
`RELEASE` SMALLINT,<br />
`BRICKID` INT,<br />
`BRICKNAME` STRING,<br />
`OBJID` INT,<br />
`TYPE` STRING,<br />
`RA` DOUBLE,<br />
`DEC` DOUBLE,<br />
`RA_IVAR` FLOAT,<br />
`DEC_IVAR` FLOAT,<br />
`DCHISQ` ARRAY<FLOAT>,<br />
`EBV` FLOAT,<br />
`FLUX_G` FLOAT,<br />
`FLUX_R` FLOAT,<br />
`FLUX_Z` FLOAT,<br />
`FLUX_W1` FLOAT,<br />
`FLUX_W2` FLOAT,<br />
`FLUX_W3` FLOAT,<br />
`FLUX_W4` FLOAT,<br />
`FLUX_IVAR_G` FLOAT,<br />
`FLUX_IVAR_R` FLOAT,<br />
`FLUX_IVAR_Z` FLOAT,<br />
`FLUX_IVAR_W1` FLOAT,<br />
`FLUX_IVAR_W2` FLOAT,<br />
`FLUX_IVAR_W3` FLOAT,<br />
`FLUX_IVAR_W4` FLOAT,<br />
`MW_TRANSMISSION_G` FLOAT,<br />
`MW_TRANSMISSION_R` FLOAT,<br />
`MW_TRANSMISSION_Z` FLOAT,<br />
`MW_TRANSMISSION_W1` FLOAT,<br />
`MW_TRANSMISSION_W2` FLOAT,<br />
`MW_TRANSMISSION_W3` FLOAT,<br />
`MW_TRANSMISSION_W4` FLOAT,<br />
`NOBS_G` INT,<br />
`NOBS_R` INT,<br />
`NOBS_Z` INT,<br />
`NOBS_W1` INT,<br />
`NOBS_W2` INT,<br />
`NOBS_W3` INT,<br />
`NOBS_W4` INT,<br />
`RCHISQ_G` FLOAT,<br />
`RCHISQ_R` FLOAT,<br />
`RCHISQ_Z` FLOAT,<br />
`RCHISQ_W1` FLOAT,<br />
`RCHISQ_W2` FLOAT,<br />
`RCHISQ_W3` FLOAT,<br />
`RCHISQ_W4` FLOAT,<br />
`FRACFLUX_G` FLOAT,<br />
`FRACFLUX_R` FLOAT,<br />
`FRACFLUX_Z` FLOAT,<br />
`FRACFLUX_W1` FLOAT,<br />
`FRACFLUX_W2` FLOAT,<br />
`FRACFLUX_W3` FLOAT,<br />
`FRACFLUX_W4` FLOAT,<br />
`FRACMASKED_G` FLOAT,<br />
`FRACMASKED_R` FLOAT,<br />
`FRACMASKED_Z` FLOAT,<br />
`FRACIN_G` FLOAT,<br />
`FRACIN_R` FLOAT,<br />
`FRACIN_Z` FLOAT,<br />
`ANYMASK_G` SMALLINT,<br />
`ANYMASK_R` SMALLINT,<br />
`ANYMASK_Z` SMALLINT,<br />
`ALLMASK_G` SMALLINT,<br />
`ALLMASK_R` SMALLINT,<br />
`ALLMASK_Z` SMALLINT,<br />
`WISEMASK_W1` TINYINT,<br />
`WISEMASK_W2` TINYINT,<br />
`PSFSIZE_G` FLOAT,<br />
`PSFSIZE_R` FLOAT,<br />
`PSFSIZE_Z` FLOAT,<br />
`PSFDEPTH_G` FLOAT,<br />
`PSFDEPTH_R` FLOAT,<br />
`PSFDEPTH_Z` FLOAT,<br />
`GALDEPTH_G` FLOAT,<br />
`GALDEPTH_R` FLOAT,<br />
`GALDEPTH_Z` FLOAT,<br />
`PSFDEPTH_W1` FLOAT,<br />
`PSFDEPTH_W2` FLOAT,<br />
`WISE_COADD_ID` STRING,<br />
`FRACDEV` FLOAT,<br />
`FRACDEV_IVAR` FLOAT,<br />
`SHAPEDEV_R` FLOAT,<br />
`SHAPEDEV_R_IVAR` FLOAT,<br />
`SHAPEDEV_E1` FLOAT,<br />
`SHAPEDEV_E1_IVAR` FLOAT,<br />
`SHAPEDEV_E2` FLOAT,<br />
`SHAPEDEV_E2_IVAR` FLOAT,<br />
`SHAPEEXP_R` FLOAT,<br />
`SHAPEEXP_R_IVAR` FLOAT,<br />
`SHAPEEXP_E1` FLOAT,<br />
`SHAPEEXP_E1_IVAR` FLOAT,<br />
`SHAPEEXP_E2` FLOAT,<br />
`SHAPEEXP_E2_IVAR` FLOAT,<br />
`FIBERFLUX_G` FLOAT,<br />
`FIBERFLUX_R` FLOAT,<br />
`FIBERFLUX_Z` FLOAT,<br />
`FIBERTOTFLUX_G` FLOAT,<br />
`FIBERTOTFLUX_R` FLOAT,<br />
`FIBERTOTFLUX_Z` FLOAT,<br />
`REF_CAT` STRING,<br />
`REF_ID` BIGINT,<br />
`REF_EPOCH` FLOAT,<br />
`GAIA_PHOT_G_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_G_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_ASTROMETRIC_EXCESS_NOISE` FLOAT,<br />
`GAIA_DUPLICATED_SOURCE` BOOLEAN,<br />
`GAIA_PHOT_BP_RP_EXCESS_FACTOR` FLOAT,<br />
`GAIA_ASTROMETRIC_SIGMA5D_MAX` FLOAT,<br />
`GAIA_ASTROMETRIC_PARAMS_SOLVED` TINYINT,<br />
`PARALLAX` FLOAT,<br />
`PARALLAX_IVAR` FLOAT,<br />
`PMRA` FLOAT,<br />
`PMRA_IVAR` FLOAT,<br />
`PMDEC` FLOAT,<br />
`PMDEC_IVAR` FLOAT,<br />
`MASKBITS` SMALLINT,<br />
`z_phot_mean` FLOAT,<br />
`z_phot_median` FLOAT,<br />
`z_phot_std` FLOAT,<br />
`z_phot_l68` FLOAT,<br />
`z_phot_u68` FLOAT,<br />
`z_phot_l95` FLOAT,<br />
`z_phot_u95` FLOAT,<br />
`z_spec` FLOAT,<br />
`survey` STRING,<br />
`training` BOOLEAN<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey-sweep'<br />
;<br />
<br />
=== Combined table ===<br />
<br />
CREATE TABLE cosmohub.legacysurvey_dr8_phz (<br />
release smallint COMMENT 'Unique integer denoting the camera and filter set used',<br />
brickid int COMMENT 'Brick ID [1,662174]',<br />
brickname string COMMENT 'Name of brick, encoding the brick sky position, eg "1126p222" near RA=112.6, Dec=+22.2',<br />
objid int COMMENT 'Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick',<br />
brick_primary boolean COMMENT 'True if the object is within the brick boundary',<br />
brightblob smallint COMMENT 'bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page',<br />
maskbits smallint COMMENT 'bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits* maps, as cataloged on the DR8 bitmasks page',<br />
type string COMMENT 'Morphological model: "PSF"=stellar, "REX"="round exponential galaxy", "DEV"=deVauc, "EXP"=exponential, "COMP"=composite, "DUP"=Gaia source fit by different model. Note that in some FITS readers, a trailing space may be appended for "PSF ", "DEV " and "EXP " since the column data type is a 4-character string',<br />
ra double COMMENT 'Right ascension at equinox J2000',<br />
dec double COMMENT 'Declination at equinox J2000',<br />
ra_ivar float COMMENT 'Inverse variance of RA (no cosine term!), excluding astrometric calibration errors',<br />
dec_ivar float COMMENT 'Inverse variance of DEC, excluding astrometric calibration errors',<br />
bx float COMMENT 'X position (0-indexed) of coordinates in brick image stack',<br />
`by` float COMMENT 'Y position (0-indexed) of coordinates in brick image stack',<br />
dchisq array<float> COMMENT 'Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.',<br />
ebv float COMMENT 'Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns',<br />
mjd_min double COMMENT 'Minimum Modified Julian Date of observations used to construct the model of this object',<br />
mjd_max double COMMENT 'Maximum Modified Julian Date of observations used to construct the model of this object',<br />
ref_cat string COMMENT 'Reference catalog source for this star: "T2" for Tycho-2, "G2" for Gaia DR2, "L2" for the LSLGA, empty otherwise',<br />
ref_id bigint COMMENT 'Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b "sourceid" for Gaia-DR2 and LSLGA',<br />
pmra float COMMENT 'Reference catalog proper motion in the RA direction',<br />
pmdec float COMMENT 'Reference catalog proper motion in the Dec direction',<br />
parallax float COMMENT 'Reference catalog parallax',<br />
pmra_ivar float COMMENT 'Reference catalog inverse-variance on pmra',<br />
pmdec_ivar float COMMENT 'Reference catalog inverse-variance on pmdec',<br />
parallax_ivar float COMMENT 'Reference catalog inverse-variance on parallax',<br />
ref_epoch float COMMENT 'Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)',<br />
gaia_pointsource boolean COMMENT 'This Gaia DR2 source is believed to be a star, not a galaxy',<br />
gaia_phot_g_mean_mag float COMMENT 'Gaia G band mag',<br />
gaia_phot_g_mean_flux_over_error float COMMENT 'Gaia G band signal-to-noise',<br />
gaia_phot_g_n_obs smallint COMMENT 'Gaia G band number of observations',<br />
gaia_phot_bp_mean_mag float COMMENT 'Gaia BP mag',<br />
gaia_phot_bp_mean_flux_over_error float COMMENT 'Gaia BP signal-to-noise',<br />
gaia_phot_bp_n_obs smallint COMMENT 'Gaia BP number of observations',<br />
gaia_phot_rp_mean_mag float COMMENT 'Gaia RP mag',<br />
gaia_phot_rp_mean_flux_over_error float COMMENT 'Gaia RP signal-to-noise',<br />
gaia_phot_rp_n_obs smallint COMMENT 'Gaia RP number of observations',<br />
gaia_phot_variable_flag boolean COMMENT 'Gaia photometric variable flag',<br />
gaia_astrometric_excess_noise float COMMENT 'Gaia astrometric excess noise',<br />
gaia_astrometric_excess_noise_sig float COMMENT 'Gaia astrometric excess noise uncertainty',<br />
gaia_astrometric_n_obs_al smallint COMMENT 'Gaia number of astrometric observations along scan direction',<br />
gaia_astrometric_n_good_obs_al smallint COMMENT 'Gaia number of good astrometric observations along scan direction',<br />
gaia_astrometric_weight_al float COMMENT 'Gaia astrometric weight along scan direction',<br />
gaia_duplicated_source boolean COMMENT 'Gaia duplicated source flag',<br />
gaia_a_g_val float COMMENT 'Gaia line-of-sight extinction in the G band',<br />
gaia_e_bp_min_rp_val float COMMENT 'Gaia line-of-sight reddening E(BP-RP)',<br />
gaia_phot_bp_rp_excess_factor float COMMENT 'Gaia BP/RP excess factor',<br />
gaia_astrometric_sigma5d_max float COMMENT 'Gaia longest semi-major axis of the 5-d error ellipsoid',<br />
gaia_astrometric_params_solved tinyint COMMENT 'which astrometric parameters were estimated for a Gaia source',<br />
flux_g float COMMENT 'model flux in gg',<br />
flux_r float COMMENT 'model flux in rr',<br />
flux_z float COMMENT 'model flux in zz',<br />
flux_w1 float COMMENT 'WISE model flux in W1W1 (AB system)',<br />
flux_w2 float COMMENT 'WISE model flux in W2W2 (AB)',<br />
flux_w3 float COMMENT 'WISE model flux in W3W3 (AB)',<br />
flux_w4 float COMMENT 'WISE model flux in W4W4 (AB)',<br />
flux_ivar_g float COMMENT 'Inverse variance of flux_g',<br />
flux_ivar_r float COMMENT 'Inverse variance of flux_r',<br />
flux_ivar_z float COMMENT 'Inverse variance of flux_z',<br />
flux_ivar_w1 float COMMENT 'Inverse variance of flux_w1 (AB system)',<br />
flux_ivar_w2 float COMMENT 'Inverse variance of flux_w2 (AB)',<br />
flux_ivar_w3 float COMMENT 'Inverse variance of flux_w3 (AB)',<br />
flux_ivar_w4 float COMMENT 'Inverse variance of flux_w4 (AB)',<br />
fiberflux_g float COMMENT 'Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_r float COMMENT 'Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_z float COMMENT 'Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fibertotflux_g float COMMENT 'Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_r float COMMENT 'Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_z float COMMENT 'Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
apflux_g array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg',<br />
apflux_r array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr',<br />
apflux_z array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz',<br />
apflux_resid_g array<float> COMMENT 'aperture fluxes on the co-added residual images in gg',<br />
apflux_resid_r array<float> COMMENT 'aperture fluxes on the co-added residual images in rr',<br />
apflux_resid_z array<float> COMMENT 'aperture fluxes on the co-added residual images in zz',<br />
apflux_ivar_g array<float> COMMENT 'Inverse variance of apflux_resid_g',<br />
apflux_ivar_r array<float> COMMENT 'Inverse variance of apflux_resid_r',<br />
apflux_ivar_z array<float> COMMENT 'Inverse variance of apflux_resid_z',<br />
mw_transmission_g float COMMENT 'Galactic transmission in gg filter in linear units [0, 1]',<br />
mw_transmission_r float COMMENT 'Galactic transmission in rr filter in linear units [0, 1]',<br />
mw_transmission_z float COMMENT 'Galactic transmission in zz filter in linear units [0, 1]',<br />
mw_transmission_w1 float COMMENT 'Galactic transmission in W1W1 filter in linear units [0, 1]',<br />
mw_transmission_w2 float COMMENT 'Galactic transmission in W2W2 filter in linear units [0, 1]',<br />
mw_transmission_w3 float COMMENT 'Galactic transmission in W3W3 filter in linear units [0, 1]',<br />
mw_transmission_w4 float COMMENT 'Galactic transmission in W4W4 filter in linear units [0, 1]',<br />
nobs_g smallint COMMENT 'Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)',<br />
nobs_r smallint COMMENT 'Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)',<br />
nobs_z smallint COMMENT 'Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)',<br />
nobs_w1 smallint COMMENT 'Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)',<br />
nobs_w2 smallint COMMENT 'Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)',<br />
nobs_w3 smallint COMMENT 'Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)',<br />
nobs_w4 smallint COMMENT 'Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)',<br />
rchisq_g float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in gg',<br />
rchisq_r float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in rr',<br />
rchisq_z float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in zz',<br />
rchisq_w1 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W1W1',<br />
rchisq_w2 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W2W2',<br />
rchisq_w3 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W3W3',<br />
rchisq_w4 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W4W4',<br />
fracflux_g float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in gg (typically [0,1])',<br />
fracflux_r float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in rr (typically [0,1])',<br />
fracflux_z float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in zz (typically [0,1])',<br />
fracflux_w1 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1 (typically [0,1])',<br />
fracflux_w2 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2 (typically [0,1])',<br />
fracflux_w3 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3 (typically [0,1])',<br />
fracflux_w4 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4 (typically [0,1])',<br />
fracmasked_g float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]',<br />
fracmasked_r float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]',<br />
fracmasked_z float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]',<br />
fracin_g float COMMENT 'Fraction of a source\'s flux within the blob in gg, near unity for real sources',<br />
fracin_r float COMMENT 'Fraction of a source\'s flux within the blob in rr, near unity for real sources',<br />
fracin_z float COMMENT 'Fraction of a source\'s flux within the blob in zz, near unity for real sources',<br />
anymask_g smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page',<br />
anymask_r smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page',<br />
anymask_z smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page',<br />
allmask_g smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page',<br />
allmask_r smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page',<br />
allmask_z smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page',<br />
wisemask_w1 tinyint COMMENT 'W1 bitmask as cataloged on the DR8 bitmasks page',<br />
wisemask_w2 tinyint COMMENT 'W2 bitmask as cataloged on the DR8 bitmasks page',<br />
psfsize_g float COMMENT 'Weighted average PSF FWHM in the gg band',<br />
psfsize_r float COMMENT 'Weighted average PSF FWHM in the rr band',<br />
psfsize_z float COMMENT 'Weighted average PSF FWHM in the zz band',<br />
psfdepth_g float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude',<br />
psfdepth_r float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude',<br />
psfdepth_z float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude',<br />
galdepth_g float COMMENT 'As for psfdepth_g but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_r float COMMENT 'As for psfdepth_r but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_z float COMMENT 'As for psfdepth_z but for a galaxy (0.45" exp, round) detection sensitivity',<br />
psfdepth_w1 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W1',<br />
psfdepth_w2 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W2',<br />
psfdepth_w3 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W3',<br />
psfdepth_w4 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W4',<br />
wise_coadd_id string COMMENT 'unWISE coadd file name for the center of each object',<br />
lc_flux_w1 array<float> COMMENT 'flux_w1 in each of up to eleven unWISE coadd epochs (AB system)',<br />
lc_flux_w2 array<float> COMMENT 'flux_w2 in each of up to eleven unWISE coadd epochs (AB)',<br />
lc_flux_ivar_w1 array<float> COMMENT 'Inverse variance of lc_flux_w1 (AB system)',<br />
lc_flux_ivar_w2 array<float> COMMENT 'Inverse variance of lc_flux_w2 (AB)',<br />
lc_nobs_w1 array<smallint> COMMENT 'nobs_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_nobs_w2 array<smallint> COMMENT 'nobs_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w1 array<float> COMMENT 'fracflux_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w2 array<float> COMMENT 'fracflux_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w1 array<float> COMMENT 'rchisq_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w2 array<float> COMMENT 'rchisq_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w1 array<double> COMMENT 'mjd_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w2 array<double> COMMENT 'mjd_w2 in each of up to eleven unWISE coadd epochs',<br />
fracdev float COMMENT 'Fraction of model in deVauc [0,1]',<br />
fracdev_ivar float COMMENT 'Inverse variance of fracdev',<br />
shapeexp_r float COMMENT 'Half-light radius of exponential model (>0)',<br />
shapeexp_r_ivar float COMMENT 'Inverse variance of shapeexp_r',<br />
shapeexp_e1 float COMMENT 'Ellipticity component 1',<br />
shapeexp_e1_ivar float COMMENT 'Inverse variance of shapeexp_e1',<br />
shapeexp_e2 float COMMENT 'Ellipticity component 2',<br />
shapeexp_e2_ivar float COMMENT 'Inverse variance of shapeexp_e2',<br />
shapedev_r float COMMENT 'Half-light radius of deVaucouleurs model (>0)',<br />
shapedev_r_ivar float COMMENT 'Inverse variance of shapedev_r',<br />
shapedev_e1 float COMMENT 'Ellipticity component 1',<br />
shapedev_e1_ivar float COMMENT 'Inverse variance of shapedev_e1',<br />
shapedev_e2 float COMMENT 'Ellipticity component 2',<br />
shapedev_e2_ivar float COMMENT 'Inverse variance of shapedev_e2',<br />
z_phot_mean FLOAT COMMENT 'photo-z derived from the mean of the photo-z PDF ',<br />
z_phot_median FLOAT COMMENT 'photo-z derived from the median of the photo-z PDF',<br />
z_phot_std FLOAT COMMENT 'standard deviation of the photo-z\'s derived from the photo-z PDF',<br />
z_phot_l68 FLOAT COMMENT 'lower bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_u68 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_l95 FLOAT COMMENT 'lower bound of the 95% confidence region, derived from the photo-z PDF',<br />
z_phot_u95 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_spec FLOAT COMMENT 'spectroscopic redshift, if available',<br />
survey STRING COMMENT 'source of the spectroscopic redshift',<br />
training BOOLEAN COMMENT 'whether or not the spectroscopic redshift is used in photometric redshift training'<br />
)<br />
CLUSTERED BY (<br />
release,<br />
brickid,<br />
objid)<br />
SORTED BY (<br />
release ASC,<br />
brickid ASC,<br />
objid ASC)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
=== Join insert ===<br />
<br />
INSERT OVERWRITE TABLE cosmohub.legacy_survey_dr8_phz<br />
SELECT<br />
dr8.*,<br />
phz.z_phot_mean,<br />
phz.z_phot_median,<br />
phz.z_phot_std,<br />
phz.z_phot_l68,<br />
phz.z_phot_u68,<br />
phz.z_phot_l95,<br />
phz.z_phot_u95,<br />
phz.z_spec,<br />
phz.survey,<br />
phz.training<br />
FROM cosmohub.legacy_survey_dr8 AS dr8<br />
LEFT JOIN tallada.dr8_sweep AS phz<br />
ON dr8.release = phz.release<br />
AND dr8.brickid = phz.brickid<br />
AND dr8.objid = phz.objid<br />
;<br />
<br />
== Queries for checking ==<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT release, brickid, objid, COUNT(release)<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
HAVING COUNT(release) > 1<br />
<br />
DR8 release brickid objid: DISTINCT(1.694.230.368) TOTAL(1.694.230.368)<br />
DR8 brickid objid: DISTINCT(1.654.429.115) TOTAL(1.694.230.368)<br />
<br />
PHZ release brickid objid: DISTINCT(1.646.664.729) TOTAL(1.646.664.897)<br />
PHZ brickid objid: DISTINCT(1.608.841.582) TOTAL(1.646.664.897)<br />
<br />
DR8: 1.694.230.368<br />
PHZ: 1.646.664.897<br />
----------------------------------------------------------------------<br />
-47.565.471</div>
Tallada
https://pwiki.pic.es/index.php?title=Euclid_mock_production_v1.9.2&diff=822
Euclid mock production v1.9.2
2020-07-14T08:03:45Z
<p>Tallada: </p>
<hr />
<div> CREATE EXTERNAL TABLE `flagship_1_9_2`(<br />
`kind` bigint, <br />
`luminosity_r01` double, <br />
`halo_x` double, <br />
`halo_y` double, <br />
`halo_z` double, <br />
`halo_vx` double, <br />
`halo_vy` double, <br />
`halo_vz` double, <br />
`halo_r` double, <br />
`true_redshift_halo` double, <br />
`halo_lm` double, <br />
`halo_n_sats` bigint, <br />
`n_gals` bigint, <br />
`abs_mag_r01` double, <br />
`abs_mag_r01_evolved` double, <br />
`luminosity_r01_evolved` double, <br />
`gr_restframe` double, <br />
`color_kind` int, <br />
`x_gal` double, <br />
`y_gal` double, <br />
`z_gal` double, <br />
`r_gal` double, <br />
`true_redshift_gal` double, <br />
`ra_gal` double, <br />
`dec_gal` double, <br />
`hpix_nest` bigint, <br />
`vx_gal` double, <br />
`vy_gal` double, <br />
`vz_gal` double, <br />
`vrad_gal` double, <br />
`delta_r` double, <br />
`observed_redshift_gal` double, <br />
`sed_ke` double, <br />
`gr_cosmos` double, <br />
`sed_cosmos` double, <br />
`ext_curve_cosmos` bigint, <br />
`ebv_cosmos` double, <br />
`is_within_cosmos` boolean, <br />
`cosmos_distance` double, <br />
`abs_mag_uv_dereddened` double, <br />
`log_ml_r01` double, <br />
`log_stellar_mass` double, <br />
`metallicity` double, <br />
`log_sfr` double, <br />
`logf_halpha_ext` double, <br />
`logf_halpha` double, <br />
`halpha_scatter` double, <br />
`loglum_halpha` double, <br />
`loglum_halpha_ext` double, <br />
`logf_dummy` double, <br />
`z_dummy` double, <br />
`logf_halpha_model1_ext` double, <br />
`logf_halpha_model1` double, <br />
`loglum_halpha_model1_ext` double, <br />
`loglum_halpha_model1` double, <br />
`logf_halpha_model3_ext` double, <br />
`logf_halpha_model3` double, <br />
`loglum_halpha_model3_ext` double, <br />
`loglum_halpha_model3` double, <br />
`logf_hbeta_model1_ext` double, <br />
`logf_hbeta_model1` double, <br />
`logf_o2_model1_ext` double, <br />
`logf_o2_model1` double, <br />
`logf_n2_model1_ext` double, <br />
`logf_n2_model1` double, <br />
`logf_o3_model1_ext` double, <br />
`logf_o3_model1` double, <br />
`logf_s2_model1_ext` double, <br />
`logf_s2_model1` double, <br />
`logf_hbeta_model3_ext` double, <br />
`logf_hbeta_model3` double, <br />
`logf_o2_model3_ext` double, <br />
`logf_o2_model3` double, <br />
`logf_n2_model3_ext` double, <br />
`logf_n2_model3` double, <br />
`logf_o3_model3_ext` double, <br />
`logf_o3_model3` double, <br />
`logf_s2_model3_ext` double, <br />
`logf_s2_model3` double, <br />
`blanco_decam_g` double, <br />
`blanco_decam_i` double, <br />
`blanco_decam_r` double, <br />
`blanco_decam_z` double, <br />
`gaia_bp` double, <br />
`gaia_g` double, <br />
`gaia_rp` double, <br />
`subaru_hsc_z` double, <br />
`jst_jpcam_g` double, <br />
`kids_g` double, <br />
`kids_i` double, <br />
`kids_r` double, <br />
`kids_u` double, <br />
`lsst_g` double, <br />
`lsst_i` double, <br />
`lsst_r` double, <br />
`lsst_u` double, <br />
`lsst_y` double, <br />
`lsst_z` double, <br />
`cfht_megacam_r` double, <br />
`cfht_megacam_u` double, <br />
`pan_starrs_i` double, <br />
`pan_starrs_z` double, <br />
`euclid_nisp_h` double, <br />
`euclid_nisp_j` double, <br />
`euclid_nisp_y` double, <br />
`euclid_vis` double, <br />
`2mass_h` double, <br />
`2mass_j` double, <br />
`2mass_ks` double, <br />
`sdss_r01` double, <br />
`l_gal` double, <br />
`b_gal` double, <br />
`mw_extinction` double, <br />
`2mass_h_odonnell_ext` double, <br />
`2mass_j_odonnell_ext` double, <br />
`2mass_ks_odonnell_ext` double, <br />
`blanco_decam_g_odonnell_ext` double, <br />
`blanco_decam_i_odonnell_ext` double, <br />
`blanco_decam_r_odonnell_ext` double, <br />
`blanco_decam_z_odonnell_ext` double, <br />
`cfht_megacam_r_odonnell_ext` double, <br />
`cfht_megacam_u_odonnell_ext` double, <br />
`euclid_nisp_h_odonnell_ext` double, <br />
`euclid_nisp_j_odonnell_ext` double, <br />
`euclid_nisp_y_odonnell_ext` double, <br />
`euclid_vis_odonnell_ext` double, <br />
`gaia_bp_odonnell_ext` double, <br />
`gaia_g_odonnell_ext` double, <br />
`gaia_rp_odonnell_ext` double, <br />
`jst_jpcam_g_odonnell_ext` double, <br />
`kids_g_odonnell_ext` double, <br />
`kids_i_odonnell_ext` double, <br />
`kids_r_odonnell_ext` double, <br />
`kids_u_odonnell_ext` double, <br />
`lsst_g_odonnell_ext` double, <br />
`lsst_i_odonnell_ext` double, <br />
`lsst_r_odonnell_ext` double, <br />
`lsst_u_odonnell_ext` double, <br />
`lsst_y_odonnell_ext` double, <br />
`lsst_z_odonnell_ext` double, <br />
`pan_starrs_i_odonnell_ext` double, <br />
`pan_starrs_z_odonnell_ext` double, <br />
`sdss_r01_odonnell_ext` double, <br />
`subaru_hsc_z_odonnell_ext` double, <br />
`blanco_decam_g_el_model1_odonnell_ext` double, <br />
`blanco_decam_g_el_model3_odonnell_ext` double, <br />
`blanco_decam_i_el_model1_odonnell_ext` double, <br />
`blanco_decam_i_el_model3_odonnell_ext` double, <br />
`blanco_decam_r_el_model1_odonnell_ext` double, <br />
`blanco_decam_r_el_model3_odonnell_ext` double, <br />
`blanco_decam_z_el_model1_odonnell_ext` double, <br />
`blanco_decam_z_el_model3_odonnell_ext` double, <br />
`gaia_bp_el_model1_odonnell_ext` double, <br />
`gaia_bp_el_model3_odonnell_ext` double, <br />
`gaia_g_el_model1_odonnell_ext` double, <br />
`gaia_g_el_model3_odonnell_ext` double, <br />
`gaia_rp_el_model1_odonnell_ext` double, <br />
`gaia_rp_el_model3_odonnell_ext` double, <br />
`subaru_hsc_z_el_model1_odonnell_ext` double, <br />
`subaru_hsc_z_el_model3_odonnell_ext` double, <br />
`jst_jpcam_g_el_model1_odonnell_ext` double, <br />
`jst_jpcam_g_el_model3_odonnell_ext` double, <br />
`kids_g_el_model1_odonnell_ext` double, <br />
`kids_g_el_model3_odonnell_ext` double, <br />
`kids_i_el_model1_odonnell_ext` double, <br />
`kids_i_el_model3_odonnell_ext` double, <br />
`kids_r_el_model1_odonnell_ext` double, <br />
`kids_r_el_model3_odonnell_ext` double, <br />
`kids_u_el_model1_odonnell_ext` double, <br />
`kids_u_el_model3_odonnell_ext` double, <br />
`lsst_g_el_model1_odonnell_ext` double, <br />
`lsst_g_el_model3_odonnell_ext` double, <br />
`lsst_i_el_model1_odonnell_ext` double, <br />
`lsst_i_el_model3_odonnell_ext` double, <br />
`lsst_r_el_model1_odonnell_ext` double, <br />
`lsst_r_el_model3_odonnell_ext` double, <br />
`lsst_u_el_model1_odonnell_ext` double, <br />
`lsst_u_el_model3_odonnell_ext` double, <br />
`lsst_y_el_model1_odonnell_ext` double, <br />
`lsst_y_el_model3_odonnell_ext` double, <br />
`lsst_z_el_model1_odonnell_ext` double, <br />
`lsst_z_el_model3_odonnell_ext` double, <br />
`cfht_megacam_r_el_model1_odonnell_ext` double, <br />
`cfht_megacam_r_el_model3_odonnell_ext` double, <br />
`cfht_megacam_u_el_model1_odonnell_ext` double, <br />
`cfht_megacam_u_el_model3_odonnell_ext` double, <br />
`pan_starrs_i_el_model1_odonnell_ext` double, <br />
`pan_starrs_i_el_model3_odonnell_ext` double, <br />
`pan_starrs_z_el_model1_odonnell_ext` double, <br />
`pan_starrs_z_el_model3_odonnell_ext` double, <br />
`euclid_nisp_h_el_model1_odonnell_ext` double, <br />
`euclid_nisp_h_el_model3_odonnell_ext` double, <br />
`euclid_nisp_j_el_model1_odonnell_ext` double, <br />
`euclid_nisp_j_el_model3_odonnell_ext` double, <br />
`euclid_nisp_y_el_model1_odonnell_ext` double, <br />
`euclid_nisp_y_el_model3_odonnell_ext` double, <br />
`euclid_vis_el_model1_odonnell_ext` double, <br />
`euclid_vis_el_model3_odonnell_ext` double, <br />
`2mass_h_el_model1_odonnell_ext` double, <br />
`2mass_h_el_model3_odonnell_ext` double, <br />
`2mass_j_el_model1_odonnell_ext` double, <br />
`2mass_j_el_model3_odonnell_ext` double, <br />
`2mass_ks_el_model1_odonnell_ext` double, <br />
`2mass_ks_el_model3_odonnell_ext` double, <br />
`sdss_r01_el_model1_odonnell_ext` double, <br />
`sdss_r01_el_model3_odonnell_ext` double, <br />
`dominant_shape` bigint, <br />
`bulge_angle` double, <br />
`disk_angle` double, <br />
`median_major_axis` double, <br />
`scale_length` double, <br />
`bulge_fraction` double, <br />
`disk_scalelength` double, <br />
`disk_nsersic` double, <br />
`disk_r50` double, <br />
`bulge_r50` double, <br />
`bulge_nsersic` double, <br />
`disk_ellipticity` double, <br />
`bulge_ellipticity` double, <br />
`disk_axis_ratio` double, <br />
`bulge_axis_ratio` double, <br />
`disk_inclination_angle` double, <br />
`bulge_inclination_angle` double, <br />
`step` bigint, <br />
`random_index` double, <br />
`halo_id` bigint, <br />
`galaxy_id` bigint)<br />
ROW FORMAT SERDE <br />
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' <br />
STORED AS INPUTFORMAT <br />
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' <br />
OUTPUTFORMAT <br />
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'<br />
;<br />
<br />
CREATE TABLE flagship_mock_1_9_2_c(<br />
`kind` bigint, <br />
`luminosity_r01` double, <br />
`halo_x` double, <br />
`halo_y` double, <br />
`halo_z` double, <br />
`halo_vx` double, <br />
`halo_vy` double, <br />
`halo_vz` double, <br />
`halo_r` double, <br />
`true_redshift_halo` double, <br />
`halo_lm` double, <br />
`halo_n_sats` bigint, <br />
`n_gals` bigint, <br />
`abs_mag_r01` double, <br />
`abs_mag_r01_evolved` double, <br />
`luminosity_r01_evolved` double, <br />
`gr_restframe` double, <br />
`color_kind` int, <br />
`x_gal` double, <br />
`y_gal` double, <br />
`z_gal` double, <br />
`r_gal` double, <br />
`true_redshift_gal` double, <br />
`ra_gal` double, <br />
`dec_gal` double, <br />
`hpix_nest` bigint, <br />
`vx_gal` double, <br />
`vy_gal` double, <br />
`vz_gal` double, <br />
`vrad_gal` double, <br />
`delta_r` double, <br />
`observed_redshift_gal` double, <br />
`sed_ke` double, <br />
`gr_cosmos` double, <br />
`sed_cosmos` double, <br />
`ext_curve_cosmos` bigint, <br />
`ebv_cosmos` double, <br />
`is_within_cosmos` boolean, <br />
`cosmos_distance` double, <br />
`abs_mag_uv_dereddened` double, <br />
`log_ml_r01` double, <br />
`log_stellar_mass` double, <br />
`metallicity` double, <br />
`log_sfr` double, <br />
`logf_halpha_ext` double, <br />
`logf_halpha` double, <br />
`halpha_scatter` double, <br />
`loglum_halpha` double, <br />
`loglum_halpha_ext` double, <br />
`logf_dummy` double, <br />
`z_dummy` double, <br />
`logf_halpha_model1_ext` double, <br />
`logf_halpha_model1` double, <br />
`loglum_halpha_model1_ext` double, <br />
`loglum_halpha_model1` double, <br />
`logf_halpha_model3_ext` double, <br />
`logf_halpha_model3` double, <br />
`loglum_halpha_model3_ext` double, <br />
`loglum_halpha_model3` double, <br />
`logf_hbeta_model1_ext` double, <br />
`logf_hbeta_model1` double, <br />
`logf_o2_model1_ext` double, <br />
`logf_o2_model1` double, <br />
`logf_n2_model1_ext` double, <br />
`logf_n2_model1` double, <br />
`logf_o3_model1_ext` double, <br />
`logf_o3_model1` double, <br />
`logf_s2_model1_ext` double, <br />
`logf_s2_model1` double, <br />
`logf_hbeta_model3_ext` double, <br />
`logf_hbeta_model3` double, <br />
`logf_o2_model3_ext` double, <br />
`logf_o2_model3` double, <br />
`logf_n2_model3_ext` double, <br />
`logf_n2_model3` double, <br />
`logf_o3_model3_ext` double, <br />
`logf_o3_model3` double, <br />
`logf_s2_model3_ext` double, <br />
`logf_s2_model3` double, <br />
`blanco_decam_g` double, <br />
`blanco_decam_i` double, <br />
`blanco_decam_r` double, <br />
`blanco_decam_z` double, <br />
`gaia_bp` double, <br />
`gaia_g` double, <br />
`gaia_rp` double, <br />
`subaru_hsc_z` double, <br />
`jst_jpcam_g` double, <br />
`kids_g` double, <br />
`kids_i` double, <br />
`kids_r` double, <br />
`kids_u` double, <br />
`lsst_g` double, <br />
`lsst_i` double, <br />
`lsst_r` double, <br />
`lsst_u` double, <br />
`lsst_y` double, <br />
`lsst_z` double, <br />
`cfht_megacam_r` double, <br />
`cfht_megacam_u` double, <br />
`pan_starrs_i` double, <br />
`pan_starrs_z` double, <br />
`euclid_nisp_h` double, <br />
`euclid_nisp_j` double, <br />
`euclid_nisp_y` double, <br />
`euclid_vis` double, <br />
`2mass_h` double, <br />
`2mass_j` double, <br />
`2mass_ks` double, <br />
`sdss_r01` double, <br />
`l_gal` double, <br />
`b_gal` double, <br />
`mw_extinction` double, <br />
`2mass_h_odonnell_ext` double, <br />
`2mass_j_odonnell_ext` double, <br />
`2mass_ks_odonnell_ext` double, <br />
`blanco_decam_g_odonnell_ext` double, <br />
`blanco_decam_i_odonnell_ext` double, <br />
`blanco_decam_r_odonnell_ext` double, <br />
`blanco_decam_z_odonnell_ext` double, <br />
`cfht_megacam_r_odonnell_ext` double, <br />
`cfht_megacam_u_odonnell_ext` double, <br />
`euclid_nisp_h_odonnell_ext` double, <br />
`euclid_nisp_j_odonnell_ext` double, <br />
`euclid_nisp_y_odonnell_ext` double, <br />
`euclid_vis_odonnell_ext` double, <br />
`gaia_bp_odonnell_ext` double, <br />
`gaia_g_odonnell_ext` double, <br />
`gaia_rp_odonnell_ext` double, <br />
`jst_jpcam_g_odonnell_ext` double, <br />
`kids_g_odonnell_ext` double, <br />
`kids_i_odonnell_ext` double, <br />
`kids_r_odonnell_ext` double, <br />
`kids_u_odonnell_ext` double, <br />
`lsst_g_odonnell_ext` double, <br />
`lsst_i_odonnell_ext` double, <br />
`lsst_r_odonnell_ext` double, <br />
`lsst_u_odonnell_ext` double, <br />
`lsst_y_odonnell_ext` double, <br />
`lsst_z_odonnell_ext` double, <br />
`pan_starrs_i_odonnell_ext` double, <br />
`pan_starrs_z_odonnell_ext` double, <br />
`sdss_r01_odonnell_ext` double, <br />
`subaru_hsc_z_odonnell_ext` double, <br />
`blanco_decam_g_el_model1_odonnell_ext` double, <br />
`blanco_decam_g_el_model3_odonnell_ext` double, <br />
`blanco_decam_i_el_model1_odonnell_ext` double, <br />
`blanco_decam_i_el_model3_odonnell_ext` double, <br />
`blanco_decam_r_el_model1_odonnell_ext` double, <br />
`blanco_decam_r_el_model3_odonnell_ext` double, <br />
`blanco_decam_z_el_model1_odonnell_ext` double, <br />
`blanco_decam_z_el_model3_odonnell_ext` double, <br />
`gaia_bp_el_model1_odonnell_ext` double, <br />
`gaia_bp_el_model3_odonnell_ext` double, <br />
`gaia_g_el_model1_odonnell_ext` double, <br />
`gaia_g_el_model3_odonnell_ext` double, <br />
`gaia_rp_el_model1_odonnell_ext` double, <br />
`gaia_rp_el_model3_odonnell_ext` double, <br />
`subaru_hsc_z_el_model1_odonnell_ext` double, <br />
`subaru_hsc_z_el_model3_odonnell_ext` double, <br />
`jst_jpcam_g_el_model1_odonnell_ext` double, <br />
`jst_jpcam_g_el_model3_odonnell_ext` double, <br />
`kids_g_el_model1_odonnell_ext` double, <br />
`kids_g_el_model3_odonnell_ext` double, <br />
`kids_i_el_model1_odonnell_ext` double, <br />
`kids_i_el_model3_odonnell_ext` double, <br />
`kids_r_el_model1_odonnell_ext` double, <br />
`kids_r_el_model3_odonnell_ext` double, <br />
`kids_u_el_model1_odonnell_ext` double, <br />
`kids_u_el_model3_odonnell_ext` double, <br />
`lsst_g_el_model1_odonnell_ext` double, <br />
`lsst_g_el_model3_odonnell_ext` double, <br />
`lsst_i_el_model1_odonnell_ext` double, <br />
`lsst_i_el_model3_odonnell_ext` double, <br />
`lsst_r_el_model1_odonnell_ext` double, <br />
`lsst_r_el_model3_odonnell_ext` double, <br />
`lsst_u_el_model1_odonnell_ext` double, <br />
`lsst_u_el_model3_odonnell_ext` double, <br />
`lsst_y_el_model1_odonnell_ext` double, <br />
`lsst_y_el_model3_odonnell_ext` double, <br />
`lsst_z_el_model1_odonnell_ext` double, <br />
`lsst_z_el_model3_odonnell_ext` double, <br />
`cfht_megacam_r_el_model1_odonnell_ext` double, <br />
`cfht_megacam_r_el_model3_odonnell_ext` double, <br />
`cfht_megacam_u_el_model1_odonnell_ext` double, <br />
`cfht_megacam_u_el_model3_odonnell_ext` double, <br />
`pan_starrs_i_el_model1_odonnell_ext` double, <br />
`pan_starrs_i_el_model3_odonnell_ext` double, <br />
`pan_starrs_z_el_model1_odonnell_ext` double, <br />
`pan_starrs_z_el_model3_odonnell_ext` double, <br />
`euclid_nisp_h_el_model1_odonnell_ext` double, <br />
`euclid_nisp_h_el_model3_odonnell_ext` double, <br />
`euclid_nisp_j_el_model1_odonnell_ext` double, <br />
`euclid_nisp_j_el_model3_odonnell_ext` double, <br />
`euclid_nisp_y_el_model1_odonnell_ext` double, <br />
`euclid_nisp_y_el_model3_odonnell_ext` double, <br />
`euclid_vis_el_model1_odonnell_ext` double, <br />
`euclid_vis_el_model3_odonnell_ext` double, <br />
`2mass_h_el_model1_odonnell_ext` double, <br />
`2mass_h_el_model3_odonnell_ext` double, <br />
`2mass_j_el_model1_odonnell_ext` double, <br />
`2mass_j_el_model3_odonnell_ext` double, <br />
`2mass_ks_el_model1_odonnell_ext` double, <br />
`2mass_ks_el_model3_odonnell_ext` double, <br />
`sdss_r01_el_model1_odonnell_ext` double, <br />
`sdss_r01_el_model3_odonnell_ext` double, <br />
`dominant_shape` bigint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)', <br />
`bulge_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`disk_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`median_major_axis` double COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)', <br />
`scale_length` double COMMENT 'disc and bulge scalelength prior', <br />
`bulge_fraction` double COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)', <br />
`disk_scalelength` double COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)', <br />
`disk_nsersic` double COMMENT 'Sersic index of the disk component', <br />
`disk_r50` double COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678', <br />
`bulge_r50` double COMMENT 'half light radius of the bulge', <br />
`bulge_nsersic` double COMMENT 'Sersic index of the bulge component', <br />
`disk_ellipticity` double COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)', <br />
`bulge_ellipticity` double COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)', <br />
`disk_axis_ratio` double COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`bulge_axis_ratio` double COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',<br />
`disk_inclination_angle` double COMMENT '', <br />
`bulge_inclination_angle` double COMMENT '', <br />
`step` bigint, <br />
`random_index` double, <br />
`halo_id` bigint, <br />
`galaxy_id` bigint<br />
)<br />
CLUSTERED BY ( <br />
halo_id, <br />
galaxy_id<br />
) <br />
INTO 128 BUCKETS<br />
ROW FORMAT SERDE <br />
'org.apache.hadoop.hive.ql.io.orc.OrcSerde' <br />
STORED AS INPUTFORMAT <br />
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' <br />
OUTPUTFORMAT <br />
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'<br />
;<br />
<br />
INSERT OVERWRITE TABLE cosmohub.flagship_mock_1_9_2_c<br />
SELECT * FROM tallada.flagship_1_9_2;<br />
<br />
ANALYZE TABLE cosmohub.flagship_mock_1_9_2_c COMPUTE STATISTICS FOR COLUMNS;<br />
<br />
CREATE TABLE tallada.flagship_mock_1_9_2_c2(<br />
`kind` bigint, <br />
`luminosity_r01` double, <br />
`halo_x` double, <br />
`halo_y` double, <br />
`halo_z` double, <br />
`halo_vx` double, <br />
`halo_vy` double, <br />
`halo_vz` double, <br />
`halo_r` double, <br />
`true_redshift_halo` double, <br />
`halo_lm` double, <br />
`halo_n_sats` bigint, <br />
`n_gals` bigint, <br />
`abs_mag_r01` double, <br />
`abs_mag_r01_evolved` double, <br />
`luminosity_r01_evolved` double, <br />
`gr_restframe` double, <br />
`color_kind` int, <br />
`x_gal` double, <br />
`y_gal` double, <br />
`z_gal` double, <br />
`r_gal` double, <br />
`true_redshift_gal` double, <br />
`ra_gal` double, <br />
`dec_gal` double, <br />
`hpix_29_nest` bigint, <br />
`hpix_13_nest` bigint, <br />
`vx_gal` double, <br />
`vy_gal` double, <br />
`vz_gal` double, <br />
`vrad_gal` double, <br />
`delta_r` double, <br />
`observed_redshift_gal` double, <br />
`sed_ke` double, <br />
`gr_cosmos` double, <br />
`sed_cosmos` double, <br />
`ext_curve_cosmos` bigint, <br />
`ebv_cosmos` double, <br />
`is_within_cosmos` boolean, <br />
`cosmos_distance` double, <br />
`abs_mag_uv_dereddened` double, <br />
`log_ml_r01` double, <br />
`log_stellar_mass` double, <br />
`metallicity` double, <br />
`log_sfr` double, <br />
`logf_halpha_ext` double, <br />
`logf_halpha` double, <br />
`halpha_scatter` double, <br />
`loglum_halpha` double, <br />
`loglum_halpha_ext` double, <br />
`logf_dummy` double, <br />
`z_dummy` double, <br />
`logf_halpha_model1_ext` double, <br />
`logf_halpha_model1` double, <br />
`loglum_halpha_model1_ext` double, <br />
`loglum_halpha_model1` double, <br />
`logf_halpha_model3_ext` double, <br />
`logf_halpha_model3` double, <br />
`loglum_halpha_model3_ext` double, <br />
`loglum_halpha_model3` double, <br />
`logf_hbeta_model1_ext` double, <br />
`logf_hbeta_model1` double, <br />
`logf_o2_model1_ext` double, <br />
`logf_o2_model1` double, <br />
`logf_n2_model1_ext` double, <br />
`logf_n2_model1` double, <br />
`logf_o3_model1_ext` double, <br />
`logf_o3_model1` double, <br />
`logf_s2_model1_ext` double, <br />
`logf_s2_model1` double, <br />
`logf_hbeta_model3_ext` double, <br />
`logf_hbeta_model3` double, <br />
`logf_o2_model3_ext` double, <br />
`logf_o2_model3` double, <br />
`logf_n2_model3_ext` double, <br />
`logf_n2_model3` double, <br />
`logf_o3_model3_ext` double, <br />
`logf_o3_model3` double, <br />
`logf_s2_model3_ext` double, <br />
`logf_s2_model3` double, <br />
`blanco_decam_g` double, <br />
`blanco_decam_i` double, <br />
`blanco_decam_r` double, <br />
`blanco_decam_z` double, <br />
`gaia_bp` double, <br />
`gaia_g` double, <br />
`gaia_rp` double, <br />
`subaru_hsc_z` double, <br />
`jst_jpcam_g` double, <br />
`kids_g` double, <br />
`kids_i` double, <br />
`kids_r` double, <br />
`kids_u` double, <br />
`lsst_g` double, <br />
`lsst_i` double, <br />
`lsst_r` double, <br />
`lsst_u` double, <br />
`lsst_y` double, <br />
`lsst_z` double, <br />
`cfht_megacam_r` double, <br />
`cfht_megacam_u` double, <br />
`pan_starrs_i` double, <br />
`pan_starrs_z` double, <br />
`euclid_nisp_h` double, <br />
`euclid_nisp_j` double, <br />
`euclid_nisp_y` double, <br />
`euclid_vis` double, <br />
`2mass_h` double, <br />
`2mass_j` double, <br />
`2mass_ks` double, <br />
`sdss_r01` double, <br />
`l_gal` double, <br />
`b_gal` double, <br />
`mw_extinction` double, <br />
`2mass_h_odonnell_ext` double, <br />
`2mass_j_odonnell_ext` double, <br />
`2mass_ks_odonnell_ext` double, <br />
`blanco_decam_g_odonnell_ext` double, <br />
`blanco_decam_i_odonnell_ext` double, <br />
`blanco_decam_r_odonnell_ext` double, <br />
`blanco_decam_z_odonnell_ext` double, <br />
`cfht_megacam_r_odonnell_ext` double, <br />
`cfht_megacam_u_odonnell_ext` double, <br />
`euclid_nisp_h_odonnell_ext` double, <br />
`euclid_nisp_j_odonnell_ext` double, <br />
`euclid_nisp_y_odonnell_ext` double, <br />
`euclid_vis_odonnell_ext` double, <br />
`gaia_bp_odonnell_ext` double, <br />
`gaia_g_odonnell_ext` double, <br />
`gaia_rp_odonnell_ext` double, <br />
`jst_jpcam_g_odonnell_ext` double, <br />
`kids_g_odonnell_ext` double, <br />
`kids_i_odonnell_ext` double, <br />
`kids_r_odonnell_ext` double, <br />
`kids_u_odonnell_ext` double, <br />
`lsst_g_odonnell_ext` double, <br />
`lsst_i_odonnell_ext` double, <br />
`lsst_r_odonnell_ext` double, <br />
`lsst_u_odonnell_ext` double, <br />
`lsst_y_odonnell_ext` double, <br />
`lsst_z_odonnell_ext` double, <br />
`pan_starrs_i_odonnell_ext` double, <br />
`pan_starrs_z_odonnell_ext` double, <br />
`sdss_r01_odonnell_ext` double, <br />
`subaru_hsc_z_odonnell_ext` double, <br />
`blanco_decam_g_el_model1_odonnell_ext` double, <br />
`blanco_decam_g_el_model3_odonnell_ext` double, <br />
`blanco_decam_i_el_model1_odonnell_ext` double, <br />
`blanco_decam_i_el_model3_odonnell_ext` double, <br />
`blanco_decam_r_el_model1_odonnell_ext` double, <br />
`blanco_decam_r_el_model3_odonnell_ext` double, <br />
`blanco_decam_z_el_model1_odonnell_ext` double, <br />
`blanco_decam_z_el_model3_odonnell_ext` double, <br />
`gaia_bp_el_model1_odonnell_ext` double, <br />
`gaia_bp_el_model3_odonnell_ext` double, <br />
`gaia_g_el_model1_odonnell_ext` double, <br />
`gaia_g_el_model3_odonnell_ext` double, <br />
`gaia_rp_el_model1_odonnell_ext` double, <br />
`gaia_rp_el_model3_odonnell_ext` double, <br />
`subaru_hsc_z_el_model1_odonnell_ext` double, <br />
`subaru_hsc_z_el_model3_odonnell_ext` double, <br />
`jst_jpcam_g_el_model1_odonnell_ext` double, <br />
`jst_jpcam_g_el_model3_odonnell_ext` double, <br />
`kids_g_el_model1_odonnell_ext` double, <br />
`kids_g_el_model3_odonnell_ext` double, <br />
`kids_i_el_model1_odonnell_ext` double, <br />
`kids_i_el_model3_odonnell_ext` double, <br />
`kids_r_el_model1_odonnell_ext` double, <br />
`kids_r_el_model3_odonnell_ext` double, <br />
`kids_u_el_model1_odonnell_ext` double, <br />
`kids_u_el_model3_odonnell_ext` double, <br />
`lsst_g_el_model1_odonnell_ext` double, <br />
`lsst_g_el_model3_odonnell_ext` double, <br />
`lsst_i_el_model1_odonnell_ext` double, <br />
`lsst_i_el_model3_odonnell_ext` double, <br />
`lsst_r_el_model1_odonnell_ext` double, <br />
`lsst_r_el_model3_odonnell_ext` double, <br />
`lsst_u_el_model1_odonnell_ext` double, <br />
`lsst_u_el_model3_odonnell_ext` double, <br />
`lsst_y_el_model1_odonnell_ext` double, <br />
`lsst_y_el_model3_odonnell_ext` double, <br />
`lsst_z_el_model1_odonnell_ext` double, <br />
`lsst_z_el_model3_odonnell_ext` double, <br />
`cfht_megacam_r_el_model1_odonnell_ext` double, <br />
`cfht_megacam_r_el_model3_odonnell_ext` double, <br />
`cfht_megacam_u_el_model1_odonnell_ext` double, <br />
`cfht_megacam_u_el_model3_odonnell_ext` double, <br />
`pan_starrs_i_el_model1_odonnell_ext` double, <br />
`pan_starrs_i_el_model3_odonnell_ext` double, <br />
`pan_starrs_z_el_model1_odonnell_ext` double, <br />
`pan_starrs_z_el_model3_odonnell_ext` double, <br />
`euclid_nisp_h_el_model1_odonnell_ext` double, <br />
`euclid_nisp_h_el_model3_odonnell_ext` double, <br />
`euclid_nisp_j_el_model1_odonnell_ext` double, <br />
`euclid_nisp_j_el_model3_odonnell_ext` double, <br />
`euclid_nisp_y_el_model1_odonnell_ext` double, <br />
`euclid_nisp_y_el_model3_odonnell_ext` double, <br />
`euclid_vis_el_model1_odonnell_ext` double, <br />
`euclid_vis_el_model3_odonnell_ext` double, <br />
`2mass_h_el_model1_odonnell_ext` double, <br />
`2mass_h_el_model3_odonnell_ext` double, <br />
`2mass_j_el_model1_odonnell_ext` double, <br />
`2mass_j_el_model3_odonnell_ext` double, <br />
`2mass_ks_el_model1_odonnell_ext` double, <br />
`2mass_ks_el_model3_odonnell_ext` double, <br />
`sdss_r01_el_model1_odonnell_ext` double, <br />
`sdss_r01_el_model3_odonnell_ext` double, <br />
`dominant_shape` bigint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)', <br />
`bulge_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`disk_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`median_major_axis` double COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)', <br />
`scale_length` double COMMENT 'disc and bulge scalelength prior', <br />
`bulge_fraction` double COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)', <br />
`disk_scalelength` double COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)', <br />
`disk_nsersic` double COMMENT 'Sersic index of the disk component', <br />
`disk_r50` double COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678', <br />
`bulge_r50` double COMMENT 'half light radius of the bulge', <br />
`bulge_nsersic` double COMMENT 'Sersic index of the bulge component', <br />
`disk_ellipticity` double COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)', <br />
`bulge_ellipticity` double COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)', <br />
`disk_axis_ratio` double COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`bulge_axis_ratio` double COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`disk_inclination_angle` double COMMENT '', <br />
`bulge_inclination_angle` double COMMENT '', <br />
`random_index` double, <br />
`halo_id` bigint, <br />
`galaxy_id` bigint<br />
)<br />
PARTITIONED BY (<br />
`step` smallint<br />
)<br />
CLUSTERED BY (<br />
hpix_13_nest<br />
)<br />
SORTED BY (<br />
hpix_13_nest ASC<br />
)<br />
INTO 128 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
INSERT OVERWRITE TABLE tallada.flagship_mock_1_9_2_c2<br />
SELECT<br />
kind, luminosity_r01, halo_x, halo_y, halo_z, halo_vx, halo_vy, halo_vz, halo_r, true_redshift_halo, halo_lm, halo_n_sats, n_gals,<br />
abs_mag_r01, abs_mag_r01_evolved, luminosity_r01_evolved, gr_restframe, color_kind, x_gal, y_gal, z_gal, r_gal, true_redshift_gal,<br />
ra_gal, dec_gal, hpix_nest, SHIFTRIGHT(hpix_nest, (29-13)*2), vx_gal, vy_gal, vz_gal, vrad_gal, delta_r, observed_redshift_gal, sed_ke, <br />
gr_cosmos, sed_cosmos, ext_curve_cosmos, ebv_cosmos, is_within_cosmos, cosmos_distance, abs_mag_uv_dereddened, log_ml_r01, <br />
log_stellar_mass, metallicity, log_sfr, logf_halpha_ext, logf_halpha, halpha_scatter, loglum_halpha, loglum_halpha_ext, logf_dummy, <br />
z_dummy, logf_halpha_model1_ext, logf_halpha_model1, loglum_halpha_model1_ext, loglum_halpha_model1, logf_halpha_model3_ext, <br />
logf_halpha_model3, loglum_halpha_model3_ext, loglum_halpha_model3, logf_hbeta_model1_ext, logf_hbeta_model1, logf_o2_model1_ext, <br />
logf_o2_model1, logf_n2_model1_ext, logf_n2_model1, logf_o3_model1_ext, logf_o3_model1, logf_s2_model1_ext, logf_s2_model1, <br />
logf_hbeta_model3_ext, logf_hbeta_model3, logf_o2_model3_ext, logf_o2_model3, logf_n2_model3_ext, logf_n2_model3, logf_o3_model3_ext, <br />
logf_o3_model3, logf_s2_model3_ext, logf_s2_model3, blanco_decam_g, blanco_decam_i, blanco_decam_r, blanco_decam_z, gaia_bp, gaia_g, <br />
gaia_rp, subaru_hsc_z, jst_jpcam_g, kids_g, kids_i, kids_r, kids_u, lsst_g, lsst_i, lsst_r, lsst_u, lsst_y, lsst_z, cfht_megacam_r, <br />
cfht_megacam_u, pan_starrs_i, pan_starrs_z, euclid_nisp_h, euclid_nisp_j, euclid_nisp_y, euclid_vis, 2mass_h, 2mass_j, 2mass_ks, <br />
sdss_r01, l_gal, b_gal, mw_extinction, 2mass_h_odonnell_ext, 2mass_j_odonnell_ext, 2mass_ks_odonnell_ext, blanco_decam_g_odonnell_ext, <br />
blanco_decam_i_odonnell_ext, blanco_decam_r_odonnell_ext, blanco_decam_z_odonnell_ext, cfht_megacam_r_odonnell_ext, <br />
cfht_megacam_u_odonnell_ext, euclid_nisp_h_odonnell_ext, euclid_nisp_j_odonnell_ext, euclid_nisp_y_odonnell_ext, euclid_vis_odonnell_ext,<br />
gaia_bp_odonnell_ext, gaia_g_odonnell_ext, gaia_rp_odonnell_ext, jst_jpcam_g_odonnell_ext, kids_g_odonnell_ext, kids_i_odonnell_ext, <br />
kids_r_odonnell_ext, kids_u_odonnell_ext, lsst_g_odonnell_ext, lsst_i_odonnell_ext, lsst_r_odonnell_ext, lsst_u_odonnell_ext, <br />
lsst_y_odonnell_ext, lsst_z_odonnell_ext, pan_starrs_i_odonnell_ext, pan_starrs_z_odonnell_ext, sdss_r01_odonnell_ext, <br />
subaru_hsc_z_odonnell_ext, blanco_decam_g_el_model1_odonnell_ext, blanco_decam_g_el_model3_odonnell_ext, <br />
blanco_decam_i_el_model1_odonnell_ext, blanco_decam_i_el_model3_odonnell_ext, blanco_decam_r_el_model1_odonnell_ext, <br />
blanco_decam_r_el_model3_odonnell_ext, blanco_decam_z_el_model1_odonnell_ext, blanco_decam_z_el_model3_odonnell_ext, <br />
gaia_bp_el_model1_odonnell_ext, gaia_bp_el_model3_odonnell_ext, gaia_g_el_model1_odonnell_ext, gaia_g_el_model3_odonnell_ext, <br />
gaia_rp_el_model1_odonnell_ext, gaia_rp_el_model3_odonnell_ext, subaru_hsc_z_el_model1_odonnell_ext, subaru_hsc_z_el_model3_odonnell_ext, <br />
jst_jpcam_g_el_model1_odonnell_ext, jst_jpcam_g_el_model3_odonnell_ext, kids_g_el_model1_odonnell_ext, kids_g_el_model3_odonnell_ext, <br />
kids_i_el_model1_odonnell_ext, kids_i_el_model3_odonnell_ext, kids_r_el_model1_odonnell_ext, kids_r_el_model3_odonnell_ext, <br />
kids_u_el_model1_odonnell_ext, kids_u_el_model3_odonnell_ext, lsst_g_el_model1_odonnell_ext, lsst_g_el_model3_odonnell_ext, <br />
lsst_i_el_model1_odonnell_ext, lsst_i_el_model3_odonnell_ext, lsst_r_el_model1_odonnell_ext, lsst_r_el_model3_odonnell_ext, <br />
lsst_u_el_model1_odonnell_ext, lsst_u_el_model3_odonnell_ext, lsst_y_el_model1_odonnell_ext, lsst_y_el_model3_odonnell_ext, <br />
lsst_z_el_model1_odonnell_ext, lsst_z_el_model3_odonnell_ext, cfht_megacam_r_el_model1_odonnell_ext, <br />
cfht_megacam_r_el_model3_odonnell_ext, cfht_megacam_u_el_model1_odonnell_ext, cfht_megacam_u_el_model3_odonnell_ext, <br />
pan_starrs_i_el_model1_odonnell_ext, pan_starrs_i_el_model3_odonnell_ext, pan_starrs_z_el_model1_odonnell_ext, <br />
pan_starrs_z_el_model3_odonnell_ext, euclid_nisp_h_el_model1_odonnell_ext, euclid_nisp_h_el_model3_odonnell_ext, <br />
euclid_nisp_j_el_model1_odonnell_ext, euclid_nisp_j_el_model3_odonnell_ext, euclid_nisp_y_el_model1_odonnell_ext, <br />
euclid_nisp_y_el_model3_odonnell_ext, euclid_vis_el_model1_odonnell_ext, euclid_vis_el_model3_odonnell_ext, <br />
2mass_h_el_model1_odonnell_ext, 2mass_h_el_model3_odonnell_ext, 2mass_j_el_model1_odonnell_ext, 2mass_j_el_model3_odonnell_ext,<br />
2mass_ks_el_model1_odonnell_ext, 2mass_ks_el_model3_odonnell_ext, sdss_r01_el_model1_odonnell_ext, sdss_r01_el_model3_odonnell_ext, <br />
dominant_shape, bulge_angle, disk_angle, median_major_axis, scale_length, bulge_fraction, disk_scalelength, disk_nsersic, disk_r50, <br />
bulge_r50, bulge_nsersic, disk_ellipticity, bulge_ellipticity, disk_axis_ratio, bulge_axis_ratio, disk_inclination_angle,<br />
bulge_inclination_angle, random_index, halo_id, galaxy_id, step<br />
FROM cosmohub.flagship_mock_1_9_2_c<br />
<br />
CREATE TABLE tallada.flagship_mock_1_9_2_s(<br />
`kind` bigint, <br />
`luminosity_r01` double, <br />
`halo_x` double, <br />
`halo_y` double, <br />
`halo_z` double, <br />
`halo_vx` double, <br />
`halo_vy` double, <br />
`halo_vz` double, <br />
`halo_r` double, <br />
`true_redshift_halo` double, <br />
`halo_lm` double, <br />
`halo_n_sats` bigint, <br />
`n_gals` bigint, <br />
`abs_mag_r01` double, <br />
`abs_mag_r01_evolved` double, <br />
`luminosity_r01_evolved` double, <br />
`gr_restframe` double, <br />
`color_kind` int, <br />
`x_gal` double, <br />
`y_gal` double, <br />
`z_gal` double, <br />
`r_gal` double, <br />
`true_redshift_gal` double, <br />
`ra_gal` double, <br />
`dec_gal` double, <br />
`ra_mag_gal` double, <br />
`dec_mag_gal` double, <br />
`kappa` double,<br />
`gamma1` double,<br />
`gamma2` double,<br />
`hpix_29_nest` bigint, <br />
`hpix_13_nest` bigint, <br />
`vx_gal` double, <br />
`vy_gal` double, <br />
`vz_gal` double, <br />
`vrad_gal` double, <br />
`delta_r` double, <br />
`observed_redshift_gal` double, <br />
`sed_ke` double, <br />
`gr_cosmos` double, <br />
`sed_cosmos` double, <br />
`ext_curve_cosmos` bigint, <br />
`ebv_cosmos` double, <br />
`is_within_cosmos` boolean, <br />
`cosmos_distance` double, <br />
`abs_mag_uv_dereddened` double, <br />
`log_ml_r01` double, <br />
`log_stellar_mass` double, <br />
`metallicity` double, <br />
`log_sfr` double, <br />
`logf_halpha_ext` double, <br />
`logf_halpha` double, <br />
`halpha_scatter` double, <br />
`loglum_halpha` double, <br />
`loglum_halpha_ext` double, <br />
`logf_dummy` double, <br />
`z_dummy` double, <br />
`logf_halpha_model1_ext` double, <br />
`logf_halpha_model1` double, <br />
`loglum_halpha_model1_ext` double, <br />
`loglum_halpha_model1` double, <br />
`logf_halpha_model3_ext` double, <br />
`logf_halpha_model3` double, <br />
`loglum_halpha_model3_ext` double, <br />
`loglum_halpha_model3` double, <br />
`logf_hbeta_model1_ext` double, <br />
`logf_hbeta_model1` double, <br />
`logf_o2_model1_ext` double, <br />
`logf_o2_model1` double, <br />
`logf_n2_model1_ext` double, <br />
`logf_n2_model1` double, <br />
`logf_o3_model1_ext` double, <br />
`logf_o3_model1` double, <br />
`logf_s2_model1_ext` double, <br />
`logf_s2_model1` double, <br />
`logf_hbeta_model3_ext` double, <br />
`logf_hbeta_model3` double, <br />
`logf_o2_model3_ext` double, <br />
`logf_o2_model3` double, <br />
`logf_n2_model3_ext` double, <br />
`logf_n2_model3` double, <br />
`logf_o3_model3_ext` double, <br />
`logf_o3_model3` double, <br />
`logf_s2_model3_ext` double, <br />
`logf_s2_model3` double, <br />
`blanco_decam_g` double, <br />
`blanco_decam_i` double, <br />
`blanco_decam_r` double, <br />
`blanco_decam_z` double, <br />
`gaia_bp` double, <br />
`gaia_g` double, <br />
`gaia_rp` double, <br />
`subaru_hsc_z` double, <br />
`jst_jpcam_g` double, <br />
`kids_g` double, <br />
`kids_i` double, <br />
`kids_r` double, <br />
`kids_u` double, <br />
`lsst_g` double, <br />
`lsst_i` double, <br />
`lsst_r` double, <br />
`lsst_u` double, <br />
`lsst_y` double, <br />
`lsst_z` double, <br />
`cfht_megacam_r` double, <br />
`cfht_megacam_u` double, <br />
`pan_starrs_i` double, <br />
`pan_starrs_z` double, <br />
`euclid_nisp_h` double, <br />
`euclid_nisp_j` double, <br />
`euclid_nisp_y` double, <br />
`euclid_vis` double, <br />
`2mass_h` double, <br />
`2mass_j` double, <br />
`2mass_ks` double, <br />
`sdss_r01` double, <br />
`l_gal` double, <br />
`b_gal` double, <br />
`mw_extinction` double, <br />
`2mass_h_odonnell_ext` double, <br />
`2mass_j_odonnell_ext` double, <br />
`2mass_ks_odonnell_ext` double, <br />
`blanco_decam_g_odonnell_ext` double, <br />
`blanco_decam_i_odonnell_ext` double, <br />
`blanco_decam_r_odonnell_ext` double, <br />
`blanco_decam_z_odonnell_ext` double, <br />
`cfht_megacam_r_odonnell_ext` double, <br />
`cfht_megacam_u_odonnell_ext` double, <br />
`euclid_nisp_h_odonnell_ext` double, <br />
`euclid_nisp_j_odonnell_ext` double, <br />
`euclid_nisp_y_odonnell_ext` double, <br />
`euclid_vis_odonnell_ext` double, <br />
`gaia_bp_odonnell_ext` double, <br />
`gaia_g_odonnell_ext` double, <br />
`gaia_rp_odonnell_ext` double, <br />
`jst_jpcam_g_odonnell_ext` double, <br />
`kids_g_odonnell_ext` double, <br />
`kids_i_odonnell_ext` double, <br />
`kids_r_odonnell_ext` double, <br />
`kids_u_odonnell_ext` double, <br />
`lsst_g_odonnell_ext` double, <br />
`lsst_i_odonnell_ext` double, <br />
`lsst_r_odonnell_ext` double, <br />
`lsst_u_odonnell_ext` double, <br />
`lsst_y_odonnell_ext` double, <br />
`lsst_z_odonnell_ext` double, <br />
`pan_starrs_i_odonnell_ext` double, <br />
`pan_starrs_z_odonnell_ext` double, <br />
`sdss_r01_odonnell_ext` double, <br />
`subaru_hsc_z_odonnell_ext` double, <br />
`blanco_decam_g_el_model1_odonnell_ext` double, <br />
`blanco_decam_g_el_model3_odonnell_ext` double, <br />
`blanco_decam_i_el_model1_odonnell_ext` double, <br />
`blanco_decam_i_el_model3_odonnell_ext` double, <br />
`blanco_decam_r_el_model1_odonnell_ext` double, <br />
`blanco_decam_r_el_model3_odonnell_ext` double, <br />
`blanco_decam_z_el_model1_odonnell_ext` double, <br />
`blanco_decam_z_el_model3_odonnell_ext` double, <br />
`gaia_bp_el_model1_odonnell_ext` double, <br />
`gaia_bp_el_model3_odonnell_ext` double, <br />
`gaia_g_el_model1_odonnell_ext` double, <br />
`gaia_g_el_model3_odonnell_ext` double, <br />
`gaia_rp_el_model1_odonnell_ext` double, <br />
`gaia_rp_el_model3_odonnell_ext` double, <br />
`subaru_hsc_z_el_model1_odonnell_ext` double, <br />
`subaru_hsc_z_el_model3_odonnell_ext` double, <br />
`jst_jpcam_g_el_model1_odonnell_ext` double, <br />
`jst_jpcam_g_el_model3_odonnell_ext` double, <br />
`kids_g_el_model1_odonnell_ext` double, <br />
`kids_g_el_model3_odonnell_ext` double, <br />
`kids_i_el_model1_odonnell_ext` double, <br />
`kids_i_el_model3_odonnell_ext` double, <br />
`kids_r_el_model1_odonnell_ext` double, <br />
`kids_r_el_model3_odonnell_ext` double, <br />
`kids_u_el_model1_odonnell_ext` double, <br />
`kids_u_el_model3_odonnell_ext` double, <br />
`lsst_g_el_model1_odonnell_ext` double, <br />
`lsst_g_el_model3_odonnell_ext` double, <br />
`lsst_i_el_model1_odonnell_ext` double, <br />
`lsst_i_el_model3_odonnell_ext` double, <br />
`lsst_r_el_model1_odonnell_ext` double, <br />
`lsst_r_el_model3_odonnell_ext` double, <br />
`lsst_u_el_model1_odonnell_ext` double, <br />
`lsst_u_el_model3_odonnell_ext` double, <br />
`lsst_y_el_model1_odonnell_ext` double, <br />
`lsst_y_el_model3_odonnell_ext` double, <br />
`lsst_z_el_model1_odonnell_ext` double, <br />
`lsst_z_el_model3_odonnell_ext` double, <br />
`cfht_megacam_r_el_model1_odonnell_ext` double, <br />
`cfht_megacam_r_el_model3_odonnell_ext` double, <br />
`cfht_megacam_u_el_model1_odonnell_ext` double, <br />
`cfht_megacam_u_el_model3_odonnell_ext` double, <br />
`pan_starrs_i_el_model1_odonnell_ext` double, <br />
`pan_starrs_i_el_model3_odonnell_ext` double, <br />
`pan_starrs_z_el_model1_odonnell_ext` double, <br />
`pan_starrs_z_el_model3_odonnell_ext` double, <br />
`euclid_nisp_h_el_model1_odonnell_ext` double, <br />
`euclid_nisp_h_el_model3_odonnell_ext` double, <br />
`euclid_nisp_j_el_model1_odonnell_ext` double, <br />
`euclid_nisp_j_el_model3_odonnell_ext` double, <br />
`euclid_nisp_y_el_model1_odonnell_ext` double, <br />
`euclid_nisp_y_el_model3_odonnell_ext` double, <br />
`euclid_vis_el_model1_odonnell_ext` double, <br />
`euclid_vis_el_model3_odonnell_ext` double, <br />
`2mass_h_el_model1_odonnell_ext` double, <br />
`2mass_h_el_model3_odonnell_ext` double, <br />
`2mass_j_el_model1_odonnell_ext` double, <br />
`2mass_j_el_model3_odonnell_ext` double, <br />
`2mass_ks_el_model1_odonnell_ext` double, <br />
`2mass_ks_el_model3_odonnell_ext` double, <br />
`sdss_r01_el_model1_odonnell_ext` double, <br />
`sdss_r01_el_model3_odonnell_ext` double, <br />
`dominant_shape` bigint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)', <br />
`bulge_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`disk_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`median_major_axis` double COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)', <br />
`scale_length` double COMMENT 'disc and bulge scalelength prior', <br />
`bulge_fraction` double COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)', <br />
`disk_scalelength` double COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)', <br />
`disk_nsersic` double COMMENT 'Sersic index of the disk component', <br />
`disk_r50` double COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678', <br />
`bulge_r50` double COMMENT 'half light radius of the bulge', <br />
`bulge_nsersic` double COMMENT 'Sersic index of the bulge component', <br />
`disk_ellipticity` double COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)', <br />
`bulge_ellipticity` double COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)', <br />
`disk_axis_ratio` double COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`bulge_axis_ratio` double COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`disk_inclination_angle` double COMMENT '', <br />
`bulge_inclination_angle` double COMMENT '', <br />
`random_index` double, <br />
`halo_id` bigint, <br />
`galaxy_id` bigint<br />
)<br />
PARTITIONED BY (<br />
`step` smallint<br />
)<br />
CLUSTERED BY (<br />
hpix_13_nest<br />
)<br />
SORTED BY (<br />
hpix_13_nest ASC<br />
)<br />
INTO 128 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
cat << EOF > lensing_step.sql<br />
SET hive.auto.convert.join.noconditionaltask.size=1;<br />
INSERT OVERWRITE TABLE tallada.flagship_mock_1_9_2_s PARTITION (step=${STEP})<br />
SELECT<br />
m.kind, m.luminosity_r01, m.halo_x, m.halo_y, m.halo_z, m.halo_vx, m.halo_vy, m.halo_vz, m.halo_r, m.true_redshift_halo, m.halo_lm,<br />
m.halo_n_sats, m.n_gals, m.abs_mag_r01, m.abs_mag_r01_evolved, m.luminosity_r01_evolved, m.gr_restframe, m.color_kind, m.x_gal,<br />
m.y_gal, m.z_gal, m.r_gal, m.true_redshift_gal, m.ra_gal, m.dec_gal, magpos.ra_mag, magpos.dec_mag, m.kappa, m.gamma1, m.gamma2,<br />
m.hpix_29_nest, m.hpix_13_nest, m.vx_gal, m.vy_gal, m.vz_gal, m.vrad_gal, m.delta_r, m.observed_redshift_gal, m.sed_ke, m.gr_cosmos, <br />
m.sed_cosmos, m.ext_curve_cosmos, m.ebv_cosmos, m.is_within_cosmos, m.cosmos_distance, m.abs_mag_uv_dereddened, m.log_ml_r01,<br />
m.log_stellar_mass, m.metallicity, m.log_sfr, m.logf_halpha_ext, m.logf_halpha, m.halpha_scatter, m.loglum_halpha, m.loglum_halpha_ext, <br />
m.logf_dummy, m.z_dummy, m.logf_halpha_model1_ext, m.logf_halpha_model1, m.loglum_halpha_model1_ext, m.loglum_halpha_model1, <br />
m.logf_halpha_model3_ext, m.logf_halpha_model3, m.loglum_halpha_model3_ext, m.loglum_halpha_model3, m.logf_hbeta_model1_ext, <br />
m.logf_hbeta_model1, m.logf_o2_model1_ext, m.logf_o2_model1, m.logf_n2_model1_ext, m.logf_n2_model1, m.logf_o3_model1_ext, <br />
m.logf_o3_model1, m.logf_s2_model1_ext, m.logf_s2_model1, m.logf_hbeta_model3_ext, m.logf_hbeta_model3, m.logf_o2_model3_ext, <br />
m.logf_o2_model3, m.logf_n2_model3_ext, m.logf_n2_model3, m.logf_o3_model3_ext, m.logf_o3_model3, m.logf_s2_model3_ext,<br />
m.logf_s2_model3, m.blanco_decam_g, m.blanco_decam_i, m.blanco_decam_r, m.blanco_decam_z, m.gaia_bp, m.gaia_g, m.gaia_rp, <br />
m.subaru_hsc_z, m.jst_jpcam_g, m.kids_g, m.kids_i, m.kids_r, m.kids_u, m.lsst_g, m.lsst_i, m.lsst_r, m.lsst_u, m.lsst_y, m.lsst_z, <br />
m.cfht_megacam_r, m.cfht_megacam_u, m.pan_starrs_i, m.pan_starrs_z, m.euclid_nisp_h, m.euclid_nisp_j, m.euclid_nisp_y, m.euclid_vis, <br />
m.2mass_h, m.2mass_j, m.2mass_ks, m.sdss_r01, m.l_gal, m.b_gal,mw_extinction, m.2mass_h_odonnell_ext, m.2mass_j_odonnell_ext, <br />
m.2mass_ks_odonnell_ext, m.blanco_decam_g_odonnell_ext, m.blanco_decam_i_odonnell_ext, m.blanco_decam_r_odonnell_ext, <br />
m.blanco_decam_z_odonnell_ext, m.cfht_megacam_r_odonnell_ext, m.cfht_megacam_u_odonnell_ext, m.euclid_nisp_h_odonnell_ext, <br />
m.euclid_nisp_j_odonnell_ext, m.euclid_nisp_y_odonnell_ext, m.euclid_vis_odonnell_ext, m.gaia_bp_odonnell_ext, m.gaia_g_odonnell_ext, <br />
m.gaia_rp_odonnell_ext, m.jst_jpcam_g_odonnell_ext, m.kids_g_odonnell_ext, m.kids_i_odonnell_ext, m.kids_r_odonnell_ext, <br />
m.kids_u_odonnell_ext, m.lsst_g_odonnell_ext, m.lsst_i_odonnell_ext, m.lsst_r_odonnell_ext, m.lsst_u_odonnell_ext, <br />
m.lsst_y_odonnell_ext, m.lsst_z_odonnell_ext, m.pan_starrs_i_odonnell_ext, m.pan_starrs_z_odonnell_ext, m.sdss_r01_odonnell_ext, <br />
m.subaru_hsc_z_odonnell_ext, m.blanco_decam_g_el_model1_odonnell_ext, m.blanco_decam_g_el_model3_odonnell_ext, <br />
m.blanco_decam_i_el_model1_odonnell_ext, m.blanco_decam_i_el_model3_odonnell_ext, m.blanco_decam_r_el_model1_odonnell_ext, <br />
m.blanco_decam_r_el_model3_odonnell_ext, m.blanco_decam_z_el_model1_odonnell_ext, m.blanco_decam_z_el_model3_odonnell_ext,<br />
m.gaia_bp_el_model1_odonnell_ext, m.gaia_bp_el_model3_odonnell_ext, m.gaia_g_el_model1_odonnell_ext, m.gaia_g_el_model3_odonnell_ext, <br />
m.gaia_rp_el_model1_odonnell_ext, m.gaia_rp_el_model3_odonnell_ext, m.subaru_hsc_z_el_model1_odonnell_ext, <br />
m.subaru_hsc_z_el_model3_odonnell_ext, m.jst_jpcam_g_el_model1_odonnell_ext, m.jst_jpcam_g_el_model3_odonnell_ext,<br />
m.kids_g_el_model1_odonnell_ext, m.kids_g_el_model3_odonnell_ext, m.kids_i_el_model1_odonnell_ext, m.kids_i_el_model3_odonnell_ext, <br />
m.kids_r_el_model1_odonnell_ext, m.kids_r_el_model3_odonnell_ext, m.kids_u_el_model1_odonnell_ext, m.kids_u_el_model3_odonnell_ext, <br />
m.lsst_g_el_model1_odonnell_ext, m.lsst_g_el_model3_odonnell_ext, m.lsst_i_el_model1_odonnell_ext, m.lsst_i_el_model3_odonnell_ext,<br />
m.lsst_r_el_model1_odonnell_ext, m.lsst_r_el_model3_odonnell_ext, m.lsst_u_el_model1_odonnell_ext, m.lsst_u_el_model3_odonnell_ext, <br />
m.lsst_y_el_model1_odonnell_ext, m.lsst_y_el_model3_odonnell_ext, m.lsst_z_el_model1_odonnell_ext, m.lsst_z_el_model3_odonnell_ext, <br />
m.cfht_megacam_r_el_model1_odonnell_ext, m.cfht_megacam_r_el_model3_odonnell_ext, m.cfht_megacam_u_el_model1_odonnell_ext,<br />
m.cfht_megacam_u_el_model3_odonnell_ext, m.pan_starrs_i_el_model1_odonnell_ext, m.pan_starrs_i_el_model3_odonnell_ext, <br />
m.pan_starrs_z_el_model1_odonnell_ext, m.pan_starrs_z_el_model3_odonnell_ext, m.euclid_nisp_h_el_model1_odonnell_ext, <br />
m.euclid_nisp_h_el_model3_odonnell_ext, m.euclid_nisp_j_el_model1_odonnell_ext, m.euclid_nisp_j_el_model3_odonnell_ext, <br />
m.euclid_nisp_y_el_model1_odonnell_ext, m.euclid_nisp_y_el_model3_odonnell_ext, m.euclid_vis_el_model1_odonnell_ext, <br />
m.euclid_vis_el_model3_odonnell_ext, m.2mass_h_el_model1_odonnell_ext, m.2mass_h_el_model3_odonnell_ext, <br />
m.2mass_j_el_model1_odonnell_ext, m.2mass_j_el_model3_odonnell_ext, m.2mass_ks_el_model1_odonnell_ext, <br />
m.2mass_ks_el_model3_odonnell_ext, m.sdss_r01_el_model1_odonnell_ext, m.sdss_r01_el_model3_odonnell_ext, m.dominant_shape, <br />
m.bulge_angle, m.disk_angle, m.median_major_axis, m.scale_length, m.bulge_fraction, m.disk_scalelength, m.disk_nsersic, m.disk_r50,<br />
m.bulge_r50, m.bulge_nsersic, m.disk_ellipticity, m.bulge_ellipticity, m.disk_axis_ratio, m.bulge_axis_ratio, m.disk_inclination_angle, <br />
m.bulge_inclination_angle, m.random_index, m.halo_id, m.galaxy_id<br />
FROM (<br />
SELECT <br />
m.kind, m.luminosity_r01, m.halo_x, m.halo_y, m.halo_z, m.halo_vx, m.halo_vy, m.halo_vz, m.halo_r, m.true_redshift_halo, m.halo_lm, <br />
m.halo_n_sats, m.n_gals, m.abs_mag_r01, m.abs_mag_r01_evolved, m.luminosity_r01_evolved, m.gr_restframe, m.color_kind, m.x_gal, <br />
m.y_gal, m.z_gal, m.r_gal, m.true_redshift_gal, m.ra_gal, m.dec_gal, l.defl_0 as kappa, l.shear_1 as gamma1, l.shear_2 as gamma2,<br />
m.hpix_29_nest, m.hpix_13_nest, m.vx_gal, m.vy_gal, m.vz_gal, m.vrad_gal, m.delta_r, m.observed_redshift_gal, m.sed_ke, m.gr_cosmos, <br />
m.sed_cosmos, m.ext_curve_cosmos, m.ebv_cosmos, m.is_within_cosmos, m.cosmos_distance, m.abs_mag_uv_dereddened, m.log_ml_r01, <br />
m.log_stellar_mass, m.metallicity, m.log_sfr, m.logf_halpha_ext, m.logf_halpha, m.halpha_scatter, m.loglum_halpha, <br />
m.loglum_halpha_ext, m.logf_dummy, m.z_dummy, m.logf_halpha_model1_ext, m.logf_halpha_model1, m.loglum_halpha_model1_ext, <br />
m.loglum_halpha_model1, m.logf_halpha_model3_ext, m.logf_halpha_model3, m.loglum_halpha_model3_ext, m.loglum_halpha_model3, <br />
m.logf_hbeta_model1_ext, m.logf_hbeta_model1, m.logf_o2_model1_ext, m.logf_o2_model1, m.logf_n2_model1_ext, m.logf_n2_model1,<br />
m.logf_o3_model1_ext, m.logf_o3_model1, m.logf_s2_model1_ext, m.logf_s2_model1, m.logf_hbeta_model3_ext, m.logf_hbeta_model3, <br />
m.logf_o2_model3_ext, m.logf_o2_model3, m.logf_n2_model3_ext, m.logf_n2_model3, m.logf_o3_model3_ext, m.logf_o3_model3, <br />
m.logf_s2_model3_ext, m.logf_s2_model3, m.blanco_decam_g, m.blanco_decam_i, m.blanco_decam_r, m.blanco_decam_z, m.gaia_bp, m.gaia_g, <br />
m.gaia_rp, m.subaru_hsc_z, m.jst_jpcam_g, m.kids_g, m.kids_i, m.kids_r, m.kids_u, m.lsst_g, m.lsst_i, m.lsst_r, m.lsst_u, m.lsst_y, <br />
m.lsst_z, m.cfht_megacam_r, m.cfht_megacam_u, m.pan_starrs_i, m.pan_starrs_z, m.euclid_nisp_h, m.euclid_nisp_j, m.euclid_nisp_y, <br />
m.euclid_vis, m.2mass_h, m.2mass_j, m.2mass_ks, m.sdss_r01, m.l_gal, m.b_gal,mw_extinction, m.2mass_h_odonnell_ext, <br />
m.2mass_j_odonnell_ext, m.2mass_ks_odonnell_ext, m.blanco_decam_g_odonnell_ext, m.blanco_decam_i_odonnell_ext, <br />
m.blanco_decam_r_odonnell_ext, m.blanco_decam_z_odonnell_ext, m.cfht_megacam_r_odonnell_ext, m.cfht_megacam_u_odonnell_ext,<br />
m.euclid_nisp_h_odonnell_ext, m.euclid_nisp_j_odonnell_ext, m.euclid_nisp_y_odonnell_ext, m.euclid_vis_odonnell_ext, <br />
m.gaia_bp_odonnell_ext, m.gaia_g_odonnell_ext, m.gaia_rp_odonnell_ext, m.jst_jpcam_g_odonnell_ext, m.kids_g_odonnell_ext, <br />
m.kids_i_odonnell_ext, m.kids_r_odonnell_ext, m.kids_u_odonnell_ext, m.lsst_g_odonnell_ext, m.lsst_i_odonnell_ext, <br />
m.lsst_r_odonnell_ext, m.lsst_u_odonnell_ext, m.lsst_y_odonnell_ext, m.lsst_z_odonnell_ext, m.pan_starrs_i_odonnell_ext, <br />
m.pan_starrs_z_odonnell_ext, m.sdss_r01_odonnell_ext, m.subaru_hsc_z_odonnell_ext, m.blanco_decam_g_el_model1_odonnell_ext, <br />
m.blanco_decam_g_el_model3_odonnell_ext, m.blanco_decam_i_el_model1_odonnell_ext, m.blanco_decam_i_el_model3_odonnell_ext, <br />
m.blanco_decam_r_el_model1_odonnell_ext, m.blanco_decam_r_el_model3_odonnell_ext, m.blanco_decam_z_el_model1_odonnell_ext, <br />
m.blanco_decam_z_el_model3_odonnell_ext, m.gaia_bp_el_model1_odonnell_ext, m.gaia_bp_el_model3_odonnell_ext,<br />
m.gaia_g_el_model1_odonnell_ext, m.gaia_g_el_model3_odonnell_ext, m.gaia_rp_el_model1_odonnell_ext, m.gaia_rp_el_model3_odonnell_ext, <br />
m.subaru_hsc_z_el_model1_odonnell_ext, m.subaru_hsc_z_el_model3_odonnell_ext, m.jst_jpcam_g_el_model1_odonnell_ext, <br />
m.jst_jpcam_g_el_model3_odonnell_ext, m.kids_g_el_model1_odonnell_ext, m.kids_g_el_model3_odonnell_ext,<br />
m.kids_i_el_model1_odonnell_ext, m.kids_i_el_model3_odonnell_ext, m.kids_r_el_model1_odonnell_ext, m.kids_r_el_model3_odonnell_ext, <br />
m.kids_u_el_model1_odonnell_ext, m.kids_u_el_model3_odonnell_ext, m.lsst_g_el_model1_odonnell_ext, m.lsst_g_el_model3_odonnell_ext, <br />
m.lsst_i_el_model1_odonnell_ext, m.lsst_i_el_model3_odonnell_ext, m.lsst_r_el_model1_odonnell_ext, m.lsst_r_el_model3_odonnell_ext,<br />
m.lsst_u_el_model1_odonnell_ext, m.lsst_u_el_model3_odonnell_ext, m.lsst_y_el_model1_odonnell_ext, m.lsst_y_el_model3_odonnell_ext, <br />
m.lsst_z_el_model1_odonnell_ext, m.lsst_z_el_model3_odonnell_ext, m.cfht_megacam_r_el_model1_odonnell_ext, <br />
m.cfht_megacam_r_el_model3_odonnell_ext, m.cfht_megacam_u_el_model1_odonnell_ext, m.cfht_megacam_u_el_model3_odonnell_ext, <br />
m.pan_starrs_i_el_model1_odonnell_ext, m.pan_starrs_i_el_model3_odonnell_ext, m.pan_starrs_z_el_model1_odonnell_ext, <br />
m.pan_starrs_z_el_model3_odonnell_ext, m.euclid_nisp_h_el_model1_odonnell_ext, m.euclid_nisp_h_el_model3_odonnell_ext,<br />
m.euclid_nisp_j_el_model1_odonnell_ext, m.euclid_nisp_j_el_model3_odonnell_ext, m.euclid_nisp_y_el_model1_odonnell_ext, <br />
m.euclid_nisp_y_el_model3_odonnell_ext, m.euclid_vis_el_model1_odonnell_ext, m.euclid_vis_el_model3_odonnell_ext, <br />
m.2mass_h_el_model1_odonnell_ext, m.2mass_h_el_model3_odonnell_ext, m.2mass_j_el_model1_odonnell_ext, <br />
m.2mass_j_el_model3_odonnell_ext, m.2mass_ks_el_model1_odonnell_ext, m.2mass_ks_el_model3_odonnell_ext, <br />
m.sdss_r01_el_model1_odonnell_ext, m.sdss_r01_el_model3_odonnell_ext, m.dominant_shape, m.bulge_angle, m.disk_angle, <br />
m.median_major_axis, m.scale_length, m.bulge_fraction, m.disk_scalelength, m.disk_nsersic, m.disk_r50, m.bulge_r50, m.bulge_nsersic, <br />
m.disk_ellipticity, m.bulge_ellipticity, m.disk_axis_ratio, m.bulge_axis_ratio, m.disk_inclination_angle, m.bulge_inclination_angle, <br />
m.random_index, m.halo_id, m.galaxy_id,<br />
udf.magnified_positions(CAST(m.ra_gal AS FLOAT), CAST(m.dec_gal AS FLOAT), l.defl_1, l.defl_2) AS magpos<br />
FROM tallada.flagship_mock_1_9_2_c2 AS m<br />
JOIN cosmohub.flagship_lensing AS l<br />
ON udf.ang2pix(13, m.ra_gal, m.dec_gal, True, True) = l.hpix<br />
AND m.step = l.step<br />
AND l.step=${STEP} AND m.step=${STEP}<br />
) AS m<br />
;<br />
<br />
for STEP in $(seq 344 400);<br />
do<br />
echo $STEP<br />
sem -j4 "beeline --color=true --showHeader=true --verbose=true --silent=false -u \<br />
'jdbc:hive2://ambarisrv02.pic.es:2181,ambarisrv03.pic.es:2181,ambarisrv01.pic.es:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2' \<br />
-f lensing_step.sql --hivevar STEP=${STEP}"<br />
done<br />
sem --wait</div>
Tallada
https://pwiki.pic.es/index.php?title=Euclid_mock_production_v1.9.2&diff=821
Euclid mock production v1.9.2
2020-07-14T08:03:20Z
<p>Tallada: </p>
<hr />
<div> CREATE EXTERNAL TABLE `flagship_1_9_2`(<br />
`kind` bigint, <br />
`luminosity_r01` double, <br />
`halo_x` double, <br />
`halo_y` double, <br />
`halo_z` double, <br />
`halo_vx` double, <br />
`halo_vy` double, <br />
`halo_vz` double, <br />
`halo_r` double, <br />
`true_redshift_halo` double, <br />
`halo_lm` double, <br />
`halo_n_sats` bigint, <br />
`n_gals` bigint, <br />
`abs_mag_r01` double, <br />
`abs_mag_r01_evolved` double, <br />
`luminosity_r01_evolved` double, <br />
`gr_restframe` double, <br />
`color_kind` int, <br />
`x_gal` double, <br />
`y_gal` double, <br />
`z_gal` double, <br />
`r_gal` double, <br />
`true_redshift_gal` double, <br />
`ra_gal` double, <br />
`dec_gal` double, <br />
`hpix_nest` bigint, <br />
`vx_gal` double, <br />
`vy_gal` double, <br />
`vz_gal` double, <br />
`vrad_gal` double, <br />
`delta_r` double, <br />
`observed_redshift_gal` double, <br />
`sed_ke` double, <br />
`gr_cosmos` double, <br />
`sed_cosmos` double, <br />
`ext_curve_cosmos` bigint, <br />
`ebv_cosmos` double, <br />
`is_within_cosmos` boolean, <br />
`cosmos_distance` double, <br />
`abs_mag_uv_dereddened` double, <br />
`log_ml_r01` double, <br />
`log_stellar_mass` double, <br />
`metallicity` double, <br />
`log_sfr` double, <br />
`logf_halpha_ext` double, <br />
`logf_halpha` double, <br />
`halpha_scatter` double, <br />
`loglum_halpha` double, <br />
`loglum_halpha_ext` double, <br />
`logf_dummy` double, <br />
`z_dummy` double, <br />
`logf_halpha_model1_ext` double, <br />
`logf_halpha_model1` double, <br />
`loglum_halpha_model1_ext` double, <br />
`loglum_halpha_model1` double, <br />
`logf_halpha_model3_ext` double, <br />
`logf_halpha_model3` double, <br />
`loglum_halpha_model3_ext` double, <br />
`loglum_halpha_model3` double, <br />
`logf_hbeta_model1_ext` double, <br />
`logf_hbeta_model1` double, <br />
`logf_o2_model1_ext` double, <br />
`logf_o2_model1` double, <br />
`logf_n2_model1_ext` double, <br />
`logf_n2_model1` double, <br />
`logf_o3_model1_ext` double, <br />
`logf_o3_model1` double, <br />
`logf_s2_model1_ext` double, <br />
`logf_s2_model1` double, <br />
`logf_hbeta_model3_ext` double, <br />
`logf_hbeta_model3` double, <br />
`logf_o2_model3_ext` double, <br />
`logf_o2_model3` double, <br />
`logf_n2_model3_ext` double, <br />
`logf_n2_model3` double, <br />
`logf_o3_model3_ext` double, <br />
`logf_o3_model3` double, <br />
`logf_s2_model3_ext` double, <br />
`logf_s2_model3` double, <br />
`blanco_decam_g` double, <br />
`blanco_decam_i` double, <br />
`blanco_decam_r` double, <br />
`blanco_decam_z` double, <br />
`gaia_bp` double, <br />
`gaia_g` double, <br />
`gaia_rp` double, <br />
`subaru_hsc_z` double, <br />
`jst_jpcam_g` double, <br />
`kids_g` double, <br />
`kids_i` double, <br />
`kids_r` double, <br />
`kids_u` double, <br />
`lsst_g` double, <br />
`lsst_i` double, <br />
`lsst_r` double, <br />
`lsst_u` double, <br />
`lsst_y` double, <br />
`lsst_z` double, <br />
`cfht_megacam_r` double, <br />
`cfht_megacam_u` double, <br />
`pan_starrs_i` double, <br />
`pan_starrs_z` double, <br />
`euclid_nisp_h` double, <br />
`euclid_nisp_j` double, <br />
`euclid_nisp_y` double, <br />
`euclid_vis` double, <br />
`2mass_h` double, <br />
`2mass_j` double, <br />
`2mass_ks` double, <br />
`sdss_r01` double, <br />
`l_gal` double, <br />
`b_gal` double, <br />
`mw_extinction` double, <br />
`2mass_h_odonnell_ext` double, <br />
`2mass_j_odonnell_ext` double, <br />
`2mass_ks_odonnell_ext` double, <br />
`blanco_decam_g_odonnell_ext` double, <br />
`blanco_decam_i_odonnell_ext` double, <br />
`blanco_decam_r_odonnell_ext` double, <br />
`blanco_decam_z_odonnell_ext` double, <br />
`cfht_megacam_r_odonnell_ext` double, <br />
`cfht_megacam_u_odonnell_ext` double, <br />
`euclid_nisp_h_odonnell_ext` double, <br />
`euclid_nisp_j_odonnell_ext` double, <br />
`euclid_nisp_y_odonnell_ext` double, <br />
`euclid_vis_odonnell_ext` double, <br />
`gaia_bp_odonnell_ext` double, <br />
`gaia_g_odonnell_ext` double, <br />
`gaia_rp_odonnell_ext` double, <br />
`jst_jpcam_g_odonnell_ext` double, <br />
`kids_g_odonnell_ext` double, <br />
`kids_i_odonnell_ext` double, <br />
`kids_r_odonnell_ext` double, <br />
`kids_u_odonnell_ext` double, <br />
`lsst_g_odonnell_ext` double, <br />
`lsst_i_odonnell_ext` double, <br />
`lsst_r_odonnell_ext` double, <br />
`lsst_u_odonnell_ext` double, <br />
`lsst_y_odonnell_ext` double, <br />
`lsst_z_odonnell_ext` double, <br />
`pan_starrs_i_odonnell_ext` double, <br />
`pan_starrs_z_odonnell_ext` double, <br />
`sdss_r01_odonnell_ext` double, <br />
`subaru_hsc_z_odonnell_ext` double, <br />
`blanco_decam_g_el_model1_odonnell_ext` double, <br />
`blanco_decam_g_el_model3_odonnell_ext` double, <br />
`blanco_decam_i_el_model1_odonnell_ext` double, <br />
`blanco_decam_i_el_model3_odonnell_ext` double, <br />
`blanco_decam_r_el_model1_odonnell_ext` double, <br />
`blanco_decam_r_el_model3_odonnell_ext` double, <br />
`blanco_decam_z_el_model1_odonnell_ext` double, <br />
`blanco_decam_z_el_model3_odonnell_ext` double, <br />
`gaia_bp_el_model1_odonnell_ext` double, <br />
`gaia_bp_el_model3_odonnell_ext` double, <br />
`gaia_g_el_model1_odonnell_ext` double, <br />
`gaia_g_el_model3_odonnell_ext` double, <br />
`gaia_rp_el_model1_odonnell_ext` double, <br />
`gaia_rp_el_model3_odonnell_ext` double, <br />
`subaru_hsc_z_el_model1_odonnell_ext` double, <br />
`subaru_hsc_z_el_model3_odonnell_ext` double, <br />
`jst_jpcam_g_el_model1_odonnell_ext` double, <br />
`jst_jpcam_g_el_model3_odonnell_ext` double, <br />
`kids_g_el_model1_odonnell_ext` double, <br />
`kids_g_el_model3_odonnell_ext` double, <br />
`kids_i_el_model1_odonnell_ext` double, <br />
`kids_i_el_model3_odonnell_ext` double, <br />
`kids_r_el_model1_odonnell_ext` double, <br />
`kids_r_el_model3_odonnell_ext` double, <br />
`kids_u_el_model1_odonnell_ext` double, <br />
`kids_u_el_model3_odonnell_ext` double, <br />
`lsst_g_el_model1_odonnell_ext` double, <br />
`lsst_g_el_model3_odonnell_ext` double, <br />
`lsst_i_el_model1_odonnell_ext` double, <br />
`lsst_i_el_model3_odonnell_ext` double, <br />
`lsst_r_el_model1_odonnell_ext` double, <br />
`lsst_r_el_model3_odonnell_ext` double, <br />
`lsst_u_el_model1_odonnell_ext` double, <br />
`lsst_u_el_model3_odonnell_ext` double, <br />
`lsst_y_el_model1_odonnell_ext` double, <br />
`lsst_y_el_model3_odonnell_ext` double, <br />
`lsst_z_el_model1_odonnell_ext` double, <br />
`lsst_z_el_model3_odonnell_ext` double, <br />
`cfht_megacam_r_el_model1_odonnell_ext` double, <br />
`cfht_megacam_r_el_model3_odonnell_ext` double, <br />
`cfht_megacam_u_el_model1_odonnell_ext` double, <br />
`cfht_megacam_u_el_model3_odonnell_ext` double, <br />
`pan_starrs_i_el_model1_odonnell_ext` double, <br />
`pan_starrs_i_el_model3_odonnell_ext` double, <br />
`pan_starrs_z_el_model1_odonnell_ext` double, <br />
`pan_starrs_z_el_model3_odonnell_ext` double, <br />
`euclid_nisp_h_el_model1_odonnell_ext` double, <br />
`euclid_nisp_h_el_model3_odonnell_ext` double, <br />
`euclid_nisp_j_el_model1_odonnell_ext` double, <br />
`euclid_nisp_j_el_model3_odonnell_ext` double, <br />
`euclid_nisp_y_el_model1_odonnell_ext` double, <br />
`euclid_nisp_y_el_model3_odonnell_ext` double, <br />
`euclid_vis_el_model1_odonnell_ext` double, <br />
`euclid_vis_el_model3_odonnell_ext` double, <br />
`2mass_h_el_model1_odonnell_ext` double, <br />
`2mass_h_el_model3_odonnell_ext` double, <br />
`2mass_j_el_model1_odonnell_ext` double, <br />
`2mass_j_el_model3_odonnell_ext` double, <br />
`2mass_ks_el_model1_odonnell_ext` double, <br />
`2mass_ks_el_model3_odonnell_ext` double, <br />
`sdss_r01_el_model1_odonnell_ext` double, <br />
`sdss_r01_el_model3_odonnell_ext` double, <br />
`dominant_shape` bigint, <br />
`bulge_angle` double, <br />
`disk_angle` double, <br />
`median_major_axis` double, <br />
`scale_length` double, <br />
`bulge_fraction` double, <br />
`disk_scalelength` double, <br />
`disk_nsersic` double, <br />
`disk_r50` double, <br />
`bulge_r50` double, <br />
`bulge_nsersic` double, <br />
`disk_ellipticity` double, <br />
`bulge_ellipticity` double, <br />
`disk_axis_ratio` double, <br />
`bulge_axis_ratio` double, <br />
`disk_inclination_angle` double, <br />
`bulge_inclination_angle` double, <br />
`step` bigint, <br />
`random_index` double, <br />
`halo_id` bigint, <br />
`galaxy_id` bigint)<br />
ROW FORMAT SERDE <br />
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' <br />
STORED AS INPUTFORMAT <br />
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' <br />
OUTPUTFORMAT <br />
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'<br />
;<br />
<br />
CREATE TABLE flagship_mock_1_9_2_c(<br />
`kind` bigint, <br />
`luminosity_r01` double, <br />
`halo_x` double, <br />
`halo_y` double, <br />
`halo_z` double, <br />
`halo_vx` double, <br />
`halo_vy` double, <br />
`halo_vz` double, <br />
`halo_r` double, <br />
`true_redshift_halo` double, <br />
`halo_lm` double, <br />
`halo_n_sats` bigint, <br />
`n_gals` bigint, <br />
`abs_mag_r01` double, <br />
`abs_mag_r01_evolved` double, <br />
`luminosity_r01_evolved` double, <br />
`gr_restframe` double, <br />
`color_kind` int, <br />
`x_gal` double, <br />
`y_gal` double, <br />
`z_gal` double, <br />
`r_gal` double, <br />
`true_redshift_gal` double, <br />
`ra_gal` double, <br />
`dec_gal` double, <br />
`hpix_nest` bigint, <br />
`vx_gal` double, <br />
`vy_gal` double, <br />
`vz_gal` double, <br />
`vrad_gal` double, <br />
`delta_r` double, <br />
`observed_redshift_gal` double, <br />
`sed_ke` double, <br />
`gr_cosmos` double, <br />
`sed_cosmos` double, <br />
`ext_curve_cosmos` bigint, <br />
`ebv_cosmos` double, <br />
`is_within_cosmos` boolean, <br />
`cosmos_distance` double, <br />
`abs_mag_uv_dereddened` double, <br />
`log_ml_r01` double, <br />
`log_stellar_mass` double, <br />
`metallicity` double, <br />
`log_sfr` double, <br />
`logf_halpha_ext` double, <br />
`logf_halpha` double, <br />
`halpha_scatter` double, <br />
`loglum_halpha` double, <br />
`loglum_halpha_ext` double, <br />
`logf_dummy` double, <br />
`z_dummy` double, <br />
`logf_halpha_model1_ext` double, <br />
`logf_halpha_model1` double, <br />
`loglum_halpha_model1_ext` double, <br />
`loglum_halpha_model1` double, <br />
`logf_halpha_model3_ext` double, <br />
`logf_halpha_model3` double, <br />
`loglum_halpha_model3_ext` double, <br />
`loglum_halpha_model3` double, <br />
`logf_hbeta_model1_ext` double, <br />
`logf_hbeta_model1` double, <br />
`logf_o2_model1_ext` double, <br />
`logf_o2_model1` double, <br />
`logf_n2_model1_ext` double, <br />
`logf_n2_model1` double, <br />
`logf_o3_model1_ext` double, <br />
`logf_o3_model1` double, <br />
`logf_s2_model1_ext` double, <br />
`logf_s2_model1` double, <br />
`logf_hbeta_model3_ext` double, <br />
`logf_hbeta_model3` double, <br />
`logf_o2_model3_ext` double, <br />
`logf_o2_model3` double, <br />
`logf_n2_model3_ext` double, <br />
`logf_n2_model3` double, <br />
`logf_o3_model3_ext` double, <br />
`logf_o3_model3` double, <br />
`logf_s2_model3_ext` double, <br />
`logf_s2_model3` double, <br />
`blanco_decam_g` double, <br />
`blanco_decam_i` double, <br />
`blanco_decam_r` double, <br />
`blanco_decam_z` double, <br />
`gaia_bp` double, <br />
`gaia_g` double, <br />
`gaia_rp` double, <br />
`subaru_hsc_z` double, <br />
`jst_jpcam_g` double, <br />
`kids_g` double, <br />
`kids_i` double, <br />
`kids_r` double, <br />
`kids_u` double, <br />
`lsst_g` double, <br />
`lsst_i` double, <br />
`lsst_r` double, <br />
`lsst_u` double, <br />
`lsst_y` double, <br />
`lsst_z` double, <br />
`cfht_megacam_r` double, <br />
`cfht_megacam_u` double, <br />
`pan_starrs_i` double, <br />
`pan_starrs_z` double, <br />
`euclid_nisp_h` double, <br />
`euclid_nisp_j` double, <br />
`euclid_nisp_y` double, <br />
`euclid_vis` double, <br />
`2mass_h` double, <br />
`2mass_j` double, <br />
`2mass_ks` double, <br />
`sdss_r01` double, <br />
`l_gal` double, <br />
`b_gal` double, <br />
`mw_extinction` double, <br />
`2mass_h_odonnell_ext` double, <br />
`2mass_j_odonnell_ext` double, <br />
`2mass_ks_odonnell_ext` double, <br />
`blanco_decam_g_odonnell_ext` double, <br />
`blanco_decam_i_odonnell_ext` double, <br />
`blanco_decam_r_odonnell_ext` double, <br />
`blanco_decam_z_odonnell_ext` double, <br />
`cfht_megacam_r_odonnell_ext` double, <br />
`cfht_megacam_u_odonnell_ext` double, <br />
`euclid_nisp_h_odonnell_ext` double, <br />
`euclid_nisp_j_odonnell_ext` double, <br />
`euclid_nisp_y_odonnell_ext` double, <br />
`euclid_vis_odonnell_ext` double, <br />
`gaia_bp_odonnell_ext` double, <br />
`gaia_g_odonnell_ext` double, <br />
`gaia_rp_odonnell_ext` double, <br />
`jst_jpcam_g_odonnell_ext` double, <br />
`kids_g_odonnell_ext` double, <br />
`kids_i_odonnell_ext` double, <br />
`kids_r_odonnell_ext` double, <br />
`kids_u_odonnell_ext` double, <br />
`lsst_g_odonnell_ext` double, <br />
`lsst_i_odonnell_ext` double, <br />
`lsst_r_odonnell_ext` double, <br />
`lsst_u_odonnell_ext` double, <br />
`lsst_y_odonnell_ext` double, <br />
`lsst_z_odonnell_ext` double, <br />
`pan_starrs_i_odonnell_ext` double, <br />
`pan_starrs_z_odonnell_ext` double, <br />
`sdss_r01_odonnell_ext` double, <br />
`subaru_hsc_z_odonnell_ext` double, <br />
`blanco_decam_g_el_model1_odonnell_ext` double, <br />
`blanco_decam_g_el_model3_odonnell_ext` double, <br />
`blanco_decam_i_el_model1_odonnell_ext` double, <br />
`blanco_decam_i_el_model3_odonnell_ext` double, <br />
`blanco_decam_r_el_model1_odonnell_ext` double, <br />
`blanco_decam_r_el_model3_odonnell_ext` double, <br />
`blanco_decam_z_el_model1_odonnell_ext` double, <br />
`blanco_decam_z_el_model3_odonnell_ext` double, <br />
`gaia_bp_el_model1_odonnell_ext` double, <br />
`gaia_bp_el_model3_odonnell_ext` double, <br />
`gaia_g_el_model1_odonnell_ext` double, <br />
`gaia_g_el_model3_odonnell_ext` double, <br />
`gaia_rp_el_model1_odonnell_ext` double, <br />
`gaia_rp_el_model3_odonnell_ext` double, <br />
`subaru_hsc_z_el_model1_odonnell_ext` double, <br />
`subaru_hsc_z_el_model3_odonnell_ext` double, <br />
`jst_jpcam_g_el_model1_odonnell_ext` double, <br />
`jst_jpcam_g_el_model3_odonnell_ext` double, <br />
`kids_g_el_model1_odonnell_ext` double, <br />
`kids_g_el_model3_odonnell_ext` double, <br />
`kids_i_el_model1_odonnell_ext` double, <br />
`kids_i_el_model3_odonnell_ext` double, <br />
`kids_r_el_model1_odonnell_ext` double, <br />
`kids_r_el_model3_odonnell_ext` double, <br />
`kids_u_el_model1_odonnell_ext` double, <br />
`kids_u_el_model3_odonnell_ext` double, <br />
`lsst_g_el_model1_odonnell_ext` double, <br />
`lsst_g_el_model3_odonnell_ext` double, <br />
`lsst_i_el_model1_odonnell_ext` double, <br />
`lsst_i_el_model3_odonnell_ext` double, <br />
`lsst_r_el_model1_odonnell_ext` double, <br />
`lsst_r_el_model3_odonnell_ext` double, <br />
`lsst_u_el_model1_odonnell_ext` double, <br />
`lsst_u_el_model3_odonnell_ext` double, <br />
`lsst_y_el_model1_odonnell_ext` double, <br />
`lsst_y_el_model3_odonnell_ext` double, <br />
`lsst_z_el_model1_odonnell_ext` double, <br />
`lsst_z_el_model3_odonnell_ext` double, <br />
`cfht_megacam_r_el_model1_odonnell_ext` double, <br />
`cfht_megacam_r_el_model3_odonnell_ext` double, <br />
`cfht_megacam_u_el_model1_odonnell_ext` double, <br />
`cfht_megacam_u_el_model3_odonnell_ext` double, <br />
`pan_starrs_i_el_model1_odonnell_ext` double, <br />
`pan_starrs_i_el_model3_odonnell_ext` double, <br />
`pan_starrs_z_el_model1_odonnell_ext` double, <br />
`pan_starrs_z_el_model3_odonnell_ext` double, <br />
`euclid_nisp_h_el_model1_odonnell_ext` double, <br />
`euclid_nisp_h_el_model3_odonnell_ext` double, <br />
`euclid_nisp_j_el_model1_odonnell_ext` double, <br />
`euclid_nisp_j_el_model3_odonnell_ext` double, <br />
`euclid_nisp_y_el_model1_odonnell_ext` double, <br />
`euclid_nisp_y_el_model3_odonnell_ext` double, <br />
`euclid_vis_el_model1_odonnell_ext` double, <br />
`euclid_vis_el_model3_odonnell_ext` double, <br />
`2mass_h_el_model1_odonnell_ext` double, <br />
`2mass_h_el_model3_odonnell_ext` double, <br />
`2mass_j_el_model1_odonnell_ext` double, <br />
`2mass_j_el_model3_odonnell_ext` double, <br />
`2mass_ks_el_model1_odonnell_ext` double, <br />
`2mass_ks_el_model3_odonnell_ext` double, <br />
`sdss_r01_el_model1_odonnell_ext` double, <br />
`sdss_r01_el_model3_odonnell_ext` double, <br />
`dominant_shape` bigint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)', <br />
`bulge_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`disk_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`median_major_axis` double COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)', <br />
`scale_length` double COMMENT 'disc and bulge scalelength prior', <br />
`bulge_fraction` double COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)', <br />
`disk_scalelength` double COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)', <br />
`disk_nsersic` double COMMENT 'Sersic index of the disk component', <br />
`disk_r50` double COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678', <br />
`bulge_r50` double COMMENT 'half light radius of the bulge', <br />
`bulge_nsersic` double COMMENT 'Sersic index of the bulge component', <br />
`disk_ellipticity` double COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)', <br />
`bulge_ellipticity` double COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)', <br />
`disk_axis_ratio` double COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`bulge_axis_ratio` double COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',<br />
`disk_inclination_angle` double COMMENT '', <br />
`bulge_inclination_angle` double COMMENT '', <br />
`step` bigint, <br />
`random_index` double, <br />
`halo_id` bigint, <br />
`galaxy_id` bigint<br />
)<br />
CLUSTERED BY ( <br />
halo_id, <br />
galaxy_id<br />
) <br />
INTO 128 BUCKETS<br />
ROW FORMAT SERDE <br />
'org.apache.hadoop.hive.ql.io.orc.OrcSerde' <br />
STORED AS INPUTFORMAT <br />
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' <br />
OUTPUTFORMAT <br />
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'<br />
;<br />
<br />
INSERT OVERWRITE TABLE cosmohub.flagship_mock_1_9_2_c<br />
SELECT * FROM tallada.flagship_1_9_2;<br />
<br />
ANALYZE TABLE cosmohub.flagship_mock_1_9_2_c COMPUTE STATISTICS FOR COLUMNS;<br />
<br />
CREATE TABLE tallada.flagship_mock_1_9_2_c2(<br />
`kind` bigint, <br />
`luminosity_r01` double, <br />
`halo_x` double, <br />
`halo_y` double, <br />
`halo_z` double, <br />
`halo_vx` double, <br />
`halo_vy` double, <br />
`halo_vz` double, <br />
`halo_r` double, <br />
`true_redshift_halo` double, <br />
`halo_lm` double, <br />
`halo_n_sats` bigint, <br />
`n_gals` bigint, <br />
`abs_mag_r01` double, <br />
`abs_mag_r01_evolved` double, <br />
`luminosity_r01_evolved` double, <br />
`gr_restframe` double, <br />
`color_kind` int, <br />
`x_gal` double, <br />
`y_gal` double, <br />
`z_gal` double, <br />
`r_gal` double, <br />
`true_redshift_gal` double, <br />
`ra_gal` double, <br />
`dec_gal` double, <br />
`hpix_29_nest` bigint, <br />
`hpix_13_nest` bigint, <br />
`vx_gal` double, <br />
`vy_gal` double, <br />
`vz_gal` double, <br />
`vrad_gal` double, <br />
`delta_r` double, <br />
`observed_redshift_gal` double, <br />
`sed_ke` double, <br />
`gr_cosmos` double, <br />
`sed_cosmos` double, <br />
`ext_curve_cosmos` bigint, <br />
`ebv_cosmos` double, <br />
`is_within_cosmos` boolean, <br />
`cosmos_distance` double, <br />
`abs_mag_uv_dereddened` double, <br />
`log_ml_r01` double, <br />
`log_stellar_mass` double, <br />
`metallicity` double, <br />
`log_sfr` double, <br />
`logf_halpha_ext` double, <br />
`logf_halpha` double, <br />
`halpha_scatter` double, <br />
`loglum_halpha` double, <br />
`loglum_halpha_ext` double, <br />
`logf_dummy` double, <br />
`z_dummy` double, <br />
`logf_halpha_model1_ext` double, <br />
`logf_halpha_model1` double, <br />
`loglum_halpha_model1_ext` double, <br />
`loglum_halpha_model1` double, <br />
`logf_halpha_model3_ext` double, <br />
`logf_halpha_model3` double, <br />
`loglum_halpha_model3_ext` double, <br />
`loglum_halpha_model3` double, <br />
`logf_hbeta_model1_ext` double, <br />
`logf_hbeta_model1` double, <br />
`logf_o2_model1_ext` double, <br />
`logf_o2_model1` double, <br />
`logf_n2_model1_ext` double, <br />
`logf_n2_model1` double, <br />
`logf_o3_model1_ext` double, <br />
`logf_o3_model1` double, <br />
`logf_s2_model1_ext` double, <br />
`logf_s2_model1` double, <br />
`logf_hbeta_model3_ext` double, <br />
`logf_hbeta_model3` double, <br />
`logf_o2_model3_ext` double, <br />
`logf_o2_model3` double, <br />
`logf_n2_model3_ext` double, <br />
`logf_n2_model3` double, <br />
`logf_o3_model3_ext` double, <br />
`logf_o3_model3` double, <br />
`logf_s2_model3_ext` double, <br />
`logf_s2_model3` double, <br />
`blanco_decam_g` double, <br />
`blanco_decam_i` double, <br />
`blanco_decam_r` double, <br />
`blanco_decam_z` double, <br />
`gaia_bp` double, <br />
`gaia_g` double, <br />
`gaia_rp` double, <br />
`subaru_hsc_z` double, <br />
`jst_jpcam_g` double, <br />
`kids_g` double, <br />
`kids_i` double, <br />
`kids_r` double, <br />
`kids_u` double, <br />
`lsst_g` double, <br />
`lsst_i` double, <br />
`lsst_r` double, <br />
`lsst_u` double, <br />
`lsst_y` double, <br />
`lsst_z` double, <br />
`cfht_megacam_r` double, <br />
`cfht_megacam_u` double, <br />
`pan_starrs_i` double, <br />
`pan_starrs_z` double, <br />
`euclid_nisp_h` double, <br />
`euclid_nisp_j` double, <br />
`euclid_nisp_y` double, <br />
`euclid_vis` double, <br />
`2mass_h` double, <br />
`2mass_j` double, <br />
`2mass_ks` double, <br />
`sdss_r01` double, <br />
`l_gal` double, <br />
`b_gal` double, <br />
`mw_extinction` double, <br />
`2mass_h_odonnell_ext` double, <br />
`2mass_j_odonnell_ext` double, <br />
`2mass_ks_odonnell_ext` double, <br />
`blanco_decam_g_odonnell_ext` double, <br />
`blanco_decam_i_odonnell_ext` double, <br />
`blanco_decam_r_odonnell_ext` double, <br />
`blanco_decam_z_odonnell_ext` double, <br />
`cfht_megacam_r_odonnell_ext` double, <br />
`cfht_megacam_u_odonnell_ext` double, <br />
`euclid_nisp_h_odonnell_ext` double, <br />
`euclid_nisp_j_odonnell_ext` double, <br />
`euclid_nisp_y_odonnell_ext` double, <br />
`euclid_vis_odonnell_ext` double, <br />
`gaia_bp_odonnell_ext` double, <br />
`gaia_g_odonnell_ext` double, <br />
`gaia_rp_odonnell_ext` double, <br />
`jst_jpcam_g_odonnell_ext` double, <br />
`kids_g_odonnell_ext` double, <br />
`kids_i_odonnell_ext` double, <br />
`kids_r_odonnell_ext` double, <br />
`kids_u_odonnell_ext` double, <br />
`lsst_g_odonnell_ext` double, <br />
`lsst_i_odonnell_ext` double, <br />
`lsst_r_odonnell_ext` double, <br />
`lsst_u_odonnell_ext` double, <br />
`lsst_y_odonnell_ext` double, <br />
`lsst_z_odonnell_ext` double, <br />
`pan_starrs_i_odonnell_ext` double, <br />
`pan_starrs_z_odonnell_ext` double, <br />
`sdss_r01_odonnell_ext` double, <br />
`subaru_hsc_z_odonnell_ext` double, <br />
`blanco_decam_g_el_model1_odonnell_ext` double, <br />
`blanco_decam_g_el_model3_odonnell_ext` double, <br />
`blanco_decam_i_el_model1_odonnell_ext` double, <br />
`blanco_decam_i_el_model3_odonnell_ext` double, <br />
`blanco_decam_r_el_model1_odonnell_ext` double, <br />
`blanco_decam_r_el_model3_odonnell_ext` double, <br />
`blanco_decam_z_el_model1_odonnell_ext` double, <br />
`blanco_decam_z_el_model3_odonnell_ext` double, <br />
`gaia_bp_el_model1_odonnell_ext` double, <br />
`gaia_bp_el_model3_odonnell_ext` double, <br />
`gaia_g_el_model1_odonnell_ext` double, <br />
`gaia_g_el_model3_odonnell_ext` double, <br />
`gaia_rp_el_model1_odonnell_ext` double, <br />
`gaia_rp_el_model3_odonnell_ext` double, <br />
`subaru_hsc_z_el_model1_odonnell_ext` double, <br />
`subaru_hsc_z_el_model3_odonnell_ext` double, <br />
`jst_jpcam_g_el_model1_odonnell_ext` double, <br />
`jst_jpcam_g_el_model3_odonnell_ext` double, <br />
`kids_g_el_model1_odonnell_ext` double, <br />
`kids_g_el_model3_odonnell_ext` double, <br />
`kids_i_el_model1_odonnell_ext` double, <br />
`kids_i_el_model3_odonnell_ext` double, <br />
`kids_r_el_model1_odonnell_ext` double, <br />
`kids_r_el_model3_odonnell_ext` double, <br />
`kids_u_el_model1_odonnell_ext` double, <br />
`kids_u_el_model3_odonnell_ext` double, <br />
`lsst_g_el_model1_odonnell_ext` double, <br />
`lsst_g_el_model3_odonnell_ext` double, <br />
`lsst_i_el_model1_odonnell_ext` double, <br />
`lsst_i_el_model3_odonnell_ext` double, <br />
`lsst_r_el_model1_odonnell_ext` double, <br />
`lsst_r_el_model3_odonnell_ext` double, <br />
`lsst_u_el_model1_odonnell_ext` double, <br />
`lsst_u_el_model3_odonnell_ext` double, <br />
`lsst_y_el_model1_odonnell_ext` double, <br />
`lsst_y_el_model3_odonnell_ext` double, <br />
`lsst_z_el_model1_odonnell_ext` double, <br />
`lsst_z_el_model3_odonnell_ext` double, <br />
`cfht_megacam_r_el_model1_odonnell_ext` double, <br />
`cfht_megacam_r_el_model3_odonnell_ext` double, <br />
`cfht_megacam_u_el_model1_odonnell_ext` double, <br />
`cfht_megacam_u_el_model3_odonnell_ext` double, <br />
`pan_starrs_i_el_model1_odonnell_ext` double, <br />
`pan_starrs_i_el_model3_odonnell_ext` double, <br />
`pan_starrs_z_el_model1_odonnell_ext` double, <br />
`pan_starrs_z_el_model3_odonnell_ext` double, <br />
`euclid_nisp_h_el_model1_odonnell_ext` double, <br />
`euclid_nisp_h_el_model3_odonnell_ext` double, <br />
`euclid_nisp_j_el_model1_odonnell_ext` double, <br />
`euclid_nisp_j_el_model3_odonnell_ext` double, <br />
`euclid_nisp_y_el_model1_odonnell_ext` double, <br />
`euclid_nisp_y_el_model3_odonnell_ext` double, <br />
`euclid_vis_el_model1_odonnell_ext` double, <br />
`euclid_vis_el_model3_odonnell_ext` double, <br />
`2mass_h_el_model1_odonnell_ext` double, <br />
`2mass_h_el_model3_odonnell_ext` double, <br />
`2mass_j_el_model1_odonnell_ext` double, <br />
`2mass_j_el_model3_odonnell_ext` double, <br />
`2mass_ks_el_model1_odonnell_ext` double, <br />
`2mass_ks_el_model3_odonnell_ext` double, <br />
`sdss_r01_el_model1_odonnell_ext` double, <br />
`sdss_r01_el_model3_odonnell_ext` double, <br />
`dominant_shape` bigint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)', <br />
`bulge_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`disk_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`median_major_axis` double COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)', <br />
`scale_length` double COMMENT 'disc and bulge scalelength prior', <br />
`bulge_fraction` double COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)', <br />
`disk_scalelength` double COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)', <br />
`disk_nsersic` double COMMENT 'Sersic index of the disk component', <br />
`disk_r50` double COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678', <br />
`bulge_r50` double COMMENT 'half light radius of the bulge', <br />
`bulge_nsersic` double COMMENT 'Sersic index of the bulge component', <br />
`disk_ellipticity` double COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)', <br />
`bulge_ellipticity` double COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)', <br />
`disk_axis_ratio` double COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`bulge_axis_ratio` double COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`disk_inclination_angle` double COMMENT '', <br />
`bulge_inclination_angle` double COMMENT '', <br />
`random_index` double, <br />
`halo_id` bigint, <br />
`galaxy_id` bigint<br />
)<br />
PARTITIONED BY (<br />
`step` smallint<br />
)<br />
CLUSTERED BY (<br />
hpix_13_nest<br />
)<br />
SORTED BY (<br />
hpix_13_nest ASC<br />
)<br />
INTO 128 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
INSERT OVERWRITE TABLE tallada.flagship_mock_1_9_2_c2<br />
SELECT<br />
kind, luminosity_r01, halo_x, halo_y, halo_z, halo_vx, halo_vy, halo_vz, halo_r, true_redshift_halo, halo_lm, halo_n_sats, n_gals,<br />
abs_mag_r01, abs_mag_r01_evolved, luminosity_r01_evolved, gr_restframe, color_kind, x_gal, y_gal, z_gal, r_gal, true_redshift_gal,<br />
ra_gal, dec_gal, hpix_nest, SHIFTRIGHT(hpix_nest, (29-13)*2), vx_gal, vy_gal, vz_gal, vrad_gal, delta_r, observed_redshift_gal, sed_ke, <br />
gr_cosmos, sed_cosmos, ext_curve_cosmos, ebv_cosmos, is_within_cosmos, cosmos_distance, abs_mag_uv_dereddened, log_ml_r01, <br />
log_stellar_mass, metallicity, log_sfr, logf_halpha_ext, logf_halpha, halpha_scatter, loglum_halpha, loglum_halpha_ext, logf_dummy, <br />
z_dummy, logf_halpha_model1_ext, logf_halpha_model1, loglum_halpha_model1_ext, loglum_halpha_model1, logf_halpha_model3_ext, <br />
logf_halpha_model3, loglum_halpha_model3_ext, loglum_halpha_model3, logf_hbeta_model1_ext, logf_hbeta_model1, logf_o2_model1_ext, <br />
logf_o2_model1, logf_n2_model1_ext, logf_n2_model1, logf_o3_model1_ext, logf_o3_model1, logf_s2_model1_ext, logf_s2_model1, <br />
logf_hbeta_model3_ext, logf_hbeta_model3, logf_o2_model3_ext, logf_o2_model3, logf_n2_model3_ext, logf_n2_model3, logf_o3_model3_ext, <br />
logf_o3_model3, logf_s2_model3_ext, logf_s2_model3, blanco_decam_g, blanco_decam_i, blanco_decam_r, blanco_decam_z, gaia_bp, gaia_g, <br />
gaia_rp, subaru_hsc_z, jst_jpcam_g, kids_g, kids_i, kids_r, kids_u, lsst_g, lsst_i, lsst_r, lsst_u, lsst_y, lsst_z, cfht_megacam_r, <br />
cfht_megacam_u, pan_starrs_i, pan_starrs_z, euclid_nisp_h, euclid_nisp_j, euclid_nisp_y, euclid_vis, 2mass_h, 2mass_j, 2mass_ks, <br />
sdss_r01, l_gal, b_gal, mw_extinction, 2mass_h_odonnell_ext, 2mass_j_odonnell_ext, 2mass_ks_odonnell_ext, blanco_decam_g_odonnell_ext, <br />
blanco_decam_i_odonnell_ext, blanco_decam_r_odonnell_ext, blanco_decam_z_odonnell_ext, cfht_megacam_r_odonnell_ext, <br />
cfht_megacam_u_odonnell_ext, euclid_nisp_h_odonnell_ext, euclid_nisp_j_odonnell_ext, euclid_nisp_y_odonnell_ext, euclid_vis_odonnell_ext,<br />
gaia_bp_odonnell_ext, gaia_g_odonnell_ext, gaia_rp_odonnell_ext, jst_jpcam_g_odonnell_ext, kids_g_odonnell_ext, kids_i_odonnell_ext, <br />
kids_r_odonnell_ext, kids_u_odonnell_ext, lsst_g_odonnell_ext, lsst_i_odonnell_ext, lsst_r_odonnell_ext, lsst_u_odonnell_ext, <br />
lsst_y_odonnell_ext, lsst_z_odonnell_ext, pan_starrs_i_odonnell_ext, pan_starrs_z_odonnell_ext, sdss_r01_odonnell_ext, <br />
subaru_hsc_z_odonnell_ext, blanco_decam_g_el_model1_odonnell_ext, blanco_decam_g_el_model3_odonnell_ext, <br />
blanco_decam_i_el_model1_odonnell_ext, blanco_decam_i_el_model3_odonnell_ext, blanco_decam_r_el_model1_odonnell_ext, <br />
blanco_decam_r_el_model3_odonnell_ext, blanco_decam_z_el_model1_odonnell_ext, blanco_decam_z_el_model3_odonnell_ext, <br />
gaia_bp_el_model1_odonnell_ext, gaia_bp_el_model3_odonnell_ext, gaia_g_el_model1_odonnell_ext, gaia_g_el_model3_odonnell_ext, <br />
gaia_rp_el_model1_odonnell_ext, gaia_rp_el_model3_odonnell_ext, subaru_hsc_z_el_model1_odonnell_ext, subaru_hsc_z_el_model3_odonnell_ext, <br />
jst_jpcam_g_el_model1_odonnell_ext, jst_jpcam_g_el_model3_odonnell_ext, kids_g_el_model1_odonnell_ext, kids_g_el_model3_odonnell_ext, <br />
kids_i_el_model1_odonnell_ext, kids_i_el_model3_odonnell_ext, kids_r_el_model1_odonnell_ext, kids_r_el_model3_odonnell_ext, <br />
kids_u_el_model1_odonnell_ext, kids_u_el_model3_odonnell_ext, lsst_g_el_model1_odonnell_ext, lsst_g_el_model3_odonnell_ext, <br />
lsst_i_el_model1_odonnell_ext, lsst_i_el_model3_odonnell_ext, lsst_r_el_model1_odonnell_ext, lsst_r_el_model3_odonnell_ext, <br />
lsst_u_el_model1_odonnell_ext, lsst_u_el_model3_odonnell_ext, lsst_y_el_model1_odonnell_ext, lsst_y_el_model3_odonnell_ext, <br />
lsst_z_el_model1_odonnell_ext, lsst_z_el_model3_odonnell_ext, cfht_megacam_r_el_model1_odonnell_ext, <br />
cfht_megacam_r_el_model3_odonnell_ext, cfht_megacam_u_el_model1_odonnell_ext, cfht_megacam_u_el_model3_odonnell_ext, <br />
pan_starrs_i_el_model1_odonnell_ext, pan_starrs_i_el_model3_odonnell_ext, pan_starrs_z_el_model1_odonnell_ext, <br />
pan_starrs_z_el_model3_odonnell_ext, euclid_nisp_h_el_model1_odonnell_ext, euclid_nisp_h_el_model3_odonnell_ext, <br />
euclid_nisp_j_el_model1_odonnell_ext, euclid_nisp_j_el_model3_odonnell_ext, euclid_nisp_y_el_model1_odonnell_ext, <br />
euclid_nisp_y_el_model3_odonnell_ext, euclid_vis_el_model1_odonnell_ext, euclid_vis_el_model3_odonnell_ext, <br />
2mass_h_el_model1_odonnell_ext, 2mass_h_el_model3_odonnell_ext, 2mass_j_el_model1_odonnell_ext, 2mass_j_el_model3_odonnell_ext,<br />
2mass_ks_el_model1_odonnell_ext, 2mass_ks_el_model3_odonnell_ext, sdss_r01_el_model1_odonnell_ext, sdss_r01_el_model3_odonnell_ext, <br />
dominant_shape, bulge_angle, disk_angle, median_major_axis, scale_length, bulge_fraction, disk_scalelength, disk_nsersic, disk_r50, <br />
bulge_r50, bulge_nsersic, disk_ellipticity, bulge_ellipticity, disk_axis_ratio, bulge_axis_ratio, disk_inclination_angle,<br />
bulge_inclination_angle, random_index, halo_id, galaxy_id, step<br />
FROM cosmohub.flagship_mock_1_9_2_c<br />
<br />
CREATE TABLE tallada.flagship_mock_1_9_2_s(<br />
`kind` bigint, <br />
`luminosity_r01` double, <br />
`halo_x` double, <br />
`halo_y` double, <br />
`halo_z` double, <br />
`halo_vx` double, <br />
`halo_vy` double, <br />
`halo_vz` double, <br />
`halo_r` double, <br />
`true_redshift_halo` double, <br />
`halo_lm` double, <br />
`halo_n_sats` bigint, <br />
`n_gals` bigint, <br />
`abs_mag_r01` double, <br />
`abs_mag_r01_evolved` double, <br />
`luminosity_r01_evolved` double, <br />
`gr_restframe` double, <br />
`color_kind` int, <br />
`x_gal` double, <br />
`y_gal` double, <br />
`z_gal` double, <br />
`r_gal` double, <br />
`true_redshift_gal` double, <br />
`ra_gal` double, <br />
`dec_gal` double, <br />
`ra_mag_gal` double, <br />
`dec_mag_gal` double, <br />
`kappa` double,<br />
`gamma1` double,<br />
`gamma2` double,<br />
`hpix_29_nest` bigint, <br />
`hpix_13_nest` bigint, <br />
`vx_gal` double, <br />
`vy_gal` double, <br />
`vz_gal` double, <br />
`vrad_gal` double, <br />
`delta_r` double, <br />
`observed_redshift_gal` double, <br />
`sed_ke` double, <br />
`gr_cosmos` double, <br />
`sed_cosmos` double, <br />
`ext_curve_cosmos` bigint, <br />
`ebv_cosmos` double, <br />
`is_within_cosmos` boolean, <br />
`cosmos_distance` double, <br />
`abs_mag_uv_dereddened` double, <br />
`log_ml_r01` double, <br />
`log_stellar_mass` double, <br />
`metallicity` double, <br />
`log_sfr` double, <br />
`logf_halpha_ext` double, <br />
`logf_halpha` double, <br />
`halpha_scatter` double, <br />
`loglum_halpha` double, <br />
`loglum_halpha_ext` double, <br />
`logf_dummy` double, <br />
`z_dummy` double, <br />
`logf_halpha_model1_ext` double, <br />
`logf_halpha_model1` double, <br />
`loglum_halpha_model1_ext` double, <br />
`loglum_halpha_model1` double, <br />
`logf_halpha_model3_ext` double, <br />
`logf_halpha_model3` double, <br />
`loglum_halpha_model3_ext` double, <br />
`loglum_halpha_model3` double, <br />
`logf_hbeta_model1_ext` double, <br />
`logf_hbeta_model1` double, <br />
`logf_o2_model1_ext` double, <br />
`logf_o2_model1` double, <br />
`logf_n2_model1_ext` double, <br />
`logf_n2_model1` double, <br />
`logf_o3_model1_ext` double, <br />
`logf_o3_model1` double, <br />
`logf_s2_model1_ext` double, <br />
`logf_s2_model1` double, <br />
`logf_hbeta_model3_ext` double, <br />
`logf_hbeta_model3` double, <br />
`logf_o2_model3_ext` double, <br />
`logf_o2_model3` double, <br />
`logf_n2_model3_ext` double, <br />
`logf_n2_model3` double, <br />
`logf_o3_model3_ext` double, <br />
`logf_o3_model3` double, <br />
`logf_s2_model3_ext` double, <br />
`logf_s2_model3` double, <br />
`blanco_decam_g` double, <br />
`blanco_decam_i` double, <br />
`blanco_decam_r` double, <br />
`blanco_decam_z` double, <br />
`gaia_bp` double, <br />
`gaia_g` double, <br />
`gaia_rp` double, <br />
`subaru_hsc_z` double, <br />
`jst_jpcam_g` double, <br />
`kids_g` double, <br />
`kids_i` double, <br />
`kids_r` double, <br />
`kids_u` double, <br />
`lsst_g` double, <br />
`lsst_i` double, <br />
`lsst_r` double, <br />
`lsst_u` double, <br />
`lsst_y` double, <br />
`lsst_z` double, <br />
`cfht_megacam_r` double, <br />
`cfht_megacam_u` double, <br />
`pan_starrs_i` double, <br />
`pan_starrs_z` double, <br />
`euclid_nisp_h` double, <br />
`euclid_nisp_j` double, <br />
`euclid_nisp_y` double, <br />
`euclid_vis` double, <br />
`2mass_h` double, <br />
`2mass_j` double, <br />
`2mass_ks` double, <br />
`sdss_r01` double, <br />
`l_gal` double, <br />
`b_gal` double, <br />
`mw_extinction` double, <br />
`2mass_h_odonnell_ext` double, <br />
`2mass_j_odonnell_ext` double, <br />
`2mass_ks_odonnell_ext` double, <br />
`blanco_decam_g_odonnell_ext` double, <br />
`blanco_decam_i_odonnell_ext` double, <br />
`blanco_decam_r_odonnell_ext` double, <br />
`blanco_decam_z_odonnell_ext` double, <br />
`cfht_megacam_r_odonnell_ext` double, <br />
`cfht_megacam_u_odonnell_ext` double, <br />
`euclid_nisp_h_odonnell_ext` double, <br />
`euclid_nisp_j_odonnell_ext` double, <br />
`euclid_nisp_y_odonnell_ext` double, <br />
`euclid_vis_odonnell_ext` double, <br />
`gaia_bp_odonnell_ext` double, <br />
`gaia_g_odonnell_ext` double, <br />
`gaia_rp_odonnell_ext` double, <br />
`jst_jpcam_g_odonnell_ext` double, <br />
`kids_g_odonnell_ext` double, <br />
`kids_i_odonnell_ext` double, <br />
`kids_r_odonnell_ext` double, <br />
`kids_u_odonnell_ext` double, <br />
`lsst_g_odonnell_ext` double, <br />
`lsst_i_odonnell_ext` double, <br />
`lsst_r_odonnell_ext` double, <br />
`lsst_u_odonnell_ext` double, <br />
`lsst_y_odonnell_ext` double, <br />
`lsst_z_odonnell_ext` double, <br />
`pan_starrs_i_odonnell_ext` double, <br />
`pan_starrs_z_odonnell_ext` double, <br />
`sdss_r01_odonnell_ext` double, <br />
`subaru_hsc_z_odonnell_ext` double, <br />
`blanco_decam_g_el_model1_odonnell_ext` double, <br />
`blanco_decam_g_el_model3_odonnell_ext` double, <br />
`blanco_decam_i_el_model1_odonnell_ext` double, <br />
`blanco_decam_i_el_model3_odonnell_ext` double, <br />
`blanco_decam_r_el_model1_odonnell_ext` double, <br />
`blanco_decam_r_el_model3_odonnell_ext` double, <br />
`blanco_decam_z_el_model1_odonnell_ext` double, <br />
`blanco_decam_z_el_model3_odonnell_ext` double, <br />
`gaia_bp_el_model1_odonnell_ext` double, <br />
`gaia_bp_el_model3_odonnell_ext` double, <br />
`gaia_g_el_model1_odonnell_ext` double, <br />
`gaia_g_el_model3_odonnell_ext` double, <br />
`gaia_rp_el_model1_odonnell_ext` double, <br />
`gaia_rp_el_model3_odonnell_ext` double, <br />
`subaru_hsc_z_el_model1_odonnell_ext` double, <br />
`subaru_hsc_z_el_model3_odonnell_ext` double, <br />
`jst_jpcam_g_el_model1_odonnell_ext` double, <br />
`jst_jpcam_g_el_model3_odonnell_ext` double, <br />
`kids_g_el_model1_odonnell_ext` double, <br />
`kids_g_el_model3_odonnell_ext` double, <br />
`kids_i_el_model1_odonnell_ext` double, <br />
`kids_i_el_model3_odonnell_ext` double, <br />
`kids_r_el_model1_odonnell_ext` double, <br />
`kids_r_el_model3_odonnell_ext` double, <br />
`kids_u_el_model1_odonnell_ext` double, <br />
`kids_u_el_model3_odonnell_ext` double, <br />
`lsst_g_el_model1_odonnell_ext` double, <br />
`lsst_g_el_model3_odonnell_ext` double, <br />
`lsst_i_el_model1_odonnell_ext` double, <br />
`lsst_i_el_model3_odonnell_ext` double, <br />
`lsst_r_el_model1_odonnell_ext` double, <br />
`lsst_r_el_model3_odonnell_ext` double, <br />
`lsst_u_el_model1_odonnell_ext` double, <br />
`lsst_u_el_model3_odonnell_ext` double, <br />
`lsst_y_el_model1_odonnell_ext` double, <br />
`lsst_y_el_model3_odonnell_ext` double, <br />
`lsst_z_el_model1_odonnell_ext` double, <br />
`lsst_z_el_model3_odonnell_ext` double, <br />
`cfht_megacam_r_el_model1_odonnell_ext` double, <br />
`cfht_megacam_r_el_model3_odonnell_ext` double, <br />
`cfht_megacam_u_el_model1_odonnell_ext` double, <br />
`cfht_megacam_u_el_model3_odonnell_ext` double, <br />
`pan_starrs_i_el_model1_odonnell_ext` double, <br />
`pan_starrs_i_el_model3_odonnell_ext` double, <br />
`pan_starrs_z_el_model1_odonnell_ext` double, <br />
`pan_starrs_z_el_model3_odonnell_ext` double, <br />
`euclid_nisp_h_el_model1_odonnell_ext` double, <br />
`euclid_nisp_h_el_model3_odonnell_ext` double, <br />
`euclid_nisp_j_el_model1_odonnell_ext` double, <br />
`euclid_nisp_j_el_model3_odonnell_ext` double, <br />
`euclid_nisp_y_el_model1_odonnell_ext` double, <br />
`euclid_nisp_y_el_model3_odonnell_ext` double, <br />
`euclid_vis_el_model1_odonnell_ext` double, <br />
`euclid_vis_el_model3_odonnell_ext` double, <br />
`2mass_h_el_model1_odonnell_ext` double, <br />
`2mass_h_el_model3_odonnell_ext` double, <br />
`2mass_j_el_model1_odonnell_ext` double, <br />
`2mass_j_el_model3_odonnell_ext` double, <br />
`2mass_ks_el_model1_odonnell_ext` double, <br />
`2mass_ks_el_model3_odonnell_ext` double, <br />
`sdss_r01_el_model1_odonnell_ext` double, <br />
`sdss_r01_el_model3_odonnell_ext` double, <br />
`dominant_shape` bigint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)', <br />
`bulge_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`disk_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`median_major_axis` double COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)', <br />
`scale_length` double COMMENT 'disc and bulge scalelength prior', <br />
`bulge_fraction` double COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)', <br />
`disk_scalelength` double COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)', <br />
`disk_nsersic` double COMMENT 'Sersic index of the disk component', <br />
`disk_r50` double COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678', <br />
`bulge_r50` double COMMENT 'half light radius of the bulge', <br />
`bulge_nsersic` double COMMENT 'Sersic index of the bulge component', <br />
`disk_ellipticity` double COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)', <br />
`bulge_ellipticity` double COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)', <br />
`disk_axis_ratio` double COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`bulge_axis_ratio` double COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`disk_inclination_angle` double COMMENT '', <br />
`bulge_inclination_angle` double COMMENT '', <br />
`random_index` double, <br />
`halo_id` bigint, <br />
`galaxy_id` bigint<br />
)<br />
PARTITIONED BY (<br />
`step` smallint<br />
)<br />
CLUSTERED BY (<br />
hpix_13_nest<br />
)<br />
SORTED BY (<br />
hpix_13_nest ASC<br />
)<br />
INTO 128 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
cat << EOF > lensing_step.sql<br />
SET hive.auto.convert.join.noconditionaltask.size=1;<br />
INSERT OVERWRITE TABLE tallada.flagship_mock_1_9_2_s PARTITION (step=${STEP})<br />
SELECT<br />
m.kind, m.luminosity_r01, m.halo_x, m.halo_y, m.halo_z, m.halo_vx, m.halo_vy, m.halo_vz, m.halo_r, m.true_redshift_halo, m.halo_lm,<br />
m.halo_n_sats, m.n_gals, m.abs_mag_r01, m.abs_mag_r01_evolved, m.luminosity_r01_evolved, m.gr_restframe, m.color_kind, m.x_gal,<br />
m.y_gal, m.z_gal, m.r_gal, m.true_redshift_gal, m.ra_gal, m.dec_gal, magpos.ra_mag, magpos.dec_mag, m.kappa, m.gamma1, m.gamma2,<br />
m.hpix_29_nest, m.hpix_13_nest, m.vx_gal, m.vy_gal, m.vz_gal, m.vrad_gal, m.delta_r, m.observed_redshift_gal, m.sed_ke, m.gr_cosmos, <br />
m.sed_cosmos, m.ext_curve_cosmos, m.ebv_cosmos, m.is_within_cosmos, m.cosmos_distance, m.abs_mag_uv_dereddened, m.log_ml_r01,<br />
m.log_stellar_mass, m.metallicity, m.log_sfr, m.logf_halpha_ext, m.logf_halpha, m.halpha_scatter, m.loglum_halpha, m.loglum_halpha_ext, <br />
m.logf_dummy, m.z_dummy, m.logf_halpha_model1_ext, m.logf_halpha_model1, m.loglum_halpha_model1_ext, m.loglum_halpha_model1, <br />
m.logf_halpha_model3_ext, m.logf_halpha_model3, m.loglum_halpha_model3_ext, m.loglum_halpha_model3, m.logf_hbeta_model1_ext, <br />
m.logf_hbeta_model1, m.logf_o2_model1_ext, m.logf_o2_model1, m.logf_n2_model1_ext, m.logf_n2_model1, m.logf_o3_model1_ext, <br />
m.logf_o3_model1, m.logf_s2_model1_ext, m.logf_s2_model1, m.logf_hbeta_model3_ext, m.logf_hbeta_model3, m.logf_o2_model3_ext, <br />
m.logf_o2_model3, m.logf_n2_model3_ext, m.logf_n2_model3, m.logf_o3_model3_ext, m.logf_o3_model3, m.logf_s2_model3_ext,<br />
m.logf_s2_model3, m.blanco_decam_g, m.blanco_decam_i, m.blanco_decam_r, m.blanco_decam_z, m.gaia_bp, m.gaia_g, m.gaia_rp, <br />
m.subaru_hsc_z, m.jst_jpcam_g, m.kids_g, m.kids_i, m.kids_r, m.kids_u, m.lsst_g, m.lsst_i, m.lsst_r, m.lsst_u, m.lsst_y, m.lsst_z, <br />
m.cfht_megacam_r, m.cfht_megacam_u, m.pan_starrs_i, m.pan_starrs_z, m.euclid_nisp_h, m.euclid_nisp_j, m.euclid_nisp_y, m.euclid_vis, <br />
m.2mass_h, m.2mass_j, m.2mass_ks, m.sdss_r01, m.l_gal, m.b_gal,mw_extinction, m.2mass_h_odonnell_ext, m.2mass_j_odonnell_ext, <br />
m.2mass_ks_odonnell_ext, m.blanco_decam_g_odonnell_ext, m.blanco_decam_i_odonnell_ext, m.blanco_decam_r_odonnell_ext, <br />
m.blanco_decam_z_odonnell_ext, m.cfht_megacam_r_odonnell_ext, m.cfht_megacam_u_odonnell_ext, m.euclid_nisp_h_odonnell_ext, <br />
m.euclid_nisp_j_odonnell_ext, m.euclid_nisp_y_odonnell_ext, m.euclid_vis_odonnell_ext, m.gaia_bp_odonnell_ext, m.gaia_g_odonnell_ext, <br />
m.gaia_rp_odonnell_ext, m.jst_jpcam_g_odonnell_ext, m.kids_g_odonnell_ext, m.kids_i_odonnell_ext, m.kids_r_odonnell_ext, <br />
m.kids_u_odonnell_ext, m.lsst_g_odonnell_ext, m.lsst_i_odonnell_ext, m.lsst_r_odonnell_ext, m.lsst_u_odonnell_ext, <br />
m.lsst_y_odonnell_ext, m.lsst_z_odonnell_ext, m.pan_starrs_i_odonnell_ext, m.pan_starrs_z_odonnell_ext, m.sdss_r01_odonnell_ext, <br />
m.subaru_hsc_z_odonnell_ext, m.blanco_decam_g_el_model1_odonnell_ext, m.blanco_decam_g_el_model3_odonnell_ext, <br />
m.blanco_decam_i_el_model1_odonnell_ext, m.blanco_decam_i_el_model3_odonnell_ext, m.blanco_decam_r_el_model1_odonnell_ext, <br />
m.blanco_decam_r_el_model3_odonnell_ext, m.blanco_decam_z_el_model1_odonnell_ext, m.blanco_decam_z_el_model3_odonnell_ext,<br />
m.gaia_bp_el_model1_odonnell_ext, m.gaia_bp_el_model3_odonnell_ext, m.gaia_g_el_model1_odonnell_ext, m.gaia_g_el_model3_odonnell_ext, <br />
m.gaia_rp_el_model1_odonnell_ext, m.gaia_rp_el_model3_odonnell_ext, m.subaru_hsc_z_el_model1_odonnell_ext, <br />
m.subaru_hsc_z_el_model3_odonnell_ext, m.jst_jpcam_g_el_model1_odonnell_ext, m.jst_jpcam_g_el_model3_odonnell_ext,<br />
m.kids_g_el_model1_odonnell_ext, m.kids_g_el_model3_odonnell_ext, m.kids_i_el_model1_odonnell_ext, m.kids_i_el_model3_odonnell_ext, <br />
m.kids_r_el_model1_odonnell_ext, m.kids_r_el_model3_odonnell_ext, m.kids_u_el_model1_odonnell_ext, m.kids_u_el_model3_odonnell_ext, <br />
m.lsst_g_el_model1_odonnell_ext, m.lsst_g_el_model3_odonnell_ext, m.lsst_i_el_model1_odonnell_ext, m.lsst_i_el_model3_odonnell_ext,<br />
m.lsst_r_el_model1_odonnell_ext, m.lsst_r_el_model3_odonnell_ext, m.lsst_u_el_model1_odonnell_ext, m.lsst_u_el_model3_odonnell_ext, <br />
m.lsst_y_el_model1_odonnell_ext, m.lsst_y_el_model3_odonnell_ext, m.lsst_z_el_model1_odonnell_ext, m.lsst_z_el_model3_odonnell_ext, <br />
m.cfht_megacam_r_el_model1_odonnell_ext, m.cfht_megacam_r_el_model3_odonnell_ext, m.cfht_megacam_u_el_model1_odonnell_ext,<br />
m.cfht_megacam_u_el_model3_odonnell_ext, m.pan_starrs_i_el_model1_odonnell_ext, m.pan_starrs_i_el_model3_odonnell_ext, <br />
m.pan_starrs_z_el_model1_odonnell_ext, m.pan_starrs_z_el_model3_odonnell_ext, m.euclid_nisp_h_el_model1_odonnell_ext, <br />
m.euclid_nisp_h_el_model3_odonnell_ext, m.euclid_nisp_j_el_model1_odonnell_ext, m.euclid_nisp_j_el_model3_odonnell_ext, <br />
m.euclid_nisp_y_el_model1_odonnell_ext, m.euclid_nisp_y_el_model3_odonnell_ext, m.euclid_vis_el_model1_odonnell_ext, <br />
m.euclid_vis_el_model3_odonnell_ext, m.2mass_h_el_model1_odonnell_ext, m.2mass_h_el_model3_odonnell_ext, <br />
m.2mass_j_el_model1_odonnell_ext, m.2mass_j_el_model3_odonnell_ext, m.2mass_ks_el_model1_odonnell_ext, <br />
m.2mass_ks_el_model3_odonnell_ext, m.sdss_r01_el_model1_odonnell_ext, m.sdss_r01_el_model3_odonnell_ext, m.dominant_shape, <br />
m.bulge_angle, m.disk_angle, m.median_major_axis, m.scale_length, m.bulge_fraction, m.disk_scalelength, m.disk_nsersic, m.disk_r50,<br />
m.bulge_r50, m.bulge_nsersic, m.disk_ellipticity, m.bulge_ellipticity, m.disk_axis_ratio, m.bulge_axis_ratio, m.disk_inclination_angle, <br />
m.bulge_inclination_angle, m.random_index, m.halo_id, m.galaxy_id<br />
FROM (<br />
SELECT <br />
m.kind, m.luminosity_r01, m.halo_x, m.halo_y, m.halo_z, m.halo_vx, m.halo_vy, m.halo_vz, m.halo_r, m.true_redshift_halo, m.halo_lm, <br />
m.halo_n_sats, m.n_gals, m.abs_mag_r01, m.abs_mag_r01_evolved, m.luminosity_r01_evolved, m.gr_restframe, m.color_kind, m.x_gal, <br />
m.y_gal, m.z_gal, m.r_gal, m.true_redshift_gal, m.ra_gal, m.dec_gal, l.defl_0 as kappa, l.shear_1 as gamma1, l.shear_2 as gamma2,<br />
m.hpix_29_nest, m.hpix_13_nest, m.vx_gal, m.vy_gal, m.vz_gal, m.vrad_gal, m.delta_r, m.observed_redshift_gal, m.sed_ke, m.gr_cosmos, <br />
m.sed_cosmos, m.ext_curve_cosmos, m.ebv_cosmos, m.is_within_cosmos, m.cosmos_distance, m.abs_mag_uv_dereddened, m.log_ml_r01, <br />
m.log_stellar_mass, m.metallicity, m.log_sfr, m.logf_halpha_ext, m.logf_halpha, m.halpha_scatter, m.loglum_halpha, <br />
m.loglum_halpha_ext, m.logf_dummy, m.z_dummy, m.logf_halpha_model1_ext, m.logf_halpha_model1, m.loglum_halpha_model1_ext, <br />
m.loglum_halpha_model1, m.logf_halpha_model3_ext, m.logf_halpha_model3, m.loglum_halpha_model3_ext, m.loglum_halpha_model3, <br />
m.logf_hbeta_model1_ext, m.logf_hbeta_model1, m.logf_o2_model1_ext, m.logf_o2_model1, m.logf_n2_model1_ext, m.logf_n2_model1,<br />
m.logf_o3_model1_ext, m.logf_o3_model1, m.logf_s2_model1_ext, m.logf_s2_model1, m.logf_hbeta_model3_ext, m.logf_hbeta_model3, <br />
m.logf_o2_model3_ext, m.logf_o2_model3, m.logf_n2_model3_ext, m.logf_n2_model3, m.logf_o3_model3_ext, m.logf_o3_model3, <br />
m.logf_s2_model3_ext, m.logf_s2_model3, m.blanco_decam_g, m.blanco_decam_i, m.blanco_decam_r, m.blanco_decam_z, m.gaia_bp, m.gaia_g, <br />
m.gaia_rp, m.subaru_hsc_z, m.jst_jpcam_g, m.kids_g, m.kids_i, m.kids_r, m.kids_u, m.lsst_g, m.lsst_i, m.lsst_r, m.lsst_u, m.lsst_y, <br />
m.lsst_z, m.cfht_megacam_r, m.cfht_megacam_u, m.pan_starrs_i, m.pan_starrs_z, m.euclid_nisp_h, m.euclid_nisp_j, m.euclid_nisp_y, <br />
m.euclid_vis, m.2mass_h, m.2mass_j, m.2mass_ks, m.sdss_r01, m.l_gal, m.b_gal,mw_extinction, m.2mass_h_odonnell_ext, <br />
m.2mass_j_odonnell_ext, m.2mass_ks_odonnell_ext, m.blanco_decam_g_odonnell_ext, m.blanco_decam_i_odonnell_ext, <br />
m.blanco_decam_r_odonnell_ext, m.blanco_decam_z_odonnell_ext, m.cfht_megacam_r_odonnell_ext, m.cfht_megacam_u_odonnell_ext,<br />
m.euclid_nisp_h_odonnell_ext, m.euclid_nisp_j_odonnell_ext, m.euclid_nisp_y_odonnell_ext, m.euclid_vis_odonnell_ext, <br />
m.gaia_bp_odonnell_ext, m.gaia_g_odonnell_ext, m.gaia_rp_odonnell_ext, m.jst_jpcam_g_odonnell_ext, m.kids_g_odonnell_ext, <br />
m.kids_i_odonnell_ext, m.kids_r_odonnell_ext, m.kids_u_odonnell_ext, m.lsst_g_odonnell_ext, m.lsst_i_odonnell_ext, <br />
m.lsst_r_odonnell_ext, m.lsst_u_odonnell_ext, m.lsst_y_odonnell_ext, m.lsst_z_odonnell_ext, m.pan_starrs_i_odonnell_ext, <br />
m.pan_starrs_z_odonnell_ext, m.sdss_r01_odonnell_ext, m.subaru_hsc_z_odonnell_ext, m.blanco_decam_g_el_model1_odonnell_ext, <br />
m.blanco_decam_g_el_model3_odonnell_ext, m.blanco_decam_i_el_model1_odonnell_ext, m.blanco_decam_i_el_model3_odonnell_ext, <br />
m.blanco_decam_r_el_model1_odonnell_ext, m.blanco_decam_r_el_model3_odonnell_ext, m.blanco_decam_z_el_model1_odonnell_ext, <br />
m.blanco_decam_z_el_model3_odonnell_ext, m.gaia_bp_el_model1_odonnell_ext, m.gaia_bp_el_model3_odonnell_ext,<br />
m.gaia_g_el_model1_odonnell_ext, m.gaia_g_el_model3_odonnell_ext, m.gaia_rp_el_model1_odonnell_ext, m.gaia_rp_el_model3_odonnell_ext, <br />
m.subaru_hsc_z_el_model1_odonnell_ext, m.subaru_hsc_z_el_model3_odonnell_ext, m.jst_jpcam_g_el_model1_odonnell_ext, <br />
m.jst_jpcam_g_el_model3_odonnell_ext, m.kids_g_el_model1_odonnell_ext, m.kids_g_el_model3_odonnell_ext,<br />
m.kids_i_el_model1_odonnell_ext, m.kids_i_el_model3_odonnell_ext, m.kids_r_el_model1_odonnell_ext, m.kids_r_el_model3_odonnell_ext, <br />
m.kids_u_el_model1_odonnell_ext, m.kids_u_el_model3_odonnell_ext, m.lsst_g_el_model1_odonnell_ext, m.lsst_g_el_model3_odonnell_ext, <br />
m.lsst_i_el_model1_odonnell_ext, m.lsst_i_el_model3_odonnell_ext, m.lsst_r_el_model1_odonnell_ext, m.lsst_r_el_model3_odonnell_ext,<br />
m.lsst_u_el_model1_odonnell_ext, m.lsst_u_el_model3_odonnell_ext, m.lsst_y_el_model1_odonnell_ext, m.lsst_y_el_model3_odonnell_ext, <br />
m.lsst_z_el_model1_odonnell_ext, m.lsst_z_el_model3_odonnell_ext, m.cfht_megacam_r_el_model1_odonnell_ext, <br />
m.cfht_megacam_r_el_model3_odonnell_ext, m.cfht_megacam_u_el_model1_odonnell_ext, m.cfht_megacam_u_el_model3_odonnell_ext, <br />
m.pan_starrs_i_el_model1_odonnell_ext, m.pan_starrs_i_el_model3_odonnell_ext, m.pan_starrs_z_el_model1_odonnell_ext, <br />
m.pan_starrs_z_el_model3_odonnell_ext, m.euclid_nisp_h_el_model1_odonnell_ext, m.euclid_nisp_h_el_model3_odonnell_ext,<br />
m.euclid_nisp_j_el_model1_odonnell_ext, m.euclid_nisp_j_el_model3_odonnell_ext, m.euclid_nisp_y_el_model1_odonnell_ext, <br />
m.euclid_nisp_y_el_model3_odonnell_ext, m.euclid_vis_el_model1_odonnell_ext, m.euclid_vis_el_model3_odonnell_ext, <br />
m.2mass_h_el_model1_odonnell_ext, m.2mass_h_el_model3_odonnell_ext, m.2mass_j_el_model1_odonnell_ext, <br />
m.2mass_j_el_model3_odonnell_ext, m.2mass_ks_el_model1_odonnell_ext, m.2mass_ks_el_model3_odonnell_ext, <br />
m.sdss_r01_el_model1_odonnell_ext, m.sdss_r01_el_model3_odonnell_ext, m.dominant_shape, m.bulge_angle, m.disk_angle, <br />
m.median_major_axis, m.scale_length, m.bulge_fraction, m.disk_scalelength, m.disk_nsersic, m.disk_r50, m.bulge_r50, m.bulge_nsersic, <br />
m.disk_ellipticity, m.bulge_ellipticity, m.disk_axis_ratio, m.bulge_axis_ratio, m.disk_inclination_angle, m.bulge_inclination_angle, <br />
m.random_index, m.halo_id, m.galaxy_id,<br />
udf.magnified_positions(CAST(m.ra_gal AS FLOAT), CAST(m.dec_gal AS FLOAT), l.defl_1, l.defl_2) AS magpos<br />
FROM tallada.flagship_mock_1_9_2_c2 AS m<br />
JOIN cosmohub.flagship_lensing AS l<br />
ON udf.ang2pix(13, m.ra_gal, m.dec_gal, True, True) = l.hpix<br />
AND m.step = l.step<br />
AND l.step=${STEP} AND m.step=${STEP}<br />
) AS m<br />
;<br />
<br />
for STEP in $(seq 344 400);<br />
do<br />
echo $STEP<br />
sem -j4 "beeline --color=true --showHeader=true --verbose=true --silent=false -u \<br />
<br />
<br />
'jdbc:hive2://ambarisrv02.pic.es:2181,ambarisrv03.pic.es:2181,ambarisrv01.pic.es:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2' \<br />
-f lensing_step.sql --hivevar STEP=${STEP}"<br />
done<br />
sem --wait</div>
Tallada
https://pwiki.pic.es/index.php?title=Euclid_mock_production_v1.9.2&diff=820
Euclid mock production v1.9.2
2020-07-14T08:02:39Z
<p>Tallada: </p>
<hr />
<div> CREATE EXTERNAL TABLE `flagship_1_9_2`(<br />
`kind` bigint, <br />
`luminosity_r01` double, <br />
`halo_x` double, <br />
`halo_y` double, <br />
`halo_z` double, <br />
`halo_vx` double, <br />
`halo_vy` double, <br />
`halo_vz` double, <br />
`halo_r` double, <br />
`true_redshift_halo` double, <br />
`halo_lm` double, <br />
`halo_n_sats` bigint, <br />
`n_gals` bigint, <br />
`abs_mag_r01` double, <br />
`abs_mag_r01_evolved` double, <br />
`luminosity_r01_evolved` double, <br />
`gr_restframe` double, <br />
`color_kind` int, <br />
`x_gal` double, <br />
`y_gal` double, <br />
`z_gal` double, <br />
`r_gal` double, <br />
`true_redshift_gal` double, <br />
`ra_gal` double, <br />
`dec_gal` double, <br />
`hpix_nest` bigint, <br />
`vx_gal` double, <br />
`vy_gal` double, <br />
`vz_gal` double, <br />
`vrad_gal` double, <br />
`delta_r` double, <br />
`observed_redshift_gal` double, <br />
`sed_ke` double, <br />
`gr_cosmos` double, <br />
`sed_cosmos` double, <br />
`ext_curve_cosmos` bigint, <br />
`ebv_cosmos` double, <br />
`is_within_cosmos` boolean, <br />
`cosmos_distance` double, <br />
`abs_mag_uv_dereddened` double, <br />
`log_ml_r01` double, <br />
`log_stellar_mass` double, <br />
`metallicity` double, <br />
`log_sfr` double, <br />
`logf_halpha_ext` double, <br />
`logf_halpha` double, <br />
`halpha_scatter` double, <br />
`loglum_halpha` double, <br />
`loglum_halpha_ext` double, <br />
`logf_dummy` double, <br />
`z_dummy` double, <br />
`logf_halpha_model1_ext` double, <br />
`logf_halpha_model1` double, <br />
`loglum_halpha_model1_ext` double, <br />
`loglum_halpha_model1` double, <br />
`logf_halpha_model3_ext` double, <br />
`logf_halpha_model3` double, <br />
`loglum_halpha_model3_ext` double, <br />
`loglum_halpha_model3` double, <br />
`logf_hbeta_model1_ext` double, <br />
`logf_hbeta_model1` double, <br />
`logf_o2_model1_ext` double, <br />
`logf_o2_model1` double, <br />
`logf_n2_model1_ext` double, <br />
`logf_n2_model1` double, <br />
`logf_o3_model1_ext` double, <br />
`logf_o3_model1` double, <br />
`logf_s2_model1_ext` double, <br />
`logf_s2_model1` double, <br />
`logf_hbeta_model3_ext` double, <br />
`logf_hbeta_model3` double, <br />
`logf_o2_model3_ext` double, <br />
`logf_o2_model3` double, <br />
`logf_n2_model3_ext` double, <br />
`logf_n2_model3` double, <br />
`logf_o3_model3_ext` double, <br />
`logf_o3_model3` double, <br />
`logf_s2_model3_ext` double, <br />
`logf_s2_model3` double, <br />
`blanco_decam_g` double, <br />
`blanco_decam_i` double, <br />
`blanco_decam_r` double, <br />
`blanco_decam_z` double, <br />
`gaia_bp` double, <br />
`gaia_g` double, <br />
`gaia_rp` double, <br />
`subaru_hsc_z` double, <br />
`jst_jpcam_g` double, <br />
`kids_g` double, <br />
`kids_i` double, <br />
`kids_r` double, <br />
`kids_u` double, <br />
`lsst_g` double, <br />
`lsst_i` double, <br />
`lsst_r` double, <br />
`lsst_u` double, <br />
`lsst_y` double, <br />
`lsst_z` double, <br />
`cfht_megacam_r` double, <br />
`cfht_megacam_u` double, <br />
`pan_starrs_i` double, <br />
`pan_starrs_z` double, <br />
`euclid_nisp_h` double, <br />
`euclid_nisp_j` double, <br />
`euclid_nisp_y` double, <br />
`euclid_vis` double, <br />
`2mass_h` double, <br />
`2mass_j` double, <br />
`2mass_ks` double, <br />
`sdss_r01` double, <br />
`l_gal` double, <br />
`b_gal` double, <br />
`mw_extinction` double, <br />
`2mass_h_odonnell_ext` double, <br />
`2mass_j_odonnell_ext` double, <br />
`2mass_ks_odonnell_ext` double, <br />
`blanco_decam_g_odonnell_ext` double, <br />
`blanco_decam_i_odonnell_ext` double, <br />
`blanco_decam_r_odonnell_ext` double, <br />
`blanco_decam_z_odonnell_ext` double, <br />
`cfht_megacam_r_odonnell_ext` double, <br />
`cfht_megacam_u_odonnell_ext` double, <br />
`euclid_nisp_h_odonnell_ext` double, <br />
`euclid_nisp_j_odonnell_ext` double, <br />
`euclid_nisp_y_odonnell_ext` double, <br />
`euclid_vis_odonnell_ext` double, <br />
`gaia_bp_odonnell_ext` double, <br />
`gaia_g_odonnell_ext` double, <br />
`gaia_rp_odonnell_ext` double, <br />
`jst_jpcam_g_odonnell_ext` double, <br />
`kids_g_odonnell_ext` double, <br />
`kids_i_odonnell_ext` double, <br />
`kids_r_odonnell_ext` double, <br />
`kids_u_odonnell_ext` double, <br />
`lsst_g_odonnell_ext` double, <br />
`lsst_i_odonnell_ext` double, <br />
`lsst_r_odonnell_ext` double, <br />
`lsst_u_odonnell_ext` double, <br />
`lsst_y_odonnell_ext` double, <br />
`lsst_z_odonnell_ext` double, <br />
`pan_starrs_i_odonnell_ext` double, <br />
`pan_starrs_z_odonnell_ext` double, <br />
`sdss_r01_odonnell_ext` double, <br />
`subaru_hsc_z_odonnell_ext` double, <br />
`blanco_decam_g_el_model1_odonnell_ext` double, <br />
`blanco_decam_g_el_model3_odonnell_ext` double, <br />
`blanco_decam_i_el_model1_odonnell_ext` double, <br />
`blanco_decam_i_el_model3_odonnell_ext` double, <br />
`blanco_decam_r_el_model1_odonnell_ext` double, <br />
`blanco_decam_r_el_model3_odonnell_ext` double, <br />
`blanco_decam_z_el_model1_odonnell_ext` double, <br />
`blanco_decam_z_el_model3_odonnell_ext` double, <br />
`gaia_bp_el_model1_odonnell_ext` double, <br />
`gaia_bp_el_model3_odonnell_ext` double, <br />
`gaia_g_el_model1_odonnell_ext` double, <br />
`gaia_g_el_model3_odonnell_ext` double, <br />
`gaia_rp_el_model1_odonnell_ext` double, <br />
`gaia_rp_el_model3_odonnell_ext` double, <br />
`subaru_hsc_z_el_model1_odonnell_ext` double, <br />
`subaru_hsc_z_el_model3_odonnell_ext` double, <br />
`jst_jpcam_g_el_model1_odonnell_ext` double, <br />
`jst_jpcam_g_el_model3_odonnell_ext` double, <br />
`kids_g_el_model1_odonnell_ext` double, <br />
`kids_g_el_model3_odonnell_ext` double, <br />
`kids_i_el_model1_odonnell_ext` double, <br />
`kids_i_el_model3_odonnell_ext` double, <br />
`kids_r_el_model1_odonnell_ext` double, <br />
`kids_r_el_model3_odonnell_ext` double, <br />
`kids_u_el_model1_odonnell_ext` double, <br />
`kids_u_el_model3_odonnell_ext` double, <br />
`lsst_g_el_model1_odonnell_ext` double, <br />
`lsst_g_el_model3_odonnell_ext` double, <br />
`lsst_i_el_model1_odonnell_ext` double, <br />
`lsst_i_el_model3_odonnell_ext` double, <br />
`lsst_r_el_model1_odonnell_ext` double, <br />
`lsst_r_el_model3_odonnell_ext` double, <br />
`lsst_u_el_model1_odonnell_ext` double, <br />
`lsst_u_el_model3_odonnell_ext` double, <br />
`lsst_y_el_model1_odonnell_ext` double, <br />
`lsst_y_el_model3_odonnell_ext` double, <br />
`lsst_z_el_model1_odonnell_ext` double, <br />
`lsst_z_el_model3_odonnell_ext` double, <br />
`cfht_megacam_r_el_model1_odonnell_ext` double, <br />
`cfht_megacam_r_el_model3_odonnell_ext` double, <br />
`cfht_megacam_u_el_model1_odonnell_ext` double, <br />
`cfht_megacam_u_el_model3_odonnell_ext` double, <br />
`pan_starrs_i_el_model1_odonnell_ext` double, <br />
`pan_starrs_i_el_model3_odonnell_ext` double, <br />
`pan_starrs_z_el_model1_odonnell_ext` double, <br />
`pan_starrs_z_el_model3_odonnell_ext` double, <br />
`euclid_nisp_h_el_model1_odonnell_ext` double, <br />
`euclid_nisp_h_el_model3_odonnell_ext` double, <br />
`euclid_nisp_j_el_model1_odonnell_ext` double, <br />
`euclid_nisp_j_el_model3_odonnell_ext` double, <br />
`euclid_nisp_y_el_model1_odonnell_ext` double, <br />
`euclid_nisp_y_el_model3_odonnell_ext` double, <br />
`euclid_vis_el_model1_odonnell_ext` double, <br />
`euclid_vis_el_model3_odonnell_ext` double, <br />
`2mass_h_el_model1_odonnell_ext` double, <br />
`2mass_h_el_model3_odonnell_ext` double, <br />
`2mass_j_el_model1_odonnell_ext` double, <br />
`2mass_j_el_model3_odonnell_ext` double, <br />
`2mass_ks_el_model1_odonnell_ext` double, <br />
`2mass_ks_el_model3_odonnell_ext` double, <br />
`sdss_r01_el_model1_odonnell_ext` double, <br />
`sdss_r01_el_model3_odonnell_ext` double, <br />
`dominant_shape` bigint, <br />
`bulge_angle` double, <br />
`disk_angle` double, <br />
`median_major_axis` double, <br />
`scale_length` double, <br />
`bulge_fraction` double, <br />
`disk_scalelength` double, <br />
`disk_nsersic` double, <br />
`disk_r50` double, <br />
`bulge_r50` double, <br />
`bulge_nsersic` double, <br />
`disk_ellipticity` double, <br />
`bulge_ellipticity` double, <br />
`disk_axis_ratio` double, <br />
`bulge_axis_ratio` double, <br />
`disk_inclination_angle` double, <br />
`bulge_inclination_angle` double, <br />
`step` bigint, <br />
`random_index` double, <br />
`halo_id` bigint, <br />
`galaxy_id` bigint)<br />
ROW FORMAT SERDE <br />
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' <br />
STORED AS INPUTFORMAT <br />
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' <br />
OUTPUTFORMAT <br />
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'<br />
;<br />
<br />
CREATE TABLE flagship_mock_1_9_2_c(<br />
`kind` bigint, <br />
`luminosity_r01` double, <br />
`halo_x` double, <br />
`halo_y` double, <br />
`halo_z` double, <br />
`halo_vx` double, <br />
`halo_vy` double, <br />
`halo_vz` double, <br />
`halo_r` double, <br />
`true_redshift_halo` double, <br />
`halo_lm` double, <br />
`halo_n_sats` bigint, <br />
`n_gals` bigint, <br />
`abs_mag_r01` double, <br />
`abs_mag_r01_evolved` double, <br />
`luminosity_r01_evolved` double, <br />
`gr_restframe` double, <br />
`color_kind` int, <br />
`x_gal` double, <br />
`y_gal` double, <br />
`z_gal` double, <br />
`r_gal` double, <br />
`true_redshift_gal` double, <br />
`ra_gal` double, <br />
`dec_gal` double, <br />
`hpix_nest` bigint, <br />
`vx_gal` double, <br />
`vy_gal` double, <br />
`vz_gal` double, <br />
`vrad_gal` double, <br />
`delta_r` double, <br />
`observed_redshift_gal` double, <br />
`sed_ke` double, <br />
`gr_cosmos` double, <br />
`sed_cosmos` double, <br />
`ext_curve_cosmos` bigint, <br />
`ebv_cosmos` double, <br />
`is_within_cosmos` boolean, <br />
`cosmos_distance` double, <br />
`abs_mag_uv_dereddened` double, <br />
`log_ml_r01` double, <br />
`log_stellar_mass` double, <br />
`metallicity` double, <br />
`log_sfr` double, <br />
`logf_halpha_ext` double, <br />
`logf_halpha` double, <br />
`halpha_scatter` double, <br />
`loglum_halpha` double, <br />
`loglum_halpha_ext` double, <br />
`logf_dummy` double, <br />
`z_dummy` double, <br />
`logf_halpha_model1_ext` double, <br />
`logf_halpha_model1` double, <br />
`loglum_halpha_model1_ext` double, <br />
`loglum_halpha_model1` double, <br />
`logf_halpha_model3_ext` double, <br />
`logf_halpha_model3` double, <br />
`loglum_halpha_model3_ext` double, <br />
`loglum_halpha_model3` double, <br />
`logf_hbeta_model1_ext` double, <br />
`logf_hbeta_model1` double, <br />
`logf_o2_model1_ext` double, <br />
`logf_o2_model1` double, <br />
`logf_n2_model1_ext` double, <br />
`logf_n2_model1` double, <br />
`logf_o3_model1_ext` double, <br />
`logf_o3_model1` double, <br />
`logf_s2_model1_ext` double, <br />
`logf_s2_model1` double, <br />
`logf_hbeta_model3_ext` double, <br />
`logf_hbeta_model3` double, <br />
`logf_o2_model3_ext` double, <br />
`logf_o2_model3` double, <br />
`logf_n2_model3_ext` double, <br />
`logf_n2_model3` double, <br />
`logf_o3_model3_ext` double, <br />
`logf_o3_model3` double, <br />
`logf_s2_model3_ext` double, <br />
`logf_s2_model3` double, <br />
`blanco_decam_g` double, <br />
`blanco_decam_i` double, <br />
`blanco_decam_r` double, <br />
`blanco_decam_z` double, <br />
`gaia_bp` double, <br />
`gaia_g` double, <br />
`gaia_rp` double, <br />
`subaru_hsc_z` double, <br />
`jst_jpcam_g` double, <br />
`kids_g` double, <br />
`kids_i` double, <br />
`kids_r` double, <br />
`kids_u` double, <br />
`lsst_g` double, <br />
`lsst_i` double, <br />
`lsst_r` double, <br />
`lsst_u` double, <br />
`lsst_y` double, <br />
`lsst_z` double, <br />
`cfht_megacam_r` double, <br />
`cfht_megacam_u` double, <br />
`pan_starrs_i` double, <br />
`pan_starrs_z` double, <br />
`euclid_nisp_h` double, <br />
`euclid_nisp_j` double, <br />
`euclid_nisp_y` double, <br />
`euclid_vis` double, <br />
`2mass_h` double, <br />
`2mass_j` double, <br />
`2mass_ks` double, <br />
`sdss_r01` double, <br />
`l_gal` double, <br />
`b_gal` double, <br />
`mw_extinction` double, <br />
`2mass_h_odonnell_ext` double, <br />
`2mass_j_odonnell_ext` double, <br />
`2mass_ks_odonnell_ext` double, <br />
`blanco_decam_g_odonnell_ext` double, <br />
`blanco_decam_i_odonnell_ext` double, <br />
`blanco_decam_r_odonnell_ext` double, <br />
`blanco_decam_z_odonnell_ext` double, <br />
`cfht_megacam_r_odonnell_ext` double, <br />
`cfht_megacam_u_odonnell_ext` double, <br />
`euclid_nisp_h_odonnell_ext` double, <br />
`euclid_nisp_j_odonnell_ext` double, <br />
`euclid_nisp_y_odonnell_ext` double, <br />
`euclid_vis_odonnell_ext` double, <br />
`gaia_bp_odonnell_ext` double, <br />
`gaia_g_odonnell_ext` double, <br />
`gaia_rp_odonnell_ext` double, <br />
`jst_jpcam_g_odonnell_ext` double, <br />
`kids_g_odonnell_ext` double, <br />
`kids_i_odonnell_ext` double, <br />
`kids_r_odonnell_ext` double, <br />
`kids_u_odonnell_ext` double, <br />
`lsst_g_odonnell_ext` double, <br />
`lsst_i_odonnell_ext` double, <br />
`lsst_r_odonnell_ext` double, <br />
`lsst_u_odonnell_ext` double, <br />
`lsst_y_odonnell_ext` double, <br />
`lsst_z_odonnell_ext` double, <br />
`pan_starrs_i_odonnell_ext` double, <br />
`pan_starrs_z_odonnell_ext` double, <br />
`sdss_r01_odonnell_ext` double, <br />
`subaru_hsc_z_odonnell_ext` double, <br />
`blanco_decam_g_el_model1_odonnell_ext` double, <br />
`blanco_decam_g_el_model3_odonnell_ext` double, <br />
`blanco_decam_i_el_model1_odonnell_ext` double, <br />
`blanco_decam_i_el_model3_odonnell_ext` double, <br />
`blanco_decam_r_el_model1_odonnell_ext` double, <br />
`blanco_decam_r_el_model3_odonnell_ext` double, <br />
`blanco_decam_z_el_model1_odonnell_ext` double, <br />
`blanco_decam_z_el_model3_odonnell_ext` double, <br />
`gaia_bp_el_model1_odonnell_ext` double, <br />
`gaia_bp_el_model3_odonnell_ext` double, <br />
`gaia_g_el_model1_odonnell_ext` double, <br />
`gaia_g_el_model3_odonnell_ext` double, <br />
`gaia_rp_el_model1_odonnell_ext` double, <br />
`gaia_rp_el_model3_odonnell_ext` double, <br />
`subaru_hsc_z_el_model1_odonnell_ext` double, <br />
`subaru_hsc_z_el_model3_odonnell_ext` double, <br />
`jst_jpcam_g_el_model1_odonnell_ext` double, <br />
`jst_jpcam_g_el_model3_odonnell_ext` double, <br />
`kids_g_el_model1_odonnell_ext` double, <br />
`kids_g_el_model3_odonnell_ext` double, <br />
`kids_i_el_model1_odonnell_ext` double, <br />
`kids_i_el_model3_odonnell_ext` double, <br />
`kids_r_el_model1_odonnell_ext` double, <br />
`kids_r_el_model3_odonnell_ext` double, <br />
`kids_u_el_model1_odonnell_ext` double, <br />
`kids_u_el_model3_odonnell_ext` double, <br />
`lsst_g_el_model1_odonnell_ext` double, <br />
`lsst_g_el_model3_odonnell_ext` double, <br />
`lsst_i_el_model1_odonnell_ext` double, <br />
`lsst_i_el_model3_odonnell_ext` double, <br />
`lsst_r_el_model1_odonnell_ext` double, <br />
`lsst_r_el_model3_odonnell_ext` double, <br />
`lsst_u_el_model1_odonnell_ext` double, <br />
`lsst_u_el_model3_odonnell_ext` double, <br />
`lsst_y_el_model1_odonnell_ext` double, <br />
`lsst_y_el_model3_odonnell_ext` double, <br />
`lsst_z_el_model1_odonnell_ext` double, <br />
`lsst_z_el_model3_odonnell_ext` double, <br />
`cfht_megacam_r_el_model1_odonnell_ext` double, <br />
`cfht_megacam_r_el_model3_odonnell_ext` double, <br />
`cfht_megacam_u_el_model1_odonnell_ext` double, <br />
`cfht_megacam_u_el_model3_odonnell_ext` double, <br />
`pan_starrs_i_el_model1_odonnell_ext` double, <br />
`pan_starrs_i_el_model3_odonnell_ext` double, <br />
`pan_starrs_z_el_model1_odonnell_ext` double, <br />
`pan_starrs_z_el_model3_odonnell_ext` double, <br />
`euclid_nisp_h_el_model1_odonnell_ext` double, <br />
`euclid_nisp_h_el_model3_odonnell_ext` double, <br />
`euclid_nisp_j_el_model1_odonnell_ext` double, <br />
`euclid_nisp_j_el_model3_odonnell_ext` double, <br />
`euclid_nisp_y_el_model1_odonnell_ext` double, <br />
`euclid_nisp_y_el_model3_odonnell_ext` double, <br />
`euclid_vis_el_model1_odonnell_ext` double, <br />
`euclid_vis_el_model3_odonnell_ext` double, <br />
`2mass_h_el_model1_odonnell_ext` double, <br />
`2mass_h_el_model3_odonnell_ext` double, <br />
`2mass_j_el_model1_odonnell_ext` double, <br />
`2mass_j_el_model3_odonnell_ext` double, <br />
`2mass_ks_el_model1_odonnell_ext` double, <br />
`2mass_ks_el_model3_odonnell_ext` double, <br />
`sdss_r01_el_model1_odonnell_ext` double, <br />
`sdss_r01_el_model3_odonnell_ext` double, <br />
`dominant_shape` bigint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)', <br />
`bulge_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`disk_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`median_major_axis` double COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)', <br />
`scale_length` double COMMENT 'disc and bulge scalelength prior', <br />
`bulge_fraction` double COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)', <br />
`disk_scalelength` double COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)', <br />
`disk_nsersic` double COMMENT 'Sersic index of the disk component', <br />
`disk_r50` double COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678', <br />
`bulge_r50` double COMMENT 'half light radius of the bulge', <br />
`bulge_nsersic` double COMMENT 'Sersic index of the bulge component', <br />
`disk_ellipticity` double COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)', <br />
`bulge_ellipticity` double COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)', <br />
`disk_axis_ratio` double COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`bulge_axis_ratio` double COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',<br />
`disk_inclination_angle` double COMMENT '', <br />
`bulge_inclination_angle` double COMMENT '', <br />
`step` bigint, <br />
`random_index` double, <br />
`halo_id` bigint, <br />
`galaxy_id` bigint<br />
)<br />
CLUSTERED BY ( <br />
halo_id, <br />
galaxy_id<br />
) <br />
INTO 128 BUCKETS<br />
ROW FORMAT SERDE <br />
'org.apache.hadoop.hive.ql.io.orc.OrcSerde' <br />
STORED AS INPUTFORMAT <br />
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' <br />
OUTPUTFORMAT <br />
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'<br />
;<br />
<br />
INSERT OVERWRITE TABLE cosmohub.flagship_mock_1_9_2_c<br />
SELECT * FROM tallada.flagship_1_9_2;<br />
<br />
ANALYZE TABLE cosmohub.flagship_mock_1_9_2_c COMPUTE STATISTICS FOR COLUMNS;<br />
<br />
CREATE TABLE tallada.flagship_mock_1_9_2_c2(<br />
`kind` bigint, <br />
`luminosity_r01` double, <br />
`halo_x` double, <br />
`halo_y` double, <br />
`halo_z` double, <br />
`halo_vx` double, <br />
`halo_vy` double, <br />
`halo_vz` double, <br />
`halo_r` double, <br />
`true_redshift_halo` double, <br />
`halo_lm` double, <br />
`halo_n_sats` bigint, <br />
`n_gals` bigint, <br />
`abs_mag_r01` double, <br />
`abs_mag_r01_evolved` double, <br />
`luminosity_r01_evolved` double, <br />
`gr_restframe` double, <br />
`color_kind` int, <br />
`x_gal` double, <br />
`y_gal` double, <br />
`z_gal` double, <br />
`r_gal` double, <br />
`true_redshift_gal` double, <br />
`ra_gal` double, <br />
`dec_gal` double, <br />
`hpix_29_nest` bigint, <br />
`hpix_13_nest` bigint, <br />
`vx_gal` double, <br />
`vy_gal` double, <br />
`vz_gal` double, <br />
`vrad_gal` double, <br />
`delta_r` double, <br />
`observed_redshift_gal` double, <br />
`sed_ke` double, <br />
`gr_cosmos` double, <br />
`sed_cosmos` double, <br />
`ext_curve_cosmos` bigint, <br />
`ebv_cosmos` double, <br />
`is_within_cosmos` boolean, <br />
`cosmos_distance` double, <br />
`abs_mag_uv_dereddened` double, <br />
`log_ml_r01` double, <br />
`log_stellar_mass` double, <br />
`metallicity` double, <br />
`log_sfr` double, <br />
`logf_halpha_ext` double, <br />
`logf_halpha` double, <br />
`halpha_scatter` double, <br />
`loglum_halpha` double, <br />
`loglum_halpha_ext` double, <br />
`logf_dummy` double, <br />
`z_dummy` double, <br />
`logf_halpha_model1_ext` double, <br />
`logf_halpha_model1` double, <br />
`loglum_halpha_model1_ext` double, <br />
`loglum_halpha_model1` double, <br />
`logf_halpha_model3_ext` double, <br />
`logf_halpha_model3` double, <br />
`loglum_halpha_model3_ext` double, <br />
`loglum_halpha_model3` double, <br />
`logf_hbeta_model1_ext` double, <br />
`logf_hbeta_model1` double, <br />
`logf_o2_model1_ext` double, <br />
`logf_o2_model1` double, <br />
`logf_n2_model1_ext` double, <br />
`logf_n2_model1` double, <br />
`logf_o3_model1_ext` double, <br />
`logf_o3_model1` double, <br />
`logf_s2_model1_ext` double, <br />
`logf_s2_model1` double, <br />
`logf_hbeta_model3_ext` double, <br />
`logf_hbeta_model3` double, <br />
`logf_o2_model3_ext` double, <br />
`logf_o2_model3` double, <br />
`logf_n2_model3_ext` double, <br />
`logf_n2_model3` double, <br />
`logf_o3_model3_ext` double, <br />
`logf_o3_model3` double, <br />
`logf_s2_model3_ext` double, <br />
`logf_s2_model3` double, <br />
`blanco_decam_g` double, <br />
`blanco_decam_i` double, <br />
`blanco_decam_r` double, <br />
`blanco_decam_z` double, <br />
`gaia_bp` double, <br />
`gaia_g` double, <br />
`gaia_rp` double, <br />
`subaru_hsc_z` double, <br />
`jst_jpcam_g` double, <br />
`kids_g` double, <br />
`kids_i` double, <br />
`kids_r` double, <br />
`kids_u` double, <br />
`lsst_g` double, <br />
`lsst_i` double, <br />
`lsst_r` double, <br />
`lsst_u` double, <br />
`lsst_y` double, <br />
`lsst_z` double, <br />
`cfht_megacam_r` double, <br />
`cfht_megacam_u` double, <br />
`pan_starrs_i` double, <br />
`pan_starrs_z` double, <br />
`euclid_nisp_h` double, <br />
`euclid_nisp_j` double, <br />
`euclid_nisp_y` double, <br />
`euclid_vis` double, <br />
`2mass_h` double, <br />
`2mass_j` double, <br />
`2mass_ks` double, <br />
`sdss_r01` double, <br />
`l_gal` double, <br />
`b_gal` double, <br />
`mw_extinction` double, <br />
`2mass_h_odonnell_ext` double, <br />
`2mass_j_odonnell_ext` double, <br />
`2mass_ks_odonnell_ext` double, <br />
`blanco_decam_g_odonnell_ext` double, <br />
`blanco_decam_i_odonnell_ext` double, <br />
`blanco_decam_r_odonnell_ext` double, <br />
`blanco_decam_z_odonnell_ext` double, <br />
`cfht_megacam_r_odonnell_ext` double, <br />
`cfht_megacam_u_odonnell_ext` double, <br />
`euclid_nisp_h_odonnell_ext` double, <br />
`euclid_nisp_j_odonnell_ext` double, <br />
`euclid_nisp_y_odonnell_ext` double, <br />
`euclid_vis_odonnell_ext` double, <br />
`gaia_bp_odonnell_ext` double, <br />
`gaia_g_odonnell_ext` double, <br />
`gaia_rp_odonnell_ext` double, <br />
`jst_jpcam_g_odonnell_ext` double, <br />
`kids_g_odonnell_ext` double, <br />
`kids_i_odonnell_ext` double, <br />
`kids_r_odonnell_ext` double, <br />
`kids_u_odonnell_ext` double, <br />
`lsst_g_odonnell_ext` double, <br />
`lsst_i_odonnell_ext` double, <br />
`lsst_r_odonnell_ext` double, <br />
`lsst_u_odonnell_ext` double, <br />
`lsst_y_odonnell_ext` double, <br />
`lsst_z_odonnell_ext` double, <br />
`pan_starrs_i_odonnell_ext` double, <br />
`pan_starrs_z_odonnell_ext` double, <br />
`sdss_r01_odonnell_ext` double, <br />
`subaru_hsc_z_odonnell_ext` double, <br />
`blanco_decam_g_el_model1_odonnell_ext` double, <br />
`blanco_decam_g_el_model3_odonnell_ext` double, <br />
`blanco_decam_i_el_model1_odonnell_ext` double, <br />
`blanco_decam_i_el_model3_odonnell_ext` double, <br />
`blanco_decam_r_el_model1_odonnell_ext` double, <br />
`blanco_decam_r_el_model3_odonnell_ext` double, <br />
`blanco_decam_z_el_model1_odonnell_ext` double, <br />
`blanco_decam_z_el_model3_odonnell_ext` double, <br />
`gaia_bp_el_model1_odonnell_ext` double, <br />
`gaia_bp_el_model3_odonnell_ext` double, <br />
`gaia_g_el_model1_odonnell_ext` double, <br />
`gaia_g_el_model3_odonnell_ext` double, <br />
`gaia_rp_el_model1_odonnell_ext` double, <br />
`gaia_rp_el_model3_odonnell_ext` double, <br />
`subaru_hsc_z_el_model1_odonnell_ext` double, <br />
`subaru_hsc_z_el_model3_odonnell_ext` double, <br />
`jst_jpcam_g_el_model1_odonnell_ext` double, <br />
`jst_jpcam_g_el_model3_odonnell_ext` double, <br />
`kids_g_el_model1_odonnell_ext` double, <br />
`kids_g_el_model3_odonnell_ext` double, <br />
`kids_i_el_model1_odonnell_ext` double, <br />
`kids_i_el_model3_odonnell_ext` double, <br />
`kids_r_el_model1_odonnell_ext` double, <br />
`kids_r_el_model3_odonnell_ext` double, <br />
`kids_u_el_model1_odonnell_ext` double, <br />
`kids_u_el_model3_odonnell_ext` double, <br />
`lsst_g_el_model1_odonnell_ext` double, <br />
`lsst_g_el_model3_odonnell_ext` double, <br />
`lsst_i_el_model1_odonnell_ext` double, <br />
`lsst_i_el_model3_odonnell_ext` double, <br />
`lsst_r_el_model1_odonnell_ext` double, <br />
`lsst_r_el_model3_odonnell_ext` double, <br />
`lsst_u_el_model1_odonnell_ext` double, <br />
`lsst_u_el_model3_odonnell_ext` double, <br />
`lsst_y_el_model1_odonnell_ext` double, <br />
`lsst_y_el_model3_odonnell_ext` double, <br />
`lsst_z_el_model1_odonnell_ext` double, <br />
`lsst_z_el_model3_odonnell_ext` double, <br />
`cfht_megacam_r_el_model1_odonnell_ext` double, <br />
`cfht_megacam_r_el_model3_odonnell_ext` double, <br />
`cfht_megacam_u_el_model1_odonnell_ext` double, <br />
`cfht_megacam_u_el_model3_odonnell_ext` double, <br />
`pan_starrs_i_el_model1_odonnell_ext` double, <br />
`pan_starrs_i_el_model3_odonnell_ext` double, <br />
`pan_starrs_z_el_model1_odonnell_ext` double, <br />
`pan_starrs_z_el_model3_odonnell_ext` double, <br />
`euclid_nisp_h_el_model1_odonnell_ext` double, <br />
`euclid_nisp_h_el_model3_odonnell_ext` double, <br />
`euclid_nisp_j_el_model1_odonnell_ext` double, <br />
`euclid_nisp_j_el_model3_odonnell_ext` double, <br />
`euclid_nisp_y_el_model1_odonnell_ext` double, <br />
`euclid_nisp_y_el_model3_odonnell_ext` double, <br />
`euclid_vis_el_model1_odonnell_ext` double, <br />
`euclid_vis_el_model3_odonnell_ext` double, <br />
`2mass_h_el_model1_odonnell_ext` double, <br />
`2mass_h_el_model3_odonnell_ext` double, <br />
`2mass_j_el_model1_odonnell_ext` double, <br />
`2mass_j_el_model3_odonnell_ext` double, <br />
`2mass_ks_el_model1_odonnell_ext` double, <br />
`2mass_ks_el_model3_odonnell_ext` double, <br />
`sdss_r01_el_model1_odonnell_ext` double, <br />
`sdss_r01_el_model3_odonnell_ext` double, <br />
`dominant_shape` bigint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)', <br />
`bulge_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`disk_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`median_major_axis` double COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)', <br />
`scale_length` double COMMENT 'disc and bulge scalelength prior', <br />
`bulge_fraction` double COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)', <br />
`disk_scalelength` double COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)', <br />
`disk_nsersic` double COMMENT 'Sersic index of the disk component', <br />
`disk_r50` double COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678', <br />
`bulge_r50` double COMMENT 'half light radius of the bulge', <br />
`bulge_nsersic` double COMMENT 'Sersic index of the bulge component', <br />
`disk_ellipticity` double COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)', <br />
`bulge_ellipticity` double COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)', <br />
`disk_axis_ratio` double COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`bulge_axis_ratio` double COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`disk_inclination_angle` double COMMENT '', <br />
`bulge_inclination_angle` double COMMENT '', <br />
`random_index` double, <br />
`halo_id` bigint, <br />
`galaxy_id` bigint<br />
)<br />
PARTITIONED BY (<br />
`step` smallint<br />
)<br />
CLUSTERED BY (<br />
hpix_13_nest<br />
)<br />
SORTED BY (<br />
hpix_13_nest ASC<br />
)<br />
INTO 128 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
INSERT OVERWRITE TABLE tallada.flagship_mock_1_9_2_c2<br />
SELECT<br />
kind, luminosity_r01, halo_x, halo_y, halo_z, halo_vx, halo_vy, halo_vz, halo_r, true_redshift_halo, halo_lm, halo_n_sats, n_gals,<br />
abs_mag_r01, abs_mag_r01_evolved, luminosity_r01_evolved, gr_restframe, color_kind, x_gal, y_gal, z_gal, r_gal, true_redshift_gal,<br />
ra_gal, dec_gal, hpix_nest, SHIFTRIGHT(hpix_nest, (29-13)*2), vx_gal, vy_gal, vz_gal, vrad_gal, delta_r, observed_redshift_gal, sed_ke, <br />
gr_cosmos, sed_cosmos, ext_curve_cosmos, ebv_cosmos, is_within_cosmos, cosmos_distance, abs_mag_uv_dereddened, log_ml_r01, <br />
log_stellar_mass, metallicity, log_sfr, logf_halpha_ext, logf_halpha, halpha_scatter, loglum_halpha, loglum_halpha_ext, logf_dummy, <br />
z_dummy, logf_halpha_model1_ext, logf_halpha_model1, loglum_halpha_model1_ext, loglum_halpha_model1, logf_halpha_model3_ext, <br />
logf_halpha_model3, loglum_halpha_model3_ext, loglum_halpha_model3, logf_hbeta_model1_ext, logf_hbeta_model1, logf_o2_model1_ext, <br />
logf_o2_model1, logf_n2_model1_ext, logf_n2_model1, logf_o3_model1_ext, logf_o3_model1, logf_s2_model1_ext, logf_s2_model1, <br />
logf_hbeta_model3_ext, logf_hbeta_model3, logf_o2_model3_ext, logf_o2_model3, logf_n2_model3_ext, logf_n2_model3, logf_o3_model3_ext, <br />
logf_o3_model3, logf_s2_model3_ext, logf_s2_model3, blanco_decam_g, blanco_decam_i, blanco_decam_r, blanco_decam_z, gaia_bp, gaia_g, <br />
gaia_rp, subaru_hsc_z, jst_jpcam_g, kids_g, kids_i, kids_r, kids_u, lsst_g, lsst_i, lsst_r, lsst_u, lsst_y, lsst_z, cfht_megacam_r, <br />
cfht_megacam_u, pan_starrs_i, pan_starrs_z, euclid_nisp_h, euclid_nisp_j, euclid_nisp_y, euclid_vis, 2mass_h, 2mass_j, 2mass_ks, <br />
sdss_r01, l_gal, b_gal, mw_extinction, 2mass_h_odonnell_ext, 2mass_j_odonnell_ext, 2mass_ks_odonnell_ext, blanco_decam_g_odonnell_ext, <br />
blanco_decam_i_odonnell_ext, blanco_decam_r_odonnell_ext, blanco_decam_z_odonnell_ext, cfht_megacam_r_odonnell_ext, <br />
cfht_megacam_u_odonnell_ext, euclid_nisp_h_odonnell_ext, euclid_nisp_j_odonnell_ext, euclid_nisp_y_odonnell_ext, euclid_vis_odonnell_ext,<br />
gaia_bp_odonnell_ext, gaia_g_odonnell_ext, gaia_rp_odonnell_ext, jst_jpcam_g_odonnell_ext, kids_g_odonnell_ext, kids_i_odonnell_ext, <br />
kids_r_odonnell_ext, kids_u_odonnell_ext, lsst_g_odonnell_ext, lsst_i_odonnell_ext, lsst_r_odonnell_ext, lsst_u_odonnell_ext, <br />
lsst_y_odonnell_ext, lsst_z_odonnell_ext, pan_starrs_i_odonnell_ext, pan_starrs_z_odonnell_ext, sdss_r01_odonnell_ext, <br />
subaru_hsc_z_odonnell_ext, blanco_decam_g_el_model1_odonnell_ext, blanco_decam_g_el_model3_odonnell_ext, <br />
blanco_decam_i_el_model1_odonnell_ext, blanco_decam_i_el_model3_odonnell_ext, blanco_decam_r_el_model1_odonnell_ext, <br />
blanco_decam_r_el_model3_odonnell_ext, blanco_decam_z_el_model1_odonnell_ext, blanco_decam_z_el_model3_odonnell_ext, <br />
gaia_bp_el_model1_odonnell_ext, gaia_bp_el_model3_odonnell_ext, gaia_g_el_model1_odonnell_ext, gaia_g_el_model3_odonnell_ext, <br />
gaia_rp_el_model1_odonnell_ext, gaia_rp_el_model3_odonnell_ext, subaru_hsc_z_el_model1_odonnell_ext, subaru_hsc_z_el_model3_odonnell_ext, <br />
jst_jpcam_g_el_model1_odonnell_ext, jst_jpcam_g_el_model3_odonnell_ext, kids_g_el_model1_odonnell_ext, kids_g_el_model3_odonnell_ext, <br />
kids_i_el_model1_odonnell_ext, kids_i_el_model3_odonnell_ext, kids_r_el_model1_odonnell_ext, kids_r_el_model3_odonnell_ext, <br />
kids_u_el_model1_odonnell_ext, kids_u_el_model3_odonnell_ext, lsst_g_el_model1_odonnell_ext, lsst_g_el_model3_odonnell_ext, <br />
lsst_i_el_model1_odonnell_ext, lsst_i_el_model3_odonnell_ext, lsst_r_el_model1_odonnell_ext, lsst_r_el_model3_odonnell_ext, <br />
lsst_u_el_model1_odonnell_ext, lsst_u_el_model3_odonnell_ext, lsst_y_el_model1_odonnell_ext, lsst_y_el_model3_odonnell_ext, <br />
lsst_z_el_model1_odonnell_ext, lsst_z_el_model3_odonnell_ext, cfht_megacam_r_el_model1_odonnell_ext, <br />
cfht_megacam_r_el_model3_odonnell_ext, cfht_megacam_u_el_model1_odonnell_ext, cfht_megacam_u_el_model3_odonnell_ext, <br />
pan_starrs_i_el_model1_odonnell_ext, pan_starrs_i_el_model3_odonnell_ext, pan_starrs_z_el_model1_odonnell_ext, <br />
pan_starrs_z_el_model3_odonnell_ext, euclid_nisp_h_el_model1_odonnell_ext, euclid_nisp_h_el_model3_odonnell_ext, <br />
euclid_nisp_j_el_model1_odonnell_ext, euclid_nisp_j_el_model3_odonnell_ext, euclid_nisp_y_el_model1_odonnell_ext, <br />
euclid_nisp_y_el_model3_odonnell_ext, euclid_vis_el_model1_odonnell_ext, euclid_vis_el_model3_odonnell_ext, <br />
2mass_h_el_model1_odonnell_ext, 2mass_h_el_model3_odonnell_ext, 2mass_j_el_model1_odonnell_ext, 2mass_j_el_model3_odonnell_ext,<br />
2mass_ks_el_model1_odonnell_ext, 2mass_ks_el_model3_odonnell_ext, sdss_r01_el_model1_odonnell_ext, sdss_r01_el_model3_odonnell_ext, <br />
dominant_shape, bulge_angle, disk_angle, median_major_axis, scale_length, bulge_fraction, disk_scalelength, disk_nsersic, disk_r50, <br />
bulge_r50, bulge_nsersic, disk_ellipticity, bulge_ellipticity, disk_axis_ratio, bulge_axis_ratio, disk_inclination_angle,<br />
bulge_inclination_angle, random_index, halo_id, galaxy_id, step<br />
FROM cosmohub.flagship_mock_1_9_2_c<br />
<br />
CREATE TABLE tallada.flagship_mock_1_9_2_s(<br />
`kind` bigint, <br />
`luminosity_r01` double, <br />
`halo_x` double, <br />
`halo_y` double, <br />
`halo_z` double, <br />
`halo_vx` double, <br />
`halo_vy` double, <br />
`halo_vz` double, <br />
`halo_r` double, <br />
`true_redshift_halo` double, <br />
`halo_lm` double, <br />
`halo_n_sats` bigint, <br />
`n_gals` bigint, <br />
`abs_mag_r01` double, <br />
`abs_mag_r01_evolved` double, <br />
`luminosity_r01_evolved` double, <br />
`gr_restframe` double, <br />
`color_kind` int, <br />
`x_gal` double, <br />
`y_gal` double, <br />
`z_gal` double, <br />
`r_gal` double, <br />
`true_redshift_gal` double, <br />
`ra_gal` double, <br />
`dec_gal` double, <br />
`ra_mag_gal` double, <br />
`dec_mag_gal` double, <br />
`kappa` double,<br />
`gamma1` double,<br />
`gamma2` double,<br />
`hpix_29_nest` bigint, <br />
`hpix_13_nest` bigint, <br />
`vx_gal` double, <br />
`vy_gal` double, <br />
`vz_gal` double, <br />
`vrad_gal` double, <br />
`delta_r` double, <br />
`observed_redshift_gal` double, <br />
`sed_ke` double, <br />
`gr_cosmos` double, <br />
`sed_cosmos` double, <br />
`ext_curve_cosmos` bigint, <br />
`ebv_cosmos` double, <br />
`is_within_cosmos` boolean, <br />
`cosmos_distance` double, <br />
`abs_mag_uv_dereddened` double, <br />
`log_ml_r01` double, <br />
`log_stellar_mass` double, <br />
`metallicity` double, <br />
`log_sfr` double, <br />
`logf_halpha_ext` double, <br />
`logf_halpha` double, <br />
`halpha_scatter` double, <br />
`loglum_halpha` double, <br />
`loglum_halpha_ext` double, <br />
`logf_dummy` double, <br />
`z_dummy` double, <br />
`logf_halpha_model1_ext` double, <br />
`logf_halpha_model1` double, <br />
`loglum_halpha_model1_ext` double, <br />
`loglum_halpha_model1` double, <br />
`logf_halpha_model3_ext` double, <br />
`logf_halpha_model3` double, <br />
`loglum_halpha_model3_ext` double, <br />
`loglum_halpha_model3` double, <br />
`logf_hbeta_model1_ext` double, <br />
`logf_hbeta_model1` double, <br />
`logf_o2_model1_ext` double, <br />
`logf_o2_model1` double, <br />
`logf_n2_model1_ext` double, <br />
`logf_n2_model1` double, <br />
`logf_o3_model1_ext` double, <br />
`logf_o3_model1` double, <br />
`logf_s2_model1_ext` double, <br />
`logf_s2_model1` double, <br />
`logf_hbeta_model3_ext` double, <br />
`logf_hbeta_model3` double, <br />
`logf_o2_model3_ext` double, <br />
`logf_o2_model3` double, <br />
`logf_n2_model3_ext` double, <br />
`logf_n2_model3` double, <br />
`logf_o3_model3_ext` double, <br />
`logf_o3_model3` double, <br />
`logf_s2_model3_ext` double, <br />
`logf_s2_model3` double, <br />
`blanco_decam_g` double, <br />
`blanco_decam_i` double, <br />
`blanco_decam_r` double, <br />
`blanco_decam_z` double, <br />
`gaia_bp` double, <br />
`gaia_g` double, <br />
`gaia_rp` double, <br />
`subaru_hsc_z` double, <br />
`jst_jpcam_g` double, <br />
`kids_g` double, <br />
`kids_i` double, <br />
`kids_r` double, <br />
`kids_u` double, <br />
`lsst_g` double, <br />
`lsst_i` double, <br />
`lsst_r` double, <br />
`lsst_u` double, <br />
`lsst_y` double, <br />
`lsst_z` double, <br />
`cfht_megacam_r` double, <br />
`cfht_megacam_u` double, <br />
`pan_starrs_i` double, <br />
`pan_starrs_z` double, <br />
`euclid_nisp_h` double, <br />
`euclid_nisp_j` double, <br />
`euclid_nisp_y` double, <br />
`euclid_vis` double, <br />
`2mass_h` double, <br />
`2mass_j` double, <br />
`2mass_ks` double, <br />
`sdss_r01` double, <br />
`l_gal` double, <br />
`b_gal` double, <br />
`mw_extinction` double, <br />
`2mass_h_odonnell_ext` double, <br />
`2mass_j_odonnell_ext` double, <br />
`2mass_ks_odonnell_ext` double, <br />
`blanco_decam_g_odonnell_ext` double, <br />
`blanco_decam_i_odonnell_ext` double, <br />
`blanco_decam_r_odonnell_ext` double, <br />
`blanco_decam_z_odonnell_ext` double, <br />
`cfht_megacam_r_odonnell_ext` double, <br />
`cfht_megacam_u_odonnell_ext` double, <br />
`euclid_nisp_h_odonnell_ext` double, <br />
`euclid_nisp_j_odonnell_ext` double, <br />
`euclid_nisp_y_odonnell_ext` double, <br />
`euclid_vis_odonnell_ext` double, <br />
`gaia_bp_odonnell_ext` double, <br />
`gaia_g_odonnell_ext` double, <br />
`gaia_rp_odonnell_ext` double, <br />
`jst_jpcam_g_odonnell_ext` double, <br />
`kids_g_odonnell_ext` double, <br />
`kids_i_odonnell_ext` double, <br />
`kids_r_odonnell_ext` double, <br />
`kids_u_odonnell_ext` double, <br />
`lsst_g_odonnell_ext` double, <br />
`lsst_i_odonnell_ext` double, <br />
`lsst_r_odonnell_ext` double, <br />
`lsst_u_odonnell_ext` double, <br />
`lsst_y_odonnell_ext` double, <br />
`lsst_z_odonnell_ext` double, <br />
`pan_starrs_i_odonnell_ext` double, <br />
`pan_starrs_z_odonnell_ext` double, <br />
`sdss_r01_odonnell_ext` double, <br />
`subaru_hsc_z_odonnell_ext` double, <br />
`blanco_decam_g_el_model1_odonnell_ext` double, <br />
`blanco_decam_g_el_model3_odonnell_ext` double, <br />
`blanco_decam_i_el_model1_odonnell_ext` double, <br />
`blanco_decam_i_el_model3_odonnell_ext` double, <br />
`blanco_decam_r_el_model1_odonnell_ext` double, <br />
`blanco_decam_r_el_model3_odonnell_ext` double, <br />
`blanco_decam_z_el_model1_odonnell_ext` double, <br />
`blanco_decam_z_el_model3_odonnell_ext` double, <br />
`gaia_bp_el_model1_odonnell_ext` double, <br />
`gaia_bp_el_model3_odonnell_ext` double, <br />
`gaia_g_el_model1_odonnell_ext` double, <br />
`gaia_g_el_model3_odonnell_ext` double, <br />
`gaia_rp_el_model1_odonnell_ext` double, <br />
`gaia_rp_el_model3_odonnell_ext` double, <br />
`subaru_hsc_z_el_model1_odonnell_ext` double, <br />
`subaru_hsc_z_el_model3_odonnell_ext` double, <br />
`jst_jpcam_g_el_model1_odonnell_ext` double, <br />
`jst_jpcam_g_el_model3_odonnell_ext` double, <br />
`kids_g_el_model1_odonnell_ext` double, <br />
`kids_g_el_model3_odonnell_ext` double, <br />
`kids_i_el_model1_odonnell_ext` double, <br />
`kids_i_el_model3_odonnell_ext` double, <br />
`kids_r_el_model1_odonnell_ext` double, <br />
`kids_r_el_model3_odonnell_ext` double, <br />
`kids_u_el_model1_odonnell_ext` double, <br />
`kids_u_el_model3_odonnell_ext` double, <br />
`lsst_g_el_model1_odonnell_ext` double, <br />
`lsst_g_el_model3_odonnell_ext` double, <br />
`lsst_i_el_model1_odonnell_ext` double, <br />
`lsst_i_el_model3_odonnell_ext` double, <br />
`lsst_r_el_model1_odonnell_ext` double, <br />
`lsst_r_el_model3_odonnell_ext` double, <br />
`lsst_u_el_model1_odonnell_ext` double, <br />
`lsst_u_el_model3_odonnell_ext` double, <br />
`lsst_y_el_model1_odonnell_ext` double, <br />
`lsst_y_el_model3_odonnell_ext` double, <br />
`lsst_z_el_model1_odonnell_ext` double, <br />
`lsst_z_el_model3_odonnell_ext` double, <br />
`cfht_megacam_r_el_model1_odonnell_ext` double, <br />
`cfht_megacam_r_el_model3_odonnell_ext` double, <br />
`cfht_megacam_u_el_model1_odonnell_ext` double, <br />
`cfht_megacam_u_el_model3_odonnell_ext` double, <br />
`pan_starrs_i_el_model1_odonnell_ext` double, <br />
`pan_starrs_i_el_model3_odonnell_ext` double, <br />
`pan_starrs_z_el_model1_odonnell_ext` double, <br />
`pan_starrs_z_el_model3_odonnell_ext` double, <br />
`euclid_nisp_h_el_model1_odonnell_ext` double, <br />
`euclid_nisp_h_el_model3_odonnell_ext` double, <br />
`euclid_nisp_j_el_model1_odonnell_ext` double, <br />
`euclid_nisp_j_el_model3_odonnell_ext` double, <br />
`euclid_nisp_y_el_model1_odonnell_ext` double, <br />
`euclid_nisp_y_el_model3_odonnell_ext` double, <br />
`euclid_vis_el_model1_odonnell_ext` double, <br />
`euclid_vis_el_model3_odonnell_ext` double, <br />
`2mass_h_el_model1_odonnell_ext` double, <br />
`2mass_h_el_model3_odonnell_ext` double, <br />
`2mass_j_el_model1_odonnell_ext` double, <br />
`2mass_j_el_model3_odonnell_ext` double, <br />
`2mass_ks_el_model1_odonnell_ext` double, <br />
`2mass_ks_el_model3_odonnell_ext` double, <br />
`sdss_r01_el_model1_odonnell_ext` double, <br />
`sdss_r01_el_model3_odonnell_ext` double, <br />
`dominant_shape` bigint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)', <br />
`bulge_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`disk_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`median_major_axis` double COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)', <br />
`scale_length` double COMMENT 'disc and bulge scalelength prior', <br />
`bulge_fraction` double COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)', <br />
`disk_scalelength` double COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)', <br />
`disk_nsersic` double COMMENT 'Sersic index of the disk component', <br />
`disk_r50` double COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678', <br />
`bulge_r50` double COMMENT 'half light radius of the bulge', <br />
`bulge_nsersic` double COMMENT 'Sersic index of the bulge component', <br />
`disk_ellipticity` double COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)', <br />
`bulge_ellipticity` double COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)', <br />
`disk_axis_ratio` double COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`bulge_axis_ratio` double COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`disk_inclination_angle` double COMMENT '', <br />
`bulge_inclination_angle` double COMMENT '', <br />
`random_index` double, <br />
`halo_id` bigint, <br />
`galaxy_id` bigint<br />
)<br />
PARTITIONED BY (<br />
`step` smallint<br />
)<br />
CLUSTERED BY (<br />
hpix_13_nest<br />
)<br />
SORTED BY (<br />
hpix_13_nest ASC<br />
)<br />
INTO 128 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
cat << EOF > lensing_step.sql<br />
SET hive.auto.convert.join.noconditionaltask.size=1;<br />
INSERT OVERWRITE TABLE tallada.flagship_mock_1_9_2_s PARTITION (step=${STEP})<br />
SELECT<br />
m.kind, m.luminosity_r01, m.halo_x, m.halo_y, m.halo_z, m.halo_vx, m.halo_vy, m.halo_vz, m.halo_r, m.true_redshift_halo, m.halo_lm,<br />
m.halo_n_sats, m.n_gals, m.abs_mag_r01, m.abs_mag_r01_evolved, m.luminosity_r01_evolved, m.gr_restframe, m.color_kind, m.x_gal,<br />
m.y_gal, m.z_gal, m.r_gal, m.true_redshift_gal, m.ra_gal, m.dec_gal, magpos.ra_mag, magpos.dec_mag, m.kappa, m.gamma1, m.gamma2,<br />
m.hpix_29_nest, m.hpix_13_nest, m.vx_gal, m.vy_gal, m.vz_gal, m.vrad_gal, m.delta_r, m.observed_redshift_gal, m.sed_ke, m.gr_cosmos, <br />
m.sed_cosmos, m.ext_curve_cosmos, m.ebv_cosmos, m.is_within_cosmos, m.cosmos_distance, m.abs_mag_uv_dereddened, m.log_ml_r01,<br />
m.log_stellar_mass, m.metallicity, m.log_sfr, m.logf_halpha_ext, m.logf_halpha, m.halpha_scatter, m.loglum_halpha, m.loglum_halpha_ext, <br />
m.logf_dummy, m.z_dummy, m.logf_halpha_model1_ext, m.logf_halpha_model1, m.loglum_halpha_model1_ext, m.loglum_halpha_model1, <br />
m.logf_halpha_model3_ext, m.logf_halpha_model3, m.loglum_halpha_model3_ext, m.loglum_halpha_model3, m.logf_hbeta_model1_ext, <br />
m.logf_hbeta_model1, m.logf_o2_model1_ext, m.logf_o2_model1, m.logf_n2_model1_ext, m.logf_n2_model1, m.logf_o3_model1_ext, <br />
m.logf_o3_model1, m.logf_s2_model1_ext, m.logf_s2_model1, m.logf_hbeta_model3_ext, m.logf_hbeta_model3, m.logf_o2_model3_ext, <br />
m.logf_o2_model3, m.logf_n2_model3_ext, m.logf_n2_model3, m.logf_o3_model3_ext, m.logf_o3_model3, m.logf_s2_model3_ext,<br />
m.logf_s2_model3, m.blanco_decam_g, m.blanco_decam_i, m.blanco_decam_r, m.blanco_decam_z, m.gaia_bp, m.gaia_g, m.gaia_rp, <br />
m.subaru_hsc_z, m.jst_jpcam_g, m.kids_g, m.kids_i, m.kids_r, m.kids_u, m.lsst_g, m.lsst_i, m.lsst_r, m.lsst_u, m.lsst_y, m.lsst_z, <br />
m.cfht_megacam_r, m.cfht_megacam_u, m.pan_starrs_i, m.pan_starrs_z, m.euclid_nisp_h, m.euclid_nisp_j, m.euclid_nisp_y, m.euclid_vis, <br />
m.2mass_h, m.2mass_j, m.2mass_ks, m.sdss_r01, m.l_gal, m.b_gal,mw_extinction, m.2mass_h_odonnell_ext, m.2mass_j_odonnell_ext, <br />
m.2mass_ks_odonnell_ext, m.blanco_decam_g_odonnell_ext, m.blanco_decam_i_odonnell_ext, m.blanco_decam_r_odonnell_ext, <br />
m.blanco_decam_z_odonnell_ext, m.cfht_megacam_r_odonnell_ext, m.cfht_megacam_u_odonnell_ext, m.euclid_nisp_h_odonnell_ext, <br />
m.euclid_nisp_j_odonnell_ext, m.euclid_nisp_y_odonnell_ext, m.euclid_vis_odonnell_ext, m.gaia_bp_odonnell_ext, m.gaia_g_odonnell_ext, <br />
m.gaia_rp_odonnell_ext, m.jst_jpcam_g_odonnell_ext, m.kids_g_odonnell_ext, m.kids_i_odonnell_ext, m.kids_r_odonnell_ext, <br />
m.kids_u_odonnell_ext, m.lsst_g_odonnell_ext, m.lsst_i_odonnell_ext, m.lsst_r_odonnell_ext, m.lsst_u_odonnell_ext, <br />
m.lsst_y_odonnell_ext, m.lsst_z_odonnell_ext, m.pan_starrs_i_odonnell_ext, m.pan_starrs_z_odonnell_ext, m.sdss_r01_odonnell_ext, <br />
m.subaru_hsc_z_odonnell_ext, m.blanco_decam_g_el_model1_odonnell_ext, m.blanco_decam_g_el_model3_odonnell_ext, <br />
m.blanco_decam_i_el_model1_odonnell_ext, m.blanco_decam_i_el_model3_odonnell_ext, m.blanco_decam_r_el_model1_odonnell_ext, <br />
m.blanco_decam_r_el_model3_odonnell_ext, m.blanco_decam_z_el_model1_odonnell_ext, m.blanco_decam_z_el_model3_odonnell_ext,<br />
m.gaia_bp_el_model1_odonnell_ext, m.gaia_bp_el_model3_odonnell_ext, m.gaia_g_el_model1_odonnell_ext, m.gaia_g_el_model3_odonnell_ext, <br />
m.gaia_rp_el_model1_odonnell_ext, m.gaia_rp_el_model3_odonnell_ext, m.subaru_hsc_z_el_model1_odonnell_ext, <br />
m.subaru_hsc_z_el_model3_odonnell_ext, m.jst_jpcam_g_el_model1_odonnell_ext, m.jst_jpcam_g_el_model3_odonnell_ext,<br />
m.kids_g_el_model1_odonnell_ext, m.kids_g_el_model3_odonnell_ext, m.kids_i_el_model1_odonnell_ext, m.kids_i_el_model3_odonnell_ext, <br />
m.kids_r_el_model1_odonnell_ext, m.kids_r_el_model3_odonnell_ext, m.kids_u_el_model1_odonnell_ext, m.kids_u_el_model3_odonnell_ext, <br />
m.lsst_g_el_model1_odonnell_ext, m.lsst_g_el_model3_odonnell_ext, m.lsst_i_el_model1_odonnell_ext, m.lsst_i_el_model3_odonnell_ext,<br />
m.lsst_r_el_model1_odonnell_ext, m.lsst_r_el_model3_odonnell_ext, m.lsst_u_el_model1_odonnell_ext, m.lsst_u_el_model3_odonnell_ext, <br />
m.lsst_y_el_model1_odonnell_ext, m.lsst_y_el_model3_odonnell_ext, m.lsst_z_el_model1_odonnell_ext, m.lsst_z_el_model3_odonnell_ext, <br />
m.cfht_megacam_r_el_model1_odonnell_ext, m.cfht_megacam_r_el_model3_odonnell_ext, m.cfht_megacam_u_el_model1_odonnell_ext,<br />
m.cfht_megacam_u_el_model3_odonnell_ext, m.pan_starrs_i_el_model1_odonnell_ext, m.pan_starrs_i_el_model3_odonnell_ext, <br />
m.pan_starrs_z_el_model1_odonnell_ext, m.pan_starrs_z_el_model3_odonnell_ext, m.euclid_nisp_h_el_model1_odonnell_ext, <br />
m.euclid_nisp_h_el_model3_odonnell_ext, m.euclid_nisp_j_el_model1_odonnell_ext, m.euclid_nisp_j_el_model3_odonnell_ext, <br />
m.euclid_nisp_y_el_model1_odonnell_ext, m.euclid_nisp_y_el_model3_odonnell_ext, m.euclid_vis_el_model1_odonnell_ext, <br />
m.euclid_vis_el_model3_odonnell_ext, m.2mass_h_el_model1_odonnell_ext, m.2mass_h_el_model3_odonnell_ext, <br />
m.2mass_j_el_model1_odonnell_ext, m.2mass_j_el_model3_odonnell_ext, m.2mass_ks_el_model1_odonnell_ext, <br />
m.2mass_ks_el_model3_odonnell_ext, m.sdss_r01_el_model1_odonnell_ext, m.sdss_r01_el_model3_odonnell_ext, m.dominant_shape, <br />
m.bulge_angle, m.disk_angle, m.median_major_axis, m.scale_length, m.bulge_fraction, m.disk_scalelength, m.disk_nsersic, m.disk_r50,<br />
m.bulge_r50, m.bulge_nsersic, m.disk_ellipticity, m.bulge_ellipticity, m.disk_axis_ratio, m.bulge_axis_ratio, m.disk_inclination_angle, <br />
m.bulge_inclination_angle, m.random_index, m.halo_id, m.galaxy_id<br />
FROM (<br />
SELECT <br />
m.kind, m.luminosity_r01, m.halo_x, m.halo_y, m.halo_z, m.halo_vx, m.halo_vy, m.halo_vz, m.halo_r, m.true_redshift_halo, m.halo_lm, <br />
m.halo_n_sats, m.n_gals, m.abs_mag_r01, m.abs_mag_r01_evolved, m.luminosity_r01_evolved, m.gr_restframe, m.color_kind, m.x_gal, <br />
m.y_gal, m.z_gal, m.r_gal, m.true_redshift_gal, m.ra_gal, m.dec_gal, l.defl_0 as kappa, l.shear_1 as gamma1, l.shear_2 as gamma2,<br />
m.hpix_29_nest, m.hpix_13_nest, m.vx_gal, m.vy_gal, m.vz_gal, m.vrad_gal, m.delta_r, m.observed_redshift_gal, m.sed_ke, m.gr_cosmos, <br />
m.sed_cosmos, m.ext_curve_cosmos, m.ebv_cosmos, m.is_within_cosmos, m.cosmos_distance, m.abs_mag_uv_dereddened, m.log_ml_r01, <br />
m.log_stellar_mass, m.metallicity, m.log_sfr, m.logf_halpha_ext, m.logf_halpha, m.halpha_scatter, m.loglum_halpha, <br />
m.loglum_halpha_ext, m.logf_dummy, m.z_dummy, m.logf_halpha_model1_ext, m.logf_halpha_model1, m.loglum_halpha_model1_ext, <br />
m.loglum_halpha_model1, m.logf_halpha_model3_ext, m.logf_halpha_model3, m.loglum_halpha_model3_ext, m.loglum_halpha_model3, <br />
m.logf_hbeta_model1_ext, m.logf_hbeta_model1, m.logf_o2_model1_ext, m.logf_o2_model1, m.logf_n2_model1_ext, m.logf_n2_model1,<br />
m.logf_o3_model1_ext, m.logf_o3_model1, m.logf_s2_model1_ext, m.logf_s2_model1, m.logf_hbeta_model3_ext, m.logf_hbeta_model3, <br />
m.logf_o2_model3_ext, m.logf_o2_model3, m.logf_n2_model3_ext, m.logf_n2_model3, m.logf_o3_model3_ext, m.logf_o3_model3, <br />
m.logf_s2_model3_ext, m.logf_s2_model3, m.blanco_decam_g, m.blanco_decam_i, m.blanco_decam_r, m.blanco_decam_z, m.gaia_bp, m.gaia_g, <br />
m.gaia_rp, m.subaru_hsc_z, m.jst_jpcam_g, m.kids_g, m.kids_i, m.kids_r, m.kids_u, m.lsst_g, m.lsst_i, m.lsst_r, m.lsst_u, m.lsst_y, <br />
m.lsst_z, m.cfht_megacam_r, m.cfht_megacam_u, m.pan_starrs_i, m.pan_starrs_z, m.euclid_nisp_h, m.euclid_nisp_j, m.euclid_nisp_y, <br />
m.euclid_vis, m.2mass_h, m.2mass_j, m.2mass_ks, m.sdss_r01, m.l_gal, m.b_gal,mw_extinction, m.2mass_h_odonnell_ext, <br />
m.2mass_j_odonnell_ext, m.2mass_ks_odonnell_ext, m.blanco_decam_g_odonnell_ext, m.blanco_decam_i_odonnell_ext, <br />
m.blanco_decam_r_odonnell_ext, m.blanco_decam_z_odonnell_ext, m.cfht_megacam_r_odonnell_ext, m.cfht_megacam_u_odonnell_ext,<br />
m.euclid_nisp_h_odonnell_ext, m.euclid_nisp_j_odonnell_ext, m.euclid_nisp_y_odonnell_ext, m.euclid_vis_odonnell_ext, <br />
m.gaia_bp_odonnell_ext, m.gaia_g_odonnell_ext, m.gaia_rp_odonnell_ext, m.jst_jpcam_g_odonnell_ext, m.kids_g_odonnell_ext, <br />
m.kids_i_odonnell_ext, m.kids_r_odonnell_ext, m.kids_u_odonnell_ext, m.lsst_g_odonnell_ext, m.lsst_i_odonnell_ext, <br />
m.lsst_r_odonnell_ext, m.lsst_u_odonnell_ext, m.lsst_y_odonnell_ext, m.lsst_z_odonnell_ext, m.pan_starrs_i_odonnell_ext, <br />
m.pan_starrs_z_odonnell_ext, m.sdss_r01_odonnell_ext, m.subaru_hsc_z_odonnell_ext, m.blanco_decam_g_el_model1_odonnell_ext, <br />
m.blanco_decam_g_el_model3_odonnell_ext, m.blanco_decam_i_el_model1_odonnell_ext, m.blanco_decam_i_el_model3_odonnell_ext, <br />
m.blanco_decam_r_el_model1_odonnell_ext, m.blanco_decam_r_el_model3_odonnell_ext, m.blanco_decam_z_el_model1_odonnell_ext, <br />
m.blanco_decam_z_el_model3_odonnell_ext, m.gaia_bp_el_model1_odonnell_ext, m.gaia_bp_el_model3_odonnell_ext,<br />
m.gaia_g_el_model1_odonnell_ext, m.gaia_g_el_model3_odonnell_ext, m.gaia_rp_el_model1_odonnell_ext, m.gaia_rp_el_model3_odonnell_ext, <br />
m.subaru_hsc_z_el_model1_odonnell_ext, m.subaru_hsc_z_el_model3_odonnell_ext, m.jst_jpcam_g_el_model1_odonnell_ext, <br />
m.jst_jpcam_g_el_model3_odonnell_ext, m.kids_g_el_model1_odonnell_ext, m.kids_g_el_model3_odonnell_ext,<br />
m.kids_i_el_model1_odonnell_ext, m.kids_i_el_model3_odonnell_ext, m.kids_r_el_model1_odonnell_ext, m.kids_r_el_model3_odonnell_ext, <br />
m.kids_u_el_model1_odonnell_ext, m.kids_u_el_model3_odonnell_ext, m.lsst_g_el_model1_odonnell_ext, m.lsst_g_el_model3_odonnell_ext, <br />
m.lsst_i_el_model1_odonnell_ext, m.lsst_i_el_model3_odonnell_ext, m.lsst_r_el_model1_odonnell_ext, m.lsst_r_el_model3_odonnell_ext,<br />
m.lsst_u_el_model1_odonnell_ext, m.lsst_u_el_model3_odonnell_ext, m.lsst_y_el_model1_odonnell_ext, m.lsst_y_el_model3_odonnell_ext, <br />
m.lsst_z_el_model1_odonnell_ext, m.lsst_z_el_model3_odonnell_ext, m.cfht_megacam_r_el_model1_odonnell_ext, <br />
m.cfht_megacam_r_el_model3_odonnell_ext, m.cfht_megacam_u_el_model1_odonnell_ext, m.cfht_megacam_u_el_model3_odonnell_ext, <br />
m.pan_starrs_i_el_model1_odonnell_ext, m.pan_starrs_i_el_model3_odonnell_ext, m.pan_starrs_z_el_model1_odonnell_ext, <br />
m.pan_starrs_z_el_model3_odonnell_ext, m.euclid_nisp_h_el_model1_odonnell_ext, m.euclid_nisp_h_el_model3_odonnell_ext,<br />
m.euclid_nisp_j_el_model1_odonnell_ext, m.euclid_nisp_j_el_model3_odonnell_ext, m.euclid_nisp_y_el_model1_odonnell_ext, <br />
m.euclid_nisp_y_el_model3_odonnell_ext, m.euclid_vis_el_model1_odonnell_ext, m.euclid_vis_el_model3_odonnell_ext, <br />
m.2mass_h_el_model1_odonnell_ext, m.2mass_h_el_model3_odonnell_ext, m.2mass_j_el_model1_odonnell_ext, <br />
m.2mass_j_el_model3_odonnell_ext, m.2mass_ks_el_model1_odonnell_ext, m.2mass_ks_el_model3_odonnell_ext, <br />
m.sdss_r01_el_model1_odonnell_ext, m.sdss_r01_el_model3_odonnell_ext, m.dominant_shape, m.bulge_angle, m.disk_angle, <br />
m.median_major_axis, m.scale_length, m.bulge_fraction, m.disk_scalelength, m.disk_nsersic, m.disk_r50, m.bulge_r50, m.bulge_nsersic, <br />
m.disk_ellipticity, m.bulge_ellipticity, m.disk_axis_ratio, m.bulge_axis_ratio, m.disk_inclination_angle, m.bulge_inclination_angle, <br />
m.random_index, m.halo_id, m.galaxy_id,<br />
udf.magnified_positions(CAST(m.ra_gal AS FLOAT), CAST(m.dec_gal AS FLOAT), l.defl_1, l.defl_2) AS magpos<br />
FROM tallada.flagship_mock_1_9_2_c2 AS m<br />
JOIN cosmohub.flagship_lensing AS l<br />
ON udf.ang2pix(13, m.ra_gal, m.dec_gal, True, True) = l.hpix<br />
AND m.step = l.step<br />
AND l.step=${STEP} AND m.step=${STEP}<br />
) AS m<br />
;<br />
<br />
for STEP in $(seq 344 400);<br />
do<br />
echo $STEP<br />
sem -j4 "beeline --color=true --showHeader=true --verbose=true --silent=false -u \<br />
'jdbc:hive2://ambarisrv02.pic.es:2181,ambarisrv03.pic.es:2181,ambarisrv01.pic.es:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2' \<br />
-f lensing_step.sql --hivevar STEP=${STEP}"<br />
done<br />
sem --wait</div>
Tallada
https://pwiki.pic.es/index.php?title=Euclid_mock_production_v1.9.2&diff=819
Euclid mock production v1.9.2
2020-07-14T08:01:49Z
<p>Tallada: </p>
<hr />
<div> CREATE EXTERNAL TABLE `flagship_1_9_2`(<br />
`kind` bigint, <br />
`luminosity_r01` double, <br />
`halo_x` double, <br />
`halo_y` double, <br />
`halo_z` double, <br />
`halo_vx` double, <br />
`halo_vy` double, <br />
`halo_vz` double, <br />
`halo_r` double, <br />
`true_redshift_halo` double, <br />
`halo_lm` double, <br />
`halo_n_sats` bigint, <br />
`n_gals` bigint, <br />
`abs_mag_r01` double, <br />
`abs_mag_r01_evolved` double, <br />
`luminosity_r01_evolved` double, <br />
`gr_restframe` double, <br />
`color_kind` int, <br />
`x_gal` double, <br />
`y_gal` double, <br />
`z_gal` double, <br />
`r_gal` double, <br />
`true_redshift_gal` double, <br />
`ra_gal` double, <br />
`dec_gal` double, <br />
`hpix_nest` bigint, <br />
`vx_gal` double, <br />
`vy_gal` double, <br />
`vz_gal` double, <br />
`vrad_gal` double, <br />
`delta_r` double, <br />
`observed_redshift_gal` double, <br />
`sed_ke` double, <br />
`gr_cosmos` double, <br />
`sed_cosmos` double, <br />
`ext_curve_cosmos` bigint, <br />
`ebv_cosmos` double, <br />
`is_within_cosmos` boolean, <br />
`cosmos_distance` double, <br />
`abs_mag_uv_dereddened` double, <br />
`log_ml_r01` double, <br />
`log_stellar_mass` double, <br />
`metallicity` double, <br />
`log_sfr` double, <br />
`logf_halpha_ext` double, <br />
`logf_halpha` double, <br />
`halpha_scatter` double, <br />
`loglum_halpha` double, <br />
`loglum_halpha_ext` double, <br />
`logf_dummy` double, <br />
`z_dummy` double, <br />
`logf_halpha_model1_ext` double, <br />
`logf_halpha_model1` double, <br />
`loglum_halpha_model1_ext` double, <br />
`loglum_halpha_model1` double, <br />
`logf_halpha_model3_ext` double, <br />
`logf_halpha_model3` double, <br />
`loglum_halpha_model3_ext` double, <br />
`loglum_halpha_model3` double, <br />
`logf_hbeta_model1_ext` double, <br />
`logf_hbeta_model1` double, <br />
`logf_o2_model1_ext` double, <br />
`logf_o2_model1` double, <br />
`logf_n2_model1_ext` double, <br />
`logf_n2_model1` double, <br />
`logf_o3_model1_ext` double, <br />
`logf_o3_model1` double, <br />
`logf_s2_model1_ext` double, <br />
`logf_s2_model1` double, <br />
`logf_hbeta_model3_ext` double, <br />
`logf_hbeta_model3` double, <br />
`logf_o2_model3_ext` double, <br />
`logf_o2_model3` double, <br />
`logf_n2_model3_ext` double, <br />
`logf_n2_model3` double, <br />
`logf_o3_model3_ext` double, <br />
`logf_o3_model3` double, <br />
`logf_s2_model3_ext` double, <br />
`logf_s2_model3` double, <br />
`blanco_decam_g` double, <br />
`blanco_decam_i` double, <br />
`blanco_decam_r` double, <br />
`blanco_decam_z` double, <br />
`gaia_bp` double, <br />
`gaia_g` double, <br />
`gaia_rp` double, <br />
`subaru_hsc_z` double, <br />
`jst_jpcam_g` double, <br />
`kids_g` double, <br />
`kids_i` double, <br />
`kids_r` double, <br />
`kids_u` double, <br />
`lsst_g` double, <br />
`lsst_i` double, <br />
`lsst_r` double, <br />
`lsst_u` double, <br />
`lsst_y` double, <br />
`lsst_z` double, <br />
`cfht_megacam_r` double, <br />
`cfht_megacam_u` double, <br />
`pan_starrs_i` double, <br />
`pan_starrs_z` double, <br />
`euclid_nisp_h` double, <br />
`euclid_nisp_j` double, <br />
`euclid_nisp_y` double, <br />
`euclid_vis` double, <br />
`2mass_h` double, <br />
`2mass_j` double, <br />
`2mass_ks` double, <br />
`sdss_r01` double, <br />
`l_gal` double, <br />
`b_gal` double, <br />
`mw_extinction` double, <br />
`2mass_h_odonnell_ext` double, <br />
`2mass_j_odonnell_ext` double, <br />
`2mass_ks_odonnell_ext` double, <br />
`blanco_decam_g_odonnell_ext` double, <br />
`blanco_decam_i_odonnell_ext` double, <br />
`blanco_decam_r_odonnell_ext` double, <br />
`blanco_decam_z_odonnell_ext` double, <br />
`cfht_megacam_r_odonnell_ext` double, <br />
`cfht_megacam_u_odonnell_ext` double, <br />
`euclid_nisp_h_odonnell_ext` double, <br />
`euclid_nisp_j_odonnell_ext` double, <br />
`euclid_nisp_y_odonnell_ext` double, <br />
`euclid_vis_odonnell_ext` double, <br />
`gaia_bp_odonnell_ext` double, <br />
`gaia_g_odonnell_ext` double, <br />
`gaia_rp_odonnell_ext` double, <br />
`jst_jpcam_g_odonnell_ext` double, <br />
`kids_g_odonnell_ext` double, <br />
`kids_i_odonnell_ext` double, <br />
`kids_r_odonnell_ext` double, <br />
`kids_u_odonnell_ext` double, <br />
`lsst_g_odonnell_ext` double, <br />
`lsst_i_odonnell_ext` double, <br />
`lsst_r_odonnell_ext` double, <br />
`lsst_u_odonnell_ext` double, <br />
`lsst_y_odonnell_ext` double, <br />
`lsst_z_odonnell_ext` double, <br />
`pan_starrs_i_odonnell_ext` double, <br />
`pan_starrs_z_odonnell_ext` double, <br />
`sdss_r01_odonnell_ext` double, <br />
`subaru_hsc_z_odonnell_ext` double, <br />
`blanco_decam_g_el_model1_odonnell_ext` double, <br />
`blanco_decam_g_el_model3_odonnell_ext` double, <br />
`blanco_decam_i_el_model1_odonnell_ext` double, <br />
`blanco_decam_i_el_model3_odonnell_ext` double, <br />
`blanco_decam_r_el_model1_odonnell_ext` double, <br />
`blanco_decam_r_el_model3_odonnell_ext` double, <br />
`blanco_decam_z_el_model1_odonnell_ext` double, <br />
`blanco_decam_z_el_model3_odonnell_ext` double, <br />
`gaia_bp_el_model1_odonnell_ext` double, <br />
`gaia_bp_el_model3_odonnell_ext` double, <br />
`gaia_g_el_model1_odonnell_ext` double, <br />
`gaia_g_el_model3_odonnell_ext` double, <br />
`gaia_rp_el_model1_odonnell_ext` double, <br />
`gaia_rp_el_model3_odonnell_ext` double, <br />
`subaru_hsc_z_el_model1_odonnell_ext` double, <br />
`subaru_hsc_z_el_model3_odonnell_ext` double, <br />
`jst_jpcam_g_el_model1_odonnell_ext` double, <br />
`jst_jpcam_g_el_model3_odonnell_ext` double, <br />
`kids_g_el_model1_odonnell_ext` double, <br />
`kids_g_el_model3_odonnell_ext` double, <br />
`kids_i_el_model1_odonnell_ext` double, <br />
`kids_i_el_model3_odonnell_ext` double, <br />
`kids_r_el_model1_odonnell_ext` double, <br />
`kids_r_el_model3_odonnell_ext` double, <br />
`kids_u_el_model1_odonnell_ext` double, <br />
`kids_u_el_model3_odonnell_ext` double, <br />
`lsst_g_el_model1_odonnell_ext` double, <br />
`lsst_g_el_model3_odonnell_ext` double, <br />
`lsst_i_el_model1_odonnell_ext` double, <br />
`lsst_i_el_model3_odonnell_ext` double, <br />
`lsst_r_el_model1_odonnell_ext` double, <br />
`lsst_r_el_model3_odonnell_ext` double, <br />
`lsst_u_el_model1_odonnell_ext` double, <br />
`lsst_u_el_model3_odonnell_ext` double, <br />
`lsst_y_el_model1_odonnell_ext` double, <br />
`lsst_y_el_model3_odonnell_ext` double, <br />
`lsst_z_el_model1_odonnell_ext` double, <br />
`lsst_z_el_model3_odonnell_ext` double, <br />
`cfht_megacam_r_el_model1_odonnell_ext` double, <br />
`cfht_megacam_r_el_model3_odonnell_ext` double, <br />
`cfht_megacam_u_el_model1_odonnell_ext` double, <br />
`cfht_megacam_u_el_model3_odonnell_ext` double, <br />
`pan_starrs_i_el_model1_odonnell_ext` double, <br />
`pan_starrs_i_el_model3_odonnell_ext` double, <br />
`pan_starrs_z_el_model1_odonnell_ext` double, <br />
`pan_starrs_z_el_model3_odonnell_ext` double, <br />
`euclid_nisp_h_el_model1_odonnell_ext` double, <br />
`euclid_nisp_h_el_model3_odonnell_ext` double, <br />
`euclid_nisp_j_el_model1_odonnell_ext` double, <br />
`euclid_nisp_j_el_model3_odonnell_ext` double, <br />
`euclid_nisp_y_el_model1_odonnell_ext` double, <br />
`euclid_nisp_y_el_model3_odonnell_ext` double, <br />
`euclid_vis_el_model1_odonnell_ext` double, <br />
`euclid_vis_el_model3_odonnell_ext` double, <br />
`2mass_h_el_model1_odonnell_ext` double, <br />
`2mass_h_el_model3_odonnell_ext` double, <br />
`2mass_j_el_model1_odonnell_ext` double, <br />
`2mass_j_el_model3_odonnell_ext` double, <br />
`2mass_ks_el_model1_odonnell_ext` double, <br />
`2mass_ks_el_model3_odonnell_ext` double, <br />
`sdss_r01_el_model1_odonnell_ext` double, <br />
`sdss_r01_el_model3_odonnell_ext` double, <br />
`dominant_shape` bigint, <br />
`bulge_angle` double, <br />
`disk_angle` double, <br />
`median_major_axis` double, <br />
`scale_length` double, <br />
`bulge_fraction` double, <br />
`disk_scalelength` double, <br />
`disk_nsersic` double, <br />
`disk_r50` double, <br />
`bulge_r50` double, <br />
`bulge_nsersic` double, <br />
`disk_ellipticity` double, <br />
`bulge_ellipticity` double, <br />
`disk_axis_ratio` double, <br />
`bulge_axis_ratio` double, <br />
`disk_inclination_angle` double, <br />
`bulge_inclination_angle` double, <br />
`step` bigint, <br />
`random_index` double, <br />
`halo_id` bigint, <br />
`galaxy_id` bigint)<br />
ROW FORMAT SERDE <br />
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' <br />
STORED AS INPUTFORMAT <br />
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' <br />
OUTPUTFORMAT <br />
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'<br />
;<br />
<br />
CREATE TABLE flagship_mock_1_9_2_c(<br />
`kind` bigint, <br />
`luminosity_r01` double, <br />
`halo_x` double, <br />
`halo_y` double, <br />
`halo_z` double, <br />
`halo_vx` double, <br />
`halo_vy` double, <br />
`halo_vz` double, <br />
`halo_r` double, <br />
`true_redshift_halo` double, <br />
`halo_lm` double, <br />
`halo_n_sats` bigint, <br />
`n_gals` bigint, <br />
`abs_mag_r01` double, <br />
`abs_mag_r01_evolved` double, <br />
`luminosity_r01_evolved` double, <br />
`gr_restframe` double, <br />
`color_kind` int, <br />
`x_gal` double, <br />
`y_gal` double, <br />
`z_gal` double, <br />
`r_gal` double, <br />
`true_redshift_gal` double, <br />
`ra_gal` double, <br />
`dec_gal` double, <br />
`hpix_nest` bigint, <br />
`vx_gal` double, <br />
`vy_gal` double, <br />
`vz_gal` double, <br />
`vrad_gal` double, <br />
`delta_r` double, <br />
`observed_redshift_gal` double, <br />
`sed_ke` double, <br />
`gr_cosmos` double, <br />
`sed_cosmos` double, <br />
`ext_curve_cosmos` bigint, <br />
`ebv_cosmos` double, <br />
`is_within_cosmos` boolean, <br />
`cosmos_distance` double, <br />
`abs_mag_uv_dereddened` double, <br />
`log_ml_r01` double, <br />
`log_stellar_mass` double, <br />
`metallicity` double, <br />
`log_sfr` double, <br />
`logf_halpha_ext` double, <br />
`logf_halpha` double, <br />
`halpha_scatter` double, <br />
`loglum_halpha` double, <br />
`loglum_halpha_ext` double, <br />
`logf_dummy` double, <br />
`z_dummy` double, <br />
`logf_halpha_model1_ext` double, <br />
`logf_halpha_model1` double, <br />
`loglum_halpha_model1_ext` double, <br />
`loglum_halpha_model1` double, <br />
`logf_halpha_model3_ext` double, <br />
`logf_halpha_model3` double, <br />
`loglum_halpha_model3_ext` double, <br />
`loglum_halpha_model3` double, <br />
`logf_hbeta_model1_ext` double, <br />
`logf_hbeta_model1` double, <br />
`logf_o2_model1_ext` double, <br />
`logf_o2_model1` double, <br />
`logf_n2_model1_ext` double, <br />
`logf_n2_model1` double, <br />
`logf_o3_model1_ext` double, <br />
`logf_o3_model1` double, <br />
`logf_s2_model1_ext` double, <br />
`logf_s2_model1` double, <br />
`logf_hbeta_model3_ext` double, <br />
`logf_hbeta_model3` double, <br />
`logf_o2_model3_ext` double, <br />
`logf_o2_model3` double, <br />
`logf_n2_model3_ext` double, <br />
`logf_n2_model3` double, <br />
`logf_o3_model3_ext` double, <br />
`logf_o3_model3` double, <br />
`logf_s2_model3_ext` double, <br />
`logf_s2_model3` double, <br />
`blanco_decam_g` double, <br />
`blanco_decam_i` double, <br />
`blanco_decam_r` double, <br />
`blanco_decam_z` double, <br />
`gaia_bp` double, <br />
`gaia_g` double, <br />
`gaia_rp` double, <br />
`subaru_hsc_z` double, <br />
`jst_jpcam_g` double, <br />
`kids_g` double, <br />
`kids_i` double, <br />
`kids_r` double, <br />
`kids_u` double, <br />
`lsst_g` double, <br />
`lsst_i` double, <br />
`lsst_r` double, <br />
`lsst_u` double, <br />
`lsst_y` double, <br />
`lsst_z` double, <br />
`cfht_megacam_r` double, <br />
`cfht_megacam_u` double, <br />
`pan_starrs_i` double, <br />
`pan_starrs_z` double, <br />
`euclid_nisp_h` double, <br />
`euclid_nisp_j` double, <br />
`euclid_nisp_y` double, <br />
`euclid_vis` double, <br />
`2mass_h` double, <br />
`2mass_j` double, <br />
`2mass_ks` double, <br />
`sdss_r01` double, <br />
`l_gal` double, <br />
`b_gal` double, <br />
`mw_extinction` double, <br />
`2mass_h_odonnell_ext` double, <br />
`2mass_j_odonnell_ext` double, <br />
`2mass_ks_odonnell_ext` double, <br />
`blanco_decam_g_odonnell_ext` double, <br />
`blanco_decam_i_odonnell_ext` double, <br />
`blanco_decam_r_odonnell_ext` double, <br />
`blanco_decam_z_odonnell_ext` double, <br />
`cfht_megacam_r_odonnell_ext` double, <br />
`cfht_megacam_u_odonnell_ext` double, <br />
`euclid_nisp_h_odonnell_ext` double, <br />
`euclid_nisp_j_odonnell_ext` double, <br />
`euclid_nisp_y_odonnell_ext` double, <br />
`euclid_vis_odonnell_ext` double, <br />
`gaia_bp_odonnell_ext` double, <br />
`gaia_g_odonnell_ext` double, <br />
`gaia_rp_odonnell_ext` double, <br />
`jst_jpcam_g_odonnell_ext` double, <br />
`kids_g_odonnell_ext` double, <br />
`kids_i_odonnell_ext` double, <br />
`kids_r_odonnell_ext` double, <br />
`kids_u_odonnell_ext` double, <br />
`lsst_g_odonnell_ext` double, <br />
`lsst_i_odonnell_ext` double, <br />
`lsst_r_odonnell_ext` double, <br />
`lsst_u_odonnell_ext` double, <br />
`lsst_y_odonnell_ext` double, <br />
`lsst_z_odonnell_ext` double, <br />
`pan_starrs_i_odonnell_ext` double, <br />
`pan_starrs_z_odonnell_ext` double, <br />
`sdss_r01_odonnell_ext` double, <br />
`subaru_hsc_z_odonnell_ext` double, <br />
`blanco_decam_g_el_model1_odonnell_ext` double, <br />
`blanco_decam_g_el_model3_odonnell_ext` double, <br />
`blanco_decam_i_el_model1_odonnell_ext` double, <br />
`blanco_decam_i_el_model3_odonnell_ext` double, <br />
`blanco_decam_r_el_model1_odonnell_ext` double, <br />
`blanco_decam_r_el_model3_odonnell_ext` double, <br />
`blanco_decam_z_el_model1_odonnell_ext` double, <br />
`blanco_decam_z_el_model3_odonnell_ext` double, <br />
`gaia_bp_el_model1_odonnell_ext` double, <br />
`gaia_bp_el_model3_odonnell_ext` double, <br />
`gaia_g_el_model1_odonnell_ext` double, <br />
`gaia_g_el_model3_odonnell_ext` double, <br />
`gaia_rp_el_model1_odonnell_ext` double, <br />
`gaia_rp_el_model3_odonnell_ext` double, <br />
`subaru_hsc_z_el_model1_odonnell_ext` double, <br />
`subaru_hsc_z_el_model3_odonnell_ext` double, <br />
`jst_jpcam_g_el_model1_odonnell_ext` double, <br />
`jst_jpcam_g_el_model3_odonnell_ext` double, <br />
`kids_g_el_model1_odonnell_ext` double, <br />
`kids_g_el_model3_odonnell_ext` double, <br />
`kids_i_el_model1_odonnell_ext` double, <br />
`kids_i_el_model3_odonnell_ext` double, <br />
`kids_r_el_model1_odonnell_ext` double, <br />
`kids_r_el_model3_odonnell_ext` double, <br />
`kids_u_el_model1_odonnell_ext` double, <br />
`kids_u_el_model3_odonnell_ext` double, <br />
`lsst_g_el_model1_odonnell_ext` double, <br />
`lsst_g_el_model3_odonnell_ext` double, <br />
`lsst_i_el_model1_odonnell_ext` double, <br />
`lsst_i_el_model3_odonnell_ext` double, <br />
`lsst_r_el_model1_odonnell_ext` double, <br />
`lsst_r_el_model3_odonnell_ext` double, <br />
`lsst_u_el_model1_odonnell_ext` double, <br />
`lsst_u_el_model3_odonnell_ext` double, <br />
`lsst_y_el_model1_odonnell_ext` double, <br />
`lsst_y_el_model3_odonnell_ext` double, <br />
`lsst_z_el_model1_odonnell_ext` double, <br />
`lsst_z_el_model3_odonnell_ext` double, <br />
`cfht_megacam_r_el_model1_odonnell_ext` double, <br />
`cfht_megacam_r_el_model3_odonnell_ext` double, <br />
`cfht_megacam_u_el_model1_odonnell_ext` double, <br />
`cfht_megacam_u_el_model3_odonnell_ext` double, <br />
`pan_starrs_i_el_model1_odonnell_ext` double, <br />
`pan_starrs_i_el_model3_odonnell_ext` double, <br />
`pan_starrs_z_el_model1_odonnell_ext` double, <br />
`pan_starrs_z_el_model3_odonnell_ext` double, <br />
`euclid_nisp_h_el_model1_odonnell_ext` double, <br />
`euclid_nisp_h_el_model3_odonnell_ext` double, <br />
`euclid_nisp_j_el_model1_odonnell_ext` double, <br />
`euclid_nisp_j_el_model3_odonnell_ext` double, <br />
`euclid_nisp_y_el_model1_odonnell_ext` double, <br />
`euclid_nisp_y_el_model3_odonnell_ext` double, <br />
`euclid_vis_el_model1_odonnell_ext` double, <br />
`euclid_vis_el_model3_odonnell_ext` double, <br />
`2mass_h_el_model1_odonnell_ext` double, <br />
`2mass_h_el_model3_odonnell_ext` double, <br />
`2mass_j_el_model1_odonnell_ext` double, <br />
`2mass_j_el_model3_odonnell_ext` double, <br />
`2mass_ks_el_model1_odonnell_ext` double, <br />
`2mass_ks_el_model3_odonnell_ext` double, <br />
`sdss_r01_el_model1_odonnell_ext` double, <br />
`sdss_r01_el_model3_odonnell_ext` double, <br />
`dominant_shape` bigint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)', <br />
`bulge_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`disk_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`median_major_axis` double COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)', <br />
`scale_length` double COMMENT 'disc and bulge scalelength prior', <br />
`bulge_fraction` double COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)', <br />
`disk_scalelength` double COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)', <br />
`disk_nsersic` double COMMENT 'Sersic index of the disk component', <br />
`disk_r50` double COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678', <br />
`bulge_r50` double COMMENT 'half light radius of the bulge', <br />
`bulge_nsersic` double COMMENT 'Sersic index of the bulge component', <br />
`disk_ellipticity` double COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)', <br />
`bulge_ellipticity` double COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)', <br />
`disk_axis_ratio` double COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`bulge_axis_ratio` double COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',<br />
`disk_inclination_angle` double COMMENT '', <br />
`bulge_inclination_angle` double COMMENT '', <br />
`step` bigint, <br />
`random_index` double, <br />
`halo_id` bigint, <br />
`galaxy_id` bigint<br />
)<br />
CLUSTERED BY ( <br />
halo_id, <br />
galaxy_id<br />
) <br />
INTO 128 BUCKETS<br />
ROW FORMAT SERDE <br />
'org.apache.hadoop.hive.ql.io.orc.OrcSerde' <br />
STORED AS INPUTFORMAT <br />
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' <br />
OUTPUTFORMAT <br />
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'<br />
;<br />
<br />
INSERT OVERWRITE TABLE cosmohub.flagship_mock_1_9_2_c<br />
SELECT * FROM tallada.flagship_1_9_2;<br />
<br />
ANALYZE TABLE cosmohub.flagship_mock_1_9_2_c COMPUTE STATISTICS FOR COLUMNS;<br />
<br />
CREATE TABLE tallada.flagship_mock_1_9_2_c2(<br />
`kind` bigint, <br />
`luminosity_r01` double, <br />
`halo_x` double, <br />
`halo_y` double, <br />
`halo_z` double, <br />
`halo_vx` double, <br />
`halo_vy` double, <br />
`halo_vz` double, <br />
`halo_r` double, <br />
`true_redshift_halo` double, <br />
`halo_lm` double, <br />
`halo_n_sats` bigint, <br />
`n_gals` bigint, <br />
`abs_mag_r01` double, <br />
`abs_mag_r01_evolved` double, <br />
`luminosity_r01_evolved` double, <br />
`gr_restframe` double, <br />
`color_kind` int, <br />
`x_gal` double, <br />
`y_gal` double, <br />
`z_gal` double, <br />
`r_gal` double, <br />
`true_redshift_gal` double, <br />
`ra_gal` double, <br />
`dec_gal` double, <br />
`hpix_29_nest` bigint, <br />
`hpix_13_nest` bigint, <br />
`vx_gal` double, <br />
`vy_gal` double, <br />
`vz_gal` double, <br />
`vrad_gal` double, <br />
`delta_r` double, <br />
`observed_redshift_gal` double, <br />
`sed_ke` double, <br />
`gr_cosmos` double, <br />
`sed_cosmos` double, <br />
`ext_curve_cosmos` bigint, <br />
`ebv_cosmos` double, <br />
`is_within_cosmos` boolean, <br />
`cosmos_distance` double, <br />
`abs_mag_uv_dereddened` double, <br />
`log_ml_r01` double, <br />
`log_stellar_mass` double, <br />
`metallicity` double, <br />
`log_sfr` double, <br />
`logf_halpha_ext` double, <br />
`logf_halpha` double, <br />
`halpha_scatter` double, <br />
`loglum_halpha` double, <br />
`loglum_halpha_ext` double, <br />
`logf_dummy` double, <br />
`z_dummy` double, <br />
`logf_halpha_model1_ext` double, <br />
`logf_halpha_model1` double, <br />
`loglum_halpha_model1_ext` double, <br />
`loglum_halpha_model1` double, <br />
`logf_halpha_model3_ext` double, <br />
`logf_halpha_model3` double, <br />
`loglum_halpha_model3_ext` double, <br />
`loglum_halpha_model3` double, <br />
`logf_hbeta_model1_ext` double, <br />
`logf_hbeta_model1` double, <br />
`logf_o2_model1_ext` double, <br />
`logf_o2_model1` double, <br />
`logf_n2_model1_ext` double, <br />
`logf_n2_model1` double, <br />
`logf_o3_model1_ext` double, <br />
`logf_o3_model1` double, <br />
`logf_s2_model1_ext` double, <br />
`logf_s2_model1` double, <br />
`logf_hbeta_model3_ext` double, <br />
`logf_hbeta_model3` double, <br />
`logf_o2_model3_ext` double, <br />
`logf_o2_model3` double, <br />
`logf_n2_model3_ext` double, <br />
`logf_n2_model3` double, <br />
`logf_o3_model3_ext` double, <br />
`logf_o3_model3` double, <br />
`logf_s2_model3_ext` double, <br />
`logf_s2_model3` double, <br />
`blanco_decam_g` double, <br />
`blanco_decam_i` double, <br />
`blanco_decam_r` double, <br />
`blanco_decam_z` double, <br />
`gaia_bp` double, <br />
`gaia_g` double, <br />
`gaia_rp` double, <br />
`subaru_hsc_z` double, <br />
`jst_jpcam_g` double, <br />
`kids_g` double, <br />
`kids_i` double, <br />
`kids_r` double, <br />
`kids_u` double, <br />
`lsst_g` double, <br />
`lsst_i` double, <br />
`lsst_r` double, <br />
`lsst_u` double, <br />
`lsst_y` double, <br />
`lsst_z` double, <br />
`cfht_megacam_r` double, <br />
`cfht_megacam_u` double, <br />
`pan_starrs_i` double, <br />
`pan_starrs_z` double, <br />
`euclid_nisp_h` double, <br />
`euclid_nisp_j` double, <br />
`euclid_nisp_y` double, <br />
`euclid_vis` double, <br />
`2mass_h` double, <br />
`2mass_j` double, <br />
`2mass_ks` double, <br />
`sdss_r01` double, <br />
`l_gal` double, <br />
`b_gal` double, <br />
`mw_extinction` double, <br />
`2mass_h_odonnell_ext` double, <br />
`2mass_j_odonnell_ext` double, <br />
`2mass_ks_odonnell_ext` double, <br />
`blanco_decam_g_odonnell_ext` double, <br />
`blanco_decam_i_odonnell_ext` double, <br />
`blanco_decam_r_odonnell_ext` double, <br />
`blanco_decam_z_odonnell_ext` double, <br />
`cfht_megacam_r_odonnell_ext` double, <br />
`cfht_megacam_u_odonnell_ext` double, <br />
`euclid_nisp_h_odonnell_ext` double, <br />
`euclid_nisp_j_odonnell_ext` double, <br />
`euclid_nisp_y_odonnell_ext` double, <br />
`euclid_vis_odonnell_ext` double, <br />
`gaia_bp_odonnell_ext` double, <br />
`gaia_g_odonnell_ext` double, <br />
`gaia_rp_odonnell_ext` double, <br />
`jst_jpcam_g_odonnell_ext` double, <br />
`kids_g_odonnell_ext` double, <br />
`kids_i_odonnell_ext` double, <br />
`kids_r_odonnell_ext` double, <br />
`kids_u_odonnell_ext` double, <br />
`lsst_g_odonnell_ext` double, <br />
`lsst_i_odonnell_ext` double, <br />
`lsst_r_odonnell_ext` double, <br />
`lsst_u_odonnell_ext` double, <br />
`lsst_y_odonnell_ext` double, <br />
`lsst_z_odonnell_ext` double, <br />
`pan_starrs_i_odonnell_ext` double, <br />
`pan_starrs_z_odonnell_ext` double, <br />
`sdss_r01_odonnell_ext` double, <br />
`subaru_hsc_z_odonnell_ext` double, <br />
`blanco_decam_g_el_model1_odonnell_ext` double, <br />
`blanco_decam_g_el_model3_odonnell_ext` double, <br />
`blanco_decam_i_el_model1_odonnell_ext` double, <br />
`blanco_decam_i_el_model3_odonnell_ext` double, <br />
`blanco_decam_r_el_model1_odonnell_ext` double, <br />
`blanco_decam_r_el_model3_odonnell_ext` double, <br />
`blanco_decam_z_el_model1_odonnell_ext` double, <br />
`blanco_decam_z_el_model3_odonnell_ext` double, <br />
`gaia_bp_el_model1_odonnell_ext` double, <br />
`gaia_bp_el_model3_odonnell_ext` double, <br />
`gaia_g_el_model1_odonnell_ext` double, <br />
`gaia_g_el_model3_odonnell_ext` double, <br />
`gaia_rp_el_model1_odonnell_ext` double, <br />
`gaia_rp_el_model3_odonnell_ext` double, <br />
`subaru_hsc_z_el_model1_odonnell_ext` double, <br />
`subaru_hsc_z_el_model3_odonnell_ext` double, <br />
`jst_jpcam_g_el_model1_odonnell_ext` double, <br />
`jst_jpcam_g_el_model3_odonnell_ext` double, <br />
`kids_g_el_model1_odonnell_ext` double, <br />
`kids_g_el_model3_odonnell_ext` double, <br />
`kids_i_el_model1_odonnell_ext` double, <br />
`kids_i_el_model3_odonnell_ext` double, <br />
`kids_r_el_model1_odonnell_ext` double, <br />
`kids_r_el_model3_odonnell_ext` double, <br />
`kids_u_el_model1_odonnell_ext` double, <br />
`kids_u_el_model3_odonnell_ext` double, <br />
`lsst_g_el_model1_odonnell_ext` double, <br />
`lsst_g_el_model3_odonnell_ext` double, <br />
`lsst_i_el_model1_odonnell_ext` double, <br />
`lsst_i_el_model3_odonnell_ext` double, <br />
`lsst_r_el_model1_odonnell_ext` double, <br />
`lsst_r_el_model3_odonnell_ext` double, <br />
`lsst_u_el_model1_odonnell_ext` double, <br />
`lsst_u_el_model3_odonnell_ext` double, <br />
`lsst_y_el_model1_odonnell_ext` double, <br />
`lsst_y_el_model3_odonnell_ext` double, <br />
`lsst_z_el_model1_odonnell_ext` double, <br />
`lsst_z_el_model3_odonnell_ext` double, <br />
`cfht_megacam_r_el_model1_odonnell_ext` double, <br />
`cfht_megacam_r_el_model3_odonnell_ext` double, <br />
`cfht_megacam_u_el_model1_odonnell_ext` double, <br />
`cfht_megacam_u_el_model3_odonnell_ext` double, <br />
`pan_starrs_i_el_model1_odonnell_ext` double, <br />
`pan_starrs_i_el_model3_odonnell_ext` double, <br />
`pan_starrs_z_el_model1_odonnell_ext` double, <br />
`pan_starrs_z_el_model3_odonnell_ext` double, <br />
`euclid_nisp_h_el_model1_odonnell_ext` double, <br />
`euclid_nisp_h_el_model3_odonnell_ext` double, <br />
`euclid_nisp_j_el_model1_odonnell_ext` double, <br />
`euclid_nisp_j_el_model3_odonnell_ext` double, <br />
`euclid_nisp_y_el_model1_odonnell_ext` double, <br />
`euclid_nisp_y_el_model3_odonnell_ext` double, <br />
`euclid_vis_el_model1_odonnell_ext` double, <br />
`euclid_vis_el_model3_odonnell_ext` double, <br />
`2mass_h_el_model1_odonnell_ext` double, <br />
`2mass_h_el_model3_odonnell_ext` double, <br />
`2mass_j_el_model1_odonnell_ext` double, <br />
`2mass_j_el_model3_odonnell_ext` double, <br />
`2mass_ks_el_model1_odonnell_ext` double, <br />
`2mass_ks_el_model3_odonnell_ext` double, <br />
`sdss_r01_el_model1_odonnell_ext` double, <br />
`sdss_r01_el_model3_odonnell_ext` double, <br />
`dominant_shape` bigint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)', <br />
`bulge_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`disk_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`median_major_axis` double COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)', <br />
`scale_length` double COMMENT 'disc and bulge scalelength prior', <br />
`bulge_fraction` double COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)', <br />
`disk_scalelength` double COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)', <br />
`disk_nsersic` double COMMENT 'Sersic index of the disk component', <br />
`disk_r50` double COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678', <br />
`bulge_r50` double COMMENT 'half light radius of the bulge', <br />
`bulge_nsersic` double COMMENT 'Sersic index of the bulge component', <br />
`disk_ellipticity` double COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)', <br />
`bulge_ellipticity` double COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)', <br />
`disk_axis_ratio` double COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`bulge_axis_ratio` double COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`disk_inclination_angle` double COMMENT '', <br />
`bulge_inclination_angle` double COMMENT '', <br />
`random_index` double, <br />
`halo_id` bigint, <br />
`galaxy_id` bigint<br />
)<br />
PARTITIONED BY (<br />
`step` smallint<br />
)<br />
CLUSTERED BY (<br />
hpix_13_nest<br />
)<br />
SORTED BY (<br />
hpix_13_nest ASC<br />
)<br />
INTO 128 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
INSERT OVERWRITE TABLE tallada.flagship_mock_1_9_2_c2<br />
SELECT<br />
kind, luminosity_r01, halo_x, halo_y, halo_z, halo_vx, halo_vy, halo_vz, halo_r, true_redshift_halo, halo_lm, halo_n_sats, n_gals,<br />
abs_mag_r01, abs_mag_r01_evolved, luminosity_r01_evolved, gr_restframe, color_kind, x_gal, y_gal, z_gal, r_gal, true_redshift_gal,<br />
ra_gal, dec_gal, hpix_nest, SHIFTRIGHT(hpix_nest, (29-13)*2), vx_gal, vy_gal, vz_gal, vrad_gal, delta_r, observed_redshift_gal, sed_ke, <br />
gr_cosmos, sed_cosmos, ext_curve_cosmos, ebv_cosmos, is_within_cosmos, cosmos_distance, abs_mag_uv_dereddened, log_ml_r01, <br />
log_stellar_mass, metallicity, log_sfr, logf_halpha_ext, logf_halpha, halpha_scatter, loglum_halpha, loglum_halpha_ext, logf_dummy, <br />
z_dummy, logf_halpha_model1_ext, logf_halpha_model1, loglum_halpha_model1_ext, loglum_halpha_model1, logf_halpha_model3_ext, <br />
logf_halpha_model3, loglum_halpha_model3_ext, loglum_halpha_model3, logf_hbeta_model1_ext, logf_hbeta_model1, logf_o2_model1_ext, <br />
logf_o2_model1, logf_n2_model1_ext, logf_n2_model1, logf_o3_model1_ext, logf_o3_model1, logf_s2_model1_ext, logf_s2_model1, <br />
logf_hbeta_model3_ext, logf_hbeta_model3, logf_o2_model3_ext, logf_o2_model3, logf_n2_model3_ext, logf_n2_model3, logf_o3_model3_ext, <br />
logf_o3_model3, logf_s2_model3_ext, logf_s2_model3, blanco_decam_g, blanco_decam_i, blanco_decam_r, blanco_decam_z, gaia_bp, gaia_g, <br />
gaia_rp, subaru_hsc_z, jst_jpcam_g, kids_g, kids_i, kids_r, kids_u, lsst_g, lsst_i, lsst_r, lsst_u, lsst_y, lsst_z, cfht_megacam_r, <br />
cfht_megacam_u, pan_starrs_i, pan_starrs_z, euclid_nisp_h, euclid_nisp_j, euclid_nisp_y, euclid_vis, 2mass_h, 2mass_j, 2mass_ks, <br />
sdss_r01, l_gal, b_gal, mw_extinction, 2mass_h_odonnell_ext, 2mass_j_odonnell_ext, 2mass_ks_odonnell_ext, blanco_decam_g_odonnell_ext, <br />
blanco_decam_i_odonnell_ext, blanco_decam_r_odonnell_ext, blanco_decam_z_odonnell_ext, cfht_megacam_r_odonnell_ext, <br />
cfht_megacam_u_odonnell_ext, euclid_nisp_h_odonnell_ext, euclid_nisp_j_odonnell_ext, euclid_nisp_y_odonnell_ext, euclid_vis_odonnell_ext,<br />
gaia_bp_odonnell_ext, gaia_g_odonnell_ext, gaia_rp_odonnell_ext, jst_jpcam_g_odonnell_ext, kids_g_odonnell_ext, kids_i_odonnell_ext, <br />
kids_r_odonnell_ext, kids_u_odonnell_ext, lsst_g_odonnell_ext, lsst_i_odonnell_ext, lsst_r_odonnell_ext, lsst_u_odonnell_ext, <br />
lsst_y_odonnell_ext, lsst_z_odonnell_ext, pan_starrs_i_odonnell_ext, pan_starrs_z_odonnell_ext, sdss_r01_odonnell_ext, <br />
subaru_hsc_z_odonnell_ext, blanco_decam_g_el_model1_odonnell_ext, blanco_decam_g_el_model3_odonnell_ext, <br />
blanco_decam_i_el_model1_odonnell_ext, blanco_decam_i_el_model3_odonnell_ext, blanco_decam_r_el_model1_odonnell_ext, <br />
blanco_decam_r_el_model3_odonnell_ext, blanco_decam_z_el_model1_odonnell_ext, blanco_decam_z_el_model3_odonnell_ext, <br />
gaia_bp_el_model1_odonnell_ext, gaia_bp_el_model3_odonnell_ext, gaia_g_el_model1_odonnell_ext, gaia_g_el_model3_odonnell_ext, <br />
gaia_rp_el_model1_odonnell_ext, gaia_rp_el_model3_odonnell_ext, subaru_hsc_z_el_model1_odonnell_ext, subaru_hsc_z_el_model3_odonnell_ext, <br />
jst_jpcam_g_el_model1_odonnell_ext, jst_jpcam_g_el_model3_odonnell_ext, kids_g_el_model1_odonnell_ext, kids_g_el_model3_odonnell_ext, <br />
kids_i_el_model1_odonnell_ext, kids_i_el_model3_odonnell_ext, kids_r_el_model1_odonnell_ext, kids_r_el_model3_odonnell_ext, <br />
kids_u_el_model1_odonnell_ext, kids_u_el_model3_odonnell_ext, lsst_g_el_model1_odonnell_ext, lsst_g_el_model3_odonnell_ext, <br />
lsst_i_el_model1_odonnell_ext, lsst_i_el_model3_odonnell_ext, lsst_r_el_model1_odonnell_ext, lsst_r_el_model3_odonnell_ext, <br />
lsst_u_el_model1_odonnell_ext, lsst_u_el_model3_odonnell_ext, lsst_y_el_model1_odonnell_ext, lsst_y_el_model3_odonnell_ext, <br />
lsst_z_el_model1_odonnell_ext, lsst_z_el_model3_odonnell_ext, cfht_megacam_r_el_model1_odonnell_ext, <br />
cfht_megacam_r_el_model3_odonnell_ext, cfht_megacam_u_el_model1_odonnell_ext, cfht_megacam_u_el_model3_odonnell_ext, <br />
pan_starrs_i_el_model1_odonnell_ext, pan_starrs_i_el_model3_odonnell_ext, pan_starrs_z_el_model1_odonnell_ext, <br />
pan_starrs_z_el_model3_odonnell_ext, euclid_nisp_h_el_model1_odonnell_ext, euclid_nisp_h_el_model3_odonnell_ext, <br />
euclid_nisp_j_el_model1_odonnell_ext, euclid_nisp_j_el_model3_odonnell_ext, euclid_nisp_y_el_model1_odonnell_ext, <br />
euclid_nisp_y_el_model3_odonnell_ext, euclid_vis_el_model1_odonnell_ext, euclid_vis_el_model3_odonnell_ext, <br />
2mass_h_el_model1_odonnell_ext, 2mass_h_el_model3_odonnell_ext, 2mass_j_el_model1_odonnell_ext, 2mass_j_el_model3_odonnell_ext,<br />
2mass_ks_el_model1_odonnell_ext, 2mass_ks_el_model3_odonnell_ext, sdss_r01_el_model1_odonnell_ext, sdss_r01_el_model3_odonnell_ext, <br />
dominant_shape, bulge_angle, disk_angle, median_major_axis, scale_length, bulge_fraction, disk_scalelength, disk_nsersic, disk_r50, <br />
bulge_r50, bulge_nsersic, disk_ellipticity, bulge_ellipticity, disk_axis_ratio, bulge_axis_ratio, disk_inclination_angle,<br />
bulge_inclination_angle, random_index, halo_id, galaxy_id, step<br />
FROM cosmohub.flagship_mock_1_9_2_c<br />
<br />
CREATE TABLE tallada.flagship_mock_1_9_2_s(<br />
`kind` bigint, <br />
`luminosity_r01` double, <br />
`halo_x` double, <br />
`halo_y` double, <br />
`halo_z` double, <br />
`halo_vx` double, <br />
`halo_vy` double, <br />
`halo_vz` double, <br />
`halo_r` double, <br />
`true_redshift_halo` double, <br />
`halo_lm` double, <br />
`halo_n_sats` bigint, <br />
`n_gals` bigint, <br />
`abs_mag_r01` double, <br />
`abs_mag_r01_evolved` double, <br />
`luminosity_r01_evolved` double, <br />
`gr_restframe` double, <br />
`color_kind` int, <br />
`x_gal` double, <br />
`y_gal` double, <br />
`z_gal` double, <br />
`r_gal` double, <br />
`true_redshift_gal` double, <br />
`ra_gal` double, <br />
`dec_gal` double, <br />
`ra_mag_gal` double, <br />
`dec_mag_gal` double, <br />
`kappa` double,<br />
`gamma1` double,<br />
`gamma2` double,<br />
`hpix_29_nest` bigint, <br />
`hpix_13_nest` bigint, <br />
`vx_gal` double, <br />
`vy_gal` double, <br />
`vz_gal` double, <br />
`vrad_gal` double, <br />
`delta_r` double, <br />
`observed_redshift_gal` double, <br />
`sed_ke` double, <br />
`gr_cosmos` double, <br />
`sed_cosmos` double, <br />
`ext_curve_cosmos` bigint, <br />
`ebv_cosmos` double, <br />
`is_within_cosmos` boolean, <br />
`cosmos_distance` double, <br />
`abs_mag_uv_dereddened` double, <br />
`log_ml_r01` double, <br />
`log_stellar_mass` double, <br />
`metallicity` double, <br />
`log_sfr` double, <br />
`logf_halpha_ext` double, <br />
`logf_halpha` double, <br />
`halpha_scatter` double, <br />
`loglum_halpha` double, <br />
`loglum_halpha_ext` double, <br />
`logf_dummy` double, <br />
`z_dummy` double, <br />
`logf_halpha_model1_ext` double, <br />
`logf_halpha_model1` double, <br />
`loglum_halpha_model1_ext` double, <br />
`loglum_halpha_model1` double, <br />
`logf_halpha_model3_ext` double, <br />
`logf_halpha_model3` double, <br />
`loglum_halpha_model3_ext` double, <br />
`loglum_halpha_model3` double, <br />
`logf_hbeta_model1_ext` double, <br />
`logf_hbeta_model1` double, <br />
`logf_o2_model1_ext` double, <br />
`logf_o2_model1` double, <br />
`logf_n2_model1_ext` double, <br />
`logf_n2_model1` double, <br />
`logf_o3_model1_ext` double, <br />
`logf_o3_model1` double, <br />
`logf_s2_model1_ext` double, <br />
`logf_s2_model1` double, <br />
`logf_hbeta_model3_ext` double, <br />
`logf_hbeta_model3` double, <br />
`logf_o2_model3_ext` double, <br />
`logf_o2_model3` double, <br />
`logf_n2_model3_ext` double, <br />
`logf_n2_model3` double, <br />
`logf_o3_model3_ext` double, <br />
`logf_o3_model3` double, <br />
`logf_s2_model3_ext` double, <br />
`logf_s2_model3` double, <br />
`blanco_decam_g` double, <br />
`blanco_decam_i` double, <br />
`blanco_decam_r` double, <br />
`blanco_decam_z` double, <br />
`gaia_bp` double, <br />
`gaia_g` double, <br />
`gaia_rp` double, <br />
`subaru_hsc_z` double, <br />
`jst_jpcam_g` double, <br />
`kids_g` double, <br />
`kids_i` double, <br />
`kids_r` double, <br />
`kids_u` double, <br />
`lsst_g` double, <br />
`lsst_i` double, <br />
`lsst_r` double, <br />
`lsst_u` double, <br />
`lsst_y` double, <br />
`lsst_z` double, <br />
`cfht_megacam_r` double, <br />
`cfht_megacam_u` double, <br />
`pan_starrs_i` double, <br />
`pan_starrs_z` double, <br />
`euclid_nisp_h` double, <br />
`euclid_nisp_j` double, <br />
`euclid_nisp_y` double, <br />
`euclid_vis` double, <br />
`2mass_h` double, <br />
`2mass_j` double, <br />
`2mass_ks` double, <br />
`sdss_r01` double, <br />
`l_gal` double, <br />
`b_gal` double, <br />
`mw_extinction` double, <br />
`2mass_h_odonnell_ext` double, <br />
`2mass_j_odonnell_ext` double, <br />
`2mass_ks_odonnell_ext` double, <br />
`blanco_decam_g_odonnell_ext` double, <br />
`blanco_decam_i_odonnell_ext` double, <br />
`blanco_decam_r_odonnell_ext` double, <br />
`blanco_decam_z_odonnell_ext` double, <br />
`cfht_megacam_r_odonnell_ext` double, <br />
`cfht_megacam_u_odonnell_ext` double, <br />
`euclid_nisp_h_odonnell_ext` double, <br />
`euclid_nisp_j_odonnell_ext` double, <br />
`euclid_nisp_y_odonnell_ext` double, <br />
`euclid_vis_odonnell_ext` double, <br />
`gaia_bp_odonnell_ext` double, <br />
`gaia_g_odonnell_ext` double, <br />
`gaia_rp_odonnell_ext` double, <br />
`jst_jpcam_g_odonnell_ext` double, <br />
`kids_g_odonnell_ext` double, <br />
`kids_i_odonnell_ext` double, <br />
`kids_r_odonnell_ext` double, <br />
`kids_u_odonnell_ext` double, <br />
`lsst_g_odonnell_ext` double, <br />
`lsst_i_odonnell_ext` double, <br />
`lsst_r_odonnell_ext` double, <br />
`lsst_u_odonnell_ext` double, <br />
`lsst_y_odonnell_ext` double, <br />
`lsst_z_odonnell_ext` double, <br />
`pan_starrs_i_odonnell_ext` double, <br />
`pan_starrs_z_odonnell_ext` double, <br />
`sdss_r01_odonnell_ext` double, <br />
`subaru_hsc_z_odonnell_ext` double, <br />
`blanco_decam_g_el_model1_odonnell_ext` double, <br />
`blanco_decam_g_el_model3_odonnell_ext` double, <br />
`blanco_decam_i_el_model1_odonnell_ext` double, <br />
`blanco_decam_i_el_model3_odonnell_ext` double, <br />
`blanco_decam_r_el_model1_odonnell_ext` double, <br />
`blanco_decam_r_el_model3_odonnell_ext` double, <br />
`blanco_decam_z_el_model1_odonnell_ext` double, <br />
`blanco_decam_z_el_model3_odonnell_ext` double, <br />
`gaia_bp_el_model1_odonnell_ext` double, <br />
`gaia_bp_el_model3_odonnell_ext` double, <br />
`gaia_g_el_model1_odonnell_ext` double, <br />
`gaia_g_el_model3_odonnell_ext` double, <br />
`gaia_rp_el_model1_odonnell_ext` double, <br />
`gaia_rp_el_model3_odonnell_ext` double, <br />
`subaru_hsc_z_el_model1_odonnell_ext` double, <br />
`subaru_hsc_z_el_model3_odonnell_ext` double, <br />
`jst_jpcam_g_el_model1_odonnell_ext` double, <br />
`jst_jpcam_g_el_model3_odonnell_ext` double, <br />
`kids_g_el_model1_odonnell_ext` double, <br />
`kids_g_el_model3_odonnell_ext` double, <br />
`kids_i_el_model1_odonnell_ext` double, <br />
`kids_i_el_model3_odonnell_ext` double, <br />
`kids_r_el_model1_odonnell_ext` double, <br />
`kids_r_el_model3_odonnell_ext` double, <br />
`kids_u_el_model1_odonnell_ext` double, <br />
`kids_u_el_model3_odonnell_ext` double, <br />
`lsst_g_el_model1_odonnell_ext` double, <br />
`lsst_g_el_model3_odonnell_ext` double, <br />
`lsst_i_el_model1_odonnell_ext` double, <br />
`lsst_i_el_model3_odonnell_ext` double, <br />
`lsst_r_el_model1_odonnell_ext` double, <br />
`lsst_r_el_model3_odonnell_ext` double, <br />
`lsst_u_el_model1_odonnell_ext` double, <br />
`lsst_u_el_model3_odonnell_ext` double, <br />
`lsst_y_el_model1_odonnell_ext` double, <br />
`lsst_y_el_model3_odonnell_ext` double, <br />
`lsst_z_el_model1_odonnell_ext` double, <br />
`lsst_z_el_model3_odonnell_ext` double, <br />
`cfht_megacam_r_el_model1_odonnell_ext` double, <br />
`cfht_megacam_r_el_model3_odonnell_ext` double, <br />
`cfht_megacam_u_el_model1_odonnell_ext` double, <br />
`cfht_megacam_u_el_model3_odonnell_ext` double, <br />
`pan_starrs_i_el_model1_odonnell_ext` double, <br />
`pan_starrs_i_el_model3_odonnell_ext` double, <br />
`pan_starrs_z_el_model1_odonnell_ext` double, <br />
`pan_starrs_z_el_model3_odonnell_ext` double, <br />
`euclid_nisp_h_el_model1_odonnell_ext` double, <br />
`euclid_nisp_h_el_model3_odonnell_ext` double, <br />
`euclid_nisp_j_el_model1_odonnell_ext` double, <br />
`euclid_nisp_j_el_model3_odonnell_ext` double, <br />
`euclid_nisp_y_el_model1_odonnell_ext` double, <br />
`euclid_nisp_y_el_model3_odonnell_ext` double, <br />
`euclid_vis_el_model1_odonnell_ext` double, <br />
`euclid_vis_el_model3_odonnell_ext` double, <br />
`2mass_h_el_model1_odonnell_ext` double, <br />
`2mass_h_el_model3_odonnell_ext` double, <br />
`2mass_j_el_model1_odonnell_ext` double, <br />
`2mass_j_el_model3_odonnell_ext` double, <br />
`2mass_ks_el_model1_odonnell_ext` double, <br />
`2mass_ks_el_model3_odonnell_ext` double, <br />
`sdss_r01_el_model1_odonnell_ext` double, <br />
`sdss_r01_el_model3_odonnell_ext` double, <br />
`dominant_shape` bigint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)', <br />
`bulge_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`disk_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`median_major_axis` double COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)', <br />
`scale_length` double COMMENT 'disc and bulge scalelength prior', <br />
`bulge_fraction` double COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)', <br />
`disk_scalelength` double COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)', <br />
`disk_nsersic` double COMMENT 'Sersic index of the disk component', <br />
`disk_r50` double COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678', <br />
`bulge_r50` double COMMENT 'half light radius of the bulge', <br />
`bulge_nsersic` double COMMENT 'Sersic index of the bulge component', <br />
`disk_ellipticity` double COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)', <br />
`bulge_ellipticity` double COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)', <br />
`disk_axis_ratio` double COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`bulge_axis_ratio` double COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`disk_inclination_angle` double COMMENT '', <br />
`bulge_inclination_angle` double COMMENT '', <br />
`random_index` double, <br />
`halo_id` bigint, <br />
`galaxy_id` bigint<br />
)<br />
PARTITIONED BY (<br />
`step` smallint<br />
)<br />
CLUSTERED BY (<br />
hpix_13_nest<br />
)<br />
SORTED BY (<br />
hpix_13_nest ASC<br />
)<br />
INTO 128 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
cat << EOF > lensing_step.sql<br />
SET hive.auto.convert.join.noconditionaltask.size=1;<br />
INSERT OVERWRITE TABLE tallada.flagship_mock_1_9_2_s PARTITION (step=${STEP})<br />
SELECT<br />
m.kind, m.luminosity_r01, m.halo_x, m.halo_y, m.halo_z, m.halo_vx, m.halo_vy, m.halo_vz, m.halo_r, m.true_redshift_halo, m.halo_lm,<br />
m.halo_n_sats, m.n_gals, m.abs_mag_r01, m.abs_mag_r01_evolved, m.luminosity_r01_evolved, m.gr_restframe, m.color_kind, m.x_gal,<br />
m.y_gal, m.z_gal, m.r_gal, m.true_redshift_gal, m.ra_gal, m.dec_gal, magpos.ra_mag, magpos.dec_mag, m.kappa, m.gamma1, m.gamma2,<br />
m.hpix_29_nest, m.hpix_13_nest, m.vx_gal, m.vy_gal, m.vz_gal, m.vrad_gal, m.delta_r, m.observed_redshift_gal, m.sed_ke, m.gr_cosmos, <br />
m.sed_cosmos, m.ext_curve_cosmos, m.ebv_cosmos, m.is_within_cosmos, m.cosmos_distance, m.abs_mag_uv_dereddened, m.log_ml_r01,<br />
m.log_stellar_mass, m.metallicity, m.log_sfr, m.logf_halpha_ext, m.logf_halpha, m.halpha_scatter, m.loglum_halpha, m.loglum_halpha_ext, <br />
m.logf_dummy, m.z_dummy, m.logf_halpha_model1_ext, m.logf_halpha_model1, m.loglum_halpha_model1_ext, m.loglum_halpha_model1, <br />
m.logf_halpha_model3_ext, m.logf_halpha_model3, m.loglum_halpha_model3_ext, m.loglum_halpha_model3, m.logf_hbeta_model1_ext, <br />
m.logf_hbeta_model1, m.logf_o2_model1_ext, m.logf_o2_model1, m.logf_n2_model1_ext, m.logf_n2_model1, m.logf_o3_model1_ext, <br />
m.logf_o3_model1, m.logf_s2_model1_ext, m.logf_s2_model1, m.logf_hbeta_model3_ext, m.logf_hbeta_model3, m.logf_o2_model3_ext, <br />
m.logf_o2_model3, m.logf_n2_model3_ext, m.logf_n2_model3, m.logf_o3_model3_ext, m.logf_o3_model3, m.logf_s2_model3_ext,<br />
m.logf_s2_model3, m.blanco_decam_g, m.blanco_decam_i, m.blanco_decam_r, m.blanco_decam_z, m.gaia_bp, m.gaia_g, m.gaia_rp, <br />
m.subaru_hsc_z, m.jst_jpcam_g, m.kids_g, m.kids_i, m.kids_r, m.kids_u, m.lsst_g, m.lsst_i, m.lsst_r, m.lsst_u, m.lsst_y, m.lsst_z, <br />
m.cfht_megacam_r, m.cfht_megacam_u, m.pan_starrs_i, m.pan_starrs_z, m.euclid_nisp_h, m.euclid_nisp_j, m.euclid_nisp_y, m.euclid_vis, <br />
m.2mass_h, m.2mass_j, m.2mass_ks, m.sdss_r01, m.l_gal, m.b_gal,mw_extinction, m.2mass_h_odonnell_ext, m.2mass_j_odonnell_ext, <br />
m.2mass_ks_odonnell_ext, m.blanco_decam_g_odonnell_ext, m.blanco_decam_i_odonnell_ext, m.blanco_decam_r_odonnell_ext, <br />
m.blanco_decam_z_odonnell_ext, m.cfht_megacam_r_odonnell_ext, m.cfht_megacam_u_odonnell_ext, m.euclid_nisp_h_odonnell_ext, <br />
m.euclid_nisp_j_odonnell_ext, m.euclid_nisp_y_odonnell_ext, m.euclid_vis_odonnell_ext, m.gaia_bp_odonnell_ext, m.gaia_g_odonnell_ext, <br />
m.gaia_rp_odonnell_ext, m.jst_jpcam_g_odonnell_ext, m.kids_g_odonnell_ext, m.kids_i_odonnell_ext, m.kids_r_odonnell_ext, <br />
m.kids_u_odonnell_ext, m.lsst_g_odonnell_ext, m.lsst_i_odonnell_ext, m.lsst_r_odonnell_ext, m.lsst_u_odonnell_ext, <br />
m.lsst_y_odonnell_ext, m.lsst_z_odonnell_ext, m.pan_starrs_i_odonnell_ext, m.pan_starrs_z_odonnell_ext, m.sdss_r01_odonnell_ext, <br />
m.subaru_hsc_z_odonnell_ext, m.blanco_decam_g_el_model1_odonnell_ext, m.blanco_decam_g_el_model3_odonnell_ext, <br />
m.blanco_decam_i_el_model1_odonnell_ext, m.blanco_decam_i_el_model3_odonnell_ext, m.blanco_decam_r_el_model1_odonnell_ext, <br />
m.blanco_decam_r_el_model3_odonnell_ext, m.blanco_decam_z_el_model1_odonnell_ext, m.blanco_decam_z_el_model3_odonnell_ext,<br />
m.gaia_bp_el_model1_odonnell_ext, m.gaia_bp_el_model3_odonnell_ext, m.gaia_g_el_model1_odonnell_ext, m.gaia_g_el_model3_odonnell_ext, <br />
m.gaia_rp_el_model1_odonnell_ext, m.gaia_rp_el_model3_odonnell_ext, m.subaru_hsc_z_el_model1_odonnell_ext, <br />
m.subaru_hsc_z_el_model3_odonnell_ext, m.jst_jpcam_g_el_model1_odonnell_ext, m.jst_jpcam_g_el_model3_odonnell_ext,<br />
m.kids_g_el_model1_odonnell_ext, m.kids_g_el_model3_odonnell_ext, m.kids_i_el_model1_odonnell_ext, m.kids_i_el_model3_odonnell_ext, <br />
m.kids_r_el_model1_odonnell_ext, m.kids_r_el_model3_odonnell_ext, m.kids_u_el_model1_odonnell_ext, m.kids_u_el_model3_odonnell_ext, <br />
m.lsst_g_el_model1_odonnell_ext, m.lsst_g_el_model3_odonnell_ext, m.lsst_i_el_model1_odonnell_ext, m.lsst_i_el_model3_odonnell_ext,<br />
m.lsst_r_el_model1_odonnell_ext, m.lsst_r_el_model3_odonnell_ext, m.lsst_u_el_model1_odonnell_ext, m.lsst_u_el_model3_odonnell_ext, <br />
m.lsst_y_el_model1_odonnell_ext, m.lsst_y_el_model3_odonnell_ext, m.lsst_z_el_model1_odonnell_ext, m.lsst_z_el_model3_odonnell_ext, <br />
m.cfht_megacam_r_el_model1_odonnell_ext, m.cfht_megacam_r_el_model3_odonnell_ext, m.cfht_megacam_u_el_model1_odonnell_ext,<br />
m.cfht_megacam_u_el_model3_odonnell_ext, m.pan_starrs_i_el_model1_odonnell_ext, m.pan_starrs_i_el_model3_odonnell_ext, <br />
m.pan_starrs_z_el_model1_odonnell_ext, m.pan_starrs_z_el_model3_odonnell_ext, m.euclid_nisp_h_el_model1_odonnell_ext, <br />
m.euclid_nisp_h_el_model3_odonnell_ext, m.euclid_nisp_j_el_model1_odonnell_ext, m.euclid_nisp_j_el_model3_odonnell_ext, <br />
m.euclid_nisp_y_el_model1_odonnell_ext, m.euclid_nisp_y_el_model3_odonnell_ext, m.euclid_vis_el_model1_odonnell_ext, <br />
m.euclid_vis_el_model3_odonnell_ext, m.2mass_h_el_model1_odonnell_ext, m.2mass_h_el_model3_odonnell_ext, <br />
m.2mass_j_el_model1_odonnell_ext, m.2mass_j_el_model3_odonnell_ext, m.2mass_ks_el_model1_odonnell_ext, <br />
m.2mass_ks_el_model3_odonnell_ext, m.sdss_r01_el_model1_odonnell_ext, m.sdss_r01_el_model3_odonnell_ext, m.dominant_shape, <br />
m.bulge_angle, m.disk_angle, m.median_major_axis, m.scale_length, m.bulge_fraction, m.disk_scalelength, m.disk_nsersic, m.disk_r50,<br />
m.bulge_r50, m.bulge_nsersic, m.disk_ellipticity, m.bulge_ellipticity, m.disk_axis_ratio, m.bulge_axis_ratio, m.disk_inclination_angle, <br />
m.bulge_inclination_angle, m.random_index, m.halo_id, m.galaxy_id<br />
FROM (<br />
SELECT <br />
m.kind, m.luminosity_r01, m.halo_x, m.halo_y, m.halo_z, m.halo_vx, m.halo_vy, m.halo_vz, m.halo_r, m.true_redshift_halo, m.halo_lm, <br />
m.halo_n_sats, m.n_gals, m.abs_mag_r01, m.abs_mag_r01_evolved, m.luminosity_r01_evolved, m.gr_restframe, m.color_kind, m.x_gal, <br />
m.y_gal, m.z_gal, m.r_gal, m.true_redshift_gal, m.ra_gal, m.dec_gal, l.defl_0 as kappa, l.shear_1 as gamma1, l.shear_2 as gamma2,<br />
m.hpix_29_nest, m.hpix_13_nest, m.vx_gal, m.vy_gal, m.vz_gal, m.vrad_gal, m.delta_r, m.observed_redshift_gal, m.sed_ke, m.gr_cosmos, <br />
m.sed_cosmos, m.ext_curve_cosmos, m.ebv_cosmos, m.is_within_cosmos, m.cosmos_distance, m.abs_mag_uv_dereddened, m.log_ml_r01, <br />
m.log_stellar_mass, m.metallicity, m.log_sfr, m.logf_halpha_ext, m.logf_halpha, m.halpha_scatter, m.loglum_halpha, <br />
m.loglum_halpha_ext, m.logf_dummy, m.z_dummy, m.logf_halpha_model1_ext, m.logf_halpha_model1, m.loglum_halpha_model1_ext, <br />
m.loglum_halpha_model1, m.logf_halpha_model3_ext, m.logf_halpha_model3, m.loglum_halpha_model3_ext, m.loglum_halpha_model3, <br />
m.logf_hbeta_model1_ext, m.logf_hbeta_model1, m.logf_o2_model1_ext, m.logf_o2_model1, m.logf_n2_model1_ext, m.logf_n2_model1,<br />
m.logf_o3_model1_ext, m.logf_o3_model1, m.logf_s2_model1_ext, m.logf_s2_model1, m.logf_hbeta_model3_ext, m.logf_hbeta_model3, <br />
m.logf_o2_model3_ext, m.logf_o2_model3, m.logf_n2_model3_ext, m.logf_n2_model3, m.logf_o3_model3_ext, m.logf_o3_model3, <br />
m.logf_s2_model3_ext, m.logf_s2_model3, m.blanco_decam_g, m.blanco_decam_i, m.blanco_decam_r, m.blanco_decam_z, m.gaia_bp, m.gaia_g, <br />
m.gaia_rp, m.subaru_hsc_z, m.jst_jpcam_g, m.kids_g, m.kids_i, m.kids_r, m.kids_u, m.lsst_g, m.lsst_i, m.lsst_r, m.lsst_u, m.lsst_y, <br />
m.lsst_z, m.cfht_megacam_r, m.cfht_megacam_u, m.pan_starrs_i, m.pan_starrs_z, m.euclid_nisp_h, m.euclid_nisp_j, m.euclid_nisp_y, <br />
m.euclid_vis, m.2mass_h, m.2mass_j, m.2mass_ks, m.sdss_r01, m.l_gal, m.b_gal,mw_extinction, m.2mass_h_odonnell_ext, <br />
m.2mass_j_odonnell_ext, m.2mass_ks_odonnell_ext, m.blanco_decam_g_odonnell_ext, m.blanco_decam_i_odonnell_ext, <br />
m.blanco_decam_r_odonnell_ext, m.blanco_decam_z_odonnell_ext, m.cfht_megacam_r_odonnell_ext, m.cfht_megacam_u_odonnell_ext,<br />
m.euclid_nisp_h_odonnell_ext, m.euclid_nisp_j_odonnell_ext, m.euclid_nisp_y_odonnell_ext, m.euclid_vis_odonnell_ext, <br />
m.gaia_bp_odonnell_ext, m.gaia_g_odonnell_ext, m.gaia_rp_odonnell_ext, m.jst_jpcam_g_odonnell_ext, m.kids_g_odonnell_ext, <br />
m.kids_i_odonnell_ext, m.kids_r_odonnell_ext, m.kids_u_odonnell_ext, m.lsst_g_odonnell_ext, m.lsst_i_odonnell_ext, <br />
m.lsst_r_odonnell_ext, m.lsst_u_odonnell_ext, m.lsst_y_odonnell_ext, m.lsst_z_odonnell_ext, m.pan_starrs_i_odonnell_ext, <br />
m.pan_starrs_z_odonnell_ext, m.sdss_r01_odonnell_ext, m.subaru_hsc_z_odonnell_ext, m.blanco_decam_g_el_model1_odonnell_ext, <br />
m.blanco_decam_g_el_model3_odonnell_ext, m.blanco_decam_i_el_model1_odonnell_ext, m.blanco_decam_i_el_model3_odonnell_ext, <br />
m.blanco_decam_r_el_model1_odonnell_ext, m.blanco_decam_r_el_model3_odonnell_ext, m.blanco_decam_z_el_model1_odonnell_ext, <br />
m.blanco_decam_z_el_model3_odonnell_ext, m.gaia_bp_el_model1_odonnell_ext, m.gaia_bp_el_model3_odonnell_ext,<br />
m.gaia_g_el_model1_odonnell_ext, m.gaia_g_el_model3_odonnell_ext, m.gaia_rp_el_model1_odonnell_ext, m.gaia_rp_el_model3_odonnell_ext, <br />
m.subaru_hsc_z_el_model1_odonnell_ext, m.subaru_hsc_z_el_model3_odonnell_ext, m.jst_jpcam_g_el_model1_odonnell_ext, <br />
m.jst_jpcam_g_el_model3_odonnell_ext, m.kids_g_el_model1_odonnell_ext, m.kids_g_el_model3_odonnell_ext,<br />
m.kids_i_el_model1_odonnell_ext, m.kids_i_el_model3_odonnell_ext, m.kids_r_el_model1_odonnell_ext, m.kids_r_el_model3_odonnell_ext, <br />
m.kids_u_el_model1_odonnell_ext, m.kids_u_el_model3_odonnell_ext, m.lsst_g_el_model1_odonnell_ext, m.lsst_g_el_model3_odonnell_ext, <br />
m.lsst_i_el_model1_odonnell_ext, m.lsst_i_el_model3_odonnell_ext, m.lsst_r_el_model1_odonnell_ext, m.lsst_r_el_model3_odonnell_ext,<br />
m.lsst_u_el_model1_odonnell_ext, m.lsst_u_el_model3_odonnell_ext, m.lsst_y_el_model1_odonnell_ext, m.lsst_y_el_model3_odonnell_ext, <br />
m.lsst_z_el_model1_odonnell_ext, m.lsst_z_el_model3_odonnell_ext, m.cfht_megacam_r_el_model1_odonnell_ext, <br />
m.cfht_megacam_r_el_model3_odonnell_ext, m.cfht_megacam_u_el_model1_odonnell_ext, m.cfht_megacam_u_el_model3_odonnell_ext, <br />
m.pan_starrs_i_el_model1_odonnell_ext, m.pan_starrs_i_el_model3_odonnell_ext, m.pan_starrs_z_el_model1_odonnell_ext, <br />
m.pan_starrs_z_el_model3_odonnell_ext, m.euclid_nisp_h_el_model1_odonnell_ext, m.euclid_nisp_h_el_model3_odonnell_ext,<br />
m.euclid_nisp_j_el_model1_odonnell_ext, m.euclid_nisp_j_el_model3_odonnell_ext, m.euclid_nisp_y_el_model1_odonnell_ext, <br />
m.euclid_nisp_y_el_model3_odonnell_ext, m.euclid_vis_el_model1_odonnell_ext, m.euclid_vis_el_model3_odonnell_ext, <br />
m.2mass_h_el_model1_odonnell_ext, m.2mass_h_el_model3_odonnell_ext, m.2mass_j_el_model1_odonnell_ext, <br />
m.2mass_j_el_model3_odonnell_ext, m.2mass_ks_el_model1_odonnell_ext, m.2mass_ks_el_model3_odonnell_ext, <br />
m.sdss_r01_el_model1_odonnell_ext, m.sdss_r01_el_model3_odonnell_ext, m.dominant_shape, m.bulge_angle, m.disk_angle, <br />
m.median_major_axis, m.scale_length, m.bulge_fraction, m.disk_scalelength, m.disk_nsersic, m.disk_r50, m.bulge_r50, m.bulge_nsersic, <br />
m.disk_ellipticity, m.bulge_ellipticity, m.disk_axis_ratio, m.bulge_axis_ratio, m.disk_inclination_angle, m.bulge_inclination_angle, <br />
m.random_index, m.halo_id, m.galaxy_id,<br />
udf.magnified_positions(CAST(m.ra_gal AS FLOAT), CAST(m.dec_gal AS FLOAT), l.defl_1, l.defl_2) AS magpos<br />
FROM tallada.flagship_mock_1_9_2_c2 AS m<br />
JOIN cosmohub.flagship_lensing AS l<br />
ON udf.ang2pix(13, m.ra_gal, m.dec_gal, True, True) = l.hpix<br />
AND m.step = l.step<br />
AND l.step=${STEP} AND m.step=${STEP}<br />
) AS m<br />
;<br />
<br />
for STEP in $(seq 344 400);<br />
do<br />
echo $STEP<br />
sem -j4 "beeline --color=true --showHeader=true --verbose=true --silent=false -u 'jdbc:hive2://ambarisrv02.pic.es:2181,ambarisrv03.pic.es:2181,ambarisrv01.pic.es:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2' -f lensing_step.sql --hivevar STEP=${STEP}"<br />
done<br />
sem --wait</div>
Tallada
https://pwiki.pic.es/index.php?title=Euclid_mock_production_v1.9.2&diff=818
Euclid mock production v1.9.2
2020-07-14T07:47:35Z
<p>Tallada: </p>
<hr />
<div> CREATE EXTERNAL TABLE `flagship_1_9_2`(<br />
`kind` bigint, <br />
`luminosity_r01` double, <br />
`halo_x` double, <br />
`halo_y` double, <br />
`halo_z` double, <br />
`halo_vx` double, <br />
`halo_vy` double, <br />
`halo_vz` double, <br />
`halo_r` double, <br />
`true_redshift_halo` double, <br />
`halo_lm` double, <br />
`halo_n_sats` bigint, <br />
`n_gals` bigint, <br />
`abs_mag_r01` double, <br />
`abs_mag_r01_evolved` double, <br />
`luminosity_r01_evolved` double, <br />
`gr_restframe` double, <br />
`color_kind` int, <br />
`x_gal` double, <br />
`y_gal` double, <br />
`z_gal` double, <br />
`r_gal` double, <br />
`true_redshift_gal` double, <br />
`ra_gal` double, <br />
`dec_gal` double, <br />
`hpix_nest` bigint, <br />
`vx_gal` double, <br />
`vy_gal` double, <br />
`vz_gal` double, <br />
`vrad_gal` double, <br />
`delta_r` double, <br />
`observed_redshift_gal` double, <br />
`sed_ke` double, <br />
`gr_cosmos` double, <br />
`sed_cosmos` double, <br />
`ext_curve_cosmos` bigint, <br />
`ebv_cosmos` double, <br />
`is_within_cosmos` boolean, <br />
`cosmos_distance` double, <br />
`abs_mag_uv_dereddened` double, <br />
`log_ml_r01` double, <br />
`log_stellar_mass` double, <br />
`metallicity` double, <br />
`log_sfr` double, <br />
`logf_halpha_ext` double, <br />
`logf_halpha` double, <br />
`halpha_scatter` double, <br />
`loglum_halpha` double, <br />
`loglum_halpha_ext` double, <br />
`logf_dummy` double, <br />
`z_dummy` double, <br />
`logf_halpha_model1_ext` double, <br />
`logf_halpha_model1` double, <br />
`loglum_halpha_model1_ext` double, <br />
`loglum_halpha_model1` double, <br />
`logf_halpha_model3_ext` double, <br />
`logf_halpha_model3` double, <br />
`loglum_halpha_model3_ext` double, <br />
`loglum_halpha_model3` double, <br />
`logf_hbeta_model1_ext` double, <br />
`logf_hbeta_model1` double, <br />
`logf_o2_model1_ext` double, <br />
`logf_o2_model1` double, <br />
`logf_n2_model1_ext` double, <br />
`logf_n2_model1` double, <br />
`logf_o3_model1_ext` double, <br />
`logf_o3_model1` double, <br />
`logf_s2_model1_ext` double, <br />
`logf_s2_model1` double, <br />
`logf_hbeta_model3_ext` double, <br />
`logf_hbeta_model3` double, <br />
`logf_o2_model3_ext` double, <br />
`logf_o2_model3` double, <br />
`logf_n2_model3_ext` double, <br />
`logf_n2_model3` double, <br />
`logf_o3_model3_ext` double, <br />
`logf_o3_model3` double, <br />
`logf_s2_model3_ext` double, <br />
`logf_s2_model3` double, <br />
`blanco_decam_g` double, <br />
`blanco_decam_i` double, <br />
`blanco_decam_r` double, <br />
`blanco_decam_z` double, <br />
`gaia_bp` double, <br />
`gaia_g` double, <br />
`gaia_rp` double, <br />
`subaru_hsc_z` double, <br />
`jst_jpcam_g` double, <br />
`kids_g` double, <br />
`kids_i` double, <br />
`kids_r` double, <br />
`kids_u` double, <br />
`lsst_g` double, <br />
`lsst_i` double, <br />
`lsst_r` double, <br />
`lsst_u` double, <br />
`lsst_y` double, <br />
`lsst_z` double, <br />
`cfht_megacam_r` double, <br />
`cfht_megacam_u` double, <br />
`pan_starrs_i` double, <br />
`pan_starrs_z` double, <br />
`euclid_nisp_h` double, <br />
`euclid_nisp_j` double, <br />
`euclid_nisp_y` double, <br />
`euclid_vis` double, <br />
`2mass_h` double, <br />
`2mass_j` double, <br />
`2mass_ks` double, <br />
`sdss_r01` double, <br />
`l_gal` double, <br />
`b_gal` double, <br />
`mw_extinction` double, <br />
`2mass_h_odonnell_ext` double, <br />
`2mass_j_odonnell_ext` double, <br />
`2mass_ks_odonnell_ext` double, <br />
`blanco_decam_g_odonnell_ext` double, <br />
`blanco_decam_i_odonnell_ext` double, <br />
`blanco_decam_r_odonnell_ext` double, <br />
`blanco_decam_z_odonnell_ext` double, <br />
`cfht_megacam_r_odonnell_ext` double, <br />
`cfht_megacam_u_odonnell_ext` double, <br />
`euclid_nisp_h_odonnell_ext` double, <br />
`euclid_nisp_j_odonnell_ext` double, <br />
`euclid_nisp_y_odonnell_ext` double, <br />
`euclid_vis_odonnell_ext` double, <br />
`gaia_bp_odonnell_ext` double, <br />
`gaia_g_odonnell_ext` double, <br />
`gaia_rp_odonnell_ext` double, <br />
`jst_jpcam_g_odonnell_ext` double, <br />
`kids_g_odonnell_ext` double, <br />
`kids_i_odonnell_ext` double, <br />
`kids_r_odonnell_ext` double, <br />
`kids_u_odonnell_ext` double, <br />
`lsst_g_odonnell_ext` double, <br />
`lsst_i_odonnell_ext` double, <br />
`lsst_r_odonnell_ext` double, <br />
`lsst_u_odonnell_ext` double, <br />
`lsst_y_odonnell_ext` double, <br />
`lsst_z_odonnell_ext` double, <br />
`pan_starrs_i_odonnell_ext` double, <br />
`pan_starrs_z_odonnell_ext` double, <br />
`sdss_r01_odonnell_ext` double, <br />
`subaru_hsc_z_odonnell_ext` double, <br />
`blanco_decam_g_el_model1_odonnell_ext` double, <br />
`blanco_decam_g_el_model3_odonnell_ext` double, <br />
`blanco_decam_i_el_model1_odonnell_ext` double, <br />
`blanco_decam_i_el_model3_odonnell_ext` double, <br />
`blanco_decam_r_el_model1_odonnell_ext` double, <br />
`blanco_decam_r_el_model3_odonnell_ext` double, <br />
`blanco_decam_z_el_model1_odonnell_ext` double, <br />
`blanco_decam_z_el_model3_odonnell_ext` double, <br />
`gaia_bp_el_model1_odonnell_ext` double, <br />
`gaia_bp_el_model3_odonnell_ext` double, <br />
`gaia_g_el_model1_odonnell_ext` double, <br />
`gaia_g_el_model3_odonnell_ext` double, <br />
`gaia_rp_el_model1_odonnell_ext` double, <br />
`gaia_rp_el_model3_odonnell_ext` double, <br />
`subaru_hsc_z_el_model1_odonnell_ext` double, <br />
`subaru_hsc_z_el_model3_odonnell_ext` double, <br />
`jst_jpcam_g_el_model1_odonnell_ext` double, <br />
`jst_jpcam_g_el_model3_odonnell_ext` double, <br />
`kids_g_el_model1_odonnell_ext` double, <br />
`kids_g_el_model3_odonnell_ext` double, <br />
`kids_i_el_model1_odonnell_ext` double, <br />
`kids_i_el_model3_odonnell_ext` double, <br />
`kids_r_el_model1_odonnell_ext` double, <br />
`kids_r_el_model3_odonnell_ext` double, <br />
`kids_u_el_model1_odonnell_ext` double, <br />
`kids_u_el_model3_odonnell_ext` double, <br />
`lsst_g_el_model1_odonnell_ext` double, <br />
`lsst_g_el_model3_odonnell_ext` double, <br />
`lsst_i_el_model1_odonnell_ext` double, <br />
`lsst_i_el_model3_odonnell_ext` double, <br />
`lsst_r_el_model1_odonnell_ext` double, <br />
`lsst_r_el_model3_odonnell_ext` double, <br />
`lsst_u_el_model1_odonnell_ext` double, <br />
`lsst_u_el_model3_odonnell_ext` double, <br />
`lsst_y_el_model1_odonnell_ext` double, <br />
`lsst_y_el_model3_odonnell_ext` double, <br />
`lsst_z_el_model1_odonnell_ext` double, <br />
`lsst_z_el_model3_odonnell_ext` double, <br />
`cfht_megacam_r_el_model1_odonnell_ext` double, <br />
`cfht_megacam_r_el_model3_odonnell_ext` double, <br />
`cfht_megacam_u_el_model1_odonnell_ext` double, <br />
`cfht_megacam_u_el_model3_odonnell_ext` double, <br />
`pan_starrs_i_el_model1_odonnell_ext` double, <br />
`pan_starrs_i_el_model3_odonnell_ext` double, <br />
`pan_starrs_z_el_model1_odonnell_ext` double, <br />
`pan_starrs_z_el_model3_odonnell_ext` double, <br />
`euclid_nisp_h_el_model1_odonnell_ext` double, <br />
`euclid_nisp_h_el_model3_odonnell_ext` double, <br />
`euclid_nisp_j_el_model1_odonnell_ext` double, <br />
`euclid_nisp_j_el_model3_odonnell_ext` double, <br />
`euclid_nisp_y_el_model1_odonnell_ext` double, <br />
`euclid_nisp_y_el_model3_odonnell_ext` double, <br />
`euclid_vis_el_model1_odonnell_ext` double, <br />
`euclid_vis_el_model3_odonnell_ext` double, <br />
`2mass_h_el_model1_odonnell_ext` double, <br />
`2mass_h_el_model3_odonnell_ext` double, <br />
`2mass_j_el_model1_odonnell_ext` double, <br />
`2mass_j_el_model3_odonnell_ext` double, <br />
`2mass_ks_el_model1_odonnell_ext` double, <br />
`2mass_ks_el_model3_odonnell_ext` double, <br />
`sdss_r01_el_model1_odonnell_ext` double, <br />
`sdss_r01_el_model3_odonnell_ext` double, <br />
`dominant_shape` bigint, <br />
`bulge_angle` double, <br />
`disk_angle` double, <br />
`median_major_axis` double, <br />
`scale_length` double, <br />
`bulge_fraction` double, <br />
`disk_scalelength` double, <br />
`disk_nsersic` double, <br />
`disk_r50` double, <br />
`bulge_r50` double, <br />
`bulge_nsersic` double, <br />
`disk_ellipticity` double, <br />
`bulge_ellipticity` double, <br />
`disk_axis_ratio` double, <br />
`bulge_axis_ratio` double, <br />
`disk_inclination_angle` double, <br />
`bulge_inclination_angle` double, <br />
`step` bigint, <br />
`random_index` double, <br />
`halo_id` bigint, <br />
`galaxy_id` bigint)<br />
ROW FORMAT SERDE <br />
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' <br />
STORED AS INPUTFORMAT <br />
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' <br />
OUTPUTFORMAT <br />
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'<br />
;<br />
<br />
CREATE TABLE flagship_mock_1_9_2_c(<br />
`kind` bigint, <br />
`luminosity_r01` double, <br />
`halo_x` double, <br />
`halo_y` double, <br />
`halo_z` double, <br />
`halo_vx` double, <br />
`halo_vy` double, <br />
`halo_vz` double, <br />
`halo_r` double, <br />
`true_redshift_halo` double, <br />
`halo_lm` double, <br />
`halo_n_sats` bigint, <br />
`n_gals` bigint, <br />
`abs_mag_r01` double, <br />
`abs_mag_r01_evolved` double, <br />
`luminosity_r01_evolved` double, <br />
`gr_restframe` double, <br />
`color_kind` int, <br />
`x_gal` double, <br />
`y_gal` double, <br />
`z_gal` double, <br />
`r_gal` double, <br />
`true_redshift_gal` double, <br />
`ra_gal` double, <br />
`dec_gal` double, <br />
`hpix_nest` bigint, <br />
`vx_gal` double, <br />
`vy_gal` double, <br />
`vz_gal` double, <br />
`vrad_gal` double, <br />
`delta_r` double, <br />
`observed_redshift_gal` double, <br />
`sed_ke` double, <br />
`gr_cosmos` double, <br />
`sed_cosmos` double, <br />
`ext_curve_cosmos` bigint, <br />
`ebv_cosmos` double, <br />
`is_within_cosmos` boolean, <br />
`cosmos_distance` double, <br />
`abs_mag_uv_dereddened` double, <br />
`log_ml_r01` double, <br />
`log_stellar_mass` double, <br />
`metallicity` double, <br />
`log_sfr` double, <br />
`logf_halpha_ext` double, <br />
`logf_halpha` double, <br />
`halpha_scatter` double, <br />
`loglum_halpha` double, <br />
`loglum_halpha_ext` double, <br />
`logf_dummy` double, <br />
`z_dummy` double, <br />
`logf_halpha_model1_ext` double, <br />
`logf_halpha_model1` double, <br />
`loglum_halpha_model1_ext` double, <br />
`loglum_halpha_model1` double, <br />
`logf_halpha_model3_ext` double, <br />
`logf_halpha_model3` double, <br />
`loglum_halpha_model3_ext` double, <br />
`loglum_halpha_model3` double, <br />
`logf_hbeta_model1_ext` double, <br />
`logf_hbeta_model1` double, <br />
`logf_o2_model1_ext` double, <br />
`logf_o2_model1` double, <br />
`logf_n2_model1_ext` double, <br />
`logf_n2_model1` double, <br />
`logf_o3_model1_ext` double, <br />
`logf_o3_model1` double, <br />
`logf_s2_model1_ext` double, <br />
`logf_s2_model1` double, <br />
`logf_hbeta_model3_ext` double, <br />
`logf_hbeta_model3` double, <br />
`logf_o2_model3_ext` double, <br />
`logf_o2_model3` double, <br />
`logf_n2_model3_ext` double, <br />
`logf_n2_model3` double, <br />
`logf_o3_model3_ext` double, <br />
`logf_o3_model3` double, <br />
`logf_s2_model3_ext` double, <br />
`logf_s2_model3` double, <br />
`blanco_decam_g` double, <br />
`blanco_decam_i` double, <br />
`blanco_decam_r` double, <br />
`blanco_decam_z` double, <br />
`gaia_bp` double, <br />
`gaia_g` double, <br />
`gaia_rp` double, <br />
`subaru_hsc_z` double, <br />
`jst_jpcam_g` double, <br />
`kids_g` double, <br />
`kids_i` double, <br />
`kids_r` double, <br />
`kids_u` double, <br />
`lsst_g` double, <br />
`lsst_i` double, <br />
`lsst_r` double, <br />
`lsst_u` double, <br />
`lsst_y` double, <br />
`lsst_z` double, <br />
`cfht_megacam_r` double, <br />
`cfht_megacam_u` double, <br />
`pan_starrs_i` double, <br />
`pan_starrs_z` double, <br />
`euclid_nisp_h` double, <br />
`euclid_nisp_j` double, <br />
`euclid_nisp_y` double, <br />
`euclid_vis` double, <br />
`2mass_h` double, <br />
`2mass_j` double, <br />
`2mass_ks` double, <br />
`sdss_r01` double, <br />
`l_gal` double, <br />
`b_gal` double, <br />
`mw_extinction` double, <br />
`2mass_h_odonnell_ext` double, <br />
`2mass_j_odonnell_ext` double, <br />
`2mass_ks_odonnell_ext` double, <br />
`blanco_decam_g_odonnell_ext` double, <br />
`blanco_decam_i_odonnell_ext` double, <br />
`blanco_decam_r_odonnell_ext` double, <br />
`blanco_decam_z_odonnell_ext` double, <br />
`cfht_megacam_r_odonnell_ext` double, <br />
`cfht_megacam_u_odonnell_ext` double, <br />
`euclid_nisp_h_odonnell_ext` double, <br />
`euclid_nisp_j_odonnell_ext` double, <br />
`euclid_nisp_y_odonnell_ext` double, <br />
`euclid_vis_odonnell_ext` double, <br />
`gaia_bp_odonnell_ext` double, <br />
`gaia_g_odonnell_ext` double, <br />
`gaia_rp_odonnell_ext` double, <br />
`jst_jpcam_g_odonnell_ext` double, <br />
`kids_g_odonnell_ext` double, <br />
`kids_i_odonnell_ext` double, <br />
`kids_r_odonnell_ext` double, <br />
`kids_u_odonnell_ext` double, <br />
`lsst_g_odonnell_ext` double, <br />
`lsst_i_odonnell_ext` double, <br />
`lsst_r_odonnell_ext` double, <br />
`lsst_u_odonnell_ext` double, <br />
`lsst_y_odonnell_ext` double, <br />
`lsst_z_odonnell_ext` double, <br />
`pan_starrs_i_odonnell_ext` double, <br />
`pan_starrs_z_odonnell_ext` double, <br />
`sdss_r01_odonnell_ext` double, <br />
`subaru_hsc_z_odonnell_ext` double, <br />
`blanco_decam_g_el_model1_odonnell_ext` double, <br />
`blanco_decam_g_el_model3_odonnell_ext` double, <br />
`blanco_decam_i_el_model1_odonnell_ext` double, <br />
`blanco_decam_i_el_model3_odonnell_ext` double, <br />
`blanco_decam_r_el_model1_odonnell_ext` double, <br />
`blanco_decam_r_el_model3_odonnell_ext` double, <br />
`blanco_decam_z_el_model1_odonnell_ext` double, <br />
`blanco_decam_z_el_model3_odonnell_ext` double, <br />
`gaia_bp_el_model1_odonnell_ext` double, <br />
`gaia_bp_el_model3_odonnell_ext` double, <br />
`gaia_g_el_model1_odonnell_ext` double, <br />
`gaia_g_el_model3_odonnell_ext` double, <br />
`gaia_rp_el_model1_odonnell_ext` double, <br />
`gaia_rp_el_model3_odonnell_ext` double, <br />
`subaru_hsc_z_el_model1_odonnell_ext` double, <br />
`subaru_hsc_z_el_model3_odonnell_ext` double, <br />
`jst_jpcam_g_el_model1_odonnell_ext` double, <br />
`jst_jpcam_g_el_model3_odonnell_ext` double, <br />
`kids_g_el_model1_odonnell_ext` double, <br />
`kids_g_el_model3_odonnell_ext` double, <br />
`kids_i_el_model1_odonnell_ext` double, <br />
`kids_i_el_model3_odonnell_ext` double, <br />
`kids_r_el_model1_odonnell_ext` double, <br />
`kids_r_el_model3_odonnell_ext` double, <br />
`kids_u_el_model1_odonnell_ext` double, <br />
`kids_u_el_model3_odonnell_ext` double, <br />
`lsst_g_el_model1_odonnell_ext` double, <br />
`lsst_g_el_model3_odonnell_ext` double, <br />
`lsst_i_el_model1_odonnell_ext` double, <br />
`lsst_i_el_model3_odonnell_ext` double, <br />
`lsst_r_el_model1_odonnell_ext` double, <br />
`lsst_r_el_model3_odonnell_ext` double, <br />
`lsst_u_el_model1_odonnell_ext` double, <br />
`lsst_u_el_model3_odonnell_ext` double, <br />
`lsst_y_el_model1_odonnell_ext` double, <br />
`lsst_y_el_model3_odonnell_ext` double, <br />
`lsst_z_el_model1_odonnell_ext` double, <br />
`lsst_z_el_model3_odonnell_ext` double, <br />
`cfht_megacam_r_el_model1_odonnell_ext` double, <br />
`cfht_megacam_r_el_model3_odonnell_ext` double, <br />
`cfht_megacam_u_el_model1_odonnell_ext` double, <br />
`cfht_megacam_u_el_model3_odonnell_ext` double, <br />
`pan_starrs_i_el_model1_odonnell_ext` double, <br />
`pan_starrs_i_el_model3_odonnell_ext` double, <br />
`pan_starrs_z_el_model1_odonnell_ext` double, <br />
`pan_starrs_z_el_model3_odonnell_ext` double, <br />
`euclid_nisp_h_el_model1_odonnell_ext` double, <br />
`euclid_nisp_h_el_model3_odonnell_ext` double, <br />
`euclid_nisp_j_el_model1_odonnell_ext` double, <br />
`euclid_nisp_j_el_model3_odonnell_ext` double, <br />
`euclid_nisp_y_el_model1_odonnell_ext` double, <br />
`euclid_nisp_y_el_model3_odonnell_ext` double, <br />
`euclid_vis_el_model1_odonnell_ext` double, <br />
`euclid_vis_el_model3_odonnell_ext` double, <br />
`2mass_h_el_model1_odonnell_ext` double, <br />
`2mass_h_el_model3_odonnell_ext` double, <br />
`2mass_j_el_model1_odonnell_ext` double, <br />
`2mass_j_el_model3_odonnell_ext` double, <br />
`2mass_ks_el_model1_odonnell_ext` double, <br />
`2mass_ks_el_model3_odonnell_ext` double, <br />
`sdss_r01_el_model1_odonnell_ext` double, <br />
`sdss_r01_el_model3_odonnell_ext` double, <br />
`dominant_shape` bigint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)', <br />
`bulge_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`disk_angle` double COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', <br />
`median_major_axis` double COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)', <br />
`scale_length` double COMMENT 'disc and bulge scalelength prior', <br />
`bulge_fraction` double COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)', <br />
`disk_scalelength` double COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)', <br />
`disk_nsersic` double COMMENT 'Sersic index of the disk component', <br />
`disk_r50` double COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678', <br />
`bulge_r50` double COMMENT 'half light radius of the bulge', <br />
`bulge_nsersic` double COMMENT 'Sersic index of the bulge component', <br />
`disk_ellipticity` double COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)', <br />
`bulge_ellipticity` double COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)', <br />
`disk_axis_ratio` double COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', <br />
`bulge_axis_ratio` double COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',<br />
`disk_inclination_angle` double COMMENT '', <br />
`bulge_inclination_angle` double COMMENT '', <br />
`step` bigint, <br />
`random_index` double, <br />
`halo_id` bigint, <br />
`galaxy_id` bigint<br />
)<br />
CLUSTERED BY ( <br />
halo_id, <br />
galaxy_id<br />
) <br />
INTO 128 BUCKETS<br />
ROW FORMAT SERDE <br />
'org.apache.hadoop.hive.ql.io.orc.OrcSerde' <br />
STORED AS INPUTFORMAT <br />
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' <br />
OUTPUTFORMAT <br />
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'<br />
;<br />
<br />
INSERT OVERWRITE TABLE cosmohub.flagship_mock_1_9_2_c<br />
SELECT * FROM tallada.flagship_1_9_2;<br />
<br />
ANALYZE TABLE cosmohub.flagship_mock_1_9_2_c COMPUTE STATISTICS FOR COLUMNS;</div>
Tallada
https://pwiki.pic.es/index.php?title=Legacy_Survey_DR8&diff=803
Legacy Survey DR8
2020-05-10T12:03:59Z
<p>Tallada: /* Join insert */</p>
<hr />
<div>== Original DR8 ==<br />
=== Raw data ===<br />
<br />
Downloaded from<br />
ftp://archive.noao.edu:/public/hlsp/ls/dr8/<br />
<br />
Directories:<br />
south/tractor<br />
north/tractor<br />
<br />
Mirrored using lftp commands:<br />
mirror -c -v -P 200 north/tractor/ /mnt/pau/north/tractor/<br />
mirror -c -v -P 200 south/tractor/ /mnt/pau/south/tractor/<br />
<br />
into a 5 TiB temporary Ceph filesystem mounted in tdm002.<br />
<br />
Checksums provided through sha256sum files, and checked using command:<br />
find * -type d | xargs -t -L1 -P24 sh -c 'cd $1; sha256sum --quiet -c *.sha256sum' cksum<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to convert fits files to parquet:<br />
<br />
path_base = '/cephfs/pic.es/cosmohub/legacysurvey'<br />
hdfs_base = '/user/tallada/data/legacysurvey'<br />
<br />
hs = ['h=north', 'h=south']<br />
rs = ['r={0:03d}'.format(x) for x in range(360)]<br />
<br />
for h in hs:<br />
for r in rs:<br />
p = os.path.join(path_base, h, r)<br />
if not os.path.exists(p):<br />
continue<br />
<br />
size = sum(os.path.getsize(os.path.join(p,f)) for f in os.listdir(p) if os.path.isfile(os.path.join(p,f)))<br />
<br />
print h, r, size <br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + p<br />
)<br />
<br />
df = df.repartition(int(math.ceil(size / (256*1024*1024.0))))<br />
<br />
df.write.parquet(os.path.join(hdfs_base, h, r))<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE tallada.dr8 (<br />
`release` smallint,<br />
`brickid` int,<br />
`brickname` string,<br />
`objid` int,<br />
`brick_primary` boolean,<br />
`brightblob` smallint,<br />
`maskbits` smallint,<br />
`type` string,<br />
`ra` double,<br />
`dec` double,<br />
`ra_ivar` float,<br />
`dec_ivar` float,<br />
`bx` float,<br />
`by` float,<br />
`dchisq` array<float>,<br />
`ebv` float,<br />
`mjd_min` double,<br />
`mjd_max` double,<br />
`ref_cat` string,<br />
`ref_id` bigint,<br />
`pmra` float,<br />
`pmdec` float,<br />
`parallax` float,<br />
`pmra_ivar` float,<br />
`pmdec_ivar` float,<br />
`parallax_ivar` float,<br />
`ref_epoch` float,<br />
`gaia_pointsource` boolean,<br />
`gaia_phot_g_mean_mag` float,<br />
`gaia_phot_g_mean_flux_over_error` float,<br />
`gaia_phot_g_n_obs` smallint,<br />
`gaia_phot_bp_mean_mag` float,<br />
`gaia_phot_bp_mean_flux_over_error` float,<br />
`gaia_phot_bp_n_obs` smallint,<br />
`gaia_phot_rp_mean_mag` float,<br />
`gaia_phot_rp_mean_flux_over_error` float,<br />
`gaia_phot_rp_n_obs` smallint,<br />
`gaia_phot_variable_flag` boolean,<br />
`gaia_astrometric_excess_noise` float,<br />
`gaia_astrometric_excess_noise_sig` float,<br />
`gaia_astrometric_n_obs_al` smallint,<br />
`gaia_astrometric_n_good_obs_al` smallint,<br />
`gaia_astrometric_weight_al` float,<br />
`gaia_duplicated_source` boolean,<br />
`gaia_a_g_val` float,<br />
`gaia_e_bp_min_rp_val` float,<br />
`gaia_phot_bp_rp_excess_factor` float,<br />
`gaia_astrometric_sigma5d_max` float,<br />
`gaia_astrometric_params_solved` tinyint,<br />
`flux_g` float,<br />
`flux_r` float,<br />
`flux_z` float,<br />
`flux_w1` float,<br />
`flux_w2` float,<br />
`flux_w3` float,<br />
`flux_w4` float,<br />
`flux_ivar_g` float,<br />
`flux_ivar_r` float,<br />
`flux_ivar_z` float,<br />
`flux_ivar_w1` float,<br />
`flux_ivar_w2` float,<br />
`flux_ivar_w3` float,<br />
`flux_ivar_w4` float,<br />
`fiberflux_g` float,<br />
`fiberflux_r` float,<br />
`fiberflux_z` float,<br />
`fibertotflux_g` float,<br />
`fibertotflux_r` float,<br />
`fibertotflux_z` float,<br />
`apflux_g` array<float>,<br />
`apflux_r` array<float>,<br />
`apflux_z` array<float>,<br />
`apflux_resid_g` array<float>,<br />
`apflux_resid_r` array<float>,<br />
`apflux_resid_z` array<float>,<br />
`apflux_ivar_g` array<float>,<br />
`apflux_ivar_r` array<float>,<br />
`apflux_ivar_z` array<float>,<br />
`mw_transmission_g` float,<br />
`mw_transmission_r` float,<br />
`mw_transmission_z` float,<br />
`mw_transmission_w1` float,<br />
`mw_transmission_w2` float,<br />
`mw_transmission_w3` float,<br />
`mw_transmission_w4` float,<br />
`nobs_g` smallint,<br />
`nobs_r` smallint,<br />
`nobs_z` smallint,<br />
`nobs_w1` smallint,<br />
`nobs_w2` smallint,<br />
`nobs_w3` smallint,<br />
`nobs_w4` smallint,<br />
`rchisq_g` float,<br />
`rchisq_r` float,<br />
`rchisq_z` float,<br />
`rchisq_w1` float,<br />
`rchisq_w2` float,<br />
`rchisq_w3` float,<br />
`rchisq_w4` float,<br />
`fracflux_g` float,<br />
`fracflux_r` float,<br />
`fracflux_z` float,<br />
`fracflux_w1` float,<br />
`fracflux_w2` float,<br />
`fracflux_w3` float,<br />
`fracflux_w4` float,<br />
`fracmasked_g` float,<br />
`fracmasked_r` float,<br />
`fracmasked_z` float,<br />
`fracin_g` float,<br />
`fracin_r` float,<br />
`fracin_z` float,<br />
`anymask_g` smallint,<br />
`anymask_r` smallint,<br />
`anymask_z` smallint,<br />
`allmask_g` smallint,<br />
`allmask_r` smallint,<br />
`allmask_z` smallint,<br />
`wisemask_w1` tinyint,<br />
`wisemask_w2` tinyint,<br />
`psfsize_g` float,<br />
`psfsize_r` float,<br />
`psfsize_z` float,<br />
`psfdepth_g` float,<br />
`psfdepth_r` float,<br />
`psfdepth_z` float,<br />
`galdepth_g` float,<br />
`galdepth_r` float,<br />
`galdepth_z` float,<br />
`psfdepth_w1` float,<br />
`psfdepth_w2` float,<br />
`psfdepth_w3` float,<br />
`psfdepth_w4` float,<br />
`wise_coadd_id` string,<br />
`lc_flux_w1` array<float>,<br />
`lc_flux_w2` array<float>,<br />
`lc_flux_ivar_w1` array<float>,<br />
`lc_flux_ivar_w2` array<float>,<br />
`lc_nobs_w1` array<smallint>,<br />
`lc_nobs_w2` array<smallint>,<br />
`lc_fracflux_w1` array<float>,<br />
`lc_fracflux_w2` array<float>,<br />
`lc_rchisq_w1` array<float>,<br />
`lc_rchisq_w2` array<float>,<br />
`lc_mjd_w1` array<double>,<br />
`lc_mjd_w2` array<double>,<br />
`fracdev` float,<br />
`fracdev_ivar` float,<br />
`shapeexp_r` float,<br />
`shapeexp_r_ivar` float,<br />
`shapeexp_e1` float,<br />
`shapeexp_e1_ivar` float,<br />
`shapeexp_e2` float,<br />
`shapeexp_e2_ivar` float,<br />
`shapedev_r` float,<br />
`shapedev_r_ivar` float,<br />
`shapedev_e1` float,<br />
`shapedev_e1_ivar` float,<br />
`shapedev_e2` float,<br />
`shapedev_e2_ivar` float<br />
)<br />
PARTITIONED BY (<br />
h string,<br />
r string<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey/dr8/'<br />
;<br />
<br />
=== Internal Table ===<br />
<br />
CREATE TABLE cosmohub.legacy_survey_dr8 (<br />
`release` smallint,<br />
`brickid` int,<br />
`brickname` string,<br />
`objid` int,<br />
`brick_primary` boolean,<br />
`brightblob` smallint,<br />
`maskbits` smallint,<br />
`type` string,<br />
`ra` double,<br />
`dec` double,<br />
`ra_ivar` float,<br />
`dec_ivar` float,<br />
`bx` float,<br />
`by` float,<br />
`dchisq` array<float>,<br />
`ebv` float,<br />
`mjd_min` double,<br />
`mjd_max` double,<br />
`ref_cat` string,<br />
`ref_id` bigint,<br />
`pmra` float,<br />
`pmdec` float,<br />
`parallax` float,<br />
`pmra_ivar` float,<br />
`pmdec_ivar` float,<br />
`parallax_ivar` float,<br />
`ref_epoch` float,<br />
`gaia_pointsource` boolean,<br />
`gaia_phot_g_mean_mag` float,<br />
`gaia_phot_g_mean_flux_over_error` float,<br />
`gaia_phot_g_n_obs` smallint,<br />
`gaia_phot_bp_mean_mag` float,<br />
`gaia_phot_bp_mean_flux_over_error` float,<br />
`gaia_phot_bp_n_obs` smallint,<br />
`gaia_phot_rp_mean_mag` float,<br />
`gaia_phot_rp_mean_flux_over_error` float,<br />
`gaia_phot_rp_n_obs` smallint,<br />
`gaia_phot_variable_flag` boolean,<br />
`gaia_astrometric_excess_noise` float,<br />
`gaia_astrometric_excess_noise_sig` float,<br />
`gaia_astrometric_n_obs_al` smallint,<br />
`gaia_astrometric_n_good_obs_al` smallint,<br />
`gaia_astrometric_weight_al` float,<br />
`gaia_duplicated_source` boolean,<br />
`gaia_a_g_val` float,<br />
`gaia_e_bp_min_rp_val` float,<br />
`gaia_phot_bp_rp_excess_factor` float,<br />
`gaia_astrometric_sigma5d_max` float,<br />
`gaia_astrometric_params_solved` tinyint,<br />
`flux_g` float,<br />
`flux_r` float,<br />
`flux_z` float,<br />
`flux_w1` float,<br />
`flux_w2` float,<br />
`flux_w3` float,<br />
`flux_w4` float,<br />
`flux_ivar_g` float,<br />
`flux_ivar_r` float,<br />
`flux_ivar_z` float,<br />
`flux_ivar_w1` float,<br />
`flux_ivar_w2` float,<br />
`flux_ivar_w3` float,<br />
`flux_ivar_w4` float,<br />
`fiberflux_g` float,<br />
`fiberflux_r` float,<br />
`fiberflux_z` float,<br />
`fibertotflux_g` float,<br />
`fibertotflux_r` float,<br />
`fibertotflux_z` float,<br />
`apflux_g` array<float>,<br />
`apflux_r` array<float>,<br />
`apflux_z` array<float>,<br />
`apflux_resid_g` array<float>,<br />
`apflux_resid_r` array<float>,<br />
`apflux_resid_z` array<float>,<br />
`apflux_ivar_g` array<float>,<br />
`apflux_ivar_r` array<float>,<br />
`apflux_ivar_z` array<float>,<br />
`mw_transmission_g` float,<br />
`mw_transmission_r` float,<br />
`mw_transmission_z` float,<br />
`mw_transmission_w1` float,<br />
`mw_transmission_w2` float,<br />
`mw_transmission_w3` float,<br />
`mw_transmission_w4` float,<br />
`nobs_g` smallint,<br />
`nobs_r` smallint,<br />
`nobs_z` smallint,<br />
`nobs_w1` smallint,<br />
`nobs_w2` smallint,<br />
`nobs_w3` smallint,<br />
`nobs_w4` smallint,<br />
`rchisq_g` float,<br />
`rchisq_r` float,<br />
`rchisq_z` float,<br />
`rchisq_w1` float,<br />
`rchisq_w2` float,<br />
`rchisq_w3` float,<br />
`rchisq_w4` float,<br />
`fracflux_g` float,<br />
`fracflux_r` float,<br />
`fracflux_z` float,<br />
`fracflux_w1` float,<br />
`fracflux_w2` float,<br />
`fracflux_w3` float,<br />
`fracflux_w4` float,<br />
`fracmasked_g` float,<br />
`fracmasked_r` float,<br />
`fracmasked_z` float,<br />
`fracin_g` float,<br />
`fracin_r` float,<br />
`fracin_z` float,<br />
`anymask_g` smallint,<br />
`anymask_r` smallint,<br />
`anymask_z` smallint,<br />
`allmask_g` smallint,<br />
`allmask_r` smallint,<br />
`allmask_z` smallint,<br />
`wisemask_w1` tinyint,<br />
`wisemask_w2` tinyint,<br />
`psfsize_g` float,<br />
`psfsize_r` float,<br />
`psfsize_z` float,<br />
`psfdepth_g` float,<br />
`psfdepth_r` float,<br />
`psfdepth_z` float,<br />
`galdepth_g` float,<br />
`galdepth_r` float,<br />
`galdepth_z` float,<br />
`psfdepth_w1` float,<br />
`psfdepth_w2` float,<br />
`psfdepth_w3` float,<br />
`psfdepth_w4` float,<br />
`wise_coadd_id` string,<br />
`lc_flux_w1` array<float>,<br />
`lc_flux_w2` array<float>,<br />
`lc_flux_ivar_w1` array<float>,<br />
`lc_flux_ivar_w2` array<float>,<br />
`lc_nobs_w1` array<smallint>,<br />
`lc_nobs_w2` array<smallint>,<br />
`lc_fracflux_w1` array<float>,<br />
`lc_fracflux_w2` array<float>,<br />
`lc_rchisq_w1` array<float>,<br />
`lc_rchisq_w2` array<float>,<br />
`lc_mjd_w1` array<double>,<br />
`lc_mjd_w2` array<double>,<br />
`fracdev` float,<br />
`fracdev_ivar` float,<br />
`shapeexp_r` float,<br />
`shapeexp_r_ivar` float,<br />
`shapeexp_e1` float,<br />
`shapeexp_e1_ivar` float,<br />
`shapeexp_e2` float,<br />
`shapeexp_e2_ivar` float,<br />
`shapedev_r` float,<br />
`shapedev_r_ivar` float,<br />
`shapedev_e1` float,<br />
`shapedev_e1_ivar` float,<br />
`shapedev_e2` float,<br />
`shapedev_e2_ivar` float<br />
)<br />
clustered by (<br />
release, brickid, objid<br />
)<br />
sorted by (<br />
release, brickid, objid<br />
)<br />
into 4096 buckets<br />
stored as orc<br />
;<br />
<br />
insert overwrite table cosmohub.legacy_survey_dr8<br />
select release, brickid, brickname, objid, brick_primary, brightblob, maskbits, type, ra, dec, ra_ivar, dec_ivar, bx, `by`, <br />
dchisq, ebv, mjd_min, mjd_max, ref_cat, ref_id, pmra, pmdec, parallax, pmra_ivar, pmdec_ivar, parallax_ivar, ref_epoch,<br />
gaia_pointsource, gaia_phot_g_mean_mag, gaia_phot_g_mean_flux_over_error, gaia_phot_g_n_obs,<br />
gaia_phot_bp_mean_mag, gaia_phot_bp_mean_flux_over_error, gaia_phot_bp_n_obs, gaia_phot_rp_mean_mag,<br />
gaia_phot_rp_mean_flux_over_error, gaia_phot_rp_n_obs, gaia_phot_variable_flag, gaia_astrometric_excess_noise,<br />
gaia_astrometric_excess_noise_sig, gaia_astrometric_n_obs_al, gaia_astrometric_n_good_obs_al,<br />
gaia_astrometric_weight_al, gaia_duplicated_source, gaia_a_g_val, gaia_e_bp_min_rp_val,<br />
gaia_phot_bp_rp_excess_factor, gaia_astrometric_sigma5d_max, gaia_astrometric_params_solved,<br />
flux_g, flux_r, flux_z, flux_w1, flux_w2, flux_w3, flux_w4, flux_ivar_g, flux_ivar_r,<br />
flux_ivar_z, flux_ivar_w1, flux_ivar_w2, flux_ivar_w3, flux_ivar_w4, fiberflux_g, fiberflux_r,<br />
fiberflux_z, fibertotflux_g, fibertotflux_r, fibertotflux_z, apflux_g, apflux_r, apflux_z,<br />
apflux_resid_g, apflux_resid_r, apflux_resid_z, apflux_ivar_g, apflux_ivar_r, apflux_ivar_z,<br />
mw_transmission_g, mw_transmission_r, mw_transmission_z, mw_transmission_w1, mw_transmission_w2,<br />
mw_transmission_w3, mw_transmission_w4, nobs_g, nobs_r, nobs_z, nobs_w1, nobs_w2, nobs_w3,<br />
nobs_w4, rchisq_g, rchisq_r, rchisq_z, rchisq_w1, rchisq_w2, rchisq_w3, rchisq_w4, fracflux_g,<br />
fracflux_r, fracflux_z, fracflux_w1, fracflux_w2, fracflux_w3, fracflux_w4, fracmasked_g,<br />
fracmasked_r, fracmasked_z, fracin_g, fracin_r, fracin_z, anymask_g, anymask_r, anymask_z,<br />
allmask_g, allmask_r, allmask_z, wisemask_w1, wisemask_w2, psfsize_g, psfsize_r, psfsize_z,<br />
psfdepth_g, psfdepth_r, psfdepth_z, galdepth_g, galdepth_r, galdepth_z, psfdepth_w1, psfdepth_w2,<br />
psfdepth_w3, psfdepth_w4, wise_coadd_id, lc_flux_w1, lc_flux_w2, lc_flux_ivar_w1, lc_flux_ivar_w2,<br />
lc_nobs_w1, lc_nobs_w2, lc_fracflux_w1, lc_fracflux_w2, lc_rchisq_w1, lc_rchisq_w2, lc_mjd_w1,<br />
lc_mjd_w2, fracdev, fracdev_ivar, shapeexp_r, shapeexp_r_ivar, shapeexp_e1, shapeexp_e1_ivar,<br />
shapeexp_e2, shapeexp_e2_ivar, shapedev_r, shapedev_r_ivar, shapedev_e1, shapedev_e1_ivar,<br />
shapedev_e2, shapedev_e2_ivar<br />
from tallada.dr8<br />
;<br />
<br />
=== Comments ===<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE release release smallint COMMENT "Unique integer denoting the camera and filter set used";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickid brickid int COMMENT "Brick ID [1,662174]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickname brickname string COMMENT "Name of brick, encoding the brick sky position, eg \"1126p222\" near RA=112.6, Dec=+22.2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE objid objid int COMMENT "Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brick_primary brick_primary boolean COMMENT "True if the object is within the brick boundary";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brightblob brightblob smallint COMMENT "bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE maskbits maskbits smallint COMMENT "bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits*maps, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE type type string COMMENT "Morphological model: PSF=stellar, REX=round exponential galaxy, DEV=deVauc, EXP=exponential, COMP=composite, DUP=Gaia source fit by different model.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra ra double COMMENT "Right ascension at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec dec double COMMENT "Declination at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra_ivar ra_ivar float COMMENT "Inverse variance of RA (no cosine term!), excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec_ivar dec_ivar float COMMENT "Inverse variance of DEC, excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE bx bx float COMMENT "X position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE by by float COMMENT "Y position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dchisq dchisq array<float> COMMENT "Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ebv ebv float COMMENT "Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_min mjd_min double COMMENT "Minimum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_max mjd_max double COMMENT "Maximum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_cat ref_cat string COMMENT "Reference catalog source for this star: T2 for Tycho-2, G2 for Gaia DR2, L2 for the LSLGA, empty otherwise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_id ref_id bigint COMMENT "Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b \"sourceid\" for Gaia-DR2 and LSLGA";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra pmra float COMMENT "Reference catalog proper motion in the RA direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec pmdec float COMMENT "Reference catalog proper motion in the Dec direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax parallax float COMMENT "Reference catalog parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra_ivar pmra_ivar float COMMENT "Reference catalog inverse-variance on pmra";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec_ivar pmdec_ivar float COMMENT "Reference catalog inverse-variance on pmdec";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax_ivar parallax_ivar float COMMENT "Reference catalog inverse-variance on parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_epoch ref_epoch float COMMENT "Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_pointsource gaia_pointsource boolean COMMENT "This Gaia DR2 source is believed to be a star, not a galaxy";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_mag gaia_phot_g_mean_mag float COMMENT "Gaia G band mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_flux_over_error gaia_phot_g_mean_flux_over_error float COMMENT "Gaia G band signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_n_obs gaia_phot_g_n_obs smallint COMMENT "Gaia G band number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_mag gaia_phot_bp_mean_mag float COMMENT "Gaia BP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_flux_over_error gaia_phot_bp_mean_flux_over_error float COMMENT "Gaia BP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_n_obs gaia_phot_bp_n_obs smallint COMMENT "Gaia BP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_mag gaia_phot_rp_mean_mag float COMMENT "Gaia RP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_flux_over_error gaia_phot_rp_mean_flux_over_error float COMMENT "Gaia RP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_n_obs gaia_phot_rp_n_obs smallint COMMENT "Gaia RP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_variable_flag gaia_phot_variable_flag boolean COMMENT "Gaia photometric variable flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise gaia_astrometric_excess_noise float COMMENT "Gaia astrometric excess noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise_sig gaia_astrometric_excess_noise_sig float COMMENT "Gaia astrometric excess noise uncertainty";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_obs_al gaia_astrometric_n_obs_al smallint COMMENT "Gaia number of astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_good_obs_al gaia_astrometric_n_good_obs_al smallint COMMENT "Gaia number of good astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_weight_al gaia_astrometric_weight_al float COMMENT "Gaia astrometric weight along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_duplicated_source gaia_duplicated_source boolean COMMENT "Gaia duplicated source flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_a_g_val gaia_a_g_val float COMMENT "Gaia line-of-sight extinction in the G band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_e_bp_min_rp_val gaia_e_bp_min_rp_val float COMMENT "Gaia line-of-sight reddening E(BP-RP)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_rp_excess_factor gaia_phot_bp_rp_excess_factor float COMMENT "Gaia BP/RP excess factor";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_sigma5d_max gaia_astrometric_sigma5d_max float COMMENT "Gaia longest semi-major axis of the 5-d error ellipsoid";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_params_solved gaia_astrometric_params_solved tinyint COMMENT "which astrometric parameters were estimated for a Gaia source";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_g flux_g float COMMENT "model flux in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_r flux_r float COMMENT "model flux in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_z flux_z float COMMENT "model flux in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w1 flux_w1 float COMMENT "WISE model flux in W1W1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w2 flux_w2 float COMMENT "WISE model flux in W2W2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w3 flux_w3 float COMMENT "WISE model flux in W3W3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w4 flux_w4 float COMMENT "WISE model flux in W4W4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_g flux_ivar_g float COMMENT "Inverse variance of flux_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_r flux_ivar_r float COMMENT "Inverse variance of flux_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_z flux_ivar_z float COMMENT "Inverse variance of flux_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w1 flux_ivar_w1 float COMMENT "Inverse variance of flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w2 flux_ivar_w2 float COMMENT "Inverse variance of flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w3 flux_ivar_w3 float COMMENT "Inverse variance of flux_w3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w4 flux_ivar_w4 float COMMENT "Inverse variance of flux_w4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_g fiberflux_g float COMMENT "Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_r fiberflux_r float COMMENT "Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_z fiberflux_z float COMMENT "Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_g fibertotflux_g float COMMENT "Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_r fibertotflux_r float COMMENT "Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_z fibertotflux_z float COMMENT "Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_g apflux_g array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_r apflux_r array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_z apflux_z array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_g apflux_resid_g array<float> COMMENT "aperture fluxes on the co-added residual images in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_r apflux_resid_r array<float> COMMENT "aperture fluxes on the co-added residual images in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_z apflux_resid_z array<float> COMMENT "aperture fluxes on the co-added residual images in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_g apflux_ivar_g array<float> COMMENT "Inverse variance of apflux_resid_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_r apflux_ivar_r array<float> COMMENT "Inverse variance of apflux_resid_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_z apflux_ivar_z array<float> COMMENT "Inverse variance of apflux_resid_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_g mw_transmission_g float COMMENT "Galactic transmission in gg filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_r mw_transmission_r float COMMENT "Galactic transmission in rr filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_z mw_transmission_z float COMMENT "Galactic transmission in zz filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w1 mw_transmission_w1 float COMMENT "Galactic transmission in W1W1 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w2 mw_transmission_w2 float COMMENT "Galactic transmission in W2W2 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w3 mw_transmission_w3 float COMMENT "Galactic transmission in W3W3 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w4 mw_transmission_w4 float COMMENT "Galactic transmission in W4W4 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_g nobs_g smallint COMMENT "Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_r nobs_r smallint COMMENT "Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_z nobs_z smallint COMMENT "Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w1 nobs_w1 smallint COMMENT "Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w2 nobs_w2 smallint COMMENT "Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w3 nobs_w3 smallint COMMENT "Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w4 nobs_w4 smallint COMMENT "Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_g rchisq_g float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_r rchisq_r float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_z rchisq_z float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w1 rchisq_w1 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W1W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w2 rchisq_w2 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W2W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w3 rchisq_w3 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W3W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w4 rchisq_w4 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W4W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_g fracflux_g float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in gg(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_r fracflux_r float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in rr(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_z fracflux_z float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in zz(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w1 fracflux_w1 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w2 fracflux_w2 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w3 fracflux_w3 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w4 fracflux_w4 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_g fracmasked_g float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_r fracmasked_r float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_z fracmasked_z float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_g fracin_g float COMMENT "Fraction of a source's flux within the blob in gg, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_r fracin_r float COMMENT "Fraction of a source's flux within the blob in rr, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_z fracin_z float COMMENT "Fraction of a source's flux within the blob in zz, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_g anymask_g smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_r anymask_r smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_z anymask_z smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_g allmask_g smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_r allmask_r smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_z allmask_z smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w1 wisemask_w1 tinyint COMMENT "W1 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w2 wisemask_w2 tinyint COMMENT "W2 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_g psfsize_g float COMMENT "Weighted average PSF FWHM in the gg band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_r psfsize_r float COMMENT "Weighted average PSF FWHM in the rr band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_z psfsize_z float COMMENT "Weighted average PSF FWHM in the zz band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_g psfdepth_g float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_r psfdepth_r float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_z psfdepth_z float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_g galdepth_g float COMMENT "As for psfdepth_g but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_r galdepth_r float COMMENT "As for psfdepth_r but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_z galdepth_z float COMMENT "As for psfdepth_z but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w1 psfdepth_w1 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w2 psfdepth_w2 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w3 psfdepth_w3 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w4 psfdepth_w4 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wise_coadd_id wise_coadd_id string COMMENT "unWISE coadd file name for the center of each object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w1 lc_flux_w1 array<float> COMMENT "flux_w1 in each of up to eleven unWISE coadd epochs (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w2 lc_flux_w2 array<float> COMMENT "flux_w2 in each of up to eleven unWISE coadd epochs (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w1 lc_flux_ivar_w1 array<float> COMMENT "Inverse variance of lc_flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w2 lc_flux_ivar_w2 array<float> COMMENT "Inverse variance of lc_flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w1 lc_nobs_w1 array<smallint> COMMENT "nobs_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w2 lc_nobs_w2 array<smallint> COMMENT "nobs_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w1 lc_fracflux_w1 array<float> COMMENT "fracflux_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w2 lc_fracflux_w2 array<float> COMMENT "fracflux_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w1 lc_rchisq_w1 array<float> COMMENT "rchisq_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w2 lc_rchisq_w2 array<float> COMMENT "rchisq_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w1 lc_mjd_w1 array<double> COMMENT "mjd_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w2 lc_mjd_w2 array<double> COMMENT "mjd_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev fracdev float COMMENT "Fraction of model in deVauc [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev_ivar fracdev_ivar float COMMENT "Inverse variance of fracdev";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r shapeexp_r float COMMENT "Half-light radius of exponential model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r_ivar shapeexp_r_ivar float COMMENT "Inverse variance of shapeexp_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1 shapeexp_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1_ivar shapeexp_e1_ivar float COMMENT "Inverse variance of shapeexp_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2 shapeexp_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2_ivar shapeexp_e2_ivar float COMMENT "Inverse variance of shapeexp_e2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r shapedev_r float COMMENT "Half-light radius of deVaucouleurs model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r_ivar shapedev_r_ivar float COMMENT "Inverse variance of shapedev_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1 shapedev_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1_ivar shapedev_e1_ivar float COMMENT "Inverse variance of shapedev_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2 shapedev_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2_ivar shapedev_e2_ivar float COMMENT "Inverse variance of shapedev_e2";<br />
<br />
== Sweep photo-z ==<br />
<br />
=== Raw data ===<br />
<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/north/sweep/<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/south/sweep/<br />
<br />
Checksums checked:<br />
<br />
$ cd north/sweep/8.0<br />
$ ls | wc -l<br />
287<br />
$ wc -l legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
286 (+ checksum file)<br />
$ sha256sum -c legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
OK<br />
<br />
$ cd north/sweep/8.0-photo-z<br />
$ ls | wc -l<br />
287<br />
$ wc -l legacysurvey_dr8_north_sweep_8.0-photo-z.sha256sum<br />
286 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_north_sweep_8.0-photo-z.sha256sum<br />
OK<br />
<br />
$ cd south/sweep/8.0<br />
$ ls | wc -l<br />
438<br />
$ wc -l legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
437 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
OK<br />
<br />
$ cd south/sweep/8.0-photo-z<br />
$ ls | wc -l<br />
438<br />
$ wc -l legacysurvey_dr8_south_sweep_8.0-photo-z.sha256sum<br />
437 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_south_sweep_8.0-photo-z.sha256sum<br />
OK<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to merge fits files and convert to parquet:<br />
<br />
path_base = '/cephfs/pic.es/astro/scratch/tallada/dr8-phz/'<br />
hdfs_base = '/user/tallada/data/legacysurvey-sweep'<br />
<br />
hs = ['south/sweep'] #, 'north/sweep']<br />
dir_src = '8.0'<br />
dir_phz = '8.0-photo-z'<br />
dir_dst = '8.0-merged'<br />
<br />
my_files = sorted(os.listdir(os.path.join(path_base, hs[0], dir_src)))<br />
<br />
def merge_phz(file_name):<br />
file_src = os.path.join(path_base, hs[0], dir_src, file_name)<br />
file_phz = os.path.join(path_base, hs[0], dir_phz, file_name[:-5]+'-pz.fits')<br />
file_dst = os.path.join(path_base, hs[0], dir_dst, file_name[:-5]+'-merged.fits')<br />
<br />
t_src = table.Table.read(file_src)<br />
t_phz = table.Table.read(file_phz)<br />
t_dst = table.hstack([t_src, t_phz], join_type='exact')<br />
<br />
os.unlink(file_dst)<br />
t_dst.write(file_dst)<br />
<br />
return file_name<br />
<br />
df = sc.parallelize(my_files, len(my_files))<br />
<br />
df.map(merge_phz).count()<br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + os.path.join(path_base, hs[0], dir_dst)<br />
) <br />
<br />
df.write.parquet(os.path.join(hdfs_base, hs[0]))<br />
<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE tallada.dr8_sweep (<br />
`RELEASE` SMALLINT,<br />
`BRICKID` INT,<br />
`BRICKNAME` STRING,<br />
`OBJID` INT,<br />
`TYPE` STRING,<br />
`RA` DOUBLE,<br />
`DEC` DOUBLE,<br />
`RA_IVAR` FLOAT,<br />
`DEC_IVAR` FLOAT,<br />
`DCHISQ` ARRAY<FLOAT>,<br />
`EBV` FLOAT,<br />
`FLUX_G` FLOAT,<br />
`FLUX_R` FLOAT,<br />
`FLUX_Z` FLOAT,<br />
`FLUX_W1` FLOAT,<br />
`FLUX_W2` FLOAT,<br />
`FLUX_W3` FLOAT,<br />
`FLUX_W4` FLOAT,<br />
`FLUX_IVAR_G` FLOAT,<br />
`FLUX_IVAR_R` FLOAT,<br />
`FLUX_IVAR_Z` FLOAT,<br />
`FLUX_IVAR_W1` FLOAT,<br />
`FLUX_IVAR_W2` FLOAT,<br />
`FLUX_IVAR_W3` FLOAT,<br />
`FLUX_IVAR_W4` FLOAT,<br />
`MW_TRANSMISSION_G` FLOAT,<br />
`MW_TRANSMISSION_R` FLOAT,<br />
`MW_TRANSMISSION_Z` FLOAT,<br />
`MW_TRANSMISSION_W1` FLOAT,<br />
`MW_TRANSMISSION_W2` FLOAT,<br />
`MW_TRANSMISSION_W3` FLOAT,<br />
`MW_TRANSMISSION_W4` FLOAT,<br />
`NOBS_G` INT,<br />
`NOBS_R` INT,<br />
`NOBS_Z` INT,<br />
`NOBS_W1` INT,<br />
`NOBS_W2` INT,<br />
`NOBS_W3` INT,<br />
`NOBS_W4` INT,<br />
`RCHISQ_G` FLOAT,<br />
`RCHISQ_R` FLOAT,<br />
`RCHISQ_Z` FLOAT,<br />
`RCHISQ_W1` FLOAT,<br />
`RCHISQ_W2` FLOAT,<br />
`RCHISQ_W3` FLOAT,<br />
`RCHISQ_W4` FLOAT,<br />
`FRACFLUX_G` FLOAT,<br />
`FRACFLUX_R` FLOAT,<br />
`FRACFLUX_Z` FLOAT,<br />
`FRACFLUX_W1` FLOAT,<br />
`FRACFLUX_W2` FLOAT,<br />
`FRACFLUX_W3` FLOAT,<br />
`FRACFLUX_W4` FLOAT,<br />
`FRACMASKED_G` FLOAT,<br />
`FRACMASKED_R` FLOAT,<br />
`FRACMASKED_Z` FLOAT,<br />
`FRACIN_G` FLOAT,<br />
`FRACIN_R` FLOAT,<br />
`FRACIN_Z` FLOAT,<br />
`ANYMASK_G` SMALLINT,<br />
`ANYMASK_R` SMALLINT,<br />
`ANYMASK_Z` SMALLINT,<br />
`ALLMASK_G` SMALLINT,<br />
`ALLMASK_R` SMALLINT,<br />
`ALLMASK_Z` SMALLINT,<br />
`WISEMASK_W1` TINYINT,<br />
`WISEMASK_W2` TINYINT,<br />
`PSFSIZE_G` FLOAT,<br />
`PSFSIZE_R` FLOAT,<br />
`PSFSIZE_Z` FLOAT,<br />
`PSFDEPTH_G` FLOAT,<br />
`PSFDEPTH_R` FLOAT,<br />
`PSFDEPTH_Z` FLOAT,<br />
`GALDEPTH_G` FLOAT,<br />
`GALDEPTH_R` FLOAT,<br />
`GALDEPTH_Z` FLOAT,<br />
`PSFDEPTH_W1` FLOAT,<br />
`PSFDEPTH_W2` FLOAT,<br />
`WISE_COADD_ID` STRING,<br />
`FRACDEV` FLOAT,<br />
`FRACDEV_IVAR` FLOAT,<br />
`SHAPEDEV_R` FLOAT,<br />
`SHAPEDEV_R_IVAR` FLOAT,<br />
`SHAPEDEV_E1` FLOAT,<br />
`SHAPEDEV_E1_IVAR` FLOAT,<br />
`SHAPEDEV_E2` FLOAT,<br />
`SHAPEDEV_E2_IVAR` FLOAT,<br />
`SHAPEEXP_R` FLOAT,<br />
`SHAPEEXP_R_IVAR` FLOAT,<br />
`SHAPEEXP_E1` FLOAT,<br />
`SHAPEEXP_E1_IVAR` FLOAT,<br />
`SHAPEEXP_E2` FLOAT,<br />
`SHAPEEXP_E2_IVAR` FLOAT,<br />
`FIBERFLUX_G` FLOAT,<br />
`FIBERFLUX_R` FLOAT,<br />
`FIBERFLUX_Z` FLOAT,<br />
`FIBERTOTFLUX_G` FLOAT,<br />
`FIBERTOTFLUX_R` FLOAT,<br />
`FIBERTOTFLUX_Z` FLOAT,<br />
`REF_CAT` STRING,<br />
`REF_ID` BIGINT,<br />
`REF_EPOCH` FLOAT,<br />
`GAIA_PHOT_G_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_G_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_ASTROMETRIC_EXCESS_NOISE` FLOAT,<br />
`GAIA_DUPLICATED_SOURCE` BOOLEAN,<br />
`GAIA_PHOT_BP_RP_EXCESS_FACTOR` FLOAT,<br />
`GAIA_ASTROMETRIC_SIGMA5D_MAX` FLOAT,<br />
`GAIA_ASTROMETRIC_PARAMS_SOLVED` TINYINT,<br />
`PARALLAX` FLOAT,<br />
`PARALLAX_IVAR` FLOAT,<br />
`PMRA` FLOAT,<br />
`PMRA_IVAR` FLOAT,<br />
`PMDEC` FLOAT,<br />
`PMDEC_IVAR` FLOAT,<br />
`MASKBITS` SMALLINT,<br />
`z_phot_mean` FLOAT,<br />
`z_phot_median` FLOAT,<br />
`z_phot_std` FLOAT,<br />
`z_phot_l68` FLOAT,<br />
`z_phot_u68` FLOAT,<br />
`z_phot_l95` FLOAT,<br />
`z_phot_u95` FLOAT,<br />
`z_spec` FLOAT,<br />
`survey` STRING,<br />
`training` BOOLEAN<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey-sweep'<br />
;<br />
<br />
=== Combined table ===<br />
<br />
CREATE TABLE cosmohub.legacysurvey_dr8_phz (<br />
release smallint COMMENT 'Unique integer denoting the camera and filter set used',<br />
brickid int COMMENT 'Brick ID [1,662174]',<br />
brickname string COMMENT 'Name of brick, encoding the brick sky position, eg "1126p222" near RA=112.6, Dec=+22.2',<br />
objid int COMMENT 'Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick',<br />
brick_primary boolean COMMENT 'True if the object is within the brick boundary',<br />
brightblob smallint COMMENT 'bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page',<br />
maskbits smallint COMMENT 'bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits* maps, as cataloged on the DR8 bitmasks page',<br />
type string COMMENT 'Morphological model: "PSF"=stellar, "REX"="round exponential galaxy", "DEV"=deVauc, "EXP"=exponential, "COMP"=composite, "DUP"=Gaia source fit by different model. Note that in some FITS readers, a trailing space may be appended for "PSF ", "DEV " and "EXP " since the column data type is a 4-character string',<br />
ra double COMMENT 'Right ascension at equinox J2000',<br />
dec double COMMENT 'Declination at equinox J2000',<br />
ra_ivar float COMMENT 'Inverse variance of RA (no cosine term!), excluding astrometric calibration errors',<br />
dec_ivar float COMMENT 'Inverse variance of DEC, excluding astrometric calibration errors',<br />
bx float COMMENT 'X position (0-indexed) of coordinates in brick image stack',<br />
`by` float COMMENT 'Y position (0-indexed) of coordinates in brick image stack',<br />
dchisq array<float> COMMENT 'Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.',<br />
ebv float COMMENT 'Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns',<br />
mjd_min double COMMENT 'Minimum Modified Julian Date of observations used to construct the model of this object',<br />
mjd_max double COMMENT 'Maximum Modified Julian Date of observations used to construct the model of this object',<br />
ref_cat string COMMENT 'Reference catalog source for this star: "T2" for Tycho-2, "G2" for Gaia DR2, "L2" for the LSLGA, empty otherwise',<br />
ref_id bigint COMMENT 'Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b "sourceid" for Gaia-DR2 and LSLGA',<br />
pmra float COMMENT 'Reference catalog proper motion in the RA direction',<br />
pmdec float COMMENT 'Reference catalog proper motion in the Dec direction',<br />
parallax float COMMENT 'Reference catalog parallax',<br />
pmra_ivar float COMMENT 'Reference catalog inverse-variance on pmra',<br />
pmdec_ivar float COMMENT 'Reference catalog inverse-variance on pmdec',<br />
parallax_ivar float COMMENT 'Reference catalog inverse-variance on parallax',<br />
ref_epoch float COMMENT 'Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)',<br />
gaia_pointsource boolean COMMENT 'This Gaia DR2 source is believed to be a star, not a galaxy',<br />
gaia_phot_g_mean_mag float COMMENT 'Gaia G band mag',<br />
gaia_phot_g_mean_flux_over_error float COMMENT 'Gaia G band signal-to-noise',<br />
gaia_phot_g_n_obs smallint COMMENT 'Gaia G band number of observations',<br />
gaia_phot_bp_mean_mag float COMMENT 'Gaia BP mag',<br />
gaia_phot_bp_mean_flux_over_error float COMMENT 'Gaia BP signal-to-noise',<br />
gaia_phot_bp_n_obs smallint COMMENT 'Gaia BP number of observations',<br />
gaia_phot_rp_mean_mag float COMMENT 'Gaia RP mag',<br />
gaia_phot_rp_mean_flux_over_error float COMMENT 'Gaia RP signal-to-noise',<br />
gaia_phot_rp_n_obs smallint COMMENT 'Gaia RP number of observations',<br />
gaia_phot_variable_flag boolean COMMENT 'Gaia photometric variable flag',<br />
gaia_astrometric_excess_noise float COMMENT 'Gaia astrometric excess noise',<br />
gaia_astrometric_excess_noise_sig float COMMENT 'Gaia astrometric excess noise uncertainty',<br />
gaia_astrometric_n_obs_al smallint COMMENT 'Gaia number of astrometric observations along scan direction',<br />
gaia_astrometric_n_good_obs_al smallint COMMENT 'Gaia number of good astrometric observations along scan direction',<br />
gaia_astrometric_weight_al float COMMENT 'Gaia astrometric weight along scan direction',<br />
gaia_duplicated_source boolean COMMENT 'Gaia duplicated source flag',<br />
gaia_a_g_val float COMMENT 'Gaia line-of-sight extinction in the G band',<br />
gaia_e_bp_min_rp_val float COMMENT 'Gaia line-of-sight reddening E(BP-RP)',<br />
gaia_phot_bp_rp_excess_factor float COMMENT 'Gaia BP/RP excess factor',<br />
gaia_astrometric_sigma5d_max float COMMENT 'Gaia longest semi-major axis of the 5-d error ellipsoid',<br />
gaia_astrometric_params_solved tinyint COMMENT 'which astrometric parameters were estimated for a Gaia source',<br />
flux_g float COMMENT 'model flux in gg',<br />
flux_r float COMMENT 'model flux in rr',<br />
flux_z float COMMENT 'model flux in zz',<br />
flux_w1 float COMMENT 'WISE model flux in W1W1 (AB system)',<br />
flux_w2 float COMMENT 'WISE model flux in W2W2 (AB)',<br />
flux_w3 float COMMENT 'WISE model flux in W3W3 (AB)',<br />
flux_w4 float COMMENT 'WISE model flux in W4W4 (AB)',<br />
flux_ivar_g float COMMENT 'Inverse variance of flux_g',<br />
flux_ivar_r float COMMENT 'Inverse variance of flux_r',<br />
flux_ivar_z float COMMENT 'Inverse variance of flux_z',<br />
flux_ivar_w1 float COMMENT 'Inverse variance of flux_w1 (AB system)',<br />
flux_ivar_w2 float COMMENT 'Inverse variance of flux_w2 (AB)',<br />
flux_ivar_w3 float COMMENT 'Inverse variance of flux_w3 (AB)',<br />
flux_ivar_w4 float COMMENT 'Inverse variance of flux_w4 (AB)',<br />
fiberflux_g float COMMENT 'Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_r float COMMENT 'Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_z float COMMENT 'Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fibertotflux_g float COMMENT 'Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_r float COMMENT 'Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_z float COMMENT 'Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
apflux_g array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg',<br />
apflux_r array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr',<br />
apflux_z array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz',<br />
apflux_resid_g array<float> COMMENT 'aperture fluxes on the co-added residual images in gg',<br />
apflux_resid_r array<float> COMMENT 'aperture fluxes on the co-added residual images in rr',<br />
apflux_resid_z array<float> COMMENT 'aperture fluxes on the co-added residual images in zz',<br />
apflux_ivar_g array<float> COMMENT 'Inverse variance of apflux_resid_g',<br />
apflux_ivar_r array<float> COMMENT 'Inverse variance of apflux_resid_r',<br />
apflux_ivar_z array<float> COMMENT 'Inverse variance of apflux_resid_z',<br />
mw_transmission_g float COMMENT 'Galactic transmission in gg filter in linear units [0, 1]',<br />
mw_transmission_r float COMMENT 'Galactic transmission in rr filter in linear units [0, 1]',<br />
mw_transmission_z float COMMENT 'Galactic transmission in zz filter in linear units [0, 1]',<br />
mw_transmission_w1 float COMMENT 'Galactic transmission in W1W1 filter in linear units [0, 1]',<br />
mw_transmission_w2 float COMMENT 'Galactic transmission in W2W2 filter in linear units [0, 1]',<br />
mw_transmission_w3 float COMMENT 'Galactic transmission in W3W3 filter in linear units [0, 1]',<br />
mw_transmission_w4 float COMMENT 'Galactic transmission in W4W4 filter in linear units [0, 1]',<br />
nobs_g smallint COMMENT 'Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)',<br />
nobs_r smallint COMMENT 'Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)',<br />
nobs_z smallint COMMENT 'Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)',<br />
nobs_w1 smallint COMMENT 'Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)',<br />
nobs_w2 smallint COMMENT 'Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)',<br />
nobs_w3 smallint COMMENT 'Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)',<br />
nobs_w4 smallint COMMENT 'Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)',<br />
rchisq_g float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in gg',<br />
rchisq_r float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in rr',<br />
rchisq_z float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in zz',<br />
rchisq_w1 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W1W1',<br />
rchisq_w2 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W2W2',<br />
rchisq_w3 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W3W3',<br />
rchisq_w4 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W4W4',<br />
fracflux_g float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in gg (typically [0,1])',<br />
fracflux_r float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in rr (typically [0,1])',<br />
fracflux_z float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in zz (typically [0,1])',<br />
fracflux_w1 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1 (typically [0,1])',<br />
fracflux_w2 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2 (typically [0,1])',<br />
fracflux_w3 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3 (typically [0,1])',<br />
fracflux_w4 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4 (typically [0,1])',<br />
fracmasked_g float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]',<br />
fracmasked_r float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]',<br />
fracmasked_z float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]',<br />
fracin_g float COMMENT 'Fraction of a source\'s flux within the blob in gg, near unity for real sources',<br />
fracin_r float COMMENT 'Fraction of a source\'s flux within the blob in rr, near unity for real sources',<br />
fracin_z float COMMENT 'Fraction of a source\'s flux within the blob in zz, near unity for real sources',<br />
anymask_g smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page',<br />
anymask_r smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page',<br />
anymask_z smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page',<br />
allmask_g smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page',<br />
allmask_r smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page',<br />
allmask_z smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page',<br />
wisemask_w1 tinyint COMMENT 'W1 bitmask as cataloged on the DR8 bitmasks page',<br />
wisemask_w2 tinyint COMMENT 'W2 bitmask as cataloged on the DR8 bitmasks page',<br />
psfsize_g float COMMENT 'Weighted average PSF FWHM in the gg band',<br />
psfsize_r float COMMENT 'Weighted average PSF FWHM in the rr band',<br />
psfsize_z float COMMENT 'Weighted average PSF FWHM in the zz band',<br />
psfdepth_g float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude',<br />
psfdepth_r float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude',<br />
psfdepth_z float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude',<br />
galdepth_g float COMMENT 'As for psfdepth_g but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_r float COMMENT 'As for psfdepth_r but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_z float COMMENT 'As for psfdepth_z but for a galaxy (0.45" exp, round) detection sensitivity',<br />
psfdepth_w1 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W1',<br />
psfdepth_w2 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W2',<br />
psfdepth_w3 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W3',<br />
psfdepth_w4 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W4',<br />
wise_coadd_id string COMMENT 'unWISE coadd file name for the center of each object',<br />
lc_flux_w1 array<float> COMMENT 'flux_w1 in each of up to eleven unWISE coadd epochs (AB system)',<br />
lc_flux_w2 array<float> COMMENT 'flux_w2 in each of up to eleven unWISE coadd epochs (AB)',<br />
lc_flux_ivar_w1 array<float> COMMENT 'Inverse variance of lc_flux_w1 (AB system)',<br />
lc_flux_ivar_w2 array<float> COMMENT 'Inverse variance of lc_flux_w2 (AB)',<br />
lc_nobs_w1 array<smallint> COMMENT 'nobs_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_nobs_w2 array<smallint> COMMENT 'nobs_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w1 array<float> COMMENT 'fracflux_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w2 array<float> COMMENT 'fracflux_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w1 array<float> COMMENT 'rchisq_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w2 array<float> COMMENT 'rchisq_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w1 array<double> COMMENT 'mjd_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w2 array<double> COMMENT 'mjd_w2 in each of up to eleven unWISE coadd epochs',<br />
fracdev float COMMENT 'Fraction of model in deVauc [0,1]',<br />
fracdev_ivar float COMMENT 'Inverse variance of fracdev',<br />
shapeexp_r float COMMENT 'Half-light radius of exponential model (>0)',<br />
shapeexp_r_ivar float COMMENT 'Inverse variance of shapeexp_r',<br />
shapeexp_e1 float COMMENT 'Ellipticity component 1',<br />
shapeexp_e1_ivar float COMMENT 'Inverse variance of shapeexp_e1',<br />
shapeexp_e2 float COMMENT 'Ellipticity component 2',<br />
shapeexp_e2_ivar float COMMENT 'Inverse variance of shapeexp_e2',<br />
shapedev_r float COMMENT 'Half-light radius of deVaucouleurs model (>0)',<br />
shapedev_r_ivar float COMMENT 'Inverse variance of shapedev_r',<br />
shapedev_e1 float COMMENT 'Ellipticity component 1',<br />
shapedev_e1_ivar float COMMENT 'Inverse variance of shapedev_e1',<br />
shapedev_e2 float COMMENT 'Ellipticity component 2',<br />
shapedev_e2_ivar float COMMENT 'Inverse variance of shapedev_e2',<br />
z_phot_mean FLOAT COMMENT 'photo-z derived from the mean of the photo-z PDF ',<br />
z_phot_median FLOAT COMMENT 'photo-z derived from the median of the photo-z PDF',<br />
z_phot_std FLOAT COMMENT 'standard deviation of the photo-z\'s derived from the photo-z PDF',<br />
z_phot_l68 FLOAT COMMENT 'lower bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_u68 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_l95 FLOAT COMMENT 'lower bound of the 95% confidence region, derived from the photo-z PDF',<br />
z_phot_u95 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_spec FLOAT COMMENT 'spectroscopic redshift, if available',<br />
survey STRING COMMENT 'source of the spectroscopic redshift',<br />
training BOOLEAN COMMENT 'whether or not the spectroscopic redshift is used in photometric redshift training'<br />
)<br />
CLUSTERED BY (<br />
release,<br />
brickid,<br />
objid)<br />
SORTED BY (<br />
release ASC,<br />
brickid ASC,<br />
objid ASC)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
=== Join insert ===<br />
<br />
INSERT OVERWRITE TABLE cosmohub.legacy_survey_dr8_phz<br />
SELECT<br />
dr8.*,<br />
phz.z_phot_mean,<br />
phz.z_phot_median,<br />
phz.z_phot_std,<br />
phz.z_phot_l68,<br />
phz.z_phot_u68,<br />
phz.z_phot_l95,<br />
phz.z_phot_u95,<br />
phz.z_spec,<br />
phz.survey,<br />
phz.training<br />
FROM cosmohub.legacy_survey_dr8 AS dr8<br />
LEFT JOIN tallada.dr8_sweep AS phz<br />
ON dr8.release = phz.release<br />
AND dr8.brickid = phz.brickid<br />
AND dr8.objid = phz.objid<br />
;<br />
<br />
== Queries for checking ==<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT release, brickid, objid, COUNT(release)<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
HAVING COUNT(release) > 1<br />
<br />
DR8 release brickid objid: DISTINCT(1.694.230.368) TOTAL(1.694.230.368)<br />
DR8 brickid objid: DISTINCT(1.654.429.115) TOTAL(1.694.230.368)<br />
<br />
PHZ release brickid objid: DISTINCT(1.646.664.729) TOTAL(1.646.664.897)<br />
PHZ brickid objid: DISTINCT(1.608.841.582) TOTAL(1.646.664.897)<br />
<br />
DR8: 1.694.230.368<br />
PHZ: 1.646.664.897<br />
----------------------------------------------------------------------<br />
-47.565.471</div>
Tallada
https://pwiki.pic.es/index.php?title=Legacy_Survey_DR8&diff=802
Legacy Survey DR8
2020-05-09T11:05:18Z
<p>Tallada: /* Sweep photo-z */</p>
<hr />
<div>== Original DR8 ==<br />
=== Raw data ===<br />
<br />
Downloaded from<br />
ftp://archive.noao.edu:/public/hlsp/ls/dr8/<br />
<br />
Directories:<br />
south/tractor<br />
north/tractor<br />
<br />
Mirrored using lftp commands:<br />
mirror -c -v -P 200 north/tractor/ /mnt/pau/north/tractor/<br />
mirror -c -v -P 200 south/tractor/ /mnt/pau/south/tractor/<br />
<br />
into a 5 TiB temporary Ceph filesystem mounted in tdm002.<br />
<br />
Checksums provided through sha256sum files, and checked using command:<br />
find * -type d | xargs -t -L1 -P24 sh -c 'cd $1; sha256sum --quiet -c *.sha256sum' cksum<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to convert fits files to parquet:<br />
<br />
path_base = '/cephfs/pic.es/cosmohub/legacysurvey'<br />
hdfs_base = '/user/tallada/data/legacysurvey'<br />
<br />
hs = ['h=north', 'h=south']<br />
rs = ['r={0:03d}'.format(x) for x in range(360)]<br />
<br />
for h in hs:<br />
for r in rs:<br />
p = os.path.join(path_base, h, r)<br />
if not os.path.exists(p):<br />
continue<br />
<br />
size = sum(os.path.getsize(os.path.join(p,f)) for f in os.listdir(p) if os.path.isfile(os.path.join(p,f)))<br />
<br />
print h, r, size <br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + p<br />
)<br />
<br />
df = df.repartition(int(math.ceil(size / (256*1024*1024.0))))<br />
<br />
df.write.parquet(os.path.join(hdfs_base, h, r))<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE tallada.dr8 (<br />
`release` smallint,<br />
`brickid` int,<br />
`brickname` string,<br />
`objid` int,<br />
`brick_primary` boolean,<br />
`brightblob` smallint,<br />
`maskbits` smallint,<br />
`type` string,<br />
`ra` double,<br />
`dec` double,<br />
`ra_ivar` float,<br />
`dec_ivar` float,<br />
`bx` float,<br />
`by` float,<br />
`dchisq` array<float>,<br />
`ebv` float,<br />
`mjd_min` double,<br />
`mjd_max` double,<br />
`ref_cat` string,<br />
`ref_id` bigint,<br />
`pmra` float,<br />
`pmdec` float,<br />
`parallax` float,<br />
`pmra_ivar` float,<br />
`pmdec_ivar` float,<br />
`parallax_ivar` float,<br />
`ref_epoch` float,<br />
`gaia_pointsource` boolean,<br />
`gaia_phot_g_mean_mag` float,<br />
`gaia_phot_g_mean_flux_over_error` float,<br />
`gaia_phot_g_n_obs` smallint,<br />
`gaia_phot_bp_mean_mag` float,<br />
`gaia_phot_bp_mean_flux_over_error` float,<br />
`gaia_phot_bp_n_obs` smallint,<br />
`gaia_phot_rp_mean_mag` float,<br />
`gaia_phot_rp_mean_flux_over_error` float,<br />
`gaia_phot_rp_n_obs` smallint,<br />
`gaia_phot_variable_flag` boolean,<br />
`gaia_astrometric_excess_noise` float,<br />
`gaia_astrometric_excess_noise_sig` float,<br />
`gaia_astrometric_n_obs_al` smallint,<br />
`gaia_astrometric_n_good_obs_al` smallint,<br />
`gaia_astrometric_weight_al` float,<br />
`gaia_duplicated_source` boolean,<br />
`gaia_a_g_val` float,<br />
`gaia_e_bp_min_rp_val` float,<br />
`gaia_phot_bp_rp_excess_factor` float,<br />
`gaia_astrometric_sigma5d_max` float,<br />
`gaia_astrometric_params_solved` tinyint,<br />
`flux_g` float,<br />
`flux_r` float,<br />
`flux_z` float,<br />
`flux_w1` float,<br />
`flux_w2` float,<br />
`flux_w3` float,<br />
`flux_w4` float,<br />
`flux_ivar_g` float,<br />
`flux_ivar_r` float,<br />
`flux_ivar_z` float,<br />
`flux_ivar_w1` float,<br />
`flux_ivar_w2` float,<br />
`flux_ivar_w3` float,<br />
`flux_ivar_w4` float,<br />
`fiberflux_g` float,<br />
`fiberflux_r` float,<br />
`fiberflux_z` float,<br />
`fibertotflux_g` float,<br />
`fibertotflux_r` float,<br />
`fibertotflux_z` float,<br />
`apflux_g` array<float>,<br />
`apflux_r` array<float>,<br />
`apflux_z` array<float>,<br />
`apflux_resid_g` array<float>,<br />
`apflux_resid_r` array<float>,<br />
`apflux_resid_z` array<float>,<br />
`apflux_ivar_g` array<float>,<br />
`apflux_ivar_r` array<float>,<br />
`apflux_ivar_z` array<float>,<br />
`mw_transmission_g` float,<br />
`mw_transmission_r` float,<br />
`mw_transmission_z` float,<br />
`mw_transmission_w1` float,<br />
`mw_transmission_w2` float,<br />
`mw_transmission_w3` float,<br />
`mw_transmission_w4` float,<br />
`nobs_g` smallint,<br />
`nobs_r` smallint,<br />
`nobs_z` smallint,<br />
`nobs_w1` smallint,<br />
`nobs_w2` smallint,<br />
`nobs_w3` smallint,<br />
`nobs_w4` smallint,<br />
`rchisq_g` float,<br />
`rchisq_r` float,<br />
`rchisq_z` float,<br />
`rchisq_w1` float,<br />
`rchisq_w2` float,<br />
`rchisq_w3` float,<br />
`rchisq_w4` float,<br />
`fracflux_g` float,<br />
`fracflux_r` float,<br />
`fracflux_z` float,<br />
`fracflux_w1` float,<br />
`fracflux_w2` float,<br />
`fracflux_w3` float,<br />
`fracflux_w4` float,<br />
`fracmasked_g` float,<br />
`fracmasked_r` float,<br />
`fracmasked_z` float,<br />
`fracin_g` float,<br />
`fracin_r` float,<br />
`fracin_z` float,<br />
`anymask_g` smallint,<br />
`anymask_r` smallint,<br />
`anymask_z` smallint,<br />
`allmask_g` smallint,<br />
`allmask_r` smallint,<br />
`allmask_z` smallint,<br />
`wisemask_w1` tinyint,<br />
`wisemask_w2` tinyint,<br />
`psfsize_g` float,<br />
`psfsize_r` float,<br />
`psfsize_z` float,<br />
`psfdepth_g` float,<br />
`psfdepth_r` float,<br />
`psfdepth_z` float,<br />
`galdepth_g` float,<br />
`galdepth_r` float,<br />
`galdepth_z` float,<br />
`psfdepth_w1` float,<br />
`psfdepth_w2` float,<br />
`psfdepth_w3` float,<br />
`psfdepth_w4` float,<br />
`wise_coadd_id` string,<br />
`lc_flux_w1` array<float>,<br />
`lc_flux_w2` array<float>,<br />
`lc_flux_ivar_w1` array<float>,<br />
`lc_flux_ivar_w2` array<float>,<br />
`lc_nobs_w1` array<smallint>,<br />
`lc_nobs_w2` array<smallint>,<br />
`lc_fracflux_w1` array<float>,<br />
`lc_fracflux_w2` array<float>,<br />
`lc_rchisq_w1` array<float>,<br />
`lc_rchisq_w2` array<float>,<br />
`lc_mjd_w1` array<double>,<br />
`lc_mjd_w2` array<double>,<br />
`fracdev` float,<br />
`fracdev_ivar` float,<br />
`shapeexp_r` float,<br />
`shapeexp_r_ivar` float,<br />
`shapeexp_e1` float,<br />
`shapeexp_e1_ivar` float,<br />
`shapeexp_e2` float,<br />
`shapeexp_e2_ivar` float,<br />
`shapedev_r` float,<br />
`shapedev_r_ivar` float,<br />
`shapedev_e1` float,<br />
`shapedev_e1_ivar` float,<br />
`shapedev_e2` float,<br />
`shapedev_e2_ivar` float<br />
)<br />
PARTITIONED BY (<br />
h string,<br />
r string<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey/dr8/'<br />
;<br />
<br />
=== Internal Table ===<br />
<br />
CREATE TABLE cosmohub.legacy_survey_dr8 (<br />
`release` smallint,<br />
`brickid` int,<br />
`brickname` string,<br />
`objid` int,<br />
`brick_primary` boolean,<br />
`brightblob` smallint,<br />
`maskbits` smallint,<br />
`type` string,<br />
`ra` double,<br />
`dec` double,<br />
`ra_ivar` float,<br />
`dec_ivar` float,<br />
`bx` float,<br />
`by` float,<br />
`dchisq` array<float>,<br />
`ebv` float,<br />
`mjd_min` double,<br />
`mjd_max` double,<br />
`ref_cat` string,<br />
`ref_id` bigint,<br />
`pmra` float,<br />
`pmdec` float,<br />
`parallax` float,<br />
`pmra_ivar` float,<br />
`pmdec_ivar` float,<br />
`parallax_ivar` float,<br />
`ref_epoch` float,<br />
`gaia_pointsource` boolean,<br />
`gaia_phot_g_mean_mag` float,<br />
`gaia_phot_g_mean_flux_over_error` float,<br />
`gaia_phot_g_n_obs` smallint,<br />
`gaia_phot_bp_mean_mag` float,<br />
`gaia_phot_bp_mean_flux_over_error` float,<br />
`gaia_phot_bp_n_obs` smallint,<br />
`gaia_phot_rp_mean_mag` float,<br />
`gaia_phot_rp_mean_flux_over_error` float,<br />
`gaia_phot_rp_n_obs` smallint,<br />
`gaia_phot_variable_flag` boolean,<br />
`gaia_astrometric_excess_noise` float,<br />
`gaia_astrometric_excess_noise_sig` float,<br />
`gaia_astrometric_n_obs_al` smallint,<br />
`gaia_astrometric_n_good_obs_al` smallint,<br />
`gaia_astrometric_weight_al` float,<br />
`gaia_duplicated_source` boolean,<br />
`gaia_a_g_val` float,<br />
`gaia_e_bp_min_rp_val` float,<br />
`gaia_phot_bp_rp_excess_factor` float,<br />
`gaia_astrometric_sigma5d_max` float,<br />
`gaia_astrometric_params_solved` tinyint,<br />
`flux_g` float,<br />
`flux_r` float,<br />
`flux_z` float,<br />
`flux_w1` float,<br />
`flux_w2` float,<br />
`flux_w3` float,<br />
`flux_w4` float,<br />
`flux_ivar_g` float,<br />
`flux_ivar_r` float,<br />
`flux_ivar_z` float,<br />
`flux_ivar_w1` float,<br />
`flux_ivar_w2` float,<br />
`flux_ivar_w3` float,<br />
`flux_ivar_w4` float,<br />
`fiberflux_g` float,<br />
`fiberflux_r` float,<br />
`fiberflux_z` float,<br />
`fibertotflux_g` float,<br />
`fibertotflux_r` float,<br />
`fibertotflux_z` float,<br />
`apflux_g` array<float>,<br />
`apflux_r` array<float>,<br />
`apflux_z` array<float>,<br />
`apflux_resid_g` array<float>,<br />
`apflux_resid_r` array<float>,<br />
`apflux_resid_z` array<float>,<br />
`apflux_ivar_g` array<float>,<br />
`apflux_ivar_r` array<float>,<br />
`apflux_ivar_z` array<float>,<br />
`mw_transmission_g` float,<br />
`mw_transmission_r` float,<br />
`mw_transmission_z` float,<br />
`mw_transmission_w1` float,<br />
`mw_transmission_w2` float,<br />
`mw_transmission_w3` float,<br />
`mw_transmission_w4` float,<br />
`nobs_g` smallint,<br />
`nobs_r` smallint,<br />
`nobs_z` smallint,<br />
`nobs_w1` smallint,<br />
`nobs_w2` smallint,<br />
`nobs_w3` smallint,<br />
`nobs_w4` smallint,<br />
`rchisq_g` float,<br />
`rchisq_r` float,<br />
`rchisq_z` float,<br />
`rchisq_w1` float,<br />
`rchisq_w2` float,<br />
`rchisq_w3` float,<br />
`rchisq_w4` float,<br />
`fracflux_g` float,<br />
`fracflux_r` float,<br />
`fracflux_z` float,<br />
`fracflux_w1` float,<br />
`fracflux_w2` float,<br />
`fracflux_w3` float,<br />
`fracflux_w4` float,<br />
`fracmasked_g` float,<br />
`fracmasked_r` float,<br />
`fracmasked_z` float,<br />
`fracin_g` float,<br />
`fracin_r` float,<br />
`fracin_z` float,<br />
`anymask_g` smallint,<br />
`anymask_r` smallint,<br />
`anymask_z` smallint,<br />
`allmask_g` smallint,<br />
`allmask_r` smallint,<br />
`allmask_z` smallint,<br />
`wisemask_w1` tinyint,<br />
`wisemask_w2` tinyint,<br />
`psfsize_g` float,<br />
`psfsize_r` float,<br />
`psfsize_z` float,<br />
`psfdepth_g` float,<br />
`psfdepth_r` float,<br />
`psfdepth_z` float,<br />
`galdepth_g` float,<br />
`galdepth_r` float,<br />
`galdepth_z` float,<br />
`psfdepth_w1` float,<br />
`psfdepth_w2` float,<br />
`psfdepth_w3` float,<br />
`psfdepth_w4` float,<br />
`wise_coadd_id` string,<br />
`lc_flux_w1` array<float>,<br />
`lc_flux_w2` array<float>,<br />
`lc_flux_ivar_w1` array<float>,<br />
`lc_flux_ivar_w2` array<float>,<br />
`lc_nobs_w1` array<smallint>,<br />
`lc_nobs_w2` array<smallint>,<br />
`lc_fracflux_w1` array<float>,<br />
`lc_fracflux_w2` array<float>,<br />
`lc_rchisq_w1` array<float>,<br />
`lc_rchisq_w2` array<float>,<br />
`lc_mjd_w1` array<double>,<br />
`lc_mjd_w2` array<double>,<br />
`fracdev` float,<br />
`fracdev_ivar` float,<br />
`shapeexp_r` float,<br />
`shapeexp_r_ivar` float,<br />
`shapeexp_e1` float,<br />
`shapeexp_e1_ivar` float,<br />
`shapeexp_e2` float,<br />
`shapeexp_e2_ivar` float,<br />
`shapedev_r` float,<br />
`shapedev_r_ivar` float,<br />
`shapedev_e1` float,<br />
`shapedev_e1_ivar` float,<br />
`shapedev_e2` float,<br />
`shapedev_e2_ivar` float<br />
)<br />
clustered by (<br />
release, brickid, objid<br />
)<br />
sorted by (<br />
release, brickid, objid<br />
)<br />
into 4096 buckets<br />
stored as orc<br />
;<br />
<br />
insert overwrite table cosmohub.legacy_survey_dr8<br />
select release, brickid, brickname, objid, brick_primary, brightblob, maskbits, type, ra, dec, ra_ivar, dec_ivar, bx, `by`, <br />
dchisq, ebv, mjd_min, mjd_max, ref_cat, ref_id, pmra, pmdec, parallax, pmra_ivar, pmdec_ivar, parallax_ivar, ref_epoch,<br />
gaia_pointsource, gaia_phot_g_mean_mag, gaia_phot_g_mean_flux_over_error, gaia_phot_g_n_obs,<br />
gaia_phot_bp_mean_mag, gaia_phot_bp_mean_flux_over_error, gaia_phot_bp_n_obs, gaia_phot_rp_mean_mag,<br />
gaia_phot_rp_mean_flux_over_error, gaia_phot_rp_n_obs, gaia_phot_variable_flag, gaia_astrometric_excess_noise,<br />
gaia_astrometric_excess_noise_sig, gaia_astrometric_n_obs_al, gaia_astrometric_n_good_obs_al,<br />
gaia_astrometric_weight_al, gaia_duplicated_source, gaia_a_g_val, gaia_e_bp_min_rp_val,<br />
gaia_phot_bp_rp_excess_factor, gaia_astrometric_sigma5d_max, gaia_astrometric_params_solved,<br />
flux_g, flux_r, flux_z, flux_w1, flux_w2, flux_w3, flux_w4, flux_ivar_g, flux_ivar_r,<br />
flux_ivar_z, flux_ivar_w1, flux_ivar_w2, flux_ivar_w3, flux_ivar_w4, fiberflux_g, fiberflux_r,<br />
fiberflux_z, fibertotflux_g, fibertotflux_r, fibertotflux_z, apflux_g, apflux_r, apflux_z,<br />
apflux_resid_g, apflux_resid_r, apflux_resid_z, apflux_ivar_g, apflux_ivar_r, apflux_ivar_z,<br />
mw_transmission_g, mw_transmission_r, mw_transmission_z, mw_transmission_w1, mw_transmission_w2,<br />
mw_transmission_w3, mw_transmission_w4, nobs_g, nobs_r, nobs_z, nobs_w1, nobs_w2, nobs_w3,<br />
nobs_w4, rchisq_g, rchisq_r, rchisq_z, rchisq_w1, rchisq_w2, rchisq_w3, rchisq_w4, fracflux_g,<br />
fracflux_r, fracflux_z, fracflux_w1, fracflux_w2, fracflux_w3, fracflux_w4, fracmasked_g,<br />
fracmasked_r, fracmasked_z, fracin_g, fracin_r, fracin_z, anymask_g, anymask_r, anymask_z,<br />
allmask_g, allmask_r, allmask_z, wisemask_w1, wisemask_w2, psfsize_g, psfsize_r, psfsize_z,<br />
psfdepth_g, psfdepth_r, psfdepth_z, galdepth_g, galdepth_r, galdepth_z, psfdepth_w1, psfdepth_w2,<br />
psfdepth_w3, psfdepth_w4, wise_coadd_id, lc_flux_w1, lc_flux_w2, lc_flux_ivar_w1, lc_flux_ivar_w2,<br />
lc_nobs_w1, lc_nobs_w2, lc_fracflux_w1, lc_fracflux_w2, lc_rchisq_w1, lc_rchisq_w2, lc_mjd_w1,<br />
lc_mjd_w2, fracdev, fracdev_ivar, shapeexp_r, shapeexp_r_ivar, shapeexp_e1, shapeexp_e1_ivar,<br />
shapeexp_e2, shapeexp_e2_ivar, shapedev_r, shapedev_r_ivar, shapedev_e1, shapedev_e1_ivar,<br />
shapedev_e2, shapedev_e2_ivar<br />
from tallada.dr8<br />
;<br />
<br />
=== Comments ===<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE release release smallint COMMENT "Unique integer denoting the camera and filter set used";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickid brickid int COMMENT "Brick ID [1,662174]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickname brickname string COMMENT "Name of brick, encoding the brick sky position, eg \"1126p222\" near RA=112.6, Dec=+22.2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE objid objid int COMMENT "Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brick_primary brick_primary boolean COMMENT "True if the object is within the brick boundary";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brightblob brightblob smallint COMMENT "bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE maskbits maskbits smallint COMMENT "bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits*maps, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE type type string COMMENT "Morphological model: PSF=stellar, REX=round exponential galaxy, DEV=deVauc, EXP=exponential, COMP=composite, DUP=Gaia source fit by different model.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra ra double COMMENT "Right ascension at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec dec double COMMENT "Declination at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra_ivar ra_ivar float COMMENT "Inverse variance of RA (no cosine term!), excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec_ivar dec_ivar float COMMENT "Inverse variance of DEC, excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE bx bx float COMMENT "X position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE by by float COMMENT "Y position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dchisq dchisq array<float> COMMENT "Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ebv ebv float COMMENT "Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_min mjd_min double COMMENT "Minimum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_max mjd_max double COMMENT "Maximum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_cat ref_cat string COMMENT "Reference catalog source for this star: T2 for Tycho-2, G2 for Gaia DR2, L2 for the LSLGA, empty otherwise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_id ref_id bigint COMMENT "Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b \"sourceid\" for Gaia-DR2 and LSLGA";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra pmra float COMMENT "Reference catalog proper motion in the RA direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec pmdec float COMMENT "Reference catalog proper motion in the Dec direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax parallax float COMMENT "Reference catalog parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra_ivar pmra_ivar float COMMENT "Reference catalog inverse-variance on pmra";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec_ivar pmdec_ivar float COMMENT "Reference catalog inverse-variance on pmdec";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax_ivar parallax_ivar float COMMENT "Reference catalog inverse-variance on parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_epoch ref_epoch float COMMENT "Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_pointsource gaia_pointsource boolean COMMENT "This Gaia DR2 source is believed to be a star, not a galaxy";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_mag gaia_phot_g_mean_mag float COMMENT "Gaia G band mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_flux_over_error gaia_phot_g_mean_flux_over_error float COMMENT "Gaia G band signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_n_obs gaia_phot_g_n_obs smallint COMMENT "Gaia G band number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_mag gaia_phot_bp_mean_mag float COMMENT "Gaia BP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_flux_over_error gaia_phot_bp_mean_flux_over_error float COMMENT "Gaia BP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_n_obs gaia_phot_bp_n_obs smallint COMMENT "Gaia BP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_mag gaia_phot_rp_mean_mag float COMMENT "Gaia RP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_flux_over_error gaia_phot_rp_mean_flux_over_error float COMMENT "Gaia RP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_n_obs gaia_phot_rp_n_obs smallint COMMENT "Gaia RP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_variable_flag gaia_phot_variable_flag boolean COMMENT "Gaia photometric variable flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise gaia_astrometric_excess_noise float COMMENT "Gaia astrometric excess noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise_sig gaia_astrometric_excess_noise_sig float COMMENT "Gaia astrometric excess noise uncertainty";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_obs_al gaia_astrometric_n_obs_al smallint COMMENT "Gaia number of astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_good_obs_al gaia_astrometric_n_good_obs_al smallint COMMENT "Gaia number of good astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_weight_al gaia_astrometric_weight_al float COMMENT "Gaia astrometric weight along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_duplicated_source gaia_duplicated_source boolean COMMENT "Gaia duplicated source flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_a_g_val gaia_a_g_val float COMMENT "Gaia line-of-sight extinction in the G band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_e_bp_min_rp_val gaia_e_bp_min_rp_val float COMMENT "Gaia line-of-sight reddening E(BP-RP)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_rp_excess_factor gaia_phot_bp_rp_excess_factor float COMMENT "Gaia BP/RP excess factor";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_sigma5d_max gaia_astrometric_sigma5d_max float COMMENT "Gaia longest semi-major axis of the 5-d error ellipsoid";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_params_solved gaia_astrometric_params_solved tinyint COMMENT "which astrometric parameters were estimated for a Gaia source";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_g flux_g float COMMENT "model flux in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_r flux_r float COMMENT "model flux in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_z flux_z float COMMENT "model flux in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w1 flux_w1 float COMMENT "WISE model flux in W1W1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w2 flux_w2 float COMMENT "WISE model flux in W2W2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w3 flux_w3 float COMMENT "WISE model flux in W3W3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w4 flux_w4 float COMMENT "WISE model flux in W4W4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_g flux_ivar_g float COMMENT "Inverse variance of flux_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_r flux_ivar_r float COMMENT "Inverse variance of flux_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_z flux_ivar_z float COMMENT "Inverse variance of flux_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w1 flux_ivar_w1 float COMMENT "Inverse variance of flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w2 flux_ivar_w2 float COMMENT "Inverse variance of flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w3 flux_ivar_w3 float COMMENT "Inverse variance of flux_w3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w4 flux_ivar_w4 float COMMENT "Inverse variance of flux_w4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_g fiberflux_g float COMMENT "Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_r fiberflux_r float COMMENT "Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_z fiberflux_z float COMMENT "Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_g fibertotflux_g float COMMENT "Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_r fibertotflux_r float COMMENT "Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_z fibertotflux_z float COMMENT "Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_g apflux_g array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_r apflux_r array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_z apflux_z array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_g apflux_resid_g array<float> COMMENT "aperture fluxes on the co-added residual images in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_r apflux_resid_r array<float> COMMENT "aperture fluxes on the co-added residual images in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_z apflux_resid_z array<float> COMMENT "aperture fluxes on the co-added residual images in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_g apflux_ivar_g array<float> COMMENT "Inverse variance of apflux_resid_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_r apflux_ivar_r array<float> COMMENT "Inverse variance of apflux_resid_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_z apflux_ivar_z array<float> COMMENT "Inverse variance of apflux_resid_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_g mw_transmission_g float COMMENT "Galactic transmission in gg filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_r mw_transmission_r float COMMENT "Galactic transmission in rr filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_z mw_transmission_z float COMMENT "Galactic transmission in zz filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w1 mw_transmission_w1 float COMMENT "Galactic transmission in W1W1 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w2 mw_transmission_w2 float COMMENT "Galactic transmission in W2W2 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w3 mw_transmission_w3 float COMMENT "Galactic transmission in W3W3 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w4 mw_transmission_w4 float COMMENT "Galactic transmission in W4W4 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_g nobs_g smallint COMMENT "Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_r nobs_r smallint COMMENT "Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_z nobs_z smallint COMMENT "Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w1 nobs_w1 smallint COMMENT "Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w2 nobs_w2 smallint COMMENT "Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w3 nobs_w3 smallint COMMENT "Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w4 nobs_w4 smallint COMMENT "Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_g rchisq_g float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_r rchisq_r float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_z rchisq_z float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w1 rchisq_w1 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W1W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w2 rchisq_w2 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W2W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w3 rchisq_w3 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W3W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w4 rchisq_w4 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W4W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_g fracflux_g float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in gg(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_r fracflux_r float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in rr(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_z fracflux_z float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in zz(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w1 fracflux_w1 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w2 fracflux_w2 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w3 fracflux_w3 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w4 fracflux_w4 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_g fracmasked_g float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_r fracmasked_r float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_z fracmasked_z float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_g fracin_g float COMMENT "Fraction of a source's flux within the blob in gg, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_r fracin_r float COMMENT "Fraction of a source's flux within the blob in rr, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_z fracin_z float COMMENT "Fraction of a source's flux within the blob in zz, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_g anymask_g smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_r anymask_r smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_z anymask_z smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_g allmask_g smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_r allmask_r smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_z allmask_z smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w1 wisemask_w1 tinyint COMMENT "W1 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w2 wisemask_w2 tinyint COMMENT "W2 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_g psfsize_g float COMMENT "Weighted average PSF FWHM in the gg band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_r psfsize_r float COMMENT "Weighted average PSF FWHM in the rr band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_z psfsize_z float COMMENT "Weighted average PSF FWHM in the zz band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_g psfdepth_g float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_r psfdepth_r float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_z psfdepth_z float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_g galdepth_g float COMMENT "As for psfdepth_g but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_r galdepth_r float COMMENT "As for psfdepth_r but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_z galdepth_z float COMMENT "As for psfdepth_z but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w1 psfdepth_w1 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w2 psfdepth_w2 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w3 psfdepth_w3 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w4 psfdepth_w4 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wise_coadd_id wise_coadd_id string COMMENT "unWISE coadd file name for the center of each object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w1 lc_flux_w1 array<float> COMMENT "flux_w1 in each of up to eleven unWISE coadd epochs (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w2 lc_flux_w2 array<float> COMMENT "flux_w2 in each of up to eleven unWISE coadd epochs (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w1 lc_flux_ivar_w1 array<float> COMMENT "Inverse variance of lc_flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w2 lc_flux_ivar_w2 array<float> COMMENT "Inverse variance of lc_flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w1 lc_nobs_w1 array<smallint> COMMENT "nobs_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w2 lc_nobs_w2 array<smallint> COMMENT "nobs_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w1 lc_fracflux_w1 array<float> COMMENT "fracflux_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w2 lc_fracflux_w2 array<float> COMMENT "fracflux_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w1 lc_rchisq_w1 array<float> COMMENT "rchisq_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w2 lc_rchisq_w2 array<float> COMMENT "rchisq_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w1 lc_mjd_w1 array<double> COMMENT "mjd_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w2 lc_mjd_w2 array<double> COMMENT "mjd_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev fracdev float COMMENT "Fraction of model in deVauc [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev_ivar fracdev_ivar float COMMENT "Inverse variance of fracdev";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r shapeexp_r float COMMENT "Half-light radius of exponential model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r_ivar shapeexp_r_ivar float COMMENT "Inverse variance of shapeexp_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1 shapeexp_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1_ivar shapeexp_e1_ivar float COMMENT "Inverse variance of shapeexp_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2 shapeexp_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2_ivar shapeexp_e2_ivar float COMMENT "Inverse variance of shapeexp_e2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r shapedev_r float COMMENT "Half-light radius of deVaucouleurs model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r_ivar shapedev_r_ivar float COMMENT "Inverse variance of shapedev_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1 shapedev_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1_ivar shapedev_e1_ivar float COMMENT "Inverse variance of shapedev_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2 shapedev_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2_ivar shapedev_e2_ivar float COMMENT "Inverse variance of shapedev_e2";<br />
<br />
== Sweep photo-z ==<br />
<br />
=== Raw data ===<br />
<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/north/sweep/<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/south/sweep/<br />
<br />
Checksums checked:<br />
<br />
$ cd north/sweep/8.0<br />
$ ls | wc -l<br />
287<br />
$ wc -l legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
286 (+ checksum file)<br />
$ sha256sum -c legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
OK<br />
<br />
$ cd north/sweep/8.0-photo-z<br />
$ ls | wc -l<br />
287<br />
$ wc -l legacysurvey_dr8_north_sweep_8.0-photo-z.sha256sum<br />
286 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_north_sweep_8.0-photo-z.sha256sum<br />
OK<br />
<br />
$ cd south/sweep/8.0<br />
$ ls | wc -l<br />
438<br />
$ wc -l legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
437 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
OK<br />
<br />
$ cd south/sweep/8.0-photo-z<br />
$ ls | wc -l<br />
438<br />
$ wc -l legacysurvey_dr8_south_sweep_8.0-photo-z.sha256sum<br />
437 (+ checksum file)<br />
$ sha256sum -w --quiet --strict -c legacysurvey_dr8_south_sweep_8.0-photo-z.sha256sum<br />
OK<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to merge fits files and convert to parquet:<br />
<br />
path_base = '/cephfs/pic.es/astro/scratch/tallada/dr8-phz/'<br />
hdfs_base = '/user/tallada/data/legacysurvey-sweep'<br />
<br />
hs = ['south/sweep'] #, 'north/sweep']<br />
dir_src = '8.0'<br />
dir_phz = '8.0-photo-z'<br />
dir_dst = '8.0-merged'<br />
<br />
my_files = sorted(os.listdir(os.path.join(path_base, hs[0], dir_src)))<br />
<br />
def merge_phz(file_name):<br />
file_src = os.path.join(path_base, hs[0], dir_src, file_name)<br />
file_phz = os.path.join(path_base, hs[0], dir_phz, file_name[:-5]+'-pz.fits')<br />
file_dst = os.path.join(path_base, hs[0], dir_dst, file_name[:-5]+'-merged.fits')<br />
<br />
t_src = table.Table.read(file_src)<br />
t_phz = table.Table.read(file_phz)<br />
t_dst = table.hstack([t_src, t_phz], join_type='exact')<br />
<br />
os.unlink(file_dst)<br />
t_dst.write(file_dst)<br />
<br />
return file_name<br />
<br />
df = sc.parallelize(my_files, len(my_files))<br />
<br />
df.map(merge_phz).count()<br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + os.path.join(path_base, hs[0], dir_dst)<br />
) <br />
<br />
df.write.parquet(os.path.join(hdfs_base, hs[0]))<br />
<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE tallada.dr8_sweep (<br />
`RELEASE` SMALLINT,<br />
`BRICKID` INT,<br />
`BRICKNAME` STRING,<br />
`OBJID` INT,<br />
`TYPE` STRING,<br />
`RA` DOUBLE,<br />
`DEC` DOUBLE,<br />
`RA_IVAR` FLOAT,<br />
`DEC_IVAR` FLOAT,<br />
`DCHISQ` ARRAY<FLOAT>,<br />
`EBV` FLOAT,<br />
`FLUX_G` FLOAT,<br />
`FLUX_R` FLOAT,<br />
`FLUX_Z` FLOAT,<br />
`FLUX_W1` FLOAT,<br />
`FLUX_W2` FLOAT,<br />
`FLUX_W3` FLOAT,<br />
`FLUX_W4` FLOAT,<br />
`FLUX_IVAR_G` FLOAT,<br />
`FLUX_IVAR_R` FLOAT,<br />
`FLUX_IVAR_Z` FLOAT,<br />
`FLUX_IVAR_W1` FLOAT,<br />
`FLUX_IVAR_W2` FLOAT,<br />
`FLUX_IVAR_W3` FLOAT,<br />
`FLUX_IVAR_W4` FLOAT,<br />
`MW_TRANSMISSION_G` FLOAT,<br />
`MW_TRANSMISSION_R` FLOAT,<br />
`MW_TRANSMISSION_Z` FLOAT,<br />
`MW_TRANSMISSION_W1` FLOAT,<br />
`MW_TRANSMISSION_W2` FLOAT,<br />
`MW_TRANSMISSION_W3` FLOAT,<br />
`MW_TRANSMISSION_W4` FLOAT,<br />
`NOBS_G` INT,<br />
`NOBS_R` INT,<br />
`NOBS_Z` INT,<br />
`NOBS_W1` INT,<br />
`NOBS_W2` INT,<br />
`NOBS_W3` INT,<br />
`NOBS_W4` INT,<br />
`RCHISQ_G` FLOAT,<br />
`RCHISQ_R` FLOAT,<br />
`RCHISQ_Z` FLOAT,<br />
`RCHISQ_W1` FLOAT,<br />
`RCHISQ_W2` FLOAT,<br />
`RCHISQ_W3` FLOAT,<br />
`RCHISQ_W4` FLOAT,<br />
`FRACFLUX_G` FLOAT,<br />
`FRACFLUX_R` FLOAT,<br />
`FRACFLUX_Z` FLOAT,<br />
`FRACFLUX_W1` FLOAT,<br />
`FRACFLUX_W2` FLOAT,<br />
`FRACFLUX_W3` FLOAT,<br />
`FRACFLUX_W4` FLOAT,<br />
`FRACMASKED_G` FLOAT,<br />
`FRACMASKED_R` FLOAT,<br />
`FRACMASKED_Z` FLOAT,<br />
`FRACIN_G` FLOAT,<br />
`FRACIN_R` FLOAT,<br />
`FRACIN_Z` FLOAT,<br />
`ANYMASK_G` SMALLINT,<br />
`ANYMASK_R` SMALLINT,<br />
`ANYMASK_Z` SMALLINT,<br />
`ALLMASK_G` SMALLINT,<br />
`ALLMASK_R` SMALLINT,<br />
`ALLMASK_Z` SMALLINT,<br />
`WISEMASK_W1` TINYINT,<br />
`WISEMASK_W2` TINYINT,<br />
`PSFSIZE_G` FLOAT,<br />
`PSFSIZE_R` FLOAT,<br />
`PSFSIZE_Z` FLOAT,<br />
`PSFDEPTH_G` FLOAT,<br />
`PSFDEPTH_R` FLOAT,<br />
`PSFDEPTH_Z` FLOAT,<br />
`GALDEPTH_G` FLOAT,<br />
`GALDEPTH_R` FLOAT,<br />
`GALDEPTH_Z` FLOAT,<br />
`PSFDEPTH_W1` FLOAT,<br />
`PSFDEPTH_W2` FLOAT,<br />
`WISE_COADD_ID` STRING,<br />
`FRACDEV` FLOAT,<br />
`FRACDEV_IVAR` FLOAT,<br />
`SHAPEDEV_R` FLOAT,<br />
`SHAPEDEV_R_IVAR` FLOAT,<br />
`SHAPEDEV_E1` FLOAT,<br />
`SHAPEDEV_E1_IVAR` FLOAT,<br />
`SHAPEDEV_E2` FLOAT,<br />
`SHAPEDEV_E2_IVAR` FLOAT,<br />
`SHAPEEXP_R` FLOAT,<br />
`SHAPEEXP_R_IVAR` FLOAT,<br />
`SHAPEEXP_E1` FLOAT,<br />
`SHAPEEXP_E1_IVAR` FLOAT,<br />
`SHAPEEXP_E2` FLOAT,<br />
`SHAPEEXP_E2_IVAR` FLOAT,<br />
`FIBERFLUX_G` FLOAT,<br />
`FIBERFLUX_R` FLOAT,<br />
`FIBERFLUX_Z` FLOAT,<br />
`FIBERTOTFLUX_G` FLOAT,<br />
`FIBERTOTFLUX_R` FLOAT,<br />
`FIBERTOTFLUX_Z` FLOAT,<br />
`REF_CAT` STRING,<br />
`REF_ID` BIGINT,<br />
`REF_EPOCH` FLOAT,<br />
`GAIA_PHOT_G_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_G_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_ASTROMETRIC_EXCESS_NOISE` FLOAT,<br />
`GAIA_DUPLICATED_SOURCE` BOOLEAN,<br />
`GAIA_PHOT_BP_RP_EXCESS_FACTOR` FLOAT,<br />
`GAIA_ASTROMETRIC_SIGMA5D_MAX` FLOAT,<br />
`GAIA_ASTROMETRIC_PARAMS_SOLVED` TINYINT,<br />
`PARALLAX` FLOAT,<br />
`PARALLAX_IVAR` FLOAT,<br />
`PMRA` FLOAT,<br />
`PMRA_IVAR` FLOAT,<br />
`PMDEC` FLOAT,<br />
`PMDEC_IVAR` FLOAT,<br />
`MASKBITS` SMALLINT,<br />
`z_phot_mean` FLOAT,<br />
`z_phot_median` FLOAT,<br />
`z_phot_std` FLOAT,<br />
`z_phot_l68` FLOAT,<br />
`z_phot_u68` FLOAT,<br />
`z_phot_l95` FLOAT,<br />
`z_phot_u95` FLOAT,<br />
`z_spec` FLOAT,<br />
`survey` STRING,<br />
`training` BOOLEAN<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey-sweep'<br />
;<br />
<br />
=== Combined table ===<br />
<br />
CREATE TABLE cosmohub.legacysurvey_dr8_phz (<br />
release smallint COMMENT 'Unique integer denoting the camera and filter set used',<br />
brickid int COMMENT 'Brick ID [1,662174]',<br />
brickname string COMMENT 'Name of brick, encoding the brick sky position, eg "1126p222" near RA=112.6, Dec=+22.2',<br />
objid int COMMENT 'Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick',<br />
brick_primary boolean COMMENT 'True if the object is within the brick boundary',<br />
brightblob smallint COMMENT 'bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page',<br />
maskbits smallint COMMENT 'bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits* maps, as cataloged on the DR8 bitmasks page',<br />
type string COMMENT 'Morphological model: "PSF"=stellar, "REX"="round exponential galaxy", "DEV"=deVauc, "EXP"=exponential, "COMP"=composite, "DUP"=Gaia source fit by different model. Note that in some FITS readers, a trailing space may be appended for "PSF ", "DEV " and "EXP " since the column data type is a 4-character string',<br />
ra double COMMENT 'Right ascension at equinox J2000',<br />
dec double COMMENT 'Declination at equinox J2000',<br />
ra_ivar float COMMENT 'Inverse variance of RA (no cosine term!), excluding astrometric calibration errors',<br />
dec_ivar float COMMENT 'Inverse variance of DEC, excluding astrometric calibration errors',<br />
bx float COMMENT 'X position (0-indexed) of coordinates in brick image stack',<br />
`by` float COMMENT 'Y position (0-indexed) of coordinates in brick image stack',<br />
dchisq array<float> COMMENT 'Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.',<br />
ebv float COMMENT 'Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns',<br />
mjd_min double COMMENT 'Minimum Modified Julian Date of observations used to construct the model of this object',<br />
mjd_max double COMMENT 'Maximum Modified Julian Date of observations used to construct the model of this object',<br />
ref_cat string COMMENT 'Reference catalog source for this star: "T2" for Tycho-2, "G2" for Gaia DR2, "L2" for the LSLGA, empty otherwise',<br />
ref_id bigint COMMENT 'Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b "sourceid" for Gaia-DR2 and LSLGA',<br />
pmra float COMMENT 'Reference catalog proper motion in the RA direction',<br />
pmdec float COMMENT 'Reference catalog proper motion in the Dec direction',<br />
parallax float COMMENT 'Reference catalog parallax',<br />
pmra_ivar float COMMENT 'Reference catalog inverse-variance on pmra',<br />
pmdec_ivar float COMMENT 'Reference catalog inverse-variance on pmdec',<br />
parallax_ivar float COMMENT 'Reference catalog inverse-variance on parallax',<br />
ref_epoch float COMMENT 'Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)',<br />
gaia_pointsource boolean COMMENT 'This Gaia DR2 source is believed to be a star, not a galaxy',<br />
gaia_phot_g_mean_mag float COMMENT 'Gaia G band mag',<br />
gaia_phot_g_mean_flux_over_error float COMMENT 'Gaia G band signal-to-noise',<br />
gaia_phot_g_n_obs smallint COMMENT 'Gaia G band number of observations',<br />
gaia_phot_bp_mean_mag float COMMENT 'Gaia BP mag',<br />
gaia_phot_bp_mean_flux_over_error float COMMENT 'Gaia BP signal-to-noise',<br />
gaia_phot_bp_n_obs smallint COMMENT 'Gaia BP number of observations',<br />
gaia_phot_rp_mean_mag float COMMENT 'Gaia RP mag',<br />
gaia_phot_rp_mean_flux_over_error float COMMENT 'Gaia RP signal-to-noise',<br />
gaia_phot_rp_n_obs smallint COMMENT 'Gaia RP number of observations',<br />
gaia_phot_variable_flag boolean COMMENT 'Gaia photometric variable flag',<br />
gaia_astrometric_excess_noise float COMMENT 'Gaia astrometric excess noise',<br />
gaia_astrometric_excess_noise_sig float COMMENT 'Gaia astrometric excess noise uncertainty',<br />
gaia_astrometric_n_obs_al smallint COMMENT 'Gaia number of astrometric observations along scan direction',<br />
gaia_astrometric_n_good_obs_al smallint COMMENT 'Gaia number of good astrometric observations along scan direction',<br />
gaia_astrometric_weight_al float COMMENT 'Gaia astrometric weight along scan direction',<br />
gaia_duplicated_source boolean COMMENT 'Gaia duplicated source flag',<br />
gaia_a_g_val float COMMENT 'Gaia line-of-sight extinction in the G band',<br />
gaia_e_bp_min_rp_val float COMMENT 'Gaia line-of-sight reddening E(BP-RP)',<br />
gaia_phot_bp_rp_excess_factor float COMMENT 'Gaia BP/RP excess factor',<br />
gaia_astrometric_sigma5d_max float COMMENT 'Gaia longest semi-major axis of the 5-d error ellipsoid',<br />
gaia_astrometric_params_solved tinyint COMMENT 'which astrometric parameters were estimated for a Gaia source',<br />
flux_g float COMMENT 'model flux in gg',<br />
flux_r float COMMENT 'model flux in rr',<br />
flux_z float COMMENT 'model flux in zz',<br />
flux_w1 float COMMENT 'WISE model flux in W1W1 (AB system)',<br />
flux_w2 float COMMENT 'WISE model flux in W2W2 (AB)',<br />
flux_w3 float COMMENT 'WISE model flux in W3W3 (AB)',<br />
flux_w4 float COMMENT 'WISE model flux in W4W4 (AB)',<br />
flux_ivar_g float COMMENT 'Inverse variance of flux_g',<br />
flux_ivar_r float COMMENT 'Inverse variance of flux_r',<br />
flux_ivar_z float COMMENT 'Inverse variance of flux_z',<br />
flux_ivar_w1 float COMMENT 'Inverse variance of flux_w1 (AB system)',<br />
flux_ivar_w2 float COMMENT 'Inverse variance of flux_w2 (AB)',<br />
flux_ivar_w3 float COMMENT 'Inverse variance of flux_w3 (AB)',<br />
flux_ivar_w4 float COMMENT 'Inverse variance of flux_w4 (AB)',<br />
fiberflux_g float COMMENT 'Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_r float COMMENT 'Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_z float COMMENT 'Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fibertotflux_g float COMMENT 'Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_r float COMMENT 'Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_z float COMMENT 'Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
apflux_g array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg',<br />
apflux_r array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr',<br />
apflux_z array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz',<br />
apflux_resid_g array<float> COMMENT 'aperture fluxes on the co-added residual images in gg',<br />
apflux_resid_r array<float> COMMENT 'aperture fluxes on the co-added residual images in rr',<br />
apflux_resid_z array<float> COMMENT 'aperture fluxes on the co-added residual images in zz',<br />
apflux_ivar_g array<float> COMMENT 'Inverse variance of apflux_resid_g',<br />
apflux_ivar_r array<float> COMMENT 'Inverse variance of apflux_resid_r',<br />
apflux_ivar_z array<float> COMMENT 'Inverse variance of apflux_resid_z',<br />
mw_transmission_g float COMMENT 'Galactic transmission in gg filter in linear units [0, 1]',<br />
mw_transmission_r float COMMENT 'Galactic transmission in rr filter in linear units [0, 1]',<br />
mw_transmission_z float COMMENT 'Galactic transmission in zz filter in linear units [0, 1]',<br />
mw_transmission_w1 float COMMENT 'Galactic transmission in W1W1 filter in linear units [0, 1]',<br />
mw_transmission_w2 float COMMENT 'Galactic transmission in W2W2 filter in linear units [0, 1]',<br />
mw_transmission_w3 float COMMENT 'Galactic transmission in W3W3 filter in linear units [0, 1]',<br />
mw_transmission_w4 float COMMENT 'Galactic transmission in W4W4 filter in linear units [0, 1]',<br />
nobs_g smallint COMMENT 'Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)',<br />
nobs_r smallint COMMENT 'Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)',<br />
nobs_z smallint COMMENT 'Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)',<br />
nobs_w1 smallint COMMENT 'Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)',<br />
nobs_w2 smallint COMMENT 'Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)',<br />
nobs_w3 smallint COMMENT 'Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)',<br />
nobs_w4 smallint COMMENT 'Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)',<br />
rchisq_g float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in gg',<br />
rchisq_r float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in rr',<br />
rchisq_z float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in zz',<br />
rchisq_w1 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W1W1',<br />
rchisq_w2 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W2W2',<br />
rchisq_w3 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W3W3',<br />
rchisq_w4 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W4W4',<br />
fracflux_g float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in gg (typically [0,1])',<br />
fracflux_r float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in rr (typically [0,1])',<br />
fracflux_z float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in zz (typically [0,1])',<br />
fracflux_w1 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1 (typically [0,1])',<br />
fracflux_w2 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2 (typically [0,1])',<br />
fracflux_w3 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3 (typically [0,1])',<br />
fracflux_w4 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4 (typically [0,1])',<br />
fracmasked_g float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]',<br />
fracmasked_r float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]',<br />
fracmasked_z float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]',<br />
fracin_g float COMMENT 'Fraction of a source\'s flux within the blob in gg, near unity for real sources',<br />
fracin_r float COMMENT 'Fraction of a source\'s flux within the blob in rr, near unity for real sources',<br />
fracin_z float COMMENT 'Fraction of a source\'s flux within the blob in zz, near unity for real sources',<br />
anymask_g smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page',<br />
anymask_r smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page',<br />
anymask_z smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page',<br />
allmask_g smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page',<br />
allmask_r smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page',<br />
allmask_z smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page',<br />
wisemask_w1 tinyint COMMENT 'W1 bitmask as cataloged on the DR8 bitmasks page',<br />
wisemask_w2 tinyint COMMENT 'W2 bitmask as cataloged on the DR8 bitmasks page',<br />
psfsize_g float COMMENT 'Weighted average PSF FWHM in the gg band',<br />
psfsize_r float COMMENT 'Weighted average PSF FWHM in the rr band',<br />
psfsize_z float COMMENT 'Weighted average PSF FWHM in the zz band',<br />
psfdepth_g float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude',<br />
psfdepth_r float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude',<br />
psfdepth_z float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude',<br />
galdepth_g float COMMENT 'As for psfdepth_g but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_r float COMMENT 'As for psfdepth_r but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_z float COMMENT 'As for psfdepth_z but for a galaxy (0.45" exp, round) detection sensitivity',<br />
psfdepth_w1 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W1',<br />
psfdepth_w2 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W2',<br />
psfdepth_w3 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W3',<br />
psfdepth_w4 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W4',<br />
wise_coadd_id string COMMENT 'unWISE coadd file name for the center of each object',<br />
lc_flux_w1 array<float> COMMENT 'flux_w1 in each of up to eleven unWISE coadd epochs (AB system)',<br />
lc_flux_w2 array<float> COMMENT 'flux_w2 in each of up to eleven unWISE coadd epochs (AB)',<br />
lc_flux_ivar_w1 array<float> COMMENT 'Inverse variance of lc_flux_w1 (AB system)',<br />
lc_flux_ivar_w2 array<float> COMMENT 'Inverse variance of lc_flux_w2 (AB)',<br />
lc_nobs_w1 array<smallint> COMMENT 'nobs_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_nobs_w2 array<smallint> COMMENT 'nobs_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w1 array<float> COMMENT 'fracflux_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w2 array<float> COMMENT 'fracflux_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w1 array<float> COMMENT 'rchisq_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w2 array<float> COMMENT 'rchisq_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w1 array<double> COMMENT 'mjd_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w2 array<double> COMMENT 'mjd_w2 in each of up to eleven unWISE coadd epochs',<br />
fracdev float COMMENT 'Fraction of model in deVauc [0,1]',<br />
fracdev_ivar float COMMENT 'Inverse variance of fracdev',<br />
shapeexp_r float COMMENT 'Half-light radius of exponential model (>0)',<br />
shapeexp_r_ivar float COMMENT 'Inverse variance of shapeexp_r',<br />
shapeexp_e1 float COMMENT 'Ellipticity component 1',<br />
shapeexp_e1_ivar float COMMENT 'Inverse variance of shapeexp_e1',<br />
shapeexp_e2 float COMMENT 'Ellipticity component 2',<br />
shapeexp_e2_ivar float COMMENT 'Inverse variance of shapeexp_e2',<br />
shapedev_r float COMMENT 'Half-light radius of deVaucouleurs model (>0)',<br />
shapedev_r_ivar float COMMENT 'Inverse variance of shapedev_r',<br />
shapedev_e1 float COMMENT 'Ellipticity component 1',<br />
shapedev_e1_ivar float COMMENT 'Inverse variance of shapedev_e1',<br />
shapedev_e2 float COMMENT 'Ellipticity component 2',<br />
shapedev_e2_ivar float COMMENT 'Inverse variance of shapedev_e2',<br />
z_phot_mean FLOAT COMMENT 'photo-z derived from the mean of the photo-z PDF ',<br />
z_phot_median FLOAT COMMENT 'photo-z derived from the median of the photo-z PDF',<br />
z_phot_std FLOAT COMMENT 'standard deviation of the photo-z\'s derived from the photo-z PDF',<br />
z_phot_l68 FLOAT COMMENT 'lower bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_u68 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_l95 FLOAT COMMENT 'lower bound of the 95% confidence region, derived from the photo-z PDF',<br />
z_phot_u95 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_spec FLOAT COMMENT 'spectroscopic redshift, if available',<br />
survey STRING COMMENT 'source of the spectroscopic redshift',<br />
training BOOLEAN COMMENT 'whether or not the spectroscopic redshift is used in photometric redshift training'<br />
)<br />
CLUSTERED BY (<br />
release,<br />
brickid,<br />
objid)<br />
SORTED BY (<br />
release ASC,<br />
brickid ASC,<br />
objid ASC)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
=== Join insert ===<br />
<br />
INSERT OVERWRITE TABLE cosmohub.legacysurvey_dr8_phz<br />
SELECT<br />
dr8.*,<br />
phz.z_phot_mean,<br />
phz.z_phot_median,<br />
phz.z_phot_std,<br />
phz.z_phot_l68,<br />
phz.z_phot_u68,<br />
phz.z_phot_l95,<br />
phz.z_phot_u95,<br />
phz.z_spec,<br />
phz.survey,<br />
phz.training<br />
FROM cosmohub.legacy_survey_dr8 AS dr8<br />
LEFT JOIN tallada.dr8_sweep AS phz<br />
ON dr8.release = phz.release<br />
AND dr8.brickid = phz.brickid<br />
AND dr8.objid = phz.objid<br />
;<br />
<br />
== Queries for checking ==<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT release, brickid, objid, COUNT(release)<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
HAVING COUNT(release) > 1<br />
<br />
DR8 release brickid objid: DISTINCT(1.694.230.368) TOTAL(1.694.230.368)<br />
DR8 brickid objid: DISTINCT(1.654.429.115) TOTAL(1.694.230.368)<br />
<br />
PHZ release brickid objid: DISTINCT(1.646.664.729) TOTAL(1.646.664.897)<br />
PHZ brickid objid: DISTINCT(1.608.841.582) TOTAL(1.646.664.897)<br />
<br />
DR8: 1.694.230.368<br />
PHZ: 1.646.664.897<br />
----------------------------------------------------------------------<br />
-47.565.471</div>
Tallada
https://pwiki.pic.es/index.php?title=Legacy_Survey_DR8&diff=801
Legacy Survey DR8
2020-05-07T12:32:28Z
<p>Tallada: </p>
<hr />
<div>== Original DR8 ==<br />
=== Raw data ===<br />
<br />
Downloaded from<br />
ftp://archive.noao.edu:/public/hlsp/ls/dr8/<br />
<br />
Directories:<br />
south/tractor<br />
north/tractor<br />
<br />
Mirrored using lftp commands:<br />
mirror -c -v -P 200 north/tractor/ /mnt/pau/north/tractor/<br />
mirror -c -v -P 200 south/tractor/ /mnt/pau/south/tractor/<br />
<br />
into a 5 TiB temporary Ceph filesystem mounted in tdm002.<br />
<br />
Checksums provided through sha256sum files, and checked using command:<br />
find * -type d | xargs -t -L1 -P24 sh -c 'cd $1; sha256sum --quiet -c *.sha256sum' cksum<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to convert fits files to parquet:<br />
<br />
path_base = '/cephfs/pic.es/cosmohub/legacysurvey'<br />
hdfs_base = '/user/tallada/data/legacysurvey'<br />
<br />
hs = ['h=north', 'h=south']<br />
rs = ['r={0:03d}'.format(x) for x in range(360)]<br />
<br />
for h in hs:<br />
for r in rs:<br />
p = os.path.join(path_base, h, r)<br />
if not os.path.exists(p):<br />
continue<br />
<br />
size = sum(os.path.getsize(os.path.join(p,f)) for f in os.listdir(p) if os.path.isfile(os.path.join(p,f)))<br />
<br />
print h, r, size <br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + p<br />
)<br />
<br />
df = df.repartition(int(math.ceil(size / (256*1024*1024.0))))<br />
<br />
df.write.parquet(os.path.join(hdfs_base, h, r))<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE tallada.dr8 (<br />
`release` smallint,<br />
`brickid` int,<br />
`brickname` string,<br />
`objid` int,<br />
`brick_primary` boolean,<br />
`brightblob` smallint,<br />
`maskbits` smallint,<br />
`type` string,<br />
`ra` double,<br />
`dec` double,<br />
`ra_ivar` float,<br />
`dec_ivar` float,<br />
`bx` float,<br />
`by` float,<br />
`dchisq` array<float>,<br />
`ebv` float,<br />
`mjd_min` double,<br />
`mjd_max` double,<br />
`ref_cat` string,<br />
`ref_id` bigint,<br />
`pmra` float,<br />
`pmdec` float,<br />
`parallax` float,<br />
`pmra_ivar` float,<br />
`pmdec_ivar` float,<br />
`parallax_ivar` float,<br />
`ref_epoch` float,<br />
`gaia_pointsource` boolean,<br />
`gaia_phot_g_mean_mag` float,<br />
`gaia_phot_g_mean_flux_over_error` float,<br />
`gaia_phot_g_n_obs` smallint,<br />
`gaia_phot_bp_mean_mag` float,<br />
`gaia_phot_bp_mean_flux_over_error` float,<br />
`gaia_phot_bp_n_obs` smallint,<br />
`gaia_phot_rp_mean_mag` float,<br />
`gaia_phot_rp_mean_flux_over_error` float,<br />
`gaia_phot_rp_n_obs` smallint,<br />
`gaia_phot_variable_flag` boolean,<br />
`gaia_astrometric_excess_noise` float,<br />
`gaia_astrometric_excess_noise_sig` float,<br />
`gaia_astrometric_n_obs_al` smallint,<br />
`gaia_astrometric_n_good_obs_al` smallint,<br />
`gaia_astrometric_weight_al` float,<br />
`gaia_duplicated_source` boolean,<br />
`gaia_a_g_val` float,<br />
`gaia_e_bp_min_rp_val` float,<br />
`gaia_phot_bp_rp_excess_factor` float,<br />
`gaia_astrometric_sigma5d_max` float,<br />
`gaia_astrometric_params_solved` tinyint,<br />
`flux_g` float,<br />
`flux_r` float,<br />
`flux_z` float,<br />
`flux_w1` float,<br />
`flux_w2` float,<br />
`flux_w3` float,<br />
`flux_w4` float,<br />
`flux_ivar_g` float,<br />
`flux_ivar_r` float,<br />
`flux_ivar_z` float,<br />
`flux_ivar_w1` float,<br />
`flux_ivar_w2` float,<br />
`flux_ivar_w3` float,<br />
`flux_ivar_w4` float,<br />
`fiberflux_g` float,<br />
`fiberflux_r` float,<br />
`fiberflux_z` float,<br />
`fibertotflux_g` float,<br />
`fibertotflux_r` float,<br />
`fibertotflux_z` float,<br />
`apflux_g` array<float>,<br />
`apflux_r` array<float>,<br />
`apflux_z` array<float>,<br />
`apflux_resid_g` array<float>,<br />
`apflux_resid_r` array<float>,<br />
`apflux_resid_z` array<float>,<br />
`apflux_ivar_g` array<float>,<br />
`apflux_ivar_r` array<float>,<br />
`apflux_ivar_z` array<float>,<br />
`mw_transmission_g` float,<br />
`mw_transmission_r` float,<br />
`mw_transmission_z` float,<br />
`mw_transmission_w1` float,<br />
`mw_transmission_w2` float,<br />
`mw_transmission_w3` float,<br />
`mw_transmission_w4` float,<br />
`nobs_g` smallint,<br />
`nobs_r` smallint,<br />
`nobs_z` smallint,<br />
`nobs_w1` smallint,<br />
`nobs_w2` smallint,<br />
`nobs_w3` smallint,<br />
`nobs_w4` smallint,<br />
`rchisq_g` float,<br />
`rchisq_r` float,<br />
`rchisq_z` float,<br />
`rchisq_w1` float,<br />
`rchisq_w2` float,<br />
`rchisq_w3` float,<br />
`rchisq_w4` float,<br />
`fracflux_g` float,<br />
`fracflux_r` float,<br />
`fracflux_z` float,<br />
`fracflux_w1` float,<br />
`fracflux_w2` float,<br />
`fracflux_w3` float,<br />
`fracflux_w4` float,<br />
`fracmasked_g` float,<br />
`fracmasked_r` float,<br />
`fracmasked_z` float,<br />
`fracin_g` float,<br />
`fracin_r` float,<br />
`fracin_z` float,<br />
`anymask_g` smallint,<br />
`anymask_r` smallint,<br />
`anymask_z` smallint,<br />
`allmask_g` smallint,<br />
`allmask_r` smallint,<br />
`allmask_z` smallint,<br />
`wisemask_w1` tinyint,<br />
`wisemask_w2` tinyint,<br />
`psfsize_g` float,<br />
`psfsize_r` float,<br />
`psfsize_z` float,<br />
`psfdepth_g` float,<br />
`psfdepth_r` float,<br />
`psfdepth_z` float,<br />
`galdepth_g` float,<br />
`galdepth_r` float,<br />
`galdepth_z` float,<br />
`psfdepth_w1` float,<br />
`psfdepth_w2` float,<br />
`psfdepth_w3` float,<br />
`psfdepth_w4` float,<br />
`wise_coadd_id` string,<br />
`lc_flux_w1` array<float>,<br />
`lc_flux_w2` array<float>,<br />
`lc_flux_ivar_w1` array<float>,<br />
`lc_flux_ivar_w2` array<float>,<br />
`lc_nobs_w1` array<smallint>,<br />
`lc_nobs_w2` array<smallint>,<br />
`lc_fracflux_w1` array<float>,<br />
`lc_fracflux_w2` array<float>,<br />
`lc_rchisq_w1` array<float>,<br />
`lc_rchisq_w2` array<float>,<br />
`lc_mjd_w1` array<double>,<br />
`lc_mjd_w2` array<double>,<br />
`fracdev` float,<br />
`fracdev_ivar` float,<br />
`shapeexp_r` float,<br />
`shapeexp_r_ivar` float,<br />
`shapeexp_e1` float,<br />
`shapeexp_e1_ivar` float,<br />
`shapeexp_e2` float,<br />
`shapeexp_e2_ivar` float,<br />
`shapedev_r` float,<br />
`shapedev_r_ivar` float,<br />
`shapedev_e1` float,<br />
`shapedev_e1_ivar` float,<br />
`shapedev_e2` float,<br />
`shapedev_e2_ivar` float<br />
)<br />
PARTITIONED BY (<br />
h string,<br />
r string<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey/dr8/'<br />
;<br />
<br />
=== Internal Table ===<br />
<br />
CREATE TABLE cosmohub.legacy_survey_dr8 (<br />
`release` smallint,<br />
`brickid` int,<br />
`brickname` string,<br />
`objid` int,<br />
`brick_primary` boolean,<br />
`brightblob` smallint,<br />
`maskbits` smallint,<br />
`type` string,<br />
`ra` double,<br />
`dec` double,<br />
`ra_ivar` float,<br />
`dec_ivar` float,<br />
`bx` float,<br />
`by` float,<br />
`dchisq` array<float>,<br />
`ebv` float,<br />
`mjd_min` double,<br />
`mjd_max` double,<br />
`ref_cat` string,<br />
`ref_id` bigint,<br />
`pmra` float,<br />
`pmdec` float,<br />
`parallax` float,<br />
`pmra_ivar` float,<br />
`pmdec_ivar` float,<br />
`parallax_ivar` float,<br />
`ref_epoch` float,<br />
`gaia_pointsource` boolean,<br />
`gaia_phot_g_mean_mag` float,<br />
`gaia_phot_g_mean_flux_over_error` float,<br />
`gaia_phot_g_n_obs` smallint,<br />
`gaia_phot_bp_mean_mag` float,<br />
`gaia_phot_bp_mean_flux_over_error` float,<br />
`gaia_phot_bp_n_obs` smallint,<br />
`gaia_phot_rp_mean_mag` float,<br />
`gaia_phot_rp_mean_flux_over_error` float,<br />
`gaia_phot_rp_n_obs` smallint,<br />
`gaia_phot_variable_flag` boolean,<br />
`gaia_astrometric_excess_noise` float,<br />
`gaia_astrometric_excess_noise_sig` float,<br />
`gaia_astrometric_n_obs_al` smallint,<br />
`gaia_astrometric_n_good_obs_al` smallint,<br />
`gaia_astrometric_weight_al` float,<br />
`gaia_duplicated_source` boolean,<br />
`gaia_a_g_val` float,<br />
`gaia_e_bp_min_rp_val` float,<br />
`gaia_phot_bp_rp_excess_factor` float,<br />
`gaia_astrometric_sigma5d_max` float,<br />
`gaia_astrometric_params_solved` tinyint,<br />
`flux_g` float,<br />
`flux_r` float,<br />
`flux_z` float,<br />
`flux_w1` float,<br />
`flux_w2` float,<br />
`flux_w3` float,<br />
`flux_w4` float,<br />
`flux_ivar_g` float,<br />
`flux_ivar_r` float,<br />
`flux_ivar_z` float,<br />
`flux_ivar_w1` float,<br />
`flux_ivar_w2` float,<br />
`flux_ivar_w3` float,<br />
`flux_ivar_w4` float,<br />
`fiberflux_g` float,<br />
`fiberflux_r` float,<br />
`fiberflux_z` float,<br />
`fibertotflux_g` float,<br />
`fibertotflux_r` float,<br />
`fibertotflux_z` float,<br />
`apflux_g` array<float>,<br />
`apflux_r` array<float>,<br />
`apflux_z` array<float>,<br />
`apflux_resid_g` array<float>,<br />
`apflux_resid_r` array<float>,<br />
`apflux_resid_z` array<float>,<br />
`apflux_ivar_g` array<float>,<br />
`apflux_ivar_r` array<float>,<br />
`apflux_ivar_z` array<float>,<br />
`mw_transmission_g` float,<br />
`mw_transmission_r` float,<br />
`mw_transmission_z` float,<br />
`mw_transmission_w1` float,<br />
`mw_transmission_w2` float,<br />
`mw_transmission_w3` float,<br />
`mw_transmission_w4` float,<br />
`nobs_g` smallint,<br />
`nobs_r` smallint,<br />
`nobs_z` smallint,<br />
`nobs_w1` smallint,<br />
`nobs_w2` smallint,<br />
`nobs_w3` smallint,<br />
`nobs_w4` smallint,<br />
`rchisq_g` float,<br />
`rchisq_r` float,<br />
`rchisq_z` float,<br />
`rchisq_w1` float,<br />
`rchisq_w2` float,<br />
`rchisq_w3` float,<br />
`rchisq_w4` float,<br />
`fracflux_g` float,<br />
`fracflux_r` float,<br />
`fracflux_z` float,<br />
`fracflux_w1` float,<br />
`fracflux_w2` float,<br />
`fracflux_w3` float,<br />
`fracflux_w4` float,<br />
`fracmasked_g` float,<br />
`fracmasked_r` float,<br />
`fracmasked_z` float,<br />
`fracin_g` float,<br />
`fracin_r` float,<br />
`fracin_z` float,<br />
`anymask_g` smallint,<br />
`anymask_r` smallint,<br />
`anymask_z` smallint,<br />
`allmask_g` smallint,<br />
`allmask_r` smallint,<br />
`allmask_z` smallint,<br />
`wisemask_w1` tinyint,<br />
`wisemask_w2` tinyint,<br />
`psfsize_g` float,<br />
`psfsize_r` float,<br />
`psfsize_z` float,<br />
`psfdepth_g` float,<br />
`psfdepth_r` float,<br />
`psfdepth_z` float,<br />
`galdepth_g` float,<br />
`galdepth_r` float,<br />
`galdepth_z` float,<br />
`psfdepth_w1` float,<br />
`psfdepth_w2` float,<br />
`psfdepth_w3` float,<br />
`psfdepth_w4` float,<br />
`wise_coadd_id` string,<br />
`lc_flux_w1` array<float>,<br />
`lc_flux_w2` array<float>,<br />
`lc_flux_ivar_w1` array<float>,<br />
`lc_flux_ivar_w2` array<float>,<br />
`lc_nobs_w1` array<smallint>,<br />
`lc_nobs_w2` array<smallint>,<br />
`lc_fracflux_w1` array<float>,<br />
`lc_fracflux_w2` array<float>,<br />
`lc_rchisq_w1` array<float>,<br />
`lc_rchisq_w2` array<float>,<br />
`lc_mjd_w1` array<double>,<br />
`lc_mjd_w2` array<double>,<br />
`fracdev` float,<br />
`fracdev_ivar` float,<br />
`shapeexp_r` float,<br />
`shapeexp_r_ivar` float,<br />
`shapeexp_e1` float,<br />
`shapeexp_e1_ivar` float,<br />
`shapeexp_e2` float,<br />
`shapeexp_e2_ivar` float,<br />
`shapedev_r` float,<br />
`shapedev_r_ivar` float,<br />
`shapedev_e1` float,<br />
`shapedev_e1_ivar` float,<br />
`shapedev_e2` float,<br />
`shapedev_e2_ivar` float<br />
)<br />
clustered by (<br />
release, brickid, objid<br />
)<br />
sorted by (<br />
release, brickid, objid<br />
)<br />
into 4096 buckets<br />
stored as orc<br />
;<br />
<br />
insert overwrite table cosmohub.legacy_survey_dr8<br />
select release, brickid, brickname, objid, brick_primary, brightblob, maskbits, type, ra, dec, ra_ivar, dec_ivar, bx, `by`, <br />
dchisq, ebv, mjd_min, mjd_max, ref_cat, ref_id, pmra, pmdec, parallax, pmra_ivar, pmdec_ivar, parallax_ivar, ref_epoch,<br />
gaia_pointsource, gaia_phot_g_mean_mag, gaia_phot_g_mean_flux_over_error, gaia_phot_g_n_obs,<br />
gaia_phot_bp_mean_mag, gaia_phot_bp_mean_flux_over_error, gaia_phot_bp_n_obs, gaia_phot_rp_mean_mag,<br />
gaia_phot_rp_mean_flux_over_error, gaia_phot_rp_n_obs, gaia_phot_variable_flag, gaia_astrometric_excess_noise,<br />
gaia_astrometric_excess_noise_sig, gaia_astrometric_n_obs_al, gaia_astrometric_n_good_obs_al,<br />
gaia_astrometric_weight_al, gaia_duplicated_source, gaia_a_g_val, gaia_e_bp_min_rp_val,<br />
gaia_phot_bp_rp_excess_factor, gaia_astrometric_sigma5d_max, gaia_astrometric_params_solved,<br />
flux_g, flux_r, flux_z, flux_w1, flux_w2, flux_w3, flux_w4, flux_ivar_g, flux_ivar_r,<br />
flux_ivar_z, flux_ivar_w1, flux_ivar_w2, flux_ivar_w3, flux_ivar_w4, fiberflux_g, fiberflux_r,<br />
fiberflux_z, fibertotflux_g, fibertotflux_r, fibertotflux_z, apflux_g, apflux_r, apflux_z,<br />
apflux_resid_g, apflux_resid_r, apflux_resid_z, apflux_ivar_g, apflux_ivar_r, apflux_ivar_z,<br />
mw_transmission_g, mw_transmission_r, mw_transmission_z, mw_transmission_w1, mw_transmission_w2,<br />
mw_transmission_w3, mw_transmission_w4, nobs_g, nobs_r, nobs_z, nobs_w1, nobs_w2, nobs_w3,<br />
nobs_w4, rchisq_g, rchisq_r, rchisq_z, rchisq_w1, rchisq_w2, rchisq_w3, rchisq_w4, fracflux_g,<br />
fracflux_r, fracflux_z, fracflux_w1, fracflux_w2, fracflux_w3, fracflux_w4, fracmasked_g,<br />
fracmasked_r, fracmasked_z, fracin_g, fracin_r, fracin_z, anymask_g, anymask_r, anymask_z,<br />
allmask_g, allmask_r, allmask_z, wisemask_w1, wisemask_w2, psfsize_g, psfsize_r, psfsize_z,<br />
psfdepth_g, psfdepth_r, psfdepth_z, galdepth_g, galdepth_r, galdepth_z, psfdepth_w1, psfdepth_w2,<br />
psfdepth_w3, psfdepth_w4, wise_coadd_id, lc_flux_w1, lc_flux_w2, lc_flux_ivar_w1, lc_flux_ivar_w2,<br />
lc_nobs_w1, lc_nobs_w2, lc_fracflux_w1, lc_fracflux_w2, lc_rchisq_w1, lc_rchisq_w2, lc_mjd_w1,<br />
lc_mjd_w2, fracdev, fracdev_ivar, shapeexp_r, shapeexp_r_ivar, shapeexp_e1, shapeexp_e1_ivar,<br />
shapeexp_e2, shapeexp_e2_ivar, shapedev_r, shapedev_r_ivar, shapedev_e1, shapedev_e1_ivar,<br />
shapedev_e2, shapedev_e2_ivar<br />
from tallada.dr8<br />
;<br />
<br />
=== Comments ===<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE release release smallint COMMENT "Unique integer denoting the camera and filter set used";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickid brickid int COMMENT "Brick ID [1,662174]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickname brickname string COMMENT "Name of brick, encoding the brick sky position, eg \"1126p222\" near RA=112.6, Dec=+22.2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE objid objid int COMMENT "Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brick_primary brick_primary boolean COMMENT "True if the object is within the brick boundary";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brightblob brightblob smallint COMMENT "bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE maskbits maskbits smallint COMMENT "bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits*maps, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE type type string COMMENT "Morphological model: PSF=stellar, REX=round exponential galaxy, DEV=deVauc, EXP=exponential, COMP=composite, DUP=Gaia source fit by different model.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra ra double COMMENT "Right ascension at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec dec double COMMENT "Declination at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra_ivar ra_ivar float COMMENT "Inverse variance of RA (no cosine term!), excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec_ivar dec_ivar float COMMENT "Inverse variance of DEC, excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE bx bx float COMMENT "X position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE by by float COMMENT "Y position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dchisq dchisq array<float> COMMENT "Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ebv ebv float COMMENT "Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_min mjd_min double COMMENT "Minimum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_max mjd_max double COMMENT "Maximum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_cat ref_cat string COMMENT "Reference catalog source for this star: T2 for Tycho-2, G2 for Gaia DR2, L2 for the LSLGA, empty otherwise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_id ref_id bigint COMMENT "Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b \"sourceid\" for Gaia-DR2 and LSLGA";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra pmra float COMMENT "Reference catalog proper motion in the RA direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec pmdec float COMMENT "Reference catalog proper motion in the Dec direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax parallax float COMMENT "Reference catalog parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra_ivar pmra_ivar float COMMENT "Reference catalog inverse-variance on pmra";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec_ivar pmdec_ivar float COMMENT "Reference catalog inverse-variance on pmdec";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax_ivar parallax_ivar float COMMENT "Reference catalog inverse-variance on parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_epoch ref_epoch float COMMENT "Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_pointsource gaia_pointsource boolean COMMENT "This Gaia DR2 source is believed to be a star, not a galaxy";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_mag gaia_phot_g_mean_mag float COMMENT "Gaia G band mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_flux_over_error gaia_phot_g_mean_flux_over_error float COMMENT "Gaia G band signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_n_obs gaia_phot_g_n_obs smallint COMMENT "Gaia G band number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_mag gaia_phot_bp_mean_mag float COMMENT "Gaia BP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_flux_over_error gaia_phot_bp_mean_flux_over_error float COMMENT "Gaia BP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_n_obs gaia_phot_bp_n_obs smallint COMMENT "Gaia BP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_mag gaia_phot_rp_mean_mag float COMMENT "Gaia RP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_flux_over_error gaia_phot_rp_mean_flux_over_error float COMMENT "Gaia RP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_n_obs gaia_phot_rp_n_obs smallint COMMENT "Gaia RP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_variable_flag gaia_phot_variable_flag boolean COMMENT "Gaia photometric variable flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise gaia_astrometric_excess_noise float COMMENT "Gaia astrometric excess noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise_sig gaia_astrometric_excess_noise_sig float COMMENT "Gaia astrometric excess noise uncertainty";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_obs_al gaia_astrometric_n_obs_al smallint COMMENT "Gaia number of astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_good_obs_al gaia_astrometric_n_good_obs_al smallint COMMENT "Gaia number of good astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_weight_al gaia_astrometric_weight_al float COMMENT "Gaia astrometric weight along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_duplicated_source gaia_duplicated_source boolean COMMENT "Gaia duplicated source flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_a_g_val gaia_a_g_val float COMMENT "Gaia line-of-sight extinction in the G band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_e_bp_min_rp_val gaia_e_bp_min_rp_val float COMMENT "Gaia line-of-sight reddening E(BP-RP)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_rp_excess_factor gaia_phot_bp_rp_excess_factor float COMMENT "Gaia BP/RP excess factor";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_sigma5d_max gaia_astrometric_sigma5d_max float COMMENT "Gaia longest semi-major axis of the 5-d error ellipsoid";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_params_solved gaia_astrometric_params_solved tinyint COMMENT "which astrometric parameters were estimated for a Gaia source";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_g flux_g float COMMENT "model flux in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_r flux_r float COMMENT "model flux in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_z flux_z float COMMENT "model flux in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w1 flux_w1 float COMMENT "WISE model flux in W1W1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w2 flux_w2 float COMMENT "WISE model flux in W2W2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w3 flux_w3 float COMMENT "WISE model flux in W3W3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w4 flux_w4 float COMMENT "WISE model flux in W4W4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_g flux_ivar_g float COMMENT "Inverse variance of flux_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_r flux_ivar_r float COMMENT "Inverse variance of flux_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_z flux_ivar_z float COMMENT "Inverse variance of flux_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w1 flux_ivar_w1 float COMMENT "Inverse variance of flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w2 flux_ivar_w2 float COMMENT "Inverse variance of flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w3 flux_ivar_w3 float COMMENT "Inverse variance of flux_w3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w4 flux_ivar_w4 float COMMENT "Inverse variance of flux_w4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_g fiberflux_g float COMMENT "Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_r fiberflux_r float COMMENT "Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_z fiberflux_z float COMMENT "Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_g fibertotflux_g float COMMENT "Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_r fibertotflux_r float COMMENT "Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_z fibertotflux_z float COMMENT "Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_g apflux_g array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_r apflux_r array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_z apflux_z array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_g apflux_resid_g array<float> COMMENT "aperture fluxes on the co-added residual images in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_r apflux_resid_r array<float> COMMENT "aperture fluxes on the co-added residual images in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_z apflux_resid_z array<float> COMMENT "aperture fluxes on the co-added residual images in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_g apflux_ivar_g array<float> COMMENT "Inverse variance of apflux_resid_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_r apflux_ivar_r array<float> COMMENT "Inverse variance of apflux_resid_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_z apflux_ivar_z array<float> COMMENT "Inverse variance of apflux_resid_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_g mw_transmission_g float COMMENT "Galactic transmission in gg filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_r mw_transmission_r float COMMENT "Galactic transmission in rr filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_z mw_transmission_z float COMMENT "Galactic transmission in zz filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w1 mw_transmission_w1 float COMMENT "Galactic transmission in W1W1 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w2 mw_transmission_w2 float COMMENT "Galactic transmission in W2W2 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w3 mw_transmission_w3 float COMMENT "Galactic transmission in W3W3 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w4 mw_transmission_w4 float COMMENT "Galactic transmission in W4W4 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_g nobs_g smallint COMMENT "Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_r nobs_r smallint COMMENT "Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_z nobs_z smallint COMMENT "Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w1 nobs_w1 smallint COMMENT "Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w2 nobs_w2 smallint COMMENT "Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w3 nobs_w3 smallint COMMENT "Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w4 nobs_w4 smallint COMMENT "Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_g rchisq_g float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_r rchisq_r float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_z rchisq_z float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w1 rchisq_w1 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W1W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w2 rchisq_w2 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W2W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w3 rchisq_w3 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W3W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w4 rchisq_w4 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W4W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_g fracflux_g float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in gg(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_r fracflux_r float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in rr(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_z fracflux_z float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in zz(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w1 fracflux_w1 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w2 fracflux_w2 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w3 fracflux_w3 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w4 fracflux_w4 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_g fracmasked_g float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_r fracmasked_r float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_z fracmasked_z float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_g fracin_g float COMMENT "Fraction of a source's flux within the blob in gg, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_r fracin_r float COMMENT "Fraction of a source's flux within the blob in rr, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_z fracin_z float COMMENT "Fraction of a source's flux within the blob in zz, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_g anymask_g smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_r anymask_r smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_z anymask_z smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_g allmask_g smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_r allmask_r smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_z allmask_z smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w1 wisemask_w1 tinyint COMMENT "W1 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w2 wisemask_w2 tinyint COMMENT "W2 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_g psfsize_g float COMMENT "Weighted average PSF FWHM in the gg band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_r psfsize_r float COMMENT "Weighted average PSF FWHM in the rr band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_z psfsize_z float COMMENT "Weighted average PSF FWHM in the zz band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_g psfdepth_g float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_r psfdepth_r float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_z psfdepth_z float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_g galdepth_g float COMMENT "As for psfdepth_g but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_r galdepth_r float COMMENT "As for psfdepth_r but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_z galdepth_z float COMMENT "As for psfdepth_z but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w1 psfdepth_w1 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w2 psfdepth_w2 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w3 psfdepth_w3 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w4 psfdepth_w4 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wise_coadd_id wise_coadd_id string COMMENT "unWISE coadd file name for the center of each object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w1 lc_flux_w1 array<float> COMMENT "flux_w1 in each of up to eleven unWISE coadd epochs (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w2 lc_flux_w2 array<float> COMMENT "flux_w2 in each of up to eleven unWISE coadd epochs (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w1 lc_flux_ivar_w1 array<float> COMMENT "Inverse variance of lc_flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w2 lc_flux_ivar_w2 array<float> COMMENT "Inverse variance of lc_flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w1 lc_nobs_w1 array<smallint> COMMENT "nobs_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w2 lc_nobs_w2 array<smallint> COMMENT "nobs_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w1 lc_fracflux_w1 array<float> COMMENT "fracflux_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w2 lc_fracflux_w2 array<float> COMMENT "fracflux_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w1 lc_rchisq_w1 array<float> COMMENT "rchisq_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w2 lc_rchisq_w2 array<float> COMMENT "rchisq_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w1 lc_mjd_w1 array<double> COMMENT "mjd_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w2 lc_mjd_w2 array<double> COMMENT "mjd_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev fracdev float COMMENT "Fraction of model in deVauc [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev_ivar fracdev_ivar float COMMENT "Inverse variance of fracdev";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r shapeexp_r float COMMENT "Half-light radius of exponential model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r_ivar shapeexp_r_ivar float COMMENT "Inverse variance of shapeexp_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1 shapeexp_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1_ivar shapeexp_e1_ivar float COMMENT "Inverse variance of shapeexp_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2 shapeexp_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2_ivar shapeexp_e2_ivar float COMMENT "Inverse variance of shapeexp_e2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r shapedev_r float COMMENT "Half-light radius of deVaucouleurs model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r_ivar shapedev_r_ivar float COMMENT "Inverse variance of shapedev_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1 shapedev_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1_ivar shapedev_e1_ivar float COMMENT "Inverse variance of shapedev_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2 shapedev_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2_ivar shapedev_e2_ivar float COMMENT "Inverse variance of shapedev_e2";<br />
<br />
== Sweep photo-z ==<br />
<br />
=== Raw data ===<br />
<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/north/sweep/<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/south/sweep/<br />
<br />
Checksums checked:<br />
<br />
sha256sum -c legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
sha256sum -c legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to merge fits files and convert to parquet:<br />
<br />
path_base = '/cephfs/pic.es/astro/scratch/tallada/dr8-phz/'<br />
hdfs_base = '/user/tallada/data/legacysurvey-sweep'<br />
<br />
hs = ['south/sweep'] #, 'north/sweep']<br />
dir_src = '8.0'<br />
dir_phz = '8.0-photo-z'<br />
dir_dst = '8.0-merged'<br />
<br />
my_files = sorted(os.listdir(os.path.join(path_base, hs[0], dir_src)))<br />
<br />
def merge_phz(file_name):<br />
file_src = os.path.join(path_base, hs[0], dir_src, file_name)<br />
file_phz = os.path.join(path_base, hs[0], dir_phz, file_name[:-5]+'-pz.fits')<br />
file_dst = os.path.join(path_base, hs[0], dir_dst, file_name[:-5]+'-merged.fits')<br />
<br />
t_src = table.Table.read(file_src)<br />
t_phz = table.Table.read(file_phz)<br />
t_dst = table.hstack([t_src, t_phz], join_type='exact')<br />
<br />
os.unlink(file_dst)<br />
t_dst.write(file_dst)<br />
<br />
return file_name<br />
<br />
df = sc.parallelize(my_files, len(my_files))<br />
<br />
df.map(merge_phz).count()<br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + os.path.join(path_base, hs[0], dir_dst)<br />
) <br />
<br />
df.write.parquet(os.path.join(hdfs_base, hs[0]))<br />
<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE tallada.dr8_sweep (<br />
`RELEASE` SMALLINT,<br />
`BRICKID` INT,<br />
`BRICKNAME` STRING,<br />
`OBJID` INT,<br />
`TYPE` STRING,<br />
`RA` DOUBLE,<br />
`DEC` DOUBLE,<br />
`RA_IVAR` FLOAT,<br />
`DEC_IVAR` FLOAT,<br />
`DCHISQ` ARRAY<FLOAT>,<br />
`EBV` FLOAT,<br />
`FLUX_G` FLOAT,<br />
`FLUX_R` FLOAT,<br />
`FLUX_Z` FLOAT,<br />
`FLUX_W1` FLOAT,<br />
`FLUX_W2` FLOAT,<br />
`FLUX_W3` FLOAT,<br />
`FLUX_W4` FLOAT,<br />
`FLUX_IVAR_G` FLOAT,<br />
`FLUX_IVAR_R` FLOAT,<br />
`FLUX_IVAR_Z` FLOAT,<br />
`FLUX_IVAR_W1` FLOAT,<br />
`FLUX_IVAR_W2` FLOAT,<br />
`FLUX_IVAR_W3` FLOAT,<br />
`FLUX_IVAR_W4` FLOAT,<br />
`MW_TRANSMISSION_G` FLOAT,<br />
`MW_TRANSMISSION_R` FLOAT,<br />
`MW_TRANSMISSION_Z` FLOAT,<br />
`MW_TRANSMISSION_W1` FLOAT,<br />
`MW_TRANSMISSION_W2` FLOAT,<br />
`MW_TRANSMISSION_W3` FLOAT,<br />
`MW_TRANSMISSION_W4` FLOAT,<br />
`NOBS_G` INT,<br />
`NOBS_R` INT,<br />
`NOBS_Z` INT,<br />
`NOBS_W1` INT,<br />
`NOBS_W2` INT,<br />
`NOBS_W3` INT,<br />
`NOBS_W4` INT,<br />
`RCHISQ_G` FLOAT,<br />
`RCHISQ_R` FLOAT,<br />
`RCHISQ_Z` FLOAT,<br />
`RCHISQ_W1` FLOAT,<br />
`RCHISQ_W2` FLOAT,<br />
`RCHISQ_W3` FLOAT,<br />
`RCHISQ_W4` FLOAT,<br />
`FRACFLUX_G` FLOAT,<br />
`FRACFLUX_R` FLOAT,<br />
`FRACFLUX_Z` FLOAT,<br />
`FRACFLUX_W1` FLOAT,<br />
`FRACFLUX_W2` FLOAT,<br />
`FRACFLUX_W3` FLOAT,<br />
`FRACFLUX_W4` FLOAT,<br />
`FRACMASKED_G` FLOAT,<br />
`FRACMASKED_R` FLOAT,<br />
`FRACMASKED_Z` FLOAT,<br />
`FRACIN_G` FLOAT,<br />
`FRACIN_R` FLOAT,<br />
`FRACIN_Z` FLOAT,<br />
`ANYMASK_G` SMALLINT,<br />
`ANYMASK_R` SMALLINT,<br />
`ANYMASK_Z` SMALLINT,<br />
`ALLMASK_G` SMALLINT,<br />
`ALLMASK_R` SMALLINT,<br />
`ALLMASK_Z` SMALLINT,<br />
`WISEMASK_W1` TINYINT,<br />
`WISEMASK_W2` TINYINT,<br />
`PSFSIZE_G` FLOAT,<br />
`PSFSIZE_R` FLOAT,<br />
`PSFSIZE_Z` FLOAT,<br />
`PSFDEPTH_G` FLOAT,<br />
`PSFDEPTH_R` FLOAT,<br />
`PSFDEPTH_Z` FLOAT,<br />
`GALDEPTH_G` FLOAT,<br />
`GALDEPTH_R` FLOAT,<br />
`GALDEPTH_Z` FLOAT,<br />
`PSFDEPTH_W1` FLOAT,<br />
`PSFDEPTH_W2` FLOAT,<br />
`WISE_COADD_ID` STRING,<br />
`FRACDEV` FLOAT,<br />
`FRACDEV_IVAR` FLOAT,<br />
`SHAPEDEV_R` FLOAT,<br />
`SHAPEDEV_R_IVAR` FLOAT,<br />
`SHAPEDEV_E1` FLOAT,<br />
`SHAPEDEV_E1_IVAR` FLOAT,<br />
`SHAPEDEV_E2` FLOAT,<br />
`SHAPEDEV_E2_IVAR` FLOAT,<br />
`SHAPEEXP_R` FLOAT,<br />
`SHAPEEXP_R_IVAR` FLOAT,<br />
`SHAPEEXP_E1` FLOAT,<br />
`SHAPEEXP_E1_IVAR` FLOAT,<br />
`SHAPEEXP_E2` FLOAT,<br />
`SHAPEEXP_E2_IVAR` FLOAT,<br />
`FIBERFLUX_G` FLOAT,<br />
`FIBERFLUX_R` FLOAT,<br />
`FIBERFLUX_Z` FLOAT,<br />
`FIBERTOTFLUX_G` FLOAT,<br />
`FIBERTOTFLUX_R` FLOAT,<br />
`FIBERTOTFLUX_Z` FLOAT,<br />
`REF_CAT` STRING,<br />
`REF_ID` BIGINT,<br />
`REF_EPOCH` FLOAT,<br />
`GAIA_PHOT_G_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_G_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_ASTROMETRIC_EXCESS_NOISE` FLOAT,<br />
`GAIA_DUPLICATED_SOURCE` BOOLEAN,<br />
`GAIA_PHOT_BP_RP_EXCESS_FACTOR` FLOAT,<br />
`GAIA_ASTROMETRIC_SIGMA5D_MAX` FLOAT,<br />
`GAIA_ASTROMETRIC_PARAMS_SOLVED` TINYINT,<br />
`PARALLAX` FLOAT,<br />
`PARALLAX_IVAR` FLOAT,<br />
`PMRA` FLOAT,<br />
`PMRA_IVAR` FLOAT,<br />
`PMDEC` FLOAT,<br />
`PMDEC_IVAR` FLOAT,<br />
`MASKBITS` SMALLINT,<br />
`z_phot_mean` FLOAT,<br />
`z_phot_median` FLOAT,<br />
`z_phot_std` FLOAT,<br />
`z_phot_l68` FLOAT,<br />
`z_phot_u68` FLOAT,<br />
`z_phot_l95` FLOAT,<br />
`z_phot_u95` FLOAT,<br />
`z_spec` FLOAT,<br />
`survey` STRING,<br />
`training` BOOLEAN<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey-sweep'<br />
;<br />
<br />
=== Combined table ===<br />
<br />
CREATE TABLE cosmohub.legacysurvey_dr8_phz (<br />
release smallint COMMENT 'Unique integer denoting the camera and filter set used',<br />
brickid int COMMENT 'Brick ID [1,662174]',<br />
brickname string COMMENT 'Name of brick, encoding the brick sky position, eg "1126p222" near RA=112.6, Dec=+22.2',<br />
objid int COMMENT 'Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick',<br />
brick_primary boolean COMMENT 'True if the object is within the brick boundary',<br />
brightblob smallint COMMENT 'bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page',<br />
maskbits smallint COMMENT 'bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits* maps, as cataloged on the DR8 bitmasks page',<br />
type string COMMENT 'Morphological model: "PSF"=stellar, "REX"="round exponential galaxy", "DEV"=deVauc, "EXP"=exponential, "COMP"=composite, "DUP"=Gaia source fit by different model. Note that in some FITS readers, a trailing space may be appended for "PSF ", "DEV " and "EXP " since the column data type is a 4-character string',<br />
ra double COMMENT 'Right ascension at equinox J2000',<br />
dec double COMMENT 'Declination at equinox J2000',<br />
ra_ivar float COMMENT 'Inverse variance of RA (no cosine term!), excluding astrometric calibration errors',<br />
dec_ivar float COMMENT 'Inverse variance of DEC, excluding astrometric calibration errors',<br />
bx float COMMENT 'X position (0-indexed) of coordinates in brick image stack',<br />
`by` float COMMENT 'Y position (0-indexed) of coordinates in brick image stack',<br />
dchisq array<float> COMMENT 'Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.',<br />
ebv float COMMENT 'Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns',<br />
mjd_min double COMMENT 'Minimum Modified Julian Date of observations used to construct the model of this object',<br />
mjd_max double COMMENT 'Maximum Modified Julian Date of observations used to construct the model of this object',<br />
ref_cat string COMMENT 'Reference catalog source for this star: "T2" for Tycho-2, "G2" for Gaia DR2, "L2" for the LSLGA, empty otherwise',<br />
ref_id bigint COMMENT 'Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b "sourceid" for Gaia-DR2 and LSLGA',<br />
pmra float COMMENT 'Reference catalog proper motion in the RA direction',<br />
pmdec float COMMENT 'Reference catalog proper motion in the Dec direction',<br />
parallax float COMMENT 'Reference catalog parallax',<br />
pmra_ivar float COMMENT 'Reference catalog inverse-variance on pmra',<br />
pmdec_ivar float COMMENT 'Reference catalog inverse-variance on pmdec',<br />
parallax_ivar float COMMENT 'Reference catalog inverse-variance on parallax',<br />
ref_epoch float COMMENT 'Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)',<br />
gaia_pointsource boolean COMMENT 'This Gaia DR2 source is believed to be a star, not a galaxy',<br />
gaia_phot_g_mean_mag float COMMENT 'Gaia G band mag',<br />
gaia_phot_g_mean_flux_over_error float COMMENT 'Gaia G band signal-to-noise',<br />
gaia_phot_g_n_obs smallint COMMENT 'Gaia G band number of observations',<br />
gaia_phot_bp_mean_mag float COMMENT 'Gaia BP mag',<br />
gaia_phot_bp_mean_flux_over_error float COMMENT 'Gaia BP signal-to-noise',<br />
gaia_phot_bp_n_obs smallint COMMENT 'Gaia BP number of observations',<br />
gaia_phot_rp_mean_mag float COMMENT 'Gaia RP mag',<br />
gaia_phot_rp_mean_flux_over_error float COMMENT 'Gaia RP signal-to-noise',<br />
gaia_phot_rp_n_obs smallint COMMENT 'Gaia RP number of observations',<br />
gaia_phot_variable_flag boolean COMMENT 'Gaia photometric variable flag',<br />
gaia_astrometric_excess_noise float COMMENT 'Gaia astrometric excess noise',<br />
gaia_astrometric_excess_noise_sig float COMMENT 'Gaia astrometric excess noise uncertainty',<br />
gaia_astrometric_n_obs_al smallint COMMENT 'Gaia number of astrometric observations along scan direction',<br />
gaia_astrometric_n_good_obs_al smallint COMMENT 'Gaia number of good astrometric observations along scan direction',<br />
gaia_astrometric_weight_al float COMMENT 'Gaia astrometric weight along scan direction',<br />
gaia_duplicated_source boolean COMMENT 'Gaia duplicated source flag',<br />
gaia_a_g_val float COMMENT 'Gaia line-of-sight extinction in the G band',<br />
gaia_e_bp_min_rp_val float COMMENT 'Gaia line-of-sight reddening E(BP-RP)',<br />
gaia_phot_bp_rp_excess_factor float COMMENT 'Gaia BP/RP excess factor',<br />
gaia_astrometric_sigma5d_max float COMMENT 'Gaia longest semi-major axis of the 5-d error ellipsoid',<br />
gaia_astrometric_params_solved tinyint COMMENT 'which astrometric parameters were estimated for a Gaia source',<br />
flux_g float COMMENT 'model flux in gg',<br />
flux_r float COMMENT 'model flux in rr',<br />
flux_z float COMMENT 'model flux in zz',<br />
flux_w1 float COMMENT 'WISE model flux in W1W1 (AB system)',<br />
flux_w2 float COMMENT 'WISE model flux in W2W2 (AB)',<br />
flux_w3 float COMMENT 'WISE model flux in W3W3 (AB)',<br />
flux_w4 float COMMENT 'WISE model flux in W4W4 (AB)',<br />
flux_ivar_g float COMMENT 'Inverse variance of flux_g',<br />
flux_ivar_r float COMMENT 'Inverse variance of flux_r',<br />
flux_ivar_z float COMMENT 'Inverse variance of flux_z',<br />
flux_ivar_w1 float COMMENT 'Inverse variance of flux_w1 (AB system)',<br />
flux_ivar_w2 float COMMENT 'Inverse variance of flux_w2 (AB)',<br />
flux_ivar_w3 float COMMENT 'Inverse variance of flux_w3 (AB)',<br />
flux_ivar_w4 float COMMENT 'Inverse variance of flux_w4 (AB)',<br />
fiberflux_g float COMMENT 'Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_r float COMMENT 'Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_z float COMMENT 'Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fibertotflux_g float COMMENT 'Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_r float COMMENT 'Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_z float COMMENT 'Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
apflux_g array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg',<br />
apflux_r array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr',<br />
apflux_z array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz',<br />
apflux_resid_g array<float> COMMENT 'aperture fluxes on the co-added residual images in gg',<br />
apflux_resid_r array<float> COMMENT 'aperture fluxes on the co-added residual images in rr',<br />
apflux_resid_z array<float> COMMENT 'aperture fluxes on the co-added residual images in zz',<br />
apflux_ivar_g array<float> COMMENT 'Inverse variance of apflux_resid_g',<br />
apflux_ivar_r array<float> COMMENT 'Inverse variance of apflux_resid_r',<br />
apflux_ivar_z array<float> COMMENT 'Inverse variance of apflux_resid_z',<br />
mw_transmission_g float COMMENT 'Galactic transmission in gg filter in linear units [0, 1]',<br />
mw_transmission_r float COMMENT 'Galactic transmission in rr filter in linear units [0, 1]',<br />
mw_transmission_z float COMMENT 'Galactic transmission in zz filter in linear units [0, 1]',<br />
mw_transmission_w1 float COMMENT 'Galactic transmission in W1W1 filter in linear units [0, 1]',<br />
mw_transmission_w2 float COMMENT 'Galactic transmission in W2W2 filter in linear units [0, 1]',<br />
mw_transmission_w3 float COMMENT 'Galactic transmission in W3W3 filter in linear units [0, 1]',<br />
mw_transmission_w4 float COMMENT 'Galactic transmission in W4W4 filter in linear units [0, 1]',<br />
nobs_g smallint COMMENT 'Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)',<br />
nobs_r smallint COMMENT 'Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)',<br />
nobs_z smallint COMMENT 'Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)',<br />
nobs_w1 smallint COMMENT 'Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)',<br />
nobs_w2 smallint COMMENT 'Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)',<br />
nobs_w3 smallint COMMENT 'Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)',<br />
nobs_w4 smallint COMMENT 'Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)',<br />
rchisq_g float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in gg',<br />
rchisq_r float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in rr',<br />
rchisq_z float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in zz',<br />
rchisq_w1 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W1W1',<br />
rchisq_w2 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W2W2',<br />
rchisq_w3 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W3W3',<br />
rchisq_w4 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W4W4',<br />
fracflux_g float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in gg (typically [0,1])',<br />
fracflux_r float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in rr (typically [0,1])',<br />
fracflux_z float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in zz (typically [0,1])',<br />
fracflux_w1 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1 (typically [0,1])',<br />
fracflux_w2 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2 (typically [0,1])',<br />
fracflux_w3 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3 (typically [0,1])',<br />
fracflux_w4 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4 (typically [0,1])',<br />
fracmasked_g float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]',<br />
fracmasked_r float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]',<br />
fracmasked_z float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]',<br />
fracin_g float COMMENT 'Fraction of a source\'s flux within the blob in gg, near unity for real sources',<br />
fracin_r float COMMENT 'Fraction of a source\'s flux within the blob in rr, near unity for real sources',<br />
fracin_z float COMMENT 'Fraction of a source\'s flux within the blob in zz, near unity for real sources',<br />
anymask_g smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page',<br />
anymask_r smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page',<br />
anymask_z smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page',<br />
allmask_g smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page',<br />
allmask_r smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page',<br />
allmask_z smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page',<br />
wisemask_w1 tinyint COMMENT 'W1 bitmask as cataloged on the DR8 bitmasks page',<br />
wisemask_w2 tinyint COMMENT 'W2 bitmask as cataloged on the DR8 bitmasks page',<br />
psfsize_g float COMMENT 'Weighted average PSF FWHM in the gg band',<br />
psfsize_r float COMMENT 'Weighted average PSF FWHM in the rr band',<br />
psfsize_z float COMMENT 'Weighted average PSF FWHM in the zz band',<br />
psfdepth_g float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude',<br />
psfdepth_r float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude',<br />
psfdepth_z float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude',<br />
galdepth_g float COMMENT 'As for psfdepth_g but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_r float COMMENT 'As for psfdepth_r but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_z float COMMENT 'As for psfdepth_z but for a galaxy (0.45" exp, round) detection sensitivity',<br />
psfdepth_w1 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W1',<br />
psfdepth_w2 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W2',<br />
psfdepth_w3 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W3',<br />
psfdepth_w4 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W4',<br />
wise_coadd_id string COMMENT 'unWISE coadd file name for the center of each object',<br />
lc_flux_w1 array<float> COMMENT 'flux_w1 in each of up to eleven unWISE coadd epochs (AB system)',<br />
lc_flux_w2 array<float> COMMENT 'flux_w2 in each of up to eleven unWISE coadd epochs (AB)',<br />
lc_flux_ivar_w1 array<float> COMMENT 'Inverse variance of lc_flux_w1 (AB system)',<br />
lc_flux_ivar_w2 array<float> COMMENT 'Inverse variance of lc_flux_w2 (AB)',<br />
lc_nobs_w1 array<smallint> COMMENT 'nobs_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_nobs_w2 array<smallint> COMMENT 'nobs_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w1 array<float> COMMENT 'fracflux_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w2 array<float> COMMENT 'fracflux_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w1 array<float> COMMENT 'rchisq_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w2 array<float> COMMENT 'rchisq_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w1 array<double> COMMENT 'mjd_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w2 array<double> COMMENT 'mjd_w2 in each of up to eleven unWISE coadd epochs',<br />
fracdev float COMMENT 'Fraction of model in deVauc [0,1]',<br />
fracdev_ivar float COMMENT 'Inverse variance of fracdev',<br />
shapeexp_r float COMMENT 'Half-light radius of exponential model (>0)',<br />
shapeexp_r_ivar float COMMENT 'Inverse variance of shapeexp_r',<br />
shapeexp_e1 float COMMENT 'Ellipticity component 1',<br />
shapeexp_e1_ivar float COMMENT 'Inverse variance of shapeexp_e1',<br />
shapeexp_e2 float COMMENT 'Ellipticity component 2',<br />
shapeexp_e2_ivar float COMMENT 'Inverse variance of shapeexp_e2',<br />
shapedev_r float COMMENT 'Half-light radius of deVaucouleurs model (>0)',<br />
shapedev_r_ivar float COMMENT 'Inverse variance of shapedev_r',<br />
shapedev_e1 float COMMENT 'Ellipticity component 1',<br />
shapedev_e1_ivar float COMMENT 'Inverse variance of shapedev_e1',<br />
shapedev_e2 float COMMENT 'Ellipticity component 2',<br />
shapedev_e2_ivar float COMMENT 'Inverse variance of shapedev_e2',<br />
z_phot_mean FLOAT COMMENT 'photo-z derived from the mean of the photo-z PDF ',<br />
z_phot_median FLOAT COMMENT 'photo-z derived from the median of the photo-z PDF',<br />
z_phot_std FLOAT COMMENT 'standard deviation of the photo-z\'s derived from the photo-z PDF',<br />
z_phot_l68 FLOAT COMMENT 'lower bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_u68 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_l95 FLOAT COMMENT 'lower bound of the 95% confidence region, derived from the photo-z PDF',<br />
z_phot_u95 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_spec FLOAT COMMENT 'spectroscopic redshift, if available',<br />
survey STRING COMMENT 'source of the spectroscopic redshift',<br />
training BOOLEAN COMMENT 'whether or not the spectroscopic redshift is used in photometric redshift training'<br />
)<br />
CLUSTERED BY (<br />
release,<br />
brickid,<br />
objid)<br />
SORTED BY (<br />
release ASC,<br />
brickid ASC,<br />
objid ASC)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
=== Join insert ===<br />
<br />
INSERT OVERWRITE TABLE cosmohub.legacysurvey_dr8_phz<br />
SELECT<br />
dr8.*,<br />
phz.z_phot_mean,<br />
phz.z_phot_median,<br />
phz.z_phot_std,<br />
phz.z_phot_l68,<br />
phz.z_phot_u68,<br />
phz.z_phot_l95,<br />
phz.z_phot_u95,<br />
phz.z_spec,<br />
phz.survey,<br />
phz.training<br />
FROM cosmohub.legacy_survey_dr8 AS dr8<br />
LEFT JOIN tallada.dr8_sweep AS phz<br />
ON dr8.release = phz.release<br />
AND dr8.brickid = phz.brickid<br />
AND dr8.objid = phz.objid<br />
;<br />
<br />
== Queries for checking ==<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM cosmohub.legacy_survey_dr8<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(release) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
) AS t<br />
<br />
SELECT COUNT(c), SUM(c) <br />
FROM (<br />
SELECT COUNT(brickid) AS c<br />
FROM tallada.dr8_sweep<br />
GROUP BY brickid, objid<br />
) AS t<br />
<br />
SELECT release, brickid, objid, COUNT(release)<br />
FROM tallada.dr8_sweep<br />
GROUP BY release, brickid, objid<br />
HAVING COUNT(release) > 1<br />
<br />
DR8 release brickid objid: DISTINCT(1.694.230.368) TOTAL(1.694.230.368)<br />
DR8 brickid objid: DISTINCT(1.654.429.115) TOTAL(1.694.230.368)<br />
<br />
PHZ release brickid objid: DISTINCT(1.646.664.729) TOTAL(1.646.664.897)<br />
PHZ brickid objid: DISTINCT(1.608.841.582) TOTAL(1.646.664.897)<br />
<br />
DR8: 1.694.230.368<br />
PHZ: 1.646.664.897<br />
----------------------------------------------------------------------<br />
-47.565.471</div>
Tallada
https://pwiki.pic.es/index.php?title=Legacy_Survey_DR8&diff=800
Legacy Survey DR8
2020-05-07T09:16:24Z
<p>Tallada: /* Sweep photo-z */</p>
<hr />
<div>== Original DR8 ==<br />
=== Raw data ===<br />
<br />
Downloaded from<br />
ftp://archive.noao.edu:/public/hlsp/ls/dr8/<br />
<br />
Directories:<br />
south/tractor<br />
north/tractor<br />
<br />
Mirrored using lftp commands:<br />
mirror -c -v -P 200 north/tractor/ /mnt/pau/north/tractor/<br />
mirror -c -v -P 200 south/tractor/ /mnt/pau/south/tractor/<br />
<br />
into a 5 TiB temporary Ceph filesystem mounted in tdm002.<br />
<br />
Checksums provided through sha256sum files, and checked using command:<br />
find * -type d | xargs -t -L1 -P24 sh -c 'cd $1; sha256sum --quiet -c *.sha256sum' cksum<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to convert fits files to parquet:<br />
<br />
path_base = '/cephfs/pic.es/cosmohub/legacysurvey'<br />
hdfs_base = '/user/tallada/data/legacysurvey'<br />
<br />
hs = ['h=north', 'h=south']<br />
rs = ['r={0:03d}'.format(x) for x in range(360)]<br />
<br />
for h in hs:<br />
for r in rs:<br />
p = os.path.join(path_base, h, r)<br />
if not os.path.exists(p):<br />
continue<br />
<br />
size = sum(os.path.getsize(os.path.join(p,f)) for f in os.listdir(p) if os.path.isfile(os.path.join(p,f)))<br />
<br />
print h, r, size <br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + p<br />
)<br />
<br />
df = df.repartition(int(math.ceil(size / (256*1024*1024.0))))<br />
<br />
df.write.parquet(os.path.join(hdfs_base, h, r))<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE tallada.dr8 (<br />
`release` smallint,<br />
`brickid` int,<br />
`brickname` string,<br />
`objid` int,<br />
`brick_primary` boolean,<br />
`brightblob` smallint,<br />
`maskbits` smallint,<br />
`type` string,<br />
`ra` double,<br />
`dec` double,<br />
`ra_ivar` float,<br />
`dec_ivar` float,<br />
`bx` float,<br />
`by` float,<br />
`dchisq` array<float>,<br />
`ebv` float,<br />
`mjd_min` double,<br />
`mjd_max` double,<br />
`ref_cat` string,<br />
`ref_id` bigint,<br />
`pmra` float,<br />
`pmdec` float,<br />
`parallax` float,<br />
`pmra_ivar` float,<br />
`pmdec_ivar` float,<br />
`parallax_ivar` float,<br />
`ref_epoch` float,<br />
`gaia_pointsource` boolean,<br />
`gaia_phot_g_mean_mag` float,<br />
`gaia_phot_g_mean_flux_over_error` float,<br />
`gaia_phot_g_n_obs` smallint,<br />
`gaia_phot_bp_mean_mag` float,<br />
`gaia_phot_bp_mean_flux_over_error` float,<br />
`gaia_phot_bp_n_obs` smallint,<br />
`gaia_phot_rp_mean_mag` float,<br />
`gaia_phot_rp_mean_flux_over_error` float,<br />
`gaia_phot_rp_n_obs` smallint,<br />
`gaia_phot_variable_flag` boolean,<br />
`gaia_astrometric_excess_noise` float,<br />
`gaia_astrometric_excess_noise_sig` float,<br />
`gaia_astrometric_n_obs_al` smallint,<br />
`gaia_astrometric_n_good_obs_al` smallint,<br />
`gaia_astrometric_weight_al` float,<br />
`gaia_duplicated_source` boolean,<br />
`gaia_a_g_val` float,<br />
`gaia_e_bp_min_rp_val` float,<br />
`gaia_phot_bp_rp_excess_factor` float,<br />
`gaia_astrometric_sigma5d_max` float,<br />
`gaia_astrometric_params_solved` tinyint,<br />
`flux_g` float,<br />
`flux_r` float,<br />
`flux_z` float,<br />
`flux_w1` float,<br />
`flux_w2` float,<br />
`flux_w3` float,<br />
`flux_w4` float,<br />
`flux_ivar_g` float,<br />
`flux_ivar_r` float,<br />
`flux_ivar_z` float,<br />
`flux_ivar_w1` float,<br />
`flux_ivar_w2` float,<br />
`flux_ivar_w3` float,<br />
`flux_ivar_w4` float,<br />
`fiberflux_g` float,<br />
`fiberflux_r` float,<br />
`fiberflux_z` float,<br />
`fibertotflux_g` float,<br />
`fibertotflux_r` float,<br />
`fibertotflux_z` float,<br />
`apflux_g` array<float>,<br />
`apflux_r` array<float>,<br />
`apflux_z` array<float>,<br />
`apflux_resid_g` array<float>,<br />
`apflux_resid_r` array<float>,<br />
`apflux_resid_z` array<float>,<br />
`apflux_ivar_g` array<float>,<br />
`apflux_ivar_r` array<float>,<br />
`apflux_ivar_z` array<float>,<br />
`mw_transmission_g` float,<br />
`mw_transmission_r` float,<br />
`mw_transmission_z` float,<br />
`mw_transmission_w1` float,<br />
`mw_transmission_w2` float,<br />
`mw_transmission_w3` float,<br />
`mw_transmission_w4` float,<br />
`nobs_g` smallint,<br />
`nobs_r` smallint,<br />
`nobs_z` smallint,<br />
`nobs_w1` smallint,<br />
`nobs_w2` smallint,<br />
`nobs_w3` smallint,<br />
`nobs_w4` smallint,<br />
`rchisq_g` float,<br />
`rchisq_r` float,<br />
`rchisq_z` float,<br />
`rchisq_w1` float,<br />
`rchisq_w2` float,<br />
`rchisq_w3` float,<br />
`rchisq_w4` float,<br />
`fracflux_g` float,<br />
`fracflux_r` float,<br />
`fracflux_z` float,<br />
`fracflux_w1` float,<br />
`fracflux_w2` float,<br />
`fracflux_w3` float,<br />
`fracflux_w4` float,<br />
`fracmasked_g` float,<br />
`fracmasked_r` float,<br />
`fracmasked_z` float,<br />
`fracin_g` float,<br />
`fracin_r` float,<br />
`fracin_z` float,<br />
`anymask_g` smallint,<br />
`anymask_r` smallint,<br />
`anymask_z` smallint,<br />
`allmask_g` smallint,<br />
`allmask_r` smallint,<br />
`allmask_z` smallint,<br />
`wisemask_w1` tinyint,<br />
`wisemask_w2` tinyint,<br />
`psfsize_g` float,<br />
`psfsize_r` float,<br />
`psfsize_z` float,<br />
`psfdepth_g` float,<br />
`psfdepth_r` float,<br />
`psfdepth_z` float,<br />
`galdepth_g` float,<br />
`galdepth_r` float,<br />
`galdepth_z` float,<br />
`psfdepth_w1` float,<br />
`psfdepth_w2` float,<br />
`psfdepth_w3` float,<br />
`psfdepth_w4` float,<br />
`wise_coadd_id` string,<br />
`lc_flux_w1` array<float>,<br />
`lc_flux_w2` array<float>,<br />
`lc_flux_ivar_w1` array<float>,<br />
`lc_flux_ivar_w2` array<float>,<br />
`lc_nobs_w1` array<smallint>,<br />
`lc_nobs_w2` array<smallint>,<br />
`lc_fracflux_w1` array<float>,<br />
`lc_fracflux_w2` array<float>,<br />
`lc_rchisq_w1` array<float>,<br />
`lc_rchisq_w2` array<float>,<br />
`lc_mjd_w1` array<double>,<br />
`lc_mjd_w2` array<double>,<br />
`fracdev` float,<br />
`fracdev_ivar` float,<br />
`shapeexp_r` float,<br />
`shapeexp_r_ivar` float,<br />
`shapeexp_e1` float,<br />
`shapeexp_e1_ivar` float,<br />
`shapeexp_e2` float,<br />
`shapeexp_e2_ivar` float,<br />
`shapedev_r` float,<br />
`shapedev_r_ivar` float,<br />
`shapedev_e1` float,<br />
`shapedev_e1_ivar` float,<br />
`shapedev_e2` float,<br />
`shapedev_e2_ivar` float<br />
)<br />
PARTITIONED BY (<br />
h string,<br />
r string<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey/dr8/'<br />
;<br />
<br />
=== Internal Table ===<br />
<br />
CREATE TABLE cosmohub.legacy_survey_dr8 (<br />
`release` smallint,<br />
`brickid` int,<br />
`brickname` string,<br />
`objid` int,<br />
`brick_primary` boolean,<br />
`brightblob` smallint,<br />
`maskbits` smallint,<br />
`type` string,<br />
`ra` double,<br />
`dec` double,<br />
`ra_ivar` float,<br />
`dec_ivar` float,<br />
`bx` float,<br />
`by` float,<br />
`dchisq` array<float>,<br />
`ebv` float,<br />
`mjd_min` double,<br />
`mjd_max` double,<br />
`ref_cat` string,<br />
`ref_id` bigint,<br />
`pmra` float,<br />
`pmdec` float,<br />
`parallax` float,<br />
`pmra_ivar` float,<br />
`pmdec_ivar` float,<br />
`parallax_ivar` float,<br />
`ref_epoch` float,<br />
`gaia_pointsource` boolean,<br />
`gaia_phot_g_mean_mag` float,<br />
`gaia_phot_g_mean_flux_over_error` float,<br />
`gaia_phot_g_n_obs` smallint,<br />
`gaia_phot_bp_mean_mag` float,<br />
`gaia_phot_bp_mean_flux_over_error` float,<br />
`gaia_phot_bp_n_obs` smallint,<br />
`gaia_phot_rp_mean_mag` float,<br />
`gaia_phot_rp_mean_flux_over_error` float,<br />
`gaia_phot_rp_n_obs` smallint,<br />
`gaia_phot_variable_flag` boolean,<br />
`gaia_astrometric_excess_noise` float,<br />
`gaia_astrometric_excess_noise_sig` float,<br />
`gaia_astrometric_n_obs_al` smallint,<br />
`gaia_astrometric_n_good_obs_al` smallint,<br />
`gaia_astrometric_weight_al` float,<br />
`gaia_duplicated_source` boolean,<br />
`gaia_a_g_val` float,<br />
`gaia_e_bp_min_rp_val` float,<br />
`gaia_phot_bp_rp_excess_factor` float,<br />
`gaia_astrometric_sigma5d_max` float,<br />
`gaia_astrometric_params_solved` tinyint,<br />
`flux_g` float,<br />
`flux_r` float,<br />
`flux_z` float,<br />
`flux_w1` float,<br />
`flux_w2` float,<br />
`flux_w3` float,<br />
`flux_w4` float,<br />
`flux_ivar_g` float,<br />
`flux_ivar_r` float,<br />
`flux_ivar_z` float,<br />
`flux_ivar_w1` float,<br />
`flux_ivar_w2` float,<br />
`flux_ivar_w3` float,<br />
`flux_ivar_w4` float,<br />
`fiberflux_g` float,<br />
`fiberflux_r` float,<br />
`fiberflux_z` float,<br />
`fibertotflux_g` float,<br />
`fibertotflux_r` float,<br />
`fibertotflux_z` float,<br />
`apflux_g` array<float>,<br />
`apflux_r` array<float>,<br />
`apflux_z` array<float>,<br />
`apflux_resid_g` array<float>,<br />
`apflux_resid_r` array<float>,<br />
`apflux_resid_z` array<float>,<br />
`apflux_ivar_g` array<float>,<br />
`apflux_ivar_r` array<float>,<br />
`apflux_ivar_z` array<float>,<br />
`mw_transmission_g` float,<br />
`mw_transmission_r` float,<br />
`mw_transmission_z` float,<br />
`mw_transmission_w1` float,<br />
`mw_transmission_w2` float,<br />
`mw_transmission_w3` float,<br />
`mw_transmission_w4` float,<br />
`nobs_g` smallint,<br />
`nobs_r` smallint,<br />
`nobs_z` smallint,<br />
`nobs_w1` smallint,<br />
`nobs_w2` smallint,<br />
`nobs_w3` smallint,<br />
`nobs_w4` smallint,<br />
`rchisq_g` float,<br />
`rchisq_r` float,<br />
`rchisq_z` float,<br />
`rchisq_w1` float,<br />
`rchisq_w2` float,<br />
`rchisq_w3` float,<br />
`rchisq_w4` float,<br />
`fracflux_g` float,<br />
`fracflux_r` float,<br />
`fracflux_z` float,<br />
`fracflux_w1` float,<br />
`fracflux_w2` float,<br />
`fracflux_w3` float,<br />
`fracflux_w4` float,<br />
`fracmasked_g` float,<br />
`fracmasked_r` float,<br />
`fracmasked_z` float,<br />
`fracin_g` float,<br />
`fracin_r` float,<br />
`fracin_z` float,<br />
`anymask_g` smallint,<br />
`anymask_r` smallint,<br />
`anymask_z` smallint,<br />
`allmask_g` smallint,<br />
`allmask_r` smallint,<br />
`allmask_z` smallint,<br />
`wisemask_w1` tinyint,<br />
`wisemask_w2` tinyint,<br />
`psfsize_g` float,<br />
`psfsize_r` float,<br />
`psfsize_z` float,<br />
`psfdepth_g` float,<br />
`psfdepth_r` float,<br />
`psfdepth_z` float,<br />
`galdepth_g` float,<br />
`galdepth_r` float,<br />
`galdepth_z` float,<br />
`psfdepth_w1` float,<br />
`psfdepth_w2` float,<br />
`psfdepth_w3` float,<br />
`psfdepth_w4` float,<br />
`wise_coadd_id` string,<br />
`lc_flux_w1` array<float>,<br />
`lc_flux_w2` array<float>,<br />
`lc_flux_ivar_w1` array<float>,<br />
`lc_flux_ivar_w2` array<float>,<br />
`lc_nobs_w1` array<smallint>,<br />
`lc_nobs_w2` array<smallint>,<br />
`lc_fracflux_w1` array<float>,<br />
`lc_fracflux_w2` array<float>,<br />
`lc_rchisq_w1` array<float>,<br />
`lc_rchisq_w2` array<float>,<br />
`lc_mjd_w1` array<double>,<br />
`lc_mjd_w2` array<double>,<br />
`fracdev` float,<br />
`fracdev_ivar` float,<br />
`shapeexp_r` float,<br />
`shapeexp_r_ivar` float,<br />
`shapeexp_e1` float,<br />
`shapeexp_e1_ivar` float,<br />
`shapeexp_e2` float,<br />
`shapeexp_e2_ivar` float,<br />
`shapedev_r` float,<br />
`shapedev_r_ivar` float,<br />
`shapedev_e1` float,<br />
`shapedev_e1_ivar` float,<br />
`shapedev_e2` float,<br />
`shapedev_e2_ivar` float<br />
)<br />
clustered by (<br />
release, brickid, objid<br />
)<br />
sorted by (<br />
release, brickid, objid<br />
)<br />
into 4096 buckets<br />
stored as orc<br />
;<br />
<br />
insert overwrite table cosmohub.legacy_survey_dr8<br />
select release, brickid, brickname, objid, brick_primary, brightblob, maskbits, type, ra, dec, ra_ivar, dec_ivar, bx, `by`, <br />
dchisq, ebv, mjd_min, mjd_max, ref_cat, ref_id, pmra, pmdec, parallax, pmra_ivar, pmdec_ivar, parallax_ivar, ref_epoch,<br />
gaia_pointsource, gaia_phot_g_mean_mag, gaia_phot_g_mean_flux_over_error, gaia_phot_g_n_obs,<br />
gaia_phot_bp_mean_mag, gaia_phot_bp_mean_flux_over_error, gaia_phot_bp_n_obs, gaia_phot_rp_mean_mag,<br />
gaia_phot_rp_mean_flux_over_error, gaia_phot_rp_n_obs, gaia_phot_variable_flag, gaia_astrometric_excess_noise,<br />
gaia_astrometric_excess_noise_sig, gaia_astrometric_n_obs_al, gaia_astrometric_n_good_obs_al,<br />
gaia_astrometric_weight_al, gaia_duplicated_source, gaia_a_g_val, gaia_e_bp_min_rp_val,<br />
gaia_phot_bp_rp_excess_factor, gaia_astrometric_sigma5d_max, gaia_astrometric_params_solved,<br />
flux_g, flux_r, flux_z, flux_w1, flux_w2, flux_w3, flux_w4, flux_ivar_g, flux_ivar_r,<br />
flux_ivar_z, flux_ivar_w1, flux_ivar_w2, flux_ivar_w3, flux_ivar_w4, fiberflux_g, fiberflux_r,<br />
fiberflux_z, fibertotflux_g, fibertotflux_r, fibertotflux_z, apflux_g, apflux_r, apflux_z,<br />
apflux_resid_g, apflux_resid_r, apflux_resid_z, apflux_ivar_g, apflux_ivar_r, apflux_ivar_z,<br />
mw_transmission_g, mw_transmission_r, mw_transmission_z, mw_transmission_w1, mw_transmission_w2,<br />
mw_transmission_w3, mw_transmission_w4, nobs_g, nobs_r, nobs_z, nobs_w1, nobs_w2, nobs_w3,<br />
nobs_w4, rchisq_g, rchisq_r, rchisq_z, rchisq_w1, rchisq_w2, rchisq_w3, rchisq_w4, fracflux_g,<br />
fracflux_r, fracflux_z, fracflux_w1, fracflux_w2, fracflux_w3, fracflux_w4, fracmasked_g,<br />
fracmasked_r, fracmasked_z, fracin_g, fracin_r, fracin_z, anymask_g, anymask_r, anymask_z,<br />
allmask_g, allmask_r, allmask_z, wisemask_w1, wisemask_w2, psfsize_g, psfsize_r, psfsize_z,<br />
psfdepth_g, psfdepth_r, psfdepth_z, galdepth_g, galdepth_r, galdepth_z, psfdepth_w1, psfdepth_w2,<br />
psfdepth_w3, psfdepth_w4, wise_coadd_id, lc_flux_w1, lc_flux_w2, lc_flux_ivar_w1, lc_flux_ivar_w2,<br />
lc_nobs_w1, lc_nobs_w2, lc_fracflux_w1, lc_fracflux_w2, lc_rchisq_w1, lc_rchisq_w2, lc_mjd_w1,<br />
lc_mjd_w2, fracdev, fracdev_ivar, shapeexp_r, shapeexp_r_ivar, shapeexp_e1, shapeexp_e1_ivar,<br />
shapeexp_e2, shapeexp_e2_ivar, shapedev_r, shapedev_r_ivar, shapedev_e1, shapedev_e1_ivar,<br />
shapedev_e2, shapedev_e2_ivar<br />
from tallada.dr8<br />
;<br />
<br />
=== Comments ===<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE release release smallint COMMENT "Unique integer denoting the camera and filter set used";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickid brickid int COMMENT "Brick ID [1,662174]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickname brickname string COMMENT "Name of brick, encoding the brick sky position, eg \"1126p222\" near RA=112.6, Dec=+22.2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE objid objid int COMMENT "Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brick_primary brick_primary boolean COMMENT "True if the object is within the brick boundary";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brightblob brightblob smallint COMMENT "bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE maskbits maskbits smallint COMMENT "bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits*maps, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE type type string COMMENT "Morphological model: PSF=stellar, REX=round exponential galaxy, DEV=deVauc, EXP=exponential, COMP=composite, DUP=Gaia source fit by different model.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra ra double COMMENT "Right ascension at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec dec double COMMENT "Declination at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra_ivar ra_ivar float COMMENT "Inverse variance of RA (no cosine term!), excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec_ivar dec_ivar float COMMENT "Inverse variance of DEC, excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE bx bx float COMMENT "X position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE by by float COMMENT "Y position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dchisq dchisq array<float> COMMENT "Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ebv ebv float COMMENT "Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_min mjd_min double COMMENT "Minimum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_max mjd_max double COMMENT "Maximum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_cat ref_cat string COMMENT "Reference catalog source for this star: T2 for Tycho-2, G2 for Gaia DR2, L2 for the LSLGA, empty otherwise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_id ref_id bigint COMMENT "Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b \"sourceid\" for Gaia-DR2 and LSLGA";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra pmra float COMMENT "Reference catalog proper motion in the RA direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec pmdec float COMMENT "Reference catalog proper motion in the Dec direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax parallax float COMMENT "Reference catalog parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra_ivar pmra_ivar float COMMENT "Reference catalog inverse-variance on pmra";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec_ivar pmdec_ivar float COMMENT "Reference catalog inverse-variance on pmdec";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax_ivar parallax_ivar float COMMENT "Reference catalog inverse-variance on parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_epoch ref_epoch float COMMENT "Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_pointsource gaia_pointsource boolean COMMENT "This Gaia DR2 source is believed to be a star, not a galaxy";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_mag gaia_phot_g_mean_mag float COMMENT "Gaia G band mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_flux_over_error gaia_phot_g_mean_flux_over_error float COMMENT "Gaia G band signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_n_obs gaia_phot_g_n_obs smallint COMMENT "Gaia G band number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_mag gaia_phot_bp_mean_mag float COMMENT "Gaia BP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_flux_over_error gaia_phot_bp_mean_flux_over_error float COMMENT "Gaia BP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_n_obs gaia_phot_bp_n_obs smallint COMMENT "Gaia BP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_mag gaia_phot_rp_mean_mag float COMMENT "Gaia RP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_flux_over_error gaia_phot_rp_mean_flux_over_error float COMMENT "Gaia RP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_n_obs gaia_phot_rp_n_obs smallint COMMENT "Gaia RP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_variable_flag gaia_phot_variable_flag boolean COMMENT "Gaia photometric variable flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise gaia_astrometric_excess_noise float COMMENT "Gaia astrometric excess noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise_sig gaia_astrometric_excess_noise_sig float COMMENT "Gaia astrometric excess noise uncertainty";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_obs_al gaia_astrometric_n_obs_al smallint COMMENT "Gaia number of astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_good_obs_al gaia_astrometric_n_good_obs_al smallint COMMENT "Gaia number of good astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_weight_al gaia_astrometric_weight_al float COMMENT "Gaia astrometric weight along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_duplicated_source gaia_duplicated_source boolean COMMENT "Gaia duplicated source flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_a_g_val gaia_a_g_val float COMMENT "Gaia line-of-sight extinction in the G band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_e_bp_min_rp_val gaia_e_bp_min_rp_val float COMMENT "Gaia line-of-sight reddening E(BP-RP)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_rp_excess_factor gaia_phot_bp_rp_excess_factor float COMMENT "Gaia BP/RP excess factor";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_sigma5d_max gaia_astrometric_sigma5d_max float COMMENT "Gaia longest semi-major axis of the 5-d error ellipsoid";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_params_solved gaia_astrometric_params_solved tinyint COMMENT "which astrometric parameters were estimated for a Gaia source";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_g flux_g float COMMENT "model flux in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_r flux_r float COMMENT "model flux in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_z flux_z float COMMENT "model flux in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w1 flux_w1 float COMMENT "WISE model flux in W1W1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w2 flux_w2 float COMMENT "WISE model flux in W2W2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w3 flux_w3 float COMMENT "WISE model flux in W3W3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w4 flux_w4 float COMMENT "WISE model flux in W4W4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_g flux_ivar_g float COMMENT "Inverse variance of flux_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_r flux_ivar_r float COMMENT "Inverse variance of flux_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_z flux_ivar_z float COMMENT "Inverse variance of flux_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w1 flux_ivar_w1 float COMMENT "Inverse variance of flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w2 flux_ivar_w2 float COMMENT "Inverse variance of flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w3 flux_ivar_w3 float COMMENT "Inverse variance of flux_w3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w4 flux_ivar_w4 float COMMENT "Inverse variance of flux_w4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_g fiberflux_g float COMMENT "Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_r fiberflux_r float COMMENT "Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_z fiberflux_z float COMMENT "Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_g fibertotflux_g float COMMENT "Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_r fibertotflux_r float COMMENT "Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_z fibertotflux_z float COMMENT "Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_g apflux_g array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_r apflux_r array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_z apflux_z array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_g apflux_resid_g array<float> COMMENT "aperture fluxes on the co-added residual images in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_r apflux_resid_r array<float> COMMENT "aperture fluxes on the co-added residual images in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_z apflux_resid_z array<float> COMMENT "aperture fluxes on the co-added residual images in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_g apflux_ivar_g array<float> COMMENT "Inverse variance of apflux_resid_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_r apflux_ivar_r array<float> COMMENT "Inverse variance of apflux_resid_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_z apflux_ivar_z array<float> COMMENT "Inverse variance of apflux_resid_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_g mw_transmission_g float COMMENT "Galactic transmission in gg filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_r mw_transmission_r float COMMENT "Galactic transmission in rr filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_z mw_transmission_z float COMMENT "Galactic transmission in zz filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w1 mw_transmission_w1 float COMMENT "Galactic transmission in W1W1 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w2 mw_transmission_w2 float COMMENT "Galactic transmission in W2W2 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w3 mw_transmission_w3 float COMMENT "Galactic transmission in W3W3 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w4 mw_transmission_w4 float COMMENT "Galactic transmission in W4W4 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_g nobs_g smallint COMMENT "Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_r nobs_r smallint COMMENT "Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_z nobs_z smallint COMMENT "Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w1 nobs_w1 smallint COMMENT "Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w2 nobs_w2 smallint COMMENT "Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w3 nobs_w3 smallint COMMENT "Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w4 nobs_w4 smallint COMMENT "Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_g rchisq_g float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_r rchisq_r float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_z rchisq_z float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w1 rchisq_w1 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W1W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w2 rchisq_w2 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W2W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w3 rchisq_w3 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W3W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w4 rchisq_w4 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W4W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_g fracflux_g float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in gg(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_r fracflux_r float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in rr(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_z fracflux_z float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in zz(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w1 fracflux_w1 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w2 fracflux_w2 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w3 fracflux_w3 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w4 fracflux_w4 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_g fracmasked_g float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_r fracmasked_r float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_z fracmasked_z float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_g fracin_g float COMMENT "Fraction of a source's flux within the blob in gg, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_r fracin_r float COMMENT "Fraction of a source's flux within the blob in rr, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_z fracin_z float COMMENT "Fraction of a source's flux within the blob in zz, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_g anymask_g smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_r anymask_r smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_z anymask_z smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_g allmask_g smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_r allmask_r smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_z allmask_z smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w1 wisemask_w1 tinyint COMMENT "W1 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w2 wisemask_w2 tinyint COMMENT "W2 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_g psfsize_g float COMMENT "Weighted average PSF FWHM in the gg band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_r psfsize_r float COMMENT "Weighted average PSF FWHM in the rr band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_z psfsize_z float COMMENT "Weighted average PSF FWHM in the zz band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_g psfdepth_g float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_r psfdepth_r float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_z psfdepth_z float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_g galdepth_g float COMMENT "As for psfdepth_g but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_r galdepth_r float COMMENT "As for psfdepth_r but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_z galdepth_z float COMMENT "As for psfdepth_z but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w1 psfdepth_w1 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w2 psfdepth_w2 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w3 psfdepth_w3 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w4 psfdepth_w4 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wise_coadd_id wise_coadd_id string COMMENT "unWISE coadd file name for the center of each object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w1 lc_flux_w1 array<float> COMMENT "flux_w1 in each of up to eleven unWISE coadd epochs (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w2 lc_flux_w2 array<float> COMMENT "flux_w2 in each of up to eleven unWISE coadd epochs (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w1 lc_flux_ivar_w1 array<float> COMMENT "Inverse variance of lc_flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w2 lc_flux_ivar_w2 array<float> COMMENT "Inverse variance of lc_flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w1 lc_nobs_w1 array<smallint> COMMENT "nobs_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w2 lc_nobs_w2 array<smallint> COMMENT "nobs_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w1 lc_fracflux_w1 array<float> COMMENT "fracflux_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w2 lc_fracflux_w2 array<float> COMMENT "fracflux_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w1 lc_rchisq_w1 array<float> COMMENT "rchisq_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w2 lc_rchisq_w2 array<float> COMMENT "rchisq_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w1 lc_mjd_w1 array<double> COMMENT "mjd_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w2 lc_mjd_w2 array<double> COMMENT "mjd_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev fracdev float COMMENT "Fraction of model in deVauc [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev_ivar fracdev_ivar float COMMENT "Inverse variance of fracdev";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r shapeexp_r float COMMENT "Half-light radius of exponential model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r_ivar shapeexp_r_ivar float COMMENT "Inverse variance of shapeexp_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1 shapeexp_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1_ivar shapeexp_e1_ivar float COMMENT "Inverse variance of shapeexp_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2 shapeexp_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2_ivar shapeexp_e2_ivar float COMMENT "Inverse variance of shapeexp_e2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r shapedev_r float COMMENT "Half-light radius of deVaucouleurs model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r_ivar shapedev_r_ivar float COMMENT "Inverse variance of shapedev_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1 shapedev_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1_ivar shapedev_e1_ivar float COMMENT "Inverse variance of shapedev_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2 shapedev_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2_ivar shapedev_e2_ivar float COMMENT "Inverse variance of shapedev_e2";<br />
<br />
== Sweep photo-z ==<br />
<br />
=== Raw data ===<br />
<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/north/sweep/<br />
wget -c --no-clobber --no-parent -r https://portal.nersc.gov/project/cosmo/data/legacysurvey/dr8/south/sweep/<br />
<br />
Checksums checked:<br />
<br />
sha256sum -c legacysurvey_dr8_north_sweep_8.0.sha256sum<br />
sha256sum -c legacysurvey_dr8_south_sweep_8.0.sha256sum<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to merge fits files and convert to parquet:<br />
<br />
path_base = '/cephfs/pic.es/astro/scratch/tallada/dr8-phz/'<br />
hdfs_base = '/user/tallada/data/legacysurvey-sweep'<br />
<br />
hs = ['south/sweep'] #, 'north/sweep']<br />
dir_src = '8.0'<br />
dir_phz = '8.0-photo-z'<br />
dir_dst = '8.0-merged'<br />
<br />
my_files = sorted(os.listdir(os.path.join(path_base, hs[0], dir_src)))<br />
<br />
def merge_phz(file_name):<br />
file_src = os.path.join(path_base, hs[0], dir_src, file_name)<br />
file_phz = os.path.join(path_base, hs[0], dir_phz, file_name[:-5]+'-pz.fits')<br />
file_dst = os.path.join(path_base, hs[0], dir_dst, file_name[:-5]+'-merged.fits')<br />
<br />
t_src = table.Table.read(file_src)<br />
t_phz = table.Table.read(file_phz)<br />
t_dst = table.hstack([t_src, t_phz], join_type='exact')<br />
<br />
os.unlink(file_dst)<br />
t_dst.write(file_dst)<br />
<br />
return file_name<br />
<br />
df = sc.parallelize(my_files, len(my_files))<br />
<br />
df.map(merge_phz).count()<br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + os.path.join(path_base, hs[0], dir_dst)<br />
) <br />
<br />
df.write.parquet(os.path.join(hdfs_base, hs[0]))<br />
<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE tallada.dr8_sweep (<br />
`RELEASE` SMALLINT,<br />
`BRICKID` INT,<br />
`BRICKNAME` STRING,<br />
`OBJID` INT,<br />
`TYPE` STRING,<br />
`RA` DOUBLE,<br />
`DEC` DOUBLE,<br />
`RA_IVAR` FLOAT,<br />
`DEC_IVAR` FLOAT,<br />
`DCHISQ` ARRAY<FLOAT>,<br />
`EBV` FLOAT,<br />
`FLUX_G` FLOAT,<br />
`FLUX_R` FLOAT,<br />
`FLUX_Z` FLOAT,<br />
`FLUX_W1` FLOAT,<br />
`FLUX_W2` FLOAT,<br />
`FLUX_W3` FLOAT,<br />
`FLUX_W4` FLOAT,<br />
`FLUX_IVAR_G` FLOAT,<br />
`FLUX_IVAR_R` FLOAT,<br />
`FLUX_IVAR_Z` FLOAT,<br />
`FLUX_IVAR_W1` FLOAT,<br />
`FLUX_IVAR_W2` FLOAT,<br />
`FLUX_IVAR_W3` FLOAT,<br />
`FLUX_IVAR_W4` FLOAT,<br />
`MW_TRANSMISSION_G` FLOAT,<br />
`MW_TRANSMISSION_R` FLOAT,<br />
`MW_TRANSMISSION_Z` FLOAT,<br />
`MW_TRANSMISSION_W1` FLOAT,<br />
`MW_TRANSMISSION_W2` FLOAT,<br />
`MW_TRANSMISSION_W3` FLOAT,<br />
`MW_TRANSMISSION_W4` FLOAT,<br />
`NOBS_G` INT,<br />
`NOBS_R` INT,<br />
`NOBS_Z` INT,<br />
`NOBS_W1` INT,<br />
`NOBS_W2` INT,<br />
`NOBS_W3` INT,<br />
`NOBS_W4` INT,<br />
`RCHISQ_G` FLOAT,<br />
`RCHISQ_R` FLOAT,<br />
`RCHISQ_Z` FLOAT,<br />
`RCHISQ_W1` FLOAT,<br />
`RCHISQ_W2` FLOAT,<br />
`RCHISQ_W3` FLOAT,<br />
`RCHISQ_W4` FLOAT,<br />
`FRACFLUX_G` FLOAT,<br />
`FRACFLUX_R` FLOAT,<br />
`FRACFLUX_Z` FLOAT,<br />
`FRACFLUX_W1` FLOAT,<br />
`FRACFLUX_W2` FLOAT,<br />
`FRACFLUX_W3` FLOAT,<br />
`FRACFLUX_W4` FLOAT,<br />
`FRACMASKED_G` FLOAT,<br />
`FRACMASKED_R` FLOAT,<br />
`FRACMASKED_Z` FLOAT,<br />
`FRACIN_G` FLOAT,<br />
`FRACIN_R` FLOAT,<br />
`FRACIN_Z` FLOAT,<br />
`ANYMASK_G` SMALLINT,<br />
`ANYMASK_R` SMALLINT,<br />
`ANYMASK_Z` SMALLINT,<br />
`ALLMASK_G` SMALLINT,<br />
`ALLMASK_R` SMALLINT,<br />
`ALLMASK_Z` SMALLINT,<br />
`WISEMASK_W1` TINYINT,<br />
`WISEMASK_W2` TINYINT,<br />
`PSFSIZE_G` FLOAT,<br />
`PSFSIZE_R` FLOAT,<br />
`PSFSIZE_Z` FLOAT,<br />
`PSFDEPTH_G` FLOAT,<br />
`PSFDEPTH_R` FLOAT,<br />
`PSFDEPTH_Z` FLOAT,<br />
`GALDEPTH_G` FLOAT,<br />
`GALDEPTH_R` FLOAT,<br />
`GALDEPTH_Z` FLOAT,<br />
`PSFDEPTH_W1` FLOAT,<br />
`PSFDEPTH_W2` FLOAT,<br />
`WISE_COADD_ID` STRING,<br />
`FRACDEV` FLOAT,<br />
`FRACDEV_IVAR` FLOAT,<br />
`SHAPEDEV_R` FLOAT,<br />
`SHAPEDEV_R_IVAR` FLOAT,<br />
`SHAPEDEV_E1` FLOAT,<br />
`SHAPEDEV_E1_IVAR` FLOAT,<br />
`SHAPEDEV_E2` FLOAT,<br />
`SHAPEDEV_E2_IVAR` FLOAT,<br />
`SHAPEEXP_R` FLOAT,<br />
`SHAPEEXP_R_IVAR` FLOAT,<br />
`SHAPEEXP_E1` FLOAT,<br />
`SHAPEEXP_E1_IVAR` FLOAT,<br />
`SHAPEEXP_E2` FLOAT,<br />
`SHAPEEXP_E2_IVAR` FLOAT,<br />
`FIBERFLUX_G` FLOAT,<br />
`FIBERFLUX_R` FLOAT,<br />
`FIBERFLUX_Z` FLOAT,<br />
`FIBERTOTFLUX_G` FLOAT,<br />
`FIBERTOTFLUX_R` FLOAT,<br />
`FIBERTOTFLUX_Z` FLOAT,<br />
`REF_CAT` STRING,<br />
`REF_ID` BIGINT,<br />
`REF_EPOCH` FLOAT,<br />
`GAIA_PHOT_G_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_G_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_BP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_MAG` FLOAT,<br />
`GAIA_PHOT_RP_MEAN_FLUX_OVER_ERROR` FLOAT,<br />
`GAIA_ASTROMETRIC_EXCESS_NOISE` FLOAT,<br />
`GAIA_DUPLICATED_SOURCE` BOOLEAN,<br />
`GAIA_PHOT_BP_RP_EXCESS_FACTOR` FLOAT,<br />
`GAIA_ASTROMETRIC_SIGMA5D_MAX` FLOAT,<br />
`GAIA_ASTROMETRIC_PARAMS_SOLVED` TINYINT,<br />
`PARALLAX` FLOAT,<br />
`PARALLAX_IVAR` FLOAT,<br />
`PMRA` FLOAT,<br />
`PMRA_IVAR` FLOAT,<br />
`PMDEC` FLOAT,<br />
`PMDEC_IVAR` FLOAT,<br />
`MASKBITS` SMALLINT,<br />
`z_phot_mean` FLOAT,<br />
`z_phot_median` FLOAT,<br />
`z_phot_std` FLOAT,<br />
`z_phot_l68` FLOAT,<br />
`z_phot_u68` FLOAT,<br />
`z_phot_l95` FLOAT,<br />
`z_phot_u95` FLOAT,<br />
`z_spec` FLOAT,<br />
`survey` STRING,<br />
`training` BOOLEAN<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey-sweep'<br />
;<br />
<br />
=== Combined table ===<br />
<br />
CREATE TABLE cosmohub.legacysurvey_dr8_phz (<br />
release smallint COMMENT 'Unique integer denoting the camera and filter set used',<br />
brickid int COMMENT 'Brick ID [1,662174]',<br />
brickname string COMMENT 'Name of brick, encoding the brick sky position, eg "1126p222" near RA=112.6, Dec=+22.2',<br />
objid int COMMENT 'Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick',<br />
brick_primary boolean COMMENT 'True if the object is within the brick boundary',<br />
brightblob smallint COMMENT 'bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page',<br />
maskbits smallint COMMENT 'bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits* maps, as cataloged on the DR8 bitmasks page',<br />
type string COMMENT 'Morphological model: "PSF"=stellar, "REX"="round exponential galaxy", "DEV"=deVauc, "EXP"=exponential, "COMP"=composite, "DUP"=Gaia source fit by different model. Note that in some FITS readers, a trailing space may be appended for "PSF ", "DEV " and "EXP " since the column data type is a 4-character string',<br />
ra double COMMENT 'Right ascension at equinox J2000',<br />
dec double COMMENT 'Declination at equinox J2000',<br />
ra_ivar float COMMENT 'Inverse variance of RA (no cosine term!), excluding astrometric calibration errors',<br />
dec_ivar float COMMENT 'Inverse variance of DEC, excluding astrometric calibration errors',<br />
bx float COMMENT 'X position (0-indexed) of coordinates in brick image stack',<br />
`by` float COMMENT 'Y position (0-indexed) of coordinates in brick image stack',<br />
dchisq array<float> COMMENT 'Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.',<br />
ebv float COMMENT 'Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns',<br />
mjd_min double COMMENT 'Minimum Modified Julian Date of observations used to construct the model of this object',<br />
mjd_max double COMMENT 'Maximum Modified Julian Date of observations used to construct the model of this object',<br />
ref_cat string COMMENT 'Reference catalog source for this star: "T2" for Tycho-2, "G2" for Gaia DR2, "L2" for the LSLGA, empty otherwise',<br />
ref_id bigint COMMENT 'Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b "sourceid" for Gaia-DR2 and LSLGA',<br />
pmra float COMMENT 'Reference catalog proper motion in the RA direction',<br />
pmdec float COMMENT 'Reference catalog proper motion in the Dec direction',<br />
parallax float COMMENT 'Reference catalog parallax',<br />
pmra_ivar float COMMENT 'Reference catalog inverse-variance on pmra',<br />
pmdec_ivar float COMMENT 'Reference catalog inverse-variance on pmdec',<br />
parallax_ivar float COMMENT 'Reference catalog inverse-variance on parallax',<br />
ref_epoch float COMMENT 'Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)',<br />
gaia_pointsource boolean COMMENT 'This Gaia DR2 source is believed to be a star, not a galaxy',<br />
gaia_phot_g_mean_mag float COMMENT 'Gaia G band mag',<br />
gaia_phot_g_mean_flux_over_error float COMMENT 'Gaia G band signal-to-noise',<br />
gaia_phot_g_n_obs smallint COMMENT 'Gaia G band number of observations',<br />
gaia_phot_bp_mean_mag float COMMENT 'Gaia BP mag',<br />
gaia_phot_bp_mean_flux_over_error float COMMENT 'Gaia BP signal-to-noise',<br />
gaia_phot_bp_n_obs smallint COMMENT 'Gaia BP number of observations',<br />
gaia_phot_rp_mean_mag float COMMENT 'Gaia RP mag',<br />
gaia_phot_rp_mean_flux_over_error float COMMENT 'Gaia RP signal-to-noise',<br />
gaia_phot_rp_n_obs smallint COMMENT 'Gaia RP number of observations',<br />
gaia_phot_variable_flag boolean COMMENT 'Gaia photometric variable flag',<br />
gaia_astrometric_excess_noise float COMMENT 'Gaia astrometric excess noise',<br />
gaia_astrometric_excess_noise_sig float COMMENT 'Gaia astrometric excess noise uncertainty',<br />
gaia_astrometric_n_obs_al smallint COMMENT 'Gaia number of astrometric observations along scan direction',<br />
gaia_astrometric_n_good_obs_al smallint COMMENT 'Gaia number of good astrometric observations along scan direction',<br />
gaia_astrometric_weight_al float COMMENT 'Gaia astrometric weight along scan direction',<br />
gaia_duplicated_source boolean COMMENT 'Gaia duplicated source flag',<br />
gaia_a_g_val float COMMENT 'Gaia line-of-sight extinction in the G band',<br />
gaia_e_bp_min_rp_val float COMMENT 'Gaia line-of-sight reddening E(BP-RP)',<br />
gaia_phot_bp_rp_excess_factor float COMMENT 'Gaia BP/RP excess factor',<br />
gaia_astrometric_sigma5d_max float COMMENT 'Gaia longest semi-major axis of the 5-d error ellipsoid',<br />
gaia_astrometric_params_solved tinyint COMMENT 'which astrometric parameters were estimated for a Gaia source',<br />
flux_g float COMMENT 'model flux in gg',<br />
flux_r float COMMENT 'model flux in rr',<br />
flux_z float COMMENT 'model flux in zz',<br />
flux_w1 float COMMENT 'WISE model flux in W1W1 (AB system)',<br />
flux_w2 float COMMENT 'WISE model flux in W2W2 (AB)',<br />
flux_w3 float COMMENT 'WISE model flux in W3W3 (AB)',<br />
flux_w4 float COMMENT 'WISE model flux in W4W4 (AB)',<br />
flux_ivar_g float COMMENT 'Inverse variance of flux_g',<br />
flux_ivar_r float COMMENT 'Inverse variance of flux_r',<br />
flux_ivar_z float COMMENT 'Inverse variance of flux_z',<br />
flux_ivar_w1 float COMMENT 'Inverse variance of flux_w1 (AB system)',<br />
flux_ivar_w2 float COMMENT 'Inverse variance of flux_w2 (AB)',<br />
flux_ivar_w3 float COMMENT 'Inverse variance of flux_w3 (AB)',<br />
flux_ivar_w4 float COMMENT 'Inverse variance of flux_w4 (AB)',<br />
fiberflux_g float COMMENT 'Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_r float COMMENT 'Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fiberflux_z float COMMENT 'Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing',<br />
fibertotflux_g float COMMENT 'Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_r float COMMENT 'Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
fibertotflux_z float COMMENT 'Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing',<br />
apflux_g array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg',<br />
apflux_r array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr',<br />
apflux_z array<float> COMMENT 'aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz',<br />
apflux_resid_g array<float> COMMENT 'aperture fluxes on the co-added residual images in gg',<br />
apflux_resid_r array<float> COMMENT 'aperture fluxes on the co-added residual images in rr',<br />
apflux_resid_z array<float> COMMENT 'aperture fluxes on the co-added residual images in zz',<br />
apflux_ivar_g array<float> COMMENT 'Inverse variance of apflux_resid_g',<br />
apflux_ivar_r array<float> COMMENT 'Inverse variance of apflux_resid_r',<br />
apflux_ivar_z array<float> COMMENT 'Inverse variance of apflux_resid_z',<br />
mw_transmission_g float COMMENT 'Galactic transmission in gg filter in linear units [0, 1]',<br />
mw_transmission_r float COMMENT 'Galactic transmission in rr filter in linear units [0, 1]',<br />
mw_transmission_z float COMMENT 'Galactic transmission in zz filter in linear units [0, 1]',<br />
mw_transmission_w1 float COMMENT 'Galactic transmission in W1W1 filter in linear units [0, 1]',<br />
mw_transmission_w2 float COMMENT 'Galactic transmission in W2W2 filter in linear units [0, 1]',<br />
mw_transmission_w3 float COMMENT 'Galactic transmission in W3W3 filter in linear units [0, 1]',<br />
mw_transmission_w4 float COMMENT 'Galactic transmission in W4W4 filter in linear units [0, 1]',<br />
nobs_g smallint COMMENT 'Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)',<br />
nobs_r smallint COMMENT 'Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)',<br />
nobs_z smallint COMMENT 'Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)',<br />
nobs_w1 smallint COMMENT 'Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)',<br />
nobs_w2 smallint COMMENT 'Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)',<br />
nobs_w3 smallint COMMENT 'Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)',<br />
nobs_w4 smallint COMMENT 'Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)',<br />
rchisq_g float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in gg',<br />
rchisq_r float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in rr',<br />
rchisq_z float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in zz',<br />
rchisq_w1 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W1W1',<br />
rchisq_w2 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W2W2',<br />
rchisq_w3 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W3W3',<br />
rchisq_w4 float COMMENT 'Profile-weighted χ² of model fit normalized by the number of pixels in W4W4',<br />
fracflux_g float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in gg (typically [0,1])',<br />
fracflux_r float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in rr (typically [0,1])',<br />
fracflux_z float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in zz (typically [0,1])',<br />
fracflux_w1 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1 (typically [0,1])',<br />
fracflux_w2 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2 (typically [0,1])',<br />
fracflux_w3 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3 (typically [0,1])',<br />
fracflux_w4 float COMMENT 'Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4 (typically [0,1])',<br />
fracmasked_g float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]',<br />
fracmasked_r float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]',<br />
fracmasked_z float COMMENT 'Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]',<br />
fracin_g float COMMENT 'Fraction of a source\'s flux within the blob in gg, near unity for real sources',<br />
fracin_r float COMMENT 'Fraction of a source\'s flux within the blob in rr, near unity for real sources',<br />
fracin_z float COMMENT 'Fraction of a source\'s flux within the blob in zz, near unity for real sources',<br />
anymask_g smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page',<br />
anymask_r smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page',<br />
anymask_z smallint COMMENT 'Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page',<br />
allmask_g smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page',<br />
allmask_r smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page',<br />
allmask_z smallint COMMENT 'Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page',<br />
wisemask_w1 tinyint COMMENT 'W1 bitmask as cataloged on the DR8 bitmasks page',<br />
wisemask_w2 tinyint COMMENT 'W2 bitmask as cataloged on the DR8 bitmasks page',<br />
psfsize_g float COMMENT 'Weighted average PSF FWHM in the gg band',<br />
psfsize_r float COMMENT 'Weighted average PSF FWHM in the rr band',<br />
psfsize_z float COMMENT 'Weighted average PSF FWHM in the zz band',<br />
psfdepth_g float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude',<br />
psfdepth_r float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude',<br />
psfdepth_z float COMMENT 'For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude',<br />
galdepth_g float COMMENT 'As for psfdepth_g but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_r float COMMENT 'As for psfdepth_r but for a galaxy (0.45" exp, round) detection sensitivity',<br />
galdepth_z float COMMENT 'As for psfdepth_z but for a galaxy (0.45" exp, round) detection sensitivity',<br />
psfdepth_w1 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W1',<br />
psfdepth_w2 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W2',<br />
psfdepth_w3 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W3',<br />
psfdepth_w4 float COMMENT 'As for psfdepth_g (and also on the AB system) but for WISE W4',<br />
wise_coadd_id string COMMENT 'unWISE coadd file name for the center of each object',<br />
lc_flux_w1 array<float> COMMENT 'flux_w1 in each of up to eleven unWISE coadd epochs (AB system)',<br />
lc_flux_w2 array<float> COMMENT 'flux_w2 in each of up to eleven unWISE coadd epochs (AB)',<br />
lc_flux_ivar_w1 array<float> COMMENT 'Inverse variance of lc_flux_w1 (AB system)',<br />
lc_flux_ivar_w2 array<float> COMMENT 'Inverse variance of lc_flux_w2 (AB)',<br />
lc_nobs_w1 array<smallint> COMMENT 'nobs_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_nobs_w2 array<smallint> COMMENT 'nobs_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w1 array<float> COMMENT 'fracflux_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_fracflux_w2 array<float> COMMENT 'fracflux_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w1 array<float> COMMENT 'rchisq_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_rchisq_w2 array<float> COMMENT 'rchisq_w2 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w1 array<double> COMMENT 'mjd_w1 in each of up to eleven unWISE coadd epochs',<br />
lc_mjd_w2 array<double> COMMENT 'mjd_w2 in each of up to eleven unWISE coadd epochs',<br />
fracdev float COMMENT 'Fraction of model in deVauc [0,1]',<br />
fracdev_ivar float COMMENT 'Inverse variance of fracdev',<br />
shapeexp_r float COMMENT 'Half-light radius of exponential model (>0)',<br />
shapeexp_r_ivar float COMMENT 'Inverse variance of shapeexp_r',<br />
shapeexp_e1 float COMMENT 'Ellipticity component 1',<br />
shapeexp_e1_ivar float COMMENT 'Inverse variance of shapeexp_e1',<br />
shapeexp_e2 float COMMENT 'Ellipticity component 2',<br />
shapeexp_e2_ivar float COMMENT 'Inverse variance of shapeexp_e2',<br />
shapedev_r float COMMENT 'Half-light radius of deVaucouleurs model (>0)',<br />
shapedev_r_ivar float COMMENT 'Inverse variance of shapedev_r',<br />
shapedev_e1 float COMMENT 'Ellipticity component 1',<br />
shapedev_e1_ivar float COMMENT 'Inverse variance of shapedev_e1',<br />
shapedev_e2 float COMMENT 'Ellipticity component 2',<br />
shapedev_e2_ivar float COMMENT 'Inverse variance of shapedev_e2',<br />
z_phot_mean FLOAT COMMENT 'photo-z derived from the mean of the photo-z PDF ',<br />
z_phot_median FLOAT COMMENT 'photo-z derived from the median of the photo-z PDF',<br />
z_phot_std FLOAT COMMENT 'standard deviation of the photo-z\'s derived from the photo-z PDF',<br />
z_phot_l68 FLOAT COMMENT 'lower bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_u68 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_phot_l95 FLOAT COMMENT 'lower bound of the 95% confidence region, derived from the photo-z PDF',<br />
z_phot_u95 FLOAT COMMENT 'upper bound of the 68% confidence region, derived from the photo-z PDF',<br />
z_spec FLOAT COMMENT 'spectroscopic redshift, if available',<br />
survey STRING COMMENT 'source of the spectroscopic redshift',<br />
training BOOLEAN COMMENT 'whether or not the spectroscopic redshift is used in photometric redshift training'<br />
)<br />
CLUSTERED BY (<br />
release,<br />
brickid,<br />
objid)<br />
SORTED BY (<br />
release ASC,<br />
brickid ASC,<br />
objid ASC)<br />
INTO 4096 BUCKETS<br />
STORED AS ORC<br />
;<br />
<br />
=== Join insert ===<br />
<br />
INSERT OVERWRITE TABLE cosmohub.legacysurvey_dr8_phz<br />
SELECT<br />
dr8.*,<br />
phz.z_phot_mean,<br />
phz.z_phot_median,<br />
phz.z_phot_std,<br />
phz.z_phot_l68,<br />
phz.z_phot_u68,<br />
phz.z_phot_l95,<br />
phz.z_phot_u95,<br />
phz.z_spec,<br />
phz.survey,<br />
phz.training<br />
FROM cosmohub.legacy_survey_dr8 AS dr8<br />
LEFT JOIN tallada.dr8_sweep AS phz<br />
ON dr8.release = phz.release<br />
AND dr8.brickid = phz.brickid<br />
AND dr8.objid = phz.objid<br />
;</div>
Tallada
https://pwiki.pic.es/index.php?title=Legacy_Survey_DR8&diff=799
Legacy Survey DR8
2020-05-07T08:51:04Z
<p>Tallada: </p>
<hr />
<div>== Original DR8 ==<br />
=== Raw data ===<br />
<br />
Downloaded from<br />
ftp://archive.noao.edu:/public/hlsp/ls/dr8/<br />
<br />
Directories:<br />
south/tractor<br />
north/tractor<br />
<br />
Mirrored using lftp commands:<br />
mirror -c -v -P 200 north/tractor/ /mnt/pau/north/tractor/<br />
mirror -c -v -P 200 south/tractor/ /mnt/pau/south/tractor/<br />
<br />
into a 5 TiB temporary Ceph filesystem mounted in tdm002.<br />
<br />
Checksums provided through sha256sum files, and checked using command:<br />
find * -type d | xargs -t -L1 -P24 sh -c 'cd $1; sha256sum --quiet -c *.sha256sum' cksum<br />
<br />
=== Parquet ===<br />
<br />
The following code snippet was used in spark to convert fits files to parquet:<br />
<br />
path_base = '/cephfs/pic.es/cosmohub/legacysurvey'<br />
hdfs_base = '/user/tallada/data/legacysurvey'<br />
<br />
hs = ['h=north', 'h=south']<br />
rs = ['r={0:03d}'.format(x) for x in range(360)]<br />
<br />
for h in hs:<br />
for r in rs:<br />
p = os.path.join(path_base, h, r)<br />
if not os.path.exists(p):<br />
continue<br />
<br />
size = sum(os.path.getsize(os.path.join(p,f)) for f in os.listdir(p) if os.path.isfile(os.path.join(p,f)))<br />
<br />
print h, r, size <br />
<br />
df = spark.read.format("fits").option("hdu", 1).load(<br />
"file://" + p<br />
)<br />
<br />
df = df.repartition(int(math.ceil(size / (256*1024*1024.0))))<br />
<br />
df.write.parquet(os.path.join(hdfs_base, h, r))<br />
<br />
=== External table ===<br />
<br />
CREATE EXTERNAL TABLE tallada.dr8 (<br />
`release` smallint,<br />
`brickid` int,<br />
`brickname` string,<br />
`objid` int,<br />
`brick_primary` boolean,<br />
`brightblob` smallint,<br />
`maskbits` smallint,<br />
`type` string,<br />
`ra` double,<br />
`dec` double,<br />
`ra_ivar` float,<br />
`dec_ivar` float,<br />
`bx` float,<br />
`by` float,<br />
`dchisq` array<float>,<br />
`ebv` float,<br />
`mjd_min` double,<br />
`mjd_max` double,<br />
`ref_cat` string,<br />
`ref_id` bigint,<br />
`pmra` float,<br />
`pmdec` float,<br />
`parallax` float,<br />
`pmra_ivar` float,<br />
`pmdec_ivar` float,<br />
`parallax_ivar` float,<br />
`ref_epoch` float,<br />
`gaia_pointsource` boolean,<br />
`gaia_phot_g_mean_mag` float,<br />
`gaia_phot_g_mean_flux_over_error` float,<br />
`gaia_phot_g_n_obs` smallint,<br />
`gaia_phot_bp_mean_mag` float,<br />
`gaia_phot_bp_mean_flux_over_error` float,<br />
`gaia_phot_bp_n_obs` smallint,<br />
`gaia_phot_rp_mean_mag` float,<br />
`gaia_phot_rp_mean_flux_over_error` float,<br />
`gaia_phot_rp_n_obs` smallint,<br />
`gaia_phot_variable_flag` boolean,<br />
`gaia_astrometric_excess_noise` float,<br />
`gaia_astrometric_excess_noise_sig` float,<br />
`gaia_astrometric_n_obs_al` smallint,<br />
`gaia_astrometric_n_good_obs_al` smallint,<br />
`gaia_astrometric_weight_al` float,<br />
`gaia_duplicated_source` boolean,<br />
`gaia_a_g_val` float,<br />
`gaia_e_bp_min_rp_val` float,<br />
`gaia_phot_bp_rp_excess_factor` float,<br />
`gaia_astrometric_sigma5d_max` float,<br />
`gaia_astrometric_params_solved` tinyint,<br />
`flux_g` float,<br />
`flux_r` float,<br />
`flux_z` float,<br />
`flux_w1` float,<br />
`flux_w2` float,<br />
`flux_w3` float,<br />
`flux_w4` float,<br />
`flux_ivar_g` float,<br />
`flux_ivar_r` float,<br />
`flux_ivar_z` float,<br />
`flux_ivar_w1` float,<br />
`flux_ivar_w2` float,<br />
`flux_ivar_w3` float,<br />
`flux_ivar_w4` float,<br />
`fiberflux_g` float,<br />
`fiberflux_r` float,<br />
`fiberflux_z` float,<br />
`fibertotflux_g` float,<br />
`fibertotflux_r` float,<br />
`fibertotflux_z` float,<br />
`apflux_g` array<float>,<br />
`apflux_r` array<float>,<br />
`apflux_z` array<float>,<br />
`apflux_resid_g` array<float>,<br />
`apflux_resid_r` array<float>,<br />
`apflux_resid_z` array<float>,<br />
`apflux_ivar_g` array<float>,<br />
`apflux_ivar_r` array<float>,<br />
`apflux_ivar_z` array<float>,<br />
`mw_transmission_g` float,<br />
`mw_transmission_r` float,<br />
`mw_transmission_z` float,<br />
`mw_transmission_w1` float,<br />
`mw_transmission_w2` float,<br />
`mw_transmission_w3` float,<br />
`mw_transmission_w4` float,<br />
`nobs_g` smallint,<br />
`nobs_r` smallint,<br />
`nobs_z` smallint,<br />
`nobs_w1` smallint,<br />
`nobs_w2` smallint,<br />
`nobs_w3` smallint,<br />
`nobs_w4` smallint,<br />
`rchisq_g` float,<br />
`rchisq_r` float,<br />
`rchisq_z` float,<br />
`rchisq_w1` float,<br />
`rchisq_w2` float,<br />
`rchisq_w3` float,<br />
`rchisq_w4` float,<br />
`fracflux_g` float,<br />
`fracflux_r` float,<br />
`fracflux_z` float,<br />
`fracflux_w1` float,<br />
`fracflux_w2` float,<br />
`fracflux_w3` float,<br />
`fracflux_w4` float,<br />
`fracmasked_g` float,<br />
`fracmasked_r` float,<br />
`fracmasked_z` float,<br />
`fracin_g` float,<br />
`fracin_r` float,<br />
`fracin_z` float,<br />
`anymask_g` smallint,<br />
`anymask_r` smallint,<br />
`anymask_z` smallint,<br />
`allmask_g` smallint,<br />
`allmask_r` smallint,<br />
`allmask_z` smallint,<br />
`wisemask_w1` tinyint,<br />
`wisemask_w2` tinyint,<br />
`psfsize_g` float,<br />
`psfsize_r` float,<br />
`psfsize_z` float,<br />
`psfdepth_g` float,<br />
`psfdepth_r` float,<br />
`psfdepth_z` float,<br />
`galdepth_g` float,<br />
`galdepth_r` float,<br />
`galdepth_z` float,<br />
`psfdepth_w1` float,<br />
`psfdepth_w2` float,<br />
`psfdepth_w3` float,<br />
`psfdepth_w4` float,<br />
`wise_coadd_id` string,<br />
`lc_flux_w1` array<float>,<br />
`lc_flux_w2` array<float>,<br />
`lc_flux_ivar_w1` array<float>,<br />
`lc_flux_ivar_w2` array<float>,<br />
`lc_nobs_w1` array<smallint>,<br />
`lc_nobs_w2` array<smallint>,<br />
`lc_fracflux_w1` array<float>,<br />
`lc_fracflux_w2` array<float>,<br />
`lc_rchisq_w1` array<float>,<br />
`lc_rchisq_w2` array<float>,<br />
`lc_mjd_w1` array<double>,<br />
`lc_mjd_w2` array<double>,<br />
`fracdev` float,<br />
`fracdev_ivar` float,<br />
`shapeexp_r` float,<br />
`shapeexp_r_ivar` float,<br />
`shapeexp_e1` float,<br />
`shapeexp_e1_ivar` float,<br />
`shapeexp_e2` float,<br />
`shapeexp_e2_ivar` float,<br />
`shapedev_r` float,<br />
`shapedev_r_ivar` float,<br />
`shapedev_e1` float,<br />
`shapedev_e1_ivar` float,<br />
`shapedev_e2` float,<br />
`shapedev_e2_ivar` float<br />
)<br />
PARTITIONED BY (<br />
h string,<br />
r string<br />
)<br />
STORED AS PARQUET<br />
LOCATION '/user/tallada/data/legacysurvey/dr8/'<br />
;<br />
<br />
=== Internal Table ===<br />
<br />
CREATE TABLE cosmohub.legacy_survey_dr8 (<br />
`release` smallint,<br />
`brickid` int,<br />
`brickname` string,<br />
`objid` int,<br />
`brick_primary` boolean,<br />
`brightblob` smallint,<br />
`maskbits` smallint,<br />
`type` string,<br />
`ra` double,<br />
`dec` double,<br />
`ra_ivar` float,<br />
`dec_ivar` float,<br />
`bx` float,<br />
`by` float,<br />
`dchisq` array<float>,<br />
`ebv` float,<br />
`mjd_min` double,<br />
`mjd_max` double,<br />
`ref_cat` string,<br />
`ref_id` bigint,<br />
`pmra` float,<br />
`pmdec` float,<br />
`parallax` float,<br />
`pmra_ivar` float,<br />
`pmdec_ivar` float,<br />
`parallax_ivar` float,<br />
`ref_epoch` float,<br />
`gaia_pointsource` boolean,<br />
`gaia_phot_g_mean_mag` float,<br />
`gaia_phot_g_mean_flux_over_error` float,<br />
`gaia_phot_g_n_obs` smallint,<br />
`gaia_phot_bp_mean_mag` float,<br />
`gaia_phot_bp_mean_flux_over_error` float,<br />
`gaia_phot_bp_n_obs` smallint,<br />
`gaia_phot_rp_mean_mag` float,<br />
`gaia_phot_rp_mean_flux_over_error` float,<br />
`gaia_phot_rp_n_obs` smallint,<br />
`gaia_phot_variable_flag` boolean,<br />
`gaia_astrometric_excess_noise` float,<br />
`gaia_astrometric_excess_noise_sig` float,<br />
`gaia_astrometric_n_obs_al` smallint,<br />
`gaia_astrometric_n_good_obs_al` smallint,<br />
`gaia_astrometric_weight_al` float,<br />
`gaia_duplicated_source` boolean,<br />
`gaia_a_g_val` float,<br />
`gaia_e_bp_min_rp_val` float,<br />
`gaia_phot_bp_rp_excess_factor` float,<br />
`gaia_astrometric_sigma5d_max` float,<br />
`gaia_astrometric_params_solved` tinyint,<br />
`flux_g` float,<br />
`flux_r` float,<br />
`flux_z` float,<br />
`flux_w1` float,<br />
`flux_w2` float,<br />
`flux_w3` float,<br />
`flux_w4` float,<br />
`flux_ivar_g` float,<br />
`flux_ivar_r` float,<br />
`flux_ivar_z` float,<br />
`flux_ivar_w1` float,<br />
`flux_ivar_w2` float,<br />
`flux_ivar_w3` float,<br />
`flux_ivar_w4` float,<br />
`fiberflux_g` float,<br />
`fiberflux_r` float,<br />
`fiberflux_z` float,<br />
`fibertotflux_g` float,<br />
`fibertotflux_r` float,<br />
`fibertotflux_z` float,<br />
`apflux_g` array<float>,<br />
`apflux_r` array<float>,<br />
`apflux_z` array<float>,<br />
`apflux_resid_g` array<float>,<br />
`apflux_resid_r` array<float>,<br />
`apflux_resid_z` array<float>,<br />
`apflux_ivar_g` array<float>,<br />
`apflux_ivar_r` array<float>,<br />
`apflux_ivar_z` array<float>,<br />
`mw_transmission_g` float,<br />
`mw_transmission_r` float,<br />
`mw_transmission_z` float,<br />
`mw_transmission_w1` float,<br />
`mw_transmission_w2` float,<br />
`mw_transmission_w3` float,<br />
`mw_transmission_w4` float,<br />
`nobs_g` smallint,<br />
`nobs_r` smallint,<br />
`nobs_z` smallint,<br />
`nobs_w1` smallint,<br />
`nobs_w2` smallint,<br />
`nobs_w3` smallint,<br />
`nobs_w4` smallint,<br />
`rchisq_g` float,<br />
`rchisq_r` float,<br />
`rchisq_z` float,<br />
`rchisq_w1` float,<br />
`rchisq_w2` float,<br />
`rchisq_w3` float,<br />
`rchisq_w4` float,<br />
`fracflux_g` float,<br />
`fracflux_r` float,<br />
`fracflux_z` float,<br />
`fracflux_w1` float,<br />
`fracflux_w2` float,<br />
`fracflux_w3` float,<br />
`fracflux_w4` float,<br />
`fracmasked_g` float,<br />
`fracmasked_r` float,<br />
`fracmasked_z` float,<br />
`fracin_g` float,<br />
`fracin_r` float,<br />
`fracin_z` float,<br />
`anymask_g` smallint,<br />
`anymask_r` smallint,<br />
`anymask_z` smallint,<br />
`allmask_g` smallint,<br />
`allmask_r` smallint,<br />
`allmask_z` smallint,<br />
`wisemask_w1` tinyint,<br />
`wisemask_w2` tinyint,<br />
`psfsize_g` float,<br />
`psfsize_r` float,<br />
`psfsize_z` float,<br />
`psfdepth_g` float,<br />
`psfdepth_r` float,<br />
`psfdepth_z` float,<br />
`galdepth_g` float,<br />
`galdepth_r` float,<br />
`galdepth_z` float,<br />
`psfdepth_w1` float,<br />
`psfdepth_w2` float,<br />
`psfdepth_w3` float,<br />
`psfdepth_w4` float,<br />
`wise_coadd_id` string,<br />
`lc_flux_w1` array<float>,<br />
`lc_flux_w2` array<float>,<br />
`lc_flux_ivar_w1` array<float>,<br />
`lc_flux_ivar_w2` array<float>,<br />
`lc_nobs_w1` array<smallint>,<br />
`lc_nobs_w2` array<smallint>,<br />
`lc_fracflux_w1` array<float>,<br />
`lc_fracflux_w2` array<float>,<br />
`lc_rchisq_w1` array<float>,<br />
`lc_rchisq_w2` array<float>,<br />
`lc_mjd_w1` array<double>,<br />
`lc_mjd_w2` array<double>,<br />
`fracdev` float,<br />
`fracdev_ivar` float,<br />
`shapeexp_r` float,<br />
`shapeexp_r_ivar` float,<br />
`shapeexp_e1` float,<br />
`shapeexp_e1_ivar` float,<br />
`shapeexp_e2` float,<br />
`shapeexp_e2_ivar` float,<br />
`shapedev_r` float,<br />
`shapedev_r_ivar` float,<br />
`shapedev_e1` float,<br />
`shapedev_e1_ivar` float,<br />
`shapedev_e2` float,<br />
`shapedev_e2_ivar` float<br />
)<br />
clustered by (<br />
release, brickid, objid<br />
)<br />
sorted by (<br />
release, brickid, objid<br />
)<br />
into 4096 buckets<br />
stored as orc<br />
;<br />
<br />
insert overwrite table cosmohub.legacy_survey_dr8<br />
select release, brickid, brickname, objid, brick_primary, brightblob, maskbits, type, ra, dec, ra_ivar, dec_ivar, bx, `by`, <br />
dchisq, ebv, mjd_min, mjd_max, ref_cat, ref_id, pmra, pmdec, parallax, pmra_ivar, pmdec_ivar, parallax_ivar, ref_epoch,<br />
gaia_pointsource, gaia_phot_g_mean_mag, gaia_phot_g_mean_flux_over_error, gaia_phot_g_n_obs,<br />
gaia_phot_bp_mean_mag, gaia_phot_bp_mean_flux_over_error, gaia_phot_bp_n_obs, gaia_phot_rp_mean_mag,<br />
gaia_phot_rp_mean_flux_over_error, gaia_phot_rp_n_obs, gaia_phot_variable_flag, gaia_astrometric_excess_noise,<br />
gaia_astrometric_excess_noise_sig, gaia_astrometric_n_obs_al, gaia_astrometric_n_good_obs_al,<br />
gaia_astrometric_weight_al, gaia_duplicated_source, gaia_a_g_val, gaia_e_bp_min_rp_val,<br />
gaia_phot_bp_rp_excess_factor, gaia_astrometric_sigma5d_max, gaia_astrometric_params_solved,<br />
flux_g, flux_r, flux_z, flux_w1, flux_w2, flux_w3, flux_w4, flux_ivar_g, flux_ivar_r,<br />
flux_ivar_z, flux_ivar_w1, flux_ivar_w2, flux_ivar_w3, flux_ivar_w4, fiberflux_g, fiberflux_r,<br />
fiberflux_z, fibertotflux_g, fibertotflux_r, fibertotflux_z, apflux_g, apflux_r, apflux_z,<br />
apflux_resid_g, apflux_resid_r, apflux_resid_z, apflux_ivar_g, apflux_ivar_r, apflux_ivar_z,<br />
mw_transmission_g, mw_transmission_r, mw_transmission_z, mw_transmission_w1, mw_transmission_w2,<br />
mw_transmission_w3, mw_transmission_w4, nobs_g, nobs_r, nobs_z, nobs_w1, nobs_w2, nobs_w3,<br />
nobs_w4, rchisq_g, rchisq_r, rchisq_z, rchisq_w1, rchisq_w2, rchisq_w3, rchisq_w4, fracflux_g,<br />
fracflux_r, fracflux_z, fracflux_w1, fracflux_w2, fracflux_w3, fracflux_w4, fracmasked_g,<br />
fracmasked_r, fracmasked_z, fracin_g, fracin_r, fracin_z, anymask_g, anymask_r, anymask_z,<br />
allmask_g, allmask_r, allmask_z, wisemask_w1, wisemask_w2, psfsize_g, psfsize_r, psfsize_z,<br />
psfdepth_g, psfdepth_r, psfdepth_z, galdepth_g, galdepth_r, galdepth_z, psfdepth_w1, psfdepth_w2,<br />
psfdepth_w3, psfdepth_w4, wise_coadd_id, lc_flux_w1, lc_flux_w2, lc_flux_ivar_w1, lc_flux_ivar_w2,<br />
lc_nobs_w1, lc_nobs_w2, lc_fracflux_w1, lc_fracflux_w2, lc_rchisq_w1, lc_rchisq_w2, lc_mjd_w1,<br />
lc_mjd_w2, fracdev, fracdev_ivar, shapeexp_r, shapeexp_r_ivar, shapeexp_e1, shapeexp_e1_ivar,<br />
shapeexp_e2, shapeexp_e2_ivar, shapedev_r, shapedev_r_ivar, shapedev_e1, shapedev_e1_ivar,<br />
shapedev_e2, shapedev_e2_ivar<br />
from tallada.dr8<br />
;<br />
<br />
=== Comments ===<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE release release smallint COMMENT "Unique integer denoting the camera and filter set used";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickid brickid int COMMENT "Brick ID [1,662174]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brickname brickname string COMMENT "Name of brick, encoding the brick sky position, eg \"1126p222\" near RA=112.6, Dec=+22.2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE objid objid int COMMENT "Catalog object number within this brick\u003b a unique identifier hash is release,brickid,objid\u003b objid spans [0,N-1] and is contiguously enumerated within each brick";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brick_primary brick_primary boolean COMMENT "True if the object is within the brick boundary";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE brightblob brightblob smallint COMMENT "bitwise mask indicating that an object is near a bright foreground source, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE maskbits maskbits smallint COMMENT "bitwise mask indicating that an object touches a pixel in the coadd/*/*/*maskbits*maps, as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE type type string COMMENT "Morphological model: PSF=stellar, REX=round exponential galaxy, DEV=deVauc, EXP=exponential, COMP=composite, DUP=Gaia source fit by different model.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra ra double COMMENT "Right ascension at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec dec double COMMENT "Declination at equinox J2000";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ra_ivar ra_ivar float COMMENT "Inverse variance of RA (no cosine term!), excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dec_ivar dec_ivar float COMMENT "Inverse variance of DEC, excluding astrometric calibration errors";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE bx bx float COMMENT "X position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE by by float COMMENT "Y position (0-indexed) of coordinates in brick image stack";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE dchisq dchisq array<float> COMMENT "Difference in χ² between successively more-complex model fits: PSF, REX, DEV, EXP, COMP. The difference is versus no source.";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ebv ebv float COMMENT "Galactic extinction E(B-V) reddening from SFD98, used to compute the mw_transmission_ columns";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_min mjd_min double COMMENT "Minimum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mjd_max mjd_max double COMMENT "Maximum Modified Julian Date of observations used to construct the model of this object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_cat ref_cat string COMMENT "Reference catalog source for this star: T2 for Tycho-2, G2 for Gaia DR2, L2 for the LSLGA, empty otherwise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_id ref_id bigint COMMENT "Reference catalog identifier for this star\u003b Tyc1*1,000,000+Tyc2*10+Tyc3 for Tycho2\u003b \"sourceid\" for Gaia-DR2 and LSLGA";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra pmra float COMMENT "Reference catalog proper motion in the RA direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec pmdec float COMMENT "Reference catalog proper motion in the Dec direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax parallax float COMMENT "Reference catalog parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmra_ivar pmra_ivar float COMMENT "Reference catalog inverse-variance on pmra";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE pmdec_ivar pmdec_ivar float COMMENT "Reference catalog inverse-variance on pmdec";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE parallax_ivar parallax_ivar float COMMENT "Reference catalog inverse-variance on parallax";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE ref_epoch ref_epoch float COMMENT "Reference catalog reference epoch (eg, 2015.5 for Gaia DR2)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_pointsource gaia_pointsource boolean COMMENT "This Gaia DR2 source is believed to be a star, not a galaxy";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_mag gaia_phot_g_mean_mag float COMMENT "Gaia G band mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_mean_flux_over_error gaia_phot_g_mean_flux_over_error float COMMENT "Gaia G band signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_g_n_obs gaia_phot_g_n_obs smallint COMMENT "Gaia G band number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_mag gaia_phot_bp_mean_mag float COMMENT "Gaia BP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_mean_flux_over_error gaia_phot_bp_mean_flux_over_error float COMMENT "Gaia BP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_n_obs gaia_phot_bp_n_obs smallint COMMENT "Gaia BP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_mag gaia_phot_rp_mean_mag float COMMENT "Gaia RP mag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_mean_flux_over_error gaia_phot_rp_mean_flux_over_error float COMMENT "Gaia RP signal-to-noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_rp_n_obs gaia_phot_rp_n_obs smallint COMMENT "Gaia RP number of observations";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_variable_flag gaia_phot_variable_flag boolean COMMENT "Gaia photometric variable flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise gaia_astrometric_excess_noise float COMMENT "Gaia astrometric excess noise";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_excess_noise_sig gaia_astrometric_excess_noise_sig float COMMENT "Gaia astrometric excess noise uncertainty";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_obs_al gaia_astrometric_n_obs_al smallint COMMENT "Gaia number of astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_n_good_obs_al gaia_astrometric_n_good_obs_al smallint COMMENT "Gaia number of good astrometric observations along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_weight_al gaia_astrometric_weight_al float COMMENT "Gaia astrometric weight along scan direction";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_duplicated_source gaia_duplicated_source boolean COMMENT "Gaia duplicated source flag";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_a_g_val gaia_a_g_val float COMMENT "Gaia line-of-sight extinction in the G band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_e_bp_min_rp_val gaia_e_bp_min_rp_val float COMMENT "Gaia line-of-sight reddening E(BP-RP)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_phot_bp_rp_excess_factor gaia_phot_bp_rp_excess_factor float COMMENT "Gaia BP/RP excess factor";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_sigma5d_max gaia_astrometric_sigma5d_max float COMMENT "Gaia longest semi-major axis of the 5-d error ellipsoid";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE gaia_astrometric_params_solved gaia_astrometric_params_solved tinyint COMMENT "which astrometric parameters were estimated for a Gaia source";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_g flux_g float COMMENT "model flux in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_r flux_r float COMMENT "model flux in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_z flux_z float COMMENT "model flux in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w1 flux_w1 float COMMENT "WISE model flux in W1W1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w2 flux_w2 float COMMENT "WISE model flux in W2W2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w3 flux_w3 float COMMENT "WISE model flux in W3W3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_w4 flux_w4 float COMMENT "WISE model flux in W4W4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_g flux_ivar_g float COMMENT "Inverse variance of flux_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_r flux_ivar_r float COMMENT "Inverse variance of flux_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_z flux_ivar_z float COMMENT "Inverse variance of flux_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w1 flux_ivar_w1 float COMMENT "Inverse variance of flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w2 flux_ivar_w2 float COMMENT "Inverse variance of flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w3 flux_ivar_w3 float COMMENT "Inverse variance of flux_w3 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE flux_ivar_w4 flux_ivar_w4 float COMMENT "Inverse variance of flux_w4 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_g fiberflux_g float COMMENT "Predicted gg-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_r fiberflux_r float COMMENT "Predicted rr-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fiberflux_z fiberflux_z float COMMENT "Predicted zz-band flux within a fiber from this object in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_g fibertotflux_g float COMMENT "Predicted gg-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_r fibertotflux_r float COMMENT "Predicted rr-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fibertotflux_z fibertotflux_z float COMMENT "Predicted zz-band flux within a fiber from all sources at this location in 1 arcsec Gaussian seeing";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_g apflux_g array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_r apflux_r array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_z apflux_z array<float> COMMENT "aperture fluxes on the co-added images in apertures of radius [0.5, 0.75, 1.0, 1.5, 2.0, 3.5, 5.0, 7.0] arcsec in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_g apflux_resid_g array<float> COMMENT "aperture fluxes on the co-added residual images in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_r apflux_resid_r array<float> COMMENT "aperture fluxes on the co-added residual images in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_resid_z apflux_resid_z array<float> COMMENT "aperture fluxes on the co-added residual images in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_g apflux_ivar_g array<float> COMMENT "Inverse variance of apflux_resid_g";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_r apflux_ivar_r array<float> COMMENT "Inverse variance of apflux_resid_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE apflux_ivar_z apflux_ivar_z array<float> COMMENT "Inverse variance of apflux_resid_z";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_g mw_transmission_g float COMMENT "Galactic transmission in gg filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_r mw_transmission_r float COMMENT "Galactic transmission in rr filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_z mw_transmission_z float COMMENT "Galactic transmission in zz filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w1 mw_transmission_w1 float COMMENT "Galactic transmission in W1W1 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w2 mw_transmission_w2 float COMMENT "Galactic transmission in W2W2 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w3 mw_transmission_w3 float COMMENT "Galactic transmission in W3W3 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE mw_transmission_w4 mw_transmission_w4 float COMMENT "Galactic transmission in W4W4 filter in linear units [0, 1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_g nobs_g smallint COMMENT "Number of images that contribute to the central pixel in gg: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_r nobs_r smallint COMMENT "Number of images that contribute to the central pixel in rr: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_z nobs_z smallint COMMENT "Number of images that contribute to the central pixel in zz: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w1 nobs_w1 smallint COMMENT "Number of images that contribute to the central pixel in W1W1: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w2 nobs_w2 smallint COMMENT "Number of images that contribute to the central pixel in W2W2: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w3 nobs_w3 smallint COMMENT "Number of images that contribute to the central pixel in W3W3: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE nobs_w4 nobs_w4 smallint COMMENT "Number of images that contribute to the central pixel in W4W4: filter for this object (not profile-weighted)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_g rchisq_g float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in gg";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_r rchisq_r float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in rr";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_z rchisq_z float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in zz";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w1 rchisq_w1 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W1W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w2 rchisq_w2 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W2W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w3 rchisq_w3 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W3W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE rchisq_w4 rchisq_w4 float COMMENT "Profile-weighted χ² of model fit normalized by the number of pixels in W4W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_g fracflux_g float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in gg(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_r fracflux_r float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in rr(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_z fracflux_z float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in zz(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w1 fracflux_w1 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W1W1(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w2 fracflux_w2 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W2W2(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w3 fracflux_w3 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W3W3(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracflux_w4 fracflux_w4 float COMMENT "Profile-weighted fraction of the flux from other sources divided by the total flux in W4W4(typically [0,1])";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_g fracmasked_g float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in gg, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_r fracmasked_r float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in rr, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracmasked_z fracmasked_z float COMMENT "Profile-weighted fraction of pixels masked from all observations of this object in zz, strictly between [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_g fracin_g float COMMENT "Fraction of a source's flux within the blob in gg, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_r fracin_r float COMMENT "Fraction of a source's flux within the blob in rr, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracin_z fracin_z float COMMENT "Fraction of a source's flux within the blob in zz, near unity for real sources";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_g anymask_g smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_r anymask_r smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE anymask_z anymask_z smallint COMMENT "Bitwise mask set if the central pixel from any image satisfies each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_g allmask_g smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in gg as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_r allmask_r smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in rr as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE allmask_z allmask_z smallint COMMENT "Bitwise mask set if the central pixel from all images satisfy each condition in zz as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w1 wisemask_w1 tinyint COMMENT "W1 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wisemask_w2 wisemask_w2 tinyint COMMENT "W2 bitmask as cataloged on the DR8 bitmasks page";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_g psfsize_g float COMMENT "Weighted average PSF FWHM in the gg band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_r psfsize_r float COMMENT "Weighted average PSF FWHM in the rr band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfsize_z psfsize_z float COMMENT "Weighted average PSF FWHM in the zz band";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_g psfdepth_g float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_g)5/(psfdepth_g) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_g))−9]−2.5[log10(5/(psfdepth_g))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_r psfdepth_r float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_r)5/(psfdepth_r) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_r))−9]−2.5[log10(5/(psfdepth_r))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_z psfdepth_z float COMMENT "For a 5σ5σ point source detection limit in gg, 5/(√psfdepth_z)5/(psfdepth_z) gives flux in nanomaggies and −2.5[log10(5/(√psfdepth_z))−9]−2.5[log10(5/(psfdepth_z))−9] gives corresponding AB magnitude";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_g galdepth_g float COMMENT "As for psfdepth_g but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_r galdepth_r float COMMENT "As for psfdepth_r but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE galdepth_z galdepth_z float COMMENT "As for psfdepth_z but for a galaxy (0.45\" exp, round) detection sensitivity";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w1 psfdepth_w1 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w2 psfdepth_w2 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w3 psfdepth_w3 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W3";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE psfdepth_w4 psfdepth_w4 float COMMENT "As for psfdepth_g (and also on the AB system) but for WISE W4";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE wise_coadd_id wise_coadd_id string COMMENT "unWISE coadd file name for the center of each object";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w1 lc_flux_w1 array<float> COMMENT "flux_w1 in each of up to eleven unWISE coadd epochs (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_w2 lc_flux_w2 array<float> COMMENT "flux_w2 in each of up to eleven unWISE coadd epochs (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w1 lc_flux_ivar_w1 array<float> COMMENT "Inverse variance of lc_flux_w1 (AB system)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_flux_ivar_w2 lc_flux_ivar_w2 array<float> COMMENT "Inverse variance of lc_flux_w2 (AB)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w1 lc_nobs_w1 array<smallint> COMMENT "nobs_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_nobs_w2 lc_nobs_w2 array<smallint> COMMENT "nobs_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w1 lc_fracflux_w1 array<float> COMMENT "fracflux_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_fracflux_w2 lc_fracflux_w2 array<float> COMMENT "fracflux_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w1 lc_rchisq_w1 array<float> COMMENT "rchisq_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_rchisq_w2 lc_rchisq_w2 array<float> COMMENT "rchisq_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w1 lc_mjd_w1 array<double> COMMENT "mjd_w1 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE lc_mjd_w2 lc_mjd_w2 array<double> COMMENT "mjd_w2 in each of up to eleven unWISE coadd epochs";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev fracdev float COMMENT "Fraction of model in deVauc [0,1]";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE fracdev_ivar fracdev_ivar float COMMENT "Inverse variance of fracdev";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r shapeexp_r float COMMENT "Half-light radius of exponential model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_r_ivar shapeexp_r_ivar float COMMENT "Inverse variance of shapeexp_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1 shapeexp_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e1_ivar shapeexp_e1_ivar float COMMENT "Inverse variance of shapeexp_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2 shapeexp_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapeexp_e2_ivar shapeexp_e2_ivar float COMMENT "Inverse variance of shapeexp_e2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r shapedev_r float COMMENT "Half-light radius of deVaucouleurs model (>0)";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_r_ivar shapedev_r_ivar float COMMENT "Inverse variance of shapedev_r";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1 shapedev_e1 float COMMENT "Ellipticity component 1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e1_ivar shapedev_e1_ivar float COMMENT "Inverse variance of shapedev_e1";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2 shapedev_e2 float COMMENT "Ellipticity component 2";<br />
ALTER TABLE cosmohub.legacy_survey_dr8 CHANGE shapedev_e2_ivar shapedev_e2_ivar float COMMENT "Inverse variance of shapedev_e2";<br />
<br />
== Sweep photo-z ==</div>
Tallada
https://pwiki.pic.es/index.php?title=JupyterHub&diff=775
JupyterHub
2020-03-24T14:34:54Z
<p>Tallada: /* Link an existing environment to Jupyter */</p>
<hr />
<div><br />
= Introduction =<br />
<br />
PIC offers a service for running Jupyter notebooks on CPU or GPU resources. This service is primarily thought for code developing or prototyping rather than data processing. The usage is similar to running notebooks on your personal computer but offers the advantage of developing and testing your code on different hardware configurations. <br />
<br />
Since the service is strictly thought for development and small scale testing tasks, a shutdown policy for the sessions has been put in place:<br />
<br />
# The maximum duration for a session is 48h.<br />
# After an idle period of 2 hours, the session will be closed. <br />
<br />
In practice that means that you should estimate the test data volume that you work with during a session to be able to be processed in less than 48 hours.<br />
<br />
= How to connect to the service =<br />
<br />
Got to [https://jupyter01.pic.es jupyter.pic.es] to see your login screen.<br />
<br />
[[File:login.png|frameless|700px|Login screen]]<br />
<br />
Sign in with your PIC user credentials. This will prompt you to the following screen.<br />
<br />
[[File:screen01.png|700px]]<br />
<br />
Here you can choose the hardware configuration for your Jupyter session. After choosing a configuration and pressing start the next screen will show you the progress of the initialisation process. Keep in mind that a job containing your Jupyter session is actually sent to the HTCondor queuing system and waiting for available resources before being started. This usually takes less than a minute but can take up to a few depending on our resource usage.<br />
<br />
[[File:screen02.png|600px]]<br />
<br />
In the next screen you can choose the tool that you want to use for your work: a Python notebook, a Python console or a plain bash terminal.<br />
<br />
[[File:screen03.png|700px]]<br />
<br />
Your python environments should appear under Notebook and Console headers. In a later section we will show you how to create a new environment and to remove an existing one.<br />
<br />
= Terminate your session and logout =<br />
<br />
It is important that you terminate your session before you log out. In order to do so, go to the top page menu "'''File''' -> '''Hub Control Panel'''" and you will see the following screen.<br />
<br />
[[File:screen04.png|600px]]<br />
<br />
Here, click on the '''Stop My Server''' button. After that you can log out by clicking the '''Logout''' button in the right upper corner.<br />
<br />
= Python virtual environments =<br />
<br />
This section covers the use of Python virtual environments with Jupyter.<br />
<br />
== Initialize conda (only if you use conda) ==<br />
<br />
Before using conda in your bash session, you have to initialize it. For access to an available conda installation, please get in contact with your project liaison at PIC. He/she will give you the actual value for the '''/path/to/anaconda''' placeholder.<br />
<br />
Log onto Jupyter and start a session. On the homepage of your Jupyter session, click on the terminal button on the session dashboard on the right to open a bash terminal.<br />
<br />
First, let's initialize conda for our bash sessions:<br />
<pre><br />
[neissner@td110 ~]$ eval "$(/path/to/anaconda/bin/conda shell.bash hook)"<br />
[neissner@td110 ~]$ conda init<br />
</pre><br />
<br />
This actually changes the .bashrc file in your home directory in order to activate the base environment on login.<br />
To avoid that the base environment is activated every time you log on to a node, run:<br />
<pre><br />
[neissner@td110 ~]$ conda config --set auto_activate_base false<br />
</pre><br />
<br />
For now you can exit the terminal.<br />
<pre><br />
[neissner@td110 ~]$ exit<br />
</pre><br />
<br />
<br />
== Link an existing environment to Jupyter ==<br />
<br />
If you want to know which environments are available for your project, please contact your project liaison at PIC. He/she will give you the values for the placeholder '''/path/to/predefined/venv/environment''' or '''/path/to/predefined/conda/environment'''. Although, you can find instructions on how to create your own environments, e.g. here: [[#Create_virtual_environments_with_venv_or_conda]], we would like to encourage the use of the predeployed environments. The main reason is the sheer size of a virtual environment which reaches easily several GB. If for any reason you need to use your own environment, make sure that the '''ipykernel''' module is installed.<br />
<br />
Log into Jupyter, start a session. From the session dashboard choose the bash terminal.<br />
<br />
Inside the terminal, activate your environment.<br />
<br />
For '''conda''' environments:<br />
<pre><br />
[neissner@td110 ~]$ conda activate /path/to/predefined/conda/environment<br />
(...) [neissner@td110 ~]$ <br />
</pre><br />
The parenthesis (...) in front of your bash prompt show the absolute path to your environment. <br />
<br />
For '''venv''' environments:<br />
<pre><br />
[neissner@td110 ~]$ . /path/to/predefined/venv/environment/bin/activate<br />
(...) [neissner@td110 ~]$ <br />
</pre><br />
<br />
Link the environment to a Jupyter kernel. For both, conda and venv:<br />
<pre><br />
(...) [neissner@td110 ~]$ python -m ipykernel install --user --name=whatever_kernel_name<br />
Installed kernelspec whatever_kernel_name in /nfs/pic.es/user/n/neissner/.local/share/jupyter/kernels/whatever_kernel_name<br />
</pre><br />
<br />
Deactivate your environment. <br />
<br />
For conda:<br />
<pre><br />
(...) [neissner@td110 ~]$ conda deactivate<br />
</pre><br />
<br />
For venv:<br />
<pre><br />
(...) [neissner@td110 ~]$ deactivate<br />
</pre><br />
<br />
Now you can exit the terminal. After refreshing the Jupyter page your whatever_kernel_name appears in the dashboard. In this example '''test''' has been used for whatever_kernel_name<br />
<br />
[[File:screen05.png|700px]]<br />
<br />
== Unlink an environment from Jupyter ==<br />
Log onto Jupyter, start a session and from the session dashboard choose the bash terminal. To remove your environment/kernel from Jupyter run:<br />
<pre><br />
[neissner@td110 ~]$ jupyter kernelspec uninstall whatever_kernel_name<br />
Kernel specs to remove:<br />
whatever_kernel_name /nfs/pic.es/user/n/neissner/.local/share/jupyter/kernels/whatever_kernel_name<br />
Remove 1 kernel specs [y/N]: y<br />
[RemoveKernelSpec] Removed /nfs/pic.es/user/n/neissner/.local/share/jupyter/kernels/whatever_kernel_name<br />
</pre><br />
Keep in mind that, although not available in Jupyter anymore, the environment still exists. Whenever you need it, you can link it again.<br />
<br />
== Create virtual environments with venv or conda ==<br />
<br />
Before creating a new environment, please get in contact with your project liaison at PIC as there may be already a suitable environment for your needs in place.<br />
<br />
If none of the existing environments suits your needs, you can create a new environment.<br />
First, create a directory in a suitable place to store the environment. For single-user environments, place them in your home under ~/env. For environments that will be shared with other project users, contact your project liaison and ask him/her for a path in a shared storage volume that is visible to all of them.<br />
<br />
Once you have the location (i.e. /path/to/env/folder), create the environment with the following commands:<br />
<br />
For '''venv''' environments '''(recommended)'''<br />
<pre><br />
[neissner@td110 ~]$ cd /path/to/env/folder<br />
[neissner@td110 ~]$ python3 -m venv your_env<br />
</pre><br />
Now you should be able to activate your environment and install additional modules<br />
<pre><br />
[neissner@td110 ~]$ cd /path/to/env/folder<br />
[neissner@td110 ~]$ . your_env/bin/activate<br />
(...)[neissner@td110 ~]$ pip install additional_module1 additional_module2 ...<br />
</pre><br />
<br />
For '''conda''' enviroments<br />
<pre><br />
[neissner@td110 ~]$ conda create --prefix /path/to/env/folder/your_env module1 module2 ...<br />
</pre><br />
The list of modules (module1, module2, ...) is optional. For instance, for a python3 environment with scipy you would specify: ''python=3 scipy'' <br />
<br />
Now you should be able to activate your environment and install additional modules<br />
<pre><br />
[neissner@td110 ~]$ conda activate /path/to/env/folder/your_env<br />
(...)[neissner@td110 ~]$ conda install additional_module1 additional_module2 ...<br />
</pre></div>
Tallada