Public PIC Wiki - User contributions [en]

JupyterHub

2022-07-28T09:10:17Z

Jcarrete:

= Introduction =

PIC offers a service for running Jupyter notebooks on CPU or GPU resources. This service is primarily thought for code developing or prototyping rather than data processing. The usage is similar to running notebooks on your personal computer but offers the advantage of developing and testing your code on different hardware configurations, as well as facilitating the scalability of the code since it is being tested in the same environment in which it would run on a mass scale.

Since the service is strictly thought for development and small scale testing tasks, a shutdown policy for the sessions has been put in place:

# The maximum duration for a session is 48h.
# After an idle period of 2 hours, the session will be closed.

In practice that means that you should estimate the test data volume that you work with during a session to be able to be processed in less than 48 hours.

= How to connect to the service =

Got to [https://jupyter.pic.es jupyter.pic.es] to see your login screen.

[[File:login.png|frameless|700px|Login screen]]

Sign in with your PIC user credentials. This will prompt you to the following screen.

[[File:Screen01.png|700px|current]]

Here you can choose the hardware configuration for your Jupyter session. Also, you have to choose the experiment (project) you are working on during the Jupyter session. After choosing a configuration and pressing start the next screen will show you the progress of the initialisation process. Keep in mind that a job containing your Jupyter session is actually sent to the HTCondor queuing system and waiting for available resources before being started. This usually takes less than a minute but can take up to a few depending on our resource usage.

[[File:screen02.png|600px]]

In the next screen you can choose the tool that you want to use for your work: a Python notebook, a Python console or a plain bash terminal.
For the Python environment (either notebook or environment) you have two default options:
* the ipykernel version of Python 3
* the XPython version of Python 3.9, this one allows you to use the integrated debugging module.

Further you see an icon with a "D" - desktop, this one starts a VNC session that allows the use of programs with graphical user interfaces.

Also, recently you can find the icon of Visual Studio, an integrated development environment.

[[File:Screen03.png|700px]]

Your python environments should appear under Notebook and Console headers. In a later section we will show you how to create a new environment and to remove an existing one.

= Terminate your session and logout =

It is important that you terminate your session before you log out. In order to do so, go to the top page menu "'''File''' -> '''Hub Control Panel'''" and you will see the following screen.

[[File:screen04.png|600px]]

Here, click on the '''Stop My Server''' button. After that you can log out by clicking the '''Logout''' button in the right upper corner.

= Python virtual environments =

This section covers the use of Python virtual environments with Jupyter.

== Initialize conda (only if you use conda) ==

Before using conda in your bash session, you have to initialize it. For access to an available conda installation, please get in contact with your project liaison at PIC. He/she will give you the actual value for the '''/path/to/anaconda''' placeholder.

Log onto Jupyter and start a session. On the homepage of your Jupyter session, click on the terminal button on the session dashboard on the right to open a bash terminal.

First, let's initialize conda for our bash sessions:
<pre>
[neissner@td110 ~]$ eval "$(/path/to/anaconda/bin/conda shell.bash hook)"
[neissner@td110 ~]$ conda init
</pre>

This actually changes the .bashrc file in your home directory in order to activate the base environment on login.
To avoid that the base environment is activated every time you log on to a node, run:
<pre>
[neissner@td110 ~]$ conda config --set auto_activate_base false
</pre>

For now you can exit the terminal.
<pre>
[neissner@td110 ~]$ exit
</pre>

== Link an existing environment to Jupyter ==

If you want to know which environments are available for your project, please contact your project liaison at PIC. He/she will give you the values for the placeholder '''/path/to/predefined/venv/environment''' or '''/path/to/predefined/conda/environment'''. Although, you can find instructions on how to create your own environments, e.g. [[#Create_virtual_environments_with_venv_or_conda | here]], we would like to encourage the use of the predeployed environments. The main reason is the sheer size of a virtual environment which reaches easily several GB. If for any reason you need to use your own environment, make sure that the '''ipykernel''' module is installed.

Log into Jupyter, start a session. From the session dashboard choose the bash terminal.

Inside the terminal, activate your environment.

For '''conda''' environments:
<pre>
[neissner@td110 ~]$ conda activate /path/to/predefined/conda/environment
(...) [neissner@td110 ~]$
</pre>
The parenthesis (...) in front of your bash prompt show the absolute path to your environment.

For '''venv''' environments:
<pre>
[neissner@td110 ~]$ . /path/to/predefined/venv/environment/bin/activate
(...) [neissner@td110 ~]$
</pre>

Link the environment to a Jupyter kernel. For both, conda and venv:
<pre>
(...) [neissner@td110 ~]$ python -m ipykernel install --user --name=whatever_kernel_name
Installed kernelspec whatever_kernel_name in
/nfs/pic.es/user/n/neissner/.local/share/jupyter/kernels/whatever_kernel_name
</pre>

Deactivate your environment.

For conda:
<pre>
(...) [neissner@td110 ~]$ conda deactivate
</pre>

For venv:
<pre>
(...) [neissner@td110 ~]$ deactivate
</pre>

Now you can exit the terminal. After refreshing the Jupyter page your whatever_kernel_name appears in the dashboard. In this example '''test''' has been used for whatever_kernel_name

[[File:screen05.png|700px]]

== Unlink an environment from Jupyter ==
Log onto Jupyter, start a session and from the session dashboard choose the bash terminal. To remove your environment/kernel from Jupyter run:
<pre>
[neissner@td110 ~]$ jupyter kernelspec uninstall whatever_kernel_name
Kernel specs to remove:
whatever_kernel_name /nfs/pic.es/user/n/neissner/.local/share/jupyter/kernels/whatever_kernel_name
Remove 1 kernel specs [y/N]: y
[RemoveKernelSpec] Removed /nfs/pic.es/user/n/neissner/.local/share/jupyter/kernels/whatever_kernel_name
</pre>
Keep in mind that, although not available in Jupyter anymore, the environment still exists. Whenever you need it, you can link it again.

== Create virtual environments with venv or conda ==

Before creating a new environment, please get in contact with your project liaison at PIC as there may be already a suitable environment for your needs in place.

If none of the existing environments suits your needs, you can create a new environment.
First, create a directory in a suitable place to store the environment. For single-user environments, place them in your home under ~/env. For environments that will be shared with other project users, contact your project liaison and ask him/her for a path in a shared storage volume that is visible to all of them.

Once you have the location (i.e. /path/to/env/folder), create the environment with the following commands:

For '''venv''' environments '''(recommended)'''
<pre>
[neissner@td110 ~]$ cd /path/to/env/folder
[neissner@td110 ~]$ python3 -m venv your_env
</pre>
Now you should be able to activate your environment and install additional modules
<pre>
[neissner@td110 ~]$ cd /path/to/env/folder
[neissner@td110 ~]$ . your_env/bin/activate
(...)[neissner@td110 ~]$ pip install additional_module1 additional_module2 ...
</pre>

For '''conda''' enviroments
<pre>
[neissner@td110 ~]$ conda create --prefix /path/to/env/folder/your_env module1 module2 ...
</pre>
The list of modules (module1, module2, ...) is optional. For instance, for a python3 environment with scipy you would specify: ''python=3 scipy''

Now you should be able to activate your environment and install additional modules
<pre>
[neissner@td110 ~]$ conda activate /path/to/env/folder/your_env
(...)[neissner@td110 ~]$ conda install additional_module1 additional_module2 ...
</pre>

JupyterHub

2022-07-27T08:45:24Z

Jcarrete:

= Introduction =

PIC offers a service for running Jupyter notebooks on CPU or GPU resources. This service is primarily thought for code developing or prototyping rather than data processing. The usage is similar to running notebooks on your personal computer but offers the advantage of developing and testing your code on different hardware configurations, as well as facilitating the scalability of the code since it is being tested in the same environment in which it would run on a mass scale.

Since the service is strictly thought for development and small scale testing tasks, a shutdown policy for the sessions has been put in place:

# The maximum duration for a session is 48h.
# After an idle period of 2 hours, the session will be closed.

In practice that means that you should estimate the test data volume that you work with during a session to be able to be processed in less than 48 hours.

= How to connect to the service =

Got to [https://jupyter.pic.es jupyter.pic.es] to see your login screen.

[[File:login.png|frameless|700px|Login screen]]

Sign in with your PIC user credentials. This will prompt you to the following screen.

[[File:Screen01.png|700px|current]]

Here you can choose the hardware configuration for your Jupyter session. Also, you have to choose the experiment (project) you are working on during the Jupyter session. After choosing a configuration and pressing start the next screen will show you the progress of the initialisation process. Keep in mind that a job containing your Jupyter session is actually sent to the HTCondor queuing system and waiting for available resources before being started. This usually takes less than a minute but can take up to a few depending on our resource usage.

[[File:screen02.png|600px]]

In the next screen you can choose the tool that you want to use for your work: a Python notebook, a Python console or a plain bash terminal.
For the Python environment (either notebook or environment) you have two default options:
* the ipykernel version of Python 3
* the XPython version of Python 3.9, this one allows you to use the integrated debugging module.

Further you see an icon with a "D" - desktop, this one starts a VNC session that allows the use of programs with graphical user interfaces.

[[File:Screen03.png|700px]]

Your python environments should appear under Notebook and Console headers. In a later section we will show you how to create a new environment and to remove an existing one.

= Terminate your session and logout =

It is important that you terminate your session before you log out. In order to do so, go to the top page menu "'''File''' -> '''Hub Control Panel'''" and you will see the following screen.

[[File:screen04.png|600px]]

Here, click on the '''Stop My Server''' button. After that you can log out by clicking the '''Logout''' button in the right upper corner.

= Python virtual environments =

This section covers the use of Python virtual environments with Jupyter.

== Initialize conda (only if you use conda) ==

Before using conda in your bash session, you have to initialize it. For access to an available conda installation, please get in contact with your project liaison at PIC. He/she will give you the actual value for the '''/path/to/anaconda''' placeholder.

Log onto Jupyter and start a session. On the homepage of your Jupyter session, click on the terminal button on the session dashboard on the right to open a bash terminal.

First, let's initialize conda for our bash sessions:
<pre>
[neissner@td110 ~]$ eval "$(/path/to/anaconda/bin/conda shell.bash hook)"
[neissner@td110 ~]$ conda init
</pre>

This actually changes the .bashrc file in your home directory in order to activate the base environment on login.
To avoid that the base environment is activated every time you log on to a node, run:
<pre>
[neissner@td110 ~]$ conda config --set auto_activate_base false
</pre>

For now you can exit the terminal.
<pre>
[neissner@td110 ~]$ exit
</pre>

== Link an existing environment to Jupyter ==

If you want to know which environments are available for your project, please contact your project liaison at PIC. He/she will give you the values for the placeholder '''/path/to/predefined/venv/environment''' or '''/path/to/predefined/conda/environment'''. Although, you can find instructions on how to create your own environments, e.g. [[#Create_virtual_environments_with_venv_or_conda | here]], we would like to encourage the use of the predeployed environments. The main reason is the sheer size of a virtual environment which reaches easily several GB. If for any reason you need to use your own environment, make sure that the '''ipykernel''' module is installed.

Log into Jupyter, start a session. From the session dashboard choose the bash terminal.

Inside the terminal, activate your environment.

For '''conda''' environments:
<pre>
[neissner@td110 ~]$ conda activate /path/to/predefined/conda/environment
(...) [neissner@td110 ~]$
</pre>
The parenthesis (...) in front of your bash prompt show the absolute path to your environment.

For '''venv''' environments:
<pre>
[neissner@td110 ~]$ . /path/to/predefined/venv/environment/bin/activate
(...) [neissner@td110 ~]$
</pre>

Link the environment to a Jupyter kernel. For both, conda and venv:
<pre>
(...) [neissner@td110 ~]$ python -m ipykernel install --user --name=whatever_kernel_name
Installed kernelspec whatever_kernel_name in
/nfs/pic.es/user/n/neissner/.local/share/jupyter/kernels/whatever_kernel_name
</pre>

Deactivate your environment.

For conda:
<pre>
(...) [neissner@td110 ~]$ conda deactivate
</pre>

For venv:
<pre>
(...) [neissner@td110 ~]$ deactivate
</pre>

Now you can exit the terminal. After refreshing the Jupyter page your whatever_kernel_name appears in the dashboard. In this example '''test''' has been used for whatever_kernel_name

[[File:screen05.png|700px]]

== Unlink an environment from Jupyter ==
Log onto Jupyter, start a session and from the session dashboard choose the bash terminal. To remove your environment/kernel from Jupyter run:
<pre>
[neissner@td110 ~]$ jupyter kernelspec uninstall whatever_kernel_name
Kernel specs to remove:
whatever_kernel_name /nfs/pic.es/user/n/neissner/.local/share/jupyter/kernels/whatever_kernel_name
Remove 1 kernel specs [y/N]: y
[RemoveKernelSpec] Removed /nfs/pic.es/user/n/neissner/.local/share/jupyter/kernels/whatever_kernel_name
</pre>
Keep in mind that, although not available in Jupyter anymore, the environment still exists. Whenever you need it, you can link it again.

== Create virtual environments with venv or conda ==

Before creating a new environment, please get in contact with your project liaison at PIC as there may be already a suitable environment for your needs in place.

If none of the existing environments suits your needs, you can create a new environment.
First, create a directory in a suitable place to store the environment. For single-user environments, place them in your home under ~/env. For environments that will be shared with other project users, contact your project liaison and ask him/her for a path in a shared storage volume that is visible to all of them.

Once you have the location (i.e. /path/to/env/folder), create the environment with the following commands:

For '''venv''' environments '''(recommended)'''
<pre>
[neissner@td110 ~]$ cd /path/to/env/folder
[neissner@td110 ~]$ python3 -m venv your_env
</pre>
Now you should be able to activate your environment and install additional modules
<pre>
[neissner@td110 ~]$ cd /path/to/env/folder
[neissner@td110 ~]$ . your_env/bin/activate
(...)[neissner@td110 ~]$ pip install additional_module1 additional_module2 ...
</pre>

For '''conda''' enviroments
<pre>
[neissner@td110 ~]$ conda create --prefix /path/to/env/folder/your_env module1 module2 ...
</pre>
The list of modules (module1, module2, ...) is optional. For instance, for a python3 environment with scipy you would specify: ''python=3 scipy''

Now you should be able to activate your environment and install additional modules
<pre>
[neissner@td110 ~]$ conda activate /path/to/env/folder/your_env
(...)[neissner@td110 ~]$ conda install additional_module1 additional_module2 ...
</pre>

Main Page

2022-05-25T09:36:01Z

Jcarrete:

Storage Department

2022-05-24T13:24:59Z

Jcarrete: Created page with "== Storage Department == * dCache.org * FileSystems"

== Storage Department ==

* [[dCache.org]]
* [[FileSystems]]

Euclid

2022-05-24T13:24:37Z

Jcarrete: Created page with "I include here all the information that was included in the Astrophysics and Cosmology Departament: * Gitlab migration *Python_Jupyter | Python enviro..."

I include here all the information that was included in the Astrophysics and Cosmology Departament:

*[[Gitlab_Migration | Gitlab migration]]
*[[Python_Jupyter | Python environments and Jupyter ]]
*[[AC_UserManual | User manual]]
*[[Catalogs | Catalogs]]
*[[Mock pipeline v2.0]]

2021-02-05T23:18:30Z

Jcarrete:

High-z galaxies

2021-02-02T10:02:33Z

Jcarrete:

High-z QSOs

2021-02-02T09:54:47Z

Jcarrete:

Standard galaxies

2021-02-02T09:47:07Z

Jcarrete:

GSIR more

2021-01-26T16:37:15Z

Jcarrete: Created page with "This is the second try to generate GSIR. Below there are the notes I wrote during the process: ---- EMPEZAMOS DE NUEVO CON 3 areas: (+- 4 en lon,lat) ---- SWF1 obsreque..."

This is the second try to generate GSIR.

Below there are the notes I wrote during the process:

----

EMPEZAMOS DE NUEVO CON 3 areas:

(+- 4 en lon,lat)

----

SWF1
obsrequestid: 52929
5.233205 16.429612 -23.885618 -13.2708
En el primer octante:
5.233205 16.429612 13.2708 23.885618

----

SWF2
obsrequestid: 1717
213.074616 251.75015 72.188664 82.03694
En el primer octante:
213.074616-180 251.75015-180 72.188664 82.03694
33.074616 71.75015 72.188664 82.03694

----

DEEP
obsrequestid: self_calib_field
321.985796 328.956117 50.480548 54.70978
En el primer octante:
321.985796-270 328.956117-270 50.480548 54.70978
51.985796 58.956117 50.480548 54.70978

----

CREO LA TABLA CLUSTERED BUCKET "_c2" A PARTIR DE LA 1.9.8_c (qué es el octante entero del DM halo catalog v1.1, pero sin flujos ni lensing)

CREATE TABLE jcarrete.flagship_mock_1_9_8_swf1_swf2_deep_c2(
`kind` BIGINT,
`luminosity_r01` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`abs_mag_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`luminosity_r01_evolved` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` INT,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`hpix_29_nest` BIGINT,
`hpix_13_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`observed_redshift_gal` DOUBLE,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha_ext` DOUBLE,
`logf_halpha` DOUBLE,
`Halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_Halpha_model3_ext` DOUBLE,
`loglum_Halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`blanco_decam_i` DOUBLE,
`sdss_r01` DOUBLE,
`euclid_nisp_h` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`dominant_shape` BIGINT,
`bulge_angle` DOUBLE,
`disk_angle` DOUBLE,
`median_major_axis` DOUBLE,
`scale_length` DOUBLE,
`bulge_fraction` DOUBLE,
`disk_scalelength` DOUBLE,
`disk_nsersic` DOUBLE,
`disk_r50` DOUBLE,
`bulge_r50` DOUBLE,
`bulge_nsersic` DOUBLE,
`inclination_angle` DOUBLE,
`disk_axis_ratio` DOUBLE,
`bulge_axis_ratio` DOUBLE,
`disk_ellipticity` DOUBLE,
`bulge_ellipticity` DOUBLE,
`random_index` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
PARTITIONED BY (
`step` smallint
)
CLUSTERED BY (
hpix_13_nest
)
SORTED BY (
hpix_13_nest ASC
)
INTO 128 BUCKETS
STORED AS ORC
;

INSERT OVERWRITE TABLE jcarrete.flagship_mock_1_9_8_swf1_swf2_deep_c2
SELECT
`kind`, `luminosity_r01`, `halo_x`, `halo_y`, `halo_z`, `halo_vx`, `halo_vy`, `halo_vz`, `halo_r`, `true_redshift_halo`, `halo_lm`, `halo_n_sats`, `n_gals`, `abs_mag_r01`, `abs_mag_r01_evolved`, `luminosity_r01_evolved`, `gr_restframe`, `color_kind`, `x_gal`, `y_gal`, `z_gal`, `r_gal`, `true_redshift_gal`, `ra_gal`, `dec_gal`, `hpix_nest`, SHIFTRIGHT(`hpix_nest`, (29-13)*2), `vx_gal`, `vy_gal`, `vz_gal`, `vrad_gal`, `delta_r`, `observed_redshift_gal`, `sed_ke`, `gr_cosmos`, `sed_cosmos`, `ext_curve_cosmos`, `ebv_cosmos`, `is_within_cosmos`, `cosmos_distance`, `abs_mag_uv_dereddened`, `log_ml_r01`, `log_stellar_mass`, `metallicity`, `log_sfr`, `logf_halpha_ext`, `logf_halpha`, `Halpha_scatter`, `loglum_halpha`, `loglum_halpha_ext`, `logf_dummy`, `z_dummy`, `logf_halpha_model1_ext`, `logf_halpha_model1`, `loglum_halpha_model1_ext`, `loglum_halpha_model1`, `logf_halpha_model3_ext`, `logf_halpha_model3`, `loglum_Halpha_model3_ext`, `loglum_Halpha_model3`, `logf_hbeta_model1_ext`, `logf_hbeta_model1`, `logf_o2_model1_ext`, `logf_o2_model1`, `logf_n2_model1_ext`, `logf_n2_model1`, `logf_o3_model1_ext`, `logf_o3_model1`, `logf_s2_model1_ext`, `logf_s2_model1`, `logf_hbeta_model3_ext`, `logf_hbeta_model3`, `logf_o2_model3_ext`, `logf_o2_model3`, `logf_n2_model3_ext`, `logf_n2_model3`, `logf_o3_model3_ext`, `logf_o3_model3`, `logf_s2_model3_ext`, `logf_s2_model3`, `blanco_decam_i`, `sdss_r01`, `euclid_nisp_h`, `l_gal`, `b_gal`, `mw_extinction`, `blanco_decam_i_odonnell_ext`, `euclid_nisp_h_odonnell_ext`, `sdss_r01_odonnell_ext`, `blanco_decam_i_el_model1_odonnell_ext`, `blanco_decam_i_el_model3_odonnell_ext`, `sdss_r01_el_model1_odonnell_ext`, `sdss_r01_el_model3_odonnell_ext`, `euclid_nisp_h_el_model1_odonnell_ext`, `euclid_nisp_h_el_model3_odonnell_ext`, `dominant_shape`, `bulge_angle`, `disk_angle`, `median_major_axis`, `scale_length`, `bulge_fraction`, `disk_scalelength`, `disk_nsersic`, `disk_r50`, `bulge_r50`, `bulge_nsersic`, `inclination_angle`, `disk_axis_ratio`, `bulge_axis_ratio`, `disk_ellipticity`, `bulge_ellipticity`, `random_index`, `halo_id`, `galaxy_id`, `step`
FROM jcarrete.flagship_mock_1_9_8_c
WHERE
(ra_gal >= 5.233205 SWF1
AND ra_gal <= 16.429612
AND dec_gal >= 13.2708
AND dec_gal <= 23.885618)
OR SWF2
(ra_gal >= 33.074616
AND ra_gal <= 71.75015
AND dec_gal >= 72.188664
AND dec_gal <= 82.03694)
OR DEEP
(ra_gal >= 51.985796
AND ra_gal <= 58.956117
AND dec_gal >= 50.480548
AND dec_gal <= 54.70978)
;

----

'''Genero los flujos en el lugar que corresponden'''

----

SWF1: /user/jcarrete/data/euclid/flagship_mock_1_9_9_swf1

SWF1
obsrequestid: 52929
5.233205 16.429612 -23.885618 -13.2708
En el primer octante:
5.233205 16.429612 13.2708 23.885618
No cojo ni las shapes ni los flujos
DEC = -DEC. RA no necesita rotar

halo_sql = """
SELECT halo_id, galaxy_id, kind, random_index,
ra_gal, -dec_gal as dec_gal,
x_gal, y_gal, z_gal, r_gal, true_redshift_gal, observed_redshift_gal,
hpix_29_nest, vx_gal, vy_gal, vz_gal, vrad_gal, delta_r, abs_mag_r01,
luminosity_r01, abs_mag_r01_evolved, sdss_r01, abs_mag_uv_dereddened, gr_restframe,
color_kind, sed_ke, gr_cosmos, sed_cosmos, is_within_cosmos, cosmos_distance,
ext_curve_cosmos, ebv_cosmos, log_ml_r01, log_stellar_mass, metallicity, log_sfr,
logf_halpha, logf_halpha_ext, halpha_scatter, loglum_halpha, loglum_halpha_ext,
logf_dummy, z_dummy, logf_halpha_model1_ext, logf_halpha_model1, loglum_halpha_model1_ext,
loglum_halpha_model1, logf_halpha_model3_ext, logf_halpha_model3, loglum_halpha_model3_ext,
loglum_halpha_model3, logf_hbeta_model1_ext, logf_hbeta_model1, logf_o2_model1_ext,
logf_o2_model1, logf_n2_model1_ext, logf_n2_model1, logf_o3_model1_ext, logf_o3_model1,
logf_s2_model1_ext, logf_s2_model1, logf_hbeta_model3_ext, logf_hbeta_model3,
logf_o2_model3_ext, logf_o2_model3, logf_n2_model3_ext, logf_n2_model3, logf_o3_model3_ext,
logf_o3_model3, logf_s2_model3_ext, logf_s2_model3,
halo_x, halo_y, halo_z, halo_vx, halo_vy, halo_vz, halo_r, true_redshift_halo,
halo_lm, halo_n_sats, n_gals, luminosity_r01_evolved, hpix_13_nest, step
FROM jcarrete.flagship_mock_1_9_8_swf1_swf2_deep_c2
WHERE ra_gal >= 5.233205
AND ra_gal <= 16.429612
AND dec_gal >= 13.2708
AND dec_gal <= 23.885618
"""

Genero la tabla externa en parquet:

/software/astro/scripts/create_table_from_parquet.sh -p /hdfs/user/jcarrete/data/euclid/flagship_mock_1_9_9_swf1/0.pq -t jcarrete.flagship_mock_1_9_9_swf1_pq -e

CREATE EXTERNAL TABLE jcarrete.flagship_mock_1_9_9_swf1_pq (
`kind` BIGINT,
`random_index` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`observed_redshift_gal` DOUBLE,
`hpix_29_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`abs_mag_r01` DOUBLE,
`luminosity_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`sdss_r01` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` BIGINT,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha` DOUBLE,
`logf_halpha_ext` DOUBLE,
`halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_halpha_model3_ext` DOUBLE,
`loglum_halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`luminosity_r01_evolved` DOUBLE,
`hpix_13_nest` BIGINT,
`step` BIGINT,
`blanco_decam_g` DOUBLE,
`blanco_decam_i` DOUBLE,
`blanco_decam_r` DOUBLE,
`blanco_decam_z` DOUBLE,
`subaru_hsc_z` DOUBLE,
`jst_jpcam_g` DOUBLE,
`kids_g` DOUBLE,
`kids_i` DOUBLE,
`kids_r` DOUBLE,
`kids_u` DOUBLE,
`lsst_g` DOUBLE,
`lsst_i` DOUBLE,
`lsst_r` DOUBLE,
`lsst_u` DOUBLE,
`lsst_y` DOUBLE,
`lsst_z` DOUBLE,
`cfht_megacam_r` DOUBLE,
`cfht_megacam_u` DOUBLE,
`pan_starrs_i` DOUBLE,
`pan_starrs_z` DOUBLE,
`2mass_h` DOUBLE,
`2mass_j` DOUBLE,
`2mass_ks` DOUBLE,
`gaia_bp` DOUBLE,
`gaia_g` DOUBLE,
`gaia_rp` DOUBLE,
`euclid_nisp_h` DOUBLE,
`euclid_nisp_j` DOUBLE,
`euclid_nisp_y` DOUBLE,
`euclid_vis` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`2mass_h_odonnell_ext` DOUBLE,
`2mass_j_odonnell_ext` DOUBLE,
`2mass_ks_odonnell_ext` DOUBLE,
`blanco_decam_g_odonnell_ext` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`blanco_decam_r_odonnell_ext` DOUBLE,
`blanco_decam_z_odonnell_ext` DOUBLE,
`cfht_megacam_r_odonnell_ext` DOUBLE,
`cfht_megacam_u_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`euclid_nisp_j_odonnell_ext` DOUBLE,
`euclid_nisp_y_odonnell_ext` DOUBLE,
`euclid_vis_odonnell_ext` DOUBLE,
`gaia_bp_odonnell_ext` DOUBLE,
`gaia_g_odonnell_ext` DOUBLE,
`gaia_rp_odonnell_ext` DOUBLE,
`jst_jpcam_g_odonnell_ext` DOUBLE,
`kids_g_odonnell_ext` DOUBLE,
`kids_i_odonnell_ext` DOUBLE,
`kids_r_odonnell_ext` DOUBLE,
`kids_u_odonnell_ext` DOUBLE,
`lsst_g_odonnell_ext` DOUBLE,
`lsst_i_odonnell_ext` DOUBLE,
`lsst_r_odonnell_ext` DOUBLE,
`lsst_u_odonnell_ext` DOUBLE,
`lsst_y_odonnell_ext` DOUBLE,
`lsst_z_odonnell_ext` DOUBLE,
`pan_starrs_i_odonnell_ext` DOUBLE,
`pan_starrs_z_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`subaru_hsc_z_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model3_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model1_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model3_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model1_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model3_odonnell_ext` DOUBLE,
`kids_g_el_model1_odonnell_ext` DOUBLE,
`kids_g_el_model3_odonnell_ext` DOUBLE,
`kids_i_el_model1_odonnell_ext` DOUBLE,
`kids_i_el_model3_odonnell_ext` DOUBLE,
`kids_r_el_model1_odonnell_ext` DOUBLE,
`kids_r_el_model3_odonnell_ext` DOUBLE,
`kids_u_el_model1_odonnell_ext` DOUBLE,
`kids_u_el_model3_odonnell_ext` DOUBLE,
`lsst_g_el_model1_odonnell_ext` DOUBLE,
`lsst_g_el_model3_odonnell_ext` DOUBLE,
`lsst_i_el_model1_odonnell_ext` DOUBLE,
`lsst_i_el_model3_odonnell_ext` DOUBLE,
`lsst_r_el_model1_odonnell_ext` DOUBLE,
`lsst_r_el_model3_odonnell_ext` DOUBLE,
`lsst_u_el_model1_odonnell_ext` DOUBLE,
`lsst_u_el_model3_odonnell_ext` DOUBLE,
`lsst_y_el_model1_odonnell_ext` DOUBLE,
`lsst_y_el_model3_odonnell_ext` DOUBLE,
`lsst_z_el_model1_odonnell_ext` DOUBLE,
`lsst_z_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model3_odonnell_ext` DOUBLE,
`2mass_h_el_model1_odonnell_ext` DOUBLE,
`2mass_h_el_model3_odonnell_ext` DOUBLE,
`2mass_j_el_model1_odonnell_ext` DOUBLE,
`2mass_j_el_model3_odonnell_ext` DOUBLE,
`2mass_ks_el_model1_odonnell_ext` DOUBLE,
`2mass_ks_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`gaia_bp_el_model1_odonnell_ext` DOUBLE,
`gaia_bp_el_model3_odonnell_ext` DOUBLE,
`gaia_g_el_model1_odonnell_ext` DOUBLE,
`gaia_g_el_model3_odonnell_ext` DOUBLE,
`gaia_rp_el_model1_odonnell_ext` DOUBLE,
`gaia_rp_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model3_odonnell_ext` DOUBLE,
`euclid_vis_el_model1_odonnell_ext` DOUBLE,
`euclid_vis_el_model3_odonnell_ext` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
STORED AS PARQUET
LOCATION '/user/jcarrete/data/euclid/flagship_mock_1_9_9_swf1'
;

----
SWF2: /user/jcarrete/data/euclid/flagship_mock_1_9_9_swf2

SWF2
obsrequestid: 1717
213.074616 251.75015 72.188664 82.03694
En el primer octante:
213.074616-180 251.75015-180 72.188664 82.03694
33.074616 71.75015 72.188664 82.0369
No cojo ni las shapes ni los flujos
RA + 180, DEC no lo toco

halo_sql = """
SELECT halo_id, galaxy_id, kind, random_index,
ra_gal + 180 as ra_gal, dec_gal as dec_gal,
x_gal, y_gal, z_gal, r_gal, true_redshift_gal, observed_redshift_gal,
hpix_29_nest, vx_gal, vy_gal, vz_gal, vrad_gal, delta_r, abs_mag_r01,
luminosity_r01, abs_mag_r01_evolved, sdss_r01, abs_mag_uv_dereddened, gr_restframe,
color_kind, sed_ke, gr_cosmos, sed_cosmos, is_within_cosmos, cosmos_distance,
ext_curve_cosmos, ebv_cosmos, log_ml_r01, log_stellar_mass, metallicity, log_sfr,
logf_halpha, logf_halpha_ext, halpha_scatter, loglum_halpha, loglum_halpha_ext,
logf_dummy, z_dummy, logf_halpha_model1_ext, logf_halpha_model1, loglum_halpha_model1_ext,
loglum_halpha_model1, logf_halpha_model3_ext, logf_halpha_model3, loglum_halpha_model3_ext,
loglum_halpha_model3, logf_hbeta_model1_ext, logf_hbeta_model1, logf_o2_model1_ext,
logf_o2_model1, logf_n2_model1_ext, logf_n2_model1, logf_o3_model1_ext, logf_o3_model1,
logf_s2_model1_ext, logf_s2_model1, logf_hbeta_model3_ext, logf_hbeta_model3,
logf_o2_model3_ext, logf_o2_model3, logf_n2_model3_ext, logf_n2_model3, logf_o3_model3_ext,
logf_o3_model3, logf_s2_model3_ext, logf_s2_model3,
halo_x, halo_y, halo_z, halo_vx, halo_vy, halo_vz, halo_r, true_redshift_halo,
halo_lm, halo_n_sats, n_gals, luminosity_r01_evolved, hpix_13_nest, step
FROM jcarrete.flagship_mock_1_9_8_swf1_swf2_deep_c2
WHERE ra_gal >= 33.074616
AND ra_gal <= 71.75015
AND dec_gal >= 72.188664
AND dec_gal <= 82.0369
"""

/software/astro/scripts/create_table_from_parquet.sh -p /hdfs/user/jcarrete/data/euclid/flagship_mock_1_9_9_swf2/0.pq -t jcarrete.flagship_mock_1_9_9_swf2_pq -e

CREATE EXTERNAL TABLE jcarrete.flagship_mock_1_9_9_swf2_pq (
`kind` BIGINT,
`random_index` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`observed_redshift_gal` DOUBLE,
`hpix_29_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`abs_mag_r01` DOUBLE,
`luminosity_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`sdss_r01` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` BIGINT,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha` DOUBLE,
`logf_halpha_ext` DOUBLE,
`halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_halpha_model3_ext` DOUBLE,
`loglum_halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`luminosity_r01_evolved` DOUBLE,
`hpix_13_nest` BIGINT,
`step` BIGINT,
`blanco_decam_g` DOUBLE,
`blanco_decam_i` DOUBLE,
`blanco_decam_r` DOUBLE,
`blanco_decam_z` DOUBLE,
`subaru_hsc_z` DOUBLE,
`jst_jpcam_g` DOUBLE,
`kids_g` DOUBLE,
`kids_i` DOUBLE,
`kids_r` DOUBLE,
`kids_u` DOUBLE,
`lsst_g` DOUBLE,
`lsst_i` DOUBLE,
`lsst_r` DOUBLE,
`lsst_u` DOUBLE,
`lsst_y` DOUBLE,
`lsst_z` DOUBLE,
`cfht_megacam_r` DOUBLE,
`cfht_megacam_u` DOUBLE,
`pan_starrs_i` DOUBLE,
`pan_starrs_z` DOUBLE,
`2mass_h` DOUBLE,
`2mass_j` DOUBLE,
`2mass_ks` DOUBLE,
`gaia_bp` DOUBLE,
`gaia_g` DOUBLE,
`gaia_rp` DOUBLE,
`euclid_nisp_h` DOUBLE,
`euclid_nisp_j` DOUBLE,
`euclid_nisp_y` DOUBLE,
`euclid_vis` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`2mass_h_odonnell_ext` DOUBLE,
`2mass_j_odonnell_ext` DOUBLE,
`2mass_ks_odonnell_ext` DOUBLE,
`blanco_decam_g_odonnell_ext` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`blanco_decam_r_odonnell_ext` DOUBLE,
`blanco_decam_z_odonnell_ext` DOUBLE,
`cfht_megacam_r_odonnell_ext` DOUBLE,
`cfht_megacam_u_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`euclid_nisp_j_odonnell_ext` DOUBLE,
`euclid_nisp_y_odonnell_ext` DOUBLE,
`euclid_vis_odonnell_ext` DOUBLE,
`gaia_bp_odonnell_ext` DOUBLE,
`gaia_g_odonnell_ext` DOUBLE,
`gaia_rp_odonnell_ext` DOUBLE,
`jst_jpcam_g_odonnell_ext` DOUBLE,
`kids_g_odonnell_ext` DOUBLE,
`kids_i_odonnell_ext` DOUBLE,
`kids_r_odonnell_ext` DOUBLE,
`kids_u_odonnell_ext` DOUBLE,
`lsst_g_odonnell_ext` DOUBLE,
`lsst_i_odonnell_ext` DOUBLE,
`lsst_r_odonnell_ext` DOUBLE,
`lsst_u_odonnell_ext` DOUBLE,
`lsst_y_odonnell_ext` DOUBLE,
`lsst_z_odonnell_ext` DOUBLE,
`pan_starrs_i_odonnell_ext` DOUBLE,
`pan_starrs_z_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`subaru_hsc_z_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model3_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model1_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model3_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model1_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model3_odonnell_ext` DOUBLE,
`kids_g_el_model1_odonnell_ext` DOUBLE,
`kids_g_el_model3_odonnell_ext` DOUBLE,
`kids_i_el_model1_odonnell_ext` DOUBLE,
`kids_i_el_model3_odonnell_ext` DOUBLE,
`kids_r_el_model1_odonnell_ext` DOUBLE,
`kids_r_el_model3_odonnell_ext` DOUBLE,
`kids_u_el_model1_odonnell_ext` DOUBLE,
`kids_u_el_model3_odonnell_ext` DOUBLE,
`lsst_g_el_model1_odonnell_ext` DOUBLE,
`lsst_g_el_model3_odonnell_ext` DOUBLE,
`lsst_i_el_model1_odonnell_ext` DOUBLE,
`lsst_i_el_model3_odonnell_ext` DOUBLE,
`lsst_r_el_model1_odonnell_ext` DOUBLE,
`lsst_r_el_model3_odonnell_ext` DOUBLE,
`lsst_u_el_model1_odonnell_ext` DOUBLE,
`lsst_u_el_model3_odonnell_ext` DOUBLE,
`lsst_y_el_model1_odonnell_ext` DOUBLE,
`lsst_y_el_model3_odonnell_ext` DOUBLE,
`lsst_z_el_model1_odonnell_ext` DOUBLE,
`lsst_z_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model3_odonnell_ext` DOUBLE,
`2mass_h_el_model1_odonnell_ext` DOUBLE,
`2mass_h_el_model3_odonnell_ext` DOUBLE,
`2mass_j_el_model1_odonnell_ext` DOUBLE,
`2mass_j_el_model3_odonnell_ext` DOUBLE,
`2mass_ks_el_model1_odonnell_ext` DOUBLE,
`2mass_ks_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`gaia_bp_el_model1_odonnell_ext` DOUBLE,
`gaia_bp_el_model3_odonnell_ext` DOUBLE,
`gaia_g_el_model1_odonnell_ext` DOUBLE,
`gaia_g_el_model3_odonnell_ext` DOUBLE,
`gaia_rp_el_model1_odonnell_ext` DOUBLE,
`gaia_rp_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model3_odonnell_ext` DOUBLE,
`euclid_vis_el_model1_odonnell_ext` DOUBLE,
`euclid_vis_el_model3_odonnell_ext` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
STORED AS PARQUET
LOCATION '/user/jcarrete/data/euclid/flagship_mock_1_9_9_swf2'
;

----

DEEP: /user/jcarrete/data/euclid/flagship_mock_1_9_9_deep

DEEP
obsrequestid: self_calib_field
321.985796 328.956117 50.480548 54.70978
En el primer octante:
321.985796-270 328.956117-270 50.480548 54.70978
51.985796 58.956117 50.480548 54.70978
No cojo ni las shapes ni los flujos
RA + 270, DEC no lo toco

halo_sql = """
SELECT halo_id, galaxy_id, kind, random_index,
ra_gal + 270 as ra_gal, dec_gal as dec_gal,
x_gal, y_gal, z_gal, r_gal, true_redshift_gal, observed_redshift_gal,
hpix_29_nest, vx_gal, vy_gal, vz_gal, vrad_gal, delta_r, abs_mag_r01,
luminosity_r01, abs_mag_r01_evolved, sdss_r01, abs_mag_uv_dereddened, gr_restframe,
color_kind, sed_ke, gr_cosmos, sed_cosmos, is_within_cosmos, cosmos_distance,
ext_curve_cosmos, ebv_cosmos, log_ml_r01, log_stellar_mass, metallicity, log_sfr,
logf_halpha, logf_halpha_ext, halpha_scatter, loglum_halpha, loglum_halpha_ext,
logf_dummy, z_dummy, logf_halpha_model1_ext, logf_halpha_model1, loglum_halpha_model1_ext,
loglum_halpha_model1, logf_halpha_model3_ext, logf_halpha_model3, loglum_halpha_model3_ext,
loglum_halpha_model3, logf_hbeta_model1_ext, logf_hbeta_model1, logf_o2_model1_ext,
logf_o2_model1, logf_n2_model1_ext, logf_n2_model1, logf_o3_model1_ext, logf_o3_model1,
logf_s2_model1_ext, logf_s2_model1, logf_hbeta_model3_ext, logf_hbeta_model3,
logf_o2_model3_ext, logf_o2_model3, logf_n2_model3_ext, logf_n2_model3, logf_o3_model3_ext,
logf_o3_model3, logf_s2_model3_ext, logf_s2_model3,
halo_x, halo_y, halo_z, halo_vx, halo_vy, halo_vz, halo_r, true_redshift_halo,
halo_lm, halo_n_sats, n_gals, luminosity_r01_evolved, hpix_13_nest, step
FROM jcarrete.flagship_mock_1_9_8_swf1_swf2_deep_c2
WHERE ra_gal >= 51.985796
AND ra_gal <= 58.956117
AND dec_gal >= 50.480548
AND dec_gal <= 54.70978
"""

----

Genero la tabla externa en parquet:

/software/astro/scripts/create_table_from_parquet.sh -p /hdfs/user/jcarrete/data/euclid/flagship_mock_1_9_9_deep/0.pq -t jcarrete.flagship_mock_1_9_9_deep_pq -e

CREATE EXTERNAL TABLE jcarrete.flagship_mock_1_9_9_deep_pq (
`kind` BIGINT,
`random_index` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`observed_redshift_gal` DOUBLE,
`hpix_29_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`abs_mag_r01` DOUBLE,
`luminosity_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`sdss_r01` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` BIGINT,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha` DOUBLE,
`logf_halpha_ext` DOUBLE,
`halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_halpha_model3_ext` DOUBLE,
`loglum_halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`luminosity_r01_evolved` DOUBLE,
`hpix_13_nest` BIGINT,
`step` BIGINT,
`blanco_decam_g` DOUBLE,
`blanco_decam_i` DOUBLE,
`blanco_decam_r` DOUBLE,
`blanco_decam_z` DOUBLE,
`subaru_hsc_z` DOUBLE,
`jst_jpcam_g` DOUBLE,
`kids_g` DOUBLE,
`kids_i` DOUBLE,
`kids_r` DOUBLE,
`kids_u` DOUBLE,
`lsst_g` DOUBLE,
`lsst_i` DOUBLE,
`lsst_r` DOUBLE,
`lsst_u` DOUBLE,
`lsst_y` DOUBLE,
`lsst_z` DOUBLE,
`cfht_megacam_r` DOUBLE,
`cfht_megacam_u` DOUBLE,
`pan_starrs_i` DOUBLE,
`pan_starrs_z` DOUBLE,
`2mass_h` DOUBLE,
`2mass_j` DOUBLE,
`2mass_ks` DOUBLE,
`gaia_bp` DOUBLE,
`gaia_g` DOUBLE,
`gaia_rp` DOUBLE,
`euclid_nisp_h` DOUBLE,
`euclid_nisp_j` DOUBLE,
`euclid_nisp_y` DOUBLE,
`euclid_vis` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`2mass_h_odonnell_ext` DOUBLE,
`2mass_j_odonnell_ext` DOUBLE,
`2mass_ks_odonnell_ext` DOUBLE,
`blanco_decam_g_odonnell_ext` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`blanco_decam_r_odonnell_ext` DOUBLE,
`blanco_decam_z_odonnell_ext` DOUBLE,
`cfht_megacam_r_odonnell_ext` DOUBLE,
`cfht_megacam_u_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`euclid_nisp_j_odonnell_ext` DOUBLE,
`euclid_nisp_y_odonnell_ext` DOUBLE,
`euclid_vis_odonnell_ext` DOUBLE,
`gaia_bp_odonnell_ext` DOUBLE,
`gaia_g_odonnell_ext` DOUBLE,
`gaia_rp_odonnell_ext` DOUBLE,
`jst_jpcam_g_odonnell_ext` DOUBLE,
`kids_g_odonnell_ext` DOUBLE,
`kids_i_odonnell_ext` DOUBLE,
`kids_r_odonnell_ext` DOUBLE,
`kids_u_odonnell_ext` DOUBLE,
`lsst_g_odonnell_ext` DOUBLE,
`lsst_i_odonnell_ext` DOUBLE,
`lsst_r_odonnell_ext` DOUBLE,
`lsst_u_odonnell_ext` DOUBLE,
`lsst_y_odonnell_ext` DOUBLE,
`lsst_z_odonnell_ext` DOUBLE,
`pan_starrs_i_odonnell_ext` DOUBLE,
`pan_starrs_z_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`subaru_hsc_z_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model3_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model1_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model3_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model1_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model3_odonnell_ext` DOUBLE,
`kids_g_el_model1_odonnell_ext` DOUBLE,
`kids_g_el_model3_odonnell_ext` DOUBLE,
`kids_i_el_model1_odonnell_ext` DOUBLE,
`kids_i_el_model3_odonnell_ext` DOUBLE,
`kids_r_el_model1_odonnell_ext` DOUBLE,
`kids_r_el_model3_odonnell_ext` DOUBLE,
`kids_u_el_model1_odonnell_ext` DOUBLE,
`kids_u_el_model3_odonnell_ext` DOUBLE,
`lsst_g_el_model1_odonnell_ext` DOUBLE,
`lsst_g_el_model3_odonnell_ext` DOUBLE,
`lsst_i_el_model1_odonnell_ext` DOUBLE,
`lsst_i_el_model3_odonnell_ext` DOUBLE,
`lsst_r_el_model1_odonnell_ext` DOUBLE,
`lsst_r_el_model3_odonnell_ext` DOUBLE,
`lsst_u_el_model1_odonnell_ext` DOUBLE,
`lsst_u_el_model3_odonnell_ext` DOUBLE,
`lsst_y_el_model1_odonnell_ext` DOUBLE,
`lsst_y_el_model3_odonnell_ext` DOUBLE,
`lsst_z_el_model1_odonnell_ext` DOUBLE,
`lsst_z_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model3_odonnell_ext` DOUBLE,
`2mass_h_el_model1_odonnell_ext` DOUBLE,
`2mass_h_el_model3_odonnell_ext` DOUBLE,
`2mass_j_el_model1_odonnell_ext` DOUBLE,
`2mass_j_el_model3_odonnell_ext` DOUBLE,
`2mass_ks_el_model1_odonnell_ext` DOUBLE,
`2mass_ks_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`gaia_bp_el_model1_odonnell_ext` DOUBLE,
`gaia_bp_el_model3_odonnell_ext` DOUBLE,
`gaia_g_el_model1_odonnell_ext` DOUBLE,
`gaia_g_el_model3_odonnell_ext` DOUBLE,
`gaia_rp_el_model1_odonnell_ext` DOUBLE,
`gaia_rp_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model3_odonnell_ext` DOUBLE,
`euclid_vis_el_model1_odonnell_ext` DOUBLE,
`euclid_vis_el_model3_odonnell_ext` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
STORED AS PARQUET
LOCATION '/user/jcarrete/data/euclid/flagship_mock_1_9_9_deep'
;

----

Ahora le añado las shapes.
Lo hago usando la pipeline y en la query llamo a las 3 tablas a la vez!

/user/jcarrete/data/euclid/flagship_mock_1_9_10_swf1_swf2_deep/

galaxy_sql = """
SELECT * FROM jcarrete.flagship_mock_1_9_9_swf1_pq
UNION ALL
SELECT * FROM jcarrete.flagship_mock_1_9_9_swf2_pq
UNION ALL
SELECT * FROM jcarrete.flagship_mock_1_9_9_deep_pq
"""

/software/astro/scripts/create_table_from_parquet.sh -p /hdfs/user/jcarrete/data/euclid/flagship_mock_1_9_10_swf1_swf2_deep/0.pq -t jcarrete.flagship_mock_1_9_10_swf1_swf2_deep_pq -e

CREATE EXTERNAL TABLE jcarrete.flagship_mock_1_9_10_swf1_swf2_deep_pq (
`kind` BIGINT,
`random_index` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`observed_redshift_gal` DOUBLE,
`hpix_29_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`abs_mag_r01` DOUBLE,
`luminosity_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`sdss_r01` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` BIGINT,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha` DOUBLE,
`logf_halpha_ext` DOUBLE,
`halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_halpha_model3_ext` DOUBLE,
`loglum_halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`luminosity_r01_evolved` DOUBLE,
`hpix_13_nest` BIGINT,
`step` BIGINT,
`blanco_decam_g` DOUBLE,
`blanco_decam_i` DOUBLE,
`blanco_decam_r` DOUBLE,
`blanco_decam_z` DOUBLE,
`subaru_hsc_z` DOUBLE,
`jst_jpcam_g` DOUBLE,
`kids_g` DOUBLE,
`kids_i` DOUBLE,
`kids_r` DOUBLE,
`kids_u` DOUBLE,
`lsst_g` DOUBLE,
`lsst_i` DOUBLE,
`lsst_r` DOUBLE,
`lsst_u` DOUBLE,
`lsst_y` DOUBLE,
`lsst_z` DOUBLE,
`cfht_megacam_r` DOUBLE,
`cfht_megacam_u` DOUBLE,
`pan_starrs_i` DOUBLE,
`pan_starrs_z` DOUBLE,
`2mass_h` DOUBLE,
`2mass_j` DOUBLE,
`2mass_ks` DOUBLE,
`gaia_bp` DOUBLE,
`gaia_g` DOUBLE,
`gaia_rp` DOUBLE,
`euclid_nisp_h` DOUBLE,
`euclid_nisp_j` DOUBLE,
`euclid_nisp_y` DOUBLE,
`euclid_vis` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`2mass_h_odonnell_ext` DOUBLE,
`2mass_j_odonnell_ext` DOUBLE,
`2mass_ks_odonnell_ext` DOUBLE,
`blanco_decam_g_odonnell_ext` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`blanco_decam_r_odonnell_ext` DOUBLE,
`blanco_decam_z_odonnell_ext` DOUBLE,
`cfht_megacam_r_odonnell_ext` DOUBLE,
`cfht_megacam_u_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`euclid_nisp_j_odonnell_ext` DOUBLE,
`euclid_nisp_y_odonnell_ext` DOUBLE,
`euclid_vis_odonnell_ext` DOUBLE,
`gaia_bp_odonnell_ext` DOUBLE,
`gaia_g_odonnell_ext` DOUBLE,
`gaia_rp_odonnell_ext` DOUBLE,
`jst_jpcam_g_odonnell_ext` DOUBLE,
`kids_g_odonnell_ext` DOUBLE,
`kids_i_odonnell_ext` DOUBLE,
`kids_r_odonnell_ext` DOUBLE,
`kids_u_odonnell_ext` DOUBLE,
`lsst_g_odonnell_ext` DOUBLE,
`lsst_i_odonnell_ext` DOUBLE,
`lsst_r_odonnell_ext` DOUBLE,
`lsst_u_odonnell_ext` DOUBLE,
`lsst_y_odonnell_ext` DOUBLE,
`lsst_z_odonnell_ext` DOUBLE,
`pan_starrs_i_odonnell_ext` DOUBLE,
`pan_starrs_z_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`subaru_hsc_z_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model3_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model1_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model3_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model1_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model3_odonnell_ext` DOUBLE,
`kids_g_el_model1_odonnell_ext` DOUBLE,
`kids_g_el_model3_odonnell_ext` DOUBLE,
`kids_i_el_model1_odonnell_ext` DOUBLE,
`kids_i_el_model3_odonnell_ext` DOUBLE,
`kids_r_el_model1_odonnell_ext` DOUBLE,
`kids_r_el_model3_odonnell_ext` DOUBLE,
`kids_u_el_model1_odonnell_ext` DOUBLE,
`kids_u_el_model3_odonnell_ext` DOUBLE,
`lsst_g_el_model1_odonnell_ext` DOUBLE,
`lsst_g_el_model3_odonnell_ext` DOUBLE,
`lsst_i_el_model1_odonnell_ext` DOUBLE,
`lsst_i_el_model3_odonnell_ext` DOUBLE,
`lsst_r_el_model1_odonnell_ext` DOUBLE,
`lsst_r_el_model3_odonnell_ext` DOUBLE,
`lsst_u_el_model1_odonnell_ext` DOUBLE,
`lsst_u_el_model3_odonnell_ext` DOUBLE,
`lsst_y_el_model1_odonnell_ext` DOUBLE,
`lsst_y_el_model3_odonnell_ext` DOUBLE,
`lsst_z_el_model1_odonnell_ext` DOUBLE,
`lsst_z_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model3_odonnell_ext` DOUBLE,
`2mass_h_el_model1_odonnell_ext` DOUBLE,
`2mass_h_el_model3_odonnell_ext` DOUBLE,
`2mass_j_el_model1_odonnell_ext` DOUBLE,
`2mass_j_el_model3_odonnell_ext` DOUBLE,
`2mass_ks_el_model1_odonnell_ext` DOUBLE,
`2mass_ks_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`gaia_bp_el_model1_odonnell_ext` DOUBLE,
`gaia_bp_el_model3_odonnell_ext` DOUBLE,
`gaia_g_el_model1_odonnell_ext` DOUBLE,
`gaia_g_el_model3_odonnell_ext` DOUBLE,
`gaia_rp_el_model1_odonnell_ext` DOUBLE,
`gaia_rp_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model3_odonnell_ext` DOUBLE,
`euclid_vis_el_model1_odonnell_ext` DOUBLE,
`euclid_vis_el_model3_odonnell_ext` DOUBLE,
`dominant_shape` BIGINT,
`bulge_angle` DOUBLE,
`disk_angle` DOUBLE,
`median_major_axis` DOUBLE,
`scale_length` DOUBLE,
`bulge_fraction` DOUBLE,
`disk_scalelength` DOUBLE,
`disk_nsersic` DOUBLE,
`disk_r50` DOUBLE,
`bulge_r50` DOUBLE,
`bulge_nsersic` DOUBLE,
`inclination_angle` DOUBLE,
`disk_axis_ratio` DOUBLE,
`bulge_axis_ratio` DOUBLE,
`disk_ellipticity` DOUBLE,
`bulge_ellipticity` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
STORED AS PARQUET
LOCATION '/user/jcarrete/data/euclid/flagship_mock_1_9_10_swf1_swf2_deep'
;

CREATE TABLE jcarrete.flagship_mock_1_9_10_swf1_swf2_deep_c2(
`halo_id` bigint COMMENT 'unique halo id coming from the Flagship dark matter halo catalog',
`galaxy_id` int COMMENT 'combined with the unique_halo_id they uniquely identify each galaxy',
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',
`random_index` float COMMENT 'random number [0 - 1) for subsampling',
`ra_gal` float COMMENT 'galaxy right ascension (degrees)',
`dec_gal` float COMMENT 'galaxy declination (degrees)',
`l_gal` float COMMENT 'galactic longitude (degrees)',
`b_gal` float COMMENT 'galactic latitude (degrees)',
`mw_extinction` float COMMENT 'EBV from file: HFI_CompMap_ThermalDustModel_2048_R1.20.fits',
`x_gal` float COMMENT 'galaxy comoving x coordinate (Mpc/h)',
`y_gal` float COMMENT 'galaxy comoving y coordinate (Mpc/h)',
`z_gal` float COMMENT 'galaxy comoving z coordinate (Mpc/h)',
`r_gal` float COMMENT 'galaxy comoving distance (Mpc/h)',
`true_redshift_gal` float COMMENT 'true galaxy redshift',
`observed_redshift_gal` float COMMENT 'observed galaxy redshift (including peculiar velocity)',
`hpix_29_nest` bigint COMMENT 'healpix pixel id using order = 29 in NESTED format',
`vx_gal` float COMMENT 'physical (peculiar) velocity in x-component (km/s)',
`vy_gal` float COMMENT 'physical (peculiar) velocity in y-component (km/s)',
`vz_gal` float COMMENT 'physical (peculiar) velocity in z-component (km/s)',
`vrad_gal` float COMMENT 'missing comment',
`delta_r` float COMMENT 'missing comment',
`abs_mag_r01` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h)',
`luminosity_r01` float COMMENT 'luminosity in the 0.1-r-band',
`abs_mag_r01_evolved` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h) including MICECATv2.0 magnitude evolution',
`sdss_r01` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1',
`abs_mag_uv_dereddened` float COMMENT 'absolute magnitude ultraviolet',
`gr_restframe` float COMMENT '(g-r) rest-frame at z=0',
`color_kind` tinyint COMMENT 'galaxy color kind: 0=red sequence 1=green valley 2=blue cloud',
`sed_ke` float COMMENT 'K-correction for each SED',
`gr_cosmos` float COMMENT 'interpolated COSMOS g-r color',
`sed_cosmos` float COMMENT 'interpolated basic COSMOS SED [0 30]',
`is_within_cosmos` boolean COMMENT 'missing comment',
`cosmos_distance` float COMMENT 'missing comment',
`ext_curve_cosmos` tinyint COMMENT 'COSMOS extinction curve [0-4]',
`ebv_cosmos` float COMMENT 'interpolated extinction strength for COSMOS extinction curve',
`log_ml_r01` float COMMENT 'logarithm of stellar_mass/luminosity ratio in the r01 SDSS band',
`log_stellar_mass` float COMMENT 'logarithm of the stellar mass (Msun/h)',
`metallicity` float COMMENT 'oxygen metallicity in units 12+logO/H',
`log_sfr` float COMMENT 'logarithm of star formation rate in (Msun/h)/year',
`logf_halpha` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: DOES NOT include extinction)',
`logf_halpha_ext` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: includes extinction)',
`halpha_scatter` float COMMENT 'missing comment',
`loglum_halpha` float COMMENT 'DEPRECATED logarithm of Halpha luminosity (erg/s/h^2) using empirical relation with UV magnitude (DOES NOT include extinction)',
`loglum_halpha_ext` float COMMENT 'missing comment',
`logf_dummy` float COMMENT 'missing comment',
`z_dummy` float COMMENT 'missing comment',
`logf_halpha_model1_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: includes extinction)',
`logf_halpha_model1` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: DOES NOT include extinction)',
`loglum_halpha_model1_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (includes extinction)',
`loglum_halpha_model1` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (DOES NOT include extinction)',
`logf_halpha_model3_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: includes extinction)',
`logf_halpha_model3` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: DOES NOT include extinction)',
`loglum_halpha_model3_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (includes extinction)',
`loglum_halpha_model3` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (DOES NOT include extinction)',
`logf_hbeta_model1_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_hbeta_model1` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_n2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_n2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o3_model1_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o3_model1` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_s2_model1_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_s2_model1` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_hbeta_model3_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_hbeta_model3` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_n2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_n2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o3_model3_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o3_model3` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_s2_model3_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_s2_model3` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',

`2mass_h` float COMMENT 'observed flux in 2MASS H-band (Euclid provided)',
`2mass_j` float COMMENT 'observed flux in 2MASS J-band (Euclid provided)',
`2mass_ks` float COMMENT 'observed flux in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g` float COMMENT 'observed flux in DES g-band (Euclid provided)',
`blanco_decam_i` float COMMENT 'observed flux in DES i-band (Euclid provided)',
`blanco_decam_r` float COMMENT 'observed flux in DES r-band (Euclid provided)',
`blanco_decam_z` float COMMENT 'observed flux in DES z-band (Euclid provided)',
`cfht_megacam_r` float COMMENT 'observed flux in CFHT r-band (Euclid provided)',
`cfht_megacam_u` float COMMENT 'observed flux in CFHT u-band (Euclid provided)',
`euclid_nisp_h` float COMMENT 'observed flux in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j` float COMMENT 'observed flux in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y` float COMMENT 'observed flux in Euclid NISP-Y band (Euclid provided)',
`euclid_vis` float COMMENT 'observed flux in Euclid VIS band (Euclid provided)',
`gaia_bp` float COMMENT 'observed flux in Gaia-BP band (Euclid provided)',
`gaia_g` float COMMENT 'observed flux in Gaia-G band (Euclid provided)',
`gaia_rp` float COMMENT 'observed flux in Gaia-RP band (Euclid provided)',
`jst_jpcam_g` float COMMENT 'observed flux in JST g-band (Euclid provided)',
`kids_g` float COMMENT 'observed flux in KIDS g-band (Euclid provided)',
`kids_i` float COMMENT 'observed flux in KIDS i-band (Euclid provided)',
`kids_r` float COMMENT 'observed flux in KIDS r-band (Euclid provided)',
`kids_u` float COMMENT 'observed flux in KIDS u-band (Euclid provided)',
`lsst_g` float COMMENT 'observed flux in LSST g-band (Euclid provided)',
`lsst_i` float COMMENT 'observed flux in LSST i-band (Euclid provided)',
`lsst_r` float COMMENT 'observed flux in LSST r-band (Euclid provided)',
`lsst_u` float COMMENT 'observed flux in LSST u-band (Euclid provided)',
`lsst_y` float COMMENT 'observed flux in LSST Y-band (Euclid provided)',
`lsst_z` float COMMENT 'observed flux in LSST z-band (Euclid provided)',
`pan_starrs_i` float COMMENT 'observed flux in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z` float COMMENT 'observed flux in Pan-STARRS z-band (Euclid provided)',
`subaru_hsc_z` float COMMENT 'observed flux in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS H-band (Euclid provided)',
`2mass_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in JST g-band (Euclid provided)',
`kids_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_odonnell_ext` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1 with MW extinction',
`subaru_hsc_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS Ks-band (Euclid provided)',
`2mass_ks_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES z-band (Euclid provided)',
`blanco_decam_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT u-band (Euclid provided)',
`cfht_megacam_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_nisp_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid VIS band (Euclid provided)',
`euclid_vis_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_bp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-RP band (Euclid provided)',
`gaia_rp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in JST g-band (Euclid provided)',
`jst_jpcam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in JST g-band (Euclid provided)',
`kids_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS g-band (Euclid provided)',
`kids_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS i-band (Euclid provided)',
`kids_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS r-band (Euclid provided)',
`kids_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS u-band (Euclid provided)',
`kids_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST g-band (Euclid provided)',
`lsst_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST i-band (Euclid provided)',
`lsst_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST r-band (Euclid provided)',
`lsst_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST u-band (Euclid provided)',
`lsst_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST z-band (Euclid provided)',
`lsst_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`pan_starrs_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in sdss r-band redshifted to z=0.1',
`sdss_r01_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in sdss r-band redshifted to z=0.1',
`subaru_hsc_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`subaru_hsc_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',

`dominant_shape` tinyint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)',
`bulge_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`disk_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`median_major_axis` float COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)',
`scale_length` float COMMENT 'disc and bulge scalelength prior',
`bulge_fraction` float COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)',
`disk_scalelength` float COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)',
`disk_nsersic` float COMMENT 'Sersic index of the disk component',
`disk_r50` float COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678',
`bulge_r50` float COMMENT 'half light radius of the bulge',
`bulge_nsersic` float COMMENT 'Sersic index of the bulge component',
`disk_ellipticity` float COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)',
`bulge_ellipticity` float COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)',
`disk_axis_ratio` float COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`bulge_axis_ratio` float COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`inclination_angle` float COMMENT 'Galaxy inclination angle (where 0 degrees = face-on and 90 degrees = edge-on). Galaxy ellipticity for disk and bulge components are computed following the recipe in https://euclid.roe.ac.uk/projects/sgsshear/wiki/SHE-SIM',
`halo_x` float COMMENT 'halo comoving x coordinate (Mpc/h)',
`halo_y` float COMMENT 'halo comoving y coordinate (Mpc/h)',
`halo_z` float COMMENT 'halo comoving z coordinate (Mpc/h)',
`halo_vx` float COMMENT 'halo x-centre velocity (km/h)',
`halo_vy` float COMMENT 'halo y-centre velocity (km/h)',
`halo_vz` float COMMENT 'halo z-centre velocity (km/h)',
`halo_r` float COMMENT 'halo comoving distance (Mpc/h)',
`true_redshift_halo` float COMMENT 'true redshift of the host halo',
`halo_lm` float COMMENT 'log10 of the FoF halo mass in Msum/h. This halo mass is computed as the particule mass multiplied by the number of particles of the FoF halo. At low masses it is interpolated to obtain a smooth distribution of halo masses',
`halo_n_sats` int COMMENT 'number of satellite galaxies from HOD',
`n_gals` int COMMENT 'DEPRECATED',
`luminosity_r01_evolved` float COMMENT 'missing comment',
`hpix_13_nest` bigint COMMENT 'missing comment')
PARTITIONED BY (
`step` smallint COMMENT 'step number in the Flagship lensing maps')
CLUSTERED BY (
hpix_13_nest)
SORTED BY (
hpix_13_nest ASC)
INTO 128 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;

----

INSERT OVERWRITE TABLE jcarrete.flagship_mock_1_9_10_swf1_swf2_deep_c2 PARTITION(step)
SELECT halo_id, galaxy_id, kind, random_index, ra_gal, dec_gal, l_gal, b_gal, mw_extinction, x_gal, y_gal, z_gal, r_gal, true_redshift_gal, observed_redshift_gal, hpix_29_nest, vx_gal, vy_gal, vz_gal, vrad_gal, delta_r, abs_mag_r01, luminosity_r01, abs_mag_r01_evolved, sdss_r01, abs_mag_uv_dereddened, gr_restframe, color_kind, sed_ke, gr_cosmos, sed_cosmos, is_within_cosmos, cosmos_distance, ext_curve_cosmos, ebv_cosmos, log_ml_r01, log_stellar_mass, metallicity, log_sfr, logf_halpha, logf_halpha_ext, halpha_scatter, loglum_halpha, loglum_halpha_ext, logf_dummy, z_dummy, logf_halpha_model1_ext, logf_halpha_model1, loglum_halpha_model1_ext, loglum_halpha_model1, logf_halpha_model3_ext, logf_halpha_model3, loglum_halpha_model3_ext, loglum_halpha_model3, logf_hbeta_model1_ext, logf_hbeta_model1, logf_o2_model1_ext, logf_o2_model1, logf_n2_model1_ext, logf_n2_model1, logf_o3_model1_ext, logf_o3_model1, logf_s2_model1_ext, logf_s2_model1, logf_hbeta_model3_ext, logf_hbeta_model3, logf_o2_model3_ext, logf_o2_model3, logf_n2_model3_ext, logf_n2_model3, logf_o3_model3_ext, logf_o3_model3, logf_s2_model3_ext, logf_s2_model3, 2mass_h, 2mass_j, 2mass_ks, blanco_decam_g, blanco_decam_i, blanco_decam_r, blanco_decam_z, cfht_megacam_r, cfht_megacam_u, euclid_nisp_h, euclid_nisp_j, euclid_nisp_y, euclid_vis, gaia_bp, gaia_g, gaia_rp, jst_jpcam_g, kids_g, kids_i, kids_r, kids_u, lsst_g, lsst_i, lsst_r, lsst_u, lsst_y, lsst_z, pan_starrs_i, pan_starrs_z, subaru_hsc_z, 2mass_h_odonnell_ext, 2mass_j_odonnell_ext, 2mass_ks_odonnell_ext, blanco_decam_g_odonnell_ext, blanco_decam_i_odonnell_ext, blanco_decam_r_odonnell_ext, blanco_decam_z_odonnell_ext, cfht_megacam_r_odonnell_ext, cfht_megacam_u_odonnell_ext, euclid_nisp_h_odonnell_ext, euclid_nisp_j_odonnell_ext, euclid_nisp_y_odonnell_ext, euclid_vis_odonnell_ext, gaia_bp_odonnell_ext, gaia_g_odonnell_ext, gaia_rp_odonnell_ext, jst_jpcam_g_odonnell_ext, kids_g_odonnell_ext, kids_i_odonnell_ext, kids_r_odonnell_ext, kids_u_odonnell_ext, lsst_g_odonnell_ext, lsst_i_odonnell_ext, lsst_r_odonnell_ext, lsst_u_odonnell_ext, lsst_y_odonnell_ext, lsst_z_odonnell_ext, pan_starrs_i_odonnell_ext, pan_starrs_z_odonnell_ext, sdss_r01_odonnell_ext, subaru_hsc_z_odonnell_ext, 2mass_h_el_model1_odonnell_ext, 2mass_h_el_model3_odonnell_ext, 2mass_j_el_model1_odonnell_ext, 2mass_j_el_model3_odonnell_ext, 2mass_ks_el_model1_odonnell_ext, 2mass_ks_el_model3_odonnell_ext, blanco_decam_g_el_model1_odonnell_ext, blanco_decam_g_el_model3_odonnell_ext, blanco_decam_i_el_model1_odonnell_ext, blanco_decam_i_el_model3_odonnell_ext, blanco_decam_r_el_model1_odonnell_ext, blanco_decam_r_el_model3_odonnell_ext, blanco_decam_z_el_model1_odonnell_ext, blanco_decam_z_el_model3_odonnell_ext, cfht_megacam_r_el_model1_odonnell_ext, cfht_megacam_r_el_model3_odonnell_ext, cfht_megacam_u_el_model1_odonnell_ext, cfht_megacam_u_el_model3_odonnell_ext, euclid_nisp_h_el_model1_odonnell_ext, euclid_nisp_h_el_model3_odonnell_ext, euclid_nisp_j_el_model1_odonnell_ext, euclid_nisp_j_el_model3_odonnell_ext, euclid_nisp_y_el_model1_odonnell_ext, euclid_nisp_y_el_model3_odonnell_ext, euclid_vis_el_model1_odonnell_ext, euclid_vis_el_model3_odonnell_ext, gaia_bp_el_model1_odonnell_ext, gaia_bp_el_model3_odonnell_ext, gaia_g_el_model1_odonnell_ext, gaia_g_el_model3_odonnell_ext, gaia_rp_el_model1_odonnell_ext, gaia_rp_el_model3_odonnell_ext, jst_jpcam_g_el_model1_odonnell_ext, jst_jpcam_g_el_model3_odonnell_ext, kids_g_el_model1_odonnell_ext, kids_g_el_model3_odonnell_ext, kids_i_el_model1_odonnell_ext, kids_i_el_model3_odonnell_ext, kids_r_el_model1_odonnell_ext, kids_r_el_model3_odonnell_ext, kids_u_el_model1_odonnell_ext, kids_u_el_model3_odonnell_ext, lsst_g_el_model1_odonnell_ext, lsst_g_el_model3_odonnell_ext, lsst_i_el_model1_odonnell_ext, lsst_i_el_model3_odonnell_ext, lsst_r_el_model1_odonnell_ext, lsst_r_el_model3_odonnell_ext, lsst_u_el_model1_odonnell_ext, lsst_u_el_model3_odonnell_ext, lsst_y_el_model1_odonnell_ext, lsst_y_el_model3_odonnell_ext, lsst_z_el_model1_odonnell_ext, lsst_z_el_model3_odonnell_ext, pan_starrs_i_el_model1_odonnell_ext, pan_starrs_i_el_model3_odonnell_ext, pan_starrs_z_el_model1_odonnell_ext, pan_starrs_z_el_model3_odonnell_ext, sdss_r01_el_model1_odonnell_ext, sdss_r01_el_model3_odonnell_ext, subaru_hsc_z_el_model1_odonnell_ext, subaru_hsc_z_el_model3_odonnell_ext, dominant_shape, bulge_angle, disk_angle, median_major_axis, scale_length, bulge_fraction, disk_scalelength, disk_nsersic, disk_r50, bulge_r50, bulge_nsersic, disk_ellipticity, bulge_ellipticity, disk_axis_ratio, bulge_axis_ratio, inclination_angle, halo_x, halo_y, halo_z, halo_vx, halo_vy, halo_vz, halo_r, true_redshift_halo, halo_lm, halo_n_sats, n_gals, luminosity_r01_evolved, hpix_13_nest, step
FROM jcarrete.flagship_mock_1_9_10_swf1_swf2_deep_pq;

La paso a cosmohub para hacer algunas pruebas!

ALTER TABLE jcarrete.flagship_mock_1_9_10_swf1_swf2_deep_c2 RENAME TO cosmohub.flagship_mock_1_9_10_swf1_swf2_deep_c2;

ANALYZE TABLE cosmohub.flagship_mock_1_9_10_swf1_swf2_deep_c2 COMPUTE STATISTICS FOR COLUMNS;

----

LAS PRUEBAS DE LAS SHAPES EN EL CATALOGO parecen estar bien:

SELECT `halo_id`, `galaxy_id`, `ra_gal`, `dec_gal`, `true_redshift_gal`, `observed_redshift_gal`,
`kind`, `abs_mag_r01`, `gr_restframe`, `blanco_decam_i`, `bulge_angle`, `disk_angle`, `dominant_shape`,
`median_major_axis`, `scale_length`, `bulge_fraction`, `disk_scalelength`, `disk_nsersic`, `disk_r50`,
`bulge_r50`, `bulge_nsersic`, `disk_ellipticity`, `bulge_ellipticity`, `disk_axis_ratio`, `bulge_axis_ratio`,
`inclination_angle`
FROM cosmohub.flagship_mock_1_9_10_swf1_swf2_deep_c2
where -2.5*log10(blanco_decam_i) -48.6 < 26.5
and ra_gal between 215.5 and 219.5
and dec_gal between 74 and 78

Uso el notebook: shape_implementation_v20200811_to_check_1.9.7.ipynb

La tabla con los parámetros de lensing sería de la siguiente manera:

CREATE TABLE cosmohub.flagship_mock_1_9_10_swf1_swf2_deep_s(
`halo_id` bigint COMMENT 'unique halo id coming from the Flagship dark matter halo catalog',
`galaxy_id` int COMMENT 'combined with the unique_halo_id they uniquely identify each galaxy',
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',
`random_index` float COMMENT 'random number [0 - 1) for subsampling',
`ra_gal` float COMMENT 'galaxy right ascension (degrees)',
`dec_gal` float COMMENT 'galaxy declination (degrees)',
`ra_mag_gal` float COMMENT 'galaxy magnified right ascension (degree)',
`dec_mag_gal` float COMMENT 'galaxy magnified declination (degree)',
`kappa` float COMMENT 'convergence',
`gamma1` float COMMENT 'shear',
`gamma2` float COMMENT 'shear',
`l_gal` float COMMENT 'galactic longitude (degrees)',
`b_gal` float COMMENT 'galactic latitude (degrees)',
`mw_extinction` float COMMENT 'EBV from file: HFI_CompMap_ThermalDustModel_2048_R1.20.fits',
`x_gal` float COMMENT 'galaxy comoving x coordinate (Mpc/h)',
`y_gal` float COMMENT 'galaxy comoving y coordinate (Mpc/h)',
`z_gal` float COMMENT 'galaxy comoving z coordinate (Mpc/h)',
`r_gal` float COMMENT 'galaxy comoving distance (Mpc/h)',
`true_redshift_gal` float COMMENT 'true galaxy redshift',
`observed_redshift_gal` float COMMENT 'observed galaxy redshift (including peculiar velocity)',
`hpix_29_nest` bigint COMMENT 'healpix pixel id using order = 29 in NESTED format',
`vx_gal` float COMMENT 'physical (peculiar) velocity in x-component (km/s)',
`vy_gal` float COMMENT 'physical (peculiar) velocity in y-component (km/s)',
`vz_gal` float COMMENT 'physical (peculiar) velocity in z-component (km/s)',
`vrad_gal` float COMMENT 'missing comment',
`delta_r` float COMMENT 'missing comment',
`abs_mag_r01` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h)',
`luminosity_r01` float COMMENT 'luminosity in the 0.1-r-band',
`abs_mag_r01_evolved` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h) including MICECATv2.0 magnitude evolution',
`sdss_r01` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1',
`abs_mag_uv_dereddened` float COMMENT 'absolute magnitude ultraviolet',
`gr_restframe` float COMMENT '(g-r) rest-frame at z=0',
`color_kind` tinyint COMMENT 'galaxy color kind: 0=red sequence 1=green valley 2=blue cloud',
`sed_ke` float COMMENT 'K-correction for each SED',
`gr_cosmos` float COMMENT 'interpolated COSMOS g-r color',
`sed_cosmos` float COMMENT 'interpolated basic COSMOS SED [0 30]',
`is_within_cosmos` boolean COMMENT 'missing comment',
`cosmos_distance` float COMMENT 'missing comment',
`ext_curve_cosmos` tinyint COMMENT 'COSMOS extinction curve [0-4]',
`ebv_cosmos` float COMMENT 'interpolated extinction strength for COSMOS extinction curve',
`log_ml_r01` float COMMENT 'logarithm of stellar_mass/luminosity ratio in the r01 SDSS band',
`log_stellar_mass` float COMMENT 'logarithm of the stellar mass (Msun/h)',
`metallicity` float COMMENT 'oxygen metallicity in units 12+logO/H',
`log_sfr` float COMMENT 'logarithm of star formation rate in (Msun/h)/year',
`logf_halpha` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: DOES NOT include extinction)',
`logf_halpha_ext` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: includes extinction)',
`halpha_scatter` float COMMENT 'missing comment',
`loglum_halpha` float COMMENT 'DEPRECATED logarithm of Halpha luminosity (erg/s/h^2) using empirical relation with UV magnitude (DOES NOT include extinction)',
`loglum_halpha_ext` float COMMENT 'missing comment',
`logf_dummy` float COMMENT 'missing comment',
`z_dummy` float COMMENT 'missing comment',
`logf_halpha_model1_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: includes extinction)',
`logf_halpha_model1` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: DOES NOT include extinction)',
`loglum_halpha_model1_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (includes extinction)',
`loglum_halpha_model1` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (DOES NOT include extinction)',
`logf_halpha_model3_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: includes extinction)',
`logf_halpha_model3` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: DOES NOT include extinction)',
`loglum_halpha_model3_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (includes extinction)',
`loglum_halpha_model3` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (DOES NOT include extinction)',
`logf_hbeta_model1_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_hbeta_model1` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_n2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_n2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o3_model1_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o3_model1` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_s2_model1_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_s2_model1` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_hbeta_model3_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_hbeta_model3` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_n2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_n2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o3_model3_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o3_model3` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_s2_model3_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_s2_model3` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`2mass_h` float COMMENT 'observed flux in 2MASS H-band (Euclid provided)',
`2mass_j` float COMMENT 'observed flux in 2MASS J-band (Euclid provided)',
`2mass_ks` float COMMENT 'observed flux in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g` float COMMENT 'observed flux in DES g-band (Euclid provided)',
`blanco_decam_i` float COMMENT 'observed flux in DES i-band (Euclid provided)',
`blanco_decam_r` float COMMENT 'observed flux in DES r-band (Euclid provided)',
`blanco_decam_z` float COMMENT 'observed flux in DES z-band (Euclid provided)',
`cfht_megacam_r` float COMMENT 'observed flux in CFHT r-band (Euclid provided)',
`cfht_megacam_u` float COMMENT 'observed flux in CFHT u-band (Euclid provided)',
`euclid_nisp_h` float COMMENT 'observed flux in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j` float COMMENT 'observed flux in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y` float COMMENT 'observed flux in Euclid NISP-Y band (Euclid provided)',
`euclid_vis` float COMMENT 'observed flux in Euclid VIS band (Euclid provided)',
`gaia_bp` float COMMENT 'observed flux in Gaia-BP band (Euclid provided)',
`gaia_g` float COMMENT 'observed flux in Gaia-G band (Euclid provided)',
`gaia_rp` float COMMENT 'observed flux in Gaia-RP band (Euclid provided)',
`jst_jpcam_g` float COMMENT 'observed flux in JST g-band (Euclid provided)',
`kids_g` float COMMENT 'observed flux in KIDS g-band (Euclid provided)',
`kids_i` float COMMENT 'observed flux in KIDS i-band (Euclid provided)',
`kids_r` float COMMENT 'observed flux in KIDS r-band (Euclid provided)',
`kids_u` float COMMENT 'observed flux in KIDS u-band (Euclid provided)',
`lsst_g` float COMMENT 'observed flux in LSST g-band (Euclid provided)',
`lsst_i` float COMMENT 'observed flux in LSST i-band (Euclid provided)',
`lsst_r` float COMMENT 'observed flux in LSST r-band (Euclid provided)',
`lsst_u` float COMMENT 'observed flux in LSST u-band (Euclid provided)',
`lsst_y` float COMMENT 'observed flux in LSST Y-band (Euclid provided)',
`lsst_z` float COMMENT 'observed flux in LSST z-band (Euclid provided)',
`pan_starrs_i` float COMMENT 'observed flux in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z` float COMMENT 'observed flux in Pan-STARRS z-band (Euclid provided)',
`subaru_hsc_z` float COMMENT 'observed flux in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS H-band (Euclid provided)',
`2mass_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in JST g-band (Euclid provided)',
`kids_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_odonnell_ext` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1 with MW extinction',
`subaru_hsc_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS Ks-band (Euclid provided)',
`2mass_ks_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES z-band (Euclid provided)',
`blanco_decam_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT u-band (Euclid provided)',
`cfht_megacam_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_nisp_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid VIS band (Euclid provided)',
`euclid_vis_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_bp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-RP band (Euclid provided)',
`gaia_rp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in JST g-band (Euclid provided)',
`jst_jpcam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in JST g-band (Euclid provided)',
`kids_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS g-band (Euclid provided)',
`kids_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS i-band (Euclid provided)',
`kids_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS r-band (Euclid provided)',
`kids_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS u-band (Euclid provided)',
`kids_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST g-band (Euclid provided)',
`lsst_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST i-band (Euclid provided)',
`lsst_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST r-band (Euclid provided)',
`lsst_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST u-band (Euclid provided)',
`lsst_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST z-band (Euclid provided)',
`lsst_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`pan_starrs_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in sdss r-band redshifted to z=0.1',
`sdss_r01_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in sdss r-band redshifted to z=0.1',
`subaru_hsc_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`subaru_hsc_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`dominant_shape` tinyint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)',
`bulge_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`disk_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`median_major_axis` float COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)',
`scale_length` float COMMENT 'disc and bulge scalelength prior',
`bulge_fraction` float COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)',
`disk_scalelength` float COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)',
`disk_nsersic` float COMMENT 'Sersic index of the disk component',
`disk_r50` float COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678',
`bulge_r50` float COMMENT 'half light radius of the bulge',
`bulge_nsersic` float COMMENT 'Sersic index of the bulge component',
`disk_ellipticity` float COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)',
`bulge_ellipticity` float COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)',
`disk_axis_ratio` float COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`bulge_axis_ratio` float COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`inclination_angle` float COMMENT 'Galaxy inclination angle (where 0 degrees = face-on and 90 degrees = edge-on). Galaxy ellipticity for disk and bulge components are computed following the recipe in https://euclid.roe.ac.uk/projects/sgsshear/wiki/SHE-SIM',
`halo_x` float COMMENT 'halo comoving x coordinate (Mpc/h)',
`halo_y` float COMMENT 'halo comoving y coordinate (Mpc/h)',
`halo_z` float COMMENT 'halo comoving z coordinate (Mpc/h)',
`halo_vx` float COMMENT 'halo x-centre velocity (km/h)',
`halo_vy` float COMMENT 'halo y-centre velocity (km/h)',
`halo_vz` float COMMENT 'halo z-centre velocity (km/h)',
`halo_r` float COMMENT 'halo comoving distance (Mpc/h)',
`true_redshift_halo` float COMMENT 'true redshift of the host halo',
`halo_lm` float COMMENT 'log10 of the FoF halo mass in Msum/h. This halo mass is computed as the particule mass multiplied by the number of particles of the FoF halo. At low masses it is interpolated to obtain a smooth distribution of halo masses',
`halo_n_sats` int COMMENT 'number of satellite galaxies from HOD',
`n_gals` int COMMENT 'DEPRECATED',
`luminosity_r01_evolved` float COMMENT 'missing comment',
`hpix_13_nest` bigint COMMENT 'missing comment'
)
PARTITIONED BY (
`step` smallint COMMENT 'step number in the Flagship lensing maps')
CLUSTERED BY (
hpix_13_nest)
SORTED BY (
hpix_13_nest ASC)
INTO 128 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;

----

HAGO EL CAMBIO EN EL SIGNO DE gamma1 CUANDO HAGO EL INSERT OVERWRITE USANDO UN IF:

IF(fs.dec_gal<0, lensing.gamma1, -lensing.gamma1)

Además hay que hacer el traslado en ra!

INSERT OVERWRITE TABLE cosmohub.flagship_mock_1_9_10_swf1_swf2_deep_s PARTITION(step)
SELECT fs.halo_id, fs.galaxy_id, fs.kind, fs.random_index, fs.ra_gal, fs.dec_gal,
CASE
WHEN fs.ra_gal > 320 THEN lensing.ra_mag_gal + 270
WHEN fs.ra_gal > 213 THEN lensing.ra_mag_gal + 180
ELSE lensing.ra_mag_gal
END,
IF(fs.dec_gal<0, -lensing.dec_mag_gal, lensing.dec_mag_gal) AS dec_mag_gal,
lensing.kappa,
IF(fs.dec_gal<0, lensing.gamma1, -lensing.gamma1) AS gamma1,
lensing.gamma2, fs.l_gal, fs.b_gal, fs.mw_extinction, fs.x_gal, fs.y_gal, fs.z_gal, fs.r_gal, fs.true_redshift_gal, fs.observed_redshift_gal, fs.hpix_29_nest, fs.vx_gal, fs.vy_gal, fs.vz_gal, fs.vrad_gal, fs.delta_r, fs.abs_mag_r01, fs.luminosity_r01, fs.abs_mag_r01_evolved, fs.sdss_r01, fs.abs_mag_uv_dereddened, fs.gr_restframe, fs.color_kind, fs.sed_ke, fs.gr_cosmos, fs.sed_cosmos, fs.is_within_cosmos, fs.cosmos_distance, fs.ext_curve_cosmos, fs.ebv_cosmos, fs.log_ml_r01, fs.log_stellar_mass, fs.metallicity, fs.log_sfr, fs.logf_halpha, fs.logf_halpha_ext, fs.halpha_scatter, fs.loglum_halpha, fs.loglum_halpha_ext, fs.logf_dummy, fs.z_dummy, fs.logf_halpha_model1_ext, fs.logf_halpha_model1, fs.loglum_halpha_model1_ext, fs.loglum_halpha_model1, fs.logf_halpha_model3_ext, fs.logf_halpha_model3, fs.loglum_halpha_model3_ext, fs.loglum_halpha_model3, fs.logf_hbeta_model1_ext, fs.logf_hbeta_model1, fs.logf_o2_model1_ext, fs.logf_o2_model1, fs.logf_n2_model1_ext, fs.logf_n2_model1, fs.logf_o3_model1_ext, fs.logf_o3_model1, fs.logf_s2_model1_ext, fs.logf_s2_model1, fs.logf_hbeta_model3_ext, fs.logf_hbeta_model3, fs.logf_o2_model3_ext, fs.logf_o2_model3, fs.logf_n2_model3_ext, fs.logf_n2_model3, fs.logf_o3_model3_ext, fs.logf_o3_model3, fs.logf_s2_model3_ext, fs.logf_s2_model3, fs.2mass_h, fs.2mass_j, fs.2mass_ks, fs.blanco_decam_g, fs.blanco_decam_i, fs.blanco_decam_r, fs.blanco_decam_z, fs.cfht_megacam_r, fs.cfht_megacam_u, fs.euclid_nisp_h, fs.euclid_nisp_j, fs.euclid_nisp_y, fs.euclid_vis, fs.gaia_bp, fs.gaia_g, fs.gaia_rp, fs.jst_jpcam_g, fs.kids_g, fs.kids_i, fs.kids_r, fs.kids_u, fs.lsst_g, fs.lsst_i, fs.lsst_r, fs.lsst_u, fs.lsst_y, fs.lsst_z, fs.pan_starrs_i, fs.pan_starrs_z, fs.subaru_hsc_z, fs.2mass_h_odonnell_ext, fs.2mass_j_odonnell_ext, fs.2mass_ks_odonnell_ext, fs.blanco_decam_g_odonnell_ext, fs.blanco_decam_i_odonnell_ext, fs.blanco_decam_r_odonnell_ext, fs.blanco_decam_z_odonnell_ext, fs.cfht_megacam_r_odonnell_ext, fs.cfht_megacam_u_odonnell_ext, fs.euclid_nisp_h_odonnell_ext, fs.euclid_nisp_j_odonnell_ext, fs.euclid_nisp_y_odonnell_ext, fs.euclid_vis_odonnell_ext, fs.gaia_bp_odonnell_ext, fs.gaia_g_odonnell_ext, fs.gaia_rp_odonnell_ext, fs.jst_jpcam_g_odonnell_ext, fs.kids_g_odonnell_ext, fs.kids_i_odonnell_ext, fs.kids_r_odonnell_ext, fs.kids_u_odonnell_ext, fs.lsst_g_odonnell_ext, fs.lsst_i_odonnell_ext, fs.lsst_r_odonnell_ext, fs.lsst_u_odonnell_ext, fs.lsst_y_odonnell_ext, fs.lsst_z_odonnell_ext, fs.pan_starrs_i_odonnell_ext, fs.pan_starrs_z_odonnell_ext, fs.sdss_r01_odonnell_ext, fs.subaru_hsc_z_odonnell_ext, fs.2mass_h_el_model1_odonnell_ext, fs.2mass_h_el_model3_odonnell_ext, fs.2mass_j_el_model1_odonnell_ext, fs.2mass_j_el_model3_odonnell_ext, fs.2mass_ks_el_model1_odonnell_ext, fs.2mass_ks_el_model3_odonnell_ext, fs.blanco_decam_g_el_model1_odonnell_ext, fs.blanco_decam_g_el_model3_odonnell_ext, fs.blanco_decam_i_el_model1_odonnell_ext, fs.blanco_decam_i_el_model3_odonnell_ext, fs.blanco_decam_r_el_model1_odonnell_ext, fs.blanco_decam_r_el_model3_odonnell_ext, fs.blanco_decam_z_el_model1_odonnell_ext, fs.blanco_decam_z_el_model3_odonnell_ext, fs.cfht_megacam_r_el_model1_odonnell_ext, fs.cfht_megacam_r_el_model3_odonnell_ext, fs.cfht_megacam_u_el_model1_odonnell_ext, fs.cfht_megacam_u_el_model3_odonnell_ext, fs.euclid_nisp_h_el_model1_odonnell_ext, fs.euclid_nisp_h_el_model3_odonnell_ext, fs.euclid_nisp_j_el_model1_odonnell_ext, fs.euclid_nisp_j_el_model3_odonnell_ext, fs.euclid_nisp_y_el_model1_odonnell_ext, fs.euclid_nisp_y_el_model3_odonnell_ext, fs.euclid_vis_el_model1_odonnell_ext, fs.euclid_vis_el_model3_odonnell_ext, fs.gaia_bp_el_model1_odonnell_ext, fs.gaia_bp_el_model3_odonnell_ext, fs.gaia_g_el_model1_odonnell_ext, fs.gaia_g_el_model3_odonnell_ext, fs.gaia_rp_el_model1_odonnell_ext, fs.gaia_rp_el_model3_odonnell_ext, fs.jst_jpcam_g_el_model1_odonnell_ext, fs.jst_jpcam_g_el_model3_odonnell_ext, fs.kids_g_el_model1_odonnell_ext, fs.kids_g_el_model3_odonnell_ext, fs.kids_i_el_model1_odonnell_ext, fs.kids_i_el_model3_odonnell_ext, fs.kids_r_el_model1_odonnell_ext, fs.kids_r_el_model3_odonnell_ext, fs.kids_u_el_model1_odonnell_ext, fs.kids_u_el_model3_odonnell_ext, fs.lsst_g_el_model1_odonnell_ext, fs.lsst_g_el_model3_odonnell_ext, fs.lsst_i_el_model1_odonnell_ext, fs.lsst_i_el_model3_odonnell_ext, fs.lsst_r_el_model1_odonnell_ext, fs.lsst_r_el_model3_odonnell_ext, fs.lsst_u_el_model1_odonnell_ext, fs.lsst_u_el_model3_odonnell_ext, fs.lsst_y_el_model1_odonnell_ext, fs.lsst_y_el_model3_odonnell_ext, fs.lsst_z_el_model1_odonnell_ext, fs.lsst_z_el_model3_odonnell_ext, fs.pan_starrs_i_el_model1_odonnell_ext, fs.pan_starrs_i_el_model3_odonnell_ext, fs.pan_starrs_z_el_model1_odonnell_ext, fs.pan_starrs_z_el_model3_odonnell_ext, fs.sdss_r01_el_model1_odonnell_ext, fs.sdss_r01_el_model3_odonnell_ext, fs.subaru_hsc_z_el_model1_odonnell_ext, fs.subaru_hsc_z_el_model3_odonnell_ext, fs.dominant_shape, fs.bulge_angle, fs.disk_angle, fs.median_major_axis, fs.scale_length, fs.bulge_fraction, fs.disk_scalelength, fs.disk_nsersic, fs.disk_r50, fs.bulge_r50, fs.bulge_nsersic, fs.disk_ellipticity, fs.bulge_ellipticity, fs.disk_axis_ratio, fs.bulge_axis_ratio, fs.inclination_angle, fs.halo_x, fs.halo_y, fs.halo_z, fs.halo_vx, fs.halo_vy, fs.halo_vz, fs.halo_r, fs.true_redshift_halo, fs.halo_lm, fs.halo_n_sats, fs.n_gals, fs.luminosity_r01_evolved, fs.hpix_13_nest, fs.step
FROM cosmohub.flagship_mock_1_9_10_swf1_swf2_deep_c2 as fs
JOIN jcarrete.flagship_mock_1_9_8_swf1_swf2_deep_s as lensing
ON fs.halo_id = lensing.halo_id AND fs.galaxy_id = lensing.galaxy_id
;

ANALYZE TABLE cosmohub.flagship_mock_1_9_10_swf1_swf2_deep_s PARTITION(step) COMPUTE STATISTICS FOR COLUMNS;

GENERO LA TABLA FITS:

CREATE TABLE jcarrete.flagship_mock_1_9_10_swf1_swf2_deep_s_magnified_fits(
`source_id` bigint COMMENT 'from deserializer',
`halo_id` bigint COMMENT 'from deserializer',
`ra` float COMMENT 'from deserializer',
`dec` float COMMENT 'from deserializer',
`ra_mag` float COMMENT 'from deserializer',
`dec_mag` float COMMENT 'from deserializer',
`z_obs` float COMMENT 'from deserializer',
`ref_mag_abs` float COMMENT 'from deserializer',
`ref_mag` float COMMENT 'from deserializer',
`bulge_fraction` float COMMENT 'from deserializer',
`bulge_r50` float COMMENT 'from deserializer',
`disk_r50` float COMMENT 'from deserializer',
`bulge_nsersic` float COMMENT 'from deserializer',
`inclination_angle` float COMMENT 'from deserializer',
`disk_angle` float COMMENT 'from deserializer',
`kappa` float COMMENT 'from deserializer',
`gamma1` float COMMENT 'from deserializer',
`gamma2` float COMMENT 'from deserializer',
`sed_template` float COMMENT 'from deserializer',
`ext_law` smallint COMMENT 'from deserializer',
`ebv` float COMMENT 'from deserializer',
`halpha_logflam_ext` float COMMENT 'from deserializer',
`hbeta_logflam_ext` float COMMENT 'from deserializer',
`o2_logflam_ext` float COMMENT 'from deserializer',
`o3_logflam_ext` float COMMENT 'from deserializer',
`n2_logflam_ext` float COMMENT 'from deserializer',
`s2_logflam_ext` float COMMENT 'from deserializer',
`av` float COMMENT 'from deserializer',
`tu_fnu_vis` float COMMENT 'from deserializer',
`tu_fnu_y_nisp` float COMMENT 'from deserializer',
`tu_fnu_j_nisp` float COMMENT 'from deserializer',
`tu_fnu_h_nisp` float COMMENT 'from deserializer',
`tu_fnu_g_decam` float COMMENT 'from deserializer',
`tu_fnu_r_decam` float COMMENT 'from deserializer',
`tu_fnu_i_decam` float COMMENT 'from deserializer',
`tu_fnu_z_decam` float COMMENT 'from deserializer',
`tu_fnu_u_megacam` float COMMENT 'from deserializer',
`tu_fnu_r_megacam` float COMMENT 'from deserializer',
`tu_fnu_g_jpcam` float COMMENT 'from deserializer',
`tu_fnu_i_panstarrs` float COMMENT 'from deserializer',
`tu_fnu_z_panstarrs` float COMMENT 'from deserializer',
`tu_fnu_z_hsc` float COMMENT 'from deserializer',
`tu_fnu_g_gaia` float COMMENT 'from deserializer',
`tu_fnu_bp_gaia` float COMMENT 'from deserializer',
`tu_fnu_rp_gaia` float COMMENT 'from deserializer',
`tu_fnu_u_lsst` float COMMENT 'from deserializer',
`tu_fnu_g_lsst` float COMMENT 'from deserializer',
`tu_fnu_r_lsst` float COMMENT 'from deserializer',
`tu_fnu_i_lsst` float COMMENT 'from deserializer',
`tu_fnu_z_lsst` float COMMENT 'from deserializer',
`tu_fnu_y_lsst` float COMMENT 'from deserializer',
`tu_fnu_u_kids` float COMMENT 'from deserializer',
`tu_fnu_g_kids` float COMMENT 'from deserializer',
`tu_fnu_r_kids` float COMMENT 'from deserializer',
`tu_fnu_i_kids` float COMMENT 'from deserializer',
`tu_fnu_j_2mass` float COMMENT 'from deserializer',
`tu_fnu_h_2mass` float COMMENT 'from deserializer',
`tu_fnu_ks_2mass` float COMMENT 'from deserializer')
PARTITIONED BY (
`hpix_5_nest` bigint)
ROW FORMAT SERDE
'es.pic.astro.hadoop.serde.RecArraySerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'es.pic.astro.hadoop.io.BinaryOutputFormat'
;

INSERT OVERWRITE TABLE jcarrete.flagship_mock_1_9_10_swf1_swf2_deep_s_magnified_fits PARTITION(hpix_5_nest)
SELECT CAST(((halo_id * 10000) + galaxy_id) AS bigint) AS SOURCE_ID,
CAST(halo_id AS bigint) AS HALO_ID,
CAST(ra_gal AS float) AS RA,
CAST(dec_gal AS float) AS `DEC`,
CAST(ra_mag_gal AS float) AS RA_MAG,
CAST(dec_mag_gal AS float) AS DEC_MAG,
CAST(observed_redshift_gal AS float) AS Z_OBS,
CAST(abs_mag_r01_evolved AS float) AS REF_MAG_ABS,
CAST(-2.5*log10(sdss_r01) - 48.6 AS float) AS REF_MAG,
CAST(bulge_fraction AS float) AS BULGE_FRACTION,
CAST(bulge_r50 AS float) AS BULGE_R50,
CAST(disk_r50 AS float) AS DISK_R50,
CAST(bulge_nsersic AS float) AS BULGE_NSERSIC,
CAST(inclination_angle AS float) AS INCLINATION_ANGLE,
CAST(disk_angle AS float) AS DISK_ANGLE,
CAST(kappa AS float) AS KAPPA,
CAST(gamma1 AS float) AS GAMMA1,
CAST(gamma2 AS float) AS GAMMA2,
CAST(sed_cosmos AS float) AS SED_TEMPLATE,
CAST(ROUND(ext_curve_cosmos) AS smallint) AS EXT_LAW,
CAST(ebv_cosmos AS float) AS EBV,
CAST(logf_halpha_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HALPHA_LOGFLAM_EXT,
CAST(logf_hbeta_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HBETA_LOGFLAM_EXT,
CAST(logf_o2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O2_LOGFLAM_EXT,
CAST(logf_o3_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O3_LOGFLAM_EXT,
CAST(logf_n2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS N2_LOGFLAM_EXT,
CAST(logf_s2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS S2_LOGFLAM_EXT,
CAST(mw_extinction AS float) AS AV,
CAST(euclid_vis_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_VIS,
CAST(euclid_nisp_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_NISP,
CAST(euclid_nisp_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_NISP,
CAST(euclid_nisp_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_NISP,
CAST(blanco_decam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_DECAM,
CAST(blanco_decam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_DECAM,
CAST(blanco_decam_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_DECAM,
CAST(blanco_decam_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_DECAM,
CAST(cfht_megacam_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_MEGACAM,
CAST(cfht_megacam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_MEGACAM,
CAST(jst_jpcam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_JPCAM,
CAST(pan_starrs_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_PANSTARRS,
CAST(pan_starrs_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_PANSTARRS,
CAST(subaru_hsc_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_HSC,
CAST(gaia_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_GAIA,
CAST(gaia_bp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_BP_GAIA,
CAST(gaia_rp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_RP_GAIA,
CAST(lsst_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_LSST,
CAST(lsst_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_LSST,
CAST(lsst_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_LSST,
CAST(lsst_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_LSST,
CAST(lsst_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_LSST,
CAST(lsst_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_LSST,
CAST(kids_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_KIDS,
CAST(kids_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_KIDS,
CAST(kids_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_KIDS,
CAST(kids_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_KIDS,
CAST(2mass_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_2MASS,
CAST(2mass_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_2MASS,
CAST(2mass_ks_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_KS_2MASS,
CAST(SHIFTRIGHT(hpix_29_nest, (29-5)*2) AS bigint) AS hpix_5_nest
FROM cosmohub.flagship_mock_1_9_10_swf1_swf2_deep_s
WHERE (logf_halpha_model3_ext > -16 OR -2.5*log10(euclid_nisp_h) - 48.6 < 26)
AND disk_axis_ratio > 0.10865
;

Los genero en una carpeta que se llama 1.9.10

Information for CosmoHub

Dark matter halo input catalog from flagship_rockstar_octant1_c, which is the Flagship dark matter halo catalog v1.1 with correct velocities.
There are three different regions to cover the current selected observations to simulate for GSIR, which are SWF1, SWF2 and the Deep field (see the following link for more details:[https://euclid.roe.ac.uk/issues/14134](https://euclid.roe.ac.uk/issues/14134)).
**Shape distributions are completely different from all previous versions (including new recipe for inclination_angle field)**.
In this version a new python method for inclination_angle was developed by Francisco and ellipticities and axis ratios come from Eric's formulas.
See the following [url](https://docs.google.com/document/d/1FRERcl5Lv2IDB6ZChTK6gcEHI2FWA9gSgYKcaDUA9EM/edit?usp=sharing) for more details.
Note that **we provide fluxes instead of magnitudes**. In order **to get magnitudes** you can enter in the "Expert mode" and estimate the magnitude as follows:
m = -2.5 * log10(flux) - 48.6
And in order to estimate magnified magnitudes:
m_mag = m + 2.5 * log10((1-kappa)*(1-kappa) - gamma1 * gamma1 - gamma2 * gamma2)

----

FITS file in CosmoHub:

{
"sql": "SELECT CAST(((halo_id * 10000) + galaxy_id) AS bigint) AS SOURCE_ID, \nCAST(halo_id AS bigint) AS HALO_ID, \nCAST(ra_gal AS float) AS RA, \nCAST(dec_gal AS float) AS `DEC`, \nCAST(ra_mag_gal AS float) AS RA_MAG, \nCAST(dec_mag_gal AS float) AS DEC_MAG, \nCAST(observed_redshift_gal AS float) AS Z_OBS, \nCAST(abs_mag_r01_evolved AS float) AS REF_MAG_ABS, \nCAST(-2.5*log10(sdss_r01) - 48.6 AS float) AS REF_MAG, \nCAST(bulge_fraction AS float) AS BULGE_FRACTION, \nCAST(bulge_r50 AS float) AS BULGE_R50, \nCAST(disk_r50 AS float) AS DISK_R50, \nCAST(bulge_nsersic AS float) AS BULGE_NSERSIC, \nCAST(inclination_angle AS float) AS INCLINATION_ANGLE, \nCAST(disk_angle AS float) AS DISK_ANGLE, \nCAST(kappa AS float) AS KAPPA, \nCAST(gamma1 AS float) AS GAMMA1, \nCAST(gamma2 AS float) AS GAMMA2, \nCAST(sed_cosmos AS float) AS SED_TEMPLATE, \nCAST(ROUND(ext_curve_cosmos) AS smallint) AS EXT_LAW, \nCAST(ebv_cosmos AS float) AS EBV, \nCAST(logf_halpha_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HALPHA_LOGFLAM_EXT_MAG, \nCAST(logf_hbeta_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HBETA_LOGFLAM_EXT_MAG, \nCAST(logf_o2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O2_LOGFLAM_EXT_MAG, \nCAST(logf_o3_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O3_LOGFLAM_EXT_MAG, \nCAST(logf_n2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS N2_LOGFLAM_EXT_MAG, \nCAST(logf_s2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS S2_LOGFLAM_EXT_MAG, \nCAST(mw_extinction AS float) AS AV,\nCAST(euclid_vis_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_VIS_MAG, \nCAST(euclid_nisp_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_NISP_MAG, \nCAST(euclid_nisp_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_NISP_MAG, \nCAST(euclid_nisp_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_NISP_MAG, \nCAST(blanco_decam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_DECAM_MAG, \nCAST(blanco_decam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_DECAM_MAG, \nCAST(blanco_decam_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_DECAM_MAG, \nCAST(blanco_decam_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_DECAM_MAG, \nCAST(cfht_megacam_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_MEGACAM_MAG, \nCAST(cfht_megacam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_MEGACAM_MAG, \nCAST(jst_jpcam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_JPCAM_MAG, \nCAST(pan_starrs_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_PANSTARRS_MAG, \nCAST(pan_starrs_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_PANSTARRS_MAG, \nCAST(subaru_hsc_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_HSC_MAG, \nCAST(gaia_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_GAIA_MAG, \nCAST(gaia_bp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_BP_GAIA_MAG, \nCAST(gaia_rp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_RP_GAIA_MAG, \nCAST(lsst_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_LSST_MAG, \nCAST(lsst_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_LSST_MAG, \nCAST(lsst_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_LSST_MAG, \nCAST(lsst_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_LSST_MAG, \nCAST(lsst_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_LSST_MAG, \nCAST(lsst_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_LSST_MAG, \nCAST(kids_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_KIDS_MAG, \nCAST(kids_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_KIDS_MAG, \nCAST(kids_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_KIDS_MAG, \nCAST(kids_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_KIDS_MAG, \nCAST(2mass_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_2MASS_MAG, \nCAST(2mass_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_2MASS_MAG, \nCAST(2mass_ks_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_KS_2MASS_MAG \nFROM cosmohub.flagship_mock_1_9_10_swf1_swf2_deep_s \nWHERE (logf_halpha_model3_ext > -16 OR -2.5*log10(euclid_nisp_h) - 48.6 < 26) \nAND disk_axis_ratio > 0.10865 \nAND SHIFTRIGHT(hpix_29_nest, (29-5)*2) = 1008"
}

----

We receive another set of filters for NISP, so we have to reprocess the catalog:

I do it using the flux_pipeline.py.

I create v1.9.11

The external parquet file:

/software/astro/scripts/create_table_from_parquet.sh -p /hdfs/user/jcarrete/data/euclid/flagship_mock_1_9_11/0.pq -t jcarrete.flagship_mock_1_9_11_pq -e

CREATE EXTERNAL TABLE jcarrete.flagship_mock_1_9_11_pq (
`kind` BIGINT,
`random_index` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`ra_mag_gal` DOUBLE,
`dec_mag_gal` DOUBLE,
`kappa` DOUBLE,
`gamma1` DOUBLE,
`gamma2` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`observed_redshift_gal` DOUBLE,
`hpix_29_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`abs_mag_r01` DOUBLE,
`luminosity_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`sdss_r01` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` BIGINT,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha` DOUBLE,
`logf_halpha_ext` DOUBLE,
`halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_halpha_model3_ext` DOUBLE,
`loglum_halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`2mass_h` DOUBLE,
`2mass_j` DOUBLE,
`2mass_ks` DOUBLE,
`blanco_decam_g` DOUBLE,
`blanco_decam_i` DOUBLE,
`blanco_decam_r` DOUBLE,
`blanco_decam_z` DOUBLE,
`cfht_megacam_r` DOUBLE,
`cfht_megacam_u` DOUBLE,
`euclid_vis` DOUBLE,
`gaia_bp` DOUBLE,
`gaia_g` DOUBLE,
`gaia_rp` DOUBLE,
`jst_jpcam_g` DOUBLE,
`kids_g` DOUBLE,
`kids_i` DOUBLE,
`kids_r` DOUBLE,
`kids_u` DOUBLE,
`lsst_g` DOUBLE,
`lsst_i` DOUBLE,
`lsst_r` DOUBLE,
`lsst_u` DOUBLE,
`lsst_y` DOUBLE,
`lsst_z` DOUBLE,
`pan_starrs_i` DOUBLE,
`pan_starrs_z` DOUBLE,
`subaru_hsc_z` DOUBLE,
`2mass_h_odonnell_ext` DOUBLE,
`2mass_j_odonnell_ext` DOUBLE,
`2mass_ks_odonnell_ext` DOUBLE,
`blanco_decam_g_odonnell_ext` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`blanco_decam_r_odonnell_ext` DOUBLE,
`blanco_decam_z_odonnell_ext` DOUBLE,
`cfht_megacam_r_odonnell_ext` DOUBLE,
`cfht_megacam_u_odonnell_ext` DOUBLE,
`euclid_vis_odonnell_ext` DOUBLE,
`gaia_bp_odonnell_ext` DOUBLE,
`gaia_g_odonnell_ext` DOUBLE,
`gaia_rp_odonnell_ext` DOUBLE,
`jst_jpcam_g_odonnell_ext` DOUBLE,
`kids_g_odonnell_ext` DOUBLE,
`kids_i_odonnell_ext` DOUBLE,
`kids_r_odonnell_ext` DOUBLE,
`kids_u_odonnell_ext` DOUBLE,
`lsst_g_odonnell_ext` DOUBLE,
`lsst_i_odonnell_ext` DOUBLE,
`lsst_r_odonnell_ext` DOUBLE,
`lsst_u_odonnell_ext` DOUBLE,
`lsst_y_odonnell_ext` DOUBLE,
`lsst_z_odonnell_ext` DOUBLE,
`pan_starrs_i_odonnell_ext` DOUBLE,
`pan_starrs_z_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`subaru_hsc_z_odonnell_ext` DOUBLE,
`2mass_h_el_model1_odonnell_ext` DOUBLE,
`2mass_h_el_model3_odonnell_ext` DOUBLE,
`2mass_j_el_model1_odonnell_ext` DOUBLE,
`2mass_j_el_model3_odonnell_ext` DOUBLE,
`2mass_ks_el_model1_odonnell_ext` DOUBLE,
`2mass_ks_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model3_odonnell_ext` DOUBLE,
`euclid_vis_el_model1_odonnell_ext` DOUBLE,
`euclid_vis_el_model3_odonnell_ext` DOUBLE,
`gaia_bp_el_model1_odonnell_ext` DOUBLE,
`gaia_bp_el_model3_odonnell_ext` DOUBLE,
`gaia_g_el_model1_odonnell_ext` DOUBLE,
`gaia_g_el_model3_odonnell_ext` DOUBLE,
`gaia_rp_el_model1_odonnell_ext` DOUBLE,
`gaia_rp_el_model3_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model1_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model3_odonnell_ext` DOUBLE,
`kids_g_el_model1_odonnell_ext` DOUBLE,
`kids_g_el_model3_odonnell_ext` DOUBLE,
`kids_i_el_model1_odonnell_ext` DOUBLE,
`kids_i_el_model3_odonnell_ext` DOUBLE,
`kids_r_el_model1_odonnell_ext` DOUBLE,
`kids_r_el_model3_odonnell_ext` DOUBLE,
`kids_u_el_model1_odonnell_ext` DOUBLE,
`kids_u_el_model3_odonnell_ext` DOUBLE,
`lsst_g_el_model1_odonnell_ext` DOUBLE,
`lsst_g_el_model3_odonnell_ext` DOUBLE,
`lsst_i_el_model1_odonnell_ext` DOUBLE,
`lsst_i_el_model3_odonnell_ext` DOUBLE,
`lsst_r_el_model1_odonnell_ext` DOUBLE,
`lsst_r_el_model3_odonnell_ext` DOUBLE,
`lsst_u_el_model1_odonnell_ext` DOUBLE,
`lsst_u_el_model3_odonnell_ext` DOUBLE,
`lsst_y_el_model1_odonnell_ext` DOUBLE,
`lsst_y_el_model3_odonnell_ext` DOUBLE,
`lsst_z_el_model1_odonnell_ext` DOUBLE,
`lsst_z_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model1_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model3_odonnell_ext` DOUBLE,
`dominant_shape` BIGINT,
`bulge_angle` DOUBLE,
`disk_angle` DOUBLE,
`median_major_axis` DOUBLE,
`scale_length` DOUBLE,
`bulge_fraction` DOUBLE,
`disk_scalelength` DOUBLE,
`disk_nsersic` DOUBLE,
`disk_r50` DOUBLE,
`bulge_r50` DOUBLE,
`bulge_nsersic` DOUBLE,
`disk_ellipticity` DOUBLE,
`bulge_ellipticity` DOUBLE,
`disk_axis_ratio` DOUBLE,
`bulge_axis_ratio` DOUBLE,
`inclination_angle` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`luminosity_r01_evolved` DOUBLE,
`hpix_13_nest` BIGINT,
`step` BIGINT,
`euclid_nisp_h` DOUBLE,
`euclid_nisp_j` DOUBLE,
`euclid_nisp_y` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`euclid_nisp_j_odonnell_ext` DOUBLE,
`euclid_nisp_y_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model3_odonnell_ext` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
STORED AS PARQUET
LOCATION '/user/jcarrete/data/euclid/flagship_mock_1_9_11'
;

Creo la ORC en cosmohub:

CREATE TABLE cosmohub.flagship_mock_1_9_11_swf1_swf2_deep_s(
`halo_id` bigint COMMENT 'unique halo id coming from the Flagship dark matter halo catalog',
`galaxy_id` int COMMENT 'combined with the unique_halo_id they uniquely identify each galaxy',
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',
`random_index` float COMMENT 'random number [0 - 1) for subsampling',
`ra_gal` float COMMENT 'galaxy right ascension (degrees)',
`dec_gal` float COMMENT 'galaxy declination (degrees)',
`ra_mag_gal` float COMMENT 'galaxy magnified right ascension (degree)',
`dec_mag_gal` float COMMENT 'galaxy magnified declination (degree)',
`kappa` float COMMENT 'convergence',
`gamma1` float COMMENT 'shear',
`gamma2` float COMMENT 'shear',
`l_gal` float COMMENT 'galactic longitude (degrees)',
`b_gal` float COMMENT 'galactic latitude (degrees)',
`mw_extinction` float COMMENT 'EBV from file: HFI_CompMap_ThermalDustModel_2048_R1.20.fits',
`x_gal` float COMMENT 'galaxy comoving x coordinate (Mpc/h)',
`y_gal` float COMMENT 'galaxy comoving y coordinate (Mpc/h)',
`z_gal` float COMMENT 'galaxy comoving z coordinate (Mpc/h)',
`r_gal` float COMMENT 'galaxy comoving distance (Mpc/h)',
`true_redshift_gal` float COMMENT 'true galaxy redshift',
`observed_redshift_gal` float COMMENT 'observed galaxy redshift (including peculiar velocity)',
`hpix_29_nest` bigint COMMENT 'healpix pixel id using order = 29 in NESTED format',
`vx_gal` float COMMENT 'physical (peculiar) velocity in x-component (km/s)',
`vy_gal` float COMMENT 'physical (peculiar) velocity in y-component (km/s)',
`vz_gal` float COMMENT 'physical (peculiar) velocity in z-component (km/s)',
`vrad_gal` float COMMENT 'missing comment',
`delta_r` float COMMENT 'missing comment',
`abs_mag_r01` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h)',
`luminosity_r01` float COMMENT 'luminosity in the 0.1-r-band',
`abs_mag_r01_evolved` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h) including MICECATv2.0 magnitude evolution',
`sdss_r01` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1',
`abs_mag_uv_dereddened` float COMMENT 'absolute magnitude ultraviolet',
`gr_restframe` float COMMENT '(g-r) rest-frame at z=0',
`color_kind` tinyint COMMENT 'galaxy color kind: 0=red sequence 1=green valley 2=blue cloud',
`sed_ke` float COMMENT 'K-correction for each SED',
`gr_cosmos` float COMMENT 'interpolated COSMOS g-r color',
`sed_cosmos` float COMMENT 'interpolated basic COSMOS SED [0 30]',
`is_within_cosmos` boolean COMMENT 'missing comment',
`cosmos_distance` float COMMENT 'missing comment',
`ext_curve_cosmos` tinyint COMMENT 'COSMOS extinction curve [0-4]',
`ebv_cosmos` float COMMENT 'interpolated extinction strength for COSMOS extinction curve',
`log_ml_r01` float COMMENT 'logarithm of stellar_mass/luminosity ratio in the r01 SDSS band',
`log_stellar_mass` float COMMENT 'logarithm of the stellar mass (Msun/h)',
`metallicity` float COMMENT 'oxygen metallicity in units 12+logO/H',
`log_sfr` float COMMENT 'logarithm of star formation rate in (Msun/h)/year',
`logf_halpha` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: DOES NOT include extinction)',
`logf_halpha_ext` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: includes extinction)',
`halpha_scatter` float COMMENT 'missing comment',
`loglum_halpha` float COMMENT 'DEPRECATED logarithm of Halpha luminosity (erg/s/h^2) using empirical relation with UV magnitude (DOES NOT include extinction)',
`loglum_halpha_ext` float COMMENT 'missing comment',
`logf_dummy` float COMMENT 'missing comment',
`z_dummy` float COMMENT 'missing comment',
`logf_halpha_model1_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: includes extinction)',
`logf_halpha_model1` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: DOES NOT include extinction)',
`loglum_halpha_model1_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (includes extinction)',
`loglum_halpha_model1` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (DOES NOT include extinction)',
`logf_halpha_model3_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: includes extinction)',
`logf_halpha_model3` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: DOES NOT include extinction)',
`loglum_halpha_model3_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (includes extinction)',
`loglum_halpha_model3` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (DOES NOT include extinction)',
`logf_hbeta_model1_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_hbeta_model1` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_n2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_n2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o3_model1_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o3_model1` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_s2_model1_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_s2_model1` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_hbeta_model3_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_hbeta_model3` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_n2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_n2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o3_model3_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o3_model3` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_s2_model3_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_s2_model3` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`2mass_h` float COMMENT 'observed flux in 2MASS H-band (Euclid provided)',
`2mass_j` float COMMENT 'observed flux in 2MASS J-band (Euclid provided)',
`2mass_ks` float COMMENT 'observed flux in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g` float COMMENT 'observed flux in DES g-band (Euclid provided)',
`blanco_decam_i` float COMMENT 'observed flux in DES i-band (Euclid provided)',
`blanco_decam_r` float COMMENT 'observed flux in DES r-band (Euclid provided)',
`blanco_decam_z` float COMMENT 'observed flux in DES z-band (Euclid provided)',
`cfht_megacam_r` float COMMENT 'observed flux in CFHT r-band (Euclid provided)',
`cfht_megacam_u` float COMMENT 'observed flux in CFHT u-band (Euclid provided)',
`euclid_nisp_h` float COMMENT 'observed flux in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j` float COMMENT 'observed flux in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y` float COMMENT 'observed flux in Euclid NISP-Y band (Euclid provided)',
`euclid_vis` float COMMENT 'observed flux in Euclid VIS band (Euclid provided)',
`gaia_bp` float COMMENT 'observed flux in Gaia-BP band (Euclid provided)',
`gaia_g` float COMMENT 'observed flux in Gaia-G band (Euclid provided)',
`gaia_rp` float COMMENT 'observed flux in Gaia-RP band (Euclid provided)',
`jst_jpcam_g` float COMMENT 'observed flux in JST g-band (Euclid provided)',
`kids_g` float COMMENT 'observed flux in KIDS g-band (Euclid provided)',
`kids_i` float COMMENT 'observed flux in KIDS i-band (Euclid provided)',
`kids_r` float COMMENT 'observed flux in KIDS r-band (Euclid provided)',
`kids_u` float COMMENT 'observed flux in KIDS u-band (Euclid provided)',
`lsst_g` float COMMENT 'observed flux in LSST g-band (Euclid provided)',
`lsst_i` float COMMENT 'observed flux in LSST i-band (Euclid provided)',
`lsst_r` float COMMENT 'observed flux in LSST r-band (Euclid provided)',
`lsst_u` float COMMENT 'observed flux in LSST u-band (Euclid provided)',
`lsst_y` float COMMENT 'observed flux in LSST Y-band (Euclid provided)',
`lsst_z` float COMMENT 'observed flux in LSST z-band (Euclid provided)',
`pan_starrs_i` float COMMENT 'observed flux in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z` float COMMENT 'observed flux in Pan-STARRS z-band (Euclid provided)',
`subaru_hsc_z` float COMMENT 'observed flux in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS H-band (Euclid provided)',
`2mass_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in JST g-band (Euclid provided)',
`kids_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_odonnell_ext` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1 with MW extinction',
`subaru_hsc_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS Ks-band (Euclid provided)',
`2mass_ks_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES z-band (Euclid provided)',
`blanco_decam_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT u-band (Euclid provided)',
`cfht_megacam_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_nisp_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid VIS band (Euclid provided)',
`euclid_vis_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_bp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-RP band (Euclid provided)',
`gaia_rp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in JST g-band (Euclid provided)',
`jst_jpcam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in JST g-band (Euclid provided)',
`kids_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS g-band (Euclid provided)',
`kids_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS i-band (Euclid provided)',
`kids_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS r-band (Euclid provided)',
`kids_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS u-band (Euclid provided)',
`kids_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST g-band (Euclid provided)',
`lsst_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST i-band (Euclid provided)',
`lsst_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST r-band (Euclid provided)',
`lsst_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST u-band (Euclid provided)',
`lsst_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST z-band (Euclid provided)',
`lsst_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`pan_starrs_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in sdss r-band redshifted to z=0.1',
`sdss_r01_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in sdss r-band redshifted to z=0.1',
`subaru_hsc_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`subaru_hsc_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`dominant_shape` tinyint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)',
`bulge_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`disk_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`median_major_axis` float COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)',
`scale_length` float COMMENT 'disc and bulge scalelength prior',
`bulge_fraction` float COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)',
`disk_scalelength` float COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)',
`disk_nsersic` float COMMENT 'Sersic index of the disk component',
`disk_r50` float COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678',
`bulge_r50` float COMMENT 'half light radius of the bulge',
`bulge_nsersic` float COMMENT 'Sersic index of the bulge component',
`disk_ellipticity` float COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)',
`bulge_ellipticity` float COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)',
`disk_axis_ratio` float COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`bulge_axis_ratio` float COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`inclination_angle` float COMMENT 'Galaxy inclination angle (where 0 degrees = face-on and 90 degrees = edge-on). Galaxy ellipticity for disk and bulge components are computed following the recipe in https://euclid.roe.ac.uk/projects/sgsshear/wiki/SHE-SIM',
`halo_x` float COMMENT 'halo comoving x coordinate (Mpc/h)',
`halo_y` float COMMENT 'halo comoving y coordinate (Mpc/h)',
`halo_z` float COMMENT 'halo comoving z coordinate (Mpc/h)',
`halo_vx` float COMMENT 'halo x-centre velocity (km/h)',
`halo_vy` float COMMENT 'halo y-centre velocity (km/h)',
`halo_vz` float COMMENT 'halo z-centre velocity (km/h)',
`halo_r` float COMMENT 'halo comoving distance (Mpc/h)',
`true_redshift_halo` float COMMENT 'true redshift of the host halo',
`halo_lm` float COMMENT 'log10 of the FoF halo mass in Msum/h. This halo mass is computed as the particule mass multiplied by the number of particles of the FoF halo. At low masses it is interpolated to obtain a smooth distribution of halo masses',
`halo_n_sats` int COMMENT 'number of satellite galaxies from HOD',
`n_gals` int COMMENT 'DEPRECATED',
`luminosity_r01_evolved` float COMMENT 'missing comment',
`hpix_13_nest` bigint COMMENT 'missing comment'
)
PARTITIONED BY (
`step` smallint COMMENT 'step number in the Flagship lensing maps')
CLUSTERED BY (
hpix_13_nest)
SORTED BY (
hpix_13_nest ASC)
INTO 128 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;

INSERT OVERWRITE TABLE cosmohub.flagship_mock_1_9_11_swf1_swf2_deep_s PARTITION(step)
SELECT halo_id, galaxy_id, kind, random_index, ra_gal, dec_gal, ra_mag_gal, dec_mag_gal, kappa, gamma1, gamma2, l_gal, b_gal, mw_extinction, x_gal, y_gal, z_gal, r_gal, true_redshift_gal, observed_redshift_gal, hpix_29_nest, vx_gal, vy_gal, vz_gal, vrad_gal, delta_r, abs_mag_r01, luminosity_r01, abs_mag_r01_evolved, sdss_r01, abs_mag_uv_dereddened, gr_restframe, color_kind, sed_ke, gr_cosmos, sed_cosmos, is_within_cosmos, cosmos_distance, ext_curve_cosmos, ebv_cosmos, log_ml_r01, log_stellar_mass, metallicity, log_sfr, logf_halpha, logf_halpha_ext, halpha_scatter, loglum_halpha, loglum_halpha_ext, logf_dummy, z_dummy, logf_halpha_model1_ext, logf_halpha_model1, loglum_halpha_model1_ext, loglum_halpha_model1, logf_halpha_model3_ext, logf_halpha_model3, loglum_halpha_model3_ext, loglum_halpha_model3, logf_hbeta_model1_ext, logf_hbeta_model1, logf_o2_model1_ext, logf_o2_model1, logf_n2_model1_ext, logf_n2_model1, logf_o3_model1_ext, logf_o3_model1, logf_s2_model1_ext, logf_s2_model1, logf_hbeta_model3_ext, logf_hbeta_model3, logf_o2_model3_ext, logf_o2_model3, logf_n2_model3_ext, logf_n2_model3, logf_o3_model3_ext, logf_o3_model3, logf_s2_model3_ext, logf_s2_model3, 2mass_h, 2mass_j, 2mass_ks, blanco_decam_g, blanco_decam_i, blanco_decam_r, blanco_decam_z, cfht_megacam_r, cfht_megacam_u, euclid_nisp_h, euclid_nisp_j, euclid_nisp_y, euclid_vis, gaia_bp, gaia_g, gaia_rp, jst_jpcam_g, kids_g, kids_i, kids_r, kids_u, lsst_g, lsst_i, lsst_r, lsst_u, lsst_y, lsst_z, pan_starrs_i, pan_starrs_z, subaru_hsc_z, 2mass_h_odonnell_ext, 2mass_j_odonnell_ext, 2mass_ks_odonnell_ext, blanco_decam_g_odonnell_ext, blanco_decam_i_odonnell_ext, blanco_decam_r_odonnell_ext, blanco_decam_z_odonnell_ext, cfht_megacam_r_odonnell_ext, cfht_megacam_u_odonnell_ext, euclid_nisp_h_odonnell_ext, euclid_nisp_j_odonnell_ext, euclid_nisp_y_odonnell_ext, euclid_vis_odonnell_ext, gaia_bp_odonnell_ext, gaia_g_odonnell_ext, gaia_rp_odonnell_ext, jst_jpcam_g_odonnell_ext, kids_g_odonnell_ext, kids_i_odonnell_ext, kids_r_odonnell_ext, kids_u_odonnell_ext, lsst_g_odonnell_ext, lsst_i_odonnell_ext, lsst_r_odonnell_ext, lsst_u_odonnell_ext, lsst_y_odonnell_ext, lsst_z_odonnell_ext, pan_starrs_i_odonnell_ext, pan_starrs_z_odonnell_ext, sdss_r01_odonnell_ext, subaru_hsc_z_odonnell_ext, 2mass_h_el_model1_odonnell_ext, 2mass_h_el_model3_odonnell_ext, 2mass_j_el_model1_odonnell_ext, 2mass_j_el_model3_odonnell_ext, 2mass_ks_el_model1_odonnell_ext, 2mass_ks_el_model3_odonnell_ext, blanco_decam_g_el_model1_odonnell_ext, blanco_decam_g_el_model3_odonnell_ext, blanco_decam_i_el_model1_odonnell_ext, blanco_decam_i_el_model3_odonnell_ext, blanco_decam_r_el_model1_odonnell_ext, blanco_decam_r_el_model3_odonnell_ext, blanco_decam_z_el_model1_odonnell_ext, blanco_decam_z_el_model3_odonnell_ext, cfht_megacam_r_el_model1_odonnell_ext, cfht_megacam_r_el_model3_odonnell_ext, cfht_megacam_u_el_model1_odonnell_ext, cfht_megacam_u_el_model3_odonnell_ext, euclid_nisp_h_el_model1_odonnell_ext, euclid_nisp_h_el_model3_odonnell_ext, euclid_nisp_j_el_model1_odonnell_ext, euclid_nisp_j_el_model3_odonnell_ext, euclid_nisp_y_el_model1_odonnell_ext, euclid_nisp_y_el_model3_odonnell_ext, euclid_vis_el_model1_odonnell_ext, euclid_vis_el_model3_odonnell_ext, gaia_bp_el_model1_odonnell_ext, gaia_bp_el_model3_odonnell_ext, gaia_g_el_model1_odonnell_ext, gaia_g_el_model3_odonnell_ext, gaia_rp_el_model1_odonnell_ext, gaia_rp_el_model3_odonnell_ext, jst_jpcam_g_el_model1_odonnell_ext, jst_jpcam_g_el_model3_odonnell_ext, kids_g_el_model1_odonnell_ext, kids_g_el_model3_odonnell_ext, kids_i_el_model1_odonnell_ext, kids_i_el_model3_odonnell_ext, kids_r_el_model1_odonnell_ext, kids_r_el_model3_odonnell_ext, kids_u_el_model1_odonnell_ext, kids_u_el_model3_odonnell_ext, lsst_g_el_model1_odonnell_ext, lsst_g_el_model3_odonnell_ext, lsst_i_el_model1_odonnell_ext, lsst_i_el_model3_odonnell_ext, lsst_r_el_model1_odonnell_ext, lsst_r_el_model3_odonnell_ext, lsst_u_el_model1_odonnell_ext, lsst_u_el_model3_odonnell_ext, lsst_y_el_model1_odonnell_ext, lsst_y_el_model3_odonnell_ext, lsst_z_el_model1_odonnell_ext, lsst_z_el_model3_odonnell_ext, pan_starrs_i_el_model1_odonnell_ext, pan_starrs_i_el_model3_odonnell_ext, pan_starrs_z_el_model1_odonnell_ext, pan_starrs_z_el_model3_odonnell_ext, sdss_r01_el_model1_odonnell_ext, sdss_r01_el_model3_odonnell_ext, subaru_hsc_z_el_model1_odonnell_ext, subaru_hsc_z_el_model3_odonnell_ext, dominant_shape, bulge_angle, disk_angle, median_major_axis, scale_length, bulge_fraction, disk_scalelength, disk_nsersic, disk_r50, bulge_r50, bulge_nsersic, disk_ellipticity, bulge_ellipticity, disk_axis_ratio, bulge_axis_ratio, inclination_angle, halo_x, halo_y, halo_z, halo_vx, halo_vy, halo_vz, halo_r, true_redshift_halo, halo_lm, halo_n_sats, n_gals, luminosity_r01_evolved, hpix_13_nest, step
FROM jcarrete.flagship_mock_1_9_11_pq;

ANALYZE TABLE cosmohub.flagship_mock_1_9_11_swf1_swf2_deep_s PARTITION(step) COMPUTE STATISTICS FOR COLUMNS;

CREATE TABLE jcarrete.flagship_mock_1_9_11_swf1_swf2_deep_s_magnified_fits(
`source_id` bigint COMMENT 'from deserializer',
`halo_id` bigint COMMENT 'from deserializer',
`ra` float COMMENT 'from deserializer',
`dec` float COMMENT 'from deserializer',
`ra_mag` float COMMENT 'from deserializer',
`dec_mag` float COMMENT 'from deserializer',
`z_obs` float COMMENT 'from deserializer',
`ref_mag_abs` float COMMENT 'from deserializer',
`ref_mag` float COMMENT 'from deserializer',
`bulge_fraction` float COMMENT 'from deserializer',
`bulge_r50` float COMMENT 'from deserializer',
`disk_r50` float COMMENT 'from deserializer',
`bulge_nsersic` float COMMENT 'from deserializer',
`inclination_angle` float COMMENT 'from deserializer',
`disk_angle` float COMMENT 'from deserializer',
`kappa` float COMMENT 'from deserializer',
`gamma1` float COMMENT 'from deserializer',
`gamma2` float COMMENT 'from deserializer',
`sed_template` float COMMENT 'from deserializer',
`ext_law` smallint COMMENT 'from deserializer',
`ebv` float COMMENT 'from deserializer',
`halpha_logflam_ext` float COMMENT 'from deserializer',
`hbeta_logflam_ext` float COMMENT 'from deserializer',
`o2_logflam_ext` float COMMENT 'from deserializer',
`o3_logflam_ext` float COMMENT 'from deserializer',
`n2_logflam_ext` float COMMENT 'from deserializer',
`s2_logflam_ext` float COMMENT 'from deserializer',
`av` float COMMENT 'from deserializer',
`tu_fnu_vis` float COMMENT 'from deserializer',
`tu_fnu_y_nisp` float COMMENT 'from deserializer',
`tu_fnu_j_nisp` float COMMENT 'from deserializer',
`tu_fnu_h_nisp` float COMMENT 'from deserializer',
`tu_fnu_g_decam` float COMMENT 'from deserializer',
`tu_fnu_r_decam` float COMMENT 'from deserializer',
`tu_fnu_i_decam` float COMMENT 'from deserializer',
`tu_fnu_z_decam` float COMMENT 'from deserializer',
`tu_fnu_u_megacam` float COMMENT 'from deserializer',
`tu_fnu_r_megacam` float COMMENT 'from deserializer',
`tu_fnu_g_jpcam` float COMMENT 'from deserializer',
`tu_fnu_i_panstarrs` float COMMENT 'from deserializer',
`tu_fnu_z_panstarrs` float COMMENT 'from deserializer',
`tu_fnu_z_hsc` float COMMENT 'from deserializer',
`tu_fnu_g_gaia` float COMMENT 'from deserializer',
`tu_fnu_bp_gaia` float COMMENT 'from deserializer',
`tu_fnu_rp_gaia` float COMMENT 'from deserializer',
`tu_fnu_u_lsst` float COMMENT 'from deserializer',
`tu_fnu_g_lsst` float COMMENT 'from deserializer',
`tu_fnu_r_lsst` float COMMENT 'from deserializer',
`tu_fnu_i_lsst` float COMMENT 'from deserializer',
`tu_fnu_z_lsst` float COMMENT 'from deserializer',
`tu_fnu_y_lsst` float COMMENT 'from deserializer',
`tu_fnu_u_kids` float COMMENT 'from deserializer',
`tu_fnu_g_kids` float COMMENT 'from deserializer',
`tu_fnu_r_kids` float COMMENT 'from deserializer',
`tu_fnu_i_kids` float COMMENT 'from deserializer',
`tu_fnu_j_2mass` float COMMENT 'from deserializer',
`tu_fnu_h_2mass` float COMMENT 'from deserializer',
`tu_fnu_ks_2mass` float COMMENT 'from deserializer')
PARTITIONED BY (
`hpix_5_nest` bigint)
ROW FORMAT SERDE
'es.pic.astro.hadoop.serde.RecArraySerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'es.pic.astro.hadoop.io.BinaryOutputFormat'
;

INSERT OVERWRITE TABLE jcarrete.flagship_mock_1_9_11_swf1_swf2_deep_s_magnified_fits PARTITION(hpix_5_nest)
SELECT CAST(((halo_id * 10000) + galaxy_id) AS bigint) AS SOURCE_ID,
CAST(halo_id AS bigint) AS HALO_ID,
CAST(ra_gal AS float) AS RA,
CAST(dec_gal AS float) AS `DEC`,
CAST(ra_mag_gal AS float) AS RA_MAG,
CAST(dec_mag_gal AS float) AS DEC_MAG,
CAST(observed_redshift_gal AS float) AS Z_OBS,
CAST(abs_mag_r01_evolved AS float) AS REF_MAG_ABS,
CAST(-2.5*log10(sdss_r01) - 48.6 AS float) AS REF_MAG,
CAST(bulge_fraction AS float) AS BULGE_FRACTION,
CAST(bulge_r50 AS float) AS BULGE_R50,
CAST(disk_r50 AS float) AS DISK_R50,
CAST(bulge_nsersic AS float) AS BULGE_NSERSIC,
CAST(inclination_angle AS float) AS INCLINATION_ANGLE,
CAST(disk_angle AS float) AS DISK_ANGLE,
CAST(kappa AS float) AS KAPPA,
CAST(gamma1 AS float) AS GAMMA1,
CAST(gamma2 AS float) AS GAMMA2,
CAST(sed_cosmos AS float) AS SED_TEMPLATE,
CAST(ROUND(ext_curve_cosmos) AS smallint) AS EXT_LAW,
CAST(ebv_cosmos AS float) AS EBV,
CAST(logf_halpha_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HALPHA_LOGFLAM_EXT,
CAST(logf_hbeta_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HBETA_LOGFLAM_EXT,
CAST(logf_o2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O2_LOGFLAM_EXT,
CAST(logf_o3_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O3_LOGFLAM_EXT,
CAST(logf_n2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS N2_LOGFLAM_EXT,
CAST(logf_s2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS S2_LOGFLAM_EXT,
CAST(mw_extinction AS float) AS AV,
CAST(euclid_vis_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_VIS,
CAST(euclid_nisp_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_NISP,
CAST(euclid_nisp_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_NISP,
CAST(euclid_nisp_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_NISP,
CAST(blanco_decam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_DECAM,
CAST(blanco_decam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_DECAM,
CAST(blanco_decam_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_DECAM,
CAST(blanco_decam_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_DECAM,
CAST(cfht_megacam_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_MEGACAM,
CAST(cfht_megacam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_MEGACAM,
CAST(jst_jpcam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_JPCAM,
CAST(pan_starrs_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_PANSTARRS,
CAST(pan_starrs_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_PANSTARRS,
CAST(subaru_hsc_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_HSC,
CAST(gaia_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_GAIA,
CAST(gaia_bp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_BP_GAIA,
CAST(gaia_rp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_RP_GAIA,
CAST(lsst_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_LSST,
CAST(lsst_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_LSST,
CAST(lsst_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_LSST,
CAST(lsst_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_LSST,
CAST(lsst_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_LSST,
CAST(lsst_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_LSST,
CAST(kids_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_KIDS,
CAST(kids_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_KIDS,
CAST(kids_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_KIDS,
CAST(kids_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_KIDS,
CAST(2mass_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_2MASS,
CAST(2mass_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_2MASS,
CAST(2mass_ks_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_KS_2MASS,
CAST(SHIFTRIGHT(hpix_29_nest, (29-5)*2) AS bigint) AS hpix_5_nest
FROM cosmohub.flagship_mock_1_9_11_swf1_swf2_deep_s
WHERE (logf_halpha_model3_ext > -16 OR -2.5*log10(euclid_nisp_h) - 48.6 < 26)
AND disk_axis_ratio > 0.10865
;

Information in CosmoHub:

Dark matter halo input catalog from flagship_rockstar_octant1_c, which is the Flagship dark matter halo catalog v1.1 with correct velocities.
There are three different regions to cover the current selected observations to simulate for GSIR, which are SWF1, SWF2 and the Deep field (see the following link for more details:[https://euclid.roe.ac.uk/issues/14134](https://euclid.roe.ac.uk/issues/14134)).
**The only difference with previous version v1.9.10 is the update of NISP filter responses.**
**Shape distributions are completely different from all previous versions (including new recipe for inclination_angle field)**.
In this version a new python method for inclination_angle was developed by Francisco and ellipticities and axis ratios come from Eric's formulas.
See the following [url](https://docs.google.com/document/d/1FRERcl5Lv2IDB6ZChTK6gcEHI2FWA9gSgYKcaDUA9EM/edit?usp=sharing) for more details.
Note that **we provide fluxes instead of magnitudes**. In order **to get magnitudes** you can enter in the "Expert mode" and estimate the magnitude as follows:
m = -2.5 * log10(flux) - 48.6
And in order to estimate magnified magnitudes:
m_mag = m + 2.5 * log10((1-kappa)*(1-kappa) - gamma1 * gamma1 - gamma2 * gamma2)

----

FITS file dataset:

{
"sql": "SELECT CAST(((halo_id * 10000) + galaxy_id) AS bigint) AS SOURCE_ID, \nCAST(halo_id AS bigint) AS HALO_ID, \nCAST(ra_gal AS float) AS RA, \nCAST(dec_gal AS float) AS `DEC`, \nCAST(ra_mag_gal AS float) AS RA_MAG, \nCAST(dec_mag_gal AS float) AS DEC_MAG, \nCAST(observed_redshift_gal AS float) AS Z_OBS, \nCAST(abs_mag_r01_evolved AS float) AS REF_MAG_ABS, \nCAST(-2.5*log10(sdss_r01) - 48.6 AS float) AS REF_MAG, \nCAST(bulge_fraction AS float) AS BULGE_FRACTION, \nCAST(bulge_r50 AS float) AS BULGE_R50, \nCAST(disk_r50 AS float) AS DISK_R50, \nCAST(bulge_nsersic AS float) AS BULGE_NSERSIC, \nCAST(inclination_angle AS float) AS INCLINATION_ANGLE, \nCAST(disk_angle AS float) AS DISK_ANGLE, \nCAST(kappa AS float) AS KAPPA, \nCAST(gamma1 AS float) AS GAMMA1, \nCAST(gamma2 AS float) AS GAMMA2, \nCAST(sed_cosmos AS float) AS SED_TEMPLATE, \nCAST(ROUND(ext_curve_cosmos) AS smallint) AS EXT_LAW, \nCAST(ebv_cosmos AS float) AS EBV, \nCAST(logf_halpha_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HALPHA_LOGFLAM_EXT_MAG, \nCAST(logf_hbeta_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HBETA_LOGFLAM_EXT_MAG, \nCAST(logf_o2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O2_LOGFLAM_EXT_MAG, \nCAST(logf_o3_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O3_LOGFLAM_EXT_MAG, \nCAST(logf_n2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS N2_LOGFLAM_EXT_MAG, \nCAST(logf_s2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS S2_LOGFLAM_EXT_MAG, \nCAST(mw_extinction AS float) AS AV,\nCAST(euclid_vis_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_VIS_MAG, \nCAST(euclid_nisp_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_NISP_MAG, \nCAST(euclid_nisp_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_NISP_MAG, \nCAST(euclid_nisp_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_NISP_MAG, \nCAST(blanco_decam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_DECAM_MAG, \nCAST(blanco_decam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_DECAM_MAG, \nCAST(blanco_decam_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_DECAM_MAG, \nCAST(blanco_decam_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_DECAM_MAG, \nCAST(cfht_megacam_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_MEGACAM_MAG, \nCAST(cfht_megacam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_MEGACAM_MAG, \nCAST(jst_jpcam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_JPCAM_MAG, \nCAST(pan_starrs_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_PANSTARRS_MAG, \nCAST(pan_starrs_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_PANSTARRS_MAG, \nCAST(subaru_hsc_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_HSC_MAG, \nCAST(gaia_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_GAIA_MAG, \nCAST(gaia_bp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_BP_GAIA_MAG, \nCAST(gaia_rp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_RP_GAIA_MAG, \nCAST(lsst_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_LSST_MAG, \nCAST(lsst_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_LSST_MAG, \nCAST(lsst_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_LSST_MAG, \nCAST(lsst_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_LSST_MAG, \nCAST(lsst_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_LSST_MAG, \nCAST(lsst_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_LSST_MAG, \nCAST(kids_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_KIDS_MAG, \nCAST(kids_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_KIDS_MAG, \nCAST(kids_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_KIDS_MAG, \nCAST(kids_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_KIDS_MAG, \nCAST(2mass_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_2MASS_MAG, \nCAST(2mass_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_2MASS_MAG, \nCAST(2mass_ks_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_KS_2MASS_MAG \nFROM cosmohub.flagship_mock_1_9_11_swf1_swf2_deep_s \nWHERE (logf_halpha_model3_ext > -16 OR -2.5*log10(euclid_nisp_h) - 48.6 < 26) \nAND disk_axis_ratio > 0.10865 \nAND SHIFTRIGHT(hpix_29_nest, (29-5)*2) = 1008"
}

Catalogs

2021-01-26T16:15:35Z

Jcarrete: /* Ground Segment Implementation Review (GSIR) */

== Euclid productions ==

==== Zurich Dark Matter Halo catalogs ====

* [[ v1.0 | v1.0 ]]

* [[ v1.1 | v1.1 Corrected Velocities ]]

==== Mock Galaxy catalogs ====

More info from Euclid wiki (Francesc):

https://euclid.roe.ac.uk/projects/sgv/wiki/SC456_#Galaxies

* [[ Euclid_mock_production_v1.6.20 | v1.6.20 ]] ([https://euclid.roe.ac.uk/issues/9153 #9153])
* [[ Euclid_mock_production_v1.7.17 | v1.7.17 ]] (SC456)
* [[ Euclid_mock_production_v1.8.1 | v1.8.1 ]]
* [[ Euclid_mock_production_v1.8.3 | v1.8.3 ]] (single octant for Helsinki)
* [[ Euclid_mock_production_v1.8.3_c | v1.8.3_c ]] (clustered table without lensing)
* [[ Euclid_mock_production_v1.8.4_s | v1.8.4_s ]] (from 1.8.3 with correct lensing and without buffer)
* [[ Euclid_mock_production_v1.8.5 | v1.8.5 ]] (Kai subset with additional magnitudes for IA)
* [[ Euclid_mock_production_v1.7.17_correct | v1.7.17_correct (v1.7.18)]] (SC456)
* [[ Euclid_mock_production_v1.7.17_deep_correct | v1.7.17_deep_correct]] (SC456)
* [[ Euclid_mock_production_v1.7.17_correct_v2 | v1.7.17_correct_v2 (v1.7.19)]] (SC456)
* [[ Euclid_mock_production_v1.7.17_deep_correct_v2 | v1.7.17_deep_correct_v2]] (SC456)
* [[ Euclid_mock_production_v1.9.2 | v1.9.2]] (GSIR)
* [[ Euclid_mock_production_v1.9.3 | v1.9.3]] (GSIR)
* [[ Euclid_mock_production_v1.9.4 | v1.9.4]] (GSIR)
* [[ Euclid_mock_production_v1.9.5 | v1.9.5]] (GSIR)
* [[ Euclid_mock_production_v1.9.6 | v1.9.6]] (GSIR)
* [[ Euclid_mock_production_v1.9.7 | v1.9.7]] (GSIR)

==== Ground Segment Implementation Review (GSIR) ====

* [[ GSIR | GSIR release process]]

* [[ GSIR_more | GSIR release process (more)]]

==== Primeval Universe Catalogs ====
* [[ QSOs | QSOs]]
* [[ High_redshift_galaxies | High-z galaxies]]

==== Scientific Challenge 8 (SC8) ====

* [[ Standard_galaxies]]
* [[ High-z_QSOs]]
* [[ High-z_galaxies]]

== MICE productions ==

* [[ MIDES Wide Field v5.0 | MIDES Wide Field v5.0 ]]
* [[ MIDES Wide Field v6.0 | MIDES Wide Field v6.0 ]]
* [[ MICE Redmagic v5.1 ]]
* [[ DESY3 MICE2 source catalog | DESY3 MICE2 source catalog ]]

== PAUS productions ==

=== Observations ===

* [[ PAUdm_forced_aperture_coadd | PAUdm forced aperture coadd ]]

==== Photoz productions ====

* [[ photoz_bcnz | bcnz_v29.parquet ]]

* [[ production_table | paudm_production_table ]]

* [[ PAUS+COSMOS_photoz | PAUS+COSMOS_photoz ]]

==== CIGALE catalogs ====

* [[ CIGALE-W3 | CIGALE W3 ]]

==== Mock Galaxy catalogs ====

* [[ Flagship-PAU | Flagship-PAU v1.8.4_v0.1 ]]

* [[ Flagship-PAU_v02 | Flagship-PAU v1.8.4_v0.2 ]]

* [[ Flagship-PAU_v03 | Flagship-PAU v1.8.4_v0.3 ]]

* [[ Durham-LC-PAU lc5.field1.63 | Durham LC lc5.field1.63 ]]

== External ==

* [[ Cosmos_2015_DR_2.1 | COSMOS 2015 DR 2.1 ]]
* [[ ZEST_v1.0 | Zest v1.0 ]]
* [[ CANDELS_2018 | CANDELS 2018 ]]
* [[ Legacy Survey DR8 | Legacy Survey DR8 ]]
* [[ Legacy Survey DR9 | Legacy Survey DR9 ]]
* [[ KiDS DR4 ]]
* [[ VIPERS ]]
* [[ The COSMOS ACS Catalog ]]
* [[ GLADE ]]
* [[ Gaia_EDR3 | Gaia EDR3 ]]

GSIR

2021-01-20T17:53:24Z

Jcarrete: Created page with " == This is a copy of the notes I took to generate GSIR == It is a long story... ---- This is the summary of the generation of the TU FITS files needed for GSIR: We start..."

== This is a copy of the notes I took to generate GSIR ==

It is a long story...

----

This is the summary of the generation of the TU FITS files needed for GSIR:

We start by generating a mock galaxy catalog of the full first octant (v1.9.8) using the full pipeline (full_pipeline.py) but only including 3 filters in the flux computation:

* euclid_nisp_h for making the magnitude limit cut and therefore make the catalog lighter.
We make the cut in this way:

df = df[
(df['euclid_nisp_h'] > flux_cut)
| (df['logf_halpha_model1_ext'] > -16)
| (df['logf_halpha_model3_ext'] > -16)
]

where flux_cut = mag_to_flux(26) So we cut all galaxies fainter than 26 in the euclid_nisp_h filter

* blanco_decam_i: needed to estimate morphological parameters

* sdss_r01: for precuation since it is the reference filter. I think we can avoid it

----

The generated catalog is already in ORC format in a table called: jcarrete.flagship_mock_1_9_8_c

I tried to generate it in a _c2 format with the same partitions and buckets than the lensing table in order to make an efficient JOIN but I made a mistake with the number of buckets.

I put 4096 and given that it is already partitioned by step, each partition is clustered in 4096 buckets. This is a huge amount of files and we had problems with that execution.

----

So finally we materialized the 2 pointings we had at that time: SWF1 and SWF2:

SWF1
Area pointing obsid: 52929
ra_range = 7.678307967139321 14.008504032860678
dec_range = -21.587506 -15.587506000000001

Area pointing obsid in 1st octant: 52929
ra_range = 7.678307967139321 14.008504032860678
dec_range = 15.587506000000001 21.587506

SWF2
Area pointing obsid: 1717
ra_range = 215.45815488783862 241.3301611121614
dec_range = 73.5904 79.5904

Area pointing obsid in 1st octant: 1717
ra_range = 35.45815488783862 61.33016111216139
dec_range = 73.5904 79.5904

I created a table with those two regions with the correct format to make the JOIN efficiently in the test platform!

CREATE TABLE jcarrete.flagship_mock_obsid_52929_1717_c2(
...
)
PARTITIONED BY (
`step` smallint
)
CLUSTERED BY (
hpix_13_nest
)
SORTED BY (
hpix_13_nest ASC
)
INTO 128 BUCKETS
STORED AS ORC
;

'''Then Pau add the lensing in the Test platform.'''
We finally have a table like this:

CREATE TABLE jcarrete.flagship_mock_obsid_52929_1717_s(
)
CLUSTERED BY (
hpix_13_nest)
SORTED BY (
hpix_13_nest ASC)
INTO 128 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;

'''Adding FLUXES'''
Once the lensing is added I run the flux_pipeline.ipynb, in which we have updated the euclid filters. ONLY THE euclid ones at this time!

I do it for the 1717 observation (at this time I do not know what to do with the area that has to be rotated to the southern hemisphere, obsid= 52929).

/software/astro/scripts/create_table_from_parquet.sh -p /hdfs/user/jcarrete/data/euclid/flagship_mock_obsid_1717_s/0.pq -t jcarrete.flagship_mock_obsid_1717_s_pq -e

CREATE EXTERNAL TABLE jcarrete.flagship_mock_obsid_1717_s_pq (
...
)
STORED AS PARQUET
LOCATION '/user/jcarrete/data/euclid/flagship_mock_obsid_1717_s'
;

I create the ORC table. This table contains "everything needed" (but note that shapes have a bug and filters of LSST are not updated yet!)

CREATE TABLE jcarrete.flagship_mock_obsid_1717_s(
...
)
PARTITIONED BY (
`step` smallint COMMENT 'step number in the Flagship lensing maps')
CLUSTERED BY (
hpix_13_nest)
SORTED BY (
hpix_13_nest ASC)
INTO 128 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;

Then I create the FITS table:

CREATE TABLE jcarrete.flagship_mock_obsid_1717_s_magnified_fits(
...
)
PARTITIONED BY (
`hpix_5_nest` bigint)
ROW FORMAT SERDE
'es.pic.astro.hadoop.serde.RecArraySerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'es.pic.astro.hadoop.io.BinaryOutputFormat'
;

----

Cuando inserto los datos en esta tabla cambio el signo de gamma1!!!!!

Esto es tras varias conversaciones con Pablo y Francisco acerca de las convenciones de MICE y de Flagship.
Sigo sin entender por qué en MICE calculan RA de la manera que lo hacen (swap entre x e y) y luego a la hora de buscar el pixel en los mapas a través de phi y theta, le meten un shift en phi de valor pi.
En cualquier caso, a la conclusión a la que llegamos es que para pasar del sistema de referencia local de Healpy al de IAU, hay que modificar gamma1 = -gamma1.

Además, cuando rotamos un area con DEC > 0 a una con DEC < 0, hay que volver a cambiarle el signo a gamma1!
Esto será "útil" para el obsid = 52929

INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_1717_s_magnified_fits PARTITION(hpix_5_nest)
...
CAST(-gamma1 as FLOAT) AS GAMMA1
)

Como se ve, los cambios en gamma1 se hacen a nivel de FITS files. Es decir, las tablas de las que parto siguen estando MAL.
Esto lo tendría que cambiar de cara a poner online el catálogo en CosmoHub! Tendrían que ser iguales.
Todo excepto, y no lo voy a hacer, los valores de x, y, z. La traslación la hago en RA,DEC

----

Es en este momento en el que alguien se da cuenta de que las shapes están mal!

Corrijo el problema que estaba en las unidades que le pasaba a la función que calculaba las elipticidades (de grados a radianes)

Y creo otra tabla usando el shape_pipeline.py: jcarrete.flagship_mock_obsid_1717_s_v2_pq

La paso a ORC: jcarrete.flagship_mock_obsid_1717_s_v2

----

Hay que volver a correr los flujos para la tabla jcarrete.flagship_mock_obsid_1717_s_v2 ya que nos han actualizado los del LSST. Ver más abajo los detalles:

----

Tenemos que:

* Corregir el fjc_miller.py. Pau tendría que hacerlo transparente y que el nombre de la clase no cambiara.

* En el notebook full_pipeline.py he metido los cambios del shape_pipeline.py y del flux_pipeline.py.

'''OJO CON LOS BUCKETS PORQUE HE PUESTO 4096 EN UNA TABLA PARTICIONADA Y ESO ES MUCHIIIIIISSSIIIMO!!!!'''

/software/astro/scripts/create_table_from_parquet.sh -p /hdfs/user/jcarrete/data/euclid/flagship_mock_1_9_8/0.pq -t jcarrete.flagship_mock_1_9_8_pq -e

CREATE EXTERNAL TABLE jcarrete.flagship_mock_1_9_8_pq (
`kind` BIGINT,
`luminosity_r01` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`abs_mag_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`luminosity_r01_evolved` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` INT,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`hpix_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`observed_redshift_gal` DOUBLE,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha_ext` DOUBLE,
`logf_halpha` DOUBLE,
`Halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_Halpha_model3_ext` DOUBLE,
`loglum_Halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`blanco_decam_i` DOUBLE,
`sdss_r01` DOUBLE,
`euclid_nisp_h` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`dominant_shape` BIGINT,
`bulge_angle` DOUBLE,
`disk_angle` DOUBLE,
`median_major_axis` DOUBLE,
`scale_length` DOUBLE,
`bulge_fraction` DOUBLE,
`disk_scalelength` DOUBLE,
`disk_nsersic` DOUBLE,
`disk_r50` DOUBLE,
`bulge_r50` DOUBLE,
`bulge_nsersic` DOUBLE,
`inclination_angle` DOUBLE,
`disk_axis_ratio` DOUBLE,
`bulge_axis_ratio` DOUBLE,
`disk_ellipticity` DOUBLE,
`bulge_ellipticity` DOUBLE,
`step` BIGINT,
`random_index` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
STORED AS PARQUET
LOCATION '/user/jcarrete/data/euclid/flagship_mock_1_9_8'
;

----

PREGUNTAR EL NUMERO DE BUCKETS!
Si no recuerdo mal era el volumen de los datos dividido entre 256MB y "pasarme" hasta el número en base 2 siguiente.

SON 3 TB de catálogo, pongo 4096 buckets

CREATE TABLE jcarrete.flagship_mock_1_9_8_c (
`kind` BIGINT,
`luminosity_r01` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`abs_mag_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`luminosity_r01_evolved` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` INT,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`hpix_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`observed_redshift_gal` DOUBLE,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha_ext` DOUBLE,
`logf_halpha` DOUBLE,
`Halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_Halpha_model3_ext` DOUBLE,
`loglum_Halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`blanco_decam_i` DOUBLE,
`sdss_r01` DOUBLE,
`euclid_nisp_h` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`dominant_shape` BIGINT,
`bulge_angle` DOUBLE,
`disk_angle` DOUBLE,
`median_major_axis` DOUBLE,
`scale_length` DOUBLE,
`bulge_fraction` DOUBLE,
`disk_scalelength` DOUBLE,
`disk_nsersic` DOUBLE,
`disk_r50` DOUBLE,
`bulge_r50` DOUBLE,
`bulge_nsersic` DOUBLE,
`inclination_angle` DOUBLE,
`disk_axis_ratio` DOUBLE,
`bulge_axis_ratio` DOUBLE,
`disk_ellipticity` DOUBLE,
`bulge_ellipticity` DOUBLE,
`step` BIGINT,
`random_index` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
CLUSTERED BY (
halo_id,
galaxy_id
)
INTO 4096 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;

INSERT OVERWRITE TABLE jcarrete.flagship_mock_1_9_8_c
SELECT * FROM jcarrete.flagship_mock_1_9_8_pq;

ANALYZE TABLE jcarrete.flagship_mock_1_9_8_c COMPUTE STATISTICS FOR COLUMNS;

----

PREGUNTAR DE NUEVO POR EL NUMERO DE BUCKETS

EL NUMERO DE BUCKETS ESTÁ MAL, ya que estoy dividiendo cada PARTITION EN 4096 buckets. ES UNA BARBARIDAD!

ESTA TABLA FALLA:

CREATE TABLE jcarrete.flagship_mock_1_9_8_c2(
`kind` BIGINT,
`luminosity_r01` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`abs_mag_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`luminosity_r01_evolved` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` INT,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`hpix_29_nest` BIGINT,
`hpix_13_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`observed_redshift_gal` DOUBLE,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha_ext` DOUBLE,
`logf_halpha` DOUBLE,
`Halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_Halpha_model3_ext` DOUBLE,
`loglum_Halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`blanco_decam_i` DOUBLE,
`sdss_r01` DOUBLE,
`euclid_nisp_h` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`dominant_shape` BIGINT,
`bulge_angle` DOUBLE,
`disk_angle` DOUBLE,
`median_major_axis` DOUBLE,
`scale_length` DOUBLE,
`bulge_fraction` DOUBLE,
`disk_scalelength` DOUBLE,
`disk_nsersic` DOUBLE,
`disk_r50` DOUBLE,
`bulge_r50` DOUBLE,
`bulge_nsersic` DOUBLE,
`inclination_angle` DOUBLE,
`disk_axis_ratio` DOUBLE,
`bulge_axis_ratio` DOUBLE,
`disk_ellipticity` DOUBLE,
`bulge_ellipticity` DOUBLE,
`random_index` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
PARTITIONED BY (
`step` smallint
)
CLUSTERED BY (
hpix_13_nest
)
SORTED BY (
hpix_13_nest ASC
)
INTO 4096 BUCKETS
STORED AS ORC
;

'''ESTA HA PETADO!'''

INSERT OVERWRITE TABLE jcarrete.flagship_mock_1_9_8_c2
SELECT
`kind`, `luminosity_r01`, `halo_x`, `halo_y`, `halo_z`, `halo_vx`, `halo_vy`, `halo_vz`, `halo_r`, `true_redshift_halo`, `halo_lm`, `halo_n_sats`, `n_gals`, `abs_mag_r01`, `abs_mag_r01_evolved`, `luminosity_r01_evolved`, `gr_restframe`, `color_kind`, `x_gal`, `y_gal`, `z_gal`, `r_gal`, `true_redshift_gal`, `ra_gal`, `dec_gal`, `hpix_nest`, SHIFTRIGHT(`hpix_nest`, (29-13)*2), `vx_gal`, `vy_gal`, `vz_gal`, `vrad_gal`, `delta_r`, `observed_redshift_gal`, `sed_ke`, `gr_cosmos`, `sed_cosmos`, `ext_curve_cosmos`, `ebv_cosmos`, `is_within_cosmos`, `cosmos_distance`, `abs_mag_uv_dereddened`, `log_ml_r01`, `log_stellar_mass`, `metallicity`, `log_sfr`, `logf_halpha_ext`, `logf_halpha`, `Halpha_scatter`, `loglum_halpha`, `loglum_halpha_ext`, `logf_dummy`, `z_dummy`, `logf_halpha_model1_ext`, `logf_halpha_model1`, `loglum_halpha_model1_ext`, `loglum_halpha_model1`, `logf_halpha_model3_ext`, `logf_halpha_model3`, `loglum_Halpha_model3_ext`, `loglum_Halpha_model3`, `logf_hbeta_model1_ext`, `logf_hbeta_model1`, `logf_o2_model1_ext`, `logf_o2_model1`, `logf_n2_model1_ext`, `logf_n2_model1`, `logf_o3_model1_ext`, `logf_o3_model1`, `logf_s2_model1_ext`, `logf_s2_model1`, `logf_hbeta_model3_ext`, `logf_hbeta_model3`, `logf_o2_model3_ext`, `logf_o2_model3`, `logf_n2_model3_ext`, `logf_n2_model3`, `logf_o3_model3_ext`, `logf_o3_model3`, `logf_s2_model3_ext`, `logf_s2_model3`, `blanco_decam_i`, `sdss_r01`, `euclid_nisp_h`, `l_gal`, `b_gal`, `mw_extinction`, `blanco_decam_i_odonnell_ext`, `euclid_nisp_h_odonnell_ext`, `sdss_r01_odonnell_ext`, `blanco_decam_i_el_model1_odonnell_ext`, `blanco_decam_i_el_model3_odonnell_ext`, `sdss_r01_el_model1_odonnell_ext`, `sdss_r01_el_model3_odonnell_ext`, `euclid_nisp_h_el_model1_odonnell_ext`, `euclid_nisp_h_el_model3_odonnell_ext`, `dominant_shape`, `bulge_angle`, `disk_angle`, `median_major_axis`, `scale_length`, `bulge_fraction`, `disk_scalelength`, `disk_nsersic`, `disk_r50`, `bulge_r50`, `bulge_nsersic`, `inclination_angle`, `disk_axis_ratio`, `bulge_axis_ratio`, `disk_ellipticity`, `bulge_ellipticity`, `random_index`, `halo_id`, `galaxy_id`, `step`
FROM jcarrete.flagship_mock_1_9_8_c;

----

ADDING lensing parameters

FIJARME E LOS COMENTARIOS QUE PONGO!!!!!
ESTA NO ES LA DEFINICIÓN FINAL!
MÁS ABAJO LA TENGO ESCRITA PARA CORRERLA

CREATE TABLE jcarrete.flagship_mock_1_9_8_s(
`halo_id` bigint COMMENT 'unique halo id coming from the Flagship dark matter halo catalog',
`galaxy_id` int COMMENT 'combined with the unique_halo_id they uniquely identify each galaxy',
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',
`random_index` float COMMENT 'random number [0 - 1) for subsampling',
`ra_gal` float COMMENT 'galaxy right ascension (degrees)',
`dec_gal` float COMMENT 'galaxy declination (degrees)',

Y ESTOS SON LOS QUE AÑADIMOS

`ra_mag_gal` float COMMENT 'galaxy magnified right ascension (degree)',
`dec_mag_gal` float COMMENT 'galaxy magnified declination (degree)',
`kappa` float COMMENT 'convergence',
`gamma1` float COMMENT 'shear',
`gamma2` float COMMENT 'shear',

`l_gal` float COMMENT 'galactic longitude (degrees)',
`b_gal` float COMMENT 'galactic latitude (degrees)',
`mw_extinction` float COMMENT 'EBV from file: HFI_CompMap_ThermalDustModel_2048_R1.20.fits',
`x_gal` float COMMENT 'galaxy comoving x coordinate (Mpc/h)',
`y_gal` float COMMENT 'galaxy comoving y coordinate (Mpc/h)',
`z_gal` float COMMENT 'galaxy comoving z coordinate (Mpc/h)',
`r_gal` float COMMENT 'galaxy comoving distance (Mpc/h)',
`true_redshift_gal` float COMMENT 'true galaxy redshift',
`observed_redshift_gal` float COMMENT 'observed galaxy redshift (including peculiar velocity)',
`hpix_29_nest` bigint COMMENT 'healpix pixel id using order = 29 in NESTED format', EN EL CATALOGO SE LLAMA hpix_nest BIGINT,
`vx_gal` float COMMENT 'physical (peculiar) velocity in x-component (km/s)',
`vy_gal` float COMMENT 'physical (peculiar) velocity in y-component (km/s)',
`vz_gal` float COMMENT 'physical (peculiar) velocity in z-component (km/s)',
`vrad_gal` float COMMENT 'missing comment',
`delta_r` float COMMENT 'missing comment',
`abs_mag_r01` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h)',
`luminosity_r01` float COMMENT 'luminosity in the 0.1-r-band',
`abs_mag_r01_evolved` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h) including MICECATv2.0 magnitude evolution',
`sdss_r01` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1',
`abs_mag_uv_dereddened` float COMMENT 'absolute magnitude ultraviolet',
`gr_restframe` float COMMENT '(g-r) rest-frame at z=0',
`color_kind` tinyint COMMENT 'galaxy color kind: 0=red sequence 1=green valley 2=blue cloud',
`sed_ke` float COMMENT 'K-correction for each SED',
`gr_cosmos` float COMMENT 'interpolated COSMOS g-r color',
`sed_cosmos` float COMMENT 'interpolated basic COSMOS SED [0 30]',
`is_within_cosmos` boolean COMMENT 'missing comment',
`cosmos_distance` float COMMENT 'missing comment',
`ext_curve_cosmos` tinyint COMMENT 'COSMOS extinction curve [0-4]',
`ebv_cosmos` float COMMENT 'interpolated extinction strength for COSMOS extinction curve',
`log_ml_r01` float COMMENT 'logarithm of stellar_mass/luminosity ratio in the r01 SDSS band',
`log_stellar_mass` float COMMENT 'logarithm of the stellar mass (Msun/h)',
`metallicity` float COMMENT 'oxygen metallicity in units 12+logO/H',
`log_sfr` float COMMENT 'logarithm of star formation rate in (Msun/h)/year',
`logf_halpha` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: DOES NOT include extinction)',
`logf_halpha_ext` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: includes extinction)',
`halpha_scatter` float COMMENT 'missing comment',
`loglum_halpha` float COMMENT 'DEPRECATED logarithm of Halpha luminosity (erg/s/h^2) using empirical relation with UV magnitude (DOES NOT include extinction)',
`loglum_halpha_ext` float COMMENT 'missing comment',
`logf_dummy` float COMMENT 'missing comment',
`z_dummy` float COMMENT 'missing comment',
`logf_halpha_model1_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: includes extinction)',
`logf_halpha_model1` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: DOES NOT include extinction)',
`loglum_halpha_model1_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (includes extinction)',
`loglum_halpha_model1` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (DOES NOT include extinction)',
`logf_halpha_model3_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: includes extinction)',
`logf_halpha_model3` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: DOES NOT include extinction)',
`loglum_halpha_model3_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (includes extinction)',
`loglum_halpha_model3` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (DOES NOT include extinction)',
`logf_hbeta_model1_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_hbeta_model1` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_n2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_n2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o3_model1_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o3_model1` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_s2_model1_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_s2_model1` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_hbeta_model3_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_hbeta_model3` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_n2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_n2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o3_model3_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o3_model3` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_s2_model3_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_s2_model3` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`blanco_decam_i` float COMMENT 'observed flux in DES i-band (Euclid provided)',
`euclid_nisp_h` float COMMENT 'observed flux in Euclid NISP-H band (Euclid provided)',
`blanco_decam_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES i-band (Euclid provided)',
`euclid_nisp_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-H band (Euclid provided)',
`blanco_decam_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES i-band (Euclid provided)',
`euclid_nisp_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-H band (Euclid provided)',
`sdss_r01_odonnell_ext` float COMMENT 'observed flux with MW extinction in in sdss r-band redshifted to z=0.1',
`sdss_r01_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in in sdss r-band redshifted to z=0.1',
`sdss_r01_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in in sdss r-band redshifted to z=0.1',
`dominant_shape` tinyint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)',
`bulge_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`disk_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`median_major_axis` float COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)',
`scale_length` float COMMENT 'disc and bulge scalelength prior',
`bulge_fraction` float COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)',
`disk_scalelength` float COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)',
`disk_nsersic` float COMMENT 'Sersic index of the disk component',
`disk_r50` float COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678',
`bulge_r50` float COMMENT 'half light radius of the bulge',
`bulge_nsersic` float COMMENT 'Sersic index of the bulge component',
`disk_ellipticity` float COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)',
`bulge_ellipticity` float COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)',
`disk_axis_ratio` float COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`bulge_axis_ratio` float COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`inclination_angle` float COMMENT 'Galaxy inclination angle (where 0 degrees = face-on and 90 degrees = edge-on). Galaxy ellipticity for disk and bulge components are computed following the recipe in https://euclid.roe.ac.uk/projects/sgsshear/wiki/SHE-SIM',
`halo_x` float COMMENT 'halo comoving x coordinate (Mpc/h)',
`halo_y` float COMMENT 'halo comoving y coordinate (Mpc/h)',
`halo_z` float COMMENT 'halo comoving z coordinate (Mpc/h)',
`halo_vx` float COMMENT 'halo x-centre velocity (km/h)',
`halo_vy` float COMMENT 'halo y-centre velocity (km/h)',
`halo_vz` float COMMENT 'halo z-centre velocity (km/h)',
`halo_r` float COMMENT 'halo comoving distance (Mpc/h)',
`true_redshift_halo` float COMMENT 'true redshift of the host halo',
`halo_lm` float COMMENT 'log10 of the FoF halo mass in Msum/h. This halo mass is computed as the particule mass multiplied by the number of particles of the FoF halo. At low masses it is interpolated to obtain a smooth distribution of halo masses',
`halo_n_sats` int COMMENT 'number of satellite galaxies from HOD',
`n_gals` int COMMENT 'DEPRECATED',
`luminosity_r01_evolved` float COMMENT 'missing comment',
`hpix_13_nest` bigint COMMENT 'missing comment' ESTE NO ESTÁAAAAA EN LA TABLA!)
PARTITIONED BY (
`step` smallint COMMENT 'step number in the Flagship lensing maps')
CLUSTERED BY (
hpix_13_nest)
SORTED BY (
hpix_13_nest ASC)
INTO 128 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;

DE NUEVO CALCULAR LOS BUCKETS!

CREATE TABLE jcarrete.flagship_mock_1_9_8_s(
`halo_id` bigint COMMENT 'unique halo id coming from the Flagship dark matter halo catalog',
`galaxy_id` int COMMENT 'combined with the unique_halo_id they uniquely identify each galaxy',
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',
`random_index` float COMMENT 'random number [0 - 1) for subsampling',
`ra_gal` float COMMENT 'galaxy right ascension (degrees)',
`dec_gal` float COMMENT 'galaxy declination (degrees)',
`ra_mag_gal` float COMMENT 'galaxy magnified right ascension (degree)',
`dec_mag_gal` float COMMENT 'galaxy magnified declination (degree)',
`kappa` float COMMENT 'convergence',
`gamma1` float COMMENT 'shear',
`gamma2` float COMMENT 'shear',
`l_gal` float COMMENT 'galactic longitude (degrees)',
`b_gal` float COMMENT 'galactic latitude (degrees)',
`mw_extinction` float COMMENT 'EBV from file: HFI_CompMap_ThermalDustModel_2048_R1.20.fits',
`x_gal` float COMMENT 'galaxy comoving x coordinate (Mpc/h)',
`y_gal` float COMMENT 'galaxy comoving y coordinate (Mpc/h)',
`z_gal` float COMMENT 'galaxy comoving z coordinate (Mpc/h)',
`r_gal` float COMMENT 'galaxy comoving distance (Mpc/h)',
`true_redshift_gal` float COMMENT 'true galaxy redshift',
`observed_redshift_gal` float COMMENT 'observed galaxy redshift (including peculiar velocity)',
`hpix_29_nest` bigint COMMENT 'healpix pixel id using order = 29 in NESTED format',
`vx_gal` float COMMENT 'physical (peculiar) velocity in x-component (km/s)',
`vy_gal` float COMMENT 'physical (peculiar) velocity in y-component (km/s)',
`vz_gal` float COMMENT 'physical (peculiar) velocity in z-component (km/s)',
`vrad_gal` float COMMENT 'missing comment',
`delta_r` float COMMENT 'missing comment',
`abs_mag_r01` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h)',
`luminosity_r01` float COMMENT 'luminosity in the 0.1-r-band',
`abs_mag_r01_evolved` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h) including MICECATv2.0 magnitude evolution',
`sdss_r01` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1',
`abs_mag_uv_dereddened` float COMMENT 'absolute magnitude ultraviolet',
`gr_restframe` float COMMENT '(g-r) rest-frame at z=0',
`color_kind` tinyint COMMENT 'galaxy color kind: 0=red sequence 1=green valley 2=blue cloud',
`sed_ke` float COMMENT 'K-correction for each SED',
`gr_cosmos` float COMMENT 'interpolated COSMOS g-r color',
`sed_cosmos` float COMMENT 'interpolated basic COSMOS SED [0 30]',
`is_within_cosmos` boolean COMMENT 'missing comment',
`cosmos_distance` float COMMENT 'missing comment',
`ext_curve_cosmos` tinyint COMMENT 'COSMOS extinction curve [0-4]',
`ebv_cosmos` float COMMENT 'interpolated extinction strength for COSMOS extinction curve',
`log_ml_r01` float COMMENT 'logarithm of stellar_mass/luminosity ratio in the r01 SDSS band',
`log_stellar_mass` float COMMENT 'logarithm of the stellar mass (Msun/h)',
`metallicity` float COMMENT 'oxygen metallicity in units 12+logO/H',
`log_sfr` float COMMENT 'logarithm of star formation rate in (Msun/h)/year',
`logf_halpha` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: DOES NOT include extinction)',
`logf_halpha_ext` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: includes extinction)',
`halpha_scatter` float COMMENT 'missing comment',
`loglum_halpha` float COMMENT 'DEPRECATED logarithm of Halpha luminosity (erg/s/h^2) using empirical relation with UV magnitude (DOES NOT include extinction)',
`loglum_halpha_ext` float COMMENT 'missing comment',
`logf_dummy` float COMMENT 'missing comment',
`z_dummy` float COMMENT 'missing comment',
`logf_halpha_model1_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: includes extinction)',
`logf_halpha_model1` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: DOES NOT include extinction)',
`loglum_halpha_model1_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (includes extinction)',
`loglum_halpha_model1` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (DOES NOT include extinction)',
`logf_halpha_model3_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: includes extinction)',
`logf_halpha_model3` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: DOES NOT include extinction)',
`loglum_halpha_model3_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (includes extinction)',
`loglum_halpha_model3` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (DOES NOT include extinction)',
`logf_hbeta_model1_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_hbeta_model1` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_n2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_n2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o3_model1_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o3_model1` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_s2_model1_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_s2_model1` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_hbeta_model3_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_hbeta_model3` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_n2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_n2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o3_model3_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o3_model3` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_s2_model3_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_s2_model3` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`blanco_decam_i` float COMMENT 'observed flux in DES i-band (Euclid provided)',
`euclid_nisp_h` float COMMENT 'observed flux in Euclid NISP-H band (Euclid provided)',
`blanco_decam_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES i-band (Euclid provided)',
`euclid_nisp_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-H band (Euclid provided)',
`blanco_decam_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES i-band (Euclid provided)',
`euclid_nisp_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-H band (Euclid provided)',
`sdss_r01_odonnell_ext` float COMMENT 'observed flux with MW extinction in in sdss r-band redshifted to z=0.1',
`sdss_r01_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in in sdss r-band redshifted to z=0.1',
`sdss_r01_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in in sdss r-band redshifted to z=0.1',
`dominant_shape` tinyint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)',
`bulge_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`disk_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`median_major_axis` float COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)',
`scale_length` float COMMENT 'disc and bulge scalelength prior',
`bulge_fraction` float COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)',
`disk_scalelength` float COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)',
`disk_nsersic` float COMMENT 'Sersic index of the disk component',
`disk_r50` float COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678',
`bulge_r50` float COMMENT 'half light radius of the bulge',
`bulge_nsersic` float COMMENT 'Sersic index of the bulge component',
`disk_ellipticity` float COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)',
`bulge_ellipticity` float COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)',
`disk_axis_ratio` float COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`bulge_axis_ratio` float COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`inclination_angle` float COMMENT 'Galaxy inclination angle (where 0 degrees = face-on and 90 degrees = edge-on). Galaxy ellipticity for disk and bulge components are computed following the recipe in https://euclid.roe.ac.uk/projects/sgsshear/wiki/SHE-SIM',
`halo_x` float COMMENT 'halo comoving x coordinate (Mpc/h)',
`halo_y` float COMMENT 'halo comoving y coordinate (Mpc/h)',
`halo_z` float COMMENT 'halo comoving z coordinate (Mpc/h)',
`halo_vx` float COMMENT 'halo x-centre velocity (km/h)',
`halo_vy` float COMMENT 'halo y-centre velocity (km/h)',
`halo_vz` float COMMENT 'halo z-centre velocity (km/h)',
`halo_r` float COMMENT 'halo comoving distance (Mpc/h)',
`true_redshift_halo` float COMMENT 'true redshift of the host halo',
`halo_lm` float COMMENT 'log10 of the FoF halo mass in Msum/h. This halo mass is computed as the particule mass multiplied by the number of particles of the FoF halo. At low masses it is interpolated to obtain a smooth distribution of halo masses',
`halo_n_sats` int COMMENT 'number of satellite galaxies from HOD',
`n_gals` int COMMENT 'DEPRECATED',
`luminosity_r01_evolved` float COMMENT 'missing comment',
`hpix_13_nest` bigint COMMENT 'missing comment'
PARTITIONED BY (
`step` smallint COMMENT 'step number in the Flagship lensing maps')
CLUSTERED BY (
hpix_13_nest)
SORTED BY (
hpix_13_nest ASC)
INTO 128 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;

TENGO QUE ENTENDER DONDE SE CALCULAN LOS BUCKETS!

PORQUE NO LOS TENGO EN MI CATALOGO DE BASE, creo!

cat << EOF > lensing_step.sql
SET hive.auto.convert.join.noconditionaltask.size=1;
INSERT OVERWRITE TABLE jcarrete.flagship_mock_1_9_8_s PARTITION (step=${STEP})
SELECT
m.`halo_id`, m.`galaxy_id`, m.`kind`, m.`random_index`, m.`ra_gal`, m.`dec_gal`, magpos.`ra_mag_gal`, magpos.`dec_mag_gal`, m.`kappa`, m.`gamma1`, m.`gamma2`, m.`l_gal`, m.`b_gal`, m.`mw_extinction`, m.`x_gal`, m.`y_gal`, m.`z_gal`, m.`r_gal`, m.`true_redshift_gal`, m.`observed_redshift_gal`, m.`hpix_29_nest`, m.`vx_gal`, m.`vy_gal`, m.`vz_gal`, m.`vrad_gal`, m.`delta_r`, m.`abs_mag_r01`, m.`luminosity_r01`, m.`abs_mag_r01_evolved`, m.`sdss_r01`, m.`abs_mag_uv_dereddened`, m.`gr_restframe`, m.`color_kind`, m.`sed_ke`, m.`gr_cosmos`, m.`sed_cosmos`, m.`is_within_cosmos`, m.`cosmos_distance`, m.`ext_curve_cosmos`, m.`ebv_cosmos`, m.`log_ml_r01`, m.`log_stellar_mass`, m.`metallicity`, m.`log_sfr`, m.`logf_halpha`, m.`logf_halpha_ext`, m.`halpha_scatter`, m.`loglum_halpha`, m.`loglum_halpha_ext`, m.`logf_dummy`, m.`z_dummy`, m.`logf_halpha_model1_ext`, m.`logf_halpha_model1`, m.`loglum_halpha_model1_ext`, m.`loglum_halpha_model1`, m.`logf_halpha_model3_ext`, m.`logf_halpha_model3`, m.`loglum_halpha_model3_ext`, m.`loglum_halpha_model3`, m.`logf_hbeta_model1_ext`, m.`logf_hbeta_model1`, m.`logf_o2_model1_ext`, m.`logf_o2_model1`, m.`logf_n2_model1_ext`, m.`logf_n2_model1`, m.`logf_o3_model1_ext`, m.`logf_o3_model1`, m.`logf_s2_model1_ext`, m.`logf_s2_model1`, m.`logf_hbeta_model3_ext`, m.`logf_hbeta_model3`, m.`logf_o2_model3_ext`, m.`logf_o2_model3`, m.`logf_n2_model3_ext`, m.`logf_n2_model3`, m.`logf_o3_model3_ext`, m.`logf_o3_model3`, m.`logf_s2_model3_ext`, m.`logf_s2_model3`, m.`blanco_decam_i`, m.`euclid_nisp_h`, m.`blanco_decam_i_odonnell_ext`, m.`euclid_nisp_h_odonnell_ext`, m.`blanco_decam_i_el_model1_odonnell_ext`, m.`blanco_decam_i_el_model3_odonnell_ext`, m.`euclid_nisp_h_el_model1_odonnell_ext`, m.`euclid_nisp_h_el_model3_odonnell_ext`, m.`sdss_r01_odonnell_ext`, m.`sdss_r01_el_model1_odonnell_ext`, m.`sdss_r01_el_model3_odonnell_ext`, m.`dominant_shape`, m.`bulge_angle`, m.`disk_angle`, m.`median_major_axis`, m.`scale_length`, m.`bulge_fraction`, m.`disk_scalelength`, m.`disk_nsersic`, m.`disk_r50`, m.`bulge_r50`, m.`bulge_nsersic`, m.`disk_ellipticity`, m.`bulge_ellipticity`, m.`disk_axis_ratio`, m.`bulge_axis_ratio`, m.`inclination_angle`, m.`halo_x`, m.`halo_y`, m.`halo_z`, m.`halo_vx`, m.`halo_vy`, m.`halo_vz`, m.`halo_r`, m.`true_redshift_halo`, m.`halo_lm`, m.`halo_n_sats`, m.`n_gals`, m.`luminosity_r01_evolved`, m.`hpix_13_nest`
FROM (
SELECT
m.`halo_id`, m.`galaxy_id`, m.`kind`, m.`random_index`, m.`ra_gal`, m.`dec_gal`, magpos.`ra_mag_gal`, magpos.`dec_mag_gal`, l.defl_0 as kappa, l.shear_1 as gamma1, l.shear_2 as gamma2, m.`l_gal`, m.`b_gal`, m.`mw_extinction`, m.`x_gal`, m.`y_gal`, m.`z_gal`, m.`r_gal`, m.`true_redshift_gal`, m.`observed_redshift_gal`, m.`hpix_29_nest`, m.`vx_gal`, m.`vy_gal`, m.`vz_gal`, m.`vrad_gal`, m.`delta_r`, m.`abs_mag_r01`, m.`luminosity_r01`, m.`abs_mag_r01_evolved`, m.`sdss_r01`, m.`abs_mag_uv_dereddened`, m.`gr_restframe`, m.`color_kind`, m.`sed_ke`, m.`gr_cosmos`, m.`sed_cosmos`, m.`is_within_cosmos`, m.`cosmos_distance`, m.`ext_curve_cosmos`, m.`ebv_cosmos`, m.`log_ml_r01`, m.`log_stellar_mass`, m.`metallicity`, m.`log_sfr`, m.`logf_halpha`, m.`logf_halpha_ext`, m.`halpha_scatter`, m.`loglum_halpha`, m.`loglum_halpha_ext`, m.`logf_dummy`, m.`z_dummy`, m.`logf_halpha_model1_ext`, m.`logf_halpha_model1`, m.`loglum_halpha_model1_ext`, m.`loglum_halpha_model1`, m.`logf_halpha_model3_ext`, m.`logf_halpha_model3`, m.`loglum_halpha_model3_ext`, m.`loglum_halpha_model3`, m.`logf_hbeta_model1_ext`, m.`logf_hbeta_model1`, m.`logf_o2_model1_ext`, m.`logf_o2_model1`, m.`logf_n2_model1_ext`, m.`logf_n2_model1`, m.`logf_o3_model1_ext`, m.`logf_o3_model1`, m.`logf_s2_model1_ext`, m.`logf_s2_model1`, m.`logf_hbeta_model3_ext`, m.`logf_hbeta_model3`, m.`logf_o2_model3_ext`, m.`logf_o2_model3`, m.`logf_n2_model3_ext`, m.`logf_n2_model3`, m.`logf_o3_model3_ext`, m.`logf_o3_model3`, m.`logf_s2_model3_ext`, m.`logf_s2_model3`, m.`blanco_decam_i`, m.`euclid_nisp_h`, m.`blanco_decam_i_odonnell_ext`, m.`euclid_nisp_h_odonnell_ext`, m.`blanco_decam_i_el_model1_odonnell_ext`, m.`blanco_decam_i_el_model3_odonnell_ext`, m.`euclid_nisp_h_el_model1_odonnell_ext`, m.`euclid_nisp_h_el_model3_odonnell_ext`, m.`sdss_r01_odonnell_ext`, m.`sdss_r01_el_model1_odonnell_ext`, m.`sdss_r01_el_model3_odonnell_ext`, m.`dominant_shape`, m.`bulge_angle`, m.`disk_angle`, m.`median_major_axis`, m.`scale_length`, m.`bulge_fraction`, m.`disk_scalelength`, m.`disk_nsersic`, m.`disk_r50`, m.`bulge_r50`, m.`bulge_nsersic`, m.`disk_ellipticity`, m.`bulge_ellipticity`, m.`disk_axis_ratio`, m.`bulge_axis_ratio`, m.`inclination_angle`, m.`halo_x`, m.`halo_y`, m.`halo_z`, m.`halo_vx`, m.`halo_vy`, m.`halo_vz`, m.`halo_r`, m.`true_redshift_halo`, m.`halo_lm`, m.`halo_n_sats`, m.`n_gals`, m.`luminosity_r01_evolved`, m.`hpix_13_nest`
udf.magnified_positions(CAST(m.ra_gal AS FLOAT), CAST(m.dec_gal AS FLOAT), l.defl_1, l.defl_2) AS magpos
FROM jcarrete.flagship_mock_1_9_8_c2 AS m
JOIN cosmohub.flagship_lensing AS l
ON udf.ang2pix(13, m.ra_gal, m.dec_gal, True, True) = l.hpix
AND m.step = l.step
AND l.step=${STEP} AND m.step=${STEP}
) AS m
;

----

'''Materializing the table:'''

----

SWF1
Area pointing obsid: 52929
ra_range = 7.678307967139321 14.008504032860678
dec_range = -21.587506 -15.587506000000001

Area pointing obsid in 1st octant: 52929
ra_range = 7.678307967139321 14.008504032860678
dec_range = 15.587506000000001 21.587506

----

SWF2
Area pointing obsid: 1717
ra_range = 215.45815488783862 241.3301611121614
dec_range = 73.5904 79.5904

Area pointing obsid in 1st octant: 1717
ra_range = 35.45815488783862 61.33016111216139
dec_range = 73.5904 79.5904

213.074616 251.75015 72.188664 82.03694

CREATE TABLE jcarrete.flagship_mock_obsid_52929_1717_c2(
`kind` BIGINT,
`luminosity_r01` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`abs_mag_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`luminosity_r01_evolved` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` INT,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`hpix_29_nest` BIGINT,
`hpix_13_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`observed_redshift_gal` DOUBLE,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha_ext` DOUBLE,
`logf_halpha` DOUBLE,
`Halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_Halpha_model3_ext` DOUBLE,
`loglum_Halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`blanco_decam_i` DOUBLE,
`sdss_r01` DOUBLE,
`euclid_nisp_h` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`dominant_shape` BIGINT,
`bulge_angle` DOUBLE,
`disk_angle` DOUBLE,
`median_major_axis` DOUBLE,
`scale_length` DOUBLE,
`bulge_fraction` DOUBLE,
`disk_scalelength` DOUBLE,
`disk_nsersic` DOUBLE,
`disk_r50` DOUBLE,
`bulge_r50` DOUBLE,
`bulge_nsersic` DOUBLE,
`inclination_angle` DOUBLE,
`disk_axis_ratio` DOUBLE,
`bulge_axis_ratio` DOUBLE,
`disk_ellipticity` DOUBLE,
`bulge_ellipticity` DOUBLE,
`random_index` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
PARTITIONED BY (
`step` smallint
)
CLUSTERED BY (
hpix_13_nest
)
SORTED BY (
hpix_13_nest ASC
)
INTO 128 BUCKETS
STORED AS ORC
;

----

1) Crear clustered _c3

2) Cambiar el nombre del campo:
hpix_13_nest_mice y calcularlo con el mice convention:

ra_mice = 90 - ra
dec_mice = dec

udf. ang2pix(order =13, ra_mice, dec_mice, nest=true, lonlat=true)

3) En Test:

INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_52929_1717_c2
SELECT
`kind`, `luminosity_r01`, `halo_x`, `halo_y`, `halo_z`, `halo_vx`, `halo_vy`, `halo_vz`, `halo_r`, `true_redshift_halo`, `halo_lm`, `halo_n_sats`, `n_gals`, `abs_mag_r01`, `abs_mag_r01_evolved`, `luminosity_r01_evolved`, `gr_restframe`, `color_kind`, `x_gal`, `y_gal`, `z_gal`, `r_gal`, `true_redshift_gal`, `ra_gal`, `dec_gal`, `hpix_nest`, SHIFTRIGHT(`hpix_nest`, (29-13)*2), `vx_gal`, `vy_gal`, `vz_gal`, `vrad_gal`, `delta_r`, `observed_redshift_gal`, `sed_ke`, `gr_cosmos`, `sed_cosmos`, `ext_curve_cosmos`, `ebv_cosmos`, `is_within_cosmos`, `cosmos_distance`, `abs_mag_uv_dereddened`, `log_ml_r01`, `log_stellar_mass`, `metallicity`, `log_sfr`, `logf_halpha_ext`, `logf_halpha`, `Halpha_scatter`, `loglum_halpha`, `loglum_halpha_ext`, `logf_dummy`, `z_dummy`, `logf_halpha_model1_ext`, `logf_halpha_model1`, `loglum_halpha_model1_ext`, `loglum_halpha_model1`, `logf_halpha_model3_ext`, `logf_halpha_model3`, `loglum_Halpha_model3_ext`, `loglum_Halpha_model3`, `logf_hbeta_model1_ext`, `logf_hbeta_model1`, `logf_o2_model1_ext`, `logf_o2_model1`, `logf_n2_model1_ext`, `logf_n2_model1`, `logf_o3_model1_ext`, `logf_o3_model1`, `logf_s2_model1_ext`, `logf_s2_model1`, `logf_hbeta_model3_ext`, `logf_hbeta_model3`, `logf_o2_model3_ext`, `logf_o2_model3`, `logf_n2_model3_ext`, `logf_n2_model3`, `logf_o3_model3_ext`, `logf_o3_model3`, `logf_s2_model3_ext`, `logf_s2_model3`, `blanco_decam_i`, `sdss_r01`, `euclid_nisp_h`, `l_gal`, `b_gal`, `mw_extinction`, `blanco_decam_i_odonnell_ext`, `euclid_nisp_h_odonnell_ext`, `sdss_r01_odonnell_ext`, `blanco_decam_i_el_model1_odonnell_ext`, `blanco_decam_i_el_model3_odonnell_ext`, `sdss_r01_el_model1_odonnell_ext`, `sdss_r01_el_model3_odonnell_ext`, `euclid_nisp_h_el_model1_odonnell_ext`, `euclid_nisp_h_el_model3_odonnell_ext`, `dominant_shape`, `bulge_angle`, `disk_angle`, `median_major_axis`, `scale_length`, `bulge_fraction`, `disk_scalelength`, `disk_nsersic`, `disk_r50`, `bulge_r50`, `bulge_nsersic`, `inclination_angle`, `disk_axis_ratio`, `bulge_axis_ratio`, `disk_ellipticity`, `bulge_ellipticity`, `random_index`, `halo_id`, `galaxy_id`, `step`
FROM jcarrete.flagship_mock_1_9_8_c
WHERE
(ra_gal >= 7.678307967139321
AND ra_gal < 14.008504032860678
AND dec_gal >= 15.587506
AND dec_gal < 21.587506)
OR
(ra_gal >= 35.45815488783862
AND ra_gal < 61.33016111216139
AND dec_gal >=73.5904
AND dec_gal < 79.5904)
;

ANALYZE TABLE jcarrete.flagship_mock_obsid_52929_1717_c2 COMPUTE STATISTICS FOR COLUMNS;

----

CREATE TABLE jcarrete.flagship_mock_obsid_52929_1717_s(
`halo_id` bigint COMMENT 'unique halo id coming from the Flagship dark matter halo catalog',
`galaxy_id` int COMMENT 'combined with the unique_halo_id they uniquely identify each galaxy',
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',
`random_index` float COMMENT 'random number [0 - 1) for subsampling',
`ra_gal` float COMMENT 'galaxy right ascension (degrees)',
`dec_gal` float COMMENT 'galaxy declination (degrees)',
`ra_mag_gal` float COMMENT 'galaxy magnified right ascension (degree)',
`dec_mag_gal` float COMMENT 'galaxy magnified declination (degree)',
`kappa` float COMMENT 'convergence',
`gamma1` float COMMENT 'shear',
`gamma2` float COMMENT 'shear',
`l_gal` float COMMENT 'galactic longitude (degrees)',
`b_gal` float COMMENT 'galactic latitude (degrees)',
`mw_extinction` float COMMENT 'EBV from file: HFI_CompMap_ThermalDustModel_2048_R1.20.fits',
`x_gal` float COMMENT 'galaxy comoving x coordinate (Mpc/h)',
`y_gal` float COMMENT 'galaxy comoving y coordinate (Mpc/h)',
`z_gal` float COMMENT 'galaxy comoving z coordinate (Mpc/h)',
`r_gal` float COMMENT 'galaxy comoving distance (Mpc/h)',
`true_redshift_gal` float COMMENT 'true galaxy redshift',
`observed_redshift_gal` float COMMENT 'observed galaxy redshift (including peculiar velocity)',
`hpix_29_nest` bigint COMMENT 'healpix pixel id using order = 29 in NESTED format',
`vx_gal` float COMMENT 'physical (peculiar) velocity in x-component (km/s)',
`vy_gal` float COMMENT 'physical (peculiar) velocity in y-component (km/s)',
`vz_gal` float COMMENT 'physical (peculiar) velocity in z-component (km/s)',
`vrad_gal` float COMMENT 'missing comment',
`delta_r` float COMMENT 'missing comment',
`abs_mag_r01` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h)',
`luminosity_r01` float COMMENT 'luminosity in the 0.1-r-band',
`abs_mag_r01_evolved` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h) including MICECATv2.0 magnitude evolution',
`sdss_r01` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1',
`abs_mag_uv_dereddened` float COMMENT 'absolute magnitude ultraviolet',
`gr_restframe` float COMMENT '(g-r) rest-frame at z=0',
`color_kind` tinyint COMMENT 'galaxy color kind: 0=red sequence 1=green valley 2=blue cloud',
`sed_ke` float COMMENT 'K-correction for each SED',
`gr_cosmos` float COMMENT 'interpolated COSMOS g-r color',
`sed_cosmos` float COMMENT 'interpolated basic COSMOS SED [0 30]',
`is_within_cosmos` boolean COMMENT 'missing comment',
`cosmos_distance` float COMMENT 'missing comment',
`ext_curve_cosmos` tinyint COMMENT 'COSMOS extinction curve [0-4]',
`ebv_cosmos` float COMMENT 'interpolated extinction strength for COSMOS extinction curve',
`log_ml_r01` float COMMENT 'logarithm of stellar_mass/luminosity ratio in the r01 SDSS band',
`log_stellar_mass` float COMMENT 'logarithm of the stellar mass (Msun/h)',
`metallicity` float COMMENT 'oxygen metallicity in units 12+logO/H',
`log_sfr` float COMMENT 'logarithm of star formation rate in (Msun/h)/year',
`logf_halpha` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: DOES NOT include extinction)',
`logf_halpha_ext` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: includes extinction)',
`halpha_scatter` float COMMENT 'missing comment',
`loglum_halpha` float COMMENT 'DEPRECATED logarithm of Halpha luminosity (erg/s/h^2) using empirical relation with UV magnitude (DOES NOT include extinction)',
`loglum_halpha_ext` float COMMENT 'missing comment',
`logf_dummy` float COMMENT 'missing comment',
`z_dummy` float COMMENT 'missing comment',
`logf_halpha_model1_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: includes extinction)',
`logf_halpha_model1` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: DOES NOT include extinction)',
`loglum_halpha_model1_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (includes extinction)',
`loglum_halpha_model1` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (DOES NOT include extinction)',
`logf_halpha_model3_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: includes extinction)',
`logf_halpha_model3` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: DOES NOT include extinction)',
`loglum_halpha_model3_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (includes extinction)',
`loglum_halpha_model3` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (DOES NOT include extinction)',
`logf_hbeta_model1_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_hbeta_model1` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_n2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_n2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o3_model1_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o3_model1` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_s2_model1_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_s2_model1` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_hbeta_model3_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_hbeta_model3` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_n2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_n2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o3_model3_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o3_model3` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_s2_model3_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_s2_model3` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`blanco_decam_i` float COMMENT 'observed flux in DES i-band (Euclid provided)',
`euclid_nisp_h` float COMMENT 'observed flux in Euclid NISP-H band (Euclid provided)',
`blanco_decam_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES i-band (Euclid provided)',
`euclid_nisp_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-H band (Euclid provided)',
`blanco_decam_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES i-band (Euclid provided)',
`euclid_nisp_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-H band (Euclid provided)',
`sdss_r01_odonnell_ext` float COMMENT 'observed flux with MW extinction in in sdss r-band redshifted to z=0.1',
`sdss_r01_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in in sdss r-band redshifted to z=0.1',
`sdss_r01_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in in sdss r-band redshifted to z=0.1',
`dominant_shape` tinyint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)',
`bulge_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`disk_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`median_major_axis` float COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)',
`scale_length` float COMMENT 'disc and bulge scalelength prior',
`bulge_fraction` float COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)',
`disk_scalelength` float COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)',
`disk_nsersic` float COMMENT 'Sersic index of the disk component',
`disk_r50` float COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678',
`bulge_r50` float COMMENT 'half light radius of the bulge',
`bulge_nsersic` float COMMENT 'Sersic index of the bulge component',
`disk_ellipticity` float COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)',
`bulge_ellipticity` float COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)',
`disk_axis_ratio` float COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`bulge_axis_ratio` float COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`inclination_angle` float COMMENT 'Galaxy inclination angle (where 0 degrees = face-on and 90 degrees = edge-on). Galaxy ellipticity for disk and bulge components are computed following the recipe in https://euclid.roe.ac.uk/projects/sgsshear/wiki/SHE-SIM',
`halo_x` float COMMENT 'halo comoving x coordinate (Mpc/h)',
`halo_y` float COMMENT 'halo comoving y coordinate (Mpc/h)',
`halo_z` float COMMENT 'halo comoving z coordinate (Mpc/h)',
`halo_vx` float COMMENT 'halo x-centre velocity (km/h)',
`halo_vy` float COMMENT 'halo y-centre velocity (km/h)',
`halo_vz` float COMMENT 'halo z-centre velocity (km/h)',
`halo_r` float COMMENT 'halo comoving distance (Mpc/h)',
`true_redshift_halo` float COMMENT 'true redshift of the host halo',
`halo_lm` float COMMENT 'log10 of the FoF halo mass in Msum/h. This halo mass is computed as the particule mass multiplied by the number of particles of the FoF halo. At low masses it is interpolated to obtain a smooth distribution of halo masses',
`halo_n_sats` int COMMENT 'number of satellite galaxies from HOD',
`n_gals` int COMMENT 'DEPRECATED',
`luminosity_r01_evolved` float COMMENT 'missing comment',
`hpix_13_nest` bigint COMMENT 'missing comment')
PARTITIONED BY (
`step` smallint COMMENT 'step number in the Flagship lensing maps')
CLUSTERED BY (
hpix_13_nest)
SORTED BY (
hpix_13_nest ASC)
INTO 128 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;

----

nano lensing_step.sql

Pego esto:

SET hive.auto.convert.join.noconditionaltask.size=1;
INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_52929_1717_s PARTITION (step=${STEP})
SELECT
m.`halo_id`, m.`galaxy_id`, m.`kind`, m.`random_index`, m.`ra_gal`, m.`dec_gal`, magpos.`ra_mag` as ra_mag_gal, magpos.`dec_mag` as dec_mag_gal, m.`kappa`, m.`gamma1`, m.`gamma2`, m.`l_gal`, m.`b_gal`, m.`mw_extinction`, m.`x_gal`, m.`y_gal`, m.`z_gal`, m.`r_gal`, m.`true_redshift_gal`, m.`observed_redshift_gal`, m.`hpix_29_nest`, m.`vx_gal`, m.`vy_gal`, m.`vz_gal`, m.`vrad_gal`, m.`delta_r`, m.`abs_mag_r01`, m.`luminosity_r01`, m.`abs_mag_r01_evolved`, m.`sdss_r01`, m.`abs_mag_uv_dereddened`, m.`gr_restframe`, m.`color_kind`, m.`sed_ke`, m.`gr_cosmos`, m.`sed_cosmos`, m.`is_within_cosmos`, m.`cosmos_distance`, m.`ext_curve_cosmos`, m.`ebv_cosmos`, m.`log_ml_r01`, m.`log_stellar_mass`, m.`metallicity`, m.`log_sfr`, m.`logf_halpha`, m.`logf_halpha_ext`, m.`halpha_scatter`, m.`loglum_halpha`, m.`loglum_halpha_ext`, m.`logf_dummy`, m.`z_dummy`, m.`logf_halpha_model1_ext`, m.`logf_halpha_model1`, m.`loglum_halpha_model1_ext`, m.`loglum_halpha_model1`, m.`logf_halpha_model3_ext`, m.`logf_halpha_model3`, m.`loglum_halpha_model3_ext`, m.`loglum_halpha_model3`, m.`logf_hbeta_model1_ext`, m.`logf_hbeta_model1`, m.`logf_o2_model1_ext`, m.`logf_o2_model1`, m.`logf_n2_model1_ext`, m.`logf_n2_model1`, m.`logf_o3_model1_ext`, m.`logf_o3_model1`, m.`logf_s2_model1_ext`, m.`logf_s2_model1`, m.`logf_hbeta_model3_ext`, m.`logf_hbeta_model3`, m.`logf_o2_model3_ext`, m.`logf_o2_model3`, m.`logf_n2_model3_ext`, m.`logf_n2_model3`, m.`logf_o3_model3_ext`, m.`logf_o3_model3`, m.`logf_s2_model3_ext`, m.`logf_s2_model3`, m.`blanco_decam_i`, m.`euclid_nisp_h`, m.`blanco_decam_i_odonnell_ext`, m.`euclid_nisp_h_odonnell_ext`, m.`blanco_decam_i_el_model1_odonnell_ext`, m.`blanco_decam_i_el_model3_odonnell_ext`, m.`euclid_nisp_h_el_model1_odonnell_ext`, m.`euclid_nisp_h_el_model3_odonnell_ext`, m.`sdss_r01_odonnell_ext`, m.`sdss_r01_el_model1_odonnell_ext`, m.`sdss_r01_el_model3_odonnell_ext`, m.`dominant_shape`, m.`bulge_angle`, m.`disk_angle`, m.`median_major_axis`, m.`scale_length`, m.`bulge_fraction`, m.`disk_scalelength`, m.`disk_nsersic`, m.`disk_r50`, m.`bulge_r50`, m.`bulge_nsersic`, m.`disk_ellipticity`, m.`bulge_ellipticity`, m.`disk_axis_ratio`, m.`bulge_axis_ratio`, m.`inclination_angle`, m.`halo_x`, m.`halo_y`, m.`halo_z`, m.`halo_vx`, m.`halo_vy`, m.`halo_vz`, m.`halo_r`, m.`true_redshift_halo`, m.`halo_lm`, m.`halo_n_sats`, m.`n_gals`, m.`luminosity_r01_evolved`, m.`hpix_13_nest`
FROM (
SELECT
m.`halo_id`, m.`galaxy_id`, m.`kind`, m.`random_index`, m.`ra_gal`, m.`dec_gal`, l.defl_0 as kappa, l.shear_1 as gamma1, l.shear_2 as gamma2, m.`l_gal`, m.`b_gal`, m.`mw_extinction`, m.`x_gal`, m.`y_gal`, m.`z_gal`, m.`r_gal`, m.`true_redshift_gal`, m.`observed_redshift_gal`, m.`hpix_29_nest`, m.`vx_gal`, m.`vy_gal`, m.`vz_gal`, m.`vrad_gal`, m.`delta_r`, m.`abs_mag_r01`, m.`luminosity_r01`, m.`abs_mag_r01_evolved`, m.`sdss_r01`, m.`abs_mag_uv_dereddened`, m.`gr_restframe`, m.`color_kind`, m.`sed_ke`, m.`gr_cosmos`, m.`sed_cosmos`, m.`is_within_cosmos`, m.`cosmos_distance`, m.`ext_curve_cosmos`, m.`ebv_cosmos`, m.`log_ml_r01`, m.`log_stellar_mass`, m.`metallicity`, m.`log_sfr`, m.`logf_halpha`, m.`logf_halpha_ext`, m.`halpha_scatter`, m.`loglum_halpha`, m.`loglum_halpha_ext`, m.`logf_dummy`, m.`z_dummy`, m.`logf_halpha_model1_ext`, m.`logf_halpha_model1`, m.`loglum_halpha_model1_ext`, m.`loglum_halpha_model1`, m.`logf_halpha_model3_ext`, m.`logf_halpha_model3`, m.`loglum_halpha_model3_ext`, m.`loglum_halpha_model3`, m.`logf_hbeta_model1_ext`, m.`logf_hbeta_model1`, m.`logf_o2_model1_ext`, m.`logf_o2_model1`, m.`logf_n2_model1_ext`, m.`logf_n2_model1`, m.`logf_o3_model1_ext`, m.`logf_o3_model1`, m.`logf_s2_model1_ext`, m.`logf_s2_model1`, m.`logf_hbeta_model3_ext`, m.`logf_hbeta_model3`, m.`logf_o2_model3_ext`, m.`logf_o2_model3`, m.`logf_n2_model3_ext`, m.`logf_n2_model3`, m.`logf_o3_model3_ext`, m.`logf_o3_model3`, m.`logf_s2_model3_ext`, m.`logf_s2_model3`, m.`blanco_decam_i`, m.`euclid_nisp_h`, m.`blanco_decam_i_odonnell_ext`, m.`euclid_nisp_h_odonnell_ext`, m.`blanco_decam_i_el_model1_odonnell_ext`, m.`blanco_decam_i_el_model3_odonnell_ext`, m.`euclid_nisp_h_el_model1_odonnell_ext`, m.`euclid_nisp_h_el_model3_odonnell_ext`, m.`sdss_r01_odonnell_ext`, m.`sdss_r01_el_model1_odonnell_ext`, m.`sdss_r01_el_model3_odonnell_ext`, m.`dominant_shape`, m.`bulge_angle`, m.`disk_angle`, m.`median_major_axis`, m.`scale_length`, m.`bulge_fraction`, m.`disk_scalelength`, m.`disk_nsersic`, m.`disk_r50`, m.`bulge_r50`, m.`bulge_nsersic`, m.`disk_ellipticity`, m.`bulge_ellipticity`, m.`disk_axis_ratio`, m.`bulge_axis_ratio`, m.`inclination_angle`, m.`halo_x`, m.`halo_y`, m.`halo_z`, m.`halo_vx`, m.`halo_vy`, m.`halo_vz`, m.`halo_r`, m.`true_redshift_halo`, m.`halo_lm`, m.`halo_n_sats`, m.`n_gals`, m.`luminosity_r01_evolved`, m.`hpix_13_nest`,
udf.magnified_positions(CAST(m.ra_gal AS FLOAT), CAST(m.dec_gal AS FLOAT), l.defl_1, l.defl_2) AS magpos
FROM jcarrete.flagship_mock_obsid_52929_1717_c2 AS m
JOIN cosmohub.flagship_lensing AS l
ON udf.ang2pix(13, m.ra_gal, m.dec_gal, True, True) = l.hpix
AND m.step = l.step
AND l.step=${STEP} AND m.step=${STEP}
) AS m
;

----

for STEP in $(seq 87 400);
do
echo $STEP
sem -j4 "beeline --color=true --showHeader=true --verbose=true --silent=false -u \
'jdbc:hive2://ambarisrv02.pic.es:2181,ambarisrv03.pic.es:2181,ambarisrv01.pic.es:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2' \
-f lensing_step.sql --hivevar STEP=${STEP}"
done
sem --wait

----

Once the lensing is added to the clustered table that contains two of the paintings, I run the flux_pipeline, in which we have updated the euclid filters!
I do it for the 1717 observation since at this time I do not know what to do with the area that has to be rotated to the southern hemisphere (which is the obsid= 52929).

----

This is done once the catalog is generated with the flux_pipeline.ipynb

/software/astro/scripts/create_table_from_parquet.sh -p /hdfs/user/jcarrete/data/euclid/flagship_mock_obsid_1717_s/0.pq -t jcarrete.flagship_mock_obsid_1717_s_pq -e

CREATE EXTERNAL TABLE jcarrete.flagship_mock_obsid_1717_s_pq (
`kind` BIGINT,
`random_index` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`ra_mag_gal` DOUBLE,
`dec_mag_gal` DOUBLE,
`kappa` DOUBLE,
`gamma1` DOUBLE,
`gamma2` DOUBLE,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`observed_redshift_gal` DOUBLE,
`hpix_29_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`abs_mag_r01` DOUBLE,
`luminosity_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`sdss_r01` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` BIGINT,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha` DOUBLE,
`logf_halpha_ext` DOUBLE,
`halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_halpha_model3_ext` DOUBLE,
`loglum_halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`dominant_shape` BIGINT,
`bulge_angle` DOUBLE,
`disk_angle` DOUBLE,
`median_major_axis` DOUBLE,
`scale_length` DOUBLE,
`bulge_fraction` DOUBLE,
`disk_scalelength` DOUBLE,
`disk_nsersic` DOUBLE,
`disk_r50` DOUBLE,
`bulge_r50` DOUBLE,
`bulge_nsersic` DOUBLE,
`disk_ellipticity` DOUBLE,
`bulge_ellipticity` DOUBLE,
`disk_axis_ratio` DOUBLE,
`bulge_axis_ratio` DOUBLE,
`inclination_angle` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`luminosity_r01_evolved` DOUBLE,
`hpix_13_nest` BIGINT,
`step` BIGINT,
`blanco_decam_g` DOUBLE,
`blanco_decam_i` DOUBLE,
`blanco_decam_r` DOUBLE,
`blanco_decam_z` DOUBLE,
`subaru_hsc_z` DOUBLE,
`jst_jpcam_g` DOUBLE,
`kids_g` DOUBLE,
`kids_i` DOUBLE,
`kids_r` DOUBLE,
`kids_u` DOUBLE,
`lsst_g` DOUBLE,
`lsst_i` DOUBLE,
`lsst_r` DOUBLE,
`lsst_u` DOUBLE,
`lsst_y` DOUBLE,
`lsst_z` DOUBLE,
`cfht_megacam_r` DOUBLE,
`cfht_megacam_u` DOUBLE,
`pan_starrs_i` DOUBLE,
`pan_starrs_z` DOUBLE,
`2mass_h` DOUBLE,
`2mass_j` DOUBLE,
`2mass_ks` DOUBLE,
`gaia_bp` DOUBLE,
`gaia_g` DOUBLE,
`gaia_rp` DOUBLE,
`euclid_nisp_h` DOUBLE,
`euclid_nisp_j` DOUBLE,
`euclid_nisp_y` DOUBLE,
`euclid_vis` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`2mass_h_odonnell_ext` DOUBLE,
`2mass_j_odonnell_ext` DOUBLE,
`2mass_ks_odonnell_ext` DOUBLE,
`blanco_decam_g_odonnell_ext` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`blanco_decam_r_odonnell_ext` DOUBLE,
`blanco_decam_z_odonnell_ext` DOUBLE,
`cfht_megacam_r_odonnell_ext` DOUBLE,
`cfht_megacam_u_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`euclid_nisp_j_odonnell_ext` DOUBLE,
`euclid_nisp_y_odonnell_ext` DOUBLE,
`euclid_vis_odonnell_ext` DOUBLE,
`gaia_bp_odonnell_ext` DOUBLE,
`gaia_g_odonnell_ext` DOUBLE,
`gaia_rp_odonnell_ext` DOUBLE,
`jst_jpcam_g_odonnell_ext` DOUBLE,
`kids_g_odonnell_ext` DOUBLE,
`kids_i_odonnell_ext` DOUBLE,
`kids_r_odonnell_ext` DOUBLE,
`kids_u_odonnell_ext` DOUBLE,
`lsst_g_odonnell_ext` DOUBLE,
`lsst_i_odonnell_ext` DOUBLE,
`lsst_r_odonnell_ext` DOUBLE,
`lsst_u_odonnell_ext` DOUBLE,
`lsst_y_odonnell_ext` DOUBLE,
`lsst_z_odonnell_ext` DOUBLE,
`pan_starrs_i_odonnell_ext` DOUBLE,
`pan_starrs_z_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`subaru_hsc_z_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model3_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model1_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model3_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model1_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model3_odonnell_ext` DOUBLE,
`kids_g_el_model1_odonnell_ext` DOUBLE,
`kids_g_el_model3_odonnell_ext` DOUBLE,
`kids_i_el_model1_odonnell_ext` DOUBLE,
`kids_i_el_model3_odonnell_ext` DOUBLE,
`kids_r_el_model1_odonnell_ext` DOUBLE,
`kids_r_el_model3_odonnell_ext` DOUBLE,
`kids_u_el_model1_odonnell_ext` DOUBLE,
`kids_u_el_model3_odonnell_ext` DOUBLE,
`lsst_g_el_model1_odonnell_ext` DOUBLE,
`lsst_g_el_model3_odonnell_ext` DOUBLE,
`lsst_i_el_model1_odonnell_ext` DOUBLE,
`lsst_i_el_model3_odonnell_ext` DOUBLE,
`lsst_r_el_model1_odonnell_ext` DOUBLE,
`lsst_r_el_model3_odonnell_ext` DOUBLE,
`lsst_u_el_model1_odonnell_ext` DOUBLE,
`lsst_u_el_model3_odonnell_ext` DOUBLE,
`lsst_y_el_model1_odonnell_ext` DOUBLE,
`lsst_y_el_model3_odonnell_ext` DOUBLE,
`lsst_z_el_model1_odonnell_ext` DOUBLE,
`lsst_z_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model3_odonnell_ext` DOUBLE,
`2mass_h_el_model1_odonnell_ext` DOUBLE,
`2mass_h_el_model3_odonnell_ext` DOUBLE,
`2mass_j_el_model1_odonnell_ext` DOUBLE,
`2mass_j_el_model3_odonnell_ext` DOUBLE,
`2mass_ks_el_model1_odonnell_ext` DOUBLE,
`2mass_ks_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`gaia_bp_el_model1_odonnell_ext` DOUBLE,
`gaia_bp_el_model3_odonnell_ext` DOUBLE,
`gaia_g_el_model1_odonnell_ext` DOUBLE,
`gaia_g_el_model3_odonnell_ext` DOUBLE,
`gaia_rp_el_model1_odonnell_ext` DOUBLE,
`gaia_rp_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model3_odonnell_ext` DOUBLE,
`euclid_vis_el_model1_odonnell_ext` DOUBLE,
`euclid_vis_el_model3_odonnell_ext` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
STORED AS PARQUET
LOCATION '/user/jcarrete/data/euclid/flagship_mock_obsid_1717_s'
;

----

CREATE TABLE jcarrete.flagship_mock_obsid_1717_s(
`halo_id` bigint COMMENT 'unique halo id coming from the Flagship dark matter halo catalog',
`galaxy_id` int COMMENT 'combined with the unique_halo_id they uniquely identify each galaxy',
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',
`random_index` float COMMENT 'random number [0 - 1) for subsampling',
`ra_gal` float COMMENT 'galaxy right ascension (degrees)',
`dec_gal` float COMMENT 'galaxy declination (degrees)',
`ra_mag_gal` float COMMENT 'galaxy magnified right ascension (degree)',
`dec_mag_gal` float COMMENT 'galaxy magnified declination (degree)',
`kappa` float COMMENT 'convergence',
`gamma1` float COMMENT 'shear',
`gamma2` float COMMENT 'shear',
`l_gal` float COMMENT 'galactic longitude (degrees)',
`b_gal` float COMMENT 'galactic latitude (degrees)',
`mw_extinction` float COMMENT 'EBV from file: HFI_CompMap_ThermalDustModel_2048_R1.20.fits',
`x_gal` float COMMENT 'galaxy comoving x coordinate (Mpc/h)',
`y_gal` float COMMENT 'galaxy comoving y coordinate (Mpc/h)',
`z_gal` float COMMENT 'galaxy comoving z coordinate (Mpc/h)',
`r_gal` float COMMENT 'galaxy comoving distance (Mpc/h)',
`true_redshift_gal` float COMMENT 'true galaxy redshift',
`observed_redshift_gal` float COMMENT 'observed galaxy redshift (including peculiar velocity)',
`hpix_29_nest` bigint COMMENT 'healpix pixel id using order = 29 in NESTED format',
`vx_gal` float COMMENT 'physical (peculiar) velocity in x-component (km/s)',
`vy_gal` float COMMENT 'physical (peculiar) velocity in y-component (km/s)',
`vz_gal` float COMMENT 'physical (peculiar) velocity in z-component (km/s)',
`vrad_gal` float COMMENT 'missing comment',
`delta_r` float COMMENT 'missing comment',
`abs_mag_r01` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h)',
`luminosity_r01` float COMMENT 'luminosity in the 0.1-r-band',
`abs_mag_r01_evolved` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h) including MICECATv2.0 magnitude evolution',
`sdss_r01` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1',
`abs_mag_uv_dereddened` float COMMENT 'absolute magnitude ultraviolet',
`gr_restframe` float COMMENT '(g-r) rest-frame at z=0',
`color_kind` tinyint COMMENT 'galaxy color kind: 0=red sequence 1=green valley 2=blue cloud',
`sed_ke` float COMMENT 'K-correction for each SED',
`gr_cosmos` float COMMENT 'interpolated COSMOS g-r color',
`sed_cosmos` float COMMENT 'interpolated basic COSMOS SED [0 30]',
`is_within_cosmos` boolean COMMENT 'missing comment',
`cosmos_distance` float COMMENT 'missing comment',
`ext_curve_cosmos` tinyint COMMENT 'COSMOS extinction curve [0-4]',
`ebv_cosmos` float COMMENT 'interpolated extinction strength for COSMOS extinction curve',
`log_ml_r01` float COMMENT 'logarithm of stellar_mass/luminosity ratio in the r01 SDSS band',
`log_stellar_mass` float COMMENT 'logarithm of the stellar mass (Msun/h)',
`metallicity` float COMMENT 'oxygen metallicity in units 12+logO/H',
`log_sfr` float COMMENT 'logarithm of star formation rate in (Msun/h)/year',
`logf_halpha` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: DOES NOT include extinction)',
`logf_halpha_ext` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: includes extinction)',
`halpha_scatter` float COMMENT 'missing comment',
`loglum_halpha` float COMMENT 'DEPRECATED logarithm of Halpha luminosity (erg/s/h^2) using empirical relation with UV magnitude (DOES NOT include extinction)',
`loglum_halpha_ext` float COMMENT 'missing comment',
`logf_dummy` float COMMENT 'missing comment',
`z_dummy` float COMMENT 'missing comment',
`logf_halpha_model1_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: includes extinction)',
`logf_halpha_model1` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: DOES NOT include extinction)',
`loglum_halpha_model1_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (includes extinction)',
`loglum_halpha_model1` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (DOES NOT include extinction)',
`logf_halpha_model3_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: includes extinction)',
`logf_halpha_model3` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: DOES NOT include extinction)',
`loglum_halpha_model3_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (includes extinction)',
`loglum_halpha_model3` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (DOES NOT include extinction)',
`logf_hbeta_model1_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_hbeta_model1` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_n2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_n2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o3_model1_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o3_model1` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_s2_model1_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_s2_model1` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_hbeta_model3_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_hbeta_model3` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_n2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_n2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o3_model3_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o3_model3` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_s2_model3_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_s2_model3` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',

`2mass_h` float COMMENT 'observed flux in 2MASS H-band (Euclid provided)',
`2mass_j` float COMMENT 'observed flux in 2MASS J-band (Euclid provided)',
`2mass_ks` float COMMENT 'observed flux in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g` float COMMENT 'observed flux in DES g-band (Euclid provided)',
`blanco_decam_i` float COMMENT 'observed flux in DES i-band (Euclid provided)',
`blanco_decam_r` float COMMENT 'observed flux in DES r-band (Euclid provided)',
`blanco_decam_z` float COMMENT 'observed flux in DES z-band (Euclid provided)',
`cfht_megacam_r` float COMMENT 'observed flux in CFHT r-band (Euclid provided)',
`cfht_megacam_u` float COMMENT 'observed flux in CFHT u-band (Euclid provided)',
`euclid_nisp_h` float COMMENT 'observed flux in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j` float COMMENT 'observed flux in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y` float COMMENT 'observed flux in Euclid NISP-Y band (Euclid provided)',
`euclid_vis` float COMMENT 'observed flux in Euclid VIS band (Euclid provided)',
`gaia_bp` float COMMENT 'observed flux in Gaia-BP band (Euclid provided)',
`gaia_g` float COMMENT 'observed flux in Gaia-G band (Euclid provided)',
`gaia_rp` float COMMENT 'observed flux in Gaia-RP band (Euclid provided)',
`jst_jpcam_g` float COMMENT 'observed flux in JST g-band (Euclid provided)',
`kids_g` float COMMENT 'observed flux in KIDS g-band (Euclid provided)',
`kids_i` float COMMENT 'observed flux in KIDS i-band (Euclid provided)',
`kids_r` float COMMENT 'observed flux in KIDS r-band (Euclid provided)',
`kids_u` float COMMENT 'observed flux in KIDS u-band (Euclid provided)',
`lsst_g` float COMMENT 'observed flux in LSST g-band (Euclid provided)',
`lsst_i` float COMMENT 'observed flux in LSST i-band (Euclid provided)',
`lsst_r` float COMMENT 'observed flux in LSST r-band (Euclid provided)',
`lsst_u` float COMMENT 'observed flux in LSST u-band (Euclid provided)',
`lsst_y` float COMMENT 'observed flux in LSST Y-band (Euclid provided)',
`lsst_z` float COMMENT 'observed flux in LSST z-band (Euclid provided)',
`pan_starrs_i` float COMMENT 'observed flux in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z` float COMMENT 'observed flux in Pan-STARRS z-band (Euclid provided)',
`subaru_hsc_z` float COMMENT 'observed flux in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS H-band (Euclid provided)',
`2mass_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in JST g-band (Euclid provided)',
`kids_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_odonnell_ext` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1 with MW extinction',
`subaru_hsc_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS Ks-band (Euclid provided)',
`2mass_ks_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES z-band (Euclid provided)',
`blanco_decam_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT u-band (Euclid provided)',
`cfht_megacam_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_nisp_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid VIS band (Euclid provided)',
`euclid_vis_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_bp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-RP band (Euclid provided)',
`gaia_rp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in JST g-band (Euclid provided)',
`jst_jpcam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in JST g-band (Euclid provided)',
`kids_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS g-band (Euclid provided)',
`kids_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS i-band (Euclid provided)',
`kids_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS r-band (Euclid provided)',
`kids_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS u-band (Euclid provided)',
`kids_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST g-band (Euclid provided)',
`lsst_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST i-band (Euclid provided)',
`lsst_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST r-band (Euclid provided)',
`lsst_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST u-band (Euclid provided)',
`lsst_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST z-band (Euclid provided)',
`lsst_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`pan_starrs_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in sdss r-band redshifted to z=0.1',
`sdss_r01_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in sdss r-band redshifted to z=0.1',
`subaru_hsc_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`subaru_hsc_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',

`dominant_shape` tinyint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)',
`bulge_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`disk_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`median_major_axis` float COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)',
`scale_length` float COMMENT 'disc and bulge scalelength prior',
`bulge_fraction` float COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)',
`disk_scalelength` float COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)',
`disk_nsersic` float COMMENT 'Sersic index of the disk component',
`disk_r50` float COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678',
`bulge_r50` float COMMENT 'half light radius of the bulge',
`bulge_nsersic` float COMMENT 'Sersic index of the bulge component',
`disk_ellipticity` float COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)',
`bulge_ellipticity` float COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)',
`disk_axis_ratio` float COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`bulge_axis_ratio` float COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`inclination_angle` float COMMENT 'Galaxy inclination angle (where 0 degrees = face-on and 90 degrees = edge-on). Galaxy ellipticity for disk and bulge components are computed following the recipe in https://euclid.roe.ac.uk/projects/sgsshear/wiki/SHE-SIM',
`halo_x` float COMMENT 'halo comoving x coordinate (Mpc/h)',
`halo_y` float COMMENT 'halo comoving y coordinate (Mpc/h)',
`halo_z` float COMMENT 'halo comoving z coordinate (Mpc/h)',
`halo_vx` float COMMENT 'halo x-centre velocity (km/h)',
`halo_vy` float COMMENT 'halo y-centre velocity (km/h)',
`halo_vz` float COMMENT 'halo z-centre velocity (km/h)',
`halo_r` float COMMENT 'halo comoving distance (Mpc/h)',
`true_redshift_halo` float COMMENT 'true redshift of the host halo',
`halo_lm` float COMMENT 'log10 of the FoF halo mass in Msum/h. This halo mass is computed as the particule mass multiplied by the number of particles of the FoF halo. At low masses it is interpolated to obtain a smooth distribution of halo masses',
`halo_n_sats` int COMMENT 'number of satellite galaxies from HOD',
`n_gals` int COMMENT 'DEPRECATED',
`luminosity_r01_evolved` float COMMENT 'missing comment',
`hpix_13_nest` bigint COMMENT 'missing comment')
PARTITIONED BY (
`step` smallint COMMENT 'step number in the Flagship lensing maps')
CLUSTERED BY (
hpix_13_nest)
SORTED BY (
hpix_13_nest ASC)
INTO 128 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;

INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_1717_s PARTITION(step)
SELECT halo_id, galaxy_id, kind, random_index, ra_gal, dec_gal, ra_mag_gal, dec_mag_gal, kappa, gamma1, gamma2, l_gal, b_gal, mw_extinction, x_gal, y_gal, z_gal, r_gal, true_redshift_gal, observed_redshift_gal, hpix_29_nest, vx_gal, vy_gal, vz_gal, vrad_gal, delta_r, abs_mag_r01, luminosity_r01, abs_mag_r01_evolved, sdss_r01, abs_mag_uv_dereddened, gr_restframe, color_kind, sed_ke, gr_cosmos, sed_cosmos, is_within_cosmos, cosmos_distance, ext_curve_cosmos, ebv_cosmos, log_ml_r01, log_stellar_mass, metallicity, log_sfr, logf_halpha, logf_halpha_ext, halpha_scatter, loglum_halpha, loglum_halpha_ext, logf_dummy, z_dummy, logf_halpha_model1_ext, logf_halpha_model1, loglum_halpha_model1_ext, loglum_halpha_model1, logf_halpha_model3_ext, logf_halpha_model3, loglum_halpha_model3_ext, loglum_halpha_model3, logf_hbeta_model1_ext, logf_hbeta_model1, logf_o2_model1_ext, logf_o2_model1, logf_n2_model1_ext, logf_n2_model1, logf_o3_model1_ext, logf_o3_model1, logf_s2_model1_ext, logf_s2_model1, logf_hbeta_model3_ext, logf_hbeta_model3, logf_o2_model3_ext, logf_o2_model3, logf_n2_model3_ext, logf_n2_model3, logf_o3_model3_ext, logf_o3_model3, logf_s2_model3_ext, logf_s2_model3, 2mass_h, 2mass_j, 2mass_ks, blanco_decam_g, blanco_decam_i, blanco_decam_r, blanco_decam_z, cfht_megacam_r, cfht_megacam_u, euclid_nisp_h, euclid_nisp_j, euclid_nisp_y, euclid_vis, gaia_bp, gaia_g, gaia_rp, jst_jpcam_g, kids_g, kids_i, kids_r, kids_u, lsst_g, lsst_i, lsst_r, lsst_u, lsst_y, lsst_z, pan_starrs_i, pan_starrs_z, subaru_hsc_z, 2mass_h_odonnell_ext, 2mass_j_odonnell_ext, 2mass_ks_odonnell_ext, blanco_decam_g_odonnell_ext, blanco_decam_i_odonnell_ext, blanco_decam_r_odonnell_ext, blanco_decam_z_odonnell_ext, cfht_megacam_r_odonnell_ext, cfht_megacam_u_odonnell_ext, euclid_nisp_h_odonnell_ext, euclid_nisp_j_odonnell_ext, euclid_nisp_y_odonnell_ext, euclid_vis_odonnell_ext, gaia_bp_odonnell_ext, gaia_g_odonnell_ext, gaia_rp_odonnell_ext, jst_jpcam_g_odonnell_ext, kids_g_odonnell_ext, kids_i_odonnell_ext, kids_r_odonnell_ext, kids_u_odonnell_ext, lsst_g_odonnell_ext, lsst_i_odonnell_ext, lsst_r_odonnell_ext, lsst_u_odonnell_ext, lsst_y_odonnell_ext, lsst_z_odonnell_ext, pan_starrs_i_odonnell_ext, pan_starrs_z_odonnell_ext, sdss_r01_odonnell_ext, subaru_hsc_z_odonnell_ext, 2mass_h_el_model1_odonnell_ext, 2mass_h_el_model3_odonnell_ext, 2mass_j_el_model1_odonnell_ext, 2mass_j_el_model3_odonnell_ext, 2mass_ks_el_model1_odonnell_ext, 2mass_ks_el_model3_odonnell_ext, blanco_decam_g_el_model1_odonnell_ext, blanco_decam_g_el_model3_odonnell_ext, blanco_decam_i_el_model1_odonnell_ext, blanco_decam_i_el_model3_odonnell_ext, blanco_decam_r_el_model1_odonnell_ext, blanco_decam_r_el_model3_odonnell_ext, blanco_decam_z_el_model1_odonnell_ext, blanco_decam_z_el_model3_odonnell_ext, cfht_megacam_r_el_model1_odonnell_ext, cfht_megacam_r_el_model3_odonnell_ext, cfht_megacam_u_el_model1_odonnell_ext, cfht_megacam_u_el_model3_odonnell_ext, euclid_nisp_h_el_model1_odonnell_ext, euclid_nisp_h_el_model3_odonnell_ext, euclid_nisp_j_el_model1_odonnell_ext, euclid_nisp_j_el_model3_odonnell_ext, euclid_nisp_y_el_model1_odonnell_ext, euclid_nisp_y_el_model3_odonnell_ext, euclid_vis_el_model1_odonnell_ext, euclid_vis_el_model3_odonnell_ext, gaia_bp_el_model1_odonnell_ext, gaia_bp_el_model3_odonnell_ext, gaia_g_el_model1_odonnell_ext, gaia_g_el_model3_odonnell_ext, gaia_rp_el_model1_odonnell_ext, gaia_rp_el_model3_odonnell_ext, jst_jpcam_g_el_model1_odonnell_ext, jst_jpcam_g_el_model3_odonnell_ext, kids_g_el_model1_odonnell_ext, kids_g_el_model3_odonnell_ext, kids_i_el_model1_odonnell_ext, kids_i_el_model3_odonnell_ext, kids_r_el_model1_odonnell_ext, kids_r_el_model3_odonnell_ext, kids_u_el_model1_odonnell_ext, kids_u_el_model3_odonnell_ext, lsst_g_el_model1_odonnell_ext, lsst_g_el_model3_odonnell_ext, lsst_i_el_model1_odonnell_ext, lsst_i_el_model3_odonnell_ext, lsst_r_el_model1_odonnell_ext, lsst_r_el_model3_odonnell_ext, lsst_u_el_model1_odonnell_ext, lsst_u_el_model3_odonnell_ext, lsst_y_el_model1_odonnell_ext, lsst_y_el_model3_odonnell_ext, lsst_z_el_model1_odonnell_ext, lsst_z_el_model3_odonnell_ext, pan_starrs_i_el_model1_odonnell_ext, pan_starrs_i_el_model3_odonnell_ext, pan_starrs_z_el_model1_odonnell_ext, pan_starrs_z_el_model3_odonnell_ext, sdss_r01_el_model1_odonnell_ext, sdss_r01_el_model3_odonnell_ext, subaru_hsc_z_el_model1_odonnell_ext, subaru_hsc_z_el_model3_odonnell_ext, dominant_shape, bulge_angle, disk_angle, median_major_axis, scale_length, bulge_fraction, disk_scalelength, disk_nsersic, disk_r50, bulge_r50, bulge_nsersic, disk_ellipticity, bulge_ellipticity, disk_axis_ratio, bulge_axis_ratio, inclination_angle, halo_x, halo_y, halo_z, halo_vx, halo_vy, halo_vz, halo_r, true_redshift_halo, halo_lm, halo_n_sats, n_gals, luminosity_r01_evolved, hpix_13_nest, step
FROM jcarrete.flagship_mock_obsid_1717_s_pq

CREATE TABLE jcarrete.flagship_mock_obsid_1717_s_magnified_fits(
`source_id` bigint COMMENT 'from deserializer',
`halo_id` bigint COMMENT 'from deserializer',
`ra` float COMMENT 'from deserializer',
`dec` float COMMENT 'from deserializer',
`ra_mag` float COMMENT 'from deserializer',
`dec_mag` float COMMENT 'from deserializer',
`z_obs` float COMMENT 'from deserializer',
`ref_mag_abs` float COMMENT 'from deserializer',
`ref_mag` float COMMENT 'from deserializer',
`bulge_fraction` float COMMENT 'from deserializer',
`bulge_r50` float COMMENT 'from deserializer',
`disk_r50` float COMMENT 'from deserializer',
`bulge_nsersic` float COMMENT 'from deserializer',
`inclination_angle` float COMMENT 'from deserializer',
`disk_angle` float COMMENT 'from deserializer',
`kappa` float COMMENT 'from deserializer',
`gamma1` float COMMENT 'from deserializer',
`gamma2` float COMMENT 'from deserializer',
`sed_template` float COMMENT 'from deserializer',
`ext_law` smallint COMMENT 'from deserializer',
`ebv` float COMMENT 'from deserializer',
`halpha_logflam_ext` float COMMENT 'from deserializer',
`hbeta_logflam_ext` float COMMENT 'from deserializer',
`o2_logflam_ext` float COMMENT 'from deserializer',
`o3_logflam_ext` float COMMENT 'from deserializer',
`n2_logflam_ext` float COMMENT 'from deserializer',
`s2_logflam_ext` float COMMENT 'from deserializer',
`av` float COMMENT 'from deserializer',
`tu_fnu_vis` float COMMENT 'from deserializer',
`tu_fnu_y_nisp` float COMMENT 'from deserializer',
`tu_fnu_j_nisp` float COMMENT 'from deserializer',
`tu_fnu_h_nisp` float COMMENT 'from deserializer',
`tu_fnu_g_decam` float COMMENT 'from deserializer',
`tu_fnu_r_decam` float COMMENT 'from deserializer',
`tu_fnu_i_decam` float COMMENT 'from deserializer',
`tu_fnu_z_decam` float COMMENT 'from deserializer',
`tu_fnu_u_megacam` float COMMENT 'from deserializer',
`tu_fnu_r_megacam` float COMMENT 'from deserializer',
`tu_fnu_g_jpcam` float COMMENT 'from deserializer',
`tu_fnu_i_panstarrs` float COMMENT 'from deserializer',
`tu_fnu_z_panstarrs` float COMMENT 'from deserializer',
`tu_fnu_z_hsc` float COMMENT 'from deserializer',
`tu_fnu_g_gaia` float COMMENT 'from deserializer',
`tu_fnu_bp_gaia` float COMMENT 'from deserializer',
`tu_fnu_rp_gaia` float COMMENT 'from deserializer',
`tu_fnu_u_lsst` float COMMENT 'from deserializer',
`tu_fnu_g_lsst` float COMMENT 'from deserializer',
`tu_fnu_r_lsst` float COMMENT 'from deserializer',
`tu_fnu_i_lsst` float COMMENT 'from deserializer',
`tu_fnu_z_lsst` float COMMENT 'from deserializer',
`tu_fnu_y_lsst` float COMMENT 'from deserializer',
`tu_fnu_u_kids` float COMMENT 'from deserializer',
`tu_fnu_g_kids` float COMMENT 'from deserializer',
`tu_fnu_r_kids` float COMMENT 'from deserializer',
`tu_fnu_i_kids` float COMMENT 'from deserializer',
`tu_fnu_j_2mass` float COMMENT 'from deserializer',
`tu_fnu_h_2mass` float COMMENT 'from deserializer',
`tu_fnu_ks_2mass` float COMMENT 'from deserializer')
PARTITIONED BY (
`hpix_5_nest` bigint)
ROW FORMAT SERDE
'es.pic.astro.hadoop.serde.RecArraySerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'es.pic.astro.hadoop.io.BinaryOutputFormat'
;

INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_1717_s_magnified_fits PARTITION(hpix_5_nest)
SELECT CAST(((halo_id * 10000) + galaxy_id) AS bigint) AS SOURCE_ID,
CAST(halo_id AS bigint) AS HALO_ID,
CAST(ra_gal AS float) AS RA,
CAST(dec_gal AS float) AS `DEC`,
CAST(ra_mag_gal AS float) AS RA_MAG,
CAST(dec_mag_gal AS float) AS DEC_MAG,
CAST(observed_redshift_gal AS float) AS Z_OBS,
CAST(abs_mag_r01_evolved AS float) AS REF_MAG_ABS,
CAST(-2.5*log10(sdss_r01) - 48.6 AS float) AS REF_MAG,
CAST(bulge_fraction AS float) AS BULGE_FRACTION,
CAST(bulge_r50 AS float) AS BULGE_R50,
CAST(disk_r50 AS float) AS DISK_R50,
CAST(bulge_nsersic AS float) AS BULGE_NSERSIC,
CAST(inclination_angle AS float) AS INCLINATION_ANGLE,
CAST(disk_angle AS float) AS DISK_ANGLE,
CAST(kappa AS float) AS KAPPA,
CAST(gamma1 AS float) AS GAMMA1,
CAST(gamma2 AS float) AS GAMMA2,
CAST(sed_cosmos AS float) AS SED_TEMPLATE,
CAST(ROUND(ext_curve_cosmos) AS smallint) AS EXT_LAW,
CAST(ebv_cosmos AS float) AS EBV,
CAST(logf_halpha_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HALPHA_LOGFLAM_EXT,
CAST(logf_hbeta_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HBETA_LOGFLAM_EXT,
CAST(logf_o2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O2_LOGFLAM_EXT,
CAST(logf_o3_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O3_LOGFLAM_EXT,
CAST(logf_n2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS N2_LOGFLAM_EXT,
CAST(logf_s2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS S2_LOGFLAM_EXT,
CAST(mw_extinction AS float) AS AV,
CAST(euclid_vis_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_VIS,
CAST(euclid_nisp_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_NISP,
CAST(euclid_nisp_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_NISP,
CAST(euclid_nisp_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_NISP,
CAST(blanco_decam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_DECAM,
CAST(blanco_decam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_DECAM,
CAST(blanco_decam_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_DECAM,
CAST(blanco_decam_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_DECAM,
CAST(cfht_megacam_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_MEGACAM,
CAST(cfht_megacam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_MEGACAM,
CAST(jst_jpcam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_JPCAM,
CAST(pan_starrs_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_PANSTARRS,
CAST(pan_starrs_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_PANSTARRS,
CAST(subaru_hsc_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_HSC,
CAST(gaia_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_GAIA,
CAST(gaia_bp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_BP_GAIA,
CAST(gaia_rp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_RP_GAIA,
CAST(lsst_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_LSST,
CAST(lsst_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_LSST,
CAST(lsst_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_LSST,
CAST(lsst_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_LSST,
CAST(lsst_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_LSST,
CAST(lsst_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_LSST,
CAST(kids_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_KIDS,
CAST(kids_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_KIDS,
CAST(kids_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_KIDS,
CAST(kids_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_KIDS,
CAST(2mass_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_2MASS,
CAST(2mass_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_2MASS,
CAST(2mass_ks_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_KS_2MASS,
CAST( SHIFTRIGHT(hpix_29_nest, (29-5)*2) AS bigint) AS hpix_5_nest
FROM jcarrete.flagship_mock_obsid_1717_s
WHERE (logf_halpha_model3_ext > -16 OR -2.5*log10(euclid_nisp_h) - 48.6 < 26)
AND disk_axis_ratio > 0.10865
;

Y luego hay que añadir los header!

'''version v41!'''

/user/jcarrete/db/flagship_mock_obsid_1717_s_magnified_fits

/pnfs/pic.es/data/astro/euclid/disk/shared/SPV/GALAXIES/v41/

----

VUELVO A REGENERAR LOS FITS FILES (otra versión: v42) con el gamma1 cambiado de signo.
SOBREESCRIBO LA TABLA jcarrete.flagship_mock_obsid_1717_s_magnified_fits.

Le cambio el signo a gamma1

gamma1 = -gamma1

INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_1717_s_magnified_fits PARTITION(hpix_5_nest)
SELECT CAST(((halo_id * 10000) + galaxy_id) AS bigint) AS SOURCE_ID,
CAST(halo_id AS bigint) AS HALO_ID,
CAST(ra_gal AS float) AS RA,
CAST(dec_gal AS float) AS `DEC`,
CAST(ra_mag_gal AS float) AS RA_MAG,
CAST(dec_mag_gal AS float) AS DEC_MAG,
CAST(observed_redshift_gal AS float) AS Z_OBS,
CAST(abs_mag_r01_evolved AS float) AS REF_MAG_ABS,
CAST(-2.5*log10(sdss_r01) - 48.6 AS float) AS REF_MAG,
CAST(bulge_fraction AS float) AS BULGE_FRACTION,
CAST(bulge_r50 AS float) AS BULGE_R50,
CAST(disk_r50 AS float) AS DISK_R50,
CAST(bulge_nsersic AS float) AS BULGE_NSERSIC,
CAST(inclination_angle AS float) AS INCLINATION_ANGLE,
CAST(disk_angle AS float) AS DISK_ANGLE,
CAST(kappa AS float) AS KAPPA,
CAST(-gamma1 AS float) AS GAMMA1,
CAST(gamma2 AS float) AS GAMMA2,
CAST(sed_cosmos AS float) AS SED_TEMPLATE,
CAST(ROUND(ext_curve_cosmos) AS smallint) AS EXT_LAW,
CAST(ebv_cosmos AS float) AS EBV,
CAST(logf_halpha_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HALPHA_LOGFLAM_EXT,
CAST(logf_hbeta_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HBETA_LOGFLAM_EXT,
CAST(logf_o2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O2_LOGFLAM_EXT,
CAST(logf_o3_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O3_LOGFLAM_EXT,
CAST(logf_n2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS N2_LOGFLAM_EXT,
CAST(logf_s2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS S2_LOGFLAM_EXT,
CAST(mw_extinction AS float) AS AV,
CAST(euclid_vis_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_VIS,
CAST(euclid_nisp_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_NISP,
CAST(euclid_nisp_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_NISP,
CAST(euclid_nisp_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_NISP,
CAST(blanco_decam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_DECAM,
CAST(blanco_decam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_DECAM,
CAST(blanco_decam_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_DECAM,
CAST(blanco_decam_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_DECAM,
CAST(cfht_megacam_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_MEGACAM,
CAST(cfht_megacam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_MEGACAM,
CAST(jst_jpcam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_JPCAM,
CAST(pan_starrs_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_PANSTARRS,
CAST(pan_starrs_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_PANSTARRS,
CAST(subaru_hsc_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_HSC,
CAST(gaia_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_GAIA,
CAST(gaia_bp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_BP_GAIA,
CAST(gaia_rp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_RP_GAIA,
CAST(lsst_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_LSST,
CAST(lsst_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_LSST,
CAST(lsst_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_LSST,
CAST(lsst_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_LSST,
CAST(lsst_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_LSST,
CAST(lsst_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_LSST,
CAST(kids_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_KIDS,
CAST(kids_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_KIDS,
CAST(kids_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_KIDS,
CAST(kids_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_KIDS,
CAST(2mass_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_2MASS,
CAST(2mass_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_2MASS,
CAST(2mass_ks_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_KS_2MASS,
CAST( SHIFTRIGHT(hpix_29_nest, (29-5)*2) AS bigint) AS hpix_5_nest
FROM jcarrete.flagship_mock_obsid_1717_s
WHERE (logf_halpha_model3_ext > -16 OR -2.5*log10(euclid_nisp_h) - 48.6 < 26)
AND disk_axis_ratio > 0.10865
;

Añado el header a la versión 42!

----

'''Apparently, to go from MICE shear convention to IAU, which is also in principle the Euclid convention (or Flagship), it is necessary to modify the sign of gamma1:'''

gamma1 = -gamma1

In addition, we agreed that the topbot parameter = 1 is not necessary and it can be removed!

There is no need to modify anything no matter the octant we are working with.

----

'''For obsid = 52929'''

He modificado el flux_pipeline.py y he añadido la query que toca y los siguientes comentarios:

# Voy de +DEC a -DEC:
# dec_gal = - dec_gal
# Quito l_gal, b_gal, mw_extinction!
# Quito todos flujos que había calculado antes:
# blanco_decam_i para las shapes
# euclid_nisp_h para el corte
# sdss_r01 que lo había puesto "por si acaso"
# Filtro el obsid = 52929
#(ra_gal >= 35.45815488783862
#AND ra_gal < 61.33016111216139
#AND dec_gal >=73.5904
#AND dec_gal < 79.5904)

# The change in the shear parameters to go from MICE convention to IAU is the following:
# gamma1 = -gamma1
# BUT the change from DEC > 0 to DEC < 0 is equivalent to (again) change the sign of gamma1.
# So I do not do anything in the shear parameters!
# This is only valid for this particular case.
# Previous version (catalog) flagship_mock_obsid_1717_s need to be fixed!
# Note that x, y, z are not converted to the new position!

El catálogo se ha generado en formato parquet en el path:

/user/jcarrete/data/euclid/flagship_mock_obsid_52929_s/

/software/astro/scripts/create_table_from_parquet.sh -p /hdfs/user/jcarrete/data/euclid/flagship_mock_obsid_52929_s/0.pq -t jcarrete.flagship_mock_obsid_52929_s_pq -e

CREATE EXTERNAL TABLE jcarrete.flagship_mock_obsid_52929_s_pq (
`kind` BIGINT,
`random_index` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`ra_mag_gal` DOUBLE,
`dec_mag_gal` DOUBLE,
`kappa` DOUBLE,
`gamma1` DOUBLE,
`gamma2` DOUBLE,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`observed_redshift_gal` DOUBLE,
`hpix_29_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`abs_mag_r01` DOUBLE,
`luminosity_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`sdss_r01` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` BIGINT,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha` DOUBLE,
`logf_halpha_ext` DOUBLE,
`halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_halpha_model3_ext` DOUBLE,
`loglum_halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`dominant_shape` BIGINT,
`bulge_angle` DOUBLE,
`disk_angle` DOUBLE,
`median_major_axis` DOUBLE,
`scale_length` DOUBLE,
`bulge_fraction` DOUBLE,
`disk_scalelength` DOUBLE,
`disk_nsersic` DOUBLE,
`disk_r50` DOUBLE,
`bulge_r50` DOUBLE,
`bulge_nsersic` DOUBLE,
`disk_ellipticity` DOUBLE,
`bulge_ellipticity` DOUBLE,
`disk_axis_ratio` DOUBLE,
`bulge_axis_ratio` DOUBLE,
`inclination_angle` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`luminosity_r01_evolved` DOUBLE,
`hpix_13_nest` BIGINT,
`step` BIGINT,
`blanco_decam_g` DOUBLE,
`blanco_decam_i` DOUBLE,
`blanco_decam_r` DOUBLE,
`blanco_decam_z` DOUBLE,
`subaru_hsc_z` DOUBLE,
`jst_jpcam_g` DOUBLE,
`kids_g` DOUBLE,
`kids_i` DOUBLE,
`kids_r` DOUBLE,
`kids_u` DOUBLE,
`lsst_g` DOUBLE,
`lsst_i` DOUBLE,
`lsst_r` DOUBLE,
`lsst_u` DOUBLE,
`lsst_y` DOUBLE,
`lsst_z` DOUBLE,
`cfht_megacam_r` DOUBLE,
`cfht_megacam_u` DOUBLE,
`pan_starrs_i` DOUBLE,
`pan_starrs_z` DOUBLE,
`2mass_h` DOUBLE,
`2mass_j` DOUBLE,
`2mass_ks` DOUBLE,
`gaia_bp` DOUBLE,
`gaia_g` DOUBLE,
`gaia_rp` DOUBLE,
`euclid_nisp_h` DOUBLE,
`euclid_nisp_j` DOUBLE,
`euclid_nisp_y` DOUBLE,
`euclid_vis` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`2mass_h_odonnell_ext` DOUBLE,
`2mass_j_odonnell_ext` DOUBLE,
`2mass_ks_odonnell_ext` DOUBLE,
`blanco_decam_g_odonnell_ext` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`blanco_decam_r_odonnell_ext` DOUBLE,
`blanco_decam_z_odonnell_ext` DOUBLE,
`cfht_megacam_r_odonnell_ext` DOUBLE,
`cfht_megacam_u_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`euclid_nisp_j_odonnell_ext` DOUBLE,
`euclid_nisp_y_odonnell_ext` DOUBLE,
`euclid_vis_odonnell_ext` DOUBLE,
`gaia_bp_odonnell_ext` DOUBLE,
`gaia_g_odonnell_ext` DOUBLE,
`gaia_rp_odonnell_ext` DOUBLE,
`jst_jpcam_g_odonnell_ext` DOUBLE,
`kids_g_odonnell_ext` DOUBLE,
`kids_i_odonnell_ext` DOUBLE,
`kids_r_odonnell_ext` DOUBLE,
`kids_u_odonnell_ext` DOUBLE,
`lsst_g_odonnell_ext` DOUBLE,
`lsst_i_odonnell_ext` DOUBLE,
`lsst_r_odonnell_ext` DOUBLE,
`lsst_u_odonnell_ext` DOUBLE,
`lsst_y_odonnell_ext` DOUBLE,
`lsst_z_odonnell_ext` DOUBLE,
`pan_starrs_i_odonnell_ext` DOUBLE,
`pan_starrs_z_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`subaru_hsc_z_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model3_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model1_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model3_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model1_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model3_odonnell_ext` DOUBLE,
`kids_g_el_model1_odonnell_ext` DOUBLE,
`kids_g_el_model3_odonnell_ext` DOUBLE,
`kids_i_el_model1_odonnell_ext` DOUBLE,
`kids_i_el_model3_odonnell_ext` DOUBLE,
`kids_r_el_model1_odonnell_ext` DOUBLE,
`kids_r_el_model3_odonnell_ext` DOUBLE,
`kids_u_el_model1_odonnell_ext` DOUBLE,
`kids_u_el_model3_odonnell_ext` DOUBLE,
`lsst_g_el_model1_odonnell_ext` DOUBLE,
`lsst_g_el_model3_odonnell_ext` DOUBLE,
`lsst_i_el_model1_odonnell_ext` DOUBLE,
`lsst_i_el_model3_odonnell_ext` DOUBLE,
`lsst_r_el_model1_odonnell_ext` DOUBLE,
`lsst_r_el_model3_odonnell_ext` DOUBLE,
`lsst_u_el_model1_odonnell_ext` DOUBLE,
`lsst_u_el_model3_odonnell_ext` DOUBLE,
`lsst_y_el_model1_odonnell_ext` DOUBLE,
`lsst_y_el_model3_odonnell_ext` DOUBLE,
`lsst_z_el_model1_odonnell_ext` DOUBLE,
`lsst_z_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model3_odonnell_ext` DOUBLE,
`2mass_h_el_model1_odonnell_ext` DOUBLE,
`2mass_h_el_model3_odonnell_ext` DOUBLE,
`2mass_j_el_model1_odonnell_ext` DOUBLE,
`2mass_j_el_model3_odonnell_ext` DOUBLE,
`2mass_ks_el_model1_odonnell_ext` DOUBLE,
`2mass_ks_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`gaia_bp_el_model1_odonnell_ext` DOUBLE,
`gaia_bp_el_model3_odonnell_ext` DOUBLE,
`gaia_g_el_model1_odonnell_ext` DOUBLE,
`gaia_g_el_model3_odonnell_ext` DOUBLE,
`gaia_rp_el_model1_odonnell_ext` DOUBLE,
`gaia_rp_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model3_odonnell_ext` DOUBLE,
`euclid_vis_el_model1_odonnell_ext` DOUBLE,
`euclid_vis_el_model3_odonnell_ext` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
STORED AS PARQUET
LOCATION '/user/jcarrete/data/euclid/flagship_mock_obsid_52929_s'
;

CREATE TABLE jcarrete.flagship_mock_obsid_52929_s(
`halo_id` bigint COMMENT 'unique halo id coming from the Flagship dark matter halo catalog',
`galaxy_id` int COMMENT 'combined with the unique_halo_id they uniquely identify each galaxy',
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',
`random_index` float COMMENT 'random number [0 - 1) for subsampling',
`ra_gal` float COMMENT 'galaxy right ascension (degrees)',
`dec_gal` float COMMENT 'galaxy declination (degrees)',
`ra_mag_gal` float COMMENT 'galaxy magnified right ascension (degree)',
`dec_mag_gal` float COMMENT 'galaxy magnified declination (degree)',
`kappa` float COMMENT 'convergence',
`gamma1` float COMMENT 'shear',
`gamma2` float COMMENT 'shear',
`l_gal` float COMMENT 'galactic longitude (degrees)',
`b_gal` float COMMENT 'galactic latitude (degrees)',
`mw_extinction` float COMMENT 'EBV from file: HFI_CompMap_ThermalDustModel_2048_R1.20.fits',
`x_gal` float COMMENT 'galaxy comoving x coordinate (Mpc/h)',
`y_gal` float COMMENT 'galaxy comoving y coordinate (Mpc/h)',
`z_gal` float COMMENT 'galaxy comoving z coordinate (Mpc/h)',
`r_gal` float COMMENT 'galaxy comoving distance (Mpc/h)',
`true_redshift_gal` float COMMENT 'true galaxy redshift',
`observed_redshift_gal` float COMMENT 'observed galaxy redshift (including peculiar velocity)',
`hpix_29_nest` bigint COMMENT 'healpix pixel id using order = 29 in NESTED format',
`vx_gal` float COMMENT 'physical (peculiar) velocity in x-component (km/s)',
`vy_gal` float COMMENT 'physical (peculiar) velocity in y-component (km/s)',
`vz_gal` float COMMENT 'physical (peculiar) velocity in z-component (km/s)',
`vrad_gal` float COMMENT 'missing comment',
`delta_r` float COMMENT 'missing comment',
`abs_mag_r01` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h)',
`luminosity_r01` float COMMENT 'luminosity in the 0.1-r-band',
`abs_mag_r01_evolved` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h) including MICECATv2.0 magnitude evolution',
`sdss_r01` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1',
`abs_mag_uv_dereddened` float COMMENT 'absolute magnitude ultraviolet',
`gr_restframe` float COMMENT '(g-r) rest-frame at z=0',
`color_kind` tinyint COMMENT 'galaxy color kind: 0=red sequence 1=green valley 2=blue cloud',
`sed_ke` float COMMENT 'K-correction for each SED',
`gr_cosmos` float COMMENT 'interpolated COSMOS g-r color',
`sed_cosmos` float COMMENT 'interpolated basic COSMOS SED [0 30]',
`is_within_cosmos` boolean COMMENT 'missing comment',
`cosmos_distance` float COMMENT 'missing comment',
`ext_curve_cosmos` tinyint COMMENT 'COSMOS extinction curve [0-4]',
`ebv_cosmos` float COMMENT 'interpolated extinction strength for COSMOS extinction curve',
`log_ml_r01` float COMMENT 'logarithm of stellar_mass/luminosity ratio in the r01 SDSS band',
`log_stellar_mass` float COMMENT 'logarithm of the stellar mass (Msun/h)',
`metallicity` float COMMENT 'oxygen metallicity in units 12+logO/H',
`log_sfr` float COMMENT 'logarithm of star formation rate in (Msun/h)/year',
`logf_halpha` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: DOES NOT include extinction)',
`logf_halpha_ext` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: includes extinction)',
`halpha_scatter` float COMMENT 'missing comment',
`loglum_halpha` float COMMENT 'DEPRECATED logarithm of Halpha luminosity (erg/s/h^2) using empirical relation with UV magnitude (DOES NOT include extinction)',
`loglum_halpha_ext` float COMMENT 'missing comment',
`logf_dummy` float COMMENT 'missing comment',
`z_dummy` float COMMENT 'missing comment',
`logf_halpha_model1_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: includes extinction)',
`logf_halpha_model1` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: DOES NOT include extinction)',
`loglum_halpha_model1_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (includes extinction)',
`loglum_halpha_model1` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (DOES NOT include extinction)',
`logf_halpha_model3_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: includes extinction)',
`logf_halpha_model3` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: DOES NOT include extinction)',
`loglum_halpha_model3_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (includes extinction)',
`loglum_halpha_model3` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (DOES NOT include extinction)',
`logf_hbeta_model1_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_hbeta_model1` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_n2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_n2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o3_model1_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o3_model1` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_s2_model1_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_s2_model1` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_hbeta_model3_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_hbeta_model3` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_n2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_n2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o3_model3_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o3_model3` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_s2_model3_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_s2_model3` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',

`2mass_h` float COMMENT 'observed flux in 2MASS H-band (Euclid provided)',
`2mass_j` float COMMENT 'observed flux in 2MASS J-band (Euclid provided)',
`2mass_ks` float COMMENT 'observed flux in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g` float COMMENT 'observed flux in DES g-band (Euclid provided)',
`blanco_decam_i` float COMMENT 'observed flux in DES i-band (Euclid provided)',
`blanco_decam_r` float COMMENT 'observed flux in DES r-band (Euclid provided)',
`blanco_decam_z` float COMMENT 'observed flux in DES z-band (Euclid provided)',
`cfht_megacam_r` float COMMENT 'observed flux in CFHT r-band (Euclid provided)',
`cfht_megacam_u` float COMMENT 'observed flux in CFHT u-band (Euclid provided)',
`euclid_nisp_h` float COMMENT 'observed flux in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j` float COMMENT 'observed flux in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y` float COMMENT 'observed flux in Euclid NISP-Y band (Euclid provided)',
`euclid_vis` float COMMENT 'observed flux in Euclid VIS band (Euclid provided)',
`gaia_bp` float COMMENT 'observed flux in Gaia-BP band (Euclid provided)',
`gaia_g` float COMMENT 'observed flux in Gaia-G band (Euclid provided)',
`gaia_rp` float COMMENT 'observed flux in Gaia-RP band (Euclid provided)',
`jst_jpcam_g` float COMMENT 'observed flux in JST g-band (Euclid provided)',
`kids_g` float COMMENT 'observed flux in KIDS g-band (Euclid provided)',
`kids_i` float COMMENT 'observed flux in KIDS i-band (Euclid provided)',
`kids_r` float COMMENT 'observed flux in KIDS r-band (Euclid provided)',
`kids_u` float COMMENT 'observed flux in KIDS u-band (Euclid provided)',
`lsst_g` float COMMENT 'observed flux in LSST g-band (Euclid provided)',
`lsst_i` float COMMENT 'observed flux in LSST i-band (Euclid provided)',
`lsst_r` float COMMENT 'observed flux in LSST r-band (Euclid provided)',
`lsst_u` float COMMENT 'observed flux in LSST u-band (Euclid provided)',
`lsst_y` float COMMENT 'observed flux in LSST Y-band (Euclid provided)',
`lsst_z` float COMMENT 'observed flux in LSST z-band (Euclid provided)',
`pan_starrs_i` float COMMENT 'observed flux in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z` float COMMENT 'observed flux in Pan-STARRS z-band (Euclid provided)',
`subaru_hsc_z` float COMMENT 'observed flux in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS H-band (Euclid provided)',
`2mass_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in JST g-band (Euclid provided)',
`kids_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_odonnell_ext` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1 with MW extinction',
`subaru_hsc_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS Ks-band (Euclid provided)',
`2mass_ks_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES z-band (Euclid provided)',
`blanco_decam_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT u-band (Euclid provided)',
`cfht_megacam_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_nisp_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid VIS band (Euclid provided)',
`euclid_vis_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_bp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-RP band (Euclid provided)',
`gaia_rp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in JST g-band (Euclid provided)',
`jst_jpcam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in JST g-band (Euclid provided)',
`kids_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS g-band (Euclid provided)',
`kids_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS i-band (Euclid provided)',
`kids_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS r-band (Euclid provided)',
`kids_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS u-band (Euclid provided)',
`kids_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST g-band (Euclid provided)',
`lsst_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST i-band (Euclid provided)',
`lsst_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST r-band (Euclid provided)',
`lsst_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST u-band (Euclid provided)',
`lsst_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST z-band (Euclid provided)',
`lsst_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`pan_starrs_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in sdss r-band redshifted to z=0.1',
`sdss_r01_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in sdss r-band redshifted to z=0.1',
`subaru_hsc_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`subaru_hsc_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',

`dominant_shape` tinyint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)',
`bulge_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`disk_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`median_major_axis` float COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)',
`scale_length` float COMMENT 'disc and bulge scalelength prior',
`bulge_fraction` float COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)',
`disk_scalelength` float COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)',
`disk_nsersic` float COMMENT 'Sersic index of the disk component',
`disk_r50` float COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678',
`bulge_r50` float COMMENT 'half light radius of the bulge',
`bulge_nsersic` float COMMENT 'Sersic index of the bulge component',
`disk_ellipticity` float COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)',
`bulge_ellipticity` float COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)',
`disk_axis_ratio` float COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`bulge_axis_ratio` float COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`inclination_angle` float COMMENT 'Galaxy inclination angle (where 0 degrees = face-on and 90 degrees = edge-on). Galaxy ellipticity for disk and bulge components are computed following the recipe in https://euclid.roe.ac.uk/projects/sgsshear/wiki/SHE-SIM',
`halo_x` float COMMENT 'halo comoving x coordinate (Mpc/h)',
`halo_y` float COMMENT 'halo comoving y coordinate (Mpc/h)',
`halo_z` float COMMENT 'halo comoving z coordinate (Mpc/h)',
`halo_vx` float COMMENT 'halo x-centre velocity (km/h)',
`halo_vy` float COMMENT 'halo y-centre velocity (km/h)',
`halo_vz` float COMMENT 'halo z-centre velocity (km/h)',
`halo_r` float COMMENT 'halo comoving distance (Mpc/h)',
`true_redshift_halo` float COMMENT 'true redshift of the host halo',
`halo_lm` float COMMENT 'log10 of the FoF halo mass in Msum/h. This halo mass is computed as the particule mass multiplied by the number of particles of the FoF halo. At low masses it is interpolated to obtain a smooth distribution of halo masses',
`halo_n_sats` int COMMENT 'number of satellite galaxies from HOD',
`n_gals` int COMMENT 'DEPRECATED',
`luminosity_r01_evolved` float COMMENT 'missing comment',
`hpix_13_nest` bigint COMMENT 'missing comment')
PARTITIONED BY (
`step` smallint COMMENT 'step number in the Flagship lensing maps')
CLUSTERED BY (
hpix_13_nest)
SORTED BY (
hpix_13_nest ASC)
INTO 128 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;

INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_52929_s PARTITION(step)
SELECT halo_id, galaxy_id, kind, random_index, ra_gal, dec_gal, ra_mag_gal, dec_mag_gal, kappa, gamma1, gamma2, l_gal, b_gal, mw_extinction, x_gal, y_gal, z_gal, r_gal, true_redshift_gal, observed_redshift_gal, hpix_29_nest, vx_gal, vy_gal, vz_gal, vrad_gal, delta_r, abs_mag_r01, luminosity_r01, abs_mag_r01_evolved, sdss_r01, abs_mag_uv_dereddened, gr_restframe, color_kind, sed_ke, gr_cosmos, sed_cosmos, is_within_cosmos, cosmos_distance, ext_curve_cosmos, ebv_cosmos, log_ml_r01, log_stellar_mass, metallicity, log_sfr, logf_halpha, logf_halpha_ext, halpha_scatter, loglum_halpha, loglum_halpha_ext, logf_dummy, z_dummy, logf_halpha_model1_ext, logf_halpha_model1, loglum_halpha_model1_ext, loglum_halpha_model1, logf_halpha_model3_ext, logf_halpha_model3, loglum_halpha_model3_ext, loglum_halpha_model3, logf_hbeta_model1_ext, logf_hbeta_model1, logf_o2_model1_ext, logf_o2_model1, logf_n2_model1_ext, logf_n2_model1, logf_o3_model1_ext, logf_o3_model1, logf_s2_model1_ext, logf_s2_model1, logf_hbeta_model3_ext, logf_hbeta_model3, logf_o2_model3_ext, logf_o2_model3, logf_n2_model3_ext, logf_n2_model3, logf_o3_model3_ext, logf_o3_model3, logf_s2_model3_ext, logf_s2_model3, 2mass_h, 2mass_j, 2mass_ks, blanco_decam_g, blanco_decam_i, blanco_decam_r, blanco_decam_z, cfht_megacam_r, cfht_megacam_u, euclid_nisp_h, euclid_nisp_j, euclid_nisp_y, euclid_vis, gaia_bp, gaia_g, gaia_rp, jst_jpcam_g, kids_g, kids_i, kids_r, kids_u, lsst_g, lsst_i, lsst_r, lsst_u, lsst_y, lsst_z, pan_starrs_i, pan_starrs_z, subaru_hsc_z, 2mass_h_odonnell_ext, 2mass_j_odonnell_ext, 2mass_ks_odonnell_ext, blanco_decam_g_odonnell_ext, blanco_decam_i_odonnell_ext, blanco_decam_r_odonnell_ext, blanco_decam_z_odonnell_ext, cfht_megacam_r_odonnell_ext, cfht_megacam_u_odonnell_ext, euclid_nisp_h_odonnell_ext, euclid_nisp_j_odonnell_ext, euclid_nisp_y_odonnell_ext, euclid_vis_odonnell_ext, gaia_bp_odonnell_ext, gaia_g_odonnell_ext, gaia_rp_odonnell_ext, jst_jpcam_g_odonnell_ext, kids_g_odonnell_ext, kids_i_odonnell_ext, kids_r_odonnell_ext, kids_u_odonnell_ext, lsst_g_odonnell_ext, lsst_i_odonnell_ext, lsst_r_odonnell_ext, lsst_u_odonnell_ext, lsst_y_odonnell_ext, lsst_z_odonnell_ext, pan_starrs_i_odonnell_ext, pan_starrs_z_odonnell_ext, sdss_r01_odonnell_ext, subaru_hsc_z_odonnell_ext, 2mass_h_el_model1_odonnell_ext, 2mass_h_el_model3_odonnell_ext, 2mass_j_el_model1_odonnell_ext, 2mass_j_el_model3_odonnell_ext, 2mass_ks_el_model1_odonnell_ext, 2mass_ks_el_model3_odonnell_ext, blanco_decam_g_el_model1_odonnell_ext, blanco_decam_g_el_model3_odonnell_ext, blanco_decam_i_el_model1_odonnell_ext, blanco_decam_i_el_model3_odonnell_ext, blanco_decam_r_el_model1_odonnell_ext, blanco_decam_r_el_model3_odonnell_ext, blanco_decam_z_el_model1_odonnell_ext, blanco_decam_z_el_model3_odonnell_ext, cfht_megacam_r_el_model1_odonnell_ext, cfht_megacam_r_el_model3_odonnell_ext, cfht_megacam_u_el_model1_odonnell_ext, cfht_megacam_u_el_model3_odonnell_ext, euclid_nisp_h_el_model1_odonnell_ext, euclid_nisp_h_el_model3_odonnell_ext, euclid_nisp_j_el_model1_odonnell_ext, euclid_nisp_j_el_model3_odonnell_ext, euclid_nisp_y_el_model1_odonnell_ext, euclid_nisp_y_el_model3_odonnell_ext, euclid_vis_el_model1_odonnell_ext, euclid_vis_el_model3_odonnell_ext, gaia_bp_el_model1_odonnell_ext, gaia_bp_el_model3_odonnell_ext, gaia_g_el_model1_odonnell_ext, gaia_g_el_model3_odonnell_ext, gaia_rp_el_model1_odonnell_ext, gaia_rp_el_model3_odonnell_ext, jst_jpcam_g_el_model1_odonnell_ext, jst_jpcam_g_el_model3_odonnell_ext, kids_g_el_model1_odonnell_ext, kids_g_el_model3_odonnell_ext, kids_i_el_model1_odonnell_ext, kids_i_el_model3_odonnell_ext, kids_r_el_model1_odonnell_ext, kids_r_el_model3_odonnell_ext, kids_u_el_model1_odonnell_ext, kids_u_el_model3_odonnell_ext, lsst_g_el_model1_odonnell_ext, lsst_g_el_model3_odonnell_ext, lsst_i_el_model1_odonnell_ext, lsst_i_el_model3_odonnell_ext, lsst_r_el_model1_odonnell_ext, lsst_r_el_model3_odonnell_ext, lsst_u_el_model1_odonnell_ext, lsst_u_el_model3_odonnell_ext, lsst_y_el_model1_odonnell_ext, lsst_y_el_model3_odonnell_ext, lsst_z_el_model1_odonnell_ext, lsst_z_el_model3_odonnell_ext, pan_starrs_i_el_model1_odonnell_ext, pan_starrs_i_el_model3_odonnell_ext, pan_starrs_z_el_model1_odonnell_ext, pan_starrs_z_el_model3_odonnell_ext, sdss_r01_el_model1_odonnell_ext, sdss_r01_el_model3_odonnell_ext, subaru_hsc_z_el_model1_odonnell_ext, subaru_hsc_z_el_model3_odonnell_ext, dominant_shape, bulge_angle, disk_angle, median_major_axis, scale_length, bulge_fraction, disk_scalelength, disk_nsersic, disk_r50, bulge_r50, bulge_nsersic, disk_ellipticity, bulge_ellipticity, disk_axis_ratio, bulge_axis_ratio, inclination_angle, halo_x, halo_y, halo_z, halo_vx, halo_vy, halo_vz, halo_r, true_redshift_halo, halo_lm, halo_n_sats, n_gals, luminosity_r01_evolved, hpix_13_nest, step
FROM jcarrete.flagship_mock_obsid_52929_s_pq

CREATE TABLE jcarrete.flagship_mock_obsid_52929_s_magnified_fits(
`source_id` bigint COMMENT 'from deserializer',
`halo_id` bigint COMMENT 'from deserializer',
`ra` float COMMENT 'from deserializer',
`dec` float COMMENT 'from deserializer',
`ra_mag` float COMMENT 'from deserializer',
`dec_mag` float COMMENT 'from deserializer',
`z_obs` float COMMENT 'from deserializer',
`ref_mag_abs` float COMMENT 'from deserializer',
`ref_mag` float COMMENT 'from deserializer',
`bulge_fraction` float COMMENT 'from deserializer',
`bulge_r50` float COMMENT 'from deserializer',
`disk_r50` float COMMENT 'from deserializer',
`bulge_nsersic` float COMMENT 'from deserializer',
`inclination_angle` float COMMENT 'from deserializer',
`disk_angle` float COMMENT 'from deserializer',
`kappa` float COMMENT 'from deserializer',
`gamma1` float COMMENT 'from deserializer',
`gamma2` float COMMENT 'from deserializer',
`sed_template` float COMMENT 'from deserializer',
`ext_law` smallint COMMENT 'from deserializer',
`ebv` float COMMENT 'from deserializer',
`halpha_logflam_ext` float COMMENT 'from deserializer',
`hbeta_logflam_ext` float COMMENT 'from deserializer',
`o2_logflam_ext` float COMMENT 'from deserializer',
`o3_logflam_ext` float COMMENT 'from deserializer',
`n2_logflam_ext` float COMMENT 'from deserializer',
`s2_logflam_ext` float COMMENT 'from deserializer',
`av` float COMMENT 'from deserializer',
`tu_fnu_vis` float COMMENT 'from deserializer',
`tu_fnu_y_nisp` float COMMENT 'from deserializer',
`tu_fnu_j_nisp` float COMMENT 'from deserializer',
`tu_fnu_h_nisp` float COMMENT 'from deserializer',
`tu_fnu_g_decam` float COMMENT 'from deserializer',
`tu_fnu_r_decam` float COMMENT 'from deserializer',
`tu_fnu_i_decam` float COMMENT 'from deserializer',
`tu_fnu_z_decam` float COMMENT 'from deserializer',
`tu_fnu_u_megacam` float COMMENT 'from deserializer',
`tu_fnu_r_megacam` float COMMENT 'from deserializer',
`tu_fnu_g_jpcam` float COMMENT 'from deserializer',
`tu_fnu_i_panstarrs` float COMMENT 'from deserializer',
`tu_fnu_z_panstarrs` float COMMENT 'from deserializer',
`tu_fnu_z_hsc` float COMMENT 'from deserializer',
`tu_fnu_g_gaia` float COMMENT 'from deserializer',
`tu_fnu_bp_gaia` float COMMENT 'from deserializer',
`tu_fnu_rp_gaia` float COMMENT 'from deserializer',
`tu_fnu_u_lsst` float COMMENT 'from deserializer',
`tu_fnu_g_lsst` float COMMENT 'from deserializer',
`tu_fnu_r_lsst` float COMMENT 'from deserializer',
`tu_fnu_i_lsst` float COMMENT 'from deserializer',
`tu_fnu_z_lsst` float COMMENT 'from deserializer',
`tu_fnu_y_lsst` float COMMENT 'from deserializer',
`tu_fnu_u_kids` float COMMENT 'from deserializer',
`tu_fnu_g_kids` float COMMENT 'from deserializer',
`tu_fnu_r_kids` float COMMENT 'from deserializer',
`tu_fnu_i_kids` float COMMENT 'from deserializer',
`tu_fnu_j_2mass` float COMMENT 'from deserializer',
`tu_fnu_h_2mass` float COMMENT 'from deserializer',
`tu_fnu_ks_2mass` float COMMENT 'from deserializer')
PARTITIONED BY (
`hpix_5_nest` bigint)
ROW FORMAT SERDE
'es.pic.astro.hadoop.serde.RecArraySerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'es.pic.astro.hadoop.io.BinaryOutputFormat'
;

INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_52929_s_magnified_fits PARTITION(hpix_5_nest)
SELECT CAST(((halo_id * 10000) + galaxy_id) AS bigint) AS SOURCE_ID,
CAST(halo_id AS bigint) AS HALO_ID,
CAST(ra_gal AS float) AS RA,
CAST(dec_gal AS float) AS `DEC`,
CAST(ra_mag_gal AS float) AS RA_MAG,
CAST(dec_mag_gal AS float) AS DEC_MAG,
CAST(observed_redshift_gal AS float) AS Z_OBS,
CAST(abs_mag_r01_evolved AS float) AS REF_MAG_ABS,
CAST(-2.5*log10(sdss_r01) - 48.6 AS float) AS REF_MAG,
CAST(bulge_fraction AS float) AS BULGE_FRACTION,
CAST(bulge_r50 AS float) AS BULGE_R50,
CAST(disk_r50 AS float) AS DISK_R50,
CAST(bulge_nsersic AS float) AS BULGE_NSERSIC,
CAST(inclination_angle AS float) AS INCLINATION_ANGLE,
CAST(disk_angle AS float) AS DISK_ANGLE,
CAST(kappa AS float) AS KAPPA,
CAST(gamma1 AS float) AS GAMMA1,
CAST(gamma2 AS float) AS GAMMA2,
CAST(sed_cosmos AS float) AS SED_TEMPLATE,
CAST(ROUND(ext_curve_cosmos) AS smallint) AS EXT_LAW,
CAST(ebv_cosmos AS float) AS EBV,
CAST(logf_halpha_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HALPHA_LOGFLAM_EXT,
CAST(logf_hbeta_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HBETA_LOGFLAM_EXT,
CAST(logf_o2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O2_LOGFLAM_EXT,
CAST(logf_o3_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O3_LOGFLAM_EXT,
CAST(logf_n2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS N2_LOGFLAM_EXT,
CAST(logf_s2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS S2_LOGFLAM_EXT,
CAST(mw_extinction AS float) AS AV,
CAST(euclid_vis_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_VIS,
CAST(euclid_nisp_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_NISP,
CAST(euclid_nisp_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_NISP,
CAST(euclid_nisp_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_NISP,
CAST(blanco_decam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_DECAM,
CAST(blanco_decam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_DECAM,
CAST(blanco_decam_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_DECAM,
CAST(blanco_decam_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_DECAM,
CAST(cfht_megacam_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_MEGACAM,
CAST(cfht_megacam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_MEGACAM,
CAST(jst_jpcam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_JPCAM,
CAST(pan_starrs_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_PANSTARRS,
CAST(pan_starrs_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_PANSTARRS,
CAST(subaru_hsc_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_HSC,
CAST(gaia_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_GAIA,
CAST(gaia_bp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_BP_GAIA,
CAST(gaia_rp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_RP_GAIA,
CAST(lsst_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_LSST,
CAST(lsst_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_LSST,
CAST(lsst_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_LSST,
CAST(lsst_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_LSST,
CAST(lsst_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_LSST,
CAST(lsst_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_LSST,
CAST(kids_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_KIDS,
CAST(kids_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_KIDS,
CAST(kids_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_KIDS,
CAST(kids_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_KIDS,
CAST(2mass_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_2MASS,
CAST(2mass_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_2MASS,
CAST(2mass_ks_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_KS_2MASS,
CAST( SHIFTRIGHT(hpix_29_nest, (29-5)*2) AS bigint) AS hpix_5_nest
FROM jcarrete.flagship_mock_obsid_52929_s
WHERE (logf_halpha_model3_ext > -16 OR -2.5*log10(euclid_nisp_h) - 48.6 < 26)
AND disk_axis_ratio > 0.10865
;

Añadimos los header con el notebook: add_fits_header_to_hive_table.py

Lo hago con la version 43. No pongo la 42 porque la voy a usar para corregir del lensing la v41 (gamma1 = -gamma1)

----

'''Los dos catálogos anteriores tienen un bug en las shapes!!!!'''

Vuelvo a calcular las shapes pero solo para la 1717.

Genero un catalog con la pipeline shape_pipeline.py que se llama: flagship_mock_obsid_1717_s_v2

Esta es la query que uso (elimino los campos de las shapes, que son los que voy a recalcular!):

SELECT halo_id, galaxy_id, kind, random_index, ra_gal, dec_gal, ra_mag_gal, dec_mag_gal, kappa, gamma1, gamma2, l_gal, b_gal, mw_extinction, x_gal, y_gal, z_gal, r_gal, true_redshift_gal, observed_redshift_gal, hpix_29_nest, vx_gal, vy_gal, vz_gal, vrad_gal, delta_r, abs_mag_r01, luminosity_r01, abs_mag_r01_evolved, sdss_r01, abs_mag_uv_dereddened, gr_restframe, color_kind, sed_ke, gr_cosmos, sed_cosmos, is_within_cosmos, cosmos_distance, ext_curve_cosmos, ebv_cosmos, log_ml_r01, log_stellar_mass, metallicity, log_sfr, logf_halpha, logf_halpha_ext, halpha_scatter, loglum_halpha, loglum_halpha_ext, logf_dummy, z_dummy, logf_halpha_model1_ext, logf_halpha_model1, loglum_halpha_model1_ext, loglum_halpha_model1, logf_halpha_model3_ext, logf_halpha_model3, loglum_halpha_model3_ext, loglum_halpha_model3, logf_hbeta_model1_ext, logf_hbeta_model1, logf_o2_model1_ext, logf_o2_model1, logf_n2_model1_ext, logf_n2_model1, logf_o3_model1_ext, logf_o3_model1, logf_s2_model1_ext, logf_s2_model1, logf_hbeta_model3_ext, logf_hbeta_model3, logf_o2_model3_ext, logf_o2_model3, logf_n2_model3_ext, logf_n2_model3, logf_o3_model3_ext, logf_o3_model3, logf_s2_model3_ext, logf_s2_model3, 2mass_h, 2mass_j, 2mass_ks, blanco_decam_g, blanco_decam_i, blanco_decam_r, blanco_decam_z, cfht_megacam_r, cfht_megacam_u, euclid_nisp_h, euclid_nisp_j, euclid_nisp_y, euclid_vis, gaia_bp, gaia_g, gaia_rp, jst_jpcam_g, kids_g, kids_i, kids_r, kids_u, lsst_g, lsst_i, lsst_r, lsst_u, lsst_y, lsst_z, pan_starrs_i, pan_starrs_z, subaru_hsc_z, 2mass_h_odonnell_ext, 2mass_j_odonnell_ext, 2mass_ks_odonnell_ext, blanco_decam_g_odonnell_ext, blanco_decam_i_odonnell_ext, blanco_decam_r_odonnell_ext, blanco_decam_z_odonnell_ext, cfht_megacam_r_odonnell_ext, cfht_megacam_u_odonnell_ext, euclid_nisp_h_odonnell_ext, euclid_nisp_j_odonnell_ext, euclid_nisp_y_odonnell_ext, euclid_vis_odonnell_ext, gaia_bp_odonnell_ext, gaia_g_odonnell_ext, gaia_rp_odonnell_ext, jst_jpcam_g_odonnell_ext, kids_g_odonnell_ext, kids_i_odonnell_ext, kids_r_odonnell_ext, kids_u_odonnell_ext, lsst_g_odonnell_ext, lsst_i_odonnell_ext, lsst_r_odonnell_ext, lsst_u_odonnell_ext, lsst_y_odonnell_ext, lsst_z_odonnell_ext, pan_starrs_i_odonnell_ext, pan_starrs_z_odonnell_ext, sdss_r01_odonnell_ext, subaru_hsc_z_odonnell_ext, 2mass_h_el_model1_odonnell_ext, 2mass_h_el_model3_odonnell_ext, 2mass_j_el_model1_odonnell_ext, 2mass_j_el_model3_odonnell_ext, 2mass_ks_el_model1_odonnell_ext, 2mass_ks_el_model3_odonnell_ext, blanco_decam_g_el_model1_odonnell_ext, blanco_decam_g_el_model3_odonnell_ext, blanco_decam_i_el_model1_odonnell_ext, blanco_decam_i_el_model3_odonnell_ext, blanco_decam_r_el_model1_odonnell_ext, blanco_decam_r_el_model3_odonnell_ext, blanco_decam_z_el_model1_odonnell_ext, blanco_decam_z_el_model3_odonnell_ext, cfht_megacam_r_el_model1_odonnell_ext, cfht_megacam_r_el_model3_odonnell_ext, cfht_megacam_u_el_model1_odonnell_ext, cfht_megacam_u_el_model3_odonnell_ext, euclid_nisp_h_el_model1_odonnell_ext, euclid_nisp_h_el_model3_odonnell_ext, euclid_nisp_j_el_model1_odonnell_ext, euclid_nisp_j_el_model3_odonnell_ext, euclid_nisp_y_el_model1_odonnell_ext, euclid_nisp_y_el_model3_odonnell_ext, euclid_vis_el_model1_odonnell_ext, euclid_vis_el_model3_odonnell_ext, gaia_bp_el_model1_odonnell_ext, gaia_bp_el_model3_odonnell_ext, gaia_g_el_model1_odonnell_ext, gaia_g_el_model3_odonnell_ext, gaia_rp_el_model1_odonnell_ext, gaia_rp_el_model3_odonnell_ext, jst_jpcam_g_el_model1_odonnell_ext, jst_jpcam_g_el_model3_odonnell_ext, kids_g_el_model1_odonnell_ext, kids_g_el_model3_odonnell_ext, kids_i_el_model1_odonnell_ext, kids_i_el_model3_odonnell_ext, kids_r_el_model1_odonnell_ext, kids_r_el_model3_odonnell_ext, kids_u_el_model1_odonnell_ext, kids_u_el_model3_odonnell_ext, lsst_g_el_model1_odonnell_ext, lsst_g_el_model3_odonnell_ext, lsst_i_el_model1_odonnell_ext, lsst_i_el_model3_odonnell_ext, lsst_r_el_model1_odonnell_ext, lsst_r_el_model3_odonnell_ext, lsst_u_el_model1_odonnell_ext, lsst_u_el_model3_odonnell_ext, lsst_y_el_model1_odonnell_ext, lsst_y_el_model3_odonnell_ext, lsst_z_el_model1_odonnell_ext, lsst_z_el_model3_odonnell_ext, pan_starrs_i_el_model1_odonnell_ext, pan_starrs_i_el_model3_odonnell_ext, pan_starrs_z_el_model1_odonnell_ext, pan_starrs_z_el_model3_odonnell_ext, sdss_r01_el_model1_odonnell_ext, sdss_r01_el_model3_odonnell_ext, subaru_hsc_z_el_model1_odonnell_ext, subaru_hsc_z_el_model3_odonnell_ext, halo_x, halo_y, halo_z, halo_vx, halo_vy, halo_vz, halo_r, true_redshift_halo, halo_lm, halo_n_sats, n_gals, luminosity_r01_evolved, hpix_13_nest, step
FROM jcarrete.flagship_mock_obsid_1717_s

----

Defino la tabla externa en parquet format del catálogo usando el script:

/software/astro/scripts/create_table_from_parquet.sh -p /hdfs/user/jcarrete/data/euclid/flagship_mock_obsid_1717_s_v2/0.pq -t jcarrete.flagship_mock_obsid_1717_s_v2_pq -e

CREATE EXTERNAL TABLE jcarrete.flagship_mock_obsid_1717_s_v2_pq (
`kind` BIGINT,
`random_index` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`ra_mag_gal` DOUBLE,
`dec_mag_gal` DOUBLE,
`kappa` DOUBLE,
`gamma1` DOUBLE,
`gamma2` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`observed_redshift_gal` DOUBLE,
`hpix_29_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`abs_mag_r01` DOUBLE,
`luminosity_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`sdss_r01` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` BIGINT,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha` DOUBLE,
`logf_halpha_ext` DOUBLE,
`halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_halpha_model3_ext` DOUBLE,
`loglum_halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`2mass_h` DOUBLE,
`2mass_j` DOUBLE,
`2mass_ks` DOUBLE,
`blanco_decam_g` DOUBLE,
`blanco_decam_i` DOUBLE,
`blanco_decam_r` DOUBLE,
`blanco_decam_z` DOUBLE,
`cfht_megacam_r` DOUBLE,
`cfht_megacam_u` DOUBLE,
`euclid_nisp_h` DOUBLE,
`euclid_nisp_j` DOUBLE,
`euclid_nisp_y` DOUBLE,
`euclid_vis` DOUBLE,
`gaia_bp` DOUBLE,
`gaia_g` DOUBLE,
`gaia_rp` DOUBLE,
`jst_jpcam_g` DOUBLE,
`kids_g` DOUBLE,
`kids_i` DOUBLE,
`kids_r` DOUBLE,
`kids_u` DOUBLE,
`lsst_g` DOUBLE,
`lsst_i` DOUBLE,
`lsst_r` DOUBLE,
`lsst_u` DOUBLE,
`lsst_y` DOUBLE,
`lsst_z` DOUBLE,
`pan_starrs_i` DOUBLE,
`pan_starrs_z` DOUBLE,
`subaru_hsc_z` DOUBLE,
`2mass_h_odonnell_ext` DOUBLE,
`2mass_j_odonnell_ext` DOUBLE,
`2mass_ks_odonnell_ext` DOUBLE,
`blanco_decam_g_odonnell_ext` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`blanco_decam_r_odonnell_ext` DOUBLE,
`blanco_decam_z_odonnell_ext` DOUBLE,
`cfht_megacam_r_odonnell_ext` DOUBLE,
`cfht_megacam_u_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`euclid_nisp_j_odonnell_ext` DOUBLE,
`euclid_nisp_y_odonnell_ext` DOUBLE,
`euclid_vis_odonnell_ext` DOUBLE,
`gaia_bp_odonnell_ext` DOUBLE,
`gaia_g_odonnell_ext` DOUBLE,
`gaia_rp_odonnell_ext` DOUBLE,
`jst_jpcam_g_odonnell_ext` DOUBLE,
`kids_g_odonnell_ext` DOUBLE,
`kids_i_odonnell_ext` DOUBLE,
`kids_r_odonnell_ext` DOUBLE,
`kids_u_odonnell_ext` DOUBLE,
`lsst_g_odonnell_ext` DOUBLE,
`lsst_i_odonnell_ext` DOUBLE,
`lsst_r_odonnell_ext` DOUBLE,
`lsst_u_odonnell_ext` DOUBLE,
`lsst_y_odonnell_ext` DOUBLE,
`lsst_z_odonnell_ext` DOUBLE,
`pan_starrs_i_odonnell_ext` DOUBLE,
`pan_starrs_z_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`subaru_hsc_z_odonnell_ext` DOUBLE,
`2mass_h_el_model1_odonnell_ext` DOUBLE,
`2mass_h_el_model3_odonnell_ext` DOUBLE,
`2mass_j_el_model1_odonnell_ext` DOUBLE,
`2mass_j_el_model3_odonnell_ext` DOUBLE,
`2mass_ks_el_model1_odonnell_ext` DOUBLE,
`2mass_ks_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model3_odonnell_ext` DOUBLE,
`euclid_vis_el_model1_odonnell_ext` DOUBLE,
`euclid_vis_el_model3_odonnell_ext` DOUBLE,
`gaia_bp_el_model1_odonnell_ext` DOUBLE,
`gaia_bp_el_model3_odonnell_ext` DOUBLE,
`gaia_g_el_model1_odonnell_ext` DOUBLE,
`gaia_g_el_model3_odonnell_ext` DOUBLE,
`gaia_rp_el_model1_odonnell_ext` DOUBLE,
`gaia_rp_el_model3_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model1_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model3_odonnell_ext` DOUBLE,
`kids_g_el_model1_odonnell_ext` DOUBLE,
`kids_g_el_model3_odonnell_ext` DOUBLE,
`kids_i_el_model1_odonnell_ext` DOUBLE,
`kids_i_el_model3_odonnell_ext` DOUBLE,
`kids_r_el_model1_odonnell_ext` DOUBLE,
`kids_r_el_model3_odonnell_ext` DOUBLE,
`kids_u_el_model1_odonnell_ext` DOUBLE,
`kids_u_el_model3_odonnell_ext` DOUBLE,
`lsst_g_el_model1_odonnell_ext` DOUBLE,
`lsst_g_el_model3_odonnell_ext` DOUBLE,
`lsst_i_el_model1_odonnell_ext` DOUBLE,
`lsst_i_el_model3_odonnell_ext` DOUBLE,
`lsst_r_el_model1_odonnell_ext` DOUBLE,
`lsst_r_el_model3_odonnell_ext` DOUBLE,
`lsst_u_el_model1_odonnell_ext` DOUBLE,
`lsst_u_el_model3_odonnell_ext` DOUBLE,
`lsst_y_el_model1_odonnell_ext` DOUBLE,
`lsst_y_el_model3_odonnell_ext` DOUBLE,
`lsst_z_el_model1_odonnell_ext` DOUBLE,
`lsst_z_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model1_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model3_odonnell_ext` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`luminosity_r01_evolved` DOUBLE,
`hpix_13_nest` BIGINT,
`step` BIGINT,
`dominant_shape` BIGINT,
`bulge_angle` DOUBLE,
`disk_angle` DOUBLE,
`median_major_axis` DOUBLE,
`scale_length` DOUBLE,
`bulge_fraction` DOUBLE,
`disk_scalelength` DOUBLE,
`disk_nsersic` DOUBLE,
`disk_r50` DOUBLE,
`bulge_r50` DOUBLE,
`bulge_nsersic` DOUBLE,
`inclination_angle` DOUBLE,
`disk_axis_ratio` DOUBLE,
`bulge_axis_ratio` DOUBLE,
`disk_ellipticity` DOUBLE,
`bulge_ellipticity` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
STORED AS PARQUET
LOCATION '/user/jcarrete/data/euclid/flagship_mock_obsid_1717_s_v2'
;

----

CREATE TABLE jcarrete.flagship_mock_obsid_1717_s_v2(
`halo_id` bigint COMMENT 'unique halo id coming from the Flagship dark matter halo catalog',
`galaxy_id` int COMMENT 'combined with the unique_halo_id they uniquely identify each galaxy',
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',
`random_index` float COMMENT 'random number [0 - 1) for subsampling',
`ra_gal` float COMMENT 'galaxy right ascension (degrees)',
`dec_gal` float COMMENT 'galaxy declination (degrees)',
`ra_mag_gal` float COMMENT 'galaxy magnified right ascension (degree)',
`dec_mag_gal` float COMMENT 'galaxy magnified declination (degree)',
`kappa` float COMMENT 'convergence',
`gamma1` float COMMENT 'shear',
`gamma2` float COMMENT 'shear',
`l_gal` float COMMENT 'galactic longitude (degrees)',
`b_gal` float COMMENT 'galactic latitude (degrees)',
`mw_extinction` float COMMENT 'EBV from file: HFI_CompMap_ThermalDustModel_2048_R1.20.fits',
`x_gal` float COMMENT 'galaxy comoving x coordinate (Mpc/h)',
`y_gal` float COMMENT 'galaxy comoving y coordinate (Mpc/h)',
`z_gal` float COMMENT 'galaxy comoving z coordinate (Mpc/h)',
`r_gal` float COMMENT 'galaxy comoving distance (Mpc/h)',
`true_redshift_gal` float COMMENT 'true galaxy redshift',
`observed_redshift_gal` float COMMENT 'observed galaxy redshift (including peculiar velocity)',
`hpix_29_nest` bigint COMMENT 'healpix pixel id using order = 29 in NESTED format',
`vx_gal` float COMMENT 'physical (peculiar) velocity in x-component (km/s)',
`vy_gal` float COMMENT 'physical (peculiar) velocity in y-component (km/s)',
`vz_gal` float COMMENT 'physical (peculiar) velocity in z-component (km/s)',
`vrad_gal` float COMMENT 'missing comment',
`delta_r` float COMMENT 'missing comment',
`abs_mag_r01` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h)',
`luminosity_r01` float COMMENT 'luminosity in the 0.1-r-band',
`abs_mag_r01_evolved` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h) including MICECATv2.0 magnitude evolution',
`sdss_r01` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1',
`abs_mag_uv_dereddened` float COMMENT 'absolute magnitude ultraviolet',
`gr_restframe` float COMMENT '(g-r) rest-frame at z=0',
`color_kind` tinyint COMMENT 'galaxy color kind: 0=red sequence 1=green valley 2=blue cloud',
`sed_ke` float COMMENT 'K-correction for each SED',
`gr_cosmos` float COMMENT 'interpolated COSMOS g-r color',
`sed_cosmos` float COMMENT 'interpolated basic COSMOS SED [0 30]',
`is_within_cosmos` boolean COMMENT 'missing comment',
`cosmos_distance` float COMMENT 'missing comment',
`ext_curve_cosmos` tinyint COMMENT 'COSMOS extinction curve [0-4]',
`ebv_cosmos` float COMMENT 'interpolated extinction strength for COSMOS extinction curve',
`log_ml_r01` float COMMENT 'logarithm of stellar_mass/luminosity ratio in the r01 SDSS band',
`log_stellar_mass` float COMMENT 'logarithm of the stellar mass (Msun/h)',
`metallicity` float COMMENT 'oxygen metallicity in units 12+logO/H',
`log_sfr` float COMMENT 'logarithm of star formation rate in (Msun/h)/year',
`logf_halpha` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: DOES NOT include extinction)',
`logf_halpha_ext` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: includes extinction)',
`halpha_scatter` float COMMENT 'missing comment',
`loglum_halpha` float COMMENT 'DEPRECATED logarithm of Halpha luminosity (erg/s/h^2) using empirical relation with UV magnitude (DOES NOT include extinction)',
`loglum_halpha_ext` float COMMENT 'missing comment',
`logf_dummy` float COMMENT 'missing comment',
`z_dummy` float COMMENT 'missing comment',
`logf_halpha_model1_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: includes extinction)',
`logf_halpha_model1` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: DOES NOT include extinction)',
`loglum_halpha_model1_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (includes extinction)',
`loglum_halpha_model1` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (DOES NOT include extinction)',
`logf_halpha_model3_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: includes extinction)',
`logf_halpha_model3` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: DOES NOT include extinction)',
`loglum_halpha_model3_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (includes extinction)',
`loglum_halpha_model3` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (DOES NOT include extinction)',
`logf_hbeta_model1_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_hbeta_model1` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_n2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_n2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o3_model1_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o3_model1` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_s2_model1_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_s2_model1` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_hbeta_model3_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_hbeta_model3` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_n2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_n2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o3_model3_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o3_model3` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_s2_model3_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_s2_model3` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',

`2mass_h` float COMMENT 'observed flux in 2MASS H-band (Euclid provided)',
`2mass_j` float COMMENT 'observed flux in 2MASS J-band (Euclid provided)',
`2mass_ks` float COMMENT 'observed flux in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g` float COMMENT 'observed flux in DES g-band (Euclid provided)',
`blanco_decam_i` float COMMENT 'observed flux in DES i-band (Euclid provided)',
`blanco_decam_r` float COMMENT 'observed flux in DES r-band (Euclid provided)',
`blanco_decam_z` float COMMENT 'observed flux in DES z-band (Euclid provided)',
`cfht_megacam_r` float COMMENT 'observed flux in CFHT r-band (Euclid provided)',
`cfht_megacam_u` float COMMENT 'observed flux in CFHT u-band (Euclid provided)',
`euclid_nisp_h` float COMMENT 'observed flux in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j` float COMMENT 'observed flux in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y` float COMMENT 'observed flux in Euclid NISP-Y band (Euclid provided)',
`euclid_vis` float COMMENT 'observed flux in Euclid VIS band (Euclid provided)',
`gaia_bp` float COMMENT 'observed flux in Gaia-BP band (Euclid provided)',
`gaia_g` float COMMENT 'observed flux in Gaia-G band (Euclid provided)',
`gaia_rp` float COMMENT 'observed flux in Gaia-RP band (Euclid provided)',
`jst_jpcam_g` float COMMENT 'observed flux in JST g-band (Euclid provided)',
`kids_g` float COMMENT 'observed flux in KIDS g-band (Euclid provided)',
`kids_i` float COMMENT 'observed flux in KIDS i-band (Euclid provided)',
`kids_r` float COMMENT 'observed flux in KIDS r-band (Euclid provided)',
`kids_u` float COMMENT 'observed flux in KIDS u-band (Euclid provided)',
`lsst_g` float COMMENT 'observed flux in LSST g-band (Euclid provided)',
`lsst_i` float COMMENT 'observed flux in LSST i-band (Euclid provided)',
`lsst_r` float COMMENT 'observed flux in LSST r-band (Euclid provided)',
`lsst_u` float COMMENT 'observed flux in LSST u-band (Euclid provided)',
`lsst_y` float COMMENT 'observed flux in LSST Y-band (Euclid provided)',
`lsst_z` float COMMENT 'observed flux in LSST z-band (Euclid provided)',
`pan_starrs_i` float COMMENT 'observed flux in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z` float COMMENT 'observed flux in Pan-STARRS z-band (Euclid provided)',
`subaru_hsc_z` float COMMENT 'observed flux in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS H-band (Euclid provided)',
`2mass_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in JST g-band (Euclid provided)',
`kids_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_odonnell_ext` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1 with MW extinction',
`subaru_hsc_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS Ks-band (Euclid provided)',
`2mass_ks_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES z-band (Euclid provided)',
`blanco_decam_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT u-band (Euclid provided)',
`cfht_megacam_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_nisp_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid VIS band (Euclid provided)',
`euclid_vis_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_bp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-RP band (Euclid provided)',
`gaia_rp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in JST g-band (Euclid provided)',
`jst_jpcam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in JST g-band (Euclid provided)',
`kids_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS g-band (Euclid provided)',
`kids_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS i-band (Euclid provided)',
`kids_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS r-band (Euclid provided)',
`kids_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS u-band (Euclid provided)',
`kids_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST g-band (Euclid provided)',
`lsst_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST i-band (Euclid provided)',
`lsst_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST r-band (Euclid provided)',
`lsst_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST u-band (Euclid provided)',
`lsst_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST z-band (Euclid provided)',
`lsst_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`pan_starrs_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in sdss r-band redshifted to z=0.1',
`sdss_r01_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in sdss r-band redshifted to z=0.1',
`subaru_hsc_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`subaru_hsc_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',

`dominant_shape` tinyint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)',
`bulge_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`disk_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`median_major_axis` float COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)',
`scale_length` float COMMENT 'disc and bulge scalelength prior',
`bulge_fraction` float COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)',
`disk_scalelength` float COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)',
`disk_nsersic` float COMMENT 'Sersic index of the disk component',
`disk_r50` float COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678',
`bulge_r50` float COMMENT 'half light radius of the bulge',
`bulge_nsersic` float COMMENT 'Sersic index of the bulge component',
`disk_ellipticity` float COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)',
`bulge_ellipticity` float COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)',
`disk_axis_ratio` float COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`bulge_axis_ratio` float COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`inclination_angle` float COMMENT 'Galaxy inclination angle (where 0 degrees = face-on and 90 degrees = edge-on). Galaxy ellipticity for disk and bulge components are computed following the recipe in https://euclid.roe.ac.uk/projects/sgsshear/wiki/SHE-SIM',
`halo_x` float COMMENT 'halo comoving x coordinate (Mpc/h)',
`halo_y` float COMMENT 'halo comoving y coordinate (Mpc/h)',
`halo_z` float COMMENT 'halo comoving z coordinate (Mpc/h)',
`halo_vx` float COMMENT 'halo x-centre velocity (km/h)',
`halo_vy` float COMMENT 'halo y-centre velocity (km/h)',
`halo_vz` float COMMENT 'halo z-centre velocity (km/h)',
`halo_r` float COMMENT 'halo comoving distance (Mpc/h)',
`true_redshift_halo` float COMMENT 'true redshift of the host halo',
`halo_lm` float COMMENT 'log10 of the FoF halo mass in Msum/h. This halo mass is computed as the particule mass multiplied by the number of particles of the FoF halo. At low masses it is interpolated to obtain a smooth distribution of halo masses',
`halo_n_sats` int COMMENT 'number of satellite galaxies from HOD',
`n_gals` int COMMENT 'DEPRECATED',
`luminosity_r01_evolved` float COMMENT 'missing comment',
`hpix_13_nest` bigint COMMENT 'missing comment')
PARTITIONED BY (
`step` smallint COMMENT 'step number in the Flagship lensing maps')
CLUSTERED BY (
hpix_13_nest)
SORTED BY (
hpix_13_nest ASC)
INTO 128 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;

INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_1717_s_v2 PARTITION(step)
SELECT halo_id, galaxy_id, kind, random_index, ra_gal, dec_gal, ra_mag_gal, dec_mag_gal, kappa, gamma1, gamma2, l_gal, b_gal, mw_extinction, x_gal, y_gal, z_gal, r_gal, true_redshift_gal, observed_redshift_gal, hpix_29_nest, vx_gal, vy_gal, vz_gal, vrad_gal, delta_r, abs_mag_r01, luminosity_r01, abs_mag_r01_evolved, sdss_r01, abs_mag_uv_dereddened, gr_restframe, color_kind, sed_ke, gr_cosmos, sed_cosmos, is_within_cosmos, cosmos_distance, ext_curve_cosmos, ebv_cosmos, log_ml_r01, log_stellar_mass, metallicity, log_sfr, logf_halpha, logf_halpha_ext, halpha_scatter, loglum_halpha, loglum_halpha_ext, logf_dummy, z_dummy, logf_halpha_model1_ext, logf_halpha_model1, loglum_halpha_model1_ext, loglum_halpha_model1, logf_halpha_model3_ext, logf_halpha_model3, loglum_halpha_model3_ext, loglum_halpha_model3, logf_hbeta_model1_ext, logf_hbeta_model1, logf_o2_model1_ext, logf_o2_model1, logf_n2_model1_ext, logf_n2_model1, logf_o3_model1_ext, logf_o3_model1, logf_s2_model1_ext, logf_s2_model1, logf_hbeta_model3_ext, logf_hbeta_model3, logf_o2_model3_ext, logf_o2_model3, logf_n2_model3_ext, logf_n2_model3, logf_o3_model3_ext, logf_o3_model3, logf_s2_model3_ext, logf_s2_model3, 2mass_h, 2mass_j, 2mass_ks, blanco_decam_g, blanco_decam_i, blanco_decam_r, blanco_decam_z, cfht_megacam_r, cfht_megacam_u, euclid_nisp_h, euclid_nisp_j, euclid_nisp_y, euclid_vis, gaia_bp, gaia_g, gaia_rp, jst_jpcam_g, kids_g, kids_i, kids_r, kids_u, lsst_g, lsst_i, lsst_r, lsst_u, lsst_y, lsst_z, pan_starrs_i, pan_starrs_z, subaru_hsc_z, 2mass_h_odonnell_ext, 2mass_j_odonnell_ext, 2mass_ks_odonnell_ext, blanco_decam_g_odonnell_ext, blanco_decam_i_odonnell_ext, blanco_decam_r_odonnell_ext, blanco_decam_z_odonnell_ext, cfht_megacam_r_odonnell_ext, cfht_megacam_u_odonnell_ext, euclid_nisp_h_odonnell_ext, euclid_nisp_j_odonnell_ext, euclid_nisp_y_odonnell_ext, euclid_vis_odonnell_ext, gaia_bp_odonnell_ext, gaia_g_odonnell_ext, gaia_rp_odonnell_ext, jst_jpcam_g_odonnell_ext, kids_g_odonnell_ext, kids_i_odonnell_ext, kids_r_odonnell_ext, kids_u_odonnell_ext, lsst_g_odonnell_ext, lsst_i_odonnell_ext, lsst_r_odonnell_ext, lsst_u_odonnell_ext, lsst_y_odonnell_ext, lsst_z_odonnell_ext, pan_starrs_i_odonnell_ext, pan_starrs_z_odonnell_ext, sdss_r01_odonnell_ext, subaru_hsc_z_odonnell_ext, 2mass_h_el_model1_odonnell_ext, 2mass_h_el_model3_odonnell_ext, 2mass_j_el_model1_odonnell_ext, 2mass_j_el_model3_odonnell_ext, 2mass_ks_el_model1_odonnell_ext, 2mass_ks_el_model3_odonnell_ext, blanco_decam_g_el_model1_odonnell_ext, blanco_decam_g_el_model3_odonnell_ext, blanco_decam_i_el_model1_odonnell_ext, blanco_decam_i_el_model3_odonnell_ext, blanco_decam_r_el_model1_odonnell_ext, blanco_decam_r_el_model3_odonnell_ext, blanco_decam_z_el_model1_odonnell_ext, blanco_decam_z_el_model3_odonnell_ext, cfht_megacam_r_el_model1_odonnell_ext, cfht_megacam_r_el_model3_odonnell_ext, cfht_megacam_u_el_model1_odonnell_ext, cfht_megacam_u_el_model3_odonnell_ext, euclid_nisp_h_el_model1_odonnell_ext, euclid_nisp_h_el_model3_odonnell_ext, euclid_nisp_j_el_model1_odonnell_ext, euclid_nisp_j_el_model3_odonnell_ext, euclid_nisp_y_el_model1_odonnell_ext, euclid_nisp_y_el_model3_odonnell_ext, euclid_vis_el_model1_odonnell_ext, euclid_vis_el_model3_odonnell_ext, gaia_bp_el_model1_odonnell_ext, gaia_bp_el_model3_odonnell_ext, gaia_g_el_model1_odonnell_ext, gaia_g_el_model3_odonnell_ext, gaia_rp_el_model1_odonnell_ext, gaia_rp_el_model3_odonnell_ext, jst_jpcam_g_el_model1_odonnell_ext, jst_jpcam_g_el_model3_odonnell_ext, kids_g_el_model1_odonnell_ext, kids_g_el_model3_odonnell_ext, kids_i_el_model1_odonnell_ext, kids_i_el_model3_odonnell_ext, kids_r_el_model1_odonnell_ext, kids_r_el_model3_odonnell_ext, kids_u_el_model1_odonnell_ext, kids_u_el_model3_odonnell_ext, lsst_g_el_model1_odonnell_ext, lsst_g_el_model3_odonnell_ext, lsst_i_el_model1_odonnell_ext, lsst_i_el_model3_odonnell_ext, lsst_r_el_model1_odonnell_ext, lsst_r_el_model3_odonnell_ext, lsst_u_el_model1_odonnell_ext, lsst_u_el_model3_odonnell_ext, lsst_y_el_model1_odonnell_ext, lsst_y_el_model3_odonnell_ext, lsst_z_el_model1_odonnell_ext, lsst_z_el_model3_odonnell_ext, pan_starrs_i_el_model1_odonnell_ext, pan_starrs_i_el_model3_odonnell_ext, pan_starrs_z_el_model1_odonnell_ext, pan_starrs_z_el_model3_odonnell_ext, sdss_r01_el_model1_odonnell_ext, sdss_r01_el_model3_odonnell_ext, subaru_hsc_z_el_model1_odonnell_ext, subaru_hsc_z_el_model3_odonnell_ext, dominant_shape, bulge_angle, disk_angle, median_major_axis, scale_length, bulge_fraction, disk_scalelength, disk_nsersic, disk_r50, bulge_r50, bulge_nsersic, disk_ellipticity, bulge_ellipticity, disk_axis_ratio, bulge_axis_ratio, inclination_angle, halo_x, halo_y, halo_z, halo_vx, halo_vy, halo_vz, halo_r, true_redshift_halo, halo_lm, halo_n_sats, n_gals, luminosity_r01_evolved, hpix_13_nest, step
FROM jcarrete.flagship_mock_obsid_1717_s_v2_pq;

La paso al esquema cosmohub

ALTER TABLE jcarrete.flagship_mock_obsid_1717_s_v2 RENAME TO cosmohub.flagship_mock_obsid_1717_s_v2;

----

'''Ahora es necesario volver a calcular los flujos! Pero únicamente para el filtro LSST!'''

Abro la pipeline de flujos y con la siguiente SQL (cojo todo menos los LSST filters):
(y solo añado el filtro LSST en la lista de air filters!)

halo_sql = """
SELECT halo_id, galaxy_id, kind, random_index, ra_gal, dec_gal, ra_mag_gal, dec_mag_gal,
kappa, gamma1, gamma2, l_gal, b_gal, mw_extinction,
x_gal, y_gal, z_gal, r_gal, true_redshift_gal, observed_redshift_gal,
hpix_29_nest, vx_gal, vy_gal, vz_gal, vrad_gal, delta_r, abs_mag_r01,
luminosity_r01, abs_mag_r01_evolved, sdss_r01, abs_mag_uv_dereddened, gr_restframe,
color_kind, sed_ke, gr_cosmos, sed_cosmos, is_within_cosmos, cosmos_distance,
ext_curve_cosmos, ebv_cosmos, log_ml_r01, log_stellar_mass, metallicity, log_sfr,
logf_halpha, logf_halpha_ext, halpha_scatter, loglum_halpha, loglum_halpha_ext,
logf_dummy, z_dummy, logf_halpha_model1_ext, logf_halpha_model1, loglum_halpha_model1_ext,
loglum_halpha_model1, logf_halpha_model3_ext, logf_halpha_model3, loglum_halpha_model3_ext,
loglum_halpha_model3, logf_hbeta_model1_ext, logf_hbeta_model1, logf_o2_model1_ext,
logf_o2_model1, logf_n2_model1_ext, logf_n2_model1, logf_o3_model1_ext, logf_o3_model1,
logf_s2_model1_ext, logf_s2_model1, logf_hbeta_model3_ext, logf_hbeta_model3,
logf_o2_model3_ext, logf_o2_model3, logf_n2_model3_ext, logf_n2_model3, logf_o3_model3_ext,
logf_o3_model3, logf_s2_model3_ext, logf_s2_model3, 2mass_h, 2mass_j, 2mass_ks, blanco_decam_g, blanco_decam_i, blanco_decam_r, blanco_decam_z, cfht_megacam_r, cfht_megacam_u, euclid_nisp_h, euclid_nisp_j, euclid_nisp_y, euclid_vis, gaia_bp, gaia_g, gaia_rp, jst_jpcam_g, kids_g, kids_i, kids_r, kids_u, pan_starrs_i, pan_starrs_z, subaru_hsc_z, 2mass_h_odonnell_ext, 2mass_j_odonnell_ext, 2mass_ks_odonnell_ext, blanco_decam_g_odonnell_ext, blanco_decam_i_odonnell_ext, blanco_decam_r_odonnell_ext, blanco_decam_z_odonnell_ext, cfht_megacam_r_odonnell_ext, cfht_megacam_u_odonnell_ext, euclid_nisp_h_odonnell_ext, euclid_nisp_j_odonnell_ext, euclid_nisp_y_odonnell_ext, euclid_vis_odonnell_ext, gaia_bp_odonnell_ext, gaia_g_odonnell_ext, gaia_rp_odonnell_ext, jst_jpcam_g_odonnell_ext, kids_g_odonnell_ext, kids_i_odonnell_ext, kids_r_odonnell_ext, kids_u_odonnell_ext, pan_starrs_i_odonnell_ext, pan_starrs_z_odonnell_ext, sdss_r01_odonnell_ext, subaru_hsc_z_odonnell_ext, 2mass_h_el_model1_odonnell_ext, 2mass_h_el_model3_odonnell_ext, 2mass_j_el_model1_odonnell_ext, 2mass_j_el_model3_odonnell_ext, 2mass_ks_el_model1_odonnell_ext, 2mass_ks_el_model3_odonnell_ext, blanco_decam_g_el_model1_odonnell_ext, blanco_decam_g_el_model3_odonnell_ext, blanco_decam_i_el_model1_odonnell_ext, blanco_decam_i_el_model3_odonnell_ext, blanco_decam_r_el_model1_odonnell_ext, blanco_decam_r_el_model3_odonnell_ext, blanco_decam_z_el_model1_odonnell_ext, blanco_decam_z_el_model3_odonnell_ext, cfht_megacam_r_el_model1_odonnell_ext, cfht_megacam_r_el_model3_odonnell_ext, cfht_megacam_u_el_model1_odonnell_ext, cfht_megacam_u_el_model3_odonnell_ext, euclid_nisp_h_el_model1_odonnell_ext, euclid_nisp_h_el_model3_odonnell_ext, euclid_nisp_j_el_model1_odonnell_ext, euclid_nisp_j_el_model3_odonnell_ext, euclid_nisp_y_el_model1_odonnell_ext, euclid_nisp_y_el_model3_odonnell_ext, euclid_vis_el_model1_odonnell_ext, euclid_vis_el_model3_odonnell_ext, gaia_bp_el_model1_odonnell_ext, gaia_bp_el_model3_odonnell_ext, gaia_g_el_model1_odonnell_ext, gaia_g_el_model3_odonnell_ext, gaia_rp_el_model1_odonnell_ext, gaia_rp_el_model3_odonnell_ext, jst_jpcam_g_el_model1_odonnell_ext, jst_jpcam_g_el_model3_odonnell_ext, kids_g_el_model1_odonnell_ext, kids_g_el_model3_odonnell_ext, kids_i_el_model1_odonnell_ext, kids_i_el_model3_odonnell_ext, kids_r_el_model1_odonnell_ext, kids_r_el_model3_odonnell_ext, kids_u_el_model1_odonnell_ext, kids_u_el_model3_odonnell_ext, pan_starrs_i_el_model1_odonnell_ext, pan_starrs_i_el_model3_odonnell_ext, pan_starrs_z_el_model1_odonnell_ext, pan_starrs_z_el_model3_odonnell_ext, sdss_r01_el_model1_odonnell_ext, sdss_r01_el_model3_odonnell_ext, subaru_hsc_z_el_model1_odonnell_ext, subaru_hsc_z_el_model3_odonnell_ext, dominant_shape, bulge_angle, disk_angle, median_major_axis, scale_length, bulge_fraction, disk_scalelength, disk_nsersic, disk_r50, bulge_r50, bulge_nsersic, disk_ellipticity, bulge_ellipticity, disk_axis_ratio, bulge_axis_ratio, inclination_angle, halo_x, halo_y, halo_z, halo_vx, halo_vy, halo_vz, halo_r, true_redshift_halo, halo_lm, halo_n_sats, n_gals, luminosity_r01_evolved, hpix_13_nest, step
FROM cosmohub.flagship_mock_obsid_1717_s_v2
"""

/software/astro/scripts/create_table_from_parquet.sh -p /hdfs/user/jcarrete/data/euclid/flagship_mock_obsid_1717_s_v3/0.pq -t jcarrete.flagship_mock_obsid_1717_s_v3_pq -e

CREATE EXTERNAL TABLE jcarrete.flagship_mock_obsid_1717_s_v3_pq (
`kind` BIGINT,
`random_index` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`ra_mag_gal` DOUBLE,
`dec_mag_gal` DOUBLE,
`kappa` DOUBLE,
`gamma1` DOUBLE,
`gamma2` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`observed_redshift_gal` DOUBLE,
`hpix_29_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`abs_mag_r01` DOUBLE,
`luminosity_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`sdss_r01` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` BIGINT,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha` DOUBLE,
`logf_halpha_ext` DOUBLE,
`halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_halpha_model3_ext` DOUBLE,
`loglum_halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`2mass_h` DOUBLE,
`2mass_j` DOUBLE,
`2mass_ks` DOUBLE,
`blanco_decam_g` DOUBLE,
`blanco_decam_i` DOUBLE,
`blanco_decam_r` DOUBLE,
`blanco_decam_z` DOUBLE,
`cfht_megacam_r` DOUBLE,
`cfht_megacam_u` DOUBLE,
`euclid_nisp_h` DOUBLE,
`euclid_nisp_j` DOUBLE,
`euclid_nisp_y` DOUBLE,
`euclid_vis` DOUBLE,
`gaia_bp` DOUBLE,
`gaia_g` DOUBLE,
`gaia_rp` DOUBLE,
`jst_jpcam_g` DOUBLE,
`kids_g` DOUBLE,
`kids_i` DOUBLE,
`kids_r` DOUBLE,
`kids_u` DOUBLE,
`pan_starrs_i` DOUBLE,
`pan_starrs_z` DOUBLE,
`subaru_hsc_z` DOUBLE,
`2mass_h_odonnell_ext` DOUBLE,
`2mass_j_odonnell_ext` DOUBLE,
`2mass_ks_odonnell_ext` DOUBLE,
`blanco_decam_g_odonnell_ext` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`blanco_decam_r_odonnell_ext` DOUBLE,
`blanco_decam_z_odonnell_ext` DOUBLE,
`cfht_megacam_r_odonnell_ext` DOUBLE,
`cfht_megacam_u_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`euclid_nisp_j_odonnell_ext` DOUBLE,
`euclid_nisp_y_odonnell_ext` DOUBLE,
`euclid_vis_odonnell_ext` DOUBLE,
`gaia_bp_odonnell_ext` DOUBLE,
`gaia_g_odonnell_ext` DOUBLE,
`gaia_rp_odonnell_ext` DOUBLE,
`jst_jpcam_g_odonnell_ext` DOUBLE,
`kids_g_odonnell_ext` DOUBLE,
`kids_i_odonnell_ext` DOUBLE,
`kids_r_odonnell_ext` DOUBLE,
`kids_u_odonnell_ext` DOUBLE,
`pan_starrs_i_odonnell_ext` DOUBLE,
`pan_starrs_z_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`subaru_hsc_z_odonnell_ext` DOUBLE,
`2mass_h_el_model1_odonnell_ext` DOUBLE,
`2mass_h_el_model3_odonnell_ext` DOUBLE,
`2mass_j_el_model1_odonnell_ext` DOUBLE,
`2mass_j_el_model3_odonnell_ext` DOUBLE,
`2mass_ks_el_model1_odonnell_ext` DOUBLE,
`2mass_ks_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model3_odonnell_ext` DOUBLE,
`euclid_vis_el_model1_odonnell_ext` DOUBLE,
`euclid_vis_el_model3_odonnell_ext` DOUBLE,
`gaia_bp_el_model1_odonnell_ext` DOUBLE,
`gaia_bp_el_model3_odonnell_ext` DOUBLE,
`gaia_g_el_model1_odonnell_ext` DOUBLE,
`gaia_g_el_model3_odonnell_ext` DOUBLE,
`gaia_rp_el_model1_odonnell_ext` DOUBLE,
`gaia_rp_el_model3_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model1_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model3_odonnell_ext` DOUBLE,
`kids_g_el_model1_odonnell_ext` DOUBLE,
`kids_g_el_model3_odonnell_ext` DOUBLE,
`kids_i_el_model1_odonnell_ext` DOUBLE,
`kids_i_el_model3_odonnell_ext` DOUBLE,
`kids_r_el_model1_odonnell_ext` DOUBLE,
`kids_r_el_model3_odonnell_ext` DOUBLE,
`kids_u_el_model1_odonnell_ext` DOUBLE,
`kids_u_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model1_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model3_odonnell_ext` DOUBLE,
`dominant_shape` BIGINT,
`bulge_angle` DOUBLE,
`disk_angle` DOUBLE,
`median_major_axis` DOUBLE,
`scale_length` DOUBLE,
`bulge_fraction` DOUBLE,
`disk_scalelength` DOUBLE,
`disk_nsersic` DOUBLE,
`disk_r50` DOUBLE,
`bulge_r50` DOUBLE,
`bulge_nsersic` DOUBLE,
`disk_ellipticity` DOUBLE,
`bulge_ellipticity` DOUBLE,
`disk_axis_ratio` DOUBLE,
`bulge_axis_ratio` DOUBLE,
`inclination_angle` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`luminosity_r01_evolved` DOUBLE,
`hpix_13_nest` BIGINT,
`step` BIGINT,
`lsst_g` DOUBLE,
`lsst_i` DOUBLE,
`lsst_r` DOUBLE,
`lsst_u` DOUBLE,
`lsst_y` DOUBLE,
`lsst_z` DOUBLE,
`lsst_g_odonnell_ext` DOUBLE,
`lsst_i_odonnell_ext` DOUBLE,
`lsst_r_odonnell_ext` DOUBLE,
`lsst_u_odonnell_ext` DOUBLE,
`lsst_y_odonnell_ext` DOUBLE,
`lsst_z_odonnell_ext` DOUBLE,
`lsst_g_el_model1_odonnell_ext` DOUBLE,
`lsst_g_el_model3_odonnell_ext` DOUBLE,
`lsst_i_el_model1_odonnell_ext` DOUBLE,
`lsst_i_el_model3_odonnell_ext` DOUBLE,
`lsst_r_el_model1_odonnell_ext` DOUBLE,
`lsst_r_el_model3_odonnell_ext` DOUBLE,
`lsst_u_el_model1_odonnell_ext` DOUBLE,
`lsst_u_el_model3_odonnell_ext` DOUBLE,
`lsst_y_el_model1_odonnell_ext` DOUBLE,
`lsst_y_el_model3_odonnell_ext` DOUBLE,
`lsst_z_el_model1_odonnell_ext` DOUBLE,
`lsst_z_el_model3_odonnell_ext` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
STORED AS PARQUET
LOCATION '/user/jcarrete/data/euclid/flagship_mock_obsid_1717_s_v3'
;

Creo la tabla en ORC:

CREATE TABLE jcarrete.flagship_mock_obsid_1717_s_v3(
`halo_id` bigint COMMENT 'unique halo id coming from the Flagship dark matter halo catalog',
`galaxy_id` int COMMENT 'combined with the unique_halo_id they uniquely identify each galaxy',
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',
`random_index` float COMMENT 'random number [0 - 1) for subsampling',
`ra_gal` float COMMENT 'galaxy right ascension (degrees)',
`dec_gal` float COMMENT 'galaxy declination (degrees)',
`ra_mag_gal` float COMMENT 'galaxy magnified right ascension (degree)',
`dec_mag_gal` float COMMENT 'galaxy magnified declination (degree)',
`kappa` float COMMENT 'convergence',
`gamma1` float COMMENT 'shear',
`gamma2` float COMMENT 'shear',
`l_gal` float COMMENT 'galactic longitude (degrees)',
`b_gal` float COMMENT 'galactic latitude (degrees)',
`mw_extinction` float COMMENT 'EBV from file: HFI_CompMap_ThermalDustModel_2048_R1.20.fits',
`x_gal` float COMMENT 'galaxy comoving x coordinate (Mpc/h)',
`y_gal` float COMMENT 'galaxy comoving y coordinate (Mpc/h)',
`z_gal` float COMMENT 'galaxy comoving z coordinate (Mpc/h)',
`r_gal` float COMMENT 'galaxy comoving distance (Mpc/h)',
`true_redshift_gal` float COMMENT 'true galaxy redshift',
`observed_redshift_gal` float COMMENT 'observed galaxy redshift (including peculiar velocity)',
`hpix_29_nest` bigint COMMENT 'healpix pixel id using order = 29 in NESTED format',
`vx_gal` float COMMENT 'physical (peculiar) velocity in x-component (km/s)',
`vy_gal` float COMMENT 'physical (peculiar) velocity in y-component (km/s)',
`vz_gal` float COMMENT 'physical (peculiar) velocity in z-component (km/s)',
`vrad_gal` float COMMENT 'missing comment',
`delta_r` float COMMENT 'missing comment',
`abs_mag_r01` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h)',
`luminosity_r01` float COMMENT 'luminosity in the 0.1-r-band',
`abs_mag_r01_evolved` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h) including MICECATv2.0 magnitude evolution',
`sdss_r01` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1',
`abs_mag_uv_dereddened` float COMMENT 'absolute magnitude ultraviolet',
`gr_restframe` float COMMENT '(g-r) rest-frame at z=0',
`color_kind` tinyint COMMENT 'galaxy color kind: 0=red sequence 1=green valley 2=blue cloud',
`sed_ke` float COMMENT 'K-correction for each SED',
`gr_cosmos` float COMMENT 'interpolated COSMOS g-r color',
`sed_cosmos` float COMMENT 'interpolated basic COSMOS SED [0 30]',
`is_within_cosmos` boolean COMMENT 'missing comment',
`cosmos_distance` float COMMENT 'missing comment',
`ext_curve_cosmos` tinyint COMMENT 'COSMOS extinction curve [0-4]',
`ebv_cosmos` float COMMENT 'interpolated extinction strength for COSMOS extinction curve',
`log_ml_r01` float COMMENT 'logarithm of stellar_mass/luminosity ratio in the r01 SDSS band',
`log_stellar_mass` float COMMENT 'logarithm of the stellar mass (Msun/h)',
`metallicity` float COMMENT 'oxygen metallicity in units 12+logO/H',
`log_sfr` float COMMENT 'logarithm of star formation rate in (Msun/h)/year',
`logf_halpha` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: DOES NOT include extinction)',
`logf_halpha_ext` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: includes extinction)',
`halpha_scatter` float COMMENT 'missing comment',
`loglum_halpha` float COMMENT 'DEPRECATED logarithm of Halpha luminosity (erg/s/h^2) using empirical relation with UV magnitude (DOES NOT include extinction)',
`loglum_halpha_ext` float COMMENT 'missing comment',
`logf_dummy` float COMMENT 'missing comment',
`z_dummy` float COMMENT 'missing comment',
`logf_halpha_model1_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: includes extinction)',
`logf_halpha_model1` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: DOES NOT include extinction)',
`loglum_halpha_model1_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (includes extinction)',
`loglum_halpha_model1` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (DOES NOT include extinction)',
`logf_halpha_model3_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: includes extinction)',
`logf_halpha_model3` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: DOES NOT include extinction)',
`loglum_halpha_model3_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (includes extinction)',
`loglum_halpha_model3` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (DOES NOT include extinction)',
`logf_hbeta_model1_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_hbeta_model1` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_n2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_n2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o3_model1_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o3_model1` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_s2_model1_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_s2_model1` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_hbeta_model3_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_hbeta_model3` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_n2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_n2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o3_model3_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o3_model3` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_s2_model3_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_s2_model3` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`2mass_h` float COMMENT 'observed flux in 2MASS H-band (Euclid provided)',
`2mass_j` float COMMENT 'observed flux in 2MASS J-band (Euclid provided)',
`2mass_ks` float COMMENT 'observed flux in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g` float COMMENT 'observed flux in DES g-band (Euclid provided)',
`blanco_decam_i` float COMMENT 'observed flux in DES i-band (Euclid provided)',
`blanco_decam_r` float COMMENT 'observed flux in DES r-band (Euclid provided)',
`blanco_decam_z` float COMMENT 'observed flux in DES z-band (Euclid provided)',
`cfht_megacam_r` float COMMENT 'observed flux in CFHT r-band (Euclid provided)',
`cfht_megacam_u` float COMMENT 'observed flux in CFHT u-band (Euclid provided)',
`euclid_nisp_h` float COMMENT 'observed flux in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j` float COMMENT 'observed flux in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y` float COMMENT 'observed flux in Euclid NISP-Y band (Euclid provided)',
`euclid_vis` float COMMENT 'observed flux in Euclid VIS band (Euclid provided)',
`gaia_bp` float COMMENT 'observed flux in Gaia-BP band (Euclid provided)',
`gaia_g` float COMMENT 'observed flux in Gaia-G band (Euclid provided)',
`gaia_rp` float COMMENT 'observed flux in Gaia-RP band (Euclid provided)',
`jst_jpcam_g` float COMMENT 'observed flux in JST g-band (Euclid provided)',
`kids_g` float COMMENT 'observed flux in KIDS g-band (Euclid provided)',
`kids_i` float COMMENT 'observed flux in KIDS i-band (Euclid provided)',
`kids_r` float COMMENT 'observed flux in KIDS r-band (Euclid provided)',
`kids_u` float COMMENT 'observed flux in KIDS u-band (Euclid provided)',
`lsst_g` float COMMENT 'observed flux in LSST g-band (Euclid provided)',
`lsst_i` float COMMENT 'observed flux in LSST i-band (Euclid provided)',
`lsst_r` float COMMENT 'observed flux in LSST r-band (Euclid provided)',
`lsst_u` float COMMENT 'observed flux in LSST u-band (Euclid provided)',
`lsst_y` float COMMENT 'observed flux in LSST Y-band (Euclid provided)',
`lsst_z` float COMMENT 'observed flux in LSST z-band (Euclid provided)',
`pan_starrs_i` float COMMENT 'observed flux in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z` float COMMENT 'observed flux in Pan-STARRS z-band (Euclid provided)',
`subaru_hsc_z` float COMMENT 'observed flux in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS H-band (Euclid provided)',
`2mass_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in JST g-band (Euclid provided)',
`kids_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_odonnell_ext` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1 with MW extinction',
`subaru_hsc_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS Ks-band (Euclid provided)',
`2mass_ks_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES z-band (Euclid provided)',
`blanco_decam_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT u-band (Euclid provided)',
`cfht_megacam_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_nisp_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid VIS band (Euclid provided)',
`euclid_vis_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_bp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-RP band (Euclid provided)',
`gaia_rp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in JST g-band (Euclid provided)',
`jst_jpcam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in JST g-band (Euclid provided)',
`kids_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS g-band (Euclid provided)',
`kids_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS i-band (Euclid provided)',
`kids_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS r-band (Euclid provided)',
`kids_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS u-band (Euclid provided)',
`kids_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST g-band (Euclid provided)',
`lsst_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST i-band (Euclid provided)',
`lsst_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST r-band (Euclid provided)',
`lsst_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST u-band (Euclid provided)',
`lsst_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST z-band (Euclid provided)',
`lsst_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`pan_starrs_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in sdss r-band redshifted to z=0.1',
`sdss_r01_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in sdss r-band redshifted to z=0.1',
`subaru_hsc_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`subaru_hsc_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`dominant_shape` tinyint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)',
`bulge_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`disk_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`median_major_axis` float COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)',
`scale_length` float COMMENT 'disc and bulge scalelength prior',
`bulge_fraction` float COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)',
`disk_scalelength` float COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)',
`disk_nsersic` float COMMENT 'Sersic index of the disk component',
`disk_r50` float COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678',
`bulge_r50` float COMMENT 'half light radius of the bulge',
`bulge_nsersic` float COMMENT 'Sersic index of the bulge component',
`disk_ellipticity` float COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)',
`bulge_ellipticity` float COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)',
`disk_axis_ratio` float COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`bulge_axis_ratio` float COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`inclination_angle` float COMMENT 'Galaxy inclination angle (where 0 degrees = face-on and 90 degrees = edge-on). Galaxy ellipticity for disk and bulge components are computed following the recipe in https://euclid.roe.ac.uk/projects/sgsshear/wiki/SHE-SIM',
`halo_x` float COMMENT 'halo comoving x coordinate (Mpc/h)',
`halo_y` float COMMENT 'halo comoving y coordinate (Mpc/h)',
`halo_z` float COMMENT 'halo comoving z coordinate (Mpc/h)',
`halo_vx` float COMMENT 'halo x-centre velocity (km/h)',
`halo_vy` float COMMENT 'halo y-centre velocity (km/h)',
`halo_vz` float COMMENT 'halo z-centre velocity (km/h)',
`halo_r` float COMMENT 'halo comoving distance (Mpc/h)',
`true_redshift_halo` float COMMENT 'true redshift of the host halo',
`halo_lm` float COMMENT 'log10 of the FoF halo mass in Msum/h. This halo mass is computed as the particule mass multiplied by the number of particles of the FoF halo. At low masses it is interpolated to obtain a smooth distribution of halo masses',
`halo_n_sats` int COMMENT 'number of satellite galaxies from HOD',
`n_gals` int COMMENT 'DEPRECATED',
`luminosity_r01_evolved` float COMMENT 'missing comment',
`hpix_13_nest` bigint COMMENT 'missing comment')
PARTITIONED BY (
`step` smallint COMMENT 'step number in the Flagship lensing maps')
CLUSTERED BY (
hpix_13_nest)
SORTED BY (
hpix_13_nest ASC)
INTO 128 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;

----

'''OJO PORQUE LE CAMBIO EL SIGNO AL GAMMA1 para que NO SE TENGA QUE HACER A NIVEL DE FITS FILES!!!! :'''

----

INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_1717_s_v3 PARTITION(step)
SELECT halo_id, galaxy_id, kind, random_index, ra_gal, dec_gal, ra_mag_gal, dec_mag_gal, kappa, -gamma1 as gamma1, gamma2, l_gal, b_gal, mw_extinction, x_gal, y_gal, z_gal, r_gal, true_redshift_gal, observed_redshift_gal, hpix_29_nest, vx_gal, vy_gal, vz_gal, vrad_gal, delta_r, abs_mag_r01, luminosity_r01, abs_mag_r01_evolved, sdss_r01, abs_mag_uv_dereddened, gr_restframe, color_kind, sed_ke, gr_cosmos, sed_cosmos, is_within_cosmos, cosmos_distance, ext_curve_cosmos, ebv_cosmos, log_ml_r01, log_stellar_mass, metallicity, log_sfr, logf_halpha, logf_halpha_ext, halpha_scatter, loglum_halpha, loglum_halpha_ext, logf_dummy, z_dummy, logf_halpha_model1_ext, logf_halpha_model1, loglum_halpha_model1_ext, loglum_halpha_model1, logf_halpha_model3_ext, logf_halpha_model3, loglum_halpha_model3_ext, loglum_halpha_model3, logf_hbeta_model1_ext, logf_hbeta_model1, logf_o2_model1_ext, logf_o2_model1, logf_n2_model1_ext, logf_n2_model1, logf_o3_model1_ext, logf_o3_model1, logf_s2_model1_ext, logf_s2_model1, logf_hbeta_model3_ext, logf_hbeta_model3, logf_o2_model3_ext, logf_o2_model3, logf_n2_model3_ext, logf_n2_model3, logf_o3_model3_ext, logf_o3_model3, logf_s2_model3_ext, logf_s2_model3, 2mass_h, 2mass_j, 2mass_ks, blanco_decam_g, blanco_decam_i, blanco_decam_r, blanco_decam_z, cfht_megacam_r, cfht_megacam_u, euclid_nisp_h, euclid_nisp_j, euclid_nisp_y, euclid_vis, gaia_bp, gaia_g, gaia_rp, jst_jpcam_g, kids_g, kids_i, kids_r, kids_u, lsst_g, lsst_i, lsst_r, lsst_u, lsst_y, lsst_z, pan_starrs_i, pan_starrs_z, subaru_hsc_z, 2mass_h_odonnell_ext, 2mass_j_odonnell_ext, 2mass_ks_odonnell_ext, blanco_decam_g_odonnell_ext, blanco_decam_i_odonnell_ext, blanco_decam_r_odonnell_ext, blanco_decam_z_odonnell_ext, cfht_megacam_r_odonnell_ext, cfht_megacam_u_odonnell_ext, euclid_nisp_h_odonnell_ext, euclid_nisp_j_odonnell_ext, euclid_nisp_y_odonnell_ext, euclid_vis_odonnell_ext, gaia_bp_odonnell_ext, gaia_g_odonnell_ext, gaia_rp_odonnell_ext, jst_jpcam_g_odonnell_ext, kids_g_odonnell_ext, kids_i_odonnell_ext, kids_r_odonnell_ext, kids_u_odonnell_ext, lsst_g_odonnell_ext, lsst_i_odonnell_ext, lsst_r_odonnell_ext, lsst_u_odonnell_ext, lsst_y_odonnell_ext, lsst_z_odonnell_ext, pan_starrs_i_odonnell_ext, pan_starrs_z_odonnell_ext, sdss_r01_odonnell_ext, subaru_hsc_z_odonnell_ext, 2mass_h_el_model1_odonnell_ext, 2mass_h_el_model3_odonnell_ext, 2mass_j_el_model1_odonnell_ext, 2mass_j_el_model3_odonnell_ext, 2mass_ks_el_model1_odonnell_ext, 2mass_ks_el_model3_odonnell_ext, blanco_decam_g_el_model1_odonnell_ext, blanco_decam_g_el_model3_odonnell_ext, blanco_decam_i_el_model1_odonnell_ext, blanco_decam_i_el_model3_odonnell_ext, blanco_decam_r_el_model1_odonnell_ext, blanco_decam_r_el_model3_odonnell_ext, blanco_decam_z_el_model1_odonnell_ext, blanco_decam_z_el_model3_odonnell_ext, cfht_megacam_r_el_model1_odonnell_ext, cfht_megacam_r_el_model3_odonnell_ext, cfht_megacam_u_el_model1_odonnell_ext, cfht_megacam_u_el_model3_odonnell_ext, euclid_nisp_h_el_model1_odonnell_ext, euclid_nisp_h_el_model3_odonnell_ext, euclid_nisp_j_el_model1_odonnell_ext, euclid_nisp_j_el_model3_odonnell_ext, euclid_nisp_y_el_model1_odonnell_ext, euclid_nisp_y_el_model3_odonnell_ext, euclid_vis_el_model1_odonnell_ext, euclid_vis_el_model3_odonnell_ext, gaia_bp_el_model1_odonnell_ext, gaia_bp_el_model3_odonnell_ext, gaia_g_el_model1_odonnell_ext, gaia_g_el_model3_odonnell_ext, gaia_rp_el_model1_odonnell_ext, gaia_rp_el_model3_odonnell_ext, jst_jpcam_g_el_model1_odonnell_ext, jst_jpcam_g_el_model3_odonnell_ext, kids_g_el_model1_odonnell_ext, kids_g_el_model3_odonnell_ext, kids_i_el_model1_odonnell_ext, kids_i_el_model3_odonnell_ext, kids_r_el_model1_odonnell_ext, kids_r_el_model3_odonnell_ext, kids_u_el_model1_odonnell_ext, kids_u_el_model3_odonnell_ext, lsst_g_el_model1_odonnell_ext, lsst_g_el_model3_odonnell_ext, lsst_i_el_model1_odonnell_ext, lsst_i_el_model3_odonnell_ext, lsst_r_el_model1_odonnell_ext, lsst_r_el_model3_odonnell_ext, lsst_u_el_model1_odonnell_ext, lsst_u_el_model3_odonnell_ext, lsst_y_el_model1_odonnell_ext, lsst_y_el_model3_odonnell_ext, lsst_z_el_model1_odonnell_ext, lsst_z_el_model3_odonnell_ext, pan_starrs_i_el_model1_odonnell_ext, pan_starrs_i_el_model3_odonnell_ext, pan_starrs_z_el_model1_odonnell_ext, pan_starrs_z_el_model3_odonnell_ext, sdss_r01_el_model1_odonnell_ext, sdss_r01_el_model3_odonnell_ext, subaru_hsc_z_el_model1_odonnell_ext, subaru_hsc_z_el_model3_odonnell_ext, dominant_shape, bulge_angle, disk_angle, median_major_axis, scale_length, bulge_fraction, disk_scalelength, disk_nsersic, disk_r50, bulge_r50, bulge_nsersic, disk_ellipticity, bulge_ellipticity, disk_axis_ratio, bulge_axis_ratio, inclination_angle, halo_x, halo_y, halo_z, halo_vx, halo_vy, halo_vz, halo_r, true_redshift_halo, halo_lm, halo_n_sats, n_gals, luminosity_r01_evolved, hpix_13_nest, step
FROM jcarrete.flagship_mock_obsid_1717_s_v3_pq;

ANALYZE TABLE jcarrete.flagship_mock_obsid_1717_s_v3 COMPUTE STATISTICS FOR COLUMNS;

Ahora puedo crear los FITS files:

CREATE TABLE jcarrete.flagship_mock_obsid_1717_s_v3_magnified_fits(
`source_id` bigint COMMENT 'from deserializer',
`halo_id` bigint COMMENT 'from deserializer',
`ra` float COMMENT 'from deserializer',
`dec` float COMMENT 'from deserializer',
`ra_mag` float COMMENT 'from deserializer',
`dec_mag` float COMMENT 'from deserializer',
`z_obs` float COMMENT 'from deserializer',
`ref_mag_abs` float COMMENT 'from deserializer',
`ref_mag` float COMMENT 'from deserializer',
`bulge_fraction` float COMMENT 'from deserializer',
`bulge_r50` float COMMENT 'from deserializer',
`disk_r50` float COMMENT 'from deserializer',
`bulge_nsersic` float COMMENT 'from deserializer',
`inclination_angle` float COMMENT 'from deserializer',
`disk_angle` float COMMENT 'from deserializer',
`kappa` float COMMENT 'from deserializer',
`gamma1` float COMMENT 'from deserializer',
`gamma2` float COMMENT 'from deserializer',
`sed_template` float COMMENT 'from deserializer',
`ext_law` smallint COMMENT 'from deserializer',
`ebv` float COMMENT 'from deserializer',
`halpha_logflam_ext` float COMMENT 'from deserializer',
`hbeta_logflam_ext` float COMMENT 'from deserializer',
`o2_logflam_ext` float COMMENT 'from deserializer',
`o3_logflam_ext` float COMMENT 'from deserializer',
`n2_logflam_ext` float COMMENT 'from deserializer',
`s2_logflam_ext` float COMMENT 'from deserializer',
`av` float COMMENT 'from deserializer',
`tu_fnu_vis` float COMMENT 'from deserializer',
`tu_fnu_y_nisp` float COMMENT 'from deserializer',
`tu_fnu_j_nisp` float COMMENT 'from deserializer',
`tu_fnu_h_nisp` float COMMENT 'from deserializer',
`tu_fnu_g_decam` float COMMENT 'from deserializer',
`tu_fnu_r_decam` float COMMENT 'from deserializer',
`tu_fnu_i_decam` float COMMENT 'from deserializer',
`tu_fnu_z_decam` float COMMENT 'from deserializer',
`tu_fnu_u_megacam` float COMMENT 'from deserializer',
`tu_fnu_r_megacam` float COMMENT 'from deserializer',
`tu_fnu_g_jpcam` float COMMENT 'from deserializer',
`tu_fnu_i_panstarrs` float COMMENT 'from deserializer',
`tu_fnu_z_panstarrs` float COMMENT 'from deserializer',
`tu_fnu_z_hsc` float COMMENT 'from deserializer',
`tu_fnu_g_gaia` float COMMENT 'from deserializer',
`tu_fnu_bp_gaia` float COMMENT 'from deserializer',
`tu_fnu_rp_gaia` float COMMENT 'from deserializer',
`tu_fnu_u_lsst` float COMMENT 'from deserializer',
`tu_fnu_g_lsst` float COMMENT 'from deserializer',
`tu_fnu_r_lsst` float COMMENT 'from deserializer',
`tu_fnu_i_lsst` float COMMENT 'from deserializer',
`tu_fnu_z_lsst` float COMMENT 'from deserializer',
`tu_fnu_y_lsst` float COMMENT 'from deserializer',
`tu_fnu_u_kids` float COMMENT 'from deserializer',
`tu_fnu_g_kids` float COMMENT 'from deserializer',
`tu_fnu_r_kids` float COMMENT 'from deserializer',
`tu_fnu_i_kids` float COMMENT 'from deserializer',
`tu_fnu_j_2mass` float COMMENT 'from deserializer',
`tu_fnu_h_2mass` float COMMENT 'from deserializer',
`tu_fnu_ks_2mass` float COMMENT 'from deserializer')
PARTITIONED BY (
`hpix_5_nest` bigint)
ROW FORMAT SERDE
'es.pic.astro.hadoop.serde.RecArraySerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'es.pic.astro.hadoop.io.BinaryOutputFormat'
;

INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_1717_s_v3_magnified_fits PARTITION(hpix_5_nest)
SELECT CAST(((halo_id * 10000) + galaxy_id) AS bigint) AS SOURCE_ID,
CAST(halo_id AS bigint) AS HALO_ID,
CAST(ra_gal AS float) AS RA,
CAST(dec_gal AS float) AS `DEC`,
CAST(ra_mag_gal AS float) AS RA_MAG,
CAST(dec_mag_gal AS float) AS DEC_MAG,
CAST(observed_redshift_gal AS float) AS Z_OBS,
CAST(abs_mag_r01_evolved AS float) AS REF_MAG_ABS,
CAST(-2.5*log10(sdss_r01) - 48.6 AS float) AS REF_MAG,
CAST(bulge_fraction AS float) AS BULGE_FRACTION,
CAST(bulge_r50 AS float) AS BULGE_R50,
CAST(disk_r50 AS float) AS DISK_R50,
CAST(bulge_nsersic AS float) AS BULGE_NSERSIC,
CAST(inclination_angle AS float) AS INCLINATION_ANGLE,
CAST(disk_angle AS float) AS DISK_ANGLE,
CAST(kappa AS float) AS KAPPA,
CAST(gamma1 AS float) AS GAMMA1,
CAST(gamma2 AS float) AS GAMMA2,
CAST(sed_cosmos AS float) AS SED_TEMPLATE,
CAST(ROUND(ext_curve_cosmos) AS smallint) AS EXT_LAW,
CAST(ebv_cosmos AS float) AS EBV,
CAST(logf_halpha_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HALPHA_LOGFLAM_EXT,
CAST(logf_hbeta_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HBETA_LOGFLAM_EXT,
CAST(logf_o2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O2_LOGFLAM_EXT,
CAST(logf_o3_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O3_LOGFLAM_EXT,
CAST(logf_n2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS N2_LOGFLAM_EXT,
CAST(logf_s2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS S2_LOGFLAM_EXT,
CAST(mw_extinction AS float) AS AV,
CAST(euclid_vis_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_VIS,
CAST(euclid_nisp_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_NISP,
CAST(euclid_nisp_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_NISP,
CAST(euclid_nisp_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_NISP,
CAST(blanco_decam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_DECAM,
CAST(blanco_decam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_DECAM,
CAST(blanco_decam_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_DECAM,
CAST(blanco_decam_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_DECAM,
CAST(cfht_megacam_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_MEGACAM,
CAST(cfht_megacam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_MEGACAM,
CAST(jst_jpcam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_JPCAM,
CAST(pan_starrs_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_PANSTARRS,
CAST(pan_starrs_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_PANSTARRS,
CAST(subaru_hsc_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_HSC,
CAST(gaia_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_GAIA,
CAST(gaia_bp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_BP_GAIA,
CAST(gaia_rp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_RP_GAIA,
CAST(lsst_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_LSST,
CAST(lsst_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_LSST,
CAST(lsst_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_LSST,
CAST(lsst_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_LSST,
CAST(lsst_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_LSST,
CAST(lsst_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_LSST,
CAST(kids_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_KIDS,
CAST(kids_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_KIDS,
CAST(kids_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_KIDS,
CAST(kids_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_KIDS,
CAST(2mass_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_2MASS,
CAST(2mass_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_2MASS,
CAST(2mass_ks_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_KS_2MASS,
CAST(SHIFTRIGHT(hpix_29_nest, (29-5)*2) AS bigint) AS hpix_5_nest
FROM jcarrete.flagship_mock_obsid_1717_s_v3
WHERE (logf_halpha_model3_ext > -16 OR -2.5*log10(euclid_nisp_h) - 48.6 < 26)
AND disk_axis_ratio > 0.10865
;

----

Añadimos los header con el notebook: add_fits_header_to_hive_table.py

En este caso vamos a cambiar la nomenclatura de las versiones en el webdavdoor.

El catálogo del que partimos es el 1.9.8. Se trata de un optante.
Pero las shapes están mal. Las corrijo, así que la versión sería 1.9.9

Llamo al directorio: flagship_mock_1_9_9_obsid_1717_v3

Paso la tabla ORC al esquema cosmohub: OJO, le meto la versión 1_9_9!!!!!!

ALTER TABLE jcarrete.flagship_mock_obsid_1717_s_v3 RENAME TO cosmohub.flagship_mock_1_9_9_obsid_1717_s_v3;

ANALYZE TABLE cosmohub.flagship_mock_1_9_9_obsid_1717_s_v3 COMPUTE STATISTICS FOR COLUMNS;

----

Description in CosmoHub

Dark matter halo input catalog from flagship_rockstar_octant1_c, which is the Flagship dark matter halo catalog v1.1 with correct velocities.

The covered area of the SWF2 observation (obsid = 1717) is:

215.46 < ra_gal < 241.33

73.59 < dec_gal < 79.5904

**Shape distributions are completely different from all previous versions (including new recipe for inclination_angle field)**.

In this version a new python method for inclination_angle was developed by Francisco and ellipticities and axis ratios come from Eric's formulas.

See the following [url](https://docs.google.com/document/d/1FRERcl5Lv2IDB6ZChTK6gcEHI2FWA9gSgYKcaDUA9EM/edit?usp=sharing) for more details.

Note that **we provide fluxes instead of magnitudes**. In order **to get magnitudes** you can enter in the "Expert mode" and estimate the magnitude as follows:

m = -2.5 * log10(flux) - 48.6

And in order to estimate magnified magnitudes:

m_mag = m + 2.5 * log10((1-kappa)*(1-kappa) - gamma1 * gamma1 - gamma2 * gamma2)

----

{
"sql": "SELECT CAST(((halo_id * 10000) + galaxy_id) AS bigint) AS SOURCE_ID, \nCAST(halo_id AS bigint) AS HALO_ID, \nCAST(ra_gal AS float) AS RA, \nCAST(dec_gal AS float) AS `DEC`, \nCAST(ra_mag_gal AS float) AS RA_MAG, \nCAST(dec_mag_gal AS float) AS DEC_MAG, \nCAST(observed_redshift_gal AS float) AS Z_OBS, \nCAST(abs_mag_r01_evolved AS float) AS REF_MAG_ABS, \nCAST(-2.5*log10(sdss_r01) - 48.6 AS float) AS REF_MAG, \nCAST(bulge_fraction AS float) AS BULGE_FRACTION, \nCAST(bulge_r50 AS float) AS BULGE_R50, \nCAST(disk_r50 AS float) AS DISK_R50, \nCAST(bulge_nsersic AS float) AS BULGE_NSERSIC, \nCAST(inclination_angle AS float) AS INCLINATION_ANGLE, \nCAST(disk_angle AS float) AS DISK_ANGLE, \nCAST(kappa AS float) AS KAPPA, \nCAST(gamma1 AS float) AS GAMMA1, \nCAST(gamma2 AS float) AS GAMMA2, \nCAST(sed_cosmos AS float) AS SED_TEMPLATE, \nCAST(ROUND(ext_curve_cosmos) AS smallint) AS EXT_LAW, \nCAST(ebv_cosmos AS float) AS EBV, \nCAST(logf_halpha_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HALPHA_LOGFLAM_EXT_MAG, \nCAST(logf_hbeta_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HBETA_LOGFLAM_EXT_MAG, \nCAST(logf_o2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O2_LOGFLAM_EXT_MAG, \nCAST(logf_o3_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O3_LOGFLAM_EXT_MAG, \nCAST(logf_n2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS N2_LOGFLAM_EXT_MAG, \nCAST(logf_s2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS S2_LOGFLAM_EXT_MAG, \nCAST(mw_extinction AS float) AS AV,\nCAST(euclid_vis_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_VIS_MAG, \nCAST(euclid_nisp_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_NISP_MAG, \nCAST(euclid_nisp_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_NISP_MAG, \nCAST(euclid_nisp_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_NISP_MAG, \nCAST(blanco_decam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_DECAM_MAG, \nCAST(blanco_decam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_DECAM_MAG, \nCAST(blanco_decam_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_DECAM_MAG, \nCAST(blanco_decam_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_DECAM_MAG, \nCAST(cfht_megacam_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_MEGACAM_MAG, \nCAST(cfht_megacam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_MEGACAM_MAG, \nCAST(jst_jpcam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_JPCAM_MAG, \nCAST(pan_starrs_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_PANSTARRS_MAG, \nCAST(pan_starrs_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_PANSTARRS_MAG, \nCAST(subaru_hsc_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_HSC_MAG, \nCAST(gaia_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_GAIA_MAG, \nCAST(gaia_bp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_BP_GAIA_MAG, \nCAST(gaia_rp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_RP_GAIA_MAG, \nCAST(lsst_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_LSST_MAG, \nCAST(lsst_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_LSST_MAG, \nCAST(lsst_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_LSST_MAG, \nCAST(lsst_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_LSST_MAG, \nCAST(lsst_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_LSST_MAG, \nCAST(lsst_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_LSST_MAG, \nCAST(kids_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_KIDS_MAG, \nCAST(kids_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_KIDS_MAG, \nCAST(kids_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_KIDS_MAG, \nCAST(kids_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_KIDS_MAG, \nCAST(2mass_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_2MASS_MAG, \nCAST(2mass_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_2MASS_MAG, \nCAST(2mass_ks_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_KS_2MASS_MAG \nFROM cosmohub.flagship_mock_1_9_9_obsid_1717_s_v3 \nWHERE (logf_halpha_model3_ext > -16 OR -2.5*log10(euclid_nisp_h) - 48.6 < 26) \nAND disk_axis_ratio > 0.10865 \nAND SHIFTRIGHT(hpix_29_nest, (29-5)*2) = 1008"
}

----

'''Generating Deep field with lensing parameters from 1.9.8!!!'''

CREATE TABLE jcarrete.flagship_mock_gsir_nir_self_calibration_c2(
`kind` BIGINT,
`luminosity_r01` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`abs_mag_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`luminosity_r01_evolved` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` INT,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`hpix_29_nest` BIGINT,
`hpix_13_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`observed_redshift_gal` DOUBLE,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha_ext` DOUBLE,
`logf_halpha` DOUBLE,
`Halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_Halpha_model3_ext` DOUBLE,
`loglum_Halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`blanco_decam_i` DOUBLE,
`sdss_r01` DOUBLE,
`euclid_nisp_h` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`dominant_shape` BIGINT,
`bulge_angle` DOUBLE,
`disk_angle` DOUBLE,
`median_major_axis` DOUBLE,
`scale_length` DOUBLE,
`bulge_fraction` DOUBLE,
`disk_scalelength` DOUBLE,
`disk_nsersic` DOUBLE,
`disk_r50` DOUBLE,
`bulge_r50` DOUBLE,
`bulge_nsersic` DOUBLE,
`inclination_angle` DOUBLE,
`disk_axis_ratio` DOUBLE,
`bulge_axis_ratio` DOUBLE,
`disk_ellipticity` DOUBLE,
`bulge_ellipticity` DOUBLE,
`random_index` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
PARTITIONED BY (
`step` smallint
)
CLUSTERED BY (
hpix_13_nest
)
SORTED BY (
hpix_13_nest ASC
)
INTO 128 BUCKETS
STORED AS ORC
;

INSERT OVERWRITE TABLE jcarrete.flagship_mock_gsir_nir_self_calibration_c2
SELECT
`kind`, `luminosity_r01`, `halo_x`, `halo_y`, `halo_z`, `halo_vx`, `halo_vy`, `halo_vz`, `halo_r`, `true_redshift_halo`, `halo_lm`, `halo_n_sats`, `n_gals`, `abs_mag_r01`, `abs_mag_r01_evolved`, `luminosity_r01_evolved`, `gr_restframe`, `color_kind`, `x_gal`, `y_gal`, `z_gal`, `r_gal`, `true_redshift_gal`, `ra_gal`, `dec_gal`, `hpix_nest`, SHIFTRIGHT(`hpix_nest`, (29-13)*2), `vx_gal`, `vy_gal`, `vz_gal`, `vrad_gal`, `delta_r`, `observed_redshift_gal`, `sed_ke`, `gr_cosmos`, `sed_cosmos`, `ext_curve_cosmos`, `ebv_cosmos`, `is_within_cosmos`, `cosmos_distance`, `abs_mag_uv_dereddened`, `log_ml_r01`, `log_stellar_mass`, `metallicity`, `log_sfr`, `logf_halpha_ext`, `logf_halpha`, `Halpha_scatter`, `loglum_halpha`, `loglum_halpha_ext`, `logf_dummy`, `z_dummy`, `logf_halpha_model1_ext`, `logf_halpha_model1`, `loglum_halpha_model1_ext`, `loglum_halpha_model1`, `logf_halpha_model3_ext`, `logf_halpha_model3`, `loglum_Halpha_model3_ext`, `loglum_Halpha_model3`, `logf_hbeta_model1_ext`, `logf_hbeta_model1`, `logf_o2_model1_ext`, `logf_o2_model1`, `logf_n2_model1_ext`, `logf_n2_model1`, `logf_o3_model1_ext`, `logf_o3_model1`, `logf_s2_model1_ext`, `logf_s2_model1`, `logf_hbeta_model3_ext`, `logf_hbeta_model3`, `logf_o2_model3_ext`, `logf_o2_model3`, `logf_n2_model3_ext`, `logf_n2_model3`, `logf_o3_model3_ext`, `logf_o3_model3`, `logf_s2_model3_ext`, `logf_s2_model3`, `blanco_decam_i`, `sdss_r01`, `euclid_nisp_h`, `l_gal`, `b_gal`, `mw_extinction`, `blanco_decam_i_odonnell_ext`, `euclid_nisp_h_odonnell_ext`, `sdss_r01_odonnell_ext`, `blanco_decam_i_el_model1_odonnell_ext`, `blanco_decam_i_el_model3_odonnell_ext`, `sdss_r01_el_model1_odonnell_ext`, `sdss_r01_el_model3_odonnell_ext`, `euclid_nisp_h_el_model1_odonnell_ext`, `euclid_nisp_h_el_model3_odonnell_ext`, `dominant_shape`, `bulge_angle`, `disk_angle`, `median_major_axis`, `scale_length`, `bulge_fraction`, `disk_scalelength`, `disk_nsersic`, `disk_r50`, `bulge_r50`, `bulge_nsersic`, `inclination_angle`, `disk_axis_ratio`, `bulge_axis_ratio`, `disk_ellipticity`, `bulge_ellipticity`, `random_index`, `halo_id`, `galaxy_id`, `step`
FROM jcarrete.flagship_mock_1_9_8_c
WHERE
(ra_gal >= 7.678307967139321
AND ra_gal < 14.008504032860678
AND dec_gal >= 15.587506
AND dec_gal < 21.587506)
OR
(ra_gal >= 35.45815488783862
AND ra_gal < 61.33016111216139
AND dec_gal >=73.5904
AND dec_gal < 79.5904)
;

ANALYZE TABLE jcarrete.flagship_mock_gsir_nir_self_calibration_c2 COMPUTE STATISTICS FOR COLUMNS;

jcarrete.flagship_mock_gsir_nir_self_calibration_c2

jcarrete.flagship_mock_gsir_nir_self_calibration_s

----

Notebooks with kappa string:

* Checking_lensing_procedure.ipynb
* assign_lensing_Flagship_1_8_3.ipynb
* assign_lensing.ipynb
* assign_lensing_pipeline_integration.ipynb
* Checking_fluxes_EL_MW.ipynb
* Checking_true_flux_SciPIC_vs_Euclid.ipynb
* DES_Y3_MICE2_source_catalog_IA_mocks.ipynb
* Euclid_bug_10759_11158_10141.ipynb
* Ingest_lensing_maps_into_Big_Data_platform.ipynb
* Lensing_test_new.ipynb
* MICECATv2.0_tests_for_SC2.ipynb
* micecat_v2_0_thumbnails_catalog.ipynb
* pruebas.ipynb
* shape_pipeline.ipynb
* thumbnail_euclid_problem.ipynb
* Thumbnail_library_test.ipynb

https://math.stackexchange.com/questions/2484830/how-to-switch-y-and-z-axis-of-a-rotation-matrix