GSIR
Jump to navigation
Jump to search
This is a copy of the notes I took to generate GSIR
It is a long story...
This is the summary of the generation of the TU FITS files needed for GSIR:
We start by generating a mock galaxy catalog of the full first octant (v1.9.8) using the full pipeline (full_pipeline.py) but only including 3 filters in the flux computation:
- euclid_nisp_h for making the magnitude limit cut and therefore make the catalog lighter.
We make the cut in this way:
df = df[
(df['euclid_nisp_h'] > flux_cut)
| (df['logf_halpha_model1_ext'] > -16)
| (df['logf_halpha_model3_ext'] > -16)
]
where flux_cut = mag_to_flux(26) So we cut all galaxies fainter than 26 in the euclid_nisp_h filter
- blanco_decam_i: needed to estimate morphological parameters
- sdss_r01: for precuation since it is the reference filter. I think we can avoid it
The generated catalog is already in ORC format in a table called: jcarrete.flagship_mock_1_9_8_c
I tried to generate it in a _c2 format with the same partitions and buckets than the lensing table in order to make an efficient JOIN but I made a mistake with the number of buckets.
I put 4096 and given that it is already partitioned by step, each partition is clustered in 4096 buckets. This is a huge amount of files and we had problems with that execution.
So finally we materialized the 2 pointings we had at that time: SWF1 and SWF2:
SWF1 Area pointing obsid: 52929 ra_range = 7.678307967139321 14.008504032860678 dec_range = -21.587506 -15.587506000000001
Area pointing obsid in 1st octant: 52929
ra_range = 7.678307967139321 14.008504032860678
dec_range = 15.587506000000001 21.587506
SWF2 Area pointing obsid: 1717 ra_range = 215.45815488783862 241.3301611121614 dec_range = 73.5904 79.5904
Area pointing obsid in 1st octant: 1717 ra_range = 35.45815488783862 61.33016111216139 dec_range = 73.5904 79.5904
I created a table with those two regions with the correct format to make the JOIN efficiently in the test platform!
CREATE TABLE jcarrete.flagship_mock_obsid_52929_1717_c2(
...
)
PARTITIONED BY (
`step` smallint
)
CLUSTERED BY (
hpix_13_nest
)
SORTED BY (
hpix_13_nest ASC
)
INTO 128 BUCKETS
STORED AS ORC
;
Then Pau add the lensing in the Test platform.
We finally have a table like this:
CREATE TABLE jcarrete.flagship_mock_obsid_52929_1717_s(
)
CLUSTERED BY (
hpix_13_nest)
SORTED BY (
hpix_13_nest ASC)
INTO 128 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;
Adding FLUXES
Once the lensing is added I run the flux_pipeline.ipynb, in which we have updated the euclid filters. ONLY THE euclid ones at this time!
I do it for the 1717 observation (at this time I do not know what to do with the area that has to be rotated to the southern hemisphere, obsid= 52929).
/software/astro/scripts/create_table_from_parquet.sh -p /hdfs/user/jcarrete/data/euclid/flagship_mock_obsid_1717_s/0.pq -t jcarrete.flagship_mock_obsid_1717_s_pq -e
CREATE EXTERNAL TABLE jcarrete.flagship_mock_obsid_1717_s_pq ( ... ) STORED AS PARQUET LOCATION '/user/jcarrete/data/euclid/flagship_mock_obsid_1717_s' ;
I create the ORC table. This table contains "everything needed" (but note that shapes have a bug and filters of LSST are not updated yet!)
CREATE TABLE jcarrete.flagship_mock_obsid_1717_s(
...
)
PARTITIONED BY (
`step` smallint COMMENT 'step number in the Flagship lensing maps')
CLUSTERED BY (
hpix_13_nest)
SORTED BY (
hpix_13_nest ASC)
INTO 128 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;
Then I create the FITS table:
CREATE TABLE jcarrete.flagship_mock_obsid_1717_s_magnified_fits( ... ) PARTITIONED BY ( `hpix_5_nest` bigint) ROW FORMAT SERDE 'es.pic.astro.hadoop.serde.RecArraySerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'es.pic.astro.hadoop.io.BinaryOutputFormat' ;
Cuando inserto los datos en esta tabla cambio el signo de gamma1!!!!!
Esto es tras varias conversaciones con Pablo y Francisco acerca de las convenciones de MICE y de Flagship. Sigo sin entender por qué en MICE calculan RA de la manera que lo hacen (swap entre x e y) y luego a la hora de buscar el pixel en los mapas a través de phi y theta, le meten un shift en phi de valor pi. En cualquier caso, a la conclusión a la que llegamos es que para pasar del sistema de referencia local de Healpy al de IAU, hay que modificar gamma1 = -gamma1.
Además, cuando rotamos un area con DEC > 0 a una con DEC < 0, hay que volver a cambiarle el signo a gamma1! Esto será "útil" para el obsid = 52929
INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_1717_s_magnified_fits PARTITION(hpix_5_nest) ... CAST(-gamma1 as FLOAT) AS GAMMA1 )
Como se ve, los cambios en gamma1 se hacen a nivel de FITS files. Es decir, las tablas de las que parto siguen estando MAL. Esto lo tendría que cambiar de cara a poner online el catálogo en CosmoHub! Tendrían que ser iguales. Todo excepto, y no lo voy a hacer, los valores de x, y, z. La traslación la hago en RA,DEC
Es en este momento en el que alguien se da cuenta de que las shapes están mal!
Corrijo el problema que estaba en las unidades que le pasaba a la función que calculaba las elipticidades (de grados a radianes)
Y creo otra tabla usando el shape_pipeline.py: jcarrete.flagship_mock_obsid_1717_s_v2_pq
La paso a ORC: jcarrete.flagship_mock_obsid_1717_s_v2
Hay que volver a correr los flujos para la tabla jcarrete.flagship_mock_obsid_1717_s_v2 ya que nos han actualizado los del LSST. Ver más abajo los detalles:
Tenemos que:
- Corregir el fjc_miller.py. Pau tendría que hacerlo transparente y que el nombre de la clase no cambiara.
- En el notebook full_pipeline.py he metido los cambios del shape_pipeline.py y del flux_pipeline.py.
OJO CON LOS BUCKETS PORQUE HE PUESTO 4096 EN UNA TABLA PARTICIONADA Y ESO ES MUCHIIIIIISSSIIIMO!!!!
/software/astro/scripts/create_table_from_parquet.sh -p /hdfs/user/jcarrete/data/euclid/flagship_mock_1_9_8/0.pq -t jcarrete.flagship_mock_1_9_8_pq -e
CREATE EXTERNAL TABLE jcarrete.flagship_mock_1_9_8_pq (
`kind` BIGINT,
`luminosity_r01` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`abs_mag_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`luminosity_r01_evolved` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` INT,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`hpix_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`observed_redshift_gal` DOUBLE,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha_ext` DOUBLE,
`logf_halpha` DOUBLE,
`Halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_Halpha_model3_ext` DOUBLE,
`loglum_Halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`blanco_decam_i` DOUBLE,
`sdss_r01` DOUBLE,
`euclid_nisp_h` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`dominant_shape` BIGINT,
`bulge_angle` DOUBLE,
`disk_angle` DOUBLE,
`median_major_axis` DOUBLE,
`scale_length` DOUBLE,
`bulge_fraction` DOUBLE,
`disk_scalelength` DOUBLE,
`disk_nsersic` DOUBLE,
`disk_r50` DOUBLE,
`bulge_r50` DOUBLE,
`bulge_nsersic` DOUBLE,
`inclination_angle` DOUBLE,
`disk_axis_ratio` DOUBLE,
`bulge_axis_ratio` DOUBLE,
`disk_ellipticity` DOUBLE,
`bulge_ellipticity` DOUBLE,
`step` BIGINT,
`random_index` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
STORED AS PARQUET
LOCATION '/user/jcarrete/data/euclid/flagship_mock_1_9_8'
;
PREGUNTAR EL NUMERO DE BUCKETS! Si no recuerdo mal era el volumen de los datos dividido entre 256MB y "pasarme" hasta el número en base 2 siguiente.
SON 3 TB de catálogo, pongo 4096 buckets
CREATE TABLE jcarrete.flagship_mock_1_9_8_c (
`kind` BIGINT,
`luminosity_r01` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`abs_mag_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`luminosity_r01_evolved` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` INT,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`hpix_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`observed_redshift_gal` DOUBLE,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha_ext` DOUBLE,
`logf_halpha` DOUBLE,
`Halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_Halpha_model3_ext` DOUBLE,
`loglum_Halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`blanco_decam_i` DOUBLE,
`sdss_r01` DOUBLE,
`euclid_nisp_h` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`dominant_shape` BIGINT,
`bulge_angle` DOUBLE,
`disk_angle` DOUBLE,
`median_major_axis` DOUBLE,
`scale_length` DOUBLE,
`bulge_fraction` DOUBLE,
`disk_scalelength` DOUBLE,
`disk_nsersic` DOUBLE,
`disk_r50` DOUBLE,
`bulge_r50` DOUBLE,
`bulge_nsersic` DOUBLE,
`inclination_angle` DOUBLE,
`disk_axis_ratio` DOUBLE,
`bulge_axis_ratio` DOUBLE,
`disk_ellipticity` DOUBLE,
`bulge_ellipticity` DOUBLE,
`step` BIGINT,
`random_index` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
CLUSTERED BY (
halo_id,
galaxy_id
)
INTO 4096 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;
INSERT OVERWRITE TABLE jcarrete.flagship_mock_1_9_8_c SELECT * FROM jcarrete.flagship_mock_1_9_8_pq;
ANALYZE TABLE jcarrete.flagship_mock_1_9_8_c COMPUTE STATISTICS FOR COLUMNS;
PREGUNTAR DE NUEVO POR EL NUMERO DE BUCKETS
EL NUMERO DE BUCKETS ESTÁ MAL, ya que estoy dividiendo cada PARTITION EN 4096 buckets. ES UNA BARBARIDAD!
ESTA TABLA FALLA:
CREATE TABLE jcarrete.flagship_mock_1_9_8_c2(
`kind` BIGINT,
`luminosity_r01` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`abs_mag_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`luminosity_r01_evolved` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` INT,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`hpix_29_nest` BIGINT,
`hpix_13_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`observed_redshift_gal` DOUBLE,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha_ext` DOUBLE,
`logf_halpha` DOUBLE,
`Halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_Halpha_model3_ext` DOUBLE,
`loglum_Halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`blanco_decam_i` DOUBLE,
`sdss_r01` DOUBLE,
`euclid_nisp_h` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`dominant_shape` BIGINT,
`bulge_angle` DOUBLE,
`disk_angle` DOUBLE,
`median_major_axis` DOUBLE,
`scale_length` DOUBLE,
`bulge_fraction` DOUBLE,
`disk_scalelength` DOUBLE,
`disk_nsersic` DOUBLE,
`disk_r50` DOUBLE,
`bulge_r50` DOUBLE,
`bulge_nsersic` DOUBLE,
`inclination_angle` DOUBLE,
`disk_axis_ratio` DOUBLE,
`bulge_axis_ratio` DOUBLE,
`disk_ellipticity` DOUBLE,
`bulge_ellipticity` DOUBLE,
`random_index` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
PARTITIONED BY (
`step` smallint
)
CLUSTERED BY (
hpix_13_nest
)
SORTED BY (
hpix_13_nest ASC
)
INTO 4096 BUCKETS
STORED AS ORC
;
ESTA HA PETADO!
INSERT OVERWRITE TABLE jcarrete.flagship_mock_1_9_8_c2 SELECT `kind`, `luminosity_r01`, `halo_x`, `halo_y`, `halo_z`, `halo_vx`, `halo_vy`, `halo_vz`, `halo_r`, `true_redshift_halo`, `halo_lm`, `halo_n_sats`, `n_gals`, `abs_mag_r01`, `abs_mag_r01_evolved`, `luminosity_r01_evolved`, `gr_restframe`, `color_kind`, `x_gal`, `y_gal`, `z_gal`, `r_gal`, `true_redshift_gal`, `ra_gal`, `dec_gal`, `hpix_nest`, SHIFTRIGHT(`hpix_nest`, (29-13)*2), `vx_gal`, `vy_gal`, `vz_gal`, `vrad_gal`, `delta_r`, `observed_redshift_gal`, `sed_ke`, `gr_cosmos`, `sed_cosmos`, `ext_curve_cosmos`, `ebv_cosmos`, `is_within_cosmos`, `cosmos_distance`, `abs_mag_uv_dereddened`, `log_ml_r01`, `log_stellar_mass`, `metallicity`, `log_sfr`, `logf_halpha_ext`, `logf_halpha`, `Halpha_scatter`, `loglum_halpha`, `loglum_halpha_ext`, `logf_dummy`, `z_dummy`, `logf_halpha_model1_ext`, `logf_halpha_model1`, `loglum_halpha_model1_ext`, `loglum_halpha_model1`, `logf_halpha_model3_ext`, `logf_halpha_model3`, `loglum_Halpha_model3_ext`, `loglum_Halpha_model3`, `logf_hbeta_model1_ext`, `logf_hbeta_model1`, `logf_o2_model1_ext`, `logf_o2_model1`, `logf_n2_model1_ext`, `logf_n2_model1`, `logf_o3_model1_ext`, `logf_o3_model1`, `logf_s2_model1_ext`, `logf_s2_model1`, `logf_hbeta_model3_ext`, `logf_hbeta_model3`, `logf_o2_model3_ext`, `logf_o2_model3`, `logf_n2_model3_ext`, `logf_n2_model3`, `logf_o3_model3_ext`, `logf_o3_model3`, `logf_s2_model3_ext`, `logf_s2_model3`, `blanco_decam_i`, `sdss_r01`, `euclid_nisp_h`, `l_gal`, `b_gal`, `mw_extinction`, `blanco_decam_i_odonnell_ext`, `euclid_nisp_h_odonnell_ext`, `sdss_r01_odonnell_ext`, `blanco_decam_i_el_model1_odonnell_ext`, `blanco_decam_i_el_model3_odonnell_ext`, `sdss_r01_el_model1_odonnell_ext`, `sdss_r01_el_model3_odonnell_ext`, `euclid_nisp_h_el_model1_odonnell_ext`, `euclid_nisp_h_el_model3_odonnell_ext`, `dominant_shape`, `bulge_angle`, `disk_angle`, `median_major_axis`, `scale_length`, `bulge_fraction`, `disk_scalelength`, `disk_nsersic`, `disk_r50`, `bulge_r50`, `bulge_nsersic`, `inclination_angle`, `disk_axis_ratio`, `bulge_axis_ratio`, `disk_ellipticity`, `bulge_ellipticity`, `random_index`, `halo_id`, `galaxy_id`, `step` FROM jcarrete.flagship_mock_1_9_8_c;
ADDING lensing parameters
FIJARME E LOS COMENTARIOS QUE PONGO!!!!! ESTA NO ES LA DEFINICIÓN FINAL! MÁS ABAJO LA TENGO ESCRITA PARA CORRERLA
CREATE TABLE jcarrete.flagship_mock_1_9_8_s( `halo_id` bigint COMMENT 'unique halo id coming from the Flagship dark matter halo catalog', `galaxy_id` int COMMENT 'combined with the unique_halo_id they uniquely identify each galaxy', `kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT', `random_index` float COMMENT 'random number [0 - 1) for subsampling', `ra_gal` float COMMENT 'galaxy right ascension (degrees)', `dec_gal` float COMMENT 'galaxy declination (degrees)',
Y ESTOS SON LOS QUE AÑADIMOS
`ra_mag_gal` float COMMENT 'galaxy magnified right ascension (degree)', `dec_mag_gal` float COMMENT 'galaxy magnified declination (degree)', `kappa` float COMMENT 'convergence', `gamma1` float COMMENT 'shear', `gamma2` float COMMENT 'shear',
`l_gal` float COMMENT 'galactic longitude (degrees)', `b_gal` float COMMENT 'galactic latitude (degrees)', `mw_extinction` float COMMENT 'EBV from file: HFI_CompMap_ThermalDustModel_2048_R1.20.fits', `x_gal` float COMMENT 'galaxy comoving x coordinate (Mpc/h)', `y_gal` float COMMENT 'galaxy comoving y coordinate (Mpc/h)', `z_gal` float COMMENT 'galaxy comoving z coordinate (Mpc/h)', `r_gal` float COMMENT 'galaxy comoving distance (Mpc/h)', `true_redshift_gal` float COMMENT 'true galaxy redshift', `observed_redshift_gal` float COMMENT 'observed galaxy redshift (including peculiar velocity)', `hpix_29_nest` bigint COMMENT 'healpix pixel id using order = 29 in NESTED format', EN EL CATALOGO SE LLAMA hpix_nest BIGINT, `vx_gal` float COMMENT 'physical (peculiar) velocity in x-component (km/s)', `vy_gal` float COMMENT 'physical (peculiar) velocity in y-component (km/s)', `vz_gal` float COMMENT 'physical (peculiar) velocity in z-component (km/s)', `vrad_gal` float COMMENT 'missing comment', `delta_r` float COMMENT 'missing comment', `abs_mag_r01` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h)', `luminosity_r01` float COMMENT 'luminosity in the 0.1-r-band', `abs_mag_r01_evolved` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h) including MICECATv2.0 magnitude evolution', `sdss_r01` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1', `abs_mag_uv_dereddened` float COMMENT 'absolute magnitude ultraviolet', `gr_restframe` float COMMENT '(g-r) rest-frame at z=0', `color_kind` tinyint COMMENT 'galaxy color kind: 0=red sequence 1=green valley 2=blue cloud', `sed_ke` float COMMENT 'K-correction for each SED', `gr_cosmos` float COMMENT 'interpolated COSMOS g-r color', `sed_cosmos` float COMMENT 'interpolated basic COSMOS SED [0 30]', `is_within_cosmos` boolean COMMENT 'missing comment', `cosmos_distance` float COMMENT 'missing comment', `ext_curve_cosmos` tinyint COMMENT 'COSMOS extinction curve [0-4]', `ebv_cosmos` float COMMENT 'interpolated extinction strength for COSMOS extinction curve', `log_ml_r01` float COMMENT 'logarithm of stellar_mass/luminosity ratio in the r01 SDSS band', `log_stellar_mass` float COMMENT 'logarithm of the stellar mass (Msun/h)', `metallicity` float COMMENT 'oxygen metallicity in units 12+logO/H', `log_sfr` float COMMENT 'logarithm of star formation rate in (Msun/h)/year', `logf_halpha` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: DOES NOT include extinction)', `logf_halpha_ext` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: includes extinction)', `halpha_scatter` float COMMENT 'missing comment', `loglum_halpha` float COMMENT 'DEPRECATED logarithm of Halpha luminosity (erg/s/h^2) using empirical relation with UV magnitude (DOES NOT include extinction)', `loglum_halpha_ext` float COMMENT 'missing comment', `logf_dummy` float COMMENT 'missing comment', `z_dummy` float COMMENT 'missing comment', `logf_halpha_model1_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: includes extinction)', `logf_halpha_model1` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: DOES NOT include extinction)', `loglum_halpha_model1_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (includes extinction)', `loglum_halpha_model1` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (DOES NOT include extinction)', `logf_halpha_model3_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: includes extinction)', `logf_halpha_model3` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: DOES NOT include extinction)', `loglum_halpha_model3_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (includes extinction)', `loglum_halpha_model3` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (DOES NOT include extinction)', `logf_hbeta_model1_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: includes extinction)', `logf_hbeta_model1` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)', `logf_o2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)', `logf_o2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)', `logf_n2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)', `logf_n2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)', `logf_o3_model1_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: includes extinction)', `logf_o3_model1` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)', `logf_s2_model1_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: includes extinction)', `logf_s2_model1` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)', `logf_hbeta_model3_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: includes extinction)', `logf_hbeta_model3` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)', `logf_o2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)', `logf_o2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)', `logf_n2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)', `logf_n2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)', `logf_o3_model3_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: includes extinction)', `logf_o3_model3` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)', `logf_s2_model3_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: includes extinction)', `logf_s2_model3` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)', `blanco_decam_i` float COMMENT 'observed flux in DES i-band (Euclid provided)', `euclid_nisp_h` float COMMENT 'observed flux in Euclid NISP-H band (Euclid provided)', `blanco_decam_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES i-band (Euclid provided)', `euclid_nisp_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-H band (Euclid provided)', `blanco_decam_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES i-band (Euclid provided)', `blanco_decam_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES i-band (Euclid provided)', `euclid_nisp_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-H band (Euclid provided)', `euclid_nisp_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-H band (Euclid provided)', `sdss_r01_odonnell_ext` float COMMENT 'observed flux with MW extinction in in sdss r-band redshifted to z=0.1', `sdss_r01_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in in sdss r-band redshifted to z=0.1', `sdss_r01_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in in sdss r-band redshifted to z=0.1', `dominant_shape` tinyint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)', `bulge_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', `disk_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', `median_major_axis` float COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)', `scale_length` float COMMENT 'disc and bulge scalelength prior', `bulge_fraction` float COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)', `disk_scalelength` float COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)', `disk_nsersic` float COMMENT 'Sersic index of the disk component', `disk_r50` float COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678', `bulge_r50` float COMMENT 'half light radius of the bulge', `bulge_nsersic` float COMMENT 'Sersic index of the bulge component', `disk_ellipticity` float COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)', `bulge_ellipticity` float COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)', `disk_axis_ratio` float COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', `bulge_axis_ratio` float COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', `inclination_angle` float COMMENT 'Galaxy inclination angle (where 0 degrees = face-on and 90 degrees = edge-on). Galaxy ellipticity for disk and bulge components are computed following the recipe in https://euclid.roe.ac.uk/projects/sgsshear/wiki/SHE-SIM', `halo_x` float COMMENT 'halo comoving x coordinate (Mpc/h)', `halo_y` float COMMENT 'halo comoving y coordinate (Mpc/h)', `halo_z` float COMMENT 'halo comoving z coordinate (Mpc/h)', `halo_vx` float COMMENT 'halo x-centre velocity (km/h)', `halo_vy` float COMMENT 'halo y-centre velocity (km/h)', `halo_vz` float COMMENT 'halo z-centre velocity (km/h)', `halo_r` float COMMENT 'halo comoving distance (Mpc/h)', `true_redshift_halo` float COMMENT 'true redshift of the host halo', `halo_lm` float COMMENT 'log10 of the FoF halo mass in Msum/h. This halo mass is computed as the particule mass multiplied by the number of particles of the FoF halo. At low masses it is interpolated to obtain a smooth distribution of halo masses', `halo_n_sats` int COMMENT 'number of satellite galaxies from HOD', `n_gals` int COMMENT 'DEPRECATED', `luminosity_r01_evolved` float COMMENT 'missing comment', `hpix_13_nest` bigint COMMENT 'missing comment' ESTE NO ESTÁAAAAA EN LA TABLA!) PARTITIONED BY ( `step` smallint COMMENT 'step number in the Flagship lensing maps') CLUSTERED BY ( hpix_13_nest) SORTED BY ( hpix_13_nest ASC) INTO 128 BUCKETS ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' ;
DE NUEVO CALCULAR LOS BUCKETS!
CREATE TABLE jcarrete.flagship_mock_1_9_8_s(
`halo_id` bigint COMMENT 'unique halo id coming from the Flagship dark matter halo catalog', `galaxy_id` int COMMENT 'combined with the unique_halo_id they uniquely identify each galaxy', `kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT', `random_index` float COMMENT 'random number [0 - 1) for subsampling', `ra_gal` float COMMENT 'galaxy right ascension (degrees)', `dec_gal` float COMMENT 'galaxy declination (degrees)', `ra_mag_gal` float COMMENT 'galaxy magnified right ascension (degree)', `dec_mag_gal` float COMMENT 'galaxy magnified declination (degree)', `kappa` float COMMENT 'convergence', `gamma1` float COMMENT 'shear', `gamma2` float COMMENT 'shear', `l_gal` float COMMENT 'galactic longitude (degrees)', `b_gal` float COMMENT 'galactic latitude (degrees)', `mw_extinction` float COMMENT 'EBV from file: HFI_CompMap_ThermalDustModel_2048_R1.20.fits', `x_gal` float COMMENT 'galaxy comoving x coordinate (Mpc/h)', `y_gal` float COMMENT 'galaxy comoving y coordinate (Mpc/h)', `z_gal` float COMMENT 'galaxy comoving z coordinate (Mpc/h)', `r_gal` float COMMENT 'galaxy comoving distance (Mpc/h)', `true_redshift_gal` float COMMENT 'true galaxy redshift', `observed_redshift_gal` float COMMENT 'observed galaxy redshift (including peculiar velocity)', `hpix_29_nest` bigint COMMENT 'healpix pixel id using order = 29 in NESTED format', `vx_gal` float COMMENT 'physical (peculiar) velocity in x-component (km/s)', `vy_gal` float COMMENT 'physical (peculiar) velocity in y-component (km/s)', `vz_gal` float COMMENT 'physical (peculiar) velocity in z-component (km/s)', `vrad_gal` float COMMENT 'missing comment', `delta_r` float COMMENT 'missing comment', `abs_mag_r01` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h)', `luminosity_r01` float COMMENT 'luminosity in the 0.1-r-band', `abs_mag_r01_evolved` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h) including MICECATv2.0 magnitude evolution', `sdss_r01` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1', `abs_mag_uv_dereddened` float COMMENT 'absolute magnitude ultraviolet', `gr_restframe` float COMMENT '(g-r) rest-frame at z=0', `color_kind` tinyint COMMENT 'galaxy color kind: 0=red sequence 1=green valley 2=blue cloud', `sed_ke` float COMMENT 'K-correction for each SED', `gr_cosmos` float COMMENT 'interpolated COSMOS g-r color', `sed_cosmos` float COMMENT 'interpolated basic COSMOS SED [0 30]', `is_within_cosmos` boolean COMMENT 'missing comment', `cosmos_distance` float COMMENT 'missing comment', `ext_curve_cosmos` tinyint COMMENT 'COSMOS extinction curve [0-4]', `ebv_cosmos` float COMMENT 'interpolated extinction strength for COSMOS extinction curve', `log_ml_r01` float COMMENT 'logarithm of stellar_mass/luminosity ratio in the r01 SDSS band', `log_stellar_mass` float COMMENT 'logarithm of the stellar mass (Msun/h)', `metallicity` float COMMENT 'oxygen metallicity in units 12+logO/H', `log_sfr` float COMMENT 'logarithm of star formation rate in (Msun/h)/year', `logf_halpha` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: DOES NOT include extinction)', `logf_halpha_ext` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: includes extinction)', `halpha_scatter` float COMMENT 'missing comment', `loglum_halpha` float COMMENT 'DEPRECATED logarithm of Halpha luminosity (erg/s/h^2) using empirical relation with UV magnitude (DOES NOT include extinction)', `loglum_halpha_ext` float COMMENT 'missing comment', `logf_dummy` float COMMENT 'missing comment', `z_dummy` float COMMENT 'missing comment', `logf_halpha_model1_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: includes extinction)', `logf_halpha_model1` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: DOES NOT include extinction)', `loglum_halpha_model1_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (includes extinction)', `loglum_halpha_model1` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (DOES NOT include extinction)', `logf_halpha_model3_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: includes extinction)', `logf_halpha_model3` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: DOES NOT include extinction)', `loglum_halpha_model3_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (includes extinction)', `loglum_halpha_model3` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (DOES NOT include extinction)', `logf_hbeta_model1_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: includes extinction)', `logf_hbeta_model1` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)', `logf_o2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)', `logf_o2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)', `logf_n2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)', `logf_n2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)', `logf_o3_model1_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: includes extinction)', `logf_o3_model1` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)', `logf_s2_model1_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: includes extinction)', `logf_s2_model1` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)', `logf_hbeta_model3_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: includes extinction)', `logf_hbeta_model3` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)', `logf_o2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)', `logf_o2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)', `logf_n2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)', `logf_n2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)', `logf_o3_model3_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: includes extinction)', `logf_o3_model3` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)', `logf_s2_model3_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: includes extinction)', `logf_s2_model3` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)', `blanco_decam_i` float COMMENT 'observed flux in DES i-band (Euclid provided)', `euclid_nisp_h` float COMMENT 'observed flux in Euclid NISP-H band (Euclid provided)', `blanco_decam_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES i-band (Euclid provided)', `euclid_nisp_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-H band (Euclid provided)', `blanco_decam_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES i-band (Euclid provided)', `blanco_decam_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES i-band (Euclid provided)', `euclid_nisp_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-H band (Euclid provided)', `euclid_nisp_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-H band (Euclid provided)', `sdss_r01_odonnell_ext` float COMMENT 'observed flux with MW extinction in in sdss r-band redshifted to z=0.1', `sdss_r01_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in in sdss r-band redshifted to z=0.1', `sdss_r01_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in in sdss r-band redshifted to z=0.1', `dominant_shape` tinyint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)', `bulge_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', `disk_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination', `median_major_axis` float COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)', `scale_length` float COMMENT 'disc and bulge scalelength prior', `bulge_fraction` float COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)', `disk_scalelength` float COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)', `disk_nsersic` float COMMENT 'Sersic index of the disk component', `disk_r50` float COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678', `bulge_r50` float COMMENT 'half light radius of the bulge', `bulge_nsersic` float COMMENT 'Sersic index of the bulge component', `disk_ellipticity` float COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)', `bulge_ellipticity` float COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)', `disk_axis_ratio` float COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', `bulge_axis_ratio` float COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)', `inclination_angle` float COMMENT 'Galaxy inclination angle (where 0 degrees = face-on and 90 degrees = edge-on). Galaxy ellipticity for disk and bulge components are computed following the recipe in https://euclid.roe.ac.uk/projects/sgsshear/wiki/SHE-SIM', `halo_x` float COMMENT 'halo comoving x coordinate (Mpc/h)', `halo_y` float COMMENT 'halo comoving y coordinate (Mpc/h)', `halo_z` float COMMENT 'halo comoving z coordinate (Mpc/h)', `halo_vx` float COMMENT 'halo x-centre velocity (km/h)', `halo_vy` float COMMENT 'halo y-centre velocity (km/h)', `halo_vz` float COMMENT 'halo z-centre velocity (km/h)', `halo_r` float COMMENT 'halo comoving distance (Mpc/h)', `true_redshift_halo` float COMMENT 'true redshift of the host halo', `halo_lm` float COMMENT 'log10 of the FoF halo mass in Msum/h. This halo mass is computed as the particule mass multiplied by the number of particles of the FoF halo. At low masses it is interpolated to obtain a smooth distribution of halo masses', `halo_n_sats` int COMMENT 'number of satellite galaxies from HOD', `n_gals` int COMMENT 'DEPRECATED', `luminosity_r01_evolved` float COMMENT 'missing comment', `hpix_13_nest` bigint COMMENT 'missing comment' PARTITIONED BY ( `step` smallint COMMENT 'step number in the Flagship lensing maps') CLUSTERED BY ( hpix_13_nest) SORTED BY ( hpix_13_nest ASC) INTO 128 BUCKETS ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' ;
TENGO QUE ENTENDER DONDE SE CALCULAN LOS BUCKETS!
PORQUE NO LOS TENGO EN MI CATALOGO DE BASE, creo!
cat << EOF > lensing_step.sql
SET hive.auto.convert.join.noconditionaltask.size=1;
INSERT OVERWRITE TABLE jcarrete.flagship_mock_1_9_8_s PARTITION (step=${STEP})
SELECT
m.`halo_id`, m.`galaxy_id`, m.`kind`, m.`random_index`, m.`ra_gal`, m.`dec_gal`, magpos.`ra_mag_gal`, magpos.`dec_mag_gal`, m.`kappa`, m.`gamma1`, m.`gamma2`, m.`l_gal`, m.`b_gal`, m.`mw_extinction`, m.`x_gal`, m.`y_gal`, m.`z_gal`, m.`r_gal`, m.`true_redshift_gal`, m.`observed_redshift_gal`, m.`hpix_29_nest`, m.`vx_gal`, m.`vy_gal`, m.`vz_gal`, m.`vrad_gal`, m.`delta_r`, m.`abs_mag_r01`, m.`luminosity_r01`, m.`abs_mag_r01_evolved`, m.`sdss_r01`, m.`abs_mag_uv_dereddened`, m.`gr_restframe`, m.`color_kind`, m.`sed_ke`, m.`gr_cosmos`, m.`sed_cosmos`, m.`is_within_cosmos`, m.`cosmos_distance`, m.`ext_curve_cosmos`, m.`ebv_cosmos`, m.`log_ml_r01`, m.`log_stellar_mass`, m.`metallicity`, m.`log_sfr`, m.`logf_halpha`, m.`logf_halpha_ext`, m.`halpha_scatter`, m.`loglum_halpha`, m.`loglum_halpha_ext`, m.`logf_dummy`, m.`z_dummy`, m.`logf_halpha_model1_ext`, m.`logf_halpha_model1`, m.`loglum_halpha_model1_ext`, m.`loglum_halpha_model1`, m.`logf_halpha_model3_ext`, m.`logf_halpha_model3`, m.`loglum_halpha_model3_ext`, m.`loglum_halpha_model3`, m.`logf_hbeta_model1_ext`, m.`logf_hbeta_model1`, m.`logf_o2_model1_ext`, m.`logf_o2_model1`, m.`logf_n2_model1_ext`, m.`logf_n2_model1`, m.`logf_o3_model1_ext`, m.`logf_o3_model1`, m.`logf_s2_model1_ext`, m.`logf_s2_model1`, m.`logf_hbeta_model3_ext`, m.`logf_hbeta_model3`, m.`logf_o2_model3_ext`, m.`logf_o2_model3`, m.`logf_n2_model3_ext`, m.`logf_n2_model3`, m.`logf_o3_model3_ext`, m.`logf_o3_model3`, m.`logf_s2_model3_ext`, m.`logf_s2_model3`, m.`blanco_decam_i`, m.`euclid_nisp_h`, m.`blanco_decam_i_odonnell_ext`, m.`euclid_nisp_h_odonnell_ext`, m.`blanco_decam_i_el_model1_odonnell_ext`, m.`blanco_decam_i_el_model3_odonnell_ext`, m.`euclid_nisp_h_el_model1_odonnell_ext`, m.`euclid_nisp_h_el_model3_odonnell_ext`, m.`sdss_r01_odonnell_ext`, m.`sdss_r01_el_model1_odonnell_ext`, m.`sdss_r01_el_model3_odonnell_ext`, m.`dominant_shape`, m.`bulge_angle`, m.`disk_angle`, m.`median_major_axis`, m.`scale_length`, m.`bulge_fraction`, m.`disk_scalelength`, m.`disk_nsersic`, m.`disk_r50`, m.`bulge_r50`, m.`bulge_nsersic`, m.`disk_ellipticity`, m.`bulge_ellipticity`, m.`disk_axis_ratio`, m.`bulge_axis_ratio`, m.`inclination_angle`, m.`halo_x`, m.`halo_y`, m.`halo_z`, m.`halo_vx`, m.`halo_vy`, m.`halo_vz`, m.`halo_r`, m.`true_redshift_halo`, m.`halo_lm`, m.`halo_n_sats`, m.`n_gals`, m.`luminosity_r01_evolved`, m.`hpix_13_nest`
FROM (
SELECT
m.`halo_id`, m.`galaxy_id`, m.`kind`, m.`random_index`, m.`ra_gal`, m.`dec_gal`, magpos.`ra_mag_gal`, magpos.`dec_mag_gal`, l.defl_0 as kappa, l.shear_1 as gamma1, l.shear_2 as gamma2, m.`l_gal`, m.`b_gal`, m.`mw_extinction`, m.`x_gal`, m.`y_gal`, m.`z_gal`, m.`r_gal`, m.`true_redshift_gal`, m.`observed_redshift_gal`, m.`hpix_29_nest`, m.`vx_gal`, m.`vy_gal`, m.`vz_gal`, m.`vrad_gal`, m.`delta_r`, m.`abs_mag_r01`, m.`luminosity_r01`, m.`abs_mag_r01_evolved`, m.`sdss_r01`, m.`abs_mag_uv_dereddened`, m.`gr_restframe`, m.`color_kind`, m.`sed_ke`, m.`gr_cosmos`, m.`sed_cosmos`, m.`is_within_cosmos`, m.`cosmos_distance`, m.`ext_curve_cosmos`, m.`ebv_cosmos`, m.`log_ml_r01`, m.`log_stellar_mass`, m.`metallicity`, m.`log_sfr`, m.`logf_halpha`, m.`logf_halpha_ext`, m.`halpha_scatter`, m.`loglum_halpha`, m.`loglum_halpha_ext`, m.`logf_dummy`, m.`z_dummy`, m.`logf_halpha_model1_ext`, m.`logf_halpha_model1`, m.`loglum_halpha_model1_ext`, m.`loglum_halpha_model1`, m.`logf_halpha_model3_ext`, m.`logf_halpha_model3`, m.`loglum_halpha_model3_ext`, m.`loglum_halpha_model3`, m.`logf_hbeta_model1_ext`, m.`logf_hbeta_model1`, m.`logf_o2_model1_ext`, m.`logf_o2_model1`, m.`logf_n2_model1_ext`, m.`logf_n2_model1`, m.`logf_o3_model1_ext`, m.`logf_o3_model1`, m.`logf_s2_model1_ext`, m.`logf_s2_model1`, m.`logf_hbeta_model3_ext`, m.`logf_hbeta_model3`, m.`logf_o2_model3_ext`, m.`logf_o2_model3`, m.`logf_n2_model3_ext`, m.`logf_n2_model3`, m.`logf_o3_model3_ext`, m.`logf_o3_model3`, m.`logf_s2_model3_ext`, m.`logf_s2_model3`, m.`blanco_decam_i`, m.`euclid_nisp_h`, m.`blanco_decam_i_odonnell_ext`, m.`euclid_nisp_h_odonnell_ext`, m.`blanco_decam_i_el_model1_odonnell_ext`, m.`blanco_decam_i_el_model3_odonnell_ext`, m.`euclid_nisp_h_el_model1_odonnell_ext`, m.`euclid_nisp_h_el_model3_odonnell_ext`, m.`sdss_r01_odonnell_ext`, m.`sdss_r01_el_model1_odonnell_ext`, m.`sdss_r01_el_model3_odonnell_ext`, m.`dominant_shape`, m.`bulge_angle`, m.`disk_angle`, m.`median_major_axis`, m.`scale_length`, m.`bulge_fraction`, m.`disk_scalelength`, m.`disk_nsersic`, m.`disk_r50`, m.`bulge_r50`, m.`bulge_nsersic`, m.`disk_ellipticity`, m.`bulge_ellipticity`, m.`disk_axis_ratio`, m.`bulge_axis_ratio`, m.`inclination_angle`, m.`halo_x`, m.`halo_y`, m.`halo_z`, m.`halo_vx`, m.`halo_vy`, m.`halo_vz`, m.`halo_r`, m.`true_redshift_halo`, m.`halo_lm`, m.`halo_n_sats`, m.`n_gals`, m.`luminosity_r01_evolved`, m.`hpix_13_nest`
udf.magnified_positions(CAST(m.ra_gal AS FLOAT), CAST(m.dec_gal AS FLOAT), l.defl_1, l.defl_2) AS magpos
FROM jcarrete.flagship_mock_1_9_8_c2 AS m
JOIN cosmohub.flagship_lensing AS l
ON udf.ang2pix(13, m.ra_gal, m.dec_gal, True, True) = l.hpix
AND m.step = l.step
AND l.step=${STEP} AND m.step=${STEP}
) AS m
;
Materializing the table:
SWF1 Area pointing obsid: 52929 ra_range = 7.678307967139321 14.008504032860678 dec_range = -21.587506 -15.587506000000001
Area pointing obsid in 1st octant: 52929 ra_range = 7.678307967139321 14.008504032860678 dec_range = 15.587506000000001 21.587506
SWF2 Area pointing obsid: 1717 ra_range = 215.45815488783862 241.3301611121614 dec_range = 73.5904 79.5904
Area pointing obsid in 1st octant: 1717 ra_range = 35.45815488783862 61.33016111216139 dec_range = 73.5904 79.5904
213.074616 251.75015 72.188664 82.03694
CREATE TABLE jcarrete.flagship_mock_obsid_52929_1717_c2(
`kind` BIGINT,
`luminosity_r01` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`abs_mag_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`luminosity_r01_evolved` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` INT,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`hpix_29_nest` BIGINT,
`hpix_13_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`observed_redshift_gal` DOUBLE,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha_ext` DOUBLE,
`logf_halpha` DOUBLE,
`Halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_Halpha_model3_ext` DOUBLE,
`loglum_Halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`blanco_decam_i` DOUBLE,
`sdss_r01` DOUBLE,
`euclid_nisp_h` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`dominant_shape` BIGINT,
`bulge_angle` DOUBLE,
`disk_angle` DOUBLE,
`median_major_axis` DOUBLE,
`scale_length` DOUBLE,
`bulge_fraction` DOUBLE,
`disk_scalelength` DOUBLE,
`disk_nsersic` DOUBLE,
`disk_r50` DOUBLE,
`bulge_r50` DOUBLE,
`bulge_nsersic` DOUBLE,
`inclination_angle` DOUBLE,
`disk_axis_ratio` DOUBLE,
`bulge_axis_ratio` DOUBLE,
`disk_ellipticity` DOUBLE,
`bulge_ellipticity` DOUBLE,
`random_index` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
PARTITIONED BY (
`step` smallint
)
CLUSTERED BY (
hpix_13_nest
)
SORTED BY (
hpix_13_nest ASC
)
INTO 128 BUCKETS
STORED AS ORC
;
1) Crear clustered _c3
2) Cambiar el nombre del campo:
hpix_13_nest_mice y calcularlo con el mice convention: ra_mice = 90 - ra dec_mice = dec
udf. ang2pix(order =13, ra_mice, dec_mice, nest=true, lonlat=true)
3) En Test:
INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_52929_1717_c2 SELECT `kind`, `luminosity_r01`, `halo_x`, `halo_y`, `halo_z`, `halo_vx`, `halo_vy`, `halo_vz`, `halo_r`, `true_redshift_halo`, `halo_lm`, `halo_n_sats`, `n_gals`, `abs_mag_r01`, `abs_mag_r01_evolved`, `luminosity_r01_evolved`, `gr_restframe`, `color_kind`, `x_gal`, `y_gal`, `z_gal`, `r_gal`, `true_redshift_gal`, `ra_gal`, `dec_gal`, `hpix_nest`, SHIFTRIGHT(`hpix_nest`, (29-13)*2), `vx_gal`, `vy_gal`, `vz_gal`, `vrad_gal`, `delta_r`, `observed_redshift_gal`, `sed_ke`, `gr_cosmos`, `sed_cosmos`, `ext_curve_cosmos`, `ebv_cosmos`, `is_within_cosmos`, `cosmos_distance`, `abs_mag_uv_dereddened`, `log_ml_r01`, `log_stellar_mass`, `metallicity`, `log_sfr`, `logf_halpha_ext`, `logf_halpha`, `Halpha_scatter`, `loglum_halpha`, `loglum_halpha_ext`, `logf_dummy`, `z_dummy`, `logf_halpha_model1_ext`, `logf_halpha_model1`, `loglum_halpha_model1_ext`, `loglum_halpha_model1`, `logf_halpha_model3_ext`, `logf_halpha_model3`, `loglum_Halpha_model3_ext`, `loglum_Halpha_model3`, `logf_hbeta_model1_ext`, `logf_hbeta_model1`, `logf_o2_model1_ext`, `logf_o2_model1`, `logf_n2_model1_ext`, `logf_n2_model1`, `logf_o3_model1_ext`, `logf_o3_model1`, `logf_s2_model1_ext`, `logf_s2_model1`, `logf_hbeta_model3_ext`, `logf_hbeta_model3`, `logf_o2_model3_ext`, `logf_o2_model3`, `logf_n2_model3_ext`, `logf_n2_model3`, `logf_o3_model3_ext`, `logf_o3_model3`, `logf_s2_model3_ext`, `logf_s2_model3`, `blanco_decam_i`, `sdss_r01`, `euclid_nisp_h`, `l_gal`, `b_gal`, `mw_extinction`, `blanco_decam_i_odonnell_ext`, `euclid_nisp_h_odonnell_ext`, `sdss_r01_odonnell_ext`, `blanco_decam_i_el_model1_odonnell_ext`, `blanco_decam_i_el_model3_odonnell_ext`, `sdss_r01_el_model1_odonnell_ext`, `sdss_r01_el_model3_odonnell_ext`, `euclid_nisp_h_el_model1_odonnell_ext`, `euclid_nisp_h_el_model3_odonnell_ext`, `dominant_shape`, `bulge_angle`, `disk_angle`, `median_major_axis`, `scale_length`, `bulge_fraction`, `disk_scalelength`, `disk_nsersic`, `disk_r50`, `bulge_r50`, `bulge_nsersic`, `inclination_angle`, `disk_axis_ratio`, `bulge_axis_ratio`, `disk_ellipticity`, `bulge_ellipticity`, `random_index`, `halo_id`, `galaxy_id`, `step` FROM jcarrete.flagship_mock_1_9_8_c WHERE (ra_gal >= 7.678307967139321 AND ra_gal < 14.008504032860678 AND dec_gal >= 15.587506 AND dec_gal < 21.587506) OR (ra_gal >= 35.45815488783862 AND ra_gal < 61.33016111216139 AND dec_gal >=73.5904 AND dec_gal < 79.5904) ;
ANALYZE TABLE jcarrete.flagship_mock_obsid_52929_1717_c2 COMPUTE STATISTICS FOR COLUMNS;
CREATE TABLE jcarrete.flagship_mock_obsid_52929_1717_s(
`halo_id` bigint COMMENT 'unique halo id coming from the Flagship dark matter halo catalog',
`galaxy_id` int COMMENT 'combined with the unique_halo_id they uniquely identify each galaxy',
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',
`random_index` float COMMENT 'random number [0 - 1) for subsampling',
`ra_gal` float COMMENT 'galaxy right ascension (degrees)',
`dec_gal` float COMMENT 'galaxy declination (degrees)',
`ra_mag_gal` float COMMENT 'galaxy magnified right ascension (degree)',
`dec_mag_gal` float COMMENT 'galaxy magnified declination (degree)',
`kappa` float COMMENT 'convergence',
`gamma1` float COMMENT 'shear',
`gamma2` float COMMENT 'shear',
`l_gal` float COMMENT 'galactic longitude (degrees)',
`b_gal` float COMMENT 'galactic latitude (degrees)',
`mw_extinction` float COMMENT 'EBV from file: HFI_CompMap_ThermalDustModel_2048_R1.20.fits',
`x_gal` float COMMENT 'galaxy comoving x coordinate (Mpc/h)',
`y_gal` float COMMENT 'galaxy comoving y coordinate (Mpc/h)',
`z_gal` float COMMENT 'galaxy comoving z coordinate (Mpc/h)',
`r_gal` float COMMENT 'galaxy comoving distance (Mpc/h)',
`true_redshift_gal` float COMMENT 'true galaxy redshift',
`observed_redshift_gal` float COMMENT 'observed galaxy redshift (including peculiar velocity)',
`hpix_29_nest` bigint COMMENT 'healpix pixel id using order = 29 in NESTED format',
`vx_gal` float COMMENT 'physical (peculiar) velocity in x-component (km/s)',
`vy_gal` float COMMENT 'physical (peculiar) velocity in y-component (km/s)',
`vz_gal` float COMMENT 'physical (peculiar) velocity in z-component (km/s)',
`vrad_gal` float COMMENT 'missing comment',
`delta_r` float COMMENT 'missing comment',
`abs_mag_r01` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h)',
`luminosity_r01` float COMMENT 'luminosity in the 0.1-r-band',
`abs_mag_r01_evolved` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h) including MICECATv2.0 magnitude evolution',
`sdss_r01` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1',
`abs_mag_uv_dereddened` float COMMENT 'absolute magnitude ultraviolet',
`gr_restframe` float COMMENT '(g-r) rest-frame at z=0',
`color_kind` tinyint COMMENT 'galaxy color kind: 0=red sequence 1=green valley 2=blue cloud',
`sed_ke` float COMMENT 'K-correction for each SED',
`gr_cosmos` float COMMENT 'interpolated COSMOS g-r color',
`sed_cosmos` float COMMENT 'interpolated basic COSMOS SED [0 30]',
`is_within_cosmos` boolean COMMENT 'missing comment',
`cosmos_distance` float COMMENT 'missing comment',
`ext_curve_cosmos` tinyint COMMENT 'COSMOS extinction curve [0-4]',
`ebv_cosmos` float COMMENT 'interpolated extinction strength for COSMOS extinction curve',
`log_ml_r01` float COMMENT 'logarithm of stellar_mass/luminosity ratio in the r01 SDSS band',
`log_stellar_mass` float COMMENT 'logarithm of the stellar mass (Msun/h)',
`metallicity` float COMMENT 'oxygen metallicity in units 12+logO/H',
`log_sfr` float COMMENT 'logarithm of star formation rate in (Msun/h)/year',
`logf_halpha` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: DOES NOT include extinction)',
`logf_halpha_ext` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: includes extinction)',
`halpha_scatter` float COMMENT 'missing comment',
`loglum_halpha` float COMMENT 'DEPRECATED logarithm of Halpha luminosity (erg/s/h^2) using empirical relation with UV magnitude (DOES NOT include extinction)',
`loglum_halpha_ext` float COMMENT 'missing comment',
`logf_dummy` float COMMENT 'missing comment',
`z_dummy` float COMMENT 'missing comment',
`logf_halpha_model1_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: includes extinction)',
`logf_halpha_model1` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: DOES NOT include extinction)',
`loglum_halpha_model1_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (includes extinction)',
`loglum_halpha_model1` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (DOES NOT include extinction)',
`logf_halpha_model3_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: includes extinction)',
`logf_halpha_model3` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: DOES NOT include extinction)',
`loglum_halpha_model3_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (includes extinction)',
`loglum_halpha_model3` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (DOES NOT include extinction)',
`logf_hbeta_model1_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_hbeta_model1` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_n2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_n2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o3_model1_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o3_model1` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_s2_model1_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_s2_model1` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_hbeta_model3_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_hbeta_model3` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_n2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_n2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o3_model3_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o3_model3` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_s2_model3_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_s2_model3` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`blanco_decam_i` float COMMENT 'observed flux in DES i-band (Euclid provided)',
`euclid_nisp_h` float COMMENT 'observed flux in Euclid NISP-H band (Euclid provided)',
`blanco_decam_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES i-band (Euclid provided)',
`euclid_nisp_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-H band (Euclid provided)',
`blanco_decam_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES i-band (Euclid provided)',
`euclid_nisp_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-H band (Euclid provided)',
`sdss_r01_odonnell_ext` float COMMENT 'observed flux with MW extinction in in sdss r-band redshifted to z=0.1',
`sdss_r01_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in in sdss r-band redshifted to z=0.1',
`sdss_r01_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in in sdss r-band redshifted to z=0.1',
`dominant_shape` tinyint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)',
`bulge_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`disk_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`median_major_axis` float COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)',
`scale_length` float COMMENT 'disc and bulge scalelength prior',
`bulge_fraction` float COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)',
`disk_scalelength` float COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)',
`disk_nsersic` float COMMENT 'Sersic index of the disk component',
`disk_r50` float COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678',
`bulge_r50` float COMMENT 'half light radius of the bulge',
`bulge_nsersic` float COMMENT 'Sersic index of the bulge component',
`disk_ellipticity` float COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)',
`bulge_ellipticity` float COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)',
`disk_axis_ratio` float COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`bulge_axis_ratio` float COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`inclination_angle` float COMMENT 'Galaxy inclination angle (where 0 degrees = face-on and 90 degrees = edge-on). Galaxy ellipticity for disk and bulge components are computed following the recipe in https://euclid.roe.ac.uk/projects/sgsshear/wiki/SHE-SIM',
`halo_x` float COMMENT 'halo comoving x coordinate (Mpc/h)',
`halo_y` float COMMENT 'halo comoving y coordinate (Mpc/h)',
`halo_z` float COMMENT 'halo comoving z coordinate (Mpc/h)',
`halo_vx` float COMMENT 'halo x-centre velocity (km/h)',
`halo_vy` float COMMENT 'halo y-centre velocity (km/h)',
`halo_vz` float COMMENT 'halo z-centre velocity (km/h)',
`halo_r` float COMMENT 'halo comoving distance (Mpc/h)',
`true_redshift_halo` float COMMENT 'true redshift of the host halo',
`halo_lm` float COMMENT 'log10 of the FoF halo mass in Msum/h. This halo mass is computed as the particule mass multiplied by the number of particles of the FoF halo. At low masses it is interpolated to obtain a smooth distribution of halo masses',
`halo_n_sats` int COMMENT 'number of satellite galaxies from HOD',
`n_gals` int COMMENT 'DEPRECATED',
`luminosity_r01_evolved` float COMMENT 'missing comment',
`hpix_13_nest` bigint COMMENT 'missing comment')
PARTITIONED BY (
`step` smallint COMMENT 'step number in the Flagship lensing maps')
CLUSTERED BY (
hpix_13_nest)
SORTED BY (
hpix_13_nest ASC)
INTO 128 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;
nano lensing_step.sql
Pego esto:
SET hive.auto.convert.join.noconditionaltask.size=1;
INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_52929_1717_s PARTITION (step=${STEP})
SELECT
m.`halo_id`, m.`galaxy_id`, m.`kind`, m.`random_index`, m.`ra_gal`, m.`dec_gal`, magpos.`ra_mag` as ra_mag_gal, magpos.`dec_mag` as dec_mag_gal, m.`kappa`, m.`gamma1`, m.`gamma2`, m.`l_gal`, m.`b_gal`, m.`mw_extinction`, m.`x_gal`, m.`y_gal`, m.`z_gal`, m.`r_gal`, m.`true_redshift_gal`, m.`observed_redshift_gal`, m.`hpix_29_nest`, m.`vx_gal`, m.`vy_gal`, m.`vz_gal`, m.`vrad_gal`, m.`delta_r`, m.`abs_mag_r01`, m.`luminosity_r01`, m.`abs_mag_r01_evolved`, m.`sdss_r01`, m.`abs_mag_uv_dereddened`, m.`gr_restframe`, m.`color_kind`, m.`sed_ke`, m.`gr_cosmos`, m.`sed_cosmos`, m.`is_within_cosmos`, m.`cosmos_distance`, m.`ext_curve_cosmos`, m.`ebv_cosmos`, m.`log_ml_r01`, m.`log_stellar_mass`, m.`metallicity`, m.`log_sfr`, m.`logf_halpha`, m.`logf_halpha_ext`, m.`halpha_scatter`, m.`loglum_halpha`, m.`loglum_halpha_ext`, m.`logf_dummy`, m.`z_dummy`, m.`logf_halpha_model1_ext`, m.`logf_halpha_model1`, m.`loglum_halpha_model1_ext`, m.`loglum_halpha_model1`, m.`logf_halpha_model3_ext`, m.`logf_halpha_model3`, m.`loglum_halpha_model3_ext`, m.`loglum_halpha_model3`, m.`logf_hbeta_model1_ext`, m.`logf_hbeta_model1`, m.`logf_o2_model1_ext`, m.`logf_o2_model1`, m.`logf_n2_model1_ext`, m.`logf_n2_model1`, m.`logf_o3_model1_ext`, m.`logf_o3_model1`, m.`logf_s2_model1_ext`, m.`logf_s2_model1`, m.`logf_hbeta_model3_ext`, m.`logf_hbeta_model3`, m.`logf_o2_model3_ext`, m.`logf_o2_model3`, m.`logf_n2_model3_ext`, m.`logf_n2_model3`, m.`logf_o3_model3_ext`, m.`logf_o3_model3`, m.`logf_s2_model3_ext`, m.`logf_s2_model3`, m.`blanco_decam_i`, m.`euclid_nisp_h`, m.`blanco_decam_i_odonnell_ext`, m.`euclid_nisp_h_odonnell_ext`, m.`blanco_decam_i_el_model1_odonnell_ext`, m.`blanco_decam_i_el_model3_odonnell_ext`, m.`euclid_nisp_h_el_model1_odonnell_ext`, m.`euclid_nisp_h_el_model3_odonnell_ext`, m.`sdss_r01_odonnell_ext`, m.`sdss_r01_el_model1_odonnell_ext`, m.`sdss_r01_el_model3_odonnell_ext`, m.`dominant_shape`, m.`bulge_angle`, m.`disk_angle`, m.`median_major_axis`, m.`scale_length`, m.`bulge_fraction`, m.`disk_scalelength`, m.`disk_nsersic`, m.`disk_r50`, m.`bulge_r50`, m.`bulge_nsersic`, m.`disk_ellipticity`, m.`bulge_ellipticity`, m.`disk_axis_ratio`, m.`bulge_axis_ratio`, m.`inclination_angle`, m.`halo_x`, m.`halo_y`, m.`halo_z`, m.`halo_vx`, m.`halo_vy`, m.`halo_vz`, m.`halo_r`, m.`true_redshift_halo`, m.`halo_lm`, m.`halo_n_sats`, m.`n_gals`, m.`luminosity_r01_evolved`, m.`hpix_13_nest`
FROM (
SELECT
m.`halo_id`, m.`galaxy_id`, m.`kind`, m.`random_index`, m.`ra_gal`, m.`dec_gal`, l.defl_0 as kappa, l.shear_1 as gamma1, l.shear_2 as gamma2, m.`l_gal`, m.`b_gal`, m.`mw_extinction`, m.`x_gal`, m.`y_gal`, m.`z_gal`, m.`r_gal`, m.`true_redshift_gal`, m.`observed_redshift_gal`, m.`hpix_29_nest`, m.`vx_gal`, m.`vy_gal`, m.`vz_gal`, m.`vrad_gal`, m.`delta_r`, m.`abs_mag_r01`, m.`luminosity_r01`, m.`abs_mag_r01_evolved`, m.`sdss_r01`, m.`abs_mag_uv_dereddened`, m.`gr_restframe`, m.`color_kind`, m.`sed_ke`, m.`gr_cosmos`, m.`sed_cosmos`, m.`is_within_cosmos`, m.`cosmos_distance`, m.`ext_curve_cosmos`, m.`ebv_cosmos`, m.`log_ml_r01`, m.`log_stellar_mass`, m.`metallicity`, m.`log_sfr`, m.`logf_halpha`, m.`logf_halpha_ext`, m.`halpha_scatter`, m.`loglum_halpha`, m.`loglum_halpha_ext`, m.`logf_dummy`, m.`z_dummy`, m.`logf_halpha_model1_ext`, m.`logf_halpha_model1`, m.`loglum_halpha_model1_ext`, m.`loglum_halpha_model1`, m.`logf_halpha_model3_ext`, m.`logf_halpha_model3`, m.`loglum_halpha_model3_ext`, m.`loglum_halpha_model3`, m.`logf_hbeta_model1_ext`, m.`logf_hbeta_model1`, m.`logf_o2_model1_ext`, m.`logf_o2_model1`, m.`logf_n2_model1_ext`, m.`logf_n2_model1`, m.`logf_o3_model1_ext`, m.`logf_o3_model1`, m.`logf_s2_model1_ext`, m.`logf_s2_model1`, m.`logf_hbeta_model3_ext`, m.`logf_hbeta_model3`, m.`logf_o2_model3_ext`, m.`logf_o2_model3`, m.`logf_n2_model3_ext`, m.`logf_n2_model3`, m.`logf_o3_model3_ext`, m.`logf_o3_model3`, m.`logf_s2_model3_ext`, m.`logf_s2_model3`, m.`blanco_decam_i`, m.`euclid_nisp_h`, m.`blanco_decam_i_odonnell_ext`, m.`euclid_nisp_h_odonnell_ext`, m.`blanco_decam_i_el_model1_odonnell_ext`, m.`blanco_decam_i_el_model3_odonnell_ext`, m.`euclid_nisp_h_el_model1_odonnell_ext`, m.`euclid_nisp_h_el_model3_odonnell_ext`, m.`sdss_r01_odonnell_ext`, m.`sdss_r01_el_model1_odonnell_ext`, m.`sdss_r01_el_model3_odonnell_ext`, m.`dominant_shape`, m.`bulge_angle`, m.`disk_angle`, m.`median_major_axis`, m.`scale_length`, m.`bulge_fraction`, m.`disk_scalelength`, m.`disk_nsersic`, m.`disk_r50`, m.`bulge_r50`, m.`bulge_nsersic`, m.`disk_ellipticity`, m.`bulge_ellipticity`, m.`disk_axis_ratio`, m.`bulge_axis_ratio`, m.`inclination_angle`, m.`halo_x`, m.`halo_y`, m.`halo_z`, m.`halo_vx`, m.`halo_vy`, m.`halo_vz`, m.`halo_r`, m.`true_redshift_halo`, m.`halo_lm`, m.`halo_n_sats`, m.`n_gals`, m.`luminosity_r01_evolved`, m.`hpix_13_nest`,
udf.magnified_positions(CAST(m.ra_gal AS FLOAT), CAST(m.dec_gal AS FLOAT), l.defl_1, l.defl_2) AS magpos
FROM jcarrete.flagship_mock_obsid_52929_1717_c2 AS m
JOIN cosmohub.flagship_lensing AS l
ON udf.ang2pix(13, m.ra_gal, m.dec_gal, True, True) = l.hpix
AND m.step = l.step
AND l.step=${STEP} AND m.step=${STEP}
) AS m
;
for STEP in $(seq 87 400);
do
echo $STEP
sem -j4 "beeline --color=true --showHeader=true --verbose=true --silent=false -u \
'jdbc:hive2://ambarisrv02.pic.es:2181,ambarisrv03.pic.es:2181,ambarisrv01.pic.es:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2' \
-f lensing_step.sql --hivevar STEP=${STEP}"
done
sem --wait
Once the lensing is added to the clustered table that contains two of the paintings, I run the flux_pipeline, in which we have updated the euclid filters! I do it for the 1717 observation since at this time I do not know what to do with the area that has to be rotated to the southern hemisphere (which is the obsid= 52929).
This is done once the catalog is generated with the flux_pipeline.ipynb
/software/astro/scripts/create_table_from_parquet.sh -p /hdfs/user/jcarrete/data/euclid/flagship_mock_obsid_1717_s/0.pq -t jcarrete.flagship_mock_obsid_1717_s_pq -e
CREATE EXTERNAL TABLE jcarrete.flagship_mock_obsid_1717_s_pq (
`kind` BIGINT,
`random_index` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`ra_mag_gal` DOUBLE,
`dec_mag_gal` DOUBLE,
`kappa` DOUBLE,
`gamma1` DOUBLE,
`gamma2` DOUBLE,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`observed_redshift_gal` DOUBLE,
`hpix_29_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`abs_mag_r01` DOUBLE,
`luminosity_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`sdss_r01` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` BIGINT,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha` DOUBLE,
`logf_halpha_ext` DOUBLE,
`halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_halpha_model3_ext` DOUBLE,
`loglum_halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`dominant_shape` BIGINT,
`bulge_angle` DOUBLE,
`disk_angle` DOUBLE,
`median_major_axis` DOUBLE,
`scale_length` DOUBLE,
`bulge_fraction` DOUBLE,
`disk_scalelength` DOUBLE,
`disk_nsersic` DOUBLE,
`disk_r50` DOUBLE,
`bulge_r50` DOUBLE,
`bulge_nsersic` DOUBLE,
`disk_ellipticity` DOUBLE,
`bulge_ellipticity` DOUBLE,
`disk_axis_ratio` DOUBLE,
`bulge_axis_ratio` DOUBLE,
`inclination_angle` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`luminosity_r01_evolved` DOUBLE,
`hpix_13_nest` BIGINT,
`step` BIGINT,
`blanco_decam_g` DOUBLE,
`blanco_decam_i` DOUBLE,
`blanco_decam_r` DOUBLE,
`blanco_decam_z` DOUBLE,
`subaru_hsc_z` DOUBLE,
`jst_jpcam_g` DOUBLE,
`kids_g` DOUBLE,
`kids_i` DOUBLE,
`kids_r` DOUBLE,
`kids_u` DOUBLE,
`lsst_g` DOUBLE,
`lsst_i` DOUBLE,
`lsst_r` DOUBLE,
`lsst_u` DOUBLE,
`lsst_y` DOUBLE,
`lsst_z` DOUBLE,
`cfht_megacam_r` DOUBLE,
`cfht_megacam_u` DOUBLE,
`pan_starrs_i` DOUBLE,
`pan_starrs_z` DOUBLE,
`2mass_h` DOUBLE,
`2mass_j` DOUBLE,
`2mass_ks` DOUBLE,
`gaia_bp` DOUBLE,
`gaia_g` DOUBLE,
`gaia_rp` DOUBLE,
`euclid_nisp_h` DOUBLE,
`euclid_nisp_j` DOUBLE,
`euclid_nisp_y` DOUBLE,
`euclid_vis` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`2mass_h_odonnell_ext` DOUBLE,
`2mass_j_odonnell_ext` DOUBLE,
`2mass_ks_odonnell_ext` DOUBLE,
`blanco_decam_g_odonnell_ext` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`blanco_decam_r_odonnell_ext` DOUBLE,
`blanco_decam_z_odonnell_ext` DOUBLE,
`cfht_megacam_r_odonnell_ext` DOUBLE,
`cfht_megacam_u_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`euclid_nisp_j_odonnell_ext` DOUBLE,
`euclid_nisp_y_odonnell_ext` DOUBLE,
`euclid_vis_odonnell_ext` DOUBLE,
`gaia_bp_odonnell_ext` DOUBLE,
`gaia_g_odonnell_ext` DOUBLE,
`gaia_rp_odonnell_ext` DOUBLE,
`jst_jpcam_g_odonnell_ext` DOUBLE,
`kids_g_odonnell_ext` DOUBLE,
`kids_i_odonnell_ext` DOUBLE,
`kids_r_odonnell_ext` DOUBLE,
`kids_u_odonnell_ext` DOUBLE,
`lsst_g_odonnell_ext` DOUBLE,
`lsst_i_odonnell_ext` DOUBLE,
`lsst_r_odonnell_ext` DOUBLE,
`lsst_u_odonnell_ext` DOUBLE,
`lsst_y_odonnell_ext` DOUBLE,
`lsst_z_odonnell_ext` DOUBLE,
`pan_starrs_i_odonnell_ext` DOUBLE,
`pan_starrs_z_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`subaru_hsc_z_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model3_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model1_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model3_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model1_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model3_odonnell_ext` DOUBLE,
`kids_g_el_model1_odonnell_ext` DOUBLE,
`kids_g_el_model3_odonnell_ext` DOUBLE,
`kids_i_el_model1_odonnell_ext` DOUBLE,
`kids_i_el_model3_odonnell_ext` DOUBLE,
`kids_r_el_model1_odonnell_ext` DOUBLE,
`kids_r_el_model3_odonnell_ext` DOUBLE,
`kids_u_el_model1_odonnell_ext` DOUBLE,
`kids_u_el_model3_odonnell_ext` DOUBLE,
`lsst_g_el_model1_odonnell_ext` DOUBLE,
`lsst_g_el_model3_odonnell_ext` DOUBLE,
`lsst_i_el_model1_odonnell_ext` DOUBLE,
`lsst_i_el_model3_odonnell_ext` DOUBLE,
`lsst_r_el_model1_odonnell_ext` DOUBLE,
`lsst_r_el_model3_odonnell_ext` DOUBLE,
`lsst_u_el_model1_odonnell_ext` DOUBLE,
`lsst_u_el_model3_odonnell_ext` DOUBLE,
`lsst_y_el_model1_odonnell_ext` DOUBLE,
`lsst_y_el_model3_odonnell_ext` DOUBLE,
`lsst_z_el_model1_odonnell_ext` DOUBLE,
`lsst_z_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model3_odonnell_ext` DOUBLE,
`2mass_h_el_model1_odonnell_ext` DOUBLE,
`2mass_h_el_model3_odonnell_ext` DOUBLE,
`2mass_j_el_model1_odonnell_ext` DOUBLE,
`2mass_j_el_model3_odonnell_ext` DOUBLE,
`2mass_ks_el_model1_odonnell_ext` DOUBLE,
`2mass_ks_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`gaia_bp_el_model1_odonnell_ext` DOUBLE,
`gaia_bp_el_model3_odonnell_ext` DOUBLE,
`gaia_g_el_model1_odonnell_ext` DOUBLE,
`gaia_g_el_model3_odonnell_ext` DOUBLE,
`gaia_rp_el_model1_odonnell_ext` DOUBLE,
`gaia_rp_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model3_odonnell_ext` DOUBLE,
`euclid_vis_el_model1_odonnell_ext` DOUBLE,
`euclid_vis_el_model3_odonnell_ext` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
STORED AS PARQUET
LOCATION '/user/jcarrete/data/euclid/flagship_mock_obsid_1717_s'
;
CREATE TABLE jcarrete.flagship_mock_obsid_1717_s(
`halo_id` bigint COMMENT 'unique halo id coming from the Flagship dark matter halo catalog',
`galaxy_id` int COMMENT 'combined with the unique_halo_id they uniquely identify each galaxy',
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',
`random_index` float COMMENT 'random number [0 - 1) for subsampling',
`ra_gal` float COMMENT 'galaxy right ascension (degrees)',
`dec_gal` float COMMENT 'galaxy declination (degrees)',
`ra_mag_gal` float COMMENT 'galaxy magnified right ascension (degree)',
`dec_mag_gal` float COMMENT 'galaxy magnified declination (degree)',
`kappa` float COMMENT 'convergence',
`gamma1` float COMMENT 'shear',
`gamma2` float COMMENT 'shear',
`l_gal` float COMMENT 'galactic longitude (degrees)',
`b_gal` float COMMENT 'galactic latitude (degrees)',
`mw_extinction` float COMMENT 'EBV from file: HFI_CompMap_ThermalDustModel_2048_R1.20.fits',
`x_gal` float COMMENT 'galaxy comoving x coordinate (Mpc/h)',
`y_gal` float COMMENT 'galaxy comoving y coordinate (Mpc/h)',
`z_gal` float COMMENT 'galaxy comoving z coordinate (Mpc/h)',
`r_gal` float COMMENT 'galaxy comoving distance (Mpc/h)',
`true_redshift_gal` float COMMENT 'true galaxy redshift',
`observed_redshift_gal` float COMMENT 'observed galaxy redshift (including peculiar velocity)',
`hpix_29_nest` bigint COMMENT 'healpix pixel id using order = 29 in NESTED format',
`vx_gal` float COMMENT 'physical (peculiar) velocity in x-component (km/s)',
`vy_gal` float COMMENT 'physical (peculiar) velocity in y-component (km/s)',
`vz_gal` float COMMENT 'physical (peculiar) velocity in z-component (km/s)',
`vrad_gal` float COMMENT 'missing comment',
`delta_r` float COMMENT 'missing comment',
`abs_mag_r01` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h)',
`luminosity_r01` float COMMENT 'luminosity in the 0.1-r-band',
`abs_mag_r01_evolved` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h) including MICECATv2.0 magnitude evolution',
`sdss_r01` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1',
`abs_mag_uv_dereddened` float COMMENT 'absolute magnitude ultraviolet',
`gr_restframe` float COMMENT '(g-r) rest-frame at z=0',
`color_kind` tinyint COMMENT 'galaxy color kind: 0=red sequence 1=green valley 2=blue cloud',
`sed_ke` float COMMENT 'K-correction for each SED',
`gr_cosmos` float COMMENT 'interpolated COSMOS g-r color',
`sed_cosmos` float COMMENT 'interpolated basic COSMOS SED [0 30]',
`is_within_cosmos` boolean COMMENT 'missing comment',
`cosmos_distance` float COMMENT 'missing comment',
`ext_curve_cosmos` tinyint COMMENT 'COSMOS extinction curve [0-4]',
`ebv_cosmos` float COMMENT 'interpolated extinction strength for COSMOS extinction curve',
`log_ml_r01` float COMMENT 'logarithm of stellar_mass/luminosity ratio in the r01 SDSS band',
`log_stellar_mass` float COMMENT 'logarithm of the stellar mass (Msun/h)',
`metallicity` float COMMENT 'oxygen metallicity in units 12+logO/H',
`log_sfr` float COMMENT 'logarithm of star formation rate in (Msun/h)/year',
`logf_halpha` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: DOES NOT include extinction)',
`logf_halpha_ext` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: includes extinction)',
`halpha_scatter` float COMMENT 'missing comment',
`loglum_halpha` float COMMENT 'DEPRECATED logarithm of Halpha luminosity (erg/s/h^2) using empirical relation with UV magnitude (DOES NOT include extinction)',
`loglum_halpha_ext` float COMMENT 'missing comment',
`logf_dummy` float COMMENT 'missing comment',
`z_dummy` float COMMENT 'missing comment',
`logf_halpha_model1_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: includes extinction)',
`logf_halpha_model1` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: DOES NOT include extinction)',
`loglum_halpha_model1_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (includes extinction)',
`loglum_halpha_model1` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (DOES NOT include extinction)',
`logf_halpha_model3_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: includes extinction)',
`logf_halpha_model3` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: DOES NOT include extinction)',
`loglum_halpha_model3_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (includes extinction)',
`loglum_halpha_model3` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (DOES NOT include extinction)',
`logf_hbeta_model1_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_hbeta_model1` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_n2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_n2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o3_model1_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o3_model1` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_s2_model1_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_s2_model1` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_hbeta_model3_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_hbeta_model3` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_n2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_n2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o3_model3_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o3_model3` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_s2_model3_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_s2_model3` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`2mass_h` float COMMENT 'observed flux in 2MASS H-band (Euclid provided)',
`2mass_j` float COMMENT 'observed flux in 2MASS J-band (Euclid provided)',
`2mass_ks` float COMMENT 'observed flux in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g` float COMMENT 'observed flux in DES g-band (Euclid provided)',
`blanco_decam_i` float COMMENT 'observed flux in DES i-band (Euclid provided)',
`blanco_decam_r` float COMMENT 'observed flux in DES r-band (Euclid provided)',
`blanco_decam_z` float COMMENT 'observed flux in DES z-band (Euclid provided)',
`cfht_megacam_r` float COMMENT 'observed flux in CFHT r-band (Euclid provided)',
`cfht_megacam_u` float COMMENT 'observed flux in CFHT u-band (Euclid provided)',
`euclid_nisp_h` float COMMENT 'observed flux in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j` float COMMENT 'observed flux in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y` float COMMENT 'observed flux in Euclid NISP-Y band (Euclid provided)',
`euclid_vis` float COMMENT 'observed flux in Euclid VIS band (Euclid provided)',
`gaia_bp` float COMMENT 'observed flux in Gaia-BP band (Euclid provided)',
`gaia_g` float COMMENT 'observed flux in Gaia-G band (Euclid provided)',
`gaia_rp` float COMMENT 'observed flux in Gaia-RP band (Euclid provided)',
`jst_jpcam_g` float COMMENT 'observed flux in JST g-band (Euclid provided)',
`kids_g` float COMMENT 'observed flux in KIDS g-band (Euclid provided)',
`kids_i` float COMMENT 'observed flux in KIDS i-band (Euclid provided)',
`kids_r` float COMMENT 'observed flux in KIDS r-band (Euclid provided)',
`kids_u` float COMMENT 'observed flux in KIDS u-band (Euclid provided)',
`lsst_g` float COMMENT 'observed flux in LSST g-band (Euclid provided)',
`lsst_i` float COMMENT 'observed flux in LSST i-band (Euclid provided)',
`lsst_r` float COMMENT 'observed flux in LSST r-band (Euclid provided)',
`lsst_u` float COMMENT 'observed flux in LSST u-band (Euclid provided)',
`lsst_y` float COMMENT 'observed flux in LSST Y-band (Euclid provided)',
`lsst_z` float COMMENT 'observed flux in LSST z-band (Euclid provided)',
`pan_starrs_i` float COMMENT 'observed flux in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z` float COMMENT 'observed flux in Pan-STARRS z-band (Euclid provided)',
`subaru_hsc_z` float COMMENT 'observed flux in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS H-band (Euclid provided)',
`2mass_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in JST g-band (Euclid provided)',
`kids_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_odonnell_ext` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1 with MW extinction',
`subaru_hsc_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS Ks-band (Euclid provided)',
`2mass_ks_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES z-band (Euclid provided)',
`blanco_decam_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT u-band (Euclid provided)',
`cfht_megacam_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_nisp_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid VIS band (Euclid provided)',
`euclid_vis_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_bp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-RP band (Euclid provided)',
`gaia_rp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in JST g-band (Euclid provided)',
`jst_jpcam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in JST g-band (Euclid provided)',
`kids_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS g-band (Euclid provided)',
`kids_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS i-band (Euclid provided)',
`kids_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS r-band (Euclid provided)',
`kids_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS u-band (Euclid provided)',
`kids_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST g-band (Euclid provided)',
`lsst_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST i-band (Euclid provided)',
`lsst_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST r-band (Euclid provided)',
`lsst_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST u-band (Euclid provided)',
`lsst_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST z-band (Euclid provided)',
`lsst_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`pan_starrs_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in sdss r-band redshifted to z=0.1',
`sdss_r01_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in sdss r-band redshifted to z=0.1',
`subaru_hsc_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`subaru_hsc_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`dominant_shape` tinyint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)',
`bulge_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`disk_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`median_major_axis` float COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)',
`scale_length` float COMMENT 'disc and bulge scalelength prior',
`bulge_fraction` float COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)',
`disk_scalelength` float COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)',
`disk_nsersic` float COMMENT 'Sersic index of the disk component',
`disk_r50` float COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678',
`bulge_r50` float COMMENT 'half light radius of the bulge',
`bulge_nsersic` float COMMENT 'Sersic index of the bulge component',
`disk_ellipticity` float COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)',
`bulge_ellipticity` float COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)',
`disk_axis_ratio` float COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`bulge_axis_ratio` float COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`inclination_angle` float COMMENT 'Galaxy inclination angle (where 0 degrees = face-on and 90 degrees = edge-on). Galaxy ellipticity for disk and bulge components are computed following the recipe in https://euclid.roe.ac.uk/projects/sgsshear/wiki/SHE-SIM',
`halo_x` float COMMENT 'halo comoving x coordinate (Mpc/h)',
`halo_y` float COMMENT 'halo comoving y coordinate (Mpc/h)',
`halo_z` float COMMENT 'halo comoving z coordinate (Mpc/h)',
`halo_vx` float COMMENT 'halo x-centre velocity (km/h)',
`halo_vy` float COMMENT 'halo y-centre velocity (km/h)',
`halo_vz` float COMMENT 'halo z-centre velocity (km/h)',
`halo_r` float COMMENT 'halo comoving distance (Mpc/h)',
`true_redshift_halo` float COMMENT 'true redshift of the host halo',
`halo_lm` float COMMENT 'log10 of the FoF halo mass in Msum/h. This halo mass is computed as the particule mass multiplied by the number of particles of the FoF halo. At low masses it is interpolated to obtain a smooth distribution of halo masses',
`halo_n_sats` int COMMENT 'number of satellite galaxies from HOD',
`n_gals` int COMMENT 'DEPRECATED',
`luminosity_r01_evolved` float COMMENT 'missing comment',
`hpix_13_nest` bigint COMMENT 'missing comment')
PARTITIONED BY (
`step` smallint COMMENT 'step number in the Flagship lensing maps')
CLUSTERED BY (
hpix_13_nest)
SORTED BY (
hpix_13_nest ASC)
INTO 128 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;
INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_1717_s PARTITION(step) SELECT halo_id, galaxy_id, kind, random_index, ra_gal, dec_gal, ra_mag_gal, dec_mag_gal, kappa, gamma1, gamma2, l_gal, b_gal, mw_extinction, x_gal, y_gal, z_gal, r_gal, true_redshift_gal, observed_redshift_gal, hpix_29_nest, vx_gal, vy_gal, vz_gal, vrad_gal, delta_r, abs_mag_r01, luminosity_r01, abs_mag_r01_evolved, sdss_r01, abs_mag_uv_dereddened, gr_restframe, color_kind, sed_ke, gr_cosmos, sed_cosmos, is_within_cosmos, cosmos_distance, ext_curve_cosmos, ebv_cosmos, log_ml_r01, log_stellar_mass, metallicity, log_sfr, logf_halpha, logf_halpha_ext, halpha_scatter, loglum_halpha, loglum_halpha_ext, logf_dummy, z_dummy, logf_halpha_model1_ext, logf_halpha_model1, loglum_halpha_model1_ext, loglum_halpha_model1, logf_halpha_model3_ext, logf_halpha_model3, loglum_halpha_model3_ext, loglum_halpha_model3, logf_hbeta_model1_ext, logf_hbeta_model1, logf_o2_model1_ext, logf_o2_model1, logf_n2_model1_ext, logf_n2_model1, logf_o3_model1_ext, logf_o3_model1, logf_s2_model1_ext, logf_s2_model1, logf_hbeta_model3_ext, logf_hbeta_model3, logf_o2_model3_ext, logf_o2_model3, logf_n2_model3_ext, logf_n2_model3, logf_o3_model3_ext, logf_o3_model3, logf_s2_model3_ext, logf_s2_model3, 2mass_h, 2mass_j, 2mass_ks, blanco_decam_g, blanco_decam_i, blanco_decam_r, blanco_decam_z, cfht_megacam_r, cfht_megacam_u, euclid_nisp_h, euclid_nisp_j, euclid_nisp_y, euclid_vis, gaia_bp, gaia_g, gaia_rp, jst_jpcam_g, kids_g, kids_i, kids_r, kids_u, lsst_g, lsst_i, lsst_r, lsst_u, lsst_y, lsst_z, pan_starrs_i, pan_starrs_z, subaru_hsc_z, 2mass_h_odonnell_ext, 2mass_j_odonnell_ext, 2mass_ks_odonnell_ext, blanco_decam_g_odonnell_ext, blanco_decam_i_odonnell_ext, blanco_decam_r_odonnell_ext, blanco_decam_z_odonnell_ext, cfht_megacam_r_odonnell_ext, cfht_megacam_u_odonnell_ext, euclid_nisp_h_odonnell_ext, euclid_nisp_j_odonnell_ext, euclid_nisp_y_odonnell_ext, euclid_vis_odonnell_ext, gaia_bp_odonnell_ext, gaia_g_odonnell_ext, gaia_rp_odonnell_ext, jst_jpcam_g_odonnell_ext, kids_g_odonnell_ext, kids_i_odonnell_ext, kids_r_odonnell_ext, kids_u_odonnell_ext, lsst_g_odonnell_ext, lsst_i_odonnell_ext, lsst_r_odonnell_ext, lsst_u_odonnell_ext, lsst_y_odonnell_ext, lsst_z_odonnell_ext, pan_starrs_i_odonnell_ext, pan_starrs_z_odonnell_ext, sdss_r01_odonnell_ext, subaru_hsc_z_odonnell_ext, 2mass_h_el_model1_odonnell_ext, 2mass_h_el_model3_odonnell_ext, 2mass_j_el_model1_odonnell_ext, 2mass_j_el_model3_odonnell_ext, 2mass_ks_el_model1_odonnell_ext, 2mass_ks_el_model3_odonnell_ext, blanco_decam_g_el_model1_odonnell_ext, blanco_decam_g_el_model3_odonnell_ext, blanco_decam_i_el_model1_odonnell_ext, blanco_decam_i_el_model3_odonnell_ext, blanco_decam_r_el_model1_odonnell_ext, blanco_decam_r_el_model3_odonnell_ext, blanco_decam_z_el_model1_odonnell_ext, blanco_decam_z_el_model3_odonnell_ext, cfht_megacam_r_el_model1_odonnell_ext, cfht_megacam_r_el_model3_odonnell_ext, cfht_megacam_u_el_model1_odonnell_ext, cfht_megacam_u_el_model3_odonnell_ext, euclid_nisp_h_el_model1_odonnell_ext, euclid_nisp_h_el_model3_odonnell_ext, euclid_nisp_j_el_model1_odonnell_ext, euclid_nisp_j_el_model3_odonnell_ext, euclid_nisp_y_el_model1_odonnell_ext, euclid_nisp_y_el_model3_odonnell_ext, euclid_vis_el_model1_odonnell_ext, euclid_vis_el_model3_odonnell_ext, gaia_bp_el_model1_odonnell_ext, gaia_bp_el_model3_odonnell_ext, gaia_g_el_model1_odonnell_ext, gaia_g_el_model3_odonnell_ext, gaia_rp_el_model1_odonnell_ext, gaia_rp_el_model3_odonnell_ext, jst_jpcam_g_el_model1_odonnell_ext, jst_jpcam_g_el_model3_odonnell_ext, kids_g_el_model1_odonnell_ext, kids_g_el_model3_odonnell_ext, kids_i_el_model1_odonnell_ext, kids_i_el_model3_odonnell_ext, kids_r_el_model1_odonnell_ext, kids_r_el_model3_odonnell_ext, kids_u_el_model1_odonnell_ext, kids_u_el_model3_odonnell_ext, lsst_g_el_model1_odonnell_ext, lsst_g_el_model3_odonnell_ext, lsst_i_el_model1_odonnell_ext, lsst_i_el_model3_odonnell_ext, lsst_r_el_model1_odonnell_ext, lsst_r_el_model3_odonnell_ext, lsst_u_el_model1_odonnell_ext, lsst_u_el_model3_odonnell_ext, lsst_y_el_model1_odonnell_ext, lsst_y_el_model3_odonnell_ext, lsst_z_el_model1_odonnell_ext, lsst_z_el_model3_odonnell_ext, pan_starrs_i_el_model1_odonnell_ext, pan_starrs_i_el_model3_odonnell_ext, pan_starrs_z_el_model1_odonnell_ext, pan_starrs_z_el_model3_odonnell_ext, sdss_r01_el_model1_odonnell_ext, sdss_r01_el_model3_odonnell_ext, subaru_hsc_z_el_model1_odonnell_ext, subaru_hsc_z_el_model3_odonnell_ext, dominant_shape, bulge_angle, disk_angle, median_major_axis, scale_length, bulge_fraction, disk_scalelength, disk_nsersic, disk_r50, bulge_r50, bulge_nsersic, disk_ellipticity, bulge_ellipticity, disk_axis_ratio, bulge_axis_ratio, inclination_angle, halo_x, halo_y, halo_z, halo_vx, halo_vy, halo_vz, halo_r, true_redshift_halo, halo_lm, halo_n_sats, n_gals, luminosity_r01_evolved, hpix_13_nest, step FROM jcarrete.flagship_mock_obsid_1717_s_pq
CREATE TABLE jcarrete.flagship_mock_obsid_1717_s_magnified_fits(
`source_id` bigint COMMENT 'from deserializer',
`halo_id` bigint COMMENT 'from deserializer',
`ra` float COMMENT 'from deserializer',
`dec` float COMMENT 'from deserializer',
`ra_mag` float COMMENT 'from deserializer',
`dec_mag` float COMMENT 'from deserializer',
`z_obs` float COMMENT 'from deserializer',
`ref_mag_abs` float COMMENT 'from deserializer',
`ref_mag` float COMMENT 'from deserializer',
`bulge_fraction` float COMMENT 'from deserializer',
`bulge_r50` float COMMENT 'from deserializer',
`disk_r50` float COMMENT 'from deserializer',
`bulge_nsersic` float COMMENT 'from deserializer',
`inclination_angle` float COMMENT 'from deserializer',
`disk_angle` float COMMENT 'from deserializer',
`kappa` float COMMENT 'from deserializer',
`gamma1` float COMMENT 'from deserializer',
`gamma2` float COMMENT 'from deserializer',
`sed_template` float COMMENT 'from deserializer',
`ext_law` smallint COMMENT 'from deserializer',
`ebv` float COMMENT 'from deserializer',
`halpha_logflam_ext` float COMMENT 'from deserializer',
`hbeta_logflam_ext` float COMMENT 'from deserializer',
`o2_logflam_ext` float COMMENT 'from deserializer',
`o3_logflam_ext` float COMMENT 'from deserializer',
`n2_logflam_ext` float COMMENT 'from deserializer',
`s2_logflam_ext` float COMMENT 'from deserializer',
`av` float COMMENT 'from deserializer',
`tu_fnu_vis` float COMMENT 'from deserializer',
`tu_fnu_y_nisp` float COMMENT 'from deserializer',
`tu_fnu_j_nisp` float COMMENT 'from deserializer',
`tu_fnu_h_nisp` float COMMENT 'from deserializer',
`tu_fnu_g_decam` float COMMENT 'from deserializer',
`tu_fnu_r_decam` float COMMENT 'from deserializer',
`tu_fnu_i_decam` float COMMENT 'from deserializer',
`tu_fnu_z_decam` float COMMENT 'from deserializer',
`tu_fnu_u_megacam` float COMMENT 'from deserializer',
`tu_fnu_r_megacam` float COMMENT 'from deserializer',
`tu_fnu_g_jpcam` float COMMENT 'from deserializer',
`tu_fnu_i_panstarrs` float COMMENT 'from deserializer',
`tu_fnu_z_panstarrs` float COMMENT 'from deserializer',
`tu_fnu_z_hsc` float COMMENT 'from deserializer',
`tu_fnu_g_gaia` float COMMENT 'from deserializer',
`tu_fnu_bp_gaia` float COMMENT 'from deserializer',
`tu_fnu_rp_gaia` float COMMENT 'from deserializer',
`tu_fnu_u_lsst` float COMMENT 'from deserializer',
`tu_fnu_g_lsst` float COMMENT 'from deserializer',
`tu_fnu_r_lsst` float COMMENT 'from deserializer',
`tu_fnu_i_lsst` float COMMENT 'from deserializer',
`tu_fnu_z_lsst` float COMMENT 'from deserializer',
`tu_fnu_y_lsst` float COMMENT 'from deserializer',
`tu_fnu_u_kids` float COMMENT 'from deserializer',
`tu_fnu_g_kids` float COMMENT 'from deserializer',
`tu_fnu_r_kids` float COMMENT 'from deserializer',
`tu_fnu_i_kids` float COMMENT 'from deserializer',
`tu_fnu_j_2mass` float COMMENT 'from deserializer',
`tu_fnu_h_2mass` float COMMENT 'from deserializer',
`tu_fnu_ks_2mass` float COMMENT 'from deserializer')
PARTITIONED BY (
`hpix_5_nest` bigint)
ROW FORMAT SERDE
'es.pic.astro.hadoop.serde.RecArraySerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'es.pic.astro.hadoop.io.BinaryOutputFormat'
;
INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_1717_s_magnified_fits PARTITION(hpix_5_nest)
SELECT CAST(((halo_id * 10000) + galaxy_id) AS bigint) AS SOURCE_ID,
CAST(halo_id AS bigint) AS HALO_ID,
CAST(ra_gal AS float) AS RA,
CAST(dec_gal AS float) AS `DEC`,
CAST(ra_mag_gal AS float) AS RA_MAG,
CAST(dec_mag_gal AS float) AS DEC_MAG,
CAST(observed_redshift_gal AS float) AS Z_OBS,
CAST(abs_mag_r01_evolved AS float) AS REF_MAG_ABS,
CAST(-2.5*log10(sdss_r01) - 48.6 AS float) AS REF_MAG,
CAST(bulge_fraction AS float) AS BULGE_FRACTION,
CAST(bulge_r50 AS float) AS BULGE_R50,
CAST(disk_r50 AS float) AS DISK_R50,
CAST(bulge_nsersic AS float) AS BULGE_NSERSIC,
CAST(inclination_angle AS float) AS INCLINATION_ANGLE,
CAST(disk_angle AS float) AS DISK_ANGLE,
CAST(kappa AS float) AS KAPPA,
CAST(gamma1 AS float) AS GAMMA1,
CAST(gamma2 AS float) AS GAMMA2,
CAST(sed_cosmos AS float) AS SED_TEMPLATE,
CAST(ROUND(ext_curve_cosmos) AS smallint) AS EXT_LAW,
CAST(ebv_cosmos AS float) AS EBV,
CAST(logf_halpha_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HALPHA_LOGFLAM_EXT,
CAST(logf_hbeta_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HBETA_LOGFLAM_EXT,
CAST(logf_o2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O2_LOGFLAM_EXT,
CAST(logf_o3_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O3_LOGFLAM_EXT,
CAST(logf_n2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS N2_LOGFLAM_EXT,
CAST(logf_s2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS S2_LOGFLAM_EXT,
CAST(mw_extinction AS float) AS AV,
CAST(euclid_vis_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_VIS,
CAST(euclid_nisp_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_NISP,
CAST(euclid_nisp_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_NISP,
CAST(euclid_nisp_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_NISP,
CAST(blanco_decam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_DECAM,
CAST(blanco_decam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_DECAM,
CAST(blanco_decam_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_DECAM,
CAST(blanco_decam_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_DECAM,
CAST(cfht_megacam_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_MEGACAM,
CAST(cfht_megacam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_MEGACAM,
CAST(jst_jpcam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_JPCAM,
CAST(pan_starrs_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_PANSTARRS,
CAST(pan_starrs_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_PANSTARRS,
CAST(subaru_hsc_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_HSC,
CAST(gaia_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_GAIA,
CAST(gaia_bp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_BP_GAIA,
CAST(gaia_rp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_RP_GAIA,
CAST(lsst_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_LSST,
CAST(lsst_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_LSST,
CAST(lsst_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_LSST,
CAST(lsst_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_LSST,
CAST(lsst_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_LSST,
CAST(lsst_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_LSST,
CAST(kids_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_KIDS,
CAST(kids_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_KIDS,
CAST(kids_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_KIDS,
CAST(kids_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_KIDS,
CAST(2mass_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_2MASS,
CAST(2mass_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_2MASS,
CAST(2mass_ks_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_KS_2MASS,
CAST( SHIFTRIGHT(hpix_29_nest, (29-5)*2) AS bigint) AS hpix_5_nest
FROM jcarrete.flagship_mock_obsid_1717_s
WHERE (logf_halpha_model3_ext > -16 OR -2.5*log10(euclid_nisp_h) - 48.6 < 26)
AND disk_axis_ratio > 0.10865
;
Y luego hay que añadir los header!
version v41!
/user/jcarrete/db/flagship_mock_obsid_1717_s_magnified_fits
/pnfs/pic.es/data/astro/euclid/disk/shared/SPV/GALAXIES/v41/
VUELVO A REGENERAR LOS FITS FILES (otra versión: v42) con el gamma1 cambiado de signo. SOBREESCRIBO LA TABLA jcarrete.flagship_mock_obsid_1717_s_magnified_fits.
Le cambio el signo a gamma1
gamma1 = -gamma1
INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_1717_s_magnified_fits PARTITION(hpix_5_nest)
SELECT CAST(((halo_id * 10000) + galaxy_id) AS bigint) AS SOURCE_ID,
CAST(halo_id AS bigint) AS HALO_ID,
CAST(ra_gal AS float) AS RA,
CAST(dec_gal AS float) AS `DEC`,
CAST(ra_mag_gal AS float) AS RA_MAG,
CAST(dec_mag_gal AS float) AS DEC_MAG,
CAST(observed_redshift_gal AS float) AS Z_OBS,
CAST(abs_mag_r01_evolved AS float) AS REF_MAG_ABS,
CAST(-2.5*log10(sdss_r01) - 48.6 AS float) AS REF_MAG,
CAST(bulge_fraction AS float) AS BULGE_FRACTION,
CAST(bulge_r50 AS float) AS BULGE_R50,
CAST(disk_r50 AS float) AS DISK_R50,
CAST(bulge_nsersic AS float) AS BULGE_NSERSIC,
CAST(inclination_angle AS float) AS INCLINATION_ANGLE,
CAST(disk_angle AS float) AS DISK_ANGLE,
CAST(kappa AS float) AS KAPPA,
CAST(-gamma1 AS float) AS GAMMA1,
CAST(gamma2 AS float) AS GAMMA2,
CAST(sed_cosmos AS float) AS SED_TEMPLATE,
CAST(ROUND(ext_curve_cosmos) AS smallint) AS EXT_LAW,
CAST(ebv_cosmos AS float) AS EBV,
CAST(logf_halpha_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HALPHA_LOGFLAM_EXT,
CAST(logf_hbeta_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HBETA_LOGFLAM_EXT,
CAST(logf_o2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O2_LOGFLAM_EXT,
CAST(logf_o3_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O3_LOGFLAM_EXT,
CAST(logf_n2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS N2_LOGFLAM_EXT,
CAST(logf_s2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS S2_LOGFLAM_EXT,
CAST(mw_extinction AS float) AS AV,
CAST(euclid_vis_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_VIS,
CAST(euclid_nisp_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_NISP,
CAST(euclid_nisp_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_NISP,
CAST(euclid_nisp_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_NISP,
CAST(blanco_decam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_DECAM,
CAST(blanco_decam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_DECAM,
CAST(blanco_decam_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_DECAM,
CAST(blanco_decam_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_DECAM,
CAST(cfht_megacam_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_MEGACAM,
CAST(cfht_megacam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_MEGACAM,
CAST(jst_jpcam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_JPCAM,
CAST(pan_starrs_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_PANSTARRS,
CAST(pan_starrs_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_PANSTARRS,
CAST(subaru_hsc_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_HSC,
CAST(gaia_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_GAIA,
CAST(gaia_bp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_BP_GAIA,
CAST(gaia_rp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_RP_GAIA,
CAST(lsst_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_LSST,
CAST(lsst_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_LSST,
CAST(lsst_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_LSST,
CAST(lsst_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_LSST,
CAST(lsst_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_LSST,
CAST(lsst_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_LSST,
CAST(kids_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_KIDS,
CAST(kids_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_KIDS,
CAST(kids_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_KIDS,
CAST(kids_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_KIDS,
CAST(2mass_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_2MASS,
CAST(2mass_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_2MASS,
CAST(2mass_ks_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_KS_2MASS,
CAST( SHIFTRIGHT(hpix_29_nest, (29-5)*2) AS bigint) AS hpix_5_nest
FROM jcarrete.flagship_mock_obsid_1717_s
WHERE (logf_halpha_model3_ext > -16 OR -2.5*log10(euclid_nisp_h) - 48.6 < 26)
AND disk_axis_ratio > 0.10865
;
Añado el header a la versión 42!
Apparently, to go from MICE shear convention to IAU, which is also in principle the Euclid convention (or Flagship), it is necessary to modify the sign of gamma1:
gamma1 = -gamma1
In addition, we agreed that the topbot parameter = 1 is not necessary and it can be removed!
There is no need to modify anything no matter the octant we are working with.
For obsid = 52929
He modificado el flux_pipeline.py y he añadido la query que toca y los siguientes comentarios:
# Voy de +DEC a -DEC: # dec_gal = - dec_gal # Quito l_gal, b_gal, mw_extinction! # Quito todos flujos que había calculado antes: # blanco_decam_i para las shapes # euclid_nisp_h para el corte # sdss_r01 que lo había puesto "por si acaso" # Filtro el obsid = 52929 #(ra_gal >= 35.45815488783862 #AND ra_gal < 61.33016111216139 #AND dec_gal >=73.5904 #AND dec_gal < 79.5904)
# The change in the shear parameters to go from MICE convention to IAU is the following: # gamma1 = -gamma1 # BUT the change from DEC > 0 to DEC < 0 is equivalent to (again) change the sign of gamma1. # So I do not do anything in the shear parameters! # This is only valid for this particular case. # Previous version (catalog) flagship_mock_obsid_1717_s need to be fixed! # Note that x, y, z are not converted to the new position!
El catálogo se ha generado en formato parquet en el path:
/user/jcarrete/data/euclid/flagship_mock_obsid_52929_s/
/software/astro/scripts/create_table_from_parquet.sh -p /hdfs/user/jcarrete/data/euclid/flagship_mock_obsid_52929_s/0.pq -t jcarrete.flagship_mock_obsid_52929_s_pq -e
CREATE EXTERNAL TABLE jcarrete.flagship_mock_obsid_52929_s_pq (
`kind` BIGINT,
`random_index` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`ra_mag_gal` DOUBLE,
`dec_mag_gal` DOUBLE,
`kappa` DOUBLE,
`gamma1` DOUBLE,
`gamma2` DOUBLE,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`observed_redshift_gal` DOUBLE,
`hpix_29_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`abs_mag_r01` DOUBLE,
`luminosity_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`sdss_r01` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` BIGINT,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha` DOUBLE,
`logf_halpha_ext` DOUBLE,
`halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_halpha_model3_ext` DOUBLE,
`loglum_halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`dominant_shape` BIGINT,
`bulge_angle` DOUBLE,
`disk_angle` DOUBLE,
`median_major_axis` DOUBLE,
`scale_length` DOUBLE,
`bulge_fraction` DOUBLE,
`disk_scalelength` DOUBLE,
`disk_nsersic` DOUBLE,
`disk_r50` DOUBLE,
`bulge_r50` DOUBLE,
`bulge_nsersic` DOUBLE,
`disk_ellipticity` DOUBLE,
`bulge_ellipticity` DOUBLE,
`disk_axis_ratio` DOUBLE,
`bulge_axis_ratio` DOUBLE,
`inclination_angle` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`luminosity_r01_evolved` DOUBLE,
`hpix_13_nest` BIGINT,
`step` BIGINT,
`blanco_decam_g` DOUBLE,
`blanco_decam_i` DOUBLE,
`blanco_decam_r` DOUBLE,
`blanco_decam_z` DOUBLE,
`subaru_hsc_z` DOUBLE,
`jst_jpcam_g` DOUBLE,
`kids_g` DOUBLE,
`kids_i` DOUBLE,
`kids_r` DOUBLE,
`kids_u` DOUBLE,
`lsst_g` DOUBLE,
`lsst_i` DOUBLE,
`lsst_r` DOUBLE,
`lsst_u` DOUBLE,
`lsst_y` DOUBLE,
`lsst_z` DOUBLE,
`cfht_megacam_r` DOUBLE,
`cfht_megacam_u` DOUBLE,
`pan_starrs_i` DOUBLE,
`pan_starrs_z` DOUBLE,
`2mass_h` DOUBLE,
`2mass_j` DOUBLE,
`2mass_ks` DOUBLE,
`gaia_bp` DOUBLE,
`gaia_g` DOUBLE,
`gaia_rp` DOUBLE,
`euclid_nisp_h` DOUBLE,
`euclid_nisp_j` DOUBLE,
`euclid_nisp_y` DOUBLE,
`euclid_vis` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`2mass_h_odonnell_ext` DOUBLE,
`2mass_j_odonnell_ext` DOUBLE,
`2mass_ks_odonnell_ext` DOUBLE,
`blanco_decam_g_odonnell_ext` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`blanco_decam_r_odonnell_ext` DOUBLE,
`blanco_decam_z_odonnell_ext` DOUBLE,
`cfht_megacam_r_odonnell_ext` DOUBLE,
`cfht_megacam_u_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`euclid_nisp_j_odonnell_ext` DOUBLE,
`euclid_nisp_y_odonnell_ext` DOUBLE,
`euclid_vis_odonnell_ext` DOUBLE,
`gaia_bp_odonnell_ext` DOUBLE,
`gaia_g_odonnell_ext` DOUBLE,
`gaia_rp_odonnell_ext` DOUBLE,
`jst_jpcam_g_odonnell_ext` DOUBLE,
`kids_g_odonnell_ext` DOUBLE,
`kids_i_odonnell_ext` DOUBLE,
`kids_r_odonnell_ext` DOUBLE,
`kids_u_odonnell_ext` DOUBLE,
`lsst_g_odonnell_ext` DOUBLE,
`lsst_i_odonnell_ext` DOUBLE,
`lsst_r_odonnell_ext` DOUBLE,
`lsst_u_odonnell_ext` DOUBLE,
`lsst_y_odonnell_ext` DOUBLE,
`lsst_z_odonnell_ext` DOUBLE,
`pan_starrs_i_odonnell_ext` DOUBLE,
`pan_starrs_z_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`subaru_hsc_z_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model3_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model1_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model3_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model1_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model3_odonnell_ext` DOUBLE,
`kids_g_el_model1_odonnell_ext` DOUBLE,
`kids_g_el_model3_odonnell_ext` DOUBLE,
`kids_i_el_model1_odonnell_ext` DOUBLE,
`kids_i_el_model3_odonnell_ext` DOUBLE,
`kids_r_el_model1_odonnell_ext` DOUBLE,
`kids_r_el_model3_odonnell_ext` DOUBLE,
`kids_u_el_model1_odonnell_ext` DOUBLE,
`kids_u_el_model3_odonnell_ext` DOUBLE,
`lsst_g_el_model1_odonnell_ext` DOUBLE,
`lsst_g_el_model3_odonnell_ext` DOUBLE,
`lsst_i_el_model1_odonnell_ext` DOUBLE,
`lsst_i_el_model3_odonnell_ext` DOUBLE,
`lsst_r_el_model1_odonnell_ext` DOUBLE,
`lsst_r_el_model3_odonnell_ext` DOUBLE,
`lsst_u_el_model1_odonnell_ext` DOUBLE,
`lsst_u_el_model3_odonnell_ext` DOUBLE,
`lsst_y_el_model1_odonnell_ext` DOUBLE,
`lsst_y_el_model3_odonnell_ext` DOUBLE,
`lsst_z_el_model1_odonnell_ext` DOUBLE,
`lsst_z_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model3_odonnell_ext` DOUBLE,
`2mass_h_el_model1_odonnell_ext` DOUBLE,
`2mass_h_el_model3_odonnell_ext` DOUBLE,
`2mass_j_el_model1_odonnell_ext` DOUBLE,
`2mass_j_el_model3_odonnell_ext` DOUBLE,
`2mass_ks_el_model1_odonnell_ext` DOUBLE,
`2mass_ks_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`gaia_bp_el_model1_odonnell_ext` DOUBLE,
`gaia_bp_el_model3_odonnell_ext` DOUBLE,
`gaia_g_el_model1_odonnell_ext` DOUBLE,
`gaia_g_el_model3_odonnell_ext` DOUBLE,
`gaia_rp_el_model1_odonnell_ext` DOUBLE,
`gaia_rp_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model3_odonnell_ext` DOUBLE,
`euclid_vis_el_model1_odonnell_ext` DOUBLE,
`euclid_vis_el_model3_odonnell_ext` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
STORED AS PARQUET
LOCATION '/user/jcarrete/data/euclid/flagship_mock_obsid_52929_s'
;
CREATE TABLE jcarrete.flagship_mock_obsid_52929_s(
`halo_id` bigint COMMENT 'unique halo id coming from the Flagship dark matter halo catalog',
`galaxy_id` int COMMENT 'combined with the unique_halo_id they uniquely identify each galaxy',
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',
`random_index` float COMMENT 'random number [0 - 1) for subsampling',
`ra_gal` float COMMENT 'galaxy right ascension (degrees)',
`dec_gal` float COMMENT 'galaxy declination (degrees)',
`ra_mag_gal` float COMMENT 'galaxy magnified right ascension (degree)',
`dec_mag_gal` float COMMENT 'galaxy magnified declination (degree)',
`kappa` float COMMENT 'convergence',
`gamma1` float COMMENT 'shear',
`gamma2` float COMMENT 'shear',
`l_gal` float COMMENT 'galactic longitude (degrees)',
`b_gal` float COMMENT 'galactic latitude (degrees)',
`mw_extinction` float COMMENT 'EBV from file: HFI_CompMap_ThermalDustModel_2048_R1.20.fits',
`x_gal` float COMMENT 'galaxy comoving x coordinate (Mpc/h)',
`y_gal` float COMMENT 'galaxy comoving y coordinate (Mpc/h)',
`z_gal` float COMMENT 'galaxy comoving z coordinate (Mpc/h)',
`r_gal` float COMMENT 'galaxy comoving distance (Mpc/h)',
`true_redshift_gal` float COMMENT 'true galaxy redshift',
`observed_redshift_gal` float COMMENT 'observed galaxy redshift (including peculiar velocity)',
`hpix_29_nest` bigint COMMENT 'healpix pixel id using order = 29 in NESTED format',
`vx_gal` float COMMENT 'physical (peculiar) velocity in x-component (km/s)',
`vy_gal` float COMMENT 'physical (peculiar) velocity in y-component (km/s)',
`vz_gal` float COMMENT 'physical (peculiar) velocity in z-component (km/s)',
`vrad_gal` float COMMENT 'missing comment',
`delta_r` float COMMENT 'missing comment',
`abs_mag_r01` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h)',
`luminosity_r01` float COMMENT 'luminosity in the 0.1-r-band',
`abs_mag_r01_evolved` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h) including MICECATv2.0 magnitude evolution',
`sdss_r01` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1',
`abs_mag_uv_dereddened` float COMMENT 'absolute magnitude ultraviolet',
`gr_restframe` float COMMENT '(g-r) rest-frame at z=0',
`color_kind` tinyint COMMENT 'galaxy color kind: 0=red sequence 1=green valley 2=blue cloud',
`sed_ke` float COMMENT 'K-correction for each SED',
`gr_cosmos` float COMMENT 'interpolated COSMOS g-r color',
`sed_cosmos` float COMMENT 'interpolated basic COSMOS SED [0 30]',
`is_within_cosmos` boolean COMMENT 'missing comment',
`cosmos_distance` float COMMENT 'missing comment',
`ext_curve_cosmos` tinyint COMMENT 'COSMOS extinction curve [0-4]',
`ebv_cosmos` float COMMENT 'interpolated extinction strength for COSMOS extinction curve',
`log_ml_r01` float COMMENT 'logarithm of stellar_mass/luminosity ratio in the r01 SDSS band',
`log_stellar_mass` float COMMENT 'logarithm of the stellar mass (Msun/h)',
`metallicity` float COMMENT 'oxygen metallicity in units 12+logO/H',
`log_sfr` float COMMENT 'logarithm of star formation rate in (Msun/h)/year',
`logf_halpha` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: DOES NOT include extinction)',
`logf_halpha_ext` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: includes extinction)',
`halpha_scatter` float COMMENT 'missing comment',
`loglum_halpha` float COMMENT 'DEPRECATED logarithm of Halpha luminosity (erg/s/h^2) using empirical relation with UV magnitude (DOES NOT include extinction)',
`loglum_halpha_ext` float COMMENT 'missing comment',
`logf_dummy` float COMMENT 'missing comment',
`z_dummy` float COMMENT 'missing comment',
`logf_halpha_model1_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: includes extinction)',
`logf_halpha_model1` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: DOES NOT include extinction)',
`loglum_halpha_model1_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (includes extinction)',
`loglum_halpha_model1` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (DOES NOT include extinction)',
`logf_halpha_model3_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: includes extinction)',
`logf_halpha_model3` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: DOES NOT include extinction)',
`loglum_halpha_model3_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (includes extinction)',
`loglum_halpha_model3` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (DOES NOT include extinction)',
`logf_hbeta_model1_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_hbeta_model1` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_n2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_n2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o3_model1_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o3_model1` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_s2_model1_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_s2_model1` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_hbeta_model3_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_hbeta_model3` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_n2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_n2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o3_model3_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o3_model3` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_s2_model3_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_s2_model3` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`2mass_h` float COMMENT 'observed flux in 2MASS H-band (Euclid provided)',
`2mass_j` float COMMENT 'observed flux in 2MASS J-band (Euclid provided)',
`2mass_ks` float COMMENT 'observed flux in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g` float COMMENT 'observed flux in DES g-band (Euclid provided)',
`blanco_decam_i` float COMMENT 'observed flux in DES i-band (Euclid provided)',
`blanco_decam_r` float COMMENT 'observed flux in DES r-band (Euclid provided)',
`blanco_decam_z` float COMMENT 'observed flux in DES z-band (Euclid provided)',
`cfht_megacam_r` float COMMENT 'observed flux in CFHT r-band (Euclid provided)',
`cfht_megacam_u` float COMMENT 'observed flux in CFHT u-band (Euclid provided)',
`euclid_nisp_h` float COMMENT 'observed flux in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j` float COMMENT 'observed flux in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y` float COMMENT 'observed flux in Euclid NISP-Y band (Euclid provided)',
`euclid_vis` float COMMENT 'observed flux in Euclid VIS band (Euclid provided)',
`gaia_bp` float COMMENT 'observed flux in Gaia-BP band (Euclid provided)',
`gaia_g` float COMMENT 'observed flux in Gaia-G band (Euclid provided)',
`gaia_rp` float COMMENT 'observed flux in Gaia-RP band (Euclid provided)',
`jst_jpcam_g` float COMMENT 'observed flux in JST g-band (Euclid provided)',
`kids_g` float COMMENT 'observed flux in KIDS g-band (Euclid provided)',
`kids_i` float COMMENT 'observed flux in KIDS i-band (Euclid provided)',
`kids_r` float COMMENT 'observed flux in KIDS r-band (Euclid provided)',
`kids_u` float COMMENT 'observed flux in KIDS u-band (Euclid provided)',
`lsst_g` float COMMENT 'observed flux in LSST g-band (Euclid provided)',
`lsst_i` float COMMENT 'observed flux in LSST i-band (Euclid provided)',
`lsst_r` float COMMENT 'observed flux in LSST r-band (Euclid provided)',
`lsst_u` float COMMENT 'observed flux in LSST u-band (Euclid provided)',
`lsst_y` float COMMENT 'observed flux in LSST Y-band (Euclid provided)',
`lsst_z` float COMMENT 'observed flux in LSST z-band (Euclid provided)',
`pan_starrs_i` float COMMENT 'observed flux in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z` float COMMENT 'observed flux in Pan-STARRS z-band (Euclid provided)',
`subaru_hsc_z` float COMMENT 'observed flux in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS H-band (Euclid provided)',
`2mass_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in JST g-band (Euclid provided)',
`kids_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_odonnell_ext` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1 with MW extinction',
`subaru_hsc_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS Ks-band (Euclid provided)',
`2mass_ks_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES z-band (Euclid provided)',
`blanco_decam_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT u-band (Euclid provided)',
`cfht_megacam_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_nisp_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid VIS band (Euclid provided)',
`euclid_vis_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_bp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-RP band (Euclid provided)',
`gaia_rp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in JST g-band (Euclid provided)',
`jst_jpcam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in JST g-band (Euclid provided)',
`kids_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS g-band (Euclid provided)',
`kids_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS i-band (Euclid provided)',
`kids_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS r-band (Euclid provided)',
`kids_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS u-band (Euclid provided)',
`kids_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST g-band (Euclid provided)',
`lsst_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST i-band (Euclid provided)',
`lsst_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST r-band (Euclid provided)',
`lsst_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST u-band (Euclid provided)',
`lsst_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST z-band (Euclid provided)',
`lsst_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`pan_starrs_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in sdss r-band redshifted to z=0.1',
`sdss_r01_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in sdss r-band redshifted to z=0.1',
`subaru_hsc_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`subaru_hsc_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`dominant_shape` tinyint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)',
`bulge_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`disk_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`median_major_axis` float COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)',
`scale_length` float COMMENT 'disc and bulge scalelength prior',
`bulge_fraction` float COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)',
`disk_scalelength` float COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)',
`disk_nsersic` float COMMENT 'Sersic index of the disk component',
`disk_r50` float COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678',
`bulge_r50` float COMMENT 'half light radius of the bulge',
`bulge_nsersic` float COMMENT 'Sersic index of the bulge component',
`disk_ellipticity` float COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)',
`bulge_ellipticity` float COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)',
`disk_axis_ratio` float COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`bulge_axis_ratio` float COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`inclination_angle` float COMMENT 'Galaxy inclination angle (where 0 degrees = face-on and 90 degrees = edge-on). Galaxy ellipticity for disk and bulge components are computed following the recipe in https://euclid.roe.ac.uk/projects/sgsshear/wiki/SHE-SIM',
`halo_x` float COMMENT 'halo comoving x coordinate (Mpc/h)',
`halo_y` float COMMENT 'halo comoving y coordinate (Mpc/h)',
`halo_z` float COMMENT 'halo comoving z coordinate (Mpc/h)',
`halo_vx` float COMMENT 'halo x-centre velocity (km/h)',
`halo_vy` float COMMENT 'halo y-centre velocity (km/h)',
`halo_vz` float COMMENT 'halo z-centre velocity (km/h)',
`halo_r` float COMMENT 'halo comoving distance (Mpc/h)',
`true_redshift_halo` float COMMENT 'true redshift of the host halo',
`halo_lm` float COMMENT 'log10 of the FoF halo mass in Msum/h. This halo mass is computed as the particule mass multiplied by the number of particles of the FoF halo. At low masses it is interpolated to obtain a smooth distribution of halo masses',
`halo_n_sats` int COMMENT 'number of satellite galaxies from HOD',
`n_gals` int COMMENT 'DEPRECATED',
`luminosity_r01_evolved` float COMMENT 'missing comment',
`hpix_13_nest` bigint COMMENT 'missing comment')
PARTITIONED BY (
`step` smallint COMMENT 'step number in the Flagship lensing maps')
CLUSTERED BY (
hpix_13_nest)
SORTED BY (
hpix_13_nest ASC)
INTO 128 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;
INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_52929_s PARTITION(step) SELECT halo_id, galaxy_id, kind, random_index, ra_gal, dec_gal, ra_mag_gal, dec_mag_gal, kappa, gamma1, gamma2, l_gal, b_gal, mw_extinction, x_gal, y_gal, z_gal, r_gal, true_redshift_gal, observed_redshift_gal, hpix_29_nest, vx_gal, vy_gal, vz_gal, vrad_gal, delta_r, abs_mag_r01, luminosity_r01, abs_mag_r01_evolved, sdss_r01, abs_mag_uv_dereddened, gr_restframe, color_kind, sed_ke, gr_cosmos, sed_cosmos, is_within_cosmos, cosmos_distance, ext_curve_cosmos, ebv_cosmos, log_ml_r01, log_stellar_mass, metallicity, log_sfr, logf_halpha, logf_halpha_ext, halpha_scatter, loglum_halpha, loglum_halpha_ext, logf_dummy, z_dummy, logf_halpha_model1_ext, logf_halpha_model1, loglum_halpha_model1_ext, loglum_halpha_model1, logf_halpha_model3_ext, logf_halpha_model3, loglum_halpha_model3_ext, loglum_halpha_model3, logf_hbeta_model1_ext, logf_hbeta_model1, logf_o2_model1_ext, logf_o2_model1, logf_n2_model1_ext, logf_n2_model1, logf_o3_model1_ext, logf_o3_model1, logf_s2_model1_ext, logf_s2_model1, logf_hbeta_model3_ext, logf_hbeta_model3, logf_o2_model3_ext, logf_o2_model3, logf_n2_model3_ext, logf_n2_model3, logf_o3_model3_ext, logf_o3_model3, logf_s2_model3_ext, logf_s2_model3, 2mass_h, 2mass_j, 2mass_ks, blanco_decam_g, blanco_decam_i, blanco_decam_r, blanco_decam_z, cfht_megacam_r, cfht_megacam_u, euclid_nisp_h, euclid_nisp_j, euclid_nisp_y, euclid_vis, gaia_bp, gaia_g, gaia_rp, jst_jpcam_g, kids_g, kids_i, kids_r, kids_u, lsst_g, lsst_i, lsst_r, lsst_u, lsst_y, lsst_z, pan_starrs_i, pan_starrs_z, subaru_hsc_z, 2mass_h_odonnell_ext, 2mass_j_odonnell_ext, 2mass_ks_odonnell_ext, blanco_decam_g_odonnell_ext, blanco_decam_i_odonnell_ext, blanco_decam_r_odonnell_ext, blanco_decam_z_odonnell_ext, cfht_megacam_r_odonnell_ext, cfht_megacam_u_odonnell_ext, euclid_nisp_h_odonnell_ext, euclid_nisp_j_odonnell_ext, euclid_nisp_y_odonnell_ext, euclid_vis_odonnell_ext, gaia_bp_odonnell_ext, gaia_g_odonnell_ext, gaia_rp_odonnell_ext, jst_jpcam_g_odonnell_ext, kids_g_odonnell_ext, kids_i_odonnell_ext, kids_r_odonnell_ext, kids_u_odonnell_ext, lsst_g_odonnell_ext, lsst_i_odonnell_ext, lsst_r_odonnell_ext, lsst_u_odonnell_ext, lsst_y_odonnell_ext, lsst_z_odonnell_ext, pan_starrs_i_odonnell_ext, pan_starrs_z_odonnell_ext, sdss_r01_odonnell_ext, subaru_hsc_z_odonnell_ext, 2mass_h_el_model1_odonnell_ext, 2mass_h_el_model3_odonnell_ext, 2mass_j_el_model1_odonnell_ext, 2mass_j_el_model3_odonnell_ext, 2mass_ks_el_model1_odonnell_ext, 2mass_ks_el_model3_odonnell_ext, blanco_decam_g_el_model1_odonnell_ext, blanco_decam_g_el_model3_odonnell_ext, blanco_decam_i_el_model1_odonnell_ext, blanco_decam_i_el_model3_odonnell_ext, blanco_decam_r_el_model1_odonnell_ext, blanco_decam_r_el_model3_odonnell_ext, blanco_decam_z_el_model1_odonnell_ext, blanco_decam_z_el_model3_odonnell_ext, cfht_megacam_r_el_model1_odonnell_ext, cfht_megacam_r_el_model3_odonnell_ext, cfht_megacam_u_el_model1_odonnell_ext, cfht_megacam_u_el_model3_odonnell_ext, euclid_nisp_h_el_model1_odonnell_ext, euclid_nisp_h_el_model3_odonnell_ext, euclid_nisp_j_el_model1_odonnell_ext, euclid_nisp_j_el_model3_odonnell_ext, euclid_nisp_y_el_model1_odonnell_ext, euclid_nisp_y_el_model3_odonnell_ext, euclid_vis_el_model1_odonnell_ext, euclid_vis_el_model3_odonnell_ext, gaia_bp_el_model1_odonnell_ext, gaia_bp_el_model3_odonnell_ext, gaia_g_el_model1_odonnell_ext, gaia_g_el_model3_odonnell_ext, gaia_rp_el_model1_odonnell_ext, gaia_rp_el_model3_odonnell_ext, jst_jpcam_g_el_model1_odonnell_ext, jst_jpcam_g_el_model3_odonnell_ext, kids_g_el_model1_odonnell_ext, kids_g_el_model3_odonnell_ext, kids_i_el_model1_odonnell_ext, kids_i_el_model3_odonnell_ext, kids_r_el_model1_odonnell_ext, kids_r_el_model3_odonnell_ext, kids_u_el_model1_odonnell_ext, kids_u_el_model3_odonnell_ext, lsst_g_el_model1_odonnell_ext, lsst_g_el_model3_odonnell_ext, lsst_i_el_model1_odonnell_ext, lsst_i_el_model3_odonnell_ext, lsst_r_el_model1_odonnell_ext, lsst_r_el_model3_odonnell_ext, lsst_u_el_model1_odonnell_ext, lsst_u_el_model3_odonnell_ext, lsst_y_el_model1_odonnell_ext, lsst_y_el_model3_odonnell_ext, lsst_z_el_model1_odonnell_ext, lsst_z_el_model3_odonnell_ext, pan_starrs_i_el_model1_odonnell_ext, pan_starrs_i_el_model3_odonnell_ext, pan_starrs_z_el_model1_odonnell_ext, pan_starrs_z_el_model3_odonnell_ext, sdss_r01_el_model1_odonnell_ext, sdss_r01_el_model3_odonnell_ext, subaru_hsc_z_el_model1_odonnell_ext, subaru_hsc_z_el_model3_odonnell_ext, dominant_shape, bulge_angle, disk_angle, median_major_axis, scale_length, bulge_fraction, disk_scalelength, disk_nsersic, disk_r50, bulge_r50, bulge_nsersic, disk_ellipticity, bulge_ellipticity, disk_axis_ratio, bulge_axis_ratio, inclination_angle, halo_x, halo_y, halo_z, halo_vx, halo_vy, halo_vz, halo_r, true_redshift_halo, halo_lm, halo_n_sats, n_gals, luminosity_r01_evolved, hpix_13_nest, step FROM jcarrete.flagship_mock_obsid_52929_s_pq
CREATE TABLE jcarrete.flagship_mock_obsid_52929_s_magnified_fits(
`source_id` bigint COMMENT 'from deserializer',
`halo_id` bigint COMMENT 'from deserializer',
`ra` float COMMENT 'from deserializer',
`dec` float COMMENT 'from deserializer',
`ra_mag` float COMMENT 'from deserializer',
`dec_mag` float COMMENT 'from deserializer',
`z_obs` float COMMENT 'from deserializer',
`ref_mag_abs` float COMMENT 'from deserializer',
`ref_mag` float COMMENT 'from deserializer',
`bulge_fraction` float COMMENT 'from deserializer',
`bulge_r50` float COMMENT 'from deserializer',
`disk_r50` float COMMENT 'from deserializer',
`bulge_nsersic` float COMMENT 'from deserializer',
`inclination_angle` float COMMENT 'from deserializer',
`disk_angle` float COMMENT 'from deserializer',
`kappa` float COMMENT 'from deserializer',
`gamma1` float COMMENT 'from deserializer',
`gamma2` float COMMENT 'from deserializer',
`sed_template` float COMMENT 'from deserializer',
`ext_law` smallint COMMENT 'from deserializer',
`ebv` float COMMENT 'from deserializer',
`halpha_logflam_ext` float COMMENT 'from deserializer',
`hbeta_logflam_ext` float COMMENT 'from deserializer',
`o2_logflam_ext` float COMMENT 'from deserializer',
`o3_logflam_ext` float COMMENT 'from deserializer',
`n2_logflam_ext` float COMMENT 'from deserializer',
`s2_logflam_ext` float COMMENT 'from deserializer',
`av` float COMMENT 'from deserializer',
`tu_fnu_vis` float COMMENT 'from deserializer',
`tu_fnu_y_nisp` float COMMENT 'from deserializer',
`tu_fnu_j_nisp` float COMMENT 'from deserializer',
`tu_fnu_h_nisp` float COMMENT 'from deserializer',
`tu_fnu_g_decam` float COMMENT 'from deserializer',
`tu_fnu_r_decam` float COMMENT 'from deserializer',
`tu_fnu_i_decam` float COMMENT 'from deserializer',
`tu_fnu_z_decam` float COMMENT 'from deserializer',
`tu_fnu_u_megacam` float COMMENT 'from deserializer',
`tu_fnu_r_megacam` float COMMENT 'from deserializer',
`tu_fnu_g_jpcam` float COMMENT 'from deserializer',
`tu_fnu_i_panstarrs` float COMMENT 'from deserializer',
`tu_fnu_z_panstarrs` float COMMENT 'from deserializer',
`tu_fnu_z_hsc` float COMMENT 'from deserializer',
`tu_fnu_g_gaia` float COMMENT 'from deserializer',
`tu_fnu_bp_gaia` float COMMENT 'from deserializer',
`tu_fnu_rp_gaia` float COMMENT 'from deserializer',
`tu_fnu_u_lsst` float COMMENT 'from deserializer',
`tu_fnu_g_lsst` float COMMENT 'from deserializer',
`tu_fnu_r_lsst` float COMMENT 'from deserializer',
`tu_fnu_i_lsst` float COMMENT 'from deserializer',
`tu_fnu_z_lsst` float COMMENT 'from deserializer',
`tu_fnu_y_lsst` float COMMENT 'from deserializer',
`tu_fnu_u_kids` float COMMENT 'from deserializer',
`tu_fnu_g_kids` float COMMENT 'from deserializer',
`tu_fnu_r_kids` float COMMENT 'from deserializer',
`tu_fnu_i_kids` float COMMENT 'from deserializer',
`tu_fnu_j_2mass` float COMMENT 'from deserializer',
`tu_fnu_h_2mass` float COMMENT 'from deserializer',
`tu_fnu_ks_2mass` float COMMENT 'from deserializer')
PARTITIONED BY (
`hpix_5_nest` bigint)
ROW FORMAT SERDE
'es.pic.astro.hadoop.serde.RecArraySerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'es.pic.astro.hadoop.io.BinaryOutputFormat'
;
INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_52929_s_magnified_fits PARTITION(hpix_5_nest)
SELECT CAST(((halo_id * 10000) + galaxy_id) AS bigint) AS SOURCE_ID,
CAST(halo_id AS bigint) AS HALO_ID,
CAST(ra_gal AS float) AS RA,
CAST(dec_gal AS float) AS `DEC`,
CAST(ra_mag_gal AS float) AS RA_MAG,
CAST(dec_mag_gal AS float) AS DEC_MAG,
CAST(observed_redshift_gal AS float) AS Z_OBS,
CAST(abs_mag_r01_evolved AS float) AS REF_MAG_ABS,
CAST(-2.5*log10(sdss_r01) - 48.6 AS float) AS REF_MAG,
CAST(bulge_fraction AS float) AS BULGE_FRACTION,
CAST(bulge_r50 AS float) AS BULGE_R50,
CAST(disk_r50 AS float) AS DISK_R50,
CAST(bulge_nsersic AS float) AS BULGE_NSERSIC,
CAST(inclination_angle AS float) AS INCLINATION_ANGLE,
CAST(disk_angle AS float) AS DISK_ANGLE,
CAST(kappa AS float) AS KAPPA,
CAST(gamma1 AS float) AS GAMMA1,
CAST(gamma2 AS float) AS GAMMA2,
CAST(sed_cosmos AS float) AS SED_TEMPLATE,
CAST(ROUND(ext_curve_cosmos) AS smallint) AS EXT_LAW,
CAST(ebv_cosmos AS float) AS EBV,
CAST(logf_halpha_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HALPHA_LOGFLAM_EXT,
CAST(logf_hbeta_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HBETA_LOGFLAM_EXT,
CAST(logf_o2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O2_LOGFLAM_EXT,
CAST(logf_o3_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O3_LOGFLAM_EXT,
CAST(logf_n2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS N2_LOGFLAM_EXT,
CAST(logf_s2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS S2_LOGFLAM_EXT,
CAST(mw_extinction AS float) AS AV,
CAST(euclid_vis_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_VIS,
CAST(euclid_nisp_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_NISP,
CAST(euclid_nisp_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_NISP,
CAST(euclid_nisp_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_NISP,
CAST(blanco_decam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_DECAM,
CAST(blanco_decam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_DECAM,
CAST(blanco_decam_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_DECAM,
CAST(blanco_decam_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_DECAM,
CAST(cfht_megacam_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_MEGACAM,
CAST(cfht_megacam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_MEGACAM,
CAST(jst_jpcam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_JPCAM,
CAST(pan_starrs_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_PANSTARRS,
CAST(pan_starrs_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_PANSTARRS,
CAST(subaru_hsc_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_HSC,
CAST(gaia_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_GAIA,
CAST(gaia_bp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_BP_GAIA,
CAST(gaia_rp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_RP_GAIA,
CAST(lsst_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_LSST,
CAST(lsst_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_LSST,
CAST(lsst_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_LSST,
CAST(lsst_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_LSST,
CAST(lsst_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_LSST,
CAST(lsst_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_LSST,
CAST(kids_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_KIDS,
CAST(kids_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_KIDS,
CAST(kids_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_KIDS,
CAST(kids_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_KIDS,
CAST(2mass_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_2MASS,
CAST(2mass_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_2MASS,
CAST(2mass_ks_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_KS_2MASS,
CAST( SHIFTRIGHT(hpix_29_nest, (29-5)*2) AS bigint) AS hpix_5_nest
FROM jcarrete.flagship_mock_obsid_52929_s
WHERE (logf_halpha_model3_ext > -16 OR -2.5*log10(euclid_nisp_h) - 48.6 < 26)
AND disk_axis_ratio > 0.10865
;
Añadimos los header con el notebook: add_fits_header_to_hive_table.py
Lo hago con la version 43. No pongo la 42 porque la voy a usar para corregir del lensing la v41 (gamma1 = -gamma1)
Los dos catálogos anteriores tienen un bug en las shapes!!!!
Vuelvo a calcular las shapes pero solo para la 1717.
Genero un catalog con la pipeline shape_pipeline.py que se llama: flagship_mock_obsid_1717_s_v2
Esta es la query que uso (elimino los campos de las shapes, que son los que voy a recalcular!):
SELECT halo_id, galaxy_id, kind, random_index, ra_gal, dec_gal, ra_mag_gal, dec_mag_gal, kappa, gamma1, gamma2, l_gal, b_gal, mw_extinction, x_gal, y_gal, z_gal, r_gal, true_redshift_gal, observed_redshift_gal, hpix_29_nest, vx_gal, vy_gal, vz_gal, vrad_gal, delta_r, abs_mag_r01, luminosity_r01, abs_mag_r01_evolved, sdss_r01, abs_mag_uv_dereddened, gr_restframe, color_kind, sed_ke, gr_cosmos, sed_cosmos, is_within_cosmos, cosmos_distance, ext_curve_cosmos, ebv_cosmos, log_ml_r01, log_stellar_mass, metallicity, log_sfr, logf_halpha, logf_halpha_ext, halpha_scatter, loglum_halpha, loglum_halpha_ext, logf_dummy, z_dummy, logf_halpha_model1_ext, logf_halpha_model1, loglum_halpha_model1_ext, loglum_halpha_model1, logf_halpha_model3_ext, logf_halpha_model3, loglum_halpha_model3_ext, loglum_halpha_model3, logf_hbeta_model1_ext, logf_hbeta_model1, logf_o2_model1_ext, logf_o2_model1, logf_n2_model1_ext, logf_n2_model1, logf_o3_model1_ext, logf_o3_model1, logf_s2_model1_ext, logf_s2_model1, logf_hbeta_model3_ext, logf_hbeta_model3, logf_o2_model3_ext, logf_o2_model3, logf_n2_model3_ext, logf_n2_model3, logf_o3_model3_ext, logf_o3_model3, logf_s2_model3_ext, logf_s2_model3, 2mass_h, 2mass_j, 2mass_ks, blanco_decam_g, blanco_decam_i, blanco_decam_r, blanco_decam_z, cfht_megacam_r, cfht_megacam_u, euclid_nisp_h, euclid_nisp_j, euclid_nisp_y, euclid_vis, gaia_bp, gaia_g, gaia_rp, jst_jpcam_g, kids_g, kids_i, kids_r, kids_u, lsst_g, lsst_i, lsst_r, lsst_u, lsst_y, lsst_z, pan_starrs_i, pan_starrs_z, subaru_hsc_z, 2mass_h_odonnell_ext, 2mass_j_odonnell_ext, 2mass_ks_odonnell_ext, blanco_decam_g_odonnell_ext, blanco_decam_i_odonnell_ext, blanco_decam_r_odonnell_ext, blanco_decam_z_odonnell_ext, cfht_megacam_r_odonnell_ext, cfht_megacam_u_odonnell_ext, euclid_nisp_h_odonnell_ext, euclid_nisp_j_odonnell_ext, euclid_nisp_y_odonnell_ext, euclid_vis_odonnell_ext, gaia_bp_odonnell_ext, gaia_g_odonnell_ext, gaia_rp_odonnell_ext, jst_jpcam_g_odonnell_ext, kids_g_odonnell_ext, kids_i_odonnell_ext, kids_r_odonnell_ext, kids_u_odonnell_ext, lsst_g_odonnell_ext, lsst_i_odonnell_ext, lsst_r_odonnell_ext, lsst_u_odonnell_ext, lsst_y_odonnell_ext, lsst_z_odonnell_ext, pan_starrs_i_odonnell_ext, pan_starrs_z_odonnell_ext, sdss_r01_odonnell_ext, subaru_hsc_z_odonnell_ext, 2mass_h_el_model1_odonnell_ext, 2mass_h_el_model3_odonnell_ext, 2mass_j_el_model1_odonnell_ext, 2mass_j_el_model3_odonnell_ext, 2mass_ks_el_model1_odonnell_ext, 2mass_ks_el_model3_odonnell_ext, blanco_decam_g_el_model1_odonnell_ext, blanco_decam_g_el_model3_odonnell_ext, blanco_decam_i_el_model1_odonnell_ext, blanco_decam_i_el_model3_odonnell_ext, blanco_decam_r_el_model1_odonnell_ext, blanco_decam_r_el_model3_odonnell_ext, blanco_decam_z_el_model1_odonnell_ext, blanco_decam_z_el_model3_odonnell_ext, cfht_megacam_r_el_model1_odonnell_ext, cfht_megacam_r_el_model3_odonnell_ext, cfht_megacam_u_el_model1_odonnell_ext, cfht_megacam_u_el_model3_odonnell_ext, euclid_nisp_h_el_model1_odonnell_ext, euclid_nisp_h_el_model3_odonnell_ext, euclid_nisp_j_el_model1_odonnell_ext, euclid_nisp_j_el_model3_odonnell_ext, euclid_nisp_y_el_model1_odonnell_ext, euclid_nisp_y_el_model3_odonnell_ext, euclid_vis_el_model1_odonnell_ext, euclid_vis_el_model3_odonnell_ext, gaia_bp_el_model1_odonnell_ext, gaia_bp_el_model3_odonnell_ext, gaia_g_el_model1_odonnell_ext, gaia_g_el_model3_odonnell_ext, gaia_rp_el_model1_odonnell_ext, gaia_rp_el_model3_odonnell_ext, jst_jpcam_g_el_model1_odonnell_ext, jst_jpcam_g_el_model3_odonnell_ext, kids_g_el_model1_odonnell_ext, kids_g_el_model3_odonnell_ext, kids_i_el_model1_odonnell_ext, kids_i_el_model3_odonnell_ext, kids_r_el_model1_odonnell_ext, kids_r_el_model3_odonnell_ext, kids_u_el_model1_odonnell_ext, kids_u_el_model3_odonnell_ext, lsst_g_el_model1_odonnell_ext, lsst_g_el_model3_odonnell_ext, lsst_i_el_model1_odonnell_ext, lsst_i_el_model3_odonnell_ext, lsst_r_el_model1_odonnell_ext, lsst_r_el_model3_odonnell_ext, lsst_u_el_model1_odonnell_ext, lsst_u_el_model3_odonnell_ext, lsst_y_el_model1_odonnell_ext, lsst_y_el_model3_odonnell_ext, lsst_z_el_model1_odonnell_ext, lsst_z_el_model3_odonnell_ext, pan_starrs_i_el_model1_odonnell_ext, pan_starrs_i_el_model3_odonnell_ext, pan_starrs_z_el_model1_odonnell_ext, pan_starrs_z_el_model3_odonnell_ext, sdss_r01_el_model1_odonnell_ext, sdss_r01_el_model3_odonnell_ext, subaru_hsc_z_el_model1_odonnell_ext, subaru_hsc_z_el_model3_odonnell_ext, halo_x, halo_y, halo_z, halo_vx, halo_vy, halo_vz, halo_r, true_redshift_halo, halo_lm, halo_n_sats, n_gals, luminosity_r01_evolved, hpix_13_nest, step FROM jcarrete.flagship_mock_obsid_1717_s
Defino la tabla externa en parquet format del catálogo usando el script:
/software/astro/scripts/create_table_from_parquet.sh -p /hdfs/user/jcarrete/data/euclid/flagship_mock_obsid_1717_s_v2/0.pq -t jcarrete.flagship_mock_obsid_1717_s_v2_pq -e
CREATE EXTERNAL TABLE jcarrete.flagship_mock_obsid_1717_s_v2_pq (
`kind` BIGINT,
`random_index` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`ra_mag_gal` DOUBLE,
`dec_mag_gal` DOUBLE,
`kappa` DOUBLE,
`gamma1` DOUBLE,
`gamma2` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`observed_redshift_gal` DOUBLE,
`hpix_29_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`abs_mag_r01` DOUBLE,
`luminosity_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`sdss_r01` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` BIGINT,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha` DOUBLE,
`logf_halpha_ext` DOUBLE,
`halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_halpha_model3_ext` DOUBLE,
`loglum_halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`2mass_h` DOUBLE,
`2mass_j` DOUBLE,
`2mass_ks` DOUBLE,
`blanco_decam_g` DOUBLE,
`blanco_decam_i` DOUBLE,
`blanco_decam_r` DOUBLE,
`blanco_decam_z` DOUBLE,
`cfht_megacam_r` DOUBLE,
`cfht_megacam_u` DOUBLE,
`euclid_nisp_h` DOUBLE,
`euclid_nisp_j` DOUBLE,
`euclid_nisp_y` DOUBLE,
`euclid_vis` DOUBLE,
`gaia_bp` DOUBLE,
`gaia_g` DOUBLE,
`gaia_rp` DOUBLE,
`jst_jpcam_g` DOUBLE,
`kids_g` DOUBLE,
`kids_i` DOUBLE,
`kids_r` DOUBLE,
`kids_u` DOUBLE,
`lsst_g` DOUBLE,
`lsst_i` DOUBLE,
`lsst_r` DOUBLE,
`lsst_u` DOUBLE,
`lsst_y` DOUBLE,
`lsst_z` DOUBLE,
`pan_starrs_i` DOUBLE,
`pan_starrs_z` DOUBLE,
`subaru_hsc_z` DOUBLE,
`2mass_h_odonnell_ext` DOUBLE,
`2mass_j_odonnell_ext` DOUBLE,
`2mass_ks_odonnell_ext` DOUBLE,
`blanco_decam_g_odonnell_ext` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`blanco_decam_r_odonnell_ext` DOUBLE,
`blanco_decam_z_odonnell_ext` DOUBLE,
`cfht_megacam_r_odonnell_ext` DOUBLE,
`cfht_megacam_u_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`euclid_nisp_j_odonnell_ext` DOUBLE,
`euclid_nisp_y_odonnell_ext` DOUBLE,
`euclid_vis_odonnell_ext` DOUBLE,
`gaia_bp_odonnell_ext` DOUBLE,
`gaia_g_odonnell_ext` DOUBLE,
`gaia_rp_odonnell_ext` DOUBLE,
`jst_jpcam_g_odonnell_ext` DOUBLE,
`kids_g_odonnell_ext` DOUBLE,
`kids_i_odonnell_ext` DOUBLE,
`kids_r_odonnell_ext` DOUBLE,
`kids_u_odonnell_ext` DOUBLE,
`lsst_g_odonnell_ext` DOUBLE,
`lsst_i_odonnell_ext` DOUBLE,
`lsst_r_odonnell_ext` DOUBLE,
`lsst_u_odonnell_ext` DOUBLE,
`lsst_y_odonnell_ext` DOUBLE,
`lsst_z_odonnell_ext` DOUBLE,
`pan_starrs_i_odonnell_ext` DOUBLE,
`pan_starrs_z_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`subaru_hsc_z_odonnell_ext` DOUBLE,
`2mass_h_el_model1_odonnell_ext` DOUBLE,
`2mass_h_el_model3_odonnell_ext` DOUBLE,
`2mass_j_el_model1_odonnell_ext` DOUBLE,
`2mass_j_el_model3_odonnell_ext` DOUBLE,
`2mass_ks_el_model1_odonnell_ext` DOUBLE,
`2mass_ks_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model3_odonnell_ext` DOUBLE,
`euclid_vis_el_model1_odonnell_ext` DOUBLE,
`euclid_vis_el_model3_odonnell_ext` DOUBLE,
`gaia_bp_el_model1_odonnell_ext` DOUBLE,
`gaia_bp_el_model3_odonnell_ext` DOUBLE,
`gaia_g_el_model1_odonnell_ext` DOUBLE,
`gaia_g_el_model3_odonnell_ext` DOUBLE,
`gaia_rp_el_model1_odonnell_ext` DOUBLE,
`gaia_rp_el_model3_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model1_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model3_odonnell_ext` DOUBLE,
`kids_g_el_model1_odonnell_ext` DOUBLE,
`kids_g_el_model3_odonnell_ext` DOUBLE,
`kids_i_el_model1_odonnell_ext` DOUBLE,
`kids_i_el_model3_odonnell_ext` DOUBLE,
`kids_r_el_model1_odonnell_ext` DOUBLE,
`kids_r_el_model3_odonnell_ext` DOUBLE,
`kids_u_el_model1_odonnell_ext` DOUBLE,
`kids_u_el_model3_odonnell_ext` DOUBLE,
`lsst_g_el_model1_odonnell_ext` DOUBLE,
`lsst_g_el_model3_odonnell_ext` DOUBLE,
`lsst_i_el_model1_odonnell_ext` DOUBLE,
`lsst_i_el_model3_odonnell_ext` DOUBLE,
`lsst_r_el_model1_odonnell_ext` DOUBLE,
`lsst_r_el_model3_odonnell_ext` DOUBLE,
`lsst_u_el_model1_odonnell_ext` DOUBLE,
`lsst_u_el_model3_odonnell_ext` DOUBLE,
`lsst_y_el_model1_odonnell_ext` DOUBLE,
`lsst_y_el_model3_odonnell_ext` DOUBLE,
`lsst_z_el_model1_odonnell_ext` DOUBLE,
`lsst_z_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model1_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model3_odonnell_ext` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`luminosity_r01_evolved` DOUBLE,
`hpix_13_nest` BIGINT,
`step` BIGINT,
`dominant_shape` BIGINT,
`bulge_angle` DOUBLE,
`disk_angle` DOUBLE,
`median_major_axis` DOUBLE,
`scale_length` DOUBLE,
`bulge_fraction` DOUBLE,
`disk_scalelength` DOUBLE,
`disk_nsersic` DOUBLE,
`disk_r50` DOUBLE,
`bulge_r50` DOUBLE,
`bulge_nsersic` DOUBLE,
`inclination_angle` DOUBLE,
`disk_axis_ratio` DOUBLE,
`bulge_axis_ratio` DOUBLE,
`disk_ellipticity` DOUBLE,
`bulge_ellipticity` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
STORED AS PARQUET
LOCATION '/user/jcarrete/data/euclid/flagship_mock_obsid_1717_s_v2'
;
CREATE TABLE jcarrete.flagship_mock_obsid_1717_s_v2(
`halo_id` bigint COMMENT 'unique halo id coming from the Flagship dark matter halo catalog',
`galaxy_id` int COMMENT 'combined with the unique_halo_id they uniquely identify each galaxy',
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',
`random_index` float COMMENT 'random number [0 - 1) for subsampling',
`ra_gal` float COMMENT 'galaxy right ascension (degrees)',
`dec_gal` float COMMENT 'galaxy declination (degrees)',
`ra_mag_gal` float COMMENT 'galaxy magnified right ascension (degree)',
`dec_mag_gal` float COMMENT 'galaxy magnified declination (degree)',
`kappa` float COMMENT 'convergence',
`gamma1` float COMMENT 'shear',
`gamma2` float COMMENT 'shear',
`l_gal` float COMMENT 'galactic longitude (degrees)',
`b_gal` float COMMENT 'galactic latitude (degrees)',
`mw_extinction` float COMMENT 'EBV from file: HFI_CompMap_ThermalDustModel_2048_R1.20.fits',
`x_gal` float COMMENT 'galaxy comoving x coordinate (Mpc/h)',
`y_gal` float COMMENT 'galaxy comoving y coordinate (Mpc/h)',
`z_gal` float COMMENT 'galaxy comoving z coordinate (Mpc/h)',
`r_gal` float COMMENT 'galaxy comoving distance (Mpc/h)',
`true_redshift_gal` float COMMENT 'true galaxy redshift',
`observed_redshift_gal` float COMMENT 'observed galaxy redshift (including peculiar velocity)',
`hpix_29_nest` bigint COMMENT 'healpix pixel id using order = 29 in NESTED format',
`vx_gal` float COMMENT 'physical (peculiar) velocity in x-component (km/s)',
`vy_gal` float COMMENT 'physical (peculiar) velocity in y-component (km/s)',
`vz_gal` float COMMENT 'physical (peculiar) velocity in z-component (km/s)',
`vrad_gal` float COMMENT 'missing comment',
`delta_r` float COMMENT 'missing comment',
`abs_mag_r01` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h)',
`luminosity_r01` float COMMENT 'luminosity in the 0.1-r-band',
`abs_mag_r01_evolved` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h) including MICECATv2.0 magnitude evolution',
`sdss_r01` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1',
`abs_mag_uv_dereddened` float COMMENT 'absolute magnitude ultraviolet',
`gr_restframe` float COMMENT '(g-r) rest-frame at z=0',
`color_kind` tinyint COMMENT 'galaxy color kind: 0=red sequence 1=green valley 2=blue cloud',
`sed_ke` float COMMENT 'K-correction for each SED',
`gr_cosmos` float COMMENT 'interpolated COSMOS g-r color',
`sed_cosmos` float COMMENT 'interpolated basic COSMOS SED [0 30]',
`is_within_cosmos` boolean COMMENT 'missing comment',
`cosmos_distance` float COMMENT 'missing comment',
`ext_curve_cosmos` tinyint COMMENT 'COSMOS extinction curve [0-4]',
`ebv_cosmos` float COMMENT 'interpolated extinction strength for COSMOS extinction curve',
`log_ml_r01` float COMMENT 'logarithm of stellar_mass/luminosity ratio in the r01 SDSS band',
`log_stellar_mass` float COMMENT 'logarithm of the stellar mass (Msun/h)',
`metallicity` float COMMENT 'oxygen metallicity in units 12+logO/H',
`log_sfr` float COMMENT 'logarithm of star formation rate in (Msun/h)/year',
`logf_halpha` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: DOES NOT include extinction)',
`logf_halpha_ext` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: includes extinction)',
`halpha_scatter` float COMMENT 'missing comment',
`loglum_halpha` float COMMENT 'DEPRECATED logarithm of Halpha luminosity (erg/s/h^2) using empirical relation with UV magnitude (DOES NOT include extinction)',
`loglum_halpha_ext` float COMMENT 'missing comment',
`logf_dummy` float COMMENT 'missing comment',
`z_dummy` float COMMENT 'missing comment',
`logf_halpha_model1_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: includes extinction)',
`logf_halpha_model1` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: DOES NOT include extinction)',
`loglum_halpha_model1_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (includes extinction)',
`loglum_halpha_model1` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (DOES NOT include extinction)',
`logf_halpha_model3_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: includes extinction)',
`logf_halpha_model3` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: DOES NOT include extinction)',
`loglum_halpha_model3_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (includes extinction)',
`loglum_halpha_model3` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (DOES NOT include extinction)',
`logf_hbeta_model1_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_hbeta_model1` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_n2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_n2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o3_model1_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o3_model1` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_s2_model1_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_s2_model1` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_hbeta_model3_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_hbeta_model3` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_n2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_n2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o3_model3_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o3_model3` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_s2_model3_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_s2_model3` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`2mass_h` float COMMENT 'observed flux in 2MASS H-band (Euclid provided)',
`2mass_j` float COMMENT 'observed flux in 2MASS J-band (Euclid provided)',
`2mass_ks` float COMMENT 'observed flux in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g` float COMMENT 'observed flux in DES g-band (Euclid provided)',
`blanco_decam_i` float COMMENT 'observed flux in DES i-band (Euclid provided)',
`blanco_decam_r` float COMMENT 'observed flux in DES r-band (Euclid provided)',
`blanco_decam_z` float COMMENT 'observed flux in DES z-band (Euclid provided)',
`cfht_megacam_r` float COMMENT 'observed flux in CFHT r-band (Euclid provided)',
`cfht_megacam_u` float COMMENT 'observed flux in CFHT u-band (Euclid provided)',
`euclid_nisp_h` float COMMENT 'observed flux in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j` float COMMENT 'observed flux in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y` float COMMENT 'observed flux in Euclid NISP-Y band (Euclid provided)',
`euclid_vis` float COMMENT 'observed flux in Euclid VIS band (Euclid provided)',
`gaia_bp` float COMMENT 'observed flux in Gaia-BP band (Euclid provided)',
`gaia_g` float COMMENT 'observed flux in Gaia-G band (Euclid provided)',
`gaia_rp` float COMMENT 'observed flux in Gaia-RP band (Euclid provided)',
`jst_jpcam_g` float COMMENT 'observed flux in JST g-band (Euclid provided)',
`kids_g` float COMMENT 'observed flux in KIDS g-band (Euclid provided)',
`kids_i` float COMMENT 'observed flux in KIDS i-band (Euclid provided)',
`kids_r` float COMMENT 'observed flux in KIDS r-band (Euclid provided)',
`kids_u` float COMMENT 'observed flux in KIDS u-band (Euclid provided)',
`lsst_g` float COMMENT 'observed flux in LSST g-band (Euclid provided)',
`lsst_i` float COMMENT 'observed flux in LSST i-band (Euclid provided)',
`lsst_r` float COMMENT 'observed flux in LSST r-band (Euclid provided)',
`lsst_u` float COMMENT 'observed flux in LSST u-band (Euclid provided)',
`lsst_y` float COMMENT 'observed flux in LSST Y-band (Euclid provided)',
`lsst_z` float COMMENT 'observed flux in LSST z-band (Euclid provided)',
`pan_starrs_i` float COMMENT 'observed flux in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z` float COMMENT 'observed flux in Pan-STARRS z-band (Euclid provided)',
`subaru_hsc_z` float COMMENT 'observed flux in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS H-band (Euclid provided)',
`2mass_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in JST g-band (Euclid provided)',
`kids_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_odonnell_ext` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1 with MW extinction',
`subaru_hsc_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS Ks-band (Euclid provided)',
`2mass_ks_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES z-band (Euclid provided)',
`blanco_decam_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT u-band (Euclid provided)',
`cfht_megacam_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_nisp_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid VIS band (Euclid provided)',
`euclid_vis_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_bp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-RP band (Euclid provided)',
`gaia_rp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in JST g-band (Euclid provided)',
`jst_jpcam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in JST g-band (Euclid provided)',
`kids_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS g-band (Euclid provided)',
`kids_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS i-band (Euclid provided)',
`kids_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS r-band (Euclid provided)',
`kids_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS u-band (Euclid provided)',
`kids_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST g-band (Euclid provided)',
`lsst_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST i-band (Euclid provided)',
`lsst_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST r-band (Euclid provided)',
`lsst_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST u-band (Euclid provided)',
`lsst_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST z-band (Euclid provided)',
`lsst_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`pan_starrs_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in sdss r-band redshifted to z=0.1',
`sdss_r01_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in sdss r-band redshifted to z=0.1',
`subaru_hsc_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`subaru_hsc_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`dominant_shape` tinyint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)',
`bulge_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`disk_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`median_major_axis` float COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)',
`scale_length` float COMMENT 'disc and bulge scalelength prior',
`bulge_fraction` float COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)',
`disk_scalelength` float COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)',
`disk_nsersic` float COMMENT 'Sersic index of the disk component',
`disk_r50` float COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678',
`bulge_r50` float COMMENT 'half light radius of the bulge',
`bulge_nsersic` float COMMENT 'Sersic index of the bulge component',
`disk_ellipticity` float COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)',
`bulge_ellipticity` float COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)',
`disk_axis_ratio` float COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`bulge_axis_ratio` float COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`inclination_angle` float COMMENT 'Galaxy inclination angle (where 0 degrees = face-on and 90 degrees = edge-on). Galaxy ellipticity for disk and bulge components are computed following the recipe in https://euclid.roe.ac.uk/projects/sgsshear/wiki/SHE-SIM',
`halo_x` float COMMENT 'halo comoving x coordinate (Mpc/h)',
`halo_y` float COMMENT 'halo comoving y coordinate (Mpc/h)',
`halo_z` float COMMENT 'halo comoving z coordinate (Mpc/h)',
`halo_vx` float COMMENT 'halo x-centre velocity (km/h)',
`halo_vy` float COMMENT 'halo y-centre velocity (km/h)',
`halo_vz` float COMMENT 'halo z-centre velocity (km/h)',
`halo_r` float COMMENT 'halo comoving distance (Mpc/h)',
`true_redshift_halo` float COMMENT 'true redshift of the host halo',
`halo_lm` float COMMENT 'log10 of the FoF halo mass in Msum/h. This halo mass is computed as the particule mass multiplied by the number of particles of the FoF halo. At low masses it is interpolated to obtain a smooth distribution of halo masses',
`halo_n_sats` int COMMENT 'number of satellite galaxies from HOD',
`n_gals` int COMMENT 'DEPRECATED',
`luminosity_r01_evolved` float COMMENT 'missing comment',
`hpix_13_nest` bigint COMMENT 'missing comment')
PARTITIONED BY (
`step` smallint COMMENT 'step number in the Flagship lensing maps')
CLUSTERED BY (
hpix_13_nest)
SORTED BY (
hpix_13_nest ASC)
INTO 128 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;
INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_1717_s_v2 PARTITION(step) SELECT halo_id, galaxy_id, kind, random_index, ra_gal, dec_gal, ra_mag_gal, dec_mag_gal, kappa, gamma1, gamma2, l_gal, b_gal, mw_extinction, x_gal, y_gal, z_gal, r_gal, true_redshift_gal, observed_redshift_gal, hpix_29_nest, vx_gal, vy_gal, vz_gal, vrad_gal, delta_r, abs_mag_r01, luminosity_r01, abs_mag_r01_evolved, sdss_r01, abs_mag_uv_dereddened, gr_restframe, color_kind, sed_ke, gr_cosmos, sed_cosmos, is_within_cosmos, cosmos_distance, ext_curve_cosmos, ebv_cosmos, log_ml_r01, log_stellar_mass, metallicity, log_sfr, logf_halpha, logf_halpha_ext, halpha_scatter, loglum_halpha, loglum_halpha_ext, logf_dummy, z_dummy, logf_halpha_model1_ext, logf_halpha_model1, loglum_halpha_model1_ext, loglum_halpha_model1, logf_halpha_model3_ext, logf_halpha_model3, loglum_halpha_model3_ext, loglum_halpha_model3, logf_hbeta_model1_ext, logf_hbeta_model1, logf_o2_model1_ext, logf_o2_model1, logf_n2_model1_ext, logf_n2_model1, logf_o3_model1_ext, logf_o3_model1, logf_s2_model1_ext, logf_s2_model1, logf_hbeta_model3_ext, logf_hbeta_model3, logf_o2_model3_ext, logf_o2_model3, logf_n2_model3_ext, logf_n2_model3, logf_o3_model3_ext, logf_o3_model3, logf_s2_model3_ext, logf_s2_model3, 2mass_h, 2mass_j, 2mass_ks, blanco_decam_g, blanco_decam_i, blanco_decam_r, blanco_decam_z, cfht_megacam_r, cfht_megacam_u, euclid_nisp_h, euclid_nisp_j, euclid_nisp_y, euclid_vis, gaia_bp, gaia_g, gaia_rp, jst_jpcam_g, kids_g, kids_i, kids_r, kids_u, lsst_g, lsst_i, lsst_r, lsst_u, lsst_y, lsst_z, pan_starrs_i, pan_starrs_z, subaru_hsc_z, 2mass_h_odonnell_ext, 2mass_j_odonnell_ext, 2mass_ks_odonnell_ext, blanco_decam_g_odonnell_ext, blanco_decam_i_odonnell_ext, blanco_decam_r_odonnell_ext, blanco_decam_z_odonnell_ext, cfht_megacam_r_odonnell_ext, cfht_megacam_u_odonnell_ext, euclid_nisp_h_odonnell_ext, euclid_nisp_j_odonnell_ext, euclid_nisp_y_odonnell_ext, euclid_vis_odonnell_ext, gaia_bp_odonnell_ext, gaia_g_odonnell_ext, gaia_rp_odonnell_ext, jst_jpcam_g_odonnell_ext, kids_g_odonnell_ext, kids_i_odonnell_ext, kids_r_odonnell_ext, kids_u_odonnell_ext, lsst_g_odonnell_ext, lsst_i_odonnell_ext, lsst_r_odonnell_ext, lsst_u_odonnell_ext, lsst_y_odonnell_ext, lsst_z_odonnell_ext, pan_starrs_i_odonnell_ext, pan_starrs_z_odonnell_ext, sdss_r01_odonnell_ext, subaru_hsc_z_odonnell_ext, 2mass_h_el_model1_odonnell_ext, 2mass_h_el_model3_odonnell_ext, 2mass_j_el_model1_odonnell_ext, 2mass_j_el_model3_odonnell_ext, 2mass_ks_el_model1_odonnell_ext, 2mass_ks_el_model3_odonnell_ext, blanco_decam_g_el_model1_odonnell_ext, blanco_decam_g_el_model3_odonnell_ext, blanco_decam_i_el_model1_odonnell_ext, blanco_decam_i_el_model3_odonnell_ext, blanco_decam_r_el_model1_odonnell_ext, blanco_decam_r_el_model3_odonnell_ext, blanco_decam_z_el_model1_odonnell_ext, blanco_decam_z_el_model3_odonnell_ext, cfht_megacam_r_el_model1_odonnell_ext, cfht_megacam_r_el_model3_odonnell_ext, cfht_megacam_u_el_model1_odonnell_ext, cfht_megacam_u_el_model3_odonnell_ext, euclid_nisp_h_el_model1_odonnell_ext, euclid_nisp_h_el_model3_odonnell_ext, euclid_nisp_j_el_model1_odonnell_ext, euclid_nisp_j_el_model3_odonnell_ext, euclid_nisp_y_el_model1_odonnell_ext, euclid_nisp_y_el_model3_odonnell_ext, euclid_vis_el_model1_odonnell_ext, euclid_vis_el_model3_odonnell_ext, gaia_bp_el_model1_odonnell_ext, gaia_bp_el_model3_odonnell_ext, gaia_g_el_model1_odonnell_ext, gaia_g_el_model3_odonnell_ext, gaia_rp_el_model1_odonnell_ext, gaia_rp_el_model3_odonnell_ext, jst_jpcam_g_el_model1_odonnell_ext, jst_jpcam_g_el_model3_odonnell_ext, kids_g_el_model1_odonnell_ext, kids_g_el_model3_odonnell_ext, kids_i_el_model1_odonnell_ext, kids_i_el_model3_odonnell_ext, kids_r_el_model1_odonnell_ext, kids_r_el_model3_odonnell_ext, kids_u_el_model1_odonnell_ext, kids_u_el_model3_odonnell_ext, lsst_g_el_model1_odonnell_ext, lsst_g_el_model3_odonnell_ext, lsst_i_el_model1_odonnell_ext, lsst_i_el_model3_odonnell_ext, lsst_r_el_model1_odonnell_ext, lsst_r_el_model3_odonnell_ext, lsst_u_el_model1_odonnell_ext, lsst_u_el_model3_odonnell_ext, lsst_y_el_model1_odonnell_ext, lsst_y_el_model3_odonnell_ext, lsst_z_el_model1_odonnell_ext, lsst_z_el_model3_odonnell_ext, pan_starrs_i_el_model1_odonnell_ext, pan_starrs_i_el_model3_odonnell_ext, pan_starrs_z_el_model1_odonnell_ext, pan_starrs_z_el_model3_odonnell_ext, sdss_r01_el_model1_odonnell_ext, sdss_r01_el_model3_odonnell_ext, subaru_hsc_z_el_model1_odonnell_ext, subaru_hsc_z_el_model3_odonnell_ext, dominant_shape, bulge_angle, disk_angle, median_major_axis, scale_length, bulge_fraction, disk_scalelength, disk_nsersic, disk_r50, bulge_r50, bulge_nsersic, disk_ellipticity, bulge_ellipticity, disk_axis_ratio, bulge_axis_ratio, inclination_angle, halo_x, halo_y, halo_z, halo_vx, halo_vy, halo_vz, halo_r, true_redshift_halo, halo_lm, halo_n_sats, n_gals, luminosity_r01_evolved, hpix_13_nest, step FROM jcarrete.flagship_mock_obsid_1717_s_v2_pq;
La paso al esquema cosmohub
ALTER TABLE jcarrete.flagship_mock_obsid_1717_s_v2 RENAME TO cosmohub.flagship_mock_obsid_1717_s_v2;
Ahora es necesario volver a calcular los flujos! Pero únicamente para el filtro LSST!
Abro la pipeline de flujos y con la siguiente SQL (cojo todo menos los LSST filters):
(y solo añado el filtro LSST en la lista de air filters!)
halo_sql = """ SELECT halo_id, galaxy_id, kind, random_index, ra_gal, dec_gal, ra_mag_gal, dec_mag_gal, kappa, gamma1, gamma2, l_gal, b_gal, mw_extinction, x_gal, y_gal, z_gal, r_gal, true_redshift_gal, observed_redshift_gal, hpix_29_nest, vx_gal, vy_gal, vz_gal, vrad_gal, delta_r, abs_mag_r01, luminosity_r01, abs_mag_r01_evolved, sdss_r01, abs_mag_uv_dereddened, gr_restframe, color_kind, sed_ke, gr_cosmos, sed_cosmos, is_within_cosmos, cosmos_distance, ext_curve_cosmos, ebv_cosmos, log_ml_r01, log_stellar_mass, metallicity, log_sfr, logf_halpha, logf_halpha_ext, halpha_scatter, loglum_halpha, loglum_halpha_ext, logf_dummy, z_dummy, logf_halpha_model1_ext, logf_halpha_model1, loglum_halpha_model1_ext, loglum_halpha_model1, logf_halpha_model3_ext, logf_halpha_model3, loglum_halpha_model3_ext, loglum_halpha_model3, logf_hbeta_model1_ext, logf_hbeta_model1, logf_o2_model1_ext, logf_o2_model1, logf_n2_model1_ext, logf_n2_model1, logf_o3_model1_ext, logf_o3_model1, logf_s2_model1_ext, logf_s2_model1, logf_hbeta_model3_ext, logf_hbeta_model3, logf_o2_model3_ext, logf_o2_model3, logf_n2_model3_ext, logf_n2_model3, logf_o3_model3_ext, logf_o3_model3, logf_s2_model3_ext, logf_s2_model3, 2mass_h, 2mass_j, 2mass_ks, blanco_decam_g, blanco_decam_i, blanco_decam_r, blanco_decam_z, cfht_megacam_r, cfht_megacam_u, euclid_nisp_h, euclid_nisp_j, euclid_nisp_y, euclid_vis, gaia_bp, gaia_g, gaia_rp, jst_jpcam_g, kids_g, kids_i, kids_r, kids_u, pan_starrs_i, pan_starrs_z, subaru_hsc_z, 2mass_h_odonnell_ext, 2mass_j_odonnell_ext, 2mass_ks_odonnell_ext, blanco_decam_g_odonnell_ext, blanco_decam_i_odonnell_ext, blanco_decam_r_odonnell_ext, blanco_decam_z_odonnell_ext, cfht_megacam_r_odonnell_ext, cfht_megacam_u_odonnell_ext, euclid_nisp_h_odonnell_ext, euclid_nisp_j_odonnell_ext, euclid_nisp_y_odonnell_ext, euclid_vis_odonnell_ext, gaia_bp_odonnell_ext, gaia_g_odonnell_ext, gaia_rp_odonnell_ext, jst_jpcam_g_odonnell_ext, kids_g_odonnell_ext, kids_i_odonnell_ext, kids_r_odonnell_ext, kids_u_odonnell_ext, pan_starrs_i_odonnell_ext, pan_starrs_z_odonnell_ext, sdss_r01_odonnell_ext, subaru_hsc_z_odonnell_ext, 2mass_h_el_model1_odonnell_ext, 2mass_h_el_model3_odonnell_ext, 2mass_j_el_model1_odonnell_ext, 2mass_j_el_model3_odonnell_ext, 2mass_ks_el_model1_odonnell_ext, 2mass_ks_el_model3_odonnell_ext, blanco_decam_g_el_model1_odonnell_ext, blanco_decam_g_el_model3_odonnell_ext, blanco_decam_i_el_model1_odonnell_ext, blanco_decam_i_el_model3_odonnell_ext, blanco_decam_r_el_model1_odonnell_ext, blanco_decam_r_el_model3_odonnell_ext, blanco_decam_z_el_model1_odonnell_ext, blanco_decam_z_el_model3_odonnell_ext, cfht_megacam_r_el_model1_odonnell_ext, cfht_megacam_r_el_model3_odonnell_ext, cfht_megacam_u_el_model1_odonnell_ext, cfht_megacam_u_el_model3_odonnell_ext, euclid_nisp_h_el_model1_odonnell_ext, euclid_nisp_h_el_model3_odonnell_ext, euclid_nisp_j_el_model1_odonnell_ext, euclid_nisp_j_el_model3_odonnell_ext, euclid_nisp_y_el_model1_odonnell_ext, euclid_nisp_y_el_model3_odonnell_ext, euclid_vis_el_model1_odonnell_ext, euclid_vis_el_model3_odonnell_ext, gaia_bp_el_model1_odonnell_ext, gaia_bp_el_model3_odonnell_ext, gaia_g_el_model1_odonnell_ext, gaia_g_el_model3_odonnell_ext, gaia_rp_el_model1_odonnell_ext, gaia_rp_el_model3_odonnell_ext, jst_jpcam_g_el_model1_odonnell_ext, jst_jpcam_g_el_model3_odonnell_ext, kids_g_el_model1_odonnell_ext, kids_g_el_model3_odonnell_ext, kids_i_el_model1_odonnell_ext, kids_i_el_model3_odonnell_ext, kids_r_el_model1_odonnell_ext, kids_r_el_model3_odonnell_ext, kids_u_el_model1_odonnell_ext, kids_u_el_model3_odonnell_ext, pan_starrs_i_el_model1_odonnell_ext, pan_starrs_i_el_model3_odonnell_ext, pan_starrs_z_el_model1_odonnell_ext, pan_starrs_z_el_model3_odonnell_ext, sdss_r01_el_model1_odonnell_ext, sdss_r01_el_model3_odonnell_ext, subaru_hsc_z_el_model1_odonnell_ext, subaru_hsc_z_el_model3_odonnell_ext, dominant_shape, bulge_angle, disk_angle, median_major_axis, scale_length, bulge_fraction, disk_scalelength, disk_nsersic, disk_r50, bulge_r50, bulge_nsersic, disk_ellipticity, bulge_ellipticity, disk_axis_ratio, bulge_axis_ratio, inclination_angle, halo_x, halo_y, halo_z, halo_vx, halo_vy, halo_vz, halo_r, true_redshift_halo, halo_lm, halo_n_sats, n_gals, luminosity_r01_evolved, hpix_13_nest, step FROM cosmohub.flagship_mock_obsid_1717_s_v2 """
/software/astro/scripts/create_table_from_parquet.sh -p /hdfs/user/jcarrete/data/euclid/flagship_mock_obsid_1717_s_v3/0.pq -t jcarrete.flagship_mock_obsid_1717_s_v3_pq -e
CREATE EXTERNAL TABLE jcarrete.flagship_mock_obsid_1717_s_v3_pq (
`kind` BIGINT,
`random_index` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`ra_mag_gal` DOUBLE,
`dec_mag_gal` DOUBLE,
`kappa` DOUBLE,
`gamma1` DOUBLE,
`gamma2` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`observed_redshift_gal` DOUBLE,
`hpix_29_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`abs_mag_r01` DOUBLE,
`luminosity_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`sdss_r01` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` BIGINT,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha` DOUBLE,
`logf_halpha_ext` DOUBLE,
`halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_halpha_model3_ext` DOUBLE,
`loglum_halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`2mass_h` DOUBLE,
`2mass_j` DOUBLE,
`2mass_ks` DOUBLE,
`blanco_decam_g` DOUBLE,
`blanco_decam_i` DOUBLE,
`blanco_decam_r` DOUBLE,
`blanco_decam_z` DOUBLE,
`cfht_megacam_r` DOUBLE,
`cfht_megacam_u` DOUBLE,
`euclid_nisp_h` DOUBLE,
`euclid_nisp_j` DOUBLE,
`euclid_nisp_y` DOUBLE,
`euclid_vis` DOUBLE,
`gaia_bp` DOUBLE,
`gaia_g` DOUBLE,
`gaia_rp` DOUBLE,
`jst_jpcam_g` DOUBLE,
`kids_g` DOUBLE,
`kids_i` DOUBLE,
`kids_r` DOUBLE,
`kids_u` DOUBLE,
`pan_starrs_i` DOUBLE,
`pan_starrs_z` DOUBLE,
`subaru_hsc_z` DOUBLE,
`2mass_h_odonnell_ext` DOUBLE,
`2mass_j_odonnell_ext` DOUBLE,
`2mass_ks_odonnell_ext` DOUBLE,
`blanco_decam_g_odonnell_ext` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`blanco_decam_r_odonnell_ext` DOUBLE,
`blanco_decam_z_odonnell_ext` DOUBLE,
`cfht_megacam_r_odonnell_ext` DOUBLE,
`cfht_megacam_u_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`euclid_nisp_j_odonnell_ext` DOUBLE,
`euclid_nisp_y_odonnell_ext` DOUBLE,
`euclid_vis_odonnell_ext` DOUBLE,
`gaia_bp_odonnell_ext` DOUBLE,
`gaia_g_odonnell_ext` DOUBLE,
`gaia_rp_odonnell_ext` DOUBLE,
`jst_jpcam_g_odonnell_ext` DOUBLE,
`kids_g_odonnell_ext` DOUBLE,
`kids_i_odonnell_ext` DOUBLE,
`kids_r_odonnell_ext` DOUBLE,
`kids_u_odonnell_ext` DOUBLE,
`pan_starrs_i_odonnell_ext` DOUBLE,
`pan_starrs_z_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`subaru_hsc_z_odonnell_ext` DOUBLE,
`2mass_h_el_model1_odonnell_ext` DOUBLE,
`2mass_h_el_model3_odonnell_ext` DOUBLE,
`2mass_j_el_model1_odonnell_ext` DOUBLE,
`2mass_j_el_model3_odonnell_ext` DOUBLE,
`2mass_ks_el_model1_odonnell_ext` DOUBLE,
`2mass_ks_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_g_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_r_el_model3_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_z_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_r_el_model3_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model1_odonnell_ext` DOUBLE,
`cfht_megacam_u_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_j_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_y_el_model3_odonnell_ext` DOUBLE,
`euclid_vis_el_model1_odonnell_ext` DOUBLE,
`euclid_vis_el_model3_odonnell_ext` DOUBLE,
`gaia_bp_el_model1_odonnell_ext` DOUBLE,
`gaia_bp_el_model3_odonnell_ext` DOUBLE,
`gaia_g_el_model1_odonnell_ext` DOUBLE,
`gaia_g_el_model3_odonnell_ext` DOUBLE,
`gaia_rp_el_model1_odonnell_ext` DOUBLE,
`gaia_rp_el_model3_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model1_odonnell_ext` DOUBLE,
`jst_jpcam_g_el_model3_odonnell_ext` DOUBLE,
`kids_g_el_model1_odonnell_ext` DOUBLE,
`kids_g_el_model3_odonnell_ext` DOUBLE,
`kids_i_el_model1_odonnell_ext` DOUBLE,
`kids_i_el_model3_odonnell_ext` DOUBLE,
`kids_r_el_model1_odonnell_ext` DOUBLE,
`kids_r_el_model3_odonnell_ext` DOUBLE,
`kids_u_el_model1_odonnell_ext` DOUBLE,
`kids_u_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_i_el_model3_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model1_odonnell_ext` DOUBLE,
`pan_starrs_z_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model1_odonnell_ext` DOUBLE,
`subaru_hsc_z_el_model3_odonnell_ext` DOUBLE,
`dominant_shape` BIGINT,
`bulge_angle` DOUBLE,
`disk_angle` DOUBLE,
`median_major_axis` DOUBLE,
`scale_length` DOUBLE,
`bulge_fraction` DOUBLE,
`disk_scalelength` DOUBLE,
`disk_nsersic` DOUBLE,
`disk_r50` DOUBLE,
`bulge_r50` DOUBLE,
`bulge_nsersic` DOUBLE,
`disk_ellipticity` DOUBLE,
`bulge_ellipticity` DOUBLE,
`disk_axis_ratio` DOUBLE,
`bulge_axis_ratio` DOUBLE,
`inclination_angle` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`luminosity_r01_evolved` DOUBLE,
`hpix_13_nest` BIGINT,
`step` BIGINT,
`lsst_g` DOUBLE,
`lsst_i` DOUBLE,
`lsst_r` DOUBLE,
`lsst_u` DOUBLE,
`lsst_y` DOUBLE,
`lsst_z` DOUBLE,
`lsst_g_odonnell_ext` DOUBLE,
`lsst_i_odonnell_ext` DOUBLE,
`lsst_r_odonnell_ext` DOUBLE,
`lsst_u_odonnell_ext` DOUBLE,
`lsst_y_odonnell_ext` DOUBLE,
`lsst_z_odonnell_ext` DOUBLE,
`lsst_g_el_model1_odonnell_ext` DOUBLE,
`lsst_g_el_model3_odonnell_ext` DOUBLE,
`lsst_i_el_model1_odonnell_ext` DOUBLE,
`lsst_i_el_model3_odonnell_ext` DOUBLE,
`lsst_r_el_model1_odonnell_ext` DOUBLE,
`lsst_r_el_model3_odonnell_ext` DOUBLE,
`lsst_u_el_model1_odonnell_ext` DOUBLE,
`lsst_u_el_model3_odonnell_ext` DOUBLE,
`lsst_y_el_model1_odonnell_ext` DOUBLE,
`lsst_y_el_model3_odonnell_ext` DOUBLE,
`lsst_z_el_model1_odonnell_ext` DOUBLE,
`lsst_z_el_model3_odonnell_ext` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
STORED AS PARQUET
LOCATION '/user/jcarrete/data/euclid/flagship_mock_obsid_1717_s_v3'
;
Creo la tabla en ORC:
CREATE TABLE jcarrete.flagship_mock_obsid_1717_s_v3(
`halo_id` bigint COMMENT 'unique halo id coming from the Flagship dark matter halo catalog',
`galaxy_id` int COMMENT 'combined with the unique_halo_id they uniquely identify each galaxy',
`kind` tinyint COMMENT 'Galaxy type: 0: CENTRAL, 1: SATELLITE, 2: QSO, 3: HIGH REDSHIFT',
`random_index` float COMMENT 'random number [0 - 1) for subsampling',
`ra_gal` float COMMENT 'galaxy right ascension (degrees)',
`dec_gal` float COMMENT 'galaxy declination (degrees)',
`ra_mag_gal` float COMMENT 'galaxy magnified right ascension (degree)',
`dec_mag_gal` float COMMENT 'galaxy magnified declination (degree)',
`kappa` float COMMENT 'convergence',
`gamma1` float COMMENT 'shear',
`gamma2` float COMMENT 'shear',
`l_gal` float COMMENT 'galactic longitude (degrees)',
`b_gal` float COMMENT 'galactic latitude (degrees)',
`mw_extinction` float COMMENT 'EBV from file: HFI_CompMap_ThermalDustModel_2048_R1.20.fits',
`x_gal` float COMMENT 'galaxy comoving x coordinate (Mpc/h)',
`y_gal` float COMMENT 'galaxy comoving y coordinate (Mpc/h)',
`z_gal` float COMMENT 'galaxy comoving z coordinate (Mpc/h)',
`r_gal` float COMMENT 'galaxy comoving distance (Mpc/h)',
`true_redshift_gal` float COMMENT 'true galaxy redshift',
`observed_redshift_gal` float COMMENT 'observed galaxy redshift (including peculiar velocity)',
`hpix_29_nest` bigint COMMENT 'healpix pixel id using order = 29 in NESTED format',
`vx_gal` float COMMENT 'physical (peculiar) velocity in x-component (km/s)',
`vy_gal` float COMMENT 'physical (peculiar) velocity in y-component (km/s)',
`vz_gal` float COMMENT 'physical (peculiar) velocity in z-component (km/s)',
`vrad_gal` float COMMENT 'missing comment',
`delta_r` float COMMENT 'missing comment',
`abs_mag_r01` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h)',
`luminosity_r01` float COMMENT 'luminosity in the 0.1-r-band',
`abs_mag_r01_evolved` float COMMENT 'absolute magnitude in the 0.1-r-band: abs_mag_r01 = Mr-5log(h) including MICECATv2.0 magnitude evolution',
`sdss_r01` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1',
`abs_mag_uv_dereddened` float COMMENT 'absolute magnitude ultraviolet',
`gr_restframe` float COMMENT '(g-r) rest-frame at z=0',
`color_kind` tinyint COMMENT 'galaxy color kind: 0=red sequence 1=green valley 2=blue cloud',
`sed_ke` float COMMENT 'K-correction for each SED',
`gr_cosmos` float COMMENT 'interpolated COSMOS g-r color',
`sed_cosmos` float COMMENT 'interpolated basic COSMOS SED [0 30]',
`is_within_cosmos` boolean COMMENT 'missing comment',
`cosmos_distance` float COMMENT 'missing comment',
`ext_curve_cosmos` tinyint COMMENT 'COSMOS extinction curve [0-4]',
`ebv_cosmos` float COMMENT 'interpolated extinction strength for COSMOS extinction curve',
`log_ml_r01` float COMMENT 'logarithm of stellar_mass/luminosity ratio in the r01 SDSS band',
`log_stellar_mass` float COMMENT 'logarithm of the stellar mass (Msun/h)',
`metallicity` float COMMENT 'oxygen metallicity in units 12+logO/H',
`log_sfr` float COMMENT 'logarithm of star formation rate in (Msun/h)/year',
`logf_halpha` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: DOES NOT include extinction)',
`logf_halpha_ext` float COMMENT 'DEPRECATED logarithm of flux of Halpha (erg/cm**2/s) using empirical relation with UV magnitude (observed: includes extinction)',
`halpha_scatter` float COMMENT 'missing comment',
`loglum_halpha` float COMMENT 'DEPRECATED logarithm of Halpha luminosity (erg/s/h^2) using empirical relation with UV magnitude (DOES NOT include extinction)',
`loglum_halpha_ext` float COMMENT 'missing comment',
`logf_dummy` float COMMENT 'missing comment',
`z_dummy` float COMMENT 'missing comment',
`logf_halpha_model1_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: includes extinction)',
`logf_halpha_model1` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 1 (observed: DOES NOT include extinction)',
`loglum_halpha_model1_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (includes extinction)',
`loglum_halpha_model1` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 1 (DOES NOT include extinction)',
`logf_halpha_model3_ext` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: includes extinction)',
`logf_halpha_model3` float COMMENT 'logarithm of flux of Halpha (erg/cm**2/s) calibrated using Pozzetti model 3 (observed: DOES NOT include extinction)',
`loglum_halpha_model3_ext` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (includes extinction)',
`loglum_halpha_model3` float COMMENT 'logarithm of Halpha luminosity (erg/s/h^2) calibrated using Pozzetti model 3 (DOES NOT include extinction)',
`logf_hbeta_model1_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_hbeta_model1` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_n2_model1_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_n2_model1` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_o3_model1_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_o3_model1` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_s2_model1_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: includes extinction)',
`logf_s2_model1` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model1 (observed: DOES NOT include extinction)',
`logf_hbeta_model3_ext` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_hbeta_model3` float COMMENT 'logarithm of flux of Hbeta (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_n2_model3_ext` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_n2_model3` float COMMENT 'logarithm of flux of [NII] 6584 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_o3_model3_ext` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_o3_model3` float COMMENT 'logarithm of flux of [OIII] 5007 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`logf_s2_model3_ext` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: includes extinction)',
`logf_s2_model3` float COMMENT 'logarithm of flux of [SII] 6716 (erg/cm**2/s) using as input model3 (observed: DOES NOT include extinction)',
`2mass_h` float COMMENT 'observed flux in 2MASS H-band (Euclid provided)',
`2mass_j` float COMMENT 'observed flux in 2MASS J-band (Euclid provided)',
`2mass_ks` float COMMENT 'observed flux in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g` float COMMENT 'observed flux in DES g-band (Euclid provided)',
`blanco_decam_i` float COMMENT 'observed flux in DES i-band (Euclid provided)',
`blanco_decam_r` float COMMENT 'observed flux in DES r-band (Euclid provided)',
`blanco_decam_z` float COMMENT 'observed flux in DES z-band (Euclid provided)',
`cfht_megacam_r` float COMMENT 'observed flux in CFHT r-band (Euclid provided)',
`cfht_megacam_u` float COMMENT 'observed flux in CFHT u-band (Euclid provided)',
`euclid_nisp_h` float COMMENT 'observed flux in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j` float COMMENT 'observed flux in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y` float COMMENT 'observed flux in Euclid NISP-Y band (Euclid provided)',
`euclid_vis` float COMMENT 'observed flux in Euclid VIS band (Euclid provided)',
`gaia_bp` float COMMENT 'observed flux in Gaia-BP band (Euclid provided)',
`gaia_g` float COMMENT 'observed flux in Gaia-G band (Euclid provided)',
`gaia_rp` float COMMENT 'observed flux in Gaia-RP band (Euclid provided)',
`jst_jpcam_g` float COMMENT 'observed flux in JST g-band (Euclid provided)',
`kids_g` float COMMENT 'observed flux in KIDS g-band (Euclid provided)',
`kids_i` float COMMENT 'observed flux in KIDS i-band (Euclid provided)',
`kids_r` float COMMENT 'observed flux in KIDS r-band (Euclid provided)',
`kids_u` float COMMENT 'observed flux in KIDS u-band (Euclid provided)',
`lsst_g` float COMMENT 'observed flux in LSST g-band (Euclid provided)',
`lsst_i` float COMMENT 'observed flux in LSST i-band (Euclid provided)',
`lsst_r` float COMMENT 'observed flux in LSST r-band (Euclid provided)',
`lsst_u` float COMMENT 'observed flux in LSST u-band (Euclid provided)',
`lsst_y` float COMMENT 'observed flux in LSST Y-band (Euclid provided)',
`lsst_z` float COMMENT 'observed flux in LSST z-band (Euclid provided)',
`pan_starrs_i` float COMMENT 'observed flux in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z` float COMMENT 'observed flux in Pan-STARRS z-band (Euclid provided)',
`subaru_hsc_z` float COMMENT 'observed flux in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS H-band (Euclid provided)',
`2mass_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_odonnell_ext` float COMMENT 'observed flux with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_odonnell_ext` float COMMENT 'observed flux with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_odonnell_ext` float COMMENT 'observed flux with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in JST g-band (Euclid provided)',
`kids_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_odonnell_ext` float COMMENT 'observed flux in sdss r-band redshifted to z=0.1 with MW extinction',
`subaru_hsc_z_odonnell_ext` float COMMENT 'observed flux with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`2mass_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction 2MASS H-band (Euclid provided)',
`2mass_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS J-band (Euclid provided)',
`2mass_ks_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in 2MASS Ks-band (Euclid provided)',
`2mass_ks_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in 2MASS Ks-band (Euclid provided)',
`blanco_decam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES g-band (Euclid provided)',
`blanco_decam_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES i-band (Euclid provided)',
`blanco_decam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES r-band (Euclid provided)',
`blanco_decam_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in DES z-band (Euclid provided)',
`blanco_decam_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in DES z-band (Euclid provided)',
`cfht_megacam_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT r-band (Euclid provided)',
`cfht_megacam_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in CFHT u-band (Euclid provided)',
`cfht_megacam_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in CFHT u-band (Euclid provided)',
`euclid_nisp_h_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_h_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-H band (Euclid provided)',
`euclid_nisp_j_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_j_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-J band (Euclid provided)',
`euclid_nisp_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_nisp_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid NISP-Y band (Euclid provided)',
`euclid_vis_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Euclid VIS band (Euclid provided)',
`euclid_vis_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Euclid VIS band (Euclid provided)',
`gaia_bp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_bp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-BP band (Euclid provided)',
`gaia_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-G band (Euclid provided)',
`gaia_rp_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Gaia-RP band (Euclid provided)',
`gaia_rp_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Gaia-RP band (Euclid provided)',
`jst_jpcam_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in JST g-band (Euclid provided)',
`jst_jpcam_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in JST g-band (Euclid provided)',
`kids_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS g-band (Euclid provided)',
`kids_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS g-band (Euclid provided)',
`kids_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS i-band (Euclid provided)',
`kids_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS i-band (Euclid provided)',
`kids_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS r-band (Euclid provided)',
`kids_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS r-band (Euclid provided)',
`kids_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in KIDS u-band (Euclid provided)',
`kids_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in KIDS u-band (Euclid provided)',
`lsst_g_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST g-band (Euclid provided)',
`lsst_g_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST g-band (Euclid provided)',
`lsst_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST i-band (Euclid provided)',
`lsst_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST i-band (Euclid provided)',
`lsst_r_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST r-band (Euclid provided)',
`lsst_r_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST r-band (Euclid provided)',
`lsst_u_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST u-band (Euclid provided)',
`lsst_u_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST u-band (Euclid provided)',
`lsst_y_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_y_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST Y-band (Euclid provided)',
`lsst_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in LSST z-band (Euclid provided)',
`lsst_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in LSST z-band (Euclid provided)',
`pan_starrs_i_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_i_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS i-band (Euclid provided)',
`pan_starrs_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`pan_starrs_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in Pan-STARRS z-band (Euclid provided)',
`sdss_r01_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in sdss r-band redshifted to z=0.1',
`sdss_r01_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in sdss r-band redshifted to z=0.1',
`subaru_hsc_z_el_model1_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model1 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`subaru_hsc_z_el_model3_odonnell_ext` float COMMENT 'observed flux including E.L. fluxes using as input model3 with MW extinction in HYPER SUPRIME CAMERA SUBARU z-band (Euclid provided)',
`dominant_shape` tinyint COMMENT 'flag to define whether the galaxy is BULGE-dominated (0) or DISK-dominated (1)',
`bulge_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`disk_angle` float COMMENT 'position of the disk rotation axis (degrees) (assumption: bulge_angle = disk_angle). TU Galaxy stamps convention for OU-SIM: the standard is to set the position angle from North to East, with the major axis aligned in Declination',
`median_major_axis` float COMMENT 'median semi-major axis exponential scalength. Profile defined as: I = I_o x exp (-radius/scalelength)',
`scale_length` float COMMENT 'disc and bulge scalelength prior',
`bulge_fraction` float COMMENT 'ratio of the flux in the bulge component to the total flux (often written B/T)',
`disk_scalelength` float COMMENT 'scalength of the disk, profile defined as: I = I_o x exp (-radius/disk_scalelength)',
`disk_nsersic` float COMMENT 'Sersic index of the disk component',
`disk_r50` float COMMENT 'half light radius of the disk. For an exponential profile (or Sersec profile with index n=1), disk_r50 = disk_scalelength * 1.678',
`bulge_r50` float COMMENT 'half light radius of the bulge',
`bulge_nsersic` float COMMENT 'Sersic index of the bulge component',
`disk_ellipticity` float COMMENT 'ellipticity of the disk defined as disk_ellipticity = (1 - disk_axis_ratio) / (1 + disk_axis_ratio)',
`bulge_ellipticity` float COMMENT 'ellipticity of the bulge defined as bulge_ellipticity = (1 - bulge_axis_ratio) / (1 + bulge_axis_ratio)',
`disk_axis_ratio` float COMMENT 'disk projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`bulge_axis_ratio` float COMMENT 'bulge projected axis ratio defined as b/a (b: semi-minor axis, a: semi-major axis)',
`inclination_angle` float COMMENT 'Galaxy inclination angle (where 0 degrees = face-on and 90 degrees = edge-on). Galaxy ellipticity for disk and bulge components are computed following the recipe in https://euclid.roe.ac.uk/projects/sgsshear/wiki/SHE-SIM',
`halo_x` float COMMENT 'halo comoving x coordinate (Mpc/h)',
`halo_y` float COMMENT 'halo comoving y coordinate (Mpc/h)',
`halo_z` float COMMENT 'halo comoving z coordinate (Mpc/h)',
`halo_vx` float COMMENT 'halo x-centre velocity (km/h)',
`halo_vy` float COMMENT 'halo y-centre velocity (km/h)',
`halo_vz` float COMMENT 'halo z-centre velocity (km/h)',
`halo_r` float COMMENT 'halo comoving distance (Mpc/h)',
`true_redshift_halo` float COMMENT 'true redshift of the host halo',
`halo_lm` float COMMENT 'log10 of the FoF halo mass in Msum/h. This halo mass is computed as the particule mass multiplied by the number of particles of the FoF halo. At low masses it is interpolated to obtain a smooth distribution of halo masses',
`halo_n_sats` int COMMENT 'number of satellite galaxies from HOD',
`n_gals` int COMMENT 'DEPRECATED',
`luminosity_r01_evolved` float COMMENT 'missing comment',
`hpix_13_nest` bigint COMMENT 'missing comment')
PARTITIONED BY (
`step` smallint COMMENT 'step number in the Flagship lensing maps')
CLUSTERED BY (
hpix_13_nest)
SORTED BY (
hpix_13_nest ASC)
INTO 128 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;
OJO PORQUE LE CAMBIO EL SIGNO AL GAMMA1 para que NO SE TENGA QUE HACER A NIVEL DE FITS FILES!!!! :
INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_1717_s_v3 PARTITION(step) SELECT halo_id, galaxy_id, kind, random_index, ra_gal, dec_gal, ra_mag_gal, dec_mag_gal, kappa, -gamma1 as gamma1, gamma2, l_gal, b_gal, mw_extinction, x_gal, y_gal, z_gal, r_gal, true_redshift_gal, observed_redshift_gal, hpix_29_nest, vx_gal, vy_gal, vz_gal, vrad_gal, delta_r, abs_mag_r01, luminosity_r01, abs_mag_r01_evolved, sdss_r01, abs_mag_uv_dereddened, gr_restframe, color_kind, sed_ke, gr_cosmos, sed_cosmos, is_within_cosmos, cosmos_distance, ext_curve_cosmos, ebv_cosmos, log_ml_r01, log_stellar_mass, metallicity, log_sfr, logf_halpha, logf_halpha_ext, halpha_scatter, loglum_halpha, loglum_halpha_ext, logf_dummy, z_dummy, logf_halpha_model1_ext, logf_halpha_model1, loglum_halpha_model1_ext, loglum_halpha_model1, logf_halpha_model3_ext, logf_halpha_model3, loglum_halpha_model3_ext, loglum_halpha_model3, logf_hbeta_model1_ext, logf_hbeta_model1, logf_o2_model1_ext, logf_o2_model1, logf_n2_model1_ext, logf_n2_model1, logf_o3_model1_ext, logf_o3_model1, logf_s2_model1_ext, logf_s2_model1, logf_hbeta_model3_ext, logf_hbeta_model3, logf_o2_model3_ext, logf_o2_model3, logf_n2_model3_ext, logf_n2_model3, logf_o3_model3_ext, logf_o3_model3, logf_s2_model3_ext, logf_s2_model3, 2mass_h, 2mass_j, 2mass_ks, blanco_decam_g, blanco_decam_i, blanco_decam_r, blanco_decam_z, cfht_megacam_r, cfht_megacam_u, euclid_nisp_h, euclid_nisp_j, euclid_nisp_y, euclid_vis, gaia_bp, gaia_g, gaia_rp, jst_jpcam_g, kids_g, kids_i, kids_r, kids_u, lsst_g, lsst_i, lsst_r, lsst_u, lsst_y, lsst_z, pan_starrs_i, pan_starrs_z, subaru_hsc_z, 2mass_h_odonnell_ext, 2mass_j_odonnell_ext, 2mass_ks_odonnell_ext, blanco_decam_g_odonnell_ext, blanco_decam_i_odonnell_ext, blanco_decam_r_odonnell_ext, blanco_decam_z_odonnell_ext, cfht_megacam_r_odonnell_ext, cfht_megacam_u_odonnell_ext, euclid_nisp_h_odonnell_ext, euclid_nisp_j_odonnell_ext, euclid_nisp_y_odonnell_ext, euclid_vis_odonnell_ext, gaia_bp_odonnell_ext, gaia_g_odonnell_ext, gaia_rp_odonnell_ext, jst_jpcam_g_odonnell_ext, kids_g_odonnell_ext, kids_i_odonnell_ext, kids_r_odonnell_ext, kids_u_odonnell_ext, lsst_g_odonnell_ext, lsst_i_odonnell_ext, lsst_r_odonnell_ext, lsst_u_odonnell_ext, lsst_y_odonnell_ext, lsst_z_odonnell_ext, pan_starrs_i_odonnell_ext, pan_starrs_z_odonnell_ext, sdss_r01_odonnell_ext, subaru_hsc_z_odonnell_ext, 2mass_h_el_model1_odonnell_ext, 2mass_h_el_model3_odonnell_ext, 2mass_j_el_model1_odonnell_ext, 2mass_j_el_model3_odonnell_ext, 2mass_ks_el_model1_odonnell_ext, 2mass_ks_el_model3_odonnell_ext, blanco_decam_g_el_model1_odonnell_ext, blanco_decam_g_el_model3_odonnell_ext, blanco_decam_i_el_model1_odonnell_ext, blanco_decam_i_el_model3_odonnell_ext, blanco_decam_r_el_model1_odonnell_ext, blanco_decam_r_el_model3_odonnell_ext, blanco_decam_z_el_model1_odonnell_ext, blanco_decam_z_el_model3_odonnell_ext, cfht_megacam_r_el_model1_odonnell_ext, cfht_megacam_r_el_model3_odonnell_ext, cfht_megacam_u_el_model1_odonnell_ext, cfht_megacam_u_el_model3_odonnell_ext, euclid_nisp_h_el_model1_odonnell_ext, euclid_nisp_h_el_model3_odonnell_ext, euclid_nisp_j_el_model1_odonnell_ext, euclid_nisp_j_el_model3_odonnell_ext, euclid_nisp_y_el_model1_odonnell_ext, euclid_nisp_y_el_model3_odonnell_ext, euclid_vis_el_model1_odonnell_ext, euclid_vis_el_model3_odonnell_ext, gaia_bp_el_model1_odonnell_ext, gaia_bp_el_model3_odonnell_ext, gaia_g_el_model1_odonnell_ext, gaia_g_el_model3_odonnell_ext, gaia_rp_el_model1_odonnell_ext, gaia_rp_el_model3_odonnell_ext, jst_jpcam_g_el_model1_odonnell_ext, jst_jpcam_g_el_model3_odonnell_ext, kids_g_el_model1_odonnell_ext, kids_g_el_model3_odonnell_ext, kids_i_el_model1_odonnell_ext, kids_i_el_model3_odonnell_ext, kids_r_el_model1_odonnell_ext, kids_r_el_model3_odonnell_ext, kids_u_el_model1_odonnell_ext, kids_u_el_model3_odonnell_ext, lsst_g_el_model1_odonnell_ext, lsst_g_el_model3_odonnell_ext, lsst_i_el_model1_odonnell_ext, lsst_i_el_model3_odonnell_ext, lsst_r_el_model1_odonnell_ext, lsst_r_el_model3_odonnell_ext, lsst_u_el_model1_odonnell_ext, lsst_u_el_model3_odonnell_ext, lsst_y_el_model1_odonnell_ext, lsst_y_el_model3_odonnell_ext, lsst_z_el_model1_odonnell_ext, lsst_z_el_model3_odonnell_ext, pan_starrs_i_el_model1_odonnell_ext, pan_starrs_i_el_model3_odonnell_ext, pan_starrs_z_el_model1_odonnell_ext, pan_starrs_z_el_model3_odonnell_ext, sdss_r01_el_model1_odonnell_ext, sdss_r01_el_model3_odonnell_ext, subaru_hsc_z_el_model1_odonnell_ext, subaru_hsc_z_el_model3_odonnell_ext, dominant_shape, bulge_angle, disk_angle, median_major_axis, scale_length, bulge_fraction, disk_scalelength, disk_nsersic, disk_r50, bulge_r50, bulge_nsersic, disk_ellipticity, bulge_ellipticity, disk_axis_ratio, bulge_axis_ratio, inclination_angle, halo_x, halo_y, halo_z, halo_vx, halo_vy, halo_vz, halo_r, true_redshift_halo, halo_lm, halo_n_sats, n_gals, luminosity_r01_evolved, hpix_13_nest, step FROM jcarrete.flagship_mock_obsid_1717_s_v3_pq;
ANALYZE TABLE jcarrete.flagship_mock_obsid_1717_s_v3 COMPUTE STATISTICS FOR COLUMNS;
Ahora puedo crear los FITS files:
CREATE TABLE jcarrete.flagship_mock_obsid_1717_s_v3_magnified_fits(
`source_id` bigint COMMENT 'from deserializer',
`halo_id` bigint COMMENT 'from deserializer',
`ra` float COMMENT 'from deserializer',
`dec` float COMMENT 'from deserializer',
`ra_mag` float COMMENT 'from deserializer',
`dec_mag` float COMMENT 'from deserializer',
`z_obs` float COMMENT 'from deserializer',
`ref_mag_abs` float COMMENT 'from deserializer',
`ref_mag` float COMMENT 'from deserializer',
`bulge_fraction` float COMMENT 'from deserializer',
`bulge_r50` float COMMENT 'from deserializer',
`disk_r50` float COMMENT 'from deserializer',
`bulge_nsersic` float COMMENT 'from deserializer',
`inclination_angle` float COMMENT 'from deserializer',
`disk_angle` float COMMENT 'from deserializer',
`kappa` float COMMENT 'from deserializer',
`gamma1` float COMMENT 'from deserializer',
`gamma2` float COMMENT 'from deserializer',
`sed_template` float COMMENT 'from deserializer',
`ext_law` smallint COMMENT 'from deserializer',
`ebv` float COMMENT 'from deserializer',
`halpha_logflam_ext` float COMMENT 'from deserializer',
`hbeta_logflam_ext` float COMMENT 'from deserializer',
`o2_logflam_ext` float COMMENT 'from deserializer',
`o3_logflam_ext` float COMMENT 'from deserializer',
`n2_logflam_ext` float COMMENT 'from deserializer',
`s2_logflam_ext` float COMMENT 'from deserializer',
`av` float COMMENT 'from deserializer',
`tu_fnu_vis` float COMMENT 'from deserializer',
`tu_fnu_y_nisp` float COMMENT 'from deserializer',
`tu_fnu_j_nisp` float COMMENT 'from deserializer',
`tu_fnu_h_nisp` float COMMENT 'from deserializer',
`tu_fnu_g_decam` float COMMENT 'from deserializer',
`tu_fnu_r_decam` float COMMENT 'from deserializer',
`tu_fnu_i_decam` float COMMENT 'from deserializer',
`tu_fnu_z_decam` float COMMENT 'from deserializer',
`tu_fnu_u_megacam` float COMMENT 'from deserializer',
`tu_fnu_r_megacam` float COMMENT 'from deserializer',
`tu_fnu_g_jpcam` float COMMENT 'from deserializer',
`tu_fnu_i_panstarrs` float COMMENT 'from deserializer',
`tu_fnu_z_panstarrs` float COMMENT 'from deserializer',
`tu_fnu_z_hsc` float COMMENT 'from deserializer',
`tu_fnu_g_gaia` float COMMENT 'from deserializer',
`tu_fnu_bp_gaia` float COMMENT 'from deserializer',
`tu_fnu_rp_gaia` float COMMENT 'from deserializer',
`tu_fnu_u_lsst` float COMMENT 'from deserializer',
`tu_fnu_g_lsst` float COMMENT 'from deserializer',
`tu_fnu_r_lsst` float COMMENT 'from deserializer',
`tu_fnu_i_lsst` float COMMENT 'from deserializer',
`tu_fnu_z_lsst` float COMMENT 'from deserializer',
`tu_fnu_y_lsst` float COMMENT 'from deserializer',
`tu_fnu_u_kids` float COMMENT 'from deserializer',
`tu_fnu_g_kids` float COMMENT 'from deserializer',
`tu_fnu_r_kids` float COMMENT 'from deserializer',
`tu_fnu_i_kids` float COMMENT 'from deserializer',
`tu_fnu_j_2mass` float COMMENT 'from deserializer',
`tu_fnu_h_2mass` float COMMENT 'from deserializer',
`tu_fnu_ks_2mass` float COMMENT 'from deserializer')
PARTITIONED BY (
`hpix_5_nest` bigint)
ROW FORMAT SERDE
'es.pic.astro.hadoop.serde.RecArraySerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'es.pic.astro.hadoop.io.BinaryOutputFormat'
;
INSERT OVERWRITE TABLE jcarrete.flagship_mock_obsid_1717_s_v3_magnified_fits PARTITION(hpix_5_nest)
SELECT CAST(((halo_id * 10000) + galaxy_id) AS bigint) AS SOURCE_ID,
CAST(halo_id AS bigint) AS HALO_ID,
CAST(ra_gal AS float) AS RA,
CAST(dec_gal AS float) AS `DEC`,
CAST(ra_mag_gal AS float) AS RA_MAG,
CAST(dec_mag_gal AS float) AS DEC_MAG,
CAST(observed_redshift_gal AS float) AS Z_OBS,
CAST(abs_mag_r01_evolved AS float) AS REF_MAG_ABS,
CAST(-2.5*log10(sdss_r01) - 48.6 AS float) AS REF_MAG,
CAST(bulge_fraction AS float) AS BULGE_FRACTION,
CAST(bulge_r50 AS float) AS BULGE_R50,
CAST(disk_r50 AS float) AS DISK_R50,
CAST(bulge_nsersic AS float) AS BULGE_NSERSIC,
CAST(inclination_angle AS float) AS INCLINATION_ANGLE,
CAST(disk_angle AS float) AS DISK_ANGLE,
CAST(kappa AS float) AS KAPPA,
CAST(gamma1 AS float) AS GAMMA1,
CAST(gamma2 AS float) AS GAMMA2,
CAST(sed_cosmos AS float) AS SED_TEMPLATE,
CAST(ROUND(ext_curve_cosmos) AS smallint) AS EXT_LAW,
CAST(ebv_cosmos AS float) AS EBV,
CAST(logf_halpha_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HALPHA_LOGFLAM_EXT,
CAST(logf_hbeta_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HBETA_LOGFLAM_EXT,
CAST(logf_o2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O2_LOGFLAM_EXT,
CAST(logf_o3_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O3_LOGFLAM_EXT,
CAST(logf_n2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS N2_LOGFLAM_EXT,
CAST(logf_s2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS S2_LOGFLAM_EXT,
CAST(mw_extinction AS float) AS AV,
CAST(euclid_vis_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_VIS,
CAST(euclid_nisp_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_NISP,
CAST(euclid_nisp_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_NISP,
CAST(euclid_nisp_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_NISP,
CAST(blanco_decam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_DECAM,
CAST(blanco_decam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_DECAM,
CAST(blanco_decam_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_DECAM,
CAST(blanco_decam_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_DECAM,
CAST(cfht_megacam_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_MEGACAM,
CAST(cfht_megacam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_MEGACAM,
CAST(jst_jpcam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_JPCAM,
CAST(pan_starrs_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_PANSTARRS,
CAST(pan_starrs_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_PANSTARRS,
CAST(subaru_hsc_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_HSC,
CAST(gaia_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_GAIA,
CAST(gaia_bp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_BP_GAIA,
CAST(gaia_rp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_RP_GAIA,
CAST(lsst_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_LSST,
CAST(lsst_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_LSST,
CAST(lsst_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_LSST,
CAST(lsst_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_LSST,
CAST(lsst_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_LSST,
CAST(lsst_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_LSST,
CAST(kids_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_KIDS,
CAST(kids_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_KIDS,
CAST(kids_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_KIDS,
CAST(kids_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_KIDS,
CAST(2mass_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_2MASS,
CAST(2mass_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_2MASS,
CAST(2mass_ks_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_KS_2MASS,
CAST(SHIFTRIGHT(hpix_29_nest, (29-5)*2) AS bigint) AS hpix_5_nest
FROM jcarrete.flagship_mock_obsid_1717_s_v3
WHERE (logf_halpha_model3_ext > -16 OR -2.5*log10(euclid_nisp_h) - 48.6 < 26)
AND disk_axis_ratio > 0.10865
;
Añadimos los header con el notebook: add_fits_header_to_hive_table.py
En este caso vamos a cambiar la nomenclatura de las versiones en el webdavdoor.
El catálogo del que partimos es el 1.9.8. Se trata de un optante. Pero las shapes están mal. Las corrijo, así que la versión sería 1.9.9
Llamo al directorio: flagship_mock_1_9_9_obsid_1717_v3
Paso la tabla ORC al esquema cosmohub: OJO, le meto la versión 1_9_9!!!!!!
ALTER TABLE jcarrete.flagship_mock_obsid_1717_s_v3 RENAME TO cosmohub.flagship_mock_1_9_9_obsid_1717_s_v3;
ANALYZE TABLE cosmohub.flagship_mock_1_9_9_obsid_1717_s_v3 COMPUTE STATISTICS FOR COLUMNS;
Description in CosmoHub
Dark matter halo input catalog from flagship_rockstar_octant1_c, which is the Flagship dark matter halo catalog v1.1 with correct velocities.
The covered area of the SWF2 observation (obsid = 1717) is:
215.46 < ra_gal < 241.33
73.59 < dec_gal < 79.5904
- Shape distributions are completely different from all previous versions (including new recipe for inclination_angle field)**.
In this version a new python method for inclination_angle was developed by Francisco and ellipticities and axis ratios come from Eric's formulas.
See the following [url](https://docs.google.com/document/d/1FRERcl5Lv2IDB6ZChTK6gcEHI2FWA9gSgYKcaDUA9EM/edit?usp=sharing) for more details.
Note that **we provide fluxes instead of magnitudes**. In order **to get magnitudes** you can enter in the "Expert mode" and estimate the magnitude as follows:
m = -2.5 * log10(flux) - 48.6
And in order to estimate magnified magnitudes:
m_mag = m + 2.5 * log10((1-kappa)*(1-kappa) - gamma1 * gamma1 - gamma2 * gamma2)
{
"sql": "SELECT CAST(((halo_id * 10000) + galaxy_id) AS bigint) AS SOURCE_ID, \nCAST(halo_id AS bigint) AS HALO_ID, \nCAST(ra_gal AS float) AS RA, \nCAST(dec_gal AS float) AS `DEC`, \nCAST(ra_mag_gal AS float) AS RA_MAG, \nCAST(dec_mag_gal AS float) AS DEC_MAG, \nCAST(observed_redshift_gal AS float) AS Z_OBS, \nCAST(abs_mag_r01_evolved AS float) AS REF_MAG_ABS, \nCAST(-2.5*log10(sdss_r01) - 48.6 AS float) AS REF_MAG, \nCAST(bulge_fraction AS float) AS BULGE_FRACTION, \nCAST(bulge_r50 AS float) AS BULGE_R50, \nCAST(disk_r50 AS float) AS DISK_R50, \nCAST(bulge_nsersic AS float) AS BULGE_NSERSIC, \nCAST(inclination_angle AS float) AS INCLINATION_ANGLE, \nCAST(disk_angle AS float) AS DISK_ANGLE, \nCAST(kappa AS float) AS KAPPA, \nCAST(gamma1 AS float) AS GAMMA1, \nCAST(gamma2 AS float) AS GAMMA2, \nCAST(sed_cosmos AS float) AS SED_TEMPLATE, \nCAST(ROUND(ext_curve_cosmos) AS smallint) AS EXT_LAW, \nCAST(ebv_cosmos AS float) AS EBV, \nCAST(logf_halpha_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HALPHA_LOGFLAM_EXT_MAG, \nCAST(logf_hbeta_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS HBETA_LOGFLAM_EXT_MAG, \nCAST(logf_o2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O2_LOGFLAM_EXT_MAG, \nCAST(logf_o3_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS O3_LOGFLAM_EXT_MAG, \nCAST(logf_n2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS N2_LOGFLAM_EXT_MAG, \nCAST(logf_s2_model3_ext + log10((1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2))) AS float) AS S2_LOGFLAM_EXT_MAG, \nCAST(mw_extinction AS float) AS AV,\nCAST(euclid_vis_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_VIS_MAG, \nCAST(euclid_nisp_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_NISP_MAG, \nCAST(euclid_nisp_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_NISP_MAG, \nCAST(euclid_nisp_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_NISP_MAG, \nCAST(blanco_decam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_DECAM_MAG, \nCAST(blanco_decam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_DECAM_MAG, \nCAST(blanco_decam_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_DECAM_MAG, \nCAST(blanco_decam_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_DECAM_MAG, \nCAST(cfht_megacam_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_MEGACAM_MAG, \nCAST(cfht_megacam_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_MEGACAM_MAG, \nCAST(jst_jpcam_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_JPCAM_MAG, \nCAST(pan_starrs_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_PANSTARRS_MAG, \nCAST(pan_starrs_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_PANSTARRS_MAG, \nCAST(subaru_hsc_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_HSC_MAG, \nCAST(gaia_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_GAIA_MAG, \nCAST(gaia_bp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_BP_GAIA_MAG, \nCAST(gaia_rp_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_RP_GAIA_MAG, \nCAST(lsst_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_LSST_MAG, \nCAST(lsst_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_LSST_MAG, \nCAST(lsst_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_LSST_MAG, \nCAST(lsst_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_LSST_MAG, \nCAST(lsst_z_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Z_LSST_MAG, \nCAST(lsst_y_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_Y_LSST_MAG, \nCAST(kids_u_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_U_KIDS_MAG, \nCAST(kids_g_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_G_KIDS_MAG, \nCAST(kids_r_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_R_KIDS_MAG, \nCAST(kids_i_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_I_KIDS_MAG, \nCAST(2mass_j_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_J_2MASS_MAG, \nCAST(2mass_h_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_H_2MASS_MAG, \nCAST(2mass_ks_el_model3_odonnell_ext*1.e23*(1/((1 - kappa)*(1 - kappa) - gamma1*gamma1 - gamma2*gamma2)) AS float) AS TU_FNU_KS_2MASS_MAG \nFROM cosmohub.flagship_mock_1_9_9_obsid_1717_s_v3 \nWHERE (logf_halpha_model3_ext > -16 OR -2.5*log10(euclid_nisp_h) - 48.6 < 26) \nAND disk_axis_ratio > 0.10865 \nAND SHIFTRIGHT(hpix_29_nest, (29-5)*2) = 1008"
}
Generating Deep field with lensing parameters from 1.9.8!!!
CREATE TABLE jcarrete.flagship_mock_gsir_nir_self_calibration_c2(
`kind` BIGINT,
`luminosity_r01` DOUBLE,
`halo_x` DOUBLE,
`halo_y` DOUBLE,
`halo_z` DOUBLE,
`halo_vx` DOUBLE,
`halo_vy` DOUBLE,
`halo_vz` DOUBLE,
`halo_r` DOUBLE,
`true_redshift_halo` DOUBLE,
`halo_lm` DOUBLE,
`halo_n_sats` BIGINT,
`n_gals` BIGINT,
`abs_mag_r01` DOUBLE,
`abs_mag_r01_evolved` DOUBLE,
`luminosity_r01_evolved` DOUBLE,
`gr_restframe` DOUBLE,
`color_kind` INT,
`x_gal` DOUBLE,
`y_gal` DOUBLE,
`z_gal` DOUBLE,
`r_gal` DOUBLE,
`true_redshift_gal` DOUBLE,
`ra_gal` DOUBLE,
`dec_gal` DOUBLE,
`hpix_29_nest` BIGINT,
`hpix_13_nest` BIGINT,
`vx_gal` DOUBLE,
`vy_gal` DOUBLE,
`vz_gal` DOUBLE,
`vrad_gal` DOUBLE,
`delta_r` DOUBLE,
`observed_redshift_gal` DOUBLE,
`sed_ke` DOUBLE,
`gr_cosmos` DOUBLE,
`sed_cosmos` DOUBLE,
`ext_curve_cosmos` BIGINT,
`ebv_cosmos` DOUBLE,
`is_within_cosmos` BOOLEAN,
`cosmos_distance` DOUBLE,
`abs_mag_uv_dereddened` DOUBLE,
`log_ml_r01` DOUBLE,
`log_stellar_mass` DOUBLE,
`metallicity` DOUBLE,
`log_sfr` DOUBLE,
`logf_halpha_ext` DOUBLE,
`logf_halpha` DOUBLE,
`Halpha_scatter` DOUBLE,
`loglum_halpha` DOUBLE,
`loglum_halpha_ext` DOUBLE,
`logf_dummy` DOUBLE,
`z_dummy` DOUBLE,
`logf_halpha_model1_ext` DOUBLE,
`logf_halpha_model1` DOUBLE,
`loglum_halpha_model1_ext` DOUBLE,
`loglum_halpha_model1` DOUBLE,
`logf_halpha_model3_ext` DOUBLE,
`logf_halpha_model3` DOUBLE,
`loglum_Halpha_model3_ext` DOUBLE,
`loglum_Halpha_model3` DOUBLE,
`logf_hbeta_model1_ext` DOUBLE,
`logf_hbeta_model1` DOUBLE,
`logf_o2_model1_ext` DOUBLE,
`logf_o2_model1` DOUBLE,
`logf_n2_model1_ext` DOUBLE,
`logf_n2_model1` DOUBLE,
`logf_o3_model1_ext` DOUBLE,
`logf_o3_model1` DOUBLE,
`logf_s2_model1_ext` DOUBLE,
`logf_s2_model1` DOUBLE,
`logf_hbeta_model3_ext` DOUBLE,
`logf_hbeta_model3` DOUBLE,
`logf_o2_model3_ext` DOUBLE,
`logf_o2_model3` DOUBLE,
`logf_n2_model3_ext` DOUBLE,
`logf_n2_model3` DOUBLE,
`logf_o3_model3_ext` DOUBLE,
`logf_o3_model3` DOUBLE,
`logf_s2_model3_ext` DOUBLE,
`logf_s2_model3` DOUBLE,
`blanco_decam_i` DOUBLE,
`sdss_r01` DOUBLE,
`euclid_nisp_h` DOUBLE,
`l_gal` DOUBLE,
`b_gal` DOUBLE,
`mw_extinction` DOUBLE,
`blanco_decam_i_odonnell_ext` DOUBLE,
`euclid_nisp_h_odonnell_ext` DOUBLE,
`sdss_r01_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model1_odonnell_ext` DOUBLE,
`blanco_decam_i_el_model3_odonnell_ext` DOUBLE,
`sdss_r01_el_model1_odonnell_ext` DOUBLE,
`sdss_r01_el_model3_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model1_odonnell_ext` DOUBLE,
`euclid_nisp_h_el_model3_odonnell_ext` DOUBLE,
`dominant_shape` BIGINT,
`bulge_angle` DOUBLE,
`disk_angle` DOUBLE,
`median_major_axis` DOUBLE,
`scale_length` DOUBLE,
`bulge_fraction` DOUBLE,
`disk_scalelength` DOUBLE,
`disk_nsersic` DOUBLE,
`disk_r50` DOUBLE,
`bulge_r50` DOUBLE,
`bulge_nsersic` DOUBLE,
`inclination_angle` DOUBLE,
`disk_axis_ratio` DOUBLE,
`bulge_axis_ratio` DOUBLE,
`disk_ellipticity` DOUBLE,
`bulge_ellipticity` DOUBLE,
`random_index` DOUBLE,
`halo_id` BIGINT,
`galaxy_id` BIGINT
)
PARTITIONED BY (
`step` smallint
)
CLUSTERED BY (
hpix_13_nest
)
SORTED BY (
hpix_13_nest ASC
)
INTO 128 BUCKETS
STORED AS ORC
;
INSERT OVERWRITE TABLE jcarrete.flagship_mock_gsir_nir_self_calibration_c2 SELECT `kind`, `luminosity_r01`, `halo_x`, `halo_y`, `halo_z`, `halo_vx`, `halo_vy`, `halo_vz`, `halo_r`, `true_redshift_halo`, `halo_lm`, `halo_n_sats`, `n_gals`, `abs_mag_r01`, `abs_mag_r01_evolved`, `luminosity_r01_evolved`, `gr_restframe`, `color_kind`, `x_gal`, `y_gal`, `z_gal`, `r_gal`, `true_redshift_gal`, `ra_gal`, `dec_gal`, `hpix_nest`, SHIFTRIGHT(`hpix_nest`, (29-13)*2), `vx_gal`, `vy_gal`, `vz_gal`, `vrad_gal`, `delta_r`, `observed_redshift_gal`, `sed_ke`, `gr_cosmos`, `sed_cosmos`, `ext_curve_cosmos`, `ebv_cosmos`, `is_within_cosmos`, `cosmos_distance`, `abs_mag_uv_dereddened`, `log_ml_r01`, `log_stellar_mass`, `metallicity`, `log_sfr`, `logf_halpha_ext`, `logf_halpha`, `Halpha_scatter`, `loglum_halpha`, `loglum_halpha_ext`, `logf_dummy`, `z_dummy`, `logf_halpha_model1_ext`, `logf_halpha_model1`, `loglum_halpha_model1_ext`, `loglum_halpha_model1`, `logf_halpha_model3_ext`, `logf_halpha_model3`, `loglum_Halpha_model3_ext`, `loglum_Halpha_model3`, `logf_hbeta_model1_ext`, `logf_hbeta_model1`, `logf_o2_model1_ext`, `logf_o2_model1`, `logf_n2_model1_ext`, `logf_n2_model1`, `logf_o3_model1_ext`, `logf_o3_model1`, `logf_s2_model1_ext`, `logf_s2_model1`, `logf_hbeta_model3_ext`, `logf_hbeta_model3`, `logf_o2_model3_ext`, `logf_o2_model3`, `logf_n2_model3_ext`, `logf_n2_model3`, `logf_o3_model3_ext`, `logf_o3_model3`, `logf_s2_model3_ext`, `logf_s2_model3`, `blanco_decam_i`, `sdss_r01`, `euclid_nisp_h`, `l_gal`, `b_gal`, `mw_extinction`, `blanco_decam_i_odonnell_ext`, `euclid_nisp_h_odonnell_ext`, `sdss_r01_odonnell_ext`, `blanco_decam_i_el_model1_odonnell_ext`, `blanco_decam_i_el_model3_odonnell_ext`, `sdss_r01_el_model1_odonnell_ext`, `sdss_r01_el_model3_odonnell_ext`, `euclid_nisp_h_el_model1_odonnell_ext`, `euclid_nisp_h_el_model3_odonnell_ext`, `dominant_shape`, `bulge_angle`, `disk_angle`, `median_major_axis`, `scale_length`, `bulge_fraction`, `disk_scalelength`, `disk_nsersic`, `disk_r50`, `bulge_r50`, `bulge_nsersic`, `inclination_angle`, `disk_axis_ratio`, `bulge_axis_ratio`, `disk_ellipticity`, `bulge_ellipticity`, `random_index`, `halo_id`, `galaxy_id`, `step` FROM jcarrete.flagship_mock_1_9_8_c WHERE (ra_gal >= 7.678307967139321 AND ra_gal < 14.008504032860678 AND dec_gal >= 15.587506 AND dec_gal < 21.587506) OR (ra_gal >= 35.45815488783862 AND ra_gal < 61.33016111216139 AND dec_gal >=73.5904 AND dec_gal < 79.5904) ;
ANALYZE TABLE jcarrete.flagship_mock_gsir_nir_self_calibration_c2 COMPUTE STATISTICS FOR COLUMNS;
jcarrete.flagship_mock_gsir_nir_self_calibration_c2
jcarrete.flagship_mock_gsir_nir_self_calibration_s
Notebooks with kappa string:
- Checking_lensing_procedure.ipynb
- assign_lensing_Flagship_1_8_3.ipynb
- assign_lensing.ipynb
- assign_lensing_pipeline_integration.ipynb
- Checking_fluxes_EL_MW.ipynb
- Checking_true_flux_SciPIC_vs_Euclid.ipynb
- DES_Y3_MICE2_source_catalog_IA_mocks.ipynb
- Euclid_bug_10759_11158_10141.ipynb
- Ingest_lensing_maps_into_Big_Data_platform.ipynb
- Lensing_test_new.ipynb
- MICECATv2.0_tests_for_SC2.ipynb
- micecat_v2_0_thumbnails_catalog.ipynb
- pruebas.ipynb
- shape_pipeline.ipynb
- thumbnail_euclid_problem.ipynb
- Thumbnail_library_test.ipynb
https://math.stackexchange.com/questions/2484830/how-to-switch-y-and-z-axis-of-a-rotation-matrix