Euclid mock production v1.7.17 correct

I estimate the correct morphological parameters for version 1.7.17, which contains several issues. The problems were reported in the Euclid redmine.

In order to correct them I use the notebook:

   /nfs/pic.es/user/j/jcarrete/python_notebooks/Euclid_mock_pipeline_production-v.1.7.17_correct.ipynb

With the notebook I create several parquet files with the correct shape parameters.

Then the JOIN with the table cosmohub.flagship_sc456_1_7_17_s does not work and we do not know exactly why.

However, a "simple" JOIN filtering with one of the healpix_id DOES work.

I copy here one of the examples run in CosmoHub that worked:

SELECT
    CAST( ((gal.halo_id * 10000) + gal.galaxy_id) AS bigint) AS SOURCE_ID,
    CAST( gal.ra_gal                              AS float) AS RA,
    CAST( gal.dec_gal                             AS float) AS DEC,
    CAST( gal.ra_gal_mag                          AS float) AS RA_MAG,
    CAST( gal.dec_gal_mag                         AS float) AS DEC_MAG,
    CAST( gal.observed_redshift_gal               AS float) AS Z_OBS,
    CAST( gal.abs_mag_r01_evolved                 AS float) AS TU_MAG_R01_SDSS_ABS,
    CAST( -2.5*log10(gal.sdss_r01) - 48.6         AS float) AS TU_MAG_R01_SDSS,
    CAST( gal.sed_cosmos                          AS float) AS SED_TEMPLATE,
    CAST( ROUND(gal.ext_curve_cosmos)             AS smallint) AS EXT_LAW,
    CAST( gal.ebv_cosmos                          AS float) AS EBV,
    CAST( gal.logf_halpha_model3_ext              AS float) AS HALPHA_LOGFLAM_EXT,
    CAST( gal.logf_hbeta_model3_ext               AS float) AS HBETA_LOGFLAM_EXT,
    CAST( gal.logf_o2_model3_ext                  AS float) AS O2_LOGFLAM_EXT,
    CAST( gal.logf_o3_model3_ext                  AS float) AS O3_LOGFLAM_EXT,
    CAST( gal.logf_n2_model3_ext                  AS float) AS N2_LOGFLAM_EXT,
    CAST( gal.logf_s2_model3_ext                  AS float) AS S2_LOGFLAM_EXT,
    CAST( shape.bulge_fraction                      AS float) AS BULGE_FRACTION,
    CAST( shape.bulge_length                        AS float) AS BULGE_LENGTH,
    CAST( shape.disk_length                         AS float) AS DISK_LENGTH,
    CAST( shape.disk_axis_ratio                     AS float) AS DISK_AXIS_RATIO,
    CAST( shape.disk_angle                          AS float) AS DISK_ANGLE,
    CAST( gal.kappa                               AS float) AS KAPPA,
    CAST( gal.gamma1                              AS float) AS GAMMA1,
    CAST( gal.gamma2                              AS float) AS GAMMA2,
    CAST( gal.mw_extinction                       AS float) AS AV,
    CAST( gal.euclid_vis_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_VIS,
    CAST( gal.euclid_nisp_y_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_Y_NISP,
    CAST( gal.euclid_nisp_j_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_J_NISP,
    CAST( gal.euclid_nisp_h_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_H_NISP,
    CAST( gal.blanco_decam_g_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_G_DECAM,
    CAST( gal.blanco_decam_r_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_R_DECAM,
    CAST( gal.blanco_decam_i_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_I_DECAM,
    CAST( gal.blanco_decam_z_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_Z_DECAM,
    CAST( gal.cfht_megacam_u_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_U_MEGACAM,
    CAST( gal.cfht_megacam_r_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_R_MEGACAM,
    CAST( gal.jst_jpcam_g_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_G_JPCAM,
    CAST( gal.pan_starrs_i_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_I_PANSTARRS,
    CAST( gal.pan_starrs_z_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_Z_PANSTARRS,
    CAST( gal.subaru_hsc_z_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_Z_HSC,
    CAST( gal.gaia_g_el_model3_odonnell_ext*1.e23  AS float)  AS TU_FNU_G_GAIA,
    CAST( gal.gaia_bp_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_BP_GAIA,
    CAST( gal.gaia_rp_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_RP_GAIA,
    CAST( gal.lsst_u_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_U_LSST,
    CAST( gal.lsst_g_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_G_LSST,
    CAST( gal.lsst_r_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_R_LSST,
    CAST( gal.lsst_i_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_I_LSST,
    CAST( gal.lsst_z_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_Z_LSST,
    CAST( gal.lsst_y_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_Y_LSST,
    CAST( gal.kids_u_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_U_KIDS,
    CAST( gal.kids_g_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_G_KIDS,
    CAST( gal.kids_r_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_R_KIDS,
    CAST( gal.kids_i_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_I_KIDS,
    CAST( gal.2mass_j_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_J_2MASS,
    CAST( gal.2mass_h_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_H_2MASS,
    CAST( gal.2mass_ks_el_model3_odonnell_ext*1.e23  AS float) AS TU_FNU_KS_2MASS
FROM cosmohub.flagship_sc456_1_7_17_s AS gal
JOIN jcarrete.flagship_1_7_17_correct_pq as shape
ON gal.halo_id = shape.halo_id
AND gal.galaxy_id = shape.galaxy_id
WHERE (gal.logf_halpha_model3_ext > -16 OR gal.euclid_nisp_h < 26)
AND gal.disk_axis_ratio > 0.10865
AND SHIFTRIGHT(gal.hpix_29_nest, (29-5)*2) = 9129

The same query DOES NOT work in hive and it is still a mistery.

Pau launched the whole list of healpix pixels of the 1.7.17 catalog version and created the FITS files. This process, instead of generating one single parquet file per healpix pixel (i.e. per partition), as it was done when Hive worked, it creates several parquet files. I had to concat all of them in one directory (per healpix pixel) and then I run the notebook:

    /nfs/pic.es/user/j/jcarrete/python_notebooks/split\ in\ healpix\ pixels.ipynb

to add the header.

FITS files are written in the /pnfs. In particular:

   /pnfs/pic.es/data/astro/euclid/disk/shared/SPV/GALAXIES/v29/