Jcarrete: Created page with "The catalog comes from: https://alexie.sites.ucsc.edu/dataproducts/ In particular I ingest the file: acs_clean.fits I use the following notebook to convert from FITS in..."

2019-12-04T15:34:04Z

Created page with "The catalog comes from: https://alexie.sites.ucsc.edu/dataproducts/ In particular I ingest the file: acs_clean.fits I use the following notebook to convert from FITS in..."

New page

The catalog comes from:

https://alexie.sites.ucsc.edu/dataproducts/

In particular I ingest the file: acs_clean.fits

I use the following notebook to convert from FITS into parquet:

/python_notebooks/Ingesting_Lethaud_COSMOS_ACS_WL_catalog_and_ACS_Kai_catalog.ipynb

There is one of the fields which is a array: FLUX_RADIUS
I separate it into 4 elements: FLUX_RADIUS_[0-3]

Create external table from beeline-hive2:

<pre>
CREATE EXTERNAL TABLE jcarrete.cosmos_acs_clean_pq (
MAG_ISO float,
MAGERR_ISO float,
MAG_ISOCOR float,
MAGERR_ISOCOR float,
MAG_PETRO float,
MAGERR_PETRO float,
PETRO_RADIUS float,
MAG_APER float,
MAGERR_APER float,
MAG_AUTO float,
MAGERR_AUTO float,
MAG_BEST float,
MAGERR_BEST float,
FLUX_AUTO float,
FLUXERR_AUTO float,
KRON_RADIUS float,
BACKGROUND float,
THRESHOLD float,
FLUX_MAX float,
ISOAREA_IMAGE int,
X_IMAGE float,
Y_IMAGE float,
XMIN_IMAGE int,
YMIN_IMAGE int,
XMAX_IMAGE int,
YMAX_IMAGE int,
XPEAK_IMAGE int,
YPEAK_IMAGE int,
ALPHAPEAK_J2000 double,
DELTAPEAK_J2000 double,
A_IMAGE float,
B_IMAGE float,
ALPHA_J2000 double,
DELTA_J2000 double,
THETA_IMAGE float,
MU_THRESHOLD float,
MU_MAX float,
ISOAREA_WORLD float,
X_WORLD double,
Y_WORLD double,
A_WORLD float,
B_WORLD float,
THETA_WORLD float,
FLAGS int,
FWHM_IMAGE float,
FWHM_WORLD float,
CXX_IMAGE float,
CYY_IMAGE float,
CXY_IMAGE float,
ELONGATION float,
CLASS_STAR float,
FIELD binary,
SE int,
IDENT int,
MU_CLASS int,
OVERLAP int,
NEARSTAR int,
MASK int,
MASKED int,
GOOD int,
CLEAN int,
UNIQUE int,
FLUX_RADIUS_0 float,
FLUX_RADIUS_1 float,
FLUX_RADIUS_2 float,
FLUX_RADIUS_3 float,
NUMBER bigint
)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
'hdfs://namenode01/user/jcarrete/data/COSMOS_ACS_catalog'
;
</pre>

Internal table:

<pre>
CREATE TABLE cosmohub.cosmos_acs_clean (
IDENT int COMMENT 'Unique number that identifies object',
NUMBER bigint COMMENT 'Running object number from SExtractor, do not use this number as a unique identification number',
MAG_ISO float COMMENT 'Isophotal magnitude (mag)',
MAGERR_ISO float COMMENT 'RMS error for isophotal magnitude (mag)',
MAG_ISOCOR float COMMENT 'Corrected isophotal magnitude (mag)',
MAGERR_ISOCOR float COMMENT 'RMS error for corrected isophotal magnitude (mag)',
MAG_PETRO float COMMENT 'Petrosian magnitude. Kron_fact = 2.5 (mag)',
MAGERR_PETRO float COMMENT 'RMS error for petrosian magnitude (mag)',
PETRO_RADIUS float COMMENT 'Petrosian radius. Pixel size is 0.03 arcseconds (pixel)',
MAG_APER float COMMENT 'Fixed aperture magnitude. Aperture = 0.6 arcseconds (mag)',
MAGERR_APER float COMMENT 'RMS error for fixed aperture mag. Aperture = 0.6 arcseconds (mag)',
MAG_AUTO float COMMENT 'Kron-like elliptical aperture magnitude (mag)',
MAGERR_AUTO float COMMENT 'RMS error for AUTO magnitude (mag)',
MAG_BEST float COMMENT 'Best of MAG_AUTO and MAG_ISOCOR (mag)',
MAGERR_BEST float COMMENT 'RMS error for MAG_BEST (mag)',
FLUX_AUTO float COMMENT 'Flux within a Kron-like elliptical aperture (count)',
FLUXERR_AUTO float COMMENT 'RMS error for AUTO flux (count)',
KRON_RADIUS float COMMENT 'Kron radius for AUTO magnitude. Pixel size is 0.03 arcseconds (pixel)',
BACKGROUND float COMMENT 'Background at centroid position (counts)',
THRESHOLD float COMMENT 'Detection threshold above background (counts)',
FLUX_MAX float COMMENT 'Peak flux above backgound (counts)',
FLUX_RADIUS_0 float COMMENT 'Half-light radius (PHOT_FLUXFRAC = 0.5) (pixel)',
FLUX_RADIUS_1 float COMMENT 'Half-light radius (PHOT_FLUXFRAC = 0.5) (pixel)',
FLUX_RADIUS_2 float COMMENT 'Half-light radius (PHOT_FLUXFRAC = 0.5) (pixel)',
FLUX_RADIUS_3 float COMMENT 'Half-light radius (PHOT_FLUXFRAC = 0.5) (pixel)',
ISOAREA_IMAGE int COMMENT 'Isophotal area (pixel^2)',
X_IMAGE float COMMENT 'Object position along x. Pixel size is 0.03 arcseconds (pixel)',
Y_IMAGE float COMMENT 'Object position along y. Pixel size is 0.03 arcseconds (pixel)',
XMIN_IMAGE int COMMENT 'Minimum x-coordinate among detected pixels. Pixel size is 0.03 arcseconds (pixel)',
YMIN_IMAGE int COMMENT 'Minimum y-coordinate among detected pixels. Pixel size is 0.03 arcseconds (pixel)',
XMAX_IMAGE int COMMENT 'Maximum x-coordinate among detected pixels. Pixel size is 0.03 arcseconds (pixel)',
YMAX_IMAGE int COMMENT 'Maximum y-coordinate among detected pixels. Pixel size is 0.03 arcseconds (pixel)',
XPEAK_IMAGE int COMMENT 'x-coordinate of the brightest pixel (pixel)',
YPEAK_IMAGE int COMMENT 'y-coordinate of the brightest pixel (pixel)',
ALPHAPEAK_J2000 double COMMENT 'Right ascension of the brightest pixel (J2000) (deg)',
DELTAPEAK_J2000 double COMMENT 'Declination of the brightest pixel (J2000) (deg)',
A_IMAGE float COMMENT 'Profile RMS along major axis. Pixel size is 0.03 arcseconds (pixel)',
B_IMAGE float COMMENT 'Profile RMS along minor axis. Pixel size is 0.03 arcseconds (pixel)',
ALPHA_J2000 double COMMENT 'Right ascension of the barycenter (J2000) (deg)',
DELTA_J2000 double COMMENT 'Declination of the barycenter (J2000) (deg)',
THETA_IMAGE float COMMENT 'Position angle (CCW/x). Defined counterclockwise (deg)',
MU_THRESHOLD float COMMENT 'Detection threshold above background (max*arcsec^-2)',
MU_MAX float COMMENT 'Peak surface brightness abouve background (max*arcsec^-2)',
ISOAREA_WORLD float COMMENT 'Isophotal area above analysis threshold (deg^2)',
X_WORLD double COMMENT 'Object position along world x axis',
Y_WORLD double COMMENT 'Object position along world y axis',
A_WORLD float COMMENT 'Profile RMS along major axis (world units)',
B_WORLD float COMMENT 'Profile RMS along minor axis (world units)',
THETA_WORLD float COMMENT 'Position angle (CCW/world-x)',
FLAGS int COMMENT 'SExtractor quality flag',
FWHM_IMAGE float COMMENT 'FWHM assuming a gaussian core. Pixel size is 0.03 arcseconds (pixel)',
FWHM_WORLD float COMMENT 'FWHM assuming a gaussian core (deg)',
CXX_IMAGE float COMMENT 'Cxx object ellipse parameter (pixel^-2)',
CYY_IMAGE float COMMENT 'Cyy object ellipse parameter (pixel^-2)',
CXY_IMAGE float COMMENT 'Cxy object ellipse parameter (pixel^-2)',
ELONGATION float COMMENT 'A_IMAGE/B_IMAGE',
CLASS_STAR float COMMENT 'SExtractor star/galaxy classifier. Prefer the MU_CLASS classifier',
FIELD string COMMENT 'Name of ACS pointing in which object is detected',
SE int COMMENT 'Detection from "hot" pass or "cold" pass (se = 0 cold (bright), se = 1 hot (faint))',
MU_CLASS int COMMENT 'classification flag: 1: Galaxies, 2: Stars, 3: Objects with mu_max higher than stars (fake objects and residual cosmic rays)',
OVERLAP int COMMENT 'overlap = 1 no overlapm if an overlap is detected: overlap = (-1)*<ident> of corresponding overlap',
NEARSTAR int COMMENT '0: object is in a star mask, 1: object is not in a star mask',
MASK int COMMENT '0: , 1: star used to make automatic masks',
MASKED int COMMENT '-1: object in manual mask (astrophysical obj), 0: object in manual mask (artefact or image defect)',
GOOD int COMMENT 'flag for lensing cuts',
CLEAN int COMMENT '0: do not use this object, 1: use this object',
UNIQUE int COMMENT 'flags faint objects ( <se> = 1 ) that are already identified in the "cold" pass, 0: object that has a "cold counterpart"'
)
CLUSTERED BY (
IDENT)
INTO 8 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
;
</pre>

<pre>
INSERT OVERWRITE TABLE cosmohub.cosmos_acs_clean
SELECT IDENT, NUMBER, MAG_ISO, MAGERR_ISO, MAG_ISOCOR, MAGERR_ISOCOR, MAG_PETRO, MAGERR_PETRO, PETRO_RADIUS, MAG_APER, MAGERR_APER, MAG_AUTO, MAGERR_AUTO, MAG_BEST, MAGERR_BEST, FLUX_AUTO, FLUXERR_AUTO, KRON_RADIUS, BACKGROUND, THRESHOLD, FLUX_MAX, FLUX_RADIUS_0, FLUX_RADIUS_1, FLUX_RADIUS_2, FLUX_RADIUS_3, ISOAREA_IMAGE, X_IMAGE, Y_IMAGE, XMIN_IMAGE, YMIN_IMAGE, XMAX_IMAGE, YMAX_IMAGE, XPEAK_IMAGE, YPEAK_IMAGE, ALPHAPEAK_J2000, DELTAPEAK_J2000, A_IMAGE, B_IMAGE, ALPHA_J2000, DELTA_J2000, THETA_IMAGE, MU_THRESHOLD, MU_MAX, ISOAREA_WORLD, X_WORLD, Y_WORLD, A_WORLD, B_WORLD, THETA_WORLD, FLAGS, FWHM_IMAGE, FWHM_WORLD, CXX_IMAGE, CYY_IMAGE, CXY_IMAGE, ELONGATION, CLASS_STAR, FIELD, SE, MU_CLASS, OVERLAP, NEARSTAR, MASK, MASKED, GOOD, CLEAN, UNIQUE
FROM jcarrete.cosmos_acs_clean_pq;
</pre>

For statistics I have to run them in HIVE1:

beeline --color=true --showHeader=true --verbose=true --silent=false -u "jdbc:hive2://ambarisrv02.pic.es:2181,ambarisrv03.pic.es:2181,ambarisrv01.pic.es:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2"

ANALYZE TABLE cosmohub.cosmos_acs_clean COMPUTE STATISTICS FOR COLUMNS

The COSMOS ACS Catalog - Revision history

Jcarrete: Created page with "The catalog comes from: https://alexie.sites.ucsc.edu/dataproducts/ In particular I ingest the file: acs_clean.fits I use the following notebook to convert from FITS in..."