AC UserManual - Revision history

Jcarrete at 17:28, 4 November 2019

2019-11-04T17:28:09Z

Jcarrete: /* Creating a python environment to work on wn-el7.astro.pic.es */

2019-06-17T11:37:55Z

Creating a python environment to work on wn-el7.astro.pic.es

Jcarrete: /* Creating a python environment to work on wn-el7.astro.pic.es */

2019-06-06T14:37:34Z

Creating a python environment to work on wn-el7.astro.pic.es

Jcarrete: /* Creating a python environment to work on wn-el7.astro.pic.es */

2019-06-06T14:36:58Z

Creating a python environment to work on wn-el7.astro.pic.es

Jcarrete: /* Creating a python environment to work on wn-el7.astro.pic.es */

2019-06-06T14:36:22Z

Creating a python environment to work on wn-el7.astro.pic.es

Jcarrete at 14:32, 6 June 2019

2019-06-06T14:32:25Z

Neissner: /* Using the HTCondor computing farm */

2019-04-25T11:06:58Z

Using the HTCondor computing farm

Neissner: /* Using the HTCondor computing farm */

2019-04-25T11:03:09Z

Using the HTCondor computing farm

Neissner: /* Using the HTCondor computing farm */

2019-04-25T11:01:59Z

Using the HTCondor computing farm

Neissner: /* Using the HTCondor computing farm */

2019-04-25T10:48:52Z

Using the HTCondor computing farm

← Older revision		Revision as of 17:28, 4 November 2019
Line 490:		Line 490:

	- paudm_input_production_id : MEMBA production_id		- paudm_input_production_id : MEMBA production_id
		+
		+	== Ingest PAUdm catalogs into CosmoHub ==
		+
		+	=== Forced aperture production (production = 941) ===
		+
		+
		+	* Get catalog from paudb (from UI):
		+
		+	psql -U readonly -W -h db.pau.pic.es dm -c "COPY (SELECT * FROM public.forced_aperture_coadd WHERE production_id = 941) TO stdout DELIMITER ',' CSV HEADER" \| lbzip2 \| pv > forced_aperture_coadd_production_id_941.csv.bz2
		+
		+	* Copy to pnfs
		+
		+	cp forced_aperture_coadd_production_id_941.csv.bz2 /pnfs/pic.es/data/astro/cosmohub/disk/raw/paudm/forced_aperture_coadd/forced_aperture_coadd_production_id_941.csv.bz2
		+
		+	* Beeline:
		+
		+	beeline --color=true --showHeader=true --verbose=true --silent=false -u "jdbc:hive2://ambarisrv02.pic.es:2181,ambarisrv03.pic.es:2181,ambarisrv01.pic.es:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-hive2"
		+
		+	* Create partition (from beeline)
		+
		+	ALTER TABLE jcarrete.paudm_forced_aperture_coadd_csv ADD IF NOT EXISTS PARTITION(production_id=941);
		+
		+	* Copy csv.bz2 to hdfs:
		+
		+	hdfs dfs -put /nfs/astro/jcarrete/sandbox/forced_aperture_coadd_production_id_941.csv.bz2 /user/jcarrete/data/paudm/forced_aperture_coadd/production_id\=941/forced_aperture_coadd_production_id_941.csv.bz2
		+
		+	* Insert overwrite cosmohub table:
		+
		+	INSERT OVERWRITE TABLE cosmohub.paudm_forced_aperture_coadd PARTITION(production_id=941)
		+	SELECT `ref_id`, `band`, `flux`, `flux_error`, `chi2`, `n_coadd`, `run`
		+	FROM jcarrete.paudm_forced_aperture_coadd_csv WHERE production_id=941;
		+
		+	=== Production table ===
		+
		+	psql -U readonly -W -h db.pau.pic.es dm -c "COPY (SELECT * FROM public.production) TO stdout DELIMITER ',' CSV HEADER" \| pv > production_table_04_11_2019.csv
		+	hdfs dfs -rm /user/jcarrete/data/paudm/production/*
		+	hdfs dfs -copyFromLocal /nfs/astro/jcarrete/sandbox/production_table_04_11_2019.csv /user/jcarrete/data/paudm/production/production_table_04_11_2019.csv
		+	INSERT OVERWRITE TABLE cosmohub.paudm_production SELECT id, input_production, pipeline, release, software_version, job_id, comments, created FROM jcarrete.paudm_production_csv WHERE id IS NOT NULL;
		+
		+	=== PAUdm photoz results from M.Eriksen file ===
		+
		+	The file is here:
		+
		+	/nfs/astro/eriksen/kcorr/bcnz_v29.parquet
		+
		+	* I use python pyarrow library to read the parquet file and saved it into csv. These are the lines in python:
		+
		+	import pyarrow.parquet as pq
		+	data = pq.read_table('/nfs/astro/eriksen/kcorr/bcnz_v29.parquet')
		+	df_new = data.to_pandas()
		+	df_new['production_id'] = 952
		+	df_new.reset_index(inplace=True)
		+	column_list = ['production_id', 'ref_id', 'zb', 'odds', 'pz_width', 'zb_mean', 'chi2', 'n_band', 'ebv', 'qz', 'best_run']
		+	df_new[column_list].to_csv('/nfs/astro/jcarrete/sandbox/parquet_bcnz_v29_some_fields.csv', header=True, index=False)
		+
		+	* Add by hand in the paudb, in the production table, the production of the new photoz data production id = 952
		+
		+	* Create partition (from beeline)
		+
		+	ALTER TABLE jcarrete.paudm_photoz_bcnz_updated_v1_1_csv ADD IF NOT EXISTS PARTITION(production_id=952);
		+
		+	* Copy csv to hdfs:
		+
		+	hdfs dfs -put /nfs/astro/jcarrete/sandbox/parquet_bcnz_v29_some_fields.csv /user/jcarrete/data/paudm/photoz_bcnz_updated_v1_1_csv/production_id\=952/parquet_bcnz_v29_some_fields.csv
		+
		+	* Insert overwrite table:
		+
		+	INSERT OVERWRITE TABLE cosmohub.paudm_photoz_bcnz_updated_v1_1 PARTITION(production_id=952)
		+	SELECT ref_id, zb, odds, pz_width, zb_mean, chi2, n_band, ebv, qz, best_run
		+	FROM jcarrete.paudm_photoz_bcnz_updated_v1_1_csv WHERE production_id = 952 AND ref_id IS NOT NULL;

@@ Line 157: / Line 157: @@
   ssh {USER}@data.astro.pic.es
-. Usually environments are all saved in the same directory (e.g. ~/env). In case it is not created:
+. Usually environments are all saved in the same directory (e.g. ~/env).
   mkdir ~/env/
   cd ~/env/
   virtualenv-3.6 {ENV_NAME}
@@ Line 168: / Line 166: @@
 . Activate environment
-  . {ENV_NAME}/bin/activate
+  source ~/env/{ENV_NAME}/bin/activate
-. Upgrade pip command:
+. Upgrade pip command (the environment has to be activated):
   pip install --upgrade pip

@@ Line 187: / Line 187: @@
   . {ENV_NAME}/bin/activate
-. Go to the directory where your notebook is and launch jupyter-notebook:
+. '''Go to the directory where your notebook is''' and launch jupyter-notebook:
   jupyter-notebook --ip=$(hostname) --no-browser

← Older revision		Revision as of 14:36, 6 June 2019
Line 177:		Line 177:

	pip install numpy scipy matplotlib astropy pandas jupyter psycopg2 psycopg2-binary		pip install numpy scipy matplotlib astropy pandas jupyter psycopg2 psycopg2-binary
−

	8. Open another terminal and login to wn-el7.astro.pic.es:		8. Open another terminal and login to wn-el7.astro.pic.es:

@@ Line 155: / Line 155: @@
 . Login to data.astro.pic.es (because virtualenv3.6 is installed there):
-  ssh {USER}@ui.pic.es
+  ssh {USER}@data.astro.pic.es
 . Usually environments are all saved in the same directory (e.g. ~/env). In case it is not created:
@@ Line 162: / Line 162: @@
 . Create a new environment (python_version = 3.6):
   cd ~/env/
   virtualenv-3.6 {ENV_NAME}
@@ Line 206: / Line 207: @@
   http://localhost:{LOCAL_PORT}
 == Working with Python environments (in UI) ==

@@ Line 145: / Line 145: @@
   Total for query: 263 jobs; 0 completed, 0 removed, 255 idle, 8 running, 0 held, 0 suspended
   Total for all users: 263 jobs; 0 completed, 0 removed, 255 idle, 8 running, 0 held, 0 suspended
 == Working with Python environments (in UI) ==

@@ Line 102: / Line 102: @@
   error = error-600-134.err
   log = log-600-134.log
- transfer_executable = false
+  request_memory=6GB
   request_cpus=2
   +experiment="des"
@@ Line 116: / Line 115: @@
   error = error-600-$(Item).err
   log = log-600-$(Item).log
   request_memory=6GB
   request_cpus=2

@@ Line 126: / Line 126: @@
 Finally we can send the job to the HTCondor scheduler:
   [neissner@ui04 ~]$ ssh submit01.pic.es 'condor_submit test.sub
-  Submitting job(s)...............................................................................................................................................
+  Submitting job(s)......................................................................................
- ............................................................................................................................
+ .................................................................................................................
 job(s) submitted to cluster 1486.

@@ Line 65: / Line 65: @@
   import bz2
   from pathlib import Path
-  import resource
+  import pycdlib
-     import pycdlib
+ import io
-     import io
+ import pyarrow
 With this information we create our virtual environment:
@@ Line 74: / Line 74: @@
   (test) [neissner@ui04 ~]$ pip install --upgrade pip
   [...]
-  (test) [neissner@ui04 ~]$ pip install numpy healpy pandas pathlib io pycdlib
+  (test) [neissner@ui04 ~]$ pip install --no-cache-dir numpy healpy pandas pathlib io pycdlib pyarrow
   [...]
   (test) [neissner@ui04 ~]$ deactivate