Difference between revisions of "Production table"
Jump to navigation
Jump to search
(Created page with "To create the file for updating the productions: psql -U postgres -W -h db.pau.pic.es dm -c "COPY (SELECT * FROM production) TO stdout DELIMITER ',' CSV" | pv > paudm_pr...") |
|||
| Line 9: | Line 9: | ||
Copy from local: | Copy from local: | ||
| − | hdfs dfs -copyFromLocal /cephfs/pic.es/astro/scratch/jcarrete/sandbox/ | + | hdfs dfs -copyFromLocal /cephfs/pic.es/astro/scratch/jcarrete/sandbox/paudm_production_datetime.csv /user/jcarrete/data/paudm/production/paudm_production_datetime.csv |
To update the table: | To update the table: | ||
INSERT OVERWRITE TABLE cosmohub.paudm_production SELECT id, input_production, pipeline, release, software_version, job_id, comments, created FROM jcarrete.paudm_production_csv; | INSERT OVERWRITE TABLE cosmohub.paudm_production SELECT id, input_production, pipeline, release, software_version, job_id, comments, created FROM jcarrete.paudm_production_csv; | ||
| + | |||
| + | There were problems with the "commment" fields. | ||
| + | This was reported in 14/09/2020. Some comments (8 entries) contained "line break" and then the number of entries in the csv file were wrong. We asked for modifying them directly in the paudb given that there are only 8 entries. | ||
Revision as of 15:27, 14 September 2020
To create the file for updating the productions:
psql -U postgres -W -h db.pau.pic.es dm -c "COPY (SELECT * FROM production) TO stdout DELIMITER ',' CSV" | pv > paudm_production_datetime.csv
To remove the old version:
hdfs dfs -rm /user/jcarrete/data/paudm/production/*
Copy from local:
hdfs dfs -copyFromLocal /cephfs/pic.es/astro/scratch/jcarrete/sandbox/paudm_production_datetime.csv /user/jcarrete/data/paudm/production/paudm_production_datetime.csv
To update the table:
INSERT OVERWRITE TABLE cosmohub.paudm_production SELECT id, input_production, pipeline, release, software_version, job_id, comments, created FROM jcarrete.paudm_production_csv;
There were problems with the "commment" fields. This was reported in 14/09/2020. Some comments (8 entries) contained "line break" and then the number of entries in the csv file were wrong. We asked for modifying them directly in the paudb given that there are only 8 entries.