https://pwiki.pic.es/index.php?action=history&feed=atom&title=PAUS-photoz_bcnz_v36 2026-04-20T15:11:18Z Revision history for this page on the wiki MediaWiki 1.35.14 https://pwiki.pic.es/index.php?title=PAUS-photoz_bcnz_v36&diff=597&oldid=prev 2020-02-06T13:00:59Z

<p></p> <table class="diff diff-contentalign-left diff-editfont-monospace" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 13:00, 6 February 2020</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l18" >Line 18:</td> <td colspan="2" class="diff-lineno">Line 18:</td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>3. Ingest into paudb:</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>3. Ingest into paudb:</div></td></tr> <tr><td class='diff-marker'>−</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del style="font-weight: bold; text-decoration: none;"></del></div></td><td colspan="2"> </td></tr> <tr><td class='diff-marker'>−</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del style="font-weight: bold; text-decoration: none;">tail -n +2 filename.csv | awk -F, '{print 866,$6,$7,$4,$5,$8,$1,$3,$2}' | tr ' ' ',' &gt; filename_reordered.csv</del></div></td><td colspan="2"> </td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>I assume I have a CSV (comma separated value) file with header, and with the fields are ordered in the same way as in the database model:</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>I assume I have a CSV (comma separated value) file with header, and with the fields are ordered in the same way as in the database model:</div></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr> <tr><td class='diff-marker'>−</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>tail -n +2 filename.csv | pv -l | psql -U postgres -h <del class="diffchange diffchange-inline">db01</del>.pau.pic.es dm - c &quot;COPY <del class="diffchange diffchange-inline">paudm.</del>photoz_bcnz FROM STDIN DELIMITER ',' CSV&quot;<del class="diffchange diffchange-inline">Once the csv is created</del></div></td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins class="diffchange diffchange-inline">    </ins>tail -n +2 filename.csv | pv -l | psql -U postgres -h <ins class="diffchange diffchange-inline">db</ins>.pau.pic.es dm -c &quot;COPY photoz_bcnz FROM STDIN DELIMITER ',' CSV&quot;</div></td></tr> </table>

Jcarrete https://pwiki.pic.es/index.php?title=PAUS-photoz_bcnz_v36&diff=596&oldid=prev 2020-02-06T13:00:08Z

<p>Created page with &quot;&#039;&#039;&#039;This is the way we ingest PAUS photo-z releases in CosmoHub and PAUdb&#039;&#039;&#039; This is the case for file: /nfs/astro/eriksen/kcorr/bcnz_v36.parquet 1. Create a new product...&quot;</p> <p><b>New page</b></p><div>'''This is the way we ingest PAUS photo-z releases in CosmoHub and PAUdb'''<br /> <br /> This is the case for file:<br /> <br /> /nfs/astro/eriksen/kcorr/bcnz_v36.parquet<br /> <br /> 1. Create a new production in the db.pau.pic.es in the production table.<br /> <br /> In this case we create production = 953, with input_production_id = 944. Some more information is needed.<br /> <br /> 2. There is a python notebook:<br /> <br /> /nfs/user/j/jcarrete/python_notebooks/PAUdm photoz parquet files.ipynb<br /> <br /> Where we convert the super large parquet file with many fields into a csv fields with the needed fields:<br /> <br /> /cephfs/pic.es/astro/scratch/jcarrete/sandbox/parquet_bcnz_v36_some_fields.csv<br /> <br /> 3. Ingest into paudb:<br /> <br /> tail -n +2 filename.csv | awk -F, '{print 866,$6,$7,$4,$5,$8,$1,$3,$2}' | tr ' ' ',' &gt; filename_reordered.csv<br /> <br /> I assume I have a CSV (comma separated value) file with header, and with the fields are ordered in the same way as in the database model:<br /> <br /> tail -n +2 filename.csv | pv -l | psql -U postgres -h db01.pau.pic.es dm - c &quot;COPY paudm.photoz_bcnz FROM STDIN DELIMITER ',' CSV&quot;Once the csv is created</div>

Jcarrete