Mock pipeline v2.0 - Revision history

Tallada at 10:17, 3 January 2020

2020-01-03T10:17:36Z

Tallada at 09:17, 29 November 2019

2019-11-29T09:17:44Z

Tallada at 09:17, 29 November 2019

2019-11-29T09:17:30Z

Tallada at 13:32, 28 November 2019

2019-11-28T13:32:18Z

Tallada: /* Installation */

2019-11-28T13:23:55Z

Installation

Tallada at 13:23, 28 November 2019

2019-11-28T13:23:21Z

Tallada: Created page with "== Dependencies == * Python 3.8 for PEP 574 (Pickle protocol 5 with out-of-band data) * Spark 3.0.0 for SPARK-28198 (Add mapPartitionsInPandas to allow an iterator of DataFra..."

2019-11-28T12:54:43Z

Created page with "== Dependencies == * Python 3.8 for PEP 574 (Pickle protocol 5 with out-of-band data) * Spark 3.0.0 for SPARK-28198 (Add mapPartitionsInPandas to allow an iterator of DataFra..."

New page

== Dependencies ==

* Python 3.8 for PEP 574 (Pickle protocol 5 with out-of-band data)
* Spark 3.0.0 for SPARK-28198 (Add mapPartitionsInPandas to allow an iterator of DataFrames)

== Installation ==

=== Development Toolset v8 ===

sudo yum install centos-release-scl
sudo yum install devtoolset-6
scl enable devtoolset-8 bash

=== Python 3.8.0 ===

Get system compilation flags using:

>>> import sysconfig
>>> sysconfig.get_config_var('CONFIG_ARGS')

Configure:

export CFLAGS="-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv"
export LDFLAGS="-Wl,-z,relro -g"
export PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig
./configure --prefix=/software/astro/centos7/python/3.8.0 --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu \
--disable-dependency-tracking --enable-ipv6 --enable-shared --with-computed-gotos=yes --with-dbmliborder=gdbm:ndbm:bdb \
--with-system-expat --with-system-ffi --enable-loadable-sqlite-extensions --with-dtrace --with-valgrind --without-ensurepip --enable-optimizations

Compile & install:

make -j 24
make install

← Older revision		Revision as of 10:17, 3 January 2020
Line 76:		Line 76:

	jupyter-lab --no-browser --ip=$(hostname)		jupyter-lab --no-browser --ip=$(hostname)
		+
		+	== Input preparation ==
		+
		+	CREATE TABLE tallada.`flagship_v1_1_halos`(
		+	`bx` tinyint,
		+	`by` tinyint,
		+	`bz` tinyint,
		+	`id` bigint,
		+	`pid` bigint,
		+	`num_p` int,
		+	`x` float,
		+	`y` float,
		+	`z` float,
		+	`vx` float,
		+	`vy` float,
		+	`vz` float,
		+	hpix_2_nest smallint,
		+	hpix_13_nest bigint
		+	)
		+	PARTITIONED BY (
		+	z_step smallint
		+	)
		+	CLUSTERED BY (
		+	hpix_2_nest
		+	)
		+	SORTED BY (
		+	hpix_13_nest
		+	)
		+	INTO 192 BUCKETS
		+	STORED AS PARQUET
		+	;
		+
		+	INSERT OVERWRITE TABLE tallada.`flagship_v1_1_halos` PARTITION(z_step)
		+	SELECT
		+	bx, `by`, bz,
		+	id, pid,
		+	num_p, x, y, z, vx, vy, vz,
		+	default.vec2pix(2, x, y, z), default.vec2pix(13, x, y, z),
		+	step
		+	FROM cosmohub.flagship_rockstar_octant1_c AS m
		+	JOIN flagship_steps AS s
		+	ON SQRT(m.xm.x + m.ym.y + m.z*m.z) BETWEEN s.r_min AND s.r_max
		+	WHERE x BETWEEN 0 AND 200
		+	AND y BETWEEN 0 AND 200
		+	AND z>=0
		+	AND pid=-1
		+	;

@@ Line 71: / Line 71: @@
   source ~/envs/mocks3/activate
-  pip install jupyterlab jupytext jupyterlab-spark-ui-tab
+  pip install jupyterlab jupytext
 Open browser to trigger rebuild:
   jupyter-lab --no-browser --ip=$(hostname)

@@ Line 72: / Line 72: @@
   pip install jupyterlab jupytext jupyterlab-spark-ui-tab
 Open browser to trigger rebuild:
   jupyter-lab --no-browser --ip=$(hostname)

← Older revision		Revision as of 13:32, 28 November 2019
Line 64:		Line 64:
	python setup.py sdist		python setup.py sdist
	pip install dist/pyspark-3.0.0.dev0.tar.gz		pip install dist/pyspark-3.0.0.dev0.tar.gz
		+
		+	=== Environment ===
		+
		+	Install JupyterLab & extensions
		+
		+	source ~/envs/mocks3/activate
		+
		+	pip install jupyterlab jupytext jupyterlab-spark-ui-tab
		+	jupyter labextension install @liorbaber/jupyterlab-spark-ui-tab
		+	jupyter serverextension enable --py spark-ui-tab
		+
		+	Open browser to trigger rebuild:
		+
		+	jupyter-lab --no-browser --ip=$(hostname)

@@ Line 39: / Line 39: @@
   /software/astro/sl6/patchelf/0.8/bin/patchelf --set-rpath /software/astro/centos7/python/3.8.0/lib/ /software/astro/centos7/python/3.8.0/bin/python3.8
-=== Pip & virtualenv ===
+Pip & virtualenv:
   cd /tmp

@@ Line 7: / Line 7: @@
 === Development Toolset v8 ===
   sudo yum install centos-release-scl
@@ Line 32: / Line 34: @@
   make -j 24
   make install