Mock pipeline v2.0

Dependencies

• Python 3.8 for PEP 574 (Pickle protocol 5 with out-of-band data)
• Spark 3.0.0 for SPARK-28198 (Add mapPartitionsInPandas to allow an iterator of DataFrames)

Installation

Development Toolset v8

sudo yum install centos-release-scl
sudo yum install devtoolset-6
scl enable devtoolset-8 bash

Python 3.8.0

Get system compilation flags using:

>>> import sysconfig
>>> sysconfig.get_config_var('CONFIG_ARGS')

Configure:

export CFLAGS="-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv"
export LDFLAGS="-Wl,-z,relro  -g"
export PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig
./configure --prefix=/software/astro/centos7/python/3.8.0 --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu \
--disable-dependency-tracking --enable-ipv6 --enable-shared --with-computed-gotos=yes --with-dbmliborder=gdbm:ndbm:bdb \
--with-system-expat --with-system-ffi --enable-loadable-sqlite-extensions --with-dtrace --with-valgrind --without-ensurepip --enable-optimizations

Compile & install:

make -j 24
make install