Building Hadoop 2.7.2, Spark 2.0, Hive 2.0.1 using Apache BigTop

Pre-requisites

  • For CentOS 7:
    • sudo yum install -y git wget tar maven ant gcc gcc-c++ make zip unzip rpm-build autoconf automake cppunit-devel hostname svn libtool chrpath fuse-devel fuse cmake lzo-devel java-1.8.0-openjdk lcms2-devel asciidoc xmlto python-devel python-setuptools libxml2-devel libxslt-devel libyaml-devel cyrus-sasl-devel sqlite-devel openldap-devel mysql-devel ivy openssl-devel zlib-devel snappy-devel jansson-devel

  • For Debian Jessie:
    • sudo apt-get install -y git python gcc g++ make ca-certificates-java ant curl dpkg-dev debhelper devscripts autoconf automake libtool libcppunit-dev chrpath liblzo2-dev libzip-dev sharutils libfuse-dev libssl-dev cmake pkg-config asciidoc xmlto python2.7-dev libxml2-dev libxslt1-dev libsqlite3-dev libldap2-dev libsasl2-dev libmysqlclient-dev python-setuptools libkrb5-dev rsync build-essential zlib1g-dev libsnappy-dev libjansson-dev fuse

  • A recent version of node.js for AArch64 - 4.2.1.
  • Protobuf 2.5.0
  • A recent version of Maven
  • Scala 2.11.1.
  • OpenJDK version 8

Hooking up the Linaro OpenJDK

We need to do symlink the CentOS cacerts in order for SSL downloads to work. Also JAVA_HOME needs to be set to the JDK base dir. Please not that only version 7 of the JDK will work.

  • wget http://openjdk.linaro.org/releases/jdk8u-server-release-1609.tar.xz

  • tar xvf jdk8u-server-release-1609.tar.xz

  • cd jdk8u-server-release-1609

  • export JAVA_HOME=$PWD

  • cd jre/lib/security/

  • rm cacerts

  • (for CentOS 7) ln --symbolic /etc/pki/java/cacerts .

  • (for Debian Jessie) ln --symbolic /etc/ssl/certs/java/cacerts .

Add the jvm to the path for Debian Jessie:

  • cd jdk8u-server-release-1609/bin

  • export PATH=$PWD:$PATH

Hooking up Scala

  • wget http://downloads.typesafe.com/scala/2.11.1/scala-2.11.1.tgz

  • tar xvf scala-2.11.1.tgz

  • cd scala-2.11.1

  • export SCALA_HOME=$PWD

Building and hooking up nodejs

  • wget https://nodejs.org/dist/v4.2.1/node-v4.2.1.tar.gz

  • tar xvf node-v4.2.1.tar.gz

  • cd node-v4.2.1

  • ./configure --prefix=/place/to/install/node

  • make -j<NUMCORES>

  • make install

  • cd /place/to/install/node/bin

  • export PATH=$PWD:$PATH

Building and hooking up Protobuf

download the patch required from https://projects.linaro.org/secure/attachment/12000/0001-Added-atomic-ops-to-make-protocbuf-2.5.0-work-with-A.patch

  • git clone https://github.com/google/protobuf.git

  • cd protobuf

  • git checkout -b 2.5.0 v2.5.0

  • git apply 0001-Added-atomic-ops-to-make-protocbuf-2.5.0-work-with-A.patch

  • ./autogen.sh

  • ./configure --prefix=/place/to/install/protobuf

  • make -j8

  • make check

  • make install

  • cd /place/to/install/protobuf/bin

  • export PATH=$PWD:$PATH

  • cd /place/to/install/protobuf/lib/pkgconfig

  • export PKG_CONFIG_PATH=$PWD

type the following to check installation: (it should output 2.5.0)

  • protoc --version

Hooking up upstream Maven 3.3.9 (for Debian Jessie only)

  • wget http://mirror.ox.ac.uk/sites/rsync.apache.org/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz

  • tar xvf apache-maven-3.3.9-bin.tar.gz

  • cd apache-maven-3.3.9/bin

  • export PATH=$PWD:$PATH

Build Procedure

  • Set JAVA_HOME to point to the pertinent JDK.

  • Set SCALA_HOME to point to the built version of Scala (I set it to: scala/build/pack).

  • Add the nodejs installation to PATH.

  • Add the protobuf installation to PATH and PKG_CONFIG_PATH.

  • I set MAVEN_OPTS to -Xmx8G -XX:MaxPermSize=1G

  • I set ANT_OPTS to -Xmx8G -XX:MaxPermSize=1G

  • For Debian Jessie I added upstream Maven to the PATH.

  • git clone https://git.linaro.org/people/ganesh.raju/bigtop-odpi.git

  • cd bigtop-odpi

Run the below commands to create packages:

  • (for CentOS 7) ./gradlew clean hadoop-rpm spark-rpm hive-rpm -Pdist,native-win -DskipTests -Dtar -Dmaven.javadoc.skip=true -PHadoop-2.7 -Pyarn -Phadoop.version=2.7.0 -Dscala-2.11 --debug

  • (for Debian Jessie) ./gradlew clean hadoop-deb spark-deb hive-deb -Pdist,native-win -DskipTests -Dtar -Dmaven.javadoc.skip=true -PHadoop-2.7 -Pyarn -Phadoop.version=2.7.0 -Dscala-2.11 --debug

LEG/Engineering/BigData/ODPi_Setup_Config_Run/ODPi_1_1_BigTop_Building (last modified 2017-01-12 15:26:30)