Build BigDL-core

BigDL-core provides a lot of native methods including math processing routines, quantized operation and so on. It will increase the performance of neural networks. The math processing routines will call Intel MKL, including gemm, add operations and so on.

BigDL-core is a JNI project, mkl2017-xeon-blas needs MKL libraries with icc and bigquant needs g++-7. We use maven + make to control the build process where maven for java and make for c/c++ code. For building BigDL-core, there should have

Download BigDL-core Source

BigDL-core source code is available at GitHub

$ git clone

By default, git clone will download the development version of BigDL-core, if you want a release version, you can use command git checkout to change the version.

Environment on CentOS (after 6.8 final)

You should build the environment with CentOS 6.8 final. BigQuant depends the GCC 7.2, and it's hard to find the a GCC 7.2 repo or rpm package that supports this version OS. So you should build it from source. Another thing you should pay attention is, you should build a new version binutils because the old version doesn't support to assemble the code generated from GCC 7.2.

git clone
git checkout gcc-7_2_0-release
./configure --prefix=/opt/gcc-7.2.0 --enable-languages=c,c++ --disable-multilib --disable-nls
make -j4 && make install
ln -s /opt/gcc-7.2.0 /opt/gcc
tar zxvf binutils-2.29.tar.gz -C /tmp/ && cd /tmp/binutils-2.29
configure --prefix=/opt/binutils-2.29
make && make install
ln -s /opt/binutils-2.29 /opt/binutils/
./configure --prefix=/opt/git-2.9.5
make -j4 && make install
ln -s /opt/git-2.9.5 /opt/git
export PATH=${GCC_7_HOME}/bin/:${PATH}
export C_INCLUDE_PATH=/opt/gcc/include/:${C_INCLUDE_PATH}
export CPLUS_INCLUDE_PATH=/opt/gcc/include/:${CPLUS_INCLUDE_PATH}

export PATH=${GIT_HOME}/bin:${PATH}

export PATH=${BINUTILS_HOME}/bin:${PATH}

Environment on Ubuntu/Debian

sudo add-apt-repository ppa:jonathonf/gcc-7.1
sudo apt-get update
sudo apt-get install gcc-7 g++-7
sudo apt-get install build-essential

Environment on Windows 10

For this platform, you should install Visual Studio 2015. We have not built successfully with much newer VS. When you do the installation, you need only forllow the default steps and click next.

Please pay attention, the MinGW must follows the link below. Otherwise it maybe can't work. By default, the make names with ming32-make.exe, and g++-7 names with g++. You should change the name at last as follows.

  1. Install Visual Studio 2015
  2. Install Intel Parallel Studio XE 2018
  3. Install MinGW
  4. Copy ming32-make.exe to make.exe
  5. Copy g++ to g++-7

At the end, open a cmd terminal and input g++-7 -v , should output like below,

Using built-in specs.
Target: x86_64-w64-mingw32
Configured with: ../../../src/gcc-7.1.0/configure --host=x86_64-w64-mingw32 --build=x86_64-w64-mingw32 --target=x86_64-w64-ingw32 --prefix=/mingw64 --with-sysroot=/c/mingw710/x86_64-710-win32-sjlj-rt_v5-rev2/mingw64 --enable-shared --enable-static --enable-targets=all --enable-multilib --enable-languages=c,c++,fortran,lto --enable-libstdcxx-time=yes --enable-threads=win32 --enable-libgomp --enable-libatomic --enable-lto --enable-graphite --enable-checking=release --enable-fully-dynamic-string --enable-version-specific-runtime-libs --enable-libstdcxx-filesystem-ts=yes --enable-sjlj-exceptions --disable-libstdcxx-pch --disable-libstdcxx-debug --enable-bootstrap --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --with-gnu-as --with-gnu-ld --with-arch-32=i686 --with-arch-64=nocona --with-tune-32=generic --with-tune-64=core2 --with-libiconv --with-system-zlib --with-gmp=/c/mingw710/prerequisites/x86_64-w64-mingw32-static --with-mpfr=/c/mingw710/prerequisites/x86_64-w64-mingw32-static --with-mpc=/c/mingw710/prerequisites/x86_64-w64-mingw32-static --with-isl=/c/mingw710/prerequisites/x86_64-w64-mingw32-static --with-pkgversion='x86_64-win32-sjlj-rev2, Built by MinGW-W64 project' --with-bugurl= CFLAGS='-O2 -pipe -fno-ident -I/c/mingw710/x86_64-710-win32-sjlj-rt_v5-rev2/mingw64/opt/include -I/c/mingw710/prerequisites/x86_64-zlib-static/include -I/c/mingw710/prerequisites/x86_64-w64-mingw32-static/include' CXXFLAGS='-O2 -pipe -fno-ident -I/c/mingw710/x86_64-710-win32-sjlj-rt_v5-rev2/mingw64/opt/include -I/c/mingw710/prerequisites/x86_64-zlib-static/include -I/c/mingw710/prerequisites/x86_64-w64-mingw32-static/include' CPPFLAGS=' -I/c/mingw710/x86_64-710-win32-sjlj-rt_v5-rev2/mingw64/opt/include -I/c/mingw710/prerequisites/x86_64-zlib-static/include -I/c/mingw710/prerequisites/x86_64-w64-mingw32-static/include' LDFLAGS='-pipe -fno-ident -L/c/mingw710/x86_64-710-win32-sjlj-rt_v5-rev2/mingw64/opt/lib -L/c/mingw710/prerequisites/x86_64-zlib-static/lib -L/c/mingw710/prerequisites/x86_64-w64-mingw32-static/lib '
Thread model: win32
gcc version 7.1.0 (x86_64-win32-sjlj-rev2, Built by MinGW-W64 project)

Environment on macOS (after Sierra / 10.12.1)

On macOS, you can install Intel Parallell Studio XE as normal. And for gcc, you can use brew.

  1. Install Intel Parallel Studio XE.
  2. Install g++-7: brew install gcc@7.

Build and Deploy

We use maven profile to control the build process. For different platforms has different profiles.

Platform Profile Command
Linux linux mvn clean package -P linux
RedHat5 rh5 mvn clean package -P rh5
macOS mac mvn clean package -P mac
Windows win64 mvn clean package -P win64
All Platforms - mvn clean package

Note, the default command whithout any profile will generate a jar package which should contain all libraries of different OS. But it requires you put the relevant libraries into the folder. For example, for biquant, you should put the .so, .dylib, .dll to bigquant/bigquant-java-<os>/target/clases.

There two ways to deploy. You can use mvn deploy -P deploy at the end.

mvn clean deploy -P 'linux' -pl 'bigquant/bigquant-java-x86_64-linux'