hadoop-2.6.0-cdh5.7.0-支持各种压缩的源码编译安装

Build instructions for Hadoop

 ----------------------------------------------------------------------------------
 Requirements:
 
 * Unix System
 * JDK 1.7+
 * Maven 3.0 or later
 * Findbugs 1.3.9 (if running findbugs)
 * ProtocolBuffer 2.5.0
 * CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
 * Zlib devel (if compiling native code)
 * openssl devel ( if compiling native hadoop-pipes )
 * Internet connection for first build (to fetch all Maven and Hadoop dependencies)
 
 ----------------------------------------------------------------------------------
 Maven main modules:
 
   hadoop                            (Main Hadoop project)
          - hadoop-project           (Parent POM for all Hadoop Maven modules.             )
                                     (All plugins & dependencies versions are defined here.)
          - hadoop-project-dist      (Parent POM for modules that generate distributions.)
          - hadoop-annotations       (Generates the Hadoop doclet used to generated the Javadocs)
          - hadoop-assemblies        (Maven assemblies used by the different modules)
          - hadoop-common-project    (Hadoop Common)
          - hadoop-hdfs-project      (Hadoop HDFS)
          - hadoop-mapreduce-project (Hadoop MapReduce)
          - hadoop-tools             (Hadoop tools like Streaming, Distcp, etc.)
          - hadoop-dist              (Hadoop distribution assembler)
 
 ----------------------------------------------------------------------------------
 Where to run Maven from?
 
   It can be run from any module. The only catch is that if not run from utrunk
   all modules that are not part of the build run must be installed in the local
   Maven cache or available in a Maven repository.
 
 ----------------------------------------------------------------------------------
 Maven build goals:
 
  * Clean                     : mvn clean
  * Compile                   : mvn compile [-Pnative]
  * Run tests                 : mvn test [-Pnative]
  * Create JAR                : mvn package
  * Run findbugs              : mvn compile findbugs:findbugs
  * Run checkstyle            : mvn compile checkstyle:checkstyle
  * Install JAR in M2 cache   : mvn install
  * Deploy JAR to Maven repo  : mvn deploy
  * Run clover                : mvn test -Pclover [-DcloverLicenseLocation=${user.name}/.clover.license]
  * Run Rat                   : mvn apache-rat:check
  * Build javadocs            : mvn javadoc:javadoc
  * Build distribution        : mvn package [-Pdist][-Pdocs][-Psrc][-Pnative][-Dtar]
  * Change Hadoop version     : mvn versions:set -DnewVersion=NEWVERSION
 
  Build options:
 
   * Use -Pnative to compile/bundle native code
   * Use -Pdocs to generate & bundle the documentation in the distribution (using -Pdist)
   * Use -Psrc to create a project source TAR.GZ
   * Use -Dtar to create a TAR with the distribution (using -Pdist)
 
  Snappy build options:
 
    Snappy is a compression library that can be utilized by the native code.
    It is currently an optional component, meaning that Hadoop can be built with
    or without this dependency.
 
   * Use -Drequire.snappy to fail the build if libsnappy.so is not found.
     If this option is not specified and the snappy library is missing,
     we silently build a version of libhadoop.so that cannot make use of snappy.
     This option is recommended if you plan on making use of snappy and want
     to get more repeatable builds.
 
   * Use -Dsnappy.prefix to specify a nonstandard location for the libsnappy
     header files and library files. You do not need this option if you have
     installed snappy using a package manager.
   * Use -Dsnappy.lib to specify a nonstandard location for the libsnappy library
     files.  Similarly to snappy.prefix, you do not need this option if you have
     installed snappy using a package manager.
   * Use -Dbundle.snappy to copy the contents of the snappy.lib directory into
     the final tar file. This option requires that -Dsnappy.lib is also given,
     and it ignores the -Dsnappy.prefix option.
 
  OpenSSL build options:
 
    OpenSSL includes a crypto library that can be utilized by the native code.
    It is currently an optional component, meaning that Hadoop can be built with
    or without this dependency.
 
   * Use -Drequire.openssl to fail the build if libcrypto.so is not found.
     If this option is not specified and the openssl library is missing,
     we silently build a version of libhadoop.so that cannot make use of
     openssl. This option is recommended if you plan on making use of openssl 
     and want to get more repeatable builds.
   * Use -Dopenssl.prefix to specify a nonstandard location for the libcrypto
     header files and library files. You do not need this option if you have
     installed openssl using a package manager.
   * Use -Dopenssl.lib to specify a nonstandard location for the libcrypto library
     files. Similarly to openssl.prefix, you do not need this option if you have
     installed openssl using a package manager.
   * Use -Dbundle.openssl to copy the contents of the openssl.lib directory into
     the final tar file. This option requires that -Dopenssl.lib is also given,
     and it ignores the -Dopenssl.prefix option.
 
    Tests options:
 
   * Use -DskipTests to skip tests when running the following Maven goals:
     'package',  'install', 'deploy' or 'verify'
   * -Dtest=<TESTCLASSNAME>,<TESTCLASSNAME#METHODNAME>,....
   * -Dtest.exclude=<TESTCLASSNAME>
   * -Dtest.exclude.pattern=**/<TESTCLASSNAME1>.java,**/<TESTCLASSNAME2>.java
 
 ----------------------------------------------------------------------------------
 Building components separately
 
 If you are building a submodule directory, all the hadoop dependencies this
 submodule has will be resolved as all other 3rd party dependencies. This is,
 from the Maven cache or from a Maven repository (if not available in the cache
 or the SNAPSHOT 'timed out').
 An alternative is to run 'mvn install -DskipTests' from Hadoop source top
 level once; and then work from the submodule. Keep in mind that SNAPSHOTs
 time out after a while, using the Maven '-nsu' will stop Maven from trying
 to update SNAPSHOTs from external repos.
 
 ----------------------------------------------------------------------------------
 Protocol Buffer compiler
 
 The version of Protocol Buffer compiler, protoc, must match the version of the
 protobuf JAR.
 
 If you have multiple versions of protoc in your system, you can set in your 
 build shell the HADOOP_PROTOC_CDH5_PATH environment variable to point to the one you 
 want to use for the Hadoop build. If you don't define this environment variable,
 protoc is looked up in the PATH.
 ----------------------------------------------------------------------------------
 Importing projects to eclipse
 
 When you import the project to eclipse, install hadoop-maven-plugins at first.
 
   $ cd hadoop-maven-plugins
   $ mvn install
 
 Then, generate eclipse project files.
 
   $ mvn eclipse:eclipse -DskipTests
 
 At last, import to eclipse by specifying the root directory of the project via
 [File] > [Import] > [Existing Projects into Workspace].
 
 ----------------------------------------------------------------------------------
 Building distributions:
 
 Create binary distribution without native code and without documentation:
 
   $ mvn package -Pdist -DskipTests -Dtar
 
 Create binary distribution with native code and with documentation:
 
   $ mvn package -Pdist,native,docs -DskipTests -Dtar
 
 Create source distribution:
 
   $ mvn package -Psrc -DskipTests
 
 Create source and binary distributions with native code and documentation:
 
   $ mvn package -Pdist,native,docs,src -DskipTests -Dtar
 
 Create a local staging version of the website (in /tmp/hadoop-site)
 
   $ mvn clean site; mvn site:stage -DstagingDirectory=/tmp/hadoop-site
 
 ----------------------------------------------------------------------------------
 Installing Hadoop
 
 Look for these HTML files after you build the document by the above commands.
 
   * Single Node Setup:
     hadoop-project-dist/hadoop-common/SingleCluster.html
 
   * Cluster Setup:
     hadoop-project-dist/hadoop-common/ClusterSetup.html
 
 ----------------------------------------------------------------------------------
 
 Handling out of memory errors in builds
 
 ----------------------------------------------------------------------------------
 
 If the build process fails with an out of memory error, you should be able to fix
 it by increasing the memory used by maven -which can be done via the environment
 variable MAVEN_OPTS.
 
 Here is an example setting to allocate between 256 and 512 MB of heap space to
 Maven
 
 export MAVEN_OPTS="-Xms256m -Xmx512m"
 
 ----------------------------------------------------------------------------------
 
 Building on OS/X
 
 ----------------------------------------------------------------------------------
 
 A one-time manual step is required to enable building Hadoop OS X with Java 7
 every time the JDK is updated.
 see: https://issues.apache.org/jira/browse/HADOOP-9350
 
 $ sudo mkdir `/usr/libexec/java_home`/Classes
 $ sudo ln -s `/usr/libexec/java_home`/lib/tools.jar `/usr/libexec/java_home`/Classes/classes.jar
 
 ----------------------------------------------------------------------------------
 
 Building on Windows
 
 ----------------------------------------------------------------------------------
 Requirements:
 
 * Windows System
 * JDK 1.7+
 * Maven 3.0 or later
 * Findbugs 1.3.9 (if running findbugs)
 * ProtocolBuffer 2.5.0
 * CMake 2.6 or newer
 * Windows SDK or Visual Studio 2010 Professional
 * Unix command-line tools from GnuWin32 or Cygwin: sh, mkdir, rm, cp, tar, gzip
 * zlib headers (if building native code bindings for zlib)
 * Internet connection for first build (to fetch all Maven and Hadoop dependencies)
 
 If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).
 Do not use Visual Studio Express.  It does not support compiling for 64-bit,
 which is problematic if running a 64-bit system.  The Windows SDK is free to
 download here:
 
 http://www.microsoft.com/en-us/download/details.aspx?id=8279
 
 ----------------------------------------------------------------------------------
 Building:
 
 Keep the source code tree in a short path to avoid running into problems related
 to Windows maximum path length limitation.  (For example, C:\hdc).
 
 Run builds from a Windows SDK Command Prompt.  (Start, All Programs,
 Microsoft Windows SDK v7.1, Windows SDK 7.1 Command Prompt.)
 
 JAVA_HOME must be set, and the path must not contain spaces.  If the full path
 would contain spaces, then use the Windows short path instead.
 
 You must set the Platform environment variable to either x64 or Win32 depending
 on whether you're running a 64-bit or 32-bit system.  Note that this is
 case-sensitive.  It must be "Platform", not "PLATFORM" or "platform".
 Environment variables on Windows are usually case-insensitive, but Maven treats
 them as case-sensitive.  Failure to set this environment variable correctly will
 cause msbuild to fail while building the native code in hadoop-common.
 
 set Platform=x64 (when building on a 64-bit system)
 set Platform=Win32 (when building on a 32-bit system)
 
 Several tests require that the user must have the Create Symbolic Links
 privilege.
 
 All Maven goals are the same as described above with the exception that
 native code is built by enabling the 'native-win' Maven profile. -Pnative-win 
 is enabled by default when building on Windows since the native components 
 are required (not optional) on Windows.
 
 If native code bindings for zlib are required, then the zlib headers must be
 deployed on the build machine.  Set the ZLIB_HOME environment variable to the
 directory containing the headers.
 
 set ZLIB_HOME=C:\zlib-1.2.7
 
 At runtime, zlib1.dll must be accessible on the PATH.  Hadoop has been tested
 with zlib 1.2.7, built using Visual Studio 2010 out of contrib\vstudio\vc10 in
 the zlib 1.2.7 source tree.
 
 http://www.zlib.net/
 
 ----------------------------------------------------------------------------------
 Building distributions:
 
  * Build distribution with native code    : mvn package [-Pdist][-Pdocs][-Psrc][-Dtar]

tar -zxvf jdk1.7.0_45.tar.gz -C /usr/java/
vi ~/.bash_profile
export JAVA_HOME=/usr/java/jdk1.7.0_45
export PATH=$JAVA_HOME/bin:$PATH
[hadoop@hadoop001 java]$ source ~/.bash_profile 
[hadoop@hadoop001 home]$ java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)

[root@hadoop001 app]# yum install -y unzip zip
[root@hadoop001 app]# cd /usr
[root@hadoop001 usr]# mkdir maven
[root@hadoop001 app]# unzip -d /usr/maven/ apache-maven-3.3.9-bin.zip 
**maven环境变量**
export MAVEN_HOME=/usr/maven/apache-maven-3.3.9
export PATH=$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
[hadoop@hadoop001 java]$ source ~/.bash_profile 
[hadoop@hadoop001 java]$ mvn -v
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00)
Maven home: /usr/maven/apache-maven-3.3.9
Java version: 1.8.0_45, vendor: Oracle Corporation
Java home: /usr/java/jdk1.8.0_45/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-514.el7.x86_64", arch: "amd64", family: "unix"

修改conf/settings.xml(将mirror换成阿里云的)
# 配置仓库路径
<localRepository>/usr/maven/repo</localRepository> 
      <mirrors>
        <mirror>
          <id>alimaven</id>
          <name>aliyun maven</name>
          <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
          <mirrorOf>central</mirrorOf>        
        </mirror>
      </mirrors>

[root@hadoop001 ~]# yum install -y gcc gcc-c++ make cmake
[root@hadoop001 ~]# yum -y install autoconf automake libtool curl g++
[root@hadoop001 ~]# yum install -y openssl openssl-devel svn ncurses-devel zlib-devel libtool
[root@hadoop001 ~]# yum install -y snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop autoconf automake

[root@hadoop001 app]# tar -zxvf protobuf-2.5.0.tar.gz -C /home/hadoop/lib/

[root@hadoop001 lib]# ll
drwxr-xr-x. 10 109965   5000 4096 Feb 27  2013 protobuf-2.5.0

# 注意：权限，解压后发现权限变化了
[root@hadoop001 lib]# chown root:root -R protobuf-2.5.0/
[root@hadoop001 lib]# ll
drwxr-xr-x. 10 root   root   4096 Feb 27  2013 protobuf-2.5.0

# 指定编译之后的安装路径，目录不需要自己去创建，编译过程中会自动创建
[root@hadoop001 protobuf-2.5.0]#  ./configure --prefix=/usr/protobuf
[root@hadoop001 protobuf-2.5.0]#  make && make install  #安装

# 添加环境变量
[hadoop@hadoop001 java]$vim ~/.bash_profile
export MAVEN_HOME=/usr/maven/apache-maven-3.3.9
export PROTOBUF_HOME =/usr/protobuf
export PATH=$PROTOBUF_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH

# 生效环境变量
[hadoop@hadoop001 java]$source ~/.bash_profile
[hadoop@hadoop001 java]$ protoc  --version 
libprotoc 2.5.0

[root@hadoop001 app]# unzip -d /home/hadoop/lib/ findbugs-1.3.9.zip
[hadoop@hadoop001 java]$ vi ~/.bash_profile
export FINDBUGS_HOME=/home/hadoop/lib/findbugs-1.3.9
export PATH=$FINDBUGS_HOME/bin:$PROTOBUF_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
[hadoop@hadoop001 java]$ source ~/.bash_profile 
[hadoop@hadoop001 java]$ findbugs -version
1.3.9

[root@hadoop001 app]# tar -zxvf hadoop-2.6.0-cdh5.7.0-src.tar.gz -C /home/hadoop/source/
[hadoop@hadoop001 source]$chown -R  hadoop:hadoop hadoop-2.6.0-cdh5.7.0/

[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ mvn -X clean package -Pdist,native -DskipTests -Dtar

[hadoop@hadoop001 hadoop-dist]$ cd target/
[hadoop@hadoop001 target]$ ll
total 565848
drwxrwxr-x. 2 hadoop hadoop        28 Apr  5 01:17 antrun
drwxrwxr-x. 3 hadoop hadoop        22 Apr  5 01:17 classes
-rw-rw-r--. 1 hadoop hadoop      1998 Apr  5 01:17 dist-layout-stitching.sh
-rw-rw-r--. 1 hadoop hadoop       690 Apr  5 01:17 dist-tar-stitching.sh
drwxrwxr-x. 9 hadoop hadoop       149 Apr  5 01:17 hadoop-2.6.0-cdh5.7.0
-rw-rw-r--. 1 hadoop hadoop 192393404 Apr  5 01:17 hadoop-2.6.0-cdh5.7.0.tar.gz
-rw-rw-r--. 1 hadoop hadoop      7314 Apr  5 01:17 hadoop-dist-2.6.0-cdh5.7.0.jar
-rw-rw-r--. 1 hadoop hadoop 386994546 Apr  5 01:18 hadoop-dist-2.6.0-cdh5.7.0-javadoc.jar
-rw-rw-r--. 1 hadoop hadoop      4856 Apr  5 01:17 hadoop-dist-2.6.0-cdh5.7.0-sources.jar
-rw-rw-r--. 1 hadoop hadoop      4856 Apr  5 01:17 hadoop-dist-2.6.0-cdh5.7.0-test-sources.jar
drwxrwxr-x. 2 hadoop hadoop        51 Apr  5 01:17 javadoc-bundle-options
drwxrwxr-x. 2 hadoop hadoop        28 Apr  5 01:17 maven-archiver
drwxrwxr-x. 3 hadoop hadoop        22 Apr  5 01:17 maven-shared-archive-resources
drwxrwxr-x. 3 hadoop hadoop        22 Apr  5 01:17 test-classes
drwxrwxr-x. 2 hadoop hadoop         6 Apr  5 01:17 test-dir

[hadoop@hadoop001 bin]$ pwd
/home/hadoop/source/hadoop-2.6.0-cdh5.7.0/hadoop-dist/target/hadoop-2.6.0-cdh5.7.0/bin
[hadoop@hadoop001 bin]$ ./hadoop checknative -a
19/04/05 07:58:50 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
19/04/05 07:58:50 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
snappy:  true /lib64/libsnappy.so.1
lz4:     true revision:99
bzip2:   true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so