Hbase支持snappy压缩格式

说明

本文的整个操作是在根帐户下进行的。所有步骤完成后,您可以根据实际需要更改目录权限。[en]The whole operation of this article is carried out under the root account. After all the steps are completed, you can change the directory permissions according to the actual requirements.
整个行动的目的有两个:[en]The purpose of the whole operation is two:

  • 安装Snappy包。安装后,路径为/usr/local/lib/。[en]* install the snappy package. After installation, the path is / usr/local/lib/.
  • 安装hadoop依赖的snappy包,安装完成后所在路径是 $HADOOP_HOME/lib/native

然后将上述两个副本分发到所有节点,第二个副本是在安装Hadoop时复制的,通常第一个分发副本操作可能会被忘记。[en]The above two copies are then distributed to all nodes, the second of which is copied when hadoop is installed, and usually the first distribution copy operation may be forgotten.

1. 安装编译环境

1.1 安装C++编译器

yum install -y automake autoconf gcc-c++ libedit libtool openssl-devel ncurses-devel

1.2 安装cmake

注意版本需要大于3
检查cmake的版本。如果是2.8.12.2,则需要卸载后单独安装。[en]Check the version of cmake. If it is 2.8.12.2, you need to uninstall it and install it separately.

cmake --version

卸载cmake[en]Uninstall cmake

yum remove cmake -y

重新安装cMake 3.9版本[en]Reinstall the cmake3.9 version

wget https://cmake.org/files/v3.9/cmake-3.9.2.tar.gz
tar zxvf cmake-3.9.2.tar.gz
cd cmake-3.9.2
./configure
make & make install

1.3 安装protobuf-2.5.0

版本必须是该版本[en]The version must be that version

tar zxvf protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
./configure
make & make install

安装完成后,注销并重新登录。[en]After the installation is complete, log out and log back in.

1.4 安装maven

yum install wget -y
wget http://mirror.bit.edu.cn/apache/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz
tar xvf apache-maven-3.5.4-bin.tar.gz

修改环境变量/etc/PROFILE[en]Modify the environment variable / etc/profile

export MAVEN_HOME=/opt/local/maven
export PATH=$PATH:$MAVEN_HOME/bin

修改设置档案仓库的位置:[en]Modify the location of the setting file warehouse:

<localrepository>/opt/local/maven/repo</localrepository>

修改镜像服务器地址:[en]Modify the image server address:

    <mirror>
      <id>alimaven</id>
      <name>aliyun maven</name>
      <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
      <mirrorof>central</mirrorof>
    </mirror>

2. 安装snappy

2.1 安装snappy

wget https://src.fedoraproject.org/repo/pkgs/snappy/snappy-1.1.3.tar.gz/7358c82f133dc77798e4c2062a749b73/snappy-1.1.3.tar.gz
tar xvf snappy-1.1.3.tar.gz
cd snappy-1.1.3
./configure
make
make install
#&#x5378;&#x8F7D;&#x547D;&#x4EE4; make uninstall

注意:安装完成后,在/usr/local/lib/目录下生成对应的snappy文件

Hbase支持snappy压缩格式

/usr/local/lib 目录下所有文件拷贝到 其他节点相同目录下。

2.2 编译hadoop-snappy(未做似乎不影响)

wget下载,或者直接从git地址 https://github.com/electrum/hadoop-snappy 手工选择download ZIP包。

wget https://github.com/electrum/hadoop-snappy/archive/refs/heads/master.zip

解压缩、编译[en]Decompress, compile

yum install unzip
unzip master.zip
cd hadoop-snappy-master
mvn package

请务必先安装Snappy,然后检查/usr/local/lib/目录下是否有对应的Snappy文件,否则此步骤不会成功打包。[en]Be sure to install snappy first, and then check that there is a corresponding snappy file in the / usr/local/lib/ directory, otherwise this step will not be packaged successfully.

Hbase支持snappy压缩格式

编译成功后,目标文件夹下如下:[en]After the compilation is successful, under the target folder is as follows:

Hbase支持snappy压缩格式

3. hadoop

hadoop编译的目的是为了获取到编译完成后的lib库,编译完成的lib库保存在路径

/opt/local/hadoop-3.1.3-src/hadoop-dist/target/hadoop-3.1.3/lib/native

然后将lib库复制到相应版本中已安装的节点。[en]Then copy the lib library to the node that has been installed in the corresponding version.
复制后,需要重新启动节点。[en]After copying, the node needs to be restarted.

3.1 编译hadoop

wget http://archive.apache.org/dist/hadoop/core/hadoop-3.1.3/hadoop-3.1.3-src.tar.gz
tar -zxvf hadoop-3.1.3-src.tar.gz
cd hadoop-3.1.3-src

hadoop-3.1.3-src目录里新建文件compile.sh,填写以下内容,并修改权限755。

vim compile.sh
export MAVEN_OPTS="-Xms256m -Xmx512m"
mvn clean package -Pdist,native -DskipTests -Dtar -rf :hadoop-common -Drequire.snappy -X

这种编译方式将Snappy包直接编译在一起,编译过程需要20多分钟。[en]This kind of compilation compiles the snappy packages directly together, and the compilation process takes more than 20 minutes.

chmod 755 compile.sh
./compile.sh

编译成功如下:[en]The successful compilation is as follows:

Hbase支持snappy压缩格式

网上还有一段增加其他依赖的编译命令如下,可以自行尝试。

#!/bin/bash
mvn package -Pdist,native,nexus -DskipTests=true -Dmaven.javadoc.skip=true -Dtar -Dcontainer-executor.conf.dir=/etc/hadoop/ -Drequire.snappy -Dbundle.snappy -Dsnappy.lib=/opt/develop/snappy/lib -Dsnappy.prefix=/opt/develop/snappy -Drequire.openssl -Dbundle.openssl -Dopenssl.lib=/opt/develop/openssl/lib -Dopenssl.prefix=/opt/develop/openssl -Drequire.bzip2=true -Dzstd.prefix=/opt/develop/zstd -Dzstd.lib=/opt/develop/zstd/lib -Dbundle.zstd -Drequire.zstd -Disal.prefix=/opt/develop/isal -Drequire.isal=true -Dbundle.isal -Disal.lib=/opt/develop/isal/lib

3.2 修改hadoop配置文件

3.2.1 修改core-site.xml

<!-- 用于压缩/解压的CompressionCodec列表,多种算法使用逗号分隔的类名  -->
<property>
    <name>io.compression.codecs</name>
    <value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>

3.2.2 修改yarn-site.xml

<!-- 压缩MapReduce作业的输出 -->
<property>
    <name>mapreduce.map.output.compress</name>
    <value>true</value>
</property>
<!-- 指定压缩算法的类名 -->
<property>
    <name>mapred.map.output.compress.codec</name>
    <value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>

4. 拷贝文件

4.1 分发snappy文件

如果2.1没做拷贝,就将 /usr/local/lib 目录下所有文件拷贝到 其他节点相同目录下

 scp -r /usr/local/lib/*  hadoop100:/usr/local/lib/

4.2 分发hadoop lib库

将hadoop目录中lib/native文件夹复制拷贝到 /opt/local/hbase/lib/目录下,需要所有节点都拷贝。

scp -r $HADOOP_HOME/lib/native hadoop100:/opt/local/hbase/lib/

5. 修改hbase配置

通过在末尾添加以下配置来修改hbase-env.sh:[en]Modify hbase-env.sh by adding the following configuration at the end:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native/:/usr/local/lib/
export HBASE_LIBRARY_PATH=$HBASE_LIBRARY_PATH:$HBASE_HOME/lib/native/:/usr/local/lib/
export CLASSPATH=$CLASSPATH:$HBASE_LIBRARY_PATH

修改后,同步所有节点上的文件。[en]After modification, synchronize the file on all nodes.

6. Hbase测试

create 'test2',{NAME=>'f1',COMPRESSION => 'gz'}
create 'test3',{NAME=>'f1',COMPRESSION => 'SNAPPY'}

Original: https://www.cnblogs.com/30go/p/16243455.html
Author: 硅谷工具人
Title: Hbase支持snappy压缩格式

原创文章受到原创版权保护。转载请注明出处:https://www.johngo689.com/6002/

转载文章受原作者版权保护。转载请注明原作者出处!

(0)

大家都在看

免费咨询
免费咨询
扫码关注
扫码关注
联系站长

站长Johngo!

大数据和算法重度研究者!

持续产出大数据、算法、LeetCode干货,以及业界好资源!

2022012703491714

微信来撩,免费咨询:xiaozhu_tec

分享本页
返回顶部
最近整理资源【免费获取】:   👉 程序员最新必读书单  | 👏 互联网各方向面试题下载 | ✌️计算机核心资源汇总