搭建hive环境
从官网http://hive.apache.org/下载hive-0.8.1-bin.tar.gz,下载完成后copy hive-0.8.1-bin.tar.gz到服务器目录下
执行tar -zxvf hive-0.8.1-bin.tar.gz 解压hive。
将hive加入环境变量,主要是方便hive命令的执行,命令如下
exprot HIVE_HOME=/home/hive-0.8.1
exprot PATH=$HIVE_HOME:$PATH
将conf下面的template文件copy一份,便于个性化配置
cp hive-default.xml.template hive-default.xml
cp hive-default.xml.template hive-site.xml
cp hive-env.sh.template hive-env.sh
cp hive-log4j.properties.template hive-log4j.properties
在hive-env.sh中添加HADOOP_HOME的安装目录地址
在hive-log4j.properties中将log4j.appender.EventCounter的值修改为org.apache.hadoop.log.metrics.EventCounter,这样就不会报WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.的警告了。
以上一切完成后,执行
root@wenbo00:/home/hive-0.8.1-bin/conf# hive
进入hive命令行模式,然后执行
hive> show tables;
OK
Time taken: 6.909 seconds
出现以上结果表示安装成功。
执行建表语句
hive> create table invites(foo INT, bar STRING) partitioned by (ds STRING);
OK
Time taken: 5.918 seconds
查看结果
hive> show tables;
OK
invites
Time taken: 0.246 seconds
加载数据
hive> LOAD DATA LOCAL INPATH '/home/wenbo/kv2.txt' OVERWRITE INTO TABLE invites PARTITION (ds='2012-03-16 17:56:25');
hive> LOAD DATA LOCAL INPATH '/home/wenbo/kv3.txt' OVERWRITE INTO TABLE invites PARTITION (ds='2012-03-16 17:57:25');
查询数据
hive> select count(*) from invites;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201203160053_0004, Tracking URL = http://wenbo00:50030/jobdetails.jsp?jobid=job_201203160053_0004
Kill Command = /home/hadoop-1.0.1/libexec/../bin/hadoop job -Dmapred.job.tracker=wenbo00:9001 -kill job_201203160053_0004
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2012-03-16 03:03:25,739 Stage-1 map = 0%, reduce = 0%
2012-03-16 03:03:37,819 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:38,825 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:39,837 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:40,852 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:41,870 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:42,879 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:43,885 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:44,898 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:45,907 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:46,914 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:47,926 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:48,933 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:49,949 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 9.74 sec
2012-03-16 03:03:50,958 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 9.74 sec
2012-03-16 03:03:51,964 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 9.74 sec
2012-03-16 03:03:52,978 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 9.74 sec
2012-03-16 03:03:53,997 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 9.74 sec
2012-03-16 03:03:55,016 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 9.74 sec
2012-03-16 03:03:56,029 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 9.74 sec
MapReduce Total cumulative CPU time: 9 seconds 740 msec
Ended Job = job_201203160053_0004
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 1 Accumulative CPU: 9.74 sec HDFS Read: 6342 HDFS Write: 4 SUCESS
Total MapReduce CPU Time Spent: 9 seconds 740 msec
OK
525
Time taken: 42.688 seconds
查询的时候利用mapreduce创建任务执行,可惜我这里的环境是三台虚拟机运行在一台windows7上面,无法发挥集群优势,才导致一个简单的查询就耗费了将近43秒。
分享到:
相关推荐
hadoop+hbase+hive集群搭建
该文档目录如下: ...1.1 基于Hadoop的数据仓库Hive学习指南 1.2实验环境 1.3实验原理 1.3.1 Hive简介 1.3.2 Hive安装 1.3.3安装并配置mysql 1.3.5 Hive简单编程实践 1.3.4 Hive的常用HiveQL操作
VM虚拟机上,安装ubantu搭建hadoop+Hive集群,步骤详细。
win10下搭建Hadoop(jdk+mysql+hadoop+scala+hive+spark),包括jdk的安装、mysql安装和配置,hadoop安装和配置,scala安装和配置,hive安装和配置,spark安装和配置。
1、内容概要:Hadoop+Spark+Hive+HBase+Oozie+Kafka+Flume+Flink+Elasticsearch+Redash等大数据集群及组件搭建指南(详细搭建步骤+实践过程问题总结)。 2、适合人群:大数据运维、大数据相关技术及组件初学者。 3、...
本文件包含hadoop集群搭建的详细步骤,包含基础环境搭建,Hadoop集群搭建,Hive搭建。小白放心食用,无坑。 其中基础环境搭建包括虚拟机安装、centos7、网络配置、xshell、notepad等环境的安装。注:本文里安装的...
很多刚入门的同学找不到版本对应关系,这里从官网整理下来,供大家参考 hadoop、hbase、hive版本对应关系.新手指导hadoop、hbase、hive版本对应关系查找表
Hadoop hbase hive sqoop集群环境安装配置及使用文档
Hadoop集群搭建及Hive的安装与使用
windows系统下eclipse集成hadoop,spark,hive开发环境
Hadoop2.x HA环境搭建Hadoop2.x HA环境搭建Hadoop2.x HA环境搭建
手把手教你进行mac搭建hadoop和hive环境
大数据笔记,包含Hadoop、Spark、Flink、Hive、Kafka、Flume、ZK...... 大数据笔记,包含Hadoop、Spark、Flink、Hive、Kafka、Flume、ZK...... 大数据笔记,包含Hadoop、Spark、Flink、Hive、Kafka、Flume、ZK.......
自己整理的Hadoop环境的一些安装,和一些简单的使用,其中包括Hadoop、hbase、hive、mysql、zookeeper、Kafka、flume。都是一些简单的安装步骤和使用,只在自己的虚拟机(Linux centOS7)上使用测试过。按照步骤一步...
适合新手,详细 01-Java环境安装 02- Eclipse下载与安装 03-VMware虚拟机的安装 04-在VMware中安装CentOS 05- Hadoop集群+ Hive+ MySQL搭建
七月在线七月在线## Note, this file is written by cloud-init on first boot of an instance
厦门大学林子雨编著-基于Hadoop的数据仓库Hive,很详细的教程
机器学习算法的实现,对Hadoop,Spark,Hive等的搭建及其使用
hadoop+spark+hive Linux centos大数据集群搭建,简单易懂,从0到1搭建大数据集群