Azkaban配置以及azkaban web服务应用

若需要相应的其他工具的配置，详细请看《hadoop大数据生态圈工具配置与应用》

文章目录

• ​​Azkaban配置​​
• ​​azkaban web服务安装​​
• ​​案例 依赖任务调度管理​​
• ​​mapreduce任务调度管理​​
• ​​案例 HIVE脚本任务调度管理​​

Azkaban配置

hadoop01进行

先yum -y install git

./gradlew build -x test

主节点

[root@hadoop01 distributions]# vim /etc/my.cnf

重启mysql

[root@hadoop01 distributions]# sudo /sbin/service mysqld restart

azkaban数据库表初始化

分别进去把这四个文件夹里build/distributions里的包解压，也可以先解压azkaban-db

解压后显示结构

启动mysql

mysql> source /export/servers/azkaban-3.51.4/azkaban-db/build/distributions/azkaban-db-0.1.0-SNAPSHOT/create-all-sql-0.1.0-SNAPSHOT.sql

azkaban web服务安装

随便一个目录

[root@hadoop01 servers]# keytool -keystore keystore -alias jetty -genkey -keyalg RSA

[root@hadoop01 distributions]# tar xzvf azkaban-web-server-0.1.0-SNAPSHOT.tar.gz -C /export/servers/azkaban

[root@hadoop01 azkaban-solo-server-0.1.0-SNAPSHOT]# cp -rf plugins/export/servers/azkaban/azkaban-web-server-0.1.0-SNAPSHOT/

[root@hadoop01 conf]# vim azkaban.properties 

添加以下内容：
# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml
# Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.use.ssl=true
jetty.maxThreads=25
jetty.port=8081
# Azkaban Executor settings
# mail settings
jetty.ssl.port=8443
jetty.keystore=keystore
jetty.password=123456
jetty.keypassword=123456
jetty.truststore=keystore
jetty.trustpassword=123456
mail.sender=
mail.host=

executor.port=12321
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban mysql settings by default. Users should configure their own username and password.
database.type=mysql
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=root
mysql.password=Dn@123456
mysql.numconnections=100



azkaban.jobtype.plugin.dir=plugins/jobtypes

传入刚开始创建的密钥

[root@hadoop01 conf]# mv /export/servers/keystore /export/servers/azkaban/azkaban-web-server-0.1.0-SNAPSHOT/

[root@hadoop01 conf]# vim azkaban-users.xml

<azkaban-users>
  <user groups="azkaban" password="azkaban" roles="admin" username="azkaban"/>
  <user password="metrics" roles="metrics" username="metrics"/>

  <user password="admin" roles="metrics,admin" username="admin"/>

  <role name="admin" permissions="ADMIN"/>
  <role name="metrics" permissions="METRICS"/>
</azkaban-users>

[root@hadoop01 conf]# vim log4j.properties 

添加以下内容：
log4j.rootLogger=INFO, Console
log4j.logger.azkaban=INFO, server
log4j.appender.server=org.apache.log4j.RollingFileAppender
log4j.appender.server.layout=org.apache.log4j.PatternLayout
#log4j.appender.server.File=logs/azkaban-webserver.log
log4j.appender.server.File=logs/azkaban-server.log

log4j.appender.server.layout.ConversionPattern=%d{yyyy/MM/dd HH:mm:ss.SSS Z} %p [%c{1}] [Azkaban] %m%n
log4j.appender.server.MaxFileSize=102400MB
log4j.appender.server.MaxBackupIndex=2
log4j.appender.Console=org.apache.log4j.ConsoleAppender
log4j.appender.Console.layout=org.apache.log4j.PatternLayout
log4j.appender.Console.layout.ConversionPattern=%d{yyyy/MM/dd HH:mm:ss.SSS Z} %p [%c{1}] [Azkaban] %m%n

[root@hadoop01 distributions]# tar xzvf azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz -C /export/servers/azkaban

[root@hadoop01 azkaban-web-server-0.1.0-SNAPSHOT]# cp -rf conf /export/servers/azkaban/azkaban-exec-server-0.1.0-SNAPSHOT/

修改三处

executor.port=12321
executor.maxThreads=50
executor.flow.threads=30

启动 azkaban executor

查看日志

两个都打开后，查看服务情况

案例 依赖任务调度管理

建立两个文件，并压缩成zip

#foo.job

type=command

command=echo foo

#bar.job

type=command

dependencies=foo

command=echo bar

[root@hadoop01 ~]# zip -r bar.zip foo.job bar.job

winscp把bar.zip取出

点击schedule

mapreduce任务调度管理

从winscp拿出两种文件

然后打包

#wordcount_mr.job

type=command

command=hadoop jar hadoop-mapreduce-examples-2.7.4.jar wordcount hdfs://hadoop01:9000/wordcount/input/wctest.txt hdfs://hadoop01:9000/wordcount/input/mrjobresult

需要统计的文本

创建项目命名如下

选择立即执行

案例 HIVE脚本任务调度管理

开启hadoop

所需数据如下：

#test.sql

use default;

drop table aztest;

create table aztest(id int,name string) row format

delimited fields terminated by ‘,’;

load data inpath ‘/aztest/hiveinput’ into table aztest;

insert overwrite directory ‘/aztest/hiveoutput’

select count(1) from aztest;

#hive.job

type=command

command=/export/servers/apache-hive-1.2.1-bin/bin/hive -f ‘test.sql’

#aztest.txt

1,allen

2,tom

3,jerry

从winscp拿出两种文件

打包hive.zip

创建项目导入zip