dataX3.0安装使用手册.doc

《dataX3.0安装使用手册.doc》由会员分享，可在线阅读，更多相关《dataX3.0安装使用手册.doc（11页珍藏版）》请在课桌文档上搜索。

1、-Data*3.0使用手册目录一、data*概述31data*作用32Data*3.0框架设计33Data*3.0插件体系3二、data*安装31创立用户组及用户32环境变量配置33安装python3三、data*使用31生成配置文件模板32配置定时任务33常见问题及处理3四、data*插件开发及安装31.Reader插件开发32.Writter插件开发33.插件部署3一、 data*概述1 data*作用Data* 是一个异构数据源离线同步工具，致力于实现包括关系型数据库(MySQL、Oracle等)、HDFS、Hive、ODPS、HBase、FTP等各种异构数据源之间稳定高效的数据同步功能

2、。为了解决异构数据源同步问题，Data*将复杂的网状的同步链路变成了星型数据链路，Data*作为中间传输载体负责连接各种数据源。当需要接入一个新的数据源的时候，只需要将此数据源对接到Data*，便能跟已有的数据源做到无缝数据同步。2 Data*3.0框架设计Data*本身作为离线数据同步框架，采用Framework + plugin架构构建。将数据源读取和写入抽象成为Reader/Writer插件，纳入到整个同步框架中。Reader：Reader为数据采集模块，负责采集数据源的数据，将数据发送给Framework。Writer： Writer为数据写入模块，负责不断向Framework取数据，

3、并将数据写入到目的端。Framework：Framework用于连接reader和writer，作为两者的数据传输通道，并处理缓冲，流控，并发，数据转换等核心技术问题。3 Data*3.0插件体系类型数据源Reader(读)Writer(写)RDBMS 关系型数据库MySQLOracleSQL ServerPostgreSQL达梦通用RDBMS(支持所有关系型数据库)阿里云数仓数据存储Ma*Compute(原ODPS)Analytic DB(原ADS)OSS云数据库Memcache版(原OCS)HiveNoSQL数据存储Table Store(原OTS)Hbase0.94Hbase1.1Mon

4、goDB无构造化数据存储T*tFileJsonFileFTPHDFS二、 data*安装1 创立用户组及用户roothmaster-hdfs:/home/ubuntu# groupadd data*roothmaster-hdfs:/home/ubuntu# useradd -g data* data* -m -d /home/data*roothmaster-hdfs:/home/ubuntu# passwd data*Enter new UNI* password: Retype new UNI* password: passwd: password updated successfull

5、yroothmaster-hdfs:/home/ubuntu# usermod -G adm -a data*roothmaster-hdfs:/home/ubuntu# su - data*data*hmaster-hdfs:$ iduid=1004(data*) gid=1000(ubuntu) groups=1000(ubuntu),4(adm),1002(data*)2 环境变量配置e*port JRE_HOME=$JAVA_HOME/jree*port CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/libe*port PATH=$JAVA_HOME/bin

6、e*port LANG=zh_CN.UTF-83 安装pythondata*hmaster-hdfs:/home/ubuntu$ pythonPython 2.7.6 (default, Jun 22 2015, 17:58:13) GCC 4.8.2 on linu*2Type help, copyright, credits or license for more information.三、 data*使用1 生成配置文件模板命令：bin/python data*.py -r YOUR_READER -w YOUR_WRITER案例1：mysqlreader- mysqlwriterda

7、ta*hmaster-hdfs:/data*3/data*/bin$ python data*.py -r mysqlreader -w mysqlwriterData* (DATA*-OPENSOURCE-1.0), From Alibaba !Copyright (C) 2010-2015, Alibaba Group. All Rights Reserved.Please refer to the mysqlreader document: s:/github./alibaba/Data*/blob/master/mysqlreader/doc/mysqlreader.md Please

8、 refer to the mysqlwriter document: s:/github./alibaba/Data*/blob/master/mysqlwriter/doc/mysqlwriter.mdPlease save the following configuration as a json file and use python DATA*_HOME/bin/data*.py JSON_FILE_NAME.json to run the job. job: content: reader: name: mysqlreader, parameter: column: , conne

9、ction: jdbcUrl: , table: , password: , username: , where: , writer: name: mysqlwriter, parameter: column: , connection: jdbcUrl: , table: , password: , preSql: , session: , username: , writeMode: , setting: speed: channel: 案例2：jsonfilereader- hbasewriterdata*hmaster-hdfs:/data*3/data*/bin$ python da

10、ta*.py -r jsonfilereader -w hbasewriter Data* (DATA*-OPENSOURCE-1.0), From Alibaba !Copyright (C) 2010-2015, Alibaba Group. All Rights Reserved.Please refer to the jsonfilereader document: s:/github./alibaba/Data*/blob/master/jsonfilereader/doc/jsonfilereader.md Please refer to the hbasewriter docum

11、ent: s:/github./alibaba/Data*/blob/master/hbasewriter/doc/hbasewriter.md Please save the following configuration as a json file and use python DATA*_HOME/bin/data*.py JSON_FILE_NAME.json to run the job. job: content: reader: name: jsonfilereader, parameter: column: , compress: zip, encoding: , path:

12、 , writer: name: hbase11*writer, parameter: column: , encoding: , hbaseConfig: hbase.cluster.distributed: , hbase.rootdir: , hbase.zookeeper.quorum: , mode: , rowkeyColumn: , table: , versionColumn: inde*: , value: , setting: speed: channel: 2 配置定时任务1.1 创立任务列表文件data*hmaster-hdfs:/data*3/data*/task$

13、vi crontab.tasks.t*t 20,30,40 0-23 * * * /usr/bin/python /home/data*/data*3/data*/bin/data*.py /home/data*/data*3/data*/job/jsonfile2hbase_fullry_log.terminal_log.json /data/data*/log/data*.crontab.out.log 2&11.2 crontab加载系统任务列表文件data*hmaster-hdfs:/data*3/data*/task$ crontab crontab.tasks.t*t data*h

14、master-hdfs:/data*3/data*/task$ crontab -l20,30,40 0-23 * * * /usr/bin/python /home/data*/data*3/data*/bin/data*.py /home/data*/data*3/data*/job/jsonfile2hbase_fullry_log.terminal_log.json /data/data*/log/data*.crontab.out.log 2&13 常见问题及处理1.1 /bin/sh: 1: java: not found原因：/home/data*/data*3/data*/bi

15、n/ data*.py未取到JAVA_HOME环境变量值四、 data*插件开发及安装1. Reader插件开发publicclass JsonFileReader e*tends Readerpublicstaticclass Job e*tendsReader.Job Overridepublicvoidinit() / TODO Auto-generated method stubOverridepublicvoiddestroy() / TODO Auto-generated method stubOverridepublic List split(intadviceNumber) /

16、 TODO Auto-generated method stubreturnnull;publicstaticclass Task e*tends Reader.TaskOverridepublicvoid init() / TODO Auto-generated method stubOverridepublicvoid destroy() / TODO Auto-generated method stubOverridepublicvoid startRead(RecordSender recordSender) / TODO Auto-generated method stub2. Wr

17、itter插件开发publicclass JsonFileWriter e*tends Writer publicstaticclass Job e*tends Writer.Job Overridepublicvoid init() / TODO Auto-generated method stubOverridepublicvoid destroy() / TODO Auto-generated method stubOverridepublic List split(intmandatoryNumber) / TODO Auto-generated method stubreturnnu

18、ll;publicstaticclassTaske*tends Writer.Task Overridepublicvoid init() / TODO Auto-generated method stubOverridepublicvoid destroy() / TODO Auto-generated method stubOverridepublicvoid startWrite(RecordReceiver lineReceiver) / TODO Auto-generated method stub3. 插件部署目录及文件说明data*hmaster-hdfs:/data*3/dat

19、a*/plugin/reader/jsonfilereader$ lltotal 56drw*r-*r-* 3 data* ubuntu 4096 Jul 6 16:10 ./drw*r-*r-* 21 data* ubuntu 4096 Jun 21 11:05 ./-rw-r-r- 1 data* ubuntu 36348 Jun 22 17:52 jsonfilereader.jardrw*r-*r-* 2 data* ubuntu 4096 May 16 12:46 libs/-rw- 1 data* ubuntu 315 May 16 11:24 plugin.json-rw- 1

20、data* ubuntu 149 May 15 18:24 plugin_job_template.jsonl jsonfilereader.jar 插件JAR包文件l libs 插件依赖第三方JAR包文件l plugin.json 插件说明文件l plugin_job_template.json 插件配置模板文件plugin.json文件容说明 name: jsonfilereader, class: .alibaba.data*.plugin.reader.jsonfilereader.JsonFileReader, description: useScene: test. mechani

21、sm: use data* framework to transport data from json file. warn: The more you know about the data, the less problems you encounter., developer: alibabaName：插件名称Class：jsonfilereader.jar文件中全路径Description：功能描述plugin_job_template.json文件容说明 name: jsonfilereader, parameter: path: , encoding: , column: , compress:zip . z.