logstash 采集mysql 到elasticsearch

logstash input output配置,logstash-mysql-user.conf

input {
  jdbc {
    jdbc_driver_library => "mysql-connector-java-5.1.47.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://192.168.254.133:3306/9tong_user"
    jdbc_user => "develop"
	jdbc_password => "testdevelop"
	use_column_value => true
	tracking_column => "modifytime"
	schedule => "* * * * *"
    statement => "select ui.uid as uid,ui.chineseName as chineseName,ui.job as job,ui.englishName as englishName,ui.avatar as avatar,ui.sex as sex,ui.company as company,ui.city as city,ui.province as province,ui.mobilePhone as mobilePhone,ui.emailAddress as emailAddress,ui.email2nd as email2nd,ui.email3rd as email3rd,ui.imQq as imQq,ui.industry as industry,ui.industryUnionCode as industryUnionCode,ui.modifyTime as modifyTime,ui.linkedInId as linkedInId,ui.sinaWeiboId as sinaWeiboId,ui.weChat as weChat,ui.vAuth as vAuth,ui.member as member,ui.personIUCode as personIUCode,ui.identityType as identityType,ui.sourceOf as sourceOf,
		(select ua.account from user_account ua where ua.uid = ui.uid) as account,
		(select ua.lastActive from user_account ua where ua.uid = ui.uid) as lastActive,
		(select us.recruitType from user_sdr us where us.uid = ui.uid) as recruitType,
		(select us.demandInfo from user_sdr us where us.uid = ui.uid) as demandInfo,
		(select us.recruitInfo from user_sdr us where us.uid = ui.uid) as recruitInfo,
		(select us.supplyInfo from user_sdr us where us.uid = ui.uid) as supplyInfo,
		(select us.recruitType from user_sdr us where us.uid = ui.uid) as recruitType,
		(select ae.version from account_extension ae where ae.uid = ui.uid) as version,
		(select ae.channel from account_extension ae where ae.uid = ui.uid) as channel
		from user_info ui  where ui.modifyTime >=:sql_last_value limit 10000"
   }
}

output {
    elasticsearch {
        hosts => [ "192.168.254.212:9200" ]
		index => "user"
		document_id => "%{uid}"
		action => "update"
		doc_as_upsert => true
    }
}

启动logstash

bin/logstash -f logstash-mysql-user.conf

(全文完)

(转载本站文章请注明作者和出处 logstash 采集mysql 到elasticsearch