Flink1.12 使用WatermarkStrategy生成时间戳
程序员文章站
2022-07-14 12:27:50
...
Flink1.12版本后建议使用assignTimestampsAndWatermarks(WatermarkStrategy)的方式生成watermark,之前使用的assignTimestampsAndWatermarks(AssignerWithPunctuatedWatermarks)以及assignTimestampsAndWatermarks(timestampAndWatermarkAssigner)均不再被推荐使用,这里是根据官网给出的小例子,仅供参考:
package demo;
import org.apache.flink.api.common.eventtime.SerializableTimestampAssigner;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows;
import org.apache.flink.streaming.api.windowing.time.Time;
import org.apache.flink.util.Collector;
import java.time.Duration;
/**
* Author: Chenghui Bai
* Date: 2021/3/4 17:48
* ProjectName: frauddetection
* PackageName: demo
* ClassName: WatermarkWCDemo
* Version:
* Description:
*/
public class WatermarkWCDemo {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStreamSource<String> inputStream = env.socketTextStream("localhost", 7777);
SerializableTimestampAssigner<String> timestampAssigner = new SerializableTimestampAssigner<String>() {
@Override
public long extractTimestamp(String element, long recordTimestamp) {
String[] fields = element.split(",");
Long aLong = new Long(fields[0]);
return aLong * 1000L;
}
};
SingleOutputStreamOperator<String> watermarkStream = inputStream.assignTimestampsAndWatermarks(WatermarkStrategy.<String>forBoundedOutOfOrderness(Duration.ofSeconds(10)).withTimestampAssigner(timestampAssigner));
watermarkStream.flatMap(new FlatMapFunction<String, Tuple2<String, Integer>>() {
@Override
public void flatMap(String value, Collector<Tuple2<String, Integer>> out) throws Exception {
String[] fields = value.split(",");
out.collect(new Tuple2<>(fields[1], 1));
}
}).keyBy(data -> data.f0).window(TumblingEventTimeWindows.of(Time.seconds(10))).sum(1).print();
env.execute("run watermark wc");
}
}
程序中,输入的nc数据形如:
1614909710,java
1614909701,python
1614909700,python
1614909702,python
下一篇: Flink1.12 通过CEP实现告警
推荐阅读
-
ms sql server中实现的unix时间戳函数(含生成和格式化,可以和mysql兼容)
-
PHP时间戳 strtotime()使用方法和技巧
-
分享8个Laravel模型时间戳使用技巧小结
-
ms sql server中实现的unix时间戳函数(含生成和格式化,可以和mysql兼容)
-
PHP时间戳使用实例代码
-
使用php 获取时间今天明天昨天时间戳的详解
-
使用CTE生成辅助表(数字或时间)等
-
mysql使用Sql获取13位时间戳
-
LocalDateTime在项目中的使用(LocalDateTime对接前端通过时间戳互转、LocalDateTime对接数据库)
-
Flink1.12 使用WatermarkStrategy生成时间戳