1、Flink快速入门(Quickstart)

Flink官方文档：https://ci.apache.org/projects/flink/flink-docs-release-1.5/quickstart/setup_quickstart.html

本文档只讲解Flink在Linux系统中的安装使用。Windows用户，请查看官方文档：

https://ci.apache.org/projects/flink/flink-docs-release-1.5/start/flink_on_windows.html

1.1下载并启动Flink

(1)检查JDK。

为了能够运行Flink，唯一的要求是安装一个有效的Java8.x。

可以通过以下命令来检查Java是否正确安装：

[[e-mail] ~]$ java -version

如果有Java8，将输出如下结果：

java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

(2)下载Flink。

Flink下载：http://flink.apache.org/downloads.html

解压下载的文件：

[[e-mail] core]$ tar -zxvf flink-1.5.1-bin-hadoop27-scala_2.11.tgz

为使用方便，将解压后的目录更名为flink：

[[e-mail] core]$ mv flink-1.5.1 flink

(3)启动Flink。

启动本地Flink集群：

[[e-mail] flink]$ ./bin/start-cluster.sh
Starting cluster.
Starting standalonesession daemon on host rdpecore1.
Starting taskexecutor daemon on host rdpecore1.

(4)验证Flink是否正常启动。

打开http://rdpecore1:8081检查Flink组件是否正常运行。Web UI中应该会显示只有一个可用的TaskManager实例。

还可以通过检查log目录中的日志文件来验证系统是否正在运行：

[[e-mail] flink]$ tail log/flink-hadoop-standalonesession-2-rdpecore1.log
2018-07-24 14:20:23,017 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - ResourceManager akka.tcp://[e-mail]:6123/user/resourcemanager was granted leadership with fencing token 00000000000000000000000000000000
2018-07-24 14:20:23,018 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Starting the SlotManager.
2018-07-24 14:20:23,052 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Dispatcher akka.tcp://[e-mail]:6123/user/dispatcher was granted leadership with fencing token 00000000-0000-0000-0000-000000000000
2018-07-24 14:20:23,052 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Recovering all persisted jobs.
2018-07-24 14:20:26,849 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Registering TaskManager 3df9cee680a5c6053d28d2cabd17a9cf under 8453da1201b81a610c18ac23f6cef4c0 at the SlotManager.

1.2阅读样例代码

GitHub上有样例SocketWindowWordCount的完整源代码。

(1)Scala代码：

https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/scala/org/apache/flink/streaming/scala/examples/socket/SocketWindowWordCount.scala

/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.flink.streaming.scala.examples.socket

import org.apache.flink.api.java.utils.ParameterTool
import org.apache.flink.streaming.api.scala._
import org.apache.flink.streaming.api.windowing.time.Time

/**
 * Implements a streaming windowed version of the "WordCount" program.
 * 
 * This program connects to a server socket and reads strings from the socket.
 * The easiest way to try this out is to open a text sever (at port 12345) 
 * using the ''netcat'' tool via
 * {{{
 * nc -l 12345
 * }}}
 * and run this example with the hostname and the port as arguments..
 */
object SocketWindowWordCount {

  /** Main program method */
  def main(args: Array[String]) : Unit = {

    // the host and the port to connect to
    var hostname: String = "localhost"
    var port: Int = 0

    try {
      val params = ParameterTool.fromArgs(args)
      hostname = if (params.has("hostname")) params.get("hostname") else "localhost"
      port = params.getInt("port")
    } catch {
      case e: Exception => {
        System.err.println("No port specified. Please run 'SocketWindowWordCount " +
          "--hostname <hostname> --port <port>', where hostname (localhost by default) and port " +
          "is the address of the text server")
        System.err.println("To start a simple text server, run 'netcat -l <port>' " +
          "and type the input text into the command line")
        return
      }
    }
    
    // get the execution environment
    val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment
    
    // get input data by connecting to the socket
    val text: DataStream[String] = env.socketTextStream(hostname, port, '\n')

    // parse the data, group it, window it, and aggregate the counts 
    val windowCounts = text
          .flatMap { w => w.split("\\s") }
          .map { w => WordWithCount(w, 1) }
          .keyBy("word")
          .timeWindow(Time.seconds(5))
          .sum("count")

    // print the results with a single thread, rather than in parallel
    windowCounts.print().setParallelism(1)

    env.execute("Socket Window WordCount")
  }

  /** Data type for words with count */
  case class WordWithCount(word: String, count: Long)
}

Java代码：

https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/socket/SocketWindowWordCount.java

/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.flink.streaming.examples.socket;

import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.functions.ReduceFunction;
import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.windowing.time.Time;
import org.apache.flink.util.Collector;

/**
 * Implements a streaming windowed version of the "WordCount" program.
 *
 * <p>This program connects to a server socket and reads strings from the socket.
 * The easiest way to try this out is to open a text server (at port 12345)
 * using the <i>netcat</i> tool via
 * <pre>
 * nc -l 12345
 * </pre>
 * and run this example with the hostname and the port as arguments.
 */
@SuppressWarnings("serial")
public class SocketWindowWordCount {

	public static void main(String[] args) throws Exception {

		// the host and the port to connect to
		final String hostname;
		final int port;
		try {
			final ParameterTool params = ParameterTool.fromArgs(args);
			hostname = params.has("hostname") ? params.get("hostname") : "localhost";
			port = params.getInt("port");
		} catch (Exception e) {
			System.err.println("No port specified. Please run 'SocketWindowWordCount " +
				"--hostname <hostname> --port <port>', where hostname (localhost by default) " +
				"and port is the address of the text server");
			System.err.println("To start a simple text server, run 'netcat -l <port>' and " +
				"type the input text into the command line");
			return;
		}

		// get the execution environment
		final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

		// get input data by connecting to the socket
		DataStream<String> text = env.socketTextStream(hostname, port, "\n");

		// parse the data, group it, window it, and aggregate the counts
		DataStream<WordWithCount> windowCounts = text

				.flatMap(new FlatMapFunction<String, WordWithCount>() {
					@Override
					public void flatMap(String value, Collector<WordWithCount> out) {
						for (String word : value.split("\\s")) {
							out.collect(new WordWithCount(word, 1L));
						}
					}
				})

				.keyBy("word")
				.timeWindow(Time.seconds(5))

				.reduce(new ReduceFunction<WordWithCount>() {
					@Override
					public WordWithCount reduce(WordWithCount a, WordWithCount b) {
						return new WordWithCount(a.word, a.count + b.count);
					}
				});

		// print the results with a single thread, rather than in parallel
		windowCounts.print().setParallelism(1);

		env.execute("Socket Window WordCount");
	}

	// ------------------------------------------------------------------------

	/**
	 * Data type for words with count.
	 */
	public static class WordWithCount {

		public String word;
		public long count;

		public WordWithCount() {}

		public WordWithCount(String word, long count) {
			this.word = word;
			this.count = count;
		}

		@Override
		public String toString() {
			return word + " : " + count;
		}
	}
}

1.3运行样例

现在，我们将运行此Flink应用程序。它将从socket读取文本，并且每5秒打印一次前5秒内每个不同单词的出现次数，即处理时间的翻滚窗口，只要文字在里面浮动。

(1)首先，我们使用netcat来启动本地服务器。

[[e-mail] flink]$ nc -l 9001

启动后，终端会一直处于等待状态。

(2)再打开一个新的终端，在其中提交Flink程序。

[[e-mail] flink]$ ./bin/flink run examples/streaming/SocketWindowWordCount.jar --port 9001
Starting execution of program
Program execution finished
Job with JobID c7b4f7bd16b96ae718bf658c8b129848 has finished.
Job Runtime: 386160 ms

(3)程序连接了socket并等待输入。可以在Web UI中检查Job是否正常运行。

(4)输入数据。

单词在5秒的时间窗口（处理时间，滚动窗口）中被计算，并被打印到stdout。监控TaskManage的输出文件，并在nc中写入一些文本（输入的文本在点击后按行发送到Flink）。

[[e-mail] flink]$ nc -l 9001
hello world
flink

(5)在.out文件中查看处理结果。

[[e-mail] flink]$ tail -f log/flink-hadoop-taskexecutor-2-rdpecore1.out
hello : 1
world : 1
flink : 1

(6)停止Flink。

[[e-mail] flink]$ ./bin/stop-cluster.sh