【Apache DolphinScheduler介绍】

Apache DolphinScheduler

分布式易扩展的可视化DAG工作流任务调度系统

 

Apache DolphinScheduler是一个分布式去中心化，易扩展的可视化DAG工作流任务调度系统。致力于解决数据处理流程中错综复杂的依赖关系，使调度系统在数据处理流程中开箱即用。

 

 

DolphinScheduler提供了许多易于使用的功能，可加快数据ETL工作开发流程的效率。其主要特点如下：

一个分布式易扩展的可视化DAG工作流任务调度系统。致力于解决数据处理流程中错综复杂的依赖关系，使调度系统在数据处理流程中开箱即用。 其主要目标如下：

• 以DAG图的方式将Task按照任务的依赖关系关联起来，可实时可视化监控任务的运行状态
• 支持丰富的任务类型：Shell、MR、Spark、SQL(mysql、postgresql、hive、sparksql),Python,Sub_Process、Procedure等
• 支持工作流定时调度、依赖调度、手动调度、手动暂停/停止/恢复，同时支持失败重试/告警、从指定节点恢复失败、Kill任务等操作
• 支持工作流优先级、任务优先级及任务的故障转移及任务超时告警/失败
• 支持工作流全局参数及节点自定义参数设置
• 支持资源文件的在线上传/下载，管理等，支持在线文件创建、编辑
• 支持任务日志在线查看及滚动、在线下载日志等
• 实现集群HA，通过Zookeeper实现Master集群和Worker集群去中心化
• 支持对Master/Worker cpu load，memory，cpu在线查看
• 支持工作流运行历史树形/甘特图展示、支持任务状态统计、流程状态统计
• 支持补数
• 支持多租户
• 支持国际化

 

 

Design Features

DolphinScheduler is a distributed and extensible workflow scheduler platform with powerful DAG visual interfaces, dedicated to solving complex job dependencies in the data pipeline and providing various types of jobs available out of the box.

Its main objectives are as follows:

• Associate the tasks according to the dependencies of the tasks in a DAG graph, which can visualize the running state of the task in real-time.
• Support various task types: Shell, MR, Spark, SQL (MySQL, PostgreSQL, hive, spark SQL), Python, Sub_Process, Procedure, etc.
• Support scheduling of workflows and dependencies, manual scheduling to pause/stop/recover task, support failure task retry/alarm, recover specified nodes from failure, kill task, etc.
• Support the priority of workflows & tasks, task failover, and task timeout alarm or failure.
• Support workflow global parameters and node customized parameter settings.
• Support online upload/download/management of resource files, etc. Support online file creation and editing.
• Support task log online viewing and scrolling and downloading, etc.
• Have implemented cluster HA, decentralize Master cluster and Worker cluster through Zookeeper.
• Support the viewing of Master/Worker CPU load, memory, and CPU usage metrics.
• Support displaying workflow history in tree/Gantt chart, as well as statistical analysis on the task status & process status in each workflow.
• Support back-filling data.
• Support multi-tenant.
• Support internationalization.