企业安全建设之路：端口扫描（下）

0×00、前言

在企业安全建设过程当中，我们也不断在思考，做一个什么样的端口扫描才能企业业务需求。同时，伴随着企业私有云、混合云以及公有云业务部署环境的不断变化，我们适当也要对扫描策略做调整。前期的端口扫描设计在/Article/60/76/2017/84312.htm

在本文各个部分又所变动。
0×01、详细设计

@1、各个模块之间的交互：
一开始都是把产品想的特别完美，
（1）  Web控制端
（2）  worker工作节点
（3）  存储扫描结果（maybe： HDFS）
这样实现起来比较麻烦，当时说使用celery做调度，后来发现，celery对django有版本要求，超过1.10版本不成。等等现实问题。其实celery也是redis做调度数据同步。有时间可以自己做。
其实Web控制端和worker可以使用数据库做交互。用户通过Web控制端设置扫描策略和查看报表。Worker读取数据库中的配置信息，执行扫描任务，把扫描结果存储到数据库。
@2、功能需求
在对端口扫描功能的选型上，为啥选择nmap，
（1）  很多商用扫描器也是集成nmap扫描结果，例如：rapid7 Vulnerability Management。
（2）  nmap扫描速度，肯定没有masscan、Zmap快，但是扫描结果有对服务banner和版本的探测，更重要的是有操作系统的探测。在云平台部署zmap等无状态扫描，会瞬间发出大量数据包，公有云EIP带宽QoS超过会立刻丢弃，对扫描结果有很大影响。
（3）  libnmap 对扫描结果解析的相对完美，方便的提取我想要的数据到数据库中。
端口扫描后，我们还能做什么？
（1）  个人认为第一需求就是对新暴发的漏洞做企业内部评估。前几天的WannaCry就是445端口对外开发又可能触发MS-17-010的RCE。这里我集成了巡风漏洞扫描组件。
（2）  评估高危端口变化趋势，也是衡量企业安全管理人员工作成果的一个手段。
（3）  对企业内部部门漏洞分布有清晰的了解
0×02、交互设计
与用户交互部分，因为是安全管理员用，所以简单做。Axure是一个好的交互工具，可以帮助你梳理业务逻辑。
按照模块分：
（1）扫描配置

（2）扫描报表

0×03、前端实现



 
（1）开发环境建立：

brew install nodejs
npm install webpack –g
npm install --global vue-cli
vue init webpack CloudPScan
cd CloudPScan
npm install
npm install vue-resource
npm install element-ui
设置代理 config/dev.index.js
module.exports = {
  //...
 dev: {
    proxyTable: {
     // proxy all requests starting with /api to https://127.0.0.1:8000
     '/api': {
       target: 'https://127.0.0.1:8000',
       changeOrigin: true,
     }
}
}

（2）创建页面路由

import Vue from 'vue'
import Routerfrom 'vue-router'
import LoginViewfrom '@/components/LoginView'
import MainViewfrom '@/components/MainView'
import ScanSettingViewfrom '@/components/ScanSettingView'
import ScanReportViewfrom '@/components/ScanReportView'
import ElementUIfrom 'element-ui'
import 'element-ui/lib/theme-default/index.css'
import VueResourcefrom 'vue-resource'
Vue.use(ElementUI)
Vue.use(Router)
Vue.use(VueResource)
export default new Router({
  routes: [
    {
      path: '/',
      name: 'LoginView',
      component: LoginView
    }
    , {
      path: '/MainView',
      name: 'MainView',
      component: MainView,
      children: [{
        name: 'ScanSettingView',
        path: '/ScanSettingView',
        component: ScanSettingView

      }, {
        name: 'ScanReportView',
        path: '/ScanReportView',
        component: ScanReportView
      }]
    }
  ]
})

（3）登陆页面

template>
  p class="logincontainer" align="center">
    p class="form-signin" >
      img  alt="云平台扫描系统">
    p>
    p class="form-signin--form" align="center">
      el-tabs>
        el-form label-position="center" @submit.native.prevent="doLogin" auto-complete="on" label-width="80px">
          el-form-item label="用户" :required ='true'>
            el-input v-model="params.username" auto-complete="on">el-input>
          el-form-item>
          el-form-item label="密码" :required ='true'>
            el-input type="password" v-model="params.password" auto-complete="on">el-input>
          el-form-item>
          el-form-item>
            el-button type="primary" native-type="submit" style="width:180px;text-align:center;">登录el-button>
            p v-if="fail" class="alert alert-danger">
              {{ msg }}
            p>
          el-form-item>
        el-form>
      el-tabs>
      p class="sl-login_copyright">
        GSGSoft Research br/>© 2017 GSGSoft Tech.
      p>
    p>
  p>
template>
script>
export default {
  name: 'LoginView'
  , data: function () {
    return {
      fail: true
      , msg: ''
      , params: {
        username: ''
        , password: ''
      }
    }
  }
  , methods: {
    doLogin () {  //这个地方的处理就忽略了，其实就是请求查询数据库是否匹配提交的账号和密码，如果匹配然后跳转
      this.$router.replace({
        path: '/MainView'
      })
    }
  }
  , created () {
  }
}
script>

（4）扫描配置

template>
  p>
    p>
    p class=panel-back>
      h4>扫描设置h4>
      el-form :model="params" label-width="68px" label-position="left" @submit.native.prevent="submit">
        el-row :gutter="50">
          el-col :span="4">
            el-form-item label="任务名称">
              el-input size="small" type="text" v-model="params.task_id">el-input>
            el-form-item>
          el-col>
          el-col :span="6">
            el-form-item label="开始IP">
              el-input size="small" type="text" v-model="params.ipconf_startip">el-input>


            el-form-item>
          el-col>
          el-col :span="6">
            el-form-item label="结束IP">
              el-input size="small" type="text" v-model="params.ipconf_endip">el-input>
            el-form-item>
          el-col>
          el-col :span="4">
            el-form-item label="调度周期">
              el-input size="small" type="text" v-model="params.looptime">el-input>
            el-form-item>
          el-col>
          el-col :span="4">
            el-button size="small" type="primary" @click.native="submit()">添加el-button>
          el-col>
        el-row>
      el-form>
    p>
    p>
    p>
      p class=panel-back>
        h4>扫描任务h4>
        el-table
          :data="logs"
          style="width: 100%">
          el-table-column
            property="task_id"
            label="任务名称"
            width="160">
          el-table-column>
          el-table-column
            label="扫描状态"
            inline-template
          >
            el-progress  :stroke-width="12" v-bind:percentage='scanstate' >el-progress>
          el-table-column>
          el-table-column
            property="ipconf_startip"
            label="扫描开始IP"
            width="130">
          el-table-column>
          el-table-column
            property="ipconf_endip"
            label="扫描结束IP"
            width="130">
          el-table-column>
          el-table-column
            inline-template
            property="cops"
            label="操作">
            p>
              el-button size="small" @click.native="StartTask(row)">启动el-button>
              el-button size="small" @click.native="DeleteTask(row)">删除el-button>
              el-button size="small" @click.native="VulTask(row)">漏洞el-button>


            p>
          el-table-column>
        el-table>
    p>
      el-pagination
        v-if="!loading"
        :current-page="offset / 20 + 1"
        @current-change="paginationChange"
        layout="prev, pager, next"
        :page-size="20"
        :total="total">
      el-pagination>
    p>
  p>
  p>
  p>
template>
export default {
  data: function () {
    return {
      logs: []
      , params: {
        task_id: ''
        , ipconf_startip: ''
        , ipconf_endip: ''
        , looptime: ''
        , scanstate: ''
      }
      , offset: 0
      , total: 0
      , count: 0
      , loading: false
      , dialogVisible: false
    }
  }
  , methods: {
    submit () {
      this.$http.post('/api/config/newtask/', this.params).then((response) => {
        if (response.data.err === 'exists') {
          this.$message.error('任务名称已经存在，请更改');
        } else {
          this.$message({
           type: 'success',
           message: '扫描任务添加完成'
          });
        }
      this.$http.get('/api/config/tasklist/id?offset=0&count=20').then((response) => {
        this.logs = response.body.items
        this.total = response.body.total
        this.loading = false
      }, () => {
        this.$message({
          type: 'warning',
          message: '网络错误'
        });
      })
      }, (response) => {
      })
    }
    , paginationChange (page) {
      console.log('page' + page)
      this.paginationRequest((page - 1) * 20, 20)
    }
    , paginationRequest (offset, count) {
      this.$http.get(`/api/config/tasklist/id?offset=${offset}&count=${count}`).then((response) => {
        this.logs = response.body.items
        this.total = response.body.total
      }, () => {
        this.$message({
          type: 'warning',
          message: '网络错误'
        });
      })
    }
  , StartTask (row) {
      this.$confirm('你确定要启动该任务?', '提示', {
        confirmButtonText: '确定',
        cancelButtonText: '取消',
        type: 'warning'
      }).then(() => {
        this.$http.post('/api/action/doscan/', row).then((response) => {
        this.$http.get('/api/config/tasklist/id?offset=0&count=20').then((response) => {
            this.logs = response.body.items


            this.total = response.body.total
            this.loading = false
        }, () => {
            this.$message({
                type: 'warning',
                message: '网络错误'
              });
        })
        if (response.data.err === 'scanning') {
          this.$message.error('任务扫描中...，请稍后');
        } else {
          row.task_id
          this.$message({
           type: 'success',
           message: '扫描任务已经启动'
          });
        }
        }, (response) => {
        })
      }).catch(() => {
        this.$message({
          type: 'info',
          message: '已取消删除任务'
        });
      });
  }
  , DeleteTask (row) {
      console.log(row.task_id)
      this.$confirm('此操作将永久删除该任务, 是否继续?', '提示', {
        confirmButtonText: '确定',
        cancelButtonText: '取消',
        type: 'warning'
      }).then(() => {
        this.$http.post('/api/config/deltask/', row).then((response) => {
        this.$http.get('/api/config/tasklist/id?offset=0&count=20').then((response) => {
            this.logs = response.body.items
            this.total = response.body.total
            this.loading = false
        }, () => {
            this.$message({
                type: 'warning',
                message: '网络错误'
              });
        })
        }, (response) => {
        })
        this.$message({
          type: 'success',
          message: '任务删除成功!'
        });
      }).catch(() => {
        this.$message({
          type: 'info',
          message: '已取消删除任务'
        });
      });
  }
  , VulTask (row) {
      this.$confirm('此操作将执行所有已知漏洞漏扫任务, 是否继续?', '提示', {
        confirmButtonText: '确定',
        cancelButtonText: '取消',
        type: 'warning'
      }).then(() => {
        this.$http.post('/api/config/vultask/', row).then((response) => {
        }, (response) => {
        })
        this.$message({
          type: 'success',
          message: '任务删除成功!'


        });
      }).catch(() => {
        this.$message({
          type: 'info',
          message: '已取消漏洞扫描任务'
        });
      });
  }
  , request(){
      this.loading = true
      this.$http.get('/api/config/tasklist/id?offset=0&count=20').then((response) => {
        this.logs = response.body.items
        this.total = response.body.total
        this.loading = false
      }, () => {
        this.$message({
            type: 'warning',
            message: '网络错误'
          });
      })
  }
  , requestPoll(){
      console.log('Poll')
      this.request()
      this.timer = window.setTimeout(this.requestPoll, 5000)
  }
  }
  , created () {
    this.loading = true
    this.dialogVisible = false
    if (this.timer) {
        window.clearTimeout(this.timer)
      }
    this.$http.get('/api/config/tasklist/id?offset=0&count=20').then((response) => {
      this.logs = response.body.items
      this.total = response.body.total
      this.loading = false
    }, () => {
      this.$message({
          type: 'warning',
          message: '网络错误'
        });
    })
  }
}
（5）扫描报表
script>
import VueHighcharts from 'vue2-highcharts'
export default{
    components: {
        VueHighcharts
    },
    data(){
      return{
        logs: [],
        options: {
          chart: {
            type: 'spline'
          },
          title: {
            text: '高危端口暴露趋势'
          },
          subtitle: {
            text: 'Source: gsgsoft.com'
          },
          xAxis: {
            categories: []
          },
          yAxis: {
            title: {
              text: '数量'
            },
          },
          tooltip: {
            crosshairs: true,
            shared: true
          },
          credits: {
            enabled: false
          },
          plotOptions: {
            spline: {
              marker: {
                radius: 4,
                lineColor: '#666666',


                lineWidth: 1
              }
            }
          },
          series: []
        },
        pieOptions: {
             chart: {
              type: 'pie',
                options3d: {
                    enabled: true,
                    alpha: 45
                }
              },
              title: {
                  text: '服务类型分布'
              },
              subtitle: {
                  text: 'Source: gsgsoft.com'
              },
              plotOptions: {
                pie: {
                    innerSize: 200,
                    depth: 45
                }
              },
              credits: {
                enabled: false
              },
              series: [{ // 这部分动态数据还没有实现，不过原理和端口数量都一样
                  name: '服务类型分布',
                  data: [
                      ['http', 117],
                      ['ssh', 34],
                      ['msrpc', 18],
                      ['mysql', 10],
                      ['ftp', 9],
                      ['ms-wbt-server', 7],
                      ['rfe', 5],
                      ['commplex-link', 5],
                      ['svnserve', 3]
                  ]
              }]
              }
        }
    },
    methods: {
    }
    , created (){
        this.$http.get('/api/config/Query/').then((response) => {


          let lineCharts = this.$refs.lineCharts;
          this.logs = response.body;
          this.options.xAxis.categories = this.logs[6]['data'];
          lineCharts.addSeries(this.logs[0],this.options.xAxis.categories);
          lineCharts.addSeries(this.logs[1]);
          lineCharts.addSeries(this.logs[2]);
          lineCharts.addSeries(this.logs[3]);
          lineCharts.addSeries(this.logs[4]);
          lineCharts.addSeries(this.logs[5]);
        }, (response) => {
        })
    }
}
script>

0×04、后端实现
（1）数据库设计
配置保存表：主要是保存用户输入的扫描配置记录，包括任务名称、扫描开始IP、扫描结束IP、扫描周期、扫描进度。

CREATE TABLE scanconf
(
   id INTEGER DEFAULT nextval('table_name_id_seq'::regclass) PRIMARY KEYNOT NULL,
   ipconf_startip TEXT,
   ipconf_endip TEXT,
   looptime INTEGER,
   task_id TEXT,
   scanstate TEXT
);
CREATE UNIQUE INDEX table_name_id_uindex ONscanconf (id);
数据保存表：包含任务名称、创建时间、IP地址、端口、服务、产品、产品版本、产品额外信息、操作系统、对应用户名称、对应的用户部门。
CREATE TABLE scanresult_20170609
(
   task_id TEXT,
   ctime TEXT,
   address TEXT,
   port TEXT,
   service TEXT,
   product TEXT,
   product_version TEXT,
    product_extrainfo TEXT,
   os TEXT,
   eip TEXT,
   business TEXT
);

漏洞类型描述：主要是把漏洞信息记录到数据库中。例如：
st2_eval     Struts2
远程代码执行  
可直接执行任意代码，
进而直接导致服务器被入侵控制。   
紧急
代码执行 

wolf@YSRC      
https://www.shack2.org/Article/1374154000.html
tag:tomcat
CREATE TABLE vultype
(
   id INTEGER DEFAULT nextval('vultype_id_seq'::regclass) PRIMARY KEY NOTNULL,
   add_time TEXT,
   filename TEXT,
   name TEXT,
   info TEXT,
   level TEXT,
   type TEXT,
   author TEXT,
   url TEXT,
   keyword TEXT
);
CREATE UNIQUE INDEX vultype_id_uindex ONvultype (id);

扫描结果保存表：例如：
x.x.21.116
heartbleed_poc
存在心脏出血漏洞   
2017-05-27 11:26:56

CREATE TABLE vulresult
(
   id INTEGER DEFAULT nextval('vulresult_id_seq'::regclass) PRIMARY KEY NOTNULL,
   address TEXT,
   vulname TEXT,
   result TEXT,
   ctime TEXT
);
CREATE UNIQUE INDEX vulresult_id_uindex ONvulresult (id);
（2）代码实现-端口扫描代码
OpenAPI部分：
Urls.py
urlpatterns = [
    url(r'^api/config/newtask/$', ConfigAPI.as_view()),
    url(r'^api/action/doscan/$', ScanAPI.as_view()),
    url(r'^api/config/tasklist/id$', ScanconfListAPI.as_view()),
    url(r'^api/config/deltask/$', ConfigDelAPI.as_view()),
]

创建扫描任务

class ConfigAPI(APIView):
    def post(self, request, format=None):
        m_task_id = request.POST.get('task_id')
        db_tasks = scanconf.objects.filter(task_id=m_task_id)
        if db_tasks.exists():
            return error(err="exists", msg="task name exists")
        else:
            ser = ScanconfSerializer(data=request.data)
            print request.data
            if ser.is_valid():
                ser.save()
                return Response(ser.data)
            return Response(ser.errors)

删除扫描任务

class ConfigDelAPI(APIView):
    def post(self, request, format=None):
        data = request.data
        m_task_id = data['task_id']
        db_tasks = scanconf.objects.filter(task_id=m_task_id).delete()
        return success("success")
启动扫描任务
class ScanAPI(APIView):
    def post(self, request, format =None):
        data = request.data
        m_task_id = data['task_id']
        print m_task_id
        db_tasks = scanconf.objects.filter(task_id=m_task_id)
        if db_tasks.exists():
            try:
                threading.Thread(target=ScanExtIP.doscan, args=(m_task_id,)).start()
            except:
                print traceback.print_exc()
            return Response("success")
        return Response("doscan failure no task in db")
列举扫描任务
class ScanconfListAPI(APIView):
    def get(self, request, format=None):
        print request.GET.get("count")
        cursor = scanconf.objects.all()
        return Response(paginate_data(request, cursor, ScanconfSerializer))

扫描执行

def Scan():
    try:
        global g_queue
        global g_task_id
        tableName = "%s_%s" % ("scanresult", time.strftime("%Y%m%d"))
        num = '0.0'
        curS = connS.cursor()
        curS.execute("update scanconf SET scanstate = %s where task_id = %s", (num, g_task_id))
        connS.commit()
        cur1 = conn1.cursor()
        while not g_queue.empty():
            item = g_queue.get()
            nm = NmapProcess(item, "-sV -O --min-rate 2000 --max-rtt-timeout 100ms")
            nm.sudo_run()
            ctime = strftime("%Y-%m-%d %H:%M:%S", gmtime())
            nmap_report = NmapParser.parse(nm.stdout)
            for scanned_hosts in nmap_report.hosts:
                print scanned_hosts.address
                if len(scanned_hosts.os.osmatch()) > 0:
                    print scanned_hosts.os.osmatch()[0]
                for serv in scanned_hosts.services:
                    if serv.state == 'open':
                        if len(scanned_hosts.os.osmatch()) > 0:
                            sql = "INSERT INTO %s (task_id,ctime, address,port,service,product,product_version,product_extrainfo,os) VALUES ('%s','%s','%s','%s','%s','%s','%s','%s','%s')"
                            sqlCmd = sql%(tableName,g_task_id,ctime,scanned_hosts.address,str(serv.port),serv.service,serv.service_dict.get("product", ""),serv.service_dict.get("version", ""),serv.service_dict.get("extrainfo", ""),scanned_hosts.os.osmatch()[0])                   

 else:
                            sql = "INSERT INTO %s (task_id,ctime, address,port,service,product,product_version,product_extrainfo,os) VALUES ('%s','%s','%s','%s','%s','%s','%s','%s','%s')"
                            sqlCmd = sql%(tableName,g_task_id,ctime,scanned_hosts.address,str(serv.port),serv.service,serv.service_dict.get("product", ""),serv.service_dict.get("version", ""),serv.service_dict.get("extrainfo", ""),'NULL')
                        cur1.execute(sqlCmd)
                        conn1.commit()
            print "size = ", g_queue.qsize()
            g_size = g_queue.qsize()
            num = 100 - round(float(g_size) / float(g_totalsize) * 100, 0)
            print num, g_size, g_totalsize
            curS = connS.cursor()
            curS.execute("update scanconf SET scanstate = %s where task_id = %s", (num, g_task_id))
            connS.commit()
        return "ok"
    except Exception,e:
        print e
        return e
def CreateTable():
    curC = connC.cursor()
    sqlCreate = "create table if not exists %s ( \
                 task_id TEXT,\
                 ctime TEXT,\
                 address TEXT,\
                 port TEXT,\
                 service TEXT,\
                 product TEXT ,\
                 product_version TEXT,\
                 product_extrainfo TEXT,\
                 os TEXT,\
                 eip TEXT,\
                 business TEXT\
                 )"
    tableName = "%s_%s"%("scanresult", time.strftime("%Y%m%d"))
    sqlCmd = sqlCreate%tableName
    curC.execute(sqlCmd)
def doscan(task_id):
    global g_queue
    global g_task_id
    listThread = []
    cur = conn.cursor()
    querySQL = "select id,ipconf_startip,ipconf_endip,looptime from scanconf WHERE task_id = '{}'".format(task_id)
    cur.execute(querySQL)
    rows = cur.fetchall()
    for row in rows:
        iplist(row[1],row[2])
    g_task_id =task_id
    conn.commit()
    conn.close()
    CreateTable()
    for i in xrange(g_threadNum):
        thread = ScanThread(Scan)
        thread.start()
        listThread.append(thread)
    for thread in listThread:
        thread.join()

        print thread
    return "ok"
漏洞扫描部分：主要是集成巡风漏洞系统的VulScan.py 只是把mongodb数据库换成了postgresql，就不在这里累述。高危端口变化趋势：这部分说一下逻辑，因为代码实在太长了。就是从数据库中查询最近7天的高危端口数据。组合成json的形式返回给全端。       
b = json.dumps([{"name": "mysql", "data": list1},                       
{"name": "ms-ql-s", "data": list2},                       
{"name": "ibm-db2", "data": list3},                       
{"name": "oracle", "data": list4},                       
{"name": "redis", "data": list5},                       
{"name": "mongodb", "data": list6},                       
{"name": "day", "data": list7}])       
return HttpResponse(b)
0×05、部署云主机的选择
由于使用了多线程，对CPU内存要求都比较高，经过综合对比选择金山云大米主机。2 core，4G内存，100G SSD，1元用7天，买4个月赠送3个月。

大致的部署架构：

nginx.conf
server
{  
  listen      80;  
  server_name x.x.10x.1x2;   
  charset     utf-8;  
  client_max_body_size 75M;   
  location /api {      
    proxy_pass https://127.0.0.1:9001;   
  }  
  location / {       
    root /var/CloudPScan/dist;       
    try_files $uri $uri/ /index.html;   
  }
}
uwsgi.ini （uwsgi使用ini文件启动）
[uwsgi]
http=127.0.0.1:9001
chdir=/var/CloudPScan/
master=True
pidfile=CloudPScan-master.pid
vacuum=True
max-requests=5000
daemonize=CloudPScan.log
env = LANG=en_US.UTF-8
wsgi-file = CloudPScan/wsgi.py
服务器安装：
yum install epel-releaseyum
install python-pip python-devel nginx gcc
pip install --upgrade pip
pip install uwsgi
systemctl start uwsgi
cd /etc/nginx/sites-enabled
vim CloudPScan.conf
sudo nginx -t
systemctl start nginxsystemctl enable nginxyum
install postgresql-serverpostgresql-devel postgresql-contrib
postgresql-setup initdbsystemctl start postgresql
pip install -U django==1.10.0
pip install djangorestframework==3.3.2
pip install requests
pip install python-libnmap
yum install nmap
systemctl stop firewalld.service

0×06、总结
整个coding的过程比较匆忙，代码中也有很多地方不完善，还请各位大牛口下留情。本文从详细设计、交互设计、前端代码实现、后端代码实现、部署等环节，完整的描述了一个产品的产生过程。最后一点想说，产品经理和程序员需要相互体谅，都不容易。