kubernetes垃圾回收器GarbageCollector Controller源码分析(二)
kubernetes版本:1.13.2
接上一节:kubernetes垃圾回收器garbagecollector controller源码分析(一)
主要步骤
garbagecollector controller源码主要分为以下几部分:
-
monitors
作为生产者将变化的资源放入graphchanges
队列;同时restmapper
定期检测集群内资源类型,刷新monitors
-
runprocessgraphchanges
从graphchanges
队列中取出变化的item
,根据情况放入attempttodelete
队列; -
runprocessgraphchanges
从graphchanges
队列中取出变化的item
,根据情况放入attempttoorphan
队列; -
runattempttodeleteworker
从attempttodelete
队列取出,尝试删除垃圾资源; -
runattempttoorphanworker
从attempttodelete
队列取出,处理该孤立的资源;
代码较复杂,便于讲的更清楚,调整了下讲解顺序。上一节分析了第1部分,本节分析第2、3部分。
runprocessgraphchanges处理主流程
来到源码k8s.io\kubernetes\pkg\controller\garbagecollector\graph_builder.go中,runprocessgraphchanges中一直死循环处理变化的资源对象:
func (gb *graphbuilder) runprocessgraphchanges() { for gb.processgraphchanges() { } }
一个协程一直循环从graphchanges队列中获取变化的资源对象,更新图形,填充dirty_queue。(graphchanges队列里数据来源于各个资源的monitors监听资源变化回调addfunc、updatefunc、deletefunc)
// dequeueing an event from graphchanges, updating graph, populating dirty_queue. //从graphchanges中获取事件,更新图形,填充dirty_queue。(graphchanges队列里数据来源于各个资源的monitors监听资源变化回调addfunc、updatefunc、deletefunc) func (gb *graphbuilder) processgraphchanges() bool { item, quit := gb.graphchanges.get() if quit { return false } defer gb.graphchanges.done(item) event, ok := item.(*event) if !ok { utilruntime.handleerror(fmt.errorf("expect a *event, got %v", item)) return true } obj := event.obj //获取该变化资源obj的accessor accessor, err := meta.accessor(obj) if err != nil { utilruntime.handleerror(fmt.errorf("cannot access obj: %v", err)) return true } klog.v(5).infof("graphbuilder process object: %s/%s, namespace %s, name %s, uid %s, event type %v", event.gvk.groupversion().string(), event.gvk.kind, accessor.getnamespace(), accessor.getname(), string(accessor.getuid()), event.eventtype) // check if the node already exists // 检查节点是否已存在 //根据该变化资源obj的uid //uidtonode维护着资源对象依赖关系图表结构 existingnode, found := gb.uidtonode.read(accessor.getuid()) if found { // this marks the node as having been observed via an informer event // 1. this depends on graphchanges only containing add/update events from the actual informer // 2. this allows things tracking virtual nodes' existence to stop polling and rely on informer events //这标志着节点已经通过informer事件 // 1.进行了观察。这取决于仅包含来自实际informer的添加/更新事件的graphchange // 2.这允许跟踪虚拟节点的存在以停止轮询和依赖informer事件 existingnode.markobserved() } switch { //gc第一次运行时,uidtonode尚且没有初始化资源对象依赖关系图表结构,所以found为false,会新增节点 case (event.eventtype == addevent || event.eventtype == updateevent) && !found: newnode := &node{ identity: objectreference{ ownerreference: metav1.ownerreference{ apiversion: event.gvk.groupversion().string(), kind: event.gvk.kind, uid: accessor.getuid(), name: accessor.getname(), }, namespace: accessor.getnamespace(), }, dependents: make(map[*node]struct{}), owners: accessor.getownerreferences(), deletingdependents: beingdeleted(accessor) && hasdeletedependentsfinalizer(accessor), beingdeleted: beingdeleted(accessor), } gb.insertnode(newnode) // the underlying delta_fifo may combine a creation and a deletion into // one event, so we need to further process the event. //底层delta_fifo可以将创建和删除组合成一个事件,因此我们需要进一步处理事件。 gb.processtransitions(event.oldobj, accessor, newnode) //uidtonode已经初始化资源对象依赖关系图表结构,所以found为true case (event.eventtype == addevent || event.eventtype == updateevent) && found: // handle changes in ownerreferences //处理ownerreferences中的更改 added, removed, changed := referencesdiffs(existingnode.owners, accessor.getownerreferences()) if len(added) != 0 || len(removed) != 0 || len(changed) != 0 { // check if the changed dependency graph unblock owners that are // waiting for the deletion of their dependents. //检查更改的依赖关系图是否取消阻止等待删除其依赖项的所有者。 gb.addunblockedownerstodeletequeue(removed, changed) // update the node itself //更新node的owner existingnode.owners = accessor.getownerreferences() // add the node to its new owners' dependent lists. //给新owner添加依赖资源列表 gb.adddependenttoowners(existingnode, added) // remove the node from the dependent list of node that are no longer in // the node's owners list. //从不再属于该资源owner列表中删除该节点。 gb.removedependentfromowners(existingnode, removed) } // 该对象正在被删除中 if beingdeleted(accessor) { existingnode.markbeingdeleted() } gb.processtransitions(event.oldobj, accessor, existingnode) //处理资源对象被删除的场景,涉及垃圾。比如,owner被删除,其依赖资源(从资源)也需要被删除掉,除非设置了orphan case event.eventtype == deleteevent: if !found { klog.v(5).infof("%v doesn't exist in the graph, this shouldn't happen", accessor.getuid()) return true } // 从图标中移除item资源,同时遍历owners,移除owner下的item资源 gb.removenode(existingnode) existingnode.dependentslock.rlock() defer existingnode.dependentslock.runlock() //如果该资源的从资源数大于0,则将该资源被删除信息加入absentownercache缓存 if len(existingnode.dependents) > 0 { gb.absentownercache.add(accessor.getuid()) } //遍历该资源的从资源加到删除队列里 for dep := range existingnode.dependents { gb.attempttodelete.add(dep) } for _, owner := range existingnode.owners { ownernode, found := gb.uidtonode.read(owner.uid) //owner没发现 或者 owner的从资源不是正在被删除(只有该资源对象的终结器为foregrounddeletion finalizer时deletingdependents被设为true,因为后台删除owner直接被删除,不会被其从资源block,故这里都不需要去尝试删除owner了) if !found || !ownernode.isdeletingdependents() { continue } // 这是让attemptodeleteitem检查是否删除了owner的依赖项,如果是,则删除所有者。 gb.attempttodelete.add(ownernode) } } return true }
该方法功能主要将对象、owner、从资源加入到attempttodelete或attempttoorphan。
1、 出队
从graphchanges队列取出资源对象,从graphbuilder.uidtonode中读取该资源节点(uidtonode维护着资源对象依赖关系图表结构),found为true时表示图表存在该资源节点;
2、switch的第一个case
如果该资源是新增或者更新触发,且该资源对象不存在于图表中,gb.uidtonode.write(n)会将其写入图标;
gb.insertnode(newnode)中的gb.adddependenttoowners(n, n.owners)方法则会遍历该资源的owner,如果其owner不存在于图标中,则新增owner的虚拟节点到图标中,并将该资源和owner产生关联。如果owner不存在时,则尝试将owner加入到attempttodelete队列中去;
// adddependenttoowners将n添加到所有者的从属列表中。如果所有者不存在于gb.uidtonode中,则将创建"虚拟"节点以表示 // 所有者。 "虚拟"节点将入队到attempttodelete,因此 // attempttodeleteitem()将根据api服务器验证所有者是否存在。 func (gb *graphbuilder) adddependenttoowners(n *node, owners []metav1.ownerreference) { //遍历owner for _, owner := range owners { //获取owner node如果不存在于图中,则加虚拟owner节点 ownernode, ok := gb.uidtonode.read(owner.uid) if !ok { // create a "virtual" node in the graph for the owner if it doesn't // exist in the graph yet. //如果图形中尚未存在,则在图表中为所有者创建“虚拟”节点。 ownernode = &node{ identity: objectreference{ ownerreference: owner, namespace: n.identity.namespace, }, dependents: make(map[*node]struct{}), virtual: true, } klog.v(5).infof("add virtual node.identity: %s\n\n", ownernode.identity) gb.uidtonode.write(ownernode) } //给owner加该资源作为依赖 ownernode.adddependent(n) //owner不存在于图中时,才往删除队列添加 if !ok { // enqueue the virtual node into attempttodelete. // the garbage processor will enqueue a virtual delete // event to delete it from the graph if api server confirms this // owner doesn't exist. //将虚拟节点排入attempttodelete。 // 如果api服务器确认owner不存在,垃圾处理器将排队虚拟删除事件以将其从图中删除。 gb.attempttodelete.add(ownernode) } } }
gb.processtransitions方法:
新item正在被删,旧item没开始被删除,且终结器为orphan finalizer加入到attempttoorphan队列;
新item正在被删,旧item没开始被删除,且终结器为foregrounddeletion finalizer,则加入到attempttodelete队列。
func (gb *graphbuilder) processtransitions(oldobj interface{}, newaccessor metav1.object, n *node) { //新的正在被删,旧的没开始被删除,且终结器为orphan finalizer if startswaitingfordependentsorphaned(oldobj, newaccessor) { klog.v(5).infof("add %s to the attempttoorphan", n.identity) //加入到orphan队列 gb.attempttoorphan.add(n) return } //新的正在被删,旧的没开始被删除,且终结器为foregrounddeletion finalizer if startswaitingfordependentsdeleted(oldobj, newaccessor) { klog.v(2).infof("add %s to the attempttodelete, because it's waiting for its dependents to be deleted", n.identity) // if the n is added as a "virtual" node, its deletingdependents field is not properly set, so always set it here. n.markdeletingdependents() for dep := range n.dependents { gb.attempttodelete.add(dep) } gb.attempttodelete.add(n) } }
3、switch的第二个case
如果该资源是新增或者更新触发,且该资源对象存在于图表中。对比ownereferences是否有变更,referencesdiffs方法里会根据uid对比,added表示新owner里有,旧owner里没有的, removed表示旧owner里有,新owner里没有的, changed表示相同uid的owner不deepequal的。
func referencesdiffs(old []metav1.ownerreference, new []metav1.ownerreference) (added []metav1.ownerreference, removed []metav1.ownerreference, changed []ownerrefpair) { //key为uid, value为ownerreference olduidtoref := make(map[string]metav1.ownerreference) for _, value := range old { olduidtoref[string(value.uid)] = value } olduidset := sets.stringkeyset(olduidtoref) //key为uid, value为ownerreference newuidtoref := make(map[string]metav1.ownerreference) for _, value := range new { newuidtoref[string(value.uid)] = value } newuidset := sets.stringkeyset(newuidtoref) //新的里有,旧的里没有的为新增(根据uid判断) addeduid := newuidset.difference(olduidset) //旧的里有,新的里没有的为删除(根据uid判断) removeduid := olduidset.difference(newuidset) //取交集, 旧的和新的里都有的owner(根据uid判断) intersection := olduidset.intersection(newuidset) for uid := range addeduid { added = append(added, newuidtoref[uid]) } for uid := range removeduid { removed = append(removed, olduidtoref[uid]) } //根据uid判断,两个uid相等的ownerreference是否deepequal,不等则加到changed for uid := range intersection { if !reflect.deepequal(olduidtoref[uid], newuidtoref[uid]) { changed = append(changed, ownerrefpair{oldref: olduidtoref[uid], newref: newuidtoref[uid]}) } } return added, removed, changed }
整体来说,owner发生变化,addunblockedownerstodeletequeue方法会判断:如果阻塞ownerreference指向某个对象被删除,或者设置为blockownerdeletion=false
,则将该对象添加到attempttodelete队列;
// if an blocking ownerreference points to an object gets removed, or gets set to // "blockownerdeletion=false", add the object to the attempttodelete queue. //如果阻塞ownerreference指向某个对象被删除,或者设置为 // "blockownerdeletion = false",则将该对象添加到attempttodelete队列。 func (gb *graphbuilder) addunblockedownerstodeletequeue(removed []metav1.ownerreference, changed []ownerrefpair) { for _, ref := range removed { //被移除的ownersreferences,blockownerdeletion为true if ref.blockownerdeletion != nil && *ref.blockownerdeletion { //依赖图表中发现,则加入删除队列 node, found := gb.uidtonode.read(ref.uid) if !found { klog.v(5).infof("cannot find %s in uidtonode", ref.uid) continue } //加入尝试删除队列删除这个owner gb.attempttodelete.add(node) } } // owners存在且发生变化,旧的blockownerdeletion为true, 新的blockownerdeletion为空或者blockownerdeletion为false则删除owner(父节点) for _, c := range changed { wasblocked := c.oldref.blockownerdeletion != nil && *c.oldref.blockownerdeletion isunblocked := c.newref.blockownerdeletion == nil || (c.newref.blockownerdeletion != nil && !*c.newref.blockownerdeletion) if wasblocked && isunblocked { node, found := gb.uidtonode.read(c.newref.uid) if !found { klog.v(5).infof("cannot find %s in uidtonode", c.newref.uid) continue } gb.attempttodelete.add(node) } } }
更新node的owner;
在依赖图表中给新owner添加该node;
在依赖图表中,被删除的owner列表下删除该节点。
gb.processtransitions方法:
新item正在被删,旧item没开始被删除,且终结器为orphan finalizer加入到attempttoorphan队列;
新item正在被删,旧item没开始被删除,且终结器为foregrounddeletion finalizer,则加入到attempttodelete队列。
4、switch的第三个case
如果该资源是删除时触发,从图表中移除item资源,同时遍历owners,移除owner下的item资源;
如果该资源的从资源数大于0,则将该资源被删除信息(uid)加入absentownercache缓存,这样处理该资源的从资源时,就知道owner不存在了。
遍历该资源的从资源加到删除队列里;
如果从图表中发现 owner或者 owner的从资源正在被删除,则尝试将owner加入到attempttodelete队列中,去尝试删除owner。
整理流程
- 当controllermanager重启时,会全量listwatch一遍所有对象,gc collector维护的uidtonode图表里各个资源对象node是不存在的,此时会走第一个switch case,构建完整关系图表,如果owner不存在则先构建虚拟owner节点,同时加入attempttodelete队列,尝试去删除这个owner,其实即使加入到attempttodelete队列,也不一定会被删除,还会进行一系列判断,这个下一节再分析;将正在删除的资源,同时finalizer为orphan的加入到attempttoorphan队列;为foreground的资源以及其从资源加入到attempttodelete队列,并将deletingdependents设置为true;
- 添加或者更新事件时,且图表中存在item资源对象时,会走第二个switch case,对item的owner变化进行判断,并维护更新图表;同理将正在删除的资源,同时finalizer为orphan的加入到attempttoorphan队列;finalizer为foreground的资源以及其从资源加入到attempttodelete队列,并将deletingdependents设置为true;
- 如果是删除事件,则会更新图表,并处理和其相关的从资源和其owner加入到attempttodelete队列。
参考:
k8s官方文档garbage-collection英文版:
依赖图标生成库gonum api文档:
graphviz下载:
https://graphviz.gitlab.io/_pages/download/download_windows.html
本公众号免费提供csdn下载服务,海量it学习资源,如果你准备入it坑,励志成为优秀的程序猿,那么这些资源很适合你,包括但不限于java、go、python、springcloud、elk、嵌入式 、大数据、面试资料、前端 等资源。同时我们组建了一个技术交流群,里面有很多大佬,会不定时分享技术文章,如果你想来一起学习提高,可以公众号后台回复【2】,免费邀请加技术交流群互相学习提高,会不定期分享编程it相关资源。
扫码关注,精彩内容第一时间推给你