字节跳动数据血缘图谱升级方案设计与实现

更多技术交流、求职机会，欢迎关注 字节跳动数据平台微信公众号，回复【1】进入官方交流群

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:b0f9ddf6-fd9c-4ffe-8c5f-4822a1a14adb

血缘图谱由 xGraph 与数据地图平台团队合作研发。xGraph 从 Dataleap 业务中孵化，从底至上完全自研，提供设计成熟的内置节点、连线、分组样式，精心打磨图分析产品中常用布局和交互，帮助用户快速搭建关系图产品。血缘图谱解决方案已沉淀到 xGraph 为更多团队复用。

数据血缘图谱介绍

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:6a6c4153-5e33-4e5f-91eb-f69146a78ee8

数据地图平台在 2021 年接入了全链路核心元数据，包括但不限于：Hive、Clickhouse、Kafka、BI 报表、BI 数据集、画像、埋点、MySQL、Abase。这些数据全部要通过数据血缘连接起来，进而可以进行影响分析、内部审计、SLA保障、归因分析、理解和查找数据、自动化推荐等操作。

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:431eeb7f-f218-4da1-ab26-cb16b45b1739

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:ceaf7a47-ad49-4935-b344-c632c760414b

需求发现

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:5138cd24-79f4-4953-a045-d58664a3c359

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:efda5b63-94f6-4750-bdb3-1da6ea7ce9ff

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:70a5af9f-b462-4ecd-ab39-c986bac94087

问题分析

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:ed18440d-d838-42eb-8e7f-9868cbbae8fa

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:95863269-3de0-401f-b23e-4062f5e34afb

节点较少，比较清晰

大量节点，查看困难

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:22dbea2d-ced8-45a9-945b-c3ee980a33f8

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:38ff7a42-d752-4e57-a0ea-cb95cff81fa3

方案设计

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:f31f086a-a907-4516-b9d8-506d5f713648

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:cf316e68-50cf-4e44-aa6a-ac4b0668ee5f

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:ecaff26c-2cc7-4420-ae37-974770cf6a38

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:c6fbae2d-5a1c-4594-8ddd-4e5cc6c7387c

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:143e0597-a42b-4a6b-a908-db23f2fd4ebb

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:3ae60d6d-2601-401e-a69d-9b3c91551f68

当用户想去找数，理解数或做归因分析时，不仅要了解一个表的上游依赖，更需要理解表的加工逻辑。因此我们在节点的连线上新增了任务信息。当用户 hover 到连线上后，连线会加粗高亮并弹出任务信息。我们还附上了大数据开发平台的对应任务链接，点击链接即可跳转到新页面查看任务逻辑详情。

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:bdcbe332-099e-429f-a649-7167f9d2a81e

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:cd9df267-6def-41fb-86c1-1f448edce25e

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:3e69670b-2994-4b31-be9a-d07115e1b4c7

技术实现

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:3ef1538d-916c-42d7-a836-9091c84a80b3

于是我们结合两者之长，选用了 React + Canvas 的混合模式来实现血缘图谱。Canvas 居于底部，仅负责画连线。React 在上层负责渲染节点响应 hover 等交互。DOM 层叠关系如下：

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:4913a232-40df-44f8-a713-0075527b57a8

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:49eeade4-8da9-4ca2-b9c5-455bd851b9e0

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:c955c234-936d-4d7c-85cd-00b885769982

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:03e874d7-cad5-4152-b089-63aff7695806

* 节点布局：根据层级和分组情况布局节点，相对应的每个节点有 { x, y, width, height 属性以确定每个节点的定位。
[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:1705ba18-d176-4556-8aef-8f7a0b359f65

* 渲染节点：根据节点的位置和分组情况用 React 渲染出每一列节点 DOM。
[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:99e78c98-5dd9-4b94-a85f-a2e3f5d7ccda

用这种混合模式的一个挑战就是 Canvas 和 DOM 的刷新率和同步率。在血缘图谱中滚动横向滚动条和每一列的纵向滚动条时 Canvas 要进行及时的刷新以保证连线和节点的相对位置一定。

• 当图谱横向滚动时，每条连线的斜率不变，只是端点左右平移了。我们可以通过更新绘图矩阵来加速这种情况下的更新，不需要去重计算每条连线的位置。具体做法是监听容器的滚动事件，根据容器的 scrollLeft 属性来更新绘图矩阵后重绘。
  [TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:75d12456-6049-4826-8044-b9839a6ccf98
  [En]

  [TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:05a9a118-b37e-43a0-8a02-63ed01f0b27f

另一个挑战是 DOM 节点在大数据量下的性能问题。通常情况下我们认为 Canvas 在大数据量渲染有更好的性能，而万级的 DOM 节点就会让用户在使用中感受到卡顿了。这时候我们想到了按需渲染。 用户在图谱可视区域中一屏能看到的节点数量是有限的，高度为 1120 的容器中，一列仅存在至多 30 个节点。如果仅渲染可见的节点，则能保证使用 过程的流畅。具体做法是在节点布局时增加以下步骤：

• 根据视口的位置（主要是图容器的横向滚动距离 scrollLeft ）和每一列的滚动距离（主要是每一列容器的纵向滚动距离 scrollTop ）计算目前的可视范围。
  [TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:2bf5a007-1bf6-4a71-89db-8fdd4f3fa23b
  [En]

  [TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:daf38a77-db8f-4580-aefd-9852c3925a22

  [TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:d6bcf073-4224-4001-917d-7ae90f1f864d

  [En]

  [TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:e019a58f-8703-407f-a52a-f20b71bb4421

在 React 渲染时更新每列容器的长度，将节点根据坐标绝对定位到正确的位 置上。看起来就跟全量渲染的效果一致，渲染效率大幅提升。

然而问题并不止于此。在进行大数据量的纵向滚动时，会发现帧率很低，交 互还是不流畅。分析得知是由于列表滚动时会在短时间内进行大量线条重计算和渲染。于是还要在 Canvas 绘制上进行优化。

我们从上图可以看到在单层节点很多的情况下，主节点与不可见节点的连线可见，但是没有任何价值，只是加重了用户对当前节点连线查看的负担。因此我们对线条也进行了渲染优化，仅当一条连线两端的节点都在可见范围中时才渲染连线，在连线的 Tooltip 上增加了来源去向的展示辅助查看。至此我们做到了在复杂情况下的流畅展示血缘数据。

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:00876036-a5dc-4efc-a982-160f3a9c3bd7

Original: https://www.cnblogs.com/bytedata/p/16734697.html
Author: 字节跳动数据平台
Title: 字节跳动数据血缘图谱升级方案设计与实现