This is the blog section.
Files in these directories will be listed in reverse chronological order.
This is the blog section.
Files in these directories will be listed in reverse chronological order.
目前支持:
调研开发中:
生产xx+应用,不断接入中
生产xx+应用,不断接入中
“让代码实现"一次编写,随处运行”。 借助Capa体系,使你的Java应用在改动量较小的情况下,拥有跨云、混合云运行的能力。"
作者简介: KevinTen,携程后端开发工程师,关注Reactive、RPC和云原生领域,对Mecha架构混合云中间件有深度实践经验。 Capa官方GitHub地址:https://github.com/capa-cloud/capa-java
在过去微服务的发展历程中,各大厂商基于SDK模式已经有相对完善的中间件体系。但在业务全球化、混合多云架构场景下,业务应用对基础设施的标准化和解耦、可迁移性以及拥抱开源成为新的诉求。 以当前的业界实践及趋势来看,ServiceMesh 这种 Sidecar 架构与体系是满足上述诉求的最佳实践。
ServiceMesh 在微服务领域已经非常流行,越来越多的公司开始在内部落地,ServiceMesh 带来的业务解耦,平滑升级等优势大大提高了中间件的迭代效率。
不过 ServiceMesh 只解决了服务间通讯的需求,而现实中的分布式应用存在更多的需求。而效仿 ServiceMesh 将应用需要的其他分布式能力外移到各种 Sidecar Runtime,这逐渐演变成了一个趋势。
本文主要对 ServiceMesh 进行回顾总结,并分享业界基于 ServiceMesh 这种 Sidecar 模式解决混合云应用开发的解决方案,最后是关于混合云应用开发模式的探讨。
在微服务架构下,基础架构团队一般会为应用提供一个封装了各种服务治理能力的 SDK,这种做法虽然保障了应用的正常运行,但缺点也非常明显,每次基础架构团队迭代一个新功能都需要业务方参与升级才能使用,尤其是 bugfix 版本,往往需要强推业务方升级,这里面的痛苦程度每一个基础架构团队成员都深有体会。
伴随着升级的困难,随之而来的就是应用使用的 SDK 版本差别非常大,生产环境同时跑着各种版本的 SDK,这种现象又会让新功能的迭代必须考虑各种兼容,就好像带着枷锁前进一般,这样随着不断迭代,会让代码维护非常困难,有些祖传逻辑更是一不小心就会掉坑里。
同时这种“重”SDK 的开发模式,导致异构语言的治理能力非常薄弱,如果想为各种编程语言都提供一个功能完整且能持续迭代的 SDK 其中的成本可想而知。
18 年的时候,Service Mesh 在国内持续火爆,这种架构理念旨在把服务治理能力跟业务解耦,让两者通过进程级别的通信方式进行交互。在这种架构模式下,服务治理能力从应用中剥离,运行在独立的进程中,迭代升级跟业务进程无关,这就可以让各种服务治理能力快速迭代,并且由于升级成本低,因此每个版本都可以全部升级,解决了历史包袱问题,同时 SDK 变“轻”直接降低了异构语言的治理门槛,再也不用为需要给各个语言开发相同服务治理能力的 SDK 头疼了。
参考资料:https://mp.weixin.qq.com/s/xokpWwTItyEW2YN4Iklmww
如果根据 “跨越鸿沟” 理论,服务网格已经跨越了 “鸿沟”,处于 “早期大众” 和 “晚期大众” 阶段之间。根据《Istio 大咖说》 观众中的反馈来看,用户已不再盲从于新技术,开始辩证的考虑 是否真的需要引入服务网格。

蚂蚁很快意识到了 Service Mesh 的价值,全力投入到这个方向,用 Go 语言开发了 MOSN 这样可以对标 envoy 的优秀数据面,全权负责服务路由,负载均衡,熔断限流等能力的建设,大大加快了公司内部落地 Service Mesh 的进度。
现在 MOSN 在蚂蚁内部已经覆盖了数千个应用、数十万容器,新创建的应用默认接入 MOSN,形成闭环。而且在大家最关心的资源占用、性能损耗方面 MOSN 也交出了一份让人满意的答卷:
RT 小于 0.2ms
CPU 占用增加 0%~2%
内存消耗增长小于 15M
由于 Service Mesh 降低了异构语言的服务治理门槛,NodeJS、C++等异构技术栈也在持续接入到 MOSN 中。
在看到 RPC 能力 Mesh 化带来的巨大收益之后,蚂蚁内部还把 MQ,Cache,Config 等中间件能力都进行了 Mesh 化改造,下沉到 MOSN,提高了中间件产品整体的迭代效率。
Mesh化是云原生落地的关键步骤。

参考资料:https://skyao.io/talk/201910-ant-finance-service-mesh-deep-practice/
一个现代分布式应用,往往会同时依赖 RPC、Cache、MQ、Config 等各种分布式能力来完成业务逻辑的处理。
当初看到 RPC 下沉的红利以后,其他各种能力也都快速下沉。
初期,大家都会以自己最熟悉的方式来开发,这就导致没有统一的规划管理,如上图所示,应用依赖了各种基础设施的 SDK,而每种 SDK 又以自己特有的方式跟 MOSN 进行交互,使用的往往都是由原生基础设施提供的私有协议,这直接导致了复杂的中间件能力虽然下沉,但应用本质上还是被绑定到了基础设施。
比如想把缓存从 Redis 迁移到 Memcache 的话,仍旧需要业务方升级 SDK,这种问题在应用上云的大趋势下表现的更为突出。
试想一下,如果一个应用要部署在云上,由于该应用依赖了各种基础设施,势必要先把整个基础设施搬到云上才能让应用顺利部署,这其中的成本可想而知。 因此如何让应用跟基础设施解绑,使其具备可移植能力,能够无感知跨平台部署是我们面临的第一个问题。
事实证明 Service Mesh 确实降低了异构语言的接入门槛,但在越来越多的基础能力下沉到 MOSN 以后,我们逐渐意识到为了让应用跟 MOSN 交互,各种 SDK 里都需要对通信协议,序列化协议进行开发,如果再加上需要对各种异构语言都提供相同的功能,那维护难度就会成倍上涨,
Service Mesh 让重 SDK 成为了历史,但对于现在各种编程语言百花齐放、各种应用又强依赖基础设施的场景来说,我们发现现有的 SDK 还不够薄,异构语言接入的门槛还不够低,如何进一步降低异构语言的接入门槛是我们面临的第二个问题。
参考资料:https://skyao.io/talk/202103-dapr-from-servicemesh-to-cloudnative/
20 年初的时候,Bilgin lbryam 发表了一篇名为 Multi-Runtime Microservices Architecture 的文章,里面对微服务架构下一阶段的形态进行了讨论。
如上图所示,作者把分布式服务的需求进行了抽象,总共分为了四大类:
生命周期(Lifecycle) 主要指应用的编译、打包、部署等事情,在云原生的大趋势下基本被 docker、kubernetes 承包。
网络(Networking) 可靠的网络是微服务之间进行通信的基本保障,Service Mesh 正是在这方面做了尝试,目前 MOSN、envoy 等流行的数据面的稳定性、实用性都已经得到了充分验证。
状态(State) 分布式系统需要的服务编排,工作流,分布式单例,调度,幂等性,有状态的错误恢复,缓存等操作都可以统一归为底层的状态管理。
绑定(Binding) 在分布式系统中,不仅需要跟其他系统通信,还需要集成各种外部系统,因此对于协议转换,多种交互模型、错误恢复流程等功能也都有强依赖。
明确了需求以后,借鉴了 Service Mesh 的思路,作者对分布式服务的架构演进进行了如下总结:
第一阶段就是把各种基础设施能力从应用中剥离解耦,通通变成独立 sidecar 模型伴随着应用一起运行。
第二阶段是把各种 sidecar 提供的能力统一抽象成若干个 Runtime,这样应用从面向基础组件开发就演变成了面向各种分布式能力开发,彻底屏蔽掉了底层实现细节,而且由于是面向能力,除了调用提供各种能力的 API 之外,应用再也不需要依赖各种各样基础设施提供的 SDK 了。
作者的思路跟我们希望解决的问题一致,我们决定使用 Runtime 的理念来解决 Service Mesh 发展到现在所遇到的新问题。
为了让大家对 Runtime 有一个更加清晰的认识,上图针对 Service Mesh 跟 Runtime 两种理念的定位、交互方式、通信协议以及能力丰富度进行了总结,可以看到相比 Service Mesh 而言,Runtime 提供了语义明确、能力丰富的 API,可以让应用跟它的交互变得更加简单直接。

dapr 是社区中一款知名的 Runtime 实现产品,活跃度也比较高。


参考资料:https://docs.dapr.io/concepts/service-mesh/#using-dapr-with-a-service-mesh

当考虑如何在已使用Service Mesh公司内部落地 dapr 时,提出了两种方案,如下:
替换:废弃掉现在的Istio,用 dapr 进行替换,这种方案存在两个问题:
a. dapr 虽然提供了很多分布式能力,但目前并不具备 Service Mesh 包含的丰富的服务治理能力。
b. Istio 已经大规模落地,并且经过了考验,直接用 dapr 来替换 Istio 稳定性有待验证。
共存:新增一个 dapr 容器,跟 Istio 以两个 sidecar 的模式进行部署。这种方案同样存在两个问题:
a. 引入一个新的 sidecar,我们就需要考虑它配套的升级、监控、注入等等事情,运维成本飙升。
b. 多维护一个容器意味着多了一层挂掉的风险,这会降低现在的系统可用性。
同样的,如果你目前正在使用 envoy 作为数据面,也会面临上述问题。
因此我们希望把 Runtime 跟 Service Mesh 两者结合起来,通过一个完整的 sidecar 进行部署,在保证稳定性、运维成本不变的前提下,最大程度复用现有的各种 Mesh 能力。此外我们还希望这部分 Runtime 能力除了跟 Istio 结合起来之外,未来也可以跟 envoy 结合起来,解决更多场景中的问题,Layotto 就是在这样的背景下诞生。
如上图所示,Layotto 是构建在 MOSN 之上,在下层对接了各种基础设施,向上层应用提供了统一的,具有各种各样分布式能力的标准 API。对于接入 Layotto 的应用来说,开发者不再需要关心底层各种组件的实现差异,只需要关注应用需要什么样的能力,然后调用对应能力的 API 即可,这样可以彻底跟底层基础设施解绑。
对应用来说,交互分为两块,一个是作为 gRPC Client 调用 Layotto 的标准 API,一个是作为 gRPC Server 来实现 Layotto 的回调,得利于gRPC 优秀的跨语言支持能力,应用不再需要关心通信、序列化等细节问题,进一步降低了异构技术栈的使用门槛。
除了面向应用,Layotto 也向运维平台提供了统一的接口,这些接口可以把应用跟 sidecar 的运行状态反馈给运维平台,方便 SRE 同学及时了解应用的运行状态并针对不同状态做出不同的举措,该功能考虑到跟 k8s 等已有的平台集成,因此我们提供了 HTTP 协议的访问方式。
除了 Layotto 本身设计以外,项目还涉及两块标准化建设,首先想要制定一套语义明确,适用场景广泛的 API 并不是一件容易的事情,为此我们跟阿里、 dapr 社区进行了合作,希望能够推进 Runtime API 标准化的建设,其次对于 dapr 社区已经实现的各种能力的 Components 来说,我们的原则是优先复用、其次开发,尽量不把精力浪费在已有的组件上面,重复造轮子。
最后 Layotto 目前虽然是构建在 MOSN 之上,未来我们希望 Layotto 可以跑在 envoy 上,这样只要应用接入了 Service Mesh,无论数据面使用的是 MOSN 还是 envoy,都可以在上面增加 Runtime能力。
Capa主要为了解决携程混合云部署开发的问题,主要面向Java技术栈。 github: https://github.com/capa-cloud/capa-java
Capa项目基于Mecha架构的设计理念,使用 富SDK模式 提供Multi-Runtime的标准API。
您可以简单的将Capa项目理解为 Dapr / Layotto 等Sidecar模式项目的SDK实现版本。
基于Mecha架构理念的Multi-Runtime,以Sidecar的方式提供标准API的功能,看起来似乎是最合理的选择。
那为什么不直接使用Dapr/Layotto等项目,而是选择开发 富SDK模式 的Capa项目呢。
概括:以Dapr为代表的Sidecar架构是未来,但现有的很多企业和系统很难一步到位的升级到Sidecar架构,富SDK架构将会长期的存在下去。
引申:面对庞大的Java系统体系,Capa项目将使用富SDK模型支持Java系统向Mecha架构过渡。在Dapr等项目成熟后,也可以无缝衔接到Sidecar架构。
选用sidecar来进行中间件支持,存在以下几个问题:

SDK模式的缺点
- 跨语言开发和维护成本高。无法支撑多语言战略,团队没有精力维护除了 Java 以外其他语言的 SDK;
- SDK升级侵入性强。客户端 SDK 版本升级推进困难,特别是遇到 Bug 的时候,彻底下线一个版本可能会花上几个月的时间,给业务带来了隐患;
- 依赖冲突
- ……
参考资料:https://github.com/dapr/dapr/issues/3261
设计思路:标准API + 可拔插可替换的SDK组件 模式
在不同的分布式中间件领域,Capa提供与具体中间件API无关的统一的标准编程API。 所以应用程序在使用Capa编程时不需要依赖任何具体的中间件API,只需要依赖Capa的标准编程API即可。
例如:
Mono<Void> result = client.publishEvent("mytopic", "my message");
在部署到不同的目标环境时,Capa将会装载标准API的不同实现类到应用程序中。当调用统一的编程API时,底层运行时会适配到不同的具体中间件SDK实现。
中间件团队需要针对不同目标环境,开发标准API在目标环境下的实现类即可;而应用代码可以拥有"一次编写,随处运行"的开发体验。
核心实现思想就是:面向接口编程 + 通过JavaSPI动态加载实现类

Maven配置伪代码示例:
<!-- 编程时直接引入API层 -->
<dependency>
<groupId>group.rxcloud</groupId>
<artifactId>capa-sdk</artifactId>
</dependency>
<profiles>
<!-- AWS云实现类 -->
<profile>aws</profile>
<dependencys>
<dependency>
<groupId>group.rxcloud</groupId>
<artifactId>capa-sdk-aws</artifactId>
</dependency>
</dependencys>
<!-- Ali云实现类 -->
<profile>ali</profile>
<dependencys>
<dependency>
<groupId>group.rxcloud</groupId>
<artifactId>capa-sdk-ali</artifactId>
</dependency>
</dependencys>
</profiles>
开发人员编程时,应用程序直接调用API层的接口进行编程。
在打包成镜像的过程中,通过maven profile加载不同云平台的jar实现。
在运行时,SDK通过Java SPI机制,动态加载API层接口的实现类。 该实现类,即为各个云平台上不同的SDK实现。
参考资料:https://www.yuque.com/docs/share/5221c27a-9d0c-44c7-8ef2-0956d3b09a32?#

如上图所示,一旦完成 Runtime API 的标准化建设,接入 Mecha 的应用天然具备了可移植性,应用不需要任何改造就可以在私有云以及各种公有云上部署,并且由于使用的是标准 API,应用也可以无需任何改造就在 Layotto / dapr / Capa 之间自由切换。
SDK模式的优点在于开发和接入的成本较低,可以快速拥有混合云部署开发的能力。但缺点也是无法忽视的,当随着Mecha Runtime的发展,未来SDK模式很可能将会迁移到更有技术优势的Mecha Runtime Sidecar模式中。
但也要考虑到性能敏感领域,例如分布式缓存领域。相关模块能接受额外 1ms 的响应延迟吗?Sidecar 带来的优势能否弥补这个问题? 所以in-process(SDK)模型在这些领域仍具备不可替代的优势。
推动API的标准化建设:使各大Mecha体系保持API层的统一,保留未来互相迁移的可能。
从未来几年来看:
就像SQL标准之于关系型数据库。
参考资料:https://www.infoq.cn/article/wjkNGoGaaHyKs7xIyTSB



在私有云的Configuraion-java-sdk(如apollo/qconfig)中,sdk本身支持监听配置变化。
但在AWS AppConfig java sdk中,本身并不支持监听配置变化。
那我们认为对于configuraion中间件领域而言,监听配置变化是level0的功能,是必须具有的功能。
所以需要在component-sdk中对其进行弥补:
以此,使用户接入时,认为具有这项能力。
能力分层模型:A云能够实现,B云通过其他方式也能实现。
高危,按条件选用。 明确告知在特定云上才有这项能力。
社区讨论:协商机制

高度定制化。不建议用户直接使用。
若要用,则使用定义好的有限枚举。 SDK实现中对该枚举进行支持。

参考资料:https://github.com/dapr/dapr/issues/2988
首先是分布式系统中经常使用的配置功能,应用一般使用配置中心来做开关或者动态调整应用的运行状态。Layotto 中配置模块的实现包括两部分,一个是对如何定义配置这种能力的 API 的思考,一个是具体的实现,下面逐个来看。
想要定义一个能满足大部分实际生产诉求的配置 API 并不是一件容易的事,dapr 目前也缺失这个能力,因此我们跟阿里以及 dapr 社区一起合作,为如何定义一版合理的配置 API 进行了激烈讨论。
目前讨论结果还没有最终确定,因此 Layotto 是基于我们提给社区的第一版草案进行实现,下面对我们的草案进行简要说明。
我们先定义了一般配置所需的基本元素:
appId:表示配置属于哪个应用
key:配置的 key
content:配置的值
group:配置所属的分组,如果一个 appId 下面的配置过多,我们可以给这些配置进行分组归类,便于维护。
此外我们追加了两种高级特性,用来适配更加复杂的配置使用场景:
label,用于给配置打标签,比如该配置属于哪个环境,在进行配置查询的时候,我们会使用 label + key 来查询配置。
tags,用户给配置追加的一些附加信息,如描述信息、创建者信息,最后修改时间等等,方便配置的管理,审计等。
对于上述定义的配置 API 的具体实现,目前支持查询、订阅、删除、创建、修改五种操作,其中订阅配置变更后的推送使用的是 gRPC 的 stream 特性,而底层实现这些配置能力的组件,我们选择了国内流行的 apollo,后面也会根据需求增加其他实现。
level1

level2

levelx…
metadata
RPC 的能力大家不会陌生,这可能是微服务架构下最最基础的需求,对于 RPC 接口的定义,我们同样参考了 dapr 社区的定义,发现完全可以满足我们的需求,因此接口定义就直接复用 dapr 的,但目前 dapr 提供的 RPC 实现方案还比较薄弱,而 MOSN 经过多年迭代,能力已经非常成熟完善,因此我们大胆把 Runtime 跟 Service Mesh 两种思路结合在一起,把 MOSN 本身作为我们实现 RPC 能力的一个 Component,这样 Layotto 在收到 RPC 请求以后交给 MOSN 进行实际数据传输,这种方案可以通过 istio 动态改变路由规则,降级限流等等设置,相当于直接复用了 Service Mesh 的各种能力,这也说明 Runtime 不是要推翻 Service Mesh,而是要在此基础上继续向前迈一步。

如果要使用原生的Capa API,您的遗留系统需要面对较大的重构工作量。
为了使迁移做到低成本,我们可以复用目前使用到的中间件API。
通过开发一个适配层项目(提供相同的注解/接口调用方式),将原中间件API的实现更改为Capa API。
如此一来,应用程序只需要更改很少的代码(例如更换注解/接口的路径名)即可迁移到Capa架构。
目前在Java语言上,Capa可以支持:

<dependency>
<groupId>com.ctrip.framework.apollo</groupId>
<artifactId>apollo-client</artifactId>
</dependency>
to
<dependency>
<groupId>group.rxcloud</groupId>
<artifactId>capa-sdk-configuration</artifactId>
</dependency>
import com.ctrip.framework.apollo.Config;
@Config("testjson.json")
private Person person;
to
import group.rxcloud.capa.config.CloudConfig;
@CloudConfig("testjson.json")
private Person person;
Q: 为什么不保持import路径和使用方式完全一致?
A: 当运行在特定云上时,需要加载对应的sdk进来;这时如果有类路径完全一致的类存在,会导致类加载冲突。
Capa(Java SDK)是面向Java应用实现Mecha架构的SDK解决方案,它目前支持以下领域:
标准API中只涵盖了作为Client调用其他服务的API。
但不同云上服务注册的框架弈有所不同(例如:dubbo/spring cloud/service mesh)。
在sidecar模式中,sidecar本身可类似service mesh一样承担服务注册的能力,通过callback API调用服务接口,但SDK模式时却不行。
所以添加了作为服务端的API:
// -- Runtime as Server
/**
* Register onInvoke method when runtime as server.
*
* @param <T> The Type of the request type, use byte[] to skip serialization.
* @param <R> The Type of the response type, use byte[] to skip serialization.
* @param methodName The actual Method to be call in the application.
* @param httpExtensions Additional fields that are needed if the receiving app is listening on HTTP, {@link HttpExtension#NONE} otherwise.
* @param onInvoke the on invoke
* @param metadata Metadata (in GRPC) or headers (in HTTP) to be received in request.
* @return A Mono Plan of register result.
*/
<T, R> Mono<Boolean> registerMethod(String methodName, List<HttpExtension> httpExtensions,
Function<T, R> onInvoke,
Map<String, String> metadata);
/**
* Register controller class when runtime as server.
*
* @param registerServerRequest the register server request
* @return A Mono Plan of register result.
*/
Mono<Boolean> registerServer(RegisterServerRequest registerServerRequest);

标准API中定义了Pub发布消息的API,而Sub订阅消息则通过callback进行定义。
sidecar通过callback回调触发Sub订阅,但对于SDK模式而言,并没有额外的进程可触发callback逻辑。
所以添加了消息订阅的API:
// -- Runtime as Subscriber
/**
* Subscribe events.
*
* @param pubsubName the pubsub name we will subscribe the event from.
* @param topicName the topicName where the event will be subscribed.
* @param metadata The metadata for the subscription.
* @return a Flux stream of subscription events.
*/
Flux<TopicEventRequest> subscribeEvents(String pubsubName, String topicName, Map<String, String> metadata);
/**
* Subscribe events.
*
* @param topicSubscription the request for topic subscription.
* @return a Flux stream of subscription events.
*/
Flux<TopicEventRequest> subscribeEvents(TopicSubscription topicSubscription);
目前KV主要使用的Redis。
而标准State API,可表达的语义较弱,无法cover redis场景。
所以暂时未使用。
规划中。
但对于跨云环境而言,很难做到全局锁,只能做到Region/云级别的锁。
目前在java中,直接使用本地进程内算法实现。
规划中。
后续应该会做,因为不同云上的文件系统使用方式不同。
规划中。
后续可能会做,因为不同云上密钥存储使用方式不同。
但目前密钥存储还未暴露给用户使用,而只是中间件在使用,所以中间件中直接引用了对应云的SDK来实现。
目前的 dapr api 并不能在保持可移植性的同时满足现实世界中应用程序的所有需求。
参考:https://github.com/mosn/layotto/issues/530
对于SQL而言:
故Capa目前仅提供了一套待讨论的SQL API: dapr【提案】数据库API设计 #3354
而在落地实践过程中,SQL关系型数据库领域采用的方案是 DAL 数据库连接技术,经二次开发支持混合云SDK模式。
目前的State API,无法支持复杂的Redis语义表达。 故Capa在Redis领域,基于 Jedis 定义了一套 Redis API
场景:中间件SDK/应用程序 需要记录指标埋点。
之前,我们可以通过 CAT 这类应用监控告警平台进行实现。
Capa复用了 OpenTelemetry 的API,来实现监控告警/指标收集的功能。

场景:触发job调度任务(规划中)
/**
* Schedule Bindings Runtimes standard API defined.
*/
public interface ScheduleRuntimes {
/**
* Invokes a Schedule Binding operation.
*
* @param appId the app id
* @param jobName the job name
* @param metadata the metadata
* @return the job flux stream
*/
Flux<Object> invokeSchedule(String appId, String jobName, Map<String, String> metadata);
}
Capa SDK本身不具有丰富的功能,更像是一个包装适配层,具体功能由下层的具体中间件来实现。
| 领域 | 云厂商 | 技术选型 | 实践经验 |
|---|---|---|---|
| RPC | Trip | Trip SOA | |
| AWS | AWS AppMesh | 重试、熔断、超时等全部从SDK中下沉到Mesh层处理 | |
| Configuration | Trip | Apollo / QConfig | |
| AWS | AWS AppConfig | 功能较为简陋,需要在SDK中进行弥补 | |
| MQ | Trip | QMQ | |
| AWS | AWS MSK Kafka | 缺少如延时消息等功能,需要借助周边系统进行弥补 | |
| Redis | Trip | CRedis | |
| AWS | AWS ElasticCache | ||
| Log/Metric/Trace | Trip | CAT | |
| AWS | AWS CloudWatch | OpenTelemetryAPI对接CloudWatch SDK | |
| DB(SQL) | Trip | DAL | 非Mecha架构 |
| AWS | DAL + AWS RDS | 非Mecha架构 | |
| Schedule | Trip | QSchedule | |
| AWS | K8S cronJob |


自定义配置类似layotto:layotto 配置下发通道与配置热加载 #500
集成在对应云的SDK中。
如果用户在调用API时明确指定了,使用redis作为store_name,那么对于没有部署redis的云平台,则无法运行。
所以对于redis这种与基础设施有关的选择,capa并不暴露给用户:
主要也是因为,capa当前都是1:1的情况,目前不存在1个领域有多种实现。
后续如果有这种情况,考虑使用configuration配置功能进行映射:
SDK中的一些逻辑是可替换的,实现上通过面向Java接口编程,在SDK中提供了接口的默认实现。
应用程序可以通过覆盖接口实现,从而实现自定义逻辑。
一方面在进行混合云改造的过程中,面临过渡期的问题。 当一部分的服务和数据在原云平台上时,新云平台将面临和原云平台进行交互的问题。
另一方面作为混合云架构,若要发挥每个云各自的优势,有可能不同服务会部署在不同的云平台上,这时也会面对跨云交互的问题。
而要实现跨云交互,主要由以下两个思路:
Mecha Runtime/SDK感知混合云,并实现跨云交互。
但可能实现起来比较复杂,设计到跨云的认证、网络打通等。
Mecha Runtime/SDK将会比较轻量,不需要感知其他云。 涉及到某个领域的跨云交互问题,交由该领域的 基础设施/周边系统 完成。
Capa相比于layotto等sidecar模型,采用SDK模式更不易引入较复杂逻辑,会导致SDK过于臃肿,且无法多语言复用。 故Capa架构中,所有跨云交互问题,都交由具体领域的周边系统完成,Capa只关注于当前云的配置和使用。
跨云服务调用的前提是,网络层面流量已经打通。 例如在AWS上,通过PrivateLink技术进行网络流量打通。
之后在服务调用中,依赖网络基础的流量转发实现:
| RPC网络基础设施 | 跨云交互能力 |
|---|---|
| K8S | ExternalService转发 |
| Istio | ServiceEntry转发 |
| Dubbo/SpringCloud/Trip SOA | MockService转发 |
| AWS AppMesh | VirtualNode DNS转发 |
| 不支持? | 搭建Nginx代理实现转发 |


使用周边系统:MQ同步工具,进行跨云消息传输。
按需申请,消息格式转换,失败重试。

DB数据使用周边系统:MySQL同步工具 ,进行跨云数据传输。
Redis数据目前不做跨云传输。
使用周边系统:XX同步工具,进行跨云的数据同步。
WebAssembly,简称 WASM,是一个二进制指令集,最初是跑在浏览器上来解决 JavaScript 的性能问题,但由于它良好的安全性,隔离性以及语言无关性等优秀特性,很快人们便开始让它跑在浏览器之外的地方,随着 WASI 定义的出现,只需要一个 WASM 运行时,就可以让 WASM 文件随处执行。
既然 WebAssembly 可以在浏览器以外的地方运行,那么我们是否能把它用在 Serverless 领域?目前已经有人在这方面做了一些尝试,不过如果这种方案真的想落地的话,首先要考虑的就是如何解决运行中的 WebAssembly 对各种基础设施的依赖问题。
目前 MOSN 通过集成 WASM Runtime 的方式让 WASM 跑在 MOSN 上面,以此来满足对 MOSN 做自定义扩展的需求。同时,Layotto 也是构建在 MOSN 之上,因此我们考虑把二者结合在一起,实现方案如下图所示:
开发者可以使用 Go/C++/Rust 等各种各样自己喜欢的语言来开发应用代码,然后把它们编译成 WASM 文件跑在 MOSN 上面,当 WASM 形态的应用在处理请求的过程中需要依赖各种分布式能力时就可以通过本地函数调用的方式调用 Layotto 提供的标准 API,这样直接解决了 WASM 形态应用的依赖问题。
目前 Layotto 提供了 Go 跟 Rust 版 WASM 的实现,虽然只支持 demo 级功能,但已经足够让我们看到这种方案的潜在价值。
此外,WASM 社区目前还处于初期阶段,有很多地方需要完善,我们也给社区提交了一些 PR共同建设,为 WASM 技术的落地添砖加瓦。
虽然现在 Layotto 中对 WASM 的使用还处于试验阶段,但我们希望它最终可以成为 Serverless 的一种实现形态,如上图所示,应用通过各种编程语言开发,然后统一编译成 WASM 文件,最后跑在 Layotto+MOSN 上面,而对于应用的运维管理统一由 k8s、docker、prometheus 等产品负责。
对于中间件领域,可以由社区定义一套执行模板,而用户可以灵活的插入自定义的回调逻辑,更具有动态扩展性。
Runtime启动多个相互隔离的 WASM Runtime ,从而隔离每个中间件组件的运行时(CPU/内存/…),避免异常的扩散和传播,互不影响。
Istio 的扩展机制使用 Proxy-Wasm 应用二进制接口(ABI)规范,提供了一套代理无关的流媒体 API 和实用功能,可以用任何有合适 SDK 的语言来实现。
扩展 Istio 的功能,满足你的特定需求,需要三个步骤:
参考资料:https://mp.weixin.qq.com/s/VAwTCIP6RhqDNoBbmE8zVA



参考资料:https://mp.weixin.qq.com/s/W9NySdKnxuQ6S917QQn3PA
socket重定向,绕过TCP/IP协议栈,
最后来看下社区的做的一些事情。
参考资料:https://zhuanlan.zhihu.com/p/435012312?utm_source=wechat_session&utm_medium=social&utm_oi=618742049890111488&utm_content=group2_article&utm_campaign=shareopn
上图列出了 Layotto 跟 dapr 现有的能力对比,在 Layotto 的开发过程中,我们借鉴 dapr 的思路,始终以优先复用、其次开发为原则,旨在达成共建的目标,而对于正在建设或者未来要建设的能力来说,我们计划优先在 Layotto 上落地,然后再提给社区,合并到标准 API,鉴于社区异步协作的机制,沟通成本较高,因此短期内可能 Layotto 的 API 会先于社区,但长期来看一定会统一。
关于如何定义一套标准的 API 以及如何让 Layotto 可以跑在 envoy 上等等事项,我们已经在各个社区进行了深入讨论,并且以后也还会继续推进。
对于 API 标准化的建设是一件需要长期推动的事情,同时标准化意味着不是满足一两种场景,而是尽可能的适配大多数使用场景。
参考OS领域当年是怎么定API的,我们可以把Runtime API设计成多层:
社区标准API
企业内部的私有API
基础设施特定API
分别对应OS领域的:
POSIX API
各种Unix-like系统自己的System Call
特殊硬件提供的特殊功能
基于这种思想,尝试让Runtime支持API插件,让用户自己实现自己的私有API
直接支持SQL
直接支持Redis
需要进一步讨论。
各个Mecha项目积极交流,合作共建,互为补充。
参考资料:https://blog.dapr.io/posts/2021/03/19/how-alibaba-is-using-dapr/
微服务架构分层逐渐形成,后端 BaaS 化,客户端轻量化,业务侧 Serverless 化,让业务更加聚焦业务开发,进一步提升研发效率。
FaaS 对多语言支持的需求强烈,并不局限于 Java。但实际上,由于成本和时间限制,用所有语言重新开发客户是不切实际的。
借助 Dapr,阿里巴巴解决了 FaaS 的多语言问题,帮助客户通过 FaaS 提高开发效率。

我们旨在消除业务开发团队进行代码更改以减少迁移期间对业务应用程序的影响的需要。在将这些遗留系统迁移到 Dapr 时,我们设计了一个 Java 适配层,它将原始 Java 调用适配到 Dapr 客户端 API。

Dapr 的可移植性将钉钉文档的上层应用与消息系统等底层基础设施解耦。从而,钉钉文档实现了不同云平台之间的平滑迁移。

参考文章:https://mp.weixin.qq.com/s/eh0Jz7YeF27bXqdBMd4MPw
中间件的客户端,均直接集成在业务进程内:

中间件能力下沉,应用和基础设施实现部分解耦:

将应用和具体基础设施的类型解耦,仅依赖标准 API 编程:

微软在 Ignite 2021 大会上发布了预览版的Azure Container Apps,这是一个完全托管的无服务器容器运行时,用于大规模构建和运行现代应用程序。
参考资料:https://www.cnblogs.com/shanyou/p/15509042.html
CNCF 托管了许多与 Dapr 紧密结合的项目。例如,Dapr 使用 gRPC 作为它的主要 RPC 堆栈。OpenTelemetry 用作 Dapr 的遥测协议,CloudEvents 用于标准化 Dapr 的 Pub/Sub API 中的消息格式,SPIFFE 用作 Dapr 的访问列表特性的身份格式,Prometheus 用作 Dapr 端点的指标格式,Dapr 支持 NATS 的 Pub/Sub 实现以及 Open Policy Agent。此外,Dapr 使用 Operator 在 Kubernetes 上原生运行,并使用 Helm 作为主要部署机制。
参考资料:https://mp.weixin.qq.com/s/8qnmL12JrESwJg3RsSelJQ

参考文档:https://www.servicemesher.com/istio-handbook/concepts/sidecar-injection.html
Sidecar模式是一种单节点、多容器的应用设计形式。Sidecar主张以额外的容器来扩展或增强主容器,而这个额外的容器被称为Sidecar容器
sidecar 模式也符合当前微服务的以下特点:

在软件架构中, Sidecar 连接到父应用并且为其添加扩展或者增强功能。Sidecar 应用与主应用程序松散耦合。它可以屏蔽不同编程语言的差异,统一实现微服务的可观察性、监控、日志记录、配置、断路器等功能。
使用 sidecar 模式部署服务网格时,无需在节点上运行代理,但是集群中将运行多个相同的 sidecar 副本。在 sidecar 部署方式中,每个应用的容器旁都会部署一个伴生容器(如 Envoy 或 MOSN),这个容器称之为 sidecar 容器。Sidecar 接管进出应用容器的所有流量。在 Kubernetes 的 Pod 中,在原有的应用容器旁边注入一个 Sidecar 容器,两个容器共享存储、网络等资源,可以广义的将这个包含了 sidecar 容器的 Pod 理解为一台主机,两个容器共享主机资源。
因其独特的部署结构,使得 sidecar 模式具有以下优势:
借助于K8S良好的可拓展性,使用sidecar模式可以享受到分布式系统中的规模化效率红利。相比于这种效率提升,我们可以容许性能上的开销。
相关文档:https://www.infoq.cn/news/58j970hvYSx2QSEvCESh

DaemonSet
sidecar的代码很简单,与写一个webserver类似。
但与 vm/k8s 的自动化集成比较复杂。
func main() {
if err := rootCmd.Execute(); err != nil {
log.Error(err)
os.Exit(-1)
}
}
手动/脚本 操作:
借助 docker-compose.yml 文件,开发人员可定义一组相关服务,通过部署命令将其部署为组合应用程序。 它还配置其依赖项关系和运行时配置。多个容器使用同一个network。

istio dockerfile:
# BASE_DISTRIBUTION is used to switch between the old base distribution and distroless base images
ARG BASE_DISTRIBUTION=debug
# Version is the base image version from the TLD Makefile
ARG BASE_VERSION=latest
# The following section is used as base image if BASE_DISTRIBUTION=debug
FROM gcr.io/istio-release/base:${BASE_VERSION} as debug
# The following section is used as base image if BASE_DISTRIBUTION=distroless
# This image is a custom built debian11 distroless image with multiarchitecture support.
# It is built on the base distroless image, with iptables binary and libraries added
# The source can be found at https://github.com/istio/distroless/tree/iptables
# This version is from commit 105e1319a176a5156205b9e351b4e2016363f00d.
FROM gcr.io/istio-release/iptables@sha256:bae9287d64be13179b7bc794ec3db26bd5c5fe3fb591c484992366314c9a7d3d as distroless
# This will build the final image based on either debug or distroless from above
# hadolint ignore=DL3006
FROM ${BASE_DISTRIBUTION:-debug}
WORKDIR /
ARG proxy_version
ARG istio_version
ARG SIDECAR=envoy
# Copy Envoy bootstrap templates used by pilot-agent
COPY envoy_bootstrap.json /var/lib/istio/envoy/envoy_bootstrap_tmpl.json
COPY gcp_envoy_bootstrap.json /var/lib/istio/envoy/gcp_envoy_bootstrap_tmpl.json
# Install Envoy.
ARG TARGETARCH
COPY ${TARGETARCH:-amd64}/${SIDECAR} /usr/local/bin/${SIDECAR}
# Environment variable indicating the exact proxy sha - for debugging or version-specific configs
ENV ISTIO_META_ISTIO_PROXY_SHA $proxy_version
# Environment variable indicating the exact build, for debugging
ENV ISTIO_META_ISTIO_VERSION $istio_version
ARG TARGETARCH
COPY ${TARGETARCH:-amd64}/pilot-agent /usr/local/bin/pilot-agent
COPY stats-filter.wasm /etc/istio/extensions/stats-filter.wasm
COPY stats-filter.compiled.wasm /etc/istio/extensions/stats-filter.compiled.wasm
COPY metadata-exchange-filter.wasm /etc/istio/extensions/metadata-exchange-filter.wasm
COPY metadata-exchange-filter.compiled.wasm /etc/istio/extensions/metadata-exchange-filter.compiled.wasm
# The pilot-agent will bootstrap Envoy.
ENTRYPOINT ["/usr/local/bin/pilot-agent"]
docker-compose:
version: '3.4'
services:
webmvc:
image: eshop/web
environment:
- CatalogUrl=http://catalog-api
- OrderingUrl=http://ordering-api
ports:
- "80:80"
depends_on:
- catalog-api
- ordering-api
catalog-api:
image: eshop/catalog-api
environment:
- ConnectionString=Server=sqldata;Port=1433;Database=CatalogDB;…
ports:
- "81:80"
depends_on:
- sqldata
ordering-api:
image: eshop/ordering-api
environment:
- ConnectionString=Server=sqldata;Database=OrderingDb;…
ports:
- "82:80"
extra_hosts:
- "CESARDLBOOKVHD:10.0.75.1"
depends_on:
- sqldata
sqldata:
image: mcr.microsoft.com/mssql/server:latest
environment:
- SA_PASSWORD=Pass@word
- ACCEPT_EULA=Y
ports:
- "5433:1433"
Kompose是个转换工具,可将 compose(即 Docker Compose)所组装的所有内容 转换成容器编排器(Kubernetes 或 OpenShift)可识别的形式。
要将 docker-compose.yml 转换为 kubectl 可用的文件,请运行 kompose convert 命令进行转换,然后运行 kubectl create -f
参考文章:https://blog.hdls.me/15564491070483.html
K8S作为云原生操作系统的定位,其设计理念是"微内核"架构。
Kubernetes 的 apiserver 一开始就有 AdmissionController 的设计,这个设计和各类 Web 框架中的 Filter 很像,就是一个插件化的责任链,责任链中的每个插件针对 apiserver 收到的请求做一些操作或校验。分类
Kubernetes 中的许多高级功能需要启用准入控制器才能正确支持该功能。
对于在数据持久化之前,拦截到 Kubernetes API server 的请求

通过创建webhook资源,利用k8s的webhook能力实现pod的自动注入。
基于 Kubernetes 的 突变 webhook 入驻控制器(mutating webhook addmission controller 的自动 sidecar 注入方式。
实现流程大致如下:
sidecarConfig, err := loadConfig(parameters.sidecarCfgFile)
pair, err := tls.LoadX509KeyPair(parameters.certFile, parameters.keyFile)
whsvr := &WebhookServer {
sidecarConfig: sidecarConfig,
server: &http.Server {
Addr: fmt.Sprintf(":%v", 443),
TLSConfig: &tls.Config{Certificates: []tls.Certificate{pair}},
},
}
// define http server and server handler
mux := http.NewServeMux()
mux.HandleFunc("/mutate", whsvr.serve)
whsvr.server.Handler = mux
// start webhook server in new rountine
go func() {
if err := whsvr.server.ListenAndServeTLS("", ""); err != nil {
glog.Errorf("Filed to listen and serve webhook server: %v", err)
}
}()
capa-injector: https://github.com/capa-cloud/capa-injector
apiVersion: v1
kind: ConfigMap
metadata:
name: sidecar-injector-webhook-configmap
data:
sidecarconfig.yaml: |
containers:
- name: sidecar-nginx
image: nginx:1.12.2
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
volumeMounts:
- name: nginx-conf
mountPath: /etc/nginx
volumes:
- name: nginx-conf
configMap:
name: nginx-configmap
由于准入控制是一个高安全性操作,所以对外在的 webhook server 提供 TLS 是必须的。作为流程的一部分,我们需要创建由 Kubernetes CA 签名的 TLS 证书,以确保 webhook server 和 apiserver 之间通信的安全性。
apiVersion: v1
kind: Secret
metadata:
name: sidecar-injector
namespace: sidecar-injector
data:
tls.crt: |
LS0tLS1CRUdJTi...
tls.key: |
LS0tLS1CRUdJTiBSU0...
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: sidecar-injector-webhook-deployment
labels:
app: sidecar-injector
spec:
replicas: 1
template:
metadata:
labels:
app: sidecar-injector
spec:
containers:
- name: sidecar-injector
image: morvencao/sidecar-injector:v1
imagePullPolicy: IfNotPresent
args:
- -sidecarCfgFile=/etc/webhook/config/sidecarconfig.yaml
- -tlsCertFile=/etc/webhook/certs/cert.pem
- -tlsKeyFile=/etc/webhook/certs/key.pem
- -alsologtostderr
- -v=4
- 2>&1
volumeMounts:
- name: webhook-certs
mountPath: /etc/webhook/certs
readOnly: true
- name: webhook-config
mountPath: /etc/webhook/config
volumes:
- name: webhook-certs
secret:
secretName: sidecar-injector-webhook-certs
- name: webhook-config
configMap:
name: sidecar-injector-webhook-configmap
apiVersion: v1
kind: Service
metadata:
name: sidecar-injector-webhook-svc
labels:
app: sidecar-injector
spec:
ports:
- port: 443
targetPort: 443
selector:
app: sidecar-injector
这个 Service 会被 MutatingWebhookConfiguration 中定义的 clientConfig 部分访问,默认的端口 spec.ports.port 需要设置为 443。
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
name: sidecar-injector-webhook-cfg
labels:
app: sidecar-injector
webhooks:
- name: sidecar-injector.morven.me
clientConfig:
service:
name: sidecar-injector-webhook-svc
namespace: default
path: "/mutate"
caBundle: ${CA_BUNDLE}
rules:
- operations: [ "CREATE" ]
apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods"]
namespaceSelector:
matchLabels:
sidecar-injector: enabled
kubectl config view –raw –minify –flatten -o jsonpath=’{.clusters[].cluster.certificate-authority-data}
Kubernetes虽然提供了多种容器编排对象,例如Deployment、StatefulSet、DeamonSet、Job等,还有多种基础资源封装例如ConfigMap、Secret、Serivce等,但是一个应用往往有多个服务,有的可能还要依赖持久化存储,当这些服务之间直接互相依赖,需要有一定的组合的情况下,使用YAML文件的方式配置应用往往十分繁琐还容易出错,这时候就需要服务编排工具。

$ helm install istio-base istio/base -n istio-system
Istio 使用 ValidatingAdmissionWebhooks 验证 Istio 配置,使用 MutatingAdmissionWebhooks 自动将 Sidecar 代理注入至用户 Pod。
它使用 MutatingWebhook 机制在 pod 创建的时候将 sidecar 的容器和卷添加到每个 pod 的模版里。

containers:
- image: docker.io/istio/examples-bookinfo-productpage-v1:1.15.0 # 应用镜像
name: productpage
ports:
- containerPort: 9080
- args:
- proxy
- sidecar
- --domain
- $(POD_NAMESPACE).svc.cluster.local
- --configPath
- /etc/istio/proxy
- --binaryPath
- /usr/local/bin/envoy
- --serviceCluster
- productpage.$(POD_NAMESPACE)
- --drainDuration
- 45s
- --parentShutdownDuration
- 1m0s
- --discoveryAddress
- istiod.istio-system.svc:15012
- --zipkinAddress
- zipkin.istio-system:9411
- --proxyLogLevel=warning
- --proxyComponentLogLevel=misc:error
- --connectTimeout
- 10s
- --proxyAdminPort
- "15000"
- --concurrency
- "2"
- --controlPlaneAuthPolicy
- NONE
- --dnsRefreshRate
- 300s
- --statusPort
- "15020"
- --trust-domain=cluster.local
- --controlPlaneBootstrap=false
image: docker.io/istio/proxyv2:1.5.1 # sidecar proxy
name: istio-proxy
ports:
- containerPort: 15090
name: http-envoy-prom
protocol: TCP
initContainers:
- command:
- istio-iptables
- -p
- "15001"
- -z
- "15006"
- -u
- "1337"
- -m
- REDIRECT
- -i
- '*'
- -x
- ""
- -b
- '*'
- -d
- 15090,15020
image: docker.io/istio/proxyv2:1.5.1 # init 容器
name: istio-init
启动的最小配置集,可以和镜像集成在一起。
可以使用K8S的配置进行管理:
控制面下发,全量/增量推送:
参考文档:https://www.infoq.cn/article/jTJGTtu2AgX74GkGif8Y


Init 容器是一种专用容器,它在应用程序容器启动之前运行,用来包含一些应用镜像中不存在的实用工具或安装脚本。
一个 Pod 中可以指定多个 Init 容器,如果指定了多个,那么 Init 容器将会按顺序依次运行。只有当前面的 Init 容器必须运行成功后,才可以运行下一个 Init 容器。当所有的 Init 容器运行完成后,Kubernetes 才初始化 Pod 和运行应用容器。
Init 容器使用 Linux Namespace,所以相对应用程序容器来说具有不同的文件系统视图。因此,它们能够具有访问 Secret 的权限,而应用程序容器则不能。
在 Pod 启动过程中,Init 容器会按顺序在网络和数据卷初始化之后启动。每个容器必须在下一个容器启动之前成功退出。如果由于运行时或失败退出,将导致容器启动失败,它会根据 Pod 的 restartPolicy 指定的策略进行重试。然而,如果 Pod 的 restartPolicy 设置为 Always,Init 容器失败时会使用 RestartPolicy 策略。
在所有的 Init 容器没有成功之前,Pod 将不会变成 Ready 状态。Init 容器的端口将不会在 Service中进行聚集。 正在初始化中的 Pod 处于 Pending 状态,但应该会将 Initializing 状态设置为 true。Init 容器运行完成以后就会自动终止。


该容器存在的意义就是让 sidecar 代理可以拦截所有的进出 pod 的流量,15090 端口(Mixer 使用)和 15092 端口(Ingress Gateway)除外的所有入站(inbound)流量重定向到 15006 端口(sidecar),再拦截应用容器的出站(outbound)流量经过 sidecar 处理(通过 15001 端口监听)后再出站。
istio-iptables [flags]
-p: 指定重定向所有 TCP 流量的 sidecar 端口(默认为 $ENVOY_PORT = 15001)
-m: 指定入站连接重定向到 sidecar 的模式,“REDIRECT” 或 “TPROXY”(默认为 $ISTIO_INBOUND_INTERCEPTION_MODE)
-b: 逗号分隔的入站端口列表,其流量将重定向到 Envoy(可选)。使用通配符 “*” 表示重定向所有端口。为空时表示禁用所有入站重定向(默认为 $ISTIO_INBOUND_PORTS)
-d: 指定要从重定向到 sidecar 中排除的入站端口列表(可选),以逗号格式分隔。使用通配符“*” 表示重定向所有入站流量(默认为 $ISTIO_LOCAL_EXCLUDE_PORTS)
-o:逗号分隔的出站端口列表,不包括重定向到 Envoy 的端口。
-i: 指定重定向到 sidecar 的 IP 地址范围(可选),以逗号分隔的 CIDR 格式列表。使用通配符 “*” 表示重定向所有出站流量。空列表将禁用所有出站重定向(默认为 $ISTIO_SERVICE_CIDR)
-x: 指定将从重定向中排除的 IP 地址范围,以逗号分隔的 CIDR 格式列表。使用通配符 “*” 表示重定向所有出站流量(默认为 $ISTIO_SERVICE_EXCLUDE_CIDR)。
-k:逗号分隔的虚拟接口列表,其入站流量(来自虚拟机的)将被视为出站流量。
-g:指定不应用重定向的用户的 GID。(默认值与 -u param 相同)
-u:指定不应用重定向的用户的 UID。通常情况下,这是代理容器的 UID(默认值是 1337,即 istio-proxy 的 UID)。
-z: 所有进入 pod/VM 的 TCP 流量应被重定向到的端口(默认 $INBOUND_CAPTURE_PORT = 15006)。
#!/bin/bash
# Forward TCP traffic on port 80 to port 8000 on the eth0 interface.
iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 80 -j REDIRECT --to-port 8000
# List all iptables rules.
iptables -t nat --list
# Use the latest Ubuntu image for the base.
FROM ubuntu:latest
# Install the iptables command.
RUN apt-get update && \
apt-get install -y iptables
# Copy the initialization script into the container.
COPY init.sh /usr/local/bin/
# Mark the initialization script as executable.
RUN chmod +x /usr/local/bin/init.sh
# Start the initialization script on container startup.
ENTRYPOINT ["init.sh"]
capa

使用单容器多进程模型。
在proxy镜像中,pilot-agent 负责的工作包括:

[/pilot-agent/main/initStatusServer]
对于 ready 检查,调用的路径为/healthz/ready, 并配合设置的端口 applicationPorts 通过 Envoy 的 admin 端口进行对应的端口进行检查,用于决定 Envoy 是否已经 ready 接受相对应的流量。
检查原理是通过本地管理端口,如 http://127.0.0.1:15000/listeners 获取 Envoy 当前监听的全部端口,然后将配置的端口 applicationPorts 在监听的端口中进行查找,来决定 Envoy 是否 ready。

检查的路径为 /url 路径,在 header 中设置 istio-app-probe-port 端口,使用 访问路径中的 url 来进行检查,最终调用的是 http://127.0.0.1:istio-app-probe-port/url,头部设置的全部参数也都会传递到别检测的服务端口上;
[/pilot-agent/main/istio_agent.NewAgent/initXdsProxy]
[/pilot-agent/main/istio_agent.NewAgent]

代理属于第三方提供,istio是对其进行了管理和拓展。
如果代理本身属于istio,是可以实现单进程模型。
由agent负责配置监听和下发。
解耦配置管理和运行时,同时可以对proxy进程进行热重启。
ip netns exec cni-bf783dac-fe05-cb35-4d5a-848449119b19 iptables -L -t nat
-A PREROUTING -p tcp -j ISTIO_INBOUND # PREROUTING全部转发到INBOUND,PREROUTING发生在流入的数据包进入路由表之前
-A OUTPUT -p tcp -j ISTIO_OUTPUT # 由本机产生的数据向外转发的
-A ISTIO_INBOUND -p tcp -m tcp --dport 22 -j RETURN # 22 15090 15021 15020的不转发到ISTIO_REDIRECT
-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT # 剩余的流量都转发到ISTIO_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006 # 转发到15006
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN # 127.0.0.6是InboundPassthroughBindIpv4,代表原地址是passthrough的流量都直接跳过,不劫持
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT #lo网卡出流量,目标地址不是localhost的,且为同用户的流量进入ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN # lo网卡出流量 非同用户的不劫持
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN # 剩下的同用户的都跳过
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT # lo网卡出流量,目标地址非本地,同用户组的流量进入ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN # lo网卡出流量非同组的不劫持
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN # 剩余的同用户的不劫持
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN # 剩余的目标地址为127的不劫持
-A ISTIO_OUTPUT -j ISTIO_REDIRECT # 剩下的都进入 ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001 # 转达到15001 outbond
COMMIT



https://jimmysong.io/blog/sidecar-injection-iptables-and-traffic-routing/
要实现 应用容器(进程) 和 Sidecar容器(进程) 之间的交互。需要完成流量交互的功能。
对于跨容器(进程)的流量交互,主要有以下两种交互模式:
proxyless sidecar 和 Servicemesh 在方式上的差异:暴露 API 还是代理通讯协议。
在 Servicemesh 中,“零侵入”是一个非常强调的特性,为此不惜引入 iptables 等流量劫持方案。“零侵入”在某些特殊场景下会发挥巨大的优势,如旧有应用不做改造的前提下接入 servicemesh。好处自然不言而喻,但零侵入也有自身的限制:客户端必须能发出符合服务器端要求的网络通讯请求,这个过程外部无法插手。
代理模式强调的是 原协议转发,应用进程无感。往往使用操作系统提供的流量劫持功能。
适合于统一的网络协议栈(HTTP),仅实现原协议层面的控制(路由、重试等)。
应用场景:envoy流量代理
发展方向:内核高性能(eBPF)

几乎没有开销是来自代理本身的逻辑。开销是通过注入代理,将网络流量重定向到它,终止连接和启动新的连接而增加的。


应用进程明确转发,协议自由切换,更丰富的应用层语义。往往使用grpc建立跨进程的链接。
适合异构协议栈,支持更丰富的功能,支持协议之上应用层语义的控制。
应用场景:multi runtime
发展方向:grpc proxyless sidecar
gRPC 项目对 xDS API 有很好的支持,也就是说你可以管理 gRPC 工作负载,而不需要同时部署 Envoy sidecar。

ServiceMesh 在微服务领域已经非常流行,越来越多的公司开始在内部落地,ServiceMesh 带来的业务解耦,平滑升级等优势大大提高了中间件的迭代效率。
不过 ServiceMesh 只解决了服务间通讯的需求,而现实中的分布式应用存在更多的需求。而效仿 ServiceMesh 将应用需要的其他分布式能力外移到各种 Sidecar Runtime,这逐渐演变成了一个趋势。

与其依靠多个代理来实现不同的目的(例如网络代理,缓存代理,绑定代理),不如使用一个 Mecha 提供所有这些能力。
Mecha 强调是“提供能力”,而不是通讯代理。
Mecha 和 Micrologic 之间的交互是开放而有 API 标准的,Mecha 和 Micrologic 之间的“协议”体现在 API 上,而不是 TCP 通讯协议。这提供了一个契机:一个统一 Micrologic 和 Mecha 之间通讯方式的契机。

面对千变万化的需求和复杂的应用环境,期望 Sidecar 本身的控制面和数据面来覆盖所有的场景显然是不现实的。强大、全面往往是因为易扩展。
使用类似K8S webhook的机制,将自定义插件注入到控制面中,作为单独的服务执行。
此类扩展可以完全无侵入的实现数据平面的增强。而且 API 的抽象屏蔽了数据平面的实现细节,扩展会具有更好的可移植性;独立进程执行和部署,具备更强的伸缩性。但是 webhook 模 也引入了大量额外的外部调用和数据交互,带来了巨大的性能开销。

WebAssembly,简称 WASM,是一个二进制指令集,最初是跑在浏览器上来解决 JavaScript 的性能问题,但由于它良好的安全性,隔离性以及语言无关性等优秀特性,很快人们便开始让它跑在浏览器之外的地方,随着 WASI 定义的出现,只需要一个 WASM 运行时,就可以让 WASM 文件随处执行。
WASM 字节码不能直接在任何 CPU 架构上执行,但由于它与机器码非常相近,因此能够以非常快的速度被 WASM 引擎(或者也可以称之为 WASM 虚拟机)翻译为对应架构的机器码,获得和机器码相近的性能。
WASM 本身是为 Web 而设计,因此天然具有跨平台支持;同时,通过 WASM 虚拟机的沙箱隔离,也使得执行 WASM 字节码相比于直接执行机器码有更高的安全性。

它是扩展提供的网络代理设计的,目标是支持通过 wasm envoy 的功能。
API 是这样一个函数,如果想要接收一个 http 请求,它需要实现开发三个接口:
Envoy 在可拓展性方面做了两方面的工作:
第一,提供了名为 lua 的特殊扩展,允许控制面通过 xDS 协议动态下发 Lua 脚本并由 Envoy 解释执行。
第二,也是本节的主题,Envoy 引入了 WASM 技术用于开发 Envoy 扩展。

Istio 的扩展机制使用 Proxy-Wasm 应用二进制接口(ABI)规范,提供了一套代理无关的流媒体 API 和实用功能,可以用任何有合适 SDK 的语言来实现。
扩展 Istio 的功能,满足你的特定需求,需要三个步骤:
Mesh: dns拦截(udp)


基于Iptables,拦截http流量。
多个iptables怎么注入?顺序?
实验性质功能,会有影响主链路的风险。
参考 dapr 的 actor 设计。
actor 并不适合集成到SDK中运行,故通过runtime提供actor api。
参考 dapr 的 binding 设计。
binding 作为拓展性质的同外部系统的交互方式,可基于runtime提供弱依赖的交互。
SaaS api 可以作为实验性质,在runtime中进行提供
尽可能少的引入依赖项,Capa作为基础能力的聚合层,不再引入额外的控制面。
使用 configuration 组件作为控制面配置下发方式。
随着云厂商的技术建设,稳定性越来越高,且同 region 的多可用区方案进一步降低了企业集中式故障的概率。但云厂商仍存在中心式服务,如 region 的网络汇聚、统一的结算系统等。
region级别的可用性,还是比较依赖云厂商的可用性,尤其是一些中心式服务,可能可用性不能达到要求。
对于用户使用时间窗口特别集中的业务,对稳定性的要求更高。比如在线素养课就是在有限时间内老师和学生完成知识传授,如果在这段时间内云服务出现故障,学生的学业受到影响,学生和老师需要重新匹配时间才能完成学习计划。健康码、打车软件这些使用集中在早晚高峰的场景也是同理。对于 SLA 要求更高的业务,选择多云多活是必然的趋势。
实际应用场景,尤其是跟人的现实活动挂钩的,对可用性要求比线上服务要更高。
但随着互联网红利不断消退,公司希望实现成本效益最大化,引入更多的供应商也是必选的手段之一。新的供应商,可以用来做数据灾备,或者用于峰值时的弹性计算,也或者按照不同业务进行切分。
不希望鸡蛋放到一个篮子里,厂商锁定可能在议价时处于劣势。而如果可以灵活选择供应商,就可以选择折扣比较多的云服务。
多活架构是用来解决稳定性问题的,但若不能做到多云各自完整的闭环,彼此之间还有千丝万缕的调用依赖,故障率反而会增加。
如果每个云都是单独的服务,并且存在互相调用问题。那么故障的几率是乘积的关系,其实是比单云可用性要更低的。
所以每个服务,都部署到多云上,将会极大地提高SLA。但在这种架构的过渡时期,则必须面对以上问题。
除了部署异构导致的故障率加剧外,脑裂加剧也是一大隐患。多家云厂商中数据存储一般采用主从的方式进行同步,master 所在的主云故障后,需要将另一边云的数据存储提主。但是主云并不是完整挂掉,运维人员无法从控制台登陆,无法将 master 降为 slave。但用户南北流量或者定时任务还在持续请求,主云仍有可能还在处理写入流量。这时候从云的切主,就会导致不可避免的脑裂。脑裂无法简单的进行修复,需要业务研发 case by case 的进行修复,代价巨大。
“脑裂"类似的场景很多,比如数据同步,以及在A云故障时,如何将数据放回B云,是值得思考的问题。
如果做不到多云的对等部署,不得不通过持续演练的方式来应对墒增,保证多云架构的有效性。一旦松懈了,很有可能导致花大精力做的架构升级,但在真正的单云故障面前不起任何作用。另外,在两次演练之间的窗口内,架构能否很好的应对单云故障也是存疑的。
虽然多云号称可以解决单云故障问题,但系统是熵增的,必须通过不断的演练来确保这一点。不然可能真的故障的时候,手忙脚乱,导致各种业务问题。
多云架构有如下优势:
* 灾难恢复,当一家云供应商出现故障时,数据存储可以从另一家云供应商进行恢复。虽然云厂商有多地域,单地域也有多可用区,但还是存在中心化的依赖。这种依赖的故障就会导致整个云的故障。后面提到的故障主要指这种类型的故障。
* 故障转移,当一家云供应商出现故障时,使用另一家云供应商承接服务,实现服务平稳不中断。
* 成本优化,任意的采购只要有了两家及以上供应商,采购方就有了充分选择和议价的能力。
* 避免供应商锁定,单一供应商,除了没有议价能力外,各种依赖也会使得变更极为困难。
* 数据主权,企业提供服务,但产生的数据产权也是归属于服务对象。服务对象既有普通的用户(行权由国家主体代为进行),也有机构。他们对产权的需求,连带的导致企业的云基础设施选择也受其限制。
* 特定服务访问,不同云有自己优势的服务,一般出现在 PaaS 层,如各式各样的数据库、大数据实时方案等。使用多云可以集各家之长。
Good!回看文章。
Capa主要解决应用的混合云问题,依据理论主要为Mecha Runtime设计理念。
Mecha Runtime目前有几个项目比较成熟。
所以了解其他Mecha Runtime项目的设计是很有必要的。
本目录主要为对Mecha Runtime项目的学习和实践。
参考文档:https://docs.dapr.io/
Dapr 允许通过链接一系列中间件组件来定义自定义处理管道。 请求在路由到用户代码之前经过所有已定义的中间件组件,然后在返回到客户机之前,按相反顺序经过已定义的中间件,如下图中所示。

参考文档:http://femas.io/doc/
参考文档:https://mosn.io/layotto/#/zh/README
Capa主要解决应用的混合云问题,涉及到对接适配多种云的底层实现。
Capa本身不提供功能,而是由底层的各种云原生的能力来提供。
所以了解云原生中常用中间件的设计是很有必要的。
本目录主要为对云原生生态中中间件的学习和实践。
跟踪控制器日志:
export APPMESH_SYSTEM_NAMESPACE=appmesh-system
kubectl logs -n "${APPMESH_SYSTEM_NAMESPACE}" -f --since 10s \
$(kubectl get pods -n "${APPMESH_SYSTEM_NAMESPACE}" -o name | grep controller)
尾特使日志:
export APPLICATION_NAMESPACE=<your namespace>
export APPLICATION=<your pod or deployment> # i.e. deploy/my-app
kubectl logs -n "${APPLICATION_NAMESPACE} "${APPLICATION_POD}" envoy -f --since 10s
查看特使配置:
export APPLICATION_NAMESPACE=<your namespace>
export APPLICATION=<your pod>
kubectl port-forward -n "${APPLICATION_NAMESPACE}" \
$(kubectl get pod -n "${APPLICATION_NAMESPACE}" | grep "${APPLICATION}" |awk '{print $1}') \
9901
然后导航到localhost:9901/索引或localhost:9901/config_dump特使配置。
重要参考资料:https://mp.weixin.qq.com/s/t1ppAQfm0cPmqhxEARB03Q
cds_egress_app-mesh_app-1-canary_prod_http_8080::observability_name::cds_egress_app-mesh_app-1-canary_prod_http_8080
cds_egress_app-mesh_app-1-canary_prod_http_8080::default_priority::max_connections::2147483647
cds_egress_app-mesh_app-1-canary_prod_http_8080::default_priority::max_pending_requests::2147483647
cds_egress_app-mesh_app-1-canary_prod_http_8080::default_priority::max_requests::2147483647
cds_egress_app-mesh_app-1-canary_prod_http_8080::default_priority::max_retries::2147483647
cds_egress_app-mesh_app-1-canary_prod_http_8080::high_priority::max_connections::1024
cds_egress_app-mesh_app-1-canary_prod_http_8080::high_priority::max_pending_requests::1024
cds_egress_app-mesh_app-1-canary_prod_http_8080::high_priority::max_requests::1024
cds_egress_app-mesh_app-1-canary_prod_http_8080::high_priority::max_retries::3
cds_egress_app-mesh_app-1-canary_prod_http_8080::added_via_api::true
cds_egress_app-mesh_app-1-canary_prod_http_8080::172.20.215.183:8080::cx_active::0
cds_egress_app-mesh_app-1-canary_prod_http_8080::172.20.215.183:8080::cx_connect_fail::0
cds_egress_app-mesh_app-1-canary_prod_http_8080::172.20.215.183:8080::cx_total::0
cds_egress_app-mesh_app-1-canary_prod_http_8080::172.20.215.183:8080::rq_active::0
cds_egress_app-mesh_app-1-canary_prod_http_8080::172.20.215.183:8080::rq_error::0
cds_egress_app-mesh_app-1-canary_prod_http_8080::172.20.215.183:8080::rq_success::0
cds_egress_app-mesh_app-1-canary_prod_http_8080::172.20.215.183:8080::rq_timeout::0
cds_egress_app-mesh_app-1-canary_prod_http_8080::172.20.215.183:8080::rq_total::0
cds_egress_app-mesh_app-1-canary_prod_http_8080::172.20.215.183:8080::hostname::app-1-canary.prod.svc.cluster.local.
cds_egress_app-mesh_app-1-canary_prod_http_8080::172.20.215.183:8080::health_flags::healthy
cds_egress_app-mesh_app-1-canary_prod_http_8080::172.20.215.183:8080::weight::1
cds_egress_app-mesh_app-1-canary_prod_http_8080::172.20.215.183:8080::region::
cds_egress_app-mesh_app-1-canary_prod_http_8080::172.20.215.183:8080::zone::
cds_egress_app-mesh_app-1-canary_prod_http_8080::172.20.215.183:8080::sub_zone::
cds_egress_app-mesh_app-1-canary_prod_http_8080::172.20.215.183:8080::canary::false
cds_egress_app-mesh_app-1-canary_prod_http_8080::172.20.215.183:8080::priority::0
cds_egress_app-mesh_app-1-canary_prod_http_8080::172.20.215.183:8080::success_rate::-1.0
cds_egress_app-mesh_app-1-canary_prod_http_8080::172.20.215.183:8080::local_origin_success_rate::-1.0
cds_egress_app-mesh_app-2-primary_prod_http_8080::observability_name::cds_egress_app-mesh_app-2-primary_prod_http_8080
cds_egress_app-mesh_app-2-primary_prod_http_8080::default_priority::max_connections::2147483647
cds_egress_app-mesh_app-2-primary_prod_http_8080::default_priority::max_pending_requests::2147483647
cds_egress_app-mesh_app-2-primary_prod_http_8080::default_priority::max_requests::2147483647
cds_egress_app-mesh_app-2-primary_prod_http_8080::default_priority::max_retries::2147483647
cds_egress_app-mesh_app-2-primary_prod_http_8080::high_priority::max_connections::1024
cds_egress_app-mesh_app-2-primary_prod_http_8080::high_priority::max_pending_requests::1024
cds_egress_app-mesh_app-2-primary_prod_http_8080::high_priority::max_requests::1024
cds_egress_app-mesh_app-2-primary_prod_http_8080::high_priority::max_retries::3
cds_egress_app-mesh_app-2-primary_prod_http_8080::added_via_api::true
cds_egress_app-mesh_app-2-primary_prod_http_8080::172.20.123.200:8080::cx_active::15
cds_egress_app-mesh_app-2-primary_prod_http_8080::172.20.123.200:8080::cx_connect_fail::0
cds_egress_app-mesh_app-2-primary_prod_http_8080::172.20.123.200:8080::cx_total::37
cds_egress_app-mesh_app-2-primary_prod_http_8080::172.20.123.200:8080::rq_active::0
cds_egress_app-mesh_app-2-primary_prod_http_8080::172.20.123.200:8080::rq_error::43
cds_egress_app-mesh_app-2-primary_prod_http_8080::172.20.123.200:8080::rq_success::781048
cds_egress_app-mesh_app-2-primary_prod_http_8080::172.20.123.200:8080::rq_timeout::0
cds_egress_app-mesh_app-2-primary_prod_http_8080::172.20.123.200:8080::rq_total::781091
cds_egress_app-mesh_app-2-primary_prod_http_8080::172.20.123.200:8080::hostname::app-2-primary.prod.svc.cluster.local.
cds_egress_app-mesh_app-2-primary_prod_http_8080::172.20.123.200:8080::health_flags::healthy
cds_egress_app-mesh_app-2-primary_prod_http_8080::172.20.123.200:8080::weight::1
cds_egress_app-mesh_app-2-primary_prod_http_8080::172.20.123.200:8080::region::
cds_egress_app-mesh_app-2-primary_prod_http_8080::172.20.123.200:8080::zone::
cds_egress_app-mesh_app-2-primary_prod_http_8080::172.20.123.200:8080::sub_zone::
cds_egress_app-mesh_app-2-primary_prod_http_8080::172.20.123.200:8080::canary::false
cds_egress_app-mesh_app-2-primary_prod_http_8080::172.20.123.200:8080::priority::0
cds_egress_app-mesh_app-2-primary_prod_http_8080::172.20.123.200:8080::success_rate::-1.0
cds_egress_app-mesh_app-2-primary_prod_http_8080::172.20.123.200:8080::local_origin_success_rate::-1.0
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::observability_name::cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::default_priority::max_connections::2147483647
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::default_priority::max_pending_requests::2147483647
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::default_priority::max_requests::2147483647
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::default_priority::max_retries::2147483647
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::high_priority::max_connections::1024
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::high_priority::max_pending_requests::1024
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::high_priority::max_requests::1024
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::high_priority::max_retries::3
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::added_via_api::true
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::127.0.0.1:15001::cx_active::66
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::127.0.0.1:15001::cx_connect_fail::0
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::127.0.0.1:15001::cx_total::514
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::127.0.0.1:15001::rq_active::0
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::127.0.0.1:15001::rq_error::3301
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::127.0.0.1:15001::rq_success::4266126
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::127.0.0.1:15001::rq_timeout::0
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::127.0.0.1:15001::rq_total::4269427
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::127.0.0.1:15001::hostname::
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::127.0.0.1:15001::health_flags::healthy
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::127.0.0.1:15001::weight::1
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::127.0.0.1:15001::region::
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::127.0.0.1:15001::zone::
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::127.0.0.1:15001::sub_zone::
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::127.0.0.1:15001::canary::false
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::127.0.0.1:15001::priority::0
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::127.0.0.1:15001::success_rate::-1.0
cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001::127.0.0.1:15001::local_origin_success_rate::-1.0
cds_egress_app-mesh_app-3-primary_prod_http_8080::observability_name::cds_egress_app-mesh_app-3-primary_prod_http_8080
cds_egress_app-mesh_app-3-primary_prod_http_8080::default_priority::max_connections::2147483647
cds_egress_app-mesh_app-3-primary_prod_http_8080::default_priority::max_pending_requests::2147483647
cds_egress_app-mesh_app-3-primary_prod_http_8080::default_priority::max_requests::2147483647
cds_egress_app-mesh_app-3-primary_prod_http_8080::default_priority::max_retries::2147483647
cds_egress_app-mesh_app-3-primary_prod_http_8080::high_priority::max_connections::1024
cds_egress_app-mesh_app-3-primary_prod_http_8080::high_priority::max_pending_requests::1024
cds_egress_app-mesh_app-3-primary_prod_http_8080::high_priority::max_requests::1024
cds_egress_app-mesh_app-3-primary_prod_http_8080::high_priority::max_retries::3
cds_egress_app-mesh_app-3-primary_prod_http_8080::added_via_api::true
cds_egress_app-mesh_app-3-primary_prod_http_8080::172.20.135.124:8080::cx_active::4
cds_egress_app-mesh_app-3-primary_prod_http_8080::172.20.135.124:8080::cx_connect_fail::0
cds_egress_app-mesh_app-3-primary_prod_http_8080::172.20.135.124:8080::cx_total::428
cds_egress_app-mesh_app-3-primary_prod_http_8080::172.20.135.124:8080::rq_active::0
cds_egress_app-mesh_app-3-primary_prod_http_8080::172.20.135.124:8080::rq_error::419
cds_egress_app-mesh_app-3-primary_prod_http_8080::172.20.135.124:8080::rq_success::33582
cds_egress_app-mesh_app-3-primary_prod_http_8080::172.20.135.124:8080::rq_timeout::0
cds_egress_app-mesh_app-3-primary_prod_http_8080::172.20.135.124:8080::rq_total::34001
cds_egress_app-mesh_app-3-primary_prod_http_8080::172.20.135.124:8080::hostname::app-3-primary.prod.svc.cluster.local.
cds_egress_app-mesh_app-3-primary_prod_http_8080::172.20.135.124:8080::health_flags::healthy
cds_egress_app-mesh_app-3-primary_prod_http_8080::172.20.135.124:8080::weight::1
cds_egress_app-mesh_app-3-primary_prod_http_8080::172.20.135.124:8080::region::
cds_egress_app-mesh_app-3-primary_prod_http_8080::172.20.135.124:8080::zone::
cds_egress_app-mesh_app-3-primary_prod_http_8080::172.20.135.124:8080::sub_zone::
cds_egress_app-mesh_app-3-primary_prod_http_8080::172.20.135.124:8080::canary::false
cds_egress_app-mesh_app-3-primary_prod_http_8080::172.20.135.124:8080::priority::0
cds_egress_app-mesh_app-3-primary_prod_http_8080::172.20.135.124:8080::success_rate::-1.0
cds_egress_app-mesh_app-3-primary_prod_http_8080::172.20.135.124:8080::local_origin_success_rate::-1.0
cds_egress_app-mesh_app-3-canary_prod_http_8080::observability_name::cds_egress_app-mesh_app-3-canary_prod_http_8080
cds_egress_app-mesh_app-3-canary_prod_http_8080::default_priority::max_connections::2147483647
cds_egress_app-mesh_app-3-canary_prod_http_8080::default_priority::max_pending_requests::2147483647
cds_egress_app-mesh_app-3-canary_prod_http_8080::default_priority::max_requests::2147483647
cds_egress_app-mesh_app-3-canary_prod_http_8080::default_priority::max_retries::2147483647
cds_egress_app-mesh_app-3-canary_prod_http_8080::high_priority::max_connections::1024
cds_egress_app-mesh_app-3-canary_prod_http_8080::high_priority::max_pending_requests::1024
cds_egress_app-mesh_app-3-canary_prod_http_8080::high_priority::max_requests::1024
cds_egress_app-mesh_app-3-canary_prod_http_8080::high_priority::max_retries::3
cds_egress_app-mesh_app-3-canary_prod_http_8080::added_via_api::true
cds_egress_app-mesh_app-3-canary_prod_http_8080::172.20.46.186:8080::cx_active::0
cds_egress_app-mesh_app-3-canary_prod_http_8080::172.20.46.186:8080::cx_connect_fail::0
cds_egress_app-mesh_app-3-canary_prod_http_8080::172.20.46.186:8080::cx_total::0
cds_egress_app-mesh_app-3-canary_prod_http_8080::172.20.46.186:8080::rq_active::0
cds_egress_app-mesh_app-3-canary_prod_http_8080::172.20.46.186:8080::rq_error::0
cds_egress_app-mesh_app-3-canary_prod_http_8080::172.20.46.186:8080::rq_success::0
cds_egress_app-mesh_app-3-canary_prod_http_8080::172.20.46.186:8080::rq_timeout::0
cds_egress_app-mesh_app-3-canary_prod_http_8080::172.20.46.186:8080::rq_total::0
cds_egress_app-mesh_app-3-canary_prod_http_8080::172.20.46.186:8080::hostname::app-3-canary.prod.svc.cluster.local.
cds_egress_app-mesh_app-3-canary_prod_http_8080::172.20.46.186:8080::health_flags::healthy
cds_egress_app-mesh_app-3-canary_prod_http_8080::172.20.46.186:8080::weight::1
cds_egress_app-mesh_app-3-canary_prod_http_8080::172.20.46.186:8080::region::
cds_egress_app-mesh_app-3-canary_prod_http_8080::172.20.46.186:8080::zone::
cds_egress_app-mesh_app-3-canary_prod_http_8080::172.20.46.186:8080::sub_zone::
cds_egress_app-mesh_app-3-canary_prod_http_8080::172.20.46.186:8080::canary::false
cds_egress_app-mesh_app-3-canary_prod_http_8080::172.20.46.186:8080::priority::0
cds_egress_app-mesh_app-3-canary_prod_http_8080::172.20.46.186:8080::success_rate::-1.0
cds_egress_app-mesh_app-3-canary_prod_http_8080::172.20.46.186:8080::local_origin_success_rate::-1.0
cds_egress_app-mesh_app-1-primary_prod_http_8080::observability_name::cds_egress_app-mesh_app-1-primary_prod_http_8080
cds_egress_app-mesh_app-1-primary_prod_http_8080::default_priority::max_connections::2147483647
cds_egress_app-mesh_app-1-primary_prod_http_8080::default_priority::max_pending_requests::2147483647
cds_egress_app-mesh_app-1-primary_prod_http_8080::default_priority::max_requests::2147483647
cds_egress_app-mesh_app-1-primary_prod_http_8080::default_priority::max_retries::2147483647
cds_egress_app-mesh_app-1-primary_prod_http_8080::high_priority::max_connections::1024
cds_egress_app-mesh_app-1-primary_prod_http_8080::high_priority::max_pending_requests::1024
cds_egress_app-mesh_app-1-primary_prod_http_8080::high_priority::max_requests::1024
cds_egress_app-mesh_app-1-primary_prod_http_8080::high_priority::max_retries::3
cds_egress_app-mesh_app-1-primary_prod_http_8080::added_via_api::true
cds_egress_app-mesh_app-1-primary_prod_http_8080::172.20.121.13:8080::cx_active::0
cds_egress_app-mesh_app-1-primary_prod_http_8080::172.20.121.13:8080::cx_connect_fail::0
cds_egress_app-mesh_app-1-primary_prod_http_8080::172.20.121.13:8080::cx_total::0
cds_egress_app-mesh_app-1-primary_prod_http_8080::172.20.121.13:8080::rq_active::0
cds_egress_app-mesh_app-1-primary_prod_http_8080::172.20.121.13:8080::rq_error::0
cds_egress_app-mesh_app-1-primary_prod_http_8080::172.20.121.13:8080::rq_success::0
cds_egress_app-mesh_app-1-primary_prod_http_8080::172.20.121.13:8080::rq_timeout::0
cds_egress_app-mesh_app-1-primary_prod_http_8080::172.20.121.13:8080::rq_total::0
cds_egress_app-mesh_app-1-primary_prod_http_8080::172.20.121.13:8080::hostname::app-1-primary.prod.svc.cluster.local.
cds_egress_app-mesh_app-1-primary_prod_http_8080::172.20.121.13:8080::health_flags::healthy
cds_egress_app-mesh_app-1-primary_prod_http_8080::172.20.121.13:8080::weight::1
cds_egress_app-mesh_app-1-primary_prod_http_8080::172.20.121.13:8080::region::
cds_egress_app-mesh_app-1-primary_prod_http_8080::172.20.121.13:8080::zone::
cds_egress_app-mesh_app-1-primary_prod_http_8080::172.20.121.13:8080::sub_zone::
cds_egress_app-mesh_app-1-primary_prod_http_8080::172.20.121.13:8080::canary::false
cds_egress_app-mesh_app-1-primary_prod_http_8080::172.20.121.13:8080::priority::0
cds_egress_app-mesh_app-1-primary_prod_http_8080::172.20.121.13:8080::success_rate::-1.0
cds_egress_app-mesh_app-1-primary_prod_http_8080::172.20.121.13:8080::local_origin_success_rate::-1.0
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::observability_name::cds_egress_app-mesh_app-100035323-canary_prod_http_8080
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::default_priority::max_connections::2147483647
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::default_priority::max_pending_requests::2147483647
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::default_priority::max_requests::2147483647
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::default_priority::max_retries::2147483647
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::high_priority::max_connections::1024
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::high_priority::max_pending_requests::1024
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::high_priority::max_requests::1024
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::high_priority::max_retries::3
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::added_via_api::true
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::172.20.20.112:8080::cx_active::0
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::172.20.20.112:8080::cx_connect_fail::0
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::172.20.20.112:8080::cx_total::0
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::172.20.20.112:8080::rq_active::0
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::172.20.20.112:8080::rq_error::0
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::172.20.20.112:8080::rq_success::0
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::172.20.20.112:8080::rq_timeout::0
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::172.20.20.112:8080::rq_total::0
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::172.20.20.112:8080::hostname::app-100035323-canary.prod.svc.cluster.local.
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::172.20.20.112:8080::health_flags::healthy
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::172.20.20.112:8080::weight::1
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::172.20.20.112:8080::region::
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::172.20.20.112:8080::zone::
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::172.20.20.112:8080::sub_zone::
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::172.20.20.112:8080::canary::false
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::172.20.20.112:8080::priority::0
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::172.20.20.112:8080::success_rate::-1.0
cds_egress_app-mesh_app-100035323-canary_prod_http_8080::172.20.20.112:8080::local_origin_success_rate::-1.0
cds_egress_app-mesh_app-4-canary_prod_http_8080::observability_name::cds_egress_app-mesh_app-4-canary_prod_http_8080
cds_egress_app-mesh_app-4-canary_prod_http_8080::default_priority::max_connections::2147483647
cds_egress_app-mesh_app-4-canary_prod_http_8080::default_priority::max_pending_requests::2147483647
cds_egress_app-mesh_app-4-canary_prod_http_8080::default_priority::max_requests::2147483647
cds_egress_app-mesh_app-4-canary_prod_http_8080::default_priority::max_retries::2147483647
cds_egress_app-mesh_app-4-canary_prod_http_8080::high_priority::max_connections::1024
cds_egress_app-mesh_app-4-canary_prod_http_8080::high_priority::max_pending_requests::1024
cds_egress_app-mesh_app-4-canary_prod_http_8080::high_priority::max_requests::1024
cds_egress_app-mesh_app-4-canary_prod_http_8080::high_priority::max_retries::3
cds_egress_app-mesh_app-4-canary_prod_http_8080::added_via_api::true
cds_egress_app-mesh_app-4-canary_prod_http_8080::172.20.115.83:8080::cx_active::0
cds_egress_app-mesh_app-4-canary_prod_http_8080::172.20.115.83:8080::cx_connect_fail::0
cds_egress_app-mesh_app-4-canary_prod_http_8080::172.20.115.83:8080::cx_total::0
cds_egress_app-mesh_app-4-canary_prod_http_8080::172.20.115.83:8080::rq_active::0
cds_egress_app-mesh_app-4-canary_prod_http_8080::172.20.115.83:8080::rq_error::0
cds_egress_app-mesh_app-4-canary_prod_http_8080::172.20.115.83:8080::rq_success::0
cds_egress_app-mesh_app-4-canary_prod_http_8080::172.20.115.83:8080::rq_timeout::0
cds_egress_app-mesh_app-4-canary_prod_http_8080::172.20.115.83:8080::rq_total::0
cds_egress_app-mesh_app-4-canary_prod_http_8080::172.20.115.83:8080::hostname::app-4-canary.prod.svc.cluster.local.
cds_egress_app-mesh_app-4-canary_prod_http_8080::172.20.115.83:8080::health_flags::healthy
cds_egress_app-mesh_app-4-canary_prod_http_8080::172.20.115.83:8080::weight::1
cds_egress_app-mesh_app-4-canary_prod_http_8080::172.20.115.83:8080::region::
cds_egress_app-mesh_app-4-canary_prod_http_8080::172.20.115.83:8080::zone::
cds_egress_app-mesh_app-4-canary_prod_http_8080::172.20.115.83:8080::sub_zone::
cds_egress_app-mesh_app-4-canary_prod_http_8080::172.20.115.83:8080::canary::false
cds_egress_app-mesh_app-4-canary_prod_http_8080::172.20.115.83:8080::priority::0
cds_egress_app-mesh_app-4-canary_prod_http_8080::172.20.115.83:8080::success_rate::-1.0
cds_egress_app-mesh_app-4-canary_prod_http_8080::172.20.115.83:8080::local_origin_success_rate::-1.0
cds_egress_app-mesh_app-4-primary_prod_http_8080::observability_name::cds_egress_app-mesh_app-4-primary_prod_http_8080
cds_egress_app-mesh_app-4-primary_prod_http_8080::default_priority::max_connections::2147483647
cds_egress_app-mesh_app-4-primary_prod_http_8080::default_priority::max_pending_requests::2147483647
cds_egress_app-mesh_app-4-primary_prod_http_8080::default_priority::max_requests::2147483647
cds_egress_app-mesh_app-4-primary_prod_http_8080::default_priority::max_retries::2147483647
cds_egress_app-mesh_app-4-primary_prod_http_8080::high_priority::max_connections::1024
cds_egress_app-mesh_app-4-primary_prod_http_8080::high_priority::max_pending_requests::1024
cds_egress_app-mesh_app-4-primary_prod_http_8080::high_priority::max_requests::1024
cds_egress_app-mesh_app-4-primary_prod_http_8080::high_priority::max_retries::3
cds_egress_app-mesh_app-4-primary_prod_http_8080::added_via_api::true
cds_egress_app-mesh_app-4-primary_prod_http_8080::172.20.60.241:8080::cx_active::0
cds_egress_app-mesh_app-4-primary_prod_http_8080::172.20.60.241:8080::cx_connect_fail::0
cds_egress_app-mesh_app-4-primary_prod_http_8080::172.20.60.241:8080::cx_total::0
cds_egress_app-mesh_app-4-primary_prod_http_8080::172.20.60.241:8080::rq_active::0
cds_egress_app-mesh_app-4-primary_prod_http_8080::172.20.60.241:8080::rq_error::0
cds_egress_app-mesh_app-4-primary_prod_http_8080::172.20.60.241:8080::rq_success::0
cds_egress_app-mesh_app-4-primary_prod_http_8080::172.20.60.241:8080::rq_timeout::0
cds_egress_app-mesh_app-4-primary_prod_http_8080::172.20.60.241:8080::rq_total::0
cds_egress_app-mesh_app-4-primary_prod_http_8080::172.20.60.241:8080::hostname::app-4-primary.prod.svc.cluster.local.
cds_egress_app-mesh_app-4-primary_prod_http_8080::172.20.60.241:8080::health_flags::healthy
cds_egress_app-mesh_app-4-primary_prod_http_8080::172.20.60.241:8080::weight::1
cds_egress_app-mesh_app-4-primary_prod_http_8080::172.20.60.241:8080::region::
cds_egress_app-mesh_app-4-primary_prod_http_8080::172.20.60.241:8080::zone::
cds_egress_app-mesh_app-4-primary_prod_http_8080::172.20.60.241:8080::sub_zone::
cds_egress_app-mesh_app-4-primary_prod_http_8080::172.20.60.241:8080::canary::false
cds_egress_app-mesh_app-4-primary_prod_http_8080::172.20.60.241:8080::priority::0
cds_egress_app-mesh_app-4-primary_prod_http_8080::172.20.60.241:8080::success_rate::-1.0
cds_egress_app-mesh_app-4-primary_prod_http_8080::172.20.60.241:8080::local_origin_success_rate::-1.0
cds_egress_app-mesh_app-5-canary_prod_http_8080::observability_name::cds_egress_app-mesh_app-5-canary_prod_http_8080
cds_egress_app-mesh_app-5-canary_prod_http_8080::default_priority::max_connections::2147483647
cds_egress_app-mesh_app-5-canary_prod_http_8080::default_priority::max_pending_requests::2147483647
cds_egress_app-mesh_app-5-canary_prod_http_8080::default_priority::max_requests::2147483647
cds_egress_app-mesh_app-5-canary_prod_http_8080::default_priority::max_retries::2147483647
cds_egress_app-mesh_app-5-canary_prod_http_8080::high_priority::max_connections::1024
cds_egress_app-mesh_app-5-canary_prod_http_8080::high_priority::max_pending_requests::1024
cds_egress_app-mesh_app-5-canary_prod_http_8080::high_priority::max_requests::1024
cds_egress_app-mesh_app-5-canary_prod_http_8080::high_priority::max_retries::3
cds_egress_app-mesh_app-5-canary_prod_http_8080::added_via_api::true
cds_egress_app-mesh_app-5-canary_prod_http_8080::172.20.49.28:8080::cx_active::0
cds_egress_app-mesh_app-5-canary_prod_http_8080::172.20.49.28:8080::cx_connect_fail::0
cds_egress_app-mesh_app-5-canary_prod_http_8080::172.20.49.28:8080::cx_total::23
cds_egress_app-mesh_app-5-canary_prod_http_8080::172.20.49.28:8080::rq_active::0
cds_egress_app-mesh_app-5-canary_prod_http_8080::172.20.49.28:8080::rq_error::0
cds_egress_app-mesh_app-5-canary_prod_http_8080::172.20.49.28:8080::rq_success::5364
cds_egress_app-mesh_app-5-canary_prod_http_8080::172.20.49.28:8080::rq_timeout::0
cds_egress_app-mesh_app-5-canary_prod_http_8080::172.20.49.28:8080::rq_total::5364
cds_egress_app-mesh_app-5-canary_prod_http_8080::172.20.49.28:8080::hostname::app-5-canary.prod.svc.cluster.local.
cds_egress_app-mesh_app-5-canary_prod_http_8080::172.20.49.28:8080::health_flags::healthy
cds_egress_app-mesh_app-5-canary_prod_http_8080::172.20.49.28:8080::weight::1
cds_egress_app-mesh_app-5-canary_prod_http_8080::172.20.49.28:8080::region::
cds_egress_app-mesh_app-5-canary_prod_http_8080::172.20.49.28:8080::zone::
cds_egress_app-mesh_app-5-canary_prod_http_8080::172.20.49.28:8080::sub_zone::
cds_egress_app-mesh_app-5-canary_prod_http_8080::172.20.49.28:8080::canary::false
cds_egress_app-mesh_app-5-canary_prod_http_8080::172.20.49.28:8080::priority::0
cds_egress_app-mesh_app-5-canary_prod_http_8080::172.20.49.28:8080::success_rate::-1.0
cds_egress_app-mesh_app-5-canary_prod_http_8080::172.20.49.28:8080::local_origin_success_rate::-1.0
cds_egress_app-mesh_app-5-primary_prod_http_8080::observability_name::cds_egress_app-mesh_app-5-primary_prod_http_8080
cds_egress_app-mesh_app-5-primary_prod_http_8080::default_priority::max_connections::2147483647
cds_egress_app-mesh_app-5-primary_prod_http_8080::default_priority::max_pending_requests::2147483647
cds_egress_app-mesh_app-5-primary_prod_http_8080::default_priority::max_requests::2147483647
cds_egress_app-mesh_app-5-primary_prod_http_8080::default_priority::max_retries::2147483647
cds_egress_app-mesh_app-5-primary_prod_http_8080::high_priority::max_connections::1024
cds_egress_app-mesh_app-5-primary_prod_http_8080::high_priority::max_pending_requests::1024
cds_egress_app-mesh_app-5-primary_prod_http_8080::high_priority::max_requests::1024
cds_egress_app-mesh_app-5-primary_prod_http_8080::high_priority::max_retries::3
cds_egress_app-mesh_app-5-primary_prod_http_8080::added_via_api::true
cds_egress_app-mesh_app-5-primary_prod_http_8080::172.20.95.84:8080::cx_active::44
cds_egress_app-mesh_app-5-primary_prod_http_8080::172.20.95.84:8080::cx_connect_fail::1482
cds_egress_app-mesh_app-5-primary_prod_http_8080::172.20.95.84:8080::cx_total::2312
cds_egress_app-mesh_app-5-primary_prod_http_8080::172.20.95.84:8080::rq_active::0
cds_egress_app-mesh_app-5-primary_prod_http_8080::172.20.95.84:8080::rq_error::6657
cds_egress_app-mesh_app-5-primary_prod_http_8080::172.20.95.84:8080::rq_success::3446132
cds_egress_app-mesh_app-5-primary_prod_http_8080::172.20.95.84:8080::rq_timeout::425
cds_egress_app-mesh_app-5-primary_prod_http_8080::172.20.95.84:8080::rq_total::3446714
cds_egress_app-mesh_app-5-primary_prod_http_8080::172.20.95.84:8080::hostname::app-5-primary.prod.svc.cluster.local.
cds_egress_app-mesh_app-5-primary_prod_http_8080::172.20.95.84:8080::health_flags::healthy
cds_egress_app-mesh_app-5-primary_prod_http_8080::172.20.95.84:8080::weight::1
cds_egress_app-mesh_app-5-primary_prod_http_8080::172.20.95.84:8080::region::
cds_egress_app-mesh_app-5-primary_prod_http_8080::172.20.95.84:8080::zone::
cds_egress_app-mesh_app-5-primary_prod_http_8080::172.20.95.84:8080::sub_zone::
cds_egress_app-mesh_app-5-primary_prod_http_8080::172.20.95.84:8080::canary::false
cds_egress_app-mesh_app-5-primary_prod_http_8080::172.20.95.84:8080::priority::0
cds_egress_app-mesh_app-5-primary_prod_http_8080::172.20.95.84:8080::success_rate::-1.0
cds_egress_app-mesh_app-5-primary_prod_http_8080::172.20.95.84:8080::local_origin_success_rate::-1.0
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::observability_name::cds_egress_app-mesh_app-100035323-primary_prod_http_8080
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::default_priority::max_connections::2147483647
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::default_priority::max_pending_requests::2147483647
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::default_priority::max_requests::2147483647
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::default_priority::max_retries::2147483647
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::high_priority::max_connections::1024
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::high_priority::max_pending_requests::1024
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::high_priority::max_requests::1024
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::high_priority::max_retries::3
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::added_via_api::true
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::172.20.218.166:8080::cx_active::0
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::172.20.218.166:8080::cx_connect_fail::0
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::172.20.218.166:8080::cx_total::0
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::172.20.218.166:8080::rq_active::0
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::172.20.218.166:8080::rq_error::0
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::172.20.218.166:8080::rq_success::0
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::172.20.218.166:8080::rq_timeout::0
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::172.20.218.166:8080::rq_total::0
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::172.20.218.166:8080::hostname::app-100035323-primary.prod.svc.cluster.local.
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::172.20.218.166:8080::health_flags::healthy
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::172.20.218.166:8080::weight::1
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::172.20.218.166:8080::region::
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::172.20.218.166:8080::zone::
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::172.20.218.166:8080::sub_zone::
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::172.20.218.166:8080::canary::false
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::172.20.218.166:8080::priority::0
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::172.20.218.166:8080::success_rate::-1.0
cds_egress_app-mesh_app-100035323-primary_prod_http_8080::172.20.218.166:8080::local_origin_success_rate::-1.0
cds_egress_app-mesh_app-2-canary_prod_http_8080::observability_name::cds_egress_app-mesh_app-2-canary_prod_http_8080
cds_egress_app-mesh_app-2-canary_prod_http_8080::default_priority::max_connections::2147483647
cds_egress_app-mesh_app-2-canary_prod_http_8080::default_priority::max_pending_requests::2147483647
cds_egress_app-mesh_app-2-canary_prod_http_8080::default_priority::max_requests::2147483647
cds_egress_app-mesh_app-2-canary_prod_http_8080::default_priority::max_retries::2147483647
cds_egress_app-mesh_app-2-canary_prod_http_8080::high_priority::max_connections::1024
cds_egress_app-mesh_app-2-canary_prod_http_8080::high_priority::max_pending_requests::1024
cds_egress_app-mesh_app-2-canary_prod_http_8080::high_priority::max_requests::1024
cds_egress_app-mesh_app-2-canary_prod_http_8080::high_priority::max_retries::3
cds_egress_app-mesh_app-2-canary_prod_http_8080::added_via_api::true
cds_egress_app-mesh_app-2-canary_prod_http_8080::172.20.160.19:8080::cx_active::0
cds_egress_app-mesh_app-2-canary_prod_http_8080::172.20.160.19:8080::cx_connect_fail::0
cds_egress_app-mesh_app-2-canary_prod_http_8080::172.20.160.19:8080::cx_total::0
cds_egress_app-mesh_app-2-canary_prod_http_8080::172.20.160.19:8080::rq_active::0
cds_egress_app-mesh_app-2-canary_prod_http_8080::172.20.160.19:8080::rq_error::0
cds_egress_app-mesh_app-2-canary_prod_http_8080::172.20.160.19:8080::rq_success::0
cds_egress_app-mesh_app-2-canary_prod_http_8080::172.20.160.19:8080::rq_timeout::0
cds_egress_app-mesh_app-2-canary_prod_http_8080::172.20.160.19:8080::rq_total::0
cds_egress_app-mesh_app-2-canary_prod_http_8080::172.20.160.19:8080::hostname::app-2-canary.prod.svc.cluster.local.
cds_egress_app-mesh_app-2-canary_prod_http_8080::172.20.160.19:8080::health_flags::healthy
cds_egress_app-mesh_app-2-canary_prod_http_8080::172.20.160.19:8080::weight::1
cds_egress_app-mesh_app-2-canary_prod_http_8080::172.20.160.19:8080::region::
cds_egress_app-mesh_app-2-canary_prod_http_8080::172.20.160.19:8080::zone::
cds_egress_app-mesh_app-2-canary_prod_http_8080::172.20.160.19:8080::sub_zone::
cds_egress_app-mesh_app-2-canary_prod_http_8080::172.20.160.19:8080::canary::false
cds_egress_app-mesh_app-2-canary_prod_http_8080::172.20.160.19:8080::priority::0
cds_egress_app-mesh_app-2-canary_prod_http_8080::172.20.160.19:8080::success_rate::-1.0
cds_egress_app-mesh_app-2-canary_prod_http_8080::172.20.160.19:8080::local_origin_success_rate::-1.0
{
"configs": [
{
"@type": "type.googleapis.com/envoy.admin.v3.BootstrapConfigDump",
"bootstrap": {
"node": {
"id": "mesh/app-mesh/virtualGateway/appmesh-gateway_prod",
"cluster": "mesh/app-mesh/virtualGateway/appmesh-gateway_prod",
"metadata": {
"aws.appmesh.task.interfaces": {
"ipv4": {
"eth0": [
"10.11.71.67/32"
],
"lo": [
"127.0.0.1/8"
]
}
}
},
"user_agent_name": "envoy",
"user_agent_build_version": {
"version": {
"major_number": 1,
"minor_number": 20
},
"metadata": {
"build.type": "RELEASE",
"revision.status": "Modified",
"ssl.version": "BoringSSL",
"build.label": "appmesh.1",
"revision.sha": "3093e5567822eedaa55a7b5da9c35ee6da6b2d1f"
}
},
"extensions": [
{
"name": "envoy.http.original_ip_detection.custom_header",
"category": "envoy.http.original_ip_detection"
},
{
"name": "envoy.http.original_ip_detection.xff",
"category": "envoy.http.original_ip_detection"
},
{
"name": "envoy.watchdog.abort_action",
"category": "envoy.guarddog_actions"
},
{
"name": "envoy.watchdog.profile_action",
"category": "envoy.guarddog_actions"
},
{
"name": "envoy.formatter.metadata",
"category": "envoy.formatter"
},
{
"name": "envoy.formatter.req_without_query",
"category": "envoy.formatter"
},
{
"name": "envoy.dynamic.ot",
"category": "envoy.tracers"
},
{
"name": "envoy.lightstep",
"category": "envoy.tracers"
},
{
"name": "envoy.tracers.datadog",
"category": "envoy.tracers"
},
{
"name": "envoy.tracers.dynamic_ot",
"category": "envoy.tracers"
},
{
"name": "envoy.tracers.lightstep",
"category": "envoy.tracers"
},
{
"name": "envoy.tracers.opencensus",
"category": "envoy.tracers"
},
{
"name": "envoy.tracers.skywalking",
"category": "envoy.tracers"
},
{
"name": "envoy.tracers.xray",
"category": "envoy.tracers"
},
{
"name": "envoy.tracers.zipkin",
"category": "envoy.tracers"
},
{
"name": "envoy.zipkin",
"category": "envoy.tracers"
},
{
"name": "envoy.rbac.matchers.upstream.upstream_ip_port",
"category": "envoy.rbac.matchers"
},
{
"name": "envoy.transport_sockets.alts",
"category": "envoy.transport_sockets.upstream"
},
{
"name": "envoy.transport_sockets.quic",
"category": "envoy.transport_sockets.upstream"
},
{
"name": "envoy.transport_sockets.raw_buffer",
"category": "envoy.transport_sockets.upstream"
},
{
"name": "envoy.transport_sockets.starttls",
"category": "envoy.transport_sockets.upstream"
},
{
"name": "envoy.transport_sockets.tap",
"category": "envoy.transport_sockets.upstream"
},
{
"name": "envoy.transport_sockets.tls",
"category": "envoy.transport_sockets.upstream"
},
{
"name": "envoy.transport_sockets.upstream_proxy_protocol",
"category": "envoy.transport_sockets.upstream"
},
{
"name": "raw_buffer",
"category": "envoy.transport_sockets.upstream"
},
{
"name": "starttls",
"category": "envoy.transport_sockets.upstream"
},
{
"name": "tls",
"category": "envoy.transport_sockets.upstream"
},
{
"name": "envoy.matching.matchers.consistent_hashing",
"category": "envoy.matching.input_matchers"
},
{
"name": "envoy.matching.matchers.ip",
"category": "envoy.matching.input_matchers"
},
{
"name": "envoy.health_checkers.redis",
"category": "envoy.health_checkers"
},
{
"name": "envoy.bandwidth_limit",
"category": "envoy.filters.http"
},
{
"name": "envoy.buffer",
"category": "envoy.filters.http"
},
{
"name": "envoy.cors",
"category": "envoy.filters.http"
},
{
"name": "envoy.csrf",
"category": "envoy.filters.http"
},
{
"name": "envoy.ext_authz",
"category": "envoy.filters.http"
},
{
"name": "envoy.ext_proc",
"category": "envoy.filters.http"
},
{
"name": "envoy.fault",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.adaptive_concurrency",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.admission_control",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.alternate_protocols_cache",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.aws_lambda",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.aws_request_signing",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.bandwidth_limit",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.buffer",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.cache",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.cdn_loop",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.composite",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.compressor",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.cors",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.csrf",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.decompressor",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.dynamic_forward_proxy",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.dynamo",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.ext_authz",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.ext_proc",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.fault",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.grpc_http1_bridge",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.grpc_http1_reverse_bridge",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.grpc_json_transcoder",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.grpc_stats",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.grpc_web",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.header_to_metadata",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.health_check",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.ip_tagging",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.jwt_authn",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.local_ratelimit",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.lua",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.oauth2",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.on_demand",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.original_src",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.ratelimit",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.rbac",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.router",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.set_metadata",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.tap",
"category": "envoy.filters.http"
},
{
"name": "envoy.filters.http.wasm",
"category": "envoy.filters.http"
},
{
"name": "envoy.grpc_http1_bridge",
"category": "envoy.filters.http"
},
{
"name": "envoy.grpc_json_transcoder",
"category": "envoy.filters.http"
},
{
"name": "envoy.grpc_web",
"category": "envoy.filters.http"
},
{
"name": "envoy.health_check",
"category": "envoy.filters.http"
},
{
"name": "envoy.http_dynamo_filter",
"category": "envoy.filters.http"
},
{
"name": "envoy.ip_tagging",
"category": "envoy.filters.http"
},
{
"name": "envoy.local_rate_limit",
"category": "envoy.filters.http"
},
{
"name": "envoy.lua",
"category": "envoy.filters.http"
},
{
"name": "envoy.rate_limit",
"category": "envoy.filters.http"
},
{
"name": "envoy.router",
"category": "envoy.filters.http"
},
{
"name": "match-wrapper",
"category": "envoy.filters.http"
},
{
"name": "envoy.transport_sockets.alts",
"category": "envoy.transport_sockets.downstream"
},
{
"name": "envoy.transport_sockets.quic",
"category": "envoy.transport_sockets.downstream"
},
{
"name": "envoy.transport_sockets.raw_buffer",
"category": "envoy.transport_sockets.downstream"
},
{
"name": "envoy.transport_sockets.starttls",
"category": "envoy.transport_sockets.downstream"
},
{
"name": "envoy.transport_sockets.tap",
"category": "envoy.transport_sockets.downstream"
},
{
"name": "envoy.transport_sockets.tls",
"category": "envoy.transport_sockets.downstream"
},
{
"name": "raw_buffer",
"category": "envoy.transport_sockets.downstream"
},
{
"name": "starttls",
"category": "envoy.transport_sockets.downstream"
},
{
"name": "tls",
"category": "envoy.transport_sockets.downstream"
},
{
"name": "envoy.bootstrap.wasm",
"category": "envoy.bootstrap"
},
{
"name": "envoy.extensions.network.socket_interface.default_socket_interface",
"category": "envoy.bootstrap"
},
{
"name": "envoy.matching.common_inputs.environment_variable",
"category": "envoy.matching.common_inputs"
},
{
"name": "request-headers",
"category": "envoy.matching.http.input"
},
{
"name": "request-trailers",
"category": "envoy.matching.http.input"
},
{
"name": "response-headers",
"category": "envoy.matching.http.input"
},
{
"name": "response-trailers",
"category": "envoy.matching.http.input"
},
{
"name": "envoy.quic.proof_source.filter_chain",
"category": "envoy.quic.proof_source"
},
{
"name": "envoy.access_loggers.file",
"category": "envoy.access_loggers"
},
{
"name": "envoy.access_loggers.http_grpc",
"category": "envoy.access_loggers"
},
{
"name": "envoy.access_loggers.open_telemetry",
"category": "envoy.access_loggers"
},
{
"name": "envoy.access_loggers.stderr",
"category": "envoy.access_loggers"
},
{
"name": "envoy.access_loggers.stdout",
"category": "envoy.access_loggers"
},
{
"name": "envoy.access_loggers.tcp_grpc",
"category": "envoy.access_loggers"
},
{
"name": "envoy.access_loggers.wasm",
"category": "envoy.access_loggers"
},
{
"name": "envoy.file_access_log",
"category": "envoy.access_loggers"
},
{
"name": "envoy.http_grpc_access_log",
"category": "envoy.access_loggers"
},
{
"name": "envoy.open_telemetry_access_log",
"category": "envoy.access_loggers"
},
{
"name": "envoy.stderr_access_log",
"category": "envoy.access_loggers"
},
{
"name": "envoy.stdout_access_log",
"category": "envoy.access_loggers"
},
{
"name": "envoy.tcp_grpc_access_log",
"category": "envoy.access_loggers"
},
{
"name": "envoy.wasm_access_log",
"category": "envoy.access_loggers"
},
{
"name": "envoy.cluster.eds",
"category": "envoy.clusters"
},
{
"name": "envoy.cluster.logical_dns",
"category": "envoy.clusters"
},
{
"name": "envoy.cluster.original_dst",
"category": "envoy.clusters"
},
{
"name": "envoy.cluster.static",
"category": "envoy.clusters"
},
{
"name": "envoy.cluster.strict_dns",
"category": "envoy.clusters"
},
{
"name": "envoy.clusters.aggregate",
"category": "envoy.clusters"
},
{
"name": "envoy.clusters.dynamic_forward_proxy",
"category": "envoy.clusters"
},
{
"name": "envoy.clusters.redis",
"category": "envoy.clusters"
},
{
"name": "envoy.tls.cert_validator.default",
"category": "envoy.tls.cert_validator"
},
{
"name": "envoy.tls.cert_validator.spiffe",
"category": "envoy.tls.cert_validator"
},
{
"name": "envoy.filters.thrift.rate_limit",
"category": "envoy.thrift_proxy.filters"
},
{
"name": "envoy.filters.thrift.router",
"category": "envoy.thrift_proxy.filters"
},
{
"name": "envoy.filters.listener.http_inspector",
"category": "envoy.filters.listener"
},
{
"name": "envoy.filters.listener.original_dst",
"category": "envoy.filters.listener"
},
{
"name": "envoy.filters.listener.original_src",
"category": "envoy.filters.listener"
},
{
"name": "envoy.filters.listener.proxy_protocol",
"category": "envoy.filters.listener"
},
{
"name": "envoy.filters.listener.tls_inspector",
"category": "envoy.filters.listener"
},
{
"name": "envoy.listener.http_inspector",
"category": "envoy.filters.listener"
},
{
"name": "envoy.listener.original_dst",
"category": "envoy.filters.listener"
},
{
"name": "envoy.listener.original_src",
"category": "envoy.filters.listener"
},
{
"name": "envoy.listener.proxy_protocol",
"category": "envoy.filters.listener"
},
{
"name": "envoy.listener.tls_inspector",
"category": "envoy.filters.listener"
},
{
"name": "dubbo.hessian2",
"category": "envoy.dubbo_proxy.serializers"
},
{
"name": "envoy.retry_priorities.previous_priorities",
"category": "envoy.retry_priorities"
},
{
"name": "envoy.rate_limit_descriptors.expr",
"category": "envoy.rate_limit_descriptors"
},
{
"name": "envoy.key_value.file_based",
"category": "envoy.common.key_value"
},
{
"name": "envoy.retry_host_predicates.omit_canary_hosts",
"category": "envoy.retry_host_predicates"
},
{
"name": "envoy.retry_host_predicates.omit_host_metadata",
"category": "envoy.retry_host_predicates"
},
{
"name": "envoy.retry_host_predicates.previous_hosts",
"category": "envoy.retry_host_predicates"
},
{
"name": "envoy.filters.udp.dns_filter",
"category": "envoy.filters.udp_listener"
},
{
"name": "envoy.filters.udp_listener.udp_proxy",
"category": "envoy.filters.udp_listener"
},
{
"name": "preserve_case",
"category": "envoy.http.stateful_header_formatters"
},
{
"name": "envoy.wasm.runtime.null",
"category": "envoy.wasm.runtime"
},
{
"name": "envoy.wasm.runtime.v8",
"category": "envoy.wasm.runtime"
},
{
"name": "envoy.request_id.uuid",
"category": "envoy.request_id"
},
{
"name": "envoy.filters.dubbo.router",
"category": "envoy.dubbo_proxy.filters"
},
{
"name": "envoy.extensions.upstreams.http.v3.HttpProtocolOptions",
"category": "envoy.upstream_options"
},
{
"name": "envoy.upstreams.http.http_protocol_options",
"category": "envoy.upstream_options"
},
{
"name": "composite-action",
"category": "envoy.matching.action"
},
{
"name": "skip",
"category": "envoy.matching.action"
},
{
"name": "default",
"category": "envoy.dubbo_proxy.route_matchers"
},
{
"name": "dubbo",
"category": "envoy.dubbo_proxy.protocols"
},
{
"name": "envoy.compression.brotli.decompressor",
"category": "envoy.compression.decompressor"
},
{
"name": "envoy.compression.gzip.decompressor",
"category": "envoy.compression.decompressor"
},
{
"name": "auto",
"category": "envoy.thrift_proxy.protocols"
},
{
"name": "binary",
"category": "envoy.thrift_proxy.protocols"
},
{
"name": "binary/non-strict",
"category": "envoy.thrift_proxy.protocols"
},
{
"name": "compact",
"category": "envoy.thrift_proxy.protocols"
},
{
"name": "twitter",
"category": "envoy.thrift_proxy.protocols"
},
{
"name": "envoy.grpc_credentials.aws_iam",
"category": "envoy.grpc_credentials"
},
{
"name": "envoy.grpc_credentials.default",
"category": "envoy.grpc_credentials"
},
{
"name": "envoy.grpc_credentials.file_based_metadata",
"category": "envoy.grpc_credentials"
},
{
"name": "envoy.extensions.http.cache.simple",
"category": "envoy.http.cache"
},
{
"name": "envoy.internal_redirect_predicates.allow_listed_routes",
"category": "envoy.internal_redirect_predicates"
},
{
"name": "envoy.internal_redirect_predicates.previous_routes",
"category": "envoy.internal_redirect_predicates"
},
{
"name": "envoy.internal_redirect_predicates.safe_cross_scheme",
"category": "envoy.internal_redirect_predicates"
},
{
"name": "envoy.compression.brotli.compressor",
"category": "envoy.compression.compressor"
},
{
"name": "envoy.compression.gzip.compressor",
"category": "envoy.compression.compressor"
},
{
"name": "auto",
"category": "envoy.thrift_proxy.transports"
},
{
"name": "framed",
"category": "envoy.thrift_proxy.transports"
},
{
"name": "header",
"category": "envoy.thrift_proxy.transports"
},
{
"name": "unframed",
"category": "envoy.thrift_proxy.transports"
},
{
"name": "envoy.ip",
"category": "envoy.resolvers"
},
{
"name": "envoy.dog_statsd",
"category": "envoy.stats_sinks"
},
{
"name": "envoy.graphite_statsd",
"category": "envoy.stats_sinks"
},
{
"name": "envoy.metrics_service",
"category": "envoy.stats_sinks"
},
{
"name": "envoy.stat_sinks.dog_statsd",
"category": "envoy.stats_sinks"
},
{
"name": "envoy.stat_sinks.graphite_statsd",
"category": "envoy.stats_sinks"
},
{
"name": "envoy.stat_sinks.hystrix",
"category": "envoy.stats_sinks"
},
{
"name": "envoy.stat_sinks.metrics_service",
"category": "envoy.stats_sinks"
},
{
"name": "envoy.stat_sinks.statsd",
"category": "envoy.stats_sinks"
},
{
"name": "envoy.stat_sinks.wasm",
"category": "envoy.stats_sinks"
},
{
"name": "envoy.statsd",
"category": "envoy.stats_sinks"
},
{
"name": "envoy.resource_monitors.fixed_heap",
"category": "envoy.resource_monitors"
},
{
"name": "envoy.resource_monitors.injected_resource",
"category": "envoy.resource_monitors"
},
{
"name": "envoy.quic.crypto_stream.server.quiche",
"category": "envoy.quic.server.crypto_stream"
},
{
"name": "envoy.filters.connection_pools.tcp.generic",
"category": "envoy.upstreams"
},
{
"name": "envoy.client_ssl_auth",
"category": "envoy.filters.network"
},
{
"name": "envoy.echo",
"category": "envoy.filters.network"
},
{
"name": "envoy.ext_authz",
"category": "envoy.filters.network"
},
{
"name": "envoy.filters.network.client_ssl_auth",
"category": "envoy.filters.network"
},
{
"name": "envoy.filters.network.connection_limit",
"category": "envoy.filters.network"
},
{
"name": "envoy.filters.network.direct_response",
"category": "envoy.filters.network"
},
{
"name": "envoy.filters.network.dubbo_proxy",
"category": "envoy.filters.network"
},
{
"name": "envoy.filters.network.echo",
"category": "envoy.filters.network"
},
{
"name": "envoy.filters.network.ext_authz",
"category": "envoy.filters.network"
},
{
"name": "envoy.filters.network.http_connection_manager",
"category": "envoy.filters.network"
},
{
"name": "envoy.filters.network.local_ratelimit",
"category": "envoy.filters.network"
},
{
"name": "envoy.filters.network.mongo_proxy",
"category": "envoy.filters.network"
},
{
"name": "envoy.filters.network.ratelimit",
"category": "envoy.filters.network"
},
{
"name": "envoy.filters.network.rbac",
"category": "envoy.filters.network"
},
{
"name": "envoy.filters.network.redis_proxy",
"category": "envoy.filters.network"
},
{
"name": "envoy.filters.network.sni_cluster",
"category": "envoy.filters.network"
},
{
"name": "envoy.filters.network.sni_dynamic_forward_proxy",
"category": "envoy.filters.network"
},
{
"name": "envoy.filters.network.tcp_proxy",
"category": "envoy.filters.network"
},
{
"name": "envoy.filters.network.thrift_proxy",
"category": "envoy.filters.network"
},
{
"name": "envoy.filters.network.wasm",
"category": "envoy.filters.network"
},
{
"name": "envoy.filters.network.zookeeper_proxy",
"category": "envoy.filters.network"
},
{
"name": "envoy.http_connection_manager",
"category": "envoy.filters.network"
},
{
"name": "envoy.mongo_proxy",
"category": "envoy.filters.network"
},
{
"name": "envoy.ratelimit",
"category": "envoy.filters.network"
},
{
"name": "envoy.redis_proxy",
"category": "envoy.filters.network"
},
{
"name": "envoy.tcp_proxy",
"category": "envoy.filters.network"
}
]
},
"dynamic_resources": {
"lds_config": {
"ads": {},
"initial_fetch_timeout": "0s",
"resource_api_version": "V3"
},
"cds_config": {
"ads": {},
"initial_fetch_timeout": "0s",
"resource_api_version": "V3"
},
"ads_config": {
"api_type": "GRPC",
"grpc_services": [
{
"google_grpc": {
"target_uri": "appmesh-envoy-management.ap-southeast-1.amazonaws.com:443",
"channel_credentials": {
"ssl_credentials": {
"root_certs": {
"filename": "/etc/pki/tls/cert.pem"
}
}
},
"call_credentials": [
{
"from_plugin": {
"name": "envoy.grpc_credentials.aws_iam",
"typed_config": {
"@type": "type.googleapis.com/envoy.config.grpc_credential.v3.AwsIamConfig",
"service_name": "appmesh",
"region": "ap-southeast-1"
}
}
}
],
"stat_prefix": "ads",
"credentials_factory_name": "envoy.grpc_credentials.aws_iam",
"channel_args": {
"args": {
"grpc.keepalive_time_ms": {
"int_value": "10000"
},
"grpc.http2.max_pings_without_data": {
"int_value": "0"
},
"grpc.keepalive_timeout_ms": {
"int_value": "20000"
}
}
}
}
}
],
"transport_api_version": "V3"
}
},
"cluster_manager": {
"outlier_detection": {
"event_log_path": "/s/stdout"
}
},
"admin": {
"access_log_path": "/tmp/envoy_admin_access.log",
"address": {
"socket_address": {
"address": "0.0.0.0",
"port_value": 9901
}
}
},
"layered_runtime": {
"layers": [
{
"name": "static_layer_0",
"static_layer": {
"envoy.reloadable_features.enable_deprecated_v2_api": true,
"envoy.features.enable_all_deprecated_features": true,
"re2.max_program_size.error_level": 1000,
"envoy.reloadable_features.http_set_tracing_decision_in_request_id": true
}
}
]
}
},
"last_updated": "2022-05-21T12:08:39.717Z"
},
{
"@type": "type.googleapis.com/envoy.admin.v3.ClustersConfigDump",
"version_info": "822736081:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440",
"dynamic_active_clusters": [
{
"version_info": "822736081:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440",
"cluster": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cds_egress_app-mesh_app-4-canary_prod_http_8080",
"type": "LOGICAL_DNS",
"connect_timeout": "1s",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 2147483647,
"max_pending_requests": 2147483647,
"max_requests": 2147483647,
"max_retries": 2147483647
}
]
},
"metadata": {
"filter_metadata": {
"aws.appmesh.aggregate_stats": {
"config": "{\"tags\":[{\"key\":\"TargetVirtualNode\",\"value\":\"app-4-canary_prod\"}]}"
}
}
},
"load_assignment": {
"cluster_name": "cds_egress_app-mesh_app-4-canary_prod_http_8080",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "app-4-canary.prod.svc.cluster.local.",
"port_value": 8080
}
}
}
}
]
}
]
}
},
"last_updated": "2022-05-21T12:08:39.918Z"
},
{
"version_info": "822736081:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440",
"cluster": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cds_egress_app-mesh_app-4-primary_prod_http_8080",
"type": "LOGICAL_DNS",
"connect_timeout": "1s",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 2147483647,
"max_pending_requests": 2147483647,
"max_requests": 2147483647,
"max_retries": 2147483647
}
]
},
"metadata": {
"filter_metadata": {
"aws.appmesh.aggregate_stats": {
"config": "{\"tags\":[{\"key\":\"TargetVirtualNode\",\"value\":\"app-4-primary_prod\"}]}"
}
}
},
"load_assignment": {
"cluster_name": "cds_egress_app-mesh_app-4-primary_prod_http_8080",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "app-4-primary.prod.svc.cluster.local.",
"port_value": 8080
}
}
}
}
]
}
]
}
},
"last_updated": "2022-05-21T12:08:39.917Z"
},
{
"version_info": "822736081:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440",
"cluster": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cds_egress_app-mesh_app-3-canary_prod_http_8080",
"type": "LOGICAL_DNS",
"connect_timeout": "1s",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 2147483647,
"max_pending_requests": 2147483647,
"max_requests": 2147483647,
"max_retries": 2147483647
}
]
},
"metadata": {
"filter_metadata": {
"aws.appmesh.aggregate_stats": {
"config": "{\"tags\":[{\"key\":\"TargetVirtualNode\",\"value\":\"app-3-canary_prod\"}]}"
}
}
},
"load_assignment": {
"cluster_name": "cds_egress_app-mesh_app-3-canary_prod_http_8080",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "app-3-canary.prod.svc.cluster.local.",
"port_value": 8080
}
}
}
}
]
}
]
}
},
"last_updated": "2022-05-21T12:08:39.913Z"
},
{
"version_info": "822736081:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440",
"cluster": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cds_egress_app-mesh_app-3-primary_prod_http_8080",
"type": "LOGICAL_DNS",
"connect_timeout": "1s",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 2147483647,
"max_pending_requests": 2147483647,
"max_requests": 2147483647,
"max_retries": 2147483647
}
]
},
"metadata": {
"filter_metadata": {
"aws.appmesh.aggregate_stats": {
"config": "{\"tags\":[{\"key\":\"TargetVirtualNode\",\"value\":\"app-3-primary_prod\"}]}"
}
}
},
"load_assignment": {
"cluster_name": "cds_egress_app-mesh_app-3-primary_prod_http_8080",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "app-3-primary.prod.svc.cluster.local.",
"port_value": 8080
}
}
}
}
]
}
]
}
},
"last_updated": "2022-05-21T12:08:39.912Z"
},
{
"version_info": "822736081:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440",
"cluster": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cds_egress_app-mesh_app-100035323-canary_prod_http_8080",
"type": "LOGICAL_DNS",
"connect_timeout": "1s",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 2147483647,
"max_pending_requests": 2147483647,
"max_requests": 2147483647,
"max_retries": 2147483647
}
]
},
"metadata": {
"filter_metadata": {
"aws.appmesh.aggregate_stats": {
"config": "{\"tags\":[{\"key\":\"TargetVirtualNode\",\"value\":\"app-100035323-canary_prod\"}]}"
}
}
},
"load_assignment": {
"cluster_name": "cds_egress_app-mesh_app-100035323-canary_prod_http_8080",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "app-100035323-canary.prod.svc.cluster.local.",
"port_value": 8080
}
}
}
}
]
}
]
}
},
"last_updated": "2022-05-21T12:08:39.915Z"
},
{
"version_info": "822736081:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440",
"cluster": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cds_egress_app-mesh_app-100035323-primary_prod_http_8080",
"type": "LOGICAL_DNS",
"connect_timeout": "1s",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 2147483647,
"max_pending_requests": 2147483647,
"max_requests": 2147483647,
"max_retries": 2147483647
}
]
},
"metadata": {
"filter_metadata": {
"aws.appmesh.aggregate_stats": {
"config": "{\"tags\":[{\"key\":\"TargetVirtualNode\",\"value\":\"app-100035323-primary_prod\"}]}"
}
}
},
"load_assignment": {
"cluster_name": "cds_egress_app-mesh_app-100035323-primary_prod_http_8080",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "app-100035323-primary.prod.svc.cluster.local.",
"port_value": 8080
}
}
}
}
]
}
]
}
},
"last_updated": "2022-05-21T12:08:39.914Z"
},
{
"version_info": "822736081:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440",
"cluster": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cds_egress_app-mesh_app-1-canary_prod_http_8080",
"type": "LOGICAL_DNS",
"connect_timeout": "1s",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 2147483647,
"max_pending_requests": 2147483647,
"max_requests": 2147483647,
"max_retries": 2147483647
}
]
},
"metadata": {
"filter_metadata": {
"aws.appmesh.aggregate_stats": {
"config": "{\"tags\":[{\"key\":\"TargetVirtualNode\",\"value\":\"app-1-canary_prod\"}]}"
}
}
},
"load_assignment": {
"cluster_name": "cds_egress_app-mesh_app-1-canary_prod_http_8080",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "app-1-canary.prod.svc.cluster.local.",
"port_value": 8080
}
}
}
}
]
}
]
}
},
"last_updated": "2022-05-21T12:08:39.910Z"
},
{
"version_info": "822736081:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440",
"cluster": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cds_egress_app-mesh_app-1-primary_prod_http_8080",
"type": "LOGICAL_DNS",
"connect_timeout": "1s",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 2147483647,
"max_pending_requests": 2147483647,
"max_requests": 2147483647,
"max_retries": 2147483647
}
]
},
"metadata": {
"filter_metadata": {
"aws.appmesh.aggregate_stats": {
"config": "{\"tags\":[{\"key\":\"TargetVirtualNode\",\"value\":\"app-1-primary_prod\"}]}"
}
}
},
"load_assignment": {
"cluster_name": "cds_egress_app-mesh_app-1-primary_prod_http_8080",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "app-1-primary.prod.svc.cluster.local.",
"port_value": 8080
}
}
}
}
]
}
]
}
},
"last_updated": "2022-05-21T12:08:39.911Z"
},
{
"version_info": "822736081:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440",
"cluster": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cds_egress_app-mesh_app-5-canary_prod_http_8080",
"type": "LOGICAL_DNS",
"connect_timeout": "1s",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 2147483647,
"max_pending_requests": 2147483647,
"max_requests": 2147483647,
"max_retries": 2147483647
}
]
},
"metadata": {
"filter_metadata": {
"aws.appmesh.aggregate_stats": {
"config": "{\"tags\":[{\"key\":\"TargetVirtualNode\",\"value\":\"app-5-canary_prod\"}]}"
}
}
},
"load_assignment": {
"cluster_name": "cds_egress_app-mesh_app-5-canary_prod_http_8080",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "app-5-canary.prod.svc.cluster.local.",
"port_value": 8080
}
}
}
}
]
}
]
}
},
"last_updated": "2022-05-21T12:08:39.909Z"
},
{
"version_info": "822736081:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440",
"cluster": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cds_egress_app-mesh_app-5-primary_prod_http_8080",
"type": "LOGICAL_DNS",
"connect_timeout": "1s",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 2147483647,
"max_pending_requests": 2147483647,
"max_requests": 2147483647,
"max_retries": 2147483647
}
]
},
"metadata": {
"filter_metadata": {
"aws.appmesh.aggregate_stats": {
"config": "{\"tags\":[{\"key\":\"TargetVirtualNode\",\"value\":\"app-5-primary_prod\"}]}"
}
}
},
"load_assignment": {
"cluster_name": "cds_egress_app-mesh_app-5-primary_prod_http_8080",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "app-5-primary.prod.svc.cluster.local.",
"port_value": 8080
}
}
}
}
]
}
]
}
},
"last_updated": "2022-05-21T12:08:39.908Z"
},
{
"version_info": "822736081:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440",
"cluster": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cds_egress_app-mesh_app-2-canary_prod_http_8080",
"type": "LOGICAL_DNS",
"connect_timeout": "1s",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 2147483647,
"max_pending_requests": 2147483647,
"max_requests": 2147483647,
"max_retries": 2147483647
}
]
},
"metadata": {
"filter_metadata": {
"aws.appmesh.aggregate_stats": {
"config": "{\"tags\":[{\"key\":\"TargetVirtualNode\",\"value\":\"app-2-canary_prod\"}]}"
}
}
},
"load_assignment": {
"cluster_name": "cds_egress_app-mesh_app-2-canary_prod_http_8080",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "app-2-canary.prod.svc.cluster.local.",
"port_value": 8080
}
}
}
}
]
}
]
}
},
"last_updated": "2022-05-21T12:08:39.917Z"
},
{
"version_info": "822736081:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440",
"cluster": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cds_egress_app-mesh_app-2-primary_prod_http_8080",
"type": "LOGICAL_DNS",
"connect_timeout": "1s",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 2147483647,
"max_pending_requests": 2147483647,
"max_requests": 2147483647,
"max_retries": 2147483647
}
]
},
"metadata": {
"filter_metadata": {
"aws.appmesh.aggregate_stats": {
"config": "{\"tags\":[{\"key\":\"TargetVirtualNode\",\"value\":\"app-2-primary_prod\"}]}"
}
}
},
"load_assignment": {
"cluster_name": "cds_egress_app-mesh_app-2-primary_prod_http_8080",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "app-2-primary.prod.svc.cluster.local.",
"port_value": 8080
}
}
}
}
]
}
]
}
},
"last_updated": "2022-05-21T12:08:39.916Z"
},
{
"version_info": "822736081:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440",
"cluster": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001",
"type": "STATIC",
"connect_timeout": "0.250s",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 2147483647,
"max_pending_requests": 2147483647,
"max_requests": 2147483647,
"max_retries": 2147483647
}
]
},
"load_assignment": {
"cluster_name": "cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "127.0.0.1",
"port_value": 15001
}
}
}
}
]
}
]
}
},
"last_updated": "2022-05-21T12:08:39.920Z"
}
]
},
{
"@type": "type.googleapis.com/envoy.admin.v3.ListenersConfigDump",
"version_info": "822736081:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440",
"dynamic_listeners": [
{
"name": "lds_ingress_0.0.0.0_8088",
"active_state": {
"version_info": "822736081:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440",
"listener": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "lds_ingress_0.0.0.0_8088",
"address": {
"socket_address": {
"address": "0.0.0.0",
"port_value": 8088
}
},
"filter_chains": [
{
"filter_chain_match": {
"destination_port": 8088
},
"filters": [
{
"name": "envoy.filters.network.http_connection_manager",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"stat_prefix": "ingress",
"rds": {
"config_source": {
"ads": {},
"resource_api_version": "V3"
},
"route_config_name": "rds_ingress_http_8088"
},
"http_filters": [
{
"name": "envoy.filters.http.router"
}
],
"tracing": {},
"access_log": [
{
"name": "envoy.access_loggers.file",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
"path": "/dev/stdout"
}
}
],
"upgrade_configs": [
{
"upgrade_type": "websocket",
"enabled": true
}
],
"normalize_path": true,
"merge_slashes": true,
"path_with_escaped_slashes_action": "KEEP_UNCHANGED"
}
}
]
}
],
"metadata": {
"filter_metadata": {
"aws.appmesh.aggregate_stats": {
"config": "{\"tags\":[]}"
}
}
},
"listener_filters": [
{
"name": "envoy.filters.listener.tls_inspector"
}
],
"traffic_direction": "INBOUND"
},
"last_updated": "2022-05-21T12:08:39.935Z"
}
},
{
"name": "lds_egress_0.0.0.0_15001",
"active_state": {
"version_info": "822736081:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440",
"listener": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "lds_egress_0.0.0.0_15001",
"address": {
"socket_address": {
"address": "0.0.0.0",
"port_value": 15001
}
},
"filter_chains": [
{
"filter_chain_match": {
"destination_port": 15001
},
"filters": [
{
"name": "envoy.filters.network.http_connection_manager",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"stat_prefix": "egress",
"rds": {
"config_source": {
"ads": {},
"resource_api_version": "V3"
},
"route_config_name": "rds_egress_http_15001"
},
"http_filters": [
{
"name": "envoy.filters.http.router"
}
],
"tracing": {},
"access_log": [
{
"name": "envoy.access_loggers.file",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
"path": "/dev/stdout"
}
}
],
"upgrade_configs": [
{
"upgrade_type": "websocket",
"enabled": true
}
],
"normalize_path": true,
"merge_slashes": true,
"path_with_escaped_slashes_action": "KEEP_UNCHANGED"
}
}
]
}
],
"metadata": {
"filter_metadata": {
"aws.appmesh.aggregate_stats": {
"config": "{\"tags\":[]}"
}
}
},
"listener_filters": [
{
"name": "envoy.filters.listener.tls_inspector"
}
],
"traffic_direction": "OUTBOUND"
},
"last_updated": "2022-05-21T12:08:39.937Z"
}
}
]
},
{
"@type": "type.googleapis.com/envoy.admin.v3.ScopedRoutesConfigDump"
},
{
"@type": "type.googleapis.com/envoy.admin.v3.RoutesConfigDump",
"dynamic_route_configs": [
{
"version_info": "822736081:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440",
"route_config": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "rds_ingress_http_8088",
"virtual_hosts": [
{
"name": "self_redirect",
"domains": [
"*"
],
"routes": [
{
"match": {
"prefix": "/flutter-server/"
},
"route": {
"cluster": "cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001",
"prefix_rewrite": "/",
"host_rewrite_literal": "app-100035407.prod"
}
},
{
"match": {
"prefix": "/foxpage-server/"
},
"route": {
"cluster": "cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001",
"prefix_rewrite": "/",
"host_rewrite_literal": "app-100035323.prod",
"timeout": "15s"
}
},
{
"match": {
"prefix": "/flutter-server"
},
"route": {
"cluster": "cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001",
"prefix_rewrite": "/",
"host_rewrite_literal": "app-100035407.prod"
}
},
{
"match": {
"prefix": "/foxpage-admin/"
},
"route": {
"cluster": "cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001",
"prefix_rewrite": "/",
"host_rewrite_literal": "app-1.prod",
"timeout": "15s"
}
},
{
"match": {
"prefix": "/foxpage-server"
},
"route": {
"cluster": "cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001",
"prefix_rewrite": "/",
"host_rewrite_literal": "app-100035323.prod",
"timeout": "15s"
}
},
{
"match": {
"prefix": "/foxpage-admin"
},
"route": {
"cluster": "cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001",
"prefix_rewrite": "/",
"host_rewrite_literal": "app-1.prod",
"timeout": "15s"
}
},
{
"match": {
"prefix": "/14524/"
},
"route": {
"cluster": "cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001",
"prefix_rewrite": "/",
"host_rewrite_literal": "app-2.prod",
"timeout": "15s"
}
},
{
"match": {
"prefix": "/19816/"
},
"route": {
"cluster": "cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001",
"prefix_rewrite": "/",
"host_rewrite_literal": "app-5.prod",
"timeout": "15s"
}
},
{
"match": {
"prefix": "/14524"
},
"route": {
"cluster": "cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001",
"prefix_rewrite": "/",
"host_rewrite_literal": "app-2.prod",
"timeout": "15s"
}
},
{
"match": {
"prefix": "/19816"
},
"route": {
"cluster": "cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001",
"prefix_rewrite": "/",
"host_rewrite_literal": "app-5.prod",
"timeout": "15s"
}
},
{
"match": {
"prefix": "/hot/"
},
"route": {
"cluster": "cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001",
"prefix_rewrite": "/hot/",
"timeout": "15s"
},
"request_headers_to_add": [
{
"header": {
"key": "x-app-mesh-virtual-service",
"value": "app-4.prod"
}
}
]
},
{
"match": {
"prefix": "/healthcheck/"
},
"route": {
"cluster": "cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001",
"prefix_rewrite": "/",
"host_rewrite_literal": "app-3.prod",
"timeout": "15s"
}
},
{
"match": {
"prefix": "/healthcheck"
},
"route": {
"cluster": "cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001",
"prefix_rewrite": "/",
"host_rewrite_literal": "app-3.prod",
"timeout": "15s"
}
},
{
"match": {
"prefix": "/"
},
"route": {
"cluster": "cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001",
"prefix_rewrite": "/",
"host_rewrite_literal": "app-3.prod",
"timeout": "15s"
}
}
]
}
]
},
"last_updated": "2022-05-21T12:08:39.943Z"
},
{
"version_info": "-1338743948:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440",
"route_config": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "rds_egress_http_15001",
"virtual_hosts": [
{
"name": "app-3.prod",
"domains": [
"app-3.prod",
"app-3.prod:15001"
],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"weighted_clusters": {
"clusters": [
{
"name": "cds_egress_app-mesh_app-3-canary_prod_http_8080",
"weight": 0
},
{
"name": "cds_egress_app-mesh_app-3-primary_prod_http_8080",
"weight": 100
}
],
"total_weight": 100
},
"timeout": "15s",
"retry_policy": {
"retry_on": "reset,refused-stream,retriable-status-codes",
"num_retries": 1,
"per_try_timeout": "5s",
"retry_host_predicate": [
{
"name": "envoy.retry_host_predicates.previous_hosts"
}
],
"host_selection_retry_max_attempts": "5",
"retriable_status_codes": [
502,
503,
504,
409
]
}
},
"metadata": {
"filter_metadata": {
"aws.appmesh.aggregate_stats": {
"config": "{\"tags\":[{\"key\":\"TargetVirtualService\",\"value\":\"app-3.prod\"}]}"
}
}
}
}
]
},
{
"name": "app-100035323.prod",
"domains": [
"app-100035323.prod",
"app-100035323.prod:15001"
],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"weighted_clusters": {
"clusters": [
{
"name": "cds_egress_app-mesh_app-100035323-canary_prod_http_8080",
"weight": 0
},
{
"name": "cds_egress_app-mesh_app-100035323-primary_prod_http_8080",
"weight": 100
}
],
"total_weight": 100
},
"timeout": "15s",
"retry_policy": {
"retry_on": "reset,refused-stream,retriable-status-codes",
"num_retries": 1,
"per_try_timeout": "5s",
"retry_host_predicate": [
{
"name": "envoy.retry_host_predicates.previous_hosts"
}
],
"host_selection_retry_max_attempts": "5",
"retriable_status_codes": [
502,
503,
504,
409
]
}
},
"metadata": {
"filter_metadata": {
"aws.appmesh.aggregate_stats": {
"config": "{\"tags\":[{\"key\":\"TargetVirtualService\",\"value\":\"app-100035323.prod\"}]}"
}
}
}
}
]
},
{
"name": "app-1.prod",
"domains": [
"app-1.prod",
"app-1.prod:15001"
],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"weighted_clusters": {
"clusters": [
{
"name": "cds_egress_app-mesh_app-1-canary_prod_http_8080",
"weight": 0
},
{
"name": "cds_egress_app-mesh_app-1-primary_prod_http_8080",
"weight": 100
}
],
"total_weight": 100
},
"timeout": "15s",
"retry_policy": {
"retry_on": "reset,refused-stream,retriable-status-codes",
"num_retries": 1,
"per_try_timeout": "5s",
"retry_host_predicate": [
{
"name": "envoy.retry_host_predicates.previous_hosts"
}
],
"host_selection_retry_max_attempts": "5",
"retriable_status_codes": [
502,
503,
504,
409
]
}
},
"metadata": {
"filter_metadata": {
"aws.appmesh.aggregate_stats": {
"config": "{\"tags\":[{\"key\":\"TargetVirtualService\",\"value\":\"app-1.prod\"}]}"
}
}
}
}
]
},
{
"name": "app-100035407.prod",
"domains": [
"app-100035407.prod",
"app-100035407.prod:15001"
]
},
{
"name": "default_backend",
"domains": [
"*"
],
"routes": [
{
"match": {
"prefix": "/",
"headers": [
{
"name": "x-app-mesh-virtual-service",
"exact_match": "app-4.prod"
}
]
},
"route": {
"weighted_clusters": {
"clusters": [
{
"name": "cds_egress_app-mesh_app-4-canary_prod_http_8080",
"weight": 0
},
{
"name": "cds_egress_app-mesh_app-4-primary_prod_http_8080",
"weight": 100
}
],
"total_weight": 100
},
"timeout": "15s",
"retry_policy": {
"retry_on": "reset,refused-stream,retriable-status-codes",
"num_retries": 1,
"per_try_timeout": "5s",
"retry_host_predicate": [
{
"name": "envoy.retry_host_predicates.previous_hosts"
}
],
"host_selection_retry_max_attempts": "5",
"retriable_status_codes": [
502,
503,
504,
409
]
}
},
"metadata": {
"filter_metadata": {
"aws.appmesh.aggregate_stats": {
"config": "{\"tags\":[{\"key\":\"TargetVirtualService\",\"value\":\"app-4.prod\"}]}"
}
}
},
"response_headers_to_remove": [
"x-app-mesh-virtual-service"
]
}
]
},
{
"name": "app-5.prod",
"domains": [
"app-5.prod",
"app-5.prod:15001"
],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"weighted_clusters": {
"clusters": [
{
"name": "cds_egress_app-mesh_app-5-canary_prod_http_8080",
"weight": 0
},
{
"name": "cds_egress_app-mesh_app-5-primary_prod_http_8080",
"weight": 100
}
],
"total_weight": 100
},
"timeout": "15s",
"retry_policy": {
"retry_on": "reset,refused-stream,retriable-status-codes",
"num_retries": 1,
"per_try_timeout": "5s",
"retry_host_predicate": [
{
"name": "envoy.retry_host_predicates.previous_hosts"
}
],
"host_selection_retry_max_attempts": "5",
"retriable_status_codes": [
502,
503,
504,
409
]
}
},
"metadata": {
"filter_metadata": {
"aws.appmesh.aggregate_stats": {
"config": "{\"tags\":[{\"key\":\"TargetVirtualService\",\"value\":\"app-5.prod\"}]}"
}
}
}
}
]
},
{
"name": "app-2.prod",
"domains": [
"app-2.prod",
"app-2.prod:15001"
],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"weighted_clusters": {
"clusters": [
{
"name": "cds_egress_app-mesh_app-2-canary_prod_http_8080",
"weight": 0
},
{
"name": "cds_egress_app-mesh_app-2-primary_prod_http_8080",
"weight": 100
}
],
"total_weight": 100
},
"timeout": "15s",
"retry_policy": {
"retry_on": "reset,refused-stream,retriable-status-codes",
"num_retries": 1,
"per_try_timeout": "5s",
"retry_host_predicate": [
{
"name": "envoy.retry_host_predicates.previous_hosts"
}
],
"host_selection_retry_max_attempts": "5",
"retriable_status_codes": [
502,
503,
504,
409
]
}
},
"metadata": {
"filter_metadata": {
"aws.appmesh.aggregate_stats": {
"config": "{\"tags\":[{\"key\":\"TargetVirtualService\",\"value\":\"app-2.prod\"}]}"
}
}
}
}
]
}
]
},
"last_updated": "2022-05-23T10:13:07.240Z"
}
]
},
{
"@type": "type.googleapis.com/envoy.admin.v3.SecretsConfigDump"
}
]
}
cluster_manager.cds.version_text: "-1338743948:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440"
control_plane.identifier: ""
http.egress.rds.rds_egress_http_15001.version_text: "-1338743948:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440"
http.ingress.rds.rds_ingress_http_8088.version_text: "-1338743948:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440"
listener_manager.lds.version_text: "-1338743948:-2086145894:cec42df27b90fab256794a8a8c943552bb7bd440"
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.assignment_stale: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.assignment_timeout_received: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.bind_errors: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.circuit_breakers.default.cx_open: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.circuit_breakers.default.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.circuit_breakers.default.rq_open: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.circuit_breakers.default.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.circuit_breakers.default.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.circuit_breakers.high.cx_open: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.circuit_breakers.high.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.circuit_breakers.high.rq_open: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.circuit_breakers.high.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.circuit_breakers.high.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.default.total_match_count: 1
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.lb_healthy_panic: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.lb_local_cluster_not_ok: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.lb_recalculate_zone_structures: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.lb_subsets_active: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.lb_subsets_created: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.lb_subsets_fallback: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.lb_subsets_fallback_panic: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.lb_subsets_removed: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.lb_subsets_selected: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.lb_zone_cluster_too_small: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.lb_zone_no_capacity_left: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.lb_zone_number_differs: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.lb_zone_routing_all_directly: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.lb_zone_routing_cross_zone: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.lb_zone_routing_sampled: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.max_host_weight: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.membership_change: 1
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.membership_degraded: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.membership_excluded: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.membership_healthy: 1
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.membership_total: 1
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.original_dst_host_invalid: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.retry_or_shadow_abandoned: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.update_attempt: 33917
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.update_empty: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.update_failure: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.update_no_rebuild: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.update_success: 33917
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_active: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_close_notify: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_connect_attempts_exceeded: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_connect_fail: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_connect_timeout: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_destroy: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_destroy_local: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_destroy_local_with_active_rq: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_destroy_remote: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_destroy_remote_with_active_rq: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_destroy_with_active_rq: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_http1_total: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_http2_total: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_http3_total: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_idle_timeout: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_max_requests: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_none_healthy: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_overflow: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_pool_overflow: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_protocol_error: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_rx_bytes_buffered: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_rx_bytes_total: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_total: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_tx_bytes_buffered: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_tx_bytes_total: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_flow_control_backed_up_total: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_flow_control_drained_total: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_flow_control_paused_reading_total: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_flow_control_resumed_reading_total: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_internal_redirect_failed_total: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_internal_redirect_succeeded_total: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_rq_active: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_rq_cancelled: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_rq_completed: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_rq_maintenance_mode: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_rq_max_duration_reached: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_rq_pending_active: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_rq_pending_failure_eject: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_rq_pending_overflow: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_rq_pending_total: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_rq_per_try_idle_timeout: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_rq_per_try_timeout: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_rq_retry: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_rq_retry_backoff_exponential: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_rq_retry_backoff_ratelimited: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_rq_retry_limit_exceeded: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_rq_retry_overflow: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_rq_retry_success: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_rq_rx_reset: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_rq_timeout: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_rq_total: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_rq_tx_reset: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.version: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.assignment_stale: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.assignment_timeout_received: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.bind_errors: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.circuit_breakers.default.cx_open: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.circuit_breakers.default.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.circuit_breakers.default.rq_open: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.circuit_breakers.default.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.circuit_breakers.default.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.circuit_breakers.high.cx_open: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.circuit_breakers.high.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.circuit_breakers.high.rq_open: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.circuit_breakers.high.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.circuit_breakers.high.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.default.total_match_count: 1
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.lb_healthy_panic: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.lb_local_cluster_not_ok: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.lb_recalculate_zone_structures: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.lb_subsets_active: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.lb_subsets_created: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.lb_subsets_fallback: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.lb_subsets_fallback_panic: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.lb_subsets_removed: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.lb_subsets_selected: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.lb_zone_cluster_too_small: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.lb_zone_no_capacity_left: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.lb_zone_number_differs: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.lb_zone_routing_all_directly: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.lb_zone_routing_cross_zone: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.lb_zone_routing_sampled: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.max_host_weight: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.membership_change: 1
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.membership_degraded: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.membership_excluded: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.membership_healthy: 1
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.membership_total: 1
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.original_dst_host_invalid: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.retry_or_shadow_abandoned: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.update_attempt: 33917
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.update_empty: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.update_failure: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.update_no_rebuild: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.update_success: 33917
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_active: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_close_notify: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_connect_attempts_exceeded: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_connect_fail: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_connect_timeout: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_destroy: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_destroy_local: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_destroy_local_with_active_rq: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_destroy_remote: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_destroy_remote_with_active_rq: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_destroy_with_active_rq: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_http1_total: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_http2_total: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_http3_total: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_idle_timeout: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_max_requests: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_none_healthy: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_overflow: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_pool_overflow: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_protocol_error: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_rx_bytes_buffered: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_rx_bytes_total: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_total: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_tx_bytes_buffered: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_tx_bytes_total: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_flow_control_backed_up_total: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_flow_control_drained_total: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_flow_control_paused_reading_total: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_flow_control_resumed_reading_total: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_internal_redirect_failed_total: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_internal_redirect_succeeded_total: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_rq_active: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_rq_cancelled: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_rq_completed: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_rq_maintenance_mode: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_rq_max_duration_reached: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_rq_pending_active: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_rq_pending_failure_eject: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_rq_pending_overflow: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_rq_pending_total: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_rq_per_try_idle_timeout: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_rq_per_try_timeout: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_rq_retry: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_rq_retry_backoff_exponential: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_rq_retry_backoff_ratelimited: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_rq_retry_limit_exceeded: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_rq_retry_overflow: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_rq_retry_success: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_rq_rx_reset: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_rq_timeout: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_rq_total: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_rq_tx_reset: 0
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.version: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.assignment_stale: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.assignment_timeout_received: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.bind_errors: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.circuit_breakers.default.cx_open: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.circuit_breakers.default.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.circuit_breakers.default.rq_open: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.circuit_breakers.default.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.circuit_breakers.default.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.circuit_breakers.high.cx_open: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.circuit_breakers.high.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.circuit_breakers.high.rq_open: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.circuit_breakers.high.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.circuit_breakers.high.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.default.total_match_count: 1
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.lb_healthy_panic: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.lb_local_cluster_not_ok: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.lb_recalculate_zone_structures: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.lb_subsets_active: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.lb_subsets_created: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.lb_subsets_fallback: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.lb_subsets_fallback_panic: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.lb_subsets_removed: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.lb_subsets_selected: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.lb_zone_cluster_too_small: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.lb_zone_no_capacity_left: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.lb_zone_number_differs: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.lb_zone_routing_all_directly: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.lb_zone_routing_cross_zone: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.lb_zone_routing_sampled: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.max_host_weight: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.membership_change: 1
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.membership_degraded: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.membership_excluded: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.membership_healthy: 1
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.membership_total: 1
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.original_dst_host_invalid: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.retry_or_shadow_abandoned: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.update_attempt: 33917
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.update_empty: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.update_failure: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.update_no_rebuild: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.update_success: 33917
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_active: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_close_notify: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_connect_attempts_exceeded: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_connect_fail: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_connect_timeout: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_destroy: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_destroy_local: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_destroy_local_with_active_rq: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_destroy_remote: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_destroy_remote_with_active_rq: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_destroy_with_active_rq: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_http1_total: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_http2_total: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_http3_total: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_idle_timeout: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_max_requests: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_none_healthy: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_overflow: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_pool_overflow: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_protocol_error: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_rx_bytes_buffered: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_rx_bytes_total: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_total: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_tx_bytes_buffered: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_tx_bytes_total: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_flow_control_backed_up_total: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_flow_control_drained_total: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_flow_control_paused_reading_total: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_flow_control_resumed_reading_total: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_internal_redirect_failed_total: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_internal_redirect_succeeded_total: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_rq_active: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_rq_cancelled: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_rq_completed: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_rq_maintenance_mode: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_rq_max_duration_reached: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_rq_pending_active: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_rq_pending_failure_eject: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_rq_pending_overflow: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_rq_pending_total: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_rq_per_try_idle_timeout: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_rq_per_try_timeout: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_rq_retry: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_rq_retry_backoff_exponential: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_rq_retry_backoff_ratelimited: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_rq_retry_limit_exceeded: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_rq_retry_overflow: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_rq_retry_success: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_rq_rx_reset: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_rq_timeout: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_rq_total: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_rq_tx_reset: 0
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.version: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.assignment_stale: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.assignment_timeout_received: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.bind_errors: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.circuit_breakers.default.cx_open: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.circuit_breakers.default.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.circuit_breakers.default.rq_open: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.circuit_breakers.default.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.circuit_breakers.default.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.circuit_breakers.high.cx_open: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.circuit_breakers.high.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.circuit_breakers.high.rq_open: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.circuit_breakers.high.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.circuit_breakers.high.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.default.total_match_count: 1
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.external.upstream_rq_200: 19
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.external.upstream_rq_2xx: 19
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.external.upstream_rq_404: 33957
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.external.upstream_rq_4xx: 33957
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.external.upstream_rq_completed: 33976
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.http1.dropped_headers_with_underscores: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.http1.metadata_not_supported_error: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.http1.requests_rejected_with_underscores_in_headers: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.http1.response_flood: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.lb_healthy_panic: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.lb_local_cluster_not_ok: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.lb_recalculate_zone_structures: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.lb_subsets_active: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.lb_subsets_created: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.lb_subsets_fallback: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.lb_subsets_fallback_panic: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.lb_subsets_removed: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.lb_subsets_selected: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.lb_zone_cluster_too_small: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.lb_zone_no_capacity_left: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.lb_zone_number_differs: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.lb_zone_routing_all_directly: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.lb_zone_routing_cross_zone: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.lb_zone_routing_sampled: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.max_host_weight: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.membership_change: 1
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.membership_degraded: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.membership_excluded: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.membership_healthy: 1
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.membership_total: 1
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.original_dst_host_invalid: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.retry.upstream_rq_503: 427
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.retry.upstream_rq_5xx: 427
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.retry.upstream_rq_completed: 427
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.retry_or_shadow_abandoned: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.update_attempt: 33917
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.update_empty: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.update_failure: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.update_no_rebuild: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.update_success: 33917
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_active: 2
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_close_notify: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_connect_attempts_exceeded: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_connect_fail: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_connect_timeout: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_destroy: 434
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_destroy_local: 434
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_destroy_local_with_active_rq: 427
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_destroy_remote: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_destroy_remote_with_active_rq: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_destroy_with_active_rq: 427
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_http1_total: 436
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_http2_total: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_http3_total: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_idle_timeout: 7
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_max_requests: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_none_healthy: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_overflow: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_pool_overflow: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_protocol_error: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_rx_bytes_buffered: 708
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_rx_bytes_total: 13036491
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_total: 436
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_tx_bytes_buffered: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_tx_bytes_total: 14190765
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_flow_control_backed_up_total: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_flow_control_drained_total: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_flow_control_paused_reading_total: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_flow_control_resumed_reading_total: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_internal_redirect_failed_total: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_internal_redirect_succeeded_total: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_200: 19
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_2xx: 19
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_404: 33957
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_4xx: 33957
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_active: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_cancelled: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_completed: 33976
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_maintenance_mode: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_max_duration_reached: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_pending_active: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_pending_failure_eject: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_pending_overflow: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_pending_total: 436
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_per_try_idle_timeout: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_per_try_timeout: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_retry: 427
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_retry_backoff_exponential: 427
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_retry_backoff_ratelimited: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_retry_limit_exceeded: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_retry_overflow: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_retry_success: 427
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_rx_reset: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_timeout: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_total: 34403
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_tx_reset: 0
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.version: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.assignment_stale: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.assignment_timeout_received: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.bind_errors: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.circuit_breakers.default.cx_open: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.circuit_breakers.default.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.circuit_breakers.default.rq_open: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.circuit_breakers.default.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.circuit_breakers.default.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.circuit_breakers.high.cx_open: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.circuit_breakers.high.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.circuit_breakers.high.rq_open: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.circuit_breakers.high.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.circuit_breakers.high.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.default.total_match_count: 1
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.lb_healthy_panic: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.lb_local_cluster_not_ok: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.lb_recalculate_zone_structures: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.lb_subsets_active: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.lb_subsets_created: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.lb_subsets_fallback: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.lb_subsets_fallback_panic: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.lb_subsets_removed: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.lb_subsets_selected: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.lb_zone_cluster_too_small: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.lb_zone_no_capacity_left: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.lb_zone_number_differs: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.lb_zone_routing_all_directly: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.lb_zone_routing_cross_zone: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.lb_zone_routing_sampled: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.max_host_weight: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.membership_change: 1
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.membership_degraded: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.membership_excluded: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.membership_healthy: 1
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.membership_total: 1
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.original_dst_host_invalid: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.retry_or_shadow_abandoned: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.update_attempt: 33917
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.update_empty: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.update_failure: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.update_no_rebuild: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.update_success: 33917
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_active: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_close_notify: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_connect_attempts_exceeded: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_connect_fail: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_connect_timeout: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_destroy: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_destroy_local: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_destroy_local_with_active_rq: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_destroy_remote: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_destroy_remote_with_active_rq: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_destroy_with_active_rq: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_http1_total: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_http2_total: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_http3_total: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_idle_timeout: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_max_requests: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_none_healthy: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_overflow: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_pool_overflow: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_protocol_error: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_rx_bytes_buffered: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_rx_bytes_total: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_total: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_tx_bytes_buffered: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_tx_bytes_total: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_flow_control_backed_up_total: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_flow_control_drained_total: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_flow_control_paused_reading_total: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_flow_control_resumed_reading_total: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_internal_redirect_failed_total: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_internal_redirect_succeeded_total: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_rq_active: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_rq_cancelled: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_rq_completed: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_rq_maintenance_mode: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_rq_max_duration_reached: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_rq_pending_active: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_rq_pending_failure_eject: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_rq_pending_overflow: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_rq_pending_total: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_rq_per_try_idle_timeout: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_rq_per_try_timeout: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_rq_retry: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_rq_retry_backoff_exponential: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_rq_retry_backoff_ratelimited: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_rq_retry_limit_exceeded: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_rq_retry_overflow: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_rq_retry_success: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_rq_rx_reset: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_rq_timeout: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_rq_total: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_rq_tx_reset: 0
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.version: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.assignment_stale: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.assignment_timeout_received: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.bind_errors: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.circuit_breakers.default.cx_open: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.circuit_breakers.default.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.circuit_breakers.default.rq_open: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.circuit_breakers.default.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.circuit_breakers.default.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.circuit_breakers.high.cx_open: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.circuit_breakers.high.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.circuit_breakers.high.rq_open: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.circuit_breakers.high.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.circuit_breakers.high.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.default.total_match_count: 1
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.lb_healthy_panic: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.lb_local_cluster_not_ok: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.lb_recalculate_zone_structures: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.lb_subsets_active: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.lb_subsets_created: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.lb_subsets_fallback: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.lb_subsets_fallback_panic: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.lb_subsets_removed: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.lb_subsets_selected: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.lb_zone_cluster_too_small: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.lb_zone_no_capacity_left: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.lb_zone_number_differs: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.lb_zone_routing_all_directly: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.lb_zone_routing_cross_zone: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.lb_zone_routing_sampled: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.max_host_weight: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.membership_change: 1
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.membership_degraded: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.membership_excluded: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.membership_healthy: 1
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.membership_total: 1
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.original_dst_host_invalid: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.retry_or_shadow_abandoned: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.update_attempt: 33917
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.update_empty: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.update_failure: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.update_no_rebuild: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.update_success: 33917
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_active: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_close_notify: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_connect_attempts_exceeded: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_connect_fail: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_connect_timeout: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_destroy: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_destroy_local: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_destroy_local_with_active_rq: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_destroy_remote: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_destroy_remote_with_active_rq: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_destroy_with_active_rq: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_http1_total: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_http2_total: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_http3_total: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_idle_timeout: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_max_requests: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_none_healthy: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_overflow: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_pool_overflow: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_protocol_error: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_rx_bytes_buffered: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_rx_bytes_total: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_total: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_tx_bytes_buffered: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_tx_bytes_total: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_flow_control_backed_up_total: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_flow_control_drained_total: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_flow_control_paused_reading_total: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_flow_control_resumed_reading_total: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_internal_redirect_failed_total: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_internal_redirect_succeeded_total: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_rq_active: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_rq_cancelled: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_rq_completed: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_rq_maintenance_mode: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_rq_max_duration_reached: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_rq_pending_active: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_rq_pending_failure_eject: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_rq_pending_overflow: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_rq_pending_total: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_rq_per_try_idle_timeout: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_rq_per_try_timeout: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_rq_retry: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_rq_retry_backoff_exponential: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_rq_retry_backoff_ratelimited: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_rq_retry_limit_exceeded: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_rq_retry_overflow: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_rq_retry_success: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_rq_rx_reset: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_rq_timeout: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_rq_total: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_rq_tx_reset: 0
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.version: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.assignment_stale: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.assignment_timeout_received: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.bind_errors: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.circuit_breakers.default.cx_open: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.circuit_breakers.default.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.circuit_breakers.default.rq_open: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.circuit_breakers.default.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.circuit_breakers.default.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.circuit_breakers.high.cx_open: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.circuit_breakers.high.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.circuit_breakers.high.rq_open: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.circuit_breakers.high.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.circuit_breakers.high.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.default.total_match_count: 1
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.lb_healthy_panic: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.lb_local_cluster_not_ok: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.lb_recalculate_zone_structures: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.lb_subsets_active: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.lb_subsets_created: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.lb_subsets_fallback: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.lb_subsets_fallback_panic: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.lb_subsets_removed: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.lb_subsets_selected: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.lb_zone_cluster_too_small: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.lb_zone_no_capacity_left: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.lb_zone_number_differs: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.lb_zone_routing_all_directly: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.lb_zone_routing_cross_zone: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.lb_zone_routing_sampled: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.max_host_weight: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.membership_change: 1
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.membership_degraded: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.membership_excluded: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.membership_healthy: 1
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.membership_total: 1
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.original_dst_host_invalid: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.retry_or_shadow_abandoned: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.update_attempt: 33917
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.update_empty: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.update_failure: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.update_no_rebuild: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.update_success: 33917
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_active: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_close_notify: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_connect_attempts_exceeded: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_connect_fail: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_connect_timeout: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_destroy: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_destroy_local: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_destroy_local_with_active_rq: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_destroy_remote: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_destroy_remote_with_active_rq: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_destroy_with_active_rq: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_http1_total: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_http2_total: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_http3_total: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_idle_timeout: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_max_requests: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_none_healthy: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_overflow: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_pool_overflow: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_protocol_error: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_rx_bytes_buffered: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_rx_bytes_total: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_total: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_tx_bytes_buffered: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_tx_bytes_total: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_flow_control_backed_up_total: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_flow_control_drained_total: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_flow_control_paused_reading_total: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_flow_control_resumed_reading_total: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_internal_redirect_failed_total: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_internal_redirect_succeeded_total: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_rq_active: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_rq_cancelled: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_rq_completed: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_rq_maintenance_mode: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_rq_max_duration_reached: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_rq_pending_active: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_rq_pending_failure_eject: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_rq_pending_overflow: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_rq_pending_total: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_rq_per_try_idle_timeout: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_rq_per_try_timeout: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_rq_retry: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_rq_retry_backoff_exponential: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_rq_retry_backoff_ratelimited: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_rq_retry_limit_exceeded: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_rq_retry_overflow: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_rq_retry_success: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_rq_rx_reset: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_rq_timeout: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_rq_total: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_rq_tx_reset: 0
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.version: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.assignment_stale: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.assignment_timeout_received: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.bind_errors: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.circuit_breakers.default.cx_open: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.circuit_breakers.default.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.circuit_breakers.default.rq_open: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.circuit_breakers.default.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.circuit_breakers.default.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.circuit_breakers.high.cx_open: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.circuit_breakers.high.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.circuit_breakers.high.rq_open: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.circuit_breakers.high.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.circuit_breakers.high.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.default.total_match_count: 1
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.lb_healthy_panic: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.lb_local_cluster_not_ok: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.lb_recalculate_zone_structures: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.lb_subsets_active: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.lb_subsets_created: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.lb_subsets_fallback: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.lb_subsets_fallback_panic: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.lb_subsets_removed: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.lb_subsets_selected: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.lb_zone_cluster_too_small: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.lb_zone_no_capacity_left: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.lb_zone_number_differs: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.lb_zone_routing_all_directly: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.lb_zone_routing_cross_zone: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.lb_zone_routing_sampled: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.max_host_weight: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.membership_change: 1
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.membership_degraded: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.membership_excluded: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.membership_healthy: 1
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.membership_total: 1
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.original_dst_host_invalid: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.retry_or_shadow_abandoned: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.update_attempt: 33917
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.update_empty: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.update_failure: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.update_no_rebuild: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.update_success: 33917
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_active: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_close_notify: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_connect_attempts_exceeded: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_connect_fail: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_connect_timeout: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_destroy: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_destroy_local: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_destroy_local_with_active_rq: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_destroy_remote: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_destroy_remote_with_active_rq: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_destroy_with_active_rq: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_http1_total: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_http2_total: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_http3_total: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_idle_timeout: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_max_requests: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_none_healthy: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_overflow: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_pool_overflow: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_protocol_error: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_rx_bytes_buffered: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_rx_bytes_total: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_total: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_tx_bytes_buffered: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_tx_bytes_total: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_flow_control_backed_up_total: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_flow_control_drained_total: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_flow_control_paused_reading_total: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_flow_control_resumed_reading_total: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_internal_redirect_failed_total: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_internal_redirect_succeeded_total: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_rq_active: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_rq_cancelled: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_rq_completed: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_rq_maintenance_mode: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_rq_max_duration_reached: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_rq_pending_active: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_rq_pending_failure_eject: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_rq_pending_overflow: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_rq_pending_total: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_rq_per_try_idle_timeout: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_rq_per_try_timeout: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_rq_retry: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_rq_retry_backoff_exponential: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_rq_retry_backoff_ratelimited: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_rq_retry_limit_exceeded: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_rq_retry_overflow: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_rq_retry_success: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_rq_rx_reset: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_rq_timeout: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_rq_total: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_rq_tx_reset: 0
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.version: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.assignment_stale: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.assignment_timeout_received: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.bind_errors: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.circuit_breakers.default.cx_open: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.circuit_breakers.default.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.circuit_breakers.default.rq_open: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.circuit_breakers.default.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.circuit_breakers.default.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.circuit_breakers.high.cx_open: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.circuit_breakers.high.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.circuit_breakers.high.rq_open: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.circuit_breakers.high.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.circuit_breakers.high.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.default.total_match_count: 1
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.external.upstream_rq_200: 4938
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.external.upstream_rq_2xx: 4938
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.external.upstream_rq_completed: 4938
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.http1.dropped_headers_with_underscores: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.http1.metadata_not_supported_error: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.http1.requests_rejected_with_underscores_in_headers: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.http1.response_flood: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.internal.upstream_rq_200: 426
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.internal.upstream_rq_2xx: 426
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.internal.upstream_rq_completed: 426
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.lb_healthy_panic: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.lb_local_cluster_not_ok: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.lb_recalculate_zone_structures: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.lb_subsets_active: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.lb_subsets_created: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.lb_subsets_fallback: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.lb_subsets_fallback_panic: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.lb_subsets_removed: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.lb_subsets_selected: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.lb_zone_cluster_too_small: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.lb_zone_no_capacity_left: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.lb_zone_number_differs: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.lb_zone_routing_all_directly: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.lb_zone_routing_cross_zone: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.lb_zone_routing_sampled: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.max_host_weight: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.membership_change: 1
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.membership_degraded: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.membership_excluded: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.membership_healthy: 1
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.membership_total: 1
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.original_dst_host_invalid: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.retry_or_shadow_abandoned: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.update_attempt: 33917
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.update_empty: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.update_failure: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.update_no_rebuild: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.update_success: 33917
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_active: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_close_notify: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_connect_attempts_exceeded: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_connect_fail: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_connect_timeout: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_destroy: 23
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_destroy_local: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_destroy_local_with_active_rq: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_destroy_remote: 23
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_destroy_remote_with_active_rq: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_destroy_with_active_rq: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_http1_total: 23
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_http2_total: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_http3_total: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_idle_timeout: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_max_requests: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_none_healthy: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_overflow: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_pool_overflow: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_protocol_error: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_rx_bytes_buffered: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_rx_bytes_total: 2817757
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_total: 23
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_tx_bytes_buffered: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_tx_bytes_total: 13417197
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_flow_control_backed_up_total: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_flow_control_drained_total: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_flow_control_paused_reading_total: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_flow_control_resumed_reading_total: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_internal_redirect_failed_total: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_internal_redirect_succeeded_total: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_200: 5364
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_2xx: 5364
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_active: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_cancelled: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_completed: 5364
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_maintenance_mode: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_max_duration_reached: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_pending_active: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_pending_failure_eject: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_pending_overflow: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_pending_total: 23
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_per_try_idle_timeout: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_per_try_timeout: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_retry: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_retry_backoff_exponential: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_retry_backoff_ratelimited: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_retry_limit_exceeded: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_retry_overflow: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_retry_success: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_rx_reset: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_timeout: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_total: 5364
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_tx_reset: 0
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.version: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.assignment_stale: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.assignment_timeout_received: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.bind_errors: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.circuit_breakers.default.cx_open: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.circuit_breakers.default.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.circuit_breakers.default.rq_open: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.circuit_breakers.default.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.circuit_breakers.default.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.circuit_breakers.high.cx_open: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.circuit_breakers.high.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.circuit_breakers.high.rq_open: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.circuit_breakers.high.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.circuit_breakers.high.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.default.total_match_count: 1
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.external.upstream_rq_200: 3229960
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.external.upstream_rq_2xx: 3229960
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.external.upstream_rq_500: 6
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.external.upstream_rq_503: 2922
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.external.upstream_rq_504: 118
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.external.upstream_rq_5xx: 3046
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.external.upstream_rq_completed: 3233006
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.http1.dropped_headers_with_underscores: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.http1.metadata_not_supported_error: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.http1.requests_rejected_with_underscores_in_headers: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.http1.response_flood: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.internal.upstream_rq_200: 276952
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.internal.upstream_rq_2xx: 276952
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.internal.upstream_rq_500: 44
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.internal.upstream_rq_503: 157
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.internal.upstream_rq_504: 11
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.internal.upstream_rq_5xx: 212
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.internal.upstream_rq_completed: 277164
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.lb_healthy_panic: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.lb_local_cluster_not_ok: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.lb_recalculate_zone_structures: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.lb_subsets_active: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.lb_subsets_created: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.lb_subsets_fallback: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.lb_subsets_fallback_panic: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.lb_subsets_removed: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.lb_subsets_selected: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.lb_zone_cluster_too_small: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.lb_zone_no_capacity_left: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.lb_zone_number_differs: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.lb_zone_routing_all_directly: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.lb_zone_routing_cross_zone: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.lb_zone_routing_sampled: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.max_host_weight: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.membership_change: 1
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.membership_degraded: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.membership_excluded: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.membership_healthy: 1
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.membership_total: 1
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.original_dst_host_invalid: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.retry.upstream_rq_503: 62
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.retry.upstream_rq_5xx: 62
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.retry.upstream_rq_completed: 62
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.retry_or_shadow_abandoned: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.update_attempt: 33917
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.update_empty: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.update_failure: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.update_no_rebuild: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.update_success: 33917
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_active: 13
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_close_notify: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_connect_attempts_exceeded: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_connect_fail: 1482
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_connect_timeout: 1482
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_destroy: 2299
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_destroy_local: 2291
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_destroy_local_with_active_rq: 487
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_destroy_remote: 8
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_destroy_remote_with_active_rq: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_destroy_with_active_rq: 487
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_http1_total: 2312
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_http2_total: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_http3_total: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_idle_timeout: 322
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_max_requests: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_none_healthy: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_overflow: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_pool_overflow: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_protocol_error: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_rx_bytes_buffered: 5052
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_rx_bytes_total: 1517682417
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_total: 2312
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_tx_bytes_buffered: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_tx_bytes_total: 8816062321
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_flow_control_backed_up_total: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_flow_control_drained_total: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_flow_control_paused_reading_total: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_flow_control_resumed_reading_total: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_internal_redirect_failed_total: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_internal_redirect_succeeded_total: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_200: 3506912
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_2xx: 3506912
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_500: 50
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_503: 3079
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_504: 129
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_5xx: 3258
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_active: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_cancelled: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_completed: 3510170
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_maintenance_mode: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_max_duration_reached: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_pending_active: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_pending_failure_eject: 6075
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_pending_overflow: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_pending_total: 6908
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_per_try_idle_timeout: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_per_try_timeout: 425
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_retry: 3399
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_retry_backoff_exponential: 3399
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_retry_backoff_ratelimited: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_retry_limit_exceeded: 3208
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_retry_overflow: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_retry_success: 191
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_rx_reset: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_timeout: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_total: 3507494
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_tx_reset: 0
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.version: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.assignment_stale: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.assignment_timeout_received: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.bind_errors: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.circuit_breakers.default.cx_open: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.circuit_breakers.default.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.circuit_breakers.default.rq_open: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.circuit_breakers.default.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.circuit_breakers.default.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.circuit_breakers.high.cx_open: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.circuit_breakers.high.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.circuit_breakers.high.rq_open: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.circuit_breakers.high.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.circuit_breakers.high.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.default.total_match_count: 1
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.lb_healthy_panic: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.lb_local_cluster_not_ok: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.lb_recalculate_zone_structures: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.lb_subsets_active: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.lb_subsets_created: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.lb_subsets_fallback: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.lb_subsets_fallback_panic: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.lb_subsets_removed: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.lb_subsets_selected: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.lb_zone_cluster_too_small: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.lb_zone_no_capacity_left: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.lb_zone_number_differs: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.lb_zone_routing_all_directly: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.lb_zone_routing_cross_zone: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.lb_zone_routing_sampled: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.max_host_weight: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.membership_change: 1
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.membership_degraded: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.membership_excluded: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.membership_healthy: 1
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.membership_total: 1
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.original_dst_host_invalid: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.retry_or_shadow_abandoned: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.update_attempt: 33917
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.update_empty: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.update_failure: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.update_no_rebuild: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.update_success: 33917
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_active: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_close_notify: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_connect_attempts_exceeded: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_connect_fail: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_connect_timeout: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_destroy: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_destroy_local: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_destroy_local_with_active_rq: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_destroy_remote: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_destroy_remote_with_active_rq: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_destroy_with_active_rq: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_http1_total: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_http2_total: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_http3_total: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_idle_timeout: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_max_requests: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_none_healthy: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_overflow: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_pool_overflow: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_protocol_error: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_rx_bytes_buffered: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_rx_bytes_total: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_total: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_tx_bytes_buffered: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_tx_bytes_total: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_flow_control_backed_up_total: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_flow_control_drained_total: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_flow_control_paused_reading_total: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_flow_control_resumed_reading_total: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_internal_redirect_failed_total: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_internal_redirect_succeeded_total: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_rq_active: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_rq_cancelled: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_rq_completed: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_rq_maintenance_mode: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_rq_max_duration_reached: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_rq_pending_active: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_rq_pending_failure_eject: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_rq_pending_overflow: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_rq_pending_total: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_rq_per_try_idle_timeout: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_rq_per_try_timeout: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_rq_retry: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_rq_retry_backoff_exponential: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_rq_retry_backoff_ratelimited: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_rq_retry_limit_exceeded: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_rq_retry_overflow: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_rq_retry_success: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_rq_rx_reset: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_rq_timeout: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_rq_total: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_rq_tx_reset: 0
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.version: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.assignment_stale: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.assignment_timeout_received: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.bind_errors: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.circuit_breakers.default.cx_open: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.circuit_breakers.default.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.circuit_breakers.default.rq_open: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.circuit_breakers.default.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.circuit_breakers.default.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.circuit_breakers.high.cx_open: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.circuit_breakers.high.cx_pool_open: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.circuit_breakers.high.rq_open: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.circuit_breakers.high.rq_pending_open: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.circuit_breakers.high.rq_retry_open: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.default.total_match_count: 1
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.external.upstream_rq_200: 836771
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.external.upstream_rq_2xx: 836771
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.external.upstream_rq_500: 43
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.external.upstream_rq_5xx: 43
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.external.upstream_rq_completed: 836814
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.http1.dropped_headers_with_underscores: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.http1.metadata_not_supported_error: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.http1.requests_rejected_with_underscores_in_headers: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.http1.response_flood: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.internal.upstream_rq_200: 24
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.internal.upstream_rq_2xx: 24
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.internal.upstream_rq_completed: 24
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.lb_healthy_panic: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.lb_local_cluster_not_ok: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.lb_recalculate_zone_structures: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.lb_subsets_active: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.lb_subsets_created: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.lb_subsets_fallback: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.lb_subsets_fallback_panic: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.lb_subsets_removed: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.lb_subsets_selected: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.lb_zone_cluster_too_small: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.lb_zone_no_capacity_left: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.lb_zone_number_differs: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.lb_zone_routing_all_directly: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.lb_zone_routing_cross_zone: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.lb_zone_routing_sampled: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.max_host_weight: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.membership_change: 1
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.membership_degraded: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.membership_excluded: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.membership_healthy: 1
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.membership_total: 1
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.original_dst_host_invalid: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.retry_or_shadow_abandoned: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.update_attempt: 33917
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.update_empty: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.update_failure: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.update_no_rebuild: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.update_success: 33917
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_active: 11
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_close_notify: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_connect_attempts_exceeded: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_connect_fail: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_connect_timeout: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_destroy: 26
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_destroy_local: 23
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_destroy_local_with_active_rq: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_destroy_remote: 3
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_destroy_remote_with_active_rq: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_destroy_with_active_rq: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_http1_total: 37
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_http2_total: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_http3_total: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_idle_timeout: 23
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_max_requests: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_none_healthy: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_overflow: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_pool_overflow: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_protocol_error: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_rx_bytes_buffered: 1965
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_rx_bytes_total: 388923067
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_total: 37
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_tx_bytes_buffered: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_tx_bytes_total: 2692902155
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_flow_control_backed_up_total: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_flow_control_drained_total: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_flow_control_paused_reading_total: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_flow_control_resumed_reading_total: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_internal_redirect_failed_total: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_internal_redirect_succeeded_total: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_200: 836795
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_2xx: 836795
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_500: 43
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_5xx: 43
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_active: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_cancelled: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_completed: 836838
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_maintenance_mode: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_max_duration_reached: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_pending_active: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_pending_failure_eject: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_pending_overflow: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_pending_total: 37
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_per_try_idle_timeout: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_per_try_timeout: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_retry: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_retry_backoff_exponential: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_retry_backoff_ratelimited: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_retry_limit_exceeded: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_retry_overflow: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_retry_success: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_rx_reset: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_timeout: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_total: 836838
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_tx_reset: 0
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.version: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.assignment_stale: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.assignment_timeout_received: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.bind_errors: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.circuit_breakers.default.cx_open: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.circuit_breakers.default.cx_pool_open: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.circuit_breakers.default.rq_open: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.circuit_breakers.default.rq_pending_open: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.circuit_breakers.default.rq_retry_open: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.circuit_breakers.high.cx_open: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.circuit_breakers.high.cx_pool_open: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.circuit_breakers.high.rq_open: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.circuit_breakers.high.rq_pending_open: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.circuit_breakers.high.rq_retry_open: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.default.total_match_count: 1
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.external.upstream_rq_200: 4071688
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.external.upstream_rq_2xx: 4071688
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.external.upstream_rq_404: 33957
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.external.upstream_rq_4xx: 33957
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.external.upstream_rq_500: 49
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.external.upstream_rq_503: 2922
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.external.upstream_rq_504: 118
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.external.upstream_rq_5xx: 3089
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.external.upstream_rq_completed: 4108734
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.http1.dropped_headers_with_underscores: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.http1.metadata_not_supported_error: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.http1.requests_rejected_with_underscores_in_headers: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.http1.response_flood: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.internal.upstream_rq_200: 277402
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.internal.upstream_rq_2xx: 277402
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.internal.upstream_rq_500: 44
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.internal.upstream_rq_503: 157
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.internal.upstream_rq_504: 11
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.internal.upstream_rq_5xx: 212
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.internal.upstream_rq_completed: 277614
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.lb_healthy_panic: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.lb_local_cluster_not_ok: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.lb_recalculate_zone_structures: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.lb_subsets_active: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.lb_subsets_created: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.lb_subsets_fallback: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.lb_subsets_fallback_panic: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.lb_subsets_removed: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.lb_subsets_selected: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.lb_zone_cluster_too_small: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.lb_zone_no_capacity_left: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.lb_zone_number_differs: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.lb_zone_routing_all_directly: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.lb_zone_routing_cross_zone: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.lb_zone_routing_sampled: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.max_host_weight: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.membership_change: 1
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.membership_degraded: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.membership_excluded: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.membership_healthy: 1
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.membership_total: 1
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.original_dst_host_invalid: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.retry_or_shadow_abandoned: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.update_attempt: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.update_empty: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.update_failure: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.update_no_rebuild: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.update_success: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_active: 16
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_close_notify: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_connect_attempts_exceeded: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_connect_fail: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_connect_timeout: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_destroy: 498
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_destroy_local: 492
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_destroy_local_with_active_rq: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_destroy_remote: 6
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_destroy_remote_with_active_rq: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_destroy_with_active_rq: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_http1_total: 514
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_http2_total: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_http3_total: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_idle_timeout: 492
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_max_requests: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_none_healthy: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_overflow: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_pool_overflow: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_protocol_error: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_rx_bytes_buffered: 5664
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_rx_bytes_total: 1923092694
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_total: 514
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_tx_bytes_buffered: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_tx_bytes_total: 11547428789
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_flow_control_backed_up_total: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_flow_control_drained_total: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_flow_control_paused_reading_total: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_flow_control_resumed_reading_total: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_internal_redirect_failed_total: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_internal_redirect_succeeded_total: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_200: 4349090
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_2xx: 4349090
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_404: 33957
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_4xx: 33957
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_500: 93
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_503: 3079
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_504: 129
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_5xx: 3301
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_active: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_cancelled: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_completed: 4386348
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_maintenance_mode: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_max_duration_reached: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_pending_active: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_pending_failure_eject: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_pending_overflow: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_pending_total: 514
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_per_try_idle_timeout: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_per_try_timeout: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_retry: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_retry_backoff_exponential: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_retry_backoff_ratelimited: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_retry_limit_exceeded: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_retry_overflow: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_retry_success: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_rx_reset: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_timeout: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_total: 4386348
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_tx_reset: 0
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.version: 0
cluster_manager.active_clusters: 13
cluster_manager.cds.init_fetch_timeout: 0
cluster_manager.cds.update_attempt: 96
cluster_manager.cds.update_failure: 86
cluster_manager.cds.update_rejected: 0
cluster_manager.cds.update_success: 9
cluster_manager.cds.update_time: 1653300787235
cluster_manager.cds.version: 17800499387772728049
cluster_manager.cluster_added: 13
cluster_manager.cluster_modified: 0
cluster_manager.cluster_removed: 0
cluster_manager.cluster_updated: 0
cluster_manager.cluster_updated_via_merge: 0
cluster_manager.update_merge_cancelled: 0
cluster_manager.update_out_of_merge_window: 0
cluster_manager.warming_clusters: 0
control_plane.connected_state: 1
control_plane.pending_requests: 0
control_plane.rate_limit_enforced: 0
envoy.overload_actions.reset_high_memory_stream.count: 0
filesystem.flushed_by_timer: 33956
filesystem.reopen_failed: 0
filesystem.write_buffered: 8823631
filesystem.write_completed: 1085260
filesystem.write_failed: 0
filesystem.write_total_buffered: 53866
grpc.ads.google_grpc_client_creation: 1
grpc.ads.streams_closed_0: 86
grpc.ads.streams_closed_1: 0
grpc.ads.streams_closed_10: 0
grpc.ads.streams_closed_11: 0
grpc.ads.streams_closed_12: 0
grpc.ads.streams_closed_13: 0
grpc.ads.streams_closed_14: 0
grpc.ads.streams_closed_15: 0
grpc.ads.streams_closed_16: 0
grpc.ads.streams_closed_2: 0
grpc.ads.streams_closed_3: 0
grpc.ads.streams_closed_4: 0
grpc.ads.streams_closed_5: 0
grpc.ads.streams_closed_6: 0
grpc.ads.streams_closed_7: 0
grpc.ads.streams_closed_8: 0
grpc.ads.streams_closed_9: 0
grpc.ads.streams_total: 87
http.admin.downstream_cx_active: 2
http.admin.downstream_cx_delayed_close_timeout: 0
http.admin.downstream_cx_destroy: 33962
http.admin.downstream_cx_destroy_active_rq: 0
http.admin.downstream_cx_destroy_local: 0
http.admin.downstream_cx_destroy_local_active_rq: 0
http.admin.downstream_cx_destroy_remote: 33962
http.admin.downstream_cx_destroy_remote_active_rq: 0
http.admin.downstream_cx_drain_close: 0
http.admin.downstream_cx_http1_active: 2
http.admin.downstream_cx_http1_total: 16985
http.admin.downstream_cx_http2_active: 0
http.admin.downstream_cx_http2_total: 0
http.admin.downstream_cx_http3_active: 0
http.admin.downstream_cx_http3_total: 0
http.admin.downstream_cx_idle_timeout: 0
http.admin.downstream_cx_max_duration_reached: 0
http.admin.downstream_cx_max_requests_reached: 0
http.admin.downstream_cx_overload_disable_keepalive: 0
http.admin.downstream_cx_protocol_error: 0
http.admin.downstream_cx_rx_bytes_buffered: 83
http.admin.downstream_cx_rx_bytes_total: 10101576
http.admin.downstream_cx_ssl_active: 0
http.admin.downstream_cx_ssl_total: 0
http.admin.downstream_cx_total: 33964
http.admin.downstream_cx_tx_bytes_buffered: 0
http.admin.downstream_cx_tx_bytes_total: 13155824700
http.admin.downstream_cx_upgrades_active: 0
http.admin.downstream_cx_upgrades_total: 0
http.admin.downstream_flow_control_paused_reading_total: 0
http.admin.downstream_flow_control_resumed_reading_total: 0
http.admin.downstream_rq_1xx: 0
http.admin.downstream_rq_2xx: 50935
http.admin.downstream_rq_3xx: 0
http.admin.downstream_rq_4xx: 0
http.admin.downstream_rq_5xx: 0
http.admin.downstream_rq_active: 1
http.admin.downstream_rq_completed: 50935
http.admin.downstream_rq_failed_path_normalization: 0
http.admin.downstream_rq_header_timeout: 0
http.admin.downstream_rq_http1_total: 50936
http.admin.downstream_rq_http2_total: 0
http.admin.downstream_rq_http3_total: 0
http.admin.downstream_rq_idle_timeout: 0
http.admin.downstream_rq_max_duration_reached: 0
http.admin.downstream_rq_non_relative_path: 0
http.admin.downstream_rq_overload_close: 0
http.admin.downstream_rq_redirected_with_normalized_path: 0
http.admin.downstream_rq_rejected_via_ip_detection: 0
http.admin.downstream_rq_response_before_rq_complete: 0
http.admin.downstream_rq_rx_reset: 0
http.admin.downstream_rq_timeout: 0
http.admin.downstream_rq_too_large: 0
http.admin.downstream_rq_total: 50936
http.admin.downstream_rq_tx_reset: 0
http.admin.downstream_rq_ws_on_non_ws_route: 0
http.admin.rs_too_large: 0
http.async-client.no_cluster: 0
http.async-client.no_route: 0
http.async-client.passthrough_internal_redirect_bad_location: 0
http.async-client.passthrough_internal_redirect_no_route: 0
http.async-client.passthrough_internal_redirect_predicate: 0
http.async-client.passthrough_internal_redirect_too_many_redirects: 0
http.async-client.passthrough_internal_redirect_unsafe_scheme: 0
http.async-client.rq_direct_response: 0
http.async-client.rq_redirect: 0
http.async-client.rq_reset_after_downstream_response_started: 0
http.async-client.rq_total: 0
http.egress.downstream_cx_active: 16
http.egress.downstream_cx_delayed_close_timeout: 0
http.egress.downstream_cx_destroy: 492
http.egress.downstream_cx_destroy_active_rq: 0
http.egress.downstream_cx_destroy_local: 0
http.egress.downstream_cx_destroy_local_active_rq: 0
http.egress.downstream_cx_destroy_remote: 492
http.egress.downstream_cx_destroy_remote_active_rq: 0
http.egress.downstream_cx_drain_close: 0
http.egress.downstream_cx_http1_active: 16
http.egress.downstream_cx_http1_total: 508
http.egress.downstream_cx_http2_active: 0
http.egress.downstream_cx_http2_total: 0
http.egress.downstream_cx_http3_active: 0
http.egress.downstream_cx_http3_total: 0
http.egress.downstream_cx_idle_timeout: 214
http.egress.downstream_cx_max_duration_reached: 0
http.egress.downstream_cx_max_requests_reached: 0
http.egress.downstream_cx_overload_disable_keepalive: 0
http.egress.downstream_cx_protocol_error: 0
http.egress.downstream_cx_rx_bytes_buffered: 50513
http.egress.downstream_cx_rx_bytes_total: 11547428789
http.egress.downstream_cx_ssl_active: 0
http.egress.downstream_cx_ssl_total: 0
http.egress.downstream_cx_total: 508
http.egress.downstream_cx_tx_bytes_buffered: 0
http.egress.downstream_cx_tx_bytes_total: 1923092694
http.egress.downstream_cx_upgrades_active: 0
http.egress.downstream_cx_upgrades_total: 0
http.egress.downstream_flow_control_paused_reading_total: 0
http.egress.downstream_flow_control_resumed_reading_total: 0
http.egress.downstream_rq_1xx: 0
http.egress.downstream_rq_2xx: 4349090
http.egress.downstream_rq_3xx: 0
http.egress.downstream_rq_4xx: 33957
http.egress.downstream_rq_5xx: 3301
http.egress.downstream_rq_active: 0
http.egress.downstream_rq_completed: 4386348
http.egress.downstream_rq_failed_path_normalization: 0
http.egress.downstream_rq_header_timeout: 0
http.egress.downstream_rq_http1_total: 4386348
http.egress.downstream_rq_http2_total: 0
http.egress.downstream_rq_http3_total: 0
http.egress.downstream_rq_idle_timeout: 0
http.egress.downstream_rq_max_duration_reached: 0
http.egress.downstream_rq_non_relative_path: 0
http.egress.downstream_rq_overload_close: 0
http.egress.downstream_rq_redirected_with_normalized_path: 0
http.egress.downstream_rq_rejected_via_ip_detection: 0
http.egress.downstream_rq_response_before_rq_complete: 0
http.egress.downstream_rq_rx_reset: 0
http.egress.downstream_rq_timeout: 0
http.egress.downstream_rq_too_large: 0
http.egress.downstream_rq_total: 4386348
http.egress.downstream_rq_tx_reset: 0
http.egress.downstream_rq_ws_on_non_ws_route: 0
http.egress.no_cluster: 0
http.egress.no_route: 0
http.egress.passthrough_internal_redirect_bad_location: 0
http.egress.passthrough_internal_redirect_no_route: 0
http.egress.passthrough_internal_redirect_predicate: 0
http.egress.passthrough_internal_redirect_too_many_redirects: 0
http.egress.passthrough_internal_redirect_unsafe_scheme: 0
http.egress.rds.rds_egress_http_15001.config_reload: 9
http.egress.rds.rds_egress_http_15001.config_reload_time_ms: 1653300787240
http.egress.rds.rds_egress_http_15001.init_fetch_timeout: 0
http.egress.rds.rds_egress_http_15001.update_attempt: 96
http.egress.rds.rds_egress_http_15001.update_empty: 0
http.egress.rds.rds_egress_http_15001.update_failure: 86
http.egress.rds.rds_egress_http_15001.update_rejected: 0
http.egress.rds.rds_egress_http_15001.update_success: 9
http.egress.rds.rds_egress_http_15001.update_time: 1653300787240
http.egress.rds.rds_egress_http_15001.version: 17800499387772728049
http.egress.rq_direct_response: 0
http.egress.rq_redirect: 0
http.egress.rq_reset_after_downstream_response_started: 0
http.egress.rq_total: 4386348
http.egress.rs_too_large: 0
http.egress.tracing.client_enabled: 0
http.egress.tracing.health_check: 0
http.egress.tracing.not_traceable: 0
http.egress.tracing.random_sampling: 4386348
http.egress.tracing.service_forced: 0
http.egress.user_agent.ios.downstream_cx_destroy_remote_active_rq: 0
http.egress.user_agent.ios.downstream_cx_total: 136
http.egress.user_agent.ios.downstream_rq_total: 136
http.ingress.downstream_cx_active: 1
http.ingress.downstream_cx_delayed_close_timeout: 7
http.ingress.downstream_cx_destroy: 4352384
http.ingress.downstream_cx_destroy_active_rq: 0
http.ingress.downstream_cx_destroy_local: 7
http.ingress.downstream_cx_destroy_local_active_rq: 0
http.ingress.downstream_cx_destroy_remote: 4352377
http.ingress.downstream_cx_destroy_remote_active_rq: 0
http.ingress.downstream_cx_drain_close: 0
http.ingress.downstream_cx_http1_active: 1
http.ingress.downstream_cx_http1_total: 4352385
http.ingress.downstream_cx_http2_active: 0
http.ingress.downstream_cx_http2_total: 0
http.ingress.downstream_cx_http3_active: 0
http.ingress.downstream_cx_http3_total: 0
http.ingress.downstream_cx_idle_timeout: 0
http.ingress.downstream_cx_max_duration_reached: 0
http.ingress.downstream_cx_max_requests_reached: 0
http.ingress.downstream_cx_overload_disable_keepalive: 0
http.ingress.downstream_cx_protocol_error: 0
http.ingress.downstream_cx_rx_bytes_buffered: 253
http.ingress.downstream_cx_rx_bytes_total: 11011819872
http.ingress.downstream_cx_ssl_active: 0
http.ingress.downstream_cx_ssl_total: 0
http.ingress.downstream_cx_total: 4352385
http.ingress.downstream_cx_tx_bytes_buffered: 0
http.ingress.downstream_cx_tx_bytes_total: 2005905035
http.ingress.downstream_cx_upgrades_active: 0
http.ingress.downstream_cx_upgrades_total: 0
http.ingress.downstream_flow_control_paused_reading_total: 0
http.ingress.downstream_flow_control_resumed_reading_total: 0
http.ingress.downstream_rq_1xx: 0
http.ingress.downstream_rq_2xx: 4349090
http.ingress.downstream_rq_3xx: 0
http.ingress.downstream_rq_4xx: 33957
http.ingress.downstream_rq_5xx: 3301
http.ingress.downstream_rq_active: 0
http.ingress.downstream_rq_completed: 4386348
http.ingress.downstream_rq_failed_path_normalization: 0
http.ingress.downstream_rq_header_timeout: 0
http.ingress.downstream_rq_http1_total: 4386348
http.ingress.downstream_rq_http2_total: 0
http.ingress.downstream_rq_http3_total: 0
http.ingress.downstream_rq_idle_timeout: 0
http.ingress.downstream_rq_max_duration_reached: 0
http.ingress.downstream_rq_non_relative_path: 0
http.ingress.downstream_rq_overload_close: 0
http.ingress.downstream_rq_redirected_with_normalized_path: 0
http.ingress.downstream_rq_rejected_via_ip_detection: 0
http.ingress.downstream_rq_response_before_rq_complete: 0
http.ingress.downstream_rq_rx_reset: 0
http.ingress.downstream_rq_timeout: 0
http.ingress.downstream_rq_too_large: 0
http.ingress.downstream_rq_total: 4386348
http.ingress.downstream_rq_tx_reset: 0
http.ingress.downstream_rq_ws_on_non_ws_route: 0
http.ingress.no_cluster: 0
http.ingress.no_route: 0
http.ingress.passthrough_internal_redirect_bad_location: 0
http.ingress.passthrough_internal_redirect_no_route: 0
http.ingress.passthrough_internal_redirect_predicate: 0
http.ingress.passthrough_internal_redirect_too_many_redirects: 0
http.ingress.passthrough_internal_redirect_unsafe_scheme: 0
http.ingress.rds.rds_ingress_http_8088.config_reload: 1
http.ingress.rds.rds_ingress_http_8088.config_reload_time_ms: 1653134919943
http.ingress.rds.rds_ingress_http_8088.init_fetch_timeout: 0
http.ingress.rds.rds_ingress_http_8088.update_attempt: 96
http.ingress.rds.rds_ingress_http_8088.update_empty: 0
http.ingress.rds.rds_ingress_http_8088.update_failure: 86
http.ingress.rds.rds_ingress_http_8088.update_rejected: 0
http.ingress.rds.rds_ingress_http_8088.update_success: 9
http.ingress.rds.rds_ingress_http_8088.update_time: 1653300787240
http.ingress.rds.rds_ingress_http_8088.version: 17800499387772728049
http.ingress.rq_direct_response: 0
http.ingress.rq_redirect: 0
http.ingress.rq_reset_after_downstream_response_started: 0
http.ingress.rq_total: 4386348
http.ingress.rs_too_large: 0
http.ingress.tracing.client_enabled: 0
http.ingress.tracing.health_check: 0
http.ingress.tracing.not_traceable: 0
http.ingress.tracing.random_sampling: 4386348
http.ingress.tracing.service_forced: 0
http.ingress.user_agent.android.downstream_cx_destroy_remote_active_rq: 0
http.ingress.user_agent.android.downstream_cx_total: 976
http.ingress.user_agent.android.downstream_rq_total: 976
http.ingress.user_agent.ios.downstream_cx_destroy_remote_active_rq: 0
http.ingress.user_agent.ios.downstream_cx_total: 1325725
http.ingress.user_agent.ios.downstream_rq_total: 1325725
http1.dropped_headers_with_underscores: 0
http1.metadata_not_supported_error: 0
http1.requests_rejected_with_underscores_in_headers: 0
http1.response_flood: 0
listener.0.0.0.0_15001.downstream_cx_active: 16
listener.0.0.0.0_15001.downstream_cx_destroy: 492
listener.0.0.0.0_15001.downstream_cx_overflow: 0
listener.0.0.0.0_15001.downstream_cx_overload_reject: 0
listener.0.0.0.0_15001.downstream_cx_total: 508
listener.0.0.0.0_15001.downstream_cx_transport_socket_connect_timeout: 0
listener.0.0.0.0_15001.downstream_global_cx_overflow: 0
listener.0.0.0.0_15001.downstream_pre_cx_active: 0
listener.0.0.0.0_15001.downstream_pre_cx_timeout: 6
listener.0.0.0.0_15001.http.egress.downstream_rq_1xx: 0
listener.0.0.0.0_15001.http.egress.downstream_rq_2xx: 4349090
listener.0.0.0.0_15001.http.egress.downstream_rq_3xx: 0
listener.0.0.0.0_15001.http.egress.downstream_rq_4xx: 33957
listener.0.0.0.0_15001.http.egress.downstream_rq_5xx: 3301
listener.0.0.0.0_15001.http.egress.downstream_rq_completed: 4386348
listener.0.0.0.0_15001.no_filter_chain_match: 0
listener.0.0.0.0_15001.worker_0.downstream_cx_active: 4
listener.0.0.0.0_15001.worker_0.downstream_cx_total: 122
listener.0.0.0.0_15001.worker_1.downstream_cx_active: 3
listener.0.0.0.0_15001.worker_1.downstream_cx_total: 131
listener.0.0.0.0_15001.worker_2.downstream_cx_active: 5
listener.0.0.0.0_15001.worker_2.downstream_cx_total: 131
listener.0.0.0.0_15001.worker_3.downstream_cx_active: 4
listener.0.0.0.0_15001.worker_3.downstream_cx_total: 124
listener.0.0.0.0_8088.downstream_cx_active: 1
listener.0.0.0.0_8088.downstream_cx_destroy: 4352384
listener.0.0.0.0_8088.downstream_cx_overflow: 0
listener.0.0.0.0_8088.downstream_cx_overload_reject: 0
listener.0.0.0.0_8088.downstream_cx_total: 4352385
listener.0.0.0.0_8088.downstream_cx_transport_socket_connect_timeout: 0
listener.0.0.0.0_8088.downstream_global_cx_overflow: 0
listener.0.0.0.0_8088.downstream_pre_cx_active: 0
listener.0.0.0.0_8088.downstream_pre_cx_timeout: 0
listener.0.0.0.0_8088.http.ingress.downstream_rq_1xx: 0
listener.0.0.0.0_8088.http.ingress.downstream_rq_2xx: 4349090
listener.0.0.0.0_8088.http.ingress.downstream_rq_3xx: 0
listener.0.0.0.0_8088.http.ingress.downstream_rq_4xx: 33957
listener.0.0.0.0_8088.http.ingress.downstream_rq_5xx: 3301
listener.0.0.0.0_8088.http.ingress.downstream_rq_completed: 4386348
listener.0.0.0.0_8088.no_filter_chain_match: 0
listener.0.0.0.0_8088.worker_0.downstream_cx_active: 0
listener.0.0.0.0_8088.worker_0.downstream_cx_total: 1089759
listener.0.0.0.0_8088.worker_1.downstream_cx_active: 0
listener.0.0.0.0_8088.worker_1.downstream_cx_total: 1085223
listener.0.0.0.0_8088.worker_2.downstream_cx_active: 0
listener.0.0.0.0_8088.worker_2.downstream_cx_total: 1089488
listener.0.0.0.0_8088.worker_3.downstream_cx_active: 1
listener.0.0.0.0_8088.worker_3.downstream_cx_total: 1087915
listener.admin.downstream_cx_active: 2
listener.admin.downstream_cx_destroy: 33962
listener.admin.downstream_cx_overflow: 0
listener.admin.downstream_cx_overload_reject: 0
listener.admin.downstream_cx_total: 33964
listener.admin.downstream_cx_transport_socket_connect_timeout: 0
listener.admin.downstream_global_cx_overflow: 0
listener.admin.downstream_pre_cx_active: 0
listener.admin.downstream_pre_cx_timeout: 0
listener.admin.http.admin.downstream_rq_1xx: 0
listener.admin.http.admin.downstream_rq_2xx: 50935
listener.admin.http.admin.downstream_rq_3xx: 0
listener.admin.http.admin.downstream_rq_4xx: 0
listener.admin.http.admin.downstream_rq_5xx: 0
listener.admin.http.admin.downstream_rq_completed: 50935
listener.admin.main_thread.downstream_cx_active: 2
listener.admin.main_thread.downstream_cx_total: 33964
listener.admin.no_filter_chain_match: 0
listener_manager.lds.init_fetch_timeout: 0
listener_manager.lds.update_attempt: 96
listener_manager.lds.update_failure: 86
listener_manager.lds.update_rejected: 0
listener_manager.lds.update_success: 9
listener_manager.lds.update_time: 1653300787238
listener_manager.lds.version: 17800499387772728049
listener_manager.listener_added: 2
listener_manager.listener_create_failure: 0
listener_manager.listener_create_success: 8
listener_manager.listener_in_place_updated: 0
listener_manager.listener_modified: 0
listener_manager.listener_removed: 0
listener_manager.listener_stopped: 0
listener_manager.total_filter_chains_draining: 0
listener_manager.total_listeners_active: 2
listener_manager.total_listeners_draining: 0
listener_manager.total_listeners_warming: 0
listener_manager.workers_started: 1
main_thread.watchdog_mega_miss: 0
main_thread.watchdog_miss: 0
runtime.admin_overrides_active: 0
runtime.deprecated_feature_seen_since_process_start: 9
runtime.deprecated_feature_use: 9
runtime.load_error: 0
runtime.load_success: 1
runtime.num_keys: 4
runtime.num_layers: 1
runtime.override_dir_exists: 0
runtime.override_dir_not_exists: 1
server.compilation_settings.fips_mode: 0
server.concurrency: 4
server.days_until_first_cert_expiring: 2147483647
server.debug_assertion_failures: 0
server.dropped_stat_flushes: 0
server.dynamic_unknown_fields: 0
server.envoy_bug_failures: 0
server.hot_restart_epoch: 0
server.hot_restart_generation: 1
server.live: 1
server.main_thread.watchdog_mega_miss: 0
server.main_thread.watchdog_miss: 0
server.memory_allocated: 10664800
server.memory_heap_size: 46137344
server.memory_physical_size: 48080486
server.parent_connections: 0
server.seconds_until_first_ocsp_response_expiring: 0
server.state: 0
server.static_unknown_fields: 0
server.stats_recent_lookups: 9129
server.total_connections: 17
server.uptime: 169791
server.version: 3183589
server.wip_protos: 0
server.worker_0.watchdog_mega_miss: 0
server.worker_0.watchdog_miss: 0
server.worker_1.watchdog_mega_miss: 0
server.worker_1.watchdog_miss: 0
server.worker_2.watchdog_mega_miss: 0
server.worker_2.watchdog_miss: 0
server.worker_3.watchdog_mega_miss: 0
server.worker_3.watchdog_miss: 0
tls_inspector.alpn_found: 0
tls_inspector.alpn_not_found: 0
tls_inspector.client_hello_too_large: 0
tls_inspector.connection_closed: 102101
tls_inspector.read_error: 0
tls_inspector.sni_found: 0
tls_inspector.sni_not_found: 0
tls_inspector.tls_found: 0
tls_inspector.tls_not_found: 4352893
vhost.app-3.prod.vcluster.other.upstream_rq_retry: 0
vhost.app-3.prod.vcluster.other.upstream_rq_retry_limit_exceeded: 0
vhost.app-3.prod.vcluster.other.upstream_rq_retry_overflow: 0
vhost.app-3.prod.vcluster.other.upstream_rq_retry_success: 0
vhost.app-3.prod.vcluster.other.upstream_rq_timeout: 0
vhost.app-3.prod.vcluster.other.upstream_rq_total: 0
vhost.app-100035323.prod.vcluster.other.upstream_rq_retry: 0
vhost.app-100035323.prod.vcluster.other.upstream_rq_retry_limit_exceeded: 0
vhost.app-100035323.prod.vcluster.other.upstream_rq_retry_overflow: 0
vhost.app-100035323.prod.vcluster.other.upstream_rq_retry_success: 0
vhost.app-100035323.prod.vcluster.other.upstream_rq_timeout: 0
vhost.app-100035323.prod.vcluster.other.upstream_rq_total: 0
vhost.app-1.prod.vcluster.other.upstream_rq_retry: 0
vhost.app-1.prod.vcluster.other.upstream_rq_retry_limit_exceeded: 0
vhost.app-1.prod.vcluster.other.upstream_rq_retry_overflow: 0
vhost.app-1.prod.vcluster.other.upstream_rq_retry_success: 0
vhost.app-1.prod.vcluster.other.upstream_rq_timeout: 0
vhost.app-1.prod.vcluster.other.upstream_rq_total: 0
vhost.app-100035407.prod.vcluster.other.upstream_rq_retry: 0
vhost.app-100035407.prod.vcluster.other.upstream_rq_retry_limit_exceeded: 0
vhost.app-100035407.prod.vcluster.other.upstream_rq_retry_overflow: 0
vhost.app-100035407.prod.vcluster.other.upstream_rq_retry_success: 0
vhost.app-100035407.prod.vcluster.other.upstream_rq_timeout: 0
vhost.app-100035407.prod.vcluster.other.upstream_rq_total: 0
vhost.default_backend.vcluster.other.upstream_rq_retry: 0
vhost.default_backend.vcluster.other.upstream_rq_retry_limit_exceeded: 0
vhost.default_backend.vcluster.other.upstream_rq_retry_overflow: 0
vhost.default_backend.vcluster.other.upstream_rq_retry_success: 0
vhost.default_backend.vcluster.other.upstream_rq_timeout: 0
vhost.default_backend.vcluster.other.upstream_rq_total: 0
vhost.app-5.prod.vcluster.other.upstream_rq_retry: 0
vhost.app-5.prod.vcluster.other.upstream_rq_retry_limit_exceeded: 0
vhost.app-5.prod.vcluster.other.upstream_rq_retry_overflow: 0
vhost.app-5.prod.vcluster.other.upstream_rq_retry_success: 0
vhost.app-5.prod.vcluster.other.upstream_rq_timeout: 0
vhost.app-5.prod.vcluster.other.upstream_rq_total: 0
vhost.app-2.prod.vcluster.other.upstream_rq_retry: 0
vhost.app-2.prod.vcluster.other.upstream_rq_retry_limit_exceeded: 0
vhost.app-2.prod.vcluster.other.upstream_rq_retry_overflow: 0
vhost.app-2.prod.vcluster.other.upstream_rq_retry_success: 0
vhost.app-2.prod.vcluster.other.upstream_rq_timeout: 0
vhost.app-2.prod.vcluster.other.upstream_rq_total: 0
vhost.self_redirect.vcluster.other.upstream_rq_retry: 0
vhost.self_redirect.vcluster.other.upstream_rq_retry_limit_exceeded: 0
vhost.self_redirect.vcluster.other.upstream_rq_retry_overflow: 0
vhost.self_redirect.vcluster.other.upstream_rq_retry_success: 0
vhost.self_redirect.vcluster.other.upstream_rq_timeout: 0
vhost.self_redirect.vcluster.other.upstream_rq_total: 0
workers.watchdog_mega_miss: 0
workers.watchdog_miss: 0
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_connect_ms: No recorded values
cluster.cds_egress_app-mesh_app-4-canary_prod_http_8080.upstream_cx_length_ms: No recorded values
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_connect_ms: No recorded values
cluster.cds_egress_app-mesh_app-4-primary_prod_http_8080.upstream_cx_length_ms: No recorded values
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_connect_ms: No recorded values
cluster.cds_egress_app-mesh_app-3-canary_prod_http_8080.upstream_cx_length_ms: No recorded values
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.external.upstream_rq_time: P0(1.0,0.0) P25(1.025,1.0220436352221) P50(1.05,1.0487082090723592) P75(1.075,1.075372782922618) P90(1.09,1.0913715272327735) P95(1.095,1.0967044420028254) P99(1.099,9.018068965517234) P99.5(1.0995,18.77846153846174) P99.9(1.0999,26.27911111111098) P100(1.1,3400.0)
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_connect_ms: P0(nan,0.0) P25(nan,0.0) P50(nan,0.0) P75(nan,0.0) P90(nan,0.0) P95(nan,0.0) P99(nan,0.0) P99.5(nan,0.0) P99.9(nan,0.0) P100(nan,0.0)
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_cx_length_ms: P0(nan,9900.0) P25(nan,94250.0) P50(nan,310000.0) P75(nan,865000.0) P90(nan,2930000.000000001) P95(nan,4629999.999999995) P99(nan,11829999.999999983) P99.5(nan,14829999.999999983) P99.9(nan,25565999.999999974) P100(nan,26000000.0)
cluster.cds_egress_app-mesh_app-3-primary_prod_http_8080.upstream_rq_time: P0(1.0,0.0) P25(1.025,1.0220436352221) P50(1.05,1.0487082090723592) P75(1.075,1.075372782922618) P90(1.09,1.0913715272327735) P95(1.095,1.0967044420028254) P99(1.099,9.018068965517234) P99.5(1.0995,18.77846153846174) P99.9(1.0999,26.27911111111098) P100(1.1,3400.0)
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_connect_ms: No recorded values
cluster.cds_egress_app-mesh_app-100035323-canary_prod_http_8080.upstream_cx_length_ms: No recorded values
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_connect_ms: No recorded values
cluster.cds_egress_app-mesh_app-100035323-primary_prod_http_8080.upstream_cx_length_ms: No recorded values
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_connect_ms: No recorded values
cluster.cds_egress_app-mesh_app-1-canary_prod_http_8080.upstream_cx_length_ms: No recorded values
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_connect_ms: No recorded values
cluster.cds_egress_app-mesh_app-1-primary_prod_http_8080.upstream_cx_length_ms: No recorded values
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.external.upstream_rq_time: P0(nan,2.0) P25(nan,4.062862796833773) P50(nan,6.031196054254007) P75(nan,8.033276450511945) P90(nan,13.557142857142853) P95(nan,22.719047619047593) P99(nan,98.80999999999995) P99.5(nan,283.100000000004) P99.9(nan,1835.3999999999967) P100(nan,2900.0)
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.internal.upstream_rq_time: P0(nan,3.0) P25(nan,4.018992248062015) P50(nan,5.002222222222223) P75(nan,6.040217391304348) P90(nan,11.200000000000017) P95(nan,21.69999999999999) P99(nan,157.4000000000001) P99.5(nan,228.70000000000007) P99.9(nan,1357.4000000000012) P100(nan,1400.0)
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_connect_ms: P0(nan,0.0) P25(nan,0.0) P50(nan,0.0) P75(nan,0.0) P90(nan,0.0) P95(nan,0.0) P99(nan,1.077) P99.5(nan,1.0885000000000002) P99.9(nan,1.0977000000000001) P100(nan,1.1)
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_cx_length_ms: P0(nan,15000.0) P25(nan,352750.0) P50(nan,358500.0) P75(nan,412499.99999999994) P90(nan,495400.00000000006) P95(nan,497700.00000000006) P99(nan,499540.00000000006) P99.5(nan,499770.00000000006) P99.9(nan,499954.00000000006) P100(nan,500000.00000000006)
cluster.cds_egress_app-mesh_app-5-canary_prod_http_8080.upstream_rq_time: P0(nan,2.0) P25(nan,4.056482525366404) P50(nan,6.019253208868145) P75(nan,8.016611295681063) P90(nan,13.436065573770497) P95(nan,22.704761904761913) P99(nan,99.17999999999984) P99.5(nan,281.8000000000029) P99.9(nan,1821.2000000000141) P100(nan,2900.0)
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.external.upstream_rq_time: P0(1.0,1.0) P25(3.043661971830986,3.0173005675888924) P50(3.1,3.0584060169513094) P75(4.093023255813954,3.0995114663137264) P90(7.016666666666667,7.009256257237018) P95(9.1,12.174262542955322) P99(31.200000000000003,24.85297527706738) P99.5(31.599999999999994,29.584275862069003) P99.9(31.92,128.9431828978602) P100(32.0,10000.0)
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.internal.upstream_rq_time: P0(2.0,1.0) P25(3.01875,3.017443884401325) P50(3.0625,3.054934545336423) P75(5.0125,3.0924252062715203) P90(5.065,7.068905908096281) P95(5.0825,12.365060093205782) P99(5.0965,22.19343347639488) P99.5(5.09825,26.729669811320676) P99.9(5.0996500000000005,40.47405263157807) P100(5.1,9800.0)
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_connect_ms: P0(nan,0.0) P25(nan,0.0) P50(nan,0.0) P75(nan,0.0)P90(nan,0.0) P95(nan,1.0549180327868852) P99(nan,2.0712500000000005) P99.5(nan,3.0616666666666674) P99.9(nan,10.169999999999959) P100(nan,11.0)
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_cx_length_ms: P0(nan,990.0) P25(nan,998.7215477996965) P50(nan,1059.599027946537) P75(nan,5044.948630136986) P90(nan,3644745.0980392154) P95(nan,3689823.529411765) P99(nan,25504999.99999988) P99.5(nan,46752500.00000005) P99.9(nan,80701000.00000001) P100(nan,90000000.0)
cluster.cds_egress_app-mesh_app-5-primary_prod_http_8080.upstream_rq_time: P0(1.0,1.0) P25(3.0423333333333336,3.0173128848600244) P50(3.098,3.058107663559403) P75(4.09360465116279,3.0989024422587823) P90(7.005,7.011589352961315) P95(9.065000000000001,12.193492287155145) P99(31.165000000000006,24.71263688475983) P99.5(31.582499999999996,29.339922969187562) P99.9(31.9165,126.35210153482933) P100(32.0,10000.0)
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_connect_ms: No recorded values
cluster.cds_egress_app-mesh_app-2-canary_prod_http_8080.upstream_cx_length_ms: No recorded values
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.external.upstream_rq_time: P0(1.0,1.0) P25(1.0351485148514852,1.032650008193971) P50(1.0702970297029704,1.0653000163879416) P75(2.014102564102564,1.0979500245819125) P90(2.0687179487179486,2.0642441219008854) P95(2.086923076923077,2.0881616541568295) P99(3.058000000000001,6.024024797487188) P99.5(5.028999999999999,6.093191105967929) P99.9(5.085800000000001,19.01027272727383) P100(5.1,750.0)
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.internal.upstream_rq_time: P0(nan,1.0) P25(nan,1.03) P50(nan,1.06) P75(nan,1.09) P90(nan,2.0533333333333332) P95(nan,2.0933333333333333) P99(nan,4.076) P99.5(nan,4.088) P99.9(nan,4.0976) P100(nan,4.1)
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_connect_ms: P0(nan,0.0) P25(nan,0.0) P50(nan,0.0) P75(nan,0.0) P90(nan,1.046) P95(nan,1.083) P99(nan,3.063) P99.5(nan,3.0814999999999997) P99.9(nan,3.0963000000000003) P100(nan,3.1)
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_cx_length_ms: P0(nan,15000.0) P25(nan,3650000.0) P50(nan,4300000.0) P75(nan,9450000.0) P90(nan,17400000.000000004) P95(nan,21700000.0) P99(nan,29740000.0) P99.5(nan,29870000.0) P99.9(nan,29974000.0) P100(nan,30000000.0)
cluster.cds_egress_app-mesh_app-2-primary_prod_http_8080.upstream_rq_time: P0(1.0,1.0) P25(1.0351485148514852,1.032649925476211) P50(1.0702970297029704,1.0652998509524219) P75(2.014102564102564,1.0979497764286328) P90(2.0687179487179486,2.0642439347860884) P95(2.086923076923077,2.0881617428486496) P99(3.058000000000001,6.024020829889238) P99.5(5.028999999999999,6.093189122168954) P99.9(5.085800000000001,19.009181818184018) P100(5.1,750.0)
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.external.upstream_rq_time: P0(1.0,1.0) P25(2.0383177570093456,3.006821652908589) P50(3.0140845070422535,3.058425962349539) P75(4.075,4.038239950124115) P90(6.005454545454546,6.080137074867902) P95(7.08,11.887640259415578) P99(15.95999999999998,24.073976911976935) P99.5(31.24000000000001,29.683882378472237) P99.9(31.848000000000013,149.38917539267146) P100(32.0,11000.0)
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.internal.upstream_rq_time: P0(3.0,1.0) P25(3.058333333333333,3.038612681206665) P50(4.025,3.077959543503773) P75(5.025,4.053480702184691) P90(6.03,8.020957661290321) P95(6.0649999999999995,12.801542989653944) P99(6.093,23.354771573604133)P99.5(6.0965,27.948482905982935) P99.9(6.0993,67.46300000000822) P100(6.1,11000.0)
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_connect_ms: P0(nan,0.0) P25(nan,0.0) P50(nan,0.0) P75(nan,0.0) P90(nan,0.0) P95(nan,1.0795652173913042) P99(nan,2.088111111111111) P99.5(nan,3.0143333333333335) P99.9(nan,3.0828666666666664) P100(nan,3.1)
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_cx_length_ms: P0(nan,14000.0) P25(nan,3633380.2816901407) P50(nan,3668450.704225352) P75(nan,4422000.0) P90(nan,16099999.999999994) P95(nan,29699999.99999999) P99(nan,72019999.99999999) P99.5(nan,91509999.99999999) P99.9(nan,105020000.00000009) P100(nan,110000000.0)
cluster.cds_ingress_app-mesh_appmesh-gateway_prod_self_redirect_http_15001.upstream_rq_time: P0(1.0,1.0) P25(2.0399532710280375,3.0094095172078967) P50(3.0182432432432433,3.060016041689934) P75(4.078409090909091,4.03974196054733) P90(6.0075,6.081804386549366) P95(7.074166666666667,11.95779825126059) P99(15.889999999999986,24.03443112176918) P99.5(31.222499999999997,29.558283279220916) P99.9(31.84450000000001,148.29730220492954) P100(32.0,11000.0)
cluster_manager.cds.update_duration: P0(nan,0.0) P25(nan,0.0) P50(nan,0.0) P75(nan,0.0) P90(nan,13.1) P95(nan,13.549999999999999) P99(nan,13.91) P99.5(nan,13.955) P99.9(nan,13.991) P100(nan,14.0)
http.admin.downstream_cx_length_ms: P0(0.0,0.0) P25(0.0,0.0) P50(0.0,2.000059998800024) P75(2.05,2.0510019799604007) P90(2.08,2.081567168656627) P95(2.09,2.091755564888702) P99(2.098,2.0999062818743623) P99.5(2.099,3.05448409893993) P99.9(2.0998,4.041400000000003) P100(2.1,879999.9999999999)
http.admin.downstream_rq_time: P0(2.0,0.0) P25(2.05,2.0552172760996292) P50(2.1,15.472178507194245) P75(19.5,17.949607548430194) P90(19.8,19.321118755608733) P95(19.9,20.23757497857755) P99(19.98,22.659392014519057) P99.5(19.99,22.8904945553539) P99.9(19.998,23.92294444444447) P100(20.0,38.0)
http.egress.downstream_cx_length_ms: P0(nan,3600000.0) P25(nan,3634647.8873239434) P50(nan,3669295.7746478873) P75(nan,4428000.0) P90(nan,16400000.000000006) P95(nan,29799999.999999993) P99(nan,72079999.99999999) P99.5(nan,91540000.00000001) P99.9(nan,105079999.99999982) P100(nan,110000000.0)
http.egress.downstream_rq_time: P0(1.0,0.0) P25(2.000704225352113,3.000075741829494) P50(3.009333333333333,3.047658772535418) P75(4.025675675675676,3.0952418032413425) P90(5.095,6.063754737214434) P95(7.041666666666667,11.74336073367112) P99(15.899999999999977,23.774528795811406) P99.5(31.224999999999994,29.316057017544008) P99.9(31.845,148.0518783068792) P100(32.0,11000.0)
http.egress.rds.rds_egress_http_15001.update_duration: P0(nan,1.0) P25(nan,1.025) P50(nan,1.05) P75(nan,1.075) P90(nan,1.09) P95(nan,1.095) P99(nan,1.099) P99.5(nan,1.0995) P99.9(nan,1.0999) P100(nan,1.1)
http.egress.user_agent.ios.downstream_cx_length_ms: P0(nan,3600000.0) P25(nan,3635106.3829787234) P50(nan,3670212.765957447) P75(nan,6950000.0) P90(nan,21799999.999999996) P95(nan,47399999.99999999) P99(nan,69840000.0) P99.5(nan,91340000.0) P99.9(nan,91868000.0) P100(nan,92000000.0)
http.ingress.downstream_cx_length_ms: P0(2.0,1.0) P25(3.04453125,4.016042704988579) P50(4.060869565217391,4.098696623511976) P75(5.087068965517242,5.0869566329451255) P90(7.0875,8.036492270395316) P95(9.03,13.393543493188965) P99(16.899999999999977,25.32678497485961) P99.5(32.224999999999994,31.00438400743826) P99.9(32.845,150.64532567049338) P100(33.0,879999.9999999999)
http.ingress.downstream_rq_time: P0(1.0,1.0) P25(2.0518,3.014656273915426) P50(3.0296610169491527,3.076407388490412) P75(4.097972972972973,4.0675015017168885) P90(6.022307692307693,6.092430411515358) P95(7.090833333333333,12.200154059227208) P99(15.889999999999986,24.158241143567352) P99.5(31.222499999999997,29.72537008481123) P99.9(31.84450000000001,148.4579743589752) P100(32.0,11000.0)
http.ingress.rds.rds_ingress_http_8088.update_duration: P0(nan,0.0) P25(nan,0.0) P50(nan,0.0) P75(nan,0.0) P90(nan,4.01) P95(nan,4.055) P99(nan,4.091) P99.5(nan,4.0955) P99.9(nan,4.0991) P100(nan,4.1)
http.ingress.user_agent.android.downstream_cx_length_ms: P0(3.0,2.0) P25(3.025,2.0838487972508593) P50(3.05,3.0480487804878047) P75(3.075,4.013362068965518)P90(3.09,4.076465517241379) P95(3.095,4.0975) P99(3.099,8.0248) P99.5(3.0995,9.056000000000001) P99.9(3.0999,140.24) P100(3.1,150.0)
http.ingress.user_agent.ios.downstream_cx_length_ms: P0(3.0,2.0) P25(4.072727272727272,4.058431164943404) P50(5.06,5.030753251260693) P75(7.011111111111111,5.098417906943508) P90(8.07,10.617400793650784) P95(11.699999999999996,14.453390714658312) P99(32.28,26.731525204653114) P99.5(32.64,30.27848526703504) P99.9(32.928,56.70566666666701) P100(33.0,11000.0)
listener.0.0.0.0_15001.downstream_cx_length_ms: P0(nan,3600000.0) P25(nan,3634647.8873239434) P50(nan,3669295.7746478873) P75(nan,4428000.0) P90(nan,16400000.000000006) P95(nan,29799999.999999993) P99(nan,72079999.99999999) P99.5(nan,91540000.00000001) P99.9(nan,105079999.99999982) P100(nan,110000000.0)
listener.0.0.0.0_8088.downstream_cx_length_ms: P0(2.0,1.0) P25(3.042063492063492,4.0145679120410716) P50(4.060294117647059,4.0973874158449375) P75(5.085593220338983,5.086013613560341) P90(7.08235294117647,8.034672148209825) P95(9.03,13.380818700369083) P99(16.899999999999977,25.314193738925084) P99.5(32.224999999999994,30.990860798759094) P99.9(32.845,150.62616858237078) P100(33.0,879999.9999999999)
listener.admin.downstream_cx_length_ms: P0(0.0,0.0) P25(0.0,0.0) P50(0.0,2.0000539665407446) P75(2.05,2.0509654014511005) P90(2.08,2.0815122623973137) P95(2.09,2.0916945493793846) P99(2.098,2.0998403789650415) P99.5(2.099,3.0524505494505503) P99.9(2.0998,4.037737500000003) P100(2.1,879999.9999999999)
listener_manager.lds.update_duration: P0(nan,0.0) P25(nan,0.0) P50(nan,0.0) P75(nan,0.0) P90(nan,11.1) P95(nan,11.549999999999999) P99(nan,11.91) P99.5(nan,11.955) P99.9(nan,11.991) P100(nan,12.0)
server.initialization_time_ms: P0(nan,220.00000000000003) P25(nan,222.50000000000003) P50(nan,225.00000000000003) P75(nan,227.50000000000003) P90(nan,229.00000000000003) P95(nan,229.50000000000003) P99(nan,229.90000000000003) P99.5(nan,229.95000000000002) P99.9(nan,229.99000000000004) P100(nan,230.00000000000003)
Envoy 支持通过配置UDP 侦听器 DNS Filter来响应 DNS 请求。
DNS 过滤器支持响应 A 和 AAAA 记录的转发查询。答案是从静态配置的资源、集群或外部 DNS 服务器中发现的。过滤器将返回最多 512 个字节的 DNS 响应。如果域配置了多个地址,或者集群配置了多个端点,Envoy 将返回每个发现的地址,但不超过上述大小限制。
配置http过滤器类型为envoy.filters.http.router,我们的用例中需要这个过滤器将地址127.0.0.1:10000上收到的请求路由到www.envoyproxy.io, 除此之外,envoy还有其他过滤器,比如带宽限制过滤器envoy.filters.http.bandwidth_limit,速率限制过滤器envoy.filters.http.ratelimit等。具体可参考:http 过滤器。
指定了cluster的名称为service_envoyproxy_io, 上文中的listener监听到的请求在37行就会被路由到该cluster,listener_0监听来自下游主机的请求后,路由到service_envoyproxy_io,service_envoyproxy_io会将该请求发送给相应的上游端点进行处理。
当 Envoy 匹配路由时,它使用以下过程:
HTTP 请求的主机或:authority标头与虚拟主机匹配。
按顺序检查虚拟主机中的 每个路由条目。如果存在匹配,则使用该路由并且不再进行路由检查。
独立地,按顺序检查虚拟主机中的每个虚拟集群。如果匹配,则使用虚拟集群并且不再进行虚拟集群检查。
通过HTTP 过滤器和 自定义集群的组合,Envoy 支持 HTTP 动态转发代理。这意味着 Envoy 可以在不事先了解所有配置的 DNS 地址的情况下执行 HTTP 代理的角色,同时仍保留 Envoy 的绝大多数优势,包括异步 DNS 解析。实施工作如下:
如果目标 DNS 主机尚未在缓存中,则动态转发代理 HTTP 过滤器用于暂停请求。
Envoy 将开始异步解析 DNS 地址,并在解析完成时解锁任何等待响应的请求。
由于 DNS 地址已在缓存中,因此不会阻止任何未来的请求。解析过程类似于逻辑 DNS服务发现类型,在任何给定时间都会记住一个目标地址。
所有已知主机都存储在动态转发代理集群中,以便它们可以显示在管理输出中。
一个特殊的负载均衡器将在转发期间根据 HTTP 主机/权限标头选择要使用的正确主机。
一段时间未使用的主机会受到将清除它们的 TTL。
当上游集群配置了 TLS 上下文时,Envoy 将自动对解析的主机名执行 SAN 验证,并通过 SNI 指定主机名。
上述实现细节意味着在稳定状态下,Envoy 可以转发大量 HTTP 代理流量,而所有 DNS 解析都在后台异步进行。此外,所有其他 Envoy 过滤器和扩展都可以与动态转发代理支持结合使用,包括身份验证、RBAC、速率限制等 。
https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers
动态代理转发
当传入连接通过 iptables REDIRECT 或 TPROXY 目标或使用代理协议重定向到 Envoy 时,可以使用原始目标集群。在这些情况下,路由到原始目标集群的请求被转发到由重定向元数据寻址的上游主机,而无需任何显式主机配置或上游主机发现。当上游主机的空闲时间超过 cleanup_interval(默认为 5000 毫秒)时,连接到上游主机的连接被池化,并且未使用的主机被刷新。如果原始目标地址不可用,则不打开上游连接。Envoy 还可以从HTTP 标头中获取原始目标。原始目标服务发现必须与原始目标负载均衡器一起使用.
此过滤器应配置为envoy.filters.listener.original_src名称。
原始源侦听器过滤器在 Envoy 的上游端复制连接的下游远程地址。例如,如果下游连接使用 IP 地址连接到 Envoy 10.1.2.3,那么 Envoy 将使用源 IP 连接到上游10.1.2.3。
易于操作是 Envoy 的主要目标之一。除了强大的统计数据和本地管理界面之外,Envoy 还能够“热”或“实时”重启自身。这意味着 Envoy 可以完全重新加载自身(代码和配置),而不会在排水过程中丢弃现有连接。热重启功能具有以下通用架构:
两个活动进程使用基本的 RPC 协议通过 unix 域套接字相互通信。所有计数器都通过 unix 域从旧进程发送到新进程,并且除了标有NeverImport的那些之外,还会传输仪表。热重启完成后,从旧进程传来的 gauge 会被清理,但 会保留server.hot_restart_generation 统计等特殊的 gauge 。
新进程在向旧进程请求侦听套接字的副本之前,会完全初始化自身(加载配置、执行初始服务发现和健康检查阶段等)。新进程开始监听,然后告诉旧进程开始排空。
在耗尽阶段,旧进程尝试正常关闭现有连接。如何完成取决于配置的过滤器。排水时间可通过 –drain-time-s选项配置,随着时间的推移,排水变得更加激进。
在耗尽序列之后,新的 Envoy 进程告诉旧的 Envoy 进程自行关闭。该时间可通过–parent-shutdown-time-s选项进行配置。
Envoy 的热重启支持被设计为即使新的 Envoy 进程和旧的 Envoy 进程在不同的容器中运行,它也能正常工作。进程之间的通信仅使用 unix 域套接字进行。
源代码分发中包含一个用 Python 编写的示例重启程序/父进程。此父进程可用于标准进程控制实用程序,例如 monit/runit/etc。
HTTP 标头可以在x-forwarded-for标头中携带请求的原始 IP 地址 。上游服务器可以使用此标头来确定下游远程地址。Envoy 也可以使用这个 header 来选择 Original Src HTTP Filter使用的 IP 地址。
HTTP 标头方法有一些缺点:
它仅适用于 HTTP。
上游主机可能不支持它。
它需要仔细配置。
Envoy 在其代码库和文档中使用以下术语:
集群:具有一组端点的逻辑服务,Envoy 将请求转发到这些端点。
下游:连接到 Envoy 的实体。这可能是本地应用程序(在边车模型中)或网络节点。在非 Sidecar 模型中,这是一个远程客户端。
端点:实现逻辑服务的网络节点。它们被分组到集群中。集群中的端点位于Envoy 代理的上游。
过滤器:连接或请求处理管道中的模块,提供请求处理的某些方面。Unix 的一个类比是小型实用程序(过滤器)与 Unix 管道(过滤器链)的组合。
过滤器链:一系列过滤器。
侦听器:Envoy 模块,负责绑定到 IP/端口、接受新的 TCP 连接(或 UDP 数据报)并协调面向下游的请求处理方面。
上游:Envoy 在转发服务请求时连接的端点(网络节点)。这可能是本地应用程序(在边车模型中)或网络节点。在非边车模型中,这对应于远程后端。

从 Envoy 初始化配置文件中,我们可以看出 Istio 中 Envoy sidecar 真正的配置实际上是由两部分组成的。Pilot-agent 在启动 Envoy 时将 xDS server 信息通过静态资源的方式配置到 Envoy 的初始化配置文件中,Envoy 启动后再通过 xDS server 获取网格中的服务信息、路由规则等动态资源。

重要参考资料!!!:https://www.servicemesher.com/istio-handbook/concepts/sidecar-traffic-route.html
Envoy 采用 listener 来接收并处理 downstream 发过来的请求,listener 采用了插件式的架构,可以通过配置不同的 filter 在 listener 中插入不同的处理逻辑。
Listener 可以绑定到 IP Socket 或者 Unix Domain Socket 上,以接收来自客户端的请求;也可以不绑定,而是接收从其他 listener 转发来的数据。Istio 利用了 Envoy listener 的这一特点,通过 VirtualOutboundListener 在一个端口接收所有出向请求,然后再按照请求的端口分别转发给不同的 listener 分别处理。
Istio 在 Envoy 中配置了一个在 15001 端口监听的虚拟入口监听器。Iptable 规则将 Envoy 所在 pod 的对外请求拦截后发向本地的 15001 端口,该监听器接收后并不进行业务处理,而是根据请求的目的端口分发给其他监听器处理。这就是该监听器取名为 “virtual”(虚拟)监听器的原因。
Envoy 是如何做到按请求的目的端口进行分发的呢? 从下面 VirtualOutbound listener 的配置中可以看到 use_original_dest 属性被设置为 true, 这表示该监听器在接收到来自 downstream 的请求后,会将请求转交给匹配该请求原目的地址的 listener(即名字格式为 0.0.0.0_ 请求目的端口 的 listener)进行处理。
如果在 Enovy 的配置中找不到匹配请求目的端口的 listener,则将会根据 Istio 的 outboundTrafficPolicy 全局配置选项进行处理。存在两种情况:
PassthroughCluster 的 TCP proxy filter,找不到匹配端口 listener 的请求会被该 TCP
proxy filter 处理,请求将会被发送到其 IP 头中的原始目的地地址。下图是 bookinfo 例子中 productpage 服务中 Enovy Proxy 的 Virutal Outbound Listener 配置。由于 outboundTrafficPolicy 的默认配置为 ALLOW_ANY,因此 listener 的 filterchain 中第二个 filter chain 中是一个 upstream cluster 为 PassthroughCluster 的 TCP proxy filter。注意该 filter 没有 filter_chain_match 匹配条件,因此如果进入该 listener 的请求在配置中找不到匹配其目的端口的 listener,就会缺省进入该 filter 进行处理。
filterchain 中的第一个 filter chain 中是一个 upstream cluster 为 BlackHoleCluster 的 TCP proxy filter,该 filter 设置了 filter_chain_match 匹配条件,只有发向 10.40.0.18 这个 IP 的出向请求才会进入该 filter 处理。10.40.0.18 是 productpage 服务自身的IP地址。该 filter 的目的是为了防止服务向自己发送请求可能导致的死循环。
该 cluster 的 type 被设置为 ORIGINAL_DST 类型, 表明任何发向该 cluster 的请求都会被直接发送到其请求中的原始目地的,Envoy 不会对请求进行重新路由。
这是一个特殊的 cluster ,其中并没有配置后端处理请求的 host。如其名字所表明的一样,请求进入该 cluster 后如同进入了一个黑洞,将被丢弃掉,而不是发向一个 upstream host。
https://www.envoyproxy.io/docs/envoy/latest/operations/admin
默认UUID格式:xxxxxx-xxx-xxx-xxx-xxxxxxx
可以看到,Envoy 中实际生效的配置是由初始化配置文件中的静态配置和从 Pilot 获取的动态配置一起组成的。因此只对 envoy-rev0 .json 进行分析并不能看到网络中流量管理的全貌。那么有没有办法可以看到 Envoy 中实际生效的完整配置呢?Envoy 提供了相应的管理接口,我们可以采用下面的命令导出 productpage-v1 服务 sidecar 的完整配置。
kubectl exec -it productpage-v1-6d8bc58dd7-ts8kw -c istio-proxy curl http://127.0.0.1:15000/config_dump > config_dump
该配置文件的内容如下:
{
"configs": [
{
"@type": "type.googleapis.com/envoy.admin.v3.BootstrapConfigDump",
"bootstrap": {},
"last_updated": "2020-03-11T08:14:03.630Z"
},
{
"@type": "type.googleapis.com/envoy.admin.v3.ClustersConfigDump",
"version_info": "2020-03-11T08:14:06Z/23",
"static_clusters": [
...
],
"dynamic_active_clusters": [
...
]
},
{
"@type": "type.googleapis.com/envoy.admin.v3.ListenersConfigDump",
"version_info": "2020-03-11T08:13:39Z/22",
"static_listeners": [
...
],
"dynamic_listeners": [
...
]
},
{
"@type": "type.googleapis.com/envoy.admin.v3.RoutesConfigDump",
"static_route_configs": [
...
],
"dynamic_route_configs": [
...
]
},
{
"@type": "type.googleapis.com/envoy.admin.v3.SecretsConfigDump",
"dynamic_active_secrets": [
...
]
}
]
}
从导出的文件中可以看到 Envoy 中主要由以下几部分内容组成:
下面我们对该配置文件中和流量路由相关的配置一一进行详细分析。
从名字可以看出这是 Envoy 的初始化配置,打开该节点,可以看到其中的内容和 envoy-rev0.json 是一致的,这里不再赘述。 需要注意的是在 bootstrap 部分配置的一些内容也会被用于其他部分,例如 clusters 部分就包含了 bootstrap 中定义的一些静态 cluster 资源。
{
"@type": "type.googleapis.com/envoy.admin.v3.BootstrapConfigDump",
"bootstrap": {
"node": {
...
},
"stats_config": {
...
},
"admin": {
...
},
"dynamic_resources": {
...
},
"static_resources": {
...
},
"tracing": {
...
}
},
"last_updated": "2020-03-11T08:14:03.630Z"
}
这部分配置定义了 Envoy 中所有的 cluster,即服务集群,cluster 中包含一个到多个 endpoint,每个 endpoint 都可以提供服务,Envoy 根据负载均衡算法将请求发送到这些 endpoint 中。
从配置文件结构中可以看到,在 productpage 的 clusters 配置中包含 static_clusters 和 dynamic_active_clusters 两部分,其中 static_clusters 是来自于 envoy-rev0.json 的初始化配置中的 prometheus_stats、xDS server、zipkin server 信息。dynamic_active_clusters 是 Envoy 通过 xDS 接口从 Istio 控制平面获取的服务信息。
其中 dynamic cluster 又分为以下几类:
这部分的 cluster 占了绝大多数,该类 cluster 对应于 Envoy 所在节点的外部服务。以 reviews 为例,对于 productpage 来说,reviews 是一个外部服务,因此其 cluster 名称中包含 outbound 字样。
从 reviews 服务对应的 cluster 配置中可以看到,其类型为 EDS,即表示该 cluster 的 endpoint 来自于动态发现,动态发现中 eds_config 则指向了ads,最终指向 static resource 中配置的 xds-grpc cluster,即 Pilot 的地址。
{
"version_info": "2020-03-11T08:13:39Z/22",
"cluster": {
"@type": "type.googleapis.com/envoy.api.v2.Cluster",
"name": "outbound|9080||reviews.default.svc.cluster.local",
"type": "EDS",
"eds_cluster_config": {
"eds_config": {
"ads": {}
},
"service_name": "outbound|9080||reviews.default.svc.cluster.local"
},
"connect_timeout": "1s",
"circuit_breakers": {},
"filters": [],
"transport_socket_matches": []
},
"last_updated": "2020-03-11T08:14:04.664Z"
}
可以通过 Pilot 的调试接口获取该 cluster 的 endpoint:
curl http://10.97.222.108:15014/debug/edsz > pilot_eds_dump
从导出的文件内容可以看到,reviews cluster 配置了3个 endpoint 地址,是 reviews 的 pod ip。
{
"clusterName": "outbound|9080||reviews.default.svc.cluster.local",
"endpoints": [
{
"lbEndpoints": [
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.40.0.15",
"portValue": 9080
}
}
},
"metadata": {},
"loadBalancingWeight": 1
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.40.0.16",
"portValue": 9080
}
}
},
"metadata": {},
"loadBalancingWeight": 1
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.40.0.17",
"portValue": 9080
}
}
},
"metadata": {},
"loadBalancingWeight": 1
}
],
"loadBalancingWeight": 3
}
]
}
对于 Envoy 来说,inbound cluster 对应于入向请求的 upstream 集群, 即 Envoy 自身所在节点的服务。对于 productpage Pod 上的 Envoy,其对应的 Inbound cluster 只有一个,即 productpage。该 cluster 对应的 host 为127.0.0.1,即回环地址上 productpage 的监听端口。由于 iptable 规则中排除了127.0.0.1,入站请求通过该 Inbound cluster 处理后将跳过 Envoy,直接发送给 productpage 进程处理。
CNCF 云原生景观全景图:https://landscape.cncf.io/?zoom=400

面对庞大的技术栈,有选择性的进行研究:
偏向 应用层 的技术栈:





云原生 应用层 技术栈,大量使用了gRPC作为网络通讯的协议/框架
尤其是go-sdk,在大量sidecar模式架构中进行使用。
Go语言gRPC技术栈

鉴于 gRPC 在应用层技术栈的广泛使用和重要性,对其有源码级了解是有价值的。
● 服务非对象、消息非引用 —— 促进微服务的系统间粗粒度消息交互设计理念,同时避免分布式对象的陷阱和分布式计算的谬误。
和 DCOM 及 EJB 完全相反的。只有数据,不包含逻辑
分布式计算谬误
- 网络可靠;
- 延迟为零;
- 带宽是无限的;
- 网络安全;
- 拓扑不变;
- 有一名管理员;
- 运输成本为零;
- 网络是同质的。
● 普遍并且简单 —— 该基础框架应该在任何流行的开发平台上适用,并且易于被个人在自己的平台上构建。它在CPU和内存有限的设备上也应该切实可行。
协议的设计应考虑全面的应用场景:边缘节点、serverless等
● 免费并且开源 —— 所有人可免费使用基本特性。以友好的许可协议开源方式发布所有交付件。
避免具体技术锁定,应是可插拔的
● 互通性 —— 该报文协议(Wire Protocol)必须遵循普通互联网基础框架。
同现有网络设施的兼容性和连通性
- over HTTP
- over TCP
- over UDP
● 通用并且高性能 —— 该框架应该适用于绝大多数用例场景,相比针对特定用例的框架,该框架只会牺牲一点性能。
要通用性,不要极端性能优化
● 分层的 —— 该框架的关键是必须能够独立演进。对报文格式(Wire Format)的修改不应该影响应用层。
抽象的编程模型,与具体实现解耦
● 负载无关的 —— 不同的服务需要使用不同的消息类型和编码,例如protocol buffers、JSON、XML和Thrift,协议上和实现上必须满足这样的诉求。类似地,对负载压缩的诉求也因应用场景和负载类型不同而不同,协议上应该支持可插拔的压缩机制。
Payload agnostic,RPC 框架不应该规定用的是什么 payload 格式
● 流 —— 存储系统依赖于流和流控来传递大数据集。像语音转文本或股票代码等其它服务,依靠流表达时间相关的消息序列。
在某种程度上,可以将其理解为类似消息系统的消息队列。
● 阻塞式和非阻塞式 —— 支持异步和同步处理在客户端和服务端间交互的消息序列。这是在某些平台上缩放和处理流的关键。
用户态线程阻塞模型性能较差,应使用NIO编程模型,支持异步非阻塞式的交互。
● 取消和超时 —— 有的操作可能会用时很长,客户端运行正常时,可以通过取消操作让服务端回收资源。当任务因果链被追踪时,取消可以级联。客户端可能会被告知调用超时,此时服务就可以根据客户端的需求来调整自己的行为。
支持重置功能(RST_STREAM 帧)
● Lameducking —— 服务端必须支持优雅关闭,优雅关闭时拒绝新请求,但继续处理正在运行中的请求。
优先级设计(Http2)
- 针对流ID
- 流级联设计
● 流控 —— 在客户端和服务端之间,计算能力和网络容量往往是不平衡的。流控可以更好的缓冲管理,以及保护系统免受来自异常活跃对端的拒绝服务(DOS)攻击。
维护缓冲区:窗口/背压/租约
基于HTTP2缓冲区的流控机制 基于应用层语义的流控
● 可插拔的 —— 数据传输协议(Wire Protocol)只是功能完备API基础框架的一部分。大型分布式系统需要安全、健康检查、负载均衡和故障恢复、监控、跟踪、日志等。实现上应该提供扩展点,以允许插入这些特性和默认实现。
● API扩展 —— 可能的话,在服务间协作的扩展应该最好使用接口扩展,而不是协议扩展。这种类型的扩展可以包括健康检查、服务内省、负载监测和负载均衡分配。
● 元数据交换 —— 常见的横切关注点,如认证或跟踪,依赖数据交换,但这不是服务公共接口中的一部分。部署依赖于他们将这些特性以不同速度演进到服务暴露的个别API的能力。
metadata与payload分离
● 标准化状态码 —— 客户端通常以有限的方式响应API调用返回的错误。应该限制状态代码名字空间,使得这些错误处理决定更清晰。如果需要更丰富的特定域的状态,可以使用元数据交换机制来提供。
● 对端平等 —— 人机交互强调 客户端->服务端 模型,机器之间交互任意一端都可作为客户端/服务端
● 长链接 —— 为服务端推送和双向流交互提供基础。
完善的应用层状态码命名空间


gRPC 引入了三个新概念:通道、远程过程调用(RPC) 和消息。三者之间的关系很简单:每个通道可能有很多 RPC,而每个 RPC 可能有很多消息。


通道是 gRPC 中的一个关键概念。HTTP/2 中的流支持在单个连接上进行多个并发会话;通道通过在多个并发连接上启用多个流来扩展这个概念。
通道代表到端点的虚拟连接,实际上可能由许多 HTTP/2 连接支持。RPC 与连接相关联。RPC 实际上是普通的 HTTP/2 流。消息与 RPC 相关联并作为 HTTP/2 数据帧发送。更具体地说,消息是在数据帧之上分层的。一个数据帧可能有很多 gRPC 消息,或者如果一个 gRPC 消息非常大它可能跨越多个数据帧。
gRPC 对 16kb 的数据帧使用 HTTP/2 默认的最大大小。超过 16kb 的消息可能跨越多个数据帧,而低于该大小的消息可能与一些其他消息共享一个数据帧。
H2 Conn:
startCall
call.sendMessage
call.halfMessage
TODO
https://juejin.cn/post/6844903984524705800 https://grpc.io/docs/what-is-grpc/core-concepts/ https://grpc.io/blog/grpc-on-http2/ https://grpc.io/blog/yikyak/ https://grpc.io/blog/grpc-load-balancing/ https://segmentfault.com/a/1190000019608421
在这些示例中,我们将介绍应用程序运行时会发生什么curl example.com。虽然在curl这里使用,但同样适用于几乎所有客户端。
当您向域发送请求时,客户端将执行 DNS 解析以将其解析为 IP 地址。无论 Istio 设置如何,都会发生这种情况,因为 Istio 只拦截网络流量;它不能改变您的应用程序的行为或发送 DNS 请求的决定。在下面的示例中,example.com解析为192.0.2.0.
$ curl example.com -v
接下来,请求将被 Istio 拦截。此时,Istio 将同时看到主机名(来自Host: example.com标头)和目标地址 ( 192.0.2.0:80)。Istio 使用此信息来确定预期目的地。 了解流量路由可以深入了解此行为的工作原理。
如果客户端无法解析 DNS 请求,请求将在 Istio 收到之前终止。这意味着如果将请求发送到 Istio 已知的主机名(例如 a VirtualService)但不发送到 DNS 服务器,则请求将失败。Istio DNS 代理可以改变这种行为。
一旦 Istio 确定了预期的目的地,它必须选择要发送到的地址。由于 Istio 的高级负载均衡功能,这通常不是客户端发送的原始 IP 地址。根据服务配置,Istio 有几种不同的方式来执行此操作。
使用客户端的原始 IP 地址(192.0.2.0在上面的示例中)。这是ServiceEntrytype resolution: NONE(默认)和headlessServices的情况。 在一组静态 IP 地址上进行负载平衡。这是ServiceEntrytype的情况,将使用resolution: STATICall ,或 standard ,将使用 all 。spec.endpointsServicesEndpoints 使用 DNS 定期解析地址,并对所有结果进行负载平衡。这是ServiceEntrytype的情况resolution: DNS。 请注意,在所有情况下,Istio 代理中的 DNS 解析都与用户应用程序中的 DNS 解析正交。即使客户端进行 DNS 解析,代理也可能会忽略解析的 IP 地址并使用自己的 IP 地址,这可能来自静态 IP 列表或通过自己的 DNS 解析(可能是相同的主机名或不同的主机名)。
DNS 代理 3分钟阅读 页面测试
除了捕获应用程序流量之外,Istio 还可以捕获 DNS 请求以提高网格的性能和可用性。在代理 DNS 时,来自应用程序的所有 DNS 请求都将被重定向到 sidecar,其中存储了本地域名到 IP 地址的映射。如果请求可以被 Sidecar 处理,它将直接向应用程序返回响应,避免往返上游 DNS 服务器。否则,请求将按照标准/etc/resolv.confDNS 配置向上游转发。
虽然 Kubernetes 为 Kubernetes 提供了开箱即用的 DNS 解析Service,但任何自定义ServiceEntry的都不会被识别。使用此功能,ServiceEntry无需自定义配置 DNS 服务器即可解析地址。对于 Kubernetes Service,DNS 响应将是相同的,但负载减少kube-dns并提高了性能。
此功能也可用于在 Kubernetes 之外运行的服务。这意味着可以解决所有内部服务,而无需笨拙的变通方法将 Kubernetes DNS 条目暴露在集群之外。
入门 默认情况下当前未启用此功能。要启用它,请使用以下设置安装 Istio:
$ cat «EOF | istioctl install -y -f - apiVersion: install.istio.io/v1alpha1 kind: IstioOperator spec: meshConfig: defaultConfig: proxyMetadata:
ISTIO_META_DNS_CAPTURE: “true”
ISTIO_META_DNS_AUTO_ALLOCATE: “true” EOF
这也可以使用proxy.istio.io/configannotation在每个 pod 的基础上启用。
使用 部署到 VM 时,默认情况下将启用基本 DNS 代理。istioctl workload entry configure DNS 捕获在行动 要尝试 DNS 捕获,首先ServiceEntry为一些外部服务设置一个:
apiVersion: networking.istio.io/v1beta1 kind: ServiceEntry metadata: name: external-address spec: addresses:
如果没有 DNS 捕获,请求address.internal可能无法解析。启用此功能后,您应该根据配置获得响应address:
$ curl -v address.internal
地址自动分配 在上面的示例中,您有一个用于发送请求的服务的预定义 IP 地址。但是,访问没有稳定地址的外部服务是很常见的,而是依赖 DNS。在这种情况下,DNS 代理将没有足够的信息来返回响应,并且需要向上游转发 DNS 请求。
这对于 TCP 流量来说尤其成问题。与基于Host标头路由的 HTTP 请求不同,TCP 携带的信息要少得多。您只能在目标 IP 和端口号上进行路由。因为后端没有一个稳定的IP,所以也不能基于它进行路由,只留下端口号,当多个ServiceEntryTCP服务共享同一个端口时,会导致冲突。
为了解决这些问题,DNS 代理还支持为ServiceEntry未明确定义地址的 s 自动分配地址。这是由ISTIO_META_DNS_AUTO_ALLOCATE选项配置的。
启用此功能后,DNS 响应将为每个ServiceEntry. 然后将代理配置为将请求匹配到此 IP 地址,并将请求转发到相应的ServiceEntry.
由于此功能会修改 DNS 响应,因此它可能无法与所有应用程序兼容。 要尝试这一点,请配置另一个ServiceEntry:
apiVersion: networking.istio.io/v1beta1 kind: ServiceEntry metadata: name: external-auto spec: hosts:
现在,发送请求:
$ curl -v auto.internal
如您所见,请求被发送到一个自动分配的地址,240.240.0.1. 这些地址将从240.240.0.0/16保留的 IP 地址范围中挑选,以避免与实际服务发生冲突。
参考文档:https://www.servicemesher.com/istio-handbook/concepts/sidecar-injection.html
Sidecar模式是一种单节点、多容器的应用设计形式。Sidecar主张以额外的容器来扩展或增强主容器,而这个额外的容器被称为Sidecar容器
sidecar 模式也符合当前微服务的以下特点:

在软件架构中, Sidecar 连接到父应用并且为其添加扩展或者增强功能。Sidecar 应用与主应用程序松散耦合。它可以屏蔽不同编程语言的差异,统一实现微服务的可观察性、监控、日志记录、配置、断路器等功能。
使用 sidecar 模式部署服务网格时,无需在节点上运行代理,但是集群中将运行多个相同的 sidecar 副本。在 sidecar 部署方式中,每个应用的容器旁都会部署一个伴生容器(如 Envoy 或 MOSN),这个容器称之为 sidecar 容器。Sidecar 接管进出应用容器的所有流量。在 Kubernetes 的 Pod 中,在原有的应用容器旁边注入一个 Sidecar 容器,两个容器共享存储、网络等资源,可以广义的将这个包含了 sidecar 容器的 Pod 理解为一台主机,两个容器共享主机资源。
因其独特的部署结构,使得 sidecar 模式具有以下优势:
借助于K8S良好的可拓展性,使用sidecar模式可以享受到分布式系统中的规模化效率红利。相比于这种效率提升,我们可以容许性能上的开销。
func main() {
if err := rootCmd.Execute(); err != nil {
log.Error(err)
os.Exit(-1)
}
}
借助 docker-compose.yml 文件,开发人员可定义一组相关服务,通过部署命令将其部署为组合应用程序。 它还配置其依赖项关系和运行时配置。多个容器使用同一个network。

version: '3.4'
services:
webmvc:
image: eshop/web
environment:
- CatalogUrl=http://catalog-api
- OrderingUrl=http://ordering-api
ports:
- "80:80"
depends_on:
- catalog-api
- ordering-api
catalog-api:
image: eshop/catalog-api
environment:
- ConnectionString=Server=sqldata;Port=1433;Database=CatalogDB;…
ports:
- "81:80"
depends_on:
- sqldata
ordering-api:
image: eshop/ordering-api
environment:
- ConnectionString=Server=sqldata;Database=OrderingDb;…
ports:
- "82:80"
extra_hosts:
- "CESARDLBOOKVHD:10.0.75.1"
depends_on:
- sqldata
sqldata:
image: mcr.microsoft.com/mssql/server:latest
environment:
- SA_PASSWORD=Pass@word
- ACCEPT_EULA=Y
ports:
- "5433:1433"
# BASE_DISTRIBUTION is used to switch between the old base distribution and distroless base images
ARG BASE_DISTRIBUTION=debug
# Version is the base image version from the TLD Makefile
ARG BASE_VERSION=latest
# The following section is used as base image if BASE_DISTRIBUTION=debug
FROM gcr.io/istio-release/base:${BASE_VERSION} as debug
# The following section is used as base image if BASE_DISTRIBUTION=distroless
# This image is a custom built debian11 distroless image with multiarchitecture support.
# It is built on the base distroless image, with iptables binary and libraries added
# The source can be found at https://github.com/istio/distroless/tree/iptables
# This version is from commit 105e1319a176a5156205b9e351b4e2016363f00d.
FROM gcr.io/istio-release/iptables@sha256:bae9287d64be13179b7bc794ec3db26bd5c5fe3fb591c484992366314c9a7d3d as distroless
# This will build the final image based on either debug or distroless from above
# hadolint ignore=DL3006
FROM ${BASE_DISTRIBUTION:-debug}
WORKDIR /
ARG proxy_version
ARG istio_version
ARG SIDECAR=envoy
# Copy Envoy bootstrap templates used by pilot-agent
COPY envoy_bootstrap.json /var/lib/istio/envoy/envoy_bootstrap_tmpl.json
COPY gcp_envoy_bootstrap.json /var/lib/istio/envoy/gcp_envoy_bootstrap_tmpl.json
# Install Envoy.
ARG TARGETARCH
COPY ${TARGETARCH:-amd64}/${SIDECAR} /usr/local/bin/${SIDECAR}
# Environment variable indicating the exact proxy sha - for debugging or version-specific configs
ENV ISTIO_META_ISTIO_PROXY_SHA $proxy_version
# Environment variable indicating the exact build, for debugging
ENV ISTIO_META_ISTIO_VERSION $istio_version
ARG TARGETARCH
COPY ${TARGETARCH:-amd64}/pilot-agent /usr/local/bin/pilot-agent
COPY stats-filter.wasm /etc/istio/extensions/stats-filter.wasm
COPY stats-filter.compiled.wasm /etc/istio/extensions/stats-filter.compiled.wasm
COPY metadata-exchange-filter.wasm /etc/istio/extensions/metadata-exchange-filter.wasm
COPY metadata-exchange-filter.compiled.wasm /etc/istio/extensions/metadata-exchange-filter.compiled.wasm
# The pilot-agent will bootstrap Envoy.
ENTRYPOINT ["/usr/local/bin/pilot-agent"]
Kompose是个转换工具,可将 compose(即 Docker Compose)所组装的所有内容 转换成容器编排器(Kubernetes 或 OpenShift)可识别的形式。
要将 docker-compose.yml 转换为 kubectl 可用的文件,请运行 kompose convert 命令进行转换,然后运行 kubectl create -f
K8S作为云原生操作系统的定位,其设计理念是"微内核"架构。
Kubernetes 的 apiserver 一开始就有 AdmissionController 的设计,这个设计和各类 Web 框架中的 Filter 很像,就是一个插件化的责任链,责任链中的每个插件针对 apiserver 收到的请求做一些操作或校验。分类
Kubernetes 中的许多高级功能需要启用准入控制器才能正确支持该功能。

通过创建webhook资源,利用k8s的webhook能力实现pod的自动注入。
基于 Kubernetes 的 突变 webhook 入驻控制器(mutating webhook addmission controller 的自动 sidecar 注入方式。
实现流程大致如下:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: "pod-policy.example.com"
webhooks:
- name: "pod-policy.example.com"
rules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE"]
resources: ["pods"]
scope: "Namespaced"
clientConfig:
service:
namespace: "example-namespace"
name: "example-service"
caBundle: "Ci0tLS0tQk...<base64-encoded PEM bundle containing the CA that signed the webhook's serving certificate>...tLS0K"
admissionReviewVersions: ["v1", "v1beta1"]
sideEffects: None
timeoutSeconds: 5
Kubernetes虽然提供了多种容器编排对象,例如Deployment、StatefulSet、DeamonSet、Job等,还有多种基础资源封装例如ConfigMap、Secret、Serivce等,但是一个应用往往有多个服务,有的可能还要依赖持久化存储,当这些服务之间直接互相依赖,需要有一定的组合的情况下,使用YAML文件的方式配置应用往往十分繁琐还容易出错,这时候就需要服务编排工具。

Istio 使用 ValidatingAdmissionWebhooks 验证 Istio 配置,使用 MutatingAdmissionWebhooks 自动将 Sidecar 代理注入至用户 Pod。
它使用 MutatingWebhook 机制在 pod 创建的时候将 sidecar 的容器和卷添加到每个 pod 的模版里。

containers:
- image: docker.io/istio/examples-bookinfo-productpage-v1:1.15.0 # 应用镜像
name: productpage
ports:
- containerPort: 9080
- args:
- proxy
- sidecar
- --domain
- $(POD_NAMESPACE).svc.cluster.local
- --configPath
- /etc/istio/proxy
- --binaryPath
- /usr/local/bin/envoy
- --serviceCluster
- productpage.$(POD_NAMESPACE)
- --drainDuration
- 45s
- --parentShutdownDuration
- 1m0s
- --discoveryAddress
- istiod.istio-system.svc:15012
- --zipkinAddress
- zipkin.istio-system:9411
- --proxyLogLevel=warning
- --proxyComponentLogLevel=misc:error
- --connectTimeout
- 10s
- --proxyAdminPort
- "15000"
- --concurrency
- "2"
- --controlPlaneAuthPolicy
- NONE
- --dnsRefreshRate
- 300s
- --statusPort
- "15020"
- --trust-domain=cluster.local
- --controlPlaneBootstrap=false
image: docker.io/istio/proxyv2:1.5.1 # sidecar proxy
name: istio-proxy
ports:
- containerPort: 15090
name: http-envoy-prom
protocol: TCP
initContainers:
- command:
- istio-iptables
- -p
- "15001"
- -z
- "15006"
- -u
- "1337"
- -m
- REDIRECT
- -i
- '*'
- -x
- ""
- -b
- '*'
- -d
- 15090,15020
image: docker.io/istio/proxyv2:1.5.1 # init 容器
name: istio-init
启动的最小配置集,可以和镜像集成在一起。
可以使用K8S的静态配置进行配置:
控制面下发,全量/增量推送:

Init 容器是一种专用容器,它在应用程序容器启动之前运行,用来包含一些应用镜像中不存在的实用工具或安装脚本。
一个 Pod 中可以指定多个 Init 容器,如果指定了多个,那么 Init 容器将会按顺序依次运行。只有当前面的 Init 容器必须运行成功后,才可以运行下一个 Init 容器。当所有的 Init 容器运行完成后,Kubernetes 才初始化 Pod 和运行应用容器。
Init 容器使用 Linux Namespace,所以相对应用程序容器来说具有不同的文件系统视图。因此,它们能够具有访问 Secret 的权限,而应用程序容器则不能。
在 Pod 启动过程中,Init 容器会按顺序在网络和数据卷初始化之后启动。每个容器必须在下一个容器启动之前成功退出。如果由于运行时或失败退出,将导致容器启动失败,它会根据 Pod 的 restartPolicy 指定的策略进行重试。然而,如果 Pod 的 restartPolicy 设置为 Always,Init 容器失败时会使用 RestartPolicy 策略。
在所有的 Init 容器没有成功之前,Pod 将不会变成 Ready 状态。Init 容器的端口将不会在 Service中进行聚集。 正在初始化中的 Pod 处于 Pending 状态,但应该会将 Initializing 状态设置为 true。Init 容器运行完成以后就会自动终止。

该容器存在的意义就是让 sidecar 代理可以拦截所有的进出 pod 的流量,15090 端口(Mixer 使用)和 15092 端口(Ingress Gateway)除外的所有入站(inbound)流量重定向到 15006 端口(sidecar),再拦截应用容器的出站(outbound)流量经过 sidecar 处理(通过 15001 端口监听)后再出站。
istio-iptables [flags]
-p: 指定重定向所有 TCP 流量的 sidecar 端口(默认为 $ENVOY_PORT = 15001)
-m: 指定入站连接重定向到 sidecar 的模式,“REDIRECT” 或 “TPROXY”(默认为 $ISTIO_INBOUND_INTERCEPTION_MODE)
-b: 逗号分隔的入站端口列表,其流量将重定向到 Envoy(可选)。使用通配符 “*” 表示重定向所有端口。为空时表示禁用所有入站重定向(默认为 $ISTIO_INBOUND_PORTS)
-d: 指定要从重定向到 sidecar 中排除的入站端口列表(可选),以逗号格式分隔。使用通配符“*” 表示重定向所有入站流量(默认为 $ISTIO_LOCAL_EXCLUDE_PORTS)
-o:逗号分隔的出站端口列表,不包括重定向到 Envoy 的端口。
-i: 指定重定向到 sidecar 的 IP 地址范围(可选),以逗号分隔的 CIDR 格式列表。使用通配符 “*” 表示重定向所有出站流量。空列表将禁用所有出站重定向(默认为 $ISTIO_SERVICE_CIDR)
-x: 指定将从重定向中排除的 IP 地址范围,以逗号分隔的 CIDR 格式列表。使用通配符 “*” 表示重定向所有出站流量(默认为 $ISTIO_SERVICE_EXCLUDE_CIDR)。
-k:逗号分隔的虚拟接口列表,其入站流量(来自虚拟机的)将被视为出站流量。
-g:指定不应用重定向的用户的 GID。(默认值与 -u param 相同)
-u:指定不应用重定向的用户的 UID。通常情况下,这是代理容器的 UID(默认值是 1337,即 istio-proxy 的 UID)。
-z: 所有进入 pod/VM 的 TCP 流量应被重定向到的端口(默认 $INBOUND_CAPTURE_PORT = 15006)。

使用单容器多进程模型。
在proxy镜像中,pilot-agent 负责的工作包括:

[/pilot-agent/main/initStatusServer]
对于 ready 检查,调用的路径为/healthz/ready, 并配合设置的端口 applicationPorts 通过 Envoy 的 admin 端口进行对应的端口进行检查,用于决定 Envoy 是否已经 ready 接受相对应的流量。
检查原理是通过本地管理端口,如 http://127.0.0.1:15000/listeners 获取 Envoy 当前监听的全部端口,然后将配置的端口 applicationPorts 在监听的端口中进行查找,来决定 Envoy 是否 ready。

检查的路径为 /url 路径,在 header 中设置 istio-app-probe-port 端口,使用 访问路径中的 url 来进行检查,最终调用的是 http://127.0.0.1:istio-app-probe-port/url,头部设置的全部参数也都会传递到别检测的服务端口上;
[/pilot-agent/main/istio_agent.NewAgent/initXdsProxy]
[/pilot-agent/main/istio_agent.NewAgent]

代理属于第三方提供,istio是对其进行了管理和拓展。
如果代理本身属于istio,是可以实现单进程模型。
由agent负责配置监听和下发。
解耦配置管理和运行时,同时可以对proxy进程进行热重启。
ip netns exec cni-bf783dac-fe05-cb35-4d5a-848449119b19 iptables -L -t nat
-A PREROUTING -p tcp -j ISTIO_INBOUND # PREROUTING全部转发到INBOUND,PREROUTING发生在流入的数据包进入路由表之前
-A OUTPUT -p tcp -j ISTIO_OUTPUT # 由本机产生的数据向外转发的
-A ISTIO_INBOUND -p tcp -m tcp --dport 22 -j RETURN # 22 15090 15021 15020的不转发到ISTIO_REDIRECT
-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT # 剩余的流量都转发到ISTIO_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006 # 转发到15006
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN # 127.0.0.6是InboundPassthroughBindIpv4,代表原地址是passthrough的流量都直接跳过,不劫持
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT #lo网卡出流量,目标地址不是localhost的,且为同用户的流量进入ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN # lo网卡出流量 非同用户的不劫持
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN # 剩下的同用户的都跳过
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT # lo网卡出流量,目标地址非本地,同用户组的流量进入ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN # lo网卡出流量非同组的不劫持
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN # 剩余的同用户的不劫持
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN # 剩余的目标地址为127的不劫持
-A ISTIO_OUTPUT -j ISTIO_REDIRECT # 剩下的都进入 ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001 # 转达到15001 outbond
COMMIT





要实现 应用容器(进程) 和 Sidecar容器(进程) 之间的交互。需要完成流量交互的功能。
对于跨容器(进程)的流量交互,主要有以下两种交互模式:
proxyless sidecar 和 Servicemesh 在方式上的差异:暴露 API 还是代理通讯协议。
在 Servicemesh 中,“零侵入”是一个非常强调的特性,为此不惜引入 iptables 等流量劫持方案。“零侵入”在某些特殊场景下会发挥巨大的优势,如旧有应用不做改造的前提下接入 servicemesh。好处自然不言而喻,但零侵入也有自身的限制:客户端必须能发出符合服务器端要求的网络通讯请求,这个过程外部无法插手。
代理模式强调的是 原协议转发,应用进程无感。往往使用操作系统提供的流量劫持功能。
适合于统一的网络协议栈(HTTP),仅实现原协议层面的控制(路由、重试等)。
应用场景:envoy流量代理
发展方向:内核高性能(eBPF)
几乎没有开销是来自代理本身的逻辑。开销是通过注入代理,将网络流量重定向到它,终止连接和启动新的连接而增加的。
应用进程明确转发,协议自由切换,更丰富的应用层语义。往往使用grpc建立跨进程的链接。
适合异构协议栈,支持更丰富的功能,支持协议之上应用层语义的控制。
应用场景:multi runtime
发展方向:grpc proxyless sidecar
gRPC 项目对 xDS API 有很好的支持,也就是说你可以管理 gRPC 工作负载,而不需要同时部署 Envoy sidecar。
ServiceMesh 在微服务领域已经非常流行,越来越多的公司开始在内部落地,ServiceMesh 带来的业务解耦,平滑升级等优势大大提高了中间件的迭代效率。
不过 ServiceMesh 只解决了服务间通讯的需求,而现实中的分布式应用存在更多的需求。而效仿 ServiceMesh 将应用需要的其他分布式能力外移到各种 Sidecar Runtime,这逐渐演变成了一个趋势。

与其依靠多个代理来实现不同的目的(例如网络代理,缓存代理,绑定代理),不如使用一个 Mecha 提供所有这些能力。
Mecha 强调是“提供能力”,而不是通讯代理。
Mecha 和 Micrologic 之间的交互是开放而有 API 标准的,Mecha 和 Micrologic 之间的“协议”体现在 API 上,而不是 TCP 通讯协议。这提供了一个契机:一个统一 Micrologic 和 Mecha 之间通讯方式的契机。
面对千变万化的需求和复杂的应用环境,期望 Sidecar 本身的控制面和数据面来覆盖所有的场景显然是不现实的。强大、全面往往是因为易扩展。
使用类似K8S webhook的机制,将自定义插件注入到控制面中,作为单独的服务执行。
此类扩展可以完全无侵入的实现数据平面的增强。而且 API 的抽象屏蔽了数据平面的实现细节,扩展会具有更好的可移植性;独立进程执行和部署,具备更强的伸缩性。但是 webhook 模 也引入了大量额外的外部调用和数据交互,带来了巨大的性能开销。

WebAssembly,简称 WASM,是一个二进制指令集,最初是跑在浏览器上来解决 JavaScript 的性能问题,但由于它良好的安全性,隔离性以及语言无关性等优秀特性,很快人们便开始让它跑在浏览器之外的地方,随着 WASI 定义的出现,只需要一个 WASM 运行时,就可以让 WASM 文件随处执行。
WASM 字节码不能直接在任何 CPU 架构上执行,但由于它与机器码非常相近,因此能够以非常快的速度被 WASM 引擎(或者也可以称之为 WASM 虚拟机)翻译为对应架构的机器码,获得和机器码相近的性能。
WASM 本身是为 Web 而设计,因此天然具有跨平台支持;同时,通过 WASM 虚拟机的沙箱隔离,也使得执行 WASM 字节码相比于直接执行机器码有更高的安全性。
Envoy 在可拓展性方面做了两方面的工作:
第一,提供了名为 lua 的特殊扩展,允许控制面通过 xDS 协议动态下发 Lua 脚本并由 Envoy 解释执行。
第二,也是本节的主题,Envoy 引入了 WASM 技术用于开发 Envoy 扩展。

Istio 的扩展机制使用 Proxy-Wasm 应用二进制接口(ABI)规范,提供了一套代理无关的流媒体 API 和实用功能,可以用任何有合适 SDK 的语言来实现。
扩展 Istio 的功能,满足你的特定需求,需要三个步骤:
Mesh: dns拦截(udp)


pip install awscli
curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin
eksctl version
curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.21.2/2021-07-05/bin/linux/amd64/kubectl
chmod +x ./kubectl
mkdir -p $HOME/bin && cp ./kubectl $HOME/bin/kubectl && export PATH=$PATH:$HOME/bin
echo 'export PATH=$PATH:$HOME/bin' >> ~/.bashrc
kubectl version --short --client
aws eks update-kubeconfig --region ap-southeast-1 --name xxx-eks-cluster
更新kube的config文件
默认使用的是客户端 用户/角色 的权限。
AWS IAM权限映射到EKS的RBCA权限上。
登录命令
kubectl run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox sh
kubectl scale deployment/nginx-proxy-deployment --replicas=2
MOSN 作为 Sidecar 使用时的流量劫持方案。
MOSN 作为 Sidecar 和业务容器部署在同一个 Pod 中时,需要使得业务应用的 Inbound 和 Outbound 服务请求都能够经过 Sidecar 处理。区别于 Istio 社区使用 iptables 做流量透明劫持,MOSN 目前使用的是流量接管方案,并在积极探索适用于大规模流量下的透明劫持方案
区别于 Istio 社区的 iptables 流量劫持方案,MOSN 使用的流量接管的方案如下:
经过上述的服务发现过程,流量转发过程就显得非常自然了:
上文通过在服务注册过程中把服务端地址替换成本机监听端口实现了轻量级的“流量劫持”,在存在注册中心,且调用端和服务端同时使用特定SDK的场景中可以很好的工作,如果不满足这两个条件,则无法流量劫持。为了降低对于应用程序的要求,需要引入透明劫持。
iptables 通过 NAT 表的 redirect 动作执行流量重定向,通过 syn 包触发新建 nefilter 层的连接,后续报文到来时查找连接转换目的地址与端口。新建连接时同时会记录下原始目的地址,应用程序可以通过(SOL_IP、SO_ORIGINAL_DST)获取到真实的目的地址。
iptables 劫持原理如下图所示:

使用 iptables 做流量劫持时存在的问题 目前 Istio 使用 iptables 实现透明劫持,主要存在以下三个问题:
上述几个问题并非在所有场景中都存在,比方说某些场景下,连接数并不多,且 NAT 表未被使用到的情况下,iptables 是一个满足要求的简单方案。为了适配更加广泛的场景,透明劫持需要解决上述三个问题。
tproxy 可以用于 inbound 流量的重定向,且无需改变报文中的目的 IP/端口,不需要执行连接跟踪,不会出现 conntrack 模块创建大量连接的问题。受限于内核版本,tproxy 应用于 outbound 存在一定缺陷。目前 Istio 支持通过 tproxy 处理 inbound 流量。
为了适配更多应用场景,outbound 方向通过 hook connect 来实现,实现原理如下:

无论采用哪种透明劫持方案,均需要解决获取真实目的 IP/端口的问题,使用 iptables 方案通过 getsockopt 方式获取,tproxy 可以直接读取目的地址,通过修改调用接口,hook connect 方案读取方式类似于tproxy。
实现透明劫持后,在内核版本满足要求(4.16以上)的前提下,通过 sockmap 可以缩短报文穿越路径,进而改善 outbound 方向的转发性能。
总结来看,如果应用程序通过注册中心发布/订阅服务时,可以结合注册中心劫持流量;在需要用到透明劫持的场景,如果性能压力不大,使用 iptables redirect 即可,大并发压力下使用 tproxy 与hook connect 结合的方案。
本文介绍 MOSN 支持平滑升级的原因和解决方案,对于平滑升级的一些基础概念,大家可以通过 Nginx vs Enovy vs Mosn 平滑升级原理解析了解。
先简单介绍一下为什么 Nginx 和 Envoy 不需要具备 MOSN 这样的连接无损迁移方案,主要还是跟业务场景相关,Nginx 和 Envoy 主要支持的是 HTTP1 和 HTTP2 协议,HTTP1使用 connection: Close,HTTP2 使用 Goaway Frame 都可以让 Client 端主动断链接,然后新建链接到新的 New process,但是针对 Dubbo、SOFA PRC 等常见的多路复用协议,它们是没有控制帧,Old process 的链接如果断了就会影响请求的。
参考:Envoy热重启
一般的升级做法就是切走应用的流量,比如自己UnPub掉服务,等待一段时间没有请求之后,升级MOSN,升级好之后再Pub服务,整个过程比较耗时,并且会有一段时间是不提供服务的,还要考虑应用的水位,在大规模场景下,就很难兼顾评估。MOSN 为了满足自身业务场景,开发了长连接迁移方案,把这条链接迁移到 New process 上,整个过程对 Client 透明,不需要重新建链接,达到请求无损的平滑升级。

基于 MOSN 本身的扩展机制,我们完成了最初版本的协议扩展接入。但是在实践过程中,我们发现这并不是一件容易的事情:
基于这些实践痛点,我们设计了 MOSN 多协议框架,希望可以降低私有协议的接入成本,加快普及 ServiceMesh 架构的落地推进。
第一个要介绍的是目前发展势头强劲的 Envoy。从图上可以看出,Envoy 支持四层的读写过滤器扩展、基于 HTTP 的七层读写过滤器扩展以及对应的 Router/Upstream 实现。如果想要基于 Envoy 的扩展框架实现 L7 协议接入,目前的普遍做法是基于 L4 filter 封装相应的 L7 codec,在此基础之上再实现对应的协议路由等能力,无法复用 HTTP L7 的扩展框架。

MOSN 的 Plugin 机制包含了两部分内容,一是 MOSN 自定义的 Plugin 框架,它支持通过在 MOSN 中实现 agent 与一个独立的进程进行交互来完成 MOSN 扩展能力的实现。二是基于 Golang 的 Plugin 框架,通过动态库(SO)加载的方式,实现 MOSN 的扩展。其中动态库加载的方式目前还存在一些局限性,还处于 beta 阶段。
MOSN 的 Plugin 框架是 MOSN 封装的一个可以让 MOSN 通过 gRPC 和独立进程进行交互的方式,它包含两部分:

MOSN 使用 Go 语言编写,Go 语言在生产效率,内存安全上有比较强的保障,同时 Go 语言在云原生时代有广泛的库生态系统,性能在 Mesh 场景下被我们评估以及实践是可以接受的。所以 MOSN 对于使用 Go、Java 等语言的公司和个人的心智成本更低。
MOSN 支持多协议框架,用户可以比较容易的接入私有协议,具有统一的路由框架;
多进程的插件机制,可以通过插件框架很方便的扩展独立 MOSN 进程的插件,做一些其他管理,旁路等的功能模块扩展;
具备中国密码合规的传输层国密算法支持;
采用 WebAssembly(Wasm) 技术,给 MOSN 实现了一个安全隔离的沙箱环境,让扩展程序能够运行在隔离沙箱之中,并对其资源、能力进行严格限制,使程序故障止步于沙箱,从而实现安全隔离的目标。本文将着重叙述 MOSN 中的 Wasm 扩展框架,并介绍我们在 Proxy-Wasm 这一代理扩展规范上的工作。

上图为 MOSN Wasm 扩展框架的整体示意图。如图所示,对于 MOSN 的任意扩展点(Codec、NetworkFilter、StreamFilter 等),用户均能够通过 Wasm 扩展框架,以隔离沙箱的形式运行自定义的扩展代码。而 MOSN 与 Wasm 扩展代码之间的交互,是通过 Proxy-Wasm 标准 ABI 来完成的。
当我们在讨论 Wasm 时,都明白 Wasm 能够提供一个安全隔离的沙箱环境,但并不是每个人都了解 Wasm 实现隔离沙箱的技术原理。这时又要搬出计算机科学中的至理名言: “计算机科学领域的任何问题都可以通过增加一个间接的中间层来解决”。Wasm 实际上也是通过引用一个“中间层”来实现的安全隔离。简单来说,Wasm 通过一个运行时(Runtime)来运行 Wasm 沙箱扩展,每个 Wasm 沙箱都有其独立的线性内存空间和一组导入/导出模块。

一方面,每个 Wasm 沙箱都有其独立的线性内存空间,其内存模型如上图所示。Wasm 代码只能通过简单的 load/store 等指令访问线性内存空间的有限部分,并通过符号(下标)的方式来间接访问函数、全局变量等,杜绝了类似 C 语言中访问任意内存地址的骚操作。同时,用于间接调用函数的符号表对于 Wasm 代码而言是只读的,从而保证 Wasm 代码的执行是受控的。此外,Wasm 沙箱的整个线性内存空间由宿主机(Wasm Runtime)分配及管理,通过严格的内存管理保证沙箱的隔离性。
另一方面,Wasm 也规定了代码中任何可能产生外部影响的操作只能通过导入/导出模块来实现。以 C 语言为例,我们可以直接通过系统调用来访问系统的环境变量、文件、网络等资源。而在 Wasm 的世界中,并不存在系统调用相关的指令,任何对外部资源的访问必须通过导入模块来间接实现。以文件读写为例,在 Wasm 中要想进行文件读写,需要宿主机提供实现文件读写功能的导入函数,Wasm 代码调用该导入函数,由宿主机间接进行文件读写,再将操作结果返回给 Wasm 扩展。在上述过程中,实际的文件读写操作由宿主机完成,宿主机对这一过程有绝对的控制权,包括但不限于只允许读写指定文件、限制读写内容、完全禁止读写等。

本小节将介绍 MOSN 具体是如何跟 Wasm 扩展程序进行交互的。先说结论: MOSN 跟 Wasm 扩展代码之间的交互采用的是社区规范: Proxy-Wasm
Proxy-Wasm 规范定义了宿主机与 Wasm 扩展程序之间的交互细节,包括 API 列表、函数调用规范以及数据传输规范这几个方面。其中,API 列表包含了 L4/L7、property、metrics、日志等方面的扩展点,涵盖了网络代理场景下所需的大部分交互点,且可以划分为宿主侧扩展和 Wasm 侧扩展点。这里简单展示规范中的部分内容,完整内容请参考 spec。
当ELB的IP变化时,DNS刷新不及时,导致499超时问题。
带有 AWS ELB 的 Nginx 代理:间歇性网关超时(499 或 502)的关键处理
往往是Nginx服务有问题
跨域中一些ngxin的配置:https://juejin.cn/post/6995374680114741279
跨域的解释:https://juejin.cn/post/6844903938232156167#heading-46
CORS:https://developer.mozilla.org/zh-CN/docs/Web/HTTP/CORS
https://kubernetes.io/zh/docs/tasks/run-application/run-stateless-application-deployment/
配置文件 /etc/nginx/conf.d
默认日志 /var/logs/nginx
配置日志 /etc/nginx/logs/
测试:nginx -t
重启:nginx -s reload
查看日志:
域名解析:
error_log logs/error_www.abc.com.log error;
如果proxy_pass中hostname使用了占位符,需要指定resovler的dns服务器ip
使用resolver指定dns服务器时,hostname要带上 .svc.cluster.local
可以查看/etc/resov.conf中的dns服务器,来指定nginx的resovler
https://www.cnblogs.com/dadonggg/p/7797281.html
https://segmentfault.com/a/1190000019894251
Capa主要解决应用的混合云问题,涉及到对接适配多种云的底层实现。
Capa本身不提供功能,而是由底层的各种云原生的能力来提供。
对于Capa的运维工作,更多的精力是在同底层的云原生能力打交道。
所以了解云原生中底层组件的DevOps是很有必要的。
本目录主要为对云原生生态中DevOps的学习和实践。
演练目标:
演练手段:
演练:
# kubectl annotate {kind} {name} experiment.chaos-mesh.org/pause=true
kubectl annotate {networkchaos} {network-delay} experiment.chaos-mesh.org/pause=true
# kubectl annotate {kind} {name} experiment.chaos-mesh.org/pause-
kubectl annotate networkchaos network-delay experiment.chaos-mesh.org/pause-
kubectl delete -f network-delay.yaml
# or delete the chaos object directly
kubectl delete {networkchaos} {network-delay}
向指定的 Pod 中注入故障,使得该 Pod 在一段时间内处于不可用的状态。
配置:
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
name: pod-failure-example
namespace: chaos-testing
spec:
action: pod-failure
mode: one
duration: '30s'
selector:
labelSelectors:
'app.kubernetes.io/component': 'tikv'
实践:
简言之,这里有几个关于使用 “Pod Failure” 混沌实验的建议:
Pod Failure 混沌实验将会改变目标 Pod 中每个容器的 image 为 “pause image”,这是一个特殊的镜像,不会执行任何操作。我们使用 gcr.io/google-containers/pause:latest 作为默认的 “pause image”,你可以在 helm values controllerManager.podChaos.podFailure.pauseImage 中更改它。
下载 “pause image” 将会消耗时间,并且这个时间将会被计入实验的持续时间中。所以你可能会发现,“实际受影响的时间” 可能会比配置的时间短。这是推荐设置可用的 “pause image” 的另一个原因。
另外一个迷惑的点是,“pause image” 可以在未配置 command 的容器中正常工作。所以,如果容器未配置 command,livenessProbe 和 readinessProbe,它将会被视为 Running 和 Ready,即使它已经被改变为 “pause image”,并且实际上不提供正常功能, 或者被视为不可用。所以建议为容器配置 livenessProbe 和 readinessProbe。
配置:
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
name: delay
spec:
action: delay
# one(表示随机选出一个符合条件的 Pod)
mode: one
selector:
namespaces:
- {default}
labelSelectors:
'app': '{web-show}'
delay:
latency: '{10ms}'
# 表示延迟时间的时间长度与前一次延迟时长的相关性。取值范围:[0, 100]
correlation: '100'
# 表示延迟时间的变化范围
jitter: '0ms'
实践:
AWSChaos 能够帮助你模拟指定的 AWS 实例发生故障的情景。目前,AWSChaos 支持以下类型的故障:
实践:
ec2级别的故障,可以由aws原生的fis来进行。
要运行实验,您首先要创建一个实验模板。实验模板是实验的蓝图。它包含实验的 操作、目标和停止条件。

AWS FIS 支持针对以下 AWS 服务的目标资源的操作:
在开始使用 AWS FIS 进行实验之前,请执行以下步骤:
参考: https://docs.aws.amazon.com/fis/latest/userguide/getting-started-iam-service-role.html
chaos-testing chaos-controller-manager-7788488bbd-dq8k8 1/1 Running 0 45h
chaos-testing chaos-controller-manager-7788488bbd-dvqdj 1/1 Running 1 (34h ago) 45h
chaos-testing chaos-controller-manager-7788488bbd-ffwss 1/1 Running 0 45h
chaos-testing chaos-daemon-4cqnm 1/1 Running 0 44h
chaos-testing chaos-daemon-djrth 1/1 Running 0 44h
chaos-testing chaos-daemon-hpc59 0/1 Pending 0 44h
chaos-testing chaos-daemon-jppg9 1/1 Running 0 44h
chaos-testing chaos-daemon-kgdwb 1/1 Running 0 44h
chaos-testing chaos-daemon-mqh5k 1/1 Running 0 44h
chaos-testing chaos-daemon-qw5cn 1/1 Running 0 44h
chaos-testing chaos-daemon-v8ptz 1/1 Running 0 44h
chaos-testing chaos-daemon-xzkm4 1/1 Running 0 44h
chaos-testing chaos-daemon-zlt67 1/1 Running 0 44h
chaos-testing chaos-dashboard-84ffc4bb9-4nxf9 1/1 Running 0 45h
chaos-daemon每个node一个,以daemonset方式运行。
控制混沌实验生效的范围
Chaos Mesh 提供了以下两种方式用于控制混沌实验生效的范围:
helm upgrade chaos-mesh chaos-mesh/chaos-mesh -n chaos-testing --set controllerManager.enableFilterNamespace=true
在开启 FilterNamespace 功能后,Chaos Mesh 将只会向包含有 chaos-mesh.org/inject=enabled 注解的命名空间注入故障。因此,在进行混沌实验之前,你需要为允许混沌实验的命名空间添加该注解,其他命名空间则受到保护不会被注入故障。
A:降级kubectl到和k8s server匹配的版本上
确保命令行执行时,具有相应的权限。
如在AWS时,需确保客户端的IAM角色具有EKS的操作权限。
Chaos Mesh 使用 Kubernetes 原生的 RBAC 功能来管理用户角色和权限。用户在创建、查看、管理混沌实验时,需要拥有 chaos-mesh.org 这个 apiGroups 下混沌实验自定义资源的相应权限。
对于AWS而言,IAM会映射到RBAC上,所以需要IAM具有权限。
在 Chaos Mesh 中,根据实验的执行流程,一个混沌实验的生命周期主要分为以下四个阶段:
可以使用 kubectl describe 命令查看此混沌实验对象的 Status 和 Events,从而确定实验结果。
kubectl describe podchaos pod-failure-tikv -n tidb-cluster
...
Status:
Conditions:
Reason:
Status: False
Type: Paused
Reason:
Status: True
Type: Selected
Reason:
Status: True
Type: AllInjected
Reason:
Status: False
Type: AllRecovered
Experiment:
Container Records:
Id: tidb-cluster/basic-tikv-0
Phase: Injected
Selector Key: .
Desired Phase: Run
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal FinalizerInited 39s finalizer Finalizer has been inited
Normal Paused 39s desiredphase Experiment has been paused
Normal Updated 39s finalizer Successfully update finalizer of resource
Normal Updated 39s records Successfully update records of resource
Normal Updated 39s desiredphase Successfully update desiredPhase of resource
Normal Started 17s desiredphase Experiment has started
Normal Updated 17s desiredphase Successfully update desiredPhase of resource
Normal Applied 17s records Successfully apply chaos for tidb-cluster/basic-tikv-0
Normal Updated 17s records Successfully update records of resource
依据混沌实验的执行流程,Status 提供了以下四类状态记录:
可以通过这四类状态记录推断出当前混沌实验的真实运行情况。例如:
当 Paused、Selected、AllRecoverd 的状态是 True 且 AllInjected 的状态是 False时,说明当前实验处在暂停状态。
当 Paused 为 True 的时,说明当前实验处在暂停状态,但是如果此时的 Selected 值为 False,那么可以进一步得出此混沌实验无法选出待测试目标。
注意 你可以从上述的四类实验记录组合中可以推导出更多的信息,例如当 Paused 为 True 的时候,说明混沌实验处在暂停状态,但是如果此时的 Selected 值为 False,那么可以进一步得出此混沌实验无法选出待测试目标。
事件列表中包含混沌实验整个生命周期中的操作记录,可以帮助确定混沌实验状态并排除问题。
复杂的注入工作流。
TODO
用户和角色类似,也是一组权限的集合,但是用户应该更多的面向 人/客户端?
角色可以理解为,一组权限的集合代表,当需要权限进行操作时,往往是基于 角色 这个实体进行赋予。
就是一组权限,或者单个权限。可以赋给 用户、角色。
由于本身网络环境是开放的,所以仅需进行IAM认证即可连接到其中。
以下为一种示例流程:
[aws]
aws_access_key_id=
aws_secret_access_key=
aws_session_token=
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export AWS_SESSION_TOKEN=
这时,作为客户端,需要使用的是 用户 这个概念,然后具有 用户 上面对应的权限。
这时,作为客户端,需要使用的是 用户 这个概念,然后具有 用户 上面对应的权限。
连接VPN后,操作步骤同A中所示。
日志、指标、告警等功能
使用 CloudWatch 查询语法进行 日志组 维度的全局日志搜索。
fields @timestamp, @message
| sort @timestamp desc
| limit 20
使用 ‘|’ 来作为 “或” 语义的分隔符,使用多个查询条件
使用 ‘#’ 在开头,作为该行查询语句的注释,可使用快捷键 “ctrl + /”
fields @timestamp, @message
| filter @message like /(?i)(queryData)/
| filter @message like /(?i)("500")/
| sort @timestamp desc
| limit 100
fields @timestamp, @message
| filter log._trace_id = '12345'
| sort @timestamp desc
| limit 20
fields @timestamp, @message
| filter kubernetes.labels.appid = '12345'
| sort @timestamp desc
| limit 20
fields @timestamp, @message
| filter kubernetes.pod_name = '12345-fb5f75b84-k5kgv'
| sort @timestamp desc
| limit 20
fields @timestamp, @message
| filter log._log_level = 'INFO'
| sort @timestamp desc
| limit 20
fields @timestamp, @message
| filter @message like 'HttpMediaTypeNotSupportedException'
| sort @timestamp desc
| limit 20
Capa主要面向Java语言进行设计。
Capa本身不提供功能,而是由底层的各种中间件的Java SDK来提供。
所以了解Java中常用中间件的设计是很有必要的。
本目录主要为对Java生态中中间件的学习和实践。
梳理Dubbo的调用链路流程,分析其设计理念和思路,以及阅读其中关键步骤的源码逻辑
在 RPC 中,Protocol 是核心层,也就是只要有 Protocol + Invoker + Exporter 就可以完成非透明的 RPC 调用,然后在 Invoker 的主过程上 Filter 拦截点。
调用流程主要围绕Protocol/Invoker/Filter三个接口进行
public interface Protocol {
<T> Exporter<T> export(Invoker<T> invoker) throws RpcException;
<T> Invoker<T> refer(Class<T> type, URL url) throws RpcException;
void destroy();
}
public interface Invoker<T> extends Node {
Result invoke(Invocation invocation) throws RpcException;
void destroy();
}
public interface Filter extends BaseFilter {
Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException;
}

ProtocolInvokerInvoker,做一些逻辑Filter,做一些逻辑Invoker,做一些逻辑从一个Client开始……

解决函数逻辑的水平拓展问题

定义接口:Response invoke(Request request);
public interface Invoker<T> extends Node {
Result invoke(Invocation invocation) throws RpcException;
void destroy();
}

如果你要写个远程调用框架,那远程调用的过程应该有一个统一的拦截接口。如果你要写一个 ORM 框架,那至少 SQL 的执行过程,Mapping 过程要有拦截接口;如果你要写一个 Web 框架,那请求的执行过程应该要有拦截接口,等等。没有哪个公用的框架可以 Cover 住所有需求,允许外置行为,是框架的基本扩展方式。这样,如果有人想在远程调用前,验证下令牌,验证下黑白名单,统计下日志;如果有人想在 SQL 执行前加下分页包装,做下数据权限控制,统计下 SQL 执行时间;如果有人想在请求执行前检查下角色,包装下输入输出流,统计下请求量,等等,就可以自行完成,而不用侵入框架内部。拦截接口,通常是把过程本身用一个对象封装起来,传给拦截器链,比如:远程调用主过程为 invoke(),那拦截器接口通常为 invoke(Invocation),Invocation 对象封装了本来要执行过程的上下文,并且 Invocation 里有一个 invoke() 方法,由拦截器决定什么时候执行,同时,Invocation 也代表拦截器行为本身,这样上一拦截器的 Invocation 其实是包装的下一拦截器的过程,直到最后一个拦截器的 Invocation 是包装的最终的 invoke() 过程;同理,SQL 主过程为 execute(),那拦截器接口通常为 execute(Execution),原理一样。当然,实现方式可以任意,上面只是举例。
public interface Filter extends BaseFilter {
Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException;
}
装饰模式/组合模式

重资源 -> 资源的管理 -> 生命周期
资源的管理:
共享、创建、释放、生命周期……..
链路治理、上下文、组装……..

在 Dubbo 的核心领域模型中:
public interface Protocol {
<T> Exporter<T> export(Invoker<T> invoker) throws RpcException;
<T> Invoker<T> refer(Class<T> type, URL url) throws RpcException;
void destroy();
}

共享资源谁持有?
缓存在哪里存储?
线程安全?
单例还是New?
任何框架或组件,总会有核心领域模型,比如:Spring 的 Bean,Struts 的 Action,Dubbo 的 Service,Napoli 的 Queue 等等。这个核心领域模型及其组成部分称为实体域,它代表着我们要操作的目标本身。实体域通常是线程安全的,不管是通过不变类,同步状态,或复制的方式。 ①
服务域也就是行为域,它是组件的功能集,同时也负责实体域和会话域的生命周期管理, 比如 Spring 的 ApplicationContext,Dubbo 的 ServiceManager 等。服务域的对象通常会比较重,而且是线程安全的,并以单一实例服务于所有调用。 ②
什么是会话?就是一次交互过程。会话中重要的概念是上下文,什么是上下文?比如我们说:“老地方见”,这里的“老地方”就是上下文信息。为什么说“老地方”对方会知道,因为我们前面定义了“老地方”的具体内容。所以说,上下文通常持有交互过程中的状态变量等。会话对象通常较轻,每次请求都重新创建实例,请求结束后销毁。简而言之:把元信息交由实体域持有③,把一次请求中的临时状态由会话域持有,由服务域贯穿整个过程。
① Invoker仅维护自己的状态,类似Actor设计模式
② Protocol初始化为单例,并且持有缓存Map
③ Invoker中存储默认元信息,临时信息存储到Invocation
=> show code
由服务域管理资源的生命周期,并委托给实体域执行
public interface Protocol {
<T> Exporter<T> export(Invoker<T> invoker) throws RpcException;
<T> Invoker<T> refer(Class<T> type, URL url) throws RpcException;
void destroy();
}
public interface Invoker<T> extends Node {
Result invoke(Invocation invocation) throws RpcException;
void destroy();
}
这里先要讲一个事件和上面拦截器的区别,拦截器是干预过程的,它是过程的一部分,是基于过程行为的,而事件是基于状态数据的,任何行为改变的相同状态,对事件应该是一致的。事件通常是事后通知,是一个 Callback 接口,方法名通常是过去式的,比如 onChanged()。比如远程调用框架,当网络断开或连上应该发出一个事件,当出现错误也可以考虑发出一个事件,这样外围应用就有可能观察到框架内部的变化,做相应适应。
public interface InvokerListener {
void referred(Invoker<?> invoker) throws RpcException;
void destroyed(Invoker<?> invoker);
}
SPI,微内核,插件化
大凡发展的比较好的框架,都遵守微核的理念。Eclipse 的微核是 OSGi, Spring 的微核是 BeanFactory,Maven 的微核是 Plexus。通常核心是不应该带有功能性的,而是一个生命周期和集成容器,这样各功能可以通过相同的方式交互及扩展,并且任何功能都可以被替换。如果做不到微核,至少要平等对待第三方,即原作者能实现的功能,扩展者应该可以通过扩展的方式全部做到。原作者要把自己也当作扩展者,这样才能保证框架的可持续性及由内向外的稳定性。
引入配置模块,担当微内核。并负责Protocol的初始化启动(Protocol再负责下层的初始化)
其他层次以插件的方式装配到config里,核心仍然是Protocol层。功能要么伪装成Invoker,要么在初始化时进行。
在 RPC 中,Protocol 是核心层,也就是只要有 Protocol + Invoker + Exporter 就可以完成非透明的 RPC 调用,然后在 Invoker 的主过程上 Filter 拦截点。


层级关系:
-system
-config
-proxy
-registry
-cluster
-monitor
-protocol
-invoker
-exchange
-...

public interface Result extends Serializable {
Result whenCompleteWithContext(BiConsumer<Result, Throwable> fn);
}
结合一开始的大概流程…
初始化的时候做了什么?集群?注册发现?代理封装?
资源如何加载?如何缓存?连接池?线程池?
……..
实际调用过程的流程?有哪些逻辑步骤?
负载均衡?重试?监控?序列化?
同步异步?流式调用?
……..


关键Invoker的逻辑:
AbstractInvoker: 3个步骤,(PR:https://github.com/apache/dubbo/pull/7952)
具体逻辑在doInvoke中
FailoverClusterInvoker当调用失败时,记录初始错误并重试其他调用程序(重试n次,这意味着最多将调用n个不同的调用程序)注意,重试会导致延迟。 故障转移
图中方法:list、route、select
什么时候会重试?非biz异常,但粒度很粗….(Issue相关)
ListenerInvokerWrapper注册invoker的listener,并进行操作,装饰模式
改进:观察者模式+生命周期
FilterChainNode实际上是Invoker,但内部保存了Filter
Invoker装饰,Filter适配,两者的组合
DubboInvoker可以看到:Invoker层持有下层的资源管理
接下来就是excahnge层的逻辑
TripleInvoker直接持有底层netty资源,比较粗糙
=> show code
层级关系:
-system
-config
-proxy
-registry
-cluster
-monitor
-protocol
-invoker
-exchange
-...

责任链:Invoker
适配器:适配到Invoker责任链中
装饰:对Invoker进行增强
………
Invoker调用链:√,整个系统的核心,插件化功能采用拓展接口的方式,各个模块功能比较明确
Lifecycle生命周期:不完全,没有完整的生命周期;对于生命周期的事件处理比较硬编码
Listener:装饰的比较硬编码,事件阶段不够全,拓展性不够好
Filter:采用装饰模型而非组合模型,拓展性较差;嵌套层级深,调试时链路不清晰
……..
SPI,微内核,插件化
装饰器模式
责任链模式
https://www.codetd.com/en/article/13249666#ServletContextListener_27
ServletContextListener 接口,它能够监听 ServletContext 对象的生命周期,当Servlet 容器启动或终止Web 应用时,会触发ServletContextEvent 事件,该事件由ServletContextListener 来处理。
https://www.codetd.com/en/article/13249666#ServletContextListener_27
Spring容器的创建Bean前后执行
InitializingBean接口为bean提供了属性初始化后的处理方法,在bean的属性初始化后都会执行该方法。
使用@PostConstruct注解一个方法来完成初始化,@PostConstruct注解的方法将会在依赖注入完成后被自动调用。
项目启动后执行,CommandLineRunner和ApplicationRunner的作用是相同的,不同在于参数的封装和没封装。可以创建多个实现CommandLineRunner和ApplicationRunner接口的类。为了使他们按一定顺序执行,可以使用@Order注解或实现Ordered接口。
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.5.9.RELEASE)
beanPostProcessor run...
servletContextListener run...
initializingBean run...
postConstruct run...
applicationRunner run...
commandLineRunner run...
……
EchoServerHandler serverHandler = new EchoServerHandler()
Handler类结构图:

EventLoopGroup group = new NioEventLoopGroup();
EventLoopGroup类结构图:

@Override
protected EventLoop newChild(Executor executor, Object... args) throws Exception {
return new NioEventLoop(this, executor, (SelectorProvider) args[0],
((SelectStrategyFactory) args[1]).newSelectStrategy(), (RejectedExecutionHandler) args[2]);
}

taskQueue: MpscChunkedArrayQueue
tailQueue: MpscChunkedArrayQueue
selectNowSupplier: selector.selectNow()
pendingTasksCallable: super.pendingTasks() + tailTasks.size()
selectProvider: SelectorProvider.provider()

ServerBootstrap b = new ServerBootstrap();



持有底层的SocketChannel:(此处的Channel为java.nio的Channel)

创建一个DefaultChannelPipeline:

Channel和ChannelPipeline是双向绑定的关系,向ChannelPipeline中通过addLast方法添加Handler。
将ChannelHandler包装进入ChannelContext, 然后ChannelPipeline持有ChannelContext的双向链表:

ChannelFuture:对Channel的各个生命周期阶段注册回调

调用的是AbstractChannel的:
public final void register(EventLoop eventLoop, final ChannelPromise promise) {
然后是:
eventLoop.execute(new Runnable() {
@Override
public void run() {
register0(promise);
}
});
eventloop在这里是前面创建的EventLoopGroup
执行前面注册的ChannelHandler初始化initChannel的动作
ServerBootstrapAcceptor
最终进入bind()操作,执行ServerSocketChannelImpl.bind()
同Server
connect操作会注册一个task给eventloop
eventloop执行task:
Bootstrap.doResolveAndConnect0()
doConnect()
channel.connect()
SocketChannelImpl.connect()
interestOps置为8



主要包括4个基本组件:
事件队列(event queue):接收事件的入口,存储待处理事件
分发器(event mediator):将不同的事件分发到不同的业务逻辑单元
事件通道(event channel):分发器与处理器之间的联系渠道
事件处理器(event processor):实现业务逻辑,处理完成后会发出事件,触发下一步操作
可以看出,相对传统轮询模式,事件驱动有如下优点:
在同一组会话/链接中,将会触发不同的事件入口时,适合事件驱动架构

1)Netty 抽象出两组线程池:BossGroup 和 WorkerGroup,也可以叫做 BossNioEventLoopGroup 和 WorkerNioEventLoopGroup。每个线程池中都有 NioEventLoop 线程。BossGroup 中的线程专门负责和客户端建立连接,WorkerGroup 中的线程专门负责处理连接上的读写。BossGroup 和 WorkerGroup 的类型都是 NioEventLoopGroup。
2)NioEventLoopGroup 相当于一个事件循环组,这个组中含有多个事件循环,每个事件循环就是一个 NioEventLoop。
3)NioEventLoop 表示一个不断循环的执行事件处理的线程,每个 NioEventLoop 都包含一个 Selector,用于监听注册在其上的 Socket 网络连接(Channel)。
4)NioEventLoopGroup 可以含有多个线程,即可以含有多个 NioEventLoop。
5)每个 BossNioEventLoop 中循环执行以下三个步骤:
5.1)select:轮训注册在其上的 ServerSocketChannel 的 accept 事件(OP_ACCEPT 事件)
5.2)processSelectedKeys:处理 accept 事件,与客户端建立连接,生成一个 NioSocketChannel,并将其注册到某个 WorkerNioEventLoop 上的 Selector 上
5.3)runAllTasks:再去以此循环处理任务队列中的其他任务
6)每个 WorkerNioEventLoop 中循环执行以下三个步骤:
6.1)select:轮训注册在其上的 NioSocketChannel 的 read/write 事件(OP_READ/OP_WRITE 事件)
6.2)processSelectedKeys:在对应的 NioSocketChannel 上处理 read/write 事件
6.3)runAllTasks:再去以此循环处理任务队列中的其他任务
7)在以上两个processSelectedKeys步骤中,会使用 Pipeline(管道),Pipeline 中引用了 Channel,即通过 Pipeline 可以获取到对应的 Channel,Pipeline 中维护了很多的处理器(拦截处理器、过滤处理器、自定义处理器等)。这里暂时不详细展开讲解 Pipeline。

1)Bootstrap 和 ServerBootstrap 分别是客户端和服务器端的引导类,一个 Netty 应用程序通常由一个引导类开始,主要是用来配置整个 Netty 程序、设置业务处理类(Handler)、绑定端口、发起连接等。
2)客户端创建一个 NioSocketChannel 作为客户端通道,去连接服务器。
3)服务端首先创建一个 NioServerSocketChannel 作为服务器端通道,每当接收一个客户端连接就产生一个 NioSocketChannel 应对该客户端。
4)使用 Channel 构建网络 IO 程序的时候,不同的协议、不同的阻塞类型和 Netty 中不同的 Channel 对应,常用的 Channel 有:
NioSocketChannel:非阻塞的 TCP 客户端 Channel(本案例的客户端使用的 Channel) NioServerSocketChannel:非阻塞的 TCP 服务器端 Channel(本案例的服务器端使用的 Channel) NioDatagramChannel:非阻塞的 UDP Channel NioSctpChannel:非阻塞的 SCTP 客户端 Channel NioSctpServerChannel:非阻塞的 SCTP 服务器端 Channel…

每个 Netty Channel 包含了一个 ChannelPipeline(其实 Channel 和 ChannelPipeline 互相引用),而 ChannelPipeline 又维护了一个由 ChannelHandlerContext 构成的双向循环列表,其中的每一个 ChannelHandlerContext 都包含一个 ChannelHandler。(前文描述的时候为了简便,直接说 ChannelPipeline 包含了一个 ChannelHandler 责任链,这里给出完整的细节。)
网络Channel对象 + 执行器 + 不同的事件处理器
processor.execute(channel(data), executor(executore))
面向对象包装Channel:执行器、自身属性、操作定义
事件处理器责任链化:pipeline
事件处理器状态机:内部状态流转和外部状态触发 (好几套状态机…channel/pipeline/bootstrap…)
内部状态流转的例子:register->register0->doregister —future完成后—> bind->dobind0 内部状态流转:bind –> pipeline.fireChannelActive —> 一系列handler等内部状态流转 —> read –> 注册read状态位 –> 进入外部事件触发
register: 接口定义,事件入口 register0: 抽象类封装(递交给eventloop) doregister: 委托底层实现
channel事件入口 --->递交给 channelpipeline(invoker定义链路操作) --->链路入口 channelhandler执行具体链路方法
--->上下文 channelcontext
channelpipelint: invoker+链路组装
佐证:channelpipeline(invoker)和handler的方法一一对应 推测:context:面向对象的封装、另开一条链路
ctx.fireChannelActive(); 链路控制
责任链需要递归无用的handler,通过context直接指向下一个执行者,优化了流程
……
……
……
Boss Worker
将待执行操作包装成命令对象,投递给eventloop线程执行
单流
三流
五流
将步骤抽象为Sink算子
原文地址:https://zhuanlan.zhihu.com/p/392401566

反应式宣言:异步 非阻塞 带回压 的方式进行流程控制
技术手段+表现形式:
我们在选择使用反应式编程时,一定要明确它的适用场景,主要包括:
首先,我们的代码是声明式的,不方便调试,错误发生时不容易定位。使用原生的API,例如不通过Spring框架而直接使用Reactor,会使情况变的更糟,因为我们自己要做很多的错误处理,每次进行网络调用都要写很多样板代码。通过组合使用 Spring 和 Reactor 我们可以方便的查看堆栈信息和未捕获的异常。由于运行的线程不受我们控制,因此在理解上会有困难。
其次,一旦编写错误导致一个Reactive回调被阻塞,在同一线程上的所有请求都会挂起。在servlet容器中,由于是一个请求一个线程,一个请求阻塞时,其它的请求不会受影响。而在Reactive中,一个请求被阻塞会导致所有请求的延迟都增加。
实践反应式宣言的编程框架,提供开箱即用的反应式编程手段
编写非阻塞、并发和可伸缩的代码是困难的。反应式编程库通过提供基于reactor模式的简单编程模型,提供了构建此类应用程序的最简单方法,这将允许应用程序高效地使用所有硬件资源。
相对于传统的基于回调和Future的异步开发方式,响应式编程更加具有可编排性和可读性,配合lambda表达式,代码更加简洁,处理逻辑的表达就像装配“流水线”,适用于对数据流的处理。
统一的编码方式,在不同的反应式网络协议和编程模型之上,构建了抽象编程层。例如gRPC和RSocket等反应式网络协议,无需了解具体协议细节,直接使用抽象层编程即可实现相关功能。
静态编译技术:GraalVM/Android

responseHandlerFuture.whenCompleteAsync((r, t) -> {
if (t == null) {
responseFuture.complete(r);
} else {
responseFuture.completeExceptionally(t);
}
}, futureCompletionExecutor);
responseFuture.addListener(() -> {...}, rpcExecutorService);
// sending with async non-blocking io
return Mono
.fromCompletionStage(() -> {
// 注①
CompletableFuture<SendEmailResponse> responseFuture = awsAsyncClient.sendEmail(sendEmailRequest);
return responseFuture;
})
// 注② thread switching: callback executor
// .publishOn(asyncResponseScheduler)
// 注③ callback: success
.map(response -> this.onSuccess(context, response))
// 注④ callback: failure
.onErrorResume(throwable -> {
return this.onFailure(context, throwable);
});
原因在于,若将其用于响应式编程中,是有局限性的。比如如下两个需要面对的问题:
具备“异步非阻塞”特性和“流量控制”能力的数据流,我们称之为反应式流(Reactive Stream)。
就像面向对象的编程思想一样,函数式编程只是一系列想法,而不是一套严苛的规定。有很多支持函数式编程的程序语言,它们之间的具体设计都不完全一样。
响应式编程(reactive programming)是一种基于数据流(data stream)和变化传递(propagation of change)的声明式(declarative)的编程范式。
本质上都是java编程语言,主要区别在于编程风格和思考逻辑的不同
组装数据流处理链的几种形式:
面向过程
面向对象
Publisher/Subscriber/Subctiption
java9
Publisher 在 PRR 中 所承担的角色也就是传统的 观察者模式 中的 被观察者对象,在 PRR 的定义也极为简单。
package org.reactivestreams;
public interface Publisher<T> {
public void subscribe(Subscriber<? super T> s);
}
Publisher 的定义可以看出来,Publisher 接受 Subscriber,非常简单的一个接口。但是这里有个有趣的小细节,这个类所在的包是 org.reactivestreams,这里的做法和传统的 J2EE 标准类似,我们使用标准的 Javax 接口定义行为,不定义具体的实现。
Subscriber 在 PRR 中 所承担的角色也就是传统的 观察者模式 中的 观察者对象,在 PRR 的定义要多一些。
public interface Subscriber<T> {
public void onSubscribe(Subscription s); ➊
public void onNext(T t); ➋
public void onError(Throwable t);
public void onComplete(); ➍
}
➊ 订阅时被调用 ➋ 每一个元素接受时被触发一次 ➌ 当在触发错误的时候被调用 ➍ 在接受完最后一个元素最终完成被调用
Subscriber 的定义可以看出来,Publisher 是主要的行为对象,用来描述我们最终的执行逻辑。
行为是由Subscriber触发的,这是一种Pull模型,如果是Publisher触发,则为Push模型。
在最基础的 观察者模式 中,我们只是需要 Subscriber 观察者 Publisher 发布者,而在 PRR 中增加了一个 Subscription 作为 Subscriber Publisher 的桥接者。
public interface Subscription {
public void request(long n); ➊
public void cancel(); ➋
}
➊ 获取 N 个元素往下传递 ➋ 取消执行
为什么需要这个对象,笔者觉得是一是为了解耦合,第二在 Reference 中有提到 Backpressure 也就是下游可以保护自己不受上游大流量冲击,这个在 Stream 编程中是无法做到的,想要做到这个,就需要可以控制流速,那秘密看起来也就是在 request(long n) 中。
中间层,主要用于解耦和流控。


常用的、简单的数据流处理链路
Netty
去除中间过程,横向改为纵向处理

将Op组装成Pipeline
生成Sink链
优化:
执行Sink方法
最核心的就是类为AbstractPipeline,ReferencePipeline和Sink接口.AbstractPipeline抽象类是整个Stream中流水线的高度抽象了源头sourceStage,上游previousStage,下游nextStage,定义evaluate结束方法,而ReferencePipeline则是抽象了过滤,转换,聚合,归约等功能,每一个功能的添加实际上可以理解为卷心菜,菜心就是源头,每一次加入一个功能就相当于重新长出一片叶子包住了菜心,最后一个功能集成完毕之后整颗卷心菜就长大了。而Sink接口呢负责把整个流水线串起来,然后在执行聚合,归约时候调AbstractPipeline抽象类的evaluate结束方法,根据是否是并行执行,调用不同的结束逻辑,如果不是并行方法则执行terminalOp.evaluateSequential否则就执行terminalOp.evaluateParallel,非并行执行模式下则是执行的是AbstractPipeline抽象类的wrapAndCopyInto方法去调用copyInto,调用前会先执行一下wrapSink,用于剥开这个我们在流水线上产生的卷心菜。从下游向上游去遍历AbstractPipeline,然后包装到Sink,然后在copyInto方法内部迭代执行对应的方法。最后完成调用

反应式编程模型在执行中主要有5条链路:
组装reactor执行链路,绑定上下游节点,在 subscribe() 之前,我们什么都没做,只是在不断的包裹 Publisher 将作为原始的 Publisher 一层又一层的返回回来。
生成Publisher
subscribe()
生成Subscriber
内存数据发送很简单,就是循环发送。而对于像数据库、RPC 这样的场景,则会触发请求的发送。
Publisher 接口中的 subscribe 方法语义上有些奇特,它表示的不是订阅关系,而是被订阅关系。即 aPublisher.subscribe(aSubscriber) 表示的是 aPublisher 被 aSubscriber 订阅。
对于中间过程的 Mono/Flux,subscribe 阶段是订阅上一个 Mono/Flux;而对于源 Mono/Flux,则是要执行 Subscriber.onSubscribe(Subscription s) 方法。
onSubscribe()
生成Subscription,并将Subscriber作为成员参数传入
request(n)
基于Subscription实现request(n)机制
doNext()/….
基于Subscription中的Subscriber实现执行机制



最后生成的对象是这样的


最后生成的对象是这样的



经过一顿分析,整个 PRR 是如何将操作整合起来的,我们已经有一个大致的了解,通过不断的包裹出新的 Subscriber 对象,在最终的 request() 行为中触发整个消息的处理,这个过程非常像 俄罗斯套娃,一层一层的将变化组合形变操作变成一个新的 Subscriber, 然后就和一个管道一样,一层一层的往下传递。

Reactor库提供了丰富的操作符,通过这些操作符你几乎可以搭建出能够进行任何业务需求的数据处理管道/流水线。如果将Reactor的执行步骤看做一条流水线,那么操作符就可以比作流水线中的一个工作台,负责处理流程中的元素,并传递给下游。 上述章节我们了解到Reactor的执行流程包括subscribe、onSubscribe、onRequest阶段,一个基本的操作符需要实现这些阶段的处理,比如连接上下游并往下传递,之后在onNext中实现该操作符具体的功能便可以完成一个操作。接下来我们通过学习一些操作符来了解
publishOn 强制下一个操作符(很可能包括下一个的下一个…)来运行在一个不同的线程上。

背压指的是当Subscriber请求的数据的访问超出它的处理能力时,Publisher限制数据发送速度的能力。

本质上背压和TCP中的窗口限流机制比较类似,都是让消费者反馈请求数据的范围,生产者根据消费者的反馈提供一定量的数据来进行流控。
反馈请求数据范围的操作,可以在Subscriber每次完成数据的处理之后,让Subscriber自行反馈;也可以在Subscriber外部对Subscriber的消费情况进行监视,根据监视情况进行反馈。 背压通常在以下场景中使用:
request(n)来通过 subscription 向上游传递 背压请求Flux<String> source = someStringSource();
source.map(String::toUpperCase)
.subscribe(new BaseSubscriber<String>() {
@Override
protected void hookOnSubscribe(Subscription subscription) {
//在开始这个流的时候请求一个元素值。
request(1);
}
@Override
protected void hookOnNext(String value) {
//执行onnext的时候向上游请求一个元素
request(1);
}
});
start: INFO [Thread-7] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler [\"http-nio-8080\"]
start: INFO [Thread-6] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
end: INFO [Thread-6] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler [\"http-nio-8080\"]
end: INFO [Thread-6] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler [\"http-nio-8080\"]
start: INFO [Thread-7] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler [\"http-nio-8080\"]
start: INFO [Thread-6] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
这是非正常关闭
正常关闭(viaport)的话会在 pause 之前有这样的一句warn日志:
org.apache.catalina.core.StandardServer await
A valid shutdown command was received via the shutdown port. Stopping the Server instance.
然后才是 pause -> stop -> destory
而这又有两种情况:
在构建现代互联网系统时,随着业务规模的增加,稳定性要求的增加,对于架构设计的挑战也随之增加。
尤其是涉及到多云多活、跨数据中心流量调度等场景,更会将系统的复杂性进一步提升。
本目录主要为对架构设计的学习和实践。
可视化呈现方式对可观测性建设和维护尤为重要。
基于以往的经验,如果可视化系统使用费力度高,往往会导致开发人员维护意愿下降,恶性循环最终导致可观测性失去敏感度和荒废。
可视化上
https://www.codetd.com/en/article/13249666#ServletContextListener_27
ServletContextListener 接口,它能够监听 ServletContext 对象的生命周期,当Servlet 容器启动或终止Web 应用时,会触发ServletContextEvent 事件,该事件由ServletContextListener 来处理。
任何大型架构或者框架,都离不开代码细节。
精益求精,本目录主要为对代码细节的学习和实践。