Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

批量添加ingress规则，最后一个ingress规则偶发性出现404错误 #1547

Closed
ishells opened this issue Nov 27, 2024 · 7 comments · Fixed by #1559
Closed

批量添加ingress规则，最后一个ingress规则偶发性出现404错误 #1547

ishells opened this issue Nov 27, 2024 · 7 comments · Fixed by #1559
Assignees
@johnlanni

Comments

@ishells
Copy link

ishells commented Nov 27, 2024 •
edited
Loading

Ⅰ. Issue Description

Ack集群 1.30.1 部署的higress 2.0.2版本

  • 1、首次报错

2.0.2 版本higress目前有650多个ingress规则，使用脚本基于helm命令一次性批量新添加了8个ingress规则之后，最后一个ingress规则的路径用wss长链测试连接会时不时出现404状态码，不知道啥情况
higress-gateway-404-log

首次出现的某一应用的偶发性404日志记录

  • 2、controller日志查看

查看某一个controller higress-core容器日志，没有明显报错。日志也显示下发成功

2024-11-26T06:43:43.216305Z	info	ingress	resource type networking.istio.io/v1alpha3/VirtualService, configs number 652
2024-11-26T06:43:43.245498Z	info	ingress	Found 0 number of basic auth
2024-11-26T06:43:43.245516Z	info	ingress	Found 0 number of envoyFilters
2024-11-26T06:43:43.249243Z	info	ads	networking.istio.io/v1alpha3/VirtualService: PUSH for node:higress-controller-c89956cf4-bnf9s.xxxx-higress resources:5 size:321.7kB
2024-11-26T06:43:43.249282Z	info	ingress	controller tracing ConstructEnvoyFilters
2024-11-26T06:43:43.249287Z	info	ingress	controller gzip ConstructEnvoyFilters
2024-11-26T06:43:43.249290Z	info	ingress	controller global-option ConstructEnvoyFilters
2024-11-26T06:43:43.249455Z	info	ingress	Append 1 configmap EnvoyFilters
2024-11-26T06:43:43.249467Z	info	ingress	resource type networking.istio.io/v1alpha3/EnvoyFilter, configs number 1
2024-11-26T06:43:43.249553Z	info	ads	networking.istio.io/v1alpha3/EnvoyFilter: PUSH for node:higress-controller-c89956cf4-bnf9s.xxxx-higress resources:1 size:836B
2024-11-26T06:43:43.249609Z	info	ads	telemetry.istio.io/v1alpha1/Telemetry: PUSH for node:higress-controller-c89956cf4-bnf9s.xxxx-higress resources:0 size:0B
2024-11-26T06:43:43.260943Z	info	ingress	resource type extensions.istio.io/v1alpha1/WasmPlugin, configs number 652
2024-11-26T06:43:43.260974Z	info	ads	extensions.istio.io/v1alpha1/WasmPlugin: PUSH for node:higress-controller-c89956cf4-bnf9s.xxxx-higress resources:0 size:0B
2024-11-26T06:43:43.260988Z	info	ads	networking.istio.io/v1alpha3/WorkloadEntry: PUSH for node:higress-controller-c89956cf4-bnf9s.xxxx-higress resources:0 size:0B
2024-11-26T06:43:52.206234Z	info	ingress	no need process, ingress xxxxqd-prod/xxxxqd-xxxx-10061-ing
2024-11-26T06:43:52.214517Z	info	ads	full push happen, reason:map[config:1]
2024-11-26T06:43:52.214535Z	info	ads	full push happen, reason:map[config:1]
2024-11-26T06:43:52.214539Z	info	ads	full push happen, reason:map[config:1]
2024-11-26T06:43:52.214543Z	info	ads	full push happen, reason:map[config:1]
2024-11-26T06:43:52.315392Z	info	ads	Push debounce stable[890] 4 for config DestinationRule/xxxxqd-prod/xxxxqd-xxxx-10061-ing-destinationrule and 3 more configs: 100.818338ms since last change, 100.825892ms since last push, full=true
2024-11-26T06:43:52.322480Z	info	ingress	resource type networking.istio.io/v1alpha3/VirtualService, configs number 652
2024-11-26T06:43:52.334549Z	info	ingress	Found 0 number of basic auth
2024-11-26T06:43:52.334560Z	info	ingress	Found 0 number of envoyFilters
2024-11-26T06:43:52.347985Z	info	ingress	resource type networking.istio.io/v1alpha3/DestinationRule, configs number 652
2024-11-26T06:43:52.348150Z	info	ingress	controller tracing ConstructEnvoyFilters
2024-11-26T06:43:52.348155Z	info	ingress	controller gzip ConstructEnvoyFilters
2024-11-26T06:43:52.348158Z	info	ingress	controller global-option ConstructEnvoyFilters
2024-11-26T06:43:52.348266Z	info	ingress	Append 1 configmap EnvoyFilters
2024-11-26T06:43:52.348272Z	info	ingress	resource type networking.istio.io/v1alpha3/EnvoyFilter, configs number 1
2024-11-26T06:43:52.355632Z	info	ingress	resource type networking.istio.io/v1alpha3/Gateway, configs number 652
2024-11-26T06:43:52.363804Z	info	ads	XDS: Pushing Services:0 ConnectedEndpoints:1 Version:2024-11-26T06:43:52Z/890
2024-11-26T06:43:52.375831Z	info	ingress	resource type networking.istio.io/v1alpha3/Gateway, configs number 652
2024-11-26T06:43:52.384514Z	info	ads	networking.istio.io/v1alpha3/Gateway: PUSH for node:higress-controller-c89956cf4-bnf9s.xxxx-higress resources:5 size:2.2kB
2024-11-26T06:43:52.390963Z	info	ingress	resource type networking.istio.io/v1alpha3/ServiceEntry, configs number 652
2024-11-26T06:43:52.390997Z	info	ads	networking.istio.io/v1alpha3/ServiceEntry: PUSH for node:higress-controller-c89956cf4-bnf9s.xxxx-higress resources:0 size:0B
2024-11-26T06:43:52.391020Z	info	ads	networking.istio.io/v1alpha3/Sidecar: PUSH for node:higress-controller-c89956cf4-bnf9s.xxxx-higress resources:0 size:0B
2024-11-26T06:43:52.391034Z	info	ads	security.istio.io/v1beta1/AuthorizationPolicy: PUSH for node:higress-controller-c89956cf4-bnf9s.xxxx-higress resources:0 size:0B
2024-11-26T06:43:52.391044Z	info	ads	security.istio.io/v1beta1/RequestAuthentication: PUSH for node:higress-controller-c89956cf4-bnf9s.xxxx-higress resources:0 size:0B
2024-11-26T06:43:52.397403Z	info	ingress	resource type networking.istio.io/v1alpha3/VirtualService, configs number 652
2024-11-26T06:43:52.408959Z	info	ingress	Found 0 number of basic auth
2024-11-26T06:43:52.408969Z	info	ingress	Found 0 number of envoyFilters
2024-11-26T06:43:52.411945Z	info	ads	networking.istio.io/v1alpha3/VirtualService: PUSH for node:higress-controller-c89956cf4-bnf9s.xxxx-higress resources:5 size:321.7kB
  • 3、问题复现与解决
    第一次出现偶发性404问题

① 脚本使用helm批量顺序安装应用(包括ingress规则)，本次脚本从10054安装到10061，8个一批休眠6分钟；在安装到10061应用时，使用postman测试10061应用的ingress规则发现偶发性出现404返回，通过手动helm uninstall 10061应用并手动helm install之后问题解决；时间线大概在2024-11-26 14:43:43之后。

② 晚些时候第二次执行脚本再次批量安装应用，本次脚本从1300安装到1395，在安装到1369应用时再次出现postman测试偶发性404返回，脚本并未暂停运行，再休眠6分钟之后，继续安装1370-1395应用，脚本自动安装了1370应用之后，1369应用的偶发性404问题不再出现。时间线大概在2024-11-26 18:09:19之后。

image

共性都是脚本某一个批次的最后一个应用会出现偶发性404，之前同样方式安装同样应用未出现过类似问题。
@johnlanni
Copy link
Collaborator

@ishells 404 对应的访问日志麻烦发一下

@ishells
Copy link
Author

ishells commented Nov 27, 2024

@ishells 404 对应的访问日志麻烦发一下
{"authority":"g.xxxxxxxxx.com","bytes_received":"0","bytes_sent":"0","downstream_local_address":"10.128.20.183:443","downstream_remote_address":"xx.xxx.xxx.xx:51529","duration":"1","istio_policy_status":"-","method":"GET","path":"/g10061","protocol":"HTTP/1.1","request_id":"04f37744-e39f-4675-a2e6-0da6713efa58","requested_server_name":"g.xxxxxxxxx.com","response_code":"404","response_flags":"NR","route_name":"-","start_time":"2024-11-26T07:16:38.015Z","trace_id":"-","upstream_cluster":"-","upstream_host":"-","upstream_local_address":"-","upstream_service_time":"-","upstream_transport_failure_reason":"-","user_agent":"-","x_forwarded_for":"xx.xxx.xxx.xx","response_code_details":"route_not_found"}

修改了公网IP和域名的一条请求404日志

@johnlanni
Copy link
Collaborator

@ishells 你说之前未出现类似问题，之前的版本是多少？
另外方便提供一下你的这个脚本吗，我这边用于复现排查

@johnlanni
Copy link
Collaborator

可以加一下我钉钉：chengtanzty

@ishells
Copy link
Author

ishells commented Nov 27, 2024

之前使用的也是2.0.2版本，版本没有变过。是执行脚本批量安装应用的过程执行过很多次，上面说的偶发404是第一次出现。

@ishells
Copy link
Author

ishells commented Nov 27, 2024

可以加一下我钉钉：chengtanzty

好的

@johnlanni johnlanni self-assigned this Nov 27, 2024
@ishells
Copy link
Author

ishells commented Dec 3, 2024

#1559 solved

@ishells ishells closed this as completed Dec 3, 2024
@johnlanni johnlanni mentioned this issue Dec 3, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@johnlanni johnlanni

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix xds cache johnlanni/higress
2 participants
@johnlanni @ishells