[Task] <add push style data collector>

[Ceilzcx]

Description

添加push方式的数据采集，类似Prometheus的pushgateway
为什么需要添加push模式？

• 方便用户自定义需要监控的数据
• 对于内网的应用，hertzbeat部署在外网无法访问内网的应用

Task List

具体描述：

• 前端创建对应的push monitor，生成monitorId作为push模块的key
• 新增push module，暴露接口供用户调用push监控数据，push模块存储和定时删除数据。
• collector模块定时拉取push模块的数据，并复用数据采集的部分逻辑，进行存储（内存和磁盘）和删除（暂时使用定时删除，有能力可以根据当前使用内存进行判断是否需要删除）
• 前端调用展示用户自定义的数据

[Ceilzcx]

[hudongdong129]

Where are the push data indicators configured？

[Ceilzcx]

not support，It's still implementing

[leo-934]

I can do this.
Please assign it to me.

[gcdd1993]

collector模块定时拉取push模块的数据，是不是无法解决hertzbeat部署在外网无法访问内网的应用。
应该要有个client，比如redis_client，定期采集redis的监控数据，往hertzbeat推送
@Ceilzcx @tomsun28

[hertzbeat]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

The collector module regularly pulls the data of the push module. Is it impossible to solve the application that hertzbeat is deployed on the external network and cannot access the internal network.
There should be a client, such as redis_client, which regularly collects redis monitoring data and pushes it to hertzbeat
@Ceilzcx @tomsun28

[leo-934]

collector模块定时拉取push模块的数据，是不是无法解决hertzbeat部署在外网无法访问内网的应用。 应该要有个client，比如redis_client，定期采集redis的监控数据，往hertzbeat推送 @Ceilzcx @tomsun28

是的，应该需要用户编写的client向push module推送数据。
push module如果部署在外网的话，网关就只需要放行出口流量。如果push module在内网则需要放行一个入口流量，以便于collect module访问push module
@gcdd1993

[hertzbeat]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

The collector module regularly pulls the data of the push module. Is it impossible to solve the application that hertzbeat is deployed on the external network and cannot access the internal network. There should be a client, such as redis_client, which regularly collects redis monitoring data and pushes it to hertzbeat @Ceilzcx @tomsun28

Yes, the client written by the user should be required to push data to the push module.
If the push module is deployed on the external network, the gateway only needs to release the egress traffic. If the push module is on the intranet, an ingress traffic needs to be released so that the collect module can access the push module
@gcdd1993

[gcdd1993]

collector模块定时拉取push模块的数据，是不是无法解决hertzbeat部署在外网无法访问内网的应用。 应该要有个client，比如redis_client，定期采集redis的监控数据，往hertzbeat推送 @Ceilzcx @tomsun28

是的，应该需要用户编写的client向push module推送数据。 push module如果部署在外网的话，网关就只需要放行出口流量。如果push module在内网则需要放行一个入口流量，以便于collect module访问push module @gcdd1993

随之而来的一个问题就是如何复用现有的collect模块，以及如何管理client的配置（下发配置至client），众多的采集协议或许可以用pf4j重构一波

[hertzbeat]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

The collector module regularly pulls the data of the push module. Is it impossible to solve the application that hertzbeat is deployed on the external network and cannot access the internal network. There should be a client, such as redis_client, which regularly collects redis monitoring data and pushes it to hertzbeat @Ceilzcx @tomsun28

Yes, the client written by the user should be required to push data to the push module. If the push module is deployed on the external network, the gateway only needs to release the egress traffic. If the push module is on the internal network, an ingress traffic needs to be released so that the collect module can access the push module @gcdd1993

A subsequent problem is how to reuse the existing collect module, and how to manage the configuration of the client (deliver the configuration to the client). Many collection protocols may be available with [pf4j](https://github.com/ pf4j/pf4j) reconstruction wave

[Ceilzcx]

collector模块定时拉取push模块的数据，是不是无法解决hertzbeat部署在外网无法访问内网的应用。 应该要有个client，比如redis_client，定期采集redis的监控数据，往hertzbeat推送 @Ceilzcx @tomsun28

是的，应该需要用户编写的client向push module推送数据。 push module如果部署在外网的话，网关就只需要放行出口流量。如果push module在内网则需要放行一个入口流量，以便于collect module访问push module @gcdd1993

随之而来的一个问题就是如何复用现有的collect模块，以及如何管理client的配置（下发配置至client），众多的采集协议或许可以用pf4j重构一波

复用应该是可以做到的，步骤一生成monitorId就是为了复用的，collector模块原本的数据就是放在一个Map里面的，key就是monitorId，当时我和tom讨论了一下，应该如何有效的让用户获取这个monitorId，同时将他作为push的类似key的效果，查询monitorId是否有效（在数据库中），确保用户上传的数据正确

[hertzbeat]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

The collector module regularly pulls the data of the push module. Is it impossible to solve the application that hertzbeat is deployed on the external network and cannot access the internal network. There should be a client, such as redis_client, which regularly collects redis monitoring data and pushes it to hertzbeat @Ceilzcx @tomsun28

Yes, the client written by the user should be required to push data to the push module. If the push module is deployed on the external network, the gateway only needs to release the egress traffic. If the push module is on the internal network, an ingress traffic needs to be released so that the collect module can access the push module @gcdd1993

A subsequent problem is how to reuse the existing collect module, and how to manage the configuration of the client (deliver the configuration to the client). Many collection protocols may be available with [pf4j](https://github.com /pf4j/pf4j) Refactor a wave

Reuse should be possible. The monitorId generated in step 1 is for reuse. The original data of the collector module is placed in a Map, and the key is monitorId. At that time, I discussed with Tom how to effectively let users Obtain this monitorId, and at the same time use it as a key-like effect of push, query whether the monitorId is valid (in the database), and ensure that the data uploaded by the user is correct

[Ceilzcx]

@tomsun28 有一个问题还是要考虑一下的，push需不需要单独部署，因为prometheus是通过一个exporter的，pushgateway单独一个模块很正常，我们的push功能其实合并也是可以的。要考虑一下单独部署有什么好处，毕竟单独部署和配置增加了用户的操作

[hertzbeat]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

@tomsun28 One question still needs to be considered, whether push needs to be deployed separately, because prometheus is through an exporter, and it is normal for pushgateway to be a single module, and our push function can actually be merged. Consider the benefits of separate deployment, after all, separate deployment and configuration increase user operations

[tomsun28]

@tomsun28 有一个问题还是要考虑一下的，push需不需要单独部署，因为prometheus是通过一个exporter的，pushgateway单独一个模块很正常，我们的push功能其实合并也是可以的。要考虑一下单独部署有什么好处，毕竟单独部署和配置增加了用户的操作

我们第一版本可以把push做成一个模块，暂时不单独部署，和其它模块一样all in one的形式。单独部署我觉得可以在以后的集群环境下，多采集器和PUSH接收器再支持多进程单独部署。感觉目前没有这个必要，单独部署增加了用户配置操作成本。

[hertzbeat]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

@tomsun28 One question still needs to be considered, whether push needs to be deployed separately, because prometheus is through an exporter, it is normal for pushgateway to be a single module, and our push function can actually be merged. Consider the benefits of separate deployment, after all, separate deployment and configuration increase user operations

In our first version, push can be made into a module, and it will not be deployed separately for the time being. It is in the form of all in one like other modules. Separate deployment I think that in the future cluster environment, multi-collectors and PUSH receivers can support multi-process separate deployment. I feel that this is not necessary at present, and separate deployment increases the cost of user configuration operations.

[l646505418]

Hi all，想问下这个模块的开发工作还有在进行吗😂

[hertzbeat]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Hi all, I would like to ask if the development of this module is still in progress 😂

[leo-934]

Hi all，想问下这个模块的开发工作还有在进行吗😂

您好，我在负责这个模块的开发，怎么了？

[hertzbeat]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Hi all, I would like to ask if the development of this module is still in progress 😂

Hello, I am in charge of the development of this module, what's wrong?

[l646505418]

没有，我只是想问我可以帮到什么么，23333

[hertzbeat]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

No, I just want to ask if I can help, 23333

[leo-934]

没有，我只是想问我可以帮到什么么，23333

啊，还好，我这应该没什么需要帮忙的，可以去看看有无其他issue？

[hertzbeat]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

No, I just want to ask if I can help, 23333

Ah, it’s okay, I don’t think there’s anything I need help with, can I go and see if there are any other issues?

[Ceilzcx]

@l646505418 这个需求是作为开源之夏的任务进行的，是不能有其他人进行commit的，你如果感兴趣可以看一下其他的，类似#1003也是类似的设计类的需要，感兴趣可以看一下

[hertzbeat]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

@l646505418 This requirement is carried out as a task of the summer of open source, and no one else can commit. If you are interested, you can take a look at other ones, similar to [#1003](https://github.com/dromara/hertzbeat /issues/1003) is also a similar design class, you can take a look if you are interested

[leim]

没有，我只是想问我可以帮到什么么，23333

啊，还好，我这应该没什么需要帮忙的，可以去看看有无其他issue？

你好，请问这个模块已经上线了吗？
我现在更新到了版本v1.5.0，从上一个版本开始，我看文档上面就已经有了关于“推送方式监控”的介绍，在docker镜像的配置文件目录中给也已经找到了 app-push.yml 的配置，并且添加配置文档也会提示名称重复，但是在操作页面中一直就没能找到配置入口。