Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add EventId in order to deduplicate #35

Open
wernerb opened this issue Dec 16, 2020 · 0 comments · May be fixed by #36
Open

Add EventId in order to deduplicate #35

wernerb opened this issue Dec 16, 2020 · 0 comments · May be fixed by #36

Comments

@wernerb
Copy link
Contributor

wernerb commented Dec 16, 2020

AWS filter logs api call has eventId per message:

events": [
    {
      "ingestionTime": 1396035394997,
      "timestamp": 1396035378988,
      "message": "ERROR Event 1",
      "logStreamName": "my-log-stream-1",
      "eventId": "31132629274945519779805322857203735586714454643391594505"
    },
    {
      "ingestionTime": 1396035394997,
      "timestamp": 1396035378988,
      "message": "ERROR Event 2",
      "logStreamName": "my-log-stream-2",
      "eventId": "31132629274945519779805322857203735586814454643391594505"
    },
    {

While I appreciate the s3 state storage I would rather just reprocess logs for the last X hours and deduplicate by sending a @metadata._id so that elasticsearch output will overwrite the event.

It would be great if the EventId is added as a field so we can further use this information in an ingest pipeline to set the document id. Unfortunately it seems beats can't set elasticsearch document id yet.
@wernerb wernerb linked a pull request Dec 17, 2020 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add Event ID wernerb/cloudwatchlogsbeat
1 participant
@wernerb