Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get processing timeline events #3241

Draft
wants to merge 14 commits into
base: master
Choose a base branch
from

Conversation

jbaptperez
Copy link

@jbaptperez jbaptperez commented Dec 5, 2024

IMPORTANT: All Pull Requests should be connected to an issue, if you don't
have an issue, please start by creating an issue and link it to the PR.

Please provide enough information so that others can review your pull request:

  • What existing problem does this PR solve?
    • Impossible to get timeline events when the one is being updated (new Plaso ingestion for the timeline).
  • What new feature is being introduced with this PR?
    • Allows to get timeline events when the timeline is being updated (new Plaso ingestion) i.e. allows to get partial events, already indexed,
    • Adds a setting to enable or disable this feature,
    • The UI clearly communicates to the user that search results against processing timelines may be incomplete and subject to change.
  • Overview of changes to existing functions if required.
    • Backend: In the Sketch model, the active_timelines property includes timelines with a status set to processing,
    • Backend: In the utils.py, the get_validated_indices function includes timelines with a status set to processing,
    • Backend: The POST /sketches/{{sketch_id}}/aggregation/explore/ endpoint includes indices which timeline has a status set to processing.

Checks

  • All tests succeed.
  • Unit tests added.
  • e2e tests added.
  • Documentation updated.

Closing issues

Closes #3219.

The yarn.lock file speed-ups an initial set-up but freezes the registry
URL, which can be different in the ~/.npmrc file when developing in a
company.
Markdown formatting, typos.
Makes the repository handle file line endings.
This helps to make it cross-platform, asserting some files are Unix-ended.
Uses the prefeed name of the Compose specification.
Changes:
- Removes the deprecated "version" field,
- Adds a toplevel "name" field (prefix of container names nad network),
- Adds a toplevel "network" field, with a common "timesketch-dev"
  network,
- Removes container names (depends on and toplevel name and service
  names),
- Do not bind to the 127.0.0.1 interface only (0.0.0.0),
- Removes useless "links" (common network),
- Refactors environment variables to don't use a YAML array,
- Removes "restart" fields to detect undesired crashes in development,
- Binds ports of other services to the host (opensearch, redis).
Adds .gitignore files.
Dramatically improves an image build in a development context.
Adapts the list to the latest version of the repository:
- Fixes moved ones,
- Deletes removed ones.
Changes are:
- Allows Docker image builds in a restricted company context (limited
  access to remote Ubuntu, Python or Node repositories) using variables,
- Centralizes variables in a .env file (not versioned),
- Adds a .env.template file as .env template with predefined variables,
- Use a distinct directory for every service dependencies,
- Use named volumes to avoid anonymous ones (PostgreSQL, Redis and
  Prometheus),
- Use a per-service environment file,
- Simplifies how development configuration files are transferred to
  Timesketch,
- Simplifies manipulation of containers using Compose CLI instead of
  the Docker one,
- Simplify and optimizes the Timesketch entrypoint,
- Updates the Bash scripts to start frontend-ng,
- Updates related documentation.

Use named volumes in Compose development

PostgreSQL, Redis and Prometheus declare volumes in their Dockerfile.
This leads to anonymous volume creations if they are not declared in
Compose.
Avoids using the system environment.
@jbaptperez jbaptperez force-pushed the get-processing-timeline-events branch from 3d33c11 to 42403d9 Compare December 5, 2024 16:24
For now, there is no setting to switch with previous behaviour.
@jbaptperez jbaptperez force-pushed the get-processing-timeline-events branch from 42403d9 to de22d67 Compare December 5, 2024 17:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Seeing data already ingested into a timeline when the related search index is being updated
1 participant