Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compose dev #3226

Open
wants to merge 13 commits into
base: master
Choose a base branch
from
Open

Compose dev #3226

wants to merge 13 commits into from

Conversation

jbaptperez
Copy link

@jbaptperez jbaptperez commented Nov 21, 2024

IMPORTANT: All Pull Requests should be connected to an issue, if you don't
have an issue, please start by creating an issue and link it to the PR.

Please provide enough information so that others can review your pull request:

  • What existing problem does this PR solve?
  • What new feature is being introduced with this PR?
    • Add to the development set-up the possibility to customise image builds (among other) using a .env file,
    • Update of the Compose file syntax to the Compose specification,
    • Other minor enhancements related to the development Compose set-up,
    • Use a Python virtual environment for Timesketch requirements.
  • Overview of changes to existing functions if required.
    • The current behaviour remain the same, but the enhancement of the development set-up leads to different paths to the Dockefiles and simplified commands to manage the development container.

Checks

  • All tests succeed.
  • Unit tests added.
  • e2e tests added.
  • Documentation updated.

Closing issues

Closes #3225.

Copy link

google-cla bot commented Nov 21, 2024

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@jbaptperez jbaptperez force-pushed the compose-dev branch 2 times, most recently from f8d06cb to fb811f1 Compare November 22, 2024 10:24
@jbaptperez jbaptperez marked this pull request as ready for review November 22, 2024 10:34
@jbaptperez jbaptperez force-pushed the compose-dev branch 12 times, most recently from 62afd82 to c310703 Compare November 28, 2024 21:57
@jbaptperez
Copy link
Author

As I had time to update the PR, I also optimized some things, like using a Python virtual environment in the Docker image.
There is a lot of another improvements to be done, but maybe in another PR?

I thought about:

  • Adding an environment variable to allow a custom place for the configuration files (with a default value to be compatible), this is necessary for the next steps,
  • Put every component into its own container,
  • Using Compose profiles to start everything or only a set of services,
  • Allow, with port mapping to localhost, using our IDE to start a process in true debug mode (gunicorn or flask, celery),
  • Use Compose development features for an optimized synchronization with the current source code.

The yarn.lock file speed-ups an initial set-up but freezes the registry
URL, which can be different in the ~/.npmrc file when developing in a
company.
Markdown formatting, typos.
Makes the repository handle file line endings.
This helps to make it cross-platform, asserting some files are Unix-ended.
Uses the prefeed name of the Compose specification.
Changes:
- Removes the deprecated "version" field,
- Adds a toplevel "name" field (prefix of container names nad network),
- Adds a toplevel "network" field, with a common "timesketch-dev"
  network,
- Removes container names (depends on and toplevel name and service
  names),
- Do not bind to the 127.0.0.1 interface only (0.0.0.0),
- Removes useless "links" (common network),
- Refactors environment variables to don't use a YAML array,
- Removes "restart" fields to detect undesired crashes in development,
- Binds ports of other services to the host (opensearch, redis).
Adds .gitignore files.
Dramatically improves an image build in a development context.
Adapts the list to the latest version of the repository:
- Fixes moved ones,
- Deletes removed ones.
Changes are:
- Allows Docker image builds in a restricted company context (limited
  access to remote Ubuntu, Python or Node repositories) using variables,
- Centralizes variables in a .env file (not versioned),
- Adds a .env.template file as .env template with predefined variables,
- Use a distinct directory for every service dependencies,
- Use named volumes to avoid anonymous ones (PostgreSQL, Redis and
  Prometheus),
- Use a per-service environment file,
- Simplifies how development configuration files are transferred to
  Timesketch,
- Simplifies manipulation of containers using Compose CLI instead of
  the Docker one,
- Simplify and optimizes the Timesketch entrypoint,
- Updates the Bash scripts to start frontend-ng,
- Updates related documentation.

Use named volumes in Compose development

PostgreSQL, Redis and Prometheus declare volumes in their Dockerfile.
This leads to anonymous volume creations if they are not declared in
Compose.
Avoids using the system environment.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Adapt the development Compose set-up to allow restricted company contexts
1 participant