Enhance the Compose development set-up

Changes are: - Allows Docker image builds in a restricted company context (limited access to remote Ubuntu, Python or Node repositories) using variables, - Centralizes variables in a .env file (not versioned), - Adds a .env.template file as .env template with predefined variables, - Use a distinct directory for every service dependencies, - Use a per-service environment file, - Simplifies how development configuration files are transferred to Timesketch, - Simplifies manipulation of containers using Compose CLI instead of the Docker one, - Simplify and optimizes the Timesketch entrypoint, - Updates the Bash scripts to start frontend-ng, - Updates related documentation.
google · Nov 28, 2024 · d314590 · d314590
1 parent 0de30e7
commit d314590
Show file tree

Hide file tree

Showing 21 changed files with 730 additions and 293 deletions.
diff --git a/.dockerignore b/.dockerignore
@@ -47,3 +47,6 @@
 
 # Exclude JetBrains IDE files
 /.idea/
+
+# Exclude Compose environment file
+/docker/dev/.env
diff --git a/.gitignore b/.gitignore
@@ -53,3 +53,6 @@ vagrant/*.log
 
 # Exclude JetBrains IDE files
 .idea/
+
+# Exclude Compose environment file
+docker/dev/.env
diff --git a/docker/dev/.env.template b/docker/dev/.env.template
@@ -0,0 +1,15 @@
+GIFT_PPA_TRACK="stable"
+GIFT_PPA_URL="https://ppa.launchpadcontent.net/gift/${GIFT_PPA_TRACK}/ubuntu"
+NODE_VERSION="18.x"
+NODE_PPA_URL="https://deb.nodesource.com/node_${NODE_VERSION}"
+NODE_NPMRC=""
+PYTHON_PIP_CONF=""
+
+TIMESKETCH_BASE_IMAGE="ubuntu:22.04"
+TIMESKETCH_CONF_DIR="/etc/timesketch"
+TIMESKETCH_SECRET_KEY="L4np0jV3yAdAFdbVzWRMaBqiFMV8FKYd+Je1WKE40o8="
+TIMESKETCH_USER="dev"
+TIMESKETCH_PASSWORD="dev"
+
+POSTGRES_USER="timesketch"
+POSTGRES_PASSWORD="password"
diff --git a/docker/dev/README.md b/docker/dev/README.md
@@ -5,49 +5,60 @@ Make sure to follow the docker [post-install](https://docs.docker.com/engine/ins
 
 NOTE: It is not recommended to try to run on a system with less than 8 GB of RAM.
 
-### Start a developer version of docker containers in this directory
+### Prepare a .env file
+
+Compose requires a `.env` file with top level environment variables to be set.
+To create it, just copy the `.env.template` file as a base.
 
 ```bash
-docker compose up -d
+cp .env.template .env
 ```
 
-The provided container definition runs Timesketch in development mode as a volume from your cloned repo. Any changes you make will appear in Timesketch automatically.
+Note the `.env` is ignored by Git: you can safely write sensitive data in it.
 
-If you see the following message you can continue
+You can optionally edit the `.env` file.
+This is useful if you need to build images with some company restrictions (accessing
+remote Ubuntu, PyPI or Node repositories).
 
-```text
-Timesketch development server is ready!
-```
-### Find out container ID for the timesketch container
+### Start a developer version of docker containers in this directory
 
 ```bash
-CONTAINER_ID="$(docker container list -f name=timesketch-dev -q)"
+docker compose up -d
 ```
 
-In the output look for CONTAINER ID for the timesketch container
-
-To write the ID to a variable, use:
-
-```bash
-export CONTAINER_ID="$(docker container list -f name=timesketch-dev -q)"
-```
+The provided container definition runs Timesketch in development mode as a volume from your cloned repo. Any changes you make will appear in Timesketch automatically.
 
-and test with
+If you see the following message you can continue
 
-```bash
-echo $CONTAINER_ID
+```text
+Timesketch development server is ready!
 ```
 
 ### Start a celery container shell
 
+Start the container in foreground (add `-d` to run in background):
+
 ```bash
-docker exec -it $CONTAINER_ID celery -A timesketch.lib.tasks worker --loglevel info
+docker compose exec timesketch \
+  celery \
+  -A timesketch.lib.tasks \
+  worker \
+  --loglevel info
 ```
 
 ### Start development webserver (and metrics server)
 
+Start the container in foreground (add `-d` to run in background):
+
 ```bash
-docker exec -it $CONTAINER_ID gunicorn --reload -b 0.0.0.0:5000 --log-file - --timeout 600 -c /usr/local/src/timesketch/data/gunicorn_config.py timesketch.wsgi:application
+docker compose exec timesketch \
+  gunicorn \
+  --reload \
+  -b 0.0.0.0:5000 \
+  --log-file - \
+  --timeout 600 \
+  -c /usr/local/src/timesketch/data/gunicorn_config.py \
+  timesketch.wsgi:application
 ```
 
 You now can access your development version at http://127.0.0.1:5000/
@@ -58,18 +69,30 @@ You can also access a metrics dashboard at http://127.0.0.1:3000/
 
 ### Non-interactive
 
-Running the following as a script after `docker compose up -d` will bring up the development environment in the background for you.
+A script applies the previous commands in background for you.
 
 ```bash
-export CONTAINER_ID="$(docker container list -f name=timesketch-dev -q)"
-docker exec $CONTAINER_ID celery -A timesketch.lib.tasks worker --loglevel info
-docker exec $CONTAINER_ID gunicorn --reload -b 0.0.0.0:5000 --log-file - --timeout 120 timesketch.wsgi:application
+docker compose up -d
+./start-frontend-ng-no-dev.sh
+```
+
+A second script starts an additional development server for the frontend
+(http://127.0.0.1:5001/).
+You need to wait a few seconds before accessing it.
+
+```bash
+docker compose up -d
+./start-frontend-ng-dev.sh
 ```
 
 ### Run tests
 
 ```bash
-docker exec -w /usr/local/src/timesketch -it $CONTAINER_ID python3 run_tests.py --coverage
+docker compose exec \
+    -w /usr/local/src/timesketch \
+    -it \
+    timesketch \
+    python3 run_tests.py --coverage
 ```
 
 That will run all tests in your docker container. It is recommended to run all tests at least before creating a pull request.

diff --git a/docker/dev/build/Dockerfile b/docker/dev/build/Dockerfile
diff --git a/docker/dev/build/docker-entrypoint.sh b/docker/dev/build/docker-entrypoint.sh
diff --git a/docker/dev/compose.yaml b/docker/dev/compose.yaml
@@ -6,26 +6,42 @@ networks:
 services:
   timesketch:
     image: us-docker.pkg.dev/osdfir-registry/timesketch/dev:latest
+    build:
+      context: ../..
+      dockerfile: docker/dev/timesketch/Dockerfile
+      args:
+        BASE_IMAGE: "${TIMESKETCH_BASE_IMAGE}"
+        GIFT_PPA_TRACK: "${GIFT_PPA_TRACK}"
+        GIFT_PPA_URL: "${GIFT_PPA_URL}"
+        NODE_VERSION: "${NODE_VERSION}"
+        NODE_PPA_URL: "${NODE_PPA_URL}"
+        NODE_NPMRC: "${NODE_NPMRC}"
+        PYTHON_PIP_CONF: "${PYTHON_PIP_CONF}"
     command: timesketch
     ports:
       - "5000:5000"
       - "5001:5001"
       - "8080:8080"
-    environment:
-      POSTGRES_USER: "timesketch"
-      POSTGRES_PASSWORD: "password"
-      POSTGRES_ADDRESS: "postgres"
-      POSTGRES_PORT: "5432"
-      OPENSEARCH_HOST: "opensearch"
-      OPENSEARCH_PORT: "9200"
-      REDIS_ADDRESS: "redis"
-      REDIS_PORT: "6379"
-      TIMESKETCH_USER: "dev"
-      TIMESKETCH_PASSWORD: "dev"
-      CHOKIDAR_USEPOLLING: "true"
-      prometheus_multiproc_dir: "/tmp/"
+    env_file:
+      - timesketch/timesketch.env
     volumes:
       - "../../:/usr/local/src/timesketch/"
+      - "./timesketch/timesketch.conf:${TIMESKETCH_CONF_DIR}/timesketch.conf:ro"
+      - "./timesketch/sigma_rules.txt:${TIMESKETCH_CONF_DIR}/sigma_rules.txt:ro"
+      - "../../data/regex_features.yaml:${TIMESKETCH_CONF_DIR}/regex_features.yaml:ro"
+      - "../../data/winevt_features.yaml:${TIMESKETCH_CONF_DIR}/winevt_features.yaml:ro"
+      - "../../data/tags.yaml:${TIMESKETCH_CONF_DIR}/tags.yaml:ro"
+      - "../../data/intelligence_tag_metadata.yaml:${TIMESKETCH_CONF_DIR}/intelligence_tag_metadata.yaml:ro"
+      - "../../data/plaso.mappings:${TIMESKETCH_CONF_DIR}/plaso.mappings:ro"
+      - "../../data/generic.mappings:${TIMESKETCH_CONF_DIR}/generic.mappings:ro"
+      - "../../data/ontology.yaml:${TIMESKETCH_CONF_DIR}/ontology.yaml:ro"
+      - "../../data/data_finder.yaml:${TIMESKETCH_CONF_DIR}/data_finder.yaml:ro"
+      - "../../data/bigquery_matcher.yaml:${TIMESKETCH_CONF_DIR}/bigquery_matcher.yaml:ro"
+      - "../../data/sigma_config.yaml:${TIMESKETCH_CONF_DIR}/sigma_config.yaml:ro"
+      - "../../data/sigma:${TIMESKETCH_CONF_DIR}/sigma:ro"
+      - "../../data/dfiq:${TIMESKETCH_CONF_DIR}/dfiq:ro"
+      - "../../data/context_links.yaml:${TIMESKETCH_CONF_DIR}/context_links.yaml:ro"
+      - "../../data/plaso_formatters.yaml:${TIMESKETCH_CONF_DIR}/plaso_formatters.yaml:ro"
     depends_on:
       - opensearch
       - postgres
@@ -35,13 +51,8 @@ services:
 
   opensearch:
     image: opensearchproject/opensearch:2.15.0
-    environment:
-      discovery.type: "single-node"
-      bootstrap.memory_lock: "true"
-      network.host: "0.0.0.0"
-      OPENSEARCH_JAVA_OPTS: "-Xms2g -Xmx2g"
-      DISABLE_INSTALL_DEMO_CONFIG: "true"
-      DISABLE_SECURITY_PLUGIN: "true" # TODO: Enable when we have migrated the python client to Opensearch as well.
+    env_file:
+      - opensearch/opensearch.env
     ports:
       - "9200:9200"
     networks:
@@ -56,9 +67,8 @@ services:
 
   postgres:
     image: postgres:13.1-alpine
-    environment:
-      POSTGRES_USER: "timesketch"
-      POSTGRES_PASSWORD: "password"
+    env_file:
+      - postgresql/postgresql.env
     ports:
       - "5432:5432"
     networks:
@@ -73,6 +83,11 @@ services:
 
   notebook:
     image: us-docker.pkg.dev/osdfir-registry/timesketch/notebook:latest
+    build:
+      context: ../..
+      dockerfile: docker/dev/notebook/Dockerfile
+      args:
+        PYTHON_PIP_CONF: "${PYTHON_PIP_CONF}"
     ports:
       - "8844:8844"
     volumes: