Skip to content

Latest commit

 

History

History
298 lines (235 loc) · 16.3 KB

README.md

File metadata and controls

298 lines (235 loc) · 16.3 KB
Raw

layman

IntegrationTests

Publishing geospatial data online through REST API.

Requirements

  • at least 3 GB RAM
  • at least 5 GB disk space
  • Linux kernel v4.11+ (because of QGIS server and its Qt 5.10+/libQt5Core dependency), for example

    • Ubuntu 18.04.2+

    • CentOS 8+

    • Fedora 29+

      You can check your kernel version with uname -r.

  • Docker Engine v20.10.13+ including Docker Compose v2+
    • installation instructions for centos 7, including docker-compose-plugin installation

Optionally

  • linux or any tool to run tasks defined in Makefile using make command
    • can be replaced by running commands defined in Makefile directly
  • git
    • can be replaced by downloading ZIP archive of the repository

Installation

git clone https://github.com/LayerManager/layman.git
cd layman
# checkout the latest release in current branch
git checkout $(git describe --abbrev=0 --tags)

Run

There are following ways how to run Layman:

Run demo

This is the easiest way how to start layman for demonstration purposes. It includes also external dependencies. However it's not meant for production and it's not safe for production. Performance might be also an issue.

# use demo settings
cp .env.demo .env

# start dockerized containers in background
make start-demo-full

Initial startup may take few minutes (download docker images, build it, run it). You are interested in container named layman. You can check its logs with command

docker logs -f layman

Wait until you see something like


[2019-10-30 13:45:36 +0000] [12] [INFO] Layman successfully started!

Then visit http://localhost/. You will see simple web client that interacts with REST API. To check if Layman is running, call GET Version.

To stop running service, run make stop-demo.

Run demo with authentication

By default, demo starts without authentication provider. You can publish layers and maps as anonymous user, but you can't log in nor publish any private data.

To be able to use authentication and publish private data, follow these steps:

  • Stop demo instance by make stop-demo
  • Copy all OAUTH_* environment variables from .env.dev to .env file.
  • In .env change environment variables: 
    LAYMAN_AUTHN_MODULES=layman.authn.oauth2,layman.authn.http_header
OAUTH2_CALLBACK_URL=http://localhost/client/authn/oauth2-provider/callback
  • Start demo with make start-demo-full-with-optional-deps

Then you can log in with automatically provided Wagtail user layman and password laymanpwd.

Configuration

Layman's source code provides settings suitable for development, testing and demo purposes. Furthermore, there exists Makefile with predefined commands for each purpose including starting all necessary services (both in background and foreground) and stoping it.

Layman's configuration is split into three levels:

  • docker-compose.*.yml files used as docker-compose configuration files with most general settings of docker containers including volume mappings, port mappings, container names and startup commands
  • .env* files with environment settings of both build stage and runtime of docker containers
  • src/layman_settings.py Python module with settings of Layman's Python modules for runtime

Files at all three levels are suffixed with strings that indicates what they are intended to:

When you are switching between different contexts (e.g. between demo and dev), always check that you are using settings intended for your context, especially

  • .env* file (check env_file properties in docker-compose.*.yml file)

Also, anytime you change .env file, remember to rebuild docker images as some environment variables affect build stage of docker images. Rebuild happens automatically as a part of make start-demo* commands.

Dependencies

Layman has many dependencies. Most of them are shipped with Layman. However, there are some external dependencies that should be treated carefully:

  • PostgreSQL & PostGIS
  • QGIS Server
  • GeoServer
  • Redis
  • Micka

These external dependencies are shipped with Layman for development, testing and demo purposes. They are grouped in docker-compose.deps*.yml files.

However, if you want to run Layman in production, it is strongly recommended to install external dependencies separately. Recommended (i.e. tested) versions are:

  • PostgreSQL 13.3 & PostGIS 3.1
  • QGIS Server 3.16.1
  • GeoServer 2.21.2
  • Redis 4.0
  • Micka 2020.014 (versions >=2020.010 probably work too)

Upgrade

Layman follows semantic versioning, so any change in MINOR and PATCH version should be backwards compatible. Still, human make mistakes, so it's recommended to backup data directories as a part of each upgrade.

General steps to upgrade layman to MINOR or PATCH version:

  1. Stop layman. For demo configuration, make stop-demo, for dev configuration make stop-dev.
  2. Backup data directories. By default, they are located at
    • layman_data
    • deps/*/data (data directories of external dependencies)
  3. Follow Upgrade requirements in Changelog of all MINOR and PATCH versions greater than your current version and lower or equal to the version you are upgrading to.
    • If you run Layman in development mode (e.g. by make start-dev), run also
      • make build-dev
      • make client-build
      • make timgen-build
      • make wagtail-build
      • make micka-build
  4. If you are expecting long-running upgrade, run standalone upgrade, otherwise Gunicorn could time out. The command depends on how you are starting Layman.
    • If you are starting Layman with make start-demo, run make upgrade-demo.
    • If you are starting Layman with make start-demo-full or make start-demo-full-with-optional-deps, run make upgrade-demo-full.
    • If you are starting Layman with make start-dev, you don't need to run standalone migration.
  5. Start Layman.

NOTE: If upgrade failed due to timeout of request to GeoServer, run make upgrade-after-timeout to finish upgrade. NOTE: If you run in problems during upgrade process, especially before v1.12.0, you can try to upgrade gradually through individual minor versions.

Run in production

To run layman in production, you need to provide external dependencies and configure layman manually.

When providing external dependencies, check their production-related documentation:

Within PostgreSQL, you need to provide one database for Layman and one database for Micka. For Layman, you also need to provide one user LAYMAN_PG_USER who needs enough privileges to create new schemas in LAYMAN_PG_DBNAME database. The user also needs access to public schema where PostGIS must be installed.

If you are using other DB schema than internal role service schema as role service, you need to provide all admin records.

Within QGIS Server, you do not need to provide anything special.

Within GeoServer, you need to provide either admin password GEOSERVER_ADMIN_PASSWORD, or one Layman user LAYMAN_GS_USER. If admin password is provided, Layman will create the Layman user automatically. URL path of the GeoServer must be /geoserver/.

Within Redis, you need to provide two databases, one for Layman, second for Layman Test Client. Connection strings are defined by LAYMAN_REDIS_URL and LTC_REDIS_URL.

Within Micka, you need to provide one user with editor privileges, whose credentials are defined by CSW_BASIC_AUTHN.

After providing external dependencies there is time to provide internal dependencies (system-level, python-level and node.js-level dependencies). You can either use our docker and docker compose configuration to generate docker images that already provides internal dependencies, or you can provide internal dependencies by yourself (if you prefer not to use docker in production).

System-level dependencies includes

Pipenv is recommended tool for installing python-level dependencies. Both Pipfile and Pipfile.lock are located in docker/ directory.

Npm is recommended tool for installing node.js-level dependencies. Both package.json and package-lock.json are located in timgen/ directory.

Next you need to choose how you deploy Layman. As Layman is Flask application, check Flask's deployment options. Layman is safe to run with multiple WSGI Flask processes and with multiple Celery worker processes.

Configure Layman using environment settings. Demo configuration is a good starting point to setup Layman for production, however it needs to be adjusted carefully.

Last, start layman and necessary services:

  • thumbnail image generator (Timgen) using npm (see startup command of timgen docker compose service)
  • Layman client using npm (see startup command of layman_client docker compose service)
  • Layman using your deployment server (see startup command of layman docker compose service)
  • Layman celery worker using python (see startup command of celery_worker docker compose service)

Run in development

Suitable for development only.

Before the first run:

# use dev settings
cp .env.dev .env

Now everything is ready to start:

# start all needed dockerized containers 
make start-dev

Initial startup may take few minutes (download docker images, build it, run it). Wait until you see something like

[2020-12-13 09:57:00,968] INFO in __init__: Layman successfully started!

in log of layman_dev container:

# see logs from Layman 
docker logs -f layman_dev

Then visit http://localhost:8000/. You will see simple web client that interacts with REST API. You can also log in with automatically provided Wagtail user layman and password laymanpwd.

To stop running service run:

# stop all needed dockerized containers 
make stop-dev

Mount some volumes as non-root user

By default, docker run all containers as root user. It's possible to change it by defining UID_GID permanent environment variable. First stop all running containers:

make stop-dev

The UID_GID variable should look like "<user id>:<group id>" of current user, e.g. UID_GID="1000:1000". As it should be permanent, you can solve it for example by adding following line to your ~/.bashrc file:

export UID_GID="1000:1000"

and restart terminal. Verification:

$ echo $UID_GID
1000:1000

Then change ownership of some directories

make prepare-dirs
sudo chown -R 1000:1000 layman_data/
sudo chown -R 1000:1000 layman_data_test/
sudo chown -R 1000:1000 src/
sudo chown -R 1000:1000 tmp/

and restart layman

make start-dev

Test

⚠️ It will delete

Default values are defined in .env.test

# test related mainly to REST API endpoints of layers and maps
make test-static
make test-separated

# other tests
make test

For more information about tests, look at test-related documentation for developers.