diff --git a/DATABASE_MANAGEMENT.md b/DATABASE_MANAGEMENT.md index 3117a61..028313d 100644 --- a/DATABASE_MANAGEMENT.md +++ b/DATABASE_MANAGEMENT.md @@ -6,31 +6,9 @@ SPDX-License-Identifier: CC-BY-4.0 ## Database Management -## Versioning Overview -![Versioning overview](./images/database/BaseX_Versioning.png) - -To achieve versioning (which is not available out-of-the-box), we need to add something smart to BaseX. This smart thing is [RESTXQ](http://exquery.github.io/exquery/exquery-restxq-specification/restxq-1.0-specification.html) in our case. - -With RESTXQ, functions can be created using xQuery and some added intelligence like variables and for-loops for example. +CoMPAS uses a PostgreSQL database. -Example RESTXQ function: - -``` -declare - %rest:path("/search") - %rest:query-param("term", "{$term}") - %rest:single -function page:search($term as xs:string) { - -}; -``` - -By using RESTXQ, a versioning mechanism can be created. So for example, in a edit (PUT) function we can do something like: When editing a already stored configuration, save it by incrementing the version and store as a separate configuration. The old configuration is stored in the archive database, the current version is replaced in the current database. - -In a get (GET) function, we can make distinction between newer and older versions using RESTXQ. By using xQuery syntax (scl[@version="1"] for example), we can get specific versions of a configuration. +## Versioning Overview ### Versioning type For type of versioning, we prefer [Semantic Versioning](https://semver.org/). This to keep versioning simple. For every changeset CoMPAS is going to ask if it's a major, minor or a patch. This way the version will be adjusted according to the user's needs. An example of distinction can be: @@ -52,88 +30,9 @@ This creates provenance, and version is one of them. The version attribute will Another solution could be [Branch Based Versioning](https://simon-maxen.medium.com/branch-based-versioning-5ebf6ca2bccb). This way, a configuration file can be 'branched', and can be 'merged' when the user think it's fine. When merging, a newer version number can be added (can be done in combination with semantic versioning). This in indeed a fancy way of versioning, but it's too complex for our use cases. We don't see users branching a configuration file and saving it for a couple of days, before merging it. Besides, this kind of versioning isn't supported in BaseX out of the box so we have to create it ourselves. When comparing added value to effort, this isn't what we want. -## Tech Talk - -### Points to remember -- home of BaseX = /srv/basex -- RESTXQ file extension = .xqm -- RESTXQPATH variable (in {home}/webapp/WEB-INF/web.xml) points to directory containing the RESTXQ modules (.xqm files) - - Default is '.', which is relative to the WEBPATH variable (which is {home}/webapp) - -### Example using RESTXQ - -- Run a BaseX container -- Use shell inside container (docker exec -it bash) -- create a RESTXQ module: vi /srv/basex/webapp/test.xqm for example -- copy paste the following code: - -``` -module namespace page = 'http://basex.org/examples/web-page'; - -declare %rest:path("hello/{$who}") %rest:GET function page:hello($who) { - - !Hello { $who }! - -}; -``` - -- You don't have to restart the container, when doing a REST request it seaches on the fly for functions. -- Do a GET request like http://localhost:8984/hello/World -- You will get a XML containing a title !Hello World! - -### Restrictions -A single database is restricted to 2 billion nodes (also, see [BaseX Statistics](https://docs.basex.org/wiki/Statistics)) -A node in this case is an XML node like an element, attribute, text, etc. - -### Sources -http://www.adamretter.org.uk/presentations/restxq_mugl_20120308.pdf - ## Database Rights In a microservice architecture, a microservice's database should be part of the implementation of that service and cannot be accessed directly by other services. This way, the service is loosely coupled and can be developed/scaled/deployed independently. -There are some patterns to keep persistent data private: -- private-tables-per-service -- schema-per-service -- database-server-per-service -As seen, 2 options are not available for BaseX because it's not a relational database. It doesn't have tables or schemas. -A Database-server-per-service pattern helps ensure that the services are lossely coupled. - -The CIM - IEC 61850 service for example get's their own database. If another service wants to get SCD files from this service, use the API of that particular service. - -### Where do we set the user privileges of Basex? -Basex has it's own [User Management](https://docs.basex.org/wiki/User_Management). - -It's pretty straight forward: Basex has Users that can be created. These users can have so-called permissions that can be applied to the user: -![BaseX permissions overview](./images/database/basex_permissions.png) - -In this overview, we see 'Global' permissions and 'Local' permissions. -In both permission groups, a higher permission includes all lower permissions. So a user with the 'Create' permission also has the 'Read' permission. - -All permissions are stored in a file called users.xml (which can be edited manually) inside the database directory, and is being parsed once BaseX is started. - -### How do we connect BaseX with a central identity repository/application? -BaseX doesn't have compatibility with a central identity repository (like Keycloak) out of the box available, but after discussing it with the BaseX community it's pretty comfortable to achieve this with RESTXQ or xQuery. There are examples available for making use of Keycloak: - -[Example with xQuery](https://code-repo.d4science.org/gCubeSystem/d4science-keycloak-themes/src/branch/master/src/utils/xquery) - -[Example with RESTXQ](./blob-files/code_examples/auth_sk.xqm) -Author: Marco Lettere. Origin: [BaseX Mailing List](https://mailman.uni-konstanz.de/pipermail/basex-talk/2021-May/016157.html) - -Full attached description about this example: - ->I attach here an example of an OIDC code grant flow implemented with RestXQ, BaseX permission and error handler. -The file includes a sort of library for performing the steps of the OIDC flow plus a minimal application that is registered as public client inside keycloak and which is what you should access from your browser by calling http://localhost:8984/authtest or http://localhost:8984/authtest/internal. -I've put into it also the logout procedure for performing the back-channel logout which closes the SSO session. -This is only a resume of a more generic and complex module but it should be useful as a howto and it should be as simple to install as copying the file to your BaseX' webapp folder. Use it as you like. - -### Is direct database access allowed within the microservices architecture? -For maintenance for example, it's of course allowed to have direct database access. There is no best practice available for this. For some things, you just need direct database access. - -If other microservices need access to the data of an other microservice, the only way (best practice) to do this is by API calls. - -Source: -https://microservices.io/patterns/data/database-per-service.html - ## Provenance Overview If the generation of a substation fails for example, we would like to know the provenance of the file. This way it's easier to get the cause. diff --git a/TECHNOLOGY.md b/TECHNOLOGY.md index dc60bed..4833dba 100644 --- a/TECHNOLOGY.md +++ b/TECHNOLOGY.md @@ -32,22 +32,13 @@ Microservices are deployed as Linux based Docker container. Advantages of deploy ## Database - BaseX -For the database BaseX](https://basex.org/) is chosen on following arguments: +For the database PostgreSQL is chosen on following arguments: **Pros** -- Native XML database -- Fully Open Source -- BSD license -- Easy to setup using available Docker image -- Cross-platform available -- Active community -- Multiple API's, like REST(ful) and HTTP -- [ACID guarantees](https://docs.basex.org/wiki/Transaction_Management) -- Many [usage examples](https://docs.basex.org/wiki/Clients) available in different programming languages +- Flexible and supported by many cloud providers **Cons** -- No clear use cases using BaseX -- Versioning is not out-of-the-box available. Need to use a second database to create 'versioning', which creates an archive database and a current database. And by using RESTXQ is relatively easy to create a versioning mechanism. BaseX gave [SirixDB](https://sirix.io/) as a good alternative in case we want a NoSQL database with versioning mechanism. +- Not a native XML database ## XML Processing ### XML Validation diff --git a/images/database/BaseX_Versioning.png b/images/database/BaseX_Versioning.png deleted file mode 100644 index 40784d8..0000000 Binary files a/images/database/BaseX_Versioning.png and /dev/null differ diff --git a/images/database/BaseX_Versioning.svg b/images/database/BaseX_Versioning.svg deleted file mode 100644 index 7811a8a..0000000 --- a/images/database/BaseX_Versioning.svg +++ /dev/null @@ -1,3 +0,0 @@ - - -
RESTXQ
RESTXQ

RESTXQ functions

getLatestConfiguration
/configurations/{name}/latest

getConfiguration
/configurations/{name}/version/{version}
RESTXQ functions...

REST Call

Example:
basex-container/configurations/substationA/latest

REST Call...
CoMPAS
CoMPAS
Current Versions
Current Versi...
Archive
Archive

Within Container multiple databases

One for the current versions, one for all the archiving versions. We want one separate database for current versions because of performance for example.
Within Container multiple databases...
Viewer does not support full SVG 1.1
\ No newline at end of file diff --git a/images/database/basex_permissions.png b/images/database/basex_permissions.png deleted file mode 100644 index 0eaa93e..0000000 Binary files a/images/database/basex_permissions.png and /dev/null differ