diff --git a/README.md b/README.md index 2d3476b..d2860e3 100644 --- a/README.md +++ b/README.md @@ -1,22 +1,19 @@ # RepoAchiever -[![Build](https://github.com/YarikRevich/ResourceTracker/actions/workflows/build.yml/badge.svg)](https://github.com/YarikRevich/ResourceTracker/actions/workflows/build.yml) +[![Build](https://github.com/YarikRevich/RepoAchiever/actions/workflows/build.yml/badge.svg)](https://github.com/YarikRevich/ResourceTracker/actions/workflows/build.yml) ![Linux](https://img.shields.io/badge/Linux-FCC624?style=for-the-badge&logo=linux&logoColor=black) ![MacOS](https://img.shields.io/badge/MacOS-8773f5?style=for-the-badge&logo=macos&logoColor=black) [![StandWithUkraine](https://raw.githubusercontent.com/vshymanskyy/StandWithUkraine/main/badges/StandWithUkraine.svg)](https://github.com/vshymanskyy/StandWithUkraine/blob/main/docs/README.md) ## General Information -A tool used for remote Git repositories achieving operations. +A tool used for remote content achieving operations. ![](./docs/high-level-design.png) -![](./docs/detailed-design.png) +![](./docs/detailed-design-raw.png) -![](./docs/workspace-design.png) - -RepoAchiever uses server-side data validation. It means, that on client side no data transformation or further analysis are made. -Only API Server calls are on client side responsibility. +![](./docs/internal-storage-design.png) ## Setup @@ -24,65 +21,208 @@ All setup related operations are processed via **Makefile** placed in the root d ### CLI -In order to build CLI it's required to execute the following command. Initially it cleans the environment and builds Java project using **Maven** +In order to build **RepoAchiever CLI** it's required to execute the following command. Initially it cleans the environment and builds Java project using **Maven** ```shell make build-cli ``` After the execution of command given above the executable will be generated and placed into **bin** folder in the root directory of the project -**CLI** build automatically places default **user.yaml** configuration file into ~/.resourcetracker/config directory. +**RepoAchiever CLI** build automatically places default **user.yaml** configuration file into **~/.repoachiever/config** directory. ### GUI -In order to build GUI it's required to execute the following command. Initially it cleans the environment and build Java project using **Maven** +In order to build **RepoAchiever GUI** it's required to execute the following command. Initially it cleans the environment and build Java project using **Maven** ```shell make build-gui ``` After the execution of command given above the executable will be generated and placed into **bin** folder in the root directory of the project -**GUI** build automatically compiles **API Server** and places both executable JAR and other dependencies into **~/.resourcetracker/bin/api-server** directory +**RepoAchiever GUI** build automatically compiles **RepoAchiever API Server** and places both executable JAR and other dependencies into **~/.repoachiever/bin/api-server** directory -It's highly recommended not to move **API Server** files from the default local directory +It's highly recommended not to move **RepoAchiever API Server** files from the default local directory ### API Server -In order to build **API Server** it's required to execute the following command. Initially it cleans the environment and build Java project using **Maven** +In order to build **RepoAchiever API Server** it's required to execute the following command. Initially it cleans the environment and build Java project using **Maven** ```shell make build-api-server ``` After the execution of command given above the executable will be generated and placed into **bin** folder in the root directory of the project +### Cluster + +In order to build **RepoAchiever Cluster** it's required to execute the following command. Initially it cleans the environment and build Java project using **Maven** +```shell +make build-cluster +``` + +After the execution of command given above the executable will be generated and placed into **bin** folder in the root directory of the project + ## Use cases -For both **CLI** and **GUI** examples, there was used the following user configuration file: +For both **RepoAchiever CLI** and **RepoAchiever GUI** examples, there was used the following user configuration file located at **~/.repoachiever/config** directory as **user.yaml**: ```yaml -requests: - - name: "first" - frequency: "10 * * * * *" - file: "/Volumes/Files/first.sh" -cloud: - provider: "aws" - credentials: - file: "/Volumes/Files/aws.csv" - region: "us-west-2" +# Represents section used to declare desired content. +content: + # Represents locations of VCS provider repositories, which are intended to be used for requested operations. + locations: + - name: "[YarikRevich]test:master" # Represents location definition name. + + # Enables additional content download of repositories of VCS provider repositories. + additional: true + + - name: "[YarikRevich]scripts:master" + additional: true + + - name: "[YarikRevich]odb:master" + additional: true + + - name: "[YarikRevich]tetris:main" + additional: true + + - name: "[YarikRevich]tmpm:master" + additional: true + + - name: "[YarikRevich]portfolio:master". + additional: true + + - name: "[YarikRevich]ai-commit-generator:master" + additional: true + + - name: "[YarikRevich]website:main" + additional: true + + - name: "[YarikRevich]vim-config:master" + additional: true + + - name: "[YarikRevich]critical_db_framework:master" + additional: true + +# Represents section used for VCS provider configuration. +service: + # Represents the selected VCS provider for RepoAchiever. The first part divided by the hyphen represents + # selected VCS core type and the second part defines external provider. + # + # Supported providers are "exporter" and "git-github". + provider: "git-github" + + # Represents section used for RepoAchiever Exporter configuration. + exporter: + # Represents address for the host of RepoAchiever Exporter. + host: "http://localhost:10000" + + # Represents credentials used for the selected VCS provider. Can be left null, if the provider does not + # require it. + credentials: + # Represents session identificator, used to distinguish different workspaces and thus separate content inside. + id: "1" + + # Represents authentication token intended to be used for VCS provider API access permission. + token: "ghp_UMz4cQAD87xXAtEbsPazlRcLw7kVPu1zF8dj" + +# Represents section used for RepoAchiever API Server configuration. api-server: - host: "http://localhost:8080" + # Represents address for the host of RepoAchiever API Server. + host: "http://localhost:8085" ``` -And the following request script file: -```shell -#!/bin/bash - -echo "Hello world!" +For **RepoAchiever API Server** there was used the following configuration file located at **~/.repoachiever/config** directory as **api-server.yaml**: +```yaml +# Represents section used for RepoAchiever API Server connection configuration. +connection: + # Represents port of RepoAchiever API Server used for connection establishment. + port: 8085 + +# Represents section used for RepoAchiever API Server internal communication configuration. +communication: + # Represents port of RMI registry used for RepoAchiever API Server to RepoAchiever Cluster communication. + port: 8089 + +# Represents section used for RepoAchiever API Server content configuration. +content: + # Represents format used for content download requests. + format: "zip" + +# Represents section used for RepoAchiever API Server diagnostics configuration. +diagnostics: + # Enables diagnostics functionality. + enabled: false + + # Represents section used for RepoAchiever diagnostics metrics configuration. + metrics: + # Represents port used for metrics endpoint. + port: 8090 + + # Represents section used for RepoAchiever diagnostics Grafana instance. + grafana: + # Represents port used for Grafana instance deployment. + port: 8091 + + # Represents section used for RepoAchiever diagnostics Prometheus instance. + prometheus: + # Represents port used for Prometheus instance deployment. + port: 8120 + + # Represents section used for RepoAchiever diagnostics Prometheus Node Exporter instance. + node-exporter: + # Represents port used for Prometheus Node Exporter instance deployment. + port: 8121 + +# Represents section used for RepoAchiever resource configuration. +resource: + # Represents a set of options used for RepoAchiever Cluster configuration. + cluster: + # Represents the highest amount of allocated workers per cluster. + max-workers: 5 + + # Represents the highest amount of downloaded content versions per each. + max-versions: 5 + + # Represents a set of options used for RepoAchiever Cluster worker configuration. + worker: + # Represents frequency of requests to selected VCS provider resources. + frequency: "0/30 * * * * ?" ``` ### CLI -![cli](./docs/example/cli.gif) +![cli](./docs/examples/cli.gif) ### GUI -![gui](./docs/example/gui.gif) \ No newline at end of file +![gui](./docs/examples/gui.gif) + +### Diagnostics dashboard + +For **RepoAchiever API Server** configuration the following section should be modified: +```yaml +# Represents section used for RepoAchiever API Server diagnostics configuration. +diagnostics: + # Enables diagnostics functionality. + enabled: true + + # Represents section used for RepoAchiever diagnostics metrics configuration. + metrics: + # Represents port used for metrics endpoint. + port: 8090 + + # Represents section used for RepoAchiever diagnostics Grafana instance. + grafana: + # Represents port used for Grafana instance deployment. + port: 8091 + + # Represents section used for RepoAchiever diagnostics Prometheus instance. + prometheus: + # Represents port used for Prometheus instance deployment. + port: 8120 + + # Represents section used for RepoAchiever diagnostics Prometheus Node Exporter instance. + node-exporter: + # Represents port used for Prometheus Node Exporter instance deployment. + port: 8121 +``` + +![diagnostics](./docs/examples/diagnostics.gif) \ No newline at end of file diff --git a/cli/src/main/java/com/repoachiever/entity/PropertiesEntity.java b/cli/src/main/java/com/repoachiever/entity/PropertiesEntity.java index a708a62..b5e26b1 100644 --- a/cli/src/main/java/com/repoachiever/entity/PropertiesEntity.java +++ b/cli/src/main/java/com/repoachiever/entity/PropertiesEntity.java @@ -17,6 +17,9 @@ public class PropertiesEntity { @Value(value = "${git.commit.id.abbrev}") private String gitCommitId; + @Value(value = "${rest-client.timeout}") + private String restClientTimeout; + @Value(value = "${config.default.location}") private String configDefaultLocation; diff --git a/cli/src/main/java/com/repoachiever/service/command/BaseCommandService.java b/cli/src/main/java/com/repoachiever/service/command/BaseCommandService.java index 09efdd2..e03ab5c 100644 --- a/cli/src/main/java/com/repoachiever/service/command/BaseCommandService.java +++ b/cli/src/main/java/com/repoachiever/service/command/BaseCommandService.java @@ -465,13 +465,7 @@ private void version( return; } - try { - versionExternalCommandService.process(configService.getConfig()); - } catch (ApiServerOperationFailureException e) { - logger.fatal(e.getMessage()); - - return; - } + versionExternalCommandService.process(configService.getConfig()); visualizationService.await(); } diff --git a/cli/src/main/java/com/repoachiever/service/command/external/version/VersionExternalCommandService.java b/cli/src/main/java/com/repoachiever/service/command/external/version/VersionExternalCommandService.java index 33d02c3..77d539d 100644 --- a/cli/src/main/java/com/repoachiever/service/command/external/version/VersionExternalCommandService.java +++ b/cli/src/main/java/com/repoachiever/service/command/external/version/VersionExternalCommandService.java @@ -7,6 +7,7 @@ import com.repoachiever.service.client.info.version.VersionInfoClientService; import com.repoachiever.service.command.common.ICommand; import com.repoachiever.service.visualization.state.VisualizationState; +import lombok.SneakyThrows; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Service; @@ -24,7 +25,7 @@ public class VersionExternalCommandService implements ICommand { /** * @see ICommand */ - public void process(ConfigEntity config) throws ApiServerOperationFailureException { + public void process(ConfigEntity config) { visualizationState.getLabel().pushNext(); VersionInfoClientService versionInfoClientService = @@ -36,11 +37,12 @@ public void process(ConfigEntity config) throws ApiServerOperationFailureExcepti visualizationState.addResult( String.format( "API Server version: %s", versionInfoResult.getExternalApi().getHash())); - } finally { - visualizationState.addResult( - String.format("Client version: %s", properties.getGitCommitId())); + } catch (ApiServerOperationFailureException ignored) { } + visualizationState.addResult( + String.format("Client version: %s", properties.getGitCommitId())); + visualizationState.getLabel().pushNext(); } } diff --git a/cli/src/main/resources/application.properties b/cli/src/main/resources/application.properties index 95536b3..e7dd9d4 100644 --- a/cli/src/main/resources/application.properties +++ b/cli/src/main/resources/application.properties @@ -1,6 +1,9 @@ # Describes Spring related properties. spring.main.web-application-type=NONE +# Describes timeout used for RepoAchiever API Server client. +rest-client.timeout=15 + # Describes location of application configuration file. config.default.location=${user.home}/.repoachiever/config/user.yaml diff --git a/docs/detailed-design-raw.md b/docs/detailed-design-raw.md index 046c305..c663a5b 100644 --- a/docs/detailed-design-raw.md +++ b/docs/detailed-design-raw.md @@ -2,75 +2,95 @@ !pragma teoz true title - Detailed design of "ResourceTracker" + Detailed design of "RepoAchiever" end title +box "External environment" #MOTIVATION actor "Client" as client +actor "Diagnostics" as diagnostics +end box box "Control plain" #MOTIVATION participant "API Server" as apiserver +database "Local storage" as localstorage + +entity "Metrics registry" as metricsregistry -box "Cloud environment" #Lavender -queue "Kafka" as kafka -participant "Kafka starter" as kafkastarter -participant "Agent" as agent -entity "Cloud provider" as cloudprovider +box "Decentralized environment" #Lavender +participant "Cluster" as cluster +entity "Worker" as worker end box end box -note over [[[[[[kafka]]]]]]: Kafka is considered to be used in persisted mode +box "Third-party environment" #MOTIVATION +entity "External service" as externalservice +end box opt "endpoints" -opt "/v1/secrets/acquire [POST]" -apiserver -> cloudprovider: validate provided credentials -cloudprovider -> apiserver: validation result +opt "/v1/content [POST]" +localstorage -> apiserver: retrieve details about all the available content for the given user end -opt "/v1/topic/logs [GET]" -apiserver -> kafka: retrieve state for the given "logs" topic -kafka -> apiserver: transform data stream according to the specified filters +opt "/v1/content/apply [POST]" +apiserver -> externalservice: validate provided credentials +apiserver -> localstorage: apply provided user configuration +apiserver -> cluster: update topology according to the new configuration end -opt "/v1/terraform/apply [POST]" -apiserver -> cloudprovider: deploy resource tracking infrastructure -apiserver -> kafkastarter: request kafka cluster startup -kafkastarter -> kafka: start kafka cluster +opt "/v1/content/withdraw [DELETE]" +apiserver -> localstorage: remove configuration for the given user +apiserver -> cluster: remove topology end -opt "/v1/terraform/destroy [POST]" -apiserver -> cloudprovider: destroy resource tracking infrastructure +opt "/v1/content/download [POST]" +localstorage -> apiserver: retrieve requested content in a form of achieve end +opt "/v1/content/clean [DELETE]" +apiserver -> localstorage: remove requested content for the given user +end +opt "/v1/content/clean/all [DELETE]" +apiserver -> localstorage: remove all content for the given user end -opt "agent execution flow" -agent -> cloudprovider: execute remote operations -agent <-- cloudprovider: remote operation result -agent --> kafka: push latest resource state to "logs" topic +opt "cluster execution flow" +cluster -> worker: allocate instances according to the provided locations +worker -> externalservice: retrieve content with the given identification +worker -> apiserver: transfer retrieved content end opt "requests" -note over client: Uses properties specified in a client\nconfiguration file located in\n a common directory -opt "credentials validation" -client -> apiserver: /v1/secrets/acquire [POST] + +opt "retrieve available content" +client -> apiserver: /v1/content [POST] +end +opt "configuration application" +client -> apiserver: /v1/content/apply [POST] end -opt "script validation" -client -> apiserver: /v1/script/acquire [POST] +opt "configuration withdrawal" +client -> apiserver: /v1/content/withdraw [DELETE] +end +opt "download content" +client -> apiserver: /v1/content/download [POST] +end +opt "clean selected content" +client -> apiserver: /v1/content/clean [DELETE] +end +opt "clean all the content for the given user" +client -> apiserver: /v1/content/clean/all [DELETE] end opt "health check" client -> apiserver: /v1/health [GET] end -opt "readiness check" -client -> apiserver: /v1/readiness [GET] -end -opt "version validation" +opt "version retrieval" client -> apiserver: /v1/info/version [GET] end -opt "infrustructure deployment" -client -> apiserver: /v1/terraform/apply [POST] +opt "topology retrieval" +client -> apiserver: /v1/info/topology [GET] end -opt "infrustructure clean up" +opt "" client -> apiserver: /v1/terraform/destroy [POST] end -opt "state retrieval" -client -> apiserver: /v1/topic/logs [GET] -end +opt "metrics retrieval" +diagnostics -> metricsregistry: / [GET] +end + end ``` \ No newline at end of file diff --git a/docs/detailed-design-raw.png b/docs/detailed-design-raw.png new file mode 100644 index 0000000..2611c48 Binary files /dev/null and b/docs/detailed-design-raw.png differ diff --git a/docs/detailed-design.png b/docs/detailed-design.png deleted file mode 100644 index 2c30540..0000000 Binary files a/docs/detailed-design.png and /dev/null differ diff --git a/docs/examples/cli.gif b/docs/examples/cli.gif new file mode 100644 index 0000000..5aca77d Binary files /dev/null and b/docs/examples/cli.gif differ diff --git a/docs/examples/diagnostics.gif b/docs/examples/diagnostics.gif new file mode 100644 index 0000000..8aa2708 Binary files /dev/null and b/docs/examples/diagnostics.gif differ diff --git a/docs/examples/gui.gif b/docs/examples/gui.gif new file mode 100644 index 0000000..787f5f9 Binary files /dev/null and b/docs/examples/gui.gif differ diff --git a/docs/high-level-design-raw.md b/docs/high-level-design-raw.md deleted file mode 100644 index 0f4c954..0000000 --- a/docs/high-level-design-raw.md +++ /dev/null @@ -1,23 +0,0 @@ -```plantuml -title - -High-level design of "RepoArchiver" - -end title - -actor "Client" - -component "Control plane" { -node "API Server" -node "Prometheus" - -cloud " Decentralized environment" { -node "Cluster" -entity "Worker" -} - -[Client] <--> [API Server]: " Send requests" -[API Server] <-> [Cluster]: " Schedule content processing" -[Cluster] <--> [Worker]: " Propagate individual requests " -[API Server] --> [Prometheus]: " Export diagnostics" -``` \ No newline at end of file diff --git a/docs/high-level-design.md b/docs/high-level-design.md new file mode 100644 index 0000000..e325b91 --- /dev/null +++ b/docs/high-level-design.md @@ -0,0 +1,33 @@ +```plantuml +title + +High-level design of "RepoAchiever" + +end title + +cloud "External environment" { +actor "Client" +actor "Diagnostics" +} + +component "Control plane" { +node "API Server" +node "Metrics registry" + +cloud " Decentralized environment" { +node "Cluster" +entity "Worker" +} +} + +cloud " Third-party environment" { +entity "External service" +} + +[Client] <--> [API Server]: " Send requests" +[API Server] <-> [Cluster]: " Schedule content processing" +[Cluster] <--> [Worker]: " Propagate individual requests " +[Worker] <--> [External service]: " Retrieve content" +[API Server] <-- [Metrics registry]: " Retrieve metrics" +[Diagnostics] --> [Metrics registry]: " Scrap metrics" +``` \ No newline at end of file diff --git a/docs/high-level-design.png b/docs/high-level-design.png index d589fff..3efc06c 100644 Binary files a/docs/high-level-design.png and b/docs/high-level-design.png differ diff --git a/docs/internal-storage-design.md b/docs/internal-storage-design.md new file mode 100644 index 0000000..87107c6 --- /dev/null +++ b/docs/internal-storage-design.md @@ -0,0 +1,16 @@ +```plantuml +@startwbs +title + +Internal storage design of "RepoAchiever" + +end title + +* Workspace unit +** Repository +*** Raw +**** N-versions +*** Additional(PRs) +**** N-versions +@endwbs +``` \ No newline at end of file diff --git a/docs/internal-storage-design.png b/docs/internal-storage-design.png new file mode 100644 index 0000000..3efc42e Binary files /dev/null and b/docs/internal-storage-design.png differ diff --git a/docs/workspace-design.md b/docs/workspace-design.md deleted file mode 100644 index 1c72563..0000000 --- a/docs/workspace-design.md +++ /dev/null @@ -1,18 +0,0 @@ -```plantuml -@startwbs -title - -Workspace design of "RepoArchiver API Server" - -end title - -* Workspace unit -** Content -*** Certain repository -**** Repository version -** Metadata -*** PRs -*** Issues -*** Releases -@endwbs -``` \ No newline at end of file diff --git a/docs/workspace-design.png b/docs/workspace-design.png deleted file mode 100644 index a53a228..0000000 Binary files a/docs/workspace-design.png and /dev/null differ