feat: Implement CRUD API for managing Harbor per-project Quota #3090

jopemachine · 2024-11-13T03:19:08Z

Resolves #1911.

Implement CRUD for controlling Harbor per-project Quota.

Important

Currently, the Project Quota API implemented in this PR is only available for the HarborV2 registry.

Details

In practice, the HarborV2 Quota API has no delete or create methods. But if you query a quota that does not exist, it returns -1. Using the update API at this point creates the quota. Updating it back to -1 resets it to an unlimited quota. Using this approach, I implemented all CRUD operations.
Quota mutations can only be executed by admin, superadmin, while quota queries are accessible to regular users if they have READ permissions.

Implementation Check list

REST API
Graphql API
Client SDK

Client code example

import asyncio
from ai.backend.client.session import AsyncSession

async def main():
    async with AsyncSession() as session:
        await session.Group.update_container_registry_quota("2de2b969-1d04-48a6-af16-0bc8adb3c831", 1200)

if __name__ == "__main__":
    asyncio.run(main())

GQL example

mutation {
  update_container_registry_quota(scope_id: "project:2de2b969-1d04-48a6-af16-0bc8adb3c831", quota: 100) {
    ok
    msg
  }
}

Reference

Access Harbor API documentation

Checklist: (if applicable)

Milestone metadata specifying the target backport version
Mention to the original issue
API server-client counterparts (e.g., manager API -> client SDK)

📚 Documentation preview 📚: https://sorna--3090.org.readthedocs.build/en/3090/

📚 Documentation preview 📚: https://sorna-ko--3090.org.readthedocs.build/ko/3090/

jopemachine · 2024-11-13T03:19:27Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

jopemachine · 2024-11-14T04:27:43Z

Remaining tasks: Adding REST API and SDK.

jopemachine · 2024-11-18T04:56:13Z

src/ai/backend/client/func/group.py

+        """
+        )
+
+        scope_id = f"project:{group_id}"


@fregataa If possible, instead of hardcoding the scope_id like this, I’d prefer to create a ProjectScope object and serialize it. However, since the ProjectScope type is currently located under the models directory, this violates the pants visibility rule.

What do you think about moving the types related to ScopeField under the common directory?

jopemachine · 2024-11-18T05:00:03Z

src/ai/backend/client/func/group.py

+        )
+
+        variables = {"id": base64(f"group_node:{group_id}")}
+        data = await api_session.get().Admin._query(query, variables)


@fregataa Registry quota READ operation can be executed even if the user is not an admin.
However, it seems that GQL queries in the current SDK can only be executed through Admin.
What do you think about adding this query, _query functions to User as well?

Co-authored-by: octodog <[email protected]>

HyeockJinKim · 2024-11-27T07:54:54Z

src/ai/backend/client/func/group.py

+    @classmethod
+    async def get_container_registry_quota(cls, group_id: str) -> int:
+        """
+        Delete Quota Limit for the group's container registry.


HyeockJinKim · 2024-11-27T08:07:50Z

src/ai/backend/manager/api/group.py

+    cors.add(app.router.add_route("POST", "/registry-quota", create_registry_quota))
+    cors.add(app.router.add_route("GET", "/registry-quota", read_registry_quota))
+    cors.add(app.router.add_route("PATCH", "/registry-quota", update_registry_quota))
+    cors.add(app.router.add_route("DELETE", "/registry-quota", delete_registry_quota))


can we use PATCH, DELETE method? @fregataa

HyeockJinKim · 2024-11-27T08:18:33Z

src/ai/backend/manager/models/gql_models/container_registry_utils.py

+    if (
+        not group_row
+        or not group_row.container_registry
+        or "registry" not in group_row.container_registry
+        or "project" not in group_row.container_registry
+    ):


make function for this condition to improve readability.

HyeockJinKim · 2024-11-27T08:20:15Z

src/ai/backend/manager/models/gql_models/container_registry_utils.py

+async def handle_harbor_project_quota_operation(
+    operation_type: Literal["create", "read", "update", "delete"],
+    db_sess: SASession,
+    scope_id: ScopeType,
+    quota: Optional[int],
+) -> Optional[int]:


Don't combine handler functions into one function that accept multiple operation_types.

github-actions bot assigned jopemachine Nov 13, 2024

github-actions bot added area:docs Documentations comp:manager Related to Manager component size:L 100~500 LoC labels Nov 13, 2024

This was referenced Nov 13, 2024

feat: Implement Image RBAC project scope #3035

Open

feat: Implement ImageNode, ImageNodes as per-project images API #3036

Draft

This was referenced Nov 13, 2024

feat: Implement m2m table for container_registries, and groups #3065

Open

feat: Implement APIs for associating, disassociating container_registries with groups #3067

Open

jopemachine added the type:feature Add new features label Nov 13, 2024

jopemachine added this to the 24.12 milestone Nov 13, 2024

jopemachine force-pushed the topic/11-06-feat_implement_per-project_images_api_based_on_rbac branch from 7b81657 to f0cc98f Compare November 13, 2024 06:34

jopemachine force-pushed the topic/11-13-feat_implement_management_api_for_controlling_harbor_per-project_quota branch from 47cacdd to a6068e1 Compare November 13, 2024 06:35

jopemachine force-pushed the topic/11-06-feat_implement_per-project_images_api_based_on_rbac branch from f0cc98f to f01260a Compare November 13, 2024 08:36

jopemachine force-pushed the topic/11-13-feat_implement_management_api_for_controlling_harbor_per-project_quota branch from f5a8af1 to 917dffd Compare November 13, 2024 08:36

jopemachine force-pushed the topic/11-06-feat_implement_per-project_images_api_based_on_rbac branch from f01260a to 6fcfeae Compare November 14, 2024 06:01

jopemachine force-pushed the topic/11-13-feat_implement_management_api_for_controlling_harbor_per-project_quota branch from 1be79c6 to 6b91f24 Compare November 14, 2024 06:01

jopemachine force-pushed the topic/11-06-feat_implement_per-project_images_api_based_on_rbac branch from 6fcfeae to 548f6f8 Compare November 14, 2024 06:05

jopemachine force-pushed the topic/11-13-feat_implement_management_api_for_controlling_harbor_per-project_quota branch from 6b91f24 to 81dbd9e Compare November 14, 2024 06:05

jopemachine force-pushed the topic/11-06-feat_implement_per-project_images_api_based_on_rbac branch from 548f6f8 to bda8f44 Compare November 14, 2024 06:37

jopemachine force-pushed the topic/11-13-feat_implement_management_api_for_controlling_harbor_per-project_quota branch from 81dbd9e to 33c1b79 Compare November 14, 2024 06:38

jopemachine force-pushed the topic/11-06-feat_implement_per-project_images_api_based_on_rbac branch from bda8f44 to 30b5684 Compare November 18, 2024 01:16

jopemachine force-pushed the topic/11-13-feat_implement_management_api_for_controlling_harbor_per-project_quota branch from 33c1b79 to f036297 Compare November 18, 2024 01:16

jopemachine force-pushed the topic/11-06-feat_implement_per-project_images_api_based_on_rbac branch from 30b5684 to 56003a4 Compare November 18, 2024 01:28

jopemachine force-pushed the topic/11-13-feat_implement_management_api_for_controlling_harbor_per-project_quota branch from f036297 to 0d5ccf2 Compare November 18, 2024 01:28

jopemachine force-pushed the topic/11-06-feat_implement_per-project_images_api_based_on_rbac branch from 56003a4 to 643e723 Compare November 18, 2024 01:33

jopemachine force-pushed the topic/11-13-feat_implement_management_api_for_controlling_harbor_per-project_quota branch from 0d5ccf2 to 408bf1f Compare November 18, 2024 01:33

jopemachine commented Nov 18, 2024

View reviewed changes

jopemachine and others added 25 commits November 27, 2024 15:20

feat: Add registry_quota to GroupNode

ed1a4b1

fix: Update UpdateQuota mutation

a1225c2

fix: Update resolve_registry_quota

c8fece0

fix: Update resolve_registry_quota

4f3448a

fix: Only authorized groups view the quota

780a250

feat: Add CreateQuota, DeleteQuota mutation

1a4bf70

chore: Rename function

63d5dbb

fix: Add exception handling for each operation

473b125

chore: Update schema

1e8ac1a

chore: Update error msg

cfd21fc

chore: Rename variable

a49f1ff

fix: Remove useless strenum

16b45d1

refactor: mutate_harbor_project_quota

3128b0a

refactor: Add read operation handling for code reuse

97a3022

feat: Add SDK for registry quota mutations

1a03789

fix: Broken CI

c118c7e

fix: Wrong object_name in ContainerRegistryNotFound

96fc374

feat: Implement REST API

a1f3c08

fix: Wrong exception handling

217a359

chore: Update comment

89ab7a7

chore: Rename types

896c44d

chore: update GraphQL schema dump

835d0c6

Co-authored-by: octodog <[email protected]>

chore: Rename news fragment

0a442c9

fix: Use BigInt

049dfe0

chore: update GraphQL schema dump

cae3fee

Co-authored-by: octodog <[email protected]>

jopemachine force-pushed the topic/11-13-feat_implement_management_api_for_controlling_harbor_per-project_quota branch from 6b415d5 to cae3fee Compare November 27, 2024 06:20

HyeockJinKim reviewed Nov 27, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Implement CRUD API for managing Harbor per-project Quota #3090

feat: Implement CRUD API for managing Harbor per-project Quota #3090

jopemachine commented Nov 13, 2024 •

edited by github-actions bot

Loading

jopemachine commented Nov 13, 2024 •

edited

Loading

jopemachine commented Nov 14, 2024

jopemachine Nov 18, 2024 •

edited

Loading

jopemachine Nov 18, 2024 •

edited

Loading

HyeockJinKim Nov 27, 2024

HyeockJinKim Nov 27, 2024

HyeockJinKim Nov 27, 2024

HyeockJinKim Nov 27, 2024

feat: Implement CRUD API for managing Harbor per-project Quota #3090

Are you sure you want to change the base?

feat: Implement CRUD API for managing Harbor per-project Quota #3090

Conversation

jopemachine commented Nov 13, 2024 • edited by github-actions bot Loading

Details

Implementation Check list

Client code example

GQL example

Reference

jopemachine commented Nov 13, 2024 • edited Loading

jopemachine commented Nov 14, 2024

jopemachine Nov 18, 2024 • edited Loading

Choose a reason for hiding this comment

jopemachine Nov 18, 2024 • edited Loading

Choose a reason for hiding this comment

HyeockJinKim Nov 27, 2024

Choose a reason for hiding this comment

HyeockJinKim Nov 27, 2024

Choose a reason for hiding this comment

HyeockJinKim Nov 27, 2024

Choose a reason for hiding this comment

HyeockJinKim Nov 27, 2024

Choose a reason for hiding this comment

jopemachine commented Nov 13, 2024 •

edited by github-actions bot

Loading

jopemachine commented Nov 13, 2024 •

edited

Loading

jopemachine Nov 18, 2024 •

edited

Loading

jopemachine Nov 18, 2024 •

edited

Loading