Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Machinery configuration via API #13056

Open
wants to merge 18 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/admin/machine.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@
The services translate from the source language as configured at
:ref:`component`, see :ref:`component-source_language`.

Per-project automatic suggestion can also be configured via the :ref:`api`.

.. seealso::

:ref:`machine-translation`
Expand Down Expand Up @@ -187,7 +189,7 @@

.. seealso::

`DeepL translator <https://www.deepl.com/translator>`_,

Check warning on line 192 in docs/admin/machine.rst

View workflow job for this annotation

GitHub Actions / Linkcheck

https://www.deepl.com/translator to https://www.deepl.com/en/translator

Check warning on line 192 in docs/admin/machine.rst

View workflow job for this annotation

GitHub Actions / Linkcheck

https://www.deepl.com/pro to https://www.deepl.com/en/pro
`DeepL pricing <https://www.deepl.com/pro>`_,
`DeepL API documentation <https://developers.deepl.com/docs>`_

Expand Down
24 changes: 24 additions & 0 deletions docs/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1235,6 +1235,30 @@ Projects
:>json string full_name: Full name of the contributor
:>json string change_count: Number of changes done in the time range


.. http:get:: /api/projects/{string:project}/machinery_settings/

.. versionadded:: 5.9

Returns automatic suggestion settings for a project, consisting of the configurations defined for each translation service installed.

:param project: Project URL slug
:type project: string
:>json object suggestion_settings: Configuration for all installed services.


.. http:post:: /api/projects/{string:project}/machinery_settings/

.. versionadded:: 5.9

Create or update the service configuration for a project.

:param project: Project URL slug
:type project: string
:form string service: Service name
:form string configuration: Service configuration in JSON


Components
++++++++++

Expand Down
4 changes: 4 additions & 0 deletions docs/changes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ Weblate 5.9
Not yet released.

**New features**
* Per-project :ref:`machine-translation-setup` can now be configured via the Project :ref:`api`.

* Added :http:get:`/api/projects/{string:project}/machinery_settings/`.
* Added :http:post:`/api/projects/{string:project}/machinery_settings/`.

* Translation memory import now supports files with XLIFF, PO and CSV formats, see :ref:`memory-user` and `import_memory` command in :ref:`manage`.
* The registration CAPTCHA now includes proof-of-work mechanism ALTCHA.
Expand Down
75 changes: 75 additions & 0 deletions weblate/api/serializers.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,13 @@

from django.conf import settings
from django.db.models import Model
from drf_spectacular.extensions import OpenApiSerializerExtension
from drf_spectacular.plumbing import build_basic_type, build_object_type
from drf_spectacular.utils import (
OpenApiExample,
extend_schema_serializer,
inline_serializer,
)
from rest_framework import serializers

from weblate.accounts.models import Subscription
Expand Down Expand Up @@ -44,6 +51,7 @@
)

if TYPE_CHECKING:
from drf_spectacular.openapi import AutoSchema
from rest_framework.request import Request

_MT = TypeVar("_MT", bound=Model) # Model Type
Expand Down Expand Up @@ -82,7 +90,7 @@
super().__init__(**kwargs)
self.lookup_field = lookup_field

def get_url(self, obj, view_name, request: AuthenticatedHttpRequest, format): # noqa: A002

Check failure on line 93 in weblate/api/serializers.py

View workflow job for this annotation

GitHub Actions / mypy

Argument 3 of "get_url" is incompatible with supertype "HyperlinkedRelatedField"; supertype defines the argument type as "Request"
"""
Given an object, return the URL that hyperlinks to the object.

Expand Down Expand Up @@ -273,7 +281,7 @@
return data


class RoleSerializer(serializers.ModelSerializer[Role]):

Check warning on line 284 in weblate/api/serializers.py

View workflow job for this annotation

GitHub Actions / API Lint

[RoleViewSet > RoleSerializer]: could not resolve serializer field "PermissionSerializer()". Defaulting to "string"
permissions = PermissionSerializer(many=True)

class Meta:
Expand Down Expand Up @@ -328,7 +336,7 @@
many=True,
read_only=True,
)
componentlists = serializers.HyperlinkedRelatedField(

Check failure on line 339 in weblate/api/serializers.py

View workflow job for this annotation

GitHub Actions / mypy

Need type annotation for "componentlists"
view_name="api:componentlist-detail",
lookup_field="slug",
many=True,
Expand Down Expand Up @@ -368,7 +376,7 @@
def __init__(self, *args, **kwargs) -> None:
super().__init__(*args, **kwargs)
user = self.context["request"].user
self.fields["defining_project"].queryset = user.managed_projects

Check failure on line 379 in weblate/api/serializers.py

View workflow job for this annotation

GitHub Actions / mypy

"Field[Any, Any, Any, Any]" has no attribute "queryset"


class ProjectSerializer(serializers.ModelSerializer[Project]):
Expand Down Expand Up @@ -397,6 +405,9 @@
credits_url = serializers.HyperlinkedIdentityField(
view_name="api:project-credits", lookup_field="slug"
)
machinery_settings = serializers.HyperlinkedIdentityField(
view_name="api:project-machinery-settings", lookup_field="slug"
)

class Meta:
model = Project
Expand All @@ -422,6 +433,7 @@
"enable_hooks",
"language_aliases",
"enforced_2fa",
"machinery_settings",
)
extra_kwargs = {
"url": {"view_name": "api:project-detail", "lookup_field": "slug"}
Expand All @@ -432,7 +444,7 @@
def get_attribute(self, instance):
if instance.linked_component:
instance = instance.linked_component
return getattr(instance, self.source)

Check failure on line 447 in weblate/api/serializers.py

View workflow job for this annotation

GitHub Actions / mypy

Argument 2 to "getattr" has incompatible type "Callable[..., Any] | str | None"; expected "str"


class RepoField(LinkedField):
Expand Down Expand Up @@ -615,7 +627,7 @@
project = kwargs["context"]["project"]

if project is not None:
self.fields["category"].queryset = project.category_set.all()

Check failure on line 630 in weblate/api/serializers.py

View workflow job for this annotation

GitHub Actions / mypy

"Field[Any, Any, Any, Any]" has no attribute "queryset"

def validate_enforced_checks(self, value):
if not isinstance(value, list):
Expand Down Expand Up @@ -673,7 +685,7 @@
if "project" in self._context:
result["project"] = self._context["project"]
elif self.instance:
result["project"] = self.instance.project

Check failure on line 688 in weblate/api/serializers.py

View workflow job for this annotation

GitHub Actions / mypy

Item "Sequence[Component]" of "Component | Sequence[Component]" has no attribute "project"

# Workaround for https://github.com/encode/django-rest-framework/issues/7489
if "category" not in result:
Expand Down Expand Up @@ -705,14 +717,14 @@
if docfile is not None or zipfile is not None:
# Validate name/slug uniqueness, this has to be done prior docfile/zipfile
# extracting
instance.clean_unique_together()

Check failure on line 720 in weblate/api/serializers.py

View workflow job for this annotation

GitHub Actions / mypy

Item "Sequence[Component]" of "Component | Sequence[Component]" has no attribute "clean_unique_together"

# Handle uploaded files
if docfile is not None:
fake = create_component_from_doc(attrs, docfile)
instance.template = attrs["template"] = fake.template

Check failure on line 725 in weblate/api/serializers.py

View workflow job for this annotation

GitHub Actions / mypy

Item "Sequence[Component]" of "Component | Sequence[Component]" has no attribute "template"
instance.new_base = attrs["new_base"] = fake.template

Check failure on line 726 in weblate/api/serializers.py

View workflow job for this annotation

GitHub Actions / mypy

Item "Sequence[Component]" of "Component | Sequence[Component]" has no attribute "new_base"
instance.filemask = attrs["filemask"] = fake.filemask

Check failure on line 727 in weblate/api/serializers.py

View workflow job for this annotation

GitHub Actions / mypy

Item "Sequence[Component]" of "Component | Sequence[Component]" has no attribute "filemask"
if zipfile is not None:
try:
create_component_from_zip(attrs, zipfile)
Expand Down Expand Up @@ -1513,3 +1525,66 @@
child=serializers.IntegerField(), source="get_celery_queues"
)
name = serializers.CharField(source="get_name")


@extend_schema_serializer(
examples=[
OpenApiExample(
"Service settings example",
value={
"service": "service_name",
"configuration": {"key": "xxxxx", "url": "https://api.service.com/"},
},
)
]
)
class SingleServiceConfigSerializer(serializers.Serializer):
service = serializers.CharField()
configuration = serializers.DictField()


@extend_schema_serializer(
examples=[
OpenApiExample(
"Service settings example",
value={
"service1": {"key": "XXXXXXX", "url": "https://api.service.com/"},
"service2": {"secret": "SECRET_KEY", "credentials": "XXXXXXX"},
},
request_only=False,
response_only=True,
)
]
)
class ProjectMachinerySettingsSerializer(serializers.Serializer):
def to_representation(self, instance: Project):
return dict(instance.machinery_settings)


class ProjectMachinerySettingsSerializerExtension(OpenApiSerializerExtension):
target_class = ProjectMachinerySettingsSerializer

def map_serializer(self, auto_schema: AutoSchema, direction):
return build_object_type(properties={"service_name": build_basic_type(dict)})


def edit_service_settings_response_serializer(*codes) -> int:
_serializers = {
200: inline_serializer(
"Simple message serializer",
fields={
"message": serializers.CharField(),
},
),
201: inline_serializer(
"Simple message serializer",
fields={
"message": serializers.CharField(),
},
gersona marked this conversation as resolved.
Show resolved Hide resolved
),
400: inline_serializer(
"Simple error message serializer",
gersona marked this conversation as resolved.
Show resolved Hide resolved
fields={"errors": serializers.CharField()},
),
}
return {code: _serializers[code] for code in codes}
213 changes: 213 additions & 0 deletions weblate/api/tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
from datetime import UTC, datetime, timedelta
from io import BytesIO

import responses
from django.core.files import File
from django.urls import reverse
from rest_framework.exceptions import ErrorDetail
Expand Down Expand Up @@ -2010,6 +2011,218 @@ def test_credits(self) -> None:
)
self.assertEqual(response.data, [])

@responses.activate
def test_install_machinery(self) -> None:
"""Test the machinery settings API endpoint for various scenarios."""
from weblate.machinery.tests import AlibabaTranslationTest, DeepLTranslationTest

# unauthenticated get
self.do_request(
"api:project-machinery-settings",
self.project_kwargs,
method="get",
code=403,
authenticated=True,
superuser=False,
)

# unauthenticated post
self.do_request(
"api:project-machinery-settings",
self.project_kwargs,
method="post",
code=403,
authenticated=True,
superuser=False,
request={"service": "weblate"},
)

# missing service
response = self.do_request(
"api:project-machinery-settings",
self.project_kwargs,
method="post",
code=400,
superuser=True,
request={},
)

# unknown service
response = self.do_request(
"api:project-machinery-settings",
self.project_kwargs,
method="post",
code=400,
superuser=True,
request={"service": "unknown"},
)

# create configuration: no form
response = self.do_request(
"api:project-machinery-settings",
self.project_kwargs,
method="post",
code=201,
superuser=True,
request={"service": "weblate"},
)

# missing required field
response = self.do_request(
"api:project-machinery-settings",
self.project_kwargs,
method="post",
code=400,
superuser=True,
request={
"service": "deepl",
"configuration": {"wrong": ""},
},
format="json",
)

# invalid field with multipart
response = self.do_request(
"api:project-machinery-settings",
self.project_kwargs,
method="post",
code=400,
superuser=True,
request={
"service": "deepl",
"configuration": "{malformed_json",
},
)

# create configuration: valid form with multipart
DeepLTranslationTest.mock_response()
response = self.do_request(
"api:project-machinery-settings",
self.project_kwargs,
method="post",
code=201,
superuser=True,
request={
"service": "deepl",
"configuration": '{"key": "x", "url": "https://api.deepl.com/v2/"}',
},
)

# create configuration: valid form with json
AlibabaTranslationTest().mock_response()
response = self.do_request(
"api:project-machinery-settings",
self.project_kwargs,
method="post",
code=201,
superuser=True,
request={
"service": "alibaba",
"configuration": {
"key": "alibaba-key-v1",
"secret": "alibaba-secret",
"region": "alibaba-region",
},
},
format="json",
)

# update configuration: incorrect method
response = self.do_request(
"api:project-machinery-settings",
self.project_kwargs,
method="post",
code=400,
superuser=True,
request={
"service": "deepl",
"configuration": {
"key": "deepl-key-v1",
"url": "https://api.deepl.com/v2/",
},
},
format="json",
)

# update configuration: valid method
response = self.do_request(
"api:project-machinery-settings",
self.project_kwargs,
method="patch",
code=200,
superuser=True,
request={
"service": "deepl",
"configuration": {
"key": "deepl-key-v2",
"url": "https://api.deepl.com/v2/",
},
},
format="json",
)

# list configurations
response = self.do_request(
"api:project-machinery-settings",
self.project_kwargs,
method="get",
code=200,
superuser=True,
)
self.assertIn("weblate", response.data)
self.assertEqual(response.data["deepl"]["key"], "deepl-key-v2")
self.assertEqual(response.data["alibaba"]["key"], "alibaba-key-v1")

# remove configuration
response = self.do_request(
"api:project-machinery-settings",
self.project_kwargs,
method="patch",
code=200,
superuser=True,
request={"service": "deepl", "configuration": None},
format="json",
)

# check configuration no longer exists
response = self.do_request(
"api:project-machinery-settings",
self.project_kwargs,
method="get",
code=200,
superuser=True,
)
self.assertNotIn("deepl", response.data)

# replace all configurations
new_config = {
"deepl": {"key": "deepl-key-v3", "url": "https://api.deepl.com/v2/"},
"alibaba": {
"key": "alibaba-key-v2",
"secret": "alibaba-secret",
"region": "alibaba-region",
},
}
response = self.do_request(
"api:project-machinery-settings",
self.project_kwargs,
method="put",
code=201,
superuser=True,
request=new_config,
format="json",
)

# check all configurations
response = self.do_request(
"api:project-machinery-settings",
self.project_kwargs,
method="get",
superuser=True,
)

self.assertEqual(new_config, response.data)


class ComponentAPITest(APIBaseTest):
def setUp(self) -> None:
Expand Down
Loading
Loading