Add cohere embedding for ai-cache #1563

ayanami-desu · 2024-12-02T10:59:29Z

Ⅰ. Describe what this PR did

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)?

Ⅳ. Describe how to verify it

docker-compose.yml

services:
  envoy:
    image: higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/gateway:v2.0.2
    entrypoint: /usr/local/bin/envoy
    command: -c /etc/envoy/envoy.yaml --component-log-level wasm:debug
    networks:
    - wasmtest
    ports:
    - "10000:10000"
    volumes:
    - ./envoy.yaml:/etc/envoy/envoy.yaml
    - ./main.wasm:/etc/envoy/main.wasm
    - ./ai.wasm:/etc/envoy/ai.wasm

networks:
  wasmtest: {}

envoy.yaml

admin:
  address:
    socket_address:
      protocol: TCP
      address: 0.0.0.0
      port_value: 9901
static_resources:
  listeners:
    - name: listener_0
      address:
        socket_address:
          protocol: TCP
          address: 0.0.0.0
          port_value: 10000
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                scheme_header_transformation:
                  scheme_to_overwrite: https
                stat_prefix: ingress_http
                # Output envoy logs to stdout
                access_log:
                  - name: envoy.access_loggers.stdout
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
                # Modify as required
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: local_service
                      domains: [ "*" ]
                      routes:
                        - match:
                            prefix: "/"
                          route:
                            cluster: deepseek
                            timeout: 300s
                http_filters:
                  - name: wasmtest
                    typed_config:
                      "@type": type.googleapis.com/udpa.type.v1.TypedStruct
                      type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
                      value:
                        config:
                          name: wasmtest
                          vm_config:
                            runtime: envoy.wasm.runtime.v8
                            code:
                              local:
                                filename: /etc/envoy/ai.wasm
                          configuration:
                            "@type": "type.googleapis.com/google.protobuf.StringValue"
                            value: |
                              {
                                "provider": {
                                  "type": "deepseek",
                                  "apiTokens": [
                                    "sk-"
                                  ]
                                }
                              }

                  - name: cache
                    typed_config:
                      "@type": type.googleapis.com/udpa.type.v1.TypedStruct
                      type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
                      value:
                        config:
                          name: cache
                          vm_config:
                            runtime: envoy.wasm.runtime.v8
                            code:
                              local:
                                filename: /etc/envoy/main.wasm
                          configuration:
                            "@type": "type.googleapis.com/google.protobuf.StringValue"
                            value: |
                              {
                                "embedding": {
                                  "type": "cohere",
                                  "serviceName": "cohere.dns",
                                  "apiKey": ""
                                },
                                "vector": {
                                  "type": "dashvector",
                                  "serviceName": "dashvector.dns",
                                  "collectionID": "first",
                                  "serviceHost": "vrs-cn-.dashvector.cn-hangzhou.aliyuncs.com",
                                  "apiKey": "sk-"
                                },
                                "cache": {
                                  "serviceName": "",
                                  "type": ""
                                }
                              }
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
   - name: deepseek
     connect_timeout: 30s
     type: LOGICAL_DNS
     dns_lookup_family: V4_ONLY
     lb_policy: ROUND_ROBIN
     load_assignment:
       cluster_name: deepseek
       endpoints:
         - lb_endpoints:
             - endpoint:
                 address:
                   socket_address:
                     address: api.deepseek.com
                     port_value: 443
     transport_socket:
       name: envoy.transport_sockets.tls
       typed_config:
         "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
         "sni": "api.deepseek.com"

   - name: outbound|443||cohere.dns
     connect_timeout: 30s
     type: LOGICAL_DNS
     dns_lookup_family: V4_ONLY
     lb_policy: ROUND_ROBIN
     load_assignment:
       cluster_name: outbound|443||cohere.dns
       endpoints:
         - lb_endpoints:
             - endpoint:
                 address:
                   socket_address:
                     address: api.cohere.com
                     port_value: 443
     transport_socket:
       name: envoy.transport_sockets.tls
       typed_config:
         "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
         "sni": "api.cohere.com"

   - name: outbound|443||dashvector.dns
     connect_timeout: 30s
     type: LOGICAL_DNS
     dns_lookup_family: V4_ONLY
     lb_policy: ROUND_ROBIN
     load_assignment:
       cluster_name: outbound|443||dashvector.dns
       endpoints:
         - lb_endpoints:
             - endpoint:
                 address:
                   socket_address:
                     address: vrs-cn-.dashvector.cn-hangzhou.aliyuncs.com
                     port_value: 443
     transport_socket:
       name: envoy.transport_sockets.tls
       typed_config:
         "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
         "sni": "vrs-cn-.dashvector.cn-hangzhou.aliyuncs.com"

Ⅴ. Special notes for reviews

CLAassistant · 2024-12-02T10:59:36Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

ayanami-desu seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

CH3CHO · 2024-12-02T11:32:30Z

plugins/wasm-go/extensions/ai-cache/embedding/cohere.go

+
+func (c *cohereProviderInitializer) CreateConfig(json gjson.Result) {
+	cohereConfig.apiKey = json.Get("apiKey").String()
+


奇怪的空行

CH3CHO · 2024-12-02T11:32:37Z

plugins/wasm-go/extensions/ai-cache/embedding/cohere.go

+	// EmbeddingTypes string
+	// Texts          []string
+	// Images         []string
+	// Truncate       string


这些注释掉的参数可以删掉吗

CH3CHO · 2024-12-02T11:33:25Z

plugins/wasm-go/extensions/ai-cache/embedding/cohere.go

+	}
+
+	headers := [][2]string{
+		{"Authorization", fmt.Sprintf("BEARER %s", cohereConfig.apiKey)},


BEARER 应该是小写吧

是大写，按照api文档来的

CH3CHO · 2024-12-02T11:36:29Z

plugins/wasm-go/extensions/ai-cache/embedding/provider.go

 )

 type providerInitializer interface {
-	ValidateConfig(ProviderConfig) error
+	CreateConfig(json gjson.Result)


方法叫 Create，但没有返回值，很奇怪

因为这个方法是初始化全局变量。不合适的话我再改改

Support cohere embedding

7aa6e69

ayanami-desu requested review from johnlanni, WeixinX and CH3CHO as code owners December 2, 2024 10:59

CH3CHO requested changes Dec 2, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cohere embedding for ai-cache #1563

Add cohere embedding for ai-cache #1563

ayanami-desu commented Dec 2, 2024

CLAassistant commented Dec 2, 2024

CH3CHO Dec 2, 2024

CH3CHO Dec 2, 2024

CH3CHO Dec 2, 2024

ayanami-desu Dec 2, 2024

CH3CHO Dec 2, 2024

ayanami-desu Dec 2, 2024


		func (c *cohereProviderInitializer) CreateConfig(json gjson.Result) {
		cohereConfig.apiKey = json.Get("apiKey").String()

Add cohere embedding for ai-cache #1563

Are you sure you want to change the base?

Add cohere embedding for ai-cache #1563

Conversation

ayanami-desu commented Dec 2, 2024

Ⅰ. Describe what this PR did

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)?

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

CLAassistant commented Dec 2, 2024

CH3CHO Dec 2, 2024

Choose a reason for hiding this comment

CH3CHO Dec 2, 2024

Choose a reason for hiding this comment

CH3CHO Dec 2, 2024

Choose a reason for hiding this comment

ayanami-desu Dec 2, 2024

Choose a reason for hiding this comment

CH3CHO Dec 2, 2024

Choose a reason for hiding this comment

ayanami-desu Dec 2, 2024

Choose a reason for hiding this comment