Skip to content

Commit

Permalink
Show subscriptions to event types in Backstage (#24)
Browse files Browse the repository at this point in the history 
* Build subscribers list

Signed-off-by: Ali Ok <[email protected]>

* Embed the consumedBy info in eventTypes

Signed-off-by: Ali Ok <[email protected]>

* Make the tests working

Signed-off-by: Ali Ok <[email protected]>

* Get dynamic client from the context

Signed-off-by: Ali Ok <[email protected]>

* go mod tidy && go mod vendor

Signed-off-by: Ali Ok <[email protected]>

* Link event mesh plugin locally

Signed-off-by: Ali Ok <[email protected]>

* Store the consumer relation in a custom annotation

Signed-off-by: Ali Ok <[email protected]>

* Process the custom annotation to emit relations

Signed-off-by: Ali Ok <[email protected]>

* Does not work - extended the ApiEntity and added a root level field

Signed-off-by: Ali Ok <[email protected]>

* Put the consumer information in the metadata as is

Signed-off-by: Ali Ok <[email protected]>

* Use Backstage Kube id annotation to fetch the consumer

Signed-off-by: Ali Ok <[email protected]>

* Improve code, add logging

Signed-off-by: Ali Ok <[email protected]>

* Add some test cases in the backend code

Signed-off-by: Ali Ok <[email protected]>

* Add more test cases in the backend code

Signed-off-by: Ali Ok <[email protected]>

* Use UnsafeGuessKindToResource

Signed-off-by: Ali Ok <[email protected]>

* Add some docs

Signed-off-by: Ali Ok <[email protected]>

* Rename `etNameMap` to `etByNamespacedName`

Signed-off-by: Ali Ok <[email protected]>

* Simplify trigger processing

Signed-off-by: Ali Ok <[email protected]>

* Add code comments

Signed-off-by: Ali Ok <[email protected]>

* Update architecture diagrams

Signed-off-by: Ali Ok <[email protected]>

* Address comments

Signed-off-by: Ali Ok <[email protected]>

* Fix provider unit tests

Signed-off-by: Ali Ok <[email protected]>

* Created separate issues for TODOs

Signed-off-by: Ali Ok <[email protected]>

* Tests for the processor

Signed-off-by: Ali Ok <[email protected]>

* Remove trailing whitespace

Signed-off-by: Ali Ok <[email protected]>

---------

Signed-off-by: Ali Ok <[email protected]>
  • Loading branch information
@aliok
aliok authored Feb 7, 2024
1 parent 413359b commit 408beed
Show file tree
Hide file tree
Showing 94 changed files with 5,827 additions and 91 deletions.
136 changes: 118 additions & 18 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,30 +14,130 @@ See [Event Mesh plugin README file](./backstage/plugins/knative-event-mesh-backe

The architecture of the plugin is as follows:
```
Kubernetes Backstage
┌────────────────────┐ ┌─────────────────────┐
│ │ │ │
│ ┌───────────────┐ │ │ Plugin │
│ │ │ │ │ ┌─────────────────┐ │
│ │ Backend ◄──┼────┐ │ │ │ │
│ │ │ │ │ │ │ ┌─────────────┐ │ │
│ └───────┬───────┘ │ │ │ │ │ │ │ │
│ │ │ └──────┼─┼─┤ Provider │ │ │
│ │ │ │ │ │ │ │ │
│ │ │ │ │ └─────────────┘ │ │
│ ┌───────▼───────┐ │ │ │ │ │
│ │ │ │ │ └─────────────────┘ │
│ │ API Server │ │ │ │
│ │ │ │ └─────────────────────┘
│ └───────────────┘ │
│ │
└────────────────────┘
Kubernetes Backstage
┌────────────────────┐ ┌───────────────────────────────────────────────┐
│ │ │ Plugin │
│ ┌───────────────┐ │ │ ┌─────────────────┐ ┌───────────────┐ │
│ │ │ │ │ │ │ │ │ │
│ │ Backend ◄──┼────┐ │ │ ┌─────────────┐ │ │ │ │
│ │ │ │ │ │ │ │ │ │ │ │ │
│ └───────┬───────┘ │ └──────┼─┼─┤ Provider ├─┼────────► │ │
│ │ │ │ │ │ │ │ │ │ │
│ │ │ │ │ └─────────────┘ │ │ │ │
│ │ │ │ │ │ │ │ │
│ ┌───────▼───────┐ │ │ │ │ │ │ │
│ │ │ │ │ │ ┌────────┼────────┤ Catalog │ │
│ │ API Server │ │ │ │ │ │ │ │ │
│ │ │ │ │ │ │ │ │ │ │
│ └───────────────┘ │ │ │ ┌──────▼──────┐ │ │ │ │
│ │ │ │ │ │ │ │ │ │
└────────────────────┘ │ │ │ Processor ├─┼────────► │ │
│ │ │ │ │ │ │ │
│ │ └─────────────┘ │ │ │ │
│ │ │ │ │ │
│ └─────────────────┘ └───────────────┘ │
└───────────────────────────────────────────────┘
```

The plugin use providers (and possibly other mechanisms) to communicate with a special backend-for-frontend.

This backend talks to the Kubernetes API server to get information about the resources in the cluster.

```mermaid
---
title: Overall
---
flowchart TD
Start --> FetchBrokers
FetchBrokers --> ProcessBrokers
ProcessBrokers --> FetchEventTypes
FetchEventTypes --> ProcessEventTypes
ProcessEventTypes --> FetchTriggers
FetchTriggers --> ProcessTriggers
```

## Processing the brokers

```mermaid
---
title: ProcessBrokers
---
flowchart LR
GetNextBroker --> CreateDTO
```

## Processing the event types

```mermaid
---
title: ProcessEventTypes
---
flowchart TD
GetEventType[Get next event type]
CheckRef{spec.ref exists?}
RefIsABrokerInTheBrokerMap{ref is a broker in the previously <br> built broker map?}
RegisterEventType[Add event type to broker DTO's `providedEventTypes` list]
DontRegisterEventType[Don't relate the event type to any broker]
Done[Done]
GetEventType --> CheckRef
CheckRef --> |Yes| RefIsABrokerInTheBrokerMap
RefIsABrokerInTheBrokerMap --> |Yes| RegisterEventType
CheckRef --> |No| DontRegisterEventType
RefIsABrokerInTheBrokerMap --> |No| DontRegisterEventType
RegisterEventType --> Done
DontRegisterEventType --> Done
```

## Processing the triggers

```mermaid
---
title: ProcessTriggers
---
flowchart TD
GetTrigger[Get next trigger]
CheckSubscriberRef{spec.subscriber.ref <br> exists?}
FetchSubscriberRef[Fetch subscriber resource]
CheckSubscriberLabel{Subscriber has the <br> Backstage label}
CheckEventType{Trigger has an <br> event type}
RegisterSingleRelation[Register `ConsumedBy` relation <br> for eventType and subscriber]
RegisterRelation[Register `ConsumedBy` relation <br> for eventType and subscriber]
Ignore[Ignore trigger]
Done[Done]
GetTrigger --> CheckSubscriberRef
CheckSubscriberRef --> |Yes| FetchSubscriberRef
FetchSubscriberRef --> CheckSubscriberLabel
CheckSubscriberLabel --> |Yes| CheckEventType
CheckEventType --> |Yes| RegisterSingleRelation
CheckEventType --> |No| FetchAllEventTypesForBroker
FetchAllEventTypesForBroker --> ForEachEventType --> RegisterRelation
RegisterSingleRelation --> Done
RegisterRelation --> Done
CheckSubscriberLabel --> |No| Ignore
CheckSubscriberRef --> |No| Ignore
Ignore --> Done
CheckSubscriberRef -.- CheckSubscriberRefNote["We can't collect subscriber information using the URL. <br> So, let's simply check the subsciber ref."]
CheckSubscriberLabel -.- CheckSubscriberLabelNote["The target is to show what resource is using what event types. <br> However, Backstage will only show the resource if it has a special label. <br> So, if that label is missing, simply ignore the subscriber."]
CheckEventType -.- CheckEventTypeNote["If the trigger has an event type filter, <br> that means the subscriber is subscribed to that event. <br> If not, the subscriber is subscribed to all events from this trigger. <br> Please note that we ignore other filtering mechanisms such as 'source'."]
CheckSubscriberRefNote:::note
CheckSubscriberLabelNote:::note
CheckEventTypeNote:::note
classDef note fill:yellow
```

#### Running the backend

The backend is a Go project that runs in a Kubernetes cluster.
Expand Down
12 changes: 12 additions & 0 deletions backends/config/100-eventmesh/100-clusterrole.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ rules:
resources:
- brokers
- eventtypes
- triggers
verbs:
- get
- list
Expand All @@ -43,3 +44,14 @@ rules:
- delete
- patch
- watch


# permissions to get subscribers for triggers
# as subscribers can be any resource, we need to give access to all resources
# we fetch subscribers one by one, we only need `get` verb
- apiGroups:
- "*"
resources:
- "*"
verbs:
- get
2 changes: 2 additions & 0 deletions backends/pkg/reconciler/eventmesh/annotations.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ var ExcludedAnnotations = map[string]struct{}{
"kubectl.kubernetes.io/last-applied-configuration": {},
}

// FilterAnnotations filters out annotations that are not interesting to provide to the Backstage plugin.
// Specifically, it filters out the annotations in ExcludedAnnotations.
func FilterAnnotations(annotations map[string]string) map[string]string {
if annotations == nil {
return nil
Expand Down
31 changes: 23 additions & 8 deletions backends/pkg/reconciler/eventmesh/broker.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,28 +4,43 @@ import (
eventingv1 "knative.dev/eventing/pkg/apis/eventing/v1"
)

// Broker is a simplified representation of a Knative Eventing Broker that is easier to consume by the Backstage plugin.
type Broker struct {
Name string `json:"name"`
Namespace string `json:"namespace"`
UID string `json:"uid"`
Labels map[string]string `json:"labels,omitempty"`
// Namespace of the broker
Namespace string `json:"namespace"`

// Name of the broker
Name string `json:"name"`

// UID of the broker
UID string `json:"uid"`

// Labels of the broker. These are passed as is.
Labels map[string]string `json:"labels,omitempty"`

// Annotations of the broker. These are passed as is, except that are filtered out by the FilterAnnotations function.
Annotations map[string]string `json:"annotations,omitempty"`
//

// ProvidedEventTypes is a list of event types that the broker provides.
// This is a list of strings, where each string is a "<namespace>/<name>" of the event type.
ProvidedEventTypes []string `json:"providedEventTypes,omitempty"`
}

func (b Broker) GetNameAndNamespace() string {
return NameAndNamespace(b.Namespace, b.Name)
// GetNamespacedName returns the name and namespace of the broker in the format "<namespace>/<name>"
func (b Broker) GetNamespacedName() string {
return NamespacedName(b.Namespace, b.Name)
}

// convertBroker converts a Knative Eventing Broker to a simplified representation that is easier to consume by the Backstage plugin.
// see Broker.
func convertBroker(br *eventingv1.Broker) Broker {
return Broker{
Name: br.Name,
Namespace: br.Namespace,
UID: string(br.UID),
Labels: br.Labels,
Annotations: FilterAnnotations(br.Annotations),
// to be filled later
// this field will be populated later on, when we have the list of event types
ProvidedEventTypes: []string{},
}
}
5 changes: 4 additions & 1 deletion backends/pkg/reconciler/eventmesh/controller.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import (
eventtypereconciler "knative.dev/eventing/pkg/client/injection/reconciler/eventing/v1beta2/eventtype"

brokerinformer "knative.dev/eventing/pkg/client/injection/informers/eventing/v1/broker"
triggerinformer "knative.dev/eventing/pkg/client/injection/informers/eventing/v1/trigger"
eventtypeinformer "knative.dev/eventing/pkg/client/injection/informers/eventing/v1beta2/eventtype"

eventinglistersv1 "knative.dev/eventing/pkg/client/listers/eventing/v1"
Expand All @@ -22,6 +23,7 @@ import (
type Listers struct {
EventTypeLister eventinglistersv1beta2.EventTypeLister
BrokerLister eventinglistersv1.BrokerLister
TriggerLister eventinglistersv1.TriggerLister
}

func NewController(ctx context.Context) *controller.Impl {
Expand All @@ -40,6 +42,7 @@ func NewController(ctx context.Context) *controller.Impl {
listers := Listers{
EventTypeLister: eventtypeinformer.Get(ctx).Lister(),
BrokerLister: brokerinformer.Get(ctx).Lister(),
TriggerLister: triggerinformer.Get(ctx).Lister(),
}

go startWebServer(ctx, listers)
Expand All @@ -56,7 +59,7 @@ func startWebServer(ctx context.Context, listers Listers) {
r := mux.NewRouter()
r.Use(commonMiddleware)

r.HandleFunc("/", EventMeshHandler(ctx, listers)).Methods("GET")
r.HandleFunc("/", HttpHandler(ctx, listers)).Methods("GET")
http.Handle("/", r)

log.Fatal(http.ListenAndServe(":8080", r))
Expand Down
1 change: 1 addition & 0 deletions backends/pkg/reconciler/eventmesh/eventmesh.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (
eventingv1beta2 "knative.dev/eventing/pkg/apis/eventing/v1beta2"
)

// Reconciler is a stub reconciler for getting the informers injected by sharedmain.
type Reconciler struct {
}

Expand Down
40 changes: 28 additions & 12 deletions backends/pkg/reconciler/eventmesh/eventtype.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,23 +4,37 @@ import (
"knative.dev/eventing/pkg/apis/eventing/v1beta2"
)

// EventType is a simplified representation of a Knative Eventing EventType that is easier to consume by the Backstage plugin.
type EventType struct {
Name string `json:"name"`
Namespace string `json:"namespace"`
Type string `json:"type"`
UID string `json:"uid"`
Description string `json:"description,omitempty"`
SchemaData string `json:"schemaData,omitempty"`
SchemaURL string `json:"schemaURL,omitempty"`
Labels map[string]string `json:"labels,omitempty"`
Namespace string `json:"namespace"`
Name string `json:"name"`
Type string `json:"type"`
UID string `json:"uid"`
Description string `json:"description,omitempty"`
SchemaData string `json:"schemaData,omitempty"`
SchemaURL string `json:"schemaURL,omitempty"`
// Labels of the event type. These are passed as is.
Labels map[string]string `json:"labels,omitempty"`
// Annotations of the event type. These are passed as is, except that are filtered out by the FilterAnnotations function.
Annotations map[string]string `json:"annotations,omitempty"`
Reference string `json:"reference,omitempty"`
// Reference is the ET's reference to a resource like a broker or a channel. It is in the format "<namespace>/<name>".
Reference string `json:"reference,omitempty"`
// ConsumedBy is a <namespace/name> list of the consumers of the event type.
ConsumedBy []string `json:"consumedBy,omitempty"`
}

func (et EventType) NameAndNamespace() string {
return NameAndNamespace(et.Namespace, et.Name)
// NamespacedName returns the name and namespace of the event type in the format "<namespace>/<name>"
func (et EventType) NamespacedName() string {
return NamespacedName(et.Namespace, et.Name)
}

// NamespacedType returns the type and namespace of the event type in the format "<namespace>/<type>"
func (et EventType) NamespacedType() string {
return NamespacedName(et.Namespace, et.Type)
}

// convertEventType converts a Knative Eventing EventType to a simplified representation that is easier to consume by the Backstage plugin.
// see EventType.
func convertEventType(et *v1beta2.EventType) EventType {
return EventType{
Name: et.Name,
Expand All @@ -32,6 +46,8 @@ func convertEventType(et *v1beta2.EventType) EventType {
SchemaURL: et.Spec.Schema.String(),
Labels: et.Labels,
Annotations: FilterAnnotations(et.Annotations),
Reference: RefNameAndNamespace(et.Spec.Reference),
Reference: NamespacedRefName(et.Spec.Reference),
// this field will be populated later on, when we have process the triggers
ConsumedBy: make([]string, 0),
}
}
Loading

0 comments on commit 408beed

Please sign in to comment.