Custom filter support for typeorm and graphql

[luca-nardelli]

Hey! In the past few days I spent some time thinking about a general solution for having custom user-defined filters ( related issues: #702 and #720 ), for now only at the typeorm level.
I'm opening this draft PR to ask for feedback on my solution. If this can work for you, I can probably keep working on this and transform it in a real PR.

Problem: It is currently not possible to define user-defined filters on entities. This means that, for example, if we're using postgis and our entity has a Point field (e.g. GPS coordinates) it's impossible to create a filter that returns all entities that are closer than X to a given point (radius search), since the only allowed operations are the ones defined by the sqlComparisonBuilder.

Assuming our entity has a location field which represents its position, I'd like to be able to filter on it with something like {location: {distanceFrom: {point: {lat: ..., lng: ...}, radius: ....}}}.

Proposed solution:
Extend the filtering mechanism to allow custom user defined filters that can be applied during the transformation of the Query object into the actual typeorm query.
A custom filter is basically the following interface:

export type CustomFilterResult = { sql: string; params: ObjectLiteral };

export interface CustomFilter<Entity = unknown, T = unknown> {
  apply(field: keyof Entity & string, cmp: string, val: T, alias?: string): CustomFilterResult;
}

(note that CustomFilterResult is the same as CmpSQLType https://github.com/luca-nardelli/nestjs-query/blob/custom-filters/packages/query-typeorm/src/query/sql-comparison.builder.ts#L9)

Each custom filter can be registered for a specific entity class, field and operation name, and is then invoked by the where builder when it's processing the filter object. Here, "operation name" basically means the individual operation of the filter (same as cmpType in the where builder)

{ numberType: { gt: 10 } }

In the above example, numberType is the field, and gt is the operation name. Filter registration happens through a new CustomFilterRegistry class that basically holds all custom filters for all entities in a Map. Storing them flat allows to easily fetch all custom filters when traversing relations in the where builder.

I started writing some code with the above idea in mind and I've managed to cobble together something that allowed me to write and pass some tests ( added here https://github.com/luca-nardelli/nestjs-query/blob/custom-filters/packages/query-typeorm/__tests__/query/where.builder.spec.ts).

More specifically, this is how it looks like at the code level for 2 different custom filters: an extension for numeric types where I want to get values that are multiples of a user provided value, and an example with a PostGIS query.

  const customFilterRegistry = new CustomFilterRegistry();
  customFilterRegistry.setFilter<TestEntity>(TestEntity, 'numberType', 'isMultipleOf', {
    apply(field, cmp, val: number, alias): CustomFilterResult {
      alias = alias ? alias : '';
      const pname = `param${randomString()}`;
      return {
        sql: `("${alias}"."${field}" % :${pname}) == 0`,
        params: { [pname]: val },
      };
    },
  });

  customFilterRegistry.setFilter<TestEntity>(TestEntity, 'fakePointType', 'distanceFrom', {
    apply(field, cmp, val: { point: { lat: number; lng: number }; radius: number }, alias): CustomFilterResult {
      alias = alias ? alias : '';
      const plat = `param${randomString()}`;
      const plng = `param${randomString()}`;
      const prad = `param${randomString()}`;
      return {
        sql: `ST_Distance("${alias}"."${field}", ST_MakePoint(:${plat},:${plng})) <= :${prad}`,
        params: { [plat]: val.point.lat, [plng]: val.point.lng, [prad]: val.radius },
      };
    },
  });

  const expectSQLSnapshot = (filter: Filter<TestEntity>): void => {
    const selectQueryBuilder = createWhereBuilder().build(
      getQueryBuilder(),
      filter,
      {},
      TestEntity, // <--- New parameter
      customFilterRegistry, // <--- New parameter
      'TestEntity',
    );
    const [sql, params] = selectQueryBuilder.getQueryAndParameters();
    expect(sql).toMatchSnapshot();
    expect(params).toMatchSnapshot();
  };

  it('should accept custom filters alongside regular filters', (): void => {
    expectSQLSnapshot({
      numberType: { gte: 1, lte: 10, isMultipleOf: 5 },
      fakePointType: { distanceFrom: { point: { lat: 45.3, lng: 9.5 }, radius: 50000 } },
    } as any);
  });

The custom filter registry here is a standard class and I am registering filters manually, but at a higher level we could register custom filters using decorators + DI. Apart from this, the most notable changes I had to make so far are the following:

• Add an explicit reference to the class currently being processed to the build() method of the where builder and to all of its inner methods.
• Add an explicit reference to the custom filter registry to the build() method of the where builder and to all of its inner methods.
• Rewrite the getRelationNamesRecursive method to not only extract relation names, but also the target class, in order to use that as index for the custom filter registry.

At the wherebuilder level, the main change happened in the withFilterComparison, where now I try to call a custom filter and I use the sqlcomparisonbuilder only as a fallback.

const sqlComparisons = opts.map((cmpType) => {
          const customFilter = customFilterRegistry?.getFilter(klass, field, cmpType);
          // If we have a registered customfilter for this cmpType, this has priority over the standard sqlComparisonBuilder
          if (customFilter) {
            return customFilter.apply(field, cmpType, cmp[cmpType], alias);
          }
          return this.sqlComparisonBuilder.build(
            field,
            cmpType,
            cmp[cmpType] as EntityComparisonField<Entity, T>,
            alias,
          );
        });
        sqlComparisons.map(({ sql, params }) => qb.orWhere(sql, params));

---

There are obviously a lot of things to do here (as an example, there's some type fixing that needs to happen because right now we are assuming that the only possible values in the Filter* types are the common ones), but for now could you @doug-martin (or someone else) have a look and tell me if you think this is an acceptable approach?

Thanks!

[coveralls]

Pull Request Test Coverage Report for Build 1326807712

• 67 of 68 (98.53%) changed or added relevant lines in 6 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.03%) to 96.374%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
packages/query-typeorm/src/query/custom-filter.registry.ts	11	12	91.67%

Totals
Change from base Build 1326721410:	-0.03%
Covered Lines:	4999
Relevant Lines:	5089

---

💛 - Coveralls

[doug-martin]

@luca-nardelli thank you for looking into this!

When I have investigated this before my hangup (I may be complicating this more than it needs to be) is approaching it generically. More specifically allowing custom operators across types.

Putting myself in an end-users shoe's, I would want to specify a generic operation (e.g. distanceFrom) so it could be used across multiple DTOs. Your solution eliminates a lot of complications from our (nestjs-query) pov but does it make our adopter's life easier? Maybe it's better to get something out there to handle this and see where it goes but our end-users are always number one in my mind.

I'd be interested in your thoughts/approach in making this more generic to allow specifying an operation instead of a (field, operation) tuple.

Great job on the PR!

[luca-nardelli]

I agree that defining common custom filters per (type, operation) rather than (field, operation) makes much more sense for standard scenarios (e.g. the distanceFrom case or getting multiples of a number).
However, thinking about some situations I had to face in the past, I would still like to have (field, operation) custom filters as well to handle some edge cases and also to define filters that do not map to real fields of the entity.

Simple example: A user has N projects, and each project has N tasks that can either be pending or complete. I would like to get all projects that have more than M pending tasks, in order to do that I could define a new custom filter on a non existing field of the project entity called pendingTasksCount with operation gte (technically all number type operations could be supported, but that's up to me to implement them) that allows me to implement that filter as a subquery added in the query builder WHERE (SELECT COUNT(*) ... ) > :val
I know that, technically, I could create a real computed field pendingTasksCount in the entity using db triggers/typeorm hooks, but for simple scenarios or when you need to get a POC out maybe defining a custom filter with the subquery is easier?

Harder example: A user belongs to N groups, projects belong to 1 group, groups can be nested inside of each other (parent-child), where users at the parent level can see the projects of all child groups, and where the task is to filter all projects based on the requesting user. if you store the parent-child relationship with a materialized path approach using e.g. JSON arrays, given a parent group ID you can get all its child groups, and then you can use that to filter projects. In order to implement this at the project level, I would probably define a custom filter isAccessibleToUser: {eq: 'id'} (ideally I'd only like isAccessibleToUser: 'id' without the eq part as I wouldn't plan on supporting more operations there) and then build the subquery at the where builder level. This filter would probably never be exposed to the client though, it would only be used in an Authorizer.

Technically though, you could argue that these filtering operations could be defined on the relation type itself 😆

All this to say that I'd still like the flexibility of manually defining filters with the (Entity, field, operation) approach (also, maybe I'd want to expose a distanceFrom filter only on one field of one entity, for example), but having to do that manually for all fields can become quite a bother for situations where you have many entities. I'll try tinkering with the code to see what's possible!

[luca-nardelli]

Idea: What do you say about explicitly defining filter classes instead of generating them? It's definitely a bit more boilerplate, but it could allow the right grade of flexibility, and potentially it could also give a bit more control to end users (maybe, if you are fetching books, you only want to filter on autor names and not author birth dates, whereas if you fetch authors directly you want to filter on both)

I made a POC some time ago (as a learning project) that was something like this:

@EntityFilter()
export class BookFilter {
  @EntityFilterProperty()
  name?: StringFilter; // StringFilter is another class with {eq: string, noteq: string, in: string[], ....}

  @EntityFilterProperty()
  numPages?: NumericFilter;

  @EntityFilterProperty()
  publishedAt?: DateFilter;

  @EntityFilterProperty({type: () => AuthorFilter})
  author?: AuthorFilter;
}

I would still have a filter registry, but registration only linked the filter class with the filter handler.

registry.registerFilter(StringFilter, handlerString);
registry.registerFilter(NumericFilter, handlerNumeric);
registry.registerFilter(DateFilter, handlerDate);

Processing of the filter object then was similar to what I did now, with the difference that I was using the filter type to invoke the right handlers.

I think this could allow both generic type filters, that are associated to the type of the field, but also highly specific filters, provided that you registered them only on the right class. The drawback, however, is that the filter class needs to be manually defined, but since it's basically just fields and annotations maybe it's acceptable?

[luca-nardelli]

Hey @doug-martin , sorry for the radio silence but I've been swamped by things and while I was able to think about this in the past weeks, I just found some time to go over the code.

Let's ignore for now my previous comment (about defining the filter classes) as it is mostly unrelated. I gave some more thoughts on the (operation) type filter as opposed to the (class, field, operation) type filter I had first implemented, and I might have found a way to allow both of them (and potentially some other categories of filters too).

Nothing changed regarding how I apply filters in the where builder, but now filters can be registered in 2 ways: global (i.e. (operation)) mode, or specific mode (where we need to specify the class and field as well).

Since we could potentially have a conflict between an (operation) filter and a (class, field, operation) filter when resolving the filter for a specific (class, field, operation) tuple (in the where builder), I simply decided to give higher priority to the most specific filter, i.e. if we define the same operation both with a global filter and a specific filter, the specific one wins.

Example from the updated tests:

// Test for (operation) filter registration (this is valid for all fields of all entities)
  customFilterRegistry.setFilter('isMultipleOf', {
    apply(field, cmp, val: number, alias): CustomFilterResult {
      alias = alias ? alias : '';
      const pname = `param${randomString()}`;
      return {
        sql: `("${alias}"."${field}" % :${pname}) == 0`,
        params: { [pname]: val },
      };
    },
  });
  // Test for (class, field, operation) filter overriding the previous operation filter on a specific field
  customFilterRegistry.setFilter<TestEntity>(
    'isMultipleOf',
    {
      apply(field, cmp, val: number, alias): CustomFilterResult {
        alias = alias ? alias : '';
        const pname = `param${randomString()}`;
        return {
          sql: `(EXTRACT(EPOCH FROM "${alias}"."${field}") / 3600 / 24) % :${pname}) == 0`,
          params: { [pname]: val },
        };
      },
    },
    { klass: TestEntity, field: 'dateType' },
  );
  // Test for (class, field, operation) filter on a virtual property 'fakePointType' that does not really exist on the entity
  customFilterRegistry.setFilter<TestEntity>(
    'distanceFrom',
    {
      apply(field, cmp, val: { point: { lat: number; lng: number }; radius: number }, alias): CustomFilterResult {
        alias = alias ? alias : '';
        const plat = `param${randomString()}`;
        const plng = `param${randomString()}`;
        const prad = `param${randomString()}`;
        return {
          sql: `ST_Distance("${alias}"."${field}", ST_MakePoint(:${plat},:${plng})) <= :${prad}`,
          params: { [plat]: val.point.lat, [plng]: val.point.lng, [prad]: val.radius },
        };
      },
    },
    { klass: TestEntity, field: 'fakePointType' },
  );

it('should accept custom filters alongside regular filters', (): void => {
    expectSQLSnapshot({
      // This has the global isMultipleOf filter
      numberType: { gte: 1, lte: 10, isMultipleOf: 5 },
      // Here, the isMultipleOf filter was overridden for dateType only
      dateType: { isMultipleOf: 3 },
      // This is a more complex filter involving geospatial queries
      fakePointType: { distanceFrom: { point: { lat: 45.3, lng: 9.5 }, radius: 50000 } },
    } as any);
  });

This setup could potentially allow us to define new filter types with intermediate specificity (e.g. a filter that works on (type, operation) rather than just operation, so that, for example we can define isMultipleOf for all numbers and all dates differently. This would be more specific than the global filter, but less specific than the field filter, and so on...

Let me know your thoughts!

[luca-nardelli]

Hey, I was able to put some further work on this and I started getting close to an external API that other devs could use in their application code.

Example:

@Entity()
// This binds the filter to ALL fields of the entity
@TypeormQueryFilter<TestEntity>({
  filter: IsMultipleOfCustomFilter,
})
// This binds the filter to only the specific fields mentioned in the fields array
@TypeormQueryFilter<TestEntity>({
  filter: IsMultipleOfDateCustomFilter,
  fields: ['dateType'],
})
export class TestEntity {
  @PrimaryColumn({ name: 'test_entity_pk' })
  testEntityPk!: string;
}

These decorators only set metadata. During NesjJsQueryTypeormModule onModuleInit I fetch that metadata and register the filters in the customFilterRegistry (which is not much different than before).

This is how a custom filter looks like:

@Injectable()
export class IsMultipleOfCustomFilter implements CustomFilter<IsMultipleOfOpType> {
  readonly operations: string[] = ['isMultipleOf'];

  apply(field: string, cmp: string, val: unknown, alias?: string): CustomFilterResult {
    alias = alias ? alias : '';
    const pname = `param${randomString()}`;
    return {
      sql: `(${alias}.${field} % :${pname}) == 0`,
      params: { [pname]: val },
    };
  }
}

I added further tests at the filter-query-builder level and at the typeorm-query-service level (had to force the timezone of the tests to UTC otherwise some tests would fail) and everything seems to check out so far.

Note that the decorators are applied at the entity level, and not at the DTO level. This is because we are still not in GraphQL-land here, but we are in Typeorm-land, that's why I wend this route.

I think I'm going in the right direction and I like this decorator based API, but I'd love to hear your thoughts on this!

Next steps:

• See how I can extract an entity field type (based on the typeorm column type I guess) and maybe allow registering custom filter on types
• See how I can register global type-based custom filters (i.e. not tied to any particular entity). Probably this can be done using another decorator, this time at the filter level
• Understand how to expose all of this at the graphql level, with the Filter interface and the graphql filter creation for the schema

[luca-nardelli]

Hey @doug-martin , I got sidetracked and didn't have time to look into this until recently.
I've simplified the whole reasoning and now I only defined 2 filter types : type-based (which works globally only based on the field type) and "virtual", which is basically a completely custom filter defined on a (DTO, field, operation) tuple. More background on this is explained in the documentation pages I added (see below)

Basically the process happens in 2 steps:

1. Define your filter at the persistence layer and register it in NestjsQueryTypeOrmModule.forFeature(), this already enables the filter but we need to expose it in the GraphQL layer as well.
2. Register your filter at the GraphQL layer, automatically updating/patching all relevant graphql types we have in the schema.

This is similar to how things like enums are registered.

As for how the external API looks like, you can have a quick look at the 2 documentation pages I wrote:

• For typeorm: https://github.com/luca-nardelli/nestjs-query/tree/8733a1d8b046e61d2912cff515dfe66a9dc62c81/documentation/docs/persistence/typeorm/custom-filters.md
• For graphql: https://github.com/luca-nardelli/nestjs-query/tree/8733a1d8b046e61d2912cff515dfe66a9dc62c81/documentation/docs/graphql/custom-filters.md

I believe we are getting close to your initial idea of "allowing custom operators across types.", but definitely let me know your thoughts on this! I could probably also use some help in naming the public API interfaces/entities, as right now I didn't spend too much time thinking about naming. You can see them in use in the doc pages or in the https://github.com/luca-nardelli/nestjs-query/tree/8733a1d8b046e61d2912cff515dfe66a9dc62c81/examples/typeorm example

[hedwiggggg]

Just wanted to mention that there is an actively developed fork: https://github.com/tripss/nestjs-query Maybe you can create this pull request there? @luca-nardelli

Because @doug-martin probably abandoned this project (see #1538)