Newbie needs some help 🙏

[cocoderus]

I'm building a searchable tree data grid in avalonia using reactive ui + dynamic data. So far, I came up with the following solution:

First, I created a LoadCommand responsible for loading and refreshing the data:

LoadCommand = ReactiveCommand.CreateFromTask(Load);

The Load method handles updating properties like IsLoading and setting the SourceCache:

private async Task Load()
{
    await Task.Run(() =>
    {
        IsLoading = true;
        
        SearchQuery = "";
        Error = null;
        
        _sourceCache.AddOrUpdate(LoadData());
        
        IsLoading = false;
    });
}

Then I built a filter and bound it to the source like so:

var filter = this.WhenAnyValue(x => x.SearchQuery)
            .Throttle(TimeSpan.FromMilliseconds(500))
            .DistinctUntilChanged()
            .Select(Filter);
        
 var cleanUp = _sourceCache.Connect()
      .Filter(filter)
      .ObserveOn(RxApp.MainThreadScheduler)
      .TransformToTree(x => x.ParentOrganizationId)
      .Transform(node => new OrganizationViewModel(node))
      .SortAndBind(out _source, SortExpressionComparer<OrganizationViewModel>.Ascending(x => x.OrganizationName))
      .DisposeMany()
      .Subscribe();

And when view is activated, I execute the LoadCommand to load the data:

this.WhenActivated(d =>
{
    LoadCommand.Execute()
        .Subscribe()
        .DisposeWith(d);
    cleanUp.DisposeWith(d);
});

This solution works ok, but there are some problems I'd like to resolve.

1. Filtering the source before transforming it into the tree structure is taking about 4 seconds for 65k records. Is this normal, or there is a more efficient method?
2. I need to display a splash screen while filtering is in progress by setting Isloading = true, and hide it once the filtering completes. I attempted the following approach:

var cleanUp = _sourceCache.Connect()
            .Filter(filter)
            .ObserveOn(RxApp.MainThreadScheduler)
            .Do(x => IsLoading = true)
            .ObserveOn(RxApp.TaskpoolScheduler)
            .ObserveOn(RxApp.MainThreadScheduler)
            .TransformToTree(x => x.ParentOrganizationId)
            .Transform(node => new OrganizationViewModel(node))
            .SortAndBind(out _source, SortExpressionComparer<OrganizationViewModel>.Ascending(x => x.OrganizationName))
            .DisposeMany()
            .Do(x => IsLoading = false)
            .Subscribe();

However, the splash screen is hidden before the LoadCommand finishes. What’s the correct approach to ensure the splash screen is displayed throughout the entire filtering process?

[JakenVeina]

Filtering the source before transforming it into the tree structure is taking about 4 seconds for 65k records. Is this normal, or there is a more efficient method?

It's taking 4 seconds for the entire load process, or just for the filtering? If this includes processing for the View layer, 4 seconds for 65k does not sound crazy to me. UI layers are generally VERY slow, and WPF and all of its cousins are no exception. Even if you're not actually drawing UI for all 65k elements (which you're absolutely not doing in 4 seconds), you're still probably binding all 65k elements up to the UI, which means the UI is creating objects for them in the background, since I don't see any virtualization here. Although, to be fair, I'm very unfamiliar with the .TransformToTree() operator.

I.E. if 4 seconds is what you're observing with your eyeballs on the screen, that's probably not something you're doing wrong. If that's something you're measuring across some sub-portion of the RX stream, particularly if it's not touching the UI, there probably is something you're doing wrong.

What’s the correct approach to ensure the splash screen is displayed throughout the entire filtering process?

I'm confused based on just the context being given here. What I see that immediately looks like the problem is that you have IsLoading = false in two different spots. The one within Load() will definitely be running before the .BindAndSort() operator finishes publishing items to the UI layer. However, your usage of IsLoading is also inconsistent. When setting it within the main RX stream, you seem to be taking care to only change IsLoading from the UI thread, but within Load() you're doing it from a background thread. Are there actually multiple different IsLoading properties in play here? Or is there a separate mechanism not shown that marshals IsLoading changes to the UI thread, making the marshaling in the RX stream redundant?

On a more general note, my approach to something like this would look pretty similar to what you have, but a little more RX-ified...

_loadCommand = ReactiveCommand.Create(...);
_itemsSource = new SourceCache<...>(...);
_items = new ObservableCollectionExtended<...>();

this.WhenActivated(disposal =>
{
    _loadCommand
        .Do(_ =>
        {
            SearchCriteria = null;
            Error = null;
        })
        .ObserveOn(RxApp.ThreadpoolScheduler)
        .Select(_ => Observable.FromAsync(GetItemsAsync))
        .Switch()
        .Subscribe(items =>
        {
            using var suspension = _itemsSource.SuspendNotifications();
            _itemsSource.Clear();
            _itemsSource.AddOrUpdate(items);
        })
        .DisposeWith(disposal);

    _itemsSource
        .Connect()
        .ObserveOn(RxApp.MainThreadScheduler)
        .Do(_ => IsLoading = true)
        .ObserveOn(RxApp.ThreadpoolScheduler)
        .Filter(this.WhenAnyValue(x => x.SearchCriteria)
            .Throttle(TimeSpan.FromMilliseconds(500))
            .DistinctUntilChanged()
            .Select(Filter))
        .TransformToTree(x => x.ParentOrganizationId)
        .Transform(node => new OrganizationViewModel(node))
        .ObserveOn(RxApp.MainThreadScheduler)
        .DisposeMany()
        .SortAndBind(_items, SortExpressionComparer<OrganizationViewModel>.Ascending(x => x.OrganizationName))
        .Subscribe(_ => IsLoading = true)
        .DisposeWith(disposal);
});

[cocoderus]

I.E. if 4 seconds is what you're observing with your eyeballs on the screen, that's probably not something you're doing wrong. If that's something you're measuring across some sub-portion of the RX stream, particularly if it's not touching the UI, there probably is something you're doing wrong.

Yes, it's the process I observed on the screen, from when I type in the textbox until the view updates. I guess it just how things work and nothing can be done here.

What I see that immediately looks like the problem is that you have IsLoading = false in two different spots. The one within Load() will definitely be running before the .BindAndSort() operator finishes publishing items to the UI layer.

You are right. This is what caused this behavior.

Are there actually multiple different IsLoading properties in play here? Or is there a separate mechanism not shown that marshals IsLoading changes to the UI thread, making the #marshaling in the RX stream redundant?

No, there is only one IsLoading and it looks like this:

private bool _isLoading;
public bool IsLoading
{
    get => _isLoading;
    set => this.RaiseAndSetIfChanged(ref _isLoading, value);
}

This property is used to notify the view that a process is running, and a splash screen should be displayed. The LoadCommand refreshes the data in _sourceCache when the button is clicked and when the view is activated for the first time. I set IsLoading inside Load to start loading data and show the splash screen as soon as the view is activated. Within the RX stream, I set IsLoading to display the splash screen when the filtering process begins and hide it once the process is completed.

Thank you for the example you provided. After experimenting with it, I was able to achieve the desired behavior. My ViewModel now looks like this:

LoadCommand = ReactiveCommand.Create(() => { });

this.WhenActivated(d =>
{
    LoadCommand
        .Do(_ =>
        {
            SearchQuery = null;
            Error = null; 
            IsLoading = true;
        })
        .ObserveOn(RxApp.TaskpoolScheduler)
        .Select(_ => LoadData())
        .Subscribe(list =>
        {
            using var suspension = _sourceCache.SuspendNotifications();
            _sourceCache.Clear();
            _sourceCache.AddOrUpdate(list);
        })
        .DisposeWith(d);

    _sourceCache
        .Connect()
        .Filter(this.WhenAnyValue(x => x.SearchQuery)
            .Throttle(TimeSpan.FromMilliseconds(500))
            .DistinctUntilChanged()
            .Select(Filter))
        .ObserveOn(RxApp.MainThreadScheduler)
        .Do(_ => IsLoading = true)
        .ObserveOn(RxApp.TaskpoolScheduler)
        .ObserveOn(RxApp.MainThreadScheduler)
        .TransformToTree(x => x.ParentOrganizationId)
        .Transform(node => new OrganizationViewModel(node))
        .DisposeMany()
        .SortAndBind(_source, SortExpressionComparer<OrganizationViewModel>.Ascending(x => x.OrganizationName))
        .Subscribe(_ => IsLoading = false)
        .DisposeWith(d);

    LoadCommand.Execute().Subscribe();
});

Although this works as expected, I feel that the initialization of LoadCommand is not right. What is the proper way to initialize it? Additionally, how should I handle exceptions that might be thrown while loading data? I could do this by subscribing to the ThrownExceptions observable within the LoadCommand when there was an actual command, but how can I handle exceptions in the reactive way?

[JakenVeina]

I guess it just how things work and nothing can be done here.

I wouldn't say this. I'd say you just need to be smarter about how much you're pushing to the UI. Unfortunately, like I mentioned before, I don't have any familiarity with .TransformToTree() or with UI trees in general, so I don't have any specific advice to offer, on optimizations. At the end of the day, you probably just want SOME kind of virtualization, where you're not sending all 65k items to the UI at once. Your next step if you want to try and optimize this may be to confirm whether I'm right about all 65k items are being processed by the UI "at once" or not.

Although this works as expected, I feel that the initialization of LoadCommand is not right. What is the proper way to initialize it?

I've not actually used Reactive UI a whole lot, believe it or not, so I don't know what all overloads there are for initializing a command. Me, I just rather like the idea of keeping all the real "logic" inside stream flows, so an "empty" command that just gets subscribed to to generate side-effects makes sense to me. You could maybe just move that first .Do() off of the command into the command action itself, that'd be functionally the same thing. Maybe there's like a separate non-generic ReactiveCommand class that's built for just being a subscribable? Your call what makes semantic sense to you.

Additionally, how should I handle exceptions that might be thrown while loading data?

Probably worth googling for "RX error handling" or similar and reading up on it, there's nothing special about errors with DD or with RX UI, but the gist is as follows:

Normal RX notifications are published via IObserver<T>.OnNext(T) and errors are published via IObserver<T>.OnError(Exception). RX operators virtually always just forward errors downstream, and treat that as a teardown operation. This is probably the biggest point to note: errors cause the whole stream to die, if not handled. Which is, of course, similar to how Exceptions work in imperative code, and the solution is the same: catch them at the lowest point in the stack (in RX-land, the earliest point in the stream) where you know how to recover. So, to expand my earlier example, that'll look roughly like...

.Select(_ => Observable.FromAsync(GetItemsAsync))
.Switch()
.Catch(error =>
{
    // Display error popup to the user?
    IsLoading = false;
    return Observable.Empty<...>();
})