"locked" proxy structure not accommodating new data

[rfgoldberg]

Hello, I've discovered an issue with proxy and show/hide columns when they're acting upon reactive datasets. In the example below, at first the dataset is not fed to the proxy until "showdata" is pressed. Even though in a shiny reactlog, the datatable is fully rendered, nothing will appear in the output object. It seems the proxy is not updating the table "shell" once data is passed to it, which results in the data not rendering. This "locked" proxy structure behavior is also observed when you pass an empty or single row dataset to the proxy, but then wish to update the table to multiple rows. With a non-NULL original dataset, the show/hide functionality works, but the top filters will not work, because as far as they know, there's only one or no row to filter. Only when a dataset of at least 2 different rows are passed to the proxy when it's first created, do the filters (where applicable to those 2 rows) work as expected. Reprex below, adapted from here.

```{r}
library(shiny)
library(DT)
library(tidyverse)

shinyApp(
  
  ui = fluidPage(
    actionButton('showdata',"Show Data"),
    actionButton('prev_five', 'Previous Cols'),
    actionButton('next_five', 'Next Cols'),
    DTOutput('tbl')),
  
  
  server = function(input, output) {
    
    cols <- reactiveValues()   
    cols$showing <- 1:3    
    
    # currently relies on observer to populate data
    observeEvent(input$showdata,{
      cols$df <- iris
    })
    
    # 1. uncomment this to observe an empty dataset fed to proxy first
    # observe({
    #   cols$df <- iris %>% filter(Species == "x")
    # })
    
    # 2. uncomment this to observe a filled dataset fed to proxy first 
    # change matrix to only one or two rows to see the difference in tp filters that appear in the DT
    # observe({
    #   # cols$df <- iris %>% filter(Species == "x")
    #   cols$df <- data.frame(matrix(1:10,nrow = 2,ncol = 5)) |>
    #     `colnames<-` (c(names(iris)))
    # })
    
    
    #show the next five columns 
    observeEvent(input$next_five, {
      #stop when the last column is displayed
      if(cols$showing[[length(cols$showing)]] < length(cols$df)) {
        hideCols(proxy, cols$showing, reset = FALSE) #hide displayed cols
        cols$showing <- cols$showing + 2
        showCols(proxy, cols$showing, reset = FALSE) #show the next five 
      } 
    })
    
    #similar mechanism but reversed to show the previous cols
    observeEvent(input$prev_five, {
      #stop when the first column is displayed
      if(cols$showing[[1]] > 1) {
        hideCols(proxy, cols$showing, reset = FALSE) #hide displayed cols
        cols$showing <- cols$showing - 2
        showCols(proxy, cols$showing, reset = FALSE) #show previous five
      } 
    })
    
   
    output$tbl = renderDT({
      datatable(isolate(cols$df),filter = "top",
                options = list(
                  columnDefs = list(list(visible = FALSE, targets = 1:length(isolate(cols$df)))), #hide all columns
                  scrollX = TRUE)  #for when many columns are visible
      )
    })
    
    proxy <- dataTableProxy('tbl')
    observe({
      DT::replaceData(proxy, cols$df, resetPaging = FALSE)
    })
    showCols(proxy, 1:3, reset = FALSE) #show the first five cols (because the columns are now all hidden)
  
  }
)


```

R version 4.3.2 (2023-10-31)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux 8.9 (Ootpa), RStudio 2023.9.0.463

Locale:
LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C
LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

Package version:
base64enc_0.1.3 bslib_0.6.1 cachem_1.0.8 cli_3.6.1 crosstalk_1.2.1 digest_0.6.33 DT_0.32
ellipsis_0.3.2 evaluate_0.23 fastmap_1.1.1 fontawesome_0.5.2 fs_1.6.3 glue_1.6.2 graphics_4.3.2
grDevices_4.3.2 highr_0.10 htmltools_0.5.7 htmlwidgets_1.6.4 httpuv_1.6.14 jquerylib_0.1.4 jsonlite_1.8.8
knitr_1.45 later_1.3.2 lazyeval_0.2.2 lifecycle_1.0.4 magrittr_2.0.3 memoise_2.0.1 methods_4.3.2
mime_0.12 promises_1.2.1 R6_2.5.1 rappdirs_0.3.3 Rcpp_1.0.11 rlang_1.1.2 rmarkdown_2.25
sass_0.4.8 stats_4.3.2 stringi_1.8.3 stringr_1.5.1 tinytex_0.49 tools_4.3.2 utils_4.3.2
vctrs_0.6.5 xfun_0.41 yaml_2.3.8

---

By filing an issue to this repo, I promise that

• I have fully read the issue guide at https://yihui.org/issue/.
• I have provided the necessary information about my issue.
  • If I'm asking a question, I have already asked it on Stack Overflow or RStudio Community, waited for at least 24 hours, and included a link to my question there.
  • If I'm filing a bug report, I have included a minimal, self-contained, and reproducible example, and have also included xfun::session_info('DT'). I have upgraded all my packages to their latest versions (e.g., R, RStudio, and R packages), and also tried the development version: remotes::install_github('rstudio/DT').
  • If I have posted the same issue elsewhere, I have also mentioned it in this issue.
• I have learned the Github Markdown syntax, and formatted my issue correctly.

I understand that my issue may be closed if I don't fulfill my promises.

[stla]

From the documentation:

When you replace the data in an existing table, please make sure the new data has the same number of columns as the current data.

So if you start with a NULL dataset, it is expected that replaceData will not work.

[rfgoldberg]

That's what option 1. is supposed to simulate. If you populate the proxy with a dataset with no rows but that has the correct number of columns, the filters still don't work when data is updated and elongated.

[stla]

I think the reason can also be found in ?replaceData:

When you have enabled column filters, you should also make sure the attributes of every column remain the same, e.g. factor columns should have the same or fewer levels, and numeric columns should have the same or smaller range, otherwise the filters may never be able to reach certain rows in the data, unless you explicitly update the filters with updateFilters().

[rfgoldberg]

Thank you so much for this clarification. I believe updateFilters is what I'm looking for, but I've run into another issue with it though. I've modified the reprex above to include the modifications you suggested and with a wider dataset as is applicable to my use case. If you run the reprex, you'll notice very long processing time between when the button to populate the data is clicked and when the table actually renders. It also takes longer for just the empty table to show up. I only observe this behavior when the top filter is implemented. Once top filter is removed, the processing speed increases dramatically to populate the DT. Is this something that should be faster? On my actual shiny dashboard, it results in about 12-15 seconds of nothing happening, for a table of 3 row x 189 cols.

## added to load wide dataset (143 variables)
dat <- chickwts %>%
  bind_rows(chickwts) %>%
  mutate(name = paste0("col",row_number())) %>%
  pivot_wider(id_cols = feed, names_from = name, values_from = weight)





shinyApp(
  
  ui = fluidPage(
    actionButton('showdata',"Show Data"),
    actionButton('prev_five', 'Previous Cols'),
    actionButton('next_five', 'Next Cols'),
    DTOutput('tbl')),
  
  
  server = function(input, output) {
    
    
    
    cols <- reactiveValues()   
    cols$showing <- 1:3    
    
    # currently relies on observer to populate data
    observeEvent(input$showdata,{
      cols$df <- dat 
    })
    
    # 1. use this to observe an empty dataset fed to proxy first
    observe({
      cols$df <- dat %>% filter(col1 == "test")
    })
    
    # 2. use this to observe a filled dataset fed to proxy first 
    # change matrix to only one or two rows to see the difference in tp filters that appear in the DT
    # observe({
    #   # cols$df <- iris %>% filter(Species == "x")
    #   cols$df <- data.frame(matrix(1:10,nrow = 2,ncol = 5)) |>
    #     `colnames<-` (c(names(iris)))
    # })
    
    
    #show the next five columns 
    observeEvent(input$next_five, {
      #stop when the last column is displayed
      if(cols$showing[[length(cols$showing)]] < length(cols$df)) {
        hideCols(proxy, cols$showing, reset = FALSE) #hide displayed cols
        cols$showing <- cols$showing + 2
        showCols(proxy, cols$showing, reset = FALSE) #show the next five 
      } 
    })
    
    #similar mechanism but reversed to show the previous cols
    observeEvent(input$prev_five, {
      #stop when the first column is displayed
      if(cols$showing[[1]] > 1) {
        hideCols(proxy, cols$showing, reset = FALSE) #hide displayed cols
        cols$showing <- cols$showing - 2
        showCols(proxy, cols$showing, reset = FALSE) #show previous five
      } 
    })
    
    
    output$tbl = renderDT({
      datatable(isolate(cols$df),filter = "top", # <- if this is commented out, this goes much more quickly
                options = list(
                  columnDefs = list(list(visible = FALSE, targets = 1:length(isolate(cols$df)))), #hide all columns
                  scrollX = TRUE)  #for when many columns are visible
      )
    })
    
    
    proxy <- dataTableProxy('tbl')
    observe({
      DT::replaceData(proxy, cols$df, resetPaging = FALSE)
      DT::updateFilters(proxy,cols$df)
    })
    showCols(proxy, 1:3, reset = FALSE) #show the first five cols (because the columns are now all hidden)
  
  }
)