[Bug]: tm_outliers can not create a report card / Error : Can't subset columns that don't exist. ✖ Column order doesn't exist.

[m7pr]

For exploratory app

1. Go to Outliers module,
2. Select categorical factor: PARAM
3. Try adding a report card.

On Add Card you will see

The card could not be added to the report. Have the outputs for the report been created yet? If not please try again when they are ready. Otherwise contact your application developer

Checked on 669_insertUI@main branches for: teal.slice. teal and teal.data

[m7pr]

This is what I see in the console

Warning in observe() :
  The card could not be added to the report. Have the outputs for the report been created yet? If not please try again when they are ready. Otherwise contact your application developer
Error : Can't subset columns that don't exist.
✖ Column `order` doesn't exist. 
 when evaluating qenv code:
ANL_OUTLIER_EXTENDED[ANL_OUTLIER_EXTENDED$is_outlier_selected, columns_index]

trace: 
 library(scda)
library(scda.2022)
library(dplyr)
library(tidyr)
library(ggExtra)
library(ggpmisc)
library(ggpp)
library(goftest)
library(gridExtra)
library(htmlwidgets)
library(jsonlite)
library(lattice)
library(MASS)
library(rlang)
library(rtables)
library(nestcolor)
library(broom)
library(colourpicker)
library(sparkline)
ADSL <- synthetic_cdisc_dataset("latest", "adsl")
ADLB <- synthetic_cdisc_dataset("latest", "adlb")
stopifnot(rlang::hash(ADSL) == "0f1586cab2e43352b9a1473f2093eb69")
stopifnot(rlang::hash(ADLB) == "a901545cf1043a604915be404481ec6e")
ADSL_raw <- ADSL
ADLB_raw <- ADLB

ADLB <- dplyr::inner_join(x = ADLB, y = ADSL[, c("STUDYID", "USUBJID"), drop = FALSE], by = c("STUDYID", "USUBJID"))
 ANL_1 <- ADLB %>% dplyr::select(STUDYID, USUBJID, PARAMCD, AVISIT, AVAL, PARAM)
ANL <- ANL_1
ANL <- ANL %>% teal.data::col_relabel(AVAL = "Analysis Value", PARAM = "Parameter")
 ANL_OUTLIER <- ANL %>% dplyr::mutate(is_outlier = {
    q1_q3 <- stats::quantile(AVAL, probs = c(0.25, 0.75))
    iqr <- q1_q3[2] - q1_q3[1]
    !(AVAL >= q1_q3[1] - 1.5 * iqr & AVAL <= q1_q3[2] + 1.5 * iqr)
}) %>% dplyr::mutate(is_outlier_selected = {
    q1_q3 <- stats::quantile(AVAL, probs = c(0.25, 0.75))
    iqr <- q1_q3[2] - q1_q3[1]
    !(AVAL >= q1_q3[1] - 3 * iqr & AVAL <= q1_q3[2] + 3 * iqr)
}) %>% dplyr::filter(is_outlier | is_outlier_selected) %>% dplyr::select(-is_outlier)
 ANL_OUTLIER_EXTENDED <- dplyr::left_join(ANL_OUTLIER, dplyr::select(ADSL, dplyr::setdiff(names(ADSL), dplyr::setdiff(names(ANL_OUTLIER), c("STUDYID", "USUBJID")))), by = c("STUDYID", "USUBJID"))
 summary_table_pre <- ANL_OUTLIER %>% dplyr::filter(is_outlier_selected) %>% dplyr::select(AVAL, PARAM) %>% dplyr::group_by(PARAM) %>% dplyr::summarise(n_outliers = dplyr::n()) %>% dplyr::right_join(ANL %>% dplyr::select(AVAL, PARAM) %>% dplyr::group_by(PARAM) %>% dplyr::summarise(total_in_cat = dplyr::n(), n_na = sum(is.na(AVAL) | is.na(PARAM))), by = "PARAM") %>% dplyr::arrange(PARAM) %>% dplyr::mutate(n_outliers = dplyr::if_else(is.na(n_outliers), 0, as.numeric(n_outliers)), display_str = dplyr::if_else(n_outliers > 
    0, sprintf("%d [%.02f%%]", n_outliers, 100 * n_outliers/total_in_cat), "0"), display_str_na = dplyr::if_else(n_na > 0, sprintf("%d [%.02f%%]", n_na, 100 * n_na/total_in_cat), "0"), order = seq_along(n_outliers))
 ANL_OUTLIER <- dplyr::left_join(ANL_OUTLIER, summary_table_pre[, c("order", "PARAM")], by = "PARAM")
ANL <- ANL %>% dplyr::left_join(dplyr::select(summary_table_pre, PARAM, order), by = "PARAM") %>% dplyr::arrange(order)
summary_table <- summary_table_pre %>% dplyr::select(PARAM, Outliers = display_str, Missings = display_str_na, Total = total_in_cat) %>% dplyr::mutate_all(as.character) %>% tidyr::pivot_longer(-PARAM) %>% tidyr::pivot_wider(names_from = "PARAM", values_from = value) %>% tibble::column_to_rownames("name")
summary_table
 g <- ANL %>% ggplot() + geom_boxplot(outlier.shape = NA) + aes(y = AVAL, x = reorder(PARAM, order)) + xlab("PARAM") + scale_x_discrete() + geom_point(data = ANL_OUTLIER, aes(x = as.factor(PARAM), y = AVAL, color = is_outlier_selected)) + scale_color_manual(values = c(`TRUE` = "red", `FALSE` = "black")) + ggplot2::labs(caption = "NEST PROJECT", color = "Is outlier?") + ggplot2::theme_gray() + ggplot2::theme(legend.position = "top")
 print(g)
 columns_index <- union(setdiff(names(ANL_OUTLIER), "is_outlier_selected"), NULL)
ANL_OUTLIER_EXTENDED[ANL_OUTLIER_EXTENDED$is_outlier_selected, columns_index]

Warning in observe() :
  The card could not be added to the report. Have the outputs for the report been created yet? If not please try again when they are ready. Otherwise contact your application developer

[donyunardi]

I no longer see this issue in our teal.gallery _main version which is using the main branch for all teal dependencies:
https://genentech.shinyapps.io/nest_exploratory_main/

@m7pr @PritDash
Could you please try the above link to confirm?

[m7pr]

@donyunardi I think _main uses releases and _dev uses @main branches :D

So it some changes are added to @main branch but there is no GitHub release, those are just visible in _dev app and not in _main app.

Right @vedhav :P?

[vedhav]

Yeah, that's true. _main apps use the released version of packages. And, _dev apps use the main branch of the packages.

Edit: _stable apps, not the _main apps. We renamed it to reduce confusion.

[donyunardi]

Thank you @m7pr @vedhav
Yes, I see the same issue in the _dev apps. I'll take a look at the PR.

[donyunardi]

Yeah, that's true. _main apps use the released version of packages.

I believe the released version of packages are in _stable. This is the link that we provided in teal.gallery website.

I also want to mention that this issue is reproducible in both _stable and _dev apps.

The error occurs when the Categorical factor is added because the order column is generated when the user selects a categorical variable:

teal.modules.general/R/tm_outliers.R

Lines 569 to 604 in 6527128

	if (length(categorical_var) > 0) {
	qenv <- teal.code::eval_code(
	qenv,
	substitute(
	expr = summary_table_pre <- ANL_OUTLIER %>%
	dplyr::filter(is_outlier_selected) %>%
	dplyr::select(outlier_var_name, categorical_var_name) %>%
	dplyr::group_by(categorical_var_name) %>%
	dplyr::summarise(n_outliers = dplyr::n()) %>%
	dplyr::right_join(
	ANL %>%
	dplyr::select(outlier_var_name, categorical_var_name) %>%
	dplyr::group_by(categorical_var_name) %>%
	dplyr::summarise(
	total_in_cat = dplyr::n(),
	n_na = sum(is.na(outlier_var_name) | is.na(categorical_var_name))
	),
	by = categorical_var
	) %>%
	# This is important as there may be categorical variables with natural orderings, e.g. AGE.
	# The plots should be displayed by default in increasing order in these situations.
	# dplyr::arrange will sort integer, factor, and character data types in the expected way.
	dplyr::arrange(categorical_var_name) %>%
	dplyr::mutate(
	n_outliers = dplyr::if_else(is.na(n_outliers), 0, as.numeric(n_outliers)),
	display_str = dplyr::if_else(
	n_outliers > 0,
	sprintf("%d [%.02f%%]", n_outliers, 100 * n_outliers / total_in_cat),
	"0"
	),
	display_str_na = dplyr::if_else(
	n_na > 0,
	sprintf("%d [%.02f%%]", n_na, 100 * n_na / total_in_cat),
	"0"
	),
	order = seq_along(n_outliers)

This is related with #574 where this statement is added to the qenv.

ANL_OUTLIER_EXTENDED[ANL_OUTLIER_EXTENDED$is_outlier_selected, columns_index]

Clicking Show R Code button would also show the same error message.
The error that occurs when adding the card is due to teal.reporter re-creating the output before adding it to the card, which causes the line to fail.