buildBench {SummarizedBenchmark}R Documentation

Make SummarizedBenchmark from BenchDesign

Description

Function to evaluate BenchDesign methods on supplied data set to generate a SummarizedBenchmark. In addition to the results of applying each method on the data, the returned SummarizedBenchmark also includes metadata for the methods in the colData of the returned object, metadata for the data in the rowData, and the session information generated by sessionInfo() in the metadata.

Usage

buildBench(bd, data = NULL, truthCols = NULL, ftCols = NULL,
  ptabular = TRUE, sortIDs = FALSE, catchErrors = TRUE,
  parallel = FALSE, BPPARAM = bpparam())

Arguments

bd

BenchDesign object.

data

Data set to be used for benchmarking, will take priority over data set originally specified to BenchDesign object. Ignored if NULL. (default = NULL)

truthCols

Character vector of column names in data set corresponding to ground truth values for each assay. If specified, column will be added to the groundTruth DataFrame for the returned SummarizedBenchmark object. If the BenchDesign includes only a single assay, the same name will be used for the assay. If the BenchDesign includes multiple assays, to map data set columns with assays, the vector must have names corresponding to the assay names specified to the post parameter at each addMethod call. (default = NULL)

ftCols

Vector of character names of columns in data set that should be included as feature data (row data) in the returned SummarizedBenchmark object. (default = NULL)

ptabular

Whether to return method parameters with each parameter in a separate column of the colData for the returned SummarizedBenchmark object, i.e. in tabular form. If FALSE, method parameters are returned as a single column with comma-delimited "key=value" pairs. (default = TRUE)

sortIDs

Whether the output of each method should be merged using IDs. If TRUE, each method must return a named vector or list. The names will be used to align the output of each method in the returned SummarizedBenchmark. Missing values will be set to NA. This can be useful if the different methods return overlapping, but not identical, results. If truthCols is also specified, and sorting by IDs is necessary, rather than specifying 'TRUE', specify the string name of a column in the data to use to sort the method output to match the order of truthCols. (default = FALSE)

catchErrors

logical whether errors produced by methods during evaluation should be caught and printed as a message without stopping the entire build process. (default = TRUE)

parallel

Whether to use parallelization for evaluating each method. Parallel execution is performed using BiocParallel. Parameters for parallelization should be specified with BiocParallel::register or through the BPPARAM parameter. (default = FALSE)

BPPARAM

Optional BiocParallelParam instance to be used when parallel is TRUE. If not specified, the default instance from the parameter registry is used.

Details

Parallelization is performed across methods. Therefore, there is no benefit to specifying more cores than the total number of methods in the BenchDesign object. By default, errors thrown by individual methods in the BenchDesign are caught during evaluation and handled in a way that allows buildBench to continue running with the other methods. The error is printed as a message, and the corresponding column in the returned SummarizedBenchmark object is set to NA. Since many benchmarking experiments can be time and computationally intensive, having to rerun the entire analysis due to a single failed method can be frustrating. Default error catching was included to alleviate these frustrations. However, if this behavior is not desired, setting catchErrors = FALSE will turn off error handling.

Value

SummarizedBenchmark object with single assay

Author(s)

Patrick Kimes

Examples

## with toy data.frame
df <- data.frame(pval = rnorm(100))
bench <- BenchDesign(df)

## add methods
bench <- addMethod(bench, label = "bonf", func = p.adjust,
                   params = rlang::quos(p = pval, method = "bonferroni"))
bench <- addMethod(bench, label = "BH", func = p.adjust,
                   params = rlang::quos(p = pval, method = "BH"))

## evaluate benchmark experiment
sb <- buildBench(bench)

## evaluate benchmark experiment w/ data sepecified
sb <- buildBench(bench, data = df)


[Package SummarizedBenchmark version 1.0.4 Index]