---
title: "Introduction_1_load_metadata"
author: "Hangjia Zhao"
date: "`r Sys.Date()`"
output: 
  BiocStyle::html_document:
  toc: true
vignette: >
  %\VignetteIndexEntry{Introduction_1_load_metadata}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
    collapse = TRUE,
    comment = "#>"
)
```

[Progenetix](https://progenetix.org/) is an open data resource that provides curated individual cancer copy number variation (CNV) profiles along with associated metadata sourced from published oncogenomic studies and various data repositories. This vignette provides a comprehensive guide on accessing and utilizing metadata for samples or their corresponding individuals within the Progenetix database. 

If your focus lies in cancer cell lines, you can access data from [*cancercelllines.org*](https://cancercelllines.org/) by setting the `domain` parameter to `"cancercelllines.org"` in `pgxLoader` function. This data repository originates from CNV profiling data of cell lines initially collected as part of Progenetix and currently includes additional types of genomic mutations.

# Load library 

```{r setup}
library(pgxRpi)
```

## `pgxLoader` function 

This function loads various data from `Progenetix` database via the Beacon v2 API with some extensions (BeaconPlus).

The parameters of this function used in this tutorial:

* `type`: A string specifying output data type. `"individuals"`, `"biosamples"`, `"analyses"`, `"filtering_terms"`, and `"counts"` are used in this tutorial.
* `filters`: Identifiers used in public repositories, bio-ontology terms, or custom terms such as `c("NCIT:C7376", "pgx:icdom-85003")`. When multiple filters are used, they are combined using AND logic when the parameter `type` is `"individuals"`, `"biosamples"`, or `"analyses"`; OR logic when the parameter `type` is `"counts"`.
* `individual_id`: Identifiers used in the query database for identifying individuals. 
* `biosample_id`: Identifiers used in the query database for identifying biosamples. 
* `codematches`: A logical value determining whether to exclude samples from child concepts 
of specified filters in the ontology tree. If `TRUE`, only samples exactly matching the specified filters will be included. 
Do not use this parameter when `filters` include ontology-irrelevant filters such as pubmed or cohort identifiers. Default is `FALSE`.
* `filter_pattern`: Optional string pattern to match against the `label` field of available filters. Only used when the parameter type is `"filtering_terms"`. Default is `NULL`, which includes all filters.
* `limit`: Integer to specify the number of returned profiles. Default is `0` (return all). 
* `skip`: Integer to specify the number of skipped profiles. E.g. if `skip = 2, limit=500`, the first 2*500=1000 profiles are skipped and the next 500 profiles are returned. Default is `0` (no skip). 
* `use_https`: A logical value indicating whether to use the HTTPS protocol. If `TRUE`, the domain will be prefixed with `"https://"`; otherwise, `"http://"` will be used. Default is `TRUE`.
* `dataset`: A string specifying the dataset to query from the Beacon response. Default is `NULL`, which includes results from all datasets.
* `domain`: A string specifying the domain of the query data resource. Default is `"progenetix.org"`.
* `entry_point`: A string specifying the entry point of the Beacon v2 API. Default is `"beacon"`, resulting in the endpoint being `"https://progenetix.org/beacon"`.
* `num_cores`: An integer specifying the number of cores to use for parallel processing during Beacon v2 phenotypic/meta-data queries from multiple domains. Default is `1`.

# Retrieve biosamples information

## Search by filters

Filters are a significant enhancement to the [Beacon](https://www.ga4gh.org/product/beacon-api/) query API, providing a mechanism for specifying rules to select records based on their field values. To learn more about how to utilize filters in Progenetix, please refer to the [documentation](https://docs.progenetix.org/common/beacon-api/#filters-filtering-terms).

The following example demonstrates how to access all available filters in Progenetix:

```{r}
all_filters <- pgxLoader(type="filtering_terms")
head(all_filters)
```

If you're interested in filters related to a specific disease or phenotype, you can use the `filter_pattern` argument to narrow down the list. For example, to search for filters related to retinoblastoma:

```{r}
query_filter <- pgxLoader(type="filtering_terms",filter_pattern="retinoblastoma")
query_filter
```

To retrieve biosamples associated with a specific disease, use appropriate filter terms. In this example, we use an [NCIt](https://ncit.nci.nih.gov) code corresponding to **retinoblastoma** (`NCIT:C7541`):

```{r}
biosamples <- pgxLoader(type="biosamples", filters = "NCIT:C7541")
# data looks like this
biosamples[1:5,]
```

The data contains many columns representing different aspects of sample information.

## Search by biosample id and individual id 

In the Beacon v2 specification, biosample id and individual id are unique identifiers for biosamples and their corresponding individuals, respectively. These identifiers can be obtained through metadata searches using filters as described above or by querying the [Progenetix search interface](https://progenetix.org/search), which provides access to the IDs used in the Progenetix database.

```{r}
biosamples_2 <- pgxLoader(type="biosamples", biosample_id = "pgxbs-kftvki7h",individual_id = "pgxind-kftx6ltu")

biosamples_2
```

It's also possible to query by a combination of filters, biosample id, and individual id.

## Access a subset of samples

By default, it returns all related samples (limit=0). You can access a subset of them 
via the parameter `limit` and `skip`. For example, if you want to access the first 10 samples
, you can set `limit` = 10, `skip` = 0. 

```{r}
biosamples_3 <- pgxLoader(type="biosamples", filters = "NCIT:C7541",skip=0, limit = 10)
# Dimension: Number of samples * features
print(dim(biosamples))
print(dim(biosamples_3))
```

## Parameter `codematches` use

Some filters, such as NCIt codes, are hierarchical. As a result, retrieved samples may 
include not only the specified filters but also their child terms.

```{r}
unique(biosamples$histological_diagnosis_id)
```

Setting `codematches` as TRUE allows this function to only return biosamples that exactly match the specified filter, excluding child terms.

```{r}
biosamples_4 <- pgxLoader(type="biosamples", filters = "NCIT:C7541",codematches = TRUE)
unique(biosamples_4$histological_diagnosis_id)
```

# Retrieve individuals information

If you want to query details of individuals (e.g. clinical data) where the samples 
of interest come from, set the parameter `type` to "individuals" and follow the same steps as above.

```{r}
individuals <- pgxLoader(type="individuals",individual_id = "pgxind-kftx26ml",filters="NCIT:C7541")
# data looks like this
tail(individuals,2)
```

# Retrieve analyses information 

If you want to know more details about data analyses, set the parameter `type` to "analyses". 
The other steps are the same, except the parameter `codematches` is not available because 
analyses data do not include filter information, even though it can be searched by filters.

```{r}
analyses <- pgxLoader(type="analyses",biosample_id = c("pgxbs-kftvik5i","pgxbs-kftvik96"))

analyses
```

# Retrieve the number of results for a specific filter

To retrieve the number of results for a specific filter in Progenetix, set the `type` parameter to `"counts"`. You can query different Beacon v2 resources by setting the `domain` and `entry_point` parameters accordingly.

```{r}
pgxLoader(type="counts",filters = "NCIT:C7541")
```

# Query from multiple data resources

You can query data from multiple resources via the Beacon v2 API by setting the `domain` and `entry_point` parameters accordingly. To speed up the process, use the `num_cores` parameter to enable parallel processing across different domains. For resources that only support `http` (e.g., local or internal network instances), set `use_https = FALSE` to avoid connection issues.

```{r}
record_counts <- pgxLoader(type="counts",filters = "NCIT:C9245",domain=c("progenetix.org","cancercelllines.org"), entry_point=c("beacon","beacon"))

record_counts
```

# Visualization of survival data

Suppose you want to investigate whether there are survival differences associated with a particular disease, for example, between younger and older patients, or based on other variables. You can query and visualize the relevant information using the `pgxMetaplot` function.

## `pgxMetaplot` function

This function generates a survival plot using metadata of individuals obtained by the `pgxLoader` function.

The parameters of this function:

* `data`: The data frame returned by the `pgxLoader` function, containing survival data for individuals. The survival state is represented by Experimental Factor Ontology in the "followup_state_id" column, and the survival time is represented in ISO 8601 duration format in the "followup_time" column.
* `group_id`: A string specifying which column is used for grouping in the Kaplan-Meier plot.
* `condition`: A string for splitting individuals into younger and older groups, following the ISO 8601 duration format. Only used if `group_id` is "age_iso".
* `return_data`: A logical value determining whether to return the metadata used for plotting. Default is FALSE.
* `...`: Other parameters relevant to KM plot. These include `pval`, `pval.coord`, `pval.method`, `conf.int`, `linetype`, and `palette` (see ggsurvplot function from survminer package)

### Example usage 

```{r}
# query metadata of individuals with lung adenocarcinoma
luad_inds <- pgxLoader(type="individuals",filters="NCIT:C3512")
# use 70 years old as the splitting condition
pgxMetaplot(data=luad_inds, group_id="age_iso", condition="P70Y", pval=TRUE)
```

It's noted that not all individuals have available survival data. If you set `return_data` to TRUE, 
the function will return the metadata of individuals used for the plot.

# Session Info

```{r echo = FALSE}
sessionInfo()
```


