github.com/broadinstitute/gatk-sv/VisualizeCnvs

Abstract

GATK-SV A structural variation discovery pipeline for Illumina short-read whole-genome sequencing (WGS) data. Table of Contents Requirements Citation Quickstart Pipeline Overview Cohort mode Single-sample mode gCNV model Generating a reference panel Module Descriptions GatherSampleEvidence - Raw callers and evidence collection EvidenceQC - Batch QC TrainGCNV - gCNV model creation GatherBatchEvidence - Batch evidence merging, BAF generation, and depth callers ClusterBatch - Site clustering GenerateBatchMetrics - Site metrics FilterBatch - Filtering MergeBatchSites - Cross-batch site merging GenotypeBatch - Genotyping RegenotypeCNVs - Genotype refinement (optional) MakeCohortVcf - Cross-batch integration, complex event resolution, and VCF cleanup Module 07 - Downstream Filtering AnnotateVcf - Annotation Module 09 - QC and Visualization Additional modules - Mosaic and de novo CI/CD Troubleshooting Requirements Deployment and execution: A Google Cloud account. A workflow execution system supporting the Workflow Description Language (WDL), either: Cromwell (v36 or higher). A dedicated server is highly recommended. or Terra (note preconfigured GATK-SV workflows are not yet available for this platform) Recommended: MELT. Due to licensing restrictions, we cannot provide a public docker image or reference panel VCFs for this algorithm. Recommended: cromshell for interacting with a dedicated Cromwell server. Recommended: WOMtool for validating WDL/json files. Alternative backends Because GATK-SV has been tested only on the Google Cloud Platform (GCP), we are unable to provide specific guidance or support for other execution platforms including HPC clusters and AWS. Contributions from the community to improve portability between backends will be considered on a case-by-case-basis. We ask contributors to please adhere to the following guidelines when submitting issues and pull requests: Code changes must be functionally equivalent on GCP backends, i.e. not result in changed output Increases to cost and runtime on GCP backends should be minimal Avoid adding new inputs and tasks to workflows. Simpler changes are more likely to be approved, e.g. small in-line changes to scripts or WDL task command sections Avoid introducing new code paths, e.g. conditional statements Additional backend-specific scripts, workflows, tests, and Dockerfiles will not be approved Changes to Dockerfiles may require extensive testing before approval We still encourage members of the community to adapt GATK-SV for non-GCP backends and share code on forked repositories. Here are a some considerations: Refer to Cromwell's documentation for configuration instructions. The handling and ordering of glob commands may differ between platforms. Shell commands that are potentially destructive to input files (e.g. rm, mv, tabix) can cause unexpected behavior on shared filesystems. Enabling copy localization may help to more closely replicate the behavior on GCP. For clusters that do not support Docker, Singularity is an alternative. See Cromwell documentation on Singularity. The GATK-SV pipeline takes advantage of the massive parallelization possible in the cloud. Local backends may not have the resources to execute all of the workflows. Workflows that use fewer resources or that are less parallelized may be more successful. For instance, some users have been able to run GatherSampleEvidence on a SLURM cluster. Data: Illumina short-read whole-genome CRAMs or BAMs, aligned to hg38 with bwa-mem. BAMs must also be indexed. Family structure definitions file in PED format. Sex aneuploidies (detected in EvidenceQC) should be entered as sex = 0. Sample Exclusion We recommend filtering out samples with a high percentage of improperly paired reads (>10% or an outlier for your data) as technical outliers prior to running GatherSampleEvidence. A high percentage of improperly paired reads may indicate issues with library prep, degradation, or contamination. Artifactual improperly paired reads could cause incorrect SV calls, and these samples have been observed to have longer runtimes and higher compute costs for GatherSampleEvidence. Sample ID requirements: Sample IDs must: Be unique within the cohort Contain only alphanumeric characters and underscores (no dashes, whitespace, or special characters) Sample IDs should not: Contain only numeric characters Be a substring of another sample ID in the same cohort Contain any of the following substrings: chr, name, DEL, DUP, CPX, CHROM The same requirements apply to family IDs in the PED file, as well as batch IDs and the cohort ID provided as workflow inputs. Sample IDs are provided to GatherSampleEvidence directly and need not match sample names from the BAM/CRAM headers. GetSampleID.wdl can be used to fetch BAM sample IDs and also generates a set of alternate IDs that are considered safe for this pipeline; alternatively, this script transforms a list of sample IDs to fit these requirements. Currently, sample IDs can be replaced again in GatherBatchEvidence. The following inputs will need to be updated with the transformed sample IDs: Sample ID list for GatherSampleEvidence or GatherBatchEvidence PED file Citation Please cite the following publication: Collins, Brand, et al. 2020. "A structural variation reference for medical and population genetics." Nature 581, 444-451. Additional references: Werling et al. 2018. "An analytical framework for whole-genome sequence association studies and its implications for autism spectrum disorder." Nature genetics 50.5, 727-736. Quickstart WDLs There are two scripts for running the full pipeline: wdl/GATKSVPipelineBatch.wdl: Runs GATK-SV on a batch of samples. wdl/GATKSVPipelineSingleSample.wdl: Runs GATK-SV on a single sample, given a reference panel Building inputs Example workflow inputs can be found in /inputs. Build using scripts/inputs/build_default_inputs.sh, which generates input jsons in /inputs/build. Except the MELT docker image, all required resources are available in public Google buckets. Some workflows require a Google Cloud Project ID to be defined in a cloud environment parameter group. Workspace builds require a Terra billing project ID as well. An example is provided at /inputs/values/google_cloud.json but should not be used, as modifying this file will cause tracked changes in the repository. Instead, create a copy in the same directory with the format google_cloud.my_project.json and modify as necessary. Note that these inputs are required only when certain data are located in requester pays buckets. If this does not apply, users may use placeholder values for the cloud configuration and simply delete the inputs manually. MELT Important: The example input files contain MELT inputs that are NOT public (see Requirements). These include: GATKSVPipelineSingleSample.melt_docker and GATKSVPipelineBatch.melt_docker - MELT docker URI (see Docker readme) GATKSVPipelineSingleSample.ref_std_melt_vcfs - Standardized MELT VCFs (GatherBatchEvidence) The input values are provided only as an example and are not publicly accessible. In order to include MELT, these values must be provided by the user. MELT can be disabled by deleting these inputs and setting GATKSVPipelineBatch.use_melt to false. Requester pays buckets Important: The following parameters must be set when certain input data is in requester pays (RP) buckets: GATKSVPipelineSingleSample.requester_pays_cram and GATKSVPipelineBatch.GatherSampleEvidenceBatch.requester_pays_crams - set to True if inputs are CRAM format and in an RP bucket, otherwise False. Execution We recommend running the pipeline on a dedicated Cromwell server with a cromshell client. A batch run can be started with the following commands: > mkdir gatksv_run && cd gatksv_run > mkdir wdl && cd wdl > cp $GATK_SV_ROOT/wdl/*.wdl . > zip dep.zip *.wdl > cd .. > echo '{ "google_project_id": "my-google-project-id", "terra_billing_project_id": "my-terra-billing-project" }' > inputs/values/google_cloud.my_project.json > bash scripts/inputs/build_default_inputs.sh -d $GATK_SV_ROOT -c google_cloud.my_project > cp $GATK_SV_ROOT/inputs/build/ref_panel_1kg/test/GATKSVPipelineBatch/GATKSVPipelineBatch.json GATKSVPipelineBatch.my_run.json > cromshell submit wdl/GATKSVPipelineBatch.wdl GATKSVPipelineBatch.my_run.json cromwell_config.json wdl/dep.zip where cromwell_config.json is a Cromwell workflow options file. Note users will need to re-populate batch/sample-specific parameters (e.g. BAMs and sample IDs). Pipeline Overview The pipeline consists of a series of modules that perform the following: GatherSampleEvidence: SV evidence collection, including calls from a configurable set of algorithms (Manta, MELT, and Wham), read depth (RD), split read positions (SR), and discordant pair positions (PE). EvidenceQC: Dosage bias scoring and ploidy estimation GatherBatchEvidence: Copy number variant calling using cn.MOPS and GATK gCNV; B-allele frequency (BAF) generation; call and evidence aggregation ClusterBatch: Variant clustering GenerateBatchMetrics: Variant filtering metric generation FilterBatch: Variant filtering; outlier exclusion GenotypeBatch: Genotyping MakeCohortVcf: Cross-batch integration; complex variant resolution and re-genotyping; vcf cleanup Module 07: Downstream filtering, including minGQ, batch effect check, outlier samples removal and final recalibration; AnnotateVcf: Annotations, including functional annotation, allele frequency (AF) annotation and AF annotation with external population callsets; Module 09: Visualization, including scripts that generates IGV screenshots and rd plots. Additional modules to be added: de novo and mosaic scripts Repository structure: /dockerfiles: Resources for building pipeline docker images /inputs: files for generating workflow inputs /templates: Input json file t