A grammar of graphics over a BAM (NA12878 insert size)
The tutorials target the JBrowse v5 beta, and the v4.3.0 release
on the download page lacks some of what they show. To install the
beta of JBrowse Web, run npm install -g @jbrowse/cli@next, then
jbrowse create jbrowse2 --branch v5.0.0-beta.11. Desktop beta builds are coming soon.
A read pair that straddles a deletion maps with a long insert, and a
heterozygous deletion halves the read depth. We plot those two fields straight
from NA12878's reads to find a deletion on chromosome 20 without a variant
caller, scan the whole chromosome for the same signature, and check the hits
against the 1000 Genomes callset. The plots come from JBrowse's mark display, a
grammar of graphics over a track: each entry in marks names a mark type, a
transform list and an encoding from feature fields to channels, as in the
Alu tutorial. The mark display is experimental, and
its config shape may change.
Prerequisites
- a JBrowse to open the figures' sessions in (Web or Desktop)
- samtools and htslib (
bgzip,tabix), for cutting the pairs out of the file and for checking a window by hand - bcftools, for reading the callset at the end
- Node.js and the JBrowse CLI, for the build script
Where the data comes from
1000 Genomes high-coverage release (Byrska-Bishop et al. 2022), GRCh38:
- NA12878's 30x CRAM, index beside it: s3.amazonaws.com/…/NA12878.final.cramhttps://s3.amazonaws.com/1000genomes/1000G_2504_high_coverage/data/ERR3239334/NA12878.final.cram
- its chromosome 20 pairs with an insert over 1 kb, cut out below and rehosted: jbrowse.org/…/NA12878.chr20.discordant_pairs.bed.gzhttps://jbrowse.org/demos/read_marks/NA12878.chr20.discordant_pairs.bed.gz
- the release's structural-variant callset: ftp.1000genomes.ebi.ac.uk/…/1KGP_3202.gatksv_svtools_novelins.freeze_V3.wAF.vcf.gzhttps://ftp.1000genomes.ebi.ac.uk/vol1/ftp/data_collections/1000G_2504_high_coverage/working/20210124.SV_Illumina_Integration/1KGP_3202.gatksv_svtools_novelins.freeze_V3.wAF.vcf.gz
- UCSC's GRCh38 cytoband table, rehosted: jbrowse.org/…/cytoBand.txthttps://jbrowse.org/genomes/GRCh38/cytoBand.txt
- a hosted config with the reference, RefSeq genes and every track below but the reads pileup: jbrowse.org/…/config.jsonhttps://jbrowse.org/demos/read_marks/config.json
Loading hg38
The CRAM decodes against the assembly the track is added to, so the assembly must be the GRCh38 sequence the reads were aligned to. The cytoband table draws each chromosome's banding in the view's overview.
Goes in the assemblies array of config.json. See Assemblies.
{
"name": "hg38",
"uri": "https://jbrowse.org/genomes/GRCh38/fasta/hg38.prefix.fa.gz",
"refNameAliases": {
"uri": "https://s3.amazonaws.com/jbrowse.org/genomes/GRCh38/hg38_aliases.txt"
},
"cytobands": "https://jbrowse.org/genomes/GRCh38/cytoBand.txt"
}jbrowse add-assembly https://jbrowse.org/genomes/GRCh38/fasta/hg38.prefix.fa.gz \
--name hg38 \
--refNameAliases https://s3.amazonaws.com/jbrowse.org/genomes/GRCh38/hg38_aliases.txt \
--config '{"cytobands":"https://jbrowse.org/genomes/GRCh38/cytoBand.txt"}'Depth as a coverage step
The window covers 30 kb of an EFCAB8 intron on chromosome 20, where the
callset says NA12878 has one copy of a 3.9 kb deletion. A bar mark over a
coverage transform draws the reads as runs of constant depth.
Goes in the tracks array of config.json. See Tracks.
{
"type": "AlignmentsTrack",
"trackId": "na12878_read_depth",
"name": "NA12878 depth (1000 Genomes, 30x)",
"uri": "https://s3.amazonaws.com/1000genomes/1000G_2504_high_coverage/data/ERR3239334/NA12878.final.cram",
"assemblyNames": ["hg38"],
"displays": [
{
"type": "LinearMarkDisplay",
"scales": { "y": { "title": "Read depth" } },
"marks": [
{
"mark": "bar",
"transform": [{ "type": "coverage" }],
"encoding": { "color": { "value": "#c8d8ee" } }
}
]
}
]
}jbrowse add-track-json '{
"type": "AlignmentsTrack",
"trackId": "na12878_read_depth",
"name": "NA12878 depth (1000 Genomes, 30x)",
"uri": "https://s3.amazonaws.com/1000genomes/1000G_2504_high_coverage/data/ERR3239334/NA12878.final.cram",
"assemblyNames": ["hg38"],
"displays": [
{
"type": "LinearMarkDisplay",
"scales": { "y": { "title": "Read depth" } },
"marks": [
{
"mark": "bar",
"transform": [{ "type": "coverage" }],
"encoding": { "color": { "value": "#c8d8ee" } }
}
]
}
]
}'In JBrowse Desktop, or in any running JBrowse Web session, open a view on this track’s assembly, then File → Open track..., choose Add track from pasted JSON, and paste:
{
"type": "AlignmentsTrack",
"trackId": "na12878_read_depth",
"name": "NA12878 depth (1000 Genomes, 30x)",
"uri": "https://s3.amazonaws.com/1000genomes/1000G_2504_high_coverage/data/ERR3239334/NA12878.final.cram",
"assemblyNames": ["hg38"],
"displays": [
{
"type": "LinearMarkDisplay",
"scales": { "y": { "title": "Read depth" } },
"marks": [
{
"mark": "bar",
"transform": [{ "type": "coverage" }],
"encoding": { "color": { "value": "#c8d8ee" } }
}
]
}
]
}Open the track at chr20:32,925,000-32,955,000.
Insert size as a point per pair
A mark display draws one y axis. Depth runs in the tens and an insert size in the thousands, so the insert goes on a second track over the same file:
- a
pointmark plotstemplate_lengthper pair - a
filterkeeps the leftmost mate, where the template length is positive, so each pair counts once, and drops the few over 8 kb - the colour is mapping quality on a ramp pinned to 0 to 60
Goes in the tracks array of config.json. See Tracks.
{
"type": "AlignmentsTrack",
"trackId": "na12878_read_marks",
"name": "NA12878 insert size (1000 Genomes, 30x)",
"uri": "https://s3.amazonaws.com/1000genomes/1000G_2504_high_coverage/data/ERR3239334/NA12878.final.cram",
"assemblyNames": ["hg38"],
"displays": [
{
"type": "LinearMarkDisplay",
"scales": { "y": { "title": "Insert size (bp)" } },
"marks": [
{
"mark": "point",
"transform": [
{
"type": "filter",
"expr": "jexl:feature.template_length > 0 && feature.template_length < 8000"
}
],
"encoding": {
"y": "template_length",
"color": {
"field": "score",
"scale": "linear",
"domainMin": 0,
"domainMax": 60,
"range": ["#bdbdbd", "#1f4e9a"],
"title": "Mapping quality"
}
}
}
]
}
]
}jbrowse add-track-json '{
"type": "AlignmentsTrack",
"trackId": "na12878_read_marks",
"name": "NA12878 insert size (1000 Genomes, 30x)",
"uri": "https://s3.amazonaws.com/1000genomes/1000G_2504_high_coverage/data/ERR3239334/NA12878.final.cram",
"assemblyNames": ["hg38"],
"displays": [
{
"type": "LinearMarkDisplay",
"scales": { "y": { "title": "Insert size (bp)" } },
"marks": [
{
"mark": "point",
"transform": [
{
"type": "filter",
"expr": "jexl:feature.template_length > 0 && feature.template_length < 8000"
}
],
"encoding": {
"y": "template_length",
"color": {
"field": "score",
"scale": "linear",
"domainMin": 0,
"domainMax": 60,
"range": ["#bdbdbd", "#1f4e9a"],
"title": "Mapping quality"
}
}
}
]
}
]
}'In JBrowse Desktop, or in any running JBrowse Web session, open a view on this track’s assembly, then File → Open track..., choose Add track from pasted JSON, and paste:
{
"type": "AlignmentsTrack",
"trackId": "na12878_read_marks",
"name": "NA12878 insert size (1000 Genomes, 30x)",
"uri": "https://s3.amazonaws.com/1000genomes/1000G_2504_high_coverage/data/ERR3239334/NA12878.final.cram",
"assemblyNames": ["hg38"],
"displays": [
{
"type": "LinearMarkDisplay",
"scales": { "y": { "title": "Insert size (bp)" } },
"marks": [
{
"mark": "point",
"transform": [
{
"type": "filter",
"expr": "jexl:feature.template_length > 0 && feature.template_length < 8000"
}
],
"encoding": {
"y": "template_length",
"color": {
"field": "score",
"scale": "linear",
"domainMin": 0,
"domainMax": 60,
"range": ["#bdbdbd", "#1f4e9a"],
"title": "Mapping quality"
}
}
}
]
}
]
}Open it under the depth track, on the same window.
Each pair in the upper group straddles the missing 3.9 kb. On an alignments
track score is the mapping quality. Hover a point for its values, or click it
to open the read.
Which reads have the long inserts
A span mark over a pileup transform stacks the reads. A formula step
writes the unsigned insert as insert, so both mates of a pair take one colour,
and a ramp pinned at 5 kb paints a spanning pair red.
Goes in the tracks array of config.json. See Tracks.
{
"type": "AlignmentsTrack",
"trackId": "na12878_read_pileup",
"name": "NA12878 reads (1000 Genomes, 30x)",
"uri": "https://s3.amazonaws.com/1000genomes/1000G_2504_high_coverage/data/ERR3239334/NA12878.final.cram",
"assemblyNames": ["hg38"],
"displays": [
{
"type": "LinearMarkDisplay",
"marks": [
{
"mark": "span",
"transform": [
{
"type": "formula",
"expr": "jexl:abs(feature.template_length)",
"as": "insert"
},
{ "type": "pileup" }
],
"encoding": {
"row": "row",
"color": {
"field": "insert",
"scale": "linear",
"domainMin": 0,
"domainMax": 5000,
"range": ["#c8d8ee", "#d62728"],
"title": "Insert size (bp)"
}
}
}
]
}
]
}jbrowse add-track-json '{
"type": "AlignmentsTrack",
"trackId": "na12878_read_pileup",
"name": "NA12878 reads (1000 Genomes, 30x)",
"uri": "https://s3.amazonaws.com/1000genomes/1000G_2504_high_coverage/data/ERR3239334/NA12878.final.cram",
"assemblyNames": ["hg38"],
"displays": [
{
"type": "LinearMarkDisplay",
"marks": [
{
"mark": "span",
"transform": [
{
"type": "formula",
"expr": "jexl:abs(feature.template_length)",
"as": "insert"
},
{ "type": "pileup" }
],
"encoding": {
"row": "row",
"color": {
"field": "insert",
"scale": "linear",
"domainMin": 0,
"domainMax": 5000,
"range": ["#c8d8ee", "#d62728"],
"title": "Insert size (bp)"
}
}
}
]
}
]
}'In JBrowse Desktop, or in any running JBrowse Web session, open a view on this track’s assembly, then File → Open track..., choose Add track from pasted JSON, and paste:
{
"type": "AlignmentsTrack",
"trackId": "na12878_read_pileup",
"name": "NA12878 reads (1000 Genomes, 30x)",
"uri": "https://s3.amazonaws.com/1000genomes/1000G_2504_high_coverage/data/ERR3239334/NA12878.final.cram",
"assemblyNames": ["hg38"],
"displays": [
{
"type": "LinearMarkDisplay",
"marks": [
{
"mark": "span",
"transform": [
{
"type": "formula",
"expr": "jexl:abs(feature.template_length)",
"as": "insert"
},
{ "type": "pileup" }
],
"encoding": {
"row": "row",
"color": {
"field": "insert",
"scale": "linear",
"domainMin": 0,
"domainMax": 5000,
"range": ["#c8d8ee", "#d62728"],
"title": "Insert size (bp)"
}
}
}
]
}
]
}Open the track at the left edge of the dip, chr20:32,936,200-32,939,200.
An unpinned ramp spans the values on screen, so in a window with no spanning
pair it would paint the longest ordinary insert red. Pinning domainMin and
domainMax keeps red for the long inserts.
Scanning chromosome 20 for clusters of long-insert pairs
Fetching every read of a chromosome overruns the byte budget, so cut the long pairs out once, one row per pair, into a BED with a header naming its columns.
# one row per pair with an insert over 1 kb, from the leftmost mate to the
# end of the insert, with the mapping quality in the score column
# -q 20 drops reads the aligner could not place; -F 0x904 drops unmapped,
# secondary and supplementary records
# REF_PATH lets htslib fetch each reference sequence the CRAM names by MD5
export REF_PATH='https://www.ebi.ac.uk/ena/cram/md5/%s'
samtools view -q 20 -F 0x904 --input-fmt-option required_fields=0x1DF NA12878.final.cram chr20 |
awk 'BEGIN { OFS = "\t"; print "#chrom", "chromStart", "chromEnd", "name", "score", "strand", "tlen" }
$7 == "=" && $9 > 1000 { print $3, $4 - 1, $4 - 1 + $9, $1, $5, "+", $9 }' |
bgzip > NA12878.chr20.discordant_pairs.bed.gz
tabix -p bed NA12878.chr20.discordant_pairs.bed.gzThe BED holds few enough rows to fetch whole at any zoom. Two tracks read it:
- a
pointper pair at the middle of its insert,tlenon y, coloured byscore; afilterunder 20 kb keeps the centromere's megabase inserts off the axis - a
barper bin counting pairs of 2 to 10 kb, on an axis pinned at 60 so the centromere saturates and a deletion's few dozen pairs stand up
Goes in the tracks array of config.json. See Tracks.
{
"type": "FeatureTrack",
"trackId": "na12878_chr20_pairs",
"name": "NA12878 chr20, pairs over 1 kb",
"uri": "https://jbrowse.org/demos/read_marks/NA12878.chr20.discordant_pairs.bed.gz",
"assemblyNames": ["hg38"],
"displays": [
{
"type": "LinearMarkDisplay",
"scales": { "y": { "title": "Insert size (bp)" } },
"marks": [
{
"mark": "point",
"transform": [
{ "type": "filter", "expr": "jexl:feature.tlen < 20000" }
],
"encoding": {
"y": "tlen",
"color": {
"field": "score",
"scale": "linear",
"domainMin": 0,
"domainMax": 60,
"range": ["#bdbdbd", "#1f4e9a"],
"title": "Mapping quality"
}
}
}
]
}
]
}jbrowse add-track-json '{
"type": "FeatureTrack",
"trackId": "na12878_chr20_pairs",
"name": "NA12878 chr20, pairs over 1 kb",
"uri": "https://jbrowse.org/demos/read_marks/NA12878.chr20.discordant_pairs.bed.gz",
"assemblyNames": ["hg38"],
"displays": [
{
"type": "LinearMarkDisplay",
"scales": { "y": { "title": "Insert size (bp)" } },
"marks": [
{
"mark": "point",
"transform": [
{ "type": "filter", "expr": "jexl:feature.tlen < 20000" }
],
"encoding": {
"y": "tlen",
"color": {
"field": "score",
"scale": "linear",
"domainMin": 0,
"domainMax": 60,
"range": ["#bdbdbd", "#1f4e9a"],
"title": "Mapping quality"
}
}
}
]
}
]
}'In JBrowse Desktop, or in any running JBrowse Web session, open a view on this track’s assembly, then File → Open track..., choose Add track from pasted JSON, and paste:
{
"type": "FeatureTrack",
"trackId": "na12878_chr20_pairs",
"name": "NA12878 chr20, pairs over 1 kb",
"uri": "https://jbrowse.org/demos/read_marks/NA12878.chr20.discordant_pairs.bed.gz",
"assemblyNames": ["hg38"],
"displays": [
{
"type": "LinearMarkDisplay",
"scales": { "y": { "title": "Insert size (bp)" } },
"marks": [
{
"mark": "point",
"transform": [
{ "type": "filter", "expr": "jexl:feature.tlen < 20000" }
],
"encoding": {
"y": "tlen",
"color": {
"field": "score",
"scale": "linear",
"domainMin": 0,
"domainMax": 60,
"range": ["#bdbdbd", "#1f4e9a"],
"title": "Mapping quality"
}
}
}
]
}
]
}Goes in the tracks array of config.json. See Tracks.
{
"type": "FeatureTrack",
"trackId": "na12878_chr20_pair_counts",
"name": "NA12878 chr20, pairs of 2 to 10 kb per bin",
"uri": "https://jbrowse.org/demos/read_marks/NA12878.chr20.discordant_pairs.bed.gz",
"assemblyNames": ["hg38"],
"displays": [
{
"type": "LinearMarkDisplay",
"scales": {
"y": { "domainMin": 0, "domainMax": 60, "title": "Pairs per bin" }
},
"marks": [
{
"mark": "bar",
"transform": [
{
"type": "filter",
"expr": "jexl:feature.tlen > 2000 && feature.tlen < 10000"
},
{ "type": "bin", "step": "auto" },
{ "type": "aggregate", "ops": [{ "op": "count" }] }
],
"encoding": { "color": { "value": "#d62728" } }
}
]
}
]
}jbrowse add-track-json '{
"type": "FeatureTrack",
"trackId": "na12878_chr20_pair_counts",
"name": "NA12878 chr20, pairs of 2 to 10 kb per bin",
"uri": "https://jbrowse.org/demos/read_marks/NA12878.chr20.discordant_pairs.bed.gz",
"assemblyNames": ["hg38"],
"displays": [
{
"type": "LinearMarkDisplay",
"scales": {
"y": { "domainMin": 0, "domainMax": 60, "title": "Pairs per bin" }
},
"marks": [
{
"mark": "bar",
"transform": [
{
"type": "filter",
"expr": "jexl:feature.tlen > 2000 && feature.tlen < 10000"
},
{ "type": "bin", "step": "auto" },
{ "type": "aggregate", "ops": [{ "op": "count" }] }
],
"encoding": { "color": { "value": "#d62728" } }
}
]
}
]
}'In JBrowse Desktop, or in any running JBrowse Web session, open a view on this track’s assembly, then File → Open track..., choose Add track from pasted JSON, and paste:
{
"type": "FeatureTrack",
"trackId": "na12878_chr20_pair_counts",
"name": "NA12878 chr20, pairs of 2 to 10 kb per bin",
"uri": "https://jbrowse.org/demos/read_marks/NA12878.chr20.discordant_pairs.bed.gz",
"assemblyNames": ["hg38"],
"displays": [
{
"type": "LinearMarkDisplay",
"scales": {
"y": { "domainMin": 0, "domainMax": 60, "title": "Pairs per bin" }
},
"marks": [
{
"mark": "bar",
"transform": [
{
"type": "filter",
"expr": "jexl:feature.tlen > 2000 && feature.tlen < 10000"
},
{ "type": "bin", "step": "auto" },
{ "type": "aggregate", "ops": [{ "op": "count" }] }
],
"encoding": { "color": { "value": "#d62728" } }
}
]
}
]
}In the counts track the aggregate step names no groupby, so it groups by the
edges the bin step above it wrote.
Open both tracks on the whole of chr20.
The bar at 34.2 Mb is a homozygous deletion, and the one at 32.9 Mb is the EFCAB8 intron above.
Checking the bars against the callset
bcftools lists every deletion over 2 kb that the callset gives NA12878 on the
chromosome:
# -s keeps one sample's genotypes; -i then keeps the rows where that sample
# carries the allele
bcftools view -s NA12878 1KGP_3202.gatksv_svtools_novelins.freeze_V3.wAF.vcf.gz chr20 |
bcftools query -i 'GT="alt" && INFO/SVTYPE="DEL" && INFO/SVLEN<-2000' \
-f '%CHROM\t%POS\t%END\t%INFO/SVLEN\t[%GT]\t%INFO/AF\t%INFO/EVIDENCE\n'Every deletion in that listing between 2 and 10 kb is a bar on the track, and the two homozygous ones are tallest. Other windows hold ten or more such pairs with no call: the chromosome start and 1.4, 2.8, 32.7 and 48.5 Mb.
To check the EFCAB8 deletion against the reads, compare the depth inside the call with the depth beside it, and count the long pairs around it:
samtools coverage -r chr20:32937680-32941583 NA12878.final.cram | cut -f 1-3,7
samtools coverage -r chr20:32930000-32937000 NA12878.final.cram | cut -f 1-3,7
samtools view -q 20 NA12878.final.cram chr20:32935000-32944000 |
awk '{ t = $9 < 0 ? -$9 : $9; if (t > 2000) big++; else if (t > 0) norm++ }
END { print norm " pairs at the library insert, " big " over 2 kb" }'Depth inside the call comes out about half the depth beside it, as a heterozygous deletion predicts. Around the call, almost every pair sits at the library insert, and a few dozen exceed 2 kb.
Reproduce it end to end
build_read_marks.sh
runs the commands above:
curl -fO https://raw.githubusercontent.com/GMOD/jbrowse-components/main/scripts/build_read_marks.sh
bash build_read_marks.sh # builds ./read_marks_build/jbrowse2
npx --yes serve read_marks_build/jbrowse2 # then open the printed URLWith no arguments the script builds the depth, insert-size and chromosome-scan
tracks over NA12878. Given your own reads,
bash build_read_marks.sh reads.cram genome.fa builds them over your file, and
CHROM picks the chromosome to scan.
See also
- Mark display
- Methylation (long-read)
- A grammar of graphics over a BED (RepeatMasker Alu age)
- Low-mappability regions (SMN)
- Structural variants (1000 Genomes)
- JBrowse web quick start
Citations
- Byrska-Bishop M, et al. High-coverage whole-genome sequencing of the expanded 1000 Genomes Project cohort including 602 trios. Cell 185:3426-3440 (2022), the reads and the structural-variant callset.
- Li H, et al. The Sequence Alignment/Map format and SAMtools. Bioinformatics 25:2078-2079 (2009), where the template length and mapping quality fields are defined.
Feedback on this tutorial is welcome: contact us.