RepeatMasker as one row per class
The tutorials target the JBrowse v5 beta, and the v4.3.0 release
on the download page lacks some of what they show. To install the
beta of JBrowse Web, run npm install -g @jbrowse/cli@next, then
jbrowse create jbrowse2 --branch v5.0.0-beta.13. Desktop beta builds are coming soon.
Repeat classes spread along a genome differently: over a window of 17q21, SINE copies fill it end to end while LINE copies come in clusters. A RepeatMasker track draws every class in one packed row of colored blocks, where that difference is hard to read. The same file opened as a multi-row feature display draws one labelled row per class.
JBrowse reads the classes from the file, so no data preparation is needed. Every genome at genomes.jbrowse.org, and any UCSC or GenArk hub config, has a RepeatMasker track to try.
Prerequisites
- a JBrowse to paste the tracks into (Web or Desktop)
- htslib (
bgzip,tabix,htsfile), for the check at the end samtools, for serving your own RepeatMasker output and Reproduce it end to end, which runs the same conversionnode, for the JBrowse CLI, which both of those need too
Where the data comes from
The figures read UCSC's RepeatMasker track for hg38 and dm6, rehosted on jbrowse.org.
Nothing to download: the track configs and the commands below read these files
by URL. The build script takes a genome of your own,
as its FASTA and RepeatMasker's .out.
The files
- hg38, read by the tabix command under Checking the rows against the file: https://jbrowse.org/ucsc/hg38/rmsk.bed.gz
- dm6, the file the diff under Reproduce it end to end checks a home-built conversion against: https://jbrowse.org/ucsc/dm6/rmsk.bed.gz
Where UCSC and GenArk hubs store the repeat class
The multi-row display's rows assigns one row per value of a field. UCSC
golden-path and GenArk hubs store the repeat class in different places:
- A UCSC golden-path assembly ships a BED whose header names its columns,
repClassamong them. That column is a feature attribute, sorowsis"repClass". - A GenArk assembly ships a
bigRmskBed, whose autoSql has no class column. The name contains the class as a suffix,L1HS#LINE/L1, sorowsderives it with a jexl expression,"jexl:split(split(feature.name,'#')[1],'/')[0]"; Multi-row feature track works through it.
In both cases JBrowse builds the rows from the values in the loaded region, so a window with no satellite repeats has no satellite row.
Switching hg38's RepeatMasker track to one row per class
We'll open hg38 on genomes.jbrowse.org and switch the track:
- Turn on RepeatMasker and type
chr17:45,700,000-45,750,000into the location box. - Display types → Multi-row feature display (painting) in the track menu.
The display partitions on repClass whenever the file has that column, and
One row per... in the same menu lists every column the loaded features have,
repFamily among them.
Pinning row order and colors in a track config
A track config can open the track in the partitioned view, and two settings keep
the rows stable as the window's class list changes. rows.domain lists the
classes that come first, in order, and a class it leaves out follows the listed
ones. rowColor is a color scale on each row's label bar, pairing each class in
its domain with a color in its range.
Goes in the tracks array of config.json. See Tracks.
{
"type": "FeatureTrack",
"trackId": "rmsk_hg38_rows",
"name": "RepeatMasker by class",
"uri": "rmsk.bed.gz",
"assemblyNames": ["hg38"],
"displays": [
{
"type": "LinearMultiRowFeatureDisplay",
"rows": {
"field": "repClass",
"domain": [
"SINE",
"LINE",
"LTR",
"DNA",
"Simple_repeat",
"Low_complexity"
]
},
"rowColor": {
"domain": [
"SINE",
"LINE",
"LTR",
"DNA",
"Simple_repeat",
"Low_complexity"
],
"range": [
"#e41a1c",
"#377eb8",
"#4daf4a",
"#984ea3",
"#ff7f00",
"#a65628"
]
},
"showRowSeparators": true
}
]
}jbrowse add-track-json '{
"type": "FeatureTrack",
"trackId": "rmsk_hg38_rows",
"name": "RepeatMasker by class",
"uri": "rmsk.bed.gz",
"assemblyNames": ["hg38"],
"displays": [
{
"type": "LinearMultiRowFeatureDisplay",
"rows": {
"field": "repClass",
"domain": [
"SINE",
"LINE",
"LTR",
"DNA",
"Simple_repeat",
"Low_complexity"
]
},
"rowColor": {
"domain": [
"SINE",
"LINE",
"LTR",
"DNA",
"Simple_repeat",
"Low_complexity"
],
"range": [
"#e41a1c",
"#377eb8",
"#4daf4a",
"#984ea3",
"#ff7f00",
"#a65628"
]
},
"showRowSeparators": true
}
]
}'In JBrowse Desktop, or in any running JBrowse Web session, open a view on this track’s assembly, then File → Open track..., choose Add track from pasted JSON, and paste:
{
"type": "FeatureTrack",
"trackId": "rmsk_hg38_rows",
"name": "RepeatMasker by class",
"uri": "rmsk.bed.gz",
"assemblyNames": ["hg38"],
"displays": [
{
"type": "LinearMultiRowFeatureDisplay",
"rows": {
"field": "repClass",
"domain": [
"SINE",
"LINE",
"LTR",
"DNA",
"Simple_repeat",
"Low_complexity"
]
},
"rowColor": {
"domain": [
"SINE",
"LINE",
"LTR",
"DNA",
"Simple_repeat",
"Low_complexity"
],
"range": [
"#e41a1c",
"#377eb8",
"#4daf4a",
"#984ea3",
"#ff7f00",
"#a65628"
]
},
"showRowSeparators": true
}
]
}rmsk.bed.gz is relative to a config.json. Replace it with its URL or its path on this computer.
A row not named in rowColor paints in the default block color. Setting
unknown in rowColor gives every unnamed row one color.
The track opens with the first display listed, so the multi-row entry makes rows
the default. Putting a bare { "type": "LinearBasicDisplay" } ahead of it keeps
the packed form as the default and leaves the rows one menu click away.
Checking the rows against the RepeatMasker file
To check the rows, count the classes in the file over the figures' window:
tabix https://jbrowse.org/ucsc/hg38/rmsk.bed.gz chr17:45,700,000-45,750,000 |
awk -F'\t' '{ n[$7]++; bp[$7] += $3 - $2 }
END { for (c in n) printf "%s\t%d\t%d bp\n", c, n[c], bp[c] }' |
sort -k3 -nrThe printed output lines up with the rows:
- Each printed class is a row with features on screen, and its bp total is the area drawn in that row.
- JBrowse fetches past the view's edges and builds rows from everything it
fetched, so an empty row can come from a repeat just outside the window: the
figures' empty
LTR?row isMamRep605, past the right edge. - A printed class with no row, or a row with features and no printed line, means the view is showing a different file.
The Unknown row is the control. Neither the rowColor above nor the
cookbook's color lookup names it, and it appears
because the rows come from the file. Pan to a window whose output has no
Unknown line and the row goes away.
The same command with $6 in place of $7 counts repFamily, the finer
partition (L1, Alu, MIR).
Serving your own RepeatMasker output
A hub's RepeatMasker BED has the repClass column the display partitions on.
The RepeatMasker .out file instead writes a single class/family field,
LINE/L1 for a repeat with both and a bare Simple_repeat for one whose family
is its class. Splitting that field in two, under a header naming the columns,
turns the .out into the file the track above reads:
# the header names the columns, so `rows: "repClass"` can name a column
# outside the BED spec
{
printf '#genoName\tgenoStart\tgenoEnd\tname\tstrand\trepFamily\trepClass\tswScore\tmilliDiv\n'
awk 'BEGIN { OFS = "\t" }
# a data row is the one starting with a bare integer, its SW score
$1 ~ /^[0-9]+$/ {
# "LINE/L1" splits; "Simple_repeat" does not, and UCSC repeats the class
# as the family for exactly those rows. Positions are 1-based inclusive
# and the strand column spells minus "C".
n = split($11, cf, "/")
print $5, $6 - 1, $7, $10, ($9 == "C") ? "-" : "+", (n > 1) ? cf[2] : cf[1], cf[1], $1, int($2 * 10 + 0.5)
}' repeats.out
} > rmsk.bed
# `sort-bed` is `sort -k1,1 -k2,2n` under LC_ALL=C, header kept on top
jbrowse sort-bed rmsk.bed | bgzip > rmsk.bed.gz
tabix -p bed rmsk.bed.gzThe bed's first seven columns match UCSC's, so the tabix | awk check above
reads the result unchanged.
Reproduce it end to end
One script converts the .out, indexes it, and writes a runnable JBrowse with
the track already set to the multi-row display,
build_repeatmasker_classes.sh:
curl -fO https://raw.githubusercontent.com/GMOD/jbrowse-components/main/scripts/build_repeatmasker_classes.sh
bash build_repeatmasker_classes.sh genome.fa repeats.out # builds ./repeatmasker_build/jbrowse2
npx --yes serve repeatmasker_build/jbrowse2 # then open the printed URLgenome.fa is the FASTA RepeatMasker was run against and repeats.out is its
.out; either may be gzipped.
On a genome UCSC also masks, compare the output with the UCSC conversion:
curl -o ucsc_rmsk.bed.gz https://jbrowse.org/ucsc/dm6/rmsk.bed.gz
diff <(gzip -dc repeatmasker_build/rmsk.bed.gz | grep -v '^#' | cut -f1-7 | sort) \
<(gzip -dc ucsc_rmsk.bed.gz | grep -v '^#' | cut -f1-7 | sort)No output means every interval, name, strand, family and class agrees with the
UCSC conversion. A disagreement usually traces to the .out quirks the awk
comments name.
See also
- Tutorial: Basic usage of genomes.jbrowse.org
- Tutorial: ChromHMM chromatin states
- User guide: Multi-row feature track
- Docs: Cookbook
Feedback on this tutorial is welcome: contact us.