RepeatMasker as one row per class

JBrowse v5 beta

The tutorials target the JBrowse v5 beta, and the v4.3.0 release on the download page lacks some of what they show. To install the beta of JBrowse Web, run npm install -g @jbrowse/cli@next, then jbrowse create jbrowse2 --branch v5.0.0-beta.17. Desktop beta builds are coming soon.

Repeat classes spread along a genome differently: over a window of 17q21, SINE copies fill it end to end while LINE copies come in clusters. A RepeatMasker track draws every class in one packed row of colored blocks, where that difference is hard to read. We:

switch hg38's RepeatMasker track to a multi-row feature display, one labelled row per class
pin the row order and colors in a track config
serve your own RepeatMasker output the same way

JBrowse reads the classes from the file, so no data preparation is needed. Every genome at genomes.jbrowse.org, and any UCSC or GenArk hub config, has a RepeatMasker track to try.

Prerequisites

a JBrowse to paste the tracks into (Web or Desktop)
htslib (bgzip, tabix), for indexing your own output
samtools, for serving your own RepeatMasker output and Reproduce it end to end, which runs the same conversion
node, for the JBrowse CLI, which both of those need too

# Debian/Ubuntu
sudo apt install samtools tabix nodejs
# or conda, which has every tool above
conda install -c bioconda -c conda-forge samtools htslib nodejs
# any platform
npm install -g @jbrowse/cli

The data

The figures read UCSC's RepeatMasker track for hg38 and dm6, rehosted on jbrowse.org.

Nothing to download: the track configs read their files by URL. The build script takes a genome of your own, as its FASTA and RepeatMasker's .out.

Where UCSC and GenArk hubs store the repeat class

The multi-row display's rows assigns one row per value of a field. UCSC golden-path and GenArk hubs store the repeat class in different places:

A UCSC golden-path assembly ships a BED whose header names its columns, repClass among them. That column is a feature attribute, so rows is "repClass".
A GenArk assembly ships a bigRmskBed, whose autoSql has no class column. The name contains the class as a suffix, L1HS#LINE/L1, so rows derives it with a jexl expression, "jexl:split(split(feature.name,'#')[1],'/')[0]"; Multi-row feature track works through it.

In both cases JBrowse builds the rows from the values in the window, so a window with no satellite repeats has no satellite row.

Switching hg38's RepeatMasker track to one row per class

We'll open hg38 on genomes.jbrowse.org and switch the track:

Turn on RepeatMasker and type chr17:45,700,000-45,750,000 into the location box.
Display types → Multi-row feature display (painting) in the track menu.

The display partitions on repClass whenever the file has that column, and One row per... in the same menu lists every column the loaded features have, repFamily among them.

Top: the track menu's Display types submenu on the UCSC RepeatMasker track; any feature track offers the multi-row display beside its default one. Bottom: the same window after picking it, one row per repeat class.

The RepeatMasker track from one packed row to a labelled row per class: the track menu's Display types, and the multi-row painting partitioning on the repeat class column in the file.

Video

Open the session the video starts in ↗, then take the steps it shows:

Every repeat in one packed lane, whatever its class
Redraw the lane as a painting from the track menu
A labelled lane per class, LINE to SINE, found in the file itself

Steps

Open this view in JBrowse Web ↗ ·

Open in JBrowse Desktop ↗

Needs JBrowse Desktop 5.0 or newer. If nothing opens, copy the link and paste it into Desktop's Open JBrowse Web link..., on the start screen or under File → Session.

https://jbrowse.org/code/jb2/main/?config=test_data%2Fconfig_demo.json&session=spec-%7B%22sessionTracks%22%3A%5B%7B%22type%22%3A%22FeatureTrack%22%2C%22trackId%22%3A%22rmsk_hg38_ucsc%22%2C%22name%22%3A%22RepeatMasker%22%2C%22assemblyNames%22%3A%5B%22hg38%22%5D%2C%22adapter%22%3A%7B%22type%22%3A%22BedTabixAdapter%22%2C%22bedGzLocation%22%3A%7B%22uri%22%3A%22https%3A%2F%2Fjbrowse.org%2Fucsc%2Fhg38%2Frmsk.bed.gz%22%2C%22locationType%22%3A%22UriLocation%22%7D%2C%22index%22%3A%7B%22indexType%22%3A%22CSI%22%2C%22location%22%3A%7B%22uri%22%3A%22https%3A%2F%2Fjbrowse.org%2Fucsc%2Fhg38%2Frmsk.bed.gz.csi%22%2C%22locationType%22%3A%22UriLocation%22%7D%7D%7D%2C%22displays%22%3A%5B%7B%22type%22%3A%22LinearBasicDisplay%22%2C%22displayId%22%3A%22rmsk_hg38_ucsc-LinearBasicDisplay%22%7D%2C%7B%22type%22%3A%22LinearMultiRowFeatureDisplay%22%2C%22displayId%22%3A%22rmsk_hg38_ucsc-LinearMultiRowFeatureDisplay%22%2C%22height%22%3A260%7D%5D%7D%5D%2C%22views%22%3A%5B%7B%22type%22%3A%22LinearGenomeView%22%2C%22assembly%22%3A%22hg38%22%2C%22loc%22%3A%22chr17%3A45%2C700%2C000-45%2C750%2C000%22%2C%22tracks%22%3A%5B%22rmsk_hg38_ucsc%22%5D%7D%5D%7D&sessionName=JBrowse+docs%3A+repeats%2Fpainting_display_switch

The short route: open the view in JBrowse Web, where File → Open track... adds your own files beside its hosted genome and tracks.

From scratch with your own data, the steps behind the figure:

Open your genome: on the JBrowse Desktop start screen click Open new genome (or Show all available genomes to pick a hosted one). For JBrowse Web, jbrowse add-assembly puts it in your config.json.This figure uses hg38, loaded from a bgzip-compressed FASTA (.fa.gz + .fai + .gzi).
Add a track: check one your config has in the track selector, or File → Open track... to paste a URL or choose a local file. This one needs a bgzip-compressed, tabix-indexed BED (.bed.gz + .tbi).This figure uses “RepeatMasker”.
Type your region of interest into the location box and press Enter.This figure is at chr17:45,700,000-45,750,000.

CLI

A figure adds its tracks to one session. The jbrowse CLI writes the same tracks into a config.json instead, where every session that opens it has them: add-track where flags cover the whole track, add-track-json where they do not.

jbrowse add-track-json '{
  "type": "FeatureTrack",
  "trackId": "rmsk_hg38_ucsc",
  "name": "RepeatMasker",
  "assemblyNames": [
    "hg38"
  ],
  "adapter": {
    "type": "BedTabixAdapter",
    "bedGzLocation": {
      "uri": "https://jbrowse.org/ucsc/hg38/rmsk.bed.gz",
      "locationType": "UriLocation"
    },
    "index": {
      "indexType": "CSI",
      "location": {
        "uri": "https://jbrowse.org/ucsc/hg38/rmsk.bed.gz.csi",
        "locationType": "UriLocation"
      }
    }
  },
  "displays": [
    {
      "type": "LinearBasicDisplay",
      "displayId": "rmsk_hg38_ucsc-LinearBasicDisplay"
    },
    {
      "type": "LinearMultiRowFeatureDisplay",
      "displayId": "rmsk_hg38_ucsc-LinearMultiRowFeatureDisplay",
      "height": 260
    }
  ]
}'

Run these where the config.json is, or add --out <dir>, and point each uri at your own file. The config needs hg38 already — jbrowse add-assembly adds yours. The location and the settings the steps carry are session state rather than track config — that half is the Spec tab, or jbrowse set-default-session.

Spec

This session spec draws the figure. It goes after &session=spec- on a JBrowse Web link that loads https://jbrowse.org/code/jb2/main/test_data/config_demo.json, which is what the Web button above opens.

{
  "sessionTracks": [
    {
      "type": "FeatureTrack",
      "trackId": "rmsk_hg38_ucsc",
      "name": "RepeatMasker",
      "assemblyNames": [
        "hg38"
      ],
      "adapter": {
        "type": "BedTabixAdapter",
        "bedGzLocation": {
          "uri": "https://jbrowse.org/ucsc/hg38/rmsk.bed.gz",
          "locationType": "UriLocation"
        },
        "index": {
          "indexType": "CSI",
          "location": {
            "uri": "https://jbrowse.org/ucsc/hg38/rmsk.bed.gz.csi",
            "locationType": "UriLocation"
          }
        }
      },
      "displays": [
        {
          "type": "LinearBasicDisplay",
          "displayId": "rmsk_hg38_ucsc-LinearBasicDisplay"
        },
        {
          "type": "LinearMultiRowFeatureDisplay",
          "displayId": "rmsk_hg38_ucsc-LinearMultiRowFeatureDisplay",
          "height": 260
        }
      ]
    }
  ],
  "views": [
    {
      "type": "LinearGenomeView",
      "assembly": "hg38",
      "loc": "chr17:45,700,000-45,750,000",
      "tracks": [
        "rmsk_hg38_ucsc"
      ]
    }
  ]
}

Notebook

The same view with jbrowse-anywidget, from the config the figure loads. Swap a track's uri for your own file.

from jbrowse_anywidget import JBrowseApp, fetch_hub

config = fetch_hub("https://jbrowse.org/code/jb2/main/test_data/config_demo.json")
app = JBrowseApp(
    assemblies=config["assemblies"],
    tracks=config.get("tracks", []) + [
        {
            "type": "FeatureTrack",
            "trackId": "rmsk_hg38_ucsc",
            "name": "RepeatMasker",
            "assemblyNames": [
                "hg38"
            ],
            "adapter": {
                "type": "BedTabixAdapter",
                "bedGzLocation": {
                    "uri": "https://jbrowse.org/ucsc/hg38/rmsk.bed.gz",
                    "locationType": "UriLocation"
                },
                "index": {
                    "indexType": "CSI",
                    "location": {
                        "uri": "https://jbrowse.org/ucsc/hg38/rmsk.bed.gz.csi",
                        "locationType": "UriLocation"
                    }
                }
            },
            "displays": [
                {
                    "type": "LinearBasicDisplay",
                    "displayId": "rmsk_hg38_ucsc-LinearBasicDisplay"
                },
                {
                    "type": "LinearMultiRowFeatureDisplay",
                    "displayId": "rmsk_hg38_ucsc-LinearMultiRowFeatureDisplay",
                    "height": 260
                }
            ]
        }
    ],
    views=[
        {
            "type": "LinearGenomeView",
            "assembly": "hg38",
            "loc": "chr17:45,700,000-45,750,000",
            "tracks": [
                "rmsk_hg38_ucsc"
            ]
        }
    ],
)
app  # display the widget

Image

jbrowse-img draws the same view as an SVG, from the command line and without a browser. Point --config at your own config, or swap a track for your own file: a data file opens as the track its name says.

cat > tracks.json <<'JSON'
[
  {
    "type": "FeatureTrack",
    "trackId": "rmsk_hg38_ucsc",
    "name": "RepeatMasker",
    "assemblyNames": [
      "hg38"
    ],
    "adapter": {
      "type": "BedTabixAdapter",
      "bedGzLocation": {
        "uri": "https://jbrowse.org/ucsc/hg38/rmsk.bed.gz",
        "locationType": "UriLocation"
      },
      "index": {
        "indexType": "CSI",
        "location": {
          "uri": "https://jbrowse.org/ucsc/hg38/rmsk.bed.gz.csi",
          "locationType": "UriLocation"
        }
      }
    },
    "displays": [
      {
        "type": "LinearBasicDisplay",
        "displayId": "rmsk_hg38_ucsc-LinearBasicDisplay"
      },
      {
        "type": "LinearMultiRowFeatureDisplay",
        "displayId": "rmsk_hg38_ucsc-LinearMultiRowFeatureDisplay",
        "height": 260
      }
    ]
  }
]
JSON

npx @jbrowse/img --config https://jbrowse.org/code/jb2/main/test_data/config_demo.json \
  --tracks tracks.json \
  --assembly hg38 --loc chr17:45,700,000-45,750,000 \
  --track rmsk_hg38_ucsc \
  --width 1920 --out figure.svg

--out figure.png writes a PNG instead, through rsvg-convert (apt install librsvg2-bin, brew install librsvg).

Agent

Hand this to a coding agent. It rebuilds the figure above headlessly, and the session file is then the thing to edit: change the location or a track's settings, or add your own track under sessionTracks, and rerun.

cat > session.json <<'JSON'
{
  "sessionTracks": [
    {
      "type": "FeatureTrack",
      "trackId": "rmsk_hg38_ucsc",
      "name": "RepeatMasker",
      "assemblyNames": [
        "hg38"
      ],
      "adapter": {
        "type": "BedTabixAdapter",
        "bedGzLocation": {
          "uri": "https://jbrowse.org/ucsc/hg38/rmsk.bed.gz",
          "locationType": "UriLocation"
        },
        "index": {
          "indexType": "CSI",
          "location": {
            "uri": "https://jbrowse.org/ucsc/hg38/rmsk.bed.gz.csi",
            "locationType": "UriLocation"
          }
        }
      },
      "displays": [
        {
          "type": "LinearBasicDisplay",
          "displayId": "rmsk_hg38_ucsc-LinearBasicDisplay"
        },
        {
          "type": "LinearMultiRowFeatureDisplay",
          "displayId": "rmsk_hg38_ucsc-LinearMultiRowFeatureDisplay",
          "height": 260
        }
      ]
    }
  ],
  "views": [
    {
      "type": "LinearGenomeView",
      "assembly": "hg38",
      "loc": "chr17:45,700,000-45,750,000",
      "tracks": [
        "rmsk_hg38_ucsc"
      ]
    }
  ]
}
JSON

npx @jbrowse/capture --instance https://jbrowse.org/code/jb2/main/ \
  --config https://jbrowse.org/code/jb2/main/test_data/config_demo.json \
  --width 1920 --height 592 \
  --spec session.json -o figure.png

Using JBrowse with AI agents covers the rest: hosted genomes to point it at, and why waiting for the render is the part that goes wrong.

Top: RepeatMasker colored by repClass in one packed row. Bottom: the same track with one row per repClass.

Pinning row order and colors in a track config

A track config can open the track in the partitioned view, and two settings keep the rows stable as the window's class list changes. rows.sort lists the classes that come first, in order, and a class it leaves out follows the listed ones. rowColor is a color scale on each row's label bar, pairing each class in its domain with a color in its range.

Config file

Goes in the tracks array of config.json. See Tracks.

{
  "type": "FeatureTrack",
  "trackId": "rmsk_hg38_rows",
  "name": "RepeatMasker by class",
  "uri": "rmsk.bed.gz",
  "assemblyNames": ["hg38"],
  "displays": [
    {
      "type": "LinearMultiRowFeatureDisplay",
      "rows": {
        "field": "repClass",
        "sort": [
          "SINE",
          "LINE",
          "LTR",
          "DNA",
          "Simple_repeat",
          "Low_complexity"
        ]
      },
      "rowColor": {
        "scale": {
          "domain": [
            "SINE",
            "LINE",
            "LTR",
            "DNA",
            "Simple_repeat",
            "Low_complexity"
          ],
          "range": [
            "#e41a1c",
            "#377eb8",
            "#4daf4a",
            "#984ea3",
            "#ff7f00",
            "#a65628"
          ]
        }
      },
      "axis": { "grid": true }
    }
  ]
}

CLI (add-track-json)

jbrowse add-track-json '{
  "type": "FeatureTrack",
  "trackId": "rmsk_hg38_rows",
  "name": "RepeatMasker by class",
  "uri": "rmsk.bed.gz",
  "assemblyNames": ["hg38"],
  "displays": [
    {
      "type": "LinearMultiRowFeatureDisplay",
      "rows": {
        "field": "repClass",
        "sort": [
          "SINE",
          "LINE",
          "LTR",
          "DNA",
          "Simple_repeat",
          "Low_complexity"
        ]
      },
      "rowColor": {
        "scale": {
          "domain": [
            "SINE",
            "LINE",
            "LTR",
            "DNA",
            "Simple_repeat",
            "Low_complexity"
          ],
          "range": [
            "#e41a1c",
            "#377eb8",
            "#4daf4a",
            "#984ea3",
            "#ff7f00",
            "#a65628"
          ]
        }
      },
      "axis": { "grid": true }
    }
  ]
}'

In the app

In JBrowse Desktop, or in any running JBrowse Web session, open a view on this track’s assembly, then File → Open track..., choose Add track from pasted JSON, and paste:

{
  "type": "FeatureTrack",
  "trackId": "rmsk_hg38_rows",
  "name": "RepeatMasker by class",
  "uri": "rmsk.bed.gz",
  "assemblyNames": ["hg38"],
  "displays": [
    {
      "type": "LinearMultiRowFeatureDisplay",
      "rows": {
        "field": "repClass",
        "sort": [
          "SINE",
          "LINE",
          "LTR",
          "DNA",
          "Simple_repeat",
          "Low_complexity"
        ]
      },
      "rowColor": {
        "scale": {
          "domain": [
            "SINE",
            "LINE",
            "LTR",
            "DNA",
            "Simple_repeat",
            "Low_complexity"
          ],
          "range": [
            "#e41a1c",
            "#377eb8",
            "#4daf4a",
            "#984ea3",
            "#ff7f00",
            "#a65628"
          ]
        }
      },
      "axis": { "grid": true }
    }
  ]
}

rmsk.bed.gz is relative to a config.json. Replace it with its URL or its path on this computer.

A row not named in rowColor paints in the default block color. unknown in rowColor sets one color for every unnamed row.

The track opens with the first display listed, so the multi-row entry makes rows the default. A bare { "type": "LinearBasicDisplay" } ahead of it keeps the packed form as the default and leaves the rows one menu click away.

The rows come from the classes in the file, so a class rowColor leaves out still gets a row, as Unknown does in the figures.

Serving your own RepeatMasker output

A hub's RepeatMasker BED has the repClass column the display partitions on. The RepeatMasker .out file instead writes a single class/family field, LINE/L1 for a repeat with both and a bare Simple_repeat for one whose family is its class. Splitting that field in two, under a header naming the columns, turns the .out into the file the track above reads:

# the header names the columns, so `rows: "repClass"` can name a column
# outside the BED spec
{
  printf '#genoName\tgenoStart\tgenoEnd\tname\tstrand\trepFamily\trepClass\tswScore\tmilliDiv\n'
  awk 'BEGIN { OFS = "\t" }
    # a data row is the one starting with a bare integer, its SW score
    $1 ~ /^[0-9]+$/ {
      # "LINE/L1" splits; "Simple_repeat" does not, and UCSC repeats the class
      # as the family for exactly those rows. Positions are 1-based inclusive
      # and the strand column spells minus "C".
      n = split($11, cf, "/")
      print $5, $6 - 1, $7, $10, ($9 == "C") ? "-" : "+", (n > 1) ? cf[2] : cf[1], cf[1], $1, int($2 * 10 + 0.5)
    }' repeats.out
} > rmsk.bed

# `sort-bed` is `sort -k1,1 -k2,2n` under LC_ALL=C, header kept on top
jbrowse sort-bed rmsk.bed | bgzip > rmsk.bed.gz
tabix -p bed rmsk.bed.gz

Reproduce it end to end

build_repeatmasker_classes.sh turns your RepeatMasker .out into a runnable JBrowse. The script:

Splits the .out's class/family field into a class column and a family column.
Writes a BED whose header names the columns UCSC's does, then bgzips and indexes it.
Downloads JBrowse and adds the track with the multi-row display selected.

curl -fO https://raw.githubusercontent.com/GMOD/jbrowse-components/main/scripts/build_repeatmasker_classes.sh
bash build_repeatmasker_classes.sh genome.fa repeats.out  # builds ./repeatmasker_build/jbrowse2
npx --yes serve repeatmasker_build/jbrowse2               # then open the printed URL

genome.fa is the FASTA RepeatMasker was run against and repeats.out is its .out; either may be gzipped.

RepeatMasker as one row per class

Prerequisites#

The data#

Where UCSC and GenArk hubs store the repeat class#

Switching hg38's RepeatMasker track to one row per class#

Pinning row order and colors in a track config#

Serving your own RepeatMasker output#

Reproduce it end to end#

See also#