JBrowse REST API and Data APIs
Writing JBrowse-compatible Web Services
Beginning in version 1.9.0, JBrowse ships with a REST data store adapter (JBrowse/Store/SeqFeature/REST) that can provide feature, sequence, and quantitative data for display by any of JBrowse's track types. To use it, a developer can implement server-side web services that satisfy the REST API it expects.
JBrowse version 1.11.0 added a REST adaptor that can look up names and name prefixes (for type-ahead completion) from REST endpoints as well (see JBrowse REST Names API below).
JBrowse REST Feature Store API
The JBrowse REST feature store requires the following server resources.
GET (base)/stats/global
Required. Returns a JSON object containing global statistics about the features served by this store.
Example:
{
"featureDensity": 0.02,
"featureCount": 234235,
"scoreMin": 87,
"scoreMax": 87,
"scoreMean": 42,
"scoreStdDev": 2.1
}
None of the attributes in the example above are required to be present. However, if the store is primarily providing positional data (such as genes), it is recommended to provide at least featureDensity (average number of features per basepair), since JBrowse uses this metric to make many decisions about how to display features. For stores that primarily provide quantitative data, it is recommended to also provide score statistics.
GET (base)/stats/region/(refseq_name)?start=123&end=456
Optional, but recommended. Get statistics for a particular region. Returns the same format as stats/global above, but with statistics that apply only to the region specified.
The refseq name URL component, and the start and end query parameters specify the region of interest. Statistics should be calculated for all features that overlap the region in question. start and end are in interbase coordinates.
NOTE: If this is not implemented, the statistics will be calculated as needed by actually fetching feature data for the region in question. If your backend *does* implement region stats, set "region_stats": true in the track or store configuration to have JBrowse use them.
GET (base)/stats/regionFeatureDensities/(refseq_name)?start=123&end=456&basesPerBin=20000
Optional, added in JBrowse 1.10.7. Get binned feature counts for a certain region, which are used only by HTMLFeatures tracks to draw density histograms at certain zoom levels. If your backend implements this endpoint, set "region_feature_densities": true in the track or store configuration to have JBrowse use it.
The refseq name URL component, and the start and end query parameters specify the region of interest. start and end are in interbase coordinates.
The basesPerBin is an integer which must be used to determine the number of bins - the endpoint may not choose its own bin size. max should be the maximum value for the density, the global maximum for the entire track.
Example returned JSON:
{
"bins": [ 51, 50, 58, 63, 57, 57, 65, 66, 63, 61,
56, 49, 50, 47, 39, 38, 54, 41, 50, 71,
61, 44, 64, 60, 42
],
"stats": {
"basesPerBin": 200,
"max": 88
}
}
Note that the stats.max attribute sets that Y-axis scale for the entire track, so should probably be set according to the global (or nearly global) max count for bins of that size.
GET (base)/features/(refseq_name)?start=234&end=5678
Required. Fetch feature data (including quantitative data) for the specified region.
The refseq name URL component, and the start and end query parameters specify the region of interest. All features that overlap the region in question should be returned. start and end are in interbase coordinates. Also, track types that display features as boxes laid out on the genome (such as HTMLFeatures, CanvasFeatures, Alignments, and Alignments2 ) require all top-level features to have a globally-unique ID, which should be set as uniqueID in the JSON emitted by the service. uniqueID can be any string or number that is guaranteed to be unique among the features being emitted by this query. It is never shown to the user.
Example return JSON:
{
"features": [
/* minimal required data */
{ "start": 123, "end": 456 },
/* typical quantitative data */
{ "start": 123, "end": 456, "score": 42 },
/* Expected format of the single feature expected when the track is a sequence data track. */
{"seq": "gattacagattaca", "start": 0, "end": 14},
/* typical processed transcript with subfeatures */
{ "type": "mRNA", "start": 5975, "end": 9744, "score": 0.84, "strand": 1,
"name": "au9.g1002.t1", "uniqueID": "globallyUniqueString3",
"subfeatures": [
{ "type": "five_prime_UTR", "start": 5975, "end": 6109, "score": 0.98, "strand": 1 },
{ "type": "start_codon", "start": 6110, "end": 6112, "strand": 1, "phase": 0 },
{ "type": "CDS", "start": 6110, "end": 6148, "score": 1, "strand": 1, "phase": 0 },
{ "type": "CDS", "start": 6615, "end": 6683, "score": 1, "strand": 1, "phase": 0 },
{ "type": "CDS", "start": 6758, "end": 7040, "score": 1, "strand": 1, "phase": 0 },
{ "type": "CDS", "start": 7142, "end": 7319, "score": 1, "strand": 1, "phase": 2 },
{ "type": "CDS", "start": 7411, "end": 7687, "score": 1, "strand": 1, "phase": 1 },
{ "type": "CDS", "start": 7748, "end": 7850, "score": 1, "strand": 1, "phase": 0 },
{ "type": "CDS", "start": 7953, "end": 8098, "score": 1, "strand": 1, "phase": 2 },
{ "type": "CDS", "start": 8166, "end": 8320, "score": 1, "strand": 1, "phase": 0 },
{ "type": "CDS", "start": 8419, "end": 8614, "score": 1, "strand": 1, "phase": 1 },
{ "type": "CDS", "start": 8708, "end": 8811, "score": 1, "strand": 1, "phase": 0 },
{ "type": "CDS", "start": 8927, "end": 9239, "score": 1, "strand": 1, "phase": 1 },
{ "type": "CDS", "start": 9414, "end": 9494, "score": 1, "strand": 1, "phase": 0 },
{ "type": "stop_codon", "start": 9492, "end": 9494, "strand": 1, "phase": 0 },
{ "type": "three_prime_UTR", "start": 9495, "end": 9744, "score": 0.86, "strand": 1 }
]
}
]
}
Configuring Tracks to Use a REST Feature Store
Example configuration for an HTMLFeatures track showing features from a REST feature store with URLs based at http://my.site.com/rest/api/base, and also adding "organism=tyrannosaurus" in the query string of all HTTP requests.
{
"label": "my_rest_track",
"key": "REST Test Track",
"type": "JBrowse/View/Track/HTMLFeatures",
"storeClass": "JBrowse/Store/SeqFeature/REST",
"baseUrl": "http://my.site.com/rest/api/base",
"query": {
"organism": "tyrannosaurus"
}
}
Other Dynamically-Servable Formats
trackList.json format
{
"tracks": [
{
"label": "my_gene_track", /* Unique, machine readable name */
"key": "Genes", /* Descriptive, meat readable name */
"type": "JBrowse/View/Track/HTMLFeatures",
"storeClass": "JBrowse/Store/SeqFeature/REST",
"baseUrl": "http://my.site.com/rest/api/base",
"query": { /* Your arbitrary set of query parameters, always sent with every request */
"organism": "tyrannosaurus", "soType": "gene"
}
},
{
"label": "my_sequence_track", /* Unique, machine readable name */
"key": "DNA", /* Descriptive, meat readable name */
"type": "JBrowse/View/Track/Sequence",
"storeClass": "JBrowse/Store/SeqFeature/REST",
"baseUrl": "http://my.site.com/rest/api/base",
"query": { /* Your arbitrary set of query parameters, always sent with every request */
"organism": "tyrannosaurus", "sequence": true
}
}
]
}
refSeqs.json format
This will be fetched from the url configured in the config.json file, such that if the config.json file specifies "?data=X/Y/Z" and is itself at "SCHEME://HOST:PORT/PATH", then JBrowse will request the url "SCHEME://HOST:PORT/PATH/X/Y/Z/seq/refSeqs.json".
[
{"name": "chr1", "start": 0, "end": 12345678},
{...}
]
Sequence data format
Retrieved from "{BASE}/features/{seqid}".
This is the REST feature store data format, but it expects just a single feature, and that feature should have sequence.
{"features": [
{"seq": "gattacagattaca" "start": 0, "end": 14}
]
JBrowse REST Names API
Starting in version 1.11.0, JBrowse can use REST web services for looking up features by name, and for type-ahead autocompletion.
GET (url)?equals=Apple1
Required. Returns JSON list of genomic locations with names that exactly match the given string.
The JSON format returned is the same as for startswith above.
GET (url)?startswith=Ap
Required. Returns JSON list of genomic locations that have names that start with the given string.
Example returned JSON:
[
{
"name" : "Apple1", // Name associated with the record. May be a secondary name of the object.
"location" : { // location information for the match
"ref" : "ctgA", // name of the reference sequence
"start" : 9999, // genomic start (interbase coords)
"end" : 11500, // genomic end (interbase coords)
"tracks" : [ // list of track labels that contain this object
"CDS"
],
"objectName" : "Apple1" // canonical/primary name of the object
}
},
...
]
Configuring JBrowse to Use REST Name Lookup
Add something like the following to jbrowse.conf:
[names]
type = REST
url = /path/to/names/rest/service
Publishing and Subscribing to JBrowse Events
JBrowse client events are implemented using the dojo/topic message bus from the Dojo library. Extensions can subscribe to particular events in order to be notified when certain UI changes happen (for example, highlighting a region generates an event, which can be latched onto with a callback that triggers a request for the server to BLAST that region against a database). In select cases, extensions can also publish events, as a way of forcing the UI into certain states or transitions (for example, events can be used in this way to force the browser to load a new track, in response to some other circumstance or notification).
JSON LazyNCList Feature Store
One data store type that JBrowse uses is a lazily-loaded nested containment list (LazyNCLists), which is an efficient format for storing feature data in pre-generated static JSON files. A nested containment list is a tree data structure in which the nodes of the tree are intervals themselves features, and edges connecting features that lie `within the bounds of (but are not subfeatures of) another feature. It has some similarities to an R tree. For more on NClists, see the Alekseyenko paper.
This data format is currently used in JBrowse 1.3 for tracks of type FeatureTrack, and the code that actually reads this format is in SeqFeatureStore/NCList.js and ArrayRepr.js.
The LazyNCList format can be broken down into two distinct subformats: the LazyNCList itself, and the array-based JSON representation of the features themselves.
Array Representation (ArrayRepr)
For speed and memory efficiency, NCList JSON represents features as arrays instead of objects. This is because the JSON representation is much more compact (saving a lot of disk space), and many browsers significantly optimize JavaScript Array objects over more general objects.
Each feature is represented as an array of the form [ class, data, data, ... ], where the class is an integer index into the store's classes array (more on that in the next section). Each of the elements in the classes array is an array representation that defines the meaning of each of the the elements in the feature array.
An array representation specification is encoded in JSON as (comments added):
{
"attributes": [ // array of attribute names for this representation
"AttributeNameForIndex1",
"AttributeNameForIndex2",
...
],
"isArrayAttr": { // list of which attributes are themselves arrays
"AttributeNameForIndexN": 1,
...
}
}
Lazy Nested-Containment Lists (LazyNCList)
A JBrowse LazyNCList is a nested containment list tree structure stored as one JSON file that contains the root node of the tree, plus zero or more "lazy" JSON files that contain subtrees of the main tree. These subtree files are lazily fetched: that is, they are only fetched by JBrowse when they are needed to display a certain genomic region.
On disk, the files in an LazyNCList feature store look like this:
# stats, metadata, and nclist root node
data/tracks/<track_label>/<refseq_name>/trackData.json
# lazily-loaded nclist subtrees
data/tracks/<track_label>/<refseq_name>/lf-<chunk_number>.json
# precalculated feature densities
data/tracks/<track_label>/<refseq_name>/hist-<bin_size>.json
...`
Where the trackData.json file is formatted as (comments added):
{
"featureCount" : 4293, // total number of features in this store
"histograms" : { // information about precalculated feature-frequency histograms
"meta" : [
{ // description of each available bin-size for precalculated feature frequencies
"basesPerBin" : "100000",
"arrayParams" : {
"length" : 904,
"chunkSize" : 10000,
"urlTemplate" : "hist-100000-{Chunk}.json"
}
},
... // and so on for each bin size
],
"stats" : [
{ // stats about each precalculated set of binned feature frequencies
"basesPerBin" : "100000", // bin size in bp
"max" : 51, // max features per bin
"mean" : 4.93030973451327 // mean features per bin
},
...
]
},
"intervals" : {
"classes" : [ // classes: array representations used in this feature data (see ArrayRepr section above)
{
"isArrayAttr" : {
"Subfeatures" : 1
},
"attributes" : [
"Start",
"End",
"Strand",
"Source",
"Phase",
"Type",
"Id",
"Name",
"Subfeatures"
]
},
...
{ // the last arrayrepr class is the "lazyClass": fake features that point to other files
"isArrayAttr" : {
"Sublist" : 1
},
"attributes" : [
"Start",
"End",
"Chunk"
]
}
],
"nclist" : [
[
2, // arrayrepr class 2
12962, // "Start" minimum coord of features in this subtree
221730, // "End" maximum coord of features in this subtree
1 // "Chunk" (indicates this subtree is in lf-1.json)
],
[
2, // arrayrepr class 2
220579, // "Start" minimum coord of features in this subtree
454457, // "End" maximum coord of features in this subtree
2 // "Chunk" (indicates this subtree is in lf-2.json)
],
...
],
"lazyClass" : 2, // index of arrayrepr class that points to a subtree
"maxEnd" : 90303842, // maximum coordinate of features in this store
"urlTemplate" : "lf-{Chunk}.json", // format for lazily-fetched subtree files
"minStart" : 12962 // minimum coordinate of features in this store
},
"formatVersion" : 1
}
Feature API and Feature Store API
Classes modeling individual sequence features conform to the Feature API (exemplified by and documented in the source for the class JBrowse/Model/SimpleFeature) by providing accessor methods for various feature attributes (start and endpoint, ID field, tags, score, parent and child relationships for modeling super- and sub-features) some of which are mandatory for the various different types of track. Specifically: “start” and “end” attributes are always required (representing 1-based closed-interval coordinates), a unique “id” attribute is required by non-quantitative tracks (CanvasFeatures, HTMLFeatures, Alignments*, etc) and a “score” attribute is used by quantitative tracks (Wiggle/*) to represent the score for the interval spanned by the feature.
By contrast, classes modeling sources of sequences and sequence features generally inherit from JBrowse/Store/SeqFeature and implement the Feature Store API including methods getGlobalStats (for global statistics about the features in the store), getRegionStats (for statistics about a particular interval), getFeatures (to query the store for features) and getReferenceSequence (to query the store for sequence data).
Typically, different Feature Stores will provide their own custom implementations of the Feature API.
Using JBrowse with Existing Web Services
Users can extend JBrowse's functionality to with their own JavaScript code using the JBrowse plugin system. For an overview of plugins and their structure, see Writing JBrowse Plugins.
To use JBrowse with an existing set of web services, users will want to implement a JBrowse Store module in JavaScript that can fetch data from them and convert it into the internal JavaScript object representations that JBrowse expects. In general terms, the steps to follow to do this would be:
- Create a new plugin using
bin/new-plugin.plor manually. - Enable the plugin by adding its name to
pluginsin the JBrowse configuration (in jbrowse_conf.json, trackList.json, in the constructor arguments in index.html, or elsewhere). - Create a new data store class in the plugin's JS directory that inherits from JBrowse/Store/SeqFeature and overrides its methods.
Example custom JBrowse store class
In plugins/MyPlugin/js/Store/SeqFeature/FooBaseWebServices.js, usable in store or track configurations as MyPlugin/Store/SeqFeature/FooBaseWebServices.
Note that the most basic class could simply have a "getFeatures" function that grabs the feature data.
/**
* Example store class that uses Dojo's XHR libraries to fetch data
* from backend web services. In the case of feature data, converts
* the data into JBrowse SimpleFeature objects (see
* JBrowse/Model/SimpleFeature.js) but any objects that support the
* same methods as SimpleFeature are fine.
*/
define([
'dojo/_base/declare',
'dojo/_base/array',
'dojo/request/xhr',
'JBrowse/Store/SeqFeature',
'JBrowse/Model/SimpleFeature'
],
function( declare, array, xhr, SeqFeatureStore, SimpleFeature ) {
return declare( SeqFeatureStore, {
constructor: function( args ) {
// perform any steps to initialize your new store.
},
getFeatures: function( query, featureCallback, finishCallback, errorCallback ) {
var thisB = this;
xhr.get( this.config.baseUrl+'my/features/webservice/url',
{ handleAs: 'json', query: query }
).then(
function( featuredata ) {
// transform the feature data into feature
// objects and call featureCallback for each
// one. for example, the default REST
// store does something like:
array.forEach( featuredata || [],
function( featureKeyValue ) {
var feature = new SimpleFeature({
data: featureKeyValue
});
featureCallback( feature );
});
// call the endCallback when all the features
// have been processed
finishCallback();
},
errorCallback
);
}
});
});
Note: other feature stores can be "derived from" or extended in different ways. The FeatureCoverage store class is a good example of a store class that uses the BAM store, but instead overrides the functionality to calculate coverage.