JBrowse 1.11.2 maintenance release

JBrowse 1.11.2 has been released, with some new controls to adjust the size of quantitative tracks (thanks to Evan Briones at University of Arizona!), added support for the new text-based tracks.conf files to generate-names.pl, and quite a few other small improvements.

Files for download:

Minor improvements

  • Added some user interface elements to set the height in pixels of a
    single quantitative track, or of all visible quantitative tracks.
    Thanks to Evan Briones for implementing this!
  • Added a JBrowse/View/FeatureGlyph/Diamond glyph that draws
    diamond-shaped features instead of boxes. Thanks to OICR Co-op
    student Kevin Mohamed for implementing this!
  • Reference sequence tracks now display a “no sequence” message
    instead of a bunch of blank blocks when the reference sequence
    basepairs aren’t available. Thanks to Kevin Mohamed for
    implementing this (issue #422).
  • Persistent session state is now stored on a per-dataset basis,
    which improves user experience when switching between multiple
    datasets in JBrowse. Thanks to Richard Hayes for pointing this
    issue out, and Kevin Mohamed for the fix (issue #410).
  • “Hide sites not passing …” settings in VCF track menus now show
    the filter’s long description when hovered over. Thanks to Keiran
    Raine for suggesting this (issue #420).
  • Tweaked display of track labels and added a slight border at the
    top edge of each track to make it more clear which track data
    belongs to. Thanks to Keiran Raine for suggesting this, and Kevin
    Mohamed for the initial implementation (issue #432).
  • Added a --config option to flatfile-to-json.pl that accepts
    additional configuration variables that will be merged into the top
    level of the track configuration. Thanks to Mikael Brandström
    Durling for the initial implementation of this.

Bug fixes

  • generate-names.pl now indexes VCF files from track definitions in
    tracks.conf files. Thanks to Paul Halle for pointing this out
    (issue #434).
  • Added a missing dependency for the server side formatting tools on
    List::MoreUtil 0.28 or higher. Thanks to Cris Lawrence and Keiran
    Raine for troubleshooting this!
Posted in Software releases | Leave a comment

JBrowse 1.11.1 maintenance release

JBrowse 1.11.1 has been released, with a number of bug fixes and some nice improvements to Alignments2 tracks. Most of the Alignments2 improvements were thought up by Keiran Raine at Sanger, thanks Keiran! Thanks also to Cris Lawrence and Richard Hayes, who were helpful in finding some other bugs and opportunities for improvements.

Files for download:

Minor improvements

  • Alignments2 tracks now include right-click menu items to view the
    location of an alignment’s mate pair or next segment in a popup or
    a new tab. Thanks to Keiran Raine for suggesting this (issue #406).

  • Alignments2 tracks now draw gaps and deletions in reads regardless
    of zoom level, as long as the alignment is at least 3 pixels wide
    in the display. Thanks to Keiran Raine for pointing out the need
    for this (issue #403).

  • Added support for a histograms.max variable for
    CanvasFeatures-based tracks that can be used to manually set the
    max value of a histogram display. Thanks to Keiran Raine for
    pointing out the need for this.

  • Added support for drawing clip markers (with their color set by
    histograms.clip_marker_color) in CanvasFeatures-based
    tracks. Thanks to Keiran Raine for pointing out the need for this
    (issue #402).

  • Canvas-based feature tracks now try to draw histograms, if
    available, when the data backend throws a data-overflow error (like
    when the BAM backend exceeds the chunkSizeLimit). Thanks to Keiran
    Raine for motivating this (issue #405).

  • Make it easier to set JS loading baseUrl by moving it into the
    initial dojo configuration. Thanks to Jillian Rowe for pointing
    out the need for this.

  • Enhanced new text-based config syntax to support arrays of values
    in a list like:

    [trackMetadata]
    sources =
      + data/mymeta.csv
      + data/more_meta.csv
    

Bug fixes

  • When a number is typed into the location box, JBrowse first checks
    if it is the name of a feature in the names index, and only
    interprets it as a coordinate if it is not found in the names
    index. Thanks to Richard Hayes for pointing this out (issue #407).

  • Fixed bug that caused client-side GFF3 tracks to appear as
    “Loading” forever if the GFF3 is malformed (like malformed GFF3
    files that are opened with the File->Open tool).

  • Fixed bug in which no default value for
    maxFeatureSizeForUnderlyingRefSeq was being set, which made
    default feature detail popups try to fetch and display a feature’s
    underlying reference sequence even if it is way too large, unless
    the variable was set explicitly in the configuration.

  • JBrowse now shows a more understandable error message when trying
    to open an uncompressed BAM file. Thanks to Keiran Raine for
    pointing this out (issue #404).

  • Fixed jbrowse.conf faceted track selector configuration examples
    not working as written. Thanks to Cris Lawrence for pointing this
    out.

  • Fixed a bug in which right-clicking on feature labels in an
    HTMLFeatures-based track did not bring up the right-click menu for
    a feature. Thanks to Cris Lawrence for pointing this out (issue #408).

Posted in Software releases | Leave a comment

JBrowse 1.11.0: hierarchical track selector, easier configuration, summary mode

JBrowse 1.11.0 is out today, with some major new features, and, as always, quite a lot of smaller bugfixes and improvements.

But first, the download links:

Firstly, JBrowse has added a third track selector type alongside the existing Simple (small) and Faceted (large) track selectors called Hierarchical.  It looks like the old Simple track selector (which was the default), except it pays attention to the “category” key in a track’s metadata, and if it is present, it organizes the tracks into nested, collapsible panes based on that.  Also, instead of the drag-and-drop paradigm used by the Simple track selector, the Hierarchical track selector turns tracks on and off by just checking and unchecking the box next to each track.  I think the checkboxes are probably easier for new users to understand, as well, the Hierarchical track selector is now enabled by default.  Those that prefer the old Simple selector can always turn it on by setting trackSelector.type to “Simple” in the jbrowse_conf.json file or the new jbrowse.conf file, which brings me to my next item.

Secondly, JBrowse now has a additional text configuration format that is designed to be easier to hand edit, and can coexist and interoperate with existing JSON configuration files.  Users of GBrowse will find the syntax very familiar, since its design borrows heavily from GBrowse’s configuration syntax.  Here’s a side-by-side comparison of what you would write in the old JSON and new text configuration formats to make JBrowse use the old Simple track selector.

jbrowse_conf.json jbrowse.conf
"trackSelector": {
  "type": "Simple"
}
[trackSelector]
type = Simple

It’s easier to configure tracks in the new format, as well.  Here’s a typical BAM Alignments2 track configuration in both formats, for comparison.

trackList.json
"tracks": [
  {
    "storeClass"  : "JBrowse/Store/SeqFeature/BAM",
    "urlTemplate" : "../../raw/volvox/volvox-sorted.bam",
    "label"       : "volvox-sorted-bam",
    "type"        : "JBrowse/View/Track/Alignments2",
    "key"         : "My BAM track"
  }
]
tracks.conf
[tracks.volvox-sorted-bam]
storeClass  = JBrowse/Store/SeqFeature/BAM
urlTemplate = ../../raw/volvox/volvox-sorted.bam
type        = JBrowse/View/Track/Alignments2
key         = My BAM track

And for the final headline feature of this release, JBrowse CanvasFeatures, CanvasVariants, and Alignments2 tracks can now be configured to show feature density histograms or coverage plots when zoomed too far out to display individual features.  They can take this quantitative data from any type of JBrowse data store (including REST stores), but most users would want to use a BigWig store.    One caveat: this isn’t a full-featured quantitative display like a Wiggle/XYPlot track, so quantitative values need to all be greater than 0, and all you can really change about how it looks is its color.  Here’s an example in the new text configuration syntax of an Alignments2 track that will display either alignments from a BAM file or a coverage plot from a BigWig file, depending on zoom level.

[tracks.my-bam-with-coverage]

storeClass  = JBrowse/Store/SeqFeature/BAM
urlTemplate = my_alignments.bam

type        = JBrowse/View/Track/Alignments2
key         = My BAM track

histograms.storeClass = JBrowse/Store/SeqFeature/BigWig
histograms.urlTemplate = my_alignments.bam.coverage.bw
histograms.description = coverage plot
histograms.color = #dcdcdc

Of course, there are many smaller improvements and bug fixes.  Read on for the complete release notes.

Release 1.11.0

Major improvements

  • Introduced density/coverage histogram support for CanvasFeatures,
    CanvasVariants, and Alignments2 tracks. These track types now
    support an optional histograms configuration subsection that can
    contain a definition for a second datastore that holds quantitative
    data (usually either coverage depth or feature density) to be
    displayed when zoomed further out than featureScale (or if
    featureScale is not set, the scale determined by the store’s
    feature density divided by maxFeatureScreenDensity). Thanks to
    Richard Hayes for pushing hard for this feature.
  • Added a new “Hierarchical” track selector that shows tracks in a
    hierarchy of collapsible categories, which is now the default track
    selector. To assign categories and subcategories to your tracks,
    set category or metadata.category attributes on each configured
    tracks in your trackList.json. Thanks to the many users who have
    requested this at one time or another.
  • JBrowse now supports a new plaintext configuration format that
    users of GBrowse will find very familiar, since it is designed to
    be very similar to it. This syntax is also much easier to
    hand-write than JSON. The JSON configuration syntax is not going
    away, and will continue to be supported.

    Thanks to Erik Derohanian for the original implementation of this
    configuration adaptor, and Richard Hayes and Keiran Raine for
    motivating the work to polish and more fully integrate it.

  • Variables in configuration files can now be based on the contents
    of other variables. For example, setting

    “myCustomVariable”: “/some/custom/path”,
    “include”: “{myCustomVariable}/conf.json”

    will try to include a configuration file located at
    “/some/custom/path/conf.json”. Interpolation is done as the final
    step in configuration loading, so variables can come from anywhere
    in the configuration.

  • When JBrowse is started, if there are no reference sequences found
    in the default dataRoot, but the dataset selector is configured,
    JBrowse shows a simple list of links to available datasets instead
    of the “Congratulations, JBrowse is on the web” page. Thanks to
    Saulo Aflitos for the idea and its initial implementation.
  • For users wishing to convert existing JSON configuration files to
    the new format, there is a new script, bin/json2conf.pl, that
    does a fair job. Run bin/json2conf.pl -? for details on how to
    use it.
  • Added a new REST backend for name lookup and autocompletion. See
    http://gmod.org/wiki/JBrowse_Configuration_Guide#JBrowse_REST_Names_API
    for details. Thanks to Erik Derohanian for implementing this, and
    Ben Booth for suggesting an API design (issue #267).
  • Major performance and scalability improvements for
    generate-names.pl. Now uses a different algorithm that is faster
    and more scalable than before, and no longer relies on BerkeleyDB
    for temporary storage. This should also alleviate the need to run
    generate-names.pl with --safeMode in Perl 5.10 and earlier. In
    fact, the --safeMode argument to generate-names.pl no longer has
    any effect. Thanks to Cris Lawrence for pointing out the
    continuing need for more scalability.

Minor improvements

  • Detail popups for CanvasVariants and HTMLVariants tracks now
    display the reference sequence itself instead of just “ref” in
    genotype displays. Thanks to Cris Lawrence for requesting this.
  • Added a “save as FASTA” button to default feature detail popups
    that downloads a FASTA file with the displayed piece of reference
    sequence (issue #299).
  • chunkSizeLimit for VCF files now defaults to 1 MiB. It used to
    be 15 MiB, which was really far too big for browsers to handle.
  • Added support for a --nameAttributes argument to
    flatfile-to-json.pl that takes a comma-separated list of feature
    attributes to index for name searching and completions, or ‘none’
    to not make names searchable.
  • Added support for a nameAttributes variable in biodb-to-json.pl
    track configurations that can be set to an array of feature
    attribute names to to index for name searching and completions, or
    ‘none’ to not make names searchable.
  • Add a --category argument to bin/wig-to-json.pl that can be used
    to set the metadata.category of a track.

Bug fixes

  • Fixed a bug in NCList data backed in which feature histograms were
    often calculated very incorrectly.
  • Fixed a bug in the VCF data backend that caused not all VCF
    features to be shown in some files at some zoom levels.
Posted in Software releases | Leave a comment

JBrowse 1.10.12 maintenance release (please upgrade!)

JBrowse 1.10.12 has been released, and we are asking all users of JBrowse 1.10.7 1.10.10 and higher to please upgrade to this release as soon as possible.

The story behind this is a bit embarassing: I discovered today that JBrowse releases 1.10.7 through 1.10.10 and 1.10.11 shipped with usage analytics reporting disabled. This is a little piece of code that tracks how many installations and users of JBrowse there are, and the data it provides is absolutely crucial for the JBrowse project, because we use it to try to convince funding agencies that JBrowse is a project worth funding.

So, yes … this release has analytics reporting re-enabled. There are also a few other minor improvements and bug fixes. Please upgrade to this ASAP if you are using JBrowse 1.10.7, 1.10.8, 1.10.9,  1.10.10 or 1.10.11, because we really need that analytics data.

Edit (Dec 11): realized just now that only 1.10.10 and 1.10.11 were affected.  Forgot to account for branching and merging last night.

Files for download:

Minor improvements

  • bam-to-json.pl and flatfile-to-json.pl now support a
    --metadata argument that can add a metadata stanza to track
    configurations they generate.
  • Multi-valued attributes in default feature detail popups are
    displayed as a string of boxes, each containing a value, to avoid
    misinterpretation. Thanks to Cris Lawrence for pointing out the
    need for this.

Bug fixes

  • Re-enabled usage analytics reporting, which had been disabled by a
    stray piece of debugging code since the 1.10.7 release.
  • Fixed a bug in which the tooltip in canvas-based feature tracks
    would sometimes display incorrect label or description text.
Posted in Development, Software releases | Leave a comment

JBrowse 1.10.11 maintenance release

JBrowse 1.10.11 has been released, with many bug fixes and better support for spliced alignments in SNPCoverage tracks.

(Download disabled.  Please download the 1.10.12 release or later instead.)

Minor improvements

  • Made the sequence track’s “zoom in to see sequence” placeholder
    take up less vertical space. Thanks to Scott Cain for pointing out
    that making it be the same height as it will eventually be when
    zoomed in to base level is silly.

  • By default highlighting features after searching for them by name
    is now turned off. Set the highlightSearchedRegions top-level
    conf variable to true to turn this back on. Turns out, most
    people seem not to like this behavior. Thanks to Gregg Helt and
    Cris Lawrence for pointing this out.

  • SNPCoverage tracks now correctly display “skipped” regions in
    alignments, such as those produced by TopHat. Thanks to Josh
    Orvis, Gustavo Cerquiera, and others for reminders that this was
    still an issue.

  • SNPCoverage tracks now provide per-strand counts of “reference”
    reads at each position, like they already were providing for
    reads with mismatches.

  • SNPCoverage tracks now accept a mismatchScale configuration
    variable that sets the viewing scale (i.e. zoom level, pixels per
    bp) above which base-level mismatches will be drawn. Defaults to
    1/10. Making this value larger can speed up SNPCoverage tracks for
    high-coverage data at the cost of needing to zoom in further to see
    mismatches.

  • setup.sh now uses curl for downloading things instead of wget,
    since curl is more widely available. Thanks to Keiran Raine for
    suggesting and implementing this (issue #393).

Bug fixes

  • Fixed a bug in which generate-names.pl would sometimes report the
    incorrect number of hashing bits in verbose output, and would
    sometimes use the number of hashing bits for an existing index even
    if that index was being regenerated. Thanks to Richard Hayes for
    pointing out the incorrect log output.

  • Fix generate-names.pl crashing on some older versions of Perl with
    an error like Bareword "POSIX::O_RDONLY" not allowed while "strict
    subs" in use
    . Thanks to Chris Childers for pointing this out.

  • Fix setup.sh failing on some older versions of Perl. It now runs
    generate-names.pl with the --safeMode flag.

  • Fixed a bug where the value display in SNPCoverage tracks would
    sometimes report “NaN%” for the reference when no reads cover a
    region.

  • Fixed a bug in which activating rubberband zooming using the SHIFT
    key while in highlighting mode would cause all the tracks to be
    dragged when attempting to highlight a region afterward. Thanks to
    Erik Derohanian for pointing out and fixing this (issue #387).

  • Fixed the location of the dojo/nls directory in release zipfiles.
    Was erroneously in src/nls, supposed to be src/dojo/nls. Thanks to
    Matt Bomhoff for pointing this out.

  • The in-memory GFF3 parser now copes with a missing newline at the
    end of a GFF3 file. Previously, the last line was ignored if it
    did not end with a newline character. Thanks to Colin Davenport
    for pointing this out (issue #394).

Posted in Software releases | Leave a comment

JBrowse 1.10.10 maintenance release

JBrowse 1.10.10 has been released, with new compression support in generate-names.pl, configurable inter-track spacing, and quite a few bug fixes.

(Download disabled.  Please download the 1.10.12 release or later instead.)

Minor improvements

  • Spacing between tracks is now configurable by setting
    view.trackPadding in the configuration. Thanks to Chenchen Zhu
    for suggesting this (issue #377).

  • If reference sequences are defined, but no tracks are yet added,
    JBrowse will start normally instead of going to the
    “Congratulations, JBrowse is on the web” page.

  • generate-names.pl now supports a --compress option that
    compresses the name index files to save server disk space. Thanks
    to Richard Hayes for pointing out the need for this (issue #378).

Bug fixes

  • generate-names.pl now uses IO::Uncompress::Gunzip instead of
    PerlIO::gzip to read compressed VCF files. This fixes a bug in
    which only the first few hundred names in a VCF were indexed.
    Thanks to Cris Lawrence for pointing this out (issue #380).

  • Fixed a bug in which generate-names.pl would crash if run with
    --incremental and no existing names index. Thanks to Richard
    Hayes for pointing this out (issue #379).

  • Fixed a bug in which generate-names.pl would sometimes choose the
    wrong number of hash bits when performing incremental updates,
    leading to the old data being lost. Thanks to Richard Hayes for
    lots of help troubleshooting this.

  • Fixed bug where other tracks are visible underneath pinned tracks
    when the display is scrolled down. Thanks to Ed Lee for pointing
    this out.

  • Fixed a bug in which tooltip does not hide after the mouse leaves a
    Wiggle track in Safari 5 and 6. Thanks to Charles Girardot for
    pointing this out.

  • Fixed a bug in which the main.css file for plugins was not being
    correctly loaded in some installations. Thanks to Matt Bomhoff for
    pointing this out.

Posted in Software releases | Leave a comment

JBrowse 1.10.9 maintenance release

JBrowse 1.10.9 has been released, with new incremental-updating support in generate-names.pl, along with some other bug fixes and small improvements.

Files for download:

Minor improvements

  • generate-names.pl now supports a --incremental or -i option
    that adds names to an existing index. Thanks to Richard Hayes for
    reminding me that this wasn’t done yet (issue #373).

  • Added a lower-performance, but more backward-compatible indexing
    backend to generate-names.pl that can be activated by passing the
    --safeMode command-line argument. The recent performance
    improvements to generate-names.pl have apparently tickled some bugs
    that are present in some installations. Thanks to Josie Reinhardt
    and GitHub user raj76 for their continued help troubleshooting
    this. If you find that name indexing is not working correctly, try
    running it again with --safeMode, and report to the mailing list
    if it helps.

  • Added support for using the JBrowse in-memory GFF3 adaptor to
    display web-accessible GFF3 files directly. See
    docs/tutorial/data_files/volvox.gff3.conf for an example
    configuration. Thanks to David Goodstein and Richard Hayes for
    motivating this.

Bug fixes

  • flatfile-to-json.pl now depends on the latest
    Bio::GFF3::LowLevel::Parser 1.8, which fixes a bug in which
    features with no ID, Parent, or Derives_from attributes were not
    being included in parsed data. Thanks to Gwendoline Andres for
    pointing this out.

  • Tweak BioPerl-handling code in biodb-to-json.pl to hopefully work
    better with BioSQL backends. Thanks to Brian Osborne for pointing
    this out.

Posted in Software releases | Leave a comment

JBrowse 1.10.8 maintenance release

JBrowse 1.10.8 has been released, with an important bugfix for generate-names.pl, better REST data store feature caching, and more.

Files for download:

Minor improvements

  • Added a new navigateTo action usable for customizing feature
    left-clicks and right-click menus. Thanks to Scott Cain for
    requesting this.

  • Added a new feature_range_cache option for the REST data backend.
    If set to true, the REST backend will more aggressively cache
    ranges of feature data. Thanks to Daniel Troesser for pointing out
    the need for this (issue #369).

  • maker2jbrowse now, by default, runs generate-names.pl to
    generate names indexes. Also added a --no_names_index option to
    turn this off. Thanks to Josie Reinhardt for making me notice this
    was missing.

  • Tweaked generate-names.pl default indexing parameters to
    emphasize indexing speed more. Now defaults to a smaller average
    file size for the on-disk JSON files it produces, which is much
    faster to generate and write (up to 8 or 10x faster). However, the
    on-disk index is about 2-2.5x larger overall.

  • Removed support for the --refids command-line argument to
    prepare-refseqs.pl, which has probably never really worked.

  • Improved prepare-refseqs.pl support for Bio::DB::Das::Chado
    database backends. Thanks to Gwendoline Andres for helping
    troubleshoot this.

Bug fixes

  • Fixed generate-names.pl making incorrect name indexes when using 16
    or more bits of hashing (when the number of index entries exceeds
    about 4 million). Thanks to Josie Reinhardt for helping
    troubleshoot this (issue #370).

  • The faceted track selector no longer refuses to display track
    unique labels even if they are explicitly included in the
    displayColumns setting.

Posted in Software releases | Leave a comment

JBrowse 1.10.7 maintenance release

JBrowse 1.10.7 has been released, with some important bug fixes and added support for displaying feature density histograms from REST stores.

Files for download:

Minor improvements

  • Added an optional stats/regionFeatureDensities endpoint to the
    REST API that makes it possible to provide binned feature counts
    that HTMLFeatures tracks can use to display feature histograms.
    Thanks to Stuart Watt and Daniel Troesser for pointing out the need
    for this (issue #365).

Bug fixes

  • Fixed a bad bug introduced in 1.10.6 in which FASTA files with line
    lengths longer than the configured chunk size were not correctly
    formatted. Thanks to Jean-Jack Riethoven for pointing this out
    (issue #363).

  • Fixed a bug introduced in 1.10.6 in which prepare-refseqs.pl would
    not respect the –noseq command-line option when using –fasta.

  • Fixed bug in which name or seq_id attributes are required for
    reference sequence features. Thanks to Daniel Troesser for
    pointing this out.

  • Fixed a bug that may have prevented some types of VCF files from
    being displayed (error message referring to inheritedFilters).

  • Fixed a bug in which “Zoom in to see feature” in Sequence tracks
    would wrap downwards below track at some zoom levels and window
    widths.

  • Fixed an off-by-one error in UTR attributes manufactured by the
    impliedUTRs mechanism of the ProcessedTranscript glyph. Thanks
    to Ben Booth for pointing this out (issue #362).

  • Fixed a bug in which score attributes of features were not being
    recorded by biodb-to-json.pl. Thanks to HongKee Moon for
    pointing this out (issue #364).

  • Removed a stray use of Carp::Always in GFF3-processing Perl code
    that may have caused problems in some installations.

Posted in Software releases | Leave a comment

JBrowse 1.10.6 maintenance release

JBrowse 1.10.6 has been released, with some important bug fixes.

Files for download:

Minor improvements

  • Made the impliedUTRs option for ProcessedTranscript and Gene
    glyph still attempt to create UTRs if only ”’one”’ of the UTRs is
    missing from a transcript. Thanks to Ben Booth for pointing out
    the need for this.

Bug fixes

  • Fixed bug in which the mouseover value displays for Wiggle and
    SNPCoverage tracks would not always be hidden when the mouse leaves
    the track.

  • Fixed a bad bug that prevented fixed-scale Wiggle and SNPCoverage
    tracks from displaying. Thanks to Jean-Jack Riethoven and Michael
    Axtell for pointing this out.

Posted in Software releases | Leave a comment