Fast and memory-efficient sampling of DNA-seq or RNA-seq FASTQ data with replacement. Useful for generating bootstrap replicates to estimate technical variance in downstream analyses, and for subsampling large datasets for testing and benchmarking.
pip install fastQpickUsing pip:
pip install git+https://github.com/pachterlab/fastQpick.gitRun fastQpick with a specified fraction and options:
fastQpick [OPTIONS] FASTQ_FILE1 FASTQ_FILE2 ...Use fastQpick in your Python code:
from fastQpick import fastQpick
fastQpick(
input_files=['FASTQ_FILE1', 'FASTQ_FILE2', ...],
...
)-
Command-line Help: Use the following command to see all available options:
fastQpick --help
-
Python API Help: Use the
helpfunction to explore the API:help(fastQpick)
Two Jupyter notebooks in notebooks/ walk through fastQpick end-to-end:
intro.ipynb— Getting started on synthetic data. Simulates a small RNA-seq experiment with known transcript abundances, draws bootstrap replicates with replacement (fraction=1.0,replacement=True), and shows that the bootstrap standard errors recover the analytic multinomial sampling error.yeast_example.ipynb— Real-data application reproducing Figure 1 of the paper. Bootstraps a paired-end yeast RNA-seq dataset (SRASRR453566), re-quantifies each replicate withkallisto, and characterizes the bootstrap distribution of the transcript abundance estimates.
To reproduce the figures exactly as they appear in the manuscript, check out the manuscript tag before running the notebooks:
git checkout manuscript- Time efficient - streams through the fastq and writes output in batches - generates a full-size (fraction=1, with replacement) bootstrap replicate of a 500M-read FASTQ in ~26 minutes in standard mode, ~56 minutes in low-memory mode, and ~35 minutes in one-pass mode (see Benchmark below).
- Memory efficient - the occurrence vector is sized to the largest per-read count actually drawn (one byte per read in the common case), and low-memory mode further avoids materializing the array of sampled indices.
- Gzip-compressed output by default, using the ISA-L-accelerated
isallibrary to keep compression from bottlenecking the write pass. Pass--disable-gzip(CLI) ordisable_gzip=True(Python API) to write plain FASTQ instead.
fastQpick is licensed under the 2-clause BSD license. See the LICENSE file for details.
We welcome contributions! Please see the CONTRIBUTING.md file for guidelines on how to get involved.