nextflow-development

Files (28)

• LICENSE.txt
• SKILL.md
• geo-sra-acquisition.md · references/
• installation.md · references/
• atacseq.md · references/pipelines/
• rnaseq.md · references/pipelines/
• sarek.md · references/pipelines/
• troubleshooting.md · references/
• check_environment.py · scripts/
• cluster_profile_generator.py · scripts/
• genomes.yaml · scripts/config/
• atacseq.yaml · scripts/config/pipelines/
• methylseq.yaml · scripts/config/pipelines/
• nanoseq.yaml · scripts/config/pipelines/
• proteinfold.yaml · scripts/config/pipelines/
• rnaseq.yaml · scripts/config/pipelines/
• sarek.yaml · scripts/config/pipelines/
• scrnaseq.yaml · scripts/config/pipelines/
• detect_data_type.py · scripts/
• generate_samplesheet.py · scripts/
• manage_genomes.py · scripts/
• nfcore_launch.py · scripts/
• sra_geo_fetch.py · scripts/
• __init__.py · scripts/utils/
• file_discovery.py · scripts/utils/
• ncbi_utils.py · scripts/utils/
• sample_inference.py · scripts/utils/
• validators.py · scripts/utils/

SKILL.md (first lines)

# nf-core Pipeline Deployment & Cluster Execution

Run nf-core bioinformatics pipelines on local machines, HPC clusters (Slurm, PBS, SGE, LSF), or Cloud Batch environments (AWS Batch, Google Cloud Batch).

## Supported nf-core Pipelines

| Data Type | Pipeline | Version | Primary Purpose |
|---|---|---|---|
| Bulk RNA-seq | `rnaseq` | 3.22.2 | Gene expression quantification & DESeq2 |
| Single-Cell RNA | `scrnaseq` | 2.7.2 | STARsolo, Alevin, 10X count matrices |
| WGS / WES | `sarek` | 3.7.1 | Germline & somatic variant calling |
| ATAC-seq | `atacseq` | 2.1.2 | Peak calling & chromatin accessibility |
| DNA Methylation | `methylseq` | 2.7.0 | Bismark / bwa-meth CpG methylation |
| Protein Structure | `proteinfold` | 1.1.1 | AlphaFold2, ColabFold, ESMFold 3D models |
| Nanopore Long-Reads | `nanoseq` | 3.1.0 | Demultiplexing, Minimap2, SV calling |

---

## HPC & Cloud Cluster Profile Generator (`scripts/cluster_profile_generator.py`)

Generate tailored `nextflow.config` files with dynamic resource escalation (auto-retry on OOM errors) and container engine bindings:

```bash
# Slurm cluster profile with Singularity/Apptainer
python scripts/cluster_profile_generator.py \
    --executor slurm \
    --partition standard \
    --account my_lab_account \
    --max-memory 256.GB \
    -o nextflow_slurm.config

# AWS Batch profile
python scripts/cluster_profile_generator.py \
    --executor awsbatch \
    --aws-queue arn:aws:batch:... \
    --work-bucket s3://my-lab-bucket/nextflow_work \
    -o nextflow_aws.config

# Google Cloud Batch profile
python scripts/cluster_profile_generator.py \
    --executor googlebatch \
    --google-project my-gcp-project \
    --google-region us-central1 \
    --work-bucket gs://my-lab-bucket/work \
    -o nextflow_gcp.config
```

Run pipelines on your cluster:
```bash
nextflow run nf-core/scrnaseq \
    -c nextflow_slurm.config \
    -profile singularity \
    --input samplesheet.csv \
    --genome GRCh38 \
    --aligner starsolo \
    --outdir ./results
```

---

## Workflow Steps

1. **Step 0: Acquire Data (GEO/SRA)**: `python scripts/sra_geo_fetch.py download GSE110004 -o ./fastq`
2. **Step 1: Check Environment**: `python scripts/check_environment.py --samplesheet samplesheet.csv --config nextflow.config`
2b. **Launch artifact**: `python scripts/nfcore_launch.py --pipeline rnaseq --samplesheet samplesheet.csv --outdir results -o run.sh --preview`
3. **Step 2: Generate Samplesheet**: `python scripts/generate_samplesheet.py ./fastq scrnaseq -o samplesheet.csv`
4. **Step 3: Generate Cluster Profile**: `python scripts/cluster_profile_generator.py --executor slurm`
5. **Step 4: Execute Pipeline**: `nextflow run nf-core/<pipeline> -c nextflow_cluster.config ...`

From HERRY423/BioNexus, Apache-2.0.

More from HERRY423/BioNexus