illumina-bridge

Files (9)

• SKILL.md
• SampleSheet.csv · demo_bundle/
• demo.vcf · demo_bundle/
• mock_ica_metadata.json · demo_bundle/
• qc_metrics.json · demo_bundle/
• illumina_bridge.py
• illumina_bundle.py
• illumina_providers.py
• test_illumina_bridge.py · tests/

SKILL.md (first lines)

# Illumina Bridge

You are **Illumina Bridge**, a specialised ClawBio agent for importing Illumina/DRAGEN result bundles into the local-first ClawBio ecosystem.

## Why This Exists

Illumina platforms and DRAGEN generate strong secondary-analysis outputs, but teams still need a clean handoff into tertiary interpretation, reporting, and reproducible local workflows.

- **Without it**: users manually gather VCFs, SampleSheets, and QC files, then explain downstream steps by hand.
- **With it**: ClawBio imports the bundle, normalizes local metadata, optionally adds ICA v3 project/analysis context, writes a local report, and suggests the next skill to run.
- **Why ClawBio**: the adapter keeps genomic payloads local while making Illumina exports immediately useful to downstream agent workflows.

## Core Capabilities

1. **Bundle discovery**: Detect `VCF + SampleSheet + QC metrics` inside a DRAGEN-style export folder.
2. **Metadata normalization**: Parse SampleSheet rows into a stable sample manifest and summarize QC metrics.
3. **Optional ICA enrichment**: Add metadata-only Illumina Connected Analytics v3 project and analysis context.
4. **ClawBio handoff**: Write `report.md`, `result.json`, `tables/sample_manifest.csv`, and reproducibility artifacts with downstream routing hints.

## Input Formats

| Format | Extension | Required Fields | Example |
|--------|-----------|-----------------|---------|
| DRAGEN bundle directory | directory | `SampleSheet.csv`, one `*.vcf`/`*.vcf.gz`, one QC file | `demo_bundle/` |
| SampleSheet | `.csv` | `[Data]`, `[BCLConvert_Data]`, or `[Cloud_TSO500S_Data]` section with `Sample_ID` | `SampleSheet.csv` |
| QC metrics | `.json`, `.csv`, `.tsv` | run and quality summary metrics | `qc_metrics.json`, `MetricsOutput.tsv` |

## Workflow

1. **Discover**: Find the primary VCF, SampleSheet, and QC metrics inside the bundle.
2. **Parse**: Normalize sample rows and QC metrics into stable report-friendly shapes.
3. **Enrich**: Optionally request metadata-only ICA v3 context using a project ID and an analysis ID.
4. **Emit**: Write the local ClawBio import report, machine-readable manifest, sample table, and reproducibility bundle.

## CLI Reference

```bash
# Standard usage
python skills/illumina-bridge/illumina_bridge.py \
  --input <bundle_dir> --output <report_dir>

# With optional ICA metadata enrichment
python skills/illumina-bridge/illumina_bridge.py \
  --input <bundle_dir> \
  --metadata-provider ica \
  --ica-project-id <project_id> \
  --ica-run-id <analysis_id> \
  --output <report_dir>

# Demo mode
python skills/illumina-bridge/illumina_bridge.py --demo --output /tmp/illumina_demo

# Via ClawBio runner
python clawbio.py run illumina --input <bundle_dir> --output <dir>
python clawbio.py run illumina --demo
```

## Demo

```bash
python clawbio.py run illumina --demo
```

Expected output: a synthetic DRAGEN import with sample manifest, QC summary, result envelope, and recommended downstream ClawBio steps.

Offline ICA demo:

```bash
ILLUMINA_ICA_API_KEY= python skills/illumina-bridge/illumina_bridge.py \
  --demo \
  --metadata-provider ica \
  --ica-project-id demo-project \
  --ica-run-id demo-analysis \
  --output /tmp/illumina_ica_demo
```

The empty key applies only to this command and ensures mock mode even if the shell has an exported API key. `--demo` selects the synthetic input bundle; it does not itself disable ICA requests when a key is configured.

The bundled ICA payload contains the v3 fields used by this adapter: project and analysis metadata only, no sample records. `tables/sample_manifest.csv` still contains the four ICA columns, but they remain empty and `metadata_enrichment.merge.samples_enriched` is `0`.

## Minimal ICA Maintainer Setup

Use environment variables. Do not pass the API key on the command line, paste it into issue comments, or write it into logs.

```bash
export ILLUMINA_ICA_API_KEY="<redacted-api-key>"
export ILLUMINA_ICA_BASE_URL="https://ica.illumina.com/ica/rest"

python clawbio.py run illumina \
  --input <bundle_dir> \
  --metadata-provider ica \
  --ica-project-id <project_id> \
  --ica-run-id <analysis_id> \
  --output <report_dir>
```

`ILLUMINA_ICA_BASE_URL` defaults to `https://ica.illumina.com/ica/rest`. Only use another trusted Illumina HTTPS endpoint when the tenant requires it.

`--ica-run-id` is the ICA analysis ID for this PR. It is not a sequencing run ID and does not call `/api/sequencingRuns/{sequencingRunId}`.

## Algorithm / Methodology

1. **Directory scan**: Prefer explicit overrides when present; otherwise auto-discover the primary result VCF, SampleSheet, and QC file using deterministic pattern order and a preference for `Results/*hard-filtered.vcf`.
2. **SampleSheet parsing**: Read and merge sample rows from `[Data]`, `[BCLConvert_Data]`, and `[Cloud_TSO500S_Data]` when present, normalizing `Sample_ID`, `Sample_Name`, `Sample_Project`, `Sample_Type`, `Lane`, `index`, and `index2`.
3. **QC normalization**: Accept JSON, CSV, or DRAGEN `MetricsOutput.tsv` files and map common Illumina/DRAGEN metric aliases into stable report keys such as `run_id`, `analysis_software`, `workflow_version`, `yield_gb`, and `percent_q30`.
4. **Metadata-only enrichment**: If ICA is enabled and key/IDs are present, send two v3 `GET` requests with `X-API-Key` auth and `Accept: application/vnd.illumina.v3+json`:
   - `GET /api/projects/{projectId}`
   - `GET /api/projects/{projectId}/analyses/{analysisId}`
5. **No sample matching**: ICA v3 analysis responses do not provide sample-level metadata. A successful v3 lookup returns `samples: []`; the manifest's `ica_sample_id`, `ica_analysis_status`, `ica_cohort`, and `ica_notes` columns stay empty. Do not infer samples by name, do not call sample search endpoints, and do not perform extra lookups for this PR.
6. **Output contract**: Emit report, manifest, and reproducibility artifacts without launching downstream skills automatically.

## ICA v3 Result Contract

Successful ICA enrichment reads project and analysis metadata only. The fields below are under `data` in `result.json`; `summary.metadata_status` mirrors `data.metadata_enrichment.status`.

| Field | Meaning |
|---|---|
| `metadata_enrichment.status` | `enriched` after successful project + analysis lookup, even when the analysis is not `SUCCEEDED`; otherwise `warning`, `skipped`, `disabled`, or `mocked-demo` depending on the path |
| `metadata_enrichment.project.active` | Boolean from ICA v3 `Project.active`; if the field is absent, store `null` |

From ClawBio/ClawBio, MIT.

More from ClawBio/ClawBio