comfyui-voice-pipeline

Files (1)

• SKILL.md

SKILL.md (first lines)

# ComfyUI Voice Pipeline

> All recommendations in this skill are subordinate to `state/inventory.json`, `foundation/hardware-profile.md`, and current version constraints.

Creates character voices through TTS/voice cloning and synchronizes them with generated video.

Before choosing a path, verify the required node, model, executable, or API integration.
Classify each candidate as local ComfyUI, external local tooling, or hosted API. A named
tool below is never evidence that it is available.

## Voice Generation Decision Tree

```
VOICE REQUEST
    |
    |-- Have reference audio of target voice?
    |   |-- Yes → Compare installed cloning/conversion engines against their versioned requirements
    |   |-- Hosted service acceptable → ElevenLabs API, if configured
    |
    |-- No reference audio?
    |   |-- Need emotion control → Select an installed engine exposing emotion controls
    |   |-- Need multi-language → Select an installed engine supporting the target language
    |   |-- Need voice design → ElevenLabs Voice Design (describe voice)
    |   |-- Quick prototype → Available default voice
    |
    |-- Need multi-speaker dialog?
    |   |-- Select an installed engine exposing speaker switching
    |
    |-- Need lip-sync?
    |   |-- Wav2Lip installed? → Validate it on the target face and footage
    |   |-- Need head movement/expression transfer → Verify SadTalker or LivePortrait availability
    |   |-- Need long-form output → Verify an installed long-form workflow and its limits
```

## Tool Reference

### Chatterbox (Local Candidate)

Verify the installed version, supported tags, reference-audio requirements, and runtime
before selecting it. Upstream benchmark claims are not local validation.

**Paralinguistic tags:**
```
[laugh] [chuckle] [sigh] [gasp] [cough] [clear throat]
[whisper] [excited] [sad] [angry] [surprised]
```

**Key parameter**: `exaggeration` (0.25-2.0) controls expressiveness.

Confirm the installed version's segment limit before splitting long content.

### F5-TTS

Verify reference length, language support, and license from the installed version's model card.

**Requirements**: Reference audio must be paired with `.wav` + `.txt` (matching transcription).

**Languages**: English, German, Spanish, French, Japanese, Hindi, Thai, Portuguese.

### TTS Audio Suite

Verify which engines, languages, and speaker-switching features the installed node exposes.

**Special features:**
- Character switching: `[CharacterName]` tags
- Language switching: `[de:Alice]`, `[fr:Bob]`
- Pause control: `[pause:1s]`
- SRT timing sync

**Integrates**: F5-TTS, Chatterbox, Higgs Audio 2, VibeVoice, IndexTTS-2, RVC.

### IndexTTS-2

Verify the installed version's emotion controls and per-segment parameters.

**Emotions**: happy, angry, sad, surprised, afraid, disgusted, calm, melancholic.

### RVC (Voice Conversion)

**Use case**: Train a model on target voice (10+ min audio), then convert any TTS output.

**Pipeline**: `Text → Any TTS → Base Audio → RVC Model → Character Voice`

Derive training duration and feature extraction from the selected RVC implementation and dataset.

### ElevenLabs (Hosted API)

Treat ElevenLabs as an external service, never as a locally installed ComfyUI model.
Use it only when the configured integration is available.

**Tiers:**
- Instant Clone: 1-minute sample, good quality
- Professional Clone: 30+ minutes (3h ideal), near-indistinguishable
- Voice Design: Describe voice in text (no sample needed)

## Voice Profile Setup

For each character, establish a voice profile in `projects/{project}/characters/{name}/profile.yaml`:

```yaml
voice:
  cloned: true
  model: "chatterbox"
  sample_file: "references/voice_sample.wav"
  settings:
    exaggeration: 1.2
    default_emotion: "neutral"
  notes: "[voice qualities, accent, pacing, and delivery]"
```

## Script Preparation

### Text Formatting for TTS

1. **Punctuation matters**: Commas create pauses, periods create stops
2. **Phonetic hints**: Spell unusual words phonetically if mispronounced
3. **Emotion cues**: Use Chatterbox tags or split by emotion for IndexTTS-2
4. **Length**: Split into 30-40 second segments for Chatterbox limit

From calebmills99/VUORSE-VORTEX, MIT.

More from calebmills99/VUORSE-VORTEX