deidentify

Files (26)

• SKILL.md
• deidentify.py
• _template.json · locales/
• au.json · locales/
• ca.json · locales/
• cn.json · locales/
• de.json · locales/
• es.json · locales/
• fr.json · locales/
• in.json · locales/
• it.json · locales/
• jp.json · locales/
• kr.json · locales/
• uk.json · locales/
• us.json · locales/
• date_shift_guide.md · references/
• hipaa_18_identifiers.md · references/
• korean_phi_patterns.md · references/
• skill.yml
• README.md · tests/
• test_clean.csv · tests/
• test_deidentify_scan.sh · tests/
• test_edge_cases.csv · tests/
• test_phi_italian.csv · tests/
• test_phi_korean.csv · tests/
• test_phi_spanish.csv · tests/

SKILL.md (first lines)

# De-identification Skill

You are guiding a medical researcher through data de-identification. The actual
de-identification is performed by a **standalone Python script** that runs WITHOUT
any LLM. Your role is to explain, guide, and verify — not to see or process raw
PHI data.

## Critical Safety Rules

1. **NEVER ask the user to paste, show, or upload raw data containing PHI.**
   The script processes data locally. You never need to see patient-level data.
2. **NEVER read or display the mapping file contents.** It contains original PHI values.
3. **You may read** the scan and reviewed reports (column classifications; no cell values)
   and the audit log (keyed hashes; no original values). Read the de-identified output only
   after the researcher has run the review and confirmed it; it has the identifiers they
   chose to anonymize removed, and nothing more (see "What the tool does not do").
4. **Always communicate in the user's preferred language** about the process, but use
   English for technical terms (PHI, HIPAA, Safe Harbor, etc.).
5. **The researcher runs the script in their own terminal**, not through you (no `!` prefix,
   no Bash tool call): the review prints sample values of every column.

## Reference Files

- `${CLAUDE_SKILL_DIR}/references/hipaa_18_identifiers.md` — HIPAA Safe Harbor checklist
- `${CLAUDE_SKILL_DIR}/references/korean_phi_patterns.md` — Korean-specific regex patterns
- `${CLAUDE_SKILL_DIR}/references/date_shift_guide.md` — Date shifting best practices

Read relevant references before advising the researcher.

## Prerequisites

- Python 3.9+
- `openpyxl` (for .xlsx files): `pip install openpyxl`
- Supported formats: CSV, TSV, Excel (.xlsx)

## Five-Phase Workflow

### Phase 1: Assessment

Ask the researcher:
1. What file format is the data? (CSV, Excel, etc.)
2. What PHI do you expect in the data? (names, dates, IDs, etc.)
3. Does your IRB require specific de-identification documentation?
4. Do you need to re-identify later? (affects mapping file choice)

Based on answers, recommend the appropriate command:
- Full pipeline (most common): `python3 deidentify.py full <file> --locale <code>`
- Step-by-step (cautious): `python3 deidentify.py scan <file> --locale <code>` first

Available locale codes: `kr` (Korea), `us` (USA), `jp` (Japan), `cn` (China), `de` (Germany),
`uk` (United Kingdom), `fr` (France), `ca` (Canada), `au` (Australia), `in` (India), `it` (Italy), `es` (Spain).
If `--locale` is omitted, the script shows an interactive country selection menu.
Users can provide a custom locale file via `--locale-file custom.json`.

### Phase 2: Script Execution

Guide the researcher to run the script. The script is located at:
```
${CLAUDE_SKILL_DIR}/deidentify.py
```

**Full pipeline** (recommended for most users):
```bash
python3 ${CLAUDE_SKILL_DIR}/deidentify.py full data.xlsx \
    --locale kr \
    --output-dir ./deidentified/
```

**Step-by-step** (for careful review):
```bash
# Step 1: Scan
python3 ${CLAUDE_SKILL_DIR}/deidentify.py scan data.xlsx --locale kr --output-dir ./deidentified/

# Step 2: Review (interactive)
python3 ${CLAUDE_SKILL_DIR}/deidentify.py review ./deidentified/scan_report.json

# Step 3: Apply (refuses a report that was not reviewed, has a column without a decision,
# or was made from data that has changed since: edit the file, then scan and review again)
python3 ${CLAUDE_SKILL_DIR}/deidentify.py apply ./deidentified/reviewed_report.json
```

**Options:**
- `--locale CODE`: Country locale for PHI patterns (kr, us, jp, cn, de, uk, fr, ca, au, in, it, es)
- `--locale-file PATH`: Custom locale JSON file (copy `locales/_template.json` to create one)
- `--auto-accept-safe`: Keep SAFE columns without showing them. Not recommended: SAFE means
  no pattern matched, not that the column holds no identifiers (a name typed into a short
  comment matches no pattern), and this option means nobody looks at those columns
- `--hash-mapping`: Store unkeyed SHA-256 hashes instead of original names/IDs in the mapping
  file. Dates and numeric IDs can be recovered from such hashes by trying every candidate, so
  mapping.json stays restricted either way
- `--mapping-in PATH` (`apply`/`full`): Reuse a previous run's mapping.json, so values already
  de-identified keep the same pseudonym, ID, and date offset. Without it, every run numbers
  from 0001: merging outputs from separate runs silently gives two patients the same pseudonym
  (and one patient different ones). A `--hash-mapping` file cannot be reused this way, since
  its keys are hashes, not original values. An imported mapping is validated before use: all
  three sections must be present, replacement values must be P/ID-plus-digits, unique within
  their section, and different from the original they replace, duplicate JSON keys are rejected,
  and date offsets must be within 1..365 days (never 0). Row-keyed offsets from
  a run with no patient key are not carried across runs, and a batch with nothing to anonymize
  keeps the prior mapping instead of erasing it. With or without it, a new pseudonym or ID skips
  every original value in the anonymized columns, the patient key column and the prior mapping,
  so a source ID already shaped like `ID0002` is never written out unchanged. An earlier batch
  cannot see values that arrive later: if source IDs share the P/ID-plus-digits form, an earlier
  batch's pseudonym can still equal a later batch's original value
- `--output-dir`: Where to save de-identified file, mapping, and audit log
- `-v/--verbose`: Enable debug logging

### Phase 3: Interactive Review Guidance

The script's terminal review has three passes:

1. **Pass 1 — Column Classification**: Each column is shown as PHI / REVIEW_NEEDED / SAFE,
   with sample values. The researcher confirms or overrides each classification.
2. **Pass 2 — Undecided Items**: Columns that weren't resolved in Pass 1 get a second look
   with more sample values displayed.
3. **Pass 3 — Final Summary**: A table of all planned actions. The researcher can edit
   individual decisions before confirming.
4. **Patient key** (only when dates will be shifted): the researcher picks the column that
   identifies the patient, or `row` if every row is a different patient. Each patient gets

From Aperivue/medsci-skills, MIT.

More from Aperivue/medsci-skills