data-science

Files (4)

• SKILL.md
• data-science-expert.md · references/
• ml-pipelines.md · references/
• statistical-methods.md · references/

SKILL.md (first lines)

# Data Science Expert

Comprehensive data science frameworks for analytics, machine learning, and data-driven decision making.

## Data Strategy

### Data Maturity Model

| Level | Name                | Characteristics                           |
| ----- | ------------------- | ----------------------------------------- |
| 1     | **Ad Hoc**          | Manual, inconsistent, siloed              |
| 2     | **Opportunistic**   | Some automation, point solutions          |
| 3     | **Systematic**      | Defined processes, governance emerging    |
| 4     | **Differentiating** | Data-driven decisions, advanced analytics |
| 5     | **Transformative**  | AI-first, competitive advantage           |

### Analytics Value Chain

```
DATA → INFORMATION → INSIGHT → ACTION → VALUE

PROGRESSION:
Descriptive: What happened?
Diagnostic: Why did it happen?
Predictive: What will happen?
Prescriptive: What should we do?
Autonomous: Self-optimizing systems
```

## Statistical Analysis

### Descriptive Statistics

```
CENTRAL TENDENCY:
- Mean: Sum / Count (sensitive to outliers)
- Median: Middle value (robust to outliers)
- Mode: Most frequent value

DISPERSION:
- Range: Max - Min
- Variance: Average squared deviation
- Standard Deviation: √Variance
- IQR: Q3 - Q1 (robust)

DISTRIBUTION SHAPE:
- Skewness: Asymmetry (0 = symmetric)
- Kurtosis: Tail heaviness (3 = normal)
```

For detailed inferential statistics and hypothesis testing, see [Statistical Methods Reference](references/statistical-methods.md).

## Machine Learning

### Algorithm Selection

| Task                         | Algorithms                                                   | When to Use                      |
| ---------------------------- | ------------------------------------------------------------ | -------------------------------- |
| **Classification**           | Logistic Regression, Random Forest, XGBoost, Neural Networks | Categorical outcomes             |
| **Regression**               | Linear Regression, Ridge/Lasso, Random Forest, XGBoost       | Continuous outcomes              |
| **Clustering**               | K-Means, Hierarchical, DBSCAN                                | Group discovery                  |
| **Dimensionality Reduction** | PCA, t-SNE, UMAP                                             | Feature reduction, visualization |
| **Anomaly Detection**        | Isolation Forest, One-Class SVM, Autoencoders                | Outlier detection                |
| **Time Series**              | ARIMA, Prophet, LSTM                                         | Sequential data                  |
| **Recommendation**           | Collaborative Filtering, Content-Based, Matrix Factorization | Personalization                  |
| **NLP**                      | Transformers, BERT, GPT                                      | Text understanding/generation    |

For detailed ML pipelines, feature engineering, and model monitoring, see [ML Pipelines Reference](references/ml-pipelines.md).

## Data Governance

### Data Governance Framework

```
GOVERNANCE PILLARS:

POLICIES:
- Data ownership
- Data classification
- Data retention
- Data access
- Data quality standards

ROLES:
- Data Owner: Accountable for data domain
- Data Steward: Day-to-day quality management
- Data Custodian: Technical implementation
- Data Consumer: End user

PROCESSES:
- Data cataloging
- Metadata management
- Data lineage
- Issue resolution
- Change management

METRICS:
- Data quality scores
- Policy compliance
- Data access requests
- Issue resolution time
```

### Data Quality Dimensions

| Dimension        | Definition                        | Measurement               |
| ---------------- | --------------------------------- | ------------------------- |
| **Accuracy**     | Correct representation of reality | % records matching source |
| **Completeness** | All required data present         | % non-null values         |
| **Consistency**  | Same across systems               | % matching across sources |
| **Timeliness**   | Available when needed             | Latency, freshness        |
| **Validity**     | Conforms to format/rules          | % passing validation      |
| **Uniqueness**   | No unwanted duplicates            | Duplicate rate            |

## Business Intelligence

### BI Architecture

```

From travisjneuman/.claude, MIT.

More from travisjneuman/.claude