literature-search-biorxiv

Files (3)

• SKILL.md
• search_by_dates.py · scripts/
• search_by_doi.py · scripts/

SKILL.md (first lines)

# bioRxiv and medRxiv Literature Search

## Prerequisites

1.  **`uv`**: Read the `uv` skill and follow its Setup instructions to ensure
    `uv` is installed and on PATH.
2.  **User Notification**: If LICENSE_NOTIFICATION.txt does not already exist in
    this skill directory then (1) prominently notify the user to check the terms
    at https://api.biorxiv.org/ and
    https://www.biorxiv.org/content/about-biorxiv and to always check the
    license of the papers retrieved by the skill for any restrictions, then (2)
    create the file recording the notification text and timestamp.

## Search Strategy Guide (Read First)

**This skill browses a date-based preprint archive. It is NOT a keyword search
engine.** Choose your approach based on what you already know:

-   **A DOI** (e.g., from a citation): Use `search_by_doi.py`. Fast and
    reliable.
-   **Approximate date + category**: Use `search_by_dates.py` with a 1–4 week
    range and `--category`.
-   **Only a topic or keywords, no date**: **Do NOT use this skill for
    discovery.** Use a keyword-capable literature skill first to find relevant
    DOIs, then return here to fetch metadata.

> **CRITICAL ANTI-PATTERN — Do NOT do this:** Do NOT attempt to search broad
> date ranges (months or years) with `--keywords` hoping to find a specific
> paper. The bioRxiv API does not support server-side keyword search. The script
> must download ALL metadata for the entire date range and filter locally in
> Python. Broad ranges will result in thousands of API calls, timeouts, and your
> request being blocked for API abuse. This is the #1 reason this skill fails.

## Core Rules

-   **Use the Wrapper**: ALWAYS execute the provided helper scripts to query the
    database rather than accessing the database directly. The scripts
    automatically enforce the required rate limit gracefully.
-   **Local Filtering (CRITICAL WARNING)**: Unlike arXiv, the bioRxiv API **does
    not support server-side keyword or author searches**. Keyword and author
    filtering is performed *locally* by the scripts after downloading all
    metadata for a specified date range. You **MUST** use narrow date ranges
    (e.g., 1-4 weeks) AND the `--category` filter when searching with
    `--keywords` or `--author`.
-   **Abstracts Excluded By Default**: To save context space in the resulting
    JSON, abstracts are stripped from the output by default. If you are
    searching by `--keywords` and want to read the abstracts of the resulting
    papers to understand their context, you **MUST** pass the
    `--include_abstracts` flag.
-   **Output Redirection**: Search commands output JSON arrays to standard
    output. Always redirect output to a file (e.g., `> results.json`) and parse
    the file separately.
-   **List Sources** If this skill is used, ensure this is mentioned in the
    output AND list the URLs of all papers that were used in producing the
    output.

## Utility Scripts

All tools enforce a cross-process rate limits and retry with backoff on failure.
To ensure you respect terms-of-service, do NOT write custom `curl` queries.

**Pagination:** The bioRxiv API returns results in pages of up to 100 papers.
The `search_by_dates.py` script automatically fetches all pages and reports
pagination progress to stderr (e.g., `[Page 2] Fetched 200/543 papers...`). The
JSON output to stdout contains the **complete** filtered result set across all
pages — no manual pagination is needed.

### 1. Search by Dates (`search_by_dates.py`)

Search for preprints within an explicit date range, optionally filtering by
category, keywords, or author.

```bash
# Broad category search over a 2-week period
uv run scripts/search_by_dates.py --server biorxiv \
  --start_date 2024-01-01 --end_date 2024-01-14 \
  --category neuroscience > results.json

# Deep keyword filtering using OR logic and including abstracts
uv run scripts/search_by_dates.py --server medrxiv \
  --start_date 2023-11-01 --end_date 2023-11-30 \
  --category infectious_diseases \
  --keywords "covid" "sars-cov-2" --match_logic OR \
  --include_abstracts > covid_papers.json

# Finding papers by a specific author in a narrow window
uv run scripts/search_by_dates.py \
  --start_date 2024-05-01 --end_date 2024-05-14 \
  --author "Smith" > smith_papers.json
```

*Required Arguments:*

-   `--start_date`: YYYY-MM-DD
-   `--end_date`: YYYY-MM-DD

*Optional Arguments:*

-   `--server`: `biorxiv` (default) or `medrxiv`
-   `--category`: A valid subject category (see below). **Highly recommended** —
    dramatically reduces the data the script must download and filter.
-   `--keywords`: List of strings to search in the title/abstract.
-   `--match_logic`: `AND` (default) or `OR` for keywords.
-   `--author`: Author name (case-insensitive string match).
-   `--include_abstracts`: Flag to include full abstracts in the JSON output.

### 2. Fetch Metadata by DOI (`search_by_doi.py`)

Retrieve the detailed JSON metadata for a single paper if you already know its
DOI. **This is the most reliable entry point.**

```bash
uv run scripts/search_by_doi.py --server biorxiv \
  --doi "10.1101/2023.08.15.551388" \
  --include_abstracts > paper_info.json
```

### Downloading Full-Text PDFs

From hanyanphysics/AI-for-Physics, Apache-2.0.

More from hanyanphysics/AI-for-Physics