threads-keyword-search

Files (2)

• SKILL.md
• extract-search-results.py · scripts/

SKILL.md (first lines)

# Threads — Keyword & Hashtag Search

> keyword + sort filter → list of matching posts with engagement metrics

## Language

All process output to user (progress updates, process notifications) follows the user's language.

## Objective

Search Threads for posts matching a keyword or hashtag and extract the results from SSR-embedded JSON, with support for top/recent sort ordering.

## Prerequisites

- A browser is open and connected via browser-act
- No login required for public search results (unauthenticated: typically 17-18 results per query, no pagination)

## Pre-execution Checks

### 1. Tool Readiness

If browser-act has been confirmed available in the current session → skip this step.

Invoke `browser-act` via Skill tool to load usage. If installation or configuration issues arise, follow its guidance to resolve then retry.

## Capability Components

> This Skill's operational boundary = what the user can manually do in their browser. It only reads data already displayed to the user on the page. JS code is encapsulated in Python files under the `scripts/` directory, invoked via `eval "$(python scripts/xxx.py {params})"`. `$(...)` is bash syntax; it is recommended to use the bash tool for execution.

### SSR: Extract keyword search results

Navigate to the search page with the keyword and sort filter, then extract all results embedded in the initial HTML:

1. `navigate https://www.threads.com/search/?q={keyword}&serp_type={filter}`
   - `{keyword}`: URL-encoded search term or hashtag (e.g., `AI` or `%23AI` for `#AI`)
   - `{filter}`: `top` (default, most relevant) or `recent` (chronologically newest)
2. `wait stable`
3. `eval "$(python scripts/extract-search-results.py '{keyword}' --filter {filter})"`

Output example:
```json
{
  "keyword": "AI",
  "filter": "top",
  "posts": [
    {
      "id": "3936653356768062022",          // post internal ID (pk)
      "code": "DZyvcdEl-W1",               // post short code for URL
      "url": "https://www.threads.com/@flower_popy/post/DZyvcdEl-W1",
      "text": "AI is changing everything...", // post text content, null if no caption
      "taken_at": 1781926571,              // Unix timestamp of post creation
      "like_count": 34,                    // number of likes
      "reply_count": 6,                    // number of direct replies
      "repost_count": 0,                   // number of reposts
      "quote_count": 0,                    // number of quote posts
      "is_reply": false,                   // true if this post is a reply
      "media_type": 19,                    // 1=photo, 2=video, 8=carousel, 19=text-only
      "has_media": false,                  // true if post contains image/video/carousel
      "user": {
        "pk": "14803522782",
        "username": "flower_popy",
        "full_name": "Flower",
        "is_verified": false
      }
    }
  ],
  "count": 17,
  "page_info": {
    "end_cursor": null,
    "has_next_page": false,
    "has_previous_page": false,
    "start_cursor": null
  }
}
```

Error handling: If `error: true` is returned, check that the search page loaded correctly by verifying the URL contains the query parameter. If `searchResults not found`, the page may have failed to render SSR data — retry `navigate` and `wait stable` once. If `count: 0`, no posts matched the keyword.

### SSR: Hashtag search

Hashtag search uses the same URL pattern. Prefix keyword with `#`:

1. `navigate https://www.threads.com/search/?q=%23{hashtag}&serp_type={filter}`
   - Example: `%23AI` for `#AI`
2. `wait stable`
3. `eval "$(python scripts/extract-search-results.py '#AI' --filter top)"`

The `#` prefix in the keyword argument is for labeling only; the URL encoding `%23` is what Threads uses to identify hashtag searches.

## Enum Parameters

[DOM] filter — values: `top` (most relevant/popular posts), `recent` (newest posts first). Set via `--filter` argument and `serp_type` URL parameter.

## Pagination

Pagination is not available for unauthenticated access on the search endpoint (`has_next_page: false` is returned regardless of result volume). Results are limited to approximately 17-18 posts per query without login. For broader coverage, run multiple searches with related keywords.

## Success Criteria

`result.count >= 1` and `result.posts[0].id != null` and `result.posts[0].user.username != null`

## Known Limitations

- Unauthenticated access: no pagination; approximately 17-18 results per keyword
- Date filtering is not supported as a URL parameter; filter by `taken_at` (Unix timestamp) client-side after extraction
- Private account posts may appear in search results but their full content may be restricted
- Search index may not include very new posts (lag of minutes to hours)

## Execution Efficiency

- **Batch orchestration**: Write a bash script to loop keywords serially within a single session. Add 1-2 second intervals between searches. For higher throughput, distribute keywords across multiple parallel browser sessions.
- **Test before batch execution**: Test with 1-2 keywords first before running full batch.
- **Reduce redundant pre-operations**: Each keyword requires a fresh `navigate` to the search URL — the search parameters are baked into the URL.
- **Error resumption**: Save results per keyword; on failure, resume from the breakpoint.

## Experience Notes

Path: `{working-directory}/browser-act-skill-forge-memories/threads-scraper-threads-keyword-search.memory.md`

From browser-act/skills, MIT.

More from browser-act/skills