Obscura: a Rust headless browser without Chromium
Obscura is a Rust headless browser for AI agents and web scraping. It runs JavaScript without Chromium and exposes CDP for Puppeteer or Playwright.
Published entries across all sections carrying the “Web scraping” tag, newest first by publication date on this site.
15 entries
Obscura is a Rust headless browser for AI agents and web scraping. It runs JavaScript without Chromium and exposes CDP for Puppeteer or Playwright.
Moli Browser is a Rust headless browser for agent page extraction and automation. It reads the DOM by default, rendering layout only for geometry or pixels.
An Anthropic IETF draft (Oct 30) lets a site opt out of AI training via one robots.txt token; Microsoft honors it from Feb 1, 2027, OpenAI within months.
Discord's 63,000+ Discoverable Servers get public web pages Nov 16; from mid-December select messages reach search results — on by default, one-switch opt-out.
freehire crawls postings straight from company ATS platforms, deduplicates them and links to the original page; MIT site, extension, API or self-hosted.
Blackwall raised $100M at $2.2B valuation on October 20. It sells bot and AI-agent filtering through hosting providers, so site owners reach it via their host.
DeepSeek opened in-house Search and Fetch APIs on Oct 17 at $0.75 per 1,000 successful calls (5 yuan in Chinese docs); tool-call billing starts Nov 1.
Meta deleted the sentence that let it train on business-site content before Oct. 30, keeping the crawl authorization, test credentials and "develop."
catbus is a pure Node.js CLI that standardizes commands and JSON output across 10 platforms including TikTok, Bilibili, and X for human users and AI agents.
Firecrawl's official MCP server: agents turn pages into Markdown or schema-shaped JSON, crawl and map sites, and click before reading. Cloud or self-hosted.
A headless browser written from scratch in Zig with a built-in MCP server. Agents read pages as Markdown, click and fill forms without spinning up Chromium.
The Fetch server from the official MCP repo: one fetch tool that turns pages into Markdown and reads them in chunks. Free, keyless and minimal.
A self-hosted web research MCP server: no paid search key, local crawling and BM25 plus embedding rerank, returning only source-linked original passages.
Reddit ends RSS on November 13 and public API access in March 2027, restricts Old Reddit to logged-in users, citing large-scale scraping and abuse.
A self-hostable, TypeScript web scraping toolkit that covers everything from static pages to JS-rendered pages across three engines, with bulk search-result collection, full-site crawling, and LLM structured extraction, all outputting LLM-friendly data.