# Ramfangzauva by Lushai Dev > Enterprise Technical SEO Intelligence Platform, Recursive Web Crawler & AI Agent Search Engine Ramfangzauva (https://lushai.dev) is an enterprise-grade web intelligence platform, technical SEO diagnostic ecosystem, and AI scraping engine created and engineered by Lushai Dev. It provides real-time multi-threaded site auditing, autonomous recursive web crawling, LLM-ready markdown extraction, SERP keyword tracking, and website security vulnerability scanning without relying on any external paid APIs. ## Chanchin & Identity (The Story of Ramfangzauva) ### Meaning & Heritage In Mizo linguistic heritage, "Ramfangzauva" translates to "The Boundless Voyager who Roams and Explores Every Territory Widely" (from *Ram* meaning land or territory, *Fang* meaning to traverse or explore, and *Zauva* meaning expansive and without boundaries). Historically in the lush mountains of Mizoram, a voyager who traversed uncharted valleys, guarded frontiers, and mapped unfamiliar terrains was known for endurance, wisdom, and relentless observation. ### Mission & The Vision of Lushai Dev Channeling this spirit of boundless exploration, **Lushai Dev** (https://lushai.dev) engineered **Ramfangzauva** to conquer the modern web. In an era where web data is locked behind expensive, fragmented services (such as Semrush for SEO audits, Ahrefs for backlinks, Firecrawl for LLM scraping, Tavily for AI search, and Snyk for security headers), Ramfangzauva unifies all these capabilities into a single, high-performance, native ecosystem. Built with Next.js 15 App Router, React 19, Supabase Postgres, Playwright, and an asynchronous multi-worker Node.js crawler matrix, Ramfangzauva delivers enterprise intelligence with zero artificial lag, strict data privacy, and state-of-the-art developer ergonomics. --- ## Core Platform Capabilities 1. **Deep Technical SEO Audit Suite**: Semrush-grade audit covering 50+ on-page signals, Core Web Vitals, canonical links, schema markup, OpenGraph, and sitemap health. 2. **AI Agent Search Engine**: Tavily & Exa alternative providing real-time Google search results formatted as pristine LLM markdown excerpts, sitelinks, and People Also Ask. 3. **Selective LLM Web Scraper**: Firecrawl alternative extracting clean markdown with selective CSS inclusion/exclusion (`includeTags` / `excludeTags`), full-page screenshots, and dynamic SPA JavaScript rendering. 4. **Automated Client Browser Actions**: Playwright-powered browser interactions (`wait`, `waitFor`, `click`, `scroll`, `write`) executed before extracting rendered DOM content. 5. **Autonomous Recursive Domain Crawler**: Deep crawler with persistent deduplicated URL queues, path regex filtering (`includePaths` / `excludePaths`), sitemap discovery strategies, and asynchronous webhook notifications. 6. **Website Security & Vulnerability Auditor**: Automated scan for SSL/TLS certificates, OWASP security headers (HSTS, CSP, X-Frame-Options), cookie flags (Secure, HttpOnly, SameSite), exposed sensitive files (`.env`, `.git`), and DNS SPF/DMARC email security records. 7. **SERP Keyword Rank Tracker**: Tracks top 100 ranking positions, search volume estimates, search intent, and featured snippets across Google and Bing. 8. **Broken Links & Redirect Detector**: High-speed scanner inspecting internal & external page links, identifying 404/410/500 errors, redirect chains (301/302), response latency, and anchor text. 9. **Keyword Matrix & SEO Density Engine**: Extracts 1-word, 2-word, and 3-word n-gram keyword frequency, prominence scores in titles/headings, search intent classification, and content optimization signals. 10. **Competitor SEO Benchmark & Gap Tracker**: Side-by-side technical SEO comparison between two apex domains, analyzing shared keyword overlaps, performance gaps, and SERP competitive positions. 11. **Visual Sitemap & Robots Directive Inspector**: Hierarchical XML sitemap tree parser, nested sitemap indexes, robots.txt directives syntax validator, and indexation status checks. 12. **Backlink Topology & Referring Domain Graph**: Interactive topology graph tracking inbound backlinks, referring domains, anchor text distribution, and dofollow vs nofollow authority ratios. 13. **Interactive Graph Explorer & Node Visualizer**: Interactive deep crawl node visualization for internal page link architectures, crawl depth levels, and orphan page detection. 14. **Asynchronous Crawler Matrix & Worker Queue**: Multi-worker background crawling matrix with live concurrency controls, task scheduling, and real-time streaming telemetry. 15. **Google SERP Pixel Width & Snippet Simulator**: Real-time Google typography pixel meter measuring title (600px desktop / 680px mobile) and snippet description (960px / 1200px) cutoff, keyword bolding, and CTR optimization. 16. **Server Access Log File Analyzer & Crawl Budget Inspector**: Server log parser (Nginx, Apache, Cloudflare, Caddy) identifying Googlebot, Bingbot, and AI crawlers (GPTBot, ClaudeBot), measuring crawl waste (4xx/5xx errors), and providing crawl budget optimization. 17. **Content Readability & Text Quality Engine**: Flesch Reading Ease score, Flesch-Kincaid Grade Level, Gunning Fog Index, semantic H1-H6 heading hierarchy outline validator, run-on sentence detector, and text-to-HTML ratio. 18. **Internal PageRank & Click Depth Simulator**: Markov power-iteration PageRank algorithm (damping factor d=0.85) calculating internal link juice flow, BFS click depth distribution levels, orphan page detection (0 inlinks), and dead-end dangling page audits. 19. **Robots.txt Sandbox & RFC 9309 Crawl Simulator**: Interactive sandbox testing any URL path against robots.txt directives across Googlebot, Bingbot, GPTBot, and ClaudeBot with RFC 9309 longest-match syntax resolution and multi-bot permissions matrix. 20. **Schema.org Rich Snippets & JSON-LD Validator**: Comprehensive parser for JSON-LD and Microdata validating structured markup against official Google Rich Results guidelines for Products, FAQ accordions, Articles, and Breadcrumbs with SERP preview simulations. --- ## Developer REST API Endpoints (v1 & SEO Diagnostics) Base URL: `https://ramfangzauva.com` ### 1. AI Search Engine (Multi-Engine SERP Matrix) - **Path**: `POST /api/v1/search` & `GET /api/v1/search` - **Description**: Real-time web search for AI agents with clean LLM markdown excerpts, People Also Ask, sitelinks, and multi-engine aggregation (DuckDuckGo, Bing, Google). - **Parameters**: - `query` (string, required): Search query or question. - `maxResults` (number, optional, default: 10): Max search results (1 - 50). - `engine` (string, optional, default: "auto"): "auto" (resilient fallback matrix), "multi" (parallel multi-provider merge), "duckduckgo", "bing", "google". - `includeDomains` (string[], optional): Filter results to specific domains (e.g. `["github.com", "nextjs.org"]`). - `excludeDomains` (string[], optional): Omit results matching domains (e.g. `["pinterest.com"]`). - `timeRange` (string, optional, default: "any"): "any", "day", "week", "month", "year". - `searchDepth` (string, optional, default: "basic"): "basic" or "advanced". - `includeContent` (boolean, optional, default: true): Extract clean Markdown body per search result. - `extractDepth` (number, optional, default: 3): Top N results to fetch full Markdown for (1 - 10). - `country` (string, optional, default: "us"): Two-letter country code. - **Enhanced Response Fields**: - `bm25Score` (number): 0-100% pure TypeScript BM25 semantic query relevance confidence score. - `knowledgeGraph` (object | null): Encyclopedic entity facts card from Wikipedia Open Knowledge API with thumbnail and extract. - `features.peopleAlsoAsk` (string[]): Real-time live Google Suggest question interrogation. - `sourcesUsed` (string[]): Upstream providers queried (DuckDuckGo, Bing, Google News, Wikipedia). ### 2. Pure Code Web Page Distiller (Tavily Extract & Readability Alternative) - **Path**: `POST /api/v1/extract` & `GET /api/v1/extract?url=...` - **Description**: Trafilatura & Mozilla Readability pure TypeScript distillation converting any live web page into pristine, high-density LLM markdown with text-density heuristics, NLP key takeaways, headings outline, and reading time estimation. - **Parameters**: - `url` (string, required): Valid target website URL. - `maxLength` (number, optional, default: 8000): Maximum markdown length. - `includeLinks` (boolean, optional, default: true): Retain clean context links. - `extractTakeaways` (boolean, optional, default: true): Generate 3-4 NLP executive key takeaways. - **Path**: `POST /api/v1/scrape` - **Description**: Scrape web pages into pristine LLM Markdown, raw HTML, links, images, and metadata. - **Parameters**: - `url` (string, required): Target website URL. - `formats` (string[], optional, default: ["markdown", "metadata"]): Desired outputs: "markdown", "metadata", "links", "images", "html". - `onlyMainContent` (boolean, optional, default: true): Strip navbars, footers, and ads. - `includeTags` (string[], optional): CSS selectors to isolate (e.g. ["article", ".main-post"]). - `excludeTags` (string[], optional): CSS selectors to remove (e.g. ["nav", "footer", ".ad-banner"]). - `renderJs` (boolean, optional, default: false): Enable headless Playwright for SPAs (React/Vue/Angular). - `screenshot` (boolean, optional, default: false): Capture base64 JPEG screenshot. - `fullPage` (boolean, optional, default: false): Capture full-length scrollable height screenshot. - `actions` (array, optional): Automated browser actions before scraping (wait, waitFor, click, scroll, write). ### 3. Website Image Harvester & Asset Scraper (Firecrawl Alternative) - **Path**: `POST /api/v1/images/extract` & `GET /api/v1/images/extract?url=...` - **Description**: Firecrawl-grade enterprise image scraper that extracts, audits, and harvests every image asset from any webpage (standard ``, lazy-loaded `data-src`/`data-original`, responsive `` & ``, CSS inline `background-image`, social `og:image`/`twitter:image`, and favicons), resolving all relative URLs into clean absolute HTTPS URLs. - **Parameters**: - `url` (string, optional): Target public website URL to scrape images from. - `html` (string, optional): Raw HTML string markup to extract images from without network fetching. - `includeBackgrounds` (boolean, optional, default: true): Extract inline CSS `background-image: url(...)` assets. - `includeIcons` (boolean, optional, default: true): Extract favicons and apple-touch-icons. - `includeOpenGraph` (boolean, optional, default: true): Extract social preview images (`og:image`, `twitter:image`). - `minWidth` (number, optional): Filter out tiny assets smaller than this pixel width. - `minHeight` (number, optional): Filter out tiny assets smaller than this pixel height. - `formatsFilter` (string[], optional): Filter results by format (`["webp", "png", "svg", "jpg"]`). - **Output Fields**: `targetUrl`, `summary` (total, unique, formats breakdown, categories), `images` (array of items with `url`, `alt`, `width`, `height`, `format`, `category`, `sourceTag`), and `allImageUrls` (flat list of absolute URLs for 1-click bulk export). ### 4. Concurrent Batch Scraper - **Path**: `POST /api/v1/batch` - **Description**: Concurrently scrape up to 10 URLs in parallel with aggregated Markdown and metadata. - **Parameters**: - `urls` (string[], required): Array of target URLs (max 10). - `formats` (string[], optional): Desired outputs: ["markdown", "metadata"]. - `onlyMainContent` (boolean, optional, default: true): Strip headers and footers. - `includeTags` (string[], optional): CSS selectors to isolate across all URLs. - `excludeTags` (string[], optional): CSS selectors to remove across all URLs. - `renderJs` (boolean, optional, default: false): Dynamic SPA rendering. ### 4. Recursive Domain Crawler - **Path**: `POST /api/v1/crawl` - **Description**: Recursively crawl internal links across an apex domain, producing deduplicated Markdown pages. - **Parameters**: - `url` (string, required): Root seed URL. - `limit` (number, optional, default: 10): Max pages to crawl (up to 30 for API, 100 for enterprise engine). - `maxDepth` (number, optional, default: 2): Max crawl depth from seed (1 - 4). - `includePaths` (string[], optional): Regex/path prefixes to crawl (e.g. ["/blog/.*", "/docs/.*"]). - `excludePaths` (string[], optional): Regex/path prefixes to skip (e.g. ["/admin/.*", "/cart/.*"]). - `sitemap` (string, optional, default: "include"): Strategy: "include", "skip", or "only". - `ignoreQueryParameters` (boolean, optional, default: false): Deduplicate canonical URLs by ignoring query strings. - `scrapeOptions` (object, optional): Scrape options per page ({ onlyMainContent, includeTags, excludeTags }). - `webhook` (object, optional): Async callback ({ url: "https://your-server.com/hook", headers?: {} }). ### 5. Fast Site Mapper - **Path**: `POST /api/v1/map` - **Description**: Ultra-fast discovery of all sub-URLs and sitemaps on a target domain without downloading heavy HTML. - **Parameters**: - `url` (string, required): Target domain URL. - `limit` (number, optional, default: 100): Maximum URLs to discover. - `search` (string, optional): Substring filter. ### 6. Structured Entity Extractor - **Path**: `POST /api/v1/extract` - **Description**: Extracts contact info (emails, phones, social links), OpenGraph metadata, JSON-LD schema, pricing tiers, and tech stack. - **Parameters**: - `url` (string, required): Target website URL. ### 7. Technical SEO Auditor (Semrush Alternative) - **Path**: `POST /api/v1/audit` - **Description**: 50+ on-page SEO diagnostic checks with Core Web Vitals, canonical validation, and indexability signals. - **Parameters**: - `url` (string, required): Target URL to audit. ### 7B. Deep Site Crawler & Technical SEO Audit Engine (Screaming Frog Alternative) - **Path**: `POST /api/v1/audit/deep` & `GET /api/v1/audit/deep` - **Description**: Screaming Frog SEO Spider alternative (100% pure code). Recursively crawls domain internal links, computes 64-bit SimHash near-duplicate content clusters via pairwise Hamming distance, simulates Core Web Vitals DOM tree depth and CLS image metrics, and computes internal PageRank link equity flow with orphan page detection. - **Parameters**: - `url` (string, required): Target root seed website URL (e.g. "https://example.com"). - `limit` (number, optional, default: 15): Maximum pages to crawl (up to 30). - `maxDepth` (number, optional, default: 2): Crawl depth level from seed (1 - 4). - `detectDuplicates` (boolean, optional, default: true): Enable 64-bit SimHash pairwise duplicate detection. - **Response Fields**: - `siteHealthScore` (number): Overall site technical health score (0 - 100) and letter grade (A+, A, B, C, D, F). - `duplicateClusters` (array): Near-duplicate content groups with SimHash hex fingerprints, pairwise Hamming distances, similarity scores, and canonical recommendations. - `linkEquityGraph` (object): Internal PageRank distribution (Markov model d=0.85), mapped link edges, top authority hub pages, and orphan pages (0 internal inlinks). - `issuesSummary` (object): Critical errors, warnings, notices, and affected URLs breakdown. - `pages` (array): Itemized crawled pages with status codes, latency, depth, canonical validation, word counts, text-to-HTML ratios, SimHashes, and Core Web Vitals DOM metrics. ### 8. Audit Quota Telemetry - **Path**: `GET /api/v1/audit` - **Description**: Returns 24-hour IP quota status, used count, and reset timestamp without consuming credits. ### 9. Website Security Auditor - **Path**: `POST /api/v1/security` (also supports `GET /api/v1/security?url=...` and quota check `GET /api/v1/security?checkQuota=true`) - **Description**: Deep live vulnerability & cryptographic security assessment with zero fake results. Probes live TLS peer certificates (SAN hostname matching, TLSv1.3, AEAD cipher, key bits/ECC curve, validity window), OWASP and modern isolation headers (HSTS preload, CSP, X-Frame-Options, Cross-Origin-Opener-Policy COOP, Cross-Origin-Resource-Policy CORP, X-Content-Type-Options, Referrer-Policy, Permissions-Policy), live CORS preflight probe (origin reflection & credential theft detection), deep sensitive file exposure (.env, .git/HEAD, database.sql, and binary ZIP magic byte verification with zero SPA false positives), Subresource Integrity (SRI) cryptographic script scan, Insecure Mixed Content detection, itemized cookie flags (Secure, HttpOnly, SameSite), and DNS security with cryptographic DNSSEC live validation via DoH, SPF strictness (-all / ~all / +all), and DMARC enforcement (p=reject / p=quarantine / p=none). - **Parameters**: - `url` (string, required in POST body or GET query): Target website URL. - `userId` (string, optional): Authenticated user ID for account monthly credit quota tracking. ### 10. SERP Keyword Rank Checker & Features Matrix (SerpApi Alternative) - **Path**: `POST /api/v1/serp/matrix` & `GET /api/v1/serp/matrix` (also `POST /api/v1/serp` & `GET /api/v1/serp`) - **Description**: Bot-resilient Google & Bing SERP position tracker with mobile user-agent rotation, client-hints emulation, and full SERP feature extraction (Featured Snippets pos #0, PAA accordions, Knowledge Graph, Video Carousel, Sitelinks, CTR Visibility score model, and Top 100 Organic Ranking Matrix) without paid proxies or external APIs. - **Parameters**: - `keyword` (string, required): Search query or target keyword. - `domain` (string, required): Target website domain (e.g. "zolinks.in"). - `device` (string, optional, default: "desktop"): "desktop" or "mobile" (emulates Pixel 8 Android 14 / iPhone 15 iOS 17). - `engine` (string, optional, default: "google"): "google", "bing", or "hybrid" (resilient multi-tier fallback). - `country` (string, optional, default: "us"): Country geolocation code. - `language` (string, optional, default: "en"): Language code. - `numResults` (number, optional, default: 100): Maximum rankings to fetch (1 - 100). - **Response Fields**: - `rank` (number | null): Organic ranking position (1-100) or null if unranked. - `visibilityScore` (number): Estimated Click-Through Rate percentage (0-100% based on Google position CTR curves). - `ownsFeaturedSnippet` (boolean): Whether target domain captures position zero. - `features` (object): Featured snippet text & URL, People Also Ask questions with snippet answers, Knowledge Graph card, Video carousel items, Related searches. - `organicResults` (array): Complete top 100 search listings with titles, URLs, domains, snippets, sitelinks, and target domain indicators. - `topCompetitors` (array): Top 10 competing domains ranking above or adjacent to target. - `telemetry` (object): Engine used, stealth browser profile, device emulated, response time ms, and fallback status. ### 11. Broken Links & Redirect Scanner - **Path**: `POST /api/seo/broken-links` - **Description**: Scan any web page to identify 404/500 broken links, dead external references, and redirect chains. - **Parameters**: - `url` (string, required): Target page URL. - `domain` (string, optional): Apex domain filter. - `limit` (number, optional, default: 30): Max links to check. ### 12. Real-time Keyword Discovery & Keyword Magic Tool - **Path**: `POST /api/v1/keywords/discover` & `POST /api/seo/keywords` - **Description**: Semrush Keyword Magic Tool & Ahrefs Keyword Explorer alternative (100% pure code). Discovers hundreds of live Google Autocomplete suggestions using Alphabet Soup wildcards (a-z, 0-9, questions, prepositions) with algorithmic Keyword Difficulty (KD 0-100%), estimated Search Volume, Search Intent classification, and CPC heuristics. Also supports on-page URL keyword density audits. - **Parameters**: - `keyword` (string, required for discovery mode): Seed keyword query. - `url` (string, required for density mode): Target website URL. - `limit` (number, optional, default: 150): Maximum keywords to discover. - `country` (string, optional, default: "us"): Two-letter country code. ### 12B. People Also Ask (PAA) Question Trees - **Path**: `POST /api/v1/keywords/paa` & `GET /api/v1/keywords/paa` - **Description**: Extracts and clusters authentic Google search question branches into four structured intent clusters: Foundational & Concepts, How-To & Execution, Comparison & Versus, and Commercial & Costs. - **Parameters**: - `keyword` (string, required): Seed search topic. ### 13. Competitor SEO Benchmark Comparison - **Path**: `POST /api/seo/competitors` - **Description**: Head-to-head competitive technical SEO benchmark and keyword gap analysis between two domains. - **Parameters**: - `urlA` (string, required): Primary domain URL. - `urlB` (string, required): Competitor domain URL. ### 14. XML Sitemap Hierarchy Parser - **Path**: `POST /api/seo/sitemap` - **Description**: Parse XML sitemaps and nested sitemap indexes, extract loc entries, change frequencies, and optionally import to crawl queue. - **Parameters**: - `url` (string, required): XML sitemap URL. - `importToQueue` (boolean, optional, default: false): Ingest discovered URLs into crawler queue. ### 15. Google SERP Pixel Width & Snippet Preview - **Path**: `POST /api/v1/serp-preview` & `GET /api/v1/serp-preview` - **Description**: Real-time Google SERP pixel width measurement (Desktop 600px / Mobile 680px Title cutoff; 960px / 1200px Description cutoff) with live URL metadata extraction, keyword bolding, character counts, and 0-100 CTR optimization score. - **Parameters**: - `title` (string, optional): Page title text to measure. - `description` (string, optional): Snippet meta description text to measure. - `url` (string, optional): Target URL (fetches live title, description, favicon & breadcrumbs if provided). - `keyword` (string, optional): Target keyword for SERP bolding and placement audits. - `device` (string, optional, default: "desktop"): "desktop" or "mobile". ### 16. Server Access Log File Analyzer - **Path**: `POST /api/v1/log-analyzer` & `GET /api/v1/log-analyzer` - **Description**: Inspects raw server access logs (Nginx Combined, Apache Common/Combined, Cloudflare JSON, Caddy) to categorize Googlebot, Bingbot, and AI crawlers (GPTBot, ClaudeBot), calculate crawl waste on 4xx/5xx dead URLs, and generate Semrush-grade crawl budget recommendations. - **Parameters**: - `logContent` (string, optional): Raw access log lines (up to 50,000 lines). - `url` (string, optional): Remote URL pointing to access log file to fetch and analyze. - `sample` (boolean, optional, default: false): Load and analyze built-in sample server access logs. ### 17. Content Readability & Heading Tree Analyzer - **Path**: `POST /api/v1/readability` & `GET /api/v1/readability` - **Description**: Flesch Reading Ease (0-100), Flesch-Kincaid Grade Level, Gunning Fog Index, Coleman-Liau, semantic H1-H6 heading hierarchy outline tree, run-on sentence detector (>25 words), and text-to-HTML ratio. - **Parameters**: - `text` (string, optional): Raw article text, markdown, or HTML to analyze. - `url` (string, optional): Public webpage URL to scrape and audit in real time. - `sample` (boolean, optional, default: false): Analyze built-in sample SEO article text. ### 18. Internal PageRank & Click Depth Simulator - **Path**: `POST /api/v1/link-equity` & `GET /api/v1/link-equity` - **Description**: Semrush-grade internal link equity calculator. Computes Markov random walk PageRank (damping factor d=0.85), BFS click depth levels from root, orphan page identification (0 inlinks), and dead-end dangling page audits. - **Parameters**: - `url` (string, optional): Target website URL to crawl and simulate internal link equity. - `maxPages` (number, optional, default: 50): Maximum internal pages to crawl (up to 100). - `sample` (boolean, optional, default: false): Simulate link equity on pre-configured SaaS site architecture graph. - `rootUrl` (string, optional): Custom root URL when providing custom graph. - `pages` (array, optional): Custom array of `{ id, url, title }` page nodes. - `edges` (array, optional): Custom array of `{ source, target }` directed internal link edges. ### 19. Robots.txt Sandbox & Crawl Simulator - **Path**: `POST /api/v1/robots-test` & `GET /api/v1/robots-test` - **Description**: Evaluates any target URL path against robots.txt directives across 10+ standard Search Engines and AI agents (Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Applebot, etc.) using RFC 9309 longest-match syntax resolution. Supports live website fetching, raw text input, and syntax linting. - **Parameters**: - `testPath` (string, optional, default: "/"): Target URL path to test permissions for. - `url` (string, optional): Domain or URL to fetch remote robots.txt from. - `robotsContent` (string, optional): Raw robots.txt text content to analyze directly. - `sample` (boolean, optional, default: false): Analyze built-in sample SaaS robots.txt. ### 20. Schema.org Rich Snippets & JSON-LD Validator - **Path**: `POST /api/v1/schema-validator` & `GET /api/v1/schema-validator` - **Description**: Validates structured data against official Google Rich Results guidelines for Product, FAQPage, Article, Organization, and BreadcrumbList schemas. Computes compliance health score, flags critical missing properties, and provides live Google SERP rich snippet simulation. - **Parameters**: - `url` (string, optional): Target public webpage URL to scrape and audit. - `schemaContent` (string, optional): Raw JSON-LD or HTML string containing `