Design Image URL Crawler

Design a web scraping system to extract image URLs from a provided list of domain URLs.

Requirements:

• Accept list of root URLs with configurable crawl depth
• Crawl pages and extract image src attributes (not actual images)
• Track job status (submitted, in-progress, completed, failed)
• Return mapping of source URLs to extracted image URLs

API Endpoints:

• POST /jobs: Submit list of URLs, returns job_id
• GET /jobs/{job_id}/status: Returns {completed: N, in_progress: M, total: T}
• GET /jobs/{job_id}/results: Returns {url: [img_urls]} mapping

Key Components:

• URL Frontier (priority queue for BFS by depth)
• HTML Parser Service (extract img tags, links)
• Job Status Store (Redis/DB)
• Results Store (DB)
• Rate Limiter (per-domain politeness)
• Distributed Worker Pool

Discussion Points:

• How to handle robots.txt and crawl delays
• URL deduplication strategies
• Failure handling and retries
• Horizontal scaling of crawlers
• Depth-first vs breadth-first trade-offs