CrawlLit

An intelligent, multi-threaded web crawler with smart priority scoring for efficient content discovery. CrawlLit uses text-based relevance scoring to prioritize pages based on your search query, making it perfect for targeted web scraping and content discovery.

What the project includes

• ** Parallel Processing**: Multi-threaded crawling with configurable worker threads for maximum speed
• ** Smart Priority Scoring**: Text-based relevance scoring that prioritizes pages matching your query
• ** Content Filtering**: Only crawls pages with relevant content based on minimum priority thresholds
• ** Comprehensive Output**: Saves results in JSON format with detailed summaries and timestamps
• ** Real-time Progress**: Live progress tracking and comprehensive logging
• ** reliable Error Handling**: Graceful handling of network issues, timeouts, and malformed content
• ** Easy to Use**: Simple command-line interface with sensible defaults
• ** Cross-platform**: Works on Windows, Linux, and macOS

Requirements

• Python 3.7 or higher
• Internet connection
• Required packages (see requirements.txt):
  • requests >= 2.28.0
  • beautifulsoup4 >= 4.11.0
  • lxml >= 4.9.0

Quick Start

Installation

1. Clone this repository:

git clone https://github.com/Tuhin-SnapD/CrawlLit.git
cd CrawlLit

2. Install dependencies:

pip install -r requirements.txt

Basic Usage

# Basic crawl with a search query
python scraper.py -q "machine learning"

# Crawl 100 pages with 8 workers
python scraper.py -w 8 -p 100 -q "python programming"

# Start from a specific website
python scraper.py -u "https://example.com" -q "tutorial"

Windows Users

Double-click run.bat for a graphical menu interface that handles setup and provides quick start options!

Command Line Options

Option	Description	Default
-w, --workers	Number of worker threads	8
-p, --pages	Maximum pages to crawl	50
-q, --query	Search query for priority scoring (required)	-
-u, --url	Base URL to start crawling from	https://httpbin.org
--min-priority	Minimum priority score (0.0-1.0)	0.1
--output-dir	Output directory for results	crawl_results
--store-links-count	Include link counts in results	False

Examples

# High-performance crawl
python scraper.py -w 16 -p 200 -q "data science" -u "https://example.com"

# Broader search with lower priority threshold
python scraper.py -q "research papers" --min-priority 0.05

# Custom output location
python scraper.py -q "tutorials" --output-dir my_results

# Include link counts in results
python scraper.py -q "documentation" --store-links-count

How It Works

1. Initialization: The crawler starts with your specified base URL and search query
2. Priority Scoring: Uses text-based matching to score pages based on relevance to your query:
   • Counts occurrences of query words in page content
   • Calculates match density (matches per word)
   • Applies bonuses for exact phrase matches and title relevance
3. Parallel Crawling: Multiple worker threads crawl pages simultaneously from a priority queue
4. Link Discovery: Extracts links from crawled pages and prioritizes them based on anchor text and surrounding context
5. Result Storage: Saves crawled data with priority scores, timestamps, and metadata

Priority Scoring Algorithm

The crawler uses a simple but effective text-based scoring system:

score = (query_word_matches / total_words) + bonuses

Bonuses are applied for:

• Exact phrase matches (+0.5)
• Title/heading matches (+0.3)

Final scores are normalized to a 0.0-1.0 range, with pages scoring below --min-priority being filtered out.

Output Files

The crawler generates several output files in the specified results directory:

JSON Results (simple_crawl_results_YYYYMMDD_HHMMSS.json)

Complete crawl data including:

• URL
• Page title
• Content preview (first 1000 characters)
• Priority score
• Timestamp
• HTTP status code
• Optional link counts (if --store-links-count is enabled)

Summary Report (simple_crawl_summary_YYYYMMDD_HHMMSS.txt)

Human-readable summary including:

• Query and base URL
• Total pages crawled
• Workers used
• Top 20 results sorted by priority score

Log Files (crawler_YYYYMMDD_HHMMSS.log)

Detailed logging of:

• Crawling progress
• Errors and warnings
• Network requests
• Performance metrics

Performance Tips

1. Workers: Use 4-16 workers depending on your system capabilities and target website

   • More workers = faster crawling, but also more resource usage
   • Too many workers may trigger rate limiting or anti-bot measures

2. Pages: Start with smaller numbers (50-100) to test performance and relevance

3. Priority Threshold:

   • Lower values (0.05-0.1) for broader crawling
   • Higher values (0.2-0.3) for focused, high-relevance results

4. Query Specificity:

   • Use specific, relevant search terms for better results
   • Multi-word queries work best

5. Target Websites:

   • Be respectful of robots.txt and rate limits
   • Some websites may block aggressive crawlers

How the system is built

Core Components

• SimpleWebScraper: Main crawler class managing the crawling process
• CrawlResult: Dataclass for storing page data
• Priority Queue: Ensures high-priority pages are crawled first
• Thread Pool: Manages concurrent page requests
• Link Parser: Extracts and scores links from HTML content

Thread Safety

All shared data structures are protected with locks to ensure thread-safe operation:

• Scraped pages set
• Results list
• Priority queue operations

Examples

Example 1: Research Paper Discovery

python scraper.py -q "neural networks" -u "https://arxiv.org" -w 10 -p 100 --min-priority 0.15

Example 2: Product Search

python scraper.py -q "wireless headphones" -u "https://www.amazon.in" -w 8 -p 50

Example 3: Documentation Crawling

python scraper.py -q "api reference" -u "https://docs.example.com" -w 4 -p 200 --min-priority 0.2

Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Legal and Ethical Considerations

• Respect robots.txt: Check and respect the robots.txt file of target websites
• Rate Limiting: Be mindful of server resources; avoid aggressive crawling
• Terms of Service: Ensure your crawling activities comply with the website's terms of service
• Copyright: Respect copyright and intellectual property rights
• Personal Use: This tool is intended for legitimate research and personal use only

License

This project is open source and available under the MIT License.

Acknowledgments

• Built with BeautifulSoup for HTML parsing
• Uses requests for HTTP operations

Support

If you encounter any issues or have questions:

1. Check the Issues page
2. Create a new issue with detailed information about your problem
3. Include error messages, command used, and Python version

---

Made with for efficient web content discovery