Scrapy-Based Website Link Crawler

This function uses the Scrapy library to crawl a website starting from a specified start_url. It extracts and prints all the links from the start_url.

Technology Stack : Scrapy, CSS selector

Code Type : Scrapy reptile

Code Difficulty : Intermediate

                
                    
def crawl_random_links(start_url):
    """
    This function uses Scrapy to crawl a website starting from a given start_url.
    It extracts all the links from the start_url and prints them.
    """
    import scrapy
    from scrapy.crawler import CrawlerProcess

    class MySpider(scrapy.Spider):
        name = "random_links_crawler"
        start_urls = [start_url]

        def parse(self, response):
            for link in response.css('a::attr(href)'):
                yield link.get()

    process = CrawlerProcess()
    process.crawl(MySpider)
    process.start()