Scrape Webpages at regular intervals automatically with python-CodePudding

How can I monthly scrape a website for products' prices and send an email if there are any changes in the prices? Is there's any way to do that automatically using Python? What libraries should I use?

CodePudding user response：

You can use crontab or airflow and set your scraper to run every month.

CodePudding user response：

You should use job queues to do this and in the simplest way, you can use RQ.

from redis import Redis
from rq import Queue
from datetime import timedelta

scrap_queue = Queue(connection=Redis())

def scrap_some_data():
    ... # my scraper code in here
def main():
    scrap_queue.enqueue_in(timedelta(minutes=30), scrap_some_data)
if __name__ == '__main__':
    main()

scrap some data function will call every 30 minutes.