Often a webmaster, marketer or SEO specialist needs to extract data from site pages and display it in a convenient form for further processing. This can be parsing prices in an online store, getting the number of likes, or extracting a content of reviews from resources of interest.
By default, most technical site audit programs collect only the contents of the H1 and H2 headers, however, if, for example, you want to collect the H5 headers, then they will already need to be extracted separately. And in order to avoid the routine manual work of parsing and extracting data from the HTML code of pages, they usually use the web scraper or parser.
The next logical step in scraping data from websites that change frequently (that is, an RSS feed displaying X items at a time) is to scrap it regularly. In the previous parsing example, we used the command line to execute code on command. However, this is not a scalable solution. To automate it, we'll add Celery to create a runtime task queue system.
Selenium is a framework designed to automate web application testing. We can write a Python script that automatically controls browser interactions such as clicking links and submitting forms. But in addition, Selenium will be very useful if we need to collect data from a page, the content of which is generated by JavaScript. In this case, the data is displayed after many Ajax requests.