There are several examples of headless browser automation tools like Selenium for Firefox, Zombie.js, and Intoli’s Remote Browser. You don’t interact with visual elements on a screen but instead, use a Command-Line Interface to make requests and automate use-cases. It is referred to as headless because there is no Graphical User Interface to interact with. Headless browser automation is a way of leveraging the ability of your browser to render Javascript code for automating use-cases, like web scraping. This is where headless browser automation and Puppeteer come in. However, seeing as most web scraping tools are designed to capture HTML code and extract data from that, you run into the problem of how to extract website data that is rendered by Javascript code. Essentially, Javascript has become the language of modern websites. When your browser makes a request, the server usually brings a response of Javascript files injected into the HTML framework. Now, this might be good news for the average internet user, but for someone who is looking to perform tasks like automated web scraping, it’s a tad inconvenient.

Over the past decade, the internet has evolved from bare-bones websites built with ordinary HTML and CSS to having complex web apps with interactive user interfaces, built using frameworks like Angular or React, that is written with Javascript.

Puppeteer github download free