Web Scrapping using python 3
Hello everyone, today I’m going to talk about web scraping using python 3. I have shared my sample code with you. Before we start the discussion, below code which I have implemented for javascript required websites. People who have heard about web scraping will know about what I’m mentioning. If javascript is not required, websites don’t want this kind of effort. And you may want geckodriver for that. I have mentioned the link for download geckodriver in here.Using this, you can save your content in a MySQL database, .csv file, or .txt file. I have used .txt for the example.
Note — All the code implemented with .findall() method depends on your website. Here I have used this development for UNICODE. if your website is in English language (ASCII) you don’t want those encode(‘utf-8’) and decode(‘utf-8’)
- lets we look on libraries we need ,
from bs4 import BeautifulSoup
import time
from selenium import webdriver
import MySQLdb
from datetime import datetime
import csv
BeautifulSoup library — pulling data out of HTML and XML files.
webdriver — WebDriver is a web automation framework that allows you to execute your tests against different browsers, not just Firefox, Chrome (unlike Selenium IDE). WebDriver also enables you to use a programming language in creating your test scripts
MySQLdb — connect with mysql database.
csv — Files in the CSV format can be imported to and exported from programs that store data in tables, such as Microsoft Excel or OpenOffice Calc.
2.lets look on our sample code.
then we looks on how we get data on <h> tags <a> tags,
above all code depend on your website.
.find_all() — Extracts a list of Tag objects that match the given criteria
3.lets save those content on .txt file ,
4.lets save those content on .csv file ,
5.lets save those content on a mySQL db ,
Now we have finished our discussion about web scrapping using python 3. Before you use this or refer this sometimes UTF-8 characters will not save on mySQL database.Those cases you will have to use mongodb.In my case it works fine.
6.Lets look on full sample code ,
Thank you all.