urllib and urllib2 are Python modules for fetching URLs. It offers a very simple interface, in the form of the urlopen function which is capable of fetching URLs. It also offers a slightly more complex interface for handling features like basic authentication, cookies, proxies and soon.
urllib2 supports fetching URLs for many URL schemes such as ftp, http -- for example “ftp” is the URL scheme of “ftp://python.org/“
Fetching URLs : urlopen()
Now it is the time to play off the theories. The simplest way to use urllib2 to open urls is as follows:
import urllib2
response = urllib2.urlopen(’http://python.org/’)
html = response.read()
You can use the above method to store the complete contents of a file into a variable without writing it to a local file...
Sending Data along with url
Sometimes you want to send data to a URL (often the URL will refer to a CGI script or other web application). With HTTP, this is often done using a GET/POST request. This is what your browser does when you submit a HTML form that you filled in on the web. In the common case of HTML forms, the data needs to be encoded in a standard way, and then passed to the Request object as the data argument. In python encoding is done using a function from the urllib library.
import urllibimport urllib2
url = ’http://www.someserver.com/cgi-bin/register.cgi’
values = {’name’ : ’Michael Foord’,
’location’ : ’Northampton’,
’language’ : ’Python’ }
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page = response.read()
This can be done manually as follows :
import urllib2
import urllib
data = {}
data[’name’] = ’Somebody Here’
data[’location’] = ’Northampton’
data[’language’] = ’Python’
url_values = urllib.urlencode(data)
print url_values
name=Somebody+Here&language=Python&location=Northampton
url = ’http://www.example.com/example.cgi’
full_url = url + ’?’ + url_values
data = urllib2.open(full_url)
Fetching files : urlretrieve()
We can use urllib's urlretrieve function to download a file over internet and save it in a new name. We can download text / image / video files using this function if the source file is available as a url.
import urlliburl=https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAEaU33AHSR1u5v4ulLwW6ll9sIy5j7GbCoFQkC6bXbnl8EQrf8NPjlVbeC_6OShG00vi0R6sT_GTXkueNsXUnZEkq2IcLwyRmIQke2JzeKrh7lWd9YR5Sjob3a9AZsfuh4ENpmt1MT8Wy/s104/ubuntu.pngurllib.urlretrieve(url, new_image.png)
Hope you have enjoyed the basics of the urllib library of python...
