python urllib library : the basics

               urllib and urllib2 are Python modules for fetching URLs. It offers a very simple interface, in the form of the urlopen function which is capable of fetching URLs. It also offers a  slightly more complex interface for  handling features like basic authentication, cookies, proxies and soon.

urllib2 supports fetching URLs for many URL schemes  such as ftp, http -- for example “ftp” is the URL scheme of “ftp://python.org/“

Fetching URLs : urlopen()

 
Now it is the time to play off the theories. The simplest way to use urllib2 to open urls is as follows:

  import urllib2
  response = urllib2.urlopen(’http://python.org/’)
  html = response.read()

You can use the above method to store the complete contents of a file into a variable without writing it to a local file... 

Sending Data along with url 

                Sometimes you want to send data to a URL (often the URL will refer to a CGI script or other web application). With HTTP, this is often done using  a GET/POST request. This is what your browser does when you submit a HTML form that you filled in on the web. In the common case of HTML forms, the data needs to be encoded in a standard way, and then passed to the Request object as the data argument. In python encoding is done using a function from the urllib library.

  import urllib

  import urllib2
  url = ’http://www.someserver.com/cgi-bin/register.cgi’
  values = {’name’ : ’Michael Foord’,
                             ’location’ : ’Northampton’,
                             ’language’ : ’Python’ }
  data = urllib.urlencode(values)
  req = urllib2.Request(url, data)
  response = urllib2.urlopen(req)
  the_page = response.read()

This can be done manually as follows :

  import urllib2
  import urllib
  data = {}
  data[’name’] = ’Somebody Here’
  data[’location’] = ’Northampton’
  data[’language’] = ’Python’
  url_values = urllib.urlencode(data)
  print url_values
  name=Somebody+Here&language=Python&location=Northampton
  url = ’http://www.example.com/example.cgi’
  full_url = url + ’?’ + url_values
  data = urllib2.open(full_url)

Fetching files : urlretrieve()

We can use urllib's urlretrieve function to download a file over internet and save it in a new name. We can download text / image / video files using this function if the source file is available as a url.

 import urllib

 url=https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAEaU33AHSR1u5v4ulLwW6ll9sIy5j7GbCoFQkC6bXbnl8EQrf8NPjlVbeC_6OShG00vi0R6sT_GTXkueNsXUnZEkq2IcLwyRmIQke2JzeKrh7lWd9YR5Sjob3a9AZsfuh4ENpmt1MT8Wy/s104/ubuntu.png 

 urllib.urlretrieve(url, new_image.png) 

Hope you have enjoyed the basics of the urllib library of python...