Cleaning up Data Scraped from the Web

Clean the HTML

HTML you find on the web can be atrocious. Here’s a quick function that can help. We make use of the `lxml`_ library. It’s very good at understanding broken HTML and will render a perfectly-formed page for your extractor functions.

You may be concerned that this is computationally wasteful. This is true, but it can reduce lots of the irritation of extracting specific information from messy HTML:

def clean_page(html, pretty_print=False):
    """
    >>> junk = "some random HTML<P> for you to try to parse</p>"
    >>> clean_page(junk)
    '<div><p>some random HTML</p><p> for you to try to parse</p></div>'
    >>> print clean_page(junk, pretty_print=True)
    <div>
    <p>some random HTML</p>
    <p> for you to try to parse</p>
    </div>
    """
    from lxml.html import fromstring
    from lxml.html import tostring
    return tostring(fromstring(html), pretty_print=pretty_print)

Converting yes/no to Boolean values

Computers are far better at interpreting Boolean values when they are consistently provided. Irrespective of the programming language, normalising these values will make any automatic comparisons much richer:

def to_bool(yes_no, none_to_false=True):
    """
    >>> to_bool('')
    False
    >>> to_bool(None):
    False
    >>> to_bool('y')
    True
    >>> to_bool('yip')
    True
    >>> to_bool('Yes')
    True
    >>> to_bool('nuh')
    False
    """
    yes_no = yes_no.strip().lower()
    if not yes_no.strip() and none_to_false:
        return False
    if yes_no.startswith('y'):
        return True
    elif yes_no.startswith('n'):
        return False

Converting numbers to the correct type

If you’re extracting numbers from HTML tables, they will each be represented as a string or Unicode, even though it would be more sensible to treat as integers or floating point numbers:

def to_int(number, european=False):
    """
    >>> to_int('32')
    32
    >>> to_int('3,998')
    3998
    >>> to_int('3.998', european=True)
    3998
    """
    if european:
        number = number.replace('.', '')
    else:
        number = number.replace(',', '')
    return int(number)

def to_float(number, european=False)
    """
    >>> to_float(u'42.1')
    42.1
    >>> to_float(u'32,1', european=True)
    32.1
    >>> to_float('3,132.87')
    3132.87
    >>> to_float('3.132,87')
    3132.87
    >>> to_float('(54.12)')
    -54.12

    Warning
    -------

    Incorrectly declaring `european` leads to troublesome results:

    >>> to_float('54.2', european=True)
    542
    """
    import string
    if european:
        table = string.maketrans(',.','.,')
        number = string.translate(number, table)
    number = number.replace(',', '')
    if number.startswith('(') and number.endswith(')'):
        number = '-' + number[1:-1]
return float(number)

If you are dealing with numbers from another region consistently, it may be appropriate to call upon the locale module. You will then have the advantage of code written in C, rather than Python:

>>> import locale
>>> locale.setlocale(locale.LC_ALL, '')
>>> locale.atoi('1,000,000')
1000000

Stripping whitespace

Removing whitespace from a string is built into many languages string. Removing left and right whitespace is highly recommended. Your database will be unable to sort data properly which have inconsistent treatment of whitespace:

>>> u'\n\tTitle'.strip()
u'Title'

Converting dates to a machine-readable format

Python is well blessed with a mature date parser, dateutil. We can take advantage of this to make light work an otherwise error-prone task.

dateutil can be reluctant to raise exceptions to dates that it doesn’t understand. Therefore, it can be wise to store the original along with the parsed ISO formatted string. This can be used for manual checking if required later.

Example code:

def date_to_iso(datestring):
    """
    Takes a string of a human-readable date and
    returns a machine-readable date string.


    >>> date_to_iso('20 July 2002')
    '2002-07-20 00:00:00'
    >>> date_to_iso('June 3 2009 at 4am')
    '2009-06-03 04:00:00'
    """
    from dateutil import parser
    from datetime import datetime
    default = datetime(year=1, month=1, day=1)
    return str(parser.parse(datestring, default=default))