otfbot.lib.urlutils

module documentation
(source)

Some functions to simplify handling of HTTP-requests

Function convert_bytes copied from http://www.5dollarwhitebox.org/drupal/node/84
Function download Uses twisted.web.client.getPage() to fetch a Page via HTTP
Function download_if_html download content, if the mimetype is text/html
Function get_headers Undocumented
Function get_page_with_header Download a web page as a string.
Function is_html Undocumented
def convert_bytes(bytes): (source)

copied from http://www.5dollarwhitebox.org/drupal/node/84

def download(url, file=None, **kwargs): (source)

Uses twisted.web.client.getPage() to fetch a Page via HTTP

Returns
A Defered which will call a Callback with the content as argument
def download_if_html(url): (source)

download content, if the mimetype is text/html

def get_headers(url): (source)

Undocumented

def get_page_with_header(url, contextFactory=None, *args, **kwargs): (source)

Download a web page as a string.

Download a page. Return a deferred, which will callback with a page (as a string) or errback with a description of the error.

See HTTPClientFactory to see what extra args can be passed.

def is_html(headers): (source)

Undocumented