Reference

class urlfetch.Response(r, **kwargs)

A Response object.

>>> import urlfetch
>>> response = urlfetch.get("http://docs.python.org/")
>>> response.total_time
0.033042049407959
>>> response.status, response.reason, response.version
(200, 'OK', 10)
>>> type(response.body), len(response.body)
(<type 'str'>, 8719)
>>> type(response.text), len(response.text)
(<type 'unicode'>, 8719)
>>> response.getheader('server')
'Apache/2.2.16 (Debian)'
>>> response.getheaders()
[
    ('content-length', '8719'),
    ('x-cache', 'MISS from localhost'),
    ('accept-ranges', 'bytes'),
    ('vary', 'Accept-Encoding'),
    ('server', 'Apache/2.2.16 (Debian)'),
    ('last-modified', 'Tue, 26 Jun 2012 19:23:18 GMT'),
    ('connection', 'close'),
    ('etag', '"13cc5e4-220f-4c36507ded580"'),
    ('date', 'Wed, 27 Jun 2012 06:50:30 GMT'),
    ('content-type', 'text/html'),
    ('x-cache-lookup', 'MISS from localhost:8080')
]
>>> response.headers
{
    'content-length': '8719',
    'x-cache': 'MISS from localhost',
    'accept-ranges': 'bytes',
    'vary': 'Accept-Encoding',
    'server': 'Apache/2.2.16 (Debian)',
    'last-modified': 'Tue, 26 Jun 2012 19:23:18 GMT',
    'connection': 'close',
    'etag': '"13cc5e4-220f-4c36507ded580"',
    'date': 'Wed, 27 Jun 2012 06:50:30 GMT',
    'content-type': 'text/html',
    'x-cache-lookup': 'MISS from localhost:8080'
}

Raises:ContentLimitExceeded

body

Response body.

Raises:ContentLimitExceeded, ContentDecodingError

close()

Close the connection.

content

cookies

Cookies in dict

cookiestring

Cookie string

classmethod from_httplib(connection, **kwargs)

Make an Response object from a httplib response object.

headers

Response headers.

Response headers is a dict with all keys in lower case.

>>> import urlfetch
>>> response = urlfetch.get("http://docs.python.org/")
>>> response.headers
{
    'content-length': '8719',
    'x-cache': 'MISS from localhost',
    'accept-ranges': 'bytes',
    'vary': 'Accept-Encoding',
    'server': 'Apache/2.2.16 (Debian)',
    'last-modified': 'Tue, 26 Jun 2012 19:23:18 GMT',
    'connection': 'close',
    'etag': '"13cc5e4-220f-4c36507ded580"',
    'date': 'Wed, 27 Jun 2012 06:50:30 GMT',
    'content-type': 'text/html',
    'x-cache-lookup': 'MISS from localhost:8080'
}

json

Load response body as json.

Raises:ContentDecodingError

links

Links parsed from HTTP Link header

next()

read(chunk_size=65536)

Read content (for streaming and large files)

Parameters:chunk_size (int) – size of chunk, default is 65536, i.e. 64KiB.

reason = None

Reason phrase returned by server.

status = None

Status code returned by server.

status_code = None

An alias of status.

text

Response body in str.

total_time = None

total time

version = None

HTTP protocol version used by server. 10 for HTTP/1.0, 11 for HTTP/1.1.

class urlfetch.Session(headers={}, cookies={}, auth=None)

A session object.

urlfetch.Session can hold common headers and cookies. Every request issued by a urlfetch.Session object will bring u these headers and cookies.

urlfetch.Session plays a role in handling cookies, just like a cookiejar.

Parameters:

• headers (dict) – Init headers.
• cookies (dict) – Init cookies.
• auth (tuple) – (username, password) for basic authentication.

cookies = None

cookies

cookiestring

Cookie string.

It’s assignalbe, and will change cookies correspondingly.

>>> s = Session()
>>> s.cookiestring = 'foo=bar; 1=2'
>>> s.cookies
{'1': '2', 'foo': 'bar'}

delete(*args, **kwargs)

Issue a delete request.

fetch(*args, **kwargs)

Fetch an URL

get(*args, **kwargs)

Issue a get request.

head(*args, **kwargs)

Issue a head request.

headers = None

headers

options(*args, **kwargs)

Issue a options request.

patch(*args, **kwargs)

Issue a patch request.

popcookie(key)

Remove an cookie from default cookies.

popheader(header)

Remove an header from default headers.

post(*args, **kwargs)

Issue a post request.

put(*args, **kwargs)

Issue a put request.

putcookie(key, value='')

Add an cookie to default cookies.

putheader(header, value)

Add an header to default headers.

request(*args, **kwargs)

Issue a request.

snapshot()

trace(*args, **kwargs)

Issue a trace request.

urlfetch.request(url, method='GET', params=None, data=None, headers={}, timeout=None, files={}, randua=False, auth=None, length_limit=None, proxies=None, trust_env=True, max_redirects=0, source_address=None, validate_certificate=None, **kwargs)

request an URL

Parameters:

• url (string) – URL to be fetched.
• method (string) – (optional) HTTP method, one of GET, DELETE, HEAD, OPTIONS, PUT, POST, TRACE, PATCH. GET is the default.
• params (dict/string) – (optional) Dict or string to attach to url as querystring.
• headers (dict) – (optional) HTTP request headers.
• timeout (float) – (optional) Timeout in seconds
• files – (optional) Files to be sended
• randua – (optional) If True or path string, use a random user-agent in headers, instead of 'urlfetch/' + __version__
• auth (tuple) – (optional) (username, password) for basic authentication
• length_limit (int) – (optional) If None, no limits on content length, if the limit reached raised exception ‘Content length is more than …’
• proxies (dict) – (optional) HTTP proxy, like {‘http’: ‘127.0.0.1:8888’, ‘https’: ‘127.0.0.1:563’}
• trust_env (bool) – (optional) If True, urlfetch will get infomations from env, such as HTTP_PROXY, HTTPS_PROXY
• max_redirects (int) – (integer, optional) Max redirects allowed within a request. Default is 0, which means redirects are not allowed.
• source_address (tuple) – (optional) A tuple of (host, port) to specify the source_address to bind to. This argument is ignored if you’re using Python prior to 2.7/3.2.
• validate_certificate (bool) – (optional) If False, urlfetch skips all the necessary certificate and hostname checks

Returns:

A Response object

Raises:

URLError, UrlfetchException, TooManyRedirects,

urlfetch.fetch(*args, **kwargs)

fetch an URL.

fetch() is a wrapper of request(). It calls get() by default. If one of parameter data or parameter files is supplied, post() is called.

urlfetch.get(url, *, method='GET', params=None, data=None, headers={}, timeout=None, files={}, randua=False, auth=None, length_limit=None, proxies=None, trust_env=True, max_redirects=0, source_address=None, validate_certificate=None, **kwargs)

Issue a get request

urlfetch.post(url, *, method='POST', params=None, data=None, headers={}, timeout=None, files={}, randua=False, auth=None, length_limit=None, proxies=None, trust_env=True, max_redirects=0, source_address=None, validate_certificate=None, **kwargs)

Issue a post request

urlfetch.head(url, *, method='HEAD', params=None, data=None, headers={}, timeout=None, files={}, randua=False, auth=None, length_limit=None, proxies=None, trust_env=True, max_redirects=0, source_address=None, validate_certificate=None, **kwargs)

Issue a head request

urlfetch.put(url, *, method='PUT', params=None, data=None, headers={}, timeout=None, files={}, randua=False, auth=None, length_limit=None, proxies=None, trust_env=True, max_redirects=0, source_address=None, validate_certificate=None, **kwargs)

Issue a put request

urlfetch.delete(url, *, method='DELETE', params=None, data=None, headers={}, timeout=None, files={}, randua=False, auth=None, length_limit=None, proxies=None, trust_env=True, max_redirects=0, source_address=None, validate_certificate=None, **kwargs)

Issue a delete request

urlfetch.options(url, *, method='OPTIONS', params=None, data=None, headers={}, timeout=None, files={}, randua=False, auth=None, length_limit=None, proxies=None, trust_env=True, max_redirects=0, source_address=None, validate_certificate=None, **kwargs)

Issue a options request

urlfetch.trace(url, *, method='TRACE', params=None, data=None, headers={}, timeout=None, files={}, randua=False, auth=None, length_limit=None, proxies=None, trust_env=True, max_redirects=0, source_address=None, validate_certificate=None, **kwargs)

Issue a trace request

urlfetch.patch(url, *, method='PATCH', params=None, data=None, headers={}, timeout=None, files={}, randua=False, auth=None, length_limit=None, proxies=None, trust_env=True, max_redirects=0, source_address=None, validate_certificate=None, **kwargs)

Issue a patch request

Exceptions

class urlfetch.UrlfetchException

Base exception. All exceptions and errors will subclass from this.

class urlfetch.ContentLimitExceeded

Content length is beyond the limit.

class urlfetch.URLError

Error parsing or handling the URL.

class urlfetch.ContentDecodingError

Failed to decode the content.

class urlfetch.TooManyRedirects

Too many redirects.

class urlfetch.Timeout

Request timed out.

helpers

urlfetch.parse_url(url)

Return a dictionary of parsed url

Including scheme, netloc, path, params, query, fragment, uri, username, password, host, port and http_host

urlfetch.get_proxies_from_environ()

Get proxies from os.environ.

urlfetch.mb_code(s, coding=None, errors='replace')

encoding/decoding helper.

urlfetch.random_useragent(filename=True)

Returns a User-Agent string randomly from file.

Parameters:filename (string) – (Optional) Path to the file from which a random useragent is generated. By default it’s True, a file shipped with this module will be used. Returns:An user-agent string.

urlfetch.url_concat(url, args, keep_existing=True)

Concatenate url and argument dictionary

>>> url_concat("http://example.com/foo?a=b", dict(c="d"))
'http://example.com/foo?a=b&c=d'

Parameters:

• url (string) – URL being concat to.
• args (dict) – Args being concat.
• keep_existing (bool) – (Optional) Whether to keep the args which are alreay in url, default is True.

urlfetch.choose_boundary()

Generate a multipart boundry.

Returns:A boundary string

urlfetch.encode_multipart(data, files)

Encode multipart.

Parameters:

• data (dict) – Data to be encoded
• files (dict) – Files to be encoded

Returns:

Encoded binary string

Raises:

UrlfetchException