Fixtures

A fixture is defined as “a piece of equipment or furniture which is fixed in position in a building or vehicle”. In our case a fixture is something attached to the action that processes an HTTP request in order to produce a response.

When processing any HTTP requests there are some optional operations we may want to perform. For example parse the cookie to look for session information, commit a database transaction, determine the preferred language from the HTTP header and lookup proper internationalization, etc. These operations are optional. Some actions need them and some actions do not. They may also depend on each other. For example, if sessions are stored in the database and our action needs it, we may need to parse the session cookie from the HTTP header, pick up a connection from the database connection pool, and - after the action has been executed - save the session back in the database if data has changed.

PY4WEB fixtures provide a mechanism to specify what an action needs so that py4web can accomplish the required tasks (and skip non required ones) in the most efficient manner. Fixtures make the code efficient and reduce the need for boilerplate code.

PY4WEB fixtures are similar to WSGI middleware and BottlePy plugin except that they apply to individual actions, not to all of them, and can depend on each other.

PY4WEB comes with some pre-defined fixtures: sessions, url signing and flash messages will be fully explained in this chapter. Database connections, internationalization, authentication, and templates will instead be just outlined here since they have dedicated chapters. The developer is also free to add fixtures, for example, to handle a third party template language or third party session logic; this is explained later in the Custom fixtures paragraph.

Using Fixtures

As we’ve seen in the previous chapter, fixtures are the arguments of the decorator @action.uses(...). You can specify multiple fixtures in one decorator or you can have multiple decorators.

Also, fixtures can be applied in groups. For example:

preferred = action.uses(session, auth, T, flash)

Then you can apply all of them at once with:

@action('index.html')
@preferred
def index():
    return dict()

Usually, it’s not important the order you use to specify the fixtures, because py4web knows well how to manage them. But there is an important exception: the Template fixture must always be the last one.

The Template fixture

PY4WEB by default uses the YATL template language and provides a fixture for it.

from py4web import action
from py4web.core import Template

@action('index')
@action.uses(Template('index.html', delimiters='[[ ]]'))
def index():
    return dict(message="Hello world")

Note: this example assumes that you created the application from the scaffolding app, so that the template index.html is already created for you.

The Template object is a Fixture. It transforms the dict() returned by the action into a string by using the index.html template file. In a later chapter we will provide an example of how to define a custom fixture to use a different template language, for example Jinja2.

Notice that since the use of templates is very common and since, most likely, every action uses a different template, we provide some syntactic sugar, and the two following lines are equivalent:

@action.uses('index.html')
@action.uses(Template('index.html', delimiters='[[ ]]'))

Notice that py4web template files are cached in RAM. The py4web caching object is described later on Caching and Memoize.

Warning

If you use multiple fixtures, always place the template as the last one. Otherwise, it will not have access to various things it needs from the other fixtures.

For example:

@action.uses(session, db, 'index.html') # right
@action.uses('index.html', session, db) # wrong

The Translator fixture

Here is an example of usage:

from py4web import action, Translator
import os

T_FOLDER = os.path.join(os.path.dirname(__file__), 'translations')
T = Translator(T_FOLDER)

@action('index')
@action.uses(T)
def index(): return str(T('Hello world'))

The string hello world will be translated based on the internationalization file in the specified “translations” folder that best matches the HTTP accept-language header.

Here Translator is a py4web class that extends pluralize.Translator and also implements the Fixture interface.

We can easily combine multiple fixtures. Here, as example, we make action with a counter that counts “visits”.

from py4web import action, Session, Translator, DAL
from py4web.utils.dbstore import DBStore
import os
db = DAL('sqlite:memory')
session =  Session(storage=DBStore(db))
T_FOLDER = os.path.join(os.path.dirname(__file__), 'translations')
T = Translator(T_FOLDER)

@action('index')
@action.uses(session, T)
def index():
    counter = session.get('counter', -1)
    counter += 1
    session['counter'] = counter
    return str(T("You have been here {n} times").format(n=counter))

Now create the following translation file translations/en.json:

{"You have been here {n} times":
  {
    "0": "This your first time here",
    "1": "You have been here once before",
    "2": "You have been here twice before",
    "3": "You have been here {n} times",
    "6": "You have been here more than 5 times"
  }
}

When visiting this site with the browser language preference set to English and reloading multiple times you will get the following messages:

This your first time here
You have been here once before
You have been here twice before
You have been here 3 times
You have been here 4 times
You have been here 5 times
You have been here more than 5 times

Now try create a file called translations/it.json which contains:

{"You have been here {n} times":
  {
    "0": "Non ti ho mai visto prima",
    "1": "Ti ho gia' visto",
    "2": "Ti ho gia' visto 2 volte",
    "3": "Ti ho visto {n} volte",
    "6": "Ti ho visto piu' di 5 volte"
  }
}

Set your browser preference to Italian: now the messages will be automatically translated to Italian.

The Flash fixture

It is common to want to display “alerts” to the users. Here we refer to them as flash messages. There is a little more to it than just displaying a message to the view, because flash messages:

  • can have state that must be preserved after redirection

  • can be generated both server side and client side

  • may have a type

  • should be dismissible

The Flash helper handles the server side of them. Here is an example:

from py4web import Flash

flash = Flash()

@action('index')
@action.uses(flash)
def index():
    flash.set("Hello World", _class="info", sanitize=True)
    return dict()

and in the template:

...
<div id="py4web-flash"></div>
...
<script src="js/utils.js"></script>
[[if globals().get('flash'):]]
<script>utils.flash([[=XML(flash)]]);</script>
[[pass]]

By setting the value of the message in the flash helper, a flash variable is returned by the action and this triggers the JS in the template to inject the message in the py4web-flash DIV which you can position at your convenience. Also the optional class is applied to the injected HTML.

If a page is redirected after a flash is set, the flash is remembered. This is achieved by asking the browser to keep the message temporarily in a one-time cookie. After redirection the message is sent back by the browser to the server and the server sets it again automatically before returning the content, unless it is overwritten by another set.

The client can also set/add flash messages by calling:

utils.flash({'message': 'hello world', 'class': 'info'});

py4web defaults to an alert class called info and most CSS frameworks define classes for alerts called success, error, warning, default, and info. Yet, there is nothing in py4web that hardcodes those names. You can use your own class names.

You can see the basic usage of flash messages in the examples app.

The Session fixture

Simply speaking, a session can be defined as a way to preserve information that is desired to persist throughout the user’s interaction with the web site or web application. In other words, sessions render the stateless HTTP connection a stateful one.

In py4web, the session object is also a fixture. Here is a simple example of its usage to implement a counter.

from py4web import Session, action
session = Session(secret='my secret key')

@action('index')
@action.uses(session)
def index():
    counter = session.get('counter', -1)
    counter += 1
    session['counter'] = counter
    return "counter = %i" % counter

The counter will start from 0; its value will be remembered and increased every time you reload the page.

_images/simple_counter.png

Opening the page in a new browser tab will give you the updated counter value. Closing and reopening the browser, or opening a new private window, will instead restart the counter from 0.

Usually the information is saved in the session object are related to the user - like its username, preferences, last pages visited, shopping cart and so on. The session object has the same interface as a Python dictionary but in py4web sessions are always stored using JSON (JWT specifically, i.e. JSON Web Token), therefore you should only store objects that are JSON serializable. If the object is not JSON serializable, it will be serialized using the __str__ operator and some information may be lost.

The information composing the session object can be saved:

  • client-side, by only using cookies (default)

  • server-side, but you’ll still need minimal cookies for identifying the clients

By default py4web sessions never expire (unless they contain login information, but that is another story) even if an expiration can be set. Other parameters can be specified as well:

session = Session(secret='my secret key',
                  expiration=3600,
                  algorithm='HS256',
                  storage=None,
                  same_site='Lax')

Here:

  • secret is the passphrase used to sign the information

  • expiration is the maximum lifetime of the session, in seconds (default = None, i.e. no timeout)

  • algorithm is the algorithm to be used for the JWT token signature (‘HS256’ by default)

  • storage is a parameter that allows to specify an alternate session storage method (for example Redis, or database). If not specified, the default cookie method will be used

  • same_site is an option that prevents CSRF attacks (Cross-Site Request Forgery) and is enabled by default with the ‘Lax’ option. You can read more about it here

If storage is not provided, session is stored in client-side jwt cookie. Otherwise, we have server-side session: the jwt is stored in storage and only its UUID key is stored in the cookie. This is the reason why the secret is not required with server-side sessions.

Client-side session in cookies

By default the session object is stored inside a cookie called appname_session. It’s a JWT, hence encoded in a URL-friendly string format and signed using the provided secret for preventing tampering. Notice that it’s not encrypted (in fact it’s quite trivial to read its content from http communications or from disk), so do not place any sensitive information inside, and use a complex secret. If the secret changes existing sessions are invalidated. If the user switches from HTTP to HTTPS or vice versa, the user session is also invalidated. Session in cookies have a small size limit (4 kbytes after being serialized and encoded) so do not put too much into them.

Server-side session in memcache

Requires memcache installed and configured.

import memcache, time
conn = memcache.Client(['127.0.0.1:11211'], debug=0)
session = Session(storage=conn)

Server-side session in Redis

Requires Redis installed and configured.

import redis
conn = redis.Redis(host='localhost', port=6379)
conn.set = lambda k, v, e, cs=conn.set, ct=conn.ttl: (cs(k, v), e and ct(e))
session = Session(storage=conn)

Notice: a storage object must have get and set methods and the set method must allow to specify an expiration. The redis connection object has a ttl method to specify the expiration, hence we monkey patch the set method to have the expected signature and functionality.

Server-side session in database

from py4web import Session, DAL
from py4web.utils.dbstore import DBStore
db = DAL('sqlite:memory')
session =  Session(storage=DBStore(db))

Warning

the 'sqlite:memory' database used in this example cannot be used in multiprocess environment; the quirk is that your application will still work but in non-deterministic and unsafe mode, since each process/worker will have its own independent in-memory database.

This is one case when a fixture (session) requires another fixture (db). This is handled automatically by py4web and the following lines are equivalent:

@action.uses(session)
@action.uses(db, session)

Server-side session anywhere

You can easily store sessions in any place you want. All you need to do is provide to the Session object a storage object with both get and set methods. For example, imagine you want to store sessions on your local filesystem:

import os
import json

class FSStorage:
   def __init__(self, folder):
       self.folder = folder
   def get(self, key):
       filename = os.path.join(self.folder, key)
       if os.path.exists(filename):
           with open(filename) as fp:
              return json.load(fp)
       return None
   def set(self, key, value, expiration=None):
       filename = os.path.join(self.folder, key)
       with open(filename, 'w') as fp:
           json.dump(value, fp)

session = Session(storage=FSStorage('/tmp/sessions'))

We leave to you as an exercise to implement expiration, limit the number of files per folder by using subfolders, and implement file locking. Yet we do not recommend storing sessions on the filesystem: it is inefficient and does not scale well.

The URLsigner fixture

A signed URL is a URL that provides limited permission and time to make an HTTP request by containing authentication information in its query string. The typical usage is as follows:

from py4web.utils import URLSigner

# We build a URL signer.
url_signer = URLSigner(session)

@action('/somepath')
@action.uses(url_signer)
def somepath():
   # This controller signs a URL.
   return dict(signed_url = URL('/anotherpath', signer=url_signer))

@action('/anotherpath')
@action.uses(url_signer.verify())
def anotherpath():
   # The signature has been verified.
   return dict()

The DAL fixture

We have already used the DAL fixture in the context of sessions but maybe you want direct access to the DAL object for the purpose of accessing the database, not just sessions.

PY4WEB, by default, uses the PyDAL (Python Database Abstraction Layer) which is documented in the next chapter. Here is an example, please remember to create the databases folder under your project in case it doesn’t exist:

from datetime import datetime
from py4web import action, request, DAL, Field
import os

DB_FOLDER = os.path.join(os.path.dirname(__file__), 'databases')
db = DAL('sqlite://storage.db', folder=DB_FOLDER, pool_size=1)
db.define_table('visit_log', Field('client_ip'), Field('timestamp', 'datetime'))
db.commit()

@action('index')
@action.uses(db)
def index():
    client_ip = request.environ.get('REMOTE_ADDR')
    db.visit_log.insert(client_ip=client_ip, timestamp=datetime.utcnow())
    return "Your visit was stored in database"

Notice that the database fixture defines (creates/re-creates) tables automatically when py4web starts (and every time it reloads this app) and picks a connection from the connection pool at every HTTP request. Also each call to the index() action is wrapped into a transaction and it commits on_success and rolls back on_error.

The Auth fixture

auth and auth.user are both fixtures that depend on session and db. Their role is to provide the action with authentication information.

Auth is used as follows:

from py4web import action, redirect, Session, DAL, URL
from py4web.utils.auth import Auth
import os

session = Session(secret='my secret key')
DB_FOLDER = os.path.join(os.path.dirname(__file__), 'databases')
db = DAL('sqlite://storage.db', folder=DB_FOLDER, pool_size=1)
auth = Auth(session, db)
auth.enable()

@action('index')
@action.uses(auth)
def index():
    user = auth.get_user() or redirect(URL('auth/login'))
    return 'Welcome %s' % user.get('first_name')

The constructor of the Auth object defines the auth_user table with the following fields: username, email, password, first_name, last_name, sso_id, and action_token (the last two are mostly for internal use).

The auth object exposes the method:auth.enable() which registers multiple actions including {appname}/auth/login. It requires the presence of the auth.html template and the auth value component provided by the _scaffold app. It also exposes the method:

auth.get_user()

which returns a python dictionary containing the information of the currently logged in user. If the user is not logged-in, it returns None and in this case the code of the example redirects to the auth/login page.

Since this check is very common, py4web provides an additional fixture auth.user:

@action('index')
@action.uses(auth.user)
def index():
    user = auth.get_user()
    return 'Welcome %s' % user.get('first_name')

This fixture automatically redirects to the auth/login page if user is not logged-in, hence this example is equivalent to the previous one.

The auth fixture is plugin based: it supports multiple plugin methods including OAuth2 (Google, Facebook, Twitter), PAM and LDAP. The Authentication and Access control chapter will show you all the related details.

Caveats about fixtures

Since fixtures are shared by multiple actions you are not allowed to change their state because it would not be thread safe. There is one exception to this rule. Actions can change some attributes of database fields:

from py4web import action, request, DAL, Field
from py4web.utils.form import Form
import os

DB_FOLDER = os.path.join(os.path.dirname(__file__), 'databases')
db = DAL('sqlite://storage.db', folder=DB_FOLDER, pool_size=1)
db.define_table('thing', Field('name', writable=False))

@action('index')
@action.uses(db, 'generic.html')
def index():
    db.thing.name.writable = True
    form = Form(db.thing)
    return dict(form=form)

Note that this code will only be able to display a form, to process it after submit, additional code needs to be added, as we will see later on. This example is assuming that you created the application from the scaffolding app, so that a generic.html is already created for you.

The readable, writable, default, update, and require attributes of db.{table}.{field} are special objects of class ThreadSafeVariable defined the threadsafevariable module. These objects are very much like Python thread local objects but they are re-initialized at every request using the value specified outside of the action. This means that actions can safely change the values of these attributes.

Custom fixtures

A fixture is an object with the following minimal structure:

from py4web import Fixture

class MyFixture(Fixture):
    def on_request(self): pass
    def on_success(self): pass
    def on_error(self): pass
    def transform(self, data): return data

If an action uses this fixture:

@action('index')
@action.uses(MyFixture())
def index(): return 'hello world'

then:

  • the on_request() function is guaranteed to be called before the index() function is called

  • the on_success() function is guaranteed to be called if the index() function returns successfully or raises HTTP or performs a redirect

  • the on_error() function is guaranteed to be called when the index() function raises any exception other than HTTP.

  • the transform function is called to perform any desired transformation of the value returned by the index() function.

Caching and Memoize

py4web provides a cache in RAM object that implements the last recently used (LRU) algorithm. It can be used to cache any function via a decorator:

import uuid
from py4web import Cache, action
cache = Cache(size=1000)

@action('hello/<name>')
@cache.memoize(expiration=60)
def hello(name):
    return "Hello %s your code is %s" % (name, uuid.uuid4())

It will cache (memoize) the return value of the hello function, as function of the input name, for up to 60 seconds. It will store in cache the 1000 most recently used values. The data is always stored in RAM.

The cache object is not a fixture and it should not and cannot be registered using the @action.uses decorator but we mention it here because some of the fixtures use this object internally. For example, template files are cached in RAM to avoid accessing the file system every time a template needs to be rendered.

Convenience Decorators

The _scaffold application, in common.py defines two special convenience decorators:

@unauthenticated
def index():
    return dict()

and

@authenticated
def index():
    return dict()

They apply all of the decorators below (db, session, T, flash, auth), use a template with the same name as the function (.html), and also register a route with the name of action followed by the number of arguments of the action separated by a slash (/).

  • @unauthenticated does not require the user to be logged in.

  • @authenticated required the user to be logged in.

They can be combined with (and precede) other @action.uses(...) but they should not be combined with @action(...) because they perform that function automatically.