Clay is a framework for building RESTful backend services using best practices. It’s a wrapper around Flask (http://flask.pocoo.org) along with several convenient, standardized, performance enhanced ways of performing common tasks like sending email and connecting to a database. Clay is available under the MIT License.

Getting Started

Writing a new service

If you’re developing a new service, you’ve come to the right place. Let’s start with the obligatory Hello World. You’ll notice that this is very similar to Flask’s Hello World example.

# This is a minimalist example of a clay view. Run it like so:
#
#   CLAY_CONFIG=simple-clay.conf clay-devserver
#
from __future__ import absolute_import
from clay import app


@app.route('/', methods=['GET'])
def hello():
    return 'Hello World!'

A few noteworthy things here that begin our list of best practices…

  • Every module starts with from __future__ import absolute_import, this is to prevent large projects from ending up with cyclical imports.

  • Along those same lines, wildcard imports like from foo import * should never be used. Always explicitly list the things you’re importing from a module.

  • from imports should be avoided when practical. In the middle of a thousand-line file, it might not be obvious that mail.send(...) is really clay.mail.send(...) and not an instance of a local object.

  • Keep reasonably close to PEP8 recommendations. Four space tabs, two newlines between top-level definitions in a module, etc. If you like keeping things to 80 columns, that’s fine, but if a line needs to be longer, nobody’s going to complain.

  • Explicitly list the methods that a route responds to. We know that app.route() defaults to GET methods only, but this is not guaranteed to be the same in future versions of Flask/Clay.

Every Clay application needs a config. Clay configs are JSON files. The clay framework itself looks for a few config variables (TODO: see below). By default, we look for files listed in $CLAY_CONFIG delimited by colons : relative to the directory the development server runs from. Here’s a simple example.

{
        "debug": {
                "enabled": true,
                "server": {
                        "host": "127.0.0.1",
                        "port": 8000
                }
        },
        "views": ["helloworld"]
}

This is the minimum configuration necessary to run the clay devserver. The first flask init section is the module name passed to the Flask app constructor. This is only used for uniquely identifying this application internally and has no bearing on anything important. It should follow the naming rules for Python modules. Here we use helloworldapp just to differentiate it from the name of our view module.

The debug server section provides a host and port for the clay devserver to listen on. No magic here, just an IP and port. Use "0.0.0.0" for the host to listen on all interfaces.

views is a list of modules to be loaded when the server starts up. This allows you to run multiple view modules simultaneously without having to manage a file with a list of routes or prefixes.

Now that we’ve got a view module and a config, we can run the clay devserver…

Running the development server

export CLAY_CONFIG=./simple-clay.conf
clay-devserver

CLAY_CONFIG is a colon delimited list of config files to be loaded, in order of precedence. For example, if CLAY_CONFIG=./common.conf:./janky.conf, the configuration dictionary from common.conf is loaded first, then update()'d with the dictionary from janky.conf.

Running a clay service in production

Clay exposes itself as a standard WSGI application in the module clay.wsgi. The CLAY_CONFIG environment variable needs to be set. All view modules listed in the config will be imported at startup.

Running a production clay service with gunicorn
export CLAY_CONFIG=./simple-clay.conf
gunicorn clay.wsgi:application

API Reference

clay.config

The clay.config module provides a simple API for accessing configuration information. Internally, all Clay modules use this module to configure themselves.

Upon import, clay.config attempts to load it’s configuration from files listed in the CLAY_CONFIG environment variable, delimited by colons. Each file’s name is examined for a known file extension and parsed by the appropriate deserializer. Currently json is supported using the standard library and yaml is supported if PyYAML is installed. As each file is parsed, it is applied to the global config with dict.update().

This module registers itself as a handler for SIGHUP and will attempt to reload it’s configuration upon receiving that signal. The configuration may also be reloaded on demand by calling config.load().

Several methods are exposed at the top level of the clay.config module and are intended to provide the config’s public API.

clay.config.get(key, default=None)

key is a period . delimited string that is recursively searched for a configuration option with that name. If this lookup fails at any level of the config hierarchy, the value of default is returned. For example, you can expect the following behavior for the given config.

Example configuration
{
        "users": {
                "admins": ["synack", "bigo"],
        }
}
>>> from clay import config
>>> config.get('users.admins')
[u"synack", u"bigo"]
>>> config.get('users.players')
None
>>> config.get('dogs')
None
>>> config.get('dogs', default=True)
True

clay.config.get_logger(name)

Returns a pre-configured logging.Logger instance identified by the given name. Depending on the framework’s environment, this logger may format and emit messages in different ways. In development, messages will be routed to the console whereas in production they might be routed to an aggregate endpoint or archive.

>>> from clay.config import config
>>> log = config.get_logger('myservice')
>>> log.debug('Now we know what\'s happening!')
myservice DEBUG Now we know what's happening!

clay.config.debug()

DEPRECATED, will be removed in a future release. Use clay.config.get('debug.enabled', False) instead.

clay.config.feature_flag(name)

Similar to clay.config.get, feature_flag is specific to things with boolean values that enable or disable functionality within your service. This method returns True if the given feature is enabled, False otherwise.

Example configuration
{
        "features": {
                "new_shiny_bits": {
                        "enabled": true
                },
                "new_scary_thing": {
                        "enabled": true,
                        "percent": 10.0
                }
        }
}

In the example above, feature_flag('new_shiny_bits') would return True and feature_flag('new_scary_thing') will only return True 10% of the time. The percent option is useful for A/B testing new features or slowly rolling out a feature for a subset of requests to gauge performance.

clay.config.load()

This method will cause the configuration to be reloaded from it’s source on demand. WARNING if a syntax error or otherwise unreadable configuration is loaded, the process calling this method will be aborted immediately via sys.exit(), this is often not desirable.

clay.mail

clay.mail.sendmail(mailto, subject, message, subtype=html, charset=utf-8, **headers)

Sends an email to the given address using the server/credentials specified in the config under smtp.

Additional SMTP headers may be set as keyword arguments, the values of which are expected to be either a subclass of basestring or an iterable of basestrings.

By default the From address is populated by the smtp.from config option. This may be overridden by passing a From kwarg.

Mail configuration
{
        "smtp": {
                "host": "smtp.example.com",
                "port": 25,
                "username": "myname",
                "password": "superseekrit"
                "from": "test@example.com"
        }
}
Sending email
from clay import mail

# Simple example
mail.sendmail('user@example.com', 'Not spam I promise!', 'A simple example')

# Complex example
mail.sendmail('user@example.com', 'Definitely not spam', CC=[
  'otheruser@example.com',
  'somedude@otherexample.com'
], BCC='outbound@example.com', From='noreply@example.com', subtype='html',
  message='<marquee><blink>YOU OBVIOUSLY LOVE OWLS</blink></marquee>')

clay.http

clay.http.request(method, uri, headers, data)

Performs an HTTP request and returns a Response object (which is just a namedtuple) with status, headers, and data attributes. This module is just a wrapper around urllib2 so any installed openers or redirect handlers will be used. See the urllib2 docs for more information (http://docs.python.org/2/library/urllib2.html).

from clay import http

response = http.request('GET', 'http://httpbin.org/ip')
if response.status != 200:
  print 'Something bad happened: ', response.data

Useful Patterns

Testing with Clay: Letting tests change clay’s configuration

Sometimes when unittesting, it may make sense to test various configuration settings. Using a base unittest like so:

Base Unittest
class CerebroTestCase(unittest.TestCase):

    def __call__(self):

        #Do pre-test stuff
        self._pre_setup()

        unittest.TestCase.__call__(self, result)

        #Do post-test stuff
        self._post_teardown()

    def _pre_setup(self):
        pass

    def _post_teardown(self):
        pass

Since clay’s configuration is stored as a dictionary, the trick is to use mock’s patch dictionary to replace the read-only dictionary with a malleable one:

Patched Unittest
class CerebroTestCase(unittest.TestCase):

    def __call__(self):
        #Copy configuration dictionary
        self.config_copy = clay.config.CONFIG.config.copy()

        #Do pre-test stuff
        self._pre_setup()

        #Patch config dictionary so tests can customize
        with mock.patch.dict(clay.config.CONFIG.config, self.config_copy, clear=True):
            unittest.TestCase.__call__(self, result)

        #Do post-test stuff
        self._post_teardown()

    def _pre_setup(self):
        pass

    def _post_teardown(self):
        pass

Finally, you can specify an attribute on a test with a dictionary of clay.config keys and values to be applied to each test (in the example below, it’s test_config), as well as a utility function so that the configuration can be manipulated for a single test:

Customizable Unittest
class CerebroTestCase(unittest.TestCase):

    def __call__(self):

        #Do pre-test stuff

        self.config_copy = clay.config.CONFIG.config.copy()

        self._pre_setup()

        #Patch config dictionary so tests can customize
        with mock.patch.dict(clay.config.CONFIG.config, self.config_copy, clear=True):
            unittest.TestCase.__call__(self, result)

        #Do post-test stuff
        self._post_teardown()

    def _pre_setup(self):
        if hasattr(self, 'test_config'):
            #Support ability for tests to have their own test settings
            for attr, value in self.test_config.items():
                self.set_config(attr, value)


    def _post_teardown(self):
        pass

    def set_config(self, key, value):
        """Helper function that tests can call to set config values."""

        val = self.config_copy
        keys = key.split('.')

        for k in keys[:-1]:
            val = val[k]

        val[keys[-1]] = value

An example test using the above patterns:

Customizable Unittest
class TestToast(CerebroTestCase):

    #These values will apply to all tests in this class
    test_config = {
        'toast.one_sided': False,
        'toast.two_sided': True
    }

    def setUp(self):
        pass

    def tearDown(self):
        pass

    def test_burnt_toast(self):
        #These values will apply for only this test
        self.set_config('toast.one_sided', True)
        self.set_config('toast.two_sided', False)

        self.toast = Toast()
        #...test continues...

Testing with Clay: Using Charlatan to manage fixtures

Charlatan is another open sourced library from Uber for managing fixtures. The following patterns provide a way to integrate the charlatan fixture model into a unittesting framework.

First add a filepath to your fixtures to your configuration file: .Fixture configuration

{
    "testing": {
        "fixture_filepath":  "cerebro/tests/fixtures.yaml"
    }
}

Charlatan only needs to load the fixtures once, so it makes sense to do it when the package is setup. Adding this to the setup_package() method in the project’s test/init.py accomplishes this when using nosetests. The following is an example from Uber’s Cerebro project:

test/__init__.py
from __future__ import absolute_import
import charlatan
import os
import clay

#Get project path
if os.environ.get("CEREBRO_HOME"):
    #Get absolute project path from environmental variables
    CEREBRO_PATH = os.path.join(os.environ["CEREBRO_HOME"], "cerebro")
else:
    #Get project path from relative path
    CEREBRO_PATH = os.path.normpath(os.path.abspath(__file__) + "../../../..")


def setup_package():
    """Set up the environment for the whole test package.

    Put here all the configuration that needs to be run only once.
    """

    #Import fixtures
    if clay.config.get('testing.fixture_filepath'):

        #Get fixture filepath
        if str(CEREBRO_PATH) not in str(clay.config.get('testing.fixture_filepath')):
            fixtures_filepath = os.path.join(CEREBRO_PATH, clay.config.get('testing.fixture_filepath'))
        else:
            fixtures_filepath = clay.config.get('testing.fixture_filepath')

        #Load fixtures
        charlatan.load(fixtures_filepath,
                models_package="cerebro.model",
                db_session=db_session)  # db_session is a sqlalchemy Session object, for saving fixtures

def teardown_package():
    pass

Once the fixtures have been loaded, add charlatan’s FixturesManagerMixin to your testcase to allow each test to access and manipulate fixtures. Generally, it saves time if each test can optionally specify a list of fixtures to be installed before each test; adding a few lines to _pre_setup takes care of the installation.

Fixture Test
class CerebroTestCase(unittest.TestCase, charlatan.FixturesManagerMixin):

    def __call__(self):

        #Do pre-test stuff

        self.config_copy = clay.config.CONFIG.config.copy()

        self._pre_setup()

        #Patch config dictionary so tests can customize
        with mock.patch.dict(clay.config.CONFIG.config, self.config_copy, clear=True):
            unittest.TestCase.__call__(self, result)

        #Do post-test stuff
        self._post_teardown()

    def _pre_setup(self):

        self.clean_fixtures_cache()

        if hasattr(self, 'test_config'):
            #Support ability for tests to have their own test settings
            for attr, value in self.test_config.items():
                self.set_config(attr, value)

        #install class fixtures
        if hasattr(self, 'fixtures'):
            self.install_fixtures(self.fixtures)


    def _post_teardown(self):
        self.clean_fixtures_cache()


    def set_config(self, key, value):
        """Helper function that tests can call to set config values."""

        val = self.config_copy
        keys = key.split('.')

        for k in keys[:-1]:
            val = val[k]

        val[keys[-1]] = value

Additionally, the full charlatan functionality is available in each unit test:

Fixture Unittest
class TestToast(CerebroTestCase):

    #These fixtures will be installed with all tests
    fixtures = ('toast', 'burnt_toast', 'bread')

    def setUp(self):
        pass

    def tearDown(self):
        pass

    def test_toastiness(self):
        is_burnt = self.toast.burnt  # installed fixtures can be accessed

        self.install_fixture('wheat_bread')  # installs wheat_bread fixture for just this test.

        toaster = self.get_fixture('toaster')  # can get fixtures manipulate them

        toaster.num_slots = 4
        toaster.save()

        #...test continues

Testing with Clay: Providing a clean test database for each test

When testing, its wise to provide each test with a database containing the correct schema, but no actual data, allowing each test to manipulate the database as necessary. This pattern involves leveraging the transaction capability of sqlalchemy’s scoped_session to accopmlish this goal.

First, add a sqlalchemy configuration for your test database to clay’s configuration file:

Test database configuration
{
    "testing": {
        "fixture_filepath":  "cerebro/tests/fixtures.yaml",
        "database": {
            "sqlalchemy.url": "postgresql://user:password@localhost:1234/test_database",
            "echo": True
        }
    }
}

Then, have the engine start when the test package is started by adding to the setup_package() function started here. The following is an example from Uber’s Cerebro project:

test/__init__.py with Test Database Engine
from __future__ import absolute_import
import charlatan
import os
import clay

from sqlalchemy import engine_from_config

#db_session is a sqlalchemy scoped_session object
#initialize_sql is a function to import Cerebro's models and initialize the sqlalchemy mappers
from cerebro.model.basics import db_session, initialize_sql


test_engine = None

#Get project path
if os.environ.get("CEREBRO_HOME"):
    #Get absolute project path from environmental variables
    CEREBRO_PATH = os.path.join(os.environ["CEREBRO_HOME"], "cerebro")
else:
    #Get project path from relative path
    CEREBRO_PATH = os.path.normpath(os.path.abspath(__file__) + "../../../..")


def setup_package():
    """Set up the environment for the whole test package.

    Put here all the configuration that needs to be run only once.
    """

    #Start test engine
    global test_engine

    test_engine = engine_from_config(clay.config.get('testing.database'))

    #Import fixtures
    if clay.config.get('testing.fixture_filepath'):

        #Get fixture filepath
        if str(CEREBRO_PATH) not in str(clay.config.get('testing.fixture_filepath')):
            fixtures_filepath = os.path.join(CEREBRO_PATH, clay.config.get('testing.fixture_filepath'))
        else:
            fixtures_filepath = clay.config.get('testing.fixture_filepath')

        #Load fixtures
        charlatan.load(fixtures_filepath,
                models_package="cerebro.model",
                db_session=db_session)  # db_session is a sqlalchemy Session object, for saving fixtures

    #Initialize SQLalchemy mappers with test engine
    initialize_sql(test_engine)

def teardown_package():
    pass

Each test will open its own connection and transaction with the database, and as part of the teardown the entire transaction will be rolled back, effectively giving each test its own copy of the database to interact with. Continuing to add to our base CerebroTestCase class, we add this logic in the _pre_setup and post_teardown methods:

Test Database Test
class CerebroTestCase(unittest.TestCase, charlatan.FixturesManagerMixin):

    def __call__(self):

        #Do pre-test stuff

        self.config_copy = clay.config.CONFIG.config.copy()

        self._pre_setup()

        #Patch config dictionary so tests can customize
        with mock.patch.dict(clay.config.CONFIG.config, self.config_copy, clear=True):
            unittest.TestCase.__call__(self, result)

        #Do post-test stuff
        self._post_teardown()

    def _pre_setup(self):
        from cerebro.tests import test_engine as engine

        # Start a new connection
        self.connection = engine.connect()

        # Begin a non-ORM transaction
        self.trans = self.connection.begin()
        # Bind the session to the connection
        db_session.configure(bind=self.connection)

        self.clean_fixtures_cache()

        if hasattr(self, 'test_config'):
            #Support ability for tests to have their own test settings
            for attr, value in self.test_config.items():
                self.set_config(attr, value)

        #install class fixtures
        if hasattr(self, 'fixtures'):
            self.install_fixtures(self.fixtures)


    def _post_teardown(self):
        # Teardown the transaction
        if hasattr(self, "connection"):
            # Rollback database
            self.trans.rollback()
            db_session.remove()
            # We have to explicitely close the connection
            self.connection.close()
            del self.connection

        self.clean_fixtures_cache()


    def set_config(self, key, value):
        """Helper function that tests can call to set config values."""

        val = self.config_copy
        keys = key.split('.')

        for k in keys[:-1]:
            val = val[k]

        val[keys[-1]] = value

Important Note: This pattern can not be used for any methods with explicitly roll back a sqlalchemy session, as it will cascade downward and rollback the test’s transaction.

Changes in 2.0.0

Package version is now semantic

This release, and all future releases of Clay, will adhere to the versioning scheme described at http://semver.org/

In summary, the major version will be incremented for backwards incompatible changes, the minor version will be incremented for feature releases containing backwards compatible changes, and the patch version will be incremented for bugfix releases.

CLAY_ENVIRONMENT is deprecated

The CLAY_ENVIRONMENT variable should no longer be used to differentiate production and development environments, but you should rather use separate files passed to CLAY_CONFIG for this purpose.

Debug flags are no longer environment based

In previous releases, if CLAY_ENVIRONMENT=development was specified, logging and devserver configuration behaved differently. This check has been replaced with the debug.enabled boolean flag, which defaults to false. You will generally want this enabled in your development configuration.

An additional debug.logging boolean has been added that sets the default log level of logger instances returned from clay.config.get_logger() to DEBUG, rather than INFO.

Default logging configuration

Clay now supports arbitrary logging configuration by setting a logging key in your configuration, with contents adhering to the dictConfig schema specified by the Python standard library.

If no logging element is found in your configuration, Clay will default to logging all messages to stderr at the WARNING level or above.

Remote logging is not automatic

In previous releases, if logging.host were specified in the config, a clay.logger.UDPHandler was initialized and all log messages were directed to that host. This option is no longer available, and the UDPHandler must be initialized using a dictConfig.

logging:
  version: 1
  handlers:
    remote:
      class: clay.logger.UDPHandler
      level: DEBUG
      host: logs.example.com
      port: 22000
  loggers:
    root:
      level: DEBUG
      handlers: remote

flask.init is no longer required

The flask.init.import_name configuration option is now optional. If not specified, the import_name defaults to clayapp.

Clay now includes unit tests

The clay framework now has internal tests that may be run with python setup.py test. These tests are contained in the tests/ directory at the top level of the project and utilize the (webtest library.