Wednesday, July 20, 2011

Configuration - part 6

This series has been about the configuration manager within the Socorro project.  This posting is about a future direction of the configuration manager.  The goal is embrace and unify other configuration sources.  If there's a module that does something better or differently than the corresponding configuration manager analog, configuration manager should be able to exploit it.

Configuration manager depends on two sources of information: the definition list and the value source list.  The definition list tells the manager about the parameters, their default values and their documentation strings.  The values list is a hierarchy of sources of values for the the parameters defined in the definition list.

The configuration manager provides its own "language" for defining configuration parameters.  The language consists of namespaces and options.  Options consist of a name, a documentation string, a default value, and a conversion function.  In the existing world of Pythonic configuration, there are already systems that have similar definition "languages".

Argparse has an API that defines such a language.  There is a direct analog between the components of the Socorro configuration language and argparse's configuration language.  However, argparse goes beyond Socorro in defining relationships between parameters.  It allows some parameters to be assigned values based on "actions" that may interact with other parameters.

Socorro's configuration manager can use argparse's definitions with an adapter.  Since argparse can serve in Socorro as both a definition source and a value source, any program already using argparse can simply drop the argparse object into configmanager.  Configmanager will delegate to argparse all the responsibilities of definitions and command line parsing. the example below is the canonical argparse demo updated to use configuration manager.  With the addition of just a few lines of code, the example has been extended to allow parameter values to come from an ini file as well as the os environment.


    import argparse
    import socorro.lib.configurationmanager as cm

    parser = argparse.ArgumentParser(description='Process some integers.')
    parser.add_argument('integers', metavar='N', type=int, nargs='+',
                        help='an integer for the accumulator')
    parser.add_argument('--sum', dest='accumulate', action='store_const',
                        const=sum, default=max,
                        help='sum the integers (default: find the max)')

    configman = cm.ConfigurationManager((parser,), (ConfigParse,
                                                    os.environ,
                                                    parser))
    config = configman.get_config()
    print config.integers
    print config.accumulate


So how will (does) configuration manager do this? For each of the two external lists, definitions and values, there are a collections of adapters.  In turn, each of the lists will be walked and the elements of the list offered to the adapter controller.  Dispatched by the identity or type of the element, an appropriate adapter is instantiated to wrap the element.  The wrapper gives the configuration manager a uniform API to read from.  Once all the sources are wrapped with an identical API, the configuration manager can treat all sources equally.

Argparse, unfortunately, isn't so easy to adapt.  It's API is a sink for its parameter definitions.  Once defined, there is no external API to read them back out.  I believe this is an unfortunate oversight and, perhaps, a violation of the build to be extended mantra that I believe is so important to good library design.  So configuration manager has to cheat and peer inside argparse's parser object.  The violation isn't too serious as it is a read only operation, however, it is still far from ideal.

In the next article of the series, I'll discuss the use of ConfigParse as a value source and why it can't be used as a definition source.

Friday, July 15, 2011

Darlingtonia californica - part 1

We're getting quite excited these days about carnivorous plants.  With the lull in the rose business, we find ourselves with free time and we're indulging in some horticultural stuff just for fun.  We've acquired many different types of the carnivores and we're experimenting on exactly how to culture them.


These are Cobra Lillies, Darlingtonia californica.  They're native to North America, but only in a few places in California and Oregon.  Like most of the carnivores, they live in boggy areas of poor "mossy" nitrogen deficient soils.  They supplement their nutritional needs by luring insects into a tube from which they're incapable or too stupid to escape.  The bugs eventually die and drop into the enzyme rich fluid at the bottom of the tube for digestion.


Why is the pot all tipped up?  Well, in order to keep these guys thriving, we want to mimic their natural environment as much as we can.  They really like to have their roots wet, but not in just standing water: the water must move.  We pump it out of the low end and back to the high end.

The main pot is filled with crushed quartz filtration sand and live sphagnum moss.  We placed a small fountain pump into a standard nursery band (small rectangular pot, 3x3x6) with its sides cut out and replaced with plastic screen. That smaller pot is buried up to the brim, providing a hollow that fills with water free of sand and moss.


The tiny pump cycles the water back to the high end of the pot.  The water is effectively filtered through the Darlingtonia's potting medium.


This is an experiment.  We had some of these plants about a decade ago, but they didn't live long.  They have very specific cultural needs and we likely failed before because we didn't meet those needs. If this small experiment works, then we'll probably try to scale it up a bit.  Why?  Well, we aren't pushing to the point where we'll go into business selling these guys.  There isn't much of a market for such a finicky plant.  This is just for fun.


Paul has started a tumblr stream about all our carnivorous plants here on the farm.  His photography is spectacular, check out his blog.

Wednesday, July 13, 2011

Greenhouse 5: part 2

It's been a month and a half since I last posted about my private garden in Greenhouse 5.  It is a greenhouse off to the east of my big production greenhouses.  It was empty this year and I decided to make it my own private garden.  I planted it in May and now things are quite lush in there now despite of our cold wet Spring and disappointingly cool Summer.
I love the overgrown look.  Those are horse tails and lemon balm growing in the floor.  That's hops in background, ornamentals in the hanging pots on the right, peppers in the middle, and hanging pots of strawberries an the left.
The large pot of flowers at the bottom of this photo is homage to my late mother.  These are the things that she would have planted around our home back in the seventies in Montana.  I like the fiery colors of the first hanging pot.
The peppers on the left are really doing well.  Behind the peppers, but in front of the hops, are flat Italian beans.  They are already producing more than we can eat.
Some peppers are just ripening and will be ready for eating soon.  I've got about eight different types - mostly sweet but I can't resist growing hot ones, too.
The watermelons and gourds are a complete failure.  Voles keep undermining them and chewing the roots off.  It is very disappointing as these were the whole idea of gardening in the greenhouse this year.  I've yet to successfully grown a watermelon.  Next year, I think I'll try them in giant pots.

There are more pictures of both the May and July photos here: Greenhouse 5 Gallery

Monday, July 04, 2011

Configuration - part 5

In the previous installment of this series about my grand unified configuration system, I showed how namespaces can be used to resolve conflicts for parameters with the same name.  These namespaces can be nested arbitrarily deeply.

This system was designed for use in the Socorro Project, Mozilla's crash reporting system.  Socorro's ecosystem consists of a flock of thirty small scripts run periodically by cron and some long running daemon processes.  Each of these scripts has its own executable file.  If you examine these files, you'll see that they're almost all exactly the same: they setup logging and set of configuration variables, import a class that defines the business logic of the script, instantiate the class, call the class' "main" function. Aside from the business logic class, it's all boiler plate.  Since we've already got all these classes, its a simple step to instrument them so that the configuration manager can dynamically load them.  That eliminates all but one copy of the boiler plate code and drops the number of executable scripts from thirty to one. Everything is controlled by configuration.

    app_definition = cm.Namespace()
    app_definition.option('_application',
                          doc='the fully qualified class of the application',
                          default=some_default_class,
                          from_string_converter=cm.class_converter
                         )

    config_manager = cm.ConfigurationManager((
app_definition,), 
                                             (ConfigParser,os.environ, getopt))
    config = config_manager.get_config()

    application_class = config._application
    app_instance = application_class(config)
    logger.info('starting %s', application_class.app_name)
    app_instance.main()


For clarity, error handling has been stripped from this example. 

The configuration manager treats a option definition with the name '_application' in a special way.  It assumes that the class will have three  attributes: a string called, app_name, app_version, and a main function.   It makes the further assumption that the constructor for the class needs the DOM-like config dictionary for initialization.

With a properly instrumented cooperating class, that is all the code required to load the app, read the configuration ini file, override any config files option values with values from the environment and then override those with any provided on the command line.

Here's an edited version of a class compatible with the application class above.

    import os.path
    import socarro.configman as cm     
    import socorro.database as sdb      
    import socorro.lib.gzip_csv as gzcsv

    class DailyCsvApp(object):

        app_name = 'daily_csv'        
        app_version = '1.1'        
        app_doc = "This app produces a csv file of the current day's crash data"

        required_config = cm.Namespace() 
        required_config.option(name='day',
 
                              doc='the date to dump (YYYY-MM-DD)',
 
                              default=dt.date.today().isoformat(),
  
                             from_string_converter=cm.date_converter)
 
       required_config.option(name='outputPath',
 
                              doc="the path of the gzipped csv output file",
 
                              default='.')
 
       required_config.option('product',
 
                              doc='the name of the product to dump',
 
                              default='Firefox')
 
       required_config.option(name='version',
 
                              doc='the name of the version to dump',
 
                              default='4.%')
 
       # get database connection option definitions from the database module
 
       required_config.update(sdb.get_required_config())
   
        
        def __init__(self, config):
 
           self.config = config   
                
        def main(self):            
            with config.database.transaction() as db_conn:     
                output_filename = '.'.join(self.config.product,
 
                                          self.config.version,
 
                                          self.config.day.isoformat())
 
               csv_pathname = os.path.join(self.config.outputPath,
 
                                           output_filename)
 
               db_query = self.construct_query(self.config.day,
 
                                               self.config.product,
 
                                               self.confg.version)
  
              with gzcsv(csv_pathname) as csv_fp:
  
                  for a_row in sdb.query(db_conn, db_query):
  
                      csv_fp.write_row(a_row)
   
 
       def construct_query(self, day, product, version):
 
           # implementation omitted for brevity            pass

In Socorro, the generic app is called simply, "app.py".  To run see the options brought in by the DailyCsvApp class:

    
    $ python app.py --_application=socorro.cron.dailycsv --help
    daily_csv 1.1
        This app produces a csv file of the current day's crash data
       --_write
        write config file to stdout (conf, ini, json) (default: None)
      --config_path
        path for config file (not the filename) (default: ./)
      --databaseHost
        the host name of the database (defailt: localhost)

      --databaseUser
        the username on the database (default: breakpad_rw)
      --databasePassword
        the user password (default: *******)
      --day
        the date to dump (YYYY-MM-DD) (default: 2011-07-04)
      --help
        print this

      --outputPath
        the path of the gzipped csv output file (default: .)
      --product
        the name of the product to dump (default: Firefox)
      --version
        the name of the version to dump (default: 4.%)

Of course, it is easy to write out an ini file by just specifying the _write directive on the command line.  The configuration manager will write out the ini to the directory specified by the config_path using the name of the app as the basename of the ini file. 
In the next installment of this multipart missive, I'll talk about using other sources like argparse as the source of option definitions.