Configuration - part 5

In the previous installment of this series about my grand unified configuration system, I showed how namespaces can be used to resolve conflicts for parameters with the same name.  These namespaces can be nested arbitrarily deeply.

This system was designed for use in the Socorro Project, Mozilla's crash reporting system.  Socorro's ecosystem consists of a flock of thirty small scripts run periodically by cron and some long running daemon processes.  Each of these scripts has its own executable file.  If you examine these files, you'll see that they're almost all exactly the same: they setup logging and set of configuration variables, import a class that defines the business logic of the script, instantiate the class, call the class' "main" function. Aside from the business logic class, it's all boiler plate.  Since we've already got all these classes, its a simple step to instrument them so that the configuration manager can dynamically load them.  That eliminates all but one copy of the boiler plate code and drops the number of executable scripts from thirty to one. Everything is controlled by configuration.

    app_definition = cm.Namespace()
    app_definition.option('_application',
                          doc='the fully qualified class of the application',
                          default=some_default_class,
                          from_string_converter=cm.class_converter
                         )

    config_manager = cm.ConfigurationManager((
app_definition,), 
                                             (ConfigParser,os.environ, getopt))
    config = config_manager.get_config()

    application_class = config._application
    app_instance = application_class(config)
    logger.info('starting %s', application_class.app_name)
    app_instance.main()


For clarity, error handling has been stripped from this example. 

The configuration manager treats a option definition with the name '_application' in a special way.  It assumes that the class will have three  attributes: a string called, app_name, app_version, and a main function.   It makes the further assumption that the constructor for the class needs the DOM-like config dictionary for initialization.

With a properly instrumented cooperating class, that is all the code required to load the app, read the configuration ini file, override any config files option values with values from the environment and then override those with any provided on the command line.

Here's an edited version of a class compatible with the application class above.

    import os.path
    import socarro.configman as cm     
    import socorro.database as sdb      
    import socorro.lib.gzip_csv as gzcsv

    class DailyCsvApp(object):

        app_name = 'daily_csv'        
        app_version = '1.1'        
        app_doc = "This app produces a csv file of the current day's crash data"

        required_config = cm.Namespace() 
        required_config.option(name='day',
 
                              doc='the date to dump (YYYY-MM-DD)',
 
                              default=dt.date.today().isoformat(),
  
                             from_string_converter=cm.date_converter)
 
       required_config.option(name='outputPath',
 
                              doc="the path of the gzipped csv output file",
 
                              default='.')
 
       required_config.option('product',
 
                              doc='the name of the product to dump',
 
                              default='Firefox')
 
       required_config.option(name='version',
 
                              doc='the name of the version to dump',
 
                              default='4.%')
 
       # get database connection option definitions from the database module
 
       required_config.update(sdb.get_required_config())
   
        
        def __init__(self, config):
 
           self.config = config   
                
        def main(self):            
            with config.database.transaction() as db_conn:     
                output_filename = '.'.join(self.config.product,
 
                                          self.config.version,
 
                                          self.config.day.isoformat())
 
               csv_pathname = os.path.join(self.config.outputPath,
 
                                           output_filename)
 
               db_query = self.construct_query(self.config.day,
 
                                               self.config.product,
 
                                               self.confg.version)
  
              with gzcsv(csv_pathname) as csv_fp:
  
                  for a_row in sdb.query(db_conn, db_query):
  
                      csv_fp.write_row(a_row)
   
 
       def construct_query(self, day, product, version):
 
           # implementation omitted for brevity            pass

In Socorro, the generic app is called simply, "app.py".  To run see the options brought in by the DailyCsvApp class:

    
    $ python app.py --_application=socorro.cron.dailycsv --help
    daily_csv 1.1
        This app produces a csv file of the current day's crash data
       --_write
        write config file to stdout (conf, ini, json) (default: None)
      --config_path
        path for config file (not the filename) (default: ./)
      --databaseHost
        the host name of the database (defailt: localhost)

      --databaseUser
        the username on the database (default: breakpad_rw)
      --databasePassword
        the user password (default: *******)
      --day
        the date to dump (YYYY-MM-DD) (default: 2011-07-04)
      --help
        print this

      --outputPath
        the path of the gzipped csv output file (default: .)
      --product
        the name of the product to dump (default: Firefox)
      --version
        the name of the version to dump (default: 4.%)

Of course, it is easy to write out an ini file by just specifying the _write directive on the command line.  The configuration manager will write out the ini to the directory specified by the config_path using the name of the app as the basename of the ini file. 
In the next installment of this multipart missive, I'll talk about using other sources like argparse as the source of option definitions.