Things Gateway - Monitoring Solar Panels

I was scanning my network recently and noticed a device that I did not recognize.  It wasn't a rogue, it turned out to be the Enphase Energy controller for my solar panels.  We had them installed years ago and I had forgotten that they connected to the local network.  They feed data back to a cloud service called Enlighten that allows an owner to track their own panels and compare with their neighbors.  I never signed up for the service as I've always been suspicious and mistrusting of cloud services that connect to things in my home.

I probed the local solar panel controller to see what ports it had active and found a Web server.  Jumping over to a browser, I hit that web server and found some rudimentary data about my panels: current generation, lifetime generation, number of microinverters, number of microinverters online, software version numbers, etc.  This was all on the home page of the Web server.

Since I wrote the Virtual Weather Station to demonstrate how easy it is to wrap a Web resource as a Web Thing and integrate it into my Things Gateway, I saw an opportunity.  If I were to scrape this Web page, I could make a Web thing very easily.  I had it running less than an hour later.  It was amazingly simple.

I knew that I'd have to scrape the Web page and Python has a beautiful tool to do that: Beautiful Soup.  I read up on the package, explored it a bit with some sample data and then wrote the code that would scrape the home page from the local Enphase Web server.

The Web page itself is very simple, a set of nested tables with tags around the data I wanted.  To get Beautiful Soup to fetch my target data, I just had to count the number of tables and the number of <td> elements. However, it should be noted that is a very fragile way of scraping a Web page. It depends on the Enphase Web server to never change the layout of the home page. For me, my Enphase software is at version D3.14.27 (fd6ee1).  Maybe my screen scrape will work on other versions of the software, maybe it won't.  If it doesn't work for you, expect to explore Beautiful Soup to find the right location of the data.

I selected current generation, lifetime generation, number of microinverters, number of microinverters online to be the properties for my Solar Panel Web Thing:
    lifetime_generation = WoTThing.wot_property(
        name='lifetime_generation',
        initial_value=0.0,
        description='Total lifetime generation in KWh',
        value_source_fn=get_enphase_data,
        units='KWh'
    )
    generating_now = WoTThing.wot_property(
        name='generating_now',
        initial_value=0.0,
        description='currently generating in KWh',
        units='KW'
    )
    microinverter_total = WoTThing.wot_property(
        name='microinverter_total',
        initial_value=0,
        description='the number of microinverters installed'
    )
    microinverters_online = WoTThing.wot_property(
        name='microinverters_online',
        initial_value=0,
        description='the number of microinverters online',
    )
Modifying the asynchronous fetch data from my Virtual Weather Station, I came up with this code that polls the Enphase Web server for the data that I chose to make up my Web Thing:

    async def get_enphase_data(self):
        async with aiohttp.ClientSession() as session:
            async with async_timeout.timeout(config.seconds_for_timeout):
                async with session.get(config.target_url) as response:
                    enphase_home_page_raw = await response.text()
        enphase_page = BeautifulSoup(enphase_home_page_raw, 'html.parser')
        # this is stupidly fragile - we're assuming this page format never
        # changes from fetch to fetch - observation has shown this to be ok
        # but don't know if that will hold over Enphase software updates.
        td_elements = enphase_page.find_all('table')[2].find_all('td')
        self.lifetime_generation = self._scale_based_on_units(
            td_elements[self._LIFETIME_GENERATION].contents[0]
        )
        self.generating_now = self._scale_based_on_units(
            td_elements[self._CURRENTLY_GENERATING].contents[0]
        )
        self.microinverter_total = td_elements[self._MICROINVERTER_TOTAL].contents[0]
        self.microinverters_online = td_elements[self._MICROINVERTERS_ONLINE].contents[0]
(see the entire source code in the enphase_solar.py source file in the pywot/demo directory)

To run this Web Thing on your own instance of the Things Gateway, you'll need to do the following first:
        
    $ sudo pip3 install configman
    $ sudo pip3 install bs4
    $ sudo pip3 install webthing
    $ git clone https://github.com/twobraids/pywot.git
    $ cd pywot
    $ export PYTHONPATH=$PYTHONPATH:$PWD
    $ cd demo

Locate your Enphase Energy Web server on your local network.  I used the program Fing on my Android phone to scan my local WiFi network.

Start the Enphase Solar Web Thing like this:
        
    $ ./enphase_solar.py --enphase_address=YOUR_LOCAL_ENPHASE_IP_ADDRESS

The program will start logging what it is doing. First you'll see all the settable parameters and their current values.  Then it'll log about setting up properties and starting servers.  Finally, it will start logging new values that it has found, by default, every five minutes.  You can change that with the --seconds_between_polling command line switch.

Then go over to your Things Gateway and press the "+" button, add the Enphase Solar Panels Thing by pressing "Save" then "Done".  Clicking the "Splat" and you'll see the current stats of your solar panels.
Now you're ready to add some rules to the Gateway to notify you of events that happen to your solar panels.  Though, I suppose turning on lights to notify yourself that your solar panels are producing a lot of power may be counterproductive.

In the next edition of this series, I hope to show how to get a Web Thing to send text messages [editor's note: delayed until version 0.5 of the Things Gateway].   You could create a rule that notified you via text if one of your microinverters on your solar panels fail.  The opportunities are endless...

Thanks to my neighbors Susan & Todd for allowing me to photograph their solar panels - they're much more photogenic than mine.