Building a Custom Input Module

Contents

Introduction
Resources
Input Module Tutorial
- Required Input Module Components
- Additional Input Module Components

Introduction

An Input module import system has been implemented in Mycodo to allow new Inputs to be created within a single file and imported into Mycodo for use within the system. An Input module can be imported into Mycodo on the Configure -> Custom Inputs page and will appear in the Add Input dropdown selection list on the Setup -> Input page.

Resources

Built-in Inputs: Mycodo/mycodo/inputs

Example Input that generates random data: minimal_humidity_temperature.py

Example Input with all options: example_all_options_temperature.py

Example Input with all options as well as per-channel options: example_all_options_temperature_with_channel_options.py

Input Module Tutorial

This will be a brief tutorial covering the components of a simple Input module with the purpose of helping you develop your own. I'll be using the MCP9808 temperature sensor Input module (located at Mycodo/mycodo/inputs/mcp9808.py) for this example, as it contains the minimum components necessary for an Input module to operate. Following this tutorial will be additional options and functions available to Input modules for advanced use.

Required Input Module Components

Measurements Dictionary

The first part of an Input module is the measurements_dict dictionary. measurements_dict contains all measurements that you might store in the measurements database for this Input. This could include measurements that the sensor is capable of measuring in addition to measurements that could be calculated. For the MCP9808, since only temperature is measured, measurements_dict looks like this:

measurements_dict = {
    0: {
        'measurement': 'temperature',
        'unit': 'C'
    }
}

Had this sensor been capable of measuring more than one condition or if calculated measurements were desired, additional dictionary entries could be added with an incrementing dictionary key (0, 1, 2, etc.). This dictionary key number is presented to the user as the channel number of the Input. For instance, a temperature/humidity sensor may be able to measure only temperature and humidity, but dew point could be calculated from these two measurements. If dew point was desired to be stored in the measurement database, this would need to be included in measurements_dict, like so:

measurements_dict = {
    0: {
        'measurement': 'temperature',
        'unit': 'C'
    },
    1: {
        'measurement': 'humidity',
        'unit': 'percent'
    },
    2: {
        'measurement': 'dewpoint',
        'unit': 'C'
    }
}

Additionally, measurements_dict also requires both the units and measurements to exist in the measurement/unit database. You are required to add measurements and units from the Configure -> Measurements page if they don't already exist, prior to importing the Input module.

For example, if there's a measurement you require but it doesn't currently exist in the database, do the following:

Add the unit:

Unit ID: valve
Unit Name: Valve State
Unit Abbreviation: Position

Add the measurement:

Measurement ID: valve_state
Measurement Name: Valve State
Measurement Units (from Unit ID, above): valve

Then, add them to measurements_dict, like so:

measurements_dict = {
    0: {
        'measurement': 'valve_state',
        'unit': 'valve'
    }
}

Input Information Dictionary

Next is the INPUT_INFORMATION dictionary, where all information about an input is contained. Let's take a look at this dictionary to see what's required for it to operate.

INPUT_INFORMATION = {
    'input_name_unique': 'MCP9808',
    'input_manufacturer': 'Microchip',
    'input_name': 'MCP9808',
    'input_library': 'Adafruit_MCP9808',
    'measurements_name': 'Temperature',
    'measurements_dict': measurements_dict,
    'url_manufacturer': 'https://www.microchip.com/wwwproducts/en/en556182',
    'url_datasheet': 'http://ww1.microchip.com/downloads/en/DeviceDoc/MCP9808-0.5C-Maximum-Accuracy-Digital-Temperature-Sensor-Data-Sheet-DS20005095B.pdf',
    'url_product_purchase': 'https://www.adafruit.com/product/1782',

    'dependencies_module': [
        ('pip-pypi', 'Adafruit_GPIO', 'Adafruit_GPIO'),
        ('pip-git', 'Adafruit_MCP9808', 'git://github.com/adafruit/Adafruit_Python_MCP9808.git#egg=adafruit-mcp9808'),
    ],

    'interfaces': ['I2C'],
    'i2c_location': ['0x18', '0x19', '0x1a', '0x1b', '0x1c', '0x1d', '0x1e', '0x1f'],
    'i2c_address_editable': False,

    'options_enabled': [
        'i2c_location',
        'period',
        'pre_output'
    ],
    'options_disabled': ['interface']
}

input_name_unique

input_name_unique is an ID for the input that must be unique from all other inputs, as this is what's used to differentiate it from others. If you're copying an Input module and using it as a template, you must change at least this name to something different before it can be imported into Mycodo.

input_manufacturer, input_name, input_library, measurements_name

input_manufacturer is the manufacturer name, input_name is the input name, input_library is the library or libraries used to read the sensor, and measurements_name is the measurement or measurements that can be obtained from the input. These are all used to generate the Input name that's presented in the list of Inputs that can be added on the Data page. These names can be anything, but it's suggested to follow the format of the other Inputs. If copying a built-in Input to modify for your own needs, it's advised to modify one or more of these values so your new Input's name will be distinguishable from the Input you copied.

measurements_dict

measurements_dict merely points to our previously-created dictionary of measurements, measurements_dict, and is required for the Input module to operate properly.

url_manufacturer, url_datasheet, url_product_purchase, url_additional

These can either be a URL string or a list of URL strings that provide links to more information about the Input. These will be used to generate the documentation in the manual.

dependencies_module

Next is dependencies_module, which contains all the software dependencies for the Input. This is structured as a list of tuples, with each tuple comprised of three strings.

The first string is used to indicate how the dependency should be installed. This can be "pip-pypi" to install a package from pypi.org with pip, "pip-git" to install a package from github.com with pip, or "apt" to install a package via apt.

The second string is the Python library name that will be attempted to be imported (if installed with pip), or the apt package name (if installed with apt), and is used to determine if the dependency is installed.

The last string is the name used to install the library, either via pip or apt. In the above example, Adafruit_GPIO is a library available via pypi.org, and will be installed via pip with the command pip install Adafruit_GPIO. Adafruit_MCP9808 is installed via pip as well, but uses a git repository instead of pypi, with the command pip install -e git://github.com/adafruit/Adafruit_Python_MCP9808.git#egg=adafruit-mcp9808. If apt is specified, the install will proceed with the command sudo apt install package_name.

When a user attempts to add this Input, Mycodo will check if these libraries are installed. If they are not installed, Mycodo will then automatically install them. See the other input modules as examples for how to properly format the dependencies list.

Note: For the "pip" install option, a package version can be specified, such as "Adafruit-GPIO==2.3.1", however this can cause conflicts if other Inputs or modules also have this package listed as a dependency and don't specify the same version number, which can cause a different version to be installed.

interface

Next is interface, and is used to determine how to communicate with the Input and what type of options to display to the user. Since this input communicates with the sensor via the I2C bus, we specify I2C. It's possible for some sensors to be able to communicate in multiple ways, such as UART, 1WIRE, SPI, FTDI, among others. For each interface you provide in this list, multiple Inputs will be listed on the Input Add dropdown. For instance, if an Input can be communicated with using both I2C and UART, then you will have the option of adding "[UART] My Input" or "[I2C] My Input". If the I2C variant is selected and added, the I2C Bus and I2C Address will be options available to the user to set. If the UART variant is added, the options for UART communication will be available to the user to set.

Add all applicable communication methods to the interfaces list. In the above example, the sensor only has an I2C interface. i2c_location defines a list of strings for all possible I2C addresses that the sensor can be found at. If these are the only addresses that the sensor allows, i2c_address_editable should be set to False. However, if the sensor allows the user to reconfigure the address to something unique, this should be set to True to display an editable box for the user to define the I2C address rather than a pre-defined dropdown list.

Note: If your Input communicated with a protocol other than I2C, it's recommended to find a built-in Input with the same communication interface to review the proper usage and formatting to properly develop your Input module.

options_enabled, options_disabled

Next is options_enabled and options_disabled, which determine which Input configuration options appear on the web interface for the user to set or view. Since we want to be able to set the relevant settings for I2C communication, we add "i2c_location" to options_enabled, which will display the option fields I2C Address and I2C Bus as options for the user to set. Additionally, the Period (seconds) is enabled to set how often the Input acquires measurements and Pre Output is enabled to allow an Output to be selected that runs during measurement acquisition. interface is disabled, meaning it will be visible with the other options, but it will be disabled and unable to be edited, allowing it indicate what interface has been selected for the Input.

The following table shows for each interface enabled, what needs to be included in options_enabled, how to set default options, as well as the variables that are available within the module.

Interfaces	options_enabled	INPUT_INFORMATION default options	variables
I2C	i2c_location	'i2c_location': ['0x64', '0x65'], 'i2c_address_editable': True	self.input_dev.i2c_address, self.input_dev.i2c_bus
UART	uart_location	'uart_location': '/dev/ttyAMA0', 'uart_baud_rate': 9600	self.input_dev.uart_location, self.input_dev.baud_rate
FTDI	ftdi_location	'ftdi_location': '/dev/ttyUSB0'	self.input_dev.ftdi_location
1WIRE	location		self.input_dev.location

There are other options that can be used within INPUT_INFORMATION. Refer to the other Input and example modules to see how they can be used.

Class: InputModule

The InputModule class contains all the functions to query measurements from the sensor itself as well as any other functions, such as those to perform calibration, calculations, or other tasks. This class inherits from the AbstractInput base class in base_input.py.

class InputModule(AbstractInput):
    """ A sensor support class that monitors the MCP9808's temperature """

    def __init__(self, input_dev, testing=False):
        super(InputModule, self).__init__(input_dev, testing=testing, name=__name__)

        self.sensor = None

        if not testing:
            self.initialize_input()

    def initialize_input(self):
        """ Initialize the MCP9808 sensor class """
        from Adafruit_MCP9808 import MCP9808

        try:
            self.sensor = MCP9808.MCP9808(
                address=int(str(self.input_dev.i2c_location), 16),
                busnum=self.input_dev.i2c_bus)
            self.sensor.begin()
            self.logger.info("Sensor initialized without causing an exception!")
        except:
            self.logger.exception("Exception encountered while initializing input!")

    def get_measurement(self):
        """ Gets the MCP9808's temperature in Celsius """
        self.return_dict = copy.deepcopy(measurements_dict)

        if not self.sensor:
            self.logger.error("Sensor not set up")
            return

        try:
            temperature_c = self.sensor.readTempC()
            self.logger.debug("Value returned from the sensor library: {}. Saving to database.".format(temperature_c))
            self.value_set(0, temperature_c)
            return self.return_dict
        except Exception as msg:
            self.logger.exception("Input read failure: {}".format(msg))

Function: init()

__init__() is executed upon the class instantiation, and is generally reserved for defining variable default values. Here, we set self.sensor to None to indicate the Input has not yet been successfully initialized.

Function: initialize_input()

initialize_input() is executed following instantiation. This is where our dependency library or other libraries (if there are any) are typically imported and any classes initialized. In the above example code, we create an instance of the class MCP9808 from our imported library and assign this object to the variable self.sensor. We initialize the class with the I2C address (input_dev.i2c_location) and I2C bus (input_dev.i2c_bus) parameters set by the user in the web user interface, and stored in input_dev of our settings database. The MCP9808 library also requires begin() to be called to properly initialize the library/sensor prior to querying the sensor for measurements.

Function: get_measurement()

Next we have the get_measurement() function. This is called every Input Period to acquire a measurement from the sensor and store it in the measurement database. This function contains the code to query the sensor itself and the function to store the measurement that was obtained.

Function: value_set(channel, value, timestamp=None)

This sets the value to store in the measurements database for the particular measurement channel. Only the measurements set with this function are stored in the measurements database following the return of the get_measurements() function. An optional timestamp may be provided as a datetime object in UTC time, otherwise the current time when get_measurements() returns is used.

In the above code, we begin with self.return_dict = copy.deepcopy(measurements_dict) that creates a blank template of our measurements dictionary. Next, we check if self.sensor is still None or if it was successfully set up in initialize_input(). Next, we check if channel 0 is enabled with self.is_enabled(0). If enabled, we call self.sensor.readTempC() to return a temperature measurement and set temperature_c to that value. We then print a line in the Damon log with the returned value (if Log Level: Debug is enabled in the Input settings on the web user interface). Last, this temperature value is passed as a parameter within self.value_set(0, temperature_c) in order to indicate we wish to store this measurement in the database. If we desired to store additional measurements, such as humidity, under key 1 of our template, we would call self.value_set(1, humidity_value), where humidity_value is a numerical value representing the humidity. Last, we return self.return_dict to indicate all measurements were successfully obtained and the Input module should save the measurements that were stored in the template to the measurements database.

Additional Input Module Components

Logging

Logging is a critical aspect of any software development. It can be used to provide feedback about what code is executing, when code is executing, and what the values of variables are, among other uses. There are three logging functions that you should become familiar with in order to effectively develop and debug Inputs: self.logging.info(), self.logging.debug(), and self.logging.exception(). All log lines will be able to be viewed in the Daemon Log on the Configure -> Mycodo Logs page. self.logging.info() will always display a log line if it's executed, at the INFO level. self.logging.debug() will only display a log line if the option "Log Level: Debug" is enabled for the Input, which is nice to be able to have several diagnostic log lines in your Input but be able to quickly disable them by disabling "Log Level: Debug". Last, self.logging.exception() is useful for printing the full traceback of an exception, as can be seen in use in the try/except block during the input initialization.

Custom Options

Beyond the ability to configure how to communicate with the Input device (e.g. I2C address, I2C Bus, UART device, UART baud rate, FTDI device, 1-Wire address), you can create your own options that will be presented for the user to configure and be available to use within the Input module as variables. Below are examples of the various types.

def constraints_pass_positive_value(mod_input, value):
    """Check if the user input is acceptable"""
    errors = []
    all_passed = True
    if value <= 0:  # Ensure value is positive
        all_passed = False
        errors.append("Must be a positive value")
    return all_passed, errors, mod_input

INPUT_INFORMATION = {
    # Custom options that can be set by the user in the web interface.
    'custom_options_message': 'This is a message displayed for custom options.',
    'custom_options': [
        {
            'id': 'variable_boolean',
            'type': 'bool',
            'default_value': True,
            'name': 'Checkbox',
            'phrase': 'Description of the checkbox option'
        },
        {
            'id': 'variable_integer',
            'type': 'integer',
            'default_value': 10,
            'constraints_pass': constraints_pass_positive_value,
            'name': 'Integer Value',
            'phrase': 'Description of the integer option'
        },
        {
            'id': 'variable_float',
            'type': 'float',
            'default_value': 5.2,
            'constraints_pass': constraints_pass_positive_value,
            'name': 'Float Value',
            'phrase': 'Description of the float option'
        },
        {  # This starts a new line for the next options
            'type': 'new_line'
        },
        {  # This message will be displayed after the new line
            'type': 'message',
            'default_value': 'Another message between options',
        },
        {
            'id': 'variable_select_dropdown',
            'type': 'select',
            'default_value': '2',
            'options_select': [
                ('1', 'Selection One'),
                ('2', 'Selection Two'),
                ('3', 'Selection Three'),
                ('5', 'Selection Four'),
            ],
            'constraints_pass': constraints_pass_measure_range,
            'name': 'Select Options',
            'phrase': 'Description of the selection option'
        },
        {
            'id': 'variable_select_device',
            'type': 'select_device',
            'default_value': '',
            'options_select': [
                'Output',
            ],
            'name': 'Select Device',
            'phrase': 'Description of the device selection'
        },
        {
            'id': 'variable_select_measurement',
            'type': 'select_measurement',
            'default_value': '',
            'options_select': [
                'Input',
                'Math'
            ],
            'name': 'Select Measurement',
            'phrase': 'Description of the measurement selection'
        }
    ]
}

class InputModule(AbstractInput):
    def __init__(self, input_dev, testing=False):
        super(InputModule, self).__init__(input_dev, testing=testing, name=__name__)

        # Initialize custom option variables to None
        self.variable_boolean = None
        self.variable_integer = None
        self.variable_float = None
        self.variable_select_dropdown = None
        self.variable_select_device_id = None
        self.variable_select_measurement_device_id = None
        self.variable_select_measurement_measurement_id = None

        # Set custom option variables to defaults or user-set values
        self.setup_custom_options(INPUT_INFORMATION['custom_options'], input_dev)

        self.logger.info("Variable values: {}, {}, {}, {}, {}, {}, {}".format(
            self.variable_boolean,
            self.variable_integer,
            self.variable_float,
            self.variable_select_dropdown,
            self.variable_select_device_id,
            self.variable_select_measurement_device_id,
            self.variable_select_measurement_measurement_id))

        self.logger.debug("Turning Output with ID {} ON".format(
            self.variable_select_device_id))
        from mycodo.mycodo_client import DaemonControl
        self.control = DaemonControl()
        self.control.output_on(self.variable_select_device_id)

        last_measurement = self.get_last_measurement(
            self.variable_select_measurement_device_id,
            self.variable_select_measurement_measurement_id,
            max_age=3600)
        self.logger.debug("Last measurement (past 3600 seconds): {}".format(
            last_measurement))

In the above code, there are options set in custom_options for the user to be able to set a boolean value, set an integer value, set a float value, select text from a dropdown, select an output device from a dropdown, and select a specific measurement of an Input or Math controller from a dropdown. Note that the option id must contain no spaces, as it will be used later to determine the variable names (which cannot have spaces). There's also the ability to add a message at the top of these options or in between options, as well as create a new line. Optionally, constraints_pass may be set to a function to verify the user input. In this example, constraints_pass_positive_value() tests whether the value entered is positive. If the user tries to save a non-positive value, the specified error is presented and the value is not saved.

When we get to __init__(), we see we must first name the custom option variables to the same name as the option id and set them to None. The only exception is the device and measurements selection options. The device option variable name must have "_id" appended to the end and the measurement option becomes two variables, one with "_device_id" appended to the end and the other with "_measurement_id" appended to the end. These correspond to either the device ID itself (Input, Outputs, PID, etc.) or the measurement ID (such as the ID of channel 0 of an Input, representing temperature). Following setting these variables to None, we execute self.setup_custom_options(), which sets them to the values the user configured in the web interface. Following this, we have a log line that will print the values of all the variables. Next we demonstrate how to use the output device variable to turn the output on. And last we demonstrate how to use the measurement variables to retrieve and print the last stored value of the selected measurement.

Abstract Base Class Functions

These are functions contained in the abstract base classes AbstractInput of base_input.py or AbstractBaseController of abstract_base_controller.py. These may be used within all Input modules because they inherit these functions.

Function: is_enabled(channel)

Parameter channel: integer, the numeric key of the measurements_dict dictionary of measurements.
Returns: True if measurement channel enabled, False if measurement channel disabled.

This function checks if the particular measurement is enabled. If it is enabled, it returns True and the indented block beyond it executes, causing a measurement to be acquired and the value to be saved to the measurement database. This has only been added to this Input for demonstration purposes. Typically, for an Input with only one measurement, there's no need to check if it's enabled. However, if you are building an Input with multiple possible measurements, you can give the user the option to select which measurements they want to query the sensor library to measure and store in the measurement database. Since acquiring measurements takes time and storing measurements takes space, disabling unneeded measurements can save both time and space. To provide the user with an option to select which measurements are enabled, add the "measurements_select" option to the options_enabled list.

Function: value_get(channel)

Parameter channel: integer, the numeric key of the measurements_dict dictionary of measurements.
Returns: The value stored with value_set() since get_measurement() was first called.

This function is useful for referencing previously-acquired measurements without having to explicitly declare variables to hold the measurements. For example, a temperature and humidity sensor will acquire the measurements temperature and humidity, but dew point can also be calculated from these two measurements. In the below code, we check whether each measurement is enabled, then use value_set() to store the measurement. Then, we check if measurements 0, 1, and 2 are all enabled, we calculate the dew point with calculate_dewpoint() and set measurement 3.

if self.is_enabled(0):
    self.value_set(0, self.read_temperature())

if self.is_enabled(1):
    self.value_set(1, self.read_humidity())

 if self.is_enabled(2) and self.is_enabled(0) and self.is_enabled(1):
    self.value_set(2, calculate_dewpoint(self.value_get(0), self.value_get(1)))

set_custom_option(option, value)

Parameter option: string, the name of the option you would like to set in the settings database
Parameter value: integer/float/string/list/dictionary, the value of the option you would like to set
Returns: None

This function is useful for setting values that can be retrieved across Input activations (for example, you can record how many times the Input has been activated, by incrementing the stored value by 1 every time the Input is initialized). Typically, when an Input is deactivated and reactivated, all variables are reinitialized. To store a value that can be acquired after deactivating and reactivating, we can store the value in the settings database. Be mindful to use an option name that is distinct from any options already stored in the database, otherwise you may overwrite that value with your own.

if self.get_custom_option("my_custom_option_name"):
    my_option = self.get_custom_option("my_custom_option_name")
else:
    my_option = 0

self.set_custom_option("my_custom_option_name", my_option + 1)

get_custom_option(option)

Parameter option: string, the name of the option you would like to get from the settings database
Returns: None if the option cannot be found or the previously-set value if the option is found

This function gets the value for the specified option from the settings database. It's advised to use an if/else statement in order to set your variable to a default value if the option isn't found (if it's the first the Input has run and your option has not yet been set for the first time).

if self.get_custom_option("my_custom_option_name"):
    my_option = self.get_custom_option("my_custom_option_name")
else:
    my_option = 0

delete_custom_option(option)

Parameter option: string, the name of the option you would like to get from the settings database
Returns: None if the option cannot be found or the previously-set value if the option is found

This deletes the entry for the specified option.

Function: filter_average(name, init_max=0, measurement=None)

Parameter name: string, a dictionary key to store measurements
Parameter init_max: integer, function must first be called with this, sets how many measurements to store/average.
Parameter measurement: integer/float, the measurement to store in a rolling list to average.
Returns: An average of past-stored measurements.

This function is used to average measurements. This is useful for smoothing out noisy signals such as from analog-to-digital converters. For instance, you can first call filter_average("light", init_max=20) in initialize_input() prior to storing several values.

light_average = None
for _ in range(20):
    light_average = self.filter_average("light", measurement=self.sensor.get_light())

self.value_set(0, light_average)

You could also merely perform a rolling average of several past measurements rather than performing multiple measurements per Input Period.

self.value_set(0, self.filter_average("light", measurement=self.sensor.get_light()))

Function: lock_acquire(lockfile, timeout)

Parameter lockfile: string, the full path to the lock file to be created.
Parameter timeout: integer, the timeout period for acquiring a lock.
Returns: True if successfully locked, False if not successfully locked

This is a locking method for gaining access to shared resources, such as devices, files, sensors, or anything else. An example if it's use is below.

if self.lock_acquire("/var/lock/lockfile_my_input", timeout=10):
    try:
        # Code to run while we have a lock.
        # Any other code also in a lockfile block like this using the same
        # lockfile cannot run until this lock has been released.
    finally:
        self.lock_release("/var/lock/lockfile_my_input")

Function: lock_locked(lockfile)

Parameter lockfile: string, the full path to the lock file.
Returns: True if locked, False if not locked

Determines if a lock is already in place for a specific lockfile.

Function: lock_release(lockfile)

Parameter lockfile: string, the full path to the lock file.
Returns: None

Releases a lock and forces the deletion of the lockfile.

Function: is_acquiring_measurement()

This function can be used to determine if an Input is currently acquiring a measurement by returning the value of self.acquiring_measurement. This entails self.acquiring_measurement is set to True prior to measurement acquisition and set to False after.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly