Mac Home Automation

XTension talks to it’s plugins via a socket connection. Currently most plugins run on the same machine as XTension but there is support for allowing the plugins to connect from external machines as well. If you’re writing your plugin in Python or other future supported scripting systems then all this work will be taken care of by the XTension object that is provided by the included scripts. This information is provided for those that might wish to implement a connection from an unsupported scripting system or a binary compiled application. Note that at this moment all communication at this level with XTension is unencrypted. No SSL support is available for this channel. The reason being that most or all of the communication at this level is between apps running on the same machine and so encrypting it doesn’t make any sense. If remote connections are necessary over the internet I would recommend using a VPN or other encrypted channel to forward the data such as an SSH tunnel.

At this moment the only examples are for connecting with Python. Since the connection to XTension is through a socket and the protocols are documented any language capable of implementing these protocols would work just fine though you might have to implemented these protocols yourself. I currently have release level plugin implementations in Python and Xojo, I also have an alpha level implementation written in Node JS for Javascript. That is a problem in releasing a plugin as you would either have to embed the NodeJS interpreter into your plugin or require that the user install it and it’s other dependencies before installing your plugin. At this moment I don’t recommend using that though there are other possibilities in the future for how I might handle javascript plugins.

A Xojo plugin is an excellent option if you require a binary application. The downsides are that the application is larger but otherwise those plugin libraries are well debugged as I use them myself quite a bit.

Another option is to wrap a command line utility in some simple python code rather than start from scratch. If the command line utility is not already installed on MacOS and licensing permits it you can include the binary utility in the plugin itself so that no other installations are necessary. This is the tack I took with the Switchbot plugin. Since controlling BluetoothLE would have required new python libraries be included and I wanted to provide a command line utility for it anyway I wrote the low level bluetooth handlers into a command line utility and included it inside some relatively simple Python code. You can examine the code as an example by downloading the switchbot plugin linked to above.

In XTension, a plugin is a separate command line application that communicates to XTension via a socket. The plugins do not share memory or CPU space with XTension itself so a misbehaving plugin is less likely to be able to degrade the performance of XTension itself. This also means that plugins will better be able to take advantage of multiple processors or other resources available on the server. Plugins which lose their communications with the device they are meant to be talking to or otherwise have errors can either recover from those themselves or they can simply quit. XTension will restart the plugin application after a short timeout and will continue to try to restart the plugin until it’s retry count is used up or until the connection is working again. This is especially useful for some devices which may not handle errors gracefully like USB devices. Due to limitations with some of their drivers it is simply not possible to gracefully recover from a USB bus hiccup or other error without quitting and restarting the host process. XTension will take care of that for you automatically if you are forced to deal with a device like that.

XTension launches your script or binary plugin based on the data in the “info.json” file. XTension will run your plugin with 3 command line parameters. No command line argument parsing is necessary, these parameters and their locations will not change for APIv2 though some plugins may require extra parameters added in the future to the end of the list, the first 3 will always be available. Note that most systems will pass you the link or name of your executable as the first parameter, so these would start at the second actual argument.

Once your plugin is launched and has found the command line params listed above it should open a TCP socket to the passed address and port. Once the connection is accepted you must send a short handshake string that contains a hello message and the connection ID number. The format is:

plugin hello message:IDNumber\r

The format of the plugin hello message is not important and is only used if there are problems in the connection. Otherwise XTension ignores this value. The string must contain only a single colon, and after the colon must be the ID number passed from the command line parameters followed by a carriage return.

Once you’ve sent that your pipe to XTension is open and you may start to receive commands if the units assigned to your interface change state.

XTension commands are sent via the pipe you just opened. All communications with XTension is through these command objects. They contain a header character that will always be “J” or “K” which controls the integer size that contains the size of the packet. You have to read the header of the command to properly parse the rest of the command. The rest of the command are key/value pairs. In the case of the command packet all keys are 4 byte strings. You can see them all in the python plugin include file “xtension_constants.py” included in the demo plugin. The value at any of those keys may be any length as each value as a length integer, either 2 or 4 bytes depending on the header, In order to parse these packets from a stream it is necessary to read the header and then either 2 or 4 more bytes for the size depending on the header. Then read the correct number of bytes and pass them off to an XTensionCommand class for parsing and creating of an XTension Command object that you can use to get and set values as well as send commands back to XTension. A useful implementation of the command would be a wrapper around a dictionary class. Received values are placed into the dictionary and to send the packet values are read out from the dictionary. Specific keys may appear in any order and are different based on the specific needs of the command being sent. You should save but ignore keys present that you are not using but not log errors if more data than necessary is present.

For packets that start with a J header all lengths are 2 byte unsigned integers. The K header is only used if the size of the packet is too large to be represented with the 2 bytes. You should be prepared to parse either packet however as larger database dumps or requested images will require the larger size.

For packets that start with a K header all lengths are 4 byte unsigned integers. It is identical to the J header packet above except for the longer lengths. You should be prepared to parse both as the K header packets will be automatically sent by XTension if a packet is too large to use the smaller sizes. For sending commands to XTension either command can be used however it’s only necessary to implement the K header for sending. All commands can be sent that way regardless of size and should you have to send a large amount of data further processing or checking the size is not necessary.

Most non-command data in XTension is stored and sent to you as xtData objects. These are basically a flattening of a dictionary of key/value pairs similar to the command stream above however the key and the value can both be any length and additional dictionaries can be embedded as well requiring a recursive parsing of embedded dictionaries. This is similar to how a JSON object might work but the xtData object in XTension predates the easy availability of JSON parsing utilities in the system so this structure is still used. I do have some work done internally to present this data in JSON format and if implementing this protocol is too complicated in your chosen system then please let me know and I will move making that available up the to do list.

xtData objects are sent flattened in a key in a command. For example if you send a command to XTension asking it to tell you about all the units that it currently has assigned to your interface they will be sent back in another Command object with a flattened xtData object stored in the key xtKeyData (as defined in the xtension_constants.py file) you can then extract that field from the command and pass it to a new xtData object to be parsed or however else you are going to parse the flattened data to your application. You should also be prepared to flatten your own xtData objects as some commands may require flattened xtData streams as part of the needed info.

Once you have requested that dump of all units assigned to your interface (or all units if you are requesting access to the entire database) you are effectively subscribed to changes to any of the internal values of those units. If anything changes in the unit a few ms seconds later an update will be generated and sent to your plugin. It will be the same layout as the initial database dump but will contain only the changed values. Your class should be able to receive these update packets and change only the included data in order to keep your plugins database of it’s units up to date. The xtData class in the xtension_plugin.py code also has a subscription method so that you can subscribe to specific keys in the database and receive a callback when they are changed in XTension. For example the demo dimmer plugin uses this to watch the status of the “dimmable” flag in the database. If a user changes a unit in XTension from dimmable to non-dimmable the database in the plugin process will be updated and the proper command will be sent to the hardware to conform the output channel to be the same. For some simpler plugins implementing this callback system may not be necessary but it is extremely handy for more full featured plugins.

Each xtData class will also contain a UUID. This is used when merging data so that embedded xtData classes can be found and the merging data routed to the proper embedded data class. The example xtData class contains a getContainerByUUID class that is used in the merge function to make sure the new data is applied to the proper container class. Embedded xtData classes are called containers in the original code and I may refer to them that way below. The UUID key will always be “_typeuuid_“

You can read the commented python code that parses and flattens these objects as well as handles the subscription to changes in the xtension_plugin.py demo code file.

All data in the xtData stream are strings. They can represent dates, colors, integer, floats or files. The type is stored as a separate entry in the dictionary with the kNamedTypePrefix appended to the key. For example if you save a string into the database with the name “myKey” and the value “myValue” it will generate 2 entries into the dictionary. The first will be the passed key and the passed value “myKey=myValue” the second will prepend the named type prefix onto the key and contain the type as a second 3 byte string. The second entry might look like “_typ_myKey=str” you can either ignore these type keys or use them to return the strings converted to the proper data types. When a value is set in a dictionary you should convert it to the proper string format and set a type entry to help the receiver know how to handle the value when it is received.

Data Type Constants:

The flattened xtData object requires recursive parsing in order to support the embedded xtData objects. See the xtData object in the xtension_plugin.py file for more info on how this might be done properly.

The initial header that identifies this packet as a flattened xtData object is 4 bytes “Xbdb” all other headers included in the stream are 1 byte long.

• Pass the stream or string or buffer to a method in the xtData class.
• Read 4 bytes and verify the Packet Header
• Read 1 byte and verify the format version
• Read 3 unused flag bytes
• begin a loop that will read until we receive the End Of Object header.
  • read 1 byte
    • if you’ve reached EOF then return
    • if it’s an End Of Object header then return
    • if it’s a Value Header then:
      • read 4 bytes of key length
      • read the above number of bytes into the Key string
      • read 4 bytes of Value length
      • read the above number of bytes into the Value
      • store the key/value pair in a dictionary or associative array or something simlar
    • if it’s an Object Header then:
      • create a new xtData object and prepare it for parsing
      • read 4 bytes that are the length of the name of the embedded object
      • read the number of bytes above and assign to a local name variable in the new xtData object
      • append the new xtData class to an array of embedded objects or other method of storing a reference to them
      • pass the stream off to the new xtData class and let it parse itself until it reaches an End Of Object header at which point it will return to you and the you can read the next header byte off the stream where it left off.

Once you’re connected you can ask XTension for your configuration info. This will include all the port names or internet addresses or any other data that is setup in the interface setup dialog or your dynamic interface embedded in the same. Sending this command also subscribes you to any changes in the settings xtData object in XTension. So if any value is edited by the user the new values will be sent as an update packet which you should merge with the existing xtData object thereby generating subscriber events for anyone that needs to know if those values change.

XTension will then respond with another command that will have the xtData object with all that information flattened into the command under the xtKeyData key.

Response:

If you have not yet received any base configuration then unflatten the xtData as sent and store it to a global or class variable. If you have already received this data and receive this command with this address again then it will contain only the changes that have been made to the settings by the user. Unflatten the changes into a new xtData object and merge it with the existing xtData class of your settings so that any changes can be sent to subscribers interested in those values.

For more detail see the plugin examples. To tell XTension that you have received a value change for a unit you would create an xtData object, insert the following key/value pairs, flatten the command and send it up the pipe. Note that all values stored in the xtCommand object are strings.

You can send a command to a Unit in XTension in 1 of 2 ways. A plugin that has received access to the entire database should include the unitqueID key in the command with the number set to the ID string from the Unit they want to control. A normal plugin with access only to it’s own units should include both the device type “tag” as specified in the info.json file as well as the address key. The command will be routed to and handled by the proper unit in XTension either way.