As the old adage goes, you can't always have what you want. I want reliable, easy to use, secure console access to the modules in this project. However, the reality is more complicated.
The ESP32's development boards have a CP2102 USB to UART adapter that is used for easy firmware updates and serial console access, and they work perfectly when the ESP32 in question is on your desk at the end of a USB cable. It doesn't work so well, when the same module is buried inside a sealed ABS box, within a Chicken house, inside a muddy pen, at 11pm, when its raining outside and you really want to know why something isn't working.
I made the conscious decision during the platform design to have a console port connected from a Raspberry Pi Zero, to the ESP32 within the main unit. I can easily connect over WiFi to the Raspberry Pi, open minicom and I've got a console. This is tried and tested and what I've used in the previous version of this project.
The front panel on the other hand is only connected via a 12 pin cable and there is no support for the console to be mapped to that, which means that I can't connect to that module if something goes wrong. Other modules I'm thinking about will have the same issue, since some of them are not physically wired to the main unit at all, so a wireless method is required.
The industry standard way of having a console to various devices is via a UART console port. I've got serial access on the modules, even if I can't get to them in all cases. The other industry standard way is via Secure Shell, (SSH), which comes as standard on Linux and is recognised as being secure. Its predecessor Terminal Emulation over Networks (TELNET) is in essence just UART over a network port. The problem with this protocol is that its incredibly insecure and therefore not a good choice and is mostly obsolete.
The other common method is Bluetooth Serial Port Profile (SPP), which is sometimes also called Serial Line Over Bluetooth (SLOB), it arrived around the Bluetooth version 2 timeframe and it takes UART data and sends it over Bluetooth. As we know, Bluetooth has a PIN code on connection and as long as you don't go for one of the standard ones 0000, 1234 or similar, then its fairly secure too.
Unfortunately, ESP32-S2's do not have Bluetooth and the ESP32-S3 has Bluetooth Low Energy (BLE), which is a newer standard (v4.0 and above), however the standards bodies decided to not provide backwards compatibility, which means there is no SPP support. Various implementations of serial over Bluetooth have been created by a number of manufacturers, but all are proprietary and none are standards based, which makes interoperability non-existent.
When is a console useful ?
Another key consideration is when is a console useful and to who ?
Serial consoles, being trivially simple to configure and needing minimal hardware readily found in nearly all electronic solutions, as long as you know which pins to look on. They are often not particularly well secured and may just go to a boot loader. This also means that they catch all the bootup messages right from the start, this makes them very useful. The downside is that you need to find them, connect to them and they are often limited to 115.2K baud, which is fast enough in most cases, but can easily overrun if lots of dialogue occurs. a hard wired console port is single user only as there is only one physical set of wires to connect to.
Network based consoles such as SSH and Telnet require that the host is booted sufficiently to have a working network stack and all the daemons working on that stack. This means that they come up late, after boot, but provide access once the system is operating. They are however multi-user.
The protocols therefore cover different needs. Network based protocols aren't particularly useful for example if you are trying to configure the WiFi SSID and password on a newly built module and the network is down. However, WiFi can reach around 50M, depending on what its going through, whereas Bluetooth is around 10M-20M, again, depending on what its going through. WiFi devices need an access points to achieve communication and I'm trying to reduce dependencies where I can, to ensure that things run if we have a power failure.
During microcontroller development, when you restart the device, any network consoles (Telnet or SSH), it will disconnect and require that you log in again, this rapidly becomes frustrating and it detracts from the development activity.
Considering all of the above how can I provide alternative ways into the modules that work in a variety of ways, does not take lots of power or need many IO pins on the ESP32 nor take up much board space whilst still being cheap enough and secure enough for the project.
After some research, the fairly old HC-05 Bluetooth modules seemed to fit the need, they provide a transparent Bluetooth SPP to UART interface and can be found for about £4 each from reputable suppliers. These modules have a 6 pin interface, 2 for power and ground, 2 for serial data transfer and 2 further pins, one that tells you when someone has connected to it (STATE) and another to put it into a programming mode (EN) so that it can be configured via an AT command syntax, which will be familiar to anyone that has used Hayes modems back in the 1980's. This module looks ideal as it can be used on any of the ESP32's and being an older product, it uses the older non BLE Bluetooth standards, which by implication means its not Low Energy (from BLE), but that's a relative term after all
Being Bluetooth based, the connection will remain during power cycles and it can therefore be available from early in the boot cycle. The presence of the connection (STATE pin) can also be used as an interrupt input to start a console on that port.
Development work was undertaken to configure up the Bluetooth module and and SD card reader, using the same code and design as used on the main board. The previously allocated SD card pins on the TFT display board were re-mapped to the micro-SD card reader and this was squeezed in on the right of the TFT display and adjacent to the rotary encoder, making access tight, but in an ideal location should it be used, since the card slot is visible and accessible, but doesn't clutter the view in the front of the box. This will need integrating properly in the next iteration of the boar, but that's a future me problem. The Bluetooth console was wired in on the touch panel pins, since those are not being used in the design.
The result was that there were now 2 consoles on the unit, one over UART and one over Bluetooth, the best of both worlds. But, what to connect that Bluetooth adapter to at the other end ?
With the range of Bluetooth is only 10-20 metres in open space and the house being further away, with walls, floors and wire mesh fences to get through, and I know that didn't go well for 433MHz remote controls on the previous iteration of the Chicken coop, yet WiFi has no problems getting through the same wire fence, when coming at it from a slightly different angle.
What can I connect the Bluetooth console to in a reliable manner but in the pen ?
The Raspberry Pi Zero 2W has a Bluetooth capability with v4.2 support, however it does have backwards compatibility for SPP, meaning that it can connect to the HC-05 module. Configuration was done, scripts were written and the connection through a couple of ABS boxes and a several centimetres of wood went fine. A console session was established - Success. There are benefits of having the Pi there, even if I want to get get it out, to save power on the other hand. This whole project is a collection of things that work and a number of compromises that have to be made.
The presence of the new console and the related SD card slot means that files can be copied, firmware can be applied in a new way and certificates can be copied over for SSH when that is compiled in and configured to the project.
Attention then turned to getting SSH up and running, however looking at the example code, the complexity of it is high and it was not obvious how it could be easily integrated to the current code, so I considered again what I'm trying to do and the relative priorities, its ultimately about having multiple methods of connecting into the units in a secure enough manner so that the project can proceed.
The Chicken coop is already on a separate VLAN with firewall separation from everything else, this was done when I initially looked at Alexa integration on the older version 1 design, but it turned out to be far too complex to hook into that in a sensible manner, due to a number of limitations in that platform, so it didn't happen. The new hardware unblocks this and its on the backlog and looking far simpler to integrate to now.
Looking at the project backlog of problems and feature that I want to bring on-line, I decided that the risk/reward for Telnet is acceptable in the short term. It is insecure as it uses plaintext across the wire, so its trivial to snoop on, however given that there is only the chicken coop hardware on that network, the worst case is that someone could get in through the other security controls (like getting on the WiFi), mapping the network, finding the device, getting through the login process, then when finally in, get quickly disappointed with what is accessible, toggle the door or reboot the system and probably leave. In the first case, the girls would kick up a fuss and we would go and see what's up and sort it out, or the MQTT messages would indicate some odd behaviour and I'd go and look, or my other network monitoring tools would detect the connection, so there is a risk, but its acceptable, compared to some of the other things I want to do first.
The Telnet library was installed, configured and works fine, all be it that you have to reconnect every time that the system reboots and that rapidly becomes frustrating, so this will not be used for general debugging, just as an alternative way in, when Bluetooth is not working. However, another route in for a console is a good thing. Another plus for Telnet is that it doesn't need to go via the Raspberry Pi.
Consoles vs logging
Up until the start of this piece of work, there was only one console and only one place for diagnostic logging output, that was the UART console. Now, there are multiple console options and many parts of the code simply write to the console. Now there are several consoles, all of them see the same details and that isn't helpful as it would be frustrating if multiple people were working on different things, or even if you are looking at the same problem via two consoles.
Work was therefore needed to separate logging from command line output, just like it would be on any proper operating system. The impact of this change becomes fairly high, after all, I also want one common code base and making edits in multiple code bases for the same thing is just a waste of time and prone to errors and the code will inevitably drift over time, so action must be taken to prevent this. This is a great example, where unified code bases and re-usable libraries would help when making such broad changes to the existing code. Hence :
- The common code must be made common across all modules, reducing rework and removing the possibilities of errors. This in turn will require another git repo for the libraries. A quick look at this shows that Platform IO kinda sorta does this to a degree. This needs more work and is not the topic if this article.
- The logging output must go to the logging destinations. Separating this opens up the possibilities of logging to other places such as SYSLOG or an MQTT topic for each node. Both are nice options and need further work.
- Any command line output must only go to the console that requested the command in the first place
- Additional testing on another piece of work showed that logging messages can flood the console and there is no way to disable them at the moment which makes it very difficult to fix the problem.
A lot of work resulted from this. That includes the following changes :
- A new command line option called logging, which allows logging to be turned on and off for this session only. This helps a lot to suppress any unwanted logging output when working on any given problem.
- Lots of code updates were made to decide on each of the output messages, is this logging or command line based ?
- This in turn means that a console handle must be passed to many functions, so that they know where to write output to. Its a bit of a hack, since there is no underlying operating system in the chicken coop. There is no concept of kernel space and user space, there is no way of isolating user sessions. Remember this is a microcontroller, not a microprocessor. There is no STDIN, STDOUT, STDERR and I'm not trying to write an operating system, although I will use operating system principles and standards when enhancing the design.
- Many bugs were identified where output went to the wrong place. This also showed that the ANSI terminal code functionality needs extending to allow the codes to be sent to the correct console as overloaded functions are currently necessary to make this work and that's not pretty and its a waste of FLASH space, plus it breaks the DRY principle. Another task goes onto the backlog.
- The changes needed to be duplicated across multiple code bases, which wasted a lot of time. The common libraries work needs to be prioritised up to prevent further time being wasted as the project continues.
The result though is that there are now many options for logging, hooks for SYSLOG and MQTT logging, multiple console sessions can be established and they are visually separated from each other, which helps in debugging. Similarly, the ability to connect in via multiple different routes gives flexibilities and workarounds to various problems.