SSH Tunnelling Functionality for Boardies MySQL Manager Postmortem

Chris Board

Jul 22, 20172 min read

On Saturday 22nd July at around 21:45 BST we noticed that Boardies MySQL Manager for Android was failing to do SSH tunnelling stating that the SSH Tunnelling was not available at that time and to try again later.

Upon investigation we discovered that the service had stopped on both of our servers hence why it was a complete failure for the SSH tunnel functionality instead of it being a partial outage.

checking through the logs it looks like it had stopped on both servers around 17th July. Although we've not found anything in the logs to indicate why the process stopped but the process should have automatically restarted.

We then investigated as to why the process didn't respawn and we noticed a typo in the configuration start script for the respawn option.

Steps Taken To Prevent This Happening Again######

We have resolved the configuration issue within the Linux start up scripts to enable the process to respawn again should the process stop unexpectedly.

We have also modified our processes start script so that as soon as the service is restarted, it will raise us an alarm to inform us that it had an unexpected restarted and look into the issue so we can release a fix.

We have also added a monitoring script which monitors the process for the SSH tunnelling plugin to check that the process is running. Should the process stop, and for some reason not be respawned we will receive an alert to inform us that the process is no longer running so we can investigate as soon as possible and restore the service.

We apologise for any inconvenience that this may have caused and will do everything to ensure nothing like this happens again, either with this service, or any of our other apps or services that we release.

Thanks
Chris Board
Boardies IT Solutions

Test Track

Are you a developer or involved in Quality Assurance Testing or User Acceptance Testing, you might be interested in Test Track

A simple and affordable test planning and management solution.