As you might be aware, on the 17th January we had an issue where our site went offline due to a browser error stating that there was a certificate issue due to the certificate being revoked.

We use a cloud VPS provider for our servers and load balancer. These servers have their own SSL ceritifcate installed for HTTPS access. We also use a service provider called Cloudflare which hides our site behind them to protect our servers from hackers, spam bots and also improve the response times by acting as CDN (Content Delivery Network) as well as many other security features.

Via Cloudflare, they provide another SSL Certificate that you see for the domain and subdomains of Boardies IT Solutions that pass through cloudflare, you'll only see our certificate if you hit the servers directly, which you shouldn't be able to and usually there's no need to. When we first saw this issue we weren't sure whether it was cloudflare related or own server related.

We modified our PCs own hosts file so that it can bypass the domains DNS so when I visit boardiesitsolutions.com it would instead resolve via the hosts file and point to one of the servers or the load balancer directly and bypass cloudflare and the site returned with no issue.

If anyone's used Cloudflare, in the DNS settings, there is a little cloud next to each A record or AAAA record. If the cloud is orange, then traffic is being routed through cloudflare, and then when required cloudflare will fetch content from the servers. However, you can turn this cloud off, and then all traffic hits our server directly and no traffic, other than DNS is passed through cloudflare. We then waited a few minutes to make sure DNS changes were updated, and we removed the override from the PCs own hosts file so it has to use DNS again and again the site worked, turn cloudflare back on, and the certificate error came back, so we knew the issue was related to cloudflare and unfortunately this is where it all became a bit of a nightmare.

Before we get into this, I want to say, I've been using Cloudflare for a long time, I believe I signed up and started using them within the first six months of Cloudflare launching which was in 2009, so been using them for some time.

We're not going to go into a huge amount of detail as to what's been going on, but the basis is, we've been either waiting for a period of time for cloudflare to respond, and having to chase up and going round in circles between them and another company being the certificate authority for the cloudflare certificate to try and determine why the certificate got revoked.

We can in theory clear the cloudflare certificate and re-issue it however, until we find out why it got revoked I'm a bit reluctant to do it in case it happens again for some reason and we're back to square 1.

What are we doing now?

At the moment we're still talking with Cloudflare and the certificate authority to establish the cause of the certificate being revoked. We've got round the issue by turning off cloudflare for the main domains, such as boardiesitsolutions.com, support.boardiesitsolutions.com and status.boardiesitsolutions.com. We've left an internal subdomain and newsletters.boardiesitsolutions.com to help diagnose the issue but we are starting to investigate closing our cloudflare account due to a couple of recent issues we've had and problems with support related to them. We don't want to do this, as I've said at the beginning they do offer a good service but from our point of view, if you are offering a service and offering support, support is a very important aspect, maybe even more important than the service itself as if you get stuck, how its dealt with and how quickly resolved is a huge importance. For example, Boardies IT Solutions, is just myself, Chris Board, we offer services, and APIs for our android app and in the future we plan to launch some other services later on this year. Although its just myself, we do offer support for what we provide to our users, and we might not be able to response immediately or help immediately, but we make it clear and we will always keep our users informed and tell them the progress of their support query, and we will, where possible resolve an issue as quickly as possible.

I don't want to go through everything we plan on doing, as one it might not be that interesting, but also we don't want to expose too much about what we're doing any potentially making us vulnerable to any hacks (we obviously do as much as we can to ensure this isn't possible).

First step will be to do some server maintenance to ensure everything is patched and up to date - this is something we do on a regular basis anyway, usually at least once a month. We will notify you of the maintenance as usual via our twitter and our status pages.

We also have a couple of other domains, https://bitsurl.co.uk and http://chrisboard.co.uk. These aren't hugely important if anything goes wrong so we we'll initially be testing these domains by moving the nameservers (which tells the domain who is DNS provider) and move the DNS to our VPS hosting provider.

As stated earlier, we are not at the moment planning on removing cloudflare, we are just conducting some preparation work and some testing in the event that we do not make any progress with this and feel that this is the best option for us to to get our site up and running at 100% again.

We will continue to provide updates about this issue via twitter and via our status pages.

If you have any questions, then please leave them in the comments or email us at [email protected].

Regards

Chris Board
Boardies IT Solutions