Issues with SSL certificate issuance for branded domains
Incident Report for iPaper

What happened:

On Friday, October 1st, 2021, we saw our SSL service failing to upload certificates to our hosting provider. We then stopped the service to investigate the cause of this.
To us, the certificates looked valid locally and had just been issued from Let’s Encrypt directly with no issues or warnings.

We then manually tried to upload the certificate to AWS and were met with a “certificate expired” error. We then reached out to AWS to confirm, and they confirmed the chain had an invalid certificate.

Why it happened:

On September 30th, 2021, Let’s Encrypt’s root certificate (DST ROOT CA X3) expired, and while Let’s Encrypt had long ago added the new root certificate (ISRG Root X1) and tried to mitigate this issue by adding a cross-signed certificate that would point to both the new and old root certificates, it would not work with some versions of OpenSSL (<= v1.02) and it seems it was failing validation with AWS for simply containing an expired certificate, so while this was not affecting our current certificates, we had no way of uploading new certificates at this point.

How we fixed it:

After a lot of research, we had no option but to remove the expired certificate and the cross-signed intermediate certificates. This would then allow us to resume creating new certificates for our customers. This decision was not taken lightly, as while it means we can issue new certificates that work on 99.99% of modern devices, it would exclude older devices, like Android version 7.1 and older.

What we are doing to prevent it from happening again:

We do not take the responsibility our customers put on us lightly, and therefore we will continue working closely with our providers to see if we can resolve this for more devices, and to prevent this from happening in the future.

We appreciate your patience and continued trust in us 🙏

Posted Nov 09, 2021 - 14:50 CET

After monitoring this for some time and being in close contact with affected customers, we feel confident that we can now close this issue.
Should you experience any problems or have any concerns please do not hesitate to reach out.

A post mortem will be made available in the coming days.
Posted Nov 05, 2021 - 14:51 CET
We have deployed a fix to resolve the challenge with expired SSL certificates.
Due to the nature of this fix, we are unable to present content via HTTPS on a range of older operating systems and applications. This includes Android devices running 7.1 or lower.
We continue to monitor the situation while working with our infrastructure providers to find a more inclusive solution.
Posted Oct 18, 2021 - 15:37 CEST
This incident affected: Services (SSL Processing Service).