Issues with SSL certificate issuance for branded domains
Incident Report for iPaper
Resolved
This incident has been resolved.
Posted Oct 18, 2021 - 15:38 CEST
Monitoring
More great news!

We are now completely operational again and serving new certificates.
At the time of this update, we have experienced just 2 end-users being unable to load catalogs. These end-users were on Android 6 and 7 (5+ year old software), and this is to be expected with the trade-off we had to make for us to get back up and running for 99.99% of our customers.

For now, we will monitor the situation and continue to work with our providers for a while before closing this incident, where a postmortem will follow shortly after.
Posted Oct 11, 2021 - 14:25 CEST
Update
Great news!

We have been working hard on a solution that would enable the remaining device combinations to access flipbooks via HTTPS.

That solution was deployed a short while ago - and it seems to be working 🥳
Do let us know if you experience any devices that are unable to connect to a flipbook due to an untrusted certificate.

We will keep monitoring this closely and continue working with our providers to ensure we are staying on top of the situation going forward.

We wholeheartedly understand how frustrating this situation has been for you, as many of your customers, sales reps and colleagues were unable to access the catalogs they have come to depend on.

For this very reason, we want to extend a loud and profound - THANK YOU - for your understanding and patience with us during this past week 🙏

We truly have the best customers!
Posted Oct 07, 2021 - 14:26 CEST
Update
Currently, the majority of traffic is coming through as normal (we're seeing a drop of less than half a percent).

We are however still seeing issues on some devices due to the chain of trust pointing to the expired root certificate.
We are working hard on a fix to help mitigate this specific issue, and we should hopefully have a working solution sometime tomorrow morning, where we will start to see fewer devices rejecting our certificates right after.

This is a global issue, and we are still working closely with our providers to coordinate this effort to the benefit of our customers, as well as manually mitigating as much as we can!

We are grateful for your extended patience 🙏
Posted Oct 06, 2021 - 20:11 CEST
Update
What is happening:
We have been experiencing problems with serving new certificates over the last few days due to us being unable to upload to our infrastructure provider where our certificates would be rejected at their end.

Why is this happening:
On September 30th, the global root certificate of our SSL provider expired. This was planned, but nonetheless ended up affecting large parts of the internet. Read more about this here: https://www.zdnet.com/article/fortinet-shopify-others-report-issues-after-root-ca-certificate-from-lets-encrypt-expires.

With the expiration of Let’s Encrypt’s global root certificate, our infrastructure provider would no longer accept the new certificates we were generating, resulting in our inability to serve new certificates to our customers.

Who/what has been affected:
Customers who were in need of new certificates or had expiring certificates on or after October 1st for Flipbooks have been impacted over the last few days. We have tried our best to accommodate as much as we could manually during this time.

How are we mitigating this issue:
We have mitigated this with a workaround in how we generate certificates to avoid failing validation with our provider while we are waiting for them to respond back and to have a solution on their end. We will be keeping a close eye on this issue going forward.

We are lowering the severity to "Partial outage" as we are now serving new certificates successfully, but we are still awaiting response from our providers before we close this.

While this mitigates by far most of the traffic we receive, we do see reports from iOS and Mac users that are unable to load certain Flipbooks. This is in most cases caused by devices that have not been updated, though in some cases this can happen even on fully updated devices. We are working closely with our hosting provider to take care of this issue.

Possible workaround for macOS:
Restart your browser to force a recheck of the certificate in your browser.
This can in some cases be enough to load the page correctly again.

Thank you for your patience and understanding during this time. 🙏
Posted Oct 05, 2021 - 20:55 CEST
Update
We are doing everything we can to resolve this challenge as swiftly as possible.

Who is impacted:
The challenge affects new certificate creations, and not existing certificates.
Only customers with certificates that are about to expire/have expired will experience this challenge.

If you have any immediate concerns or issues with your Branded Domains, please reach out to Customer Care and we will help you any way we can.

We appreciate your continued patience 🙏
Posted Oct 04, 2021 - 16:12 CEST
Identified
Issue has been identified and a solution is being worked on. Until resolved no new certificates will be issued, but current certificates will remain active.
Posted Oct 01, 2021 - 13:19 CEST
Investigating
We are currently investigating problems with issuing SSL certificates for branded domains.
Posted Oct 01, 2021 - 11:30 CEST
This incident affected: Services (SSL Processing Service).