Intermittent flipbook load failures
Incident Report for iPaper

What happened:

Thursday afternoon around 16:25 CET, the AppServers request queues started filling up across all servers, and an alert was dispatched to our operations team.

Why it happened:

We had rolled out new database changes earlier in the week, these along with other factors like automated database backups and high traffic resulted in slower queries for enrichments, when we reached peak traffic Thursday, the queries were long enough that the request queues started to get full very quickly.

How we fixed it:

Our operations team quickly reacted to the error codes we were seeing and scaled up to double capacity to mitigate the immediate issue. Shortly after the new servers came online we were seeing request queues starting to drop, and issues were resolved for customers.

We then identified the database query in question, and it was resolved to a point where we see a significant lower load than previous.

What we are doing to prevent it from happening again:

We are bringing in additional monitoring, and are looking closely for similar degradations.

Posted Feb 04, 2022 - 17:56 CET

We are resolving this, as we are seeing no other issues.
Posted Jan 27, 2022 - 18:44 CET
We experienced intermittent issues with our app servers, we have doubled our capacity and seeing no more errors.
Posted Jan 27, 2022 - 17:07 CET
We are currently seeing intermittent load issues of flipbooks

Our team is investigating and hope to have full functionality restored as soon as possible.

We will keep you updated as we progress.
Posted Jan 27, 2022 - 16:36 CET
This incident affected: Flipbooks.