All systems are operational

About This Site

This is the technical status page of combahton GmbH which represents past incidents and planned maintenances.

Past Incidents

Thursday 23rd January 2020

No incidents reported

Wednesday 22nd January 2020

SSD Cloud Storage - Planned Upgrade, scheduled 1 day ago

We will carry out a upgrade of our SSD Cloud Storage, which is related to the storage node ssd-gluster01-04. The upgrade is required to increase the performance of the SSDs attached to this node by adding a faster HBA Controller.

During the last days, we saw a increased amount of IOwait, which will be at least mitigated with this change. No downtime is to be expected, I/O will be provide by remaining storage nodes during the maintenance. You might see a slightly slower I/O for some specific instances within the next 24 hours after the upgrade of a node has been carried out.

Update 12:30: We are starting the upgrade of ssd-gluster04. Storage environment appears to be stable.
Update 15:30: The upgrade of ssd-gluster04 has been carried out successfully, performance increased by 300%. We are now waiting for the volume to be synchronised with the other storage nodes.

Cloud Nodes FFM2 Unexpected restart - kvm01.cloud01 - FFM2

Around 5:20 pm an unexpected restart of a node (kvm01) in our cloud cluster occurred. The cause is still being investigated. All customers were automatically migrated and started on another node, there was no noticeable downtime.
Update 19:21: The issue was related to a firmware bug. We have disabled the faulty feature, node is rebooting and will be tested further.

Tuesday 21st January 2020

No incidents reported

Monday 20th January 2020

No incidents reported

Sunday 19th January 2020

No incidents reported

Saturday 18th January 2020

Core Network Network Disturbance - FFM3

From 04:15 pm till 04:50 pm, we had a series of packetloss on our Uplinks to Core-Backbone and RETN. The issue was caused by a very large ddos attack affecting one ouf our customers. The attack filled both uplinks to nearly 100% usage. We are currently assessing the event and will implement further protection mechanisms to avoid this from happening again in the future.

We are deeply sorry for any issues caused to our customers! - We will keep you updated about further technical improvements.

Update 18:30: We have internally discussed the incident and implemented technical improvements to avoid this from happening again. There are also planned upgrades for our Uplinks to Core-Backbone and RETN, which will be implemented until the end of January. However, this doesnt mean, you have to expect such issue to happen again until the upgrades are implemented.

Update 18:45: We have decided to implement additional monitoring checks, which will be done until the end of the day. The checks will cover our smokeping probes, which will alert once there is a specific amount of packetloss for specific uplinks. In that way, we will ensure additional coverage and extended insight, which would have helped to minimize the impact of this incident.

Friday 17th January 2020

No incidents reported