A Series of Tubez Networks Services https://status.aseriesoftubez.com/ 2026-04-05T17:57:57+00:00 A Series of Tubez Networks Services <![CDATA[TOR1 - Packet Loss (Resolved)]]> https://status.aseriesoftubez.com/incident/80/ As of 09:12 EDT this issue have been resolved.

]]> 2023-06-15T14:07:31+00:00 A Series of Tubez Networks Services <![CDATA[TOR1 - Packet Loss]]> https://status.aseriesoftubez.com/incident/79/ At around 21:20 EDT we have been noticing packetloss at our Tor1 location.

We will post updates here while we have more information.

]]> 2023-06-15T02:24:20+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 - Core Network Upgrade (Completed)]]> https://status.aseriesoftubez.com/incident/78/ The scheduled network upgrade has been completed.

]]> 2023-03-25T05:27:37+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 - Core Network Upgrade (In progress)]]> https://status.aseriesoftubez.com/incident/77/ Scheduled network upgrade is currently in progress. We will provide updates as necessary.

]]> 2023-03-25T04:00:00+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 - Core Network Upgrade]]> https://status.aseriesoftubez.com/incident/76/ We will be performing a scheduled network upgrade in our Tor1 facility from 12AM until 2AM EDT on Saturday, March 25, 2023.

During this time some services will be unavailable for a brief period.

]]> 2023-03-24T04:31:00+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 - Power Outage (Resolved)]]> https://status.aseriesoftubez.com/incident/75/ All systems are now back online in our Tor1 facility.

]]> 2022-06-17T07:25:00+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 - Power Outage (Power Restored)]]> https://status.aseriesoftubez.com/incident/74/ Power have been restored at the Tor1 facility.

Restoring all services.

]]> 2022-06-17T06:23:00+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 - Power Outage (Major Outage)]]> https://status.aseriesoftubez.com/incident/73/ All systems have been safely shutdown due to power outage. Total network outage.

]]> 2022-06-17T05:55:00+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 - Power Outage]]> https://status.aseriesoftubez.com/incident/72/ Battery backup used up. All systems being shutdown.

]]> 2022-06-17T05:50:00+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 - Power Outage]]> https://status.aseriesoftubez.com/incident/71/ We are currently experiencing a power outage at our Tor1 facility. We are currently running on battery backup.

]]> 2022-06-17T05:14:00+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 Storage Server - Outage (Resolved)]]> https://status.aseriesoftubez.com/incident/70/ Tor1 Storage Server has been stable for the past 12 hours.

This incident has been resolved.

]]> 2021-11-23T03:16:02+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 Storage Server - Outage (Monitoring)]]> https://status.aseriesoftubez.com/incident/69/ We have identified the cause of the Tor1 Storage Server. The boot disk hanged for unknown reason and we are currently monitoring the saturation.

All services are now restored and running.

]]> 2021-11-22T15:02:40+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 Storage Server - Outage]]> https://status.aseriesoftubez.com/incident/68/ There is currently an outage on the Tor1 Storage server which is also affecting our backups and mail server service.

We are currently investigating the cause of the issue.

]]> 2021-11-22T14:49:05+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 Outage - Resolved]]> https://status.aseriesoftubez.com/incident/67/ Utility power returned and all servers booted up correctly.

]]> 2021-08-23T22:58:00+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 Outage]]> https://status.aseriesoftubez.com/incident/66/ We are currently experienceing an powet outage at out Tor1 datacenter.

All UPS have been depreciated.

]]> 2021-08-23T21:32:09+00:00 A Series of Tubez Networks Services <![CDATA[Scheduled Network Maintenance - TOR1 (Completed)]]> https://status.aseriesoftubez.com/incident/65/ The scheduled maintenance has been completed.

]]> 2021-07-25T05:02:33+00:00 A Series of Tubez Networks Services <![CDATA[Scheduled Network Maintenance - TOR1 (In progress)]]> https://status.aseriesoftubez.com/incident/64/ Scheduled maintenance is currently in progress.

We will provide updates as necessary.

]]> 2021-07-25T04:30:07+00:00 A Series of Tubez Networks Services <![CDATA[Scheduled Network Maintenance - TOR1]]> https://status.aseriesoftubez.com/incident/63/ Upcoming scheduled maintenance notice

We will be preforming firmware upgrades on our Core Routers in the TOR1 data centre on Sunday, July 25th, at 00:30 EDT until Sunday, July 25th, at 1:30 EDT.

During this time all TOR1 services will be unavailable, estimated downtime will be less than 30 minutes.

Components affected:

  • DNS Panel
  • Email
  • NTP
  • Virtualization Cluster at TOR1 Location
  • Web01.tor
]]> 2021-07-21T04:43:00+00:00 A Series of Tubez Networks Services <![CDATA[Virtualization Environment - KVM01-tor Software Upgrade (Completed)]]> https://status.aseriesoftubez.com/incident/62/ Scheduled maintenance completed.

]]> 2021-04-11T04:47:01+00:00 A Series of Tubez Networks Services <![CDATA[Virtualization Environment - KVM01-tor Software Upgrade (In progress)]]> https://status.aseriesoftubez.com/incident/61/ Scheduled maintenance is currently in progress.

We will provide updates as needed.

]]> 2021-04-11T04:11:02+00:00 A Series of Tubez Networks Services <![CDATA[Virtualization Environment - KVM01-tor Software Upgrade]]> https://status.aseriesoftubez.com/incident/60/ We will be upgrading the hypervisor software on our Virtualization Environment.

During this time the DNS manager, MailNinja email server, and all virtual machines on KVM01-tor will be safely shutdown and rebooted after the maintenance.

The scheduled a maintenance window is Sunday April 11, at 00:20 EDT to 01:20 EDT.

]]> 2021-04-04T02:33:29+00:00 A Series of Tubez Networks Services <![CDATA[MailNinja.aseriesoftubez.com - Maintenance (Competed)]]> https://status.aseriesoftubez.com/incident/59/ MailNinja.aseriesoftubez.com Docker volume storage have been moved to block storage and offsite backup of the block storage will sync hourly.

Email service is now operating normally.

]]> 2021-03-14T06:17:30+00:00 A Series of Tubez Networks Services <![CDATA[MailNinja.aseriesoftubez.com - Maintenance]]> https://status.aseriesoftubez.com/incident/58/ We will be starting the email server maintenance.

MailNinja.aseriesoftubez.com will be unavailable during this time. All incoming emails will be queued on mx2.aseriesoftubez.com and delivered after the maintenance.

Maintenance will end no later than March 14th, at 04:00 EDT.

]]> 2021-03-14T05:00:00+00:00 A Series of Tubez Networks Services <![CDATA[MailNinja.aseriesoftubez.com - Maintenance Reminder]]> https://status.aseriesoftubez.com/incident/57/ This is a reminder that we have scheduled a maintenance window this Sunday March 14th, at 00:00 EST to 04:00 EDT.

We will move the Docker storage from ZFS running in the VM itself to block storage formatted with ext4.

During the maintenance MailNinja.aseriesoftubez.com will be unreachable and incoming email will be queued on mx2.aseriesoftubez.com and delivered after the maintenance.

The total downtime will be less than 3 hours.

]]> 2021-03-11T23:03:00+00:00 A Series of Tubez Networks Services <![CDATA[MailNinja.aseriesoftubez.com - Backup Snapshot Issue (Resolved)]]> https://status.aseriesoftubez.com/incident/56/ We have resolved the issue and restored all the emails missing during that time.

We will do a final filesystem maintenance on March 14th, at 00:00 EDT.

]]> 2021-03-08T06:18:35+00:00 A Series of Tubez Networks Services <![CDATA[MailNinja.aseriesoftubez.com - Backup Snapshot Issue]]> https://status.aseriesoftubez.com/incident/55/ While snapshots were being purged service was running fine but after snapshots were purged we ran a ZFS scrub on the pool which caused the kernel to hang at boot at 12:30PM EST similar to this issue.

We have restored a backup from this morning at 5:41:13AM EDT.

Once the old VM is back up and running we will do a diff of the mailboxes directories and restore those files on the new VM.

Later this week we will schedule a window to do some filesystem maintenance to prevent this from happening again.

]]> 2021-03-07T19:02:00+00:00 A Series of Tubez Networks Services <![CDATA[MailNinja.aseriesoftubez.com - Backup Snapshot Issue (Identified)]]> https://status.aseriesoftubez.com/incident/54/ We have identified the cause of the ZFS replication issue.

There are large amounts of snapshots on the system causing degraded performance which caused a back log of jobs that are queued up and not competing.

We are currently purging the snapshots and limiting the amount kept to avoid this from happening in the future.

]]> 2021-03-07T08:52:00+00:00 A Series of Tubez Networks Services <![CDATA[MailNinja.aseriesoftubez.com - Backup Snapshot Issue]]> https://status.aseriesoftubez.com/incident/53/ We are currently investigating an issue causing ZFS snapshots in our mail server not being sent to our backup server.

There will be some connection timeout / errors as we fix this issue.

We will provide update as they are available.

]]> 2021-03-07T05:48:00+00:00 A Series of Tubez Networks Services <![CDATA[Server Migration - NS2 (Resolved)]]> https://status.aseriesoftubez.com/incident/52/ Migration took longer than expected but NS2 DNS server is now up and running.

]]> 2021-01-05T04:38:00+00:00 A Series of Tubez Networks Services <![CDATA[Server Migration - Email (Resolved)]]> https://status.aseriesoftubez.com/incident/51/ Migration took longer than expected but the backup mail server is now up and running.

]]> 2021-01-05T04:34:36+00:00 A Series of Tubez Networks Services <![CDATA[Server Migration - Email]]> https://status.aseriesoftubez.com/incident/49/ Our backup mail server is being migrated to a new physical server on January 2nd to January 3rd.

During this time the backup mail server will be unavailable.

Our regular monthly Mailcow upgrade will be moved to January, 10th during our regular maintenance window 1:00AM to 3:00AM EST.

]]> 2021-01-02T05:00:00+00:00 A Series of Tubez Networks Services <![CDATA[Server Migration - NS2]]> https://status.aseriesoftubez.com/incident/50/ Our NS2 DNS server is being migrated to a new physical server on January 2nd to January 3rd.

During this time NS2 will be unavailable.

DNS requests to the other DNS servers will be unaffected.

]]> 2021-01-02T05:00:00+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 Network Outage (Resolved)]]> https://status.aseriesoftubez.com/incident/48/ All services are operating normally and issue has been resolved.

]]> 2020-07-27T01:47:42+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 Network Outage]]> https://status.aseriesoftubez.com/incident/47/ Our upstream provider have fixed the issue all services should now be back to normal.

We will continue to monitor the network for issues.

]]> 2020-07-26T21:10:00+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 Network Outage]]> https://status.aseriesoftubez.com/incident/46/ We are currently experiencing an major outage. We will update here when updates are available.

]]> 2020-07-26T16:43:00+00:00 A Series of Tubez Networks Services <![CDATA[Scheduled Network Maintenance - TOR1 (Completed)]]> https://status.aseriesoftubez.com/incident/45/ The scheduled maintenance has been completed.

]]> 2020-05-23T02:54:00+00:00 A Series of Tubez Networks Services <![CDATA[Scheduled Network Maintenance - TOR1 (In progress)]]> https://status.aseriesoftubez.com/incident/44/ Scheduled maintenance is currently in progress.

We will provide updates as necessary.

]]> 2020-05-23T02:00:00+00:00 A Series of Tubez Networks Services <![CDATA[Scheduled Network Maintenance - TOR1]]> https://status.aseriesoftubez.com/incident/43/ Upcoming scheduled maintenance notice

We will be preforming firmware upgrades on our Core Routers in the TOR1 data centre on Friday, May 22nd, at 22:00 EDT until Friday, May 22nd, at 23:00 EDT.

During this time all TOR1 services will be unavailable, estimated downtime will be less than 30 minutes.

Components affected:

  • DNS Panel
  • Email
  • NTP
  • Virtualization Cluster at TOR1 Location
  • Web01.tor
]]> 2020-05-07T03:48:00+00:00 A Series of Tubez Networks Services <![CDATA[Virtualization Environment Outage - Resolved]]> https://status.aseriesoftubez.com/incident/39/ The cause of the outage was due to a new monitoring software we implemented on kvm01-tor which caused the system's SMBUS to hang which then caused the host OS to also hang which we needed to do an emergency reboot.

We resolved the issue by unloading the w83795 kernel module to avoid the software and system to query those affected sensors.

We are sorry for the inconveniences this may have caused.

]]> 2020-03-02T19:43:00+00:00 A Series of Tubez Networks Services <![CDATA[Virtualization Environment Outage]]> https://status.aseriesoftubez.com/incident/38/ We have issued an emergency reboot on kvm01-tor and all services should be returning to normal.

We are continuing to monitor this issue.

]]> 2020-03-02T08:20:00+00:00 A Series of Tubez Networks Services <![CDATA[Virtualization Environment Outage]]> https://status.aseriesoftubez.com/incident/33/ We are currently experiencing an outage on our Virtualization Environment node kvm01-tor

It is also affecting our Email and DNS management panel.

We are currently investigating the cause of the issue and restoring service.

]]> 2020-03-02T07:50:00+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 Storage Server - Software Upgrade (Done)]]> https://status.aseriesoftubez.com/incident/32/ Software upgrade is now done and all services returned to normal.

]]> 2020-02-11T01:47:45+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 Storage Server - Software Upgrade]]> https://status.aseriesoftubez.com/incident/31/ We are now currently upgrading Tor1 Storage Server software.

Accessing the storage server and creating or restoring backups on our Virtualization Environment in the Tor1 location is currently unavailable.

]]> 2020-02-11T01:06:14+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 Storage Server - Software Upgrade]]> https://status.aseriesoftubez.com/incident/30/ We will be upgrading Tor1 Storage Server software on February 10th, 2020 from 20:00 to 21:00 EST.

During the above window, the storage server and creating or accessing backups on our Virtualization Environment in the Tor1 location will be unavailable.

]]> 2020-02-10T02:32:00+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 Network Outage (Resolved)]]> https://status.aseriesoftubez.com/incident/28/ This issue have been resolved all services are operating normally.

Timeline of events:

8:05 AM EST - Link to upstream provider went down

8:50 AM EST - Link to upstream provider restored

9:30 AM EST - Upstream provider confirmed there is a widespread issue with their vendor

8:10 PM EST - Update from upstream provider that their vendor have two major CO pieces of equipment failed today and had to be replaced.

]]> 2019-11-10T06:41:00+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 Network Outage]]> https://status.aseriesoftubez.com/incident/27/ There was a problem with our upstream provider and issue have been resolved.

We will continue to monitor the network for issues.

]]> 2019-11-09T14:10:00+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 Network Outage]]> https://status.aseriesoftubez.com/incident/26/ We are currently experiencing a network outage at our Tor1 Datacenter and currently investigating the cause.

]]> 2019-11-09T13:05:00+00:00 A Series of Tubez Networks Services <![CDATA[TCP Selective Acknowledgment (SACK) vulnerability (complete)]]> https://status.aseriesoftubez.com/incident/24/ The maintenance for tonight is now complete.

Due to the nature of this security issue there may be some additional short-notice security updates in the coming weeks.

]]> 2019-06-21T05:30:00+00:00 A Series of Tubez Networks Services <![CDATA[TCP Selective Acknowledgment (SACK) vulnerability]]> https://status.aseriesoftubez.com/incident/23/ In response to the TCP Selective Acknowledgment (SACK) vulnerability security issues recently reported in the Linux Kernel (CVEs: CVE-2019-11477, CVE-2019-11478, and CVE-2019-11479), we will be performing critical security updates to the Virtualization Environment Cluster. We will be applying these updates tonight, on Friday, Jun 21, 2019 at 01:00 EDT.

There will be a brief periods when Web, Email Hosting, and DNS panel become temporarily unavailable for a few minutes.

For customers with unmanaged services to stay safe you will need to take action yourself to ensure you have the latest operating system patches installed. You will then need to reboot your servers.

Due to this being a critical security vulnerability we apologize for the short notice of this maintenance window.

]]> 2019-06-21T01:04:00+00:00 A Series of Tubez Networks Services <![CDATA[Security update for Virtualization Environment Cluster (complete)]]> https://status.aseriesoftubez.com/incident/22/ The maintenance for tonight is now complete. Due to the nature of this security issue there may be some additional short-notice security updates in the coming weeks.

]]> 2019-05-17T03:26:00+00:00 A Series of Tubez Networks Services <![CDATA[Security update for Virtualization Environment Cluster]]> https://status.aseriesoftubez.com/incident/21/ In response to the MDS security issues recently reported in Intel CPUs (CVE-2018-12126, CVE-2018-12127, CVE-2018-12130, CVE-2019-11091), we will be performing critical security updates to the Virtualization Environment Cluster. We will be applying these updates tonight, on Thursday, May 16, 2019 at 23:00 EDT.

There will be a brief periods when Web, Email Hosting, and DNS panel become temporarily unavailable for a few minutes.

For customers with unmanaged services to stay safe you will need to take action yourself to ensure you have the latest operating system patches installed. You will then need to reboot your servers.

Due to this being a critical security we apologize for the short notice of this maintenance window.

]]> 2019-05-16T15:00:00+00:00 A Series of Tubez Networks Services <![CDATA[Emergency Network Maintenance - Tor1]]> https://status.aseriesoftubez.com/incident/20/ Fan have been replaced and all services are now operational.

]]> 2019-04-17T19:24:00+00:00 A Series of Tubez Networks Services <![CDATA[Emergency Network Maintenance - Tor1]]> https://status.aseriesoftubez.com/incident/19/ A fan have failed on our core switch and it is being replaced.

All services will still be functional except for our main website and dedicated servers.

We estimate this will be resolved in an hour.

]]> 2019-04-17T17:34:00+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 Packetloss]]> https://status.aseriesoftubez.com/incident/15/ Issue resolved.

]]> 2019-01-13T03:02:49+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 Packetloss]]> https://status.aseriesoftubez.com/incident/14/ There is currently some packetloss at our Tor1 Datacenter that we are investigating.

]]> 2019-01-13T00:51:00+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 Storage Server Hardware Maintenance]]> https://status.aseriesoftubez.com/incident/13/ Tor1 Storage Server hardware maintenance has been completed.

]]> 2018-12-22T01:47:00+00:00 A Series of Tubez Networks Services <![CDATA[Tor1 Storage Server Hardware Maintenance]]> https://status.aseriesoftubez.com/incident/12/ A Series of Tubez Networks will be doing maintenance on the storage server in the Tor1 datacentre.

During the maintenance the storage server will be unavailable.

We are making the maintenance window from 21/12/2018 20:00 to 21/12/2018 23:00 EST but we are hoping it will not take this long.

]]> 2018-12-20T23:28:21+00:00