Emergency Server Maintenance (Resolved)
  • Priority - Critical
  • Affecting Server - WPHS Cloud
  • Services Affected
    All VPS nodes.

    Impact
    We have been tracking the CPU hardware vulnerability (code names Meltdown and Spectre, CVE designations CVE-2017-5753, CVE-2017-5754 and CVE-2017-5715) very closely and have been informed that kernel patches have been released. Due to the critical nature of these security issues, we have scheduled an emergency maintenance window.

    Details
    The emergency maintenance window has been scheduled to initiate at 12am PST on Saturday 6th of January 2018. Due to the large number of servers, we anticipate that the maintenance period, baring no complications will end at 8am PST. It is anticipated that each server will experience a 15 minute to 30 minute outage while it is rebooted for the kernel upgrade to apply. This alert will be updated with any complications during the maintenance period and with updates as each server is taken offline.

    Support
    Our support team will be available for assistance 24/7. For questions, concerns, or to report a problem, please contact our Support Team:

    To open a trouble ticket: [email protected]

    ==============
    Update 1 - 2:49AM PST 5th January 2018: Alert opened.
    Update 2 - 11:38PM PST 5th January 2018: The maintenance will start as planned in around 20 minutes from now.
    Update 3 - 12:00AM PST 6th January 2018: The following servers have been taken offline now for the kernel upgrade: vpsserver, vpsserver01, vpsserver02, vpsserver03, vpsserver04, vpsserver05, vpsserver06, vpsserver07, vpsserver08, vpsserver09, vpsserver10
    Update 4 - 12:15AM PST 6th January 2018: The following servers have been taken offline now for the kernel upgrade: vpsserver11, vpsserver12, vpsserver13, vpsserver14, vpsserver15
    Update 5 - 12:27AM PST 6th January 2018: The following servers have been taken offline now for the kernel upgrade: vpsserver16, vpsserver17, vpsserver18, vpsserver19, vpsserver20
    Update 6 - 12:38AM PST 6th January 2018: The following servers have been taken offline now for the kernel upgrade: vpsserver21, vpsserver22, vpsserver24, vpsserver25
    Update 7 - 12:45AM PST 6th January 2018: The following servers have been taken offline now for the kernel upgrade: vpsserver28, vpsserver29, vpsserver30
    Update 8 - 12:55AM PST 6th January 2018: The following servers have been taken offline now for the kernel upgrade: vpsserver31, vpsserver32, vpsserver33, vpsserver35
    Update 9 - 1:02AM PST 6th January 2018: The following servers have been taken offline now for the kernel upgrade: vpsserver36, vpsserver38, vpsserver39, vpsserver40
    Update 10 - 1:10AM PST 6th January 2018: The following servers have been taken offline now for the kernel upgrade: vpsserver41, vpsserver42, vpsserver44, vpsserver45
    Update 11 - 1:13AM PST 6th January 2018: All servers have been rebooted except for 6 servers that we are waiting for their backups to complete. We are aware that vpsserver06 has not come back online yet, it is taking a longer time than expected to complete the reboot process.
    Update 12 - 1:42AM PST 6th January 2018: vpsserver06 has been confirmed to be back online.
    Update 13 - 1:52AM PST 6th January 2018: We are troubleshooting an issue with vpsserver38 where it is not completely online yet, this issue doesn't affect connectivity to the VPS's on that server though.
    Update 14 - 12:11PM PST 6th January 2018: The following servers have been taken offline now for the kernel upgrade: vpsserver23, vpsserver26, vpsserver27, vpsserver37
    Update 15 - 12:51PM PST 6th January 2018: vpsserver23, vpsserver26, vpsserver27, vpsserver37 are back online. Please note that we continue to deal with an issue affecting vpsserver38, managing VPS's on that node is not possible at the moment however the VPS's themselves are online. vpsserver34 & vpsserver43 are the last two servers we are waiting for backups to complete before we proceed with applying the kernel update to those two servers.
    Update 16 - 5:15PM PST 7th January 2018: We have taken the last server offline, vpsserver34, for the kernel upgrade as it has completed the backup process now.
    Update 17 - 6:34PM PST 7th January 2018: We have confirmed that the issue with vpsserver38 has been resolved aswell. A this point the kernel updates on all VPS Nodes has completed, this resolves the imminent threat to our network. Further updates will be sent out via email over the course of the next 24 hour periods with further steps.

  • Date - 06/01/2018 10:00 - 01/08/2018 04:37
  • Last Updated - 01/08/2018 04:37
Power Advisory Notification (Resolved)
  • Priority - Low
  • Affecting Server - WPHS Cloud
  • Services Affected
    All Dedicated Servers, VPS's and Shared Hosting in DC-1 North Carolina.

    Impact
    At the moment there is no impact - this is an advisory notification.

    Details
    At approximately 7:00PM EST we have been alerted that our datacenter in North Carolina had lost power and was running on generator power. The initial ETA for resolution of this was 8:00PM EST however as of now (9:30PM EST) there has not been any resolution and power may be lost over the night. This alert will be updated with the latest progress.

    Support
    Our support team will be available for assistance 24/7. For questions, concerns, or to report a problem, please contact our Support Team:

    To open a trouble ticket: [email protected]

    ==============
    Update 1 - 9:30PM EST: We are opening this alert as an advisory notification at the moment, we are taking all measures necessary to ensure that our services continue running without interruption. We will be updating this alert with any further updates as necessary.

    Update 2 - 8:50AM EST: We are closing this alert as street power has been restored approximately 5 hours ago with no further anomalies detected, no downtime occurred and our infrastructure remained stable throughout the event.

  • Date - 10/17/2017 04:46
  • Last Updated - 10/17/2017 15:52
Network Outage (Resolved)
  • Priority - Critical
  • Affecting Server - WPHS Cloud
  • Services Affected
    All VPS's in our Lenoir Datacenter.

    Impact
    Customers with services in this datacenter will experience timeouts while we investigate this issue.

    Details
    At approximately 12:24AM PST our monitoring systems detected a full network outage affecting all services hosted in Lenoir. This alert will be updated with the latest progress.

    Support
    Our support team will be available for assistance 24/7. For questions, concerns, or to report a problem, please contact our Support Team:

    To open a trouble ticket: [email protected]

    ==============
    Update 1 - 12:38AM PST: We are still investigating this outage and hope to have further updates shortly.
    Update 2 - 1:13AM PST: We are continuing our investigation at the moment on this issue.
    Update 3 - 2:13AM PST: Our datacenter technicians have identified an issue with our primary distribution switches, we are working on bringing all our services online now and servers should start becoming responsive shortly.
    Update 4 - 2:20AM PST: The servers are now coming back online, we will update our network status page once all servers have confirmed to be back online.
    Update 4 - 2:36AM PST: Servers are continuing to come back online, we currently have 178 servers offline that we are working to restore connectivity.
    Update 5 - 2:40AM PST: Servers are continuing to come back online, we currently have 55 servers offline that we are working to restore connectivity.
    Update 6 - 2:53AM PST: We continue to work on restoring connectivity to the remaining 55 servers.
    Update 7 - 3:04AM PST: All servers are confirmed to be back online.

  • Date - 06/28/2017 10:24
  • Last Updated - 06/28/2017 15:43
Emergency Network Maintainance (Resolved)
  • Priority - Critical
  • Affecting Server - WPHS Cloud
  • Services Affected
    All VPS's residing on vpsserver21, vpsserver22.

    Impact
    Customers on the above nodes will experience up to 30 minutes of downtime while we perform an emergency replacement of a failing blade on one of our switches.

    Details
    The above VPS nodes, will be offline starting at 10am PST. Servers will be taken offline for up to 30 minutes while the failing blade is replaced. We anticipate the downtime to be less than 30 minutes however we're scheduling downtime of up to 30 minutes barring further complications. This alert will be updated with the latest progress.

    Support
    Our support team will be available for assistance 24/7. For questions, concerns, or to report a problem, please contact our Support Team:

    To open a trouble ticket: [email protected]

    =================

    Update 10:09am PST: The network maintenance has started.
    Update 10:23am PST: The faulty switch has been replaced, the outage was just 5 minutes at the end.

  • Date - 04/11/2017 20:03
  • Last Updated - 04/11/2017 20:36
Connectivity Issues - Newark - Linode (Resolved)
  • Priority - Critical
  • We're currently aware of connectivity issues affecting Linodes in Newark and are investigating

    Identified - We're working with our upstream provider to isolate and resolve a minor routing issue affecting a subset of Linodes in Newark. 

    Monitoring - At this time we believe that the routing issue affecting a subset of Linodes in Newark has been resolved and we're currently monitoring for any further problems. 
    Feb 10, 09:40 UTC

    Resolved - At this time we believe that the routing issue in Newark has been fully resolved. If we receive further information on the cause of this routing issue from our transit provider, we will update this post with a post-mortem. 
    Feb 10, 15:48 UTC
     

  • Date - 02/10/2016 07:27 - 02/10/2016 15:48
  • Last Updated - 02/10/2016 19:44
Dallas DC Issue (Resolved)
  • Priority - Critical
  • Affecting Server - WPHS Cloud
  • Currently having an issue in the Dallas DC. Working to fix...

    Dallas DC issue has been resolved. 

  • Date - 11/24/2015 03:22 - 11/24/2015 03:25
  • Last Updated - 11/24/2015 03:24