12/9/2016 Provo Data Center Outage

By Fatcow
December 14, 2016

On Friday, December 9th at approximately 4:00 pm (ET) we experienced a network outage at our Provo Data Center located in Utah. This resulted in service disruption for Virtual Private Server (VPS) and Dedicated customers as well as some of our shared customers served out of that facility.

With any significant event that affects our customers, we conduct an extensive examination to understand the root cause and develop a course of action to improve our systems and procedures. To that end, we wanted to provide a synopsis of the situation that occurred and our reassurance that we are working diligently to proactively mitigate and prevent future outages.

Here’s what happened: In an effort to address network instability affecting a small segment of customers, a configuration change was applied which impacted a core router in an unintended manner — specifically, applying a traffic filter in the router towards an aggregate switch resulted in an unexpected network response. This triggered a spanning tree loop and resulted in the loss of gateway response for this critical network segment. The change was immediately reverted, but the instability remained. We worked through every segment of the network to identify the loop while attempting to minimize disruption. Our engineering team and the service vendor wrote and applied the proper Bridge Protocol Data Unit (BPDU) filters and service was methodically restored.   


Network instability lasted approximately 16 hours for some customers while the servers/switches were restored to normal functionality. While we attempted to keep customers apprised as information became available, we realize we need to do more.  

So, here’s what’s happening: We are in the process of modifying the current network architecture to prevent or reduce the impact of any particular network event or device failure by improving Virtual Local Area Network (VLAN) segmentation. Part of the strategy includes creating smaller sub segments of the network so we can isolate problems and manage infrastructure. We are also investing in switch upgrades to allow support of packet filtering throughout our entire infrastructure and most importantly, dedicating significant financial resources to hire additional industry experts to join our team.

Outages disrupt your life and your business. We understand – we take our responsibility to you very seriously.

Please allow me to take this opportunity to thank you for your business and provide my personal assurance that we are dedicated to meeting our commitment to you.

 

Sincerely,
rontrbg

Ron LaSalvia
President & Chief Operating Officer, Endurance International Group

 

compass-high-res

This entry was posted on Wednesday, December 14th, 2016 at 7:58 pm and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.

73 Responses to “12/9/2016 Provo Data Center Outage”

  1. Lawrence Says:

    I have had numerous hosting providers. Fatcow has always been the best, and the fact you took the time to tell us what happened and what the resolution is means a great deal. Thank you.

  2. http://bit.ly/2hBUZfm Says:

    Undeniably believe that that you said. Your favourite reason seemed
    to be on the internet the easiest thing to bear in mind of.
    I say to you, I certainly get annoyed while people think about concerns that they just do not understand about.
    You controlled to hit the nail upon the highest and
    also outlined out the entire thing with no need side-effects , other people can take
    a signal. Will likely be back to get more.
    Thank you

  3. cara merawat rambut Says:

    when it rained try you absorb and feel that I am there beside you, calculate how much rainwater that fell from the sky, as it is also a love and compassion for you, try denagarkan and impregnated in each of rainwater that, you definitely will never believe that every point of the rain water there is engraved the name of both of us, and every point that also take a look at the rain splash terpercikkan how much every point was, and how far spark that fell, just as it was also my love for you

  4. home base healthcare Says:

    Hello, its nice paragraph on the topic of media print,
    we all be aware of media is a fantastic source of information.

  5. سئو سایت Says:

    keep it up! your post is very nice. yeah your job is well done

  6. 2008 dodge avenger ecm location Says:

    Hello Dear, are you in fact visiting this web page on a regular basis,
    if so then you will absolutely take good experience.

  7. pest control products Says:

    Great article.

  8. tank trouble Says:

    I know how it’s an awesome game so I real want to Play it haha

  9. bejeweled Says:

    Sure, it’s a good thing to enjoy your work, but there is such a thing as enjoying it too much

  10. dodge stratus engine computer Says:

    There is certainly a great deal to know about this issue.
    I love all of the points you have made.

  11. Dani Says:

    First off I want to say excellent blog! I had a quick question in which I’d like to ask
    if you don’t mind. I was interested to find out how you
    center yourself and clear your mind before writing. I’ve had a hard time clearing my
    mind in getting my thoughts out there. I truly do enjoy writing but it just seems like the first 10 to 15
    minutes are generally lost just trying to figure out how
    to begin. Any recommendations or tips? Thanks!

  12. madden 17 coins Says:

    Thank you for the gopd writeup. It in fact was a amusement account it.
    Look advanced to far added agreeable from you! By the way, how
    culd we communicate?

  13. Islamic Vashikaran Totke In Quran Says:

    I read your Post and impressed your written skills you have such Great through.

  14. ecm for dodge ram 1500 Says:

    Good day! I could have sworn I’ve been to this website before but after
    checking through some of the post I realized it’s new to me.

    Anyhow, I’m definitely delighted I found it and I’ll be book-marking and
    checking back frequently!

  15. Vegas Escorts Girls Says:

    Glad to know you can learn from that experience. Thanks for posting.

  16. Good to Know Says:

    These things are understandable and it’s good to see you have taken all the necessary steps to correct the problem. It’s nice to know you’re a reliable provider overall and that the company is on top of these things. Keep up the good work.

  17. jesse rogers Says:

    Great post. Bookmarked your site

  18. Cheap Dissertation Writing Service Says:

    It is amazing with any important occasions that have an effect and good work. i am visiting your site since 2014 and i got lots of tips from your blogs.

  19. click here Says:

    good If all fatcow users have the android device because they launched the app called Mobdro to their users for upcoming video streams.

  20. Cinema Box Says:

    great idea with lots of free channels on cinemabox app. Windows PC and Kodi are also supportable.

  21. dodge caravan computer problems Says:

    I savor, result in I discovered just what I was having a
    look for. You have ended my four day long hunt! God Bless
    you man. Have a nice day. Bye

  22. suplemen fitnes Says:

    thank you very much for the information …

  23. super smash flash 2 Says:

    This is such a great resource that you are providing and you give it away for free. I love seeing that understand the value of providing a quality resource for free. :) :)

  24. bloons tower defense 5 Says:

    Wow. cool post. I’d like to write like this too – taking time and real hard work to make a great article… but I put things off too much and never seem to get started. Thanks though.

  25. slither io Says:

    if you like click to play this games slither io

  26. wings io Says:

    click to play this games wings io for free

  27. Adsense CPM Says:

    keep it up! your post is very nice. yeah your job is well done

  28. online dissertation help uk Says:

    Exceptional post anyway I was thinking about whether you could compose a little more on this point? I’d be exceptionally appreciative in the event that you could expand a tad bit facilitate.

  29. Maybelle Villarreal Says:

    tarjeta plastico

  30. dodge cummins ecm ground Says:

    Very nice blog post. I absolutely love this site.
    Keep it up!

  31. Black Magic in Islam and Its Cure Says:

    Good post. It is really help to us. Its give us lots of interest and pleasure. Its opportunity are so fantastic and working style so speedy. Its really a good article. It gives me lots of pleasure and interest!

  32. hospital management doha Says:

    Your place is valueble for me. Thanks!…

  33. тристайни Says:

    This really answered my problem, thank you!

  34. самолетни билети до кипър Says:

    You made some decent points there. I looked on the internet for the issue and found most individuals will go along with with your website.

  35. Max Yetsko Says:

    foder conas peludas

  36. недвижими имоти стара загора къщи Says:

    I am often to blogging and i really appreciate your content. The article has really peaks my interest. I am going to bookmark your site and keep checking for new information.

  37. Erwin Soqui Says:

    how much is coldstone

  38. Sheri Snuggs Says:

    Great information. I found fatcow.com by chance (stumbleupon).I have book-marked it for later!

  39. недвижими имоти кърджали кара Says:

    I discovered your blog site on google and check a few of your early posts. Continue to keep up the very good operate. I just additional up your RSS feed to my MSN News Reader. Seeking forward to reading more from you later on!…

  40. недвижими имоти българия www imot bg Says:

    The next time I read a blog, I hope that it doesnt disappoint me as much as this one. I mean, I know it was my choice to read, but I actually thought youd have something interesting to say. All I hear is a bunch of whining about something that you could fix if you werent too busy looking for attention.

  41. backlinks cheap Says:

    Is using a copyright material as a reference to write articles illegal?

  42. Antiquariat Jaipur Says:

    Thanks a lot of for share the informative and useful blog. keep it up…

  43. help with assignment Says:

    I put stock in the effortlessness and benevolence of God so yes I trust in the force of the deathbed admission. What individuals need to completely comprehend is that not everybody has the chance to have a deathbed admission. Passing can be instant to the point that there is no opportunity to admit ones sins and ask absolution.

  44. ym-kan.com Says:

    Hi there! I understand this is somewhat off-topic however I had to ask.
    Does running a well-established website like yours take a large amount of work?
    I am completely new to blogging but I do write in my journal daily.

    I’d like to start a blog so I can share my own experience and views online.
    Please let me know if you have any kind of ideas or tips for new aspiring bloggers.
    Thankyou!

  45. corrupt financial advisors Says:

    Ahaa, its nice dialogue regarding this article here at this web site,
    I have read all that, so now me also commenting here.

  46. Quality Assurance Qatar Says:

    Youre so cool! I dont suppose Ive read anything like this before. So nice to find somebody with some original thoughts on this subject. realy thank you for starting this up. this website is something that is needed on the web, someone with a little originality. useful job for bringing something new to the internet!

  47. lack of motivation Says:

    Hello there! I know this is kinda off topic but I was wondering if you
    knew where I could find a captcha plugin for my comment form?
    I’m using the same blog platform as yours and I’m having difficulty finding one?
    Thanks a lot!

  48. Notepam Says:

    Great post!

  49. B.tech in Electronics & Communication Says:

    I just delighted to be here and to read this perfect post ever.Thanks for share this with us.

  50. Tory Says:

    Hiya very nice web site!! Guy .. Beautiful ..
    Amazing .. I’ll bookmark your blog and take the feeds additionally?
    I’m satisfied to find a lot of useful info right here in the post,
    we need develop more techniques on this regard, thanks for sharing.

    . . . . .

  51. stonedgirls.eu Says:

    Hurrah! At last I got a website from where I be capable of in fact take useful facts concerning my study and knowledge.

  52. CSSD Management Qatar Says:

    Oh my goodness! an amazing article dude. Thank you However I am experiencing issue with ur rss . Don’t know why Unable to subscribe to it. Is there anyone getting identical rss problem? Anyone who knows kindly respond. Thnkx

  53. BROCKSTAR BY BRIAN NELSON WILLIS Says:

    Look this ripoffreport from ripoffreport.com Don Tigert Don Tigert Don Tigert Don Tigert Don Tigert Don Tigert sala rodd sala rodd sala rodd sala rodd sala rodd sala roddBRIAN WILLIS Brockstar,BROCKSTAR BY BRIAN NELSON WILLIS,Richard Ellenbogen – Beverly Hills Body,LuxSauna Alan Symons

  54. BRIAN WILLIS Brockstar Says:

    sala rodd sala rodd park ward park ward park ward park ward park ward park ward park ward Look this ripoffreport from ripoffreport.com Don Tigert Don Tigert Don Tigert Don Tigert Don Tigert Don Tigert sala rodd sala rodd sala rodd sala rodd sala rodd sala rodd sala rodd sala rodd park ward park ward park ward park ward park ward park ward park ward Look this ripoffreport from ripoffreport.com Don Tigert Don Tigert Don Tigert Don Tigert Don Tigert Don Tigert sala rodd sala rodd sala rodd sala rodd sala rodd sala rodd sala rodd sala rodd park ward park ward park ward park ward park ward park ward park ward nationwide home comfort AGS Capital

  55. Antiquariat Jaipur Says:

    Great post || Thanks a lot of for share the informative and useful blog. keep it up…

  56. lucky patcher Says:

    keep up the good work!!!

  57. Red Pepper Benefits Says:

    Capsaicin is what produces the heat in chili peppers. When applied topically as a cream, gel or patch, the capsaicin provides relief by depleting substance P, which is a neurotransmitter that sends pain-related messages to the brain. Capsaicin may reduce pain from osteoarthritis, rheumatoid arthritis and fibromyalgia. A 2010 German study showed that joint pain decreased by about 50 percent in patients who used capsaicin cream, which can be obtained from serrano peppers. Initially, it may cause a little burning or stinging as substance P is released and eventually depleted, but with regular use, the substance P does not build up again and the burning sensation should not occur.

  58. Sarah Clarke Says:

    Dating when you are over 30 can seem tough. We all had plans to be married by 26 right? Well, thanks to 30 Plus Dating, 30 is the new 20! So make this new years start with a bang

  59. dynamic health apple vinegar complete with apple pectin Says:

    When I originally commented I clicked the -Notify me when new comments are added- checkbox and now each time a comment is added I get four emails with the same comment. Is there any way you can remove me from that service? Thanks!

  60. Best SEO Company in Chennai Says:

    very impressive post for making good quality link without harm own website…

  61. Mark Says:

    It is appropriate time to make some plans for the longer term and it is time to be happy.
    I have read this put up and if I may I want to suggest you some attention-grabbing things or tips.

    Perhaps you could write next articles relating to this article.
    I want to read more things approximately it!

  62. www.floridalottery.site Says:

    What a material of un-ambiguity and preserveness of precious know-how regarding
    unpredicted feelings.

  63. Ofelia Says:

    Heya i am for the first time here. I came across this board and
    I find It really useful & it helped me out much. I hope to give something back and help others
    like you aided me.

  64. Viral Rang Says:

    Thanks a lot of for share the informative and useful blog. keep it up…

  65. http://4sigmas.com.br/ Says:

    Hi there! I know this is kinda off topic but I
    was wondering which blog platform are you using for this site?
    I’m getting sick and tired of WordPress because I’ve had problems with hackers
    and I’m looking at options for another platform.

    I would be awesome if you could point me in the direction of a good platform.

  66. Dissertation Writing Service UK Says:

    I place stock in the ease and generosity of God so yes I confide in the drive of the deathbed affirmation. What people need to totally grasp is that not everyone has the opportunity to have a deathbed affirmation. Passing can be moment to the point that there is no chance to concede ones sins and ask vindication.

  67. How to get return lost love back spells Says:

    Your blog is really an inspiration to many. I’ll be looking forward for more of your posts. Keep it up!

  68. thuglife video maker for pc Says:

    I strongly appreciate your post good content. By the way, I love your style writing.

  69. lucky patcher app Says:

    lucky patcher analyses all the apps that are being present in your device and take actions accordingly for instance it can either modify the permissions or remove the ads,backing up the data and extracting apk files for those apps that are paid afterwards that can be installed.!!

  70. Jio TV Says:

    thanks for your sharing, it’s actually what i need now.

  71. Продажба Тристаен апартамент в София Студентски град Says:

    Your place is valueble for me. Thanks!…

  72. Adult Wedsites Design Says:

    I have heard many good things about the Endurance International Group. Thank you for posting this.

  73. Adult Websites Design Says:

    Thank you for sharing.

Leave a Reply