Amazon AWS S3 outage

March 1st, 2017
Posted in Monitoring, Industry News

Amazon AWS S3 outage

 

Yesterday Amazon's S3 web-based storage service experienced a major outage, which left a lot of websites and applications not working properly or not working at all. Not all Amazon customers were affected, but a lot of sites relying on storage in the US EAST region experienced timeouts or loss of functionality. Among those sites are Giphy, Medium, Slack, Quora and DownDetector.

Below is the offical timeline of the outage from the Amazon’s AWS service health dashboard. As you can see even the Amazon's own health dashboard was malfunctiong due to this problem

[RESOLVED] Increased Error Rates for Amazon S3
Update at 2:08 PM PST: As of 1:49 PM PST, we are fully recovered for operations for adding new objects in S3, which was our last operation showing a high error rate. The Amazon S3 service is operating normally.
Update at 1:12 PM PST: S3 object retrieval, listing and deletion are fully recovered now. We are still working to recover normal operations for adding new objects to S3.
Update at 12:52 PM PST: We are seeing recovery for S3 object retrievals, listing and deletions. We continue to work on recovery for adding new objects to S3 and expect to start seeing improved error rates within the hour.
Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue.

Some of our clients experienced issues due to this problem, but we were able to quickly identify the issue. Switching to other working platforms prevented huge losses of revenue and decreased their downtime.

Security notification for CVE-2015-0235 (GHOST vulnerability)

January 28th, 2015
Posted in Industry News, WebSitePulse News, Tech

A significant Linux vulnerability that allows remote code execution to Linux server(s) was announced late yesterday, named GHOST: CVE-2015-023. Full details of the vulnerability are available at http://www.openwall.com/lists/oss-security/2015/01/27/9. While the issue has been fixed as early as Mar 21, 2013 it was not marked as a security threat and as a result the patch was not backported to most of the stable and long-term-support distributions like RHEL, Centos, Ubuntu 12.04 etc which left them vulnerable.

Updates for CentOS are already available in the Updates repository so a simple "yum update" will install the required patches to mitigate this vulnerability.

Qualys have provided a simple C program to test if a machine is vulnerable

cat > GHOST.c << EOF
#include <netdb.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>

#define CANARY "in_the_coal_mine"

struct {
  char buffer[1024];
  char canary[sizeof(CANARY)];
} temp = { "buffer", CANARY };

int main(void) {
  struct hostent resbuf;
  struct hostent *result;
  int herrno;
  int retval;

  /*** strlen (name) = size_needed - sizeof (*host_addr) - sizeof (*h_addr_ptrs) - 1; ***/
  size_t len = sizeof(temp.buffer) - 16*sizeof(unsigned char) - 2*sizeof(char *) - 1;
  char name[sizeof(temp.buffer)];
  memset(name, '0', len);
  name[len] = '\0';

  retval = gethostbyname_r(name, &resbuf, temp.buffer, sizeof(temp.buffer), &result, &herrno);

  if (strcmp(temp.canary, CANARY) != 0) {
    puts("vulnerable");
    exit(EXIT_SUCCESS);
  }
  if (retval == ERANGE) {
    puts("not vulnerable");
    exit(EXIT_SUCCESS);
  }
  puts("should not happen");
  exit(EXIT_FAILURE);
}
EOF

$ gcc GHOST.c -o GHOST
$ ./GHOST

We have verified that all WebSitePulse servers have latest updates installed and are not vulnerable.

 

 

 

 

Security Notification for SSLv3 POODLE Vulnerability

October 16th, 2014
Posted in Industry News, WebSitePulse News

As you probably know, a number of news sources, corporations, and the OpenSSL team reported yesterday 14 October 2014 that version 3 of Secure Sockets Layer (SSLv3) is vulnerable at the protocol level. More information about the vulnerability can be found here -  CVE-2014-3566.

To prevent any potential leaks from this vulnerability we have immediately disabled SSLv3 on all our web servers including the API endpoints. Our monitoring agents are not affected by this change and will continue to support SSLv3 for the time being in order to be able to monitor properly servers that do support SSLv3 only. We are urging all customers to disable SSLv3 on hosts interacting with the our services as soon as possible and upgrade to use Transport Layer Service (TLS).

Here are a few samples how to configure your potentially vulnerable services and disable SSLv3.

 

Apache

Change all SSLProtocol directives in your httpd config to

ALL -SSLv2 -SSLv3

and restart the server.

Nginx

Add/edit the the following text to your server directive

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

If you can't  disable SSL 3.0 entirely,  there is TLS_FALLBACK_SCSV  patch that can help avoid the attack, if both the client and the server support it.

 

Facebook suffers global outage

June 19th, 2014
Posted in Monitoring, Industry News, Tech

facebook.com went down for aboutt 15 minutes between 4:00 and 4:15 AM EST. Users trying to connect to the site were seeing an error message "Sorry, something went wrong".  The issue has been confirmed by multiple locations around the world and as far as we can tell all Facebook users were affected.

 

 

The Facebook Platform Status showed a sharp increase in API response times for the time of the outage.

Facebook Platform Status

The last information posted by Facebook is

Sitewide issue resolved

Earlier this morning, we experienced an issue that prevented use of the API for a brief period of time. We resolved the issue quickly, and we are now back to 100%. We're sorry for any inconvenience this may have caused.

We are awaiting an official statement and more details from Facebook and we will update this post as soon as more information is available.