Summary

On August 18, 2011 our network was target of a distributed denial of service attack from a large number of hosts in Pakistan and India. The attack started around 18:30 UTC while monitoring coped with degraded performance between 19:00 and 20:20 UTC. After intentionally bringing down our portal in order to raise the check frequency to normal levels things went back to normal and messages queued up for delivery were sent out via the remote gateways.

With help of our hosting provider RackSpace, our team was able to mitigate the attack using blacklists and identify the IP’s being targeted, allowing us to bring back the portal pages. As of writing the attack is still ongoing and showing a 3 to 6-fold increase in our usual traffic pattern. We are continuing to take proactive measures in order to react to possible changes in the situation.

What we have learned so far

DDoS attacks are difficult to control in general, but we’ve learned a lot from these events. The biggest issue was that our fail-over location was not able to activate itself as the core services were still running. We will be investigating how we can improve this situation without causing unnecessary duplicate probes or alarms to be sent out.

Secondarily, we learned that our main portal services are located too close to the core monitoring services in our network, and as such one may affect the other. We’re planning to physically separate these services now, so that we do not have to bring down our portal in the future in order to free bandwidth for the monitoring services.

That said, I want to give a huge thanks to the stand-by team (Kalina, Dimi and Stratos) who greatly helped reducing the impact of the attack so far by working as a team on several different tracks in parallel. I also want to thank RackSpace for the support from their knowledgeable and fanatical support team.

Timeline

• 18:34 UTC Response team was first alerted about reduced connectivity to our systems (30-60% packet loss).
• 18:46 UTC Contacted RackSpace support.
• 18:59 UTC RackSpace identified the issue as a DDoS attack from the Pakistan/India region, they added an initial set of /16′s to our blacklist in an attempt to mitigate the attack.
• 19:20 UTC Continuously adding /24 subnets to our blacklist.
• 20:01 UTC Discussed placement of an additional protection layer with RackSpace to fence off the attack. But these measures would take would take up to 3 hours to set up.
• 20:20 UTC Intentionally brought down the portal website to free up resources for core monitoring services.
• 21:03 UTC Identified the target IP addresses and brought those down.
• 21:10 UTC Rerouted all services on the identified IP’s elsewhere.
• 21:10 UTC Verified pending alerts from the last 30 minutes were now being sent out correctly.
• 21:30 UTC Brought back the web services excluding the targeted IP’s.
• 22:56 UTC Brought back affected Jabber services and verified XMPP alerts being sent out.
• 09:15 UTC Fixed a redirect problem on the watchmouse.com domain.

Thanks for your understanding, we will update this post as noteworthy events arrive.

Performance transparency is critical for both small and large companies alike, which is why we’re pleased to announce the introduction of a new product feature to our WatchMouse monitoring services today – the WatchMouse Joomla widget!

The new widget enables Joomla users to easily publish their WatchMouse Public Status Page/s within the Joomla CMS system by simply installing an open source component and module. The Joomla component uses the WatchMouse API to download the monitoring results, and push them directly to a Joomla website, letting users display live availability and performance information on their Joomla-built website.

The new WatchMouse Joomla widget allows Joomla users, developers and site designers to:

• Publish hourly, daily or weekly availability and performance data
• Display data using a range of maps, charts and graphs
• Adjust the look and feel using CSS or use a selection of pre-existing styles which can be tweaked

View a live sample Providing a simple way for Joomla users to display the status of their critical services can give any size company immediate transparency with their users. We aim to create and introduce more Public Status Page widgets for organizations like Joomla who are the backbone for millions of websites including Tumblr, WordPress, Blogger and more. A WatchMouse Public Status Page (free to WatchMouse subscribers) is a web page that informs customers on the status of a website or service. It can reduce costly customer service interactions and create goodwill with end users. A Public Status Page shows the current status of a specified selection of online services and can display updates and public announcements for customers. The pages are hosted on the Amazon cloud infrastructure, ensuring that a company’s status pages are highly scalable. It also ensures that status pages continue to be publicly available even if a company’s main site or service is not. To get started:

• Sign up for a free 30-day trial or log into your existing WatchMouse account
• Set up a Public Status Page following the instructions published at the bottom of this page
• Download the widget from our Joomla page
• Login to your Joomla site and navigate to Extensions -> Install/Uninstall
• Click Browse, locate the component’s zip file and click the Upload File & Install button
• Click Browse again and locate the module’s zip file and click the Upload File & Install button
• Your installation is complete, navigate to Components -> Watchmouse PSP Widget and check our tutorial

As World IPv6 day dawned in Australia on June 8, we posted a blog reporting that a dismal 2.6% of the world’s top 500 websites supported IPv6 on their main “www” host (a.k.a. dual stack). As the sun rose across the rest of the world we continued measuring and found an astonishing difference.

85 dual stacked websites

Several hours ago, we reported that of the top 500 websites only 13 were dual stacked (in tech talk, they have an AAAA DNS record next to the usual A record). As World IPV6 day continued to dawn across the rest of the world and additional 72 website started to show signs of life over IPv6; taking the total to 85. Although it was a dramatic increase over a single day the figures means that only 17% of the top 500 websites are IPv6 ready.

At 4:00 UTC today, the following hosts resolve to an IPv6 address next to the typical IPv4 address.

These hosts are:

With an additional 72 websites added in the last day, many hosts have clearly waited until World IPv6 Day to enable their websites to be reached over IPv6.

Some stats over time

The below chart shows that on World IPv6 day, only 17% of top 500 websites supports IPv6 on their main “www” host (a.k.a. dual stack), while 11% supports IPv6 on a dedicated host.

It’s worth noting that quite a number of the investigated hosts either have a dead website, or one that is down (unavailable) more than it is up. We excluded such websites from the results below.

Conclusion:

IPv6 has undoubtedly gaining some traction but the penetration remains disappointing as very few are fully functional websites.

Monitoring is our bizz, also over IPv6

We were curious to find out if IPv6 has influence on the performance of websites, so we monitored all hosts with our IPv6 monitoring solution from September 2010 until the end of April 2011.

The results are stated below:

We can make two observations:

• Resolving IP addresses for IPv6 seems to go slightly faster
• Websites on IPv4 load faster than their IPv6 counterpart

We don’t wish to draw any immediate conclusions from these preliminary findings as many factors can play a role; possibly the most important factor being that IPv6 websites are different from the IPv4 websites and more experimental in nature (e.g not up to scale etc.).

The complete list

Below you can see all hosts that support some form of IPv6 access:

We have more stats available: if you are interested please leave a comment below!

We like measuring stuff, so we did an inquiry into the top 500 trafficked global Internet websites to see which ones have adopted the new IPv6 protocol. We have continuously tracked these 500 host names from September 2010 leading up to today’s World IPv6 day. Surprising, when we started tracking, only one had IPv6 on their “www” host name: www.free.fr, a French broadband access provider.

From one to 13 dual stack websites

Since September, we have found that 13 hosts have started to show signs of life on IPv6. Today the following hosts resolve to an IPv6 address next to the typical IPv4 address (in tech talk, they now have an AAAA DNS record next to the usual A record).

These hosts are:

• www.free.fr
• www.bit.ly
• www.facebook.com
• www.fbcdn.net
• www.naver.com
• www.engadget.com
• www.aol.com
• www.mapquest.com
• www.hostgator.com
• www.zynga.com
• www.wordreference.com
• www.mozilla.com
• www.t-online.de

Note that some of these websites have reverted to IPv4 since we first saw IPv6 connectivity.

Dedicated IPv6 websites growing as well

Next to the “www” host names we also searched for hosts especially set up for IPv6 by looking for all sorts of varieties on the main domain e.g.

• www6.domain.com
• ipv6.domain.com
• www.ipv6.domain.com
• www.v6.domain.com

Using this search method we found many more results; as of today, our trackers detected 64 hosts with an IPv6 address. The growth in the number of dedicated IPv6 test websites is shown below:

It’s unlikely to be a coincidence that about half of all hosts that can be reached over IPv6 were added in May, a month before today’s World IPv6 Day.

Some stats

Of the top 500 sites only 2.6% supports IPv6 on their main “www” host (a.k.a. dual stack), while 9.8% supports it on a dedicated host, just for IPv6.

It’s worth noting that quite a number of the investigated hosts either have a dead website, or one that is down (unavailable) more than it is up. We excluded such websites from the results below.

Conclusion:

IPv6 has undoubtedly gaining some traction with today’s World IPv6 Day but the penetration remains disappointing as very few are fully functional websites.

Monitoring is our bizz, also over IPv6

We were curious to find out if IPv6 has an influence on the performance of websites, so we monitored all hosts with our IPv6 monitoring solution and are intrigued to find the following:

We can make two observations:

• Resolving IP addresses for IPv6 seems to go slightly faster
• Websites on IPv4 load faster than their IPv6 counterpart

We don’t wish to draw any immediate conclusions from these preliminary findings as many factors can play a role; possibly the most important factor being that IPv6 websites are different from the IPv4 websites and more experimental in nature (e.g not up to scale etc.).

The complete list

Below you can see all hosts that support some form of IPv6 access:

We have more stats available; if you are interested please leave a comment below!

We’re pleased to announce that we’ve joined the Apdex Alliance as a member and bronze sponsor. If you haven’t heard of the Apdex (Application Performance Index) Alliance, they are an organization that supports an open standard – developed by an alliance of companies – that defines a uniform method to report, benchmark, and track enterprise application performance from a user satisfaction perspective.

The Apdex generates a numerical measure of user satisfaction and is the first user experience metric that is comparable across all transactional applications.

As part of our partnership, we’ve integrated the Apdex into our performance monitoring suite, including the creation of a widget that all of our subscribers can now use. Subscribers can link their performance monitors to the widget and have a customized performance-testing indicator on their site. Apdex charts and reports are also now available in the WatchMouse subscriber dashboard.

The Apdex and its standardized method of performance benchmarking are well-known and well-respected in the IT world. We’re pleased to be integrating their user satisfaction data, which perfectly complements our performance testing and monitoring.

See below for an example of an Apdex integrated widget. If you are a WatchMouse customer and would like to create your own widget you can contact us.

Happy monitoring!

Live Apdex Report

 

Apdex Report is powered by WatchMouse website monitoring.

WatchMouse is mainly known for it’s great monitoring service, the quality of it’s checkpoint grid and it’s accurate testing methodology; no question about that. Though, WatchMouse bundles a few more services together with it’s main product, that are sometimes well hidden. One of these services is the Vulnerability Scanner and this is what I’d like to introduce with this blog post.

Most probably, your online business infrastructure includes a few web servers, maybe some ssh and ftp servers, definitely some web applications, web services and who knows what else! All of these services are usually well-guarded gateways to your business. On the other hand, they are just applications written by humans and, most probably, not bug free. Some of their bugs can cause vulnerabilities that hackers may discover and try to exploit with unknown consequences. I guess you are already aware of all this; your business is probably protected by firewalls and intrusion detection systems; you are also likely to have launched a few security tests and your sure you’re fine! Sure..but for how long?

If you take a look at this page you will see that vulnerabilities are discovered every day. You should be monitoring the security of your online business quite often if you really want to feel safe.

WatchMouse Vulnerability Scanning offers exactly that; we make sure we track all known vulnerabilities and we provide tests for each one. We do this by keeping a large database of vulnerabilities and we update it every day; then for each known vulnerability we install a scan module that can detect the new vulnerability and we silently introduce it into the next run of your scan-monitor. Take a look at this page for a list of all scan modules WatchMouse Vulnerability Scanner includes. You could think of our scanner as a hacker emulator.

Setting up a vulnerability scan monitor is quite easy. If you have a WatchMouse account and haven’t tried the Vulnerability Scanner yet, you can activate a trial here. Ten Scan Credits will be added to your existing account, allowing up to 10 standard scans.

Next you need to create a scan monitor; you can do that on the Vulnerability Scan Settings page. Click on the “Add scan” button at the bottom of the page and a simple form should appear. When setting up or editing a vulnerability scan, you can choose the type of scan to be performed (see field “Scan type”). Also, don’t forget to click on the little question marks next to each form element as they will provide some useful tips!
The available scan types are:

• Standard (available in 30 day trial)
• Intrusive
• Intrusive with Denial of Service attacks
• Standard – WEB (available in 30 day trial)
• Intrusive – WEB
• Intrusive with Denial of Service attacks – WEB

The scan types with the “WEB” suffix will scan only the web-server and web applications on your server for XSS and SQL Injection vulnerabilities.

Each scan type consumes a different amount of scan credits with “standard” needing only 1 scan credit. The check interval of your scan monitor can be set to once per week and up to once per day. If, for example, you scan a server once per week (say every Sunday) with a standard scan, then you would need about 4 credits per month. With the current credit prices, you can scan your server on a weekly basis for less than 15 US Dollars per month!

Finally, don’t forget to add an alert contact! That will be used to send alerts when new vulnerabilities are discovered!

After you save your scan monitor, you need to confirm it before you can really use it. This is a security procedure that ensures that no one other than you can test your servers using the WatchMouse Vulnerability Scanner. It requires you to add a confirmation ticket in a file and place that file in your root folder of your web server. For larger companies or for individuals that need to scan a large amount of servers, WatchMouse can pre-activate a fixed number of IP addresses if you provide us with papers that certify the ownership.
Right after you activate your monitor, you will be able to either launch scans directly through the console or wait until the scan scheduler picks it up. After the first run, you will get alerted (via the alert contact you’ve setup earlier) if the scanner detects any important issues. We classify the issues as “informational”, “warnings” or “holes” with the later two considered as important.

Finally, the reporting console provides all the tools you would need to learn and  manage all discovered issues. A live demo of this console can be seen here. So what are you waiting for? Go ahead a give it a try.

Written by Dimitris Balaouras

‘WatchMouse Weekly’ tweets and corresponding blog posts aims to be an introduction with tips and tricks for getting the most out of your WatchMouse monitoring. For all ‘WatchMouse Weekly’ blog posts go here.

Custom Reports offer a great way of sending relevant performance monitoring data to the right contact.

Do your WebMasters need a daily report of confirmed errors? Perhaps a monthly Management report, containing your availability and performance data, would allow your team to spot trends and check SLA compliance? Whatever performance monitoring information your organisation requires, you’ll be able to create a Custom Report to meet it.

To get started, sign-in to your WatchMouse account. Enter your “Reports” dashboard and then the “Custom Reports” tab. For this area you can:

• Click the “add” button to create a new report. The options are numerous! At a minimum you need to provide a title, select a graph, select which monitors and click “save”. By clicking the ‘add’ button, you can insert as many graphs as you like into a single report.
• Modify an existing report by clicking on the name of the report
• Add or change recipients and the reporting frequency. To do this, click the grey triangle and select from your dropdown list the individual/s or group/s that you would like the report to go to. Select the reporting frequency and “save”. (For instruction on how to add or modify your contacts see the inaugural WatchMouse Weekly post)
• The “Actions” menu on the right hand side additionally allows you to: edit, rename, deactivate, delete or preview a report.

Custom Reports are sent as PDF files and the email body contain a summery of all monitors that are included in the report.

All WatchMouse subscriptions include Custom Reports. To check how many Custom Report are included in your current subscription, enter your “Account” Dashboard. Details are found within the “Subscription” tab. From here you can also click the blue “change” link to purchase additional reports.

Custom Reports make it possible to automatically send appropriate performance information to the right contact/s. If you have any questions about this feature please contact us via the HelpDesk from your WatchMouse console.

Written by Simone Maier

‘WatchMouse Weekly’ tweets and corresponding blog posts aims to be an introduction with tips and tricks for getting the most out of your WatchMouse monitoring. For all ‘WatchMouse Weekly’ blog posts go here.

Social networking sites are a global phenomenon. Millions now go online on a daily basis to engage in one or more social networks including Facebook, Twitter and LinkedIn.

What’s the problem with that? With millions of site visitors and complicated web pages with exhausting amounts of content coming from multiple sources, these sites can slow to an Internet snail’s pace.

Why speed matters?

• Web visitors have really short attention spans and high expectations. They will abandon a website in nano-seconds if it lags.
• Google is now using load times as a factor in search placement. Believe it or not, this still matters even to the social network giants.
• “Bad will” or brand damage happens at the speed of light. Sites start to get sluggish, people talk, start tweeting and it’s all over the Internet.

So how are these sites stacking up? We recently monitored the page load time performance of a public profile page of 22 of the world’s top social sites using our real browser monitoring product. We tested these sites from April 6 through April, 20, 2011 using the combination of measuring a profile page using real browsers, which we believe gives us the best representation of actual performance from a real user’s perspective.

Fifty percent of the sites had slow load times. Facebook at 1091 milliseconds, blew away the competition by a long shot and had the fastest page load time during the reporting period, which is fairly impressive considering it also has the most traffic. Coming in last was the ailing MySpace at 7859 milliseconds followed closely by Friendster at 6473 milliseconds and Posterous at 5973 milliseconds.

We use two performance limits to decide if a website’s load time is good, ok or bad: two and four seconds, based on the research conducted by Akamai in 2009. Anything two seconds or under is considered good. Anything over four seconds is considered bad.

Facebook has set a standard and shows that speed can be achieved regardless of traffic and page complexity. Speed still isn’t a top priority for a lot of these very popular sites, and with 50% of the sites being too slow, there is still a lot of room for improvement.

• You can test any website using our free tool: http://loads.in/
• This survey is based on WatchMouse Real Browser Monitoring.
• The current status of all of the social network sites monitored can be seen here: http://social.downornot.com/

Not many people seem to know about this, but our alerting systems can in fact be set up to call any URL in the escalation chain. Doesn’t that sound cool? (If not, then please read the sentence again until it does!) OK, so now that we agree it’s cool, I think this feature can use some promotion, as I’ve checked the numbers and discovered that only 0.1% of our customers are currently using it.

As Simone already wrote in her opening post, you can configure your monitors to escalate a problem to different people inside your company via email, SMS, Jabber/XMPP, …, or more publicly via RSS or even Twitter. But what if you don’t want someone to be alerted, but something, like an internal system or machine, instead? This is what an action URL’s can be used for: They notify another remote system about the alert via the HTTP protocol.

To set one up, click on the “Contacts” tab in either the “Monitoring” or “Reporting” dashboards. Press “new contact” and select the contact type “action” from the first drop down:

Then provide the Action URL that you want our systems to touch in case this alert is triggered, and finally click “save”. The Action URL has now been set up, and it can be used as any other normal contact in the rest of our systems. For example, it could be used as the initial element in an escalation group, to trigger an advanced warning. Technically, our systems will issue an HTTP POST request to the URL you provide, with the (customised) parametrised alert text sent as the form-urlencoded request body, in UTF-8, for example:

monitor=Monitor1&host=www.mycompany.com&type=browser&since=2011-04-19+17:00

By reading the request body on the server side, the alert can be interpreted and acted upon.

As always, we recommend you have different types of contacts in an alerting group, to ensure that important alerts will reach you even when the action URL itself is down.

Enjoy! And do let us know what you are using action URL’s for…

Pieter Ennes
VP Engineering
WatchMouse

Pieter is too modest to write his own biography so I (Simone Maier) am writing it for him…Pieter is a physicist who’s jumped ship and become WatchMouse’s VP of Engineering. As the linchpin of our company, he manages our technical and development teams. In addition to being modest, he’s also one of the smartest guys we know. (Don’t bother queuing ladies, he’s already got a lovely girlfriend).

‘WatchMouse Weekly’ tweets and corresponding blog posts aims to be an introduction with tips and tricks for getting the most out of your WatchMouse monitoring. For all ‘WatchMouse Weekly’ blog posts go here.

No tips or tricks this week as the WatchMouse team is engrossed in our bi-annual DCTC (Don’t Change The Code) conference. For this DCTC we’ve congregated in Utrecht, Netherlands which is the official HQ of WatchMouse.

While Pieter, our VP of Engineering, is trying to rein in our ideas and keep us on track to deliver some major new enhancements, Stan our CEO, has been busy planning a Meet the Mice event. If you live in or near this gorgeous pedal powered city, please come along and join us tomorrow for a drink at Springers’.