Mouse in the House

Posted by simone on May 13th, 2011

WatchMouse is mainly known for it’s great monitoring service, the quality of it’s checkpoint grid and it’s accurate testing methodology; no question about that. Though, WatchMouse bundles a few more services together with it’s main product, that are sometimes well hidden. One of these services is the Vulnerability Scanner and this is what I’d like to introduce with this blog post.

Most probably, your online business infrastructure includes a few web servers, maybe some ssh and ftp servers, definitely some web applications, web services and who knows what else! All of these services are usually well-guarded gateways to your business. On the other hand, they are just applications written by humans and, most probably, not bug free. Some of their bugs can cause vulnerabilities that hackers may discover and try to exploit with unknown consequences. I guess you are already aware of all this; your business is probably protected by firewalls and intrusion detection systems; you are also likely to have launched a few security tests and your sure you’re fine! Sure..but for how long?

If you take a look at this page you will see that vulnerabilities are discovered every day. You should be monitoring the security of your online business quite often if you really want to feel safe.

WatchMouse Vulnerability Scanning offers exactly that; we make sure we track all known vulnerabilities and we provide tests for each one. We do this by keeping a large database of vulnerabilities and we update it every day; then for each known vulnerability we install a scan module that can detect the new vulnerability and we silently introduce it into the next run of your scan-monitor. Take a look at this page for a list of all scan modules WatchMouse Vulnerability Scanner includes. You could think of our scanner as a hacker emulator.

Setting up a vulnerability scan monitor is quite easy. If you have a WatchMouse account and haven’t tried the Vulnerability Scanner yet, you can activate a trial here. Ten Scan Credits will be added to your existing account, allowing up to 10 standard scans.

Next you need to create a scan monitor; you can do that on the Vulnerability Scan Settings page. Click on the “Add scan” button at the bottom of the page and a simple form should appear. When setting up or editing a vulnerability scan, you can choose the type of scan to be performed (see field “Scan type”). Also, don’t forget to click on the little question marks next to each form element as they will provide some useful tips!
The available scan types are:

• Standard (available in 30 day trial)
• Intrusive
• Intrusive with Denial of Service attacks
• Standard – WEB (available in 30 day trial)
• Intrusive – WEB
• Intrusive with Denial of Service attacks – WEB

The scan types with the “WEB” suffix will scan only the web-server and web applications on your server for XSS and SQL Injection vulnerabilities.

Each scan type consumes a different amount of scan credits with “standard” needing only 1 scan credit. The check interval of your scan monitor can be set to once per week and up to once per day. If, for example, you scan a server once per week (say every Sunday) with a standard scan, then you would need about 4 credits per month. With the current credit prices, you can scan your server on a weekly basis for less than 15 US Dollars per month!

Finally, don’t forget to add an alert contact! That will be used to send alerts when new vulnerabilities are discovered!

After you save your scan monitor, you need to confirm it before you can really use it. This is a security procedure that ensures that no one other than you can test your servers using the WatchMouse Vulnerability Scanner. It requires you to add a confirmation ticket in a file and place that file in your root folder of your web server. For larger companies or for individuals that need to scan a large amount of servers, WatchMouse can pre-activate a fixed number of IP addresses if you provide us with papers that certify the ownership.
Right after you activate your monitor, you will be able to either launch scans directly through the console or wait until the scan scheduler picks it up. After the first run, you will get alerted (via the alert contact you’ve setup earlier) if the scanner detects any important issues. We classify the issues as “informational”, “warnings” or “holes” with the later two considered as important.

Finally, the reporting console provides all the tools you would need to learn and  manage all discovered issues. A live demo of this console can be seen here. So what are you waiting for? Go ahead a give it a try.

Written by Dimitris Balaouras

‘WatchMouse Weekly’ tweets and corresponding blog posts aims to be an introduction with tips and tricks for getting the most out of your WatchMouse monitoring. For all ‘WatchMouse Weekly’ blog posts go here.

Posted by simone on May 3rd, 2011

Custom Reports offer a great way of sending relevant performance monitoring data to the right contact.

Do your WebMasters need a daily report of confirmed errors? Perhaps a monthly Management report, containing your availability and performance data, would allow your team to spot trends and check SLA compliance? Whatever performance monitoring information your organisation requires, you’ll be able to create a Custom Report to meet it.

To get started, sign-in to your WatchMouse account. Enter your “Reports” dashboard and then the “Custom Reports” tab. For this area you can:

• Click the “add” button to create a new report. The options are numerous! At a minimum you need to provide a title, select a graph, select which monitors and click “save”. By clicking the ‘add’ button, you can insert as many graphs as you like into a single report.
• Modify an existing report by clicking on the name of the report
• Add or change recipients and the reporting frequency. To do this, click the grey triangle and select from your dropdown list the individual/s or group/s that you would like the report to go to. Select the reporting frequency and “save”. (For instruction on how to add or modify your contacts see the inaugural WatchMouse Weekly post)
• The “Actions” menu on the right hand side additionally allows you to: edit, rename, deactivate, delete or preview a report.

Custom Reports are sent as PDF files and the email body contain a summery of all monitors that are included in the report.

All WatchMouse subscriptions include Custom Reports. To check how many Custom Report are included in your current subscription, enter your “Account” Dashboard. Details are found within the “Subscription” tab. From here you can also click the blue “change” link to purchase additional reports.

Custom Reports make it possible to automatically send appropriate performance information to the right contact/s. If you have any questions about this feature please contact us via the HelpDesk from your WatchMouse console.

Written by Simone Maier

‘WatchMouse Weekly’ tweets and corresponding blog posts aims to be an introduction with tips and tricks for getting the most out of your WatchMouse monitoring. For all ‘WatchMouse Weekly’ blog posts go here.

Posted by pieter on April 20th, 2011

Not many people seem to know about this, but our alerting systems can in fact be set up to call any URL in the escalation chain. Doesn’t that sound cool? (If not, then please read the sentence again until it does!) OK, so now that we agree it’s cool, I think this feature can use some promotion, as I’ve checked the numbers and discovered that only 0.1% of our customers are currently using it.

As Simone already wrote in her opening post, you can configure your monitors to escalate a problem to different people inside your company via email, SMS, Jabber/XMPP, …, or more publicly via RSS or even Twitter. But what if you don’t want someone to be alerted, but something, like an internal system or machine, instead? This is what an action URL’s can be used for: They notify another remote system about the alert via the HTTP protocol.

To set one up, click on the “Contacts” tab in either the “Monitoring” or “Reporting” dashboards. Press “new contact” and select the contact type “action” from the first drop down:

Then provide the Action URL that you want our systems to touch in case this alert is triggered, and finally click “save”. The Action URL has now been set up, and it can be used as any other normal contact in the rest of our systems. For example, it could be used as the initial element in an escalation group, to trigger an advanced warning. Technically, our systems will issue an HTTP POST request to the URL you provide, with the (customised) parametrised alert text sent as the form-urlencoded request body, in UTF-8, for example:

monitor=Monitor1&host=www.mycompany.com&type=browser&since=2011-04-19+17:00

By reading the request body on the server side, the alert can be interpreted and acted upon.

As always, we recommend you have different types of contacts in an alerting group, to ensure that important alerts will reach you even when the action URL itself is down.

Enjoy! And do let us know what you are using action URL’s for…

Pieter Ennes
VP Engineering
WatchMouse

Pieter is too modest to write his own biography so I (Simone Maier) am writing it for him…Pieter is a physicist who’s jumped ship and become WatchMouse’s VP of Engineering. As the linchpin of our company, he manages our technical and development teams. In addition to being modest, he’s also one of the smartest guys we know. (Don’t bother queuing ladies, he’s already got a lovely girlfriend).

‘WatchMouse Weekly’ tweets and corresponding blog posts aims to be an introduction with tips and tricks for getting the most out of your WatchMouse monitoring. For all ‘WatchMouse Weekly’ blog posts go here.

Posted by simone on April 11th, 2011

No tips or tricks this week as the WatchMouse team is engrossed in our bi-annual DCTC (Don’t Change The Code) conference. For this DCTC we’ve congregated in Utrecht, Netherlands which is the official HQ of WatchMouse.

While Pieter, our VP of Engineering, is trying to rein in our ideas and keep us on track to deliver some major new enhancements, Stan our CEO, has been busy planning a Meet the Mice event. If you live in or near this gorgeous pedal powered city, please come along and join us tomorrow for a drink at Springers’.

Posted by admin on April 6th, 2011

Along with the big list of protocols you can monitor using the WatchMouse service and its global infrastructure, you can also execute “transaction application” tests or as they are more commonly referred to, “functional” tests.

Before going through the steps on how to create and upload such a script to your WatchMouse account, lets briefly see what transaction application testing actually is.

Transaction Application Testing

On top of testing the availability and the performance of a website or web application (non-functional testing), you can also test the individual components of it such as, a login procedure, the results of a search in a form, an article submission and so on.

Transaction application testing differs from non-functional system testing in that, with transaction application testing you have to specify and test the functions that the web service is expected to perform.

Creating a transaction testing script

WatchMouse uses the JMeter scripting engine to run transaction application tests.  A JMeter script is like a browser which executes steps that test the functionality of a web application. Note however, that JMeter does not support all the actions supported by browsers, for example it doesn’t execute JavaScript functions.

To create a valid JMeter script we strongly suggest to use the Badboy windows application, which can be downloaded here, with Badboy you can easily:

• Record the actions you want your script to perform, in a browser environment
• Replay the actions you recorded to validate the script functionality
• Export the script to .jmx format, so you can open it with the JMeter application or
• Upload the script directly from Badboy to your WatchMouse account

Exporting your Badboy script to JMeter correctly might require some customization, due to a few differences between Badboy and JMeter execution:

• JMeter doesn’t execute JavaScript, so in order to simulate JavaScript functionality you might need to pass values (for example a session ID) from one call to the other, manually. You can do this by saving a specific value, after an HTTP request, in a variable and use this variable in subsequent HTTP requests.
• Badboy executes its actions in a linear fashion while JMeter needs to define a scope for every action (element). So for example, if you add an assertion element, to match a text which appears after a login procedure (i.e. the text “log out”), in JMeter you should add that element as a child of the login HTTP request rather than putting it after the request in the list of calls.
• Unlike Badboy, JMeter doesn’t download the embedded elements and assets of a web page (images, css and JavaScript included files etc.). It only tests the functionality of it. You can enable downloading of embedded elements by choosing the corresponding setting in the JMeter application.

Uploading your scripts to WatchMouse

To upload the script to your WatchMouse account you have to:

• Create a new monitor
• Choose “script” in the “type” dropbox
• Upload the script, using the upload form
• Save your monitor

The WatchMouse engine will check the validity of your script and then create the new monitor.

NOTE: Due to the number of calls a script monitor performs, WatchMouse has a default timeout of 20 seconds for these type of monitors. You can adjust the timeout, according to your script, in the monitor “expert mode” settings.

Getting Help from us

You can find a set of example scripts we have created for reference, which test different kinds of applications (SOAP, OAuth, HTTP authentication) here: WatchMouse JMeter repository

We are also happy to help to construct the scripts. Just send the script to helpdesk AT watchmouse.com along with a small description of the difficulties you are facing and we will fix the script for you.

We hope this post will help you understand, as we do in WatchMouse, the importance of transaction monitor testing and also the fun of creating such tests for you websites and web applications.

Post by Nikos Prodromidis: I am a QA Tester and Junior Developer at WatchMouse. I joined the team in June 2009. I find the process of making and understanding functional tests for web applications (i.e. scripting) very interesting and innovative, also I like learning and implementing new technologies.

‘WatchMouse Weekly’ tweets and corresponding blog posts aims to be an introduction with tips and tricks for getting the most out of your WatchMouse monitoring. For all ‘WatchMouse Weekly’ blog posts go here.

Posted by mark on March 29th, 2011

Posted by stan on March 22nd, 2011

We have a complete API at WatchMouse, which is used by our partners, clients, and ourselves too. It can be used to:

• integrate monitoring functionality in other services, like a DNS failover that one of our clients implemented
• base web widgets on, like the Apdex widget,
• create new web services with, like Loads.in, and DownOrNot,
• build mobile apps like the iPhone cloudstatus app.

The unabridged version of the manual is on apidoc.watchmouse.com. To get you started with the API, however, I thought to do a short walk-thru and show you some tips and tricks along the way.

First tip is the API endpoint, http://api.watchmouse.com/1.6/ (1.6 being the current version). You use this in your code obviously, but the nice thing here is that you can also use it in a web browser to experiment with it. Just click it and you see all calls that are currently supported. Now click on cp_list for example, and you will see a short description, the price (free in this case), and the parameter list together with the type of each parameter, if it’s optional or not, and the default values.

To retrieve a list of all monitoring stations as an XLS file, just enter xls in the ‘format’ field and hit ‘exec’. Or to get one random station that supports FTP checks, fill in 1 for the ‘limit’ parameter and ftp for the ‘protocol’ parameter, hit ‘exec’ again and a XML document will be returned with the requested information.

Tip 2: By default, all call return an XML document. The structure of that XML document is documented just below the ‘exec’ button. However, by entering some string in the ‘callback’ field, a JSONP value is returned, which is very convenient when creating HTML+JavaScript widgets. Do give it a try with this cp_list call now!

Now the cp_list call is fairly simple and does not need any authentication, but when you want to access information in your account, e.g. to get the current status of one of your monitors, you obviously do need to authenticate first. For this the API supports both sessions and permanent authentication tokens. The sessions are managed with acct_login and acct_logout, while the permanent access token is obtained by a call to acct_token. Both acct_login and acct_token return an authentication string, named the ‘nkey’. For convenience, these calls also set a cookie so you don’t actually have to provide the ‘nkey’ when playing around with the API web interface.

So try it now and obtain a token using the acct_token call (provide your username and password, and never hand out that token to anyone else!) and then visit rule_get to get the last 50 checks that were performed in your account (don’t forget to set the ‘reverse’ parameter to ‘y’ or you will get the oldest 50).

Okay, that’s it for now. Do play around and let us know what you think. And if you need help or have a request, just open a ticket in the helpdesk and we’ll follow up asap.

Post by Stan van de Burgt. Stan is CEO and co-founder of WatchMouse. Stan writes regular expressions for fun, and Python is his favorite language du jour.

Posted by mark on March 15th, 2011

When setting up a new monitor, we monitor by default from all stations that support the selected protocol or monitor type. In most cases that means your site or server is monitored from all our, currently, 56 stations.

Depending on your situation and requirements, this default might be desired, but maybe it is not. Deciding how to pick your monitoring locations is pretty straightforward:

• If you have a global audience we recommend to use the default setting. In that case a random monitoring station is selected for each individual check.
• If you have a global audience and would like to monitor evenly from all locations you will have to change the scheduling algorithm to “sequential”, see below how to do that.
• If you have an audience in multiple countries, but not all, simply make a custom selection of the stations in the countries you are interested in.
• If your visitors come from a single county, simply pick a station from that country, and change the scheduling algorithm to “master”.

So that was no rocket science right? Next: how to actually set that up.

First of all the default setting. Here you specify the setting for all new monitors you create. Simply go to the Account preferences and find the “checkpoint selection”. Select all the stations you want to be used in your monitoring pool. Note that a minimum of three stations is required. The reason is that for some monitoring errors (like time-out’s) we perform a second opinion check from another location than the one that reported an error to prevent false alerts.

Existing monitors will not be affected by the changes you made in the account preferences.

To change individual monitors, go to the Monitor settings and click on a monitor (or create a new one). The “Checkpoint order algorithm” and the “Checkpoint selection” settings can be found in the “Expert mode”.

The “Checkpoint order algorithm” determines how the monitoring scheduler operates. The following settings are possible:

• Random: a random checkpoint is chosen each time this monitor is checked. This is the default.
• Master: the first check is always done from the checkpoint specified by you. If an error occurs, the second-opinion check is performed from a random (other) checkpoint.
• Sequential: all checkpoints are used in a fixed order (round robin)
• Sticky: same as random, but when a checkpoint detects an error, the monitor will be checked only from that location until the error disappears.

Selecting a specific set of stations is done at the “Checkpoint selection”. Simply check the check-box and you’ll find the same view as in the account preferences, enabling you to select the stations you want to participate for this specific monitor.

Please leave a comment if you have questions about this or open a ticket at the helpdesk.

Post by Mark Pors. Mark is CTO and co-founder of WatchMouse. His favorite editor is emacs, but he hardly gets to use it nowadays.

‘WatchMouse Weekly’ tweets and corresponding blog posts aims to be an introduction with tips and tricks for getting the most out of your WatchMouse monitoring. For all ‘WatchMouse Weekly’ blog posts go here.

Posted by simone on March 8th, 2011

I am sure that many of you reading this post already know what loads.in is. If you have used loads.in before then skip to the paragraph “Getting the most out of Loads.in”.

Quick Introduction to Loads.in

Loads.in is a service that gives you the ability to see how fast your (or any) website loads in a real browser from over 50 locations worldwide. You can read more and test our service by visiting loads.in You can also visit the post How Fast Does Your Website Load – Here and Abroad?

Getting the most out of Loads.in

So now you know how how many seconds it takes to load your site from locations all over the world. Loads.in also provides snapshots of the webpage as it loads. But, is that enough? Of course not, so we provide you with a waterfall chart for each result based on the browser profile. I believe that many of you may not really know how to read these waterfall charts so, stay turned to what follows. I advise you to load your site using loads.in, click on the waterfall chart icon and continue reading.

How to read a Waterfall Chart

Each row in a waterfall chart represents a different object such as text, image, CSS, JavaScript files. As you will see, there are some objects that load simultaneously, the number of simultaneous downloads depends on the browser’s settings. Remember that each browser renders a site differently. Using Loads.in you can verify your site’s load time using different browser profiles.

Each object requires time to be loaded which can be analyzed in the waterfall chart.

• The green bar represents the connect time, which is the time that the server needs to set up a TCP connection
• The bright pink bar represents the blocking time, which is the time taken while the object waits for another files to be completely downloaded
• The purple bar represents the waiting time, which is the time to first byte: the time until the browser receives the first byte of the object from server
• The bright purple bar represents the receiving time, which is the time the browser needs to receive the whole file

Additionally, there are two vertical lines:

• The blue vertical line shows “DOM is loaded”: when the unformatted text and HTML markup have being loaded
• The red vertical line shows “Page loaded”: when all assets of the page including images, CSS, JavaScript etc. have loaded but before the user’s JavaScript has been being executed

For a fast webpage you want:

1. As few rows as possible
2. The “DOM is loaded” and “Page loaded” vertical lines to occur as early as possible and be as close together as possible.

You can read more about understanding waterfall charts in these four articles:

• How to understand your site’s performance via waterfall chart
• Waterfalls 401: Advanced analysis for non-beginners
• How to read the AOL PageTest waterfall chart
• Firebug Net Panel Timings

Post by Ziogas Chris. I am the youngest (and most fun) web developer at WatchMouse. I started coding seriously when I was 15 years old and from then on, I’ve live & dream in this world. I always want to add new technologies to my back-pack and use it on new projects. I am pleased that WatchMouse helps me to search & learn new technologies.

Posted by simone on March 1st, 2011

Setting up a WatchMouse Public Status Page is a simple task performed from the WatchMouse website.  There are also a few nice articles that walk through the whole procedure and can be found at http://www.watchmouse.com/en/feature/public-status-page.html or download the User Manual here: http://www.watchmouse.com/assets/docs/WatchMouse_PSP_Guide.pdf.

What might not be obvious is the logic behind the Public Status Page that indicates performance issues or a service disruption. In this post, I will reveal this little secret and show you how to tweak the algorithm.

Two parameters are predominantly taken into account when measuring the performance of a monitor: “first limit” and “second limit”. Both those parameters can be configured in the monitor setup pages under the “monitoring” dashboard, after switching to the “expert mode”.

If the total time of a public monitor stays below the first limit, the server is performing well. If it totals to a value between the first and second limit, the server is considered to perform poorly. Above the second limit, the performance is considered bad.

A WatchMouse Public Status Page uses both these parameters to identify performance issues and service disruptions.
For the history, it compares the average total time of each day with those parameters. The current performance measurement is based on exponential weighted average of most recent check results.

Setting up these parameters correctly is very important for your Public Status Page. Having them too low will result in a Public Status Page that continuously indicates performance issues whereas having them them too high will hide performance issues from your visitors which, they will eventually find out anyway.
If you haven’t already tuned these parameters, I’d strongly recommend that you do so after considering the following tips:

• Get to know your monitors; check the performance charts under the “reports” dashboard.
• Set the first limit slightly higher than the average total time of your monitors.
• Set the second limit close to the total time it takes to load during a high traffic period.

For example: if you see that the average page load for a specific monitor is 4 seconds, set the first performance limit to 5000ms and the second limit to 8000ms. You can always check your Public Status Page to ensure the performance icons reflect what you had in mind. If not, you now know how to fix it!

For any questions or assistance just leave a comment or contact us through the help desk.

Post by Dimitris Balaouras. I’m the Lead Programmer at WatchMouse. I joined this great team of nerds back in 2006 and I have remained a true fan of WatchMouse ever since. Passionate about software engineering, I enjoy programming more than anything. I’m based in Greece and recently moved from the crowded Athens to Larisa, a small town in Northern Greece where I can code in peace