Tools to analyze the web pages

Here are six tools that can analyzes web pages and tells you why they are slow. Use the following tools to:

• Make your site faster.
• Debug site problem, especially client side and server side stuff.
• Better user experience.
• Improve the web.

#1: Yahoo! YSlow

The Firebug extension for Firefox allows you to debugging, editing, and monitoring of any website's CSS, HTML, DOM, and JavaScript. YSlow works with the firebug extension:

YSlow analyzes web pages and suggests ways to improve their performance based on a set of rules for high performance web pages. YSlow is a Firefox add-on integrated with the Firebug web development tool. YSlow grades web page based on one of three predefined ruleset or a user-defined ruleset. It offers suggestions for improving the page's performance, summarizes the page's components, displays statistics about the page, and provides tools for performance analysis, including Smush.i and JSLint.

Fig.01 Yahoo! Yslow Providing Overall Score For Cyberciti.biz (click to enlarge)

If you apply tips provided by YSlow, your corporate web site or personal blog can be load pretty faster as compare to old version.

Fig.02: YSlow Components Level Report (click to enlarge)

This is useful to find out if Apache or Lighttpd compressing (gzipping) files or not.

Fig.03: YSlow! Graphical Representation of Various Components

• Download Firebug and YSlow
• Developer homepage: Yahoo YSlow

#2: Google Page Speed

Page Speed is an open-source Firefox/Firebug Add-on. You can use Page Speed to evaluate the performance of yoir web pages and to get suggestions on how to improve them.

Fig.04: Google Page Speed in Action (click to enlarge)

Fig.05: Google Page Speed Suggestions

• Download Firebug and Page Speed
• Developer homepage: Google Page Speed

#3: Pagetest (IE specific tool)

This tool only works with MS Internet Explorer. From the project web page:

Pagetest is an open source tool for measuring and analyzing web page performance right from your web browser. AOL developed Pagetest internally to automate load time measurement of its many websites, and it has evolved into a powerful tool for web developers and software engineers in testing their web pages and getting instant feedback. We decided to release it to the grander web development community to further help evolve it into an even more useful - and free - web performance tool.

Fig.06: Waterfall Of My Web Page Load Performance Using (click to enlarge)

• Download Pagetest for MS-IE
• An online version of pagetest is available at webpagetest.org

#4: HTTP Server Benchmarking Tool

ab is a tool for benchmarking your Apache Hypertext Transfer Protocol (HTTP) server. It is designed to give you an impression of how your current Apache installation performs. This especially shows you how many requests per second your Apache installation is capable of serving. See how to use ab command.
httperf is a tool to measure web server performance. It speaks the HTTP protocol both in its HTTP/1.0 and HTTP/1.1 flavors and offers a variety of workload generators.  Following command causes httperf to create a connection to host www.cyberciti.biz send and receive the reply, close the connection, and then print some performance statistics.
$ httperf --hog --server www.cyberciti.biz
Sample Outputs:

httperf --hog --client=0/1 --server=www.cyberciti.biz --port=80 --uri=/ --send-buffer=4096 --recv-buffer=16384 --num-conns=1 --num-calls=1
httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE
Maximum connect burst length: 0
Total: connections 1 requests 1 replies 1 test-duration 0.236 s
Connection rate: 4.2 conn/s (236.0 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 236.0 avg 236.0 max 236.0 median 235.5 stddev 0.0
Connection time [ms]: connect 47.0
Connection length [replies/conn]: 1.000
Request rate: 4.2 req/s (236.0 ms/req)
Request size [B]: 70.0
Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (0 samples)
Reply time [ms]: response 38.0 transfer 151.0
Reply size [B]: header 242.0 content 26976.0 footer 2.0 (total 27220.0)
Reply status: 1xx=0 2xx=1 3xx=0 4xx=0 5xx=0
CPU time [s]: user 0.01 system 0.22 (user 6.3% system 93.6% total 99.9%)
Net I/O: 112.9 KB/s (0.9*10^6 bps)
Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

Following is Like above, except that a total of 100 connections are created and that connections are created at a fixed rate of 10 per second:
# httperf --hog --server www.cyberciti.biz --num-conn 100 --ra 10 --timeout 5
Sample Outputs:

httperf --hog --timeout=5 --client=0/1 --server=www.cyberciti.biz --port=80 --uri=/ --rate=10 --send-buffer=4096 --recv-buffer=16384 --num-conns=100 --num-calls=1
httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE
Maximum connect burst length: 1
Total: connections 100 requests 100 replies 100 test-duration 10.089 s
Connection rate: 9.9 conn/s (100.9 ms/conn, <=4 concurrent connections)
Connection time [ms]: min 186.7 avg 193.6 max 302.3 median 187.5 stddev 20.8
Connection time [ms]: connect 36.4
Connection length [replies/conn]: 1.000
Request rate: 9.9 req/s (100.9 ms/req)
Request size [B]: 70.0
Reply rate [replies/s]: min 9.8 avg 9.9 max 10.0 stddev 0.1 (2 samples)
Reply time [ms]: response 39.5 transfer 117.7
Reply size [B]: header 242.0 content 26976.0 footer 2.0 (total 27220.0)
Reply status: 1xx=0 2xx=100 3xx=0 4xx=0 5xx=0
CPU time [s]: user 0.34 system 9.75 (user 3.4% system 96.6% total 99.9%)
Net I/O: 264.1 KB/s (2.2*10^6 bps)
Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

• Download httppref utility for UNIX like operating systems.
• See ab and httppref man page for more details.

#5: Full Page Test

The Full Page Test loads a complete HTML page including all objects (images, CSS, JavaScripts, RSS, Flash and frames/iframes). It mimics the way a page is loaded in a web browser. The load time of all objects is shown visually with time bars.

Fig.07: Pingdom page test in action

• Pingdom Tools - Full page test

#6: UNIX wget or fetch Utility

wget is used to retrieve the file(s) pointed to by the URL(s) on the command line. It can tell you exact time it spent to download your files:
$ wget http://www.cyberciti.biz/files/test.pdf
$ wget http://www.cyberciti.biz/
Sample Outputs:

--2009-07-15 22:09:05--  http://www.cyberciti.biz/
Resolving www.cyberciti.biz... 74.86.48.99
Connecting to www.cyberciti.biz|74.86.48.99|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `index.html'
    [   <=>                                                                                                               ] 26,976      38.0K/s   in 0.7s
2009-07-15 22:09:07 (38.0 KB/s) - `index.html' saved [26976]

Tools to measure the website performance

Google Page Speed

Page Speed is an open-source Firefox/Firebug Add-on. Webmasters and web developers can use Page Speed to evaluate the performance of their web pages and to get suggestions on how to improve them.

Page Speed performs several tests on a site’s web server configuration and front-end code. These tests are based on a set of best practices known to enhance web page performance. Webmasters who run Page Speed on their pages get a set of scores for each page, as well as helpful suggestions on how to improve its performance.

Yahoo! YSlow

YSlow analyzes web pages and tells you why they’re slow based on the best practices for high performance web sites. YSlow is a Firefox add-on integrated with the popular Firebug web development tool. This tool has helped improve the performance on over 50 Yahoo! properties by 25-50%.  Here is a nice video introduction and usage of YSlow.

Introducing YSlow @ Yahoo!7 Video

Httperf

Httperf is a tool for measuring web server performance. It provides a flexible facility for generating various HTTP workloads and for measuring server performance. The focus of httperf is not on implementing one particular benchmark but on providing a robust, high-performance tool that facilitates the construction of both micro- and macro-level benchmarks.
It does so much more for it’s simplicity.  It can simulate multiple sessions and transactions including replaying ‘live’ traffic from your web access log.   Check out this introductory document below on how it can help measure performance.
httperf – A Tool for Measuring Web Server Performance –

Pagetest (IE only)

The facts remain, IE is still very popular and widely used.  This tool is only available on IE browsers but nevertheless it is very powerful.  Pagetest is an open source tool for measuring and analyzing web page performance right from your web browser.  AOL developed Pagetest internally to automate load time measurement of its many websites, and it has evolved into a powerful tool for web developers and software engineers in testing their web pages and getting instant feedback.  Check out this great video on how to analyze and optimize your website for performance.

Optimizing Web Performance with AOL Pagetest from Dave Artz on Vimeo.

• You can also use an online version of Pagetest here.

Apache JMeter

Apache JMeter is open source software, a 100% pure Java desktop application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions. Here is a quick video tutorial on using Jmeter.

It may be used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types. You can use it to make a graphical analysis of performance or to test your server/script/object behavior under heavy concurrent load.

Selenium

Selenium IDE is an integrated development environment for Selenium tests. It is implemented as a Firefox extension, and allows you to record, edit, and debug tests. Selenium IDE includes the entire Selenium Core, allowing you to easily and quickly record and play back tests in the actual environment that they will run.

Selenium IDE is not only recording tool: it is a complete IDE. You can choose to use its recording capability, or you may edit your scripts by hand. With autocomplete support and the ability to move commands around quickly, Selenium IDE is the ideal environment for creating Selenium tests no matter what style of tests you prefer.

WebInject

WebInject is a free tool for automated testing of web applications and web services. It can be used to test individual system components that have HTTP interfaces (JSP, ASP, CGI, PHP, AJAX, Servlets, HTML Forms, XML/SOAP Web Services, REST, etc), and can be used as a test harness to create a suite of [HTTP level] automated functional, acceptance, and regression tests. A test harness allows you to run many test cases and collect/report your results. WebInject offers real-time results display and may also be used for monitoring system response times.

WebInject can be used as a complete test framework that is controlled by the WebInject User Interface (GUI). Optionally, it can be used as a standalone test runner (text/console application) which can be integrated and called from other test frameworks or applications.

Siege

Siege is an http load testing and benchmarking utility. It was designed to let web developers measure their code under duress, to see how it will stand up to load on the internet. Siege supports basic authentication, cookies, HTTP and HTTPS protocols. It lets its user hit a web server with a configurable number of simulated web browsers.  Here is a pretty good article on how to use Siege.

Http load

http_load runs multiple http fetches in parallel, to test the throughput of a web server. However unlike most such test clients, it runs in a single process, so it doesn’t bog down the client machine. It can be configured to do https fetches as well.  Sample usage:

% ./http_load -rate 5 -seconds 10 urls
49 fetches, 2 max parallel, 289884 bytes, in 10.0148 seconds
5916 mean bytes/connection
4.89274 fetches/sec, 28945.5 bytes/sec
msecs/connect: 28.8932 mean, 44.243 max, 24.488 min
msecs/first-response: 63.5362 mean, 81.624 max, 57.803 min
HTTP response codes:
code 200 -- 49

HTTP Test Tool

HTTP Test Tool is a script based tool for testing and benchmarking web applications, web servers, proxy servers and web browsers.  httest can emulate clients and servers even in the same test script. Emulating servers is a unique feature, very usefull for testing proxy servers, web browsers and HTTP clients.

ApacheBench

ApacheBench is a command line computer program for measuring the performance of HTTP web servers. It was designed to give an idea of the performance that a given Apache installation can provide. In particular, it shows how many requests per second the server is capable of serving.  The ab tool comes bundled with your standard Apache, you can find it in apache install bin directory.

OpenSTA

OpenSTA is a distributed software testing architecture designed around CORBA, it was originally developed to be commercial software by CYRANO. The current toolset has the capability of performing scripted HTTP and HTTPS heavy load tests with performance measurements from Win32 platforms. However, the architectural design means
OpenSTA is a distributed software testing architecture designed around CORBA, it was originally developed to be commercial software by CYRANO. The current toolset has the capability of performing scripted HTTP and HTTPS heavy load tests with performance measurements from Win32 platforms. However, the architectural design means it could be capable of much more.

OpenWebLoad

OpenWebLoad is a tool for load testing web applications. It aims to be easy to use and providing near real-time performance measurements of the application under test. This is particulary useful when you are doing optimization as you can see the impact of your changes almost immediately.

Pylot

Open source tool by Corey Goldberg for generating concurrent http loads. Define test cases in an XML file – specify requests – url, method, body/payload, etc – and verifications. Verification is by matching content to regular expressions and with HTTP status codes. HTTP and HTTPS (SSL) support. Monitor and execute test suites from GUI (wxPython), and adjust load, number of agents, request intervals, rampup time, test duration. Real-time stats and error reporting are displayed.

JCrawler

An open-source stress-testing tool for web apps; includes crawling/exploratory features. User can give JCrawler a set of starting URLs and it will begin crawling from that point onwards, going through any URLs it can find on its way and generating load on the web application. Load parameters (hits/sec) are configurable via central XML file; fires up as many threads as needed to keep load constant; includes self-testing unit tests. Handles http redirects and cookies; platform independent.

Curl-Loader

Open-source tool written in ‘C’, simulating application load and behavior of tens of thousand HTTP/HTTPS and FTP/FTPS clients, each with its own source IP-address. In contrast to other tools curl-loader is using real C-written client protocol stacks, namely, HTTP and FTP stacks of libcurl and TLS/SSL of openssl. Activities of each virtual client are logged and collected statistics include information about: resolving, connection establishment, sending of requests, receiving responses, headers and data received/sent, errors from network, TLS/SSL and application (HTTP, FTP) level events and errors.

FunkLoad

FunkLoad is a functional and load web tester, written in Python, whose main use cases includes Functional testing of web projects and regression testing as well. It?s also capable of performance testing your app by loading the web application and monitoring your servers which helps to pinpoint bottlenecks, giving a detailed report of performance measurement. FunkLoad also expose bugs that do not surface in cursory testing, like volume testing or longevity testing.

FWPTT

fwptt it’s a Web application tester program for load testing web applications. It can record normal and ajax requests. I tested it on asp.net applications, but it should work with jsp, php or other.

Hammerhead – Web Testing Tool

Hammerhead is a web site coverage, HTTP load generator, HTTP benchmarking, and stress testing tool. It has been designed to emulate multiple users from multiple IP addresses at maximum speed.

Raw Load Tester

The application calls the URL you select as many times as you choose and tells you how long it took the server to respond. It writes some additional runtime details to the PHP log file so you can optionally do more granular analysis afterwards. Although the server processes most of the statistics, all URL requests come from the browser. You can run as many browsers and workstations simultaneously as you want.

Apache Flood

Flood is a profile-driven HTTP load tester. In layman’s terms, it means that flood is capable of generating large amounts of web traffic. Flood’s flexibility and power arises in its configuration syntax. It is able to work well with dynamic content.

LoadSlim

LoadSim is a web application load simulator. It allows you to create simulations and have those simulations run against your webserver.

HttpUnit

Open source Java program for accessing web sites without a browser.  Ideally suited for automated unit testing of web sites when combined with a Java unit test framework such as JUnit. Emulates the relevant portions of browser behavior, including form submission, basic http authentication, cookies and automatic page redirection, and allows Java test code to examine returned pages as text, an XML DOM, or containers of forms, tables, and links. Includes ServletUnit to test servlets without a servlet container.

IeUnit

IeUnit is a simple framework to test logical behaviors of web pages. It helps software engineers to create, organize and execute functional unit tests.

Imprimatur

Imprimatur is designed to be the simplest web application functional testing tool that could possibly work. It sends HTTP requests to the application, and then validates the responses using regular expressions. The tests are written in an XML file. Imprimatur supports most of HTTP including: GET, POST, PUT, HEAD and DELETE methods, HTTP authentication, setting and checking of HTTP headers, and file uploads.

Apache Benchmarking tools

httperf is a benchmarking tool that measures the HTTP request throughput of a web server. The way it achieves this is by sending requests to the server at a fixed rate and measuring the rate at which replies arrive. Running the test several times and with monotonically increasing request rates, one can see the reply rate level off when the server becomes saturated, i.e., when it is operating at its full capacity.

autobench is a Perl wrapper around httperf. It runs httperf a number of times against a Web server, increasing the number of requested connections per second on each iteration, and extracts the significant data from the httperf output, delivering a CSV format file which can be imported directly into a spreadsheet for analysis/graphing.

openload is a load testing tool for Web applications. It simulates a number of concurrent users and it measures transactions per second (a transaction is a completed request to the Web server) and response time.

Howto: Performance Benchmarks a Webserver

Apache Benchmark Procedures

• You need to use same hardware configuration and kernel (OS) for all tests
• You need to use same network configuration. For example, use 100Mbps port for all tests
• First record server load using top or uptime command
• Take at least 3-5 readings and use the best result
• After each test reboot the server and carry out test on next configuration (web server)
• Again record server load using top or uptime command
• Carry on test using static html/php files and dynamic pages
• It also important to carry out test using the Non-KeepAlive and KeepAlive (the Keep-Alive extension to provide long-lived HTTP sessions, which allow multiple requests to be sent over the same TCP connection) features
• Also don't forget to carry out test using fast-cgi and/or perl tests

Cross Browser compatibility testing

Cross Browser Testing Tools

Browsershots

Browsershots is a popular free service that will provide screen shots of a page from just about any browser from all of the common operating systems. All you have to do is enter your URL and select the browsers and operating systems that you want to test. Browsershots will then enter your information into the queue, and in a few minutes you should have the screen shots.
There is a paid option that will give you priority over the free requests. When the service is busy it can take a while to get the free screen shots, but you can pay $15 per month for priority processing.

BrowserCam

BrowserCam provides several different services for browser testing. There is a browser capture option that will give you screen shots from any browser and any operating system. There is also a device capture option for working with Blackberry and PDAs. The Remote Access service will test JavaScript DHTML, forms and other dynamic functionality on any platform.
BroswerCam is a paid service with options for daily, monthly, or yearly. They also offer a free trial that will allow you to test the screen capture service by getting up to 200 screen captures within the 24-hour trial.

NetMechanic Browser Photo

Brower Photo from NetMechanic is a paid service that will allow you to test different versions of Internet Explorer, Opera, Firefox and Safari on Windows, Mac and Linux. You can test on various screen resolutions as well.
Browser Photo has a nice print-friendly feature that will help you to show clients or bosses how you’ve test the site and how it appears on different browsers. You can get Browser Photo for $15 for one-time use, or $150 per domain per year for unlimited use.

Litmus

Litmus provides testing on all major browsers and also produces bug reporting to help you identify and correct any errors. Once the testing is complete you can produce a compatibility report to share with clients. In addition to website testing, Litmus also offers testing of HTML emails in all major email clients, and also runs your messages through spam filters and identifies any problems so you can get more of your messages read.
There is a limited version of Litmus available for free. Additionally, there is a day pass with full access (minus spam analysis) for $24, an individual subscription for $49 per month, or a team subscription for $199 per month.

IE NetRenderer

IE NetRenderer is a free tool that’s great for testing websites in Internet Explorer, typically the most troublesome browser. With IE NetRenderer all you have to do is choose a version of IE and enter the URL that you want to test, and you’ll instantly get a screen shot of the page being tested.

BrowserCamp

BrowserCamp allows you to test the compatibility of your design with Mac OS X browsers. You can quickly enter the URL that you want to test and get a full length screen shot at several different screen widths. Additionally, for full testing you can pay $3 for a two-day subscription, or $99 for a one-year subscription, with a few options in between.

IE Tester from DebugBar

IE Tester from DebugBar s a free web browser that allows you to have the rendering and JavaScript engines of IE8 beta 2, IE7 IE 6 and IE5.5 on Vista and XP, as well as the installed IE in the same process.

CrossBrowserTesting

CrossBrowserTesting.com will allow you to test your website, including AJAX and JavaScript. The service is free for less than 5 minutes per session, or you can buy credits for additional minutes. There are no monthly subscription fees, just pay for what you use.

ieCapture

ieCapture is a free tool for getting screen shots from websites in Internet Explorer. It’s very simple, just enter a URL and in a moment your screen shot will appear.

iPhoney

iPhoney is a free iPhone web simulator for designers. To use iPhoney you’ll have to download the program, but there is no cost.

Multiple IE testing tool

---

Download multiple IE here: http://tredosoft.com/Multiple_IE

Spoon

---

http://spoon.net/browsers/

Adobe® BrowserLab

---

This is a high quality browser compatibility online service from Adobe. BrowserLab is ideal for professional web designers and web developers to ensure that their website is compatible on multiple browsers and operating systems.
Adobe® BrowserLab Service

Compatibility Testing

1. Cross Browser compatibility testing 

2. Network Compatibility Testing