Wednesday, October 22, 2008

QTP vs Selenium - my take

This is a little comparison I did of the two premier tools in the market.





































































































































SNOFeaturesQTPSeleniumExplanation
1Actual end user simulation, Is the test conducted
using this tool equivalent to an end user action?
YESNOSelenium performs actions in the background on the
browser. It modifies the DOM structure of the HTML page in order to
perform actions on the page. To be more precise it executes javascript on UI objects within the webpage to perform actions like click, type, select etc. This is the reason why you can execute tests
with the browser minimized. QTP claims to perform end user simulation, in other words executing QTP scripts is equivalent to a person performing those steps manually on the application.
2Support for most UI ComponentsYESYESCertain events, methods and Object properties are
not supported by selenium. But broadly most UI components are
supported. QTP requires extra addins (plugins, not free) to work with dotNet components.
3UI-Object management & storageYESNOQTP comes built-in with object repository. Object
repository management is quite easy in QTP. Selenium has no such built in
feature but objects can be managed using UI-Element user extension. Other custom solutions like properties files can also be used in selenium. But for all such solutions the
map file in selenium has to be hand-coded i.e unlike QTP the objects won't be recorded/added automatically .
4Support for Dialog BoxesYESNO
QTP supports all kinds of IE
dialog boxes. There is partial support for dialog boxes in Selenium.
Some actions like retrieving the tittle of he dialog box can't be performed in selenium
5Support for file upload scenarioYESNO
This is the scenario where you upload a file located in your local machine onto the application using 'browse file' dialog or something similar. Selenium does not support this scenario due to JavaScript restriction. QTP has no problem whatsoever simulating this scenario.
6Support for web browsersYESYESQTP supports IE & Firefox. Selenium supports
IE, Firefox, Safari and Opera and a few more browsers. But either tools are far far away from full cross-browser support. Don't expect scripts created using browser to run flawlessly in another browser.
7
Object Recognition Parameters, Recognition on the
basis of WYSWYG (what you see is what you get)
YESNOSelenium recognizes objects on the basis of the
DOM structure of the HTML Page. The UI objects in selenium have vague
descriptions and don't comply with WYSWYG policy. QTP recognises and names objects based on properties which are more visible and obvious and are hence the objects have user friendly names.
8Object Oriented Language support &
Scalability (as in Integration with External tools utilities and libraries).
NOYESSelenium Supports JAVA, dot net and many other
industry standard programming languages. QTP supports vb script only.
9Integration with test management toolWith HP Quality Center and TestDirector NoQTP integrates seamlessly with QC and TD. Test management and mapping the manual testing process with automation becomes a lot easier with this integration. I have not yet heard of any test management tool that can integrate seamlessly with Selenium. keep an eye out for Bromide though.
10

Types of application supported
Web,Windows(DotNet,VB, Powerbuilder, TCL/TK), Terminal Emulation, Command promptWeb OnlyQTP wins this one hands down. This is one of the main reason why selenium can't even be considered in many cases. Imagine real-time applications like trading terminals, risk management applications built in TCL/TK and powerbuilder. QTP supports most of these interfaces. Selenium on the other hand can work only on applications that open up inside a browser. But aren't most applications moving to the browser based platform? :)
11Support for operating system/platformsNOYESSelenium Supports JAVA and hence can be used in Windows PC or MAC or UNIX. Using selenium you can test your web application in all the above platforms. QTP supports Windows only.
12
Ease of creation of Scripts
ExcellentGoodSelenium IDE Recorder is not as powerful as QTP
but is good for a free tool, many actions are not recorded by the IDE and
have to be manually entered.
13

Technical Support
YESNOQTP offers technical support by phone and mail, HP also have a web-forum.
QTP user community is vast and questions posted on online forums get
answered quickly. Selenium being an open source tool has no official tech
support, the user community is small, less-active and questions on forums
seldom get answered. But the community is growing day by day as the tool gains acceptance.
14

Cost
Costly, $5,000 per seat license when i last checkedFreeMany people want to switch to selenium because it's free. But cost isn't really a factor when your client's a investment bank :)
15

Test Development Environment
Reasonable but not the bestBest in the worldWhen you are working on selenium you have the option of using wide range of IDEs like Eclipse, Netbeans, Visual Studio etc depending on your choice of development language. If you are a developer then you must have developed a taste for rich IDEs and switching to a environment given by a test tool may be hard for you. QTP tests can only be developed in QTP.
16

Integration with development process
No real integration possibleYesTests developed using selenium can be easily part of the development project. Using tools like cruise control Continuous Integration is easier with Selenium. But don't get too caught up with this feature, it's really not that important to be integrated with the development process but it's nice to have.
17

Future in terms of usability and acceptance
Will have it's presence in non-web based application testingWill be the market leader in web application testingThe future bodes well for selenium because it's free, supports all programming languages/platforms and is immensely scalable and expandable due to it being free and open source. Many pundits have predicted that it will completely conquer the web testing market in the next 5 years. It being free makes a huge difference especially when times are hard like what we have now. QTP on the other hand is the current market leader and I think that it will have it's presence for long due to it's user friendliness and support for interfaces other than web.


25 Comments:

venu9955 said...

Good Work, thank u.

Mahendra Kumar. said...

Mahesh,

Thx or this information. Can you please throw some light on the concept of managing objects in Selenium (say, can we simulate GUI Map files/QTP's Test Repositorie's/Silk Test's frame files using selenium)??

-Mahendra.

Mahesh said...

Hi mahendra,
You have 2 options when it comes to object management in Selenium
1. UI-Elements user extension
read all about it here
http://ttwhy.org/home/blog/2007/05/12/selenium-ui-element-locator/

2.Create your own custom solution (For example store the mapping between physical and logical names in .txt file like usernametxtbox=ct100_usr_txt
and have refer the logical name in your scripts.

type usernametxtbox "admin"

In this case you would have to read the map file every time an instruction is to be executed and retrieve the physical name from it.

good luck to you.

Anonymous said...

Ithink your comparison has undervalued selenium a lot.Here are my arguments :
1:MultiBrowser support.In todays cloud based world,if you intend your product to be used by only a handful of people,then you are losing a serious marketshare.2: Ability to run on most OS. If MS Windows is where your realm ends then no probs.3: Ability to add custom functionality. At some stage this will be needed.If you are a person who doesnt mind mindless workaround then no probs because ideally to be able to test what you mean needs AI. 4: I agree on 'WYSWYG' but people who run automated build-regression cycles frequently especiialy agile process based companies to be able to run without knowing the state of the browser (min or max) is a gift.5: Except usability testing all testing is done by engineers.A bit of coding doesnt really matter

Anonymous said...

Basically there are two major factors:

1. Programming skills: If you have a technically skilled QA team, Selenium can do wonders with the help of a programming language

2. Open Source: QTP is insanely expensive, even in this age of competition from various open source tools.

Also, would like to mention about the support - Try reaching QTP (HP) support by phone and/or email and you will realize how well of a response you get. But having a huge community of users and forums, it is indeed easier to get QTP questions answered

Mahesh said...

New points added as per suggestion.

Anonymous said...

For QTP users, moving to Sahi (http://sahi.co.in) may be more beneficial than Selenium. We switched from QTP to Sahi because we need parallel execution of tests and a recorder on IE. Our testers are not programmers so Sahi was a better fit than Selenium. Anyway, Selenium recorder works only on IE, so it was not useful to us.

Mahesh Narayanan said...

I don't think that is correct. Selenium will support parallel execution if used with a framework like TestNG. Parallelism has nothing do with selenium but is a property exibited by the testing framework used. Selenium supports recording in firefox and playback in many browsers (IE,firefox,opera,safari,konqueror etc).

Braidy said...

Hi Mahesh. Good Points.

I have put these in my blog with your reference:

http://www.qtp10.com/2009/08/qtp-vs-selenium.html

Mahesh Narayanan said...

Hi Braidy, Could you please take it off your site? I put in a lot of effort in this article and don't think it's fair.

Millan said...
This comment has been removed by the author.
Millan said...

Its really nice about PowerBuilder. Thanks for sharing this infornation .

Doug said...

Since this article is rather old, has anyone done an update of this info?

Mahesh Narayanan said...

Doug, I have been updating this from time to time. Do you see anything that may need correction?

Anonymous said...

I'm really interested in knowing, as a fresher should i learn selenium or qtp?
which of these tools would be more popular in recent time?

Adam said...

Thanks for this comparison Mahesh. Could you please provide your own conclusion as to which of the 2 software would you recommend to learn?

Thanks!

Adam from QTP Tutorial

Mahesh Narayanan said...

Selenium!

Anonymous said...

Hi Mahesh,

Can Selenium be used for backend data validation for data driven tests?

Thanx

Anonymous said...

Last question is also important for me.. pls find time to reply.

Secondly.. how we can built user friendly custom test report as we have within QTP...

Also can you pls share some selenium scripts which can be reused to validate actual result data.

Unknown said...

Hi Mahesh,

Can you please share some article on selenium and QC integration? I heard that it is possible using OTA ad VAPI-XP test, but can you please confirm it?

Thanks,
Sneha

Ramachand said...

If you are only testing web applications, I would stay away from QTP.

If you don't know how to code, go learn it first. You will need it if you really want to learn test automation.

The only bogus advantage QTP has when testing webapps is its record-and-playback functionality which is probably one of the least bad ones in the market. The reason I call it a bogus advantage is because it gives you a false sense of getting the job done. Record-and-playback leads to unmaintainable test frameworks.

Some might argue that it's possible to create a decent framework using QTP, and they might be right. But for that, you'll have to know your code, which beats the one advantage of the tool.

Go for either Selenium-WebDriver (and stay away from Selenium IDE) or Watir-WebDriver. I find Watir's API to be cleaner and more descriptive than Selenium's, but that's only personal preference. The one disadvantage I see in Watir is that you're stuck with writing code in Ruby -- which I don't see so much as a disadvantage since I really enjoy writing Ruby, but you might if you're already familiar with other languages that Selenium supports.

Other tools you'll find very useful are Firebug on Firefox (and the other similar developer tools for other browsers) and some XPath helpers to get you started with object location.

I also recommend you to check the following discussion on this:

http://softwareqatestings.com/forum/automated-testing/1652-selenium-vs-qtp-i-have-been-doing-manual-testing-more-than-1-year-web.html

Anonymous said...

Hi, do we have the latest comparison long with pros and cons of the QTP and Selenium since the original dates to 2008-2010.Kinldy help in pointing to the right/updated URL/link. Thank you

Soumya Lokura said...

Excellent blog. Highly appreciated

muthu said...

nice blog. Good comparison chart which I expected.

Shruti Ramalingam said...

Thanks for the great information in your blog Selenium Training in Chennai