CMMI - SQA Audit

SQA audit is the audit conducted by the SQA of the project / function.

From the CMMi point of view below are the four goals:

1. SQA activities are planned.

2. Adherence of software products and activities to the applicable and defined standards, procedures, and requirements are verified objectively.

3. All affected groups and individuals are informed of software quality assurance activities and results.

4. Non-compliance issues that cannot be resolved within the software project are addressed by senior management.

Commitment needed: The project follows a written organizational policy for implementing software quality assurance (SQA). This policy typically specifies that:

1. The SQA function is in place on all software projects and functions.

2. The SQA group has a reporting channel to senior management that is independent of

- Project manager
- Software engineering group
- And other software-related groups

3. Senior management should periodically reviews the SQA activities and results.

Four Abilities:
1. A group that is responsible for coordinating and implementing SQA for the project (i.e., the SQA group) exists.

2. Adequate resources and funding are provided for performing the SQA activities:

- A manager is assigned specific responsibilities for the project’s SQA activities.
- A senior manager, who is knowledgeable in the SQA role and has the authority to take appropriate oversight actions, is designated to receive and act on software non-compliance items.
- Tools to support the SQA activities are made available.

3. Members of the SQA group are trained to perform their SQA activities.

4. The members of the software project receive orientation on the role, responsibilities, authority, and value of the SQA group.

Eight Activities:
1. An SQA plan is prepared for the software project according to a documented procedure. This procedure typically specifies that:

- The SQA plan is developed in the early stages of, and in parallel with, the overall project planning.
- The SQA plan is reviewed by the affected groups and individuals.
- The SQA plan is managed and controlled.

2. The SQA group’s activities are performed in accordance with the SQA plan. The plan covers:

- Responsibilities and authority of the SQA group.
- Resource requirements for the SQA group (including staff, tools, and facilities).
- Schedule and funding of the project’s SQA group activities.
- The SQA group’s participation in establishing the software development plan, standards, and procedures for the project.
- Evaluations to be performed by the SQA group.
- Audits and reviews to be conducted by the SQA group.
- Project standards and procedures to be used as the basis for the SQA group’s reviews and audits.
- Procedures for documenting and tracking non-compliance issues to closure.
- Documentation that the SQA group is required to produce.
- Method and frequency of providing feedback to the software group and other software-related groups on SQA activities.

3. The SQA group participates in the preparation and review of the project’s software development plan, standards, and procedures.

- The SQA group provides consultation and review of the plans, standards, and procedures with regard to compliance to organizational policy, compliance to externally imposed standards and requirements (e.g., standards required by the statement of work), standards that are appropriate for use by the project, topics that should be addressed in the software development plan, and other areas as assigned by the project.
- The SQA group verifies that plans, standards, and procedures are in place and can be used to review and audit the software project.

4. The SQA group reviews the software engineering activities to verify compliance:

- The activities are evaluated against the software development plan and the designated software standards and procedures.
- Deviations are identified, documented, and tracked to closure.
- Corrections are verified.

5. The SQA group audits designated software work products to verify compliance.

- The deliverable software products are evaluated before they are delivered to the customer.
- The software work products are evaluated against the designated software standards, procedures, and contractual requirements.
- Deviations are identified, documented, and tracked to closure
- Corrections are verified.

6. The SQA group periodically reports the results of its activities to the software engineering group.

7. Deviations identified in the software activities and software work products are documented and handled according to a documented procedure. This procedure typically specifies that:

- Deviations from the software development plan and the designated project standards and procedures are documented and resolved with the appropriate software task leaders, software managers, or project manager, where possible.
- Deviations from the software development plan and the designated project standards and procedures not resolvable with the software task leaders, software managers, or project manager are documented and presented to the senior manager designated to receive non-compliance items.
- Non-compliance items presented to the senior manager are periodically reviewed until they are resolved.
- The documentation of non-compliance items is managed and controlled

8. The SQA group conducts periodic reviews of its activities and findings with the customer’s SQA personnel, as appropriate.

Measurement: Measurements are made and used to determine the cost and schedule status of the SQA activities.

Three Verifications:
1. The SQA activities are reviewed with senior management on a periodic basis.

2. The SQA activities are reviewed with the project manager on both a periodic and event-driven basis.

3. Experts independent of the SQA group periodically review the activities and software work products of the project’s SQA group.

Sample Checklist of SQA audit:

1. Is the Work order Amended for the change in PM or extension of Project?

2. Are the latest versions of Development Plans and Estimates Approved?

3. Is the schedule updated?

4. Are the planned efforts for the current phase (from schedule), consistent with the effort distribution on SQAP and the latest approved estimate?

5. Is the actual start and end dates for the activities completed since last SQA audit?

6. Are the Internal Review meetings held and Minutes of Meeting prepared?

7. Are the minutes of meeting circulated to affected support groups?

8. Are the Efforts captured in the effort capturing tool?

9. Is there any evidence for implementing the corrective action identified on the previous metrics sheets?

10. Are the metrics tracked and the metrics sheet updated, as per the frequency of analysis in Software Quality Assurance Plan?

11. Is the Master List of Project Document updated?

12. Are Change Control Forms available for the changes in Configurable Items?

13. Are staffing Plan, Risk Plan updated?

14. Are weekly status reports being sent to Sr. Management?

15. Are status reports being sent to customer as per Software Project Management Plan?

16. Is project back up taken? Is any Back up restoration check conducted?

17. Has DAR (Decision Analysis and Resolution) been performed as planned?

18. Has CAR (Casual Analysis and Resolution) been performed at end pf phases?

19. Is Requirements Trace ability report updated?

20. Are the relevant audits conducted for completed activities?

21. Are peer review records available for the work products developed since last audit? Are the peer review findings tracked to closure?

22. Are updated test cases with actual results available?

23. Are the tools, standards and templates used for the current development activity, identified in the relevant plans?

24. Are the Non compliances for the audits last conducted closed?

10 Open Source Mobile Test Automation Tools

Here is a list of top ten open source mobile test automation tools to allow you to automate mobile apps and mobile web. Some open source tools for mobile automation can be used for both Android and iOS as well as Native, Web and Hybrid.

Appium(Android and iOS)

Appium is an open-source tool for automating native, mobile web, and hybrid applications on iOS and Android platforms.Native apps are those written using the iOS or Android SDKs.Mobile web apps are web apps accessed using a mobile browser (Appium supports Safari on iOS and Chrome or the built-in ‘Browser’ app on Android). Hybrid apps have a wrapper around a “webview” — a native control that enables interaction with web content. Download Appium



Calabash (Android and iOS)

Calabash is an automated acceptance testing framework for mobile apps. Calabash could be compared to Selenium WebDriver. However, it is important to realize that interacting with a web app from a desktop computer is vastly different than interacting with a native app using a touch screen. Calabash provides APIs that are specialized to native apps running on touch screen devices.Download Calabash



Frank (iOS)

Frank allows you to write structured text test/acceptance tests/requirements (using Cucumber) and have them execute against your iOS application. Frank also includes a powerful “app inspector” (called Symbiote) that you can use to get detailed information on your running app. Download Frank



MonkeyTalk (Android and iOS)

MonkeyTalk automates real, functional interactive tests for iOS and Android apps – everything from simple “smoke tests” to sophisticated data-driven test suites. Native, mobile, and hybrid app, real devices or simulators. Download MonkeyTalk



iOS UI Automation (iOS)

Use the Automation instrument to automate user interface tests for your iOS app through test scripts that you write. These scripts simulate user actions by calling UI Automation, a JavaScript programming interface that specifies actions to be performed in your app as it runs. During the tests, the system returns log information to you.
When you automate tests of UI interactions, you free critical staff and resources for other work. In this way you minimize procedural errors and shorten the amount of time needed to develop product updates. More information



Robotium (Android)
Robotium is an Android test automation framework that has full support for native and hybrid applications. Robotium makes it easy to write powerful and robust automatic black-box UI tests for Android applications. With the support of Robotium, test case developers can write function, system and user acceptance test scenarios, spanning multiple Android activities. Download Robotium



iOS-driver (iOS)
Automate any IOS native, hybrid, or mobile web application using the Selenium / WebDriver API. ios-driver is fully compatible with the Selenium / Webdriver API. IOS automation is therefore as easy as automation for a browser.
ios-driver fully integrates with Selenium Grid so you can reuse your existing web automation infrastructure including your helper and utility classes (i.e. data creation, page objects etc.)
Download iOS-driver



Ui Automator (Android)

The uiautomator testing framework lets you test your user interface (UI) efficiently by creating automated functional UI testcases that can be run against your app on one or more devices.
The uiautomator API is bundled in the uiautomator.jar file under the /platforms/ directory. The API includes these key classes, interfaces, and exceptions that allow you to capture and manipulate UI components on the target app. More information



KeepItFunctional (iOS)

KIF, which stands for Keep It Functional, is an iOS integration test framework. It allows for easy automation of iOS apps by leveraging the accessibility attributes that the OS makes available for those with visual disabilities.
KIF builds and performs the tests using a standard XCTest testing target. Testing is conducted synchronously in the main thread (running the run loop to force the passage of time) allowing for more complex logic and composition. Download KIF



Selendroid (Android)

Selendroid is a test automation framework which drives off the UI of Android native and hybrid applications (apps) and the mobile web. Tests are written using the Selenium 2 client API.Download Selendroid

Automated Mobile Testing Tools

Automating the testing of mobile applications is especially challenging as mobile devices are less accessible and less open than standard desktop environments or web-based applications. Vendors of automated testing tools usually offer specialized modules or additional tools as part of their products designed for mobile applications. There are also a couple of free open source tools that you can use to automate your mobile application tests.

Commercial

DeviceAnywhere

DeviceAnywhere is available in multiple editions and allows you to test your apps and websites on different devices and automate your mobile tests.

Commercial

Ranorex

Ranorex can be used to automatically test mobile applications on iOS, Android and Windows 8 devices. Ranorex allows you to directly record your tests on your device to build your tests.

Commercial

eggPlant

eggPlant is another test automation tool that also supports various mobile development platforms such as iOS, Android and Windows Mobile/Phone.

Commercial

Silk Mobile

Silk Mobile is a test automation tool for web, iOS, Android, Blackberry and Windows Phone devices, which allows you to build and run your automated tests without having to jailbreak your devices.

Commercial

SeeTest

SeeTest is a test automation tool for iOS, Android, Blackberry and Windows Phone. It allows you to record your tests on real devices and can be used to build cross-platform test automation suites.

Open Source

MonkeyTalk

MonkeyTalk is an open source test automation tool for mobile devices. It can be used to automate real, functional interactive tests for native, mobile, and hybrid iOS and Android apps, everything from simple smoke tests to more complex data-driven test suites.

Open Source

NativeDriver

NativeDriver is an implementation of the WebDriver API which drives of the UI of a native application rather than a web application. The project is working on drivers for Android, iOS and Windows/Windows Phone devices and applications.

OPEN SOURCE! THE TOP 10 IOS AND ANDROID AUTOMATION TOOLS

• Appium (iOS/Android) is a splendid open source tool used for automation of native and web as well as hybrid apps. What I mean by that is the fact that it is good for automation of apps written on original Android or iOS SDKs (native), web applications that are accessible via a mobile browser (web) and applications that are wrapped around the ‘webview’- the native control feature which enables various interactions with web contents (hybrid).
• Calabash (iOS/Android) is an acceptance testing automation framework. It may be compared with the famous Selenium Web Driver. Yet every tester has to realize the difference between testing the app on a desktop machine and on a touchscreen devise. Especially when dealing with a native application. Calabash may provide the necessary API full of ‘touchscreening’ experiences.
• Frank (iOS) actually allows you to write structure text tests as well as acceptance tests and requirements (via Cucumber) and all that fun will be executed against your iOS app. You can also get all the required info about your running app via Frank’s Symbiote, the app inspector in other words.
• MonkeyTalk (iOS/Android) can automate all possible real and functional interactive tests. Everything from little “Smoke Tests” to the tough data-driven test suits. It does bot real device and simulation tests with native and hybrid as well as mobile apps.
• iOS UI Automation (for iOS)  This one is an instrument. The one you can use for automating your user interface tests through some written test-scripts.
• Robotium (Android)  A great tool for both native and hybrid apps. Robotium makes black box testing fairly easier. With it test case developers are free to write function and system as well as user acceptance test scenarios.
• iOS Driver (iOS)  is capable of complete integration with the Selenium grid. Thus you will have the possibility of reusing the already existing wen automation infrastructure. This also includes the helper as well as the utility cases.
• UI Automator (Android) This UI Automation framework allows you to test your UI via creation of automated functional test cases. They will be able of running against your app on a single or multiple devices.
• KeepItFunctional (iOS) KIF is an integration test framework. KIF is building and performing cases with a standard XCTest testing target.
• Selendroid (Android) is but one more test automation framework. It is driven of the UI of both native and hybrid apps as well as the mobile web. So the tests will be written via Selenium 2 client API.

15 Best Test Management Tools for Software Testers

Test Management encompasses anything and everything that we do as testers.  Our day-to-day activities include:

1. Creating and maintaining release/project cycle/component information
2. Creating and maintaining the test artifacts specific to each release/cycle that we have- requirements, test cases, etc.
3. Establishing traceability and coverage between the test assets
4. Test execution support – test suite creation, test execution status capture, etc.
5. Metric collection/report-graph generation for analysis
6. Bug tracking/defect management

1) QMetry:

QMetry is a total test management tool that you can use to create requirements, test cases (test suites) that can be run on multiple platforms and defects. It integrates seamlessly with many leading bug-tracking and automation environments making a good candidate for use in most situations. It is a commercial product, with a 30 day free trial available. Visit the site for more information at:http://www.qmetry.com/

2) TestRail:

TestRail is a centralized test case management tool – you can use it to create test cases and test suites, track execution and report metrics. Additionally, it integrates with many issue tracking tools that makes requirements from external systems to be linked to test cases in TestRail; bugs can also be created in the external systems and links can be established to the corresponding test case. It comes with a HTTP-based API to integrate with the automated test results. One most common integration is with Gemini, which is a incident/ticket management system(supports agile too). It is a commercial product with a free trial available at:http://www.gurock.com/testrail/

3) JIRA:

JIRA is tool that makes an appearance anytime there is a discussion on any-management process- for all the right reasons. JIRA has 2 add-ons that support the test management process.

  a) Zephyr: All the aspects that you would expect of a typical tool of this type are supported. You can create tests/test suites/test cycles/bugs/reports and so on. You could have an additional add-on, ZAPI for automation integration.  Along with the initial JIRA license you would have to pay for Zephyr to use it. ($10 for 10 users a month). There is a free trail available too. Check out information about it at:https://marketplace.atlassian.com/plugins/com.thed.zephyr.je

  b) Go2Group SynapseRT: This tool has all the test management features but the primary focus is on requirement based testing. It can be used for projects where it makes more sense to track your progress in terms of the completion and/or success-failure of a certain requirement as opposed to test cases. Traceability is of higher priority with this add-on. Along with the initial JIRA license you would have to pay for this add-on too. ($10 for 10 users a month). There is a free trail available at:https://marketplace.atlassian.com/plugins/com.go2group.jira.plugin.synapse

4) XQual

XQual delivers one of the most advanced Test Management solution. From this unique tool, you’ll be able to manage your releases, requirements (incl. risks), specifications, tests, campaigns and bugs.

It can be integrated with all the continuous integration platforms and it can drive ANY kind of test: 5 different interfaces for manual testing and 40+ connectors with all the best automation frameworks on the market: Selenium, QTP, AutoIt, JMeter, Ranorex, TestComplete, JUnit, TestPartner, Sahi and many more. Integration with many third-party requirement or bug-tracking systems is also possible. There is also a free community edition available here:http://www.xqual.com/support/download.html

4) qTest:

Developed by QASymphony, qTest is one of the cloud-based test management tools that has all the typical key features. With the help of qTest Connector, it can integrate with JIRA for an entire end-to-end QA solution – but that is not all, it also integrates with other tools like Bugzilla, FogBugz, Rally etc. It is not open sourced but very affordable. Check out information and pricing at:http://www.qasymphony.com/qtest.html

5) TestLodge:

This is a comprehensive test case management tool that has 4 key aspects – Test Plan, requirements, Test suites/cases and test runs. So, as you can see, it has everything it needs to be able to manage test cases for you. For all the other operations, it integrates with the many main stream incident/issue management tools to provide a comprehensive solution. It is a commercial product, for a free trial visit: http://www.testlodge.com/

6) HP ALM/Quality center:

HP QC has been one of the most used test management softwarefor many years. It has all the features necessary and in many ways, it is the standard against which the other tools are measured. Even though it is one of the high-end tools, economically, it still remains to be very popular. Check out http://www8.hp.com/in/en/software-solutions/software.html?compURI=1172141 for trial and other information.

7) Zephyr:

This is the same Zephyr that we discussed earlier as being an add-on to JIRA. The reason why it deserves an independent discussion is because it can be used as an individual product also. For free trial and more info: http://www.getzephyr.com/

8) Testuff:

Testuff is Saas Test Management tool that has many cool features. The typical test case management features are a given. Besides that, it has a cool video upload capability for a defect. Integrates with a huge list of bug-trackers, some of which are, Bugzilla, JIRA, YouTrack, Mantis among others. It has an API that supports automation tools like QTP, Rational Robot, Selenium, TestComplete etc. Best of all, it is very affordable. Check out the features and pricing information at: http://www.testuff.com/

9) Test Collab:

A web based test case management tool that states its speed to be one of its key features due to its ‘Ajax’ified interface(in the exact words of the software makers). It is simple to use, integrates with all major bug/issue/incident trackers. Customizable and has a good reporting facility. It is a commercial product and information is available at: http://testcollab.com/

10) Gemini:

One of the key components of this tool is supporting ‘Testing & QA’ along with the other aspects like Project Planning, issue tracking etc. Using this tool you could create test plans, test cases, test runs, traceability, test run reports etc. There are also various integrations and extensions available. It is a commercial product with a free starter pack available at:http://www.countersoft.com/solutions/testing/

11) PractiTest:

------------

It is an end-to-end QA and Test management software. You can organize your requirements, create tests, run tests, tracks bugs etc, using this tool. It integrates very well with three of the leading incident management tools like JIRA, Bugzilla and RedMine. It is not open sourced but is quite affordable. For more feature and pricing information, check: http://www.practitest.com/

12) TestLink:

This is one of the very few open source test management toolsavailable for use in the market.  It is a web- based tool with typical features like, requirement management, test case creation and maintenance, test runs, tracking bugs, reports, integration with common issue trackers etc. To download, visit:http://sourceforge.net/projects/testlink/

13) QAComplete:

QAComplete is one of the most powerful test management tools that we have.  This suits agile/traditional, manual/automation projects excellently.  You could use it in integration to QTP and TestComplete. For automation projects, you could schedule the test runs and run them remotely on any registered hosts. There is also a detailed release management feature that provides for better analysis. It works along with major bug-trackers and with source control tools – Subversion, Perforce and CVS. Given all the features, it is a little pricey. There is a free trial available though. Check out all its features at: http://smartbear.com/products/qa-tools/test-management/

14) Silk Central:

This is a test management product by Borland. Once you have Silk Central with you there is a nothing left for you to ask for anything else. It is robustness personified, though it is pricey.  All the features for coverage, traceability, reporting, test creation, running – it has got them.  It integrates with many source control and issue tracking system. There are additional plug-ins to extend its capabilities to automation testing using QTP, WinRunner etc. It comes with a video capture feature and supports SAP testing. This product is really cool. Try it at: http://www.borland.com/products/silkcentral/

15) IBM Rational Quality Manager:

A test Management product that has all the typical features – Test planning, test design, test execution, tracking and reporting. It integrates with many of the rational products for automation, source control and bug tracking activities. It is a commercial product. Check out its features, pricing and other information at: http://www-03.ibm.com/software/products/en/ratiqualmana

Additional tools:

The below are some more tools that are worth mentioning:

16) VersionOne: A commercial product primarily catering to agile projects, this has a test management module along with planning, reporting and others, with all the typical features. Check out:http://www.versionone.com/product/agile-test-management/

17) TestPad:  This tool’s motto is “Spend more time actually testing”. The primary concept of this tool is – checklists. In your test plan you could have a series of checklists (tests) that can be as detailed or as short as possible. It is perfect for exploratory testing. It is commercial and you can try it at: https://ontestpad.com/

18) Aptest: a web based, commercial product that has all the typical features you would expect along with the typical integrations with issue trackers. http://www.aptest.com/atm2/

19) SpiraTest: A complete QA solution is what this product is. Instead of having your requirements, tests, defects in separate systems, this tool has it all in one place. It integrates with unit and automation testing frameworks among other things. It is commercial, though not very expensive.https://www.inflectra.com/SpiraTest/Default.aspx

20) Meliora TestLab: This is a simple to use product for requirement management, test-creating & running, defect management, workflow optimization, and integration with JIRA etc.  Commercial again, check it out at: https://www.melioratestlab.com/

21) SmarteQM: A complete life cycle management tool that provides the complete end-to-end test process support and integrates with other SmartSoft functional test tools.  It is web based and commercial.http://www.smartesoft.com/products_smarteQM.php

22) Test Run:  a web based commercial test management process that is easy and simple to use. It has all that you need to create test plans, execute them successfully and report. Integrates with JIRA and LightHouse. http://runtestrun.com/

23) Test Wave: A test management tool that needs no installation, is web based, simple and commercial (affordable). This tool provides you a facility to import your already existing requirements/test assets from excel sheets. It also comes with an inbuilt defect trackerhttp://www.testwave.co.uk/

24) Enterprise Tester: This test management tool supports both agile and traditional projects. Integrates with JIRA and also works great for automation testing with QTP, Selenium, RFT etc. This is a really cool commercial tool. Check it out at:http://enterprisetester.com/

25) TestLog: A very comprehensive end-to-end test management tool that is easy to install and configure due to its XML database. It allows documentation of both automation and manual test cases. It also comes with a web interface for remote access.  This product is commercial too. http://www.testlog.com/

26) QaTraq:  An open source test process control tool that can be used to create test cases, running them, recording results etc.http://sourceforge.net/projects/qatraq/

Roles and responsibilities of SQA in CMMI

SQA (Software Quality Assurance) Engineer in an Organization going for CMMI Assessment has various responsibilities. SQA Team Structure depends on the overall strength of the organization and number of people involved with process implementations (in Software Development or Services Organization). SQA is the mainly responsible for leading the Quality Initiative in the CMMI organization and is responsible for conducting many associated process improvement activities in the organization.

SQA is involved with these CMMI Process Improvement activities:

• Process Improvement Planning
• Conducting Audit of products, projects and services
• Development, modification and updating of a CMMI Compliant QMS
• Implementation Checks and Audits and the Follow-up Audits
• Conducting Training for Process Implementers according to their roles
• Training to Project Team Members on core activities like Project Management, Estimation, Reviews etc.
• Metrics Analysis Report preparation
• Preparation for SCAMPI Assessment Documentation and Training

All these activities are targeted towards the Continuous Process Improvement in the organization using CMMI Framework.

SEPG/QAG qualities required for CMMI

SEPG/QAG Group, here are some of the qualities required in a SEPG Team Member in a CMMI Organization:

• CMMI Knowledge is an added advantage. Knowledge of other Quality Framework and Standards makes you a pro.
• Subject Matter Expertise – you should have good knowledge and skill set in your field. It may in Software Development, Requirements Gathering, Software Design, Testing, Quality or any other field. But should understand Software Development Life Cycle models.
• Should have worked with good number of projects and people.
• Understand Processes and know about Process Improvement. Should have experience in assisting project teams in process implementation.
• Excellent communication and interpersonal skills.
• Good Team Player.
• Should be self motivated and capable of motivating others.
• Should be enthusiast about the process improvement and the organization.
• Should take part in SEPG meetings actively.
• Should complete the tasks on time, assigned by SEPG Team.