What is API Testing?
An API (Application Programming Interface) is a collection of software functions and procedures, called API calls, that can be executed by other software applications. Application developers code that links to existing APIs to make use of their functionality. This link is seamless and end-users of the application are generally unaware of using a separately developed API.
During testing, a test harness-an application that links the API and methodically exercises its functionality-is constructed to simulate the use of the API by end-user applications. The interesting problems for testers are:
1. Ensuring that the test harness varies parameters of the API calls in ways that verify functionality and expose failures. This includes assigning common parameter values as well as exploring boundary conditions.
2. Generating interesting parameter value combinations for calls with two or more parameters.
3. Determining the content under which an API call is made. This might include setting external environment conditions (files, peripheral devices, and so forth) and also internal stored data that affect the API.
4. Sequencing API calls to vary the order in which the functionality is exercised and to make the API produce useful results from successive calls.
An API (Application Programming Interface) is a collection of software functions and procedures, called API calls, that can be executed by other software applications. API testing is mostly used for the system which has collection of API that needs to be tested. The system could be system software, application software or libraries.
API testing is different from other testing types as GUI is rarely involved in API Testing. Even if GUI is not involved in API testing, you still need to setup initial environment, invoke API with required set of parameters and then finally analyze the result.
Setting initial environment become complex because GUI is not involved. It is very easy to setup initial condition in GUI, In most cases you can find out in a glance whether system is ready or not. In case of API this is not the case, you need to have some way to make sure that system is ready for testing.
This can be divided further in test environment setup and application setup. Things like database should be configured, server should be started are related to test environment setup. On the other hand object should be created before calling non static member of the class falls under application specific setup.
Initial condition in API testing also involves creating conditions under which API will be called. Probably, API can be called directly
or it can be called because of some event or in response of some exception.
Main Challenges of API Testing can be divided into following categories.
· Parameter Selection
· Parameter combination
· Call sequencing
How to test API's?
API Test development is a simple process with following
steps: You also, need to know about an API test case
specification before proceeding with test case creation.
1. Create / Select the Suite in which you want to add the
API test case(s)
2. Choose Test Development Mode ( Isolation or Sequence )
3. Develop Test case(s) for the desired API methods
4. Configure Application Control Parameters
5. Configure Test Conditions
6. Configure Method Validation
7. Execute API test
8. View Test Reports
9. Filter API test cases
10. Sequence API test cases
API Testing: Testing in Layers
For the past few weeks my test automation class at the University of Washington
This seems a bit confusing at first but then we realize that a single method (or function) may be a unit, or a component, or may (although unlikely) be the ‘system.’ A collection of methods wrapped in a library or DLL that interact to meet a functional requirement is a component. Rather than a developer having to call each method individually to achieve some usually repetitive functional outcome from the library that functionality is usually exposed via a call to a single API.
In this situation the students are testing a public API in a single library (DLL) that calls several methods to produce a randomly generated string (outcome) based on parameterized property values. The interface is the single API call (and the property variables) in an automated test, so we might consider the DLL that contains this API as the ‘system’ under test based on Beizer’s definition of component. Also, since students don’t actually see the underlying code, API testing in this context is ‘black-box’ testing.
The debate of who is responsible for API or component testing is tangential to the practice. I promote API testing because even in today’s extreme programming and TDD development lifecycle models we testers are still finding way too many ‘functional’ bugs (as opposed to behavioral bugs) during the integration and system levels of testing.
Also, generally (not all) software is designed and developed in layers similar to this simplified illustration. In well-designed, more easily testable projects the business logic or logical functionality is (should be) contained in classes or libraries (DLLs), and the public methods or APIs in those libraries know nothing about the user interface. and visa versa. End user inputs at the GUI are marshaled to the business logic layer via event handlers and properties (get/set accessors in C#) in different classes.
So, it shouldn’t be a surprise to anyone that certain categories of functional issues are more easily exposed at the API level of testing as opposed to testing through the UI. In fact, sometimes the UI properties and event handler layers in one project may actually mask some bugs in the APIs which aren’t exposed until much later when someone else uses that API in a different application or feature. The value of API testing is that a lot of functional testing can be performed very early in the project cycle, and functional testing can progress while the UI layer is unstable or in flux.
I sometimes think we are stuck testing from the end-user’s perspective. It seems that we often approach testing by trying to expose both behavioral type bugs and functional type bugs by testing completely through the UI. But, if we think of testing in layers the same way many products are developed then I wonder if we could better focus our test designs to target specific categories of functional bugs earlier and concentrate on behavioral issues and end-2-end customer scenarios when we have a more stable UI?
Points to take care while planning for API Testing:
1. Ensuring that the test harness varies parameters of the API calls in ways that verify functionality and expose failures. This includes assigning common parameter values as well as exploring boundary conditions.
2. Generating interesting parameter value combinations for calls with two or more parameters.
3. Determining the content under which an API call is made. This might include setting external environment conditions (files, peripheral devices, and so forth) and also internal stored data that affect the API.
4. Sequencing API calls to vary the order in which the functionality is exercised and to make the API produce useful results from successive calls.
By analyzing the problems listed above, a strategy needs to be formulated for testing the API. The API to be tested would require some environment for it to work. Hence it is required that all the conditions and prerequisites understood by the tester. The next step would be to identify and study its points of entry. The GUIs would have items like menus, buttons, check boxes, and combo lists that would trigger the event or action to be taken. Similarly, for APIs, the input parameters, the events that trigger the API would act as the point of entry. Subsequently, a chief task is to analyze the points of entry as well as significant output items. The input parameters should be tested with the valid and invalid values using strategies like the boundary value analysis and equivalence partitioning. The fourth step is to understand the purpose of the routines, the contexts in which they are to be used. Once all this parameter selections and combinations are designed, different call sequences need to be explored.
The steps can be summarized as following:
1. Identify the initial conditions required for testing.
2. Identify the parameters – Choosing the values of individual parameters.
3. Identify the combination of parameters – pick out the possible and applicable parameter combinations with multiple parameters.
4. Identify the order to make the calls – deciding the order in which to make the calls to force the API to exhibit its functionality.
5. Observe the output.
API Testing and How it is different from testing other common software interfaces like GUI testing
Application programmable Interfaces (APIs) are collections of software functions or procedures that can be used by other applications to fulfill their functionality. APIs provide an interface to the software component. These form the critical elements for the developing the applications and are used in varied applications from graph drawing packages, to speech engines, to web-based airline reservation systems, to computer security components.
Each API is supposed to behave the way it is coded, i.e. it is functionality specific. These APIs may offer different results for different type of the input provided. The errors or the exceptions returned may also vary. However once integrated within a product, the common functionality covers a very minimal code path of the API and the functionality testing / integration testing may cover only those paths. By considering each API as a black box, a generalized approach of testing can be applied. But, there may be some paths which are not tested and lead to bugs in the application. Applications can be viewed and treated as APIs from a testing perspective.
There are some distinctive attributes that make testing of APIs slightly different from testing other common software interfaces like GUI testing:
Testing APIs requires a thorough knowledge of its inner workings - Some APIs may interact with the OS kernel, other APIs, with other software to offer their functionality. Thus an understanding of the inner workings of the interface would help in analyzing the call sequences and detecting the failures caused.
Adequate programming skills - API tests are generally in the form of sequences of calls, namely, programs. Each tester must possess expertise in the programming language(s) that are targeted by the API. This would help the tester to review and scrutinize the interface under test when the source code is available.
Lack of Domain knowledge – Since the testers may not be well trained in using the API, a lot of time might be spent in exploring the interfaces and their usage. This problem can be solved to an extent by involving the testers from the initial stage of development. This would help the testers to have some understanding on the interface and avoid exploring while testing.
No documentation – Experience has shown that it is hard to create precise and readable documentation. The APIs developed will hardly have any proper documentation available. Without the documentation, it is difficult for the test designer to understand the purpose of calls, the parameter types and possible valid/invalid values, their return values, the calls it makes to other functions, and usage scenarios. Hence having proper documentation would help test designer design the tests faster.
Access to source code – The availability of the source code would help tester to understand and analyze the implementation mechanism used; and can identify the loops or vulnerabilities that may cause errors. Thus if the source code is not available then the tester does not have a chance to find anomalies that may exist in the code.
Time constraints – Thorough testing of APIs is time consuming, requires a learning overhead and resources to develop tools and design tests. Keeping up with deadlines and ship dates may become a nightmare.
Testing of API calls can be done in isolation or in Sequence to vary the order in which the functionality is exercised and to make the API produce useful results from these tests. Designing tests is essentially designing sequences of API calls that have a potential of satisfying the test objectives. This in turn boils down to designing each call with specific parameters and to building a mechanism for handling and evaluating return values.
Thus designing of the test cases can depend on some of the general questions like
· Which value should a parameter take?
· What values together make sense?
· What combination of parameters will make APIs work in a desired manner?
· What combination will cause a failure, a bad return value, or an anomaly in the operating environment?
· Which sequences are the best candidates for selection? etc.
Unit Testing Vs API Testing:
Unit Testing and API Testing:
- Unit testing is almost always an activity that is owned by the development team; developers are expected to build unit tests for each of their code modules (these are typically classes, functions, stored procedures, or some other ‘atomic’ unit of code), and to ensure that each module passes its unit tests before the code is included in a build.
- API testing, is typically an activity owned by the QA team, a team other than the author of the code. API tests are often run after the build has been created, and it is common that the team of the tests do not have access to the source code; they are essentially creating black box tests against an API rather than the traditional GUI.
An API (Application Programming Interface) is a collection of functions and procedures, called API calls that can be executed by other software applications.
Key difference between Unit and Testing
- Unit tests are typically designed by the developers to verify the functionality of each unit. The scope of unit testing often does not consider the system-level interactions of the various units; the developers simply verify that each unit in isolation performs as it should.
- API testing, like other activities owned by the QA team, must consider the ‘full’ functionality of the system, as it will be used by the end user (in this case, another program). This means that API tests must be far more extensive than unit tests, and take into consideration the sorts of ‘scenarios’ that the API will be used for, which typically involve interactions between several different modules within the application.
How to approach the API testing task?
- API testing is a testing activity that happens to require some coding, and is usually beyond the scope of what developers should be expected to do. (We need to own this activity)
- Traditional testing techniques such as equivalence classes and boundary analysis are also applicable to API testing, so even if we are not too comfortable with coding,we can still design good API tests.
- We will not be able to test all possible scenarios that are possible to use with your API. Rather we can focus on various flavours of data type and forced error testing
Main Challenges of API Testing can be divided into following categories:
• Parameter Selection
• Parameter combination
• Call sequencing
API Call testing with PHP
First, if you just want to check out the test, here’s the code in a handy .phps file. It’s the sample test I’ll be describing.
01 | public function testSiteStatus() | |
02 | { | |
03 | // This is our object to send | |
04 | $postData = json_encode(array( | |
05 | 'request'=>array('action'=>array( | ||
06 | 'type'=>'status','data'=>array( | ||
07 | 'test_string'=>'my test')) | |
08 | ))); | |
09 | ||
10 | // This is what the object that comes back should look like | |
11 | $response = json_encode(array( | ||
12 | 'dt'=>date('r'),'test_string'=>'my test' | ||
13 | )); | |
14 | ||
15 | // Build the settings for the connection | |
16 | $settings=array( | |
17 | 'location' => '/api/site', | |
18 | 'host' => 'ji-enygma.localhost', | |
19 | 'postData' => $postData, | |
20 | 'outputFormat' => 'json', | |
21 | 'headers' => array( | |
22 | 'Content-Type'=>'text/json' | |
23 | ) | |
24 | ); | |
25 | ||
26 | // Make the request and check the response! | |
27 | $this->post($settings) | |
28 | ->assertEquals($response); | |
29 | } |
The large part of the example test is made up of the “setup” to even make the test happen. In $postData I’ve created a JSON message based on what the Joind.in API isexpecting for the Site/Status request and pushed it into a string. The second JSON object, $response, mimics what the API’s response will look like. The $settingsarray is the more verbose way of specifying settings for the ActionPost request (including the additional headers). The Joind.in API requires that you give it a Content-Type of the message you’re giving it – it’s not a very good guesser.
The next two lines do all of the work – make the post request to the remote API, return the message and check to see if it’s the same as $response. Simple, right?
If you wanted to get a little more fancy with checking, you can use the “paths” in theassertContains to look inside of a JSON object. So, you could do something like:
1 | <?php | |
2 | ||
3 | $this->post($settings) | |
4 | ->assertContains('look for me','message_list/message/message1'); | |
5 | ||
6 | ?> | |
That will look at “message1″ in this JSON to see if it contains that “look for me” string:
{”message_list”:{”message”:{”message1″:”I think you should look for me”,”message2″:”but not here”}}}
What is the difference between WebService and API?
API contains classes and Interfaces just like a program.
A web service is a form of API (Application Programming Interface).
An API is used by a computer programmer to establish a link between software applications. This interface can take several forms, a web service is just one of these.
There are several types of web service. SOAP (Simple Object Access Protocol) is one of the most common. The API takes the form of a service description (WSDL) which is used to automatically generate the program code which makes the connection.
Here is a link to some web service examples:
and also here some other information:
An API is an Application Programming Interface, it's the means by which third parties can write code that interfaces with other code. A Web Service is a type of API, almost always one that operates over HTTP (Though some, like SOAP, can use alternate transports, like SMTP). The Official W3C Definitionmentions that Web Services don't necessarily use HTTP, but this is almost always the case and is usually assumed unless otherwise mentioned.
As an example of another type of API, one written in C for use on a local machine, see the Linux Kernel API.
As far as protocol goes, a Webservice API almost always uses HTTP (hence the Web part), and definitely involves communication over a network. APIs in general can use any means of communication they wish. The Linux kernel API for example uses Interrupts to invoke the system calls that comprise its API for calls from userspace.
Basically, a webservice is a method of communication between two machines while an API is an exposed layer allowing you to program against something.
You could very well have an API and the main method of interacting with that API is via a webservice.
The technical definitions (courtesy of Wikipedia) are:
API -
An application programming interface (API) is a set of routines, data structures, object classes and/or protocols provided by libraries and/or operating system services in order to support the building of applications.
Webservice
A Web service (also Web Service) is defined by the W3C as "a software system designed to support interoperable machine-to-machine interaction over a network"
Think of Web service as a web api. API is such a general term now so a web service is an interface to functionality, usually business related, that you can get to from the network over a variety of protocols.
In a generic sense an webservice IS a API over HTTP. They often utilize JSON or XML, but there are some other approaches as well.
|
Web Services API Testing
If your development plans include an API accessible over the Internet via web services, QualityLogic can provide services to assist you in thoroughly testing the functionality and performance of your API. QualityLogic has a long history of testing APIs for some of the largest players in the tech industry. Although web services are traditionally thought of as XML over SOAP, a far broader range of technologies are used by our customer base to implement their Web APIs, including protocols and data formats such as REST, Atom, GData, RSS, XML-RPC, KML, JSON, KML-GeoRSS, OpenSearch, RDF, YAML, and many others. Regardless of the implementation technologies used for your API, QualityLogic can help with your testing.
QualityLogic’s API testing methodology includes the following steps:
· API Specifi cation Review – A detailed review of the API specification and any
related use case documentation. This review of the API specification from a test perspective typically uncovers numerous errors in the implementation before a single test case is written.
· Test Specification Development – A written test specification is developed detailing the test conditions and expected results for each test case.
· Test Framework Development – Although we use standard open source tools like SoapUI and JMeter for our testing, many times a set of static resources need to be developed as a perquisite to automated test case development.
· Test Case Development – This is the actual coding of the test scenarios. Defects are reported to you as they are uncovered during the test case development.
· Test Execution and Reporting – As API test cases are typically automated, regression testing is typically quick, with most of the effort being placed in problem isolation and reporting.
There are two distinct types of testing for your Web Services API:
Functional Testing and Performance Testing
Functional Testing
Functional testing takes an external perspective of the API, using the API specification and published Use Cases to determine the valid inputs and expected outputs. This is classic “black box” testing, without an intimate knowledge of the API’s internal structure.
A typical API test developed should cover the following areas:
· Baseline Tests – These tests exercise each API method in isolation.
Boundary conditions , Repeating elements , Combinations of parameter values, Default value assumptions (i.e., optional parameters), Data types and sizes, Correct return tags and values
· Task-Oriented Tests – Multiple API calls, are made replicating common tasks and real-world scenarios documented in the Use Cases. Characteristics of these tests are as follows:
Dependencies between API calls
Calling order
Repetitive transactions
State transitions
Propagation of data to external systems
· Forced Error Tests - These tests contain typical error scenarios, such as missing required elements, empty content, and content exceeding maximum limits, across a representative sampling of the API methods. Test assertions include:
Correct error messages
Fallback behavior
Transaction rollback behaviour
Performance Testing
Performance testing a web services API encompasses a wide range of activities, including creating a test environment, setting realistic performance targets, developing test scenarios, generating high quality test input data, test execution, and
root cause analysis. QualityLogic can help with some or all of these activities depending on your needs. Key performance metrics for web services API performance testing include response times, throughput in terms of API calls and data volume, transaction errors, server and network utilization, and a variety of other
key indicators.
These metrics are gathered as the result of the following types of
Performance testing activities conducted:
· Load Testing – The web services API is loaded up to the target number of transactions per second to determine if the response times and other key performance metrics meet targets. Typically this testing is done in a number of stages, including establishing a baseline for each transaction, load testing each transaction independently, and finally load testing entire groups of transactions.
· Stress Testing – The load on the web services API is ramped up until some part of the supporting infrastructure fails. The purpose of this testing is to determine how much margin there is between the expected traffic volumes and the point at which the infrastructure starts to fail.
· Endurance (Soak) Testing – The web services API is subjected to typical loads over extended periods of time, with occasional traffic spikes typical of actual usage model. The intent is to identify problems that occur only after extended uptime, such as memory leaks.
· Scalability Testing – A series of load tests are run with varying infrastructure configurations to determine how best to scale in order to deal with traffic loads. This can be a key part of determining the appropriate production hardware necessary to deal with both current and future traffic volumes.
API Testing Examples:
No comments:
Post a Comment