Why we created API Usability Testing
There are many application programming interfaces (APIs) in the world and counting. When using the term APIs, we include web APIs, SDKs (software development kits), libraries, frameworks, development kits, and toolkits. At the moment of writing, just ProgrammableWeb alone lists about 11000 web APIs: http://www.programmableweb.com/apis/directory. In view of this wealth of available products, we could assert that to be successful in the saturated market it is not enough to provide a rich functionality set or to be reliable only, but it is necessary to be usable from a software developer’s point of view — or, in another words, provide good developer experience. Because the usability problem for software with a graphical user interface (GUI) could be a significant barrier for customers, such an API usability problem could lead to significant customer outflow. Studies show that being able to provide complete and accurate documentation (actually it is one of the most important parts of API usability) is the first priority for developers: http://blog.programmableweb.com/2013/01/07/api-consumers-want-reliability-documentation-and-community/.
Personally being developers and working with many APIs, we often suffer from API usability issues (labyrinthine documentation, homegrown documentation systems, bad examples, less-than-obvious API design, unreasonable error messages, etc.), so we decided to help API providers to deliver really usable APIs and enable them to view their API through the eyes of pragmatic developers who are absolutely new to the API, just want to “get things done,” and do not want to spend nights figuring out how the API works.
What we do
The general idea is to do the following:
- Step 1. Prepare a list of tasks (use cases) that software developers are likely to do with the API and prepare appropriate post-study questions.
- Step 2. Ask the participants (actually they are software developers) to complete the tasks and answer the questions. During the work, the developers write down all the problems they encounter, documentation issues, misunderstandings, remarks, suggestions, questionable design patterns, incoherencies, inconsistencies, bugs, possible design issues, and so on. Source codes for each task are available too.
- Step 3. A customer receives reports from every participant. So, based on the reports a customer could see general issues for usability of the API, documentation issues, “bottlenecks,” and other pitfalls that are encountered by developers.
Our customers do not have to bother with recruiting, educating, or managing developers; we do it all by ourselves. If necessary, we help to specify tasks (use cases) and prepare post-study questions that best fit your needs. If necessary, we analyze the developers’ reports and provide a list of “actionable” recommendations for improving a developer’s experience with the API. Also, by your request we could invite developer experience experts to review your API and provide additional recommendations. Read more about how it works here http://www.apiusabilitytesting.com/how-it-works/.
Our participants are not usability experts; they are just ordinary software developers with different experience and qualifications. They work for different companies all over the world, from small companies to large enterprises, and develop different kinds of software on a daily basis. We carefully select them and work with them to gain the best results.
When all participants finish the testing you will be provided with reports, each of which includes source codes for every task, notes for every task, and answers to post-study questions. As mentioned above, notes include descriptions of any problems they encounter, documentation issues, misunderstandings, suggestions, and so on. An example report is here: http://www.apiusabilitytesting.com/reports/e63b9391-9de7-4ee8-b8b7-303b3153e556/CsharpDotNET/report.html. A customer will discover how developers interact, how they react, what they think and why, and what the problems of using the API are.
We are in a consistent process of improving our services. Your feedback is very important to us. Please let us know your ideas, suggestions, or anything else: firstname.lastname@example.org.