In this post, I am going to describe the process of an API usability review in more detail. Also, I am going to show you how we analyze results and build an actionable recommendations list (I mentioned this feature in the previous post) for a real API. I will describe step-by-step how we prepare for the testing, how we do the testing and handle results, how we present the results for our customers, and so on.
1. Preparation for the testing
1.1 Define tasks for the participants
The first step in preparing the study is to define tasks for the participants. Actually, there are two options here:
- A customer provides tasks and post-study questions.
- We analyse an API and suggest tasks and post-study questions.
Many API owners prefer to provide the tasks themselves, because they know their APIs and know the most common use cases. But since we don’t have any particular customers for reviewing Soundcloud API, we had to prepare tasks and post-study questions ourselves.
We hadn’t worked with Soundcloud API before, so we had to learn the API first: how it works, whether there are some SDKs, what the common possibilities are, and so on. Finally, we came up with the following tasks:
- Create a simple application that returns list of tracks for an account.
- Upload the track sample1.mp3 to the account.
- Add an ability to get the track sample1.mp3 by your application.
- Update a description of the track sample1.mp3.
- Implement search by word “classic” and print list of the search results.
1.2 Prepare post-study questions
The second step is to prepare post-study questions. We collected a list of the most common post-study questions (you can review it here), but we also added a few new questions:
- Name at least three of the most difficult problems you encountered during task execution and how you solved them (documentation, forums, stackoverflow, help from a friend, …).
- Have you experienced any unexpected errors? If yes, did you find the error messages useful?
- Did you find that some classes or methods behave non-obviously or unexpectedly?
- Name at least three ways we can improve the documentation.
- Name at least three ways we can improve design of the API.
1.3 Define platforms on which the tasks should be done
The third step is to define the platforms on which the tasks should be done.
We decided to work with three developers for this study. So we had two options:
- Involve three developers for a single platform, e.g., Ruby.
We chose the second option because we’re pretty sure that it is more useful to review a single piece of API by three developers than to review three different pieces of API by one developer, because we can get information that is more useful from analyzing the “intersection” of the developers’ problems. Say, for example, three developers report the same problem for a single piece of API; then the problem definitely needs attention.
Therefore, we decided to involve three Ruby developers in the review.
Actually, the first step when we get tasks and questions from a customer is to review the API, understand the tasks and questions, and complete the tasks with our internal team (not participants). From our experience, the latter is absolutely necessary before sending the tasks to developers, because sometimes the tasks are not possible to complete (e.g., we don’t have access to some files or web resources, or we need some critical details from a customer). Therefore, we completed the tasks and ensured that they are doable and ready to be sent to the developers. To make the developers’ life easier, we prepared all the necessary accounts and files: three Gmail accounts, three Soundcloud accounts, and default tracks.
We have several Ruby developers in our team who were available to do the study. They all are real developers with different experience and qualifications. Below is a table that gives a summary of the participants’ experiences and development systems:
|Participant #1||7 years total: 6 years – ASP.NET, 1 year – Ruby/Ruby on Rails||Microsoft Windows 7 x64; JetBrains RubyMine 6.0.3; Ruby 2.0|
|Participant #2||12 years total: PHP, Ruby, Ruby on Rails, SQL, HTML5, CSS3.||Windows 7 x86, Ruby 1.9.3.|
One more point to note: From our experience, it is absolutely necessary to have a moderator during the testing sessions. If a developer encounters some “blocking” problem (e.g., he/she needs some account or mobile numbers, or some software), he/she pauses the work. Then we fix the problem, and then the developer continues the work.
So we sent the tasks to the developers. It was a rare case, but nobody experienced any “blocking” problem.
3. Analyzing results and the actionable recommendations list
Let’s review the results. We got three reports from every developer:
As you can see, each report contains a link to the source code for every task; participants’ notes, remarks, and suggestions; ideas for each task; time spent on each task; and answers to post-study questions. Our next step is to analyze the reports and provide an actionable, prioritized list of recommendations for API enhancements.
Some notes on recommendations: Our recommendations are based on developers’ reports only. We don’t include our personal opinions, ideas, or suggestions. We just analyze the reports and draw conclusions. Again, if three out of three developers report the issue, we should definitely consider it as a problem that should be fixed.
We analysed the results and prepared the following document: recommendations.
I embedded some tables from the document below.
The table below shows the distribution of time spent on a task by each participant:
The following table summarizes the recommendations for improving the developer experience of the API based on problems discovered during the study:
As you can see, despite Soundcloud API’s developer-friendliness, there are some problems of differing importance that probably should be taken in account by the API provider.
Of course, we reviewed only a small portion of the Soundcloud API: the Ruby SDK. To fully analyze the developer experience issues of the Soundcloud API, it is necessary to involve more developers and study more portions of the API. But I’m pretty sure that even this small review highlighted the most serious issues that developers experience with the Ruby SDK.