Evolving an API: Programmers are People Too
By popular (e.g. one) demand, here are the notes to the lightning talk I gave at the last Google Open-Source Jam on my experiences in API design.
A User Interface for Programmers
An API is a user interface for programmers. That means you have to develop it like a user-interface: iteratively, adapting to how the users actually use and misuse the interface in real life. In an open-source project we can't afford to run usability experiments, hiding behind one-way mirrors observing how users work with our software. Instead we have to react to explicit user feedback, bug reports and feature requests on our project's mailing lists, forums and issue trackers. Because programmers are passive aggressive and love to moan about things, they'll post negative feedback on their blogs instead of discussing it on the mailing lists. Google Blog Search is an excellent tool for finding out what programmers really think about your API.
But... programmers focus on solutions, not goals, so their feature requests will be worded in terms of how they think they would change the API to support whatever it is they want to do. You have to engage in a lot of arduous back-and-forth to find out what they are trying to achieve, rather than what they think they would change in your code. Often you (and they) discover that their goal can be achieved with the current API but they haven't noticed because of their solution myopia. If it turns out your API does not do what they want, you will be better placed to work out the correct way of extending the API or push back on the request because it would violate the API's conceptual integrity.
Conceptual Guide
An API encapsulates a conceptual model for representing and solving problems in the API's application domain. It guides programmers towards what the API designer considers to be the "right" way of doing things and away from the "wrong" way.
But... although programmers know there are no silver bullets, they love their golden hammers! They've learned your API and now they want to apply it to problems that are unrelated or only vaguely related. They'll submit bug reports, feature requests and patches aplenty as they encounter difficulties. To steal a colleague's joke: "if all you have is a hammer, everything looks like a thumb". You have to be willing to push back on requests that would violate the conceptual integrity of your API, otherwise it turns into an inconsistent mess that cannot communicate any useful understanding about the problem domain. It seems a strange idea to some programmers, but just because they use your API doesn't mean that they cannot not use another API at the same time.
System of Names
A clear and consistent system of names help people understand the conceptual model. For an API, the names should be chosen to make the calling code as readable as possible, even if it makes the implementation of the API less readable. It's really useful to discuss naming ideas on the mailing lists before adding new features to find out if what you think is easy to understand actually confuses people.
But... people often don't take notice of names or understand their meaning, especially when they speak a different language from the API implementors. I wrote an example class in Java that was called UnsafeHackConcreteClassImposteriser. I'd have thought that the phrase "unsafe hack" in the name was enough to warn programmers away from using it as anything more than an example, but apparently not. Someone still used it in their project and complained that it didn't work exactly as they wanted.
Extensibility
Users will always want to adapt your (object-oriented) API to their needs. If you don't define clear plug-in points they will adapt your API by inheriting your classes and overriding internal methods. That is brittle: it will increase your support overhead or reduce your ability to change the code. Define plug-in interfaces to be focused on one thing, so that they remain stable. For example, the Hamcrest Matcher interface has only two methods and has not significantly changed since it was first defined over 8 years ago. To ensure that code that uses the API is as readable as possible, plug-in points should be seamless: code that plugs user defined objects into the API should not look different than code that uses the built-in objects.
But... if you provide extension points, programmers will want you to maintain their extensions for them. Your users will contribute loads obscure extensions that make sense in their projects but don't have a wide applicability. If you adopt every contributed extension you will end up with an enormous maintenance overhead. Therefore, find out which are the most popular plug-in points and keenest users. Spin off libraries of extensions as external projects and get your keen users to maintain them. E.g. the Hamcrest matchers used to be part of the jMock project but now they're maintained by Neil and Joe. Thanks chaps!
Diagnostics
Error messages are part of the API. Take notice of the errors that confuse users the most (using the mailing lists, issue tracker and blog searches) and improve error messages to reduce confusion. This will reduce the support load in the long term.
But... there is a tension between error diagnostics and generalisation. The more generic and pattern-tastic is your implementation, the harder it is to determine what the user was trying to do that caused an error, and so the harder it is to generate error messages that make sense to the user. Sometimes you have to sacrifice generality for better error messages.
Examples
To save time, make documentation extensible so that you can grow it piecemeal without it obviously being under construction. It's easiest to generate reference documentation with Javadoc.
But... programmers don't always want reference documentation. Javadocs are not useful for people trying to learn an API. Programmers want canned solutions and concrete examples that they can copy into their project and adapt to their needs. The best form of documentation for both you the implementor and your users is HOWTO documents, cookbook recipes and FAQs. The more flexible and extensible the API, the more useful recipes and HOWTOs are because applying the API may require some lateral thinking to understand how the features of the API are used to solve problems.