Why reusable REST APIs are changing the game
REST APIs make our lives easier – but we’re still in the dark ages when it comes to making our APIs general purpose, portable and reusable. Ben Busse shows us the common pitfalls of hand-coding custom REST APIs and the architectural advantages and technical characteristics of reusable REST APIs.
Where I work at DreamFactory, we designed and built some of the very first applications that used web services on Salesforce.com, AWS and Azure. Over the course of ten years, we learned many painful lessons trying to create the perfect RESTful backend for our portfolio of enterprise applications.
When a company decides to start a new application project, the “business” team first defines the business requirements and then a development team builds the actual software. Usually there is a client-side team that designs the application and a server-side team that builds the backend infrastructure. These two teams must work together to develop a REST API that connects the backend data sources to the client application.
SEE ALSO: How to export data to REST
One of the most laborious aspects of the development process is the “interface negotiation” that occurs between these two teams (see figure 1). Project scope and functional requirements often change throughout the project, affecting API and integration requirements. The required collaboration is complex and encumbers the project.
Dungeon master development: Complex mazes of custom, handcrafted APIs
You can get away with slow, tedious interface negotiation if you’re just building one simple application. But what if you need to ship dozens, hundreds or even thousands of API-driven applications for employees, partners and customers? Each application requires a backend, APIs, user management and security, and you’re on a deadline.
Building one-off APIs and a custom backend for each and every new application is untenable. Mobile is forcing companies to confront this reality (or ignore it at their own peril). With the acceptance of BYOD (“bring your own device”) and the proliferation of mobile devices, the modern enterprise may need hundreds or even thousands of mobile applications. Backend integration, custom API development, backend security and testing comprise the lion’s share of a typical enterprise mobile application project (more than half of the time on average).
Most enterprises today are woefully unable to address API complexity at its root cause. Mobile projects typically have new requirements that were not anticipated by the existing REST APIs that are now in production. You could expand the scope of your existing API services, but they are already in production.
So the default option is to create a new REST API for each new project! The API building process continues for each new app with various developers, consultants and contractors. The result is custom, one-off APIs that are highly fragmented, fragile, hard to centrally manage and often insecure. The API dungeon is an ugly maze of complexity (see figure 2).
- Custom, manually coded REST APIs for every new application project, written with different tools and developer frameworks.
- REST APIs are hardwired to different databases and file storage systems.
- REST APIs run on different servers or in the cloud.
- REST APIs have different security mechanisms, credential strategies, user management systems and API parameter names.
- Data access rights are confused, user management is complex and application deployment is cumbersome.
- The system is difficult to manage, impossible to scale and full of security holes.
- API documentation is often non-existent. Often, companies can’t define what all the services do, or even where all of the endpoints are located.
The future: reusable REST APIs
The core mistake with the API dungeon is that development activity starts with business requirements and application design, and then works its way back to server-side data sources and software development. This is the wrong direction.
The best approach is to identify the data sources that need to be API-enabled and then create a comprehensive and reusable REST API platform that supports general-purpose application development (see figure 3).
There are huge benefits to adopting a reusable REST API strategy.
- APIs and documentation are programmatically generated and ready to use.
- There’s no need to keep building server-side software for each new application project.
- Client-side application design is decoupled from security and administration.
- The “interface negotiation” is simplified.
- Development expenses and time to market are dramatically reduced.
- Developers don’t have to learn a different API for each project.
- RESTful services are no longer tied to specific pieces of infrastructure.
- Companies can easily move applications between servers and from development to test to production.
Technical characteristics of a reusable API
This sounds good in theory, but what are the actual technical characteristics of reusable REST APIs? And how should reusable APIs be implemented in practice? The reality is that there’s no obvious way to arrive at this development pattern until you’ve tried many times the wrong way, at which point it’s usually too late.
DreamFactory tackled the API complexity challenge for over a decade, built a reusable API platform internally for our own projects and open sourced the platform for any developer to use. We had to start from scratch many times before hitting on the right design pattern that enables our developers to build applications out of general-purpose interfaces.
There are some basic characteristics that any reusable REST API should have:
- REST API endpoints should be simple and provide parameters to support a wide range of use cases.
- REST API endpoints should be consistently structured for SQL, NoSQL and file stores.
- REST APIs must be designed for high transaction volume, hence simply designed.
- REST APIs should be client-agnostic and work interchangeably well for native mobile, HTML5 mobile and web applications.
A reusable API should have the attributes below to support a wide range of client access patterns:
- Noun-based endpoints and HTTP verbs are highly effective. Noun-based endpoints should be programmatically generated based on the database schema.
- Requests and responses should include JSON or XML with objects, arrays and sub-arrays.
- All HTTP verbs (GET, PUT, DELETE, etc.) need to be implemented for every use case.
- Support for web standards like OAuth, CORS, GZIP and SSL is also important.
It’s crucially important to have a consistent URL structure for accessing any backend data source. The File Storage API should be a subset of the NoSQL API, which should be a subset of the SQL API (see figure 4).
Parameter names should be reused across services where possible. This presents developers with a familiar interface for any data source. The API should include automatically generated, live interactive documentation that allows developers to quickly experiment with different parameters (see figure 5).
In general, the structure of the request URL and associated parameters needs to be very flexible and easy to use, but also comprehensive in scope. Looking at the example below, there is a base server, an API version, the backend database (the API name) and a particular table name in the request URL string. Then the parameters specify a filter with a field name, operator and value. Lastly, an additional order parameter sorts the returned JSON data array (see figure 6).
A huge number of application development scenarios can be implemented just with the filter parameter. This allows any subset of data to be identified and operated on. For example, objects in a particular date range could be loaded into a calendar interface with a filter string (see figure 7).
Complex logical operations should also be supported and the filter string interface needs to protect against SQL injection attacks. Other database-specific features include:
- Pagination and sorting
- Rollback and commit
- Role-based access controls on tables
- Role-based access controls on records
- Stored functions and procedures
A comprehensive reusable REST API should also support operations on arrays of objects, but you can also specify related objects as a URL parameter. This allows complex documents to be downloaded from a SQL database and used immediately as a JSON object. The data can be edited along with the (see figure 8) objects. When committed back to the server, all of the changes are updated including parent, child and junction relationships between multiple tables. This flexibility supports a huge number of very efficient data access patterns.
Some of the special cases that you might want to implement with server-side scripting include:
- Custom business logic
- Workflow triggers
- Formula fields
- Field validation
- Web service orchestration
REST API complexity is an important problem for companies building API-driven applications. The tendency to build new APIs for each new project has negative consequences over time. Adopting a REST API platform strategy with reusable and general-purpose services addresses this problem and provides many benefits in terms of more agile development and quicker time to value.