Introduction of JSON/HTTP interfaces , SOA Enterprise Architecture

Introduction of JSON/HTTP interfaces , SOA Enterprise Architecture

Browser-based applications became more sophisticated, and mechanisms were introduced to write richer, more responsive web pages. These took advantage of the maturing client-side scripting capabilities of browsers, and also their ability to perform background HTTP requests using techniques, such as AJAX, to retrieve data without interrupting the user experience with a page reload.

The web pages were typically requesting data specific to the page via the page’s associated web server. SOAP/HTTP requests typical of SOA were hard to handle in JavaScript, and too heavyweight to send over an often low bandwidth Internet connection. It quickly became popular to make more granular data requests and, if possible, change to the JSON data format native to JavaScript as shown in red in Figure 5.

Figure 5. Fine-grained exposure for rich browser applications

Fine-grained exposure for rich browser applications

Free from the restrictions of the SOAP standards, these interfaces can consider alternative ways to simplify how the “actions” or “operations” to be performed on data were to be represented. In an interesting throwback to some of the earlier roots of the web, the original intent behind HTTP protocol was unearthed. Significant aspects of the HTTP standard were designed around the architectural principles of Representational State Transfer (REST) documented by Roy Fielding in 2000. From this, a more simplistic entity based style of interaction was derived. The interaction style recommends the use of the common HTTP verbs (POST, GET, PUT, DELETE) in a similar way to the common database interaction verbs (create, read, update, delete). The worldwide web recognizes these verbs in a first class way in order to provide implicit benefits such as caching. It also uses the URL path to navigate the relationships between the data entities.

In a rather over simplified example, you could imagine the addition of an item to an order might be performed by an HTTP “POST” to a URL that looked something like the following URL:

The JSON formatted data in the body of the HTTP request looked as follows:

{ "title" : "Romeo and Juliet",
   "note" " "Special Edition",
   "quantity : 1,
   "price" : 9.99 }

Where the URL describes the specific data entity (typically referred to as the “resource”), and the use of the HTTP verb “POST”, meaning it is a “create” operation for a new order item. To carry the same information in SOAP, it might have looked more like the following: HTTP/1.1

<?xml version="1.0"?>
<soap:Envelope xmlns:soap="http://..." soap:encodingStyle="http://...">
  ...SOAP headers...
  <soap:Body xmlns:m="">
      <m:order orderid="123456"
          <m:title>Romeo and Juliet</m:title>
          <m:note>Special Edition</m:note>

These JSON/HTTP based interfaces provide some useful simplifications in comparison to the more increasingly complex SOAP standards that came before them. However, SOAP has a broader set of standards, which can accomplish many things that these interface cannot. They are used by a different audience and not all of those standards are necessary in that space.

There were, at least initially, some limits to the reusability of these new interfaces. Due to the same origin policy implemented by browsers, web page based applications found it difficult (though not impossible) to make HTTP requests to interfaces offered by other companies. This means that the most common initial use of these JSON/HTTP based interfaces was between a company’s web pages and their own enterprise data. However, techniques such as proxies, JSONP, and standards, such as CORS, ease these restrictions, and enabled these interfaces to be re-used more widely and for “web API” to become a reality.

What is a web API?

There is no formal definition for exactly what is meant by “web API”, just as there wasn’t for “web services” before it, but let’s do our best to nail it down.

Simplistically and loosely speaking, a “web API” typically refers to functions and data exposed in the following way:

  • Exposed over HTTP(S)
  • Uses the HTTP protocol “RESTfully”
  • Uses JSON as the data format
  • Is available over the Internet

For anyone creating a new web API today, this is likely where you start. However, this definition is over simplistic on a number of levels:

  • Not all web APIs use JSON: Most APIs use JSON as the data format, but some provide XML as an alternative format, or even exclusively. In theory, anything that HTTP can respond with could be valid. If you include MIME types, this could mean PDF files, for example, though this broader usage is less common.
  • Not all web APIs are public: As we will see in a later section, APIs are not only exposed and used on the public Internet. However, it is fair to say that the Internet usage has driven much of the agreement on style, usage, and the supporting products and frameworks.
  • Not all web APIs use HTTPs’ RESTful properties directly: There are many Internet facing SOAP/HTTP interfaces and it would be hard to deny that these are also web APIs in some form. They are probably less “RESTful” and more heavyweight to use. However, many SOAP/HTTP interfaces have subsequently introduced JSON/HTTP “RESTful” counterparts.
  • Few web APIs are completely RESTful: The use of JSON/HTTP in web APIs means that they are certainly more RESTful than what came before them. They are, therefore, often referred to as “REST” interfaces. However, in reality most only adhere to a subset of the REST recommendations described in the original material on the subject. A common complaint leveled at APIs claiming to be RESTful is that they rarely provide links recommended by the HATEOS approach.
  • HTTPS is strongly recommended: HTTPS is certainly preferable, and many would say mandatory for web APIs. Payloads often contain private data, and credentials used to access the web API are usually confidential.

So, a new, more lightweight protocol and interaction style is available, but this alone does not warrant a revolution in real-time data integration.

What was the trigger for web APIs to come of age?

The significant shift came around 2007 when smartphones with easy to access “app” stores became mainstream. Mobile application (“app”) development became commonplace and accessible to a huge audience of developers. With some notable exceptions, apps can rarely do much on their own. They need to interact with the world of data around them. Developers can write much more powerful apps if they had simple ways to incorporate access the data and functionality provided by other companies.

It was not just demand from mobile app developers, but developers of richer web sites also needed broader and easier access to data. However, mobile typically brought a high volume of new web API consumers. They were not held back by some of the security restrictions that had challenged the use of APIs in browser-based applications. So, you can say that the introduction of web APIs came about from to two key influences: demand and capability.

  • New ways of making money: A new profitable funding model driven by, but not limited to, the prevalence of a new generation of mobile service consumers in the form of applications for phones, tablets, watches, and more, all needing access to live data and function.
  • Matured capability: The standards, technology, and methods for exposing business function have matured by a decade of work on re-usable service exposure. For example, exposure protocols and data formats have been trialed and matured. Exposure gateways for APIs are now available as a first-class component with a well-understood remit.

How are web APIs different from what came before?

It is in the new demand, and its associated funding model, where the big difference lies. The audience for the exposed business functions are beyond the walls of the enterprise. As shown in Figure 6, SOA services were more typically exposed within the enterprise, generally based on a funnel of projects and their respective known requirements. Web APIs are typically exposed externally, to an often unknown and potentially enormous user base for often highly innovative and unexpected uses.

Figure 6. New factors affecting exposure of services outside the enterprise

New factors affecting exposure of services outside the enterprise

If a web API can provide data that is useful to one application, it is probably useful to others. Put those services (or APIs) on the web, and there is suddenly a whole Internet of potential consumers of the web API who can reach out to many previously untapped customer segments. There lies the new funding model. There is an opportunity to monetize these exposed business functions. The web API becomes a new “product” provided by the organization.

The return on investment might be in many different forms:

  • Direct revenue: For example, a web API for purchasing goods.
  • Indirect revenue: For example, earning a commission by providing aggregation services to other providers.
  • Cost saving: For example, apps that enable self-service in order to scale down an expensive customer service.
  • Marketing: For example, putting product information in front of a wider customer base.

What is for certain is that new markets can be reached by crowd-sourcing the innovations of application designers beyond the walls of your enterprise.

How will you cater for this significant change in the requirements for your integration architecture?

How is the exposure componentry different for web APIs?

Based on our earlier definition of web APIs, you can see there are three key aspects that will fundamentally change when you expose external web APIs compared to enterprise services:

  1. Beyond the enterprise: Web API consumers are not part of the enterprise, so they are beyond your direct influence and control.
  2. Numerous consumers: A significantly higher number of potential consumers of your APIs than you have for an enterprise service.
  3. Competitive marketplace: Consumers have alternatives provided by other companies if your web API does not meet their expectations. Within the enterprise with SOA, they may have only had one option.

These key differences result in a huge number of amendments in how you architect and design web APIs. In this tutorial, we will focus primarily on the differences in the architecture. Architecturally, you clearly still need some form of exposure gateway, but the requirements for that gateway have some new elements.

As you will recall from earlier in the tutorial, re-use always comes at a cost. As shown in Figure 7, your exposure capability needs to be extended beyond the core SOA requirements.

Figure 7. New capabilities for exposure APIs outside the enterprise

New capabilities for exposure APIs outside the enterprise

Let’s have a look at two of the primary aspects of those new challenges:

  • Partner management: You now have a potentially enormous pool of mobile application designers that may want to experiment with your web APIs. If you do not make that extremely easy and attractive, your web API “product” will soon be overtaken by your competitors. How do you set up new relationships with these new partners? How do you keep a track on who has access to the functions? You may have external parties with a dependency on your web APIs as a fundamental part of their business. How do you establish and monitor what level of service they need? Partner management needs to be a first-class function provided by the web API exposure components. Due to the volume of potential partners, it needs to be self-service. It also needs to recognize partners and to monitor and control their usage according to their agreed entitlement plans.
  • Security: Clearly, exposing web APIs over a public medium such as the Internet means a whole new level of security concerns, from the various payload borne attacks, such as XML threats to denial of service attacks on throughput or connections. You must also reliably authenticate your partner’s applications to control their service levels effectively.

This increased complexity has resulted in the emergence of what is now known as API Management. Web API Management is an architectural intent rather than an individual component, although there are, of course, products specifically designed for that intent. It enables an organization to simply, yet securely, expose and manage a web API. It combines a more robust and secure gateway with new capabilities relating to partner management.

Figure 8 shows the primary components required to implement API Management, and the typical roles involved.

Figure 8. Example of a component model for API Management

Example of a component model for API Management

All of these roles were loosely present in SOA initiatives in one form or another, but they were often less formally implemented. The public facing nature of web APIs has forced their maturity. The typical set of roles are:

  • API Product Manager: This role establishes marketable web APIs, prepares and administers “plans” for their use, and evaluates the success of the web APIs using historical analytics.
  • API Developer: This role provides the substance behind the web API façade by configuring the connectivity and data mapping through to the backend systems that provide the actual data and function.
  • Application Developer: This role explores the web APIs on offers via an “API Portal”, and signs up to use them via one of the plans defined by the API Product Manager.
  • Operations: This role monitors and manages the web APIs on a day-to-day basis, ensuring they are meeting the service levels defined by the plan.

What is the architectural “substance” behind the API façade?

The web API Gateway is just the façade or exposure point of the web API. It provides none of the actual function or data provided by the web API. This brings us full circle to the integration architecture that has evolved within the enterprise – the growth from siloed applications through point-to-point communications and hub and spoke middleware, to potentially implementing an SOA.

Indeed, there is a strong dependency on how an enterprise has evolved when it comes to deciding where the underlying implementation of a web APIs should come from. Figure 9 shows examples of the most common options:

  1. Re-exposing an existing enterprise service
  2. Exposing a new integration mediated through the hub
  3. Exposing an interface already offered by a provider system
Figure 9. Examples of options for web API implementation

Examples of options for web API implementation

It is tempting to think that re-exposing existing services (Option A in Figure 9) is the most common. After all, SOA has been around for more than a decade, surely most companies will already have a suite of services available. Exposing those will be the most efficient way of monetizing that prior investment in integration. While there are certainly some organizations that have done this, it is less common than might be expected due to any number of the following reasons:

  • Integration maturity: As noted earlier, it is common for services to have only been exposed in specific portions of the business where they were of value. For the other parts of the business, other integration patterns, such as hub and spoke or even point-to-point, may have remained perfectly adequate.
  • Granularity: Since enterprise services were typically created based on known business requirements, they are often fairly coarse-grained, bringing back a relatively large data graph, and perhaps including many child objects. These operations are too heavyweight for the highly responsive applications required on mobile devices, especially taking into count the often variable bandwidth of the device.
  • Security: Where enterprise services perform an updating operation, it is often performed in an enterprise specific way; for example, making assumptions about the trustworthiness of the caller, the safety of the channel, or using internal only mechanisms for security tokens.
  • Relevance: Data and function, that are interesting or marketable inside the company, are often either irrelevant or inappropriate for external exposure. What we are often seeking with web APIs is a completely new market opportunity; a new concept, probably exposing completely different data.

As a result, Option B in Figure 9 is likely to be common and web APIs are implemented using integrations. Perhaps they are re-using components and adapters within the integration hub itself, but not re-using existing services.

Option C is currently rare because the desired web API is almost certainly different than the existing application. Web API gateways typically have a limited data translation capabilities However, once the integration logic goes much beyond basic data mapping and simple aggregation, the web API Gateway is architecturally an inappropriate place for this logic.

Cloud-based API Management

In the current era, where there is a desire to reduce the internal infrastructure footprint, enterprises are looking for capabilities that can be more easily sourced from cloud-based providers. API Management is a good example of this. Both the web API gateway and API Management capability have clear responsibilities so they can be easily separated from the internal architecture as shown in Figure 10. Since the aim is to expose them to external parties, hosting the web APIs from a cloud-based provider makes some sense.

Figure 10. Cloud-based API Management service provider

Cloud-based API Management service provider

Although relevant to any enterprise, this cloud-based API Management particularly suites companies that are “born on the web”. For example, a start-up company that has chosen to have no internal infrastructure and host everything in cloud-based environments. Cloud-based API Management enables them to provide a single unified exposure point for consumers to find their web APIs, regardless of where they are hosted in the cloud.

With this approach, you need to consider the following:

  • Access control: How do you safely provide the cloud-based API Management service access to your internal middleware or indeed operational systems? Clearly, some form of secure connector is required, but how much control are you prepared to put in the hands of an external party?
  • Latency: Consumers of your web APIs will now have to perform two hops over the Internet before they reach your site – one to the web API provider, and then another to your organization. Web APIs tend to be more “chatty”, meaning you often do more calls to achieve the same purpose. As such, a greater portion of end users wait time relates to the latency of the communication. Depending on the global reach of the service and how busy the web API interactions typically are, this extra latency could be significant. API Management solutions often provide caching options to reduce this latency for requests that are appropriate for that pattern.

APIs within the enterprise – the next phase in SOA?

The new style of “RESTful” JSON/HTTP interfaces are more lightweight, well-suited, and supported by modern programming languages and frameworks. The API Management gateway products are increasingly well-matured. Why use all that goodness within the enterprise too? Many now see an internal API Management layer as an alternative, and in some cases, a more effective way of exposing data and function within the enterprise – an extension to the Service Exposure Gateway shown back in Figure 4.

Figure 11 shows a minimalist architectural configuration that enables a single point of administration of both internal and external APIs. This has the added advantage that internal consumers can directly consume the externally exposed APIs without having to route out to the Internet. In reality, enterprises will often still prefer to have separate internal and external API gateways for a more robust separation of concerns.

Figure 11. APIs for internal and external consumers

APIs for internal and external consumers

So for some, APIs have become the next stage in their journey towards a service-oriented architecture within the company as well as the opportunity they present for external exposure.


Published by Admin

Amine HALLAM Talend Expert and Ex-Talend employee with 8 years experience with talend technologies,for more information please contact me here.

Leave a comment

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.