These are the news items I've curated in my monitoring of the API space that have some relevance to the API definition conversation and I wanted to include in my research. I'm using all of these links to better understand how the space is making APIs available across many regions around the world.19 Apr 2018
I wrote about Werner Vogel of Amazon’s post considering the impact of cloud regions a couple weeks back. I feel that his post captured an aspect of doing business in the cloud that isn’t discussed enough, and one that will continue to drive not just the business of APIs, but also increasingly the politics of APIs. Amidst increasing digital nationalism, and growing regulation of not just the pipes, but also platforms, understanding where your APIs are operating, and what networks you are using will become very important to doing business at a global scale.
It is an area I’m adding to my list of machine readable API definitions I’d like to add to the APIs.json stack. The goal with APIs.json is to provide a single index where we can link to all the essential building blocks of a APIs operations, with OpenAPI being the first URI, which provides a machine readable definition of the surface area of the APIs. Shortly after establishing the APIs.json specification, we also created API Commons, which is designed to be a machine readable specification for describing the licensing applied to an API, in response to the Oracle v Google API copyright case. Beyond that, there hasn’t been many other machine readable resources, beyond some existing API driven solutions used as part of API operations like Github and Twitter. There are other API definitions like Postman Collections and API Blueprint that I reference, but they are in the same silo as OpenAPI operates within.
Most of the resources we link to are still human-centered URLs like documentation, pricing, terms of service, support, and other essential building blocks of API operations. However, the goal is to evolve as many of these as possible towards being more machine readable. I’d like to see pricing, terms of services, and aspects of support become machine readable, allowing them to become more automated and understood not just at discovery, but also at runtime. I’m envisioning that regions should be added to this list of currently human readable building blocks that should eventually become machine readable. I feel like we are going to be needing to make runtime decisions regarding API regions, and we will need major cloud providers like Amazon, Azure, and Google to describe their regions in a machine readable way–something that API providers can reflect in their own API definitions, depending on which regions they operate in.
At the application and client level, we are going to need to be able to quantify, articulate, and switch between different regions depending on the user, type of resources being consumed, and business being conducted. While this can continue being manual for a while, at some point we are going to need it to become machine readable so it can become part of the API discovery, as well as integration layers. I do not know what this machine readable schema will look like, but I’m sure it will be defined based upon what AWS, Azure, and Google are already up to. However, it will quickly need to become a standard that is owned by some governing body, and overseen by the community and not just vendors. I just wanted to plant the seed, and it is something I’m hoping will grow over time, but I’m sure it will take 5-10 years before something takes roots, based upon my experience with OpenAPI, APIs.json, and API Commons.
Werner Vogels shared a great story looking back at 10 years of compartmentalization at AWS, where he talks about the impact Amazon has made on the landscape by allowing for the deployment of resources into different cloud regions, zones, and jurisdictions. I agree with him regarding the significant impact this has had on how we deliver infrastructure, and honestly isn’t something that gets as much recognition and discussion as it should. I think this is partly due to the fact that many companies, organizations, institutions, and governments are still making their way to the cloud, and aren’t far enough in their journeys to be able to sufficiently take advantage of the different availability zones.
In Werner’s piece he focuses on the availability, scalability, and redundancy benefits of operating in different zones. Which I think gets at the technical benefits of this benefit of operating infrastructure in the cloud, but there are also significant business, and even political considerations at play here. As the web matures, the business and political implications of being able to operate precisely within a specific region, zone, and jurisdiction is becoming increasingly important. Sure, you want your API infrastructure to be reliable, redundant, and failover when there has been an outage in a specific region, but increasingly clients are asking for APIs to be delivered close to where business occurs, and regulatory bodies are beginning to mandate that digital business gets done within specific borders as well.
Regions have become a top level priority for Amazon, Azure, and Google. Clearly, they are also becoming a top level priority for their customers who operate within their clouds. It is one of those things I notice evolving across the technology landscape and have felt the need to pay attention to more as I see more activity and chatter. I’ve begun documenting which regions each of the cloud providers are operating in, and have been increasing the number of stories I’m writing about the potential for API providers, as well as API service providers. So it was good to see Werner reflecting on the significant role regions have played in the evolution of the cloud, and backing up what I’m already feeling and seeing across the sector.
While Werner focused on the technical benefits, I think the political, legal, and regulatory benefits will soon dwarf the technical ones. While the web has enjoyed a borderless existence for the last 25 years, I think we are going to start seeing things change in the next decade. Making cloud regions more about maintaining control over your countries digital assets, how you generate tax revenue, and defend the critical digital infrastructure of your nation from your enemies. The cloud providers who are empowering companies, organizations, institutions, and government agencies to securely, but flexibly operate in multiple regions are going to be in a good position. Similarly, the API providers, and service providers who behave in a similar way, delivering API resources in a multi-cloud way, are going to emerge as the strongest players in the API economy.
I’m slowly categorizing all the APIs I find who are offering up some sort regional availability as part of their operations. With the easy of deployment using leading cloud services, it is something I am beginning to see more frequently. However, there is still a wide variety of reasons why an API provider will invest in this aspect of their operations, and I’m looking to understand more about what these motivations are. Sometimes it is because they are serving a global audience, and latency kills the experience, but other times I’m seeing it is more about the maturity of the API provider, and they’ve have such a large user base that they are getting more requests to deliver resources closer to home.
The most recent API provider I have come across who is offering regional API endpoints is from Riot Games, the makers of League of Legends, who offers twelve separate regions for you to chose from, broken down using a variety of regional subdomains. The Riot Games API provides a wealth of meta data around their games, and while they don’t state their reasons for providing regional APIs, I’m guessing it is to make sure the meta data is localized to whichever country their customers are playing in. Reducing an latency across networks, making the overall gaming and supporting application experience as smooth and seamless as possible. Pretty standard reasons for doing regional APIs, and providing a simple example of how you do this at the DNS level.
RIot Games also provides a regional breakdown of the availability of their regional endpoints on their API status page, adding another dimension to the regional API delivery conversation. If you are providing regional APIs, you should be monitoring them, and communicating this to your consumers. This is all pretty standard stuff, but I’m working to document every example of regional APIs I come across as part of my research. I’m considering adding a separate research area to track on the different approaches so I can publish a guide, and supporting white papers when I have enough information organized. All part of my work to understand how the API business operates, and is expanding. Showcasing how the leaders are delivering resources via APIs in a scalable way.
I’ve been deploying two project using AWS API Gateway, Lambda, and Amazon RDS lately. I’ve become so sold on this approach to deploying APIs as part of this work, that I am evolving my own internal API process to use the same approach. The technical aspect of serverless plus the gateway definitely convinced me of the potential, but it was also the usage of AWS IAM which sealed the deal for me. I’m all too aware of how much my API security lacks as a one person shop, something that I also see reflected in my client operations, and I’d rather be offloading security to AWS than ending up taking the hit on it down the road.
While deploying my project using AWS API Gateway, and Lambda, I was faced with the question regarding which zone I should be deploying the APIs in. It is the first time I’ve been faced with the opportunity to deploy my APIs into multiple zones. Sure, I could have deployed my servers into any AWS zone before, but for some reason now that I’m doing with AWS API Gateway, and Lambda, the opportunity seemed more of a possibility. I’ve pitched it to my client to consider an east as well as a west coast API deployment, so that we can give developers the choice in the documentation to choose which availability zone they’d like to use in their application. Before I make the proposal I’m going to deploy some prototypes, and do some benchmark testing, and see what the benefits are.
Even if I end up publishing APIs into separate regions, I still have the backend database to content with. Where do I put the database, and how to I replicate between zones. Amazon RDS gives me the tools to tackle this, but historically I would only do this just for backup, not for actual redundancy, as well as performance gains. Amazon zones have been a staple of the cloud since early days, but I still have many clients who only operate in the east coast region. I’m thinking I will begin to push for a multi-region approach, and see if I can’t convince some of them to start thinking bigger. Getting to know where their customers are, and delivering infrastructure closer where the resources are needed.
I’m rolling out some new APIs as part of my own infrastructure. Most of my APIs are retail, as well as wholesale APIs. Meaning you can use the APIs I’ve published, or I’ll deploy them specifically for you, in your own AWS account. I think I’m going to make east / west an option for my retail APIs, and ALL the AWS regions an option for the wholesale APIs. I’m seeing geographical region as another consideration when I’m thinking about how I should be breaking down my APIs, into smaller more bite-size chunks. I see geographical region as a variable in the host for my APIs, just like I’d add any other variable in the path. Opening up a whole new set of possibilities when it comes to not just API deployment, but also API reliability and performance.
As I continue to drink the AWS Kool-Aid I’m questioning my increased dependence on the platform. However, it gets harder to deny when they bring security to the table, and increased performance, availability, and reliability to my API stack, and the APIs I’m delivering for my clients. There is no way I could afford to do APIs at scale, across multiple regions without AWS. I think back to the day when I would secure my own T1s, colo-facility, and servers. Even with the threat of vendor lock-in, I do not want to ever go back to that world. I enjoy the benefits of AWS. I’m just going to try and keep things as well defined as simple, microservice APIs, so that I can migrate to Google, Azure, or if necessary my own infrastructure. As much as I’d like to reduce my dependencies on major cloud providers, in the current online environment I just can’t. Being able to operate in any region around the globe is a pretty significant benefit to doing business online.
I have a number of friends who worship markets, and love to tell me that we should be allowing them to just work things out. They truly believe in the magical powers of markets, that they are great equalizers, and work out all the worlds problems each day. ALL the folks who tell me this are dudes, with 90% being white dudes. From their privileged vantage point, markets are what brings balance and truth to everything–may the best man win. Survival of the fittest. May the best product win, and all that that delusion.
From my vantage point markets work things out for business leaders. Markets do not work things out for people. Markets don’t care about people with disabilities. Markets don’t see education and healthcare any differently than it sees financial products and commodities–it just works to find the most profit it possibly can. Markets work so diligent and blindly towards this goal, it will even do this to its own detriment, while believers think this is just how things should be–the markets decided.
I see Internet connectivity as a great example of markets working things out. We’ve seen consolidation of network connections into the hands of a few cable and telco giants. These market forces are looking to work things out and squeeze every bit of profit out of it’s networks that it can, completely ignoring the opportunities that are available when the networks operate at scale, and freely operate to protect everyone’s benefits. Instead of paying attention to the bigger picture, these Internet gatekeepers are all about squeezing every nickel they can for every bit of bandwidth that is currently being transmitted over the network.
The markets that are working the Internet out do not care if the bits on the network are from a school, a hospital, or you playing an online game and watching videos–it just wants to meter and throttle them. It may care just enough to understand where it can possible charge more because it is a matter of life or death, or it is your child’s education, so you are willing to pay more, but as far as actually equipping our world with quality Internet–it could care less. Cable providers and telco operators are in the profit making business, using the network that drives the Internet, even at the cost of the future–this is how short sighted markets are.
AT&T, Verizon, and Comcast do not care about the United States remaining competitive in a global environment. They care about profits. AT&T, Verizon, and Comcast do not care about folks in rural areas possessing quality broadband to remain competitive with metropolitan areas. They care about profits. In these games, markets may work things out between big companies, deciding who wins and loses, but markets do not work things out for people who live in rural areas, or depend on Internet for education and healthcare. Markets do not work things out for people, they work things out for businesses, and the handful of people who operate these businesses.
So, when you tell me that I should trust that markets will work things out, you are showing me that you do not care about people. Except for those handful of business owners who are hoping you will some day be in the club with. Markets rarely ever work things out for average people, let alone people of color, with disabilities, and beyond. When you tell me about the magic of markets, you are demonstrating to me that you don’t see these layers of society. Which demonstrates your privilege, your lack of empathy for the humans around you, while also demonstrating how truly sad your life must be, because it is lacking in meaningful interactions with a diverse slice of the life we are living on this amazing planet.
I have been revisiting my earlier work on an API rating system. One area that keeps coming up as I’m working is around the availability of APIs in a variety of regions, and the cloud platforms that are driving them. I have talked about regional availability of APIs for some time now, keeping an eye on how API providers are supporting multiple regions, as well as the expanding world of cloud computing that is powering these regional examples of providing and consuming APIs.
I have been watching Amazon rapidly expand their available regions, as well as Google and Microsoft racing to catch up. But I am starting to see API providers like Digital Ocean providing APIs for getting at geographic region information, and Amazon provides API methods for getting the available regions for Amazon EC2 compute–I will have to check if this is standard across all services. Twilio has regions for their API client, and Runscope has a region API for managing how you run API tests from a variety of regions. The role of geographic regions when it comes to providing APIs, as well as consuming APIs is increasingly part of the conversation when you visit the most mature API platforms, and something that keeps coming up on my radar.
We are still far from the average company being able to easily deploy, deprecate, and migrate APIs seamlessly across cloud providers and geographic regions, but as APIs become smaller and more modular, and cloud providers add more regions, and APIs to support automation around these regions, we will begin to see more decisions being made at deploy and run time regarding where you want to deploy or consume your API resources. To be able to do this we are going to need a lot more data and common schema regarding the what geographic regions are available for deployment, what services operate in which regions, and other key considerations about exactly where our resources should operate. This is why I’m revisiting this work, to see what I can do to get API service providers to share more data from either the API provider or consumer side of the equation.
I am considering adding an area of my research dedicated to API regions, aggregating examples of how geographic regions are playing a role in API operations. I’m thinking region availability will be playing just as significant role as performance, plans, security, reliability, and other areas of the API lifecycle when it comes to deciding where you deploy or consume your APIs. It feels like another one of the aspects of API operations that will overlap with many stops along the API lifecycle–not just deployment. One of the areas of the API lifecycle I’m increasingly thinking about that will affect geographic API decisions is regulations, and how governments are dictating what is acceptable when it comes to the storage, transmission, and access of digital resources. It feels like early notions of what the World Wide Web has been for the last 25 years is about to be blown out of the water, with the influences of digital nationalism, regulation, or even the Internet moving off planet, and increasingly driven by satellite infrastructure.
If you think there is a link I should have listed here feel free to tweet it at me, or submit as a Github issue. Even though I do this full time, I'm still a one person show, and I miss quite a bit, and depend on my network to help me know what is going on.