I'm thrilled to announce Veneto, a suite of API development tools that I've been planning for months. My experience in API development has brought me years of learning and yearning for better tools. It has all culminated in this project, and I'm eager to share it.
This post only serves to announce it to the world. It's hard to establish a reliable timeline, but I'm working to get a v0.1 out the door as quickly as I can. I'm very happy about all of the work that I've put into preparing for this project, though. I'm proud to say that I have a solid path forward and I'm more convinced than ever of my mission, which is something I've never really achieved at this scale.
So what is it?
The goal of this project is to facilitate robust web API design with well-defined schemas and hypermedia (more on that below), and to improve the developer experience of consuming those APIs. There are three areas of focus:
- The Veneto Language: An ergonomic and expressive domain modeling language to define data formats and API flows, which powers:
- The Veneto Test Client: A practical, user-friendly test client that adapts to the specifics of the user's domain model. This will first be implemented as a VSCode extension.
- The Veneto Toolkit: Code generation and maintenance tools to help developers create and use APIs without being encumbered by boilerplate
The first release to come will be the Test Client, initially as a VSCode extension. I aim to use Rust as much as possible, including for the VSCode extension; this is new territory for me, but it's been done before, so I'm stoked to give it a go.
Wow, why do all of that?
I thought you might ask! There's a lot of existing tooling for documenting and working with HTTP APIs, and I've found myself frustrated over the years with their inflexibility. Throughout this post I'll mainly be picking on OpenAPI and Postman, since those are the most popular alternatives in my experience, but I've been around the block during my research for this project and I haven't quite found what I'm looking for anywhere else.
For starters, let's focus on type systems and data schemas. OpenAPI is based on JSON Schema, which gets the job done, but I believe there's lots of room for improvement. Data Types are a first-class citizen in the Veneto language, which includes an IDL and a fully-fledged type system, complete with generics and soon to include sum types and monadic optionals. All of this being in a novel language means that you will be able to easily hash out an entire domain model in your text editor, complete with inline error checking and creature comforts like jumping to definition. Since this is meant to be an entire ecosystem, I've got big plans for the Veneto Toolkit involving client-side deserialization helpers, especially for TypeScript web applications.
That segues nicely into the HTTP portion of the domain model. First, let's talk about hypermedia.
Hypermedia
Hypermedia is a decades-old concept, notably championed in Dr. Roy Fielding's dissertation in which he proposed REST. As mentioned earlier, Veneto is not REST, but I'm a firm believer in the power of hypermedia - it was the initial motivation behind the creation of this project.
Here's a few tidbits from the project's Manifesto. (Right now the project is just me, but the whole notion space is written in this funny nosism in anticipation that someone will join me along the way.)
HTTP is the hypertext transfer protocol, and the World Wide Web was designed for documents that link to other documents. Rendering hypermedia, resources containing complete links to other resources, is immensely powerful and criminally underutilized in APIs. This principle is more controversial to most people, but if you bear with us you’ll see what we mean.
URI templating, the practice of creating request URIs by substituting variables into a template, is abundant among API designers, and we believe it to be inefficient. An API is a conversation, and there’s plenty more benefit to driving that conversation with hypermedia:
- Flexibility: URI templating creates an unnecessary coupling of client and server, which hinders longevity and precludes flexibility.
- Hypermedia simplifies client-side development by forming a powerful abstraction layer.
- Hypermedia provides the added benefit of discoverability as it predefines valid API flows. These are automatic for users of the Test Client, and they save a lot of time and effort in communicating and making decisions for client-side application developers.
- Dynamic hypermedia unlocks a powerful, maintainable, and robust system for synchronizing business logic between client and server without additional coupling or reimplementing the same logic on all clients.
From another page in the Notion space, elaborating on the benefits of using hypermedia:
API Integration
URI templating is messy and cumbersome. To evaluate a URI template, the client must have all of the relevant parameters at the call site, which introduces unnecessary data handling requirements.
A library for consuming a hypermedia API can implement generic methods that handle the relevant use cases with minimal fuss and minimal boilerplate.
Discoverability
By definition, hypermedia representations always contain links to related resources, meaning that the client never has to figure out where it can go from any given point in a request flow. This saves a lot of work, both in setting up a test client and implementing a client application, since the problem of evaluating URI templates based on the current state of the application completely goes away.
Test clients like Postman rely on the user to manually extract relevant data from responses, and create separate request types for each of the possible steps in an API flow. Moreover, they preclude the possibility of using hypermedia because whenever the user clicks an embedded link in a response, it completely resets the context of the application, so that any user-defined variables or configuration are lost. This was one of the original driving forces behind the creation of Veneto.
The same problem exists in application development too: while creating an API integration, all possible flows need to be created by the developer so that URI templates can be evaluated. Hypermedia saves a lot of time and effort by making this problem irrelevant.
Dynamic Hypermedia
In general, client applications have to be built to support all possible functionality, but not all of it is available at any given time. This is the consequence of the application’s business logic. No matter what, this business logic has to be implemented on the server side - be it for security reasons (e.g. the user is not allowed to modify a certain resource) or purely logical ones (e.g. the user cannot delete a resource since another resource depends on it), the server must prevent invalid operations from happening.
This application state must be somehow conveyed to the end user for the sake of a clean user experience - the UI needs to accurately represent what actions are allowed and not allowed, and the end user needs to see an intuitive error message if something goes wrong. Without hypermedia, the developer will often need to duplicate this business logic on the client side. Not only is this a massive waste of time in the development cycle, but it also hinders longevity as the business logic needs to be updated in multiple places if anything ever changes.
Hypermedia can be dynamic to address this. For example, the server can provide a list of possible actions as links to the client, which can have different types or properties. All the developer needs to do is program the client to understand what those properties mean, and then the client is fully prepared to understand all of those possibilities forever, regardless of how they may change on the server side.
These snippets gloss over other benefits of using hypermedia, particularly related to longevity and ease of maintenance, but I hope that gives you at least some idea.
Everything above is extremely abstract, but the longer I've been developing APIs the stronger my faith in it has become. I look forward to providing more concrete examples and use cases soon.
How it fits in
The above section demonstrates some shortcomings with the existing test client alternatives. That alone was originally the motivation for this project, but the scope of it quickly grew to include the domain modeling language, which has special constructs for hypermedia that I can't wait to show off. The Test Client will work directly with the user's domain model to provide a seamless user experience, and the Toolkit will do the same to make development a breeze.
These ideas deserve an ecosystem built for them. With this project, I hope to change the way we design and use APIs for the better.
This is all probably a lot to take in, and the hypermedia aspect in particular may have some of you scratching your heads. It's also quite an ambitious scope (believe me, I've been fighting scope creep this whole time, and this is the best I could do). I've put a lot of effort into distilling exactly what this project is and why; all of this is a work in progress, and I've been constantly refining the project's goals and principles. More discussion on all of this, especially the justification for taking on a project of this magnitude, can be found on the Manifesto.
What happens now?
The project has a public roadmap for all to see. Here's a simpler page that contains a filtered list of objectives for the current release (which at the time of writing is the v0.1, the first pre-release version). Right now I'm working on the language parser, which is an early prerequisite for everything else I plan to accomplish.
If any of this sounds cool to you, feel free to share your thoughts with me at jake@jakeconley.com! If you're interested in joining me along the way, please don't hesitate to contact me. Once the project is more mature, we'll have a better-defined roadmap and some tasks ready to go for any contributor, but for now the best way is to holler at me so we can make a plan.
Thanks so much for reading this, and I hope you share my enthusiasm. This project has been a long time coming, and I'm stoked to start delivering results in earnest. Y'all can also expect to see a devlog on this site detailing my experience along the way.