Aaron Francis

Learn more →

Transcript

Andrew

00:00:01 – 00:00:04

I wasn't thinking we were gonna jump to the hard stuff right off the bat, but

Aaron

00:00:13 – 00:00:14

Hey, Andrew.

Andrew

00:00:14 – 00:00:16

Hey, Aaron. What's going on?

Aaron

00:00:16 – 00:00:23

You know, not too much, except that we have a ton to talk about. So I say we just dive right in today. How does that sound?

Andrew

00:00:23 – 00:00:27

I think that sounds about right. So what do you wanna talk about today?

Aaron

00:00:28 – 00:00:41

I think on our docket is APIs. So we've got how it's done in Laravel, how it's done in Rails, maybe, like, our platonic ideal of how it should be done, all that kind of stuff. So there's a whole bunch to cover.

Andrew

00:00:41 – 00:01:20

Yeah. So I put this one on the docket. I have been working on API stuff recently in Bullet Train, and it ended up going a lot deeper than I expected. And it made me kinda realize, like, there is a lot to unpack here. It also did get me interested in looking at the way that this is sort of done in Laravel or some of the tooling that's available in Laravel and other ecosystems as well because when I'm working on an API, I'm not just concerned with Ruby on Rails people that are using it, so it kinda naturally got me looking at other ecosystems and got me thinking, I wanna unpack this.

Andrew

00:01:21 – 00:01:30

I can talk a little bit about what we have available in Rails, but I wanna hear more how the same problems are being solved in other ecosystems as well.

Aaron

00:01:31 – 00:01:43

Yeah. So I think we have a couple different places we could start. Do we wanna narrow it down between producing APIs and consuming APIs? Because we could talk about both. Do you wanna start with producing maybe?

Andrew

00:01:44 – 00:01:58

Yeah. I think that's a good place to start. So I think the topic really does start very simply. So simply that you could say every Rails application has an API by default. Right?

Andrew

00:01:58 – 00:02:42

Because APIs run over HTTP and every route, every controller that you generate by default in Rails has a HTML response type that's typically rendered out with Herb templates by default, and then we also have a JSON content type that can be returned by those same controllers. And so we have this Vue thing called JBuilder that instead of rendering HTML renders JSON and so we're already done. That's how we do APIs in Rails by default, but there's so much more after that. How does that work in Laravel? Is it the same, you get one by default, or you have to take action to get an API?

Aaron

00:02:43 – 00:03:01

I think we have to take the same amount of action that y'all do, which is very little. You know, by default in Laravel, you're usually returning a view from your controller. So you're saying like, here's the view path and here's the data I want to pass into the view. And then, you know, you go off and render the view. That's by default.

Aaron

00:03:01 – 00:03:20

That's what most people do. You don't have to return a view. You can return something that is either JSON able or to array able. We have a common interface called responsible. And so if something can be turned into a response, you can return that from the controller.

Aaron

00:03:21 – 00:03:49

You know, it really depends on what their front end stack, but I think what most people end up doing is returning a view, and then there's also a way to check to see if the request wants JSON. I think that's what it's actually called. So you can say, like, if request wants JSON, you can return something different than the view. So it sounds like we have the same idea that y'all have. You can switch on whatever the request is requesting.

Andrew

00:03:50 – 00:04:17

Now does that happen automatically when you this kinda gets into a deeper thing with scaffolding or whatever, but when we scaffold a resource in Rails, it gets both of those things by default. So it gets the web view, like the browser view, and it also gets the API, JBuilder, and the JSON response type if that's what the client's asking for. Does it happen automatically in Laravel or you have to add it?

Aaron

00:04:17 – 00:04:56

So I'm not sure I totally understand the question, so I'll answer it as I understand it, and then you can tell me if I was close. When you create a new model in Laravel, you could return that model directly from a controller. So there's no secondary thing that needs to be generated. When you scaffold or generate a new controller, you're left with basically an empty method in the controller, and you would have to add an if statement to see if the request wants JSON and decide what to return if that request wants JSON. So nothing really happens automatically, but you can just return a model directly if you want.

Andrew

00:04:56 – 00:05:08

Yeah. That's interesting. So we have generators. We differentiate between generators and scaffolding. So we can generate a controller, and it's the empty type of controller that you were talking about.

Andrew

00:05:08 – 00:05:33

When we do a scaffold, it's generating the controller, the herb template for the HTML view, the j builder template for the JSON response, and the logic in the controller that says if they requested HTML, render the view. If they requested application JSON, I think is what the content type is. Is that what the content type is?

Aaron

00:05:33 – 00:05:34

Yeah. I think so.

Andrew

00:05:35 – 00:05:59

It then render the Jbuilder template or whatever. Or actually, it does that automatically. So based on the content type, I think it will pick up on whichever template it's supposed to render. But that happens automatically in scaffolding. And that's the difference between our generators, so you can generate a model, you can generate a controller, and scaffolding, which is this very complete version of a generator.

Aaron

00:06:00 – 00:06:09

Yeah. That makes sense. We don't have anything like that first party. I am sure that there are packages. I think there's one called blueprint, which is Jason McCreery's thing.

Aaron

00:06:09 – 00:06:29

I think it does that kind of thing. So I guess ours would not be scaffolding. It would be more generating. And there is one more level of abstraction that you can generate, and that's called a resource. And so while you're able to just return a model from an endpoint, that would be like our active record representation of a model.

Aaron

00:06:29 – 00:07:06

So you can just say user first and return it from an endpoint. You can generate a layer of misdirection called a resource, and then the resource gives you more control over how that is serialized to the consumer. So you would say instead of return user first, you would pass the user into the user resource, and that's a pretty light layer that does more like attribute mapping, loading relationships if they're asked for hiding certain attributes, whatever. So we have those kind of that you can break that association just a little bit.

Andrew

00:07:06 – 00:07:12

That is mind blowing to me. Can you unpack that a little bit more?

Aaron

00:07:12 – 00:07:16

Sure. Yeah. Okay. 1st, tell me which part is mind blowing, and I'll try to hit that.

Andrew

00:07:16 – 00:07:33

The whole, like, include this association Uh-huh. That feels like really batteries included kinda stuff that we only get from I mean, I wasn't thinking we were gonna jump to the hard stuff right off the bat. But yeah. Like, I don't get that. Okay.

Andrew

00:07:33 – 00:07:35

Okay. Cool. Explain more.

Aaron

00:07:35 – 00:07:37

Yeah. I'll build up from the bottom.

Andrew

00:07:37 – 00:07:43

Okay. So build up from the bottom, but the content the response is what? Serialized JSON?

Aaron

00:07:44 – 00:08:09

So from the very bottom, our active record representation is called Eloquent. Those are our models. On an Eloquent model, you can say which attributes are hidden when it comes time to serialize and send to the outside world. So on the user model, you would say, like, hidden equals and then it's an array and you would say, like, password and session token, whatever you want. I don't know.

Aaron

00:08:10 – 00:08:33

Those would be then hidden when you call to array on that model to, like, send it out to the front end. Right? So you have this really, like, crude level of control over what gets sent to the front end. But if you need more fine grained control, what you would do is you would generate a resource. And so basically, it's a one to one mapping.

Aaron

00:08:33 – 00:08:50

Every model would have a resource. You could have multiple resources, but let's just keep it one to 1. So you have a user model and a user resource. And the resource accepts the model as, like, it's I think it's only argument. The constructor accepts the model.

Aaron

00:08:50 – 00:09:13

And then in the resource, you have this dedicated place to decide what is the external representation to the world of this internal model. So if you're mucking around and changing attribute names or doing something, like, totally bizarre with mapping values, you would do all of that in the resource and represent that to the world as your API.

Andrew

00:09:13 – 00:09:20

Erin, clarification. In PHP, when you say array, do arrays have keys?

Aaron

00:09:21 – 00:09:40

When I say array, everything is an array. No. It's not that bad, but it's close. So you could have an array that is numeric indexed, and that's just what I think most people think of as an array. You can also have an array that is string keyed, and that's probably what most people think of as, like, a map, I would think.

Andrew

00:09:40 – 00:09:41

We call them hashes.

Aaron

00:09:41 – 00:09:47

A hash. Perfect. Yes. So in PHP, an array is both. It's either associative or numeric.

Andrew

00:09:48 – 00:09:50

Right. Right. I forgot that. It's been a minute.

Aaron

00:09:51 – 00:09:55

Before I go into relations, does, like, that concept kinda make sense?

Andrew

00:09:55 – 00:10:19

Most frequently in the rails world that is called a serializer. So we have these Totally. But we don't have a first party serialization library. We have a bunch of competing ones and JBuilder isn't really a serializer. I mean, I guess it is in that the net result of it is JSON, but it's not a class.

Andrew

00:10:19 – 00:10:32

It's a view. It's just a different type of view than like an herb template or something like that. Yeah. Which doesn't even feel natural to me. That's new for me because I've been using these serialization libraries for so long.

Andrew

00:10:32 – 00:10:45

But either way, I think generally speaking, whether it's a serializer or like a templating thing that you're doing, I think ultimately, for me, personally, that falls into the view part of MVC.

Aaron

00:10:46 – 00:10:57

Yeah. That makes sense. And so this one that I'm describing is first party. I'll mention the the 3rd party one in a second. But when it comes to relationships, so, like, let's say a user has many blog posts.

Aaron

00:10:57 – 00:11:29

So user has many posts. You can, in this resource, say when the posts relationship has been loaded, here's how I want you to serialize the posts. And so when you pass that user resource out to the person who requested it, it might have a full collection of posts underneath it if the developer has decided to load the posts. So if the post relationship hasn't been loaded, it just goes out totally just a user. Otherwise, it has a collection in it.

Aaron

00:11:30 – 00:11:38

And that gives us a nice way to determine, like, how much should we hydrate this response based on, you know, whatever they requested.

Andrew

00:11:39 – 00:11:57

Wow. That's unlike the way that I've ever seen that problem solved in the rails world. It's never based on has this collection been loaded yet. That's not even a state that I think we really think about. We have these ARL queries that are Sure.

Andrew

00:11:58 – 00:12:08

When you hit dot first or dot all or start iterating over it, then some loading happens. But you don't think about that as being a state that you went into.

Aaron

00:12:09 – 00:12:29

Yeah. So that when loaded thing is an option. So you can say, populate this sub collection when the relationship is loaded. You can also just say populate it always. But it's really helpful in the case where the requestor is in charge of which relationships get loaded.

Aaron

00:12:29 – 00:13:07

You can then eager load those relationships as you pull out the main model and then pass it to a resource, and that will be fully populated. And so it leaves us in control of deciding, like, do I leave it up to the requester? Do I brute force it always? And then this resource is also responsible for determining how a collection of models gets sent out to the front end too. So you have you have the one that's like how is a single user serialized and represented, and then you have how is a collection of users serialized and represented, and that's where you can add meta stuff, links, that kind of thing.

Aaron

00:13:07 – 00:13:17

And so this resource is your thin layer of abstraction between internal model representation and external API representation.

Andrew

00:13:18 – 00:13:42

Alright. So everything that we just talked about is like the table stakes of creating an API. And I think it's, like, fundamentally the not interesting part, Definitely, it's on brand for us to compare and contrast even on the really basic stuff like how do you do it in Laravel? How do we do it in Rails? A lot of similarity there, slightly different approach in some things.

Andrew

00:13:42 – 00:14:22

But I also feel like this is not why I thought APIs would be an interesting topic to talk about. Once you are in this world and you realize I need an API for my application that I've developed, and, yes, I've got these tools from my framework that help me generate an API or create an API or that help you provide an API to your users. It actually kinda opens a Pandora's box, doesn't it? Because right? So in Bullet Train, for years, we had a pretty straightforward approach to this.

Andrew

00:14:22 – 00:15:21

It wasn't necessarily Rails native. We were using a serialization library that we had adopted that was popular at the time, not so maintained now. But we had a version of this and it was very simple, very somewhat naive, didn't handle things like pagination or any type of filtering or Rails was providing by default. But I got to the point where the applications that were being built with bullet train were getting bigger and bigger, and the people that were developing them, it was a stated goal to have a platform esque API interoperability of the software that they were building and the other tools that their customers were using in service of the business or the organization or the government or whatever. So that sent me back to the drawing board.

Andrew

00:15:21 – 00:16:11

This is 2021. The current state of API tools that were available in the Rails ecosystem and what sort of standards and conventions had developed across software platforms. And I have to say it was a bit of a wild journey, and I don't think I actually, on the first go around, I think I got it completely wrong. Now being on the other side of that and having found some resolution in that phase of, like, trying to figure out what the right thing is to do by default for people who are building bullet train applications. Being on the other side of that process now, it made me realize there is a lot to unpack here, and a lot of things that people can think about.

Andrew

00:16:11 – 00:16:31

And not saying that there's necessarily right answers, but I wanted us to have the conversation that basically served as a primer for what I wish I had, a discussion that I could have listened to when I was starting that journey reevaluating API design in bullet train back at the beginning of 2021.

Aaron

00:16:32 – 00:16:48

Okay. So we talked a little bit about some of the technical details on my side, but let's talk a little more broadly, maybe more generally. So you mentioned it being the wild, wild west. I think there are a few standards as standards go. Of course, there are a few of them.

Aaron

00:16:48 – 00:17:09

We should probably talk about JSON API, OpenAPI, that kind of stuff. I feel like versioning is always a really interesting one. And then maybe some stuff like producing client libraries or API documentation generators, that kind of stuff. So do any of those pique your interest and you wanna start with any of those?

Andrew

00:17:09 – 00:17:12

Yes. All of them. I can't

Aaron

00:17:12 – 00:17:14

start with all of them.

Andrew

00:17:16 – 00:17:36

That's the reality. Right? It's a huge surface area of things. As soon as you get to that point, there's a huge number of things that you have to think about and start making decisions about. And it's actually really a difficult situation to be in because once you make a decision and expose it to your users, you are no longer in control of that interface.

Andrew

00:17:36 – 00:17:48

Like, you need to maintain that interface. The expectation is that there's backwards compatibility and all of that. So the stakes are so much higher for API design. Right?

Aaron

00:17:49 – 00:18:09

Yeah. This is why I kind of argue for put it off as long as you can. And then when you do need to do it, make the surface area as small as you can. So by putting it off as long as you can, you'll know why people need the API, and you'll be able to design it most correctly. It still won't be absolutely correct.

Aaron

00:18:10 – 00:18:25

And then make the surface area as small as you can because once you put it out in the world and people start consuming it, like you said, you're stuck. So that's my argument for, like, just keep that surface area small for as long as possible.

Andrew

00:18:27 – 00:19:15

Yeah. I don't disagree with that. So you've heard me say before, I think every web application should have a REST API and webhooks. But maybe that can be stated differently that you should aspire to provide a REST API and webhooks to users when they have found your software useful, make that available to them, tell them that you want to do that so that they can use your piece of software. It's really great at the thing that it does but they can connect it and glue it whether it's via Zapier or automation like no code tools or their own scripts that they're writing in any language, their little glue code packages, putting these things together.

Andrew

00:19:15 – 00:19:41

I think that's the ideal that your customers wouldn't be locked into your platform, but they can use it as a building block with other tools that they're using. Yeah. I can't disagree with what you said in terms of stretching yourself too thin, especially if you're doing it the way that you and I would recommend people do it. Default to bootstrapping, default to staying as lean as possible, I think you might be on to something there.

Aaron

00:19:42 – 00:20:05

So I think with that framework in mind, the next question is how do you keep your API forwards compatible? Like, how do you make it as flexible as possible from the beginning so that when you do have to make changes, they're not breaking changes, which would cause you to version? And I have a few ideas here, but I'd be curious what you think.

Andrew

00:20:05 – 00:20:50

I think for me, the simple second step to you have an API because of the tooling that your framework provides. The second step is introduce some mechanism for versioning so that you know the thing that you're giving to your customers now, you have a reasonable path forward for both breaking that interface and maintaining backwards compatibility. And it's not a trivial problem to solve, but I think it's probably the first one so that you're mitigating that risk that exists when you first make this API available to users and potentially need to scrap everything you've done and rethink it 6 months down the line. What were your thoughts?

Aaron

00:20:51 – 00:21:41

So I have two thoughts for I don't know if forward compatibility is a thing, so we'll call it flexibility. My two thoughts are, 1, if you are returning a collection of resources, so if you're just, like, hitting the user's endpoint to get users back, I think it is probably most flexible to nest that under a top level key. So instead of just returning users as a bare JSON array, you would return it as, like, data or records or users, and that gives you the future possibility to put some meta information at that top level. Because if you just return the array, you're host. Like, you can only ever return a JSON array, and you can't sneak in stuff like next page or previous page or stuff like that.

Aaron

00:21:41 – 00:21:43

Does that make sense?

Andrew

00:21:43 – 00:22:24

It does. We specifically don't do that in bullet train applications because Rails doesn't do it by default in Rails. So even though the structure of our API controllers is different because we're trying to solve for problems like versioning and stuff like that, which we can talk about, The payload that we return is just a bare array of resources, and for pagination, we actually put that information in the headers. So that's in the response headers because the response headers provide a good place as well to do things like returning a link header with rel next and rel previous or whatever. For us, it's always rel next.

Andrew

00:22:24 – 00:22:26

Like, we're doing cursor based pagination, so

Aaron

00:22:27 – 00:22:27

Mhmm.

Andrew

00:22:27 – 00:23:05

It's just, like, forward, forward, forward. And then also as a easier to parse way of indicating to people that these results are paginated, we also provide just the ID of the last object that's returned so that they can pass that in as a after equals this ID and they can get the next page of results. So it makes it a little easier for developers that are maybe interacting with the API directly to compose the next URL that they need or whatever if they don't wanna parse that header, but that's how we solve that.

Aaron

00:23:06 – 00:23:30

That's interesting. I don't know how I feel about putting all the meta in the response headers. I don't think I'm against it, but I don't know how I feel about it. But the pagination thing was my next point. So the 2 broadly accepted forms of pagination are limit offset, which comes with its own problems, and cursor based, which is basically like show me all the ones after this record.

Aaron

00:23:31 – 00:24:17

For flexibility and forward compatibility, I would say even if you're doing offset limit pagination, you should encode that into the response as a cursor. So here's what I mean. If you're gonna send out a response of users and it's 0 through a100 and you're gonna do offset limit pagination, and so you wanna say the offset is a 100. I don't think you should send out offset a 100. I think you should send out cursor and then encode offset 100 so that if you do ever switch to cursor based pagination, everyone who has implemented this already, their clients are still going to work.

Aaron

00:24:18 – 00:24:19

Does that make sense?

Andrew

00:24:20 – 00:24:37

I don't have an opinion about that. But we are on this beautiful journey right now, and there are many little side quests that we can go on and talk about things. Can you just give people a quick primer on offset versus cursor based pagination and the pros and cons of both?

Aaron

00:24:37 – 00:25:01

So the easiest way to do pagination is to say, give me records 0 through a 100, and then on the next page, give me 101 through 200. And you just skip the first 100 and take the next 100. That's offset limit. It is super easy to implement because you just need to know what page are you on and how many per page are you showing. It has a lot of problems.

Aaron

00:25:02 – 00:25:15

One is that it's really expensive. The deeper you get, the database is just, like, throwing away a bunch of stuff. That's fine. The bigger one is you can potentially miss a bunch of records

Andrew

00:25:16 – 00:25:16

Yep. While

Aaron

00:25:16 – 00:25:59

you're paginating through responses. So if the order is shifting around and on a live application, it always is. If the order is shifting around, then when you get to page 3 or 4 or 5 or even page 2, you might get duplicate records that you've already seen, or you might miss records based on them moving to the 1st page while you are asking for the 2nd page. Cursor based pagination is a little bit different in that you're not saying, I'm on page 5 and there are 100 per page, so I'm gonna start with 500. With cursor based pagination, it's a little bit harder to implement because you have to keep track of a little bit of state.

Aaron

00:25:59 – 00:26:34

So if I send out page 1 of users, I also need to tell you the last ID that I sent out. And then when you ask for page 2, you have to tell me the last ID that you have seen. So every time you're asking for the next page, you have to tell me the last one that you've seen, and then the server can calculate the next one you should see. And typically that's with IDs, but it can also be with, like, created at. Tell me the time stamp of the last one you've seen and I'll give you the one after that.

Andrew

00:26:35 – 00:26:46

Alright. So I think that's enough of a primer for everybody on the pagination thing. I think worth covering while we're there. But generally, the recommendation would be start with cursor based pagination. Right?

Aaron

00:26:46 – 00:27:00

Yes. And if you're not going to, fake it. I think that's my point. If you're not gonna implement cursor based pagination, which is fine, and you're doing limit offset pagination, send out your offset as a cursor.

Andrew

00:27:00 – 00:27:03

Okay. Interesting idea. I've never heard that before.

Aaron

00:27:03 – 00:27:27

Okay. So pagination, very near and dear to my heart. What I don't know a whole lot about and I always find fascinating when it shows up on Hacker News is versioning and keeping old versions alive and versioning in the URL versus the header, like, all of that kind of stuff, I always find really interesting. So what are y'all doing on the bullet train side?

Andrew

00:27:27 – 00:28:03

So for me, I'm very deep in a sort of rails mindset. That's just where I hover, that's where I'm thinking all the time, and so when I look at solving a problem like this, I'm gonna try to solve it with the tools that I have at hand. So for me that was controller namespacing and route namespacing which is probably why I ended up with like forward slash API forward slash v 1, v 2, whatever. That was sort of maybe just picked up on that as an old school convention. I think Stripe's URLs have, like, forward slash API v 1 v 2.

Andrew

00:28:03 – 00:28:34

Although, actually, I think their all their API URLs are v 1. I could be wrong about this, but I think all their API URLs are v 1. And then maybe the version is actually sent as a header or something like that. So who knows if that's just a legacy thing. But in bullet train, all of our routes are, like, forward slash API forward slash v one, which means that's different than the rails default, which is that normally your API responses, these JSON responses are coming back from your standard controllers.

Andrew

00:28:34 – 00:29:25

That's not just namespaced in the URL, that's actually the controllers as well. So the controllers are like API colon colon v one colon colon and then whatever the resource controller is. So that's a variation from the default rails. But the reason why that helps, the way that that helps you version your API is that anytime you do this very sparingly but if you have a real break in the interface of your API, not like you added more stuff, stuff that actually breaks what was already there, you have the ability to wholesale copy your controller's directory from v 1 to v 2, go through, update them all with v 2. That's a crazy amount of code duplication.

Andrew

00:29:25 – 00:30:03

Right? But what it ensures is that the v one controllers are locked in place. And I don't just copy the controller's directory, I also copy the controller tests. So all of the API tests that we have for v one now are locked in place, and they will always let me know if the v one version of the API's surface area broke as a result of other things that aren't versioned. So all the serializers or JBuilder templates in our case, all of the controllers, all of the tests are all locked.

Andrew

00:30:03 – 00:30:43

But you still have other dependencies like models that are unversioned and can result in a break in the v one API, but the tests will let you know. And so then what the controllers provide is a place where you can go in and massage the response so that it maintains backwards compatibility. Even if your domain model has changed, you can massage the response so that you're still providing a valid response according to what you promised people in v one of your API. I think that's probably the simplest way to solve that problem, just using the tools on hand in the Rails stack.

Aaron

00:30:43 – 00:31:23

Okay. That makes me feel good because that's exactly how I would do it, and that's exactly what I think would probably be easiest to maintain. I really like the copying of the tests or, I guess, the leaving alone of the v one test, whatever you call it, because that does ensure that your external contract stays the same. And when you do rename model attributes or whatever, your tests will break and you can, in our case, change that in the v one resource or whatever. I also don't have strong opinions, but lean slightly towards the URL saying v 1 and v 2.

Aaron

00:31:23 – 00:31:58

I know there's probably a better way to do or a different way to do that with headers, but I really like the clarity and the explicitness of having that in the URL. And then that does make it easy for us as well to translate that that directly into controllers and controller paths and everything. So we would literally also have a v 1, v 2 folder. And I just like how explicit and boring that is. That feels really easy to reason about, and I oftentimes will choose the easiest thing to reason about.

Aaron

00:31:58 – 00:32:09

So that whole explanation makes a ton of sense. You said that there is another way that y'all aren't doing, but you have seen be successful elsewhere. So what is that way?

Andrew

00:32:09 – 00:32:55

Yeah. So an alternative that I'm aware is out there that people do and have success with is using some type of middleware. So just allow your API to change internally, still have some mechanism for verifying that the responses aren't changing over versions, and where they are, implement the massaging as part of some type of middleware that is external to the application, I guess. But possibly on very large teams this would be valuable because you could give that responsibility to somebody different. Then all of a sudden people can do whatever they need to do with the domain model, and with the current version of the API, they can push it into the future very aggressively and delegate the responsibility.

Andrew

00:32:56 – 00:32:59

But in theory, I think you could do the same thing with controllers as well.

Aaron

00:32:59 – 00:33:28

So 2 of the really good APIs I've seen are Stripe, obviously, and I think their versions, if I recall correctly, are date based. They do, like, 2018 0101 is a version endpoint. The other one is I've worked with Vercel's API, and they aggressively version each individual endpoint. And so I'll be working with the files endpoint, and it's like version 16. And you can see it in the URL.

Aaron

00:33:28 – 00:33:56

It's like files v 16 upload or whatever. And then you're on the site's API, and it's v 4. And so that was a little interesting to me because you're not versioning the whole API, which I guess gives you flexibility to push certain endpoints forward without feeling like, okay. This is our one version that we're gonna do for the next 2 years. What are all the changes we need to batch into it?

Aaron

00:33:56 – 00:34:04

And so I don't have enough experience to, like, have a strong feeling on that, but I did find it really interesting because I don't see that very often.

Andrew

00:34:04 – 00:34:07

Yeah. Same. I don't have an opinion about that.

Aaron

00:34:07 – 00:34:39

Something that I don't have an opinion on is these API specs. So I know of at least 2 big ones. There's OpenAPI, JSON API. I don't remember if Swagger's specifically a documentation thing, but I remember a long time ago, years years ago, going to a conference called Glucon, and they were talking about how API specifications are the future and every app will be self documenting. And I just I don't see it, and I really just don't have an opinion on it.

Aaron

00:34:39 – 00:34:41

Is that something that you care about?

Andrew

00:34:41 – 00:34:58

Yes. I care about this deeply because, for this reason, everything that we've talked about up until this point still does not deliver an API to customers that you built it for. Right? So you now have controllers. You have your versioning sorted out.

Andrew

00:34:58 – 00:35:22

You know how you're gonna do pagination. At this point, you still have to get on the phone with somebody and explain to them how to use your API. There is a whole other thing that happens, and the frameworks also aren't helping with that. Even though we've used a lot of their tooling to build the API, you haven't helped anybody yet because they don't know how to interface with it.

Aaron

00:35:22 – 00:35:27

Listen, I'm not getting on the phone to describe an API to anybody. So what's your what's your solution here?

Andrew

00:35:27 – 00:35:41

Okay. So you touched on a couple of things. You touched on Swagger and OpenAPI, and those are related. So OpenAPI is the successor to what was originally called Swagger. And Gotcha.

Andrew

00:35:41 – 00:36:17

That is effectively like a manifest, a machine readable format, and a standard for describing your endpoints, the request structure, the response structure, the headers, all of that descriptions. And I would say the primary benefit, there's 2 benefits that people get out of OpenAPI. We're up at I think version 3.1 now. So OpenAPI 3.1 document is what it's called or documents. 1 is being able to auto generate human readable documentation.

Andrew

00:36:18 – 00:37:09

So 2 services off the top of my head that provide that, and there may be open source scripts and things like that as well, but the 2 hosted services for this that I think of are Readocly is the one that I use and prefer, and then there's readme.com is another version of this. And they are able to provide more than just, oh, let's tell people about your API. They can add additional tooling to it like, okay, well, if we have your API keys, you can do like example request and response, you can play with the API in the browser, all of this stuff. So that is the benefit of OpenAPI, the successor of Swagger. And then the other thing that you mentioned was JSON API, which is not related and technically not mutually exclusive of using OpenAPI.

Andrew

00:37:10 – 00:37:48

The other benefit that you get from having OpenAPI, the other type of generation that in theory you should be able to do, And actually it's not in theory, you literally can do this. I don't necessarily love the results as a Rails developer and I don't think you'd be really happy with them as a Laravel developer. But in theory, you can auto generate API clients for every platform. So you can use this document to generate the code that is like the Ruby gem for interfacing with your API or the PHP composer package. Did I get that right?

Andrew

00:37:48 – 00:38:21

You got it. So the PHP composer package that will help people interface with your API. You should be able to auto generate those things. In my experience, the results aren't great and they feel like those packages were designed by developers that came from other ecosystems where it's like maybe that's how you think about resources. But in our worlds where there's just this really heavy batteries included model layer, like a very robust domain model, the results leave a little something to be desired.

Andrew

00:38:21 – 00:38:33

And there's other tooling available both in Laravel and Rails that does a better job, I think, and that's active resource. But in theory, that's something that you should be able to do with OpenAPI.

Aaron

00:38:34 – 00:38:48

Okay. So that tells you how much I know about these things because I didn't know OpenAPI was the successor to Swagger and is different than JSON API. So tell me, what is JSON API if it's not mutually exclusive to OpenAPI?

Andrew

00:38:49 – 00:39:02

Alright. So real briefly on JSON API. I don't wanna throw any shade here or anything. There are people who really love JSON API. We tried to live with it for a year.

Andrew

00:39:03 – 00:40:17

It was the original approach that we took for the new version of Bullet Train's API, And the reality was like the idea of JSON API is that you think of our earlier conversations that we had where you're like having to figure out a whole bunch of things like how are you going to request that you want that collection or how are you going to do pagination, how are you going to do nested resources, all of this stuff. The idea was that if you came up with a standard for how to structure that stuff in the payloads and the headers of your API, then you could create automated tooling that would make it easier to interact with APIs. You could also have auto generated client libraries, things like that. I think in practice, and some people had warned me people actually whose opinions I really respect bullet train we adopted JSON API in 2021. Sort of hoping to realize the benefits of a lot of pre written client libraries and things like that.

Andrew

00:40:17 – 00:41:04

But we went down that road and we tried to live with it for a bit, and the reality was that the net result of trying to adhere to JSON API's requirements, not suggestions, but, like, really, you're not doing the thing unless you structure it this way, resulted in APIs that when you were just looking at them naturally without a bunch of tooling support were really, really ugly, hard to look at, hard to parse, hard to work with. Users really gave us feedback. Not just bullet train users, but people using APIs that were built with bullet train. What is wrong with your API? And then you say to them, oh, that's because it's JSON API.

Andrew

00:41:04 – 00:41:47

We're gonna get all these benefits. Now all of a sudden, there's an intermediary between you and your customers, which is the health and strength of the JSON API tooling ecosystem. And if they happen to be on a platform where the JSON API tooling is stale or not maintained or not exciting or doesn't actually solve the problem very well, all of a sudden you now have an API that's a pain in the butt for people to work with. And so ultimately, at the end of the day, after living with it for about a year, we bumped our API version from v 1 to v 2, and said we're just going back to what Rails does normally. And we'll cross each of the bridges as we come to them.

Andrew

00:41:48 – 00:42:15

And the minute we did that, our open API 3.1 document got so much simpler, our API docs got so much easier to understand, and we leaned into a specification that provides a certain amount of benefit, but then decided to effectively go ad hoc on the other side and do something that looks a lot more like what you see in a lot of Rails applications.

Aaron

00:42:16 – 00:42:54

This is exactly how I feel about the auto generated stuff. Like, when you were talking about the clients that can be auto generated from JSON API, I'm just thinking, like, I've seen those before, and they are a thin, dumb wrapper around an HTTP client. And it's like, as someone who is developing an API, I don't want that to be the experience of the consumer. Like, I wanna give them a bunch of the built in developer experience niceties and not just be like, oh, yeah. The user's method just hits the user's end point.

Aaron

00:42:54 – 00:43:20

Of course, it does. Do something interesting for me. And I don't know if it's because I feel like I'm a special snowflake, but those strict standards have always felt to me like this is the bare minimum, and it leaves a lot to be desired from the experience point of view. It might technically work, but from the experience point of view, I don't wanna ship this auto generated client personally. So, yeah, that doesn't surprise me.

Aaron

00:43:20 – 00:43:32

I'm glad the open API still works for generating documentation. I have felt in the past, auto generated docs also leave something to be desired, but I haven't dived in too much there.

Andrew

00:43:33 – 00:44:14

You know, your comments about the promise of auto generated client libraries, that to me is still something that I think we really need. I have heard about how this problem is solved at some really large companies that are well known for providing best of breed APIs and platform support on every platform. For me, that is the end game of all of my desire for APIs to be available. It was never about the developer of the application. It's always been about the consumer of the application.

Andrew

00:44:14 – 00:44:54

A developer developing an application doesn't need an API. It's all about the customer's customer, you know, in the bullet train world. The customer's customer's experience working with this app. Because if it's delightful to interact with the platform, that means that we gave somebody a tool that is a better tool to build a business on because it has built in customer satisfaction if we have gone to the nth degree to make the customer's customer's experience fantastic. That's a goal across the entire framework, but when it comes to APIs, I think there is an ability to do this.

Andrew

00:44:55 – 00:45:34

For us, it is not gonna be based on a standard or a convention that is broadly accepted. It's going to be based on the fact that we know what the structure of bullet train applications looks like, the way that the name spacing is done, the way that the API works. And so we have an ability to use another tool that I would actually encourage everybody to mess around with, and that's active resource. And when I say that, everybody should mess around with it. That's because, yes, we have active resource in Rails, but it has been mimicked and ported to other platforms as well.

Andrew

00:45:34 – 00:45:50

So Shopify has a active resource port for Python that I believe their API client is based on. There is a PHP version of it called Trucker which doesn't have a lot of recent activity, but it looks sick.

Aaron

00:45:51 – 00:45:53

Interesting. I've never heard of that.

Andrew

00:45:53 – 00:46:49

And the reason I am aware of this is because I've been looking at it recently because we want to solve this problem for people that are building with bullet train. But here's the gist, active resource allows you to wrap a well structured REST API, what we typically think of as being a REST API, and interface with it and interact with it much the same way you would your own domain model within your Rails application or your Laravel application. Right? So when you think of doing like team dot findone in rails or team dot users in your API client, that's what Stripe does so well. Their API clients are like charge dot retrieve whatever, the ID, or invoice.charges.

Andrew

00:46:51 – 00:47:09

And you can see all the charge attempts that have been made to service that invoice. That sort of native object oriented interfacing with the domain model, that's what I feel like interacting with APIs should be like, and it's possible using this active resource tooling.

Aaron

00:47:09 – 00:47:28

That's interesting. I've never heard of active resource. I've never heard of Trucker. There is a package in our ecosystem called saloon that's really good and is useful for wrapping up HTTP calls like that and providing it with a high level kinda domain e interface, but I don't think it's explicitly active resource.

Andrew

00:47:28 – 00:47:35

Take a look at Trucker real quick. Look at the read me. It's incredible. Tell me you don't look at this and love it.

Aaron

00:47:35 – 00:47:52

It really does look like it's an eloquent model, which is our active record implementation, an eloquent model backed by an API with all the same methods. So, yeah, this is really interesting. The second thing that stands out to me is the last commit was 8 years ago.

Andrew

00:47:54 – 00:48:15

Yes. That is a problem. So that's a problem. That's a legitimate problem. The real magic of this one to me in active resource for rails is when you're like team dot projects dot new, and then you do a dot save on the resulting object of that, and it just posts to the API and shows up in the app.

Aaron

00:48:15 – 00:48:17

Yeah. That's pretty compelling.

Andrew

00:48:17 – 00:49:02

It is compelling. So the journey that we're on right now in the bullet train world is working with contractors like a Laravel contractor, a Python contractor, and a Node. Js contractor to use these libraries to do the same thing that we've done in Active Resource for Rails, which is based on the open API doc, but not the standard, just information that we can glean from the open API doc that we auto generate for people in Bullet Train, go and create a client library for Ruby, for Laravel, for Python, for Node. Js. And when somebody pushes up to CI and does a release, if that document has changed, regenerate the clients if that's the right way to do it.

Andrew

00:49:02 – 00:49:24

There's some technical discussion about how to solve this problem, but push a new release of those 2 Ruby Gems, 2 Composer, 2 Pip, 2 NPM, and do basically what we all know Stripe is doing, but we all suspect Stripe is not using some open tooling for this. This is something that they do internally so well.

Aaron

00:49:24 – 00:49:25

Yeah. Totally.

Andrew

00:49:25 – 00:49:49

But bring that level of developer experience to the masses so that we can also provide this high level so that we can attain to that Stripe esque platform API so that all of us can ship better software for the world that people can use to do things in real life and put them all together and just sorta elevate the level that everybody is doing their work at.

Aaron

00:49:50 – 00:50:14

Yeah. That's really interesting. I'll be curious to look at the output of the Laravel one because that would be pretty awesome to have that experience. And I think that last thing you said about so everybody can do real work, I feel like that's a topic for another day that I have a lot of opinions on about balancing what is technically correct. Like maybe JSON API is technically correct.

Aaron

00:50:14 – 00:50:55

Balancing that with what's the best experience for people. Maybe JSON API is technically correct. I have a null opinion on that because I've never used it, but I really, really, really care about giving users, however you define that, users the best experience. And I think we can get trapped a lot into holy wars of, like, how do people actually work versus how should they work? And we need to really focus, in my opinion, on how do people actually work, how can we provide them tooling for how they actually work and maybe bring them along to what platonic ideal there is.

Aaron

00:50:55 – 00:51:04

But I think focusing on the developer, the consumer experience is highly underrated and gives you a pretty big competitive advantage.

Andrew

00:51:05 – 00:51:14

Before we wrap this up, I wanna just point out something that I love about one of your products, Hammerstone Refine.

Aaron

00:51:15 – 00:51:18

Tell me. I'd love to hear things you love about my products.

Andrew

00:51:18 – 00:51:24

So if you were to guess, what's the intersection between APIs and Hammerstone Refine? What would you say?

Aaron

00:51:25 – 00:51:34

Yeah. I would say that people are typically looking for collections of models, and they want to give some sort of filtering criteria

Andrew

00:51:44 – 00:52:34

when you look at the ability to filter collections in Stripe via API. It is more limited than what you can do filtering collections in the web interface. Filtering via API, it is complicated, and they do not have, maybe I'm wrong on this, but last time I checked, there is not feature parity between how you can filter in the web interface and by the API. 1 of the beautiful technical details about the way that Refine is implemented is that you have this concept. So that everybody knows Refine is this product that allows you to basically expose filter criteria and then have users specify their filter criteria in your application.

Andrew

00:52:35 – 00:53:23

So bullet train will never build a filtering feature because we just point people to refine and say, go use that, it's worth every penny. This will save you a crazy amount of time, and this is a surface area that we would never want to be part of our framework. We wanna play well with others, so go use refine. When people do that in a bullet train application One of the amazing implementation details of refine is it has this concept of stabilizing the criteria that somebody has specified. So you can save that in the database, you can put it in the URL, you have these different stabilizers that basically convert the structure of their query into all these different formats.

Andrew

00:53:24 – 00:53:58

One of those formats is JSON. Mhmm. So what we have been able to do with APIs built in bullet train on applications that are also using refine, we have been able to allow users to construct that structure as needed in their API integration. This is crazy. There's 2 parts to this because it sounds unallowable, and then it instantly becomes amazing.

Andrew

00:53:59 – 00:54:27

We allow users to construct a refined query in this JSON structure that they can then pass into the API for their automations and specify filter criteria. What that means right off the bat, you have filtering parity between the web interface and the API. It's the same filter. We don't do much better than Stripe, but we're doing this one thing better than Stripe because of Hammerstone Refined. Right?

Andrew

00:54:27 – 00:54:27

Aaron

00:54:28 – 00:54:29

You and me.

Andrew

00:54:29 – 00:54:39

The developer who's actually thinking of that is gonna be like, Yeah, but those structures are gonna be crazy. Right? Yeah. They're not pretty. They're not intended for somebody to, like, look at and construct by hand.

Andrew

00:54:39 – 00:55:26

That's why there's a web interface for them. Here's the beautiful part. By the time we're done, there's gonna be a little button in the web interface for people who have API access that says, you've constructed this query, you know, search for this user by email address in the web interface. Export that to JSON. Now you have it so you can copy and paste it into your API logic, and maybe you programmatically provide the email address or whatever you need to be dynamic about this logic, but you sure as heck did not manually construct and have to figure out what the right structure is for the hammerstone query.

Andrew

00:55:27 – 00:55:32

You built it in the web interface. You exported it, and it's compatible with the API.

Aaron

00:55:33 – 00:55:39

That is really cool. I did not know y'all were working on that. That is amazing. I love that.

Andrew

00:55:39 – 00:55:43

Yeah. So watch out for that. I think it is absolutely next

Aaron

00:55:44 – 00:56:03

level. So we're gonna wrap it here. But if you have been talking back to us the whole time or shouting at your podcast player, please tell us on Twitter. We have been very clear about where we have opinions and where we don't, and so we would love to hear other people's opinions. I feel like that's a way that we all get better.

Aaron

00:56:03 – 00:56:09

So don't be shy. Find us on Twitter. Tell us your thoughts on APIs, and that is it for today.

Andrew

00:56:10 – 00:56:13

Framework Friends is edited by Paul Barr at Peachtree Sound.

Aaron

00:56:14 – 00:56:17

Our intro music was created by Corey Griffin.

Andrew

00:56:17 – 00:56:25

You can find us at frameworkfriends.com. Andrew's on Twitter at Andrew Culver. And Aaron is on Twitter at aarondfrancis.