Aaron Francis

Learn more →

Transcript

Andrew

00:00:00 – 00:00:01

So what are

Andrew

00:00:01 – 00:00:02

we talking about today?

Andrew

00:00:02 – 00:00:03

Let's talk about testing.

Andrew

00:00:03 – 00:00:06

Testing. Okay. Where to start?

Andrew

00:00:16 – 00:00:57

This was an interesting one because I feel like both you and I have a lot of experience with it. We've both worked on our own kind of like tooling and experiments and things like that related to testing. And probably, we could just compare and contrast what the situation is in Laravel, what it is in Rails. And then so much of testing is common in every ecosystem, especially in terms of philosophies and all of that stuff. So it seemed to me we could just go through all of that stuff and try to maybe learn something from each other or figure out where the common ground is or the common experiences stuff we've learned in the course of our career so far?

Andrew

00:00:58 – 00:01:13

Yeah. Absolutely. So I'll give you a brief overview of the Laravel side. I feel like we have a pretty strong testing culture, which may surprise people to hear that about a PHP framework. But, again, Laravel is not PHP of old.

Andrew

00:01:13 – 00:01:37

It's a modern, fully out kind of software development framework. So our culture is pretty good on testing. We also have a lot of first party built in Laravel testing, either helpers or frameworks or stuff like that. So I know that on y'all side, y'all have kind of a split that I wanna hear about between, I actually don't remember, RSpec and maybe Minitest. Is that right?

Andrew

00:01:37 – 00:01:38

Yeah. You've got it.

Andrew

00:01:38 – 00:01:46

Okay. I nailed it. On our side, we don't really have that. Like, we have PHPUnit is our thing. That's it.

Andrew

00:01:46 – 00:02:17

And that's not Laravel specific. That's generic PHP. But then as of recently, there's a new kind of player in the game and that's called Pest. And that is a little bit different than I think y'all's split because Pest actually sits on top of PHPUnit. So if you have a PHPUnit test suite, you can just bring in Pest and gradually change your style to Pest, and Pest is a lot more functional, less boilerplate y.

Andrew

00:02:17 – 00:02:39

And so it's really just like a preference thing, and I haven't yet switched over to Pest. I still just write all my stuff in PHPUnit. But I think that's interesting because it's not a one or the other. You can use both in the same app, unlike, I think, y'all's split between the 2. So ours is very streamlined in terms of what people use.

Andrew

00:02:39 – 00:02:44

It's either PHPUnit, which is probably, like, 90%, or some people layer Pest in on top.

Andrew

00:02:45 – 00:02:49

So where does Laravel Dusk fit into that equation? What's Dusk?

Andrew

00:02:49 – 00:02:50

Yeah.

Andrew

00:02:50 – 00:02:51

How does that relate to PHPUnit?

Andrew

00:02:52 – 00:03:24

So PHPUnit is totally outside of Laravel. Laravel does use it. Pest came out of Laravel, but now is kind of broader. Dusk is Laravel specific and Laravel only, and it is our browser automation, browser testing, integration, systems testing, whatever. So Dask is a first party Laravel package written and maintained by the Laravel team that automates web browsers for acceptance testing, I guess you would call it.

Andrew

00:03:24 – 00:03:29

Your PHPUnit test suite is distinct from your DUSK test suite?

Andrew

00:03:30 – 00:03:43

Interesting question. You would have all your DUSK tests alongside your unit and integration. So, like, when you're running DUSK tests, all of your assertions and all of that stuff is still PHPUnit.

Andrew

00:03:43 – 00:03:44

Right.

Andrew

00:03:44 – 00:03:58

So what Dusk does is it gives you, like, the methods that say go to this page, click this button, assert that you see this test. Right. Assert that you see this text. Yep. And then all of the assertions under the hood are just PHPUnit assertions.

Andrew

00:03:59 – 00:04:14

Yeah. So So that pattern matches for what we have in Rails, which is and you alluded to this, we've got Minitest and we've got Rspec. Those are like 2 testing frameworks. Minitest is the one that is first party in Rails. That's your default.

Andrew

00:04:15 – 00:04:52

And then it seemed like when I first started doing Rails development, I felt like everybody used Rspec. So, it was like the first thing you did was get rid of Minitest and replace it with RSpec. And I think that's less and less now. I've seen more and more people you can find examples of this of people that used to use Rspec but then worked on a project where it was just mini test, and they are like, it's really not a big deal. There's not a ton of value add for using Rspec or whatever.

Andrew

00:04:53 – 00:05:25

I saw somebody say recently, I think I might actually prefer miniTasks. And for me, it's interesting because I was an RSpec person originally for years. And then when I started building bullet train, I sort of thought I should minimize the number of decisions that I'm making to deviate from the rails standard. Like, I should just try to be as rails default as possible. And so, that was the thing that brought me back to mini test was just realizing, like, I want as many people as possible to adopt bullet train.

Andrew

00:05:25 – 00:05:38

And so the fewer barriers that I put in front of people for adoption, the fewer barriers there are to adoption, the better it will be for bullet train. And so I've been on mini tests for the past 4 or 5 years now.

Andrew

00:05:38 – 00:05:54

Why are there 2? Are they fundamentally different? And did the rails team write a new one when our spec was around? Or did it go the other way? Did the rails team write mini test and then somebody was like, like, oh, let's have a generic one.

Andrew

00:05:54 – 00:05:58

I'm gonna write rspec. It just seems it seems odd to me to have 2 in the ecosystem.

Andrew

00:05:59 – 00:06:17

Yeah. So originally, there actually was a little bit of back and forth. So in Rails, we had test unit, which was the original thing. And then there was such a strong testing culture, I think, in Rails. So I recall, like, when Rails was sort of ramping up, I was still writing PHP.

Andrew

00:06:17 – 00:06:44

And there was this emphasis growing on, hey, we should all be doing more testing. And so, in PHP, that was with PHPUnit. And so, we're advocating like, oh, we should be writing more tests and even test driven development and all these philosophies and methodologies and things like that. And there was a ton of that in the rails community. The rails community was very testing forward or whatever you wanna call it.

Andrew

00:06:44 – 00:07:28

And I think that because it was relatively new I'm not even sure if that's accurate because it's been so many years. And I should also say I was very early in my career at that time, but Rails definitely gave off a aura of being sort of testing forward. And I think because a lot of those ideas hadn't solidified yet, there were multiple people taking a stab at testing tooling. And so even back then, it wasn't just Rspec versus test unit or Minitest whenever that kinda came into. I don't even know when Minitest started, but it eventually became adopted as like the rail standard.

Andrew

00:07:29 – 00:07:56

But there were also things like cucumber behavior driven development and like these other ideas that were floating around that haven't really stood the test of time. They're not really still a thing that I hear anybody talking about. I think of Cucumber in particular. But I remember when I was working at my first Rails consultancy job over 10 years ago now, all of these were exciting. So there was RSpec and there was Cucumber and but there was just this huge emphasis on testing.

Andrew

00:07:56 – 00:08:18

And to a degree, test driven development was like a popular philosophy. So I think it was probably just that. It's interesting that there was a first party solution and then people were still proposing, oh, no. Let's take a totally different approach to it. But it was probably just because there was a strong laboratory kind of thing going on where everybody was experimenting with ideas.

Andrew

00:08:19 – 00:08:35

Yeah. That again surprises me that there are these 2 totally viable ways to do it in the rails ecosystem after, you know, all the rails makes the decisions for you kind of thing to have those 2, like, super legit ways and people are kinda torn on which one to use. That surprises me.

Andrew

00:08:35 – 00:08:47

Yeah. So I think the big difference between the 2 from my vantage point is that our spec is like a DSL. So it's this language. It's like this metalanguage or metaprogramming thing on top of

Andrew

00:08:47 – 00:08:52

Y'all love DSLs. That is such a ruby thing to have DSLs for your DSLs.

Andrew

00:08:53 – 00:09:07

Yeah. And I'm not sure I love it. I remember one of the benefits of mini test or whatever. Maybe it was a benefit of test unit back in the day or whatever. But there was something kind of nice about stuff just being classes.

Andrew

00:09:08 – 00:09:08

Mhmm.

Andrew

00:09:08 – 00:09:33

Right? So if you're just classes, you get the benefit. This is actually my biggest beef with DSLs generally is I'm an object oriented programmer. I'm really, really good at creating tooling around either inheritance or composition or whatever, just using those core language constructs. And so when you start going deep into DSLs, I feel like you lose some of the accessibility of some of that object oriented tooling.

Andrew

00:09:33 – 00:10:07

And here's an interesting one. What I have found is as you dig into a lot of the DSLs that exist in the Ruby and the Rails ecosystem, they're all kind of implemented slightly differently. They all implement their own DSL layer of, like, how their DSL is implemented. Depending on how it's implemented, you can get into some really weird situations where, like, I'm putting a debugger in this code to try to understand how to extend something or build a developer tool. Maybe I'm just not great at debugging.

Andrew

00:10:07 – 00:10:17

I think I'm pretty good at debugging though. And, like, I can't figure out I'm in some weird scope. I can't access the variables that I'm expecting. Where's that property? How's this stuff structured?

Andrew

00:10:18 – 00:10:27

And it's because in order to implement their DSL, they've implemented this massive non normative layer of abstraction. And so that's my biggest beef with DSLs.

Andrew

00:10:28 – 00:10:33

You know the meme, the simpleton on the left and the Jedi on the right, the bell curve meme?

Andrew

00:10:33 – 00:10:33

Yeah. Yeah. Yeah.

Andrew

00:10:33 – 00:10:48

But I think that's how I feel about object oriented programming. They're object oriented on both sides. It's like just use classes, and in the middle, it's DSLs and all of that stuff. I don't know if I'm on the left or the right, but I'm definitely team just cheese classes. Yeah.

Andrew

00:10:48 – 00:11:01

I think you're probably on the right now. But I agree. That's basically how I feel about it as well. And so maybe that's why some of those people are coming back and saying, like, this isn't all bad. So, yeah, that's basically, like, the tooling as it is now.

Andrew

00:11:01 – 00:11:14

People are typically using our spec or mini test. And our system tests are, like you said, we have a thing called Capybara. There's, like, the headless browser drivers behind it, but Capybara sounds the same as what Dusk is.

Andrew

00:11:15 – 00:11:20

Yeah. It's kind of the orchestration layer between the headless browser and your app, it sounds

Andrew

00:11:20 – 00:11:21

like. Exactly.

Andrew

00:11:21 – 00:11:24

Is that first party or is that a third party gem?

Andrew

00:11:24 – 00:11:26

No. That's a third party gem.

Andrew

00:11:26 – 00:11:26

Okay.

Andrew

00:11:27 – 00:11:42

It's kind of the only show in town and then where you get into different stuff on the back end. So behind Capybara, and this is probably the same for Dusk, you gotta have Selenium in play. Selenium's there somewhere?

Andrew

00:11:42 – 00:11:47

That is a good question. I don't know who does the driving for Dusk.

Andrew

00:11:48 – 00:12:24

So the default that a lot of people use in Rails these days is Selenium. But we have Caprete, I think is how it's pronounced, is an alternative, and it's Chrome specific. So it interfaces with a different API to interface with Chrome, and apparently, it's supposed to have some speed benefits or whatever. Bullet Train supports both. So we have Selenium, which can do Chrome, Safari, Edge, all these headless browsers on the back end, but it uses kind of like a industry standard interface for communicating with the browsers.

Andrew

00:12:24 – 00:12:38

And then Capri is just Chrome based on some proprietary interface that they've got. And the reality is, like, we implemented it because we thought it would be faster. And really, when all was said and done, it it's kinda the same.

Andrew

00:12:38 – 00:12:58

Yeah. I just looked at ours, and so underneath desk is PHP WebDriver and PHP WebDriver is a Facebook project that is the Selenium bindings for PHP. Yeah. If you get all the way down, it's Selenium, which I think is probably the case for all of these web browser automation.

Andrew

00:12:58 – 00:13:21

Awesome. So this is perfect because I think what it illustrates is that in Laravel, in Rails, we're basically dealing with the same foundations, which is the same on a lot of stuff. A lot server side rendered, whatever, all of that. But the same in testing, like, there is just a lot of commonality. And so I think probably there are a lot of similar lessons that have been learned in those ecosystems.

Andrew

00:13:22 – 00:13:40

And you can quickly kind of eject like, when you're talking about testing stuff, you quickly eject from a, like, I'm speaking as a Rails developer, and you eject to a level where it's like, no, we're just talking about software testing generally, and the principles are largely the same across these different ecosystems.

Andrew

00:13:41 – 00:14:20

Yeah. I think that's right. And I think the implementations differ slightly, and I think we get some benefits I think maybe a lot of benefits from having Dusk be first party because there are certain things that the Laravel team has built in. Like, you can call a method to log in as the browser, and what it does, I think, is it exposes, like, this secret controller that passes off through. So, like, if you're running a test and you wanna test it as an admin instead of going to the login screen and typing in admin with the browser automation, There's, like, a secret method you can use in your testing, and so it's pretty nice to have it so tightly integrated like that.

Andrew

00:14:21 – 00:14:38

But I think you're right. Maybe the more interesting thing is talking about, like, okay, we both have these browser automation tools. So at what level I know this is near and dear to your heart. Like, at what level are we testing? Are we testing at this micro unit level, at this integration level, or at this browser level?

Andrew

00:14:38 – 00:14:42

And, like, where kinda in the stack should we be testing, I guess?

Andrew

00:14:42 – 00:14:49

Yeah. I think that's, like, a whole can of worms to unpack, but what a great can of worms. Let's do it.

Andrew

00:14:49 – 00:14:53

And I know that you have a whole lot of thoughts. You've even written some libraries on it.

Andrew

00:14:53 – 00:15:10

That's true, actually. So I think the thing I wanted to ask you about first was what is the current state of test driven development in the Laravel ecosystem? Is it still considered to be a best practice?

Andrew

00:15:10 – 00:15:26

That is a good question. It's hard to say. I think it's hard for me to get a general survey on the community. I know that Adam Wathan did his TDD course, which influenced a lot of people, myself included. It was a great course, really made the case for it.

Andrew

00:15:26 – 00:15:49

I think the Laravel community in particular is excessively pragmatic. I think it has a lot of roots in, like, bootstrapper ethos. I think the community is less what I would say are, like, architecture astronauts, and I say that with air quotes as a pejorative. Let's just fight about architecture and more let's just build stuff. Yep.

Andrew

00:15:50 – 00:16:21

And so I wouldn't say that TDD is a religion in the Laravel space, but I also wouldn't say that it's totally eschewed. Like, people that like TDD do TDD, and there are a couple of teachers, like our friend, Matthias, does some TDD content for Laravel. And it's well received and people like it and people understand the benefit, But I've also seen a lot of people be like, hey, it doesn't matter if you TDD or not. Just write some tests and whatever works for your style works.

Andrew

00:16:21 – 00:16:38

Yeah. So I think in rails I can't speak for the rails community or anything like that because I think I'm so biased now against TDD generally. Oh, hot take. Yeah. And don't want anybody to get their backs up when I say that because I'm the same as what you just said.

Andrew

00:16:39 – 00:17:03

Everybody just needs to do whatever works for them. And I totally get one of the arguments that comes back when you say like, oh, I don't like TDD. There will be folks that say, I couldn't do some of the object oriented design or whatever. Like, I feel like I couldn't do it without TDD, and TDD helps bring out the right level of abstraction or whatever. Totally fine.

Andrew

00:17:03 – 00:17:22

I think in years past, I might argue against that and say, like, yeah. But I also think the things that TDD requires also ends up creating visible artifacts of, like, you trying to adhere to the system or whatever. And technically, I just made that argument by saying it out loud. I get it. Yep.

Andrew

00:17:22 – 00:17:48

But actually, I don't care. I just don't care anymore because whatever people need to do to do their best work and feel confident with what they've produced, have at it. What I'm against is finger wagging Mhmm. Where we're telling people, like, if you didn't do your work in a certain way, it is therefore a subpar. And in that, again, people will say, wait.

Andrew

00:17:48 – 00:18:27

You're telling people, like, don't write tests or, like, it's okay to ship code without tests? And that's not what I said. What I'm saying is I don't think and I not only do I not think you need to write your tests upfront, I think it may have negative utility. It may actually be less valuable than just quickly iterating at a product level, creating something, shipping it to customers, and then or internal stakeholders or yourself, whoever, whatever. But build the thing, use it, refine it, And then once you know, oh, this is the thing that I want to protect from regression, then write tests.

Andrew

00:18:27 – 00:19:06

Now, you're free to ask yourself at what level should I write those tests? If you're doing TDD, you're gonna have a lot of unit tests because you're driving things from the bottom up. Whereas if you test after, you're sure that this is something that you even want to protect from regression, and you didn't find that you needed to do any intermediate tests just to clear the thing up in your own brain, well, if you wait until the end, now you have the option of being like, wait, what if I just test it from the top down? What if I test it from the top down? And that's where system tests start to be pretty dang appealing?

Andrew

00:19:07 – 00:19:16

Yeah. I'm kinda the same way. I don't adhere personally to TDD. I don't think that there's a problem with it. It just doesn't match kind of what I prefer.

Andrew

00:19:16 – 00:19:57

And what I prefer is to kinda just jam for a little while and see where it goes because oftentimes, I don't know exactly where it's going, and I don't know exactly, honestly, what the feature is supposed to do. Of course, it's context dependent on the app or the project or whatever. For example, I did a stream just maybe yesterday or the day before where I was trying to fix a bug in Fast Paginate, which is a composer package that I have, and I started with a test. I'm not a TDD person, but I knew exactly what the problem was because somebody had opened an issue. And so I started with a test to prove that that was the problem because then that guided me of, like, where I need to go to fix it.

Andrew

00:19:58 – 00:20:15

But I couldn't reproduce it without the test, and so that was a case of, like, okay. Well, this makes perfect sense for TDD because they've told me there's a problem. If I can make a test to prove it, then I'm halfway there. And it turns out writing that test was 90% of the problem because I couldn't reproduce it, and that's just a great use case for TDD.

Andrew

00:20:16 – 00:20:42

Yeah. So it's interesting with system tests, it really depends on what your final product is of the thing that you yourself are shipping. Like sometimes a system test isn't appropriate because you are just shipping a widget for some other developer to compose, take that thing and some other thing and integrate them and whatever. So you might, at that point, like, it's not system tests. It might be whatever you wanna call them, integration tests or unit tests or whatever.

Andrew

00:20:43 – 00:21:12

You're testing at a different level, but you're basically testing around your area of responsibility. And for a lot of us that are full stack developers, the system test is the one that sort of fully encapsulates all of our responsibilities. Everything front end, all of the little opponents like UI components, all of that is our responsibility, and the most comprehensive way to test that is system test.

Andrew

00:21:13 – 00:21:45

I totally agree that it depends on what you're shipping. So, like, this thing that I was streaming is a very low level package whose interface is just code, And so it makes sense for me to do more unit testy things there. On the other hand, with Torchlight, the syntax highlighting API, I don't have a single unit test. I have maybe a 150 integration tests or whatever we're deciding to call them. I have another hot take that I don't really care what we call the tests.

Andrew

00:21:45 – 00:22:12

But I have a 150 tests that verify if I give the API this block of code with these options, I get this HTML back. And however that happens under the hood, I don't really care, but what I want to prove a 100% of the time is that when users give me code, I give them highlighted code back and it's correct. And so for that product, I don't have a single unit test, but I have a whole bunch of tests.

Andrew

00:22:12 – 00:22:43

So one of the places where I think there's still also room for interpretation, I tend to lean into system tests super hard, and I only go lower. I only write integration tests or unit tests, those more isolated tests where I absolutely need to. And so I'll push as much as I can. So for example, the authentication system tests for Bullet Train, it just covers an incredible number of components. It's testing the UI.

Andrew

00:22:43 – 00:23:10

It's testing that our style sheets are compiling, it's testing Devise, it's testing and not at a granular level, but it's ensuring that our Devise integration is working properly. In that system test, some people will say, I just wanna do the happy path. So just do the happy path and then all the other stuff we'll do in unit tests, probably because of speed or whatever. Mhmm. For me, I actually use my system tests to even test as many edge cases.

Andrew

00:23:11 – 00:23:47

If I can get that edge case to present in the browser in some sort of measurable or verifiable way, then I'll use system tests for that as well. Just because I like and enjoy hovering at that level, like, as high level as possible. So in bullet train, the authentication system tests, first thing it does is fails to log in, and then it fails on some other conditions and whatever. And if it was ever a problem that, oh, that's making this test really long, therefore slowing down the entire test suite, then I could break it up. I could do those as separate system tests.

Andrew

00:23:47 – 00:24:13

But I think the thing that really changed the game for me was when I started running my tests in parallel. That was the game changer because now all of a sudden, the only test that I have to worry about is my one longest test. And anything that's less than my one longest test, it really doesn't matter. And I should just do it in the way that is most, sort of productive for me as a developer.

Andrew

00:24:13 – 00:24:51

So that's interesting. You say you only have to worry about your one longest test. We have parallel testing in Laravel, and it is, surprise, surprise, first party. It's built on top of a package called ParaTest, but they sit on top of it and add all the correct stuff so that it, like, refreshes the Laravel dependency injection container and makes a new testing database, and so it just makes life, like, incredibly easy. But it does not, as far as I'm aware, it does not do that thing which I assume whatever you're using does, which times the tests and puts the slowest one on its own.

Andrew

00:24:51 – 00:24:55

Is that what you're suggesting your parallel tests do?

Andrew

00:24:55 – 00:25:25

Yeah. So this is incredible to me because I assumed that everybody had the same tool. And so, I am going to shout out a product very strongly for what they're amazing at. In the same breath, basically, put them on notice because I think this tool actually would be something that you could implement across every testing ecosystem. The logic is transferable from ecosystem to ecosystem.

Andrew

00:25:25 – 00:25:52

When we were chatting about this before, you just didn't have that look of recognition when I mentioned their name. Like, I've never heard of that ever before, And I looked it up and sure enough, they are Rails and JavaScript. But most generally speaking, they're Rails specific. And it's knapsack and specifically it's knapsack pro. So what knapsack provided, it was this gem, open source I think.

Andrew

00:25:52 – 00:26:31

It was this gem that you would run your test suite and it would return metrics for how long each test took to run so that you could use that information to distribute your tests. And so you'd have one test that maybe took 45 seconds. Say it was a super long running system test. Well, you would know I put that test on its own parallel node. Here's how I can distribute all the other tests so that they also run-in 45 seconds or less And if you are able to parallelize your tests enough, you will always run your test suite as fast as your one slowest test.

Andrew

00:26:31 – 00:26:31

Mhmm.

Andrew

00:26:31 – 00:26:56

And so, Knapsack provided that as an open source gem, and then they launched a hosted version of that that would integrate with your CI systems or whatever called Knapsack Pro. And that's what I use. So my sort of canonical bullet train testing infrastructure is CircleCI plus knapsack pro. CircleCI allows you to scale up to as many nodes as you want. You want a 100, 200, whatever.

Andrew

00:26:56 – 00:27:20

They just have servers on tap for you. And knapsack pro automatically looks at every build, and it's responsible for distributing your tests. You don't have to do it manually in your config or anything like that. It'll reach out to their server and say, these are the tests we're running, and it'll reply and say, split them up on the nodes like this. And they provide all the tooling.

Andrew

00:27:20 – 00:27:59

It's all automatic. And, like, even better, in bullet train, we have all this stuff preconfigured. So it took me years and tens of 1,000 of dollars really, if I'm being honest about it. Because I bought custom servers, I bought the Apple Silicon stuff when it first came out because I realized it was improving my test suite by 30%, whatever. I was in a sort of ruthless pursuit of developer productivity, and what it all resolved to when all was said and done, and AWS bills were huge, and Apple Silicon was sold to people locally on Facebook Marketplace, whatever.

Andrew

00:27:59 – 00:28:26

When the whole thing was said and done, the most bang for my buck was CircleCI and Knapsack Pro and just as many parallel nodes as possible. And so I think our test suite on bullet trains down to, like, under 3 minutes. I don't know what it is right now, but it hovers there. And I have visibility on some projects with a huge test suite, and it's under 7, I think, right now.

Andrew

00:28:26 – 00:28:27

Wow.

Andrew

00:28:27 – 00:28:30

And that's because of just the parallel node stuff.

Andrew

00:28:30 – 00:28:46

Yeah. There's a lot there. I wanna highlight one thing that you said about Apple Silicon. You bought a bunch of Mac Minis and ran your own little testing farm for a little while, and it was awesome. And I'm so bummed that you had to sell it off, but that was one of my favorite Andrew things.

Andrew

00:28:46 – 00:29:03

Yeah. Yeah. I mean, when I said ruthless pursuit, that's exactly what I was referring to. I just felt like this was a big pain point for people at a certain point in time. And I was talking to people and they had these super long running test suites, like 15 minutes, 30 minutes, and huge bills.

Andrew

00:29:03 – 00:29:31

Like, I was talking to people that were spending big, big dollars on CI every month, and it made me think that there was a business there, and so I pursued it. It was something Adam Palazzi and I worked on together called Rails CI. And the big pitch there, the idea was twofold. We wanna run your tests as fast as possible no matter what out of the box every time. And I wanted it to be 0 config.

Andrew

00:29:32 – 00:29:45

And we did that. We built a little CI service, and it worked. And when stuff went wrong, when I broke something, my AWS bill was, like, $2,000 in a month. Like, it was Yep. That kind of stuff started

Andrew

00:29:45 – 00:29:47

It's a commodity business too.

Andrew

00:29:47 – 00:30:17

And all of the competitors that we were looking out at were raising huge amounts of money. And you realize this is a commodity. It's gonna be very challenging to compete with these people that are basically reselling compute with a layer of product on top of it. But fundamentally, I don't wanna start a VC backed business, and I think it's gonna be very, very difficult to compete. And maybe there was something there, but I mean, you gotta kinda pick your battles and you can't do everything.

Andrew

00:30:17 – 00:31:03

And so once I realized that on 0 configuration, I can give people 0 configuration in Bullet Train. So, like, for the people that I'm primarily trying to serve with Bullet Train, I can give them 0 config because CircleCI allows you to have, like, infrastructure as code. A dot YAML file pre builds or pre configures your entire kind of build pipeline. So I can optimize that and then combine that with knapsack pro, and this is actually a solved problem. In my desire to launch products for Rails developers, I may have been a little blinded to that, but it's actually better to just use off the shelf stuff instead of creating something new because the overhead of creating something new, it was substantial.

Andrew

00:31:03 – 00:31:25

Yeah. I remember that. That was one of my favorite sagas, but I remember that being a lot. On our side, I think my favorite stack depends on what I'm doing. So if I'm building a Laravel application, I'm gonna use Chipper CI all day long because that is a CI service built specifically for Laravel CI.

Andrew

00:31:25 – 00:31:52

You can just pick your PHP version, you can pick your database, you can pick your database and just go. And it's great, and I love it. And I think I saw that knapsack pro has an API, so we may have to tell our chipper CI friend, Chris, that he's gotta get knapsack pro integrated because that would totally rule. So that's like I'm building an application. Beyond that, if I'm building a package, which is what I do a lot of nowadays, it's GitHub Actions all day long.

Andrew

00:31:52 – 00:32:31

And the reason is matrices. Because when I'm building a package, I'm testing against maybe 2 or 3 versions of Laravel, maybe 2 or 3 versions of PHP, and maybe 2 or 3 versions of some other dependent package because I don't know where this package is gonna end up. And so I have to ensure that wherever it ends up, it works. I'll push something to GitHub and literally 60 different, not tests, but 60 different containers that then run their couple 100 tests spin up, and it's insane. I imagine I'm gonna pay for it at some point, point, but I haven't hit whatever the limits are, I haven't hit it yet.

Andrew

00:32:32 – 00:32:57

And so being able to run all those different configurations and, like, I think really importantly have fine grain control over which ones I can skip. Like, I know PHP 7.4 doesn't work with Laravel 9, so I can skip that matrix configuration, and it's incredible. It gives me so much power to be able to feel confident that these libraries I'm shipping don't break in weird apps.

Andrew

00:32:58 – 00:33:16

Yeah. GitHub actions, that blows my mind. I don't know and if anybody knows, I don't know how to do that on CircleCI off the top of my head. Like, our configuration's very specific. Test on this version of all the dependencies and stuff like that.

Andrew

00:33:16 – 00:33:41

But, yeah, that would be incredible and valuable, especially in library development, which we have a lot of. We have like a whole bunch of libraries that we've extracted including, for example, like magic test, that library that is both available in Rails and in Laravel for writing system tests. When you're shipping stuff like that, testing against all those different versions and dependencies and stuff is super valuable.

Andrew

00:33:41 – 00:34:10

Yeah. I also use it for, we mentioned it briefly, but testing the Torchlight API. And there, I do something a little wacky. So all the Torchlight rendering is done via JavaScript, so it's all done via Node. And so when I push to main on the Torchlight engine, the rendering engine, it executes the tests 3 different times, 3 complete separate environments.

Andrew

00:34:10 – 00:34:53

The first one is it spins up a little bitty node server locally, and it just runs the tests that say, like, if I give you this code block, I get this HTML back. Great. So if those tests pass, those are super cheap, super fast, if those tests pass, then it goes into the next block of the GitHub action which deploys the entire app to fly dot io plus AWS Lambda. So it deploys like a staging server, and then because I'm testing an API, it runs the identical staging server URL. This helps me know that I didn't muck something up with, like, my fly dot toml file.

Andrew

00:34:53 – 00:35:26

Even though the tests passed locally, it doesn't matter if I take down the API. I can't go back to people and say, well, the test passed locally, so I'm sorry your site is down, but they passed locally. So I deploy it to staging, run all the tests again against staging, and then it shuts staging down, which is kinda awesome, and then deploys it to production. And then just for giggles, it runs all the tests against production too. Because let's say I push to production and somehow, which I don't know how, somehow I broke something there.

Andrew

00:35:26 – 00:35:46

I would at least like to know. I don't have auto rollback built in, but I would at least like to know, hey, the test just failed against production. You should probably do something about it. So that GitHub action, I can't tell you how confident that makes me feel inside to just push to main and not really worry about it because it's not gonna make it all the way out if it doesn't work.

Andrew

00:35:46 – 00:36:04

Yeah. So I feel like and maybe there's more, but the three things that have come up, sort of axises on which we are trying to perform some calculus or optimize for these different, sometimes conflicting priorities, confidence.

Andrew

00:36:04 – 00:36:08

Confidence. Confidence. Confidence is everything I want for my tests.

Andrew

00:36:09 – 00:36:44

So just like how I couldn't sleep at night when I was running rails c I and we had real folks using it. This was in their developer pipeline as, like, beta customers or whatever. And you've got, like, 30 engineers on a team that are if this thing falls over, you're screwing up a lot of people from being able to ship code. So confidence, that's an example of bad confidence. When you don't have confidence that your infrastructure's up, that your code's good, whatever, as a developer in some ways or especially as a business operator, it can wreck your life Yes.

Andrew

00:36:44 – 00:37:20

Because you've just got this low grade headache all the time. I wonder if my stuff's broken. I was hanging out with somebody recently and they're like, yeah, I have to carry my laptop around with me all the time now because they had some bad experiences and they're just like, I just have to have my laptop with me. And maybe at some point, they can structurally kind of build out their business in a way that they don't have to have their laptop with them because their confidence is at the level where they're like, no, it's gonna be fine through some mix of I checked my work, my infra is good, and there's other people that can help me take care of stuff when something goes wrong. But confidence is a big one.

Andrew

00:37:20 – 00:37:56

I think productivity is probably the other one to describe, which is that the speed of that feedback loop. I think that there's definitely value in continuing to pursue what can we do to actually, this leads into something interesting. What can we do to improve the feedback loop? Speed up the feedback loop. When you've got 30 or 40 developers working on something, you don't want them hanging out for 15 minutes making jokes about what they're doing while they're waiting for the build to break

Andrew

00:37:56 – 00:37:56

Yep.

Andrew

00:37:56 – 00:38:23

And then rinse, repeat another 15 minutes, whatever. So that's definitely an angle where there's real work to be done to improve. And if you can create tooling that helps with that problem, that's gonna be valuable to people, like the knapsack pros and other things that hopefully are available in across all the ecosystems. That's, like, another one. And maybe the third and final one is the tooling around actually writing tests.

Andrew

00:38:23 – 00:39:11

And we haven't talked about that a lot. I think probably most people's experience writing tests in Laravel or Rails. Probably a lot of that is like writing commands in PHPUnit against the DUSK interface or in Minitest or Rspec against the Capybara APIs. And you're, like, writing these commands that some robot is gonna run over and over and over on a headless browser and make sure that the thing is doing its thing, make sure everything's working the way that it's supposed to. But I actually found that to be one of the big pain points with system tests was that they were exceedingly slow in the feedback loop of writing the test.

Andrew

00:39:11 – 00:39:28

Mhmm. And so there was, like, this resistance or this friction in actually writing system tests that was kind of a pain in the butt. Like, oh, I'm looking at the browser and then I'm trying to type the command and it's like, oh, I didn't find that label. And you're like, oh, well shoot. Why didn't it find that label?

Andrew

00:39:29 – 00:39:52

So you're debugging as you go, and it's this very high friction thing. And that was the experience that made me realize, like, this is actually another missing piece of the testing story. So I want confidence. I want as much confidence as possible, which means writing more tests. The problem is the tests were taking me so long to write.

Andrew

00:39:53 – 00:40:29

If you're trying to really ship product, if you're optimizing for shipping product, there's now this resistance to writing tests because that just takes more time. And so now, it's slowing you down. And so, in addition to speeding up the running of tests, speeding up the writing of tests is another critical thing if you want to increase your level of confidence. Because if it's fast to write tests, if it's fast to test an edge case, then you're going to do it. And you'll increase the surface area that your tests are protecting you from regression.

Andrew

00:40:29 – 00:41:11

So that was why it became critical, I think, for me to identify new tools for writing tests. And the first one that I found was this thing called Capicorder, which is a play on Capybara. And Capicorder was this plugin for Chrome where you could hit record and, like, start clicking around the interface, and it would generate the Capicorder commands that you could then paste into a test file or whatever, and then write your tests sort of interactively that way. And that was really interesting, and this is the thing that kinda kills me a bit. Capricorn never got adoption in the Rails ecosystem.

Andrew

00:41:12 – 00:41:37

It was around since, like, 2012. It was a developer in Poland that created it, and it never really got mainstream adoption or promotion. And one of the reasons why I suspect that happened was if you're doing TDD, you can't use Capricor because Capricor, you can only do after the interface already exists.

Andrew

00:41:37 – 00:41:38

Yep.

Andrew

00:41:38 – 00:41:58

I could be wrong on that, but it was interesting that even on Capicorder's readme file, there was a sort of disclaimer talking about how, hey, I know that this is so bad because it's not BD. Yeah. I know test first is what we're supposed to do, but, like No. That breaks me hard. I know.

Andrew

00:41:58 – 00:42:39

It broke my heart too because it was such an incredible tool that I think actually illustrates what the sort of atmosphere Mhmm. Around testing was in the early 2000 tens. That was the climate that somebody could create such a cool tool and yet feel like they needed to disclaim it that way. So it never really got adoption, and then it didn't get updated and stuff like that. So when I initially set out to find somebody to kinda fork it and maintain it or whatever, And in the end, we ended up not using that code at all, but created the spiritual successor to it or a spiritual successor to it, which was magic test.

Andrew

00:42:39 – 00:42:50

That was the sort of genesis of magic test was realizing, like, if I wanna increase my confidence, I need to increase the productivity with which I write tests, not just run them.

Andrew

00:42:50 – 00:43:07

Yeah. And then, Matthias picks it up and brought it over to Laravel as well. And so we have an implementation of magic tests for DUSK over there. And what it does is you basically start writing a test. You open your IDE, you start a test, and then you put some command in there.

Andrew

00:43:07 – 00:43:35

I don't remember what ours is. I think ours is, like, browser arrow magic or something. And what that command will do is it'll stop the test from running and open the browser, and you can start typing in and clicking buttons and doing actions. And in the background, it's just recording all of those actions that you, the human, are taking. And then I think it dumps you into, like, a debugger and you can type a few commands like flush or dump or commit or something like that.

Andrew

00:43:35 – 00:44:13

And then when you're done, you hit exit and it writes all of that to your test file. So wherever you had the browser arrow magic thing and you type flush or dump or whatever it is, it puts all of those test actions right into your test file. And so, basically, it's a test recorder without using a Chrome extension that just follows whatever the human does. So it's kind of insane. I like what that does because that, one, makes you super fast at writing these kinds of tests, but 2, also lets you throw away the tests a lot easier because the barrier to entry to writing those tests is not so high.

Andrew

00:44:14 – 00:44:38

So then you're not weirdly connected to these tests and feel like they're such a pain to write. I can't just trash this one because I can't change the UI because I have this test, and I can't trash the test because what a pain. It's like, no. These tests are easy, so therefore, they're cheap, and I can just throw it away if it no longer makes sense. So you can correct any of that about magic tests that I got wrong, but I think I got that pretty close.

Andrew

00:44:38 – 00:44:54

Yeah. Yep. That's exactly you represented it. And you made a point that I hadn't really thought of before. I've made that same point about scaffolding, where if you get the domain model wrong, you can go back and redo it because all this UI stuff was no effort anyway once you realized you got it wrong.

Andrew

00:44:54 – 00:45:31

But you're absolutely right about that with the test that you can throw away. It's interesting too when you wait until the very end to write your tests for something once you're sure you want it to be protected from regression because it also improves in a way that same sort of like, yeah, the tests aren't in my way. When you write so many tests first, it creates friction when you go to make a change because now that all those tests, all those unit tests fail. When you figure out refactor while all these tests fail. Well, when I refactor my code with system tests, I don't get test failures because the UI never changed.

Andrew

00:45:31 – 00:45:44

I just refactored some code. But if I wrote unit tests around that stuff, it's almost like a concrete around the thing that you created. Some people like that. Sounds a little claustrophobic to me. I like to move around a little more easily than that.

Andrew

00:45:45 – 00:46:06

And it depends, again, on what you're testing. The concrete is great if people are depending on your actual function signatures and implementations. That concrete is really helpful. The concrete is terrible if people just want highlighted code back, and I need to make the highlighted code correct. And so, again, it just depends.

Andrew

00:46:06 – 00:46:10

There's no one answer, of course. It just depends on the thing that you're shipping.

Andrew

00:46:10 – 00:46:29

Yeah. So I think when you're writing your test later, it sets you up to say yes. When a stakeholder says, hey, why don't we change this or can we try this or whatever? Because you haven't already written tests around it, you're gonna be more likely to say, yeah, let's give it a shot because it's no trouble to you. There was no test written around it anyway.

Andrew

00:46:29 – 00:46:44

You don't have to change that thing in 2 places. You only have to change it in 1, and then you can write tests around it after. So yeah. I think it's a great situation right now. I think in terms of testing infrastructure and tooling and libraries and stuff like that.

Andrew

00:46:44 – 00:47:23

Well, I will say this, if magic test appeals to anybody, I am trying to hire somebody to work with magic test full time. Both this is a senior level position, but I'll just put it out there and folks can reach out via DM on Twitter or whatever. Take a look at magic test. If you enjoy being at the frontier of software testing or you really just enjoy testing philosophy and all of that stuff, this is an opportunity to basically do software testing as a senior level engineer and work on open source testing tool sets full time, definitely reach out. I would be remiss if I didn't say that because it's actually on my to do list to post about that publicly.

Andrew

00:47:24 – 00:47:34

But even the stuff that we've currently got is great, and it's such an improvement over what the landscape was, like, 10 years ago. So I'm excited for that.

Andrew

00:47:35 – 00:47:57

I think kind of wrap it up a little bit. My point of view would be write as many tests as you find useful that give you comfort. I think there's a lot of argument about what should we call the tests. If you do this type of thing, then that automatically becomes this type of test, and that's a bad thing. There's just a lot of argument about everything all the time, testing included.

Andrew

00:47:58 – 00:48:26

My point of view would just be write the tests that make you feel comfortable for the domain that you're in, and write as many as you want slash as many as you can. I get so much relief when I feel like I can push to main and know that I'm not gonna break anything, and so that's kinda where I come down on it. I feel like we usually come down in the same spot. But, yeah, I would say push back against anyone that's religious or dogmatic about what you should or shouldn't do in your kind of setup.

Andrew

00:48:26 – 00:48:46

Yep. I totally agree with that. One thing I'll be interested to circle back around with you sometime to talk about are fixtures and seeds and factories. I think it's pretty similar in Laravel in and Rails in terms of the tooling available. Like some of the other things that we talked about today, there are different approaches.

Andrew

00:48:46 – 00:48:59

It's interesting because I don't think people are so dogmatic about it, but it is a similar thing where different people have found things that work for them. And I feel like that's fun to noodle on. So but probably too much to unpack today, so maybe revisit it.

Andrew

00:48:59 – 00:49:07

Probably for today, but I'm just gonna put a little teaser out there. We have first party factories, surprising no one. So maybe we'll get to talk about it at some point.

Andrew

00:49:07 – 00:49:21

Yeah. Whereas in Rails, the first party thing is fixtures and people use FactoryBot for factories and Yep. In Bullet Train, we do something totally different. But yeah, I really enjoyed this, man. Thanks for taking time to chat about it.

Andrew

00:49:21 – 00:49:26

And thank you everybody that's listening for taking time to share it with us.

Andrew

00:49:26 – 00:49:27

Talk to you next

Andrew

00:49:27 – 00:49:31

time. Framework Friends is edited by Paul Barr at Peachtree Sound.

Andrew

00:49:32 – 00:49:34

Our intro music was created by Corey Griffin.

Andrew

00:49:34 – 00:49:44

You can find us at frameworkfriends.com. Andrew's on Twitter at Andrew Culver. And Aaron is on Twitter at aaron d Francis.