Riding Sidecar with Aaron Francis

August 21, 2021

Jake Bennett and I talk about about my project Sidecar.
Learn more →

Transcript

Jake
00:00:00 – 00:00:01
Hey. I'm Jake Bennett.
Aaron
00:00:01 – 00:00:02
I'm Aaron Francis.
Jake
00:00:02 – 00:00:06
And we're missing Michael Dorinda, but welcome to the North Meet South web podcast.
Aaron
00:00:08 – 00:00:09
I I am not Aaron,
Jake
00:00:09 – 00:00:10
how's it going, man?
Aaron
00:00:10 – 00:00:11
It's going well.
Jake
00:00:11 – 00:00:23
Yeah. Pardubrinda. He's not on the show today. So we actually Aaron and I have talked a couple times just through Twitter about having him on the show. Michael and I referenced this in our previous episode, episode 99.
Jake
00:00:24 – 00:00:36
And so we are actually top secret. This is the 100th episode. We're releasing it as the 101st episode because the centennial has to be me and Michael. Right? I mean, come on.
Jake
00:00:36 – 00:00:36
That's not fair.
Aaron
00:00:36 – 00:00:42
Even in your in your 99th episode, you said, well, we can't have Aaron on the next one because that would be 100.
Jake
00:00:44 – 00:00:48
Exactly. Exactly. And we're doing it anyway, but it's okay. It's okay. We'll call
Aaron
00:00:48 – 00:00:49
it 101. That's fine.
Jake
00:00:49 – 00:01:07
We'll call it 101. So, Aaron, for those, who might not know you, why don't you introduce yourself? Tell us a little bit about what are the things that you're working on. And you got it seems like looking at your Twitter profile, going through some of the stuff, you got a lot of things that you're working on. So just give us your, yeah, give us your elevator pitch.
Jake
00:01:07 – 00:01:09
Who are you? What do you what do you do?
Aaron
00:01:09 – 00:01:20
Yeah. So like I said, my name is Aaron Francis. I live in Texas. So I'm not I am, I guess, the south part of north meets south today, but not not quite as far. But I I live in Texas.
Aaron
00:01:21 – 00:01:42
I'm a Laravel software developer. I actually have a full time job. I work as CTO and COO at a property tax company. So very, very thrilling stuff. But most of what I talk about on Twitter is not property tax because nobody cares about
Jake
00:01:43 – 00:01:43
that.
Aaron
00:01:44 – 00:01:49
So I've been so we recently my wife and I recently had twins. So, like
Jake
00:01:50 – 00:01:51
Oh, congratulations. That's awesome, man.
Aaron
00:01:51 – 00:01:54
Like, 2 months ago recently. So
Jake
00:01:54 – 00:01:55
What? Wow.
Aaron
00:01:55 – 00:02:17
Yeah. So I was on paternity leave for, like, 7 or 8 weeks, and I was working just 2 hours a day for, like, my full time job. And so that's where a that's where a lot of, like, the creative the burst of creative energy came from because I was like, I can kind of, you know, between feeding and changing and not sleeping,
Jake
00:02:18 – 00:02:19
I can work on Exactly.
Aaron
00:02:20 – 00:02:28
Work on whatever I want. And so it kind of just all this stuff that was in my head of, like, stuff I wanna do kinda just burst forth.
Jake
00:02:28 – 00:02:29
Found an outlet. Yeah.
Aaron
00:02:29 – 00:02:30
I did. Found
Jake
00:02:30 – 00:02:31
an outlet of time and
Aaron
00:02:31 – 00:03:02
Yeah. Yeah. And so one of the one of the things that I've been working on, for probably, gosh, probably, like, 2 years now, is is the main it's that's, like, the main side project, and then each side project has its own side project. But the main side project is, it's like a a filter builder, you know, when you're, like, on Yeah. Like, on, you know, Nike, and you're, like, I want shoes that are size 12 and are
Jake
00:03:02 – 00:03:03
Yes. Yes.
Aaron
00:03:04 – 00:03:17
You know, Air Max twos and are black. And you can kinda, like, build your own basically, build your own query. So I've been working on that with a friend for about 2 years now.
Jake
00:03:17 – 00:03:18
Okay.
Aaron
00:03:18 – 00:03:40
And he's got a full time job and a wife and a kid. I've got a full time job. The where we're at now is the Laravel one is actually totally done, but we picked up a client who does rails. And so we're rewriting the whole thing in rails. So it's like That's that's
Jake
00:03:40 – 00:03:41
just how it goes. Right?
Aaron
00:03:41 – 00:03:59
Just can't finish. Like, I just I had a tweet recently that was like the last, you know, the best part about being 90% done is you're halfway there. Like, you're halfway finished. And so that's kinda how it feels right now with with the filter builders. Like, man, we're so close, but then this client was like, hey.
Aaron
00:03:59 – 00:04:03
We'll pay you to do it in rails, and then you get to keep the rails version. Like
Jake
00:04:03 – 00:04:04
There you go.
Aaron
00:04:04 – 00:04:05
Okay. Great.
Jake
00:04:05 – 00:04:09
That's awesome. So is this gonna be a paid this is gonna be a paid product for you guys?
Aaron
00:04:09 – 00:04:22
Yeah. Exactly. So we're gonna do Cool. Open source front ends. So we have a Vue front end, and then he's writing my business partner partner, Sean, is writing the, what do they call it?
Aaron
00:04:22 – 00:04:25
Hotwire? I think that's rails 1. Hotwire.
Jake
00:04:25 – 00:04:26
Yeah. Hotwire for Rails.
Aaron
00:04:26 – 00:04:32
Yeah. He's doing the Hotwire front end, and then we'll have a Laravel and a Rails back end. And so
Jake
00:04:32 – 00:04:33
Back end. There you go.
Aaron
00:04:33 – 00:04:41
Yeah. It's gonna be kinda like I I guess, kinda like the inertia model where it's like we'll adapt to whatever stack you have. Like, if you want
Jake
00:04:42 – 00:04:42
That makes sense.
Aaron
00:04:42 – 00:04:51
If you want Laravel and Vue or Rails and React or Laravel and React, like, we'll try to do it all, basically. And the back ends will be paid. Yeah.
Jake
00:04:51 – 00:04:56
Yeah. That's awesome. I love that. And, I'm trying to remember. Is it, like, Hammerstone?
Aaron
00:04:56 – 00:05:01
Yeah. Exactly. Hammerstone dotdev. Yeah. And so it's me Very cool.
Aaron
00:05:01 – 00:05:03
It's me and my friend Sean, and we met,
Jake
00:05:03 – 00:05:04
I think, we met
Aaron
00:05:04 – 00:05:20
at Bacon Biz Conf, which was like a micro comp but smaller. We met, like, in 2012, and we've been Internet friends ever since. And he's Very cool. Front end genius, and I'm not. And so we work really we work really well together.
Jake
00:05:20 – 00:05:40
That does work well. Yeah. That it's I find that, like, I'm I I feel like I prefer the back end, and then, yes, I am still sort of searching for the guy who's the front end guru who's just, you know, knows it all Yeah. And, can do it in their sleep. Wilbur Powery, who we had on our team for a while Mhmm.
Jake
00:05:40 – 00:05:51
Was real he was he was pretty close to being that guy. He was he was the dude. But, alas, he has moved on to greener pastures. So now I'm on the search again. So Yeah.
Jake
00:05:51 – 00:05:52
You just have to want to keep up with it, and it's just
Aaron
00:05:52 – 00:05:54
Right. It's Man. It's so true. Tough.
Jake
00:05:54 – 00:06:00
Yeah. Yep. I mean, even Laravel moves so fast, but the front end is insanely fast. Yeah. It's wild.
Jake
00:06:00 – 00:06:15
And there's so many new tools. You really have to be I feel like you're you're coming into this age now, Aaron. Welcome to the dad's club, by the way. Thanks. You're gonna you're gonna get to the point now where you are going to have to be more choosy about what things you actually pursue Yeah.
Jake
00:06:15 – 00:06:34
And invest in. And so you may find that you become a late adopter on things where you kind of, like, let the people who have a little bit more time play with the stuff until and then, you know, you wait until it's, like, 9 months, a year afterwards and see which stuff is still around and people are still talking about, then you sort of pick it up because you'd still have the time. I don't have
Aaron
00:06:34 – 00:06:36
Or the energy. Stuff. Yeah.
Jake
00:06:36 – 00:06:38
All the time anymore. Yeah. Yeah. Absolutely.
Aaron
00:06:38 – 00:06:50
So somebody asked me the other day about ES build, and I was like, man, I've just barely wrapped my head around Webpack. I have no idea what ES build even is. So Right. Sorry. I'm like,
Jake
00:06:51 – 00:07:00
if it's not Laravel mix, I haven't used it. Right. So it's like, yeah, just. Oh, man. I can follow a tutorial to get Tailwind set up in a in a, you know, project without Laravel mix.
Jake
00:07:00 – 00:07:04
But, like, outside of that, I'm kinda like, yeah. I just, like, stick to my wheelhouse stuff.
Aaron
00:07:04 – 00:07:05
You know? Exactly.
Jake
00:07:06 – 00:07:21
But, yeah, I wanted to have you on because of your package sidecar. But then I did sort of a a little bit of, Aaron Francis deep dive and, like, went down the hole. I watched you put together your whole shed quarters, which you're sitting in right now, which is very awesome. And I was like, wow. This is super cool.
Jake
00:07:21 – 00:07:43
This dude's a super hard worker, insanely productive. And then I went and read your blog posts, and it was like, property tax, whatever it is, but this dude has to deal with some, like, real world world crap, like, importing massive CSVs at all hours of the night. And I was like, this dude sounds like the kind of guy that, does the sort of stuff that we do at my job. Right? It's it's not sexy.
Aaron
00:07:43 – 00:07:44
It's not sexy.
Jake
00:07:44 – 00:08:06
It's not, like, crazy front end, you know, like, all this fun, like, you know, you know, SPAs and all that, but it's, like, real world stuff and, like, it's the businesses that have been around that are still making money that are you know, they just need automation and and whatever. And so I love it. Like, there were some really good tips you had about, I'm trying to remember, like, these running these fake daemons or
Aaron
00:08:06 – 00:08:08
Pseudo daemons. Yeah.
Jake
00:08:08 – 00:08:14
Yeah. The Pseudo daemons. That was a really good read. And so I was, like, maybe that's a strategy we could use. That seems really interesting.
Jake
00:08:16 – 00:08:21
So I feel like we could talk about, like, a 100 different things. Mhmm. So fan of your work to say that.
Aaron
00:08:21 – 00:08:21
No big.
Jake
00:08:21 – 00:08:24
I appreciate it. Some stuff I want to read through. Yeah. No. Absolutely.
Jake
00:08:24 – 00:08:35
I feel like you've got a lot of really cool stuff to say, which is part of the reason why I wanted to have you on. Just be like, hey, folks. Go check out Aaron's stuff. Oh, thanks. But also the sidecar thing is really, really cool.
Jake
00:08:35 – 00:08:54
So, we've talked about it on Laravel News. We talked about a tiny bit on North Main South, but I would love for you to kinda explain maybe some of like, maybe first of all, tell us what it is. What is Sidecar exactly? And then what was sort of the motivation behind it for you and how you're currently using it?
Aaron
00:08:54 – 00:09:24
Yeah. Totally. So Sidecar is, a Laravel package. So it's Laravel specific, and it allows you to, it facilitates, building, packaging, deploying, and executing AWS Lambda functions from your Laravel application. So it's basically like the 80% use case of AWS Lambda when it comes to Laravel.
Aaron
00:09:24 – 00:09:32
So, like, if if Laravel mix was webpack but easier, Sidecar is Lambda but way easier.
Jake
00:09:33 – 00:09:34
Mhmm. Yeah.
Aaron
00:09:34 – 00:10:02
So I think so where it came from was I was I had some stuff running on vapor, which I love. Total huge fan of vapor because I don't, like you said, I don't have the time or energy to, you know, futs about with a lot of stuff. So vapor rules. I love it. I needed I needed, like, 1, JavaScript, like, 1 node process in the background.
Aaron
00:10:02 – 00:10:35
And I was like because I I was trying to do you know, every side project has a side project. I was writing docs for Hammerstone, and I was trying to get the syntax highlighting just right because it looked stupid when it was wrong. And so I wrote, you know, a JavaScript. I found a Versus code engine JavaScript syntax highlighter, but it's all in Node. And with Vapor, you don't have, like, you don't have just a server you can shell out to and, like, run a node process and then hop back into PHP.
Aaron
00:10:35 – 00:10:57
Right? If you were to do the Docker deployments, maybe, whatever, but I I don't know Docker, and I didn't have time. So I thought, okay. Well, I know I can do like, I know Lambda has a node runtime, and Vapor is on Lambda. So maybe I could just use Lambda and kinda, like, hand off my JavaScript to a different Lambda and get the result back.
Aaron
00:10:57 – 00:11:26
And so I tried that, and surprisingly, it worked really well. And I was like, wait. I didn't have to set up an API gateway. I didn't have to set up the, event streams or anything like that to to get to Lambda and back because the AWS SDK for, PHP provides a method called invoke. So it's like call the call the Lambda function.
Jake
00:11:27 – 00:11:27
Call the Lambda.
Aaron
00:11:27 – 00:11:48
Yeah. And so when I saw that, I was like, wait. This should be, like, this should be way easier. We should be able to especially now as we're moving like, I see a lot of people moving to vapor, then you realize that you're missing that ability to do stuff with node. It should be way easier to just say, here's a function call it.
Aaron
00:11:48 – 00:11:49
Give me the result.
Jake
00:11:50 – 00:11:50
Mhmm.
Aaron
00:11:50 – 00:12:06
And so it really, like so I was doing that. I was doing this all, like, goofy behind the scenes invoke and, you know, get the response back. And I saw, Jonathan Renick tweet something about inertia server side rendering.
Jake
00:12:06 – 00:12:07
Right.
Aaron
00:12:07 – 00:12:22
Right. He he said, like, basically, we start an express server in the background, and we send a request to it, and it renders the view and, you know, as HTML and sends it back. I was like, oh, that's exactly, like, that's exactly what I'm doing. Yeah.
Jake
00:12:22 – 00:12:24
What you just did. Yeah.
Aaron
00:12:24 – 00:12:48
And so I I tweeted back to him. I have it pulled up here because I'm reading it now, and it just sounds so insane. And I said, I'm serving Laravel on vapor, and I'm hitting a JavaScript function in a Lambda from PHP using Lambda dot invoke from the PHP SDK. And he tweeted back and was like, dude, I don't even know what that means. But can you help us figure out inertia on on vapor?
Aaron
00:12:50 – 00:13:03
And that's that's when I like I, like, resurfaced from my deep dive of serverless stuff. I was like, oh, yeah. This is cool. Like, I could I could do something here. And so that was the impetus.
Aaron
00:13:03 – 00:13:10
When he tweeted back, dude, I have no idea what you're talking about. I was like, okay. Like, I can make this easier for people.
Jake
00:13:10 – 00:13:11
Yeah. Let's make this easy.
Aaron
00:13:11 – 00:13:37
Yeah. And so that's that's kinda what I did. So what it does is, like, all you have to do is, like, you you have a class, you know, a PHP class that defines what files you want to ship to Lambda. And it's like, you know, your JavaScript files and maybe some node modules or whatever, And then it gives you some commands, and you can say, sidecar deploy, sidecar activate, and that'll do everything on AWS. So it it sets it all up.
Aaron
00:13:37 – 00:13:48
It's just like vapor, and then it'll, like, it'll alias your functions, and it puts in you know, you can put in environment variables and all that stuff. So it's like vapor, but way, way, way, way, way, way smaller.
Jake
00:13:49 – 00:13:50
The mini version of Vapor.
Aaron
00:13:50 – 00:13:53
It does one thing. Yeah. And it's not PHP related at all.
Jake
00:13:54 – 00:14:15
Yeah. I think the thing that was really cool for me is I've looked at, like okay. I have a need for a thing, like, that would be a Lambda, but there's a couple things I want to avoid. Number 1, I can't really give all of my developers access to be able to just create Lambdas Mhmm. Out in our AWS instance.
Jake
00:14:16 – 00:14:23
And, also, there's no way to version control those because they're just out there.
Aaron
00:14:23 – 00:14:24
Mhmm.
Jake
00:14:24 – 00:14:26
And so I'm now limiting myself
Aaron
00:14:36 – 00:14:38
Like, how do you deploy them in sync with your application?
Jake
00:14:39 – 00:14:42
Correct. Correct. So all of those things. So it's been, like, okay. Yeah.
Jake
00:14:42 – 00:14:53
Lambdas are cool, but it's not really anything I can use. Right. And this sort of blows the doors off that because it does all that for you. Mhmm. So maybe you could walk us through a little bit.
Jake
00:14:53 – 00:15:06
So I've looked at the package. Mhmm. I've talked about it before. I'll give you my very basic understanding of it, and then you can tell me what I'm saying wrong and how I'm not Perfect. Explaining it correctly.
Jake
00:15:06 – 00:15:30
Hit me. So you have a class that extends a class in your package, right, which basically says, hey. This is a Lambda invocable class sort of deal. So let's just let's just say that for, sake of argument, it's going to be a Lambda that's going to roll through an s three bucket, look for zips, and unzip them. Okay?
Jake
00:15:31 – 00:16:14
We'll call it s s three unzipper. So this s three unzipper class, and what I'm going to do is I'm going to define inside of there. And, again, I'm not looking at the documentation, so I couldn't tell you exactly what it is. But inside of there, at some point, I'm going to define, I have a JavaScript no JavaScript sort of file over here somewhere that I want you to take and put into Lambda or into AWS as a Lambda function that I want to run. And then I have another method that I'm going to have called execute spot where I can say there are a few locations that the script has the ability to access within my Laravel application.
Jake
00:16:14 – 00:16:33
So there might be a couple spots where it needs to be able to write files to or read files from, and so I have the ability to do that as well. So I define all that. And then when I'm deploying my application during my continuous integration pipeline, I'm going to call sidecar colon deploy. That's going to take that JavaScript. It's going to push it up to Lambda.
Jake
00:16:33 – 00:16:56
It's going to create a new Lambda function. And then within my Laravel application, when I want to invoke that Lambda function, I'm going to call that s3 unzipper class, and I'm going to somehow invoke that thing, with any arguments that I need to pass through to it. And it's going to do that. It's going to make that request. It's going to use that, you know, that, Amazon, the AWS SDK that says Lambda invoke.
Jake
00:16:56 – 00:17:07
It's gonna get that, get the result, and return it back to me. Is that a relatively decent understanding of how it works or where are the areas I'm missing and I'm fuzzy and I'm not completely clear on?
Aaron
00:17:07 – 00:17:22
Very, very good. I would, if we can, for the sake of argument, change the example a little bit because you can you can do s 3 unzipping relatively easy with, you know, fly system or
Jake
00:17:22 – 00:17:22
True story.
Aaron
00:17:23 – 00:17:51
Stuff we have in Laravel. And so the the stuff that I've seen so far is, like, image generation, which is real real easy in JavaScript and a lot harder in PHP. And so your your, your timeline is exactly right. So what you have is you've got your JavaScript script. You have a file that says, here's, like, how I here's how I create an image.
Aaron
00:17:51 – 00:17:54
And a lot of people are using this for, like, open graphs Yeah.
Jake
00:17:54 – 00:17:55
That makes sense.
Aaron
00:17:55 – 00:18:17
Tags. Right? So every blog post, they want a new dynamic image, and there are a million of these services that have sprung up. And so you define you write this function that generates an image, and it's a JavaScript file. And then in your Laravel app, like you said, you have a class that extends, it extends I think it's called sidecar function.
Aaron
00:18:18 – 00:18:35
And you say, okay. The name of my JavaScript file is image generator dotjs, whatever, and it lives inside your Laravel app. And then you can say, I want it to run on node14.x. Right? So you can define your runtime as well.
Aaron
00:18:35 – 00:18:57
And then you can give it, like, you know, 3 gigs of memory. So you can define how much memory the Lambda has as well. And so that determines how fast it is, all that kind of stuff. And then you're absolutely right. You call, sidecar deploy, and and what that does is that does half of, what you may consider a deploy.
Aaron
00:18:57 – 00:19:29
It does half of it. So what it does is it takes all of the files that you've decided are necessary to to build that image, and that could be the entire node modules folder. If there's enough room on Lambda, you could do that, or it could be a subset. Right? And what it does is it takes them all, and it puts it into a zip file, and it uploads it to an s 3 bucket, and then it goes to Lambda and creates the function and says, you know, the the code package is in this s three bucket.
Aaron
00:19:29 – 00:19:36
The handler is this. The runtime is this. The memory is this. Please make this function. And so it makes it on Lambda.
Aaron
00:19:37 – 00:20:02
So that's half of it. The second half is you can call sidecar activate, which then turns the function live. Right? And so that helps when you've got let's say you're you're building all of this stuff in your CI and you deploy it to Lambda, but then it takes another 3 or 4 minutes for Envoyeur to finish or whatever. And you don't want your functions
Jake
00:20:02 – 00:20:03
to be
Aaron
00:20:03 – 00:20:32
out of sync because then your Laravel app is new or is old and your Lambda is new. Right? So then you call sidecar activate and it switches it live, And then you can just call, from PHP, you would call, like, o g image colon colon execute. And you can pass through, like, the title of your blog post, the URL where it can be found, and anything you pass through to that execute function ends up in the landline
Jake
00:20:32 – 00:20:32
wild person.
Aaron
00:20:33 – 00:20:37
In the JavaScript world. And so yeah. So you can play can
Jake
00:20:37 – 00:20:38
you just invoke some random
Aaron
00:20:38 – 00:20:40
function, but you can
Jake
00:20:40 – 00:20:43
actually pass arguments to it from PHP. Yeah. That's wild.
Aaron
00:20:43 – 00:21:02
You can pass right, which obviously means you could do it from form inputs. You can do it from query strings. You could do it from from anything. Right? So then then you've got basically, like, a tiny little Netlify that just, you know, like, goes through your Laravel app out to Lambda and then back through.
Aaron
00:21:03 – 00:21:34
And so what that would do is it would go to Lambda, say, here's the data that I'd like you to add to this, you know, image generator. Your JavaScript handler would run, generate the image, and then give it back to your Laravel application to do whatever you want with it. So now you've got, you know, this base 64 encoded image or whatever, and you can cache it and then return it to the browser. And so you've just generated an image on a Lambda without having to worry about, like, setting up the serverless framework on your local computer
Jake
00:21:34 – 00:21:54
and deploying. No. That makes total sense, and that is amazing. I love it. I think the reason why I use that s 3 unzip example is because we have had, like, instances where I'm trying to think I'm trying to remember exactly what the hang up was.
Jake
00:21:54 – 00:22:28
Like, if it was the fact that it was timing out or something or if it was that I can't remember exactly what it was, but there was a way really easily, like, you know, if you go look at, like, if you go Google, like, s 3 unzip, whatever, there's pretty much like a little Lambda function you just copy, paste, like, drop in there, like, boom. And then you just, like, basically point it at the directory and just goes just handles it. It goes everything unzips, handles it, pushes over there. Because we had some, like, relatively large zip files, and it's just taking forever. So, anyway, there like you said
Aaron
00:22:28 – 00:22:42
I think, yeah, I think you might be right because I think I think bandwidth between Lambda and s 3 is also free. Not only is it fast, but it's it's, you know, it's free. And so maybe maybe that would be a great reason to use a label to get your s 3 buckets that way.
Jake
00:22:42 – 00:23:11
I remember that being part of the reason why I was like, oh, this could actually be really interesting for that particular reason. But I love the idea that you're talking about, which is, image generation or even, you know, Spacy has, what's it called? Browser shot. It's like I can't remember if that's what a Spasie browser shot, but, essentially, the idea of being able to take, you know, snapshots of images. And right now, you sort of the some of the pain in the butt about that is you have to have Node and all the environment stuff all set up just perfectly on your actual server.
Jake
00:23:12 – 00:23:24
And instead of having to do that you could probably just offload that to Lambda and say hey Lambda already has all that set up and all that good to go just ship it over there and let it handle it. Right? Yep. Which is cool too. So yeah.
Jake
00:23:24 – 00:24:00
No. And then the other idea is, so you've worked with Jonathan on this, the server side rendering stuff. Trying to remember sort of the trail that went down, which was and I'm I don't think this has to do with sidecar necessarily, but some of the idea of, like, even being able to do feature tests on inertia using that server side rendering sort of idea, where they were rendering the stuff from the server and then doing these, you know, Laravel, dusk sort of tests, on that server side rendered stuff. Anyway, I'm not sure if that was even part of the conversation you guys had, but I just remember it being part of what I saw on Twitter. Maybe not between you 2, but in any case.
Aaron
00:24:00 – 00:24:45
No. That was that was a, server side rendering conversation, but wasn't specific to to Sidecar. The the place that I slotted in to that conversation was people being either unable or unable in this in the case of vapor or unwilling to run a node process in the background to do server side rendering. And so there is a company I forget I forget the name of the company, but the guy's name is Greg Skirman, I think. And his company is is live with inertia server side rendering powered by side car, and they're running, I mean, they're running thousands and thousands of requests through, and they're proxying it all through Lambda and giving it back as a rendered HTML.
Aaron
00:24:46 – 00:24:48
And it's That is so cool. It's crazy.
Jake
00:24:48 – 00:24:49
So freaking cool.
Aaron
00:24:49 – 00:24:52
It's Yeah. It's so cool. Yeah.
Jake
00:24:52 – 00:25:20
I wanted to talk maybe a little bit about these, pseudo daemons as well just because I thought this was such a novel idea. Yeah. And so this idea that so, basically, maybe you could tell us I need almost, like, recap the blog post. But Mhmm. Some of the challenges you were having is you guys have these massive CSV files that you need to pull in, at different times of the day in order to be able to update you, like, your tax tables or your property tax tables or whatever it might be.
Jake
00:25:20 – 00:25:34
Right? And so, man, I honestly do feel like there was there was another one where you were talking about how you handle CSV imports and all the lessons you've learned the hard way about handling CSV's, and it's, like, anybody exactly.
Aaron
00:25:34 – 00:25:35
Don't trust anybody.
Jake
00:25:35 – 00:25:51
Had to deal with the heartache of having to import CSV's from a vendor, a client, whatever, the government, which is kinda like what you're dealing with. Yep. Exactly. That's what we do. Just some of the tips you had in there, I thought were so good.
Jake
00:25:51 – 00:26:24
So if you're listening to this and you have to deal with CSVs, you should definitely go read this blog post that Aaron's got out there, talking about validating the headers before you assume that everything is the same. Right? And saving yourself a lot of headache in that or, checking to make sure that you don't have, like, some blank field in in the first call or in, you know, in the first set of records and just screwing up your entire process, or looking for these delimiters that could be, interpreted as, like, a line return or a new row. Right? So oh my word.
Jake
00:26:24 – 00:26:44
Just so many good things that I've run into, and I was like, yes. Like, all of this stuff is so good. Like, it's like, it's like when you're listening to a comedian and they take this scenario that you've been through, like, a 100 times, and they, like, say out loud what you've always thought, but you could never put it into words. Like, that is so true. And it's so true.
Jake
00:26:44 – 00:26:52
This is that blog post. It's like, oh, my word. All of these things, but it's written down in text format. I needed this. Like, I wanna share this to the world.
Jake
00:26:52 – 00:26:53
So
Aaron
00:26:54 – 00:26:59
Oh, man. I'm so pumped to hear that because it is not sexy stuff. Real life.
Jake
00:26:59 – 00:27:01
That makes me so real life. Feel good. Yeah.
Aaron
00:27:01 – 00:27:03
It's, like, so unprocessed.
Jake
00:27:03 – 00:27:14
Like, I can't even imagine how much time you spent working on that blog post to put all that stuff together. It's a thankless task. Right? Because that's the thing is, like, you never know who's gonna read that or if anybody. Right?
Jake
00:27:14 – 00:27:15
I have read it. It is amazing.
Aaron
00:27:16 – 00:27:23
And then you yeah. You you look at Google Google Analytics, and you're like, oh, wow. 18 people read it. That's really great. Out there.
Jake
00:27:24 – 00:27:34
Yeah. It's it's really good stuff. So, anyway, that was sort of the impetus. It's like, you have these massive CSVs that you're importing. And so, dealing with, like, how do you handle timeouts?
Jake
00:27:34 – 00:28:00
When's the best time to do them? How do you, is there a way to, like, instead of checking every 5 minutes, can I just import it immediately? Like, what, you know, how do I how do I do that in an efficient way without, you know, using crazy amounts of memory? And so this you you have this idea of, like, pseudo daemons. So I I will give a quick disclaimer, which is that this is probably one of those things that's a little bit too nuanced just to share in a podcast format and have it be completely understandable.
Jake
00:28:00 – 00:28:13
You might have to go read the blog post. But even if you just gave, like, a little bit of a teaser of, like, here's the idea behind it, I would love to just sort of maybe whet people's appetites for, like, what it is that you're trying to accomplish and and how you went about doing it.
Aaron
00:28:13 – 00:28:35
Yeah. For sure. So, background daemons are processes that are kept alive by something, you know, supervisor or whatever. So pseudo daemons, which is what I named this it's a Laravel package. Pseudo daemons are like daemons, but you don't have to modify any server configuration anywhere.
Aaron
00:28:35 – 00:29:05
So it's a process that's actually kept alive by Laravel's scheduler. So you put it in like, you you write it as a command, just a standard Laravel command. And then I think you add a trait called, you know, is Pseudo daemon or run as Pseudo daemon or something like that. And then you just schedule it to run every minute in the Laravel, scheduler. But what it does is it keeps itself alive.
Aaron
00:29:05 – 00:29:31
And then if it ever dies, it gets picked up the next minute by the Laravel scheduler. So it's kinda nice because it's like you're getting the advantages of, you know, a supervised process that keeps coming back up should it ever die. But you don't have to go into forge and go into because, again, that's outside of your version control. That's outside of your visibility. That's like it's it's just in a place where I'm not.
Aaron
00:29:31 – 00:29:49
And so the pseudo daemon is alright. Let's put it all in code. Let's just keep it running. And if it dies, it's only gonna be dead for, you know, maximum of 59 seconds because Laravel's gonna pick it up in, at the minute mark. And so it's a package that facilitates that, basically.
Aaron
00:29:49 – 00:30:24
Just keep it alive and run it over and over and over and over. And so the way that we use it at Resolute, which is the the property tax company, is we run I think we run 4 or 5 importers, as pseudo daemons. So that should anyone in the company, should anyone drop in a a new import, it'll get picked up immediately. So it's not gonna wait around for, you know, every 5 minutes or even every 1 minute if it were on a schedule. So that's that's one good part is it gets picked up immediately.
Aaron
00:30:25 – 00:30:39
The obvious question is, why not just run your imports as Laravel jobs? Right? Because imports can take an hour and a half. And so if you've ever, like, configured your horizon, configuration
Jake
00:30:40 – 00:30:40
A time out.
Aaron
00:30:40 – 00:30:46
To, like, have a super, yeah, a super long time out, you know that you have to,
Jake
00:30:46 – 00:30:47
like to set up a queue.
Aaron
00:30:47 – 00:31:04
You've gotta set up a different retry after. The retry after always bites me in the butt somehow. It's like I set it to 20 minutes, but it should have been, you know, 19 minutes and 59 seconds or whatever. And, you know, some of some of our import imports literally take
Jake
00:31:04 – 00:31:04
Oh my gosh.
Aaron
00:31:05 – 00:31:30
6 and a half hours. And so it's like, I'm not gonna set up I'm not gonna set up a queue that has a 6 and a half hour time out. I'm just I'm not gonna do that. And so these these Pseudodemons are just they're just console commands that just keep going. And so what they do is they they they sit there and they look over and over and over, and there's a there's, like, a a sleep, so it doesn't kill everything.
Aaron
00:31:30 – 00:31:49
It's like a 7 second sleep or something. Just looks over and over and over for new imports. And when it finds a new import, it picks it up and it builds the handler, and then it starts running. And that can run for 6 hours. And then when it's done, it just dies, and then it gets picked up the next minute by the Laravel scheduler.
Aaron
00:31:49 – 00:32:12
And that that pseudo daemon starts over again, and it starts looking for imports. And so it's really, really, really nice because it's super fast, and I don't have to worry about cues timing out or dying or getting picked up twice, which has happened to us before. So that's kind of the impetus behind it and how we use it, and it's it's great to be able
Jake
00:32:12 – 00:32:26
to interesting things here. One of the things I think is interesting is the little bit of hackery. I don't wanna call it a hack because it's not really hack, but it's sort of a hack. Right? Which is that you have this idea of don't yeah.
Jake
00:32:26 – 00:32:26
There you go.
Aaron
00:32:26 – 00:32:27
Call it creativity.
Jake
00:32:28 – 00:32:34
Overlap idea. Right? So, like, in the scheduler, you have this idea of Mhmm. You run this command every minute, don't overlap. Right?
Jake
00:32:34 – 00:32:44
And what that does is it sort of puts a mutex in place that says, go look in the cache and see if this thing is running. You know, so once the scheduler comes up, it looks at it and says,
Aaron
00:32:44 – 00:32:45
hey Benco.
Jake
00:32:45 – 00:32:54
Give me the mutex of this command. So look at the name of the command, look at when it's scheduled to run, etcetera, etcetera. See if it's already running. If it's already running, don't bother dispatching it again. It's already running.
Jake
00:32:54 – 00:33:03
Don't worry about it. So so your little pseudo daemon kicks off every minute, doesn't die. Just run run run run run run run run run run run run run run run run. It comes up a minute. It says, nope.
Jake
00:33:03 – 00:33:08
Already running. Don't worry about running it again. Run run run run run run run run run run run run run run run run run run run run run run run run run run run run. Right? And then if it ever dies, then it does pick it up again.
Jake
00:33:09 – 00:33:28
So what you said is true. Like, right now, you know, in scheduler, and I know I know, Fredrik van der Hirten, Spasi, again, does have a blog post out there about running scheduler on, like, a sub minute intervals. Like, there's he has something out there. I haven't looked into it a ton, But that's the idea. Right?
Jake
00:33:28 – 00:33:38
Your your thing runs, sleeps for 7 seconds, runs again, sleeps for 7 seconds, runs again. Or I think that's configurable too. Right? You can actually configure how long it runs for. But It is.
Jake
00:33:38 – 00:33:46
That's how it works. So for any of you out there being, like, how does that exactly work? That's how it works. So it runs runs runs runs at a minute. Check, oh, it's already running.
Jake
00:33:46 – 00:34:12
Don't bother spawning a new process, just keep running. Other question I have is then, so if you have a pseudo daemon running, it's not a general purpose pseudo daemon. So when you say you have 6 of them running, the reason you have 6 of them running is because if, you know, one of them is running and it's specifically set to say, let's import the property taxes from, you know, southern the southern United States. I have no idea. Right?
Jake
00:34:13 – 00:34:31
If that thing's running for 6 and a half hours because it's picked it up, it can't pick up anything else during that 6 and a half hours. It's running. Right? And it's going to continue to run. So if you want to have it as something else, pick up another thing, you have to have another Pseudomon that's specific to that particular thing.
Jake
00:34:31 – 00:34:46
Right? So that might be one word of of note there, is that, you you know, you kinda have to have a specific process for each one because it's gonna be you know, if it's gonna run for 6 and a half hours, it can't do anything during that 6, 7 6 and a half hours except for the thing that it's doing. Right? You know what I mean? It's not like it's good.
Jake
00:34:46 – 00:34:54
It's not like you get a fresh process every instance, and it's gonna kick it to the queue and start working it in the background. It's not gonna happen. It's just that one Correct. Pseudo daemon that's running. Nope.
Jake
00:34:55 – 00:35:10
But the other thing that you said is, again, like, this is not something you have to put in your cron, and then it's gone. You have no idea it exists. So oops. We're moving servers from this AWS box in the east to another AWS box in the west. Totally forgot to move that cron job over with that server.
Jake
00:35:10 – 00:35:16
It's all in version control. It all moves with the code, which is super handy, and it's visible to everybody. Everybody can tell that it's there.
Aaron
00:35:16 – 00:35:17
Exactly.
Jake
00:35:17 – 00:35:38
So, yeah, so many great things there. This is, we actually have something that we do where we run every minute, and we just say it's called, I think it's just, like, file watcher. It's a command that's just called file watcher. We basically have this location that we watch, this s 3 bucket that we watch, and we say, if any files come into this bucket that are named any of these things, dispatch this command and take care of that particular file. Right?
Jake
00:35:38 – 00:35:54
So we say, like, here's the file pattern match that you should look for. So it's called end of day call recordings dot, you know, zip. When you see that, run this command, like kick it out. But you do have to wait for a minute, and sometimes it's really annoying to have to wait, you know what I mean? It's like, okay, when is it gonna happen, right?
Jake
00:35:54 – 00:36:07
Well, if you just had one of these pseudo names, it would just say, you know, pick it up as soon as it comes in, kick up that job to the queue, you're done. Right? So that, like, sub minute interval Mhmm. Checking was is would be pretty handy. So anyway yeah.
Jake
00:36:07 – 00:36:08
Lots of cool stuff.
Aaron
00:36:09 – 00:36:36
Yeah. That's that's one of the things, like, working in working in these, you know, non sexy kind of businesses, you you end up with a lot of, like, nontechnical users. And so that's one of the benefits of this Pseudo daemon thing is one of them, processes our internal submissions, like our internal forms. And I would have people that would be like, hey. I submitted this form, and it it hasn't processed yet.
Aaron
00:36:36 – 00:36:49
And I'm like, well, it runs, you know, before the demons. It runs every minute. Yeah. So just, like, give it a second. And the the bad part was, like, it would it would pick up the form and validate it.
Aaron
00:36:49 – 00:37:19
And then if it was invalid, then it would put it back down, and another process would pick it up and try to res auto resolve the form errors. And if it could, it would put it back down and then get picked up for validation and then get picked up for saving to the database. So if you only run it every minute Right. We're talking 5 minutes, and the person is on the phone with the client waiting to see, like, did the form go through? And so when it's a pseudo daemon, it's like, it all happens within, you know, 10 seconds and you can tell the client, great, we got it.
Jake
00:37:19 – 00:37:19
Right.
Aaron
00:37:19 – 00:37:37
And and you're good. And the other thing that I like about this process is it's very vanilla Laravel. Like, it doesn't require, you know, any async PHP. It doesn't spawn background processes. It's just like it's a goofy scheduled command Right.
Aaron
00:37:37 – 00:37:45
That that stays alive. And so it's very straightforward. There's no, like, react, PHP, or anything like that. It's just a command.
Jake
00:37:45 – 00:37:47
That can run on a default
Aaron
00:37:47 – 00:37:48
And that's kinda why
Jake
00:37:48 – 00:37:53
I like it. You know, whatever. You can run a Pseudomon on that. Bingo. Really simple.
Jake
00:37:53 – 00:37:54
I love that too. Yeah. Exactly.
Aaron
00:37:54 – 00:37:54
It's like
Jake
00:37:54 – 00:38:12
again, it's like one of those things. Less headache, like, when you're moving things around. Right? Those are just, you know, we all have those applications where it's like, remember, if you're moving this, you have to install LDAP, you have to install ImageNet or whatever it's called. I think I can never be like I wanna say ImageMagick, but it's not ImageMagick.
Jake
00:38:12 – 00:38:18
It's like a magic or something like that. Right? You gotta install these different things. Mhmm. And it's so nice when you just have something like this.
Jake
00:38:18 – 00:38:22
It's like, oh, yeah. Yeah. If you have you have PHP 8, you're good. You're all set to go. Right?
Jake
00:38:22 – 00:38:23
It's all done.
Aaron
00:38:24 – 00:38:28
Yeah. Is there is the rest of your kernel running? Okay. Then you're fine. Like Exactly.
Aaron
00:38:28 – 00:38:29
It's it's okay.
Jake
00:38:30 – 00:38:43
Yeah. Well, dude, Aaron, it has been an awesome chat. I have, learned a ton. And like I said, recent, but big fan of your work. I feel like you're doing some really cool stuff, and, I I told you the other day.
Jake
00:38:43 – 00:38:49
I was like, I read through your whole shed quarters blog. You're like, the whole thing? Like, literally, the whole thing? I was like, no. Seriously.
Jake
00:38:49 – 00:38:56
Like, I sat down for I don't know. It was probably 15 minutes. I I just skimmed it, but I looked at all the pictures mostly. We all like to see the pictures. You know what I mean?
Jake
00:38:56 – 00:39:08
Yeah. And so, it was a fun adventure to read through that. And then but, yeah, I'm gonna dig into your blog post stuff for sure because I feel like it's a it's a gold mine there. And, appreciate you putting your work out there. And if anybody has not checked out your stuff, they should definitely do so.
Jake
00:39:08 – 00:39:12
So check out Aaron Francis. Aaron, what's the easiest way for people to keep up with you?
Aaron
00:39:13 – 00:39:22
On Twitter, Aaron, a a r o n, d as in Daniel, Francis, and then, my website is just Aaron Francis, the middle initial. So
Jake
00:39:22 – 00:39:24
Hey. Anything else you wanna shout out
Aaron
00:39:24 – 00:39:24
before we
Jake
00:39:24 – 00:39:25
wrap this up?
Aaron
00:39:27 – 00:39:32
No. Thanks for having me on, and congrats on 101 man. Episodes. That.
Jake
00:39:33 – 00:39:42
Thanks everybody for tuning in. You can find shown us for this at northmeetsouth.audio/101. Follow Aaron. Check out his blog. Lots of good stuff on there.
Jake
00:39:42 – 00:39:51
Of course, if you liked the show, rate it up in your podcast real choice. 5 stars would be super appreciated. Until next time, folks. And we have Michael Dorinda back. We'll see you later.
Me

Thanks for reading! My name is Aaron and I write, make videos, and generally try really hard.

If you ever have any questions or want to chat, I'm always on Twitter.

You can find me on YouTube on my personal channel or the Try Hard Studios channel.

If you love podcasts, I got you covered. You can listen to me on Mostly Technical.