Episode Transcript
[00:00:00] Speaker A: Welcome to Within WordPress, the podcast about people inside and outside of WordPress. With us today is somebody you in I think in the, the most interesting way is the person who are we are going to talk to is extremely good in testing. And you might wonder, testing what? But I'll let, I'll let Kevin answer that question. Welcome, Kev.
[00:00:28] Speaker B: Hi.
I guess I should introduce myself. I'm Kevin Ohashi and I run Review Signal, is the main company I've known for running, which is a web hosting review website.
And the thing that really took off within the reviews that I do is called WordPress hosting benchmarks, where every, almost every year I would take a large cohort of many of the largest players in WordPress hosting, and a whole bunch of small ones too, and run them through a rigorous set of tests and publish the results.
And that's the main thing I do in the kind of WordPress world.
I do some stuff too, and maybe we can talk about those later, but that's the main one.
[00:01:24] Speaker A: We can. And I'm curious for that, because when I say testing, you kind of explain already.
But the yearly results of what you publish have been a benchmark for me since forever.
How long have you been running Review Signal?
[00:01:45] Speaker B: So Review Signal launched in or on September 25, 2011, I think was the exact date.
[00:01:56] Speaker A: There you go.
[00:01:56] Speaker B: The benchmarks, that's over a decade published in 2012. I think the first ones were maybe March 2012.
[00:02:06] Speaker A: Yep.
[00:02:08] Speaker B: I'm sure if I'm wrong, somebody on the Internet will correct me.
[00:02:15] Speaker A: Little sidestep. I heard, I, I heard people in these types of recordings and podcasts, they add stuff that's factually incorrect just to get people commenting on it and saying, no, no, no, that was. And, but you're wrong. And you know, I think that's hilarious. But I don't, I, I'm not going to, I'm not going to say that that's what you were doing right here. But what I am going to say, that's a very long time of testing.
Can you explain us a little bit of how that works? Because, Mr. Ohashi, you have over that long decade done a lot of testing.
How does that start? How did you come up with the idea? Let's start there.
[00:02:55] Speaker B: Okay. So the history of the benchmarks, they weren't exactly my idea originally.
They were essentially a benchmark met between myself and Jeff King, who was something, something of hosting at GoDaddy.
I think he the top guy for hosting at GoDaddy. I think they launched a new product and he Told me that he thought it was as good as anything out there on the market.
And I think it was probably over email. But you know, you can imagine the raised eyebrow of really?
And so I was like, are you willing to put your money where your mouth is?
Of course there was no money and I did it all for free, but figuratively.
And I think there was seven or eight companies when I had this idea. I went around to who was on there at the time. I think grabbed a digital Ocean siteground, maybe a WP engine.
I could look back and find the list, but it was a handful of the players that actually catered to WordPress a little bit. And, well, I don't think DigitalOcean actually did at that time, but they had a image and so threw it in there. And yeah, I ran some very, very basic benchmarking, load testing against them and published the results.
And at that moment, GoDaddy's hosting was about on par with everything else at the upper end, which was a fun surprise.
But as soon as I published this, it got so much attention and traffic and hosting companies started coming out of the woodworks like, oh, do us, we need to be in here too. And I feel like I started the next one virtually immediately afterwards with more companies. I think it almost doubled roughly in size. And I was just, how quick can I get this out?
And after that it was like, this is too much. I can't just constantly keep doing this, let's maybe make this once a year kind of thing.
That's how they were born.
[00:05:19] Speaker A: Yeah, it's not a. So most people think that you're maybe just turning on a plugin somewhere or have a script running on the server and just kind of turn it on, let it run for a couple hours and then call it a day.
But I don't know, maybe that's how it started. I don't know. But I've. I've been on the receiving end of.
Yeah, yeah, I've been on the receiving end of having instructions on what to set up so you could test.
When I was working at. At servo.
Can you. Can you kind of run us through, like, what is the total set of things you test and the duration and stuff like that? Because I think that's probably one of the things that might not be super clear. When you just go to the website, it just states the results basically, but they don't, you know, the full grasp of what is your magic as far as you can divulge.
[00:06:12] Speaker B: Well, I divulge it all if anyone wants the Full magic. All they have to do is go to wphostingbenchmarks.com and. And then there's a methodology methodology page that you can find in the navigation that explains hopefully everything that I'm doing.
But the main components are.
Well, there's two kind of tests in what I do. There's impacting and non impacting tests. Impacting tests are the ones that I use to determine the kind of ranking, award status, rankings award status because I don't actually rank them in any order except by grouping. These ones did well, these did pretty well. And then there's everybody else.
So those are the impacting ones and then there's the non impacting metrics where I think these are interesting to record.
I think some people might get value out of them or see a way to use them to help either improve their services or figuring out which service might better suit them.
But I don't think they are universally applicable to news as a determinant in how we place the companies.
So the ones that impact the two, there's two, there's the load testing and the uptime.
Can you keep a server online the whole time and does it respond to large amounts of traffic? If you can pass those criteria, you get a good grade in with the benchmarks than there are the other ones. The other ones are stuff like. One of them is a plugin I just threw together.
It's wp. What is it? WP Performance Tester. And it was designed to test the CPU and test the database of the hosting that you're on. And it gives you a neat little score, which is nice, but there are problems or limitations, I should say, with using it, which is why it's not an impacting thing.
If you have everything running on one server, your database and the web server, the communication from the web server to the database is a lot quicker.
So if you're measuring the latency of that, it's going to look a lot faster.
However, as you scale, you don't generally want to have everything all together on one thing sometimes and you come up with, you know, scalable distributed architecture. And that could look really bad if you're just measuring single threaded performance.
So it would be really unfair to say, you know, this is a cluster of hosting that can handle billion users or something, you know, silly, but it's going to be 50 milliseconds slower because every request has to go through some clustering. Could be all over the world level clustering versus one machine that's single threaded in it.
[00:09:55] Speaker A: Yeah, in a Way I would imagine any cloud hosting, meaning just whatever hosting it is, but deployed on any of the known cloud providers. So Azure, aws, Google, they inherently have a distributed way of, you know, their services. The database server is a particular server, the web server is another particular server.
I would imagine that's something you're running quite quickly when you start testing like this.
[00:10:34] Speaker B: Those differences, the different architectures. Yeah. I mean based on the results of some of these, on that test specifically, I can kind of guess what kind of architecture we're looking at when you know, I see it's doing thousands of queries in a second versus a few hundred or it's really bad. That's also an option, just really terrible performance.
But hopefully that's not the case.
[00:11:05] Speaker A: Five queries per second.
[00:11:07] Speaker B: Right. You discover if that's the case when you send a lot of simulated users and if it actually holds up as you send thousands of users, it's probably distributed. And you know, that slowness is not really an issue when you're looking at the real world versus if it crashes immediately, you know. Oh, maybe that is just the performance.
[00:11:31] Speaker A: Yeah, yeah. So when the tests are done on the load basically with that plugin.
[00:11:43] Speaker B: Do.
[00:11:43] Speaker A: You have like criteria of where it should be to even be considered part of your test or how does that work?
[00:11:52] Speaker B: Do you mean the, the plugin results that test the database and the.
[00:11:56] Speaker A: Yep.
[00:11:57] Speaker B: Cpu?
No. So that's done after a company has already signed up and all I'm doing is just recording the data. It doesn't impact how, how they place. It's just interesting. It's the same with like I run the webpage test from all over the world and record the response times. You know, just run it to see.
[00:12:20] Speaker A: Yep.
[00:12:20] Speaker B: Are these companies effectively, are they using a cdn? Are we going to see, you know, really good performance in North America, Europe or wherever they happen to be based? And terrible. When I use an Australian location or an Africa location.
It's just sort of, it's interesting to know but I don't think that's the determinant of this is a good host or a bad host. There's different use cases.
Yeah.
[00:12:48] Speaker A: Because that kind of was my follow up question. Like there's so many ways that you can measure. Right. And we'll get to some of the stress testing later. But the, just the distributed part.
The part.
Sorry. And the next component is what is part of the stack.
In some cases that includes a layer of cloud layer or varnish.
How, how are you determining the, the things you, you are.
You Know, when does it become. This is too important to not ignore. To ignore or do this. I. I need to have this part. Or do I Compare this from one to one? So there's a. There's a bunch of popular WordPress dedicated hosting companies who have, for instance, that Cloudflare layer.
I would imagine they score on the whole, they would score better than those that don't.
[00:13:51] Speaker B: Be interesting.
[00:13:52] Speaker A: Long pause.
[00:13:55] Speaker B: Yeah, I'm not sure. I've never broken it down exactly like that. And when you say a Cloudflare layer, do we mean Cloudflare specifically or a cdn?
[00:14:09] Speaker A: Well, I think I mean a Cloudflare specifically as in their.
Yes, they're cdn, but they also have the.
[00:14:22] Speaker B: Whole.
[00:14:24] Speaker A: Caching layer specifically for HTML. So you can move it off server to Cloudflare's layer. That I don't. I mean, technically, is that still. Would we still consider that cdn?
[00:14:36] Speaker B: I mean, isn't that what a CDN is? Effectively, it's putting offloading content onto someone else's server to serve it from there.
[00:14:46] Speaker A: I mostly process CDN as the content being the assets and everything that go into that bucket.
[00:14:54] Speaker B: I think that's the like original definition. And I, to me at least I feel like we've evolved where content.
[00:15:01] Speaker A: Are you calling me old?
[00:15:02] Speaker B: Everything.
[00:15:03] Speaker A: Are you calling me old?
[00:15:05] Speaker B: I will let you call yourself whatever you like.
[00:15:10] Speaker A: That is a smart answer. I'll call myself old. I am on the Internet professionally for, for at least a few decades. And we'll just leave it at that.
No, but I mean that in that. Sure. That's part of the architecture. Right. So it's, it's an advantage of a hosting company if they choose to have something that is available.
And don't quote me on the amount of data centers Cloud for it has now, but it's above in the three hundreds, I think somewhere.
It's like a ton.
[00:15:40] Speaker B: That would be an obvious advantage disclaimer.
You know that I have a bunch of Cloudflare stock and pretty happy about it.
So anything is not a. I know.
So you know where I'm coming from.
[00:15:54] Speaker A: Yeah, yeah.
[00:15:55] Speaker B: I think they're a great tech company.
I don't know if it's unique in terms of CDN companies. I haven't looked lately, but I figured fastly and bunny and some of these other ones can probably do similar things. And I know some companies have rolled their own CDNs to try and replicate Cloudflare, so.
[00:16:18] Speaker A: Yeah, but you, you might do it on the CDN CDN side of things, but the whole suite that's built on top of it inside of Cloudflare. My word, that's going to take a lot of time, money and energy to get that up to at least some assemblance of, of a same level.
No, but I was what, what I was getting at that if you, if you have that part of your stack, you, you have an advantage. Right? So.
But I think you kind of gave the answer already. You don't factor that into.
Do I compare these? Yes or no.
[00:16:54] Speaker B: So yeah, I don't compare them and break them apart. Of this company uses this stack, this one doesn't. The way I look at it is I'm thinking about it from a consumer standpoint where at the end of the day the end user effectively doesn't care of what you've stuck in the middle there. It's the results, you know, what do they see, what is their experience? And if you can replicate the essential parts of the Cloudflare stack, the whole suite would be probably impossible.
But if you can build a good CDN and maybe integrate that a little nicer into WordPress, I could see maybe some hosts might be capable of doing something like that or building extra on top of Cloudflare because it's pretty configurable.
I really care about the end result and that's why the big test is the load testing. It's looking at what happens when you send thousands of users at these sites and what is the experience of those users they're getting.
Everything else we're measuring at individual or low scales. There's a lot of caveats like is this a distributed architecture? Is this not. Do they have a cdn?
You know, but at the end of the day, until you're actually looking at as close to real world experience and performance, all those other things are just kind of hints at what's going on.
[00:18:38] Speaker A: Yeah, I can see that.
For me, I think the problem, the probably the, the thing but again I'm. I'm not your typical consumer and wanting to understand these results, but I think the three things that would make me curious is like are you using Cloud for part of your stack?
Do you have Object Cache Pro turned on by default?
Combined with Redis for instance, and Til's other product relay is also a good indication of, you know, quite a bit of performance gain to be had there.
Those would be things I would find interesting because without them it would tell me a different story, I guess, I don't know, maybe that's an assumption.
[00:19:27] Speaker B: Deep in there. There is actually a link to that data that you could filter it down if you wanted. I don't know if you. Do you have the website open right now?
[00:19:39] Speaker A: I don't, but I can certainly look it up.
[00:19:43] Speaker B: So if you go to the homepage and then you click on.
So it says View Full Results under each price tier there, there's a yellow button of View Full Results that gives you the full results for that. Price tiers, benchmarks.
There's a table. The first table you see the companies and products.
[00:20:09] Speaker A: Which URL am I using?
[00:20:11] Speaker B: WP Hosting benchmarks.
[00:20:18] Speaker A: Yeah. So for those of you listening in on this, can you repeat it one more time? You go WP Hosting Benchmarks.
[00:20:27] Speaker B: Go to WP Hosting Benchmarks.
And if you look down or scroll down slightly, there's a View Full Results button. It's a big yellow button and there's one for each price tier that I do. So it says 2023 participating hosting companies, WordPress or WooCommerce. And there's price tier and then there's that big yellow button of View Full Results.
And so if you click that, the first table you see is the companies and the products that were submitted.
It's pretty basic, you know, it gives you the basic high level stuff that most people are looking at.
At the very bottom of the table there's a little link that says View Full Products table.
I'm guessing almost nobody ever clicks this, but it's there.
But that has all the things like CDN and I just put what the companies put in when they submit it to me. So you can actually see if you clicked on the cheapest, the under 25 and earn that. You can see a bunch of them wrote they have cloudflare CDN or Enterprise Cloudflare cdn and there should be another column all the way somewhere in the right there about caching technologies.
So I can see one of these uses ocp, parentheses redis.
Yeah, Object Cache Pro. I wrote it out fully.
Yeah. So you can. That data is actually there.
[00:22:04] Speaker A: That's fun.
[00:22:04] Speaker B: Although almost nobody has probably ever noticed that.
My goal is to make this as transparent as possible and hide nothing. But the biggest challenge when you have so much information and data is.
[00:22:19] Speaker A: Yeah, I know.
[00:22:20] Speaker B: Consumable and consumable to what audience?
And it's funny, this is hard. I'm not the normal user like you're talking about the regular consumer. Actually I think you are the regular user of my website because none of the consumers are looking at it. It's all the very nerdy techy developers and people actually care about scale.
Even Though I'm building this with what does the end user want in mind? Like, what matters to an end user?
I think it's mostly the developers, decision makers, the people who are deeply into WordPress that come and read this. Some consumers do, but the ones who get really into it are the people like you.
[00:23:10] Speaker A: Okay, you got me.
That's a different profile. So the reason I kind of assume this is I send my clients to this, to this page so they can figure out why I'm choosing what I'm choosing.
But I may be not your typical, typical one in sending this through and in general, I think so. One of the reasons I particularly like your approach is that you are basing every single point of data on researched data, data.
And if you in general just Google any which version you can think of of WordPress hosting, reviews or comparisons, or any, literally any version of any search term related to that, you'll end up in 99.9% of the search queries. You'll end up with something that is entirely and purely affiliate driven and driven, but also drivel because it's just someone's preference or whomever pays them the most.
[00:24:21] Speaker B: The best case scenario is that it's just some guy's preference and it only goes downhill from there.
[00:24:29] Speaker A: Okay, do explain. What scenarios do we have?
[00:24:33] Speaker B: Well, in terms of the reviews, I mean, that's why I started Review Signal. The idea was I've been, I may have been doing Internet stuff about as long as you have. I'm not sure when you started. I started in 90.
[00:24:47] Speaker A: 96.
Yeah, 2-1-96.
[00:24:50] Speaker B: Sorry, phone call just took over the audio there. Spam calls. Thank you.
So I started Review Signal, not the benchmarks, because after more than 10 years of building stuff, recommending hosting, you know, just doing general web stuff, I was really frustrated that still the best advice was, oh, go check like web hosting talk and just see which ones people like at this moment.
And it's like, how can that still be the only way? You know, just the current read of the room, is it just changing like this? Nobody trusts any of these reviews.
How can that be better?
So that's why I started Review Signal. And other than a couple attempts that I've seen people make, there really hasn't been any serious ones. The best attempt I saw was basically a guy copied what I did. The original one was scraping, scraping Twitter and analyzing the sentiment and opinions about companies.
And yeah, yeah, he just did. He put zero affiliate links and he published it. I was a little bit hurt because I do have affiliate Links because I need to eat and wanted to try and do something that was honest, transparent and you know, afforded a roof over my head.
But he did it, no affiliate links and got. It was wild. Everybody's like oh this is great like and whatever. And I'm just like I just compared the data. I was like hey look, it's the same if you didn't trust me before.
But that's actually we're friends now. It's. He sold it I think to one of those SEO spammy companies and I would never look at the content there. Now I think it just redirects. But yeah, the best case scenario is it's some guy's opinion of. It's well intentioned of. Here's what I use. From my experience, I like this and that's the best case a lot of times.
[00:27:21] Speaker A: What's the worst case scenario?
[00:27:22] Speaker B: I mean the worst case scenario is it's just pure.
Well the, I was going to say purely affiliate driven but I think the worst is when it's actually just owned by the company pretending not to be referring to them.
Yeah, some companies have and probably do although that do actively. I'd have to probably do a little research to find.
Yeah, they create their own review sites or sponsor somebody to create sites that happen to put them at the top and recommend them.
[00:27:59] Speaker A: Yeah, I mean this is the sleazy part of the Internet I guess where there's no real validation of what it is that I'm seeing and, and the intentions that are, are behind building the thing that I'm seeing.
I, I've. I've seen many dishonest, factually incorrect reviews to the point that you know, again, again I'm not the one googling these things but the results that you're seeing are just not close to what the actual truth is. Which yeah, basically why I like so much how you're doing what you're doing and with the intent that you're doing it.
If you look at the.
Let's assume the best case scenario. Right. So somebody does it because of an opinion.
Do you have any thoughts on. On how they're ranking in general? Because they're obviously not doing it the way you are doing.
Do you, do you have any.
[00:29:12] Speaker B: I mean a lot of them are very, they're just opinions. A single opinion, a single data point.
And it's not that it's invalid, it's that it's statistically insignificant.
And so when I thought about what I was doing, my biggest goal in my work is extracting myself out as Much as possible, my opinion about any company is the goal is to get rid of it and I will work with almost any company and put them all through equal treatment.
And I've even had complaints about that from some people. It's sort of like, oh, you know, why do you work with large negatively associated with company, you know, like you do reviews, shouldn't you know better? And it's sort of like that may be true that they are not as well liked or don't provide as good services as other people out there.
But A, they're massive. B, you have to treat them, they are relevant, you have to work with them and treat them.
You know, it's not that they're, well, most of them are not doing anything nefarious and scummy at least.
[00:30:43] Speaker A: Yeah.
[00:30:44] Speaker B: It's just often just pricing kind of stuff.
[00:30:48] Speaker A: Yeah. I think those nefarious days are on the whole kind of done with hosting companies. I mean the bigger you get, the.
[00:30:58] Speaker B: Less they have to do it. But I think you still probably see it with smaller younger companies. Not all, but the incentive and their rewards are there. That if it works and you can build off of it and then there's no consequences is the other problem.
[00:31:20] Speaker A: Isn't that part of what the cool kids nowadays call growth hacking? Would you categorize it under or nefarious to me is like.
[00:31:38] Speaker B: There'S a line and I know some companies cross it.
[00:31:43] Speaker A: Yeah.
[00:31:44] Speaker B: Some people probably would brag about it and put it on their cv.
I would not be thrilled to know that. And I think the problem. Could you give one concrete example?
I mean.
[00:31:59] Speaker A: Oh yeah, yeah, yeah.
[00:32:00] Speaker B: Because you know, the admins of these communities do not like being astroturfed by fake employees pushing that.
[00:32:08] Speaker A: Yeah.
[00:32:09] Speaker B: And they're not the only ones who do this. Lots of companies and it'd be interesting to compare ban lists from these communities. But that's not exactly public.
[00:32:19] Speaker A: Yeah, yeah, yeah, I think, yeah, I, I mean I, I, I remember these types of stories.
I, I've never paid too much attention to it because I, because I, I go about finding my hosting companies in a very different way anyway, so they didn't change CEOs.
[00:32:36] Speaker B: I actually had a sit down with two other staff. Not at the cloud fest that just happened, but the one before which was interesting.
It felt like they were saying all the right things but to me words are very cheap. It's proof is in the pudding and because of what I've written, I even, I think I told them this. I was like, I have relationships with a lot of These admins. Because when you write stuff like this, you know, these. The admin people. And when these kind of things happen, they tip me off. They're like, hey, like, have you seen this shady thing that's happened in web hosting? And I told them upfront, you know, I've been told, you know, similar things were still happening within the past year. Year at that point.
So you're saying the right things to me, but the. What's happening in the team people, you know, you may at the top say that we're doing something else, but if you have 50 people working for you, the way people go about things doesn't necessarily change overnight. And. Yeah, yeah.
[00:33:58] Speaker A: And it's hard to track as well, I guess.
[00:34:01] Speaker B: Yeah.
[00:34:02] Speaker A: Especially if you look at the.
Yeah.
[00:34:05] Speaker B: Oh, I was gonna say especially like at the scale of, you know, large teams. And then if you start using aliases, because I'm assuming people are gonna get smarter and smarter and now it's just gonna be AI, you know, have an AI agent go out and write 500 reviews for my company, please.
[00:34:29] Speaker A: That's horrible.
That side of AI.
[00:34:33] Speaker B: Welcome to the future.
Yeah.
[00:34:36] Speaker A: I had a conversation with a client today who is relatively low tech, but understands his website. It's an e Commerce Store. WooCommerce fully grasps what he's doing, but finds it difficult to translate that into how do I use new technologies and stuff. So he asked me, can I. Can I have a call with you and just sit down and talk about what I could do with AI?
And the way the questions were asked, you know, somebody was whispering in his ear, like, dude, you really need to start using AI. And it can do this and this and this and this. And all the things I kept hearing were not necessarily in. In the continuation as we just referenced AI, but certainly on the edge of.
So I said, look, what, What?
[00:35:25] Speaker B: Gray.
[00:35:27] Speaker A: Gray. Gray. Let's call it gray.
I said, what do you. What. What do you want to stand for? Like, the product descriptions that you have, the blog posts that you write.
If you find it difficult to write those, then AI is wonderful as a solution. Just have IT generate examples of what that could be. You read them, you parse them in your head, and then you write your own.
And if there's a sentence or two or three you can't figure out, you just say, I kind of want to say this, but I don't know how to best say it. AI will help you with that perfectly.
And I said, if you just stick to that, you'll write much more content much faster. Much better.
And that's a perfect example of how AI can help you. And then there's a whole bunch of other stuff like have it fill in all the alt images, text and descriptions, SEO meta descriptions, all these things. Sure, AI have, have @ it.
Not going to hurt you.
But then you know, that's the white side of the, the, the, the, the scale that he presented to me like and, but all these other things, don't, don't use it because in, in you may get away with it for a while, but at some point people are going to find out that this is not how it's supposed to be. This is not you, you, you're, you're playing with, you know, the, the sentiment here and, and you're manipulating people in a way that they didn't, you know, they don't expect that from this particular site. It's very much trust based and they. Okay, I get it, I get it. Yeah, well, you know, I'm just trying to figure out where I need to devote my attention.
That's a good example of how AI potentially, if he hadn't asked, he would have just started doing all these things. And then, you know, the web deteriorates pretty fast if you have automate automated ways of doing this, whether that's a review for a hosting company, whether that's a review for a product. And I think you can kind of guess where this was going.
You're just faking it and you're going to get caught at some point.
[00:37:37] Speaker B: That'S you feel like you hope that it will be caught because that would be how what should happen.
Whether you actually tbd.
But I agree with you and that path that you're talking about is something that I've been dealing with for a while for one of my big sites that I work on is So I run Portugal.com and people always ask, oh, why don't you just use AI to like write all this content? And yeah, I will say, at least as far as I know, I have a team of writers who write all the content and there shouldn't be any AI.
And the reasoning is real. People read this website and are going to go do things. So if I write, I actually for the record, write, have written 0 articles on there because I'm not Portuguese and I don't live there and have no business advising anyone about what they should see. Do you know, I am not an expert.
[00:38:55] Speaker A: I have to segue into the question that I need to ask now. I have to segue it. Why do you own Portugal.com.
[00:39:04] Speaker B: So the first business I ever got involved with was domain names way, way back. That was the first online business I was in.
I'm still in that world. And at the end of 2021, for some reason I decided to click on one of the dozens of emails that, that I get. You know, this is a great domain auction. You have to go, you know, this is the inventory. And I clicked it and I saw Portugal.com on the list and thought, huh, that actually doesn't seem crazy because most of the domains that people are always trying to sell, they're either crap or the reserve price is just insanely high that it doesn't make sense that anyone want to buy it.
But Portugal.com was listed there at what I thought was a reasonable reserve. And I went, huh, let me just take a minute. And one of the other businesses I'm involved with is called Name Bio, which is the largest database of domain name sales. So if you're trying to do comps. So as I, I guess I have equity in this company and built tech with it, I was like, I have access to all this. Let me look up what all the countries that have ever sold for and you know, most, there's not many of them and most of them sold 10, 20 years ago and for a lot of money higher than this. So I thought, huh, that seems like a good investment. But I don't have that kind of money.
So I told a friend about it who was in Bitcoin. So clearly gambling on a domain name feels like a drop of water in a large bucket. And they were like, oh yeah, let's do it. And so he went for it and won the auction, sort of sat there and thought, okay, what now?
Well, can I develop it? Because it would be a waste.
I don't think we expect to win.
So it's just like, oh, what now? Okay, I don't want to waste.
Yeah, that's kind of like literally the reaction of like, oh, it's like the dog who chases, you know, after some, the car. Like, what do you do when you catch it?
Yeah, so we got there and it was like, okay, well I'd be up for developing it. It's like, I'm busy, I have other things to do.
You go for it basically. Here's the keys.
[00:41:57] Speaker A: Yeah, that's funny.
[00:41:59] Speaker B: And so over about two weeks, had to like overnight build this site because it used to be developed too.
And so I didn't want to lose the SEO value, so I had to like recreate a 20 year old site just Unprompted and as quick as possible.
And yeah, been running it since. Got a team together, a team of writers who do a great job.
They're all either Portuguese or live there. Expats living there. And it's just, it's an English language site for people who want to visit, move or live there.
[00:42:37] Speaker A: Interesting. That's funny how these site. So you, you've actually done it. You've, you've, you have a side project and it's working.
I mean that alone should inspire people because I don't, I don't want to start explaining you how many potential side projects I have on my list of things to do.
I'm, I'm, I don't have the time to build them. But that, that sounds like an achievement. Good on you, Kevin.
[00:43:07] Speaker B: It was sort of an unexpected detour that I'm very happy that ended up happening and I've certainly learned a lot about Portugal in the meantime.
We, after we purchased it, my or my girlfriend and I went there in the summer and drove around virtually the entire country, north to south, stopping in.
[00:43:34] Speaker A: And you were like, I own you. I own you.
[00:43:39] Speaker B: That maybe not that far. Just. Yeah.
Trying to check out all the cities, get pictures and media content for, you know, we're going to do guides.
[00:43:51] Speaker A: Oh, you were smart about it as well. So it was a business expense.
Good, good, good.
[00:43:57] Speaker B: I mean why else would you stop in some tiny little village to take a picture of a thousand year old castle and then do it 400 more times?
Yeah, yeah.
[00:44:10] Speaker A: Funny.
I.
Let me.
I think the, the thing. Because we're segueing back to the, the testing.
The, the thing I am extremely curious about is you have a methodology that you have published as well on how you test.
So the, the stress test, not the low test from within, but the actual stress test.
What can you, what can you tell me about that?
[00:44:48] Speaker B: What would you like to know? Like the history of it or what it is?
Like what?
I can elaborate all you want. Where would you like?
[00:44:56] Speaker A: All right. Okay, so I, I'll give you more direction. So the, the, the, the stress test that you do does a particular set of things.
What are they? And yeah, what's it called? What is K6? What is, you know, what is the, the mix of things that you have.
[00:45:17] Speaker B: So there's three stress tests and they're each documented.
I call one of them the Lodestorm test.
That's named after the previous load testing platform that I used because it allowed more complicated load testing. And they were a partner sponsor for many, many years until they went out of business.
So that's affectionately to me at least known as the Lodestorm test.
And what that test did is it simulates somewhat normal WordPress user behavior.
So it would go to a homepage, it would log in as a user, it would boot, browse around different pages. It's very basic.
This is what most people do on the Internet. They go around, click some stuff and sometimes log in.
That's the load storm test.
There's a static one which literally just hits the homepage to generate as much load as possible. That is the unofficially, I call it the did you install caching test.
If you installed any sort of static caching, that says, okay, somebody's requested that page recently, we don't need to go hit anything else and could just serve it.
Yeah, let's make sure.
If you fail that test, and I don't think any companies have failed it so recently, but somebody's going to go.
[00:46:59] Speaker A: Have it tell me.
[00:47:00] Speaker B: Oh, of course, yeah, they have before.
There is the occasional failure on it.
There shouldn't be. At this point, it's 2025, you should have a static cache, but you have to check.
You can't just assume. You have to test everything. So we test, I say we, I test that. And the third one is eCommerce. Yeah, machines.
The many machines.
And then there's the WooCommerce tests which because we do WordPress and WooCommerce specific and the WooCommerce is similar to the Lodestorp test except it runs WooCommerce specific patterns like add to cart viewing products because it's a different, entirely different WordPress site. Because it's WooCommerce.
[00:47:53] Speaker A: Yep.
There's a lot more going on.
[00:47:56] Speaker B: Yeah, that's by far the most intensive one.
[00:48:03] Speaker A: And then from the, you have the load storm and so what you mentioned for the static, that is the K6 part.
[00:48:12] Speaker B: So they all run K6 now.
So lodestorm is affectionately named after the old company that went out of business because I used to use multiple load testing tools.
K6 I would use for the static test. Lodestorm I would use for the more advanced ones. But when it went out of business, I migrated everything into Lodestorm. So.
But I just like to keep the naming.
[00:48:41] Speaker A: I guess that makes sense.
What I like about K6.
Yeah, well, there's nothing wrong with this nostalgia. I still love the cars from the 80s and the 90s the most as well. So, you know, you know, who, who am I to judge what, what I like about K6 is that it? It's actually free for anyone to use, right?
[00:49:02] Speaker B: It is open source. K6 is entirely open source.
You can download, use it for free, run it. It's super easy and it's, it's opinionated. It's for programmers. It's. You write code to emulate the behavior you want and I like that as a coder.
[00:49:27] Speaker A: So yeah, can you, can it do more than stress test? Can it actually do user cart flow like from a user going into a product, add it to cart, check whatever's next and then, and then, and then whatever step you define. Does it go that far?
[00:49:45] Speaker B: I believe so. I mean that's what I do with the woocommerce test is go to a product page, add it to cart, go to checkout, you can have it.
Anything a browser can do, it's designed that it should be able to.
It even has extensions that I think you can actually make it use a browser and actually run it through a browser with Selenium, I'm guessing, or WebDriver. Yeah, I haven't actually done that before through the actual browser. I think it'll be a lot more resource intensive to scale up, but it is capable of it.
[00:50:27] Speaker A: I think.
I'm trying to remember there's a tool, I forgot the name that allows you to spin up all these types of different browsers that then emulate traffic under the different browser behavior. You know, mostly to simulate real traffic and see where edge cases are. I forgot what the name was, but it sounds very similar. Like you would you. By what you just said, K6 could be turned into something like that.
[00:51:07] Speaker B: I mean that's the beauty of it. It's open source.
You can turn it into whatever you like.
You can go get the source code and make it. You can build extensions and that actually kind of segues into the project that I'm working on now because there's K6 the open source and then there's K6 Cloud which was the common monetization. Let's offer a commercial offering to provide services around our open source product.
So K6 Cloud would let you run the test from the cloud, distribute them, save all the data, visualize the results.
So they got.
That's who I used to use for the benchmarks every year they were a sponsor. I really liked the team there. They were great.
Um, but they were acquired by Grafana not too long ago and have been integrated into Grafana Cloud and Grafana is, doesn't have that same. They're a billion dollar unicorn company, not a scrappy Startup and so that not at all like flexibility and like sponsorship, everything kind of went out the door and the prices increased dramatically.
And so one reason I picked K6 is that core of it is open source and you can take it and build whatever you want on top of it. And that was always my concern after going through probably two or three different load testing companies over all this time, they keep going out of business or something happens.
I picked K6 because it was open source and you know, worst case scenario, you still have the source code and you can build what you need.
And with the prices increasing so much, it has made the running the benchmarks untenable.
I did the rough math just for the to buy the same number of credits that I used the year before. It would cost over a quarter million dollars.
[00:53:19] Speaker A: Are you kidding me?
[00:53:20] Speaker B: Yeah.
So damn, somebody wanted to pony up an extraordinary amount of money that, you know, maybe I take and just run away and retire to some little island instead of running some benchmarks.
Yeah, that is ridiculous. Already on an island.
Exactly. And so I, for a fraction of that, that cost, I can take the open source K6 and build my own platform to do this. And that's exactly what we've done.
So I have a project now called Orderly Ape.
It's orderly ape.com and yeah, it's on GitHub. It is quietly alpha launched and maybe by the time you publish this interview it'll be more loudly or maybe this is the making it more loudly launched.
[00:54:22] Speaker A: Let's make it so.
[00:54:24] Speaker B: Yeah, and so yeah, Orderly Ape is basically a K6 cloud replacement.
It lets you run your own K6.
[00:54:36] Speaker A: Tests.
[00:54:39] Speaker B: Distribute them across as many hosts as you want. Of course you have to set up all of these stuff and it's on your own infrastructure, so you own the data it has. It uses Grafana too because that's also still open source, the core and you can see the results.
So yeah, we went and built a.
I mean it's not as polished as their product. It's an open source product and it has been very minimally funded. I discovered how difficult it is to raise money for open source.
Still looking for any more companies to step up and help cover the cost. If this seems like would you rather have complete ownership over your load testing thing or pay like five or ten grand a month to run it on someone else's cloud?
So I work with hosting companies so infrastructure is virtually free.
I thought this would be really easy sell. Apparently harder than I thought to pull money out of hosting companies.
Maybe you can speak to that, ah.
[00:55:54] Speaker A: I don't know how much sway I have over them, but I can certainly try.
[00:55:58] Speaker B: Oh, I meant the difficulty of pulling the money out of hosting companies.
[00:56:04] Speaker A: Oh, okay.
[00:56:05] Speaker B: If you want to get, get, if you want to help fundraise, I'm all for it, but I, I was just going for the salary first.
[00:56:13] Speaker A: Yeah, yeah, yeah. You want to use my physical strength. I get it, I get it.
You can donate down for it as well. Yeah.
[00:56:24] Speaker B: Oh, that's funny.
[00:56:26] Speaker A: No, but have you just orderly Ape now is fully open source. Is there any thinking in term making that into a affordable.
[00:56:39] Speaker B: Zero?
[00:56:40] Speaker A: Let's call it none. Like no. None.
[00:56:42] Speaker B: Nope. I have watched multiple load testing companies that I worked with that I thought were stable come and go. They all seem to follow this trajectory where it seems like it's incredibly difficult to make a business out of load testing. And I've seen even more tiny little startups come try and disappear into the ether. And I don't understand why other than people don't seem to want to pay for it, at least long term.
Yeah, nobody seems to be able to crack this.
I also, I don't want to work on that. Like, I would rather have it be open source. I work with hosting companies.
Let's. Open source is great because all that extra value that's being captured is being captured by the people doing stuff. I like that. I would love for them to have that and if they could put a little bit in so that we can make this a little better or contribute or whatever, that would be amazing. But yeah, I have no interest in running a load testing business. I would rather a dozen people take what we're building and build their own. I hope they have grand ambitions and can crack it, use us as a base, contribute back and make it even better.
[00:58:08] Speaker A: I get that. I get that. The, the reason I asked.
Or you didn't. You didn't even let me finish the question.
[00:58:14] Speaker B: Oh, sorry. But I get that, that exact question a lot. Like why don't you just raise money for it? Like we didn't.
[00:58:22] Speaker A: Yeah. So the, the reason I asked is the, the, the comparison that I made is, is to Obsidian. I don't know if you know that note.
It's a note taking app and it's open, it's, it's as open as cheap. And, and because it costs zero.
But there are ways to sort of contribute to it. As in if you want the sync, which is you can do it any other way but you know, you pay a, you pay a license fee and that's it, it's just no biggie. It's more like there's an easy way to donate to that project.
Um, and, and that sort of is what popped in my head. Like, you know, that that be. That might work, like not necessarily turn it into a whole full fledged company and whatnot. But, and I, I'm, I'm a huge fan of the Obsidian platform because of how much people are adding value to it and how you set it up and hot little plugins building extensions on top of it. You know, it kind of reminds me of the, the base principle you just explained. But if you do not want to do it, that's also good because you know, Open Source I think is, is the, the, the beating heart of the Internet.
And you know, I for one owe a career to it. So if you can call it a career for all the ones who just heard me say that, dude, that's a career you're doing right there.
[00:59:44] Speaker B: I got you.
I mean at some level, if you're on the Internet and make a living off of it, at some level that's open source. I mean the servers that are, we're talking through right now are almost surely Linux, the programming language. This is all written. Yeah.
[01:00:05] Speaker A: Well, Windows, but part of, parts of Windows have Open Source libraries in it.
[01:00:12] Speaker B: Okay, fair enough.
[01:00:13] Speaker A: It, it's, it's there, it's everywhere.
[01:00:16] Speaker B: Oh, I agree. I mean there's going to be components. Yeah.
And yeah, I, that's, I love Open Source. I released code before I even knew what the license. I think I probably still have code that nobody should ever run by the way out there somewhere.
You know, I got excited because I wrote like a hit counter and curl and you know, different images to do the scrolling by.
[01:00:48] Speaker A: Yeah, yeah.
[01:00:48] Speaker B: You know, pretty sure I never chose a license. It was just, hey, I made this cool thing here.
[01:00:55] Speaker A: Have fun worlds.
[01:00:56] Speaker B: Yeah, yeah, yeah.
[01:00:59] Speaker A: No, Open Source is the, in my opinion, the beating, the beating heart of the whole of the Internet. And there's a, there's a fine line where it, when you switch from having Open Source and then abusing whatever you have for just material gains, I not go off into the political side of things, of what that might implicate, but more like if people really understood, and I, I really mean like John Q. Public, if they really understood what the influence of Open Source was like proper, like fully, like most of us who are on the Internet fully understand, I think we would have a different world.
I honestly think we would have a different world because people would get it. Why are we doing this. This is why we're doing this. And everybody would. Of course you do. It makes sense. This is how we progress.
[01:02:01] Speaker B: If they could, if the whole world population could understand open source and the ethos of sharing and giving, I don't think we'd have most of the problems we have in the world today in general, even without open source, if that a level of caring and empathy existence.
[01:02:19] Speaker A: Yeah, my point exactly. But I.
[01:02:21] Speaker B: But I agree with you.
[01:02:24] Speaker A: Yeah.
Well, been an interesting and fun conversation. Kevin, thank you so much for showing us a little bit behind the curtains of what it is and what it takes to review hosting companies. Thank you so much, man.
[01:02:41] Speaker B: Oh, thank you for having me. It was an absolute pleasure.
[01:02:45] Speaker A: Absolutely. It was. Thank you.
