Beyond Data Engineering ft. Joe Reis

0:00So I'm about to meet Joe Reis, the co-author of Fundamentals of Data Engineering, and of course I'm running late. Well, yeah, maybe he's running late, what do you think? Look who I got on the bike. Hey, what's up, how's it going?

0:26Yo, it's Appraiser. What's up, cheers, good to see you. I'll do a quick intro, and you correct me if I'm in the good or not. So you are a DJ for a couple of years. I've been DJing since the late 90s, early 2000s. You have a couple of tracks on your SoundCloud, right? Yeah, yeah, a couple old sets, yeah.

0:51And yeah, I think that's pretty much it about you, right? That's it, yeah, that's all I do, I just make beats. So, of course, you're pretty well-known for the best-selling book on Fundamentals of Data Engineering, co-author, right? Yeah, co-author, Matt Housley was my co-author. Matt Housley was your co-author. And you define yourself as a recovering data scientist.

1:15We're going to get there afterwards. And speaker, professor, and of course, podcaster, and content creator, of course. Yeah, yeah. And your famous podcast, Monday Morning Data Chat, has more than 100 episodes, I think? Something like that? I think the official number is like 124, but we've actually recorded more than that.

1:38Okay, and you started also your own podcast. Yeah, so I have my own podcast, The Joe Reis Show, so that's also one. Just because the Monday Morning Data Chat was basically booked until now September. And I still like doing podcasts and talking to cool people, so I figured I'll just do my own. Nice, amazing. Let's start with something simple. What's the scary thing in data?

2:02The scary thing? Like the scary thing? Yeah. Okay. I think the scary thing is often the data itself. So you just never know what you're going to get yourself into with that. I feel like data is usually kind of a secondary consideration at a lot of companies.

2:20And so if you work with, quote, data, you're usually working with the aftermath of a developer, maybe, or an API that gives you data. And it may or may not be useful for you, but it's your job to figure out how to make it useful. And so I think that's always the big question mark.

2:35The other big question mark is, I would say, the other scary thing would be just how are you going to work with the business, and are you going to get buy-in from the business for what you're trying to do? And sometimes the first part I mentioned, the data, and the second part, the business, sometimes these are related, and sometimes they're not. But I think these are the two big ones I see.

2:57How do you see the importance of good communication for a data engineer and, let's say, a technical person? How important is that communication there compared, I would say, to a standard software engineer role, let's say, as a front-end for a mobile app?

3:16So data, it's basically the same, I think, as software, but it has sort of after-effects, too, because data is more disparate in its use cases. You might be using it for reports. You might be using it for a model. You might be using it for this, that, or the other thing. And so I think that's the big difference is just how are stakeholders going to receive and perceive the utility of the data they're getting. That's, I think, the big difference.

3:40But stakeholder management is one of those things where I think it just exists up and down the data lifecycle, whether you're a software engineer. I mean, it exists outside of data as well. It's just like that's part of your job is to do stakeholder management. How that's done at a company or on a team, I think it really depends on that company and that team. There's not really a standard way to do it. But in our book, each chapter we talk about who you work with. Yeah.

4:02At the end of each lifecycle chapter. So we cover basically, okay, so just be considerate of who's upstream and downstream for you, and here's some things that you should look for. Maybe know about their needs. They should know about your needs. I think that's the big mistake I see with a lot of data teams is there's not that kind of cross-communication with the upstream and downstream users.

4:24Yeah, and I think it might be also, and you did that pretty well in the book. I really like it at the beginning. There is a lot of context, a lot of terminology. And I think that's the biggest challenge in the data world is that there is so much wormy terminology that people get lost. There is the manager coming from the other side, and I was working in data.

4:48That's a lot. And be able to navigate to that. I think it's helping a lot to say, okay, this is your role. This is how I communicate with you. And actually, I should communicate with you because you were mentioning earlier people getting data from an API.

5:05They might change some stuff and not discuss downstream because they're not aware of the usage of it. They typically don't know and often aren't incentivized to care. Yeah. That's the other part. Do you have incentives? If you're an upstream producer of data, do you have incentives to care about the downstream users of your data? Often not.

5:25You're incentivized by your brand if you're a software engineer, which doesn't typically include working with a data team. And if it does, it's just like, we'll get you data. That's your problem. So it's interesting. I see this changing, though, solely by Shirley, but it's still a big scary thing. How do you see that changing? Why is that there is a trend?

5:49I think because applications in general are becoming more, quote, data powered. So the feedback loop between data is going back into the application now. And so I think that's forcing software engineers to, I think, be cognizant and work with the data team rather than just sort of lob it over the wall. Yeah. But lobbing it over the wall still happens, I would say, more times than not, though.

6:11Going to that theory where you define yourself as a recovering data scientist, why is that so? I got hooked on data. I made some bad choices in my life. But I'm getting better. I just take it day by day, and hopefully I'll be off data soon.

6:31What I saw in my career doing, quote, data science, which I actually have no idea what that is anymore, but all too often what I saw is companies would hire people like me to do machine learning and jump straight into the fancy stuff. But I noticed there was no infrastructure built typically to facilitate machine learning. Often no data upon which to do science.

6:55Often I felt like data scientists were set up to fail, and I still feel this is the case. I was talking to a bunch of people, machine learning engineers and data scientists, and I think half of them said, I've been hired, but I really don't have a clear directive or data. But we've been hired because this is the initiative. It's the hot thing to do, so let's do it.

7:14So I became a recovering data scientist because I just felt like the notion of data science is a bit abused and overblown and, I think, over-exaggerated. At the same time, there's really cool stuff that's being done with data science, obviously, but my personal experience was such that I got kind of jaded on it. And that's how I got into data engineering, just out of necessity because I had to do my job. There is a lot of kids like this.

7:38I've known countless of data engineers that come from data science. It was inevitable for them because they spend most of their time doing data pipeline cleaning. They start to enjoy it. They say, okay, that could be my job full-time. Yeah, and it's interesting. You see that a lot.

8:01Some people actually like the engineering more. Some people don't, and that's fine. But I've realized it's more foundational. Data engineering is to facilitate data science and analytics. The opposite isn't true. You couldn't really hire a data scientist and expect awesome things just to happen without that core foundation.

8:22You can get some probably exploratory data analysis out of it and, I think, start getting an understanding of your data, but that's not really the same thing as productionizing. But you also learn what you can productionize, I suppose, by doing the exploration. But in that case, if you're just exploring for the sake of it to find data stuff to do,

8:41then I would say that your company who hired you probably was maybe a bit misguided in terms of what they were trying to accomplish. But it's difficult also for a company to know their data matter to you. You're talking a bit of that on your book, right? Yeah. And I think it's a really nice concept.

9:01There is also another company that tests how mature you are with a few questions. They test your company or you as a person? No, basically, you as a company, you go there and you do the test. And it's a few questions to say how mature you are compared to the average.

9:21And I think it's a reality check that some companies start to have now. Maybe that's why data engineers start to explode because they say, oh, hey, we cannot have 10 data scientists and just one data engineer. Maturity assessments are interesting, too, because I sometimes feel like some of them are good

9:42and then some of them are obviously biased towards getting you to sign as a contract. So it's interesting. But I feel like the things I look for is what results are you really achieving, no matter your size of the team, right? Because I've seen small teams be super effective, like SEAL Team 6 level, and giant teams be just absolutely ineffective.

10:01What would be the first data role you would suggest to hire for a startup? I think it depends on the type of product and service that the startup is providing. Let's say like a standard mobile app for B2C customer. Yeah. I mean, it's interesting.

10:24So you obviously want to be able to track core metrics for success. And so that's probably the first person I'd be hiring is somebody who's able to help drive the product. Assuming it's a product-led growth company, somebody who's able to drive those types of metrics, because that's really quantifiable and it has a direct impact on the business. So building in the feedback loops within the app to collect those metrics

10:47and help tune the behavior of the app itself for a better customer experience. I think that's the kind of person I would hire first, whoever that is. So more a product-head person with a data background that's comfortable to navigate to this.

11:07Yeah. I think just developing. You really want to develop that flywheel in your application early so you can even drive success. If that were the company that I had started, that's probably the person I'd. . . Whatever that role is, I just need that person to do that thing. I don't know what it's called. Product, data, scientist, engineer, unicorn person or something.

11:32But I always find that you're productive as a data person when you can tie data directly back to making your product or service better in a positive direction. So let's talk a bit about your book. Why did you first start writing a book? Boredom. No, I'm just kidding.

11:52I felt like, I don't know, for a long time, Matt and I talked about writing a book and I'd always wanted to write one. It was kind of the COVID project. Some people got a dog, which I also did. Some people learned to bake bread and Matt and I decided to write a book. But don't be a best-selling book. That's quite serious. Well, yeah, obviously. No, I mean, but. . .

12:12I mean, O'Reilly, it was funny. So O'Reilly, when we originally pitched the idea, they actually told us not to write the book because they said it was going to be really hard. Because you're trying to describe an entire field, right? So that was what was really fascinating. But we ultimately convinced them, look, I mean, we're going to do this with or without you. So, you know, it seems to exist. It'll be great for, you know, for your portfolio. And they agreed. So we ultimately did it.

12:34It was a great, you know, great process of writing. And, you know, O'Reilly's got a world-class development team. And so I was really pleased by it. But what we really wanted to do was write a book, I would say, that could stand the test of time as much as you can in technology. Things change, but we didn't want to write, you know, data engineering with, you know, platform X, Y, or Z, or technology A, B, C, or something like that. Like, that's already been done.

12:57And those books have an expiration date. So the hard part was really unpacking, like, you know, in five to ten years, what's not going to change. And so that's where we came up with the idea of the lifecycle. I can't say it's like a novel idea, but it was like a very obvious one that I didn't see people talking about. The undercurrents was maybe a bit more interesting. So you focus on, like, across the lifecycle, what's not going to change.

13:19As far as, like, security, for example, you probably want to do that everywhere. Data management, architecture, engineering, and so forth. So that was an interesting thought process. I mean, it really forced you to understand the field in a way that I hadn't seen people think about or describe. Yeah. And in that sense, like, why did you learn after writing the book, aside from, like, the writing experience and so on as a writer, but more, like, on the data thing?

13:44Like, did you have, like, an aha moment when you—because you have to do a lot of research, you know, to— There was a lot of research. —to bridge the gap that's— Yeah. —from your experience. So, yeah. What was, like, the biggest learning? The biggest learning is, I mean, you just find out how much you don't know, I think, when you're writing a book, right? Like, you think you know going into writing a book that, oh, yeah, this will be cake.

14:08I know data engineering, and you kind of find out you probably don't know, you know, much at all. That was my impression. I think Matt felt the same way. You know, and it's not like we, you know, it's not like we came into it, like, as absolutely, you know, buffoons who didn't know anything. I mean, you know, we'd been doing this for a while. But it really pushed our limits and pushed us to, you know, challenge our thoughts.

14:29And one of the things we did to help write the book, too, is we did, you know, the Monday morning data chat podcasting. I mean, that was, like, one of the motives for doing that was we could talk to smart people, you know, understand what makes them think, and, you know, their thoughts and what makes them tick, and, you know, hopefully get some ideas that we could use in the book, as well as challenge some of the ideas we had, too. I mean, I can't tell you how many times we're like, yeah, we're totally wrong on this.

14:51And because of this discussion or because of this thing we read, like, you know, changed our minds. So I think the thing you've got to do is have very flexible thinking, you know, and be willing to leave your ego at the door because you're going to get. And the other thing I liked about it, too, was, so when you write a book, you get technical reviewers. So we had a giant bench of these people. But, you know, you want to find people who are going to beat the crap out of you pretty hard.

15:14So you don't want people who are going to be light because the Internet is, like, a much darker and harsher place than your reviewers are going to be. So, you know, we found people who were just borderline, you know, definitely on the spectrum of just, like, being very blunt and harsh. So, but you want that. Yeah, that was nice. So really the podcast helped you drive the content of the book. Absolutely.

15:38Yeah. And part of the research, right? Yeah. You know, if you go into a podcast with the idea that you're going to learn something from somebody, which is what I always do, it's why I like doing it, then, you know, I think it's a healthy attitude to have. Yeah. Would you recommend someone that's, like, into tech content at some point to write a book or not? Because it's really tedious.

16:01I think if you have the time and you have the disposition to do it and I think the persistence and commitment to want to do a book, then sure. It's not for everybody, and I would say, like, most people probably shouldn't do a book. It's so much work. Yeah. So, like, you know, it's going to be about a year of commitment, that minimum.

16:22You know, maybe you can write it sooner, but, you know, typically it's just a giant chunk of your time. So what's the opportunity cost? Okay. You know, that's how I look at it, so. And what would be the first advice you would give to someone that would like to write a technical book? Try and find an angle that's different, right, and that's uniquely yours.

16:44So try and fill out a book proposal template, like O'Reilly has those. Just fill them out, even if you're going to self-publish. I think it forces you to think through the business plan of your book. Okay. What is it? Who is it for? What are the competing titles? Why are you the person to write this? Who's the audience, you know? So. Yeah. Yeah, I would say that's go through that process, and then be open about your idea.

17:08I think a lot of people try and hide their ideas, like, oh, you know, somebody's going to take them. Nobody's going to steal your idea, trust me. It's like nobody's going to write a book on your idea. It's like a stardust, right? Your idea is the easiest thing in the universe. Execution is the hard part. Like, trust me, nobody's going to write a book. It's so much work. I mean, people are like, oh, I can't give away the idea for my book. I'm like, you can act that way, right?

17:30You can be secretive, but let me tell you, nobody's going to knock it off anyway. Yeah, so what is it? In fact, you want to have your ideas challenged, right? That's why I've been open about the new book I'm writing on data modeling. It's like, challenge the ideas. Yeah. So I'm talking to a lot of the top people in the world in data modeling, and that's forced me to challenge a lot of my ideas about the book, and I think it's going to make it better as a result.

17:53If I came into it with my ego maxed out and I would say, you know, I know exactly what needs to be written, it's like, you know, you could. But I think nowadays content's a bit different. It's a lot more about community involvement in some degree, at least I think so. You can have novel ideas, but I think that you can iterate a lot quicker if you get feedback on them. Yeah.

18:17Versus like locking yourself away and, you know. But yeah, you know, but it's the same with like other types of content, right? So if you're going to make a course, if you're going to, you know. So I would say like start cheap, like make posts. Yeah. On LinkedIn, Twitter, whatever. And then start to connect the dots. Connect the dots and test the ideas out. So, you know, I've been having some friends do this with a book they want to write.

18:39I'm like, why don't you just test parts of the idea out, see who's interested in this, what's the pushback. The thing you don't want to do is invest a bunch of time into a book and then release it. Yeah. Because without getting any feedback. And does your publisher already like advise you on this or is mostly like you coming out with the new way of like.

19:02Because I feel like there is like really an old way of writing a book. Yeah. And a new way, like as you mentioned, where it's more coming from the community. They've been posting and logging and they say, OK, I'm going to recap kind of what I've been doing. I mean, I think I take a different approach than traditional publishers. I was talking to some people who've been editors for a long time. And I just think I have a different opinion about how a book should be written these days.

19:26The approach I take, you know, we mentioned DJing earlier. It's like I take very much like one of my friends mentioned, like you promote your book like a club promoter. Yeah. I used to do that. So it's like you just you hustle and you think of different ways to, you know, to get the word out. Right. And it's like it's probably an approach nobody's really done. I mean, I know other authors that have. But it's like you got to be nimble and you got to be different these days.

19:50There's so much there's so many books and so many ideas. And the thing is, people don't read as much. Yeah. So you got to give people a reason to care. I'm guilty. Yeah. Well, then. But it's like, you know, if you're making a course, it's the same kind of thing like, you know Cutting through the noise is huge. I think you just need to be Like really fine-tuned and I think it's like LinkedIn you can get analytics on your posts like who yeah Who's looking at it? How many views and also like titles company geography?

20:15Yes, you can say yeah, you can do way more fast iteration on that rather than Yeah, for me what's always like scary of writing a boost is like is that's kind of a one shot as you mentioned Like yeah, if I look at classic publishing like okay, you write the book and it's out there Wow Like having a newsletter or blog post you can iterate change the topic say what's words, you know

20:39I tend to find really your voice your way of writing and Then I feel like writing a book after that like when I write like so much blogs and so on then it's like There is little failure a risk of failure plus you have an audience that you know Like there is a couple of fans that will anyway by the book. That's just it And so the next book I'm writing is probably I'm not gonna start releasing early versions of the chapters on substack

21:04Yeah, yeah, that's interesting. Right so you can build an audience that way Yeah, you can kind of get proof of what's working what's not and have a billion a community around the idea I mean, it's yeah, that's the idea Yeah I saw Steven Sinofsky did this so he was a guy that worked at a Microsoft back in the day and he wrote a Hardcore software, which is a history of working at Microsoft on like word and the office products and stuff And I thought it was brilliant. He wrote the whole book on Substack. Okay. Yeah, it was awesome

21:30So that's that's an approach and I think I'm gonna try it out some like, you know It it was interesting last night, you know I was a quick aside like I was in Munich hanging out this club with a DJ named DJ hell So if you know if you know who he is, he's like 60 years old now, but he was a first DJ in Germany and I believe in Europe So he helped he was like single-handedly the guy that created like this whole dance music scene that you have now

21:55Okay so as we sat from there for like an hour just talking about like, you know creativity and staying relevant and that was So so cool because he had some really good advice and it's something I believe in too which is like you constantly have to be reinventing what you're doing and pushing yourself into scary areas and yeah You know and that's that's how you stay relevant you don't stay relevant over time by doing the same stuff doing podcast outside

22:19Yeah, next to the boats next to the boat. That's one thing. I will tell the audience too is like He got to remember there's about five to ten percent of people who no matter what will just hate you. Yeah for whatever reason You know, they just want to see you fail And you just got to get used to it, you know No matter what you do like somebody's gonna hate it

22:38Somebody's gonna hate you personally and like the sooner you can accept that and move on the better off you'll be so yeah What's your reading strategy today reading? Yeah reading strategy to to To stay up to date in the data words. Yeah, I read a bunch of articles. So I usually queue up like probably 30 to 50 articles on my iPad and just read them over a weekend. Although I have a spin bike now

23:02So I'm back at home. We'll just set it there for an hour I'll just like plow through you know anywhere from five to ten articles So on books, I'm probably reading like one to two books a week. Oh, so I Read really fast. I do have very high comprehension as well. But it's just it's something I've always done I'm just uh, if you looked at my office or my Kindle

23:22There's like thousands of books on the Kindle and it's like my book my floor is like littered with books. It's pretty disgusting actually How do you sort like really good content and I why would be the advice to someone starting You know their data carrier and they want to stay up to date I would say study study wisely and read widely to like don't be afraid of articles

23:46I would say you'll know within the first minute or two whether this article is gonna click with you or not But I would say be pretty ruthless with them You know, you don't need to finish it. Just get the core ideas skim through it first Understand if it contains something useful. Yeah, that's like yeah often like When I when I can understand most of the article just with the title and the subtitle Yeah, I felt like okay, that's interesting to read. Yeah, and then

24:11There's a lot of clickbait though, right? So people people like, you know that the ten ways to become the type Python program of the world or something like that and it's like But you know, you never know until you read then again, there's there's articles that were You know that the GPT-3 paper was interesting to read but that was called like, you know Like a few shot learning or something like that like something that's very unintuitive But that changes the world right? So you just never know like what you're gonna run into

24:34I do read a lot of white papers, too So I think that's like the one thing where I differ from a lot of people as I read articles, but I read white papers Yeah, actually not the first one. I heard and I think it's really interesting like the one advice I received was like Take a few white papers, and if you don't understand anything, it's okay Yeah, and I think that's what scared people was like, okay

24:57I don't have a PhD to read this, but I think a lot of time you can get some idea out of it Yeah What's the source? So I mean I tend to learn from first principles, right? So it's you know with transformers, you know, for example, go read the original 2017 Google paper You understand transformers read the GPT papers and now you pretty much understand like what's going on in the generative AI world You know at an atomic level

25:22I think that's that's how I learn because I think if you can learn from first principles You have a foundation upon which to build you know a body of knowledge, but you know It's a hard part too, especially when you're starting out in your career because you want to You don't know where to begin and it's intimidating, right? And so you don't know where to start but I would say just don't be afraid to dive in and read widely to like read You know, it's up in computer science read stuff even in like front-end developments

25:46Like, you know, I still read like JavaScript articles just because I'm interested. Yeah. Yeah, it's cool I mean me too. Like what other advice would you give to someone starting his data engineering career aside from Reading widely. Get my book and yeah But you know get starting on a data engineering career you know find a good good support group find a mentor maybe and you know, and You know just be patient

26:11I mean the process is long and you understand knowledge accumulates over time the skills accumulate over time and it's got compound interest like the the gains you see on day one or It'll be small. It'll be frustratingly tiny And so you just got to know that over time if you put in the work, you're gonna see the payoff Also builds your network like that's that's a big one where I'd say like your network is your network is the old cliche goes But it's very true

26:34Like the opportunities I'll be given to you in your career or typically not you were going into playing for a job It's like people saying hey There's a job over here. Are you interested right? If I set yourself up so opportunities find you, you know building a network building your knowledge building your skills and building a good reputation That's the other piece to where it's like your reputation is everything in this world and the you have to do everything you can To like maintain that reputation and build it

26:58You know as Warren Buffett says it takes, you know decades to build a reputation and takes it You know a few minutes to destroy it. So there is one thing I want to ask Because you're from the u. s. Yeah from Europe So I heard the u. s. Innovates and the EU regulates. Yeah, so what's your thoughts about that?

27:18I I think you know, I've been spending more time in Europe and I think the differences between the u. s. And the EU are very stark The u. s. Moves fast. I would say the u. s. Also is very the land of extremes. It's very high and very low There's not really a middle ground. Whereas here it feels like it's more kind of even-keeled you guys don't go that fast You know, you guys move probably slower than I would like but it is what it is

27:43You know, it was interesting, you know at the conference I was speaking at we when the Parliament members of the EU was talking about You know GDP r2 was coming out soon So that was an interesting take I would say on you know, the European way of regulating artificial intelligence and data One thing I think I heard him say was that you know Every data set had to be unbiased

28:05Wow, yeah, which I was like, I mean it sounds nice in theory but I have no idea how you're gonna either detect that or enforce this and so Things like that. I feel are still understanding how GDP r1 is working Yeah, most people are right and so that's the whole thing where I feel like, you know in in the u. s I'm sure we're gonna have our own a regulation But you know, it'll be lobbied to death by all the big companies and so it's just it's different

28:29But I you know, there's something that I really like about, you know Europe as well But I feel like the there's definitely a lot more regulation which I think has its pros and cons, you know And just like the u. s. Is I would say, you know lack of regulation in these cases It also has its pros and cons. It's just I wouldn't say anyone's better than the other. It's just different ways of approaching things. So Thank you very much for a great time as we're really fun. Enjoy your rest. Good luck out there