Carlos:             Compañeros! Welcome to another edition of SQL Data Partners Podcast. I am Carlos L Chacon, your host. This is Episode 199, compañeros. We’re almost there, the big Episode 200. So, this is going to be our last episode in “Season 3”, whatever that means, and Season 4 will start soon. We are going to take a couple of weeks off in between this last episode and the next one, change some things up. And we are excited to have on the program today, Adam Lenda.

Adam:             Hey, how you guys doing today? It’s good to be here.

Carlos:             Yes, we’re happy to have you, Adam. We’re going to get into our topic in just a moment, but we also recognize Kevin Feasel.

Kevin:              Hello.

Carlos:             And Eugene Meidinger.

Eugene:           Howdy.

Carlos:             So, our topic today is Technical Debt. So, while we’re all experiencing the quarantine, well, except for Sweden, I guess, a lot of meetings have gone to virtual meetings and Adam, in the Richmond Technology Group put on a meeting. And this is, again, another one of those topics that I’ve been very interested in circling back around to, and now I have the person to help me do it, so Adam, thanks for talking with us about it.

Adam:             I have to clarify. I am not an expert on anything.

Carlos:             And we, I further have to clarify, Adam is a developer.

Adam:             Yes, yes.

Carlos:             We won’t hold that against him.

Adam:             I did get some nice mentoring from a really kind DBA once, so I do all my own SQL queries and I do most of the t-SQL tuning for our company, actually.

Carlos:             Does his own stunts. Very good. Okay, so before we get into our topic, I do have a couple of shout-outs, for those who are showing a little love for the podcast. Enigma Labs, Erik Binggeser and Mark Cretenson– Mike– oh, I said Mark, didn’t I?

Kevin:              Oh, you really messed up. Mike Chrestensen.

Carlos:             Mike Chrestensen.

Kevin:              Yeah, yeah, my one shout-out in 199 episodes and Carlos he just mangles that one.

Carlos:             Man, tripped over the finish line.

Kevin:              Just straight down the tube.

Eugene:           You act like you’re surprised. Carlos is usually so good at these names.

Carlos:             So now, why are we giving a shout-out to Mike?

Kevin:              Mike is a buddy of mine out of the Tri-Pass area. He has been listening to the show and is our one viewer.

Carlos:             Beyond my mother, right?

Kevin:              Yeah, exactly, exactly.

Carlos:             Thanks Mike! And we also give a shout-out to Alvin Ashcroft. So, he posts his Morning Dew blog. He posts our stuff and so thanks to Alvin. So, I am curious, compañeros, if you want to let me know on social media, how often do you actually read those little daily pressers or whatever? There’s a couple of folks still out there. I’m just not on social media as much as I used to be, but I feel like I used to see them a lot more often. Maybe I just get mentioned more, who knows, but I’m curious to know, you guys actually click on those or is it only when your name is mentioned?

Kevin:              Only when my name is mentioned. I just like reading about myself.

Adam:             I’m unqualified to talk about social media in any way shape or form. I try to avoid it, and actually, only recently because of the COVID thing is only reason I’ve been on there lately. But it’s actually interesting, I reconnected with Andy Leonard and some other people, like some of my mentors I hadn’t talked to in a long time, I got into conversations with them and it wouldn’t have happened if it hadn’t been for COVID. So, it’s not all bad, I guess, you know?

Carlos:             Always look for the bright side of life. I have the SQL Server Radio guys in the back of my mind and mad props, and oh my gosh, I can see his face but his name just– so Matan’s not there anymore. Oh, Guy. So huge shout-out to Guy. He’s always sing– well, I shouldn’t say always; I don’t listen to every episode. But he’s always singing and my hat’s off to him. Okay, so to not digress any further, compañeros, our show notes for today’s episode is going to be at sqldatapartners.com/techdebt or sqldatapartners.com/199. Okay, so this is interesting. Our topic today, Technical Debt, when I was working with Steve Stedman, and his tool, he actually has a technical debt analyzer.

Adam:             That’s interesting.

Carlos:             And so yeah, this was a little bit interesting, and I think we throw around that word ‘technical debt’ a lot.

Adam:             Yes, yes, yes.

Carlos:             And listening to Adam and admittedly, I didn’t get to hear everything of the whole entire session, but some of the points that he made really resonated with me. And a lot of times I tend to think of technical debt is either old stuff that is not supported, so again, in the database realm, right, ala SQL 2000 would be an example of technical debt. Or “I wouldn’t have built it that way”, so therefore it’s technical debt, right?

Adam:             The mental model gap. Yes, the mental model gap, absolutely. But that’s the one that drives me nuts, and it’s the one I was worst at when I got started my career, frankly, that I came in, and I came from a computer science background, and I was working with guys who were more of an information services type background. And we just had different ways of solving problems and I would drop into some code and I’ll be like (sharp inhale), and it was a learning curve, actually. But it was a good one for me to slowly ramp up on, I just wish it hadn’t taken 10 years before I finally got really good at understanding which was which.

Carlos:             You’ve entered the no-judgement zone. Some of us still haven’t quite figured some of that stuff out. But are there other scenarios or other definitions we need to include before we kind of get into it?

Eugene:           Well, so I’m not the expert here, but my understanding of technical debt was that the whole point of the analogy is you’re doing something that’s cheaper now in terms of time, but you’re going to pay interest on later on.

Carlos:             Okay, so that speaks to the current state, like something I’m currently doing that I know is going to be “unsupportable” or whatever.

Eugene:           Right. Yeah, you’re doing something in a way that’s faster, easier, quicker, whatever.

Adam:             It’s a compromise, yeah.

Eugene:           You’re collecting something now, you’re taking on debt, but later on you’re going to be paying back interest and eventually you’re going to have to pay down that principle by fixing it. So, I remember whenever I was in high school learning how to program, I think I still have my graphing calculator somewhere, and it’s just painful to look at the code that I used to write because I didn’t entirely understand a for-loop kind of thing. So, I remember there’s some program somewhere with like 30 different if-then statements, that would have been just so much better if I had like an array.

Adam:             I mean, what’s a decorator when you’re 13? Right?

Eugene:           Yeah, right, exactly. I’m programming in TI-Basic, like I don’t know any of this stuff.  That’s a lot of what I think about in technical debt is you’re doing things, especially like copy-paste code, you’re doing stuff that’s quicker and faster and easier now, but the reason it’s debt as opposed to maybe just like crappy code or whatever it is that you’re going to have to pay interest payments down the road.

Adam:             Okay so, how about you, Kevin? Where’s your head at?

Kevin:              So, there is one definition of it that I don’t like, and actually it comes from a really good book Working Effectively with Legacy Code by Michael Feathers.

Eugene:           Oh wait, is that the one where “it’s technical debt if it doesn’t have unit tests”?

Kevin:              Exactly, code without unit tests.

Adam:             How about this? If it’s not making money, it’s technical debt. That’s hyperbole by far. So, to me, technical debt is friction. That’s fundamentally what it comes down to, that where you have friction in your architecture that is impacting your ability to get things delivered, that’s how I identify it. And typically, the way I like to draw the nice bright line is I like to have a good well-defined coding standard, and I understand that’s going to be very different from where I’m coming from as a software architect where you guys are coming from with databases. But to me, that’s managing that friction and then it’s a little bit about defining where that friction came from. So, I tend to break it down into essentially, authorized new technical debt, which I think is exactly what you were referencing, Eugene, where you’ve had a conversation with the client and the client is buying that debt, which is significant, because that means when it’s time to pay to clean it up later, you can come back to them and be, “hey, you know, we did this because you told us to, right?” And then, I think where the minority of technical debt should come from, but this is not universally true, is from that unauthorized new debt which is where developer didn’t read the coding standards or the person doing the code review didn’t feel like enforcing them that day or somehow you ended up with that variation inconsistency across your site or your codebase and that results in each developer coming in to maintain it having to re-read or relearn or figure out what’s going on in this file that’s not following the same patterns as all the other files in your codebase, hopefully. So, to me and we were talking a little bit before about mental models, I really think that the thing that takes me the most time coming into a new codebase is I have to build the mental picture in my head of how all of the tables relate, how do all of the classes interoperate, where are the different layers touching or making sure that everybody’s staying communication in your layer or lower and that kind of stuff when it comes into your architecture. So, getting all of that built up in my head is super easy if everything has been consistently implemented across the architecture. If every time you’re going to fetch data, it’s a function that starts with the word ‘get’. Like to me, that’s a super simple coding standard, but if you do that and you enforce it rigorously, then the first time you run into something called ‘get or create’, it’s like, “oh, okay, well that’s going to load or initialize one if it’s not there.” So, it kind of makes sense to kind of start taking these lower-level pieces and building them up into higher order functions to put together your more complicated functionality. And that was actually one of the hardest things I had communicating to my team and I have no idea how to communicate that part other than you have to get into like SSIS packages calling each other or something to that effect.

Carlos:             So now we’ve kind of defined it or given some ideas. Now, Kevin, just to circle back here, you gave a definition you didn’t like. Is there another one that we should consider before we move on?

Kevin:              No, actually, I generally like the way Adam’s been going so far with this.

Eugene:           It’s what I think the Supreme Court came up with for what counts as pornography. It’s like you’ll know it when you see it, is that– or no, obscenity, I think was the definition, technically.

Kevin:              Yeah, it was obscenity, yeah. Okay, so that does drive to a serious question, unfortunately.

Eugene:           I’m sorry.

Kevin:              It’s unfortunate that I’m the one who had to bring up a serious question. This is two episodes in a row where Eugene has usurped my role.

Eugene:           I’ve just gone off the rails, guys, I’m sorry.

Carlos:             Blood sugar must be low.

Kevin:              So, we’re gonna fix that.

Adam:             What’s my role?

Kevin:              We’re going to fix that with Episode 200. So, okay, off of Eugene’s question, how do you differentiate what is appropriately considered technical debt versus I just don’t like this?

Adam:             I had a slide, actually that said, “if you’ve ever said things like this,” and there were a bunch of quotes that you would hear from people, like that, and that is, to me, a definition of professionalism. At the end of the day if you are a professional in the IT world, you will have to maintain things that other people wrote, and they wrote in a way that you’re not comfortable with. And if you’re not willing to accept that and get into the mind of the person who wrote it and figure it out and do the best you can with it, that’s going to be a career problem for you. Because you cannot show up at every new job and be like, “oh well, that sucks, I have to rewrite it.” And I’ve seen developers do this and I seen them do it over and over again. And the thing that I ca–

Carlos:             I’ve done it.

Adam:             Yes.

Carlos:             I mean, and again, using like the database, I mean, how many times have you seen like, you know, whatever, and then all the sudden you see TBL Name, and then you see some other naming convention?

Kevin:              Oh, I’m going to go with the stored procedure with seven nested cursors.

Adam:             How about this? I wrote a rules engine that dynamically executed SQL functions to evaluate something on the order of, I think it was like 270 different rules in this giant matrix. And I was like, “well, it’s a spreadsheet and I have a table, so if I just put all this into a table–”

Eugene:           Checks out, yeah.

Adam:             That was a tragedy right there, but admittedly, it was very early in my career and I’d just learned how to do dynamic SQL and I got excited with it and I went, and I had a hammer and I was looking for a nail. And I think to your point, Kevin, it really comes down to having someone on the team who has drawn the bright line, so whoever your technical leadership is. Sometimes that’s you, yourself and I, but sometimes there’s someone there that you can turn to. And for me, now that I am the technical authority, I try to find other people to run my ideas against, as well, to kind of like figure out when I’m off the reservation. But most importantly, my job in a technical leadership role is to do the hardest thing possible, which is to get my people to read the documentation on how to write their code and to do it in such a way that they don’t feel stifled and enslaved to the coding standard. I still want them to be creative problem solvers, but getting that down and then communicating with them, like, consistency will set us free. The more consistent we are, the more efficient we are. The more efficient we are, the easier it is to come back and fix these problems. So, one of the lines from my presentation was “always choose consistently bad over intermittently good.”

Carlos:             Yeah, and that can be a challenging idea, because it’s like “hey”. And I think that this is probably one area, and I’m making generalizations here, but my experience, our development friends, development meaning your structural language programmer, do better than the data–

Adam:             We do the stuff that doesn’t actually matter.

Carlos:             Yeah. Do better than the folks in the data world, in terms of documenting, best practices and processes, because generally they’ll have something put together. It’s equally, or it’s very hard when you’re like, “okay well, what do I reference?”  and there’s nothing to take a peek at.

Adam:             So, Kevin’s point, earlier, about what do you do when you’re stuck with something you just hate? I think that that statement “consistently bad is better than intermittently good”, I think that that same statement starts to apply, because if you stick with what they were doing, and you stay within the model that they laid out. Like for example, I really don’t like some of the design decisions that were made before I got to the company I’m at, but by sticking with them consistently, I’m then able to get to a point where we’re like, “alright, we’ve been limping along around this problem for a while, I’m able to finally go through that strategy that you saw me outline in the talk and go back to my management and be like, “I need to deal with this and I need to deal with this because of these reasons that actually affect your ROI.” And that’s really, I think, where you get that budget. You’ve got to tie it back to their motivation. And once you’re able to do that, you can get that ‘all hands on deck’ day, which is how we did it. It was actually two days, where we went through and refactored every one of our controllers to support dependency injection. Because I had stepped away from the stuff I was doing, so I could get this really, really old, like 2005 not-MVC-MVC framework. I could incorporate it with a dependency injection container and that gave us the ability to start quarantining and extracting logic out into clean code and still inject the cleaned up service back in for use. So that was like a fundamental part of how I quarantined and cleaned up that, but I had to get the budget and then I had to have the entire team on deck. But because we had been consistent, we hadn’t started using controllers from a different framework, we hadn’t started using PHP flat files, we had stuck with what was unpleasant, it was a real quick, like 30 minute conversation on the phone. “I need you to do these things and we’ve got to do them everywhere. You take A-C, I’ll take D-E,” and that’s how we knocked it out.

Carlos:             Yeah, so that  brings up the point in terms of how you then tackle. So, you talked about these, and I’ve already forgot what you called them, days, they were called. Take out days?

Adam:             Oh, oh, all hands. All hands on deck. Yeah.

Carlos:             All hands take out? I’m looking here, at Eugene’s salad fries, and I’m thinking–

Adam:             It’s cool, it’s right after lunch, I’m peckish.

Eugene:           That’s nice.

Carlos:             So, you had two days that were dedicated. Now how often would you have these? Is this like once a year?

Adam:             We’ve been evolving. There’s a big constraint on it, which is to say, I have to get the architectural changes ready. So, I had to get like the dependency injection container, which required me to refactor CodeIgniter and anyone on here who’s ever been inside the system folder for CodeIgniter will understand the horror that was. I actually called my boss at one point and I said, “I don’t want to.” I said, “I just don’t want to do this. It’s not worth it.” Anyhow, so I managed to get that to where it would play nice, and then once those pieces, the foundational place was ready, then we could jump and that’s when everybody jumped in. So, the constraint there is whoever you trust to make those large weeping architectural implementations, the stuff that’s just too big for one person or to break up amongst a team because it’s going to have too much nuance, too much research required, you know, just gnarly stuff. So, I get that kind of stuff bushwhacked down, and then I can come back to the team. And we’re trying to get into a cycle of one day every month we dedicate that day to go take on whatever I have targeted as the next item that’s impeding our ability to get out of CodeIgniter and onto Symfony, which is the replacement framework we’re moving to.

Carlos:             Sure. Okay, so regular sessions, the first time you asked for two days, there was a need to, I would say some grassroots organization that needed to happen beforehand, and then you’re able to divvy some of that up. Yeah, so why don’t piecemeal approaches work?

Adam:             Oh, that’s actually one of the slides I didn’t have in the deck, because it added too much time, was “clean as you go is a fallacy” in my opinion. Because when you say we’re going to clean as we go, what you’re essentially saying is we’re only going to clean the things that we are going to touch. And in the course of most business, there are paths through your business that are going to get touched a lot, and what I saw is we’d go make a pass, that path would be 5% better. And before we replicated that 5% out to the rest of the codebase, we’d come make another pass on that same feature again, because we had another change order for it and “well, while we’re here, we’ll do this next step, too.” And all we were really doing was increasing the gap between the stuff that hadn’t been touched yet and the stuff that we were touching on a regular basis. So, I came to that conclusion to say, if we’re not ready to make this change codebase-wide, then we’re just not going to try and start that change yet. And don’t get me wrong, I learned that lesson the hard way, because I watch those gaps get bigger and bigger and eventually, I just said, “enough’s enough, we’re not doing this anymore.”

Carlos:             Yeah. Yeah, so, I guess Kevin, Eugene, your thoughts on, now you’ve got it, how do you ha– methodologies are?

Eugene:           Well, so again I’m going to steal Kevin’s job as a devil’s advocate.

Kevin:              Derail your–

Eugene:           I don’t even know why he’s here anymore.

Adam:             Cause he’s got this– he’s got the gravitas.

Carlos:             Oh, man.

Eugene:           I’m sorry, unemployment’s up–

Kevin:              Guess which character get killed off.

Eugene:           Yeah, Season 3 finale, cliffhanger. No. Yeah, so devil’s advocate, remember how we talked about how Kevin doesn’t like the definition of ‘technical debt is any code that doesn’t have a unit test’? But you talked about like not cleaning up as you go, but isn’t that kind of the golden promise in the sky of unit tests? That if you have test coverage of your code, then you can make changes and you can clean as you go along, and you’re going to know that it’s working because you have those tests in place? Isn’t that kind of the promise that we were given whenever they invented TDD?

Adam:             I admire that you’re that far up the mountain, dude.

Eugene:           Well, I don’t actually have to write code. I do business intelligence.

Adam:             So, CodeIgniter is fundamentally untestable. Even the systems folder that I had to refactor was full of exit statements, which are hard exits.

Eugene:           Oh wow, okay.

Adam:             And PHP, that kills your test unit sweep. So, the only way we were able to get any kind of testing at all was to do integration testing. Full, like we recreate a controller, we new up a post, we pa– oh, and I’m not kidding when I say that like to test a payment processor, of which we have seven, but to do one type of check on one payment processor takes around, if I denormalized all the code, it’d be about three thousand lines of code to set everything up, create the product, create the vendor, create the affiliates, create the relationships. And by the time you’re done, you have this immense test that is great, until you change one thing, anywhere. So, from our perspective, we rely on selenium tests for their super critical features, that’s the JavaScript-driven kind of like click-through testing, with browser automation.

Eugene:           Right, it’s the Uber driverless cars of browsers and browser testing.

Adam:             Right, exactly, exactly. So, for mission-critical features, we’ll take the time to build those tests, and then on the other side of that, we have the integration tests. And so for me, getting to a point where I now have about 30% of our codebase is unit test-able, feels like a massive win. Because we’re talking about 200 thousand lines of code that’s been under active development between 4 and 9 developers for 8 years. So, for the first two years I was still just trying to get people on the same page. So yeah, we’ve got a lot of debt in there, and those integration tests definitely keep us from cutting our own face off, occasionally. But really, yes, absolutely. Getting to the unit test point is amazing and great and if you have the ability to kind of do that, “I’m going to make a change, I’m going to improve the unit test”, you can do that. But I still think what you’ll end up doing is, it’s almost like you’ve got the fast lane of change and you got the slow lanes of change. And what we really care about is the cognitive friction on the developer. That I hire a guy, I’m not going to hire him just to work in the slow lane, because he’ll quit, or hire him to work in the fast only. I’ve got to have a a full codebase developer. So, I want it to be as consistent as possible across it, so that he’s not spending time, or she’s not spending time adapting how they think about the problem to fit, “oh, that’s how we did it in 2008. This is how we do it now. And that’s how we used to do in 2015.” Because each time they have to shift gears like that they’re stopping and re-provisioning all that memory space, and to me, that’s a huge impediment.

Eugene:           What do you think, Kevin?

Kevin:              So, that said, it seems like there’s still going to be drift over time because you mention 2008, 2015, 2018, as things do change, even if you’re staying within a general pattern, the way that you write or the way that different people write code, you’ll have some people who are going to just have small enough differences that the passcode review, like I’ll write my functions, I’ll have smaller functions than developer two, who has slightly larger functions. It’s not absurd, it’s not like 2,000 lines in a function, but where we may not exactly agree on where precisely those boundaries live, over time you still have that spread out or as I convert people to my thoughts.

Adam:             The indoctrination approach.

Carlos:             They drink the Kool-Aid that I offer them every day.

Kevin:              As I kidnap their families and ransom.

Adam:             No, I hear you. I actually had a major falling out with a developer once. He doesn’t work with us anymore. He’s a smart guy, very capable guy, definitely did not put a lot of priority on interpersonal skills, and you know, when I came to him and I tried to have this conversation to say like, “look, I understand that what you did was not just better, it was more efficient.  I understand you’re right on multiple points, and I want to write code the way you’re writing the code, but what I need you to understand is that across my codebase there are literally like over a hundred and fifty controllers and some of those controllers feel should have been like 12 by themselves, but they’re overloaded.” And across all of that noise, if he comes in and starts– and a single-page web app is great, it’s very responsive, it’s awesome, but it’s the only one in our entire codebase. And to me, it was very much a matter of like, “who’s going to maintain that? Who’s going to go on vacation? Who’s going to work on it? If you build something that only you know how to maintain, or you build something that doesn’t conform to how I’ve trained everyone else to do things, even though we’re doing suboptimally, then you’re essentially saying you’re the only one who’s going to be efficient and quick in that solution. And that’s going to put an undue burden on your time, and I need your time to come and work on these more important things that are going to help move the entire codebase forward cohesively.” So, when I run into those personality differences, or those creative freedom differences, is what I essentially classified him as. If I have someone who is creative enough that I can trust them to solve problems independently, I want to pull them up to work on that bushwhacking we talked about earlier. I want them helping me get the big structural changes out of the way, because that’s something where I can point them at a problem and be like, “look I don’t care how you get our sessions compatible with Symfony, just go do it. Because once you lay that pattern in place, I can just say that’s how it is from now on and we can move on.” So, I think as a personality management issue, finding the challenges in your codebase to let them go do, because I don’t have a problem, like, I’ll go follow my coding standards and go do some boilerplate stuff. I understand why the consistency’s important, so, I’ll do the tedious stuff in order to keep that high value resource engaged and interested in solving the more interesting problems in the codebase. Problem is, if you have all, and I did this too, I hired a round-table of developers where everybody was a senior dev. We didn’t get very much done.

Carlos:             Yeah, so that’s interesting. As I think about my statement, here, it appears that it all goes back to standards, in a sense. We have to agree on the way that we are going to do things. Sure, maybe there are some minor changes in there, but if it’s close enough at least that continues to help move things forward. Yeah, there may still be some technical debt issues, but the ability to overcome them, or the jump to overcome them, or the ability for one team member to then jump in and take on something that the other one has done will be easier; the bar will be lower to do that. Is that fair?

Adam:             Yeah, fundamentally, code should not die of old age. Code only dies of inconsistent implementation within the codebase. It dies of maintainability issues, unless you have like a forms app and you need it to be a web app, rewriting is usually a pitfall and it’s so fraught, I’ve watched several rewrites in my career. Every one of them has been partially successful. And it’s super hard to get them right, and so from my perspective, I think I’m starting to specialize my career around this idea of, “yes, you have a legacy app, yes, you have bad debt. As long as it’s not COBOL we can probably work with it,” and we can probably figure out a renovation path for that code. And it’s such a broad subject, too, and I feel like I would really like to hear more about how you guys apply– like, what do you guys see as debt in your day-to-day? Like what are you guys looking to identify and how would you manage it, because I would be interested?

Kevin:              First, back to derailing on my part.

Adam:             Bring it on, man.

Kevin:              So, take that, Eugene. I’m going to say that the perfect example of Adam’s point that rewriting never works is look at the Madden series. That’s been one rewrite after another that just took it further downhill every single time. It looks prettier but it’s so bad when they rewrite it. Actually, I think that a lot of technical debt in the database world is around poorly designed databases where, “oh, we’re just going to– you know, we’re going to have a table that has– yeah we’re not going to concern ourselves with normalization, we’re not going to add these keys here because– or we forgot to think about the keys.”

Adam:             Hey, I’m 6th normal form ride or die, man.

Eugene:           He named his dog Bosco or Codboy.

Kevin:              I’m with you to 5. I’m with you to 5.

Adam:             No, I’m on 5, too, I just–

Carlos:             And if we can get three down, I’d be happy.

Kevin:              Yeah, well, Boyce-Codd.

Eugene:           That’s what I was trying to say.

Kevin:              Three’s not good enough. Boyce-Codd or nothing. But Boyce-Codd normal form aside, this is where we make a lot of rash table decisions and end up having to write complicated queries to deal with the fact later on, or we find out that, “oh, this thing that you promised me would always be one-to-many has now become many-to-many and now we have to find a way to jam it in somehow, so we’ll just add another column that’s like a binary column where I’m going to store the binary serialized array.”

Carlos:             What’s the worst that can happen?

Adam:             I have dealt with serialized arrays in columns. I have dealt with someone who tried to do a tree structure but did it wrong and we’re still having to live with it to this day. I’ve dealt with even just on like naming schemes where they don’t do pluralization correctly to indicate relationships between tables. Like every time you’re saying something, you’re hitting something that I’ve had to deal with at my– yes, absolutely.

Kevin:              Or bit logic where it’s like this column is an integer that represents wealth. If it’s divisible by 32, then it’s going to have this flag turned on. So, flags and using the logical ampersand operator to figure out what things are on.

Adam:             Because one more bit of column was going to make that big of a difference to flag that value.

Eugene:           So, taking kind of a step back for a second, I think one of the key differences between dealing with technical debt with like say, C# or PHP or F# or whatever, versus databases–

Adam:             F# has no debt.

Eugene:           Well, any kind of programming language versus databases is that if your definition of technical debt is stuff that’s giving you friction, well, the friction is different. That mu coefficient that you learned in physics back in high school, that’s different in databases, Because you think about it, in an ideal world you can take a class in C# and run ReSharper and change all the innards and because you have hopefully an interface, as long as that surface area remains the same, it doesn’t matter what you do on the inside. Ignoring coding standards and the stuff you talk about. And the problem is, in the database world, once you’ve built a table structure and you define that schema, it might as well be set in stone for how hard it is to change. You decided that you were going to use the money datatype to store money? Well, that’s what we’re using for the next 10 years.

Adam:             Right. I swear I’ll always use integer from now on. I’ve been bit so many times. Literally, like two weeks ago, I had somebody do a migration that was nine four and nobody could make more than $999,000. Great, great, thank you, thank you, appreciate that.

Eugene:           That’s just CEO pay control. I don’t see a problem with that. No one in the company should get paid more than a million dollars.

Adam:             Well, when it’s people using the product to make money, they get kind of cranky.

Eugene:           No, no, no, it’s a political statement. That’s what that developer was was doing.

Adam:             Oh, okay. Occupy Database.

Eugene:           Yeah, exactly. But I think that’s the big differentiator, because the consistency thing still applies. And I’ve lived the pain of this home-grown ERP solution built by a fire protection company, and it was like a hermit crab shell where it just grew around the organism. And so like, in the customer table, you’d have like Email and you’re like, “oh, this is the customer’s email, right?” It’s like, “oh no, we’re using it for the URL now,” or something like that. I think I might have a backwards. It’s like, “yeah we’re just going to reuse columns, because it’s too hard to change stuff.”

Kevin:              Address Line 3 never is an address line 3. I don’t think there really is in real life.

Eugene:           Nope. So yeah, the consistency things still applies to both, but I think how easy it is to change some things versus others is dramatically different. And so suddenly, that effort up front for design is way, way more important, I think, in the database world, when you’re talking about your relationships and you’re talking about your datatypes of stuff, than maybe it is for the programming language world, where it’s a lot easier to change something to a flow to an int. I mean for me, I’m a fake programmer, so I just change it and see what everything broke.

Adam:             Hey, unit coverage.

Eugene:           And it’s like, “oh, these five files won’t compile anymore.” You go fix it; you’re done. Right? So, I don’t know.

Adam:             So, I hear you, and to me, it had to be a comprehensive strategy. We had to start by communicating clearly about what’s a gap, what’s a bug, what’s debt, and what type of debt was it? So, that unauthorized, that authorized, and then that inherited, and all authorized debt pretty much instantly becomes inherited. And when you’re dealing with inherited debt, my position is that if it not impacting your ability to efficiently deliver the business need, it’s okay. Now, if it does not align with your coding standard, yes, it’s debt. It should be documented as debt, it should be labeled as debt, don’t do it this way. But, as long as you’re making those smart compromises to empower your client to be more successful, you’re going to have, hopefully, more deliverable deadlines. Because I used to be the guy who insisted on turning in correct code and just working however many hours were required, and it’s a fool’s errand. And I can go into the reasons why, but I don’t want to eat up the rest of the show. So, the point being that that kind of overachiever mentality of ‘it must be perfect code’ was one of the things I had to shed, and I had to shed it the hard way, unfortunately. And so, having come past that and now gotten to the point where I’m willing to live with debt, provided its debt that served a purpose, or it’s debt that’s on the agenda to be cleaned up. And so, I really see the architecture role, like when you talk about like bad table design is an insurmountable problem. I have three cases I can name off the top of my head where the person who built it didn’t think through what an invoice should actually be, so I’ve been working with this partial invoice structure for a long time. And so there all these like optional 1-0, or 1-1-0 tables hanging off of it now, and it’s a mess. So, yes, I think if you’re going to try and mitigate debt up front, to your point, Eugene, I think getting your data architecture correct is your number one priority. I know that it’s the thing I put the most time into when I’m actually cutting code. When it comes down, I’ll go through the API we’re going to interact with. I’m going to make sure I understand what everything needs to be and how it needs to be organized into our system, and I’m going to get those tables right. Or, like when we had to go add GDPR support, I mean nothing was built with the idea of GDR support and now we’ve got to go back through and find everywhere in our database that has that kind of information in it, and that was a huge challenge. So, you can’t possibly plan for everything, but at the very minimum, you know, I like to use the nouns and verbs like the ‘has many’ and stuff. I try to talk to my client about that to kind of get them to articulate to me how they see things. And that goes back to what you said, “if you tell me it’s one-to-many and then you come back and tell me it’s many-to-many, you’re going to need to pay to fix that. I’m sorry, I’m not a miracle worker.” And I think that it’s super important that we don’t put ourselves in this position of trying to be God for Tech. Because it’s super easy to get like, “well, if I do this, this, this and this, I can maybe make that work.” And that serotonin hit you get for being a hero in that impossible situation, that will lead you down a very dark path. And it’s much better to turn around and look to them and say, “look, it may be possible to keep going down this path, but I highly advise you to not do it, and here’s why.” And talk about it up front, that way you can get the authorization to clean it up later.

Carlos:             Yeah. There was an episode, we talked about the superhero syndrome, where we like to swoop in and be like, “oh yeah, I can fix it.” If we’re always kind of fighting that stuff, we’re never really moving forward, cause we’re always just–

Adam:             It’s the same reason Twitter’s addictive. It really is. You get a retweet, like, “hey got a re–” you know, like it’s that same thing. It really is. And I have my own self esteem issues and everything from when I was a little nerd geek, and now if I get a chance to be the hero, come up, cape blowing in the wind? “Yes, I saved your database, yes.” That’s such a great feeling, but the cost that goes with it, because if you’ve got a fundamentally broken data relationship, and you do pull off some miracle, what’s the maintainability on that going to be, going forward? I will almost guarantee you that if you have just had a hero moment, you’ve just guaranteed vacations are going to be hard for you to take. And it’s generally a good idea to avoid that kind of setup. I’m a passionate programmer, I’m a hobby programmer. You look at my stuff, like it’s quality code complete, if I’m writing some Go line for fun at home. But at work, that’s a different set of standards. And I really feel like this talk’s entire focus was about evolving mindsets to be like, “how do I approach this as a professional here to do a job, rather than as an enthusiast who’s passionate about this subject. Because it’s really easy for us to get confused or for me at least to get confused on that.

Carlos:             Right. Now we are getting close to time here, and I’ll let you have closing thoughts of course, but one of the things I always like to try to do is okay, so now how do we– I don’t want to simplify it, but how do we fix it?  What are some are the takeaways to be like, “okay hey, how could I better wrangle this?” It sounds like you gave us at least one idea. So obviously standards, number one, that’s important.

Adam:             Enforced, yeah.

Carlos:             So number two, you have to have someone who’s willing to take on the mantle of, “okay, if the standard’s not there or if we need to change it, I’m going to make that decision, or at least have the authority to and then percolate that the rest of the team,” and be like, “hey, I’m going to make some decisions here.” Yeah, it will take input, I’m not saying you do it on an island, necessarily, but at least that person is responsible for taking that and then set aside some time to do that. Is that a–

Adam:             Yeah, so, I think communication is the number one thing. You’ve got to be communicating with the client up front. Next you’ve got to have good technical leadership, like you’re describing there. Which I think comes down to both propagating the coding standard, the one responsible for when it has to evolve, and also a group of people who are willing to collectively represent everyone when it’s time to enforce code reviews. Because I’m not there reviewing code on behalf of my opinion. I’m reviewing code on behalf of everybody else on the team who doesn’t want to have a hard time maintaining your stuff, so like I never should have let that single page web app go to production. I should have made him take that out, because now, nobody wants to touch it. And then, so for me, communication, technical leadership and the manager willing to actually put people on performance improvement and eventually let people go if they’re not willing to get on the same page. Because it’s just like any other team; if you’re not using the same playbook, you’re doomed to fail. And then from those two points I kind of go to a third point, but before that, I want to hit these three maxims. Which is to say, always choose consistently bad over inconsistently good, because that’s going to help you be more efficient, and with efficiency comes the opportunity to refactor that consistent problem, and it’s much easier to solve a consistent problem than an inconsistent problem. The next one is, every new hire, even a new architect is responsible for learning the existing coding standard. Their job is to show up and wrap their brain around the mental model that was in use before they got there. Because otherwise, if they show up and they start laying down the law and changing standards, all they’re doing is creating more variation, more debt, more friction. So, from my perspective, my job, if I got hired tomorrow to go work at such and such a place, the first thing I got to do is show up and learn everything they’re doing and how they’re doing it and get on that playbook. Because I can’t help them make smart decisions until I fully integrate with where they are currently. And then the third maxim is that, if we’re doing our jobs correctly, 99% of technical debt should have been bought and paid for by the client who knowingly made the compromise to get to market faster, and that other one percent should be somebody is on a performance improvement plan or on their way out the door. We have to be compassionate and willing to help nurture people. I’m not saying you fire them the first time they violate the standard, but the point being that if someone can’t get on a documented standard, maybe try improving the standard. If that doesn’t work, try and get them to pay more attention to it, and if that doesn’t work, well then, we need to replace you with someone who can. Because, I’ve been in a situation where like somebody stuck around too long and they kept deviating from the standard time over time over time over time, and all of that code is now my problem, because nobody else wants to touch it. So those three maxims are kind of the guiding point. And I think that if you follow the communication requirement, the technical leadership and enforcement, the clarity around the standard, then the third thing you get from all that is you get a client who’s empowered to be more successful in their business. And really at the end of the day, that’s what we’re getting paid for, that’s what we get promotions for, that’s what we get bonuses for, that’s how we get time off, what not. And it’s also how we build trust with them. And if we’re focused, very, very focused on getting that business value to them in the most efficient means possible, even when it cuts across our particular grain, that’s when you’re going to see that client love you and they’re going to be crazy about working with you. And then the flip side of that is if we’ve done that for the client, it empowers us to spend the time building maintainability where it matters, for example. Because if I go in and I try and make every feature maximally maintainable, one of those features is going to matter more than the other; we talked earlier about those feature paths that get hit the most. And some of those paths do merit a higher coding standard than some of the other ones. And it’s okay to have that documented as long as it’s documented to say like, in these particular areas, like where I’ve got seven payment processors, it’s one of the only places in the application where we have to use the ability to plug in different kinds of payment processors and they all have to have the same interface. I try to avoid that kind of architectural complexity most other places in the app, but where it’s needed, it’s needed. And if you’ve spent your budget across the entire problem, you’re not going to be able to focus on the stuff that really matters. And then if you’ve built the relationship with the client, you’ve done all this stuff, I think that empowers you to come back and say, “look, I’ve been here for you, I’ve been executing for you, I’m being honest with you when I say that this is actually a problem that is actually slowing us down and impacting your ROI.” You will never get a client who pushes back against their own ROI, but you got to build the trust and you’ve got to not have used that. You’ve got to be very realistic and I would almost say harsh with yourself, going back to some of the points that you gentlemen, Eugene and Kevin raised. If someone is just not feeling this particular codebase, then maybe there’s another role for them, maybe there’s another place for them, but like a team is a team. And maybe it’s just because I played football, but we couldn’t go out there and like somebody wants to run an ISO and somebody wants to run a post, like that don’t fly. So, I think that summarizes kind of where I was trying to get to with the strategy of the whole thing.

Carlos:             There you go.

Adam:             And yeah, I’m happy– people can hit me up on LinkedIn and stuff. I can’t take 15 minutes for everybody, but if someone has a problem that they want some help with, hit me up. I’ll be more than happy to at least hear him out and do a FaceTime and beer or something while we talk it out.

Carlos:             There you go. Now let’s see, we’ll hit it again, but your LinkedIn is?

Adam:             Oh, just /in/Adam.Lenda. So, (spells out) and you’ll find me real fast that way, or [email protected] Either of those will get to me. If you put something about the podcast in the header of the message, that will help my eye pick it out of the background noise.

Carlos:             There you go. Yes, he even connected with me, compañeros, so he’s very nice. Okay, so other final thoughts, maybe Kevin, Eugene? You guys want to jump in here?

Eugene:           Yeah, I mean I think it’s interesting, because a lot of the talk focused on consistency across the board and definitely whenever I’m used to thinking about technical debt, I’m thinking about not necessarily inconsistency but shortcuts and that sort of thing. But I think both are definitely relevant and I think was a good discussion, for sure.

Kevin:              Yeah, it was an enjoyable discussion.

Carlos:             Should we do SQL Family?

Kevin:              Let us.

Adam:             Sure.

Eugene:           Yep.

Carlos:             Let’s do it.

Adam:             Do you guys have to go first, or I just got to catch up on back episodes?

Eugene:           We’re already family. You have to get into the club.

Adam:             This is the beat-in part, right?

Eugene:           This is the hazing.

Carlos:             This is the initiation.

Eugene:           If you don’t pick the right superpower, you’re out!

Adam:             Fair warning, 10 years of martial arts training. Bring it.

Carlos:             Your episode will not go live unless you answer these questions correctly.

Kevin:              You have 48 seconds.

Adam:             All right, so all-time favorite movie, Deadpool. I keep hoping that if I were talented enough, I’d be able to say all the stuff that goes through my head, but which my brain currently has to filter out.

Kevin:              What did you think of Deadpool 2?

Carlos:             Oh boy.

Adam:             It was also very good, but I think the first time will always be your first time and it really blew my mind the first time I watched it. And I was like I’ve been basically trying to do that professionally my whole life. It’s just that my skills didn’t quite justify it, hence the firing that I mentioned earlier.

Kevin:              I’m so glad they got Ryan Reynolds to do that.

Adam:             Oh my, it was great. So, a place I want to visit? I’m actually working on a plan to get all my debt and stuff squared away over the course of the next few years. I plan to try and do like three months in a different city, you know, every 3 months and I want to definitely start with like China, Japan and Thailand, because I’ve been– so much of my code has been fueled by their cuisine, I figure I should learn something about their culture.

Carlos:             Well, you didn’t say Italian, so I guess, you know, there’s no spaghetti code.

Adam:             Well, you know, you give me all that pasta and I’m going to go take a nap and I don’t think any project manager’s ever been like, “here, let me bring you something to make you sleepy.”

Carlos:             So, a food that reminds you of your childhood.

Adam:             Yes, my mom used to make taco salads like twice a week and there was a time when I got sick of it and I couldn’t stand it but now I make them myself. So, Mama’s cooking’s always Mama’s cooking, right?

Carlos:             There you go. Now so are there are specific toppings you have on your taco salad?

Adam:             So, I go to the grocery store and I look around and I’m like, “what could possibly be good on this? What might not even be good on this, but I want to find out?”

Carlos:             Interesting, okay, so you’re looking for variation.

Adam:             I cook with a very open mind, often for, you know, live dangerously, so it’s a lot of fun. A lot of fusion going on there, yeah.

Carlos:             Yeah, my kids have beat some of that creativity out of me.

Adam:             Oh yeah, the whole like chicken nuggets forever thing, right?

Carlos:             Yeah.

Adam:             I have a new nephew who’s that way.

Carlos:             Okay, now we mentioned you’re a developer and I should say you’ve been messing with– oh my gosh.

Adam:             Well, let’s put it this way–

Carlos:             What’s the name of the database?

Adam:             MySQL.

Carlos:             MySQL.

Adam:             MySQL’s been my main squeeze for 8 years. Not by choice. You don’t get to refactor your data, right?

Carlos:             That’s technical debt.

Eugene:           Hopefully it’s on strict mode.

Carlos:             That’s technical debt.

Eugene:           I pray for you that it’s on strict mode.

Adam:             Guys, at this point, I know it so well. Like I know all of its little quirks and stuff.

Carlos:             If you need a migration, let me know.

Adam:             Yeah, well, there are definitely conversations we can have. But, for me that was a really hard shift because I came from SQL Server 2008 at the time, and it was what I had the most experience with, and I cut my teeth on 2000, way back, I guess. And Mark Spinek over at ComSys, back when it was ComSys, really took me under his wing and trained me up to be like 60% developer, 40% DBA, was kind of his goal. I think I got maybe 30% of the way there, but it’s really borne out. I understand queries really well, I understand data architecture, like the normalization, like we were making jokes about normal forms and I’m a developer who can do that, yay! So, I am essentially our data architect and DBA for our system, because I have enough skill to fake it and then when I run up against something I can’t do, I get a real professional in to help. Yeah, so that’s my background. And frankly, I’d love to go back to the Microsoft world. I miss F#, I miss C#, I miss .NET. I like compiled languages, I love static analysis, these are all tools you can use to help write better code, more consistent code.

Carlos:             There you go. So now you’ve been out of it for a little while, so if you need to skip this question we can, but if there’s one thing you’d want to change about SQL Server, what would it be?

Adam:             Its price point.

Carlos:             There you go.

Adam:             Because if I could get it on a price point that my CIO would swallow, he’d be choking that down today. I’m just saying, cause like the most interesting thing I ever did with SQL Server was an app that was partially connected. We had people deep down inside buildings doing surveys and they didn’t always have an internet signal. So, we had a master server with all the backups and everything, and then they would come back out of the hole and they would have to merge rep back, so we had this whole like reconciliation process. There were libraries. It was like, this was a capability. Microsoft said, “we have two different ways to do this. You can do it this way with these trade-offs, you can do it this way with these trade–” MySQL was just like, “hm.” No help, no help. You know, just dot dot dot.

Carlos:             Good luck with that.

Adam:             So, I very much enjoyed SQL Server and I do miss it.

Carlos:             There you go. So now, what’s the best piece of career advice you’ve ever received?

Adam:             Again, I gotta go back to Mark Spinek. Very influential point in my career, I guess, but he said, “be a developer to doesn’t need a project manager.” And I think that there’s a lot of depth there, I think that some of what I had in my talk about a great developer keeps their eye on the ROI, that kind of like not having to have someone check behind you to make sure you’re getting your stuff done. These are the things that make you leadership material. These are the things that make you worth more than your peers, to give you the greater opportunities to go work on more interesting projects. Like if you’re bored where you’re at, try and solve that problem and see if you can’t find yourself in some more interesting positions because of it.

Carlos:             Sure. Okay Adam, we appreciate you being on the show today. We do have one more question for you.

Adam:             Alright, bring it on.

Carlos:             If you could have one superhero power what would it be and why do you want it?

Adam:             This might start a whole new show, but time both creates and destroys all things, so if I could manipulate time, I could have the two things in the world that I want most, and one which is more time to work on my theory of artificial consciousness.

Carlos:             Oh wow, there we go. Okay.

Adam:             Which is tongue-in-cheek to some extent, because I’m not a big data scientist, I’m not an artificial intelligence researcher, but I’ve been working on the problem on my spare time for like 10-15 years just for fun and it’s just a hobby project of mine and it’ll probably never go anywhere, but if it does, I will be Tony Stark.

Carlos:             There you go. You’ve got Jarvis already.

Adam:             That’s the goal.

Carlos:             Well, awesome, very good. Thanks for being on the program today. We appreciate your insights and some of your thoughts.

Adam:             I haven’t had this much fun since COVID started, man.

Carlos:             Well, we do aim to please around here. Okay, I think that’s going to do it for today’s episode. Thanks for tuning in. If you want to connect with Adam or the rest of us, you may do so. We’ll get to that in just a second. Again, we have our show notes link, we have a post that Adam gave us about falsehood programmers believe about addresses, kind of going back to– not data quality.

Adam:             Data structures, yeah.

Carlos:             Data structures, thank you. And I think this actually started off topic, but we were talking about French fries on salads and so there’s going to be a link to there about Pittsburgh salad French fries.

Eugene:           Yeah, our local cuisine.

Carlos:             Our show notes will be at sqldatapartners.com/techdebt and if people want to get in touch with you, Adam how can they do so?

Adam:             [email protected], just say something about the podcast in the subject line. I’ll set up a filter to grab on that, and then LinkedIn is also good. Same spelling, /in/adam.lenda. And generally speaking, there aren’t very many Lenda’s spelt that way, so you should have no problems finding me.

Carlos:             There we go. Eugene?

Eugene:           Yeah, you can find me on Twitter at sqlgene.

Carlos:             Kevin?

Kevin:              You can find my details by doing a bitwise operator against the system tables. That’ll give you my Twitter handle, but I won’t actually answer it.

Adam:             Look out for that row-level permissions locking on that. You gotta be, you know–

Carlos:             That’s the secret, ssh, you’re not supposed to tell.

Adam:             Oh yeah, right?

Carlos:             And compañeros, you can reach out to me on LinkedIn. I am at Carlos L Chacon. Compañeros, just a reminder, we will be taking a break between this episode and Episode 200, but we will be back. Thanks again for tuning in, we hope that you are safe and finding something to do while we’re all cooped up. And one day, compañeros, I hope to see you on the SQL Trail.