Episode 128: Database Doing DevOps

Episode 128: Database Doing DevOps

Episode 128: Database Doing DevOps 1300 1100 Carlos L Chacon

To Devops or not to Devops, that is the question.  We’ve talked before about how the developers have all the cool new tools, but this is slowly changing.  For those in development or dare I say agile (gasp!) environments there is more need for automation, but getting there can be a real challenge.  In this episode, we chat with John Morehouse about how he actually put into practice automated deployments for the database, some of his challenges, and how long it took to get there.

Are you trying to implement continuous delivery for the database?  Let me know in the comments below or by social media.  I am very interested to hear your experiences.

Episode Quotes

“One of things that has to change when it comes to Database Continuous Delivery is the DBAs have to get out of their chair and go talk to the developers.”

“Database Continuous Delivery is not something that you can install, flip a switch and woohoo, voila, you’re done!”

“The test process or framework is just as important as the application code or the database code.”

“Don’t notify me when things are successful. Notify me when things fail.”

“Communication is going to be a huge component of getting this into place.”

Listen to Learn

00:40     Intro
01:35     Compañero Shout-Outs
02:57     Intro to the guest and topic
05:18     DBA vs Developer roles
06:26     Get your database in source control
08:01     DBAs have to communicate with developers
11:46     What role am I going to play?
13:43     What is going to be best for this environment?
15:43     Who should lead the charge
19:09     Normal life still has to happen while you get things going
21:42     Dedicated models are a good idea and testing is critical
27:40     Security and who should have access to what
31:29     Why use a dedicated model?
33:19     Hopefully you have a good recovery strategy
34:52     SQL Family Questions
39:39     Closing Thoughts

Alex Yates blog post: http://workingwithdevs.com/delivering-databases-migrations-vs-state/

About John Morehouse

John Morehouse is currently a Consultant with Denny Cherry & Associates living in Louisville, Kentucky. John lead the Omaha SQL Server user group for 7 years and is now a leader of the Louisville SQL Server/Power BI user group. He is honored to be a Microsoft Data Platform MVP, 2016 Idera Ace, Friend of Red Gate since 2015, Sentry One PAC member & Community Ambassador.  He is also a blogger, avid tweeter, and a frequent speaker at SQLSaturday as well as other conferences.  In his spare time, you can usually find John on Twitter (@sqlrus) as well as chasing his two young sons around the house.

*Un-transcribed Introduction*

Carlos:             Compañeros! Welcome aboard the SQL Trail once again. This is Episode 128. It’s good to have you listening to the program today. Our topic is DevOps in the Database. This topic was suggested by Christian Satnic, and it is one that many folks are grappling with. So, we’re interested to have this conversation today. I think we will continue to evolve this conversation in future episodes. I can already see a couple of additional subtopics, if you will, that might come out of today’s conversation. Our guest is John Morehouse.  He is a DBA from Kentucky, who is now working with Denny Cherry and Associates. He was able to implement devops in the database in one of his previous employers. So, we’re going to talk a bit about that experience, and some of the challenges that he faced, in addition to some of the benefits that he gained once he was finally able to put this into place.

We do have a couple of compañero shout-outs today. We want to give a shout-out to Kelly Armento, Jeff Nichols, Asrar Ahsan, and Peter Shore. So, this episode and last week’s episode, episode 127, answers, or I feel ties in well to one of the questions that I frequently get. And this is “what am I, as a DBA, going to be doing in 5 years?” So, there’s no question that last week’s episode on analytics tied into this idea, and I think everyone’s aware that we’re seeing this explosion of new analytic tools and opportunities and so I think there’s going to continue to be a movement there. The integration of some of these analytic tools into the database, I think, provides some interesting opportunities for some of us, and I think today’s episode does very much the same. Now, there are lots of opportunities for devops in the database, just because there is so much confusion around the subject. I think there is a bit of a lack of leadership, and the tooling has not been where it was. And again, I think it’s getting better. It’s also not an easy process to implement, but I think you can still get in on the ground floor, if you will, and provide a lot of value to organizations because there is still such a need out there.

Our URL for today’s episode is going to be sqldatapartners.com/devops or at sqldatapartners.com/128. So, let’s go ahead and jump into the conversation with John.


Carlos:             Well, John, welcome to the program.

John:               Thanks, thanks for having me.

Carlos:             Yeah, it is a pleasure. And thanks for joining us from Louisville. Today we’re going to be talking about devops for the database and that idea of maybe doing more with less, maybe a little bit faster. On this program we have talked about, and my compañeros know that devops from the application side, getting a lot of love, there’s lots of tools out there. But from the database side, it tends to get stuck, and I think that’s for a couple of reasons. One is that the tooling wasn’t necessarily there. One is I think there was some uncertainty from the DBA community about what to do or it was just learning curve. But then also we deal with the fact that the database has a state. It’s not like code, like “oh, run this code, run that code”. There’s more than just code that I have to take care of. And those three things can be tough to deal with. So, I guess, just kind of jumping in here, how then are we supposed to tackle some of these things?

John:               That’s a really good question and I would probably say “slowly”. And a little back-story, when I started looking at Database Continuous Delivery, I was working for an organization in Omaha, Nebraska before I moved to Louisville, and our deployments, from a database perspective, were really slow and cumbersome. I was one of four database architects and we had 10 application development teams with probably about 10 people per team, so we basically had four operational DBAs that supported 100 people, give or take, ranging from managers to developers to QA. And each team had their own DBA as well. So, we managed deployments for a lot of people, and when I left the organization, at that point in time, we were doing deployments probably every day.

Carlos:             Okay, sure, a lot of stuff going on.

John:               Lot of stuff going on. But what I noticed was, we as DBAs, were always not trusting us to what the changes were going into production.

Carlos:             Yeah, that gate-keeper mentality.

John:               And that’s very true, that’s part of our job. We want to protect production to make sure nothing volatile goes in there to make things go bump in the night and I have to wake up at 3 o’clock in the morning and go fix stuff.

Carlos:             Exactly. Yeah, there’s that rub, so from the database perspective, our job is to make sure that it stays up, but from the developer’s job, they’re hired to make sure that things change.

John:               And change as fast as the business will allow it. And so, they want to change as fast as the business wants to allow it, but the DBAs want to slow it down because we want to check the scripts, or dropping logins or we’re granting the intern SA rights and production. Not being malicious, but things just sneak in by fault of the development cycle. And so, part of that is that we as DBAs, in order to start going down that path, we have to start looking at the toolset and figure out how to trust the tools. What I usually recommend, the very first thing, is if your databases aren’t in source control, get them into source control. There’s a couple different products you can use. Redgate’s got a SQL source control that plugs into Management Studio. It’s really slick. That’s probably one of my favorites. I’ve also used Database Projects from Microsoft and Visual Studio, also a great solution.

Carlos:             Right, and I think there what I’d like to see, one of the tricks there, is that exact scenario. So, from the DBA perspective, they like using the Redgate because it fit into the SQL Server Management Studio, but the developers are used to using Visual Studio. And so, we did have a little bit of tug-of-war, because unfortunately, at least at the time that I was working on it, this may have changed, so check your current documentation, compañeros, but you had to kind of choose one or the other. If you were going to build a project, you had to go with one or the other. So again, those are just compromises that you’re going to have to start looking at from the very beginning.

John:               And that’s very true. I will tell you, though, in my experience, I’ve implemented or helped to implement Database Lifecycle Management, Continuous Delivery in two organizations now. One in Omaha and one here in Louisville. And I actually went around and talked to the developers and said “hey, when you do SQL database work, what are you using?” And I will bet you that 85% of them, yeah, they’re doing C Sharp code in Visual Studio, but they’re just as unfamiliar with a database project in Visual Studio, so they would actually switch over to Management Studio.

Carlos:             Oh, interesting. Well, then you were very lucky.

John:               So, what my recommendation would be is, go talk to your developers. That’s one of the other things that have to change when it comes to Database Continuous Delivery is the DBAs have to get out of their chair and go talk to the developers. We have to break down that communication barrier where we don’t want to talk to anybody, all we do is say no. We actually have to have those conversations with them to find out how they’re doing the work to get us out of the picture.

Carlos:             That’s right, so if we want to remove ourselves or make this faster, it’s definitely a team sport at this point.

John:               Absolutely, again, we want to see all the changes that are going into place, but the real outcome of this should be, we are removed from the process, because we are the bottleneck. I’ve never been in an organization where the DBAs outnumber the developers. Ever.

Carlos:             Yes, interesting how that works.

John:               I don’t know of anybody where the DBAs outnumber the developers. So, if you have 100 developers that are moving at a very fast pace, we can’t keep up. That’s just supply and demand.

Carlos:             And I would caveat there, so some people might be saying “well, we have deployment folks”. We have people that basically just manage the deployments and I think that’s very, very common, but I also think that’s kind of a bandaid to the situation. We’re not really fixing the problem there.

John:               That’s totally a bandaid. When I went through the first iteration of this, back in 2012, 13, somewhere around that timeframe, I was a diehard DBA. I still am a diehard DBA. When I start to have the conversation with my developers and my enterprise architecture team about “how can we make this better” and “oh I’ve got this idea, let’s call Redgate, see what they offer”. And Redgate actually sent two individuals over to Omaha. One is Alex Yates, who I highly recommend, DLM Consultants for Continuous Delivery, and another gentleman by the name of Sean Newham. And when they walked in, I went “there’s no way I’m going to buy off on this. There’s too many parts, there’s too many pieces, security and things changing. I gotta say no.” But really, after two and a half days of literally spending two days in a meeting room together, myself, the two gentlemen from Redgate and some developers from one of our AppDev teams, by the end of the two days, I was completely sold. The process completely sold me because it gets me out of the picture. But the trick is that I have to trust the tools.

Carlos:             Sure. And is there enough reporting there so that you feel like you can still be part of the process and I think there’s that idea that deployment’s going to happen and, all of a sudden, they’ll be like “hey, can you fix this”, kind of a thing.

John:               Right, and so we actually went through that process and really Database Continuous Delivery is not something that you can install, flip a switch and “woohoo, voila, you’re done!”

Carlos:             No, that’s right, it’s not just a single tool going to get you there.

John:               Right, so this has really got to be a long-term investment for the organization. But again, the outcome is that you remove the DBAs from the picture, and I really want to be able to give my developers that I support the ability to check in code, database code, have some unit tests run against it, it gets validated and tested, everything’s groovy, let it go all the way up the chain to almost production, and then somebody click a button. Or even better, I want to be able to schedule it. I want to be able to schedule that database release to go out when I’m at the bar drinking bourbon. That’s really what I want. And I want to get a notification that says, “deployment X, Y and Z was successful.” “Sweet, oh barkeep, I need another bourbon.” Not that I drink bourbon all the time, but that’s really what me, as again, a diehard DBA, it makes my job easier to have Database Continuous Delivery. Why wouldn’t we want that?

Carlos:             I think there is that idea of, “what role am I going to play? If I can’t have my finger on the release, then what exactly am I doing?”, because backups and some of those other things, those day-to-day things just kind of get old. And so, I think that there’s some, you feel like you’re relevant when you’re actually executing some of those things.

John:               Probably more often than not, if you’re looking at doing Database Continuous Delivery, you are probably a larger organization. If you are a small shop and it’s you and you have 5 developers, you can manage that. Things probably aren’t changing as rapidly, which would cause you to look at a solution like Database Continuous Delivery or Database Lifecycle Management.

Carlos:             I also think maybe long-term, jumping to the next place, one of the things that we’ve talked about on this program before is the evolving role of the DBA. And I think that the continuous integration components, they give you another specialization, so analytics is not the only route. You could go into this continuous delivery and there could be a role for you that would still allow you to play with data, but perhaps not have the title of DBA, even.

John:               That’s true. devops has been a buzzword around the community for some time but it’s still fresh. Desired State Configuration is another one that has been cropping up lately is “oh, we’ve got to have DSC and we’re going to put everything in devops and whatever”. But definitely, having some experience with implementing Database Lifecycle Management, or Database Continuous Delivery would definitely be a boost to your resume. And knowing the tools and the players in the community that can help you implement that, and I’ll be the first to admit, I don’t know everything about it, but I know who to call.

Carlos:             Yeah, that’s right. Even with the tools, there are still some options or maybe I’ll call it culture and process that you still have to then figure out. What’s going to be the best for this environment? And different places or different projects, even, are going to have different needs.

John:               Absolutely, and that’s where I would recommend that if you’re not familiar with it, you get somebody like Alex involved. At my most recent employer, I started the ball rolling to get DLM installed and configured and working, and really what you have to do is you have to get the consultant, in this case it was Alex. He came on-site, I got myself, Alex, a couple developers, some QA folks, and we literally sat in a room for two or three days, again, and really went through the process of “how do we release code today? Not only just C Sharp code, but database code. What are our pain-points?”, let Alex take that information, look at some solutions. He came back and we actually put together a proof of concept of various solutions, and then we let everybody kind of play around with it, of the group.

Carlos:             And when you say DLM, you’re talking about the Redgate Database Lifecycle Management tool, right?

John:               When I say DLM, I mean just database lifecycle management in general. Redgate does have some tools that will help manage lifecycle management.

Carlos:             Like the dashboard that they have.

John:               Yeah, they’ve got a suite of things that come with the developer toolkit pack. You have to buy that license. And again, it’s not necessarily cost-effective up front, licensing always costs money, but the end result was that after Alex was on-site for a week, we came away with a very solid solution that worked for our organization, and it made sense to management all the way down to me as a lowly DBA, that this all works. But we had to have that conversation, we had to get developers involved, we had to get QA involved, management, DBAs. We literally all sat in a room and had to have that conversation.

Carlos:             So now I’m curious. I’ve had a similar situation where I saw the need to move a little faster. Some of our folks were spending a lot of time on deployments, and I thought “okay, hey, let’s move this forward”. It was in a situation like “okay, our first step should be source control. Let’s get it in there.” But then going to that next step I, from the data team basically, had a hard time corralling everybody together. And while the heads were nodding in the right direction, there was really nobody leading that charge. So, I guess in your experience, who does need to lead this and say, “okay, guys we have to buy in”?

John:               In this case, and actually both cases, I was probably the one leading the charge, or pushing it.

Carlos:             Oh, interesting. Because you think about, again, we want to put in this process, but it’s going to require additional toolsets, which means I’m going to have to come up with spend and how we’re going to do some of these things, even the testing components. Those aren’t necessarily all in your hands. But you still were kind of leading that, and like, “hey, let’s make this happen”?

John:               That’s very true. Again, this was not something that I flipped a switch and it magically was delicious. It was a long process. My most recent employer, it actually took me probably 14 months, a year. A year to 14 months or so of really sitting back and going “hey, this piece of the deployment process doesn’t work for the DBAs. It causes headaches for the developers.” In the back of my mind, I’m going “well, I we did DLM or Database Continuous Delivery, it would fix that or it won’t fix this,” and kind of look at that. But I also would start to have conversations with the leadership as well. Part of my role as the database architect was to look 6 months, 12 months, 18 months, 2 years, 3 years down the road and say, “where do we want to be?” and it was part of my job to help drive us there. I started to introduce certain concepts to certain people that I knew would be receptive. I knew what developers would listen to my advice and I would hit at stuff and show them things and they’d be like, “oh wow, that’s totally cool.” “Yeah, let’s go down that path. Let’s go.” It was a long and drawn-out process to get the ball rolling. But once it got rolling and everybody saw the value of it, I would bet you that most developers, they don’t want to talk to me anyways, because, as a DBA, I tend to say “no” with valid reasons. But if we have a process where they can just make changes and we have some pre-set unit test of some fashion in place, that will validate whether the code is good or bad, and they don’t have to talk to their DBA, that’s gravy. They can just check stuff in, it goes and we schedule a release. “Sweet, cool. Let’s go to the bar.”

Carlos:             Yes, and I know that we are going to have another conversation in a future episode about doing some of that testing, but there are multiple pieces in there, and I think it’s interesting to give your, you know, that it takes a long time. A year, it’s not eternity or anything, but it’s not again, something that you could do on your own. It’s going to take talking with people and that team effort. And I thought it was also interesting, kind of tweaking the process. “Okay, here’s initially what we thought it would be, but then it’s not working for these people. How do we have to change it?” So, there is some iteration, even to the process of then trying to do these deployments. The frustrating piece is that normal life still has to happen. We can’t just put deployments on hold for a year while we figure this thing out. And think that’s another challenge that we tend to have, is that for all intents and purposes, we’re kind of using two systems in parallel. And for a lot of people it can feel like they’re doubling up on work.

John:               And that’s very true. So, two thoughts I have about that. The organization I worked at in Omaha, we actually picked a complete development team and said, “sorry boys and girls, you are going to do the new process. You are the guinea pig.” And from a DBA perspective, it was a little bit harder, because I couldn’t do my normal deployment process with that team because they’re going to use the new shiny, fancy process. So, I did have to juggle both ways. And that is just an understood risk that the organization has to make the decision to do. Then the other thing I would mention is that it really does depend on your culture and management. I’m very thankful that the last two employers I had, the culture was such that, “okay, yes, it’s going to cost X, Y and Z to do the licensing and get the consultant in here, but we want to be able to deliver to our customers in a faster, more consistent, stable manner, including the database.” And so, we want the management, organizationally, wants to invest in that. If you have a management style that is always fighting fires and they never give you time or resources to actually do fire-prevention, you’ll never get there.

Carlos:             Right, I think it is interesting that you mentioned that you had the four groups I think you mentioned, but you only started with one. So, it’s kind of a small, you might call it a trial, you might call it whatever it is.

John:               It was proof of concept.

Carlos:             Yeah, proof of concept, there we go. “Let’s not roll this thing out to everybody. Let’s minimize the exposure.” But at the same time, hopefully then you’ll get a few people that know that they have to do it and you’ll get some champions when it does get rolled out to everybody.

John:               And that’s exactly what happened. The lead developer, he was very much involved in the process. And the team actually found that it was much easier. They didn’t have to talk to me anymore, basically. So again, it removed me as a DBA out of the process, made their jobs easier, made my job easier, once we got past the proof of concept piece. So, they would turn around and go to the other development teams and say, “this is really cool”.

Carlos:             So, let me ask this, and interestingly enough, this actually comes from an Alex Yates blog post that we’ll link to up on the show notes. But how do you get around some of the challenges, and the scenario is let’s just take a stored procedure. So, the stored procedure has to change, and you have to add a column to the output of the stored procedure. And in the background, or in the meantime, you’re also changing, and his example was he actually had to change the table name. That’s pretty drastic, I feel like, but maybe a VIEW. Maybe a VIEW that the stored procedure uses also is being changed by another group. So, I feel like it’s those kinds of things that are going to give us the most heartburn. So how do you go about attacking some of those issues?

John:               That’s always very true. There’s always going to be those edge cases where things might go haywire. In that scenario, one of the things I think is really critical is most organizations that I have experienced working with is that they have what’s called a shared development model. Meaning there’s one SQL Server, one database somewhere and everybody hits the exact same thing. And we had that in one of my previous organizations that I worked at. We had 10 AppDev teams and we had one server, one database, and I could potentially have all 10 teams trying to make changes to the exact same database all at once. You want to talk about mass chaos, dogs and cats living together. It was just like hitting your head against a brick wall, sometimes and no fault to them. But there’s another model called the dedicated model, where every developer has a local copy of a database, and they make changes locally. So that helps remove some of that conflict of multiple changing the exact same thing. The trick, though, is that really, continuous delivery forces people to talk to each other. So, if we know that we have two AppDev teams that are working on the same feature, the same database, same project, same product, whatever, and they’re making changes, in a database in source control, they have to communicate in some fashion. “Hey, I need a pull request from TFS. I made a change, I gotta test it, I’m gonna commit it. Oh, I committed this, I need to go tell Bob the developer that I made this change.” And in theory, your testing harness will help capture a lot of those conflicts.

Carlos:             In theory, yeah.

John:               Right, in theory. So that’s the other trick to this from a database perspective, rollbacks of the application code are easy. Databases, not quite so much. A little bit harder to recover from a failed deployment, a large deployment. That’s why continuous delivery should be small deployments, in my opinion.

John:               So, part of this process, and I don’t think that we, as DBAs or even data professionals do a really good job of doing any type of testing. All of us, we write store procedures, we write functions. You know, table functions, scalar functions, whatever, cursors, code, and we just throw it out there. We might take a look at an execution plan or whatever, but we probably don’t really test what changes does that really introduce into the environment as a whole. And so, there are some tools out there, tSQLt is one of them that we could use to actually evaluate and test our SQL code to make sure that we’re not breaking other stuff. You know, if you’ve had any involvement with an AppDev team, they’re almost always talking about testing coverage, unit tests, how do we test that, automated tests, UI tests. We test everything under the sun except for the database.

Carlos:             Yeah, and again, that’s one of those spaces that gives a lot of people heartburn because they feel like they’re creating more work for themselves. And sometimes when the example tests are so small, you’re like “why are we testing this?” Because you want to make it simple to understand, but again, test upon test upon test upon test and all of a sudden you have something that’s a little more complex.

John:               Right, the testing process or framework is just as important as the application code or the database code, my opinion. From a process perspective, and that’s one thing that I was really consistent on is that we have to be able to test the database code to make sure that we’re not breaking stuff. And then we, as DBAs, we have to trust to that process from start to finish, that the process is valid and vetted. It’s not something that I would just plug in, flip a switch and walk away. First couple deployments, I’m going to watch like a hawk until I’m comfortable with, “okay, we’ve released, successfully, 15 times all the way up to just before production. I clicked a few buttons, there’s a gate there that allows me to review things before I push the button to release it.” Then after that, I’m going to start trusting it and go “bye developers”. The exception is: don’t notify me when things are successful. Notify me when things fail. So now I’ve kind of switched the pendulum the other way of I’m always involved with the deployment, so I know if it’s successful or not when I push F5 and actually deploy that code, now I want to go back to “just call me when it breaks”. That’s pretty much all I want.

Carlos:             So, it is interesting that you talk about this process. Yes, there is the technology and ultimately, it’s very nice to get to that position where “yeah, just email me when it fails”. But to get there, I think it does take a lot of interaction, there’s quite a bit of culture and a little bit of work. There is no easy button, here. But in the end, just like a user using an application to allow them to do something easier, if you’re able to put in that work, then everyone wins in the end.

John:               Agreed, totally agree.

Carlos:             Very cool. So, we didn’t necessarily talk– you mentioned security earlier, but are there considerations that we need to make when it comes to security and who gets to access what, basically?

John:               Yeah, I think that’s a really interesting tangent or sub-topic of this continuous delivery. I personally think that DLM actually helps you to secure your environment even more. And the reason I say that is in most organizations, let’s say you have four environments. You have a development environment, a QA, an integration or staging environment, and then production. Usually, development is wide open. QA is probably locked down a little bit more, and then your staging integration environment is locked down even tighter, and/or as close to production-like as possible. And then hopefully your production environment is completely isolated, it’s locked down like Fort Knox, and that’s all good. This is another added benefit of doing DLM and continuous delivery is, if we have tools in place that handle the process for supplying code, and the tools do the work, nobody needs access to any environment, except to maybe read. I’ve been in organizations before where the lower environment, because everybody has SA rights, things seem to magically change. Log-ins get dropped, databases get dropped because people don’t know what they’re doing. Data gets corrupted, data gets sucked down from production because they have read-access in production that it gets put it into an unsecured location. And that’s tough to manage, but it happens, it does. But if we have the tools in place and we force consistency across all teams, including the DBAs, that the only way we’re going to get changes into the environment, all environments, is through this process, which is the way it should be, as a DBA, I can remove and say, “you know what, AppDev team? You don’t need the ability to change tables or drop databases or SA rights anymore. I’m going to give it to the tools.”

Carlos:             Yeah, so the tools, you have to promote it, put it in source control, create the release, and then the release is what will write to the database.

John:               Correct. And the other thing is that that process also forces the DBAs, we have to follow the same process.

Carlos:             Yeah, so you’re going to have to have a little self-control, there.

John:               You have to have two things, I think. And this is what I’ve told my application teams, at the most recent employer is, I have to follow the same suit in two fashions. One, if production’s down, I want to fix production, but if I have to modify a database or whatnot, it’s on me to go back and add it to source control and push it back up. If production’s down, my concern is not, “oh, I’ve got to follow this process”. While I can, and that really depends on what the process looks like in terms of do you have a lot of gate-keepers.

Carlos:             Severity of the issue, all of those kind of things, yeah.

John:               Yeah, there’s a whole slew of things. My job is to make sure production is back up and running, servicing customers, making us money. I really like to get my paycheck, so that’s my primary concern. So, if I do have to make a change to production, live, it’s on me to go back and add it to source control. And if I don’t, then I’m on the hook for it. And we know that I’m on the hook for it because hopefully I’m communicating what I had to fix with somebody. So, somebody knows that it was John that made that change in production, “oh, but it’s not in source control, guess who’s at fault?”

Carlos:             Yeah, you gotta do that. And there is some “true up” that might need to happen. Your process may need to take a peek at that and hopefully you would discover that as well.

John:               Sure, and you can do that. There’s tools out there that, you know, Redgate has that schema compare tool. You can obviously automate that to do comparison between production and a target and say, “oh, John added this to the database.”

Carlos:             Yeah, give me all the changes.

John:               Yeah, “here’s a whole slew of changes that John did that’s not in source control. Oh, that’s a problem.”

Carlos:             So, I’m curious. In that scenario, I feel like I would get– I guess I’m used to the data team pushing back, but I’d feel like I get some devs pushing back there. Was that a battle of wills to get the developers basically out of the development environment?

John:               So, there’s always pushback. When you take stuff away, there’s always concerns, I’ll put it that way, because I’ve been in that situation. So, what happens is, you’ve got this critical database that everybody’s working on, all 10 AppDev teams are working on something, and somebody makes a change at a server level that affects all 10 AppDev teams. So, if they can’t do work, what did we just do? We just ground the entire process to a halt and 10 different AppDev teams with 100 people, that’s a lot of money in terms of salary, wasted hours, until we fix it.

Carlos:             Yes, so that shared model would be much more important.

John:               It is.

Carlos:             No, I wouldn’t say much more important, but the–

John:               It’s more dangerous.

Carlos:             More dangerous, there you go.

John:               So, if we had the dedicated model and everybody’s making changes to their local laptop, and they decided to, for some unknown reason, corrupt their local instance, the only person they affected was them. One person out of 100. I can handle that. But then that one person hasn’t affected the work process of the other 99. So, having a dedicated model as well as being able to lock down all the other environments, and allowing the tools to actually drive the deployment process, to me, makes my job easier. If you can’t change anything, then I know you can’t break anything. That’s pretty common sense.

Carlos:             Or if it does get broken, it will be in the process, and I can revert.

John:               Yes. Yeah, and hopefully you have a good recovery strategy. Even with continuous delivery, we still have to have a recovery strategy of some process, and you can even bake that into the process. We did that at the previous employer. The process was that as a DBA, I wanted a rollback script. So, my development team that was part of the proof of concept actually used the tools and figured out how to do it and the continuous integration process would actually do a compare between the target and the source. What they’re pushing from, to, generate an actual rollback script.

Carlos:             Based on that, interesting.

John:               Based off that, and would actually deliver it to me. As a DBA, that actually made my job even easier, so when deployments did fail on occasion, I had an instant rollback script that I could go back and review and execute if I wanted to, or you could probably even bake that into the process itself. “Oh, the deployment failed, we’re automatically going to roll back to this script.” That’s even better, get me out of the picture.

Carlos:             That’s right, even better. That’s right. Well, awesome. John, thanks so much. I think interesting discussion. Obviously, lots more we could talk about. Lots of different areas to go down. But big takeaways at least for me, are culture, culture’s important, having that champion and then I guess in with culture, but that idea, you have to communicate. It’s not a tools-only exercise.

John:               Yep, very much true. Communication’s going to be a huge component of getting this into place. Very much so.

Carlos:             So, shall we go ahead and do SQL Family?

John:               Sure!

Carlos:             So how did you first get started with SQL Server?

John:               I am your typical accidental DBA. Many, many moons ago, I was actually working for a consulting firm in Omaha, and I was actually more of a jack of all trades. I can’t believe I’m going to say this, but I was actually a Lotus Notes admin and developer. That’s no longer on my resume. But I was also the webmaster, so I managed the website and the website changes for our organization. I worked on the internal IT staff, and I actually had a co-worker or colleague, actually is a good friend of mine, and he managed the SQL Server. And he left the organization for greener pastures, and I actually went to my boss and said, “hey, I want to take that over”. I kind of jumped into it just like everybody else, accidentally. In 2007, I actually took a job with a consulting firm as a SQL engineer. I really liked SQL. I knew that there would always be a need for somebody who could work with data moving forward. I did not want to be on the Help Desk for the rest of my life, so I made the conscious decision to, at that point, actually pursue a full-time career in SQL Server.

Carlos:             Very cool. So, all from, I guess you can thank your buddy that left.

John:               Yes, and I did, actually. I’m pretty sure I took him out to dinner or something, “thank you for leaving.”

Carlos:             Yeah, that’s funny. So, now having worked with SQL Server as long as you have, if there was one thing you could change about SQL Server, what would it be?

John:               You know, I thought about this over lunch, and this might be really comical, but get rid of NOLOCK. It drives me absolutely crazy. That is the worst nomenclature that Microsoft could have picked. We all see it everywhere, and I’ve tried to come up with something, “oh, let’s change availability groups, let’s change DR”, and I went, “no, really?” Probably the one thing that I complain about is NOLOCK. It drives me absolutely crazy.

Carlos:             So just changing the name of the function, even?

John:               Yeah, change it to DirtyReads. Change it to something other than NOLOCK, because I think if you’ve been a DBA at probably any level for a period of time, you run across other individuals, “oh, we’re going to write this massive, really important query for a report that goes to the CFO that decides our bonuses, and we’re going to use NOLOCK through the whole thing.” And then you have to have that conversation with them, “well, okay, this is what really NOLOCK means.” We’re going to have dirty reads or potentially dirty reads. It’s a financial report, let’s not have dirty reads given to the CFO with financials. Let’s make sure the data’s actually accurate. I would change NOLOCK to something else.

Carlos:             There you go. What’s the best piece of career advice you’ve ever received?

John:               That’s another excellent question. I don’t know that I’ve ever received any best piece. The one thing that I remember, back in 2007, 8, 9, 10, somewhere around there, is that you have to get out and speak and be comfortable in speaking to technical topics. I can’t stress enough that getting out to speak and blogging and getting involved in SQL Community, for sure, or any technical community, whether it’s SQL or Oracle, or MySQL or .Net or whatever. Getting involved has been completely amazing for my career.

Carlos:             Well, there you go. Our last question for you today. If you could have one superhero power, what would it be and why do you want it?

John:               That’s a really good question. And I thought about that over lunch, too, and I really didn’t have a good answer. If I had to pick one, I’d probably say the ability to fly. And part of the reason is that I grew up in a military family, my dad was Air Force. He was a pilot. I spent a lot of my time in my younger years living on an Air Force base next to the flight line. I can remember as a kid, jets are flying over the house and things rattle and whatnot. I always loved to fly. Getting on an airplane today still astounds me, even though I understand the physics of flight, flying through the air in an airplane, this enclosed metal tube, has always just fascinated me. And so, I think I would want to fly and have that experience.

Carlos:             Yeah, very cool. Well, awesome. John, thanks so much for being here and sharing with us about continuous delivery.

John:               Awesome, thanks for having me, I really appreciate it.

Carlos:             So, I mentioned in the beginning that I think there are lots of opportunities in this space, but I believe that culture plays an important role, here. So, he mentioned some of the tooling. I think, again, it is getting better, but I think if you’re going to do devops in the database, you’re going to need a complete package of skills. The technical components are only a small sliver. I think you have to add cheerleading, project management, debating skills, persistence, and a good message to be able to convey the benefits while you go through some of these challenges or hurdles, to have all of those skills to be able to deliver on this devops opportunity. Now, obviously, if you have a leader or if you have someone else who can bear some of that burden with you, I think that’s going to help, but that ability to be able to work with your team, juggle two ways, two processes during a time. I know John mentioned a little bit of that, and that’s a challenge. That’s not something easy or simple for a lot of people to do. It can be difficult. And then I think, again, through that persistence, seeing the vision and knowing that going through this process, you’ll come out better in the long run, it can be tough to stick with. I know that I’ve worked with organizations in the past, it hasn’t happened. I think there’s a lot of failure around trying to implement some of these things. You’re seeing the same thing on the code side, now trying to implement that into the database and some of the other additional complexities that are there with the database, again, add to the challenges in this space. However, the tooling is getting better. It is possible. I think there is some more conversation about it. The pains are growing bigger and bigger, and so organizations are wanting to solve this problem.

So, I’m interested in hearing from you, compañeros. I want to know, those of you who have been able to do some devops in the database, how has that gone, what’s your experience been? Again, John mentioned that it took him a little over a year to finally get some of those things in. and again, every organization is going to be a little bit different. They all have their own unique needs. So, I’m interested to hear some of your challenges, which you can leave on social media or connect with me on LinkedIn. Always interested to do that. I am @carloslchacon. I think that’s going to do it for today’s episode. Thanks again, compañeros, for tuning in, and thanks Christian for the suggestion. We’re always interested in knowing more about what you want us to be talking about. And, compañeros, I’ll see you on the SQL Trail.


Leave a Reply

Back to top