Episode 93: Making Mistakes

We all make them and sometimes it can be painful.  Making and sharing mistakes is one of the most powerful tools for learning; However, most of us do all they can to avoid and hide those mistakes.  In this episode, Steve and I talk about making mistakes and share some thoughts around how we might improve how to respond when we don’t do something quite right.

Listen to Learn

  • Two rather public mistakes and how the companies addressed them.
  • Why we are so quick to point the finger.
  • What you should do when you make a mistake.
  • Thoughts on ways to avoid mistakes

” open=”off” use_border_color=”off” border_color=”#ffffff” border_style=”solid”]

Episode 92: SSDT

At one time there were two different products named SSDT and that was confusing for lots of folks. and we’re pleased to have Kevin Cunnane, the program manager for SSDT, as our guest.  We wanted to ask him about some of the new database features in Visual Studio and what kind of problems they are looking to solve.

SSDTWe think it is awesome the Microsoft folks will take some time our there their schedules to talk with us and we think you will this episode an interesting discussion on the future of database tooling.

 

 

 Episode Quote

“If you’re doing any kind of adhoc development, Visual Studio is very complementary and it will work well.”

“Visual Studio components are much more focused on kind of the developer scenarios and light weight DBA scenarios.”

Listen to Learn

  • SQL Server Data Tools
  • The BIDS and Visual Studio components of SSDT
  • Visual Studio source control and schema compare
  • Differences of SQL Server Management Studio and Visual Studio

Kevin on Twitter
Kevin on LinkedIn
Kevin on Channel9

About Kevin Cunnane

Kevin Cunnane is a Senior Software Engineer working on SQL Server client tooling. He has been creating developer tools in the NLP space and for the last five years in the SQL Server team at Microsoft. He is also involved in client tools (mostly Visual Studio), command line tools, SQL Server Data Tools, APIs for DacFx and the Microsoft Azure SQL Database Management Portal.

 

Transcription: SSDT

Carlos: So Kevin welcome to the program.

Kevin: Thanks very much for having me on.

Carlos: Yes it’s great to able to talk with the Microsoft folks and so we appreciate you take a little bit of time to chat with us today. Why don’t you first go ahead and introduce yourself and tell us a little bit about your role at Microsoft.

Kevin: Sure. So I’ve been working for the SQL server team for about five and a half six years now. I’m an engineer working directly on the SQL server tools and over my time I worked on a bunch of different areas but I’ve kept coming back towards developer tools specifically the SQL server data tools that shipped into Visual Studio. So, well sometimes I work on other things that’s all is kind of something that keep coming back to you. And that’s what I’m hoping to be here to talk about today.

Carlos: Yes actually that is our, our topic is the SQL server data tools. Now as we get in to this and if this is something new to you, you can maybe fast forward for just a minute or two but we want to make a clarification potentially. Because at one point it was a bit confusing as to the names SQL server data tools and what they meant. So there was kind of two veins. One of which we were familiar with so there was a, the biz, the BIDS for the Business Intelligence Development Tools which have been renamed SQL server data tools. And then there was a Visual Studio components that if you wanted to do some BI things on the visual studio – there was that. So I guess help us understand what are we when we say SSDT now, what is it that we’re talking about?

Kevin: Sure so, the reasoning now behind having two SSDT brands was a little bit accidental. There was a grand plan back in 2012 to unify all of these SQL tools and business intelligence and database relational into one product. Ultimately it didn’t quite land in terms of timing so it ended up with two products. The core SSDT which was the relational projects, and SSDT BI for business intelligence which a lot of people thought of just as SSDT. So with SQL server 2016, we actually took a big step and back and actually did that work to align the two together. So what you get now is as you install Visual Studio, you still got the database project and relational tool built-in since it’s really used by everyone. But if you go to the web and you search for SSDT what you’ll get is a package containing all of the Business Intelligence tools along with the Relational Database tools in one package. So it finally gets packed to that thing up. All of the SQL tools you might want to use in Visual Studio are SSDT. So and when you install them you get the choice of which ones that you want to actually use.

Steve: So then for instance, I mean if somebody was previously using the Business Intelligence Development Studio and they were working on reports. Is that what installs now as part of Visual Studio or is that for instance part of the SSDT add-on component?

Kevin: So for all of the BI tools, they are just part of the SSDT add-on. So they’re not shipped with Visual Studio just because they are a little bit more optional. They’re heavily used but they have always shipped that outside of Visual Studio. I would note though that in Visual Studio 2017, they actually have the reporting services and analysis services project types as extensions on the Visual Studio gallery. So if you don’t want to go outside of Visual Studio and find it on the web, you can actually search in extensions and updates and look for the online updates and find it them right there. So that’s another improvement going forward to make it a little bit more lightweight and easy to get your tools.

Carlos: So in this new environment, you mentioned right the, that idea that the tools are going to join forces or they’re going to become one product. It’s caused a little bit of heartburn if you will from the data perspective which is most of our listeners come from. And they’re used to using SQL Server Management Studio and now we’re kind of being led into the Visual Studio side. I guess, thoughts on how we can, I guess as you backup a little, that pill seems to be a little bit harder to swallow than maybe originally, you know I thought I think you know we as IT people sometimes we are very stubborn in the way that we go about things with this much as things changed. So I guess I’m curious to get some of your thoughts on what can we, yeah what are we doing or how can we make that adoption a little bit easier for those who have been so traditionally used to just work in SQL Server Management Studio.

Kevin: Sure, we’ll say that SQL Server Management Studio is still hugely popular and really will continue to be the flagship of all of our tools. And if you are serious about SQL server there’s, just the odds are that you’re going to use that. That said some of the more modern practices such as DevOps continuous integration making it easy to automatically deploy, those aren’t covered in SQL server management studio so you, you do need to step outside your comfort zone a little bit if you want to truly embrace those. But there’s a nice sliding scale of where you want to go on that. If you look at the, the database projects systems we have lots of functionality for producing these deployable packages that can be used in continuous integration but you can also develop the scripts and have them ready to publish and hand off to a DBA or more data focused developer on your team. So that they can comfortably deploy using SSMS or using sqlcmd from the command line and do that in a more controlled fashion. So that’s the first sliding step is you partner with other people on your team. They hand you off the assets and you go there and then moving on towards you know getting Git integration or TSF integration into your workflow and being getting more comfortable with that. It can definitely be a little bit, little bit more of clickstop on the way. So you don’t have to jump right in, you can partner with others and find ways to do this, and there are a lot of you know third party tools and also with the new Visual Studio team services, Visual, TFS 2015 and 2017, they’re making it really easy to set up these processes so that you don’t have to be a coder or somebody deep in doing these things just to get a continuous integration or continuous deployment process going. And that you’d hopefully keep it all pretty comfortable without actually getting deep into Visual Studio coding all of those things.

Carlos: Well I think it’s more, you’re seeing more and more opportunities jobs right in the all these SAS companies or organizations that are doing development that need the DevOps environment. The data folks are going to have to suck it up in the sense and you know and get familiar with that because it does make that process just so much easier and I think of getting into, using the SQL server projects even as an example. It gives you another reference outside of just the database so as you wanted to you know move those objects like you mentioned being able to hand-off the script. If that’s even a maybe another variation of that is I can then go to source control and use that as a comparison before I actually ended up deploying those changes. So it gives me another way to review what’s happening or the changes that are going inside of the database. And ultimately for that reason I think it’s a good thing to know and to become familiar with.

Kevin: That’s for sure and there, people build up all sorts of different solutions for this. Like you mentioned just having changes being compared against different versions of your source controlled data and produce scripts from that is something that a number of more serious users of our system actually do develop really nice systems are end up because it gives them that control that they want, assurance for data folks and but also makes it very easy to rapidly apply changes. And you know one of the big things with the DevOps movement especially for data is just being able to deploy your data changes at the same time as your application has huge benefits. It really helps speed things up but application developers are really not going always going to do the right thing. You need the help from data folks to make sure that you’re designing things correctly. And so getting involved earlier on that DevOps process, reviewing the changes and code review etc can really help make the whole team more productive.

Steve: Okay, so speaking of the source control side of it, so then is it with Visual Studio or with SSDT that you get the option to be able to like commit your database in the source control.

Kevin: Sure, so it comes with Visual Studio, Visual Studio Community and up for 2015 has source control. If you just install SSDT on its own, we put in a minimal version of Visual Studio just with the bits that we need. And unfortunate for 2015 that did not include source control packages, our source, support for source control. Now with 2017 we’re working on a similar, you know like external updater that will install just the pieces we need. And we are hoping to get some basic source controls support in there so that would come back to where we were in 2013 and earlier versions. But that is a little bit to be determined but ideally if you can use either one of the Express Editions of Visual Studio 2015, which includes the SQL server data tools for relational projects or the community or up. Those all have source control.

Steve: Okay. So I haven’t used that source control through Visual Studio in the past but I have used for instance the Redgate SQL Source Control through SQL server management studio. And that was one area that you could go and just point at the database, it would scan the entire database, put the whole thing in the source control, broken out by object types, tables, stored procedures, functions, views whatnot. Is that similar? I mean is that what the source control does in the SSDT in Visual Studio side of it as well or does it do something different there?

Kevin: It’s quite similar but it is a little bit of a change from the Redgate Source Control treats the databases as the source of truth still so it manages things in source control but you keep going back to the database for reference. With database projects in Visual Studio, you, if you want to start with the database you can import the contents into a project. And that just pulls all of the files locally into your machine where you can see them just like source control organize and then you will check those into Source Control. But those files we have a build process to validate all of the things that are happening in all of the contents. So often when you first import you’ll actually find all of the things that are broken in your database. Most commonly stored procedures and views that refer to tables or columns that are no longer there and somehow they’re still working but if you ever try to deploy it to a different database those will fail. So that’s pretty common. So yeah with SSDT it’s a much more project-centric view of the world where you start with the project and then you deploy from there. That’s the key change compared to Redgate Source Control.

Carlos: And then because I’ve worked with it a little bit you know, deciding what you’re going to do with your security as well because you know you’re going to import those users and the owners of objects get imported. And so what’re the things you’re going to find out pretty quickly is do you have a good structure for who owns all those objects and in your schemas and things like that. Because as you know, as we’ve mentioned kind of moving around from environment to environment, you’re going to see issues if that’s not setup well.

Kevin: That’s correct. One thing that we have improved on is we have much better controls when deploying on what to ignore or leave be when actually publishing. So you can pull in your user’s permissions, etcetera for reference but you can actually ignore them during deployments so that they don’t interfere and if you do have different systems for different deployment environments. It’s a little bit easier to configure that now.

Steve: Okay. So then beyond the Source control side of it. I mean what are the most common things that people use in the SQL server data tools to do beyond what you would get in Management Studio or in Visual Studio?

Steve: I guess what I’m thinking there is that if you, if there’s a DBA out there who’s been using SQL Server Management Studio for a long time and they’re comfortable with it and they do most what they needed to do today with that mechanism, what are the things that you really need to use SSDT to be able to do that you can’t do in Management Studio?

Kevin: Sure. Especially for DBAs there’s a couple of hidden features that are worth the price of admission we’d like to say. Particularly Schema Compare is built-in to Visual Studio and it’s free. And that lets you compare two databases contents to each other, database to project system or a DACPAC which is the compiled version of that. And so there’s great benefits there and if you want to be able to see hey what’s the difference between staging and production in terms of my database contents. That will give you that view and make it super easy to understand that. Similarly we have data compare for if you do want to compare for example hey what’s the reference data looking like in these two tables that are in different databases. And making sure they’re up to date or identifying issues. Those two are really huge features that a DBA might not be aware of but give then a massive amount of benefit.

Steve: Okay, excellent. I know those are two pieces that I have in the past looked at like third party add-ons trying to achieve and if that’s something, I mean I know the Schema Compare has been in there with the Visual Studio but didn’t realize that Data Compare was in there as well. Nice.

Kevin: Yeah. So both of those are there, I mean there’s a lot of the other functionality is around lightweight you know project system development but those two are useful for everybody. And schema compare is actually now supported in our command line tool which is called SQLPackage that lets you deploy all of your database project information up to a database where you can also do a number of other actions. And one of those is you can get a saved schema compare file that you’ve developed in Visual Studio and run it through this process or indeed using APIs you can actually build up entire workflows on top of the schema compare engine since we now just have a very simple API for it. So if you want to do some very simple logic and if you’re comfortable with a little bit of C# coding, very little that kind that you would do in a Power Shell script, you can do amazing things with that.

Steve: Very nice. So with that I mean if someone want to just build a script that would tell them what’s different between two servers and two SQL servers and it would run daily and send out an email. I mean is that the kind of thing you’re talking about or it’ll be really easy to do with that SQLPackage and a little bit of scripting?

Kevin: Exactly. You can do it directly with SQLPackage where you just send that straight to report and pre-configure what type of things to exclude; or if you needed to be more dynamic and respond to the result before formatting it for a result. That’s where you might write a little script that iterate it over the results and did it in a more controlled fashion.

Carlos: Yeah, and this an area that I think for still, even though it’s been in the couple of versions of Visual Studio is well, still appears to not be getting a lot of love and you know as we go to different conferences or talk with others that are using the projects. You ask like who’s using it and it’s kind of crickets. Now maybe I’m not in the right circles admittedly but I think there is a lot of value to moving to the SQL server projects and you know integrating with the source control you know components. And then of course you know you talked about the data, the data comparison components. Now one of the things, I guess maybe, I don’t know for some of the developer side but what gets included with Visual Studio now is actually a LocalDB copy. Do we need to be concerned about database sprawl as we look to be using Visual Studio?

Kevin: That’s a good question. The nice thing with LocalDB is that when you’re not using it for a long time it will stop running. It spins down and just keeps a very lightweight listener that will let it spin off again. So it’s not going to be using resources on your machine, that’s for sure. In terms of database sprawl and all of the databases you’re going to connect to there, it’s only if you got a local project develop, in development that will create a temporary database under LocalDB for you.

Steve: So then with the LocalDB there is that intended for like desktop applications that are going to be deployed and they would just use a local database or is it really more intended just for development purposes to.

Kevin: It can be used for either. We have people building, using LocalDB for like that desktop applications lightweight as a replacement for SQL CE or other lightweight database options. So that’s one of the scenarios for it. The reason it’s shipped inside Visual Studio is it really helps with your local development and debug loop. So every time you make a minor change in a project in SSDT, if you had it will deploy it up to LocalDB. So it can verify the deployment works, you can you know edit the data in there and compare it, verify everything works. And you can even run unit test against it to make sure that everything’s working okay. So that’s one of the most common scenarios. Similarly, for people who don’t want to use database projects, the web application development inside Visual Studio uses LocalDB too. It uses that as like a reference database. And when it’s publishing it gets a snapshot of that using our technology and publishes it up. So there are a lot of different ways to develop for SQL inside Visual Studio and it just depends on your workflow. Those web apps tend to be very simple it gets you bootstrap, you’re not worrying about a DBA for those and then often if you scale them up, that’s when you might want to move over to a dedicated SQL project or set a migration scripts which are much more controlled approach to go in through things either of those. But for rapid development LocalDB is great for these kind of scenarios.

Steve: I mean this sort of interesting is, sounds like again kind of that rapid development and I can’t help but think of kind of the open source maybe mentality or like, “Hey, I want to try something out and see that it’s working.” I know that again on a database side SQL server has made some changes to I’ll say I use the word compete with the MySQL because it is open source right from the data science perspective, analytics components. MySQL was being adopted much faster than SQL servers then they made some changes here. I almost feel like there’s a similar component there to Visual Studio’s, you know let’s put the tools in here so that people can quickly kind of see this is what they want to get bigger and then they can use something, something else that they need. Is that a fair,assessment there?

Kevin: Yeah I think that’s fair. And we are also by the way working to be more transparent and open source in our tools in general primarily with our new multi OS tools. So we have a visual studio code extension where it’s being develop fully in the open with SSMS and SSDT people have actually asked us about can we just open source this and they have pretty complex build system there’s some legacy stuff in there that would be hard to make that possible but we are trying to embrace that in any way we can in terms of extensibility points, making it easy for you to contribute, making it easier to raise issues and like that as well, just making it easier to adopt whatever tool that you use to deploy we want to be there and make it easy for you to develop SQL server.

Carlos: Now, you’ve mentioned you said before right SQL Server Management Studio is not going away. It will be there for those who want to continue to use it, one of the things that we also know I noticed in the Visual Studio component is that the SQL Server Object Explorer is now in Visual Studio. So I guess to reverse the roles a little bit, maybe let’s say I’m new, I’m a new DBA, I’m kind of coming up could Visual Studio be the tool that I learn in and grow in to manage my database?

Kevin: I think being honest it would be a little hard. We’ve had people asking why don’t you merge this two? But there’s a huge amount of functionality in SQL Server Management Studio around backing up, around different types of profiling, etc., There’s so many different piece of functionality there that you might need to use as a DBA that I think if you are actively managing databases, SQL Server Management Studio is still the tool that you should use. But if you’re doing any kind of adhoc development or want to explore that kind of side of things Visual Studio is very complementary and it will work well.

Carlos: It sounds like a tool, so the wizarding components that it gives you that’s what’s going to continue to be in SQL Server Management Studio, from the ability to get to the database and kind of specific to T-SQL statements and things like that, maybe there’s more options available.

Kevin: That’s correct, it’s just a little bit richer in that, the Visual Studio components are much more focused on kind of the developer scenarios they have all the common things that you might need to use if you’re a developer or a light weight DBA but once you get into managing, dozens or hundreds of databases and doing them at scale, there is just certain things that SQL Server Management Studio will do that I think people are going to want to install it and use it.

Steve: Sounds like there’s a hand full of different versions out there than with a SSDT and Visual Studio 2017, 2015, 2016 which ever.

Kevin: 2015 for that one, 2013, 2012, 2010 we’ve been in them all. Yeah, that’s correct. So one of the things that it is important to note is that all we actively develop on and ship into are the most recent two versions of Visual Studio, so that means that what’s Visual Studio 2017 at, we’re going to keep shipping updates into that with new features similarly Visual Studio 2015 will keep shipping features into that until there’s a new version of Visual Studio and for Visual Studio 2013 and older we will do security patching and maintenance and all of those things if needed but it won’t get any new features. So that means in particular if you want to use the latest versions of SQL Server, SQL Server 2016 or above and SQL Server VNext which our latest RCs are letting you use or if you use Azure SQL DB which tends to keep iterating and moving on and moving forward you should be using one of Visual Studio 2015 or Visual Studio 2017.

Steve: Okay, I know that’s an area you’ve mentioned the Azure databases, that scenario that I ran into using the Azure parallel data warehousing where I needed to get a specific version of the Visual Studio that was different than what I was normally using to be able to use the tools to connect to that location.

Kevin: That’s correct, so for that, if you want to use Azure SQL Data Warehouse you should be using Visual Studio 2015 or above and you should just update the SQL server data tools to the latest version. We’ve had support for quite a number of versions now but obviously with visual studio, you know 2015 came out 2 years ago, SQL Data Warehouse came in 2016, so naturally you just need to update to get support for that. And one other thing to just call on this is that with the SSDT for VS 2015 release and onwards, we do now have backwards compatibility support for previous SQL Server versions not just in the database projects where they always were but the all of BI project types, so the database project type analysis services and reporting services all have support officially for SQL Server 2008 and up and for integration services it’s SQL server 2012 and up, so if you do want to use this with older versions in the past you needed to be tied to a specific Visual Studio version, now our recommendation is just get the latest and will support your SQL Server.

Steve: For people who need to work on multiple versions of SQL Server that’s certainly a big win there, I know for instance working with SSRS a few times I’ve had to have several different installs to be able to work on different SQL Servers at different client sites.

Kevin: Well one thing that I like to bring up is ignore column order support which is a major feature people have asked for and just to get advertising out there that this is now in SQL Server Data Tools, if that’s cool? Great I’d like to talk about that briefly, we added in our most recent version of our SSDT or release candidate support for ignore column order so people have been asking for a long time and a little background is that if your deploying from a database project we try to be very accurate about making what’s in your project system be what’s in the database. The challenge with that is that people often add in your columns in the list kind of alphabetically or they just add one in the middle which they don’t realize will cause a data movement operation because you can’t do that via T-SQL syntax, you need to always add to the end. So we finally brought back support for ignoring column order when deploying which means that if all the columns are there it doesn’t matter what order they are in we will leave them be and we’ll add new columns to the end rather than doing a much longer running operation. So we’d encourage people to try the release candidate and we do have the full version will be coming to Visual Studio 2015 as a GA within the next month or two so keep your eye out for that it will go the regular update channels.

Carlos: Okay and that makes me feel a little bit better, so I think previously the 2015 version there was this option in the project which made my heart stop a little bit where it would go through the process of creating a new table, moving the data over, dropping the old table and all the constraints and everything that was needed. And I was like, “Ooshh”.

Kevin: Yeah, and just to point out we do have so, one thing that it’s not built in to detect those kind of changes and warn you about it. We do have people who built that though on top of our extensibilities, so if you do want to do it, it’s there but it helps as well as we say trust but verify where before deploying to production in particular just make sure that somebody does review the deployment script. And if you see those kind of things it’s unlikely to be what you intended so go back, check this option to ignore column order or redesign it and realize that this is a breaking change. So we have extensibility where you can actually do that programmatically but that’s a little bit more cumbersome than some automatic processes, so just people can just build those on top.

Steve: So then if you’re doing the ignore column order rather than the old process of renaming the table and then creating a new table and throwing it up and swapping them around, it’s just going to drop the new column at the end of the table if you add one?

Kevin: Correct, it will just do an alter table add column and your good to go.

Steve: Very nice, that will be much faster, assuming that it support the applications using it are fine with that layout.

Carlos: So I did want to circle back around for a moment because you have talked about the different versions that got pushed out even we talked about 2012 or 2010, 2012, 2013, 2015 and so there’s which is great and everybody is excited to see the “New Microsoft” and the speed of what you guys moving and its mind boggling sometimes. But that does beg the question a little bit you mentioned that the new features get into the last two versions of visual studio. It can go pretty quickly there, I guess thoughts on how or the organization would attack the idea of keeping up with the newest versions?

Kevin: That’s a good question, well one of the things that’s been happening which will hopefully make it a little easier is that with Visual Studio 2017 in particular, the installation has become a lot more light weight and installs a lot less global components, so you’re less likely to run into problems if you do try out the new version while still having the older version on your machine. What we suggest is even with this you’re still getting about a 3, 4 year process were you need to update at some point during that if you’re going up to the lasts SQL Server version. When it comes to build processes etc, one thing is that we now have a new get package with all of our MS build components to make is easier to have kind of unknown version when you’re doing your proper CICD so that it builds standard version that can avoid some of the trouble there that you get into with these where at least your build process will always be consistent no matters who’s developing on which version. The other one is that probably what we’re seeing is that usually because it’s backwards compatible you can have a handful of people in the team try at 2017 even if you’re still in 2013. Stuff still works certainly on the database project side. I think some of the other projects types you may need to be prepared to do a one way upgrade so it needs to be more blunt. For the database project type you should in the vast majority of cases be fine trying it out getting maybe a mini team to do it on a certain project and then adopting it once you realized there are no issues. And also, a lot of things have gone through more of a subscription based models so if you have Visual Studio 2013 it should be pretty easy to update without any pain straight up to 2017 and use it there.

Carlos: Got you. Well, awesome. Kevin, thanks so much for being with us today. Any last thoughts before we move to SQL Family?

Kevin: You know, I’d encourage people if you haven’t tried it already, try out the RC. I mean it has support for SQL Server vNext and one of the most fun parts of that is that we now have docker support for containers so you can install SQL vNext in a few seconds without it squashing anything on your machine which is the best part of it. So just try it out there. You’ve got no risk of damaging anything and pretty exciting to do that and give us feedback on it. We’re going towards GA so we’re focusing just on the quality of the tools that will support this even though obviously it will be a while yet still SQL vNext is released. We want to be there with the tools ahead of time. So everybody please write that out. Please write Visual Studio 2017 and the tools in there as well. And we’re looking forward to hearing you feedback on it.

Steve: Well, let’s move on to SQL Family then. So Kevin, how did you get started with SQL Server?

Kevin: That’s a great question. So I have been a tools guy for a while. Before I join Microsoft I’ve been working in different company developing tools as part of a larger set of things but I really enjoy that process of kind of getting the complex requirements on deep things. And when I came to join Microsoft I had a few different areas. But this one really descended by far the most interesting. So I joined and I knew nothing about SQL. I barely done any database development so it was a little bit of a trash course learning all about it from the schema point of view. And luckily on the team that I worked on, you have to learn all breadth and depth of how you can configure SQL server in terms of schema changes. That was really fun and I just love that process and I really like that kind of people we worked with in the user base. So that’s been very rewarding for me.

Carlos: Now, if you could change one thing about SQL Server what would it be?

Kevin: That’s a good question. From my side I wish it was easier sometimes to add new features into the tools that support the latest features, so we have some really amazing things that makes stuff super simple for people such as temporal tables. Unfortunately it can take a lot of effort to make the tool support that so people don’t realize that’s why sometimes we don’t always have as many new features in the tools as we’d like because we’re supporting these new SQL Server versions. And with SQL Server 2016 in particular, there were just so many features which people should try at which we supported, OS encrypted, temporal, just a huge number. So I wish it was a little cheaper to do that just so that we could add more value for people. But on the plus side people get to try out of all these new things and it should all work pretty smoothly for them. So I do upgrade and do try these things because they’re pretty awesome.

Carlos: Very cool.

Steve: What is the best piece of career advice that you’ve received?

Kevin: That one has, I mean, my first manager here, his best piece of life advice was buy a house because the market will be crazy and I regret not following that one. In terms of career of advice, I think the biggest one really was don’t be afraid to just on quality and always good feedback in terms of code reviews, and that’s just have been a huge benefit whenever I’ve done it. Just always be willing to ship a week later but have it be good and I think that’s done for pretty much everything.

Steve: Ok.

Carlos: Especially for somebody who is working in the tools, I’m sure you can appreciate that as well.

Kevin: Yeah, you know, we don’t like to do hot fixes. The good thing is when we have to do them occasionally and we can turn things around in a day now which is amazing. So we have shipped and gone, “Oopps”, something is wrong and by the next morning we have a new version that’s been patched which is great but we really don’t like doing that. It hurts our customers and it hurts us.

Carlos: Sure. Last question for you Kevin is, if you could have one superhero power, what would it be and why do you want it?

Kevin: I think I would like to have the power to turn back time sometimes, and just redo those decisions we’re like, “Oh, this is definitely what we need to do and going to it.” Specifically sometimes we get a very principled on we must support a certain scenario and then we spend weeks trying to do it and then realizing, “Ok, we could have done this much quicker at different way or avoided it all doing together.” I think gotten more than in my career if I could turn back time and change those decisions a little bit.

Steve: You could even turn back time and have bought a house when it was recommended by your manager.

Kevin: Yeah, I should have listened to him. He was a very smart manager and that’s the piece of advice for everyone if you could back to 2011, that would be good time to do.

Steve: Alright.

Carlos: Well, Kevin, thank you so much for being on the show with us today.

Kevin: It has been a real pleasure. Thanks for having me on.                                         

Steve: Yeah. Thanks a lot, Kevin.

Episode 91: DBA Tools

We all want an easy button.  It is human nature.  We heard quite a bit about how easy PowerShell will make everything, but for those of us who aren’t programmers, it can be a bit intimidating to get started.  Luckily for us, we have what is shaping up to be the closest thing to an easy button for DBAs I have found for PowerShell tools from the folks at dbatools.io.  We invite some of the team to chat with us about their tool, how they got started and the types of problems they are looking to solve.

DBA ToolsIn this episode, Chrissy will be talking about how DBA Tools started from just being a simple migration tool until it became a useful tool solving various SQL related issues.  The team will discuss the other various situations where DBA Tools can really be helpful and how they manage contributions from over 30 people which leads us to talk about GitHub, the open source community, and their experiences in updating versions of DBA Tools and how they encourage contributors in the project.

Chrissy, Rob, Constantine, and Aaron were super excited to talk with us and we loved their energy and think you will enjoy this episode.

 Episode Quote

“The features that are now inside of DBA tools, honestly, I would describe them as really awesome.” – Constantine
“I promised you this is the best code ever used and that you will ever have.” – Aaron
“It is important to us that people do feel welcomed and that their codes gets merged in.” – Chrissy

Listen to Learn

  • The tools the dbatools team has put together
  • How the team goes about deciding what gets into the code
  • Examples of how this makes PowerShell more accessible
  • The challenges of having a community tool
Chrissy LeMaire

on Twitter
LinkedIn
GitHub
MVP Profile
Blog

Rob Sewell

on Twitter
LinkedIn
GitHub
SQL Server Central

Constantine Kokkinos

Twitter
LinkedIn
Blog

Aaron Nelson

LinkedIn
Twitter
GitHub
Channel 9
blog

About Chrissy LeMaire

Chrissy describes herself as a Cajun living in Belgium and a SQL Server DBA and PowerShell MVP with nearly 20 years of experience.  She is also the colead for the SQL PASS PowerShell Virtual Chapter and lead for the Belgian PowerShell User Group.  A fan of Linux and Open Source since I was first introduced them back in the 90’s, right around the time I moved to California to work in tech and has worked with PowerShell since 2005.

Transcription: DBA Tools

Carlos: Awesome, well today we are in for a real treat. We have the DBA tools team or “the brain child” of that project here in the studio with us today. So, we want to say, “Hello to you all!” This is probably the biggest group we ever had on the podcast so we are excited about that.

Chrissy: You are welcome.

Carlos: The more the merrier and we are going to prove that today. And so I guess I’ll kick it off with you Chrissy. Why don’t you tell us a little about these projects, how it got started and then we can kind of go around and well go from there.

Chrissy: Alright, so DBA tools actually didn’t start out how it is now. Initially way back in 2014 I created a bunch of migration commands and I made a little project on Github called SQL Migration, and then I started adding a couple of things like get SQL Server key or dug into the registry and it grab that information, and then I just kept adding more and overtime I realized that it can be more than just a migration tool so I called it DBA tools and then I just started marketing it that way as a general Power Shell tool for DBA’s and it really took off from there.

Carlos: All of you, maybe Rob you want to jump in? When you got involved with the project were you looking at it from a migration prospective?

Rob: Absolutely not. I’ve always been a SQL and Power Shell person so Chrissy and I got together and created this SQL collaborative Github collection which is where DBA tools lives on Github and obviously I knew her from partial conference and from SQL conferences and so we got together and start working from there on.

Steve: So Rob, how long after the original work that Chrissy had done was it that you got involve with the project?

Rob: So Chrissy, when did it change from being migrations to being DBA tools? I’m not sure about the exact date of that.

Chrissy: So the official, I think that I saw its birthday was sometime around September 2015 as when I changed the name of the module and then really everything in April of 2016 I registered the domain. In doing that I wanted to make it more approachable for DBA’s and for non developers because Github is not the place for people who are kind of apprehensive about Power Shell and so from there we started Slack. And I would have to say that that was about the time that it really started to take off whenever we started the SQL community Slack channel. So the first command that was introduce by the community was Claudio Silvia he is from Portugal and he gave us expense SQL T Log responsibly and then the command after that was whenever Rob contributed remove SQL database safely and that was around April or May of 2016.

Carlos: Ok. Well, I think it would be worth kind of noting here because Rob kind of mentioned that he was a Power Shell person so really quickly let’s go through and kind of talk about what were your expertise, how kind of you got involve with the project and kind of where you were coming from? So Constantine, you actually want to start it for us?

Constantine: Yeah sure actually I had almost no Power Shell experience. I came from a DBA’s stand point and I work as a production DBA. And honestly, the reason why I saw the kind of interest in DBA tools and why I was interested in it was because I didn’t have that experience and the migration videos, the intros that Chrissy had put together were so good in such a small package that I said “I need to learn this!” This absolutely calls me to learn Power Shell more than anything I have ever before really. So I had a lot of T-SQL experience. I had programmed silly things in pure SQL and I saw this and went, “I want to throw that all on the trash and replace it with this.”

Carlos: Ok that’s interesting. So you’ve talked about migration and kind of how it got started. You still feel like this is primarily a migration tool?

Constantine: I don’t actually and I think that’s part of why it is increasing adoption is because you said earlier with the whole, “Thanks but no thanks.” In reality for a large portion of the DBA tools existence, Chrissy has been extremely accepting of what those DBA tools do. We’ve had a lot of conversations about what are we really trying to solve here. What problems are we solving? Do we go beyond that? Where does that lay? I think one of the benefits of DBA tools, as we’ve said as long as it‘s really is solving a DBA problem it probably belongs in DBA tools and that why a lot of people who don’t have all of the experience to contribute their individual pieces and make it better.

Rob: Is that anything that will help make DBA life easier and that isn’t something like to be provided by Microsoft through the partial SQL Server module. So we won’t build at database user or these who generic commands will do more complex useful things in that.

Carlos: Ok. How would you describe some of the features that are now inside of DBA tools then?

Constantine: Well honestly, I would describe them as really awesome. I mean, that’s a very basic thing. But the reality is, one of the things we started to deal with and kind of struggle with but also find good solutions for, is the idea that we have a ton of Power Shell commands here doing a ton of different things. And so some of our big contributors and Slack have actually put together some in browser things and also some Power Shell base help things if you were more into that to help you narrow down on keywords or related pieces. You know Power Shell has some pieces to help you with that but again we are not really targeting the heavy amazing Power Shell users. We’re targeting people who are just starting and who want to build something great and their environment that not always someone who’s going to know how to use the help command even.

Chrissy: Absolutely

Carlos: I think you’ve described to me perfectly. What are the reasons I was so interested on having you guys on and talking a little bit about this. So if someone doesn’t need to know too much about Power Shell obviously, you said, SQL Server they can download your stuff. How do we get it started?

Chrissy: Ok, I’ll jump and answer this one. If you have a dbatools.io there’s going to be a download section that you’ll see at the top and what I did was I created the site thinking of my own favorite open source projects. So you have Notepad++ for example, when you go there what do you want to see? You want to see download, so I put the download there and I also didn’t want to burden people. What I like about Notepad++ is that I’ll just download a zip or an msi. I don’t want to mess with anything and so we have like 4 different methods for downloading DBA tools. The first one is like really direct. We have it in the Power Shell gallery for systems that do support the Power Shell gallery so just be install-module.dbatools. But if you don’t have a modern system then you can also just copy and paste a single command and as long as you have either SQL Server Management Studio 2008 and above installed or the SQL Management Objects (SMO), it’s just going to download, put it where it needs to go, extract it into the proper directory and then you’ll have it. So it’s a really straightforward download and that enables you to have over a 150 commands with a single command.

Carlos: Very cool and just to clarify because I’m not familiar with the Power Shell gallery this must require additional setup before you could use that route?

Chrissy: So the Power Shell gallery its built into Windows 10 and it allows you to kind  of like if you are familiar with Linux and apt-get and all of their package managers. This is Power Shells package manager and there’s a gallery where a whole bunch of software publishers have placed all of their stuff and it makes it easy to download. It’s kind of like a command line windows store so instead of being in the Windows store we’re in the Power Shell gallery. And if you have older systems then we have links on our website that you can just click to and you install a couple of things and then it will work that way. And the install is something that is really important to is because it is migration and we are expecting this to work on legacy systems. And so it works on Power Shell version 3 back all the way to Windows 7, so it works all the way you know from Windows 7 and above. And anytime that we have the next that comes out we test our commands on that and fortunately Microsoft has been really standardize in a way that they support SMO that our commands really just works with the next version of SQL Server.

Carlos: Right I think to their credit as much grief as we sometimes give to Microsoft that whole idea of backwards compatibility, it really does work nicely

Chrissy: I loved it! Totally!

Steve: So, on those legacy systems, one of the things you mentioned was the SQL Server Management Studio and the SMO objects, with that if you don’t have the right ones there and you trying to do the install is it going to tell you that? Or are you going to end up in a bad situation?

Chrissy: You know I actually, I can’t remember why I decided on SQL Server Management Studio 2008. I might have tried it with 2005 but basically it just going to throw the same errors that you have whenever you try to use SQL Server Management Studio because really DBA tools is kind of like SQL Server Management Studio at a command line.

Steve: Ok got it.

Carlos: And if you’re starting using 2005, do yourself a favor. Upgrade!

Chrissy: But you know which really awesome is that a lot of times people may not want to upgrade because they’re like, “Oh my God, this migration process is so daunting and I have this hundred point checklist.” And then if you just use Power Shell if it is possible in your environment then it really simplifies that entire process.

Rob: It certainly does and the important part of all of that is that that’s only in the client machine. The machine that’s running tools but you can have older versions as your servers but the client machine the one that you look onto to do your DBA work that’s one that needs to have the right version partial and the right version of Management Studio.

Chrissy: Absolutely. I actually did a test, if you go to dbatools.io/install, there’s a video there where I migrated, I backup a bunch of databases on Windows 2000 server that cannot have Power Shell installed on there and that was really to emphasize what Rob was saying which is Power Shell doesn’t even need to exist on the server. What’s really important is the client so that makes it a lot more simplified.

Constantine: Yeah, that’s more basically just running queries. It’s just abstracting a lot of SQL.

Chrissy: Yep

Rob: And if you look, if you run Profiler against the SQL Server when you are using Power Shell SMO to connect to it, all you see is T-SQL. It’s not doing anything magic.

Steve: So that video that you mentioned about migrating from SQL Server 2000, I watched that and I was just amaze on how easy it was to do, or how easy you made it look to run through all of that. You just run in the command and it move things over. So if someone’s out there and they are stuck with a SQL Server 2000 box and they want to move to something newer. How big of a jump can you make with that using the DBA tools to do that migration?

Chrissy: By default, we kind of go as max as possible I have a lab that setup that has 15, maybe 20, different kind of instances. So what they would have to do first, Microsoft does not even allow the path from 2000 to anything above 2008. So if you want to go 2016, that’s actually would be super simple. You would first migrate to 2008 and then migrate to 2016 from there, and of course you have to test your applications and everything like that. But we do make it possible in that way.

Steve: And have you seen that type of a jump, from 2000 to 2008 then to 2016? Have you seen that go off smoothly? Or do people usually have a lot of code migrations they have to deal with?

Chrissy: I would have to imagine that there would probably be a lot of code migrations. Unfortunately, I don’t get a ton of direct feedback about those legacy systems. It’s mostly just like, “Hey, thanks! That was super awesome.” But there wasn’t details like, “Uh, but we really had issues in our applications side”, because really no matter how you do it we kind of just automate the way that you’re going to do it anyway. You’re going to make a backup and then you’re going to restore it and then you’re going to make a back up and then you’re going to restore it. It’s just kind of makes that a lot more straightforward. Also what I really like is that we do introduce some protections. So, you know, if Microsoft doesn’t really recommend you do detach and attach from 2000 to 2008 then you have to use dash force command to say, “No, that’s really what I want to do”, even though Microsoft doesn’t really want me to do it.

Constantine: Yeah, and then a lot of cases there are pieces completely missing from database roles or server-level roles in SQL 2000 compared to SQL 2016. And so most of the times we do best-effort, what the equivalents are or we need to prompt the user and basically say we had an inflection point where you need to make a decision.

Carlos: Right, okay. So part of the process is as I watched one of those videos and go in through it. I just want to go try it out myself here but was the, you do the migration from like your 2012 to the vNext video I think I watched. And it just showed you run it, it does the backup, it does the restore, it moves the users, moves the jobs, moves the database mail configuration. But in doing that, one of the things that I was thinking through there is, in the demo of course it’s smaller databases and it goes really fast. But if you’re in the position where you have larger databases that are going to take a while to copy, are there any options or is there anything that would like do some transaction log backups and get it caught up to a point in time? Or is there an option to?

Chrissy: I loved that question. We’re working on it and that was something that I actually thought that I had to have for the 1.0 update. I was like I don’t even want to go 1.0 until we can go really Enterprise. And we actually decide that we had so many commands in this really awesome framework that’s going to come in 1.1 or 1.2. We are currently testing a mirror. I think it’s like invoke DBA Database Mirroring. You know, to make it easy to, I’m sorry it was the log shipping, to make it easier to do that, that is something that is on the agenda but currently we don’t support.

Steve: Okay. I think, well even without that it’s incredibly valuable.

Chrissy: Right because what you can also do is just, is do the -norecovery and that’ll really help simplify your process as well. And while we’re talking about this I did test instance sure. I have personally migrated 500GB databases but people have reported back that they’ve had 2TB databases that have worked with no issue so we do go really as high as possible.

Steve: Okay. Great I look forward to trying that out.

Carlos: So you’ve talked quite a bit about the migration piece and I guess maybe, I don’t know, Aaron if you’re still there we’ll invite you to join the conversation here. Give me an example of how you’re using the DBA tools aside from the migration component.

Aaron: So one of the things I loved about the DBA tools that I was able to get in just under the deadline for the cutoff for 1.0 was actually stole some code out of SMMS and it’s the code that goes in, if you right click on a database that do the report and you want to expand all option there to show off their were any auto growth or auto shrink events. I stole the code out there to find the auto growth events and the reason I did that was it’s great information but I want to be able to see the entire instance all at once. So what I did was just wrote a quick Power Shell function, and you know, started up my pull request to the DBA tools project, got it merged in and now everyone’s able to get this kind of information across the entire instance with a single command. And that only took me about an hour and a half of my time you know from start to finish. So if you want to take something that I needed, that I thought was useful for lots of people, and you know, add it to the project and now thousands upon thousands of people can use it.

Chrissy: Aaron brings up a really important part that I really love. My favorite thing about DBA tools, there’s a couple things. The first thing is that we see an awesome blog post, “Oh yeah that person make such a good point” and we take that T-SQL and we stick it into a command and now just across the board everyone can use it as soon as they update DBA tools. And the other thing is I, what Constantine had talked about earlier is that you know here’s a bunch of DBAs who have issues or problems and they solved it using Power Shell and then they can put it in DBA tools and now every DBA across the world can easily access the thing that they did. And that was something that Aaron shared with us so he had an issue, he solved it, and then he contributed it to the project.

Aaron: Well, other things I loved about DBA tools is when the SSRS team was putting out Power Shell commands for SSRS. They were doing it just kind of like script-based. And the fact that Chrissy going to all this effort and put together this public project gave us the standing to be able to say, “Hey look, we don’t think you’re doing it right and we think you should do it more like this.” And then they had a look at what Chrissy was recommending. And it took them a, you know day or two of deliberating but they finally came back and said, “You know, yep, that sounds right.” We’re going to do it like you suggest. And I had an article published in and SQL tips just last week on how to deploy the SQL Server, SQL performance dashboard from the Tiger team using only Power Shell. It will download the reports for you with the module, download the module for the SSRS commands, download the SQL file, deploy the reports, deploy the SQL file that you need to be able to get all these information on this performance matrix. And it’ll run across your entire Enterprise for you. One script then it does it all, and very excited to make it that much easier for people to use this free tools. Like the Tiger team spent this time getting it out there so that people could use it. And then we’ve just share, greatly upped the number of people who were able to download and used it. They actually told me the number I don’t think I’m supposed to share it but it’s a big number.

Carlos: So this is kind of where again you know my mind, the knuckle-dragging Neanderthal that I am. You know, I hear, “Power Shell bla bla bla. Oh one script it will do all of these.” “Oh okay. Now, all of a sudden I’m very very interested.” And you know, ultimately for me it’s lot about that value right instead of having to do these ten things right that I have to do before, yeah ability to have one script, one way to do it and execute it that, that’s really cool.

Constantine: Sure, so yeah what I basically wanted to bring up was talking a little bit more about that value because, and that reusability, because I have you know run many scripts within the community within the SQL Server community there’s the blitz scripts, there’s all these scripts, there’s all sort of pieces that are reproducible and reusable but there’s still a lot of code going on. And there’s always the question of what is that code quality? And so a lot of people use some sort of social currency to determine, “Oh everyone’s says all these scripts are good I can use that. It won’t destroy my environment. But I’m not going to necessarily review every line of code.” And when you see a project like this moving your old SQL server, it is really important to have that kind of currency because if everyone is doing it the same way, if everyone’s using that option SMMS, you can too. It is okay. And yeah that’s really why one of the big reasons I like DBA tool become popular is because we can kind of encode the best practices for the community and if they don’t think those are the best they can come back and help us fix them.

Chrissy: Absolutely.

Steve: So then with that process if someone decides to jump in and help you fix it and then they do the work, they do a pull request, do most of those type of things make it in to the project? Or do you end up weeding out a lot of those along the way based off of the code that you process?

Chirssy: It is very important to our team that people feel welcomed to DBA tools so if you go to dbatools.io/team it actually says “The team is me. The team is you. We really encourage pull requests.” So if somebody has, if they submit a pull request that doesn’t quite meet our standards what we will do is kindly mentor them and say and you know, I really emphasize like hey be nice to new people. And people, you know the team really jumps in and they are super nice and they’re like hey you could’ve done it this way. What about thinking about it from this perspective and you can go through any of our pull request and see that. So in addition to them being able to you know like submit the pull request they also get to learn along the way. And Constantine, it was so amazing working with them because like he had said, he hadn’t had a lot of experience with Power Shell and I did his code review for him. And he said that in one code review he had learned about two years’ worth of Power Shell knowledge.

Constantine: That’s not a joke either. I had spent you know, not directed effort but 2 years picking up random pieces of Power Shell and putting them into production even in some cases, and then sat down and Chrissy basically, in the most polite way, set me straight. And said “Here’s things you want to do and here’s why.” And I spent 2 weeks just reading the code she had given me versus the code I had submitted. And learned so much, it’s why I’m here today.

Carlos: Wow, that’s really spectacular because that’s not always the norm in the open source community or even in GitHub. And I think that the way, the way you’re doing that is spectacular.

Aaron: I promised you this is the best code ever used and that you will ever have. You come out and feeling good even though somebody like of ripped your code to bits, it would’ve been done in a way that says “Okay, that’s not how we do things we’ve learned along the way that there’s a better way of doing things or this is more perform well if you look at this. And we put examples of those up on, on the website as well but we’re very much about team and then community and family. It’s supposed to be fun. We’re doing this in our spare time so you don’t want to feel like you’re under a boss who is glaring at you with all this highness. It’s something that we pride ourselves on.

Chrissy: Yeah it’s very important that people feel welcome then any time that we have to close a pull request, if I had to guess, I think that we’re up to over 500 pull requests. And maybe 5 at most have been closed without being merged. That’s the part that hurts and it does you know like “Hey we actually have this command. It’s this one”. And so it is important to us that people do feel welcomed and that their codes gets merged in. We actually, anytime that we make a release, you’ll see it we added 4 new contributors to the repository. We want people to understand GitHub that we want people to understand Power Shell and we want them to feel like a part of the community and to not be you know a just because they don’t necessarily know Power Shell right away.

Chrissy: Oh nice. Very nice.

Aaron: And part of that as a team. Yeah we’ve got some people who know good Power Shell and we’ve got some amazing T-SQL and DBA people. But we also need people who can write documentation. We need people who can help with testing and continuous integration and all those other parts of it. So we we’re not just about writing some code, we’re about writing the good code for you. We were making sure that when it goes out there this is good as it can possibly be or from a community in our spare time project.

Steve: So as you mentioned the community and the spare time side of it. One thing that I see with the lot of projects like that is that they tend to wander a bit. But I’m curious with the DBA tools. Do you have a vision or a goal of where you see this going? Like a year from now or 2 years from now?

Chrissy: Yeah, I do. I would like it to be the standard in everyone’s toolkit. I would like hundreds of commands. We are currently at I think a 175. We’ve only announced about 150 because the process to make the webpages and everything like that it takes a little bit. But we are already setting the platform for being able to explore hundreds of commands. So with exchange, their module that came from Microsoft and SharePoint, their module that came from Microsoft, they have 700 commands. And currently I think the SQL Server module has a little less than 100. We are making it so that they’re very easy to find. There’s going to be a lot of tagging with categories through their website. But I do expect that this will be part of and I’ve been told but really kind people who are emailing me saying I’ve seen this become, it’s becoming a standard for DBAs. And that’s what I would really like because it does, it simplifies database administration and for me it makes it super exciting. It’s fun. I don’t have to like with test DBAs last backup. It’s no longer a burden to have to test my backups. It’s so fun to sit there and watch it work. It, you know it’ll perform a restore and then it will perform a verify and then it’ll make sure or sorry, checkDB or check table. It’ll make sure that everything works and then it’ll drop it and it just does all the work for you. And you sit there and you like man I enjoyed testing my backups.

Aaron: I’ve got a nice fuzzy feeling that everything is okay.

Chrissy: I can sleep at night. It’s true, it’s so nice.

Carlos: Without adding to my workload right and then I have to stay late at all.

Chrissy: Yeah you can automate this. I automate my SQL server migrations. I set that on a scheduled task and then I check it in the morning. It’s amazing.

Steve: Very interesting.

Carlos: I think I’m going to have to watch the first one.

Chrissy: Yeah you watch the first one, you watch the second one, the third one, you grab on a beer you like whatever. I’ll comeback. It worked to, a few times I’m cool with it. That’s where I’m at now.

Steve: Yep. It’s really interesting because you always hear about how the role of the SQL server DBA is changing over time. And about how people always need to learn more and keep up one the latest and build more into their toolset to be able to be more productive and not fall along way as things evolve. And it seems like the DBA tool set here would be a really good way of keeping up and keeping up your skills and continuing to grow as a DBA.

Chrissy: Absolutely and you, not only because of Power Shell but what’s really, what I really love about this is that it’s the whole DevOps process. For a while we were most of the DevOps process but we didn’t have testing and then Rob came in and like put in like a whole weekends worth of work and now we even have pester test running across the board on every single one of our commands. And so if you want to learn PowerShell, GitHub, and DevOps this is really an ideal project to work on.

Steve: Very nice.

Carlos: Very cool. Well we do appreciate the conversation. I know that we want to continue with some of the community things but should we go ahead and do SQL family now?

Chrissy: Sure.

Carlos: So Chrissy, tell us how you first got started with SQL server.

Chrissy: Alright. I actually wrote this out because I have a terrible memory and I was thinking back, it was actually a good time. So back in 1999, I worked as a startup as a Linux and Windows administrator but our DBA wasn’t super helpful. He got let go and the Dev team asked me to fill in and honestly I have always been curious about SQL server. You know when you go to the text section of the bookstore was always my favorite, and I’d always see SQL books. And I was like man oh, you know Oracle and then I installed Oracle I was like nah. But then I installed SQL server and that was really awesome. I loved that I have always been primarily administrator but I’ve also like development and that really SQL server serves both of those needs. So I picked it up I loved it and I became a web developer and DBA for the startup and then I went from there.

Steve: Great. So if you could change one thing about SQL server, what would it be?

Chrissy: What would your guess that I would like to see more about with SQL server?

Carlos: I’d say Power Shell command.

Chrissy: Yeah, absolutely. I would actually love to see more Power Shell commands from Microsoft. I would like to see the module opensourced and put up on both GitHub and the Power Shell gallery. And in addition I would also like the SQL Server Management Studio, their grid little thing, it’s not nearly as powerful as PowerShell’s Out Grid view which allows not only sorting but filtering in so many ways. I would really like to see that brought into SQL Server Management Studio.

Carlos: Interesting. We’re actually going to have a conversation with the program manager of the SQL Server Management Studio so have to make sure he pass that along.

Steve: And I would agree that whole grid results set in the Management Studio is very limited.

Aaron: If you can imagine the question, do we have x in this table and your table is 30 columns wide. You, you do that query and put that entire grid here. You can type in and you’ll get your answer in seconds.

Chrissy: Not just the column but the yeah the data as well. It’s incredible.

Aaron: So within a 30 column wide or 50 column wide table and you need to you know, you’re asked it’s is this data released, you know. I’ve got email addresses in for example. You could search, find hundreds or search hundreds or thousands of rows in a single table just for the data in it in Out Grid view because it’s so quick.

Carlos: What’s the best piece of career advice you’ve ever received?

Chrissy: Back in the dot com. At that startup that I had previously mentioned, our CEO, he had 2 doctorates in engineering. I think it was like Mechanical Engineering and Computer Engineering from an Ivy League school. He didn’t mind us call him doctor, he was super awesome. He told me to drop out of college because he said this is the dot com and this will never happen again. So if you drop out now and you gained experience whenever this busts and it will, you will be able to get a job. And so I talked to my parents about it they weren’t excited and then I talked to him about it and I was like I don’t know. And he said, “How much do you think that you would make if you drop out, sorry, but once you graduated?” And so I told him and he said, “Okay, I’ll give you that.” And I was like, “Alright, bye.” And so I am, so I dropped out of school and then I went back years later when I could afford it. I got my Bachelors at the University of San Francisco and then after that I still because I knew that I could hold a full time job and go to school. I enrolled in my Masters program and I’m currently 3 classes away from getting my Masters. So that was actually my best career advice ever was to drop out and then go back whenever really whenever school suited me.

Carlos: Very cool.

Steve: Okay. So if you could have one superhero power, what would it be and why would you want it?

Chrissy: Alright, so my superhero power would be regeneration. And the reason I want it is because I’m a hypochondriac and I’m always worrying about getting some disease or getting to an accident. And if I had regeneration powers then I wouldn’t have to worry about that anymore because it would just fix itself.

Steve: Nice.

Episode 90: DBCC CheckDB

DBCC CheckDB.  It is a simple command, but one that can cause database contention, dba stress, confusing results, dba elation, and everything in between.  At one time we will all have to face the results of having a corruption issue and we’re delighted to have Sean McCown with us again to discuss the ins and outs of the issues of this command. If you can remember, Sean was our guest in Episode 41. He is an SQL Server MVP with more than 15 years experience.

We talk with Sean about how frequent issues we have with maintenance like we often we should run CheckDB on our databases or what happens when I can only check one database during my window. While we don’t end up discussing the command as much, we definitely review all aspects of making sure it can run in ALL environments.

 Episode Quote

“I would say that the bigger issue is probably going to be something like snapshot space for your disk”

“When the rubber hits the road I’m still running CheckDB and CheckTable but it’s all about the process behind it”

Listen to Learn

  • Pros and cons of DBCC CheckDB
  • Scenarios to do CheckDB on databases
  • Issues with CheckDB
  • Running CheckTable
  • Minion Tools

Sean on Twitter
Minion CheckDB

SQL Server in the News
Data Access Libraries can affect performance

About Sean McCown

Sean is an SQL Server MVP and the founder of MidnightDBA.com. He is into SQL Server since 1995 and has been working with various databases such as DB2, Oracle, Sybase among others. He is also an editor and the sole DBA expert for InfoWorld Magazine. He spoke in various engagements talking about backups, SSIS and PowerShell. Sean is also a co-owner and principal consultant of MidnightSQL Consulting and the author of the world famous Minion tools.

Transcription: Check DBCC

Carlos: Sean, welcome to the program.

Sean: Thanks, guys.

Carlos: Yes, nice to have you back. Episode 41, you were with us and since then you made a couple of changes, added Steve to the program, and it’s nice of you to come back and chat with us.

Sean: Is this my second or third time on?

Carlos: Oh, I thought it was just the second time.

Sean: It’s possible.

Carlos: Yeah, you’re just a popular guy. You’re just everywhere, here and there, everywhere.

Sean: Oh, God, I wish.

Carlos: Well, so today, we’re going to go backwards a little bit and we’re going to be talking about a command and then we’re going to back into the why we want to be using it and some of the issues around it, so our conversation is going to focus on DBCC CheckDB.

Sean: Oh, one of my favorites.

Carlos: There you go.

Sean: An oldie birdie.

Carlos: That’s right, and the infamous or what made, maybe not what made him but definitely what added to what Paul Randall is today. He likes to frequently give credit for building the CheckDB command, at least, the original one, I think it has gone through some iterations since then.

Sean: I mean he rebuilt it on 05. He didn’t build the original one that I’m aware of. He just did the major overhaul in SQL 2005, right?

Carlos: Oh, the way I remember him telling the story was that he and his team built it originally.

Steve: I think what I’d heard was closer to what Sean was saying, that he redo of it in 2005.

Sean: Yeah, because he was always talking about how it was really messy and how bunch of the error messages were kind of fubarred and they didn’t really give much info on how he went through and straighten all that stuff out in 2005.

Carlos: Got you. Ok, well then, there you go. So I stand corrected. So he made it a little bit better and so ultimately the question is why would we be wanting to run the DBCC CheckDB command?

Sean: Wow, that’s a big question. The general answer, the lightest answer is because stuff happens when you’re dealing with media and typically magnetic media. All kinds of things can go wrong in those rights whether those things that go wrong come from the magnetic media themselves or some kind of memory corruption, sometimes CPU errors can cause things since you can’t physically see it, right? It’s not like sending out a tray of food where you can see that the order served is in disarray, right? You need something to tell you when thing aren’t right or you’re going to be making very bad decisions and you need something that helps you fix on to. I mean, people take that kind of thing for granted but it’s just as important now as it ever was.

Steve: So then with the CheckDB process when you say when things aren’t right, I mean, what type of things are going to pop up that aren’t right that you are going to know about because of CheckDB?

Sean: Did you say what are some of the problems that you wouldn’t know about because of it?

Steve: No, no, what are the things that you’re going to find out about when you run CheckDB if you had problems that you maybe hadn’t run it in the past?

Sean: Right, so you can have problems like out of range errors, so you get data that is out of range for a specific column. That used to be really common in the lower version of SQL. I haven’t seen that kind of thing as much, so the data purity. So that used to be a really big thing when I was doing upgrades from like 70 to 2000, and from 2000 to 2005, and maybe even a little bit on 2005 but it was a lot better then. Where you got an upgrade and you start getting all these PK violations because they didn’t do such a good job in checking primary keys and there are things just flat out like page linking problems. Same thing happens, right, two pages don’t get linked properly and therefore when you go to run a query it can’t find the data it needs and you get an error because the link that it has on the page doesn’t point to anywhere. You know, and of course they are the same problems with index pages as well. Those are clearly the easier ones to find. And then of course then your disastrous ones where you’ve got problems and pages for, and the system pages for the database itself. There are different classes of those problems. I mean, Steve, you’re a big CheckDB guy so you know all about that kind of stuff.

Steve: Oh yeah. So I guess one of the things, I mean, when I see that oftentimes people are either not running CheckDB at all or they’re not running it frequent enough. And I guess frequent enough is one of those sort of variable terms. But, I mean, how often should people, and from your perspective, should people be checking their databases with CheckDB?

Sean: Well, you know, there’s actually a third one there. People, they don’t run it at all, they don’t run it often enough, or they run it and they don’t check the results. That’s a really big one. Well, I’m sure we’ll talk about that more later. But as far as how often should they be running it? Man, that varies so much. I mean, what does often enough mean? I would say if you’ve got a small database where you’ve got plenty of resources and really you’re not in danger of affecting a workload, run it two or three times a day. I mean, you’re not going to hurt anything, right? It’s not like re-indexing where it’s actually going to, or you could actually mess stuff up.

Steve: So then, like let’s say, though you’ve got a much larger database and it’s really not feasible to run it that frequent.

Sean: As often as you can.

Steve: I had conversations sometimes where people say, “Well, we can only run it, we can only afford to run it once a month on each database because things are just too big.” One of the things I commonly see there is that well they’re maybe running it once a month but they only have three weeks of full back ups. If they’re doing something like that they might not have back up history far enough to go back to where that last CheckDB was. How would you come back, I don’t know, how would you address that?

Sean: Are you baiting me?

Steve: No, no, those just come up in conversations. I was just saying what would you take beyond that when somebody can’t run it. I mean, it’s so infrequent that they don’t have a back up period long enough to go back in the database.

Sean: I know that. I’m just saying, I’m just asking if you’re baiting me because that’s one of the big situations that I was adamant about handling in Minion CheckDB. I wondered if you’re baiting me and to see how long I could go without saying Minion CheckDB.

Carlos: So let me jump in here I guess just for a second as the knuckle dragger Neanderthal of the panel here. We’ve talked about some of these scenarios. I don’t think we explicitly called out corruption although I think all of the scenarios that you listed could be classified as a corruption type issue. But the constraint or the reason people don’t want to run it, you talked about impact, because it has to read everything don’t we kind of get into a memory constraint issue? Because generally our databases are going to be larger than the memory that we have available so we’re going to be causing pressure because we’re going to be flushing things out of the buffer cache.

Sean: You know, usually the pressure I’ve seen for CheckDB isn’t as much memory as it is disk and some CPU running there as well depending on how much CheckDB is going to multithread of you’re on Enterprise. But I would say that the bigger issue is probably going to be something like snapshot space for your disk. I haven’t personally ever run out of memory when running a CheckDB.

Carlos: Well then what I mean by that is that the reason I can’t run it I’m going to affect my applications because now I’m flushing things out of the buffer cache that I need for my application because I’m running CheckDB.

Sean: No, because it’s going to take an internal snapshot and it really runs it against that internal snapshot. So really when we say it’s going to affect external application it’s going to be through CPU and disk. Let’s say that, one of the big scenarios especially on a really busy system with a really big database is that it’s going to run for quite some time. Let’s say it takes 5 or 6 hours to run a CheckDB which not only isn’t all that uncommon, it’s really not all that long for a really big database. And if you don’t have enough disk space then you’re going to expand that snapshot. It’s going to keep on expanding until it fills up the disk and then either it’s going to crash or it’s going to keep your database from being able to expand when it needs to and then you’re going to halt your processes and start rolling stuff back so that and some CPU area. Maybe Steve know something about this that I don’t but I just don’t see flushing the buffer as big of a deal because it’s running against that snapshot.

Carlos: Ok, well so then, I guess I need to ask another question because obviously there’s something about this process that I did not understand. When I hear snapshot, right, I generally think about I’m making, not necessarily a copy but as things change I’m going to keep the history of that, and then the database continues, and then my snapshot grows larger as more changes happen to the database.

Sean: Correct.

Carlos: This is what happens when I’m running DBCC CheckDB?

Sean: Yes. Only it’s internal snapshot that SQL itself runs. It creates a snapshot behind the scenes for you.

Carlos: So depending on the frequency of change in my database then that snapshot is going to increase and that is what’s going to cause me the disk outage that you mentioned.

Sean: Correct, and Steve was trying to say something in the background that I’m really interested on what he’s going to say.

Steve: Right, so what I was going to get at there was around the whole squeeze on memory like Carlos is talking about. The thing I’ve seen is when you’ve got a system where the database is significantly larger than what can fit in memory and queries generally being run that are doing the best they can to deal with the page life expectancy and keep, the right things are staying in the memory for them to run. Oftentimes, what I see with that is sort of if you chart the page life expectancy you’re sort of see the chart pattern where it grows and grows and grows until that CheckDB runs. And then that pushes everything out because everything is needed to be loaded in to be scanned, and the page life expectancy drops to zero when that CheckDB runs. I look at that usually as an indication to be good to add more memory to the system. However, that’s not always feasible in all environments.

Carlos: Ok. So it could affect the three biggies, right? It could affect memory, CPU and disk. But disk obviously because we’re pulling that whole entire database off of the disk and be the snapshotting process I’m getting reads and writes so that’s probably the biggest concern, the first concern potentially.

Sean: I really don’t think it pulls in the entire database at once. It probably goes on time.

Carlos: Oh sure.

Steve: No, it will bring it in page by page just as pages would be normally used. But as it’s doing that it’s pushing out something that hasn’t been used for a little bit through the normal buffer pool process. And you end up pushing a lot of pages out in that process that may have been reused had CheckDb not been run.

Sean: Sure, sure but that doesn’t necessarily mean it’s going to lead to memory pressure.

Steve: No, and I guess, I wasn’t saying that it’s going to push it to the point you’re running out of memory. I’m just saying it’s pushing to the point that you’re causing a lot more I/O because things have been pushed out of memory to make room for what’s being brought in when CheckDB gets run.

Sean: Sure.

Carlos: Yeah, exactly. I mean, so in a certain scenario so would it be fair to say in a sense you could compare that similar to almost like a restart? Because all that stuff just gets pushed out. Maybe not that extreme.

Sean: Sure, let’s go with that.

Steve: Ok, so then one of the things, I mean that comes up is a lot of people say, I can’t run DBCC CheckDB in my whole database. It’s not just good enough there. It’s not going to work. I mean, what are the main reasons you see that people say that or people are looking at they just can’t do it.

Sean: Right, so I’d say there are two big reasons I would say that I see for that. One of them is that that disk filling up that we talked about. Right, Because their databases is just too big and too busy and it won’t run in significant time that the snapshot doesn’t fill up. And that’s a really big problem. I’ve talked to lots of people who just flat out don’t run CheckDBs anymore because of that alone. And another one is that it just takes so long for it to run even if they can withstand the disk, it still just takes so long to run that you know, that’s a drain on resources, right, I mean, ultimately that is a drain on resources. And they just can’t have that resources drain for that long, or you know, they have a thing where they need to reboot now and then or something, something, something, right? They just can’t have a process that runs for literally two days.

Steve: Yeah, you got to exceed your maintenance window, right?  The shops talk about, okay, you can do all your maintenance kind of in this window, and then, always extra things, you’re taking away from the system, therefore, can’t do it.

Sean: Exactly.

Carlos: Okay, so earlier, one of the things you mentioned was that one of the problems is that people who are running CheckDB and they’re not checking the results. I know that if you click off CheckDB and then SQL Server Management Studio Window, you’re going to get results, are all going to show up red on your screen if you’re interactively running it. However, if you’ve got it running as a job what are your best options for knowing the failure or success of that CheckDB process?

Sean: Right, so you’ve got to look at the job itself, right, the job could fail itself and all this depends on how you’ve written your CheckDB routine, right? If you’re like on built in maintenance plans doesn’t it push it to the error log? I think you see those errors in the error log.

Carlos: That’s normally what I see, you run it as a job and that’s how I’m going to know that it failed, is in my job failed.

Sean: Right.

Steve: Right, and I know that whether it’s a maintenance plan or a job or just interactively running it, whenever a CheckDB hits a problem it does right to the error log.

Sean: Right, but.

Steve: There’s also going to be things that show up in your job history as a failure if it’s running as a job or a maintenance plan there.

Sean: Right, and now, I don’t know about you guys but I come from a lot of big environments where we’ve had, you know, a thousand servers, 2 thousand servers and tens and tens of thousands of databases. I don’t know about you guys but I really really want to search the log for something vaguely CheckDB related every week or every day or every something. And then not only that but then go inside of there and parse the results out and try to figure out exactly what the problem was, right? So that’s what I see most, right? The guys who have a little bit more on the ball will run it with table results flag which is better because now you can at least get the table results but you have to build on a whole process for that thing too because that whole table results thing has to be managed and it didn’t help. For the longest time table results was undocumented. Like, somebody was going to bring down the server with table result or something. I never did, I still don’t know why it was undocumented for so long. But the ones that are really on the ball we will run with table results and at least get in a table but like I said, you know, you have to build the whole process around that and it’s not just as easy as, “I’m going to run this with the table results flag and that’s it because I’ve got tens of thousands of databases now.” And to me, any process that I develop has to work against one and two thousand servers with tens of thousands of databases or it’s not going to do me any good for the five databases I have. Right, I mean, we’re being asked as DBA’s to do more, and more, and more and, you know, to spend my time going around and checking all of these things and then parsing the results. Who’s got time for that crap? You know?

Carlos: Yeah, and that’s really an interesting point there because the thing, I mean, I look at the DB, people talk about, “What’s the DBA job going to be in the future?” And I think that it’s something that’s always changing where we’re going with it. But I look at the people who maybe haven’t change in the last 15 years of SQL Server and they’re the kind of people who their job consist of everyday looking at error logs. And mainly scanning those error logs and determining are there any problems that you have to deal with. Whereas the DBAs who were evolving and going forward in the future are the ones who were building the automated processes so you don’t have to manually do that. That you can focus on real issues rather than reading error logs.

Sean: Yeah, because you know, there are two aspects. There’s discovering the problem, reading the problem, solving the problem. And so you don’t want to have to go out. You want this stuff to be as easy as possible especially with corruption. You want to know about it as soon as you possibly can and you want to know the nature of the issues as soon as you possibly can as well. I think that sending CheckDB results to the log is ridiculous. I think sending deadlock information to the log is ridiculous. Am I going to do with deadlock information in the log. I mean, I’ve got this XML graph kind of thing and taking up 37 lines in my log. What am I going to do with that?

Carlos: Sure. You know, it will be a little bit more proactive.

Sean: Just proactive but Microsoft needs to do a better job at making the stuff that we need to monitor accessible. I mean, one of the worst things they did in x events was make the whole thing XML driven. I was one of the worst they have done because the average DBA doesn’t know really anything at all about XML. And so it’s great that I can take that stuff and save it to a file. Now, I’ve got all this XML in a file what am I suppose to do with it. Now, DBAs have to go to a 2-week course just to be able to query the XML data out of there. Thanks for that. I could at least import server side tracing profile data into a table and query it. But here you got to go through some major hoops to get it in the same format. They haven’t done a good job about taking the stuff that we absolutely have to have and making it really accessible to people that are managing on a large scale.

Carlos: So Steve, I’m sensing that we should have asked our SQL Family question little sooner.

Steve: Yeah, that’s good point. We’ll come back to that at the end then.

Carlos: That’s right.

Steve: Yup, so then I guess back to the question around, “Are we baiting you?”, and that’s the time we might start baiting you on. I mean, now that we’ve talked about CheckDB and some of the pros and cons and difficulties and challenges. You’ve come out something here with your Minion CheckDB.

Carlos: Is there something better that we can run?

Sean: Yeah, I like to think it something better. I targeted Minion CheckDB for those large shops with DBAs that are super busy they don’t have time for searching through the logs or for building their own process with table results, and also for the super large databases where you don’t have time to run CheckDB, you can’t run CheckDB because your database is too big. So it was specifically designed with those guys in mind.

Carlos: Well, so now is it different from DBCC CheckDB? Or is it using the same command just kind of slicing it up and giving us more options.

Sean: Yeah, no, it’s all about the process. When the rubber hits the road I’m still running CheckDB and CheckTable but it’s all about the process behind it. For instance, let’s talk about one of the things that I’m particularly proud of process was. We’ll talk about the large database that Steve mentioned where they just don’t run CheckDB because their database is too big. So one of the things you can do with Minion is you can set it up to do a rotating CheckTable instead. So you can say, I want to do 5 CheckTables a night or 500 CheckTables a night until all the tables are done and then start over again. So you’re doing all of the objects. Because what is a CheckDB really? Is it a CheckDB or is it the equivalent operations?

Carlos: Right, all the pieces underneath.

Steve: Although it’s more than just CheckTable and all the tables.

Sean: Of course, it is.

Steve: I think CheckTable and all the tables get’s you started on a big amount of the areas where you’re most likely to have issues.

Sean: Yeah, but you can also run the other smaller operations like as well. But if you want to check your objects and you just can’t because it takes three days then do a hundred at a time, 500 at a time.

Carlos: So how do you get around that? I’ll use smaller numbers so I can wrap my head around it. So I have 10 tables and I can only do 5 at a time. So my maintenance window let’s just say is I can do 5 tables once a week. So it’s going to take me two weekends to get through all ten tables.

Sean: Yes.

Carlos: So when the change happens, so do 1 through 5, weekend number #1. Weekend #2 I’m doing 6 through 10 but corruption happens in Table #1. I’m not going to find that until the next day. I guess going back to Steve’s point, I just take that into consideration and make sure that my backups exceed the time that I can get the total database done.

Sean: Correct.

Steve: Go ahead Sean.

Sean: Another scenario that I do with that is I allow you to automatic remote CheckDBs. So you can set it up to every single night. It will take the latest backup and restore it to a remote box and run a CheckDB on it and send the results back to the prod box.

Carlos: Got you, which is will be awesome particularly maybe not if you have thousands but if you’re in the 20s, 50s, maybe of databases would be a nice way to do it.

Sean: Well, you don’t do that for every database. I mean, every database isn’t that big. But you do it on the ones that can’t do and the other ones you do local. And, you know, so I give you three or four options. Three options I think for restoring your backup remotely and I give you a number of options for the type of remote restore or the type of remote job you want to do. I mean, it’s incredibly configurable with just three, or four, or five columns worth of data in a settings table and then it just goes. I’ve got people doing it and they set it up and once they got the permissions work out and everything that was the biggest thing, right, it was the permissions between the two servers. Once they got that worked out they found that the process itself just runs.

Steve: So then Carlos, going back to your example of small 10 tables but you can only do half of them each week. And I think around that scenario if you’re only able to do half of them per week because that’s your maintenance window. If you were doing full CheckDB previously I think the likelihood of be you never going to actually do CheckDB so even if you’re only finding out a week behind or two weeks behind that something has been encountered that’s a problem, corruption for instance. It’s better to know about it within two weeks rather than never knowing about it.

Carlos: Or when it usually tells you, “Hey, I’m getting this error.”

Steve: Right, right, and you found out you had corruption 18 months ago and it just sort of crept through that amount of time.

Sean: Well, the cool thing is you can even set it up to run your important tables every night. So you could say, I want to run these guys on a schedule but I also want to run my really important tables every single day because these are the guys that I have to have, right? And it’s really easy to exclude tables or exclude schemas or something like that so you can really target it. And really importantly you can order the databases and order the tables as well. So in your maintenance window even if you’re doing all of your tables at once which is another scenario, right? We’ve got the concept of what I call, what do I call it, dynamic snapshots, so getting around that problem of your snapshot file filling up and just killing your CheckDB operation. Well, you can put your on snapshot on there so can create your own disk and put your own snapshot in there but if you don’t have a disk laying around that is big enough, you can switch to CheckTables. But even if you’re doing your own snapshot and CheckTables you’re still going to fill that up. So with dynamic snapshots, what we do is we allow you to say, you know what, every 30 minutes I want you to drop the snapshot and create a new one, and then continue doing the CheckTables.  So if you’re going to do a thousand tables and it’s going to take you say 7 hours to get through it. Every half hour it’s going to drop that snapshot assuming that it’s after an operation or if one table takes an hour and a half there is nothing I can do about that. if it’s over that half our it will drop that snapshot, create a new one, and then carry on with the rest of the tables and that’s going to keep you from filling up that snapshot space, that disk drive. That’s another scenario that we threw in there for this big database problem, right.

Steve: So one of the scenarios that I’ve come across a couple of times that has been challenging has been when you have a larger database that takes a while that run CheckDB against it, and so you’re trying to do CheckTables or other options. But then that one large database, although it’s got a few tables in it there’s one table that really takes up like that 95% of the entire database. So then you run CheckTable against that one really big table and it ends up taking almost as long as CheckDB would have taken to begin with. So are there any options or anything that you see of how you would deal with that differently? And then perhaps run it on another server?

Sean: Well, in that case, I mean how important is that table?

Steve: Well, usually when it’s that size it’s something that’s pretty darn important and it’s just that big because some software along the way wasn’t architected to correctly warehouse things or break it out into another tables.

Sean: Right, well I mean it could just be an archive table that’s sitting there and it doesn’t get data but once a day, or once a week, or once a month, or something. It’s not really that important of a table functionally. The data just needs to be there so there are two different scenarios that you’re talking about. You’re talking about the archive table that’s just sitting there getting bigger and bloating your CheckDBs. And the one that is an actually active table that they just refused to archive and, you know, we’ve all seen it, right? So in that first scenario it’s really just to exclude that table from your CheckTable process and only run it let’s say once a month or once every after two months or something. I mean, that’s really easy to do. The second one is a little bit harder because that is an active table and so you’ve got a couple of things you can do there. You could put it in its own file group and run CheckFileGorup. I like that solution a lot less because I don’t think you really gain anything from that. But yeah, you would either run the CheckDB remotely or you could as well run CheckTable against all of the other tables every day and only save that one for like the weekend or one a month or something like that if it’s going to take a really long time then do that one by itself. Or since it is 95% then run a full CheckDB or full Checktable process on the weekend or once every two weeks or something. But for your dailies, or even better yet you can give it, I want to say a perfunctory. But you could run it with physical only more often than doing a full data check. Just make sure that physical only will just make sure that all the page structures and everything are proper. It won’t bother checking any of the other data types stuff. So it’s a much faster check because it’s a lesser check. But at least you’ll know that the tables themselves have integrity and not necessarily the data. And I’ve got somebody doing that and by the way that’s really easy to do. And Minion is to say on Monday through Friday or on Weekdays I want to run this with physical only, on Saturday I want to run a full CheckTable on it.

Steve: Yup, ok. So another area that’s come up in the bait here and there as I’ve done things with the database corruption and what not is around doing DBCC CheckDB on TempDB. And that’s one that I’ve seen a lot of really intense arguments or one or the other whether you should do it or whether you should not do it. And I’m really just curious of what your thoughts are on checking TempDB.

Sean: Of course you should do it. You know, TempDB is central to everything that you do. I mean so much stuff uses TempDB these days. And if you’re pushing all of your Temp tables through there and a lot of your spill results and you’ve got corruption in there or more importantly you’ve got it on a disk that is causing corruption in there. And you’re going to now rely on the results from these operations. You need to know if TempDB is corrupt. I mean, what you do about it is for debate depending on what’s going to happen. I mean, of course, easiest thing or the only thing that you can really do is delete the files and restart SQL. You’ve got no choice, you can’t really run a pair on TempDB. But yeah, I think because that the data isn’t persistent it doesn’t mean that it can’t be creating corrupt results and just your entire environment and results and what not that are coming through.

Steve: Perfect. And although I tried to ask that question in a very middle of the road state of trying to not waste my opinion one way or the other. I completely agree with your answer there.

Sean: You did a good job.

Steve: And I think that next time I’m in an argument over this with someone I’ll say refer back to Episode 90 of the sqldatapartners podcast and there’s a brief discussion on that.    

Sean: You can tell them I said so, that’s right.

Carlos: I of course agree with you too Steve if that matters but I’m not sure I’ll get a vote.

Sean: You will get a vote we just may not count it.

Steve: Alright, so then I guess back on sort of the topic of the Minion CheckDB side of things. I mean, there’s a lot of people running CheckDB, there’s a lot of people running their own custom CheckDB scripts, there’s people that are running the Ola Hallengren’s database integrity script. So really what I’m trying to understand is who out there is going to get the most value out of the Minion CheckDB product versus one of these other ways of doing it.          

Sean: Dude, I hate to sound like that way but absolutely everybody. The amount of logging that we’ve thrown into this product is obscene. And I don’t mean obscene in a figurative. I mean it is obscene.

Carlos: So you mentioned some of that, so kind of downplayed it a little bit the role or the usefulness of going to the error log. So when you talk about reporting what does that mean? I’m getting an email message, there’s a dashboard, what does that mean?

Sean: So, it means that we store everything in log tables instead of in files or in the log. So we make it really easy to get to.

Carlos: So you can then query it. I mean, is there built in processing that’s going to help notify me like, “Hey, your biggest table has corruption in it.”

Sean: So we don’t get into the alerting business at this level. We save that for our Enterprise product. Because I don’t know how you want to alert. But I’ve made it as simple as possible. I mean, there’s even a flag that says to fail the job because we handle all the errors the job won’t fail so you have to tell us specifically I want to fail the job. Some people don’t want the job to fail. Some people have a separate process that go out there so there are network monitors that can consume SQL queries. Or maybe you want to add a separate step to the job that queries the table and says, “Email me if any errors occurred.” But the point is that we give the errors in a very consumable state. We even tell you how many allocation and consistency errors. We tell you how long it took. We tell you what the last status was. We tell you whether it was a remote job or not. We tell you what server it ran on. We give you an obscene amount of information. There are probably 40 or 50 columns in an hour log details table, logs for absolutely everything. And I would say that’s the biggest advantage that we have over everybody else is again in that process, in the processing of the information and the jobs because we even allow you to do things like, let’s say, that you’ve got even, you know, that you’re using Ola and you’ve got 300 database on a server. And they’re all relatively medium size, right? You have to do all those serially. Or you have to create different jobs and then put them in, 10 or 20 at a time in the parameters.

Carlos: Do the leg work to carve it up yourself.

Sean: To split that up so that you can run those concurrently. You can just, I mean, Minion is already multithreaded so you just say run 10 at a time and it will split them up and it will run 10 at a time.

Carlos: That’s bags the question, when do you start seeing, were you introducing problems by doing so many at the same time. Have you even seen any?

Sean: Like resource wise?

Carlos: Yeah. When is too many too many?

Sean: That’s going to depend on your server and everything involved.

Carlos: That’s fair, too shady.

Sean: I gave you a fully automatic machine gun. What you choose to shoot with it is your problem. One thing I despise is tools that try to be smarter than me. I may not ever use a multithreaded process because this one big server won’t support it most of the time. But then there’s one time when we’re down for our quarterly maintenance and they say, “You’ve got 2 hours to do anything you need while we do all of these other stuffs in our app boxes that hit the SQL box.” And I’ll say, “Ok, I’m going to run 5 CheckDBs at once and get them all done in 2 hours.” You know, because I’ve got the box to myself. You may not use that normally but you may use it three or four times a year when something like that comes up. And you know there’s just so much other stuff in there, right? You can do the same thing with that same rotation we talked about with CheckTables. You can do a set of tables and then do another set of tables and then do another set of tables. You can do CheckDBs that way too so you could rotate those 300 databases; 10 a week or 10 a night. Or you could say you can also run it base off of time. You can run CheckDBs for an hour a night and then just pick up where you left off with the next database tomorrow so it will just roll through all those guys on a timed basis as well.

Carlos: In that scenario is it keeping a snapshot. I’m getting a new snapshot, right? If it has to break the database up so I guess one hour a night, my process takes two hours, which means I need two nights to get through the whole database. I’m getting a new snapshot, right?

Sean: It depends.

Carlos: Oh, ok.

Sean: If you’re running time to CheckDBs which only work off of a rotation which makes sense of think about it then it’s only going to run the CheckDBs that it can do in that time. So you can’t CheckDB half a database.

Carlos: Got you, ok got you.

Sean: But if you’re running CheckTables then it will run through as many of the tables in an hour as it can. Then in the snapshot settings if you’re creating your own snapshot that is, right. If it’s an internal snapshot you have no say there. But If you’re creating your own snapshot then you can tell it to leave the snapshot or to delete the snapshot when it’s done, and then you’ll just get another one tomorrow so you’ve got options there.

Carlos: Another question I had and this goes to something. I want to say it was Grant Fritchey said it but it’s giving me a little bit of heartburn and that is when you have corruption his comment was, “You want to find out why it happened.” And this kind of goes back into their TempDB discussion and you mentioned the disks. So my question is, how do you normally go about identifying why corruption happens.

Sean: I would say that depends. There are a number of things that can cause corruption. Almost in my experience, most of the time it’s going to be a spinning disk. But it can be cause by you copying a database over the wire and it gets corrupted and the bits get corrupted over the wire. Or you’ve got bad memory and the memory is starting to go out and that will do it. I would say disk and memory are the top 2 but I see disk pretty more often.

Carlos: I think I’ve also seen one scenario, I guess, in that a client called me up. They were having some corruption and they’re trying to do like a big bulk import. I don’t know exactly what happened but it sounded like somebody just pushed a button and powered down, you know, hard stop the machine. Something like that caused the problem as well.

Sean: Right, and that can usually be boiled down to probably something being written incorrectly to the disk.

Carlos: Right, so some kind of transaction error or something.

Sean: Right. So to answer your question directly, I kind of preface it there. So to answer your question directly usually if I find like a corrupt table I’ll go on and fix it, and just kind of move on but watch it. Because things happen, stuff happens. But if I start seeing another corruption within a reasonable time say another month, maybe two then I’ll say, “Ok, we need to figure out because we’ve gotten two corruptions inside of a few weeks. And this is just too often.” So I’ll have one of the guys run a disk check and then I’ll have one of them run a memory check. You need to specifically check the disk, and specifically check the memory to make sure that they’re ok. Because you’re right, you do need to know where it is coming from. And I want to say most of the time that’s going to be disk. The thing is I’ve had them come back before and say, “Well, the disk came back clean.” I’m like, “Well, that’s great but you’ve corrupted three tables this week.” I had this conversation with somebody six months ago. “But you’ve corrupted three tables this week. The disk is not ok.” We just ran a full disk check. Well, either run it again or run a different check because you’re not ok. And after going back a few rounds with them they would get the vendor who would either show them a different way to do it or give them a different tool, and bam, the bad sector show up.

Carlos: Sure. There are some flag or some option that they didn’t have before that.

Sean: Check the controller. It could be the controller as well. But something there in the disk process, in the disk write process is not right. And if you stick to your guns, I’m saying this to the DBAs who are going to be listening, right. In this case, you got to stick to your guns. Data doesn’t just corrupt on its own. It just doesn’t get tired so you need to stick to your guns and don’t let them come back and say, “Oh well, we didn’t find anything.” Because you know what, I have found quite often that when they say they do something. Like these guys are network guys, “Oh well, we didn’t find anything.” They actually didn’t look.

Steve: That brings up an interesting story. I mean, I work for a client years ago where. Yeah, there was no reason that they could that they were getting corruption and it just happened. And after looking into it a little bit more, well, it turned out that it just happened because the SQL Server was running a desk without any kind of UPS or back up or anything like that. And every time there was a lightning storm, or a power outage or any kind of problem, it would basically just completely power the whole thing off. And surprise, surprise, you have magically a corruption at that point.

Sean: I had that same problem. One of my first DBA gigs, true DBA gigs was at Pilgrim’s Pride and they had SQL boxes and these were 65 boxes, but still. They had SQL boxes in the kill rooms and so it was just a SQL box sitting on a milk crate with a piece of plastic over it covered in splatters of blood.

Carlos: What?

Sean: And I was just like you’ve got to be kidding me.

Carlos: When you’re talking about kill rooms you’re talking about they’re killing turkeys in this.

Sean: Pilgrim’s Pride, well they process chicken.      

Steve: Oh boy.

Sean: Enterprise wide they process back then about 600,000 chickens an hour, 24/7.

Steve: Oh my gosh! So the server was in the same room.

Sean: The server was in the same room because they have to weigh the chicken at every stage so they want to know what’s the weight of feathers is, and what’s the weight of the guts is, they shift all that stuff out, right? The weight of the feed is, they got to get all that stuff out of this so they got to get the weight and find out exactly what’s what. Yeah, you’ve got to have a SQL box right there by the scales.

Carlos: There you go. So because I feel like I’m going to be left out if I don’t tell a story. Now, I can’t tell you directly back to corruption however being from Richmond, Virginia. Philip Morris, this is the headquarters for tobacco industry and it used to be, I think they’ve finally changed this policy because they were pro tobacco people you could smoke anywhere even in the server rooms. They had pretty good ventilation systems supposedly but that was always the fun story that people would tell is that, “You could walk anywhere and smoke and whatever tobacco product you could use it anywhere in the building.”

Sean: How gross? So to go back to your question a little bit we kind of got sidetrack a little bit and it’s just the nature of how I talk. I would say back to who would benefit from using Minion. You know, it’s out of the gate when you install it. It installs what the job and it installs with schedule. So if you’re not a very good DBA, you just hit the installer and it does everything and it will be up and running and you’ll have CheckDBs. But it’s also configurable for like the world’s greatest masters to come in there and tweak all the bells and whistles and get absolutely everything out of it they want. And one of the big advantages you get especially with all the logging is you get a history at your fingertips of all of the corruption you’ve ever had on that server in that database on that table. And you can say, “Has this ever happen in this table before?” And you can back and look. It gives you a huge advantage of something that you don’t get elsewhere. If you run the query to look at the last time the database was CheckDBed, SQL stores that information in a page. But it doesn’t store anything about CheckTables. So if you’ve got a database where you’re even rotating CheckTables every month and it takes you like 2 weeks or 4 weeks to get through all the tables. That database will show that it has never been CheckDBed eventhough it has. So having all of these here you could easily query, “Oh well, these tables were done here.” And you know when all of the tables have been done you can count that as a CheckDB.  So you could write and SP that tells you the last time it had a full CheckDB which was did all the tables get done last month then this was the last time the tables got done. SQL won’t ever know it but Minion will and we’re going to do more for that as we go through the versions. I mean, this is just version 1 and it was a very ambitious version. I mean, we’ve got so many features in there. It’s ridiculous. One of the things I like about it the most is how, I’d have to say something as corny as we crossed the finished line on a lot of these features. But say you’re doing a remote CheckDB and you’re restoring last night’s backup. On the prod box, I’ll show you in the log, in the status column that it’s restoring to that remote server and I’ll even give you the preset complete of the restore. And as it’s doing the CheckDB on that remote server I’ll even give you the preset complete of that CheckDB on the remote server here. So if you have 5 databases on Prod 1 then they are all being remote CheckDBed to five different servers. You will see the status and get the results back on prod one. You don’t have to go around to all those different servers and check all that stuff manually. You get it back on Prod where it belongs because that’s not a CheckDB of Dev 1 or QA 1. It’s a CheckDB of the Prod 1 database. It’s just being run on over there. So the results belong back on Prod 1. And when I want to see status that status belongs back on Prod 1. I shouldn’t have to go all over BFE to find this information. I think we’ve crossed the finish line really well on some of that stuff like that. We really made it as user friendly and as complete as we possibly can.

Steve: Ok, I think that’s a very in-depth answer to the question I asked but definitely good stuff there. 

Sean: You know I never shut up about my product. You bought that. Think I was a snake when you picked me up?

Steve: I think that wraps it up for all the questions we have to cover. Anything else you want to hit on before we move in to the SQL Family questions?

Sean: No, I better not. I’ll just go and go and go. I’m still so incredibly in love with CheckDB. Well, I’m in love with two things. I’m in love with CheckDB as a product and I’m in love with not having to be developing it anymore. It took me so long to write this and I’m just thrilled to be working on something else.

Carlos: I thought it was available earlier, and you mentioned like shipping it out like last week or.

Sean: February 1.

Carlos: February 1, yeah, so was that actually the first release?

Sean: Yes.

Carlos: Oh, got you. Ok, I guess I’ve just been seeing the stickers or whatever previous to it.

Sean: Yeah, we tease it a little bit. We had the stickers printed a long time ago because we know it was going to be released. I just didn’t know. It took me like a year, a year and a half to finish it, something like that. I mean, you know when I first started it was like, “Hey, CheckDB. This will be easy because there’s not nearly enough in there. There isn’t really much in there as there was in backup. What could possibly go wrong?” But then when you start adding multithreading and mind you this is T-SQL multithreading. I mean, I’m not breaking out in the .NET or Power Shell or anything. This is all done in T-SQL. So we’re doing multithreading, and then the rotations, and then the remote stuff and it adds up. I way over complicate everything so.

Carlos: Well, I heard an interesting thought and talking with one of the Microsoft guys, and all this rapid release things they talked about. And his comment was, “Releasing is the feature.”

Sean: Yeah, I like that.

Carlos: I’m like, you know, yeah because you just get kind of a trap, “Oh, it’s not good enough, it’s not…” You know what, let’s just release and will fix it later.

Steve: Alright, so let’s go on to the SQL Family questions then.

Sean: Do it!

Steve: So how did you first get started with SQL Server?

Sean: Man, I’ll give you a shorter version of the story as I can. So I was a French chef for many years and followed a French chef all around the world, and the latest bistro that we were in closed.

Carlos: This is in France or this is in the States?          

Sean: This was in the States. We were back here at that point. And I was like, “Man, I don’t know what I’m going to do. I’m not really looking forward to moving again.” Because chefs have to move around if you want to get any good experience. And I was kind of tired of moving around and I didn’t want to end up in a subpar hotel or something opening cans on the line. So one of the waiters said, “Well, you know, my sister works for Stream. And I was like, “Ok, great, a field magazine. Great.” And he goes, “No, no, they do tech support.” I was like, “What do I know about tech support? I’m a chef.” And he says, “You know, they’ll train you.” And I was like, “Oh, really?” So I had my own computer and ended up knowing a little bit more that I thought. And being in French kitchens, and being who I am a linguist and all that, I ended up working for HP on the French line. So I was French tech support for Windows for HP. And so I stayed there for a while and then left there and went to this young start up called Health Web and it was a SQL based product for health care where you could put all your patient information up online. And they were like one of the first ones that did this and now it’s everywhere. But they were one of the first ones who did this and I was their tech support. Well, the application was written really well because the guy really knew what he was doing. And so most of the problems that we had ended up being SQL related and this was like SQL 60 and 65. We had one customer at that time. They were hiring me ahead of time and so I had nothing to do. And there was this whole wall of shelves in the back room with all of these computer parts on there. So I built 4 or 5 boxes and put them under my desk and network them and just started learning SQL, and IIS, and Windows. And I didn’t really know that much, right, but I had a guy there who was willing to take the time to show me some stuff. He really mentored me. So I just sat there all day long with the SQL training kits, you know, the official training kits they give for the cert exams back then, and just went through those things and work all the examples and did them different ways. And every day when I came in I would break all my boxes down. All the way down from Windows, and reinstall Windows, reinstall SQL, create databases, load them with data, do their exercises and do the same thing tomorrow because repetition is king. I worked there for like a year and a half or so and every problem we ever had was always SQL related. It was never the application. So they got sold and they offered me a DBA job in Denver. And I was like, “I just bought a house 2 weeks ago. I’m not moving in Denver. I just bought a house. You could have told me that you were selling the company 2 weeks ago.” But it did kind of thought in my head because I had then looking for, you know, when we find out about the sell we have been looking for help desk gigs because I was help desk guy. That’s what I was. But when they came a couple of weeks later and said, “You could move to Denver and be a DBA.” I can’t move but then I was like, “Hey, why don’t I start looking for DBA gigs here.” And I ended up getting hired on at Pilgrim’s and being a helpdesk guy with a lot of lab time under my belt and just a little bit of actual and no real production trouble shooting, no real production database running just running it as a help desk guy answering a question every now and then and I was really pretty green if you ask me. I mean, looking back on it I was a new.

Carlos: We are all were.

Sean: We were all dumb in the mid 90s, right? But he hired me on the spot; he said that I was the best candidate they had ever seen. And I was like, “Really, me?” I guess because I read the book. And the rest is history, I kind of fell into databases. I wasn’t going to be a DBA. I wasn’t even going to be in computers.  I was going to be a chef my whole life.

Steve: Very interesting, French chef to SQL DBA.

Sean: Right, so that’s my story.

Carlos: Now, we talked a little bit about this earlier and perhaps we can revisit it. If you could change one thing about SQL Server what would it be?

Sean: Oh jeez, man, I would say the thing that’s most important to me most of the time is they don’t make information as easy to get at as they think they do or as they’d like even when they do something to help somebody, like the new clone database thing. If you ask me that kind of thing, that thing was half asked. I don’t like the way they do most of the stuff. But anyway, I would say make the information easier to get at. Like there is no reason whatsoever why last CheckDB date shouldn’t be in sys databases. There is no reason why you can’t have table properties and say the last CheckTable date or something like that. Some of this information is just so incredibly hard to get at. Why is it so hard to get table size? Why is it so hard to find some of this information? Even the last time an SP was updated or the last time a table was written to. They don’t make the stuff easy. You got to jump through some major hoops to get some of this stuff. So why isn’t this stuff easier to get at? Why do we have to jump through so many hoops or basic information about the database, right? And I think CheckDB information being the theme here is the best example of that. Why do I have to do this ridiculous DBCC page thing and parse and all of that crap to find out when the last time a database was CheckDBed. You know when it was. Put it in the sys databases and move on. Another one is why is it so hard to find out the last time a database was used. The last time somebody used the database or use a table or something like that to find out if the database is even still in use. SQL should be automatically keeping the list of all the applications that hit it so that you could easily query which applications have hit it and from what server? That stuff shouldn’t be a process that I have to write, SQL should just monitor that stuff and keep a history of it for you.

Steve: Yeah, definitely agree on that one.

Sean: And it should automatically have a firewall. You should be automatically be able to say. I only want these applications to be able to query SQL. Be able to lock anybody who’s not DBA out of every single access except through the intended application. I shouldn’t have to buy a third party product or write something really thick to do that for me. You should just have a list of applications that have connected. Let’s say, click, click, click, add one, I want you to do this, I want you to add this one to it and then that’s it. For anybody who’s not a DBA, DBAs can do whatever they want, but don’t let anybody hit it with Excel. Don’t let anybody hit it with Crystal Reports. That’s what the other boxes for. This one is for this application and that’s it.

Steve: Alright, so we’re into our last SQL Family question.

Sean: So you get me riled up about things.

Steve: We’re not trying very hard either.

Sean: I know, right, it’s so easy.

Steve: Yup. If you could have one superhero power what would it be and why would you want it?

Sean: Oh, God, what did I say last time, Carlos? Do you remember?

Carlos: I’ve had to look it up.

Sean: You know, right now, I’m going to say I wish. It’s split between two. And I really can’t decide which one. I would either want to be able to read minds.

Carlos: Oh yes, I’m pretty sure that’s what you chose all the dirty laundry that comes along with it.

Sean: Or I would want to be able to see details by the way. Not just as vague BS that you see in movies but details into the reasonable future. I don’t want to see vague shapes and colors and find out what that means as I get there, that’s ridiculous. I want to be able to see a day or a weekend to the future and know what’s going to happen and be able to be prepared. Imagine in our field how cool that would be, to know that a problem is coming and have seen the fix already and to be able to do something about it. You’d be a superstar in every sense of the world.

Carlos: Awesome, very rich stock picker as well. 

Sean: Yes exactly, that hadn’t crossed my mind.

Carlos: Warren Buffet it your heart out.

Sean: I know right.

Carlos: Well, awesome. Sean thanks so much for coming on the program today. It has been a blast as always.

Steve: Definitely been fun. Thanks, Sean.

Sean: I love this topic. I’d like to see you guys again for re-index. I’m starting now. And we didn’t even get to talk about the new stuff in Minion back up but we’ll save that for another one.

Carlos: There you go. You guys got another 45 episodes or so.

Sean: Yeah, exactly. You can’t make this the Sean show. Thanks a lot guys!

Steve: Alright, thank you.

Carlos: Thanks again, Sean.

Episode 89: Lone DBA

One of the neat benefits of putting the podcast together is talking to a wide spectrum of folks with varied backgrounds.  As we get more feedback from listeners, the idea of the Lone DBA emerged.  The Lone DBA is someone who is the only person responsible for database administration at their company.  These tasks may have been assigned or they may have simply been required to help support an application.  Steve and I thought we should explore this a bit and to help give us some context, we invited Monica Rathbun to chat with us about some of the challenges she’s faced.

Besides the challenges, Monica shares tactics and techniques to juggle many tasks while being a lone DBA. This includes using social media as a way of building connections with different experts, getting the right tools for the organization’s needs and making your boss your supporter. Have your own perspective on this?  Let us know in the show notes for the episode.

Episode Quotes

“To be a lone DBA, you’ve got to have 48 arms attached to you, doing 20 things at a time”
“It is important for the leaders in the community to mentor and pull people in”
“Making network connections gears you towards your future career”

Listen to Learn

  • Techniques, tactics and challenges of being a lone DBA
  • Using social media in building network connections
  • Managing huge amount of task as a lone DBA
  • Setting the right tools for the organization

US News DBA Job Outlook
Monica on Twitter
Monica’s Blog

SQL in the news
Near-real-time replication from SQL Server to cloud

About Monica Rathbun

Lone DBAMonica is currently a Sr. Database Administrator/BI Architect at Massimo Zanetti Beverage, USA in Norfolk Virginia. She has been a Lone DBA for 15 years, working with all aspects of SQL Server and Oracle. She is co-leader for the Hampton Roads SQL Server User Group and can be found on Twitter daily. When she’s not busy with work, you will find her playing taxi for her two daughters back and forth to dance classes.

Transcription: Lone DBA

Carlos: This is Carlos Chacon.

Steve: I’m Steve Stedman.

Sean: I’m Sean McCown.

Carlos: Sean, welcome to the program.

Sean: Thanks, guys.

Carlos: Yes, nice to have you back. Episode 41, you were with us and since then you made a couple of changes, added Steve to the program, and it’s nice of you to come back and chat with us.

Sean: Is this my second or third time on?

Carlos: Oh, I thought it was just the second time.

Sean: It’s possible.

Carlos: Yeah, you’re just a popular guy. You’re just everywhere, here and there, everywhere.

Sean: Oh, God, I wish.

Carlos: Well, so today, we’re going to go backwards a little bit and we’re going to be talking about a command and then we’re going to back into the why we want to be using it and some of the issues around it, so our conversation is going to focus on DBCC CheckDB.

Sean: Oh, one of my favorites.

Carlos: There you go.

Sean: An oldie birdie.

Carlos: That’s right, and the infamous or what made, maybe not what made him but definitely what added to what Paul Randall is today. He likes to frequently give credit for building the CheckDB command, at least, the original one, I think it has gone through some iterations since then.

Sean: I mean he rebuilt it on 05. He didn’t build the original one that I’m aware of. He just did the major overhaul in SQL 2005, right?

Carlos: Oh, the way I remember him telling the story was that he and his team built it originally.

Steve: I think what I’d heard was closer to what Sean was saying, that he redo of it in 2005.

Sean: Yeah, because he was always talking about how it was really messy and how bunch of the error messages were kind of fubarred and they didn’t really give much info on how he went through and straighten all that stuff out in 2005.

Carlos: Got you. Ok, well then, there you go. So I stand corrected. So he made it a little bit better and so ultimately the question is why would we be wanting to run the DBCC CheckDB command?

Sean: Wow, that’s a big question. The general answer, the lightest answer is because stuff happens when you’re dealing with media and typically magnetic media. All kinds of things can go wrong in those rights whether those things that go wrong come from the magnetic media themselves or some kind of memory corruption, sometimes CPU errors can cause things since you can’t physically see it, right? It’s not like sending out a tray of food where you can see that the order served is in disarray, right? You need something to tell you when thing aren’t right or you’re going to be making very bad decisions and you need something that helps you fix on to. I mean, people take that kind of thing for granted but it’s just as important now as it ever was.

Steve: So then with the CheckDB process when you say when things aren’t right, I mean, what type of things are going to pop up that aren’t right that you are going to know about because of CheckDB?

Sean: Did you say what are some of the problems that you wouldn’t know about because of it?

Steve: No, no, what are the things that you’re going to find out about when you run CheckDB if you had problems that you maybe hadn’t run it in the past?

Sean: Right, so you can have problems like out of range errors, so you get data that is out of range for a specific column. That used to be really common in the lower version of SQL. I haven’t seen that kind of thing as much, so the data purity. So that used to be a really big thing when I was doing upgrades from like 70 to 2000, and from 2000 to 2005, and maybe even a little bit on 2005 but it was a lot better then. Where you got an upgrade and you start getting all these PK violations because they didn’t do such a good job in checking primary keys and there are things just flat out like page linking problems. Same thing happens, right, two pages don’t get linked properly and therefore when you go to run a query it can’t find the data it needs and you get an error because the link that it has on the page doesn’t point to anywhere. You know, and of course they are the same problems with index pages as well. Those are clearly the easier ones to find. And then of course then your disastrous ones where you’ve got problems and pages for, and the system pages for the database itself. There are different classes of  those problems. I mean, Steve, you’re a big CheckDB guy so you know all about that kind of stuff.

Steve: Oh yeah. So I guess one of the things, I mean, when I see that oftentimes people are either not running CheckDB at all or they’re not running it frequent enough. And I guess frequent enough is one of those sort of variable terms. But, I mean, how often should people, and from your perspective, should people be checking their databases with CheckDB?

Sean: Well, you know, there’s actually a third one there. People, they don’t run it at all, they don’t run it often enough, or they run it and they don’t check the results. That’s a really big one. Well, I’m sure we’ll talk about that more later. But as far as how often should they be running it? Man, that varies so much. I mean, what does often enough mean? I would say if you’ve got a small database where you’ve got plenty of resources and really you’re not in danger of affecting a workload. Hell, run it two or three times a day. I mean, you’re not going to hurt anything, right? It’s not like re-indexing where it’s actually going to, or you could actually mess stuff up.

Steve: So then, like let’s say, though you’ve got a much larger database and it’s really not feasible to run it that frequent.

Sean: As often as you can.

Steve: I had conversations sometimes where people say, “Well, we can only run it, we can only afford to run it once a month on each database because things are just too big.” One of the things I commonly see there is that well they’re maybe running it once a month but they only have three weeks of full back ups. If they’re doing something like that they might not have back up history far enough to go back to where that last CheckDB was. How would you come back, I don’t know, how would you address that?

Sean: Are you baiting me?

Steve: No, no, those just come up in conversations. I was just saying what would you take beyond that when somebody can’t run it. I mean, it’s so infrequent that they don’t have a back up period long enough to go back in the database.

Sean: I know that. I’m just saying, I’m just asking if you’re baiting me because that’s one of the big situations that I was adamant about handling in Minion CheckDB. I wondered if you’re baiting me and to see how long I could go without saying Minion CheckDB.

Carlos: So let me jump in here I guess just for a second as the knuckle dragger Neanderthal of the panel here. We’ve talked about some of these scenarios. I don’t think we explicitly called out corruption although I think all of the scenarios that you listed could be classified as a corruption type issue. But the constraint or the reason people don’t want to run it, you talked about impact, because it has to read everything don’t we kind of get into a memory constraint issue? Because generally our databases are going to be larger than the memory that we have available so we’re going to be causing pressure because we’re going to be flushing things out of the buffer cache.

Sean: You know, usually the pressure I’ve seen for CheckDB isn’t as much memory as it is disk and some CPU running there as well depending on how much CheckDB is going to multithread of you’re on Enterprise. But I would say that the bigger issue is probably going to be something like snapshot space for your disk. I haven’t personally ever run out of memory when running a CheckDB.

Carlos: Well then what I mean by that is that the reason I can’t run it I’m going to affect my applications because now I’m flushing things out of the buffer cache that I need for my application because I’m running CheckDB.

Sean: No, because it’s going to take an internal snapshot and it really runs it against that internal snapshot. So really when we say it’s going to affect external application it’s going to be through CPU and disk. Let’s say that, one of the big scenarios especially on a really busy system with a really big database is that it’s going to run for quite some time. Let’s say it takes 5 or 6 hours to run a CheckDB which not only isn’t all that uncommon, it’s really not all that long for a really big database. And if you don’t have enough disk space then you’re going to expand that snapshot. It’s going to keep on expanding until it fills up the disk and then either it’s going to crash or it’s going to keep your database from being able to expand when it needs to and then you’re going to halt your processes and start rolling stuff back so that and some CPU area. Maybe Steve know something about this that I don’t but I just don’t see flushing the buffer as big of a deal because it’s running against that snapshot.

Carlos: Ok, well so then, I guess I need to ask another question because obviously there’s something about this process that I did not understand. When I hear snapshot, right, I generally think about I’m making, not necessarily a copy but as things change I’m going to keep the history of that, and then the database continues, and then my snapshot grows larger as more changes happen to the database.

Sean: Correct.

Carlos: This is what happens when I’m running DBCC CheckDB?

Sean: Yes. Only it’s internal snapshot that SQL itself runs. It creates a snapshot behind the scenes for you.

Carlos: So depending on the frequency of change in my database then that snapshot is going to increase and that is what’s going to cause me the disk outage that you mentioned.

Sean: Correct, and Steve was trying to say something in the background that I’m really interested on what he’s going to say.

Steve: Right, so what I was going to get at there was around the whole squeeze on memory like Carlos is talking about. The thing I’ve seen is when you’ve got a system where the database is significantly larger than what can fit in memory and queries generally being run that are doing the best they can to deal with the page life expectancy and keep, the right things are staying in the memory for them to run. Oftentimes, what I see with that is sort of if you chart the page life expectancy you’re sort of see the chart pattern where it grows and grows and grows until that CheckDB runs. And then that pushes everything out because everything is needed to be loaded in to be scanned, and the page life expectancy drops to zero when that CheckDB runs. I look at that usually as an indication to be good to add more memory to the system. However, that’s not always feasible in all environments.

Carlos: Ok. So it could affect the three biggies, right? It could affect memory, CPU and disk. But disk obviously because we’re pulling that whole entire database off of the disk and be the snapshotting process I’m getting reads and writes so that’s probably the biggest concern, the first concern potentially.

Sean: I really don’t think it pulls in the entire database at once. It probably goes on time.

Carlos: Oh sure.

Steve: No, it will bring it in page by page just as pages would be normally used. But as it’s doing that it’s pushing out something that hasn’t been used for a little bit through the normal buffer pool process. And you end up pushing a lot of pages out in that process that may have been reused had CheckDb not been run.

Sean: Sure, sure but that doesn’t necessarily mean it’s going to lead to memory pressure.

Steve: No, and I guess, I wasn’t saying that it’s going to push it to the point you’re running out of memory. I’m just saying it’s pushing to the point that you’re causing a lot more I/O because things have been pushed out of memory to make room for what’s being brought in when CheckDB gets run.

Sean: Sure.

Carlos: Yeah, exactly. I mean, so in a certain scenario so would it be fair to say in a sense you could compare that similar to almost like a restart? Because all that stuff just gets pushed out. Maybe not that extreme.

Sean: Sure, let’s go with that.

Steve: Ok, so then one of the things, I mean that comes up is a lot of people say, I can’t run DBCC CheckDB in my whole database. It’s not just good enough there. It’s not going to work. I mean, what are the main reasons you see that people say that or people are looking at they just can’t do it.

Sean: Right, so I’d say there are two big reasons I would say that I see for that. One of them is that that disk filling up that we talked about. Right, Because their databases is just too big and too busy and it won’t run in significant time that the snapshot doesn’t fill up. And that’s a really big problem. I’ve talked to lots of people who just flat out don’t run CheckDBs anymore because of that alone. And another one is that it just takes so long for it to run even if they can withstand the disk, it still just takes so long to run that you know, that’s a drain on resources, right, I mean, ultimately that is a drain on resources. And they just can’t have that resources drain for that long, or you know, they have a thing where they need to reboot now and then or something, something, something, right? They just can’t have a process that runs for literally two days.

Steve: Yeah, you got to exceed your maintenance window, right?  The shops talk about, okay, you can do all your maintenance kind of in this window, and then, always extra things, you’re taking away from the system, therefore, can’t do it.

Sean: Exactly.

Carlos: Okay, so earlier, one of the things you mentioned was that one of the problems is that people who are running CheckDB and they’re not checking the results. I know that if you click off CheckDB and then SQL Server Management Studio Window, you’re going to get results, are all going to show up red on your screen if you’re interactively running it. However, if you’ve got it running as a job what are your best options for knowing the failure or success of that CheckDB process?

Sean: Right, so you’ve got to look at the job itself, right, the job could fail itself and all this depends on how you’ve written your CheckDB routine, right? If you’re like on built in maintenance plans doesn’t it push it to the error log? I think you see those errors in the error log.

Carlos: That’s normally what I see, you run it as a job and that’s how I’m going to know that it failed, is in my job failed.

Sean: Right.

Steve: Right, and I know that whether it’s a maintenance plan or a job or just interactively running it, whenever a CheckDB hits a problem it does right to the error log.

Sean: Right, but.

Steve: There’s also going to be things that show up in your job history as a failure if it’s running as a job or a maintenance plan there.

Sean: Right, and now, I don’t know about you guys but I come from a lot of big environments where we’ve had, you know, a thousand servers, 2 thousand servers and tens and tens of thousands of databases. I don’t know about you guys but I really really want to search the log for something vaguely CheckDB related every week or every day or every something. And then not only that but then go inside of there and parse the results out and try to figure out exactly what the problem was, right? So that’s what I see most, right? The guys who have a little bit more on the ball will run it with table results flag which is better because now you can at least get the table results but you have to build on a whole process for that thing too because that whole table results thing has to be managed and it didn’t help. For the longest time table results was undocumented. Like, somebody was going to bring down the server with table result or something. I never did, I still don’t know why it was undocumented for so long. But the ones that are really on the ball we will run with table results and at least get in a table but like I said, you know, you have to build the whole process around that and it’s not just as easy as, “I’m going to run this with the table results flag and that’s it because I’ve got tens of thousands of databases now.” And to me, any process that I develop has to work against one and two thousand servers with tens of thousands of databases or it’s not going to do me any good for the five databases I have. Right, I mean, we’re being asked as DBA’s to do more, and more, and more and, you know, to spend my time going around and checking all of these things and then parsing the results. Who’s got time for that crap? You know?

Carlos: Yeah, and that’s really an interesting point there because the thing, I mean, I look at the DB, people talk about, “What’s the DBA job going to be in the future?” And I think that it’s something that’s always changing where we’re going with it. But I look at the people who maybe haven’t change in the last 15 years of SQL Server and they’re the kind of people who their job consist of everyday looking at error logs. And mainly scanning those error logs and determining are there any problems that you have to deal with. Whereas the DBAs who were evolving and going forward in the future are the ones who were building the automated processes so you don’t have to manually do that. That you can focus on real issues rather than reading error logs.

Sean: Yeah, because you know, there are two aspects. There’s discovering the problem, reading the problem, solving the problem. And so you don’t want to have to go out. You want this stuff to be as easy as possible especially with corruption. You want to know about it as soon as you possibly can and you want to know the nature of the issues as soon as you possibly can as well. I think that sending CheckDB results to the log is ridiculous. I think sending deadlock information to the log is ridiculous. The hell am I going to do with deadlock information in the log. I mean, I’ve got this XML graph kind of thing and taking up 37 lines in my log. What am I going to do with that?

Carlos: Sure. You know, it will be a little bit more proactive.

Sean: Just proactive but Microsoft needs to do a better job at making the stuff that we need to monitor accessible. I mean, one of the worst things they did in x events was make the whole thing XML driven. I was one of the worst they have done because the average DBA doesn’t know really anything at all about XML. And so it’s great that I can take that stuff and save it to a file. Now, I’ve got all this XML in a file what the hell am I suppose to do with it. Now, DBAs have to go to a 2-week course just to be able to query the XML data out of there. Thanks for that. I could at least import server side tracing profile data into a table and query it. But here you got to go through some major hoops to get it in the same format. They haven’t done a good job about taking the stuff that we absolutely have to have and making it really accessible to people that are managing on a large scale.

Carlos: So Steve, I’m sensing that we should have asked our SQL Family question little sooner.

Steve: Yeah, that’s good point. We’ll come back to that at the end then.

Carlos: That’s right.

Steve: Yup, so then I guess back to the question around, “Are we baiting you?”, and that’s the time we might start baiting you on. I mean, now that we’ve talked about CheckDB and some of the pros and cons and difficulties and challenges. You’ve come out something here with your Minion CheckDB.

Carlos: Is there something better that we can run?

Sean: Yeah, I like to think it something better. I targeted Minion CheckDB for those large shops with DBAs that are super busy they don’t have time for searching through the logs or for building their own process with table results, and also for the super large databases where you don’t have time to run CheckDB, you can’t run CheckDB because your database is too big. So it was specifically designed with those guys in mind.

Carlos: Well, so now is it different from DBCC CheckDB? Or is it using the same command just kind of slicing it up and giving us more options.

Sean: Yeah, no, it’s all about the process. When the rubber hits the road I’m still running CheckDB and CheckTable but it’s all about the process behind it. For instance, let’s talk about one of the things that I’m particularly proud of process was. We’ll talk about the large database that Steve mentioned where they just don’t run CheckDB because their database is too big. So one of the things you can do with Minion is you can set it up to do a rotating CheckTable instead. So you can say, I want to do 5 CheckTables a night or 500 CheckTables a night until all the tables are done and then start over again. So you’re doing all of the objects. Because what is a CheckDB really? Is it a CheckDB or is it the equivalent operations?

Carlos: Right, all the pieces underneath.

Steve: Although it’s more than just CheckTable and all the tables.

Sean: Of course, it is.

Steve: I think CheckTable and all the tables get’s you started on a big amount of the areas where you’re most likely to have issues.

Sean: Yeah, but you can also run the other smaller operations like as well. But if you want to check your objects and you just can’t because it takes three days then do a hundred at a time, 500 at a time.

Carlos: So how do you get around that? I’ll use smaller numbers so I can wrap my head around it. So I have 10 tables and I can only do 5 at a time. So my maintenance window let’s just say is I can do 5 tables once a week. So it’s going to take me two weekends to get through all ten tables.

Sean: Yes.

Carlos: So when the change happens, so do 1 through 5, weekend number #1. Weekend #2 I’m doing 6 through 10 but corruption happens in Table #1. I’m not going to find that until the next day. I guess going back to Steve’s point, I just take that into consideration and make sure that my backups exceed the time that I can get the total database done.

Sean: Correct.

Steve: Go ahead Sean.

Sean: Another scenario that I do with that is I allow you to automatic remote CheckDBs. So you can set it up to every single night. It will take the latest backup and restore it to a remote box and run a CheckDB on it and send the results back to the prod box.

Carlos: Got you, which is will be awesome particularly maybe not if you have thousands but if you’re in the 20s, 50s, maybe of databases would be a nice way to do it.

Sean: Well, you don’t do that for every database. I mean, every database isn’t that big. But you do it on the ones that can’t do and the other ones you do local. And, you know, so I give you three or four options. Three options I think for restoring your backup remotely and I give you a number of options for the type of remote restore or the type of remote job you want to do. I mean, it’s incredibly configurable with just three, or four, or five columns worth of data in a settings table and then it just goes. I’ve got people doing it and they set it up and once they got the permissions work out and everything that was the biggest thing, right, it was the permissions between the two servers. Once they got that worked out they found that the process itself just runs.

Steve: So then Carlos, going back to your example of small 10 tables but you can only do half of them each week. And I think around that scenario if you’re only able to do half of them per week because that’s your maintenance window. If you were doing full CheckDB previously I think the likelihood of be you never going to actually do CheckDB so even if you’re only finding out a week behind or two weeks behind that something has been encountered that’s a problem, corruption for instance. It’s better to know about it within two weeks rather than never knowing about it.

Carlos: Or when it usually tells you, “Hey, I’m getting this error.”

Steve: Right, right, and you found out you had corruption 18 months ago and it just sort of crept through that amount of time.

Sean: Well, the cool thing is you can even set it up to run your important tables every night. So you could say, I want to run these guys on a schedule but I also want to run my really important tables every single day because these are the guys that I have to have, right? And it’s really easy to exclude tables or exclude schemas or something like that so you can really target it. And really importantly you can order the databases and order the tables as well. So in your maintenance window even if you’re doing all of your tables at once which is another scenario, right? We’ve got the concept of what I call, what the hell do I call it, dynamic snapshots, so getting around that problem of your snapshot file filling up and just killing your CheckDB operation. Well, you can put your on snapshot on there so can create your own disk and put your own snapshot in there but if you don’t have a disk laying around that is big enough, you can switch to CheckTables. But even if you’re doing your own snapshot and CheckTables you’re still going to fill that up. So with dynamic snapshots, what we do is we allow you to say, you know what, every 30 minutes I want you to drop the snapshot and create a new one, and then continue doing the CheckTables.  So if you’re going to do a thousand tables and it’s going to take you say 7 hours to get through it. Every half hour it’s going to drop that snapshot assuming that it’s after an operation or if one table takes an hour and a half there is nothing I can do about that. if it’s over that half our it will drop that snapshot, create a new one, and then carry on with the rest of the tables and that’s going to keep you from filling up that snapshot space, that disk drive. That’s another scenario that we threw in there for this big database problem, right.

Steve: So one of the scenarios that I’ve come across a couple of times that has been challenging has been when you have a larger database that takes a while that run CheckDB against it, and so you’re trying to do CheckTables or other options. But then that one large database, although it’s got a few tables in it there’s one table that really takes up like that 95% of the entire database. So then you run CheckTable against that one really big table and it ends up taking almost as long as CheckDB would have taken to begin with. So are there any options or anything that you see of how you would deal with that differently? And then perhaps run it on another server?

Sean:  Well, in that case, I mean how important is that table?

Steve: Well, usually when it’s that size it’s something that’s pretty darn important and it’s just that big because some software along the way wasn’t architected to correctly warehouse things or break it out into another tables.

Sean: Right, well I mean it could just be an archive table that’s sitting there and it doesn’t get data but once a day, or once a week, or once a month, or something. It’s not really that important of a table functionally. The data just needs to be there so there are two different scenarios that you’re talking about. You’re talking about the archive table that’s just sitting there getting bigger and bloating your CheckDBs. And the one that is an actually active table that they just refused to archive and, you know, we’ve all seen it, right? So in that first scenario it’s really just to exclude that table from your CheckTable process and only run it let’s say once a month or once every after two months or something. I mean, that’s really easy to do. The second one is a little bit harder because that is an active table and so you’ve got a couple of things you can do there. You could put it in its own file group and run CheckFileGorup. I like that solution a lot less because I don’t think you really gain anything from that. But yeah, you would either run the CheckDB remotely or you could as well run CheckTable against all of the other tables every day and only save that one for like the weekend or one a month or something like that if it’s going to take a really long time then do that one by itself. Or since it is 95% then run a full CheckDB or full Checktable process on the weekend or once every two weeks or something. But for your dailies, or even better yet you can give it, I want to say a perfunctory. But you could run it with physical only more often than doing a full data check. Just make sure that physical only will just make sure that all the page structures and everything are proper. It won’t bother checking any of the other data types stuff. So it’s a much faster check because it’s a lesser check. But at least you’ll know that the tables themselves have integrity and not necessarily the data. And I’ve got somebody doing that and by the way that’s really easy to do. And Minion is to say on Monday through Friday or on Weekdays I want to run this with physical only, on Saturday I want to run a full CheckTable on it.

Steve: Yup, ok. So another area that’s come up in the bait here and there as I’ve done things with the database corruption and what not is around doing DBCC CheckDB on TempDB. And that’s one that I’ve seen a lot of really intense arguments or one or the other whether you should do it or whether you should not do it. And I’m really just curious of what your thoughts are on checking TempDB.

Sean: Of course you should do it. You know, TempDB is central to everything that you do. I mean so much stuff uses TempDB these days. And if you’re pushing all of your Temp tables through there and a lot of your spill results and you’ve got corruption in there or more importantly you’ve got it on a disk that is causing corruption in there. And you’re going to now rely on the results from these operations. You need to know if TempDB is corrupt. I mean, what you do about it is for debate depending on what’s going to happen. I mean, of course, easiest thing or the only thing that you can really do is delete the files and restart SQL. You’ve got no choice, you can’t really run a pair on TempDB. But yeah, I think because that the data isn’t persistent it doesn’t mean that it can’t be creating corrupt results and your entire environment and results and what not that are coming through.

Steve: Perfect. And although I tried to ask that question in a very middle of the road state of trying to not waste my opinion one way or the other. I completely agree with your answer there.

Sean: You did a good job.

Steve: And I think that next time I’m in an argument over this with someone I’ll say refer back to Episode 90 of the sqldatapartners podcast and there’s a brief discussion on that.    

Sean: You can tell them I said so, that’s right.

Carlos: I of course agree with you too Steve if that matters but I’m not sure I’ll get a vote.

Sean: You will get a vote we just may not count it.

Steve: Alright, so then I guess back on sort of the topic of the Minion CheckDB side of things. I mean, there’s a lot of people running CheckDB, there’s a lot of people running their own custom CheckDB scripts, there’s people that are running the Ola Hallengren’s database integrity script. So really what I’m trying to understand is who out there is going to get the most value out of the Minion CheckDB product versus one of these other ways of doing it.          

Sean: Dude, I hate to sound like that way but absolutely everybody. The amount of logging that we’ve thrown into this product is obscene. And I don’t mean obscene in a figurative. I mean it is obscene.

Carlos: So you mentioned some of that, so kind of downplayed it a little bit the role or the usefulness of going to the error log. So when you talk about reporting what does that mean? I’m getting an email message, there’s a dashboard, what does that mean?

Sean: So, it means that we store everything in log tables instead of in files or in the log. So we make it really easy to get to.

Carlos: So you can then query it. I mean, is there built in processing that’s going to help notify me like, “Hey, your biggest table has corruption in it.”

Sean: So we don’t get into the alerting business at this level. We save that for our Enterprise product. Because I don’t know how you want to alert. But I’ve made it as simple as possible. I mean, there’s even a flag that says to fail the job because we handle all the errors the job won’t fail so you have to tell us specifically I want to fail the job. Some people don’t want the job to fail. Some people have a separate process that go out there so there are network monitors that can consume SQL queries. Or maybe you want to add a separate step to the job that queries the table and says, “Email me if any errors occurred.” But the point is that we give the errors in a very consumable state. We even tell you how many allocation and consistency errors. We tell you how long it took. We tell you what the last status was. We tell you whether it was a remote job or not. We tell you what server it ran on. We give you an obscene amount of information. There are probably 40 or 50 columns in an hour log details table, logs for absolutely everything. And I would say that’s the biggest advantage that we have over everybody else is again in that process, in the processing of the information and the jobs because we even allow you to do things like, let’s say, that you’ve got even, you know, that you’re using Ola and you’ve got 300 database on a server. And they’re all relatively medium size, right? You have to do all those serially. Or you have to create different jobs and then put them in, 10 or 20 at a time in the parameters.

Carlos: Do the leg work to carve it up yourself.

Sean: To split that up so that you can run those concurrently. You can just, I mean, Minion is already multithreaded so you just say run 10 at a time and it will split them up and it will run 10 at a time.

Carlos: That’s bags the question, when do you start seeing, were you introducing problems by doing so many at the same time. Have you even seen any?

Sean: Like resource wise?

Carlos: Yeah. When is too many too many?

Sean: That’s going to depend on your server and everything involved.

Carlos: That’s fair, too shady.

Sean: I gave you a fully automatic machine gun. What you choose to shoot with it is your problem. One thing I despise is tools that try to be smarter than me. I may not ever use a multithreaded process because this one big server won’t support it most of the time. But then there’s one time when we’re down for our quarterly maintenance and they say, “You’ve got 2 hours to do anything you need while we do all of these other stuffs in our app boxes that hit the SQL box.” And I’ll say, “Ok, I’m going to run 5 CheckDBs at once and get them all done in 2 hours.” You know, because I’ve got the box to myself. You may not use that normally but you may use it three or four times a year when something like that comes up. And you know there’s just so much other stuff in there, right? You can do the same thing with that same rotation we talked about with CheckTables. You can do a set of tables and then do another set of tables and then do another set of tables. You can do CheckDBs that way too so you could rotate those 300 databases; 10 a week or 10 a night. Or you could say you can also run it base off of time. You can run CheckDBs for an hour a night and then just pick up where you left off with the next database tomorrow so it will just roll through all those guys on a timed basis as well.

Carlos: In that scenario is it keeping a snapshot. I’m getting a new snapshot, right? If it has to break the database up so I guess one hour a night, my process takes two hours, which means I need two nights to get through the whole database. I’m getting a new snapshot, right?

Sean: It depends.

Carlos: Oh, ok.

Sean: If you’re running time to CheckDBs which only work off of a rotation which makes sense of think about it then it’s only going to run the CheckDBs that it can do in that time. So you can’t CheckDB half a database.

Carlos: Got you, ok got you.

Sean: But if you’re running CheckTables then it will run through as many of the tables in an hour as it can. Then in the snapshot settings if you’re creating your own snapshot that is, right. If it’s an internal snapshot you have no say there. But If you’re creating your own snapshot then you can tell it to leave the snapshot or to delete the snapshot when it’s done, and then you’ll just get another one tomorrow so you’ve got options there.

Carlos: Another question I had and this goes to something. I want to say it was Grant Fritchey said it but it’s giving me a little bit of heartburn and that is when you have corruption his comment was, “You want to find out why it happened.” And this kind of goes back into their TempDB discussion and you mentioned the disks. So my question is, how do you normally go about identifying why corruption happens.

Sean: I would say that depends. There are a number of things that can cause corruption. Almost in my experience, most of the time it’s going to be a spinning disk. But it can be cause by you copying a database over the wire and it gets corrupted and the bits get corrupted over the wire. Or you’ve got bad memory and the memory is starting to go out and that will do it. I would say disk and memory are the top 2 but I see disk pretty more often.

Carlos: I think I’ve also seen one scenario, I guess, in that a client called me up. They were having some corruption and they’re trying to do like a big bulk import. I don’t know exactly what happened but it sounded like somebody just pushed a button and powered down, you know, hard stop the machine. Something like that caused the problem as well.

Sean: Right, and that can usually be boiled down to probably something being written incorrectly to the disk.

Carlos: Right, so some kind of transaction error or something.

Sean: Right. So to answer your question directly, I kind of preface it there. So to answer your question directly usually if I find like a corrupt table I’ll go on and fix it, and just kind of move on but watch it. Because things happen, stuff happens. But if I start seeing another corruption within a reasonable time say another month, maybe two then I’ll say, “Ok, we need to figure out because we’ve gotten two corruptions inside of a few weeks. And this is just too often.” So I’ll have one of the guys run a disk check and then I’ll have one of them run a memory check. You need to specifically check the disk, and specifically check the memory to make sure that they’re ok. Because you’re right, you do need to know where it is coming from. And I want to say most of the time that’s going to be disk. The thing is I’ve had them come back before and say, “Well, the disk came back clean.” I’m like, “Well, that’s great but you’ve corrupted three tables this week.” I had this conversation with somebody six months ago. “But you’ve corrupted three tables this week. The disk is not ok.” We just ran a full disk check. Well, either run it again or run a different check because you’re not ok. And after going back a few rounds with them they would get the vendor who would either show them a different way to do it or give them a different tool, and bam, the bad sector show up.

Carlos: Sure. There are some flag or some option that they didn’t have before that.

Sean: Check the controller. It could be the controller as well. But something there in the disk process, in the disk write process is not right. And if you stick to your guns, I’m saying this to the DBAs who are going to be listening, right. In this case, you got to stick to your guns. Data doesn’t just corrupt on its own. It just doesn’t get tired so you need to stick to your guns and don’t let them come back and say, “Oh well, we didn’t find anything.” Because you know what, I have found quite often that when they say they do something. Like these guys are network guys, “Oh well, we didn’t find anything.” They actually didn’t look.

Steve: That brings up an interesting story. I mean, I work for a client years ago where. Yeah, there was no reason that they could that they were getting corruption and it just happened. And after looking into it a little bit more, well, it turned out that it just happened because the SQL Server was running a desk without any kind of UPS or back up or anything like that. And every time there was a lightning storm, or a power outage or any kind of problem, it would basically just completely power the whole thing off. And surprise, surprise, you have magically a corruption at that point.

Sean: I had that same problem. One of my first DBA gigs, true DBA gigs was at Pilgrim’s Pride and they had SQL boxes and these were 65 boxes, but still. They had SQL boxes in the kill rooms and so it was just a SQL box sitting on a milk crate with a piece of plastic over it covered in splatters of blood.

Carlos: What?

Sean: And I was just like you’ve got to be kidding me.

Carlos: When you’re talking about kill rooms you’re talking about they’re killing turkeys in this.

Sean: Pilgrim’s Pride, well they process chicken.      

Carlos: Chicken, oh my God.

Steve: Oh boy.

Sean: Enterprise wide they process back then about 600,000 chickens an hour, 24/7.

Steve: Oh my gosh! So the server was in the same room.

Sean: The server was in the same room because they have to weigh the chicken at every stage so they want to know what’s the weight of feathers is, and what’s the weight of the guts is, they shift all that stuff out, right? The weight of the feed is, they got to get all that stuff out of this so they got to get the weight and find out exactly what’s what. Yeah, you’ve got to have a SQL box right there by the scales.

Carlos: There you go. So because I feel like I’m going to be left out if I don’t tell a story. Now, I can’t tell you directly back to corruption however being from Richmond, Virginia. Philip Morris, this is the headquarters for tobacco industry and it used to be, I think they’ve finally changed this policy because they were pro tobacco people you could smoke anywhere even in the server rooms. They had pretty good ventilation systems supposedly but that was always the fun story that people would tell is that, “You could walk anywhere and smoke and whatever tobacco product you could use it anywhere in the building.”

Sean: How gross? So to go back to your question a little bit we kind of got sidetrack a little bit and it’s just the nature of how I talk. I would say back to who would benefit from using Minion. You know, it’s out of the gate when you install it. It installs what the job and it installs with schedule. So if you’re not a very good DBA, you just hit the installer and it does everything and it will be up and running and you’ll have CheckDBs. But it’s also configurable for like the world’s greatest masters to come in there and tweak all the bells and whistles and get absolutely everything out of it they want. And one of the big advantages you get especially with all the logging is you get a history at your fingertips of all of the corruption you’ve ever had on that server in that database on that table. And you can say, “Has this ever happen in this table before?” And you can back and look. It gives you a huge advantage of something that you don’t get elsewhere. If you run the query to look at the last time the database was CheckDBed, SQL stores that information in a page. But it doesn’t store anything about CheckTables. So if you’ve got a database where you’re even rotating CheckTables every month and it takes you like 2 weeks or 4 weeks to get through all the tables. That database will show that it has never been CheckDBed eventhough it has. So having all of these here you could easily query, “Oh well, these tables were done here.” And you know when all of the tables have been done you can count that as a CheckDB.  So you could write and SP that tells you the last time it had a full CheckDB which was did all the tables get done last month then this was the last time the tables got done. SQL won’t ever know it but Minion will and we’re going to do more for that as we go through the versions. I mean, this is just version 1 and it was a very ambitious version. I mean, we’ve got so many features in there. It’s ridiculous. One of the things I like about it the most is how, I’d have to say something as corny as we crossed the finished line on a lot of these features. But say you’re doing a remote CheckDB and you’re restoring last night’s backup. On the prod box, I’ll show you in the log, in the status column that it’s restoring to that remote server and I’ll even give you the preset complete of the restore. And as it’s doing the CheckDB on that remote server I’ll even give you the preset complete of that CheckDB on the remote server here. So if you have 5 databases on Prod 1 then they are all being remote CheckDBed to five different servers. You will see the status and get the results back on prod one. You don’t have to go around to all those different servers and check all that stuff manually. You get it back on Prod where it belongs because that’s not a CheckDB of Dev 1 or QA 1. It’s a CheckDB of the Prod 1 database. It’s just being run on over there. So the results belong back on Prod 1. And when I want to see status that status belongs back on Prod 1. I shouldn’t have to go all over BFE to find this information. I think we’ve crossed the finish line really well on some of that stuff like that. We really made it as user friendly and as complete as we possibly can.

Steve: Ok, I think that’s a very in-depth answer to the question I asked but definitely good stuff there. 

Sean: You know I never shut up about my product. You bought that. Think I was a snake when you picked me up?

Steve: I think that wraps it up for all the questions we have to cover. Anything else you want to hit on before we move in to the SQL Family questions?

Sean: No, I better not. I’ll just go and go and go. I’m still so incredibly in love with CheckDB. Well, I’m in love with two things. I’m in love with CheckDB as a product and I’m in love with not having to be developing it anymore. It took me so long to write this and I’m just thrilled to be working on something else.

Carlos: I thought it was available earlier, and you mentioned like shipping it out like last week or.

Sean: February 1.

Carlos: February 1, yeah, so was that actually the first release?

Sean: Yes.

Carlos: Oh, got you. Ok, I guess I’ve just been seeing the stickers or whatever previous to it.

Sean: Yeah, we tease it a little bit. We had the stickers printed a long time ago because we know it was going to be released. I just didn’t know. It took me like a year, a year and a half to finish it, something like that. I mean, you know when I first started it was like, “Hey, CheckDB. This will be easy because there’s not nearly enough in there. There isn’t really much in there as there was in backup. What could possibly go wrong?” But then when you start adding multithreading and mind you this is T-SQL multithreading. I mean, I’m not breaking out in the .NET or Power Shell or anything. This is all done in T-SQL. So we’re doing multithreading, and then the rotations, and then the remote stuff and it adds up. I way over complicate everything so.

Carlos: Well, I heard an interesting thought and talking with one of the Microsoft guys, and all this rapid release things they talked about. And his comment was, “Releasing is the feature.”

Sean: Yeah, I like that.

Carlos: I’m like, you know, yeah because you just get kind of a trap, “Oh, it’s not good enough, it’s not…” You know what, let’s just release and will fix it later.

Steve: Alright, so let’s go on to the SQL Family questions then.

Sean: Do it!

Steve: So how did you first get started with SQL Server?

Sean: Man, I’ll give you a shorter version of the story as I can. So I was a French chef for many years and followed a French chef all around the world, and the latest bistro that we were in closed.

Carlos: This is in France or this is in the States?          

Sean: This was in the States. We were back here at that point. And I was like, “Man, I don’t know what I’m going to do. I’m not really looking forward to moving again.” Because chefs have to move around if you want to get any good experience. And I was kind of tired of moving around and I didn’t want to end up in a subpar hotel or something opening cans on the line. So one of the waiters said, “Well, you know, my sister works for Stream. And I was like, “Ok, great, a field magazine. Great.” And he goes, “No, no, they do tech support.” I was like, “What the hell do I know about tech support? I’m a chef.” And he says, “You know, they’ll train you.” And I was like, “Oh, really?” So I had my own computer and ended up knowing a little bit more that I thought. And being in French kitchens, and being who I am a linguist and all that, I ended up working for HP on the French line. So I was French tech support for Windows for HP. And so I stayed there for a while and then left there and went to this young start up called Health Web and it was a SQL based product for health care where you could put all your patient information up online. And they were like one of the first ones that did this and now it’s everywhere. But they were one of the first ones who did this and I was their tech support. Well, the application was written really well because the guy really knew what he was doing. And so most of the problems that we had ended up being SQL related and this was like SQL 60 and 65. We had one customer at that time. They were hiring me ahead of time and so I had nothing to do. And there was this whole wall of shelves in the back room with all of these computer parts on there. So I built 4 or 5 boxes and put them under my desk and network them and just started learning SQL, and IIS, and Windows. And I didn’t really know that much, right, but I had a guy there who was willing to take the time to show me some stuff. He really mentored me. So I just sat there all day long with the SQL training kits, you know, the official training kits they give for the cert exams back then, and just went through those things and work all the examples and did them different ways. And every day when I came in I would break all my boxes down. All the way down from Windows, and reinstall Windows, reinstall SQL, create databases, load them with data, do their exercises and do the same thing tomorrow because repetition is king. I worked there for like a year and a half or so and every problem we ever had was always SQL related. It was never the application. So they got sold and they offered me a DBA job in Denver. And I was like, “I just bought a house 2 weeks ago. I’m not moving in Denver. I just bought a house. You could have told me that you were selling the company 2 weeks ago.” But it did kind of thought in my head because I had then looking for, you know, when we find out about the sell we have been looking for help desk gigs because I was help desk guy. That’s what I was. But when they came a couple of weeks later and said, “You could move to Denver and be a DBA.” I can’t move but then I was like, “Hey, why don’t I start looking for DBA gigs here.” And I ended up getting hired on at Pilgrim’s and being a helpdesk guy with a lot of lab time under my belt and just a little bit of actual and no real production trouble shooting, no real production database running just running it as a help desk guy answering a question every now and then and I was really pretty green if you ask me. I mean, looking back on it I was a new.

Carlos: We are all were.

Sean: We were all dumb in the mid 90s, right? But he hired me on the spot; he said that I was the best candidate they had ever seen. And I was like, “Really, me?” I guess because I read the book. And the rest is history, I kind of fell into databases. I wasn’t going to be a DBA. I wasn’t even going to be in computers.  I was going to be a chef my whole life.

Steve: Very interesting, French chef to SQL DBA.

Sean: Right, so that’s my story.

Carlos: Now, we talked a little bit about this earlier and perhaps we can revisit it. If you could change one thing about SQL Server what would it be?

Sean: Oh jeez, man, I would say the thing that’s most important to me most of the time is they don’t make information as easy to get at as they think they do or as they’d like even when they do something to help somebody, like the new clone database thing. If you ask me that kind of thing, that thing was half asked. I don’t like the way they do most of the stuff. But anyway, I would say make the information easier to get at. Like there is no reason whatsoever why last CheckDB date shouldn’t be in sys databases. There is no reason why you can’t have table properties and say the last CheckTable date or something like that. Some of this information is just so incredibly hard to get at. Why is it so hard to get table size? Why is it so hard to find some of this information? Even the last time an SP was updated or the last time a table was written to. They don’t make the stuff easy. You got to jump through some major hoops to get some of this stuff. So why isn’t this stuff easier to get at? Why do we have to jump through so many hoops or basic information about the database, right? And I think CheckDB information being the theme here is the best example of that. Why do I have to do this ridiculous DBCC page thing and parse and all of that crap to find out when the last time a database was CheckDBed. You know when it was. Put it in the sys databases and move on. Another one is why is it so hard to find out the last time a database was used. The last time somebody used the database or use a table or something like that to find out if the database is even still in use. SQL should be automatically keeping the list of all the applications that hit it so that you could easily query which applications have hit it and from what server? That stuff shouldn’t be a process that I have to write, SQL should just monitor that stuff and keep a history of it for you.

Steve: Yeah, definitely agree on that one.

Sean: And it should automatically have a firewall. You should be automatically be able to say. I only want these applications to be able to query SQL. Be able to lock anybody who’s not DBA out of every single access except through the intended application. I shouldn’t have to buy a third party product or write something really thick to do that for me. You should just have a list of applications that have connected. Let’s say, click, click, click, add one, I want you to do this, I want you to add this one to it and then that’s it. For anybody who’s not a DBA, DBAs can do whatever they want, but don’t let anybody hit it with Excel. Don’t let anybody hit it with Crystal Reports. That’s what the other boxes for. This one is for this application and that’s it.

Steve: Alright, so we’re into our last SQL Family question.

Sean: So you get me riled up about things.

Steve: We’re not trying very hard either.

Sean: I know, right, it’s so easy.

Steve: Yup. If you could have one superhero power what would it be and why would you want it?

Sean: Oh, God, what did I say last time, Carlos? Do you remember?

Carlos: I’ve had to look it up.

Sean: You know, right now, I’m going to say I wish. It’s split between two. And I really can’t decide which one. I would either want to be able to read minds.

Carlos: Oh yes, I’m pretty sure that’s what you chose all the dirty laundry that comes along with it.

Sean: Or I would want to be able to see details by the way. Not just as vague BS that you see in movies but details into the reasonable future. I don’t want to see vague shapes and colors and find out what that means as I get there, that’s ridiculous. I want to be able to see a day or a weekend to the future and know what’s going to happen and be able to be prepared. Imagine in our field how cool that would be, to know that a problem is coming and have seen the fix already and to be able to do something about it. You’d be a superstar in every sense of the world.

Carlos: Awesome, very rich stock picker as well. 

Sean: Yes exactly, that hadn’t crossed my mind.

Carlos: Warren Buffet it your heart out.

Sean: I know right.

Carlos: Well, awesome. Sean thanks so much for coming on the program today. It has been a blast as always.

Steve: Definitely been fun. Thanks, Sean.

Sean: I love this topic. I’d like to see you guys again for re-index. I’m starting now. And we didn’t even get to talk about the new stuff in Minion back up but we’ll save that for another one.

Carlos: There you go. You guys got another 45 episodes or so.

Sean: Yeah, exactly. You can’t make this the Sean show. Thanks a lot guys!

Steve: Alright, thank you.

Carlos: Thanks again, Sean.