Episode 112: Keeping up with patching

When listener Mel Vargas first suggested the topic of patching, I was not sure this would work as a topic; however, the Equifax story had just developed and there are many other security related issues that could be prevented with patching.  Match that with our guest, Robert Davis, who just happened to publish some articles on patching and I decided we needed to do this episode.

I should never have doubted this as topic.  While we are a bit more protected in SQL Server than others, the security threat is still real and this is something everyone has to go through.  Robert presents us with some interesting details on how he goes about patching and we think you will find the episode compelling.

Episode Quotes

“You not being able to be patched for the new stuff, you are really opening yourself up to a variety of attacks.”

“We’re all on the same team ultimately and the success of the application really benefits all of us.”

“There isn’t a security reason to be diligent about patching SSMS.”

“The really good DBA, really proactive one, gets overlooked because everybody else in the company doesn’t see the fires that you’re preventing.”

“Learning to speak up for myself and to publicize the things I do definitely was one of the best things I had to learn how to do.”

Listen to Learn

01:08   Episode Topic: Patching
02:19   Why should we care about patching and what it helps to do?
06:26   Maintenance window, automated patching and system center
07:56   Octopus: Patching Automation Tool, automation tools
2:52   Deploying and rebooting, and restarting services
15:48   Do you really need to update SQL Server Management Studio for security reason?
19:44   Cumulative updates in Azure
22:34   Windows patches and SQL Server patches, failover cluster, availability groups
26:39   Patching testing
29:07   Scenarios and issues when trying to do install updates or patching
36:32   SQL Family questions

About Robert Davis

Robert is a SQL Server Certified Master, MVP, and has spent 17+ years honing his skills in security, performance tuning, SQL development, high availability, and disaster recovery. He served as PM for the SQL Server Certified Master Program at Microsoft Learning, and in various roles at Microsoft specializing in SQL Server administration, development, and architecture. He currently works as a Database Engineer at BlueMountain Capital Management where he spends a vast majority of his time tuning massively parallel queries. Robert feeds his passion for security by acting as co-leader of the PASS Security Virtual Chapter.

Transcription: Keeping up with patching

*Untranscribed introductory portion*

Carlos: This is Carlos Chacon.

Steve: I am Steve Stedman.

Robert: I am Robert Davis.

Carlos: Robert, welcome to the program.

Robert: Thank you.

Carlos: Yeah, it’s good to have you. Thanks for taking a little time of your busy schedule all the way from New York City to come and chat with us today, so thanks for stopping by.

Robert: Thanks for having me, coming at you from the heart of Manhattan.

Carlos: Wow. Yes, deep downtown.

Steve: Well, I know over the years on different topics that I’ve googled on around SQL Server often times some article or post that you’ve written has come up and definitely appreciate that, it’s always been good content. I want to say thank you.

Robert: Wow, thank you for that feedback.

Carlos: Yeah, and it’s nice to have you on the program today. Ultimately our topic is going to revolve around patching. Maybe not the most sexy of topics but as administrators something that we all have to live with, and I think has only accelerated with the “new” Microsoft and not only we’re getting patches now, we’re actually getting versions and all these things. And so keeping up with all of these can be laborious at times. And so we wanted to start this conversation with talking a little bit about patching. And maybe let’s just start with the basics, right? You know, there’s an inevitable or at least in the past it has been I’m going to wait to upgrade until I get SP1, right? And then, we’ll go forward with the upgrade. And so I think a lot of people look at patching for lots of different reasons. I guess let’s just go ahead and hit on a couple of ideas of why should care about patching and what it helps to do.

Robert: I’d like to just point out the fact that we had a couple of major events very recently in the computing world where there were vast instances of people’s servers being encrypted and it was utilizing very very old vulnerabilities that have been patched years ago that the people affected by them were the people that weren’t patching their servers.

Carlos: Or even learning older versions of older operating systems and things like that and then just have them patched.

Robert: Right, exactly. Those were two major events that could have been stopped right away if people just have their systems patched up to date.

Steve: Interesting. So I mean I have come across situations where people have said, “Oh we’ve got SQL Server 2005”, running on whatever older version of Windows Server it is that they are happen to be running on, “It does everything we need for our business and we don’t need to spend money on patching or updating it because it’s getting the job done for us.” What would you day to someone with the comment like that.

Robert: I would say to those people that if for these really old versions that are no longer supported that means you’re not getting patches for these SQL instances anymore. When you see things like these encryption attacks or SQL-Injection attacks and a lot of these know vulnerabilities attacking people’s systems. In fact, just recently we had a reoccurrence of a mass Slammer attack. If you remember, Slammer hit SQL Server 2000 pretty hard, and suddenly it made a comeback earlier this year. And obviously the people that are going to be affected by that are the people running SQL 2000 that aren’t patched and aren’t getting patches anymore. So by not being on the current version and you not being able to be patched for the new stuff, you are really opening yourself up to a variety of attacks.

Steve: Yup.

Carlos: And so then this kind of gets in to the heart of I think some of our conversations so some of those vulnerabilities. So the worms that were introduced earlier this year there was the patches available 30 days prior. So if you would patch in those 30 days it would have been fine. Some were much older, then we get into that idea of how often or how frequently should we applying them and the testing that goes around with patching? And again I say this is not, I don’t know if anybody that really enjoys patching. I don’t think I’ve met a person that person, right? So I guess thoughts on how can we introduce a process or a culture even to get those patches in and get them tested and maybe what’s necessary to kick the tires, because we know that you as much as we try a lot of times our lower environments aren’t quite the same as our higher environments and that’s a problem for a lot of people.

Robert: Yeah, exactly. But I think really the key to getting people on all our kaitens of regular patching is to introduce automation to the process. Obviously if you’re up late at night installing patches manually waiting for them to finish and then rebooting is going to be a boring tedious task and nobody is going to want to do it. We use system center to push patches out to all of our servers, and with system center we have the ability to. If we have some exception like we don’t want it to be automatically patched and rebooted in the middle of the night we can exclude certain server so we can manually do some certain server but otherwise at a certain point our automation kicks in says, these servers aren’t patched, patch and then reboot them starting at 11PM on Saturday night.

Carlos: There you go, so kind of finding that maintenance window and then making that happen. So you mentioned the system center, right and obviously if you have that product and I’m sure it’s doing a lot more than patching there is some inventory kind of helping you with your server environments just in general. Have you played around with any of the other patching automation tools by chance?

Robert: Yeah, actually when I worked at Microsoft we used Octopus for our patching processes. We would have Octopus, it would push the patch files out to each of the servers and install them and do anything. Octopus is great because if you need to do anything like stop a service before you patch and then restart it after you patch, you can build that right into the Octopus script. So for example, say if we needed to stop a web service that’s getting the web, the database server before we patched and rebooted it we could do that into the Octopus deployment script. It would stop the web services, install the patches and reboot it and then restart the web services on the web servers. It could do the whole, anything that could be automated that could do and the really great thing about Octopus too is we could build into it. Some sort of some basic testing at the end of the deployment like it could go in and send a request to the web service and check to see if it gets a valid response back so that we know. Yeah, we know that the web service. We would setup like some dummy process on the web server that Octopus could hit it would make a database call and it will just return a value, and if it returns the expected value then we know that the database and the web servers are all functioning properly.

Steve: Very interesting. I mean, I have worked in environments were Octopus deploys are used but I’ve never really thought about using it for updates. Kind of makes me want to go back and take a look at it now.

Carlos: Right, well I think it’s a good point too that Robert makes in the sense that you want to automate those but then with automation comes with responsibility in the sense of then going and testing some of those things. I guess there’s a little bit of that fear particularly if we’re going to start restarting services. You want to make sure everything comes up, and so automating the patch almost like it’s step one, right? And then there’s building out the test afterwards to complete the process.

Steve: And that’s the interesting thing being on the DBA side of the world. Oftentimes you don’t always understand that entire application or web service or whatever it may be that’s using the database and know how to even tell if it’s working correctly. And I think that if you can have the tools that automated it that would really be great.

Robert: Absolutely and it definitely wasn’t something that we had done on our own. We had some amazing test people and developers for the application I worked on at Microsoft and they were really proactive about helping us build things like this and creating the web service call that we’ve made then do a dummy transaction in the database and make sure everything was working.

Carlos: It takes a village to do automated patching.

Robert: Absolutely. It often gets lost but we have to remember Dev Team, DBA Team, whoever we are we’re all on the same team ultimately and the success of the application really benefits all of us.

Carlos: Sure, it’s almost taking me back to the conversation that we have with Richard, Steve, about building trust our teams. This is kind of an interesting scenario where it gets to be a win win. One because we get to apply the patches that we want for security purposes but then application people can be assured that the application is still available when we’re done with it.

Steve: Right. And if you have that all automated then it’s less extra hours in the middle of the night for all of those people involved as well.

Carlos: Exactly, which I think everyone would agree is a good thing. I wanted to ask about deploying as well and restarting, you mentioned restarting the services. Is that across the board? I think with SQL Server we tend to it a little bit more just because SQL Server is running, we’re applying that so there’s file dependencies that sometimes get lock and things. Kind of by rule are you restarting your services after you patch them?

Robert: Typically, it’s easy to determine whether or not the automation process you can configure it to reboot it if reboot is recommended. You know, like some patches will say you must reboot to complete installing this patch. Other patches like say applying a CU to SQL Server won’t require you to reboot but then applying the new version of SSMS probably will require you to reboot. Most of the automation or the system center patching that we use gets that message back as to whether or not reboot as needed and then we’ll reboot as needed. You can also override that if you have systems that you don’t want to be rebooted automatically. For example, if you have a system that has a known issue where like maybe when it reboots all the drives don’t always come back online. I have seen cases like that where until they got configure out why some of the drives weren’t coming online they had configure it to not reboot automatically. But in the past without having that when we schedule for patching we to take into account that reboot maybe required and so we schedule Maintenance Window according to how long we think it will take if a reboot is going to be required. And so if you got the time, in the past we just install it automatically whether it says to be rebooted or not, reboot it. And there have been few updates from SQL Server, few CUs, or service packs where it gave me no message saying it needed to be rebooted, so we didn’t reboot. And then the next time I go in there to try and install an update, I then later go in to try and install the SSMS update and it says, can’t install there is a reboot pending. Like, why didn’t you tell me? So if you got the time scheduled and you don’t and you just want to play it safe. I’d say, yeah go ahead and schedule the reboot if you don’t foresee it causing you any problems.

Steve: Alright, so one of the things you mentioned there was the patching of SSMS or SQL Server Management Studio and one of the things that’s interesting around that is that, I mean it’s used to be the Management Studio was part of the SQL server install and if you updated SQL Server you would get your Management Studio patches at the same time. But now Management Studio is detached, running separately and with a completely different patch cycle and we’re getting much more frequent updates there but the other thing is it’s not a very smooth update process like a lot of the Microsoft products desktop applications for instance. You will get a message that says, there is an update required or come part of the Windows update process. But with Management Studio pretty much you got to always download it and then install that update. I’ve talked to a lot of people who, I don’t know, they kind of a little bit bothered at how annoying Management Studio is with monthly updates. And I’m just curious what your take is on updating Management Studio? Because some people I’ve talked to they come across to say, “Well, I’m not going to update Management Studio every month because I don’t need those new features.” But do people really need to update management studio often for security purposes?

Robert: With SSMS there’s really not a big security issue with the SSMS. Nobody is going to take away SSMS on your machine and be able to do bad things with it. Like they could took over your SQL Server instance. So there isn’t a security reason to be diligent about patching SSMS. But they are trying to work out a lot with the process and a lot of the recent versions of SSMS have had bugs in them. And there maybe things you don’t notice right away for specific features that you rarely use but you can definitely look at the update to see what they’ve done and determine if it’s something you need to do. But for myself every time an SSMS update comes up I update the SSMS I have on my desktop. I rarely ever use SSMS on the servers themselves. I like to have SSMS on the server in case there’s a critical issue and the only way that you can connect is via the dedicated admin connection. So I would like to have the SSMS on the server but I almost never use it so I don’t update the SSMS every time on the servers, every time an update for it comes up. But what I will do is when a new CU or SP comes out then I’m going to apply it to my instances. I’ll get the latest version of SSMS and I’ll install that as the same time as I do the CU and updates. I know that SSMS is on the servers themselves is at least as current as the last patch that was applied for the SQL Server itself.

Steve: Yup. Ok, great.

Carlos: Ok, that’s an interesting approach.

Steve: So I knew you mentioned accumulative updates there as well. It used to be like maybe 8 or 10 years ago that cumulative updates weren’t really as safe as they are today it seemed like, and that people would generally just stick with service packs when they are doing patching unless there was a specific reason that they needed cumulative update. But it seems like today cumulative updates are strongly recommended compared to where they were in the past. I’m just curious on what your feelings are on that or your experiences.

Robert: There’s used to be a big topic of debate amongst DBA’s whether or not to install CUs. A lot of people said, they always install the latest CUs. I was one of those that always said, “I only install a CU if it has something that specifically fixes an issue I’m having.” For the very reason that they didn’t get CUs used to not get full regression testing. When they were working on 2016 one of the things they change is that CUs not get the same full regression testing that service packs get.

Carlos: There you go, that’s the change right?

Robert: Yeah, so for me that really took that element of risk out of it. So now I will install a CU at my first opportunity to perform CU updates on the instances.

Carlos: I think that’s the effect that Azures had on the environment and that they need to be able to test those to support the Azure environments.

Robert: Absolutely. And everybody who owns Azure gets those updates automatically already so it was really one of the reasons say had the building the same level of trust for their CUs as they do for service packs because if you’re in Azure whether they’ve taken that choice out of your hand you’re getting the updates whether you want them or not. It’s really one of the selling points of Azures is that you don’t have to worry about the updates. It’s all taken care of for you and it happens automatically without you having to do anything.

Carlos: Right, also true. So before we go on with the other scenarios because I had a lot of questions on, so are you bundling your Windows patches and your SQL Server patches or do you like to do those separately?

Robert: We’re not bundling them, in that sense they are separate.

Carlos: Bundles is the wrong word. I guess what I mean to say, ok it’s patching weekend I want to do all my Windows updates and my SQL Server updates at the same time.

Robert: Yeah, so that’s really the kaitens we’ve gotten into now, and all of our production databases are on availability groups. The way we do it is we always patch the inactive nodes one weekend and do all the patching of the inactive nodes and then we failover our availability groups so that the patch instances is now the currently active node. And then we will run with that for a couple of weeks and then we’ll do the other CU. Unless there are some issue that we feel we need to update sooner we’ll do that another weekend a couple of weeks later and then throwback again.

Carlos: Ok, and then back into this next architecture if you will. I mean, so many people, they are stand alone where there’s clusters, now availability groups. And it sounds like if you’re familiar with the clustering idea that same concept, I can apply my service pack to one, flip it over, apply it to the other, and then either to flip it back or whatever your policy is there. That hasn’t change much with the availability groups.

Robert: Exactly. I mean with the availability groups you still want to patch your inactive node just like you would a failover cluster, failover and then do the nodes that are left that used to be active and then now inactive.

Steve: But it also sounds like either way with that whether it’s an availability group or a failover cluster that you’re delaying the time between when you patch one half of it versus when you come back and do the rest of them, so that you can have time to know if something went wrong and the ability to flip back to the old one if something did go wrong.

Robert: Absolutely, and because the way we do it where we break it up with the nodes to different weekends. If we do run into an issue like if there is a breaking change in the new version which is a lot likely to happen with the service pack than with the CU, but if there is a breaking change and the secondary can’t continue syncing data because there are some upgrade in the database that it’s trying to do but it can’t because it’s the secondary. The type of behavior you would see if you had a major mismatch between the nodes, you then have the option because the unpatch side is now the inactive one. Typically, you don’t even need to wait till the middle of the night. If you say, “Ok I’ve got to get this patch right now.” You’re not going to cause a major outage by now going in patching the secondary to make it match the active database.

Carlos: Yes, then you’re back in sync and hopefully that fixes the issue and they can continue to replicate.

Robert: Absolutely.

Carlos: Yeah, very cool. And then testing, right? We talk about automated deployments, right? Any specific test that you do around admittedly and I guess you know, maybe I shouldn’t admit this, but I think a lot of times my testing has been, “Oh, I install in a lower environments.” The proverbial, wait for something to blow up, and then if I don’t hear anything I move forward, right? Thoughts around patching testing?

Robert: That’s definitely one of the things I’ve always done is I start with dev and then move on to test and then move on to production. So like when the new CU or SP comes out I’ll install it on our dev machines the same day that we get it. And that gives us some time to see if any problems arise there before we go there. Also, usually if there’s a major problem with the CU or SP usually you hear about it within two or three days of the patch going public. Because a lot of people are start installing it on the devs environments right away and if there are problems they start finding it almost immediately. By the time we get around to patching the first set of production servers not only I’ve had it running on dev and test for awhile. A lot of other people have two and a lot of these people are very vocal when they find problems and really put it out on social media, which is one of the things I love about things like Twitter because you really, and some may post out, “Oh, this new CU is breaking my machines don’t install it.” Even if it’s somebody you don’t follow people will start retweeting it and you’ll see it pretty quickly on social media.

Steve: Yeah, that’s interesting because that really shows how valuable social media can be in that case where, I mean a lot of people look at social media in the workplace as a time snake in complete ways but when you’re using it for things like that it’s incredibly valuable. Ok, so one of the things I really want to ask about and it sounds you’ve done a lot of patching, do you have any horror stories either that you’ve seen happen or you’ve heard of happening where somebody tried to do a service pack or an update or a new version of SQL Server and something just went horribly wrong.

Robert: I have seen a few of those. There were some well known bugs that got hit in CUs and service packs where the upgrade would fail, the first part of the upgrade would fail but then not always but sometimes if there are system updates that have to be upgraded. You know, it will run a script to upgrade those objects and it puts the database in a state where if at that point it fails it puts it in a state where it won’t come unwind because the database is marked as in an upgrade. So if the CU or SP fails at that point it will uninstall the upgrade but it doesn’t revert that database and take it out of that upgrading state. It requires manual intervention. Obviously, one of the easiest ways to fix it is to, and we only know this because we’ve had to go through these issues where the servers won’t start.

Carlos: Life experience here.

Robert: Yeah, find the script, try to run, start the server with only master running and run the upgrade script and let it complete and maybe it will come late when you run it manually, if not it may give you a clear error, pin you exactly where it was. Last time I ran into this is because they made some changes to the job systems, the way they track jobs in MSDB. And part of their script was it deleted all of system categories by default and then inserted a bunch of new system categories, and had the IDs for those categories hard coded expecting that the likelihood that somebody has created a custom ID or custom category and that it’s using one of these IDs were going to use. They basically just assume that those category IDs wouldn’t be taken up, so we had custom categories that we had created and those IDs were in use and so the script failed because there is a unique constrain on the column. These were really kind of basic things we would yell at our developers if they didn’t check for if they’re going to insert hard coded values and not check to see if they exist already. But those get reported, they fixed those issues, and we haven’t seen that kind of thing…

Carlos: Click, yeah, record again. I apologize, Robert. If you’ll just go ahead and just pick up that story again talking about the change they made to the schedules or the categories then we can keep going.

Robert: Yes, so last time I encountered this issue myself was because they were upgrading the job system in MSDB which involved, their script would delete all the system categories from the categories table and then reinsert a bunch of new categories. The new categories was actually about as twice as many categories and they just made the assumption that the IDs, the hard coded IDs, they were inserting for the categories wouldn’t have been taken up by any user created category IDs. But we had created several categories of our own so when the script ran it had the unique constrain error because some of the IDs they were trying to insert were already in the table, and so it failed in the middle of the upgrade. So I found the script that they ran, ran it manually, got the error that it was generating. And I say, ok, so I went there. I dumped out the data from the table into something so I could fix it later, and then just delete, emptied out the table, and rerun the script and it completed successfully. These issues have all been reported to the team. I particularly gave them a hard time over that one because that’s the kind of stuff we would yell at our developers if they did. Like, why are you inserting hard coded values in not checking to see if they exist already?

Carlos: Or nothing else at least go to like 500 or something, right?

Robert: Right. But they took all that information in and they address those problems and these things never happened for a really long time. The only recent occurrences I’ve heard of these issues happen for people that are installing old updates. Like somebody is installing, “Oh look I just found out there’s Service Pack 3 for SQL 2005. Let me install that”, and oops it fails.

Steve: Interesting as you say that. I mean can we even get service packs for 2005 anymore? I remember recently somebody looking for one and we couldn’t find it anywhere on the Microsoft side.

Robert: Right, so you can’t get it through self service, you can’t find it yourself, but if you need to you can call my reserve support and they can get you old service packs.

Steve: Ok.

Robert: They are just going to ask you why you don’t upgrade. Obviously they are going to want to know why you don’t just upgrade to supported version. But they only keep the current versions of the updates online for you to download, so if you need really an old one you’re going to have to call support or I sometimes see people posting it for money. “Service Pack 3 for 2005, does somebody have a copy?” I see a lot of people are exchanging old updates that way.

Steve: Ok, interesting.

Carlos: Well, awesome, very good conversation, Robert. We do appreciate it and some good food for thought there. Shall we go ahead and do SQL Family?

Robert: Sounds good to me.

Steve: Ok, so Robert, how did you first get started with SQL Server?

Robert: You know, I’ve been in love with computers for as long as I can remember. But I didn’t go into IT straight away when I got out in the real world. I actually worked with the [term unclear – 36:58] disabled people for the first 10 years of my, that was my first career for 10 years. I worked with the [term unclear – 37:06] disabled people. But I was really longing for something different and still have that passion about computers. I’d started doing freelancing work, was doing Perl scripting and other types of web development. And eventually I decided, ok, it’s time to make the jump into the IT field for real. And so I started looking for position as a web developer. That’s the cool job that everybody wants though, right? Everybody come out of college wants to be a web developer. Competition was very stiff for web developers. I went to several interviews and ultimately I got offered a position as a database developer. And I thought, ok, I’ll do this for awhile and try to work make my way into web development world there. I worked for that company for 6 years and over the years, they were a very small company and their needs change and sometimes I would do web development. Sometimes I would do application developments, sometimes I would do database development. But we came to a point three years into my tenure there where our DBA had left the company 6 months earlier and they had never filled his role. And they were telling me that they thought they were at a point where they could make me a web developer full time. And at that point I realized I’d no longer want to be a web developer. I really enjoyed working with the databases. So I made a counter proposal, told them, at that time I was the head of the database development team. So I proposed that I take on the role of a DBA and still continue being the head of the database development team instead of doing web development full time. So that’s how I got my first DBA position. So that’s how I really got into it.

Carlos: Yeah, funny how things change, right, over time.

Steve: That point of enlightenment when you realized you don’t want to be the web developer. I can remember a similar point in my career as well. It’s an interesting time.

Robert: I like to tell people I’m a recovering web developer.

Steve: Oh, I like that.

Carlos: Now, of all the experience you’ve had in SQL Server and SQL Server is changing all the time. But if there’s one thing you could change about SQL Server what would it be?

Robert: What I would really like to do is there is some really horrendous features in there that I would love to see go away.

Carlos: Just take them out and shot.

Robert: Right. But that’s not likely to happen. If I had to narrow it down to just one thing, I would say get rid of all the features that have been deprecated for years.

Carlos: Pull the plug already. Robert: I mean as much as I love database mirroring. I mean I wrote a book on database mirroring. Get rid of it. It has been deprecated for much longer than things are supposed to be deprecated. I mean there’s an extremely long list of things that have been deprecated since 2008, 2012. It is like get rid of these things already. We have people that are still using them and the logic they use is, they said that were going to get rid of it after two versions and they still haven’t, so it’s obvious that they are never going to. Why should I bother switching to the new stuff? People are still using the SQL 2000 system tables instead of the current catalog.

Steve: Right, right, so force deprecation. I like it. So what is the best piece of career advice that you’ve ever received?

Robert: I would say the best piece of career advice I ever received was that you have to be your own advocate. Early in my career this was something I really had a hard time with. I’ve never been one to go out and say, “Look what I did. Isn’t this great?” But one of the hard things of being a DBA is if you’re really good at your job nobody notices. People only notice the DBAs who are always fighting fires. If things are always breaking and you’re constantly in reactive mode trying to fix, people are like, “Oh, he is a great DBA because he is always fixing out problems.” And then there’s the guy who’s, his servers are not experiencing these problems because he is proactive and he prevents things from happening, so people don’t see all the work he does. And so the really good DBA, really proactive one, gets overlooked because everybody else in the company doesn’t see the fires that you’re preventing. And so they think you just sit back there and do nothing all day. Learning to speak up for myself and to publicize the things I do definitely was one of the best things I had to learn how to do.

Steve: Yeah, interesting. Yeah, I can see that how the firefighter reactionary DBA can say I fixed these ten crisis situations, where the proactive DBA it can be more challenging to say that I prevented these ten could be crisis situations, right? That’s really what it comes down to. That’s the difference between the reactionary and proactive. Ok, cool.

Carlos: Robert, our last question for you today. If you could have one superhero power what would it be and why do you want it?

Robert: If I could have one superhero power I want to be able to freeze time.

Carlos: Oh, freeze time, so almost like be able to walk through it where nobody else is moving type of thing.

Robert: Absolutely. Time is frozen for everybody else that I may be able to walk about and do things.

Carlos: Yeah, very handy when the boss is in your cube, right? When is the database is going to be up?

Robert: Exactly, so I think goes down and people bugging you, you could freeze time and get it fixed and then unfreeze and say, “It is fixed.” “What do you mean?” I think the challenge with that power though would be keeping it in the spectrum of superhero power as opposed to super villain power.

Steve: Yes, very good point. So many of those types of powers could be abused easily.

Carlos: That’s right. Well, awesome. Robert, thanks so much for coming on the program today we do appreciate it.

Robert: Thanks for having me guys.

Carlos: It’s been great.

Episode 111: How do you prepare for conferences?

Sometimes just getting approval to attend a conference is a challenge, but getting the green light isn’t the only thing you should plan for.  Sure, you want to learn some things–but a google search can teach you too.  In fact this week we are hearing about new features and functionality from the Ignite event–and I am catching all of it in my bunny slippers.

Why attend a conference then?

Conference season is upon us and Steve and I share some thoughts on why we attend conferences and some of the strategies we have used in the past.

Show Notes: SQLDP TS EP111

 

Episode Quotes

“There is an evolution from the time you go to your first one… after 10 you’re maybe a little bit more of a veteran there. But that doesn’t mean there are still not preparation steps.”

“We should all be looking to get something out of the conference and not just, ‘Well, I want to go there and learn something.’”

“Plan ahead, sign up for what you can ahead of time but also be ready to adapt to the last minute if things change.”

“We want to reach out to these people but we want to be careful about how we approach them and talk about something positive.”

“If you’re just trying to spam people that’s a different objective and you’ll be in a different circle.”

Listen to Learn

02:45   Today’s Topic: Conference Tips and Tricks
03:48   The right mindset in going to conferences
04:27   Find your people
05:11   Connect through social media postings
07:12   Check your plans and papers
10:34   Goals: Why are you attending the conference?
12:13   Finding your people, scenarios of networking with people
16:46   Never dine alone
20:03   Right place, right time, valuable personal interaction
20:35   Meeting Famous People: Do’s and Don’ts
30:02   What to do after the whole conference is over? Whose circle do I want to get into?

 

Transcription: How do you prepare for conferences?

*Untranscribed introductory portion*

Carlos: Companeros welcome to another episode, Episode 111. It’s good to be with you today.

Steve: Hey Carlos, this episode is on conference tips and tricks. How are you doing?

Carlos: Hi, I’m doing well. Thank you. Yes, conference season is upon us and everybody is talking about conferences and we figured, hey we have this podcast so why don’t we join the conversation?

Steve: Yup. Alright, but before we jump into that, do we have any companero shout outs?

Carlos: We do. Shout out to Douglas Kemp, he chimed in says, he enjoys the show and normally listens the day they come out. Of course we appreciate, let us know that there is somebody on the other side listening.

Steve: Yup, sounds good. So we also had one from Russell Johnson and he chimed in about Episode 109 on certifications. What he said was one of the things that he uses is the Brain Dump sites sort of the questions that people have remembered or written down or copied from the test to see what he gets wrong. Meaning, he had try them out and see what’s wrong and then he goes and learns it. Part of the point that he made on that was it’s not just about memorizing those one’s that you don’t know. It’s about taking it as an item that you don’t know and you’ve got to go and learn it.

Carlos: Exactly, and that’s always the balance, right? That’s always a tough balance because you just wanted to get to the certification, and sometimes you’re willing to push learning off to the side to make that happen. As you can get that review or get feedback, learning is always good thing and we all need to learn, continue to learn. Ok, so when this episode comes out, of course we are talking about conferences. I’m sure the Companero Conference will come out but you have today, today is the last registration day to register for Companero Conference so you’re kind of a bit close companeros. Let me just say, you haven’t registered yet.

Steve: Sign up now, or forever hold your peace.

Carlos: Forever hold your peace, that’s right.

Steve: Or until next time.

Carlos: Don’t say that, Steve.

Steve: No, we want to people to sign up now and next time I guess what I’m trying to say.

Carlos: There you go. Hey, maybe you’re 2 for 1 special. Ok, there you go. The shownotes for today’s episode are at sqldatapartners.com/conferencetips.

Steve: Or at sqldatapartners.com/111.

Carlos: So ultimately, again as you mentioned our conversation is around preparing for conferences and kind of getting the most out of conferences. I think there is an evolution from the time you go to your first one. And now you have, you know that first few then maybe you get 5 under your belt then you get 10 under your belt. I think after 10 you’re maybe a little bit more of a veteran there. But that doesn’t mean there are still not preparation steps. And as we were going through this and comparing notes before we started recording I think we recognize that there is some changes. Not every conference I’m going to prepare for the same way and some of the outcomes that I’m looking for in a conference are going to vary from year to year even if it’s the same conference.

Steve: Right, and I think the things we cover here, I mean they could apply to any conference whether it’s a weeklong conference that you have to go overseas for or whether it’s a SQL Saturday in your own backyard.

Carlos: That’s right. I think we should all be looking to get something out of the conference and not just, “Well, I want to go there and learn something.” I don’t think we are maximizing the value of the conference if that is all that we’re there for.

Steve: Right, and I would hope that everyone is there to actually get something out of it besides just getting out of the offices as well.

Carlos: Exactly! Oh man. Yeah, because if you have that mindset you’ll probably not listen to this podcast.

Steve: Oh yeah, exactly.

Carlos: Ok so for preparation. Preparation is before the event, when that preparation starts again can vary. But one of the things that I’m looking at, depending on what the conference is, but basically the people that you want to meet. Who is it that you’re looking to interact with? So these could be speakers, so you’re looking over session material and you’re like, “Oh, that’s interesting, I want to chat with that speaker.” But if they are posting social media information it’s basically they’re or connecting with those people who you think will be attending, and that you want to further or connect with or network with at the event.

Steve: Right, and I think part of that social media, the side to figuring out who’s going via the social media is also posting that you’re going and things that you’re doing and other people may reach out to you as well.

Carlos: Sure, for example Companero Conference we’re using the #compaconf just to let people know that you’re there. You may end up on other people’s radar as well. And social media is that great advantage because it allows you to connect with others or find out something about them before you actually get there and potentially even give you something to talk about. Like, “Oh, hey, yeah I saw your Twitter profile and you like to do…” whatever it might be.

Steve: I remember at PASS Summit last year, I posted that I was going and somebody who uses Database Health Monitor and he had been using it for quite a while had reached to me. Wanted to meet up and ask me some questions, so we met up and chatted then I ended up running into him probably four more times just as we crossed paths throughout the rest of the conference.

Carlos: Yeah, that’s an interesting perspective. Of course, we are going to flip it out and kind of look at his perspective, so he is a user of a piece of software. He reached out to the creator of that software and was able to have a conversation with him at a conference. That’s pretty valuable, right? Just a little social media like, “Hey, I’d love to connect.” Now, again, it takes two to tango and not every tweet to all the people are going to go responded to although it is amazing how available some of these people are. But again, kind of putting yourself out there a little bit and trying to connect you will be surprised who you can chat with.

Steve: Oh yeah, everyone is there just like you usually looking for someone to chat with.

Carlos: Exactly. So the next one, checking your plans and papers.

Steve: Oh yeah, this is one, you want to make sure ahead of time you’ve had everything is lined up and ready. And I have a sort of an embarrassing story to share on this. Hopefully it doesn’t prevent me from getting invited to any overseas SQL Satrudays in the future or out of the country SQL Saturdays. But a couple of years ago, it was around the time I was working on the Database Corruption Challenge, and I had plans to go to SQL Saturday in Vancouver, British Columbia, Canada which I live just across the border from there and it’s like usually about an hour drive with traffic so I thought, “No big deal. It’s all good.” I want to go and I was a speaker, and I was going up Friday night, and Randolph West who is a friend, he’s been on the show. He and I had a planned to meet up and chat because we knew each other through the Corruption Challenge. I man had the whole weekend booked out with SQL Saturday and dinners and whatnot. It turned out that when I got in my car and I got to the border, the customs official from Canada pointed out that my passport had expired.

Carlos: Wouldn’t let you leave, Steve?

Steve: Wouldn’t let me in, which is interesting because they turn you around at that point and you go back. This sort of this no man’s land between the two borders once you’ve left with the U.S. and/or Canada. I couldn’t get into Canada but then I went back to the U.S. and they gave me even more grief about my passport being expired. But they eventually let me in, and then I had to contact everyone at the event to say, “Guys, I screwed up, my passport expired”, and I didn’t realized it. I didn’t get to go to the event. I didn’t get to meet Randolph that time. I ended up talking with him in a later time, and then Mike fall. He eventually jumped in and did the last minute session to cover the session that I couldn’t do because I was not allowed in the country at that point.

Carlos: Oh man. Yeah, that’s maybe probably an extreme example but I think even like your hotel, summit obviously is the behemoth conference that it is. There are lots of other after parties or events. They have the game event this year, so just making sure that you have those reservations ahead of time because if you wait to the last minute like, “Oh yeah, I’ll kind of wait and see what happens.” The tickets will be gone and then you’ll be left like, “Well, now what do I do?”

Steve: But the flip side of that is if you do wait for the last minute look for things that are coming up and we’ll cover that a little bit later, but plan ahead, sign up for what you can ahead of time but also be ready to adapt to the last minute if things change.

Carlos: And this is kind of a cultural issue almost in our conferences. And I think summit particularly where the number of people that just go. I’m not trying to bad mouth, it’s not negative per se, but a lot of people just go just to hang out. They are not necessarily trying to take away sometimes the same things that we’re going to be talking about here because our next item is goals, right? What is the goal of the conference? Why are you attending? For some people it’s just, I just want to get like a family reunion of sorts. Just get together talk with other and whatever happens happens. For others, that may not be the case. We want to get out with a specific goal. I can actually remember the first summit I went to which I think was 2013 it was in Charlotte, so I’m on the East Coast. Like, hey you have to get down to this East Coast but other than it being summit and everybody kept talking how great it was, I had no purpose in going. That kind of reflected in what I then chose to do at the event because I wasn’t prepared, I hadn’t thought about it. I was king of constantly winging it if you will. You know, again, there are pluses and minuses to that tact and it’s really just kind of more of a cultural fit there.

Steve: Yup. You know that was my first summit as well, and I didn’t have a lot of idea of why I wanted to be there. I know, I’d publish the book on common table expressions just before that and the publisher had a booth there and I did a book signing. But that was really kind of the only non-summit type, I mean the only sort of extracurricular type activity that I did. I didn’t know about all these other events or things to and whatnot.

Carlos: Right. Now, this kind of then comes into the idea of, ok, well so now we’ve prepared or we fail to prepare and we’re now at the event. Now there are two modes of thinking a bit about this so we’re going to take both of them, and one approach is to find your people. So conferences are just getting bigger, right, at least we’re finding, SQL Saturdays I guess that exception. They are kind of that middle conference. We have a unique community there but for the most part conferences are like 5,000+ people. The idea that you’re going to network with 5,000 people over the course of three days is just ridiculous, so you need to find your people. Everyone is not your people. Some years that might change or the subject to the conference might determine who your people are going to be. I know and some cases I’ve had my people, “Hey, I want to get better at public speaking. Let me go find those people.” So the year I launched the podcast it was I need to connect with Microsoft people to get them to be guess on the show so that was what my focus was, connecting with some of those. And I spent probably most of my time in the Microsoft area just waiting for all the PMs and whatnot to come in because I want to introduce myself to every single product managers that was going to be there in that summit.

Steve: I can remember a couple of years ago when I did the Database Corruption Challenge, part of it was all the winners got a t-shirt, database corruption challenge t-shirt. And because a lot of them were from overseas and they were coming the past Summit I thought, well I just meet up with them and just make sure that I drop that off to them while I see them at some point. So sort of finding my people on that one was going out and finding all those participants and it as almost like an Easter egg hunt to find all these people that I had talked with before and say, “I’ll drop you a t-shirt when I see you there.” There’s only ten of them but it was a lot of work to find all ten of those people during the conference.

Carlos: Yes, I bet it was. And another way to do that is to be so bold as to officially organize something. Now we’ve seen people organize and the two that come to mind are Steve Jones and Andy Warren. It seems like they are always planning something and they have some specific niche. Another example is Andy Mallen, who we’ve had on the program as well. I believe it was last year or it may have been a year before but he organized an LGBT event. So again, kind of finding your people and he went so far as to actually throw out and event and say we’re hanging out. So that’s another way to kind of find your people.

Steve: And I think that there are so many things happening and so many people there that no matter what your interest are you’re going to be able to find other people that have similar interest in whatever it is that you’re doing.

Carlos: Exactly. But the idea is you have to decide who those people are going to be and then its effort to go and find them. Some of the other things that we will talk about, it’s fine to just go and kind of row with the flow but then you can’t then knock the conference for not providing what it is that you weren’t looking for when you really didn’t know.

Steve: Oh yeah, yeah, this is an interesting one because I can think of conferences that I used to go to like more than 5 years ago it always seems, like before I didn’t really understood the thing that I’d go to conference and kind of the conference get done and multi-day conference and I’ll always end up in a restaurant by myself or picking up some food, take it back to the hotel room. It was always just sort of like kind of boring at that point in the afterhours. I think that the whole never dining alone, there are so many opportunities to meet up with other people who are probably going to be dining alone as well and just get to know them a little bit. And I think like back you mentioned Steve Jones, and Andy Warren, last year they had just a dinner event where you show up and you meet people and they get like a group of six people together and everyone is going off to dinner together, and it was a lot of fun. I got to meet some people that I probably would have not met if it hadn’t been for that but it really sort of changes what you get out of the conference because you just get exposed to more people.

Carlos: A great example for me was I was at Ignite down at Atlanta. I was eating, met some folks down and say, “Hey, can I sit here?” Trying to strike a conversation and it just so happen most of them were getting up to leave as I was sitting down. So it was me and this other fellow, and then Lora Rogers who you may know. She is a SharePoint person she is very active in that community sits down at the table. I didn’t know her. We strike up a conversation, she is there, she wants to connect with other people. She is like, “Hey, can I sit with you?” You know, we start talking about SharePoint and I knew that I wanted to have a conversation on SharePoint and so I said, “Hey, are you interested in talking with me on my podcast about SharePoint?” And she said, “You know what I don’t think I’m the best person for that.” But she introduced me to Todd Klindt, and that’s how, and he came on the program and we had a great discussion with him. And so it was very interesting that again just in that conversation the connections that can be made if you’re willing to go out and then say hello and talk a little bit about yourself.

Steve: Yup, absolutely. And I think that the worst feeling at a conference I think is to be at that table by yourself or to be sit down at a table and realized that nobody at the whole table is talking. I always like to kind of break the ice a little bit and just say, “Hi, I’m Steve. Who are you?” and just get things going. It doesn’t always work but it helps a lot. Yeah, that’s the predicament wherein we’re mostly kind of introverts so nobody wants to go out on that But the reality is that most of us are scared to death. We sit down on that table, “Hi, I’m Carlos.” You know, it’s like, what’s going to happen now? And you have to embrace that awkwardness and as you mentioned it doesn’t always work out. But that’s ok because when it does it’s really nice.

Steve: Yup.

Carlos: And I guess that kind of leads us into right place, right time. I feel like this is a little bit the opposite of finding your own people, and maybe not the opposite but instead of planning and trying to find people this is more of the happy accident.

Steve: Oh yeah, and I think you’re going to run into people you know or people that you want to know that you would maybe never run into anywhere else. Like PASS Summit I think was last year I ran into a friend from Australia that I’d known online and through when I came back to the Corruption Challenge originally but I’d known, but we’d never actually really met in person before that. We got to sit down and talk business for like an hour and a half at the PASS Summit. We didn’t talk about technology at all. It was all about the consulting business, and it was the right place, the right time and hadn’t been plan we just bump into each other and sat down and chatted and it was awesome.

Carlos: Yeah, that’s right. So sometimes you have to know when to embrace that opportunity. It could come with a sponsor or with a speaker with another attendee, and so yet always tough to gage but if both of you are getting value on that conversation and it continues to go, you know, let that moment marinate for a little bit because getting to it would be very very difficult.

Steve: I think that so often at the conference it’s all around the schedule and getting that next session. I think that used to be the way I was but I’ve found that it’s always better to, unless it’s a session that you’re speaking at obviously or one that you really want to go to. It’s always better to maybe go in late or skip a session if there’s some personal interaction there that’s valuable because, I mean, you can always watch a recorded session earlier.

Carlos: That’s right, so the learning part is still available, right, that connection part may not be available again. Now when I met people, one of the things that I like to do, and I don’t do it for everybody. If you don’t see me viewing your profile don’t feel bad. But one of the things I like to do after I meet somebody and particularly one that I think I want to follow up with is then either connecting with or at least viewing the profile of that person on LinkedIn. One of the things I like about LinkedIn is it keeps that history. I can tell who, like where I was been and who’s looked at me so it makes that connection a little bit easier. I don’t think we get that same ability like on Facebook or on Twitter unless we’re going to follow them which again is another option. But with the information that’s available on LinkedIn I can know a bit more about that person from a business perspective which is generally what I am more focused on.

Steve: Yup. Yeah, and I think that connection so that you have a way to follow up later if you need can be incredibly valuable.

Carlos: That’s right. I guess I wouldn’t hesitate to follow people. Like, oh yeah we had conversation with that person, let me follow, and then again we’ll get to the after the conference we can make changes there.

Steve: But I think part of the worst case on that is when you connect with someone and they’re just pure marketing. Yeah, and they sort of just launch into you with their ad campaign and buy, buy, buy. I mean, that’s the wrong way to do it I think.

Carlos: That’s right. That’s always rough. In a that’s going to happen, you know, the value, the up side is so much bigger than a couple of people trying to spam you because you can always just disconnect with those people.

Steve: That’s true.

Carlos: You know, don’t limit your chances, which is going to happen, and again you just have to understand because it’s a social medium you’re going to have interactions that and not all of them are going to be exactly what you want. Another one, so putting work aside again can be tough particularly with a long conference particularly like summit. You’re going to do precons, that can be whole week. So checking in a certain time maybe put them in the schedule can be something but if you’re trying to balance the two you’re just hurting yourself in my mind.

Steve: Oh yeah, couple of examples there, I remember last year at PASS Summit we had a podcast recording that was scheduled and partway through the day I left, came back to my hotel room and we did the recording but that was something that I probably could have pushed off for a couple of days until we were done. But those kinds of things happen. The other thing I remember is a friend of mine at the conference. I saw him once and I didn’t see him ‘till like the end of the week, and I said, “Where you’ve been?” And he’s like, “Oh, but we’re having troubles back at the office.” And I’ve been pretty much the whole conference I’ve been in my hotel room working. And I just thought, wow, ok. I mean, obviously you got to deal with the issues that come up back home but hopefully you can get enough in place so you don’t have to do that because at that point the whole conference is kind of a waste.

Carlos: Right, and it’s funny, so there’s system down issues. That’s probably you can’t neglect but if your company has paid for you to go to the conference put your oath of office email on, right? I mean push back a little bit on that. Unless you get the “Thou shall do this”. It’s ok to say, you know what, yes I understand you having that problem I’ll come back to it. You have to say it nicely but put the tales out there so that they know, “Hey, I’m at this conference I’m trying to learn. I have the following goals associated with this conference. If it’s urgent enough to take me away from the goals that I’m working on right now then escalate, I’ll be happy to help you. Otherwise, try to push it off, and then the last piece that we have or topics for during the event is this idea of meeting famous people.

Steve: Hoo, how exciting.

Carlos: Yes, so more than likely and summit is again using that conference we have the superstars of the SQL Server community, and you’re going to bumped into them. So what do you do?

Steve: Or what don’t you do?

Carlos: What don’t you do, yes, maybe we will just start with that and admittedly this goes back, I’m remembering because I think this was probably now that I think about it, so Kendra Little published a post and this was probably just before the 2013 conference because I was reading about going to the conference and what to expect. You know, Kendra is one of those people that people would recognize and know the name. And she wrote an article about people coming up to her conferences and of course she has been putting out all this content but people will come up to her and just kind of say the weirdest things like, “You were shorter than I thought you were” or “Your hair is longer than I thought it would be” or whatever, right? And it’s like, huh. Now obviously we recognize or we feel like we know those people because they put out content. We read their blogs. We watch their videos. We feel like maybe we’re friends, and that’s fine and that’s great. We want to reach out to these people but we want to be careful about how we approach them and talk about something positive. You wouldn’t just go up to a stranger and say, “Wow, you are a lot shorter than I thought you would be.” To role reverse for a minute, that’s what you are, you are stranger and they are trying to make a connection, and if you insult them not a way to make connections.

Steve: Yeah, that’s interesting. I know I mean there are times that I’ve been to conferences and I think, “Oh, wow, I’ve seen that person online or on Twitter or whatever and I’ve been kind of like a little bit afraid to approach them.”

Carlos: Right. And we are all are, right?

Steve: Yeah, and I think that don’t be afraid but don’t be like super interrupting either.

Carlos: Exactly, and don’t expect to get an hour. You walk up to somebody and don’t expect him to hang out with you for the next hour. I think this goes back a little bit to the right time, right place. As you may just have to find a person at a point where they’re willing to socialize, right? If they are trying to get to a session for example all you may be able to do is say, “Hey, I’m Carlos. I love your stuff. Thank you for… it’s great to meet you.” That’s all you’ll get and then maybe in some other opportunity you’ll to do it a bit longer. I think some of those more social times. Again if you happen to find someone with an empty seat for a meal those are great times because that person is kind of bound at that point, and you can take your conversation in different places potentially.

Steve: Yup. Oh yeah, so then after the event and you’ve just come home, maybe you’ve recovered with some extra sleep in a day or two. What do you going to do next?

Carlos: Yeah, so this is I think where the most valuable nuggets come in. And of course the learning is important and we want to have goals and things that we will implement in our environments particularly for technical conferences. But one of the things I like to think about is, and again think about it from an outsiders perspective is whose circle do I want to try to get in to. Right, so assuming that I’ve reached to people on social media before, I’ve tried to find my people at the conference, now I’ve had interactions with them. If you haven’t had an interaction with them then you can just cross those people off your list. The idea of then getting into their circle now after the conference is not going to happen. But if you’ve had those interactions with your people, “Ok, now who do I want to keep with”, and then the idea of, “Ok, how do I lunch this conversation?” What things did we talk about at the conference, like the reference or maybe what additional questions do I have that would help begin the conversation.

Steve: Right, and with that I think though you always want to make sure that you’re respectful and you do it tactfully.

Carlos: You know, that’s right. And so I guess I’ve phrased that in a way of questions. What can I ask them? It’s the opposite of, “Hey, we met. I’m a great guy and I can do all these things and here is my résumé. What do you think about this?” That won’t help either. But it’s engagement process, asking the questions, “Hey, do you remember me?”

Steve: Yeah. “No, no, who are you, Carlos?”

Carlos: Then again if you can gage, you get one email, then you go for the second email. Again, how do I engage in a community, their community? When could we next interact? Things like that. Maybe it’s commenting on a blog or on a LinkedIn article or whatever it might be just to continue the engagement.

Steve: Yup, but not in a stalkery or spammy kind of way whatsoever.

Carlos: You know, that’s right. I mean, if you’re talking about establishing a relationship and getting inside of circle. If you’re just trying to spam people that’s a different objective and you’ll be in a different circle, right? In the do not contact file.

Steve: Yeah, and I remember one from awhile ago where somebody made contact. They had some technical questions, I helped them out. And the next thing you know, every tweet pumping their product was mentioning me on Twitter just trying to sort of call me out to get access to the handful of followers that I have. I thought, wow that really turned ugly fast.

Carlos: Block, right, you know.

Steve: Yeah.

Carlos: Yeah, that’s unfortunate. I guess we should remember right, that we are using social medium, we’re going to have interactions that we don’t care for. But we shouldn’t try not to have them because the upside is just too valuable, too great.
Steve: Yup.

Carlos: Anyway, companeros those are our ideas for conference preparations. Some experiences we’ve had along the way. Of course we have conference season is upon us and for those who are coming to the Companero Conference we hope that you’ll apply a little bit of this. We’re going to be doing some different things as far as the social component just from an organization perspective. In year one we don’t have a great way of putting out all the social information before hand but we will be creating a channel in the SQL Server community workspace in Slack to help communicate and some other things like that. We hope that you’ll continue to reach out and start chatting with other attendees. If you’re at the summit or any of the other conferences of course come up and say hello to us. We would love to engage and connect with you.

Steve: Yeah, we’re not famous enough yet to be completely overwhelmed by people saying “Hi”. So just come up and say “Hi” and tell us who you are and how you’ve come to know us?

Carlos: That’s right. But if you say, “Hey, that bald spot is a little bit bigger than I thought it would be”, you can just. Not the best opening line. Let me just throw that out there.

Steve: I’ll send those people your way, Carlos.

Carlos: Thanks, Steve. Ok, as always, companeros we want to hear from you. You can let us know. On our podcast page, we had a couple of engagements there. We haven’t had anybody lately. You can actually leave a message. We’ll probably introduce a new segment next week. We might try to a little bit more. The shownotes for today’s episode will be at sqldatapartners.com/conferencetips.

Steve: Or at sqldatapartners.com/111.

Carlos: And as always, you can reach out to us on social media. You can connect with me, I’m on LinkedIn @carloslchacon.

Steve: And I’m on LinkedIn @stevestedman and we’ll see you on the SQL trail.

Episode 110: How do I build trust with my team?

As you work with other people, you will experience resistance when you suggest changes and want to implement something new.  Why is this?

One podcast listener shared an experience where a problem was occurring.  They suggested a solution.  The idea was rejected.  No solution presented itself and the idea was submitted various times.  When nothing else was working, the idea was finally implemented.

The suggested change successfully helped the situation.  Victory!

Later, another issue arose and a similar suggestion was made.  The expectation was the suggestion would be well received; however, this was not the case.  The idea was still met with hostile resistance.

Have you faced a situation like this before?  Why wasn’t there a trust built because of the first recommendation?

Our conversation for today’s episode centers around the idea of building trust in the teams you work with.  As most of our listeners are in the data platform space, we thought it would be a good idea to reach outside our community to folks we might receive the most push-back from–developers.  We are happy to have Richard Campbell on the show with us today.  Richard is the co-host of the .NET rocks podcast and you might recognize him from one of his many Channel9 videos.

We chat with Richard about how we can build trust and how some common ways we attempt to show authority can backfire on us and actually cause more problems.  We talk about some of the ways those we trust have gone about earning our trust.  I know you’ll enjoy this episode.

 Episode Quotes

“Mandates are one thing, execution is another.”

“Being successful is about persuading people to want to do it in a way that is usable by everyone”

“Position or power actually undermines your ability to persuade.”

Listen to Learn

04:07 What makes up trust in a team?
05:07  DBA’s responsibility in keeping the data safe and aligning business goals
08:37 Setting team standards, collaboration, being flexible to the needs of the team
11:52 How to be effective and create influence by giving value to the people in the team?
13:35 Becoming a consultant that takes away the tension of the problem inside the team
15:20 Understanding the whole person, the whole effort, the whole team
17:01 Attitudes when encountering failures
19:35 Other ways on how do you save yourself from quitting?
20:06 Why lunch is the most powerful moment of the workday in terms of improving trust?
25:30 Understanding the other team’s workflow
27:00 Discussion on best practice, showing the work, and getting an effective team
31:45 Team building activities and time together in strengthening the team
38:28 No level of position or power enhances the ability to persuade people to do something
40:46 SQL Family questions

About Richard Campbell

Richard the co-host of popular podcasts .NET Rocks (www.dotnetrocks.com) and RunAs Radio (www.runasradio.com).  Richard  is the founder of the Humanitarian Toolbox (www.htbox.org), an organization designed to let developers around the world donate their skills to disaster relief organizations by building open source software.

Today Richard is a consultant and advisor to a number of successful technology firms as well as the co-owner and content planner of the DevIntersection (www.devintersection.com) group of conferences.

Transcription : How do I build trust with my team?

*Untranscribed introductory portion*

Carlos: Richard, welcome to the program.

Richard: Thanks guys. I’m happy to be on board. Thanks for inviting me.

Carlos: Yes, all the other podcast that you’re hosting and have been on we’re happy that you came to our side of the pond and if you will and willing to chat with us a little bit today.

Richard: Hey, it’s nice to just be a guest because I know from some experience that hosting is hard work. You got to remember and keep track of a lot of things. I just have to answer questions like this is luxury.

Steve: Nice.

Carlos: How many podcast episodes would you say you’ve done?

Richard: Like little over 2,000.

Carlos: Wow, I was thinking closer to 1,200.

Richard: There are 1,465 .NET Rocks! episodes. But I came on board at show 100 so there is 1,300 and something. There are 545 RunAs episodes so that gets us up into the 1,800 range, and there is about 150 Tablet Show episodes.

Carlos: Oh, there you go. That’s right, I forgotten.

Richard: Yeah, so we are coming right around 2000. Now the Tablet Show was this little interval in 2011 to 2014 when it didn’t seem like .NET rock that much, and we hedging our bet but we were still doing .NET Rocks! but we are also building another show, just in case because it seems like Microsoft was moving away from .NET for awhile there. And then it was clear they weren’t, we just rolled the Tablet Show back into .NET Rocks!.

Steve: Ok, interesting.

Carlos: Well then we’re happy to announce this is your 2000th episode.

Richard: Yeah, ok, I’ll let you have that number. There you go.

Carlos: Well, so Richard, I think many listeners are going to know from the .NET Rock show. We actually have and one of the reasons we want to have you on the show today was a listener chimed in and had a question about trust and we thought that we might role play a little or talk a little bit about how this idea of trust works in teams in our environment. The reader actually goes to say, “I’ve been listening to your podcast, collecting best practices and I’m the DBA in an organization. I’ve only been there in a couple of months.” But they had the SQL Server, it was having problems. I was recommending changes to the SQL Server based on best practices, right, things we’ve been talking about in our program and others. The developers were like, “No, no”, because they’ve kind of managed the database, were responsible for that environment. Well, finally they relented because they were having so many problems and they had an improvement in that server.

Richard: Wow.

Carlos: That’s right, so everybody goes happy. Well, then he said, “Ok, well hey let’s adopt this as best practice and let’s go put this on server #2.” And they say, “Whoah, whoah, whoah. Time out!”

Richard: Don’t get crazy now.

Carlos: That’s right, and so his question was if I was able to show value and demonstrate that I could be of assistance why didn’t that apply to a more general trust of me and my recommendations ultimately. And so this kind of becomes then the basis of our conversation today, right, building trust among teams and particularly our audience is our SQL Server folks, and how they can work with developers like yourself to build that trust and to kind of move the organization forward. And so I guess we’ll start there and what do you think makes up trust in a team.

Richard: Well, looking at your particular scenario my concern always is folks look at one win and call it random chance. It kind of takes three wins before folks really like, “Hey you keep being right.” Like maybe there is an actual trend here.
Carlos: Sure. There is that saying that I like, “A data point of one does not a trend to make.”

Richard: Yeah, right. That’s kind of a narrow curve you’re looking out there, so it’s usual to build more points. I think it takes a lot of patience to be kind enough to allow people to learn themselves. So let them collect empirical data around their own mistakes. The challenge is to sit down your hands and not say, “I told you so.” At the same time I think it is also important, I think it is a trap we followed to is tech people is to keep talking about the practices around the technology rather than practices around them. The responsibility of a DBA is to protect and curate data irrespective of the technology involved. In the end this is an asset of the business. The developers are building software to allow us to bring that data in and perhaps to do some analysis on it, but ultimately the hat you really wear is the one that says, “I am responsible for making sure this data is safe and thriving and useful.”

Steve: Yup, and I think with that data that you’re keeping safe and thriving and useful is oftentimes considered by many of the organizations like their crown jewels. I mean without that they would have significant business impact if it was damaged or gone.

Richard: I hope that folks see that although a lot of folks don’t. That’s where, you know, when you talk about breaking down walls between teams. It’s not talking about technology; it’s talking about business goals, and so being able to speak in those terms so that people get aligned. When we really talk about that alignment we’re all pulling in the same way. It’s that we recognize, “Hey, our data storage is worthless without your software being able to feed into them.” So I don’t want to be an obstacle to your ability to put that data in. That data is only valuable if it’s usable and it’s safe so that’s why we have security and why we care about organization of the data and whatever form that looks like. It’s about making sure that data is actually valuable. You can’t succeed as a DBA without allowing your developers to be productive, that’s a necessary part of the equation. But vice versa if they are writing software that doesn’t store data effectively their software is not that useful either.

Carlos: We’ve talked about this on the show before kind of going from the CIO perspective or the managers they want the technology team to understand that the business value, what problems are we trying to solve and I think to you point sometimes we get sidetrack a bit by the technology and fighting over. You know, what we are or not going to implement when really we should be aligning what we do with those business goals.

Richard: And often we have pain too, scars. We’ve had a data corruption, or we’ve had a failed recovery, or we’ve had a badly behave piece of software where we… on a table. And so you can be a little jumpy.

Steve: Yup and I think some of that too comes from history, and I think with the example of this listener I think it was someone who was new to an organization. They probably had someone there before, I don’t know anything about the background but there is obviously some reason the developers were a little bit shy of making changes. Maybe they had a bad experience before that and it might take a lot of successful wins before that trust will be regained.

Richard: Yeah, and one win does not a trend made. Like, congratulations, you are walking the path but you got to keep walking it. And at least now you get to refer back to some success that’s already happened in the organization, some talking points. Blanket concepts around overall practices, those take longer to grow, and I think especially development organizations, you need to look at that from the perspective of how do you folks normally set standards around your practices because often we have a lot of teams where there isn’t a standard set of development practices. Essentially you’re going have to work from team to team to see how they like to interact, how they want to work with data, how they want to work with the system as a whole and essentially tailor a data access solutions to the team.

Carlos: So that sounds like something that might have to be, I don’t know, maybe not mandated but ultimately you have to be a little bit flexible. I guess maybe the first component of trust is being flexible to the needs of the team and ultimately being, which maybe against your traditional idea of a DBA, but being flexible to their needs rather than being quite so rigid in certain instances.

Richard: I think you need to be aware of how those teams currently work. If you do have an organization with say a strong enterprise architect group that does want a press standards upon everyone then talking to an individual team about a standard doesn’t make sense. You do want to go talk to those architects about how they developed those standards, how those measurements have done and how they are implemented so that you work within the system rather than inventing something new. And if it is team based, if we’ve got an environment where “agile” or we’ve got a variety of teams working in different ways there’s many ways that happens and works that way then it’s more of a team based. So I think there should be an awareness of how everybody works makes a huge difference in your ability to convince one way the other. I really want to emphasize this idea and you said this like, “Mandates are one thing, execution is another.” I’ve seen plenty of cases where we have a set of documents of this is how we build software and this is how we interact with data except for that part where nobody actually writes that code that way, those are great documents. You’ve got to actually look at how the organization actually functions as well as what it says. And being successful is about persuading people to want to do it in a way that is usable by everyone. That they are productive, you are productive and you achieve that real goal of delivering the data, so I would much rather persuade than mandate.

Carlos: Right. I think and so we kind of almost get into a soft skills discussion because this is again not about technology but this is about how to make friends and influence people.

Richard: I don’t disagree and I’ve done this long enough now in one form or another. It would basically come into conclusion that there is no situation where the technology can’t do this. This is invariably can the team do it? Yeah, this is marriage counseling.

Carlos: As a database administrator we think that we have kind of add some value to the team. First thing we want to do in establishing trust is really just getting away of the land and asking the questions, what are the problems people having, how do they go about doing things and then determining whether or not you can influence or make a change in some of those processes.

Richard: Absolutely, and so this is an old Stephen Covey’s 7 Habits of Highly Effective People thing. If you want to change anything, you want to influence anyone; first you have to understand before you’re going to be understood. So spending that time understanding how the team actually works. What motivates them, how they build their code, how they interact with data first and foremost that’s the first step because once you have a grip on that you can finally speak a language about how data is handled inside of the team and reflect that back on how you see it arriving in the database. That understanding is a step in trust too. The first hurdle to get over is this idea that they matter, and that they know you see them as people that matter.

Carlos: And that they are more important than the problem. It’s just kind of an area where the internet kind of fails us in the sense of we can publish this information out there, best practices, this is the way you “should” be doing things. But then we try to force that on people and say, “Hey, this is crazy. This is the worst environment I’ve ever seen. How do you guys work like this?” You’re not making any friends there.

Richard: Well, I’ve seen plenty of consultants where their names actually come from con game and insult. It is super easy to tear down other people’s work. The reality is nobody goes to work with intent to ruin things like today is the day I’ll make a mess of everything. That’s not the goal. I start with the fact that people generally have good intentions and then make the best decisions they could with the information they had at that time. Invariably when we look back we have more information and so we’re always going to see it differently. But if you recognize that you made the best decision with what you knew at that time and the constraints that you’re working in, some of which are going to be visible and some of which are not, then you’re a little bit kinder. I think the number one thing I’ve said most of the time when working with teams as an external is, “Guys, what I see here is very normal.” I have the good fortune to work with lots of different teams, there are always problems. The question is how well do we recognize them and what do we want to do about them. It’s just super normal. And just take that energy out of you’re not a disaster, this isn’t the worst thing I’ve ever seen. Like suck the air out of that talk about. We are all trying to do the best we could with what we had, here is where we are. We know we can get better. We can’t do it all at once. It doesn’t have to be immediately. These are incremental improvements overtime while maintaining the requirements, whatever they has to be productive and continue to ship features.

Carlos: Sure, right, so kind of taking out the Facebook approach if you will. Facebook everybody always post the best things, all the highlights and then you compare yourself to that and you’re like, “Ahh, I’m not good enough.”

Richard: No, and you always, out of the 150 store procedures you wrote you’re going to show me your best one but the other 149 exist. I think that’s part of this being a trusted team is a group of people that know we have good days and bad days. We have our strongest work and our weakest work but as a whole we deliver. Understanding the whole person, the whole effort, the whole team, that’s the value proposition. It’s hard to get there. We want to only look at shiny but we have to look at the hole.

Steve: And I think really the key to what you said there was understanding the team and understanding what the team is expecting. An example of that is doing consulting. We’ve work for a lot of different teams and some team I’ve work with, their number one goals is stability in uptime and if you’re pushing out a new feature and something goes wrong they would rather you abort the deployment of that new feature that means it’s going to give stability in uptime versus whether the new feature goes out on time. Others I’ve worked with, it’s all about we got to get the new feature out on time, and if you don’t get the new feature out, well there’s a problem.

Richard: Well, and often you’ll see those two positions inside the same team. The person who gets rewarded for shipping new features is really not worried about uptime and the person who gets rewarded for uptime features other worst thing that could happen to your uptime.

Steve: Right, and I think that you may get an organization and I’ve seen this where you say, “We try to do this but something unexpected happened. We rolled back and aborted that project or that deployment and here is the plan of how we’re going to take it on next time.” And some people respect that and other people look at it as, “Wow, you just slowed me down. You got in my way.”

Richard: Yeah, and you also find places where any kind of failure at all is such a big deal you never want to talk about it again. We cover up the fact that roll back happened.

Steve: Yup.

Carlos: Right, then we get into situations so analysis paralysis, so people are so afraid of making that mistake that they just don’t do it.

Richard: Yeah, deploying more software, making those updates would be the worst thing that could possibly happen which is just not worth the risk.

Carlos: Exactly.

Richard: I think it’s a challenge to get to a place, you know, we talked about modern environments. What we’re really pressing on is not know failures but minimum pain from failures and rapid recovery. Fail but don’t fail catastrophically that you’re able to be able to test it in the field. You know you can get back quickly. And it’s not a big deal to do that recovery that you know when you fail. It’s not like science pop out of the server going, “Opps.” How do you actually know there is a problem? If you have to wait until a customer complains then you know it’s been going on for way too long, so how are we measuring success or failure? Do we actually know when we’ve done it right?

Carlos: So thought around, let’s say again to our audience say database administrators we do have a couple that maybe managing other database administrators or team leads something along those lines but most of them I’m going to say maybe don’t have the authority if you will to start again say, “Hey, this is the way we should start doing things.”

Richard: And let’s be clear you are never going to have that authority. Get over it. It isn’t going to happen, so don’t count on it. Don’t wish for it, find other ways. You need to be able to get to that end goal of being able to have safer, reliable data that’s valuable to the organization. You’re never going to get it with the stick of power so you’re going to have to do it in a different way.

Carlos: Yeah, so I guess let’s talk about some of those other ways. Obviously we talked about reaching out when you’re on a team that is dysfunctional. I guess thoughts around how do you save yourself from just throwing up your hands and saying, “Yeah, we were not moving forward. I’m out.”

Richard: Well, and lots of people you either quit you go somewhere else, or you check out. Just tell me what you want. I will do whatever you want then nothing is my fault. And if you want to be more positive than that, more constructive than that
you have to start establishing some relationships and some trust that you have inside. And I would argue, you know, we described a very simple thing, understand the development process. Well, turns out that’s really hard to do because understanding development process needs interrupting a lot of people that are working, and they will be annoyed with you. You can’t do it during the workday. You have to do it around that and at the simplest level. The one window you’ve got is lunch. You know, when you give it workday the only time you really can talk to people where you won’t be take them away from work is lunch. And on top that, humans are hardwired to trust those they break bread with. Lunch is the most powerful moment of the workday in terms of improving trust. We just don’t take it seriously enough because we’re hungry.

Carlos: And how frequent it is that we meet people consulting for the first time over a meal. I think it would apply, I feel like maybe I have applied this in other scenarios but it takes them getting used to. It can be awkward, right, the first couple of times.

Richard: Absolutely, so just go and ask, “Can we have lunch together?” One of my measures of sort of the health of an overall team is to say who’s having lunch with whom. If devs are only having lunch with devs, and DBAs are only having lunch with DBAs, and ops guys have only lunch with ops guys. That’s an isolation. You actually want to encourage folks eating with each other. A lunch and learn where we talk through broad concepts on software and not a particular technology. More of where we want this application to go, where we want ops and data and dev to be together in a room. You want an interesting experience you put together three lunch and learns for each one of those team, all teams go to it as long as you’re buying pizza people are going to show up, right? Each team presents their view of an application, so developers are talking about the feature they’re building the stuff that they’ve deployed so far, the DBAs are talking about how data goes in, how data goes out, how data it’s currently organized, how it’s backed up and restored, operations talking about network traffic, how security works, how it goes to the firewall, because generally speaking the other team just don’t know what those guys do all day. I don’t understand why your job is hard. And so just having a moment where each of the other teams gets to see the folks that do the other piece and go, “Oh, that’s tricky.” The average dev has never seen a network diagram. Their network diagram is client, cloud, or internet server, maybe database behind it, so they reduce the operation guys entire job to a couple of lines. And the DBA guys to that cylinder on the drawing. There is more to it than that, and so actually starting to understand it a more deeply that’s the beginnings of trust, of some insight into the system.

Carlos: Can you think about all the effort we put in to just learning technology, you could take a lunch or a couple bunch of lunch, you mentioned three, to go to the pains of actually learning what your environment looks like.

Richard: And learning what your data storage environment looks like and what a recovery looks like. I want you to get to a place, I’m trying to banished the concept of that’s easy to everybody because everything is easy once you know how to do it. I want you to understand and remember these things are hard. And the side effect of that would you say to someone, if somebody ask you about a tech, oh that’s easy. You hurt yourself in two ways. One is you’ve undermined your value. Clearly you can do it because it’s not hard, and then same time you’ve also harmed the person you ask because this like, well you might be stupid because this is easy. Saying easy is just a terrible thing to say. There is no upside to it. Now, you don’t want to go, “Oh my god, that’s so terribly hard you’re going to die.” But it’s like this takes effort but it’s worth it. It’s a much more reasonable thing to say. All of our jobs take effort.

Steve: You know, I can remember being asked recently about a specific test that I have done and they phrase it as, this is what we want you to do. That’s going to be easy isn’t it? And how much does it take you to get it done? The administrator at a very high level without even exploring any of the details on it, and as soon as they said it’s going to be easy that makes it very challenging to be able to come back with anything that’s going to be realistic to their expectations.

Richard: Yeah, it’s terribly undermining, and not actually constructive. Either you lie or you lie, like you don’t have any good answers at this point. And even the process of estimating effectively is going to force people to actually understand more which often they don’t want to do. So just this encouragement to try and understand more to see what’s easy and what’s hard. I mean, there is another interesting truth which when we start poking at each other’s workflows there are skills that each team has that can make those other workflows simpler. Devs tend to solve stuff with code so often what they really understand a workflow on a DBA side there is like there’s some things we could do in code that would simplify that process for you now that I understand it. So there is often low hanging fruit between the teams that until we have some understanding of how everybody does their work, you don’t see that. We do have different view points on the same project. Once we see each other’s viewpoints you pick some simple things.

Steve: I think the key there is seeing the other person’s viewpoint, and almost like you’re in their shoes to understand what they are trying to say. And so often that’s what’s lacking. People just say this is what I’m going to do because I’m a DBA, or this is what we do because we are developers.

Richard: Well, we hide behind phrases like best practices. I don’t have to think about your need because I’ve got best practice. And again, it’s the other form of easy. You’re undermining the other person’s work, so why are you surprised when they resist you?

Carlos: Alright, that is an interesting question.

Richard: Let’s tackle it this way. We will only get to call it a best practice when all of us agree it is.

Carlos: Right, so the team itself has to agree on what the best practice is not you bringing in some external source and deciding for everyone, “Hey, this is the best practice.”

Richard: And it’s not like you have to invent from scratch. There is no problem with bringing something externally and saying, “Hey, I read this from this credible source. They recommend these things. How would this fit with our plan? Do you see value in it?”

Carlos: Right.

Richard: I hate coming with such certainty, and it’s a disease we have as technology people. We’ve had careers. We’ve had our time in school where we liked the result. We want the answer. We hated showing the work. Remember that, how much you hated showing the work. Here’s the bad news, when you actually want to influence people you got to show the work. And we keep coming with solutions and people want to understand the path. That’s why you present the best practice rather than a set of ideas that get to a result. One is showing the solution, the other one is showing the work. And without the work you can’t influence.

Steve: Yup, interesting. I can think of this specific example where I ran into that, where I was debating with a development team on the use of GUID as primary keys and the clustered index. Of course these were random GUIDs not sequential GUIDs, and I went and I found some blogpost from Paul Randall and Kimberly Tripp talking about this. And I went and examine them, wrote up an example that showed exactly what they’re article said and showed it to development team and then they came and said, “Oh, that’s just some blogger talking about it.” I mean everyone out there got an opinion on it. And the mistake I made was I didn’t set who Paul and Kimberly were in that example. They were just two unknown bloggers to this developer.
Richard: Not the guy who wrote DBCC.

Steve: Yeah, exactly. And I completely blew it on that one because I didn’t set the stage for where the data was coming from that I was using in my proof there.

Richard: And you can leverage that blogpost to map it to the problem, we’re seeing these issues. Let’s face it, like there is lots of controversy around randomly generated GUIDs. It’s not free. It has certain advantages for certain circumstances and it has a certain cost. Sort of upfront by looking it both, so the things to say, is this a cost we’re willing to pay to get this benefit. I do simply don’t believe in the one right way. I’ve had too many years writing too much software to actually believe there is a right way. There is the way that works at that time with the tools and skills that we have. We can get better but if there was one way through these journeys we’d all be taking it. The fact that a bunch of us have been successful in a bunch of different ways is sort of proof, and that no tool will save you. You take the same set of tools in a different problem and fail.

Carlos: Yeah, we look at that in data all the time, maybe like to make fun of the reporting tools. This new reporting tool is not going to help you unless you’ve got good plumbing, good infrastructure underneath. How many times do we try to do the same thing like what we’re talking about, the best practice or some other solution to say, “Hey, this new thing is what’s going to save us.”

Richard: Yeah. No, I want the big red button and then when I hit it, everything goes well.

Carlos: Exactly.

Richard: It’s a lovely dream, it’s just not true. In the end the tools cannot save you, it comes down to people in practices and since people make the practices it’s only the people. That’s all there is, you have to deal with the people that you’re working with. And if you understand them and see what their strengths are, value those strengths and put that puzzle together, put those pieces together so that everybody’s strengths are emphasize and their weaknesses are compensated for. That’s when you get an effective team.

Steve: So one of the things that I’ve seen in organizations were like management will try and get cross functional teams, operations and developers to work better together is they’ll do some kind of an outing that involves trust exercises and team building and that stuff. What do you think? That does kind of thing help or is that?

Richard: Yes. I’ll tell you why. It’s almost not the activity, it is the time together. In fact, the more mundane the activity, the better in some respects. Packing boxes at a food bank is not intellectual work. And the good news is then you don’t have to focus on it, maybe you’ll start talking to each other.

Steve: Or even if you took the whole group for an afternoon of bowling. It might have the same effect.

Richard: Yeah, same effect for that very much the same reason. And again, introducing food into the equation because it does plug into our brains to connect more, time together without the pressure on the work. Now, I’ve gone an angle on this as well. A few years ago help setup a charity called Humanitarian Toolbox where we build open source software for disaster relief organizations. In my mind it was, I know developers want to volunteer their time in a constructive way and use their skills. What I realized was it was also a very powerful team building exercise because yes we’re still writing codes but we are not writing the code from the office and we’re not on those deadlines. We’re in this volunteer mode. And so the dynamics of the team shift around and you sort of see each other differently. All of those exercises external to our regular day to day work, so the pressure of delivery, the pressure of a 40-hour work week are off and we’re in that different mode, that invariably can strengthen a team.

Steve: Ok.

Carlos: And I think some might complain now that you’re asking to do extra things.

Richard: Without a doubt, and this is part of understanding people. Look if you’ve got a 3-year old and a 5-year old at home you probably too busy to take a weekend off. Those kids need care. I think it is part of a good organization to recognize where our team members. How do we support them? On the other hand they love to have a weekend when they didn’t were constantly managing kids. So the fact that you’re offering some child minding as part of that equation at the bowling alley? Brilliant! How do I actually get the whole team to show up and be engaged? Well, if I understand them, I know what motivates them. I know what challenges they have so we can work on those things.

Carlos: And I think if we take the approach of I am making an investment in the team, I am trying to form those relationships and not, “Oh my gosh, another meeting,” just something I have to slug through that’s not helpful either.

Richard: The team building exercise will continue until morale… Really, we’re going to mandate that? One of the organizations I worked with, there was several people that were very bent out of shape about leftover food. They would take it down to the soup kitchen they would make sure utilize, and that to me was cute. Like there’s a strong social conscience in at least a portion of this team and so when you put energy into that, let that be more visible, it was very easy to say, “Hey, most of us care about this stuff. What would you think about doing an afternoon at the soup kitchen packing sandwiches?” It was already a thing where we’re
aware was important to many of our team. So validation there, add some time away, a good practice, doing a little corporate citizenship were our company did something collectively, you know, that was important. And the side of that was increased trust in the team, a shared set of values outside of today’s work. And some time to finding out who’s really good with plastic wrapping and who sucks at it.

Carlos: And then all the inside jokes that come along with that.

Richard: Exactly. You know, everyone of those inside jokes is a reminder of trust. Every nickname kindly given is a reminder of trust. That’s what good teams are built from.

Steve: Right, ok good stuff.

Carlos: Yeah, now I feel like we need to go to the soup kitchen, Steve.

Richard: Well, you’ve got to find the thing that works for your team, whatever that may be. Part of this is exploring the values that exist within your team outside of the day to day work. I’ve done Habitat for Humanity as well. My father was electrical engineer. I know how to do wiring so I was wildly productive and enjoyed myself. And everybody was surprised, I’m kind of a geeky guy. And then you give me a pair of lineman strippers and some rowmax, and I’m putting light switches in, surprise.

Carlos: Yeah, that’s right. And I think I guess, and again we’ve kind of been coming it from the vantage point while you may not have the ability to put those things together in your interviews and your annual interviews or when you’re asking for feedback, right, these are the type of things you should probably pushing for and making those suggestions. Even going back to the lunch and learn, I’m a huge fan, I don’t know why more organizations don’t do it, and I guess they look at this like a cost but again for a couple of pizzas. I mean, it’s well worth the investment.

Richard: Sure. You can do this ROI. My goal here is to make a stronger team; team is more productive for the price for a bit of food, it’s ridiculous. There is an argument about wanting to take a break, time away. But taking a break to think about something else is still a break. So it’s just what does that look like. I know a group of developers that what they did on their break was cycling. They loved riding their bikes. And so the fact that we would help organize for them a place for them go riding over lunch maybe huge difference to them. Again, valuing the distinctive aspects of those teams so that they can get stronger and get that level of validation that we care about the whole person, not just the 8 hours.

Carlos: There you go.

Steve: Yup. Good point.

Carlos: Yeah, Richard thanks for those insights. I mean so great points. Again, no easy button, right? But there are ways to go about.

Richard: And I guarantee you it is very persuasive in your own mind that someday I’ll get this promotion I’ll be able to make this happen. There is no level of power that will make it happen. CEOs all the time throw up team building exercises and nobody shows up. Just because you’re the boss doesn’t mean it’s just going to happen. You will never do this from a position or power. And I would argue that a position or power actually undermines your ability to persuade.

Steve: Yup. I would completely agree on that one. I’ve seen people get into that position or power where they believed they had the ability to command and totally loss the ability to persuade anyone to do anything.

Richard: They were probably more persuasive before they got the promotion because they actually try to persuade. Leaning on your title it’s never going to work. Just actually care enough to understand and help people understand you, you’ll get results. Nothing is fast, nothing is simple, it just takes time and it’s worth it because the improvements last. Once you get into this righteous cycle of a stronger team adding new members to the team, it’s easier to hire. People recognize a healthy, happy team and want to be a part of it. When you already have a culture that people value more people adapt to that culture. This is a righteous cycle, it gets easier over time. The first steps are the hardest steps but further down the path, you wonder how you were never there before.

Carlos: Right, I think using that title again, if the approach is different like you mentioned that idea, so there’s that saying, “People don’t care how much you know until they know how much you care”, kind of going back to the title right. If the approach then changes from caring about people to kind of getting things done, “Hey, I’m going to implement this thing”, then again you’ve lost, you’re kind of coming with the best practice again and you’re back to square one.

Richard: Well, it’s amazing how much people build like that because nobody uses this like, “Wow, that’s really well built. I’m sure you’re enjoying it all alone.”

Carlos: That’s right. Well, awesome. Richard, thank you so much. Shall we go ahead and do SQL Family now?
Richard: Absolutely.

Steve: Yeah, let’s do that. So what was your first experience using SQL Server?

Richard: Well, I’m the old guy, right, so I actually had the 421 installation on NT3 1, the non GUI version of NT. It was bomb proof, man. That was a solid piece of hardware. I think it was probably 486, maybe a 386 and just ran every day, all command line, total vintage; and the app I was writing was Visual Basic 2 with VBX library called SQL Sombrero.

Steve: Oh, I hadn’t heard that one.

Carlos: I like the name.

Richard: SQL Sombrero, that is like some vintage stuff. When we got to VB3 then we’d actually have RDO which was a data objects layer for talking to SQL Server, remote data objects but before that it was third party to actually get to SQL Server. But that’s old, that’s original client-server stuff, man. It was great great software to run. It was very straightforward but getting all the pieces put together and getting it working. That’s actually pre ODBC, that’s how old that is. It would be later that I would have ODBC errors.

Carlos: Oh man!

Steve: Yeah, there were no ODBC errors then.

Carlos: No ODBC.

Richard: There was no ODBC. Microsoft developed ODBC because they were trying to get traction to SQL space and everybody has their own drivers. The whole idea of ODBC was a universal driver. I had a customer once tell me, “I want you to get ODBC up all these machines because all it does is generate errors.”

Carlos: Wow, so all that time working with SQL Server and with .NET, right? SQL Server has changed quite a bit since those days, right?

Richard: Oh for sure, yeah.

Carlos: If there was one thing that you wish you could change about SQL Server, what would it be?

Richard: Well, even though I understand why .NET existed inside SQL Server I really wished I didn’t. I’m happy that it’s off by default. You should leave it that way. The number of scenarios where running .NET inside of SQL Server is a good idea are so vanishingly small. And of course Microsoft can never take it away because people have taken dependencies on it now. Like that’s going back to 2005, and their justification was this was a big data problem long before big data was really a thing. And today we would solve it in much more coherent ways but here we have this feature that just persists and it’s painful. It’s so dangerous. And anytime I have some a pop on the head, maybe I’ll just to turn on .NET. Don’t do it!

Carlos: Step away, step away.

Richard: Step away from the process, don’t touch it.

Steve: And I’ve never seen that work well by using .NET in the database.

Richard: It’s a very tricky thing to do and honestly I get so old now I don’t know it makes sense. There are so many better ways to solve problems that that was supposed to address than with .NET inside SQL Server.

Steve: Ok, so what is the best piece of career advice that you’ve ever received? You are not your customer. And I mean that from a perspective of your developers are not a DBA, and you’re a DBA, so just presume they are not thinking like you do. You’re not the customer, and so every time you think you know you’re wrong. You have to ask the question. You have to be influenced. You don’t use the software the same way the customer uses the software. You don’t operate the database the way that anybody else looks at the database. Your viewpoint is unique to your space. You might be able to talk to peers but your peers aren’t your customers either, so just stop fooling yourself. You have to ask questions.

Steve: Sounds like good advice.

Carlos: Richard, our last question for you today. If you can have one superhero power what would it be and why and why do you want it?

Richard: Well, I guess I want to fly. Doesn’t everybody want to fly? It’s almost a cut out
00:45:00 really. That way I could tell the airlines where to go and fly myself. I’m in metal tubes in the sky a lot and I’d like to do less of that and still get the chance to work with the people I get to work with, go to places I can go.

Carlos: There you go.

Steve: Isn’t that your superhero power Carlos, that you wanted?

Carlos: It is, yeah.

Richard: Fly or invisibility. I don’t want to do time travel because I don’t want to be responsible for anything, right? It’s really those three, that kind of great superpower.

Carlos: It depends on which plane I’m in, how much I can work with my laptop or my tablet right but that would be the one thing, the next thing to figure out once you can fly is how do I get to get some work done flying around.

Richard: Well, obviously that’s going to be enjoyable enough that we don’t care to do anything else. Never that simple.

Carlos: Well, awesome. Richard thank you so much for being with us, today.

Richard: Guys, tons of fun. Thanks so much for having me up.

Steve: Thanks Richard. It was a great time.

Richard: You bet.

Carlos: Yes, we do appreciate it.

Richard: My pleasure.

Episode 109: To Certify Or Not to Certify?

In episode 64 we interviewed Patrick Thomas, the program manager for the MCP program which includes certifications.  There continues to be quite a bit of discussion around certifications, so it only makes sense that we give our take on the matter.  Both Steve and I are certified, so this may give away some of our thoughts.

Certification is a tricky thing.  Much of the value you get out of a certification is dependent on where you on in your career and what your next step is going to be.  This is going to be different for every person, so this decision is one that only you can make.  This can make a decision on which certification to get and even when to get it a challenge.

Our conversation really revolves around what a certification will get you and what it won’t help you with.  Do you agree with our list? Leave your comments below.

Here is our list of Pros and Challenges with certifications

Pros
  1. Measure your skills
  2. Foot in the door
  3. Exposure
  4. Confidence

Challenges

  1. May not really measure your skills
  2. Not the only way to show skills
  3. Lots of prep work, with little way to gauge your progress
  4. Lateral Help
  5. Text Anxiety

Episode Quotes

“Having those certifications listed on your résumé may make you stand out as a little bit better.”

“It gives you exposure to more stuff that you may not have thought about trying in the past.”

“There are just a lot of features to know, and then try to create a test that can cover all of that is just daunting task.”

“The blogging… or the speaking at SQL Saturdays allowing you to create that portfolio, certifications are no longer the only way to do that.”

Listen to Learn

00:28   Introduction of the topic: to certify or not to certify
01:41   Companero shout outs
03:28   Updates about the upcoming Companero Conference
04:20   A little teaser of who will be next episode’s speaker
05:39   Show notes links
05:56   The pros of certification
06:10   Pro #1: Giving you an opportunity to measure your skills
08:30   Pro #2: Getting your foot on the door
10:51   Pro #3: Giving confidence to the technology workers and the employers
11:45   Pro #4: Exposure
12:28   The challenges of certification
12:37   Challenge #1: It may not help you measure your skills
13:48   Challenge #2: Your exam exposure and the employers expected exposure may differ
15:53   Challenge #3: Creating a test for too many features will be a very daunting task
18:53   Challenge #4: It’s not the only way to show your skills
22:18   challenge #5: There is a lot of prep work required that is very time consuming
26:16   Challenge #6: Lateral help
27:08   Challenge #7: Test anxiety

Transcription: To Certify Or Not to Certify?

*Untranscribed introductory portion*

Carlos: Hello companeros! Welcome back to Episode 109. It’s good to be with you again.

Steve: Yup, it’s good to be here. This week’s topic is to certify or not to certify.

Carlos: Yes, as we get back in here to the studio, Steve and I are chatting today. We kind of came with this topic. We’ve talked a bit about certification in the past but we thought we would explore this idea a little bit and kind of talk about some pros and cons of why you may or may not want to certify.

Steve: Yeah, I think what’s interesting with this is there are a lot of pros and a lot of cons and a lot of varying opinions out there depending on who you talked to.

Carlos: Sure, sure and maybe we’re going to squash some of this conversation but the reality is that you got to have them. I mean I have passed certification exams, you’ve passed certification exams. You know, we might gripe about them a bit.  Our conversation is not going to, I maybe used the wrong phrase there earlier, maybe not why you wouldn’t certify but maybe some of the challenges you will face around certification, that’s probably a better model there. I don’t think there is too many people out there saying, “Oh, don’t do that.”

Steve: Right, but before we get into the details there. Do we have any companero shout outs to mention this week?

Carlos: Yes, we do have a couple of companero shout outs, one coming from New York, so Alexander [name unclear – 1:53]. Sorry Alexander. I apologize. I should have practice that before we went on. But you know who you are, so Alexander is a long time listener. Got us after a little bit or he reached out to us when we had that little pause and encouraged us to get back on the air and to keep going, and so we appreciate that.

Steve:        That pause by the way was our summer vacation. It wasn’t like us giving up on it or anything.

Carlos: Yes, that’s true.

Steve: We also had a shout out from Aaron Hayes, and he just wanted to reach out and say that he appreciates our time for the podcast and he always listens to it on his commute and it keeps him up to date with some of the latest stuff in SQL Server. Thanks Aaron we appreciate that and Alexander.

Carlos: Yeah, we actually appreciate all those comments. Steve and I were talking before we started here. You know, we push this out into the internet, we laugh and we joke and we have a good time with it and then all we kind of see is download numbers. And so it’s nice to get just a little comment even if it’s that. We hopefully could sometimes engage in longer conversations but just to have people let us know that they are listening we appreciate it.

Steve: It’s definitely good. We don’t always know from those download numbers whether people are enjoying it or whether they are just thinking we’re crazy or what.

Carlos: Yeah, that’s right. Hey, check out what Carlos and Steve said in this, you’re never going to believe it. Oh boy! And we should note, so Aaron is actually going to join us at the Companero Conference.

Steve: Oh, speaking of which, what’s happening with that now?

Carlos:    It is coming up, and companeros you have two weeks left to register. So next episode comes out, that would be the 13th, September 27th is out cut off day for registration to just allow us to get ready. Yeah, so we hope to see you there. If you haven’t checked out the website – companeroconference.com, we will be down in Norfolk. If you are a regular listener to the podcast I think we’ve gone through the idea, we have a great lineup of speakers, we’re going to be doing peer conference. We have structured and unstructured time. We’re going to be going out on a cruise and so we think we have put together something that’s compelling and hope you’ll join us.

Steve: Alright, and this week it’s just you and I on the episode, Carlos. But next week who do we have that will be joining us?

Carlos: So next week I’m looking forward to Richard Cambell, so the Richard Cambell from the .NET Rocks podcast. Probably you’re familiar with them. He’s been in lots of different areas. You see him a lot on the Channel 9 as well. They are involved with the SQL Intersection or Dev Intersection Conference, and so he’s kind of all over the place. We had a user or a listener requested idea for building trust on teams and so I thought, hey let’s reach out to a non DBA, a developer type, and kind of get their take. I had a conversation, it was fascinating.

Steve: Yup, and I think that’s an episode that we’ve already recorded but it will be coming a week from now. I had a good time with that. It was interesting because he is probably the person that was on the episode who has had more podcast and experience than anyone we’ve had on any episode.

Carlos: Oh, no question.

Steve:  Because he’s done I think something like over 2000 podcast episodes.

Carlos: That’s right. In fact, we claimed that that episode was his 2000th episode. We claim that number. Yeah, that’s a lot of podcast. So for this episode on certification, if you want to get to the show notes, the show notes for today’s episode is sqldatapartners.com/certification.

Steve: Or sqldatapartners.com/109 for the episode number.

Carlos: Yeah, so I guess let’s go ahead and jump into that conversation, right. So we have this idea of to certify or some challenges that you face in certification. So let’s go ahead and start with the pros. Number one, certification is giving you an opportunity to measure your skills.

Steve: Yeah, it’s interesting because looking at that it’s one of those that even just preparing for it you can go through and see these are the things, the topics that maybe included on the exam. And you can go through and figure out, “Ok, where I am strong and where I am weak. What areas do I need to improve on?” It’s sort of a good way to build out your experience or build out your knowledge so that you’re a little bit more rounded on that specific certification.

Carlos: Yeah, that’s right. I mean, certainly you’re getting a broad exposure to all of the features and technologies that are in the database. Most of the time they could get sold in a book, you could take a peek there. I think some of the prep tools are getting a bit better to give you a better chance to get your hands on some of those things before kind of putting them in your environment.

Steve: I think an example of that for me was several years ago when I took the 70-461 exam. And I went through the prep material ahead of time, the whole XML in the database was a piece that I hadn’t done anything with at that point in time. But after doing the prep work I realized, wow there is a lot you can do there. Now, I’ve done a little bit with it since then but it’s not one of those things that I would have necessarily just stumbled across in my day to day work.

Carlos: Exactly. I think that kind of goes back to this idea, again we’ve talked a lot about on this program is sort of marriage of the technology and the business requirements, and that is as you begin to become more familiar what your business is looking for, the problems that they have, right. How can the technology help solve some of those problems? I think is you are able to combine those two then you have a very successful, you have a very interesting package there. You can be very successful as you begin to connect those dots.

Steve: Yup, yes indeed.

Carlos: So Pro #2 is getting your foot in the door.

Steve: Right, so what exactly do you mean by that?

Carlos: Yes, so I think a lot of times, and I knew this is the case for me particularly when you’re new, having the certification allows you to be able to say, “Well, hey, I’ve passed an exam. I kind of know what I’m talking about.” And so as you look for employment it just seems like particularly in the DBA space, the data platform space certifications are important to employers.

Steve: Yup, and I think that seems that’s an area that’s important at that time that you’re writing your résumé or preparing to jump to that next position. Just having those certifications listed on your résumé may make you stand out as a little bit better than someone else who doesn’t have those on their résumé.

Carlos: That’s right. I think it’s particularly prevalent or most needful. Let me try it again, sorry, Julien. It is especially important when you’re talking about career changes, right? If this is in technology so for example you’ve been a developer and you want to come over to data platform or networking, same thing and system administration. Or if you haven’t been in technology and you want to come in. You know, certifications are that way. Again, if nothing else just demonstrate that you have the vocabulary to be able to speak the language.

Steve: Yup, and I think an example of that, I worked with a guy who have been doing mostly like tech support type work but he wanted to do some more database thing so anytime there was a database project he could take he did it but then he wanted to prove that he knew enough about database so sort of jump out of that support role so he went and did the 70-461 exam. Took him maybe a year of prep work to get ready for that from where he was at and he eventually passed it and sort of prove that he could play in that area.

Carlos: Right. I think that’s maybe another point that we should make is confidence there. We didn’t have that on our list, maybe we should make that as another point. Even I remember going back and talking it in Episode 64 when we were talking with Patrick Thomas who was over the Microsoft Certified Professional Program at Microsoft, and that idea of they are trying to give confidence to the technology workers, not only the workers but also the employers. And so I think again, we’re going to circle back to this and sometimes those goals or what they think of the exams don’t always line up but it is important that both of them are looking to them and again want to gain that confidence. And then our last one which we kind of touched on a bit already I think is just exposure. You know, we mentioned getting into the new features. What’s there I think particularly now as new things begin to come out. What’s in the database that you’re not using and could you take advantage of it. I think exams help to do that really well.

Steve: Right, and I think I kind of jump to go on that under the measure your skill section with my story there on XML. But yeah, I mean it give you exposure to more stuff that you may not have thought about trying in the past.

Carlos: So those are our pros and interestingly enough we have a few more cons, or may not cons again, but challenges. We should change it to challenges. So as much as we talked about measuring your skills, our first challenge is that it doesn’t really help you measure your skills.

Steve: Right, and I think that might be better phrased as it may not help you measure your skills. I mean, the fact is there is some confusion out there and there’s things people can do to get the certification without really knowing what they are doing. For instance, people can cheat by going out and finding some of the actual exam questions online. I know after I took one of the exams there was a question that I wanted to look into a little bit more and I did some googling on it. And I find out after I had passed the exam that wow there is this exact questions that are published out there. And really if you have a good memorization skill you could go through all of those and memorize them and not know anything about the topic and passed the test. I hope that doesn’t happen very often but I’m sure it happens occasionally.

Carlos: Right, kind of along those same lines is the idea that we talked about employers wanting the certification, so what the exam is going to give you exposure to and what your employer thinks a certification is going to give you exposure to maybe slightly different. They are not always in harmony.

Steve: Yup. Now, I can think of an example there where I knew someone who was newer to SQL Server they had been working with it for a couple of years and they went and passed the 70-461 exam, and after passing that the employer’s take on it was that they could now do everything, anything with SQL Server, and that person could do all the skills of someone with like 20 years of experience. In a conversation with that person we discover that there is a lot of things that you do every day that aren’t part of that 70-461 exam test. Like everything that’s covered in 462 for instance.

Carlos: Yeah, and this is kind where the blessings and curses. Even when we think about SQL Server, right? It’s great for us because there are so many features that come with that but then when we talk about SQL Server we’re lumping in SSIS, SSAS, administration, performance. I mean there are just too many tools in there, and then from an employer’s perspective they are like, “Oh, SQL Server certified. You should now understand all of these things.” You’re like, “Well…” Yeah, it could be a bit challenging.

Steve: Yeah, and I think part of the key to that is really understanding what that certification has covered and being able to convey that in a way that whoever is looking at it, like your employer, understand what that means.

Carlos: Right. Another one that I added here and that is with the new versions, there are just too many features. The database now, I mean it can slice, it can dice, it can chop your onions, serve your veggies and all the other stuff as well. And so it’s interesting I think, and again we’ll see how those certifications come out but the 461, 462, 463 are still going back to 2012, 2014. Even there were two versions removed from that they haven’t quite abandoned that and I think it’s going to be challenging just with the sheer number of features that we have. I don’t know if certifications are going to change. Maybe we have new exams, smaller exams, pocket exams. I’m not sure how that will come out but I think there are just a lot of features to know, and then try to create a test that can cover all of that is just daunting task.

Steve: You know, I think some of those features that they included in the test are almost like features that the marketing team for SQL Server wants to have covered so the people know how to use the latest and greatest rather than knowing how to use what they need to use to get their job done. And those can be very demanding on occasion.

Carlos: Or the ones like you mentioned even the XML that kind of continue to get in there every version, then you’re like, “Really guys?” Having said that I guess, I should say full disclosure. I have actually participated in with an organization that reviews the tests and we go through, so on the actual exams and then on the practice test, practice exams. And writing test questions is really hard. And so you have four people, that at least four people, on this call with the moderator going over a question and all the answers and we’re debating back and forth, and we’re looking at documentation and trying to make sure the wording is just right and then you change something and you’re like, “Oh well then that’s going to affect this other thing. And what if they think of this and how they are going to interpret that.” It’s very difficult and I think sometimes we might complain a little bit about how the exam questionnaires are worded. But believe they are spending a lot of time and energy to try to get those questions just right.
Steve: Yeah, they certainly are. I think that we oftentimes forget that when we’re taking a test or when we’re practicing for a test.

Carlos: Sure, right, because I think as we get to later, when we’re in there sometime all that time and energy doesn’t help us and that can be frustrating.

Steve:  Yup. So another one of the cons that we have on our list is it’s not the only way to show your skill and that there are a lot of other things that you may be able to do and maybe even be able to do with less of the time investment to be able to show your skills. There’s whether it’s blogging, or speaking, or taking on a side project, or looking on open source project or any of those kind of things might be just as valuable in showing that you know what you’re doing versus taking a test.

Carlos: Yeah, that’s right. I mean, our pros was that idea of getting the foot into the door. Right, having that on your résumé, that’s not the only thing that you could put on your résumé to differentiate you. I think we’re going to continue to see it more obviously in the developer space. We’re seeing it a bit more but even like Github, right, so what your Github handle and allowing employers to go out there and looked at the code you’ve checked in, you’ve participated in, whatnot. That is another way to increase your profile and I think that to our advantage and it hasn’t been too far in the past that I’ve actually need to work through recruiters and like try to find a job. Oh man, what a hassle that it. Like I can almost, the back of my throat is like having this vile taste in it remembering what that process was like. Luckily, that’s changing a little bit and so it’s more of extending your network and trying to connect with other people. That will be the best and easiest way, maybe it’s not the easiest way to get a job, but obviously the most successful path. And so all of these other things like Steve mentioned, the blogging and whatnot or the speaking at SQL Saturdays allowing you to create that portfolio, certifications are no longer the only way to do that.

Steve: Right, and I think that may vary greatly with every employer too. I mean I had a job interview I remember several years ago where it’s like, “We don’t want to see your résumé. What we want to see is your Github profile.” And that was it, that’s all they wanted to look as a starting point. And at that point in time I hadn’t really done anything publicly on Github but I’ve done lots of other public stuff but I just completely struck out on that one. They didn’t care about certifications but then you go into another place and they do care about certifications. Certifications are the only thing, you got to be a little bit more well rounded there I think.

Carlos: Yeah, exactly. It’s not a one size fits all. And the other component there is what type of organization do I want to work for, right? I think that plays a role into how you build your portfolio. What’s going to be attractive to them is also attractive to you. That increase is a likelihood that you’re going to go there and be happy, and be able to contribute to the team.

Steve: And you know, it’s interesting that interview that they only wanted to see your Github profile. I was a bit disappointed when that happened but then I later learned more about it and realized I’m kind of glad that I didn’t get that job after I learned more. Another of the cons on the list is really around there is a lot of prep work required that is very time consuming. Now, it may be people are at a different skill levels or different points in their career that that time consuming part may vary. But I think that no matter who you are there is some prep work required to go in and make sure you know everything to be ready for that test. And for some people that might be a year of prep work if they are new, for others it might be a few days or week of review to be able to just get up to speed on that.

Carlos: Whoah, a few days! That’s pretty aggressive, Steve.

Steve: Well, I guess it depends like if someone has taken the exams over different versions for the last 15 years they may have just to catch up in latest things before going and taking that next exam.

Carlos:  Well, it’s funny that you say that because that actually happened to me on the 461 exam. I thought it’s writing SQL, like how hard could that be? So my challenge there was I actually really didn’t study. That was the problem I used to, “Ha, I’ll go in there and I’ll just do it.” And you have to have a 700 to pass and let’s just say my score was in the 600 somewhere, right. I was like, “Argghh”.

Steve: Let me guess, did the merge statements throw you off in the exam?

Carlos: There was a group of questions that I guess shall remain nameless that I think threw me off, that ended up spelling disaster there for me. But then I’m like, “Oh, ok I got a bone up on that.” So I guess I made the backwards mistake of instead of using the test material to find that out I used the actual exam to find that out.

Steve: Yeah. Which you know, I mean that’s one way to do it because oftentimes you can get a retake that’s free or not very expensive.

Carlos: Or reduced.

Steve: And with that, I mean not all the prep material is all that is prep up to be. I think that I did some work for a book publisher a few years ago. That was on the prep work for the DBA series, the 70-462. One of the criteria that they had for everyone who was contributing on that book and video series was that you could never have taken the 70-642 exam because for people who would actually taken the exam they sort of lead in to like giving away exact clues of here’s how you pass the test. So in order to get around the legal hurdle of getting sued in some way they said, “Well you just have to study what Microsoft says is going to be on the exam and then we write a book around that.” I did eventually after doing the work on that book. I did eventually go take the test and it was much easier to take after preparing that book or work on that video work for them but I can see that there were things that are very different in the exam versus what you sort of expect from reading some of the prep material ahead of time.

Carlos: Right. Yeah, that’s one of the challenges with the prep material. I’ve used the books before and it’s like they just go over everything and you’re like, “Huh…”

Steve: And they have to go over everything because you don’t exactly what’s going to be on the exam.

Carlos: Exactly, exactly, that’s right. And then the way they get worded and all the other stuff I guess that’s another hurdle, but just a sheer amount of material that you have to go over is a challenge. Let’s see, so the next one we have is lateral help.

Steve: So really that’ sort of saying that in most situations it’s probably not going to help you in your current position but it may help you if you are jumping somewhere.

Carlos: Right. Or you’re looking for something, either promotion. It seems like it’s a plus. I mean, it’s not to say that education is not good. We’re specifically kind of honing in on the certifications. Obviously listening to this podcast is a way to increase your knowledge and what’s in your skill set. But yeah, it seems and this is just a generalization but it seems in our experience it’s been, I want my certifications for my next job not so much for the one I’m currently in.

Steve: Right, so I think leads to another one on the con list is test anxiety. And I think that there is a lot of great DBAs and database programmers out there who are just awesome at what they do but they don’t always do well on test.

Carlos: And the format of the tests are changing. I mean, they are difficult, right, I mean to put things in order. It’s like A through J. It is knowing some of the syntax. It’s very very tricky. Now, this actually takes me back to Episode 10, the very very beginning, Thomas Frank, who runs a blog for college students. We actually chatted with him about some ideas of preparing for exams and kind of dealing with some that test anxiety, and so that was kind of interesting. You’ll have to forgive that episode that was one of the very very early ones so the audio quality is not the greatest there. But if that’s something that you’re thinking about it may be worth taking a listen to that Episode 10.
Steve: So another one of the cons as well is you can just keep taking the test until you pass, meaning as long as you’re willing to pay for it. I remember another exam that I’ve taken different from technology here was I’m a volunteer firefighter and EMT, and when I took the EMT Exam, that was way more stressful and way more detailed than anything I have done with the SQL Exam. But the criteria with that was if you don’t pass it you’ve got to wait a time period before you can retake it, so I was extremely motivated to make sure I pass it the first time. Whereas with the SQL Test, if you don’t pass it go back the next day or the day after that and take it again. It depends on your budget how often you can take it I guess.

Carlos: That’s right and luckily so there was I think was the 463 Exam, the Data Warehouse Exam. Oh man, I took that one at least three times, maybe four times, so good and bad. I mean, some of that again is just you try and balance what you need to know with what you have to get to pass the exam.

Steve: Yeah, and I think that and maybe that’s an approach you could take. You could just go take it and find out what you don’t know and then fill in the cracks. But also if you do it that way maybe you don’t enough and you just keep retaking it and filling in the cracks you’re really teaching yourself what you need to pass the test, not teaching yourself what you need to be a rockstar in that area for instance.

Carlos: Exactly, also true.

Steve: Which might be the way to do it for 463.

Carlos: Unless Data Warehouse is your thing, right?

Steve: Exactly, yes yes.

Carlos: Ok, so I guess the recap. Our pros, we have measure your skills, foot in the door, exposure and confidence.

Steve: And our cons were around it does not necessarily measure your skills, not the only way to show your skills, there is a lot of prep work that could be time consuming, it may help you if you’re jumping laterally but not in your current position, those with testing anxiety may have the challenge with it, and you can just keep retaking it until you pass.

Carlos: So there you go companeros, what do you think our list? Of course we are interested in your feedback. You can drop as a line on the show notes page which is going to be sqldatapartners.com/certification.

Steve: And you know just to follow up on all these. I mean as we’ve been through the pros and cons, and we hit the cons as our second half of the list basically. I mean overall I think it’s a good thing. I just think that there is a lot of positives and negatives with anything.

Carlos: Sure, and I think we change cons to challenges, right? So we are not saying that it’s a con to take the exam. Again, we’re both in there, so just challenges. Yes, so what do you think of that list. We would be interesting in hearing from you and comments that you have. Of course if you have other ideas or things we should be talking about on this podcast please let us know. You can reach out to us. I am on Linkedin @carloslchacon.

Steve: And I’m on Linkedin as stevestedman. And we’ll see you on the SQL trail.

Episode 108: The Future of the Relational Database

After a brief hiatus, we are back on the air to continue the conversation and let me tell you–we have a great conversation lined up for this episode.  The discussion around what will happen to the relational database, and by extension us as administrators continues to get quite a bit of traction.  Even within SQL Server, we are starting to see more features that don’t fit the traditional relational mode and a podcast listener inquired about getting our thoughts.  As I thought about a guest for this episode, I didn’t want to get someone tied to a product.  They, like me, would be biased and I wanted to get someone a bit removed from the situation.

Our guest today is Andrew Snodgrass, the research vice president at Directions and we chat about the future of the relational database and what the future of the data environment we manage might look like.  I hope you will find his insights valuable as an outsider.  While we don’t get into the specifics of what databases are mostly like to be around, Andrew does give us administrator some ideas on what technologies we should start exploring.

What are your thoughts around the future of the relational database?  Join the conversation and let us know!

 Episode Quotes

“So these things have come out as a natural result of trying to solve a problem that we weren’t able to actually do with SQL Server.”

“I think what’s going to happen is SQL Server is going to be there for traditional structured applications and database.”

“The great thing is that we can operationalize an R script in SQL Server.”

“If you’re SQL Server and you want to look at big data… you’re going to learn Hadoop.”

Listen to Learn

01:41 Thoughts about database evolution and transactional system
02:05 Ideas about big data and database tools and services
05:40 Single platform for data: is it a fair approach?
12:00 JSON data, storage cost and compute cost
12:48 Effect on the Data Team
14:15 R script on SQL Server
14:46 Role of the Express version increasing in organizations
16:44 Azure services, data lake analytics, U-SQL on organizations
19:05 Technical skills that are robust in line with big data and data warehousing
21:33 Segway about Power BI
22:28 Azure vs Containers
27:13  Andrew’s advice for executives investing in their data platform
30:57 SSIS, SSAS, Power BI
35:56 SQL Family questions

About Andrew Snodgrass

Andrew SnodgrassAndrew Snodgrass is the vice president of research for Directions, a organization that help guides IT executives on technologies, strategies, product roadmaps, and licensing policies.  Andrew leads the research and analysis of emerging trends in enterprise applications (primarily ERP and collaboration tools) and database management technologies with a focus on Big Data, business intelligence solutions, cross-platform mobile application development, and cloud hosting services.  Her is currently and adjunct professor for the Albers school of business and Economics at Seattle University.

Transcription: The Future of the Relational Database

*Untranscribed introductory portion*

Carlos: Andrew, welcome to the program.

Andrew: Thanks!

Carlos: Yeah, we appreciate you coming and taking some time to be with us today. We have kind of an interesting topic. Ultimately, our audience, our SQL Server folks, and we think about SQL Server, you know, we think about transactional system. It’s been around for a while although it has definitely evolved a bit. But one of the common questions that we get is what about all these unstructured data and even in SQL Server now we are starting to see that influx sort of kind of with XML and now we’ve got JSON, 2017 is going to have graph databases, Python, R. All of a sudden we are starting to see this intersection of what we could traditionally think of unstructured and structured data commingling in the same environment that has a lot of people of questioning; and not to mention the rise of other databases, right? It’s 2017 and Microsoft comes out with a new database technology and they are not the only ones. There are databases almost sprout out as often as applications all trying to solve a particular problem. And so ultimately our conversation today is going to revolve around this evolution if you will and kind of where the transactional system is going to end up. So I guess maybe take us through some of your thoughts there and kind of what you’re seeing and maybe what some of your customers are asking you about the same idea?

Andrew: Sure, along that line I think some of the rise of the M structure, the big data, semi structured, all these stuff have come around naturally and I think that the introduction of database tools and other services and capabilities have come about because as traditional structured, traditional relational database guys keep trying to force fit unstructured stuff into structured databases. And so these things have come out as a natural result of trying to solve a problem that we weren’t able to actually do with SQL Server. And so I don’t know that it’s necessarily that we’re seeing any kind of decline in traditional databases or in the relational industry that we’ve always been dealing with. I think those types of workloads continue to exist. I think they are still going to continue to exist and you still got financial systems, you still got procurement systems, and these things lend themselves well to a traditional, relational structured database model. But they don’t do everything. One of the things that I’d always look at was how do we handle situations where you need some kind of lightweight database management system. And SQL Server despite some attempts to give us lightweight database management didn’t always do a good job of it and so when JSON came about, XML you could say a little bit, but really when JSON came about and the idea of saying let’s go mobile. Let’s push data out to self describing. These seem like natural solutions and now that we’re seeing SQL Server come back and say, “Well, we’re going to try to make so that you can tap those sources.” I don’t know that I would necessarily think that SQL Server is trying to be everything although Microsoft might like to portray it that way. I think that SQL Server is trying to tap everything. And so it’s one of these things where I’ve got my traditional product list, my customer list, all my transactions that are in a nice, structured environment. We are working an application that people know and trust and love but I need to be able to get into other areas. I need to grab data from other areas and push data out to other solutions. So they are putting these tools into our SQL Server environments so that we can interact with those things intelligently and more seamlessly so we don’t have to push to an ETL tool or we don’t have to do some kind of extraction and then modify the data to push it out somewhere else and then bring it back. We can kind of put all these things into one environment and control it there.

Carlos: We’ve definitely drunk the KoolAid. I’m Microsoft kind of through and through if
(00:05:00) you will. Is that a fair strategy as I go and try and talk with businesses? One of the things that I like about this idea is that at least it gives me a common tool set from someone who’s been in SQL Server now for 15 years. That I don’t have to go out and learn all these other platforms, right? Now, obviously there are some differences, R or Python, so there is a learning curve but at least from I know the data in SQL Server there are some administrative things that I don’t have to relearn. Things like that and I think it will be easier. But it gives me the ability on a single platform to have all of that data. I mean, is that a fair idea or a fair approach?

Andrew: I think a fair assessment on it is I think that from an additional enterprise standpoint, I think it’s very attractive and I like it personally as well, large organizations, medium organizations, anybody that has applications that they’ve developed over the years where they’ve invested. This isn’t just an investment in skills and application creation and the rest of that but they actually have servers and they have licenses and all kinds of other stuffs that go along with that and the idea of saying, “If I can take advantage of what I already have and expand on it instead of trying to recreate it. That sounds very attractive. Here is a great analogy on it. There was somewhat type of when we went from client-server to web-based apps. There was this need to recreate all these applications so they work in a browser and it would have been nice if we could have kept our applications to simply have more kind of browser. And that’s kind of what we’re getting at here is we have all of our data sources, all of our capabilities, but we want to extend it. And that doesn’t mean you need to learn how to write web apps and it doesn’t mean you need to learn JSON and convert data. It doesn’t mean that if you want to go to Hadoop you’re going to have to learn Polybase and understand how to do a scale out compute environment and being able to tap those resources and manage that data. You might need to learn some MapReduce at the same time. But this is learning additional things and not replacing what you already have. And that’s I think part of the attraction of beefing up SQL Server to do more than it was before.

Steve: And I guess the way I’ve seen this happen a few times that I have talked with different people is that they’re using SQL Server, they’ve been using SQL Server in their shop for years and then they realized. “Ok, now we want to do unstructured data.” Big data as they often times end up calling it. And then they decide, “Ok, well we’re going to go with Cassandra or DocumentDB or CouchDB or DynamoDB.” Or one of those that is totally separate from anything that is SQL Server. I find that sort of the story goes with those oftentimes is that it’s kind of a decision. Are we kind of keep doing SQL or are we going to go with this other database or are we going to do a merge there or sort of using both sides of it. IT sounds like what you’re describing here is taking advantage of what’s being added into SQL Server so that we don’t have to go and look at those other technologies. Is that where you’re going?

Andrew: No. Actually, Steve, I don’t think that’s kind of the case. I think the merge is what’s going to happen ultimately. It’s kind of like buying a stereo system that’s an 011. You basically get a very generic set of components. If you really want the best you buy separate components. And I think what’s going to happen is SQL Server is going to be there for traditional structured applications and database and stuff but people are still going to be looking at Cassandra, cloudera, Hadoop, any of these for a true multiprocessing, parallel processing capability that they really don’t want to spend the money to try to make SQL Server do. The idea of that is if you go to a bid data solution you have a distributed computing environment with a lot of servers out there. Each with some local storage so that the computing the query happens locally on the data and you use parallel processing to do a lot queries across a lot of unstructured data at once to return the said information. If you try to do SQL Server to do a
(00:10:00) scale out and that same type of configuration your licensing cost alone is going to kill you. It doesn’t make any sense to say that you’re going to have 50 or a thousand servers each running a SQL Server Standard/Enterprise edition license. Oh my god, I can’t imagine the amount of cost that you’re going to have there. So what you’re actually looking at is you got a SQL Server environment that’s very powerful that has some really good SQL Server capabilities on it that uses Polybase to go out and tap that unstructured data that’s actually in a proper Cassandra, Hadoop deployment. If you really need to get in big data you’re going to have to look at a proper big data solution to host the data and run the compute. But you don’t have to run those stand alone. You don’t have to just do a MapReduce to get a data set that then you have to go through Integration Services or some other ETL in order to get in what your structured data. You will be able to do that directly from SQL Server so that you can query your structured information, your product list, your customer list and you can do sentiment analysis against a bunch of unstructured data out in Cassandra and bring those together in real time. That’s where I think the merging and the intelligence is going to be.
Carlos: Interesting. I think kind of with that a common point is like IoT, right? So you’re collecting, so all these sensors have all these information. SQL Server probably may not make the most sense for the repository for all of that information, right? But you have these sensors, you know, caches that on the web which you can then pull in to a Hadoop cluster or whatever else and then start to do that aggregation or some of that analytics on it and then pull in kind of the summary information and that maybe what you store in SQL Server and then of course then there is the reporting on top of all of that.

Andrew: Right, and that follows, Carlos, in the same line with the idea of JSON. It’s great that you can some native JSON stuff within SQL and you can push data out and pull data in. But from a storage cost and compute cost do you really want to store JSON data in blob columns within the SQL database? You are going to run some cost and performance issues that you might not want to deal with so I don’t think that’s necessarily the road with the JSON side either. I think store it where it make sense, where it is optimized and where it belongs. But I want to do stuff with that data and I want to do it in real time and that’s where bringing it into SQL Server and attaching it to business data make sense.

Carlos: So then let’s talk a little bit about the effect, are the data team just going to get bigger? I mean so we have like cache and some of this other caching layer that the programmers are already kind of doing. So now we start adding Hadoop and we’ve actually done an episode on Polybase. Now Polybase is still kind of an emerging technology. They kind of came up with the v1 but those worlds are still kind of very different so does that mean that I’m now going to have a SQL Server person, I’m going to have a Hadoop person, I’m going to have a Cassandra person or are you seeing organizations say, “Ok, guys. Now, it’s time to go learn Hadoop and you’re going to be responsible for both.”

Andrew: In some, and it depends on the size of the company. The really intelligent ones are investing in data architecture. It doesn’t matter just of saying we want big data you know everybody says that. How do you do that and how do you do it without spending too much.

Carlos: So the plumbing, the connections have become very very important. How’s that data is going to move and flow.

Andrew: Right, and so and the same thing is true now with machine learning. It’s great to say that we have R and Python in 2017 but what are you running this against? And what type of performance that you’re going to take on your servers with it. The greatest thing there isn’t suddenly we are able to do R scripts within SQL Server. The great thing is that we can operationalize an R script in SQL Server. And it’s that I’m going to R Studio. I’m going to build my script and I’m going to test it out in my data. I want to make sure everything is the right way and then I’m going to put this into a stored procedure and make it part of my application so that it performs the R script on the data, the right data, at the right time and then gives my users something useful.

Carlos: Exactly. I think now, so having said that do you see the role of the Express version increasing in organizations? I feel like adding some of that functionality
(00:15:00) to Express was a direct result of the rise of open source and particularly MySQL. And so to that idea now that I have the capabilities, and when we start creating these models maybe we don’t need 10 years worth of data. I need a month’s worth, a quarter’s worth. And then I can push that off unto an Express version let my data scientist or analyst or to whoever run all the items they want and then once they’ve decided, “Oh, I think this is where we want to go.” Then again kind of that integration, that’s in SQL Server already I could just move that into my production environment and have a more seamless experience.

Andrew: That’s a really interesting question on that. Most of our customers tend to be larger enterprises and so they have their MSDN Visual Studio subscriptions and even the data scientist they are putting MSDN Enterprise edition on or Standard and running it from there. I don’t know any answer for that. I think you probably could.

Carlos: Sure, the limitations that you’re going to run into. You know, 10GB or whatever. But again it’s that I can move that off my environment with the flexibility I get the tools so I guess I’m kind of interested to see how that plays off or plays out. Obviously, if you are Fortune 5,000 Company maybe that doesn’t makes sense but for these smaller organizations they have a couple of people trying to work on this. It will be interesting to see what happens I think.

Andrew: You know, on that line opening the door on that one for some of the may have customers both large and small we are seeing the use of Azure services spinning up capability there, moving data out, testing it out and then scaling down is a way of getting around local deployments and testing. Like Azure SQL Data Warehouse, all of a sudden you’ve got somebody who has 10 or 15 years or 20 years with a historical equipment data or performance data and they push this up to an Azure service and then run one of the analytics services against it. You know, they are doing some machine learning and some bigger analytics that they would never attempt on premises.

Carlos: Sure, because once they’re done they could just toss that out or spin it down.

Andrew: You know, and along those lines you’ve got Azure Data Lake analytics which was supposed to be originally just for going against the Data Lake storing environment with a bunch of unstructured data and all of a sudden they said, “Well, you know, we could tap into SQL database. We could tap into blob storage. We could tap into data warehouse.” And Data Lake analytics is all based on U-SQL and so we’ve got a really nice environment here for people that are comfortable with traditional SQL Server query capabilities to go out and run some really cool analytics and learn just a few new functionalities that are required for U-SQL and you got a scalable system out there to run these queries on that you don’t have to do on premises.

Carlos: Yeah, it’s very cool. I still think that some of that is like you mentioned for the larger organizations. I think as we start talking with people, I mean, there are those who are putting quite a bit of time to learn the U-SQL and take a look at Data Lakes. And I think obviously the bar is lowering but that’s still an investment right to take the time to learn some of those things and understand how they work. I guess like everything, all the technology continues to evolve and change.

Steve: Yup. It’s interesting because when we had our podcast talking about building your technical skills few weeks ago, one of the things we talked about was sort of the type of things that you can learn that stick around for a long time and the type of things that you can learn that are maybe a little bit riskier but they may go away or they may change rapidly. And it seems like some of the unstructured and big data and data warehousing pieces today are kind of within that riskier category because they are still sort of changing quite a bit. I mean are there any of these that you see that are kind of more robust to the point that or more mature might be the better word to the point that they are not changing quickly
(00:20:00) these days?

Andrew: Yeah, actually. Especially with the folks that are going to be listening that are SQL Server folks. You know, along that line if you take a look at Microsoft’s database landscape and say you’ve got structured, unstructured and semi and everything in between, the two that you would go into that are not SQL Server would be Hadoop and JSON. Hadoop is underpinning so much to the big data investment that Microsoft does into that if you want in SQL Server you’re probably comfortable to start learning Hadoop and understanding that environment. That was the whole concept behind Polybase was to touch Hadoop. If you actually look at the Polybase scale out concepts, this is a Hadoop cluster methodology for doing compute nodes. If you look at the Azure services you got Data Lake which is a Hadoop environment on the backend. You got Cosmos DB which is actually a Hive environment in the back end, and then you’ve got HP Insight which is multiple Hadoop type clusters that you can choose from there to do any kind of scale out environment you want based on the type of workload. Yeah, if you’re SQL Server and you want to look at big data and you want to say comfortable with Microsoft you’re going to learn Hadoop.

Carlos: Ok, interesting. There you go companeros. You’ve been warned. You heard here first.

Andrew: Yeah, and while you’re at it, learn some Power BI because Microsoft is investing in that.

Carlos: Oh yes, that is true. What it’s interesting I think that still kind of goes back to the plumbing. We’re actually starting to see traditional SQL Server people/folks trying to spin up Power BI environments and you’re like, “Hey, really cool reporting tool.” You know, the functionality is there, web browser and all that but then tickets are a little too big and they are like, “Hey, how to make this go faster?” And you’re like, “pull that out .csv file.”

Andrew: And then you do a Power BI premium where you have dedicated capacity or you end up doing a Power BI report server on premises.

Carlos: Yeah, that’s just coming out and it will be interesting to see that integration with Reporting Services that continues to evolve. I wanted to circle back quickly if we could because you mentioned Azure. We’ve actually talked a little bit about containers and containers particularly in the Linux world has been all the rage but it almost seems like maybe, I don’t know if competition is the right word, because ultimately the technology that they can give are different. Well, Azure versus containers. So if you’re a large organization and have an investment or haven’t made the move to Azure to that whole spin up for the way idea. I feel like that’s what containers are also trying to provide you. So is it an on premise Azure competition there when we talk about containers?
Andrew: Like yeah, that’s an interesting one. Maybe I’m too old struggling to find out a really good set of examples of doing containerization of apps and saying we’re going to put this out here especially when it is database related app. It seems like there is an off light you’re going to put in that container in order to make it work. But you know it’s interesting in Azure right. Microsoft’s whole move towards Linux anyway with SQL Server 2017 coming on Linux and the containers and all the rest of that. Azure has the Azure container services which has been out there for a little while, a year or a year and half, that allows you to spend up VMs and containerize things and then everything in it. But they’ve also just released a thing and it’s in preview right now called Azure Container Instances. And this is one where it’s a true platform as a service type. They provide complete control in the infrastructure so you don’t know the backend VMs. All you do is you’re seeing computing resources. I mean this many CPUs, and this so much RAM and here is my container image. Pow! Put it up there. Scale it as much as you want, run it as long as you want and then kill it. You don’t worry about the VMs or any of the backend stuff. You can throw SQL Server into this, and you don’t know if it’s Windows or Linux and doesn’t really matter because there is web access to it anyway. You get the URL and you just run it and you go for it. And so that’s another one where Microsoft is just trying to say we don’t really care on the
(00:25:00) platform. We know it’s the service you want and so here is how to go about doing the container and you don’t care what’s happening on the backend side of things.

Carlos: Now, I feel like the best example, and so we had Andrew on the program, different Andrew containing about container. The thinking was and I guess I’m inclined to believe it is that, at least from a database perspective. I can’t speak to the apps. You could do lots of different things with the apps but the best or the classic case for containers in the database is a development workshop. If you’re building like .com type of thing, right? If you’re a actively doing development that whole idea of, “Ok, once you’ve published or once you’ve pushed code. Ok now, let me take that snapshot, create a container, push that down back to all the developers.” So now I know they are developing against what is in production and it helps eliminates some of this migration issue.

Andrew: Yeah, that sounds very internal.

Carlos: I don’t think we’re quite, although I have heard a few people talking about it, I don’t think we’re quite saying that we’re going to throw containers at production databases just yet but.

Andrew: Yeah, you know, sometimes I hear so much about containers right now and it takes me back when we first heard about big data and then about artificial intelligence and it’s like, “Oh, I’ve got to get me some of that. I don’t know what I’m going to do with it but I got to get me some of that.”
Carlos: That’s right, so having said that what your ultimately with the directions, your organization, your consulting CIOs on technology choices. Now, obviously your mileage will vary right and it depends. It’s going to come into here but what’s your, I know we covered some of this pieces, and I guess you’ve already mentioned if you feel comfortable with SQL Server, you feel pretty safe gambling on Hadoop, not gambling but learning Hadoop. You know, taking the time to that could be your next foray. What else are you advising these executives in making investments in their data platform?

Andrew: So the things that we’re pushing or talking people about now is really along the lines of how do you take advantage of not only big data and machine learning capabilities…
Andrew: There you go they are implementing IoT and so that’s one of the things that we tell the folks is not necessarily to jump in because these guys aren’t going to do major investments or major overhauls. These are like organizations that have to plan out years in advance of what they are going to be spending their money and then get authorization for it and have to justify it. They’re getting a lot of CFOs and other CxOs that are reading too many magazines on airplanes that are coming back and saying, “Why aren’t we in big data? Why aren’t we doing artificial intelligence? What about IoT for our organization? How come we are not doing better analytics and all these stuff?” And so we’re trying to show them what Microsoft approaches to a SQL Server and data overall, and in fact we’re doing one here in another week on business analytics and business intelligence and how this are all working in the vision going forward and where the investments are happening and where should you spend your time now and where you should do it in the future? And so we think machine learning is actually far more mature than people like to think to this. Especially in a traditional business environment, there is a lot of stuff that is out there that the community has developed over the years either on R or Python. Now is the time. If you were ignoring it before saying we got to wait and see, now is the time to jump into it. The other area that we’re telling them to get into is taking a look
(00:30:00) now at better business intelligence and data sources on premises. This whole Power BI Report Server if you guys haven’t spend time in it we’re telling them to bring that in and take a look at it. One of the other one that is kind of surprising is any traditional ETL workloads that you have going on were you’ve been suing integration services for years. You know, ETL is changing, the idea that SQL Server’s database engine has the ability to go out and tap these resources in real time and do transformations. You might not be doing ETL the way you did for the last 10 years. Integration services, I don’t see getting a whole lot of investment. I see a lot more going into analysis services. We’re going to do real time queries and being stuff together and transform it there not in Integration Services. And so I think that’s going to be kind of a change for organization. You know, and that’s going to be one if you’ve invested in SSIS you’re not going to go out and swap it next month or next year. This is going to be like a multiyear thing. New stuff we bring in, we’re going to actually bring in directly. We are not going to run it through an ETL.

Carlos: Well, now see you’ll forgive me I guess maybe that my connections aren’t quite there. How am I getting the data? When I think of SSIS, I am thinking about it is that glue that takes me from my transactional system, into my cube, or it’s doing some transformation there for me. There are additional components now. I mean, we’ve talked about Polybase, so SSAS, the analysis services, now has these connectors to actually go out and grab this information in an ample time.

Andrew: Oh yeah. 2017 baby! Here we go, so Analysis Services in 2017 supports the M language. So all of the sudden all of the stuff you we’re doing in Power Query and Excel and you’re bringing stuff in and grabbing different sources, and maybe going out and getting a file list instead of just another data source that’s all coming in 2017 Analysis Services. So the big reason for that isn’t necessarily to make analysis more functional which was kind of nice outcome from it but they wanted to bring Power BI, and Power BI is all based in Power Query. And so in order to support that with Analysis Services in the new Power BI report service thing which is basically Reporting Services plus they had to bring in the Power Query engine capabilities.

Carlos: Ok, so now there is another dot connected because I know at least the initial version of Power BI Desktop, so the new version that you can download and install locally that only supports connections to SSAS. And I guess it makes more sense now if SSAS has a bit more functionality in it for you to take advantage of some of these other data sources because I think initially when I looked at that I thought, “Oh, that’s all going to be all cube data. I don’t want to always do that.”

Andrew: Yeah exactly, and it’s not. So what is Power BI report server when you bring it down and you install it basically we work on that reporting services plus because it’s just really the same thing. There are a few other things. You’re on a modern life cycle for updates which I don’t particularly like but that’s reality. But you can host your PBIX files and so you’ve got that capability. But the PBIX files that’s the only one that’s limited right now, the rest of the RDL stuff and KPIs and all the roster are full capability. So the PBIX files that you have there can only go through an analysis services cube which actually winds up being your gateway which then you have to think of in that scenario analysis services being like SSIS. It’s the glue that goes out and grabs everything and brings it in, right? Now, that’s the first three to Power BI report server hosting so you’ll see that expanding. They are going to all kinds of data sources for PBIOX files so I don’t expect that to be limited for more than a few months.
Carlos: Ok, very good. Andrew, thank you so much for visiting with us, great conversation. I wish we could go on for a bit longer but I know that we need to wrap it up here. I guess last thoughts. Steve, do you have any last questions.

Steve: No. I think that there is just a lot going in this space and keeping on top of it is going to be interesting I think. Like we are talking with Power BI so Microsoft is investing so much in Power BI these days that everything it touches is sure to expand down the road in the near future. So I think that’s a pretty safe bet. I
(00:35:00) think there is a lot of interesting things to come here in the near future.

Andrew: Have you guys ever wanted to talk about that one in a separate one. Power BI is one of the other cover, not for this broadcast obviously but in the future.

Carlos: Yes, of course we’d love to. Again, particularly for this conversation we wanted to have somebody who was obviously familiar kind of covering that beat if you will but didn’t work for Microsoft.

Andrew: They are not even a customer of ours.

Carlos: To see just how that continues to play out. I think we touched on it a bit but we as database administrators we have our homework cut out for us. I think it will be fun to see how that continues to evolve as time goes on. Shall we go ahead and do SQL Family?

Steve: Yeah, let’s do that, so Andrew how did you first get started using SQL Server?

Andrew: Up until 2008 I was really kind of a Sybase kind of guy. I know it takes me back a while. We had a need for standalone database engines and Sybase was one of the early ones that allowed you to have a decent relational database environment that you can have on a separate machine. We have a bunch of travelers so they needed that and I did not want to go into Access or FoxPro which is where we originally started. So there’s Sybase and then as things progress we saw the direction for SQL Server and that became our standard and we moved over in 2008 and never looked back.

Steve: Ok, interesting.

Carlos: There you go. Now, as you’ve been covering SQL Server since then and then working with it. If you could change one thing about SQL Server what would it be?

Andrew: Yeah, you don’t like this. I changed Management Studio actually.

Carlos: Oh boy! Here we go. You know, because they have broken that out. They have their own development cycles now. Let’s see. What don’t you like about SQL Management Studio?

Andrew: Well, I like a lot that’s in it. I think the thing that would benefit them is as if they would make it easier for new entrance. One of the things that I find that is most difficult with new developers and new database people we bring in isn’t learning SQL or learning database structures. They’re getting all kinds of education on that. But setting it up, accessing it, using it, you know, some wizards something in there for the new guys to be able to get in and use it would be nice.

Carlos: Sure, kind of updating that almost to like a web type interface which is a little more intuitive. We’ve have all those buttons everywhere and you’re like what kind of I’m looking at.

Andrew: Yeah, and they got some stuff that’s nice but few years back we would seek new people and say, “If you really want to see how to build a query on this? Let’s do this in Access. See this nice wizard, we draw all these tables together.” Oh look at that that great query. You know, the query works on but into Management Studio and running it as a query there, “Look it works.” Why can’t they do that there?

Steve: Very good point. And I think, I mean personally I love what you can do with Management Studio but I have definitely seen the challenge when you have someone who is first time ever using it. They sort of get dumped in and don’t know where to go. There is this huge hierarchy in the preview in the left side that you can expand. How do I get to other query? Can be a little the bit daunting in the beginning?

Andrew: Oh yeah, the wizards for setting up jobs was nice. Being able to go through and take a look at especially some of the new views and performance capabilities. These are all great stuff but I got new people that we would like to get engrained in this just to make their lives a little easier so they get addicted as well.

Steve: Alright, so what is the best piece of career advice that you have received?

Andrew: Early on someone told me, “Don’t get worked up about office politics.” I said, “Look, focus on doing a good job and the people you really want to work for will notice that and promote you and those who don’t care about the good work you do.”

Carlos: Good things will come to those who wait.

Andrew: Oh yeah, and if they don’t then you’re in the wrong place.

Carlos: Sure, yeah, and then think of all the time that you don’t have to spend worrying and talking about, “Oh, what are they going to do with.” You know, I don’t like this person.

Andrew: Exactly, yeah. Just don’t worry about it. Just go to your job.

Carlos: Very good! I like it.

Steve: You know, that’s one of the things that I like about on the consulting side is
(00:40:00) you’re there to do the job and not to be in the politics. I like being on that side of it.

Carlos: Andrew, our last question for you today. If could have one superhero power what would it be and why do you want it?

Andrew: I know he gets ridiculed all the time especially on big bang but I really like Aqua Man. And his ability to breathe under water so that I could basically do scuba without any kind of limits. That would kind of eminent. Lots of people is cool too but this ability to breath underwater that’s my thing.

Steve: Hey, if you figure that out let me know. I’ll go with you.

Andrew: Yeah, but not around here. It’s too cold. We’re going to go south.

Carlos: I said that would be the last question but just one follow up. Where is your favorite place to scuba dive then?

Andrew: That’s going to be the Caribbean and basically anywhere, 7 mile off coast of Mexico, any of it. Just go down about 70 to 100 feet and forget the world exists.

Carlos: And you’re there. Wow! Andrew thanks so much for talking with us today. We really appreciate it.

Andrew: Oh, you bet guys, that’s a lot of fun.

Episode 107: BI for the Little Guy

BI for the little guyData analytics is all anyone seems to talk about anymore.  We’ve even caught the bug at SQL Data Partners and have started working with data platform MVP César Oviedo from Costa Rica.  César has been helping us on some projects and we thought it was time we introduced him to everyone.  On the Spanish front, he has been very busy putting out YouTube videos and publishing content for BI LATAM (Latin America).  We invite him on the show to introduce himself and our conversation turns to the idea of BI for the little guy–smaller organizations that need to take advantage of analytics and how the ecosystem is changing to support them.

We are happy to introduce César to you and know you will enjoy hearing from in this episode and future episodes.

 Episode Quotes

“The problem that business intelligence has had in the past is the thinking… Ok, the BI is for Enterprise. The BI is for huge companies.”

“Again a small or medium or huge company needs an architecture behind to support all your solution.”

“Another piece that is rally fancy today is data science… Everybody is trying to understand and talk about data science.”

“I think the days of the DBA who’s only responsibility is backups have come and gone.”

Listen to Learn

00:30 A little background on César
02:00 General view of today’s topic – Business Intelligence
02:52 Companero Shoutouts
03:59 Announcements about the Companero Conference
06:44 Microsoft released updated drivers for PHP to connect to new SQL Server
08:41 SQL Server 2017 – High availability support for SSRS
10:46 Why should small and medium organizations care about business intelligence?
14:14 What are the available options that you can take advantage of some business intelligence?
17:25 Tips or suggestions on PowerBI
21:12 Other things to look at when implemnting business intelligence: data warehouse, data lake etc.
23:51 About analytics and data science, and the pendulum swing of job roles
27:00 SQL Family questions

YouTube: https://www.youtube.com/bilatam
Facebook: https://www.facebook.com/BusinessIntelligenceLatam/
LinkedIn: https://www.linkedin.com/in/cesaroviedo/

About César Oviedo

César OviedoCésar Oviedo is a Data Platform MVP with more than 15 years of experience working in multidisciplinary projects running roles such as Scrum Master, Project Manager, BI Developer and DB Administrator. With experience leading teams and projects with more than 30 members. Currently focused on Data Science and Big Data platforms. Community Manager at “Business Intelligence LATAM” (BI PASS Chapter Leader for Costa Rica and PowerBI User Community), international speaker and trainer for Data and Cloud technologies.

Transcription: BI for the Little Guy

*Untranscribed introductory portion*

Carlos: Companeros, welcome to Episode 107. We’re switching things up a bit again today as we tend to do from time to time. Joining us today is going to be Cesar Oviedo from SQL Data Partners.

Steve: Hey, SQL Data Partners I know that company. Wait, that’s our company.

Carlos: Yes, so Cesar is going to start working with us. He is helping us out with a few projects. So Cesar is from Costa Rica and he is an MVP in Business Intelligence down there so you can see where he is going to be picking up some slack for us. So Cesar welcome to the podcast.

Cesar: Hi Carlos. Hi Steve. A little scared you know. It’s my first time in a podcast.

Carlos: Well, yes, so it’s a little funny or ironic that you say that because obviously after a hundred episodes we’ve gotten a little bit use of this but I know you have created your own Spanish language version but you’re doing kind of a webcast of sorts there. You’ve done, what, 6 or 7 episodes then?

Cesar: Yeah, it’s around 5, but it’s kind of hard to find speakers.

Carlos: Yes.

Cesar: I don’t have one hundred and others. I am working on it.

Carlos: So now that you’ve come on the English language version, I’ll send Steve down to the Spanish language version. We can do that next time.

Steve: Oh boy, that’ll interesting. I’ll have my translator dictionary handy.

Carlos: Yeah, I’m sure Google Translate will work just fine. Just speak into the mic and it will translate everything you want to say. So, Cesar is going to be helping us out. We don’t know that he will be on every episode. We’re gonna try to get him in on as many as we can but he will be joining us from time to time and so it will be good to talk with you further, Cesar. Ultimately out conversation today is going to revolve around Cesar’s focus and that’s business intelligence. So we are going to chat a little bit about why small and medium businesses might need business intelligence and of course from the Microsoft perspective, how they might be able to take advantage of that and some of the new resources that are available to them. So before we do that.

Steve: Yes, do we have any companero shout outs this week, Carlos?

Carlos: We do, and I’ve checked this name twice because he accused me of saying it wrong last time, but Alberto Lopez. Alberto is giving us some love for Episode 104. It’s interesting how kind of the comments come in. Somebody tagged Eugene. So, Episode 104 was the Episode with Eugene. We were talking about learning and how to learn and lots of people thought that it was a very interesting episode and that idea of the half-life. You know, learning radioactivity idea, got a few people talking. So thanks to Nicky, and I’m just going to leave it at Nicky because I cannot pronounce that last name. Bert Wagner and Nicole Gaudreau for chiming in for Episode 104 as well.

Steve: Episode 104 was a lot of fun and on the non-technical side.

Carlos: That’s right. It’s something that we all face with, we kind of deal with on a regular basis.

Steve: And we have the Companero Conference coming up in a couple of months.

Carlos: That is right. We are getting very, very close. It’s starting to get exciting. I guess when this episode goes out we will be at 60 days or so, so we are getting close. Of course, if you are a listener of this podcast and you want to come you can use a discount code, companero, and that will get you 15% or $75 off. We will remind you that on August 16th will be the next price increase so the conference will increase $100. So, if you want to beat that and apply the $75 to that you can do that before August 16th.

Steve: So, to get the absolute best price use the discount code and do it before August 16th and you’ll be in.

Carlos: That’s right, and with your favorite companeros. We’re even going to try to convince Cesar that he needs to come.

Cesar: Of course, I would love to go there.

Carlos: Hey, that’s right. Who wouldn’t love to go? So, I won’t give too much away but I
do want to start. It’s not on the website, so again you’re king of getting some insider news here. But one of the activities, one of the things that we’re going to do at the conference is give you a way to be able to test your SQL Server knowledge with some of the other attendees or against the other attendees. So we think we’ve put together kind of a fun little activity spearheaded by Mindy who was one of our speakers and that should be great fun.

Steve: Yeah, that does sound like a lot of fun. I remember Mindy talking about that a while back so I’m excited to see how that goes.

Carlos: That’s right. And I guess let’s see by now the website has been updated a bit, like with most things as this is your one, we kind of iterated a bit on the website and so we’ve made some changes; hopefully made the ability to access the speakers and the session information a little bit easier. So we hope that you’ll check that out and let us know what you think. So, it’s been a little while but I think it’s time for some SQL Server in the News.

Steve: So, Microsoft has just released updated drivers for PHP to be able to connect and use SQL Server, and you might be wondering, “Why is this important?” Well, for anyone who is out there coding web applications in PHP, and they are trying to get to the SQL Server database, they need to use these drivers to be able to talk to the database. Without that, the drivers have been there for a while but Microsoft has come up with lots of iterations over the years for them and they have come a long way from the beginning. Every time they come out it’s always good to see that there is a new batch of patches and stability fixes along the way that make it even better. If you’re using PHP to talk to SQL Server you might want to look at those new drivers and consider upgrading.

Carlos: Yeah, and I think they continue to invest in “non” Microsoft technologies. It’s not just .NET connecting to SQL Server anymore. Now that SQL Server is in Linux. You know, I was actually on the Ruby Rogue’s podcast talking about connecting Ruby on Rails to SQL Server. Of course, I guess PHP has been there for a while but I think they are trying to increase that adoption, trying to reduce the barriers the developers have to be able to use SQL Server as a data store; which is kind of fun to see because I think then from an administrator perspective that might mean that there are more opportunities to then support or be part of different projects than I was previously.

Steve: Yes, indeed.

Carlos: Another change or another update, so in conjunction with SQL Server 2017 but I think it’s substantial and that is this idea of scale-out support or basically high availability support for SSRS. If you’ve known or even for a long time coming using clusters, SQL Server cluster, and then you have Reporting Services. That didn’t really work all that well. I said reporting services, and then I guess I should add also in addition Integration Services and being able to use that Catalog DB so the Catalog DB can be used in availability group scenarios so basically it can move. It could be redirected and your packages will still work.

Steve: Then with that you don’t need to be deploying the packages then to every server because now that’s going to be aware availability groups and you just have to deploy it to the availability group using the SSIS DB and you’ll be in better shape.

Carlos: Right, you can leverage that availability group because you started deploying to the catalog whereas before it was a little bit prohibitive because that couldn’t get replicated. Now it can.

Steve: Alright, very nice.

Carlos: Ok, so the URL for today’s episode is going to be sqldatapartners.com/BI, for business intelligence.

Steve: Or sqldatapartners.com/107 for our episode number.

Carlos: And so with that let’s go ahead and turn our attention over to our guest. Cesar,
welcome again. Ultimately, we are talking about business intelligence, kind with a focus for small and medium organizations. So I guess first tell me why should these organizations even care about business intelligence?

Cesar: You know, actually the answer for this question is pretty simple, everybody has data. The problem that business intelligence has had in the past is that, I don’t know, maybe around 10 years ago when I started working in business intelligence everything was, “Ok, the BI is for Enterprise. The BI is for huge companies.” I guess the main problem is that the companies are still considering BI is like a fancy feature or something that maybe it’s not for me. But to answer your question, everybody has data, so why we will not have the features or the mechanisms to get high quality data and to get your insights. Because again it’s different if you have a simple report that is showing you the sales, yesterday’s sales or something like that or the weekly sales or something like that, very simple stuff.

Carlos: Right, because everybody have data we should be able to interrogate that data and ask it questions. But they’ve been on the fence or behind because of the complexity. Right? You have to have a ton of people and you need to spend a ton of money on hardware and software. It gets real complicated, real fast.

Cesar: Yeah, but I always say, I am not working for Microsoft, or I’m not working for Tableau, or any other BI providers. Basically, I am a SQL Data Partners person, consultant, so basically. I always say, the main problem behind all the difficulties for the small and medium companies to get BI or new features is the provider’s stuff. Because it is the same thing as, it’s Microsoft’s fault, it’s Oracle’s fault because you as a small or medium company usually doesn’t have the money to make your own research. If you want the data, if you want your insights, but you I don’t know, let’s say some company comes to you and talk beautiful things and start talking about Big Data and Data Lake, and Cloud, and a lot of different stuff. Of course this provider is searching for his own business, but you as a small or medium company, you don’t have the money to buy those kind of fancy features but you still have the needs.

Carlos: Well, so obviously we are Microsoft-centric. Let me take another little sip of my Kool-Aid here, kind of focused on the SQL Server and the Microsoft Stack. You’ve mentioned Data Lake and Cloud, and Business Intelligence is a word we’ve been hearing a lot about. I guess the easy button approach is the right word. But what kinds of options are available to me now that I can take advantage of some Business Intelligence?

Cesar: The quick answer for your question is PowerBI. It’s basically, “Ok, I have
PowerBI.” It’s an open and free tool. But the problem here, again, is if you’re a company and you have your needs to create your dashboard or whatever, you just download your PowerBI. “Ok, what I can do to make it happen in a good way? What can I do to have a real dashboard that can handle my information to grow with my data?” Basically, this is something really important here because you can go to and download your PowerBI desktop or just create that PowerBI.com account, but what is the problem with that? Ok, I need the training and I need some basic knowledge about how it works. And let’s say a medium company, probably I have a lot of data and what about the data warehouse? The data warehouse is not something that is in PowerBI. It’s something different. It’s something about SQL Server. Again, it’s something that needs to be considered. It’s not only the tool. It’s the development of the projects.

Carlos: So the easy button is not quite there, right? I mean, you mentioned PowerBI being available so they can start using some of this data and I actually think back and we’re going to use one of the first projects we actually worked together on was an organization managing hotel data. And so they had all these information from spreadsheets and third parties that they were getting this stuff from, and they were just sucking it up into PowerBI and I thought, “Hey, we’ve arrived. We are using business intelligence. Yay!” And then they started adding more and more spreadsheets and it just got slower and it didn’t quite work for them. So while they may be able to use the reporting capabilities, you might still need somebody to help you kind of connect the dots from an architecture perspective.

Cesar: Yeah, absolutely. Again, it’s kind of hard to find a low performance in PowerBI. But again, this customer did it. I know that there is a lot of different people or companies out there trying to make it happen, trying to make his own dashboard and without any kind of support of anybody, so it’s a really, really interesting thing.

Steve: So then let’s say a small to medium size company wants to do business intelligence and create a business dashboard and they download PowerBI and they are ready to get started but they don’t know where to go. Do you have any tips or suggestions there of how to get to that next level?

Cesar: Yeah. First you have to understand that if you are doing anything, you can stop talking about PowerBI stuff. If you download something from internet and looks very simple, cheap, awesome, there is something, you know, some piece is pending.

Steve: It’s almost too good to be true.

Cesar: Yeah, exactly, exactly. This is exactly what I mean. Basically, PowerBI is this kind of stuff because it’s awesome. It’s really simple to use. But if the tool is free and I can create a dashboard in a few minutes, basically, let’s call it, again, a small, medium or huge company, you need architecture behind to support all your solution. The first step, or the first advice is, reconsider the steps, and consider that a piece is pending.

Carlos: So even though they’ve made this tool available it might not include all the pieces that you need to get up and going. I mean, not completely. They are trying to make this easy, right? I guess I’m putting on my Microsoft hat for a second, and it’s very, very cool. You can’t disagree that it’s not very cool and pretty easy to use. I think that kind of extension of, you know, if you’ve been working with Excel that leap or that jump to PowerBI is pretty simple, right?

Cesar: Yeah, it’s pretty simple. Actually, one of the podcasts they did in the past, one of them with the most traffic was the Data Warehouse with Jesus (name unclear) from Mexico was a really, really interesting point of view, was some really good conversation about this topic about, “Ok, there is PowerBI and everybody is loving PowerBI. What is happening? Why still we have problems with the data? The companies are not creating power business intelligence?” It’s not that simple. You need solutions, you need development, you need architecture to support your design. So basically, this is really, really important.

Carlos: And I know we’ve talked about this program before but this idea of being
able to solve the business problem and understand that. I think this is a perfect marriage of that technology and the business problem. So, you might not be “writing” the reports anymore. But you can still help with some of the plumbing, right, connecting the dots. And I think if you’re willing to do that and speak business, if you will, if you’re a technology person, and to be able to help those business owners with that then there will be lots of opportunities there for you. So, I guess we talked about PowerBI and the free components. Other plumbing or other things that organizations might need to look at as they start implementing some business intelligence?

Cesar: Yeah, of course. We talked a little about data warehouse but some people listen now about it or maybe just understand data warehouse as a huge database or something like that. So again, one of the primary companies in a real business intelligence solution, and this transformation and stuff so basically what I mean with transformation, I am talking about Integrations Services or if I am 100% Cloud I can consider data factory. Again, there is a lot of different mechanisms to do a lot of different things but again transformation is really important with integration services or data factory, as well I can use Data Lake, HDInsight, anything, you know, the big data stuff is growing as well.

Carlos: A little bit scary, right? I mean, I guess you hear that Data Lake, and again, granted, I’ve only seen demos and whatnot but I’m not sure I would associate the easy button and Data Lake just yet.

Cesar: No, it’s not for everyone but this is part of what I’m trying to talk. You know, it’s not for everyone and you have to understand that. If you’re a small company or you are a medium company you need that Data Lake or you don’t need that HDInsight to start working with your BI stack.

Carlos: I guess I would say, I would caveat that slightly and say existing company because there are plenty of new organizations, you know, ala .coms that might make a lot more sense for, but again those are primarily technology companies to begin with. And I think they would have a little bit more the staff and whatnot to be able to comprehend and implement some of those pieces.

Steve: So for instance, if you’re a .com or a startup or you have the developers and the IT staff that’s already doing the work in the Azure environment jumping into the Data Lake or the HDInsights or other things may not be that far off, I think. Is that what you’re trying to say?

Carlos: Yes, that’s right. Ok, very cool. Let’s see, what else?

Cesar: Another piece that is really fancy today is data science and stuff, you know, everybody is trying to understand and talk about data science.

Carlos: That’s right. I don’t know how about there in Costa Rica but one of the things the salaries are starting to swing towards the analytics and the data science way. So it used to be that from a salary perspective, database administration was kind of where it’s at and we’re starting to pendulum swing a little bit there and those salaries are starting to get really big. And so you’re starting to see people jump ship and want to learn more about that.

Cesar: Yeah, but you know what is interesting here because in SQL Server 2016 now you have the R components, and Python is for the 17. Right now you have R in PowerBI so it’s like, “Ok, something is changing.” You have to understand that a lot of additional features for data science is coming to the regular tools. So I always have that really coming, I don’t know, is existence question about is that the real disposition have a real future or will be part of everything? Because it’s in PowerBI so who will use this scripting, a PowerBI user? I don’t think so, doesn’t have the knowledge. Probably a BI developer? Maybe. And a real data scientist will say, “Ok, but this is a dashboard”, or “this is not the scripting. I really love the scripting stuff and the R scripting.” It’s like, hmm.

Carlos: Right. It would be interesting to see kind of how the job roles continue to evolve there, based on the technology that becomes available.

Cesar: Yeah.

Steve: I think with that there will probably end up being some crossover between the roles that a traditional DBA has to do versus a data scientist and there will be some overlap there.

Cesar: Now that you are talking about the DBA. The DBA have a huge challenge right here because everything is changing in his box. You know, it’s changing everything. Now we have little other different components the DBA have to handle.

Steve: I think the days of the DBA whose only responsibility is backups have come and gone.

Cesar: Yeah.

Carlos: Well, awesome. Well, obviously this is I think is just the beginning of the conversation that we’ll have and we’ll, of course, have you back on and continue this discussion in other episodes. Should we go ahead and jump into the SQL Family?

Steve: Let’s do it. So Cesar, how did you first get started with SQL Server?

Cesar: As you know, as everybody in the podcast know I’m from Costa Rica. So basically, here in Costa Rica, if I am in the near city basically you are a software developer, basically. I used to say, when I started working as a software developer I really hate to be a software programmer. It’s like, ok this is not for me. And it’s because I really love to interact with the final users and understand the requirements, and this kind of stuff, that software programming is not part of that. You know, software programming is more like, “Ok, you have requirements or in this case or histories from a scrum or whatever.” But this is not for me. I really love to interact with the customers and understand their real insights inside the business. This is why I just decide to move to anything else and I find SQL Server, actually DTS in SQL Server 2000, and ok, wow this is for me, I’m going to start growing. Basically, that is why I am here.

Steve: Exciting. I can remember using DTS on a number of different things back then.

Cesar: I don’t know if the word is “wow”.

Carlos: So, it’s funny, having just recently had to install, I guess that was 2000, right? But installed 2005, kind of taking me back a little bit there, right? One of the questions we’re going to ask you here is what do you want to change about SQL Server? Having had to deal with the 2005 installation here recently, I can assure you that I’m very grateful for the changes that have been made since those early days.

Steve: That is so true.

Carlos: So, having said that, if you could change one thing about SQL Server what would it be?

Cesar: You know, one of the primary stuff that, I always say that the SQL Server has to change was the, let’s say the integration with the BI stack was really strange. Right now with the PowerBI, with Data Lake, with BI, with HDInsight and Data Factory and others, you can see that the data platform stack is growing but outside the SQL Server. So for me, this is something that is already happening, and it’s something really, really interesting.

Carlos: May I, you know, knuckle-dragging Neanderthal right here, so the integrations between the engine and the PowerBI?

Cesar: No. I mean, if you go, let’s say, business intelligence solution in 2005 or 2008 you will find that you only have Integration Services, Reporting Services and Analysis Services, so basically everything is inside a box. Everything is inside the SQL Server.

Carlos: It comes with SQL Server. Like it is part of the, it’s almost like an attachment, an add-on.

Cesar: Exactly. Right now, this is really, really different because there is a lot of different features and tools that are outside of the SQL Server. So for me, this is awesome. This is the way Microsoft can grow PowerBI month per month, so every time it is changing and adding new features that of course, if process is inside, DVD installation will be impossible.

Carlos: We’re interjecting here another question, so do you see like they’ve done with Management Studio, which I guess is one thing. But currently, and correct me if I’m wrong, but Reporting Services and Integration Services still, you have to go through the SQL installer. Do you see, this is just conjecture on our part, we’re not saying this is going to happen. But could you see a day when there is an Installation Services installation or SQL Server Reporting Services installation that are kind of their own products.

Cesar: Yes and no. It’s kind of tricky question because if you tried to search for different tools or different options instead of Integration Services you will find Data Lake. So basically, it’s like Integration Services is the big guy and Data Factory is the newborn baby and it’s just starting to grow and understand what is the business and how to do everything. It’s really strange or really difficult for a new tool to handle or to be a stand-alone tool. You can see that. If you learn to create Integration Services in 2005 and you go to 2016 or 2017, you will find that basically it’s the same. There is only a few changes. Why is that? Basically, because Integration Services is a great tool, have a lot of different features, and basically do what needs to do.

Carlos: Yeah, so I guess I wonder, and one of the reason I guess I give pause to that, and again I’m not saying it will become its own product, but with the advent of Polybase, right? We know that Polybase is going to start connecting to more than just Hadoop. Might you see something like an integration or that middle tier which is what SSIS does, right, to kind of become its own, “Hey, I don’t rely on SQL Server necessarily, anymore. I can start doing integrations with different technologies”. I don’t know, it should be interesting.

Cesar: Yeah.

Steve: Okay, so, what is the best piece of career advice that you’ve ever received?

Cesar: Ok. When I started working with Business Intelligence here in Costa Rica, it was really hard to find, it’s like, “Ok, I am a BI guy.” Basically I just decided, “Ok, I go back to be a software developer, start searching for new opportunities.” I just have objective to be a fulltime BI developer. This is something that, it’s advice for myself, it’s something just to go on with what you want to do and search for a new specialization. Again, I will say I decided that my specialization will be business intelligence but I can choose as well software developer. I can be an architect or whatever as well in the software area. So, just search for specialties.

Carlos: Once you know what you want, go for it.

Cesar: Yeah, exactly.

Carlos: Our last question for you today Cesar. If you could have one superhero power, what would it be and why do you want it?

Cesar: It’s a really strange question. You know, I really love the Marvel superheroes and one of my favorites is, I have two maybe, probably. The first one is Wolverine. Doesn’t have any reason why, just because he’s cool.

Carlos: Yeah, kick butt and take names.

Cesar: Yeah, exactly, and some kind of immortality or something, but it’s not real immortality, it’s like getting old really go slow–

Carlos: Like auto-healing.

Cesar: Yeah, exactly, the healing factor and stuff like that. And the other is Dr. Strange. Some mystical stuff and doesn’t have any relationship with SQL Server. Is that an issue?

Carlos: No, that’s ok. Awesome. Well, thanks so much for joining us and of course we look forward to working more with you and having you back on the podcast.

Cesar: Thank you.

Steve: Yup. Thanks for joining us.

Carlos: Ok, so Steve and I need to brush up a little bit more on our Business Intelligence and Cesar is going to help us do that. Of course, if you have other topics or suggestions for things we should be talking on the podcast you can reach out to us on any of the social media options. You can also connect with us on LinkedIn. I am @carloslchacon.

Cesar: You can find me at LinkedIn @cesaroviedo.

Steve: And I’m on LinkedIn @stevestedman, and we’ll see you on the SQL Trail.

Episode 106: Temporal Tables

User: What was that record before it was updated?
Me: I don’t know.  Maybe I could restore a backup?
User: Really? You have to do all that? It is just one record.
Me: (under my breath) Yes, a record YOU updated.

If you have ever had a conversation like this you know how difficult it can be to implement auditing of records as they change.  With temporal tables, a new feature, we have the ability to track point in time information about a record without the huge expense of setting up auditing and tracking.  While you won’t want to use this on every table, when there are tables with sensitive data you want to audit from time to time, this feature will come in handy.  We are happy to have Randolph back on the program with us to talk about this new feature.

Episode Quotes

“That ability to kind of see what was there before it was updated, that seems to be the focus of temporal tables.”

“It’s all effective dated, so every row on a temporal table has to have a start and an end timestamp.”

“You don’t know how much space it’s going to use until you start doing some actual real life testing… That’s the biggest limitation for me.”

“When you create a temporal table the history table only takes the clustering index, so mainly the primary and the structure of the table.”

Listen to Learn

03:28 Temporal Tables
05:44 Auditing relative to user tracking
08:20 What are other reasons that people putting in temporal tables?
09:49 Start time and end time parameters for temporal tables
11:39 Using system_time in querying
12:50 Recovering data
13:35 Flushing the history table
14:49 Temporal Table in Management Studio
16:17 History table and clustered index
18:01 Modifying/Altering Temporal Tables
21:17 Data consistency check dealing with corruption
22:44 Scenarios where SYSTEM_VERSIONING = ON
27:50 Other limitations of temporal table
28:39 Is system time applicable when querying joined temporal tables?
31:06 When should you use temporal tables?
32:23 Tips on current feeding side of temporal tables
43:52 SQL Family questions

About Randolph West

IndexesRandolph West solves technology problems with a focus on SQL Server and C#. He is a Microsoft Data Platform MVP who has worked with SQL Server since the late 1990s. When not consulting, he can be seen acting on the stage and screen or doing voices for independent video games.

Transcription: Temporal Tables

*Untranscribed introductory portion*

Carlos: Ok. Well, Randolph, welcome again to our program.

Randolph: Thank you for having me. I’m surprised I survived the last round but this is going to be interesting.

Carlos: Yes, we actually split that into two. We had you on for a double header and so it’s been about 30 episodes and so we wanted to have you back. Of course we’ve been planning with you for the upcoming conference. The Companero Conference and so we’re excited to have you speak there.

Randolph: I’m excited to speak as well. It will be very cool because the cloud is all very rara shiny but some reasons why you shouldn’t just jump into that.

Carlos: Sure. It’s interesting you bring that up, so I just happen to be reading today, so Microsoft used to be $4 billion, they said $15 billion last year in the cloud.

Steve: So we will be covering how to appropriately take advantage of that $15 billion.

Randolph: That is exactly right.

Carlos: Being milked for a big percentage of it.

Randolph: Well, some of that budget is going towards trying to convince people to go across to the clouds so it’s an interesting time.

Carlos: That’s true. But today ultimately our topic is temporal tables.

Randolph: Temporal tables, yup, one of my favorite features of 2016 and upcoming 2017.

Carlos: Right, and I think ultimately or what we’re kind of getting at here is they try to give us some additional ways to do auditing, right? They tried with some like change data capture and maybe even some other components. I think we all have a lot of the home grown components for audit but that ability to kind of see what was there before it was updated and whatnot. That seems to be the focus of temporal tables. Would you agree with that?

Randolph: I would agree up to a point because temporal tables do not keep track of the user that performed the change.

Carlos: Got you. Ok, very good. So that is kind of the distinction when we talk about the word audit, that the person itself was not captured only the value.

Randolph: Yeah, so temporal tables I’ll quote kind of what Microsoft is saying is that it is for auditing in forensics but you don’t know who the user is that made the change so you will still have to keep track of that. But it also helps, and this is where I really like it for, data that is accidentally change or accidentally deleted. Plus it also provides a point in time analysis of what the data looked like in the past, so you could say, “Show me what the table looked like at midnight three weeks ago”, and then I can see what changed during that time using trend analysis or some type of complex query and we can analyze what’s happening over time using the data that’s in the history table compared against the data that is in the current table.

Steve: Yup, and you know that’s a real important feature there I think would be on the auditing side of it because so often you’ll get a report that shows like the last 30 days sales or something like that and then once a couple of days have passed it may have change drastically and oftentimes you don’t have a good ways of seeing what that looked like last week, or the week before for comparison and temporal table seem like a great way to be able to do that.

Randolph: That’s why it’s my favorite feature of 2016 because now exactly that you can back in time and say, “Hey, this record changed.” And we can see exactly when it changed and if you are keeping track of your logins and things like that this is where the auditing comes in but also with the tracking changes in a more sensible way. There are obviously limitations to how this works because you’re eventually going to run out of space but we can get into that if you like.

Carlos: Do you think, I guess, are those features going to continue to be separate do you think and I guess we’re kind of prognostic getting a little bit here, right? Does the user going to make it to the temporal table feature or is that going to be remain outside of what this feature set is.

Randolph: I think if they want to pursue an auditing feature then they do need to include the user at some stage. Saying that though there are a lot of systems that don’t use system IDs for tracking users. They have their own. For example, when you log-in into a web application your user account may just be a table entry inside the user table so a row and a table. There is no necessarily way a good way of checking which user that is from the temporal table point of view because it’s not a system user that accessing it because there is only one system user if you like. They probably will bring it in eventually. I don’t know. I’m no using any MVP magic here. I don’t know. But they probably will bring it in but at the same time you should be checking that anyway if you have your own table of users that you’re using to check logins. I think you can combine the two and see who’s doing.

Carlos: Well, that’s an interesting point, I guess one that I haven’t considered so having not use temporal tables before. But, yeah, that idea of let’s just account has access to the application but then the application talks to the database with a single account. That architecture, that should change a bit to take advantage of some of that, right?

Randolph: Well, I don’t think you have to change the architecture. I think you just have to be aware that the users accessing the data are internal to the database or to the SQL instance and in that respect you would have to keep track of them separately. Your architecture must have to change but I don’t see temporal tables’ architecture changing if they do add system user tracking at some stage.

Carlos: What are other reasons that people putting in temporal tables?

Randolph: The big reason for me is history, keeping track of history. So one of the systems I built awhile ago was to keep track of all records changes that happen in the system. So whenever a row is deleted or updated, I keep a copy of what it’s used to look like in an auditing table and then if I ever needed to go back into the history I would have to go and query with horrific XML queries to try and figure out exactly what is going on. Because for example if you’re keeping track of all the history for all the tables, you can’t just go and create a beautiful table structure on that, you have to use XML. You’d have to shred the XML to go and read that, and that would either require an index that is 5x the size of your data or a very slow query as it goes read everything. So what this does is in a way that you would create a trigger on a table to keep track of the old row and then store it somewhere. This does exactly the same thing out of the cover. So when you create a temporal table and then you create the history table that is linked to that whenever a row is deleted or updated in the current table the old row will be kept in a history table just like you would do with a normal trigger. The functionality out of the cover is exactly the same.

Carlos: The advantage is it’s going to give you a little bit more detail. It’s going to add that begin and start date or the date parameter too. It’s going to take care of all that.

Randolph: It’s all effective dated, so every row on a temporal table has to have a start and an end timestamp. It’s datetime2, the full 8 bytes, so datetime2 goes down to 100 nanoseconds. That is the smallest granularity and that’s the granularity you have to use for system date times for the temporal tables. So there is a start time and an end time. When you create the table the first time your default value for your create time will be system date time, so current date time if you like. And then the end time will be some time in the future that you’ll never see in real life so the year 9999 December 31st for example. And then if you see a row in the history table that has an end time and then you do not find that row in your current table that means it was deleted. Otherwise if the row exists and the current table it was never deleted, it was just changed. So a little bit of trickiness to understand how things work but it’s all date time specific.

Steve: So then take the example of someone who is not quite on SQL Server 2016 yet and they want to use something like this. They could just create the same thing with their own trigger and with the similar naming convention on the start time and end time and all of that. However, where we get challenging it seems like we haven’t talked about querying it would be when you’re trying to query that data.

Randolph: Well yeah, so the query uses a different type of querying than we’re used to. There is a keyword that you have to use, system_time, so you have to use the keyword system_time in your query so, SELECT whatever from the table FOR SYSTEM_TIME, and then you can do an is at a certain timestamp. You could use a BETWEEN. There is a number of different things that you can use that will automatically figure out the timestamps are between the current table and the history table.

Steve: And to me that seems where some of the real value is and that even though it just triggers under the scenes and all that the value is the way that SQL Server has built that in to be able to query for specific point in time or range or set of values.

Randolph: Yeah, and that’s a value for me as well so I can go and see exactly what the data looked like at that specific point in time down to the 100 nanoseconds. So if we find out that somebody deleted a whole bunch of rows we can go back to the exact point in time before it happened. We don’t have to go and restore an entire database or a file group restore for doing Enterprise edition. But we don’t have to do that anymore. We can now go into the history table, recover all those rows, then flush the history table because otherwise you’re going to have 3x the data that you originally had, and I’ll explain that as well. So if you’re recovering data you go into the history table, you get those rows out, you flush the history table, and then you input the rows back in to the current table then you’ve got your consistent data before the delete happened and you reconnect the history table to the current table again.

Steve: Ok, so you use the term flush the history table there. What exactly does that do?

Randolph: Well, the history table will grow and grow and grow until you are run out of disk space. That’s by design so there is no aging out of the data. You have to do that yourself which is good because if you’re auditing you decide how long you want to keep your data for. Same that though the data is stored in a compressed formats. So they use page level compression on the history table. And because since 2016 Service Pack 1 we don’t have to worry about compression because it’s part of the product in Standard Edition as well so we can have page compression for the history table so it will use less space than it would normally which is one of the advantages of the feature on 2016.

Carlos: Now, remind me, because I guess one of the things that we’ve kind of implicitly said here is that we have a table, now we want to add this history to it. We’re going to use an ALTER Table that’s going to change and we’re going to say, “Hey, this is now the temporal table.” It’s going to create this history table. Now, we
mentioned querying it but from seeing that history table like in Management Studio, is that visible to me or is that still an object that I’m managing or is that kind of just behind the scenes.

Randolph: You can see a temporal table in Management Studio as long as you obviously have the right permissions to view them but it will show up as what’s called a system versioned temporal table because the system is versioning the table. I mean, this is Microsoft’s naming conventions between O and love. What you’ll see in Management Studio is you’ll have the current table. Let’s say for example, dbo.Account, and then in brackets behind it will say system versioned, and then if you expand that out it will show you the history table under that first. And then it will show you the columns keys constraints and all of that stuff that you’re used to. So the history table, you can name it yourself, and put it in its own schema which is what I recommend. Let’s say you have a file group that is on slower storage or cheaper storage whatever and then you put your schema attached to that file group and then you put the history stuff inside that schema then you can manage the history in a different file group which might cost you less money. It will be a little bit slower but it’s history data so you would not have it in the first place.

Carlos: You’re not expecting super fast performance.

Randolph: Exactly. And because it’s compressed you might actually offset some of that performance problem by having it compressed and then you’ll have the list of columns that would look exactly the same as they would in the current table except you cannot obviously have a clustered index. I’m going to change that statement. You can have a clustered index but you cannot have a primary key in the history table because the primary key is in the current table. So the history table does have a clustered index it just doesn’t have a primary key because the primary key is in the current table. Otherwise, the structure is identical.

Steve: And I think the key behind that is that when you’re inserting into that history table if you had the primary key on the same thing as the original table it wouldn’t allow you to put those multiple rows in there.

Randolph: Correct, so what’s happening in the history table is that you have a clustered index that is just not unique because it’s in the pinned only way of writing to the history table.

Steve: So then if we’ve got that history table and I want to clear out some old history in there can I just go in and delete data out of that table?

Randolph: You have to break the link between the current and the history table to be able to do that which isn’t complicated at all. There is just a command and if your editor will bear with me. It’s probably going to be Carlos, right?

Carlos: No, Julien.

Randolph: Hi Julien! Bear with me here.

Steve: ALTER TABLE table SET (SYSTEM_VERSIONING = OFF).

Randolph: Yes, that’s the one, Steve. So modifying temporal tables, you have to have administrative rights to be able to alter a table. You need the admin role to do that. So you cannot alter the objects until you have administrative rights so that’s the first thing. Anybody can read and write from the temporal table and write to the temporal table but only the administrator can modify it. And what you do there is: ALTER TABLE name of the table SET (SYSTEM_VERSIONING = OFF). Then you can do whatever you like inside the temporal table with the full understanding that your auditing trail is now broken.

Carlos: Sure so almost like a maintenance window type.

Randolph: Exactly. So this is exactly the same thing as changing your database from full login model to simple login model is the recovery model so now you have basically truncated or change your history so now you will have to recreate that connection once you’re done flashing the table or deleting rows from there that you need or changing column names or changing columns themselves, deleting columns, updating columns, whatever. Because you may change the structure of your current table and you want to have that change reflected in the history because unfortunately although you can do an ALTER TABLE which changes a column or drops a column on the current table that will take place on the history table. But if you have decided to keep the history of a certain column and you do an ALTER TABLE on the current it will drop that column from the history which you don’t necessarily want. Then you have to break the connection, go into the history table, SET that column to NULL, and then recreate the connection again once you drop the column from the current table. So that way you can keep history of what the column used to have in it but then the column does not exist anymore in the current table so you still got that history retained. Does that make sense or does it just confused you all?

Carlos: No, no.

Steve: I think it make sense.

Randolph: Ok, good. There is also another feature which I have never turned off. When you create the SYSTEM_VERSIONING on a table you can alter an existing table and create a temporal table out of it, which Carlos alluded to earlier, or you can create a new table from scratch which is a temporal table. And to change an existing table you have to ALTER the tables SET (SYSTEM_VERSIONING = ON) then you can assign it to history table name if you want to create something in your own schema that we spoke about. And then there is another feature called data consistency check = ON. The documentation says this should always be ON to ensure that the history table follows the correct structure format as the primary table. I’m not sure why you are going to turn that OFF. So I’m not sure why the feature is even presented as an option.

Carlos: Right, because you made a change to that table, data type change most likely or I guess you can add a column.

Randolph: But why have the history table if you’re not checking the data consistency? I’m not sure why that’s there but I’m just letting you know that it is there. It is required with the ALTER TABLE statement if you’re going to use SYSTEM_VERSIONING = ON but I’m not quite sure why it’s there.

Steve: So one thing and I haven’t use the data consistency check one way or the other but one thing I was reading about on a blog post was that it was there to help deal with corruption at some point which immediately I perked up a little bit and thought, “Well I need to understand this one.” But I haven’t had a chance to dive into that yet. Have you heard anything about using that around corruption?

Randolph: I have not but I would imagine that if you have a corrupt table, and this is all conjecture on my part. If you have a corrupt table and you have a history table that contains all the history, the history should probably be fine because it’s stored in different file group for example or different part of the file anyway. You would have access to that history without necessarily reading or writing whatever is in the current table using just standard queries. I believe that the work was a bug in one of the Cumulative Updates did fix a bug in data consistency check for certain tables, so I’m not sure if that’s what you’re referring to, Steve.

Steve: Yup, and I guess that’s something that I need to look into a little bit more.

Randolph: Because I do know that there was in Cumulative Update 1 there was a bug with data consistency check so maybe that’s what you’re referring to.

Steve: So let’s take an example here then. Let’s say we create a table and we have SYSTEM_VERSIONING = ON and we have the history table associated with that then we insert a single row into that table. Does that row then exist in the table as well as the history table at that point or does it wait to get to the history table until it ends up being changed?

Randolph: An INSERT will only affect the current table. The only time that the history table is involved is if you modify that data. So when you insert a new row into a table there is no reason for it to have changes at all so you’ll only see it in the current table and then when you modify it the timestamp will be kept track of and the original row with the start time that you created the row in and an end time of when it was changed will show up in the history table.

Steve: Ok, got it. So then if you’ve got a table then for instance it’s something like bank account transactions which is probably the type of table that’s not going to change frequently. It will have a lot of inserts but very rarely I would hope our transactions in the past being modified.

Randolph: Well, I’m going to stop here and say that’s a bad accounting practice if you’re changing because you should be posting changes in an append only model with accounting.

Steve: Sure, exactly. But if you wanted to track to see if anything was actually changing it, you could turn this on, go back and look and see what point in time things were changing if there were. But then I was going to sort of flip it around to a different example of let’s say you have a table that’s like your users table and that user has username, and a last login date where every time that user logs-in it will update the last login date. That would be one that if temporal tables were turned on every time the user log-in it would be inserting another row into that history table, and it could grow very quickly over time.

Randolph: It could, yes. And that’s why the page compression is going to be very helpful from that point of view because that data should compress very well. But saying that it may not be a good choice for a temporal table although it might be a good choice if that’s what you want to do is audit and see how often people are logging in and things like that.

Steve: Right. So then in the case that you have a very narrow table, maybe there’s just username and last login date, something like that, primary key, whatever you need there too as well but if that’s probably going to work better in a history table than if you’ve got a user table that might have a 150 user attributes to find in different columns. Again, a poorly defined table there because when it logs to the history table, of course it has login the entire contents of the row. Is that correct?

Steve: Correct, so wide tables are going to be slower because it’s exactly the same effect as if you were taking from that and using a trigger to write it to a different table. It’s still going to be a wide table, possibly wide columns as well so it’s going to take awhile to write and you are going to have a performance impact on your current table because it has to commit the row to the history table first obviously. Let me just confirm that. Actually I do a blog series in November 2015 so it’s fairly fresh in my mind. I just want to double check here. The row will be modified in the current table and then it will be modified in the history table. It’s part of the same transaction though.

Steve: Alright, so are there other limitations that people should be aware if they are going to try this out. The big thing for me is running out of space because it’s very easy if you’re not use to what kind of data or what the…

Steve: I think we’re relying on Julien a lot on this one, Carlos.

Randolph: Well, how many times did we pause the last one? That was pretty crazy.

Steve: Quite a few.

Randolph: Can you repeat the question, Steve.

Steve: Ok, what I was just going was other limitations?

Randolph: The big one for me is the disk space because you’re going to run out of space quickly. The other problem is if you are using this in a staging environment and you have more than one table with a foreign key relationship and both of them are temporal tables. You’re going to have a hard time trying to pin down history queries to say what the data looked like at this specific point in time because of those foreign key relationships so it gets a little bit hairy in that respect. You don’t necessarily want to have every table in your database a temporal table because it’s going to make your queries complicated and it’s also going to slow you down because it’s going to be querying the history over time.

Steve: Ok, so in the SELECT statement then when we were querying for temporal data where say FOR system time as of and specific date time too. Now, if you do that in SELECT FROM one table INNER JOIN another temporal table, so we have two of them there. Is that system time parameter going to apply to both of the tables and the join?

Randolph: What a fantastic question. I’m going to say that that time is specific to each table. What I would do is I would run an experiment and then find out because I don’t know.

Steve: Ok, it’s one of those new features where there is a lot of experimentation to do there I think.

Randolph: Where the challenge comes in is you may have history on a lookup table for example. So the lookup table has a whole bunch of stuff that might change once in a while and you don’t necessarily look at the history when you’re joining that to something else, but you might have to do that and it gets a little but hairy. It’s not difficult, it’s just time consuming for the engine to go back into the history table as well, and then because when you do a temporal query. I’m going to call it a temporal query and not a history query. When you do a temporal query against your table it’s going to combine the results from your history and your current. If you’re asking for everything so it’s going to do basically UNION ALL or a UNION because the data shouldn’t be the same. When you combine the two, if you’re going to do a point in time query it’s going to go and look in the history to see if anything is there so it’s always looking in that history table. So if that is on slow performing hardware then it’s going to take longer, that’s kind of where I’m getting to.

Steve: Ok, that makes sense.

Randolph: I was just going to say when should you use temporal tables? You should use temporal tables on data that needs to be tracked and that you were tracking in an interesting way before. For example, my system that was keeping track of the XML data version of history. I needed to check that for auditing reasons. That is a good reason to use temporal tables. But I wouldn’t just go turning it on everywhere because eventually you’re going to forget and eventually you’re going to run out of disk space somewhere.

Steve: Yes, and hopefully you know about that before you run out. Hopefully you have alerting in place.

Randolph: Right, you should have alerting in space but if you’re playing with it on a test environment or something and you forget about it. You may need a massive amount of space to handle that data. Some people might have to retain 7 or 10 years of history. And then you may have to age in the manual process anyway. So you don’t know how much space it’s going to use until you start doing some actual real life testing. That’s the biggest thing that I have to say. That’s the biggest limitation for me. I think I said that a hundred times.

Steve: How about any other tips on sort of current feeding side of temporal tables?

Randolph: Well, back to the disk space that’s being used. If you are creating a lot of churn and you’re temporal table is growing excessively you may want to think about going into the history, breaking the connection and deleting or aging out data that is older than a certain number of months or years. And it’s got indexes on it so you might want to do some index maintenance as well. You can apply indexes to columns inside the history table manually if you like by the way which is pretty cool. But you cannot change the clustering key so you may want to do some index maintenance from time to time on the history tables.

Steve: Ok, so if you have a history table associated with your temporal table and you add indexes, non-clustered indexes to that for instance, do they automatically end up in the history table or do you need to go and add those manually if they are needed?

Randolph: When you create a temporal table the history table only takes the clustering index, so mainly the primary and the structure of the table. They should be no reason for it to take the indexes across because that’s additional maintenance that’s not necessary because you’re not necessarily be querying that table a lot. So it will only copy the structures, so the columns themselves and whatever your primary key is will become a clustered index on the other side. There will not be any indexes that get copied across. If you do want to query that table frequently you would have to add your own indexes.

Steve: Right, so in that example if we are querying for a certain point in time and filtering on a column that needed an index, if that history table got large it could be quite slow if we don’t add that index themselves.

Randolph: Yes, and there is a whole can of worms here with filtered indexes which don’t always work the way you expect them to which we won’t get into. I think we covered that in Episode 81.

Steve: Yeah, great, great. So any other current feeding tips to add at this point?

Randolph: When you make a change to your current table the change if it is structural, so you’re deleting a column or modifying a column may affect the temporal table as well, the history table. And the reason for that is because it wants to keep a consistent check between the two. That’s the data consistency check that’s happening. So if there is data in the history table that doesn’t satisfy the requirements of the new column, for example you may have just deleted everything from the current table and then alter the column. If it is attached to the history table still and you do and alter column and it then tries to change the data type, if the history table contains a data type that is incompatible with the new change it will fail.

Steve: Oh, so you either purge the history table or know that column in the history table perhaps in order to be able to modify it in the original table.

Randolph: For me personally I would break that connection completely, rename that history table so that it’s archived and then create a new history table going forward if the data types are that significantly different because effectively it’s a new table. If you do want to go and query that history you still can by querying the history table directly in the archived version. But if you’re going to be making structural changes to your current table, I would break the connection, rename the history table and just create a new history table going forward. The other thing that I wanted to mention, we touched on very briefly earlier, is if somebody deletes, let’s say runs a delete statement against the current table without a WHERE clause so every single row gets deleted. What you will find is that the original version of that row for every single row will find its way into the history table. So if you’ve got 10 rows, that’s fine. If you’ve got 10 million rows, you’re going to have 10 million rows in your history table as well. So if you want to recover that data you can just copy that over from the history table into the current table but I do recommend breaking the connection first because any modifications you make to any rows after that will also be written back to the history table and you still got millions or 10 million changes in that history table. So either flush those rows out first or rename the table or do something but don’t try and go and copy those 10 million rows back into the current table from the history table without doing some maintenance on the history table because otherwise you’re still going to have those 10 million records there waiting in the history forever and ever.

Steve: Yup, so it really seems like it just keeps coming back to the history table can get really big depending on what you do and it’s maybe slow if you don’t index, it may run you out of disk space. I mean there may be any number of issues associated with that history table becoming gigantic.

Randolph: Well, it’s a usual story, how much are you querying it? Does that mean we’re going to be using a buffer pool for that table? All of those things that go along with that because even though it is compressed on disk whenever you read it into memory it’s going to be uncompressed.

Steve: So another question around this. Let’s say you’re querying your original table and you’re not using the system_time for system time parameter, you’re just doing a direct query of your source table and your temporal table without the history. Does that work exactly the same as it did if you did not have a temporal table?

Randolph: That’s next in question because it brings up another thing that I didn’t mention. Yes, it works exactly the same as if nothing had changed. So when you create a new temporal table you can create the start and end time using the datetime2 data type and you can mark those columns as hidden. There is another feature in 2016 for hidden columns and this is specifically for supporting this feature. So you hide those columns so that any queries that do, and you shouldn’t do those what people do a SELECT *. Those columns will not be visible in a SELECT *. They will not be visible in any existing applications that don’t know that they are there because the system is managing them. It’s as if you’ve created a column with a default value because that’s effectively what it is.

Steve: But those have to be specifically marked as hidden for them to not show up in the SELECT *.

Randolph: Correct. But if you do that and you can do that very easily when you’re doing the CREATE or the ALTER table to make it a temporal table, you can mark those columns as hidden. So it has no effect on any current applications or any queries that you might be using currently. It will work exactly the same way as it did before. So unless you’re using that system_time query keyword it’s going to be carry on reading from the current table as if nothing else has happened.

Steve: Ok, great.

Carlos: Good deal, lots of good stuff today. As we’re going to finish up here, I know we’ve done SQL Family before, but can we go ahead and hit it again?

Randolph: We can do it again because my answer may have changed.

Steve: So Randolph how did you first get started using SQL Server?

Randolph: Accidentally, as with everybody I know. Nobody I know has ever wanted to be a DBA and then they realize they’re being silly because it’s actually what they wanted to do all along. I used Access, I used MySQL, I used Ingres. I used SQL Server when I was working with PeopleSoft back in the 90’s. I didn’t know what it was. It’s just one of those things, oh this just keeps your data, it’s in an Excel
spreadsheet basically. Lots and lots of Excel spreadsheets. And then I started reading about corruption because that happened to two of the clients. The databases either got dropped or corrupted in some way and I start of getting interested in that way. And basically it’s just, it was there, it was there, it was there and let me have a look at it and then I started loving it. Access was really my entry point into SQL Server because Access was so terrible.

Steve: You know, that’s so interesting because at first glance if you don’t know databases very well, Access looks pretty darn cool.

Randolph: Access has some very nice features for front end development for a data backend. Well, it did. I mean it is 2017 now.

Steve: Sure, but back in time it did.

Carlos: Well, they have it all in one application too, right? Two things.

Randolph: Exactly, and it was easier to use in FoxPro. It was cheaper than FoxPro and you didn’t have to be a programmer to write a query because it had that cool query designer that you could do. I believe SQL Server has the same thing but I never used it. It was just really neat to be able to picture your data with a graphical interface.

Steve: I wonder if Microsoft realized that access is really a stepping stone to get to SQL Server from… I’m sure they do.

Randolph: It’s a gateway drug.

Carlos: If you could change one thing about SQL Server what would it be?

Randolph: Last time I think I said that I want to do a TempDB for every database. I still kind of think I want that but I’m not sure if that’s the one thing that I want to change now because firstly Azure SQL Database has come a long way and in a very short time so if I would change one thing. It would be making Azure Database cheaper. Because I think there is a lot of potential for people to use for single application usage. For example using an Azure Database to replace what people use MySQL for now and that would be web applications and things like that. If it were a little bit more cost effective with all the feature set that they have because everything that’s in SQL Server 2016 is in Azure SQL Database including temporal tables on the Premium Edition and in-memory OLTP and all those things. So I think temporal tables are actually supported on the Basic Version for what it’s worth.

Carlos: I think you’re right. It’s interesting because you look at the, what’s the edition that only allows you to go 10GB.

Randolph: That’s Express.

Carlos: Express, thank you.

Randolph: I have a funny story about Express off the topic. A customer of mine had a customer because they are a vendor. A customer was running their product in Express Edition. They had 20 databases and of course Express has auto close enabled by default so a typical diagnostic session which takes 5 minutes normally run for 2 hours because it had auto close on every single database. So every time a query run there we close the connection then open it again for the next query. It was fantastic. So what were you are going to say about Express Edition?

Carlos: Well, I guess I’ll say because it’s free, right? At least my impression is of making like Service Pack 1 some of those analytic features available in the Express Edition. That is a direct response to MySQL ecosystem, so I think to your point of like, “Well, hey that’s free.” Now there is compute, right, so I guess there should be some cost there. But a database under 10GB should be pretty close to free, right?

Randolph: Agreed, but…

Carlos: That might be the change there. But I don’t think a lot of people are going to take that into consideration because even if they had to pay a penny it’s still I had to pay something and this is “free”.

Randolph: Exactly. The other cost with Express Edition which I love is the 1GB limit of memory so non sense.

Carlos: Oh right, I forgot about that.

Randolph: MySQL doesn’t have that limit obviously but it is a strong consideration for not using Express Edition. You’ve only got 1GB of RAM.

Carlos: Is that still a case in 2016?

Randolph: Yes.

Carlos: Ok.

Randolph: 4 CPU cores or one socket which are very small, 1GB of RAM and 10GB database size.

Carlos: Got you.

Randolph: I would like to see a better costed version of Azure SQL Database for the Standard level if you would like. And get rid of Basic because I don’t see the point of Basic. 5 DTUs isn’t enough for anybody. I’m sure I just coined a new phrase. And then just have Standard and Premium, and have the Standard do more for less basically because I think the adoption would be higher if they did that.

Carlos: Well, they’ve been coming down. I know the sizes have been getting larger and so, yeah, and so maybe another year or so we’ll get there.

Randolph: Maybe by the time this podcast goes up.

Steve: You’ll never know. What’s the best piece of career advice that you’ve received?

Randolph: I’ve been working for so long I probably forgotten most of what I’ve heard but the one piece of advice I give to people because I do mentoring as well is to say, “Be honest”. So if you screw up tell somebody that you screwed up. It’s much easier to handle a problem if you’re completely honest and say, “This is what happened, this is how I created this problem.” And then it’s much easier to fix than lying and say, “Oh, I don’t know how that happened”, or “I did not kick the power supply out of the server”, or “No, I did not accidentally bumped this hard drive off the disk”, you know stuff like that. Just be honest and it’s just easier for people to handle the fallout. And if you’re honest it shows that you are human and there is nothing wrong with doing something wrong. I make mistakes all the time the difference is I own up to them. I’m not like, I’m not going to say it. I’m going to get political. I’m going to stop here.

Carlos: But it’s interesting on that one because I’ve heard people say that they are sometimes afraid to be honest because the environment they work in is too aggressive or management doesn’t have your back on anything.

Randolph: Well, then I wouldn’t be working in that environment. And being honest and saying, “Yes, I did this”, gets you fired. Well, then any place is better than working in an environment like that. Toxic work places are very bad for your health and you will die young.

Steve: Yup. And I think what I was going to say around that is that if you’re on that toxic work environment, and I’ve been there in the past, honesty is even better at that point. I mean, it’s always good but it’s even more important at that point because you’ll know exactly where you’ll stand.

Randolph: It’s refreshing. It might not be the outcome you expected but it is refreshing for everybody to say, “Well, this person is telling the truth and they’re not going to compromise whatever for the sake of looking good.”

Steve: Yup, very good advice.

Carlos: If you could have one superhero power, what would it be and why do you want it?

Randolph: Last time I said, I have a superpower wherever I walk into a server room a machine will either crash or not, that’s a superhero power I didn’t want. But there’s been a whole lot of superhero movies recently, a whole lot of stuff that I don’t enjoy because everybody is destroying everything. The one superhero I do like is Batman because he is not a superhero. He is just a guy. He does have compromised morals and he’s very very wealthy. I don’t want to be wealthy like him because that also corrupts you. But I do like the idea of just being a decent person. I’m not necessarily killing bad guys but if you see something going wrong, speak up say something. You don’t have to be a superhero to be Batman. So you don’t have to be a superhero to say, “That’s wrong, this is not cool and do something about it.”

Carlos: There you go.

Steve: Yes. I like that.

Carlos: Thanks so much for being on the program today.

Randolph: Thanks for having me and I’m sure this would have been a lot longer without the edits. Thank you for making me sound good.

Carlos: No, it’s great to have you on again and chat with you. And yeah, we look forward to seeing you in October.

Randolph: Sounds good. I’m looking forward to the Companero Conference in Virginia, in Norfolk, on the 4th and 5th of October, 2017. And it’s a single track so you just have to stay in the same room and listen to people explain how awesome SQL
server and the cloud is.

Steve: We’ll see you there.

Carlos: That’s right, you took the words out of my mouth.

Episode 105: When is enough, enough?

 

How can you tell when a change is enough of a change?  How do you know you didn’t make too big of a change?  Steve and I discuss some thoughts around what you might look for when you make certain changes.  Some of the components we cover include Max degree of parallelism, cxpackets, virtual log files, the number of log files for a database, backup retention and memory.

Episode Quotes

“Then I think on the cost threshold for parallelism, again it’s going to take a little bit of knowing your system.”

“What it really depends on oftentimes in most businesses is how much money you are losing for every minute that that database is down.”

“If your system is performing great no reason to even talk about more memory unless it’s growing and you’re planning for the future.”

Listen to Learn

01:37 About the Companero Conference
02:54 Max degreee of parallelism, knowing the sweet spot
06:28 Max degree of paralellism , number of cores
10:08 CXPACKETS Waits
12:05 Cost threshold for parallelism
14:54 Virtual Log Files (VLF), growth and issues
18:53 What makes VLFs a bad thing?
23:23 How do you know if you have enough, not enough or too many?
29:00  Number of log files for a database
30:39 Backup retention periods and scenarios
37:35 More on memory
41:50  Page Life Expectancy

Transcription: When is enough, enough?

*Untranscribed introductory portion*

Carlos: Companeros welcome to Episode 105. It’s good to have you on the program again today. Steve, how is it going?

Steve: It’s good. Today’s topic is on when is enough enough. So Carlos, when is enough enough.

Carlos: Yes, that’s right, so this question was asked to me personally during a session as we were talking a little bit about parallelism and the change made to the cost resolve for parallelism. So the question was, ok now that you’ve made this change how do you know that you’ve raised it too high and so I thought we could talk a little bit about kind of extension of this idea from base lining which we’ve talked a little bit about in previous episodes, kind of best practices and now as we start putting in some of these things some of the sweet spots if you will and how we know that we’re in that sweet spot things to look at. So that’s kind of our topic for today.

Steve: Alright, so do we have any companero shout outs to mention?

Carlos: We do. We want to give a companero shout out to Daniel Temensken. He is a PFE at Microsoft. Daniel says, “Thanks for what you do.” He tries to plug-in when possible for us to keep up the good work, and so Daniel we’re thankful you Microsoft guys are keeping an eye on us and keeping us in line.

Steve: I don’t know about the keeping us in line I haven’t heard any feedback on that side yet. We’ll see.

Carlos: It’s only good and I don’t filter all of that to you, only the bad stuff. Ok, we’re still planning for the Companero Conference in October and one of our speakers, Mindy, she’s been on the podcast earlier and while we were chatting with her she gave us a little information about herself and what she’s going to be presenting at the conference.

Steve: Yes, I think one of the things I like about the conference is the whole single track concept and I’m going to get to see all of the presenters this time. Well, I’m not going to see me unless there is a mirror. But I’ll get to see all the sessions and I’m looking forward to seeing Mindy’s session as well.

Carlos: Yeah, that’s right. I think it’s going to be good. Again, just the ability to be able to review kind of see things from a different perspective. I think it’s going to set up a lot of questions for people to continue their learning. So let’s see as we get into the episode today. The show notes for today’s episode will be at sqldatapartners.com/enough.

Steve: Or at sqldatapartners.com/105 for the episode number.

Carlos: Yeah. I guess let’s go ahead and jump into this conversation. We have a couple of items or setting if you will that we’ll talk about. The first one, let’s go ahead and kind of tackle the max degree of parallelism. How do we know when we’re in a sweet spot?

Steve: Alright, time out for a second.

Carlos: Sure.

Steve: I’m getting a bunch of noise coming from somewhere. Hold on I need to close the door or something.

Carlos: Sure, sure.

Steve: Alright. It’s my neighbor has a leaf blower. Can you hear any background noise because of that?

Carlos: Yeah, I was going to say I could not hear it but I was talking at the same time, but nothing really stood out.

Steve: Ok, it sounds like he has moved around the other side of his house now. Sorry about that, so let’s jump back into the max degree of parallelism.

Carlos: Alright, I think this idea so I guess we’ll start with kind of the best practice and that is. What we’re looking at is. I guess we should start this conversation with all of these answers are going to start with it depends, right?

Steve: Yes, it does.

Carlos: And so we’re going to talk about some of these things in kind of lose general generic terms but that doesn’t mean you should necessarily adopt them as gospel. Obviously, your individual scenarios or your environments are going to
dictate some of this and so we’ll try to address a handful of them but we won’t be able to address every scenario.

Steve: Yup, but as far as max degree of parallelism, I don’t like to talk about that alone. I like to always talk about cost threshold for parallelism at the same time because I think the two of these goes so hand in hand. I think that there is a lot of confusion and just sort of downright bad defaults on both of this. And I guess both of these originated somewhere in the 90’s back SQL Server 65 or 7 or somewhere around there. They may have made a lot of sense 25 years ago. But the defaults you have on those today are not so great and I think that it comes down to the fact that hardware has evolved a whole lot since then.

Carlos: Yeah, Adam Machanic has his great line there where he says that these only make sense if you jump to a time machine and go back to guy who developed this at Microsoft, and use his computer to execute your queries.

Steve: Right, right. And unfortunately or maybe fortunately in this case we don’t have a time machine. So the default for the max degree of parallelism is zero which means unlimited parallelism. The default for the cost threshold for parallelism is five which means any query that has an estimated plan of greater than five it will be considered as a parallel query.

Carlos: Sure. The reason for max degree of parallelism is zero is because. As you begin to install SQL Server and you could think about it from their perspective, they don’t know how many CPUs you have, right? And I guess dynamically trying to figure that out and all the different scenarios that you might have as far as how much stuff you have on the box with SQL Server which I think is listening to a degree if you’re probably listening to the show. SQL Servers by itself, again there are going to be exceptions to that and so you’re going to see that conundrum that they are in there.

Steve: Yup, so the max degree of parallelism there is a lot of different opinions out there on what it should be set to. But most of them fortunately are kind of in the same range and typically what I see on this and what I like is close to, and I say close to because there is a lot of opinions there, but close to a max setting of 8. Or a max setting of the maximum numbers of cores you have or the max number of cores per numa node, and whichever number is smaller out of those.

Carlos: Right, so there is a little bit of Math involved there, right? It’s all the variations that are going to play into that. So taking into example that number 8, right, if you only have four CPUs, right you’re number is going to be four.

Steve: Right, but if you have four CPUs that are split amongst two numa nodes which usually numa is not that common with that few of CPUs. But if it was you would want to set it to two in that case. Sort of part of it around there, I mean usually with smaller number of CPUs like less than six or eight CPUs you’re probably going with the number of CPUs 99% of the time unless you have numa enabled. But when you get into SQL Servers with many CPUs this is where it can get really interesting. So if you’ve got a server with 50 CPUs, or 30, or 80 CPUs or even more than that. What could happen here is that a small query that probably shouldn’t go parallel could go parallel and could be run on all of those CPUs. I need to start saying this differently. They may be running all those cores because generally it’s not CPUs. It is multiple cores per CPU. So be run amongst all those cores so and that can be really inefficient. And generally what I see even on
servers with lots of cores is that 8 is a good setting. I mean it’s what a lot of people are recommending. It’s kind of the best practice to use there as a maximum because what you get into is when you have more than 8 there is a lot more work involved in bringing back all of the results from each of those different CPUs or different cores doing the work. There is a lot of work involved in sort of collating the results and waiting and if one of them takes a little bit longer there’s maybe a lot of waste going on there. You’ll see that show up because you may have a lot of CXPACKET Waits which maybe we should take a second to talk about that because CXPACKET is one of those wait types that is oftentimes there is a lot of misinformation out there around it. Some people say, “Yeah, I don’t have to worry about about CXPACKET Waits that’s just means that parallelism is occurring.” For the most part that’s probably true most of the time but you can get into a point where there are things that will cause excessive CXPACKET wait, one of them being incorrect settings on your parallelism. And when I say on your parallelism it’s really on your max degree of parallelism setting and your cost threshold for parallelism setting because those kind of go hand in hand there.

Carlos: I guess if we’re going to jump here so that question is we’ve made the change. How have you decided to make your change? You’ve now made it, how do you know, like how can you determine that you’re in a better place. I think there are a couple of components there. One I think the overall percentage of waits because CXPACKET will probably continue to be in there but you should see other waits to kind of float up. There should be a better mix if that’s the right word if for example CXPACKET waits was taken up 99% of all the waits. That should float down a little bit.

Steve: Yup, and usually what I look at there is as if more than maybe 50% of your waits is based on CXPACKET that maybe there is something there you should take a look at. If it’s more than 90 there’s probably there you should take a look at. If it’s up to 99% then there certainly something you should be looking at.

Carlos: Then I think on the cost threshold for parallelism, again it’s going to take a little bit of knowing your system but you just want to go and take a peek at, which is easier said than done but kind of getting in and knowing those queries what the big queries are and ensuring that they’re still parallelizing, right? And then do you feel like there are queries that are taking up lots of CPU that might benefit from parallelism and then again that’s kind of one of kind of decision as to whether try to tweak that or not.

Steve: Yup, and I think that there is kind of a balance there because you don’t want every query to go parallel. But the flip side of that is you don’t want every query to be single threaded either. And what the cost threshold for parallelism is really doing is it’s saying at what point, at how expensive of a cost are we going to decide that parallel processing in that query maybe better, and the default for that is 5. And if you look at most queries that you are running today I would assume that most people that are running queries or looking at queries for performance, they’re going to have a cost of more than 5. There sure there are some quick ones that don’t, and what that means is most of those queries are going to be processed in parallel. And that may not be a bad thing depending on your server but there are some queries that are low cost but they are greater than 5, so maybe they have a cost somewhere between 5 and 50 or 5 and 60, that they would probably run more efficiently if they weren’t ran in parallel.

Carlos: Well that’s the tricky thing I mean even though they exceed that threshold, parallelism is available to them but that doesn’t mean they will always run in parallel.

Steve: That’s right because you’ll end up with two plans. You will have a parallel plan and a non-parallel plan and with that the query optimizer will decide which one is better. I mean oftentimes from my experience it will go with the parallel plan at that point but not always.

Carlos: Got you. Yeah, again that’s another tough one. The answer there is that you’re
going to have to know a little more about your, watch your environment and see how the changed that you’ve made, how it’s affecting your numbers and then even drilling down to the query level if you will to determine if more changes are needed. So from there we’re going to jump into log files.

Steve: Alright, so the thing that usually comes up around the conversation of log files is virtual log files or VLF. And just a little bit of background before we jump into the enough is enough part of it but the way VLFs work is that as your log file grows, every time the log file grows additional VLFs are created or these are segments inside of that log file that are broken up that can be used individually one at a time. So when SQL Server is writing into the log and it always does this sequentially it will pick a next available VLF chunk that it can write to. It will fill that up and then move on to the next one, and the next one, and the next one after that. And those stay used, well if you’re in full recovery model, they will stay in used until you have a log backup. Now, if you’re in simple recovery model they will stay in used until all of the transactions that are in that like they are using that VLF until all the transactions have completed.

Carlos: And then that can be marked as “deleted” or being able to be over it.

Steve: Yeah, available for reuse basically.

Carlos: Available for reuse that’s a better word.

Steve: So what happens though and this part varies a little bit between different versions of SQL Server. SQL Server 2014 and 2016 did a little bit better job in how virtual log files grow but it used to be that when your log file grew if it was smaller than a certain size that growth would have four virtual log files associated with it. I think that size was around 256MB. Now, if it was between like 256MB and a gig you ended up with 8 virtual log files. And then if the growth was greater than a gig you ended up with 16 virtual log files. So with SQL Server trying to guess based off the size of that file growth how it could chunk that up appropriately so that you get sizes that would be more reasonable to use. And then with SQL Server 2016 and 2014 there was some changes around that so that when log files grow, with the smaller growth sizes you would oftentimes only get one VLF rather than several VLFs at that point. But the problem that you run into is that a lot of people or lot of databases have some growth settings initially and a lot of the defaults would either grow by 1MB or 1% with a starting size of 10MB. And as it grew you would end up with all kinds of really tiny virtual log files. And what that meant is that if you have a big transaction it would have to span multiple of these VLF files that were really tiny.

Carlos: Sure, and again kind of more work because it had to deal with multiple little files to get in and out.

Steve: Yup, I mean depending on how database was created it will oftentimes if you are an accidental DBA you don’t know about VLFs you might never have checked this but I’ve seen some that have had over 100,000 VLFs in a single log file. But why is that bad? Part of understanding when enough is enough is to know well what makes that a bad thing? I mean it’s just a lot of chunks in a big file. But what makes that bad is a couple of things. One is that when a transaction runs it needs to use multiple of those VLF chunks in order to be able to write to the log, and with big transactions you got to have multiple of them in use which may make it harder for the SQL engine to be able to write everything it needs to happen there. But the flip side of that and this is the real killer is that when you’ve got
that many and you try and restore your database SQL Server has to go through and recreate all of those VLF chunks when you do the restore. So part of restoring the database is allocates the data file, allocates the log file, and while it’s allocated in the log file it’s writing all those chunks. And I’ve seen restores of databases that took 8-10 hours because the VLF count, and then in the same database after reducing the VLF to a more reasonable number took an hour to do the restore.

Carlos: Wow! You know, obviously disk speed and all of that comes into play as well, right?

Steve: Oh yeah, and that’s just the same database in the same disk doing the comparison there. I’ve seen it be 8-10 times long because of a lot of VLFs.

Carlos: That’s an interesting point because normally we talk about performance, like we’re talking about application performance. Like my user is trying to get information out of the system, right, that scenario. But in this case one of the bigger performance killers is like an RTO perspective. I now need to restore that thing, “Well, I’m not going to be able to get the performance to meet the expectations of my users”, and that could be a problem.

Steve: Yup, and that’s one of those. I always try and think about that as if the system is down and you have upper management looking over your shoulder or calling you on the phone continuously saying, “Is it done yet?” And you’re just there twiddling your thumbs waiting for the log file allocate and not being able to tell them. I mean you’re thinking, “Oh, it’s got to be done soon.” But it could 6 or 7 or 8 hours before it’s done. I think that’s one of those things that misunderstanding of the VLFs there could lead to what people end up referring to as an RGE or a Resume Generating Event where you oftentimes, if you tell management you got a 2-hour recovery time but it turns out to be 10, that may be the end of your job depending on where you work.

Carlos: Sure. Now I hope that they will give them a little bit of leeway as long as they can get backupped. Now if they can’t get backupped that’s a different story. But yeah, that would be a rough place to be if they were that tight with the leash there.

Steve: And what it really depends on oftentimes in most businesses is how much money you are losing for every minute that that database is down.

Carlos: Exactly, that’s right.

Steve: I remember years ago when I worked at Amazon.com one of the things there that they measure for any outage was how much money did we lose. And if that money is a few hundred dollars it could be very different than if it was a few hundred dollars.
Carlos: Right. Yeah, I know that’s true. I think it helps put things in perspective. And again that kind of goes back to the culture of recognizing the value of things being up and then hopefully if that’s the window and you’re pricing things that way which I think again as administrators we could probably do a better job of saying, “Hey, what, you know what…” And I guess I’ll use the ChannelAdvisor guys those have access to selling things outside is a little bit easier. Customers purchasing products that’s easier to tally that downside or the cost there. But to be able to calculate that and say, “Hey, look you know what guys if we can’t do this then we’re going to lose X number of dollars. It’s going to cost Y to put it in.” The ROI does make sense at that point kind of a thing.

Steve: So then with virtual log files, how do you know if you have enough, not enough or too many?

Carlos: Yeah, that’s a good question. I was thinking about this and you put it on the list. I mean, so in my mind and luckily right, as knuckle dragging Neanderthal I haven’t had too many experiences having problem with this, Maybe at least that I have recognized. I think as long as I have a more consistent size that’s kind of where I feel better about things. What about you, Steve, when do you think enough is enough there?

Steve: Well, ok so really what it comes down to is how long does it take you to restore that database. I mean, that’s the key thing that I look at on VLFs. So if you have an environment where you’re doing a regular restore from production to a test server or a development server, that’s a great way to know. But if you don’t have that hopefully you’re in an environment where you test your backups. And if you have a backup that takes you let’s say an hour to run but 6 hours, assuming it’s similar hardware then that could be an indication that you might have a VLF problem there. However, that alone is not the only indicator. I have a script that I created and you can get to that on stevestedman.com/vlf, and it does the DBCC LOGINFO command and then puts that into a temp table and then does some sort of visualization with sort of a character based bar chart in the query output window.

Carlos: In the result, you could actually see the size of the individual files and kind of gives you an indicator as to how big they are.

Steve: Yup, yup, and with that, and again there’s a lot of opinions out there, it always comes to it depends. But my rule of thumb is that any time it’s over 1,000 VLF files in any one database or any one log file associated with that database that’s something that usually I want to deal with that right away. Anytime it’s over a couple of hundred or maybe 300 VLFs in a single log file that’s something that I like to deal with but it’s not super urgent. And just keeping it somewhere in that range. I think opinions will vary but I think most people who have experience with VLFs would agree that more than 1,000 VLFs can be an issue. And that many will also agree that more than 500 is something that wants attention as well.

Carlos: Right, and I think that’s a great point is that you have a metric to go back on and that is your restore time, right? So that can be your benchmark. That can be your feel good. You know now obviously there’s only so much performance you can squeeze out of that, right? It’s not going to go from 10 hours to 5 minutes I don’t think.

Steve: No. But it may go from 10 hours to 1 hour, and that could make the difference between staying up all night to deal with an issue or at least getting some sleep that night. Now the flip side of that is you don’t want to have too few of VLFs either. I mean, if you have too few like let’s say you only have 16 VLFs in your whole system. That would mean you only had 16 chunks of the log file, and when of them, depending on how things cross over between the VLFs that they would me marked as in use and they would stay as in use until all transactions touching them were complete or until they have been backupped. It’s kind of a balance there.

Carlos: And it also depends on the size, right? I mean if I have a new database, even like your DBA database that you have your own internal stuff in, 16 might be fine.

Steve: Oh yeah, absolutely.

Carlos: Then I have the production system with all my transactions on it that might be a different story. And again, I guess, I lean a little back on that because if I’m growing them in equal chunks, if I grow the log in equal chunks, then I guess I’m kind of trusting a little bit that they are going to grow in the best way, and the equal sizes they have equal number of VLFs so a little protection there and the database kind of indicating how many VLFs there will be for the size that I’ve specified there.

Steve: Right, but I think part of that comes back too to having your log size large enough that it doesn’t have to auto grow over time.

Carlos: Yeah, great point. That’s right.

Steve: The auto grow is kind there as in case of emergency this is what we want to grow but hopefully we’ve got that log file big enough that it’s not regularly growing. So do we want to talk about how we fix those?

Carlos: No, I think we’ve kind of mentioned them in different places or we can comeback on them, I mean, even in your performance tip. I think we’ll say that on different episode.

Steve: Ok.

Carlos: The only thing there on log files is so number of log files. We talk a little bit about sometimes dividing up or creating multiple data files. TempDB is kind of the classic example although it can be use in our other databases as well. But what do you think, number of log files?

Steve: It’s simple, one per database. The reasoning behind that is that log files are written to sequentially and whether you have one or you have 20 it’s still only ever be writing into one log file at a time.

Carlos: Yeah, exactly. So that idea of being able to use thread, parallelize, or how you want to think of that, being able to use each of those files without having to wait on something else won’t apply to the log.

Steve: Right, and I think the misconception there is that if I create two or three or four log files that it will be load balance between them as transactions are written but it doesn’t work that way. It just starts on one and uses until there’s no longer available space on it and we go to the second one if there is a second one. And it really just doesn’t make any sense especially if they are on the same drive. It really just doesn’t make any sense to have multiple files. I don’t believe I have ever come across a use case with SQL Server where I would have recommended multiple log files.

Carlos: Right, interesting.

Steve: So enough is enough, one is enough, two is too many on the number of log files for a database.

Carlos: That’s right. So another I want to go to and that is backup retention periods. We talked about, so ultimately we get into a scenario where our backups might be taking a little bit longer because we have two much, well I guess one history but I guess we’re using the word retention is how long we should keep them for?

Steve: Right, and then there is this sort of pack rat view of that which we keep them forever. And then there is the we’re tight on disk space view which means we keep them for as little as possible.

Carlos: Or just delete the ones that I don’t need so I can make room for the ones that I do need.

Steve: Right, right, so hopefully there is some balance in the middle there that can find. I think that from the DBA perspective, I mean I would like to have as many around as I possibly can because when something goes wrong that you’ll need a backup to recover from for instance corruption, oftentimes people don’t catch it immediately that day. It might be even a week or depending on your monitoring longer than that before you know that there is something wrong that you might need to go pull something out of the backup. Now, usually if you need to do a full recovery from a backup and you’re just going to abort the current database, usually you know those events pretty quick. But it’s the type of event where you realized something was missing, maybe some rows out of a table were deleted three weeks ago and you really need to restore that database and just pull those rows back in. It’s hard to know exactly what that retention should be.

Carlos: Right. I’ve worked for a couple of .coms and another scenario is not just the data but it’s either a data change or the table of table change, right? So a column gets added, a column gets removed, whatever and then all of a sudden because the reality is this change control process maybe got side step or whatever because it’s not good history. Nobody can remember exactly when the change got made and now all of a sudden it’s affecting things, so that ability to be able to go back and restore one from a couple of weeks ago and say, “Well, it was either this way or not this way as of this state.” I feel like been able to at least get things off of my plate by being able to provide that information.

Steve: I think one of the things that comes down to there on the backups and how many is enough is sort of a granularity based on how old the backups are. So for instance for me if I had unlimited space or a lot more space than I ever use for database backups, I probably wouldn’t keep every single backup. And most of the backup solutions that you have built in with SQL Server or other scripts available out there, they don’t really consider the granularity of backups over time, meaning you have a small retention window. After X number of days all backups are going to be deleted, or maybe after X number of days all your full backups are deleted, after different period your differential backups are deleted and after different period log backups are deleted. Hopefully you keep your full backups around the longest because you can’t use the other two types without those. But one of the things that I’ve come across when dealing with corruption is somebody is able to discover a corruption and it’s been in their system for months. And I’ll ask, “Well, do you have a backup from when before the corruption occurred?” And then the answer is oftentimes “No”. But sometimes somebody else will say, “I’ve got a backup from 18 months ago.” Will that help that all? And sometimes depending on how fast your data is growing or where the corruption occurs and 18 month old backup might be nice to at least see what was there or compare to what was there on older data on the database. So one of the things I like to do in a perfect scenario is maybe, like let’s say you’ve got enough space for one month of full backups but rather than keeping around a month of full backups taken let’s say every single day, which most people don’t have the space for that. But let’s say you did I’d rather keep around a week of full backups and then second week would have maybe backups from every other day and then the 2nd to 4th week or the 3rd and 4th week you might have backups once a week and then your retention period beyond that would be maybe a single full backup once a month that you keep around.

Carlos: Something like that, right.

Steve: Yeah, and I know that’s a lot harder to manage, and it might be as simple as you just have a job that once a month it takes one of your backups and copies it off to different network location.

Carlos: Right, we used to do that for auditing like the end of the year backup, you know, database for auditing purposes. So that auditors would come April, we have to keep this thing around for a time.

Steve: And my experience has been working with different areas of management is that oftentimes they don’t always understand exactly what it means when you talk about the backup retention, and when you explain to them that if we have a 2-week backup retention that means we have zero backups that are older than two weeks. Oftentimes that creates a bit of fear there that will lead to possible solutions.

Carlos: And of course one of the classic entrances into the cloud because nobody wants to go out and buy more disk space for just backups. The disk space has become very commoditized, very cheap in the cloud and so being able to store that information for a little bit longer. That’s kind of a lot of people’s first taste with getting a cloud technologies into their environment.

Steve: Yup, and I think that backups are great way to start with it.

Carlos: Well, let’s see we have a couple of others. I think I want to skip to TempDB. Do you want to touch CPU and memory? Or you want to call this an episode.

Steve: Yeah, I think it would be good. I mean you’re saying we want to skip TempDB. Yeah, because enough is enough on CPU and memory, I’d always take more.

Carlos: Yeah, that’s right. So ok, let’s hit on memory then just for a second. Yes, enough is enough, that’s a great question for that and there is this saying I hear somewhere that more memory will cover up a multitude of sins in the coding and all these issues that you could have in the database.

Steve: Yeah, a great way to fix I/O problems is with more memory.

Carlos: Yeah, exactly that’s right and the database just lives in the memory then no I/O problems.

Steve: So interesting on that I worked on a SQL Server on a few years ago where it was having some performance problems and it had, if I remember correctly, 64GB of RAM and they increased the memory from 64GB to 512GB of RAM.

Carlos: Oh, wow, so that’s substantial.

Steve: Yeah, very substantial.

Carlos: Exactly a license leap I think. I feel like 128GB is the.

Steve: We’re already on Enterprise Edition. Yeah, but that kind of a jump and at that point in time I think the cost for the memory was somewhere around $10,000 +/-. But that basically got rid of all the I/O issues that we were having there. Well, it still had a few I/O issues when you restarted the instance because nothing was cached in RAM. But once the instance is up and running for a bit it had so much stuff cached in RAM. I mean it was so much faster because it never had to go to disk to get anything. Of course it had to go to disk when there are rights but most of the performance issues around reads on this database, and it just took care of all the performance issues at that point. That was probably 3 or 4 years that continued to run without any significant performance issues simply by adding that much memory. Now, the side was effect was it also made it so that some of the developers not have to worry about performance tuning. So that when the database eventually grew to the point that that wasn’t enough memory for it, well they may have had more difficult performance issues to deal with at that point.

Carlos: Sure, and then it all comes down crashing down at that point. You’re probably talking about another system to go to the next level of memory at that point because computers are getting more robust. You know, once you start talking about terabytes of memory those are different systems.

Steve: Yup absolutely, so the question on how much enough is enough. Well, on that specific system where we set to 512GB of RAM. The thing I noticed that when it was normally running, for probably the first year that that was running it never exceeded about 400GB of memory used.

Carlos: Oh, that’s interesting because that was probably the size of the database.

Steve: Yeah, when on that one when everything was cached up and whatever the size of the database plus Temp tables and everything it was using, I mean it really didn’t exceed 400GB. But then a few years later it grew and it eventually got up and hit that limit but what that told me, when sort of it had that flat line or right around 400GB just sort of it, was that we perhaps bought too much memory and I kind of bite my tongue as I say that because it’s hard to say too much memory. But the fact that we never used more than 400GB indicated that if we had put 400GB of RAM in there that would have been enough.

Carlos: Sure, the application would have worked just as fine.

Steve: At that point in time.

Carlos: Yup, at that point. Yeah, so how do you know when enough is enough? I mean, obviously I think there is the indicator that people have lots of different thoughts about but the Page Life Expectancy (PLE) which I think has kind of been kicked around a little bit. I think for better or for worst somebody wrote a whitepaper at Microsoft to kind of came up with a recommendation that was quickly adapted as the standard. So in environments where we don’t have 512GB of memory, how do we know when enough is enough?

Steve: Yeah, and I think that comes down to sort of balancing the Page Life Expectancy with the page faults and knowing when something has to be bought in from disk versus when it’s able to be read from memory and looking at the overall performance. I mean if your system is performing great no reason to even talk about more memory at that point unless it’s growing and you’re planning for the future. But if you’re having performance issues there, whatever it is, whether you’re doubling from 4GB to 8GB, or doubling from 32GB to 64GB. Memory is oftentimes a cheap way compared to alternatives to improve that performance. So one of the things I like to look at because you brought the Page Life Expectancy was to watch how that Page Life Expectancy grows over time. One of the things that if you chart it over time and how it’s growing. If you’re running a regular CheckDB job, oftentimes that will skew your numbers because when CheckDB runs it of course has to bring the entire database in the memory bit by bit. And oftentimes when that happens you’ll end up pushing a lot of pages out of memory and it will skew the numbers on your chart there. But if you weren’t running CheckDB, how long does your Page Life Expectancy continue to grow? And if you chart that you could can sort of see pattern where it will grow and grow and grow and then at some event happens. And that event might be CheckDB or it might be nightly ETL process or it might be some job that runs that has to pull in a whole lot of data on a regular basis. But if it continues to grow and grow and grow throughout the day until you hit a few of these events that tells me that most of the time you’ve got enough memory and it’s only those certain events that you hit that would have prevented it from being read from disk if that data was cached in memory. And if those things are happening in the middle of the night or they are not impacting the actual business at that point. Yeah, no big deal, I wouldn’t worry about it. But if it’s the kind of thing where it’s an ETL that kicks off at midnight and it runs through until 10AM and it is impacting business in the morning well you may want to consider more memory to help with that. Let me just finish one thing on that first. So that being said though, you would want to understand that it’s definitely a memory constraint before throwing memory at it. Because I’ve seen people throw memory at long running ETL jobs and then they found out that it has no improvement because the bottleneck is not the memory, it’s something else in the system.

Carlos: I guess I’ll kind of go along with that a little bit in to take a peek at, obviously you don’t want to take a look at memory consumption, but the thing that I found out helpful is also just taking a look at the chattiness of the disk. You know, one of the things that I have found is that databases will kind of get grouped together and you have one database that’s kind of like the mission critical or very important. It’s rising in this rank, maybe not mission critical but it is becoming more and more important. But then you’ve talked on these other databases that are kind of one offs or not as “important” and then you found out that they are the ones that kind of being chatty, or they are taking up more space than they should. Those are the situations where for the system that you’re most interested in maybe you have enough and it’s just a matter of moving that lesser database of somewhere else so that the system that you care about is not impacted.

Steve: Yup, that’s a great example and that’s a great reason to have different instances running in different VMs, and that you could constraint some of those less important databases to not chew up the memory that way.

Carlos: Right. Ok, so I think that’s going to be our episode for today – Enough is Enough.

Steve: Enough is enough.

Carlos: Yes. Do you agree with our list? Give us your thoughts, let us know. You can leave you comments on social media or the website at sqldatapartners.com/enough or on sqldatapartners.com/105.

Steve: Oh, I just totally tripped there. Didn’t I?

Carlos: That’s fine.

Steve: Sorry, Julien. I’m making this hard for you to edit today.

Carlos: So let us know if you agree with our list. If you want to leave us a comment, you can do so on social media or you can do so at sqldatapartners.com/enough.

Steve: Sorry, I am just blowing it today because I was thinking about our LinkedIn and Twitter at the end. Let’s do one more take on that because I just spaced. Oh, sorry.

Carlos: No, that’s fine. I guess is there anything else we need to include before we end or do we want to wrap up there as well?

Steve: No, I think we could just wrap up there because we’ve kind of already included everything.

Carlos: Ok, sounds good. So that’s our list, let us know if you agree. If you have other comments you can reach out to us on social media or you can leave as a comment at sqldatapartners.com/enough.

Steve: Or at sqldatapartners.com/105.

Carlos: You can always connect with us on LinkedIn. We love hearing your comments and connecting with more of you. You can reach me I’m @carloslchacon.

Steve: And you can get me on LinkedIn @stevestedman, and we’ll see you on the SQL trail.

Episode 104: Keeping up with Technology

Do you have any experience with [Insert random technology]?  Your heart starts to race and your palms get a little sweaty.  You don’t want to say no–we’re tech folks–we know stuff, but there are so many new things to learn!  How are you supposed to keep up with it all? In this episode, we chat with Eugene Meidinger about his thoughts on keeping up and his ideas on the most important learning components.

Episode Quotes

“Keeping up with technology itself, like it’s impossible.”

“One of the important things is having awareness on what the problem is and what the challenges are.”

“One of the things that we’re afraid of is our skills decaying.”

Listen to Learn

01:08 How do you keep up with technology?
01:43 Eugene’s points on keeping up with technology
05:20 People who keep up with technology
06:13 How to stay relevant when it seems impossible to keep up with technology?
07:28 Generalization and specialization
13:03 Developing mastery and expertise
15:40 Steve’s experience in teaching a DBA class at a university
17:04 Generalization examples, job interview process
18:14 Rich mental model
20:25 Analogy of keeping up with technology as radioactive decay
23:00 Three things to have a longer “half life” with IT knowledge
26:30 Big Data or Pokémon site
29:20 Things that last: People Skills
30:31 The idea of having a periodic table of skills
31:30 Understanding theory, fundamentals and internals
35:03 Discussion summary
37:03 SQL Family questions

Compañero Conference
How the SQL CAT team monitors databases on Linux
Big Data or Pokémon?
Eugene on Twitter
Eugene on LinkedIn
Eugene’s Blog

About Eugene Meidinger

Starting out as an accidental DBA and developer, Eugene Meidinger now focuses primarily on BI consulting. He has been working with SQL Server for 5 years now, and is certified in Querying and Administering SQL Server 2012. He is a Pluralsight author on Power BI and also co-leads the Pittsburgh Power BI user group.

 

Transcript: Keeping up with Technology

*Untranscribed introductory portion*

Carlos: Eugene, welcome to the program.

Eugene: Thank you! I’m very excited to be here.

Carlos: Yes, it’s good having you. You have been a guest perhaps unofficially before on a couple of our panels when we were up in Pittsburgh and then Baltimore. You contributed to the conversation, we’d met, started talking and we want to get you on the program so thanks for being here.

Eugene: Yeah, definitely.

Steve: I guess we should say, welcome back.

Eugene: Well, I’m happy to be playing a starring role. I’m still mad at myself the first time because you’re supposed to say your name whenever they gave you the mic and I forgot to do that, so I’m just like Guest Speaker 3 or something like that on the first one.

Steve: The unknown voice with no credit.

Carlos: Yes, so we apologize. But thank you for being here and chatting with us today. This is actually an interesting topic and we had this as a SQL Family question and I thought that you had an interesting take on this. So the question that we have is how do you keep up with technology? It’s all over the place and of course we’ve even introduced since then kind of SQL Server in the News segment and it’s just amazing all of the different things that continue to come out of Microsoft. Let alone all the other companies out there. So I’ll ask you the question, let’s get started here. How do you keep up with technology?

Eugene: I think you have to just not sleep ever and then you’ll be fine. But for everyone else, anyone who happens to have a family or a significant other, or kids, hobbies, or just regular human body you’re not going to do a very good job of keeping up with technology. I think in many ways it’s not a very well defined goal. I think it’s more of an emotional fear. I think we’re afraid of two things. I think we’re afraid of losing our job or becoming so irrelevant or obsolete that we can’t easily change jobs anymore, that’s the first thing. I think there is a large number of us who fear becoming that COBOL developer who’s never updated his resume, and maybe has a good job now but there is a recession or they want to work somewhere else and they’re out of luck. I think that’s a fear that’s driving a lot of us but then the other question or the other fear.

Carlos: And I think there’s one, maybe lesser fear but I feel it’s kind of out there is, you know whatever, a social situation, “Hey, what it is that you do?” “I do COBOL.” And tech setting and they know what that is and you’re going to get the, “You’re really old.”

Eugene: I can tell you something. I’m still technically at my 20s and I don’t put VB6 on my resume but I know how to write VB6 for similar reasons.

Steve: So are we putting VB6 then on the same category as COBOL now?

Eugene: I would say technologies that I want to avoid.

Eugene: No, Ruby is basically a VB6 with some prettier syntax. I mean you could make the argument. Yeah, no, it’s definitely look down upon for being behind so one main thing is you want to keep your job. But then also you want to keep your friends and family, right? Because I joked earlier that, ok well, you could spend all of your waking hours reading books and watching videos and doing all the stuff and you probably do a good job of keeping up with technology but for me personally 9:00-10:00 PM is sacrosanct. We do not mess with that time. That is our date hour. Me and my wife are hanging out no matter what.

Carlos: Very good.

Eugene: Yeah. It’s important and so there’s balance. I think really what people want to know is how do I keep my job? How do I do it in a way that doesn’t cause me all this grief and anxiety and frustration? Keeping up with technology itself, like it’s impossible. I mean, you follow all the different things that are coming out with Azure. They just talked about CosmosDB where they took DocumentDB and then they slapped on four other different no SQL database models, right? And you’ve got SQL Server 2017. I really hope we’re not switching to an annual model. But they put Python in there. They’ve got all these other changes going on. There’s just all these different stuffs and so you’ll look at all of the things and I just don’t
[00:05:00] think, the way people to find it’s possible to keep up. There really is just too much stuff. Maybe 30 years ago you can keep up with SQL but today you can’t and if you count everything, if you count all these different changes.

Carlos: Yeah, this question perplexed me for a while, and I actually asked it when I was on SQL Cruise which is another reason why we’ve been inspired to do the Companero Conference because I was impressed and I felt like there were a couple of people that did a pretty good job of keeping up. But I’m not sure, and not to say that they are not keeping up, but the more that you follow them, the more that you kind of see some niching going on and the more that you see content sharing, right, so they’re kind of sharing what other people are doing. Similar to what we’re doing here. We don’t know all the technologies but we’re bringing people who do and can talk about it. So that’s one interesting facet that I’ve seen there. Sorry Steve, you’re going to say something?

Steve: I was just going to say given all these, I mean, it’s nearly impossible to keep with all technology or even all things in SQL Server. But you need to keep up but you need to keep your job as you said and keep your friends and family. So what do you do? How do you go about staying relevant at that point?

Eugene: I think one of the important things is having awareness on what the problem is and what the challenges are. I think there are a couple of different sources of where this is actually a challenge, so one of the things that we’re afraid of is our skills decaying. We’re afraid of being that COBOL developer and our knowledge becoming less and less relevant over time. That’s one challenge. There is a challenge where we’re worried about all these new technologies. I think the cliché example is JavaScript Frameworks. It seems like there is a new one every 6 months and you don’t know which is the right horse to bet on, which is the right choice to make. I think two really big things, just talking about generalization and specialization. In my mind, specialization is how you pay the bills. You have to pick a specialization and a degree of specialization. You need to figure out, “Ok, what do I want to go deep on?” And it doesn’t have to be Itzik Bengan deep. It doesn’t have to be David Klee deep where you’ve picked one singular thing and you are the “world’s expert”. But you have to pick something to go deep on and so that’s going to require focus. Focus on terms of what things are you not learning, what is your specialization, just setting aside time and that’s going to pay for the food today, that’s going to pay the bills today. But then the other piece that hole like, do I learn Angular kind of piece or in the data world, do I learn R, do I learn Python, do I learn Docker? That’s going to make sure that you get paid 10 years from now. Generalization makes sure that you put food on the table a decade from now. And that’s less about focus and that’s more about time. When you listen to podcast you get this exposure and you’re generalizing. You’re dealing with these unknown unknowns. I think the very first step is deciding do you have a problem where you don’t have enough specialization? Have you not gone deep enough or is the problem that you need to generalize more? Do you need to be more aware of what’s out there? I think for a lot of people they are scared of all the new stuff but really they still need to make sure that they know where they want to go and where they want to focus on for their career. I think the first thing you need to do is decide what’s my actual problem? Do I need to go deeper or do I need to go wider? And what am I doing to deal with that.

Steve: And to complicate it even more, I mean in some cases it might be do I need to do both – go deeper and wider. And that could be more subjective.

Carlos: When I think about it, I feel like at least going through the U.S. education system, right? The three of us have gone to college and that’s kind of the route that we took. You get some exposure there so that’s kind of the generalization if you will. You start in Information Technology you get your first tech job. From there, I
[00:10:00] think the most important thing is to go deep. Pick a couple of areas and that could be in a couple of different ways so tech stack. But also even just like an application of stack. More and more we hear from the CIOs and some of the things they are looking for in addition to the tech is I want to know the business. So kind of understanding the pain points and how technology solves those things. And I think once you kind of get deep and again like you’ve mentioned, just one area then it will be easier because you understand the full gamut. It will be, “Ok, where do I want to go next?” How can I take what I know and then apply it to the next hurdle or the next specialization area?

Eugene: Yeah, I definitely agree with you there. I mean, I think for a lot of people like if you are out of college your mission is to get past working at helpdesk. Your job is not to be learning Docker right now. Your job is probably not to be learning PowerShell or Hadoop or whatever the cool new next thing is. You’re right, when you’re coming out of college your job is to get enough specialization that people want to pay you money to do something. But part of that going deep too like you said is that. You know, I do martial arts and there is definitely a big difference between no belt, and white belt, and green belt, and all these different things. And I’m a green belt right now so I’m halfway there at the school that I go to. Sometimes you have to learn how to develop mastery in something. If you’ve never become an expert in area, again I’m not talking like elite top 1% expert. To me expertsy starts whenever you first present to your local user group or you write a bunch blog post; anything where the stuff has to go in through your eyes and come back out your mouth that’s starting to develop expertise. It’s on the far end of it.

Carlos: I guess I’ll throw another option there because I’m a big fan of Lunch and Learns. I think unfortunately managers don’t buy into it. The culture is, “Oh yeah, Lunch and Learn, you go bring your own lunch and make some poor shmuck present on something.” I wish that they would just say, you know what, again it could be like small groups pay the whatever it is, bring in pizza whatever, right, so that you can come and learn this. But that would be another option to say, “Hey, co-workers or group, I’ve learned something.” In fact, Doug Parnell, who is going to be speaking at the Companero Conference. One of the criteria they have for where you can go to conferences or get other training is his ability to able to bring that back and then explain to the group what it is that he learned which is interesting. So that’s not deep specialization. It’s just I’ve listen to it, I have some comprehending, and now I’m going to get at least further enough along that I can now explain it to somebody else.

Eugene: Yeah. Anything that’s going to be testing your mental model or something is going to have you that. And like I’m saying, I think that when you learn how to develop a certain level of mastery that becomes repeatable. Like you said, when you come out of college you need to learn how to go deep and once you’ve done that successfully and you’ve actually gone truly deep somewhere then now when you switch over to Hadoop or something like that you can do that. For me, I get that with speaking where the first couple of presentations that I gave there was a lot of fear and anxiety, and a lot of work. And now I’m at the point where I understand kind of the backbone of a good presentation and so it’s a lot easier for me to say, “Oh, I need to give a presentation on Power Query in two weeks or something like that.” And start putting together that outline, putting together that structure because I know what goes into it. Just the same exact thing with understanding what goes in to actually developing mastery somewhere even if that’s a journey man level so to speak and not a true expert.

Steve: So interesting, with that really the key is on developing that first thing that you’ve mastered. It’s not mastering it. It’s figuring out the process of how to master it so that you can then translate that to the next thing you have to learn.

Eugene: Yeah, absolutely. I think a big part of that like we talked about is understanding the difference between, all these different learning things. Are they giving you exposure or are they giving you mastery? Are they helping you with those unknown unknowns, like “Oh, I didn’t know that Spark was a thing.” Or are they helping you develop more of a mental model of how that stuff works and I think
[00:15:00] the big dividing line for that in a lot of cases is is it active learning? Is it something where you have to write or type or speak or code or something so that you can actually test that model that’s in your head. Because you can read all the books in the world or listen to all the blogs, or listen to all the podcasts but you need to have the rubber hit the road at some point, and that’s truly how you develop a sense of mastery and expertise somewhere. Again why I say that I think mastery starts with that first user group presentation or that first blog post because that’s something that really test your knowledge. Make sure you actually understand it at all.

Steve: Interesting. I can think of an example on that occurred in my experience was about 10 years ago I was asked to help teach a class at a local university, and it was just a DBA class and it was not the 70-461 but it would have been the equivalent of what the 70-461 exam was then. Then like right I was about to start doing it the person who’s going to help out bailed out on it so I was all on my own to go teach this 10-week class at the university. And for me that was an incredible learning experience because it pushed me beyond what I knew at that point and it made me learn at not to the point that I could just talk about those things but to the point that I can actually teach those things. And I think that was one of those things that jumping into it I never expected that to happen but I had to go deep on a whole lot of topics over that 10-week period. By the time I came out of it I was at a whole different level on what I knew about those kinds of things. I think your example of being able to take it as input and then give it as output through a presentation is a great way to learn at least in my experience.

Carlos: Then the next benefit. I have to think again kind of because now that you’ve mastered. You know, you’ve done that specialization as you go into the generalization components if you will so i.e. talking with others at a conference, listening to the podcast, talking to a vendor, talking to a co-worker, a potential employer and things like that. You can then pick up on how their topic whether that’s a technology, an idea, a process, how that overlaps into what you already know or how it doesn’t, and then be able to speak to that to help that conversation continue to flow. I guess I’m thinking more of a job interview process because that’s kind of what we were started with as job security, “I’m afraid, can I get a job?” And I can’t say that I’ve gotten all the jobs that I’ve ever applied for. That’s not true. But I feel that ability to be able to speak to the things that they have brought up has definitely been at least something that they had to consider in looking at different applicants.

Eugene: Talking about that job interview, even just talking with people, I think that by having a rich mental model, a rich understanding of something it gives you the capacity for analogy, even if it’s an awkward analogy or strained analogy, at least gives you that option. A good example is all this big data stuff. At some point I want to start learning about Hadoop, and Spark, and all these other technologies, and right now I’m still at that exposure phase. I don’t know pretty much anything but when I start looking into them. You know, I was joking with Kevin Feasel, one of your big podcasts cameos that wait a minute, Scala is just like Haskell but different, or F# but different. Or that Spark is basically a service bus but different, or Hadoop is kind of like whatever the SQL data warehouse project is, that appliance kind of thing that they sell. I forgot the exact name. It’s like Parallel Data Warehouse or that sort of thing. So whenever you have some area that you gone that richness with when someone talks about something in a completely different area you at least have the option to go, “Well, I don’t know anything about that but from what you’ve said it sounds a lot like X.” Or even something simple. When you understand how a transaction log works with SQL Server you’re going to be able to make some really good guesses about how it
[00:20:00] probably works with MySQL, or PostGres, or Oracle, or something like that. There is a lot of those things that will translate. And even if it’s not a one-to-one translation at least now you have a jumping board whereas if you are a jack of all trades you don’t really have a good way to tell if that comparison, that analogy feels right or not.

Carlos: Yeah, interesting. Now, to jump back in here you kind of have an interesting analogy with keeping up with technology. You model it after radioactive decay.

Eugene: I do. Well, I think it’s a good way to think about it because again if we talk about the beginning and how keeping up with technology is this nebulous anxious sort of thing. It makes me think a lot about when we talk about the cloud. Which originally was just some guy going, “Oh, this internet thing is undefined I’m just going to draw a cloud.” And we decided that’s our branding, right? That’s our marketing plan. Keeping up with technology is whatever it makes me not feel so nervous at night when I go to bed that I’m going to lose my job. That is keeping up with technology. I wanted some mathematical way because I’m a giant nerd of thinking about this, of actually working through this. And to me radioactive decay makes a lot of sense because when you’re dealing with, let’s say you have a pound of Uranium. I’m no physicist but I learn some basics in school. You’ve got a pound of Uranium. That Uranium is going to have something called a half life which simply put is just how long to have half of it. You could apply that to bunch of things but radioactive materials are pretty consistent and that half life is stable. And so I think that IT knowledge also has a half life. Now, what you say it is can vary. Now, Allen White, he says that every 5 years you have to retool yourself. I remember one the first time I was on this podcast he said that and I said, “Well, I’ve been doing this for five years does that mean I have to start over?” But in college I would joke about the same thing. I’d say, “Half of what you know is useless in five years.” And that’s how it really feels. And maybe it’s 10 years or 20 but the idea remains, but let’s say it is five. Well, you can mathematically model that, right? You can say, “Ok, what percentage would I retain each year so that in five years I’ve only have half of that knowledge.” And it turns out that percentage is 87%. That means that if you know 100 things that are not COBOL. You know, 100 things that are still relevant today then if your half life, your IT half life is five years, that means that 13 of them either fell out of your head or no longer applicable, right? 13 of them are either VB6 or something you haven’t done so long you forget how to do words, or DTS or whatever.

Carlos: You kind of know it but you wouldn’t want to be asked to do it again.

Eugene: Right, and so that kind of gives you a way forward because if you think of it that way then we’ve got three knobs that we can twist to try and improve how much stuff we know so that we’ve got a longer half life ourselves, a longer shelf life, whatever you want to think of it as. The first option is that you just learn more stuff. You just shove as much stuff in as you can.

Carlos: So instead of 100, it’s 150.

Eugene: Right, exactly. If you need to learn 13 things a year just to tread water then if you can learn 20 or 40 or 50 or whatever then the total amount of relevant knowledge you have is going to increase. Do you want to go deeper into that right now or do you want to go through all of the three.

Carlos: Let’s go to the other three things. I think that would be good.

Eugene: Ok. The second knob that you have is you can learn more of the right things so that’s about having a better focus. That’s about having a plan. That’s about improving the signal to noise ratio because you can spend 160 hours in your entire week reading Twitter and Hacker News but you’re going to learn about local elections or Go Lang or Rust or some local startup or what Zuckerberg is up to this week. Even the technology things may not be relevant to where you want to go or what fits your knowledge or just there’s a lot of junk out there. There is a lot of low quality materials so if the first thing is learn more things. The second thing is to learn more of the right things. Learn more of the things that fit what you want.

Carlos: So staying away from bleeding edge stuff and away until you start to see some more adoption. Maybe early adopter is the phase. You’re like, “Ok, that’s what I
[00:25:00] will jump on to because I’m seeing it more widely used.”

Eugene: Yeah, I think one of the strategies with dealing with the bleeding edge stuff is make a low investment with that. So that’s why stuff like this podcast is so great because you can spend an hour while you’re doing something else and get enough to be conversational at bleeding edge technology and then later on you can figure out, “Ok, this fit with my career. Now, I want to go deep.” So that’s the second thing is just learn the right things. The third know that we have is that radioactive decay, that how quickly does my knowledge become obsolete and that relates to what you just said as well is learn things that last. Learn things that last longer. So things that don’t last are stuff tied to a specific version. So the exact feature set that happens to be in SQL 2005 is perhaps not too useful to you. But understanding how to use some of those features that came in there or understanding some of those advance window functions that came with 2012. That is going to last longer. Certain types of technologies are just immature. Again, I joke about stuff like Angular where they’ve been breaking releases every 6 months but you have that big data space. It’s the hot new thing but I’ll tell you what there is a great site called like Big Data or Pokémon and it will give you a big… It’s true!

Carlos: Nah, I have to look it up.

Eugene: Go and look it up. So it will give you a name like Empowla, or Horsea. I forget some of the other ones. And they’ll say, “Is this a big data program or is this a Pokémon?” And then you’ll click on a button and it will tell you if you’re right or wrong. And you’re going to be wrong a lot of the time. It’s true. It’s great. It’s the best site ever.

Carlos: Ok, here we go. So I’m here, https://pixelastic.github.io/pokemonorbigdata/. We’ll put it up in the show notes. So the first name is Horsea. I happen to be a Pokémon player for the kids, for my children. I have 5 kids.

Eugene: Sure. Yeah, family bonding. I get it.

Carlos: That’s right. So Horsea, big data or Pokémon?

Eugene: Are you asking me?

Carlos: Yeah. I’m asking to the group.

Eugene: I’m pretty sure that one is a Pokémon.

Carlos: Yeah, I’m going Pokémon too. Steve?

Steve: Yeah, I’ll with the group on that one. I’ve never heard of that big data.

Carlos: Yeah, here we go. It is a Pokémon. Ok here we go, Vulpix.

Eugene: Ok, that’s definitely a Pokémon.

Carlos: Definitely a Pokémon.

Eugene: I had a try with it. I promise you.

Carlos: Here is a softball one, Hadoop.

Eugene: That is a big data.

Carlos: That’s definitely a big data. Here we go, it’s a native one that I’m not sure of anyway, Spoink.

Steve: I’m going to guess it’s a big data.

Eugene: Yeah, that’s sounds like something is going to make it for a big data company.
Steve: Sounds like a tech thing.

Carlos: Oh, it is a Pokémon. Look at that. Ok, that is funny. So I don’t know if I should thank you or send you a nasty email now that you’ve introduce me to the site because I’m going to have to go through and.

Eugene: It depends on how much time you waste.

Carlos: Exactly.

Eugene: So the point that I was making with that is that when you have so many of these big data technologies, even within Hadoop you’ve got all these goofy names. You got Pig, and Sqoop, and Flume, and Hive and HTFS and all that stuff. Because it’s immature you don’t want to make a huge time investment. These are things that are going to decay quickly because it’s going to be like some sort of ultimate battle and by the end of it one is going to standing with the crown. And you don’t know which one it is right now.

Carlos: Now, there’s a lot more players in it but it almost reminds me of, what was it? Blu-ray? And what was that technology?

Eugene: It was something like DVD, HD DVD or something.

Carlos: Yeah, DVD or something.

Eugene: Yeah, exactly or even going back VHS and betamax and all that kind of stuff. And so bleeding edge technologies are something that don’t last. But let’s talk about what things do last and we had to some of these things. But things you’re going to learn that last. One of the biggest one is people skills. People do not change or if they do it’s much much slower in terms of centuries than it is the years with technology.

Carlos: So decades, generation.

Eugene: Grammar doesn’t change that quickly. I can promise you. So if you’re going to learn how to write a good email or I have a blog post about how to write a good abstract, you know, that’s going to last the test of time along at the same time,
[00:30:00] speaking – public speaking skills. You guys do consulting and I’ve learned myself that if you can stand up in front of 50 people and pretend like you know what you’re talking about you can do it too. Learning the trick of, “Well, I don’t know but I think it will work this way I’ll get back to you. I’ll give you an answer.” Those kind of soft skills are timeless, truthfully.

Carlos: The thing you’re intuiting there is we’re just making the stuff up, aren’t you?

Eugene: No. I think I implied it. I don’t know if I intuited but the distinction is lost on me.

Steve: So it would really be really interesting as we go through these different items if there was like a periodic table of skills that you could look and say, “Well, the half life on public speaking is 200 years. But the half life on big data is 9 months.” And try and do a comparison that way to figure out, “Ok, if you need to increase your skills overall.” What are the ones that you can either increase or going to last for a long time versus what can you learn quickly that might be a risk but it may pay off in the short term but you know what it’s going to be different 5 years from now.

Eugene: Yeah. I would say the people skills are definitely the noble gases of the skill world because they are not reactive. They last forever. But another thing that last long is I think, you know we talk about it going deep, understanding theory, fundamentals and internals. Going that one layer below and understanding how something actually works because it’s so much easier to tranche for that. But it also lets you make certain guesses and inferences. I’ll give you a perfect example. I have literally thanked Paul Randall twice for his transaction log course because it saved me so much for understanding that like for example dealing with availability groups. If you don’t know how the transaction log works on an internal level, availability groups are such a huge pain because you’re like, “Why I can’t sync this?” Or you say, “Do I have to take backups on both sides?” But if you understand how it actually works then you can intuit a lot of things. You can intuit, “Ok, if I’m taking a backup right now is the transaction log going to keep growing while I’m still doing the backup or will it stop?” That kind of stuff. So we talked about three different things: learn more things, learn the right things and then learn things that last. Things that last is going to come down to the deep stuff fundamentals, internals, some of the hands off stuff. And then it’s going to be those people skills. It’s how to write, how to read, it’s how to communicate, it’s how to learn in general, that kind of stuff. So those are I think the three different approaches you can take because the first two increase just your inputs, and then the last one decreases that radioactive decay. So if you know 100 things, if your half life, if you can shift that from 5 years to 6 years. If you can make that tiny little shift then still learning just 13 things a year, you’re going to end up knowing a 120 instead of 100. So slowing that decay you’re going to know more relevant stuff as a result over time.

Steve: Interesting. As you say that I think I’m really glad I’m not a JavaScript developer because I think the half life there would be…

Eugene: 6 months.

Steve: If even that maybe.

Eugene: Like I said, I know that Angular is coming out with like build number changes like full number changes. I think the plan is supposed to be every 6 months or something like that. And I’m still mad about SQL is coming out every 2 years so I don’t know how I will deal with that.

Carlos: Yeah, that’s right. Different worlds, right? You know the dev ops level on that side.

Eugene: It’s sneaking over the SQL world for sure all the dev ops.

Carlos: It’s well on its way. Well Eugene, great information and I guess we should note that if people wanted to extend the conversation a little bit or actually here you present this, your presentation at the GroupBy Conference would be available
[00:35:00] and I’m sure it will be posted by the time this goes out actually.

Eugene: Yeah, we expect so.

Carlos: We’ll make sure that’s included in our show notes as well.

Steve: So I guess then just to kind of wrap it up at this point.

Eugene: Yeah.

Steve: Well, before going to SQL Family, just summarize a little bit of where we’re at.

Eugene: Oh sure. Yeah, ok, I can do that. Just to summarize everything you have to figure out, “Ok, what it is my real problem?” Is it that I need to go deeper with things or do I need to be learning more things? And then if I’m going deeper I need more focus. I need a plan. I need scheduled time because doing active learning is hard. It requires focus. That’s the fuel for deep learning. The fuel for generalization and broad learning is time. But you can listen to podcast while you’re exercising or doing the dishes or commuting. You can learn some of these things without giving it your full attention. And you don’t often want to give it your full attention because it’s so volatile. But really a lot of it comes down to three big things. If it’s like this radioactive decay where our knowledge is continually fading in relevancy, you can either learn more things which means putting in more time, more energy, or more money in some sort of way. You can learn the right things by say leaning on curation making sure you’re dealing stuff that’s good quality or having a plan and making sure that stuff fits in within your plan. Or you can learn things that are going to last longer; that are going to last more than five years and not become irrelevant, that aren’t just a hot new thing. Generally, that comes down to going truly deep and learning internals or fundamentals or theory. Or means learning people skills or business skills, things that haven’t changed nearly so rapidly over the past 10, 20, 30 years, things that sometimes don’t change for generations. So that would be my general advice with trying to keep up with technology. You may not be able to truly keep up with technology but you can find a way to keep your job and keep your friends without so much angst and so much anxiety.

Steve: Alright, very cool.

Carlos: Good stuff.

Eugene: Yeah.

Carlos: Shall we go ahead and do SQL Family?

Eugene: Sounds good to me.

Steve: Let’s do it. So how did you first get started with SQL Server.

Eugene: So it was largely by accident if I’m being honest. I took a database course in college and that was using MySQL as backend. I was a TA for that class later and so the different professor was using Access. And then later I did a non-credit intern project and did all the development work and that was using MySQL. Up until my first long term job, my current job, no experience with SQL, didn’t know it was a thing. And then I’m looking for a job after my first one and the job says .NET/SQL developer. And I’m like great, I always want to do software engineering, do a lot of programming, this would be perfect. Well, I thought it’s going to be 80% .NET and 20% SQL and it was flipped. Half of that was DBA stuff and I remember my first month looking, googling up the differences between a view, a stored procedure, and a function because I didn’t know any of that at the time. I could do my SELECT *, I could do my WHERE and that was about it. But I just learn on the job and I got involved and then I find out that, “Oh, user groups are a thing.” And I start going to local SQL user group in Pittsburgh and then I found out SQL Saturdays are a thing. I’ll tell everyone here. Don’t go to the after party because you’ll end up as a speaker. I got cornered by Gina Walters who was running the group and Rick Heiges who was a former PASS board member, and they’re like, “You should present.” And I said, “I’m not qualified.” And they said, “You should present anyway.” And so I gave my first presentation on execution plans. I was terrified but I loved it and I just kept going from there.

Steve: Alright, good stuff.

Carlos: Now, in all that time working with SQL Server, if there is one thing you could change about it what would it be?

Eugene: I know this had been said before but licensing. I would change licensing. If there was just one simple guy like I get, ok we got like Free, Express and we’ve got Standard, and Enterprise. Microsoft wants their money they see Oracle doing their thing I get it. But then you’re throwing stuff like, ok if you have a cold standby, that one is free. Well, in 2014 we change that now you have to have software assurance for it to be free but the moment you start taking backups, you’re doing production works so doesn’t count anymore and all these little nuisances are just really overwhelming. So licensing by far I would change.

Carlos: And then if you have that license you could take it to the cloud, but then you
[00:40:00] have to

Eugene: Yeah, now you got hybrid.

Carlos: Failing over, and if you’re in the cloud for too long and that’s different licensing.

Steve: Yeah. That’s definitely one that would be worth straining out a little bit. So what’s the best piece of career advice that you have ever received?

Eugene: I’ll give you two because the best piece of career advice I know of I got out of a book so I don’t know if I’d count that receiving it but there’s really great book that I was given by a friend in my first job, and it’s How to Have Confidence and Power in Dealing with People which sounds really fancy but it’s a lot of common sense stuff of just how to work with people and talk with people and that kind of stuff. For someone who is this introverted nerd who didn’t know how to work with other people it was big. And the biggest thing out of that book, the best career advice that I’ve ever found in my career is “paraphrase what people say”. Repeat it back to them to make sure you’re on the same page. Just ask, “Hey, do you mind if I paraphrase that to make sure we’re on the same page.” And then just repeat back what you heard because there are so many times that you heard something different than they said and even of you got it right it lets them know, “Ok, he understood”, and they can relax a little bit so that’s been huge for me. As for received, probably definitely something that’s recent and sticks in my mind is from Erin Stellato where I talked to her about, “Hey, I want to get a job and big data or data analytics or something like that.” And she said, “Make the job that you want to have.” In the sense that instead of thinking, oh I’m going to have to find some other job. Well, I can look for opportunities to say, “Hey boss, I did a little bit of R with some of our internal metrics and here is what I’m able to find.” Or just something that shape the job that I’m already in to something more of what I wanted to be three years from now or something like that. That’s something huge.

Steve: Ok, great.

Carlos: And not to bang this drum companeros here, forgive me. But I think that idea is if you can tie the technology to a business scenario I would be willing to wage your 99% of the time you’re going to get to do that project. You know, assuming budgets and all of that stuff are all in order. But if you can prove value to the business by it, much easier scenario, much easier conversation than, “Hey, I want to do big data.” I have this problem I think I can solve it. Now having said all that our last question for you today, Eugene, if you could have one superhero power what would it be and why you want it?

Eugene: Yeah, I’m tear with this question because I’d want to be able to learn mildly useful things really quickly. Because I feel like most superpowers would be just way too obvious, way too intrusive like, Carlos, if you’re flying around the work or whatever people are going to notice and then you’ve got paparazzi and all this kind of stuff, right?

Carlos: Got you. There you go.

Eugene: Or if you’re some super genius that you can just touch a computer and tell what’s wrong then people are going, the FBI is just going to kidnap you and dissect you and figure out what’s going on. But there are all these minor little skills that I mentioned and that are useful but no one would go, “Hmm, I wonder what happened to him?” Like I want to learn lip reading someday or lock picking or something that my wife and I are learning right now is sign language. Like she is fully capable hearing, no problems there at all. Well, ok maybe sometimes she can’t hear me as well. But we’re learning sign language because one it’s just this cool thing. But two it legitimately is something useful in these occasional situations. So if you are in a loud concert or you’re 30 feet away from each other you can still communicate. And right now our repertoire is pretty limited. We mostly can say, “Hey, I’m going to the rest room.” “Oh, look at that cute child.” But we still get some value out of it right now. So my superpower would be learning all these like mildly useful little skills really easily but nothing that would attract notice by any authorities or other people.

Carlos: Lots of attention.

Eugene: Yeah, right.

Carlos: So I’ll second you there on the sign language. My wife and I took a class while we were in college together. It hasn’t been super useful outside of teaching our kids when they were growing up some sign language like terrible tooth time they can’t quite talk. They want to communicate that’s been the best thing there for it but yeah, super cool. Eugene, thank you so much for being on the program today.

Eugene: You’re very welcome. It was a pleasure.

Steve: Thanks, Eugene, really enjoyed it.

Episode 103: Plan Reuse

When we write our queries to the database, SQL Server has to go and figure out the best way to go and bring back the data you asked for. A query plan is created to help SQL Server remember how to get the data for this query. It takes time to make these queries, so the database wants to limit the number of times it has to create the plans so it will try to reuse the plan as much as possible.

Our topic for today’s episode is query plan reuse and the pros and cons with this concept. We will also touch on the concept of parameter sniffing–a technique SQL Server uses to try and figure out the best values to use for the execution plan with the hopes the plan will help the most queries. Special thanks to James Youkhanis for the suggestion.

 Episode Quote

“The concept behind this is it’s there to make things a little bit faster by reusing cache plans.”

“Parameter sniffing is a good thing because without it SQL Server wouldn’t be able to optimize your plan for any kind of parameter. But occasionally it goes wrong.”

“I think it kind of comes down again to kind of knowing your system and understanding the problem”

“Optimized for adhoc workloads is one of those parameters that we most of the time will recommend people turn on”

Listen to Learn

4:53  SQL Server in the News
5:00  Ola Hallengren scripts now on GitHub
6:45 What is plan cache?
7:48 Description of T-SQL and its execution plan
10:15  Scenario in regards to statistics and indexes, and data types
11:30  One-time use query plan cache
12:22  SQL Server and the importance of memory
12:50  A specific problem with one-time use query
12:55 Parameterization
17:30  Parameter sniffing
20:25  Stored procedure and plan cache, parameter sniffing issues
23:55  Options to solve parameter sniffing issues, recompiling
27:28  Controlling plan cache size
28:10  Plan cache and flash array
29:27  Idea of ad-hoc workloads
32:30  Needs parameter reports and examples
38:15  One-time use query reports
38:50  Instance level memory report
39:40  More about hints, recompiling and plan guides

 

Transcription: Plan Reuse

*Untranscribed introductory portion*

Carlos: So companeros, welcome to Episode 103. Thanks for tuning again to another great episode.

Steve: Yeah, Episode 103. Wow, so this episode is on cache plans and plan reuse which is one of my favorite topics in SQL Server. I know I’ve had a lot of lively debates over the years around this.

Carlos: Yeah, that’s right. I think it’s one of those things where from a performance perspective so indexing and some of the other objects and then you got to figure out to the internals of how SQL Server works. Yeah, it can a bit confusing a little.

Steve: And this topic came to us from James and he suggested that we talk about plan cache and plan reuse. I want to thank him for that suggestion.

Carlos: Yeah, we apologize. He was suggesting, gosh it’s been, I’m embarrassed to say how long have it’s been but it was during a time when we had a slew of interviews kind of lined up. It was kind of push to the back there but we’re glad to finally circle back around to it. We have a couple of shout outs this episode.

Steve: Yes, so one shout out came from sqlgambo on Twitter and this was in regard to a post that I had tweeted about Database Health Monitor, and he came back and said why I am building these tools of mine. I guess he hadn’t seen Database Health Monitor and then he and I actually chatted through private messages on Twitter for quite a bit and learn some stuff about what he is working on. And learn that he really likes with Database Health Monitor, so yeah, good to connect there.

Carlos: Yeah, very cool and it’s kind of interesting. We were talking before we started recording here about the history of database health monitor and how it kind of started from SQL Server reporting services reports. I was there, I was building my own home grown and came across Database Health Monitor. Lots of other tools out there, that’s one of the very nice things about the community and making those things available.

Steve: Yup, definitely.

Carlos: So Companero Conference coming up again on October – October 4th and 15th. Interestingly enough I was just at Nashville for a health conference this week trying to make some connections with some hospitals, and one of the things that stood up to me was the importance of unstructured time. So in that conference they were trying to because it had a mix of panels and speakers, and then vendor type sessions which weren’t horrible but they are still vendor sessions. So they were just trying to get through them. They kind of plow through it and they had this conference in a day and a half, a little bit more. I didn’t felt we had enough time just to talk, like let’s understand what it is. Are you being affected by this problem to be able to keep the conversation going? Anyways, so I thought give me some perspective on this idea of creating some structured content from over a session or from a panel. But they aren’t giving the unstructured time for people just to be able to talk, connect, right? Where are there similarities and commonalities? What I might want to pick up with this person after the conference. What conversations do I want to maybe talk about tomorrow or things like that? And so I found that again kind of single track conference, other way we’re going to do ours that I would have enjoyed a little bit more unstructured time with some folks.

Steve: Interesting. Yeah, I think that that unstructured time can be more valuable relationship building time that you have at a conference. And I think going to a conference and just getting the material, I mean, really you could do a lot of that on YouTube. But going there and making those contacts with people and being able to have time and talk about what it is you’re doing or when we do a session on performance tuning to have time to talk afterwards about issues you’ve ran into or problems you’re trying to work on can be incredibly valuable.

Carlos: I agree. I am looking forward to our conference and putting that on. We hope Companeros will join us October 4th and 5th in Norfolk, Virginia. You could take a look at companeroconfernce.com and we’ll make sure we have the link on the show notes page again as well.

Steve: And now on to SQL Server in the News.

Carlos: This has been up for a couple of weeks now but thought it was interesting. Many of you may be using Ola Hallengren scripts. He has decided to put them out on GitHub, kind of made them available. Obviously they are free to download but the now the difference being that you can actually suggest changes to his code and I know that a couple of people have done that already so it would be interesting to see what happens with those scripts as a result. Kind of going back to our community episode we talk a little bit about this, so it would be interesting to see what happens.

Steve: You know, we didn’t really talk much about Ola scripts on the community episode because it wasn’t really a community contribution project. It was sort of something he has built over time but now that it’s out on GitHub maybe it will become something more amazing than it is already based on community contribution.

Carlos: I believe it came out on GitHub after our episode so I wonder if we are not influencing people out there, Steve.

Steve: Yeah, who knows.

Carlos: Can we take credit for that, you know, SQL community you can thank us for Ola putting his stuff on GitHub.

Steve: Whether we were the cause or not.

Carlos: Yeah, sorry, butterfly effect or something. Ok, so today’s episode can be found at sqldatapartners.com/plancache.

Steve: Or at sqldatapartners.com/103 for our episode number.

Carlos: So again, ultimately what we are talking about is execution plans, plan cache, and plan reuse. So first, I guess let me back up and kind of from the 10,000 level view, what is it that we talk about when we are talking about plan cache.

Steve: Well from the high level it’s basically the plan cache is a piece of memory or junk of memory in SQL Server that keeps track of every query plan as it gets compiled. And if your queries are written in a way that it can reuse the plan it can then grab and reuse one of those existing plans rather than having to recompile that every time. And the concept behind this is it’s there to make things a little bit faster by reusing cache plans.

Carlos: Right, so allow me to go back up just a slight bit higher and take the idea that we use T-SQL to tell the database what it is that we want out of the database. So T-SQL has been described as the language of what is it that we want but it doesn’t tell the database how to go get it. So the database has to decide how best to do that and when the query comes in like as you mentioned, it’s going through a process to decide, “Ok, well this is what you want. How do I need to go and get that data?” And so as a result it’s going through a look and say, “Ok, well I think this is the best way and I want to create a plan.” I want to create an execution plan or a way to go and get this so that way every time you want it I will know how to go and get it. There could be many different ways almost like kind of going from Point A to Point B, lots of different ways to get there. And it has to figure out which way is the best or at least which is its going to use on a regular basis.

Steve: Right, and it can be a very expensive step in the process that compiling and figuring out how it’s going to go about getting the data.

Carlos: Well, it’s interesting. So they mentioned expensive and I guess kind of this but there is a cap, a couple of milliseconds, which all of a sudden I can’t remember how many it is, that it will what that is. And I thought, well gosh milliseconds isn’t sound all that long of a time but I think it’s all a matter of how busy your server is and then how many executions are coming to SQL Server as well.

Steve: Yup. And I guess to put it in perspective with milliseconds there. I mean last week I was working on a query with a client where we were trying to increase or decrease the run time from about 800 millisecond down to about 150 milliseconds. And milliseconds could make a big difference there, and this was a query that was being run continuously like through website traffic, web service
traffic and all kinds of things. It was being hit quite often so a difference between 150 milliseconds and 800 milliseconds meant a lot to the performance of the system.

Carlos: Sure, that’s one of the great point, right, is the frequency not just of all of the queries but of that specific query, because if it had to recompile every single time and you’re adding milliseconds on there then you’re just kind of piling everything back up and it’s going to go and redo a lot of that work every single time.

Steve: Yup, and the work that it’s doing there is it’s going out and it’s looking at statistics and what indexes are available, and what data are you looking for, and what are you filtering on. And it puts all those things together to figure out what is the best way that a SQL Server engine can you go and get your result set for you. And if it wasn’t so smart it will just say, I want to go out and I’m going to do a full table scan on every table you’re looking at and give you your results brought back together on your JOINS. But that just wouldn’t cut it in today’s world. Maybe databases 20, 30 years ago might have done that but today there is so much going on with indexes and big tables and different ways to get the data. There is a lot of options there to look at. I mean if you got a table with 30 indexes on it versus a table with 2 indexes there might be more work that has to happen there when it’s figuring out what is the best plan to use.

Carlos: Sure, and then we kind of get into data types, right? That plays a role as well. There a lot of things that it has to look at and consider.

Steve: Yup, so what happens after that plan gets compiled is it gets put into this memory location called the plan cache and those plans are kept around with the hope, SQL Server hoping, that they will be reused so it doesn’t have to do that work again but sometimes they never get reused. What you end up with is you end up with what could be called the one-time use query plan cache where if things are changing in the query and they are not identical you end up with all this one time use queries in the plan cache that can kind of clog things up and sometimes push other things out of the plan cache that would be useful to be reuse.

Carlos: Like you mentioned, going back to that idea, so the plan cache is a space of memory where your data has to store as well, SQL Server read everything from memory, right? So has to be able to read that stuff there and so if you’re not using it that mean that there are, memory is kind of a vital thing and a finite thing as well, that you are using those resources in a way that’s not helping your SQL Server go faster.

Steve: Right, so think of it this way. Let’s say we had a table called ‘podcast’. And in there we have the list of all of the podcasts that we know about and in there is a column called ‘host’. And you just said, SELECT * from podcasts WHERE host = ‘Carlos’ If you run that query it’s going to create, I mean by the first time you run it it’s going to create a cache plan and then if I come along and run that same query a moment or two later or a few minutes later and I run the exact same query SELECT * from podcasts WHERE host = ‘Carlos’ It’s going to not have to recompile that. It’s just going to use that already compiled plan and it’s going to save some time there. But then if we change it up and Carlos runs the query to say SELECT * from podcasts WHERE host = ‘Carlos’ and I say SELECT * from podcast WHERE host = ‘Steve’ that’s going to be two different plans because the queries are different. By different if you just look at the text of that entire query and if it’s not exactly identical meaning turns in the same place, space and the layout is exactly the same and the difference is we’ve changed the WHERE filter to say Carlos or Steve, that’s shows up two different plans in the cache. Now imagine if this was like, SELECT * from customers WHERE customer_name = ‘Fred’. Or customer_name = ‘Mary’, or customer_name = any of the 10 million customers you have in your website. You could end up with many many of these one-time use queries that may not or maybe they get used once or twice or three times while that customer is there but they end up chewing a whole lot of memory, and the way you get around that is you use parameterization. And the way that works is instead of saying, SELECT * from podcasts WHERE host = ‘Carlos’ you say SELECT * from podcast WHERE host =, a parameter. And that parameter is there and when that query gets run, whether you are running it through code you’ve written, website code, or reporting services or wherever. Instead of passing through the text string of Carlos or Steve as the parameter it passes through a separate parameter that says compile this without knowing what the parameter is necessarily. Just compile it and then we’ll fix up that parameter after the plan has been compiled.

Carlos: Then one of the ways that it goes and figures out but maybe parameter it should use as a default, is it well look at those statistics to say, “Ok well, I see Carlos is in here 100 times, Steve is in here 10 times.” Like, “Huh, Ok, I see there is a tilt towards Carlos so I’m going to assume that more people are going to query Carlos than Steve.” So I’m going to make some decisions potentially based on that distribution. This can be a good thing or this can be a bad thing.

Steve: Yup. An example of that, think of it as if you were looking up for customers by zip code. And now imagine that you are in a small town running a business and maybe that small town has one zip code, maybe two zip codes, and most of your customers are local so most of the time when you’re looking up customers you’re looking them up based off of those one or two local zip codes. Well, then you get a customer that is somewhere on the other side of the country in a different zip code. It might be that 99% of all of your customer are in one or two zip codes but then you have this 1% that are in other zip codes. What can happen with that is that the plan can assume that most of the time that’s being run with something that requires more work because it has to scan more customers based off that zip code distribution but then when you run it for that one customer without one zip code that doesn’t match your local zip code. It could get to that customer with less work but it doesn’t because it goes through the normal path of that pre-compiled plan to find the information there. There is a term for that and it’s called parameter sniffing. Where when a plan gets compiled the first time it gets compiled it looks what are the parameters that are being passed in and it figures out a good plan that’s going to work for those parameters.

Carlos: Based on everything that I know as of the data that’s all in these tables what is the highest probability of what’s going to come in and let me make it easiest for kind of the 80-20 rule if you will. That’s where I’m going to go and try to get that.

Steve: Yup, so then let’s say you have that customer example by zip code and you are looking it up by zip code and the very first time or when the plan gets compiled you use one of these odd zip codes that’s not a very common zip code in your system. It may then look at it and say, “Ok, there is only a very small percentage of the rows in our result set that use that zip code so what we’re going to do is we’re going to do an index seek right to that zip code location.”

Carlos: An anomaly in that sense. Like, “Oh, all the data must become an anomaly.”

Steve: Yes, but then if it was the first time you compile that plan and it was one of the common zip codes it may look at that in an example of instead doing a seek which we go right much quicker to where you’re going for the smaller set of data. It may say, well a majority of the rows in this table are in this zip code so instead of doing an index seek we may do an index scan or even a table scan because that’s going to be the most optimal way to bring back everything that you are looking for based off of that initial parameters. So what you end up with is that when that plan gets compiled and that’s compiled the first time you run it or if it gets pushed out of memory or somebody flags it to be recompiled that the next time it is run it gets recompiled. But if that time that it gets recompiled if you’ve got good parameters that represent a regular data set in your system you get really good performance out of your query. But then if it happens to get recompiled with one of those unusual parameters that causes your plan to do something different then what is optimal most of the time you could end up with a really inefficient plan, that ends up bogging down the system or really hurting the overall performance.

Carlos: Sure, and what gets a little bit weird and you may think I guess why would this affect me, this should be a problem. If you’ve ever seen scenarios where one time query runs pretty fast and then all of a sudden it doesn’t and then maybe later in the afternoon it runs again fast. That’s a common symptom if you will.

Steve: Yup. And I guess with that the common scenario that I see as a freelance consultant as Carlos and I are. I see that you will be working with a client and they’ll come and they’ll say, “Things have really slowed down. We haven’t changed anything on the database and something has really slowed down on this one specific process or job.” Or whatever it is that’s calling into to use this query. And then you’ll go and look at it and you’ll find here is the stored procedure that’s the root cause of the problem. You’ll copy it out. You’ll copy and paste it and put it into your Management Studio and change it around so you can run it, maybe not as a stored procedure but in line. And you run it and everything runs great because with the changes you’ve made to go run that in line it gets a different plan. And then you look and you think when you look at weight statistics and something like and you can see that like at noon today that’s when things just tipped and it went bad; and prior to that everything was running great. So what could have happen to cause that and what often happens is that you have a stored procedure that gets pushed out of the plan cache for some reason. The next time it is ran it is run with this unusual parameters sets which causes it to get a bad plan and then every other call into that starts using that bad plan. Or it was good plan for that one parameter but it’s a plan for the other parameters. And I’ve seen that take a stored procedure that normally runs in under a second and cause it to run like 7-8 minutes when it’s something more complex. Then everyone they hear that and they grip and say, “Oh, the database is horrible and everything broken.” And this is all really because of the parameters sniffing and parameter sniffing is a good thing because without it SQL Server wouldn’t be able to optimize your plan for any kind of parameter. But occasionally it goes wrong and you end up with the wrong plan based off of plan reuse on that stored procedure.

Carlos: And you may be thinking how big of a problem could this be. Now, I won’t say it’s the only reason but the new feature of the query store feature is basically trying to solve this very problem. And so it’s big enough of a problem that Microsoft has built a tool to help you combat it. And I think it’s one of those things that I think a lot of, at least I can remember going back and being frustrated and I think a lot of times it probably had to do with parameter sniffing.

Steve: Yup, and most of the time I ran into parameter sniffing issues it starts with customer service department who’s working with customers getting lots and lots of complaints that something has gone wrong or the system is down and not responsive. And then it leads to then eventually jumping into the database and finding, ok here is the problem, and then there is a lot of different things you can do to help mitigate the issue with that point. I mean one of them is people will put the option recompile on a query or the recompile hints on a stored procedure and that can cause more problems or different problems.

Carlos: Well, let me explain that. Let’s just go through that just for a little bit. So now I guess what we’re saying is to mitigate that you can tell as it goes through the process of actually creating the plan you can get, there is some we call hints, because a stronger word is plan guide but that is slight different. We’ll touch that in a minute but you can tell SQL Server, “Hey, SQL Server you should do it my way.” Or in the example of the recompile you’re saying, “Hey, every time you run this I want you to recompile.” I don’t want you to use the plan you have available to you. I want you to throw that plan away every time and figure out how to do it again. And so that’s an option that you have but there are risks if you will associated with that and you want to be careful about how you go about or when you go about implementing that option.

Steve: Yup, and one of the knee jerk reactions that I often see when I explain to people that, “Well, here is what happened. You got a parameter sniffing issue. The query got a bad plan, or the stored procedure got a bad plan. Here is what we did to fix it. We force it to recompile or we change the query around.” They often think, “Well, can we just force all of our stored procedures to recompile so we’ll never have this issue again.”

Carlos: Yeah, exactly. How do I eliminate the problem from never happening again.

Steve: Yup. And then the answer to that is, well you could but however by doing that you would eliminate any of the benefit that you get from the plan cache and be able to reuse plans. And depending on the system and the performance there, that could make things far worst over time depending on the load. So there are things you can do. You can go in and perhaps change the queries or change how things are working in the stored procedure or maybe have a stored procedure that calls another stored procedure that does something different. I mean there is a lot of different, there are probably 20 different ways you could go to figure out how to do this right. I guess we don’t have time to go through all of those now but sort of the knee jerk reaction is just make it so that everyone recompiles and that’s not a good thing to do.

Carlos: Well at least by default. I mean that may be an option that you pursue but don’t make that your first choice.

Steve: Oh yeah. And I’ve certainly used that for a single stored procedure or a couple of problematic stored procedures. Used the recompile option on them and every time they are run they get recompiled. And it’s just because of how they’re written and they are in positions where they could be rewritten but the overhead of recompiling those is cheaper than the time it would take to go rewrite those.

Carlos: Right, but I think it kind of comes down again to kind of knowing your system and understanding what the, so you understand the problem, “I believe I have a parameter sniffing issue.” What do I know about this either query, procedure, view, whatever it is and do I know any history about it? Can I go find some of that out to then understand what makes the most sense?

Steve: Yup. And we could probably go for hours on parameter sniffing but let’s shift back a little bit to sort of the generic plan cache topic now. So one of the things that often comes up with the plan cache is people say, “Well, how do I control the size of the plan cache?” And, you can’t. It’s something that’s dynamically sized by the SQL Server internally and it depends a lot on the server load and what memories are available. One way to control it is just put more memory one the server but that’s not a really good answer.

Carlos: Well, another feature that they added, I’m forgetting the edition, I want to say was it 2014? It seems like it was older than 2016 but maybe I’m remembering wrong. And that is when they added the ability to add flash, so if you have flash on the SQL Server you could actually expand the plan cache to use that flash array to give you more space when you have an issue like this. So to kind of indicate the gravity of problem Microsoft is putting solutions out there around the plan cache and its size. Even if they are not giving you the controls of like in Oracle to say this is what it should be.

Steve: Right. So the best way I’ve found to deal with the plan cache if you’ve got stuff that is getting pushed out or a lot on one-time use queries in there and things like that, it is to better understand what’s in there and then it might be and I’ve worked on systems that I have tens of thousands of different queries run against them and then it turns out there is a dozen queries that are really the big offenders in hogging up the plan cache was one-time use queries. And you can go in and work and optimized those dozen queries to use parameters or do whatever needs to be done there. And oftentimes with a small amount of work you can have a really big impact on the plan cache there.

Carlos: This is where the setting, is that the right word, for optimized for adhoc workloads comes in. So this idea of this adhoc is that, “Hey, I have a one-time use query. I have a whole bunch of those. What I’m going to do is instead capturing or keeping the full blown execution plan is I’m going to just keep a little stub. The first time it gets run and then when it get run the second time then I’ll keep the whole thing and make use of the being able to run it more frequently.

Steve: Yup and that optimized for adhoc workloads is one of those parameters that we most of the time will recommend people turn on.

Carlos: Yeah. I know we’ve talked about it before. I only ever heard of one person complaining about it which we actually talked about, like it was in Episode 99. Mindy brought it up, you were in the panel in Baltimore. I remember Wayne Sheffield talking that he’d seen some CPU spike but I think. Again, obviously you have to test in your environment, right? But it seems like it’s almost one of those standard features that you can enable now.

Steve: Yup, and that’s why I said we almost always recommend it. Not always but almost always. So then I guess, I mean as far as understanding what’s going on with your plan cache, and I know we talked about Database Health Monitor a little bit earlier but in the very beginning when I first created Database Health Monitor some of the very first reports that I built were around understanding the plan cache because I was working on an environment where it wasn’t well understood and I needed a way to built a show what’s going on with the plan cache.

Carlos: Sure. And I think at least in my first interactions with that are giving you the top queries because it will keep some statistics about the plans and their executions. You can go and start to interrogate that a little bit. Generally, from a performance perspective that was the first time I remember kind of going and taking a look. It’s like, well what are the top plans by CPU, or by memory or just how long it ran, something like that.

Steve: Yup. And there are four reports in there that I usually look at that are right around understanding the plan cache, and one of them is just called the plan cache report and it’s a pre database report. And what it will do is it will show you the first 500 largest plans in the plan cache and it will show you how big they are. So you can go and see, “Oh wow, we’ve got 12 plans that are very similar that are taking up 30K each. And you do the Math and you add it all up and realized, wow some of this add up real quick to taking up a lot of your cache. Another one that’s really handy is the needs parameters reports. And what it does it goes through it and analyzes the queries that are in the plan cache and it looks for things that could be parameterized and then it groups all of those together. So if you had 1,000 queries let’s say customer name in them that was hard coded in the query, it will go through and say that by fixing this one query it would reduce your plan cache from a thousand instances of that same or similar plan to be one reusable instance.

Carlos: Now let me ask you a question on that because I guess this is where I’m drawing a blank here, it was a gap, right because I thought that even. So we talked a little bit about stored procedures versus adhocs so views or inline queries. But I thought even though I was in line query and I’ve been written with like framework or something. If the SQL Server gets that query it’s still going to try parameterized it. Even those on store procedure I guess is what I’m saying. Obviously, the very reasons why I would have that but in that scenario what do you then go about doing to solve for them?

Steve: Well, take the example earlier where we’re querying the podcast tables, SELECT * from podcasts WHERE host = ‘Carlos’ or host = ‘Steve’. If you are running that code and it’s actually running that exact query hard coded from whatever application it is. But that’s what’s ending up in the plan. It’s hard coded with Carlos or Steve in there. That is taking up just for those two queries, two or sometimes four, plan cache entries. And let me just clarify that when I say it’s two, that’s the obvious, one for the query that is looking for Carlos one for the ones looking for Steve. But sometimes you will get a parallel and a non parallel version of that query in there so sometimes a single query will have two different plans in the cache. But to go back to what you’re looking for there. If the application is passing through hard coded strings like that each one that it’s passing through will get a different plan, so then that’s really all the needs parameters report does is it goes and finds the items in the plan cache that are very similar with everything besides the parameters.

Carlos: So I guess let me ask the question this way then.

Steve: I don’t think I answered that, did I?

Carlos: I think you did answer it. I think I asked the question wrong.

Steve: Ok.

Carlos: So I am misunderstanding and SQL Server will not in every instance try to parameterize your adhoc queries.

Steve: Yes, that is correct. And the way to tell that is to look at what’s in the plan cache. And if what’s in the plan cache contains those hard coded values for a name for instance, then they haven’t been parameterized. Or the other way to look at it is if you run it with two different parameter or two different names in that value do you get two copies of that in your plan cache? And if it is then it is not parameterizing it.

Carlos: Ok.

Steve: Now with that, I guess the thing I like to do is find the ones that are the most commonly ones that need to be parameterized. This only works if you have access to the code because if you’re running an off the shelf application where you can’t change any of the code you might not be able to do this. But if you are a development organization and you’re building an application, if you can go in and find that these are the queries that end up using the most out of the plan cache. They have big plans and they are called thousands of times and then you can figure out which one needs parameterization and go parameterized a couple of those you can often times have a big impact on the amount of those one-time or low use plans that are in the cache.

Carlos: Again, to connect the dots here. There is where we’re actually going to the code and instead of using that hard coding you are going to use like sp_execute or changing the way that you are making that call to the database.

Steve: Right, and I mean for instance if you are working in pretty much any programming language that I’ve ever seen or work with that allows parameterization, you usually pass through some kind of a parameter value like a variable name in place of what were you would be filtering on that hard coded string, and then in the code you say, here is the query patch it up with these parameters and then execute it.

Carlos: So the developers have to make one more step in putting that dynamic, I’m assuming it’s a dynamic query linking it all together. Before they send it to the database they need to make one more step to help us out of it.

Steve: Then we can have a whole other conversation for probably an hour for parameterization and the impacts that it has on preventing SQL injection or helping prevent SQL injection. I mean there is another benefits to parameterization just besides the performance. Maybe we’ll save that for another time. So another report, there are a couple of others that I look at is the one-time use query report. Did I already mentioned that one?

Carlos: I think we may have touched, we talked about it but this is just to show us how many queries have just been executed in one time.

Steve: Yup, and that’s a handy way to see how many of these are there. And if you look at your database and there’s 2 or 3 or a couple of dozen, you probably don’t have to worry about it. But if you find out that there are thousands of them there then maybe it’s something you need to look into.

Carlos: And then that’s where that optimized adhoc workload comes in.

Steve: And then the other report in Database Health Monitor that I really like to use and understand is the instance level memory report where you can go in and see how much memory is being used by each database, but it also shows you how much memory is being used by the plan cache. And it’s interesting, on some server the plan cache might be using more memory than some of your databases are. I mean it depends on the size of your database and performance and things, and the load, but it’s just good to understand how big it is. And I guess I said earlier you can’t really control the size of it but you can control the size of it by reducing the amount of one time used queries either through optimized adhoc workloads or by adding parameters.

Carlos: Influencing what’s in there.

Steve: Yup, yup.

Carlos: So another thing we will touch on, we talked about it earlier, as we talk about the ability to manipulate a little of this, right? So through using hints, recompiling, the other one is plan guides but we also want to say. Again, take it for what it’s worth but in my years as a database administrator I’ve only seen hints and guides used in two instances. And so I think sometimes particularly on the forms we kind of see that get rushed, again this idea of, “I want to make sure it never happens again so I’m going to put this very extreme process in place.” When maybe testing it out a little bit would be a better approach.

Steve: Yup. And I think in my experience I’ve seen that hints are used quite often but I’ve see that plan guides are used very infrequently. I just want to take a second to jump back to a previous podcasts where I talked about one of my things I would change around SQL Server I think was the term hints on plan guides. And that hints aren’t really hints, they are really commands. And plan guides aren’t really guides, they are commands that’s say, “You will perform this way.”

Carlos: Yeah, exactly.

Steve: I look at hints and plan guides oftentimes the first thing I’ll do on performance tuning is pull the hint out and see how it performs without it, and oftentimes things improve. But I think that they are kind of emergency band-aid type response that when you’re out of other options it maybe something to consider as a short term solution.

Carlos: Sure, and I don’t mean to say they shouldn’t be use. When they are appropriate they are appropriate. But I think, again kind of the whole rebooting the server, right? Like, “Oh it’s slow, let’s reboot it.” Stepping away from the knee jerk reaction. There are going to be instances where they are called for and where you’re going to be the hero for implementing them.

Steve: Yup, and I think plan guides are amazing, they are extremely awesome but they are extremely dangerous and I don’t want anyone to think of this podcast what we are talking about here is recommendation to say go try out plan guides for your performance tuning. If you’re hearing that you should translate to go learn all you can about plan guides before you ever try it because there are some negatives in there. If you apply a plan guide it may cause trouble when you try and recompile a stored procedure that’s being utilized or the plan guide is associated with.

Carlos: And again kind of going back to query store, that’s the flip side or other angle is going to help you understand is, “Hey, this query is running with a plan guide or a hint, just so you know.” It has been difficult. It’s something you really have to spend a time with so again, the knuckle dragging Neanderthal that I am to kind of understand, ok what’s going on here? How are these plans changing over time? So it does take some practice and just hanging in there. So you will get a little bit frustrated but hang in there and eventually it will come.

Steve: Yup. So I think just kind of a quick recap overall so basically the SQL Server plan cache is where all the compiled query plans are stored. They get compiled at the first time they are used or when someone indicates that it needs to be recompiled. And it’s kind of sized dynamically by SQL Server. You don’t really have a lot of control over that and there are some things you can do to adjust that like optimized for adhoc workloads and parameterization. And I think hints and plan guides can oftentimes cause trouble but they are kind of a last ditch attempt to try and fix queries.

Carlos: So again, ultimately we would like your feedback on this and one of the areas and kind of talking, reviewing this topic again is we would like to try to make Database Health Monitor better so we’re listening to your feedback. We’d love for you to take a peek at those reports. We’ll make sure that we put them on the show notes page and list them there. We’d like to get some feedback, so as you use them what do you like about them? What else do you want to see? How do you use them? Is there some other tool that you’re using to look at the plan cache. We would be interested in hearing from you about that and you can use any of the options of social media to leave us a comment or a thought there.

Steve: Or you can leave on the podcast show notes page as well.

Carlos: That’s right. So our episode URL today is sqldatapartners.com/plancache. Again thanks again for tuning in to this episode. If you want to connect with us on LinnkedIn I am @carloslchacon.

Steve: Or you can find me on LinkedIn @stevestedman.

Episode 102: Monitoring Availability Groups

One of the newer features in SQL Server is availability groups, which can help solve a number of business problems.  As administrators, availability groups introduce some complexity as we are tasked to make sure the Recovery Point Objective (RPO) and Recovery Time Objective (RTO) can be meet for these servers.  The complexity comes because we have different instances that must work together, but they don’t always see eye to eye.  In this episode, we chat with Tracy Boggiano of ChannelAdvisor about how they go about monitoring their availability groups and the pros and cons of the out-of-the-box tools.  Our discussion touches on the availability group process and Tracy has posted her scripts on her blog for you to use as you look at reviewing your environments.  I think you will enjoy this discussion.

 Episode Quote

“You just need to allocate the resources and play with the stuff in your staging environment and make sure you have resources”

“I much prefer having a query and using PowerShell and just running in the multiple instances

“We use a third party monitoring solution for our monitoring rather getting a whole bunch of SQL agent alerts”

Listen to Learn

– Monitoring availability groups
– Data synchronization on availability groups
– Asynchronous and synchronous mode
– A review of RTO, RPO and SLA
– Errors and agent alerts

Tracy on Twitter
Tracy on LinkedIn
Tracy’s Blog

About Tracy Boggiano

Tracy BoggianoTracy is a Database Administrator for ChannelAdvisor. She has spent over 20 years in IT and has been using SQL Server since 1999 and is currently certified as a MCSE Data Platform. She also tinkered with databases in middle school to keep her sports card collection organized. She blogs at tracyboggiano.com. Her passion outside of SQL Server is volunteering with foster children as their advocate in court through casaforchildren.org.

Transcription: Monitoring Availability Groups

*Un-transcribed introductory portion*

Carlos: Tracy, welcome to the program.

Tracy: Thank you for having me.

Carlos: Yeah, it’s great to have another companera here with us. You are going to join us on October for the Companero Conference and so we appreciate that, and for you being on the show today.

Tracy: I appreciate you having me.

Carlos: Yeah, and I’ve been telling Steve we’re going to get through all those ChannelAdvisor folks. Now, we get to cross another one off our list.

Steve: Are there any DBAs there that we haven’t talk to yet?

Tracy: There is a couple left. Yes.

Steve: Ok.

Carlos: In fact, one of them I actually, so I met when I was down there in. He is a big fan of PSSDiag and so I talk with him about maybe coming on and seeing if he could convert me.

Steve: Oh, that would be interesting.

Carlos: Yeah. I’m like, it seems like a little. Anyway, that’s for another episode.

Carlos: Oh boy! Yeah, I’m not sure. Not today’s episode.

Steve: Speaking of which today’s episode is on monitoring availability groups.

Carlos: Yeah, that’s right. I think this is going to be near and dear to anybody who is trying to setup availability groups and ultimately that idea that you have DR scenario/situation and you want to make sure that the data is getting over to the secondary node. That you’re not going to lose that data that you kind of meeting your SLAs and so being able to ensure that just makes a lot of sense. And so, Tracy, it might be helpful why don’t we take a minute and just review that data synchronization process as we talk about availability groups? What are the components and what are some of the pieces that we need to be looking at?

Tracy: Ok. Well, the first step is that the log has to flush to the disk, and you have a log cache that it caches the records into that it has to send over to your secondary server. And it is stored in this area called the log capture area and it sends across the network, and at some point it gets an acknowledge commit whether you are in synchronous at asynchronous mode and that depends on what mode you’re in. Once the log is received on the other side it is stored in another cache into disk. And on the other side you have a redo thread that sits there and replace the pages back into disk. And you have a performance counters that capture your logs and queues, size, and the rate that it’s sent, and you also have the redo size and redo sync rates that are all captured and performance counters on both sides so you can monitor those.

Carlos: Right. Now, so we talk about asynchronous versus synchronous, right? So when I send it over, writes to that second node and then writes to the log, that hardening process. If it’s in synchronous, once it writes the logs, is that when I get the acknowledgment back? Or does it actually go through the redo process before it will acknowledge that?

Tracy: Once it hardens to the log it is sent back as committed. And when you’re in async it sits and hardens on your primary it’s considered committed.

Carlos: Ok.

Steve: So when we are working with that what are the things that we really want to monitor or keep tracking to make sure things are working well.

Tracy: One of the first things you want to make sure you don’t have is a lot of network latency. A lot of times this technology especially if you’re in async mode you’re looking at a DR situation or you had your secondary site in a secondary location like an AWS or different data center. You don’t want to have too much network latency. The other being that you want to make sure that your secondary storage aren’t slow and not able to process the data as fast as you need them to. Some companies like to go skimpy on their secondary servers and not have them as their primary servers. And they can get behind just because they don’t have
enough memory or CPU resources. So you want to keep an eye on those redo queues and make sure if it’s actually able to keep up or not. And that’s where it’s important to keep up with your SLAs and make sure that you’re actually meeting those or not.

Carlos: Right. Now, this is very interesting concept because you think, “Ok well, all that secondary server has got to do is process the log”, but on busy systems that can be quite a bit of work. And to keep up with that and again, ChannelAdvisor, you guys get to play with all the cool toys that some of the higher transaction rate systems you actually needed a pretty hefty server, secondary server, just to be able to keep up with that even though it wasn’t actually doing any “OLTP” workload.

Tracy: Yes, we had a system with a 256GB of memory on the primary side that we still needed 64GB of memory on the secondary side just to process the redo log to keep it current. I mean, we still are able to take 25% out. You know, we try to go less than that and we were trying to run in the cloud which can be a little bit more expensive and we weren’t able to do it. We started off at 16GB and it just wasn’t performing.

Carlos: Now, was that synchronous or asynchronous?

Tracy: Asynchronous.

Steve: So then when you say it wasn’t performing there, was it just that the continuous load was so great that it couldn’t keep up? Or was it that there were big sets of transactions that were kicking off at some point in time that were causing it to backlog and not catch up?

Tracy: We just had too many transactions going. We were hitting, 20,000-30,000 transactions per second and it just backing up on the secondary side. One we bumped up the memory it’s plainly a memory problem just trying to read all those pages into the buffer pool so it could update it. And once we bumped it up to 64GB it was able to keep up.

Steve: So then with that secondary backing up, if it’s backing up to the point that it can never caught up, what is the outcome there in the availability group? Does it eventually crash because of that or is it just very delayed?

Tracy: It’s just very delayed. It’s behind to the point you are not able to failover if you’re meeting your SLAs. In this instance we were just testing our AGs for ourselves. It wasn’t a production instance for us as far as we wanted to failover to it. It was just us testing our AGs on a production instance but not for a disaster recovery situation. But for us to see what we would need in order to setup a DR situation and we discovered that, “Hey, we’re going to have to allocate some resources to it.”

Carlos: Alright, now that’s another interesting question because you are failing over and it was just your test, so when we talk about not keeping up what is your window there? Is it 1 minute, 5 minutes, an hour, a day?

Tracy: Our service level agreement is set to 4 hours. But we were finding that the server is up to a day behind.

Carlos: So again, just to put in some of the perspective some of that, right? I wasn’t like you were asking for milliseconds or something like that. I mean 4 hours is significant, right?

Tracy: Yeah. I say you just need to allocate the resources and play with the stuff in your staging environment and make sure you have resources. We played in staging but we don’t have the transactions per second on the current staging that we have in production.

Steve: So I know when you’re monitoring presentation there are a few acronyms that you brought up along the way being RTO and RPO and SLA and how those apply in the availability groups scenario. Can we maybe talk a little bit on those and cover what that means to the availability group with the RPO and RTO and how
those apply to SLAs.

Tracy: Yes, the RPO is your Recovery Point Objective and that tells you how much data you are able to lose so that measures how many megabytes, or gigabytes, or data you are allowed to use so how much data has change that you can lose. Your RTO is how much time you can lose, so that’s how long it is going to take to do that redo on the other side. So if it says it’s a day behind that’s a day you have to wait for that redo log to play. These two are measured differently. You could have a gig of data behind but it will only take an hour to replay it or it could be the reverse, it could be a day behind and a gig of data to replay. It depends on how your system is setup. And those two combined to make your SLAs, your Serious Level Agreement, what your businesses agreed to allow you to lose data and how much time they are allowing you to recover that data.

Steve: Ok, great, so when we look at the built-in dashboard for availability groups. How well does that really do the job for you to be able to monitor and know what’s going on?

Tracy: When you are looking at a single AG it pretty much does the job for you. If you’re in our environment where you have 100+ databases, 100+ servers and you’re looking to have AGs on all of those, not so well. But you have to look at, you have to connect everyday and then to see what’s going on.

Carlos: So kind of broad brush strokes, it’s cumbersome. You have to connect to every single one of them.

Tracy: Yeah, you have to connect to every server and bring up that gone down through the dashboard in Studio Manager and bring it up. That’s why I much prefer having a query and using PowerShell and just running in the multiple instances and see which ones are on trouble, and going and checking those out. But overall is you have a couple to manage. The dashboard shows you what you need to know.

Carlos: Ok, so it’s pretty good there. It’s kind of meet your needs from an individual availability group or perspective but just that once you start getting with more than one then you guys are going to start rolling your own.

Tracy: Yes.

Carlos: So I guess talk us through some of the ways or some of the things that you’re looking at to accomplish that?

Tracy: Well, we’ve used the queries that the dashboard runs in the background. And one thing we’ve created is some PowerShell command lines that we can run and then it returns in a data grid and then we can sort and filter. And we can just run that again to any of our AGs that we have and fun data. We also have more focus onto our team who loads a lot of stuff into Grafana for us so we can see data.

Carlos: Yeah, you guys likes to meddle, who likes to tinker.

Tracy: He has our performance monitor, counter talking about before our logs and queues, our redo queues and the rates all logged into Grafana for us so we can go there and view them by instance if we need to. So we’ve got a couple of different solutions that we have that we run with right now.

Steve: Yup, so when we talk about your PowerShell scripts that you’ve got there. Are those something that’s internal and private or is that something that’s available for other people who take a look at.

Tracy: I actually have a blogpost to my website that you can pretty much plug in any SQL script you want into and run against multiple SQL Server and turn it into grid, so that you can run any troubleshooting script you want and it will run across any list of SQL Server that you provide and return it in a grid.

Steve: Ok, we’ll have to include that in our show notes, link to that.

Carlos: Well, another kind of interesting we’ll call it a hack because I like to call things hacks today. But that little cheat if you will of have a SQL Server 2 something. You pull up the manager, or the wizard, or the dashboard, whatever and then you pull up profile, I’m assuming or extended events, if you so choose and then you just figure out how it’s getting all its information and then you use that down the line.
I think that’s a great way to be able to grab that stuff and understand what the SQL Server team has put together to look at.

Tracy: Yeah, I saw what I did. First, I pulled up the dashboard. I’ve seen it only comes with like 5 columns and it’s like that’s not very useful. It has a add/remove columns out to the side so I started looking at the columns names and I was like, “Hmm, these are the things that are useful to me and that’s what I went into the DMVs to find.” And that’s what I added to my queries.

Carlos: Interesting. So I am curious and maybe it’s different because I know you guys are on AWS. So in Episode 76, we had Jimmy May on and he was talking about some of his testing experience and the improvements they’ve made particularly in their redo log and whatnot. It just made me think are there scenarios or real situations or is there anything that you can do so let’s just say maybe the scenario would be, I guess it’s not really a cluster, so patching. This may not work but I want to use the patching scenario. Maybe not exactly what I’m looking for but you need to failover. You need to move from one node to other for whatever reason. And let’s say that you’ve gotten behind, what are you going to do or what kind of things are you looking for to help you get back up to speed? Does that make sense?

Tracy: Yeah. In our situation, the AGs we currently have it run in production are synchronous on premise ones because we don’t have our DR stuff that’s setup currently for AGs. We have a different solution for DR at the moment. So those are synchronous and up to date but if you were doing an async environment because we have done this to do migrations. And to migrate from 2014 to 2016, we use AGs, so we set up AGs as asyncs and then when we got ready to failover we set them in syncs. Like the day before so that we get caught up and then we did the failover. It would be the same though with patching. You would go ahead and patch your server, set it to sync, but it force it to catch up, and once it’s caught up you’d failover to the patch server.

Carlos: Got you, ok.

Steve: Ok, so then I guess one of things that I’m interested in is that with the monitoring that you’re doing it sounds like you’re doing your own beyond what the built-in is just so you can get across multiple servers all at once. But are there additional things that you’re monitoring that aren’t part of what you would see through the normal dashboard there.

Tracy: The only thing I say I’m doing differently is the rate. The rates in the dashboard aren’t updated and laugh because Microsoft only updates those when data is being sent across. But if you’re not actively sending data the rates aren’t updated so you got some monitoring around your rates. It may not always be accurate so I do the monitoring base on the performance counters DMV rather than the DMV that just stores the rates for availability groups. So I hava a as part of my downloads for my presentation I have a thing that captures it, captures it again and actually calculates the rates that you can see the difference between what DMV for availability groups say and what rate actually is. Just in case you need to know what the rate or what the real rate of actually use as occurring. But other than that it’s pretty much straight what the queries are running on the dashboard.

Carlos: Ok, great.

Steve: Now, there is another feature or option that is available to us to be a little more proactive if you will as far as letting SQL Server notify us when it has a problem. A lot of us are going to be familiar with the alerts. I don’t know if they are called agent alerts but these are the 19-25 or 20-25 and then those like 18-32, 18-24 etcetera because of problem breathing from disk and things like that. SQL Server can say, “Hey, I had this problem, just thought you should know.” We have a
couple of those for availability groups, right?

Tracy: Yes, I think there are about 5 of them. There’s an error 1480 that lets you know if the server failed over. There is an alert for when the data movement has suspended for some reason. I’ve seen it suspended because the disk is out of space for example. There is also one that will take when it resumes. So if it decides to resume by itself or somebody actually puts a button and resumes it. It will tell you if the AG goes offline for some reason and if you’re in synchronous mode it will tell if the AG is not ready for an automatic failover. You got to set up for automatic failover. All those numbers are available using my, you can look those up online or download my slides.

Carlos: Sure, and I suppose something like that, not keeping up some tinkering will need to be involved to kind of make sure that you have the right secondary formula because I would have imagine you don’t want to be getting a lot of those emails. You get a couple of like, “Ok, more memory than the secondary”, or something, right?

Tracy: Yeah, that’s why you wouldn’t configure your secondary for automatic failover if you didn’t want to know if it wasn’t ready for automatic failover for example.

Carlos: Well, fair point. And I guess the automatic failover. Those are probably going to be a little more similar in size.

Tracy: Yeah. You definitely don’t want to have those set up exactly the same. That way when it failed over you got the exact same performance going on because those are going to hopefully happen at 3:00 in the morning. And you necessarily wouldn’t get paid when something fails over automatically either because hopefully that’s what you wanted so you could sleep. That’s why we invented AGs, to sleep.

Steve: Yeah. Nice, so when you’re monitoring your availability groups are there any weight statistics that you normally take a look at or keep an eye on just to see how things are going?

Tracy: There are a few that you can keep an eye on. The one to watch out for is there’s one the HADR_SYNC_COMMIT to tell if it’s taken a while to commit on your side or not. I’ve seen that one pop up when it’s just waiting to commit. We know so a lot when we were low on memory. The other one is write log. That will typically occur if it’s taken a long time to write a log on your secondary. Other than that I haven’t seen any in particular there pop up in our environment.

Steve: Ok, so with the sync commit is that when it’s doing the synchronous or asynchronous mode and it’s doing the commit and you wouldn’t see that if you’re running an async mode.

Tracy: I’ve seen it in async and sync. It’s just whenever it’s doing a commit.

Steve: Yup, got it. Alright. And then as far as any extended events, is there anything we should be aware of or keep an eye on there when we’re working with availability groups.

Tracy: Mostly extended events it creates always on health session by default for always on and everything you need is pretty much in there as far as events that you need including all those agent alerts that we’re talking about.

Carlos: It creates a separate event, separate monitor or it just includes them in the default extended event trace.

Tracy: It creates a separate one for you called always on and it includes all the events that you need to know including all those alerts, the agent alerts we were talking about. Those are all included in there and as part of the demos that I do at the presentation. I’ve got queries that query those out and we actually as part of our internal monitor just query those directly rather than receive SQL agent alerts. We use a third party monitoring solution for our monitoring rather getting a whole bunch of SQL agent alerts.

Carlos: Yeah, that’s right. I mean once you get so so many instances you’re going to want kind of a complete solution there.

Tracy: We don’t want so many emails.

Carlos: Yeah, exactly. Well awesome, Tracy, great information on availability groups. We do appreciate it.

Tracy: No problem.

Carlos: Should we go ahead and do SQL Family?

Tracy: Sure thing.

Steve: So Tracy, how did you first get started with SQL Server?

Tracy: Well, I was a developer at a company that wanted some reports and Access wasn’t good in it as far as querying stuff so I imported some data into SQL. Well, I had to learn a lot of stuff. I then I just kind of got hooked ever since. I took some training classes on it and next thing I know I was a DBA. I’m not a developer anymore.

Steve: Very nice. It’s interesting to see how often Access is the lead into SQL Server for people.

Carlos: Yup. Tracy, I know you guys are using a lot of features over there at ChannelAdvisor but if you could change one thing about SQL Server what would it be?

Tracy: Right now, I wish hackathon would quit writing so much to my error logs. I’m getting gigabyte log files every night on my system drives and they’re driving me crazy.

Carlos: Oh wow! That is a lot of data.

Tracy: And it’s just informational messages.

Carlos: And there’s no flag or nothing to do to turn that stuff off.

Tracy: Nope.

Tracy: I’ll stick with that one for right now.

Carlos: And so what’s your say about that? Surely you’ve, he knows about it.

Tracy: I don’t know. Brian has mentioned though. I’m not sure what Microsoft said about it yet.

Steve: Alright. What is the best piece of career advice that you’ve ever received?

Tracy: Just to always keep learning stuff. Feel change a lot in IT. What you know now is going to change. As you can tell SQL Server. We’re getting releases like every year. And now we go learn a new operating system.

Carlos: Yeah, never stop learning. Tracy, our last and favorite question for you today, if you could have one superhero power what would it be and why do you want it?

Tracy: I want to be able to teleport.

Carlos: Teleportation, ok, and why is that?

Tracy: I spend a lot of time in my car doing volunteer work and it will save me a lot of time.

Carlos: There you go, very good, awesome. Tracy thanks again for being in the program with us today.

Tracy: Ok, thanks for having me.

Steve: Yup. Thanks, Tracy.

Episode 100: Role Reversal

Something a bit strange happened in episode 100–almost like something out of the twilight zone, but don’t take our word for it.  Check out our latest episode as we tell some of the stories that led up today.

SQL Server Podcast

Transcription: Role Reversal

*Untranscribed introductory portion*

Kevin: Hello friends, welcome to the SQL Data Partners podcast. The podcast dedicated to SQL server related topics which is designed to help you become more familiar with what’s out there, how you might use those features or ideas, and how you might apply them in your environments. I’m your host Kevin Feasle. Today on the podcast we have two special interview guest, Carlos L. Chacon and Steve Stedman. Carlos and Steve, welcome to the program!

Carlos: Oh, thanks for having us, Kevin.

Steve: Thanks, Kevin, this is exciting.

Kevin: Absolutely, so we’re on Episode 100 of the SQL Data Partners podcast and oddly enough you also have a podcast. It’s weird how that works, huh.

Carlos: Yes, there is an interesting turn of events here.

Kevin: Carlos, what made you decide to start podcasting?

Carlos: Wow, that’s a great question and I guess I will say that the front if I knew how much time and effort it was going to take I don’t think I would have started it. So I knew that I wanted to engage other folks and start talking about SQL Server in a kind of a long form way. I’ve been doing a bit of blogging. Ultimately, looking to help my consulting practice or re-launch it really in a way. And so that kind of content marketing, so taking the long view of having content available to people kind of interact with find, search engine optimization, things like that. I’ve been doing some blogging. I tried to do some videos and just found that difficult. At that time there were only two SQL Server podcasts in iTunes which is the main place where people go to find podcasts and then there are lots of apps that will carry the podcast that are in iTunes and Google Play has come out there. Ultimately, I thought, “Gosh! There is only these two.” And Greg’s SQL Down Under hadn’t new episodes hadn’t been there for a while and so I take in John Lee Dumas’s course on podcasting and thought, “Hey, you know what. Why not, right?” Let me jump in, let’s see what happens here. I guess I will try to do 10 episodes. So before I actually officially launched I’ll do 10 recordings, see if I like it. See if I can actually get 10 people to sit down with me and talk. And what’s weird, so I started the podcast when I was in Costa Rica. I took my family over there for two months and we were down there. And while I’ve done my first interview actually in Argentina at a SQL Saturday, I kind of officially started doing interviews in Costa Rica. So that’s the kind of the long answer to why I started it. I though there weren’t very many people at that time doing podcasting and I thought I would give it a try and kind of see what happened, and wanted to commit to do it for one year.

Kevin: So when it comes to things that are time consuming, things that are kind of beneath the iceberg, what are the most time consuming parts of creating a podcast episode?

Carlos: Steve, I think you will attest to this. The first is just getting use to hearing your own voice.

Steve: Sure, and realizing when you’re doing that that you don’t always have to do it over again and just because it doesn’t quite sound the way you were hoping it sounded.

Carlos: Right. You don’t have that ability to do the editing, right? In the written word you could edit it, “Oh, that doesn’t sound quite right, let me go back. Let me tweak that.” As you do that audio-wise, just hearing yourself repeat the same thing over and over gets a little cumbersome. You know, trying to remove all the “uhms” and “ahs” and whatnot. In the beginning I wasn’t using an editor, I am now. That actually happened in Episode 29. That started happening and so I would have to edit my own. So first is scheduling the guest, picking the topic, creating the agenda, actually having the interview, making sure that I had questions so the prep work associated there, then editing it, writing show notes, getting links together for the show notes page; so those are some of the pieces that are involved. But the biggest piece in the beginning, again, I had those 10 episodes and I had told people that in August 2015 is when I would first start. So August came around and people started asking me, “Hey, have you launched that podcast yet?” So literally, again I was then in Costa Rica, that week I spent getting everything ready, and did a lot of editing. And that was really probably the biggest piece in the beginning that just took so long was just listening to everything again trying to figure out, “Ok, is this ok to keep?” Again, you don’t
know what people are expecting. You don’t want to disappoint the people that you’ve interviewed, you know, all those things. Those are the components I wanted to.

Kevin: Yeah, I remember really early on when we first got the other I think it was Episode 13. You had a little piece of paper where you’re writing down, ok this mini minutes, this mini seconds, that’s where somebody said something really bad. You got to cut that word out.
Carlos: Right. No, exactly, yes so I guess it’s interesting that way that processes changed a little bit. We’ve gone some good feedback from the show and now the processes, we actually just record it, you know how it is. I take it off for transcription and then I get the transcription back and I edit the transcription and then Julien our editor, great guy, will actually then edit out anything that I don’t want there. I mean, in addition to all the “uhms” and “whatnot” which he does I think a great job of. So that’s some of that how that processes changed a little bit. Because when I was doing it, yeah, I wanted to write that down because I wanted to try to speed that process up.

Steve: Just to add a little bit more on the time consuming parts of it. I mean, the one that Carlos does most of the time is the scheduling of the guest. I know that one takes up a lot of his time. But then once we have the guest scheduled it’s a couple of different recording sessions that we go through in order to get an episode out. We’ll have the session with the guest which can be one or more guest and that’s usually at least a week before the podcast airs. And sometimes this is much as 3 or 4 weeks at a time when we have a lot in the queue there. But that’s usually about, I don’t know, a half hour to an hour of preparation time we go through there to be ready to talk about whatever the topic is. And then it’s usually about a half hour to an hour of actual recording time, and that’s gives us the section that’s the part that we are talking with the guest. And then about a week before the podcast airs, usually that’s the Thursday before the podcast airs; we do our intro and closing. And that’s where we go in and we talk about the SQL Server in the News. We talk about any mentions that we’ve had out there and then we go in and sort of digest what we talk about with the guest at the end as well. I think that’s usually about an hour of time to put that together.

Carlos: Yeah, that’s true.

Steve: And then once we’ve done that part, or maybe Carlos you can jump in with any additional time but it’s kind of handed off to the process through the editor and through the assistant that we have in getting that all published.

Carlos: Right, yeah I mean, I guess thinking back just because we do have that process now which helps quite a bit but there is still each of those individual pieces to take some time.

Kevin: Oh, I can imagine.

Steve: And then once it’s out then there’s promoting it. And I don’t know I always get around doing it myself but we try and do what we can to promote the podcast through Twitter, LinkedIn or places like that so people know that there’s new episode.

Kevin: Cool, so next question. I’ll start with you, Steve. What episode was your favorite?

Steve: Wow, well, if you would ask me a couple of weeks ago I would have had a different answer but I think Episode 99 I thought was one of the favorites that I’ve gone. If I said that previously before Episode 99, it would have been the indexing episode that we did with Randolph West. But just the whole impostor syndrome conversation that we had with Mindy in Episode 99 that was different than a lot of things we talk about before and I love it.

Kevin: Yeah, I just listened to it yesterday. It was great. Well done, Mindy!

Carlos: Yes, she did a great job.

Kevin: Very much so. So Carlos what was your favorite episode?

Carlos: Gosh, you know that is a tough question.

Kevin: Choose among your children.

Carlos: Yeah, that’s right, exactly. So generally because I am a more the merrier type of person the ones that I have really enjoyed been the ones where we’ve had kind of a panel type discussion. Right, so I think about Episode 59 where we had Andy Mallon and Mariano Kovo on. I think about episode when we had the panel from the DBA Tools folks on.

Steve: Oh, that’s was Episode 91.

Carlos: Yeah, 91.

Kevin: That one was a lot of fun too.

Carlos: Right, so those, even the one that we did which ironically enough you and
Jonathan had that great interchange and I didn’t get it in the program but the ones that we do with the SQL Saturdays where we have multiple people kind of giving their input or thoughts around. I mean, again, not that the individual interviews aren’t fun but by getting different perspectives just makes the conversation flow much easier. Different things come up that Steve and I haven’t talk about beforehand and it enables the conversation to go in different places.

Kevin: Nice. So Carlos, I’ll start with you this time. What has been the most pleasant surprise for you in the making of the show?

Carlos: I think probably the continued relationships that I have been able to have with the guests. Now, that’s not to say that all of the guests are now my best friends because that’s not true. But for the most part, I’m just looking here at the list here; I have continued conversations with my former guests in some ways, shape or form, so I’ve really enjoyed that. I think being able to connect with folks that I wouldn’t otherwise have been able to do.

Kevin: How about for you, Steve?

Steve: I think it’s a lot of the same lines as what Carlos said. But I would go a little bit further to say it’s not just the guest but it’s also the guest and the listeners in that there’s been a lot of listeners who have reached out to me and connected on LinkedIn. I mean a lot of people follow on Twitter. But it’s really nice when somebody connects and you make that personal connection there and getting to know people and sort of extending the reach of who you know in the SQL community.

Carlos: I guess I will add one thing there and that is there had been more than one guest I’ve reached out to and they’re like, “You want me to do what?” I guess I’ll point one out so in Episode 45, so Wolf, up in Pittsburg. He was the nervous wreck, and I said that lovingly. He did not think he had the chops basically which is again ironic for a guy like him. So it took me a while to convince him to, “Hey, let’s do this. Let’s make it happen.” And then when he finally did to kind of see that boost in confidence it was well received. We had some comments on it so that was very gratifying as well.

Kevin: Very nice, so let’s switch gears entirely away from podcasts. Want to talk a little bit about consulting, so both of you are now independent consultants? Yes?

Carlos: Yes.

Steve: Yes.

Kevin: How long have you guys been independent, on your own or together, independent together?

Carlos: Sure, so I’ll let you go first, Steve.

Steve: Ok, and it’s a complicated answer because it has changed over as different things have happened. But I originally started as an independent consultant about 12-13 years ago. And when I did that, I mean, it was going well and then I ended up with one client that sort of ended up taking up all of my time. And then after about 2 years of being “independent” with only one client, they brought me as a regular employee, and I was there for about 7 years. And then it was about 2¼ years ago that that ended and I went back to true freelancer at that point. I said, “I don’t want to go and get a regular full time job because that’s not for me. I like the challenges of consulting and working with lots of different clients.” And then it was about, so I did that. I started my own company doing that, Stedman Solutions, and that’s been doing great. And then about a year ago, Carlos asks me to join him on the podcast. Not in any more of a business relationship than that but I joined and started helping in the podcast, and then about six months ago, maybe 8 months ago was when we decided that we would merge together between what the two of us do much more closely. Now, I still have some clients I work with under my old brand name that Stedman Solutions. But most of the
new work that we are taking on is under the SQL Data Partners brand doing independent consulting there.

Carlos: Yeah, so for me this is my third attempt.

Kevin: Third time’s a charm.

Carlos: Yeah, that’s right, third time’s a charm. In fact, Steve and I were just talking about this earlier and that is one of the things that I wanted to do is make money in the way that I wanted to make money which can be difficult. And so I kind of got fits and starts. I’ve told people before that, so originally I started consulting because I saw other consultants making very high hourly rate. And while lots of people do the hourly rate thing and that’s all very nice and great and whatnot. Just because you have a great understanding of SQL Server at least does not necessarily mean that you will make a great consultant or business owner, entrepreneur and that’s really the most important key is to stop thinking yourself as a database person and just start thinking yourself as an entrepreneur because those things are different and they get attacked differently and so that was part of my learning curve in this kind of stops and starts.

Kevin: Ok, so let’s say we have somebody in the audience who says I’m ready to go independent. Any of my employers who are listening I’m not that person in the audience. But if somebody in the audience is saying, “I’m ready to go independent and hey you just told me that being an entrepreneur is a completely different story. Well, what types of things do I need to think about before I take the plunge?”

Carlos: Marketing. So what kind of problems are you going to solve? From the tech perspective, as a full time employee, people come to us with problems whether that’s a ticket, an alert, but the work comes to us. So now as a consultant the question is how are you going to find the work and what type of work are you going to respond to, and making sure that you understand what that work is and can describe it to other people.

Steve: Yup, I think I’ll echo the same thing there. And I think that when I talk about how this is really my second time in the independent consulting where I had been doing it before and then it turned into a single client. Part of the reason that happened was at that point in time I didn’t know what I was doing and how to go out and make contact with those new clients, how to meet the new customer. And I think that’s something you can do and you can practice and work with is just who is in your network or who do you know that you can make contact with that could be providing you work. It’s surprising that there is people that I have come across I end up doing work with that I never would have necessarily considered as a perspective client in the past. But I think other things to think about for someone who wants to jump out and give it a try on their own is the security behind it.

Carlos: Or lack thereof.

Steve: Exactly, or lack thereof it. Now, I think that when you have a regular full time job most of the time there’s the illusion that it’s fairly secure. And I used that term “the illusion” because whatever happens in people’s lives, full time jobs can come to an end at any point whether it is company going out of business or a layoff or just someone knocking in along with their manager, that job can come to an end. But you generally have a lot more protection legally in different ways as a full time employee, and you have much more security, and that you know if things get slow for the company odds are that you’re still going to be getting a paycheck 2-3 weeks from now. It’s never guaranteed but with a full time position that’s pretty stable. You know that in every so many days you get a paycheck and it’s generally for about the same amount. And I think that when you go into the consulting arena that changes significantly because you run into what they call bench time or a point where you don’t have enough work for a while. And that comes back to finding your customers and marketing and reducing that bench time. But when you’ve got that bench time you’ve got to have, depending on how you’re paying yourself because the customers pay your business and then you pay yourself out of your business, you’ve got to have a buffer there so that when you do have short times that are either bench time or a period where it’s hard to get payments from clients that you can cover it. And I think it would be different for maybe a single person versus someone who is married with kids. But I know that if what I’m doing if suddenly I stop having money to contribute to my family my wife gets a bit worried about that. Alright, so part of what I do to help mitigate that is one you need to have a little bit of savings in place so that if you got a 2-week period where all of the clients decided they’re going to be a little bit late on payments you can wither that out without having a lot of financial pain right there. And then the other is, I mean around that is you’ve got to be kind of really hard with the customers when they are late. And I know that’s a challenging to do but to be able to come back and say, “I can’t keep on working on this project if you’re not going to pay.” Fortunately, it doesn’t come to that often but I think just being in a position of financial stability and I like to use the number of having 6 months of your bare minimum cash that you need to survive in the bank in order to start out doing consulting because when you start out, you are going to make mistakes. You’re going to have more expenses than you need. But there is going to be a lot of things that are challenging in that first 6 months and a lot of them are going to come down to financial challenges.

Carlos: Yeah, and I think just to echo there with Steve. Talking about that transition from the tech space to entrepreneur space so the soft skills becomes much more important there. So he mentioned kind of dealing with client payments but that whole process of just interacting with people. Once you go independent you are just no longer interacting with technology, that Idea is dead, right? Your clients are people and you have to satisfy their kind of needs first if you will.

Kevin: Right, so what point do you guys engage services of say a lawyer or an accountant?

Steve: Oh, great question. Do you want to take that or do you want me to jump in, Carlos?

Carlos: Yeah, so from the accounting perspective, from Day 1, I wanted an accountant there to at least be able to handle some of those things. So kind of goes back to economics if you have taken Economics course. You know, one country makes coconuts really well and the other one does bananas, they trade, so that is kind of the idea of hiring an accountant unless accounting is your business. Get somebody to help you with some of those things because the IRS does not mess around, at least in the United States. I can always imagine for other countries so you don’t want to get started off in a bad foot there.

Steve: And I little more on that, I mean, I don’t want to be an accountant that’s why I work in SQL Server. I wanted to thank SQL Server. If I really want to do accounting I probably would have would taken Accounting in college and gone that direction. Because of that, I mean, there’s a lot of people out there who are great at what they do with accounting and I would rather engage an accountant when it’s appropriate than try and learn all that on my own. Now, that being said, it doesn’t mean that I want to be completely illiterate on the accounting and financial side either. And I think that there are some tools out there like QuickBooks online that make it so that a lot of the stuff that you might normally need a bookkeeper for that you can do yourself. And then you can engage an accountant when it comes to tax time and all the appropriate times that you need to use an accountant there. Interesting story I mean on this when I first started back into freelance a couple of years ago I engage an accountant that gave some really bad advice. It didn’t feel quite right at that time but it came from my accountant so I believed it and then later I found out it was bad advice and that it made my first year’s taxes very challenging to get done that year. And looking back I don’t work with that accountant anymore but I work with accountants and I do a little bit more checking backgrounds and get a better understanding of who they are before working with them.

Carlos: From the legal side, generally, that’s just in the review process so it’s going to vary state by state and of course obviously country by country what the requirements are for setting up a business. Generally, so at least with me I had an attorney just kind of review some of those things or at least consult to make sure I was doing the right things. My accountant actually helps quite a bit with some of the legwork to help reduce some of that cost.

Steve: Yup, and I think that, I mean the key is use lawyers as needed. And I think there’s a lot of people who gripe at lawyers in what they do but when the time comes when you really need a lawyer. I mean again I don’t want to be a lawyer myself. I don’t even want to try to attempt that. But it’s good money spent usually because you’re in a position that you have to use a specific expertise that you don’t have.

Carlos: Yeah.

Kevin: Ok.

Carlos: And nothing else again kind of those soft skills relationships you want to be on speaking terms with someone before you have a need for their services. You’ll want to shop that around or get somebody you’re feel comfortable with rather than somebody that you have to have because you have no other choice or alternative.

Steve: Yes, that’s a very good point.

Kevin: Cool, so let’s talk a little bit about Database Corruption Challenge. Steve, what made you come up with this idea?

Steve: Wow, alright, it was interesting and I think that there is a lot of detail that Carlos actually asked me on this on Episode 12 where I first was on the podcast. It started out initially because I do lot of blogging on SQL Server topics. It started that I wanted to share some of my knowledge about database corruption and fixing it and I started writing a blog post about how to fix corruption by pulling data in from non-clustered indexes to try and figure out what was missing. And I realized that anybody could do that. I mean anybody could write a post like that so I thought, “Well, I change it up a little bit.” I’ll go and actually create a single corrupt database and I’ll put that in the blog post as a training exercise to see if somebody, to see people interested in trying to solve that. That was a Saturday. I think I did that on a Saturday morning and I threw it out. I put it on Twitter and a few things. I said, “Ok, no big deal. Nobody found it interesting.” And about 8 hours later though it got some traffic and that Brent Ozar picked it up and he decided he was going to jump in and solve it, and he solved it pretty darn quick. It think his story was he and his fiancé at that point were trying to head out to dinner when he saw this and he stopped what he was doing and fix the corruption before going to dinner. That might have cause a little bit of trouble, maybe been a little bit for dinner but he was the first to solve the first week of the corruption challenge and then he tweeted about it, and that sort of got the fire going there a little bit around more people being interested in it because I think he has a little bit more of reach on Twitter than I do.

Carlos: He can move the internet numbers that’s for sure.

Steve: Yup. After he solved it then a handful of other people jumped in to solve it and it’s at that point I realized, “Hey, this is really interesting. There is a lot of interest here. I’m going to do another one.” And then I kind of quickly made some rules and said, “Well, I could do this for 10 weeks.” And that was my initial plan, 10 weeks, but it turned out to be 10 competitions over about every 10-14 days not
every single week, and it just kind of grew from there. There were about 60-70 people who actively participated week after week and it just kind of evolved at that point. It wasn’t that I ever like sat down and thought, “Hmm, I’m going to build this Corruption Challenge.” It was just sort of a blog post that evolved and became the Corruption Challenge.

Kevin: Yeah. I remember it being a big deal and it’s still really interesting to go back because those corruption issues they still happen today.

Steve: Yup, oh yeah, and I think today I get a lot of traffic if you go to the stevestedman.com/corruption you can get to all the blog posts that I’ve done as well as all the 10 weeks of the corruption challenge. Check it out there and I get a lot of people that even though it’s been 2 years people are still learning from it, and I think almost everything that I cover in the Corruption Challenge is still valid today even in the latest versions of SQL Server.

Kevin: How much did you learn during that challenge? You started out obviously the first database you knew how to do that. You put the example together. When we got to some of the later databases did you know already all that stuff beforehand or did you have to go research more corruption, reasons for corruption?

Steve: Oh, yeah, I certainly did not know all of that when I started. I knew a lot of it but it’s one thing to know about a type of corruption and it’s a new another level to know enough about it to go be able to create it in the database that can then be backed up and distributed to people to try and fix it themselves. And there was sometimes where I thought, “Ok, well here is something I know what the corruption is but it took me 4-5 hours to go and actually build a test database that had that kind of corruption in it.”

Carlos: Right, and then to make sure that, you know, can I fix this. Is this fixable, right?

Steve: Yup, and then I think that the people who participated actively in the Corruption Challenge were incredible to be able to learn from. And I know that the participants in the first few weeks were very helpful but they were also very critical in a positive helping kind of way if anything that I tried wasn’t quite right. And there was one or two of the weeks that I put out a corrupt database and then somebody pointed out some flaw and then I have to go back and correct it in order to make it so it could actually be fixed someone.

Kevin: So of the solutions that you got, what was the most unexpected and interesting solution?

Steve: The most interesting and unusual one that I came across was Patrick Flynn, and I think he is from New Zealand. And I think it was for week 4 or 5 somewhere around there in the competition. It was one that, it was a particularly nasty corruption scenario but what he did, and one of the reason I loved it because I like CTEs, and I actually wrote a book on Common Table Expressions a while ago but it really use CTEs creatively. It was one that I actually adapted and I use it in my presentation at PASS Summit last year on database corruption. But what he did is using some temp tables and CTEs; he was able to use the DBCC page command to extract all of the data in horrible binary format into temporary tables. And then from there used CTEs to manipulate and extract all the data out of those temporary tables and reconstituted into INSERT statements to rebuild the table from scratch. I mean, if we had an hour I could walk you through the demo how it works. There were a lot of really awesome solutions but that’s the one that just jumps out at me as, wow that one was vastly superior. Not vastly superior, it was the most interesting and the one that I enjoyed working through the most. Part of the process when I did that challenge was, it was a competition people would see who could be the first one to solve it so I would throw the Corruption Challenge out there and then usually after Week 2, within about an hour, I’d start getting people submitting solutions and I would have to go through and confirm that their solution actually worked. And that one probably took me the longest amount of time to understand that it worked because it was so interesting and I just wanted to dive in and totally understand every single thing it was doing. I love that example, that’s my favorite out of all of them.

Kevin: Very nice. Let’s switch gears again, we’re going to talk about a very nice conference. Carlos, why did you pick such a hard name to pronounce for Compa Con?

Carlos: Compa Con. Yes, well I didn’t consult you, number one. And then I guess have you tried finding a URL lately, right?

Kevin: This is true.

Carlos: Ultimately, this is an extension, will be honed to be an extension of the podcast. This idea of bringing people together, talking about SQL Server in different ways, you know, ways that people might be using today or think of ways they haven’t consider with new features. You know, just different ways to attack different problems. Like Patrick’s solution for the corruption challenge, sharing that type of information. And so actually before I launched the podcast I wanted a name for the people who listen to the podcast. Kind of create a sense of community and that idea of companero kind of came to mind. I put a little video of this together out on Twitter or on YouTube rather. So companero is a Spanish word for companion and as a missionary for my church I had a companion and so we were companeros. And this person, we worked together, you know, 24 hours a day and this is for 2-year commitment. And so having good companions along the road just help things goes smoother and so again that was this kind of idea for the podcast of we want to get people together to talk about helping you get from one path to the other. And Steve and I are both actually big scouters which we didn’t find out until kind of after we started talking and so that idea of being on the trail, right? You know, known paths versus unknown paths and if you have a guide just how much simpler that makes everything. And so that’s ultimately where the idea of Companero Conference came from and then we’ve been developing that idea with the hopes that people will com. Right, you get access to folks that maybe you don’t know but we’ve. Now, I hate to use the word vetting, it’s not like, you know.

Kevin: Extreme vetting.

Carlos: Yeah, everyone’s records, IRS, background checks, all that stuff know it. These are people that we feel comfortable inviting I guess is the word to share because they knew they would be willing to share some of their experiences and do so in a way that would be positive for those who came. We hope that people will come, get some short experiences, get some help, would be able to ask questions with things that they haven’t yet face. But also then be able to when I get to a trail or scenario that they haven’t experienced before that they’ll be able to reach out and ask more than just Google.

Kevin: So, Steve, what are you looking forward to with Compa Con?

Steve: The biggest thing I’m looking forward to there is being able to meet more the people that we interact with on the podcast and meet them in person. I mean, and whether it’s the speakers that are going to be there or the attendees as well. I mean, I’m excited about the business venture of course in doing the conference but really what it comes down to is getting to know the people. Yeah, that’s it for me there.

Carlos: Alright, I will say one another thing and that is I remember again being a full time employee and not using my training budget normally because the budget was not high enough to go to some of these other conferences like PASS Summit that required to travel across the country and things. And so we wanted to, it’s like could we create something that people could afford within the budgets that they have and still come to something that’s not somebody opening up a book and you’re getting That’s not helpful, I mean. And so that was another element to that is again through the listeners they were getting value out of the podcast. We thought, “Ok well, what value can they get when we get together and can they leverage some of those budgets in a way that it will get approved, that meets the criteria of a conference and also allows them to expand their network a bit.”

Steve: Another thing to add to that that I’m really excited about too with the conference is the office hours concept. I think that quite often you go to a conference, you sit in a session for an hour or half a day or whatever it may be with the speaker and then when that’s over, it’s over. You go back to work a couple of days later and you try and use some of the things you’ve learned. Whereas with this we’ve nearly end of the conference we have an office hours slot where you will be able to meet with any of the speakers that are there to be able to discuss, or talk, or find out more about the topics that we are covered in their presentation. And I think to me that seems like a lot of fun.

Carlos: Yeah, and because the way the setup is we’re going to sprinkle that in with little bit of hands on learning. So yeah, that will be a slightly different take because I think it will be more authentic. One of the things that we are trying to do, I hate to use the word “can”, and we’ll have some scenarios where people can walkthrough individually. But we are hoping that most of this growth is kind of organic in the sense of, “Hey, you know what, Kevin, like I know you are talking about security I’d like you to show me this security thing. Can you walkthrough with this with me?” And then people just start talking, conversations in sew and you’re getting, “Yeah, let’s take a look at that. Here is how you do this.” So still kind of “hands-on” but it’s organic.

Kevin: So the conference itself will be October 4th and 5th in Norfolk, Virginia. I hear there is something involved with a boat?

Carlos: Yes, we’re going to have an evening cruise, so down there and all of a sudden I can’t remember the name of the river but we are very close to the Chesapeake Bay. One of the rivers that shoots off of the bay and of course Norfolk is a big naval yard and there is lots of traffic in that area so it will be very pleasant and it will be in the evening, the sun will be going down so will get to go out two hours out on the boat. We will actually eat dinner there as well and have a little bit of fun. There will be a top deck open air, you can go out and just hang out, again have some conversation or there will be dancing. So there’s three levels, in the second level we will have food and dancing and the third level is just kind of relaxing, you know, enjoy the weather.

Steve: And you are welcome to come along even if you don’t want to be part of the dancing.

Carlos: Yes, that’s right. We want to be very introvert friendly and so while we can’t get that third section just to ourselves. If it’s everyone’s intention we can definitely go over and push everybody outside.

Kevin: I’m claiming the nice spot against the wall. So sounds it’s going to be a blast. How about we talk about SQL Family now?

Carlos: Let’s do it.

Kevin: Ok, so how did you first get started with SQL Server? I’m going to start with Carlos for this one.

Carlos: I think I have the atypical answer, the accidental DBA kind of fits, so I want to be in networking. Networking is what I wanted to do. I did an internship for Cisco Systems. The company that I’d work for was purchased by Cisco Systems and so I wanted to do networking. That’s what I wanted to do. I went to college, I wanted to get my CCNA, all that stuff. My first job was working for a small consulting firm both kind of doing their internal IT. So it was 15 consultants, so I was doing things like email, networking, security and then setting up environments for the consultants so they could test things and whatnot, and SQL Server kind of came along with that as they were doing some of the applications. One of the consultants leaves and goes to work for the State and he calls me a couple of months later and he’s like, “Hey, they have this database administrator position. I think you should apply.” And again I’m harking back to my college days so I took two database courses. I hated both of them. It was adjunct faculty, felt very and I was like, “No way. Like, you’re crazy, right?” And he call me back and he’s like, “Hey, we are having a hard time filling the slot like I think you should consider.” I was like, “I don’t even know how to be a DBA. Like, I don’t really know anything about it.” And he’s like, “Well, this is what they pay.”
And I was like, “Oh, interesting.” Again I was at a job right out of college. I graduated in 2002 right in the end of the .com bubble so I felt fortunate actually to have a job at entry level. And so I said, well you know what. It was a significant jump from where I was. And I said, “Ok, I’ll do it.” They had SQL Server in Oracle there so they had an Oracle DBA and I applied and got the job and so basically went to the Oracle DBA and say, “Hey, how do you this?, and he showed me. And then I have to go figure out how to do it in SQL Server. That’s kind of how that started.

Kevin: Interesting, so how about you, Steve?

Steve: Well, just to echo one of the same thing as Carlos said with databases and classes in college. I had a couple of databases classes in college and I hated them. I could not stand database work the way that it was taught in the university at that point in time. But while I was in college I ended up getting a 9-month long internship working at Microsoft and this was in 1990 when Windows 3.0 had just released and just to set the timeframe there. And everyone they get hired was like in computer science and all from the local universities. They were brought in to work in tech support for Windows 3.0 right after it was released. And I learned a lot there but I didn’t want to work in tech support, and I wanted to be a programmer. And so I did everything I could to try and move from that position and I ended up taking on or working with a couple of other people in an internal project to go create some tools that were needed for the tech support team to use there. And lo and behold there was this database thing that Microsoft had just started selling that they suggest that we use and I never heard of it. And I said, “Well, what we need to do to get you speed on this is send you to Microsoft University which was an internal training course they had then. And for a week long class on how to use this thing called transact SQL. So on December 12th of 1990, I received a certificate that said, I’m qualified to use T_SQL.

Kevin: For the record, I do not have that certificate. I got qualified.

Steve: Yes, and so that’s sort of an Easter egg that I put on my blog. My parents found this in their house, this certificate like 20+ years later, and they gave it to me a couple of years ago and I scanned it in and I put a copy of that on my blog as a blog entry from 1990 even though blogs didn’t exist in 1990. Alright, if you check out stevestedman.com, you can scroll back in time and find that there if you’re looking for something that’s maybe a bit funny to look at. But anyway, so that was a 9-month long gig at Microsoft and then I went back to school and I went to do another internship and back to school and on the jobs and all that. And it seem like every job that I ended up at I ended up needing to do something with SQL Server. And then it just sort of evolved into more and more database work and I finally realized I didn’t want to be a programmer; I wanted to do database side of things. I mean, I still do programming but it is all database related programming now, and it just evolved into the DBA role and I had other jobs along the way like I ended up as a CTO at one point and I realized I don’t really like that as much. I want to go back and do more database work. Started all at Microsoft in 1990 and it just kind of evolved from there.

Kevin: Interesting. So sticking with you, Steve, if you could change one thing about SQL
Server what would it be?

Steve: The way that the check_db command works. Meaning, when it runs it goes out and scans your entire database to make sure that there is no integrity issues there, no corruption. And the problem is a lot of people don’t run that because it takes too long to run. And if there was way to say what I want to do is I want to run check_db but run it for an hour and check as much as you can possibly check and then keep track if that and then tomorrow night I’m going to kick it off for another one hour and continue that check process. That would really be a cool change that would probably help a lot with the way that people do their database checks. I know there’s ways to sort of simulate that by saying I’m going to check some of the tables but if you get to the point where you got a database with just one gigantic table, a way to run it for certain amount of time and then pick up later would be pretty awesome.

Kevin: Makes sense. Carlos, if you could change one thing about SQL Server what would it be and why would it be expanding in PolyBase?

Carlos: Yeah, you took the words out of my mouth there, Kevin. Yeah, you know, it’s funny so I was thinking a little bit about this so I went and answer some of this in Episode 0. But we’ve changed the SQL Family questions since then so this is not something that I guess I’ve had to address and of course I think one of the big things we’ve talked about, so SQL Server setup, even some Episode 98, right the first things you change, lots of things in there. As I was thinking about this, Steve and I were talking, so I’m not a user of it yet but I guess it makes me nervous so I guess I’m not sure there is quite a change yet but something that I hope that they do and that is with the introduction of services or languages like R and PolyBase and who knows what’s coming that they give me the administrator. The knuckle dragging Neanderthal that I am who is not a great programmer, you know, trying to not drown in PowerShell. Give me good tools so that I can understand and be able to react to when other people are using those languages in the database. I realized that’s kind of a tall order but help me help other people because I’m a bit nervous about some of that adoption as it increases.

Kevin: Ok, so sticking with you. What is the best piece of career advice that you have received?

Carlos: I’m not sure if it’s the best but the one that often comes back to and that is, “The money will come.” So when I graduated in 2002, that first job I was making roughly 25% lower than I thought I would be making coming out of college. I was a bit frustrated, right? Even when I moved after a couple of years, in fact that job I took as a Database Administrator position, they actually lowered the pay grade because they couldn’t increase my salary by a certain percentage to fill this thing and so. Anyway, I felt like I was, you know, that initial job my wages were lower than I wanted to be and I was expressing some frustration and the comment was, “The money will come.” If you’ll do the best that you can, invest in, kind of harkening back to our. Well, in an episode that hasn’t been released yet, so Episode 104 we’re going to talk with Eugene that idea of going deep. So go deep and get to be good at something. Get to be good at solving a problem becoming that go to person in your organization for certain problems and building trust and then good things will happen. You know, I’m not a millionaire, there is a limit there. However, we were talking about family I have 5 children. My oldest just turned 15 and my youngest is 2 so some of this risk and some of these other things I have to consider them as well. But as you continue to plot along, as you continue to keep your eye on the ball, whatever cliché you want to use there, then good things will happen and I think that has probably been the best piece of career advice there.

Kevin: Got you, so how about you, Steve? What is the best career advice that you have ever received?

Steve: I think the best advice and it kind of comes down to two but the first one is, “There is no such thing as can’t.” When somebody tells you that they can’t do something or that you can’t do something because technically it can’t be done or whatever that’s just an excuse to go and figure out how to do it. Now, maybe there is an exception to that if there is like personnel rules or things like that and
say you can’t do this things, yeah you should follow those. But when it comes to technology when people tell you that something can’t be done, I’ve always looked at it as a challenge to figure out, “Ok, well how can I do that?” The other career advice I think comes from Yoda, from the original, one of the earlier Star Wars movies and it talked about, “There is no try, there’s only do.” I don’t like to try things. I mean, I’ll try a new flavor of ice cream or I’ll try something new on the menu but I like to do things. And to say that you’re going to try something to me often times, like I’ll try and do that for you, I’ll try and get that job done whatever it may be. It’s kind of an excuse to say, “Well, I tried but I can’t do it, so that leads back to the can’t. No such thing as can’t and there is no such thing as try there is only do.

Kevin: No can’t and no try. Alright, so Steve, if you could have one superhero power what would it be and why?

Steve: Oh gosh! I answered this on Episode 12 and I don’t remember what my answer was but I’m going to go with time travel on this one. Because I think if you could go back in time, I don’t think I’d be interested in going forward in time necessarily but if you could back in time, I guess it would have to come forward to get back to where I am. But if you could back in time and learn from mistakes that have been made either by yourself, or from others, or even hundreds of years ago mistakes that have been made just to experience and see what people have done, I think would be an amazing superhero power.

Kevin: Carlos, can you top that?

Carlos: Yeah, top that. Well, I don’t know that I can top it but mind would definitely be different. So in Episode 70, Travis Wright, he kind of brought up this. He said like, “You know, everybody always talks about kind of the supernatural but like some ability that they would possess.” And he said, “Well, I would think the ability to control technology would be very powerful because then you could get it to do all the stuff.” And you wouldn’t have to worry about flaming suits, or hitting somebody when you go back in time. Your matter dimensional is smashing together whatever. And so I think that as I thought about that so that superhero power of being able to, and I always set a camera with a movie. I want to say it’s terminator but it doesn’t seem right. I think they’re actually putting him in the car and get the ATM to spit out. But I feel like there is some movie out there that they walk up and they get the ATM to just start spitting out money. And so something like that although obviously I would do everything ethical, right, nothing immoral like that.

Kevin: Especially on the record.

Carlos: Especially on the record, that’s right. I think that would be my because then also if I could control technology, I don’t know getting some big drone or something like because previously it was flying. I figure that I can get a technology to zoom me around the place pretty quickly.

Kevin: That’s fair, so thank you very much for coming over to the podcast tonight, Steve Stedman and Carlos “skynet” Chacon.

Steve: And thank you for hosting. This has been great.

Carlos: Yes, Kevin, this has been fun. Thanks for having us!

Kevin: Alright, thanks everybody! Take care now. So that was Carlos and Steve today. It was pleasure having them on and hopefully you enjoyed. If you want to see more go to sqldatapartners.com/100 and please follow us on social media, so we’re @sqldatapartners on Twitter, /sqldatapartners on Facebook, also on LinkedIn. Review us on your favorite podcast platform like iTunes or Stitcher and we’ll see you on the SQL trail.