Have you heard the buzz around the Cortana Intelligence Suite? It seems like it is all the rage these days and Microsoft is coming out with lots of new features in this space. In Episode 62, Steve and I interview Melissa Coates, AKA “SQLChick”. We chat about the history of Cortana, about the tools it encompasses, and why you might implement parts of the suite in your organization.
Compañeros, listen to learn…
- Which tools are officially part of the Cortana Intelligence Suite (there are almost a dozen!)
- The history of Cortana, from Halo all the way to the digital assistant
- The tools you need to get started
- How to use the Cortana Gallery to jump-start your data project
- Why predictive analytics is a hot trend that’s only going to grow
- The skills you should cultivate if you want to implement Cortana
- If Cortana will replace traditional data warehousing and best practices
Episode 62 Quote:
“So a DBA that’s fluent in scripting will likely do really well with a service like Azure Data Factory, which is JSON-based. Even for deployments from say dev through production using the ARM or the Azure Resource Manager templates, which are also JSON-based, will really probably benefit from a DBA-type being involved. ” – Melissa Coates on the people skills needed for Azure Intelligence Suite
Melissa Coates is a Solution Architect with BlueGranite, a consulting company that specializes in several areas including data warehousing, big data, enterprise reporting, business intelligence and advanced analytics. She’s also a regular PASS speaker on topics like PowerBI and Cortana Intelligence Suite. and On her blog, Melissa writes about all things data intelligence. Connect with her on Twitter or LinkedIn.
Transcription: Ask me anything--Cortana Intelligence Suite
*Untranscribed introductory portion*
Carlos: So Melissa, welcome to the show!
Melissa: Thank you very much.
Carlos: Yeah, it’s great to have you. Another east coast person. We kind of have East Coast – West Coast as far as who can we get on the show. I think our east coasts are leading by just a hair. But ultimately our conversations today are going to revolve around the Cortana intelligence Suite. And of course, those of us who are running Windows 10, when we click down there and it says “Ask Me Anything”, are we using the Cortana Intelligence Suite or is it more than that?
Melissa: Excellent question. So the Cortana Intelligence Suite is actually a huge collection of services in Azure for the purposes of providing big data and analytical solutions. So the suite consists of: azure data factory, data catalog, azure SQL data warehouse, azure data lake which is actually a composite of three services, azure machine learning, stream analytics, event hub, PowerBI, cognitive services, the bot framework, and finally the Cortana digital assistant like you just mentioned. So although they named the suite after Cortana, the digital assistant is just one small part. She originated as a character in Halo, as a smart artificial intelligence character that can learn and adapt. And then she was the inspiration for the digital assistant in Windows. And now this suite of tools is named after her because she symbolizes the contextualized intelligence they hope to achieve with the suite of tools.
Melissa: So having said all that, there will absolutely be other Azure services as part of an overall solution which are not officially considered part of the Cortana Intelligence Suite umbrella and we absolutely expect that things like Azure SQL Database, Blob Storage, virtual machines, and so forth are really commonly used as well.
Carlos: That sounds like a super big umbrella, right? You need a lot of services and components in there and you know, we’ve had episodes on some of those different pieces and I feel like, oh my gosh. Big data, you know, that expansion of functionality and services and technology. And I feel like Cortana is just blown that up even bigger. How do people even get started with that and even decide, “Yes, this is something I could be using?”
Melissa: The idea behind it is building intelligence and automated solutions. For example, a company is interested in building a fraud detection system. Something like that would usually encompass several of those services within azure. Um, and then another common thing that we’re seeing an awful lot nowadays is predictive analytics. So predicting credit risk and customer retention and hospital readmissions, when equipment will require maintenance. The opportunities are endless. So in terms of getting started the ultimate goal is for there to be preconfigured solutions and templates that minimize the need from developing from the ground up every single time. Now they’re in the early stages of that, but there are a few to be found in the Cortana Intelligence Suite Gallery.
Carlos: And so tell me, the gallery is just that frame work of “here’s those bits and components that you can take off the shelf and start using?
Steve: So to get started with those bits and components off the shelf there, I imagine that you can pick and choose from the features you described there. You don’t have to be using all of them in order to be working in Cortana, is that right?
Melissa: Correct, correct. For instance, there is a particular solution for equipment maintenance. So that might include something like stream analytics to pick up the data from the telemetry from the machine and send it to some sort of predictive database. Also send it to PowerBI machine learning so it can turn around and do predictions form the health of that machine and when it may need maintenance the next time. It definitely doesn’t mean that everything has to be used. The gallery evolved from an azure machine learning gallery, so that’s most of what’s out there right now. But longer-term, the vision is for that to be built up more so than where it is currently.
Steve: So then, is it the idea behind the gallery is that those are the pieces you’d start with and then modify them for your environment? Or is it example pieces like the AdventureWorks database?
Melissa: Good question. So both. And so what you can do is deploy some of these assets to your own azure subscription and then obviously you’d have to do some tweaks for you own credentials etc etc. So yes, the goal is to give you some pieces to get started and accelerator type of thing.
Steve: Okay, great.
Carlos: So then I guess take us through a scenario where, I guess you talked about fraud, right? But as you try and go and use some of this, one is identifying… my question is: organizations would start using Cortana because one, they have a scenario. Two, how do they goa bout putting together those pieces to make it all work. I mean, people are involved and yo mentioned some of the technologies. I guess help me put together that recipe a bit more
Melissa: I think you just asked about tools and people. You mentioned tools first, and there’s a lot of them. There’s a post on my blog at sqlchick.com that talks about setting up a workstation in Cortana Intelligence suite development. So in a nutshell, depending on the services you’re going to use. You basically want management studio, vs with some important extension, SQL server data tools. There’s an azure sdk which Is important for some of the services. There are some tools that you might not install on day one that you’ll probably want at some point. Things like azure storage explorer or azcopy or azure PowerShell. And then there’s also an azure feature pack for integration services if you need it. Lots and lots of different tools for client side in addition to somethings over on the portal side as well
Carlos: So Compañeros, we’ll have that link in addition to some of the other articles on her site at SQLDataParnters.com/Cortana and there you can go and click the link and it sounds like there are quite a few things you need to get set up right.
Melissa: Yep, yep. And in term of the people side of things, the way I see it is that since it’s such a wide variety of services and so forth, that there are numerous skill sets that are really going to be needed because sometimes these solutions get big and complex. So a DBA that’s fluent in scripting will likely do really well with a service like Azure Data Factory, which is JSON-based. Even for deployments from say dev through production using the ARM or the Azure Resource Manager templates, which are also JSON-based, will really probably benefit from a DBA-type being involved. Conversely, a DBA traditionally isn’t likely to be as familiar with algorithms in the machine learning space. So the data scientist or statistician type of person usually best qualified there. And then we’ve got a piece such as Azure SQL Data Warehouse, which is based on SQL Server but is on that MPP, massively parallel architecture, which means that there’s really some important distinctions to be aware of which kind of justifies a deep data warehousing specialist. And then there’s’ the portion of the service that rely on custom coded solutions. So definitely a team effort since it’s so broad.
Carlos: No doubt. So you’re a Solutions Architect with Blue Granite. And it sounds like, if an organization is going to implement one of these, kind of hold on to your seats, because there’s going to be a lot of people involved and it’s going to be a fairly complex. Is that fair?
Melissa: It is. It is. And I know that there is definitely the paradigm of less administration in the cloud. However, when we’re moving data and we’re introducing a lot of services and they’re integrating with each other and we’ve got different security layers. You know, it just does get complex. That’s the way it is.
Carlos: So then, I guess if someone wanted to get into this- and there are a lot of different components – if someone has dabbled in the Azure Data Factory or maybe the Azure Data Lake components what would they need to be looking at to say, “okay, I’m interested in being the solutions architect or being more involved in this analytics and see if I can use this.” What skills sets should they be looking to get better at?
Melissa: That’s a good question. And so, usually when someone asks me something like this I say something along the lines of, “What would make your normal day job better?” for example, putting analytics on top of data you’re already gathering. So figuring out which use case would make the most sense for you twofold. So you’re learning and you’re actually sort of helping yourself in your primary job as well. As far as going through the list, figuring out which particular one falls into that category. In terms of a really obvious place to start, the easiest and fastest entry point would be PowerBI. Because a lot of the other systems are kind of a bigger endeavor in some cases.
Steve: Okay so let’s say you’ve done that and you’ve tried PowerBI and you have some data out there in an Azure Data Warehouse. Then what really makes this Cortana versus just being able t pull that data out of your data warehouse in PowerBI?
Fair enough and you’ve got to remember that Cortana intelligence suite is really a marketing term for the collection of these services.
Melissa: Yeah, yeah. So, whether or not you use services that are technically under the umbrella or not kind of doesn’t matter to me as long as it’s the right tool for the job.
Carlos: I feel like we’ve been duped a little bit, Steve.
Steve: No, by the marketing teams, right?
Carlos: Right, by the marketing teams. Cortana intelligence suite, it’s really just all these products that we’ve looped together than congratulations! You’re in the cloud.
Melissa: Well, now, in all fairness let’s say that you really do want to integrate with Cortana on your windows 10 machine. In PowerBI, if you’ve seen this thing called QNA, or Natural Language Querying, where you type into a text box “sales by quarter by division” and it renders a visual on the fly, right? The next step to that is you talk to your laptop and you say, “Hey Cortana! Show me sales by quarter…” and basically if you’ve got your machine set up, you’ll get your PowerBI visual on the fly. That’s one of the first integrations we’re seeing through Cortana that’s a more auditory way of delivering it.
Carlos: So ultimately, Cortana will be the engine to interface and ask these questions through to get to the back end.
Melissa: I don’t know the answer to that, but that would be a really cool goal to shoot for.
Carlos: Interesting. So where are we in the spectrum of where this lives? Again, you just talked about some of the facets of using a couple features here and a couple of features there. We know with azure things change pretty rapidly. For example in episode 61, we had AZ on and we were talking about U-SQL, which is in this family umbrella as well. That’s still in preview, it came out at the beginning of the year. I guess where do you kind of get a feel for, I’ll just say, the stability of the environments and the functionality supported in the umbrella?
Melissa: Yep, yep. So you’re right, some of the services are still in preview like the two azure data lake services. Although it’s third piece of azure data lake is HD insight and it’s actually a pretty mature product now. Cognitive services and the bot framework are in previews as well. So overall a lot of the services are still pretty young and integration between the services is still evolving as well. That definitely represents a little bit of a hurdle. And as we kind of alluded to earlier, the prebuilt solutions and those templates there are all still evolving as well. And then, one thing that I think we BI folks are really feeling is that there’s a lot of new design patterns associated with these tools and these cloud services, so best practices are still emerging. So, a really good example of this is Azure Data Factory. It’s an Azure Data orchestration service and it is absolutely not SSIS for the cloud. Because it’s so fundamentally different, the patterns that we follow and the best practices we follow are not the same and those are all still evolving and that’s something that we’re kind of learning as we go.
Carlos: Sure. And I think even that whole idea of streaming analytics in that space does continue to evolve. I think form the traditional BI environment, so the Kimball methodology and cubes and things like that. That is kind of being turned and the processing power that’s available via the cloud, it has a different way of slicing and dicing all of that.
Melissa: True, true. I still believe with the data lakes in play as an overall data strategy. I believe that there’s a place for a data warehouse. That predictable reporting, that organizational reporting, and the people that really just need to run parameterized reports and that kind of stuff. I still think the Kimball methodology is a sound framework for people to learn and so forth. But you’re right, we don’t necessarily put as much data in the data warehouse or in the semantic layer or in cubes anymore as we kind of used to do in the past by default basically.
Steve: So then when we talk about features, you mentioned that there’s a bunch of them in preview and a few that are more stable or more robust. Are there any features really missing at this point that if people are wanting to try this out that they should wait for?
Melissa: So, there’s a number of things that as you go down this road, that you would find. So one of my big suggestions would be do a proof of concept or a small project to prove things out. For instance, one thing that comes to mind is something that we discovered a few weeks ago on a client project. So Azure Active Directory integration with azure SQL data warehouse and Azure SQL Database has gone GA and is now supported for integrated AUTH. Well –
Carlos: I feel like there’s a but here…
Melissa: What we’ve learned is that this support only extends to SQL Server Data tools and Management Studio. So what we had wanted to do is use PowerBI as a front end reporting tool in direct query mode to that SQL Data Warehouse or SQL Database using Integrated Authentication and it’s ust not supported yet. And that’s the sort of thing you want to learn early on. So small POCs when things are young and growing and moving fast.
Steve: I think that proof of concept idea sounds like a really good way to start. Jump in, do something small, make sure all the steps work and we’ll at least know if it’s going to do what we’re looking for.
Melissa: Yep, absolutely. So I’m on a project right now where we’re basically laying the foundation for a data lake strategy and it’s a data lake store. So we learned that oh yeah, since it’s in preview, it’s not HIPAA compliant yet. So any HIPAA data we need to make sure it goes over to the BLOB store for that, you know, all those sorts of things while it’s maturing and growing.
Carlos: Yeah, wow, that would be a lot to keep up with, you know, because certain features can only work with certain data sets… then all of a sudden you throw HIPAA compliance on there and it gets complicated really fast.
Melissa: Yep, absolutely. They did do a good job in the Azure Trust Center of you know, documenting what’s compliant with what by each service. So we have a reference. But you’re right, there’s a lot ot keep up with these days. It makes me crazy, but in a fun and good way.
Carlos: Well there you go. So I guess, what kinds of questions have you found most interesting about getting into the Cortana Intelligence Suite and what have you enjoyed most about working in that area?
Melissa: Yeah, so I kind of live to learn new things all the time. That’s kind of what makes me tick. And so it’s kind of opened up this big wide door of that and being a BI developer background, I don’t necessarily have a deep background in installation and tuning and that kind of thing for the on-prem SQL Server world. So for someone like me, it’s almost a way to hit the reset button and say “Okay, I can learn this cloud stuff kind of from the ground up,”. And granted a whole of our on-prem knowledge is useful, especially with our platform-as-a-service, some things give me a leg up there.
Carlos: No doubt, very cool.
Steve: So then if you’re going to jump into this, do you really need a whole IT team behind you to really jump into this? Or is it something really anybody can jump into to start working with Cortana?
Melissa: Yeah, yeah I think it depends on how far you want to take things. So for instance, a current customer of mine wanted to do three different subscriptions for Dev, Test, and Prod rather than segregating by resource groups. Things like, the ARM templates and deployments between environments. Things like managing firewalls and security credentials. Even scaling up and down and just the logging and auditing and that sort of thing. And granted, a lot of that stuff is much, much easier in Azure, but it’s not normally the stuff that your business user wants to pay a lot of attention to.
Melissa: So I think that yes, IT is still a relevant role.
Steve: Yep. Okay.
Carlos: So I guess last thoughts about Cortana Intelligence Suite.
Melissa: Hmm. I got nothing.
Steve: Okay, well there you go.
Carlos: Should we do SQL Family?
Steve: Yes, let’s do SQL Family. So with technology changing so quickly, how do you keep up with all the changes?
Melissa: It is really hard with the pace of change these days. And I’m a generalist, not so much a specialist, which makes it even harder. So although some people kind of think it’s old school, I do still have my RSS feed of bloggers that I follow.
Carlos: Oh wow.
Melissa: Hey! I’m also on Twitter but it’s hit or miss. I’m not a great multitasker so I don’t usually have it up during the workday so I use Twitter more for useful links and information. And then I also use the “Pocket” app so if I see a link somewhere on Twitter I’ll send it to Pocket to read later which sometimes becomes a bit of a black hole. I have good intentions when I file it away.
Carlos: If you could change one thing about SQL Server, and we’ll extend that into the BI space of course, what would it be?
Melissa: I forgot to tell you to nix this question. I couldn’t think of anything really good, sorry.
Carlos: Okay! You’re just so happy with it.
Melissa: Well, I couldn’t think of anything like super awesome. Like absolutely wonderful suggestion.
Carlos: So there’s no change, like, you know what I really wish it would do this. I want it to be the color purple; I want my database to be mauve.
Melissa: Hmmm. Nope, not that one.
Steve: Alright so let’s head into the next one. What is the best piece of career advice that you’ve received?
Melissa: Someone said to me years ago, “You’ll never be caught up.” And not to let it make you crazy. So just how to mentally deal with it because there’s always too much to do and not enough time to do it. And that’s the thing that I guess I’ve remembered that conversation many times over the years.
Carlos: So does that mean that it helps you specialize in a certain, focus on a certain thing and don’t’ worry about the periphery? Or how do you go about taking that advice to heart I guess?
Melissa: Personally, I need to make lists. If things are floating around in my head that does kind of make me crazy. But if I have a list and I know what’s the top priority, then I’m okay. I can have lots of things to do and not feel the need to work all weekend necessarily because okay, I’m organized and I know what’s coming next. I want to say, and I don’t remember this for sure, but I want to say he found me in the office on a Sunday when he told me this. Although now the question is, why was he in the office on a Sunday?
Carlos: Mysteries, mysteries.
Steve: Very good point.
Carlos: So Melissa, our last question for you today. If you could have one superhero power, what would it be and why would you want it?
Melissa: I think I would like the ability to selectively see the future, only the things I want to know of course.
Carlos: Interesting. How would you go about deciding what you’d want to know?
Melissa: Hmm. Good question. I don’t know.
Steve: That might just start with lottery tickets and go from there.
Carlos: You can figure the rest out at that point, right? Use Cortana for the rest.
Carlos: Melissa, thanks so much for being on the show today.
Melissa: Thanks for having me, that was fun guys.