Canvas Breach Breakdown: What 9,000+ Outages Teach Us About SaaS Risk

When the Canvas LMS went down, thousands of institutions came to a halt, right in the middle of finals. In this episode, we break down what really happened, what data may have been exposed, and why this incident is a wake-up call for every organization relying on SaaS platforms.

From vendor risk and contract blind spots to business continuity failures, we unpack the real lessons security leaders need to hear, and what you should be doing right now to prepare for the next breach.

Transcript

SPEAKER_02 0:00

And I had a customer that reached out, one of our customers reached out to me with a screenshot saying, um, hey, we had a user navigate to this training website that we use, and there's this shiny hunters, and we all take a screenshot and send it to us. And they were super concerned. And this wasn't even a higher ed. Yeah. This was an organization that just utilized it for some internal trainings. Yeah. Yeah. And um that so that spawned like a whole like uh what's going on? And it's like, hey, let's this is a third party, you guys are okay, this is what this means, kind of thing. But it's kind of interesting that they had no idea, and this had been already, you know, in the news for a while.

SPEAKER_00 0:42

You are now listening to the Secure AF Podcast.

SPEAKER_02 0:51

Welcome to another episode of the Secure AF Podcast. Uh, I will be your host today, Andrew, uh security engineering lead here at Alias Cybersecurity. And joining me today is my colleague and mentor, uh, Jonathan Kemet. How are you doing, sir? Mentor. I like this. Yeah, I was I would call you mentor for sure. He's our CISO here at Alias Cybersecurity.

SPEAKER_01 1:11

Uh I only teach good things sometimes.

SPEAKER_02 1:14

Usually, yeah. Um, and today we're going to be talking about a subject that is all over uh cybersecurity headlines today for a good reason. And that is the Canvas uh platform. Would you would you call it a data breach? Would we call it a breach at this point?

SPEAKER_01 1:32

I I would say yeah, it's a breach. Incidentally, in most cases, uh the breach would require notification, and this is going to require notification. Yep.

SPEAKER_02 1:42

Yeah, for sure. That's fair. Um, and this uh breach incident, breach, um, it affected a lot of organizations. Yes, a lot of organizations. So uh it's just to start out, uh, for listeners who may not be familiar with what exactly the Canvas platform is, um, who tip who typically uses so Canvas is a learning management system.

SPEAKER_01 2:09

Um, so when you imagine someone going to school and their instructors are putting up content for them up online that they can go in and read or watch or submit uh papers or submit stuff, uh that's through the learning management system. And and generally speaking, there's three or four large learning management systems throughout all of education. Uh, Canvas is one of them. It's a company called Infrastructure and no Instructure. Um, I always get that messed up. I always say Canvas, that's how everyone calls it. Uh, but the company is called Instructure. And uh it was it's it's very well used. I mean, and it's not just uh higher ed or K through 12, a lot of organizations use the same tool within their own training programs, right? Uh, and using it as a learning management system. That's what it's designed for. Um, when you go on their website, you're gonna see some really big names at the bottom. There's a lot of people who use it.

SPEAKER_02 3:09

Yeah, yeah. And I mean, my mother-in-law, who's a librarian at elementary school here, you know, we were over at their place yesterday and she said, Okay, what happened with Canvas? Uh so I mean, a lot of organizations, uh higher education, elementary, like they were down.

SPEAKER_01 3:28

Yeah. It was hard down. It was hard down for a little while. It's we've been having issues with it for about a week all last week. Uh, but they took it completely offline Thursday afternoon and it came back up in the middle of the night.

SPEAKER_02 3:42

Right. So, how critical is Canvas for day-to-day operations for those organizations like universities, enterprises, or just training programs?

SPEAKER_01 3:52

Um, well, for universities in some K through 12s, they were hard down because um one of the things that happened during COVID is school systems started pushing all their classes online. And at that time, they chose the different learning management systems that they did. Some did Blackboard, some did D2L, some did Canvas, you know, whatever it was. And so they they, in fact, I know some schools had a specific mandate that every one of their classes would have representation in the learning management system. Well, what that turned into is that is how the class is run. The that is how the students get in there, that's how the faculty get in there and get their student roster, that's how students uh get their content, that's how they submit their content. It is really that's how where their grades come from, their grades get downloaded from their the the LMS into their student information system. I mean, it's critical. Right. Um, the the afternoon it was down, the hard down. Uh there was universities that literally just shut down classes. They didn't, they couldn't submit anything, they couldn't do anything. Now, one of the problems for this particular incident and this particular timing is it was finals. You know, it was the last couple of weeks of school. This hit them hard. They were trying to get those last grades in, and it was it was very impactful.

SPEAKER_02 5:16

Right. And that kind of launches me into my last question here uh on the setting the stage portion. Um, from a CISO lens, how would you classify Canvas, like mission critical, sensitive, or both?

SPEAKER_01 5:30

Oh, I I absolutely call it critical infrastructure. Critical infrastructure for the organizations that are using it to that degree, it would be considered uh critical infrastructure, right? Critical systems. Uh, because the primary role of education, you know, K through 12 or higher ed is to provide academic services to teach students. And if their their only way to teach is through that learning management system and it's down, that's a problem. Now, when I was going to school, we did have a version of learning management. Now, this was a very long time ago. I got lots of gray. We had a version of learning management, but we only had a handful of classes in it. Right. When I was at the university, when I was a CISO there, we probably had a half to three quarters of our classes in it. But again, after COVID, because of the what happened at COVID, everyone had to have an online presence. Well, learning management systems was that online presence, so they just moved everything into it. So now, I mean, there are institutions that don't have any class that is not in the LMS to some degree, and most of their classes are entirely in the LMS. That means their content, they don't they don't even have PowerPoint presentations anymore, like outside of the the the LMS. So it's not like you're going to a class and they bring a computer up and a PowerPoint. It's they log in and all that content is there. So it is absolutely critical for those institutions.

SPEAKER_02 6:58

Yeah, and that's another perfect transition into talking about the dependency and like risk surface here. Oh, yeah. So um you mentioned, you know, during COVID, there was a huge shift um just trying to survive in most cases. Yep. Um, and that gets into the question of how heavily are these organizations relying on on SaaS platforms like Canvas compared to on-prem systems today.

SPEAKER_01 7:25

It's all SaaS. I mean, most of the organizations are moving to they either have moved uh because of COVID andor other things, or they are moving. Um, and you know, when we think about higher eds or even K through 12s, and what we're talking about is either Microsoft or Google. Yeah um they're running an LMS of some kind, their um their ERP and student information system is probably SaaS. Some institutions still have on-prem things, but a lot of the organ or a lot of the providers are saying, we're not gonna support that in the next two years. You have to move off to our SaaS platform, which makes sense from that vendor perspective because they don't want to keep maintaining old products, old on-prem things. So they're just saying, okay, we're not gonna do any more updates to it. You know, you don't have to move, but you're not gonna be using the service anymore. So I would say most organizations probably have 70 to 90 percent in some version of a SaaS. Right. Um for those that aren't, they're getting there. I mean, they're they're they're working to that. I would say in five years, uh probably 90 to 95% is all gonna be SaaS products.

SPEAKER_02 8:43

So what risks come with centralizing so much operational and user data into a single LMS platform?

SPEAKER_01 8:53

Well, the risks there's a couple different risks. Number one, you have no control over it. Right.

SPEAKER_02 8:58

You're outsourcing that security.

SPEAKER_01 8:59

Yeah, you you're everything is on that vendor. So you've got to have the right vendor management program, the right contracts, the right check-ins, all those different things. But then you're just praying that the vendor does good stuff. Right. I mean, we saw in this situation, you know, we've had 9,000 institutions go down, or somewhere around that 9,000. There is nothing they could have done. There is absolutely, and I've tried to get this across to the institutions, there was nothing they could have done any different, unless we talk about business continuity. That's a little different. But in terms of the the attack, the threat, the threat profile, the vulnerabilities, that was all on the vendor. The vendor is the one that manages it. So when you're looking at risk, that is high impact, hopefully low probability. So when you're doing your risk matrix, the impact versus probability, um, it's high impact for sure, hopefully low probability, uh, but it's not zero. And in this particular case, it was very clearly the vendor had a problem, it had a huge impact on the institution. Now, for higher eds, there's only a few things that really have a negative impact short term. You can affect higher ed long term, but let's say you had the internet go down, you know, at the institution, probably a couple of days before it actually caused harm of some uh monetary harm or something else. Um, but that's not the way with LMS when all of your teaching capability is in that tool and it goes down, even just for us in our tool, that is your entire business. So that one is huge, huge impact. So you do a couple things. I mean, you have to evaluate that risk just like that. What is the impact? Huge. What is the probability? Probably pretty small, but not zero. So you have to put in your business continuity in to make sure that you take that into account. Hopefully, it never happens, but we saw here it did. So, and and I think there's a lot of work, and this is as far as I can remember, this is the first time that we've had a major LMS outage like this. I think so. Um, and it no one was expecting it. Yeah.

SPEAKER_02 11:23

So, in your experience, are organizations generally overconfident in these SaaS providers' security controls?

SPEAKER_01 11:33

That's a good question. I don't think they're overcompetent. I think they just don't know. They don't know. I mean, I you know, I I work with organizations all the time in their vendor management protocols, and they just they don't ask questions, they don't have an MSA in there as it relates to uptime. You know, you need to have an SLA, in some cases, OLAs, you've got to have um, you know, what is it that they're doing in terms of their implementation of security controls? What are they expecting risks to be? Now, you can't prepare for everything. I mean, this it's not even reasonable to think that. But some of the more common things you want to have an alternative for. So uh I can almost guarantee you that from this point forward, whenever someone is asking, you know, what are your security controls? Another question they're gonna ask any LMS, or at least I hope they do, is what happens if your system goes down? How do I get my data so I can finish my semester? How do I finish finals? How do I do this? What options are we going to have? And I'll tell you, no one has a good answer to that yet.

SPEAKER_02 12:38

All right, so let's talk about the attack itself. All right. Okay. So at high level, uh, what do we know so far about this canvas breach?

SPEAKER_01 12:50

So, what we know is based off what we're being told. Uh, so make sure that everyone understands that. So this is coming from Canvas and from what we have found out on the internet for as much as that's worth. So we believe that the attacker is called Shiny Hunter, um, or that is the attacker that did the attack. Uh, from the company, it was there's a tool called Free for Teacher that Canvas provides. And this is not their normal licensing structure for higher rate institutions or K through 12, it's something else, but it's for a teacher that may not have a big license, they could get into this tool and use it and create for their classes. Um, what we're being told from the company is there was a problem within that tool that allowed the attackers to get in and then do effectively a privesque or some version of vulnerability to get into the back end systems of that tool. So we don't know a lot of details that hasn't been released. Uh, as of today, um, this would be 5-11 of 2026. I know that the uh person who does our podcast may not like me saying that out loud, but the reality is it's important here. As of today, we know the attackers got into that system and the company is running an incident response investigation to figure out what all has happened. And they said that they're going to release at least parts of that to the customer base. So hopefully in the next couple of days, we're going to have more information about what they do know and how that has affected the rest of the systems.

SPEAKER_02 14:38

Yeah, and I think right now they're saying that the exposed data based on confirmed statements or names, email addresses, uh, student IDs, and uh chats in chat. So that would be internal messages between Canvas users. Yes, that's what they're reporting.

SPEAKER_01 14:56

That's that is what they're reporting. Um now I would say this that the Department of Education still counts that information to be FERPA information. So um there will always be a spin on this on any sort of announcement out to the world about what is what is out there and what you're going to need to do. And you know, people will try to downplay as much as they can. I mean, they're saying things such as um there was no social security numbers, there was no credit card numbers, there was no okay, great, but the information that was released, um, the name, the the ID numbers, the chats that can still be considered FERPA data. So there is still requirements that organizations are gonna need to do uh breach notifications, DOE notifications, all kinds of different things. So, again, this is what we're being told. Yeah, um, more information will come out in the future after the investigation is complete or after they you know do as much as they can. Um, and I believe that the end date for the threat actor is tomorrow to release the information. Now, one of the things that you know we do this a lot in terms of working with the attackers and working with the organizations, you know, that the that middle piece there, sometimes we don't know everything until they've released the data. And then sometimes we release they release the data and like, oh, that was a lot more than what we were expecting. So or we were even seeing exactly. So we don't know right now, we don't know everything. So as soon as the attackers, if they're going to release the data, then we'll have at least that information to some extent after they finish the investigation at the organization level at the canvas level, we might have some more information there. We just don't know right now for sure.

SPEAKER_02 16:52

Gotcha. So do we know if the motivation behind this attack? Like, was it financial? Was it um espionage hacktivism or opportunistic?

SPEAKER_01 17:04

Um, I I think it was financial. Um, I mean, I think that they've contacted them saying, hey, uh, give us money, we're gonna get rid of your data. Um I there has been reports that individual users have been reached out to. We don't know if these are scams, we don't know if it's really the threat actor, but again, it's a uh give us money and we'll delete your data type thing. So I think at the moment, we believe it is a financial attack.

SPEAKER_02 17:32

Do we know how this breach was initially detected? Like did the provider come out, or was this like the customers were going, hey, we're can access our stuff. We're seeing this pop-up. What's happened?

SPEAKER_01 17:44

Um, so I've heard a couple of different things. Some people there the the the threat actor did post information on websites, yeah, you know, that they had gotten into it. Um there was supposedly information provided to them, like a ransomware note or uh some sort of communication saying, Hey, we got in give us your money. Um, to my knowledge, at least none of my clients ever saw anything from a usability perspective. Nothing changed in their versions or in their client or in their uh in their web interfaces. They didn't see any of that. They're going off what they were being told by the leadership by from Canvas and by what they're seeing in the news.

SPEAKER_02 18:27

Now I had a customer that reached out, one of our customers reached out to me with a screenshot saying, uh, hey, we had a user navigate to this training website that we use, and there's this shiny hunters, and we'll take a screenshot and send it to us. And they were super concerned, and this wasn't even a higher ed. Yeah, this was an organization that just utilized it for some internal trainings. Yeah, yeah. And um, that so that spawned like a whole like what's going on? And it's like, hey, let's this is a third party, you guys are okay, this is what this means, kind of thing. But it's kind of interesting that they had no idea, and this had been already you know in the news for a while. Yeah.

SPEAKER_01 19:07

Um well, just like any other incident, people are gonna try to downplay it because they don't want notoriety. Yeah, um, and in this situation, the the tool itself is used so widely, it's also wrapped up in other names. So, I mean, they'll take the functionality and they'll put a different name on it for that organization or brand it however they need to. So people may not have known that it was Canvas, right? Because that was not how they call it. It's just an LMS to them. Um, but most of my people that I talked to, their first notice was a notice from Canvas that something had happened, which you know that came out, I believe, on May 1st uh was the first one that we saw. And my talking about my client. And uh there was three or four notices after that. Every every other day or so, we had gotten notices with a little bit more information about it. But of course, we saw stuff on the news, there's stuff from other organizations, uh, stuff on X. I mean, there's all kinds of different areas where people were posting things, which can make it difficult because you don't ever know what is truthful and what's not. Even some of the information we were getting from the organization, it's not that they were lying, they just didn't know. And so you you don't want to infer anything or assume anything. You just you're trying to work with the best information you have at the time. For sure.

SPEAKER_02 20:34

And so what does this incident tell us about uh detecting blind spots in SaaS environments as an organization?

SPEAKER_01 20:43

Well, it's really hard. It's really hard within the SaaS because you have to rely on the SaaS to do that. But the blind spots for me as a CISO is going to be business continuity. If this system is not available, um, how do we function?

SPEAKER_02 20:58

And I think going beyond just this event, that is a common problem or common just theme that we see doing as in a response, where they have this tool that they utilize that they've invested 100% of into zero backup strategy. Like, no, like, yeah, hey, if our one tool goes down or breaks or disappears tomorrow, but but that's not how it's sold.

SPEAKER_01 21:28

Right. I mean, when you think about it, software as a service was sold as we've got all these data centers and we have all this redundancy, we're always up. Right. I mean, think about AWS or Azure or anything, that that's how it's sold is you know, you have full redundancy, so we're always up. So we the the customers have been sold this, it's just and and the salesperson may not have ever said it, right? But the fact that you're thinking as a That's how they're thinking about it. For sure. So, and I'll and I'll tell you, Canvas was not down because of a resource limitation. Right. It was not down because their servers went down, it was down because they turned it off. They they went into a maintenance point, which don't get me wrong, I think it was the right thing to do for them. They were turning it off to get the attacker out. 100% agree with that. If I was there, that's probably what I would have done as well. You know, and I don't want to second guess the CISO there. You know, I'm sure that they were all running, you know, crazy just to get things functional. But all we know is they intentionally turned it off. Now, going back to that risk thing, how do you know your your vendor's not going to do that for you and you have no control over it? Right. Now, even if they had an SLA saying they won't do that, or they'll have 99.999%, you know, five nines uptime, I mean, some bad things happen, and the best thing to do is turn it off. Um you can't get around that. And since you don't have any control over it, all you what you can do as the institution is okay, how do I continue providing the functions? Now, let's be real about this. This was higher ed and K through 12s, and this was learning. This is academic stuff, and it was very impactful to them. For sure. But it wasn't a hospital that was trying to provide patient treatment, it wasn't infrastructure, you know, oil and gas and electricity and water and things like that. That when it's down, you're you're talking minutes before impact is felt. So huge impact for sure, but it wasn't as bad as maybe another industry they got here. Or critical industry. Right. So, I mean, we got to balance it. Um, for my clients, you know, I'm gonna recommend, I am recommending that we do implement some better business continuity. How do we teach these things? How do we still provide services even if the SaaS goes down? I'm not saying don't use it. I'm just saying, well, let's, what are our alternatives? Right. You know, can we do something different? Maybe we have a backup of some content that, well, we'll just get them online and we'll do it anyway. You know, I I'll say this you know, one of my clients had it was a major impact. Of course, it's the time of year. And uh we were talking, it was late one night that we were still working on this, seven or eight, nine, ten o'clock, whenever it was. And one of the things I got them to understand is we don't make any decisions right now. You know, they had just turned it off. You know, Canvas had just turned the systems off. Uh, let's say four o'clock, whenever it was. I don't remember now. And so we were planning, you know, okay, what if it doesn't come back on? You know, what's worst case scenario? It's not coming back on. We have faculty members leaving, we have students trying to graduate. You know, what what is it we're going to have to do? It's like, okay, I 100% agree with thinking about this. Don't make any changes. Right. Don't make any changes because we don't know yet. And let's see what they do. They're affecting 9,000 institutions. If they were going to be down for a full day, they would absolutely lose market share.

SPEAKER_02 25:32

Right.

SPEAKER_01 25:33

So everything came back on about 11 or 12, is what we think. Uh, we got notices at about three in the morning saying everything was back up. And they were able to go through the next day fully functional. Now we didn't know much, but we knew that the tool was functional and were able to finish out the semester, finish out that week. Um I do appreciate them not being down long, even though I understand why they were down, but it allowed the institution to be able to finish out the last little pieces that they needed.

SPEAKER_02 26:08

Yeah. So we've talked about the initial impact to organizations, but what kind of downstream risks are we going to see from this? Um, and I'm thinking about, you know, based on what we know, what data may have left, uh, or possibly more, kind of depending on what they gain access to. Uh, but down downstream risk, kind of thinking about like credential reuse, um, any kind of future phishing campaigns, yep. Uh, or uh identity theft.

SPEAKER_01 26:37

Like, I I think all of those are absolutely going to happen. You know, I think students, the students' names and their emails are going to are out there now. Um, I think attackers or maybe copycat attackers are going to say, hey, give us X amount of money and we'll delete your data. You know, they might say We've seen that in the past. Absolutely, we have. Um, so I think that that is that's going to happen. Um, I think one of the things that no one is really talking about right now, but when Shiny Hunter did this, this is huge impact for higher ed. I think other LMSs are going to be targeted much more heavily now because now they saw what the impact was. So I think they're going to go after them. So if I was an LMS group, I would be absolutely learning from this. And I would be making sure that there you've got segmentation, you've got MFA, you've got anything and everything you can think of to deploy in your environment to prevent something like this, and have strategies for if it were to happen. So I think that will absolutely happen. Um, yeah, maybe have some pen testers come in. Yes, pen testers and risk assessments now it's and contract management. I some of these contracts are pretty bad and they don't have SLAs in them, they don't have any of these things. So I think that those are going to be, you know, on the uptick. The other thing that we saw from another data breach from about a year and a half ago was information that was gathered in that data breach was used to create fake accounts in the higher ed, which was then used to get federal financial aid. So it was uh financial aid fraud, but it was real students. It was the real student information. So I think that as that gets out there, you might see an uptick in fraudulent applications into institutions trying to get in to get a user account to be used for other things, or to even apply for federal financial aid.

SPEAKER_02 28:31

Absolutely. Yeah, I didn't think about that one. So that's a really good point there. So let's get into your favorite topic of this. What kind of regulatory or compliance issues are these affected organizations going to be looking at? Well, they have to report.

SPEAKER_01 28:50

I mean, no matter what this is, even though this is a third-party vendor, you are a controller. So a higher institution, even though this is your vendor that got compromised, you're the controller of the data. You have to do the reporting. You have to report it to the regulatory side, you're gonna have to report it to the users as a hitch breach notification laws, all the state laws and even country laws, meaning if you have an international student base, you still have to report this. Now, the reporting is pretty simple. Hey, our vendor got hit. Now, the DOE absolutely knows about this. They are still going to want you to report because it's your responsibility to report this appropriately. You will have to do the brief notification, so you're gonna have to let your students know. And you have to say, it wasn't us, it was our vendor, but we're letting you know what this happened because there still has to be a risk assessment done at that student level. The student has to look at it and go, okay, well, maybe I am going to see a lot more phishing attempts. Maybe I am going to see this stuff, maybe you know, password reuse, whatever. So they're gonna have to do a risk assessment of themselves if they know it happened. Now, I think everyone pretty much knows it happened, but you have to do those official notifications. You have to document it, you have to get it out to them, you have to you use your best process to get that out there. Um, you also will have to do remediation processes. How do you prevent this in the future? Now, this is vendor management. It's the vendor that had the problem, but you have to do vendor management processes to help mitigate this problem in the future. So I would absolutely be using this as a tabletop example, as some process that you have a playbook for of what happens when it happens again. What did we learn from it? How do we resolve this? You know, what is the timing? Maybe you have to if you have a four-hour response time from your vendor to let you know that there's an issue, well, then you can have a six hour before you send out a note to your campus. So, what does your playbook say about this? This happened. Now, I put stuff like this on my tabletops all the time, and everyone's like, Oh, that will never happen. It did happen. Yeah. So let's build a playbook because it will probably happen again.

SPEAKER_02 31:02

That's funny to me on the the several tabletops that I've been with you, which I love doing on. Uh, I love your hey, you're not here, or you got hit by a train coming in and you're not here. But how many times have we been on an incident and the guy we need to talk to is on vacation? Arupa.

SPEAKER_01 31:19

Yeah. Well, we just had one where the the uh a stakeholder that we needed to talk to was on a cruise ship. Yeah. It's like, okay, well, that's out. Yeah.

SPEAKER_02 31:28

Yeah. So you just, you know, you gotta have backup plans, you gotta have your chain of command um for sure. So we've kind of talked about like where the responsibility falls between uh the platform provider and the customer. So, what should organizations realistically expect from their SaaS providers uh just in terms of security and transparency?

SPEAKER_01 31:51

Well, okay. I'm gonna say it a very specific way, they should expect what the contract says. Right. Now, you could have elevated expectations depending on if it's a critical function or not, but whatever that expectation is, it needs to be in the contract. So if it's a certain amount of time before they report it to you or they report to you certain information, whatever it is, you want to make sure that is in the contract. Now, for higher ed, uh, and even some for uh K through 12, uh, you have a 24-hour requirement before you have to report things to the Department of Ed and the State Department of Ed and all kinds of different things. So 24 hours. So you really need to know something of a suspected incident within that 24 hours. That needs to be in the contract. One of the other things that we found out is we one of my clients, we didn't know who the notifications were going to. Right. So the organization, Canvas in this case or instructor, was sending out notifications. We didn't know who was supposed to receive those. Now, sometimes it's the person who signed it, sometimes it's a specific person, but is that specific person still there? So those sorts of things come into play. So you need to know what your contract says, and then when a situation happens like this, read your contract because that contract is going to tell you what to expect, when to expect it, and if you don't like it, get a new contract. Right. When it renews, you make sure that stuff is in there. Contract management is huge, and I I harp on this a lot because we have been in incidents where I asked to see the contract, and they're like, Oh, we don't have it. Then you have no expectation of anything at that point. You need to put those expectations in before you need them. So that's what I would say is you're going to get hopefully what the contract says. Sometimes not. Sometimes they will not give you that information. Because here's kind of a red flag there, though. Well, but I mean, we have seen it ourselves where people are so tied up, and you know, you're you you've got a system scene. It started on May 1st in terms of the first notification. I'm sure their teams were working 24-7 for six or seven days straight. Right. They're not thinking about the communication side, they're just trying to get it figured out. So sometimes they don't. I I I've got one story, I usually do a lot of stories, but this is the one story. We had a data breach of a SaaS, and we were notified like three months later. Woof. Um, it went to someone that didn't really know what to do with it, so they ignored the email in terms of coming into the organization. An IT person just literally happened to see it on the screen when he remoted into her computer and says, That's odd. Why what is that email? So that finally got to me. I'm reading it and going, Oh god, what what no one told us about this? Well, skip ahead. Um, three weeks. We were very early in terms of communicating with the that team, and I was able to actually get a meeting with their CISO. Now, this would be like getting a meeting with the CISO at Canvas, right? Really difficult, but I I was just fortunate. I had lots of lawyers on our side, lots of lawyers on our side their side, but I got to talk to him. So we got on a phone call. I had about 10 minutes, I asked him a bunch of questions. He actually did a good job. Yeah, everything he did was right. I I was like, okay, I appreciate that. He did what was right, I understand why he did, why he did it. It was fine. Well, he got off, and then the CEO for that division got on, and the question, the final question from the our lawyers was okay, the contract, the the contract says you will tell us within three days. It took you like 60 or 90 days or whatever it was to notify us. Why that is a completely legitimate question. Yeah, contract says this, you didn't do this. And his response, now I'm I'm off at this point. I've my my camera's off, my mic is off, I'm done. And he goes, Well, I'm really glad you asked that question because I've been doing my research and it doesn't look like anybody ever meets those requirements in that contract. And then he turns to me and he goes, Well, Jonathan, what do you think about that? I I turn my camera on and I turned on my uh my mic and go, Well, I'm really glad you asked that because I teach a contracts course for cybersecurity professionals at various conferences. And so my question back to you is if we don't have to follow this piece of the contract, do we have to follow any of the contract? Right. They didn't like that question. Of course not. And their lawyers cut the call. Just nope, fuck. So the contract is really important, and I harp on this. I mean, people make fun of me because I like reading contracts. Well, the reason I like reading contracts is I don't want to ever get caught off guard. I want to know exactly what is in that contract, what we have to do, what we're supposed to do, what we don't have to do. Your contract, your master service agreement, whatever you call it, is critical. You spend the time up front so that when you have problems later, you have a contract that's going to back you. Yeah.

SPEAKER_02 37:44

It's part of the having a plan. Exactly. Exactly. So we've discussed a lot about lessons learned already just through this conversation. But what role would like logging and monitoring play here, especially when visibility is limited?

SPEAKER_01 38:02

Um, so for my clients, what I told them in the very beginning is review your logging to see if because we didn't know anything, see if anything was coming back. Anything was affecting our systems that looks odd. Anything in terms of the data coming from Canvas back to us looks odd. So we were trying, I mean, your logging is you is you when you don't know what's going on, your logging is key. I mean, that's how you deal with this. So um, in the very first day, first hours, we were absolutely looking at that logging. Yeah. Um, we couldn't tell anything because it wasn't that sort of an attack, but we didn't know that. All we know is that there was a problem. Our number one priority is to protect our systems. And that's where we went in. We started looking at the logs of our systems to see if any problems, any user accounts, any data, anything looked odd so that we could deal with it.

SPEAKER_02 38:58

Right. So let's let's talk about some practical recommendations for security. Okay. Some of them we've already touched on, um, like talking about like what these organizations could do or what steps they would take if uh they use Canvas or similar platforms. But how could an organization strengthen their incident response plans for third-party breaches like this?

SPEAKER_01 39:24

Well, first they need to realize that it is the risk. I mean, what risks there are. So you need to plan that in, you have a risk assessment around that. You need to, of course, have that the contract side of what's going to happen during, you know, the during the situation where how they're going to communicate. Um, but it really does come down to business continuity. I mean, you need to understand that if the risk, here's the risk. A lot of people will look at the risk of the data got compromised, and that's bad, and we have to deal with that. But the other risk is you couldn't teach classes, right? You couldn't graduate, you couldn't do this. One of the things I asked one of our clients was, What did when you can't do a final, how do you deal with that? Do you postpone it till next week? Do you just say, Well, whatever grade you get is what you got, right? Um, what is it? And they were like, We've never missed a final. I'm like, I know we missed finals during COVID.

SPEAKER_02 40:22

So but my question is, these or I mean, Canvas hasn't been around forever, right? These organizations had been doing I mean LMS's have been around for probably 20 years. Yeah, I know, but before I mean schools have been around longer than 20 years. Well, well, just what were they before? You know, this is this gets into like my like I don't know why I think just having like the the simplest backup plan is the most effective. Like what how did we do it 20 years ago before the LMS?

SPEAKER_01 40:54

Well, but but here's the difference. I mean, I understand the backup plan, but schools can't be redundant, they don't have the resources to be redundant. So if you have all of your content in your LMS, you're not gonna maintain your content somewhere else. That's redundant, right? That's that's more effort, more backup, more whatever. The whole intent is to get it to where the the tool itself is the redundancy. In this particular case, they took that down. So I get it. Um, and one of my questions was well, why can't you just do a Zoom call and teach anyway? Yeah, I mean, you have the content, you should know what the syllabus should say, what the content is. Well, yeah, the syllabus was in the LMS. They didn't have the syllabus, they didn't know what content. Now, this was finals, so this they weren't actually teaching like that. They were trying to get their files in and get their grades out of there. Uh, but I will say I had one client who uh they still maintain gradebooks. Yeah, yeah. If for those of us who are old, the old grade books that were green and they had the lines in there, they still maintain those. So they had all the grades. Now we had a way to get those grades into this the student information system, um, but very few people have gradebooks anymore. Yeah. So it that's the thing is you've got to have the backups, absolutely, but they don't have the resources to be redundant, so a lot of them don't do that. Okay.

SPEAKER_02 42:14

Let's get into executive and board level perspectives. Okay. Because I know you're gonna have some great answers here. Because these are the conversations I don't want to be in. Yeah. So, how should CISOs communicate incidents like this to executives and the boards?

SPEAKER_01 42:31

If this is the first time your board is hearing about your issue, yeah, let's say this is the very first time. Yeah, this is this is a problem. Um your board, when you're talking to your board, they should not, you should not be they should not be surprised that a vendor going down has caused such an impact to the organization. That should be in your risk register, right? People should know, oh, these are critical systems. If they go down, critical functions fail. So uh if you've never done that before, uh, you're gonna take a hit on the chin. Not because they went down, but because people were not aware of it. So communication here is key. You need to get in early, you need to explain to the board here are all the systems that, if they go down, have a negative impact on us being able to function. They need to be aware of that. Now they're gonna ask you, well, what are you doing to mitigate that risk? And some things you can do and some things you can't do, but that's a risk appetite issue of what will the board allow you to do? Now, the board says, Nope, you have to have full redundancy of your LMS. Okay, so we're gonna have two one is Blackboard, one is Canvas, right? And you're gonna load your your classes in both, and people can go to both, and that's really, really difficult in higher ed. Absolutely. So

SPEAKER_02 44:00

Then you have people who are like, I want to go do this and it's not here. Exactly. It's like you can't hold me accountable for that.

SPEAKER_01 44:06

Right. Your system is messed up. But I think it is fair to let I mean, you have to let the board know that this is a risk. Right. Now, if they know it's a risk and it happens, what you're doing is you're going to the board quickly and going, Okay, here's where we're at. This is what we're communicated with. This is what our expectations are based off the contract. Here's where we're at in terms of notifications and communications with legal, with regulatory, with whatever. You have your playbook ready to go. You should have a playbook for every single critical function you do. That if it's not there, what are you doing to keep the business running? Right. Now, if you have that playbook, you're walking in with that playbook. You walk in there, you go, here's what we know. Here's everything that they've sent us. This is what we've seen in ours. Here's what we're doing next with the playbook. I think that is the most critical piece that any system, any security professional or AC so when they walk into that board, they need to have that stuff ready to go. If you just walk in getting ready to take a hit, it's like, oh, well, I don't know anything, but I'll go in there. Well, yeah, you're gonna take a hit. And you're probably not gonna be asked back because if you're not providing value to the board, right, they're not gonna have anything. Right. So that's what I would say is you should prepare the board beforehand, that these are critical functions. I'm talking months, years ahead. They should be completely aware of what your critical functions are. And if Canvas goes offline, this is the problem. That way they know about it. And they're like, okay, got it. We understand that's critical. We understand we can't do anything about that. Well, we will continue on.

SPEAKER_02 45:42

Yeah, it just gets into understanding the risks.

SPEAKER_01 45:45

Exactly.

SPEAKER_02 45:46

Exactly. Have there been any risk framing that resonated most when discussing these third-party uh breaches with your clients?

SPEAKER_01 45:58

Of the two that I I always go after is the regulatory side and the impact to the users. Right. You know, what what is going to be the impact to the students in this? Not from the fact they couldn't get their grades in. What is going to be the impact of that data being lost? You know, I I'm I will say it, you know, every single presentation, our job is to protect people. So I always make sure that when we have a data breach, I make the leadership completely aware of what risks we have to the organization for sure. But also what are the risks to the customers or the students or the the patients, whatever? Because I I think that is really important for us to understand. For sure.

SPEAKER_02 46:38

All right. So let's get into some closing thoughts now. Okay. Uh couple questions. One, what's one thing every security leader or Cisho CISO that should what should they take away from this incident?

SPEAKER_01 46:54

This is a vendor. I mean, this is what happens. You put that much trust in a vendor, a vendor can absolutely take down your business.

SPEAKER_02 47:02

What should an organization be doing today to prepare for the next SaaS breach?

SPEAKER_01 47:07

You need to understand all of your SaaS, what are critical functions, and then what is your business continuity around those business, those business functions, those critical functions that will make sure you can continue to work even if this goes down.

unknown 47:21

Okay.

SPEAKER_02 47:22

Uh and are we getting better or worse at handling these types of incidents as an industry from your experience? Or about the same?

SPEAKER_01 47:34

I don't think that's a fair question because I don't think I think we're having more of them, but I don't believe we're getting better. It's not that we're getting worse, but I don't think we're getting we've been kind of static about this. We're we are static in how we do things as an industry. Yeah. Um and then the incidents are happening more often and maybe even bigger impact. So in a way, we're getting worse because we're not improving. Okay. Um, so that's probably how I would phrase that.

SPEAKER_02 48:04

And last question. What is one question every CISO should ask their SaaS vendors?

SPEAKER_01 48:12

How are you going to make sure that we can continue to do business?

SPEAKER_02 48:17

It's kind of an important one. Yeah. I mean, because like we talked about, like when you're outsourcing all of your security to them, you know, they'll give you the everything's redundant. Yep, yep. But maybe ask them straight up, what happens if Shiny Hunters gets in and locks your stuff out? What do we do? Like, what do you do? Well, I mean, they took it down themselves. Yeah. The company took it down. So well, yeah, but it's because people were navigating to their sources and getting pop-ups that Yeah, yeah. I mean, did they see the notification? Yeah, they were Shiny Hunters was they were roasting them, unfortunately.

SPEAKER_01 48:53

As they should, yeah, as they should. So no, uh that's not fair. I I I said that wrong. I don't want everybody. No, no, no. I think we ought to leave this in because I think this is really important because when we're talking from a security perspective, yeah, we have a dark humor about this. Oh, for sure. I think you have to, you have to, yeah, but that comes across in different ways to the audience sometimes. I don't want anybody attacked. I don't. And the fact that our customers were down. I have a lot of good people that I work with on a regular, including myself. I don't sleep a lot anyway, but I didn't sleep a lot that that night everything was down. We were trying to make sure we had plans for it to come up. So I don't ever want people to go down, I don't want them to be attacked. But more importantly, I want people to implement security controls to make sure they're not attacked. And if someone gets into your system, I get they're going to make an announcement saying, hey, yeah, we're in. Yeah. Because some people, some people will deny it. So the obnoxiousness of the threat actors can be helpful to security people because they can say, ah, we know they were here.

SPEAKER_02 50:06

You know, there's no doubt about that. And and to kind of piggyback off that, like as a in the sock, I kind of look at it the same way where we uh I've talked about it on the podcast that I do. Like when I'm saying this ransomware group is out there and they're killing it, or like I've got respect for them, it doesn't mean like I like what they do. Yeah, you don't like what they're doing, but they got skills. But they they do it or they have such an approach in it that's like it's it's kind of amazing the way that they they they change the face of what we know. I mean, that's I mean, not to say that we're always behind, like our goal is to be ahead of them at all all times, but if you don't know what the attack vectors are, you know, you're just kind of identifying your risks and trying your best. So um, yeah, I get that. It's it's just uh um they're good at what they do, you know. Yeah, almost I won't I wouldn't say to like our benefit, but like the way it changes, um it makes us think differently as well.

SPEAKER_01 51:10

Yeah, we have to think like the adapters. Yeah, I mean, and and you're right, it's a uh you give them respect because I mean these people are being successful. Yeah, our job is to be just as good. Yeah and I'm just not just saying us, you know, here at alias, it's everybody who manages the system. If you have data of people or your systems could hurt someone if they go down, it's your job to be as good as or better than the attackers. That's just the reality. This is this is the job, this is security, you know, it's not funsies, it's security. Yep. So these things are important. We have to be better than the attackers, and when the attackers best us, okay, I get it, let's get better.

SPEAKER_02 52:00

Yeah. And not to harp on like get a pen test again or anything. Um but I mean, I see it with me, you know, we work with different socks and stuff too. We help socks that are they're growing and learning, we try to help them harden uh their their security environment. Um, and a lot of the times it's just it's just lack of awareness. Yep. Um that coupled with like alert fatigue and there's nobody really going in and um helping to mitigate you know the type of activity that's happening. So when they when a real alert comes in, it's like like that person who received the email who didn't know what to do with it, like uh we run into a lot of that. So um I think just from a SOC perspective as well, we see this third-party attack, and it's um you vet your vendors, you know, echoing everything that you said, look at your contracts, um, and just from a security, just specifically like SOC team perspective, um just be aware, like try to get a baseline of like at least if hey, we see this activity coming in, we haven't seen it before. Yep, look into it. Yeah. Um, I I've again we've run into so many instances where you we've got an organization that's like, uh, we've seen something similar before. So we're just right, well, we'll just we'll just monitor this. And yeah. Um again, the these guys shift in their um their attack methods so frequently. Um you just gotta stay on top of the curve. You know, you gotta be out there, take up and read, you know, like go look for information. All right, so final closing thought, and then we can we can be done. Yeah, I think we're gonna get hooked here soon. Oh, yeah, she's yeah, probably. So I what is what would be I guess just from a CISO perspective, like what is the one if there's one thing you would want organizations or a a a fellow CISO to take away from this, from this incident, what would you hope that is?

SPEAKER_01 54:20

Even though it's not your fault, you still have to clean up the mess.

SPEAKER_02 54:23

All right, that's fair. And that is a wrap on this episode of the Secure AF Podcast. Uh, thank you so much for joining us. If you have questions about this incident, want to talk to uh us about CISO stuff or pen tests or whatever, uh hit us up on our social media or via our website. Uh, as always, thanks for your time. Bye.

SPEAKER_00 54:54

The Secure AF Podcast is a production of alias Cybersecurity. Visit us online at aliascybersecurity.com. All rights reserved.