Oct 142014

A former Maricopa County Community College District employee alleges executive leadership closed their eyes to a report on their database security conducted after their massive data breach in 2013 so they would have plausible deniability in any litigation. As a result, the employee alleges, the findings were never shared with those tasked with securing MCCCD’s data assets. 

In November 2013, Maricopa County Community College District (MCCCD) disclosed that they had been informed by the FBI that 14 databases with personal information had been found up for sale on the Internet. The potential compromise of 2.5 million students’, employees’ and vendors’ personal and financial information currently stands as the largest breach ever in the education sector.

As part of its continuing investigation into that breach, DataBreaches.net recently disclosed parts of a report issued by Stach & Liu in 2011 after an earlier hacking incident. Failure to properly remediate that breach had been cited as a factor in the 2013 breach. Of special relevance now, MCCCD’s external counsel had asserted that MCCCD administration at the highest levels never even knew of the report’s existence until after the 2013 breach. Their claim was disputed by former employee Earl Monsour, who stated he had delivered the report to the Vice Chancellor for ITS.

Today, DataBreaches.net can reveal that following the massive 2013 breach, there was a database security assessment that MCCCD has not shared with its own personnel nor the public.  Will MCCCD leadership claim they have never seen this report, too?  According to a former employee, if the Chancellor and executive leadership do claim they never saw this report, it is because MCCCD did not want to see it for fear it could hurt them in litigation.

According to the former employee who spoke with DataBreaches.net, Oracle had been brought in to assess database security following the 2013 breach, but  MCCCD subsequently tried to stop Oracle from delivering their report to MCCCD:

(MCCCD) made it clear that if they did not see it (the report), they could deny it… they then put Oracle on notice they were going to go after them as this was going to cause harm to their case.

There is no mention of retaining Oracle in MCCCD board minutes following the 2013 breach and no mention of Oracle conducting any security assessment in the timeline of steps Wilson Elser stated the District took following the breach. Wilson Elser’s timeline, submitted in November 2013, names Stach & Liu and Kroll Advisory Services and describes their roles, but never mentions Oracle.

The former employee claims that to MCCCD’s great upset, the Oracle report was delivered, but MCCCD leadership did not look at it:

To be clear – no one at MCCCD leadership saw this… did not want to see it… did not want it on their servers…  they were pissed to the max that this document was sent to MCCCD. The legal teams did everything in their power to never let this see the light of day… and it has not.  Therefore, nothing that was recommended by Oracle was done as part of the official MCCCD remediation plan.

It is one thing for lawyers to claim a report is privileged or work product and exempt from public disclosure or disclosure to any adversary in litigation. It is quite another thing for those responsible for securing tons of personal information to intentionally not read a report they presumably paid for and that might contain important vulnerabilities or problems that should be addressed to prevent future breaches. Should there be another massive data breach, and should it be determined that the vulnerabilities had been identified in Oracle’s 2013 report, the consequences to the District and taxpayers could be significant.

So what was in Oracle’s June 2013 “Database Security Healthcheck”  that MCCCD’s leadership allegedly did not even look at?

Because this blog does not want to provide hackers with a roadmap to attack MCCCD if MCCCD still has not adequately secured its network and systems, the full report will not be published here at this time. DataBreaches.net will, however, note just some of the problems the report identified (without the elaboration or recommendations that were provided in the report). The categorization as “severe,” “significant,” and “moderate” are Oracle’s labels:

    • “Network Not Secure” (Severe Risk category)
    • “Default Application Accounts PW Not Changed” (Severe Risk category)
    • “Unsecured Access to Servers” (Significant Risk category)
    • “No Tool SQL Injection Prevention” (Moderate Risk category)

As noted above, the preceding are just some of Oracle’s findings included in their report. In many cases, Oracle’s report described MCCCD’s then-current security for an identified issue as “none.”

Lack of Transparency a Long-Standing Problem at MCCCD?

Some of the issues raised in Oracle’s June 2013 report are the same issues Oracle raised in its April 2008 “Insights” report. To be fair to MCCCD, many of the issues raised in the 2008 report required vendor solutions or solutions were not even available at the time. The April 2008 report was submitted to MCCCD one month before MCCCD experienced an unrelated data leak in Peoplesoft due to a programming error that allowed any user to query the database for any of millions of users. Although MCCCD claimed that the exposure only affected people with the last name of “Gilford,”  former employees tell DataBreaches.net that they believe that the entire database could have been queried during the few weeks before the error was detected.

In 2011, MCCCD experienced a breach involving a MySQL database on public-facing web servers controlled by the Marketing Department.  While that breach was relatively small as such breaches go, sources tell DataBreaches.net that they felt “lucky” it wasn’t worse, and knew that if they did not secure the web servers, the next breach could be much worse. And it did get worse, they say, because MCCCD administrators ignored or rejected the advice of employees who tried to secure the system and who repeatedly urged MCCCD to quickly replace the badly compromised web servers. Yet, despite the fact that it had still had not replaced the compromised web server that had been brought back online, and despite the fact that its monitoring system was in shambles, the Vice-Chancellor of ITS gave a report to the Governing Board in March 2012 where the Board minutes reflect he asserted that the District was “very consistent” with the industry and things were “very good about where we are but we have a lot of work to do.” Nowhere in his presentation did he reveal how serious the situation actually was.

But it was not just the Governing Board who were being misled or given incomplete information by leadership about security concerns. In the process of investigating the 2013 breach, DataBreaches.net heard from several employees who said that MCCCD never shared Oracle’s 2008 report with them or Stach and Liu’s 2011 report after the 2011 breach. Why did MCCCD never share the Oracle 2008 with its ITS personnel at the time? Why didn’t leadership share the Stach & Liu 2011 report with those attempting to remediate the 2011 breach? [CORRECTION: The S&L report was reportedly shared with  the two ITS leaders of the Systems and Networking departments (who were responsible for the compromised server) as well as with the Vice Chancellor of ITS.]

As one of the former employees who quit in frustration put it:

I have to ask what motivation could they have to withhold information which could have been used to make the environment more secure?  We are not security experts, and designing and maintaining a secure environment costs money – not just applications but FTE’s.  We could not even get an upgrade done…any change associated with expenditure was vetoed almost immediately dating from 2010.

Truth is not really a part of the equation at MCCCD and I doubt it ever will be…

So despite Stach & Liu’s post-2011-incident report, despite state audits that repeatedly raised concerns about security controls that MCCCD had not addressed satisfactorily, despite internal reports and emails from ITS personnel urging MCCCD to deal with the problems more urgently, despite repeated requests for an external audit of the ITS department and the deteriorating work environment that resulted in MCCCD losing approximately 50% of its ITS personnel, and despite briefings of Chancellor Glasper and all members of his Vice-Chancellor group, MCCCD leadership did not ensure that the security issues were effectively addressed, all to the detriment of the millions of people whose personal information the District stored, and to the detriment of students and taxpayers.

The former employee writes:

“The question I have is this, what did they do in response to this ‘in your face’ information? The short answer was not much… if anything… They …. chose to move forward in a normal business fashion… no fire alarm, no urgency, nothing.”

And then, after the largest hack ever in the education sector, MCCCD’s Chancellor and high-level administrators allegedly tried to avoid acknowledging and disclosing Oracle’s  security assessment of their database security – to protect themselves?  Why wasn’t this report shared with everyone involved in trying to secure personally identifiable information in the wake of the 2013 breach?

DataBreaches.net reached out to MCCCD and asked them to respond to allegations that MCCCD tried to evade Oracle’s June 2013 report for litigation reasons.  They were also asked whether Chancellor Glasper and high-level administrators and the Governing Board ever read the Oracle 2013 report, and if so, when. After two requests, MCCCD’s District Director of Marketing and Communications responded that he was having trouble helping people identify the report in question, and could DataBreaches.net provide more details about the report. DataBreaches.net provided a few additional details, but never heard back from MCCCD after that correspondence last night. If a response or statement is received, this post will be updated.


Sep 302014

In January 2011, DataBreaches.net reported that login credentials for the Maricopa County Community Colleges District (MCCCD) were up for sale on the black market. That month, the FBI also contacted Maricopa to alert them to the breach. In response to the incident, MCCCD brought in Stach & Liu (now Bishop Fox) to investigate and make recommendations.

Following MCCCD’s  second – and massive – data breach in 2013, this blogger filed a formal complaint with the FTC (pdf) asking them to investigate MCCCD’s data security and to take action to protect the financial and personal information of students, vendors, and staff. Yesterday, EPIC filed a supplemental complaint (pdf), also calling on the FTC to investigate and take action against MCCCD to protect personal information.

Today, in the first of what is expected to be a series of reports, DataBreaches.net is disclosing what Stach & Liu found in their preliminary investigation of the 2011 breach. In February 2011, they reported that the Application level, they found:

  • Comprehensively insecure code with systemic critical flaws like file inclusion and injection
  • Hundreds of attacks per month as far back as analysis was performed for file inclusion and SQL injection; and
  • No secure coding practices identified whatsoever

For the compromised web server, they found:

  • All LAMP systems are running out-of-date versions
  • 5 LAMP servers were backdoored with web shells
  • Up to 29 virtual hosts were stored on a single server
  • Blank passwords SSH keys were used to connect between systems; and
  • Shared filesystems were used bewteen public web servers

Their findings from investigation at the database level revealed:

  • 29% of passwords were re-used
  • 21% of passwords were cracked in seconds
  • 2 databases were poisoned with credentials to FTP
  • Up to 140 databases were stored on a single server; and
  • Databases dating back to 2000 used cleartext or unsalted passwords with FILE privileges

Note: the databases referred to above were those residing on the compromised webservers and other virtual servers. These servers were managed by the server team and MCCCD’s Marketing Department – not the employees MCCCD subsequently fired and tried to blame for everything.

The Stach & Liu report pointed out the consequences of what they had found:

  • MCCCD had already experienced a data breach and loss
  • MCCCD systems had been included in a botnet
  • Backdoors were installed on servers
  • Servers had been abused for Pharma marketing (spam); and
  • MCCCD could experience brand and reputation damage

Stach & Liu made a number of recommendations for next steps in terms of assessment and remediation. As subsequent posts will illustrate, documents provided to DataBreaches.net – most of which have not been shared with the public under open records requests – indicate that MCCCD never fully implemented the recommendations of its consultants, its own employees, or state auditors who, year after year, noted data security control concerns.

Sep 302014

Earlier this year, I reached out to the Electronic Privacy Information Center (EPIC), to encourage them to join DataBreaches.net in filing a complaint with the FTC concerning the massive data breach at Maricopa County Community Colleges District (MCCCD).  I am pleased to see that they have done so, agreeing with me that MCCCD is covered by the Safeguards Rule and that the FTC can enforce it in the education sector.

EPIC has announced:

EPIC has filed a complaint with the Federal Trade Commission concerning the loss of personal information of almost 2.5 m current and former students, employees, and vendors in Maricopa County. According to EPIC, the District’s failure to maintain a comprehensive information security program led to a “massive breach of names, addresses, phone numbers, e-mail addresses, Social Security numbers, dates of birth, certain demographical information, and enrollment, academic, and financial aid information.” EPIC further alleges the District violated the Federal Trade Commission’s Safeguards Rule by failing to protect students financial information. EPIC’s complaint follows a similar complaint by DataBreaches.net. EPIC said that, “many education institutions in the United States are subject to the Safeguards Rule. The District’s case is a particularly egregious example of the risk of failing to safeguard sensitive personal information.” For more information, see EPIC: Student Privacy.

A copy of DataBreaches.net’s complaint to the FTC can be found here (7 MB, pdf).

In the weeks to come, DataBreaches.net will be releasing more documents and files that indicate that MCCCD administration was repeatedly warned about the security risks that it had not satisfactorily addressed, leaving student financial information and others’ personal information at ongoing risk of theft and misuse.

Jul 222014

It appears that Maricopa County Community College District (MCCCD) is doubling down on trying to throw employees under the bus in the wake of its 2013 breach affecting 2.5 million. According to a web site created by the attorney for the employees:

The MCCCD Administration is accusing Mr. Corzo of not doing a job that wasn’t his to do, being responsible for systems he wasn’t supposed to be responsible for, knowing about a document that was never shared with him, not communicating upwards when he repeatedly did so, and not doing enough during an incident in 2011 when he was onsite, working with his staff and others to help MCCCD address a small security breach.  In 2013 when the second and larger breach took place, Mr. Corzo was no longer assigned to any supervisory or database duties. The ERPs at MCCCD that Mr. Corzo was responsible for were never compromised in 2011. A small database residing on the main maricopa webservers was compromised. This database was the responsibility of the marketing department and the network and server team at MCCCD not Mr. Corzo’s team.

Read more on  Maricopa Security Breach.

The residents, taxpayers, and governing board of MCCCD should not allow this travesty to continue. Documentation provided by Mr.Corzo and others raises serious questions about both due process and the accuracy of the administration’s accusations.

As I’ve said before, this case calls for an independent investigation – by Arizona’s state legislature, the state attorney general, Congress, and the Federal Trade Commission. The 2.5 million who have been at risk of identity theft deserve no less. The employees who claim they have been scapegoated and falsely accused deserve no less. And the taxpayers and students of Maricopa County who are now paying more tuition because of the breach costs deserve no less.

Will the MCCCD governing board agree with the chancellor’s recommendation to terminate Mr. Corzo’s employment when the board meets tonight, or will they actually read his lengthy annotated response to the charges and give him an opportunity to testify to them and to call the witnesses he has always sought to call?  For the sake of MCCCD and fairness, I hope it’s the latter.

Update: See coverage by Mary Beth Faller in today’s Arizona Republic.

May 172014
President Truman with his "The buck stops here" plaque on his desk.

(SOURCE: Wikipedia)

President Truman had a sign on his desk that said, “The buck stops here.” We could use more of that accountability when it comes to data breaches in the education sector.

Back in 2006, when I first began blogging about data breaches on PogoWasRight.org, I covered a series of breaches at Ohio University. One of the things that made the Ohio U. situation newsworthy was that the university publicly fired two IT Managers.  The firings made sense to some, who suggested that having heads roll might be a smart public relations move to show that the university took the breach seriously.

But shouldn’t the heads that roll be the heads that were responsible? The two Ohio University employees were subsequently found to have had no responsibility for the breaches. Stunningly, even though a grievance committee recommended reinstatement and an apology, the provost decided she  would not rescind the firing because they “”failed in their responsibility for designing and maintaining a secure network.”

Firing employees for not providing a secure environment after you’ve ignored their recommendations that might have prevented the breaches seemed somewhat unfair to me.

And that’s what seems to be happening again in the aftermath of the Maricopa County Community College District (MCCCD) breach that I’ve been covering on this blog since last year.

When MCCCD finally – seven months after they were informed of the breach  – issued a statement and started notifying those affected, their notification to state attorneys general blamed IT employees who allegedly failed to live up to MCCCD standards and obstructed the investigation into the 2011 breach, allegedly thereby leading to the 2013 breach.

In the wake of the massive data breach, a number of employees resigned or were forced out. Based on information I’ve continued to review in my investigation,  I suspect there probably were grounds to hold a few of them somewhat responsible. But what is concerning to me is that MCCCD initiated disciplinary proceedings against two employees – Miguel Corzo and Earl Monsour – who wouldn’t be forced out because they had done nothing wrong and refused to become scapegoats for MCCCD’s mismanagement of its IT department and data security.

It is Ohio University all over again.

Based on MCCCD’s organizational chart for its ITS department in 2011, neither Corzo nor Monsour had any responsibility for the web servers that were  compromised in 2011. After the breach, they were asked to help and they tried repeatedly to get MCCCD to deploy appropriate security programs and controls that would have prevented the 2013 breach. Indeed, their efforts to address MCCCD’s inadequate security programs and policies began years before the first breach.

– In 2009, Corzo authored a strategic report to the District that made numerous recommendations that would be considered industry standard. His recommendations were allegedly dismissed by Vice Chancellor Kahkedjian.

– After the January, 2011 breach, Corzo, Monsour, and others, including Martin Gang (who left MCCCD in 2011),  quickly identified the problems leading to the 2011 breach and what needed to be done to remediate it. They repeatedly tried to get MCCCD to implement the recommendations of external consultants and ITS personnel.

– When MCCCD didn’t address the security issues in a timely fashion, Corzo and Monsour filed an oversight report. MCCCD allegedly did not respond to it. Nor did MCCCD appear to implement recommendations in a state audit that had noted deficiencies and concerns – recommendations that MCCCD said they agreed with and would implement.

– Not giving up in their efforts to address MCCCD’s serious data security deficiencies, Corzo and Monsour escalated the matter by filing a  grievance report in 2012. MCCCD allegedly did not respond to the grievance report, either.  Neither has their Governing Board, to whom the grievance report was recently escalated.

Not surprisingly, then, in  2013, two years after it had suffered a similar breach that it had not fully remediated, MCCCD suffered a  massive data breach that affected 2.5 million.

And MCCCD pointed the finger at two employees who had no responsibility for the first breach and had tried repeatedly and tirelessly to get MCCCD to implement effective policies and programs? Employees who weren’t even there in 2013?

Enough, already!

Inspection of the approximately 1,000 incidents in DataLossDB.org involving higher education institutions in the U.S. reveals that the MCCCD breach in 2013 was the largest data security breach ever reported by a U.S. institution of higher education.

Has MCCCD and its governing board accepted responsibility or said, “The buck stops here?”

No, they have not. They have seemingly tried to deflect blame to two employees who tried to protect customer and consumer information. And while MCCCD has tried to claim that a consultant’s report following the 2011 breach was never given to MCCCD at the “highest levels,” their claim has been loudly refuted by at least three employees who affirm that the report was given to the Vice-Chancellor of ITS at the time.

Yes, it would probably be appropriate to have some heads roll in this case, but if heads roll, it should start at the top – with the Chancellor and Vice-Chancellor – where there seems to have been serious failures in management. They need to be held accountable for failing to respond to repeated warnings and for failure to ensure that millions of people’s personal and financial information was adequately secured.

Frustratingly, while MCCCD is already facing several potential class-action lawsuits and is spending millions on security upgrades, credit monitoring services, lawyers, and consultants, MCCCD has so far escaped any federal regulators because no federal agency investigates or enforces data security in the education sector.

That needs to change. It’s high time the federal government took breaches in the education sector as seriously as it takes breaches in the business sector, the financial, and the healthcare sector.

Universities collect and store a tremendous amount of personal, financial, and health information.  This year, parents and privacy advocates have created waves throughout the country about the importance of protecting student data in the k-12 sector. Many of the same issues apply to secondary education.

If the FTC can put businesses under a 20-year monitoring plan, and if the FTC can go after Wyndham for repeated breaches and inadequate security, it should have the authority to hold universities accountable for data security, too.

Ask not where the buck stops, MCCCD. It stops with thee.

And this blogger is going to do what she can  to ensure that Congress and federal regulators understand that they can no longer sit on the sidelines and just hope that student data are adequately secured.

Just like Congress called Target officials in to answer questions about their massive breach, there should be a Congressional hearing about MCCCD’s data breaches. And if Congress really wants to understand how 2.5 million students, vendors, and employees wound up at lifetime risk of identity theft, it should have the FBI, MCCCD’s chancellor, vice-chancellor, Corzo and Monsour testify.  And in a second panel, they should have someone who can talk about breaches in the education sector, a representative from the U.S. Education Department, and a representative from the FTC to talk about what they currently can and cannot do with respect to enforcing privacy and data security in the education sector.

Will any member of Congress do this? If you agree it should be done, feel free to forward this commentary to your Senator and Representative.