Earlier today, President Obama announced a series of reforms to address abuses by the National Security Agency. We were heartened to see Obama recognized that the NSA has gone too far in trampling the privacy rights of people worldwide. In his speech, the President ensured that National Security Letters would not come with perpetual gag orders, brought new levels of transparency and fairness to the FISA court, and ended bulk collection of telephone records by the NSA. However, there is still much more to be done.

We’ve put together a scorecard showing how Obama’s announcements stack up against 12 common sense fixes that should be a minimum for reforming NSA surveillance. Each necessary reform was worth 1 point, and we were willing to award partial credit for steps in the right direction. On that scale, President Obama racked up 3.5 points out of a possible 12.

1. Stop mass surveillance of digital communications and communication records.

Score: .2

There are three types of mass surveillance that we know about that we were using to evaluate Obama’s promises in this category: surveillance of millions of phone records under Section 215 of the PATRIOT Act; surveillance of Internet communications internationally under Section 702 of the FISA Amendments Act; and surveillance of communications overseas under Executive Order 12333.

In order to score a full point in this category, Obama would have needed to declare that the executive branch would no longer be using any of these authorities to engage in mass surveillance. He tackled only one of these issues somewhat: the surveillance of telephony metadata under Section 215 of the Patriot Act. Specifically, he acknowledged the recommendations of his review group that the government cease to collect and maintain a database of all Americans’ telephone records. He is ending that program, which is laudable. However, he left open the door to having telecom companies or another third party maintain a similar set of mass data, so even as to 215, we could not give him the full ⅓ of the point.

2. Protect the privacy rights of foreigners.

Score: .3

All too often, the NSA’s official position is that foreigners—or anybody deemed sufficiently likely to not be a “U.S. person”—are not given any legal protections under surveillance laws. This situation is unacceptable and out of line with international human rights law, as we’ve put forth in our Necessary and Proportionate Principles, now supported by over 300 organizations worldwide. We demanded that individualized targeting be conducted for non-US persons.

Obama nodded a bit to this situation, and proposed that some reforms be made, but did not give real specifics. While he also did not acknowledge any legal obligations, he did recognize a “special obligation” on U.S. intelligence agencies, and specifically called out a new, higher standard on eavesdropping on foreign leaders. But that’s not enough: privacy consideration should not be a privilege afforded only to top officials. Given these small steps forward but ongoing problems, we’ve given Obama .3 points in this category.

3. No data retention mandate.

Score: 0

Obama’s review group recommended that the telephone metadata surveillance program be taken away from the government, suggesting that a third party or even telecom companies themselves be responsible for maintaining a searchable list of our calling records. This approach—mandating companies act as Big Brother’s little helper—won’t alleviate the serious privacy concerns with maintaining a digital record of every call we make.

We had hoped that Obama would make clear that he would reject any form of mandatory data retention. Instead, Obama acknowledged some of the concerns with a data retention mandate but called for  “options for a new approach that can match the capabilities and fill the gaps that the Section 215 program was designed to address, without the government holding this metadata itself.” He never specifically rejected the idea of forcing companies or a third party to hold this data, and so he does not receive a point in this category.

4. Ban no-review National Security Letters.

Score: .5

The President gets half a point here, since he endorsed ending the permanent gag orders that accompany administrative subpoenas known as National Security Letters, under which the FBI can on its own demand information about you from your communications service providers. We still need specifics, and the details really matter—even fixed-length gags would violate the First Amendment, for example, and gags would still need to be approved by courts—but this was a good and necessary step. Obama didn’t get the other half, though, because he did not agree with EFF and his own review panel that NSLs should only issue after judicial approval. Early in 2014, EFF will ask the 9th Circuit Court of Appeals to find, like the District Court for the Northern District of California already did, that the NSL statute is unconstitutional in its current form.

5. Stop undermining Internet security.

Score: 0

The NSA’s systematic efforts to weaken and sabotage the encryption and security technology make us all less safe. But in contrast to his review group’s recommendations to stop those practices, Obama was silent on the issue. That silence is disappointing, as this is a critical problem that has not just undermined the privacy of millions around the world, but poisoned our collective trust in institutions that depend most on it. Zero points.

6. Oppose the FISA Improvements Act.

Score: 1

The FISA Improvements Act seeks to codify into law the NSA’s controversial and illegal practice of collecting and storing the telephone records of hundreds of millions of Americans. While Obama’s administration had earlier indicated support for the bill, today’s announcement made clear that Obama was not going to support this program going forward and thus was not supporting the FISA Improvements Act. We would have preferred it if Obama had stated clearly that he would veto any bill that attempts to codify mass telephone metadata surveillance, but we felt this was good enough to merit a point.

7. Reject the third party doctrine.

Score: 0

The third party doctrine is an outdated and deeply problematic legal theory that wipes out many of the privacy protections we could otherwise enjoy. It’s the shaky foundation on which some of the most invasive programs by the NSA and other law enforcement agencies rest. Obama should have said that we have a reasonable expectation of privacy in data even though we’ve trusted third party service providers with it—instead, he was silent on the issue.

8. Provide a full public accounting of our surveillance apparatus.

Score: .5

In our criteria, we asked that Obama “appoint an independent committee to give a full public accounting of surveillance programs that impact non-suspects around the world” and that this committee “directly engage whistleblowers like Thomas Drake, William Binney, Edward Snowden and others, and include independent technological experts.”  For this category, we awarded Obama with a half point because he did appoint his counsel, John Podesta, to lead “a comprehensive review of big data and privacy.” However, it remains to be seen whether this committee will actually provide a full public accounting or engage with the whistleblowers who have much to contribute.

9. Embrace meaningful transparency reform.

Score: 0

Fundamental to all of the problems surrounding NSA spying is the fact that the government’s notorious secrecy shields it from any sort of meaningful oversight or accountability. This appears, among other places, in the overclassification of documents that should not actually be secret, in the executive branch’s ruthless campaign against whistleblowers, and in its continued abuse of the “state secrets” privilege in the courtroom. Obama could have announced changes to these secrecy standards, embracing transparency as a default, and making some good on his now laughable election promise to be “the most transparent administration in history.” Instead we got nothing.

10. Reform the FISA court.

Score: 1

We gave Obama a full point for these reforms, since he embraced both independent advocates for the FISA court and an annual process of review of FISC decisions for declassification. While we would like the review to be more current, and there is much to be done to ensure that the independent advocacy panel has a real, unfettered role, Obama’s announcement indicated a good direction on both.

11. Protect national security whistleblowers.

Score: 0

Obama was clear: “One thing I’m certain of, this debate will make us stronger.” And there is little question that this debate would not have happened without the evidence brought to light by Snowden and other whistleblowers. It might seem that Obama would have some recognition that, but for these individuals, we would not be having this important debate.

Sadly, Obama’s speech today gave no indication of a change in strategy in his administration’s war on whistleblowers. If Obama welcomes this debate, he should stop his attack on the people who have risked so much to help make it happen.

12. Give criminal defendants all surveillance evidence.

Score: 0

It’s a cornerstone of our justice system that the accused have the right to see all the evidence against them. That made it very alarming when we learned that the NSA was collecting intelligence and then laundering it into criminal investigations by the Drug Enforcement Agency and other law enforcement groups. This practice conflicts with the protections enshrined in the Fifth and Sixth amendments, and should be stopped immediately. While Attorney General Holder has promised to review the cases, the Administration has not promised to ensure that everyone whose information was shared with law enforcement agencies by the NSA ultimately gets notice. Obama didn’t mention this necessary measure in his speech, and gets no points.