In today’s diverse and crowded landscape of messaging systems, what are the most secure and usable tools? The Electronic Frontier Foundation (EFF), a digital rights group based in San Francisco, CA, has been considering this question for a long time. Their most prominent initiative in this regard has been the 2014 release of the Secure Messaging Scorecard (SMS), a 7-criteria evaluation of ‘usable security’ in messaging systems. While the 2014 version of the SMS (now 1.0) displays a number of apparently straightforward criteria, a first look into the backstage shows that the selection and formulation of these criteria has been anything but linear, something that has been made particularly evident by the EFF’s recent move to renew and update the SMS. Indeed, in a digital world where the words security and privacy are constantly mobilized with several different meanings, it seems relevant to analyse the SMS’s first release, and the subsequent discussions and renegotiations of it, as processes that co-produce particular definitions of security, of defence against surveillance, and of privacy protection. This article argues that, by means of the SMS negotiations around the categories that are meaningful to qualify and define encryption, the EFF is in fact contributing to the shaping of what makes a ‘good’ secure messaging application, and what constitutes a ‘good’ categorization system to assess (usable) security, able to take into account all the ‘relevant’ aspects – not only technical but social and economic.
In today’s diverse and crowded landscape of messaging systems, what are the most secure and usable tools? The Electronic Frontier Foundation (EFF), a digital rights group based in San Francisco, has been considering this question for a long time. Their most prominent initiative in this regard has been the 2014 release of the Secure Messaging Scorecard (SMS),
While the 2014 version of the SMS (now 1.0) displays a number of apparently straightforward criteria – including, but not limited to, encryption of data in transit, ability to verify contacts’ identities, available documentation for security design assessment and whether a code audit has happened in the recent past – a first look into the backstage shows that the selection and formulation of these criteria has been anything but linear.
Indeed, in a digital world where the words security and privacy are constantly mobilized with several different meanings – even within the same debates and by a priori alike actors – it seems relevant to analyse the SMS’s first release, and the subsequent discussions and renegotiations of it, as processes that de-stabilize, negotiate and possibly re-stabilize particular definitions of security, of defence against surveillance, and of privacy protection. This article argues that, by means of the SMS negotiations around the categories that are meaningful to qualify and define encryption, the EFF is in fact contributing to shape what makes a ‘good’ secure messaging application and what constitutes a ‘good’ measurement system to assess (usable) security, able to take into account all the ‘relevant’ aspects – not only technical but social and economic.
The article draws primarily from original qualitative interviews conducted with members of the EFF and on the examination of the first version of the SMS, as well as other documentary sources. From a theoretical standpoint, the article draws upon an important body of work in the field of science and technology studies (STS) that has addressed the ‘making of’ systems of classification, categorization and measurement as a crucial component of human interaction and governance processes (notably
As Geoffrey Bowker and Susan Leigh Star remind us in their seminal work
From an STS perspective, classification and categorization processes are strictly linked to the shared perception different actors are able to have of themselves as belonging to a community. In many cases, they highlight the boundaries that exist between communities and constitute the terrain where they might get closer – or drift further apart:
‘Information technologies used to communicate across the boundaries of disparate communities […] These systems are always heterogeneous. Their ecology encompasses the formal and the informal, and the arrangements that are made to meet the needs of heterogeneous communities—some cooperative and some coercive’ (ibid, 286).
Categorization processes, as Goodwin (
Borrowing from Cole (
Membership in a community of practice has as its sine qua non an increasing familiarity with the categories that apply to all of these. As the familiarity deepens, so does one’s perception of the object as strange or of the category itself as something new and different (
We will see how, in the highly unstable environment that is the EFF’s pioneering attempt to categorize secure messaging tools with a view to providing guidance on their quality, the embryo of a community of practice that goes beyond the relatively homogeneous group of cryptography developers starts to be born. It includes users of different expertise, trainers and civil liberties activists, and simultaneously reveals the manifold points of friction between these groups.
Classifications and categorizations, Bowker and Star later point out (ibid, 319) are also ‘powerful technologies’, whose architecture is simultaneously informatic and moral. Due to their embeddedness in working infrastructures, they can become relatively invisible as they progressively stabilize, without losing their power. Thus, categorization systems should be acknowledged as a significant site of political, ethical and cultural work – three aspects that our analysis of the SMS negotiations will unfold. In these three respects, categories are performative (
Before we move on to analysing the categorization system this paper is about – and what we believe it brings to the understanding of this special issue’s theme of ‘rethinking privacy’ – it is useful to give some elements of context about the ensemble of tools, and more broadly the field, it seeks to address. We will not engage here in a literature review about encryption, security and privacy, for two reasons. First, because volumes have been and are being written on each of these topics, and it would be a daunting task to even start acknowledging them with any degree of completeness. Second, because the aim of this paper is not to assess the validity of the arguments supporting different scholarly appreciations of each topic and position ourselves among them. Instead, we wish to show how understandings of ‘good’ encryption, security and privacy emerge – more often than not, in a non-academic and bottom-up fashion – from a specific categorization attempt that, because of its pioneering role, raises a number of questions, tensions and reflections among and across the interested actors. That being said, this section will briefly address the current state of the field of secure messaging tools.
Whilst it of course predates the Snowden revelations – but, certainly, was further spurred by them – secure messaging is a lively and constantly evolving ecosystem of standardized and non-standardized projects. Developers seek, in particular, to apply the technique of end-to-end encryption
According to a post-Snowden systematization of knowledge paper on secure messaging, the field currently suffers from the ‘lack of a clear winner in the race for widespread deployment and the persistence of many lingering unsolved research problems’, as well as discrepancies between ‘grandiose claims’ and actual provided security – an issue that tools such as the SMS have attempted to tackle (
End-to-end encrypted messaging tools are currently at the centre of a powerful ‘double’ narrative. If on one hand the discourse on empowerment and better protection of fundamental civil liberties is very strong, several projects show in parallel a desire, or a need, to defend themselves from allegations of links to terrorism (
This paper is supported by a qualitative methodology and draws from the research conducted within the H2020 project NEXTLEAP
As for the present paper focusing on the Electronic Frontier Foundation, we started our research by consulting the appropriate documents, which will be extensively cited (Version 1.0 of the SMS, online discussions and blog posts discussing it). Once the ‘check back soon’ notice was made public by the EFF (see Figure
This article focuses on the negotiations around the technical properties selected as part of the categorization, seen primarily through the viewpoint of EFF members. In doing so, its primary focus is not on the Electronic Frontier Foundation as a ‘political economy actor’, although it should be acknowledged that the relationships of this non-profit organization vis-à-vis other similar organizations, the private sector, political institutions in the United States and the ‘crypto’ technical community have contributed to the selection of the particular set of encrypted secure messaging applications that made it to the SMS, and to what could be considered as an ‘over-inclusion’ of US-based companies, perhaps to the detriment of apps from other regions or language spaces.
As for ethical guidelines, a specific protocol was developed in order to protect the privacy of our respondents. We let users and developers suggest a secure tool of communication of their choice to us if they wished to do the interview online. The interview was recorded with an audio recorder isolated from the Internet. We used a dedicated encrypted hard drive to store the interviews. Before the interview we asked our respondents to carefully read two user-consent forms related to the study and ask all the questions regarding their privacy, their rights and our methodology. The two forms were written in collaboration with UCL usability researchers and based on the European General Data Protection Regulation. The documents included an Information Sheet and an Informed Consent Form. The first document explained the purpose of the interview, described the research project and clearly mentioned the sources of funding for the project; provided information on the length of the interview, but also information about the researcher, including her email, full name, academic affiliation and the address of the research institution. The second form described the procedures regarding data processing methods, the period and conditions of data storage; it emphasized the right of the interviewees to demand, at any moment, to withdraw their data from the research. A copy of each document was given to the interviewee. Different forms were used for users, trainers and developers. We respected the right of our interviewees to refuse answering a specific question and choose the degree of anonymization.
In November 2014, the Electronic Frontier Foundation released its Secure Messaging Scorecard. The SMS was announced as the first step of an awareness campaign aimed at both companies and users – a tool that, while abstaining from formal endorsement of particular products, aimed at providing guidance and reliable indications that ‘projects are on the right track’, in an increasingly complex landscape of self-labelled ‘secure messaging products’, in providing ‘actual […] security’.
The EFF closely links the initiative to the Snowden revelations, mentioning that privacy and security experts have repeatedly called on the public to adopt widespread encryption in recent years; however, Snowden’s whistle-blowing on governments’
It kind of came out of conversations at EFF […we] got a lot of queries with people asking which messaging tools they should use. So there have been a couple of projects in the past, like the guide ‘Who has your back’, a project that is a sort of scorecard of Terms of Service and how companies handle data, that’s where the idea came from… try to use the same approach to put information out there that we thought was useful about different messaging tools.
In the EFF’s view, adoption of encryption to such a wide extent ‘boils down to two things: security and usability’. It is necessary that both things go hand in hand, while in most instances, the EFF observes a trade-off between the two:
Most of the tools that are easy for the general public to use don’t rely on security best practices–including end-to-end encryption and open source code. Messaging tools that are really secure often aren’t easy to use; everyday users may have trouble installing the technology, verifying its authenticity, setting up an account, or may accidentally use it in ways that expose their communications.
Citing collaborations with notable civil liberties organizations such as ProPublica and the Princeton University Research Center on Information and Technology Policy, the EFF presents the SMS as an examination of dozens of messaging technologies
The Secure Messaging Scorecard Version 1.0.
The table includes intuitive symbology and colours to account for the presence or the lack of a requirement, and a filter giving the possibility to display only specific tools, or get a bird’s eye view of all the examined tools in alphabetical order.
Thus, as we can see, the political and ‘quasi-philosophical rationale’ behind the SMS is clearly presented. It is interesting to acknowledge, in this context, that the ‘Methodology’ section in the presentation page is very matter-of-factual, presenting the final result of the category and criteria selection, but very little information is shared by EFF on the thought processes and negotiations that led to the selection. The page states that
Yet, as researchers exploring the social and communicational dimensions of encryption from an STS perspective, we hypothesized that selecting and singling out these categories had been anything but linear, and the backstage of these choices was important to explore – not merely to assess the effectiveness of the SMS, but also as a way to explore the particular definition of encryption and security the EFF was inherently promoting as it was pushing the project forward. Indeed, it seemed to us that by fostering the SMS project, and due to its central role as an actor in the preservation of civil liberties on the Internet, the EFF was not merely acknowledging and trying to accumulate information on a ‘state of things’ in the field of encryption and security but it was contributing to shaping the field, the chosen categories to define ‘actual’ or ‘usable’ security being a very important part of
While the SMS’s main presentation page focused on the criteria in their final form, there were some indications of the EFF’s ongoing reflections on the ‘making of’ such criteria, and their presence on the Internet. An early November 2014 post by Chief Computer Scientist Peter Eckersley, ‘What Makes a Good Security Audit?’, acknowledged that the foundation had ‘gotten a lot of questions about the auditing column in the Scorecard’.
In addition, musings on the SMS came from several actors in the encryption/security community at large, some of them disputing the lack of hierarchy among the criteria, the phrasing of some of them, and even the messaging tools’ alphabetical ranking,
In early August 2016, these reflections seemed to have reached a turning point as the EFF ‘archived’ its SMS main page, labelled it a ‘Version 1.0’, announced it would be back in the near future with a new, improved and ‘more nuanced’ guide to encryption tools, and made an explicit reference to the ongoing revisions that had characterized and still were characterizing their categorization work: ‘Though all of those criteria are necessary for a tool to be secure, they can’t guarantee it; security is hard, and some aspects of it are hard to measure’
Secure messaging scorecard main page since August 2016 and as of February 8, 2017 (
And thus, Version 1.0 of the SMS was no more, except in the form of an archive preserved ‘for purely historical reasons’ of which the EFF itself discourages further use.
Indeed, the first SMS appears to have had a performative effect on the community: the EFF has a central role as a protector of online civil liberties, and setting up a scorecard in this field was a pioneering effort in its own kind. In an otherwise mostly critical discussion thread among tech-savvy users,
Interestingly, and despite the EFF’s announced intentions, we can retrace some early ambiguity and perhaps confusion amongst the SMS’s intended target audience. The simplicity and linearity of the grid, the symbols used, the way categories were presented – all these aspects could indeed lead the encryption technical community to think that it was aimed at both developers and users (confirmed earlier by R1, as well), and perhaps primarily at users. However, there are other indications that it might have been the other way around – the primary target being to foster good security practices among a wide number of developer teams, and usability being an eventual target. According to R2,
Ultimately, and despite the EFF’s warnings,
In the interviews we conducted with them, EFF members describe how, since the early days of the SMS’s first version, there has been an ongoing process of thinking back on its different categories, considering how they could be revisited, as well as the scorecard/‘grid’ itself as a device. Taking into consideration what actors in the encryption community considered ‘errors’ (shortcomings, misleading presentations, approximations, problematic inferences), and revisiting its own doubts and selection processes during the making of 1.0, the EFF team is currently analysing how the choice of these categories built towards specific understandings of encryption and security, considering how they could evolve – and with them, the definitions of ‘good encryption’ and ‘good security’ these categories present/suggest to the world. This third and last section addresses this ongoing process.
A first, fundamental level at which the reflection is taking place is the choice of the format itself. As this section’s opening quote by R1 shows, the feedback on the first version of the SMS has revealed that in the attempt to be of use to both developers and users, the scorecard might have ended up as problematic for both. But this time, says R2,
We are very likely to do the 2nd version of the scorecard but it is not a 100% sure thing at this point. And that’s connected to the fact that we are completely changing the way the scorecard is set up, we are definitely abandoning the sort of grid of specific check boxes […] A table seems to present cold hard facts in this very specific way. It is very easy to be convinced and it’s very official […] we are definitely going towards something more organic in that way, something that can capture a lot more nuance. […] there’s a lot more that makes a good tool besides six checkmarks.
Through the words of R1 and R2, we see how this intended additional ‘nuance’ is actually taking place not by eliminating categories, but by revisiting them. The v2 of the SMS, as currently envisaged, is supposed to divide the messaging tools in a set of different groups (R2 calls them ‘tiers’): the first group will include tools that are unconditionally recommended, and the last group is meant to convey an ‘avoid at all costs’ message (e.g. for those tools that have no end-to-end encryption and present clear text on the wire). Within each group the tools would again be presented alphabetically, and not internally ranked; however, each of them is going to be accompanied by a paragraph describing it in more detail, instead of visual checkmarks, to the specific benefit of users:
Interestingly, the EFF team plans to keep the first tier… empty. This has strong implications for the definition of good encryption and security – basically implying that it has not yet been achieved in the current reality of the secure messaging landscape, and as of yet, a mix of usability and strong security is still an ideal to struggle for:
There’s nothing there because every app has some sort of a problem. WhatsApp has this issue of sharing data with Facebook so depending on your threat model is not great, Signal we thought a lot about reliability problems particularly outside the US, so it’s not a great option. Those are scenarios where we’d say, those are maybe the best from what we can recommend, so they end up in ‘adequate secure messaging tools’ which is the next tier down, because they are the best tools that exist but we still want to emphasize to people that it’s not 100% NSA-proof and you will never be safe for life, because we think that’s overselling them.
In passing, the EFF team is defining what according to them is a ‘perfect’ secure messaging tool
The graphic and spatial organization of the second version of SMS radically differs from 1.0’s table. First of all, moving from a table to a list implies a different way of working with the information and undermines the idea of a direct, linear and quantified comparison offered by the table. A list of tiers with nuanced descriptions of different apps and their properties offers room for detailed and qualitative explanations, while tables tend to put different tools on the same surface, thus creating an illusion of commensurability and immediate quantified comparison. As R2 says:
It’s definitely gonna be a list-like, it’s definitely not gonna be a table. We may or may not allow people to filter on some of these criteria. For example if you wanna see just iPhone ones we may put a little button where you can click on the top and only show only iPhone tools or only show tools that use Axolotl or something like that if you wanna to. But it will be more like a list: here is the first group, here’s the next group, here’s the next group, here’s the group you should never use.
The idea of ‘filters’ adds a certain degree of user agency in the classification of tools: the lists become modulable as users may set up a criteria that could graphically reorganize data providing cross-tier comparisons. This offers a different way of both classifying the data and ‘making sense of it’, than a table. The latter is, as Jack Goody puts it, a graphically organized dataset with hard structure leaving little room for ambiguity (
Another problem posed by the grid format, that the new version of the SMS seeks to address, is the projected equivalence of the different criteria, from the open source release of the code, to its audit, to the type of encryption. The ‘checklist’ and ‘points’ system creates the impression of an artificial equality between these criteria, while in fact, the presence of some rather than others has different impacts on security and privacy, in particular for users in high-risk contexts:
[If you were] someone who does not know anything about crypto or security, you would look at this scorecard and would say, any two tools that have, say, five out of seven checkmarks are probably about the same. When if… one of those tools actually had end 2 end encryption and the other did not, even though they both had code audit or the code was available, or things like that, their security obviously isn’t the same. That was basically the flaw we found, the flaw that existed in using
The EFF team also realizes that some developers and firms proposing secure messaging tools have, while presenting their tools, bent the grid to their own advantage – more precisely, they have presented a high conformity to the SMS as a label of legitimacy. Again, R2 emphasizes that this is a problem particularly in those cases when users lack the technical background or expertise to build their own hierarchy of the criteria’s relative importance, according to their needs or threat model (see 4.2):
[There] were tools that just said on their website that say we got 5 out of 6 on EFF SMS so we are great […] some of the tools tried to sneak advantage out of it, and I can’t really blame them because it was designed in this way. […] For many users […] Even if you know what end2end means but you don’t necessarily know… you’re not in this realm… is having a code audit more important than, say, forward secrecy? […] the answer of course is that it depends on your “threat model”. But there’s nothing about the threat model or what you’re protecting against in the [1.0] scorecard, which is part of the problem.
A piece of recent research on user understanding of the SMS, undertaken by Ruba Abu-Salma, Joe Bonneau and other researchers with the collaboration of University College London, shows that indeed, users seem to have misunderstood the scorecard in several respects. Four out of seven criteria raise issues: ‘participants did not appreciate the difference between point-to-point and E2E encryption, and did not comprehend forward secrecy or fingerprint verification’ (
The additional expertise needed by the user to understand the difference between various criteria and their importance emerges as a crucial flaw of the v1 grid. One of the keys for a ‘new and improved’ SMS, thus, seems to be the fact of taking user knowledge seriously, as a cornerstone of the categorization: for it to be meaningful, users must identify and analyse their respective ‘threat model’ – i.e. identify, enumerate and prioritize potential threats in their environment, in this case digital environment. R3 remarks that this is one of the core objectives the EFF in several of its projects beyond the revision of the SMS – and notes that there are no direct tools to indicate which threat model one has, but users need to uncover the right indicators in their specific contexts of action:
We’re not answering what someone’s threat model is, we just help guide them in their direction of what to read. We can say like journalists might have good secure communication tools that they might wanna protect their data, but we can’t say how much of a threat any journalists are under because different journalists have different threats.
R2 points out that the same words, used to identify a particular threat, might have very different meanings or implications depending on the geopolitical context – the more ‘qualitative’, descriptive part of the new tier-organized SMS should be useful to trigger the right reflexes in this regard:
It’s harder to say in 3 words what is your threat model. […] even if it’s easy to distinguish a threat model “I just want more privacy” versus “I am worried of a state-level actor”, there’s still a difference between “I am worried of a state-level actor in Syria” vs. “I am worried of a state-level actor in Iran” versus China versus US… Am I worried about all of them or a particular actor? I think it’s a lot harder to capture it in the grid. We’re hoping that tiers will capture that to some degree.
There is no universally appropriate application, so the new SMS should take the diversity of the users – and the corresponding diversity of their threat models – as a starting point. R2 again resorts to specific case-examples to illustrate this point:
There’s also totally different types of people who are coming to this thing. Some of them are Ukrainian journalists working in a war zone, and some people are hipster San-Francisco wealthy middle-class people in the US who don’t really have to worry about the safety of their lives being dependent on their messages, but still wanna do something, still value privacy, and we might recommend Signal for one and WhatsApp for the others because they’re already on Facebook so it does not really change anything for them, Facebook already knows everything about them, so it already has their contacts. So including this information and being able to tell people look you don’t really have to completely change your life if your threat model isn’t demanding it, you can use this tool or here’s the slight change you can do. I think it’s helpful, it makes more sense. Not everyone has to put a tin foil hat and create an emergency bunker. Lots of people do, but not everybody. Tailoring it to the right people. I think that would be great to have an app that we would recommend to everyone because it’s usable and easy, popular and secure and everything else but since it’s not there I think it’s useful to tailor things, tailor the threat model.
If, for the user, the choice of a strong secure messaging tool is in this vision strictly linked to the understanding of his or her threat-model, for its part the EFF acknowledges that if there is no universally appropriate application, so it goes for the definition of what constitutes ‘good’ encryption – ‘good’ security and privacy. Beyond the strength of specific technical components, the qualities of being ‘secure’ and ‘private’ extend to an appreciation of the geographical and political situation, of the user’s knowledge and expertise of whether privacy and security are or are not a matter of physical and emotional integrity, which can only be contextually defined and linked to a particular threat model. A high-quality secure messaging tool may not necessarily always be the one that provides the strongest privacy, but the one that empowers users to achieve precisely the level of privacy they need.
The efforts to revise the SMS cannot, for the EFF team, do without a comparison with other categorization systems. On the one hand, the new version of the SMS will interact with the Surveillance Self-Defense Guide, developed by the EFF itself and designed to aid in ‘defending yourself and your friends from surveillance by using secure technology and developing careful practices.’
Sample of ‘user paths’ on the SSD home page (
According to R3, this approach based on facilitation, induction and personalization informs more broadly the recent EFF efforts, and goes back to identifying the right level of relative security for the right context:
‘We don’t distinguish threat models, we give tools to help users figure out what are their threat models. […] I still would not say we were putting an answer to the question out there. The key to the guide that we’ve created is that we want people to start with
The EFF is also looking at categorization systems in the same field produced by other actors – acknowledging that SMS 1.0 has been, in turn, an inspiration for some of them and that there seems to be a need, generally identified in the field, of tools providing guidance in the increasingly complex landscape of secure messaging.
The alternative, user-centred approach arises not only as a result of internal reflections on the first version of SMS, but also because of how parallel categorization attempts were done by other actors promoting online civil liberties. Our recent interview with an informational security trainer also shows a turn in pedagogical methods from tools to threat model evaluation:
A final set of evolutions moves from the building of the categorization system itself to how proof of the tools’ conformity to the guidance provided will be requested by EFF and provided by the developers of the tools, and the encryption/security community at large. Indeed, part of the early criticisms of the first SMS did not have to do with the format of the grid, but were due to the opacity of the ways in which its ‘binary’ recommendations were evidence-supported.
In SMS 1.0, ‘green lights’ had sometimes been awarded for specific criteria as a result of the private correspondence between R1 and the developers, says R2:
In particular, one of the criteria that had raised more objections in terms of provided evidence was the review of the code (‘What does “security design properly documented” even mean?’, a developer had commented).
Finally, to support the arguments provided in the new SMS, the EFF team hopes to build on the competencies of the encryption community of practice – cryptographers, professors, people in industry and trainers. R2 remarks:
This article has sought to examine the role of categorization and classification attempts in both the first and the foreseen second version of the SMS. In doing so, it has analysed how, by challenging, re-examining and re-shaping the categories that are meaningful to define the quality of secure messaging tools, the EFF has sparked a ‘global crypto discussion’
Indeed, on one hand, the EFF’s activities, epitomized by the SMS and its revisions, seem to contribute to the ‘opportunistic turn’ in encryption (
However, while calling upon developers for improved usability – demanding that the technical crypto community make some properties, such as key verification, easy for users and independent from user agency – the EFF puts users at the core of the new categorization system, and in doing so, entrusts them with an important decision-making responsibility. To put it in R2’s words,
In the end, according to R1, a ‘good’ secure messaging tool
E.g. the discussion of the code audit criterion at Peter Eckersley, ‘What Makes a Good Security Audit?’, EFF Deeplinks, 8 November 2014,
Borrowing from the conclusions of a survey conducted within the first year of the NEXTLEAP project, partly summarized in (
Communication system where only the communicating users can read the messages, thanks to encryption keys only known to them.
The full list of projects, methodology of interviews (including questionnaires), as well as key findings from these interviews are presented in a dedicated paper (
More detailed explanations of our ethical guidelines are available in (
Ibid.
And we will come back to them extensively.
Eckersley, supra note 2.
Ibid.
Discussion thread on Hacker News,
Daniel Hodson & Matt Jones, ‘EFF secure messaging scorecard review’, ELTTAM blog, 11 August 2016,
Zelijka Zorz, ‘How the EFF was pushed to rethink its Secure Messaging Scorecard’, HelpNetSecurity, August 11, 2016,
Cf.
This was also brought up in the Hacker News SMS-related thread: ‘Isn’t it a bit strange, that there is no such thing as that scoreboard produced by an international group of universities and industry experts, with a transparent documentation of the review process and plenty of room for discussion of different paradigms?’ (see
Ibid.
See
Ibid.
This work is supported by the European Union’s Horizon 2020 Framework Programme for Research and Innovation (H2020-ICT-2015, ICT-10-2015) under grant agreement nº 688722 – NEXTLEAP.
The authors have no competing interests to declare.