Quality of Service Measurement Project – MoQoS

by

The project MoQoS (Measurement of Quality of Service) facilitated cooperation among national telecom regulators across Europe. It did so by creating, publishing and reusing open data related to the Quality of Service of broadband Internet. Connecting Europe Facility of the European Union co-financed the project.

The project was based on the principle of enhanced cooperation according to the Article 20, 42(6), 46 of the Treaty on European Union, and the Article 326 – 334 of the Treaty on the Functioning of the European Union and the “Memorandum of Cooperation in the field QoS and spectrum management”, initiated by the Czech Telecommunication Office.

Specure’s contribution

Specure GmbH developed and implemented a platform for Quality of Service (QoS) measurements of mobile and fixed-line broadband internet connections. Likewise, the project partners, national regulators from Slovakia, Czechia and Slovenia, all contributed to the solution specification. The features and methodology are also in line with BEREC recommendations regarding QoS measurements. Further, the solution is based on the open-source platform Open Nettest. In particular, it includes a web page, mobile apps for the most prevalent mobile platforms – iOS and Android; and a server-side software. In short, the solution provides benefits to various stakeholders’ groups, including the National Regulatory Authorities from the participating countries: Slovakia, Czech Republic and Slovenia; and national regulators from other European and CEPT member countries.

Press coverage

Awards and Recognisions

Government award for project MoQoS
2nd price in E-Government Competition in Czech Republic (2017)

Project MoQoS won the second place in e-government The Best 2017-ICT competition. The crowdsourcing project developed in partnership also with the Czech Telecommunication Office succeeded in category of central projects.

Activities

Activity 1: Project Management

Project management included series of activities leading to a successful implementation of the project. In particular, the project management at the project partners level included the financial management, human resource management, risk management, and quality management.

Project meetings

Given the size of the consortium, project meetings represented a significant tool of the project management. In total, we held 7 meetings during the project, 2 of which were on-line:

  • Kick-off meeting, 26–27 January 2017 in the Czech Republic (Prague),
  • Teleconference 1: 28 March 2017,
  • 15–16 June 2017 in Austria (Mayerhofen),
  • 14–15 November 2017 in Slovenia (Jeruzalem),
  • Teleconference 2: 11 January 2018,
  • 19–20 June 2018 in Slovakia (Štrbské Pleso) and
  • the final meeting on 20 December 2018 in Vienna (Austria).

Activity 2: Dissemination and Deployment

The main abjective of this activity was the marketing of the project and its results. We aimed these activities at three key target groups – End users, National Telecom Regulators and European public bodies. Each of these target groups needed its own communication strategy and own communication measures.

At the beginning of the project, the project members decided to create the project acronym MoQoS. It stands for Measurement of Quality of Service (of the broadband internet). The project partners also developed a project logo, new website and created marketing materials (i.e. leaflets). We prepared and distributed a regular newsletter to communicate the project news and achievements.

MoQoS conferences

The project members talked about MoQoS at various conferences, including:

  • Internet in State Administration and Local Government (ISSS – Internet ve státní správě a samosprávě),
  • the Annual Conference Open Data in Public Administration (Výroční konference otevřená data ve veřejné správě),
  • OPEN DATA EXPO and
  • EMEA CTO Telecoms Forum.

The project was also presented in a range of media, including the Czech Television, printed journals and blogs. The project partners also published several press releases. The journals that published articles aout MoQoS included

  • Telecompaper (7 articles),
  • Respekt (wit a special insert dedicated to open data) and
  • School Management (Řízení školy).

The Czech Telecommunication Office has repeatedly talked about the project in its Monitoring Report.

End Users

One of the ideas behind crowdsource measurement is to empower the end-users. So, we focused the communication activities towards the end-users mainly on marketing of mobile apps. We addressed this group during talks at conferences and in articles, both online in printed media.

Other stakeholders

To address the other stakeholders’ needs, we organized three stakeholders meetings:

The project members and other National Regulatory Authorities (NRAs), non-members of the project, were the main participants in these conferences. We saw NRAs from Austria, Serbia and Norway participating.

Targeting NRAs, we presented the project at the 2nd Meeting of the “BEREC Contact Network” and at the “Evolving Regulatory Frameworks for Digital Future” organized by ITU.

The last stakeholders meeting was at the same time the final project-closure meeting.

The community events

Czech Supreme Audit Office and the Czech Telecommunication Office hosted several hackathons which among others also utilised the open data from the project.

Public bodies responsible for open data in European countries

In our communication, we targeted the national bodies responsible for open data, including the institutions responsible for the operation of the national open data portals. We presented the project conference as an example of best practice in cross-border cooperation on several occasions. This includes the Open data Expo in March 2017 and the National Open Data Conference in November 2017 in Prague.

Deployment

The project strived for closer cooperation with similar projects and initiatives. In addition, we also aimed for the active usage of project data. Therefore, we established cooperation with the project „Mapping of fixed and mobile Broadband Services in Europe“ (SMART 2014/0016). This initiative used the data from project MoQoS as one of its sources. MoQoS project representatives participated in several stakeholders meetings of this project.

The core software modules from project MoQoS were re-used in the „Standard connectivity“ tool. This tool was developed in cooperation with the Czech Ministry of Regional Development. It was used for monitoring the internet parameters at selected beneficiaries of structural funds, such as schools. It enabled an easy verification of the selected conditions. In return, the obtained data on the internet connection quality extended the resulting data set of our project. 12 067 tests were run at more than 2 000 schools (organisations) during the project in total.

Activity 3: Setting-up of European standard for open-data related to QoS of high-speed Internet

The Czech Telecommunication Office sought close cooperation with key entities in the standardization field. These include BEREC (The Body of European Regulators for Electronic Communications), ETSI (European Telecommunications Standards Institute) and ITU (International Telecommunication Union).

The cooperation with BEREC was crucial. We had to consider several important documents and recommendations during the project implementation. One of them were the Guidelines on the Implementation by National Regulators of European Net Neutrality Rules (BoR(16)127). These Guidelines worked out the requirements arising from the EU Regulation No. 2015/2120.

In cooperation with the NRAs, Berec published the NetNeutrality Regulatory Assessment Methodology (BoR (17)178) and Net neutrality measurement tool specification (BoR (17)179). The members of the managing team of the MoQoS project from CTU and AKOS actively contributed to drafting the above-mentioned documents and thus contributed towards the future harmonization of the QoS and QoE measurements at a European level. The project has also carried out an analysis of similar applications used by NRAs in other European countries.

Standardisation

The standardization activities and cooperation with BEREC directly influenced especially the tools for obtaining crowdsourcing data and the measurement methodology, which implements the BEREC´s recommendation in the area of net neutrality. The standardization and harmonization activities had an impact on the output data, mainly the unification of the licences in accordance with the Commission notice 2014/C 240/01 “Guidelines on recommended standard licence, datasets and charging for the reuse of documents“. All the project partners, as well as the cooperating NRAs from Austria and Serbia, agreed on using the CC BY 4.0 licence.

Activity 4: Developing mobile app for crowdsourcing data

In order to ensure data collection with the help of end users (crowdsourcing) the project partners developed the Open Nettest application, which is available for both Android and iOS operating systems. We localised the apps into more than 10 languages. The national telecommunication regulators created their own, branded versions of the apps.

Open Nettest mobile app and its localization in the Czech Republic, Slovakia and Slovenia

Open Nettest allows users to obtain information on the current QoS of their Internet connection in the context of net neutrality and provides them with comprehensive information, including statistical data. The measurement methodology encompasses over 70 unique tests with more than 80 parameters measured (e. g. TCP-/UDP-ports, a VOIP/jitter test, proxy tests, DNS tests). The methodology of verification of individual parameters and the overall scope is then in full compliance with the BEREC guidelines and recommendations, including the parameters of IAS quality monitoring system.

Client features

The application for iOS and Android itself offers its users the possibility to run the test as well as other useful features, e.g. the history of measurement including detailed results, a synchronization of measurements between two different devices (e.g. a browser and a mobile device), and a well-arranged map with the measurement results of other users. The obtained data (test results) are in accordance with the personal data protection regulation (esp. GDPR) anonymized. The user has to understand and accept the usage of their privacy data. In addition, we do not publish the following: Measure ID of the client, sync code and sync group of the clients, the complete IP address, Wi-Fi network name (SSID) and Wi-Fi numeric identifier (BSSID).

The web application is implemented in HTML5 using Javascript with Websockets. It is intended as an alternative to mobile applications, primarily for the users of personal computer/notebooks. Thanks to the properties of the technology used it is not possible to send UDP-packets. Especially the measurement for Delay variation (sometimes called Jitter) and packet loss will make sense only when using UDP. For that reason, these two parameters cannot be measured in browsers.

Apps key metrics

By 31 December 2018, more than 85.000 users downloaded the Open Nettest mobile app. In addition, the users downloaded the national instances of the app more than 30,000 times. More than 2,000 users activated the QoS measurements on their TURRIS routers. To sum up, a total of more than 115,000 users participated in the crowdsourcing data collection during the project. In addition, users in practically all EU members states downloaded and used the free Open Nettest mobile app.

Activity 5: Developing framework for crowdsourcing data from fixed-line

Hardware probes

In the project, we wanted to provide tools for quality of service measurement regardless the access technology used. Therefore, we developed, tested and deployed a tool for fixed line measurements too. The tool is based on a router TURRIS. Thanks to its release under an open licence (GPLv3), the tool may be used on other hardware with the OpenWrt operation system. The source code is available at the CZ.NIC´s GitLab.

Turris Omnia router, V2017

The measuring of individual parameters of QoS itself is based on the methodology used in a mobile app and the web test. Based on the communication with NRAs, and with the Czech Telecommunication Office in particular, this tool offers sufficient possibilities to verify basic parameters of the line in relation to consumer rights based on the EU Regulation No. 2015/2120. From this point of view, mainly relevant for the users are the upload and download speed, ping, name of ISP and the technology used. In the web interface of the router, each user may set times to run tests on a scheduled basis.

Turris Omnia – Configuration interface

Open Data

There were more than 2 000 users participating in the pilot testing, who took 1,811,878 tests in total from the introduction of the tool in November 2017 until the end of the project (31 December 2018). The success of the pilot project is proved by the statistic, according to which the users performed 4.6 tests per second on average.

Subsequently, we published the anonymized open data data obtained from the user measurements on the Czech national open data portal. From here, the data are further harvested to the European open data portal.

Part of the project was also customizing the hardware probes (the TURRIS routers) and the development of the script used for professional measurements by the NRAs, mostly in trains and in measuring cars. Therefore the project was also able to obtain data for professional measuring with the aim of e.g. verification of fulfilling the conditions the individual operators promised during auction bids.

Based on the evaluation of the pilot project, the project partnerts will further development of the tool in cooperation with the NRAs..

Activity 6: Data publication, aggregation and visualisation

The objective of this activity was to make available the data from the measurements at different levels, taking
into account the different type of users and their interests.
One of the main aims of the project was to make accessible the obtained data to both consumers and the professional users.
All the project partners, as well as the cooperating NRAs from Austria and Serbia, agreed on using the CC BY 4.0 licence for publishing their respective countrz data.

End users

The interactive map, with its visualisation of aggregated data, is intended mainly for end users. They may easily verify the quality of mobile broadband coverage in a specific location in this map. The map uses as a source the data created in mobile apps of the individual NRAs (i.e. Czech Republic, Slovakia and Slovenia) and also the data from the Open Nettest mother application. Besides the data obtained from project partners the map uses also sources from other partner applications, namely the Serbian Ratel and Austrian RTR. The data originating in national measurements are then at disposal at the respective pages of individual national tools, i.e. NetMetr in the Czech Republic, Merač Internetu in Slovakia and AKOS Test Net in Slovenia. As the users often measure data also during their stay abroad it is possible to obtain measurement data from more than 100 countries of the world.

Open Nettest – Map View

Profesional users

For professional users who want to reuse the obtained data are intended the machine readable data published in non-proprietary open format. These data are published either on web pages of the individual national authority (e.g. NetMetr, Merač Internetu, Akos Test Net) or on an open data portal. Czech Telecommunication Office publishes the relevant data set on the Czech national open data portal, from which they are later harvested by the European open data portal.

MoQoS Project Data on the European Data Portal

Activity 7: Data analysis

This activity focused on the analysis of the data obtained via the crowdsourced measurements. The analysis of the data would enable experts from the national regulatory authorities and other interested parties (Internet Service Providers, analysts, researchers and even end-users) to evaluate the quality of the Internet Access Services and monitor different aspects that influence their performance. The data console developed during the project shows the potential of this type of solutions.

MoQoS Data Source

During the project it became obvious that comparing data from different sources is not an easy task. Even if it is possible to normalise the data can be to standard performance and quality parameters, such as download speed, upload speed, latency, jitter and packet loss for instance, the methodology of collecting and presenting the data varies significantly by source.
Very few sources publish raw data directly produced by the measurement tool. Mostly, the data is normalised and aggregated. Different tools may use different normalisation metrics (e.g. download speed between 2Mbit and 10Mbit) and aggregation scale (NUTS3 level, squares 100 by 100m).

MoQoS Console

MoQoS Data Console is a tool that enables in-depth analysis and visualization of QoS broadband data. During the project, we analysed data provided from various country portals that use Nettest-like, crowdsourced methodology.

MoQoS Console Visualisation

The major challenge was to import data from heterogenous sources. This was a challenge, since the data structure was similar but not exactly the same for all the sources. We designed the MoQoS console in a way that it is also possible to import data from third party sources. This, however, was not in the scope of the project.

MoQoS filters for reports

The MoQoS Console allows for visualizing of both basic and QoS measurement data, applying various geographical and time & date filters. The users can filter data and the resulting reports by countries and NUTS regions. Trends over time are easily recognizable using the chart widgets on the overview screen.

White Spots Visualisation

Based on the suggestion of AKOS Slovenia, we added the “White spots” as a visualisation option. This functionality enables to monitor the broadband QoS in rural areas, or other regions with sub-standard mobile internet coverage. AKOS defined several such regions in Slovenia; and these are displayed on the map with distinct borders. If there are enough measurements, the corresponding colouring indicates the quality of the broadband internet in that area.

White Spots on the map of Slovenia

Conclusion

The project created a cross-border platform for collecting, publishing and reusing open data about the quality of high-speed Internet. The national telecom regulators from three European countries – the Czech Republic, Slovakia and Slovenia – cooperated closely during the project.

In order to obtain data from end users, Specure and Martes developed an open-source mobile application Open Nettest. We localised the app into several European languages. Project members created their own branded version of the apps: Czech Republic (NetMetr), Slovakia (Merač Internetu) and Slovenia (AKOS Test Net). We used TURRIS routers and the corresponding web interface by CZ-NIC for measurements of fixed lines and WiFi networks.

The following schema illustrates the various data sources.

Schema of data collection

Publication of data

Publication of data in the form of open-data supporting their future use is ensured by individual partners. A key part of the project wa publishing a data set on the European Data Portal. This data set is harvested from the Czech national portal of open data, which takes aggregated data obtained from individual national applications, the Open Nettest app and at the same time from collaborating national regulators from Norway and Serbia. By 31 December 2018 the Open Nettest data set contained in more than 2,500,000 entries, representing five-times more over the pre-defined goal of 250,000 tests. The visualised data allowing end users an easy check of the quality of the coverage at a given place are available in an interactive map on Open Nettest Portal and also on web pages of the individual national instances (NetMetr, Merac Internetu, AKOS Test Net).

Supporting European policies and cross-border collaboration

The key for the project´s implementation was the EU Regulation (EU) 2015/2120 of the European Parliament and of the Council. It laid down measures concerning open internet access and related recommendations by BEREC. The mobile app performs a total of more than 70 various QoS tests based on the BEREC recommendations.

Contributors

For the list of project contributors and copyright owners, please refer to the following URLs:

Project summary

Customer: European Union
Date of realisation: 01/2017 – 12/2018

Quality of Service project Co-financed by Connecting Europe Facility of the European Union
Co-financed by Connecting Europe Facility of the European Union