An International Workshop on Competitions, Usability, Benchmarks, Evaluation, and Standardisation for Runtime Verification Tools
Held in conjunction with the 17th International Conference on Runtime Verification (RV 2017)
The goal of this workshop is to provide a venue to discuss ongoing efforts to improve how we evaluate and compare tools for runtime verification. This workshop invites submissions that contribute to this discussion (see below). The workshop replaces the Runtime Verification Competition this year, giving an opportunity for reflection and planning for the future.
- We invite two kinds of submissions: tool overview papers of existing tools and position papers
- Attendance at the workshop is not compulsory for submission, but encouraged
- The workshop will be integrated into RV 2017 to engage with the wider RV community
Over the last three years, beginning in 2014, the Competition on Runtime Verification (CRV) has compared 14 different runtime verification tools using over 100 different benchmarks. It has motivated the development of new tools and extensions of existing ones. Thanks to the time and effort of the various organisers and participants it has successfully provided a platform to discuss how we evaluate and compare our tools. However, we are aware that the competition has not served all members of the community and required significant effort from its participants and therefore we have decided not to run the competition in the same form in 2017.
This workshop has two aims.
- Firstly, to kickstart an online repository of tool descriptions that focus on usability, rather than implementation. Much effort within Runtime Verification is spent developing tools. The planned repository will help advertise these tools within and beyond the community and to document their existence. To this end, this workshop invites tool overview papers (details below) that present an opportunity to demonstrate the diverse range of available RV tools and to advertise their advanced features and capabilities.
- Secondly, to provide a forum in which the future of the competition, and wider efforts to improve how we evaluate RV tools, can be discussed in a structured and informed way. There have been various informal discussions at the previous three RV conferences and in other settings. By inviting position papers (details below) this workshop aims to establish the issues and possible directions before holding a structured panel session during RV 2017.
RELATION TO RV 2017
There is no overlap in scope between this workshop and RV 2017. Any papers containing original technical developments should be submitted to RV 2017 rather than this workshop as the workshop focuses on tool reviews (containing existing work) and position statements. If there is any uncertainty please contact the workshop chairs.
The workshop will be integrated into RV 2017. The workshop activities will include a poster presentation session and a panel decision, both of which will be scheduled within the main conference program. A consequence of this is that there will be no separate registration for the workshop. To attend the workshop activities it will be necessary to attend RV 2017. Although, attendance is not a requirement for submission.
We invite two forms of submission: Tool Overview papers and Position papers. All submissions will be subject to a lightweight review by the PC to ensure a reasonable standard and to provide constructive feedback to improve the quality of the submission.
Tool Overview Papers
These should describe an existing tool using a minimum of 5 pages. The paper should at least describe how to obtain the tool, the RV problem the tool is aiming to solve, the key defining features of the tool and relevant references. Additionally, we might expect it to include some of the following:
- The history of the tool
- High-level overviews of key aspect such as the input language or architecture from a user’s perspective
- Usability details undocumented elsewhere
- Examples demonstrating key features
- Details of case studies or applications to real world problems
- Discussion of features particular to the tool
- Summaries (rather than detailed tables) of experimental results
- Analysis of the kinds of problem the tool is suited for and those it is less-well suited for
Ideally, the tool and related material (e.g. benchmarks) will be available online and linked to in the submission. The paper should not cover any significant new contributions (these can be submitted as tool papers to RV 2017) and should rely on previous research and tool papers to provide further details. Note that there is no upper page limit however the number of pages used should reflect the level of detail given. Submissions of 2 pages giving minimal details would be suitable in a situation where the tool is well documented elsewhere but an entry in the repository is still desired. All accepted submissions will be invited for presentation at a special Poster session during RV 2017.
Initial submissions should use a minimum of 2 pages to explore a particular position related to the evaluation, comparison or standardisation of Runtime Verification tools (and benchmarks). Topics may include, but are not limited to:
- What should a RV benchmark look like?
- Can we have a common specification language for RV? If so, what should it look like?
- Is execution time the most important performance criteria? What might be more important?
- How can we evaluate hardware monitoring tools?
- What are we doing wrong in evaluation? Can we fix this?
- What can be borrowed form other communities?
A selection of position papers will be used to structure a discussion panel to be held at RV 2017. Again, there is no upper page limit however the number of pages used should reflect the level of detail given. There will be an opportunity to update the paper based on discussions before inclusion in post-proceedings (see below).
Publishing and Submission
Contributions will be published as a post-proceedings volume in the Open-Access Scopus-Indexed EasyChair Kalpa Series. To be eligible for inclusion in the post-proceedings, papers should be at least 5 pages. Please see the relevant information for authors when preparing your paper.
Please submit contributions to the following EasyChair page.
- Submission Deadline 1 July 2017
- Abstracts: 8 July 2017
- Final Submission: 15 July 2017
- Notification 1 August 2017
- Workshop at RV 2017 13-16 September 2017
- Post-proceedings deadline 14 October 2017
Tool Overview Papers
- Sylvain Hallé and Raphael Khoury. Event Stream Processing with BeepBeep 3
- Christian Colombo and Gordon Pace. Runtime Verification with Larva
- David Basin, Srdjan Krstic and Dmitriy Traytel. Aerial: Almost Event-Rate Independent Algorithms for Monitoring Metric Regular Properties
- Shaun Azzopardi, Christian Colombo, Jean Paul Ebejer, Edward Mallia and Gordon Pace. Runtime Verification using Valour
- Julien Signoles, Nikolai Kosmatov and Kostyantyn Vorobyov. E-ACSL, a Runtime Verification Tool for Safety and Security of C Programs (tool paper)
- Nicolas Rapin. ARTiMon Monitoring Tool, The Time Domains
- Bernd Finkbeiner, Peter Faymonville, Hazem Torfah, Sebastian Schirmer and Maximilian Schwenger. Lola: A Tool for Stream-based Monitoring
- Christian Drabek and Gereon Weiss. DANA – Description and Analysis of Networked Applications
- David Basin, Felix Klaedtke and Eugen Zalinescu. The MonPoly Monitoring Tool
- Wei Dou, Domenico Bianculli and Lionel Briand. TemPsy-Check: a Tool for Model-driven Trace Checking of Pattern-based Temporal Properties
- Kristin Yvonne Rozier and Johann Schumann. R2U2: Tool Overview
- Sylvain Hallé, Raphael Khoury and Sebastien Gaboury. A Few Things We Heard About RV Tools (Position Paper)
- Julien Signoles. Online Runtime Verification Competitions: How To Possibly Deal With Their Issues (position paper)
- Domenico Bianculli and Srdjan Krstic. On the Risk of Tool Over-tuning in Run-time Verification Competitions (position paper)
- Svetlana Jaksic, Martin Leucker, Dan Li and Volker Stolz. COEMS — open traces from the industry
- Kristin Yvonne Rozier. On the Evaluation and Comparison of Runtime Verification Tools for Hardware and Cyber-Physical Systems
A separate program for RV-CuBES can be found here.
For local and general organisation please see RV 2017.
Program Committee Chairs
Giles Reger, University of Manchester, UK
Klaus Havelund, NASA Jet Propulsion Laboratory, USA
Ezio Bartocci, TU Wien, Austria
Domenico Bianculli, SnT Centre – University of Luxembourg, Luxembourg
Borzoo Bonakdarpour, McMaster University, Cananda
Christian Colombo, University of Malta, Malta
Ylies Falcone, Univ. Grenoble Alpes, Inria, Laboratoire d’Informatique de Grenoble, France
Adrian Francalanza, University of Malta, Malta
Sylvain Hallé, Université du Québec à Chicoutimi, Canada
Felix Klaedtke, NEC Europe Ltd.
Daniel Thoma, University of Lübeck, Germany
Dmitriy Traytel, ETH Zürich, Switzerland
Gordon Pace, University of Malta, Malta
Cesar Sanchez, IMDEA Software Institute, Spain
Leonardo Mariani, University of Milano Bicocca, Italy
Julien Signoles, CEA LIST, France
Tarmo Uustalu, Tallin University of Technology, Estonia