An International Workshop on Competitions, Usability, Benchmarks, Evaluation, and Standardisation for Runtime Verification Tools
Held in conjunction with the 17th International Conference on Runtime Verification (RV 2017)
The goal of this workshop is to provide a venue to discuss ongoing efforts to improve how we evaluate and compare tools for runtime verification. This workshop invites submissions that contribute to this discussion (see below). The workshop replaces the Runtime Verification Competition this year, giving an opportunity for reflection and planning for the future.
- We invite two kinds of submissions: tool overview papers of existing tools and position papers
- Attendance at the workshop is not compulsory for submission, but encouraged
- The workshop will be integrated into RV 2017 to engage with the wider RV community
Over the last three years, beginning in 2014, the Competition on Runtime Verification (CRV) has compared 14 different runtime verification tools using over 100 different benchmarks. It has motivated the development of new tools and extensions of existing ones. Thanks to the time and effort of the various organisers and participants it has successfully provided a platform to discuss how we evaluate and compare our tools. However, we are aware that the competition has not served all members of the community and required significant effort from its participants and therefore we have decided not to run the competition in the same form in 2017.
This workshop has two aims.
- Firstly, to kickstart an online repository of tool descriptions that focus on usability, rather than implementation. Much effort within Runtime Verification is spent developing tools. The planned repository will help advertise these tools within and beyond the community and to document their existence. To this end, this workshop invites tool overview papers (details below) that present an opportunity to demonstrate the diverse range of available RV tools and to advertise their advanced features and capabilities. This will be hosted at https://www.rv-competition.org.
- Secondly, to provide a forum in which the future of the competition, and wider efforts to improve how we evaluate RV tools, can be discussed in a structured and informed way. There have been various informal discussions at the previous three RV conferences and in other settings. By inviting position papers (details below) this workshop aims to establish the issues and possible directions before holding a structured panel session during RV 2017.
RELATION TO RV 2017
There is no overlap in scope between this workshop and RV 2017. Any papers containing original technical developments should be submitted to RV 2017 rather than this workshop as the workshop focuses on tool reviews (containing existing work) and position statements. If there is any uncertainty please contact the workshop chairs.
The workshop will be integrated into RV 2017. The workshop activities will include a poster presentation session and a panel decision, both of which will be scheduled within the main conference program. A consequence of this is that there will be no separate registration for the workshop. To attend the workshop activities it will be necessary to attend RV 2017. Although, attendance is not a requirement for submission.
We invite two forms of submission: Tool Overview papers and Position papers. All submissions will be subject to a lightweight review by the PC to ensure a reasonable standard and to provide constructive feedback to improve the quality of the submission. All accepted papers will be archived on the https://www.rv-competition.org website.
Tool Overview Papers
These should describe an existing tool using a minimum of 2 pages. The paper should at least describe how to obtain the tool, the RV problem the tool is aiming to solve, the key defining features of the tool and relevant references. Additionally, we might expect it to include some of the following:
- The history of the tool
- High-level overviews of key aspect such as the input language or architecture from a user’s perspective
- Usability details undocumented elsewhere
- Examples demonstrating key features
- Details of case studies or applications to real world problems
- Discussion of features particular to the tool
- Summaries (rather than detailed tables) of experimental results
- Analysis of the kinds of problem the tool is suited for and those it is less-well suited for
Ideally the tool and related material (e.g. benchmarks) will be available online and linked to in the submission. The paper should not cover any significant new contributions (these can be submitted as tool papers to RV 2017) and should rely on previous research and tool papers to provide further details. Note that there is no upper page limit however the number of pages used should reflect the level of detail given. Submissions of 2 pages giving minimal details would be suitable in a situation where the tool is well documented elsewhere but an entry in the repository is still desired. All accepted submissions will be invited for presentation at a special Poster session during RV 2017.
These should use a minimum of 2 pages to explore a particular position related to the evaluation, comparison or standardisation of Runtime Verification tools (and benchmarks). Topics may included, but are not limited to:
- What should a RV benchmark look like?
- Can we have a common specification language for RV? If so, what should it look like?
- Is execution time the most important performance criteria? What might be more important?
- How can we evaluate hardware monitoring tools?
- What are we doing wrong in evaluation? Can we fix this?
- What can be borrowed form other communities?
A selection of position papers will be used to structure a discussion panel to be held at RV 2017. Again, there is no upper page limit however the number of pages used should reflect the level of detail given.
Based on the quantity and quality of the submissions we may consider producing a Journal article summarising the discussion, or publish a post-proceedings of revised papers.
- Submission Deadline 1 July 2017
- Notification 1 August 2017
- Workshop at RV 2017 13-16 September 2017
For local and general organisation please see RV 2017.
Program Committee Chairs
Giles Reger, University of Manchester, UK
Klaus Havelund, NASA Jet Propulsion Laboratory, USA
Program Committee (not yet complete)
Ezio Bartocci, TU Wien, Austria
Domenico Bianculli, SnT Centre – University of Luxembourg, Luxembourg
Borzoo Bonakdarpour, McMaster University, Cananda
Christian Colombo, University of Malta, Malta
Ylies Falcone, Univ. Grenoble Alpes, Inria, Laboratoire d’Informatique de Grenoble, France
Sylvain Hallé, Université du Québec à Chicoutimi, Canada
Felix Klaedtke, NEC Europe Ltd.
Dmitriy Traytel, ETH Zürich, Switzerland
Gordon Pace, University of Malta, Malta
Cesar Sanchez, IMDEA Software Institute, Spain
Leonardo Mariani, University of Milano Bicocca, Italy
Julien Signoles, CEA LIST, France