Tutorials

ISO 29119: The New International Software Testing Standards

This tutorial shall introduce the new international Software Testing standards (ISO/IEC/IEEE 29119), briefly covering the following questions:

• What do these new standards provide?
• How do standards authors justify the inclusion of content into standards – and how should this change?
• What is most likely to stop ISO 29119 being adopted by the testing industry?

In May 2007 ISO formed a working group to develop new standards on software testing – a new area for ISO – these standards will start being published in mid-2013. This initiative is closely-supported by IEEE and BSI, both of which have donated existing standards as source documents to the project (these standards will be retired when the new standards are published).

There are currently six new software testing standards in development:

• Concepts and Terminology (ISO/IEC/IEEE 29119-1)
• Test Processes (ISO/IEC/IEEE 29119-2
• Test Documentation (ISO/IEC/IEEE 29119-3)
• Test Techniques (ISO/IEC/IEEE 29119-4)
• Keyword-Driven Testing (ISO/IEC/IEEE 29119-5)
• Test Assessment (ISO/IEC 33063)

This presentation describes the content of the standards, their development and the difficulties encountered in creating standards that are applicable to all organizations (from the smallest to the largest) and all types of project (from agile to traditional safety-critical). The challenge of creating new testing standards when quite disparate parts of the industry (e.g. some context-driven testers and parts of the defence industry) oppose the concept is also covered.
ISO 29119 has already been released in draft form for review (and subsequently been updated based on literally thousands of comments) and is already being used within a number of multi-national organizations. These organizations are already seeing the benefits of reusing the well-defined processes and documentation provided by a standard reflecting current industry best practices.

Stuart Reid is Chief Technology Officer at Testing Solutions Group. He has 30 years experience in IT, working in development, testing and education in application areas ranging from safety-critical to financial and media. Stuart supports the worldwide testing community as the convener of the ISO Software Testing Working Group, which is developing the new ISO 29119 standard. He chairs the BCS Specialist Group in Software Testing and founded ISTQB to promote worldwide software testing qualifications. Stuart present keynotes, tutorials and track sessions at conferences worldwide. He won the European Testing Excellence award in 2001, and regularly writes magazine articles on software testing.

Security testing techniques - Applications to industrial case studies

Presentation of the tutorial - Ina Schieferdecker and Fraunhofer Fokus

Model-based Security Testing (MBST) is a relatively new field and especially focuses on a systematic and efficient specification, generation, or documentation of test objectives, security test cases, and security test suites.  In particular, the combination of security modeling and test generation approaches is still a challenge in research and of high interest for industrial applications. The ITEA project DIAMONDS develops efficient and automated MBST methods for highly secure systems in multiple industrial domains (e.g. banking, automotive, telecommunication, industrial automation etc.). Amongst others, the project has a special focus on advanced model-based security testing methods that combine different security testing techniques to obtain improved results. Model-based fuzz testing allows for the automated and semi-automated detection of vulnerabilities related to missing input validation and the detection of flaws in dedicated security controls.

Autonomous testing techniques based on automatic monitoring supports security validation of larger network infrastructures, and risk-based security testing and its combination with security test pattern provide good guidance on identifying the most appropriate testing approaches and best practices to address specific vulnerabilities. In this tutorial, we will present selected approaches and tools from the DIAMONDS project and show their application to industrial case studies. In the final session, we will summarize the experiences and results that have been made in the case studies.

Evolution of testing techniques: from active to passive testing - Ana Cavalli, Institut Mines-Telecom

Testing techniques are used to check if a given system implementation satisfies its specification or some predefined properties. These testing techniques can be active, based on the execution of specific test sequences against the implementation under test, or passive, based on the observation of the exchange of messages (input and output events) of the implementation under test during run-time. In the last years an important research activity has been taken place on the definition of monitoring techniques based on passive testing. In this talk, we will present the evolution of these testing techniques, their advantages and limitations. We will also illustrate the application of these techniques to the security testing of real case studies.

Ana R. Cavalli received the Doctorat d'Etat es Mathematics Sciences and Informatics in 1984 from the University of Paris VII, France. From 1985 to 1990, she was a staff research member of CNET(Centre National d'Etudes des Telecommunications), where she worked on software engineering and formal description techniques. Since 1990, she joined as professor the Institut Telecom/Telecom SudParis (TSP). Currently, A.R. Cavalli is responsible at TSP of the research group "Protocols and Services Validation" and the Software-Networks department. She is a member of the CNRS Samovar research laboratory. She has been chair of several conferences (IFIP FORTE/PSTV, IEEE ICNP, IFIP TESTCOM, IFIP CFIP, IEEE ICST) and published more than 200 papers. Her research interests include validation methods for Internet protocols and services; methods for conformance, interoperability and security testing; monitoring techniques based on testing and the application of these methodologies to industrial applications. More details can be found on her webpage.

Active testing techniques - Bruno Legeard, SmartTesting & UFC/Femto-st and Alexandre Vernotte, UFC/Femto-st

This talk focuses on dynamic application security testing techniques which include black-box web application vulnerability scanners, fuzzing techniques and emerging model-based security testing approaches. A first part of the talk provides a taxonomy and a survey of the various techniques of dynamic application security testing. A second part presents in more details web application vulnerability scanners, which are currently the most prominent tools in the state for the practice for vulnerability discovery. We will shows the strengths and limits of such commercial or open-source tools. Then, a third part introduces and demonstrates an approach of model-based security testing based on behavioral models and test patterns.

Professor at the University of Franche-Comté/Femto-st Institute, Bruno Legeard is co-founder and senior scientist at Smartesting. He is internationally recognized as an expert and a well known speaker in the model-based testing field. He is strongly experienced in deploying model-based testing solutions both in enterprise information systems area and in the embedded systems field. In 2007, Bruno Legeard authored with Dr. Mark Utting the first industry-oriented book on model-based testing, "Practical Model-Based Testing: A Tools Approach", Morgan & Kaufmann Publisher.

Alexandre Vernotte is a Computer Science PhD Student at the Department of Computer Science for Complex Systems at Femto-St Institute (University of Franche-Comté), under supervision of Prof. Bruno Legeard and Dr. Fabien Peureux. Alexandre is currently working on vulnerability discovery techniques to improve the efficiency and effectiveness of the model-based vulnerability testing process of web applications. Before starting his PhD, I graduated with a Research Master's degree at the University of Franche-Comté.

Fuzz testing techniques - Ari Takanen, Codenomicon and Fabien Duchene, INPG Grenoble

Fuzzing is a next generation technique for zero-day vulnerability discovery, and is used in security auditing, penetration testing and product security processes in software development. Fuzzing feeds a program, device or system with malformed and unexpected input data in order to find critical crash-level defects. Fuzzing is highly effective in finding new vulnerabilities in any communication software. Fuzzing is a black-box testing technique that does not require any access to the source code of the system under test. The tests can be conducted against any system, whether it is internally built or developed by third parties. It can also be used in any phase of the software life-cycle, from development into acceptance testing.

In this presentation, we will analyze latest fuzzing techniques and several different use cases for fuzzing. Different fuzzing techniques have different efficiency profiles. Although random "dumb" mutation fuzzing has been in use since 90s, smart generational model-based fuzzing is relatively new security testing technique for finding critical security problems in any type of communication software. The next generation fuzzing methodologies are based on model-based testing where tests are designed, generated and executed automatically. All products we tested in our case studies failed under fuzzing. And there are no false positives in fuzzing, each issue found is always security critical. The presentation is loosely based on Ari's book on fuzzing, published by Artech House in 2008.

The second part of the presentation will focus on Evolutionary Fuzzing, in which the fuzzing process is driven by a Genetic Algorithm (GA). GA is itself guided by an heuristic that is fault and test harnessing dependent: the fitness function. It will list examples of such functions, some recent advances and directions for future work in Evolutionary Fuzzing.

Ari Takanen, founder and CTO of Codenomicon, has been active in the field of software security research since 1998 focusing on information security issues in next-generation networks and security critical environments. In his work he aims to ensure that new technologies gain public trust by providing means of measuring and solidifying the quality of networked software. Ari Takanen is one of the people behind the PROTOS research project, which studied information security and reliability errors in e.g. WAP, SNMP, LDAP, VoIP implementations. Ari is the author of several papers on security, and is a frequent speaker at security and testing conferences, leading universities and international corporations. He is also the author of two books on VoIP security and security testing.

Passive testing techniques - Wissam Mallouli, Montimage

Network monitoring is a laborious challenging task that is vital for a network operator, a service provider or a corporate network infrastructure in order to keep the network operation stable, smooth and safe. Monitoring provides valuable real time and historical information to understand the network usage trends and dynamics and thus detect misbehaviours and attacks. The vulnerabilities introduced by this “open world”: Critical infrastructures are more than ever open to the Internet, the dematerialization of corporate IT and the success of cloud services are pushing towards proactive mechanisms for detecting and preventing anomalies. In this context, Deep Packet Inspection (DPI) is considered as a catalyser in the shift towards advanced monitoring. DPI is the process of capturing network traffic, analysing and inspecting it closely to determine accurately what is really happening in the network. In this presentation, we will present an events-based network monitoring solution part of MMT tool that inspects network traffic against a set of security properties denoting both security rules and attacks. This solution has been applied to an industrial case study provided by Thales Group that consists of a set of QoS-aware ad-hoc radio communication protocols.

Dr. Wissam Mallouli is currently a research & development engineer at Montimage France. He received his Masters degree from the Evry Val d’Essonne University in 2005 and his PhD in computer science from Telecom and Management SudParis (France) in 2008. His topics of interest cover formal testing and monitoring of functional behaviours and security aspects of distributed systems and networks. He worked in several European and French research projects. He also participates to the program/organizing committees of numerous national and international conferences. He published more than 20 papers in conference proceedings, books and journals. More details can be found on his webpage.

Security test catalogue - Traceability and Risk-Oriented security testing - Ina Schieferdecker, Jürgen Großmann and Fraunhofer Fokus

Security testing is a domain in which a lot of knowledge has been collected from a significant amount of research work done in recent years. Numerous guidelines and best practices have been identified and are used at several instances. The definition of Security Test Patterns is a novel and promising activity for facilitating the reuse of known security testing solutions to recurring problems in the security testing domains. In this tutorial we will present security test pattern that have been developed in the DIAMONDS project and demonstrate a step-wise approach that integrates the application of test pattern with risk-based security testing. The approach starts with the systematic identification of the security problems and their impacts. For this we use the CORAS modelling language to concisely describe threat scenarios and vulnerabilities as well as the associated risks for the organisation or user. On basis of this initial risk assessment we continue with the selection of appropriate security testing approaches and techniques by associating security test pattern to the individual threat scenarios and vulnerabilities. The selection of security test pattern is driven by the kind and content of the risk analysis artefacts and the related risks are used to weight and prioritize the derived test cases. Finally, we assess the relationship between test results and risk analysis in the sense that we are able to control the coverage of risks by tests and test results. The overall approach is supported by means of a traceability framework, which allows for following the traces between the risk analysis artefacts (threat scenarios, vulnerabilities) and the testing artefacts (test pattern, test results).

Industrial case studies - Presentation and testing results: ITrust, Thales, G&D, DonierC, Norse, etc.

• ITrust Consulting: Reverse engineering and testing code execution with Malwasm - Hugo Caron (20 min)

  Malwasm is a "offline debugger", it was designed to help people that do reverse engineering and in particular malware analysis. It is based on cuckoo sandbox to run the malware and log all activities with pintool. After the execution you can replay all ASM instructions in your browser like with other debugger but you can also go back in the past of the execution if you miss something. malwasm will be presented and demonstrated.

  Hugo Caron is in charge of penetration testing projects and IT audit. He is in charge of hacking methodology concerning the European Diamonds project. He is the administrator of our Tricktester platform: an OS based on a Linux distribution with many tools to realize pentests and developed during a European project. He also contributed to writing of exploits and of modules for Metasploit, an open source tool dedicated to intrusion testing and automation.

• Thales: A case study - Michel Bourdellès (20 min)

  We will present a combined approach of active testing and monitoring techniques and their application to a Radio protocol Study designed by Thales Communications & Security. This work has been performed in the framework of the DIAMONDS project and it is the result of the collaboration of THALES with three SMEs (Montimage, SmartTesting and FSCOM). The application of the proposed approach will be illustrated with the validation of several security properties and intrusion attacks.

  Michel Bourdellès acts as the interface between the collaborative projects activities and Software Radio Protocol department products design at Thales Communications & Security. Michel Bourdellès is in particular involved in projects related to (1) the improvement of the RTES design process, protocols intrusion detection security extension as in the ITEA DIAMONDS project, and protocol resource optimization. Michel Bourdellès received a Ph.D. degree in Computer Science from the "Ecole des Mines de Paris" in 1999 and joined THALES in 2000.