PQC Algorithm Testing Framework

** Currently on hold!**

The complexity and criticality of cryptographic algorithms requires extensive testing to ensure that implementations works correctly, and to handles unexpected behavior in a secure way. Furthermore one implementation needs to be tested on a variety of different systems to ensure it’s cross-system functionality. Moreover, it is important to increase the accessibility of cryptographic software development for a broader audience to profit from new ideas and contributions. Thus, through developing crypto-agility methods, a broader range of developers from different fields working on cryptographic software solutions might bring a huge benefit in applying well-known software engineering principles to the development process.

The tool aims to archive the following goals:

Accessibility and ease-of-use

The tool will be easy to install, preferably with little to no custom configuration requirements or at least with an extensive explanation and documentation attached to any configuration requirements. Its implementation should be largely independent of specific system requirements to make it available for as many users as possible.

The testing and validation operations it performs must be well documented and descriptions on what is tested, why it is tested and how these test goals are accomplished is a crucial part for such a tool. This might also require a somewhat limited set and complexity of tests to execute so that a user can be sufficiently sure of the correctness of their implementation without being overwhelmed by the mass or complexity of tests.

Furthermore a commonly usable API / data exchange format is required so that everyone developing a cryptographic algorithm knows what is needed to test their implementation.

Extendability

To follow and contribute to the idea of cryptographic agility the tool must also be easily extendable in many ways.

Algorithms-extendability:

If a new type of algorithm is proposed and implemented it should be easy to add the reference implementation of this algorithm to the tool so it is able to test other implementations for compatibility with the reference implementation.

Reference-extendability:

If an updated version of a reference implementation is available or a new implementation proves itself and replaces the reference implementation over time it should be possible to easily substitute or combine (multiple reference impls) the implementations in the framework to ensure it operates using the latest standards.

Language-extendability:

The framework aims to support multiple language implementations. Hence it needs a commonly usable, strictly defined data API and possibly some simple API adapters for different languages to use for communication.

Test-set-extendability:

The framework aims to be simple to use and understand hence the amount and type of tests should be somewhat restricted in the basic variant. However, for more advanced users it might be desirable to be able to extend the set or type of tests executed by the framework with custom tests. The framework aims to offer this custom test specification in a limited way.

System-extendability:

A critical thing to test are cryptographic implementations running on i.e. embedded devices. The framework should be able to function as a testing endpoint (i.e. over a network) for embedded systems to test the basic functionality of implementations running on such devices.