[libre-riscv-dev] random testing is insufficient

[Luke Kenneth Casson Leighton]

On Friday, July 5, 2019, Jacob Lifshay <programmerjake at gmail.com> wrote:

> I noticed that the code for testing 32-bit floating-point operations
> is generating 10 pairs of uniformly random 32-bit integers,
> reinterpreting them as 32-bit floats, then verifying that the
> floating-point HW produces the correct values for those 10 pairs. This
> testing is insufficient to catch even obvious errors, such as always
> returning 0 when any input is a NaN (made up flaw).


>
See test_add.py, corresponding test_add16.py and test_add64.py, and div and
mul variants of the same.

They have not been de-cut-and-paste-itis-ified entirely as the multi in/out
code has.

The code that uses runfp (and goes through to InputTest) is designed as a
very quick test, hence the short runtime, so that iterative development is
not adversely conpromised.

test_add.py etc are not designed to be run automatically: they generate
hundreds of thousands of values, and target specific areas as well as run
regression tests.

The regression tests are, obviously, first, so that the iterative
development process is, again, not adversely compromised.

If those pass it moves on to a FULL matrix of NaN against INF against Zero
(etc), then moves on to random testing of each category, before finally
dropping into full random testing.

However what it is NOT designed to do, where InputTest is, is to test the
muxid.  test_add.py (etc) are designed to cover a fixed length pipeline.

Actually they originate from jon dawson's code and are designed to cover a
FSM, one thing at a time.

The code underwent a huge amount of morphing that was critical to keeping
the code stable as it transitioned from FSM to pipeline.

I also made an effort to keep the FSM code running as well.

L.



-- 
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68