[Libre-soc-dev] microwatt grows up LCA2021

[Luke Kenneth Casson Leighton]

On Tuesday, February 9, 2021, Paul Mackerras <paulus at ozlabs.org> wrote:
> On Mon, Feb 08, 2021 at 04:50:50PM +0000, Luke Kenneth Casson Leighton
wrote:
>
>> happy to take a look and help review.
>
> Here are the finalAnalysisReport.txt results of two STS runs, each
> on 1,000 sequences of 1,000,000 bits.

also remember
* 1,000 on 100k then
* 10,000 on 100k then if those pass
* jump to 10,000 of 1e6

basically crank it up one order of magnitude at a time [but remember that
the Lempel Ziv test only shows up as "flawed" at these higher numbers of
runs].


>  I left out the universal
> statistical test because it is giving p=0 on any input.

drat.  probably because of some 64 bit thing.  i was running STS perfectly
fine on QTY AMD Opteron systems.

>
> First run:
>
>
> 109  96 115  87 100 102 101  90  95 105  0.693142    979/1000 *
NonOverlappingTemplate
>  98  98 116 106  94  74 103  87 108 116  0.088226    989/1000
NonOverlappingTemplate
>  97  92  94 103  98 109 112  94 114  87  0.587274    990/1000

> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - -
> The minimum pass rate for each statistical test with the exception of the
> random excursion (variant) test is approximately = 980 for a
> sample size = 1000 binary sequences.


Ok so notice the asterisk? that indicates a failure.  remember, some
quantity of "failures" (tests that statistically are borderline) are going
to occur...

... but STS if you look at the paper they mathematically *calculate and
predict the number of expected failures*

(a variant of the test of tests of tests thing)

for that run it has fallen below the acceptable *quantity* of failures to
be considered "safe"

thus this run must be considered an indicator of a catastrophic failure in
the algorithm.

yes you need to be that draconian.

now, it's only by one (the success rate is 980, the pass rate was 979) so
it means that the algorithm is very close to being "good".

now, if you re-run it, at the other sizes / partitions you should keep an
eye out for that.  if it happens again, particularly at the 10k runs, then
that's confirmation that there's a serious problem.

this is very subtle, these tests look for micro-artefacts at different
frequencies and different types of patterns (in effect).

it takes time to find the flaws.  i ran these tests literally hundreds of
times over a six month period.

l.


-- 
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68