[libre-riscv-dev] extremely busy crowdsupply update started

[Jean-Paul Chaput]

Hello Jock,


First, thank you for the time you spend analysing the Coriolis code and
that detailed report. I will try to make a comprehensive answer about why
Coriolis is the way it is. 

* My job description at the University is not only to develop Coriolis,
  but be the system admistrator of a Linux network (150 desktop &
  servers) and manage industrial ASIC toolchains (Cadence,Mentor,Synopys).
  So I usually can only spend 70% of my time on Coriolis.

* One problem in a tool like Coriolis is that you need three kind of
  skills to really be efficient:

  1. A good understanding of the specifics of making an ASIC.
  2. Algorithmic knowledge to tackle efficiently the problems in (1).
  3. Programming skills in at least two language C++ & Python,
     plus a bit of software engineering.

  So, I have knowledge in all three, but cannot became a top notch
  expert in all of them. This also makes it difficult to have interns.

* In some parts, namely the detailed router, the algorithm is a genuine
  one, developing and testing it is a full time job by itself. It is not
  even finished yet. But, when finished it will allow lots of fancy things
  like seamless analog/digital designs and taking into account fancy
  constraints to avoid side-channel attacks on crypto chips (for example).

* It has a past... Some decisions where taken more than 10 years ago in
  a diffrent context and I simply cannot afford to rewrite the software
  every times something shift.

* Nevertheless, I'm deeply commited to write software as cleanly as possible,
  and will make improvements when I can. So I will very carefully read your
  observations. But, unless you want to commit yourself for the next ten years,
  I need a code that I intimately understand.


> Detailed version (The Rant)
> ===========================
> 
> 1. Installation
> 
> Coriolis 2 can be built and installed two ways:
> 
>   a) inside the home directory (“Fixed directory tree”), using a helper script [1],
>   b) as a package, either RPM or DEB [2].
> 
> I followed the (a) route, and it actually went well except for a couple
> of quirks, that is considering the number of dependencies. As I
> understand, the (b) way is not well maintained for now.

  Coriolis is too unstable yet. Using packages would meant to rebuild them
  at every git push, and having the users do not forget to reinstall it
  afterwards. I think Staf Verhaegen is making some trials.
    Ruben Undheim, an official Debian packager, is making DEB of Coriolis,
  you may contact him to see if you can set up some continuous integration
  there.
    My goal is to have something "stable" after the november run in 180nm.


> The method (a) can be used per se, at least on Debian GNU/Linux, or
> within chroot or LXC environments.
> 
> Unfortunately, the Python extensions are coupled with the rest of the
> Coriolis, so it is impossible to use the advantages of 'virtualenv' or
> similar environments with Coriolis.

  I'm not familiar with virtualenv and I don't understand how the
  C++ Python modules can be separated from the Python extension themselves.

> Thus, entering the development with Coriolis for pythonistas is
> complicated by the combination of two factors:
> 
> - lack of native packages for their OSes,
> - lack of PyPI package or pip/virtualenv support.
> 
> I understand that Coriolis is a niche product, so we can not really
> expect much in terms of packaging for popular OSes. So it may be better
> for Coriolis to take another route:
> 
> - decouple Python extensions from the main product,
> - provide Python extensions as a pip package (via PyPI or git),
> - supply a script and/or instructions for building the main product
> with the pip package.

  Breaking things in multiple components introduce complexity that you
  must manage afterwards. I can see a lot of troubles when people will
  have mismatched versions of the C++ core and the PyPI extensions.
  And lots of nigthmarish debug problems.
    Native packages would be better, and can be done, once it is stable
  enough.
  

> This is actually not a unique approach. For example, xapian-haystack
> ships a script for building a whole Xapian search engine (C++) in
> PYTHONPATH [3].
> 
> 2. Documentation
> .
> All the basics are covered by Coriolis User's Guide [4] (general
> workflow and GUI) and Python Tutorial [5], that are available online
> and offline, in PDF and HTML formats.
> 
> Unfortunately, Coriolis Python API don't have its own reference guide.
> There is a good reference on Hurricane C++ API [6] though. Since Python
> API just copies the C++ API (which is a terrible desicion by itself,
> see the explanation in 3.2), this reference alone has the great part of
> both C++ and Python APIs covered. But other parts of the Python API
> (most importantly, Etesian and Katana) stays undocumented beyond the
> basics.

  For Etesian & Katana, this is just due to the lack of time. Also mixed
  with a lack of stability. It is very frustrating to write a doc which
  is no longer relevant three month later. The basic doc of Katana is
  in fact in Kite. Kite is the digital only version of the router while
  Katana is the mixed one. And it is in flux, so I postponed the port
  of the doc.
  


> 3. API
> 
> 3.1. Supported Python versions
> 
> The Coriolis Python API is a Python 2 extension.
> 
> Python 2.7 was released in 2010, and it was meant to be the last 2.x
> release back then. In the following years Python 2 had no new features
> other than backports from Python 3. As of January 2020 Python 2 is
> considered unsupported by PSF. [7]
> 
> Still clinging to outdated and abandoned language and not even having a
> (publicly available, at least) transition plan seems a bit scary to me.

  I understand very well. I'm not clinging to it, I just have had not time
  yet to make the transition. If nothing unexpected happens I hope to do
  the migration sometimes in the next three monthes. Here I must admit
  I'm not proud of the way the C++/Python wrapper is written, but it do
  works exactly like I want to... Here again with lots of unusual constraints
  that I see no way to make swig or the like to understand.


> 3.2. Best practices of Python APIs
> 
> To have a new API mimic the old API written in another programming
> language seems like an excellent idea at first. You don't have to
> design anything. Don't have to think of naming. You can reuse
> documentation and tests. The new API is easy to use, if you already got
> used to the old one. Everybody's happy, right?
> 
> Well, everybody except the end users whom the new API is created for.

  Not so fast here, we need a definition of "end users". In my lab,
  the people using Coriolis are mostly ASIC people, that is electrical
  engineers. They use the minimal and simplest possible subset of any
  programming language.
    You may look at the relationship between C++ and SystmeC. SystemC
  is comparable to nMigen but under the form of a C++ library. They
  don't event tell them it's C++. SystemC is usually presented as a
  language on it's own (sometimes they talk about C).
    In fact, and this is a trend I've seen emerge with the rise of FPGA
  for some time, you have *two* kind of audience.
  
  * Electrical engineers or guys comming "from the hardware", with limited
    programming skills. They describe their design as series of blocks.
    They want to tightly control how and what they generate.
  * Computer Science types, or guys coming "from the software", with
    advanced programming skills. They tend to see their design in term
    of classes and functions, with lot of implicit mechanisms.
    Typical example is Chisel or SpinalHDL.

  At this point I have no clear answer about what the "best" language
  would be. But I can tell you for sure that many "hardware guys" are
  literally repelled by languages like Chisel.
    It's a discussion we are currently having at the lab. Like the one
  that did take place in this list a little while ago.
  

> Each programming language has its own best practices, and most of the
> time they either contradict the ones accepted by another language users
> or just don't have analogues in that language. Your new users will see
> your API as alien, cumbersome, or just poorly designed. They may have
> to cram their way around the least expected features. They may have to
> write wrappers around your API to reduce the astonishment factor [8]
> for the others. But most likely they will simply walk away.

  Another point to take into account, is that, at first, the Python
  extention was made to be able to do "rapid prototyping". That is
  quicly develop and test under Python, then, convert back to C++.
  So keeping both interfaces in synch as much as possible was to
  reduce that work.
    And, as if for myself only, I do constant back and forth between
  Python and C++, so I have one less "translation" to manage.
  

> Here I'll try to give a run-down of Coriolis API features that scares
> Python developers (“bad”) along with their attractive and comfy
> alternatives (“good”).
> 
> 3.2.1. Singletons
> 
> You don't need a special methods to create and access a unique object
> in Python, since you can just import it.
> 
> Bad: af = CRL.AllianceFramework.get()
> 
> Good: from CRL import af

  OK.
  

> 3.2.2. Getters and setters
> 
> Getters and setters are considered an antipattern (not only in Python),
> so they should be replaced by properties or collection-like interfaces.
> 
> Bad:
>   env = af.getEnvironment()
>   env.setCLOCK('^ck$|m_clock|^clk$')
> 
> Good:
>   af.env['CLOCK'] = '^ck$|m_clock|^clk$'

  OK. But introduce some rework at Python/C++ level.
  

> 3.2.3. Data types
> 
> Python use duck typing, so you should avoid any direct type indication
> or querying in Python.
> 
> Bad: Cfg.getParamInt('katana.vTracksReservedLocal').setInt(6)
> 
> Good: Cfg['katana.vTracksReservedLocal'] = 6

  The Cfg module is a shame. Is is unecessarily separated from the rest
  and the only one using boost. Creating one of the most painful
  dependency only for it.
    As those variables are created then stored in C++, I need to be
  able to guess their type right. integer, float, sometimes enum,
  and percentage... If you can show me an elegant solution, I will
  take it.
  

> 3.2.4. Syntactic sugar
> 
> It's easier to account for the certain things that goes in pairs
> (open/close, create/destroy) with context managers or decorators.
> 
> Bad:
>   UpdateSession.open()
>   do_my_thing()
>   UpdateSession.close()
> 
> Good:
>   with UpdateSession():
>       do_my_thing()
> 
> Also good in some cases:
>   @update_session
>   def do_my_thing():
>       ...

  OK. I will look into that.


> 3.2.5. Self-documenting
> 
> Python have a built-in 'help()' function, that returns a docstring for
> a given object. Python C extensions can emulate a docstring for use
> with 'help()'. Static syntax checkers and IDEs also rely on this API.
> 
> It is very important that this emulated docstring contains not only the
> description of the function or method, but also a signature (parameters
> and return value) in reST format, as recommended by PEP287 [9], or in
> older Epytext format at worst. This will greatly facilitate the use of
> Python C extension with modern IDEs.
> 
> The rules have slightly changed since Python 3.4, but the idea remains.

  The main reason here is still the lack of time. But there may be
  another problem. My python interface allow to mirror in Python
  the C++ function overload, but Python only allows one set of
  parameters, is that true or I'm lagging on Python features again?
  So how to describe a function with more than one parameter set?
  

> 3.2.6. Naming conventions
> 
> The Python API must adhere to naming conventions accepted in Python. I
> can not stress enough the importance of this point, but let us address
> the styling issues in general in 3.3.
> 
> 3.3. Styling
> 
> I have to admit that Coriolis Python style is quite consistent, but it
> looks like the only purpose of this consistency is to contradict the
> PEP 8 [10] in any possible way. [11]
> 
> You might think that style is not that important, but after 15 minutes
> spent in search of the 'ChipRoute' class definition, diagnosing and
> restarting IDE, checking Moon phase and whatnot, find that 'ChipRoute' 
> is not a class, but a module… How well the end user could perform with
> that in mind?

  Quite the opposite, I do think style is important. The goal of style
  is readability and it is mine. But I don't think you can say that there
  is *one* good style and all others are bad. And even the reference style
  may evolve... It is easy to slip from "the style I'm accustomed to" and
  "the right style". I understand the advantages of using a style shared
  by many peoples, but I won't blindly comply (I'm both French and a bit
  stubborn). I will try to find time to read the PEP8, but I have a lot
  of work to do...
    I've already seen this debate about C++ coding, and there are
  definetely good and bad styles (or lack of) but I won't say there is
  "the good style". Among good styles, there are different tradeoffs
  that may or may not suits you.

 
> Another thing is that it's nearly impossible to lure a junior developer
> into a project that neglects styling. In modern development,
> proprietary and opensource alike, an ability to write a consistent,
> readable, uniform code is very much appreciated, and junior developers
> expect to acquire good professional habits in their first projects. And
> style is the first thing they learn to distinguish in someone else's
> code.

  Maybe it has changed in the company world, but for what I've seen
  in the ASIC world, code cleanliness is still a long way home...
  But again, it may be different company we are talking about.
  I talk of Cadence/Mentor/Synopsys, if you have the opportunity,
  take a look.

  A last reflexion about PyCharm, it is very well for people working
  only on Python code. But for my part, I edit C++, C, Python, ReST,
  cmake, LaTeX, Yaml and whatnot, so I use Emacs (vim could also do).
  It would be inefficient to have one IDE/editor per language.

  
  Anyway, this is very valuable input, and I will take it into
  account as soon as I can, given my current workload.


  Best regards,
  

> [1] http://coriolis.lip6.fr/pages/users-guide.html#id16
> [2] http://coriolis.lip6.fr/pages/users-guide.html#id19
> [3] https://github.com/notanumber/xapian-haystack
> [4] http://coriolis.lip6.fr/pages/users-guide.html
> [5] http://coriolis.lip6.fr/pages/python-tutorial.html
> [6] http://coriolis.lip6.fr/doc/hurricane/index.html
> [7] https://www.python.org/dev/peps/pep-0373/
> [8] https://en.wikipedia.org/wiki/Principle_of_least_astonishment
> [9] https://www.python.org/dev/peps/pep-0287/
> [10] https://www.python.org/dev/peps/pep-0008/
> [11] 
> 
https://www.opensourceshakespeare.org/views/plays/play_view.php?WorkID=hamlet&Act=2&Scene=2&Scope=scene&LineHighlight=1307#1307


-- 

      .-.     J e a n - P a u l   C h a p u t  /  Administrateur Systeme
      /v\     Jean-Paul.Chaput at lip6.fr
    /(___)\   work: (33) 01.44.27.53.99              
     ^^ ^^    cell:      06.66.25.35.55   home: 09.65.29.83.38

    U P M C   Universite Pierre & Marie Curie
    L I P 6   Laboratoire d'Informatique de Paris VI
    S o C     System On Chip