SEP advanced function/profiling in Scilab 6

classic Classic list List threaded Threaded
5 messages Options
Clément David-3 Clément David-3
Reply | Threaded
Open this post in threaded view
|

SEP advanced function/profiling in Scilab 6

Dear devs,

I started working on re-introducing profiling functions into Scilab 6.0;
these functions will behave very similarly to the Scilab 5 ones but have
been renamed for consistency and their arguments will slightly differs
(macro value vs macro as string). Thanks to Samuel's mail [2], I wrote a SEP
[1] that might finally fix that miss, please comment and give feedbacks !

[1]: https://wiki.scilab.org/SEP%20profiling%20in%20Scilab%206.1
[2]:
http://mailinglists.scilab.org/New-profiling-module-code-coverage-td4034048.
html

Thanks,

--
Clément

_______________________________________________
dev mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/dev

smime.p7s (5K) Download Attachment
Samuel GOUGEON Samuel GOUGEON
Reply | Threaded
Open this post in threaded view
|

Re: SEP advanced function/profiling in Scilab 6

Hello Clément,

Glad to see that restoring handy profiling facilities is awake.
I am afraid i don't catch all from the SEP. Here is a short list of remarks that still make me wondering:
  • In 5.5.2, the former functions were about BOTH coverage and timing. Displaying both informations in the 2 first columns was very handy.
    The current SEP seems to present 2 separated subsets of functions, as if things are planned to become separated. This is quite unclear.

  • Reforging things should be a major opportunity to suppress the splitting in many functions just vs the action.
    Hence, the former add_profiling, remove_profiling, reset_profiling were painfull because inventing, maintaining, documenting and using 3 separate functions just because of ONE parameter is changing -- the action -- imo was imo meaningless. This is a low-level kind of design, whereas Scilab is a high-level language, the language should be integrated: I rather expect using
    profile(action, target [, options])  with for instance action = "on"|"off"|"reset"|"disp"|"html"|"plot"...
    In this way, things are open. If other actions have to be added latter, the prototype won't change, and no need to have a Nth function and Nth documentation page in 5 natural languages.

  • In the SEP, the fact that restoring a display in text mode in the console is planned does not explicitly appear. Nor the planned order of columns of results. Both aspects are really killing details. When improving the code, i never felt the need of having pleasant colored reports. But, daily, the display in console was my first and almost only usage.

  • "instrumentation" is some hardly understandable devs jargon.

Best regards

Samuel


Le 30/11/2018 à 15:02, Clément David a écrit :
Dear devs,

I started working on re-introducing profiling functions into Scilab 6.0;
these functions will behave very similarly to the Scilab 5 ones but have
been renamed for consistency and their arguments will slightly differs
(macro value vs macro as string). Thanks to Samuel's mail [2], I wrote a SEP
[1] that might finally fix that miss, please comment and give feedbacks !

[1]: https://wiki.scilab.org/SEP%20profiling%20in%20Scilab%206.1
[2]:
http://mailinglists.scilab.org/New-profiling-module-code-coverage-td4034048.
html

Thanks,

--
Clément


_______________________________________________
dev mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/dev



_______________________________________________
dev mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/dev
Clément David-3 Clément David-3
Reply | Threaded
Open this post in threaded view
|

Re: SEP advanced function/profiling in Scilab 6

Hello Samuel and thanks for your comments,

 

I updated the SEP with more information clarifying some aspects. These `profile` functions are not supposed to replace the existing `coverage` ones. The underlying execution counters are re-used for the two implementations but the usage might slightly be different: produce a static report for the coverage or display performance information while developing. I reword some aspect to clarify the display and show methods, the “instrumentation”

 

At first, I want to map the Scilab 5 features before trying to push enhancement or API modification so I used similar names as in Scilab 5. The main issue with switching API is the associated effort to port existing Scilab 5 toolset to the new API. As you propose, a Matlab-like `profile`  might be a later extension.

 

Best,

 

--

Clément

 

 

From: dev <[hidden email]> On Behalf Of Samuel Gougeon
Sent: Sunday, December 2, 2018 8:54 PM
To: List dedicated to the development of Scilab <[hidden email]>
Subject: Re: [Scilab-Dev] SEP advanced function/profiling in Scilab 6

 

Hello Clément,

Glad to see that restoring handy profiling facilities is awake.
I am afraid i don't catch all from the SEP. Here is a short list of remarks that still make me wondering:

  • In 5.5.2, the former functions were about BOTH coverage and timing. Displaying both informations in the 2 first columns was very handy.
    The current SEP seems to present 2 separated subsets of functions, as if things are planned to become separated. This is quite unclear.
  • Reforging things should be a major opportunity to suppress the splitting in many functions just vs the action.
    Hence, the former add_profiling, remove_profiling, reset_profiling were painfull because inventing, maintaining, documenting and using 3 separate functions just because of ONE parameter is changing -- the action -- imo was imo meaningless. This is a low-level kind of design, whereas Scilab is a high-level language, the language should be integrated: I rather expect using
    profile(action, target [, options])  with for instance action = "on"|"off"|"reset"|"disp"|"html"|"plot"...
    In this way, things are open. If other actions have to be added latter, the prototype won't change, and no need to have a Nth function and Nth documentation page in 5 natural languages.
  • In the SEP, the fact that restoring a display in text mode in the console is planned does not explicitly appear. Nor the planned order of columns of results. Both aspects are really killing details. When improving the code, i never felt the need of having pleasant colored reports. But, daily, the display in console was my first and almost only usage.
  • "instrumentation" is some hardly understandable devs jargon.

Best regards

Samuel


Le 30/11/2018 à 15:02, Clément David a écrit :

Dear devs,
 
I started working on re-introducing profiling functions into Scilab 6.0;
these functions will behave very similarly to the Scilab 5 ones but have
been renamed for consistency and their arguments will slightly differs
(macro value vs macro as string). Thanks to Samuel's mail [2], I wrote a SEP
[1] that might finally fix that miss, please comment and give feedbacks !
 
[1]: https://wiki.scilab.org/SEP%20profiling%20in%20Scilab%206.1
[2]:
http://mailinglists.scilab.org/New-profiling-module-code-coverage-td4034048.
html
 
Thanks,
 
--
Clément




_______________________________________________
dev mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/dev

 


_______________________________________________
dev mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/dev

smime.p7s (5K) Download Attachment
Samuel GOUGEON Samuel GOUGEON
Reply | Threaded
Open this post in threaded view
|

Re: SEP advanced function/profiling in Scilab 6

Hello Clément,

Thanks for this second version, indeed much clearer to me. This one brings new questions and comments.
  • The planned ability to target a single function, or a whole library, or all loaded libraries goes beyond the portage to Scilab 6. This is already a great improvement.
    About possible targets, here are some arising open questions (they are not requests, just questions for more information) :
    => Will it be possible to target a user-defined function out of libraries (defined with deff, function (in console), exec, jdeff..)?

  • prof = profileGetInfo(): the architecture/hierarchy to access to the results is unclear to me. This is an important point in term of usages. This point is one of major ones hindering the usage of slint(), as reported on bugzilla and on users@. It would be a pity to have the same issue for profiling results. Examples showing how to address most interesting results would be highly welcome.
    • Example : How to get totalTime for all functions of a given library (as a first step to focus on hot functions).
      Will it be something like
      sel = prof.FunctionTable.ModuleName=="myModule";
      prof.FunctionTable(sel).FunctionName
      prof.FunctionTable(sel).TotalTime
      or like
      prof.FunctionTable.<ModuleName>.<FunctionName>.TotalTime
      or like
      prof.FunctionTable.<FunctionName>.TotalTime
      prof.FunctionTable.<FunctionName>.ModuleName

    • I do not catch why there are 2 distinct 1st level fields FunctionTable and FunctionCoverage, instead of having a tenth field FunctionCoverage directly at the first level. This first level FunctionTable/FunctionCoverage complicates the addressing, and its motivation is unclear.

    • To me, the most handy first level addressing should be directly through <functionName>:
      prof.myFun.TotalTime
      prof.myFun.Coverage
      etc. This is much simpler and straightforward than with the syntaxes presented hereabove.


    • About fields names
      • "ModuleName" might be abbreviated into Module or Library without loss of meaning and clarity. If the expected value is the name of the library (e.g. corelib), then the field name should be Library instead of Module or ModuleName. Otherwise, ModuleName will refer to core, not to corelib. This would prevent securely addressing information from the library (the name of a module is not clearly defined. Is it the name of its SCI/<module> directory, or the name of its gateway, or...?
      • "FirstLine" : is it the "Prototype"?
      • InstructionsCount, BranchesCount, PathsCount :
        • these fields would be new results, not available in Scilab 5. For the time being, I do not understand what they represent nor what they can be used to.
        • Does "executed lines count" stand for only lines that are actually executed, or does it stand for all lines, with a count 0 for not executed ones?

  • The case of nested functions (a function defined in a function, etc) is not presented. There were some issues in Scilab 5 about them. See for instance the bug 12104 or the bug 12105.

  • The case of internal functions defined in the same .sci file after the main function is not presented. This is a new topic, since the Scilab 6 processing is new with respect to these local dependencies. Will profiling these internal dependencies be possible?

  • "The main issue with switching API is the associated effort to port existing Scilab 5 toolset to the new API. As you propose, a Matlab-like `profile`  might be a later extension."
    I am sorry, i do not clearly understand this point. Anyway, the API is changed, since functions are renamed, and the format of results is completely changed.
    Writting a profile() macro routing to the internal profileEnable() etc would be somehow let's say ~3% of the time, by far the easiest part of the forge, compared to the implementation/rewritting of hard code. But it would ease usages and maintenance and documentation. Skipping it looks like a bit self censorship, self limiting impact of a great work.
    At least, not splitting the documentation in N separated pages, but building a single help page presenting all profiling functions and usages would be better.

Thanks again for all your efforts and work.

Best regards
Samuel

Le 03/12/2018 à 16:31, Clément David a écrit :

Hello Samuel and thanks for your comments,

 

I updated the SEP with more information clarifying some aspects. These `profile` functions are not supposed to replace the existing `coverage` ones. The underlying execution counters are re-used for the two implementations but the usage might slightly be different: produce a static report for the coverage or display performance information while developing. I reword some aspect to clarify the display and show methods, the “instrumentation”

 

At first, I want to map the Scilab 5 features before trying to push enhancement or API modification so I used similar names as in Scilab 5. The main issue with switching API is the associated effort to port existing Scilab 5 toolset to the new API. As you propose, a Matlab-like `profile`  might be a later extension.

 

Best,

 

--

Clément

 

 

From: dev [hidden email] On Behalf Of Samuel Gougeon
Sent: Sunday, December 2, 2018 8:54 PM
To: List dedicated to the development of Scilab [hidden email]
Subject: Re: [Scilab-Dev] SEP advanced function/profiling in Scilab 6

 

Hello Clément,

Glad to see that restoring handy profiling facilities is awake.
I am afraid i don't catch all from the SEP. Here is a short list of remarks that still make me wondering:

  • In 5.5.2, the former functions were about BOTH coverage and timing. Displaying both informations in the 2 first columns was very handy.
    The current SEP seems to present 2 separated subsets of functions, as if things are planned to become separated. This is quite unclear.
  • Reforging things should be a major opportunity to suppress the splitting in many functions just vs the action.
    Hence, the former add_profiling, remove_profiling, reset_profiling were painfull because inventing, maintaining, documenting and using 3 separate functions just because of ONE parameter is changing -- the action -- imo was imo meaningless. This is a low-level kind of design, whereas Scilab is a high-level language, the language should be integrated: I rather expect using
    profile(action, target [, options])  with for instance action = "on"|"off"|"reset"|"disp"|"html"|"plot"...
    In this way, things are open. If other actions have to be added latter, the prototype won't change, and no need to have a Nth function and Nth documentation page in 5 natural languages.
  • In the SEP, the fact that restoring a display in text mode in the console is planned does not explicitly appear. Nor the planned order of columns of results. Both aspects are really killing details. When improving the code, i never felt the need of having pleasant colored reports. But, daily, the display in console was my first and almost only usage.
  • "instrumentation" is some hardly understandable devs jargon.

Best regards

Samuel


Le 30/11/2018 à 15:02, Clément David a écrit :

Dear devs,
 
I started working on re-introducing profiling functions into Scilab 6.0;
these functions will behave very similarly to the Scilab 5 ones but have
been renamed for consistency and their arguments will slightly differs
(macro value vs macro as string). Thanks to Samuel's mail [2], I wrote a SEP
[1] that might finally fix that miss, please comment and give feedbacks !
 
[1]: https://wiki.scilab.org/SEP%20profiling%20in%20Scilab%206.1
[2]:
http://mailinglists.scilab.org/New-profiling-module-code-coverage-td4034048.
html
 
Thanks,
 
--
Clément




_______________________________________________
dev mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/dev

 



_______________________________________________
dev mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/dev



_______________________________________________
dev mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/dev
Clément David-3 Clément David-3
Reply | Threaded
Open this post in threaded view
|

Re: SEP advanced function/profiling in Scilab 6

Hello Samuel and happy new year,

I will try to answer your question in-line to clarify the current description (and implementation).
I also updated the wiki page with the latest information (mainly a split I will detail below).

 
> => Will it be possible to target a user-defined function out of libraries (defined with deff,
> function (in console), exec, jdeff..)?

Yes of course, from the interpreter point of view, a Scilab function (in whatever way we use to
define it) is a specific value we instrument or retrieve the value from.

> sel = prof.FunctionTable.ModuleName=="myModule";
> prof.FunctionTable(sel).FunctionName
> prof.FunctionTable(sel).TotalTime

Yep, that's almost it :

 sel = prof.FunctionTable.ModuleName == "myModule";
 prof.FunctionCoverage(sel).FunctionName
 prof.FunctionCoverage(sel).TotalTime


> I do not catch why there are 2 distinct 1st level fields FunctionTable and FunctionCoverage,
> instead of having a tenth field FunctionCoverage directly at the first level. This first level
> FunctionTable/FunctionCoverage complicates the addressing, and its motivation is unclear.
>
> To me, the most handy first level addressing should be directly through <functionName>:
> prof.myFun.TotalTime
> prof.myFun.Coverage
> etc. This is much simpler and straightforward than with the syntaxes presented hereabove.

I described a way to store functions descriptions, functions statistics and line coverage in
separate fields as this is really different information. For exemple, to store function description
or statistics I used a struct which is the most easy way to select (as in the above snippet) a
module or a specific function whereas to store line coverage (per function) in a compact way I used
a list containing information for each function and *per line*. So really the final indexing is not
the same for this line coverage.

At the end and after some interaction with our current customer, we ended up with three different
information : static per function (file names, module, etc..), dynamic per function (number of
calls, total time, etc...), dynamic per function per line (time and counter). So a tlist with three
fields : FunctionTable, FunctionCoverage and LineCoverage.

> About fields names
> "ModuleName" might be abbreviated into Module or Library without loss of meaning and clarity. If
> the expected value is the name of the library (e.g. corelib), then the field name should be
> Library instead of Module or ModuleName. Otherwise, ModuleName will refer to core, not to corelib.
> This would prevent securely addressing information from the library (the name of a module is not
> clearly defined. Is it the name of its SCI/<module> directory, or the name of its gateway, or...?

I agree with the "core" vs "corelib" however this value is stored as a string so this is really a
name. I will switch to "LibraryName" to ease understanding and be coherent with the libraryinfo
documentation [1].

[1]: https://help.scilab.org/docs/6.0.1/en_US/libraryinfo.html

> "FirstLine" : is it the "Prototype"?

This field is the first line of the function (the one with the `function` keyword) and is the same
as:
 * used on error (in Scilab 5 or 6)
 * within Scinotes

This is also used to avoid generating header lines on LineCoverage.

> InstructionsCount, BranchesCount, PathsCount :
> these fields would be new results, not available in Scilab 5. For the time being, I do not
> understand what they represent nor what they can be used to.
> Does "executed lines count" stand for only lines that are actually executed, or does it stand for
> all lines, with a count 0 for not executed ones?

This is basically counters for coverage, the executed instruction/branches/path and non executed
instruction/branches counts for a specific function are reported. This is not really "lines count"
but expression count (executable statement in the code) thus is more precise than lines based
coverage but harder to visualize.

> The case of nested functions (a function defined in a function, etc) is not presented. There were
> some issues in Scilab 5 about them. See for instance the bug 12104 or the bug 12105.
>
> The case of internal functions defined in the same .sci file after the main function is not
> presented. This is a new topic, since the Scilab 6 processing is new with respect           to
> these local dependencies. Will profiling these internal dependencies be possible?

Yes this is handled and was an important remark from our customer. Both the sub function (in the
same macro file) or inner functions (within another function) are handled as independent functions
and could be retrieved by FunctionTable.FileName and a specific FunctionTable.ParentIndex (index to
its parent function).

Inner functions are also reported as non executable lines on the parent function LineCoverage.

> "The main issue with switching API is the associated effort to port existing Scilab 5 toolset to
> the new API. As you propose, a Matlab-like `profile`  might be a later extension."
> I am sorry, i do not clearly understand this point. Anyway, the API is changed, since functions
> are renamed, and the format of results is completely changed.
> Writting a profile() macro routing to the internal profileEnable() etc would be somehow let's say
> ~3% of the time, by far the easiest part of the forge, compared to the implementation/rewritting
> of hard code. But it would ease usages and maintenance and documentation. Skipping it looks like a
> bit self censorship, self limiting impact of a great work.
> At least, not splitting the documentation in N separated pages, but building a single help page
> presenting all profiling functions and usages would be better.

Well, each Scilab function is supposed to have its own associated help page furthermore having the
full description. However I agree that having a single `profile` page will ease overall
understanding, its usage and its maintenance.

Thanks for your remarks,

--
Clément
_______________________________________________
dev mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/dev