[Scilab-users] Headless graphics in scilab: time & memory issues

classic Classic list List threaded Threaded
9 messages Options
Antoine Monmayrant Antoine Monmayrant
Reply | Threaded
Open this post in threaded view
|

[Scilab-users] Headless graphics in scilab: time & memory issues

Hello all,

In these days of remote working, I need to use again scilab in headless mode to generate graphics.

In other words, I connect  in command line only to a remote server where scilab is installed and I want it to generate graphics using a combination of driver, xinit and xend.

There seems to be some issues with driver/xinit/xend: huge memory overhead and long computation time that makes it impossible to generate some plots in headless mode.

For example, a simple Sgrayplot that takes ~1 second to plot and save to a png in normal mode takes more than 200 seconds and huge amount of ram in headless mode (see attached test script).

I also have some plots that works without a problem in normal mode, but fail in headless mode because of some memory limitation:

xend: An error occurred: Unable to create export file, not enough memory. Decreasing the number of elements or the size of the figure should fix this error.

Is there a way to get around these problems?

Thanks for your help,

Antoine







_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users

xend_test.sce (666 bytes) Download Attachment
Antoine Monmayrant Antoine Monmayrant
Reply | Threaded
Open this post in threaded view
|

Re: Headless graphics in scilab: time & memory issues

Hello again,

Just to add to my previous message: increasing nx or ny in my script can crash scilab during the call to xend().
On my small local machine, this occurs for:

nx=4000;
ny=3000;

for which scilab takes ~7 seconds to create the png in "normal" mode...

Can any of you confirm this bug?

Antoine

On 03/12/2020 09:48, Antoine Monmayrant wrote:

Hello all,

In these days of remote working, I need to use again scilab in headless mode to generate graphics.

In other words, I connect  in command line only to a remote server where scilab is installed and I want it to generate graphics using a combination of driver, xinit and xend.

There seems to be some issues with driver/xinit/xend: huge memory overhead and long computation time that makes it impossible to generate some plots in headless mode.

For example, a simple Sgrayplot that takes ~1 second to plot and save to a png in normal mode takes more than 200 seconds and huge amount of ram in headless mode (see attached test script).

I also have some plots that works without a problem in normal mode, but fail in headless mode because of some memory limitation:

xend: An error occurred: Unable to create export file, not enough memory. Decreasing the number of elements or the size of the figure should fix this error.

Is there a way to get around these problems?

Thanks for your help,

Antoine







_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users

_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users
Dang Ngoc Chan, Christophe Dang Ngoc Chan, Christophe
Reply | Threaded
Open this post in threaded view
|

Re: {EXT} Re: Headless graphics in scilab: time & memory issues

Hello Antoine,

> De : Antoine Monmayrant
> Envoyé : jeudi 3 décembre 2020 10:00
>
> Can any of you confirm this bug?

I don't know what you mean by headless mode but I get the same error as you error when I run your script (except it is in Franch):
"xs2png : Impossible de créer le fichier d'export, pas assez de mémoire disponible. Diminuer le nombre d'éléments ou la taille de la figure devrait corriger cette erreur."

Regards

--
Christophe Dang Ngoc Chan
Mechanical calculation engineer

General
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error), please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users
mottelet mottelet
Reply | Threaded
Open this post in threaded view
|

Re: Headless graphics in scilab: time & memory issues

In reply to this post by Antoine Monmayrant

Hello Antoine,

Exporting in headless mode (all drivers but the default "Rec") is fundamentally different from the default "Rec" mode where graphics are actually rendered on screen on a joGL Canvas. In this case there is virtually no overhead when exporting this Canvas to a bitmap image format (png,jpeg,gif,...). In headless mode the rendering cannot use joGL and there is the same overhead as when you try to export a figure to a vectorial format (svg,pdf,eps) :

nx=4000;
ny=300;
nc=100;

x=1:nx;
y=[1:ny].';
z=(y*x);

driver Rec
h=scf();
h.color_map=hotcolormap(nc);
Sgrayplot(x,y,z.');
xs2pdf(h,"screen.pdf")

at line    13 of executed file /Users/mottelet/Desktop/xend_test.sce

xs2pdf: Unable to create export file, not enough memory. Decreasing the number of elements or the size of the figure should fix this error.

S.

Le 03/12/2020 à 09:48, Antoine Monmayrant a écrit :

Hello all,

In these days of remote working, I need to use again scilab in headless mode to generate graphics.

In other words, I connect  in command line only to a remote server where scilab is installed and I want it to generate graphics using a combination of driver, xinit and xend.

There seems to be some issues with driver/xinit/xend: huge memory overhead and long computation time that makes it impossible to generate some plots in headless mode.

For example, a simple Sgrayplot that takes ~1 second to plot and save to a png in normal mode takes more than 200 seconds and huge amount of ram in headless mode (see attached test script).

I also have some plots that works without a problem in normal mode, but fail in headless mode because of some memory limitation:

xend: An error occurred: Unable to create export file, not enough memory. Decreasing the number of elements or the size of the figure should fix this error.

Is there a way to get around these problems?

Thanks for your help,

Antoine







_______________________________________________
users mailing list
[hidden email]
https://antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/lists.scilab.org/mailman/listinfo/users
-- 
Stéphane Mottelet
Ingénieur de recherche
EA 4297 Transformations Intégrées de la Matière Renouvelable
Département Génie des Procédés Industriels
Sorbonne Universités - Université de Technologie de Compiègne
CS 60319, 60203 Compiègne cedex
Tel : +33(0)344234688
http://www.utc.fr/~mottelet

_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users
Antoine Monmayrant Antoine Monmayrant
Reply | Threaded
Open this post in threaded view
|

Re: Headless graphics in scilab: time & memory issues


On 03/12/2020 10:46, Stéphane Mottelet wrote:
> In headless mode the rendering cannot use joGL and there is the same
> overhead as when you try to export a figure to a vectorial format
> (svg,pdf,eps) :


OK thanks for the clarification.
That's a weird and unexpected limitation, no?
I would not have guessed that going headless and using a bitmap format
would result in facing this limitation.
Moreover, there is still a memory issue as I can crash scilab when
pushing things a bit further.

Antoine

_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users
mottelet mottelet
Reply | Threaded
Open this post in threaded view
|

Re: Headless graphics in scilab: time & memory issues


Le 03/12/2020 à 11:01, Antoine Monmayrant a écrit :
>
> On 03/12/2020 10:46, Stéphane Mottelet wrote:
>> In headless mode the rendering cannot use joGL and there is the same
>> overhead as when you try to export a figure to a vectorial format
>> (svg,pdf,eps) :
>
>
> OK thanks for the clarification.
> That's a weird and unexpected limitation, no?

No. In fact fast offscreen rendering is possible in joGL. In my previous
answer I should have said "In headless mode the rendering *does not* use
joGL". The pity is that the main author of Scilab Graphics Java backend
has left the team long time ago. It would have been an easy task for him.

If your project is hosted on a Linux server, I would advise you to
redirect the display of your Scilab instance to a virtual frame buffer
by using any flavor of VNC (TurboVNC is nice as the server can use the
hardware GPU capabilities of the server). In this case you can stick to
the Rec driver and j xs2png (or any bitmap variant) will be fast. That's
what I used with xmllab many years ago...

S.

> I would not have guessed that going headless and using a bitmap format
> would result in facing this limitation.
> Moreover, there is still a memory issue as I can crash scilab when
> pushing things a bit further.
>
> Antoine
>
> _______________________________________________
> users mailing list
> [hidden email]
> https://antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/lists.scilab.org/mailman/listinfo/users 
>

--
Stéphane Mottelet
Ingénieur de recherche
EA 4297 Transformations Intégrées de la Matière Renouvelable
Département Génie des Procédés Industriels
Sorbonne Universités - Université de Technologie de Compiègne
CS 60319, 60203 Compiègne cedex
Tel : +33(0)344234688
http://www.utc.fr/~mottelet

_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users
mottelet mottelet
Reply | Threaded
Open this post in threaded view
|

Re: Headless graphics in scilab: time & memory issues

The xmllab is actually still working, try e.g.

http://marguerite.lmac.utc.fr/~xmllab/cgi-bin/XMLlabServer.cgi?timing=no&lang=english&simulation=/home/xmllab/scilab-5.5.2/share/scilab/contrib/xmllab/1.7.6//examples/Image-Processing/clown.xml&action=launch

S.

Le 03/12/2020 à 11:20, Stéphane Mottelet a écrit :

>
> Le 03/12/2020 à 11:01, Antoine Monmayrant a écrit :
>>
>> On 03/12/2020 10:46, Stéphane Mottelet wrote:
>>> In headless mode the rendering cannot use joGL and there is the same
>>> overhead as when you try to export a figure to a vectorial format
>>> (svg,pdf,eps) :
>>
>>
>> OK thanks for the clarification.
>> That's a weird and unexpected limitation, no?
>
> No. In fact fast offscreen rendering is possible in joGL. In my
> previous answer I should have said "In headless mode the rendering
> *does not* use joGL". The pity is that the main author of Scilab
> Graphics Java backend has left the team long time ago. It would have
> been an easy task for him.
>
> If your project is hosted on a Linux server, I would advise you to
> redirect the display of your Scilab instance to a virtual frame buffer
> by using any flavor of VNC (TurboVNC is nice as the server can use the
> hardware GPU capabilities of the server). In this case you can stick
> to the Rec driver and j xs2png (or any bitmap variant) will be fast.
> That's what I used with xmllab many years ago...
>
> S.
>
>> I would not have guessed that going headless and using a bitmap
>> format would result in facing this limitation.
>> Moreover, there is still a memory issue as I can crash scilab when
>> pushing things a bit further.
>>
>> Antoine
>>
>> _______________________________________________
>> users mailing list
>> [hidden email]
>> https://antispam.utc.fr/proxy/2/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/lists.scilab.org/mailman/listinfo/users 
>>
>
--
Stéphane Mottelet
Ingénieur de recherche
EA 4297 Transformations Intégrées de la Matière Renouvelable
Département Génie des Procédés Industriels
Sorbonne Universités - Université de Technologie de Compiègne
CS 60319, 60203 Compiègne cedex
Tel : +33(0)344234688
http://www.utc.fr/~mottelet

_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users
Samuel GOUGEON Samuel GOUGEON
Reply | Threaded
Open this post in threaded view
|

Re: Headless graphics in scilab: time & memory issues

In reply to this post by Antoine Monmayrant
Le 03/12/2020 à 09:59, Antoine Monmayrant a écrit :

Hello again,

Just to add to my previous message: increasing nx or ny in my script can crash scilab during the call to xend().
On my small local machine, this occurs for:

nx=4000;
ny=3000;

for which scilab takes ~7 seconds to create the png in "normal" mode...

Can any of you confirm this bug?


The crash is due to a too small Java Heap of memory.
When increasing it through the user's Preferences from 256 MB to 1 GB, the PNG export works as fast as expected.
I get i nice png image from 4000x3000 with
--> tic(); Sgrayplot(x,y,z.'); toc()
 ans  =
   8.1342045

--> tic(); xs2png(0, "test.png"); toc
 ans  =
   0.4060765

Best regards
Samuel


_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users
Samuel GOUGEON Samuel GOUGEON
Reply | Threaded
Open this post in threaded view
|

Re: Headless graphics in scilab: time & memory issues

In reply to this post by Antoine Monmayrant
figure("visible","off")  allows to avoid using driver(), but the same memory consumption is then observable, when exporting the Sgrayploy with xs2png().
By the way, at least on my system, the final figure.png file is reddish.
nx=4000;
ny=300;
nc=100;

x=1:nx;
y=[1:ny].';
z=(y*x);

h = figure("visible","off");
h.color_map=hotcolormap(nc);
Sgrayplot(x,y,z.');
xs2png(h,"C:/windows/temp/figure.png")


Le 03/12/2020 à 09:48, Antoine Monmayrant a écrit :

Hello all,

In these days of remote working, I need to use again scilab in headless mode to generate graphics.

In other words, I connect  in command line only to a remote server where scilab is installed and I want it to generate graphics using a combination of driver, xinit and xend.

There seems to be some issues with driver/xinit/xend: huge memory overhead and long computation time that makes it impossible to generate some plots in headless mode.

For example, a simple Sgrayplot that takes ~1 second to plot and save to a png in normal mode takes more than 200 seconds and huge amount of ram in headless mode (see attached test script).

I also have some plots that works without a problem in normal mode, but fail in headless mode because of some memory limitation:

xend: An error occurred: Unable to create export file, not enough memory. Decreasing the number of elements or the size of the figure should fix this error.

Is there a way to get around these problems?

Thanks for your help,

Antoine







_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users



_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users