[Scilab-users] feedback on removing 1 row and 1 column

classic Classic list List threaded Threaded
8 messages Options
Carrico, Paul-2 Carrico, Paul-2
Reply | Threaded
Open this post in threaded view
|

[Scilab-users] feedback on removing 1 row and 1 column

Dear All

 

I’ve been using successfully Scilab for years, to develop my projects; I do not remember if I’ve ever had to deal with huge matrixes but that’s the case today.

 

Using “csvRead”, I got a matrix with 2.3 million of rows and 6 columns (it took about 240 seconds to read it J); nevertheless I’ve been surprised when I tried to remove the first row and the last column using basically:

MAT(1,:) = []; MAT(:,$) = [];

 

Indeed I needed to kill the process after about 10 minutes (same result both under Windows 10 and Linux – RAM not fully used).

 

By comparison, the same process is instantaneous under Python using Numpy (numpy.delete) using of course the same matrix.

 

A feedback I wanted to share ….

 

PaulEXPORT CONTROL :

« Cet email ne contient pas de données techniques »

« This email does not contain technical data »

 

 


_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users
mottelet mottelet
Reply | Threaded
Open this post in threaded view
|

Re: feedback on removing 1 row and 1 column

Le 30/04/2019 à 08:44, Carrico, Paul a écrit :

Dear All

 

I’ve been using successfully Scilab for years, to develop my projects; I do not remember if I’ve ever had to deal with huge matrixes but that’s the case today.

 

Using “csvRead”, I got a matrix with 2.3 million of rows and 6 columns (it took about 240 seconds to read it J); nevertheless I’ve been surprised when I tried to remove the first row and the last column using basically:

This one is quasi instantaneous, although it needs to move all the variable content in memory

MAT(1,:) = [];

But this stucks Scilab, you are right:

MAT(:,$) = [];

athough it should the faster of both (large number of rows but no copy needed, trim the block in memory should be enough).

@Samuel: I think we already have something similar signaled in Bugzilla

S.

 

Indeed I needed to kill the process after about 10 minutes (same result both under Windows 10 and Linux – RAM not fully used).

 

By comparison, the same process is instantaneous under Python using Numpy (numpy.delete) using of course the same matrix.

 

A feedback I wanted to share ….

 

PaulEXPORT CONTROL :

« Cet email ne contient pas de données techniques »

« This email does not contain technical data »

 

 


_______________________________________________
users mailing list
[hidden email]
https://antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/lists.scilab.org/mailman/listinfo/users


-- 
Stéphane Mottelet
Ingénieur de recherche
EA 4297 Transformations Intégrées de la Matière Renouvelable
Département Génie des Procédés Industriels
Sorbonne Universités - Université de Technologie de Compiègne
CS 60319, 60203 Compiègne cedex
Tel : +33(0)344234688
http://www.utc.fr/~mottelet

_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users
Alain LAMY Alain LAMY
Reply | Threaded
Open this post in threaded view
|

Re: feedback on removing 1 row and 1 column

Hi,

>>@Samuel: I think we already have something similar signaled in Bugzilla

I think the bug is this one:

https://bugzilla.scilab.org/show_bug.cgi?id=15087

 

Alain

 


_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users
mottelet mottelet
Reply | Threaded
Open this post in threaded view
|

Re: feedback on removing 1 row and 1 column

Thank you Alain.

S.

Le 30/04/2019 à 10:37, Lamy Alain a écrit :

Hi,

>>@Samuel: I think we already have something similar signaled in Bugzilla

I think the bug is this one:

https://bugzilla.scilab.org/show_bug.cgi?id=15087

 

Alain

 


_______________________________________________
users mailing list
[hidden email]
https://antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/lists.scilab.org/mailman/listinfo/users


-- 
Stéphane Mottelet
Ingénieur de recherche
EA 4297 Transformations Intégrées de la Matière Renouvelable
Département Génie des Procédés Industriels
Sorbonne Universités - Université de Technologie de Compiègne
CS 60319, 60203 Compiègne cedex
Tel : +33(0)344234688
http://www.utc.fr/~mottelet

_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users
mottelet mottelet
Reply | Threaded
Open this post in threaded view
|

Re: feedback on removing 1 row and 1 column

In reply to this post by Carrico, Paul-2
Le 30/04/2019 à 08:44, Carrico, Paul a écrit :

Dear All

 

I’ve been using successfully Scilab for years, to develop my projects; I do not remember if I’ve ever had to deal with huge matrixes but that’s the case today.

 

Using “csvRead”, I got a matrix with 2.3 million of rows and 6 columns (it took about 240 seconds to read it J); nevertheless I’ve been surprised when I tried to remove the first row and the last column using basically:

MAT(1,:) = []; MAT(:,$) = [];

in the meantime (before we fix this), you can do

MAT = MAT(:,1:$-1);

which will be way faster !

S.

 

Indeed I needed to kill the process after about 10 minutes (same result both under Windows 10 and Linux – RAM not fully used).

 

By comparison, the same process is instantaneous under Python using Numpy (numpy.delete) using of course the same matrix.

 

A feedback I wanted to share ….

 

PaulEXPORT CONTROL :

« Cet email ne contient pas de données techniques »

« This email does not contain technical data »

 

 


_______________________________________________
users mailing list
[hidden email]
https://antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/lists.scilab.org/mailman/listinfo/users


-- 
Stéphane Mottelet
Ingénieur de recherche
EA 4297 Transformations Intégrées de la Matière Renouvelable
Département Génie des Procédés Industriels
Sorbonne Universités - Université de Technologie de Compiègne
CS 60319, 60203 Compiègne cedex
Tel : +33(0)344234688
http://www.utc.fr/~mottelet

_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users
Carrico, Paul-2 Carrico, Paul-2
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: feedback on removing 1 row and 1 column

Thanks all

 

Indeed, getting the complete block works fine and it’s instantaneous

 

Paul

 

De : users [mailto:[hidden email]] De la part de Stéphane Mottelet
Envoyé : mardi 30 avril 2019 12:41
À : [hidden email]
Objet : [EXTERNAL] Re: [Scilab-users] feedback on removing 1 row and 1 column

 

Le 30/04/2019 à 08:44, Carrico, Paul a écrit :

Dear All

 

I’ve been using successfully Scilab for years, to develop my projects; I do not remember if I’ve ever had to deal with huge matrixes but that’s the case today.

 

Using “csvRead”, I got a matrix with 2.3 million of rows and 6 columns (it took about 240 seconds to read it J); nevertheless I’ve been surprised when I tried to remove the first row and the last column using basically:

MAT(1,:) = []; MAT(:,$) = [];

in the meantime (before we fix this), you can do

MAT = MAT(:,1:$-1);

which will be way faster !

S.

 

Indeed I needed to kill the process after about 10 minutes (same result both under Windows 10 and Linux – RAM not fully used).

 

By comparison, the same process is instantaneous under Python using Numpy (numpy.delete) using of course the same matrix.

 

A feedback I wanted to share ….

 

PaulEXPORT CONTROL :

« Cet email ne contient pas de données techniques »

« This email does not contain technical data »

 

 



_______________________________________________
users mailing list
[hidden email]
https://antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/lists.scilab.org/mailman/listinfo/users

 

-- 
Stéphane Mottelet
Ingénieur de recherche
EA 4297 Transformations Intégrées de la Matière Renouvelable
Département Génie des Procédés Industriels
Sorbonne Universités - Université de Technologie de Compiègne
CS 60319, 60203 Compiègne cedex
Tel : +33(0)344234688
http://www.utc.fr/~mottelet

_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users
mottelet mottelet
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: feedback on removing 1 row and 1 column

Hello Paul,

A fix is under review for master at https://codereview.scilab.org/#/c/20965/, I hope it will be reviewed and merged soon (timings are done on my machine, an old mid-2010 MacPro) :

--> MAT=rand(2.3e6,6);
--> tic;MAT(:,$)=[];toc
 ans  =

   0.051402

--> MAT=rand(2.3e6,6);
--> tic;MAT(:,[1 3])=[];toc
  ans  =

   0.046928

S.

Le 30/04/2019 à 14:04, Carrico, Paul a écrit :

Thanks all

 

Indeed, getting the complete block works fine and it’s instantaneous

 

Paul

 

De : users [[hidden email]] De la part de Stéphane Mottelet
Envoyé : mardi 30 avril 2019 12:41
À : [hidden email]
Objet : [EXTERNAL] Re: [Scilab-users] feedback on removing 1 row and 1 column

 

Le 30/04/2019 à 08:44, Carrico, Paul a écrit :

Dear All

 

I’ve been using successfully Scilab for years, to develop my projects; I do not remember if I’ve ever had to deal with huge matrixes but that’s the case today.

 

Using “csvRead”, I got a matrix with 2.3 million of rows and 6 columns (it took about 240 seconds to read it J); nevertheless I’ve been surprised when I tried to remove the first row and the last column using basically:

MAT(1,:) = []; MAT(:,$) = [];

in the meantime (before we fix this), you can do

MAT = MAT(:,1:$-1);

which will be way faster !

S.

 

Indeed I needed to kill the process after about 10 minutes (same result both under Windows 10 and Linux – RAM not fully used).

 

By comparison, the same process is instantaneous under Python using Numpy (numpy.delete) using of course the same matrix.

 

A feedback I wanted to share ….

 

PaulEXPORT CONTROL :

« Cet email ne contient pas de données techniques »

« This email does not contain technical data »

 

 



_______________________________________________
users mailing list
[hidden email]
https://antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/lists.scilab.org/mailman/listinfo/users

 

-- 
Stéphane Mottelet
Ingénieur de recherche
EA 4297 Transformations Intégrées de la Matière Renouvelable
Département Génie des Procédés Industriels
Sorbonne Universités - Université de Technologie de Compiègne
CS 60319, 60203 Compiègne cedex
Tel : +33(0)344234688
http://www.utc.fr/~mottelet

_______________________________________________
users mailing list
[hidden email]
https://antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/lists.scilab.org/mailman/listinfo/users


-- 
Stéphane Mottelet
Ingénieur de recherche
EA 4297 Transformations Intégrées de la Matière Renouvelable
Département Génie des Procédés Industriels
Sorbonne Universités - Université de Technologie de Compiègne
CS 60319, 60203 Compiègne cedex
Tel : +33(0)344234688
http://www.utc.fr/~mottelet

_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users
Carrico, Paul-2 Carrico, Paul-2
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: feedback on removing 1 row and 1 column

Thanks Stéphane for your support

 

Paul

 

De : users [mailto:[hidden email]] De la part de Stéphane Mottelet
Envoyé : vendredi 3 mai 2019 14:35
À : [hidden email]
Objet : Re: [Scilab-users] [EXTERNAL] Re: feedback on removing 1 row and 1 column

 

Hello Paul,

 

A fix is under review for master at https://codereview.scilab.org/#/c/20965/, I hope it will be reviewed and merged soon (timings are done on my machine, an old mid-2010 MacPro) :

 

--> MAT=rand(2.3e6,6);
--> tic;MAT(:,$)=[];toc
 ans  =

   0.051402

 

--> MAT=rand(2.3e6,6);
--> tic;MAT(:,[1 3])=[];toc

  ans  =

   0.046928

 

S.

 

Le 30/04/2019 à 14:04, Carrico, Paul a écrit :

Thanks all

 

Indeed, getting the complete block works fine and it’s instantaneous

 

Paul

 

De : users [[hidden email]] De la part de Stéphane Mottelet
Envoyé : mardi 30 avril 2019 12:41
À : [hidden email]
Objet : [EXTERNAL] Re: [Scilab-users] feedback on removing 1 row and 1 column

 

Le 30/04/2019 à 08:44, Carrico, Paul a écrit :

Dear All

 

I’ve been using successfully Scilab for years, to develop my projects; I do not remember if I’ve ever had to deal with huge matrixes but that’s the case today.

 

Using “csvRead”, I got a matrix with 2.3 million of rows and 6 columns (it took about 240 seconds to read it J); nevertheless I’ve been surprised when I tried to remove the first row and the last column using basically:

MAT(1,:) = []; MAT(:,$) = [];

in the meantime (before we fix this), you can do

MAT = MAT(:,1:$-1);

which will be way faster !

S.

 

Indeed I needed to kill the process after about 10 minutes (same result both under Windows 10 and Linux – RAM not fully used).

 

By comparison, the same process is instantaneous under Python using Numpy (numpy.delete) using of course the same matrix.

 

A feedback I wanted to share ….

 

PaulEXPORT CONTROL :

« Cet email ne contient pas de données techniques »

« This email does not contain technical data »

 

 




_______________________________________________
users mailing list
[hidden email]
https://antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/lists.scilab.org/mailman/listinfo/users

 

-- 
Stéphane Mottelet
Ingénieur de recherche
EA 4297 Transformations Intégrées de la Matière Renouvelable
Département Génie des Procédés Industriels
Sorbonne Universités - Université de Technologie de Compiègne
CS 60319, 60203 Compiègne cedex
Tel : +33(0)344234688
http://www.utc.fr/~mottelet



_______________________________________________
users mailing list
[hidden email]
https://antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/lists.scilab.org/mailman/listinfo/users

 

-- 
Stéphane Mottelet
Ingénieur de recherche
EA 4297 Transformations Intégrées de la Matière Renouvelable
Département Génie des Procédés Industriels
Sorbonne Universités - Université de Technologie de Compiègne
CS 60319, 60203 Compiègne cedex
Tel : +33(0)344234688
http://www.utc.fr/~mottelet

_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users