[Scilab-users] Trouble with reading whitespace separated csv

classic Classic list List threaded Threaded
5 messages Options
Richard llom Richard llom
Reply | Threaded
Open this post in threaded view
|

[Scilab-users] Trouble with reading whitespace separated csv

Hello,
I'm trying to read in a file with whitespace separated values, with
[log_data, comments] = csvRead(log_file,ascii(32),'.','double');
However I'm constantly getting this error:
Warning: Inconsistency found in the columns. At line 2, found 11 columns
while the previous had 5.
but with changing column numbers...

Any clue whats going on?
richard


Data sample:
16800 23200  1  1  1  -4.2 1009 143  2.2 3  2.5  92    0    0 191 -292  2
16800 23200  1  1  2  -5.4 1009 178  1.8 1  2.5  96    0    0 189 -291  2
16800 23200  1  1  3  -6.1 1008 195  1.6 0  2.4  99    0    0 188 -289  2
16800 23200  1  1  4  -6.1 1008 182  1.6 0  2.4  99    0    0 182 -288  2
16800 23200  1  1  5  -5.9 1008 168  1.9 0  2.4  99    0    0 185 -286  2
16800 23200  1  1  6  -6.4 1007 146  1.8 0  2.4  99    0    0 182 -284  2
16800 23200  1  1  7  -4.7 1006 157  3.2 0  2.5  98    0    0 175 -282  2
16800 23200  1  1  8  -4.2 1006 156  3.5 3  2.6  95    0    0 196 -282  2
16800 23200  1  1  9  -3.9 1006 164  3.9 3  2.6  92    0    0 216 -281  2
16800 23200  1  1 10  -3.4 1005 151  4.0 5  2.6  89    1   28 230 -282  2
16800 23200  1  1 11  -3.1 1004 129  3.5 6  2.6  86   32   70 247 -285  2
16800 23200  1  1 12  -1.4 1004 148  5.2 4  2.6  80  146   54 245 -289  2
16800 23200  1  1 13  -0.8 1003 150  5.1 6  2.6  74   24   97 248 -294  2
...



--
Sent from: http://mailinglists.scilab.org/Scilab-users-Mailing-Lists-Archives-f2602246.html
_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users
Rafael Guerra Rafael Guerra
Reply | Threaded
Open this post in threaded view
|

Re: Trouble with reading whitespace separated csv

Hi,

 

fscanfMat reads your sample file with not problem:

 

[log_data, comments] = fscanfMat(log_file)

 

csvRead has several bugs reported in Scilab 5 series, your case may be one.

 

Regards,

Rafael

 

-----Original Message-----
From: users [mailto:[hidden email]] On Behalf Of Richard llom
Sent: Friday, November 17, 2017 5:42 PM
To: [hidden email]
Subject: [Scilab-users] Trouble with reading whitespace separated csv

 

Hello,

I'm trying to read in a file with whitespace separated values, with

[log_data, comments] = csvRead(log_file,ascii(32),'.','double');

However I'm constantly getting this error:

Warning: Inconsistency found in the columns. At line 2, found 11 columns

while the previous had 5.

but with changing column numbers...

 

Any clue whats going on?

richard

 

 

Data sample:

16800 23200  1  1  1  -4.2 1009 143  2.2 3  2.5  92    0    0 191 -292  2

16800 23200  1  1  2  -5.4 1009 178  1.8 1  2.5  96    0    0 189 -291  2

16800 23200  1  1  3  -6.1 1008 195  1.6 0  2.4  99    0    0 188 -289  2

16800 23200  1  1  4  -6.1 1008 182  1.6 0  2.4  99    0    0 182 -288  2

16800 23200  1  1  5  -5.9 1008 168  1.9 0  2.4  99    0    0 185 -286  2

16800 23200  1  1  6  -6.4 1007 146  1.8 0  2.4  99    0    0 182 -284  2

16800 23200  1  1  7  -4.7 1006 157  3.2 0  2.5  98    0    0 175 -282  2

16800 23200  1  1  8  -4.2 1006 156  3.5 3  2.6  95    0    0 196 -282  2

16800 23200  1  1  9  -3.9 1006 164  3.9 3  2.6  92    0    0 216 -281  2

16800 23200  1  1 10  -3.4 1005 151  4.0 5  2.6  89    1   28 230 -282  2

16800 23200  1  1 11  -3.1 1004 129  3.5 6  2.6  86   32   70 247 -285  2

16800 23200  1  1 12  -1.4 1004 148  5.2 4  2.6  80  146   54 245 -289  2

16800 23200  1  1 13  -0.8 1003 150  5.1 6  2.6  74   24   97 248 -294  2

...

 


_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users
Samuel GOUGEON Samuel GOUGEON
Reply | Threaded
Open this post in threaded view
|

Re: Trouble with reading whitespace separated csv

In reply to this post by Richard llom
Hello Richard,

Le 17/11/2017 à 17:41, Richard llom a écrit :
Hello,
I'm trying to read in a file with whitespace separated values, with
[log_data, comments] = csvRead(log_file,ascii(32),'.','double');
However I'm constantly getting this error:
Warning: Inconsistency found in the columns. At line 2, found 11 columns
while the previous had 5.
but with changing column numbers...

Any clue whats going on?

Using space as a column separator is a very bad choice. For instance, in the sample that you have posted, the 3 consecutive following rows

16800 23200  1  1 10  -3.4 1005 151  4.0 5  2.6  89    1   28 230 -282  2
16800 23200  1  1 11  -3.1 1004 129  3.5 6  2.6  86   32   70 247 -285  2
16800 23200  1  1 12  -1.4 1004 148  5.2 4  2.6  80  146   54 245 -289  2

are then read by csvRead() as (if "," were the separator instead)
16800,23200,,1,,1,10,,-3.4,1005,151,,4.0 5,,2.6,,89,,,,1,,,28,230,-282,,2
16800,23200,,1,,1,11,,-3.1,1004,129,,3.5 6,,2.6,,86,,,32,,,70,247,-285,,2
16800,23200,,1,,1,12,,-1.4,1004,148,,5.2 4,,2.6,,80,,146,,,54,245,-289,,2

with many empty columns and a shift, that yields an expected error.
So, there is no bug here from csvRead(). As noted by Rafael, csvRead() does not fit to the
job here. Since your data are only numerical, fscanfMat() should work instead, indeed.

Samuel


_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users
Rafael Guerra Rafael Guerra
Reply | Threaded
Open this post in threaded view
|

Re: Trouble with reading whitespace separated csv

Hi Samuel,

 

Not a bug but maybe it could be enhanced for the conversion=’double’ case by treating consecutive spaces or tabs as a single separator?

 

Regards,

Rafael

 

From: users [mailto:[hidden email]] On Behalf Of Samuel Gougeon
Sent: Saturday, November 18, 2017 5:07 PM
To: Users mailing list for Scilab <[hidden email]>
Subject: Re: [Scilab-users] Trouble with reading whitespace separated csv

 

Hello Richard,

Le 17/11/2017 à 17:41, Richard llom a écrit :

Hello,
I'm trying to read in a file with whitespace separated values, with
[log_data, comments] = csvRead(log_file,ascii(32),'.','double');
However I'm constantly getting this error:
Warning: Inconsistency found in the columns. At line 2, found 11 columns
while the previous had 5.
but with changing column numbers...
 
Any clue whats going on?


Using space as a column separator is a very bad choice. For instance, in the sample that you have posted, the 3 consecutive following rows


16800 23200  1  1 10  -3.4 1005 151  4.0 5  2.6  89    1   28 230 -282  2
16800 23200  1  1 11  -3.1 1004 129  3.5 6  2.6  86   32   70 247 -285  2
16800 23200  1  1 12  -1.4 1004 148  5.2 4  2.6  80  146   54 245 -289  2


are then read by csvRead() as (if "," were the separator instead)
16800,23200,,1,,1,10,,-3.4,1005,151,,4.0 5,,2.6,,89,,
,,1,,,28,230,-282,,2
16800,23200,,1,,1,11,,-3.1,1004,129,,3.5 6,,2.6,,86,,
,32,,,70,247,-285,,2
16800,23200,,1,,1,12,,-1.4,1004,148,,5.2 4,,2.6,,80,,146,,,54,245,-289,,2

with many empty columns and a shift, that yields an expected error.
So, there is no bug here from csvRead(). As noted by Rafael, csvRead() does not fit to the
job here. Since your data are only numerical, fscanfMat() should work instead, indeed.

Samuel


_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users
Richard llom Richard llom
Reply | Threaded
Open this post in threaded view
|

Re: Trouble with reading whitespace separated csv

Hi all,
thanks for the replies, fscanfMat() does work fine.

I suspected something like this with the changing spaces between the values,
but had hoped csvread would concentrate multiple separators - but then again
in an csv with several ',' one would indeed expect it to treat these as
indvidual columns, so I think the behavior is fine.

Regarding the file format, it is external, so not a choice I have...
best regards
richard



--
Sent from: http://mailinglists.scilab.org/Scilab-users-Mailing-Lists-Archives-f2602246.html
_______________________________________________
users mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/users