"\r\n" in output from a gcc compiled app and shell input

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

"\r\n" in output from a gcc compiled app and shell input

Ian Puleston
Hi,

I have an XML parsing program that I wrote, and have an application of it
where it is run from a shell script. In that shell script I set a shell
variable to a list of words that the program outputs in parsing the XML
file, which it prints one-per line, and the shell script line to do that
looks like this:
    words=`xmlParse .....`
    for word in ${words}; do
        ...
    Done

This works fine when I test it running it from bash. However, when its used
in the field its run from a minimal msys install with sh.exe but no bash,
and when run from sh.exe it doesn't work. The problem is that the xmlParse
program, which I compile under msys with gcc 4.3, outputs a "\r\n" on the
end of each line of output and it seems that bash accepts the '\r' in
processing the "words=`xmlParse .....`" line, but sh.exe does not. To work
around the problem I had to change that line to:
    words=`xmlParse ..... | sed 's/\r//'`

This seems a bit messed up (OK, admittedly the whole Windows newline thing
is messed up) but I would have thought that a program compiled with default
settings under msys would produce output that would work with the msys shell
by default.

Is there a way to tell gcc to compile the program to generate Unix style
line endings in output on a stream opened in text mode (I can't find any
option for that) or a way to tell sh.exe to treat "\r\n" as "\r"?

Ian








------------------------------------------------------------------------------
Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM)
software. With Adobe AIR, Ajax developers can use existing skills and code to
build responsive, highly engaging applications that combine the power of local
resources and data with the reach of the web. Download the Adobe AIR SDK and
Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com
_______________________________________________
MinGW-users mailing list
[hidden email]

You may change your MinGW Account Options or unsubscribe at:
https://lists.sourceforge.net/lists/listinfo/mingw-users

_______________________________________________
This list observes the Etiquette found at
http://www.mingw.org/Mailing_Lists.
We ask that you be polite and do the same.

Most annoying abuses are:
1) Top posting
2) HTML/MIME encoded mail
3) Improper quoting
4) Improper trimming
Reply | Threaded
Open this post in threaded view
|

Re: "\r\n" in output from a gcc compiled app and shell input

Keith Marshall
On Tuesday 03 February 2009 20:01:23 Ian Puleston wrote:
> Hi,
>
> I have an XML parsing program that I wrote, ...
>
> This works fine when I test it running it from bash. However, when
> its used in the field its run from a minimal msys install with
> sh.exe but no bash, and when run from sh.exe it doesn't work.

MSYS' sh.exe *is* bash, (and AFAIK, it always has been), although by
invoking it as `sh' it runs in `sh' compatibility mode; (for more on
this, see the bash man page -- search for it online, if you don't
have a local copy).  Even on your minimal MSYS install, you can

  ( cd /bin && ln sh.exe bash.exe )

to get a fully featured bash implementation.

> The problem is that the xmlParse program, which I compile under msys
> with gcc 4.3, ...

It's mostly irrelevant that you compile under MSYS; unless you use
msysDVLPR, (which would mandate gcc-2.95.3), you are still creating a
native MSW binary.

> .. outputs a "\r\n" on the end of each line of output
> and it seems that bash accepts the '\r' in processing the
> "words=`xmlParse .....`" line, but sh.exe does not.

More likely, I think, that you have different *versions* of sh.exe on
the two systems.  IIRC, MSYS' bash-3.x doesn't deal with `\r\n' line
endings in quite the same way as the older bash-2.x in MSYS-1.0.10,
and it does avoid some bugs in the bash-2.x filtering code -- there
is at least one known, and unresolved bug in bash-2.x, which is
specific to command substitution with CRLF output.

> To work around the problem I had to change that line to:
>     words=`xmlParse ..... | sed 's/\r//'`

Better, surely, would be:

  words=`xmlParse ... | dos2unix`

You will find dos2unix in the mingw-utils package, or a minimal, less
sophisticated implementation at: http://preview.tinyurl.com/aqcjyx

> This seems a bit messed up (OK, admittedly the whole Windows
> newline thing is messed up) but I would have thought that a program
> compiled with default settings under msys would produce output that
> would work with the msys shell by default.

Why would you think that?  Unless you are running msysDVLPR, (which is
only intended for those who actually *develop* MSYS itself), MSYS is
no more than a convenient command line environment, which, among
other things, may be used for building native MSW applications, with
conventions and attributes common to all such applications.

> Is there a way to tell gcc to compile the program to generate Unix
> style line endings in output on a stream opened in text mode

Text mode, binary mode; who cares?  Unix makes no such distinction.  
Open the stream in binary mode, and write text -- *you* get to
control whether the line ending is CRLF, or just LF, without any
interference from the runtime I/O routines.

> (I can't find any option for that)

For the pre-opened stdin, stdout and stderr streams ...

1) link with /mingw/lib/binmode.o, or
2) call _setmode( fd, _O_BINARY ) before using the stream

For streams you open explicitly, just do so in _O_BINARY mode.

> or a way to tell sh.exe to treat "\r\n" as "\r"?

No, I don't know of any way to do that either.

--

Regards,
Keith.

------------------------------------------------------------------------------
Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM)
software. With Adobe AIR, Ajax developers can use existing skills and code to
build responsive, highly engaging applications that combine the power of local
resources and data with the reach of the web. Download the Adobe AIR SDK and
Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com
_______________________________________________
MinGW-users mailing list
[hidden email]

You may change your MinGW Account Options or unsubscribe at:
https://lists.sourceforge.net/lists/listinfo/mingw-users

_______________________________________________
This list observes the Etiquette found at
http://www.mingw.org/Mailing_Lists.
We ask that you be polite and do the same.

Most annoying abuses are:
1) Top posting
2) HTML/MIME encoded mail
3) Improper quoting
4) Improper trimming
Reply | Threaded
Open this post in threaded view
|

Re: "\r\n" in output from a gcc compiled app and shellinput

Weddington, Eric
In reply to this post by Ian Puleston
 

> -----Original Message-----
> From: Ian Puleston [mailto:[hidden email]]
> Sent: Tuesday, February 03, 2009 1:01 PM
> To: 'MinGW Users List'
> Subject: [Mingw-users] "\r\n" in output from a gcc compiled
> app and shellinput
>
> Hi,
>
> I have an XML parsing program that I wrote,

As a side note, why reinvent the wheel?:
<https://sourceforge.net/projects/xmlstar>

------------------------------------------------------------------------------
Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM)
software. With Adobe AIR, Ajax developers can use existing skills and code to
build responsive, highly engaging applications that combine the power of local
resources and data with the reach of the web. Download the Adobe AIR SDK and
Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com
_______________________________________________
MinGW-users mailing list
[hidden email]

You may change your MinGW Account Options or unsubscribe at:
https://lists.sourceforge.net/lists/listinfo/mingw-users

_______________________________________________
This list observes the Etiquette found at
http://www.mingw.org/Mailing_Lists.
We ask that you be polite and do the same.

Most annoying abuses are:
1) Top posting
2) HTML/MIME encoded mail
3) Improper quoting
4) Improper trimming
Reply | Threaded
Open this post in threaded view
|

Re: "\r\n" in output from a gcc compiled app and shellinput

Ian Puleston
> -----Original Message-----
> From: Weddington, Eric
>
> > I have an XML parsing program that I wrote,
>
> As a side note, why reinvent the wheel?:
> <https://sourceforge.net/projects/xmlstar>

Thanks - before embarking on it I spent some time searching on the web and
couldn't find anything, so I wish I'd known about that.

> -----Original Message-----
> From: Keith Marshall
>
> > This works fine when I test it running it from bash.
> > and when run from sh.exe it doesn't work.
>
> MSYS' sh.exe *is* bash, (and AFAIK, it always has been),
>
> > .. outputs a "\r\n" on the end of each line of output
> > and it seems that bash accepts the '\r' in processing the
> > "words=`xmlParse .....`" line, but sh.exe does not.
>
> More likely, I think, that you have different *versions* of sh.exe on
> the two systems.

Thanks for that info. I searched for all msys sh.exe files on my system and
they all report their version as "bash version 3.1.0". So I tried removing
the sed call that I'd added to my script to strip out the '\r' and now I
can't get the problem to happen again. So whatever was causing it, it
apparently wasn't bash versus sh after all.

Thanks,
Ian





------------------------------------------------------------------------------
Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM)
software. With Adobe AIR, Ajax developers can use existing skills and code to
build responsive, highly engaging applications that combine the power of local
resources and data with the reach of the web. Download the Adobe AIR SDK and
Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com
_______________________________________________
MinGW-users mailing list
[hidden email]

You may change your MinGW Account Options or unsubscribe at:
https://lists.sourceforge.net/lists/listinfo/mingw-users

_______________________________________________
This list observes the Etiquette found at
http://www.mingw.org/Mailing_Lists.
We ask that you be polite and do the same.

Most annoying abuses are:
1) Top posting
2) HTML/MIME encoded mail
3) Improper quoting
4) Improper trimming