Showing posts with label character. Show all posts
Showing posts with label character. Show all posts

Wednesday, March 28, 2012

Strange character [Â] appearing in file.

Hi All,

I have an ASP.NET application which opens a text file and then saves it again.
This saved file is then passed into another software suite (not ASP.NET) for processing.
This secondary suite expects the lines in the file to be a certain length (legacy).

All text is transferred between the files correctly, except for where a uk pound sign [£] appears in the original file.

This character is NOT transferredusing the default StreamReader/Writer.

I havetriedchanging the encoding of both the StreamReader and the StreamWriter butcannot find one which does the job as expected, changing the reader toASCII and leaving the default writer is the closest but not withoutissues.
Where thiscondition occurs, the resulting saved file, in the place where theoriginal pound sign was displayed shows 2 (!) characters ?£.

This extra ? [alt + 0194] character is not seen by Notepad, not visiblewhen debugging, but is there, as it affects the fixed length recordprocessing mentioned above, and can be seen in other text editors suchas Wordpad, DocPad etc.

Please help, this is causing some despair!

Thanks and Regards,
Ric

The following code sample demonstrates my point...
[A file c:\testInput.txt is also required with a line in it that contains a £ sign]
eg. 111111111111111£1111111111
[A file c:\testOutput.txt is created to demonstrate the point.]

Sub Main()

'file access vars
Dim fsRead As FileStream
Dim fsWrite As FileStream
Dim sr As StreamReader
Dim sw As StreamWriter

'The file to read from
Dim inputFile As String = "c:\testInput.txt"
Dim outputFile As String = "c:\testOutput.txt"

'Create the FileStream : INPUT
Try
fsRead = New FileStream(inputFile, FileMode.Open, FileAccess.Read,FileShare.Read)
Catch ex As Exception
End Try

'Create the StreamReader
Try
sr = New StreamReader(fsRead, System.Text.Encoding.ASCII)
Catch ex As Exception
End Try

'Create the FileStream : OUTPUT
Try
fsWrite = New FileStream(outputFile, FileMode.OpenOrCreate,FileAccess.Write, FileShare.ReadWrite)
Catch ex As Exception
End Try

'Create the StreamWriter
Try
sw = New StreamWriter(fsWrite)
Catch ex As Exception
End Try

Dim s As String
Dim replaced As String
While sr.Peek <> -1

'read a line form the input file
s = sr.ReadLine()

'substitute the misread ? characters into £ signs
replaced = Replace(s, "?", "£")

'move to the end of the file
sw.BaseStream.Seek(0, SeekOrigin.End)

'and write the string to the output file
sw.Write(replaced & vbCrLf)

End While

'Clear up...
Try
sw.Flush()
sw.Close()
fsWrite.Close()

sr.Close()
fsRead.Close()

Catch ex As Exception
End Try

'and dispose
sw = Nothing
sr = Nothing
fsRead = Nothing
fsWrite = Nothing

End Sub

Solution:

When reading/writing plain text files using file streams on a Windows box you should specify an Encoding.

The Encoding can be created using the following code:
[1252 is the Western Windows codepage]

Dim encAs System.Text.Encoding
enc = System.Text.Encoding.GetEncoding(1252)

You can then specify this encoding to be used when creating the StreamReader/StreamWriter.

All characters (even the uk pound sign £) will then be correctly read/written by the streams.

Hope this helps,
Ric

Strange character transformations

Hi,

I have an ASP.Net website, which allows users to upload a file which is
then inserted into a database.

This is all fine until it reads a line with the string +Anu in it.
It transforms this to this char ? (which, if Googled for, is
described as Unicode Character 'LATIN SMALL LETTER TURNED R WITH HOOK'
(U+027B) or, in Phonetics, as a 'Retroflex approximant'.)

Has anyone seen this behaviour before, and know how to stop it?
The code's simple - here's an example. The ? appears in the output
where the input is +Anu - it's transformed before I can touch it!

using (StreamReader sr = new StreamReader(strFile,
System.Text.Encoding.UTF7)) {
// Read and display lines from the file until the end of the file is
reached.
while ((line = sr.ReadLine()) != null) {
Response.Write(line);
}
}

Regards

AdamLooks like an encoding issue, alright.
Have you tried using the StreamReader constructor that does not require a
character encoding?

"CyberSpyders@.gmail.com" wrote:

Quote:

Originally Posted by

Hi,
>
I have an ASP.Net website, which allows users to upload a file which is
then inserted into a database.
>
This is all fine until it reads a line with the string +Anu in it.
It transforms this to this char ? (which, if Googled for, is
described as Unicode Character 'LATIN SMALL LETTER TURNED R WITH HOOK'
(U+027B) or, in Phonetics, as a 'Retroflex approximant'.)
>
Has anyone seen this behaviour before, and know how to stop it?
The code's simple - here's an example. The ? appears in the output
where the input is +Anu - it's transformed before I can touch it!
>
using (StreamReader sr = new StreamReader(strFile,
System.Text.Encoding.UTF7)) {
// Read and display lines from the file until the end of the file is
reached.
while ((line = sr.ReadLine()) != null) {
Response.Write(line);
}
}
>
Regards
>
Adam
>
>


Graven,

I'm not sure how a 4 letter string like this could be seen as an
encoding issue, but I will certainly give it a go. Thanks for the
suggestion.

Adam

Graven wrote:

Quote:

Originally Posted by

Try to use plain latin-1 encoding. I think it's an unicode
normalization issue, but don't know if StreamReader performs it by
default.
>
>
CyberSpyders@.gmail.com wrote:

Quote:

Originally Posted by

Hi,

I have an ASP.Net website, which allows users to upload a file which is
then inserted into a database.

This is all fine until it reads a line with the string +Anu in it.
It transforms this to this char ? (which, if Googled for, is
described as Unicode Character 'LATIN SMALL LETTER TURNED R WITH HOOK'
(U+027B) or, in Phonetics, as a 'Retroflex approximant'.)

Has anyone seen this behaviour before, and know how to stop it?
The code's simple - here's an example. The ? appears in the output
where the input is +Anu - it's transformed before I can touch it!

using (StreamReader sr = new StreamReader(strFile,
System.Text.Encoding.UTF7)) {
// Read and display lines from the file until the end of the file is
reached.
while ((line = sr.ReadLine()) != null) {
Response.Write(line);
}
}

Regards

Adam


Larry,

You were spot on - changing to UTF8 stopped this transformation. Thanks

It's not quite solved my problem though.
The file is a Text file, each line being a series of files delimited by
the character, as this was unliekley to ever appear in the actual
data.

Unfortunately, UTF8 encoding strips these characters completely. ASCII
encoding, on the other hand, replaces them with ?

Oh the joy of character encoding.

Regards

Adam

Larry Lard wrote:

Quote:

Originally Posted by

This is why you are seeing what you are seeing. UTF7 encodes characters
outside the printable 7 bit range using UTF16 then modified base64, with
+ as the indicator mark for this encoding. I haven't checked, but I
imagine +Anu is the UTF7 encoding of that character. You shouldn't use a
UTF7 reader to read a file that you don't know for sure was produced by
a UTF7 writer.
>
The correct way to read the file depends on what kind of file it is. If
it is text of an unknown encoding, there is no way to be absolutely
sure, but UTF8 is a good starting point. If it's binary data, you
shouldn't be using a TextReader class at all.

Thursday, March 22, 2012

Strange issue with system.web.mail

Hello,

I have a strange issue when sending e-mails using system.web.mail. For an unknown reason every 990 character is a space in my message body. Not a big issue? Well, bigger then you can think. If the 990 character is in the middle of a HTML-tag, the message body is constructed wrong.

I tested this with the script you can find below. Try it yourself to see the result. I use Microsoft .NET Framework Version:1.1.4322.2300; ASP.NET Version:1.1.4322.2300. My provider doesn't have another version available for me.

<%@dotnet.itags.org. Import Namespace = "System.Text" %>
<%@dotnet.itags.org. Import Namespace = "System.Web.Mail" %>
<script runat="server">
Sub SubmitEmail(sender As Object, e As EventArgs)
Dim sMsg as String
sMsg ="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"
sMsg+="123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-"

doEmail("name.to@dotnet.itags.org.yourhost.com", "name.from@dotnet.itags.org.yourhost.com", sMsg, "Test e-mail")
End Sub

Function doEmail(sTo as String, sFrom as String, sMsg as String, sSubject as String) 'Email component
Dim objEmail as New MailMessage
objEmail.To=sTo 'TO
objEmail.FROM=sFrom 'From

objEmail.SUBJECT=sSubject 'Subject
objEmail.BODY=sMsg 'Link Message body
objEmail.BodyFormat = MailFormat.HTML 'MailFormat.Text = RichText & MailFormat.HTML = HTML
SmtpMail.SmtpServer ="localhost" 'SMTP server
SmtpMail.Send(objEmail) 'Send message
End Function
</script>
<form runat="server">
<asp:Button id="Submit" text="Submit e-mail" OnClick="SubmitEmail" runat="server" />
</form>

Is this a known issue? What is the solution? I prefer not buying a e-mail component?

Thanks for your suggestions.

I don't know the answer - but have you tried stringbuilder. I have read about problems using string ( mostly performance related ) and that stringbuilder is the best choice ( for most string building situations)

This issue is solved. I found the solution here:

http://www.systemwebmail.com/faq/4.4.5.aspx

Usually you have a line in the body that is too long. The email rfc's state that a line should not contain more than 1000 characters. Try inserting a CrLf (vbCrLf for VB.NET or "\r\n" for C#) in your message body to break up long lines.