Wednesday, March 28, 2012

Strange bug: application dies (partially) after restart

I did more investigation, and the problem is due to a deadlock (I 'm posting
the issue separately on the VJ# NG because it seems to be specific to J#).
The fact that it only seemed to happen after a hot restart was only a
coincidence. Since my initial post, we also saw the problem after a cold
start.

Bruno

"Bruno Jouhier [MVP]" <bjouhier@dotnet.itags.org.club-internet.fr> a crit dans le message de
news:eFeXAEdFEHA.700@dotnet.itags.org.TK2MSFTNGP09.phx.gbl...
> Hi Alvin,
> Thanks for the reply. Our logs don't say anything. Looks like the stateful
> requests are not even delivered to the application any more because
> otherwise we would either be getting a response, or an exception (and we
log
> them all). But I need to attach a debugger to verify this and see what's
> really going on inside. So, I need to do more investigation on my side,
but
> I was wondering if somebody had experienced something similar.
> Bruno.
> "Alvin Bruney [MVP]" <vapor at steaming post office> a crit dans le
message
> de news:Ow$vzWaFEHA.576@dotnet.itags.org.TK2MSFTNGP11.phx.gbl...
> > The restart is key to having the application object be more robust. It's
> not
> > something that is going to get removed. You can clamor for an option
> switch
> > though which may turn it off.
> > What does your event logs say during these funny errors?
> > --
> > Regards,
> > Alvin Bruney [ASP.NET MVP]
> > Got tidbits? Get it here...
> > http://tinyurl.com/3he3b
> > "Bruno Jouhier [MVP]" <bjouhier@dotnet.itags.org.club-internet.fr> wrote in message
> > news:O4ARZfWFEHA.2560@dotnet.itags.org.TK2MSFTNGP12.phx.gbl...
> > > We are observing a strange phenomenon on our ASP .NET applications:
> > > > If the applications are cold started (after a server reboot or an II
> > > restart), they run for days without any problem.
> > > > But if they get restarted while IIS is running (because the web.config
> > file
> > > got modified, or because the DLLs got replaced by new ones), they run
ok
> > for
> > > a certain period (a few hours if the load is light, only a few minutes
> if
> > > the load is heavier) and then the server stops responding to some
> requests
> > > (seems like it continues to generate stateless pages, but does not
> > generate
> > > any stateful pages any more, but I have to investigate what is really
> > > happening in more details).
> > > > We observed this strange behavior on our two ASP .NET applications,
and
> on
> > > different servers. The applications are based on .NET Framework 1.1
and
> > > Visual J#.
> > > > This is not critical because we can restart IIS every time we modify
the
> > > config file or upgrade the DLLs but I'd like to understand what is
going
> > on.
> > > Did anyone else experience a similar problem? Is there a fix?
> > > > Also, it would be great if there was a way to prevent ASP NET from
> > > restarting the application every time the config file or the DLLs are
> > > modified. Some programs (anti-virus, backup) change the last access
time
> > on
> > > these files and they cause an application restart (accessing the file
is
> > all
> > > it takes, the application restarts even if the files are not
modified),
> > > which is very annoying (especially if the application stops responding
> > > afterwards, as described above).
> > > This automatic restart feature is nice during development but I don't
> find
> > > it so nice on a production server because I don't like the idea of
> having
> > a
> > > Web App restart because some background maintenance task is touching
its
> > > files. Does anybody know how to prevent this?
> > > > Bruno.
> >Hi Bruno,

Thanks for your followup. I'm glad that you've firgured out the problem.
Since the problem is specific to J#, have you found any means to workaround
it? Hope you'll soon thoroughly resolve this problem. If there is anything
I can help, please feel free to post here.Thanks.

Regards,

Steven Cheng
Microsoft Online Support

Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)

Get Preview at ASP.NET whidbey
http://msdn.microsoft.com/asp.net/whidbey/default.aspx
Hi Steven,

The problem is in the J# Date, SimpleDateFormat and TimeZone classes.
Synchronization seems to be seriously broken and I have a number of
tracebacks and a small repro that demonstrate this (I'm going to post more
stuff on the J# newsgroup).

First, I have tried to fix it by synchronizing around a global object before
calling these JDK methods. But this only works if I have one application
hosted in aspnet_wp.exe. With 2 applications, I get deadlocks between
threads from the 2 apps (very weird, I have posted more details on this one
on microsoft.public.dotnet.vjsharp).

So, I am in the process of recoding all our date and time stuff with the
..NET APIs. This should fix it, and it should also be faster (fortunately, we
had our own Date and Timestamp classes and everything is encapsulated there,
so the recoding is very local).

BTW: The .NET stuff is very good overall but it is rather poor when it comes
to TimeZones. .NET only knows about the machine's timezone and UTC time.
This is not enough for a server app where you want every user to see and
input dates and times formatted in his own timezome. I posted a suggestion
on the Whidbey NG to get extensions to this: APIs to enumerate time zones,
find them by name, and also a "thread" variable that would keep track of the
current timezone and influence the way DateTime are interpreted in the
current thread (something like TimeZone.CurrentTimeZone that would work very
much like CultureInfo.CurrentCulture). I'm repeating it here because I think
that this is a real hole in the .NET API, and an API like that would make my
life easier.

Bruno.

"Steven Cheng[MSFT]" <v-schang@.online.microsoft.com> a crit dans le message
de news:Y7LgaErGEHA.3244@.cpmsftngxa06.phx.gbl...
> Hi Bruno,
> Thanks for your followup. I'm glad that you've firgured out the problem.
> Since the problem is specific to J#, have you found any means to
workaround
> it? Hope you'll soon thoroughly resolve this problem. If there is anything
> I can help, please feel free to post here.Thanks.
>
> Regards,
> Steven Cheng
> Microsoft Online Support
> Get Secure! www.microsoft.com/security
> (This posting is provided "AS IS", with no warranties, and confers no
> rights.)
> Get Preview at ASP.NET whidbey
> http://msdn.microsoft.com/asp.net/whidbey/default.aspx
Hi Bruno,

Thanks for your followup. Since you mentioned that you've generated a small
repro, I think it'll be helpful if you can make the repro contains the
necessary stuffs and attached it to us. Also a definite repro steps is also
necessary, thus, we can try to do some indepth research on ourside.

In addtion, I also have met the TimeZone problems of the .NET framework
classes before since it doesn't provide enought interfaces to process
TimeZone with client cultureinfo. Sometimes we even need to build our own
custom LookUp Table to mapping cultureinfo with certain TimeZone info.
Anyway, we'll also forward this suggestion to the related production team.
Hope it will soon be improved in the following release. Thanks.

Regards,

Steven Cheng
Microsoft Online Support

Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)

Get Preview at ASP.NET whidbey
http://msdn.microsoft.com/asp.net/whidbey/default.aspx
Hi Steven,

I have posted the repro on the microsoft.public.dotnet.vjsharp NG because it
is really specific to J#.

I will also post some tracebacks of other deadlocks that I got on the J# NG
but I'll do it later because I don't have time to format and explain them
right now.

The deadlock that I have posted is really very very simple: one thread calls
TimeZone.getDefault() and the other one calls Calendar.getInstance(), and
they manage to deadlock each other (not always but at least one run out of
4).

Bruno.

"Steven Cheng[MSFT]" <v-schang@.online.microsoft.com> a crit dans le message
de news:MU4eTS7GEHA.2224@.cpmsftngxa06.phx.gbl...
> Hi Bruno,
> Thanks for your followup. Since you mentioned that you've generated a
small
> repro, I think it'll be helpful if you can make the repro contains the
> necessary stuffs and attached it to us. Also a definite repro steps is
also
> necessary, thus, we can try to do some indepth research on ourside.
> In addtion, I also have met the TimeZone problems of the .NET framework
> classes before since it doesn't provide enought interfaces to process
> TimeZone with client cultureinfo. Sometimes we even need to build our own
> custom LookUp Table to mapping cultureinfo with certain TimeZone info.
> Anyway, we'll also forward this suggestion to the related production team.
> Hope it will soon be improved in the following release. Thanks.
> Regards,
> Steven Cheng
> Microsoft Online Support
> Get Secure! www.microsoft.com/security
> (This posting is provided "AS IS", with no warranties, and confers no
> rights.)
> Get Preview at ASP.NET whidbey
> http://msdn.microsoft.com/asp.net/whidbey/default.aspx
Hi Bruno,

Thanks for your reply. From your further description, I've got that the
problem is specific to J#. And I hope you'll soon resolve the problem.
Also, if you think there is anything we still can help, please feel free
post here or mail me via the mail address in my signature(remove the
"online"). Thanks.

Regards,

Steven Cheng
Microsoft Online Support

Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)

Get Preview at ASP.NET whidbey
http://msdn.microsoft.com/asp.net/whidbey/default.aspx

0 comments:

Post a Comment