ColdFusion 8 Random Slowness? Downgrade the JVM!

Posted At : February 4, 2008 8:27 AM | Posted By : Mark
Related Categories: ColdFusion

I upgraded our main production server to ColdFusion 8 during the Christmas holidays. We had been running it on our beta server for several weeks without issue. Everything seemed to be humming along fine, until the student population returned. We began having intermittent slowness, causing intermittent timeouts, and headaches with several of our apps that wouldn't finish processing because of the timeouts (e.g. credit card transactions!). Tracking down the problem proved quite difficult. We would tried tweaking network settings (firewall settings, VLAN settings, traffic priority), increasing timeouts on some pages (to 120 seconds!) to make sure they complete, etc. While some of these changes seemed to help, the problem did not go away.

I began to suspect that ColdFusion's default Java 6 might be to blame since many of the timeouts were occurring with reactor-based applications. I added some JVM logging, but not being a JVM garbage collection log guru, I couldn't really tell if that was really the issue or not.

In a bit of frustration, I decided to back-rev the JVM as Sean Corfield suggested. I decided to go with Sun's latest 1.5 JRE, version 1.5.0-14. I installed it, set the JVM (or here for JRUN installs), and restarted ColdFusion. ColdFusion hummed along as normal, and after checking some internal indicators over the weekend, it appears that the random slowness/timeouts have disappeared.

Hopefully this issue can be resolved in either a ColdFusion updater or the next version of Sun's JVM (it's supposed to be fixed in 1.7, but they apparently haven't back-revved it into 1.6).

Comments (12) | Print | Send | del.icio.us | Digg It! | Linking Blogs | 2492 Views
Comments (Comment Moderation is enabled. Your comment will not appear until approved.)
[Add Comment]
How was your experience on doing the upgrade to ColdFusion 8? Did you run into any issues?

I'm about to upgrade my production server to CF 8 from CF 7. I've done the updgrade on several development computers, but this is the first time I'll roll it out on our production server which runs Windows Server 2003 and IIS 6.
# Posted By Bruce | 2/4/08 11:17 AM
Mark, which blog post of Sean's suggests downgrading the JVM? Thanks!
# Posted By Tom Mollerus | 2/4/08 11:24 AM
@Bruce - I don't recall changing any code to make it CF8 compatible. The Adobe team did a great job keeping this release backwards compatible and drop-in ready. One of my co-workers was dealing with some nuances in how the CF Report Builder interpreted some code, but I think that is an unusual edge case that most people would not run into. It's been great being able to use the new tags, syntax, etc.! I would recommend load-testing your site though to see if you need to backrev your JDK during the upgrade.

@Tom - It's linked at "Java 6" in my post, but here it is again in "full view" - http://corfield.org/blog/index.cfm/do/blog.entry/e...
# Posted By Mark Mazelin | 2/4/08 12:05 PM
Here's an article I forgot to reference in my blog post:
http://www.compoundtheory.com/?action=displayPost&...
# Posted By Mark Mazelin | 2/4/08 5:09 PM
Mark, this is an interesting and important post. I actually did some fairly comprehensive load-testing of the 1.5 and 1.6 JVM's and found 1.5 to be less optimal, of course this is often very application specific.

Do you still have any copies of the JVM Garbage Collection logs you collected?
# Posted By Mike Brunt | 2/4/08 10:33 PM
@Mike:

I do still have the GC log--it ran for several days and is about 75 MB. I wish I understood it better (at all)! I can send it to you if you are interested...

Mark
# Posted By Mark Mazelin | 2/5/08 8:30 AM
Mark, I could certainly look at that. Can you zip it and see what size we get then email me at mbrunt at alagad dot com. Depending on the zipped file size we might need to use ftp.
# Posted By Mike Brunt | 2/5/08 2:24 PM
Mark I got your log and split it into smaller chunks and would like to know what your jvm.config settings were when you were creating this log. Also was the whole of this log created with Java 1.6 or Java 1.5 or does it cover both. I thought it was good to do this as comments on your blog so we can hopefully help the community.

My early observation is that you are running out of Permgen memory often that is why I would like to see the jvm.config file. I also note there are Full Garbage collections every 10-11 minutes on average, which is not that bad.
# Posted By Mike Brunt | 2/5/08 5:51 PM
Mark, I did get your jvm.config file and had a quick question , you have this argument set...

-Djava.awt.headless=true

Do you have a particular reason for setting this?
# Posted By Mike Brunt | 2/6/08 3:58 PM
Mike:

I don't recall purposefully setting that. To my knowledge, that's how CF8 came. This is running on linux, so it may be necessary for the graphics/imaging stuff to work properly.
# Posted By Mark Mazelin | 2/6/08 4:45 PM
I would be interested in seeing how this conversation plays out. was the headless true the cause of the issue? Were any other issues found?
# Posted By Paul S | 2/12/08 5:13 PM
We're having very similar issues running CF 8,0,1,195765 on Linux/JBOSS with Java Version 1.5.0_16. Is there any chance that backing down to Java 1.5.0_14 would help? Is anyone else running CF on JBOSS? Any chances that this could this be part of our problem?
# Posted By Mark Kecko | 9/10/08 12:00 AM
[Add Comment]
BlogCFC was created by Raymond Camden. This blog is running version 5.8.