Overview

CloudPlatform's underlying Java architecture is susceptible to a leap-second addition bug. When a leap-second is added to the NTP time sync servers, it can cause the Java VM to go off in the weeds (500%+ CPU usage). The last time this occurred was June 30th, 2012; it will occur again on Dec 31st, 2012. The end result to the user is a very non-responsive CloudPlatform UI and API for any existing and running CloudPlatform deployment.

More information about this bug can be found at:

http://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/

http://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/comment-page-1/#comment-1472

http://remibergsma.wordpress.com/2012/07/01/high-cpu-load-on-cloudstack-management-servers-after-leap-second-3062012235959-utc/

Leap-second Workaround

The workaround for this issue is:

1. SSH into the management server
2. Run the following command:
   
   [root@managementserver ~]# date ; date -s "`date -u`" ; date
   Fri Jul 13 14:26:52 CDT 2012
   Fri Jul 13 14:26:52 CDT 2012
   Fri Jul 13 14:26:52 CDT 2012

Note: this is an issue with the Linux kernel and Java, not with CloudPlatform in particular.