[Building Sakai] Proposal for dynamic loading of ServerConfigurationService and component manager primitives

Fri May 27 16:44:05 PDT 2011

I certainly support that. It's an issue we've seen a number of times, although I have to say that in many cases the properties we need to change do work.

I don't think we're the only ones who are getting into a situation where Sakai can't ever be down. The danger with putting up a service that's potentially vital to the primary mission of the University is that people actually use it that way…

However I also encourage people to set up ways to do deploys unobtrusively. I've found that to do that really transparently requires a patch to login. Most load balancers let you set the weight of a  server to 0 without interrupting existing sessions. You then can wait for a day or two, and shouldn't have anyone left. The problem is that the load balancer's idea of a session is typically not completely synced to Sakai's. (Part of this has to do with the fact that a new JSESSIONID is issued at login, so you can't actually use JSESSIONID as the balance for persistence.) So I've found it necessary to have a setting for a server that is going down. When someone logs it, it clears the cookie that the load balancer adds and then redirects the browser back to the same URL. It then goes to a server whose weight is non-zero. The details depend upon load balancer and setup, of course.

With this in place, we can bring up a second set of servers and wait for the old set to timeout. Ideally users will see one downtime a year, when we move to a new version of Sakai. (In practice they see more than one, because our machine room power is unreliable. A UPS doesn't help when the wiring from the UPS to the machine room blows up.)

On May 27, 2011, at 7:02:40 PM, John Bush wrote:

> Right the idea is that the tools/services would register with the
> config service and say hey, I've got this property and it can be
> reloaded.  Then the tool/services have that want to participate
> rejigger their code to handle it if they want to.  Like don't cache
> stuff, recheck the serverconfigurationservice etc.  For the more
> complex cases, I'm considering a notification option, where when the
> property is changed a notification could be set out.  This about for
> example the case where you have two properties, a port and server
> location, and at init time you create a URL from that.  A notification
> then could call out to you, to let you recreate the URL.
> 
> Actually where people call ServerConfigurationService, I think that is
> quite straightforward, as proven by the ucd work.   If got a prototype
> of some basic like that working already.
> 
> I'm more interested right now in extending this to the Spring
> properties, as that is where we personally have a lot of config.
> 
> Right now, I'm trying to move this guys approach into the sakai
> component manager and see if that might work,
> http://www.wuenschenswert.net/wunschdenken/archives/127
> 
> 
> 
> On Fri, May 27, 2011 at 10:53 AM, Thomas Amsler <tpamsler at ucdavis.edu> wrote:
>> If the tool/service stores the property locally, then you won't be
>> able to change them dynamically. Tools and services would have to be
>> adjusted to always get the property per access. We did that for all
>> the tools/services for which we needed dynamic property loading.
>> -- Thomas
>> 
>> On Fri, May 27, 2011 at 9:51 AM, Duffy Gillman <duffy at rsmart.com> wrote:
>>> I think this complexity was the point of registering the properties. (Please
>>> correct me if I'm wrong, John) One can already change these values at
>>> runtime but there is no guarantee that the code that uses the properties
>>> will re-read the values. But, if one is forced to register properties as
>>> dynamic presumably inspection and changes to code can be done on a per-tool
>>> or per-component basis. One registers a property as dynamic only when one is
>>> sure that the code it supports is prepared to handle changes to the
>>> property's value. For properties whose change would require
>>> re-initialization of resources or whose change might cause inconsistency one
>>> might opt not to make those properties dynamic.
>>> My $.02...
>>>    Duffy Gillman
>>>    Sr. Software Engineer
>>>    The rSmart Group, Inc.
>>> On Fri, May 27, 2011 at 6:43 AM, Charles Hedrick <hedrick at rutgers.edu>
>>> wrote:
>>>> 
>>>> I agree that this is a good idea. We *really* like to avoid restarts.
>>>> 
>>>> I note that it's possible already to set properties dynamically. Here's a
>>>> JSP that will look at and change a property. In many cases it works. The
>>>> problem is that a lot of code reads properties when the service initializes.
>>>> So doing this for real will involve changing code throughout Sakai so that
>>>> it reads the configuration when it needs it. In many cases the change is
>>>> easy. I'm sure there are a few where other decisions depend upon the
>>>> configuration, and can't easily be changed midstream. (The reason for
>>>> getClass().getMethod().invoke() is that getProperties is a private method.
>>>> By using introspection you can access and use private attributes and methods
>>>> from classes that shouldn't be able to access them.)
>>>> 
>>>> <%@ page
>>>> import="org.sakaiproject.component.api.ServerConfigurationService" %><%
>>>> %><%@ page import="org.sakaiproject.component.cover.ComponentManager" %><%
>>>> %><%@ page import="java.beans.Introspector" %><%
>>>> %><%@ page import="java.util.Properties" %><%
>>>> 
>>>> // example of how to change a config parameter without restarting
>>>> 
>>>> ServerConfigurationService serv = (ServerConfigurationService)
>>>> ComponentManager
>>>>     .get(ServerConfigurationService.class);
>>>> 
>>>> Properties props =
>>>> (Properties)serv.getClass().getMethod("getProperties",null).invoke(serv,null);
>>>> 
>>>> out.println(props.get("content.html.forcedownload"));
>>>> //out.println(props.setProperty("content.html.forcedownload", "false"));
>>>> //out.println(props.get("content.html.forcedownload"));
>>>> 
>>>> %>
>>>> 
>>>> 
>>>> 
>>>> On May 26, 2011, at 3:11 PM, John Bush wrote:
>>>> 
>>>>> Please see https://jira.sakaiproject.org/browse/KNL-739 for more
>>>>> detail.  I've started thinking about a way we could optionally let
>>>>> folks register properties and allow for changes without a tomcat
>>>>> restart. Its still a work in progress, welcome feedback and thoughts
>>>>> on this.
>>>>> 
>>>>> The working doc is here:
>>>>> 
>>>>> https://docs.google.com/a/rsmart.com/document/pub?id=1uR11MvnG4Ja60plH8QPpVJnsHklB6_mRZ6hcNSEXqYw
>>>>> 
>>>>> --
>>>>> John Bush
>>>>> 602-490-0470
>>>>> _______________________________________________
>>>>> sakai-dev mailing list
>>>>> sakai-dev at collab.sakaiproject.org
>>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>> 
>>>>> TO UNSUBSCRIBE: send email to
>>>>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>>>>> "unsubscribe"
>>>> 
>>>> _______________________________________________
>>>> sakai-dev mailing list
>>>> sakai-dev at collab.sakaiproject.org
>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>> 
>>>> TO UNSUBSCRIBE: send email to
>>>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>>>> "unsubscribe"
>>> 
>>> 
>>> _______________________________________________
>>> sakai-dev mailing list
>>> sakai-dev at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>> 
>>> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org
>>> with a subject of "unsubscribe"
>>> 
>> 
> 
> 
> 
> -- 
> John Bush
> 602-490-0470

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3817 bytes
Desc: not available
Url : http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20110527/63123a67/attachment.bin 