[Building Sakai] Proposal for dynamic loading of ServerConfigurationService and component manager primitives

Steve Swinsburg steve.swinsburg at gmail.com
Tue Jul 12 23:51:46 PDT 2011 

I just came across the Apache Commons Configuration library which makes working with java.util.Properties much easier. One neat thing is does is it can watch for file changes and reload them (and write back to the file if you want).

http://commons.apache.org/configuration/userguide/howto_filebased.html

I'm using it to load some properties from a file, seems pretty neat so far. Could be useful as a way to get sakai.properties reloaded without a restart, with minimal changes to the ServerConfigurationService.

I haven't checked out John B's work on this yet though.

cheers,
Steve




On 16/06/2011, at 4:32 PM, John Bush wrote:

> In case you missed my tweets couple videos up on this subject
> 
> Demo
> http://www.youtube.com/watch?v=wSW-dxezVAM
> 
> Tech Talk
> http://www.youtube.com/watch?v=1SET3x0vaSg
> 
> Hoping to get code in state to share soon.  What happens when you are
> left behind from sakai conference is you end up holding down the fort
> on supporting all the production systems :(  That is eating up all my
> creative time, oh well, maybe tomorrow.
> 
> On Tue, Jun 7, 2011 at 3:39 PM, John Bush <john.bush at rsmart.com> wrote:
>> On Tue, Jun 7, 2011 at 11:54 AM, Earle Nietzel <earle.nietzel at gmail.com> wrote:
>>> I'd like to add some thoughts (some may have been listed already):
>>> 
>>> * There could be a DefaultPropertyConfiguration that is used for tools
>>> which do not make use of the new feature and would continue to act the
>>> way they currently do. The contrary could be something like
>>> DynamicPropertyConfiguration.
>>> 
>>> * Ideally all properties set via ComponentManager or
>>> ServiceConfigurationService should implement a common contract so that
>>> all properties can be managed from one service.
>> 
>> Agreed.  I think maybe having a new Properties impl is the way to do
>> this.  That plays well will both pathways.
>> 
>>> * If you have more than one application server there should be a way
>>> to update these properties on all application servers. Note some
>>> properties may need to be different on other servers. Possibly a
>>> distributed cache for DynamicProperty(s).
>> 
>> I was thinking the dynamic values are stored in database.  Local
>> caches read from the database, possibly have a distributed way to
>> invalidate.  Looking into Jgroups for that, maybe JMS, but I'm
>> thinking might be too heavy.
>> 
>>> * Changing properties from a UI (i.e. Configuration Viewer) would
>>> probably be ideal but others are helpful i.e. (JMX, MBean, WebService,
>>> SakaiShell, etc.).
>> 
>> Yes, I was thinking that gets built into the Config Viewer UI.  There
>> would be a service for doing it other ways.
>> 
>>> * For those services or tools that need to be notified when an update
>>> occurs an observer/listener pattern could be used or some other
>>> pattern.
>> 
>> yes, I have this working already.
>> 
>>> * For performance there probably should be a cache to store properties.
>> 
>> absolutely
>> 
>>> * Testing - Having a TestableProperty feature would probably help with testing.
>>> 
>>> Please add to the list or comment.
>>> 
>>> Did someone start a confluence page yet? Would be great to have all
>>> the input there.
>> 
>> No I need to move everything in there.  I'm hoping this week to get to
>> that, update where I'm at, and check some prototype code in before
>> conference.
>> 
>>> 
>>> Earle
>>> 
>>> 
>>> 
>>> On Fri, Jun 3, 2011 at 4:44 PM, John Bush <john.bush at rsmart.com> wrote:
>>>> Ok, I actually just got this to work in Sakai for spring injected
>>>> properties using this guy's approach:
>>>> http://www.wuenschenswert.net/wunschdenken/archives/127
>>>> 
>>>> I actually think I could extend this to hot redeploy a spring bean as
>>>> well, by listening for components.xml changes.  This wouldn't be safe
>>>> for production, but super handy for development.
>>>> 
>>>> I need to do some clean up and build in some other stuff, then I'll
>>>> setup a branch so others can take a look.
>>>> 
>>>> On Fri, May 27, 2011 at 4:44 PM, Hedrick Charles <hedrick at rutgers.edu> wrote:
>>>>> I certainly support that. It's an issue we've seen a number of times, although I have to say that in many cases the properties we need to change do work.
>>>>> 
>>>>> I don't think we're the only ones who are getting into a situation where Sakai can't ever be down. The danger with putting up a service that's potentially vital to the primary mission of the University is that people actually use it that way…
>>>>> 
>>>>> However I also encourage people to set up ways to do deploys unobtrusively. I've found that to do that really transparently requires a patch to login. Most load balancers let you set the weight of a  server to 0 without interrupting existing sessions. You then can wait for a day or two, and shouldn't have anyone left. The problem is that the load balancer's idea of a session is typically not completely synced to Sakai's. (Part of this has to do with the fact that a new JSESSIONID is issued at login, so you can't actually use JSESSIONID as the balance for persistence.) So I've found it necessary to have a setting for a server that is going down. When someone logs it, it clears the cookie that the load balancer adds and then redirects the browser back to the same URL. It then goes to a server whose weight is non-zero. The details depend upon load balancer and setup, of course.
>>>>> 
>>>>> With this in place, we can bring up a second set of servers and wait for the old set to timeout. Ideally users will see one downtime a year, when we move to a new version of Sakai. (In practice they see more than one, because our machine room power is unreliable. A UPS doesn't help when the wiring from the UPS to the machine room blows up.)
>>>>> 
>>>>> 
>>>>> 
>>>>> On May 27, 2011, at 7:02:40 PM, John Bush wrote:
>>>>> 
>>>>>> Right the idea is that the tools/services would register with the
>>>>>> config service and say hey, I've got this property and it can be
>>>>>> reloaded.  Then the tool/services have that want to participate
>>>>>> rejigger their code to handle it if they want to.  Like don't cache
>>>>>> stuff, recheck the serverconfigurationservice etc.  For the more
>>>>>> complex cases, I'm considering a notification option, where when the
>>>>>> property is changed a notification could be set out.  This about for
>>>>>> example the case where you have two properties, a port and server
>>>>>> location, and at init time you create a URL from that.  A notification
>>>>>> then could call out to you, to let you recreate the URL.
>>>>>> 
>>>>>> Actually where people call ServerConfigurationService, I think that is
>>>>>> quite straightforward, as proven by the ucd work.   If got a prototype
>>>>>> of some basic like that working already.
>>>>>> 
>>>>>> I'm more interested right now in extending this to the Spring
>>>>>> properties, as that is where we personally have a lot of config.
>>>>>> 
>>>>>> Right now, I'm trying to move this guys approach into the sakai
>>>>>> component manager and see if that might work,
>>>>>> http://www.wuenschenswert.net/wunschdenken/archives/127
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Fri, May 27, 2011 at 10:53 AM, Thomas Amsler <tpamsler at ucdavis.edu> wrote:
>>>>>>> If the tool/service stores the property locally, then you won't be
>>>>>>> able to change them dynamically. Tools and services would have to be
>>>>>>> adjusted to always get the property per access. We did that for all
>>>>>>> the tools/services for which we needed dynamic property loading.
>>>>>>> -- Thomas
>>>>>>> 
>>>>>>> On Fri, May 27, 2011 at 9:51 AM, Duffy Gillman <duffy at rsmart.com> wrote:
>>>>>>>> I think this complexity was the point of registering the properties. (Please
>>>>>>>> correct me if I'm wrong, John) One can already change these values at
>>>>>>>> runtime but there is no guarantee that the code that uses the properties
>>>>>>>> will re-read the values. But, if one is forced to register properties as
>>>>>>>> dynamic presumably inspection and changes to code can be done on a per-tool
>>>>>>>> or per-component basis. One registers a property as dynamic only when one is
>>>>>>>> sure that the code it supports is prepared to handle changes to the
>>>>>>>> property's value. For properties whose change would require
>>>>>>>> re-initialization of resources or whose change might cause inconsistency one
>>>>>>>> might opt not to make those properties dynamic.
>>>>>>>> My $.02...
>>>>>>>>    Duffy Gillman
>>>>>>>>    Sr. Software Engineer
>>>>>>>>    The rSmart Group, Inc.
>>>>>>>> On Fri, May 27, 2011 at 6:43 AM, Charles Hedrick <hedrick at rutgers.edu>
>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> I agree that this is a good idea. We *really* like to avoid restarts.
>>>>>>>>> 
>>>>>>>>> I note that it's possible already to set properties dynamically. Here's a
>>>>>>>>> JSP that will look at and change a property. In many cases it works. The
>>>>>>>>> problem is that a lot of code reads properties when the service initializes.
>>>>>>>>> So doing this for real will involve changing code throughout Sakai so that
>>>>>>>>> it reads the configuration when it needs it. In many cases the change is
>>>>>>>>> easy. I'm sure there are a few where other decisions depend upon the
>>>>>>>>> configuration, and can't easily be changed midstream. (The reason for
>>>>>>>>> getClass().getMethod().invoke() is that getProperties is a private method.
>>>>>>>>> By using introspection you can access and use private attributes and methods
>>>>>>>>> from classes that shouldn't be able to access them.)
>>>>>>>>> 
>>>>>>>>> <%@ page
>>>>>>>>> import="org.sakaiproject.component.api.ServerConfigurationService" %><%
>>>>>>>>> %><%@ page import="org.sakaiproject.component.cover.ComponentManager" %><%
>>>>>>>>> %><%@ page import="java.beans.Introspector" %><%
>>>>>>>>> %><%@ page import="java.util.Properties" %><%
>>>>>>>>> 
>>>>>>>>> // example of how to change a config parameter without restarting
>>>>>>>>> 
>>>>>>>>> ServerConfigurationService serv = (ServerConfigurationService)
>>>>>>>>> ComponentManager
>>>>>>>>>     .get(ServerConfigurationService.class);
>>>>>>>>> 
>>>>>>>>> Properties props =
>>>>>>>>> (Properties)serv.getClass().getMethod("getProperties",null).invoke(serv,null);
>>>>>>>>> 
>>>>>>>>> out.println(props.get("content.html.forcedownload"));
>>>>>>>>> //out.println(props.setProperty("content.html.forcedownload", "false"));
>>>>>>>>> //out.println(props.get("content.html.forcedownload"));
>>>>>>>>> 
>>>>>>>>> %>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On May 26, 2011, at 3:11 PM, John Bush wrote:
>>>>>>>>> 
>>>>>>>>>> Please see https://jira.sakaiproject.org/browse/KNL-739 for more
>>>>>>>>>> detail.  I've started thinking about a way we could optionally let
>>>>>>>>>> folks register properties and allow for changes without a tomcat
>>>>>>>>>> restart. Its still a work in progress, welcome feedback and thoughts
>>>>>>>>>> on this.
>>>>>>>>>> 
>>>>>>>>>> The working doc is here:
>>>>>>>>>> 
>>>>>>>>>> https://docs.google.com/a/rsmart.com/document/pub?id=1uR11MvnG4Ja60plH8QPpVJnsHklB6_mRZ6hcNSEXqYw
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> John Bush
>>>>>>>>>> 602-490-0470
>>>>>>>>>> _______________________________________________
>>>>>>>>>> sakai-dev mailing list
>>>>>>>>>> sakai-dev at collab.sakaiproject.org
>>>>>>>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>>>>>>> 
>>>>>>>>>> TO UNSUBSCRIBE: send email to
>>>>>>>>>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>>>>>>>>>> "unsubscribe"
>>>>>>>>> 
>>>>>>>>> _______________________________________________
>>>>>>>>> sakai-dev mailing list
>>>>>>>>> sakai-dev at collab.sakaiproject.org
>>>>>>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>>>>>> 
>>>>>>>>> TO UNSUBSCRIBE: send email to
>>>>>>>>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>>>>>>>>> "unsubscribe"
>>>>>>>> 
>>>>>>>> 
>>>>>>>> _______________________________________________
>>>>>>>> sakai-dev mailing list
>>>>>>>> sakai-dev at collab.sakaiproject.org
>>>>>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>>>>> 
>>>>>>>> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org
>>>>>>>> with a subject of "unsubscribe"
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> John Bush
>>>>>> 602-490-0470
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> John Bush
>>>> 602-490-0470
>>>> _______________________________________________
>>>> sakai-dev mailing list
>>>> sakai-dev at collab.sakaiproject.org
>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>> 
>>>> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
>>>> 
>>> _______________________________________________
>>> sakai-dev mailing list
>>> sakai-dev at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>> 
>>> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
>>> 
>> 
>> 
>> 
>> --
>> John Bush
>> 602-490-0470
>> 
> 
> 
> 
> -- 
> John Bush
> 602-490-0470
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> 
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"

More information about the sakai-dev mailing list