[sakai2-tcc] Annotation based injection within Sakai components

[Matthew Jones]

I was watching this interview the other day from Eric Liu where he talked
said "government should be "*big on the what* and small on the how.". This
is ideally how the TCC should be as John and Chuck have said. I think some
of the problem with this is that many roles of TCC members overlap as
people who will actually be doing the work as people making the decisions.
And many "what" ideas have already been attempted to be turned into "how"
ideas and the only viable option was to form a new project and spend
millions of dollars on it (both of which are not realistic).

While David Adam's point about wanting "Session Failover" is something we
all want, the previous attempt at implementing it lead to a dead-end mostly
from what I remember many tools have unserializable sessions with huge
attributes and basically any tool written in JSF was problematic. Here was
the writeup on that, after significant time was spent on it.

https://confluence.sakaiproject.org/display/TERRA/Sakai+Session+Clustering+Problems+and+Ideas


Too bad we can't go back and tell ourselves not to write things in JSF! So
it's either spent a lot more time on a re-writing these tools (will happen
someday anyway), keep trying to make it work, or consider this problem not
worth pursuing. Both options that involve solving the problem will be very
expensive.

This isn't even a problem users notice, the bigger one is that session
state results in users not being able to open more than one tab. You have
no idea (or maybe you do) about how many users want to open one assignment
in one tab and another in another tab. In Sakai this is impossible because
your state is kept in the session on the server so if you try to do
something, it has an effect on whatever you did last. Back in the world of
a single tab browser (Internet Explorer 6) that existed when Sakai was
created this was a great idea. After 2008 though this was just confusing
and useless to the majority of users.

I think both the session problem and the state persistence go together.

Maybe figuring out how to fix some of that might get the back button to
work after 6 years? ;)

On Thu, May 17, 2012 at 12:42 AM, John Bush <john.bush at rsmart.com> wrote:

> Regarding session replication.  I think you should talk to folks at
> unicon who did some work there for wiley a few years back.  I think
> the approach was on top of terracotta.  Some of that made it into
> core, but I don't really know the state of that anymore.
>
> There was a lot of good brainpower thrown at that problem.  What I
> remember was that sakai's model objects have references back to the
> service they came from and this caused a giant nightmare for any type
> of session replication attempting to make that stuff transient was
> impossible due to classloading issues, and that is how they ended up
> on terracotta.  Or something like that, probably others over there are
> better versed.
>
> The skinny is, I agree being more stateless is a big step forward in
> that direction, but this problem may prove to be hard to resolve.
> There may be other ways to attack this problem now.  Its worth
> exploration, for the reasons you've outlined.
>
> On Wed, May 16, 2012 at 9:31 PM, Adams, David <da1 at vt.edu> wrote:
> > Not that I'm on the technical committee (not even sure this message will
> make it through) but as an interested party on the operations side, I think
> Chuck's list of fixing some of the biggest deployment and scaling headaches
> ought to be high on the list, as they could mostly be accomplished without
> breaking tools. In particular, issues surrounding event message passing
> between servers (sounds like rSmart has a solution here), and finding a way
> to make user state transferable between app servers (I assume cleaning up
> the Velocity tools would go a long way towards this goal), so that we can
> take servers offline without waiting hours for active sessions to drain (or
> so that server crashes don't destroy hundreds of user sessions at a time)
> are key, as is runtime configurability.
> >
> > As for Chuck's mention of the authz tables, here at VT we had about 18
> months of intermittent reliability problems (ie, complete system crashes
> across all app servers that cropped up in just seconds) that we can trace
> 90% of back to database bottlenecks due to authz queries piling up waiting
> for shared memory latches to be freed in Oracle--with every new user
> request wanting to run a dozen or more of those queries, if there was any
> slowdown whatsoever, the app servers would exhaust their DB pools in
> seconds, no matter how much headroom we gave them. (We solved the problem
> by moving from a 16-core server to an 80-core server... Oracle correlates
> the available shared memory latches to the core count.)
> >
> > While those queries are heavy on the joins, and we found that we could
> make improvements in throughput by, for example, hardcoding the role keys
> for .anon and .auth roles into the primary authz query, and so I think
> Chuck's proposal of eliminating the string lookups on every query is a good
> idea, ultimately those fixes, for Oracle at least, don't solve the
> contention problem, which is all on the sakai_realm_rl_fn table itself.
> With just three integer fields, you can cram a lot of 12- (or 24-) byte
> rows into an 8k memory block, and removing the joins from the equation only
> buys you so much. A better approach, I believe, would be to find ways of
> either massively reducing the required number of records in that table by
> expanding the role of something like the !site.helper realm, but with the
> ability to remove positions at higher levels. So each site type ought to
> have a basic template of permissions for each role, but instead of copying
> those out to each realm individually, that
> >  default list ought to be the most commonly used set and there should be
> no reason for most sites to have their own permissions at all. And of the
> ones that do end up changing permissions, a short list of additions and
> deletions could be cached upon a visit to a site and merged with the site
> type defaults. Combined with a more reliable cache thanks to a better event
> system, it ought to be possible to keep active database queries to a
> minimum.
> >
> > Alternatively, make available an alternative authz provider that stores
> its data in some other database system. The realm-role-function triplet
> lookup seems well suited to some kind of NoSQL datastore, and just
> eliminating the authz load from the database would drop our query load by
> 40+%.
> >
> > A similar system makes sense for presence/persistent chat. Those are the
> kind of features that schools like VT end up turning off as soon as we run
> into performance issues because of the high load in terms of app server
> memory and database query volume. If those components could be moved out of
> the primary Sakai Java VM and database pair and into an associated but
> separate system, we operations folks would be a lot happier about the
> scalability, reliability, and performance of CLE.
> >
> > David Adams
> > Director, Systems Integration and Support
> > Virginia Tech Learning Technologies
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai2-tcc/attachments/20120517/395da1dc/attachment.html