[Building Sakai] ContentHosting cache?

[David Horwitz]

Thanks Jim I suspected something like that was at work - I came across
this while doing some pondering on our uses of caches and how we might
better manage memory in Sakai.


Your information however did point me towards understanding why we have
memory problems on complete index rebuilds on Sakai. In short the index
rebuild is a single thread that runs and will retrieve all the items for
the indexer. As a result thread local is never cleared. Some debugs I
stuck in content hosting illustrate this:

>
> 2010-08-16 10:37:06,891  INFO Timer-2
> org.sakaiproject.content.impl.BaseContentService - objects in
> threadlocal:2516
> 2010-08-16 10:37:07,068  INFO Timer-2
> org.sakaiproject.search.indexer.impl.SearchBuilderQueueManager -
> Rebuild for 4e24fe05-1e3f-4fd6-8997-4687066b8576 (54/65661)
> 2010-08-16 10:37:07,413  INFO Timer-2
> org.sakaiproject.content.impl.BaseContentService - objects in
> threadlocal:2542
> 2010-08-16 10:37:10,646  INFO Timer-3
> org.sakaiproject.email.impl.BasicEmailService - Preparing to send the
> mail digests for 20100815220000000]20100816220000000
> 2010-08-16 10:37:12,603  INFO Timer-2
> org.sakaiproject.search.indexer.impl.SearchBuilderQueueManager -
> Rebuild for 74f29440-acb0-4af8-88b8-a23b1950b7c7 (55/65661)
> 2010-08-16 10:37:12,910  INFO Timer-2
> org.sakaiproject.content.impl.BaseContentService - objects in
> threadlocal:4000
> 2010-08-16 10:37:13,156  INFO Timer-2
> org.sakaiproject.search.indexer.impl.SearchBuilderQueueManager -
> Rebuild for 83ef05b6-f445-40f8-a4b1-1ec05641c56c (56/65661)
> 2010-08-16 10:37:16,609  INFO Timer-2
> org.sakaiproject.content.impl.BaseContentService - objects in
> threadlocal:4074
> 2010-08-16 10:37:16,758  INFO Timer-2
> org.sakaiproject.search.indexer.impl.SearchBuilderQueueManager -
> Rebuild for e71f3b9a-6de8-459b-9ff6-17738f68ed3e (57/65661)
> 2010-08-16 10:37:20,358  INFO Timer-2
> org.sakaiproject.content.impl.BaseContentService - objects in
> threadlocal:4101
> 2010-08-16 10:37:30,413  INFO Timer-2
> org.sakaiproject.search.indexer.impl.SearchBuilderQueueManager -
> Rebuild for 65ced45d-a6bf-463e-000d-6355ed2f448f (58/65661)
> 2010-08-16 10:37:30,664  INFO Timer-2
> org.sakaiproject.content.impl.BaseContentService - objects in
> threadlocal:5882
> 2010-08-16 10:37:31,293  INFO Timer-2
> org.sakaiproject.search.indexer.impl.SearchBuilderQueueManager -
> Rebuild for 4d066f74-1cec-4a8e-bb15-99fab9af7a1b (59/65661)
> 2010-08-16 10:37:31,472  INFO Timer-2
> org.sakaiproject.content.impl.BaseContentService - objects in
> threadlocal:6084

I will be putting some code to clear threadlocal in the
SearchBuilderQueueManager

Regards

D


On 08/15/2010 04:14 PM, Jim Eng wrote:
> That cache had some problems, as I recall, and was not being used by the time I worked on CHS issues.  But I can't tell you what the problems were.   
>
> CHS does a lot of thread-local caching (or did when I last looked) of data relevant to processing the user's current request.  It also saves too much info in session state, which has the effect of caching info for a specific user across requests.  As with many things in Sakai, the permissions and volatility of data make caching difficult.  CHS needs to answer questions for different users in different ways depending on the specific users' permissions, group membership, etc.  And inappropriate caching (as in session state) can result in a user being denied access to resources when permissions or availability change.  I suspect some of these issues may have come up with that early attempt at caching in CHS.
>
> I'm just saying it's not as easy as turning on caching. 
>
> Jim
>
>
> On Aug 15, 2010, at 4:39 AM, David Horwitz wrote:
>
>   
>>  Hi All,
>>
>> While reviewing our production cache performance and pondering where 
>> Sakai migh benefit from more caches I discovered that ContentHosting has 
>> a cache but that its switched off by default.  I can't seem to find any 
>> documentation on this cache and wonder if there is a reason why it 
>> shouldn't be switched on?
>>
>> For the record it can be switched on with:
>>
>> caching at org.sakaiproject.content.api.ContentHostingService=true
>>
>>
>> Regards
>>
>> David
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
>>
>>
>>     
>