[Building Sakai] Resources keeling over with 1000+ files in a folder

[Warwick Chapman]

Ok, wait, "load balancer" being reverse proxy... If that's the case, then
yes we use Pound as a reverse proxy to just 1 instance of Tomcat/Sakai.

I will check the timeout settings there.

-- Warwick Bruce Chapman | +27 83 7797 094 | http://warwickchapman.com


On Wed, Dec 8, 2010 at 1:59 PM, Stephen Marquard <stephen.marquard at uct.ac.za
> wrote:

> Then I don't know why you're getting a 500 response. Do you get it
> always, or sometimes?
>
> It might be worth trying WebDAV and the /access/content/group/siteid/
> view to see if those are also affected.
>
> Regards
> Stephen
>
>
> --
> Stephen Marquard, Learning Technologies Co-ordinator
> Centre for Educational Technology, University of Cape Town
> http://www.cet.uct.ac.za
> Email / IM (Jabber/XMPP): stephen.marquard at uct.ac.za
> Phone: +27-21-650-5037 Cell: +27-83-500-5290
>
>
> >>> Warwick Chapman <warwickchapman at gmail.com> 12/8/2010 12:42 AM >>>
> Except we've got no load balancers (yet) ...
>
> -- Warwick Bruce Chapman | +27 83 7797 094 | http://warwickchapman.com
>
>
> On Tue, Dec 7, 2010 at 3:00 PM, Stephen Marquard
> <stephen.marquard at uct.ac.za
> > wrote:
>
> > It could be a timeout value in the loadbalancer configuration, which
> is
> > the longest time it will wait to get a reply back from the tomcat
> > worker.
> >
> > Try accessing the tomcat directly at the http/s port (i.e. not via
> the
> > loadbalancer) and see if that behaves differently.
> >
> > When we ran into it, it was slow but usable when one user accessed a
> > folder like that (1000+ files) but caused general slowness when we
> had a
> > whole class of users trying to do that at the same time.
> >
> > Regards
> > Stephen
> >
> > Stephen Marquard, Learning Technologies Co-ordinator
> > Centre for Educational Technology, University of Cape Town
> > http://www.cet.uct.ac.za
> > Email/IM/XMPP: stephen.marquard at uct.ac.za
> > Phone: +27-21-650-5037 Cell: +27-83-500-5290
> >
> >
> >
> > >>> Warwick Chapman <warwickchapman at gmail.com> 12/7/2010 2:39 PM >>>
> > So what causes the "Internal Server Error"?  Is there not a timeout
> > value I
> > can adjust somewhere?
> >
> > -- Warwick Bruce Chapman | +27 83 7797 094 |
> http://warwickchapman.com
> >
> >
> >
> > On Tue, Dec 7, 2010 at 2:20 PM, Stephen Marquard
> > <stephen.marquard at uct.ac.za
> > > wrote:
> >
> > > Hi Warwick,
> > >
> > > We ran into this issue as well. It produces very high query
> volumes
> > to
> > > the database.
> > >
> > > The best workaround at this stage is to reorganize the files into
> a
> > > deeper hierarchy so you don't get so many in one folder.
> > >
> > > Regards
> > > Stephen
> > >
> > >
> > > --
> > > Stephen Marquard, Learning Technologies Co-ordinator
> > > Centre for Educational Technology, University of Cape Town
> > > http://www.cet.uct.ac.za
> > > Email / IM (Jabber/XMPP): stephen.marquard at uct.ac.za
> > > Phone: +27-21-650-5037 Cell: +27-83-500-5290
> > >
> > >
> > > >>> Warwick Chapman <warwickchapman at gmail.com> 12/7/2010 1:50 PM
> >>>
> > > Hi All
> > >
> > > We are using Umthombo resources to store several thousands PDFs
> > > relating to
> > > our candidate nomination process.  The forms are sorted into
> folders
> > > for the
> > > province each nominee comes from.  In South Africa, we have a
> great
> > > many
> > > more nominees in one province than in any other and this has
> > resulted
> > > in
> > > that folder getting more than 1000+ PDFs alone.
> > >
> > > Listing that folder has become a pain.  It loads very slowly as
> > > frequently
> > > returns an HTTP500.  I cannot find anything obvious in the
> > > catalina.out
> > >
> > > Any thoughts?  Can I change Resources to list a few items per
> page?
> > >
> > > -- Warwick Bruce Chapman | +27 83 7797 094 |
> > http://warwickchapman.com
> > >
> > >
> > >
> > >
> > >
> > > ###
> > > UNIVERSITY OF CAPE TOWN
> > >
> > > This e-mail is subject to the UCT ICT policies and e-mail
> disclaimer
> > > published on our website at
> > > http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable
> > from
> > > +27 21 650 9111. This e-mail is intended only for the person(s) to
> > whom
> > > it is addressed. If the e-mail has reached you in error, please
> > notify
> > > the author. If you are not the intended recipient of the e-mail
> you
> > may
> > > not use, disclose, copy, redirect or print the content. If this
> > e-mail
> > > is not related to the business of UCT it is sent by the sender in
> > the
> > > sender's individual capacity.
> > >
> > > ###
> > >
> > >
> >
> >
> >
> >
> > ###
> > UNIVERSITY OF CAPE TOWN
> >
> > This e-mail is subject to the UCT ICT policies and e-mail disclaimer
> > published on our website at
> > http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable
> from
> > +27 21 650 9111. This e-mail is intended only for the person(s) to
> whom
> > it is addressed. If the e-mail has reached you in error, please
> notify
> > the author. If you are not the intended recipient of the e-mail you
> may
> > not use, disclose, copy, redirect or print the content. If this
> e-mail
> > is not related to the business of UCT it is sent by the sender in
> the
> > sender's individual capacity.
> >
> > ###
> >
> >
>
>
>
>
>
> ###
> UNIVERSITY OF CAPE TOWN
>
> This e-mail is subject to the UCT ICT policies and e-mail disclaimer
> published on our website at
> http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from
> +27 21 650 9111. This e-mail is intended only for the person(s) to whom
> it is addressed. If the e-mail has reached you in error, please notify
> the author. If you are not the intended recipient of the e-mail you may
> not use, disclose, copy, redirect or print the content. If this e-mail
> is not related to the business of UCT it is sent by the sender in the
> sender's individual capacity.
>
> ###
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20101210/1e059038/attachment.html