[Building Sakai] alternative to webdav in sakai

Wed Mar 9 13:27:33 PST 2011

I agree with these comments, I don't think you're being paranoid. There were
too many easy to reproduce edge cases that really could start up a nearly
denial of service back with the last patch that was not tested. Only the
"happy path" of a regular zip file with a few files in it was tested. If you
loaded up a zip file with thousands (hundreds of thousands?) of zero byte
files then the performance was much different. Also I remember being able to
generate some file names that generated exceptions that back then were
silently logged.

Having this process that has a good chance of taking longer than a few
second in the request thread is a long standing problem of Sakai. Something
like this (a task that is no other workflow depends on it's completion and
is potentially long running) really needs to be backgrounded or moved off to
some type of job processing server. And when it's processed, ideally, the
user notified that their (long running) job is complete. No mechanism
currently exists for this in Sakai. :(

It's also unfortunate that tomcat or java has no (seemingly easy?) way to
set any type of thread limits or timeouts like php (set_time_limit) does
with execution time. This might at least make administrators feel a little
safer. I guess you can setup an ExecutorService and run your function in
that? Still, gets into some pretty complex stuff for a "simple" feature.

I agree that this feature would be really awesome for a user but seems like
a scary addiition from an operations perspective.

On Wed, Mar 9, 2011 at 4:07 PM, Noah Botimer <botimer at umich.edu> wrote:

> Sure. When the original patch was merged, it came with a number of quality
> and completeness problems.
>
> https://jira.sakaiproject.org/browse/SAK-16036 and
> https://jira.sakaiproject.org/browse/KNL-155 capture the work that was
> done in the closing moments of the 2.6 release to disable the feature. The
> problems were discovered late and the fix was an ugly combination of
> properties and commented code.
>
> (I refer to this pretty broken code lying latent in trunk without a quality
> plan or much notice as being "parachuted in".)
>
> Perhaps the work done for KNL-273 addresses most of the problems, but I am
> not comfortable without seeing a standalone, cohesive write-up of the
> feature and its behavior. Tracking this across a handful of SAK/KNL tickets,
> some emails, and fifty-some JIRA comments just doesn't draw a clear picture.
>
> Specific technical questions are around the library used, performance,
> request-thread processing, mime-type settings, content scanning, and quota
> ramifications. We may or may not have a clear implementation plan that
> addresses these. I am especially concerned about the processing in the
> request thread and the impact on the database pool.
>
> Maybe I'm paranoid or maybe these have all been sufficiently addressed. But
> I would strongly urge those interested to do a complete and accurate
> write-up outside of the smattering of JIRA tickets, so we can make a
> conscious community decision.
>
> By the way, many thanks to Bryan and whomever else for working on a real
> user need.
>
> Thanks,
> -Noah
>
> On Mar 9, 2011, at 3:16 PM, Sam Ottenhoff wrote:
>
> > I know nothing about the previous issues or the mitigation that was
> > done.  SAK-800 comments don't seem to reflect any local catastrophes.
> > Can you give us some background?
> >
> > --Sam
> >
> > On 3/9/2011 3:11 PM, Noah Botimer wrote:
> >> With all due respect to the work done and the spoken need, I think there
> are still significant concerns with this approach. It solves a class of
> problem and introduces another.
> >>
> >> I am not familiar with the extent of the work done recently, but I am
> very leery of SAK-800. I will be disappointed if it gets parachuted in
> again. Even our mitigation was sloppy last time.
> >>
> >> This absolutely requires review and discussion on list and the TCC
> should form an opinion of the best path. This is not simple bug fixing or
> minor feature enhancement, given the technical and policy issues arising.
> >>
> >> None of this is to discourage movement forward, but it is a humble
> request that special care be taken in an area that has really stung us
> before.
> >>
> >> Thanks,
> >> -Noah
> >>
> >> On Mar 9, 2011, at 12:25 PM, Bryan Holladay wrote:
> >>
> >>> Thanks David... Anyone who wants it can take it.  I re-assigned it to
> the KERNEL TEAM in mid November.
> >>>
> >>> -Bryan
> >>>
> >>> On Mar 9, 2011, at 12:17 PM, DAVID ROLDAN MARTINEZ wrote:
> >>>
> >>>> Bryan,
> >>>>
> >>>> I offer myself to do this (or provide you help if you prefer to manage
> this yourself) and leave hardcoded messages out of kernel so that they will
> be handled at resources tool. But, as I'm not a member of the Kernel-team, I
> would like to get their feedback before to start working.
> >>>>
> >>>> Cheers,
> >>>>   David
> >>>> ________________________________________
> >>>> De: sakai-dev-bounces at collab.sakaiproject.org [
> sakai-dev-bounces at collab.sakaiproject.org] En nombre de Bryan Holladay [
> holladay at longsight.com]
> >>>> Enviado el: miércoles, 09 de marzo de 2011 17:51
> >>>> Para: Adam Marshall
> >>>> CC: sakai-dev at collab.sakaiproject.org
> >>>> Asunto: Re: [Building Sakai] alternative to webdav in sakai
> >>>>
> >>>> I've fixed this issue in https://jira.sakaiproject.org/browse/KNL-273,
> along with other issues with the unzip facility.
> >>>>
> >>>> The only reason it hasn't been put into trunk is a best code practices
> issue I wasn't able to figure out.  It's noted in the comments and is
> waiting for someone with a better understanding of the kernel code to look
> at it.
> >>>>
> >>>> Other than that, it's a good patch and works fine and would be safe to
> run.
> >>>>
> >>>> Thanks,
> >>>> Bryan
> >>>>
> >>>>
> >>>> On Mar 9, 2011, at 11:41 AM, Adam Marshall wrote:
> >>>>
> >>>> We use the unzip facility – it works OK for extraction but is not good
> for creating a zip file. You cannot zip from the root folder!
> >>>>
> >>>> From: sakai-dev-bounces at collab.sakaiproject.org<mailto:
> sakai-dev-bounces at collab.sakaiproject.org>  [mailto:
> sakai-dev-bounces at collab.sakaiproject.org] On Behalf Of Sam Ottenhoff
> >>>> Sent: 09 March 2011 16:35
> >>>> To: sakai-dev at collab.sakaiproject.org<mailto:
> sakai-dev at collab.sakaiproject.org>
> >>>> Subject: Re: [Building Sakai] alternative to webdav in sakai
> >>>>
> >>>> https://jira.sakaiproject.org/browse/SAK-800
> >>>>
> >>>> SAK-800 and KNL-273 would allow users to upload a compressed archive
> and have it unrolled in Resources.
> >>>>
> >>>> --Sam
> >>>>
> >>>> On 3/9/2011 11:19 AM, John Bush wrote:
> >>>> Over the years we've had continued problems with webdav.  The varying
> behaviors of different clients, the special authentication that doesn't play
> nice with SSO, and numerous other types of issues.  I'm beginning to think
> about alternatives that would allow for big bulk uploads that are more
> reliable.  Thinking about some type of browser plugin, flash or otherwise.
>  Has anyone else done some research into this already?
> >>>>
> >>>> --
> >>>> John Bush
> >>>> 602-490-0470
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>>
> >>>> sakai-dev mailing list
> >>>>
> >>>> sakai-dev at collab.sakaiproject.org<mailto:
> sakai-dev at collab.sakaiproject.org>
> >>>>
> >>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >>>>
> >>>>
> >>>>
> >>>> TO UNSUBSCRIBE: send email to
> sakai-dev-unsubscribe at collab.sakaiproject.org<mailto:
> sakai-dev-unsubscribe at collab.sakaiproject.org>  with a subject of
> "unsubscribe"
> >>>>
> >>>> _______________________________________________
> >>>> sakai-dev mailing list
> >>>> sakai-dev at collab.sakaiproject.org<mailto:
> sakai-dev at collab.sakaiproject.org>
> >>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >>>>
> >>>> TO UNSUBSCRIBE: send email to
> sakai-dev-unsubscribe at collab.sakaiproject.org<mailto:
> sakai-dev-unsubscribe at collab.sakaiproject.org>  with a subject of
> "unsubscribe"
> >>>>
> >>> _______________________________________________
> >>> sakai-dev mailing list
> >>> sakai-dev at collab.sakaiproject.org
> >>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >>>
> >>> TO UNSUBSCRIBE: send email to
> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
> >>>
> >>>
> >> _______________________________________________
> >> sakai-dev mailing list
> >> sakai-dev at collab.sakaiproject.org
> >> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >>
> >> TO UNSUBSCRIBE: send email to
> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
> > _______________________________________________
> > sakai-dev mailing list
> > sakai-dev at collab.sakaiproject.org
> > http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >
> > TO UNSUBSCRIBE: send email to
> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
> >
> >
>
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to
> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20110309/a407faf8/attachment.html 