[Building Sakai] SAK-800

Bryan Holladay holladay at longsight.com
Wed Mar 16 14:33:06 PDT 2011 

There has been some recent conversation lately about this... KNL-273 is an extension of SAK-800... below are some of the threads:


------------------------------------------


I went ahead and altered the patch for KNL-273.  I've added some new properties that will limit the size of a zip file that you can expand.  This is configurable and I feel that this is a good low tech approach to preventing a DOS from large zip files.  Also, you can choose to only have the ability to "Expand a zip" option and turn off the "Compress to zip" option (or visa versa)

the new sakai.properties are:

//used to limit the size of a zip file it will try to expand
resources.zip.expand.max=1000 (default) 

//used to either turn on/off just expanding zips
resources.zip.enable.expand=false  (default) 

//used to either turn on/off just compressing zips
resources.zip.enable.compress=false  (default) 

//also the prop to turn on both or default to the other props
resources.zip.enable=false  (default) 



The new patch is attached to the jira.  There is one for kernel1.1.x and trunk

-Bryan


On Mar 16, 2011, at 3:37 PM, Sam Ottenhoff wrote:

> last message in my inbox on this thread
> 
> -------- Original Message --------
> Subject:	Re: [Building Sakai] [maint] alternative to webdav in sakai
> Date:	Thu, 10 Mar 2011 17:21:42 +0000
> From:	Adam Marshall <adam.marshall at oucs.ox.ac.uk>
> To:	Nicola Monat-Jacobs <nicola at longsight.com>, John Bush <john.bush at rsmart.com>
> CC:	sakai-dev at collab.sakaiproject.org <sakai-dev at collab.sakaiproject.org>, <mt at collab.sakaiproject.org> <mt at collab.sakaiproject.org>
> 
> 
> I think an unzip (with no zip) would still be very useful ineed.
>  
> adam
>  
> From: sakai-dev-bounces at collab.sakaiproject.org [mailto:sakai-dev-bounces at collab.sakaiproject.org] On Behalf Of Nicola Monat-Jacobs
> Sent: 10 March 2011 16:54
> To: John Bush
> Cc: <mt at collab.sakaiproject.org>; sakai-dev at collab.sakaiproject.org
> Subject: Re: [Building Sakai] [maint] alternative to webdav in sakai
>  
> As far as I can tell from reading the JIRAs, the issues with SAK-800 were on the zipping side, rather than the unzipping side, right? Unless there was an issue with the 'upload and unzip a zip into resources' functionality that I'm not catching in the various JIRAs, what about just implemented that part of the functionality? It seems a shame to reject a feature because it's partner misbehaves.
>  
> Bulk uploading was the original use case that you were looking at, right John?
>  
> - Nicola
>  
> On Mar 9, 2011, at  3:51 PM, John Bush wrote:
> 
> 
> I spent a little time looking for an alternative today that doesn't involve webdav or SAK-800 (you guys aren't making me feel fuzzy about that idea).  I can't find a flash widget that will let you attach folders, and I think that is key, for people who are uploading say a bunch of html content that is all linked together.
> 
> I've settled on jupload which is an applet:
> http://jupload.sourceforge.net/
> 
> It does drag and drop, supports attaching folders, and can do a http PUT, which I'm not sure will be valuable or not.
> 
> I'm going to spend some time prototyping something that would say plug this into the upload page in resources.  We'll see where this goes...
> 
> On Wed, Mar 9, 2011 at 2:27 PM, Matthew Jones <jonespm at umich.edu> wrote:
> I agree with these comments, I don't think you're being paranoid. There were too many easy to reproduce edge cases that really could start up a nearly denial of service back with the last patch that was not tested. Only the "happy path" of a regular zip file with a few files in it was tested. If you loaded up a zip file with thousands (hundreds of thousands?) of zero byte files then the performance was much different. Also I remember being able to generate some file names that generated exceptions that back then were silently logged.
>  
> Having this process that has a good chance of taking longer than a few second in the request thread is a long standing problem of Sakai. Something like this (a task that is no other workflow depends on it's completion and is potentially long running) really needs to be backgrounded or moved off to some type of job processing server. And when it's processed, ideally, the user notified that their (long running) job is complete. No mechanism currently exists for this in Sakai. :(
>  
> It's also unfortunate that tomcat or java has no (seemingly easy?) way to set any type of thread limits or timeouts like php (set_time_limit) does with execution time. This might at least make administrators feel a little safer. I guess you can setup an ExecutorService and run your function in that? Still, gets into some pretty complex stuff for a "simple" feature.
>  
> I agree that this feature would be really awesome for a user but seems like a scary addiition from an operations perspective.
>  
> 
> On Wed, Mar 9, 2011 at 4:07 PM, Noah Botimer <botimer at umich.edu> wrote:
> Sure. When the original patch was merged, it came with a number of quality and completeness problems.
> 
> https://jira.sakaiproject.org/browse/SAK-16036 and https://jira.sakaiproject.org/browse/KNL-155 capture the work that was done in the closing moments of the 2.6 release to disable the feature. The problems were discovered late and the fix was an ugly combination of properties and commented code.
> 
> (I refer to this pretty broken code lying latent in trunk without a quality plan or much notice as being "parachuted in".)
> 
> Perhaps the work done for KNL-273 addresses most of the problems, but I am not comfortable without seeing a standalone, cohesive write-up of the feature and its behavior. Tracking this across a handful of SAK/KNL tickets, some emails, and fifty-some JIRA comments just doesn't draw a clear picture.
> 
> Specific technical questions are around the library used, performance, request-thread processing, mime-type settings, content scanning, and quota ramifications. We may or may not have a clear implementation plan that addresses these. I am especially concerned about the processing in the request thread and the impact on the database pool.
> 
> Maybe I'm paranoid or maybe these have all been sufficiently addressed. But I would strongly urge those interested to do a complete and accurate write-up outside of the smattering of JIRA tickets, so we can make a conscious community decision.
> 
> By the way, many thanks to Bryan and whomever else for working on a real user need.
> 
> Thanks,
> -Noah
> 
> On Mar 9, 2011, at 3:16 PM, Sam Ottenhoff wrote:
> 
> > I know nothing about the previous issues or the mitigation that was
> > done.  SAK-800 comments don't seem to reflect any local catastrophes.
> > Can you give us some background?
> >
> > --Sam
> >
> > On 3/9/2011 3:11 PM, Noah Botimer wrote:
> >> With all due respect to the work done and the spoken need, I think there are still significant concerns with this approach. It solves a class of problem and introduces another.
> >>
> >> I am not familiar with the extent of the work done recently, but I am very leery of SAK-800. I will be disappointed if it gets parachuted in again. Even our mitigation was sloppy last time.
> >>
> >> This absolutely requires review and discussion on list and the TCC should form an opinion of the best path. This is not simple bug fixing or minor feature enhancement, given the technical and policy issues arising.
> >>
> >> None of this is to discourage movement forward, but it is a humble request that special care be taken in an area that has really stung us before.
> >>
> >> Thanks,
> >> -Noah
> >>
> >> On Mar 9, 2011, at 12:25 PM, Bryan Holladay wrote:
> >>
> >>> Thanks David... Anyone who wants it can take it.  I re-assigned it to the KERNEL TEAM in mid November.
> >>>
> >>> -Bryan
> >>>
> >>> On Mar 9, 2011, at 12:17 PM, DAVID ROLDAN MARTINEZ wrote:
> >>>
> >>>> Bryan,
> >>>>
> >>>> I offer myself to do this (or provide you help if you prefer to manage this yourself) and leave hardcoded messages out of kernel so that they will be handled at resources tool. But, as I'm not a member of the Kernel-team, I would like to get their feedback before to start working.
> >>>>
> >>>> Cheers,
> >>>>   David
> >>>> ________________________________________
> >>>> De: sakai-dev-bounces at collab.sakaiproject.org [sakai-dev-bounces at collab.sakaiproject.org] En nombre de Bryan Holladay [holladay at longsight.com]
> >>>> Enviado el: miércoles, 09 de marzo de 2011 17:51
> >>>> Para: Adam Marshall
> >>>> CC: sakai-dev at collab.sakaiproject.org
> >>>> Asunto: Re: [Building Sakai] alternative to webdav in sakai
> >>>>
> >>>> I've fixed this issue in https://jira.sakaiproject.org/browse/KNL-273, along with other issues with the unzip facility.
> >>>>
> >>>> The only reason it hasn't been put into trunk is a best code practices issue I wasn't able to figure out.  It's noted in the comments and is waiting for someone with a better understanding of the kernel code to look at it.
> >>>>
> >>>> Other than that, it's a good patch and works fine and would be safe to run.
> >>>>
> >>>> Thanks,
> >>>> Bryan
> >>>>
> >>>>
> >>>> On Mar 9, 2011, at 11:41 AM, Adam Marshall wrote:
> >>>>
> >>>> We use the unzip facility – it works OK for extraction but is not good for creating a zip file. You cannot zip from the root folder!
> >>>>
> >>>> From: sakai-dev-bounces at collab.sakaiproject.org<mailto:sakai-dev-bounces at collab.sakaiproject.org>  [mailto:sakai-dev-bounces at collab.sakaiproject.org] On Behalf Of Sam Ottenhoff
> >>>> Sent: 09 March 2011 16:35
> >>>> To: sakai-dev at collab.sakaiproject.org<mailto:sakai-dev at collab.sakaiproject.org>
> >>>> Subject: Re: [Building Sakai] alternative to webdav in sakai
> >>>>
> >>>> https://jira.sakaiproject.org/browse/SAK-800
> >>>>
> >>>> SAK-800 and KNL-273 would allow users to upload a compressed archive and have it unrolled in Resources.
> >>>>
> >>>> --Sam
> >>>>
> >>>> On 3/9/2011 11:19 AM, John Bush wrote:
> >>>> Over the years we've had continued problems with webdav.  The varying behaviors of different clients, the special authentication that doesn't play nice with SSO, and numerous other types of issues.  I'm beginning to think about alternatives that would allow for big bulk uploads that are more reliable.  Thinking about some type of browser plugin, flash or otherwise.  Has anyone else done some research into this already?
> >>>>
> >>>> --
> >>>> John Bush
> >>>> 602-490-0470
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>>
> >>>> sakai-dev mailing list
> >>>>
> >>>> sakai-dev at collab.sakaiproject.org<mailto:sakai-dev at collab.sakaiproject.org>
> >>>>
> >>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >>>>
> >>>>
> >>>>
> >>>> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org<mailto:sakai-dev-unsubscribe at collab.sakaiproject.org>  with a subject of "unsubscribe"
> >>>>
> >>>> _______________________________________________
> >>>> sakai-dev mailing list
> >>>> sakai-dev at collab.sakaiproject.org<mailto:sakai-dev at collab.sakaiproject.org>
> >>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >>>>
> >>>> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org<mailto:sakai-dev-unsubscribe at collab.sakaiproject.org>  with a subject of "unsubscribe"
> >>>>
> >>> _______________________________________________
> >>> sakai-dev mailing list
> >>> sakai-dev at collab.sakaiproject.org
> >>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >>>
> >>> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
> >>>
> >>>
> >> _______________________________________________
> >> sakai-dev mailing list
> >> sakai-dev at collab.sakaiproject.org
> >> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >>
> >> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
> > _______________________________________________
> > sakai-dev mailing list
> > sakai-dev at collab.sakaiproject.org
> > http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >
> > TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
> >
> >
> 
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> 
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
>  
> 
> _______________________________________________
> mt mailing list
> mt at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/mt
> 
> 
> 
> 
> -- 
> John Bush
> 602-490-0470
> _______________________________________________
> mt mailing list
> mt at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/mt
>  
> <Attached Message Part.txt>

_______________________________________________
sakai-dev mailing list
sakai-dev at collab.sakaiproject.org
http://collab.sakaiproject.org/mailman/listinfo/sakai-dev

TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"






On Mar 16, 2011, at 5:25 PM, Adrian Fish wrote:

> Is anybody actively looking at SAK-800? If not, I will as this would be 
> incredibly useful for LessonBuilder as you'll be able to upload zips 
> generated by tools such as Camtasia, Articulate and Wimba Create and 
> then just link to them. We've also had the usual WebDAV grief ...
> 
> Cheers,
> Adrian.
> 
> -- 
> ==================================
> Adrian Fish
> Software Engineer
> Centre for e-Science
> Bowland Tower South C Floor
> Lancaster University
> Lancaster
> LA1 4YW
> email: a.fish at lancaster.ac.uk
> 
> http://confluence.sakaiproject.org/display/YAFT/Yaft
> http://confluence.sakaiproject.org/display/CLOG/Home
> http://confluence.sakaiproject.org/display/BBB/Home
> 
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> 
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20110316/463bf785/attachment.html

More information about the sakai-dev mailing list