[Building Sakai] SAK-800

Adrian Fish a.fish at lancaster.ac.uk
Wed Mar 16 15:20:38 PDT 2011 

Hi Bryan,

I'll attempt to apply the patch to my kernel and see how I get on. 
Thanks for the info, I should have read the JIRA more closely ...

Cheers,
Adrian.

On 16/03/2011 21:33, Bryan Holladay wrote:
> There has been some recent conversation lately about this... KNL-273 
> is an extension of SAK-800... below are some of the threads:
>
>
> ------------------------------------------
>
>
> I went ahead and altered the patch for KNL-273.  I've added some new 
> properties that will limit the size of a zip file that you can expand. 
>  This is configurable and I feel that this is a good low tech approach 
> to preventing a DOS from large zip files.  Also, you can choose to 
> only have the ability to "Expand a zip" option and turn off the 
> "Compress to zip" option (or visa versa)
>
> the new sakai.properties are:
>
> //used to limit the size of a zip file it will try to expand
> resources.zip.expand.max=1000 (default)
>
> //used to either turn on/off just expanding zips
> resources.zip.enable.expand=false  (default)
>
> //used to either turn on/off just compressing zips
> resources.zip.enable.compress=false  (default)
>
> //also the prop to turn on both or default to the other props
> resources.zip.enable=false  (default)
>
>
>
> The new patch is attached to the jira.  There is one for kernel1.1.x 
> and trunk
>
> -Bryan
>
>
> On Mar 16, 2011, at 3:37 PM, Sam Ottenhoff wrote:
>
>> last message in my inbox on this thread
>>
>> -------- Original Message --------
>> Subject: 	Re: [Building Sakai] [maint] alternative to webdav in sakai
>> Date: 	Thu, 10 Mar 2011 17:21:42 +0000
>> From: 	Adam Marshall <adam.marshall at oucs.ox.ac.uk>
>> To: 	Nicola Monat-Jacobs <nicola at longsight.com>, John Bush 
>> <john.bush at rsmart.com>
>> CC: 	sakai-dev at collab.sakaiproject.org 
>> <sakai-dev at collab.sakaiproject.org>, <mt at collab.sakaiproject.org> 
>> <mt at collab.sakaiproject.org>
>>
>>
>>
>> I think an unzip (with no zip) would still be very useful ineed.
>> adam
>> *From:*sakai-dev-bounces at collab.sakaiproject.org [mailto:sakai-dev-bounces at collab.sakaiproject.org] 
>> *On Behalf Of *Nicola Monat-Jacobs
>> *Sent:* 10 March 2011 16:54
>> *To:* John Bush
>> *Cc:* <mt at collab.sakaiproject.org>; sakai-dev at collab.sakaiproject.org
>> *Subject:* Re: [Building Sakai] [maint] alternative to webdav in sakai
>> As far as I can tell from reading the JIRAs, the issues with SAK-800 
>> were on the zipping side, rather than the unzipping side, right? 
>> Unless there was an issue with the 'upload and unzip a zip into 
>> resources' functionality that I'm not catching in the various JIRAs, 
>> what about just implemented that part of the functionality? It seems 
>> a shame to reject a feature because it's partner misbehaves.
>> Bulk uploading was the original use case that you were looking at, 
>> right John?
>> - Nicola
>> On Mar 9, 2011, at  3:51 PM, John Bush wrote:
>>
>>
>> I spent a little time looking for an alternative today that doesn't 
>> involve webdav or SAK-800 (you guys aren't making me feel fuzzy about 
>> that idea).  I can't find a flash widget that will let you attach 
>> folders, and I think that is key, for people who are uploading say a 
>> bunch of html content that is all linked together.
>>
>> I've settled on jupload which is an applet:
>> http://jupload.sourceforge.net/
>>
>> It does drag and drop, supports attaching folders, and can do a http 
>> PUT, which I'm not sure will be valuable or not.
>>
>> I'm going to spend some time prototyping something that would say 
>> plug this into the upload page in resources.  We'll see where this 
>> goes...
>>
>> On Wed, Mar 9, 2011 at 2:27 PM, Matthew Jones <jonespm at umich.edu 
>> <mailto:jonespm at umich.edu>> wrote:
>> I agree with these comments, I don't think you're being paranoid. 
>> There were too many easy to reproduce edge cases that really could 
>> start up a nearly denial of service back with the last patch that was 
>> not tested. Only the "happy path" of a regular zip file with a few 
>> files in it was tested. If you loaded up a zip file with thousands 
>> (hundreds of thousands?) of zero byte files then the performance was 
>> much different. Also I remember being able to generate some file 
>> names that generated exceptions that back then were silently logged.
>> Having this process that has a good chance of taking longer than a 
>> few second in the request thread is a long standing problem of Sakai. 
>> Something like this (a task that is no other workflow depends on it's 
>> completion and is potentially long running) really needs to be 
>> backgrounded or moved off to some type of job processing server. And 
>> when it's processed, ideally, the user notified that their (long 
>> running) job is complete. No mechanism currently exists for this in 
>> Sakai. :(
>> It's also unfortunate that tomcat or java has no (seemingly easy?) 
>> way to set any type of thread limits or timeouts like php 
>> (set_time_limit) does with execution time. This might at least make 
>> administrators feel a little safer. I guess you can setup 
>> an ExecutorService and run your function in that? Still, gets into 
>> some pretty complex stuff for a "simple" feature.
>> I agree that this feature would be really awesome for a user but 
>> seems like a scary addiition from an operations perspective.
>>
>> On Wed, Mar 9, 2011 at 4:07 PM, Noah Botimer <botimer at umich.edu 
>> <mailto:botimer at umich.edu>> wrote:
>> Sure. When the original patch was merged, it came with a number of 
>> quality and completeness problems.
>>
>> https://jira.sakaiproject.org/browse/SAK-16036 and 
>> https://jira.sakaiproject.org/browse/KNL-155 capture the work that 
>> was done in the closing moments of the 2.6 release to disable the 
>> feature. The problems were discovered late and the fix was an ugly 
>> combination of properties and commented code.
>>
>> (I refer to this pretty broken code lying latent in trunk without a 
>> quality plan or much notice as being "parachuted in".)
>>
>> Perhaps the work done for KNL-273 addresses most of the problems, but 
>> I am not comfortable without seeing a standalone, cohesive write-up 
>> of the feature and its behavior. Tracking this across a handful of 
>> SAK/KNL tickets, some emails, and fifty-some JIRA comments just 
>> doesn't draw a clear picture.
>>
>> Specific technical questions are around the library used, 
>> performance, request-thread processing, mime-type settings, content 
>> scanning, and quota ramifications. We may or may not have a clear 
>> implementation plan that addresses these. I am especially concerned 
>> about the processing in the request thread and the impact on the 
>> database pool.
>>
>> Maybe I'm paranoid or maybe these have all been sufficiently 
>> addressed. But I would strongly urge those interested to do a 
>> complete and accurate write-up outside of the smattering of JIRA 
>> tickets, so we can make a conscious community decision.
>>
>> By the way, many thanks to Bryan and whomever else for working on a 
>> real user need.
>>
>> Thanks,
>> -Noah
>>
>> On Mar 9, 2011, at 3:16 PM, Sam Ottenhoff wrote:
>>
>> > I know nothing about the previous issues or the mitigation that was
>> > done.  SAK-800 comments don't seem to reflect any local catastrophes.
>> > Can you give us some background?
>> >
>> > --Sam
>> >
>> > On 3/9/2011 3:11 PM, Noah Botimer wrote:
>> >> With all due respect to the work done and the spoken need, I think 
>> there are still significant concerns with this approach. It solves a 
>> class of problem and introduces another.
>> >>
>> >> I am not familiar with the extent of the work done recently, but I 
>> am very leery of SAK-800. I will be disappointed if it gets 
>> parachuted in again. Even our mitigation was sloppy last time.
>> >>
>> >> This absolutely requires review and discussion on list and the TCC 
>> should form an opinion of the best path. This is not simple bug 
>> fixing or minor feature enhancement, given the technical and policy 
>> issues arising.
>> >>
>> >> None of this is to discourage movement forward, but it is a humble 
>> request that special care be taken in an area that has really stung 
>> us before.
>> >>
>> >> Thanks,
>> >> -Noah
>> >>
>> >> On Mar 9, 2011, at 12:25 PM, Bryan Holladay wrote:
>> >>
>> >>> Thanks David... Anyone who wants it can take it.  I re-assigned 
>> it to the KERNEL TEAM in mid November.
>> >>>
>> >>> -Bryan
>> >>>
>> >>> On Mar 9, 2011, at 12:17 PM, DAVID ROLDAN MARTINEZ wrote:
>> >>>
>> >>>> Bryan,
>> >>>>
>> >>>> I offer myself to do this (or provide you help if you prefer to 
>> manage this yourself) and leave hardcoded messages out of kernel so 
>> that they will be handled at resources tool. But, as I'm not a member 
>> of the Kernel-team, I would like to get their feedback before to 
>> start working.
>> >>>>
>> >>>> Cheers,
>> >>>>   David
>> >>>> ________________________________________
>> >>>> De: sakai-dev-bounces at collab.sakaiproject.org 
>> <mailto:sakai-dev-bounces at collab.sakaiproject.org> [sakai-dev-bounces at collab.sakaiproject.org 
>> <mailto:sakai-dev-bounces at collab.sakaiproject.org>] En nombre de 
>> Bryan Holladay [holladay at longsight.com <mailto:holladay at longsight.com>]
>> >>>> Enviado el: miércoles, 09 de marzo de 2011 17:51
>> >>>> Para: Adam Marshall
>> >>>> CC: sakai-dev at collab.sakaiproject.org 
>> <mailto:sakai-dev at collab.sakaiproject.org>
>> >>>> Asunto: Re: [Building Sakai] alternative to webdav in sakai
>> >>>>
>> >>>> I've fixed this issue in 
>> https://jira.sakaiproject.org/browse/KNL-273, along with other issues 
>> with the unzip facility.
>> >>>>
>> >>>> The only reason it hasn't been put into trunk is a best code 
>> practices issue I wasn't able to figure out.  It's noted in the 
>> comments and is waiting for someone with a better understanding of 
>> the kernel code to look at it.
>> >>>>
>> >>>> Other than that, it's a good patch and works fine and would be 
>> safe to run.
>> >>>>
>> >>>> Thanks,
>> >>>> Bryan
>> >>>>
>> >>>>
>> >>>> On Mar 9, 2011, at 11:41 AM, Adam Marshall wrote:
>> >>>>
>> >>>> We use the unzip facility – it works OK for extraction but is 
>> not good for creating a zip file. You cannot zip from the root folder!
>> >>>>
>> >>>> From: sakai-dev-bounces at collab.sakaiproject.org 
>> <mailto:sakai-dev-bounces at collab.sakaiproject.org><mailto:sakai-dev-bounces at collab.sakaiproject.org 
>> <mailto:sakai-dev-bounces at collab.sakaiproject.org>> 
>>  [mailto:sakai-dev-bounces at collab.sakaiproject.org 
>> <mailto:sakai-dev-bounces at collab.sakaiproject.org>] On Behalf Of Sam 
>> Ottenhoff
>> >>>> Sent: 09 March 2011 16:35
>> >>>> To: sakai-dev at collab.sakaiproject.org 
>> <mailto:sakai-dev at collab.sakaiproject.org><mailto:sakai-dev at collab.sakaiproject.org 
>> <mailto:sakai-dev at collab.sakaiproject.org>>
>> >>>> Subject: Re: [Building Sakai] alternative to webdav in sakai
>> >>>>
>> >>>> https://jira.sakaiproject.org/browse/SAK-800
>> >>>>
>> >>>> SAK-800 and KNL-273 would allow users to upload a compressed 
>> archive and have it unrolled in Resources.
>> >>>>
>> >>>> --Sam
>> >>>>
>> >>>> On 3/9/2011 11:19 AM, John Bush wrote:
>> >>>> Over the years we've had continued problems with webdav.  The 
>> varying behaviors of different clients, the special authentication 
>> that doesn't play nice with SSO, and numerous other types of issues. 
>>  I'm beginning to think about alternatives that would allow for big 
>> bulk uploads that are more reliable.  Thinking about some type of 
>> browser plugin, flash or otherwise.  Has anyone else done some 
>> research into this already?
>> >>>>
>> >>>> --
>> >>>> John Bush
>> >>>> 602-490-0470
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>>
>> >>>> sakai-dev mailing list
>> >>>>
>> >>>> sakai-dev at collab.sakaiproject.org 
>> <mailto:sakai-dev at collab.sakaiproject.org><mailto:sakai-dev at collab.sakaiproject.org 
>> <mailto:sakai-dev at collab.sakaiproject.org>>
>> >>>>
>> >>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>> >>>>
>> >>>>
>> >>>>
>> >>>> TO UNSUBSCRIBE: send email to 
>> sakai-dev-unsubscribe at collab.sakaiproject.org 
>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org><mailto:sakai-dev-unsubscribe at collab.sakaiproject.org 
>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org>>  with a 
>> subject of "unsubscribe"
>> >>>>
>> >>>> _______________________________________________
>> >>>> sakai-dev mailing list
>> >>>> sakai-dev at collab.sakaiproject.org 
>> <mailto:sakai-dev at collab.sakaiproject.org><mailto:sakai-dev at collab.sakaiproject.org 
>> <mailto:sakai-dev at collab.sakaiproject.org>>
>> >>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>> >>>>
>> >>>> TO UNSUBSCRIBE: send email to 
>> sakai-dev-unsubscribe at collab.sakaiproject.org 
>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org><mailto:sakai-dev-unsubscribe at collab.sakaiproject.org 
>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org>>  with a 
>> subject of "unsubscribe"
>> >>>>
>> >>> _______________________________________________
>> >>> sakai-dev mailing list
>> >>> sakai-dev at collab.sakaiproject.org 
>> <mailto:sakai-dev at collab.sakaiproject.org>
>> >>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>> >>>
>> >>> TO UNSUBSCRIBE: send email to 
>> sakai-dev-unsubscribe at collab.sakaiproject.org 
>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a subject 
>> of "unsubscribe"
>> >>>
>> >>>
>> >> _______________________________________________
>> >> sakai-dev mailing list
>> >> sakai-dev at collab.sakaiproject.org 
>> <mailto:sakai-dev at collab.sakaiproject.org>
>> >> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>> >>
>> >> TO UNSUBSCRIBE: send email to 
>> sakai-dev-unsubscribe at collab.sakaiproject.org 
>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a subject 
>> of "unsubscribe"
>> > _______________________________________________
>> > sakai-dev mailing list
>> > sakai-dev at collab.sakaiproject.org 
>> <mailto:sakai-dev at collab.sakaiproject.org>
>> > http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>> >
>> > TO UNSUBSCRIBE: send email to 
>> sakai-dev-unsubscribe at collab.sakaiproject.org 
>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a subject 
>> of "unsubscribe"
>> >
>> >
>>
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org 
>> <mailto:sakai-dev at collab.sakaiproject.org>
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to 
>> sakai-dev-unsubscribe at collab.sakaiproject.org 
>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a subject 
>> of "unsubscribe"
>>
>>
>> _______________________________________________
>> mt mailing list
>> mt at collab.sakaiproject.org <mailto:mt at collab.sakaiproject.org>
>> http://collab.sakaiproject.org/mailman/listinfo/mt
>>
>>
>>
>>
>> -- 
>> John Bush
>> 602-490-0470
>> _______________________________________________
>> mt mailing list
>> mt at collab.sakaiproject.org <mailto:mt at collab.sakaiproject.org>
>> http://collab.sakaiproject.org/mailman/listinfo/mt
>> <Attached Message Part.txt>
>
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org 
> <mailto:sakai-dev at collab.sakaiproject.org>
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to 
> sakai-dev-unsubscribe at collab.sakaiproject.org 
> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a subject 
> of "unsubscribe"
>
>
>
>
>
>
> On Mar 16, 2011, at 5:25 PM, Adrian Fish wrote:
>
>> Is anybody actively looking at SAK-800? If not, I will as this would be
>> incredibly useful for LessonBuilder as you'll be able to upload zips
>> generated by tools such as Camtasia, Articulate and Wimba Create and
>> then just link to them. We've also had the usual WebDAV grief ...
>>
>> Cheers,
>> Adrian.
>>
>> -- 
>> ==================================
>> Adrian Fish
>> Software Engineer
>> Centre for e-Science
>> Bowland Tower South C Floor
>> Lancaster University
>> Lancaster
>> LA1 4YW
>> email: a.fish at lancaster.ac.uk <mailto:a.fish at lancaster.ac.uk>
>>
>> http://confluence.sakaiproject.org/display/YAFT/Yaft
>> http://confluence.sakaiproject.org/display/CLOG/Home
>> http://confluence.sakaiproject.org/display/BBB/Home
>>
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to 
>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of 
>> "unsubscribe"
>

-- 
==================================
Adrian Fish
Software Engineer
Centre for e-Science
Bowland Tower South C Floor
Lancaster University
Lancaster
LA1 4YW
email: a.fish at lancaster.ac.uk

http://confluence.sakaiproject.org/display/YAFT/Yaft
http://confluence.sakaiproject.org/display/CLOG/Home
http://confluence.sakaiproject.org/display/BBB/Home

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20110316/58388ab6/attachment.html

More information about the sakai-dev mailing list