[Building Sakai] SAK-800
Sam Ottenhoff
ottenhoff at longsight.com
Wed Mar 16 20:08:18 PDT 2011
If you are able to test the patch on a local install, any feedback would
be appreciated. With just the unzip functionality enabled on a dev
install, I have not run into any issues with the new functionality. I
have tested with very large ZIPs, zero-byte files, empty folders, etc.
Please leave any testing feedback in JIRA: KNL-273.
--Sam
On 3/16/2011 7:33 PM, Nicolas Lupien wrote:
> I was looking at the same thing as you today. As a user, i can't wait
> to see this in the main release.
>
> We've just made the switch from Moodle to Sakai and this seems to be
> the hardest part.
>
> Nicolas Lupien
>
>
>
> On Wed, Mar 16, 2011 at 6:20 PM, Adrian Fish <a.fish at lancaster.ac.uk
> <mailto:a.fish at lancaster.ac.uk>> wrote:
>
> Hi Bryan,
>
> I'll attempt to apply the patch to my kernel and see how I get on.
> Thanks for the info, I should have read the JIRA more closely ...
>
> Cheers,
> Adrian.
>
>
> On 16/03/2011 21:33, Bryan Holladay wrote:
>> There has been some recent conversation lately about this...
>> KNL-273 is an extension of SAK-800... below are some of the threads:
>>
>>
>> ------------------------------------------
>>
>>
>> I went ahead and altered the patch for KNL-273. I've added some
>> new properties that will limit the size of a zip file that you
>> can expand. This is configurable and I feel that this is a good
>> low tech approach to preventing a DOS from large zip files.
>> Also, you can choose to only have the ability to "Expand a zip"
>> option and turn off the "Compress to zip" option (or visa versa)
>>
>> the new sakai.properties are:
>>
>> //used to limit the size of a zip file it will try to expand
>> resources.zip.expand.max=1000 (default)
>>
>> //used to either turn on/off just expanding zips
>> resources.zip.enable.expand=false (default)
>>
>> //used to either turn on/off just compressing zips
>> resources.zip.enable.compress=false (default)
>>
>> //also the prop to turn on both or default to the other props
>> resources.zip.enable=false (default)
>>
>>
>>
>> The new patch is attached to the jira. There is one for
>> kernel1.1.x and trunk
>>
>> -Bryan
>>
>>
>> On Mar 16, 2011, at 3:37 PM, Sam Ottenhoff wrote:
>>
>>> last message in my inbox on this thread
>>>
>>> -------- Original Message --------
>>> Subject: Re: [Building Sakai] [maint] alternative to webdav in
>>> sakai
>>> Date: Thu, 10 Mar 2011 17:21:42 +0000
>>> From: Adam Marshall <adam.marshall at oucs.ox.ac.uk>
>>> <mailto:adam.marshall at oucs.ox.ac.uk>
>>> To: Nicola Monat-Jacobs <nicola at longsight.com>
>>> <mailto:nicola at longsight.com>, John Bush <john.bush at rsmart.com>
>>> <mailto:john.bush at rsmart.com>
>>> CC: sakai-dev at collab.sakaiproject.org
>>> <mailto:sakai-dev at collab.sakaiproject.org>
>>> <sakai-dev at collab.sakaiproject.org>
>>> <mailto:sakai-dev at collab.sakaiproject.org>,
>>> <mt at collab.sakaiproject.org> <mailto:mt at collab.sakaiproject.org>
>>> <mt at collab.sakaiproject.org> <mailto:mt at collab.sakaiproject.org>
>>>
>>>
>>>
>>> I think an unzip (with no zip) would still be very useful ineed.
>>> adam
>>> *From:*sakai-dev-bounces at collab.sakaiproject.org
>>> <mailto:sakai-dev-bounces at collab.sakaiproject.org> [mailto:sakai-dev-bounces at collab.sakaiproject.org]
>>> *On Behalf Of *Nicola Monat-Jacobs
>>> *Sent:* 10 March 2011 16:54
>>> *To:* John Bush
>>> *Cc:* <mt at collab.sakaiproject.org>
>>> <mailto:mt at collab.sakaiproject.org>;
>>> sakai-dev at collab.sakaiproject.org
>>> <mailto:sakai-dev at collab.sakaiproject.org>
>>> *Subject:* Re: [Building Sakai] [maint] alternative to webdav in
>>> sakai
>>> As far as I can tell from reading the JIRAs, the issues with
>>> SAK-800 were on the zipping side, rather than the unzipping
>>> side, right? Unless there was an issue with the 'upload and
>>> unzip a zip into resources' functionality that I'm not catching
>>> in the various JIRAs, what about just implemented that part of
>>> the functionality? It seems a shame to reject a feature because
>>> it's partner misbehaves.
>>> Bulk uploading was the original use case that you were looking
>>> at, right John?
>>> - Nicola
>>> On Mar 9, 2011, at 3:51 PM, John Bush wrote:
>>>
>>>
>>> I spent a little time looking for an alternative today that
>>> doesn't involve webdav or SAK-800 (you guys aren't making me
>>> feel fuzzy about that idea). I can't find a flash widget that
>>> will let you attach folders, and I think that is key, for people
>>> who are uploading say a bunch of html content that is all linked
>>> together.
>>>
>>> I've settled on jupload which is an applet:
>>> http://jupload.sourceforge.net/
>>>
>>> It does drag and drop, supports attaching folders, and can do a
>>> http PUT, which I'm not sure will be valuable or not.
>>>
>>> I'm going to spend some time prototyping something that would
>>> say plug this into the upload page in resources. We'll see
>>> where this goes...
>>>
>>> On Wed, Mar 9, 2011 at 2:27 PM, Matthew Jones <jonespm at umich.edu
>>> <mailto:jonespm at umich.edu>> wrote:
>>> I agree with these comments, I don't think you're being
>>> paranoid. There were too many easy to reproduce edge cases that
>>> really could start up a nearly denial of service back with the
>>> last patch that was not tested. Only the "happy path" of a
>>> regular zip file with a few files in it was tested. If you
>>> loaded up a zip file with thousands (hundreds of thousands?) of
>>> zero byte files then the performance was much different. Also I
>>> remember being able to generate some file names that generated
>>> exceptions that back then were silently logged.
>>> Having this process that has a good chance of taking longer than
>>> a few second in the request thread is a long standing problem of
>>> Sakai. Something like this (a task that is no other workflow
>>> depends on it's completion and is potentially long running)
>>> really needs to be backgrounded or moved off to some type of job
>>> processing server. And when it's processed, ideally, the user
>>> notified that their (long running) job is complete. No mechanism
>>> currently exists for this in Sakai. :(
>>> It's also unfortunate that tomcat or java has no (seemingly
>>> easy?) way to set any type of thread limits or timeouts like php
>>> (set_time_limit) does with execution time. This might at least
>>> make administrators feel a little safer. I guess you can setup
>>> an ExecutorService and run your function in that? Still, gets
>>> into some pretty complex stuff for a "simple" feature.
>>> I agree that this feature would be really awesome for a user but
>>> seems like a scary addiition from an operations perspective.
>>>
>>> On Wed, Mar 9, 2011 at 4:07 PM, Noah Botimer <botimer at umich.edu
>>> <mailto:botimer at umich.edu>> wrote:
>>> Sure. When the original patch was merged, it came with a number
>>> of quality and completeness problems.
>>>
>>> https://jira.sakaiproject.org/browse/SAK-16036 and
>>> https://jira.sakaiproject.org/browse/KNL-155 capture the work
>>> that was done in the closing moments of the 2.6 release to
>>> disable the feature. The problems were discovered late and the
>>> fix was an ugly combination of properties and commented code.
>>>
>>> (I refer to this pretty broken code lying latent in trunk
>>> without a quality plan or much notice as being "parachuted in".)
>>>
>>> Perhaps the work done for KNL-273 addresses most of the
>>> problems, but I am not comfortable without seeing a standalone,
>>> cohesive write-up of the feature and its behavior. Tracking this
>>> across a handful of SAK/KNL tickets, some emails, and fifty-some
>>> JIRA comments just doesn't draw a clear picture.
>>>
>>> Specific technical questions are around the library used,
>>> performance, request-thread processing, mime-type settings,
>>> content scanning, and quota ramifications. We may or may not
>>> have a clear implementation plan that addresses these. I am
>>> especially concerned about the processing in the request thread
>>> and the impact on the database pool.
>>>
>>> Maybe I'm paranoid or maybe these have all been sufficiently
>>> addressed. But I would strongly urge those interested to do a
>>> complete and accurate write-up outside of the smattering of JIRA
>>> tickets, so we can make a conscious community decision.
>>>
>>> By the way, many thanks to Bryan and whomever else for working
>>> on a real user need.
>>>
>>> Thanks,
>>> -Noah
>>>
>>> On Mar 9, 2011, at 3:16 PM, Sam Ottenhoff wrote:
>>>
>>> > I know nothing about the previous issues or the mitigation
>>> that was
>>> > done. SAK-800 comments don't seem to reflect any local
>>> catastrophes.
>>> > Can you give us some background?
>>> >
>>> > --Sam
>>> >
>>> > On 3/9/2011 3:11 PM, Noah Botimer wrote:
>>> >> With all due respect to the work done and the spoken need, I
>>> think there are still significant concerns with this approach.
>>> It solves a class of problem and introduces another.
>>> >>
>>> >> I am not familiar with the extent of the work done recently,
>>> but I am very leery of SAK-800. I will be disappointed if it
>>> gets parachuted in again. Even our mitigation was sloppy last time.
>>> >>
>>> >> This absolutely requires review and discussion on list and
>>> the TCC should form an opinion of the best path. This is not
>>> simple bug fixing or minor feature enhancement, given the
>>> technical and policy issues arising.
>>> >>
>>> >> None of this is to discourage movement forward, but it is a
>>> humble request that special care be taken in an area that has
>>> really stung us before.
>>> >>
>>> >> Thanks,
>>> >> -Noah
>>> >>
>>> >> On Mar 9, 2011, at 12:25 PM, Bryan Holladay wrote:
>>> >>
>>> >>> Thanks David... Anyone who wants it can take it. I
>>> re-assigned it to the KERNEL TEAM in mid November.
>>> >>>
>>> >>> -Bryan
>>> >>>
>>> >>> On Mar 9, 2011, at 12:17 PM, DAVID ROLDAN MARTINEZ wrote:
>>> >>>
>>> >>>> Bryan,
>>> >>>>
>>> >>>> I offer myself to do this (or provide you help if you
>>> prefer to manage this yourself) and leave hardcoded messages out
>>> of kernel so that they will be handled at resources tool. But,
>>> as I'm not a member of the Kernel-team, I would like to get
>>> their feedback before to start working.
>>> >>>>
>>> >>>> Cheers,
>>> >>>> David
>>> >>>> ________________________________________
>>> >>>> De: sakai-dev-bounces at collab.sakaiproject.org
>>> <mailto:sakai-dev-bounces at collab.sakaiproject.org> [sakai-dev-bounces at collab.sakaiproject.org
>>> <mailto:sakai-dev-bounces at collab.sakaiproject.org>] En nombre de
>>> Bryan Holladay [holladay at longsight.com
>>> <mailto:holladay at longsight.com>]
>>> >>>> Enviado el: miércoles, 09 de marzo de 2011 17:51
>>> >>>> Para: Adam Marshall
>>> >>>> CC: sakai-dev at collab.sakaiproject.org
>>> <mailto:sakai-dev at collab.sakaiproject.org>
>>> >>>> Asunto: Re: [Building Sakai] alternative to webdav in sakai
>>> >>>>
>>> >>>> I've fixed this issue in
>>> https://jira.sakaiproject.org/browse/KNL-273, along with other
>>> issues with the unzip facility.
>>> >>>>
>>> >>>> The only reason it hasn't been put into trunk is a best
>>> code practices issue I wasn't able to figure out. It's noted in
>>> the comments and is waiting for someone with a better
>>> understanding of the kernel code to look at it.
>>> >>>>
>>> >>>> Other than that, it's a good patch and works fine and would
>>> be safe to run.
>>> >>>>
>>> >>>> Thanks,
>>> >>>> Bryan
>>> >>>>
>>> >>>>
>>> >>>> On Mar 9, 2011, at 11:41 AM, Adam Marshall wrote:
>>> >>>>
>>> >>>> We use the unzip facility – it works OK for extraction but
>>> is not good for creating a zip file. You cannot zip from the
>>> root folder!
>>> >>>>
>>> >>>> From: sakai-dev-bounces at collab.sakaiproject.org
>>> <mailto:sakai-dev-bounces at collab.sakaiproject.org><mailto:sakai-dev-bounces at collab.sakaiproject.org
>>> <mailto:sakai-dev-bounces at collab.sakaiproject.org>>
>>> [mailto:sakai-dev-bounces at collab.sakaiproject.org
>>> <mailto:sakai-dev-bounces at collab.sakaiproject.org>] On Behalf Of
>>> Sam Ottenhoff
>>> >>>> Sent: 09 March 2011 16:35
>>> >>>> To: sakai-dev at collab.sakaiproject.org
>>> <mailto:sakai-dev at collab.sakaiproject.org><mailto:sakai-dev at collab.sakaiproject.org
>>> <mailto:sakai-dev at collab.sakaiproject.org>>
>>> >>>> Subject: Re: [Building Sakai] alternative to webdav in sakai
>>> >>>>
>>> >>>> https://jira.sakaiproject.org/browse/SAK-800
>>> >>>>
>>> >>>> SAK-800 and KNL-273 would allow users to upload a
>>> compressed archive and have it unrolled in Resources.
>>> >>>>
>>> >>>> --Sam
>>> >>>>
>>> >>>> On 3/9/2011 11:19 AM, John Bush wrote:
>>> >>>> Over the years we've had continued problems with webdav.
>>> The varying behaviors of different clients, the special
>>> authentication that doesn't play nice with SSO, and numerous
>>> other types of issues. I'm beginning to think about
>>> alternatives that would allow for big bulk uploads that are more
>>> reliable. Thinking about some type of browser plugin, flash or
>>> otherwise. Has anyone else done some research into this already?
>>> >>>>
>>> >>>> --
>>> >>>> John Bush
>>> >>>> 602-490-0470
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>>
>>> >>>> sakai-dev mailing list
>>> >>>>
>>> >>>> sakai-dev at collab.sakaiproject.org
>>> <mailto:sakai-dev at collab.sakaiproject.org><mailto:sakai-dev at collab.sakaiproject.org
>>> <mailto:sakai-dev at collab.sakaiproject.org>>
>>> >>>>
>>> >>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> TO UNSUBSCRIBE: send email to
>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org><mailto:sakai-dev-unsubscribe at collab.sakaiproject.org
>>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org>> with a
>>> subject of "unsubscribe"
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>> sakai-dev mailing list
>>> >>>> sakai-dev at collab.sakaiproject.org
>>> <mailto:sakai-dev at collab.sakaiproject.org><mailto:sakai-dev at collab.sakaiproject.org
>>> <mailto:sakai-dev at collab.sakaiproject.org>>
>>> >>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>> >>>>
>>> >>>> TO UNSUBSCRIBE: send email to
>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org><mailto:sakai-dev-unsubscribe at collab.sakaiproject.org
>>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org>> with a
>>> subject of "unsubscribe"
>>> >>>>
>>> >>> _______________________________________________
>>> >>> sakai-dev mailing list
>>> >>> sakai-dev at collab.sakaiproject.org
>>> <mailto:sakai-dev at collab.sakaiproject.org>
>>> >>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>> >>>
>>> >>> TO UNSUBSCRIBE: send email to
>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a
>>> subject of "unsubscribe"
>>> >>>
>>> >>>
>>> >> _______________________________________________
>>> >> sakai-dev mailing list
>>> >> sakai-dev at collab.sakaiproject.org
>>> <mailto:sakai-dev at collab.sakaiproject.org>
>>> >> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>> >>
>>> >> TO UNSUBSCRIBE: send email to
>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a
>>> subject of "unsubscribe"
>>> > _______________________________________________
>>> > sakai-dev mailing list
>>> > sakai-dev at collab.sakaiproject.org
>>> <mailto:sakai-dev at collab.sakaiproject.org>
>>> > http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>> >
>>> > TO UNSUBSCRIBE: send email to
>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a
>>> subject of "unsubscribe"
>>> >
>>> >
>>>
>>> _______________________________________________
>>> sakai-dev mailing list
>>> sakai-dev at collab.sakaiproject.org
>>> <mailto:sakai-dev at collab.sakaiproject.org>
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>
>>> TO UNSUBSCRIBE: send email to
>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a
>>> subject of "unsubscribe"
>>>
>>>
>>> _______________________________________________
>>> mt mailing list
>>> mt at collab.sakaiproject.org <mailto:mt at collab.sakaiproject.org>
>>> http://collab.sakaiproject.org/mailman/listinfo/mt
>>>
>>>
>>>
>>>
>>> --
>>> John Bush
>>> 602-490-0470
>>> _______________________________________________
>>> mt mailing list
>>> mt at collab.sakaiproject.org <mailto:mt at collab.sakaiproject.org>
>>> http://collab.sakaiproject.org/mailman/listinfo/mt
>>> <Attached Message Part.txt>
>>
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> <mailto:sakai-dev at collab.sakaiproject.org>
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to
>> sakai-dev-unsubscribe at collab.sakaiproject.org
>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a
>> subject of "unsubscribe"
>>
>>
>>
>>
>>
>>
>> On Mar 16, 2011, at 5:25 PM, Adrian Fish wrote:
>>
>>> Is anybody actively looking at SAK-800? If not, I will as this
>>> would be
>>> incredibly useful for LessonBuilder as you'll be able to upload
>>> zips
>>> generated by tools such as Camtasia, Articulate and Wimba Create
>>> and
>>> then just link to them. We've also had the usual WebDAV grief ...
>>>
>>> Cheers,
>>> Adrian.
>>>
>>> --
>>> ==================================
>>> Adrian Fish
>>> Software Engineer
>>> Centre for e-Science
>>> Bowland Tower South C Floor
>>> Lancaster University
>>> Lancaster
>>> LA1 4YW
>>> email: a.fish at lancaster.ac.uk <mailto:a.fish at lancaster.ac.uk>
>>>
>>> http://confluence.sakaiproject.org/display/YAFT/Yaft
>>> http://confluence.sakaiproject.org/display/CLOG/Home
>>> http://confluence.sakaiproject.org/display/BBB/Home
>>>
>>> _______________________________________________
>>> sakai-dev mailing list
>>> sakai-dev at collab.sakaiproject.org
>>> <mailto:sakai-dev at collab.sakaiproject.org>
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>
>>> TO UNSUBSCRIBE: send email to
>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a
>>> subject of "unsubscribe"
>>
>
> --
> ==================================
> Adrian Fish
> Software Engineer
> Centre for e-Science
> Bowland Tower South C Floor
> Lancaster University
> Lancaster
> LA1 4YW
> email:a.fish at lancaster.ac.uk <mailto:a.fish at lancaster.ac.uk>
>
> http://confluence.sakaiproject.org/display/YAFT/Yaft
> http://confluence.sakaiproject.org/display/CLOG/Home
> http://confluence.sakaiproject.org/display/BBB/Home
>
>
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> <mailto:sakai-dev at collab.sakaiproject.org>
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to
> sakai-dev-unsubscribe at collab.sakaiproject.org
> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a
> subject of "unsubscribe"
>
>
>
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20110316/d83d3416/attachment.html
More information about the sakai-dev
mailing list