[Building Sakai] SAK-800
Adrian Fish
a.fish at lancaster.ac.uk
Thu Mar 17 04:00:04 PDT 2011
Initially, all is well with unzipping although I'm starting to realise
what I need to do to get stuff like Wimba Create packages uploaded.
Wimba Create produces a little website with all the pages containing
relative links to each other. So, not only do I need to unzip the
content, I probably need to write a base tag into every page so the
links work.
Has this been done already, to anybody's knowledge?
Cheers,
Adrian.
On 17/03/2011 03:08, Sam Ottenhoff wrote:
> If you are able to test the patch on a local install, any feedback
> would be appreciated. With just the unzip functionality enabled on a
> dev install, I have not run into any issues with the new
> functionality. I have tested with very large ZIPs, zero-byte files,
> empty folders, etc.
>
> Please leave any testing feedback in JIRA: KNL-273.
>
> --Sam
>
> On 3/16/2011 7:33 PM, Nicolas Lupien wrote:
>> I was looking at the same thing as you today. As a user, i can't wait
>> to see this in the main release.
>>
>> We've just made the switch from Moodle to Sakai and this seems to be
>> the hardest part.
>>
>> Nicolas Lupien
>>
>>
>>
>> On Wed, Mar 16, 2011 at 6:20 PM, Adrian Fish <a.fish at lancaster.ac.uk
>> <mailto:a.fish at lancaster.ac.uk>> wrote:
>>
>> Hi Bryan,
>>
>> I'll attempt to apply the patch to my kernel and see how I get
>> on. Thanks for the info, I should have read the JIRA more closely ...
>>
>> Cheers,
>> Adrian.
>>
>>
>> On 16/03/2011 21:33, Bryan Holladay wrote:
>>> There has been some recent conversation lately about this...
>>> KNL-273 is an extension of SAK-800... below are some of the
>>> threads:
>>>
>>>
>>> ------------------------------------------
>>>
>>>
>>> I went ahead and altered the patch for KNL-273. I've added some
>>> new properties that will limit the size of a zip file that you
>>> can expand. This is configurable and I feel that this is a good
>>> low tech approach to preventing a DOS from large zip files.
>>> Also, you can choose to only have the ability to "Expand a zip"
>>> option and turn off the "Compress to zip" option (or visa versa)
>>>
>>> the new sakai.properties are:
>>>
>>> //used to limit the size of a zip file it will try to expand
>>> resources.zip.expand.max=1000 (default)
>>>
>>> //used to either turn on/off just expanding zips
>>> resources.zip.enable.expand=false (default)
>>>
>>> //used to either turn on/off just compressing zips
>>> resources.zip.enable.compress=false (default)
>>>
>>> //also the prop to turn on both or default to the other props
>>> resources.zip.enable=false (default)
>>>
>>>
>>>
>>> The new patch is attached to the jira. There is one for
>>> kernel1.1.x and trunk
>>>
>>> -Bryan
>>>
>>>
>>> On Mar 16, 2011, at 3:37 PM, Sam Ottenhoff wrote:
>>>
>>>> last message in my inbox on this thread
>>>>
>>>> -------- Original Message --------
>>>> Subject: Re: [Building Sakai] [maint] alternative to webdav in
>>>> sakai
>>>> Date: Thu, 10 Mar 2011 17:21:42 +0000
>>>> From: Adam Marshall <adam.marshall at oucs.ox.ac.uk>
>>>> <mailto:adam.marshall at oucs.ox.ac.uk>
>>>> To: Nicola Monat-Jacobs <nicola at longsight.com>
>>>> <mailto:nicola at longsight.com>, John Bush <john.bush at rsmart.com>
>>>> <mailto:john.bush at rsmart.com>
>>>> CC: sakai-dev at collab.sakaiproject.org
>>>> <mailto:sakai-dev at collab.sakaiproject.org>
>>>> <sakai-dev at collab.sakaiproject.org>
>>>> <mailto:sakai-dev at collab.sakaiproject.org>,
>>>> <mt at collab.sakaiproject.org>
>>>> <mailto:mt at collab.sakaiproject.org>
>>>> <mt at collab.sakaiproject.org> <mailto:mt at collab.sakaiproject.org>
>>>>
>>>>
>>>>
>>>> I think an unzip (with no zip) would still be very useful ineed.
>>>> adam
>>>> *From:*sakai-dev-bounces at collab.sakaiproject.org
>>>> <mailto:sakai-dev-bounces at collab.sakaiproject.org> [mailto:sakai-dev-bounces at collab.sakaiproject.org]
>>>> *On Behalf Of *Nicola Monat-Jacobs
>>>> *Sent:* 10 March 2011 16:54
>>>> *To:* John Bush
>>>> *Cc:* <mt at collab.sakaiproject.org>
>>>> <mailto:mt at collab.sakaiproject.org>;
>>>> sakai-dev at collab.sakaiproject.org
>>>> <mailto:sakai-dev at collab.sakaiproject.org>
>>>> *Subject:* Re: [Building Sakai] [maint] alternative to webdav
>>>> in sakai
>>>> As far as I can tell from reading the JIRAs, the issues with
>>>> SAK-800 were on the zipping side, rather than the unzipping
>>>> side, right? Unless there was an issue with the 'upload and
>>>> unzip a zip into resources' functionality that I'm not catching
>>>> in the various JIRAs, what about just implemented that part of
>>>> the functionality? It seems a shame to reject a feature because
>>>> it's partner misbehaves.
>>>> Bulk uploading was the original use case that you were looking
>>>> at, right John?
>>>> - Nicola
>>>> On Mar 9, 2011, at 3:51 PM, John Bush wrote:
>>>>
>>>>
>>>> I spent a little time looking for an alternative today that
>>>> doesn't involve webdav or SAK-800 (you guys aren't making me
>>>> feel fuzzy about that idea). I can't find a flash widget that
>>>> will let you attach folders, and I think that is key, for
>>>> people who are uploading say a bunch of html content that is
>>>> all linked together.
>>>>
>>>> I've settled on jupload which is an applet:
>>>> http://jupload.sourceforge.net/
>>>>
>>>> It does drag and drop, supports attaching folders, and can do a
>>>> http PUT, which I'm not sure will be valuable or not.
>>>>
>>>> I'm going to spend some time prototyping something that would
>>>> say plug this into the upload page in resources. We'll see
>>>> where this goes...
>>>>
>>>> On Wed, Mar 9, 2011 at 2:27 PM, Matthew Jones
>>>> <jonespm at umich.edu <mailto:jonespm at umich.edu>> wrote:
>>>> I agree with these comments, I don't think you're being
>>>> paranoid. There were too many easy to reproduce edge cases that
>>>> really could start up a nearly denial of service back with the
>>>> last patch that was not tested. Only the "happy path" of a
>>>> regular zip file with a few files in it was tested. If you
>>>> loaded up a zip file with thousands (hundreds of thousands?) of
>>>> zero byte files then the performance was much different. Also I
>>>> remember being able to generate some file names that generated
>>>> exceptions that back then were silently logged.
>>>> Having this process that has a good chance of taking longer
>>>> than a few second in the request thread is a long standing
>>>> problem of Sakai. Something like this (a task that is no other
>>>> workflow depends on it's completion and is potentially long
>>>> running) really needs to be backgrounded or moved off to some
>>>> type of job processing server. And when it's processed,
>>>> ideally, the user notified that their (long running) job is
>>>> complete. No mechanism currently exists for this in Sakai. :(
>>>> It's also unfortunate that tomcat or java has no (seemingly
>>>> easy?) way to set any type of thread limits or timeouts like
>>>> php (set_time_limit) does with execution time. This might at
>>>> least make administrators feel a little safer. I guess you can
>>>> setup an ExecutorService and run your function in that? Still,
>>>> gets into some pretty complex stuff for a "simple" feature.
>>>> I agree that this feature would be really awesome for a user
>>>> but seems like a scary addiition from an operations perspective.
>>>>
>>>> On Wed, Mar 9, 2011 at 4:07 PM, Noah Botimer <botimer at umich.edu
>>>> <mailto:botimer at umich.edu>> wrote:
>>>> Sure. When the original patch was merged, it came with a number
>>>> of quality and completeness problems.
>>>>
>>>> https://jira.sakaiproject.org/browse/SAK-16036 and
>>>> https://jira.sakaiproject.org/browse/KNL-155 capture the work
>>>> that was done in the closing moments of the 2.6 release to
>>>> disable the feature. The problems were discovered late and the
>>>> fix was an ugly combination of properties and commented code.
>>>>
>>>> (I refer to this pretty broken code lying latent in trunk
>>>> without a quality plan or much notice as being "parachuted in".)
>>>>
>>>> Perhaps the work done for KNL-273 addresses most of the
>>>> problems, but I am not comfortable without seeing a standalone,
>>>> cohesive write-up of the feature and its behavior. Tracking
>>>> this across a handful of SAK/KNL tickets, some emails, and
>>>> fifty-some JIRA comments just doesn't draw a clear picture.
>>>>
>>>> Specific technical questions are around the library used,
>>>> performance, request-thread processing, mime-type settings,
>>>> content scanning, and quota ramifications. We may or may not
>>>> have a clear implementation plan that addresses these. I am
>>>> especially concerned about the processing in the request thread
>>>> and the impact on the database pool.
>>>>
>>>> Maybe I'm paranoid or maybe these have all been sufficiently
>>>> addressed. But I would strongly urge those interested to do a
>>>> complete and accurate write-up outside of the smattering of
>>>> JIRA tickets, so we can make a conscious community decision.
>>>>
>>>> By the way, many thanks to Bryan and whomever else for working
>>>> on a real user need.
>>>>
>>>> Thanks,
>>>> -Noah
>>>>
>>>> On Mar 9, 2011, at 3:16 PM, Sam Ottenhoff wrote:
>>>>
>>>> > I know nothing about the previous issues or the mitigation
>>>> that was
>>>> > done. SAK-800 comments don't seem to reflect any local
>>>> catastrophes.
>>>> > Can you give us some background?
>>>> >
>>>> > --Sam
>>>> >
>>>> > On 3/9/2011 3:11 PM, Noah Botimer wrote:
>>>> >> With all due respect to the work done and the spoken need, I
>>>> think there are still significant concerns with this approach.
>>>> It solves a class of problem and introduces another.
>>>> >>
>>>> >> I am not familiar with the extent of the work done recently,
>>>> but I am very leery of SAK-800. I will be disappointed if it
>>>> gets parachuted in again. Even our mitigation was sloppy last time.
>>>> >>
>>>> >> This absolutely requires review and discussion on list and
>>>> the TCC should form an opinion of the best path. This is not
>>>> simple bug fixing or minor feature enhancement, given the
>>>> technical and policy issues arising.
>>>> >>
>>>> >> None of this is to discourage movement forward, but it is a
>>>> humble request that special care be taken in an area that has
>>>> really stung us before.
>>>> >>
>>>> >> Thanks,
>>>> >> -Noah
>>>> >>
>>>> >> On Mar 9, 2011, at 12:25 PM, Bryan Holladay wrote:
>>>> >>
>>>> >>> Thanks David... Anyone who wants it can take it. I
>>>> re-assigned it to the KERNEL TEAM in mid November.
>>>> >>>
>>>> >>> -Bryan
>>>> >>>
>>>> >>> On Mar 9, 2011, at 12:17 PM, DAVID ROLDAN MARTINEZ wrote:
>>>> >>>
>>>> >>>> Bryan,
>>>> >>>>
>>>> >>>> I offer myself to do this (or provide you help if you
>>>> prefer to manage this yourself) and leave hardcoded messages
>>>> out of kernel so that they will be handled at resources tool.
>>>> But, as I'm not a member of the Kernel-team, I would like to
>>>> get their feedback before to start working.
>>>> >>>>
>>>> >>>> Cheers,
>>>> >>>> David
>>>> >>>> ________________________________________
>>>> >>>> De: sakai-dev-bounces at collab.sakaiproject.org
>>>> <mailto:sakai-dev-bounces at collab.sakaiproject.org> [sakai-dev-bounces at collab.sakaiproject.org
>>>> <mailto:sakai-dev-bounces at collab.sakaiproject.org>] En nombre
>>>> de Bryan Holladay [holladay at longsight.com
>>>> <mailto:holladay at longsight.com>]
>>>> >>>> Enviado el: miércoles, 09 de marzo de 2011 17:51
>>>> >>>> Para: Adam Marshall
>>>> >>>> CC: sakai-dev at collab.sakaiproject.org
>>>> <mailto:sakai-dev at collab.sakaiproject.org>
>>>> >>>> Asunto: Re: [Building Sakai] alternative to webdav in sakai
>>>> >>>>
>>>> >>>> I've fixed this issue in
>>>> https://jira.sakaiproject.org/browse/KNL-273, along with other
>>>> issues with the unzip facility.
>>>> >>>>
>>>> >>>> The only reason it hasn't been put into trunk is a best
>>>> code practices issue I wasn't able to figure out. It's noted
>>>> in the comments and is waiting for someone with a better
>>>> understanding of the kernel code to look at it.
>>>> >>>>
>>>> >>>> Other than that, it's a good patch and works fine and
>>>> would be safe to run.
>>>> >>>>
>>>> >>>> Thanks,
>>>> >>>> Bryan
>>>> >>>>
>>>> >>>>
>>>> >>>> On Mar 9, 2011, at 11:41 AM, Adam Marshall wrote:
>>>> >>>>
>>>> >>>> We use the unzip facility – it works OK for extraction but
>>>> is not good for creating a zip file. You cannot zip from the
>>>> root folder!
>>>> >>>>
>>>> >>>> From: sakai-dev-bounces at collab.sakaiproject.org
>>>> <mailto:sakai-dev-bounces at collab.sakaiproject.org><mailto:sakai-dev-bounces at collab.sakaiproject.org
>>>> <mailto:sakai-dev-bounces at collab.sakaiproject.org>>
>>>> [mailto:sakai-dev-bounces at collab.sakaiproject.org
>>>> <mailto:sakai-dev-bounces at collab.sakaiproject.org>] On Behalf
>>>> Of Sam Ottenhoff
>>>> >>>> Sent: 09 March 2011 16:35
>>>> >>>> To: sakai-dev at collab.sakaiproject.org
>>>> <mailto:sakai-dev at collab.sakaiproject.org><mailto:sakai-dev at collab.sakaiproject.org
>>>> <mailto:sakai-dev at collab.sakaiproject.org>>
>>>> >>>> Subject: Re: [Building Sakai] alternative to webdav in sakai
>>>> >>>>
>>>> >>>> https://jira.sakaiproject.org/browse/SAK-800
>>>> >>>>
>>>> >>>> SAK-800 and KNL-273 would allow users to upload a
>>>> compressed archive and have it unrolled in Resources.
>>>> >>>>
>>>> >>>> --Sam
>>>> >>>>
>>>> >>>> On 3/9/2011 11:19 AM, John Bush wrote:
>>>> >>>> Over the years we've had continued problems with webdav.
>>>> The varying behaviors of different clients, the special
>>>> authentication that doesn't play nice with SSO, and numerous
>>>> other types of issues. I'm beginning to think about
>>>> alternatives that would allow for big bulk uploads that are
>>>> more reliable. Thinking about some type of browser plugin,
>>>> flash or otherwise. Has anyone else done some research into
>>>> this already?
>>>> >>>>
>>>> >>>> --
>>>> >>>> John Bush
>>>> >>>> 602-490-0470
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> _______________________________________________
>>>> >>>>
>>>> >>>> sakai-dev mailing list
>>>> >>>>
>>>> >>>> sakai-dev at collab.sakaiproject.org
>>>> <mailto:sakai-dev at collab.sakaiproject.org><mailto:sakai-dev at collab.sakaiproject.org
>>>> <mailto:sakai-dev at collab.sakaiproject.org>>
>>>> >>>>
>>>> >>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> TO UNSUBSCRIBE: send email to
>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org><mailto:sakai-dev-unsubscribe at collab.sakaiproject.org
>>>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org>> with a
>>>> subject of "unsubscribe"
>>>> >>>>
>>>> >>>> _______________________________________________
>>>> >>>> sakai-dev mailing list
>>>> >>>> sakai-dev at collab.sakaiproject.org
>>>> <mailto:sakai-dev at collab.sakaiproject.org><mailto:sakai-dev at collab.sakaiproject.org
>>>> <mailto:sakai-dev at collab.sakaiproject.org>>
>>>> >>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>> >>>>
>>>> >>>> TO UNSUBSCRIBE: send email to
>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org><mailto:sakai-dev-unsubscribe at collab.sakaiproject.org
>>>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org>> with a
>>>> subject of "unsubscribe"
>>>> >>>>
>>>> >>> _______________________________________________
>>>> >>> sakai-dev mailing list
>>>> >>> sakai-dev at collab.sakaiproject.org
>>>> <mailto:sakai-dev at collab.sakaiproject.org>
>>>> >>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>> >>>
>>>> >>> TO UNSUBSCRIBE: send email to
>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a
>>>> subject of "unsubscribe"
>>>> >>>
>>>> >>>
>>>> >> _______________________________________________
>>>> >> sakai-dev mailing list
>>>> >> sakai-dev at collab.sakaiproject.org
>>>> <mailto:sakai-dev at collab.sakaiproject.org>
>>>> >> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>> >>
>>>> >> TO UNSUBSCRIBE: send email to
>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a
>>>> subject of "unsubscribe"
>>>> > _______________________________________________
>>>> > sakai-dev mailing list
>>>> > sakai-dev at collab.sakaiproject.org
>>>> <mailto:sakai-dev at collab.sakaiproject.org>
>>>> > http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>> >
>>>> > TO UNSUBSCRIBE: send email to
>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a
>>>> subject of "unsubscribe"
>>>> >
>>>> >
>>>>
>>>> _______________________________________________
>>>> sakai-dev mailing list
>>>> sakai-dev at collab.sakaiproject.org
>>>> <mailto:sakai-dev at collab.sakaiproject.org>
>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>
>>>> TO UNSUBSCRIBE: send email to
>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a
>>>> subject of "unsubscribe"
>>>>
>>>>
>>>> _______________________________________________
>>>> mt mailing list
>>>> mt at collab.sakaiproject.org <mailto:mt at collab.sakaiproject.org>
>>>> http://collab.sakaiproject.org/mailman/listinfo/mt
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> John Bush
>>>> 602-490-0470
>>>> _______________________________________________
>>>> mt mailing list
>>>> mt at collab.sakaiproject.org <mailto:mt at collab.sakaiproject.org>
>>>> http://collab.sakaiproject.org/mailman/listinfo/mt
>>>> <Attached Message Part.txt>
>>>
>>> _______________________________________________
>>> sakai-dev mailing list
>>> sakai-dev at collab.sakaiproject.org
>>> <mailto:sakai-dev at collab.sakaiproject.org>
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>
>>> TO UNSUBSCRIBE: send email to
>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a
>>> subject of "unsubscribe"
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mar 16, 2011, at 5:25 PM, Adrian Fish wrote:
>>>
>>>> Is anybody actively looking at SAK-800? If not, I will as this
>>>> would be
>>>> incredibly useful for LessonBuilder as you'll be able to upload
>>>> zips
>>>> generated by tools such as Camtasia, Articulate and Wimba
>>>> Create and
>>>> then just link to them. We've also had the usual WebDAV grief ...
>>>>
>>>> Cheers,
>>>> Adrian.
>>>>
>>>> --
>>>> ==================================
>>>> Adrian Fish
>>>> Software Engineer
>>>> Centre for e-Science
>>>> Bowland Tower South C Floor
>>>> Lancaster University
>>>> Lancaster
>>>> LA1 4YW
>>>> email: a.fish at lancaster.ac.uk <mailto:a.fish at lancaster.ac.uk>
>>>>
>>>> http://confluence.sakaiproject.org/display/YAFT/Yaft
>>>> http://confluence.sakaiproject.org/display/CLOG/Home
>>>> http://confluence.sakaiproject.org/display/BBB/Home
>>>>
>>>> _______________________________________________
>>>> sakai-dev mailing list
>>>> sakai-dev at collab.sakaiproject.org
>>>> <mailto:sakai-dev at collab.sakaiproject.org>
>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>
>>>> TO UNSUBSCRIBE: send email to
>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a
>>>> subject of "unsubscribe"
>>>
>>
>> --
>> ==================================
>> Adrian Fish
>> Software Engineer
>> Centre for e-Science
>> Bowland Tower South C Floor
>> Lancaster University
>> Lancaster
>> LA1 4YW
>> email:a.fish at lancaster.ac.uk <mailto:a.fish at lancaster.ac.uk>
>>
>> http://confluence.sakaiproject.org/display/YAFT/Yaft
>> http://confluence.sakaiproject.org/display/CLOG/Home
>> http://confluence.sakaiproject.org/display/BBB/Home
>>
>>
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> <mailto:sakai-dev at collab.sakaiproject.org>
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to
>> sakai-dev-unsubscribe at collab.sakaiproject.org
>> <mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a
>> subject of "unsubscribe"
>>
>>
>>
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email tosakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
>
>
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
--
==================================
Adrian Fish
Software Engineer
Centre for e-Science
Bowland Tower South C Floor
Lancaster University
Lancaster
LA1 4YW
email: a.fish at lancaster.ac.uk
http://confluence.sakaiproject.org/display/YAFT/Yaft
http://confluence.sakaiproject.org/display/CLOG/Home
http://confluence.sakaiproject.org/display/BBB/Home
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20110317/4a48d502/attachment.html
More information about the sakai-dev
mailing list