[Building Sakai] IMS Common Cartridge import (Also IMS CP import)

Tue Jul 6 07:48:35 PDT 2010

The "Import from Archive File" feature inside Site Info do respect the
setting of "content.upload.max" configuration variable. However, there is a
recent JIRA fix for SAK-18722 "Site Info generates error when
content.upload.max > 2G", which helps with big file (>2G) upload. 

The file content is read into memory, not using streaming. Maybe Stephen
Marquard would have some usage data from UCT for large file imports.

Thanks,

- Zhen

On Tue, 6 Jul 2010 09:03:31 +1000, Steve Swinsburg
<steve.swinsburg at gmail.com> wrote:
> It is my understanding that the max upload size is applied in the
> RequestFilter via:
> 
>
org.apache.commons.fileupload.servlet.ServletFileUpload.setSizeMax(uploadMax);
> 
> There are ways to get around this on a per-webapp basis by supplying some
> init-params to the RequestFilter config in the webapp's web.xml (see
> Javadocs if interested) but looking at the sakai-site-manage-tool it is
> just a standard RequestFilter configuration so the value from
> sakai.properties should be being used. That said, Site Info aggregates a
> number of webapps into one so the one that is used for the IMS CP import
> might be in a different context.
> 
> OIf course you could just try to upload a package larger than the size
you
> have configured and see what happens, then you'll be sure :)
> 
> cheers,
> Steve
> 
> On 06/07/2010, at 1:34 AM, Adam Marshall wrote:
> 
>> we think the limit is tested further up the chain than CHS but aren't
>> sure.
>> 
>> adam
>> 
>> | -----Original Message-----
>> | From: Mark Norton [mailto:markjnorton at earthlink.net]
>> | Sent: 05 July 2010 15:18
>> | To: Adam Marshall
>> | Cc: sakai-dev at collab.sakaiproject.org
>> | Subject: Re: [Building Sakai] IMS Common Cartridge import (Also IMS CP
>> | import)
>> | 
>> | I'm not quite sure what you are referring to when you say "site info".
>> | I'm also not sure where the limits are tested.  Obviously CHS tests
the
>> | limits, but the actual testing may be in deeper, as part of the file
>> | upload method used.  I was using RSF and the Apache commons upload
>> | package.  Generally HTTP doesn't care
>> | 
>> | - Mark
>> | 
>> | Adam Marshall wrote:
>> | > thanks Mark.
>> | >
>> | > Can I just confirm that your understanding is that the upload limit
>> | applies to site info as well as resources?
>> | >
>> | > adam
>> | >
>> | > | -----Original Message-----
>> | > | From: Mark Norton [mailto:markjnorton at earthlink.net]
>> | > | Sent: 05 July 2010 14:20
>> | > | To: Adam Marshall
>> | > | Cc: sakai-dev at collab.sakaiproject.org
>> | > | Subject: Re: [Building Sakai] IMS Common Cartridge import (Also
IMS
>> | CP
>> | > | import)
>> | > |
>> | > | The work was done on software not posted to Sakai svn, so no
patch.
>> | > | Sorry.  On the server side, when you have the IO stream to the
>> | incoming
>> | > | file, read using buffered IO stream, write to a temp file.  Temp
>> | file
>> | > | can then be used as needed.
>> | > |
>> | > | - Mark
>> | > |
>> | > | Adam Marshall wrote:
>> | > | > do you have a patch for this?
>> | > | >
>> | > | > adam
>> | > | >
>> | > | > | -----Original Message-----
>> | > | > | From: Mark Norton [mailto:markjnorton at earthlink.net]
>> | > | > | Sent: 05 July 2010 13:57
>> | > | > | To: Adam Marshall
>> | > | > | Cc: sakai-dev at collab.sakaiproject.org
>> | > | > | Subject: Re: [Building Sakai] IMS Common Cartridge import
(Also
>> | IMS
>> | > | CP
>> | > | > | import)
>> | > | > |
>> | > | > | I ran into file upload memory issues in some of my past
>> | projects.
>> | > | The
>> | > | > | best workaround around I came up with was to buffer the I/O to
>> | a
>> | > | temp
>> | > | > | file.  Not always a fast solution, it does seem to work even
on
>> | > | large
>> | > | > | file sizes.
>> | > | > |
>> | > | > | - Mark Norton
>> | > | > |
>> | > | > | Adam Marshall wrote:
>> | > | > | > Dear Community
>> | > | > | >
>> | > | > | > We've just implemented a facility to allow IMS Content
>> | Packages
>> | > | to be
>> | > | > | uploaded via the "Import from file" in Site Info.
>> | > | > | >
>> | > | > | > I have some questions:
>> | > | > | >
>> | > | > | > 1/ does site info obey the "60 meg upload limit" (ie, the
max
>> | > | upload
>> | > | > | limit set in properties)?  For IMS CC or CP courses with video
>> | this
>> | > | > | will be highly restrictive.
>> | > | > | >
>> | > | > | > 2/ We think that file uploads are read into memory so, if we
>> | > | > | increased the upload limit to (say) 200MB this would possibly
>> | cause
>> | > | > | performance problems. Is there a way to get the uploads to
load
>> | to
>> | > | disc
>> | > | > | instead? Does anybody know how to do this?
>> | > | > | >
>> | > | > | > Is there any other useful info that could be passed along to
>> | us?
>> | > | > | >
>> | > | > | > adam
>> | > | > | >
>> | > | > | > PS Our IMS CP upload facility works quite well! We will make
>> | > | > | available to commuinity once we've tidied up the last few
>> | remaining
>> | > | > | issues.
>> | > | > | > _______________________________________________
>> | > | > | > sakai-dev mailing list
>> | > | > | > sakai-dev at collab.sakaiproject.org
>> | > | > | > http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>> | > | > | >
>> | > | > | > TO UNSUBSCRIBE: send email to sakai-dev-
>> | > | > | unsubscribe at collab.sakaiproject.org with a subject of
>> | "unsubscribe"
>> | > | > | >
>> | > | > | >
>> | > | > | >
>> | > | > | >
>> | > | > | > =======
>> | > | > | > Email scanned by PC Tools - No viruses or spyware found.
>> | > | > | > (Email Guard: 7.0.0.18, Virus/Spyware Database: 6.15360)
>> | > | > | > http://www.pctools.com/
>> | > | > | > =======
>> | > | > | >
>> | > | > | >
>> | > | > |
>> | > | > |
>> | > | > |
>> | > | > |
>> | > | > |
>> | > | > | =======
>> | > | > | Email scanned by PC Tools - No viruses or spyware found.
>> | > | > | (Email Guard: 7.0.0.18, Virus/Spyware Database: 6.15360)
>> | > | > | http://www.pctools.com/
>> | > | > | =======
>> | > | >
>> | > | >
>> | > | >
>> | > | >
>> | > | > =======
>> | > | > Email scanned by PC Tools - No viruses or spyware found.
>> | > | > (Email Guard: 7.0.0.18, Virus/Spyware Database: 6.15360)
>> | > | > http://www.pctools.com/
>> | > | > =======
>> | > | >
>> | > | >
>> | > |
>> | > |
>> | > |
>> | > |
>> | > |
>> | > | =======
>> | > | Email scanned by PC Tools - No viruses or spyware found.
>> | > | (Email Guard: 7.0.0.18, Virus/Spyware Database: 6.15360)
>> | > | http://www.pctools.com/
>> | > | =======
>> | >
>> | >
>> | >
>> | >
>> | > =======
>> | > Email scanned by PC Tools - No viruses or spyware found.
>> | > (Email Guard: 7.0.0.18, Virus/Spyware Database: 6.15360)
>> | > http://www.pctools.com/
>> | > =======
>> | >
>> | >
>> | 
>> | 
>> | 
>> | 
>> | 
>> | =======
>> | Email scanned by PC Tools - No viruses or spyware found.
>> | (Email Guard: 7.0.0.18, Virus/Spyware Database: 6.15360)
>> | http://www.pctools.com/
>> | =======
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>> 
>> TO UNSUBSCRIBE: send email to
>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>> "unsubscribe"