[Building Sakai] IMS Common Cartridge import (Also IMS CP import)

[Adam Marshall]

I'm a little hazy on the details but it turns out that there is a Sakai property which allows the file upload limit via Site Info to be set differently to resources. 

adam

| -----Original Message-----
| From: Zhen Qian [mailto:zqian at umich.edu]
| Sent: 06 July 2010 15:49
| To: Steve Swinsburg
| Cc: Adam Marshall; sakai-dev at collab.sakaiproject.org
| Subject: Re: [Building Sakai] IMS Common Cartridge import (Also IMS CP
| import)
| 
| 
| The "Import from Archive File" feature inside Site Info do respect the
| setting of "content.upload.max" configuration variable. However, there
| is a
| recent JIRA fix for SAK-18722 "Site Info generates error when
| content.upload.max > 2G", which helps with big file (>2G) upload.
| 
| The file content is read into memory, not using streaming. Maybe
| Stephen
| Marquard would have some usage data from UCT for large file imports.
| 
| Thanks,
| 
| - Zhen
| 
| On Tue, 6 Jul 2010 09:03:31 +1000, Steve Swinsburg
| <steve.swinsburg at gmail.com> wrote:
| > It is my understanding that the max upload size is applied in the
| > RequestFilter via:
| >
| >
| org.apache.commons.fileupload.servlet.ServletFileUpload.setSizeMax(uplo
| adMax);
| >
| > There are ways to get around this on a per-webapp basis by supplying
| some
| > init-params to the RequestFilter config in the webapp's web.xml (see
| > Javadocs if interested) but looking at the sakai-site-manage-tool it
| is
| > just a standard RequestFilter configuration so the value from
| > sakai.properties should be being used. That said, Site Info
| aggregates a
| > number of webapps into one so the one that is used for the IMS CP
| import
| > might be in a different context.
| >
| > OIf course you could just try to upload a package larger than the
| size
| you
| > have configured and see what happens, then you'll be sure :)
| >
| > cheers,
| > Steve
| >
| > On 06/07/2010, at 1:34 AM, Adam Marshall wrote:
| >
| >> we think the limit is tested further up the chain than CHS but
| aren't
| >> sure.
| >>
| >> adam
| >>
| >> | -----Original Message-----
| >> | From: Mark Norton [mailto:markjnorton at earthlink.net]
| >> | Sent: 05 July 2010 15:18
| >> | To: Adam Marshall
| >> | Cc: sakai-dev at collab.sakaiproject.org
| >> | Subject: Re: [Building Sakai] IMS Common Cartridge import (Also
| IMS CP
| >> | import)
| >> |
| >> | I'm not quite sure what you are referring to when you say "site
| info".
| >> | I'm also not sure where the limits are tested.  Obviously CHS
| tests
| the
| >> | limits, but the actual testing may be in deeper, as part of the
| file
| >> | upload method used.  I was using RSF and the Apache commons upload
| >> | package.  Generally HTTP doesn't care
| >> |
| >> | - Mark
| >> |
| >> | Adam Marshall wrote:
| >> | > thanks Mark.
| >> | >
| >> | > Can I just confirm that your understanding is that the upload
| limit
| >> | applies to site info as well as resources?
| >> | >
| >> | > adam
| >> | >
| >> | > | -----Original Message-----
| >> | > | From: Mark Norton [mailto:markjnorton at earthlink.net]
| >> | > | Sent: 05 July 2010 14:20
| >> | > | To: Adam Marshall
| >> | > | Cc: sakai-dev at collab.sakaiproject.org
| >> | > | Subject: Re: [Building Sakai] IMS Common Cartridge import
| (Also
| IMS
| >> | CP
| >> | > | import)
| >> | > |
| >> | > | The work was done on software not posted to Sakai svn, so no
| patch.
| >> | > | Sorry.  On the server side, when you have the IO stream to the
| >> | incoming
| >> | > | file, read using buffered IO stream, write to a temp file.
| Temp
| >> | file
| >> | > | can then be used as needed.
| >> | > |
| >> | > | - Mark
| >> | > |
| >> | > | Adam Marshall wrote:
| >> | > | > do you have a patch for this?
| >> | > | >
| >> | > | > adam
| >> | > | >
| >> | > | > | -----Original Message-----
| >> | > | > | From: Mark Norton [mailto:markjnorton at earthlink.net]
| >> | > | > | Sent: 05 July 2010 13:57
| >> | > | > | To: Adam Marshall
| >> | > | > | Cc: sakai-dev at collab.sakaiproject.org
| >> | > | > | Subject: Re: [Building Sakai] IMS Common Cartridge import
| (Also
| >> | IMS
| >> | > | CP
| >> | > | > | import)
| >> | > | > |
| >> | > | > | I ran into file upload memory issues in some of my past
| >> | projects.
| >> | > | The
| >> | > | > | best workaround around I came up with was to buffer the
| I/O to
| >> | a
| >> | > | temp
| >> | > | > | file.  Not always a fast solution, it does seem to work
| even
| on
| >> | > | large
| >> | > | > | file sizes.
| >> | > | > |
| >> | > | > | - Mark Norton
| >> | > | > |
| >> | > | > | Adam Marshall wrote:
| >> | > | > | > Dear Community
| >> | > | > | >
| >> | > | > | > We've just implemented a facility to allow IMS Content
| >> | Packages
| >> | > | to be
| >> | > | > | uploaded via the "Import from file" in Site Info.
| >> | > | > | >
| >> | > | > | > I have some questions:
| >> | > | > | >
| >> | > | > | > 1/ does site info obey the "60 meg upload limit" (ie,
| the
| max
| >> | > | upload
| >> | > | > | limit set in properties)?  For IMS CC or CP courses with
| video
| >> | this
| >> | > | > | will be highly restrictive.
| >> | > | > | >
| >> | > | > | > 2/ We think that file uploads are read into memory so,
| if we
| >> | > | > | increased the upload limit to (say) 200MB this would
| possibly
| >> | cause
| >> | > | > | performance problems. Is there a way to get the uploads to
| load
| >> | to
| >> | > | disc
| >> | > | > | instead? Does anybody know how to do this?
| >> | > | > | >
| >> | > | > | > Is there any other useful info that could be passed
| along to
| >> | us?
| >> | > | > | >
| >> | > | > | > adam
| >> | > | > | >
| >> | > | > | > PS Our IMS CP upload facility works quite well! We will
| make
| >> | > | > | available to commuinity once we've tidied up the last few
| >> | remaining
| >> | > | > | issues.
| >> | > | > | > _______________________________________________
| >> | > | > | > sakai-dev mailing list
| >> | > | > | > sakai-dev at collab.sakaiproject.org
| >> | > | > | > http://collab.sakaiproject.org/mailman/listinfo/sakai-
| dev
| >> | > | > | >
| >> | > | > | > TO UNSUBSCRIBE: send email to sakai-dev-
| >> | > | > | unsubscribe at collab.sakaiproject.org with a subject of
| >> | "unsubscribe"
| >> | > | > | >
| >> | > | > | >
| >> | > | > | >
| >> | > | > | >
| >> | > | > | > =======
| >> | > | > | > Email scanned by PC Tools - No viruses or spyware found.
| >> | > | > | > (Email Guard: 7.0.0.18, Virus/Spyware Database: 6.15360)
| >> | > | > | > http://www.pctools.com/
| >> | > | > | > =======
| >> | > | > | >
| >> | > | > | >
| >> | > | > |
| >> | > | > |
| >> | > | > |
| >> | > | > |
| >> | > | > |
| >> | > | > | =======
| >> | > | > | Email scanned by PC Tools - No viruses or spyware found.
| >> | > | > | (Email Guard: 7.0.0.18, Virus/Spyware Database: 6.15360)
| >> | > | > | http://www.pctools.com/
| >> | > | > | =======
| >> | > | >
| >> | > | >
| >> | > | >
| >> | > | >
| >> | > | > =======
| >> | > | > Email scanned by PC Tools - No viruses or spyware found.
| >> | > | > (Email Guard: 7.0.0.18, Virus/Spyware Database: 6.15360)
| >> | > | > http://www.pctools.com/
| >> | > | > =======
| >> | > | >
| >> | > | >
| >> | > |
| >> | > |
| >> | > |
| >> | > |
| >> | > |
| >> | > | =======
| >> | > | Email scanned by PC Tools - No viruses or spyware found.
| >> | > | (Email Guard: 7.0.0.18, Virus/Spyware Database: 6.15360)
| >> | > | http://www.pctools.com/
| >> | > | =======
| >> | >
| >> | >
| >> | >
| >> | >
| >> | > =======
| >> | > Email scanned by PC Tools - No viruses or spyware found.
| >> | > (Email Guard: 7.0.0.18, Virus/Spyware Database: 6.15360)
| >> | > http://www.pctools.com/
| >> | > =======
| >> | >
| >> | >
| >> |
| >> |
| >> |
| >> |
| >> |
| >> | =======
| >> | Email scanned by PC Tools - No viruses or spyware found.
| >> | (Email Guard: 7.0.0.18, Virus/Spyware Database: 6.15360)
| >> | http://www.pctools.com/
| >> | =======
| >> _______________________________________________
| >> sakai-dev mailing list
| >> sakai-dev at collab.sakaiproject.org
| >> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
| >>
| >> TO UNSUBSCRIBE: send email to
| >> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
| >> "unsubscribe"