[Building Sakai] Unable to upload large file/resource, but still smaller than content.upload.max?

[Steve Swinsburg]

Hi Will,

Correct, if using filesystem storage, you'll have a record in the db
pointing to the resource on the filesystem. I'm not across running up Sakai
on Windows so will leave the settings to someone else. Can you run some
monitoring on your Sakai JVM to see what happens when you upload a file that
causes the issue?

cheers,
Steve

On Thu, Nov 19, 2009 at 5:40 AM, will at serensoft.com <will at serensoft.com>wrote:

> Hi Steve --
>
> Filesystem for storage, not the database. Using Mysql--yikes!
>
> I suspect the JAVA_OPTS needs some TLC. Tomcat is running as a service
> in 64-bit windows:
> Initial memory pool: 512
> Maximum memory pool: 3600
> -Xms3600m
> -Xmx3600m
> -XX:MaxPermSize=512m
> -XX:+UseAdaptiveSizePolicy
> -XX:+UseConcMarkSweepGC
> -XX:+UseParNewGC
> -Dcatalina.base=C:\Tomcat55
> -Dcatalina.home=C:\Tomcat55
> -Djava.endorsed.dirs=C:\Tomcat55\common\endorsed
> -Djava.io.tmpdir=C:\Tomcat55\temp
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> -Djava.util.logging.config.file=C:\Tomcat55\conf\logging.properties
>
> Any obvious glitches there?
>
>
> # from sakai.properties
> content.upload.max=50
> bodyPath at org.sakaiproject.content.api.ContentHostingService
> =\\\\192.168.9.99\\SakaiResources
>
> We are using MySQL, but if we were putting the binary data in the
> database, it would certainly kill the project (we timed an 8mb upload
> at 1m43s in the DB, vs. 3s in the filesystem).
>
> In theory, here, the only entry in the database is a reference to
> where the real file is, in the file system. Right?
>
>
>
> On Tue, Nov 17, 2009 at 5:24 PM, Steve Swinsburg
> <steve.swinsburg at gmail.com> wrote:
> > Hi Will,
> > As Remi said, to make sure your DB isn't restricted, if using MySQL and
> > storing content in the DB, set the max_allowed_packet to something big. I
> > blogged about this
> > here:
> http://steve-on-sakai.blogspot.com/2008/11/sakai-mysql-and-storing-content-in.html
> > This tidbit should probably be in the install guide as its come up a
> number
> > of times before.
> > You'll also need to make sure you have enough resources available to the
> > JVM. What are your JAVA_OPTS? With a 1.5-2Gb heap (wasn't any more than
> > that), we've successfully uploaded files around 400Mb to an offsite
> Oracle
> > cluster.
> > As for adding content via an InputStream rather than byte[], looks
> > like http://jira.sakaiproject.org/browse/KNL-325 is the one and the fix
> > version is in Kernel trunk v1.1.0-beta04 (not 2.6.x):
> >
> > cheers,
> > Steve
> >
> >
> > On Wed, Nov 18, 2009 at 8:53 AM, Branden Visser <branden at uwindsor.ca>
> wrote:
> >>
> >> Charles,
> >>
> >> Content streaming is a work-in-progress. Some tickets that come to mind
> >> are:
> >>
> >> http://jira.sakaiproject.org/browse/SAK-6306
> >> http://jira.sakaiproject.org/browse/KNL-325
> >> http://jira.sakaiproject.org/browse/SAK-16806
> >>
> >> Thanks,
> >> Branden
> >>
> >> Charles Hedrick wrote:
> >> > Why is this writing from a byte array to a file? Why aren't we using
> the
> >> > facilities to patch the input stream directly into a file output
> stream?
> >> > We want to use Sakai to support multimedia work, so we want to allow
> >> > uploads of several hundred MB. We'd probably allow 1 GB if we could.
> >> > However the resulting huge byte arrays are unfriendly for Java's heap.
> >> >
> >> >
> >> > On Nov 17, 2009, at 3:22 PM, will at serensoft.com wrote:
> >> >
> >> >> org.sakaiproject.content.impl.DbContentService - failed to write file
> >> >> from byte-array to file
> >> >
> >> >
> >> >
> ------------------------------------------------------------------------
> >> >
> >> > _______________________________________________
> >> > sakai-dev mailing list
> >> > sakai-dev at collab.sakaiproject.org
> >> > http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >> >
> >> > TO UNSUBSCRIBE: send email to
> >> > sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> >> > "unsubscribe"
> >> _______________________________________________
> >> sakai-dev mailing list
> >> sakai-dev at collab.sakaiproject.org
> >> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >>
> >> TO UNSUBSCRIBE: send email to
> >> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> >> "unsubscribe"
> >
> >
> > _______________________________________________
> > sakai-dev mailing list
> > sakai-dev at collab.sakaiproject.org
> > http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >
> > TO UNSUBSCRIBE: send email to
> sakai-dev-unsubscribe at collab.sakaiproject.org
> > with a subject of "unsubscribe"
> >
>
>
>
> --
> will trillich
> "Tis the set of the sails / And not the gales / That tells the way we
> go." -- Ella Wheeler Wilcox
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20091119/b9e0c332/attachment.html