[Building Sakai] Sakai as a large file dropbox

Sat Oct 17 03:26:17 PDT 2009

Sakai trunk (not yet 2.6 / 1.0.x) has support for partial content downloads. I've tested it with some large files (>2G) with Free Download Manager (which opens multiple connections each retrieving different ranges of the file) and didn't notice any particular problems, but that wasn't really a stress-test as such.

Regards
Stephen 

>>> "Berg, A.M." <A.M.Berg at uva.nl> 10/17/2009 11:38 AM >>> 
Hi,

For our main Learning Management System which is not yet Sakai, we found that we have potential problems with downloads of large files. The problem is not the file takes memory, but if the client stops and resumes the download then the CPU of the Tomcat server increases and so to the CPU of the Apache process infront of the tomcat server. If enough clinets do that then we have load issues. Looking in the apache log we see a related HTTP status of 206 (Partial Content. The server has fulfilled the partial GET request for the resource). This behaviour happens rarely, but as course sizes increases and file size also and with the advent of more and more multimedia content we expect a higher incident rate. We have seen this for content types such as MP3,PPT and PDF. We have seen this caused by Firefox, Sarfai and RealMedia player. Hopefully, Sakai does not have a similar soft spot.

Alan

Alan Berg

Senior Developer / Quality Assurance
Group Education and Research Services
Central Computer Services
University of Amsterdam

http://home.uva.nl/a.m.berg

-----Original Message-----
From: sakai-dev-bounces at collab.sakaiproject.org on behalf of Branden Visser
Sent: Fri 16-10-2009 22:13
To: Dave Ross
Cc: Sakai Developers
Subject: Re: [Building Sakai] Sakai as a large file dropbox

SAK-16806 and SAK-16838 suggest there might still be problems with 
sucking the whole file into memory in 2.6, file-system or not.

Keep your eye on those heaps ;)

Thanks,
Branden

Dave Ross wrote:
> Right - but what version of Sakai?
> 
> After 6306, large uploads do not consume memory that way (as long as
> they are being written to a filesystem and not to a db)
> 
> 
> On 10/8/09, Steve Swinsburg <steve.swinsburg at gmail.com> wrote:
>> In out situation, when someone uploaded a file that was rather large
>> (but under 2Gb) we ran out of memory. We had a small available heap of
>> ~500Mb so limited file uploads to 400Mb. Anything much over that,
>> uploads would fail with a memory related stacktrace.
>>
>> Steve
>>
>>
>>
>> On 09/10/2009, at 3:23 AM, Dave Ross wrote:
>>
>>> To my knowledge, the 2gig limit was a Java datatype issue - not an
>>> available heap issue. Files up to 2gig in size have been possible
>>> since SAK-6306 and related tickets were resolved (in 2.4.x).
>>>
>>> -Dave
>>>
>>> On Thu, Oct 8, 2009 at 4:35 AM, Steve Swinsburg <steve.swinsburg at gmail.com
>>>
>>>> wrote:
>>> We were restricted by the amount of memory available and adding more
>>> was proving to be a hurdle so we hadn't gotten around to getting to
>>> the 'very large file support' going. But with that squared away, and
>>> the new kernel, very large files should be fine (in 2.7).
>>>
>>> cheers,
>>> Steve
>>>
>>>
>>>
>>>
>>> On 08/10/2009, at 7:05 PM, David Horwitz wrote:
>>>
>>>> Actually files larger than 2G are only possible with the latest 1.1
>>>> range of kernels. This is due to the getContentLength() method in
>>>> contenthosting returning an int. Support for file >2G will be
>>>> something 2.7 will therefore support
>>>>
>>>> D
>>>>
>>>> Steve Swinsburg wrote:
>>>>> At NCeSS in the UK we have an upload limit of around 400Mb. The
>>>>> only limiting factor is the amount of JVM heap. We have people
>>>>> wanting to upload files in the order of Gb's which should be doable.
>>>>>
>>>>> cheers,
>>>>> Steve
>>>>>
>>>>>
>>>>>
>>>>> On 08/10/2009, at 5:47 AM, Dave Ross wrote:
>>>>>
>>>>>> Once we moved to 2.5.x, in our instance "You can only upload 1000
>>>>>> MB worth of files at one time"
>>>>>>
>>>>>> We've tested various .iso files and yes it works.
>>>>>>
>>>>>> File storage is NTFS share in front of a SAN.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Oct 7, 2009 at 2:24 PM, Michael Wenk <mjwenk at ucdavis.edu>
>>>>>> wrote:
>>>>>> Branden Visser wrote:
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> What kind of limits do you put on your file upload sizes? We're
>>>>>> running
>>>>>>> Oracle 10g, storing our content in the database, and our
>>>>>> content field
>>>>>>> currently still has LONG RAW data-type.
>>>>>>>
>>>>>>> Aside from using LONG RAW instead of LOB and disk-space, is there
>>>>>>> anything that would keep us from being able to set, say, a
>>>>>> 100Mb file
>>>>>>> size limit?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Branden
>>>>>>> _______________________________________________
>>>>>>> sakai-dev mailing list
>>>>>>> sakai-dev at collab.sakaiproject.org
>>>>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>>>>
>>>>>>> TO UNSUBSCRIBE: send email to
>>>>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>>>>  with a subject of "unsubscribe"
>>>>>> At UC Davis, we set a file limit of 120 MB.  We use the file
>>>>>> system(AFS
>>>>>> in our case) to store the data.
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> _______________________________________________
>>>>>> sakai-dev mailing list
>>>>>> sakai-dev at collab.sakaiproject.org
>>>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>>>
>>>>>> TO UNSUBSCRIBE: send email to
>>>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>>>>  with a subject of "unsubscribe"
>>>>>>
>>>>>> _______________________________________________
>>>>>> sakai-dev mailing list
>>>>>> sakai-dev at collab.sakaiproject.org
>>>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>>>
>>>>>> TO UNSUBSCRIBE: send email to
>>>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>>>>  with a subject of "unsubscribe"
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> sakai-dev mailing list
>>>>> sakai-dev at collab.sakaiproject.org
>>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>>
>>>>> TO UNSUBSCRIBE: send email to
>>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>>>  with a subject of "unsubscribe"
>>>
>>> _______________________________________________
>>> sakai-dev mailing list
>>> sakai-dev at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>
>>> TO UNSUBSCRIBE: send email to
>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>  with a subject of "unsubscribe"
>>>
>>
> 
_______________________________________________
sakai-dev mailing list
sakai-dev at collab.sakaiproject.org
http://collab.sakaiproject.org/mailman/listinfo/sakai-dev

TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"