[Building Sakai] Sakai as a large file dropbox

[Branden Visser]

 JDBC allows you to stream data to the database. Are there some limitations to it that rule it out?

Thanks,
Branden
 
-----sakai-dev-bounces at collab.sakaiproject.org wrote: -----

To: Steve Swinsburg <steve.swinsburg at gmail.com>, Sakai Developers <sakai-dev at collab.sakaiproject.org>
From: Dave Ross <dave.ross at gmail.com>
Sent by: sakai-dev-bounces at collab.sakaiproject.org
Date: 10/09/2009 08:03PM
Subject: Re: [Building Sakai] Sakai as a large file dropbox

Ah yes - no way to write to database storage without the entire file
being in memory.

MSSQL 2008 has a file stream data type which you can use to get around
that limitation.

On 10/9/09, Steve Swinsburg <steve.swinsburg at gmail.com> wrote:
> 2.5.x, a revision as of early this year. But, they were being written
> to the DB not the filesystem. So this is the limiting factor then.
>
> cheers.
>
>
> On 10/10/2009, at 3:39 AM, Dave Ross wrote:
>
>> Right - but what version of Sakai?
>>
>> After 6306, large uploads do not consume memory that way (as long as
>> they are being written to a filesystem and not to a db)
>>
>>
>> On 10/8/09, Steve Swinsburg <steve.swinsburg at gmail.com> wrote:
>>> In out situation, when someone uploaded a file that was rather large
>>> (but under 2Gb) we ran out of memory. We had a small available heap
>>> of
>>> ~500Mb so limited file uploads to 400Mb. Anything much over that,
>>> uploads would fail with a memory related stacktrace.
>>>
>>> Steve
>>>
>>>
>>>
>>> On 09/10/2009, at 3:23 AM, Dave Ross wrote:
>>>
>>>> To my knowledge, the 2gig limit was a Java datatype issue - not an
>>>> available heap issue. Files up to 2gig in size have been possible
>>>> since SAK-6306 and related tickets were resolved (in 2.4.x).
>>>>
>>>> -Dave
>>>>
>>>> On Thu, Oct 8, 2009 at 4:35 AM, Steve Swinsburg
>>>> <steve.swinsburg at gmail.com
>>>>
>>>>> wrote:
>>>> We were restricted by the amount of memory available and adding more
>>>> was proving to be a hurdle so we hadn't gotten around to getting to
>>>> the 'very large file support' going. But with that squared away, and
>>>> the new kernel, very large files should be fine (in 2.7).
>>>>
>>>> cheers,
>>>> Steve
>>>>
>>>>
>>>>
>>>>
>>>> On 08/10/2009, at 7:05 PM, David Horwitz wrote:
>>>>
>>>>> Actually files larger than 2G are only possible with the latest 1.1
>>>>> range of kernels. This is due to the getContentLength() method in
>>>>> contenthosting returning an int. Support for file >2G will be
>>>>> something 2.7 will therefore support
>>>>>
>>>>> D
>>>>>
>>>>> Steve Swinsburg wrote:
>>>>>>
>>>>>> At NCeSS in the UK we have an upload limit of around 400Mb. The
>>>>>> only limiting factor is the amount of JVM heap. We have people
>>>>>> wanting to upload files in the order of Gb's which should be
>>>>>> doable.
>>>>>>
>>>>>> cheers,
>>>>>> Steve
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 08/10/2009, at 5:47 AM, Dave Ross wrote:
>>>>>>
>>>>>>> Once we moved to 2.5.x, in our instance "You can only upload 1000
>>>>>>> MB worth of files at one time"
>>>>>>>
>>>>>>> We've tested various .iso files and yes it works.
>>>>>>>
>>>>>>> File storage is NTFS share in front of a SAN.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Oct 7, 2009 at 2:24 PM, Michael Wenk <mjwenk at ucdavis.edu>
>>>>>>> wrote:
>>>>>>> Branden Visser wrote:
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> What kind of limits do you put on your file upload sizes? We're
>>>>>>> running
>>>>>>>> Oracle 10g, storing our content in the database, and our
>>>>>>> content field
>>>>>>>> currently still has LONG RAW data-type.
>>>>>>>>
>>>>>>>> Aside from using LONG RAW instead of LOB and disk-space, is
>>>>>>>> there
>>>>>>>> anything that would keep us from being able to set, say, a
>>>>>>> 100Mb file
>>>>>>>> size limit?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Branden
>>>>>>>> _______________________________________________
>>>>>>>> sakai-dev mailing list
>>>>>>>> sakai-dev at collab.sakaiproject.org
>>>>>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>>>>>
>>>>>>>> TO UNSUBSCRIBE: send email to
>>>>>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>>>>> with a subject of "unsubscribe"
>>>>>>>>
>>>>>>>
>>>>>>> At UC Davis, we set a file limit of 120 MB.  We use the file
>>>>>>> system(AFS
>>>>>>> in our case) to store the data.
>>>>>>>
>>>>>>> Mike
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> sakai-dev mailing list
>>>>>>> sakai-dev at collab.sakaiproject.org
>>>>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>>>>
>>>>>>> TO UNSUBSCRIBE: send email to
>>>>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>>>>> with a subject of "unsubscribe"
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> sakai-dev mailing list
>>>>>>> sakai-dev at collab.sakaiproject.org
>>>>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>>>>
>>>>>>> TO UNSUBSCRIBE: send email to
>>>>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>>>>> with a subject of "unsubscribe"
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> sakai-dev mailing list
>>>>>> sakai-dev at collab.sakaiproject.org
>>>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>>>
>>>>>> TO UNSUBSCRIBE: send email to
>>>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>>>> with a subject of "unsubscribe"
>>>>
>>>>
>>>> _______________________________________________
>>>> sakai-dev mailing list
>>>> sakai-dev at collab.sakaiproject.org
>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>
>>>> TO UNSUBSCRIBE: send email to
>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>> with a subject of "unsubscribe"
>>>>
>>>
>>>
>>
>> --
>> Sent from my mobile device
>
>

-- 
Sent from my mobile device
_______________________________________________
sakai-dev mailing list
sakai-dev at collab.sakaiproject.org
http://collab.sakaiproject.org/mailman/listinfo/sakai-dev

TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20091010/d5b3b442/attachment.html