[Building Sakai] Sakai as a large file dropbox

[Dave Ross]

>From what I've read there are a lot of database and/or driver specific
details to getting that to work, and support on MySql is limited. It's
probably doable, but just a lot easier when working with a separate
filesystem based binary storage mechanism.

On 10/10/09, Branden Visser <branden at uwindsor.ca> wrote:
>  JDBC allows you to stream data to the database. Are there some limitations
> to it that rule it out?
>
> Thanks,
> Branden
>
> -----sakai-dev-bounces at collab.sakaiproject.org wrote: -----
>
> To: Steve Swinsburg <steve.swinsburg at gmail.com>, Sakai Developers
> <sakai-dev at collab.sakaiproject.org>
> From: Dave Ross <dave.ross at gmail.com>
> Sent by: sakai-dev-bounces at collab.sakaiproject.org
> Date: 10/09/2009 08:03PM
> Subject: Re: [Building Sakai] Sakai as a large file dropbox
>
> Ah yes - no way to write to database storage without the entire file
> being in memory.
>
> MSSQL 2008 has a file stream data type which you can use to get around
> that limitation.
>
> On 10/9/09, Steve Swinsburg <steve.swinsburg at gmail.com> wrote:
>> 2.5.x, a revision as of early this year. But, they were being written
>> to the DB not the filesystem. So this is the limiting factor then.
>>
>> cheers.
>>
>>
>> On 10/10/2009, at 3:39 AM, Dave Ross wrote:
>>
>>> Right - but what version of Sakai?
>>>
>>> After 6306, large uploads do not consume memory that way (as long as
>>> they are being written to a filesystem and not to a db)
>>>
>>>
>>> On 10/8/09, Steve Swinsburg <steve.swinsburg at gmail.com> wrote:
>>>> In out situation, when someone uploaded a file that was rather large
>>>> (but under 2Gb) we ran out of memory. We had a small available heap
>>>> of
>>>> ~500Mb so limited file uploads to 400Mb. Anything much over that,
>>>> uploads would fail with a memory related stacktrace.
>>>>
>>>> Steve
>>>>
>>>>
>>>>
>>>> On 09/10/2009, at 3:23 AM, Dave Ross wrote:
>>>>
>>>>> To my knowledge, the 2gig limit was a Java datatype issue - not an
>>>>> available heap issue. Files up to 2gig in size have been possible
>>>>> since SAK-6306 and related tickets were resolved (in 2.4.x).
>>>>>
>>>>> -Dave
>>>>>
>>>>> On Thu, Oct 8, 2009 at 4:35 AM, Steve Swinsburg
>>>>> <steve.swinsburg at gmail.com
>>>>>
>>>>>> wrote:
>>>>> We were restricted by the amount of memory available and adding more
>>>>> was proving to be a hurdle so we hadn't gotten around to getting to
>>>>> the 'very large file support' going. But with that squared away, and
>>>>> the new kernel, very large files should be fine (in 2.7).
>>>>>
>>>>> cheers,
>>>>> Steve
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 08/10/2009, at 7:05 PM, David Horwitz wrote:
>>>>>
>>>>>> Actually files larger than 2G are only possible with the latest 1.1
>>>>>> range of kernels. This is due to the getContentLength() method in
>>>>>> contenthosting returning an int. Support for file >2G will be
>>>>>> something 2.7 will therefore support
>>>>>>
>>>>>> D
>>>>>>
>>>>>> Steve Swinsburg wrote:
>>>>>>>
>>>>>>> At NCeSS in the UK we have an upload limit of around 400Mb. The
>>>>>>> only limiting factor is the amount of JVM heap. We have people
>>>>>>> wanting to upload files in the order of Gb's which should be
>>>>>>> doable.
>>>>>>>
>>>>>>> cheers,
>>>>>>> Steve
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 08/10/2009, at 5:47 AM, Dave Ross wrote:
>>>>>>>
>>>>>>>> Once we moved to 2.5.x, in our instance "You can only upload 1000
>>>>>>>> MB worth of files at one time"
>>>>>>>>
>>>>>>>> We've tested various .iso files and yes it works.
>>>>>>>>
>>>>>>>> File storage is NTFS share in front of a SAN.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Oct 7, 2009 at 2:24 PM, Michael Wenk <mjwenk at ucdavis.edu>
>>>>>>>> wrote:
>>>>>>>> Branden Visser wrote:
>>>>>>>>> Hi everyone,
>>>>>>>>>
>>>>>>>>> What kind of limits do you put on your file upload sizes? We're
>>>>>>>> running
>>>>>>>>> Oracle 10g, storing our content in the database, and our
>>>>>>>> content field
>>>>>>>>> currently still has LONG RAW data-type.
>>>>>>>>>
>>>>>>>>> Aside from using LONG RAW instead of LOB and disk-space, is
>>>>>>>>> there
>>>>>>>>> anything that would keep us from being able to set, say, a
>>>>>>>> 100Mb file
>>>>>>>>> size limit?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Branden
>>>>>>>>> _______________________________________________
>>>>>>>>> sakai-dev mailing list
>>>>>>>>> sakai-dev at collab.sakaiproject.org
>>>>>>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>>>>>>
>>>>>>>>> TO UNSUBSCRIBE: send email to
>>>>>>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>>>>>> with a subject of "unsubscribe"
>>>>>>>>>
>>>>>>>>
>>>>>>>> At UC Davis, we set a file limit of 120 MB.  We use the file
>>>>>>>> system(AFS
>>>>>>>> in our case) to store the data.
>>>>>>>>
>>>>>>>> Mike
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> sakai-dev mailing list
>>>>>>>> sakai-dev at collab.sakaiproject.org
>>>>>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>>>>>
>>>>>>>> TO UNSUBSCRIBE: send email to
>>>>>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>>>>>> with a subject of "unsubscribe"
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> sakai-dev mailing list
>>>>>>>> sakai-dev at collab.sakaiproject.org
>>>>>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>>>>>
>>>>>>>> TO UNSUBSCRIBE: send email to
>>>>>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>>>>>> with a subject of "unsubscribe"
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> sakai-dev mailing list
>>>>>>> sakai-dev at collab.sakaiproject.org
>>>>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>>>>
>>>>>>> TO UNSUBSCRIBE: send email to
>>>>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>>>>> with a subject of "unsubscribe"
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> sakai-dev mailing list
>>>>> sakai-dev at collab.sakaiproject.org
>>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>>
>>>>> TO UNSUBSCRIBE: send email to
>>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>>> with a subject of "unsubscribe"
>>>>>
>>>>
>>>>
>>>
>>> --
>>> Sent from my mobile device
>>
>>
>
> --
> Sent from my mobile device
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org
> with a subject of "unsubscribe"
>

-- 
Sent from my mobile device