[Building Sakai] Data Archiving?

[Geng, Kelly]

Thanks Matt. That's very helpful.

Kelly


On Thu, Feb 14, 2013 at 10:59 PM, Matthew Jones <matthew at longsight.com>wrote:

> Yea, I did remember using the bodyVolumes improving things. I think the
> sys admins also created new volumes yearly to keep newer files out of the
> old volumes. It would still read from the old volumes that were accessible
> but only write new content to the new volumes. The only files that would go
> into the old volumes would be older updated files which was *very*
> infrequently changed content. Then most of the time during the backup is
> processing old files looking for recently modified files and just sending
> the stuff from the newest bodyVolume, which usually goes pretty quickly,
> even for that much space.
>
> You could probably even have something set to watch for filesystem
> modification events, either with Splunk [1] or inotify, keep track of all
> the files that were modified and avoid the scanning entirely. I guess it
> just depended on how much dev the sys admins wanted to put into it, or how
> much was automated by the existing software. It seems like the good backup
> software would be doing that by now though.
>
> [1]
> http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitorchangestoyourfilesystem
>
>
> On Thu, Feb 14, 2013 at 10:38 PM, Geng, Kelly <gengx at miamioh.edu> wrote:
>
>> Sorry I didn't make it clear: I was talking about backing up file
>> system.We are already using  the bodyPath property in sakai.properties to
>> store away binary data on the file system. We are also using Tivoli to do
>> incremental backup. The issue I found out later with us is that we are not
>> using volumes and thus all data are being backed up daily, instead of the
>> rotation backup like some universities do. I guess UMich is also doing the
>> rotation backup where you only back up more current volumes more frequently
>> while leaving the relatively "old" volumes backed up less frequently? I'll
>> talk to our sys admin and try this strategy. Thanks!
>>
>> Kelly
>>
>>
>>
>> On Wed, Feb 13, 2013 at 10:17 PM, Matthew Jones <matthew at longsight.com>wrote:
>>
>>> As long as you're under 24 hours you're good right?! :)
>>>
>>> If your files are on the filesystem as Chuck suggested, I'd probably
>>> just change my backup strategy to only backup files modified since the last
>>> backup ran. At Michigan, we had/have Tivoli Storage Manager available which
>>> backed up about 1GB of changed files (out of about a 100GB partition) in
>>> about 5 minutes. Some setup with rsync would probably do the same thing.
>>> What are your admins using now?
>>>
>>> Generally any effort to clean up files in the past (removing old sites,
>>> finding the biggest files) only gained around 20-25% disk space, so really
>>> wasn't worth the time since it was filled up so fast again.
>>>
>>>
>>> On Wed, Feb 13, 2013 at 10:06 PM, Charles Severance <csev at umich.edu>wrote:
>>>
>>>> Kelly - this is kind of a basic and obvious question - have you moved
>>>> your blobs out of your database and onto a file system?   If not, you
>>>> should.
>>>>
>>>> /Chuck
>>>>
>>>> On Feb 13, 2013, at 5:27 PM, Geng, Kelly wrote:
>>>>
>>>> Hi Dev-list,
>>>>
>>>> I wonder whether schools which have been running Sakai for a few years
>>>> have started to archive away course data, especially data on the hard disk?
>>>> Our System Admins have started to complain about it taking 14 hours for a
>>>> daily backup to finish. We've run Sakai just over 2 years and have 2.5 T
>>>> data on the disk. Any ideas or practice to recommend?
>>>>
>>>> Thanks,
>>>> --
>>>> Kelly
>>>> _______________________________________________
>>>> sakai-dev mailing list
>>>> sakai-dev at collab.sakaiproject.org
>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>
>>>> TO UNSUBSCRIBE: send email to
>>>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>>>> "unsubscribe"
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> sakai-dev mailing list
>>>> sakai-dev at collab.sakaiproject.org
>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>
>>>> TO UNSUBSCRIBE: send email to
>>>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>>>> "unsubscribe"
>>>>
>>>
>>>
>>
>>
>> --
>> Kelly
>>
>
>


-- 
Kelly
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20130215/968a686d/attachment.html