[Building Sakai] Data Archiving?

[Matthew Jones]

Yea, I did remember using the bodyVolumes improving things. I think the sys
admins also created new volumes yearly to keep newer files out of the old
volumes. It would still read from the old volumes that were accessible but
only write new content to the new volumes. The only files that would go
into the old volumes would be older updated files which was *very*
infrequently changed content. Then most of the time during the backup is
processing old files looking for recently modified files and just sending
the stuff from the newest bodyVolume, which usually goes pretty quickly,
even for that much space.

You could probably even have something set to watch for filesystem
modification events, either with Splunk [1] or inotify, keep track of all
the files that were modified and avoid the scanning entirely. I guess it
just depended on how much dev the sys admins wanted to put into it, or how
much was automated by the existing software. It seems like the good backup
software would be doing that by now though.

[1]
http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitorchangestoyourfilesystem


On Thu, Feb 14, 2013 at 10:38 PM, Geng, Kelly <gengx at miamioh.edu> wrote:

> Sorry I didn't make it clear: I was talking about backing up file
> system.We are already using  the bodyPath property in sakai.properties to
> store away binary data on the file system. We are also using Tivoli to do
> incremental backup. The issue I found out later with us is that we are not
> using volumes and thus all data are being backed up daily, instead of the
> rotation backup like some universities do. I guess UMich is also doing the
> rotation backup where you only back up more current volumes more frequently
> while leaving the relatively "old" volumes backed up less frequently? I'll
> talk to our sys admin and try this strategy. Thanks!
>
> Kelly
>
>
>
> On Wed, Feb 13, 2013 at 10:17 PM, Matthew Jones <matthew at longsight.com>wrote:
>
>> As long as you're under 24 hours you're good right?! :)
>>
>> If your files are on the filesystem as Chuck suggested, I'd probably just
>> change my backup strategy to only backup files modified since the last
>> backup ran. At Michigan, we had/have Tivoli Storage Manager available which
>> backed up about 1GB of changed files (out of about a 100GB partition) in
>> about 5 minutes. Some setup with rsync would probably do the same thing.
>> What are your admins using now?
>>
>> Generally any effort to clean up files in the past (removing old sites,
>> finding the biggest files) only gained around 20-25% disk space, so really
>> wasn't worth the time since it was filled up so fast again.
>>
>>
>> On Wed, Feb 13, 2013 at 10:06 PM, Charles Severance <csev at umich.edu>wrote:
>>
>>> Kelly - this is kind of a basic and obvious question - have you moved
>>> your blobs out of your database and onto a file system?   If not, you
>>> should.
>>>
>>> /Chuck
>>>
>>> On Feb 13, 2013, at 5:27 PM, Geng, Kelly wrote:
>>>
>>> Hi Dev-list,
>>>
>>> I wonder whether schools which have been running Sakai for a few years
>>> have started to archive away course data, especially data on the hard disk?
>>> Our System Admins have started to complain about it taking 14 hours for a
>>> daily backup to finish. We've run Sakai just over 2 years and have 2.5 T
>>> data on the disk. Any ideas or practice to recommend?
>>>
>>> Thanks,
>>> --
>>> Kelly
>>> _______________________________________________
>>> sakai-dev mailing list
>>> sakai-dev at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>
>>> TO UNSUBSCRIBE: send email to
>>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>>> "unsubscribe"
>>>
>>>
>>>
>>> _______________________________________________
>>> sakai-dev mailing list
>>> sakai-dev at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>
>>> TO UNSUBSCRIBE: send email to
>>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>>> "unsubscribe"
>>>
>>
>>
>
>
> --
> Kelly
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20130214/85ea86ef/attachment.html