| View previous topic :: View next topic |
| Author |
Message |
William Kenworthy
Guest
|
 Posted: Sun Nov 16, 2008 4:40 pm Post subject: Re: [gentoo-user] Back up a server in real-time |
 |
|
It actually does a very fast, space efficient backup management.
As for restore, each backup is effectively an uncompressed fully
accessible mirror image so you can manually copy/tar/... etc back for
the full system, or individual files. The developers basically say we
have a working backup system where you can restore using standard tools
if needed, but we'll make it easier when we can get around to it.
Having versioned backups can be REALLY handy at times. Space efficiency
typically stabilises at 2x archive size depending on delta.
Downside it can hammer a disk - reiserfs is fine but ext2/3 was flaky.
moriah ~ # esearch dirvish
[ Results for search key : dirvish ]
[ Applications found : 1 ]
* app-backup/dirvish
Latest version available: 1.2.1
Latest version installed: 1.2.1
Size of downloaded files: [no/bad digest]
Homepage: http://www.dirvish.org/
Description: Dirvish is a fast, disk based, rotating network
backup system.
License: OSL-2.0
Its worth a read on how its done - using smarts rather than brute force!
BillK
On Sun, 2008-11-16 at 11:58 +0200, Alan McKinnon wrote:
| Quote: | On Sunday 16 November 2008 11:04:41 William Kenworthy wrote:
dirvish is in portage.
tar/rsync/cp are not really backups but manual copies. If you want true
backup you need scripts to handle the extra functionality for things
like versioning, archive management and the all important restore.
Dirvish is excellent on all but restore.
Which raises the question:
What's the point of it then?
OTOH, it has to be better than legato. Nothing could be worse than legato.
-- |
William Kenworthy <billk@iinet.net.au>
Home in Perth! |
|
| |
|
 Back to top |
Mick
Guest
|
| Posted: Sun Nov 16, 2008 6:00 pm Post subject: Re: [gentoo-user] Back up a server in real-time |
|
|
On Sunday 16 November 2008, Alan McKinnon wrote:
| Quote: | On Sunday 16 November 2008 02:08:42 Mick wrote:
On Saturday 15 November 2008, Dale wrote:
Mick wrote:
Without gentoo-wiki my knowledge level is rather poor (just like my
memory!)
What would you use to back up a running server without taking it off
line?
I keep mine simple, cp -auv paths/you/want/to/backup back/up/to It has
works so far. Thought about doing a cron job but that complicates
things. :/
Thank you all for the suggestions and for the link to the wiki! I've got
some reading to do. ;-)
Whenever I have used tar to back up a whole OS I used it with a LiveCD.
This was to make sure that files and their metadata were not being
changed while I was tar'ing them.
Are you saying that I can actually fire up tar/rsync and back up in real
time?
Yes. Unix does some RealSmartThings(tm) when using files. The name is just
a pointer to the actual file, represented by an inode. Once you have an
inode open, it stays open until everything using it closes it. So you can
add/delete/copy/move files by name with impunity as you then just move
names around. Contrast this with other inferior systems, like say Windows
for example, which has a built-in self-destruct button when you try this...
|
Sure, but isn't there a problem with atime mtime metadata when you carry out a
backup in real time and then restore from it?
| Quote: | I was gravitating towards using LVM snapshot and then tar'ing that to an
external USB drive.
This is the preferred way, as you get a consistent snapshot frozen at a
point in time. This deals nicely with inconsistencies caused by files
changing while you are backing up other ones.
|
Right, that's what I was thinking too. What does restoring from a backed up
snapshot involve?
--
Regards,
Mick |
|
| |
|
| Back to top |
William Kenworthy
Guest
|
| Posted: Sun Nov 16, 2008 6:40 pm Post subject: Re: [gentoo-user] Back up a server in real-time |
|
|
On Sun, 2008-11-16 at 12:39 +0200, Alan McKinnon wrote:
| Quote: | On Sunday 16 November 2008 12:31:16 William Kenworthy wrote:
* app-backup/dirvish
Latest version available: 1.2.1
Latest version installed: 1.2.1
Size of downloaded files: [no/bad digest]
Homepage: http://www.dirvish.org/
Description: Dirvish is a fast, disk based, rotating network
backup system.
License: OSL-2.0
Its worth a read on how its done - using smarts rather than brute force!
So the basic premise is that backups are done frequently and need to be
efficient, whereas restores are only done in the event of a mistake. Less
efficient restores is then a fair trade?
I see people get this wrong all the time at work. They assume that backups are
some sort of rapid shared storage system. Which of course it isn't.
|
I wouldnt call it fair trade, rather "its working now and we'll make it
better it eventually."
I agree, backups should be just that - protection from mistakes/lost
data. However this is from someone who has the archives mounted via nfs
and can browse via filemanager an retrieve files that way via a desktop.
Currently I have my freeruner mobile phone, mythtv critical files and a
number of systems such as laptops backed up this way and its been very
reliable and useful, both as automatic/unattended and manually run.
Its neat in that you only need rsync and ssh (though its possible to use
rsh and the like) on the client. You only need dirvish on the server.
Starting to sound like a salesman for it! - but I have been very happy
with it. I used to use custom rsync scripts, different backup managers
and even tivoli backup at work. Dirvish is best for my usage.
BillK |
|
| |
|
| Back to top |
Alan McKinnon
Guest
|
| Posted: Sun Nov 16, 2008 7:10 pm Post subject: Re: [gentoo-user] Back up a server in real-time |
|
|
On Sunday 16 November 2008 13:51:23 Mick wrote:
| Quote: | Yes. Unix does some RealSmartThings(tm) when using files. The name is
just a pointer to the actual file, represented by an inode. Once you have
an inode open, it stays open until everything using it closes it. So you
can add/delete/copy/move files by name with impunity as you then just
move names around. Contrast this with other inferior systems, like say
Windows for example, which has a built-in self-destruct button when you
try this...
Sure, but isn't there a problem with atime mtime metadata when you carry
out a backup in real time and then restore from it?
|
With a restore, you have really just two options:
- consider the file being restored to be a new file and set the *time to now
- consider it a full restore and set them the same as what's in the backup
root or the file's owner is permitted to do the latter
There's a third option which makes little sense: set the *time of the restored
file to be the same as whatever file it is repalcing on disk. But this is
mostly silly as the file data is now inconsistent with the recorded times
| Quote: | I was gravitating towards using LVM snapshot and then tar'ing that to
an external USB drive.
This is the preferred way, as you get a consistent snapshot frozen at a
point in time. This deals nicely with inconsistencies caused by files
changing while you are backing up other ones.
Right, that's what I was thinking too. What does restoring from a backed
up snapshot involve?
|
The backup is just a backup, the fact that it was made from a frozen disk
snapshot is irrelevant. So you would restore it in the usual manner for the
backup format/method in use
--
alan dot mckinnon at gmail dot com |
|
| |
|
| Back to top |
Alan McKinnon
Guest
|
| Posted: Sun Nov 16, 2008 7:20 pm Post subject: Re: [gentoo-user] Back up a server in real-time |
|
|
On Sunday 16 November 2008 14:36:22 William Kenworthy wrote:
| Quote: | I wouldnt call it fair trade, rather "its working now and we'll make it
better it eventually."
I agree, backups should be just that - protection from mistakes/lost
data. However this is from someone who has the archives mounted via nfs
and can browse via filemanager an retrieve files that way via a desktop.
Currently I have my freeruner mobile phone, mythtv critical files and a
number of systems such as laptops backed up this way and its been very
reliable and useful, both as automatic/unattended and manually run.
|
OK I see where you are coming from.
When you say backup/restore to me, I think in terms of what I'm used to -
massive tape archives holding many multi-TB of data for thousands of hosts.
What you are describing is indeed a backup, but in my world I'd probably call
it something else.
Perhaps I should have been more explicit and said "Disaster Recovery Archival
Backup" instead of "backup"... :-)
--
alan dot mckinnon at gmail dot com |
|
| |
|
| Back to top |
Jorge Peixoto de Morais N
Guest
|
| Posted: Sun Nov 16, 2008 11:20 pm Post subject: Re: [gentoo-user] Back up a server in real-time |
|
|
On Sat, Nov 15, 2008 at 10:08 PM, Mick <michaelkintzios@gmail.com> wrote:
| Quote: | On Saturday 15 November 2008, Dale wrote:
Mick wrote:
Without gentoo-wiki my knowledge level is rather poor (just like my
memory!)
What would you use to back up a running server without taking it off
line?
I keep mine simple, cp -auv paths/you/want/to/backup back/up/to It has
works so far. Thought about doing a cron job but that complicates
things. :/
Thank you all for the suggestions and for the link to the wiki! I've got some
reading to do. ;-)
Whenever I have used tar to back up a whole OS I used it with a LiveCD. This
was to make sure that files and their metadata were not being changed while I
was tar'ing them.
Are you saying that I can actually fire up tar/rsync and back up in real time?
|
Please read tar's texinfo manual, at least the section
5 Performing Backups and Restoring Files
It is short, and nicely divided in subsections making it is easy to
select you need/want to read. Not like a man page, which for me is
useful as a reference but horrible at teaching how to use the program
(unless the program is quite simple).
--
Software is like sex: it is better when it is free - Linus Torvalds |
|
| |
|
| Back to top |
Joerg Schilling
Guest
|
| Posted: Mon Nov 17, 2008 10:10 pm Post subject: Re: [gentoo-user] Back up a server in real-time |
|
|
William Kenworthy <billk@iinet.net.au> wrote:
| Quote: | dirvish is in portage.
tar/rsync/cp are not really backups but manual copies. If you want true
backup you need scripts to handle the extra functionality for things
like versioning, archive management and the all important restore.
|
star has everything you need for backups and it is based on the standard
archive format.
Jörg
--
EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
js@cs.tu-berlin.de (uni)
schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily |
|
| |
|
| Back to top |
Mark Somerville
Guest
|
| Posted: Wed Nov 19, 2008 10:10 pm Post subject: Re: [gentoo-user] Back up a server in real-time |
|
|
On Sat, Nov 15, 2008 at 07:45:04PM +0000, Mick wrote:
| Quote: | Without gentoo-wiki my knowledge level is rather poor (just like my memory!)
What would you use to back up a running server without taking it off line?
|
I like rdiff-backup, which gives incremental backups over rsync.
Mark |
|
| |
|
| Back to top |
Harry Putnam
Guest
|
| Posted: Wed Nov 19, 2008 11:20 pm Post subject: [gentoo-user] Re: Back up a server in real-time |
|
|
Mark Somerville <mark@scottishclimbs.com> writes:
| Quote: | On Sat, Nov 15, 2008 at 07:45:04PM +0000, Mick wrote:
Without gentoo-wiki my knowledge level is rather poor (just like my memory!)
What would you use to back up a running server without taking it off line?
I like rdiff-backup, which gives incremental backups over rsync.
|
I just started playing with rdiff-backup. If you are familiar with
rsnapshot maybe you'll know if rdiff-backup is better in some way.
I've been using rsnapshot for quite a while but haven't had to do a
serious restore (knock on wood).
The small scale restores I've needed seemed to be something of a pita.
I guess you have to search the rsnapshot created hardlinks for the
version you want manually. At least that is what I ended up doing.
There may be better ways but rsnapshot doesn't supply any help in that
regard far as I know. Other than the biggest part of having created
the file you need that is. |
|
| |
|
| Back to top |
Dirk Heinrichs
Guest
|
| Posted: Thu Nov 20, 2008 8:35 am Post subject: Re: [gentoo-user] Re: Back up a server in real-time |
|
|
Am Mittwoch 19 November 2008 18:15:01 schrieb ext Harry Putnam:
| Quote: | I guess you have to search the rsnapshot created hardlinks for the
version you want manually. At least that is what I ended up doing.
|
Yes, that's true. But although it may be a pain to search through it, it's
already a good step forward to even have that capability. AFAIK no traditional
backup system has content search capabilities. One can usually only tell it to
restore a set of files as of a specific date. You have to restore first, then
check the content and iterate that until you finally found the version you
want.
Now, with those rsync based backup tools, you get a plain copy of your data
somewhere, so you can reverse that process. You can search through it using
find/grep/awk/whatever (or even use one of those desktop search engines to
index the whole backup like you would do for your home directory) and then
restore exactly the version you want.
Bye...
Dirk
--
Dirk Heinrichs | Tel: +49 (0)162 234 3408
Configuration Manager | Fax: +49 (0)211 47068 111
Capgemini Deutschland | Mail: dirk.heinrichs@capgemini.com
Wanheimerstraße 68 | Web: http://www.capgemini.com
D-40468 Düsseldorf | ICQ#: 110037733
GPG Public Key C2E467BB | Keyserver: wwwkeys.pgp.net |
|
| |
|
| Back to top |
Mick
Guest
|
| Posted: Mon Dec 01, 2008 5:00 am Post subject: Re: [gentoo-user] Back up a server in real-time |
|
|
On Monday 17 November 2008, Joerg Schilling wrote:
| Quote: | William Kenworthy <billk@iinet.net.au> wrote:
dirvish is in portage.
tar/rsync/cp are not really backups but manual copies. If you want true
backup you need scripts to handle the extra functionality for things
like versioning, archive management and the all important restore.
star has everything you need for backups and it is based on the standard
archive format.
|
Thanks Joerg, I have just had a quick look at star. Would you know if there
is a LiveCD that has star - or will I be able to restore a fs from an archive
created with star, but using tar to do it as it comes in e.g. Knoppix?
--
Regards,
Mick |
|
| |
|
| Back to top |
Joerg Schilling
Guest
|
| Posted: Mon Dec 01, 2008 5:10 am Post subject: Re: [gentoo-user] Back up a server in real-time |
|
|
Mick <michaelkintzios@gmail.com> wrote:
| Quote: | On Monday 17 November 2008, Joerg Schilling wrote:
William Kenworthy <billk@iinet.net.au> wrote:
dirvish is in portage.
tar/rsync/cp are not really backups but manual copies. If you want true
backup you need scripts to handle the extra functionality for things
like versioning, archive management and the all important restore.
star has everything you need for backups and it is based on the standard
archive format.
Thanks Joerg, I have just had a quick look at star. Would you know if there
is a LiveCD that has star - or will I be able to restore a fs from an archive
created with star, but using tar to do it as it comes in e.g. Knoppix?
|
Did you check grml?
Jörg
--
EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
js@cs.tu-berlin.de (uni)
schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily |
|
| |
|
| Back to top |
|
