Difference between revisions of "Hardware Issues"

Latest revision as of 05:22, 7 February 2006

Setting up Backup for Linux box (esp with DVD)

Initial Post of question to message board (hardhats)

I need a reliable backup solution for my linux server.
 Here are methods I have explored, unsuccessfully so
far.  Let me first mention that our office also has a
new Windows Server running our billing package.  My
initial thoughts were to copy my linux data to the
Windows server, and then let their backup process save
my data.

1. Install Services for Linux (SFU) put out by Windows
onto our Windows server.  I then created a NFS share. 
I can view the directory contents of the Windows share
from my linux box, but for the life of me I can't get
write permission.  You have to run a Windows
namemapper to map linux users to Windows users, and
then give that user write permission.  Sounds easy.  I
can't get it to work.  The permission system on NTFS
is a bit complex to me.

2. Run a Samba server on my linux box.  When I do
this, I can see the drive from a Windows laptop, and I
might even be able to write to it (I can't remember). 
But when I try to find the file when browsing from the
Windows Server, I can't find it.  Not sure if I have
to join Windows "domain" to work.  This descends into
a murky realm for me of ActiveDirectories.  Also, I'm
not sure if my iptables is filtering out the samba
ports etc.

3. Run a Samba client on my linux box.  I haven't
tried this, but again I worry that I would have to
discover how to join the Windows "domain" and be
validated.

4. Get scp to work on my Windows server, and scp the
needed data up.  This would probably work, but I would
need to use and 'expect' script to supply the password
etc if it is going to be automated.  This seems a bit
klunky, and will probably be my last resort.

5. Install a DVD writer on my linux box.  This is what
I am currently trying to do.  I successfully installed
a new USB 2.0 IO card, and have now plugged the drive
in.  I'm lost as to what to do now.  I have read
multiple web sites, and I'm overwhelmed.  Right off
the bat the install demos talk about possible kernel
upgrades etc.  It seems that cdrtools is one route to
use.  I would have to purchase a commercial license. 
The money is not such an issue, but I would want to be
sure it was going to work.

6. Purchase and install tape backup system for the
linux box.  This may well be a great options, but I
have no experience with this.


If anyone has any helpful thoughts or pointers (esp
regarding option 5 above), I would be very grateful.

Kevin

Samba is not my expertise, so I won't presume to give you debugging 
advice.  I will say that I have been able to get Samba and Windows 
working quite well together to share files and printers on multiple 
occasions.  When I have had problems using Linux as a server, it is 
usually because on the Linux side, I have not created the users with 
smbpasswd program (just because they exist in /etc/passwd doesn't mean 
that they exist for Samba - although I think there is now an option to 
propagate changes back and forth).  You can also webmin and swat to 
manage Samba via a browser.

Since Windows file attributes are not the same as Linux, I only 
recommend using Windows to backup your Linux files by first creating a 
tar archive on Linux and moving the tar archive to Linux.  By the way, 
to move a single file from Linux to Windows, you don't need Samba, just 
use WinSCP3 on Windows.

Getting a USB DVD burner to work on any contemporary Linux should be 
trivial.  When I got it working, I just powered down the PC, plugged in 
the DVD-ROM drive (a Sony drive that Chris Richardson put inside a box 
that provided power and interface electronics), powered up the DVD and 
powered up the PC.  K3b (my burning software) recognized the DVD 
immediately.  That PC runs the Debian Sarge installed via a Knoppix 3.6 
Linux live CD.  What Linux distribution and release are you running and 
what kernel?  If you connect a USB flash drive to the PC, is it 
recognized?  [In theory, you need not power it down, but just to 
simplify debugging, I like to start from a cold power up.]

I recommend creating a tar archive for backup.  Remember to backup your 
journal files too.

...
-- Bhaskar

>use.  I would have to purchase a commercial license. 
>The money is not such an issue, but I would want to be
>sure it was going to work.
>

As Bhaskar mentioned k3b is an excellent solution for burning CD and 
DVD's.
On this 64 bit Debian/Sarge laptop I did

# apt-get install k3b dvd+rw-tools cdrdao

You have the option of creating a seperate group (cdrom I think) for 
burning - or you can assign the permissions to the various tools during 
the setup when you first call the application.

I am not aware of any need for a commercial license for any of the 
above.  I would also expect the USB device to be well supported with 
most of the recent stock kernels.  Just as an aside, with 2.6.x you 
don't need to use SCSI emulation for IDE CDROM drives, which is nice.

>6. Purchase and install tape backup system for the
>linux box.  This may well be a great options, but I
>have no experience with this.
>  
>
My first experience with tape backup with GNU/Debian was with an old 
Colorado floppy based mini-tape backup system with the ftape drivers - 
it was slow, but worked perfectly (restore and backup).

If you add a SCSI tape backup you will have a huge array of software 
solutions - from something as simple as 'tos' (tape oriented 
backups-perl driven) to enterprise tools like Amanda. I have used 
various HP SCSI backup systems in the past and don't recall any 
problems.

And as mentioned, if you have questions about Samba and Windows John 
Terpstra's series of books and essays are excellent. BTW, he is a first 
rate speaker if you ever have the chance of attending one of his 
lectures or seminars, you will really get a huge boost up with regards 
to networking in heterogenous environments.

I hope this is of some help. --Ismet

Whoa, quite a bit here...

1.  Windows bites, do it our way or the highway.
2.  Samba - Get yourself the book by John Terpstra - Samba 3 By Example 
- 
Practical Exercises to Successful Deployment.
and/or 
Download my smb.conf tutorial file and have a go at joining that 
domain....
http://oswizards.com/samba-instruct.conf
If you would like I can give you access to one of my Linux Admin 
courses online.  You can study the Samba material to see if it clarifies things 
a little.  Samba is a huge subject, especially if you are going to deploy 
with 500+ uses.  I highly recommend the book. 

I don't think that will solve your problems though, you need to backup 
the Linux server.
3.  See number 2 above.
4.  You can share your private key/identity with the client and you 
will not have to supply a password when you execute your command.  You could do 
this with linux boxes easy, windows would be a hassle.  In Linux I could 
issue the command,  scp -r /var/data/* jerome@someotherhost.org:mydata/

5.  Oh yeah... you could install a DVD writer, CDR or an additional HDD 
or both.

Here are a couple tools for you.  Mondo Resuce 
http://www.microwerks.net/~hugo/about/about.html

I use another command line program from a cron job called rdiff-backup 
- 
http://www.nongnu.org/rdiff-backup/  It can take a snapshot of your 
filesystem whenever you tell it to, it also keeps diffs on all files 
that have been changed since the last backup.  You can actually restore a 
file from a specific date and time.  I snapshot at 3:10am onto a spare drive 
in the machine, I also snapshot my databases every hour.  You can use 
Mondo to 
create an entire system backup onto CD, HDD, NFS or DVD - 
http://www.ccp14.ac.uk/ccp14admin/linux-server/mondorescue/dvd_mondo.html 

Keep yourself clear of Windows and your choices will be plentiful and 
your difficulty will be diminished.

Setup another Linux Samba Server for the Windows clients/users to store 
their data on, then back that one up with the tools above. 

Have Fun!!!... and test your backups regularly to make sure you are 
indeed backing up your data.
--Mark Street

I just recalled another tool/method that you might want to look at - 
namely rsync

Here is a nice summary of one method employing it with automated 
scripts:

http://www.tux.org/~tbr/rsync/rsynchowto.html

Rsync is a fantastic tool - especially when combined with ssh.

Ismet / Ismet B. Kursunoglu, MD, FCCP

> 1. Install Services for Linux (SFU) put out by Windows
> onto our Windows server.  
<...snipped...>

Linux + NTFS is experimental at best still.  Do not know much about the
SFU deal from M$, but would not put too much trust in it for my obvious
prejudices. ;-)

> 2. Run a Samba server on my linux box.  
<...snipped...>

Samba on the Linux box will allow the M$ boxes to see and mount it as a
share potentially.  Possible idea, not the greatest though.

> 3. Run a Samba client on my linux box.  
<...snipped...>

If you are using RH (or one of the Fedora or CentOS forks), there are
some built in RH specific tools for Samba (redhat-config-samba for RH).
Not too hard using them.  Otherwise, there are a few HOW-To's on the
Samba subject.

> 4. Get scp to work on my Windows server, and scp the
> needed data up.  
<...snipped...>

This is actually one of the easier solutions you have presented so far
that could run unattended potentially.

scp is part of the SSH package typically (along with sftp).  This would
be one of your easiest solutions.  Tarball (with gz or bz2 compression
depending on the amount of data you wish to back up), scp the tarball 
to remote backup location directory.  You can make a simple script and 
cron job to do this on a regular basis.

> 5. Install a DVD writer on my linux box. 
<...snipped...>
> 

There is k3b which also noted by Bhaskar is freeware.  Nice tool 
overall once you get the DVD up and running, can handle CD and DVD burning.

At issue, you will be starting a large DVD collection potentially in
time.

> 6. Purchase and install tape backup system for the
> linux box.  This may well be a great options, but I
> have no experience with this.
> 

Old school, but possible via Amanda,

And finally...another Linux based solution presented is Bacula.  I
personally have not played with it (yet), but one of my guys is using 
it
for one of our clients on a regular basis.  This is a better or more
modern idea than Amanda, allows for secure file transfer to a remote
machine, and such.  http://www.bacula.org/ for their web site, and
yes...it is cross platformed as well.

--- Crawford, The Linux ETC Company

Here is something else you may want to consider: Unison - a file sync
tool for Unix and Windows

http://www.cis.upenn.edu/~bcpierce/unison/

--Jim Drash

For clarity, Unison, rsync, or any technique that copies files or parts
of files when the database is in use will not work very well, and the
copy will be corrupt.  This is because GT.M is constantly updating the
database when it is in use (unless you freeze updates).

Let's separate the two issues under consideration.  One is to get a 
copy of the database and journal files (backup), and the other is moving
backups to offline storage (archival).

Backup when there is no active GT.M process is trivial - just copy the
database files (unison, rsync and other techniques will work).  There
are (at least) two ways to backup when GT.M processes are active.  One
is to use mupip backup and the other involves using a mirrored disk.
(Promotional moment: attend my workshop in the Boston area on April 6
for hands-on training on backup - it's free but you have to RSVP!)

There are many options for archival, as discussed in this thread.

-- Bhaskar

>Thanks for your suggestions below.  The one issue
> about scp'ing the data, though, is that I would have
> to have an "scp server" running on the windows box.  I
> don't know how to set that up.  I have downloaded a
> program for windows, pscp that can pull the data from
> the linux box when run on the windows box.  It would
> be nice to be able to push it though.
> --Kevin

Cygwin will give you the sshd part-

http://pigtail.net/LRP/printsrv/cygwin-sshd.html

and this might be a good solution to limit access to scp for your 
scripts -

http://www.devguy.com/fp/scp/

--Ismet

> My impression is that K3B is very smart, much smarter than I am, 
thank goodness!  But, you generally need to start K3b as root if you want 
to consistently have joy in Mudville!
> -- Nancy Anthracite

So true, it takes some tweaking to get permissions just right for a 
mere mortal user to record CD's.  Let us not forget that the GUI tools are 
merely front ends for the command line tools from cd/DVD record tools.

http://fy.chalmers.se/~appro/linux/DVD+RW/
http://cdrecord.berlios.de/old/private/cdrecord.html

A system administrator has much more flexibility with command line 
tools when it comes to automating tasks.

> On Tuesday 22 March 2005 09:13 am, Bhaskar, KS wrote:
> Most likely, a DVD connected via a USB port will show up as a SCSI
> device, with a name like /dev/sda or /dev/sdb.  However, to burn a
> CD/DVD, you don't mount it (there is no file system to mount on a 
blank disk).
> --Bhaskar

It depends on the linux distribution but most newer distributions 
recognize USB CD drives as SCSI devices, they are labeled as /dev/scd0

I worked with a Latitude laptop this weekend and installed Debian Sarge 
and Fedora Core 3, both distros recognized the attached USB CD/RW as 
/dev/scd0.

Your USB mass storage devices, jump drives, cameras, etc. will show up 
as /dev/sda, /dev/sdb, etc.

> Generally, if you power up the USB drive first, then boot the PC, 
k3b
> will find and recognize the drive.  Also, as root, you can execute
> "cdrecord --scanbus" to find the "SCSI" drive number for your CD 
drive.

Most newer Linux distributions automatically recognize USB devices and 
make them available on boot.  Again, it depends on your Linux distro as to 
how much work will be involved.

> Mark and/or Crawford may want to correct me here, or expand on the
> explanation, because my understanding of how USB is mapped to SCSI, 
and
> how CD/DVD-ROM drives are handled on Linunx is shallow.

There is some difference how CD/DVD devices are handled in the version 
2.4 Linux kernel and 2.6 version, especially USB devices.  2.4 kernel has a 
SCSI emulation layer that was rather clunky, ide_scsi which allowed IDE 
devices to emulate SCSI devices, Linus didn't consider the interface elegant so it 
was reworked in 2.6.  There are some hard feelings on both sides between 
Linux and the author of cdrecord.  Too technical for my ears.
-- Mark Street, RHCE

Tape Backup *may* be considered old school, 
but there's a reason why it's still popular.  

I recommend to my clients that they spend the 
extra dollars up front on a real server, because 
then the majors will install a SCSI card and 
a DAT tape drive for doing backups.

SCSI, because every DAT drive is going to plug into
it, and there are ZERO emulation issue.

DAT rather than Travan, because of the media costs.  

A DAT tape is 5 bucks, a Travan is 50.  With DAT tapes, 
you can have NERDVANA:

Two full weeks of backups ( Mon1 - Thu1 & Mon2 - Thu2 )
A MONTH of Friday backups ( FriA - FriD )
and a perpetual archive of month ends

by just pulling the tape on the first, 
and replacing it with a blank...

The Auditors will LOVE you for that...

And even though I'm an open source advocate, 
I recommend a commercial product for doing 
the automated nightly backups.  When a 
harddrive dies, *I* don't want to be trying 
to figure out the easiest way to do a full 
metal restore onto a differently sized harddisk.  

24 hour support on you backup/restore product
is a very good thing.

Mike

mikelieman@gmail.com