NVBackup alpha 8 for 1.00

[pruss]

My open source backup utility for NVFS devices is progressing nicely.

I'm going down from a beta to an alpha because I've overhauled a lot of code and added compression. The compression is on by default. This increases the time of a full backup by about 70%, but greatly speeds up updates of existing backup because the compressed files have CRCs and file dates that NVBackup can use to see what has been changed.

The compression is gzip standard. Note that although NVBackup puts out gzip standard compressed files, and you can decompress them with gzip, you should not use gzip to compress old backups and expect them to work with NVBackup. NVBackup currently only reads the gzip files it produces (but any conforming gzip utility should read the files NVBackup puts out.)

Don't use this for anything important! Make a backup with another utility first, and ideally put it on a separate card. USE AT YOUR OWN RISK!!!

http://sourceforge.net/project/showfiles.php?group_id=92189

[Greek]

Alex: you should remove the "alpha" tag, it´s rock solid on my TX! :D

1-Backup time was reduced from 25 to 20 mins in this new version.
2-Update time was reduced from 3 to 2 mins in this version.
3-Restore took only 2 mins. I like the fact that the app checks changes between the handheld and the SD, so it only updates what it changed. (I did not hard reset, just changed some memos, apointments and moved an app).
4-Backup size was reduced from 50MB to 22MB!!!!! :)
5-Compressed files were easily opened in my PC with WinRAR. Opening the gzip files show me (with prcs) that they are a "File", they are not a "NVBackup.prc" but "NVBackup_0000000", for example. If I rename it to NVBackup.prc, the program can´t be installed to my Palm, I get an error: "System Error, Failed to Copy" with WiFile and HotSync refused to install the application. Will it be a .prc if I change any of the "Compress:" Preferences?

Finally, if you get some free time... can you add selective Backup/Restore of the files you choose?

I must thank you for being generous by releasing this great app as a freeware, and even more for releasing the code so programmers can learn a lot from you.

Once again, thanks a lot! :)

Regards,

[pruss]

The backup files are in an undocumented (I think) format. This is quite a bit like prc/pdb, but with ARM byte order and some additional header data that I have not yet figured out. So I am currently unable to convert them to prc/pdb. Such conversion would slow things down anyway and reduce reliability. Eventually, I guess, I should write a PC-side program to convert to prc/pdb.

Depending on your SD card speed, compressed backup can be faster because less data is written. It depends on your CPU and card speeds what it is faster. On my TX and my 1gb SD card, non-compressed is faster. And compression doesn't give me quite as much because a fair bit of what I have is already compressed with the same compression method (e.g., myKbd skins or Plucker docs).

I want to do another scheduled nightly backup before I call this a beta. This was a real nuisance to get running, since I didn't know much about zlib/gzip before I started.

Alex

[pruss]

Selective backup/restore is a nice idea. Currently, the way to do a selective restore is to copy the selected files to a directory whose name ends with " (incomplete)", and then restore that. The " (incomplete)" (with a space before it) is crucial, because normally NVBackup restores to the system state stored in the directory in question, and if crucial files are missing in that directory, NVBackup will ensure they're missing after restore. With " (incomplete)", NVBackup adds the files in the backup dir, but doesn't delete other files.

Restore/Update is much faster because of the CRC. I could make it even faster by relying solely on the file time and size, but I don't want to do that (daylight savings might screw it up, for instance).

The CRC and de/compression code is all ARM, and based on zlib.

Interestingly, NVBackup does not use ANY undocumented APIs. This makes one hope that it'll work on new devices.

By the way, I really need an icon for it. Anybody with artistic talents?

[Greek]

Thanks for the explanations. It´s great to have a talented developer teaching us. (At least those of us who like these machines).

I will test also scheduled backup and come back with my report.

I wish I could have artistic talent to make you a cool icon, but I don´t even know how to draw on paper, much less on a computer. :)

Regards,

[PinCushionQueen]

AFAIK this new version works :) So far I've done a complete backup, updated and restored without a hard reset (I verfied the restore by changing my DIA/kbd and deleting an app before restoring). So far so good!

I've also continued to be able get the schedule feature to work. I'll have to wait until tonight (my middle of the night scheduled tasks) to know for sure... I've been verifying the alarms by Palm Internals and they've always been there. There must be something else interferring with the alarm but it is not consistant and therefore difficult to pin point the cause.

I just completed a new backup at "4" compression and the size = 17.4 (down from 17.9 @ "2" compression, down from 34.6 @ no compression) But you don't think that going any higher that 5 will do any good? Right?

I had changed the compression from 2 to 4 and did an update but the size did not change from the original (I wasn't really expecting it to - just curious if it would).


What does "slow but safer update/restore" option do?

Thanks again for this great app!! :D

[pruss]

AFAIK this new version works :) So far I've done a complete backup, updated and restored without a hard reset (I verfied the restore by changing my DIA/kbd and deleting an app before restoring). So far so good!

Great!

I just completed a new backup at "4" compression and the size = 17.4 (down from 17.9 @ "2" compression, down from 34.6 @ no compression) But you don't think that going any higher that 5 will do any good? Right?

That's my feeling. I did some experimentation and got a tiny benefit in going from 5 to 9. I don't think it's worth it.

What does "slow but safer update/restore" option do?

Thanks again for this great app!! :D

With this option off, to check whether to copy a file (either when updating or restoring), the uncompressed version is checksummed, and this checksum is compared against the compressed version. And file size and file date are also compared. This is fast as the compressed version doesn't need to be decompressed or read. With this option on, a full compare is done. This is slower.

There is one chance in four billion that two different files will have the same CRC. When one adds the date and size check, the chance of failure (i.e., of two files being counted as matching when they're not) is infinitesimal. (After all, the date check alone should always work unless you set the system date back at some point which might cause collisions.)

In the official release I may remove this option not to confuse the user. I hate relying on probabilistic data (despite being a probability theorist!), but it's much faster this way.

[pruss]

I wish I could have artistic talent to make you a cool icon, but I don´t even know how to draw on paper, much less on a computer. :)

I think I'm going to put an SD card in my flatbed scanner, and then photoshop "NVBackup" onto the image.


By the way, how important to people are restore points? And what sort of user interface should I have? One easy way to do this is to have two UPDATE functions: an overwrite update and a cumulative (??what's a good name for this) update. The overwrite update just replaces the latest data in the backup with the latest data on the device. The cumulative update adds new data, but makes a new restore point. And one could specify whether the daily backup does one or the other.

Would I then have two update buttons? But there is no space for it. I always hate working out the user interface.

And I may not want to do it anyway. :-) It depends on whether I can think up a nice directory structure for it.

Alex
p.s. One reason this is free is that I don't want the legal liabilities of making a commercial backup program. I am guessing that backup programs are held to higher standards than other software. That said, it's got to be reliable enough for me to use it.

[Greek]

If it is reliable for you, then it´s more than reliable for most of us. :D

I did a scheduled backup, everything went fine. I just picked cycle size 1, so it was an update. I prefered this option to see if there was any problem. I think it´s more complicated for the program to do an update than a new backup set. ;)

I am not quite sure of restore points; who will remember which would be the best point to get back to? I prefer a selective restore, to choose the databases to restore from a list. If I deleted a memo by mistake or think that I need it again, then I would only restore the Memo database and leave the rest untouched.

I liked your idea for the icon, if you want and have time, please make the small icon too, as I prefer to see the apps by List (instead of Icons). :)

Regards,

[MetaView]

Hm, just something small and fast. Maybe someone other will refine it?

[pruss]

Thanks for the icons!

I put them in and I added a selective restore (menu | restore | selective). I haven't tested on an actual device yet--just the simulator. But here it is, as beta 9:

http://sourceforge.net/project/showfiles.php?group_id=92189&package_id=186018

[PinCushionQueen]

What is "Low memory restore" used for exactly?

Also just want to clarify - if I delete an app and then go into selective restore and select the files to that app, NVBackup will restore only that app and the rest will be left unchanged? Correct? What if I don't delete the app first but just try to restore over the existing version (like to use for a roll back to a previous version of an app)? Do you know if these will work like this or should I just test them?

[pruss]

Normally, you need free memory to do a restore. Specifically, you need the amount of free memory equal to the size of the largest file you're restoring. This can be a problem if you're restoring a very large file. (Nothing disastrous should happen if you don't have the memory--that largest file should just be skipped.)

Standard restore is designed to handle failure nicely--so if you reset or run out of batteries mid way through a reset, you will likely have a partial restore, but hopefully it'll be a functioning system. But this requires that I not delete a file until a replacement has been written.

Low memory restore is the current trick to get around this--it wipes the existing files prior to the restore. This means that if you get a restore failure, you'll have to do a hard reset, but it also means you don't need any more space than is actually available.

I need to automate this in some way, maybe through a pre-scan that decides on the appropriate restore mode depending on the free space available. And I need to make sure that the code that does this is really reliable.

[pruss]

if I delete an app and then go into selective restore and select the files to that app, NVBackup will restore only that app and the rest will be left unchanged? Correct?

That's the theory.

What if I don't delete the app first but just try to restore over the existing version (like to use for a roll back to a previous version of an app)?

It's supposed to work. And because of the way NVBackup works, it shouldn't matter whether the app is in use, resident, locked, set to read-only, etc.

[Greek]

I must thank you once again. Selective Restore worked great, deleted a couple of Memos, then restored and they were back.

I think it would be enough to write what "Low Memory Restore" is in the Help or the "i" icons, that way, you don´t have to code anything extra. :)

And also, I want to thank Henk for the icons, even in List mode (small icons) they look great. ;)

I can´t think in any other improvements except for Selective Backup. That way, one could be able to create a special set with certain databases (which will occupy less space). For example, I would like to create a Set for the PIM databases and maybe updating them much more frequently than the full backup.

Finally, will it work on non NVFS devices? (I don´t have one right now to try), or will NVBackup check if the device is NVFS?

Thanks once again for the app, for the improvements suggested and for teaching us. :)

Regards,

[pruss]

I think it would be enough to write what "Low Memory Restore" is in the Help or the "i" icons, that way, you don´t have to code anything extra. :)


I should do the extra coding. The user shouldn't have to think about HOW to restore.

I can´t think in any other improvements except for Selective Backup. That way, one could be able to create a special set with certain databases (which will occupy less space). For example, I would like to create a Set for the PIM databases and maybe updating them much more frequently than the full backup.

That would require me to implement a user interface for backup comments, so you'd know what's where. And I'd have to have a new update mode that only rewrites the files that are already there. (Currently, if you updated a partial backup, it would dump all the "missing" files in.)

Finally, will it work on non NVFS devices?

No, and I have no plans for that. That's a completely different kettle of fish, and I have no need for it. (On my NX70, I use the Clie Backup utility.)

But, yes, it checks for the presence of NVFS.

Best wishes,
Alex

[MetaView]

Well, I've installed the latest beta (9?) and it's the first time I've actually used it. What shall I say: My TX is in a reset loop... I can break out using the no-notify reset. But as soon as I do a normal softreset the reset loop is back. It hapens just after I made the backup, which finished without error (as far as I've seen it). Doing a restore of this backup with NVBackup didn't help. I will try an hardreset + NVBackup tomorrow. Or is there anything else I should do before?

To add: DebugPrefs tells me: SystemMgr.c, Line 4190, Error launching application. If I remember correctly it is the same I've got after I've tried (by accident) the freaking DAA (from the Zod) on my TX.

[pruss]

Well, I've installed the latest beta (9?) and it's the first time I've actually used it. What shall I say: My TX is in a reset loop... I can break out using the no-notify reset. But as soon as I do a normal softreset the reset loop is back. It hapens just after I made the backup, which finished without error (as far as I've seen it). Doing a restore of this backup with NVBackup didn't help. I will try an hardreset + NVBackup tomorrow. Or is there anything else I should do before?

To add: DebugPrefs tells me: SystemMgr.c, Line 4190, Error launching application. If I remember correctly it is the same I've got after I've tried (by accident) the freaking DAA (from the Zod) on my TX.

Wow, that's bad. Sorry.

Suggestion: As a first test, when you do the warm reset, delete NVBackup from Program Memory (you might want to put a copy in \Palm\Launcher on the card for later use).

You might also try the low-memory restore option in NVBackup.

After you get your system back up, there are other tests we can do.

[MetaView]

Moving NVBackup to card didn't help. Doing a low-memory restore didn't help either. Shall I do an hardreset?

BTW: I've just seen there was error reading for some files on restore.
BTW2: The way to enter the time for the scheduled backup is quite difficult to read for europeans. Do you mind to switch to a 24h clock?

[pruss]

Well, it sure looks like it's not reliable for everyone... The read error is bad. It might be worth copying the files from the SD to a local directory and see if gunzip (or equivalent) can read all the files or if they're corrupt.

[pruss]

Henk sent me his backup file, with the proviso that I don't call his friends in the middle of the night. It looks like some AirSAM* files (what are these?) were corrupt and were causing the reset loop. I do not know if NVBackup corrupted them, but given that I never got corruption on backup before, I am wondering if some other app caused the problem. But who knows?

On the bright side, the read errors are fixed in beta 10. And I've made NVBackup copy itself to the card for good measure whenever one does a full or update backup. It's only a couple of more seconds, but should make recovery easier.

b10 is at handypalmstuff.sf.net

[pruss]

Actually I just got a reset loop when restoring data. Oops. Something's wrong. I suspect the decompression code. Renamed beta 10 to alpha 10.

[pruss]

beta 11 seems much better. Now out. Use at your own risk.

[willbewill]

It looks like some AirSAM* files (what are these?) were corrupt and were causing the reset loop.

I believe AirSAM files are used by Versamail?

[raptorNL]

Correct. What they do exactly, I'm not sure, but they are definitely related to VersaMail. Should have something to do with MS ActiveSync/Exchange.

[Greek]

b11 (reads b10 in the title and in the About) ran without problems in my TX. Same for Restore.

Regards,

[pruss]

It runs fine for me, too. And I've done some actual tests. (The Test Mode in Options makes NVBackup pretend that internal data is stored in /PALM_DM on the external card, and so you can feed data in and compare it with what goes out. I did a round trip test on a bunch of zero-filled and empty files, of different sizes, and things seemed to have worked fine so far. I may need to do some further testing, though. I haven't tried cases that correlate with read buffer size in nasty ways.)

[pruss]

Did some testing. Seems to pass them all.