Serious TX (and others?) bug

[pruss]

Over the past week or so, I've been having mysterious problems on my TX. Finally, after some emails with Dmitry, it became clear. Because of the way NVFS is implemented, on the TX it is impossible to have more than 1200 databases in Device Memory (i.e., more than 1200 files in \PALM_DM on the BUILTIN drive). This is basically a bug--they could have allocated memory using a different API, and then all would have been well. There does not appear to be any easy software way out.

It does not matter how much Device Memory you have free. I have 23% free. It's the number of databases, not the amount of space taken up that is the issue.

Once the limit is reached, creating a new database will fail AND various other stability problems will result.

The fix to this would be easy if one had source code to the OS. In other words, Palm can easily fix this. If they want to.

If Palm had made this limit public, then developers might have tried to minimize the number of databases that their apps generate. For instance, at least one ebook reader app (I think eReader) generates TWO extra databases for each ebook. Plucker generates one. All of these could be collated by the developers into a single database.

Question: How does one report bugs to Palm?

[Shrink]

Question: How does one report bugs to Palm?
Reporting is easy, getting them to read and pay attention to your report is impossible.
alan

[pruss]

Reporting is easy, getting them to read and pay attention to your report is impossible.
alan

Is there an email address where one can try? Or does anyone have a friendly Palm contact?

[janos]

Is there an email address where one can try? Or does anyone have a friendly Palm contact?

No chance, Alex. (This is also my reply to your e-mails.)

There are several similar bugs/features that are here since ages with no chance to correct anything.

This one is particularly nice:

If an app registers as a handler for some file type (e.g. a viewer of .jpg files) and if this app gets deleted (or isntalled to the card) without removing the registration, then

1. One of the API calls related to the registrations (don't remember which one by heart) fails. That's ok, but Palm OS forgets to close the prefs database.

2. Any subsequent trial (usually fully unrelated) to open preferences produces a fatal alert.

If you looked into Palm OS 4 sources, you would identify the bug within a few minutes. The correction would not take longer either.

We were among the first ones trying to alert Palm. Since then many other developers discovered it, too.

Today status:
Ben Combee wrote an article how to avoid this problem - simply apps have to react to the card installation.

Several apps contain specific code dealing with the problem. We did it in Explorer, later Reset Doctor implemented a similar strategy and maybe some other apps. I guess Rob even wrote an article on Wikipedia.

Our support costs due to this "feature" were quite considerable. Plus many users must have been angry with Resco developers for getting these alerts.

Palm must have suffered, too.

Despite this all the bug was never corrected and successfuly persists since the Palm OS 4 years.

Catch XXII

[alt236]

If Palm had made this limit public, then developers might have tried to minimize the number of databases that their apps generate.

The limit is public. At least, it's in the TX manual (page 609 on paper - 631 in the pdf).
But Palm did not state it anywhere else as far as I have seen nor was there a developers bulletin of some sort regarding it...

[pruss]

Hey, you're right. Since it's documented, I guess it's not a bug. It is a silly limit, though. And I doubt the OS gracefully handles the limit.

[pruss]

Our support costs due to this "feature" were quite considerable. Plus many users must have been angry with Resco developers for getting these alerts.

Maybe then Resco should include a patch for this? (Maybe even build it into products.) Or is it too hard to patch?

[alt236]

Hey, you're right. Since it's documented, I guess it's not a bug. It is a silly limit, though. And I doubt the OS gracefully handles the limit.

Indeed, it acts unpredictably. And of course, there is no descriptive error message telling you that you have reached the limit.

At best, you will get a generic "memory full" message every time an app tries to create a new db - and then you will usually get a crash (ok, that's because most apps don't handle the possibility of not being able to create a db).

At worst, you just get semi-random crashes with no explanation.

[pruss]

Indeed, it acts unpredictably. And of course, there is no descriptive error message telling you that you have reached the limit.

At best, you will get a generic "memory full" message every time an app tries to create a new db - and then you will usually get a crash (ok, that's because most apps don't handle the possibility of not being able to create a db).

Some apps, not unreasonably, assume that if there is free memory space, then a db can be created. :-)

[alt236]

It does make sense and it is a valid assumption :)
The only other common fs that can throw something similar to you (under normal use) is FAT with the 512 file limit at the root dir.

Its just that, as you said, its a silly bug and for a non-expert user it can be quite frustrating especially since the error message is so vague...

[nepacsman]

Thanks for bringing this to the community's attention. Is there a way to test a non-Tx device like a Treo to see if it has the same problem? I am a heavy treo 755 user and would like to know in advance if I will have this problem.

[I love my mom]

I used to have that problem, however, I cleaned out several hundred files, and programs with lots of DBs I can always store on the card with something such as ZLauncher, moving them to RAM only when I need to.

You could write a patch for DmFindDatabase, etc. that would allow you to search one folder on the memory card for the file and move it to memory if available. This would, as you know, cause large speed problems though.

[alt236]

Ok, I wrote an app which creates db's until an error is encountered.

Now, the limit seems to be relatively soft, e.g. its not always on the 1200 mark.
I've gotten it on the 1211 and the 1264 count (although 1211 was more common).

Sometimes, the app will catch the error (I will get my own error dialogue), and sometimes it will not (the system will pre-empt my error handling).

In addition, many times after I got the error message and I tried to make new db's my calls were successful although no db was created.

So "unpredictable" is the word of the day...

[pruss]

The limit is 1200 files in NVFS. It may be that the databases you create are stored in RAM, where there is no limit. The problem occurs once they're stored to flash.

[pruss]

Dmitry's suggestion to me was to archive (uncompressed for speed) databases for the problematic apps. Extract them when the app runs, and then re-archive them afterwards.

[alt236]

The limit is 1200 files in NVFS. It may be that the databases you create are stored in RAM, where there is no limit. The problem occurs once they're stored to flash.
Ah, yes. That should be it.

Dmitry's suggestion to me was to archive (uncompressed for speed) databases for the problematic apps. Extract them when the app runs, and then re-archive them afterwards.
If the heavy part of the (de)compression is done in arm the its should be fine. Otherwise for large files the performance hit would be noticeable...
Also, this method could break hotsync unless you decompress everything when you get the hotsyncstart signal -which would also be noticable for large files...

EDIT: But yes, I can't see any alternative unless we swap files from card which could be even slower.

[PalmSole]

I guess this will force some of us to do some spring cleaning: more often for some than others :eek: Thanks for bringing this to the attention of the masses.

[_Em]

Anyone up for writing a background app/prefs panel that alerts you when you're getting close to the limit? Could be a useful quickfix until someone comes up with something better (like Dmitry patching it with his filesystem driver ;) )