Odd memory behaviour in BBEdit 8.7.2

[Clytie Siddall]

Hi everyone :)

I don't know if I've hit a memory ceiling with BBEdit, or whether it
should be able to handle very large text files.

I was editing my gettext Compendium today. A Compendium is a list of
all the unique strings you have translated. This file is over 30 MB in
size.

Although I was only selecting, deleting and adding small amounts of
text in this file, I experienced increasingly long save times, and
some of my simple operations also had a spinning beachball.

Peek-A-Boo, a system monitor program, kept showing virtual memory
thrashing.

Eventually, BBEdit told me it did not have enough memory to delete a
couple of lines of text. Console says:
___
24/03/08 6:56:33 PM [0x0-0x1f61f6].com.barebones.bbedit[54415]
BBEdit(54415,0xa012efa0) malloc: *** mmap(size=57163776) failed (error
code=12)
24/03/08 6:56:33 PM [0x0-0x1f61f6].com.barebones.bbedit[54415] ***
error: can't allocate region
24/03/08 6:56:33 PM [0x0-0x1f61f6].com.barebones.bbedit[54415] *** set
a breakpoint in malloc_error_break to debug
___

I can't find a separate log for BBEdit, so sorry I can't supply more
information. Console didn't seem to have anything else on its mind at
the time.

I had put off installing the latest lot of system updates until I
finished editing this file, so I'm only running the penultimate 10.5.2
at the moment on my MacBook.

...

Next day, after installing the system update and restarting my
MacBook, I have the same problem after editing (and searching) for a
while. I only have two copies of my large file open, in separate
windows. I have reduced the size of the file by editing it.
___
25/03/08 4:53:30 PM SpamSieve[449] MJTWorker.m:65 -workerThread:]
<MJTGrowlWorker: 0x18160550> process: NSCFDictionary
25/03/08 4:53:36 PM SpamSieve[449] MJTWorker.m:65 -workerThread:]
<MJTGrowlWorker: 0x18160550> process: NSCFDictionary
25/03/08 4:56:19 PM [0x0-0x1c01c].com.barebones.bbedit[156]
BBEdit(156,0xa04b7fa0) malloc: *** mmap(size=37347328) failed (error
code=12)
25/03/08 4:56:19 PM [0x0-0x1c01c].com.barebones.bbedit[156] *** error:
can't allocate region
25/03/08 4:56:19 PM [0x0-0x1c01c].com.barebones.bbedit[156] *** set a
breakpoint in malloc_error_break to debug
25/03/08 4:56:25 PM [0x0-0x1c01c].com.barebones.bbedit[156]
BBEdit[156]: [FC] Testing....70
25/03/08 4:56:25 PM [0x0-0x1c01c].com.barebones.bbedit[156]
BBEdit[156]: [FC] Corem;km;m;m;m;m;lm;lm <NULL>
25/03/08 4:57:18 PM [0x0-0x1c01c].com.barebones.bbedit[156]
BBEdit(156,0xa04b7fa0) malloc: *** mmap(size=37347328) failed (error
code=12)
25/03/08 4:57:18 PM [0x0-0x1c01c].com.barebones.bbedit[156] *** error:
can't allocate region
25/03/08 4:57:18 PM [0x0-0x1c01c].com.barebones.bbedit[156] *** set a
breakpoint in malloc_error_break to debug
___

I'm restarting BBEdit, to get it to reset its memory requirements.

...

Some editing later... the same problem:
___
25/03/08 10:10:45 PM [0x0-0x81081].com.barebones.bbedit[1686]
BBEdit[1686]: [FC] Testing....91
25/03/08 10:10:45 PM [0x0-0x81081].com.barebones.bbedit[1686]
BBEdit[1686]: [FC] Corem;km;m;m;m;m;lm;lm <NULL>
25/03/08 10:12:38 PM [0x0-0x81081].com.barebones.bbedit[1686]
BBEdit[1686]: [FC] Testing....92
25/03/08 10:12:38 PM [0x0-0x81081].com.barebones.bbedit[1686]
BBEdit[1686]: [FC] Corem;km;m;m;m;m;lm;lm <NULL>
25/03/08 10:13:01 PM [0x0-0x81081].com.barebones.bbedit[1686]
BBEdit[1686]: [FC] Testing....93
25/03/08 10:13:01 PM [0x0-0x81081].com.barebones.bbedit[1686]
BBEdit[1686]: [FC] Corem;km;m;m;m;m;lm;lm <NULL>
25/03/08 10:13:16 PM [0x0-0x81081].com.barebones.bbedit[1686]
BBEdit[1686]: [FC] Testing....94
25/03/08 10:13:16 PM [0x0-0x81081].com.barebones.bbedit[1686]
BBEdit[1686]: [FC] Corem;km;m;m;m;m;lm;lm <NULL>
25/03/08 10:13:19 PM [0x0-0x81081].com.barebones.bbedit[1686]
BBEdit(1686,0xa04b7fa0) malloc: *** mmap(size=37158912) failed (error
code=12)
25/03/08 10:13:19 PM [0x0-0x81081].com.barebones.bbedit[1686] ***
error: can't allocate region
25/03/08 10:13:19 PM [0x0-0x81081].com.barebones.bbedit[1686] *** set
a breakpoint in malloc_error_break to debug
___

Is there some way I can increase BBEdit's memory allocation? Or is
this a different problem?

Thanks for any help you can offer with this. :)

from Clytie

Vietnamese Free Software Translation Team
http://vnoss.net/dokuwiki/doku.php?id=projects:l10n

[Dennis]

Post by Clytie Siddall
I don't know if I've hit a memory ceiling with BBEdit, or whether it
should be able to handle very large text files.

From the BBEdit FAQ:

BBEdit and TextWrangler represent open documents as Unicode, which
uses two bytes for each character. Combined with an internal Mac OS X
limitation, this means that you can open files up to 384 megabytes in
size (402,653,184 characters).

Post by Clytie Siddall
I was editing my gettext Compendium today. A Compendium is a list of
all the unique strings you have translated. This file is over 30 MB
in size.
Although I was only selecting, deleting and adding small amounts of
text in this file, I experienced increasingly long save times, and
some of my simple operations also had a spinning beachball.

I've been doing some work today on a 352 MB file (over 5.6 million
lines and 369 million characters) and haven't seen any problems. The
file is slower to open and save, and some operations take a bit
longer, but for the most part BBEdit has been very responsive. In
fact, for most operations I've been using (column selections, Prefix/
Suffix Lines, Sort Lines, Quick Search, Copy/Paste, and lots of
scrolling) I can't even tell I'm working on an exceptionally large
file. Kudos to Bare Bones for creating such a reliable text editing
tool.

So I don't know what might be the cause of the problems you're seeing,
but I don't think it's a BBEdit file size limit. Does the problem
persist through a reboot of your machine?

-Dennis

[Clytie Siddall]

Dennis, thanks for your helpful reply. :)

Post by Dennis

Post by Clytie Siddall
I don't know if I've hit a memory ceiling with BBEdit, or whether
it should be able to handle very large text files.

Ouch. I should have read that. Sorry. Too much fog in my brain
nowadays. :(

Post by Dennis
BBEdit and TextWrangler represent open documents as Unicode, which
uses two bytes for each character. Combined with an internal Mac OS
X limitation, this means that you can open files up to 384 megabytes
in size (402,653,184 characters).

Even my compendium mightn't get that far. I do like a challenge. :D

Post by Dennis

Post by Clytie Siddall
I was editing my gettext Compendium today. A Compendium is a list
of all the unique strings you have translated. This file is over 30
MB in size.
Although I was only selecting, deleting and adding small amounts of
text in this file, I experienced increasingly long save times, and
some of my simple operations also had a spinning beachball.

I've been doing some work today on a 352 MB file (over 5.6 million
lines and 369 million characters) and haven't seen any problems. The
file is slower to open and save, and some operations take a bit
longer, but for the most part BBEdit has been very responsive. In
fact, for most operations I've been using (column selections, Prefix/
Suffix Lines, Sort Lines, Quick Search, Copy/Paste, and lots of
scrolling) I can't even tell I'm working on an exceptionally large
file. Kudos to Bare Bones for creating such a reliable text editing
tool.
So I don't know what might be the cause of the problems you're
seeing, but I don't think it's a BBEdit file size limit. Does the
problem persist through a reboot of your machine?

Yes, I said that in the original message. I had the problem, but
noticed (Software Update) that I had a system update to install, so
installed it via reboot. I continued to have the same problem.

Does anyone here understand what BBEdit is reporting in the Console?
Here's the most recent instance:
___
27/03/08 9:49:01 PM [0x0-0xa10a1].com.barebones.bbedit[2029]
BBEdit(2029,0xa04b7fa0) malloc: *** mmap(size=36990976) failed (error
code=12)
27/03/08 9:49:01 PM [0x0-0xa10a1].com.barebones.bbedit[2029] ***
error: can't allocate region
27/03/08 9:49:01 PM [0x0-0xa10a1].com.barebones.bbedit[2029] *** set a
breakpoint in malloc_error_break to debug
___

It looks like memory allocation to me, but that's all I can work out.

Rebooting BBEdit fixes the problem for a while.

from Clytie

Vietnamese Free Software Translation Team
http://vnoss.net/dokuwiki/doku.php?id=projects:l10n

[Aaron Hsu]

Hey There,

--On March 28, 2008 4:35:52 PM +1030 Clytie Siddall

Post by Clytie Siddall
Does anyone here understand what BBEdit is reporting in the Console?

You may have answered this already, but just exactly how much memory is
BBEdit using when you start, and when you start to open this file and
during? Also, how much system memory do you have, and how much is used?
--
Aaron Hsu <***@sacrideo.us> | Jabber: ***@jabber.org
``Government is the great fiction through which everybody endeavors to
live at the expense of everybody else.'' - Frederic Bastiat

[Clytie Siddall]

Hi Aaron (and list) :)

Post by Aaron Hsu
Hey There,

Post by Clytie Siddall
Does anyone here understand what BBEdit is reporting in the Console?

You may have answered this already, but just exactly how much memory
is BBEdit using when you start, and when you start to open this file
and during? Also, how much system memory do you have, and how much
is used?

I didn't mention it, sorry, because I have heaps of memory. BBEdit,
despite its complaints, appears to be using only a small amount of
memory, while I have 2 GB RAM on my MacBook. Lots of free RAM. At
least half of my 180 GB hard disk is also free.

So it isn't a case of not enough memory, but of somehow not being able
to use it. :S

I've repaired permissions a second time, and so far today haven't had
any problems with the same file. So maybe that sorted it out, somehow.

<grump>I don't like these problems that seem to fix themselves: I
prefer to know what is actually wrong.</grump>

Thanks for your help. :)

from Clytie

Vietnamese Free Software Translation Team
http://vnoss.net/dokuwiki/doku.php?id=projects:l10n

[Richard Carr]

I develop and maintain a product that has well over 200 source
modules. I program in a proprietary language and don't have a source
control system; BBEdit is a wonderful tool for managing all this.

One small nit: When I update and save a source module in BBEdit, the
"Date Modified" doesn't get updated in the Finder window. I always
keep my Finder window sorted by Date Modified to find the latest
updated modules for FTP'ing to a host, and just for knowing what I've
changed recently.

But, for some reason, there has always been a significant delay in
updating the dates on BBEdit files. I'm not even sure what triggers
the update.

But I do see that if I update a Microsoft Word document, the change
is reflected immediately in the Finder window. What's the difference
between BBEdit and Word? Is there anything I can do?

Regards,
Richard Carr

[Doug McNutt]

I develop and maintain a product that has well over 200 source modules. I program in a proprietary language and don't have a source control system; BBEdit is a wonderful tool for managing all this.
One small nit: When I update and save a source module in BBEdit, the "Date Modified" doesn't get updated in the Finder window. I always keep my Finder window sorted by Date Modified to find the latest updated modules for FTP'ing to a host, and just for knowing what I've changed recently.
But, for some reason, there has always been a significant delay in updating the dates on BBEdit files. I'm not even sure what triggers the update.
But I do see that if I update a Microsoft Word document, the change is reflected immediately in the Finder window. What's the difference between BBEdit and Word? Is there anything I can do?

AppleScript

tell window "name" of application "Finder"
update every file
end tell

Finder is notorious for being slow to update windows. It started with OS neXt. I usually have problems with things like curl and it's easy to run the AppleScript using the osascript tool with a << "here" document that contains the script.
--
--> From the U S of A, the only socialist country that refuses to admit it. <--

[Johan Solve]

Post by Doug McNutt

But I do see that if I update a Microsoft Word document, the change is reflected immediately in the Finder window. What's the difference between BBEdit and Word? Is there anything I can do?

AppleScript
tell window "name" of application "Finder"
update every file
end tell

The question about what Word does that BBEdit doesn't is certainly valid.

But as BBEdit's menus are attachable it is possible to attach a variant of the above script to the Save menu command, to force Finder to update the folder window where the document is.

This will work when using the File -> Save command (or keyboard shortcut), but not when saving the file through the window palette or the document drawer.
--
Johan Sölve [FSA Member, Lasso Partner]
Web Application/Lasso/FileMaker Developer
MONTANIA SOFTWARE & SOLUTIONS
http://www.montania.se mailto:joh-***@montania.se
(spam-safe email address, replace '-' with 'a')

[Richard Carr (Carr Scott Software)]

I created a new user on my Mac and tried to copy over all the prefs,
but didn't succeed.

I can't get CMD-H to mean "Find Selected Text" instead of Hide. Does
anyone remember the secret?

Thanks,
Richard

[Dennis Cox]

Check the Menus section of the Preferences.


On 4/8/08 1:25 AM, "Richard Carr (Carr Scott Software)"

Post by Richard Carr (Carr Scott Software)
I created a new user on my Mac and tried to copy over all the prefs,
but didn't succeed.
I can't get CMD-H to mean "Find Selected Text" instead of Hide. Does
anyone remember the secret?
Thanks,
Richard

[Richard Carr (Carr Scott Software)]

I knew it was in there somewhere! Thank you!

Post by Dennis Cox
Check the Menus section of the Preferences.
On 4/8/08 1:25 AM, "Richard Carr (Carr Scott Software)"

Post by Richard Carr (Carr Scott Software)
I created a new user on my Mac and tried to copy over all the prefs,
but didn't succeed.
I can't get CMD-H to mean "Find Selected Text" instead of Hide. Does
anyone remember the secret?
Thanks,
Richard

--
------------------------------------------------------------------
Have a feature request? Not sure the software's working correctly?
List FAQ: <http://www.barebones.com/support/lists/bbedit_talk.shtml>
List archives: <http://www.listsearch.com/BBEditTalk.lasso>

[James Madill]

Post by Richard Carr (Carr Scott Software)
I created a new user on my Mac and tried to copy over all the prefs,
but didn't succeed.
I can't get CMD-H to mean "Find Selected Text" instead of Hide. Does
anyone remember the secret?

BBEdit --> Preferences, then select Menus. There, choose the Search
Menu twistie to access the "Find Selected Text" shortcut definition.

Post by Richard Carr (Carr Scott Software)
Thanks,
Richard
--
------------------------------------------------------------------
Have a feature request? Not sure the software's working correctly?
List FAQ: <http://www.barebones.com/support/lists/bbedit_talk.shtml>
List archives: <http://www.listsearch.com/BBEditTalk.lasso>

James Madill
***@madill.us

[Joe Chellman]

Post by Clytie Siddall
Although I was only selecting, deleting and adding small amounts of
text in this file, I experienced increasingly long save times, and
some of my simple operations also had a spinning beachball.

The first thing I thought of, at least as far as performance goes
with large files, is whether you have soft wrap turned OFF for the
big files. The performance difference is pretty substantial. I
don't know if it will help you here, but it might be worth a shot.

As for all the console messages, I have no idea, but you should
definitely have emailed support about this. Have you done that?

Joe
--
Joe Chellman
http://www.chellman.org/
http://www.chellman.org/pgp_key.txt