[development] why locale import is slow and a memory eater?

[Gabor Hojtsy]

Dries Buytaert wrote:
>> And here comes the question. Now, if a PO file is broken for any  reason,
>> the real SQL import process is skipped. But whether the file is broken
>> at some point is only possible to tell, if we parse the file up to  that
>> point. If the SQL changes are made incrementally, then the PO file  would
>> end up half-imported. Question is if it is a problem or not. I think
>> this is a minor problem, since you are supposed to upload a  translation
>> which you are about to use, so if your po file is broken it is not a
>> problem that it is half imported, you should fix and try again.
> 
> Good analysis; it matches my own observations.  I don't think a  partial
> import is a problem.  Quite the contrary; I'd rather have 99%  of the
> strings translated than 0% of the strings.  I could translate  the
> offending strings manually, or if I'm lucky, the offending  strings are
> only accessible for the administrator, or they are part  of some
> disabled feature(s).

Ok. Just some quick data on how much an import eats now with the
upcoming Drupal 4.7.0 Hungarian translation (now 357899 bytes):

in _locale_import_read_po():

  memory before read:               5923056
  memory after read:                6281152
  memory after split to lines:      7299384
  memory after $strings generation: 8616160

Back in _locale_import_po(), which calls the above function:

  memory after return: 7285416

This shows that reading in the ~350k file results in roughly that size
of memory increase, but then splitting it results in a megabyte of added
memory usage (this is the overhead of the array), then the structured
storage takes 1.4M (it is a bit of more complicated array). After
return, it is clearly visible that the 1.4M structure is kept, the
others are wiped, resulting in 7.2M.

Goba