[development] why locale import is slow and a memory eater?

Gabor Hojtsy gabor at hojtsy.hu
Sat Dec 31 13:17:23 UTC 2005

Dries Buytaert wrote:
>> And here comes the question. Now, if a PO file is broken for any  reason,
>> the real SQL import process is skipped. But whether the file is broken
>> at some point is only possible to tell, if we parse the file up to  that
>> point. If the SQL changes are made incrementally, then the PO file  would
>> end up half-imported. Question is if it is a problem or not. I think
>> this is a minor problem, since you are supposed to upload a  translation
>> which you are about to use, so if your po file is broken it is not a
>> problem that it is half imported, you should fix and try again.
> Good analysis; it matches my own observations.  I don't think a  partial
> import is a problem.  Quite the contrary; I'd rather have 99%  of the
> strings translated than 0% of the strings.  I could translate  the
> offending strings manually, or if I'm lucky, the offending  strings are
> only accessible for the administrator, or they are part  of some
> disabled feature(s).

Ok. Just some quick data on how much an import eats now with the
upcoming Drupal 4.7.0 Hungarian translation (now 357899 bytes):

in _locale_import_read_po():

  memory before read:               5923056
  memory after read:                6281152
  memory after split to lines:      7299384
  memory after $strings generation: 8616160

Back in _locale_import_po(), which calls the above function:

  memory after return: 7285416

This shows that reading in the ~350k file results in roughly that size
of memory increase, but then splitting it results in a megabyte of added
memory usage (this is the overhead of the array), then the structured
storage takes 1.4M (it is a bit of more complicated array). After
return, it is clearly visible that the 1.4M structure is kept, the
others are wiped, resulting in 7.2M.


More information about the development mailing list