[development] why locale import is slow and a memory eater?
gabor at hojtsy.hu
Sat Dec 31 13:17:23 UTC 2005
Dries Buytaert wrote:
>> And here comes the question. Now, if a PO file is broken for any reason,
>> the real SQL import process is skipped. But whether the file is broken
>> at some point is only possible to tell, if we parse the file up to that
>> point. If the SQL changes are made incrementally, then the PO file would
>> end up half-imported. Question is if it is a problem or not. I think
>> this is a minor problem, since you are supposed to upload a translation
>> which you are about to use, so if your po file is broken it is not a
>> problem that it is half imported, you should fix and try again.
> Good analysis; it matches my own observations. I don't think a partial
> import is a problem. Quite the contrary; I'd rather have 99% of the
> strings translated than 0% of the strings. I could translate the
> offending strings manually, or if I'm lucky, the offending strings are
> only accessible for the administrator, or they are part of some
> disabled feature(s).
Ok. Just some quick data on how much an import eats now with the
upcoming Drupal 4.7.0 Hungarian translation (now 357899 bytes):
memory before read: 5923056
memory after read: 6281152
memory after split to lines: 7299384
memory after $strings generation: 8616160
Back in _locale_import_po(), which calls the above function:
memory after return: 7285416
This shows that reading in the ~350k file results in roughly that size
of memory increase, but then splitting it results in a megabyte of added
memory usage (this is the overhead of the array), then the structured
storage takes 1.4M (it is a bit of more complicated array). After
return, it is clearly visible that the 1.4M structure is kept, the
others are wiped, resulting in 7.2M.
More information about the development