I agree with Greg's and others remarks about documentation and showing difference with other modules... and you can see this is already a fascinating discussion because it adressess so many needs.<br><br>But the bottom line is you should not feel you have to jump through hoops to post an early version! Go ahead and refactor later.<br>
<br>Victor<br><br><div class="gmail_quote">On Sat, Aug 1, 2009 at 12:47 PM, Pierre Rineau <span dir="ltr"><<a href="mailto:pierre.rineau@makina-corpus.com">pierre.rineau@makina-corpus.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Oh, and I should precise, it may won't be released until we deliver our<br>
project to client (which is, if I remember well maybe in over a month).<br>
Because their needs can evolve, some architectural choices may too.<br>
<br>
But don't worry, any changes on what I explained shouldn't be more that<br>
some higher abstraction level and new features.<br>
<br>
To answer about documentation, I'll try to write some and also give some<br>
diagrams, but as DataSync relies on the system, it may be not that easy<br>
to install. I'll see in some days when I'll start deploying it on a<br>
preprod environment.<br>
<font color="#888888"><br>
Pierre.<br>
</font><div class="im"><br>
On Sat, 2009-08-01 at 15:50 +0200, Pierre Rineau wrote:<br>
</div><div><div></div><div class="h5">> Hello all,<br>
><br>
> Working on some custom project for my company, I developed a module to<br>
> do massive migration between sites.<br>
><br>
> This module uses a full OO layer.<br>
><br>
> Its internal mechanism is based on abstracting objects to migrate from a<br>
> master site to clients. This abstraction defines how to construct object<br>
> dependency tree and how to serialize objects.<br>
><br>
> Object implementation (node, user, taxonomy, whatever) is really simple<br>
> to use, it's only 3 methods classes (register dependencies, save, and<br>
> update) using some kind of custom registry for developer to save/get<br>
> back data before and after serialization.<br>
><br>
> All error handling is exception oriented, and lower software layers<br>
> won't fail on higher layers unrecoverable errors.<br>
><br>
> Object fetching is based on a push/pull mechanism. Server push the sync<br>
> order, client responds OK or not. If OK, it creates a job using DataSync<br>
> module which allow it to run as CLI thread (which won't hurt the web<br>
> server, and allow us a larger memory limit at run time). DataSync module<br>
> uses MySQL transactions (shame it's only MySQL compliant, but I hope it<br>
> will evolve, I'm thinking about PostgreSQL).<br>
><br>
> During the DataSync job execution, client will pull an original set of<br>
> content, and browsing it will do incremental dependencies fetching (by<br>
> pulling again server), based on xmlrpc (fetching component is also<br>
> abstracted, and could be any other communication method than xmlrpc).<br>
><br>
> To be unobtrusive on the system, smart unset() is done after building a<br>
> dependencies subtree, and there is a recursion breaker in case of<br>
> circular dependencies.<br>
><br>
> This module was created because the deploy module seems to be so<br>
> unstable, I did not want to risk client's production sites to run with<br>
> it. I started implementation of some sort of "deploy plan", using<br>
> profile based on views, you construct a set of views, saved them in a<br>
> profile, then all objects that these views reference will be<br>
> synchronized.<br>
><br>
> Right now, the module fully synchronize taxonomy and content types,<br>
> partially synchronize users (including core account information and<br>
> passwords), and I have a small bug left to handle with nodes (revision<br>
> problem I think).<br>
><br>
> There might be a performance or overhead problem with this conception<br>
> with a very large amount of data, it could break easily. The only way to<br>
> be sure it won't break is I think to migrate stuff with a numerous small<br>
> set of data. But the problem doing this is that it will be really hard<br>
> to keep the transactional context of DataSync module.<br>
><br>
> There is a lot of other custom goodies coming.<br>
><br>
> First thing is, what do you think about such module, should I commit it<br>
> on <a href="http://drupal.org" target="_blank">drupal.org</a>? Is there people interested?<br>
><br>
> And, now that I described the module, what name should I give him,<br>
> considering the fact I'll probably commit it on <a href="http://drupal.org" target="_blank">drupal.org</a>, if people<br>
> are interested.<br>
><br>
> I though about "YAMM" (Yet Another Migration Module), or YADM (Yet<br>
> Another Deployment Module).<br>
><br>
> The fact is there is *a lot* of modules which want to do the same thing<br>
> as this one, I just want a simple an expressive name.<br>
><br>
> Pierre.<br>
><br>
<br>
</div></div></blockquote></div><br>