[development] Solving the dev->staging->live problem

[Sam Boyer]

On Saturday 09 August 2008 15:43:35 Adrian Rossouw wrote:
> On 09 Aug 2008, at 10:31 PM, arthur wrote:
> > That being said, I think it's pretty clear that many people in the
> > Drupal community are really interested in a solution for this. A
> > sense of agreement of how this problem could be tackled could really
> > foster progress. Perhaps I'm being naive, but it does seem like some
> > of the tools that various folks in the community are using are
> > getting closer to being quite functional for lots of people. With
> > some focused development time (a sprint perhaps?), it seems as
> > though some of the more daunting pieces (reverting changes for
> > example) might be approachable.
>
> indeed, which is why we are having this discussion.
>
> > Anyway, I was just hoping to bring some awareness to one of the
> > tools that does exist now, and could potentially serve as a model to
> > build from. Of course, I'm not even the author, so maybe Greg should
> > chime in :)
>
> If we intend to build from it, we need to identify the issues relating
> to it, and the work required to make it work perfectly, which is all I
> stated.
>
> We need to inspect all the ways people are currently working around
> this, so we can choose the best mechanism to focus on.

Given that heyrocker/Greg is now at Palantir, deploy is on our minds...let me 
try to not steal any of his thunder :)

So we've talked a bit about deploy and this general problem of maintaining 
multiple different instances of a given site, where those instances may have 
different purposes/states within the overall workflow; allowing for data & 
changes to flow to/from/around those various different instances is the killer 
problem. I think we came up with a 'holy grail' that had dev state(s), then 
qa, staging, and prod. Primary keys are the big killer there, of course, and 
while I have considerable admiration for the folks who've come up with working 
models for ensuring no primary key conflicts (which afaik are either even/odd 
or start-at-key-X approaches), I'd hope there's agreement that we could do 
better. We've batted some thoughts around, but I'll leave it up to Greg on 
whether or not he feels like they're ripe for public digestion yet - this is 
his baby =)

I will say, though, that there's a basic approach present in deploy which 
_does_ have the potential to become a systemic solution to this problem (IMO): 
it's pluggable. Instead of trying to figure out what data should or shouldn't 
be transferred (which is essentially what we're doing when we pick out parts 
of our db dumps to push/pull), it just presents an API and lets the modules 
implementing it decide the logic that ought to govern synchronizing data 
between various site states. That, to me, seems like the approach most likely 
to end with positive results: deploy abstracts synchronization complexities 
into a simple API. Modules implementing it just have to answer some fairly 
basic questions about how their data is structured, and deploy takes care of 
the rest.

Sam