Page MenuHomePhabricator

Add recipes for migrating downstream branches of git mirrors
ClosedPublic

Authored by greened on Jan 10 2019, 8:56 AM.

Details

Summary

Add some common recipes for downstream users developing on top of the existing git mirrors. These instructions show how to migrate local branches to the monorepo.

Diff Detail

Repository
rL LLVM

Event Timeline

greened created this revision.Jan 10 2019, 8:56 AM
danilaml added inline comments.Jan 10 2019, 9:23 AM
docs/Proposals/GitHubMove.rst
879 ↗(On Diff #181074)

upstreamhttps

908 ↗(On Diff #181074)

Perhaps adding somewhere before or after this elaborate explanation that the script also supports migrating other types of mirrors/repos would be enough to cover those cases (like migrating lnt/test-suite/etc forks). It was very straight--forward in my experience so built in help should be enough.

I think this complex task of migrating a downstream fork with important history should go into its own document, with just a reference here. It's very important for those few who need to do it, but I don't think it'll be something that a large number of people will need to do.

I would expect the *most common* case is just:

There's some outstanding uncommitted work in:

  • an SVN checkout, or
  • a git clone of the old separate repositories with one or more branches, or
  • a git clone of the old monorepo with one or more branches

and I want to keep rebasing it onto master until I can get it committed.

These migration tools are rather scary and overkill for that sort of use.

greened updated this revision to Diff 181083.EditedJan 10 2019, 10:19 AM

Noted that when zipping, any non-LLVM repositories added as submodules need to be imported into the monorepo.

Added instructions to clean up the final monorepo.

greened marked an inline comment as done.Jan 10 2019, 10:20 AM
greened marked an inline comment as done.
greened added inline comments.
docs/Proposals/GitHubMove.rst
908 ↗(On Diff #181074)

That's a good idea. I haven't tackled test-suite and the like yet.

I could certainly put these instructions in a separate document, but where should it go? This is a one-time operation and so it doesn't make sense to put it in with the regular LLVM docs. It's not a proposal, per se, so I don't think it belongs in docs/Proposals other than as an addition to a related proposal, which is this document.

greened added inline comments.Jan 10 2019, 10:24 AM
docs/Proposals/GitHubMove.rst
879 ↗(On Diff #181083)

This should reference the final URL of the monorepo, once it's been blessed.

I could certainly put these instructions in a separate document, but where should it go? This is a one-time operation and so it doesn't make sense to put it in with the regular LLVM docs. It's not a proposal, per se, so I don't think it belongs in docs/Proposals other than as an addition to a related proposal, which is this document.

I dunno. :) If you can't find another place to put it, I'm fine to have it here for the moment; I don't want to block on perfection, we can rearrange things later on.

Do a lot of people use the old monorepo for development? We don't so I haven't developed any process for migrating from that. I'll leave that to someone else to add.

Migrating from SVN is way more painful and again, I'll leave that to someone else to document. :)

Obviously I'm biased, but I don't think these tools are all that scary. git-rebaseing multiple downstream merge commits on top of upstream master is way more scary to me. You need a relatively new git to do it properly and even then I don't know how well it's tested. If the downstream branches don't have any merges it's much easier but how likely is that? We merge from upstream regularly and I know of other downstream users that do the same.

Maybe I'm misreading the zeitgeist but as @bogner suggested in D53414, downstream branches with merges is almost certainly the most likely scenario.

jyknight added a comment.EditedJan 10 2019, 10:44 AM

Do a lot of people use the old monorepo for development? We don't so I haven't developed any process for migrating from that. I'll leave that to someone else to add.

Migrating from SVN is way more painful and again, I'll leave that to someone else to document. :)

Obviously I'm biased, but I don't think these tools are all that scary. git-rebaseing multiple downstream merge commits on top of upstream master is way more scary to me. You need a relatively new git to do it properly and even then I don't know how well it's tested. If the downstream branches don't have any merges it's much easier but how likely is that? We merge from upstream regularly and I know of other downstream users that do the same.

Maybe I'm misreading the zeitgeist but as @bogner suggested in D53414, downstream branches with merges is almost certainly the most likely scenario.

Oh yes, it's 100% the most likely scenario for downstream forks with history (e.g. shared internally to others in their org). But what I mean is that there's a lot of individual developers with their own private work-in-progress local *UNSHARED* repositories and checkouts, which will likely have no (important/intentional) merge commits because they're just working on patches to submit upstream.

Those migrations are going to be the most common, and such people shouldn't think they need to use these migration tools when a simple git rebase or format-patch/apply would likely be more appropriate.

greened updated this revision to Diff 181102.Jan 10 2019, 10:52 AM

Fixed various typos.

Oh yes, it's 100% the most likely scenario for downstream forks with history (e.g. shared internally to others in their org). But what I mean is that there's a lot of individual developers with their own private work-in-progress local *UNSHARED* repositories and checkouts, which will likely have no (important/intentional) merge commits because they're just working on patches to submit upstream.

Those migrations are going to be the most common, and such people shouldn't think they need to use these migration tools when a simple git rebase or format-patch/apply would likely be more appropriate.

Ah, yes, I completely agree with that. Our developers here are going to want to do that for their own local branches (or we could just force them to push before we migrate) so I have some need to develop a process for them.

fedor.sergeev added inline comments.Jan 10 2019, 11:44 AM
docs/Proposals/GitHubMove.rst
970 ↗(On Diff #181102)

typo: octpous

Thanks for the effort!

docs/Proposals/GitHubMove.rst
1173 ↗(On Diff #181102)

locaol->local

1203 ↗(On Diff #181102)

I dont understand semantics of this line.
$git_tool ?
! run ?

greened updated this revision to Diff 181134.Jan 10 2019, 12:53 PM

Fixed more typos.

greened marked 4 inline comments as done.Jan 10 2019, 12:54 PM
greened added inline comments.
docs/Proposals/GitHubMove.rst
1203 ↗(On Diff #181102)

Heh. Copied a little too literally from my personal script. :)

greened updated this revision to Diff 181136.Jan 10 2019, 12:57 PM
greened marked an inline comment as done.

Updated with blessed monorepo URL.

greened marked an inline comment as done and an inline comment as not done.Jan 10 2019, 12:58 PM

Is there anything substantial left here?
Since we already have a blessed repo it makes sense to push this, even if it is not 200% ready.

docs/Proposals/GitHubMove.rst
914 ↗(On Diff #181136)

Does it make sense to add --source-kind=split argument to migrate-downstream-fork.py here?
Just in case that it fails to detect ...

Anyway, just checked this recipe on our simple downstream repo (that is a split kind that has only llvm project in it).
Worked fluently for me. Thanks to all folks involved!

fedor.sergeev added inline comments.Jan 18 2019, 10:03 AM
docs/Proposals/GitHubMove.rst
1092 ↗(On Diff #181136)

You forgot the closing ')' before ';' - for git for-each-ref invocation.

greened updated this revision to Diff 183942.Mon, Jan 28, 12:41 PM

Updated for new and improved zipping tool.

greened updated this revision to Diff 183944.Mon, Jan 28, 12:45 PM

Fix typos

greened marked an inline comment as done.Mon, Jan 28, 12:45 PM
greened updated this revision to Diff 184094.Tue, Jan 29, 8:03 AM

Added --source-kind=split to the migrate-downstream-fork.py example.

greened marked an inline comment as done.Tue, Jan 29, 8:03 AM
greened updated this revision to Diff 184095.Tue, Jan 29, 8:15 AM

Note important considerations with respect to upstream merges after zipping.

fedor.sergeev accepted this revision.Mon, Feb 4, 2:17 AM

So, what are the plans with this?
I'm happy with it as is.

This revision is now accepted and ready to land.Mon, Feb 4, 2:17 AM
Herald added a project: Restricted Project. · View Herald TranscriptMon, Feb 4, 2:17 AM
This revision was automatically updated to reflect the committed changes.