Opened 11 years ago

Closed 7 years ago

#621 closed task (fixed)

Migrate to git

Reported by: Olly Betts Owned by: Olly Betts
Priority: normal Milestone: 1.4.1
Component: Other Version:
Severity: normal Keywords:
Cc: Blocked By:
Blocking: Operating System: All

Description (last modified by Olly Betts)

Still to sort out:

  • xapian-maintainer-tools/make-xapian-svn-snapshot-tarballs
  • xapian-maintainer-tools/svn-tag-release
  • Trac - enable git integration
  • Website
    • /bleeding
    • links to files in SVN via trac.xapian.org/browser
  • Buildbot
  • Make SVN repo read-only
  • Enable pushing to master git repo
  • Dismantle SVN to git mirroring machinery

Once we've moved, we can drop handling for SVN from:

  • bootstrap (needs pushing)
  • xapian-core/HACKING (git-svn requirement, etc)
  • website (/bleeding)

Also to update:

  • debian packaging - e.g. Vcs-Browser and Vcs-svn in control files

Miscellaneous:

  • Trac - currently we're patching 1.0.2 - we should sort out feeding these upstream, or doing them via a plugin (if that's possible):
    • Allowing r12345 and [12345] to keep working for SVN, while allowing [103357b] to work for git
    • Allow 7 hex digits for a git commit hash (as that's what git's abbreviated commit hash is by default) - trac requires 8 by default: http://trac.edgewall.org/ticket/11992
  • Rename svn-ci to xapian-commit
  • Consider a switch to gitolite or Simon Tatham's stuff
  • Migrate website from CVS (!) to git
  • Hook up git commits to xapian-commits list? Is this still useful? Nobody seems to think so~~

Change History (27)

comment:1 by Olly Betts, 11 years ago

Description: modified (diff)

git-tag-release written based on svn-tag-release (but not yet tested).

git equivalent of make-xapian-svn-snapshot-tarballs close to working.

comment:2 by Olly Betts, 11 years ago

Description: modified (diff)

comment:3 by Olly Betts, 11 years ago

Description: modified (diff)

comment:4 by Olly Betts, 11 years ago

Description: modified (diff)

comment:5 by Olly Betts, 11 years ago

Description: modified (diff)

comment:6 by Olly Betts, 10 years ago

Milestone: 1.3.21.3.3
Status: newassigned

comment:7 by Olly Betts, 10 years ago

Description: modified (diff)

comment:8 by Olly Betts, 9 years ago

Description: modified (diff)

Trac git support enabled

comment:9 by Olly Betts, 9 years ago

Description: modified (diff)

Migrated buildbot in r18372.

comment:10 by Olly Betts, 9 years ago

Description: modified (diff)

comment:11 by Olly Betts, 9 years ago

Description: modified (diff)

comment:12 by Olly Betts, 9 years ago

Description: modified (diff)

comment:13 by Olly Betts, 9 years ago

Description: modified (diff)
Owner: changed from Olly Betts to James Aylett
Status: assignednew

The next step which needs doing involves adding users to a group on atreus, so reassigning to james for that.

comment:14 by James Aylett, 9 years ago

Status: newassigned

comment:15 by James Aylett, 9 years ago

Description: modified (diff)
Owner: changed from James Aylett to Olly Betts
Status: assignednew

Pushing to git for atreus users via the xapian-git group should now work; reassigning to Olly.

comment:16 by Olly Betts, 9 years ago

Description: modified (diff)
Status: newassigned

Works after a chmod -R g+w of the relevant files.

Longer term we may want to move to using gitolite or Simon's stuff to provide a bit more control.

comment:17 by Olly Betts, 9 years ago

Description: modified (diff)

comment:18 by Olly Betts, 9 years ago

Description: modified (diff)

Pushed the patch for 7 character abbreviated git commit hashes upstream.

comment:19 by Olly Betts, 9 years ago

Milestone: 1.3.31.3.4

comment:20 by Olly Betts, 9 years ago

Description: modified (diff)

The website is now in git, hosted on git.xapian.org and mirrored to github. I noted down what I had to do to achieve this in README in the xapian-git directory on atreus, so it'll be easier to repeat.

Last edited 9 years ago by Olly Betts (previous) (diff)

comment:21 by Olly Betts, 9 years ago

Description: modified (diff)

xapian-commits is no longer getting traffic. Not sure if it actually still useful, but we should at least email it to (a) canvas the ~20 subscribers, and (b) so the archives show why it's no longer active if we let it lie. And if it is to die, we should disable posting so spammers can't get to it (though it is fairly well protected).

comment:22 by Olly Betts, 9 years ago

Milestone: 1.3.41.3.5

Emailed xapian-commits and xapian-discus about the list. Given we've a couple of useful steps forward, I'm bumping the rest to 1.3.5.

comment:23 by Olly Betts, 8 years ago

Description: modified (diff)
Milestone: 1.3.51.4.x

We concluded the commits list wasn't worth ressurecting.

This is not a blocker for releasing 1.4.0.

comment:24 by Olly Betts, 8 years ago

So with the server switch, we're now on trac 1.0.13 but without the local patch and without the SVN revision id links working (as there's no copy of the SVN repo). The lack of the SVN repo means that the revision number/commit hash disambiguation is no longer an issue for us.

The notable upshot is that you can't abbreviate git commit hashes to 7 characters now (only 8+), which is irritating as the default in git is 7 (when 7 is unique). 7 is short enough that large repos tend to have collisions, so it's probably wise to increase this as a local setting, but that means people need to actually do it. I don't think there's a way to configure this in the repo such that you get it just by cloning, but I may be wrong.

Perhaps this isn't worth carrying a patch for - I opened a bug against trac which seems like it may eventually get dealt with.

I think I'm also happy with how git access is set up now - once migration is complete we'll have anon access via https (secure, more likely to work through firewalls) or the git protocol (which seems to have few advantages these days), and write access via ssh, so we can probably cross off "Consider a switch to gitolite or Simon Tatham's stuff".

comment:25 by James Aylett, 8 years ago

At least at this point I agree we don't need to think about things like gitolite. Simon's restricted git shell probably isn't appropriate for us, as it maps closely onto how they merge things, and just using the normal restricted git-shell for the git write account (which we now do) should be fine.

We're using a xapian-specific install of trac, in the same way as the previous machine, so we can carry a local patch if we feel it's valuable. However (via an article linked in the trac ticket) I note that 7 character is no longer unique across the whole of Xapian:

$ git rev-list --all --abbrev=0 --abbrev-commit |
>   awk '{ a[length] += 1 } END { for (len in a) print len, a[len] }'
4 1894
5 16105
6 2708
7 193
8 8

So we may be better off advising people to set core.abbrev in git clone to 8. We could have bootstrap warn if it's in a git clone with the default (git config core.abbrev will return the empty string). I'm unaware of any way of "pushing" configuration on clone.

I'm not convinced from the trac ticket that they'll get to it any time soon unless someone chips in.

Last edited 8 years ago by Olly Betts (previous) (diff)

comment:26 by Olly Betts, 7 years ago

I've added a command to find the git commit corresponding to a given SVN revision number to https://xapian.org/bleeding :

git log --grep '^git-svn-id: .*@12345 ' `git branch --list -r 'origin/svn/*'`

So we may be better off advising people to set core.abbrev in git clone to 8. We could have bootstrap warn if it's in a git clone with the default (git config core.abbrev will return the empty string). I'm unaware of any way of "pushing" configuration on clone.

I've just written a simple check for the value and then if it's < 8 and not set on the repo itself, to set it to 8 on the repo. But if we already have collisions at length 8, that's clearly too short. I think we should go with 12 - that's not so long as to be clumsy, but long enough that we are unlikely to see collisions for a significant time. Using the full SHA1 is preferable in places (e.g. commit messages and the backporting git notes I've recently started adding).

I'm not convinced from the trac ticket that they'll get to it any time soon unless someone chips in.

I'm happy I've at least led that horse to water.

comment:27 by Olly Betts, 7 years ago

Milestone: 1.4.x1.4.1
Resolution: fixed
Status: assignedclosed

066bfa03586d3b50682c8a6cf9284b2b2ae65b56 implements encouraging use of core.abbrev=12 as described above.

With that, we can at last close this ticket - there's nothing else that we still want to do here.

I intend to backport the core.abbrev bootstrap commit to both the release branches, so setting milestone appropriately.

Note: See TracTickets for help on using tickets.