MM-21722 - Repository synchronization tool

[tasdomas]

Summary

This PR adds a proof-of-concept implementation of a tool for synchronizing mattermost plugin repositories with the template repository (mattermost-plugin-starter-template).

Ticket Link

See https://mattermost.atlassian.net/browse/MM-21722

[tasdomas]

Running against mattermost-plugin-demo:

➜  mattermost-plugin-starter-template git:(sync) go run ./build/sync/main.go ./build/sync/plan.yml ../mattermost-plugin-demo
FAILED	.circleci/config.yml: check failed, file "/home/d/src/mattermost-plugin-demo/.circleci/config.yml" has been altered
UPDATED	.editorconfig
FAILED	Makefile: check failed, file "/home/d/src/mattermost-plugin-demo/Makefile" has been altered
UPDATED	build
FAILED	go.mod: check failed, file "/home/d/src/mattermost-plugin-demo/go.mod" has been altered
FAILED	public/hello.html: check failed, file "/home/d/src/mattermost-plugin-demo/public/hello.html" has been altered
FAILED	server/configuration.go: check failed, file "/home/d/src/mattermost-plugin-demo/server/configuration.go" has been altered
FAILED	server/plugin.go: check failed, file "/home/d/src/mattermost-plugin-demo/server/plugin.go" has been altered
FAILED	server/plugin_test.go: check failed, path "server/plugin_test.go" does not exist
FAILED	webapp/.babelrc: check failed, path "webapp/.babelrc" does not exist
UPDATED	webapp/.eslintrc.json
FAILED	webapp/i18n/en.json: check failed, file "/home/d/src/mattermost-plugin-demo/webapp/i18n/en.json" has been altered
FAILED	webapp/package.json: check failed, file "/home/d/src/mattermost-plugin-demo/webapp/package.json" has been altered
FAILED	webapp/src/index.js: check failed, file "/home/d/src/mattermost-plugin-demo/webapp/src/index.js" has been altered
FAILED	webapp/tsconfig.json: check failed, path "webapp/tsconfig.json" does not exist
FAILED	webapp/webpack.config.js: check failed, file "/home/d/src/mattermost-plugin-demo/webapp/webpack.config.js" has been altered

[tasdomas]

Followups for this:

• implement format-specific merge actions (e.g. for go.mod)
• implement more intelligent non-existant file handling (a file could have been deleted on the target repository or it may have never been part of it, if it was introduced on the template repository recently)
• add a make sync target to checkout the template repository and run sync against the plugin
• investigate update dependencies (updating ./build may require updating go.mod)
• store which commit of the template repository a plugin has been sync'ed with

To make a long story short - proper, developer-friendly synchronization is complicated.

[mattermod]

This issue has been automatically labelled "stale" because it hasn't had recent activity.
A core team member will check in on the status of the PR to help with questions.
Thank you for your contribution!

/cc @jasonblais @hanzei

[jfrerich]

Thanks, @tasdomas. I will do some iteration on change requests. Before diving into the code, I was looking at the files that were in plan.yml and how the file is organized. I also created some small code nitpicks before I decided to review the plan.yml

[jfrerich]

This is an optional file that will not be used in plugins 99% of the time. Perhaps we could add an action that warns when the file exists and notes to the author that they might remove this dir/file.

This also suggests adding a .plan.yml file in a repository that will allow setting action ignores to the script. An example is if a plugin actually uses this file they would continue getting the warning.

[tasdomas]

That makes sense. I think it'd be ok to address this in a follow-up.

[jfrerich]

For .go files, it is going to be very difficult to do a compare and merge. I think the best we could do is add some AST scripts to check for certain issues.

I'm only thinking aloud here:

• known plugin hooks are in specific files or exist?
• use of known helper functions that could be used?

[tasdomas]

That seems more like a checker/linter type tool - I think it makes sense to make that a separate command, instead of integrating it with the sync tool.

[jfrerich]

This is a collection of files categorized by types and how they should be handled. I'm only making a comment for informational purposes and tracking

server/ Go Files

• could add some checks, but this is going to be difficult
• Plugin Specific
  • path: go.mod
  • path: server/configuration.go
  • path: server/plugin.go
  • path: server/plugin_test.go

webapp/ Files

• only compare if plugin has webapp/ dir
  • path: webapp/.babelrc
  • path: webapp/webpack.config.js
  • path: webapp/tsconfig.json
  • path: webapp/.eslintrc.json
  • path: webapp/src/index.js

json Files

• Plugin Specific
  • path: webapp/package.json

Should warn that they don't exists (Good to have for plugins)

• path: webapp/i18n/en.json

OPTIONAL Files (warn, but don't fail)

• path: public/hello.html

Should match Demo Plugin not starter template

• path: .circleci/config.yml

Should match starter-template 100%

• path: Makefile
• path: .editorconfig

[mickmister]

Hi @tasdomas, thanks for all of the great work in this PR. Do you plan on continuing the project? If not, I would like to continue the work in a separate PR.

[tasdomas]

@mickmister - sorry for the late reply. I was sort of expecting reviews for the PR.

Give me a day or so to at least clean up the PR and address the existing comments. Then you can take over.

[mickmister]

@mickmister - sorry for the late reply. I was sort of expecting reviews for the PR.

@tasdomas No problem at all. I wrongly assumed you did not intend to continue with the PR, but looking at the history it is absolutely due for reviews. There is a ton of work here that has not been reviewed for quite some time. It is up to you whether you choose to continue with this PR. I can give this a review sometime this week, please let me know what you think.

Thank you for all you've done here. This is an unprecedented tool that will make a huge difference.

[tasdomas]

@hanzei, @mickmister, @jfrerich - I've updated the PR with your suggestions.

Also, I've noticed that the root go.mod file references go 1.12, but ci is using 1.14.

[mickmister]

Would it be possible to support an exclude field in plan.yml near paths, in order to exclude certain files/folders? build/sync for example should only be in the starter-template, and not in the other plugin repos. So when the build folder is sync'd, the build/sync folder comes along too.

[tasdomas]

Would it be possible to support an exclude field in plan.yml near paths, in order to exclude certain files/folders? build/sync for example should only be in the starter-template, and not in the other plugin repos. So when the build folder is sync'd, the build/sync folder comes along too.

Sure. I'd like to handle that in a follow-up though.

[mickmister]

@tasdomas Do you mind putting the go.mod file in the root dir of the build folder? I think we want to keep all the build deps in one file. Other than that, I think this is good to merge as I'm making followup tickets for the other portions.

Repo sync tool: Support excludes field in actions #121
Repo sync tool: Partial equality checks/merges #122
Repo sync tool: Script to check if a target repo needs to be sync'd #123

@hanzei What do you think about merging this, given these followup tickets?

[tasdomas]

@mickmister - I've updated the PR.

[hanzei]

Awesome work @tasdomas 🎉 Sorry for the huge delay.

[hanzei]

@mickmister Absolutely. Let's get this merged.

[tasdomas]

@hanzei, @mickmister, @ben, @jfrerich thank you for reviewing this!

Time for 🍾 :)

[hanzei]

Heads up: I've posted some follow up thoughts in https://community-release.mattermost.com/core/pl/tkyb38n9hib3jby3t1g64w1xia