Skip to content

Home

Jump to bottom
Eric Harmeling edited this page Jul 23, 2020 · 33 revisions

This page provides instructions for common docs tasks. It's intended for internal CockroachDB contributors working from branches in the docs repo rather than from external forks.

Note: With git, there are many ways to reach the same outcome. The git steps here are based on Jesse's current workflow. If there's a better/more efficient way to do something, please let him know. You can also use the GitHub Desktop app.

Start Writing

When you're starting a new docs project:

  1. Move to your copy of the docs repo: cd src/github/cockroachdb/docs.

  2. Make sure you're on the master branch: git checkout master.

    This helps to prevent your new branch from inheriting commits from another branch. If that happens, though, you can always remove them.

  3. Get any recent updates to master: git pull.

  4. Create a new branch: git checkout -b "<your branch name>".

  5. Write/edit markdown files in your text editor.

  6. Build a local version of the docs to view your changes:

    • Use make standard to build CockorachDB docs.
    • Use make cockroachcloud to build Managed CockroachDB docs.

  7. Check the files you've changed: git status.

  8. Stage your changes for commit: git add . (stage all changed files) or git add <filename> (stage a specific file).

  9. Commit your changes: git commit -m "<concise message describing changes>".

  10. If you've made lots of little commits that aren't meaningful on their own, squash some of them.

  11. Start the Review workflow.

Get Reviewed

Once you're ready to have work reviewed:

  1. Push your local branch to the server (first time): git push -u origin <branch name>.

  2. Go to the docs repo in the UI, create a pull request with a brief description.

    TeamCity runs the PR through Jekyll, syncs the output to an S3 test environment, and posts a link to the test docs as a comment in the PR. It's a good idea to check that everything looks good there before moving on.

  3. In the PR, assign your initial reviewer, generally the CockroachDB engineer responsible for, or knowledgable about, the product area you're documenting.

  4. Edit your local branch based on feedback.

  5. Push edits to the server: git push origin <branch name>.

  6. In the PR, assign someone from Docs to do a language/structure review. For new features, it's a good idea to assign someone from Product as well.

  7. Once you get LGTM from the reviewers, merge the PR and delete the remote branch.

  8. Switch to master: git checkout master.

  9. Update master: git pull.

  10. Delete the local PR branch: git branch -d <branch name>.

Publish

When you merge a PR, the changes are automatically published to our GitHub Pages account.

Release Notes

See the cockroach wiki.

Squash Commits

While working, you'll likely make lots of little commits that aren't meaningful on their own. So before opening a PR, it's friendly to squash your commits down to one or a few meaningful ones. Here's how:

  1. Switch to your branch: git checkout <branch name>.
  2. Run: git rebase -i master. This will list your branch's commits in vi.
  3. Press i.
  4. Reorder the commit lines so the commits you want to retain are at the top.
  5. For each commit that you want to squash/meld into the ones at the top, change pick to s or squash.
  6. Press esc + :wq + enter. This will save your changes and open a view of your commit messages in vi.
  7. Press i.
  8. If necessary, update the message of each commit you're keeping.
  9. Press esc + :wq + enter.

Note: After squashing commits, to push to an existing remote branch, you'll need to use the --force flag, e.g., git push --force origin <branch name>.

Remove Commits

If you've somehow inherited unwanted commits from another branch, or if you simply want to remove a commit from your branch:

  1. Switch to your branch: git checkout <branch name>.
  2. Run: git rebase -i master. This will list your branch's commits in vi.
  3. Press i.
  4. For each commit that you want to remove, change pick to d or drop.
  5. Press esc + :wq + enter. This will save your changes and open a view of your commit messages in vi.
  6. Press i.
  7. If necessary, update the message of each commit you're keeping.
  8. Press esc + :wq + enter.

Note: After removing commits, to push to an existing remote branch, you'll need to use the --force flag, e.g., git push --force origin .

Generated Docs

When support for new SQL statements or functions is added or updated, in addition to documenting the updates, you need to generate new SQL syntax diagrams or function syntax tables. After generating the diagrams/tables, you need to add, commit, and push the changes to the docs repo as a part of the documentation PR. When syntax or usage is updated, make sure that you regenerate the diagrams/tables and replace old ones!

We generate diagrams and tables from source code using an internal tool called docgen. When you build cockroach from source, you build a number of other binaries included in the cockroach repo, including docgen.

To build cockroach (and docgen) from source:

  1. Go to your local copy of the cockroach repo.
  2. $ git checkout master
  3. $ git pull
  4. $ make (this takes a few minutes)

You can then install docgen, which places the binary in your GOPATH, and enables you to call it from any directory:

$ go install ./pkg/cmd/docgen

To save time, you can also just build docgen by itself:

  1. Go to your local copy of the cockroach repo.
  2. $ git checkout master
  3. $ git pull
  4. $ make bin/docgen
  5. $ go install ./pkg/cmd/docgen

After docgen is installed, you can use it to generate diagrams and tables. docgen has two commands: grammar and functions. docgen grammar generates diagrams. docgen functions generates functions. See SQL Diagrams and Functions for usage details. You can call also docgen --help for basic usage instructions.

SQL Diagrams

To generate SQL diagrams, use docgen grammar. docgen grammar has two commands: svg and bnf. docgen grammar svg calls a Railroad Diagram generator tool that converts a BNF specification of SQL syntax to XHTML-formatted diagrams. docgen grammar svg takes the diagram generator's XHTML output and converts it to proper HTML that we can use on our website. docgen grammar bnf generates new BNF files from SQL grammar source code.

Before you generate the diagrams, you need to generate the most up-to-date BNF files with docgen grammar bnf:

$ docgen grammar bnf <path to BNF output>

The cockroach makefile calls docgen grammar bnf to generate the BNF files to cockroach/docs/generated/sql/bnf:

bin/.docgen_bnfs: bin/docgen
  docgen grammar bnf docs/generated/sql/bnf --quiet
  touch $@

So, every time you build cockroach from source, new BNF files are generated. Note, however, that the Makefile does not call docgen grammar svg to generate new diagrams from the BNF files. You need to do this manually.

By default, docgen grammar svg calls the Railroad Diagram generator API. This API is undocumented, and subject to change. As of 2/10/2020, we strongly recommend downloading the Railroad JAR file from the employee Google Drive. This JAR file is also undocumented, but it is not subject to change (it's just a file sitting in our Google Drive). To get the JAR file:

  1. Navigate to Google Drive.
  2. Log in with your company email.
  3. Search "Railroad".
  4. Download the Railroad.jar file.

To use the Railroad JAR file, you must have Java 8 installed, and its JRE directory must be set as your JAVA_HOME environment variable!

To set up Java 8 on macOS:

  1. $ brew cask install adoptopenjdk8
  2. Add the following line to the top of your ~/.bash_profile: export JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home
  3. Source the .bash_profile and check your Java version: $ source ~/.bash_profile $ java -version You should get `openjdk version "1.8.x" for the Java version.

After you install the Railroad JAR and Java 8, you are ready to generate new diagrams. To generate new diagrams, run docgen grammar svg with the railroad and filter flags:

$ docgen grammar svg <path to BNF files> <path to diagram output> --railroad=<path>/Railroad.jar --filter=<BNF file to filter on>

The first argument after svg specifies where the BNF files are located (usually in cockroachdb/cockroach/docs/generated/sql/bnf). The second argument specifies where to generate the diagrams (usually in cockroachdb/docs/_includes/v20.1/sql/diagrams/). The railroad flag tells docgen to use the JAR file instead of the API, and specifies where the Railroad.jar file is located. The filter flag filters the output on a specific BNF file name. If you specify no filter, docgen will generate diagrams for all of the grammar (we don't recommend this).

When you generate new diagrams, make sure that you copy them over to the docs repo, in docs/_includes/<version>/sql/diagrams/ for each version of CockroachDB for which the changes were made!

To make our SQL diagrams more usable and intuitive, you can also:

For info about generating SQL diagrams and making custom edits to diagrams, see SQL Grammar Railroad Diagram Changes.

Functions

The cockroach makefile calls docgen to generate function table markdown files to docs/generated/sql:

When you generate new markdown tables, make sure that you copy them over to the docs repo, in docs/_includes/<version>/sql/ for each version of CockroachDB for which the changes were made.

To generate new function tables manually using docgen:

  1. cd $HOME/go/bin
  2. ./docgen functions $HOME/go/src/github.com/cockroachdb/docs/_includes/<version>/sql for each version of CockroachDB for which the changes were made.

docgen pulls the description strings and function syntax from the function definitions in cockroach/pkg/sql/sem/builtins/. funcs.go handles the function tables with function syntax for all supported functions.

Updating the Interactive SQL Help Text

When support for new SQL statements is added to cockroach, or existing SQL statement support is updated, the pkg/sql/parser/sql.y SQL grammar file is updated. Part of this update to the grammar file includes adding or editing the help text that describes the syntax and behavior of a particular statement. At the bottom of the help text is a link to that statement's documentation page, written as %SeeAlso: WEBDOCS/statement.html.

Prior to v20.2, this help text was entirely the responsibility of the engineer(s) working on the statement. Because statements are often added before documentation is available, and because engineering is not responsible for user-facing SQL statement documentation, the links added could point to the wrong page, or to no page at all. Starting in v20.2, if a documentation page does not exist for a statement, there will be no doc link. It is the writer's responsibility to add the link in the help text.

When you add a documentation page for a new SQL statement, or when you update the page name for a SQL statement, you must add/update the link to that page in the statement's help text in pkg/sql/parser/sql.y. To do this:

  1. Create a new branch on your local fork of the cockroach repo. Name it something like "docs-link-create-foo".
  2. Open pkg/sql/parser/sql.y, and find the CREATE FOO statement help text (it should start with // %Help: CREATE FOO).
  3. Locate the end of the help text (i.e., a break in the lines starting with //), and insert the following two lines: 
    //
// %SeeAlso: WEBDOCS/create-foo.html
    where create-foo.md is the name of the markdown file documenting the CREATE FOO statement.
  4. Save the file, stage and commit the change following the commit message guidelines (the commit title should begin with "docs:"), and then push the change to a new remote branch on your cockroach fork.
  5. Open a pull request from the remote branch on your cockroach fork. In the PR description, link to the docs issue for the new statement, and, if it makes sense, to the feature PR in cockroach.

Tip: Look over the help text for a SQL statement before you start documenting the statement. This can help you come up with examples, better understand usage, and focus on which syntax/behavior aspects are the most important.

Note: When you change the name of a SQL statement docs file (or any docs file for that matter), always add redirects from the file's previous name!

Cluster Settings and Session Variables

Cluster settings and session variables are primarily documented on the cluster settings page and the SHOW session variable and SET session variable pages. The session variables table(there is one table, included in both SET and SHOW pages) needs to be updated manually. The cluster settings table is generated to a settings.html file when you make cockroach from source. This file generates rows for all "public" cluster settings (i.e., cluster settings that won't destabilize the cluster - settings that users are expected to change). When you want to update this table in the docs, just fetch, pull, and make cockroach from source, and then copy and paste the generated settings.html file to the docs repo.

Add Search Synonyms

TBD

Using GitHub Desktop

You can use the GitHub Desktop app to do most of the Docs workflow, but in an easier-to-understand UI instead of the command line.

GitHub Desktop Setup

To install GitHub Desktop:

  1. Navigate to GitHub Desktop.
  2. Click Download for macOS (or the relevant link for your OS).
  3. Follow the instructions to install.
  4. Sign in to your GitHub account.

If you have the Docs repo already on your computer:

  1. Navigate to File > Add Local Repository....
  2. Choose the repo file.
  3. Click Add Repository. The Docs repo will display in the left column.

If you don't have the Docs repo:

  1. In GitHub Desktop, navigate to File > Clone Repository.
  2. On the GitHub.com tab, find cockroachdb/docs.
  3. Choose a location on your computer to save the Docs files.
  4. Click Clone. The Docs repo will display in the left column.

Create a Branch

  1. On master, Fetch origin to get any recent updates.
  2. Create a new branch off of master.
    • Go to Branch > New Branch.
    • Enter a unique name for your branch.
    • Click Create Branch.

Start Writing

  1. Write / edit markdown files in your text editor.
  2. Save your edits. A list of local changes will display in the left column of GitHub Desktop.

Commit Changes

When you want to save your changes to your branch, go to GitHub Desktop and commit your changes to your branch:

  1. Enter a Summary, a concise message describing changes.
  2. Enter a Description (optional).
  3. Click Commit to [your-branch-name].

Get Reviewed

Once you're ready to have work reviewed:

  1. Click Publish branch to push your commits to the server.
  2. Go to Branch > Create Pull Request.
  3. GitHub will load in your browser. Click Create Pull Request.
  4. In the PR, assign your initial reviewer, generally the CockroachDB engineer responsible for, or knowledgable about, the product area you're documenting.
  5. Edit your local branch based on feedback.
  6. Commit changes
  7. To push edits to the server, click Push Origin.
  8. In the PR, assign someone from Docs to do a language/structure review. For new features, it's a good idea to assign someone from Product as well.
  9. Once you get LGTM from the reviewers, merge the PR and delete the remote branch.
  10. In GitHub Desktop, switch to master by using the Current Branch dropdown.
  11. Click Fetch Origin to update your local version of master.
  12. Delete the local PR branch.
  • In GitHub Desktop, make sure your current branch is the PR branch.
  • Navigate to Branch > Delete.
  • Click Delete.
Clone this wiki locally