Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prod deploy #2719

Merged
merged 34 commits into from
Oct 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
01256a1
feat: support remote config overrides (#2704)
sweatybridge Sep 24, 2024
11c5004
fix: disable security opts for db test on bitbucket runner (#2705)
avallete Sep 24, 2024
2ad2d33
fix: Bump studio to the latest image version 20240923 (#2706)
ivasilov Sep 24, 2024
bb28463
chore: use _supabase database for internals
avallete Sep 24, 2024
5e94687
chore: fix tests mocks
avallete Sep 24, 2024
5904062
Merge branch 'develop' into avallete/chore-create-new-supabase-manage…
avallete Sep 24, 2024
e6788f3
chore: add realtime schema as well
avallete Sep 24, 2024
76ef3d2
fix: make the start work for postgres 14 and 13
avallete Sep 24, 2024
fd04d07
feat: add custom seed path to config (#2702)
avallete Sep 25, 2024
edb664a
Merge branch 'develop' into avallete/chore-create-new-supabase-manage…
avallete Sep 25, 2024
371a163
chore: add bitbucket build canary badge
avallete Sep 25, 2024
d5a34fc
chore: add gitlab canary badge
avallete Sep 25, 2024
936f5df
chore: add bitbucket build canary badge (#2709)
avallete Sep 25, 2024
49fd57f
Merge branch 'develop' into avallete/chore-create-new-supabase-manage…
avallete Sep 25, 2024
1623aa9
fix: Bump up studio image (#2711)
filipecabaco Sep 26, 2024
2822808
Merge branch 'develop' into avallete/chore-create-new-supabase-manage…
avallete Sep 26, 2024
ccf214d
chore: restore realtime to postgres database
avallete Sep 27, 2024
e911718
chore: remove relatime migration
avallete Sep 27, 2024
3115143
Update internal/db/start/start.go
avallete Sep 27, 2024
b64a8c8
chore: use _supabase database for internals (#2707)
avallete Sep 27, 2024
c8e553f
fix: bump edge-runtime to 1.58.11 (#2718)
nyannyacha Oct 1, 2024
4ee741f
chore(deps): bump github.com/containers/common from 0.60.2 to 0.60.4 …
dependabot[bot] Oct 1, 2024
3bb06fb
chore: glob seed paths when loading config (#2726)
sweatybridge Oct 4, 2024
0efc21d
chore: Bump studio to the latest image version 20240930 (#2720)
ivasilov Oct 4, 2024
333a2ca
fix: email templates for monorepos (#2723)
avallete Oct 4, 2024
9365527
fix: inspect storage container before seeding buckets
sweatybridge Oct 6, 2024
73b591c
chore: use healthy enum from docker api
sweatybridge Oct 6, 2024
b045039
chore: remove syslog config
sweatybridge Oct 6, 2024
887b5bb
feat: track seeded files in migration table
sweatybridge Oct 4, 2024
b906fc5
chore: reduce code duplication
sweatybridge Oct 4, 2024
dffb8d6
chore: fix unit tests
sweatybridge Oct 4, 2024
b3efd48
chore: prompt before pushing seed files
sweatybridge Oct 6, 2024
c805fc6
chore: create seed table separately
sweatybridge Oct 7, 2024
e642265
chore(deps): bump bin-links from 4.0.4 to 5.0.0 (#2712)
dependabot[bot] Oct 7, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Supabase CLI (v1)

[![Coverage Status](https://coveralls.io/repos/github/supabase/cli/badge.svg?branch=main)](https://coveralls.io/github/supabase/cli?branch=main)
[![Coverage Status](https://coveralls.io/repos/github/supabase/cli/badge.svg?branch=main)](https://coveralls.io/github/supabase/cli?branch=main) [![Bitbucket Pipelines](https://img.shields.io/bitbucket/pipelines/supabase-cli/setup-cli/master?style=flat-square&label=Bitbucket%20Canary)](https://bitbucket.org/supabase-cli/setup-cli/pipelines) [![Gitlab Pipeline Status](https://img.shields.io/gitlab/pipeline-status/sweatybridge%2Fsetup-cli?label=Gitlab%20Canary)
](https://gitlab.com/sweatybridge/setup-cli/-/pipelines)

[Supabase](https://supabase.io) is an open source Firebase alternative. We're building the features of Firebase using enterprise-grade open source tools.

Expand Down
2 changes: 1 addition & 1 deletion cmd/db.go
Original file line number Diff line number Diff line change
Expand Up @@ -276,7 +276,7 @@ func init() {
pushFlags := dbPushCmd.Flags()
pushFlags.BoolVar(&includeAll, "include-all", false, "Include all migrations not found on remote history table.")
pushFlags.BoolVar(&includeRoles, "include-roles", false, "Include custom roles from "+utils.CustomRolesPath+".")
pushFlags.BoolVar(&includeSeed, "include-seed", false, "Include seed data from "+utils.SeedDataPath+".")
pushFlags.BoolVar(&includeSeed, "include-seed", false, "Include seed data from your config.")
pushFlags.BoolVar(&dryRun, "dry-run", false, "Print the migrations that would be applied, but don't actually apply them.")
pushFlags.String("db-url", "", "Pushes to the database specified by the connection string (must be percent-encoded).")
pushFlags.Bool("linked", true, "Pushes to the linked project.")
Expand Down
2 changes: 1 addition & 1 deletion cmd/start.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ func validateExcludedContainers(excludedContainers []string) {
// Sort the names list so it's easier to visually spot the one you looking for
sort.Strings(validContainers)
warning := fmt.Sprintf("%s The following container names are not valid to exclude: %s\nValid containers to exclude are: %s\n",
utils.Yellow("Warning:"),
utils.Yellow("WARNING:"),
utils.Aqua(strings.Join(invalidContainers, ", ")),
utils.Aqua(strings.Join(validContainers, ", ")))
fmt.Fprint(os.Stderr, warning)
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ require (
github.com/charmbracelet/bubbletea v0.25.0
github.com/charmbracelet/glamour v0.7.0
github.com/charmbracelet/lipgloss v0.12.1
github.com/containers/common v0.60.2
github.com/containers/common v0.60.4
github.com/danieljoos/wincred v1.2.1
github.com/deepmap/oapi-codegen/v2 v2.2.0
github.com/docker/cli v27.2.1+incompatible
Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -200,8 +200,8 @@ github.com/containerd/console v1.0.4-0.20230313162750-1ae8d489ac81 h1:q2hJAaP1k2
github.com/containerd/console v1.0.4-0.20230313162750-1ae8d489ac81/go.mod h1:YynlIjWYF8myEu6sdkwKIvGQq+cOckRm6So2avqoYAk=
github.com/containerd/log v0.1.0 h1:TCJt7ioM2cr/tfR8GPbGf9/VRAX8D2B4PjzCpfX540I=
github.com/containerd/log v0.1.0/go.mod h1:VRRf09a7mHDIRezVKTRCrOq78v577GXq3bSa3EhrzVo=
github.com/containers/common v0.60.2 h1:utcwp2YkO8c0mNlwRxsxfOiqfj157FRrBjxgjR6f+7o=
github.com/containers/common v0.60.2/go.mod h1:I0upBi1qJX3QmzGbUOBN1LVP6RvkKhd3qQpZbQT+Q54=
github.com/containers/common v0.60.4 h1:H5+LAMHPZEqX6vVNOQ+IguVsaFl8kbO/SZ/VPXjxhy0=
github.com/containers/common v0.60.4/go.mod h1:I0upBi1qJX3QmzGbUOBN1LVP6RvkKhd3qQpZbQT+Q54=
github.com/containers/storage v1.55.0 h1:wTWZ3YpcQf1F+dSP4KxG9iqDfpQY1otaUXjPpffuhgg=
github.com/containers/storage v1.55.0/go.mod h1:28cB81IDk+y7ok60Of6u52RbCeBRucbFOeLunhER1RQ=
github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
Expand Down
6 changes: 3 additions & 3 deletions internal/db/diff/diff_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ func TestRun(t *testing.T) {
JSON(types.ContainerJSON{ContainerJSONBase: &types.ContainerJSONBase{
State: &types.ContainerState{
Running: true,
Health: &types.Health{Status: "healthy"},
Health: &types.Health{Status: types.Healthy},
},
}})
apitest.MockDockerStart(utils.Docker, utils.GetRegistryImageUrl(utils.Config.Realtime.Image), "test-shadow-realtime")
Expand Down Expand Up @@ -267,7 +267,7 @@ func TestDiffDatabase(t *testing.T) {
JSON(types.ContainerJSON{ContainerJSONBase: &types.ContainerJSONBase{
State: &types.ContainerState{
Running: true,
Health: &types.Health{Status: "healthy"},
Health: &types.Health{Status: types.Healthy},
},
}})
gock.New(utils.Docker.DaemonHost()).
Expand Down Expand Up @@ -303,7 +303,7 @@ At statement 0: create schema public`)
JSON(types.ContainerJSON{ContainerJSONBase: &types.ContainerJSONBase{
State: &types.ContainerState{
Running: true,
Health: &types.Health{Status: "healthy"},
Health: &types.Health{Status: types.Healthy},
},
}})
gock.New(utils.Docker.DaemonHost()).
Expand Down
1 change: 1 addition & 0 deletions internal/db/dump/dump.go
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ func dumpData(ctx context.Context, config pgconn.Config, schema, excludeTable []
// "storage",
// "supabase_functions",
"supabase_migrations",
// TODO: Remove in a few version in favor of _supabase internal db
"_analytics",
"_realtime",
"_supavisor",
Expand Down
88 changes: 67 additions & 21 deletions internal/db/push/push.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ import (
"github.com/jackc/pgconn"
"github.com/jackc/pgx/v4"
"github.com/spf13/afero"
"github.com/supabase/cli/internal/migration/apply"
"github.com/supabase/cli/internal/migration/up"
"github.com/supabase/cli/internal/utils"
"github.com/supabase/cli/pkg/migration"
Expand All @@ -29,39 +28,75 @@ func Run(ctx context.Context, dryRun, ignoreVersionMismatch bool, includeRoles,
if err != nil {
return err
}
if len(pending) == 0 {
var seeds []migration.SeedFile
if includeSeed {
seeds, err = migration.GetPendingSeeds(ctx, utils.Config.Db.Seed.SqlPaths, conn, afero.NewIOFS(fsys))
if err != nil {
return err
}
}
var globals []string
if includeRoles {
if exists, err := afero.Exists(fsys, utils.CustomRolesPath); err != nil {
return errors.Errorf("failed to find custom roles: %w", err)
} else if exists {
globals = append(globals, utils.CustomRolesPath)
}
}
if len(pending) == 0 && len(seeds) == 0 && len(globals) == 0 {
fmt.Println("Remote database is up to date.")
return nil
}
// Push pending migrations
if dryRun {
if includeRoles {
fmt.Fprintln(os.Stderr, "Would create custom roles "+utils.Bold(utils.CustomRolesPath)+"...")
if len(globals) > 0 {
fmt.Fprintln(os.Stderr, "Would create custom roles "+utils.Bold(globals[0])+"...")
}
fmt.Fprintln(os.Stderr, "Would push these migrations:")
fmt.Fprint(os.Stderr, utils.Bold(confirmPushAll(pending)))
if includeSeed {
fmt.Fprintln(os.Stderr, "Would seed data "+utils.Bold(utils.SeedDataPath)+"...")
if len(pending) > 0 {
fmt.Fprintln(os.Stderr, "Would push these migrations:")
fmt.Fprint(os.Stderr, confirmPushAll(pending))
}
} else {
msg := fmt.Sprintf("Do you want to push these migrations to the remote database?\n%s\n", confirmPushAll(pending))
if shouldPush, err := utils.NewConsole().PromptYesNo(ctx, msg, true); err != nil {
return err
} else if !shouldPush {
return errors.New(context.Canceled)
if len(seeds) > 0 {
fmt.Fprintln(os.Stderr, "Would seed these files:")
fmt.Fprint(os.Stderr, confirmSeedAll(seeds))
}
if includeRoles {
if err := apply.CreateCustomRoles(ctx, conn, fsys); err != nil {
} else {
if len(globals) > 0 {
msg := "Do you want to create custom roles in the database cluster?"
if shouldPush, err := utils.NewConsole().PromptYesNo(ctx, msg, true); err != nil {
return err
} else if !shouldPush {
return errors.New(context.Canceled)
}
if err := migration.SeedGlobals(ctx, globals, conn, afero.NewIOFS(fsys)); err != nil {
return err
}
}
if err := migration.ApplyMigrations(ctx, pending, conn, afero.NewIOFS(fsys)); err != nil {
return err
if len(pending) > 0 {
msg := fmt.Sprintf("Do you want to push these migrations to the remote database?\n%s\n", confirmPushAll(pending))
if shouldPush, err := utils.NewConsole().PromptYesNo(ctx, msg, true); err != nil {
return err
} else if !shouldPush {
return errors.New(context.Canceled)
}
if err := migration.ApplyMigrations(ctx, pending, conn, afero.NewIOFS(fsys)); err != nil {
return err
}
} else {
fmt.Fprintln(os.Stderr, "Schema migrations are up to date.")
}
if includeSeed {
if err := apply.SeedDatabase(ctx, conn, fsys); err != nil {
if len(seeds) > 0 {
msg := fmt.Sprintf("Do you want to seed the remote database with these files?\n%s\n", confirmSeedAll(seeds))
if shouldPush, err := utils.NewConsole().PromptYesNo(ctx, msg, true); err != nil {
return err
} else if !shouldPush {
return errors.New(context.Canceled)
}
if err := migration.SeedData(ctx, seeds, conn, afero.NewIOFS(fsys)); err != nil {
return err
}
} else if includeSeed {
fmt.Fprintln(os.Stderr, "Seed files are up to date.")
}
}
fmt.Println("Finished " + utils.Aqua("supabase db push") + ".")
Expand All @@ -71,7 +106,18 @@ func Run(ctx context.Context, dryRun, ignoreVersionMismatch bool, includeRoles,
func confirmPushAll(pending []string) (msg string) {
for _, path := range pending {
filename := filepath.Base(path)
msg += fmt.Sprintf(" • %s\n", filename)
msg += fmt.Sprintf(" • %s\n", utils.Bold(filename))
}
return msg
}

func confirmSeedAll(pending []migration.SeedFile) (msg string) {
for _, seed := range pending {
notice := seed.Path
if seed.Dirty {
notice += " (hash update)"
}
msg += fmt.Sprintf(" • %s\n", utils.Bold(notice))
}
return msg
}
17 changes: 14 additions & 3 deletions internal/db/push/push_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ package push

import (
"context"
"crypto/sha256"
"encoding/hex"
"os"
"path/filepath"
"testing"
Expand Down Expand Up @@ -146,7 +148,7 @@ func TestPushAll(t *testing.T) {

t.Run("throws error on roles failure", func(t *testing.T) {
// Setup in-memory fs
fsys := &fstest.OpenErrorFs{DenyPath: utils.CustomRolesPath}
fsys := &fstest.StatErrorFs{DenyPath: utils.CustomRolesPath}
path := filepath.Join(utils.MigrationsDir, "0_test.sql")
require.NoError(t, afero.WriteFile(fsys, path, []byte{}, 0644))
// Setup mock postgres
Expand All @@ -161,21 +163,30 @@ func TestPushAll(t *testing.T) {
})

t.Run("throws error on seed failure", func(t *testing.T) {
digest := hex.EncodeToString(sha256.New().Sum(nil))
seedPath := filepath.Join(utils.SupabaseDirPath, "seed.sql")
utils.Config.Db.Seed.SqlPaths = []string{seedPath}
// Setup in-memory fs
fsys := &fstest.OpenErrorFs{DenyPath: utils.SeedDataPath}
fsys := afero.NewMemMapFs()
require.NoError(t, afero.WriteFile(fsys, seedPath, []byte{}, 0644))
path := filepath.Join(utils.MigrationsDir, "0_test.sql")
require.NoError(t, afero.WriteFile(fsys, path, []byte{}, 0644))
// Setup mock postgres
conn := pgtest.NewConn()
defer conn.Close(t)
conn.Query(migration.LIST_MIGRATION_VERSION).
Reply("SELECT 0").
Query(migration.SELECT_SEED_TABLE).
Reply("SELECT 0")
helper.MockMigrationHistory(conn).
Query(migration.INSERT_MIGRATION_VERSION, "0", "test", nil).
Reply("INSERT 0 1")
helper.MockSeedHistory(conn).
Query(migration.UPSERT_SEED_FILE, seedPath, digest).
ReplyError(pgerrcode.NotNullViolation, `null value in column "hash" of relation "seed_files"`)
// Run test
err := Run(context.Background(), false, false, false, true, dbConfig, fsys, conn.Intercept)
// Check error
assert.ErrorIs(t, err, os.ErrPermission)
assert.ErrorContains(t, err, `ERROR: null value in column "hash" of relation "seed_files" (SQLSTATE 23502)`)
})
}
32 changes: 10 additions & 22 deletions internal/db/reset/reset.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import (
"strings"
"time"

"github.com/docker/docker/api/types"
"github.com/docker/docker/api/types/container"
"github.com/docker/docker/api/types/network"
"github.com/docker/docker/errdefs"
Expand Down Expand Up @@ -54,9 +55,11 @@ func Run(ctx context.Context, version string, config pgconn.Config, fsys afero.F
return err
}
// Seed objects from supabase/buckets directory
if utils.Config.Storage.Enabled {
if err := start.WaitForHealthyService(ctx, 30*time.Second, utils.StorageId); err != nil {
return err
if resp, err := utils.Docker.ContainerInspect(ctx, utils.StorageId); err == nil {
if resp.State.Health == nil || resp.State.Health.Status != types.Healthy {
if err := start.WaitForHealthyService(ctx, 30*time.Second, utils.StorageId); err != nil {
return err
}
}
if err := buckets.Run(ctx, "", false, fsys); err != nil {
return err
Expand Down Expand Up @@ -97,11 +100,6 @@ func resetDatabase14(ctx context.Context, version string, fsys afero.Fs, options
return err
}
defer conn.Close(context.Background())
if utils.Config.Db.MajorVersion > 14 {
if err := start.SetupDatabase(ctx, conn, utils.DbId, os.Stderr, fsys); err != nil {
return err
}
}
return apply.MigrateAndSeed(ctx, version, conn, fsys)
}

Expand All @@ -112,10 +110,6 @@ func resetDatabase15(ctx context.Context, version string, fsys afero.Fs, options
if err := utils.Docker.VolumeRemove(ctx, utils.DbId, true); err != nil {
return errors.Errorf("failed to remove volume: %w", err)
}
// Skip syslog if vector container is not started
if _, err := utils.Docker.ContainerInspect(ctx, utils.VectorId); err != nil {
utils.Config.Analytics.Enabled = false
}
config := start.NewContainerConfig()
hostConfig := start.NewHostConfig()
networkingConfig := network.NetworkingConfig{
Expand All @@ -132,15 +126,7 @@ func resetDatabase15(ctx context.Context, version string, fsys afero.Fs, options
if err := start.WaitForHealthyService(ctx, start.HealthTimeout, utils.DbId); err != nil {
return err
}
conn, err := utils.ConnectLocalPostgres(ctx, pgconn.Config{}, options...)
if err != nil {
return err
}
defer conn.Close(context.Background())
if err := start.SetupDatabase(ctx, conn, utils.DbId, os.Stderr, fsys); err != nil {
return err
}
if err := apply.MigrateAndSeed(ctx, version, conn, fsys); err != nil {
if err := start.SetupLocalDatabase(ctx, version, fsys, os.Stderr, options...); err != nil {
return err
}
fmt.Fprintln(os.Stderr, "Restarting containers...")
Expand Down Expand Up @@ -171,6 +157,8 @@ func recreateDatabase(ctx context.Context, options ...func(*pgx.ConnConfig)) err
Statements: []string{
"DROP DATABASE IF EXISTS postgres WITH (FORCE)",
"CREATE DATABASE postgres WITH OWNER postgres",
"DROP DATABASE IF EXISTS _supabase WITH (FORCE)",
"CREATE DATABASE _supabase WITH OWNER postgres",
},
}
return sql.ExecBatch(ctx, conn)
Expand Down Expand Up @@ -210,7 +198,7 @@ func restartServices(ctx context.Context) error {
services := listServicesToRestart()
result := utils.WaitAll(services, func(id string) error {
if err := utils.Docker.ContainerRestart(ctx, id, container.StopOptions{}); err != nil && !errdefs.IsNotFound(err) {
return errors.Errorf("Failed to restart %s: %w", id, err)
return errors.Errorf("failed to restart %s: %w", id, err)
}
return nil
})
Expand Down
Loading
Loading