Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Full SSL Support for all Databases #5027

Open
wants to merge 90 commits into
base: next
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
90 commits
Select commit Hold shift + click to select a range
3f582a1
feat(migration): Add `ssl_certificates` table and model
peaklabs-dev Jan 29, 2025
214a7a0
feat(migration): Add ssl setting to `standalone_postgresqls` table
peaklabs-dev Jan 29, 2025
875d1d4
feat(ui): Add ssl settings to Postgres ui
peaklabs-dev Jan 29, 2025
92a4b5f
feat(db): add ssl mode to Postgres URLs
peaklabs-dev Jan 29, 2025
b124904
feat(db): setup ssl during Postgres start
peaklabs-dev Jan 29, 2025
be49502
Merge branch 'coollabsio:main' into feat-db-ssl
peaklabs-dev Jan 29, 2025
9f93499
fix(ssl): permission of ssl crt and key inside the container
peaklabs-dev Jan 30, 2025
edddbc8
feat(migration): encrypt local file volumes content and paths
peaklabs-dev Jan 30, 2025
429453a
fix(ui): make sure file mounts do not showing the encrypted values
peaklabs-dev Jan 30, 2025
2ac9147
chore(migration): remove unused columns
peaklabs-dev Jan 30, 2025
3632f29
feat(ssl): ssl generation helper
peaklabs-dev Jan 30, 2025
5460018
chore(ssl): improve code in ssl helper
peaklabs-dev Jan 30, 2025
b53d3d0
fix(ssl): make default ssl mode require not verify-full as it does no…
peaklabs-dev Jan 30, 2025
d280f11
feat(ssl): migrate to `ECC`certificates using `secp521r1`
peaklabs-dev Jan 30, 2025
3418845
feat(ssl): improve SSL helper
peaklabs-dev Jan 30, 2025
22c26cd
chore(migration): ssl cert and key should not be nullable
peaklabs-dev Jan 30, 2025
e1245f4
fix(ui): select component should not always uses title case
peaklabs-dev Jan 31, 2025
90a93ce
feat(ssl): add a Coolify CA Certificate to all servers
peaklabs-dev Jan 31, 2025
503e1ff
feat(seeder): Call CA SSL seeder in prod and dev
peaklabs-dev Jan 31, 2025
0915303
feat(ssl): Add Coolify CA Certificate when adding a new server
peaklabs-dev Jan 31, 2025
34216af
fix(db): SSL certificates table and model
peaklabs-dev Jan 31, 2025
fab7300
feat(installer): create CA folder during installation
peaklabs-dev Jan 31, 2025
02475c5
feat(ssl): improve SSL helper
peaklabs-dev Jan 31, 2025
85c777d
feat(ssl): use new improved helper for SSL generation
peaklabs-dev Jan 31, 2025
7406ee6
chore(ssl): rename CA cert to `coolify-ca.crt` because of conflicts
peaklabs-dev Jan 31, 2025
ab1833b
feat(ui): Add CA cert UI
peaklabs-dev Jan 31, 2025
6d0291a
feat(ui): new copy button component
peaklabs-dev Jan 31, 2025
4eba1d2
feat(ui): use new copy button component everywhere
peaklabs-dev Jan 31, 2025
4305ba5
fix(migration): ssl certificates table
peaklabs-dev Feb 2, 2025
30343b0
feat(ui): improve server advanced view
peaklabs-dev Feb 3, 2025
a1e650e
chore: rename ca crt folder to ssl
peaklabs-dev Feb 3, 2025
3cf758e
fix(databases): fix database name users new `uuid` instead of DB one
peaklabs-dev Feb 3, 2025
5f357e3
fix(database): fix volume and file mounts and naming
peaklabs-dev Feb 3, 2025
498bf04
feat(migration): add CN and alternative names to DB
peaklabs-dev Feb 3, 2025
9d9fbd6
feat(databases): add CA SSL crt location to Postgres URLs
peaklabs-dev Feb 3, 2025
72a2f79
feat(ssl): improve ssl generation
peaklabs-dev Feb 3, 2025
f871c10
Merge branch 'next' into feat-db-ssl
peaklabs-dev Feb 3, 2025
fba95c3
fix(migration): store subjectAlternativeNames as a json array in the db
peaklabs-dev Feb 3, 2025
2fbb898
feat(ssl): regenerate SSL certs job
peaklabs-dev Feb 3, 2025
cd335e9
fix(ssl): make sure the subjectAlternativeNames are unique and stored…
peaklabs-dev Feb 3, 2025
5351092
feat(ssl): regenerate certificate and valid until UI
peaklabs-dev Feb 3, 2025
fd5b749
chore(ui): improve valid until handling
peaklabs-dev Feb 3, 2025
c3a440a
fix(ui): certificate expiration data is null before starting the DB
peaklabs-dev Feb 4, 2025
6de76ca
fix(deletion): fix DB deletion
peaklabs-dev Feb 4, 2025
3c62130
fix(ssl): improve SSL cert file mounts
peaklabs-dev Feb 4, 2025
da148f9
feat(ssl): regenerate CA cert and all other certs logic
peaklabs-dev Feb 4, 2025
d6a39f2
fix(ssl): always create ca crt on disk even if it is already there
peaklabs-dev Feb 4, 2025
3f857c6
feat(ssl): Add full MySQL SSL Support
peaklabs-dev Feb 4, 2025
80fc7c7
fix(ssl): use mountPath parameter not a hardcoded path
peaklabs-dev Feb 4, 2025
8f2b45c
fix(ssl): use 1 instead of on for mysql
peaklabs-dev Feb 4, 2025
e81ed1a
feat(ssl): Add full MariaDB SSL support
peaklabs-dev Feb 4, 2025
a3c4f86
fix(ssl): do not remove SSL directory
peaklabs-dev Feb 5, 2025
1003858
feat(ssl): Add `openssl.conf` to configure SSL extension properly
peaklabs-dev Feb 5, 2025
7666cec
fix(ssl): wrong ssl cert is loaded to the server and UI error when re…
peaklabs-dev Feb 5, 2025
ba24630
fix(ssl): make sure when regenerating the CA cert it is not overwritt…
peaklabs-dev Feb 5, 2025
951a454
fix(ssl): regenerating certs for a specific DB
peaklabs-dev Feb 5, 2025
806d9af
feat(ssl): improve SSL generation and security a lot
peaklabs-dev Feb 5, 2025
852be5f
feat(ssl): check for SSL renewal twice daily
peaklabs-dev Feb 5, 2025
844f401
feat(ssl): Add SSL relationships to all DBs
peaklabs-dev Feb 5, 2025
367eebc
feat: Add full SSL support to MongoDB
peaklabs-dev Feb 5, 2025
6eabfd5
feat/fix(ssl): fix some issues and improve ssl generation helper
peaklabs-dev Feb 6, 2025
1a4c2c3
fix(ssl): fix MariaDB and MySQL need CA cert
peaklabs-dev Feb 6, 2025
f92c170
feat(ssl): ability to create `.pem` certs and add `clientAuth` to `ex…
peaklabs-dev Feb 7, 2025
35cd957
fix(ssl): add mount path to DB to fix regeneration of certs
peaklabs-dev Feb 7, 2025
69a6010
fix(ssl): fix SSL regeneration to sign with CA cert and use mount path
peaklabs-dev Feb 7, 2025
6a52f51
fix(ssl): get caCert correctly
peaklabs-dev Feb 7, 2025
8360067
fix(ssl): remove caCert even if it is a folder by accident
peaklabs-dev Feb 7, 2025
62fb2c2
fix(ssl): ger caCert and `mountPath` correctly
peaklabs-dev Feb 7, 2025
8a45c24
fix(ui): only show Regenerate SSL Certificates button when there is a…
peaklabs-dev Feb 7, 2025
bd33f65
feat(ssl): new modes for MongoDB and get `caCert` and `mountPath` cor…
peaklabs-dev Feb 7, 2025
a539bfd
fix(ssl): server id
peaklabs-dev Feb 7, 2025
cd63760
fix(ssl): when regenerating SSL certs the cert is not singed with the…
peaklabs-dev Feb 7, 2025
c1e7a57
fix(ssl): adjust ca paths for MySQL
peaklabs-dev Feb 7, 2025
5b347f3
fix(ssl): remove mode selection for MariaDB as it is not supported
peaklabs-dev Feb 7, 2025
aad717d
fix(ssl): permission issue with MariDB cert and key and paths
peaklabs-dev Feb 7, 2025
7b30b1a
feat(ssl): Full SSL support for Redis
peaklabs-dev Feb 7, 2025
c7840bd
Merge branch 'next' into feat-db-ssl
peaklabs-dev Feb 7, 2025
484fc51
fix(ssl): rename Redis mode to verify-ca as it is not verify-full
peaklabs-dev Feb 8, 2025
5c12f72
feat: New mode implementation for MongoDB
peaklabs-dev Feb 10, 2025
6b6a9f5
fix(ui): remove unused mode for MongoDB
peaklabs-dev Feb 10, 2025
792b1b8
faet(migration): Add SSL fields to database tables
peaklabs-dev Feb 10, 2025
4547647
feat(ssl): improve Redis and remove modes
peaklabs-dev Feb 10, 2025
90e681e
feat: Full SSL support for DrangonflyDB
peaklabs-dev Feb 10, 2025
3e95387
Full: SSL Support for KeyDB
peaklabs-dev Feb 10, 2025
268fca3
feat: SSL notification
peaklabs-dev Feb 10, 2025
43adb74
Merge branch 'next' into feat-db-ssl
peaklabs-dev Feb 11, 2025
0a738e6
fix(ssl): KeyDB port and caCert args are missing
peaklabs-dev Feb 11, 2025
4fdd567
fix(ui): enable SSL is not working correctly for KeyDB
peaklabs-dev Feb 11, 2025
d74c578
fix(ssl): add `--tls` arg to DrangflyDB
peaklabs-dev Feb 11, 2025
f288852
fix(notification): always send SSL notifications
peaklabs-dev Feb 11, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 106 additions & 6 deletions app/Actions/Database/StartDragonfly.php
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

namespace App\Actions\Database;

use App\Helpers\SslHelper;
use App\Models\SslCertificate;
use App\Models\StandaloneDragonfly;
use Lorisleiva\Actions\Concerns\AsAction;
use Symfony\Component\Yaml\Yaml;
Expand All @@ -16,24 +18,74 @@ class StartDragonfly

public string $configuration_dir;

private ?SslCertificate $ssl_certificate = null;

public function handle(StandaloneDragonfly $database)
{
$this->database = $database;

$startCommand = "dragonfly --requirepass {$this->database->dragonfly_password}";

$container_name = $this->database->uuid;
$this->configuration_dir = database_configuration_dir().'/'.$container_name;

$this->commands = [
"echo 'Starting database.'",
"echo 'Creating directories.'",
"mkdir -p $this->configuration_dir",
"echo 'Directories created successfully.'",
];

if (! $this->database->enable_ssl) {
$this->commands[] = "rm -rf $this->configuration_dir/ssl";
SslCertificate::where('resource_type', $this->database->getMorphClass())
->where('resource_id', $this->database->id)
->delete();
$this->database->fileStorages()
->where('resource_type', $this->database->getMorphClass())
->where('resource_id', $this->database->id)
->get()
->filter(function ($storage) {
return in_array($storage->mount_path, [
'/etc/dragonfly/certs/server.crt',
'/etc/dragonfly/certs/server.key',
]);
})
->each(function ($storage) {
$storage->delete();
});
} else {
$this->commands[] = "echo 'Setting up SSL for this database.'";
$this->commands[] = "mkdir -p $this->configuration_dir/ssl";

$server = $this->database->destination->server;
$caCert = SslCertificate::where('server_id', $server->id)->where('is_ca_certificate', true)->first();

$this->ssl_certificate = SslCertificate::where('resource_type', $this->database->getMorphClass())
->where('resource_id', $this->database->id)
->first();

if (! $this->ssl_certificate) {
$this->commands[] = "echo 'No SSL certificate found, generating new SSL certificate for this database.'";
$this->ssl_certificate = SslHelper::generateSslCertificate(
commonName: $this->database->uuid,
resourceType: $this->database->getMorphClass(),
resourceId: $this->database->id,
serverId: $server->id,
caCert: $caCert->ssl_certificate,
caKey: $caCert->ssl_private_key,
configurationDir: $this->configuration_dir,
mountPath: '/etc/dragonfly/certs',
);
}
}

$container_name = $this->database->uuid;
$this->configuration_dir = database_configuration_dir().'/'.$container_name;

$persistent_storages = $this->generate_local_persistent_volumes();
$persistent_file_volumes = $this->database->fileStorages()->get();
$volume_names = $this->generate_local_persistent_volumes_only_volume_names();
$environment_variables = $this->generate_environment_variables();
$startCommand = $this->buildStartCommand();

$docker_compose = [
'services' => [
Expand Down Expand Up @@ -70,27 +122,55 @@ public function handle(StandaloneDragonfly $database)
],
],
];

if (! is_null($this->database->limits_cpuset)) {
data_set($docker_compose, "services.{$container_name}.cpuset", $this->database->limits_cpuset);
}

if ($this->database->destination->server->isLogDrainEnabled() && $this->database->isLogDrainEnabled()) {
$docker_compose['services'][$container_name]['logging'] = generate_fluentd_configuration();
}

if (count($this->database->ports_mappings_array) > 0) {
$docker_compose['services'][$container_name]['ports'] = $this->database->ports_mappings_array;
}

$docker_compose['services'][$container_name]['volumes'] ??= [];

if (count($persistent_storages) > 0) {
$docker_compose['services'][$container_name]['volumes'] = $persistent_storages;
$docker_compose['services'][$container_name]['volumes'] = array_merge(
$docker_compose['services'][$container_name]['volumes'],
$persistent_storages
);
}

if (count($persistent_file_volumes) > 0) {
$docker_compose['services'][$container_name]['volumes'] = $persistent_file_volumes->map(function ($item) {
return "$item->fs_path:$item->mount_path";
})->toArray();
$docker_compose['services'][$container_name]['volumes'] = array_merge(
$docker_compose['services'][$container_name]['volumes'],
$persistent_file_volumes->map(function ($item) {
return "$item->fs_path:$item->mount_path";
})->toArray()
);
}

if (count($volume_names) > 0) {
$docker_compose['volumes'] = $volume_names;
}

if ($this->database->enable_ssl) {
$docker_compose['services'][$container_name]['volumes'] = array_merge(
$docker_compose['services'][$container_name]['volumes'] ?? [],
[
[
'type' => 'bind',
'source' => '/data/coolify/ssl/coolify-ca.crt',
'target' => '/etc/dragonfly/certs/coolify-ca.crt',
'read_only' => true,
],
]
);
}

// Add custom docker run options
$docker_run_options = convertDockerRunToCompose($this->database->custom_docker_run_options);
$docker_compose = generateCustomDockerRunOptionsForDatabases($docker_run_options, $docker_compose, $container_name, $this->database->destination->network);
Expand All @@ -102,12 +182,32 @@ public function handle(StandaloneDragonfly $database)
$this->commands[] = "echo '{$readme}' > $this->configuration_dir/README.md";
$this->commands[] = "echo 'Pulling {$database->image} image.'";
$this->commands[] = "docker compose -f $this->configuration_dir/docker-compose.yml pull";
if ($this->database->enable_ssl) {
$this->commands[] = "chown -R 999:999 $this->configuration_dir/ssl/server.key $this->configuration_dir/ssl/server.crt";
}
$this->commands[] = "docker compose -f $this->configuration_dir/docker-compose.yml up -d";
$this->commands[] = "echo 'Database started.'";

return remote_process($this->commands, $database->destination->server, callEventOnFinish: 'DatabaseStatusChanged');
}

private function buildStartCommand(): string
{
$command = "dragonfly --requirepass {$this->database->dragonfly_password}";

if ($this->database->enable_ssl) {
$sslArgs = [
'--tls',
'--tls_cert_file /etc/dragonfly/certs/server.crt',
'--tls_key_file /etc/dragonfly/certs/server.key',
'--tls_ca_cert_file /etc/dragonfly/certs/coolify-ca.crt',
];
$command .= ' '.implode(' ', $sslArgs);
}

return $command;
}

private function generate_local_persistent_volumes()
{
$local_persistent_volumes = [];
Expand Down
147 changes: 134 additions & 13 deletions app/Actions/Database/StartKeydb.php
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

namespace App\Actions\Database;

use App\Helpers\SslHelper;
use App\Models\SslCertificate;
use App\Models\StandaloneKeydb;
use Illuminate\Support\Facades\Storage;
use Lorisleiva\Actions\Concerns\AsAction;
Expand All @@ -17,26 +19,77 @@ class StartKeydb

public string $configuration_dir;

private ?SslCertificate $ssl_certificate = null;

public function handle(StandaloneKeydb $database)
{
$this->database = $database;

$startCommand = "keydb-server --requirepass {$this->database->keydb_password} --appendonly yes";

$container_name = $this->database->uuid;
$this->configuration_dir = database_configuration_dir().'/'.$container_name;

$this->commands = [
"echo 'Starting database.'",
"echo 'Creating directories.'",
"mkdir -p $this->configuration_dir",
"echo 'Directories created successfully.'",
];

if (! $this->database->enable_ssl) {
$this->commands[] = "rm -rf $this->configuration_dir/ssl";
SslCertificate::where('resource_type', $this->database->getMorphClass())
->where('resource_id', $this->database->id)
->delete();
$this->database->fileStorages()
->where('resource_type', $this->database->getMorphClass())
->where('resource_id', $this->database->id)
->get()
->filter(function ($storage) {
return in_array($storage->mount_path, [
'/etc/keydb/certs/server.crt',
'/etc/keydb/certs/server.key',
]);
})
->each(function ($storage) {
$storage->delete();
});
} else {
$this->commands[] = "echo 'Setting up SSL for this database.'";
$this->commands[] = "mkdir -p $this->configuration_dir/ssl";

$server = $this->database->destination->server;
$caCert = SslCertificate::where('server_id', $server->id)->where('is_ca_certificate', true)->first();

$this->ssl_certificate = SslCertificate::where('resource_type', $this->database->getMorphClass())
->where('resource_id', $this->database->id)
->first();

if (! $this->ssl_certificate) {
$this->commands[] = "echo 'No SSL certificate found, generating new SSL certificate for this database.'";
$this->ssl_certificate = SslHelper::generateSslCertificate(
commonName: $this->database->uuid,
resourceType: $this->database->getMorphClass(),
resourceId: $this->database->id,
serverId: $server->id,
caCert: $caCert->ssl_certificate,
caKey: $caCert->ssl_private_key,
configurationDir: $this->configuration_dir,
mountPath: '/etc/keydb/certs',
);
}
}

$container_name = $this->database->uuid;
$this->configuration_dir = database_configuration_dir().'/'.$container_name;

$persistent_storages = $this->generate_local_persistent_volumes();
$persistent_file_volumes = $this->database->fileStorages()->get();
$volume_names = $this->generate_local_persistent_volumes_only_volume_names();
$environment_variables = $this->generate_environment_variables();
$this->add_custom_keydb();

$startCommand = $this->buildStartCommand();

$docker_compose = [
'services' => [
$container_name => [
Expand Down Expand Up @@ -72,34 +125,67 @@ public function handle(StandaloneKeydb $database)
],
],
];

if (! is_null($this->database->limits_cpuset)) {
data_set($docker_compose, "services.{$container_name}.cpuset", $this->database->limits_cpuset);
}

if ($this->database->destination->server->isLogDrainEnabled() && $this->database->isLogDrainEnabled()) {
$docker_compose['services'][$container_name]['logging'] = generate_fluentd_configuration();
}

if (count($this->database->ports_mappings_array) > 0) {
$docker_compose['services'][$container_name]['ports'] = $this->database->ports_mappings_array;
}

$docker_compose['services'][$container_name]['volumes'] ??= [];

if (count($persistent_storages) > 0) {
$docker_compose['services'][$container_name]['volumes'] = $persistent_storages;
$docker_compose['services'][$container_name]['volumes'] = array_merge(
$docker_compose['services'][$container_name]['volumes'] ?? [],
$persistent_storages
);
}

if (count($persistent_file_volumes) > 0) {
$docker_compose['services'][$container_name]['volumes'] = $persistent_file_volumes->map(function ($item) {
return "$item->fs_path:$item->mount_path";
})->toArray();
$docker_compose['services'][$container_name]['volumes'] = array_merge(
$docker_compose['services'][$container_name]['volumes'] ?? [],
$persistent_file_volumes->map(function ($item) {
return "$item->fs_path:$item->mount_path";
})->toArray()
);
}

if (count($volume_names) > 0) {
$docker_compose['volumes'] = $volume_names;
}

if (! is_null($this->database->keydb_conf) || ! empty($this->database->keydb_conf)) {
$docker_compose['services'][$container_name]['volumes'][] = [
'type' => 'bind',
'source' => $this->configuration_dir.'/keydb.conf',
'target' => '/etc/keydb/keydb.conf',
'read_only' => true,
];
$docker_compose['services'][$container_name]['command'] = "keydb-server /etc/keydb/keydb.conf --requirepass {$this->database->keydb_password} --appendonly yes";
$docker_compose['services'][$container_name]['volumes'] = array_merge(
$docker_compose['services'][$container_name]['volumes'] ?? [],
[
[
'type' => 'bind',
'source' => $this->configuration_dir.'/keydb.conf',
'target' => '/etc/keydb/keydb.conf',
'read_only' => true,
],
]
);
}

if ($this->database->enable_ssl) {
$docker_compose['services'][$container_name]['volumes'] = array_merge(
$docker_compose['services'][$container_name]['volumes'] ?? [],
[
[
'type' => 'bind',
'source' => '/data/coolify/ssl/coolify-ca.crt',
'target' => '/etc/keydb/certs/coolify-ca.crt',
'read_only' => true,
],
]
);
}

// Add custom docker run options
Expand All @@ -112,6 +198,9 @@ public function handle(StandaloneKeydb $database)
$this->commands[] = "echo '{$readme}' > $this->configuration_dir/README.md";
$this->commands[] = "echo 'Pulling {$database->image} image.'";
$this->commands[] = "docker compose -f $this->configuration_dir/docker-compose.yml pull";
if ($this->database->enable_ssl) {
$this->commands[] = "chown -R 999:999 $this->configuration_dir/ssl/server.key $this->configuration_dir/ssl/server.crt";
}
$this->commands[] = "docker compose -f $this->configuration_dir/docker-compose.yml up -d";
$this->commands[] = "echo 'Database started.'";

Expand Down Expand Up @@ -177,4 +266,36 @@ private function add_custom_keydb()
instant_scp($path, "{$this->configuration_dir}/{$filename}", $this->database->destination->server);
Storage::disk('local')->delete("tmp/keydb.conf_{$this->database->uuid}");
}

private function buildStartCommand(): string
{
$hasKeydbConf = ! is_null($this->database->keydb_conf) && ! empty($this->database->keydb_conf);
$keydbConfPath = '/etc/keydb/keydb.conf';

if ($hasKeydbConf) {
$confContent = $this->database->keydb_conf;
$hasRequirePass = str_contains($confContent, 'requirepass');

if ($hasRequirePass) {
$command = "keydb-server $keydbConfPath";
} else {
$command = "keydb-server $keydbConfPath --requirepass {$this->database->keydb_password}";
}
} else {
$command = "keydb-server --requirepass {$this->database->keydb_password} --appendonly yes";
}

if ($this->database->enable_ssl) {
$sslArgs = [
'--tls-port 6380',
'--tls-cert-file /etc/keydb/certs/server.crt',
'--tls-key-file /etc/keydb/certs/server.key',
'--tls-ca-cert-file /etc/keydb/certs/coolify-ca.crt',
'--tls-auth-clients optional',
];
$command .= ' '.implode(' ', $sslArgs);
}

return $command;
}
}
Loading