BigW Consortium Gitlab
Skip to content
Projects
Groups
Snippets
Help
This project
Loading...
Sign in / Register
Toggle navigation
G
gitlab-ce
Project
Overview
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
Registry
Registry
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Commits
Issue Boards
Open sidebar
Forest Godfrey
gitlab-ce
Commits
90ddf140
Commit
90ddf140
authored
Oct 06, 2015
by
Jacob Vosmaer
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Reduce disk IO during SQL backup
By using light gzip compression we can save a lot of disk IO during the backup.
parent
d3734fbd
Show whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
26 additions
and
31 deletions
+26
-31
database.rb
lib/backup/database.rb
+24
-29
manager.rb
lib/backup/manager.rb
+2
-2
No files found.
lib/backup/database.rb
View file @
90ddf140
...
...
@@ -2,26 +2,27 @@ require 'yaml'
module
Backup
class
Database
attr_reader
:config
,
:db_
dir
attr_reader
:config
,
:db_
file_name
def
initialize
@config
=
YAML
.
load_file
(
File
.
join
(
Rails
.
root
,
'config'
,
'database.yml'
))[
Rails
.
env
]
@db_
dir
=
File
.
join
(
Gitlab
.
config
.
backup
.
path
,
'db
'
)
@db_
file_name
=
File
.
join
(
Gitlab
.
config
.
backup
.
path
,
'database.sql.gz
'
)
end
def
dump
FileUtils
.
rm_rf
(
@db_dir
)
# Ensure the parent dir of @db_dir exists
FileUtils
.
rm_f
(
db_file_name
)
compress_rd
,
compress_wr
=
IO
.
pipe
compress_pid
=
spawn
(
*
%W(gzip -1 -c)
,
in:
compress_rd
,
out:
[
db_file_name
,
'w'
,
0600
])
compress_rd
.
close
FileUtils
.
mkdir_p
(
Gitlab
.
config
.
backup
.
path
)
# Fail if somebody raced to create @db_dir before us
FileUtils
.
mkdir
(
@db_dir
,
mode:
0700
)
success
=
case
config
[
"adapter"
]
dump_pid
=
case
config
[
"adapter"
]
when
/^mysql/
then
$progress
.
print
"Dumping MySQL database
#{
config
[
'database'
]
}
... "
# Workaround warnings from MySQL 5.6 about passwords on cmd line
ENV
[
'MYSQL_PWD'
]
=
config
[
"password"
].
to_s
if
config
[
"password"
]
s
ystem
(
'mysqldump'
,
*
mysql_args
,
config
[
'database'
],
out:
db_file_name
)
s
pawn
(
'mysqldump'
,
*
mysql_args
,
config
[
'database'
],
out:
compress_wr
)
when
"postgresql"
then
$progress
.
print
"Dumping PostgreSQL database
#{
config
[
'database'
]
}
... "
pg_env
...
...
@@ -30,48 +31,42 @@ module Backup
pgsql_args
<<
"-n"
pgsql_args
<<
Gitlab
.
config
.
backup
.
pg_schema
end
s
ystem
(
'pg_dump'
,
*
pgsql_args
,
config
[
'database'
],
out:
db_file_name
)
s
pawn
(
'pg_dump'
,
*
pgsql_args
,
config
[
'database'
],
out:
compress_wr
)
end
report_success
(
success
)
abort
'Backup failed'
unless
success
compress_wr
.
close
success
=
[
compress_pid
,
dump_pid
].
all?
{
|
pid
|
Process
.
waitpid
(
pid
);
$?
.
success?
}
$progress
.
print
'Compressing database ... '
success
=
system
(
'gzip'
,
db_file_name
)
report_success
(
success
)
abort
'Backup failed
: compress error
'
unless
success
abort
'Backup failed'
unless
success
end
def
restore
$progress
.
print
'Decompressing database ... '
success
=
system
(
'gzip'
,
'-d'
,
db_file_name_gz
)
report_success
(
success
)
abort
'Restore failed: decompress error'
unless
success
decompress_rd
,
decompress_wr
=
IO
.
pipe
decompress_pid
=
spawn
(
*
%W(gzip -cd)
,
out:
decompress_wr
,
in:
db_file_name
)
decompress_wr
.
close
success
=
case
config
[
"adapter"
]
restore_pid
=
case
config
[
"adapter"
]
when
/^mysql/
then
$progress
.
print
"Restoring MySQL database
#{
config
[
'database'
]
}
... "
# Workaround warnings from MySQL 5.6 about passwords on cmd line
ENV
[
'MYSQL_PWD'
]
=
config
[
"password"
].
to_s
if
config
[
"password"
]
s
ystem
(
'mysql'
,
*
mysql_args
,
config
[
'database'
],
in:
db_file_name
)
s
pawn
(
'mysql'
,
*
mysql_args
,
config
[
'database'
],
in:
decompress_rd
)
when
"postgresql"
then
$progress
.
print
"Restoring PostgreSQL database
#{
config
[
'database'
]
}
... "
pg_env
s
ystem
(
'psql'
,
config
[
'database'
],
'-f'
,
db_file_name
)
s
pawn
(
'psql'
,
config
[
'database'
],
in:
decompress_rd
)
end
decompress_rd
.
close
success
=
[
decompress_pid
,
restore_pid
].
all?
{
|
pid
|
Process
.
waitpid
(
pid
);
$?
.
success?
}
report_success
(
success
)
abort
'Restore failed'
unless
success
end
protected
def
db_file_name
File
.
join
(
db_dir
,
'database.sql'
)
end
def
db_file_name_gz
File
.
join
(
db_dir
,
'database.sql.gz'
)
end
def
mysql_args
args
=
{
'host'
=>
'--host'
,
...
...
lib/backup/manager.rb
View file @
90ddf140
...
...
@@ -150,11 +150,11 @@ module Backup
private
def
backup_contents
folders_to_backup
+
[
"uploads.tar.gz"
,
"builds.tar.gz"
,
"backup_information.yml"
]
folders_to_backup
+
[
"
database.sql.gz"
,
"
uploads.tar.gz"
,
"builds.tar.gz"
,
"backup_information.yml"
]
end
def
folders_to_backup
folders
=
%w{repositories
db
}
folders
=
%w{repositories}
if
ENV
[
"SKIP"
]
return
folders
.
reject
{
|
folder
|
ENV
[
"SKIP"
].
include?
(
folder
)
}
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment