pythonを用いて、Linuxのホームディレクトリを並列処理（Thread）でバックアップするプログラム
| 登録日 | ：2024/09/16 18:41 |
|---|---|
| カテゴリ | ：Python基礎 |
Homeディレクトリに多数のユーザがいる場合、バックアップに時間がかかる。
バックアップサーバとメインサーバ間はインフィニバンドで接続してデータ転送速度が良いため、IOバウンドがボトルネックとなる。そのため、できる限り帯域を効率的に用いるには、Threadなどでバックアップ処理を並列化したほうが良い。
Home領域の増減もあるため、pythonを用いて自動的にhome領域のディレクトリを取得して、Threadでバックアップ処理を実施するプログラムを考えてみた。
フォルダ構成
[root@nis1 backup_script]# ll
-rwxr-xr-x. 1 root root 7452 9月 16 18:40 backup_homedir.py
drwxr-xr-x. 4 root root 55 9月 16 17:18 config
drwxr-xr-x. 2 root root 66 9月 16 17:19 log
drwxr-xr-x. 3 root root 59 9月 16 17:48 script
バックアップログラム
まず、バックアップを行うシェル
script/backup_script_args.sh
#!/bin/bash
# 使用方法を表示する関数
usage() {
echo "Usage: $0 <source_directory>"
echo "Example: $0 /home/user/data"
exit 1
}
# 引数チェック
if [ $# -ne 1 ]; then
usage
fi
# バックアップ元とバックアップ先のディレクトリを設定
SOURCE="$1"
DEST="/backup_dir"
# SOURCEディレクトリの存在確認
if [ ! -d "$SOURCE" ]; then
echo "Error: Source directory '$SOURCE' does not exist."
exit 1
fi
# 日付と曜日を取得
DATE=$(date +%Y-%m-%d)
DOW=$(date +%u) # 曜日を数字で取得 (1-7, 1が月曜)
# 日曜日（7）の場合はフルバックアップ
if [ "$DOW" -eq 7 ]; then
# フルバックアップディレクトリの作成
BACKUP_DIR="$DEST/full/$SOURCE"
mkdir -p "$BACKUP_DIR"
# フルバックアップの実行
rsync -av --delete "$SOURCE/" "$BACKUP_DIR"
else
# 増分バックアップディレクトリの作成
BACKUP_DIR="$DEST/incrementa/$SOURCE"
mkdir -p "$BACKUP_DIR"
# 最新のフルバックアップを見つける
LATEST_FULL="$DEST/full/$SOURCE"
# 増分バックアップの実行
rsync -av --delete --link-dest="$LATEST_FULL" "$SOURCE/" "$BACKUP_DIR"
fi
echo "Backup completed successfully."
次にpythonプログラム
backup_homedir.py
#!/usr/bin/python3
from abc import ABC, abstractmethod
import subprocess
from subprocess import PIPE
import queue
import threading
import logging
import time
import datetime
import signal
import os
import sys
import gc
import socket
dir_path = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.append(dir_path)
from config import settings
"""
created by Tagawa.
submit shell command
version 2024.09.17.1
"""
logging.basicConfig(
filename=settings.LOG_FILE,
level=logging.INFO,
format='%(threadName)s: %(message)s')
logger = logging.getLogger(__name__)
logger.debug({'add path': dir_path})
class ShellCommand(object):
def __init__(self, dt_now, timeout: int, command: str):
self.stdout = False
self.stderr = False
self.returncode = False
self.command = False
self._timeout = timeout
self._command = command
self._dt_now = dt_now
self._command_result = False
self._errlog = False
def submit_command(self, command):
self.command = command
result = subprocess.run(self.command, shell=True, stdout=PIPE, stderr=PIPE)
self.stdout = result.stdout.decode('utf-8')
self.stderr = result.stderr.decode('utf-8')
self.returncode = result.returncode
if result.returncode != 0:
raise Exception(self.stderr)
def execute_command(self):
try:
self.submit_command(self._command)
self._command_result = self.stdout
except Exception as e:
self._command_result = self.stderr
self._errlog = str(e)
logger.error({
'time': self._dt_now,
'status': 'failed',
'action':'ExceuteShellComand',
'error': self._errlog,
'command': self._command})
class ShellCommand_signal_timeout(ShellCommand):
def _signal_handler(self, signum, frame):
logger.error({
'statud': "Time out!",
'action': 'ExecuteShellCommand_signal'})
raise Exception("No route to host")
def execute_command(self):
signal.signal(signal.SIGALRM, self._signal_handler)
signal.alarm(self._timeout)
try:
self.submit_command(self._command)
self._command_result = self.stdout
except Exception as e:
self._command_result = self.stderr
self._errlog = str(e)
logger.error({
'time': self._dt_now,
'status': 'failed',
'action':'ExceuteShellComand_signal',
'error': self._errlog,
'command': self._command})
class FetchHomeDir(object):
def __init__(self, dt_now, timeout, home):
self._dt_now = dt_now
self._timeout = timeout
self._home = home
self._status = None
self._command = 'ls -a ' + home
self.shell = ShellCommand(dt_now, timeout, self._command)
self.homedirs = []
def run_command(self):
self.shell.execute_command()
#print({'return command result': self.shell._command_result})
if not self.shell._errlog and self.shell._command_result != "":
self._status = 'success'
homedirs = self.shell._command_result.split('\n')
for _home in homedirs[2:]:
# skip '.', '..'
if _home != "":
_path = self._home + '/' + _home
#print(_path)
self.homedirs.append(_path)
else:
self._status = 'failed'
logger.error({
'time': self._dt_now,
'status': self._status,
'action': FetchHomeDir,
'command': self._command_sinfo,
'home': self._home})
if settings.DEBUG:
#print(f'{self._status}: {__file__} from {self._home}')
print(f'{self._status}: FetchHomeDir from {self._home}')
class IThreadWorker(ABC):
def __init__(self, dt_now, queue, num_of_thread, timeout):
self.dt_now = dt_now
self.queue = queue
self.num_of_thread = num_of_thread
self.timeout = timeout
self.command = None
def run(self):
ts = []
for _ in range(self.num_of_thread):
t = threading.Thread(target=self.worker)
t.start()
ts.append(t)
[self.queue.put(None) for _ in range(len(ts))]
[t.join() for t in ts]
@abstractmethod
def worker(self):
logging.debug('start')
while True:
item = self.queue.get()
if item is None:
break
print({'thread': item})
self.some_process()
self.queue.task_done()
logging.debug('end')
def some_process(self):
pass
class ThreadHomeDirChecker(IThreadWorker):
def __init__(self, dt_now, queue, num_of_thread, timeout, backup_script):
super().__init__(dt_now, queue, num_of_thread, timeout)
#self.command = 'ls -l '
self.command = backup_script + ' '
self.result = []
def worker(self):
logging.debug('start')
while True:
path_dir = self.queue.get()
if path_dir is None:
break
self.check_home_dir(path_dir)
self.queue.task_done()
logging.debug('end')
def check_home_dir(self, path_dir):
try:
_command = self.command + path_dir
_shell = ShellCommand(self.dt_now, self.timeout, _command)
_shell.execute_command()
logger.info({
'date': self.dt_now,
'path': path_dir,
'result': _shell._command_result.split('\n'),
})
except Exception as e:
print({'command Error': str(e)})
logger.error({
'time': self.dt_now,
'status': 'failed',
'action': 'ThreadHomeDirChecker',
'message': str(e),
'path': path_dir})
"""
test code
"""
def test_shell_command():
timeout = settings.TIMEOUT
dt_now = datetime.datetime.now().strftime('%Y/%m/%d %H:%M:%S')
home = '/home'
check_home_dir = FetchHomeDir(dt_now, timeout, home)
check_home_dir.run_command()
print({'result': check_home_dir.homedirs})
if __name__ == '__main__':
#test_shell_command()
home = settings.HOME_DIR
timeout = settings.TIMEOUT
threads = settings.THREADING_NUM
backup_script = settings.BACKUP_SCRIPT
dt_now = datetime.datetime.now().strftime('%Y/%m/%d %H:%M:%S')
fetch_home_dir = FetchHomeDir(dt_now, timeout, home)
homedirs_queue = queue.Queue()
start = time.time()
logger.info({'backup start date': dt_now})
fetch_home_dir.run_command()
for homedir in fetch_home_dir.homedirs:
homedirs_queue.put(homedir)
thread_homedir_checker = ThreadHomeDirChecker(
dt_now, homedirs_queue, threads, timeout, backup_script)
thread_homedir_checker.run()
end = time.time()
logger.info({'thread action': 'end'})
print('thread time: {: 4f}\n'.format(end - start))
logger.info({
'time': dt_now,
'action': 'threads',
'status': 'finished',
'elapsed time': 'time: {: 4f}'.format(end - start)})
del fetch_home_dir,thread_homedir_checker,homedirs_queue
gc.collect()
configファイル
config/settings.py
"""
created by Tagawa
version 2024.9.17.1
Please Change Option
number of threading -> integer
Timeout -> integer
"""
LOG_FILE = "/root/tools/python/backup_script/log/check_result.log"
BACKUP_SCRIPT = "/root/tools/python/backup_script/script/backup_script_args.sh"
THREADING_ON = True
THREADING_NUM = 4
TIMEOUT = 1
DEBUG = True
TEST_CODE = False
HOME_DIR = '/home'
実行ログ
logフォルダに以下のように結果が出力される
[root@nis1 backup_script]# ll log
合計 284
-rw-r--r--. 1 root root 268143 9月 16 18:40 check_result.log
[root@nis1 backup_script]# cat log/check_result.log
MainThread: {'thread action': 'end'}
MainThread: {'time': '2024/09/16 20:45:58', 'action': 'threads', 'status': 'finished', 'elapsed time': 'time: 1.392312'}
MainThread: {'backup start date': '2024/09/16 20:47:45'}
Thread-1: {'date': '2024/09/16 20:47:45', 'path': '/home/n1001', 'result': ['sending incremental file list', './', '', 'sent 292 bytes received 23 bytes 630.00 bytes/sec', 'total size is 945 speedup is 3.00', 'Backup completed successfully.', '']}
Thread-2: {'date': '2024/09/16 20:47:45', 'path': '/home/n1002', 'result': ['sending incremental file list', './', '', 'sent 616 bytes received 25 bytes 1,282.00 bytes/sec', 'total size is 10,453 speedup is 16.31', 'Backup completed successfully.', '']}
Thread-3: {'date': '2024/09/16 20:47:45', 'path': '/home/n1003', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-4: {'date': '2024/09/16 20:47:45', 'path': '/home/n1004', 'result': ['sending incremental file list', './', '', 'sent 258 bytes received 23 bytes 562.00 bytes/sec', 'total size is 570 speedup is 2.03', 'Backup completed successfully.', '']}
Thread-5: {'date': '2024/09/16 20:47:45', 'path': '/home/n1005', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-6: {'date': '2024/09/16 20:47:45', 'path': '/home/n1006', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-7: {'date': '2024/09/16 20:47:45', 'path': '/home/n1007', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-8: {'date': '2024/09/16 20:47:45', 'path': '/home/n1008', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-1: {'date': '2024/09/16 20:47:45', 'path': '/home/n1009', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-9: {'date': '2024/09/16 20:47:45', 'path': '/home/n1010', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-2: {'date': '2024/09/16 20:47:45', 'path': '/home/n1011', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-3: {'date': '2024/09/16 20:47:45', 'path': '/home/n1013', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-10: {'date': '2024/09/16 20:47:45', 'path': '/home/n1012', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-4: {'date': '2024/09/16 20:47:45', 'path': '/home/n1014', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-5: {'date': '2024/09/16 20:47:45', 'path': '/home/n1015', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-6: {'date': '2024/09/16 20:47:45', 'path': '/home/n1016', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-7: {'date': '2024/09/16 20:47:45', 'path': '/home/n1017', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-8: {'date': '2024/09/16 20:47:45', 'path': '/home/n1018', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-1: {'date': '2024/09/16 20:47:45', 'path': '/home/n1019', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-9: {'date': '2024/09/16 20:47:45', 'path': '/home/n1020', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-2: {'date': '2024/09/16 20:47:45', 'path': '/home/n1021', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-3: {'date': '2024/09/16 20:47:45', 'path': '/home/n1022', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-10: {'date': '2024/09/16 20:47:45', 'path': '/home/n1023', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-4: {'date': '2024/09/16 20:47:45', 'path': '/home/n1024', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-5: {'date': '2024/09/16 20:47:45', 'path': '/home/n1025', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-6: {'date': '2024/09/16 20:47:45', 'path': '/home/n1026', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-7: {'date': '2024/09/16 20:47:45', 'path': '/home/n1027', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-8: {'date': '2024/09/16 20:47:45', 'path': '/home/n1028', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-1: {'date': '2024/09/16 20:47:45', 'path': '/home/n1029', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-9: {'date': '2024/09/16 20:47:45', 'path': '/home/n1030', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-2: {'date': '2024/09/16 20:47:45', 'path': '/home/n1031', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-3: {'date': '2024/09/16 20:47:45', 'path': '/home/n1032', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-10: {'date': '2024/09/16 20:47:45', 'path': '/home/n1033', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-4: {'date': '2024/09/16 20:47:45', 'path': '/home/n1034', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-5: {'date': '2024/09/16 20:47:45', 'path': '/home/n1035', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 147.33 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-6: {'date': '2024/09/16 20:47:45', 'path': '/home/n1036', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 147.33 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-7: {'date': '2024/09/16 20:47:45', 'path': '/home/n1037', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-8: {'date': '2024/09/16 20:47:45', 'path': '/home/n1038', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-1: {'date': '2024/09/16 20:47:45', 'path': '/home/n1039', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-9: {'date': '2024/09/16 20:47:45', 'path': '/home/n1040', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-2: {'date': '2024/09/16 20:47:45', 'path': '/home/n1041', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-3: {'date': '2024/09/16 20:47:45', 'path': '/home/n1042', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-10: {'date': '2024/09/16 20:47:45', 'path': '/home/n1043', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-4: {'date': '2024/09/16 20:47:45', 'path': '/home/n1044', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-5: {'date': '2024/09/16 20:47:45', 'path': '/home/n1045', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-6: {'date': '2024/09/16 20:47:45', 'path': '/home/n1046', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-7: {'date': '2024/09/16 20:47:45', 'path': '/home/n1047', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-8: {'date': '2024/09/16 20:47:45', 'path': '/home/n1048', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-1: {'date': '2024/09/16 20:47:45', 'path': '/home/n1049', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-9: {'date': '2024/09/16 20:47:45', 'path': '/home/n1050', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-2: {'date': '2024/09/16 20:47:45', 'path': '/home/n1051', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-3: {'date': '2024/09/16 20:47:45', 'path': '/home/n1052', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-10: {'date': '2024/09/16 20:47:45', 'path': '/home/n1053', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-4: {'date': '2024/09/16 20:47:45', 'path': '/home/n1054', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-5: {'date': '2024/09/16 20:47:45', 'path': '/home/n1055', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-6: {'date': '2024/09/16 20:47:45', 'path': '/home/n1056', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-7: {'date': '2024/09/16 20:47:45', 'path': '/home/n1057', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-8: {'date': '2024/09/16 20:47:45', 'path': '/home/n1058', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-1: {'date': '2024/09/16 20:47:45', 'path': '/home/n1059', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-9: {'date': '2024/09/16 20:47:45', 'path': '/home/n1060', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-2: {'date': '2024/09/16 20:47:45', 'path': '/home/n1061', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-3: {'date': '2024/09/16 20:47:45', 'path': '/home/n1062', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-10: {'date': '2024/09/16 20:47:45', 'path': '/home/n1063', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-4: {'date': '2024/09/16 20:47:45', 'path': '/home/n1064', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-5: {'date': '2024/09/16 20:47:45', 'path': '/home/n1065', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-6: {'date': '2024/09/16 20:47:45', 'path': '/home/n1066', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-7: {'date': '2024/09/16 20:47:45', 'path': '/home/n1067', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-8: {'date': '2024/09/16 20:47:45', 'path': '/home/n1068', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-1: {'date': '2024/09/16 20:47:45', 'path': '/home/n1069', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-9: {'date': '2024/09/16 20:47:45', 'path': '/home/n1070', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-2: {'date': '2024/09/16 20:47:45', 'path': '/home/n1071', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-3: {'date': '2024/09/16 20:47:45', 'path': '/home/n1072', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-10: {'date': '2024/09/16 20:47:45', 'path': '/home/n1073', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-4: {'date': '2024/09/16 20:47:45', 'path': '/home/n1074', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-5: {'date': '2024/09/16 20:47:45', 'path': '/home/n1075', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-6: {'date': '2024/09/16 20:47:45', 'path': '/home/n1076', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-7: {'date': '2024/09/16 20:47:45', 'path': '/home/n1077', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-8: {'date': '2024/09/16 20:47:45', 'path': '/home/n1078', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-1: {'date': '2024/09/16 20:47:45', 'path': '/home/n1079', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-9: {'date': '2024/09/16 20:47:45', 'path': '/home/n1080', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-2: {'date': '2024/09/16 20:47:45', 'path': '/home/n1081', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-3: {'date': '2024/09/16 20:47:45', 'path': '/home/n1082', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-10: {'date': '2024/09/16 20:47:45', 'path': '/home/n1083', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-4: {'date': '2024/09/16 20:47:45', 'path': '/home/n1084', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-5: {'date': '2024/09/16 20:47:45', 'path': '/home/n1085', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-6: {'date': '2024/09/16 20:47:45', 'path': '/home/n1086', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-7: {'date': '2024/09/16 20:47:45', 'path': '/home/n1087', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-8: {'date': '2024/09/16 20:47:45', 'path': '/home/n1088', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-1: {'date': '2024/09/16 20:47:45', 'path': '/home/n1089', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-9: {'date': '2024/09/16 20:47:45', 'path': '/home/n1090', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-2: {'date': '2024/09/16 20:47:45', 'path': '/home/n1091', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-3: {'date': '2024/09/16 20:47:45', 'path': '/home/n1092', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-10: {'date': '2024/09/16 20:47:45', 'path': '/home/n1093', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-4: {'date': '2024/09/16 20:47:45', 'path': '/home/n1094', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-5: {'date': '2024/09/16 20:47:45', 'path': '/home/n1095', 'result': ['sending incremental file list', './', '', 'sent 429 bytes received 23 bytes 904.00 bytes/sec', 'total size is 694 speedup is 1.54', 'Backup completed successfully.', '']}
Thread-6: {'date': '2024/09/16 20:47:45', 'path': '/home/n1096', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-7: {'date': '2024/09/16 20:47:45', 'path': '/home/n1097', 'result': ['sending incremental file list', './', '', 'sent 198 bytes received 22 bytes 440.00 bytes/sec', 'total size is 535 speedup is 2.43', 'Backup completed successfully.', '']}
Thread-8: {'date': '2024/09/16 20:47:45', 'path': '/home/n1098', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-1: {'date': '2024/09/16 20:47:45', 'path': '/home/n1099', 'result': ['sending incremental file list', './', '', 'sent 199 bytes received 22 bytes 442.00 bytes/sec', 'total size is 535 speedup is 2.42', 'Backup completed successfully.', '']}
Thread-9: {'date': '2024/09/16 20:47:45', 'path': '/home/n1100', 'result': ['sending incremental file list', './', '', 'sent 280 bytes received 23 bytes 606.00 bytes/sec', 'total size is 585 speedup is 1.93', 'Backup completed successfully.', '']}
Thread-2: {'date': '2024/09/16 20:47:45', 'path': '/home/user01', 'result': ['sending incremental file list', './', '', 'sent 8,713 bytes received 160 bytes 17,746.00 bytes/sec', 'total size is 15,753,697 speedup is 1,775.46', 'Backup completed successfully.', '']}
MainThread: {'thread action': 'end'}
MainThread: {'time': '2024/09/16 20:47:45', 'action': 'threads', 'status': 'finished', 'elapsed time': 'time: 1.268222'}
[root@nis1 backup_script]#
Apendix
バックアップスクリプトについて
このスクリプト部分は増分バックアップを実行するための重要な手順を示しています。以下に各行の説明を記します：
BACKUP_DIR="$DEST/incrementa/$SOURCE"
増分バックアップの保存先ディレクトリを設定します。
$DESTはバックアップの基本ディレクトリ、incrementaは増分バックアップ用のサブディレクトリ、$SOURCEはバックアップ元のパスを表します。
mkdir -p "$BACKUP_DIR"
増分バックアップ用のディレクトリを作成します。
-pオプションにより、必要な親ディレクトリも同時に作成されます。
LATEST_FULL="$DEST/full/$SOURCE"
最新のフルバックアップのパスを設定します。
これは増分バックアップの基準点となります。
# 増分バックアップの実行
rsync -av --delete --link-dest="$LATEST_FULL" "$SOURCE/" "$BACKUP_DIR"
実際の増分バックアップを実行するrsyncコマンドです。
-a：アーカイブモードで、ファイルの属性を保持します。
-v：詳細な出力を表示します。
--delete：送信元にないファイルを宛先から削除します。
--link-dest="$LATEST_FULL"：最新のフルバックアップと変更がないファイルはハードリンクを作成します。
"$SOURCE/"：バックアップ元のディレクトリです。
"$BACKUP_DIR"：バックアップ先のディレクトリです。
このスクリプトは、効率的な増分バックアップを実現します。最新のフルバックアップと比較して変更されたファイルのみをコピーし、変更のないファイルはハードリンクを作成することで、ストレージ容量を節約しつつ、完全なバックアップを維持します
# フルバックアップの実行
rsync -av --delete "$SOURCE/" "$BACKUP_DIR"
このコマンドはフルバックアップを実行するためのrsyncコマンドです。各部分について説明します：
rsync: ファイル同期ツールです。
-av:
-a: アーカイブモードを意味し、再帰的にディレクトリをコピーし、シンボリックリンク、パーミッション、タイムスタンプ、所有者、グループ情報を保持します。
-v: 詳細な出力を提供します。
--delete:
このオプションは、送信元（$SOURCE）に存在しないファイルやディレクトリを宛先（$BACKUP_DIR）から削除します。
これにより、バックアップ先が送信元の正確なコピーになります。
"$SOURCE/":
バックアップ元のディレクトリです。
末尾のスラッシュ（/）は、ディレクトリの内容をコピーすることを意味します。
"$BACKUP_DIR": バックアップ先のディレクトリです。
このコマンドの動作：
$SOURCEディレクトリの内容を$BACKUP_DIRに完全にコピーします。
既存のファイルは更新され、新しいファイルが追加されます。
$SOURCEに存在しないファイルは$BACKUP_DIRから削除されます。
詳細な進行状況が表示されます。
注意点：
--deleteオプションは慎重に使用する必要があります。誤って重要なファイルを削除する可能性があります。
初回実行時や大規模な変更がある場合、処理に時間がかかる可能性があります。
セキュリティのため、重要なデータをバックアップする際は暗号化された通信チャネルの使用を検討してください。
このコマンドは、$SOURCEの完全で正確なバックアップを$BACKUP_DIRに作成するのに適しています。