FileSync - Advanced File Synchronization CLI Tool

A powerful, multi-threaded file synchronization tool written in Go with advanced features including real-time monitoring, progress tracking, and a modern web interface.

Features

• Multi-threaded file operations - Fast synchronization using worker pools
• Real-time progress tracking - Beautiful progress bars with detailed statistics
• File filtering - Include/exclude patterns for selective synchronization
• Configuration management - YAML/JSON configuration files
• Comprehensive logging - Multiple log levels with structured output
• Web interface - Modern dashboard for monitoring and configuration
• File watching - Real-time file system monitoring with automatic sync
• Cross-platform - Works on Windows, macOS, and Linux

Installation

Prerequisites

• Go 1.21 or later
• Git

Build from Source

# Clone the repository
git clone <repository-url>
cd filesync

# Build the binary
go build -o filesync .

# Make it executable (Linux/macOS)
chmod +x filesync

Cross-platform Build

# Build for multiple platforms
GOOS=windows GOARCH=amd64 go build -o filesync.exe .
GOOS=linux GOARCH=amd64 go build -o filesync .
GOOS=darwin GOARCH=amd64 go build -o filesync .

Quick Start

Basic Synchronization

# Sync files from source to destination
filesync sync --source /path/to/source --destination /path/to/backup

# Use multiple workers for faster sync
filesync sync --source ./data --destination ./backup --workers 8

# Dry run to see what would be copied
filesync sync --source ./data --destination ./backup --dry-run

File Filtering

# Exclude certain file types
filesync sync --source ./data --destination ./backup --exclude "*.tmp" --exclude "*.log"

# Include only specific file types
filesync sync --source ./data --destination ./backup --include "*.txt" --include "*.md"

Web Interface

# Start the web interface
filesync web --port 8080

# Access at http://localhost:8080

File Watching

# Watch directories for changes
filesync watch --config watch.yaml --port 8080

Configuration

FileSync uses YAML configuration files. The default configuration file is located at ~/.filesync.yaml.

Example Configuration

default:
  workers: 4
  verbose: false
  log_level: info
  exclude:
    - "*.tmp"
    - "*.log"
    - ".git/*"
  include: []

sync_jobs:
  - name: "daily-backup"
    source: "/home/user/documents"
    destination: "/backup/documents"
    workers: 4
    exclude:
      - "*.tmp"
      - "*.log"
    include: []
    schedule: "0 2 * * *"  # Daily at 2 AM
    enabled: true

  - name: "photos-sync"
    source: "/home/user/photos"
    destination: "/backup/photos"
    workers: 2
    exclude:
      - "*.thumb"
    include: []
    schedule: "0 3 * * *"  # Daily at 3 AM
    enabled: true

watch:
  directories:
    - path: "/home/user/documents"
      destination: "/backup/documents"
      patterns:
        - "*.txt"
        - "*.md"
      recursive: true
  port: 8080

Command Reference

Global Options

• --config <file> - Specify configuration file (default: ~/.filesync.yaml)
• --verbose, -v - Enable verbose output

Sync Command

filesync sync [options]

Required Options:

• --source, -s <path> - Source directory
• --destination, -d <path> - Destination directory

Optional Options:

• --workers, -w <number> - Number of worker threads (default: 4)
• --dry-run - Show what would be done without making changes
• --exclude, -e <pattern> - File patterns to exclude (can be specified multiple times)
• --include, -i <pattern> - File patterns to include (can be specified multiple times)

Watch Command

filesync watch [options]

Required Options:

• --config, -c <file> - Watch configuration file

Optional Options:

• --port, -p <number> - Web interface port (default: 8080)

Web Command

filesync web [options]

Optional Options:

• --host <host> - Web server host (default: localhost)
• --port, -p <number> - Web server port (default: 8080)

Web Interface

The web interface provides a modern dashboard for monitoring and managing FileSync operations.

Features

• Real-time monitoring - Live updates via WebSocket
• Job management - Create, edit, and manage sync jobs
• Manual sync - Trigger synchronization manually
• Configuration - Update default settings
• Activity logs - View recent file system activity
• Statistics - Monitor sync performance and status

Access

1. Start the web server: filesync web --port 8080
2. Open your browser: http://localhost:8080
3. Use the interface to manage jobs and monitor activity

File Patterns

FileSync supports glob patterns for include/exclude filters:

• *.txt - All text files
• *.{jpg,png,gif} - Image files
• temp/* - All files in temp directory
• **/*.log - All log files recursively
• !important.txt - Exclude specific file

Logging

FileSync provides comprehensive logging with multiple levels:

• DEBUG - Detailed debugging information
• INFO - General information about operations
• WARN - Warning messages
• ERROR - Error messages

Log Configuration

default:
  log_level: info  # debug, info, warn, error
  verbose: false

Performance Tips

1. Use appropriate worker count - More workers for faster sync, but consider system resources
2. Filter files - Use include/exclude patterns to skip unnecessary files
3. Monitor disk I/O - High I/O operations can impact system performance
4. Use dry-run - Test configurations before running actual sync

Troubleshooting

Common Issues

1. Permission denied - Ensure proper file permissions
2. Disk space - Check available space in destination
3. Network issues - For remote destinations, verify connectivity
4. File locks - Some files may be locked by other processes

Debug Mode

Enable debug logging for detailed troubleshooting:

filesync sync --source ./data --destination ./backup --verbose

Development

Project Structure

filesync/
├── cmd/           # CLI commands
├── internal/      # Internal packages
│   ├── config/    # Configuration management
│   ├── logger/    # Logging utilities
│   ├── sync/      # Core synchronization logic
│   ├── watcher/   # File system watcher
│   └── web/       # Web interface
├── web/           # Web interface files
├── main.go        # Application entry point
└── README.md      # This file

Building

# Development build
go build -o filesync .

# Release build with optimizations
go build -ldflags="-s -w" -o filesync .

# Cross-platform builds
make build-all

Testing

# Run all tests
go test ./...

# Run tests with coverage
go test -cover ./...

# Run specific package tests
go test ./internal/sync

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Submit a pull request

Support

For issues and questions:

1. Check the troubleshooting section
2. Search existing issues
3. Create a new issue with detailed information

Roadmap

• Scheduled job execution
• Incremental backup support
• Cloud storage integration
• Encryption support
• Backup verification
• Performance metrics
• Plugin system