Configuration and extensibility

[devwraithe]

• Allow configuration of various parameters (e.g., number of threads, crawl depth).
• Design for extensibility to add new features or modify existing ones.