Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 55 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,18 @@

# PROCESS WATCH
* License: (Apache 2.0), Copyright (C) 2025, Author Phil Chen
* The author or owner of this repository is not liable for damages or losses arising from your use or inability to use the code.
* The author or owner of this repository is not liable for damages or losses arising from your use or inability to use the repository/code.

## DESCRIPTION
Process Watch monitors processes and workflows in your Linux system for anomalies or situations which when arise trigger predetermined actions you designate.
Process Watch monitors individual processes in your Linux or macOS system for metrics such as CPU, Memory, Threads, File Descriptors. You can set a predetermined action of your choosing when a threshold is met.

This is useful for systems issues, automating troubleshooting, provisioning, scaling, and much more.
Process Watch is useful for diagnosing systems issues caused by specific processes, automating troubleshooting, triggering scaling, and much more.

The Open Source version of Process Watch is meant to be managed manually or with a provisioning/configuration management tool. There are future plans for a single pane of glass dashboard for easier visibility.

## PREREQUISITES
Python 3.11 or higher
* Operating Systems: Ubuntu 20.04+, macOS 15.4.1+
* Python 3.10.9+ and pip 22.3.1+

## INSTALLATION

Expand All @@ -23,59 +26,94 @@ Python 3.11 or higher
5. `/usr/local/process-watch/tools/config_tool.py`
6. `cd /usr/local/process-watch/systemd`
7. `sudo cp process_watch.service /etc/systemd/system/`
* (Make sure to edit the process_watch.service file with your chosen user and group)
8. `sudo systemctl daemon-reload`
9. `sudo systemctl enable process_watch`
10. `sudo systemctl start process_watch`

## CONFIGURATION

Configuring Process Watch is done through a configuration tool located in the tools directory. The configuration tool will give you several options to create or manage configurations.
Configuring Process Watch is done through a command line configuration tool located in the tools directory. The configuration tool will give you several options to create or manage configurations.

1. `python3 /usr/local/process-watch/tools/config_tool.py`
`python3 /usr/local/process-watch/tools/config_tool.py`

<img src="documentation/config-tool-menu.png" alt="Config Tool Menu" width="400" height="213">
<img src="documentation/config-tool-menu.png" alt="Config Tool Menu">

### Configure Process Profile Configuration
### Configure Process Profiler Configuration

Process profilers are the different functionalities Process Watch offers based on process utilization. Any time you make a new configuration via the configuration tool you must restart Process Watch by executing `sudo systemctl restart process_watch` for the new config to take effect.

You can access the Process Profile Configuration menu from the main menu by choosing the option "Create a Process Profile Configuration".
You can access the Process Profiler Configuration menu from the main menu by choosing the option "Create a Process Profiler Configuration".

<img src="documentation/configuration-creation-sub-menu.png" alt="Configuration Creation Menu" width="400" height="245">
<img src="documentation/configuration-creation-sub-menu.png" alt="Configuration Creation Menu">

### Create General Process Profiler

The General Process Profiler, will monitor the process you dictate at the interval you specify and write CPU, RSS memory usage, file descriptor, and thread count log entry in the logs directory under the naming convention `your-file-name-gen-profiler.log`.

You can create a General Process Profiler via the Process Profile Configuration menu by choosing the option "Create General Process Profiler".
You can create a General Process Profiler via the Process Profiler Configuration menu by choosing the option "Create General Process Profiler".

See the following example below for configuration options:

<img src="documentation/general-process-profiler-settings.png" alt="General Process Profiler Settings" width="500" height="223">
<img src="documentation/general-process-profiler-settings.png" alt="General Process Profiler Settings">

### Create Memory Process Profiler

The Memory Process Profiler, will monitor the memory utilization for the process you dictate at the interval you specify and write a log entry when you exceed a memory threshold you define in the logs directory under the naming convention `your-file-name-mem-profiler.log`. Memory Process Profiler can also take action when the memory threshold is exceeded. We recommend you write a shell script for the wanted action or actions upon exceeding the memory threshold.

You can create a Memory Process Profiler via the Process Profile Configuration Menu by choosing the option "Create Memory Process Profiler".
**Actions**

If you choose to have an action triggered we recommend you write a shell script for the action and put it in the actions directory. The shell script can be bash or language of your choosing as long as its **executable** and can be called by its path (ex. `/usr/local/process-watch/actions/xyz-action.sh`), also ensure the script has an interpreter defined (ex. `#!/bin/bash`). Example actions could be restarting the process, collecting troubleshooting info, notification or anything you need during when the threshold is met.

You can create a Memory Process Profiler via the Process Profiler Configuration Menu by choosing the option "Create Memory Process Profiler".

See the following example below for configuration options:

<img src="documentation/memory-process-profiler-settings.png" alt="Memory Process Profiler Settings">

### Create Thread Process Profiler

The Thread Process Profiler, will monitor the thread utilization for the process you dictate at the interval you specify and write a log entry when you exceed a thread threshold you define in the logs directory under the naming convention `your-file-name-thrd-profiler.log`. Thread Process Profiler can also take action when the thread threshold is exceeded. We recommend you write a shell script for the wanted action or actions upon exceeding the thread threshold.

**Actions**

If you choose to have an action triggered we recommend you write a shell script for the action and put it in the actions directory. The shell script can be bash or language of your choosing as long as its **executable** and can be called by its path (ex. `/usr/local/process-watch/actions/xyz-action.sh`), also ensure the script has an interpreter defined (ex. `#!/bin/bash`). Example actions could be restarting the process, collecting troubleshooting info, notification or anything you need during when the threshold is met.

You can create a Thread Process Profiler via the Process Profiler Configuration Menu by choosing the option "Create Thread Process Profiler".

See the following example below for configuration options:

<img src="documentation/memory-process-profiler-settings.png" alt="Memory Process Profiler Settings" width="750" height="334">
<img src="documentation/thread-process-profiler-settings.png" alt="Thread Process Profiler Settings">

### Create File Descriptor Process Profiler

The File Descriptor Process Profiler, will monitor the file descriptor utilization for the process you dictate at the interval you specify and write a log entry when you exceed a file descriptor threshold you define in the logs directory under the naming convention `your-file-name-fds-profiler.log`. File Descriptor Process Profiler can also take action when the file descriptor threshold is exceeded. We recommend you write a shell script for the wanted action or actions upon exceeding the file descriptor threshold.

You can create a File Descriptor Process Profiler via the Process Profile Configuration Menu by choosing the option "Create File Descriptor Process Profiler".
**Actions**

If you choose to have an action triggered we recommend you write a shell script for the action and put it in the actions directory. The shell script can be bash or language of your choosing as long as its **executable** and can be called by its path (ex. `/usr/local/process-watch/actions/xyz-action.sh`), also ensure the script has an interpreter defined (ex. `#!/bin/bash`). Example actions could be restarting the process, collecting troubleshooting info, notification or anything you need during when the threshold is met.

You can create a File Descriptor Process Profiler via the Process Profiler Configuration Menu by choosing the option "Create File Descriptor Process Profiler".

See the following example below for configuration options:

<img src="documentation/file-descriptor-process-profiler-settings.png" alt="File Descriptor Process Profiler Settings">

### Create Global System Processes Profiler

The Global System Processes Profiler, will monitor all processes globally and find the top 5 consumers of Memory (RSS), CPU, Threads, and File Descriptors and log them in the logs directory under the naming convention `glbl-sys-procs-profiler.log`.

You can create a File Descriptor Process Profiler via the Process Profiler Configuration Menu by choosing the option "Create File Descriptor Process Profiler".

See the following example below for configuration options:

<img src="documentation/memory-process-profiler-settings.png" alt="Memory Process Profiler Settings" width="750" height="334">
<img src="documentation/global-system-processess-profiler-settings.png" alt="Global System Processes Profiler Settings">


### Quick Notes

* If you are using systemd and the supplied `process_watch.service` file make sure to change the user and group to your choosing.
* Logs are stored in `/usr/local/process_watch/logs`
* Each profiler has its own log file.
* If you need to run Process Watch as a non-root user simply modify the systemd file swapping root for your user.
* If you need access to see the contents of individual Profiler Configuration files they are located at `/usr/local/process_watch/watch_list` under the configuration file name you provided upon creation.
* CPU actions are not their yet because we are still looking at the proper situation to take action on a process CPU utilization since often times it will spike to 100% however that doesn't mean a process needs action taken.
8 changes: 8 additions & 0 deletions actions/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# PROCESS WATCH ACTIONS DIRECTORY

**Actions**

You can put your action shell scripts here for easier locating, however this is not mandatory, action shell scripts can be called from anywhere.

The shell script can be bash or language of your choosing as long as its executable and can be called by its path (ex. `/usr/local/process-watch/actions/action.sh`), also ensure the script has an interpreter defined (ex. `#!/bin/bash`). Example actions could be restarting the process, collecting troubleshooting info, notification or anything you need during when the threshold is met.

Binary file modified documentation/configuration-creation-sub-menu.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions systemd/process_watch.service
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ Description=Process Monitoring and Management
After=network.target

[Service]
User=root
Group=root
User=YOURUSERHERE
Group=YOURUSERHERE
WorkingDirectory=/usr/local/process_watch
ExecStart=/usr/bin/python3 /usr/local/process_watch/process_watch.py

Expand Down
8 changes: 4 additions & 4 deletions tools/config_tool.py
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,8 @@ def display_menu():
print(f"\n{BRIGHT_GREEN}PROCESS WATCH CONFIGURATION CREATION OPTIONS{RESET}\n")
print("1. Create General Process Profiler")
print("2. Create Memory Process Profiler")
print("3. Create File Descriptor Profiler")
print("4. Create Thread Profiler")
print("3. Create Thread Process Profiler")
print("4. Create File Descriptor Process Profiler")
print("5. Create Global System Processes Profiler")
print("6. Exit")
watch_list_choice = input(f"\n{BRIGHT_CYAN}Enter your choice (1-5):{RESET}\n")
Expand All @@ -117,11 +117,11 @@ def display_menu():
break
if watch_list_choice == '3':
clear_screen()
file_descriptor_process_profiler()
thread_process_profiler()
break
if watch_list_choice == '4':
clear_screen()
thread_process_profiler()
file_descriptor_process_profiler()
break
if watch_list_choice == '5':
clear_screen()
Expand Down