GeoSync

GeoSync is a simple command-line application that can be used to synchronize your GIS and style files on disk with a running GeoServer’s catalog, so that they can be served up using web mapping protocols.

GeoSync can be used to:

Create workspaces
Add layers to new or existing workspaces (along with their associated coveragestores, coverages, datastores, and featuretypes)
Add layergroups to new or existing workspaces and associate layers with them
Add styles to new or existing workspaces (or update existing styles with new definitions)
Assign styles to layers using pattern matching on layer names
Remove workspaces (along with all of the objects associated with them)

GeoSync works by scanning the directory trees under the data-dir that you give it for GIS files and under the style-dir that you give it for style files. For each GIS or style file that it finds, GeoSync will compute the set of REST API calls necessary to load that layer or style into a running GeoServer instance and will then send those requests to the specified GeoServer using the authentication credentials that you provide.

At this time, GeoSync supports the following GIS file formats:

GeoTIFF
Shapefile
ImageMosaic

At this time, GeoSync supports the following style file formats:

CSS

Installation Requirements

Java Development Kit (version 11+)
Clojure CLI tools
PostgreSQL (version 12) (needed for ImageMosaic support)
PostGIS (version 3) (needed for ImageMosaic support)

PostgreSQL Database Setup

PostgreSQL needs to be installed on the machine that will be hosting the GeoServer to support ImageMosaic rasters. Once the PostgreSQL database server is running on your machine, you can initialize a GeoSync database in it by performing the following steps.

Configure Database Connection Settings in config.edn

GeoSync looks for a file called config.edn in the top level directory of this repository and reads it in to load the database connection settings it will use for communicating with your PostgreSQL server.

A sample file called config.default.edn is provided with this repository to get you started. To begin, simply copy it to config.edn. Then edit it as necessary for your local setup.

Create and Populate the Database

Once your config.edn file has been prepared, you can navigate to the top level repository directory and run the database build command as follows:

clojure -M:build-db build-all

This will begin by creating a new database and role based on your settings in config.edn and then add the postgis extension to it. Next, the script will populate the database with the functions that are necessary for storing and processing GeoServer’s data.

It’s important to note that when you are dealing with ImageMosaic workspaces, you must make sure to use GeoSync to deregister them. The GeoServer web UI gives you an option to deregister workspaces using its interface, but due to the ImageMosaic data stored in GeoSync’s PostgreSQL database, you must use GeoSync to deregister any ImageMosaic workspaces in order to make sure the database is updated properly.

Performance Settings (Optional)

If you want to improve the performance of your PostgreSQL server, one way is to visit the pgtune website and input your system settings. This will generate a set of configuration options that you can add to the postgresql.conf file in your system-specific PostgreSQL data directory.

If you would like to keep these settings separate for your own reference, you can add them to a file called performance.conf that you should place in your PostgreSQL data directory. You can then import these settings into postgresql.conf by adding this line to the end of that file:

include = 'performance.conf'

Note that the PostgreSQL server will need to be restarted for these changes to take effect.

Testing

GeoSync comes with a test suite. You can run it with this command:

clojure -M:test-runner

As we improve and expand upon the codebase, more tests will be written in order to increase code-coverage and make sure that we don’t introduce any regressions.

Quickstart

To compile and run the application, navigate to the top level project directory and run:

clojure -M:run

This will print out the following usage message:

geosync: Load a nested directory tree of GeoTIFFs and Shapefiles into a running GeoServer instance.
Copyright © 2020-2025 Spatial Informatics Group, LLC.

Usage:
  -c, --config-file EDN           Path to an EDN file containing a map of these parameters
  -g, --geoserver-rest-uri URI    URI of your GeoServer's REST extensions
  -u, --geoserver-username USER   GeoServer admin username
  -p, --geoserver-password PASS   GeoServer admin password
  -w, --geoserver-workspace WS    Workspace name to receive the new GeoServer layers
  -d, --data-dir DIR              Path to the directory containing your GIS files
  -s, --style-dir DIR             Path to the directory containing your style files
  -a, --action ACTION             GeoServer action: either "add" or "remove". Required in CLI mode.
  -O, --overwrite-styles          If true, already existing styles will have their definitions overwritten
  -A, --autostyle-layers          If true, GeoSync will match layers with existing styles based on the style and layer names
  -h, --geosync-server-host HOST  Hostname to advertise in server responses
  -P, --geosync-server-port PORT  Server port to listen on for incoming requests
  -o, --log-dir PATH              Path to log files

You can run GeoSync in one of three ways:

Pass options on the command line.

# CLI register mode: Load (add) a single directory tree into a GeoServer workspace and exit
clojure -M:run -g http://localhost:8080/geoserver/rest -u admin -p geoserver -w demo -a "add" -d /data

# CLI deregister mode: Remove a single workspace from GeoServer and exit
clojure -M:run -g http://localhost:8080/geoserver/rest -u admin -p geoserver -w demo -a "remove"

# Server mode: Listen on a port for JSON requests (see section "Server Mode" below for more info)
clojure -M:run -g http://localhost:8080/geoserver/rest -u admin -p geoserver -h geosync.mydomain.org -P 31337

Please note that the -d option must receive a file path that is available on the JVM classpath. The easiest way to accomplish this is to use an absolute path for your filesystem. Also note that without the -c option there are certain options/features that you can not make use of (such as :styles and :layer-groups).

Pass the -c option with an EDN file containing all of the other options formatted as an EDN map of keywords to values. You may also pass the optional :styles and :layer-groups options in this way. See resources/sample-config.edn for an example.
```
clojure -M:run -c resources/sample-config.edn
    
```
Pass the -c option with an EDN config file and also pass any other options on the command line that should override the settings in the EDN file.
```
clojure -M:run -c resources/sample-config.edn -w demo -a "add" -d /data
    
```

Operation Modes

GeoSync comes with two main operation modes:

CLI
Server

The CLI mode is simple to use and understand as it launches a new process, performs a single action, and exits.

The server mode can execute the same actions as the CLI mode, but rather than performing an action at start up, it simply binds to the user-specified port and listens for incoming requests. Each time a request is received, it is sorted into either a priority or standby job queue. Jobs are then executed in FIFO order with all priority jobs being completed before any standby jobs are processed.

The server mode is particularly useful if you need to submit many GeoSync requests over brief timespans. Instead of spawning one GeoSync process per action (and thus repeatedly incurring a startup time penalty and consuming more and more memory per process), you can start a single GeoSync server and send all of your jobs to it. They will be prioritized and run all from within one process with a fixed size memory footprint.

More details on these operation modes are provided below.

CLI Mode

The CLI mode is the simplest operation mode of GeoSync. It allows its user to perform on-demand additions or removals of GIS layers and styles to workspaces on the specified GeoServer.

While using GeoSync via the terminal, you can specify options using the flags and switches of the CLI itself or by providing a .edn configuration file. These options will be merged at run-time in order to produce the final version of the configuration. Options specified in the CLI take precedence over the options specified in the configuration file.

In CLI mode, it’s mandatory to specify the following arguments in the configuration file or via the command line:

geoserver-rest-uri
geoserver-username
geoserver-password
geoserver-workspace
action

If action is set to "add", you will also need to specify either or both of the following two parameters:

data-dir
style-dir

If style-dir is provided, you may also choose to set overwrite-styles to true or false.

You can also set autostyle-layers option to true. If you do, GeoSync will automatically assign existing or new styles to the layers being loaded.

Refer to the usage message in the Quickstart section above for more details on each argument.

Server Mode

In server-mode, GeoSync will listen for incoming requests on the port specified with the -P option. Each request should be a single newline-terminated JSON object matching one of these forms:

Add all GIS files under a directory tree as layers under a (possibly new) workspace

{"action":             "add",
 "dataDir":            "/data",
 "geoserverWorkspace": "demo",
 "responseHost":       "my.server.org",
 "responsePort":       5555}

Remove all objects under the specified workspace(s) from GeoServer and delete the workspace(s)

The geoserverWorkspace parameter can be plain text or a regex pattern. If a plain text string is provided, an exact match will be performed to identify the workspace to be removed. If a regex pattern is provided, all workspaces matching the regex will be deleted.

{"action":             "remove",
 "geoserverWorkspace": "demo",
 "responseHost":       "my.server.org",
 "responsePort":       5555}

Because GeoServer updates can take awhile to complete, these requests are processed asynchronously. This means that the network connection will be closed as soon as the incoming request is read from the client. In order to get a response message back from the GeoSync server about whether the requested action succeeded or failed, each request includes responseHost and responsePort fields. After the request has been processed, GeoSync will send a newline-terminated JSON response there containing all of the original request fields merged with these additional fields:

{status:       0,
 message:      "GeoSync: Workspace updated.",
 responseHost: "geosync.mydomain.org",
 responsePort: 31337}

Note that responseHost and responsePort from the request is overriden by the new responseHost and responsePort values that correspond to the GeoSync server’s host and port.

In this response, the status code is 0 on success, 1 on error, or 2 for a progress message. The message string will explain the success, error, or progress state of the associated request.

GeoSync includes all request fields in the response object to enable the client to match each asynchronous response message they receive with the correct request that they had sent out earlier. To make this easier, it is strongly recommended that each request include a uniquely identifying field, such as an id or request label.

Each valid request will be processed sequentially by the GeoSync server because GeoServer only processes REST requests in a single thread. When multiple incoming GeoSync requests are received while one is currently being processed, the new requests will be added to a job queue and processed in the order in which they are received. A status 2 (progress message) response will be sent to the client notifying them of their place in the queue. Once processed, a final status 0 (success) or 1 (error) response will be sent to the client indicating the outcome of their request.

Logging

You will probably want to redirect the output from these commands to a text file for reviewing after the fact as they report each REST call sent out from GeoSync and the corresponding response received from the GeoServer.

clojure -M:run ... &> out.txt

Alternatively, if you pass in a path to a log directory with either -o on the command line or via :log-dir in your config.edn file, all logs will be redirected to the provided directory. This directory will hold up to ten different log files where each log file corresponds to a date where GeoSync logs were output. The format of these log files is: YYYY-MM-DD.log.

Styles

:styles

There are a few ways to interface with GeoServer styles using GeoSync. The most simple way is by using a :styles key-value pair in your .edn configuration file. To use this key-value pair, the value associated with the :styles key should be a vector of maps. Each map in the vector should contain a :layer-pattern key with an associated string or regular expression. This string or regular expression should correspond with the types of layers that you want to apply specific vector or raster styles to. It should also either contain a :vector-style key, a :raster-style key, or both (for scenarios where similarly named layers could either be a vector or raster layer). The values associated with both the :vector-style and :raster-style keys can either be a string or a vector of unique strings. Each string should correspond with the exact name of a style that already exists on the GeoServer. Whenever GeoSync matches a layer with the specific :layer-pattern you provided, it will automatically add any :vector-style or :raster-style you specified with these keys (depending on the type of layer – which will be determined automatically by GeoSync). Please see resources/sample-config.edn for a concrete example of how to use the :styles key.

:style-dir

If you don’t wish to manually add any of the styles laid out in your :styles key to your GeoServer, it’s possible to sync styles to GeoServer using the :style-dir key in your .edn file. When you configure a :style-dir (also available with the -s or --style-dir option in the CLI), GeoSync will recursively look up all .css files that are present in this directory and add them to GeoServer for you. Note that this step happens before any layers are registered, so you can leverage the use of the :styles key without having to add any styles manually to your GeoServer.

If you don’t provide a :style-dir, you’ll have to make sure that any styles you use inside of the :styles key have already been manually added to the GeoServer. Note that any styles added automatically using this key will be tied to the same workspace that you provide via :geoserver-workspace in your .edn file (or with -w / --geoserver-workspace) from the command line. If you remove that workspace, all styles associated with that workspace will also be removed.

If you wish to sync styles to a unique “styles” workspace that isn’t tied to any other layers so that they aren’t removed when you need to remove layers, you can configure your .edn file to use the :style-dir key and not the :data-dir key. In this case, GeoSync would just load the CSS styles found under :style-dir into the GeoServer under the :geoserver-workspace specified in your .edn config file. If you include both a :style-dir and a :data-dir, then your CSS styles and your GIS layers will both be added under the same :geoserver-workspace.

When using this option, styles will be created on GeoServer with their name prepended with the workspace name, using : as separator. Example, if your workspace is named sierra-nevada and your style habitat-connectivity, the final style name will be sierra-nevada:habitat-connectivity. This is because GeoServer has a long standing bug that will prevent it from assigning the correct style if you have multiple styles with the same name (even across different workspaces).

:overwrite-styles

If you wish to overwrite any already existing GeoServer styles with updates to the styles made inside of the .css files in your :style-dir, you can leverage a parameter called :overwrite-styles (also available with the -O or --overwrite-styles option in the CLI). If set to true, any GeoServer styles will be overwritten if and only if the style already exists in the provided :geoserver-workspace and we have a style file of the same name in :style-dir. If both of these cases are hit, the GeoServer style will be overwritten with the latest style provided in the corresponding .css file. Defaults to false if not specified.

This option is not available when :style-dir is not set.

:autostyle-layers

If you have a large number of styles and layers, you can use the autostyle-layers option of GeoSync. Instead of declaratively matching each layer-pattern with the corresponding style, you can rely on this feature to do the work for you.

To use it, you must set autostyle-layers to true (using the CLI or the configuration file).

GeoSync will fetch all existing styles in the specified geoserver-workspace and will match each newly added GIS layer with any style whose name appears in the end of the layer name. This string matching is case insensitive.

Examples:

Layer	Style	Match?
sierra-nevada-fires	sierra-nevada:fires	Y
SierraNevada_Tier2_AnnualBurnProbability	sierra-nevada:annualburnprobability	Y
SierraNevada_Tier2_AnnualBurnProbability	sierra-nevada:TIER2_ANNUALBURNPROBABILITY	Y
SierraNevada_Tier2_AnnualBurnProbability	sierra-nevada:TIER1_ANNUALBURNPROBABILITY	N

Using Custom Projections

If your GIS data uses a custom projection that is not known to the GeoServer, you will need to add it manually before running GeoSync. Documentation on adding custom projections to GeoServer can be found here:

https://docs.geoserver.org/latest/en/user/configuration/crshandling/customcrs.html

ImageMosaic Support

In addition to registering individual GeoTIFFs and Shapefiles, GeoSync can also register a directory of GeoTIFFs as a single ImageMosaic layer. This can be particularly useful for timeseries rasters in which your data is stored as a directory of single-band GeoTIFFs with one file per timestep.

In order for a directory to be detected as an ImageMosaic data source by GeoSync, it must include these three text files:

datastore.properties
indexer.properties
timeregex.properties

The required contents of these files is provided in the following subsections.

datastore.properties

Estimated\ extends=true
jndiRefevrenceName=java\:comp/env/jdbc/postgres
validate\ connections=true
Connection\ timeout=10
SPI=org.geotools.data.postgis.PostgisNGJNDIDataStoreFactory

Note that datastore.properties also needs a schema property, but this will be inserted by GeoSync by using either the :geoserver-workspace attribute from config.edn (in CLI mode) or the geoserverWorkspace attribute from a JSON request (in server mode).

indexer.properties

TimeAttribute=ingestion
Schema=*the_geom\:Polygon,location\:String,ingestion\:java.util.Date
PropertyCollectors=TimestampFileNameExtractorSPI[timeregex](ingestion)

Note that indexer.properties also needs a Name property, but this will be inserted by GeoSync based on the the directory structure.

timeregex.properties

regex=([0-9]{8}_[0-9]{6}),format=yyyyMMdd_HHmmss

This regex must match the file naming convention used for all the timeseries GeoTIFFs in your ImageMosaic directory. The example given above would match files with the following names:

some_layer_20210801_120000
some_layer_20210801_130000
some_layer_20210801_140000

If you use a different file naming convention for your timestamps, make sure to update timeregex.properties accordingly.

File Watcher

In server-mode, you may optionally specify a file path to be monitored. Any folders added or removed with a valid directory structure (as specified in the folder-name->regex map) will be automatically registered or deregistered from the GeoServer. See the file-watcher entry in resources/sample-config.edn for an example.

Here’s an explanation of the :file-watcher configuration:

dir: A path to a readable directory.
folder-name->regex: A map of folder names to regexes. The regexes are used to parse the appropriate workspace name from the file path. Note that for any folder name not included in this map, no action will be taken (even if the file watcher detects an event).

Depending on the number of files you are attempting to watch, you will likely have to increase your inotify limit. You can find your current inotify limit by running this command:

cat /proc/sys/fs/inotify/max_user_watches

To set a new limit temporarily, run:

sudo sysctl fs.inotify.max_user_watches=$NUMBER_OF_FILES
sudo sysctl -p

To make this limit permanent, run:

echo fs.inotify.max_user_watches=$NUMBER_OF_FILES | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

Layer Rules

You may optionally specify layer rules associated with specific workspaces to add to GeoServer. To do so, you must provide a :layer-rules key value pair to your GeoSync config file. The :layer-rules key expects a vector of maps as its value, where each map contains a GeoServer workspace regex and a vector of maps of layer rules to add whenever that workspace regex is matched upon adding a new workpace. Each :workspace-regex is assumed to be unique from every other :workspace-regex because you can add as many :associated-rules as you like to each regex. For an example of what this looks like, see resources/sample-config.edn.

Note that currently the only layer rules that can be added are those that are associated with a specific workspace. Rules associated with global layer groups are not supported at this time. If you wish to add a layer rule associated with every workspace (by using the wildcard * character), you can do so manually via the Data Security page on the GeoServer web UI. Each :layer-rule inside of the :associated-rules vector of maps must contain “geoserver-workspace” as the beginning portion of the layer rule. This is replaced by the actual GeoServer workspace being added by GeoSync. The :role associated with each :layer-rule can contain multiple roles by separating each role with a comma (e.g. “ROLE_ONE,ROLE_TWO”).

When a workspace is removed, all existing layer rules are compared to the workspace being removed. Each existing layer rule that applies to the workspace being removed is deleted from GeoServer along with the workspace.

GeoFence Rules

You may optionally specify GeoFence rules associated with specific workspaces to add to GeoServer. To do so, you must provide a :geofence-rules key value pair to your GeoSync config file. The :geofence-rules key expects a vector of maps as its value, where each map contains a GeoServer workspace regex and optional vectors of data and admin rules to add whenever that workspace regex is matched upon adding a new workspace.

Each :workspace-regex must be unique from every other :workspace-regex in both the :layer-rules and :geofence-rules sections, as GeoSync uses the first matching regex to determine which rules to apply.

For an example of what this looks like, see resources/sample-config.edn.

When a workspace is removed, all existing GeoFence data and admin rules associated with that workspace are deleted from GeoServer along with the workspace.

GeoFence Data Rules

GeoFence Data Rules control access to layers and workspaces by role, username, service, request, and IP address range. They work in the same way as GeoServer’s default data security rules (see “Important GeoFence Notes” below). Use Data Security Rules when you’re controlling data access.

The :data-rules vector contains maps that specify GeoFence data rules. Each data rule requires :user-name, :role-name, and :access keys, and may optionally include :priority, :address-range, :valid-after, :valid-before, :service, :request, and :layer keys. The :access value must be either “ALLOW” or “DENY”.

GeoFence Admin Rules

GeoFence Admin Rules are used to manage administrative permissions, not data access . These include permissions such as who can edit workspaces, stores, and layers. Use Admin Rules when you want to restrict configuration/edit access to the GeoServer admin interface.

The :admin-rules vector contains maps that specify GeoFence admin rules. Each admin rule requires :user-name, :role-name, and :access keys, and may optionally include :priority and :address-range keys. The :access value must be either “ADMIN” or “USER”.

Important GeoFence Notes

In order to use GeoFence rules you need to first install the GeoFence plugin on your GeoServer.
The default behavior of GeoFence is to deny access to all resources for all users besides the admin user. GeoFence does not affect the default admin functionality.
GeoFence overrides the standard GeoServer data security/layer rules management system. This means you should not use both :layer-rules and :geofence-rules in your config.edn file. If you do, any :layer-rules will be ignored by the GeoServer.
There is an undocumented bug where the Layer Preview functionality stops working for any non-admin users, even if they have explicit “ALLOW” access to specific layers. This does not affect their ability to make requests to these layers, just the ability to see them via the Layer Preview section of the GeoServer web UI.

UberJAR

To build GeoSync as an UberJAR (generated in this repository’s target directory), run the following command:

clojure -X:build-uberjar

If you would like to build this in an isolated container, install Guix for your system and then run this command:

./geosync-shell.sh -- clojure -X:build-uberjar

Once you have created the UberJAR, you can run it with the following command:

java -jar $PATH_TO_JAR_DIR/geosync-$VERSION-standalone.jar $CLI_ARGS

Vulerability checking

To check for vulerabilities in this project, use clojure -M:clj-watson scan -p deps.edn -s

License and Distribution

GeoSync is distributed by Spatial Informatics Group, LLC. under the terms of the Eclipse Public License version 2.0 (EPLv2). See the LICENSE file in this directory for more information.

Name		Name	Last commit message	Last commit date
Latest commit History 388 Commits
.github		.github
logs		logs
resources		resources
src		src
test		test
.gitignore		.gitignore
LICENSE		LICENSE
README.org		README.org
channels.scm		channels.scm
config.default.edn		config.default.edn
deps.edn		deps.edn
geosync-shell.sh		geosync-shell.sh
manifest.scm		manifest.scm

License

pyregence/geosync

Folders and files

Latest commit

History

Repository files navigation

GeoSync

Installation Requirements

PostgreSQL Database Setup

Configure Database Connection Settings in config.edn

Create and Populate the Database

Performance Settings (Optional)

Testing

Quickstart

Operation Modes

CLI Mode

Server Mode

Logging

Styles

:styles

:style-dir

:overwrite-styles

:autostyle-layers

Using Custom Projections

ImageMosaic Support

datastore.properties

indexer.properties

timeregex.properties

File Watcher

Layer Rules

GeoFence Rules

GeoFence Data Rules

GeoFence Admin Rules

Important GeoFence Notes

UberJAR

Vulerability checking

License and Distribution

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 1

Packages 0

Uh oh!

Contributors 8

Uh oh!

Languages

Packages