The PostgreSQL server software is currently only supported for the Linux and MacOS + "emphasis">Linux and macOS architectures. Elasticsearch server software must be obtained separately. Small local file-based databases are supported on all platforms via an embedded H2 database engine. The BSim client diff --git a/Ghidra/Features/BSim/src/main/help/help/topics/BSim/CommandLineReference.html b/Ghidra/Features/BSim/src/main/help/help/topics/BSim/CommandLineReference.html index c3f3bfa17a..6b66cbe78e 100644 --- a/Ghidra/Features/BSim/src/main/help/help/topics/BSim/CommandLineReference.html +++ b/Ghidra/Features/BSim/src/main/help/help/topics/BSim/CommandLineReference.html @@ -56,11 +56,12 @@
bsim_ctl is a command-line utility for - starting and stopping a BSim server using the PostgreSQL back-end that is prepackaged with - the Ghidra distribution. All commands must be run on the machine hosting the server. - Optional parameters for a given command are indicated by square brackets '[' and ']'. - Options with an '=' character require a user specified value. If the value string requires - space characters, it should be enclosed in double quotes.
+ starting and stopping a BSim server using the PostgreSQL back-end. The utility cannot be + used with either an Elasticsearch server or a local H2 database. + All commands must be run on the machine hosting the server. + Optional parameters for a given command are indicated by square brackets '[' and ']'. + Options with an '=' character require a user specified value. If the value string requires + space characters, it should be enclosed in double quotes.Generates function signatures and metadata for all program files retrieved from a Ghidra Server repository or project as specified by a Ghidra URL. The generated signatures may be retained as XML "sigs_" files within a specified XML storage - directory and/or commited to a specified BSim database specified with the bsim=bsimURL option. If an XML storage directory is not specified, a BSim URL must be specified to which the data will be committed.
@@ -439,7 +440,7 @@Commit previously generated signatures and metadata (see - signaturerepo) to a BSim repository. A URL specifying the BSim + generatesigs) to a BSim repository. A URL specifying the BSim repository and a path to a directory containing the "sigs_" XML files to commit are required.
@@ -459,7 +460,7 @@ and metadata generated (see generatesigs). Only metadata: names, function tags, categories, etc. are changed. Signatures are not affected. The generated updates may be retained as XML "update_" files within a specified XML - storage directory and/or commited to a specified BSim database specified with the + storage directory and/or committed to a specified BSim database specified with the bsim=bsimURL option. If an XML storage directory is not specified, a BSim URL must be specified to which the data will be committed. @@ -615,8 +616,8 @@ not enough to uniquely specify the executable.--printselfsig - If specified, each - function listed will be prefixed by a calculated self-significance score. This value is - expressed as a decimal value.
+ function listed will be prefixed by a calculated self-significance score. This score is + expressed as a floating-point value.--callgraph - If specified, a list
of all library functions called by the identified executable will be listed after
@@ -751,7 +752,7 @@
*.gpr locator file must be specified with the project name. The project name
should exclude any .gpr/.rep suffix. Only the '/' character should be used as a
directory separator. In addition, when running on Windows, the directory path should
- include its drive desigation preceeded by a '/' (e.g., ghidra:/C:/mydir/myproject?/folderA/folderB).
For local file URLs, the absolute path the H2 database *.mv.db file
must be specified without the *.mv.db extension. Only the '/' character should be
used as a directory separator. In addition, when running on Windows, the directory path
- should include its drive desigation preceeded by a '/' (e.g., file:/C:/mydir/mydb).
There are two choices for deploying a shared server for the BSim Database: PostgreSQL or - Elasticsearch. In addition, a local file-based database may be employed which utilizes an - integrated H2 Database engine. This file-based database is intended for smaller datasets - and its use is limited to a single process.
+ Elasticsearch. Both options support multiple users and allow multiple simultaneous connections + to the remote server. A single database can ingest large datasets, while + still maintaining short query times. ++ Alternately, a single user can create a BSim database on their local file-system, + without a server, by utilizing the H2 database engine integrated into Ghidra. This option + is intended for querying against small datasets and does not require installation of + additional server software.
-PostgreSQL software, including the extension necessary for BSim signature indexing, - comes prepackaged with the Ghidra distribution. It runs on a single host and makes - efficient use of whatever CPU, memory, and disk resources are made available to it. - PostgreSQL is a highly robust and capable server that should perform well on minimally - configured workstations up to high-end production hardware.
+A PostgreSQL server, which must be built with a BSim specific extension, + runs on a single host and makes efficient use of whatever CPU, memory, and disk resources + are made available to it. PostgreSQL is a robust and capable server that should perform well + on minimally configured workstations up to high-end production hardware. Source for the + BSim extension to PostgreSQL is included as part of the Ghidra installation, but the + PostgreSQL source may need to be obtained separately by the database administrator. + See “Building the Server” +
-An Elasticsearch BSim plug-in is included with the Ghidra distribution, but the core - server software must be obtained separately by the database administrator. Elasticsearch is - a scalable text search and analytics database. It automatically distributes itself across - machines in a cluster, allowing individual database queries and requests to be serviced in - parallel. Support for BSim in Elasticsearch should still be considered in prototype, but - all major functionality has been implemented, and the BSim schema takes full advantage of - Elasticsearch as a distributed database.
+An Elasticsearch server, which must have a BSim specific plug-in installed, runs + as a scalable database that automatically distributes itself across machines in a cluster, + allowing individual database queries and requests to be serviced in parallel. The Elasticsearch + BSim plug-in is included with the Ghidra installation, but the core server software must be obtained + separately by the database administrator. + See “Installing the Plug-in” +
BSim clients included in the base Ghidra distribution can interface to any of these - databases.
+ databases. Users that just want to connect to an existing shared server via a BSim client do not need to + install any server software themselves.The base Ghidra distribution comes with the PostgreSQL software and the extensions - necessary for supporting a BSim database. The PostgreSQL server is most easily managed - using the bsim_ctl command-line script. When - bsim_ctl start is run for the first time (see - below), the PostgreSQL software is unpacked, depending on the host OS, to either
+$(ROOT)/Ghidra/Features/BSim/os/linux64/postgresql
- OR |
-
https://www.postgresql.org/ftp/source/v15.3
+ |
+
The steps to build the PostgreSQL server with the BSim extension then are:
-$(ROOT)/Ghidra/Features/BSim/os/osx64/postgresql1) If not already present, place the PostgreSQL source distribution file
+ postgresql-15.3.tar.gz in the Ghidra installation at
BSim will not operate with PostgreSQL without the Ghidra specific extensions, but - otherwise the provided installation is standard. It can be configured just like any other - stand-alone PostgreSQL server. PostgreSQL is highly configurable, and there are no direct - restrictions on modifying the configuration values. A default configuration is provided - with this installation that has been tuned specifically for the BSim Database - application, so in practice there may be little reason to modify it. But there are a few - standard configuration values for the server that might need adjusting. These do impact - important aspects of the server, like the amount of memory allocated to the server and - access restrictions.
+$(ROOT)/Ghidra/Features/BSim/postgresql-15.3.tar.gz
+ |
+
2) From the command-line, within the same directory, run the script make-postgres.sh
cd $(ROOT)/Ghidra/Features/BSim
+ |
+
./make-postgres.sh
+ |
+
Additional packages or software may need to be installed on the host OS in order for the + build to complete successfully, OpenSSL in particular is required for BSim. For the + full list of PostgreSQL software dependencies, refer to:
+ +https://www.postgresql.org/docs/current/install-requirements.html
+ |
+
Once the build has completed successfully, + the bsim_ctl command-line script is ready to use + for starting a server (see + “Starting and Stopping the Server”). + The PostgreSQL server software will run out of the Ghidra installation at
+ +$(ROOT)/Ghidra/Features/BSim/build/os/linux64/postgresql
+ OR |
+
$(ROOT)/Ghidra/Features/BSim/build/os/osx64/postgresql |
+
Other than having the extension itself, a BSim enabled PostgreSQL server is completely standard, + and can be configured like any other stand-alone PostgreSQL server. There are no direct restrictions + on modifying the configuration values. A default configuration is provided with this installation that + has been tuned specifically for the BSim Database application, so in practice there may be little reason to + modify it. But there are a few standard configuration values for the server that might need + adjusting. See + “Additional Configuration”.
+This controls which ciphers the server allows when negotiating a connection. - The defaults are reasonable, but administrators may want more control. The - setting 'TLSv1.2', for instance, can be used to be compliant with the latest - TLS standard.
+This controls the minimum SSL/TLS protocol version used when the server negotiates a connection. + The current default is 'TLSv1.2'
A full description of how to configure an Elasticsearch cluster, including how to - start and stop the server, is beyond the scope of this document. In particular, the bsim_ctl command-line, as described in “PostgreSQL Configuration”, does not apply to - Elasticsearch. Complete documentation is available on-line from the Elasticsearch - website.
+A full description of how to configure an Elasticsearch cluster is beyond the scope of + this document. In particular, the bsim_ctl + command-line, as described in “PostgreSQL Configuration”, does not apply to + Elasticsearch. Complete documentation for administering a database is available on-line + from the Elasticsearch website.
+ +The following discussion describes how to set up a toy, or single node, server, using the
+ free and open Elasticsearch distribution. This distribution includes a REST API for administering
+ a database, which can be accessed using curl commands or some other method
+ to send HTTP requests directly to the node.
+
Assuming the add-on has been unpacked, the plug-in can be installed to a single node using the elasticsearch-plugin command in the @@ -521,6 +601,44 @@ up.
The open Elasticsearch distribution starts with up with password authentication
+ enabled by default. When a node is started up for the first time, as described above, an
+ elastic user is created with a randomly
+ generated password that is reported, once, to the console. For a toy deployment, it may
+ be convenient to add additional users via curl commands. The
+ following example creates a user named ghidrauser
+ with a default password "changeme", using the elastic users credentials.
+ The generated password for the elastic user must be substituted for the XXXXXX
+ at the beginning of the command.
+
+curl -k -u elastic:XXXXXX -X POST "https://localhost:9200/_security/user/ghidrauser?pretty" -H 'Content-Type: application/json' -d'
+{
+ "password" : "changeme",
+ "roles" : [ "viewer" ],
+ "full_name" : "Ghidra User",
+ "email" : "ghidrauser@example.com"
+}
+'
+
+ Elasticsearch uses the concept of roles to grant access privileges to particular users. The + built-in role viewer, as in the example above, can be used to grant users + read-only access to a database. The built-in superuser role grants + administrator privileges.
+It is possible to create tailored database configuration templates so that - implementors have a permanent and accessible record of a particular set-up and don't need + implementers have a permanent and accessible record of a particular set-up and don't need to repeatedly issue bsim setmetadata and bsim addexecategory when creating a database. Other aspects of a database can also be manipulated, like weighting schemes and diff --git a/Ghidra/Features/BSim/src/main/help/help/topics/BSim/FeatureWeight.html b/Ghidra/Features/BSim/src/main/help/help/topics/BSim/FeatureWeight.html index 5eb4a220b5..a828cffe4f 100644 --- a/Ghidra/Features/BSim/src/main/help/help/topics/BSim/FeatureWeight.html +++ b/Ghidra/Features/BSim/src/main/help/help/topics/BSim/FeatureWeight.html @@ -36,8 +36,8 @@
The BSim Database uses a standard Feature - Vector approach to compare and index software functions. A The BSim Database uses a feature vector + approach to compare and index software functions. A feature is an abstraction that simply means a single element or attribute that can be compared quantitatively between two objects. The set of possible features used by a particular approach is fixed, and any object being examined is viewed as @@ -250,7 +250,7 @@ to achieve a high confidence for a small function, for single matches viewed in isolation. Of course a medium to low confidence threshold may be enough to produce a unique match if the database is small, and a medium to high confidence threshold may - still produce occasional false positives if the database is very large.
+ still produce occasional false positives even if the database is very large.To generate features and metadata on an existing repository, use the bsim generatesigs command. Signatures may be written as - XML files to a local directory and/or comitted directly to a specified BSim database. If - not immediately comitting to a database and only storing the XML files an appropriate + XML files to a local directory and/or committed directly to a specified BSim database. If + not immediately committing to a database and only storing the XML files an appropriate database config= may be specified in lieu of a BSim database URL (bsimURL) if database specific executable categories and function tags are not utilized. Use of the config= option does not require a running BSim server.
@@ -170,13 +170,16 @@ generated XML files by adding the explicit keyword --overwrite as another parameter. -Both the Ghidra Server and the BSim server must be running in order for this command - to succeed, as the BSim server provides configuration information that may be relevant to - the signature generation process, such as database specific executable categories or - function tags. Assuming this is the same template used to create the database, the BSim - server no longer needs to be running. As in the example above, configuration information - is pulled from the BSim server and signatures are generated from the Ghidra Server - executables.
+In general, both the Ghidra Server and the BSim server must be running in order for + bsim generatesigs command to succeed, + as the BSim server provides configuration information that may be relevant to + the signature generation process, such as database specific executable categories or + function tags. As in the example above, configuration information + is pulled from the BSim server and signatures are generated from the Ghidra Server + executables. If the config= + option is used, assuming the template it specifies is the same one used to create the + database and there are no executable categories or function tags, the BSim server + does not need to be running.
This command takes XML signature files in /xmldirectory
- and writes the metadata in them to a BSim database. All the executable, function, and
- feature vector records are committed to their appropriate tables and all the indexing is
- updated if supported. The URL refers to the BSim database rather than the Ghidra Server.
- The URL cannot be extended with a path. Any executable paths are already encoded within
- the XML file data.
Every executable described within the XML files has a repository and path @@ -495,15 +498,16 @@ public void adjustTags(Address myaddress) throws Exception { -
The BSim server currently has minimal maintenance functionality. Substantial changes - and additions may occur in the near term. It is possible currently to use the SQL command - line tool psql bundled with PostgreSQL in - order to make changes directly to the tables. But for very large modifications to the - database, the best option may be to recreate the database, which is slightly less onerous - than it sounds. The most CPU intensive part of the ingest process, Ghidra's - auto-analysis, typically does not need to be rerun across everything. Regenerating the - metadata files and reimporting takes much less time. Additional efficiency may be gained - by dropping and then regenerating the main index after (re)ingesting. (See below)
+The bsim script provides a minimal number + of maintenance commands for a BSim server, described below. For a PostgreSQL server, it + is possible to use the bundled SQL command line tool + psql in order to make changes directly to + the tables. But for very large modifications to the database, the best option may be to + recreate the database, which is slightly less onerous than it sounds. The most CPU + intensive part of the ingest process, Ghidra's auto-analysis, typically does not need to + be rerun across everything. Regenerating the metadata files and reimporting takes much + less time. Additional efficiency may be gained by dropping and then regenerating the main + index after (re)ingesting. (See below)
NOTE: Applies to PostgrSQL or Elasticsearch databases only
+NOTE: Applies to PostgreSQL or Elasticsearch databases only
For those users performing large ingests or who find themselves rebuilding the database frequently, it is possible to drop the main index, ingest data, then recreate @@ -649,8 +653,8 @@ public void adjustTags(Address myaddress) throws Exception {
The time it takes to rebuild depends directly on the number of functions that have been ingested. For very large collections, rebuilding can take hours or days. The - database can still be accessed while the index is dropped, but query times will likely - be too long for the database to be usable.
+ database can still be accessed while the index is dropped, but queries may take + much longer to complete.NOTE: Applies to PostgrSQL databases only
+NOTE: Applies to PostgreSQL databases only
A maintainer can issue the bsim prewarm command to prepopulate RAM with commonly accessed portions of a @@ -706,7 +710,7 @@ public void adjustTags(Address myaddress) throws Exception { existing BSim database will be incompatible with both the client and server from a new release.
-Unfortunately, the only option to upgrade in these case is to reingest the executables +
Unfortunately, the only option to upgrade in these cases is to reingest the executables into a new BSim database. Frequently the first two stages of ingest (See Ingesting Executables), importing executables to a Ghidra Server and running auto-analysis,