diff --git a/Ghidra/Configurations/Public_Release/src/global/docs/WhatsNew.md b/Ghidra/Configurations/Public_Release/src/global/docs/WhatsNew.md index d8eae8d5a2..d5cdaaa55d 100644 --- a/Ghidra/Configurations/Public_Release/src/global/docs/WhatsNew.md +++ b/Ghidra/Configurations/Public_Release/src/global/docs/WhatsNew.md @@ -43,11 +43,11 @@ libraries and operating systems (e.g., CentOS 7.x) may also run into compatibili launching native executables such as the Decompiler and GNU Demangler which may necessitate a rebuild of native components. -**NOTE:** Ghidra Server: The Ghidra 12.0 server is compatible with Ghidra 9.2 and later Ghidra +**NOTE:** Ghidra Server: The Ghidra 12.0 server is compatible with Ghidra 11.3.2 and later Ghidra clients, although the presence of any newer link-files within a repository may not be handled properly by client versions prior to 12.0 which lack support for the new storage format. Ghidra 12.0 clients that introduce new link-files into a project will not be able to add such files into version -control if connected to older Ghidra Server versions. +control if connected to older Ghidra Server versions. **NOTE:** Ghidra Server: Due to potential Java version differences, it is recommended that Ghidra Server installations older than 10.2 be upgraded. Those using 10.2 and newer @@ -64,18 +64,19 @@ process that will provide better results than prior Ghidra versions. You might fresh import of any program you will continue to reverse engineer to see if the latest Ghidra provides better results. -## Project Link Files +## Project Data Link Files Support for link-files within a Ghidra Project has been significantly expanded with this release and with it a new file storage type has been introduced which can create some incompatibilities if projects and repositories containing such files are used by older version of Ghidra or the Ghidra Server. -Previously only external folder and file links were supported through the use of a Ghidra URL. With -12.0 the ability to establish internal folder and file links has been introduced. The new storage -format avoids the use of a database and relies only on a light-weight property file. Internal -project links also allow for either absolute or relative links. Due to the fact that Ghidra allows -a folder or file to have the same pathname, some ambiguities can result. It is highly recommended -that the use of conflicting folder and file pathnames be avoided. +Previously, only external folder and file links were supported through the use of a Ghidra URL. With +12.0 the ability to establish internal folder and file links has been introduced. A new storage +format was adopted for link-files which avoids the use of a database and relies only on a +light-weight property file only. Internal project links also allow for either absolute or relative +links. Due to Ghidra allowing a folder and file to have the same pathname, some ambiguities can +result for Ghidra URL usage. It is highly recommended that the use of conflicting folder and file +pathnames be avoided. The use of internally linked folders and files allows batch import processing to more accurately reflect the native file-system and its use of symbolic links which allow for the same content to @@ -89,9 +90,7 @@ Additional Ghidra API methods have been provided or refined on the following cla link-files: `DomainFolder`, `DomainFile`, `LinkFile`, `LinkHandler`, `DomainFileFilter`, `DomainFileIterator`, etc. -...TO BE CONTINUED... - -## Filesystem Mirroring +## Importer Filesystem Mirroring An option has been added to mirror the local filesystem when importing programs and their libraries. Programs and libraries that exist on the local filesystem as symbolic links will have both their corresponding link file and resolved program file mirrored in the project. Filesystem mirroring @@ -113,7 +112,7 @@ We've added an experimental Z3-based symbolic emulator, which runs as an "auxili concrete emulator, effectively constructing what is commonly called a "concolic" emulator. The symbolic emulator creates Z3 expressions and branching constraints, but it only follows the path determined by concrete emulation. This is most easily accessed by installing the "SymbolicSummaryZ3" -extension (**File** -> **Install Extensions**) and then enabling the `Z3SummaryPlugin` in the +extension (**File -> Install Extensions**) and then enabling the `Z3SummaryPlugin` in the Debugger or Emulator tool, which includes a GUI for viewing and sorting through the results. The Z3 emulator requires z3-4.13.0, available from https://github.com/Z3Prover/z3. Other versions may work, but our current test configuration uses 4.13.0. Depending on the release and your platform, the @@ -138,10 +137,11 @@ that merely integrate with emulation should consider the compositional/callback- Extensions that incorporate new domains (e.g. Z3) or novel behaviors (e.g. JIT) should continue using inheritance. -## Other Improvements - + Added the ability to toggle the displaying of function variables (parameters and locals) that are - normally displayed just below the function signature. The variables display can be turned on/off - globally or individually per function. +## Hide Function Variables +Added the ability to toggle the display of function variables (parameters and locals) within +the Code Browser Listing just below the function signature. The Variables display can be turned +on/off globally via the popup menu toggle action (**Function -> Show/Hide All Variables**) or for +individual functions via an adjacent expand/collapse(+/-) icon. ## Additional Bug Fixes and Enhancements Numerous other new features, improvements, and bug fixes are fully listed in the diff --git a/Ghidra/Extensions/BSimElasticPlugin/README.md b/Ghidra/Extensions/BSimElasticPlugin/README.md index b7c224b41e..b7807d5a1e 100644 --- a/Ghidra/Extensions/BSimElasticPlugin/README.md +++ b/Ghidra/Extensions/BSimElasticPlugin/README.md @@ -77,8 +77,8 @@ details. ## Version -The current BSim plug-in was tested with Elasticsearch version `8.8.1`. A change to the -Elasticsearch scripting interface, starting with version `7.15`, makes the BSim plug-in incompatible +The current BSim plug-in was tested with Elasticsearch version `8.19.7`. A change to the +Elasticsearch scripting interface, starting with version `8.16`, makes the BSim plug-in incompatible with previous versions, but the lsh plug-in jars may work without change across later Elasticsearch versions. @@ -93,7 +93,7 @@ elasticsearch/plugin-descriptor.properties The file format is fairly simple: edit the line ``` -elasticsearch.version=8.8.1 +elasticsearch.version=8.19.7 ``` The plugin may work with other nearby versions, but proceed at your own risk. diff --git a/Ghidra/Extensions/BSimElasticPlugin/build.gradle b/Ghidra/Extensions/BSimElasticPlugin/build.gradle index b0ee3f42a9..fdf2c379b1 100755 --- a/Ghidra/Extensions/BSimElasticPlugin/build.gradle +++ b/Ghidra/Extensions/BSimElasticPlugin/build.gradle @@ -92,8 +92,8 @@ task elasticPluginZip(type: Zip) { destinationDirectory = file("build/data") } -// Currently targeting elasticsearch-8.8.1 which by default runs with java 20 -compileElasticPluginJava.options.release = 20 +// Currently targeting elasticsearch-8.19.7 which by default runs with java 25 +compileElasticPluginJava.options.release = 21 compileElasticPluginJava.dependsOn copyGenericTask compileElasticPluginJava.dependsOn copyUtilityTask diff --git a/Ghidra/Extensions/BSimElasticPlugin/contribZipExclude/plugin-descriptor.properties b/Ghidra/Extensions/BSimElasticPlugin/contribZipExclude/plugin-descriptor.properties index bb6c329c67..4445fdfd65 100755 --- a/Ghidra/Extensions/BSimElasticPlugin/contribZipExclude/plugin-descriptor.properties +++ b/Ghidra/Extensions/BSimElasticPlugin/contribZipExclude/plugin-descriptor.properties @@ -2,5 +2,5 @@ description=Feature Vector Plugin version=1.0 name=lsh classname=org.elasticsearch.plugin.analysis.lsh.AnalysisLSHPlugin -java.version=1.11 -elasticsearch.version=8.8.1 +java.version=1.21 +elasticsearch.version=8.19.7 diff --git a/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/VectorCompareScriptFactory.java b/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/VectorCompareScriptFactory.java index e47b274018..7e063061de 100755 --- a/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/VectorCompareScriptFactory.java +++ b/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/VectorCompareScriptFactory.java @@ -4,9 +4,9 @@ * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at - * + * * http://www.apache.org/licenses/LICENSE-2.0 - * + * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. @@ -73,6 +73,11 @@ public class VectorCompareScriptFactory implements ScoreScript.Factory { return false; } + @Override + public boolean needs_termStats() { + return false; + } + private static int scanForFeatures(byte[] buffer, int offset) throws IOException { int i = 0; while (i < FEATURES_NAME.length()) { diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/ScoreScript.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/ScoreScript.java index 01f5111046..d514b93786 100644 --- a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/ScoreScript.java +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/ScoreScript.java @@ -5,9 +5,9 @@ * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at - * + * * http://www.apache.org/licenses/LICENSE-2.0 - * + * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. @@ -39,6 +39,8 @@ public abstract class ScoreScript { public interface LeafFactory { boolean needs_score(); + boolean needs_termStats(); + ScoreScript newInstance(DocReader reader) throws IOException; } diff --git a/Ghidra/Features/PyGhidra/src/main/py/README.md b/Ghidra/Features/PyGhidra/src/main/py/README.md index 1daf1b703d..b6700043a5 100644 --- a/Ghidra/Features/PyGhidra/src/main/py/README.md +++ b/Ghidra/Features/PyGhidra/src/main/py/README.md @@ -220,9 +220,9 @@ def program_loader() -> "ProgramLoader.Builder": """ ``` -### pyghidra.monitor() +### pyghidra.task_monitor() ```python -def monitor( +def task_monitor( timeout: Optional[int] = None ) -> "PyGhidraTaskMonitor": """ @@ -288,22 +288,22 @@ with pyghidra.open_project(os.environ["GHIDRA_PROJECT_DIR"], "ExampleProject", c for f in fs.files(lambda f: "os/" in f.path and f.name.startswith("decompile")): loader = loader.source(f.getFSRL()).projectFolderPath("/" + f.parentFile.name) with loader.load() as load_results: - load_results.save(pyghidra.monitor()) + load_results.save(pyghidra.task_monitor()) # Analyze the windows decompiler program for a maximum of 10 seconds with pyghidra.program_context(project, "/win_x86_64/decompile.exe") as program: analysis_props = pyghidra.analysis_properties(program) with pyghidra.transaction(program): analysis_props.setBoolean("Non-Returning Functions - Discovered", False) - analysis_log = pyghidra.analyze(program, pyghidra.monitor(10)) - program.save("Analyzed", pyghidra.monitor()) + analysis_log = pyghidra.analyze(program, pyghidra.task_monitor(10)) + program.save("Analyzed", pyghidra.task_monitor()) # Walk the project and set a property in each decompiler program def set_property(domain_file, program): with pyghidra.transaction(program): program_info = pyghidra.program_info(program) program_info.setString("PyGhidra Property", "Set by PyGhidra!") - program.save("Setting property", pyghidra.monitor()) + program.save("Setting property", pyghidra.task_monitor()) pyghidra.walk_programs(project, set_property, program_filter=lambda f, p: p.name.startswith("decompile")) # Load some bytes as a new program @@ -312,7 +312,7 @@ with pyghidra.open_project(os.environ["GHIDRA_PROJECT_DIR"], "ExampleProject", c loader = pyghidra.program_loader().project(project).source(my_bytes).name("my_bytes") loader = loader.loaders("BinaryLoader").language("DATA:LE:64:default") with loader.load() as load_results: - load_results.save(pyghidra.monitor()) + load_results.save(pyghidra.task_monitor()) # Run a GhidraScript pyghidra.ghidra_script(f"{os.environ['GHIDRA_SCRIPTS_DIR']}/HelloWorldScript.java", project) @@ -564,6 +564,8 @@ __3.0.0:__ via the `support/launch.properties` file), it will restore `sys.modules` to its prior state after a PyGhidra script is run so the next time the script is run, it freshly loads all of its imported modules again. This is experimental and should only be enabled if necessary. +* Changed JPype dependency to be fixed at version 1.5.2 to avoid + [a possible Windows crash on 1.6.0](https://github.com/jpype-project/jpype/issues/1316) __2.2.1:__ * PyGhidra now launches with the current working directory removed from `sys.path` to prevent diff --git a/Ghidra/Features/PyGhidra/src/main/py/pyproject.toml b/Ghidra/Features/PyGhidra/src/main/py/pyproject.toml index 034167774a..63fcb9c5a5 100644 --- a/Ghidra/Features/PyGhidra/src/main/py/pyproject.toml +++ b/Ghidra/Features/PyGhidra/src/main/py/pyproject.toml @@ -30,7 +30,7 @@ classifiers = [ "Programming Language :: Python :: 3.13", ] dependencies = [ - "Jpype1>=1.5.2", + "Jpype1==1.5.2", "packaging" ] diff --git a/Ghidra/Features/PyGhidra/src/main/py/src/pyghidra/__init__.py b/Ghidra/Features/PyGhidra/src/main/py/src/pyghidra/__init__.py index a1d38d9c59..aa02f54044 100644 --- a/Ghidra/Features/PyGhidra/src/main/py/src/pyghidra/__init__.py +++ b/Ghidra/Features/PyGhidra/src/main/py/src/pyghidra/__init__.py @@ -58,7 +58,7 @@ from pyghidra.core import run_script, open_program from pyghidra.api import start, started from pyghidra.api import open_project, open_filesystem, consume_program, program_context, analyze from pyghidra.api import ghidra_script, transaction, analysis_properties, program_info -from pyghidra.api import program_loader, monitor, walk_project, walk_programs +from pyghidra.api import program_loader, task_monitor, walk_project, walk_programs from pyghidra.launcher import DeferredPyGhidraLauncher, GuiPyGhidraLauncher, HeadlessPyGhidraLauncher from pyghidra.script import get_current_interpreter from pyghidra.version import ApplicationInfo, ExtensionDetails @@ -69,6 +69,6 @@ __all__ = [ "DeferredPyGhidraLauncher", "ExtensionDetails", "GuiPyGhidraLauncher", "HeadlessPyGhidraLauncher", "start", "started", "open_project", "open_filesystem", "consume_program", "program_conext", "analyze", "ghidra_script", "transaction", - "analysis_properties", "program_info", "program_loader", "monitor", "walk_project", + "analysis_properties", "program_info", "program_loader", "task_monitor", "walk_project", "walk_programs" ] diff --git a/Ghidra/Features/PyGhidra/src/main/py/src/pyghidra/api.py b/Ghidra/Features/PyGhidra/src/main/py/src/pyghidra/api.py index ecb1a87b40..b5b13d608e 100644 --- a/Ghidra/Features/PyGhidra/src/main/py/src/pyghidra/api.py +++ b/Ghidra/Features/PyGhidra/src/main/py/src/pyghidra/api.py @@ -97,7 +97,7 @@ def open_filesystem( service = FileSystemService.getInstance() fsrl = service.getLocalFS().getLocalFSRL(File(path)) - fs = service.openFileSystemContainer(fsrl, monitor()) + fs = service.openFileSystemContainer(fsrl, task_monitor()) if fs is None: raise ValueError(f'"{fsrl}" is not a supported GFileSystem!') return fs @@ -131,7 +131,7 @@ def consume_program( df = project_data.getFile(path) if df is None: raise FileNotFoundError(f'"{path}" does not exist in the Project') - dobj = df.getDomainObject(consumer, True, False, monitor()) + dobj = df.getDomainObject(consumer, True, False, task_monitor()) program_cls = Program.class_ if not program_cls.isAssignableFrom(dobj.getClass()): dobj.release(consumer) @@ -174,7 +174,7 @@ def analyze( from ghidra.app.plugin.core.analysis import AutoAnalysisManager if monitor is None: - monitor = monitor() + monitor = task_monitor() with transaction(program, "Analyze"): GhidraScriptUtil.acquireBundleHostReference() @@ -236,7 +236,7 @@ def ghidra_script( controls = ScriptControls( PrintWriter(stdout_string_writer, True), PrintWriter(stderr_string_writer, True), - monitor() + task_monitor() ) script.setScriptArgs(script_args) script.execute(state, controls) @@ -300,7 +300,7 @@ def program_loader() -> "ProgramLoader.Builder": from ghidra.app.util.importer import ProgramLoader return ProgramLoader.builder() -def monitor( +def task_monitor( timeout: Optional[int] = None ) -> "PyGhidraTaskMonitor": """