mirror/ghidra
1
0
Fork 0
mirror of https://github.com/NationalSecurityAgency/ghidra.git synced 2026-09-28 17:11:11 -09:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/GT-2897_adamopolous_automated_install_script'

Browse Source
This commit is contained in:
Ryan Kurtz
2019-06-27 11:57:23 -04:00
parent c8ae78ee59 037b5df87b
commit 79973565fd
7 changed files with 579 additions and 177 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@@ -3,6 +3,7 @@ ghidra.repos.config
# Misc files produced while executing application # Misc files produced while executing application
repositories/ repositories/
flatRepo/
Ghidra/.ghidraSvrKeys Ghidra/.ghidraSvrKeys
wrapper.log* wrapper.log*

329
DevGuide.md
View File

@@ -6,13 +6,16 @@ The following is a list of dependencies, in no particular order.
This guide includes instructions for obtaining many of these at the relevant step(s). This guide includes instructions for obtaining many of these at the relevant step(s).
You may not need all of these, depending on which portions you are building or developing. You may not need all of these, depending on which portions you are building or developing.
* JDK 11 - We test and build using OpenJDK 11.0.2. * Java JDK 11 - Free long term support (LTS) versions of JDK 11 are provided by:
- https://jdk.java.net/11/ - AdoptOpenJDK
- https://adoptopenjdk.net/releases.html?variant=openjdk11&jvmVariant=hotspot
- Amazon Corretto
- https://docs.aws.amazon.com/corretto/latest/corretto-11-ug/downloads-list.html
* Eclipse - It must support JDK 11. Eclipse 2018-12 or later should work. Other IDEs may work, but we have not tested them. * Eclipse - It must support JDK 11. Eclipse 2018-12 or later should work. Other IDEs may work, but we have not tested them.
- https://www.eclipse.org/downloads/ - https://www.eclipse.org/downloads/
* Gradle 5.0 - Later versions may work, but you'll need to modify our version check. * Gradle 5.0 - Later versions may work, but you'll need to modify our version check.
- https://gradle.org/next-steps/?version=5.0&format=bin - https://gradle.org/next-steps/?version=5.0&format=bin
* A C/C++ compiler - We use GCC on Linux, Xcode (Clang) on macOS, and Visual Studio 2017 on Windows, . * A C/C++ compiler - We use GCC on Linux, Xcode (Clang) on macOS, and Visual Studio 2017 on Windows.
- https://gcc.gnu.org/ - https://gcc.gnu.org/
- https://developer.apple.com/xcode/ - https://developer.apple.com/xcode/
- https://visualstudio.microsoft.com/downloads/ - https://visualstudio.microsoft.com/downloads/
@@ -56,7 +59,7 @@ In Eclipse, select Window -> Prefereces (Eclipse -> Preferences on macOS), then
Install Gradle, add it to your `PATH`, and ensure it is launched using JDK 11. Install Gradle, add it to your `PATH`, and ensure it is launched using JDK 11.
## Setup Source and Dependency Repositories ## Setup Source Repository
You may choose any directory for your working copy, but these instructions will assume you have cloned the source to `~/git/ghidra`. You may choose any directory for your working copy, but these instructions will assume you have cloned the source to `~/git/ghidra`.
Be sure to adjust the commands to match your chosen working directory if different than suggested: Be sure to adjust the commands to match your chosen working directory if different than suggested:
@@ -67,58 +70,80 @@ cd ~/git
git clone git@github.com:NationalSecurityAgency/ghidra.git git clone git@github.com:NationalSecurityAgency/ghidra.git
``` ```
## Setup Build Dependency Repository
Ghidra's build uses artifacts named as available in Maven Central and Bintray JCenter, when possible. Ghidra's build uses artifacts named as available in Maven Central and Bintray JCenter, when possible.
Unfortunately, in some cases, the artifact or the particular version we desire is not available. Unfortunately, in some cases, the artifact or the particular version we desire is not available.
So, in addition to mavenCentral and jcenter, you must configure a flatDir-style repository for manually-downloaded dependencies. So, in addition to mavenCentral and jcenter, you must configure a flat directory-style repository for
manually-downloaded dependencies.
Create `~/.gradle/init.d/repos.gradle` with the following contents: The flat directory-style repository can be created and populated automatically by a provided script,
or manually by downloading the required dependencies. Choose one of the two following methods:
* [Automatic script instructions](#automatic-script-instructions)
* [Manual download instructions](#manual-download-instructions)
```groovy ### Automatic Script Instructions
ext.HOME = System.getProperty('user.home') The flat directory-style repository can be setup automatically by running a simple Gradle script.
Navigate to `~/git/ghidra` and run the following:
allprojects {
repositories {
mavenCentral()
jcenter()
flatDir name:'flat', dirs:["$HOME/flatRepo"]
}
}
``` ```
gradle --init-script gradle/fetchDependencies.gradle init
```
The Gradle task to be executed, in this case _init_, is unimportant. The point is to have Gradle execute
the `fetchDependencies.gradle` script. If it ran correctly you will have a new `~/git/ghidra/flatRepo/`
directory populated with the following jar files:
* AXMLPrinter2
* csframework
* dex-ir-2.0
* dex-reader-2.0
* dex-reader-api-2.0
* dex-tools-2.0
* dex-translator-2.0
* dex-writer-2.0
* hfsx
* hfsx_dmglib
* iharder-base64
Create the `~/flatRepo` folder to hold the manually-downloaded dependencies: There will also be a new archive files at:
* ~/git/ghidra/Ghidra/Features/GhidraServer/build/`yajsw-stable-12.12.zip`
* ~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/`PyDev 6.3.1.zip`
* ~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/`cdt-8.6.0.zip`
If you see these, congrats! Skip to [building](#building-ghidra) or [developing](#developing-ghidra). If not, continue with manual download
instructions below...
### Manual Download Instructions
Create the `~/git/ghidra/flatRepo/` directory to hold the manually-downloaded dependencies:
```bash ```bash
mkdir ~/flatRepo mkdir ~/git/ghidra/flatRepo
``` ```
If you prefer not to modify your user-wide Gradle configuration, you may use #### Get Dependencies for FileFormats:
Gradle's other init script facilities, but you're on your own.
## Get Dependencies for FileFormats:
Download `dex-tools-2.0.zip` from the dex2jar project's releases page on GitHub. Download `dex-tools-2.0.zip` from the dex2jar project's releases page on GitHub.
Unpack the `dex-*.jar` files from the `lib` directory to `~/flatRepo`: Unpack the `dex-*.jar` files from the `lib` directory to `~/git/ghidra/flatRepo`:
```bash ```bash
cd ~/Downloads # Or wherever cd ~/Downloads # Or wherever
curl -OL https://github.com/pxb1988/dex2jar/releases/download/2.0/dex-tools-2.0.zip curl -OL https://github.com/pxb1988/dex2jar/releases/download/2.0/dex-tools-2.0.zip
unzip dex-tools-2.0.zip unzip dex-tools-2.0.zip
cp dex2jar-2.0/lib/dex-*.jar ~/flatRepo/ cp dex2jar-2.0/lib/dex-*.jar ~/git/ghidra/flatRepo/
``` ```
Download `AXMLPrinter2.jar` from the "android4me" archive on code.google.com. Download `AXMLPrinter2.jar` from the "android4me" archive on code.google.com.
Place it in `~/flatRepo`: Place it in `~/git/ghidra/flatRepo`:
```bash ```bash
cd ~/flatRepo cd ~/git/ghidra/flatRepo
curl -OL https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/android4me/AXMLPrinter2.jar curl -OL https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/android4me/AXMLPrinter2.jar
``` ```
## Get Dependencies for DMG: #### Get Dependencies for DMG:
Download `hfsexplorer-0_21-bin.zip` from www.catacombae.org. Download `hfsexplorer-0_21-bin.zip` from www.catacombae.org.
Unpack the `lib` directory to `~/flatRepo.`: Unpack the `lib` directory to `~/git/ghidra/flatRepo`:
```bash ```bash
cd ~/Downloads # Or wherever cd ~/Downloads # Or wherever
@@ -127,14 +152,79 @@ mkdir hfsx
cd hfsx cd hfsx
unzip ../hfsexplorer-0_21-bin.zip unzip ../hfsexplorer-0_21-bin.zip
cd lib cd lib
cp csframework.jar hfsx_dmglib.jar hfsx.jar iharder-base64.jar ~/flatRepo/ cp csframework.jar hfsx_dmglib.jar hfsx.jar iharder-base64.jar ~/git/ghidra/flatRepo/
``` ```
## Import Gradle Project #### Get Dependencies for GhidraServer
If you want just to build Ghidra, you may skip ahead to Building Ghidra. Building the GhidraServer requires "Yet another Java service wrapper" (yajsw) version 12.12.
Otherwise, import Ghidra into Eclipse using the integrated BuildShip plugin. Download `yajsw-stable-12.12.zip` from their project on www.sourceforge.net, and place it in:
Select File -> Import, expand Gradle, and select "Existing Gradle Project." `~/git/ghidra/Ghidra/Features/GhidraServer/build`:
```bash
cd ~/Downloads # Or wherever
curl -OL https://sourceforge.net/projects/yajsw/files/yajsw/yajsw-stable-12.12/yajsw-stable-12.12.zip
mkdir -p ~/git/ghidra/Ghidra/Features/GhidraServer/build/
cp ~/Downloads/yajsw-stable-12.12.zip ~/git/ghidra/Ghidra/Features/GhidraServer/build/
```
#### Get Dependencies for GhidraDev
Building the GhidraDev plugin for Eclipse requires the CDT and PyDev plugins for Eclipse.
Download `cdt-8.6.0.zip` from The Eclipse Foundation, and place it in:
`~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/`:
```bash
cd ~/Downloads # Or wherever
curl -OL 'http://www.eclipse.org/downloads/download.php?r=1&protocol=https&file=/tools/cdt/releases/8.6/cdt-8.6.0.zip'
curl -o 'cdt-8.6.0.zip.sha512' -L --retry 3 'http://www.eclipse.org/downloads/sums.php?type=sha512&file=/tools/cdt/releases/8.6/cdt-8.6.0.zip'
shasum -a 512 -c 'cdt-8.6.0.zip.sha512'
mkdir -p ~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/
cp ~/Downloads/cdt-8.6.0.zip ~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/
```
Download `PyDev 6.3.1.zip` from www.pydev.org, and place it in the same directory:
```bash
cd ~/Downloads # Or wherever
curl -L -o 'PyDev 6.3.1.zip' https://sourceforge.net/projects/pydev/files/pydev/PyDev%206.3.1/PyDev%206.3.1.zip
cp ~/Downloads/'PyDev 6.3.1.zip' ~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/
```
## Building Ghidra
Before building, you may want to update the version and release name.
These properties are kept in `~/git/ghidra/Ghidra/application.properties`.
To build the full package, use Gradle:
```bash
gradle buildGhidra
```
The output will be placed in `~/git/ghidra/build/dist/`.
It will be named according to the version, release name, build date, and platform.
To test it, unzip it where you like, and execute `./ghidraRun`.
__NOTE:__ Unless pre-built manually, the Eclipse GhidraDev plugin will not be included
in the build. In addition, some other supporting data will also be missing.
See the sections below for instructions on how to produce these components.
You may also be able to copy some of these already-built components from a previous official distribution.
## Developing Ghidra
### Prepare the Environment
From the project root, execute:
```bash
gradle prepDev
```
The `prepDev` tasks primarily include generating some source, indexing our built-in help, and unpacking some dependencies.
### Import Eclipse Projects
To develop/modify Ghidra, import Ghidra into Eclipse using the integrated BuildShip plugin.
Select __File -> Import__, expand Gradle, and select "Existing Gradle Project."
Select the root of the source repo as the root Gradle project. Select the root of the source repo as the root Gradle project.
Be sure to select Gradle 5.0, or point it at your local installation. Be sure to select Gradle 5.0, or point it at your local installation.
You may see build path errors until the environment is properly prepared, as described below. You may see build path errors until the environment is properly prepared, as described below.
@@ -147,32 +237,11 @@ From the project root:
gradle eclipse gradle eclipse
``` ```
Select File -> Import, expand General, and select "Existing Projects into Workspace." Select __File -> Import__, expand General, and select "Existing Projects into Workspace."
Select the root of the source repo, and select "Search for nested projects." Select the root of the source repo, and select "Search for nested projects."
Select all, and Finish. Select all, and Finish.
You may see build path errors until the environment is properly prepared, as described below. You may see build path errors until the environment is properly prepared, as described below.
## Prepare the Environment
From the project root, execute:
```bash
gradle prepDev -x yajswDevUnpack eclipse
```
The `prepDev` tasks primarily include generating some source, indexing our built-in help, and unpacking some dependencies.
Regarding `yajswDevUnpack`, please see the relevant sections on GhidraServer below.
For now, we exclude the unpack task.
Optionally, to pre-compile all the language modules, you may also execute:
```bash
gradle sleighCompile
```
Refresh the projects in Eclipse.
You should not see any errors at this point, and you can accomplish many development tasks.
However, some features of Ghidra will not be functional until further steps are taken.
### Building the natives ### Building the natives
Some of Ghidra's components are built for the native platform. Some of Ghidra's components are built for the native platform.
@@ -202,136 +271,68 @@ gradle buildNatives_win64
This will build the decompiler, the demangler for GNU toolchains, the sleigh compiler, and (on Windows only) the PDB parser. This will build the decompiler, the demangler for GNU toolchains, the sleigh compiler, and (on Windows only) the PDB parser.
## Run Ghidra from Eclipse ### Pre-compile Language Modules (optional)
Optionally, to pre-compile all the language modules, you may also execute:
```bash
gradle sleighCompile
```
If the language modules are not pre-compiled, Ghidra will compile them at run time on an as-needed basis.
### Import and Build GhidraDev project (optional)
Developing the GhidraDev Eclipse plugin requires the _Eclipse PDE (Plug-in Development Environment)_, which
can be installed via the Eclipse marketplace. It is also included in the _Eclipse IDE for RCP and RAP Developers_.
To generate the GhidraDev Eclipse projects, execute:
```
gradle eclipse -PeclipsePDE
```
Import the newly generated GhidraDev projects into Eclipse.
__Note:__ If you are getting compilation errors related to PyDev and CDT, go into Eclipse's preferences,
and under _Target Platform_, activate _/Eclipse GhidraDevPlugin/GhidraDev.target_.
See `~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build_README.txt`
for instructions on how to build the GhidraDev plugin.
### Run/Debug Ghidra from Eclipse
To run or debug Ghidra from Eclipse, use the provided launch configuration (usually under the "Run" or "Debug" buttons). To run or debug Ghidra from Eclipse, use the provided launch configuration (usually under the "Run" or "Debug" buttons).
If the launcher does not appear, it probably has not been marked as a favorite. If the launcher does not appear, it probably has not been marked as a favorite.
Click the dropdown next to the "Run" button and select "Run Configurations." Click the dropdown next to the "Run" button and select "Run Configurations."
Then expand "Java Application" on the left to find the "Ghidra" launcher. Then expand "Java Application" on the left to find the "Ghidra" launcher.
## Building Supporting Data
# Building Ghidra Some features of Ghidra require the curation of rather extensive databases.
To build the full Ghidra distribution, you must also build the GhidraServer.
## Get Dependencies for GhidraServer
Building the GhidraServer requires "Yet another Java service wrapper" (yajsw) version 12.12.
Download `yajsw-stable-12.12.zip` from their project on www.sourceforge.net, and place it in a directory named:
`ghidra.bin/Ghidra/Features/GhidraServer/`. Note that `ghidra.bin` must be a sibling of `ghidra`:
```bash
cd ~/Downloads # Or wherever
curl -OL https://sourceforge.net/projects/yajsw/files/yajsw/yajsw-stable-12.12/yajsw-stable-12.12.zip
mkdir -p ~/git/ghidra.bin/Ghidra/Features/GhidraServer/
cp ~/Downloads/yajsw-stable-12.12.zip ~/git/ghidra.bin/Ghidra/Features/GhidraServer/
```
Use Gradle to unpack the wrapper for development.
From your clone:
```bash
gradle yajswDevUnpack
```
## Building the Package
Before building, you may want to update the version and release name.
These properties are kept in `Ghidra/application.properties`.
If you want it included, you must also build the GhidraDevPlugin module first.
Some supporting data will also be missing.
See the sections below for instructions to produce these components.
You may also be able to copy some of this data from a previous official distribution.
To build the full package, use Gradle:
```bash
gradle buildGhidra
```
The output will be placed in `build/dist/`.
It will be named according to the version, release name, build date, and platform.
To test it, unzip it where you like, and execute `./ghidraRun`.
# Building Supporting Data
Some features of Ghidra require the curation of rather extensive data bases.
These include the Data Type Archives and Function ID Databases, both of which require collecting header files and libraries for the relevant SDKs and platforms. These include the Data Type Archives and Function ID Databases, both of which require collecting header files and libraries for the relevant SDKs and platforms.
Much of this work is done by hand. Much of this work is done by hand.
The archives included in our official builds can be found in the [ghidra-data] repository. The archives included in our official builds can be found in the __[ghidra-data]__ repository.
## Building Data Type Archives ### Building Data Type Archives
This task is often done manually from the Ghidra GUI, and the archives included in our official build require a fair bit of fine tuning. This task is often done manually from the Ghidra GUI, and the archives included in our official build require a fair bit of fine tuning.
From a CodeBrowser window, select File -> Parse C Source. From a CodeBrowser window, select __File -> Parse C Source__.
From here you can create and configure parsing profiles, which lists headers and pre-processor options. From here you can create and configure parsing profiles, which lists headers and pre-processor options.
Then, click "Parse to File" to create the Data Type Archive. Then, click _Parse to File_ to create the Data Type Archive.
The result can be added to an installation or source tree by copying it to `Ghidra/Features/Base/data/typeinfo`. The result can be added to an installation or source tree by copying it to `~/git/ghidra/Ghidra/Features/Base/data/typeinfo`.
## Building FID Databases ### Building FID Databases
This task is often done manually from the Ghidra GUI, and the archives included in our official build require a fair bit of fine tuning. This task is often done manually from the Ghidra GUI, and the archives included in our official build require a fair bit of fine tuning.
You will first need to import the relevant libraries from which you'd like to produce a FID database. You will first need to import the relevant libraries from which you'd like to produce a FID database.
This is often a set of libraries from an SDK. This is often a set of libraries from an SDK.
We include a variety of Visual Studio platforms in the official build. We include a variety of Visual Studio platforms in the official build.
From a CodeBrowser window, select File -> Configure. From a CodeBrowser window, select __File -> Configure__.
Enable the "Function ID" plugins, and close the dialog. Enable the "Function ID" plugins, and close the dialog.
Now, from the CodeBrowser window, select Tools -> Function ID -> Create new empty FidDb. Now, from the CodeBrowser window, select __Tools -> Function ID -> Create new empty FidDb__.
Choose a destination file. Choose a destination file.
Now, select Tools -> Function ID -> Populate FidDb from programs. Now, select __Tools -> Function ID -> Populate FidDb__ from programs.
Fill out the options appropriately and click OK. Fill out the options appropriately and click OK.
If you'd like some details of our fine tuning, take a look at `Ghidra/Features/FunctionID/building_fid.txt`. If you'd like some details of our fine tuning, take a look at `~/git/ghidra/Ghidra/Features/FunctionID/data/building_fid.txt`.
# Developing / Building the GhidraDev Plugin
First, install the Eclipse Plugin Development Environment (PDE).
By default, the GhidraDev project is excluded from the build.
To enable it, uncomment it in `settings.gradle`.
```bash
${EDITOR:-vi} settings.gradle
```
You will need some additional runtime dependencies:
## Get Dependencies for GhidraDev
Building the GhidraDev plugin for Eclipse requires the CDT and PyDev plugins for Eclipse.
Download `cdt-8.6.0.zip` from The Eclipse Foundation, and place it in a directory named:
`ghidra.bin/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/`. Note that
`ghidra.bin` must be a sibling of `ghidra`.
```bash
cd ~/Downloads # Or wherever
curl -OL 'http://www.eclipse.org/downloads/download.php?r=1&protocol=https&file=/tools/cdt/releases/8.6/cdt-8.6.0.zip'
curl -o 'cdt-8.6.0.zip.sha512' -L --retry 3 'http://www.eclipse.org/downloads/sums.php?type=sha512&file=/tools/cdt/releases/8.6/cdt-8.6.0.zip'
sha512sum -c 'cdt-8.6.0.zip.sha512'
mkdir -p ~/git/ghidra.bin/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/
cp ~/Downloads/cdt-8.6.0.zip ~/git/ghidra.bin/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/
```
Download `PyDev 6.3.1.zip` from www.pydev.org, and place it in the same directory:
```bash
cd ~/Downloads # Or wherever
curl -OL https://sourceforge.net/projects/pydev/files/pydev/PyDev%206.3.1/PyDev%206.3.1.zip
cp ~/Downloads/'PyDev 6.3.1.zip' ~/git/ghidra.bin/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/
```
Use Gradle to unpack the dependencies.
Note that these tasks will not work until you enable the GhidraDev project in `settings.gradle`.
From your clone:
```bash
gradle cdtUnpack pyDevUnpack
```
## Import the GhidraDev Project
If you're using BuildShip, simply refresh the Gradle project in Eclipse.
If you're not using BuildShip, re-run `gradle eclipse` and import the new project.
[ghidra-data]: https://github.com/NationalSecurityAgency/ghidra-data

9
Ghidra/Features/GhidraServer/build.gradle
View File

@@ -25,7 +25,14 @@ addExports([
]) ])
CopySpec yajswCopySpec = copySpec { CopySpec yajswCopySpec = copySpec {
from(zipTree("${BIN_REPO}/Ghidra/Features/GhidraServer/${yajswRelease}.zip")) { File localFile = file("build/${yajswRelease}.zip")
File binFile = file("${BIN_REPO}/Ghidra/Features/GhidraServer/${yajswRelease}.zip")
// First check if the file was downloaded and dropped in locally. If not, check in the bin
// repo.
def yajswZipTree = localFile.exists() ? zipTree(localFile) : zipTree(binFile)
from(yajswZipTree) {
include "${yajswRelease}/lib/core/**" include "${yajswRelease}/lib/core/**"
include "${yajswRelease}/lib/extended/**" include "${yajswRelease}/lib/extended/**"
include "${yajswRelease}/templates/**" include "${yajswRelease}/templates/**"

34
GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build.gradle
View File

@@ -42,22 +42,14 @@ dependencies {
compileJava.enabled = false compileJava.enabled = false
jar.enabled = false jar.enabled = false
File libraryJarDestDir = file("build/data")
File pyDevSourceZipFile = file("${BIN_REPO}/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/PyDev 6.3.1.zip")
File cdtSourceZipFile = file("${BIN_REPO}/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/cdt-8.6.0.zip")
File pyDevDestDir = file("build/data/buildDependencies/pydev")
File cdtDestDir = file("build/data/buildDependencies/cdt")
task utilityJar(type:Copy) { task utilityJar(type:Copy) {
destinationDir libraryJarDestDir destinationDir file("build/data")
from { project(':Utility').jar } // using closure to delay until all projects evaluated from { project(':Utility').jar } // using closure to delay until all projects evaluated
} }
task launchSupportJar(type:Copy) { task launchSupportJar(type:Copy) {
destinationDir libraryJarDestDir destinationDir file("build/data")
from { project(':LaunchSupport').jar } // using closure to delay until all projects evaluated from { project(':LaunchSupport').jar } // using closure to delay until all projects evaluated
} }
@@ -65,13 +57,22 @@ task launchSupportJar(type:Copy) {
task pyDevUnpack(type:Copy) { task pyDevUnpack(type:Copy) {
description "Unpack PyDev plugin archive for development use" description "Unpack PyDev plugin archive for development use"
group "Development Preparation" group "Development Preparation"
File pyDevDestDir = file("build/data/buildDependencies/pydev")
// Without this, the copyTask will unzip the file to check for "up to date" // Without this, the copyTask will unzip the file to check for "up to date"
onlyIf { onlyIf {
!pyDevDestDir.exists() !pyDevDestDir.exists()
} }
File localFile = file("build/PyDev 6.3.1.zip")
File binFile = file("${BIN_REPO}/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/PyDev 6.3.1.zip")
from zipTree(pyDevSourceZipFile) // First check if the file was downloaded and dropped in locally. If not, check in the bin
// repo.
def pyDevZipTree = localFile.exists() ? zipTree(localFile) : zipTree(binFile)
from pyDevZipTree
exclude "**/.project", "**/.pydevproject" exclude "**/.project", "**/.pydevproject"
destinationDir pyDevDestDir destinationDir pyDevDestDir
@@ -80,13 +81,22 @@ task pyDevUnpack(type:Copy) {
task cdtUnpack(type:Copy) { task cdtUnpack(type:Copy) {
description "Unpack CDT plugin archive for development use" description "Unpack CDT plugin archive for development use"
group "Development Preparation" group "Development Preparation"
File cdtDestDir = file("build/data/buildDependencies/cdt")
// Without this, the copyTask will unzip the file to check for "up to date" // Without this, the copyTask will unzip the file to check for "up to date"
onlyIf { onlyIf {
!cdtDestDir.exists() !cdtDestDir.exists()
} }
File localFile = file("build/cdt-8.6.0.zip")
File binFile = file("${BIN_REPO}/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/cdt-8.6.0.zip")
from zipTree(cdtSourceZipFile) // First check if the file was downloaded and dropped in locally. If not, check in the bin
// repo.
def cdtZipTree = localFile.exists() ? zipTree(localFile) : zipTree(binFile)
from cdtZipTree
destinationDir cdtDestDir destinationDir cdtDestDir
} }

13
build.gradle
View File

@@ -39,6 +39,19 @@ allprojects {
} }
} }
/*********************************************************************************
* Use flat directory-style repository if flatRepo directory is present.
*********************************************************************************/
if (file("flatRepo").isDirectory()) {
allprojects {
repositories {
mavenCentral()
jcenter()
flatDir name: "flat", dirs:["$rootProject.projectDir/flatRepo"]
}
}
}
/********************************************************************************* /*********************************************************************************
* load properties from Ghidra/application.properties file * load properties from Ghidra/application.properties file
*********************************************************************************/ *********************************************************************************/

1
gradle/certification.manifest
View File

@@ -4,6 +4,7 @@ distributableGPLModule.gradle||GHIDRA||||END|
distributableGhidraExtension.gradle||GHIDRA||||END| distributableGhidraExtension.gradle||GHIDRA||||END|
distributableGhidraModule.gradle||GHIDRA||||END| distributableGhidraModule.gradle||GHIDRA||||END|
externalGhidraExtension.gradle||GHIDRA||||END| externalGhidraExtension.gradle||GHIDRA||||END|
fetchDependencies.gradle||GHIDRA||||END|
helpProject.gradle||GHIDRA||||END| helpProject.gradle||GHIDRA||||END|
jacocoProject.gradle||GHIDRA||||END| jacocoProject.gradle||GHIDRA||||END|
javaProject.gradle||GHIDRA||||END| javaProject.gradle||GHIDRA||||END|

369
gradle/fetchDependencies.gradle Normal file
View File

@@ -0,0 +1,369 @@
/*******************************************************************************
* fetchDependencies.gradle *
* *
* Fetches/downloads required dependencies that aren't available in the *
* standard online repositories (eg: maven) and configures a flat *
* directory-style respository that points to them. This should be run *
* immediately after cloning the Ghidra repository before any other gradle *
* tasks are run. *
* *
* Specifically, this task: *
* *
* 1. Downloads various dependencies required by the ghidra build and *
* puts them in <ghidra repo>/build/downloads/. From here they are *
* unzipped and/or copied to their final locations. The files to be *
* downloaded: *
* - dex-tools-2.0.zip *
* - AXMLPrinter2.jar *
* - hfsexplorer-0_21-bin.zip *
* - yajsw-stable-12.12.zip *
* - cdt-8.6.0.zip *
* - PyDev 6.3.1.zip *
* *
* 2. Creates a directory at <ghidra repo>/flatRepo which is used as a *
* flat directory-style respository for the files extracted above. *
* *
* usage: from the command line in the main ghidra repository *
* directory, run the following: *
* *
* gradle --init-script gradle/fetchDependencies.gradle init *
* *
* Note: When running the script, files will only be downloaded if *
* necessary (eg: they are not already in the build/downloads/ *
* directory). *
* *
*******************************************************************************/
import java.util.zip.*;
import java.nio.file.*;
import java.security.MessageDigest;
import org.apache.commons.io.*;
import org.apache.commons.io.filefilter.*;
ext.HOME_DIR = System.getProperty('user.home')
ext.REPO_DIR = ((Script)this).buildscript.getSourceFile().getParentFile().getParentFile()
ext.FLAT_REPO_DIR = new File(REPO_DIR, "flatRepo")
ext.DOWNLOADS_DIR = new File(REPO_DIR, "build/downloads")
// Stores the size of the file being downloaded (for formatting print statements)
ext.FILE_SIZE = 0;
// The URLs for each of the dependencies
ext.DEX_ZIP = 'https://github.com/pxb1988/dex2jar/releases/download/2.0/dex-tools-2.0.zip'
ext.AXML_ZIP = 'https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/android4me/AXMLPrinter2.jar'
ext.HFS_ZIP = 'https://sourceforge.net/projects/catacombae/files/HFSExplorer/0.21/hfsexplorer-0_21-bin.zip'
ext.YAJSW_ZIP = 'https://sourceforge.net/projects/yajsw/files/yajsw/yajsw-stable-12.12/yajsw-stable-12.12.zip'
ext.PYDEV_ZIP = 'https://sourceforge.net/projects/pydev/files/pydev/PyDev%206.3.1/PyDev%206.3.1.zip'
ext.CDT_ZIP = 'http://www.eclipse.org/downloads/download.php?r=1&protocol=https&file=/tools/cdt/releases/8.6/cdt-8.6.0.zip'
// The MD5s for each of the dependencies
ext.DEX_MD5 = '032456b9db9e6059376611553aecf31f'
ext.AXML_MD5 = '55d70be9862c2b456cc91a933c197934'
ext.HFS_MD5 = 'cc1713d634d2cd1fd7f21e18ae4d5d5c'
ext.YAJSW_MD5 = 'e490ea92554f0238d74d4ef6161cb2c7'
ext.PYDEV_MD5 = '06263bdef4917c49d8d977d12c2d5073'
ext.CDT_MD5 = '8e9438a6e3947d614af98e1b58e945a2'
// Number of times to try and establish a connection when downloading files before
// failing
ext.NUM_RETRIES = 2
// Set up a maven repository configuration so we can get access to Apache FileUtils for
// copying/deleting files.
initscript {
repositories {
mavenCentral()
}
dependencies {
classpath 'commons-io:commons-io:2.5'
}
}
// This is where the real flow of the script starts...
try {
createDirs()
populateFlatRepo()
}
finally {
cleanup()
}
/**
* Creates the directories where the dependencies will be downloaded and stored
*/
def createDirs() {
if (!DOWNLOADS_DIR.exists()) {
DOWNLOADS_DIR.mkdirs()
}
if (!FLAT_REPO_DIR.exists()) {
FLAT_REPO_DIR.mkdirs()
}
}
/**
* Downloads a file from a URL. If there is a problem connecting to the given
* URL the attempt will be retried NUM_RETRIES times before failing.
*
* Progress is shown on the command line in the form of the number of bytes
* downloaded and a percentage of the total.
*
* Note: We do not validate that the number of bytes downloaded matches the
* expected total here; any discrepencies will be caught when checking
* the MD5s later on.
*
* @param url the file to download
* @param filename the local file to create for the download
*/
def download(url, filename) {
println("File: " + url)
BufferedInputStream istream = establishConnection(url, NUM_RETRIES);
assert istream != null : " ***CONNECTION FAILURE***\n max attempts exceeded; exiting\n"
FileOutputStream ostream = new FileOutputStream(filename);
def dataBuffer = new byte[1024];
int bytesRead;
int totalRead;
while ((bytesRead = istream.read(dataBuffer, 0, 1024)) != -1) {
ostream.write(dataBuffer, 0, bytesRead);
totalRead += bytesRead
print("\r")
if (FILE_SIZE.equals("unknown")) {
print(" Downloading: " + totalRead + " of " + FILE_SIZE)
}
else {
int pctComplete = (totalRead / FILE_SIZE) * 100
print(" Downloading: " + totalRead + " of " + FILE_SIZE + " (" + pctComplete + "%)")
}
System.out.flush()
}
println("")
istream.close();
ostream.close();
}
/**
* Attemps to establish a connection to the given URL.
*
* @param url the site to connect to
* @param retries the number of times to attempt to reconnect if there is a failure
* @return the InputStream for the URL
*/
def establishConnection(url, retries) {
for (int i=0; i<retries; i++) {
try {
println(" Connect attempt " + (i+1) + " of " + retries)
URLConnection conn = new URL(url).openConnection();
FILE_SIZE = conn.getContentLength();
if (FILE_SIZE == -1) {
// This can happen if there is a problem retrieving the size; we've seen it happen
// in testing.
FILE_SIZE = "unknown"
}
return new BufferedInputStream(new URL(url).openStream());
}
catch (Exception e) {
println(" Connection error! " + e)
}
}
}
/**
* Unzips a file to a directory
*
* @param sourceDir the directory where the zip file resides
* @param targetDir the directory where the unzipped files should be placed
* @param zipFileName the name of the file to unpack
*/
def unzip(sourceDir, targetDir, zipFileName) {
def zip = new ZipFile(new File(sourceDir, zipFileName))
zip.entries().findAll { !it.directory }.each { e ->
(e.name as File).with { f ->
if (f.parentFile != null) {
File destPath = new File(targetDir.path, f.parentFile.path)
destPath.mkdirs()
File targetFile = new File(destPath.path, f.name)
targetFile.withOutputStream { w ->
w << zip.getInputStream(e)
}
}
}
}
}
/**
* Downloads and stores the necessary dependencies in the local flat repository.
*
* If the dependency already exists in the downloads folder (DOWNLOADS_DIR) and has the
* proper checksum, it will NOT be re-downloaded.
*/
def populateFlatRepo() {
// 1. Download all the dependencies and verify their checksums. If the dependency has already
// been download, do NOT download again.
File file = new File(DOWNLOADS_DIR, 'dex-tools-2.0.zip')
if (!DEX_MD5.equals(generateChecksum(file))) {
download (DEX_ZIP, file.path)
validateChecksum(generateChecksum(file), DEX_MD5);
}
file = new File(DOWNLOADS_DIR, 'AXMLPrinter2.jar')
if (!AXML_MD5.equals(generateChecksum(file))) {
download (AXML_ZIP, file.path)
validateChecksum(generateChecksum(file), AXML_MD5);
}
file = new File(DOWNLOADS_DIR, 'hfsexplorer-0_21-bin.zip')
if (!HFS_MD5.equals(generateChecksum(file))) {
download (HFS_ZIP, file.path)
validateChecksum(generateChecksum(file), HFS_MD5);
}
file = new File(DOWNLOADS_DIR, 'yajsw-stable-12.12.zip')
if (!YAJSW_MD5.equals(generateChecksum(file))) {
download (YAJSW_ZIP, file.path)
validateChecksum(generateChecksum(file), YAJSW_MD5);
}
file = new File(DOWNLOADS_DIR, 'PyDev 6.3.1.zip')
if (!PYDEV_MD5.equals(generateChecksum(file))) {
download (PYDEV_ZIP, file.path)
validateChecksum(generateChecksum(file), PYDEV_MD5);
}
file = new File(DOWNLOADS_DIR, 'cdt-8.6.0.zip')
if (!CDT_MD5.equals(generateChecksum(file))) {
download (CDT_ZIP, file.path)
validateChecksum(generateChecksum(file), CDT_MD5);
}
// 2. Unzip the dependencies
unzip(DOWNLOADS_DIR, DOWNLOADS_DIR, "dex-tools-2.0.zip")
unzipHfsx()
// 3. Copy the necessary jars to the flatRepo directory. Yajsw, CDT, and PyDev go directly into
// the source repository.
copyDexTools()
copyAXML()
copyHfsx()
copyYajsw()
copyPyDev()
copyCdt()
}
/**
* Generates the md5 for the given file
*
* @param file the file to generate the checksum for
* @return the generated checksum
*/
def generateChecksum(file) {
if (!file.exists()) {
return null
}
MessageDigest md = MessageDigest.getInstance("MD5");
md.update(Files.readAllBytes(Paths.get(file.path)));
byte[] digest = md.digest();
StringBuilder sb = new StringBuilder();
for (byte b : digest) {
sb.append(String.format("%02x", b));
}
return sb.toString();
}
/**
* Compares two checksums and generates an assert failure if they do not match
*
* @param sourceMd5 the checksum to validate
* @param expectedMd5 the expected checksum
*/
def validateChecksum(sourceMd5, expectedMd5) {
assert(sourceMd5.equals(expectedMd5));
}
/**
* Unzips the hfsx zip file
*/
def unzipHfsx() {
def hfsxdir = getOrCreateTempHfsxDir()
unzip (DOWNLOADS_DIR, hfsxdir, "hfsexplorer-0_21-bin.zip")
}
/**
* Copies the dex-tools jars to the flat repository
*
* Note: This will only copy files beginning with "dex-"
*/
def copyDexTools() {
FileUtils.copyDirectory(new File(DOWNLOADS_DIR, 'dex2jar-2.0/lib/'), FLAT_REPO_DIR, new WildcardFileFilter("dex-*"));
}
/**
* Copies the AXMLPrinter2 jar to the flat repository
*/
def copyAXML() {
FileUtils.copyFile(new File(DOWNLOADS_DIR, 'AXMLPrinter2.jar'), new File(FLAT_REPO_DIR, "AXMLPrinter2.jar"));
}
/**
* Copies the necessary hfsx jars to the flat repository
*/
def copyHfsx() {
FileUtils.copyFile(new File(DOWNLOADS_DIR, "hfsx/lib/csframework.jar"), new File(FLAT_REPO_DIR, "csframework.jar"));
FileUtils.copyFile(new File(DOWNLOADS_DIR, "hfsx/lib/hfsx_dmglib.jar"), new File(FLAT_REPO_DIR, "hfsx_dmglib.jar"));
FileUtils.copyFile(new File(DOWNLOADS_DIR, "hfsx/lib/hfsx.jar"), new File(FLAT_REPO_DIR, "hfsx.jar"));
FileUtils.copyFile(new File(DOWNLOADS_DIR, "hfsx/lib/iharder-base64.jar"), new File(FLAT_REPO_DIR, "iharder-base64.jar"));
}
/**
* Copies the yajswdir zip to its location in the GhidraServer project.
*/
def copyYajsw() {
FileUtils.copyFile(new File(DOWNLOADS_DIR, "yajsw-stable-12.12.zip"), new File(REPO_DIR, "Ghidra/Features/GhidraServer/build/yajsw-stable-12.12.zip"));
}
/**
* Copies the pydev zip to its bin repository location
*/
def copyPyDev() {
FileUtils.copyFile(new File(DOWNLOADS_DIR, 'PyDev 6.3.1.zip'), new File(REPO_DIR, "GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/PyDev 6.3.1.zip"));
}
/**
* Copies the cdt zip to its bin repository location
*/
def copyCdt() {
FileUtils.copyFile(new File(DOWNLOADS_DIR, 'cdt-8.6.0.zip'), new File(REPO_DIR, "GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/cdt-8.6.0.zip"));
}
/**
* Creates a temporary folder to house the hfsx zip contents
*
* @return the newly-created hfsx directory object
*/
def getOrCreateTempHfsxDir() {
def hfsxdir = new File (DOWNLOADS_DIR, "hfsx")
if (!hfsxdir.exists()) {
hfsxdir.mkdir()
}
return hfsxdir;
}
/**
* Performs any cleanup operations that need to be performed after the flat repo has
* been populated.
*/
def cleanup() {
// Uncomment this if we want to delete the downloads folder. For now, leave this and
// depend on a gradle clean to wipe it out.
//
//if (DOWNLOADS_DIR.exists()) {
// FileUtils.deleteDirectory(DOWNLOADS_DIR)
//}
}