mirror of
https://github.com/NationalSecurityAgency/ghidra.git
synced 2024-11-25 13:42:06 +00:00
Merge remote-tracking branch 'origin/GT-2897_adamopolous_automated_install_script'
This commit is contained in:
Ryan Kurtz
commit
79973565fd
1
.gitignore
vendored
1
.gitignore
vendored
@ -3,6 +3,7 @@ ghidra.repos.config
|
||||
|
||||
# Misc files produced while executing application
|
||||
repositories/
|
||||
flatRepo/
|
||||
Ghidra/.ghidraSvrKeys
|
||||
wrapper.log*
|
||||
|
||||
|
||||
329
DevGuide.md
329
DevGuide.md
@ -6,13 +6,16 @@ The following is a list of dependencies, in no particular order.
|
||||
This guide includes instructions for obtaining many of these at the relevant step(s).
|
||||
You may not need all of these, depending on which portions you are building or developing.
|
||||
|
||||
* JDK 11 - We test and build using OpenJDK 11.0.2.
|
||||
- https://jdk.java.net/11/
|
||||
* Java JDK 11 - Free long term support (LTS) versions of JDK 11 are provided by:
|
||||
- AdoptOpenJDK
|
||||
- https://adoptopenjdk.net/releases.html?variant=openjdk11&jvmVariant=hotspot
|
||||
- Amazon Corretto
|
||||
- https://docs.aws.amazon.com/corretto/latest/corretto-11-ug/downloads-list.html
|
||||
* Eclipse - It must support JDK 11. Eclipse 2018-12 or later should work. Other IDEs may work, but we have not tested them.
|
||||
- https://www.eclipse.org/downloads/
|
||||
* Gradle 5.0 - Later versions may work, but you'll need to modify our version check.
|
||||
- https://gradle.org/next-steps/?version=5.0&format=bin
|
||||
* A C/C++ compiler - We use GCC on Linux, Xcode (Clang) on macOS, and Visual Studio 2017 on Windows, .
|
||||
* A C/C++ compiler - We use GCC on Linux, Xcode (Clang) on macOS, and Visual Studio 2017 on Windows.
|
||||
- https://gcc.gnu.org/
|
||||
- https://developer.apple.com/xcode/
|
||||
- https://visualstudio.microsoft.com/downloads/
|
||||
@ -56,7 +59,7 @@ In Eclipse, select Window -> Prefereces (Eclipse -> Preferences on macOS), then
|
||||
|
||||
Install Gradle, add it to your `PATH`, and ensure it is launched using JDK 11.
|
||||
|
||||
## Setup Source and Dependency Repositories
|
||||
## Setup Source Repository
|
||||
|
||||
You may choose any directory for your working copy, but these instructions will assume you have cloned the source to `~/git/ghidra`.
|
||||
Be sure to adjust the commands to match your chosen working directory if different than suggested:
|
||||
@ -67,58 +70,80 @@ cd ~/git
|
||||
git clone git@github.com:NationalSecurityAgency/ghidra.git
|
||||
```
|
||||
|
||||
## Setup Build Dependency Repository
|
||||
|
||||
Ghidra's build uses artifacts named as available in Maven Central and Bintray JCenter, when possible.
|
||||
Unfortunately, in some cases, the artifact or the particular version we desire is not available.
|
||||
So, in addition to mavenCentral and jcenter, you must configure a flatDir-style repository for manually-downloaded dependencies.
|
||||
So, in addition to mavenCentral and jcenter, you must configure a flat directory-style repository for
|
||||
manually-downloaded dependencies.
|
||||
|
||||
Create `~/.gradle/init.d/repos.gradle` with the following contents:
|
||||
The flat directory-style repository can be created and populated automatically by a provided script,
|
||||
or manually by downloading the required dependencies. Choose one of the two following methods:
|
||||
* [Automatic script instructions](#automatic-script-instructions)
|
||||
* [Manual download instructions](#manual-download-instructions)
|
||||
|
||||
```groovy
|
||||
ext.HOME = System.getProperty('user.home')
|
||||
|
||||
allprojects {
|
||||
repositories {
|
||||
mavenCentral()
|
||||
jcenter()
|
||||
flatDir name:'flat', dirs:["$HOME/flatRepo"]
|
||||
}
|
||||
}
|
||||
### Automatic Script Instructions
|
||||
The flat directory-style repository can be setup automatically by running a simple Gradle script.
|
||||
Navigate to `~/git/ghidra` and run the following:
|
||||
```
|
||||
gradle --init-script gradle/fetchDependencies.gradle init
|
||||
```
|
||||
The Gradle task to be executed, in this case _init_, is unimportant. The point is to have Gradle execute
|
||||
the `fetchDependencies.gradle` script. If it ran correctly you will have a new `~/git/ghidra/flatRepo/`
|
||||
directory populated with the following jar files:
|
||||
* AXMLPrinter2
|
||||
* csframework
|
||||
* dex-ir-2.0
|
||||
* dex-reader-2.0
|
||||
* dex-reader-api-2.0
|
||||
* dex-tools-2.0
|
||||
* dex-translator-2.0
|
||||
* dex-writer-2.0
|
||||
* hfsx
|
||||
* hfsx_dmglib
|
||||
* iharder-base64
|
||||
|
||||
Create the `~/flatRepo` folder to hold the manually-downloaded dependencies:
|
||||
There will also be a new archive files at:
|
||||
* ~/git/ghidra/Ghidra/Features/GhidraServer/build/`yajsw-stable-12.12.zip`
|
||||
* ~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/`PyDev 6.3.1.zip`
|
||||
* ~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/`cdt-8.6.0.zip`
|
||||
|
||||
If you see these, congrats! Skip to [building](#building-ghidra) or [developing](#developing-ghidra). If not, continue with manual download
|
||||
instructions below...
|
||||
|
||||
### Manual Download Instructions
|
||||
|
||||
Create the `~/git/ghidra/flatRepo/` directory to hold the manually-downloaded dependencies:
|
||||
|
||||
```bash
|
||||
mkdir ~/flatRepo
|
||||
mkdir ~/git/ghidra/flatRepo
|
||||
```
|
||||
|
||||
If you prefer not to modify your user-wide Gradle configuration, you may use
|
||||
Gradle's other init script facilities, but you're on your own.
|
||||
|
||||
## Get Dependencies for FileFormats:
|
||||
#### Get Dependencies for FileFormats:
|
||||
|
||||
Download `dex-tools-2.0.zip` from the dex2jar project's releases page on GitHub.
|
||||
Unpack the `dex-*.jar` files from the `lib` directory to `~/flatRepo`:
|
||||
Unpack the `dex-*.jar` files from the `lib` directory to `~/git/ghidra/flatRepo`:
|
||||
|
||||
```bash
|
||||
cd ~/Downloads # Or wherever
|
||||
curl -OL https://github.com/pxb1988/dex2jar/releases/download/2.0/dex-tools-2.0.zip
|
||||
unzip dex-tools-2.0.zip
|
||||
cp dex2jar-2.0/lib/dex-*.jar ~/flatRepo/
|
||||
cp dex2jar-2.0/lib/dex-*.jar ~/git/ghidra/flatRepo/
|
||||
|
||||
```
|
||||
|
||||
Download `AXMLPrinter2.jar` from the "android4me" archive on code.google.com.
|
||||
Place it in `~/flatRepo`:
|
||||
Place it in `~/git/ghidra/flatRepo`:
|
||||
|
||||
```bash
|
||||
cd ~/flatRepo
|
||||
cd ~/git/ghidra/flatRepo
|
||||
curl -OL https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/android4me/AXMLPrinter2.jar
|
||||
```
|
||||
|
||||
## Get Dependencies for DMG:
|
||||
#### Get Dependencies for DMG:
|
||||
|
||||
Download `hfsexplorer-0_21-bin.zip` from www.catacombae.org.
|
||||
Unpack the `lib` directory to `~/flatRepo.`:
|
||||
Unpack the `lib` directory to `~/git/ghidra/flatRepo`:
|
||||
|
||||
```bash
|
||||
cd ~/Downloads # Or wherever
|
||||
@ -127,14 +152,79 @@ mkdir hfsx
|
||||
cd hfsx
|
||||
unzip ../hfsexplorer-0_21-bin.zip
|
||||
cd lib
|
||||
cp csframework.jar hfsx_dmglib.jar hfsx.jar iharder-base64.jar ~/flatRepo/
|
||||
cp csframework.jar hfsx_dmglib.jar hfsx.jar iharder-base64.jar ~/git/ghidra/flatRepo/
|
||||
```
|
||||
|
||||
## Import Gradle Project
|
||||
#### Get Dependencies for GhidraServer
|
||||
|
||||
If you want just to build Ghidra, you may skip ahead to Building Ghidra.
|
||||
Otherwise, import Ghidra into Eclipse using the integrated BuildShip plugin.
|
||||
Select File -> Import, expand Gradle, and select "Existing Gradle Project."
|
||||
Building the GhidraServer requires "Yet another Java service wrapper" (yajsw) version 12.12.
|
||||
Download `yajsw-stable-12.12.zip` from their project on www.sourceforge.net, and place it in:
|
||||
`~/git/ghidra/Ghidra/Features/GhidraServer/build`:
|
||||
|
||||
```bash
|
||||
cd ~/Downloads # Or wherever
|
||||
curl -OL https://sourceforge.net/projects/yajsw/files/yajsw/yajsw-stable-12.12/yajsw-stable-12.12.zip
|
||||
mkdir -p ~/git/ghidra/Ghidra/Features/GhidraServer/build/
|
||||
cp ~/Downloads/yajsw-stable-12.12.zip ~/git/ghidra/Ghidra/Features/GhidraServer/build/
|
||||
```
|
||||
|
||||
#### Get Dependencies for GhidraDev
|
||||
|
||||
Building the GhidraDev plugin for Eclipse requires the CDT and PyDev plugins for Eclipse.
|
||||
Download `cdt-8.6.0.zip` from The Eclipse Foundation, and place it in:
|
||||
`~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/`:
|
||||
|
||||
```bash
|
||||
cd ~/Downloads # Or wherever
|
||||
curl -OL 'http://www.eclipse.org/downloads/download.php?r=1&protocol=https&file=/tools/cdt/releases/8.6/cdt-8.6.0.zip'
|
||||
curl -o 'cdt-8.6.0.zip.sha512' -L --retry 3 'http://www.eclipse.org/downloads/sums.php?type=sha512&file=/tools/cdt/releases/8.6/cdt-8.6.0.zip'
|
||||
shasum -a 512 -c 'cdt-8.6.0.zip.sha512'
|
||||
mkdir -p ~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/
|
||||
cp ~/Downloads/cdt-8.6.0.zip ~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/
|
||||
```
|
||||
|
||||
Download `PyDev 6.3.1.zip` from www.pydev.org, and place it in the same directory:
|
||||
|
||||
```bash
|
||||
cd ~/Downloads # Or wherever
|
||||
curl -L -o 'PyDev 6.3.1.zip' https://sourceforge.net/projects/pydev/files/pydev/PyDev%206.3.1/PyDev%206.3.1.zip
|
||||
cp ~/Downloads/'PyDev 6.3.1.zip' ~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/
|
||||
```
|
||||
|
||||
## Building Ghidra
|
||||
|
||||
Before building, you may want to update the version and release name.
|
||||
These properties are kept in `~/git/ghidra/Ghidra/application.properties`.
|
||||
|
||||
To build the full package, use Gradle:
|
||||
|
||||
```bash
|
||||
gradle buildGhidra
|
||||
```
|
||||
|
||||
The output will be placed in `~/git/ghidra/build/dist/`.
|
||||
It will be named according to the version, release name, build date, and platform.
|
||||
To test it, unzip it where you like, and execute `./ghidraRun`.
|
||||
|
||||
__NOTE:__ Unless pre-built manually, the Eclipse GhidraDev plugin will not be included
|
||||
in the build. In addition, some other supporting data will also be missing.
|
||||
See the sections below for instructions on how to produce these components.
|
||||
You may also be able to copy some of these already-built components from a previous official distribution.
|
||||
|
||||
## Developing Ghidra
|
||||
|
||||
### Prepare the Environment
|
||||
|
||||
From the project root, execute:
|
||||
|
||||
```bash
|
||||
gradle prepDev
|
||||
```
|
||||
The `prepDev` tasks primarily include generating some source, indexing our built-in help, and unpacking some dependencies.
|
||||
|
||||
### Import Eclipse Projects
|
||||
To develop/modify Ghidra, import Ghidra into Eclipse using the integrated BuildShip plugin.
|
||||
Select __File -> Import__, expand Gradle, and select "Existing Gradle Project."
|
||||
Select the root of the source repo as the root Gradle project.
|
||||
Be sure to select Gradle 5.0, or point it at your local installation.
|
||||
You may see build path errors until the environment is properly prepared, as described below.
|
||||
@ -147,32 +237,11 @@ From the project root:
|
||||
gradle eclipse
|
||||
```
|
||||
|
||||
Select File -> Import, expand General, and select "Existing Projects into Workspace."
|
||||
Select __File -> Import__, expand General, and select "Existing Projects into Workspace."
|
||||
Select the root of the source repo, and select "Search for nested projects."
|
||||
Select all, and Finish.
|
||||
You may see build path errors until the environment is properly prepared, as described below.
|
||||
|
||||
## Prepare the Environment
|
||||
|
||||
From the project root, execute:
|
||||
|
||||
```bash
|
||||
gradle prepDev -x yajswDevUnpack eclipse
|
||||
```
|
||||
The `prepDev` tasks primarily include generating some source, indexing our built-in help, and unpacking some dependencies.
|
||||
Regarding `yajswDevUnpack`, please see the relevant sections on GhidraServer below.
|
||||
For now, we exclude the unpack task.
|
||||
|
||||
Optionally, to pre-compile all the language modules, you may also execute:
|
||||
|
||||
```bash
|
||||
gradle sleighCompile
|
||||
```
|
||||
|
||||
Refresh the projects in Eclipse.
|
||||
You should not see any errors at this point, and you can accomplish many development tasks.
|
||||
However, some features of Ghidra will not be functional until further steps are taken.
|
||||
|
||||
### Building the natives
|
||||
|
||||
Some of Ghidra's components are built for the native platform.
|
||||
@ -202,136 +271,68 @@ gradle buildNatives_win64
|
||||
|
||||
This will build the decompiler, the demangler for GNU toolchains, the sleigh compiler, and (on Windows only) the PDB parser.
|
||||
|
||||
## Run Ghidra from Eclipse
|
||||
### Pre-compile Language Modules (optional)
|
||||
|
||||
Optionally, to pre-compile all the language modules, you may also execute:
|
||||
|
||||
```bash
|
||||
gradle sleighCompile
|
||||
```
|
||||
|
||||
If the language modules are not pre-compiled, Ghidra will compile them at run time on an as-needed basis.
|
||||
|
||||
### Import and Build GhidraDev project (optional)
|
||||
|
||||
Developing the GhidraDev Eclipse plugin requires the _Eclipse PDE (Plug-in Development Environment)_, which
|
||||
can be installed via the Eclipse marketplace. It is also included in the _Eclipse IDE for RCP and RAP Developers_.
|
||||
To generate the GhidraDev Eclipse projects, execute:
|
||||
|
||||
```
|
||||
gradle eclipse -PeclipsePDE
|
||||
```
|
||||
|
||||
Import the newly generated GhidraDev projects into Eclipse.
|
||||
|
||||
__Note:__ If you are getting compilation errors related to PyDev and CDT, go into Eclipse's preferences,
|
||||
and under _Target Platform_, activate _/Eclipse GhidraDevPlugin/GhidraDev.target_.
|
||||
|
||||
See `~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build_README.txt`
|
||||
for instructions on how to build the GhidraDev plugin.
|
||||
|
||||
### Run/Debug Ghidra from Eclipse
|
||||
|
||||
To run or debug Ghidra from Eclipse, use the provided launch configuration (usually under the "Run" or "Debug" buttons).
|
||||
If the launcher does not appear, it probably has not been marked as a favorite.
|
||||
Click the dropdown next to the "Run" button and select "Run Configurations."
|
||||
Then expand "Java Application" on the left to find the "Ghidra" launcher.
|
||||
|
||||
## Building Supporting Data
|
||||
|
||||
# Building Ghidra
|
||||
|
||||
To build the full Ghidra distribution, you must also build the GhidraServer.
|
||||
|
||||
## Get Dependencies for GhidraServer
|
||||
|
||||
Building the GhidraServer requires "Yet another Java service wrapper" (yajsw) version 12.12.
|
||||
Download `yajsw-stable-12.12.zip` from their project on www.sourceforge.net, and place it in a directory named:
|
||||
`ghidra.bin/Ghidra/Features/GhidraServer/`. Note that `ghidra.bin` must be a sibling of `ghidra`:
|
||||
|
||||
```bash
|
||||
cd ~/Downloads # Or wherever
|
||||
curl -OL https://sourceforge.net/projects/yajsw/files/yajsw/yajsw-stable-12.12/yajsw-stable-12.12.zip
|
||||
mkdir -p ~/git/ghidra.bin/Ghidra/Features/GhidraServer/
|
||||
cp ~/Downloads/yajsw-stable-12.12.zip ~/git/ghidra.bin/Ghidra/Features/GhidraServer/
|
||||
```
|
||||
|
||||
Use Gradle to unpack the wrapper for development.
|
||||
From your clone:
|
||||
|
||||
```bash
|
||||
gradle yajswDevUnpack
|
||||
```
|
||||
|
||||
## Building the Package
|
||||
|
||||
Before building, you may want to update the version and release name.
|
||||
These properties are kept in `Ghidra/application.properties`.
|
||||
|
||||
If you want it included, you must also build the GhidraDevPlugin module first.
|
||||
Some supporting data will also be missing.
|
||||
See the sections below for instructions to produce these components.
|
||||
You may also be able to copy some of this data from a previous official distribution.
|
||||
|
||||
To build the full package, use Gradle:
|
||||
|
||||
```bash
|
||||
gradle buildGhidra
|
||||
```
|
||||
|
||||
The output will be placed in `build/dist/`.
|
||||
It will be named according to the version, release name, build date, and platform.
|
||||
To test it, unzip it where you like, and execute `./ghidraRun`.
|
||||
|
||||
# Building Supporting Data
|
||||
|
||||
Some features of Ghidra require the curation of rather extensive data bases.
|
||||
Some features of Ghidra require the curation of rather extensive databases.
|
||||
These include the Data Type Archives and Function ID Databases, both of which require collecting header files and libraries for the relevant SDKs and platforms.
|
||||
Much of this work is done by hand.
|
||||
The archives included in our official builds can be found in the [ghidra-data] repository.
|
||||
The archives included in our official builds can be found in the __[ghidra-data]__ repository.
|
||||
|
||||
## Building Data Type Archives
|
||||
### Building Data Type Archives
|
||||
|
||||
This task is often done manually from the Ghidra GUI, and the archives included in our official build require a fair bit of fine tuning.
|
||||
From a CodeBrowser window, select File -> Parse C Source.
|
||||
From a CodeBrowser window, select __File -> Parse C Source__.
|
||||
From here you can create and configure parsing profiles, which lists headers and pre-processor options.
|
||||
Then, click "Parse to File" to create the Data Type Archive.
|
||||
The result can be added to an installation or source tree by copying it to `Ghidra/Features/Base/data/typeinfo`.
|
||||
Then, click _Parse to File_ to create the Data Type Archive.
|
||||
The result can be added to an installation or source tree by copying it to `~/git/ghidra/Ghidra/Features/Base/data/typeinfo`.
|
||||
|
||||
## Building FID Databases
|
||||
### Building FID Databases
|
||||
|
||||
This task is often done manually from the Ghidra GUI, and the archives included in our official build require a fair bit of fine tuning.
|
||||
You will first need to import the relevant libraries from which you'd like to produce a FID database.
|
||||
This is often a set of libraries from an SDK.
|
||||
We include a variety of Visual Studio platforms in the official build.
|
||||
|
||||
From a CodeBrowser window, select File -> Configure.
|
||||
From a CodeBrowser window, select __File -> Configure__.
|
||||
Enable the "Function ID" plugins, and close the dialog.
|
||||
Now, from the CodeBrowser window, select Tools -> Function ID -> Create new empty FidDb.
|
||||
Now, from the CodeBrowser window, select __Tools -> Function ID -> Create new empty FidDb__.
|
||||
Choose a destination file.
|
||||
Now, select Tools -> Function ID -> Populate FidDb from programs.
|
||||
Now, select __Tools -> Function ID -> Populate FidDb__ from programs.
|
||||
Fill out the options appropriately and click OK.
|
||||
|
||||
If you'd like some details of our fine tuning, take a look at `Ghidra/Features/FunctionID/building_fid.txt`.
|
||||
|
||||
# Developing / Building the GhidraDev Plugin
|
||||
|
||||
First, install the Eclipse Plugin Development Environment (PDE).
|
||||
By default, the GhidraDev project is excluded from the build.
|
||||
To enable it, uncomment it in `settings.gradle`.
|
||||
|
||||
```bash
|
||||
${EDITOR:-vi} settings.gradle
|
||||
```
|
||||
|
||||
You will need some additional runtime dependencies:
|
||||
|
||||
## Get Dependencies for GhidraDev
|
||||
|
||||
Building the GhidraDev plugin for Eclipse requires the CDT and PyDev plugins for Eclipse.
|
||||
Download `cdt-8.6.0.zip` from The Eclipse Foundation, and place it in a directory named:
|
||||
`ghidra.bin/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/`. Note that
|
||||
`ghidra.bin` must be a sibling of `ghidra`.
|
||||
|
||||
```bash
|
||||
cd ~/Downloads # Or wherever
|
||||
curl -OL 'http://www.eclipse.org/downloads/download.php?r=1&protocol=https&file=/tools/cdt/releases/8.6/cdt-8.6.0.zip'
|
||||
curl -o 'cdt-8.6.0.zip.sha512' -L --retry 3 'http://www.eclipse.org/downloads/sums.php?type=sha512&file=/tools/cdt/releases/8.6/cdt-8.6.0.zip'
|
||||
sha512sum -c 'cdt-8.6.0.zip.sha512'
|
||||
mkdir -p ~/git/ghidra.bin/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/
|
||||
cp ~/Downloads/cdt-8.6.0.zip ~/git/ghidra.bin/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/
|
||||
```
|
||||
|
||||
Download `PyDev 6.3.1.zip` from www.pydev.org, and place it in the same directory:
|
||||
|
||||
```bash
|
||||
cd ~/Downloads # Or wherever
|
||||
curl -OL https://sourceforge.net/projects/pydev/files/pydev/PyDev%206.3.1/PyDev%206.3.1.zip
|
||||
cp ~/Downloads/'PyDev 6.3.1.zip' ~/git/ghidra.bin/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/
|
||||
```
|
||||
|
||||
Use Gradle to unpack the dependencies.
|
||||
Note that these tasks will not work until you enable the GhidraDev project in `settings.gradle`.
|
||||
From your clone:
|
||||
|
||||
```bash
|
||||
gradle cdtUnpack pyDevUnpack
|
||||
```
|
||||
|
||||
## Import the GhidraDev Project
|
||||
|
||||
If you're using BuildShip, simply refresh the Gradle project in Eclipse.
|
||||
If you're not using BuildShip, re-run `gradle eclipse` and import the new project.
|
||||
|
||||
[ghidra-data]: https://github.com/NationalSecurityAgency/ghidra-data
|
||||
If you'd like some details of our fine tuning, take a look at `~/git/ghidra/Ghidra/Features/FunctionID/data/building_fid.txt`.
|
||||
|
||||
@ -25,7 +25,14 @@ addExports([
|
||||
])
|
||||
|
||||
CopySpec yajswCopySpec = copySpec {
|
||||
from(zipTree("${BIN_REPO}/Ghidra/Features/GhidraServer/${yajswRelease}.zip")) {
|
||||
File localFile = file("build/${yajswRelease}.zip")
|
||||
File binFile = file("${BIN_REPO}/Ghidra/Features/GhidraServer/${yajswRelease}.zip")
|
||||
|
||||
// First check if the file was downloaded and dropped in locally. If not, check in the bin
|
||||
// repo.
|
||||
def yajswZipTree = localFile.exists() ? zipTree(localFile) : zipTree(binFile)
|
||||
|
||||
from(yajswZipTree) {
|
||||
include "${yajswRelease}/lib/core/**"
|
||||
include "${yajswRelease}/lib/extended/**"
|
||||
include "${yajswRelease}/templates/**"
|
||||
|
||||
@ -42,22 +42,14 @@ dependencies {
|
||||
compileJava.enabled = false
|
||||
jar.enabled = false
|
||||
|
||||
File libraryJarDestDir = file("build/data")
|
||||
|
||||
File pyDevSourceZipFile = file("${BIN_REPO}/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/PyDev 6.3.1.zip")
|
||||
File cdtSourceZipFile = file("${BIN_REPO}/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/cdt-8.6.0.zip")
|
||||
|
||||
File pyDevDestDir = file("build/data/buildDependencies/pydev")
|
||||
File cdtDestDir = file("build/data/buildDependencies/cdt")
|
||||
|
||||
task utilityJar(type:Copy) {
|
||||
destinationDir libraryJarDestDir
|
||||
destinationDir file("build/data")
|
||||
|
||||
from { project(':Utility').jar } // using closure to delay until all projects evaluated
|
||||
}
|
||||
|
||||
task launchSupportJar(type:Copy) {
|
||||
destinationDir libraryJarDestDir
|
||||
destinationDir file("build/data")
|
||||
|
||||
from { project(':LaunchSupport').jar } // using closure to delay until all projects evaluated
|
||||
}
|
||||
@ -65,13 +57,22 @@ task launchSupportJar(type:Copy) {
|
||||
task pyDevUnpack(type:Copy) {
|
||||
description "Unpack PyDev plugin archive for development use"
|
||||
group "Development Preparation"
|
||||
|
||||
File pyDevDestDir = file("build/data/buildDependencies/pydev")
|
||||
|
||||
// Without this, the copyTask will unzip the file to check for "up to date"
|
||||
onlyIf {
|
||||
!pyDevDestDir.exists()
|
||||
}
|
||||
|
||||
File localFile = file("build/PyDev 6.3.1.zip")
|
||||
File binFile = file("${BIN_REPO}/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/PyDev 6.3.1.zip")
|
||||
|
||||
from zipTree(pyDevSourceZipFile)
|
||||
// First check if the file was downloaded and dropped in locally. If not, check in the bin
|
||||
// repo.
|
||||
def pyDevZipTree = localFile.exists() ? zipTree(localFile) : zipTree(binFile)
|
||||
|
||||
from pyDevZipTree
|
||||
exclude "**/.project", "**/.pydevproject"
|
||||
|
||||
destinationDir pyDevDestDir
|
||||
@ -80,13 +81,22 @@ task pyDevUnpack(type:Copy) {
|
||||
task cdtUnpack(type:Copy) {
|
||||
description "Unpack CDT plugin archive for development use"
|
||||
group "Development Preparation"
|
||||
|
||||
File cdtDestDir = file("build/data/buildDependencies/cdt")
|
||||
|
||||
// Without this, the copyTask will unzip the file to check for "up to date"
|
||||
onlyIf {
|
||||
!cdtDestDir.exists()
|
||||
}
|
||||
|
||||
File localFile = file("build/cdt-8.6.0.zip")
|
||||
File binFile = file("${BIN_REPO}/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/cdt-8.6.0.zip")
|
||||
|
||||
from zipTree(cdtSourceZipFile)
|
||||
// First check if the file was downloaded and dropped in locally. If not, check in the bin
|
||||
// repo.
|
||||
def cdtZipTree = localFile.exists() ? zipTree(localFile) : zipTree(binFile)
|
||||
|
||||
from cdtZipTree
|
||||
|
||||
destinationDir cdtDestDir
|
||||
}
|
||||
|
||||
13
build.gradle
13
build.gradle
@ -39,6 +39,19 @@ allprojects {
|
||||
}
|
||||
}
|
||||
|
||||
/*********************************************************************************
|
||||
* Use flat directory-style repository if flatRepo directory is present.
|
||||
*********************************************************************************/
|
||||
if (file("flatRepo").isDirectory()) {
|
||||
allprojects {
|
||||
repositories {
|
||||
mavenCentral()
|
||||
jcenter()
|
||||
flatDir name: "flat", dirs:["$rootProject.projectDir/flatRepo"]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/*********************************************************************************
|
||||
* load properties from Ghidra/application.properties file
|
||||
*********************************************************************************/
|
||||
|
||||
@ -4,6 +4,7 @@ distributableGPLModule.gradle||GHIDRA||||END|
|
||||
distributableGhidraExtension.gradle||GHIDRA||||END|
|
||||
distributableGhidraModule.gradle||GHIDRA||||END|
|
||||
externalGhidraExtension.gradle||GHIDRA||||END|
|
||||
fetchDependencies.gradle||GHIDRA||||END|
|
||||
helpProject.gradle||GHIDRA||||END|
|
||||
jacocoProject.gradle||GHIDRA||||END|
|
||||
javaProject.gradle||GHIDRA||||END|
|
||||
|
||||
369
gradle/fetchDependencies.gradle
Normal file
369
gradle/fetchDependencies.gradle
Normal file
@ -0,0 +1,369 @@
|
||||
/*******************************************************************************
|
||||
* fetchDependencies.gradle *
|
||||
* *
|
||||
* Fetches/downloads required dependencies that aren't available in the *
|
||||
* standard online repositories (eg: maven) and configures a flat *
|
||||
* directory-style respository that points to them. This should be run *
|
||||
* immediately after cloning the Ghidra repository before any other gradle *
|
||||
* tasks are run. *
|
||||
* *
|
||||
* Specifically, this task: *
|
||||
* *
|
||||
* 1. Downloads various dependencies required by the ghidra build and *
|
||||
* puts them in <ghidra repo>/build/downloads/. From here they are *
|
||||
* unzipped and/or copied to their final locations. The files to be *
|
||||
* downloaded: *
|
||||
* - dex-tools-2.0.zip *
|
||||
* - AXMLPrinter2.jar *
|
||||
* - hfsexplorer-0_21-bin.zip *
|
||||
* - yajsw-stable-12.12.zip *
|
||||
* - cdt-8.6.0.zip *
|
||||
* - PyDev 6.3.1.zip *
|
||||
* *
|
||||
* 2. Creates a directory at <ghidra repo>/flatRepo which is used as a *
|
||||
* flat directory-style respository for the files extracted above. *
|
||||
* *
|
||||
* usage: from the command line in the main ghidra repository *
|
||||
* directory, run the following: *
|
||||
* *
|
||||
* gradle --init-script gradle/fetchDependencies.gradle init *
|
||||
* *
|
||||
* Note: When running the script, files will only be downloaded if *
|
||||
* necessary (eg: they are not already in the build/downloads/ *
|
||||
* directory). *
|
||||
* *
|
||||
*******************************************************************************/
|
||||
|
||||
import java.util.zip.*;
|
||||
import java.nio.file.*;
|
||||
import java.security.MessageDigest;
|
||||
import org.apache.commons.io.*;
|
||||
import org.apache.commons.io.filefilter.*;
|
||||
|
||||
ext.HOME_DIR = System.getProperty('user.home')
|
||||
ext.REPO_DIR = ((Script)this).buildscript.getSourceFile().getParentFile().getParentFile()
|
||||
ext.FLAT_REPO_DIR = new File(REPO_DIR, "flatRepo")
|
||||
ext.DOWNLOADS_DIR = new File(REPO_DIR, "build/downloads")
|
||||
|
||||
// Stores the size of the file being downloaded (for formatting print statements)
|
||||
ext.FILE_SIZE = 0;
|
||||
|
||||
// The URLs for each of the dependencies
|
||||
ext.DEX_ZIP = 'https://github.com/pxb1988/dex2jar/releases/download/2.0/dex-tools-2.0.zip'
|
||||
ext.AXML_ZIP = 'https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/android4me/AXMLPrinter2.jar'
|
||||
ext.HFS_ZIP = 'https://sourceforge.net/projects/catacombae/files/HFSExplorer/0.21/hfsexplorer-0_21-bin.zip'
|
||||
ext.YAJSW_ZIP = 'https://sourceforge.net/projects/yajsw/files/yajsw/yajsw-stable-12.12/yajsw-stable-12.12.zip'
|
||||
ext.PYDEV_ZIP = 'https://sourceforge.net/projects/pydev/files/pydev/PyDev%206.3.1/PyDev%206.3.1.zip'
|
||||
ext.CDT_ZIP = 'http://www.eclipse.org/downloads/download.php?r=1&protocol=https&file=/tools/cdt/releases/8.6/cdt-8.6.0.zip'
|
||||
|
||||
// The MD5s for each of the dependencies
|
||||
ext.DEX_MD5 = '032456b9db9e6059376611553aecf31f'
|
||||
ext.AXML_MD5 = '55d70be9862c2b456cc91a933c197934'
|
||||
ext.HFS_MD5 = 'cc1713d634d2cd1fd7f21e18ae4d5d5c'
|
||||
ext.YAJSW_MD5 = 'e490ea92554f0238d74d4ef6161cb2c7'
|
||||
ext.PYDEV_MD5 = '06263bdef4917c49d8d977d12c2d5073'
|
||||
ext.CDT_MD5 = '8e9438a6e3947d614af98e1b58e945a2'
|
||||
|
||||
// Number of times to try and establish a connection when downloading files before
|
||||
// failing
|
||||
ext.NUM_RETRIES = 2
|
||||
|
||||
// Set up a maven repository configuration so we can get access to Apache FileUtils for
|
||||
// copying/deleting files.
|
||||
initscript {
|
||||
repositories {
|
||||
mavenCentral()
|
||||
}
|
||||
dependencies {
|
||||
classpath 'commons-io:commons-io:2.5'
|
||||
}
|
||||
}
|
||||
|
||||
// This is where the real flow of the script starts...
|
||||
try {
|
||||
createDirs()
|
||||
populateFlatRepo()
|
||||
}
|
||||
finally {
|
||||
cleanup()
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates the directories where the dependencies will be downloaded and stored
|
||||
*/
|
||||
def createDirs() {
|
||||
if (!DOWNLOADS_DIR.exists()) {
|
||||
DOWNLOADS_DIR.mkdirs()
|
||||
}
|
||||
if (!FLAT_REPO_DIR.exists()) {
|
||||
FLAT_REPO_DIR.mkdirs()
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Downloads a file from a URL. If there is a problem connecting to the given
|
||||
* URL the attempt will be retried NUM_RETRIES times before failing.
|
||||
*
|
||||
* Progress is shown on the command line in the form of the number of bytes
|
||||
* downloaded and a percentage of the total.
|
||||
*
|
||||
* Note: We do not validate that the number of bytes downloaded matches the
|
||||
* expected total here; any discrepencies will be caught when checking
|
||||
* the MD5s later on.
|
||||
*
|
||||
* @param url the file to download
|
||||
* @param filename the local file to create for the download
|
||||
*/
|
||||
def download(url, filename) {
|
||||
|
||||
println("File: " + url)
|
||||
BufferedInputStream istream = establishConnection(url, NUM_RETRIES);
|
||||
assert istream != null : " ***CONNECTION FAILURE***\n max attempts exceeded; exiting\n"
|
||||
|
||||
FileOutputStream ostream = new FileOutputStream(filename);
|
||||
def dataBuffer = new byte[1024];
|
||||
int bytesRead;
|
||||
int totalRead;
|
||||
while ((bytesRead = istream.read(dataBuffer, 0, 1024)) != -1) {
|
||||
|
||||
ostream.write(dataBuffer, 0, bytesRead);
|
||||
totalRead += bytesRead
|
||||
|
||||
print("\r")
|
||||
if (FILE_SIZE.equals("unknown")) {
|
||||
print(" Downloading: " + totalRead + " of " + FILE_SIZE)
|
||||
}
|
||||
else {
|
||||
int pctComplete = (totalRead / FILE_SIZE) * 100
|
||||
print(" Downloading: " + totalRead + " of " + FILE_SIZE + " (" + pctComplete + "%)")
|
||||
}
|
||||
System.out.flush()
|
||||
}
|
||||
println("")
|
||||
|
||||
istream.close();
|
||||
ostream.close();
|
||||
}
|
||||
|
||||
/**
|
||||
* Attemps to establish a connection to the given URL.
|
||||
*
|
||||
* @param url the site to connect to
|
||||
* @param retries the number of times to attempt to reconnect if there is a failure
|
||||
* @return the InputStream for the URL
|
||||
*/
|
||||
def establishConnection(url, retries) {
|
||||
for (int i=0; i<retries; i++) {
|
||||
try {
|
||||
println(" Connect attempt " + (i+1) + " of " + retries)
|
||||
URLConnection conn = new URL(url).openConnection();
|
||||
FILE_SIZE = conn.getContentLength();
|
||||
if (FILE_SIZE == -1) {
|
||||
// This can happen if there is a problem retrieving the size; we've seen it happen
|
||||
// in testing.
|
||||
FILE_SIZE = "unknown"
|
||||
}
|
||||
return new BufferedInputStream(new URL(url).openStream());
|
||||
}
|
||||
catch (Exception e) {
|
||||
println(" Connection error! " + e)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Unzips a file to a directory
|
||||
*
|
||||
* @param sourceDir the directory where the zip file resides
|
||||
* @param targetDir the directory where the unzipped files should be placed
|
||||
* @param zipFileName the name of the file to unpack
|
||||
*/
|
||||
def unzip(sourceDir, targetDir, zipFileName) {
|
||||
def zip = new ZipFile(new File(sourceDir, zipFileName))
|
||||
|
||||
zip.entries().findAll { !it.directory }.each { e ->
|
||||
(e.name as File).with { f ->
|
||||
if (f.parentFile != null) {
|
||||
File destPath = new File(targetDir.path, f.parentFile.path)
|
||||
destPath.mkdirs()
|
||||
File targetFile = new File(destPath.path, f.name)
|
||||
targetFile.withOutputStream { w ->
|
||||
w << zip.getInputStream(e)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Downloads and stores the necessary dependencies in the local flat repository.
|
||||
*
|
||||
* If the dependency already exists in the downloads folder (DOWNLOADS_DIR) and has the
|
||||
* proper checksum, it will NOT be re-downloaded.
|
||||
*/
|
||||
def populateFlatRepo() {
|
||||
|
||||
// 1. Download all the dependencies and verify their checksums. If the dependency has already
|
||||
// been download, do NOT download again.
|
||||
File file = new File(DOWNLOADS_DIR, 'dex-tools-2.0.zip')
|
||||
if (!DEX_MD5.equals(generateChecksum(file))) {
|
||||
download (DEX_ZIP, file.path)
|
||||
validateChecksum(generateChecksum(file), DEX_MD5);
|
||||
}
|
||||
|
||||
file = new File(DOWNLOADS_DIR, 'AXMLPrinter2.jar')
|
||||
if (!AXML_MD5.equals(generateChecksum(file))) {
|
||||
download (AXML_ZIP, file.path)
|
||||
validateChecksum(generateChecksum(file), AXML_MD5);
|
||||
}
|
||||
|
||||
file = new File(DOWNLOADS_DIR, 'hfsexplorer-0_21-bin.zip')
|
||||
if (!HFS_MD5.equals(generateChecksum(file))) {
|
||||
download (HFS_ZIP, file.path)
|
||||
validateChecksum(generateChecksum(file), HFS_MD5);
|
||||
}
|
||||
|
||||
file = new File(DOWNLOADS_DIR, 'yajsw-stable-12.12.zip')
|
||||
if (!YAJSW_MD5.equals(generateChecksum(file))) {
|
||||
download (YAJSW_ZIP, file.path)
|
||||
validateChecksum(generateChecksum(file), YAJSW_MD5);
|
||||
}
|
||||
|
||||
file = new File(DOWNLOADS_DIR, 'PyDev 6.3.1.zip')
|
||||
if (!PYDEV_MD5.equals(generateChecksum(file))) {
|
||||
download (PYDEV_ZIP, file.path)
|
||||
validateChecksum(generateChecksum(file), PYDEV_MD5);
|
||||
}
|
||||
|
||||
file = new File(DOWNLOADS_DIR, 'cdt-8.6.0.zip')
|
||||
if (!CDT_MD5.equals(generateChecksum(file))) {
|
||||
download (CDT_ZIP, file.path)
|
||||
validateChecksum(generateChecksum(file), CDT_MD5);
|
||||
}
|
||||
|
||||
// 2. Unzip the dependencies
|
||||
unzip(DOWNLOADS_DIR, DOWNLOADS_DIR, "dex-tools-2.0.zip")
|
||||
unzipHfsx()
|
||||
|
||||
// 3. Copy the necessary jars to the flatRepo directory. Yajsw, CDT, and PyDev go directly into
|
||||
// the source repository.
|
||||
copyDexTools()
|
||||
copyAXML()
|
||||
copyHfsx()
|
||||
copyYajsw()
|
||||
copyPyDev()
|
||||
copyCdt()
|
||||
}
|
||||
|
||||
/**
|
||||
* Generates the md5 for the given file
|
||||
*
|
||||
* @param file the file to generate the checksum for
|
||||
* @return the generated checksum
|
||||
*/
|
||||
def generateChecksum(file) {
|
||||
if (!file.exists()) {
|
||||
return null
|
||||
}
|
||||
MessageDigest md = MessageDigest.getInstance("MD5");
|
||||
md.update(Files.readAllBytes(Paths.get(file.path)));
|
||||
byte[] digest = md.digest();
|
||||
StringBuilder sb = new StringBuilder();
|
||||
for (byte b : digest) {
|
||||
sb.append(String.format("%02x", b));
|
||||
}
|
||||
|
||||
return sb.toString();
|
||||
}
|
||||
|
||||
/**
|
||||
* Compares two checksums and generates an assert failure if they do not match
|
||||
*
|
||||
* @param sourceMd5 the checksum to validate
|
||||
* @param expectedMd5 the expected checksum
|
||||
*/
|
||||
def validateChecksum(sourceMd5, expectedMd5) {
|
||||
assert(sourceMd5.equals(expectedMd5));
|
||||
}
|
||||
|
||||
/**
|
||||
* Unzips the hfsx zip file
|
||||
*/
|
||||
def unzipHfsx() {
|
||||
def hfsxdir = getOrCreateTempHfsxDir()
|
||||
unzip (DOWNLOADS_DIR, hfsxdir, "hfsexplorer-0_21-bin.zip")
|
||||
}
|
||||
|
||||
/**
|
||||
* Copies the dex-tools jars to the flat repository
|
||||
*
|
||||
* Note: This will only copy files beginning with "dex-"
|
||||
*/
|
||||
def copyDexTools() {
|
||||
FileUtils.copyDirectory(new File(DOWNLOADS_DIR, 'dex2jar-2.0/lib/'), FLAT_REPO_DIR, new WildcardFileFilter("dex-*"));
|
||||
}
|
||||
|
||||
/**
|
||||
* Copies the AXMLPrinter2 jar to the flat repository
|
||||
*/
|
||||
def copyAXML() {
|
||||
FileUtils.copyFile(new File(DOWNLOADS_DIR, 'AXMLPrinter2.jar'), new File(FLAT_REPO_DIR, "AXMLPrinter2.jar"));
|
||||
}
|
||||
|
||||
/**
|
||||
* Copies the necessary hfsx jars to the flat repository
|
||||
*/
|
||||
def copyHfsx() {
|
||||
FileUtils.copyFile(new File(DOWNLOADS_DIR, "hfsx/lib/csframework.jar"), new File(FLAT_REPO_DIR, "csframework.jar"));
|
||||
FileUtils.copyFile(new File(DOWNLOADS_DIR, "hfsx/lib/hfsx_dmglib.jar"), new File(FLAT_REPO_DIR, "hfsx_dmglib.jar"));
|
||||
FileUtils.copyFile(new File(DOWNLOADS_DIR, "hfsx/lib/hfsx.jar"), new File(FLAT_REPO_DIR, "hfsx.jar"));
|
||||
FileUtils.copyFile(new File(DOWNLOADS_DIR, "hfsx/lib/iharder-base64.jar"), new File(FLAT_REPO_DIR, "iharder-base64.jar"));
|
||||
}
|
||||
|
||||
/**
|
||||
* Copies the yajswdir zip to its location in the GhidraServer project.
|
||||
*/
|
||||
def copyYajsw() {
|
||||
FileUtils.copyFile(new File(DOWNLOADS_DIR, "yajsw-stable-12.12.zip"), new File(REPO_DIR, "Ghidra/Features/GhidraServer/build/yajsw-stable-12.12.zip"));
|
||||
}
|
||||
|
||||
/**
|
||||
* Copies the pydev zip to its bin repository location
|
||||
*/
|
||||
def copyPyDev() {
|
||||
FileUtils.copyFile(new File(DOWNLOADS_DIR, 'PyDev 6.3.1.zip'), new File(REPO_DIR, "GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/PyDev 6.3.1.zip"));
|
||||
}
|
||||
|
||||
/**
|
||||
* Copies the cdt zip to its bin repository location
|
||||
*/
|
||||
def copyCdt() {
|
||||
FileUtils.copyFile(new File(DOWNLOADS_DIR, 'cdt-8.6.0.zip'), new File(REPO_DIR, "GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/cdt-8.6.0.zip"));
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a temporary folder to house the hfsx zip contents
|
||||
*
|
||||
* @return the newly-created hfsx directory object
|
||||
*/
|
||||
def getOrCreateTempHfsxDir() {
|
||||
def hfsxdir = new File (DOWNLOADS_DIR, "hfsx")
|
||||
if (!hfsxdir.exists()) {
|
||||
hfsxdir.mkdir()
|
||||
}
|
||||
|
||||
return hfsxdir;
|
||||
}
|
||||
|
||||
/**
|
||||
* Performs any cleanup operations that need to be performed after the flat repo has
|
||||
* been populated.
|
||||
*/
|
||||
def cleanup() {
|
||||
// Uncomment this if we want to delete the downloads folder. For now, leave this and
|
||||
// depend on a gradle clean to wipe it out.
|
||||
//
|
||||
//if (DOWNLOADS_DIR.exists()) {
|
||||
// FileUtils.deleteDirectory(DOWNLOADS_DIR)
|
||||
//}
|
||||
}
|
||||
Loading…
Reference in New Issue
Block a user