Merge remote-tracking branch

'origin/GP-3155_caheckman_PR-2810_Pokechu22_countleadingzeros'
(Closes #2810)
This commit is contained in:
Ryan Kurtz
2023-03-24 14:27:51 -04:00
94 changed files with 3604 additions and 3451 deletions

View File

@@ -62,7 +62,7 @@ task unpackFidDatabases {
// Relative to the 'workingDir' Exec task property.
def installPoint = "../help/help"
task buildFidDocumentationPdf(type: Exec) {
task buildFidHelpPdf(type: Exec) {
workingDir 'src/main/doc'
@@ -91,7 +91,7 @@ task buildFidDocumentationPdf(type: Exec) {
cp $installPoint/topics/FunctionID/images/*.png $buildDir/images
echo '** Building FunctionID.fo **'
xsltproc --output $buildDir/fid_withscaling.xml --stringparam profile.condition "withscaling" /usr/share/sgml/docbook/xsl-stylesheets/profiling/profile.xsl fid.xml 2>&1
xsltproc --output $buildDir/fid_withscaling.xml --stringparam profile.condition "withscaling" commonprofile.xsl fid.xml 2>&1
xsltproc --output $buildDir/FunctionID.fo fid_pdf.xsl $buildDir/fid_withscaling.xml 2>&1
echo '** Building FunctionID.pdf **'
@@ -137,10 +137,12 @@ task buildFidDocumentationPdf(type: Exec) {
* A build (ex: 'gradle buildLocal') will place the html files in the distribution.
* There is an associated, auto-generated clean task.
**/
task buildFidDocumentationHtml(type: Exec) {
task buildFidHelpHtml(type: Exec) {
workingDir 'src/main/doc'
def buildDir = "../../../build/html"
// 'which' returns the number of failed arguments
// Using the 'which' command first will allow the task to fail if the required
// executables are not installed.
@@ -158,9 +160,10 @@ task buildFidDocumentationHtml(type: Exec) {
rm -f $installPoint/topics/FunctionID/*.html
echo '** Building html files **'
xsltproc --output $buildDir/fid_noscaling.xml --stringparam profile.condition "noscaling" /usr/share/sgml/docbook/xsl-stylesheets/profiling/profile.xsl fid.xml 2>&1
xsltproc --output $buildDir/fid_noscaling.xml --stringparam profile.condition "noscaling" commonprofile.xsl fid.xml 2>&1
xsltproc --stringparam base.dir ${installPoint}/topics/FunctionID/ fid_html.xsl $buildDir/fid_noscaling.xml 2>&1
sed -i -e '/Frontpage.css/ { p; s/Frontpage.css/languages.css/; }' ${installPoint}/topics/FunctionID/*.html
rm ${installPoint}/topics/FunctionID/index.html
sed -i -e '/DefaultStyle.css/ { p; sQhref=".*"Qhref="../../shared/languages.css"Q; }' ${installPoint}/topics/FunctionID/*.html
echo '** Done. **'
"""

View File

@@ -4,6 +4,7 @@ Module.manifest||GHIDRA||||END|
data/building_fid.txt||GHIDRA||||END|
data/common_symbols_win32.txt||GHIDRA|||Symbols used to generate fiddb files distributed with Ghidra|END|
data/common_symbols_win64.txt||GHIDRA|||Symbols used to generate fiddb files distributed with Ghidra|END|
src/main/doc/commonprofile.xsl||GHIDRA||||END|
src/main/doc/fid.xml||GHIDRA||||END|
src/main/doc/fid_common.xsl||GHIDRA||||END|
src/main/doc/fid_html.xsl||GHIDRA||||END|

View File

@@ -0,0 +1,6 @@
<?xml version='1.0'?>
<xsl:stylesheet
xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:import href="http://docbook.sourceforge.net/release/xsl/current/profiling/profile.xsl"/>
</xsl:stylesheet>

View File

@@ -41,7 +41,7 @@ from Microsoft Visual Studio for the x86 processor. These have been broken apart
separate Function ID databases, based on 32-bit or 64-code and the version of Visual Studio.
Within each database, there are a two library variants -- one for debug versions and one for production.
</para>
<sect2>
<sect2 id="hashing">
<title>Hashing</title>
<para>
Function ID works by calculating a cumulative hash over all the machine <emphasis>instructions</emphasis>
@@ -82,7 +82,7 @@ is robust against changes due to linking; the <emphasis role="bold">specific has
helps distinguish between closely related variants of a function.
</para>
</sect2>
<sect2>
<sect2 id="parentchild">
<title>Parents and Children</title>
<para>
When Function ID examines a function, its parent and child functions are also considered
@@ -92,7 +92,7 @@ the full hashes of the functions will be identical, but the system will try to m
the two subfunctions, allowing it to distinguish between the two.
</para>
</sect2>
<sect2>
<sect2 id="libraries">
<title>Libraries</title>
<para>
Within a Function ID database, functions are grouped into <emphasis>libraries</emphasis>,
@@ -144,7 +144,7 @@ the analyzer will still report a single match but will leave off the fields it
couldn't distinguish.
</para>
</sect2>
<sect2>
<sect2 id="singlematches">
<title>Single Matches</title>
<para>
A <emphasis role="bold">Single Match</emphasis> for a function occurs under the following conditions:
@@ -283,7 +283,7 @@ increasing its overall score.
If there are still more than one potential match, the highest assigned score is
used to filter out matches with lower scores.
</para>
<sect2>
<sect2 id="matchingfunction">
<title>Matching Function Names</title>
<para>
If there are still multiple potential matches once thresholds have been applied to the
@@ -416,7 +416,7 @@ to populate the database.
<imagedata condition="withscaling" fileref="images/PopulateFidDbFromPrograms1.png" width="100%" contentwidth="4in" contentdepth="3.033175in" align="center"/>
</imageobject>
</mediaobject>
<sect3>
<sect3 id="dialogfields">
<title>Dialog Fields</title>
<para>
<informalexample>
@@ -519,7 +519,7 @@ most commonly called within the library. This list can be used to create a
</section>
<section id="preparelibraries">
<title>Preparing Libraries for a Function ID Database</title>
<sect2>
<sect2 id="programlocation">
<title>Location of Programs</title>
<para>
All functions going into a single Function ID <emphasis>Library</emphasis> must already be imported and analyzed
@@ -530,7 +530,7 @@ the root for the library. The process acts recursively, so there can be addition
but all programs to be included in the library must be under the one root.
</para>
</sect2>
<sect2>
<sect2 id="analysis">
<title>Analysis</title>
<para>
All programs must be analyzed enough to have recovered the bodies of all the functions that are to be included
@@ -572,7 +572,7 @@ function that is declared as a match by the analyzer but has the incorrect symbo
As with any classification algorithm, it is generally not possible to eliminate this kind
of error completely, but with Function ID there are some mitigation strategies.
</para>
<sect2>
<sect2 id="causes">
<title>Causes</title>
<para>
False positives for the most part only happen with small functions.
@@ -611,7 +611,7 @@ There are two related causes with Function ID:
In either case, Function ID can apply a symbol that is misleading for the analyst.
</para>
</sect2>
<sect2>
<sect2 id="mitigation">
<title>Mitigation via Threshold</title>
<para>
All mitigation strategies, to some extent, trade-off false positives for
@@ -758,7 +758,7 @@ section and check the box next to "FidDebugPlugin".
The Function ID Debug Plug-in introduces the following actions to the
<emphasis role="bold">Tools -> Function ID</emphasis> menu.
</para>
<sect2>
<sect2 id="readonly">
<title>Create Read-only Database</title>
<para>
Users can convert the read/write (.fidb) database into the a read-only (.fidbf) form. This is
@@ -787,7 +787,7 @@ the <command>RETURN</command> key, with the cursor and focus still in the desire
<imagedata condition="withscaling" fileref="images/FIDSearch.png" width="100%" contentwidth="4in" contentdepth="3.3397in" align="center"/>
</imageobject>
</mediaobject>
<sect3>
<sect3 id="searchfields">
<title>Search Fields</title>
<para>
<informalexample>
@@ -845,7 +845,7 @@ the <command>RETURN</command> key, with the cursor and focus still in the desire
</informalexample>
</para>
</sect3>
<sect3>
<sect3 id="resultwindow">
<title>Result Window</title>
<para>
Invoking a search will bring up the <emphasis>Result Window</emphasis>, presenting a row for
@@ -901,7 +901,7 @@ other columns:
</variablelist>
</informalexample>
</para>
<sect4>
<sect4 id="editmenu">
<title>Edit Menu</title>
<para>
The <emphasis>Result Window</emphasis> supports a small number of actions under the
@@ -962,7 +962,7 @@ strategy.
</sect4>
</sect3>
</sect2>
<sect2>
<sect2 id="tableviewer">
<title>Table Viewer</title>
<para>
This invokes an extremely low-level view into the underlying tables that back a

View File

@@ -2,7 +2,7 @@
<xsl:stylesheet
xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:import href="/usr/share/sgml/docbook/xsl-stylesheets/html/chunk.xsl"/>
<xsl:import href="http://docbook.sourceforge.net/release/xsl/current/html/chunk.xsl"/>
<xsl:include href="fid_common.xsl" />
@@ -40,9 +40,15 @@
</xsl:element>
</xsl:template>
<xsl:template name="body.attributes">
<!-- Remove all BODY attributes so that CSS stylesheet can provide everything -->
</xsl:template>
<xsl:param name="suppress.navigation" select="1"/> <!-- Turn off header/footer navigation links -->
<xsl:param name="use.id.as.filename" select="1"/> <!-- Split up into files based on id attribute -->
<xsl:param name="html.stylesheet" select="'../../shared/Frontpage.css'"/> <!-- Use our custom cascading style sheet -->
<xsl:param name="html.stylesheet" select="'help/shared/DefaultStyle.css'"/> <!-- Use our custom cascading style sheet -->
<xsl:param name="chunk.section.depth" select="0"/>

View File

@@ -2,7 +2,7 @@
<xsl:stylesheet
xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:import href="/usr/share/sgml/docbook/xsl-stylesheets/fo/docbook.xsl"/>
<xsl:import href="http://docbook.sourceforge.net/release/xsl/current/fo/docbook.xsl"/>
<xsl:template match="table" mode="label.markup"/>

View File

@@ -3,24 +3,14 @@
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<title>Function ID</title>
<link rel="stylesheet" type="text/css" href="help/shared/DefaultStyle.css">
<link rel="stylesheet" type="text/css" href="help/shared/languages.css">
<meta name="generator" content="DocBook XSL Stylesheets V1.78.1">
<link rel="stylesheet" type="text/css" href="../../shared/languages.css">
<meta name="generator" content="DocBook XSL Stylesheets V1.79.1">
<link rel="home" href="index.html" title="Function ID">
<link rel="up" href="index.html" title="Function ID">
<link rel="prev" href="index.html" title="Function ID">
<link rel="next" href="FunctionIDPlugin.html" title="Function ID Plug-in">
</head>
<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
<div class="navheader">
<table width="100%" summary="Navigation header">
<tr><th colspan="3" align="center">Function ID</th></tr>
<tr>
<td width="20%" align="left"></td>
<th width="60%" align="center"><EFBFBD></th>
<td width="20%" align="right"><EFBFBD><a accesskey="n" href="FunctionIDPlugin.html">Next</a>
</td>
</tr>
</table>
<hr>
</div>
<div class="chapter">
<body><div class="chapter">
<div class="titlepage"><div><div><h1 class="title">
<a name="FunctionID"></a>Function ID</h1></div></div></div>
<div class="mediaobject" align="center"><table border="0" summary="manufactured viewport for HTML img" style="cellpadding: 0; cellspacing: 0;" width="100%"><tr><td align="center"><img src="images/FIDmatch.png" align="middle" width="723" height="267"></td></tr></table></div>
@@ -57,7 +47,7 @@ Within each database, there are a two library variants -- one for debug versions
</p>
<div class="sect2">
<div class="titlepage"><div><div><h3 class="title">
<a name="idm140323472630336"></a>Hashing</h3></div></div></div>
<a name="hashing"></a>Hashing</h3></div></div></div>
<p>
Function ID works by calculating a cumulative hash over all the machine <span class="emphasis"><em>instructions</em></span>
that make up the body of a function. For each function, two different 64-bit hashes are computed: a
@@ -90,7 +80,7 @@ helps distinguish between closely related variants of a function.
</div>
<div class="sect2">
<div class="titlepage"><div><div><h3 class="title">
<a name="idm140323472620992"></a>Parents and Children</h3></div></div></div>
<a name="parentchild"></a>Parents and Children</h3></div></div></div>
<p>
When Function ID examines a function, its parent and child functions are also considered
as a way of disambiguating multiple matches. For example, suppose two functions have identical
@@ -101,7 +91,7 @@ the two subfunctions, allowing it to distinguish between the two.
</div>
<div class="sect2">
<div class="titlepage"><div><div><h3 class="title">
<a name="idm140323472619376"></a>Libraries</h3></div></div></div>
<a name="libraries"></a>Libraries</h3></div></div></div>
<p>
Within a Function ID database, functions are grouped into <span class="emphasis"><em>libraries</em></span>,
which are intended to be recognizable named software components
@@ -141,7 +131,7 @@ couldn't distinguish.
</div>
<div class="sect2">
<div class="titlepage"><div><div><h3 class="title">
<a name="idm140323472607824"></a>Single Matches</h3></div></div></div>
<a name="singlematches"></a>Single Matches</h3></div></div></div>
<p>
A <span class="bold"><strong>Single Match</strong></span> for a function occurs under the following conditions:
</p>
@@ -268,7 +258,7 @@ used to filter out matches with lower scores.
</p>
<div class="sect2">
<div class="titlepage"><div><div><h3 class="title">
<a name="idm140323472572880"></a>Matching Function Names</h3></div></div></div>
<a name="matchingfunction"></a>Matching Function Names</h3></div></div></div>
<p>
If there are still multiple potential matches once thresholds have been applied to the
match scores, the remaining matches will be grouped based on function names. If
@@ -284,22 +274,5 @@ parameter information is stripped.
</p>
</div>
</div>
</div>
<div class="navfooter">
<hr>
<table width="100%" summary="Navigation footer">
<tr>
<td width="40%" align="left"></td>
<td width="20%" align="center"><EFBFBD></td>
<td width="40%" align="right"><EFBFBD><a accesskey="n" href="FunctionIDPlugin.html">Next</a>
</td>
</tr>
<tr>
<td width="40%" align="left" valign="top">Function ID<49></td>
<td width="20%" align="center"></td>
<td width="40%" align="right" valign="top"><EFBFBD>Function ID Plug-in</td>
</tr>
</table>
</div>
</body>
</div></body>
</html>

View File

@@ -3,24 +3,13 @@
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<title>Function ID Debug Plug-in</title>
<link rel="stylesheet" type="text/css" href="help/shared/DefaultStyle.css">
<link rel="stylesheet" type="text/css" href="help/shared/languages.css">
<meta name="generator" content="DocBook XSL Stylesheets V1.78.1">
<link rel="stylesheet" type="text/css" href="../../shared/languages.css">
<meta name="generator" content="DocBook XSL Stylesheets V1.79.1">
<link rel="home" href="index.html" title="Function ID">
<link rel="up" href="index.html" title="Function ID">
<link rel="prev" href="FunctionIDPlugin.html" title="Function ID Plug-in">
</head>
<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
<div class="navheader">
<table width="100%" summary="Navigation header">
<tr><th colspan="3" align="center">Function ID Debug Plug-in</th></tr>
<tr>
<td width="20%" align="left">
<a accesskey="p" href="FunctionIDPlugin.html">Prev</a><EFBFBD></td>
<th width="60%" align="center"><EFBFBD></th>
<td width="20%" align="right"><EFBFBD></td>
</tr>
</table>
<hr>
</div>
<div class="chapter">
<body><div class="chapter">
<div class="titlepage"><div><div><h1 class="title">
<a name="FunctionIDDebug"></a>Function ID Debug Plug-in</h1></div></div></div>
<p>
@@ -54,7 +43,7 @@ The Function ID Debug Plug-in introduces the following actions to the
</p>
<div class="sect2">
<div class="titlepage"><div><div><h3 class="title">
<a name="idm140323472459600"></a>Create Read-only Database</h3></div></div></div>
<a name="readonly"></a>Create Read-only Database</h3></div></div></div>
<p>
Users can convert the read/write (.fidb) database into the a read-only (.fidbf) form. This is
the more efficient final form used directly by the Function ID analyzer. The .fidbf form is
@@ -80,7 +69,7 @@ the <span class="command"><strong>RETURN</strong></span> key, with the cursor an
<div class="mediaobject" align="center"><table border="0" summary="manufactured viewport for HTML img" style="cellpadding: 0; cellspacing: 0;" width="100%"><tr><td align="center"><img src="images/FIDSearch.png" align="middle" width="315" height="263"></td></tr></table></div>
<div class="sect3">
<div class="titlepage"><div><div><h4 class="title">
<a name="idm140323472451184"></a>Search Fields</h4></div></div></div>
<a name="searchfields"></a>Search Fields</h4></div></div></div>
<p>
</p>
<div class="informalexample"><div class="variablelist"><dl class="variablelist">
@@ -118,7 +107,7 @@ the <span class="command"><strong>RETURN</strong></span> key, with the cursor an
</div>
<div class="sect3">
<div class="titlepage"><div><div><h4 class="title">
<a name="idm140323472437088"></a>Result Window</h4></div></div></div>
<a name="resultwindow"></a>Result Window</h4></div></div></div>
<p>
Invoking a search will bring up the <span class="emphasis"><em>Result Window</em></span>, presenting a row for
each matching function record. Columns list properties of the function and correspond
@@ -162,7 +151,7 @@ other columns:
</p>
<div class="sect4">
<div class="titlepage"><div><div><h5 class="title">
<a name="idm140323472422400"></a>Edit Menu</h5></div></div></div>
<a name="editmenu"></a>Edit Menu</h5></div></div></div>
<p>
The <span class="emphasis"><em>Result Window</em></span> supports a small number of actions under the
<span class="bold"><strong>Edit</strong></span> menu that allow the user to change the
@@ -203,7 +192,7 @@ strategy.
</div>
<div class="sect2">
<div class="titlepage"><div><div><h3 class="title">
<a name="idm140323472406032"></a>Table Viewer</h3></div></div></div>
<a name="tableviewer"></a>Table Viewer</h3></div></div></div>
<p>
This invokes an extremely low-level view into the underlying tables that back a
Function ID database. It can be invoked on any attached database. A window is brought up
@@ -214,22 +203,5 @@ present readable values. The only meaningful table is likely to be the
</p>
</div>
</div>
</div>
<div class="navfooter">
<hr>
<table width="100%" summary="Navigation footer">
<tr>
<td width="40%" align="left">
<a accesskey="p" href="FunctionIDPlugin.html">Prev</a><EFBFBD></td>
<td width="20%" align="center"><EFBFBD></td>
<td width="40%" align="right"><EFBFBD></td>
</tr>
<tr>
<td width="40%" align="left" valign="top">Function ID Plug-in<69></td>
<td width="20%" align="center"></td>
<td width="40%" align="right" valign="top"><EFBFBD></td>
</tr>
</table>
</div>
</body>
</div></body>
</html>

View File

@@ -3,26 +3,14 @@
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<title>Function ID Plug-in</title>
<link rel="stylesheet" type="text/css" href="help/shared/DefaultStyle.css">
<link rel="stylesheet" type="text/css" href="help/shared/languages.css">
<meta name="generator" content="DocBook XSL Stylesheets V1.78.1">
<link rel="stylesheet" type="text/css" href="../../shared/languages.css">
<meta name="generator" content="DocBook XSL Stylesheets V1.79.1">
<link rel="home" href="index.html" title="Function ID">
<link rel="up" href="index.html" title="Function ID">
<link rel="prev" href="FunctionID.html" title="Function ID">
<link rel="next" href="FunctionIDDebug.html" title="Function ID Debug Plug-in">
</head>
<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
<div class="navheader">
<table width="100%" summary="Navigation header">
<tr><th colspan="3" align="center">Function ID Plug-in</th></tr>
<tr>
<td width="20%" align="left">
<a accesskey="p" href="FunctionID.html">Prev</a><EFBFBD></td>
<th width="60%" align="center"><EFBFBD></th>
<td width="20%" align="right"><EFBFBD><a accesskey="n" href="FunctionIDDebug.html">Next</a>
</td>
</tr>
</table>
<hr>
</div>
<div class="chapter">
<body><div class="chapter">
<div class="titlepage"><div><div><h1 class="title">
<a name="FunctionIDPlugin"></a>Function ID Plug-in</h1></div></div></div>
<p>
@@ -132,7 +120,7 @@ to populate the database.
<div class="mediaobject" align="center"><table border="0" summary="manufactured viewport for HTML img" style="cellpadding: 0; cellspacing: 0;" width="100%"><tr><td align="center"><img src="images/PopulateFidDbFromPrograms1.png" align="middle" width="422" height="320"></td></tr></table></div>
<div class="sect3">
<div class="titlepage"><div><div><h4 class="title">
<a name="idm140323472541088"></a>Dialog Fields</h4></div></div></div>
<a name="dialogfields"></a>Dialog Fields</h4></div></div></div>
<p>
</p>
<div class="informalexample"><div class="variablelist"><dl class="variablelist">
@@ -204,7 +192,7 @@ most commonly called within the library. This list can be used to create a
<a name="preparelibraries"></a>Preparing Libraries for a Function ID Database</h2></div></div></div>
<div class="sect2">
<div class="titlepage"><div><div><h3 class="title">
<a name="idm140323472516592"></a>Location of Programs</h3></div></div></div>
<a name="programlocation"></a>Location of Programs</h3></div></div></div>
<p>
All functions going into a single Function ID <span class="emphasis"><em>Library</em></span> must already be imported and analyzed
somewhere within a single Ghidra repository (shared or non-shared). Multiple libraries contained within
@@ -216,7 +204,7 @@ but all programs to be included in the library must be under the one root.
</div>
<div class="sect2">
<div class="titlepage"><div><div><h3 class="title">
<a name="idm140323472513952"></a>Analysis</h3></div></div></div>
<a name="analysis"></a>Analysis</h3></div></div></div>
<p>
All programs must be analyzed enough to have recovered the bodies of all the functions that are to be included
in the library. Generally, the easiest way to accomplish this is to run Ghidra's default auto-analysis.
@@ -260,7 +248,7 @@ of error completely, but with Function ID there are some mitigation strategies.
</p>
<div class="sect2">
<div class="titlepage"><div><div><h3 class="title">
<a name="idm140323472502064"></a>Causes</h3></div></div></div>
<a name="causes"></a>Causes</h3></div></div></div>
<p>
False positives for the most part only happen with small functions.
There are two related causes with Function ID:
@@ -291,7 +279,7 @@ In either case, Function ID can apply a symbol that is misleading for the analys
</div>
<div class="sect2">
<div class="titlepage"><div><div><h3 class="title">
<a name="idm140323472493552"></a>Mitigation via Threshold</h3></div></div></div>
<a name="mitigation"></a>Mitigation via Threshold</h3></div></div></div>
<p>
All mitigation strategies, to some extent, trade-off false positives for
<span class="bold"><strong>false negatives</strong></span>, which are functions that should have
@@ -391,23 +379,5 @@ script.
</p>
</div>
</div>
</div>
<div class="navfooter">
<hr>
<table width="100%" summary="Navigation footer">
<tr>
<td width="40%" align="left">
<a accesskey="p" href="FunctionID.html">Prev</a><EFBFBD></td>
<td width="20%" align="center"><EFBFBD></td>
<td width="40%" align="right"><EFBFBD><a accesskey="n" href="FunctionIDDebug.html">Next</a>
</td>
</tr>
<tr>
<td width="40%" align="left" valign="top">Function ID<49></td>
<td width="20%" align="center"></td>
<td width="40%" align="right" valign="top"><EFBFBD>Function ID Debug Plug-in</td>
</tr>
</table>
</div>
</body>
</div></body>
</html>