In this short post, we will show how the debuggers API can be used to monitor an app execution, hook into various key methods and classes of the standard Java cryptography SPI, and extract input and output data, as they flow in and out encryption/decryption routines.
Very handy to retrieve encrypted data used within an app or exchanged with a remote server. 1 Check out the following video to see what we are talking about:
int update(byte[] input, int inputOffset, int inputLen, byte[] output)
When any of the hooked method is called, the associated hook onEntry method is executed, which will dump interesting input parameters
When the same hooked method returns, the associated hook onExit method is executed, which will dump interesting exit parameters and return value
The hook here consists of a double breakpoint, one triggered when a method is entered, another one, when it exits.
A hook on doFinal() capturing plain text data just before it gets encrypted
The code for that Java plugin is fairly simple. More hooks could be easily added, and hooks in native libraries could be set up in a similar fashion. Lastly, always keep in mind that the API in general (and this plugin in particular) can be leveraged by UI or headless clients. Automate things away if you need to.
The one and only entry-point for developer resources is our Developer Portal. Do not hesitate to reach out, publicly or privately, if you have issues or pointed questions. Thank you.
Dynamic execution monitoring can be achieved in several ways. Debugging a target is one of them. ↩
Dalvik and native code debugging has been supported in JEB since version 2.2:
Linux ELF and Windows PE code object support (32-bit and 64-bit)
Disassemblers for Intel x86 and x86-64, ARM 32-bit (including floating point support), ARM 64-bit (Aarch64) and MIPS.
Debuggers for Android Dalvik virtual machines and native Android processes
This post presents the details and caveats pertaining to the Android debuggers, shows how to debug APK’s Dalvik and native code, and explains the debuggers limitations.
An on-going debugging session of Android Dalvik and native ARM code.
Introduction
Debugging closed-source Android apps has historically been challenging at best, close to impossible at worst. JEB took a stab at solving this problem by abstracting away a wealth of low-level details and caveats related to debugging so that analysts can focus on the Dalvik code and associated decompiled Java source, as well as native code.
The Android debuggers make the task of reverse-engineering complex apps, e.g. those using a mix and bytecode and machine code, finally possible in practice. With the rise of app protectors and obfuscators, support for full-scale debugging has become more and more pressing. Earlier in February, we published a video highlighting one major feature of these debuggers: the ability to seamlessly debug to-and-from Dalvik and native code. We will explain in details how to use the features highlighted in the video.
Another area we will explore is the debugging API. The debuggers abide to the JEB IDebuggerUnit family set of interface. They can be used to automate debugging tasks, and allow for easy integration in analysis pipelines.
Requirements
The JEB Android debuggers run on all JEB-supported platforms (Windows, Linux, macOS). Please verify the following before attempting to start a debugging session:
Make sure to have the Android SDK installed. Ideally, you also want to have either ANDROID_SDK_ROOT or ANDROID_SDK environment variable pointing to the SDK folder.
Enable Developer options and allow USB debugging on the intended physical target device. (Debugging is enabled by default on the emulators.) On physical devices running Android 4.2 and above, one way to make sure of that is to run the adb devices command. If the device is shown as unauthorized, a pop-up on your phone will appear to request authorization.
Run the app in an emulator. Emulators have the ro.debuggable property set to 1. This means they will debug all apps, regardless of the debuggable flag in the Manifest.
Use a rooted phone. A rooted phone will allow you to modify the ro.debuggable property, and change it from 0 (standard on production devices) to 1. The rooting process is out-of-scope for this document: it is device specific and rooting instructions can easily be found online. As for ro.debuggable, we will explain how to change this system property in a separate blog entry.
Unpack/modify/repack your app. (Update: JEB can do this for you) Depending on whether the Manifest is obfuscated or not, this may be the simplest option. If the Manifest is obfuscated, the repacking process may not work. Another caveat applies: signing. You will have to sign the repacked app using your own key; be aware of the implications if you choose that option. 1
Caveat: Native code in non-debuggable apps
When it comes to debugging native code of non-debuggable apps on a rooted phone or emulator, other limitations apply. 2 JEB tries its best at abstracting them away. However, things might be unstable depending on which phone and OS is being used. Do not hesitate to let us know if you encounter issues.
Note that most of our tests are done on Nexus devices running vanilla Android 5.1 and 6.0. Using similar devices for debugging will likely reduce the chances of running into corner-case problematic situations.
Starting a debugging session
Currently, JEB can start Android debugging sessions only when analyzing APK files. If your main artifact in JEB is an orphan DEX file, the UI client will refuse to start a debugging session.
First, retrieve your target APK and get the app ready for debugging:
Make sure the APK matches the one that will be executed on the target phone. You can download the APK using adb3:
adb shell pm list packages -f to retrieve a list of packages and the associated path to APK
adb pull <pathToAPK> to download the APK
Start the app on the phone
Via the App Launcher for instance, if attaching to an already running app is an acceptable scenario
If you want the app to wait for the debuggers to attach to it before it starts executing any code, you can run something like: adb shell am start -D -S -n <packageName>/<activityName>
A popup will be displayed on the phone, indicating it is waiting for a debugger to attach to the VM
Second, in the RCP desktop client:
Start an analysis of the APK file
Open a view of a main DEX file
Once the focus is on the DEX view, open the Debugger menu, and click on Start…
The Debugger/Start, used to start or attach a debugger, is available once the code view of a support Code unit has the focus. Here, the focus was on Dalvik bytecode.
In the Attach dialog window:
Select the target phone and the target process that matches your app, and click Attach.
Unless you tick the “Suspend all threads”, The app will be immediately be run/resumed after attaching.
The process filter is normally filled out with the APK package name. Simply press enter to filter out entries.
(No longer the case) Your entry must have a D flag. This flag indicates that the target device will accept incoming debugger-attach requests to the target process. If you are trying to attach to an entry that does not have this flag, the operation will fail.
After attaching, the app, you should see one or two additional nodes in the Project tree view.
If the app does not contain native code: there will be one node, representing the Dalvik VM debugger
If the app contains native libraries (*.so files in lib/ folders): there will be an additional node to represent the native code debugger
When a debugger is successfully attached, the corresponding node has a light green background.
Example: Two debugger nodes (VM, Process) currently not yet attached to the targetExample: One debugger node (Dalvik only) currently attached to the target
Views and layout
Open the VM debugger views by double-clicking the VM unit node. At this point, you will want to customize your layout: debugger views can seriously clutter the workspace area. See an example of customized layout below:
This customized layout shows: – the code hierarchies (Dalvik, Native) in the lower left corner – the VM debugger views stacked on the top right corner – the Process debugger views stacked on the lower right corner – the Console and Logs at the bottom
Layouts can be customized via the Window menu; more details can be found in a previous blog entry.
The debuggers should now be attached.
The Process debugger is never paused after attaching
The VM debugger is paused if and only if the “suspend threads” option box was ticked
Keep in mind that pausing the Process debugger (ie, suspending the native threads) will obviously freeze the higher-level Dalvik VM!
Next up, let’s review the debugger controls and controls.
Basic debugger controls via the UI
Active debugger
The most important thing to remember about debugger controls is that the UI controls affect the debugger related to the view currently focused.
Unlike most tools, JEB allows multiple debuggers and debugging sessions to take place at once. Therefore, be mindful of which debugger is being controlled when pressing menu entries or toolbar buttons: If the focus is within the DEX view or any VM debugger view, the controls are connected to the VM debugger; if the focus is within a code view connected to the Process debugger, the controls are connected to the Process debugger.
Controls
Basic debugger controls can be accessed via the Debugger menu or the toolbar area. They allow:
Attaching, detaching, terminating the process
Pausing and resuming the process and, possibly, its individual threads
The toolbar contains a subset of the most common and useful controls, that are also accessible via the Debugger menu.
Not all controls can or are implemented for all debuggers. Currently for instance, pausing individual threads of the Process debugger is not possible. When a control is not available, depending on which control it is and the severity of the failed operation, the user may be unable to activate it (eg, grayed button), receive an error in the logger, or receive a pop-up error in the client.
Breakpoints can be set/unset using the handy Control+B (or Command+B) shortcut. An icon is displayed in the left vertical bar of a code view to represent enabled/disabled breakpoints .
One enabled and one disabled breakpoints.
Debugger views
Here are some of the views rendered by the UI client when visualizing a debugger unit. (Other views were added over time.)
The Threads view displays thread identifiers, status (running, suspended, waiting, etc.) as well as the stackframes when a thread is paused. Depending on the target processor, there may be one or more stackframes, showing the location (program counter register or address) of the current thread.
The Breakpoints view displays active and inactive code breakpoints. (More on breakpoints and breakpoint types later.)
The Locals view shows the generic variables registers. They can be virtual slots of a VM, registers of a native process, complex variables inferred by the decompiler, etc.
Debugger views of a running Dalvik VMDebugger views of a paused Dalvik VM. Stackframes are visible, along with some local variables.
Every debugger has specifics that are relevant to the target being debugged. While the JEB API and front-end are trying to abstract the nitty-gritty details away, there are times when generic controls are not enough. In the next section, we discuss how users can issue such commands via the debugger console.
In the case of the Dalvik VM, the Locals view can be used to display complex objects or arrays, as is shown below:
Variables view of the top-level frame of a suspended Dalvik thread
In the case of local variables, the type of a Dalvik slot (v0, v1, etc. ) is usually inferred thanks to the Dalvik decompiler. A JEB build that does not ship with the decompiler will not be able to display most frame variables accurately.
Live variables overlays
When a thread is paused, the debuggers (native as well as Dalvik’s) provide overlay information when the mouse cursor hovers over registers, variables, class fields, or any other visual element that holds data.
Overlay on a Dalvik frame variableOverlay on a class instance field
In the case of the Dalvik debugger, overlays also work in Java decompiled views.
Advanced controls via the console
The debugger units make use of the IUnit.getCommandInterpreter method to provide clients with command interpreters to execute advanced debugger commands, that may not be readily made available by graphical clients.
In the UI client, command interpreters offered by units are accessible via the Console tab. Once the Android debuggers are attached, switch over to the Console view, and type list. This command will list all command interpreters currently attached to the console:
Two interpreters are made available, one by the VM debugger, another one by the Process debugger
An interpreter has a numeric id as well as the name of the unit that created it. Switch to an interpreter with the use <id|name> command.
Switching to the interpreter connected to the Process debugger
The special command help, available in all interpreter contexts, lists all commands made available by the interpreter currently in use.
Functions provided by the Process debugger interpreter
In this example, we can see that the Process debugger offers ways to read and write to memory, set registers, and also issue low-level GDB commands (use this option carefully).
Settings
The Android debuggers offer options to control low-level debugger parameters, such as ports and timeouts.
The .parsers.dbug_apk.* engines options
If you wish to disable native debuggers entirely, set the DoNotUseNativeDebugger to true.
API for Scripting and Automation
Debugger modules in JEB implement the set of interfaces contained in the com.pnfsoftware.jeb.core.units.code.debug public package. The principal interface in this package is IDebuggerUnit. Plugins, scripts, or third-party clients wishing to automate the usage of debuggers can us these well-defined interfaces. The official UI client uses this public API. Anything that the UI client does (and more) can be done and/or automated by third-party code.
Within the next couple of weeks, we will upload sample code on our GitHub repository demonstrating how to use the JEB Debugger API.
Sample code making use the of the JEB Debugger API
More on scripting: https://www.pnfsoftware.com/blog/crypto-monitoring-android-debuggers-api/
-nicolas
A technical implication is that apps performing health checks such as signature verification can easily detect that they have been signed by an unauthorized key. But then again, running an app on a rooted phone or an emulator is also something easily detectable. Each method has its advantages and shortcomings, be aware of them. ↩
They mostly have to do with the run-as Android utility. JEB ships with a modified version of the utility to allow debugging the native code part of non-debuggable apps. ↩
We strongly recommend our users to get familiar with the Android system tools and debugging tools ecosystem, in particular adb, am, and pm. ↩
Toggling breakpoints on and off is currently not available in decompiled views. ↩
Recently, we came across a new malware which seems to be a module of a recent Android trojan named dubbed Golem.
Golem has been found in several countries and hundreds of thousands of phones have already been infected, according to reports.
We performed detailed analysis of the malware using JEB, the operations achieved by the malware can be divided into several steps:
Step 1
When user start the phone or unlock the screen or light the screen, the malware will automatically download a file named “conf_plugin.txt” which contains configuration information like “update”, “md5”, “url”, etc.
Step 2
Then the malware will check if there is a jar file named “ic.jar” in phone memory, if not or if its md5 is different from the md5 in “conf_plugin.txt” (which means the local dex is different from the dex in remote server), malware will download the dex.
Step 3
After that, the malware will install and run the dex and execute the “onCreate” function in the “com.facebook.mini.service.RunService” class.
The complete process can be represented by the graph below:
Based on the analysis, the malware can automatically download, launch and run application without user’s authorization. The downloaded apps will run with the set of permissions already requested by the downloader:
Through this malware, the attacker can easily get your personal information, contacts or even bank accounts and passwords. Also, the attacker can remotely control your phone to open specified application and perform some bad operations to make illicit profits.
The code is obfuscated, and most strings are encrypted. The string encryption algorithm is trivial, but ever-changing across classes: bytes are incremented or decremented by constant values, either stored in a default decryptor method, or retrieved via calls to other methods. The result is something quite annoying to handle if you decide to perform a serious static analysis of the file.
Encrypted string buffers in Triada. Decryption routines can be seen in the decompiled class on the right-hand side.
Our intern Ruoxiao Wang wrote a very handy decryption script for Triada. It needs customizing (the decryption keys are not automatically retrieved) on a per-class basis, but the overall effort is a couple of seconds versus hours spending doing tedious and repetitive semi-manual work.
The script will decrypt the encrypted byte arrays and replace the decompiled Java fields supposedly holding the final strings by their actual value, as seen in the picture below.
Decrypted strings. Comments (in the left-side red box) indicate the string use was not found via xrefs. The right-side red box shows updated String fields after decryption.
The script can also be used as a tutorial on how to use the JEB Java AST API to look for and modify the AST of decompiled code. (More examples be seen on our GitHub sample script repo.)
The release of JEB 2.1.2 is being distributed to our customers today and tomorrow. We thought it would be a good time to present/recap some of the UI changes that were introduced since version 2.1.
Layouts
The RCP client comes with a default layout that has the Project view on the left-hand side, the Logger and Console at the bottom, and a large empty workspace area in the center. The layout can (and should!) be customized to fit your analysis needs.
Drag views around by their title areas. Expand a view to full-screen by double clicking on its title area. Minimize or maximize view groups using the icons located in the view trimbar. (Circled in red in the picture below.)
Customized layout with a code hierarchy on the lower-left corner.
Since you may want to have different layouts for different use cases, layouts can be duplicated and customized. You can achieve this via the Window/New Layout… menu option.
Auto-sync the Project tree selection
Enable this feature via the double-arrow icon located in the Project Explorer view. (As seen on the picture below.) When enabled, the simple selection of a unit element in the tree will automatically bring up the associated unit view. No need for Enter, no need for double-click: a simple selection is enough.
This option is especially useful when navigating large swarm of resource files, eg pictures.
Open same-type unit in same views
When enabled, a unit of view X will be opened in an already existing view representing another unit of the same type (X).
This option is extremely useful when opening many views of the same type, but only the last one is important: example, when decompiling and navigating code.
Navigating a text view with a non-sticky caret
By default, the navigation of a text view in JEB2 may be a bit confusing: due to the way very large buffers are handled by these views, it is often more resource-efficient to keep the caret on its viewport location. That means that, upon scrolling up or down, the caret will visually remain where it is.
When highlighting interactive items, and wanting to keep track of other related items across the buffer, that default behavior is not ideal: it is better to maintain the caret position within the buffer, as opposed to within the viewport.
Use Control (Control on Mac) +Shift + Up|Down to keep the caret where it is when scrolling up/down.
More to come
We will keep this entry updated as we add more how-to and gotchas regarding the RCP client user interface. If you have questions or requests, feel free to email us at support@pnfsoftware.com.
Update (9/13/2017): we open-sourced the PDF plugin. A compiled JAR binary is also available.
We have released version 0.2.9 of our PDF analyzer plugin for JEB2. This release adds support for XFA (XML Forms Architecture) fragment streams reconstruction and parsing.
In the following example, a malicious PDF file contains two XFA streams encoded with the unusual CCITTFFax encoder. Once decoded, JEB2 reassembles the decoded contents into a unit “32 0”. The XFA contains a malicious JavaScript snippet, also visible as a separate unit:
Reconstructed XFA data showing a malicious JavaScript snippet.Notifications reported also show a dangerous Open action.
The malicious PDF file examined in this entry is available on VirusTotal. SHA256: e108432dd9dad6ff57c8de6e907fd6dd25b62673bd4799fa1a47b200db5acf7c
JEB 2.1 is just around the corner! Users with a valid subscription should expect software updates today or tomorrow. This major update represents the maturation of JEB 2.0, and paves the way for JEB 2.2, which will introduce modules for x86 and ARM.
The following is a non-exhaustive list of notable changes.
Navigation bars in text views
The navigation bar is interactive (zoom in and out for finer granularity, visualize currently loaded text area, etc.) and customizable. It is connected to the metadata manager of an interactive unit. Client code (eg, in plugins or scripts) can manipulate this metadata: add, remove, query metadata and metadata groups, etc.
The navigation bar can be seen on the right-hand side
Java AST API
The newly introduced com.pnfsoftware.jeb.core.units.code.java API package allows compliant Java source units to offer direct manipulation of AST code to clients. Units produced by our native Dalvik decompiler are obviously compliant. Plugins and scripts may use this API to implement complex refactoring/code-cleanup operations, as was demonstrated with JEB 1.x in the past.
The IJavaSourceUnit interface is your entry-point to AST elements
Note that the Java AST API has changed significantly relative to JEB1’s. A few missing features will be implemented in future service releases of JEB 2.1.x (eg, tagging) but overall, it is more powerful than JEB1’s: most objects can be constructed and modified, AST elements that were not offered by the older API, such as Annotations, are now accessible, etc. – not to mention, those units are now persisted! (See our next section.)
Semi-related to the above paragraph, we are glad to announce that the decompiled Java code (and all the modifications applied to it via the Java AST API) are now persisted to the JDB2 file when saving a project to file.
The persistence mechanism has undergone significant changes and fixes, and therefore, some JDB2 generated by JEB 2.0 might not be compatible with JEB 2.1.
The IGraphicalClientContext has been augmented to support more operations, such as enumerating views and fragments, setting the focus on a specific view, setting or retrieving the active address and active items, etc.
A sample UI script showing how to start an asynchronous, interruptible task
The official RCP client implements the UI-API. We are planning to add more UI primitives in the upcoming maintenance releases.
API changes
Here is an incomplete list of API changes that took place between the latest 2.0 and the initial 2.1 releases:
Most protected members of the AbstractUnit hierarchy of skeleton classes have been privatized. The current guidelines is to use the provided getters, setters, and other accessor methods.
Side-note: As always, we encourage plugin developers to use abstract implementations instead of implementing interfaces from scratch, when abstracts are available.
Units offer a way to persist or not-persist their children units when saving a project to disk. Changes took place in IUnit.addChild() and co. methods.
The formatter (aka, the unit output manager) was also revamped: it can now yield transient and persisted documents. Write-access is also permitted, which means that plugins and scripts can add documents (such as texts, tables, or trees), and request that they’re persisted in the JDB2 database. We uploaded sample scripts here:
The client notifications are now called ClientNotification, to avoid potential confusion with another type of notification used within JEB (the unit notifications).
Next up
We are planning a few maintenance updates before the release of JEB 2.2. The currently planned release date for JEB 2.2 is early February 2016.
We will keep you posted on this very blog. Stay tuned, and a happy holiday season to all.
PS: as an early Christmas gift, we have uploaded a new third-party plugin on our public repository. Check out the Antivirus Quarantines File Extractor plugin. We currently support Kaspersky KLQ quarantine files only, but are planning to add more soon. If you’d like to contribute, please send us an email.
In this blog post, we show how JEB2 can be used as a building block of a file analysis system. We will show how to use the Core API to create a headless client. That client will scan PDF files using the JEB2 PDF Analysis Module. Basics of the IUnit and co. interfaces is also demonstrated.
Sample execution output produced by the PDF Scanner
As this slide deck shows, the back-end and front-end components of JEB2 are separated. The official RCP desktop client uses the JEB2 Core API; other front-ends, like the PDF scanner, can be built using that same API.
JEB2 HL Architecture Diagram
Creating an Eclipse project
Let’s get started by creating a new code project. We will show how to do this in Eclipse.
0- Check your license of JEB2. Make sure to use a license that supports third-party client creation and the loading of third-party plugin. If you haven’t done so, download and drop the PDF module in your coreplugins/ sub-directory.
2- Create a new Java project. The Java source folder should be rooted in the src/ directory.
3- Add the JEB2 back-end as a JAR dependency. The back-end software is contained in the file bin/cl/jeb.jar located within your installation folder. You may also want to link that JAR to the API documentation, contained in the doc/apidoc.jar file, or online at https://www.pnfsoftware.com/jeb/apidoc
Your Package Explorer view should now look like:
Package explorer view after setting up dependencies
5- Set up the execution options. The required Java properties for execution (jeb.engcfg and jeb.lickey) can be set in the Run Configurations panel (accessible via the Run menu). Example:
Example of a Run configuration
6- Open the com.pnf.pdfscan.PDFScanner source file. You are ready to execute main().
We are retrieving the top-most unit only in this example
Analyze the unit (see assessPdf())
Close the project
[Note: A detailed explanation of the above concepts (core, engines, project, artifacts, units, etc.) is outside the scope of this tutorial. Refer to our Developer Portal for more information.]
Snippet of scanFiles()
The assessPdf() method evaluates PDF units. The evaluation performed by this sample scanner is trivial: we collect the notifications created by the PDF plugin during the analysis of the file, and see if they meet basic criteria.
About the Unit Notifications:
Any JEB2 plugin can attach notifications to its units. The PDF plugin does so. Notifications are meant to pin-point noteworthy areas of a unit or artifiact.
A notification has a “dangerosity level” ranging from 0 to 100. It also has a description, an optional address to point to which area of the unit the notification is associated with, etc.
The API offers standard notification types, ranging from “Interesting area” to “Definitely Malicious”.
Standard notification levels offered in the NotificationType enum
A PDF unit can contain several types of notifications. Example include: corrupt areas in stream; multiple encoding of stream; JavaScript; password-protected stream; invalid/illegal entries in stream; etc.
Our simple scanner reports a file as suspicious if it contains at least 2 notifications that have a level >= 70 (POTENTIALLY_HARMFUL). These thresholds can be tweaked in the source code.
The assessPdf() routine
The screenshot below is a sample output produced by the PDF scanner:
Conclusion
The intent of this entry is to shed some light on the process of writing third-party clients for JEB2, as well as what and how to use notifications reported by Units. We encourage you to visit our Developer Portal to find additional documentations as well as the reference Javadoc of the API.
The latest release of JEB2, version 2.0.14, introduces a feature familiar to JEB1 users: client scripts written in Python.
Both Standard and Business licenses permit running scripts. They can be written using the Python 2.5 or 2.7 syntax and features, and are executed by Jython. (A Jython stand-alone package is required to run scripts. We recommend version 2.5. Download it and drop it in your JEB2 scripts/ sub-directory.)
Feature-wise, scripts use the standard JEB2 core APIs. They are also using the client API, available in the com.pnfsoftware.jeb.client.api package. As usual, refer to our Developer Portal and Javadoc website for API reference and usage.
A client script implements the IScript interface. Upon execution, the script run() entry-point method is provided an IClientContext or derived object, such as an IGraphicalClientContext for UI clients. (The official RCP desktop client falls in the latter category.)
Here is the simplest of all scripts:
from com.pnfsoftware.jeb.client.api import IScript
class JEB2SampleScript(IScript):
def run(self, ctx):
print('Hello, JEB2')
Within the official desktop client, scripts can be executed via the File, Scripts menu item.
Finally, remember that scripts are meant to execute small, light-weight actions. Heavy lifting operations (such as parsing or background event-driven tasks) should be implemented by back-end plugins in Java.
