Category Archives: Low Level

Android NDK Libraries Signatures

In this blog post, we present a new batch of native signatures released with JEB3 to identify Android Native Development Kit (NDK) libraries.

First, let’s briefly give some context. The Android NDK is a set of tools allowing developers to embed compiled C/C++ code into their Android applications. Thus, developers can integrate existing native code libraries, develop performance-sensitive code in C/C++ or obfuscate algorithms with native code protectors.

In practice, native code within Android applications comes in the form of ELF shared libraries (“.so”); the native methods can then be called from Java using Java Native Interface (JNI), which we described in a previous blog post.

NDK Pre-Built Libraries

Android NDK provides some pre-built libraries that can be linked against. For example, there are several C++ Standard Template Library (STL) 1 , or the Zlib decompression library.

As an example, let’s compile a “hello world” Android NDK C++ library with NDK r17. By default, the C++ implementation will be gnustl — the default choice before NDK r18.

Here is the C++ code:

When compiled with Android Studio’s default settings, libraries are linked dynamically, and libgnustl_shared.so is directly included in the application — because it is not a system library –, for each supported Application Binary Interface (ABI).

Files hierarchy of the Android application containing our “hello world” native library

If we open the ARM library we can pretty easily understand the — already convoluted — logic of our “hello world” routine, thanks to the names of gnustl external API calls:

Control-flow graph of ARM “hello world” with gnustl dynamically linked. Note that JEB displays mangled names when API calls correspond to external routines.

Now, Android NDK also provides static versions for most of the pre-built libraries. A developer — especially a malware developer wishing to hinder analysis — might prefer to use those.

When compiled in static mode, gnustl library is now ‘included’ in our native library, and here is our “hello world” routine:

Control-flow graph of ARM “hello world” with gnustl statically linked. Subroutines bear no specific names.

In this case, the analysis will be slowed down by the numerous routine calls with no specific names; each of this subroutine will need to be looked at to understand the whole purpose.

This brings us to a common reverse-engineering problem: is there a way to automatically identify and rename static library code, such that the analyst can focus on the application code?

JEB3 NDK Signatures

That’s when JEB native signatures come to the rescue! Indeed JEB3 now provides signatures for the following Android NDK  static libraries:

  • gnustl
  • libc++
  • STLport
  • libc
  • libmath
  • zlib

We provide signatures for ARM/ARM64 ABIs (including all variants like arm-v7a, arm-v7a-hard, thumb or ARM mode, etc) of these libraries, from NDK r10 to NDK r18.

These signatures are built in a similar fashion to our x86/x64 Visual Studio native signatures, and are intended to be “false-positive free”, which means a match should be blindly trustable. Note that JEB users can create their own signatures directly from the UI.

So, within JEB, if we open our statically-linked library with the signatures loaded, gnustl library routines are identified and renamed:

Control-flow graph of ARM “hello world” with gnustl statically linked and NDK signatures loaded. Subroutines have been renamed.

Note: the attentive reader might have noticed some “unk_lib_subX” routines in the previous image. Those names correspond to cases where several library routines match the routine. The user can then see the conflicting names in the target routine and use the most suitable one.

Due to the continuous evolution of compilers and libraries, it is not an easy task to provide up-to-date and useful signatures, but we hope this first NDK release will help our users. Nevertheless, more libraries should certainly be signed in the future, and we encourage users to comment on that  (email, Twitter, Slack).

  1.  NDK C++ support is a turbulent story, to say the least. Historically, different implementations of C++ have been provided with the NDK (gnustl, STLport, libc++,…), each of them coming with a different set of features (exceptions handling, RTTI…). Since the very recent r18 version (released in september 2018) Android developers must now use only libc++.

JEB3 Auto-Signing Mode

In this video we introduce a novel JEB 3.0 feature: auto-signing mode for native code.

In a nutshell, when this mode is activated all modifications made by users to native code in JEB (renaming a routine, adding a comment, etc) are “signed”.

The newly created signatures can then be loaded against another executable, and all the information of the original analysis will be imported if the same code is recognized. Therefore, the user only needs to analyze each routine once.

Without further ado, here is the video, which begins by introducing native signatures before showcasing auto-signing:

As usual, feel free to reach out to us (email, Twitter, Slack) if you have questions or suggestions.

Reverse Engineering WebAssembly

Note: Download the demo of “JEB Decompiler for WebAssembly” here.

We published a paper deep-diving into WebAssembly from a reverse engineer point of view (wasm format, bytecode, execution environment, implementation details, etc.).

The paper annex details how JEB can be used to analyze and decompiler WebAssembly modules.

Code and decompilation view of a WebAssembly module

Thank you – Nicolas.

Having Fun with Obfuscated Mach-O Files

Last week was the release of JEB 2.3.7 with a brand new parser for Mach-O, the executable file format of Apple’s macOS and iOS operating systems. This file format, like its cousins PE and ELF, contains a lot of technical peculiarities and implementing a reliable parser is not a trivial task.

During the journey leading to this first Mach-O release, we encountered some interesting executables. This short blog post is about one of them, which uses some Mach-O features to make reverse-engineering harder.

Recon

The executable in question belongs to a well-known adware family dubbed InstallCore, which is usually bundled with others applications to display ads to the users.

The sample we will be using in this post is the following:

57e4ce2f2f1262f442effc118993058f541cf3fd: Mach-O 64-bit x86_64 executable

Let’s first take a look at the Mach-O sections:

Figure 1 – Mach-O sections

Interestingly, there are some sections related to the Objective-C language (“__objc_…”). Roughly summarized, Objective-C was the main programming language for OS X and iOS applications prior the introduction of Swift. It adds some object-oriented features around C, and it can be difficult to analyze at first, in particular because of its way to call methods by “sending messages”.

Nevertheless, the good news is that Objective-C binaries usually come with a lot of meta-data describing methods and classes, which are used by Objective-C runtime to implement the message passing. These metadata are stored in the “__objc_…” sections previously mentioned, and the JEB Mach-O parser process them to find and properly name Objective-C methods.

After the initial analysis, JEB leaves us at the entry point of the program (the yellow line below):

Figure 2 – Entry point

Wait a minute… there is no routine here and it is not even correct x86-64 machine code!

Most of the detected routines do not look good either; first, there are a few objective-C methods with random looking names like this one:

Figure 3 – Objective-C method

Again the code makes very little sense…

Then comes around 50 native routines, whose code can also clearly not be executed “as is”, for example:

Figure 4 – Native routine

Moreover, there are no cross-references on any of these routines! Why would JEB disassembler engine – which follows a recursive algorithm combined with heuristics – even think there are routines here?!

Time for a Deep Dive

Code Versus Data

First, let’s deal with the numerous unreferenced routines containing no correct machine code. After some digging, we found that they are declared in the LC_FUNCTION_STARTS Mach-O command – “command” being Mach-O word for an entry in the file header.

This command provides a table containing function entry-points in the executable. It allows for example debuggers to know function boundaries without symbols. At first, this may seem like a blessing for program analysis tools, because distinguishing code from data in a stripped executable is usually a hard problem, to say the least. And hence JEB, like other analysis tools, uses this command to enrich its analysis.

But this gift from Mach-O comes with a drawback: nothing prevents miscreants to declare function entry points where there are none, and analysis tools will end up analyzing random data as code.

In this binary, all routines declared in LC_FUNCTION_STARTS command are actually not executable. Knowing that, we can simply remove the command from the Mach-O header (i.e. nullified the entry), and ask JEB to re-analyze the file, to ease the reading of the disassembly. We end up with a much shorter routine list:

Figure 5 – Routine list

The remaining routines are mostly Objective-C methods declared in the metadata. Once again, nothing prevents developers to forge these metadata to declare method entry points in data. For now, let’s keep those methods here and focus on a more pressing question…

Where Is the Entry Point?

The entry point value used by JEB comes from the LC_UNIXTHREAD command contained in the Mach-O header, which specifies a CPU state to load at startup. How could this program be even executable if the declared entry point is not correct machine code (see Figure 2)?

Surely, there has to be another entry point, which is executed first. There is one indeed, and it has to do with the way the Objective-C runtime initializes the classes. An Objective-C class can implement a method named “+load” — the + means this is a class method, rather than an instance method –, which will be called during the executable initialization, that is before the program main() function will be executed.

If we look back at Figure 5, we see that among the random looking method names there is one class with this famous +load method, and here is the beginning of its code:

Figure 6 – +load method

Finally, some decent looking machine code! We just found the real entry point of the binary, and now the adventure can really begin…

That’s it for today, stay tuned for more technical sweetness on JEB blog!

Firmware Exploitation with JEB: Part 1

In this series of blog posts I will show how JEB’s MIPS decompiler 1 can help you find and exploit software vulnerabilities in embedded devices. To do so, we will use Praetorian’s Damn Vulnerable Router Firmware (DVRF) written by b1ack0wl.

DVRF is a custom firmware made to run on a Linksys E1550 router containing a bunch of memory corruption vulnerabilities. The goal of the DVRF is to serve as a playground to learn exploitation on the MIPS architecture. As far as I know, there are no write-ups of the challenges on the Internet.

For the readers interested in testing the challenges by themselves, I suggest to follow the DVRF tutorial, and getting a complete MIPSEL Debian QEMU image as it allows the usual exploit development workflow on Linux, without any limits on the available tools.

Recon

First things first, I extracted the binaries from the firmware with binwalk. Let’s then do some recognition on the first challenge file:


file stack_bof_01

stack_bof_01: ELF 32-bit LSB executable, MIPS, MIPS32 version 1 (SYSV), dynamically linked, interpreter /lib/ld-uClibc.so.0, not stripped

After loading it in JEB we can see several interesting functions:

Among some classic libc interesting routines (system, strcpy…), I noticed the aptly named “dat_shell” function.

As we see here, this function congratulates you for solving the challenge then spawns a shell with the call to system. We now know that we want to redirect the execution flow to the dat_shell function.

Next, we see that the binary calls “strcpy” and that may just be a textbook case of buffer overflow. So let’s check the main function, where strcpy is called.

First, it checks if we provided a command-line argument, and welcomes us. Second, it copies user input in a local variable and prints what we entered. Finally, it tells us to “Try Again” and then returns. Fortunately, strcpy does not check its input size, which results in a stack buffer overflow as the challenge’s name indicates.

Building the Exploit…

As you would do in a similar situation on a x86 binary, let’s first run the binary inside a debugger with a large parameter to verify the overflow.

To do this, I started a gdbserver on my QEMU VM and attached to it with JEB’s debugger interface (see the debugging manual for more info). In MIPS ISA, the return address from a routine call is stored in a specific register called $ra, which is also filled from the stack as you normally see on x86. It then jumps to that saved return address.

In our binary, we confirm that the return address is user-controlled by providing a large parameter — a series of 0x4F bytes –, and displaying the registers state after the strcpy call:

Let’s check the stackframe that I reconstructed to calculated the appropriate padding. You can access that view with the Ctrl+Alt+k shortcut in the function of your choice. I changed the type of the variable buf to a char array of all the available size between the start of the variable and the next one. This gave me 200 bytes.

The variables var04 and var08 are in fact the saved return address and the saved frame pointer of the main function. The result is that this offset is at 204 bytes because we fill the buffer with 200 bytes and overwrite the save frame pointer with four more. Let’s try the following exploit:


#!/usr/bin/python

padding = "O"* 204

dat_shell_addr = "\x50\x09\x40" # Partial overwrite with little-endian arch

payload = padding + dat_shell_addr

with open("input", "wb") as f:

f.write(payload)

…Is Not So Easy

Surprisingly, our dummy exploit makes the program segfaults at the address 0x400970 — within the dat_shell routine. Let’s take a look at this address in JEB native view:

We can see here a memory access to the address computed by adding the offset 0x801C to the global pointer register $gp. The problem here is that $gp was initially set at the beginning of the routine from the $t9 register (see 0x4000958 above).

So, where does the value in $t9 comes from? The answer lies in the way routines are usually called on MIPS (the calling convention): the $t9 register is first set to the address of the target routine, and is then branched to, for example with a jalr $t9 instruction (see MIPS ISA p.50). The global pointer $gp is then initialized with $t9 and serves to compute various offsets, in particular to other functions that will be called, hence the fact that it absolutely needs to be correct.

In other words, if the value of $t9 is not the address of dat_shell when executing this routine, there is a good chance an invalid memory access will happen during the routine execution. To build a successful exploit, we need to load an arbitrary value from the stack into $t9 and then branch to it, as it was a real function call.

To do so, we need a “gadget”, that is a series of instructions implementing the previously described behavior that we can jump to. In the search of this gadget, let’s first check what dynamic libraries are loaded with the “libs” debugger command.

Luckily, we have three libraries loaded at fixed memory addresses: libc.so.0, libgcc_s.so.0 and ld-uClibc.so.0.

Interlude: ROP Gadget Finder Plugin for JEB

Using gadgets is a common need to build Return-Oriented-Programming (ROP) exploits, so I decided to develop a gadget finder plugin 2. Also, rather than searching gadgets from native instructions I decided to use JEB Intermediate Representation (IR), such that I could find gadgets on all architectures handled by JEB transparently.

The end result is that when loading the three previously mentioned libraries in JEB, the plugin creates a view with all the gadgets:

The output is free of duplicated gadgets and alphabetically ordered to ease the process of finding interesting gadgets.

So, how does it work exactly? Using JEB’s API, the plugin converts the native code to the IR used in the first stage of our decompilation pipeline. At this stage, all the side-effects of the native instructions are exposed and no optimizations have been made yet.

To find gadgets — a series of instructions ending with a branch –, we simply search for the assignments on the program counter register and iterate backwards until another assignment on that register is made. The last step is to filter out relative jumps — which can not really be controlled during an exploit — and we got ourselves a good list of ROP gadgets.

Again, this method works on all architectures as it is using only the IR. As an example, here is the same code running on an ARMv7 binary:

The published code can be found here:

Interlude End

Back to our challenge, using our plugin on the libc library, I found the following gadget at offset 0x6b20:

It copies a value from the top of the stack into the $t9 register, and branches to the $t9 register… perfect!

The plan is therefore to use the vulnerable strcpy to execute this gadget first, such that dat_shell address will be called as a normal routine call would do. After deactivating Address Space Layout Randomization (ASLR) on our test machine, we can use the previously found libc base address for the exploit. The final exploit looks like this:

#!/usr/bin/python

import struct

# LW $t9, 0($sp); JALR $t9;

gadget_offset = 0x6b20

libc_base = 0x77eea000

gadget_addr = struct.pack("<I", libc_base + gadget_offset)

payload = ""

payload += "A"*204 # padding

payload += gadget_addr

payload += "\x50\x09\x40"

with open("input", "wb") as f:
    f.write(payload)

And here we go!


Special thanks to @b1ack0wl for the challenges and help and to @yrp604 for the review. This post was also co-authored with our own @joancalvet.

Stay tuned for more MIPS exploitation blog posts!

  1. In this blog, we use JEB 2.3.3, which will roll out the week of Aug 21-25.
  2. The gadget finder plugin will be published on GitHub later this week.

Android O and DEX 38: Dalvik Opcodes for Dynamic Invocation

Android O – API level 26 – upgrades the DEX format in order to provide support for dynamic invocation via two new Dalvik opcodes: invoke-polymorphic and invoke-custom.1

In this post, we will:

  1. Do a brief recap of how dynamic invocation is achieved in Java
  2. Present the changes made to the DEX file format in Android O
  3. Explain what the new dynamic invocation instructions can do and how they work
  4. Show code samples to generate DEX version 38 files
  5. Have a quick look at dynamic invocations in the context of app obfuscation

JEB 2.3-beta handles DEX v38. That includes API support for programmatic access to the new pools and invoke-polymorphic, invoke-custom instructions, via the IDexUnit entry point interface. We will describe in details the Android-related additions shipping in JEB 2.3.1 in a separate blog post.

Java’s invokedynamic

As a reminder, and quoting from the official Oracle doc (emphasis mine):

“[invokedynamic] improves implementations of compilers and runtime systems for dynamic languages on the JVM. It does this by allowing the language implementer to define custom linkage behavior. This contrasts with other JVM instructions such as invokevirtual, in which linkage behavior specific to Java classes and interfaces is hard-wired by the JVM.”

If that sounded like gibberish to you, you may want to get up to speed on dynamic invocation in Java – in particular, read the javadoc of MethodHandle and CallSite. We will (re)explain a bit in this post, but it is definitely not the main purpose of it. On top of the official Oracle doc as well as the original JSR, I recommend this article from the author of ByteBuddy.

Back in the Dalvik world

Up until DEX v35/v37, the way to invoke code in Dalvik was through one of the 5 invocation instructions:

  • invoke-virtual for virtual methods (Java’s invokevirtual)
  • invoke-static for static methods (Java’s invokestatic)
  • invoke-interface for methods called on interface types (Java’s invokeinterface)
  • invoke-super for super-class methods  (Java’s invokespecial)
  • invoke-direct for constructors  (Java’s invokespecial, again)
invoke-virtual Ljava/lang/String;->length()I, v

Each one of these takes a method item, which specifies a type (class or interface) as well as a method reference – ie, the “hard-wiring” part mentioned in the above quote. Java is statically typed, and the bytecode reflects that.2 That is, until invokedynamic was introduced with Java 7.

So, what is the Dalvik equivalent of Java’s invokedynamic?

Actually, there are 4 (2×2):

  • invoke-polymorphic (as well as invoke-polymorphic/range), which does “half” of what invokedynamic can do;
  • invoke-custom (as well as invoke-custom/range), which does the other, more powerful “half”.

invoke-custom requires additional pool elements, namely method handle items and call site items. Let’s walk over the DEX format additions to support those additional pools.

DEX version 38 changes

Most DEX files have version number 35. Android Nougat introduced version 37, which did not bring any structural changes (the new version code indicated support for Java 8’s default methods). If you were wondering why Dalvik did not have the equivalent of JVM’s invokedynamic, well, brace yourself: DEX version 38 is coming.

The header magic is now “DEX\x0A038\x00”. The updated file layout shows two additional pools: call_site_ids and method_handles.

However, the header size is still 70h bytes, and therefore, contains neither the offset to, nor the count of items, for those pools. Where are they?

Let’s turn to the DEX map. Sure enough, new types were introduced: TYPE_CALL_SITE_ID_ITEM (7) and TYPE_METHOD_HANDLE_ITEM (8). We can parse the map, find those two entries, and start parsing the pools.

  • call site item is essentially an array of DEX Values. The array contains at least 3 entries:
    • a method handle index (as in: a Java MethodHandle) to a bootstrap linker method;
    • a dynamic method name, the one to be dynamically resolved
    • a dynamic method type (as in: a DEX prototype);
    • additional arguments. More on this later when we discuss invoke-custom.
  • A method handle item contains:
    • a type, indicating whether the method handle is a method invoker or a field accessor;
    • and a method id or field id, depending on the aforementioned type.

As far as other changes go, obviously, the DEX Value entries can be of two additional types: VALUE_METHOD_TYPE (0x15) that references the prototypes pool, and VALUE_METHOD_HANDLE (0x16) that references the method handles pool. (Note that there is no VALUE_CALL_SITE.)

Now, let’s see how those pools are used by the new invoke instructions, and how those instructions work.

Dalvik’s invoke-polymorphic

Below are the specifications of invoke-polymorphic taken from Android Source:

invoke-polymorphic MH.invoke, prototype, {mh, [args]}

invoke-polymorphic is used to invoke a method handle using one of two @PolymorphicSignature 3 methods of the MethodHandle object: invoke() or invokeExact(). It takes at least 3 arguments:

  • A method reference to either MethodHandle.invoke or MethodHandle.invokeExact (MH.invoke)
  • The prototype of the method to be executed
  • A method handle (mh) of the target

See the example below: MethodHandle.invoke() is used on the method handle v0; the target method has the prototype (I)Object. Therefore, v1 is of type int; the return value will be of type Object.

invoke-polymorphic used in a Dalvik version 38 file

The return type as well as parameter types are specified in the prototype item, instead of a static method item — hello, polymorphism. Of course, the target method handle must reference a method of such type, either exactly, if MethodHandle.invokeExact() is used, or have compatibility with the type (via  conversion operations), using MethodHandle.invoke().

Wait, That looks like a normal invocation!

You would be semi-right to think so. After all, we are executing invoke() or invokeExact() the old fashion way here… so, why need an additional opcode? First, remember that those methods have polymorphic signatures; their prototype is determined at compile-time. Therefore, there are two options (using the example above):

  • either the bytecode references an invoke with an (I)Object prototype: in this case, we could simply call invoke-virtual on an artificial invoke(I)Object. This is the case with the Java bytecode: invokevirtual is used;
  • or the bytecode references the generic invoke([Object)Object: in this case, the invocation would require an additional prototype argument. Hence the requirement for a new invoke opcode. This is the case with the Dalvik bytecode: invoke-polymorphic was created. It  takes not one, but two pool indexes.4

Can’t I do the same with reflection?

You may be wondering what the point of these convoluted constructions is… After all, couldn’t we do the same with reflection? The answer is mostly yes, however, remember that invokedynamic has a different goal than introspection: the goal of invokedynamic is to provide an efficient low-level primitive meant to execute dynamic call sites, and therefore, enable the implementation of dynamic languages on top of the JVM.

Practically, and as far as Java goes,  they enable the implementation of Java 8 lambdas without the use of pre-compiled anonymous inner classes.

Also practically, true polymorphism means we are no longer dealing with the auto-boxing casts associated with Reflection API calls. MethodHandle.invoke() is a very particular method – as said above, it is has a polymorphic  signature, inferred at compile-time based from the types of arguments and return value provided in the call. Nothing like actual code to show what we mean here.

Sample Code

The example below has a triple-purpose:

  • Set up your environment to generate DEX v38 files;
  • Generate DEX files containing invoke-polymorphic instructions;
  • Compare MethodHandle.invoke() vs reflection.

First, download Android Studio 2.3. It turns out that at the time of writing, compiling DEX v38 with a non-Jack toolchain (using AS 3.0) produces invalid DEX files.  Make sure to use the Android Gradle plugin 2.3.3 or above. Finally, make sure to use SDK level 26 in your module-level Gradle script, and add the “o-b2” option to allow the generation of the new invoke instructions (thank you, Tsuyoshi Murakami):

android.defaultConfig.jackOptions {
    enabled = true
    additionalParameters 'jack.android.min-api-level': 'o-b2'
}

Now, let’s compile this 30-line activity class:

public class MainActivity extends Activity {

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        String text = "";
        try {
            text = String.format("dynamic=%s reflect=%s", execDynamically(), execViaReflection());
        }
        catch(Throwable e) {
            text = e.toString();
        }
        Log.d("DexVer38", text);
        TextView tv = new TextView(this);
        tv.setText(text);
        setContentView(tv);
    }

    static Object execDynamically() throws Throwable {
        return MethodHandles.lookup().findStatic(MainActivity.class,"foo",
                MethodType.methodType(int.class, String.class, int.class)).invoke("hello", 2);
    }

    static Object execViaReflection() throws Throwable {
        return MainActivity.class.getDeclaredMethod("foo", String.class, int.class).invoke(null, "hello", 2);
    }

    static int foo(String s, int i) {
        return s.charAt(i);
    }
}

Both execDynamically() and execViaReflection() methods eventually invoke foo(“hello”, 2) and return its result:

$ adb logcat -s "DexVer38"
[...] D DexVer38: dynamic=108 reflect=108

However, while the polymorphic MethodHandle.invoke() of execDynamically truly takes a String as first argument, an int as second argument, and returns an int; we know it is not the case with the non-polymorphic invocation used by the Method.invoke(): casts are in place to box/unbox the int primitives to/from an Integer object.

Open the resulting DEX file in JEB:

MethodHandle and invoke-polymorphic vs Reflection and invoke-cirtual

Carefully look at the disassembly of both methods:

  • invoke-polymorphic’s MethodHandle.invoke handles any prototype, as long as the referenced method matches it
  • reflection’s Method.invoke is called using a traditional invoke, and therefore, its arguments must be a an array of Object, and its return value an Object — hence, the casts.

I hope this sheds some light on invoke-polymorphic, in terms of MethodHandle uses and resulting differences in the bytecode.

Dalvik’s invoke-custom

Below are the specifications of invoke-custom taken from Android Source:

invoke-custom callsite, {arguments}

Dalvik’s invoke-custom ~= Java’s invokedynamic

Before we explain the mechanics behind invoke-custom, remember that unlike the legacy invoke-xxx instructions, it does not take a reference to the LType;->method() that will be executed. Both will be determined at run-time.

The invoke-custom instruction first resolves and then invokes a call site:

  • Initially, an invoke-custom instruction is an unlinked state: its call site has yet to be created. It is the resolution stage:
    • The runtime checks if a CallSite object exists for the provided callsite index
    • If not, a new CallSite object is created using the data provided by the call site item at the corresponding pool index, via a bootstrap linker method
    • The invoke-custom is now in a linked state
  • When the invoke-custom is in a linked site, the CallSite object’s MethodHandle is invoked.

The following diagram summarizes the bootstrap process of linking an unlinked invoke-custom:

Delaying the resolution and creation of the callsite until runtime allows the VM to take the decision of which type and which method should the execution flow be dispatched to.

Bear in mind that in standard Java, crafting explicit code using dynamic invocation is currently not possible. That limitation can be circumvented with custom toolchains (such as Android’s Jack, as we’ll see below). However, a prime candidate for implicit use of dynamic invocations are of course lambdas. Lambda functions have been supported since Android Nougat and are currently compiled using virtual invocations. It is safe to say that we should see lambdas using invoke-custom in the near future, maybe as early as the release candidate of Android O.

Sample Code

Currently, crafting high-level Java code that produces invoke-custom is convoluted and artificial — unfortunately, lambdas are still desugared into statically invoked methods of synthetic inner classes.

Two possible options are:

  1. Crafting Dalvik code manually , or via a custom tool, or via a bytecode manipulation library. It is outside the scope of this post;
  2. Use the soon-to-be-deprecated Jack toolchain and custom Jack annotations to generate bootstrap methods.

Using the second approach, we can generate code that contains correct call site item pools. However, at the moment, those DEX files do not pass the Verifier.

That being said, the generated bytecode looks fine. Have a look at the sample below:

public class MainActivity extends Activity {

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        String text = "";
        try {
            text = "" + execCustom();
        }
        catch(Throwable e) {
            text = formatThrowable(e);
        }
        Log.d("DexVer38", text);
        TextView tv = new TextView(this);
        tv.setText(text);
        setContentView(tv);
    }

    public static String formatThrowable(Throwable t) {
        Writer writer = new StringWriter();
        PrintWriter out = new PrintWriter(writer);
        t.printStackTrace(out);
        return writer.toString();
    }

    public static Character execCustom() throws Throwable {
        return foo("hello", Integer.valueOf(2));
    }

    @CalledByInvokeCustom(
            invokeMethodHandle = @LinkerMethodHandle(kind = MethodHandleKind.INVOKE_STATIC,
                    enclosingType = MainActivity.class,
                    name = "linkerMethod",
                    argumentTypes = {MethodHandles.Lookup.class, String.class, MethodType.class}),
            name = "foo",
            returnType = Character.class,
            argumentTypes = {String.class, Integer.class})
    static Character foo(String s, Integer i) {
        return s.charAt(i);
    }

    private static CallSite linkerMethod(MethodHandles.Lookup caller, String name, MethodType methodType)
            throws NoSuchMethodException, IllegalAccessException {
        return new ConstantCallSite(caller.findStatic(caller.lookupClass(), name, methodType));
    }
}

Using the CalledByInvokeCustom annotation, we can specify that foo() must be dynamically invoked. The code is a bit artificial, and the linked method trivial, but see how the seemingly static call to foo() in execCustom() was compiled to the following bytecode:

invoke-custom used in a Dalvik version 38 file

Note that the JEB syntax for invoke-custom call sites is temporary and subject to change. At the moment, the pool’s call site is displayed within double curly brackets:

{{ MethodReference / DynamicMethodName / DynamicPrototype / [additional arguments, …] }}

JEB will decompile those constructs to an invocation of the bootstrap linker method, followed by a call to invoke() on the returned CallSite’s method handle. In a real environment, the bootstrap method would be executed just once. Indeed, high-level Java code cannot reflect all forms and uses of those low-level constructs.

Keep in mind that invoke-custom‘s purpose is much broader than this dummy example. As said in the previous section, we should expect it to initially be used when generating Java 8’s lambdas. They may not be extremely popular – not yet – in traditional Java programming circles, but Google’s big push on Kotlin for Android O, including:

  • Kotlin integration in Android Studio, facilitating adoption;
  • Kotlin full compatibility with Java, allowing mixed code base during migration;
  • Kotlin’s affinities with dynamically-typed languages;

may be indicators that invoke-custom (and invoke-polymorphic) will be used to power new language features for Android app development in the near future.

Dynamic invocation used for obfuscation

Finally, let’s conclude this post with a note on obfuscation, and generally, unintended, unplanned, or at least non-primary use cases, for MethodHandle.

Just like reflection has been heavily used by all5 Dalvik protectors and obfuscators to hide API calls and make static code flow analysis difficult, we should expect MethodHandle and CallSite to be used in similar ways.

MethodHandle objects have more restrictions than pure reflection though, eg, in terms of the scope of what can be retrieved. Obviously, they cannot be used to retrieve types dynamically — which means there is no equivalent to Reflection’s Class.forName(“…”). However, they can be used to retrieve handles on methods, constructors, and fields, and therefore could be mixed in with standard reflection-based obfuscation techniques.

As for invoke-custom: parsing and analysis of the call site items pool will be required to retrieve references to boostrap linker methods, and determine their effect on code.

So, exciting times ahead! We should all be excited to see those new dynamic invoke opcodes used by apps in the future, as well as the potential they bear in terms of new languages (or more realistically, new language features) that they can provide for Android app development.

  1. And their /range counterparts. Essentially, this update is the Android implementation of JSR292
  2. That point is debated; however, the invoke call sites exhibit the static nature of type binding in the bytecode.
  3. The PolymorphicSignature annotation is defined within MethodHandle and visible only to types declared  in the java.lang.invoke package
  4. Both ways are conceptually valid. In the first case, we are assuming that an infinity of MethodHandle.invoke signatures exist. In the second case, we consider that MethodHandle.invoke true prototype is ([Object)Object; that means we must provide the actual prototype separately, via a new opcode.
  5. The Dalvik verifier is quite strict and limits the classes of obfuscation that can be applied onto bytecode.