IDA has become the standard for modern disassemblers used in the reverse engineering community. Hex-Rays is a popular plugin to IDA which further simplifies the binary analysis by decompiling native code into a C-like pseudocode. However, Hex-Rays’s strength goes beyond its decompilation quality. It is its overall seamless integration with the interactive disassembler that makes it an invaluable reversing tool.

This article demonstrates how to use the extensive, but often not self-evident, functionality provided by IDA SDK in order to put together a plugin with Hex-Rays-like capabilities. In fact, many of the possibilities discussed here are probably not used outside of Hex-Rays itself.

The scope of this article is limited to the GUI-related features within IDA SDK, not the decompilation itself. The article expects the reader has a good knowledge of IDA plugin writing. It focuses on the advanced features it aims to introduce. The basic functionality comprehensible from IDA SDK’s headers or examples may be omitted.

Both IDA and IDA SDK used in this article are of version 7.5. The article discusses the fundamental principles to accomplish the given tasks. The complete working example can be found in an associated GitHub repository . The repository may be improved or updated to work with further IDA SDK versions. Examples in this article will not!

What can Hex-Rays do?

Hex-Rays is a native IDA plugin written in C++ by the guys behind IDA itself. As such, it perfectly uses IDA SDK to gain the following capabilities:

• Syntax highlighted C-like pseudocode in an IDA-native subview.

Implementing a decompiler itself is outside the scope of this article. Therefore, we are going to mock it. All we need is an interface capable of decompiling an address into a function:

class Decompiler
public:
   static Function* decompile(ea_t ea);
};Code language: C++ (cpp)

The demo plugin in the repository is able to decompile only main() and ack() functions from the provided ack.x86.gcc.O0.g.elf binary. Their decompiled code is hardcoded in the decompiler module. This is sufficient for our demonstration purposes.

2. YX coordinates

Before we look into function representation, we need to introduce two important building blocks. The first one being YX coordinates:

struct YX
   /// lines
   std::size_t y = YX::starting_y;
   /// columns
   std::size_t x = YX::starting_x;
   inline static const std::size_t starting_y = 1;
   inline static const std::size_t starting_x = 0;
   YX(std::size_t y = starting_y, std::size_t x = starting_x);
};Code language: C++ (cpp)

These objects will represent a position into an IDA custom viewer. Because we are going to display a source code, we index lines (Y) from 1 and columns (X) from 0.

3. Tokens

Another piece essential to build functions is Token:

struct Token
   enum class Kind
      NEW_LINE = 0,
      WHITE_SPACE,
      PUNCTUATION,
      OPERATOR,
      ID_VAR,
      ID_MEM,
      ID_LAB,
      ID_FNC,
      ID_ARG,
      KEYWORD,
      TYPE,
      PREPROCESSOR,
      INCLUDE,
      LITERAL_BOOL,
      LITERAL_INT,
      LITERAL_FP,
      LITERAL_STR,
      LITERAL_SYM,
      LITERAL_PTR,
      COMMENT,
   Kind kind;
   std::string value;
   ea_t ea;
   const std::string& getColorTag() const;
};Code language: C++ (cpp)

It represents one lexical unit in the decompiled source code. Each such unit has a kind (e.g. ID_FNC ), value (e.g. main ), and an associated disassembly address (e.g. 0x8048577). See this wiki page for the detailed explanation. There is also a method returning an IDA color tag associated with the Token.

4. Functions

Finally, the representation of a decompiled function:

class Function
public:
   const std::string& getName() const;
   ea_t getStart() const;
   ea_t getEnd() const;
   /// Token at YX.
   const Token* getToken(YX yx) const;
   /// YX of the first token.
   YX min_yx() const;
   /// YX of the last token.
   YX max_yx() const;
   /// YX of the token before the token on the given YX.
   YX prev_yx(YX yx) const;
   /// YX of the token after the token on the given YX.
   YX next_yx(YX yx) const;
   /// [Starting] YX of the token which contains the given YX.
   YX adjust_yx(YX yx) const;
   /// Entire colored line containing the given YX.
   /// I.e. concatenation of all the tokens with y == yx.y
   std::string line_yx(YX yx) const;
   /// Address of the given YX.
   ea_t yx_2_ea(YX yx) const;
   /// Addresses of all the XYs with y == yx.y
   std::set<ea_t> yx_2_eas(YX yx) const;
   /// [The first] XY with the given address.
   YX ea_2_yx(ea_t ea) const;
   /// Is address inside this function?
   bool ea_inside(ea_t ea) const;
   /// Lines with associated addresses.
   std::vector<std::pair<std::string, ea_t>> toLines() const;
private:
   std::string _name;
   ea_t _start;
   ea_t _end;
   std::map<YX, Token> _tokens;
   /// Multiple YXs can be associated with the same address.
   /// This stores the first such XY.
   std::map<ea_t, YX> _ea2yx;
};Code language: C++ (cpp)

A function is a list of tokens with a name, start address, and end address. Tokens are indexed by their [starting] YX coordinates. YX coordinates are indexed by their disassembly addresses. Apart from the simple getters, there is a bunch of methods dealing with coordinates, addresses, and lines. These will get quite handy in a moment.

5. Places

Places (i.e. objects derived from abstract place_t ) denote locations of data displayed in viewers. IDA SDK defines several classes that could be used to display certain data. simpleline_place_t for string lines, idaplace_t for disassembly locations, hexplace_t for hex dump lines, etc.

If none of these is suitable for the user’s application, the plugin author can implement a custom derivation of the class. In our case, we create a demo_place_t class suited for representing locations and displaying tokens of decompiled functions:

class demo_place_t : public place_t
private:
   Function* _fnc = nullptr;
   YX _yx;
public:
   demo_place_t(Function* fnc, YX yx);
   YX yx() const;
   std::size_t y() const;
   std::size_t x() const;
   const Token* token() const;
   Function* fnc() const;
// Inherited from place_t.
public:
   /// Generate short location description used @ status bar.
   /// use: toea(), yx()
   void idaapi print(qstring* out_buf, void* ud) const override;
   /// Map the location to a number.
   /// use: y()
   uval_t idaapi touval(void* ud) const override;
   /// Clone the location.
   place_t* idaapi clone(void) const override;
   /// Copy the specified location object to the current object.
   void idaapi copyfrom(const place_t* from) override;
   /// Map a number to a location.
   place_t* idaapi makeplace(
      void* ud, 
      uval_t y, 
      int lnnum) const override;
   /// Compare two locations.
   /// use: yx()
   int idaapi compare2(const place_t* t2, void* ud) const override;
   /// Adjust the current location to point @ displayable obj.
   void idaapi adjust(void* ud) override;
   /// Move to the previous displayable location.
   /// use: Function::prev_yx()
   bool idaapi prev(void* ud) override;
   /// Move to the next displayable location.
   /// use: Function::next_yx()
   bool idaapi next(void* ud) override;
   /// Are we at the first displayable object?
   /// use: Function::min_yx()
   bool idaapi beginning(void* ud) const override;
   /// Are we at the last displayable object?
   /// use: Function::max_yx()
   bool idaapi ending(void* ud) const override;
   /// Generate text lines for the current location.
   /// use:: Function::line_yx()
   int idaapi generate(
      qstrvec_t* out,
      int* out_deflnnum,
      color_t* out_pfx_color,
      bgcolor_t* out_bgcolor,
      void* ud,
      int maxsize) const override;
   /// Serialize this instance.
   void idaapi serialize(bytevec_t* out) const override;
   /// De-serialize into this instance.
   bool idaapi deserialize(
      const uchar** pptr, 
      const uchar* end) override;
   /// Get the place's ID.
   int idaapi id() const override;
   /// Get this place type name.
   const char* idaapi name() const override;
   /// Map the location to an ea_t.
   /// use: Function::yx_2_ea()
   ea_t idaapi toea() const override;
   /// Rebase the place instance.
   bool idaapi rebase(const segm_move_infos_t&) override;
   /// Visit this place, possibly 'unhiding' a section of text.
   place_t* idaapi enter(uint32*) const override;
   /// Leave this place, possibly 'hiding' a section of text.
   void idaapi leave(uint32) const override;
}Code language: C++ (cpp)

YX is the location component, Function is the data component. Using YX, we can easily extract the information needed to implement the place_t interface from the associated Function. demo_place_t is both YX-aware and EA-aware location, something that none of the existing SDK places is.

User defined places need to be registered. We use the PCF_EA_CAPABLE flag to indicate our place is EA-aware. This will enable some handy out-of-the-box features later. Also, since SDK 7.5, all new plugins should use PCF_MAKEPLACE_ALLOCATES flag:

static demo_place_t _template(nullptr, YX());
demo_place_t::ID = register_place_class(
   &_template,
   PCF_EA_CAPABLE | PCF_MAKEPLACE_ALLOCATES,
   &PLUGIN
);Code language: C++ (cpp)

Unfortunately, place_t objects are by default considered to be just a line cursors (i.e. Y-sensitive). Special cases that are also X-sensitive need to implement the custom_viewer_adjust_place_t callback and fine-tune the X location. Otherwise, moving the cursor horizontally in a line will not change the X coordinate, nor the corresponding address:

/// Called whenever the user moves the cursor around.
/// See custom_viewer_adjust_place_t 
void idaapi cv_adjust_place(
   TWidget* v, 
   lochist_entry_t* loc, 
   void* ud)
   // Set X coordinate in loc->place()
   // using loc->renderer_info().pos.cx
}Code language: C++ (cpp)

In cases like these, SDK also recommends to implement a callback named custom_viewer_get_place_xcoord_t . It determines if two places are on the same line, and should prevent unnecessary viewer refreshes:

/// Does the line pointed to by pline include pitem,
/// and if so at what X coordinate?
/// See custom_viewer_get_place_xcoord_t
int idaapi cv_get_place_xcoord(
   TWidget* v,
   const place_t* pline,
   const place_t* pitem,
   void* ud)
   // Compare Y coordinates of pline and pitem.
}Code language: C++ (cpp)

6. Code viewer

Now we have everything to create a viewer displaying decompiled functions in plugins Context::run() method:

ea_t ea = get_screen_ea();
fnc = Decompiler::decompile(ea);
demo_place_t min(fnc, fnc->min_yx());
demo_place_t max(fnc, fnc->max_yx());
demo_place_t cur(fnc, fnc->ea_2_yx(ea));
custViewer = create_custom_viewer(
   title,        // title
   &min,         // minplace
   &max,         // maxplace
   &cur,         // curplace
   nullptr,      // rinfo
   &this,        // ud
   &ui_handlers, // handlers
   &this,        // cvhandlers_ud
   nullptr       // parent widget
codeViewer = create_code_viewer(custViewer);
set_code_viewer_is_source(codeViewer);
display_widget(codeViewer, WOPN_DP_TAB | WOPN_RESTORE);Code language: C++ (cpp)

ui_handlers is a set of custom viewer handlers:

static const custom_viewer_handlers_t ui_handlers(
   cv_keyboard,          // keyboard
   nullptr,              // popup
   nullptr,              // mouse_moved
   nullptr,              // click
   cv_double,            // dblclick
   nullptr,              // current position change
   nullptr,              // close
   nullptr,              // help
   cv_adjust_place,      // adjust_place
   cv_get_place_xcoord,  // get_place_xcoord
   cv_location_changed,  // location_changed
   nullptr               // can_navigate
);Code language: C++ (cpp)

We have already discussed cv_adjust_place and cv_get_place_xcoord . The other handlers will be examined in a moment.

With the code presented so far we get a plugin capable of displaying colored decompiled functions, reacting on both Y and X movements, and aware of disassembly addresses associated with the shown lexical units.

SDK allows to implement another powerful mechanism on custom places – location conversions. These are used for view synchronization, and if a custom place is PCF_EA_CAPABLE (which our is), then also for address navigation:

/// Register location converter.
static demo_place_t _template(nullptr, YX());
static idaplace_t _idaplace;
register_loc_converter(
   _template.name(),
   _idaplace.name(),
   place_converter
);Code language: C++ (cpp)

/// Implement location converter.
lecvt_code_t idaapi place_converter(
        lochist_entry_t* dst,
        const lochist_entry_t& src,
        TWidget* view)
   // demo_place_t -> idaplace_t
   if (src.place()->name() == _template.name())
      // set dst->set_place() according to src.place()
   // idaplace_t -> demo_place_t
   else if (src.place()->name() == _idaplace.name())
      auto idaEa = src.place()->toea();
      demo_place_t* current = get_custom_viewer_place(/*...*/);
      if (current->fnc()->ea_inside(idaEa))
         // set new place in the currently displayed function
      else if (Function* fnc = Decompiler::decompile(idaEa))
         // set new place in the newly decompiled function
}Code language: C++ (cpp)

Converter’s only job is to set the new place in the destination entry. If such a place is in the newly decompiled function, it is not enough. We also need to switch viewer’s content to this function. We do so in the location changed handler:

/// See custom_viewer_location_changed_t
void idaapi cv_location_changed(
        TWidget* v,
        const lochist_entry_t* was,
        const lochist_entry_t* now,
        const locchange_md_t& md,
        void* ud)
   Context* ctx = static_cast<Context*>(ud);
   auto* oldp = dynamic_cast<const demo_place_t*>(was->place());
   auto* newp = dynamic_cast<const demo_place_t*>(now->place());
   if (oldp->fnc() != newp->fnc())
      demo_place_t min(newp->fnc(), newp->fnc()->min_yx());
      demo_place_t max(newp->fnc(), newp->fnc()->max_yx());
      set_custom_viewer_range(ctx->custViewer, &min, &max);
      ctx->fnc = newp->fnc();
}Code language: C++ (cpp)

Changing the content is as easy as setting a new viewer’s range.

With these additions, we can now synchronize our view with IDA disassembly view.

We demonstrate these in two different but similar scenarios. First, we want to decompile and display a new function, if we double click on its call statement. The catch is, we want to do it only for functions and not for other lexical elements:

/// Implement double-click callback.
bool idaapi cv_double(TWidget* cv, int shift, void* ud)
   demo_place_t* place = get_custom_viewer_place(/*...*/);
   auto* token = place->token();
   if (token->kind != Token::Kind::ID_FNC)
      return false;
   func_t* fnc = find_fnc_by_name(token->value);
   // Jumping to function's start address triggers
   // the place converter - i.e. possible decompilation.
   jumpto(fnc->start_ea, -1, UIJMP_ACTIVATE);
}Code language: C++ (cpp)

Second, we want to have different right-click-popup options depending on the kind of token under the cursor:

/// Make our plugin class an event listener.
class Context : public plugmod_t, public event_listener_t
public:
   virtual ssize_t idaapi on_event(ssize_t code, va_list va) override;
}Code language: C++ (cpp)

/// Hook our class to user interface events.
bool idaapi Context::run(size_t)
   hook_event_listener(HT_UI, this);
}Code language: C++ (cpp)

/// Implement the event listener callback.
ssize_t idaapi Context::on_event(int code, va_list va)
   switch (code)
      // Attach our actions to context menu.
      case ui_populating_widget_popup:
         demo_place_t* place = get_custom_viewer_place(/*...*/);
         auto* token = place->token();
         if (token->kind == Token::Kind::ID_FNC)
            // something
         else if (token->kind == Token::Kind::ID_VAR)
            // something else
         break;
}Code language: C++ (cpp)

In both cases, we can ask IDA what is the current viewer place. Using its YX we get the token at these coordinates in the associated function. It is then easy to use the token to make context-sensitive actions.

It is now possible to query custom viewer’s synchronization group, and interfere in viewer’s line rendering:

/// Implement the event listener callback.
ssize_t idaapi Context::on_event(int code, va_list va)
   switch (code)
      // Line rendering.
      case ui_get_lines_rendering_info:
         auto* demoSyncGroup = get_synced_group(custViewer);
         demo_place_t* demoPlace = get_custom_viewer_place(/*...*/);
         // Get callback info.
         lines_rendering_output_t* out = 
               va_arg(va, lines_rendering_output_t*);
         TWidget* view = va_arg(va, TWidget*);
         lines_rendering_input_t* info = 
               va_arg(va, lines_rendering_input_t*);
         // Check view synced with our view is being rendered.
         if (info->sync_group != demoSyncGroup)
            return false;
         // Get all the addresses associated with the current
         // line in our view.
         auto eas = demoPlace->fnc()->yx_2_eas(demoPlace->yx());
         // Fill 'out' with entries highlighting lines with 
         // addresses from 'eas' set.
         break;
}Code language: C++ (cpp)

Note: This highlighting implementation works a bit differently than the one in Hex-Rays plugin. It collects and highlights all the addresses represented by the current line in our demo viewer. Hex-Rays on the other hand seems to highlight whole continuous blocks delimited by the surrounding lines.

How to use this article?

Well, however you wish! We have shown how to use advanced IDA SDK features related to places and movements to create a dummy decompilation plugin. Hopefully, the examples in this article, or the full example in our GitHub repository , will help you make your plugins more interactive.

We will use the mechanisms shown here in our upcoming RetDec IDA plugin v1.0 to create a free and open decompilation plugin on par with Hex-Rays. Check out the project if you want to see a complex real-world usage example, or if you just need a free decompiler for IDA or Radare2 .

Know Your YARA Rules Series: #5 Everything You Need to Know About Regular Expressions in YARA Know Your YARA Rules Series: #4 YARA-X: The Next Generation Know Your YARA Rules Series: #3 Watch Out For Releases Know Your YARA Rules Series: #2 Rewrite Your Rules