Databases and Information Systems Commons^™

Learning And Understanding User Interface Semantics From Heterogeneous Networks With Multimodal And Positional Attributes, Meng Kiat Gary Ang, Ee-Peng Lim

Research Collection School Of Computing and Information Systems

User interfaces (UI) of desktop, web, and mobile applications involve a hierarchy of objects (e.g., applications, screens, view class, and other types of design objects) with multimodal (e.g., textual and visual) and positional (e.g., spatial location, sequence order, and hierarchy level) attributes. We can therefore represent a set of application UIs as a heterogeneous network with multimodal and positional attributes. Such a network not only represents how users understand the visual layout of UIs but also influences how users would interact with applications through these UIs. To model the UI semantics well for different UI annotation, search, and evaluation tasks, …