Readme
(2024-04-30 v3.6)
(2023-12-18 v3.5)
ResponseEvent: regex matching, fuzzy
matching, and vector encoding matching.VhIoWrapper and TapActionWrapper. Added support to SCROLL
and TYPE to TapActionWrapper.RemoteSimulator. In order to reduce the delay of network
transfering, enabled action batch to send and execute a group of actions
and enabled resizing the image before and after transferring to shrink the
transferred data.ResponseEvent to annotation tool.For more details, please see our Change Log and Documents.
OR-type virtual event combining
multiple types of event sources.Please see our Change Log and Document.
Please see our Change Log and Document.
ResponseEvent). Now enables the agent to generate response to human user
and parses episode signales from it. This will enable interaction tasks like
question-answering, retrieval, etc.Please see our Change Log, Usage Document, and Task Definition Document.
Mobile-Env is a interaction platform for evaluation and training of the interaction of information user interface (InfoUI). Our paper is available at arXiv.
Mobile-Env is developed based on AndroidEnv. The agent can take the screenshot and the view hierarchy (disabled defaultly for the long latency) as the observation and take a touch or type a token as the action to interact with the Android apps. Several episode signals like step instructions, rewards, or the episode end will be informed during interaction at some crucial steps. A so-called crucial step may be opening a target page, srolling to a correct area, etc. and is depending on the specific task definition.
The proposed WikiHow task set is available at the Hugging Face Platform.
Mobile-Env is a flexible, adaptable, and easily-extendable platform for InfoUI interaction with the following features:
Install from PyPI:
pip insall mobile-env-rl
or clone the repository and build locally.
git clone https://github.com/X-LANCE/Mobile-Env
cd Mobile-Env
pip install .
Several Docker images with well-configured Android AVD are also available.
Before loading the Mobile-Env environment, you will need to set up an Android
Emulator device. Then you can load the
environment with some existing task definitions and start your experiments. A
detailed guidance is provided in Evaluating and Traning Agents on
Mobile-Env. Several examples with a random agent or a
human agent is also provided under examples.
To extend a new environment for Mobile-Env, the environment designer needs to prepare the app package and ensure that the package manages to launch and run on some versions of Android Emulator. If the app requires varying online data, the necessary data should be crawled and dumped and then be replayed for a consistent evaluation. In such case, the designer is supposed to validate the certain effectiveness of certificate unpinning plan for the package. As regards to extend new tasks, task definition files are just required. Detailed instructions can be found in Extending a New Environment (App) or a New Task Based on Mobile-Env.
Several demo task definitions are provided under demos. Three of them are
migrated from AndroidEnv:
classic_2048.m.textproto - Classic 2048
game.accessibility_forwarder_clock_set_timer.m.textproto - A simple task
requiring the agent to reset a running
timer.systemui_egg_land_default.m.textproto - Flappy
Droid.
An open-sourced implementation of classic game, Flappy Bird.Another one, openmoneybox.add_billings.textproto is defined upon an
open-sourced billing app,
OpenMoneyBox.
Details are referred to in the task definition files.
We also developed an annotation tool for the human demonstrations, and a suite of template tool to auto-generate task definitions according to templates and to combine multiple task definitions to form a multi-step task. The details are referred to in Miscellaneous Auxiliary Tools.
The data are measured under the configuration below:
| Item | Avg Time | Time Std Dev |
|---|---|---|
TOUCH action | 410.50µs | 64.71µs |
LIFT action | 412.30µs | 84.18µs |
TEXT action | 1.30s | 0.28s |
| screenshot capturing | 19.94ms | 21.47ms |
| invocation of Sentence Transformer（all-MiniLM-L12-v2） | 8.51ms | 0.17ms |
| VH capturing | 2.53s | 1.90s |
| invocation of EasyOCR | 0.44s | 0.08s |
When only an app of WikiHow 2.9.6 is running, the Android emulator occupies 6,031 MiB of virtual memory and 3,444 MiB of residual memory.
This library is developed and maintained by SJTU X-Lance. The corresponding paper is available at https://arxiv.org/abs/2305.08144.
If you find Mobile-Env useful in your research, you can cite the project using the following BibTeX:
@article{DanyangZhang2023_MobileEnv,
title = {{Mobile-Env}: An Evaluation Platform and Benchmark for LLM-GUI Interaction},
author = {Danyang Zhang and
Hongshen Xu and
Zihan Zhao and
Lu Chen and
Ruisheng Cao and
Kai Yu},
journal = {CoRR},
volume = {abs/2305.08144},
year = {2023},
url = {https://arxiv.org/abs/2305.08144},
eprinttype = {arXiv},
eprint = {2305.08144},
}
FAQs
A Universal Platform for Training and Evaluation of Mobile Interaction
We found that mobile-env-rl demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
NIST's new AI Risk Management Framework aims to enhance the security and reliability of generative AI systems and address the unique challenges of malicious AI exploits.
Security News
This episode of the Risky Biz podcast discusses how the rise of small open source packages and the shift towards individual maintainers makes the ecosystem more vulnerable to supply chain attacks.
Product
Streamline your login process and enhance security by enabling Single Sign-On (SSO) on the Socket platform, now available for all customers on the Enterprise plan, supporting 20+ identity providers.