规则密集环境下的解释意识经验重放

人类环境通常由明确而复杂的规则集来调节。将强化学习（RL）代理整合到这样的环境中，促使了学习机制的发展，这些机制在规则密集和异常多的环境中表现良好，如在规范的道路上自主驾驶。在本文中，我们提出了一种组织经验的方法，即把经验缓冲区划分为以每个解释为基础的群组。我们提出了与模块化规则集和9个学习任务兼容的离散和连续导航环境。对于具有可解释规则集的环境，我们将基于规则的解释转换成基于案例的解释，将状态转换分配到用解释标记的集群中。这使我们能够以课程和任务为导向的方式对经验进行抽样，重点关注事件的稀有性、重要性和意义。我们给这个概念贴上了解释-意识（XA）的标签。我们用集群内和集群间的优先级来进行XA经验重放（XAER），并引入XA兼容的DQN、TD3和SAC版本。与传统的优先体验重放基线相比，这些算法的XA版本的性能一直很好，这表明解释工程可以用来代替具有可解释特征的环境的奖励工程。

原文题目：Explanation-Aware Experience Replay in Rule-Dense Environments

原文：Human environments are often regulated by explicit and complex rulesets. Integrating Reinforcement Learning (RL) agents into such environments motivates the development of learning mechanisms that perform well in rule-dense and exception-ridden environments such as autonomous driving on regulated roads. In this paper, we propose a method for organising experience by means of partitioning the experience buffer into clusters labelled on a per-explanation basis. We present discrete and continuous navigation environments compatible with modular rulesets and 9 learning tasks. For environments with explainable rulesets, we convert rule-based explanations into case-based explanations by allocating state-transitions into clusters labelled with explanations. This allows us to sample experiences in a curricular and task-oriented manner, focusing on the rarity, importance, and meaning of events. We label this concept Explanation-Awareness (XA). We perform XA experience replay (XAER) with intra and inter-cluster prioritisation, and introduce XA-compatible versions of DQN, TD3, and SAC. Performance is consistently superior with XA versions of those algorithms, compared to traditional Prioritised Experience Replay baselines, indicating that explanation engineering can be used in lieu of reward engineering for environments with explainable features.