The embedding-based reasoning methods map entities and relationships in KGs to a low dimensional vector space, allowing their semantic and structural information to be expressed in the form of geometric relationships, thereby efficiently handling reasoning tasks in large-scale KGs. Early methods such as TransE [11], TransH [12] and DistlMult [13] focused on studying geometric relationships such as translation, projection, and linear transformation between embedded vectors of entities and relationships, providing efficient and semantic knowledge representation for knowledge reasoning. Lu et al. [14] proposed the DensE method, which decomposes each relationship into rotation and scaling operators in three-dimensional Euclidean space, which can better represent composite relationships. Pavlovic et al. [15] proposed the ExpressivE method, which uses the spatial relationships of super parallel quadrilaterals to represent the semantic structural information between entities and relationships, providing intuitive geometric explanations for knowledge reasoning tasks. Although embedding based reasoning methods have shown good performance in large-scale KGs reasoning tasks, they are difficult to handle relationships with multi-level semantics or nonlinear structures, and have weak modeling capabilities for complex relationships.

The path-based reasoning methods utilizes the powerful expressive power of neural networks to optimize the reasoning process, which can effectively handle reasoning tasks in large-scale KGs. Dettmers et al. [16] proposed a multi-layer convolutional network mode called ConvE for link prediction, which captures the relationship information between entities through convolutional networks and effectively reasons over large-scale KGs. Yang et al. [17] proposed the hyper relation aware multi-view model HyRel, which learns the globally transferable structure of a graph to reason about unseen graphs, offering high flexibility and usability. Xiong et al. [18] proposed a reinforcement learning framework for learning multi-hop relational paths, which effectively handles complex reasoning tasks with ambiguous answers and scales to large KGs. The path-based reasoning methods perform well in large-scale KG reasoning, but the model requires a large number of training parameters, high computational complexity, and demand significant computing and storage resources.

The superposition and entanglement properties of quantum computing give it unique advantages in solving certain computational tasks that traditional computers do not have. Chen et al. [10] explored the potential of quantum computing to enhance reinforcement learning and demonstrated the potential of quantum algorithms in improving time and space complexity in simple environments. Chen et al. [4] proposed the asynchronous training of advantage actor-critic variational quantum policies, which reduces the training time of quantum reinforcement learning. As the application of quantum computing in reinforcement learning progresses, quantum reinforcement learning has been extensively explored and applied across various fields. Kim et al. [5] integrated quantum reinforcement learning into reusable rocket control systems, enhancing both computational efficiency and model stability. Ansere et al. [6] proposed a Quantum-empowered Deep Reinforcement Learning (Qe-DRL) approach to enhance computational learning speed and task processing efficiency for IoT devices under quantum uncertainty and time-varying channel conditions. In addition, quantum reinforcement learning is widely used in fields such as biochemistry and medicine, and has shown good performance. However, the above application scenarios are generally relatively simple, usually requiring only a limited number of qubits and small state and action spaces. KGs typically contain large states and action spaces. To reduce the computational complexity of reasoning models in large-scale KGs, this paper proposes a quantum reinforcement learning based knowledge reasoning model QIRL, which greatly reduces the training parameters of the model by utilizing quantum advantages. The performance of the proposed model is validated on universal datasets. In contrast to emerging LLM-based reasoning methods [9], our approach emphasizes computational efficiency and parameter reduction via quantum-inspired techniques.

The QIRL method updates the parameters of the quantum policy network by continuously interacting with the environment to find the optimal reasoning paths. In this paper, the environment is modeled as a Markov Decision Process (MDP) and the interaction dynamics between the agent and the KG are specified. MDP can be represented by tuple ⟨S,R,P,R′⟩, where S={s1,s2,⋯,sn} represents the continuous state space of the KG as input to the quantum policy network. R={r1,r2,⋯,rn} represents the set of relationships in the KG, that is, the set of all next optional actions output by the quantum policy network. P(Si+1=s,|Si=s,Ri=r) is a state transition probability function that represents the probability of the QIRL policy network transitioning from state s to state s′ upon selecting action r. R′⁢(s,r) is the reward function for each pair (s,r), designed to encourage the agent to learn the optimal strategy. The specific settings of the QIRL method are as follows:

Actions: The actions of the quantum policy network can be represented by the relationship r in the KG. For the entity pair (hi,ti), starting from the head entity hi, the agent selects the action with the highest probability as the next action ri based on the output action probability distribution of the quantum policy network, to extend the path until reaching the real tail entity ti; To ensure the consistency of the output dimension of the strategy network, this paper defines the action space as all relationships and their inverse relationships in the KG, and the dimension of the action space is consistent with the number of quantum bits in the quantum circuit.

Figure 1.

Mimic octopus.

For each entity pair (hi,ti), a random breadth first search (BFS) is used to find all paths between the entity pairs. This paper introduces a random mechanism in the BFS algorithm, which randomly selects an intermediate node instead of directly searching for paths between the head entity hi and tail entity ti. Then perform two BFS between (hi,i⁢n⁢t⁢e⁢ri) and (i⁢n⁢t⁢e⁢ri,ti) to search for the correct paths faster and improve the convergence speed of the model. For entity pairs (h,t), randomly select the inter mediate entity i⁢n⁢t⁢e⁢r and perform BFS between (h,i⁢n⁢t⁢e⁢r) and (i⁢n⁢t⁢e⁢r,t) to find paths. Entities are usually associated with multiple different relationships. Assuming that the head entity h connected to the new entity e1 through the relationship r1, and then e1 connected to the entity i⁢n⁢t⁢e⁢r through the relationship r2, a reasoning path r1→r2 is obtained. i⁢n⁢t⁢e⁢r connected to the new entity e2 through relationship r3, and e2 is connected to the tail entity t through relationship r4. Therefore, the reasoning path is r3→r4. Combining the two reasoning paths together, the final reasoning path between (h,t) is P=r1→r2→r3→r4.

This paper conducts supervised training on the quantum policy network of QIRL method based on the paths between entity pairs searched by random BFS method. The quantum strategy network maximizes the expected cumulative reward by continuously updating the parameters of the quantum circuit w. The expected cumulative reward J⁢(w) is expressed as follows:

J ⁢ ( w ) = ∑ i ∑ r ∈ R π ⁢ ( r | ξ ⁢ ( s i ) ; w ) ⁢ R s i , r i ′

where Rsi,ri′ represents the reward value obtained by selecting action ri in state si. For each supervised path, the agent receives a reward of +1 for each successful search and updates the quantum circuit parameters using the approximate gradient of the Monte Carlo strategy gradient:

∇ w J ⁢ ( w ) = ∑ i ∑ r ∈ A π ⁢ ( r | ξ ⁢ ( s i ) ; w ) ⁢ ∇ w log ⁡ π ⁢ ( r | ξ ⁢ ( s i ) ; w ) = ∇ w ⁢ ∑ i log ⁡ π ⁢ ( r = r i | ξ ⁢ ( s i ) ; w )

where ri belongs to path P, which represents the action with the highest probability of the quantum circuit output in step i.

In the pretraining process of the quantum strategy network mentioned above, the reasoning paths obtained often contain a large amount of redundant information and similar paths. To improve reasoning efficiency, this paper retrains the quantum policy network using a reward function to find a more efficient reasoning path controlled by the reward function. For each entity pair (hi,ti), the agent selects an action from the head entity hi based on the output value of the quantum policy function π⁢(r|ξ⁢(si);w) to expand the reasoning paths. If the selected action cannot connect to any entity, the agent receives a negative reward and remains in its original state. Since agents follow the quantum policy function to extend the reasoning paths, they will not get stuck due to repeating incorrect steps. This paper improves training efficiency by limiting the maximum length of the reasoning paths. If the agent finds the correct tail entity ti within the maximum path length, a new reasoning path is generated. Conversely, if the agent fails to locate the correct tail entity ti within the constrained path length, the current training iteration is terminated. The gradient function of the quantum strategy network during the retraining process is as follows:

∇ w J ⁢ ( w ) = ∇ w ⁢ ∑ i log ⁡ π ⁢ ( r = r i | ξ ⁢ ( s i ) ; w ) ⁢ R t ⁢ o ⁢ t ⁢ a ⁢ l ′

The retraining process of the model is shown in Algorithm 1, where w is the parameters of the quantum circuit, ξ⁢(st) is the quantum amplitude encoding of the input state st, and Mn⁢e⁢g represents the set of negative steps.

Entity reasoning refers to reasoning about missing entity information from known entities and their relationships through the relationships or attributes between entities in a KG. Due to the large number of complex relationships between entities in KGs, conducting path searches one by one will result in high time costs. The bidirectional path constrained search algorithm [18] is an efficient path search algorithm that can simultaneously search for paths from both positive and negative directions, significantly reducing the number of search paths. Therefore, this paper adopts a bidirectional path constrained search algorithm to search for paths between entity pairs, significantly reducing the search space and improving the efficiency of path search.

For an entity pair (h,t), the reasoning path P′ trained by the QIRL method is used as a logical formula, starting from the head entity h and tail entity t respectively, and gradually expanding the reasoning paths through a bidirectional path constraint search algorithm. If the intersection of the path entities in both directions of the bidirectional path constraint search algorithm is not empty, the true tail entity can be successfully found based on the reasoning path P′. Otherwise, the tail entity cannot be successfully reasoned.

Assuming there is a reasoning path P=r1→r2, perform bidirectional path constraint search on entity pair (h,t). Starting from the head entity h and connecting to the entity e through relationship r1, the forward entity collection is l⁢e⁢f⁢t={e}. At this point, if the length of the forward entity set l⁢e⁢f⁢t is 1 and the length of the reverse entity set r⁢i⁢g⁢h⁢t is 0, the next step is to expand r⁢i⁢g⁢h⁢t. Starting from the tail entity t and connecting to the entity e through relationship r2, r⁢i⁢g⁢h⁢t={e}, at this point, the intersection of l⁢e⁢f⁢t and r⁢i⁢g⁢h⁢t is not empty, the tail entity is successfully reasoned out.

To evaluate the performance of the QIRL method in entity prediction tasks, partial KG data was extracted from two benchmark datasets, Kinship and YAGO3-10, for testing. The Kinship dataset is small but logically complex, testing the model's reasoning ability with sparse data. The YAGO3-10 dataset is large and diverse, evaluating the model's scalability and generalization.The increase in the number of relationships in knowledge reasoning enhances the completeness of reasoning paths and improves the accuracy of knowledge reasoning. Constrained by the computational hardware's processing capabilities, the quantum reinforcement learning model in this paper contains 16 qubits and can handle 16 relationships in the dataset. To find the reasoning path more efficiently, breadth first search is performed from both the head entity and tail entity directions. The action space includes the relationships between entities and their inverse relationships. The statistical data of the extracted dataset is shown in Table 1:

Table 1

Statistical data of the dataset.

The QIRL model parameters proposed in this paper include the quantum bit count n⁢u⁢mq⁢u⁢b⁢i⁢t⁢s of the quantum circuit, the learning rate l⁢r of the model training, the entity embedding dimension e⁢m⁢b⁢e⁢d⁢d⁢i⁢n⁢gd⁢i⁢m, the reinforcement learning discount rate γ, the maximum number of attempted steps for reasoning path search m⁢a⁢xl⁢e⁢n⁢g⁢t⁢h, etc., where num_qubits is the number of relationships contained in the dataset. To improve the generalization ability and accuracy of the model, this paper generates positive and negative samples of the test set data based on pra [25]. This paper uses a TransE based method to train embedding vector representations of entities and relationships. The optimal parameters for the model are e⁢m⁢b⁢e⁢d⁢d⁢i⁢n⁢gd⁢i⁢m=100, γ=0.99 and m⁢a⁢xl⁢e⁢n⁢g⁢t⁢h=50.

This paper compares embedding-based methods and path-based methods to explore their performance on the dataset used in this paper. Embedding-based methods include translation-based method TransE [11], rotation-based method RotalE [19] and complex vector space-based method CompleEx [20]. Path-based methods include neural network-based and logical reasoning combined method NeuralLP [21], deep learning-based and relational reasoning combined method DRUM [22], rule-based method RulE [23], neural network-based and symbolic reasoning combined method NCRL [24], and reinforcement learning-based method DeepPath [18]. In order to ensure the fairness of the experiment., this paper used the publicly released source code of each model and adopted the best hyperparameters provided in the original text for the experiment.

Table 2

Comparison of relationship path numbers.

Figure 2.

Comparison of relationship path numbers.

(a)

Relationship Path Numbers of YAGO3-10

Figure 2 shows the entity reasoning MAP values of the QIRL method on different relationships between the Kinship and YAGO3-10 datasets, and Table 2 lists the number of reasoning paths found by the QIRL method on the Kinship and YAGO3-10 datasets. The agent found more reasoning paths in the Kinship dataset than in the YAGO3-10 dataset, which means the agent can find strongly correlated reasoning paths in the YAGO3-10 dataset and filter out similar or unrelated ones, but it is difficult to find the most relevant reasoning paths in the Kinship dataset. In addition, when the number of reasoning paths is too small, it is also difficult to find sufficient reasoning evidence to obtain reasoning results.

As shown in Table 3, the test results of QIRL method on YAGO3-10 are better than those on Kinship, because the correlation between entities in Kinship is low, and QIRL cannot find sufficient reasoning evidence on Kinship. However, in YAGO3-10, the correlation between entities is high and the QIRL method can find sufficient reasoning evidence for reasoning, thus obtaining good test results.

Table 3

Reasoning results.

Table 4

Comparison of parameter numbers in different models.

Table 5

Comparison of model performance on accuracy and parameter number.

Compared with embedding-based methods, path-based methods such as QIRL perform worse on Kinship. Embedding-based methods are reasoning through distance or similarity measures. The relationships in Kinship are mostly simple and symmetrical, and most relationships can be directly learned through geometric relationships in the embedding space. However, path-based methods such as QIRL do not fully utilize the simple structures in the data, so embedding-based models can better leverage their advantages. On YAGO3-10, the performance of embedding-based methods significantly decreases due to the complexity of YAGO3-10 relationship reasoning tasks. At this point, path-based methods such as QIRL can better capture the complex relationships between entities.

The performance of QIRL on Kinship and YAGO3-10 is inferior to path-based methods such as RulE and NCRL. RulE learns explicit logical rules for reasoning and directly utilizes the inherent structure and relationships in the data. Therefore, it can achieve better results on Kinship dataset with clear regularity in relationships. NCRL combines neural networks and logical reasoning, using graph neural networks to process graph structured data, which can effectively learn potential patterns between entities and relationships. Therefore, it can achieve good results in the YAGO3-10 dataset with complex relationships and no clear rules.

Table 4 shows the number of network training parameters for different models. Through comparative analysis, it can be clearly seen that compared to other methods, the QIRL algorithm requires significantly fewer parameters in the model training process, only requiring hundreds of parameters to effectively train the model and achieve excellent results. This feature enables the QIRL algorithm to significantly reduce the computational resource consumption of the model while maintaining efficient training. By significantly reducing the number of parameters, QIRL not only demonstrates significant advantages in memory and computing resources, but also effectively avoids problems such as overfitting, thereby improving the model's generalization ability and robustness in practical applications. Overall, the QIRL algorithm has improved model performance and significantly enhanced computational efficiency by optimizing the number of parameters, demonstrating its potential in complex tasks.

As shown in Table 5, QIRL significantly reduces the number of model parameter while maintaining high model accuracy. In contrast, other methods typically increase the training parameters to improve accuracy or sacrifice model accuracy in order to reduce the number of parameter. Table 5 compares exclusively neural network-based methods to ensure parameter computability.