Deep reinforcement learning enhances artistic creativity: The case study of program art students integrating computer deep learning

2.1 Neural network architecture

DQN combines deep learning with reinforcement learning to effectively learn and optimize strategies in complex tasks [16,17]. The following are the main elements of DQN’s neural network architecture [18,19].

2.2 Input and output

In the procedural art generation task, the experiment uses DQN as a key component to guide the art generation process. The following is a detailed description of the input and output of DQN in this study [28].

2.3 Reinforcement learning parameters

Compared with ordinary neural networks, DQN introduces a series of special parameters and techniques, such as experience playback, goal network, ε-greedy strategy, and Q-Learning goal. The introduction of these parameters and techniques is mainly due to the particularity of Q network [31]. In order to optimize Q network training so that it can stably estimate Q value functions, thereby improving the performance of reinforcement learning tasks, and helping DQN to perform well when dealing with reinforcement learning problems in discrete action spaces, the following DRL parameters need to be considered in detail in procedural art generation tasks.

2.4 Network training

3.1 Dataset preparation

The preparation of the dataset is a worthy part of the experiment when performing the task of procedural art generation. High-quality datasets can affect the performance of DQNs and the quality of the generated artwork. In this study, WikiArt, an open source dataset, is chosen for the experiment. The dataset is a large dataset of paintings, which contains a large number of art works, covering various periods, styles, and artists. A total of 195 paintings by artists are included, including 42,129 for training and 10,628 for testing, for a total of 52,757. The works are divided into eight categories, including landscapes, figures, still life, and abstractions. It is a collection of data that is well worth exploring for art lovers, researchers, or practitioners. This experiment would focus on the experiment of two types of artistic data: color field painting and minimalism.

The selection of the WikiArt dataset is crucial for the success of this study, as it provides rich and diverse art samples for DQN. This dataset is chosen because it has several key characteristics that make it an ideal choice for programming art generation tasks. First, WikiArt contains a wide range of art works from different historical periods, which means it covers a variety of art styles from classical to modern. This diversity provides DQN with rich learning resources, enabling the network to learn and imitate various different artistic styles. In addition, the WikiArt dataset offers a wide variety of art works, ranging from landscape paintings, figure paintings, still life paintings to more abstract art forms such as abstract and minimalist art. These different types make the dataset not only visually diverse but also equally rich in concepts and presentation techniques. Especially, the art styles of color block painting and minimalism, due to their unique visual features and expressive techniques, provide a challenge for the DQN model and an opportunity for researchers to evaluate the model’s ability to handle different art styles. In order to ensure the quality and representativeness of the dataset, strict standards are adopted in the selection process of WikiArt. The selected artwork not only needs to have sufficient visual resolution and clarity, but also needs to ensure diversity and balance between samples. This means that the number of works in each category needs to be representative enough to avoid bias in the learning process of the model. Finally, the historical depth and cultural breadth of the WikiArt dataset provide a comprehensive artistic worldview for the DQN model. By learning and analyzing the works in this dataset, DQN can not only imitate existing artistic styles, but also create new forms of artistic expression based on this foundation.

Although WikiArt covers a wide range of painting art styles for the purposes of this article, its format is not suitable as a data input for DQN. Therefore, translating the WikiArt dataset into a data format suitable for DQN model inputs became an urgent problem. The experiment needs to conduct data preprocessing and feature extraction on the collected dataset to match the experiment format and model.

Since DQN combines Q-learning, a reinforcement learning algorithm, the model takes the environmental state as the input and the action as the output. It is sensitive to the scale and distribution of the input, so it is necessary to process the environmental status and scale of the original data first.

1. Data cleaning: In the collected data, it needs to detect and remove low-quality images or abnormal data in the dataset. Data with damage, incomplete information, large image contrast differences, and obvious edge defects need to be cleaned. It can detect whether there are duplicate art works in the dataset, ensuring the diversity of the data. On this basis, this article proposes a new method for image classification of art works.

2. Normalization of image attributes: Standardization requires all images to have the same format, such as Joint Photographic Experts Group or Portable Network Graphics. This can ensure consistent identification of image data by subsequent models. It can adjust the size and resolution of images to maintain consistency. This method not only reduces the computational complexity of the system, but also ensures the consistent dimensionality of the input data of the system. The color, brightness, contrast, and other parameters of an image can be standardized to have the same visual characteristics during training. On this basis, the analysis of relevant metadata (author, style, creation time, etc.) can be used to achieve accurate and consistent metadata, providing a basis for subsequent data annotation and reward functions.

3. Feature extraction: Next it is necessary to extract features related to states, actions, rewards, and the next state from each artwork.

State: It can use the current picture to represent a state. Finally, the convolutional neural network can be used to extract the features of the image and carry out feature extraction. This eigenvector is the state expression of DQN.

Action: In procedural art generation, action is represented as editing or transforming an existing art image. These techniques include color adjustment, style change, shape adjustment, blur effect, and line enhancement. Each action can be assigned a unique identifier and mapped to a form that the model can understand. In the experiment, for the adjustment of color, style change, shape adjustment, blur effect, and line enhancement of the five editing operations, this study uses 0–5 integers for each operation.

Reward: The design of the reward function determines how the model evaluates the generated artwork. Rewards are defined based on factors such as the quality of the generated work, its closeness to the target style, and its innovation. If the generated work closely matches the target style, a higher reward can be allocated.

Next state: The next state is the Art Nouveau image after the action is performed. Based on the editing of the current state by the action, a new image can be generated and used as the next state.

3.2 Parameter tuning

When using DQN for training, this article adjusts the following hyperparameters to optimize the performance of the algorithm: Learning rate ( ), greedy strategy ( ), experience playback buffer size (buffer size), discount factor ( ), and maximum training epoch (Epoch). The experiment tested these five parameters separately, and the specific parameters are shown in Table 1:

Among the above parameters, the learning rate determines the step size at which the model updates weights in each iteration. Greedy strategy is a balance between exploring new ideas and utilizing known effective strategies. The size of the experience playback buffer is used to store and reuse previous experiences; the discount factor is used to balance the importance of current rewards with future rewards. Maximum number of training rounds: the maximum number of steps or rounds specified for the training algorithm to run is an important hyperparameter.

3.3 Training process

The training process of DQN in program art generation:

1. Initialize the network and experience replay buffer: Before training, the weights and optimizers of the DQN neural network can be initialized. It needs to create an experience replay buffer to store samples of previous states, actions, rewards, and the next state for training purposes.

2. Environmental interaction: During each training cycle (epoch), DQN interacts with the program art generation environment. The model selects an action from the current environmental state, executes the action, and observes the reward returned by the environment and the next state. This process is repeated until the predetermined number of time steps are reached or the generated artwork is considered complete.

3. Experience replay: At the end of each time step, the status, behavior, rewards, and data for the next stage are saved in an experience replay cache. Its main purpose is to disrupt the interrelationships between data and enhance the stability of training. In each batch of training, a random batch would be selected from an experienced playback buffer for training.

4. Calculate Q value and target value: Existing neural networks can be used to calculate Q value for each behavior in each state. These Q values are calculated based on the weighting of the current network. At the same time, the target Q value is calculated based on the weight of the target network (usually a copy of the current network, regularly updated). After completing various actions, the target Q value is used to evaluate long-term benefits.

5. Calculate loss function: By calculating the target Q value and combining it with the actual Q value of the existing network, the corresponding loss function is obtained. The experiment used MSE to measure the difference between the predicted values of the model and the actual indicator values.

6. Optimize the network and update the target network: Backpropagation algorithm can be used and combined with an optimizer to update the weights in the network to minimize the loss function in the network. The image generation algorithm can be optimized and the weights of the target network can be periodically updated. Under a given number of training steps, the weights of the target network can be copied, which can improve the stability and efficiency of training.

7. Repeat training cycle: The above steps can be repeated until the predetermined number of training cycles is reached or the stop condition (performance convergence) is met. The number of training cycles is usually a hyperparameter that needs to be adjusted based on the complexity of the task and computational resources.

In this study, the reasons for selecting specific hyperparameters and their impact on model performance are the core components of experimental design. Choosing the appropriate learning rate α is crucial as it determines the speed at which the model absorbs new information in each iteration. An excessively high learning rate may cause the model to fail to converge during the training process, while an excessively low learning rate may slow down the training progress. In the experiment, different learning rates are tested to find the optimal balance point, enabling the model to learn effectively without losing valuable information too quickly. The selection of the greedy strategy value ε is crucial for balancing the exploration and utilization of the model. A higher ε value encourages the model to explore more, while a lower value makes the model more inclined to utilize known information. Determining the appropriate ε value can help the model better find a balance between the two, thereby achieving optimal performance during the learning process. The size of the experience replay buffer is also a key factor. A larger buffer can store more experience, providing a richer learning material for the model. However, an excessively large buffer may cause the model to use outdated information during training, affecting learning efficiency. The setting of the discount factor γ affects the model’s emphasis on future rewards. A higher γ value makes the model more focused on long-term rewards, while a lower value makes the model more inclined to pursue immediate rewards. The adjustment of this parameter helps to control the behavior strategy of the model, making it more suitable for specific artistic creation tasks.

Next experimental testing would be conducted on the various parameters selected above, with different interval ranges set for each parameter. The purpose is to select the most suitable set of parameters for the experiment. First, the learning rate parameter was analyzed and three different settings were set, namely, 0.01, 0.005, and 0.001. The other parameters are temporarily initialized using the established parameters, with set to 0.01, Buffer size set to 10,000, set to 0.95, and epoch selected as 500 for the experiment.

Next was tested, with four parameters set: 0.5, 0.1, 0.05, and 0.01. The variance of performance uncertainty under different parameters, exploration-exploitation balance, and convergence speed were collected. The higher the value of variance, the greater the volatility of model performance. The convergence speed refers to the speed at which a model achieves optimal or near optimal performance during the training process. Exploring utilizing balance is a strategy in which algorithms find a balance between exploring new actions (finding potentially better strategies) and utilizing known actions (proven effective strategies). If the value of the exploration-exploitation balance is 0.3, it means that the algorithm has a 30% probability of choosing to explore new actions, and a 70% probability of choosing to utilize known and proven actions.

This article would analyze buffer size. The experience replay buffer is used to store experience samples of previously observed states, actions, rewards, and the next state of the agent. The experiment selected 10,000, 20,000, and 30,000 as parameter choices, as shown in Table 2.

In Table 2, it can be seen that with the continuous increase in buffer size, there has been varying degrees of growth in stability, memory usage, and training time.

From the perspective of precision rate, the precision rate with a buffer of 20,000 is 0.927, which is higher than the precision rate of the other two parameters. The discount factor (γ) determines the importance of future rewards. A larger discount factor places greater emphasis on long-term rewards, while a smaller discount factor places greater emphasis on immediate rewards. Based on different γ values, the model was tested for reward, learning speed, stability standard deviation, and exploration-exploitation balance, as shown in Table 3.

From Table 3, it can be seen that among the three different γ values, the comprehensive performance of the 0.69 interval is higher than the other two. From the perspective of obtaining rewards, the reward with an γ value of 0.69 is 2,763, which is much higher than the γ value of 0.99 and not significantly different from the γ value of 0.89. From the perspective of learning speed, the γ value of 0.69 is 192, the γ value of 0.89 is 172, and the γ value of 0.99 is 96. It can be seen that the γ value of 0.69 is higher than the other two; from the perspective of stability standard deviation, the stability standard deviation of γ value 0.69 is 0.18, the stability standard deviation of γ value 0.89 is 0.21, the stability standard deviation of γ value 0.99 is 0.34, and the stability standard deviation of γ value 0.69 is leading in the experiment. Exploration-exploitation balance also presents the same effect: the γ value of 0.69 is 0.30, the γ value of 0.89 is 0.35, and the γ value of 0.99 is 0.58. The experiment finally determined that the γ value is 0.69.

Epoch defines the number of times the entire training dataset is fully transmitted to the neural network model in the neural network model, and sets appropriate parameter ranges to achieve effective training and excellent performance of the model. The initial training frequency given in the experiment is 500 to avoid wasting resources and time caused by early training. After determining other complex parameters, this parameter can be adjusted and two different epochs can be set for experimentation.

During the process of Epoch from 500 to 1,500 and then to 3,000, both the training and validation sets exhibited similar curves in terms of loss value and accuracy. It can be clearly seen that with the increase in epoch, the trend of these two performance indicators gradually flattens out. Finally, the final epoch of 1,500 was selected for the experiment.

Based on the above tests for each parameter, the optimal parameters were determined in the experiment as shown in Table 4.

4.1 Experimental results

In this study, the trained DQN model not only demonstrated its effectiveness in color block painting and minimalist styles but was also applied to various other art styles to demonstrate its adaptability and versatility. In the experiment, the model was used to generate impressionist style artworks. After inputting classic works of impressionism, the model can output images with typical impressionist features, such as blurred contours, vivid light and shadow effects, and vivid color applications. These works reflect the unique handling of light and color by impressionists. Similarly, the model was also tested for its effectiveness in surrealist styles. Surrealist works are known for their dreamy scenes and illogical images. The model generated a series of imaginative and creative works in this style of testing, showcasing the unique dreamlike and irrational expression of surrealism. In addition, in the testing of abstract art, the model successfully created a series of abstract works that showcase the non-concrete aesthetics of abstract art through the free combination of color, shape, and line. This indicates that the DQN model can understand and mimic the core concepts of abstract art, i.e., conveying emotions and concepts through visual language.

This article would test the tested model and input different styles of painting works into the model. Through learning and testing the model, the task of program art generation can be achieved, as shown in Figure 1.

Figure 1

Generation of two styles.

In Figure 1, the left side shows the target image of the input model, and the right side shows the output image of the model. It can be seen that for the model trained in the experiment, after inputting art images of a specific style, the model would output results of a specific style.

The experiment also collected five DQN performance indicators for each of the two styles, and recorded the average reward for each experiment, which is the average reward obtained by the model in the environment. Convergence indicates whether DQN has successfully converged to a stable strategy. If it converges, it is marked as “convergent,” otherwise it is marked as “unstable.” The exploration rate records the probability of the model choosing random actions in each experiment, which gradually decreases with training; Q-value convergence indicates whether the Q-value has successfully converged during the training process. If the Q value converges, it can be marked as “convergent”; stability is used to evaluate the stability and reproducibility of DQN performance. If the performance is stable, it can be marked as “stable,” otherwise it is marked as “unstable.” DQN was applied in the process of artistic creation, and various methods were utilized to improve and innovate existing models. More complex and customized network architectures were used to better capture and simulate the subtle features of specific artistic styles. For art styles with complex textures and rich details, deeper network structures were designed and specially designed levels and activation functions were added to improve the model’s learning and generation capabilities on these details. Considering the impact of the exploration rate of the model on performance, the learning process was optimized by dynamically adjusting the exploration strategy. An adaptive exploration rate adjustment mechanism based on performance indicators was implemented to enable the model to conduct extensive exploration in the initial stage, and then gradually reduce exploration based on learning progress and performance improvement, in order to improve learning efficiency and stability.

In Table 5, experiment numbers 1–5 represent the evaluation results during the process of drawing color field painting styles. Experiment numbers 6–10 show the evaluation results during the process of drawing minimalist styles. From the entire table, it can be seen that the overall average reward remains between 2,800 and 3,300, with a minimum exploration rate of 0.01.

At the end of the experiment, three scoring items were selected: generation speed, interpretability measurement evaluation, and creativity analysis to obtain user satisfaction and feedback on the generated work. Among them, the generation speed measures the efficiency of the model in generating artistic works. This indicator is crucial for practical applications, especially in scenarios where a large number of works need to be generated quickly. An efficient model can produce high-quality works in a short period of time, which is an important measure for practicality and commercial applications. Interpretability measurement evaluation focuses on whether the artwork generated by the model can be understood and appreciated by human audiences. This indicator reflects the model’s ability to capture and imitate subtle differences and complexity in human artistic creation. An artwork generated by a highly interpretable model is more comparable to the works of human artists and is more easily accepted and appreciated by the audience. Creativity analysis measures whether a model can produce novel, unique, and creative works of art. This indicator is particularly important as it directly relates to the originality and innovation of artistic works. A highly creative model can break through the limitations of traditional thinking and style, generating works of art with innovative elements and unique styles. The experiment collected the evaluation scores of 100 viewers, with a maximum score of 100 points. The average values of these three items are shown in Figure 2.

Figure 2

Program art generation rating.

In Figure 2, the horizontal axis coordinates represent two different styles of art works (one is color field painting and the other is minimalism). From Figure 2, it can be seen that the generation speed, interpretability, and creativity scores of color field paintings are 83.2, 93.5, and 86.3, respectively. The generation speed, interpretability, and creativity scores of minimalism are 86.6, 91.5, and 82.1, respectively.

4.2 Discussion

From Table 5, it can be seen that the average reward of the experiment is around 3,000 under both stable and convergent conditions. However, it can be observed that the average reward fluctuates significantly under unstable convergence and unstable stability conditions. The minimum reward is 2,865, and the maximum reward is 3,275. At the same time, the exploration rate is higher than other groups, with a fluctuation of around 0.2.

In the ratings shown in Figure 1, the ratings for color field painting and minimalism are shown separately. From the perspective of generation speed, it can be seen that the former has a slightly lower score of 83.2 than the latter, but in terms of interpretability and creativity, the score of color field painting is higher than that of minimalism. In terms of explanatory indicators and innovation ability, due to the manually drawn dataset, there are limitations in novelty and innovation ability.

From the data in Table 5 and the ratings in Figure 2, it can be observed that there are differences in the performance of models with different artistic styles, which are mainly influenced by various factors. Color block painting and minimalist style have significant differences in visual characteristics. Color block paintings usually contain more details and color variations, which may provide DQN with richer learning materials, thereby making the model perform better in this style. In contrast, the minimalist style is characterized by simplicity and limited color usage, which may result in models facing fewer changes during the learning process, and may explain its relatively low ratings in interpretability and creativity. The different exploration rates also affect the performance of the model. A higher exploration rate means that the model attempts more new actions, leading to fluctuations in performance in specific tasks. In experiments, unstable convergence and stability are often accompanied by higher exploration rates, which leads to fluctuations in the average reward.

In this study, the advantage of the DQN model lies in its ability to learn and adapt to the characteristics of different art styles, generating works of art with high aesthetic value. By adjusting key parameters, the model can flexibly transition between different styles, demonstrating good adaptability and creativity. However, the model also has limitations. For example, high exploration rates may lead to unstable performance, while the model’s dependence on the dataset means that its creativity and novelty are limited by the provided training data. The aim of this article is to improve the model’s adaptability and creativity to different art styles by adopting more advanced network architectures and learning algorithms. Second, by enhancing and diversifying the training dataset, the learning range of the model can be expanded, improving its ability to generate novel and creative works. Finally, the exploration strategy of the model is further optimized to reduce performance fluctuations and improve the overall stability and reliability of the model while ensuring learning efficiency. Through these improvements, the application effect of the model in program art generation has been significantly improved.