1 Introduction

Nowadays, images play a vital role on online social networks (OSN), since an image says more than a thousand words, and is not restricted by the barrier of language. However, the easy access to image editing software tools makes it extremely simple to alter the content of images, or to create new ones, so that material may be manipulated to serve personal ambition or propaganda, e.g. reusing past images from unrelated contexts, or explicitly tampering with media content. As a result, seeing is no longer believing [1].

Image forensics, which focuses on physical content, is an effective technique to detect whether images are modified. But many image operations are benign, performed for fun or for artistic value, such as visible watermarking in Fig. 1(a), and photo retouching and repair in Fig. 1(b). Most images on the social networks usually have been processed to enhance posting effect or resize, such as image enhancement, median filtering and JPEG compressing, all of which are regarded as doctored images in the field of image forensics, even though the essential content of these images are not changed. It is different from malicious tampering in rumor images. Therefore, how to differentiate innocent operation and malicious tampering is our main focus, that is to say, we want to evaluate the influence of image tampering.

Fig. 1.
figure 1

Two examples of benign modification. (left) An image with visible watermarking. (right) An old photo and its repair version.

Full size image

So what features are best for evaluating the importance of tampering content? The first is whether the virtual content is changed, i.e., whether the core and main content are changed. The saliency features are usually seen as important content of an image [2, 3]. So in this paper, based on image forensics techniques, we use saliency mechanism to evaluate the influence of image tampering. Meanwhile, we set some evaluation indexes and corresponding thresholds. If the index of altered content in the saliency map exceeds preset threshold, it is seen as malicious operation, otherwise, it is seen as innocent or uncertain.

In the rest of the paper, after introducing related works about information credibility and image forensics, and our motivation, we describe the proposed method about influence evaluation of image tampering content, and finally we present experimental results and discussion.

2 Related Work

Some microblogs and websites have the function of identifying false information, such as [4,5,6], and most are human-based. However, for today’s OSN big data, human-based method is not effective. Recently there have been many efforts to study information credibility in microblogs, ranging from automated algorithms to model and predict credibility of users [7] and messages [8,9,10]. However, with the exception of [11], little research has focused on image credibility. Also, there is yet no work on content-based image credibility. Image credibility verification on OSN is a fairly challenging problem, and it is based on image forensic technology. In the field of image forensics, there are many excellent algorithms to detect image operations, such as seam carving, copy-move [12,13,14], near-duplicate [15], median filtering [16], JPEG compression [17], and image composition [18]. However, most of these methods do not evaluate the degree of importance of tampered parts.

Fig. 2.
figure 2

Images and their saliency maps. (a) Original image; (b) retouching image; (c) malicious tampered image; (d)–(f) are saliency maps of (a)–(c).

Full size image

In Fig. 2, (a) and (c) are downloaded from Internet [19]. Obviously, the (c), which has been retweeted excessively [20], is the malicious doctored version of (a) and (b) is the benign retouching version of (a) and (d)–(f) are saliency maps of (a)–(c) by CA attention algorithm [3], respectively. We can see that innocent retouching alters the image saliency at a tiny scale, while malicious tampering alters the image saliency enormously. Therefore, we will evaluate the influence of image tampering via how much operations affect saliency content of an image.

3 Proposed Method

As shown in the previous parts, we will use saliency mechanism to evaluate the influence of image tampering. Firstly, we give assumptions of our model, then we present 3 indexes to evaluate the importance of modified content, and at last the evaluation pipeline is proposed.

3.1 Assumptions of the Model

The importance evaluation of image tampering is based on following assumptions.

  1. (1)

    The bigger area of modified content, the stronger influence of the content;

  2. (2)

    The more salient pixels of modified content, the stronger influence of the content.

3.2 Evaluation Indexes

Based on the assumptions in Sect. 3.1, 3 indexes are proposed in this part. Supposing we have an image of size \(M \times N\) which has been modified, p(i, j) is the probability of corresponding pixel value in position (i, j) and \(v_s(i, j)\) is the saliency value of the pixel in position (i, j).

  1. (1)

    Ratio of modified area \(R_a\)

    $$R_a = \frac{C'}{C}$$

    where \(C'\) is modified area, and C is original image area.

  2. (2)

    Ratio of saliency value per unit area \(E_s\)

    $$E_s = (\frac{v_s'}{v_s}) / R_a$$

    where

    $$v_s = \sum _{i,j}^{M,N}v_s(i,j)$$
    $$v_s' = \sum _{i,j}^{M,N}v_s'(i,j)$$

    \(v_s'(i,j)\) is the saliency value of modified region in position (i, j).

  3. (3)

    Ratio of saliency information amount per unit area \(E_{cs}\)

    $$E_{cs} = (\frac{I_{cs}'}{I_{cs}}) / R_a$$

    where

    $$I_{cs} = \sum _{i,j}^{M,N}I_c(i,j)v_s(i,j)\sum _{i,j}^{M,N}-log_2p(i,j)v_s(i,j)$$
    $$I_{cs}' = \sum _{i,j}^{M,N}I_c'(i,j)v_s'(i,j)\sum _{i,j}^{M,N}-log_2p'(i,j)v_s'(i,j)$$

    \(p'(i,j)\) is the probability of corresponding pixel value of modified region in position (i, j).

The tampering influence of an image is defined as a numeric value, and a threshold is used to determine whether it is benign or malicious. For index \(R_a\), the bigger \(R_a\), the stronger influence of the content. For indexes \(R_s\) and \(R_{cs}\), we set confidence value according to confidence level 95%. When the index is in interval [0, 0.95), the image is benign; when in [0.95, 1.05], uncertain; and when in (1.05, \(\beta \)], malicious, where \(\beta > 1.05\).

3.3 Evaluation Pipeline

Given an image, the evaluation pipeline is as follows:

  1. (1)

    Detecting modified regions by corresponding forensics algorithm;

  2. (2)

    Generating saliency map via a series of attention models;

  3. (3)

    Computing the indexes presented in Sect. 3.2;

  4. (4)

    Classification and analyzing the influence of image tampering.

4 Experiments

In this part, we use forgery detection dataset [14] to evaluate the performance of our method, and utilize 5 classical attention models (Itti [2], CA [3], GBVS [21], SIM [22], SUN [23]) to generate saliency maps respectively. First, we ask 6 forensic experts to assess the tampering images in the dataset and average the scores. Then we use coincidence degree between human-based assessments and evaluation indexes to test the effectiveness of the proposed method.

Fig. 3.
figure 3

Line chart of human assessment sore under sorted \(R_a\).

Full size image

4.1 Impact of Different Modified Area

In the experimental dataset, the \(R_a\) lies in the interval [1.3%, 13.71%]. Sorting \(R_a\) in ascending order, the human-based assessment score is shown in Fig. 2. According to Sect. 3.1 assumption, the bigger area of modified content, the higher influence of the tampering. However, as Fig. 3 shows, the line chart does not have a explicit tendency or direction. That is to say, in this dataset, there is no direct relationships between the influence of different tampering images and \(R_a\).

4.2 Coincidence Degree

The coincidence degree between human-based assessments and evaluation indexes using different attention models is shown in Table 1.

Table 1. Coincidence degree under different attention models.
Full size table

As shown in Table 1, in general, \(E_s\) and \(E_{cs}\) all have fine coincidence degree with human-based tampering assessments using these 5 classical attention models, and the index \(E_{cs}\) has higher coincidence degree than \(E_s\), and CA attention model is better than the other 4 attention models. Then we utilize these 5 attention algorithms on Fig. 2(a), (b) and (c), and get their saliency maps shown in Fig. 4. We can see that CA saliency map of the retouching image is most similar to that of the original image, and also could differentiate malicious tampering one.

Fig. 4.
figure 4

Images and their saliency maps.

Full size image

4.3 Different-Level Modification Within One Image

To further demonstrate the effectiveness of our proposed method, we alter an image with different modifications as shown in Fig. 5. There are 9 images with different modification from Fig. 5. Top left image (\(E_s\) and \(E_{cs} > 1\)) is gotten from dataset [14], other 8 images are modified with different number of ships by copy-move operation.

Then CA algorithm are utilized to detect saliency features, and corresponding line chart of indexes \(R_a\), \(\frac{v_s'}{v_s}\) and \(\frac{I_{cs}'}{I_{cs}}\) are shown in Fig. 6.

Fig. 5.
figure 5

Modified images with different number of ships.

Full size image

As Fig. 6 shows, when \(E_s\) and \(E_{cs} > 1\), the bigger \(R_a\), \(\frac{v_s'}{v_s}\) and \(\frac{I_{cs}'}{I_{cs}}\), the higher level modification and the greater influence. This suggests that we can assign an influence score via indexes to each image such that greater influence corresponds to a higher score. In the line chart \(\frac{I_{cs}'}{I_{cs}}\), there appears downward inflection points. The corresponding images shown in Fig. 6 are not easy to differentiate by human eyes at the first glimpse. That is to say, the modification is uncertain about innocent or malicious. Of course, this is just a specific case not covering all situations.

Fig. 6.
figure 6

Line chart of indexes \(R_a\), \(\frac{v_s'}{v_s}\) and \( \frac{I_{cs}'}{I_{cs}}\).

Full size image

4.4 Classic Doctored Image Testing

In this part, we select two famous doctored images whose semantic content has been heavily altered, and get their saliency maps by CA algorithm, as shown in Fig. 7. We can see that the tampering part (the dashed ellipses) is conspicuous in their saliency maps. That is to say, the saliency content of these two forgery images are both heavily tampered, and it testifies that saliency mechanism could evaluate the influence of image tampering.

Fig. 7.
figure 7

Two famous tampered images and their saliency maps.

Full size image

5 Conclusion and Future Work

Information with maliciously tampered images on OSN will encourage the propagation of rumors. Giving early alerts of fake news can prevent further spreading of malicious content on social media. Although image tampering analysis cannot solve all problems of rumor dissemination, it could address the issue of information with doctored images as early as possible.

In this work, we attempt to evaluate the importance of tampering content via saliency mechanism and achieve some results. The performance of the method is affected by human-based assessment with a prior knowledge, image content description, subjective factor and so forth. Also, whether attention models could catch the “saliency content” affects the experimental results. In the future work, we will study image tampering influence in the real situation, generating saliency map based on the context inspiring state-of-the-art caption algorithms and text analysis methods to connect image and text.