Benchmarking Scientific Image Forgery Detectors

Abstract

The field of scientific image integrity presents a challenging research bottleneck given the lack of available datasets to design and evaluate forensic techniques. The sensitivity of data also creates a legal hurdle that restricts the use of real-world cases to build any accessible forensic benchmark. In light of this, there is no comprehensive understanding on the limitations and capabilities of automatic image analysis tools for scientific images, which might create a false sense of data integrity. To mitigate this issue, we present an extendable open-source algorithm library that reproduces the most common image forgery operations reported by the research integrity community: duplication, retouching, and cleaning. We create a large scientific forgery image benchmark (39,423 images) with enriched ground truth using this library and realistic scientific images. All figures within the benchmark are synthetically doctored using images collected from creative commons sources. While collecting the source images, we ensured that the they did not present any suspicious integrity problems. Because of the high number of retracted papers due to image duplication, this work evaluates the state-of-the-art copy-move detection methods in the proposed dataset, using a new metric that asserts consistent match detection between the source and the copied region. All evaluated methods had a low performance in this dataset, indicating that scientific images might need a specialized copy-move detector. The dataset and source code are available at https://github.com/phillipecardenuto/rsiil.

Appendix

Computer Science and Digital Forensics Glossary

• 1. Algorithm Library: A collection of algorithms and routines used by computers to run a software.

• 2. Benchmark: The process of assessing the performance of a method using a standard dataset and metric.

• 3. Brute Force Algorithm: Solving a problem trying each possible solution with exhaustive search.

• 4. Color Histogram: The frequency representation of each pixel color intensity in an image.

• 5. Computer Vision: An interdisciplinary area from Computer Science dedicated to inferring, understanding, and producing digital images and videos using computational intelligence.

• 6. Dataset: In the scope of this work, a collection of images.

• 7. Deep Fakes: Synthetic and realistic images generated by an Artificial Intelligence.

• 8. Detection Map: A map informing, for each pixel from an input image, whether the pixel is doctored or not according to a tampering detection method.

• 9. Detection Probability Map: A map related to an image that indicates the probability of each pixel being doctored.

• 10. False Negative: In the tampering detection scope, the result of predicting a forged data as pristine.

• 11. False Positive: In the tampering detection scope, the result of predicting a pristine data as doctored.

• 12. Ground Truth: The annotated information from data. In this work, we also use this term as the annotated map that pinpoints the location of each doctored pixel from an image.

• 13. True Positive: In the tampering detection scope, the result of correctly predicting a forgery.

• 14. True Negative: In the tampering detection scope, the result of correctly predicting pristine data.

• 15. JavaScript Object Notation (JSON): A typical file format in Computer Science for sharing object data in a human-readable text.

• 16. Metadata: A data description providing information regarding its storage, content, and creation.

• 17. Model Fine-tuning: The process of performing small adjustments in a pre-trained model to improve its performance for a specific task.

• 18. Model Training: The phase of an Artificial Intelligence algorithm in which a model learns how to accomplish a task.

• 19. Object-Mask or Object map: A mask/map that pinpoints the location of each foreground object within an image.

Evaluation Plots and Output Results

This section presents all scores organized in radar plots from the copy-move forgery detection benchmark. We also present output samples from all methods for each evaluated modality.

Fig. 16

a CMFD Simple Figure Evaluation. Busternet has the best average performance for Simple Figures because its polygon contains almost all the others (only OVERSEG is not contained by Busternet’s polygon). The shrinking polygons area from $\text{ F1-score}_{TP}$ to $\text{ F1-score}_{CTP}$ indicates that all methods show inconsistencies in their detection map. All metrics in percentage. b CMFD Inter-Panel Figure Evaluation on different DVS. In this plot, each method fares differently for each modality. The polygons from SIFT-NN and SURF-NN have a larger area than the other methods, indicating their robustness to more operations than the other methods. The shrinking polygons area from $\text{ F1-score}_{CTP}(DVS-1)$ to $\text{ F1-score}_{CTP}(DVS-3)$ indicates that, the higher the caption degree of visual signs, the lower the method’s effectiveness will be. All metrics in percentage. c CMFD Intra-Panel Figure Evaluation on different DVS. All methods show low performance and concentrate on the center of the radar, which indicates this is a challenging modality. The best method in this modality is OVERSEG with scores lower than 3.0 for the types of evaluated forgeries. All metrics in percentage. Evaluation Baseline Results. Within the parenthesis of each copy-move modality, there is the transformation used during the copy-move forgery. All F1-scores presented in this figure are in percentage format. The best result for each duplication modality is indicated with the color of the respective detector (e.g., in the left plot of a the value 16.83 relative to Copy-move (Random) is colored with Busternet’s color (orange), indicating that Busternet is the best method in this modalilty). a Result for Single Figure Evaluation using $\text{ F1-score}_{TP}$ and $\text{ F1-score}_{CTP}$. b Result for Inter-Panel Figure Evaluation using $\text{ F1-score}_{CTP}$ across all degrees of visual signs, indicated by the number in its subtitle (i.e., DVS1 for degree of visual signs equals to one). c Result for Intra-Panel Figure Evaluation using $\text{ F1-score}_{CTP}$ across all degrees of visual signs

Figure 16 presents a radar graph visualization based on the evaluation tables in which the forgery modalities are arranged in the radius axes. Each CMFD methods’ result is represented by a different color in the radar chart. In this visualization, we inserted the score of each method along the modality axis (e.g., copy-move with flip), which start from the radar center (score zero) to its border (highest score); thus, the farther the method point (colorpoint) is from the center, the better the method for the axis copy-move modality will be. After inserting all points of a method for each copy-move modality, we connected those points, which results in a polygon. The larger the polygon area, the better the method’s robustness among different forgery modalities. Also, comparing each detector’s robustness to the operations, this visualization type helps identify possible complementary behaviors among different methods. As an example, consider Fig. 19 left panel. In this case, we have five modalities being compared (e.g., Copy-Move with Flip, Cleaning with Brute-Force, and Copy-Move with Translation). This chart shows the results of seven methods represented by each polygon color (see legend on the right). The best method in this figure is OVERSEG (in red), while the two worse methods (HFPM and SIFT-NN) are in superposition at the center (smaller areas).

Evaluation I: Simple Forgery Figures

Figure 16a shows the evaluation of Simple Figure Forgery Detection with $\text{ F1-score}_{TP}$ (left plot) and $\text{ F1-score}_{CTP}$ (right char), all scores in percentage (i.e. within a range from zero to one hundred).

Figure 17 presents output detection samples for all methods applied to the evaluated Simple forgery modalities. This figure includes $\text{ F1-score}_{CTP}$ of each detected map next to each method’s name in the figure.

Evaluation II: Inter-Forgery Compound Figures

Figure 16b shows the evaluation of Inter-Forgery Compound Figure Detection using $\text{ F1-score}_{CTP}$. In this modality, the radar visualization helped us observe some complementary performance among the chosen detectors. For instance, SURF-NN and Zernike-PM show a complementary behavior to copy-move with rotation and retouching. The flipped copy-move, Splicing, and Overlap forgery proved the most challenging forgeries. In addition, the visual signs are shown to have a noticeable impact in this scenario, reducing by up to seven points from degree 1 to degree 3 for some detectors.

Figure 18 presents output detection samples for all methods applied to the evaluated Inter-Panel forgery modalities with degree of visual sign equals to one.

Evaluation III: Intra-Forgery Compound Figures

Figure 16c shows the evaluation of Intra-Forgery Compound Figure Detection using $\text{ F1-score}_{CTP}$. In this modality, the detectors scored lower than three percent on $\text{ F1-score}_{CTP}$ for all evaluated operations. In this figure, the polygon from OVERSEG contains all the others, indicating that OVERSEG had the best performance in this modality.

Figure 18 presents output detection samples for all method applied to the evaluated Inter-Panel forgery modalities with degree of visual sign equal one (Fig. 19).

Fig. 17

Comparative Simple Forgery duplication detection output per modality. The purple color represents a pristine/non-suspect region and each other color in the ground-truth and detection maps represents a different ID assigned to each object and its copies. Inside the parenthesis of each method, we insert the $\text{ F1-score}_{CTP}$ metric in percentage

Fig. 18

Comparative Compound Inter-Panel Forgery duplication detection output per modality. The purple color represents a pristine/non-suspect region and each other color in the ground-truth and detection maps represents a different ID assigned to each object and its copies. Inside the parenthesis of each method, we insert the $\text{ F1-score}_{CTP}$ metric in percentage

Fig. 19

Comparative Compound Intra-Panel Forgery duplication detection output per modality. The purple color represents a pristine/non-suspect region and each other color in the ground-truth and detection maps represents a different ID assigned to each object and its copies. Inside the parenthesis of each method, we insert the $\text{ F1-score}_{CTP}$ metric in percentage