User:NorwegianBlue/Kladd

Silent data corruption of video files

I've had an incident affecting my collection of photographs and videos that worries me, and that I hope someone here is able to help me understand. I had five videos (.3gp format) recorded on an HTC phone (HTC Incredible S S710e) bought January 2012. The videos were recorded May 2012. I recently discovered they were corrupt. Jpegs in the same directory appeared to be ok. Two other videos recorded with the same phone a couple of months later, which were stored in a different directory, were ok. Below (collapsed) is a summary of the troubleshooting I've done so far.

05.05.2012 17:25        50 019 626 VIDEO0002.3gp 01.10.2014 14:54        50 019 849 VIDEO0002_corrupt.3gp 05.05.2012 17:42         1 686 133 IMAG0237.jpg 01.10.2014 14:54         1 696 534 IMAG0237_modified.jpg
 * In my most recent backups, the files were corrupt, but I retrieved the intact files from a backup from June 2013, along with the jpegs that I had kept in the same directory.
 * I've found that the sizes of the corrupt videos are larger than the originals:
 * Although there appears to be no problem with the jpegs, I've found that they too have increased in size:
 * Using exiftool, I've found some differences in the metadata, both of the jpegs and the videos:


 * Videos:
 * A tag called "Handler Type" in the original is missing (or rather, moved, see below) in the corrupt file. It has the value "Video track" in the original.
 * The tag called "Handler Description" immediately follows in the original, and is untouched in the corrupt file (value: "VideoHandle").
 * Later in the corrupt file, the following section appears:

Handler Type                   : Metadata Encoding Time                  : 2012:05:05 18:52:54Z Media Class Secondary ID       : Unknown Content Media Class Primary ID         : Video
 * Finally, the movie data offset is different, 810040 vs 810263.
 * When the movie data offset is taken into account, the size of the binary video+audio data sections are exactly equal (49209586 bytes) in the original and corrupted files in the example. So I isolated the tails of the files, to check if the binary video+audio data were equal. They weren't. It is conceivable that they would have been identical if I'd had a way of assembling separate and continuous audio and video streams. But when compared as 49209586 byte binary files, they were totally different.
 * I restored the entire foto directory from the 2013 backup on a separate disk, and programmatically deleted all files that had identical md5 signatures to files in my current Photo/Video directory, and then examined the files that were left in the restored backup. I'm not through examining (it's a huge task), but I found one similar instance, in which .MOV files recorded on a Canon EOS 500 camera had been modified in the data section. The exiftool-readable stuff was identical. The mofified .MOV files were playable, and I noticed no visible or audible differences between the original and modified file. See below regarding the accompanying *.THM files, which really are small JPEGS.
 * JPEGS
 * The tag "Interoperability Index: R98 - DCF basic file (sRGB)" present in the original is gone in the modified file.
 * Later, there is a section "Padding: (Binary data 2060 bytes, use -b option to extract)" in the modified file, that is not present in the original.
 * Thumbnail offsite and length are increased in the modified file.
 * There is a section in the modified file:
 * There is a section in the modified file:

About : uuid:faf5bdd5-ba3d-11da-ad31-d33d75182f1b Date Acquired: 2013:01:26 00:12:31
 * That is not present in the original.
 * In the second observation mentioned above (modified but playable .MOV files from a Canon EOS 500), I had renamed the *.THM files to *.jpeg. These had also been modified. The most conspicuous changes were that Exif byte order had been changed from Little-endian (Intel, II) to Big-endian (Motorola, MM), and that the block
 * In the second observation mentioned above (modified but playable .MOV files from a Canon EOS 500), I had renamed the *.THM files to *.jpeg. These had also been modified. The most conspicuous changes were that Exif byte order had been changed from Little-endian (Intel, II) to Big-endian (Motorola, MM), and that the block

About        : uuid:faf5bdd5-ba3d-11da-ad31-d33d75182f1b Date Acquired : 2013:01:21 23:46:36
 * had been introduced. Note that the uuid is identical to the one above, and that the date is different. It is puzzling that the dates are close in time, but nevertheless before the undamaged backup was taken.
 * A web search for similar problems came up with two that resemble the problems I have experienced:
 * http://feedback.photoshop.com/photoshop_family/topics/_3gp_corruption_error_in_lightroom_4_beta
 * http://answers.microsoft.com/en-us/windows/forum/windows_7-pictures/windows-live-photo-gallery-corrupting-3gp-files/62c48718-4957-4566-b4f3-f1e93d933e2d
 * http://answers.microsoft.com/en-us/windows/forum/windows_7-pictures/windows-live-photo-gallery-corrupting-3gp-files/62c48718-4957-4566-b4f3-f1e93d933e2d

Working hypothesis: The troubleshooting clearly shows that this is not a case of random data corruption caused by faulty disks or cosmic rays, and I find it very unlikely that it is caused intentionally by malware. I probably at some point in time, have accessed the files with a program installed on my PC which silently modifies both their metadata and the way video and audio chunks are laid out in video files. My top suspect was XnView, which I use a lot. However, I've tried to reproduce the problem with XnView, but my currently installed version does not modify the files it accesses in its file explorer. Other candidates are Windows itself, with its image and video display functionality. I've tried accessing the files with the windows 7 image viewer and the windows Xp viewer (from a virtual machine), without being able to reproduce the problem. I suspect that the uuid referred to in the details of the troubleshooting, faf5bdd5-ba3d-11da-ad31-d33d75182f1b identifies the culprit. A web search for the uuid leads to various images, but does not appear to be strongly associated with reports of data corruption. I have not been able to identify the program associated with the uuid, a search at https://mikolajapp.appspot.com/uuid/ returns no results for {faf5bdd5-ba3d-11da-ad31-d33d75182f1b}. I have used regedit to search for the uuid in my Win 7 installation, and in the Xp virtual machine, with no hits. This does not exclude that it is a previous version of a currently installed program.

And that's where I am right now. The incident certainly feeds my paranoia. If I had known the reason, I would have been in a position to eliminate the problem.

Suggestions on how to proceed to reach an accurate diagnosis of the cause of the data corruption would be highly appreciated. I believe I have mentioned the possible suspects, but should add that I use Adobe Lightroom and Exiftool a lot, and previously used jhead and a Canon program for adjusting white balance etc of raw files.

In particular, it would be helpful if someone could find out what program the uuid faf5bdd5-ba3d-11da-ad31-d33d75182f1b is associated with.

Thank you in advance!

--NorwegianBluetalk 11:01, 19 October 2014 (UTC)