News/One Bad Ass Hedgehog - Shadow the Hedgehog

Hidden Palace and Last Minute Continue PROUDLY present...

Shadow the Hedgehog (May 27, 2005 prototype)
Shadow the Hedgehog (Sep 15, 2005 prototype)
Shadow the Hedgehog (Sep 22, 2005 prototype)

More Gifts and Stocking Stuffers:
Billy Hatcher and the Giant Egg (Jan 12, 2006 prototype)
Sega Classics Collection (Jun 29, 2005 prototype)
Sonic Gems Collection (Jun 21, 2005 prototype)
Sonic Gems Collection (Jul 4, 2005 prototype)
Sonic Mega Collection Plus (Aug 17, 2004 prototype)
Sonic Riders: Zero Gravity (Sep 1, 2007 12.26 prototype)
Sonic Riders: Zero Gravity (Sep 1, 2007 12.40 prototype)
Sonic Riders (Dec 2, 2005 prototype)
Sonic's Schoolhouse (May 16, 1996 prototype)

Discuss this release on our Discord server!
And check out the Twitch stream as it happened!

Hello everyone! It’s that special time of the year again, where we celebrate the holidays with yet another yuletide collection of prototypes. Apparently, we weren’t very well behaved this year because Santa decided to drop off a couple of stinkers down the chimney! Keeping with the holiday spirit, we can think of no better way to spread the holiday cheer than by sharing our Christmas load with the lot of you. May God have mercy on all of your mortal souls!

Courtesy of our friends at Last Minute Continue, we present the E3 prototype of Shadow the Hedgehog for the Nintendo GameCube! Aside from some other goodies, we have a proper dump of “Beta 4” of Shadow the Hedgehog as well! Ow the edge!

In all seriousness, we know this one has been a long time coming. Circumstances that were a little out of our control and the rabbit hole we ended up falling into led us down the path to really take our time with this one. We hope the wait was worth it. The story that led to the ultimate preservation of this prototype and the others that were offered to us is an extremely long one that has (un)fortunately brought us to today. Yes, it’s going to be one of those articles, so sit tight, grab a hot chocolate and a semi-automatic rifle, and enjoy the look back at one of the more bizarre moments in Sonic Team history.

Spiraling After the Fall

It’s difficult to decide what to think of Sonic the Hedgehog these days. You could make an argument that Sonic games have been less than stellar longer than there have been momentous successes. He’s been around for a while; over the course of over thirty years, he has over 100 titles under his belt. Like any other character that has been around for that long, inevitably, Sonic games would eventually see some misses. That’s not to say that Sonic was unfamiliar with failure even during a time when he was at his most popular, but there’s definitely at least some consensus as to when the series noticeably took a downturn for the worse.

That fateful day in January of 2001, when Sega would announce its plans to go third-party, would be a very big turning point for the company brand. When Sega had the Dreamcast, they had their own playground to do what they pleased and to have full control over the kind of experiences they wanted to offer to their audience. When Sega became a third-party company, choosing to publish and develop titles for other hardware manufacturers rather than create hardware to self-publish on, they were no longer in the captain’s chair. They had to find themselves answering to a lot more people than they had before. Despite their promise to ensure that Sega’s capability of designing and publishing great software would be uncompromised going forward, as the years would pass, that promise would be underdelivered at best.

That’s the one thing that isn’t always obvious. A lot changes when a company has to shed part of its identity to continue to exist. Many people walked out in the months following the third-party announcement, many of them who were the creative individuals who were responsible for decisions that created some of the most memorable games on Sega’s platforms. The resources to experiment and create those experiences dwindled, and it would become much harder to create newer gameplay experiences just for the fun of it (do you think Sega would’ve created a game like Seaman in 2003?). Sega couldn’t hold on to the identity or the momentum they had carried with them through the Dreamcast’s short life. That, combined with the need to make money to pay off many millions in debt, meant that there were many things the company just simply couldn’t take for granted anymore.

In some ways, what happened to the Sonic franchise was a direct representation of what was going on with Sega at the time. With the upcoming Sega Dreamcast, Sega had a resurgence of creative energy with a desire to spend it on as many innovative venues as possible. You could see that straight away with the original Sonic Adventure, possibly the most ambitious game Sonic Team would ever produce. Sonic Adventure was a hyper-creative moment for Sonic Team. Instead of simply translating the traditional 2D side scrollers into 3D or taking inspiration from the likes of Super Mario 64, Sonic Team decided to up the ante across the board. Sonic would get a radically different redesign with voice acting, an attempt at cinematic presentation with high stakes story telling, embracing new technology such as internet connectivity and the VMU, implementing their own take on Tamagotchi and Pokemon through the creation of the A-LIFE Chao Garden system, filled with a large world that can be explored with six different characters while listening to possibly one of the most diverse soundtracks ever created for a video game at the time. The Dreamcast had to be different; it couldn’t just be iterative. It needed to be different. And so Sega decided to be different through its software. And it was undeniably a wonderful time for everyone in the audience.

Then Sonic Team branched off and decided to tackle Phantasy Star Online, one of the world’s first console online role-playing games (at least one that can be considered successful and recognizable). While they were being innovative with creating new experiences with online play, the other half of Sonic Team decided to move back to California to create the spiritual successor to Sega Technical Institute (STI) by forming Sonic Team USA. While there, they decided to develop a direct sequel to Sonic Adventure in the form of Sonic Adventure 2. Not satisfied with simply creating a “bigger” offering that was the same as its predecessor, the newly formed Sonic Team kept the creative juices going and decided to keep innovating and rethinking how it would go about presenting its gameplay, story, and aesthetics.

But then - something changed. When Sega decided to throw in the towel on being a hardware company, there was something somber hanging in the air at the time. While Sega continued to release titles on the Dreamcast for about one more year, the announcement that Sega would go third-party happened to be right near the end of development on Sonic Adventure 2. That feeling of whether or not there would be another Sonic game after Sonic Adventure 2, at least to the standards we were becoming familiar with from the company at the time, was a very real one. Nevermind the idea of a Sonic game on a Nintendo platform of all things, the idea that Sega simply wouldn’t be able to produce high-quality Sonic titles anymore was definitely a fear from many fans who, at the time, literally grew up with the series when it was first released in 1991.

Sonic Adventure 2: Battle and Sonic Advance would eventually be released on Nintendo platforms, to much a Sega fanboy’s chagrin. But nonetheless, Sonic Adventure 2 ended up being a high note for not just the Dreamcast but also Sonic Team and Sega. Throughout 2002, many of the titles that were originally in development for the Dreamcast were eventually released on competing platforms. Just as Sega had promised at the time, the software was superb and fun to play - from the great Super Monkey Ball series on the Nintendo GameCube, Virtua Fighter 4 on the Sony PlayStation 2, and a whole offering of many titles on the untried Xbox from Microsoft. For many of us Sega fans, this was a relief that even though Sega wasn’t necessarily going to play king of the hill anymore, they could still produce titles that could hang with the big boys. If they can keep that momentum going, then maybe getting out of the whole hardware business was going to work out after all.

But there was one issue - where was Sonic? And more importantly, what can we expect from the company to succeed Sonic Adventure 2?

The year 2002 came and went without a word of what was to come. When Sonic fans entered 2003, believe it or not, at this point, this was proving to be one of the longer waits for a new Sonic title, at least not for a very long time. In time for E3 2003, Sonic Team would finally break their silence and announce their latest title since Sonic Adventure 2, just two years prior, called Sonic Heroes.

First impressions of the game were promising, a game that was seen as a bit of a return to form. Takashi Iizuka, the game’s director, would go on to state that the game would be created to appeal to a much wider demographic in comparison to previous titles. The game would feature a wide selection of characters by introducing a team-based game mechanic, ranging from the usual Sonic, Tails, and Knuckles, to characters fans haven’t seen in almost an entire decade (the Chaotix). Two new characters, Cream the Rabbit and E-123 Omega, were introduced to balance out the roster so that there’s something for everyone, both new to the franchise and old. The gameplay itself, while it utilizes the team mechanic at its core for all characters, is still relatively traditional, featuring mostly platforming with the occasional breaks in combat. The game follows a traditional two-act structure per stage, with a wide variety of colorful stages ranging from ocean views, forests, metropolises, canyons, and haunted houses. The soundtrack continues in the sound direction found in the Sonic Adventure games, using a combination of different music genres to flesh out the soundtrack once again. Ultimately, Sonic Heroes at the very least proved that the talent was still strong at Sonic Team, and that they still could create entertaining software (unless you played the Sony PlayStation 2 port).

However, Sonic Heroes can also be seen as something of an underwhelming disappointment, depending on how you look at it. The game’s story takes a significant backseat to everything else, something that had a lot more thought put into it in the past titles from the Adventure series. While the team mechanic was something relatively new to the series, the game overall was built on things that had already been established before. There was no internet connectivity, no Chao Garden, and not even any innovations to speak of – something that Sonic Team was more willing to embrace when they were still playing in their own sandbox with the Sega Dreamcast. There isn’t even a lot of replayability outside of getting ranks in the stages, which result in the player ending up with absolutely nothing in return. Sonic Adventure 2 gave you a 3D version of Green Hill Zone to unlock and more player skins for the versus mode, for crying out loud! There’s nothing to gain from getting any of the emblems in this game! People were starting to think that Sonic Team had run out of ideas.

That’s not to say Sonic Heroes was an “awful” game, but when you consider what had come before from the same developers, you can’t deny that if you were there for the game at the time when it was new, it was very underwhelming. It didn’t push anything forward other than having a wide appeal. In a sea of Sonic titles made today, it’s not easily apparent that the game is necessarily bad. But at the time, people's expectations were a lot higher. Sonic Heroes was definitely the first Sonic title that seemed to indicate that Sonic Team decided to start looking backwards, rather than forwards. It was around this time that the developers began having concerns about hoping “older” as well as “new” players would be able to pick up the game and enjoy it, while iteratively retaining what seemed to work in the past and throwing away what didn’t, rather than develop on an entertainer’s instinct.

Many of the people in the audience at the time had grown up with the franchise since its inception, its inception, mind you, includes some of the best games ever released. There was no “Sonic Cycle”, there was no threat of another “Sonic 2006”, at least not yet. But when Sonic Heroes came out, it was the first time in a while that a noticeable downgrade had taken place. Despite Sonic Heroes selling well (especially on the GameCube), it was vastly underrated by the gaming press at the time. It’s not a bad game, it just wasn’t all that it should’ve been or could be, especially in comparison to its predecessors, which had been released just a few years prior.

But perhaps that was just a product of circumstance. After all, Sega and Sonic Team had to go through a lot of sudden changes in a very short amount of time when Sega decided to quit the hardware business. To be able to create anything after almost an entire corporate restructure in about 3 years is nothing short of a miracle. So Sonic Heroes, despite whatever shortcomings you might gather from it at the time, could easily be forgiven. After the release of Sonic Heroes, the only question that was on everyone’s mind was, “So what’s on the horizon next?”

Sonic Team USA would eventually become rebranded as Sega Studios USA. Sega Studios USA began work on the studio’s next game as a follow-up to Sonic Heroes while Sonic Team in Japan was busy working on Phantasy Star Universe and supporting the release of Phantasy Star Online: Blue Burst. Most of the people who had worked on Sonic Heroes and Sonic Adventure 2 would continue to work on the next game. Takashi Iizuka took up the director’s chair once again with Jun Senoue as the game’s sound director and Yuji Naka as the producer. The game would take itself back to the style and theme of Sonic Adventure 2, returning to a slightly more mature setting and story.

The initial render of Black Doom. By Barret Meeker.

According to an interview Iizuka gave with IGN before the game’s release, the upcoming project got started when the figureheads at Sega/Sonic Team wanted to expand the Sonic franchise by offering a more mature game with features that were never included in a Sonic game before. Coupled with the fact that Shadow the Hedgehog was a fan favorite and that Iizuka himself wanted to create a game centered around Shadow, it was a perfect opportunity. In the same interview, Iizuka mentions that while there are a variety of different mediums they drew inspiration from, they found inspiration from films such as Underworld, Constantine, and the Terminator series while creating the concept art for the game. “The goal is to match the graphical world design with the dark mysteries that surround Shadow”, says Iizuka in an interview with NOM UK Magazine.

Eggman's base. Concept art by Barrett Meeker.

The game is a sequel of sorts to Sonic Adventure 2. Shadow the Hedgehog, who suffers from amnesia, still has trouble remembering the past and why he came into existence. While trying to figure out his reasons for existing, a race of aliens known as the Black Arms appears from out of nowhere, terrorizing the city under the leadership of Black Doom, who seems to know Shadow and makes allusions to a promise Shadow had made to him in the past. With nothing to go on but this unknown figure who apparently knows him, Shadow begins his quest to figure out his identity. The game has multiple different routes that the player can take that result in up to ten different endings, which must be unlocked before the true ending can be seen.

The goal of the game is to complete each of the three objectives in each stage to get on a path that leads to one of the ten endings. The game’s mission system is based on a “morality” system, in which Shadow has the choice to do something purely good, purely evil, or something for his own purpose. Some endings can only be unlocked by following a specific path, so this gives an incentive for the player to keep replaying stages by doing different missions to advance down a different path with its own set of stages. Not being satisfied with just being a traditional platformer like the original Sonic the Hedgehog games, Shadow the Hedgehog mixes platforming with third-person gun play in the style of Jak 2 and 3. Aside from guns, there are debris, sticks, poles, and other kinds of weapons that Shadow can use to aid in his quest. Shadow can also ride in vehicles on some stages, with the ability to interact with his environment. However, unlike previous games, this is the first game in a very long time in the franchise in which you only control one character with one particular playstyle for the entire game.

The game would be the very first to feature some fantastic animated cutscenes by Blur Studio, which is undeniably one of the standout parts of the game. The cutscenes were so good that Blur Studio would not only be the go-to studio for multiple video game studios for prerendered cutscenes for game franchises such as Star Wars and Call of Duty. But, they ironically would go on to provide the animation for all three Sonic the Hedgehog movies, including the Knuckles mini-series! Out of everything to come out of the production of this game, this is oddly the biggest success story.

The upcoming game would also share in a round of controversies regarding the decision to replace all of the English voice cast who became synonymous with the series since they started actively giving Sonic a voice in Sonic Adventure. The original cast who had stuck with the series since Sonic Adventure were replaced with the voice cast from the 4Kids English dub of Sonic X. The decision was believed by fans to have been likely done when the original voice actor for Dr. Eggman, Deem Bristow, passed away in early 2005. However, it’s also possible that it was an attempt by Sega to bring consistency in the voice cast between the animated show and the video games. The result of this decision led to a huge amount of backlash from fans over the release of the game, with petitions to Sega hoping to reinstate the original voice cast to reprise their roles in the upcoming game. Unfortunately, the true reason for the decision to replace the cast is unknown.

Thankfully, it wouldn’t be long before the world would witness what would become the previous game’s successor. On March 8, 2005, in celebration of Sonic the Hedgehog being inducted into the “Walk of Game” (a take on Hollywood, California’s Walk of Fame, which was hosted inside the Sony Metreon entertainment shopping center), Sega decided to celebrate Sonic’s 14th anniversary with the announcement of Shadow the Hedgehog, slated for release in “Holiday 2005”. "I want to thank you all very much for this honor. I'm happy about Sonic's success, but I'm even more excited about the future," said Takashi Iizuka, before showing off…this…

To give you an idea of just how utterly bizarre this announcement was, please take a look at the announcement trailer above. The trailer begins with a wholesome romp through Sonic’s early history with clips from some of his games like Sonic 2, Sonic Adventure, and Sonic CD. Then, out of the blue, a rain of bullet holes fills the screen to the sound of shattered glass before cutting to a shot of Shadow the Hedgehog walking toward the camera while holding a firearm. The trailer continues with scenes of Shadow shooting up monsters in a drab, dark, and gritty world and causing destruction everywhere he goes. Sometime into the trailer, the game’s marketing catch phrase, “Hero or villain? You decide," is revealed. And, to top it all off, a cut to Shadow facing the camera once again before he fires his handgun directly into the camera, right into the audience’s face.

Words cannot describe what people were feeling at the time. People legitimately thought it was an April Fool’s joke, and given the context of what the gaming landscape was like at the time, you would’ve thought so too. People also thought that because the game had guns and vehicles in it, it was going to take more inspiration from Grand Theft Auto than the released game would actually contain. We know that for some of our readers it’s difficult to place oneself back in an era that one was never directly part of. But you have to remember that the context in which a game like this is released to the ultimate reaction from the audience who had stuck with the franchise during a time when it wasn’t presenting itself like the way it did was a bit of a shock, which is an understatement.

Around the time that Sonic Heroes was wrapping up development, a very particular game had arrived and caused quite a stir in the scene for having many moments of ultraviolence in it. Grand Theft Auto 3 (Vice City and San Andreas by extension) were some of the most influential games to be released that generation, and gave many developers and producers the assumption that gamers wanted more “mature” games that dealt with more heavy-handed matters like gun violence and crime. But it wasn’t the fact that the games were violent and mature that made them fun; it was that it was fun to drive around role-playing as a criminal in a huge open world that was modeled after an exaggerated interpretation of real life. But for some reason, it was the violence and grotesqueness that caught on and gave other developers the impression that gamers wanted more “mature” games for their “mature” tastes.

The thing with video games that are runaway successes is that they give developers and producers superficial ideas of what the consumer “wants” to play, which they then go on to copy. When Final Fantasy VII came out and was a huge success, for instance, it signaled to a lot of developers that RPGs and narrative-based games were the thing to capitalize on. When Metal Gear Solid came out, people in charge thought that the audience wanted more games with a cinematic flair. When Pokémon came out and was a huge success, everyone wanted to draw some inspiration from the monster battling/capturing/raising mechanics. This isn’t necessarily a bad thing, as there are a lot of great games that were released that drew inspiration from the popular games that came before it.

But Shadow the Hedgehog is one of many other games released around that time that wears its influences on their sleeves with almost no shame. It wasn’t enough to have a game in a similar style to Sonic the Hedgehog, just with Shadow in it and a continuation of the Adventure 2 story. Oh no. Shadow the Hedgehog has to have guns, vehicles, and profanities in it for the kiddies. Because that’s bad ass, and that’s what the kids want. How else could Grand Theft Auto 3 and Vice City sell so well? A marketing savant could claim that “it’s not that the games themselves were fun, it was fun specifically because it had guns and vehicles with a hint of ultraviolence in it”. Of course! After all, that’s what people wanted to do in 2004-2005. Just run around in a car and shoot things. Therefore, every game had to have some element of that for it to be deemed worthy of being made.

Shadow the Hedgehog was released on various dates in November 2005, depending on the territory and system. The game would sell around 1.45 million copies in overseas sales outside of Japan in its first year, and roughly 2 million copies worldwide by 2007. This was enough to justify the game getting later platinum/greatest hits rereleases, but unfortunately, not enough to warrant any ports of the game on future systems besides a release on the PlayStation Store as a downloadable PS2 Classic in Japan and somewhat compatible with the Xbox 360’s backwards compatibility program with original Xbox titles.

Despite being a relatively decent success for Sega, they had decided to deem this endeavor a failure. Critics across almost every media outlet refused to give the game a rating anywhere near even a middle grade. Critics rightfully derided the attempts at making a more mature interpretation of Sonic as something “painfully dumb”, as Game Informer staff writer Matt Helgeson said. Critics weren’t convinced that the majority of people, let alone long-time Sonic fans, would find anything appealing about the game to mention as a legitimate selling point. While the act of firing with weapons and running around is fun, the game’s controls, mission management system, and level design are extremely frustrating. Most missions are some variant of “kill and destroy X of something”, “get to the end of the stage”, or “collect X of something”, but the level design doesn’t encourage players to willingly want to backtrack all the way to the beginning of the stage to find something they missed (WHICH IS VERY EASY TO DO WE MIGHT ADD). Why they took out the radar from the final game, we’ll never know.

The best way to describe Shadow the Hedgehog is “try hard”. There’s no doubt that Sonic Team wanted to create a Shadow the Hedgehog game with interesting mechanics and a more mature focus in comparison to its predecessors. The issue comes from its execution and implementation of the ideas themselves, and most importantly, how it not only compares with its predecessors but also with its contemporaries. A lot, and we mean, a LOT of games released around this time were aiming for the mature audience vibe by introducing ham-fisted attempts at making their games more mature by introducing elements that were synonymous with the concept of being mature (no thanks, mostly in part because of the Grand Theft Auto series). From what we’ve seen from development screenshots and prototype builds before the final release, someone really felt the need to focus solely on what could be gotten away with and what was deemed too “edgy” in execution. Apparently, for instance, Shadow the Hedgehog wondering where that “damn” fourth Chaos Emerald was very important to get across, but saying the word “hell” in an opening cutscene and having aliens have red blood was just too much! That’s not to say that you can’t have a Sonic game with gun play and swear words in it, but when every other platform game released at the time felt the need to do the same thing, it’s hard to believe that there wasn’t at least one person during development that was too focused on what was being released on store shelves rather than think outside the box and come up with something at least somewhat original.

In hindsight, that’s what made the release of Shadow the Hedgehog ultimately frustrating for long-time fans of the franchise. When Sonic was born, the series was a trend setter - pushing the technological envelope while releasing really fun and appealing games that were so influential that it was able to catapult a relatively unknown company into having a good chunk of market share for a brief moment in time. Now, it became clear to everyone that from this moment on, Sonic was officially in the market of being a trend follower, which doesn’t carry the same prestige as it used to have. It was embarrassing, if not a little sad. This seems to have been around the time when Sonic Team/Sega became more concerned with not just their contemporaries but also with their own past, by drawing more and more from Sonic’s legacy rather than their own ambitions as time goes on. If Sonic Heroes wasn’t something people necessarily wanted, then Shadow the Hedgehog unequivocally wasn’t it either.

But hey, Shadow the Hedgehog might not have been very good. The next-gen consoles are upon us. Surely, this time, Sonic Adventure 3 is finally on its way, and it’s going to be grea-

…oh no

The Last Chance

Sometime in 2022, Last Minute Continue acquired a few items that were rescued from an industrial skip destined to be thrown out. Among the items were a collection of discs of various builds from games dating from 2004 to 2005 for various consoles. The discs hadn’t been used in almost two decades and were difficult to read. The discs that might have been able to be salvaged, particularly the NR development discs to be played on GameCube development hardware, couldn’t be dumped by them, as they lacked a NR reader at the time. Aside from the discs that could’ve been read if given the right equipment, the discs that were nearly undumpable were set aside.

We were offered a chance to take a crack at dumping them sometime in 2023. When we assessed what we were sent, we noticed that most of the discs that were difficult to read were possibly the result of bad burns, rather than disc rot, as was originally expected from nearly 20-year-old recordable media. This doesn’t necessarily mean that the discs are unreadable; it’s just that most drives would have a hard time reading the discs, at least initially. And, it would take many, many, many attempts and different drives that could read the discs in different ways in order to pull as much data off of them as possible.

Coincidentally, when Last Minute Continue originally acquired the discs, we were doing research into reading Dreamcast GD-R discs without the need for original hardware and a System Disc 2. In the pursuit of figuring this out, we acquired many drives both from our own expense and also through many generous donations from friends on social media and within the community! We have a ton of Plextors (even some Premium2s!), M-DISC drives, drives from almost every vendor that made them, and even some odd balls to try and discover the edge cases. Aside from a computer hardware recycler, we had the largest amount of diverse optical disc drives known to man!

So we got to work. Most of the discs were actually easy to read, given the right drive, and maybe just a few retries and a little bit of patience. Thankfully, they didn’t put up much of a fight…save for one disc…

When Last Minute Continue possessed the discs, they released at least one prototype from the lot - “Beta 4” of Shadow the Hedgehog for the Microsoft Xbox. The disc was among the ‘undumpable’ discs that couldn’t be read perfectly by any of the drives that Last Minute Continue had on hand. In an attempt to at least salvage the prototype, the original dump was made available, and an attempt to Frankenstein together data from other versions to try to create a playable version was made, and was eventually released to everyone later during the year.

Unfortunately, while the game was playable, huge swaths of data had to be replaced using data from the only other available release - the final version. So there was a chance that more content that was different in Beta 4 in comparison to the final build would be permanently lost. Because the disc couldn’t be dumped properly, there would be no way for the dump in its current state to be DAT’d into a data set like Redump or TOSEC - even if DiscImageCreator was originally used to make the dump in the beginning.

When we assessed the disc for ourselves, we thought that it would be a similar story to most other hard-to-read discs. You just have to find the right drive for the job! The general rule of thumb with recordable discs is that it’s always a good idea to use the same drive that burned the disc to read the disc. While it probably was a little out of the question that we would possess the same model of DVD drive used to burn the disc, we could probably get away with using a similar drive from either the same vendor or the same time period.

Surely out of the 200 drives (and counting) that we had in our possession, one of those drives could read the disc at least somewhat perfectly, right? …right?

The Recovery Effort

Ohhhhh boy. Where do we even begin with this one?

This was the reason this release took so long. To say we were down a rabbit hole when we tried to get this disc completely dumped would be a massive understatement. This disc took us countless hours, days, weeks, and months to make at least some semblance of progress. And whenever we got close to some progress, we would hit a standstill and would have to rethink our strategy or come up with some solution that didn’t previously exist to progress just a little bit further. The amount of work that went into just this one disc is enough to make anyone blush. You think you've got what it takes to repair one bad disc? Strap yourselves in and find out what it took us months to pull off.

When people say they’ve tried almost “everything” to do something, of course, they don’t always mean “everything”. There are always options and ways to get something done (whether or not it would be done “right” is a different matter). But believe us when we say, we’ve tried almost everything. We went to great lengths to try to repair this disc so that all the contents that were mastered on it could be preserved as much as possible. The issue with this disc is that no matter what drive we used, almost every drive couldn’t read some sectors on the disc. Out of all the drives in our collection, not a single one could actually read the disc completely. There were one or two that came somewhat close to reading most of the sectors on the disc (which we’ll get to later), but none of them could read the disc completely. Just what exactly was going on here?

We always strive to dump discs the “proper” way, as mandated by Redump.org’s dumping standards, and at the time, we were limited to just a single piece of software - DiscImageCreator. While there were other options, DiscImageCreator was the only notable software that Redump was willing to accept dumps for at the time. Fortunately, unlike CD-ROM-based media, our options for dumping non-specialized DVDs were virtually as many DVD drives as had actually existed. This helped tremendously as we weren’t limited to just a few Plextor or special M-DISC Blu-Ray drives like we would’ve been if we were trying to dump a CD. In this case, since most of the drives in our collection could read DVDs, we had a lot of options to choose from.

However, eventually we started to realize that even if we did find the right drive that returned the least amount of errors, we would be stuck having to contend with the fact that we couldn’t create a “full” dump without being able to at least “refine” the dump over and over again. Refining a dump means that you constantly retry certain sectors on the disc over and over until eventually the drive returns valid sector data, which can then be appended to the disc image before moving on to the next problematic sector that needs refinement. DiscImageCreator unfortunately didn’t have this option, at least not for generic DVD dumps.

Thus, we encountered our first roadblock. “How can we dump this disc using multiple drives without having to create possibly hundreds of disc images?” Well, before we had no choice but to create hundreds of disc images, we tried using a piece of software called ddrescue. Ddrescue is an open source data recovery tool primarily used for disk drives, but it can also be used to recover optical media formats like DVDs and CDs as well. Instead of having to create hundreds of dumps, we could just keep refining one dump with ddrescue until all sectors come back good. We began our refinement by creating an initial dump using one drive, then we let the same drive try to correct the errors by retrying each sector that couldn’t be read. We let this go on for a few hours, then when it seemed like the total number of errors wasn’t going down, we’d switch to another drive and start again. We’d do this for about a few hours, at least initially, until we could get a good feel for how effective the specific drive was going to be at returning good data reads.

At first, the results were a bit promising. We started with roughly 1800 errors in total in the beginning, using ddrescue. At first, swapping to a new drive would knock out roughly 100 sectors or so by the time we wanted to swap to another drive. However, it became clear that eventually the corrected returns would start to slow down in general, and we would be left with quite a few sectors that simply were going to be a pain to correct. After using roughly 50 drives to correct the disc, we ended up with roughly 1000 sectors left to correct.

Then all of a sudden, we started noticing something odd. Very odd. After using what felt like a thousand drives to correct the disc, usually getting at least a few sectors corrected at a time, we simply couldn’t correct any more sectors. This is worrisome, but we suspected that given that we noticed that it didn’t appear that the disc had signs of rotting in it (which is usually noticeable by shining a light through the disc surface to see if any holes can let light escape), that what we were seeing was more likely due to weakened dye either as a result of aging or bad storage conditions, or that it was a bad burn. While both case scenarios were likely, due to the nature of trying to read a DVD using a DVD drive, we really couldn’t be sure as to the reasons why.

However, our concerns were about to be greatly increased when we started noticing something far more worrying. After a while, we started noticing that the disc itself couldn’t be read in any drive. Not just drives we hadn’t used yet, but even in tried and true drives that the disc had been read with many times before. What was going on? Surely it couldn’t be anything seri-

OH FUCK

Unfortunately, while trying to read the disc in the many drives that we had in our collection, one drive decided to practically destroy the disc from ever being read again. If you’re wondering what the above image actually is, we actually don’t know for sure ourselves. It’s not a scratch, we can tell you that. At first, we suspected that it was a head crash due to a loose part inside the drive itself. After taking the disc to a shop that could resurface it to try and buff out what we assumed were just scratches, we discovered to our absolute horror that those markings on the disc were actually permanent. It’s almost as if the drive head or laser practically erased or obliterated the innermost part of the disc. The closer you are to the middle of the disc, the closer you are to the disc’s “lead-in” area, or, the area in which a disc’s track layout (if you were using a disc with tracks that is) or some other form of information that a drive would use to initialize itself to prepare reading the disc that was inserted. This would explain why the disc would no longer read; the actual lead-in area was practically destroyed.

Right here would’ve been the end of this disc’s story. The original disc, even as we speak, can no longer be read by itself. Unless you had a magic drive blessed by the angels, or at least a high-powered electron microscope, there’s no way that any drive would be able to recognize the disc anymore.

However, to get around the lead-in issue, we were fortunate enough to educate ourselves in the art of disc swapping through the use of a “trap disc” from our experiments in trying to discover new ways of dumping Dreamcast discs. The concept behind the method to read a Dreamcast disc using a traditional CD drive is that you can fool the drive into thinking the disc is a lot larger than it actually is and has more tracks than it actually does by creating a “trap disc” that has a certain table-of-contents (TOC) that is used to fool a drive by swapping the trap disc with another disc while the drive still thinks the trap disc is still inserted. This is done by inserting the trap disc into a drive, letting the drive initialize and recognize the disc, then once the drive recognizes the disc and its TOC, you “stop” the drive and forcibly eject the drive tray manually (or you can remove the top of the drive itself) to swap the disc out with the disc you are trying to read. With the “fake” TOC that was present in the trap disc still loaded in the drive’s memory, it will force the drive to think that the disc you have inserted is still the trap disc. If your original trap disc had a certain TOC that described a certain track layout, you could use this to your advantage to force the drive to read sections off a disc that go way beyond the disc that you are trying to read. The reason this is helpful for Sega Dreamcast discs is that since a traditional CD drive will only acknowledge the TOC of the low density part on the disc, and not the high density, you can use a trap disc that has the TOC of the high density part so that you can force the drive to “seek” to where the unreadable high density section is where all the game data is stored. Pretty neat, huh?

Well, it might be neat, but how does it help us with Shadow? Simple, since we had a “complete”, albeit bad, dump of the Shadow disc, we could just burn one of the dumps we made that represents the whole disc onto another DVD-R, then use it to initialize the drive we want to use with the TOC of the original disc that could not be read due to the head crash. This would bypass the need for the drive to attempt to read the lead-in area off the disc we were trying to dump. As a side note, thankfully, the problematic sectors we’ve been trying to reread and correct are actually far away from the area of the damage itself, so we were able to save ourselves in the end.

However, we still had the issue where apparently no drive in our collection seemed to want to read any more data off the disc. Even though we had roughly 100 more drives to go, one thing that seemed clear was that unless we encountered a miracle, there was no way we were going to read every sector off this disc. At the very least, the dump we had now contained a lot more data in comparison to the original unmodified dump that Last Minute Continue had created. We tried all kinds of drives, from DVD drives, Blu-Ray drives, slim drives, SATA/IDE drives, to even specific vendors like the tried and true Plextor, Panasonic, NEC, Philips, Samsung, and many more. We even tested the disc in the original drive used to burn the disc, the Pioneer DVR-105 (which, believe it or not, is metadata stored in the lead-in in almost every DVD-R that is burned), and still, we could not get the number of errors to go down. The only thing that we could do is hope and pray for a miracle…

Then, something happened. We tried one seemingly random, unassuming drive from the pile, and all of a sudden, the number of errors went down. Way down. Going from 1000 errors to just 330 errors in just an hour or two of use. What the…?

After what seemed like an eternity, we had finally discovered a drive that the disc seemed to actually like! When we inserted the disc inside the Lite-On XJ-HD166S, all of a sudden, the disc almost seemed to have become readable again. At first, we thought we were being hoodwinked. What if the DVD drive was giving false corrections? What if it was returning bad, incorrect data from some cache or buffer because of how screwed up the disc is? Well, when we looked at the data that was being returned by the drive, we confirmed that the data coming back was actually legitimate. We finally found it, a drive from the heavens, a drive that would actually read the disc!

Or so we thought…

To be honest, we have no idea why out of all the drives we had in our collection, this weird non-outlier from the pile was the one that resulted in so many errors being fixed. Your guess is as good as ours. Was the drive’s firmware just really good at handling errors? Was the laser or the photodiode inside the disc drive just extremely powerful? Did the planets align at just the right moment in that given time frame that allowed just the right atmosphere to exist to allow the disc the right conditions to be read? We don’t know! We did make an effort to experiment with the drive further to see if it was possibly due to how this particular drive that we received was calibrated, specifically the drive’s potentiometer (pot), which is responsible for giving the drive’s laser more or less power. The only reasonable thing we could think of would be the drive’s pot, as traditionally, a user could potentially make their disc drives read problematic discs better if they give their drive more power by adjusting the pot.

Before experimenting with our golden goose, we bought a second drive to make sure we weren’t going crazy. We could confirm that even a second drive could read the sectors off the disc with identical corrected returns in comparison to the original drive we had. But what does that mean for the pot? Well, even after adjusting the pot, we unfortunately couldn’t change the effectiveness of the drive at reading errors. The drive performed identically to what it was doing to the disc before. We simply had a drive that had just the right conditions to allow most of the disc to be read. But just this disc! Weird.

In the end, we were able to go from 1000 errors to just around 330 errors with this drive alone. Unfortunately, our luck with the drive had run out, and we had to keep moving on with the rest of the drives in our collection, which wasn’t a lot by this point. After just a few more drives, we had officially exhausted our supply of drives. And thus, we were at a standstill once again, with nowhere to go. What’s next?

In a final last-ditch effort using traditional means, we wanted to experiment with an idea that was based on drive behavior that we noticed after starting and stopping a drive while trying to read this disc at least a thousand times by this point. We noticed that certain disc drives might use a “burst” of high power initially when tasked to begin reading a certain sector on command. Traditionally, when discs are dumped, the discs are dumped sequentially by sector until the last sector that’s reported by the TOC has been successfully dumped. However, you wouldn’t normally dump a disc by dumping a targeted range of specific sectors at a given time. When we asked a drive using DiscImageCreator to just give us a block of sectors that contained the sector we wanted to get data for, there was a chance that the drive would actually give us the correct data for it! Yes, that’s right, by taking the list of sectors we still had to get correct reads for and making a batch file to run with DiscImageCreator, we asked the software to dump specific sectors with a certain amount of retries rather than just dump the entire disc again. For some reason, despite having used about a hundred disc drives by this point and countless errors at retrying the same erroneous sectors again and again, this method allowed some sectors to be read instantly.

But only about 30 more sectors were corrected using this method…

Understandably, we were quite frustrated with the situation. Despite the countless resources and manhours trying to get just one disc to dump completely, we felt like we were not much better off than when we started. Even after resurfacing the disc multiple times and adjusting the pot on our drives in a desperate attempt at trying to make the disc read better, we were still no better off than where we were at the start of this madness. The worst part was feeling that there was undoubtedly more data that could be read from the disc itself, but for some reason or another, the drives that we were using to read the data off the disc just could not, or would not, give it to us. We didn’t feel like the data wasn’t there, despite our feelings from earlier when we thought that the errors on the disc were the result of a bad burn. We felt that even with whatever weakened dye on the disc, the data was there on the disc, but we just didn’t have as much control over the situation as we thought. That’s when we figured out that we were seemingly at the mercy of the drives themselves.

Unsatisfied with the results of our endeavors thus far, we didn’t give up - not quite yet anyhow. We decided that in order to get much further with this disc, we had to start getting creative. We needed to explore certain topics that aren’t as readily understood, so we could better understand just what exactly we’re dealing with. We needed to go beyond the drives themselves and start thinking more about what exactly a DVD is, and to use our newfound knowledge to move even further. And so, we knew at this point we had no other option but to go down one level deeper.

We needed to go raw.

Demystifying DVDs

When we exhausted every drive and piece of software that we had at our fingertips, we knew we had no choice but to start looking into how everything really worked. Just what exactly is stored on a DVD? What is the function of a DVD drive’s firmware? What would it take to go beyond our current understanding of what we were dealing with? And, more importantly, what can we theoretically do at a lower level that would help us rid ourselves of this god-forsaken disc?

Well, before we can even attempt to do something at a lower level, we need to first understand just what exactly we’re working with. Just what’s stored on a DVD anyway?

Before we can answer that question, we first have to understand how compact discs in general work. DVDs are more similar to compact discs than you think. If we can understand how compact discs work in conjunction with a disc drive, we’ll be one step closer to understanding how DVDs work by association.

A compact disc (CD) is a small, flat, optical disc capable of storing digital data using microscopic pits and lands on one small reflective surface layer that when an infrared laser is shined on the written surface area while the disc is spinning, light refracts off the surface and ultimately onto a photodiode that interprets the alternating light patterns into something similar to computer binary (ones and zeros). The storage size of a disc is synonymous with how dense the pits and lands on a disc can be written to. The higher the density of pits and lands on the surface area of a disc, the more data that can be written to it, with the cost of having to utilize a more sensitive laser, especially when rotating the disc at higher speeds. Pits and lands are written in one continuous spiral track starting near the center of the disc and moving outward toward the disc’s edge.

([https://www.iasa-web.org/book/export/html/3846 Source: IASA-Web)

A compact disc can be divided into three physical sections. First, there is the lead-in area, which is primarily used in all forms of compact discs to store the table of contents (TOC) for the disc, which is used to describe where specific tracks are stored on the disc and how long they are. Next, there is the “program area”, which is where the user data-related recording fields are stored (the user data itself, header information, error correction data, error detection data, and subcode). Finally, the lead-out area is primarily used to designate an area on the disc in which the recorded information has ended.

Depending on the track format, information contained within each section on the disc is stored in a fundamental, fixed-size unit (i.e a small “slice” within a track) called a sector. Depending on the track itself, sectors can come in different sizes and contain different types of information. Audio CDs that follow the Red Book standard contain digital audio divided into sectors 2352 bytes in size followed by subcode data, which is non-audio control data interweaved with the audio data, which is stored in 8 channels (P-W, but primarily using the P and Q channels) to tell players things such as the track start/end points, times, and CD-Text (artist, title, etc). The subcode contained within the lead-in is primarily used to detail the actual TOC of the disc. Data tracks, which primarily use the Yellow Book standard, are comprised of 2352-sized “raw” scrambled sectors, containing “sync” bytes, time-based address, sector mode value, checksum (for error detection), some reserved bytes, 2048 bytes of actual user data, and Reed-Solomon error correction (RSPC) bytes. Unlike audio tracks, which utilize no error correction or error correction data, data tracks contain both EDC (error detection code) and ECC (error correction code), depending on the mode of the data track. For Mode 1 data sectors, all of the parts that we described for a data sector are present. For Mode 2 data sectors, ECC data is optional as a means to provide a solution to those who want to write more to a disc at the cost of the ability to correct errors.

All data written on a disc is “scrambled” before it is written, and “descrambled” after it has been read. The sole purpose of scrambling data is to help the photodiode inside a disc drive easily discern between two distinct bits in succession on a disc by introducing variability to the data. Audio tracks are written scrambled to a disc, while data tracks are similarly scrambled like audio tracks before they are written to a disc. The scrambling and descrambling of tracks, as well as the positioning, decoding, and control over the disc, is handled primarily by the disc drive (notably, the drive’s firmware). Since audio tracks lack error detection and error correction, a disc drive can utilize read errors in the form of surface error scanning (for detecting physical defects such as scratches) called C1/C2 errors. C1/C2 error detection uses surface error scanning to make a “best guess” attempt at what the missing data is, then supplies the missing data to the end user. The difference between C1 and C2 is that C1 only analyzes one data frame at a time, while C2 can analyze many interleaved frames at once. How things like C1/C2 error detection, disc and media formats, and laser calibration are handled are all done by the disc drive itself, and can vary from drive to drive as well as vendor to vendor.

This is a basic rundown on what a compact disc is, what is expected from the format, and what is ultimately available to the disc drive itself for handling the disc. While some of the more technical differences and details are not as well-known as compact discs, DVDs follow roughly the same ideas that were established with the compact disc format. But each aspect of a DVD carries a certain set of differences in comparison to the qualities that make up compact discs that we just described.

A DVD is physically a higher-density CD. It is fundamentally very similar, with changes allowing for more data reads and writes while improving the readability and error correctability established using the standards that were utilized before for compact discs. The difference in a DVD laser, for example, is that it uses a red laser rather than an infrared one, which operates at a smaller wavelength and spot size, which helps read the higher density pits that are written on the disc’s surface (which are written at roughly half the size in comparison to compact discs). A DVD can consist of one or two data layers, where the second layer is semi-transparent. Since the compact disc format was originally audio-centric when it was created, the data track aspects of the format are rather tacked on. However, DVDs focus more on the concept of storing digital data itself, doing away with audio-centric ideas such as tracks, time-based addressing, and subchannels that carry control information in favor of addressing and control structures within pure data blocks.

Source: ECMA 274 2nd Edition (June 1999)

As you can imagine, since DVDs do away with the concept of tracks and subcodes, while the purpose of the lead-in is relatively the same as compact discs, a DVD’s lead-in consists of different sections of ranged PSN values (which are like addresses, further described in the next paragraph) that have unique purposes. First, there is an “Initial Zone” which consists of all 00s for the sole purpose of locking the spindle speed and achieving focus and tracking when the disc is being recognized by the drive. The reason this area is all 00 isn’t that it’s a waste of space on the disc, but rather it provides a predictable signal that can’t lead to false sync patterns, which will allow the drive to easily lock the spindle speed. The next area is the “Reference Code Zone”, which, for the sake of brevity, is meant to help calibrate the drive to the particular disc’s special optical quirks (like reflectivity normalization, compensating for jitter, tuning, etc). The first Buffer Zone is meant to separate the calibration patterns established before this region from the control data established in the next region within the lead-in in the event of potential misalignment from the drive head or partial reads from corrupt decoding. The “Control Data Zone” describes the physical aspects of the disc itself in a region called the Physical Format Information (PFI) region, like what category of disc it is (DVD-ROM, DVD-R, etc), the disc size, the maximum data rate, the number of layers, and the start and end PSN addresses. The actual manufacturing information of the disc (Manufacturer ID, Disc ID/Serial, and other optional mastering information) is stored in another section called the Disc Manufacturing Information (DMI) region. Both regions are repeated 192 times over and over to prevent the possibility of corruption and unreadability - it’s very, very, very important that the drive can decode and read this, as without it, it has no idea where the Data Zone is (where user data is stored). Finally, a second Buffer Zone is primarily used to separate the Control Data Zone from the Data Zone.

Source: DVD Demystified (McGraw Hill)

The actual user data on the DVD (LBA 0, or PSN $30000) is stored in the Data Zone. Sector sizes are a little bit confusing to talk about when talking about DVDs, as there are different layers of abstraction regarding sectors, and each layer of abstraction can refer to a different sector size depending on what it’s meant for. A normal sector, as seen by software at the user level, consists of 2048 bytes of pure user data. A “logical” sector, which exists only at the host interface, is 2064 bytes in size and consists of the 2048 bytes of user data, plus a 12-byte header, and a 4-byte EDC. A “recording” sector is a sector that is actually encoded on the disc itself and consists of 2366 bytes (all the parts mentioned before plus 302 bytes of ECC data). An “unmodulated physical sector” is 2418 bytes in size and consists of all the bytes mentioned before, but with 52 additional “sync” bytes, which provide stability through alignment of the interleaved PI/PO decoding. Finally, a “physical” sector is 4836 bytes in size (which is just double the size of an unmodulated physical sector), which is modulated to ensure the minimum and maximum pit/land lengths, clock recovery, and focus/tracking (it’s similar in practice to how CDs use EFM).

In short, when we talk about sector data as it is written on the DVDs themselves (not physically written as a pit and land on the disc surface), we’re talking about “recording” sectors. As such, a recording sector consists of 12 bytes of header data, 2048 bytes of user data, 4 bytes of EDC, and 302 bytes of ECC data (Parity Inner (PI) and Parity Outer (PO)).

First, let’s talk about the header. The first byte in the 12-byte header is the “Sector ID”, which details what region of the disc the sector belongs to (00 is the data zone, for example). Then, the next three bytes are the Physical Sector Number (PSN), which is a hexadecimal number that increases incrementally from sector to sector, describing an absolute physical address of the sector. The next two bytes are the ID Error Detection (IED) field, which is actually a form of both EDC and ECC for the first four bytes of the header (the sector ID and PSN), which is based on rs(6,4) code and capable of repairing up to one byte in the first four bytes of the header. Next is the CPR_MAI field, which contains bitfields that describe the copy protection (CSS, no-copy, copy-once, etc), the disc type, usage permissions, regions, etc - it’s usually used for DVD Video as a method to enforce format rules.

Then you have 2048 bytes of user data, scrambled for the reasons mentioned before. The best way to look at the sector as a whole is to think of each sector as 12 “rows” consisting of 172 bytes each. After each 172-byte row is 10 bytes of ECC data called Parity Inner (PI), which is based on Reed-Solomon and applied to both the header and scrambled user data per row within the sector itself. Then, after the user data and parity inner data, is the 4-byte EDC, which is calculated over the unscrambled user data only. Then, finally, Parity Outer (PO) is another form of ECC that is applied by “column” that spans over an entire block of multiple sectors stacked horizontally, or in other words, a group of 16 sectors. Altogether, this adds up to 2366 bytes of recorded sector data.

DVDs have a large number of improvements over compact discs. As we described before, not only is the laser used to read the pits better, but it also allows for higher-density pits. The methods of error correction are vastly improved. In fact, would you believe that every single sector on a DVD has ECC and EDC? That includes the lead-in as well! Every sector is laid out in the same manner, no matter what area or zone you’re in. Fields that make up the individual sector aren’t repurposed for other reasons because of different zones either. Everything is consistent and follows a strict standard.

For compact discs, much of the information that we described can easily be gleaned through the help of certain disc drives and software. Some drives can read data tracks “as audio” which can give you a scrambled raw output of what is physically written to on the disc. Some drives give you the ability to extract the raw lead-in data too. You can even acquire the C2 error bit information to determine what areas within the sector have physical abnormalities to help detect the presence of errors caused by physical issues, such as scratches or intentional errors caused by DRM or mastering mistakes. This data is not only helpful for the (admittedly enthusiastic) end user to have in the pursuit of preserving the data from the disc, as with only the user data from the disc, there’s a lot that wouldn’t otherwise be able to be identified, specifically errors. This is one of the reasons why groups like Redump exist, and why we try to follow Redump’s dumping standards. Not only do you try to acquire as much important data from an optical disc as possible, but you also ensure that the data can be used to verify whether or not the dump is actually accurate.

However, DVDs were the beginning of the end for transparency regarding the actual data that is written on the disc. Whether it’s for the sake of simplicity or just a weird flex in an effort to pursue “copyright protection”, a DVD drive, when it comes to reading DVDs for the end user do absolutely everything from descrambling, error detection, and error correction, so the end user can never see it. The only thing the end user has to do is issue a simple SCSI command, and the unscrambled, pure 2048-byte user data for that given LBA (or decimal sector number) is given right to the user…if the data’s good, that is. If there’s something wrong with the data, the very LAST thing the drive wants to do is give it to the end user to see. “Scary scrambled user data with error correction data and whatnot? It’s just too much for the vast majority! So keep it simple.”

And for good reason. Compact discs were kind of retrofitted for storing different kinds of data, so there was a bit of a mess that ensued when formats that were based on compact discs came about. Standards like Yellow and Red Book were introduced, but people could decide whether or not they wanted to follow them or hack them to shreds to fit their needs. Vital areas on the disc, like subcode, lead-in, and track pregap, could be manipulated by disc producers to do atypical things that might cause issues depending on where you insert your disc. For example, what if you had a disc that had both data and audio tracks, where the data track was the first track on the disc, and you inserted said disc into an audio disc player? You would blow your eardrums and speakers out the second the first track played! You had single-session discs and even multi-session discs to make things even more confusing. Recordable discs had different dyes, and many manufacturers had their own method of creating recordable discs that could only be remedied by drive manufacturers by hacking methods of detecting certain brands and types of discs so that the drives could calibrate to the specific dyes and reflective materials used for the discs. By the late 90s, the whole compact disc landscape was a hellish mess.

There absolutely needed to be set standards if a new, upcoming format was to be introduced. And that format had to be strict. You couldn’t easily deviate from the format anymore because you didn’t have any control over what ultimately went onto a DVD. Someone who produced content on DVD didn’t have direct control over what got burned in the lead-in. They couldn’t manipulate ECC/EDC data or create some twisted form of sector to store data in anymore. DVD recordable disc manufacturers couldn’t easily make a weird variant of a DVD-R that was marketed as something more than a DVD-R, either. The worst thing that happened was that Nintendo altered the scrambling algorithm so that consumer DVD drives couldn't recognize or return data from GameCube and Wii DVDs. But even with Nintendo’s discs, they couldn’t change what a sector consisted of or the error correction/error detection code either! If you didn’t need to have control over it, you didn’t get access to it.

DVDs sacrifice transparency for consistency, where they don't expose all the data burned on a DVD to anything but the drive itself. This is what the problem ultimately is regarding dumping DVDs. This is the problem with dumping Shadow the Hedgehog’s Beta 4 prototype using conventional means. The conventional means are doing exactly what they’re meant to do. The drive is clearly reading something off the disc, but because the data might have some errors in it, it doesn’t want to give us anything unless it’s squeaky clean. It doesn’t matter how many times you reread a sector, you won’t get a different result unless you do something to the disc or just some random act of god. You don’t have control over it, but the drive’s firmware sure does!

Dumping Raw (sort of)

If we’re to get anything from this disc without using an electron microscope, we’re going to have to tackle the firmware inside the drive itself.

As you have read, DVDs carry a lot of nice information. They carry error detection code for detecting errors in the user data. They hold two sets of error correction code for correcting that sector information within the sector itself and even across multiple sectors. The sector header itself is protected and is easy to understand and seek with. There’s nothing about this data that we don’t want! However, the drive doesn’t want to give you this data. Nope, can’t have it. It’s not because you simply can’t, it’s just that you don’t need it. Who needs a separate method of evaluating whether the data returned from the drive is good? Just trust the drive itself; it takes care of everything for you! This is a limitation of drive firmware, so every drive is supposed to be equally incapable of exposing the extra data.

However, there’s a little secret that isn’t exactly new, nor is it entirely unknown. But there is a way to get this data, at least for some drives to varying degrees.

Disc drives, like any other kind of integrated device, have their own CPU/DSP and memory modules to serve as a cache/buffer and a place to store other temporary values during run time. Like most other SCSI devices, disc drives support a wide variety of common SCSI commands that have been standard for almost four decades. Most SCSI commands are standard across SCSI devices, while some commands are specific to the realm of disc drives specifically, and there are vendor-specific SCSI commands that are specific to the drive and manufacturer, like certain debugging SCSI commands.

Dumping from the drive’s cache or buffer isn’t exactly new, and is predominantly the method used to dump most discs in the enthusiast world. Experiments in dumping discs from cache date back to the days of the Nintendo Wii, when it was discovered that while it appeared that drives couldn’t “read” Wii or GameCube discs, there were some that somehow actually did, at least without disc swapping. Part of the reason why GameCube and Wii discs couldn’t be dumped using a traditional consumer-grade disc drive was that part of the disc’s lead-in appeared malformed to most drives. However, some drive firmwares “assume” things about some discs that you insert, and can actually recognize at least some parts of a disc to get things to a point where you can actually attempt to issue SCSI commands to the drive in an attempt to read them. While normal DVD reading SCSI commands like READ12 wouldn’t return any data, it was discovered that by using certain SCSI commands that return data from the drive’s cache or buffer, the drives did in fact read data off the disc. It was never a fact that the drives physically couldn’t read the actual data from the disc; it’s just that it couldn’t descramble the data that was read and properly validate it using EDC. All it took was just a certain command that allowed someone to read data off the drive’s cache. Again, this was all just a hindrance because of the disc drive’s firmware once again!

One of those memory-reading commands, READ BUFF (0x3C), is a common SCSI command that can return the contents of a SCSI device’s memory, broken down by pages. Most drives out there have support for this command, which can help spill some unintended data to the user on request. If you get the parameters just right, you can get the data that was read by the drive before it's returned to the user. Since we know what a raw DVD sector consists of, we can easily find the right parameters needed for the command to find the right combination that returns the data we want.

However, there's a catch.

Because READ BUFF is implemented at the firmware level, despite every drive having some level of support for the command, the command varies from vendor to vendor, drive to drive. So one combination of parameters might not necessarily return the data you want in one drive, even if it worked on another drive model. The second issue is that drives will store only what's necessary for the drive's firmware itself to perform what it needs to do on the data that's read from the disc. As such, some drives return variable, but somewhat fixed/standard data sizes per sector. So even if the data is found using READ BUFF, you need to determine how much data is returned per sector that's stored. Sometimes 2366 bytes with some padding are stored, sometimes 2064, etc. Sometimes data can be returned already unscrambled, sometimes data can be stored scrambled but with only error detection data. Sometimes you get everything. It's a mess.

Raw sector returns on the cache can be classified into types based on the size of the sector on the cache and what can ultimately be gathered from it. Here are the types reserved so far:

DVD sector types

Type	Size	Scrambled?	Data	Notes
Type 1	2064 bytes	Not scrambled	No PO/PI data. EDC and sync only.
Type 1a	2064 bytes	Not scrambled	No PO/PI data. EDC and sync only.	Offset by x amount of bytes in cache for some reason.
Type 2	2236 bytes	Scrambled	EDC, sync, and PO bytes only.	No interweaved PI for some reason. PI data is stored in a separate table in the drive’s memory after the area where the disc’s other sector is stored.
Type 3	2304 bytes	Scrambled	EDC, sync, but no PO/PI bytes it seems.	"00 FF FF FF FF FF FF FF FF FF" interweaved every 181 bytes, 11/12 times per group of 2304 sectors.
Type 4	2384 bytes	Scrambled	EDC, sync, PO/PI data included.	Includes 18 extra bytes of discardable, drive-specific garbage in every group of 2384 bytes at the end.
Type 4a	2384 bytes	Scrambled	EDC, sync, PO/PI data included.	Includes 18 extra bytes of discardable, drive-specific garbage in every group of 2384 bytes at the end. Segmented, data accessed with different READ BUFF parameters, observable with only one drive so far (LG GH20LS10).
Type 5	2392 bytes	Scrambled	EDC, sync, PO/PI data included.	Includes 26 bytes added in the PI/PO areas in groups of 2 bytes that seem pointless. Not part of raw sector spec.
Type 6	2816 bytes	Scrambled	EDC, sync, PO/PI data included.	We theorize the extra 450 bytes of data per group of 2816 bytes might be debugging-specific data used for the drive, but might contain some useful information about the disc at this sector that's not part of the raw sector data spec.

So ultimately, the best drives are the ones that return at least 2366 bytes of verifiable data onto the cache. This will include the sync information, EDC value, and PO/PI data used for correction. Sometimes drives will even keep uncorrected or partially corrected data from L-EC-related SCSI errors in the cache. These drives are useful in cases where EDC fails, but we still want at least some data, which is exactly what we want with Shadow. These drives are also useful for dumping Nintendo Wii/GameCube discs, since Nintendo changed the scrambling algorithm for those discs, which cause the EDC check to fail every single time, even though the data itself was actually read perfectly and doesn’t have any actual errors in the data itself.

Now that we had a method for reading the recording sectors off a disc by reading straight from the drive’s cache, we could finally look at the data that the drives were refusing to give us! When it comes to preservation, something is always better than nothing. It was initially frustrating to us that we weren’t getting any data from the drive in the event of an uncorrectable error, even though the data was at least partially read. The data itself could’ve had one or two bytes that couldn’t be corrected, but that by itself is enough reason for the disc drive to throw away almost an entire block of sectors! And so we began creating another round of a few hundred dumps of the disc’s surviving recording sectors in the hopes of understanding what exactly was going on.

However, while we had a method for getting more data off the disc, we didn’t exactly have a plan to use it quite yet. Trying to dump any disc “raw” is niche by itself, but DVDs were somewhat uncharted territory even for most enthusiasts, and there weren’t any available applications that even utilized most of the data that was coming from the drive’s cache. From our experiences in dealing with optical media thus far, we knew that having at least some data was better than nothing at all. But having more data than what was intended for normal users to see gives us an advantage in creating applications that can utilize the data itself, besides the insight as to what was going on with the disc under the hood. Unlike the pure, unscrambled user data that the drive wanted us to see, we now have both a method of confirming the disc’s integrity (EDC) as well as a method for attempting to correct and restore the disc’s integrity.

When a disc drive is reading a disc, it only stores what’s been read for the current session. Aside from the disc’s information from the CDZ area in the lead-in, the drive can only store a megabyte or two of the disc’s sectors in memory at a time. One major limitation of a disc drive is that it’s only good for a disc that it can actually read; it’s not a magical restoration device whose sole functionality is to restore bad discs. Once you remove power from a disc drive, you have to start all over again, regardless of what’s been read.

But having this data returned to us in user-land can have possible advantages. No drive reads a bad disc the same way, so despite having hundreds of individual dumps of the same disc done in many different drives, the data that was returned by each drive could theoretically offer more (or less) data than the next, and most importantly, in different areas that might’ve been more problematic than other disc drives could handle. Any one dump by itself isn’t necessarily useful, but in theory, if we could combine something from every dump combined with a method to validate the data being combined, it may be possible to take the partially correct bytes from every dump and combine them into a working disc dump that can be completely validated by the included EDC data!

Our first thought when presented with this data was, “What if we could take all the information that was gathered from every dump and somehow mend the data back together?” Given that no tools existed that took advantage of this data, we had to develop a new tool completely from scratch that could give this (and potentially many other discs) one final chance at being rescued.

Introducing DiscImageMender (DIM)

And thus, DiscImageMender (DIM) was born!

DiscImageMender (DIM) is a tool that can use error correction codes present on CDs and DVDs and mend the good sectors/blocks of various disc image dumps together to create the best possible version of that disc image (i.e., one that’s error-free).

When dumping discs, there’s always a chance that the process of dumping the disc will generate imperfect dumps. Imperfect dumps can occur due to faulty media, misconfigured software, broken firmware, bad drives, and more. However, sometimes discs can be further dumped using different drives and software that can yield different results, some of which can be less imperfect than others. While it might not be possible to get an error-free dump every time, it might be possible to reconstruct a near-perfect image of a disc by utilizing the data from each subsequent dump in which its integrity can be verified.

Disc Image Mender can combine verifiably good data from “like” disc images and data that can be repaired/fixed using included error correction data that might be part of the disc. This way, we can have a disc image that is free from errors as much as possible. No other tool exists that can do this, as most people will just stick with the dump with the least number of errors without realizing that every dump they make could have its own unique correct/incorrect sectors.

A report generated after mending with DIM

Here is the way DIM works. First, it will scan an input directory for similar disc images with verification data. The tool will assume that all images are meant for the same disc. Once all of the discs have been found, DIM will then go through each disc image and verify the integrity of each sector in the image. For instance, if the input is an ISO 9660 formatted disc image, the program will utilize the ECC/EDC data from each dump to detect problematic sectors on the disc. The status of each sector/block from each dump is then recorded and saved until all disc images have been processed. Finally, the tool will then combine all the good sector/block data into a new file. Depending on the image type, the tool will then verify the bad sector/blocks to see which can be repaired and regenerated using the included verification data. Once the repaired sectors have been merged into the new file, the tool will then append the bad sectors/blocks that are unchanged between each dump. At the end of the process, the tool will report the final statistics on the newly repaired image (how many sectors are fixed, how many are still bad, duration, etc).

Luckily, at the same time, we were working on another project involving CD/DVD internals and the structure of their raw data. Throughout the years, we have dumped thousands of discs. Over time, an alarming trend began to emerge – most recordable discs, i.e., CD-Rs, DVD-Rs, etc., began degrading. More and more discs we got were in a state of rot, and we could only pull partial dumps of them.

Back in 2016, we worked on a bigger lot of Dreamcast GD-Rs, and some of the discs were in pretty bad shape. Unfortunately, back then, the methods of dumping Dreamcast discs were rather crude, and while we could get dumps, we didn't have a great way of knowing what percentage of a dump was correct. Fortunately, the CD format includes quite a lot of error correction and detection data. In fact, a 650MB disc is actually over 2GB in size, 650MB being the "data" and the rest being multiple error correction mechanisms and metadata.

As we mentioned earlier in this article, on a CD, every sector of 2048 bytes has 304 extra bytes attached that consist of Reed-Solomon error correction codes (ECC), called P and Q. These are a clever bit of polynomial math that let you recover lost bytes of data, provided you have some of the other bytes. It's the same algorithm that powers QR codes and error correction in RAR and PAR files. This was very helpful because it provided a robust way of checking a sector for errors, and the "degree" to which the sector was bad. The idea was that we could dump a game many times and pick the "good" sectors.

Over time, this became increasingly important as we started to work on bigger and bigger lots of discs, and at the same time, the problem of disc degradation became worse. In the year of COVID, we worked on a group buy lot that had a few unruly discs that just would not give us playable dumps, and we had an idea. Could we use the ECC codes themselves to repair some of the damaged data?

A report generated after mending with DIM

The core idea of DIM was based on the fact that sectors are arranged in rectangular blocks of 2048 bytes, with P codes correcting each "column" and Q codes correcting each “diagonal” slice of data. Thus, we could go through every P column vector and correct a byte or two, then go through each Q diagonal, correct a byte or two, in a loop until we corrected as much as was feasible. With this, we were able to correct a bunch of previously bad sectors.

An image of a disc with rot assessed with DM. Boo.

Unfortunately, bad discs tend to have a lot of bad sectors, so ECC repairing just isn’t enough Then we had the idea of combining this with the previous idea of merging known-good sectors from multiple dumps. So, let's say we dump a disc 100 times. For each sector, we pick the best dump possible for that sector, and if there isn't one, we pick the one that can be best repaired. With this, we were finally able to fully repair a previously unplayable build!

Yay!

You can download our corrected dump of Shaolin here (an extra Christmas treat for those who read this far into the article!).

So the idea worked for CDs. The story for DVDs gets even more complicated, however.

While this tool is effective with images that already include some form of verification data, it won’t be effective as it is against other types of disc images that do not have verification data baked in. For instance, DVD images that utilize UDF do not have any error correction data at all, and unlike CDs, there wasn’t a known way to dump the DVD error correction data at all, so there was no way to verify if a disc image sourced from a DVD is good or not, as the only way errors can be detected is if the software can adequately report the errors at dump time.

However, since we have discovered that various drives can actually dump the hidden ECC and EDC data from a recording sector on a DVD, this problem can be resolved by incorporating some changes into DIM’s algorithm for DVDs!

For the DVD part of DIM’s algorithm, the code first builds an empty matrix representing a DVD block, then pulls the “columns” and “rows” from all the dumps detected by DIM, as described earlier when we were talking about a 2366-byte-sized recording sector while populating the new matrix with those values. If there are conflicting passing ECC blocks, it first:

• Prefers blocks where there are duplicates across multiple dumps.
• Merges common bytes among ECC blocks whose code appears the most across all dumps.
• Merge common bytes across all dumps for that ECC block that passed their check, if any bytes tie, merge using common bytes across all ECC blocks, pass or fail.

The reason this is needed is that we were unfortunately seeing false positive (PI) ECC passes in some dumps, where the same code passed ECC, but the data was different across different dumps. The reason we suspect this happens is that some drives apply ECC in the event of a failing EDC and leave the result of the ECC application attempt in the cache. This way, we can catch this before utilizing the data for anything else. (We’re fairly certain this isn’t an issue with DIM itself, as we've tested multiple Reed-Solomon implementations and verified that the ECC checks do in fact pass every time.) Once the matrix was populated with good ECC blocks, we then filled the rest of the bytes with the common bytes that were found across all the dumps that were provided to DIM.

In total, we present five different merged sector types - common bytes only, PI + common bytes, PO + common bytes, PI + PO + common bytes, and PO + PI + common bytes. The last two swap the order in which PI and PO will overwrite each other.

A report generated after mending with DIM

At the end of the day, we were able to pretty much resolve almost all of the remaining sectors that were left to be corrected from the original dump. Unfortunately, there were about a dozen or so sectors left that couldn’t be corrected even with DIM. Luckily, this is one instance where using data from another disc comes in handy. However, rather than just replacing the file wholesale, we can actually determine whether or not the sector data we’re about to replace would work in conjunction with what was stored in the original sector. Since the EDC value in a sector corresponds to the unscrambled 2048-byte user data, we simply take the data from another build or the final version that we can clearly see was meant to go in the original sector, and just see if it matches the EDC value that was reported the most in every single dump of that sector. If it does, then we can safely merge it knowing what it actually contained! Before, we wouldn’t have been able to know if there was a byte or two that was different in a file that would cause the file to not match retail. However, with EDC data being provided, we can know for sure now that our corrections and replacements would make sense.

It took almost a full year, but after all of our hard work and trial and error, we were finally able to have a fully confirmed, working version of the disc image. The disc itself was put through the wringer, and possibly accumulated at least 500 to 1000 separate dumps in various sizes. While the work was hard, the knowledge that we gathered from the experience was invaluable and taught us quite a bit about disc formats. Now, we’re comfortable that we have the knowledge and expertise to handle any kind of disc to try and get the most out of a disc. At that point, we were confident that the repaired image represented the most complete and accurate form of the disc that could realistically be recovered.

…that is, until someone on eBay happened upon another copy of the same prototype we have been trying to dump for almost a year. We bought it, of course, and it turned out that it matched the repaired build one to one, without a byte out of place. But hey, at least we verified the method works

OH GOD DAMN IT

Reflecting on Failures

Despite the overall negativity that this article has, please don’t take it the wrong way. Despite everything that has been said or done in the name of this franchise, it’s amazing and, to be honest, extremely hard to believe that the Sonic the Hedgehog brand is still going, even despite all of the massive pitfalls the franchise has fallen victim to. To think that there were so many close calls and blunders despite everything up until this point, that some individuals at Sega decided to keep going, to keep making Sonic games despite reactions from critics and die-hard fans.

The Sonic franchise exists in a different world now, a world that makes whatever feelings someone used to have about blunders such as Shadow the Hedgehog and Sonic 2006 look alien in comparison. To think there was a time we thought Sonic 2006’s load times were the worst things imaginable! Little did we know that those kinds of load times would seem fast in comparison to some of the load times that modern games can ask of us!

Even the mere concept of video game movies being remotely successful is amazing to think about. There was a time when a theatrical movie adaptation of a video game was a sure failure in every aspect. To think that not only can a movie based on a video game be tremendously successful enough to include Jim Carrey and Keanu Reeves as your stars, but that you got to make three (possibly more) of them, and have Jim Carrey star in all three! And it’s a Sonic movie that did it no less.

The Sonic franchise of today is nothing like it was over 30 years ago, but it’s also a far cry from how it used to be perceived even 20 years ago. Sonic’s doing alright for himself, even if it feels like he has to look behind him more often than he looks in front of him. That’s not to say that he’s impervious to bad games and horrible ideas, and that we may never get another “Shadow the Hedgehog” or “Sonic 2006” ever again. But we can’t say that life would be any better without those games either. The fact that even Sonic’s past blunders can still find traction with a new, younger audience is a Christmas miracle all in itself. God bless him and everyone.

Conclusion

First of all, we deeply apologize to those who have been waiting for us to act on this for an extremely long time. We’ll admit that we sat on it for a bit, but as you can see from this article, this particular lot took a lot of time and effort over a long period of time to see through to the end. We hope it was worth it to you as it was to us. We’ll have more on the way soon, so just sit tight!

But before we go, we believe some thanks are in order. First, we’d like to thank Last Minute Continue for allowing us to dump these discs and being patient with us to have us try everything we could to save them. A HUGE thanks to Sewer56 for once again doing a huge amount of research into the prototype and for helping stream the prototype for all of you to enjoy (be sure to check in on TCRF for a much more in-depth research article on the E3 prototype as well as the Beta 4 prototype).. We’d like to take this opportunity to thank every single person who has generously donated CD/DVD/Blu Ray drives to help aid in our research in dumping not just this disc, but for many others (as you will soon see). And thank you to the donators that helped acquire some of these discs. Thanks to Mystyle48 for helping us acquire the Sonic Schoolhouse prototype. This could not have been done without any of you, and is nothing short of a Christmas miracle. Huzzah!

As you can see, we have a couple of extra Christmas goodies aside from the Shadow discs. The prototypes featured today are a combination of the lot sent to us by Last Minute Continue a few years ago, plus a few of our own that we kept over the years.
The Shadow, Sonic Gems, Billy Hatcher, and Sega Classics Collection discs are all from Last Minute Continue, while Sonic’s Schoolhouse (funny story about how we got this one!), Sonic Mega Collection, Sonic Riders (+ Zero Gravity), and the redump of Shadow the Hedgehog Beta 4 come from us. A special note on Shadow the Hedgehog’s E3 build. There were two NR discs that were part of the lot, but it looks like both discs are bad. We’re working on trying to get a full dump of one of the discs (or something using both to mend with), but it’s a low priority since the data that needs to be dumped exist in just the padding data on the disc - the actual game data is unaffected so it’s perfectly playable, it just can’t be accepted into Redump at this time. The Sonic Gems Collection discs are, unfortunately, the only discs that are definitely broken in some way, but we’re still looking into trying to get the most out of the discs to try and repair them. But we’re offering them today until we can get a better dump someday soon. All the other dumps were dumped using “redumper”, a new replacement as the go-to for making Redump submissionable dumps.

And finally, since we are nearing the end of the year after all, thank you to every single one of you who have contributed to the site, released prototypes, and just generally supported the whole endeavor. This year was pretty exciting, and we saw a lot of open chapters come to a close. Here’s hoping the next year will bring even more surprises, answers, and fun!

Happy Holidays, everyone!

Until next time! Stay frosty. 8-)

Two lovers are playing good Sonic games on a holy night.