Saturday, August 03, 2019

EXCLUSIVE: What the Hell Happened to BadBlue News?

Update 8/4: BadBlue News has been restored.

Caution: this is a really techno-geeky article describing what in the Sam Hill happened to BadBlue and the progress made thus far in restoring it

I've gotten a fair number of emails over the last few days expressing concern that the news aggregator is not updating news stories.

Now: a little bit of background on the site. It is a full automated news-gathering site that leverages social media (e.g., Twitter) to determine which stories are trending. Its database consists of over 750 news sites that are scanned pretty much continuously for "hot" stories.

I created BadBlue in 2011, initially working with the great John Hawkins (then of Right Wing News and now of Brass Pills), but eventually going out on my own. I used Amazon Web Services (AWS), the powerful cloud hosting platform, to spin up several Linux machines to host the content. If memory serves, the original launch of BadBlue was in January of 2012, more than seven years ago.

Twitter Makes a Change

When I noticed, with the help of several readers, that BadBlue wasn't updating I pulled some log files and determined my programmatic connection to Twitter (also called an API) was failing to connect. Mystified, I checked the @BadBlueNews Twitter account; it was fine, not suspended or impeded in any way. So that was a relief.

In researching the Twitter API, it hit me: Twitter had a new requirement, which they turned on in late July. This requirement was to use a new encryption standard called TLS 1.2, versus the original version called TLS 1.0.

So, my challenge was to upgrade the encryption library, called OpenSSL, to a more recent version. Not a problem, right?

First attempt: figure out a way just to update OpenSSL to a more modern version. Bam! No dice, there didn't appear to be any way to upgrade OpenSSL in place since it was such an old version.

Second attempt: upgrade the entire AMP stack (Apache-MySQL-PHP) on the Linux box, which would include OpenSSL. Bammity Bam! Sorry, Luther, the version of Linux was so old, it was incompatible with an upgrade of the AMP stack.

Third attempt: okay, I'll just upgrade the entire LAMP stack (Linux operating system plus AMP)! I use a service called Bitnami (quite good, actually) to manage my Linux servers. The one drawback with Bitnami is that they tightly control the images they distribute. There are no upgrades permitted in the Bitnami ecosystem. I can kind of understand that, but it's very painful in situations like these when I need just one library updated. So, Bammity Bam Again!

Ding-Ding: It's Round Two!

Alright, f-it, I thought. Time to just provision a brand new server. Since I use Bitnami to provision my cloud instances, I went straight there and launched a new LAMP machine. Bammity-Freaking-Bam! The error message simply stated that my AWS machines were so old (2011), Bitnami couldn't launch a new instance.

Their advice on fixing this? Create a brand new AWS account, so I would get the latest VPC (Virtual Private Cloud) features, required to provision new machines.

Okay, done. Now, Bitnami, let's build a new LAMP box from scratch!


I went back to the Bitnami control panel and punched out a new LAMP box. Success! Now it was time to begin porting over the BadBlue database. Bamapalooza! The scripts I had used to create tables and indexes in MySQL had syntax errors and wouldn't run? What in the hell?


After reviewing the DDL, the field name RANK was no longer legal (I guess it's become a keyword, or perhaps it's not gender-friendly). All of the references to RANK in the database description language and the code itself had to be changed to RANKING.

Best of luck, dumbass

Of course, I've gone on far too long screeching this digital diatribe.

Suffice it to say that I've overcome a few challenges and have yet to solve the latest: PHP no longer supports the MySQL programming library, and has moved instead to its own version called MySQLi (improved). My next task is to convert over and validate all of the code that references mysql_ to mysqli_. Good times, good times.

If you've read this far, congratulations. I suspect some are using this tale as a drug-free cure for insomnia.

* * * * * * * * *

By Crom's word, I hope to complete the upgrade process in the next several days, so please bear with me.


Anonymous said...

Just proof that going into the cloud can be just as frustrating as self-hosting

Anonymous said...


Matt Harris said...

I had thought that Bad Blue wasn't updating... sorry about your woes in getting it fixed.

Anonymous said...

Good luck, and thank you for the update and the work you put into BadBlue.

- Taqiyyologist

Anonymous said...

Thanks MUCH for your service - one of my Go-To sources...

Chris Jr.

Anonymous said...

Gotta keep an eye on updates whatever your environment. Just a note that I wrote a bunch of stuff for Bitnami (good stuff) about a year ago, but for Azure. Oh well. Not sure being on that platform would have helped either.

Anonymous said...

Regardless of its tech issues, bad blue has been subverted,hijacked, by junk sites that are nothing more then click bait.

Whoopie said...

Doug, I think I woulda' killed myself after the first BAM! Definitely by BAM! #2. Hang in there brother. Thanks for your effort, Bad Blue is the best aggregator of trending links.

Unknown said...

Good luck, I really appreciate your site. I'm an old guy that has dealt with computers since 1962. I could not deal with todays complexity.

cj said...

I, for one, really look forward to seeing Badblue up and running again. Absolutely my favorite site, which I check in on several times per day. Often, I'll have 25 - 30 Badblue links open simulataneously. Then move over to the sidebar links. My sympathies for your technical issues. Fix one thing and the fix blows up another - been there, done that. What a headache! Soon, hopefully, soon.

vanderleun said...

Doug, You have always been one of the foundations of the blue empire. I salute you.

Anonymous said...

Good luck Doug...I am another BadBlue fan

Anonymous said...

Thanks so much for keeping your fans updated as to what's happened. Bad Blue is my homepage, so the drop off in updates was concerning. Good luck and I look forward to Bad Blue up and aggregating again.

Heeter said...

Please please fix this. I am months ahead of everyone I know on news and it is because of Badblue. There is NO other site that is so easy to use and has such good content.

Anonymous said...

Please, please fix this. I am months ahead of everyone I know on news and it is because of Badblue. There is NO other site with the content and ease of use.

Anonymous said...

I really miss your website, all the news in one place. Please hurry up and get back to reporting the news.

Anonymous said...

Sure do miss your website, no one else even comes close.