Menu Close

Little mistakes, big failures.

Recently the Playstation iOS App received an update to enable Remote Play with the PS4, I already have the Playstation Remote Play.apk installed and working on some Android root and I liked the idea of remote play. Someone commented that the Android version no longer needs to be root and no longer need to be hacked from Sony Xperia, that the app works fine on any Android.

Curious I thought of installing on an Asus Tinkerboad with Android that I have at home, but out of sheer laziness I got an Amazon Fire Stick 4K that was on my desk and I tried to install it on it, “is also Android eh? And don’t need to be root.” … Do you know when some ALARM in your mind screams clearly DO NOT DO THIS?

I not only tried to install as I first installed the entire Google Services package, Google Store, Google Framework, Google Login… Huge mistake.

1st Failure: Amazon Fire Stick 4K has become a brick.
= Learned: Paying more attention to the instincts, I remembered that I could not install any of this on the Amazon Stick and only did the installation for mental laziness.
= Time Wasted: 15 minutes of copying and installing files.

After the problems with inoReader on March 1st and the migration to Tiny Tiny RSS (aka tt-rss), my personal home server demonstrated that it does not have the ability to run Docker + Postgresql + TT-RSS, Docker failed often the NAS, CPU crashed at 100% for any reason, it was not a stable solution for the long term …

I decided to try to make the TT-RSS server work in AWS, initially it worked very well, navigation responding fast, updating the feeds with frequency, I was adding more resources: Bring the whole page, take the photos/images of the articles, convert youtube links for direct-videos, adding new feeds (I think my list of feeds is currently around 600).

Until it crashed everything, crashed tt-rss, crashed PHP, crashed, crashed Remote Desktop, the whole instance on the AWS server was incommunicado!

Among several attempts to make the server communicate again I ended up clicking STOP / START on the instance of AWS EC2… I did not know but this is really a bad idea, when I clicked START the server started but in a completely different instance with a new IP …

It may not seem like a big deal, but when you have 1 server for multiple different domains it is an ungrateful job to have to manually correct and update all the DNS entries for the new IP.

2nd Failure: Turn off / on the server instance on the AWS EC2
= Learning: Never repeat this mess again
= Time wasted: 16 hours with the site offline, reinstalling Windows update 1809, correcting DNS entries, configuring the settings in TT-RSS.

As soon as the site returned on air the problems did not end, even removing the TT-RSS this time the performance was unbearably slow all the time, PHP-CGI FastCGI was consuming 100% CPU all the time, it was basically impossible to navigate the pages of the site often disconnecting the browser with slowness that causes over 1 minute waiting. There was nothing in PHP-Errors.LOG, nothing that would show a real application or database error and no form of debugging, something was consuming PHP massively and I could not pinpoint the problem.

After hours of work and maintenance, I upgraded PHP to version 7.3.3 and apparently now things are stable and lighter. Apparently the problem was not from the site itself but I discovered in the IIS logs that a bot (Yandex) decided to flood the site by opening almost every page of the site at the same time and when it failed it tried again, it was almost a DDOS for whoever has a T2.Micro Free Tier server (1 Core 2.4Ghz and 1GB Ram) on Amazon AWS.

3rd Failure: To lose hours trying to debug PHP-CGI caused by an external problem.
= Learned: Check the IIS logs and other processes even when the problem occurs in a particular process. If the PHP-CGI start to working in something, there is no way to find out which process started the call.
= Time wasted: 8 hours.


Comments are welcome.