Curating dead websites
With that many websites, which were added in a span of 3 months, I started experiencing the inevitable and universal problem of any web directory: some websites started to die.
I decided to tackle this issue sooner than later with a simple system:
- A website can be declared
dead.- When a website is dead, it goes in a graveyard. For example, here is the ~/social graveyard. Of course, a dead website won't appear anymore in other sections, including the Surf Mode.
- A website can be brought back to life if it becomes
alive again.
It can also be permanently archived when the curator is certain the website
cannot be resuscitated.To help the curator handle dead websites, there is a new background job that pings websites every day. That job marks websites
unreachable in case of failure and unmark them when they
become reachable again.I decided against a 100% automated process because it's not trivial to automatically detect that a website is
unreachable. For example,
the website may require a challenge to access or block crawler requests.With the solution I implemented, curating dead websites takes minutes per week, which is good enough for now.