robots.txt and archiveteam.org...
lists at boyandin.info
Sun Jul 7 02:22:02 CEST 2019
I believe this subject is way off the mailing list, but just my 5 cents.
1. GDPR, as any other bloated, convoluted, written in inhuman juridical
language law, mostly benefits two kinds of people: lawyers and
government-related officials. It incurs a lot of ado and expenses, gives
vast grounds for power abuse and so on and so forth.
As a side effect, it somewhat helps ordinary people to control the usage
of their personal data. Since data lifespan on the Net is hardly
controllable in whole, the abuse potential of GDPR is limitless. Cheer
the politicians for this excellent masterpiece of legislation.
As many such laws (the closest example of similarly inadequate law is
Russian Federal Law #152, "On personal data") are introduced worldwide,
they will strike a lethal blow to majority of small and medium
businesses, and cripple the base of normal human communication.
Let just watch the process and enjoy the show.
2. The "Robot Exclusion Protocol", as it's defined in its text, is
advisory only. It is not mandatory for any kind of data transmission.
Thus any claims or demands about following its statements are void. You
may ask, not to demand.
Any entity trying to transmit data over Net can't be reliably *and*
efficiently identified as human being (or a bot). Thus, it's quite easy
to imitate bot/human being, which makes the robots.txt a lame excuse for
lack of efficient control over which data should be taken by which
Simply stating, if you don't want your digital crap being available to
anyone, don't make it publicly available.
robots.txt usage was weird and strange in many cases. I remember several
WordPress versions which silently changed, when installed, robots.txt to
disable all page indexing. Also, you cannot magically demand to remove
downloaded and stored locally data just by altering your robots.txt at
will. That's pure nonsense.
Although I do not like, in many cases, the wording Archive Teams uses,
in this given case I think they are, generally, right.
Listo Factor via Gnupg-users wrote 2019-07-06 19:06:
> On 7/5/19 10:13 AM, Wiktor Kwapisiewicz via Gnupg-users -
> gnupg-users at gnupg.org wrote:
>> As for robots.txt not all archiving sites respect it:
> Thanks for posting the link. To quote from the text there:
>> What this situation does, in fact, is cause many more problems than it
>> solves - catastrophic failures on a website are ensured total
>> destruction with the addition of ROBOTS.TXT. Modifications, poor
>> choices in URL transition, and all other sorts of management work can
>> lead to a loss of historically important and relevant data. Unchecked,
>> and left alone, the ROBOTS.TXT file ensures no mirroring or reference
>> for items that may have general use and meaning beyond the website's
> This is both stupid and arrogant. It is precisely the owner of the
> website and data contain therein to decide what is and what isn't of
> "general use and meaning beyond the website's context", not of some
> aggregator/archiver's management.
> GDPR has indeed changed the nature of Internet forever, and it is for
> the better. If Google was put in its place (well, at least first steps
> have been made..) by the EU, surely it will be possible to force other,
> lesser operators of "archived information" to toe the line. Among
> to respect the straight and simple Robot Exclusion Protocol. It is not
> at all something difficult to do.
> Gnupg-users mailing list
> Gnupg-users at gnupg.org
More information about the Gnupg-users