![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
shadowscastSo, as I posted yesterday, Yahoo will soon be shutting down much of the functionality of its groups. I found out yesterday afternoon and have been scrambling to figure out what's going on and what to do about it ever since (apart from the time spent working, commuting, parenting, and going to the dentist to get a filling replaced!).
I'm going to do a links roundup here (mostly for my own benefit so that I can close some tabs in my browser!—but if anybody else finds this useful, that's great), and then I'm going to report on what I've personally done, what I'm planning to do, and what I'm still trying to figure out.
1) Yahoo's official announcement.
Understand what's changing in Yahoo Groups
Note: Yesterday (2019-10-17) it said that it would no longer be possible to upload files after October 21st, and today it says October 28th. So it looks like the situation is evolving. But it still says that the following features will go away on December 14th:
2) Info about PG Offline, which seems to be emerging as the go-to method for saving the contents of Yahoo groups.
In a Reddit thread about the closure of Yahoo! groups, a person identifying themselves as the author of PG Offline has posted:
So it sounds like 1) we absolutely have the software creator's blessing to use the software free for this purpose (yay!), and 2) if I understand correctly, the software will still be usable when the 14-day trial period has elapsed, it's just that you can't download anything else from Yahoo after that point.
PG Offline also has its own support message board, and right now the place is hopping with people talking about the Yahoo shutdown and sharing advice about archiving the contents of groups. The creator of the software is actively posting and answering people's questions.
PG Offline user forum
3) Fannish discussion and info.
rahirah: Yahoo Groups is going away
morgandawn: Yahoo Groups Shutting Down
tozka: Signal Boost: Yahoo Groups Shutting Down
A google doc about how to use PG Offline, written from a fannish perspective, which also discusses fannish archiving, AO3 and the Open Doors project:
PGOffline 4 Walkthrough (WIP)
(I don't know who is creating and editing this document;yourlibrarian sent me a direct link to it, and in fact the document has been updated in the hours since I first saw it.)
The Open Doors Project. (The Open Doors project of the Organization for Transformative Works is dedicated to preserving fanworks for the future, particularly fanworks which are at risk of disappearance.)
The particular reason that I've done such a deep dive into all this is that I'm the current list admin for a Yahoo-hosted fannish mailing list/group, RatBoat. It created on September 5, 2000, and was pretty active from 2001-2003. Its message archive contains a lot of fic, and a lot of fannish discussion. The fic is mostly (but not entirely) also hosted at a fandom-specific archive called The Agency; the archived fannish discussion, of course, exists nowhere else except possibly in the email accounts of various list members.
Yesterday I followed Yahoo's instructions to "request a download of my data," and I got an email from them saying "We will notify you when your download is ready." It is totally unclear to me what exactly they mean by "my data," or what format it will be in. I figured it was worth a try, though! Anyway, it's now been more than 24 hours and I haven't heard back from them yet, which leads me to believe that they will be sending me my data on cuneiform tablets, painstakingly hand-crafted by monks. (Update 2019-10-27: I finally did receive the data. I describe the results in this entry.)
Tonight I downloaded PG Offline and used it to download all of the messages, files, and photos associated with the RatBoat group. It was super easy! I followed the video tutorial (really the first four minutes were enough) with a few glances at the fannish walkthrough guide for extra reference. So that, at least, is done.
Update: I have run into a limitation with the PG Offline free trial.
PGO lets you export your messages in five formats: SQLite dump, MySQL dump, plain text (pipe delimited HTML), plain text (pipe delimited text), and multiple HTML files with 50 messages per file.
I've done the first four (even though I don't really know what the first two are; the second two produce an output which is human-readable but not very user-friendly). However, the trial version of the software will only allow the exporting of 1000 messages total in HTML files.
Update 2 (2019-10-18): I've become aware of another limitation of the PG Offline free trial version, and I've also learned something else that I can do to read the data PG Offline downloaded.
First, the limitation: it seems that when the trial version expires, the export functions will stop working. Source: a post by the author of PG Offline, on the user support forum.
(Of course, there's always the option of paying the $25 for the software.)
Another way to read the data: following some clues in other messages on the PG Offline user support forum, I discovered that PG Offline had saved the whole message log from my group as a .db3 file (it was a bit hard to find on my Windows 10 computer, but I found it). The .db3 is not, it turns out, a proprietary format. Other software can read it. Following the PG Offline author's advice that I found in another thread on the user support forum, I downloaded a free and open-source program called SQLiteStudio from: https://sqlitestudio.pl/index.rvt?act=download
It was a little confusing, but I did manage to open up the file ygroups.db3 in SQLiteStudio, and was rewarded with a giant table where each line had cells with the message number, date, subject, and content of a message. So in other words, everything's there and it's accessible.
Okay, so now the archive is on the hard drive of my computer. Now what?
There are three separate questions, really: what about the fics, what about the files, and what about the discussions?
Most of my thoughts are about the fics.
According to the google doc linked above: "Open Doors - the fanworks-rescuing committee of the Organization for Transformative Works - would be happy to help import fanworks from Yahoo Groups to Archive of Our Own. But they can only do so with the permission of a group administrator."
In the case of RatBoat, I am the group administrator. So it's my call.
But it's not clear to me what I should do.
The fanworks are nearly all archived at The Agency. The RatBoat mailing list always carried the notice in its intro file: "All fic posted here will be archived at The Agency unless the author requests otherwise."
In practice, the last time anything was uploaded to The Agency was on July 31st, 2005. However, very little fic was posted to the list after that date (and most of it was mine, LOL!). In fact, in 2010 when I finally finished a story that had been posted there as a WIP in 2003, I emailed the site admin to ask them to post the complete version, and I never heard back.
Some of the authors who posted fic to the RatBoat mailing list have (like me) uploaded their own fic to AO3 in the intervening years. Many of them haven't, but of those who haven't, most stories are archived at The Agency. Only a few stories (but a non-zero number) exist in the RatBoat message archives but not at either AO3 or The Agency.
It would be a fair amount of work to go through the mailing list message history to find all the fic, and then figure out what's not archived elsewhere. But on the other hand, I could do it, it would just take some time. There are exactly 2540 old messages (I know because I just downloaded them all). It's not an insurmountable task.
There's no active Once A Thief fandom anymore, but there are at least a handful of people still reading messages to the mailing list; I know because I posted new fic recently, and got a couple of responses. There's also still a community on LiveJournal (of which I am also the moderator). So I guess pretty much the first thing I should do is post to the mailing list (while I still can!) and to the LJ comm, and let people know what's happening.
I'm thinking maybe I should also contact the Open Doors people, advise them of my particular situation, and ask for their advice.
The RatBoat Yahoo group had some files, too, and honestly I'm not even sure what's in there. I think a lot of it is just screencaps, which are less important nowadays (since the series was finally released on DVD). There are some photo manips, which I'm really not sure I would have the right to repost elsewhere. And then there are some cool, quirky little things, like floor plans somebody drew up of the characters' apartments (so useful for fic writing!).
As for the old discussion threads—I'd hate to see them vanish from the internet, but I really don't know what to do with them!
I'm going to do a links roundup here (mostly for my own benefit so that I can close some tabs in my browser!—but if anybody else finds this useful, that's great), and then I'm going to report on what I've personally done, what I'm planning to do, and what I'm still trying to figure out.
Links roundup:
1) Yahoo's official announcement.
Understand what's changing in Yahoo Groups
Note: Yesterday (2019-10-17) it said that it would no longer be possible to upload files after October 21st, and today it says October 28th. So it looks like the situation is evolving. But it still says that the following features will go away on December 14th:
Files
Polls
Links
Photos
Folders
Calendar
Database
Attachments
Conversations
Email Updates
Message Digest
Message History
2) Info about PG Offline, which seems to be emerging as the go-to method for saving the contents of Yahoo groups.
In a Reddit thread about the closure of Yahoo! groups, a person identifying themselves as the author of PG Offline has posted:
evildrome
If you want to backup the posts, photos & files from this group you can use my program called PG Offline.
It has a 14 day free trial which will give ample time to download your group.
http://www.personalgroupware.com/downloads.htm
Handy "how to" video.
https://www.youtube.com/watch?v=e4PIumS33e0
The software has become effectively free.
It has a 14 day trial after which you can no longer download from YG! but you can still read and search the downloaded archive.
So, if you can DL your group in 14 days, you’ve won a watch haven’t you?
The lack of a download facility after the group has been deleted by Yahoo isn’t exactly going to be an issue is it?
Cheers,
Wilson Logan (PGO Creator).
So it sounds like 1) we absolutely have the software creator's blessing to use the software free for this purpose (yay!), and 2) if I understand correctly, the software will still be usable when the 14-day trial period has elapsed, it's just that you can't download anything else from Yahoo after that point.
PG Offline also has its own support message board, and right now the place is hopping with people talking about the Yahoo shutdown and sharing advice about archiving the contents of groups. The creator of the software is actively posting and answering people's questions.
PG Offline user forum
3) Fannish discussion and info.
rahirah: Yahoo Groups is going awaymorgandawn: Yahoo Groups Shutting Downtozka: Signal Boost: Yahoo Groups Shutting DownA google doc about how to use PG Offline, written from a fannish perspective, which also discusses fannish archiving, AO3 and the Open Doors project:
PGOffline 4 Walkthrough (WIP)
(I don't know who is creating and editing this document;
yourlibrarian sent me a direct link to it, and in fact the document has been updated in the hours since I first saw it.)The Open Doors Project. (The Open Doors project of the Organization for Transformative Works is dedicated to preserving fanworks for the future, particularly fanworks which are at risk of disappearance.)
What I've Done So Far:
The particular reason that I've done such a deep dive into all this is that I'm the current list admin for a Yahoo-hosted fannish mailing list/group, RatBoat. It created on September 5, 2000, and was pretty active from 2001-2003. Its message archive contains a lot of fic, and a lot of fannish discussion. The fic is mostly (but not entirely) also hosted at a fandom-specific archive called The Agency; the archived fannish discussion, of course, exists nowhere else except possibly in the email accounts of various list members.
Yesterday I followed Yahoo's instructions to "request a download of my data," and I got an email from them saying "We will notify you when your download is ready." It is totally unclear to me what exactly they mean by "my data," or what format it will be in. I figured it was worth a try, though! Anyway, it's now been more than 24 hours and I haven't heard back from them yet, which leads me to believe that they will be sending me my data on cuneiform tablets, painstakingly hand-crafted by monks. (Update 2019-10-27: I finally did receive the data. I describe the results in this entry.)
Tonight I downloaded PG Offline and used it to download all of the messages, files, and photos associated with the RatBoat group. It was super easy! I followed the video tutorial (really the first four minutes were enough) with a few glances at the fannish walkthrough guide for extra reference. So that, at least, is done.
Update: I have run into a limitation with the PG Offline free trial.
PGO lets you export your messages in five formats: SQLite dump, MySQL dump, plain text (pipe delimited HTML), plain text (pipe delimited text), and multiple HTML files with 50 messages per file.
I've done the first four (even though I don't really know what the first two are; the second two produce an output which is human-readable but not very user-friendly). However, the trial version of the software will only allow the exporting of 1000 messages total in HTML files.
Update 2 (2019-10-18): I've become aware of another limitation of the PG Offline free trial version, and I've also learned something else that I can do to read the data PG Offline downloaded.
First, the limitation: it seems that when the trial version expires, the export functions will stop working. Source: a post by the author of PG Offline, on the user support forum.
(Of course, there's always the option of paying the $25 for the software.)
Another way to read the data: following some clues in other messages on the PG Offline user support forum, I discovered that PG Offline had saved the whole message log from my group as a .db3 file (it was a bit hard to find on my Windows 10 computer, but I found it). The .db3 is not, it turns out, a proprietary format. Other software can read it. Following the PG Offline author's advice that I found in another thread on the user support forum, I downloaded a free and open-source program called SQLiteStudio from: https://sqlitestudio.pl/index.rvt?act=download
It was a little confusing, but I did manage to open up the file ygroups.db3 in SQLiteStudio, and was rewarded with a giant table where each line had cells with the message number, date, subject, and content of a message. So in other words, everything's there and it's accessible.
What to do next:
Okay, so now the archive is on the hard drive of my computer. Now what?
There are three separate questions, really: what about the fics, what about the files, and what about the discussions?
Most of my thoughts are about the fics.
According to the google doc linked above: "Open Doors - the fanworks-rescuing committee of the Organization for Transformative Works - would be happy to help import fanworks from Yahoo Groups to Archive of Our Own. But they can only do so with the permission of a group administrator."
In the case of RatBoat, I am the group administrator. So it's my call.
But it's not clear to me what I should do.
The fanworks are nearly all archived at The Agency. The RatBoat mailing list always carried the notice in its intro file: "All fic posted here will be archived at The Agency unless the author requests otherwise."
In practice, the last time anything was uploaded to The Agency was on July 31st, 2005. However, very little fic was posted to the list after that date (and most of it was mine, LOL!). In fact, in 2010 when I finally finished a story that had been posted there as a WIP in 2003, I emailed the site admin to ask them to post the complete version, and I never heard back.
Some of the authors who posted fic to the RatBoat mailing list have (like me) uploaded their own fic to AO3 in the intervening years. Many of them haven't, but of those who haven't, most stories are archived at The Agency. Only a few stories (but a non-zero number) exist in the RatBoat message archives but not at either AO3 or The Agency.
It would be a fair amount of work to go through the mailing list message history to find all the fic, and then figure out what's not archived elsewhere. But on the other hand, I could do it, it would just take some time. There are exactly 2540 old messages (I know because I just downloaded them all). It's not an insurmountable task.
There's no active Once A Thief fandom anymore, but there are at least a handful of people still reading messages to the mailing list; I know because I posted new fic recently, and got a couple of responses. There's also still a community on LiveJournal (of which I am also the moderator). So I guess pretty much the first thing I should do is post to the mailing list (while I still can!) and to the LJ comm, and let people know what's happening.
I'm thinking maybe I should also contact the Open Doors people, advise them of my particular situation, and ask for their advice.
The RatBoat Yahoo group had some files, too, and honestly I'm not even sure what's in there. I think a lot of it is just screencaps, which are less important nowadays (since the series was finally released on DVD). There are some photo manips, which I'm really not sure I would have the right to repost elsewhere. And then there are some cool, quirky little things, like floor plans somebody drew up of the characters' apartments (so useful for fic writing!).
As for the old discussion threads—I'd hate to see them vanish from the internet, but I really don't know what to do with them!
(no subject)
Date: 2019-10-18 06:08 pm (UTC)(no subject)
Date: 2019-10-19 06:28 pm (UTC)(no subject)
Date: 2019-10-18 06:36 pm (UTC)(no subject)
Date: 2019-10-19 06:33 pm (UTC)They also asked me some questions about the privacy settings of the group; I'm going to go respond to that email now.