Store each article in exactly one file as plain text format

[ncarkaci]

Use file parameter and store each article in exactly one file as plain text format. This means that there no entities and frequent other HTML tags in file. The files are grouped by their first letter and stored in directory which named as the files first letter.

Fixed #139

[markdimi]

Is this feature merged with the master?

[ncarkaci]

No, wait still in pull request status

[markdimi]

Too bad. I have used that feature from your version and I think it works well. Only think I have found out is that a few HTML tags were left in the text. I have tested this in Non-English version of Wikipedia (Greek). 2017-11-15 16:22 GMT+02:00 Necmettin Çarkacı <notifications@github.com>:

…

No, wait still in pull request status — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#141 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE77mO4agmsO5ZD5m5AxaCi4smchkaICks5s2vOigaJpZM4PVmkz> .

[ncarkaci]

Thanks! I used base branch code to remove html tags. My contribution includes that store each file into per file and group them into directory as a first letter. I test the code in non-english version of wiki (turkish). It's pass the test.

[markdimi]

The one file per article and the alphabetical folders, are good features. Hope they get merged. I only saw one html tag in a file. It was a <poem> tag. Now I am using it with the English version of Wikipedia. I was counting of cleaning the html tags on my own. If there aren't any, that would be perfect. Thanks you for your work. 2017-11-19 20:33 GMT+02:00 Necmettin Çarkacı <notifications@github.com>:

…

Thanks! I used base branch code to remove html tags. My contribution includes that store each file into per file and group them into directory as a first letter. I test the code in non-english version of wiki (turkish). It's pass the test. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#141 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE77mMKD9qRKzHiMWJMPWwXWekkgcUX2ks5s4HRmgaJpZM4PVmkz> .

[markdimi]

I just open some English documents that I have used your version of Wikiextractor and I can definitely see some html tags. Just and FYI. 2017-11-20 9:29 GMT+02:00 Markos Dimijazz <dimitsas.markos@gmail.com>:

…

The one file per article and the alphabetical folders, are good features. Hope they get merged. I only saw one html tag in a file. It was a <poem> tag. Now I am using it with the English version of Wikipedia. I was counting of cleaning the html tags on my own. If there aren't any, that would be perfect. Thanks you for your work. 2017-11-19 20:33 GMT+02:00 Necmettin Çarkacı ***@***.***>: > Thanks! I used base branch code to remove html tags. My contribution > includes that store each file into per file and group them into directory > as a first letter. I test the code in non-english version of wiki > (turkish). It's pass the test. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#141 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AE77mMKD9qRKzHiMWJMPWwXWekkgcUX2ks5s4HRmgaJpZM4PVmkz> > . >

[PanderMusubi]

@attardi or other maintainers, please, merge this PR to get this bug fixed and with that revive WikiExtractor development. There are a lot of forks with good improvements because of the value of WikiExtractor. If they are never merged these additions unfortunately don't find their way to the normal end-user.

[ncarkaci]

The conflicts were resolved

[HjalmarrSv]

I did something similar for cirrus-extract.py that uses the already template expanded cirrus dumps. With their ups (complete) and downs (no cfhoices regarding content).

[bluelephant825]

@ncarkaci would you mind updating your code (33 commits behind attardi:master). It no longer works, at least for Python 3+. To start with, lines 77 and 78 should be commented out for Python 3+ as Python 3 has no sys.setdefaultencoding() function. It cannot be reinstated by reload(sys) like it can on Python 2 (which you really shouldn't do in any case).
Since the default on Python 3 is UTF-8 already, there is no point in leaving those statements in.

Furthermore, the results produced are empty text files. Your write function is expecting a string and receives bytes instead!

This gets fixed by removing or commenting out line 603 which reads:
line = line.encode('utf-8')