<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>wikipedia | Nerdpress.org</title>
	<atom:link href="https://nerdpress.org/tag/wikipedia/feed/" rel="self" type="application/rss+xml" />
	<link>https://nerdpress.org</link>
	<description>...dev, tech problems and solutions.</description>
	<lastBuildDate>Wed, 28 Sep 2011 08:05:35 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=6.9.4</generator>
	<item>
		<title>2 ways to your offline wikipedia</title>
		<link>https://nerdpress.org/2011/09/28/2-ways-to-your-offline-wikipedia/</link>
					<comments>https://nerdpress.org/2011/09/28/2-ways-to-your-offline-wikipedia/#comments</comments>
		
		<dc:creator><![CDATA[Ivo Bathke]]></dc:creator>
		<pubDate>Wed, 28 Sep 2011 08:05:35 +0000</pubDate>
				<category><![CDATA[Uncategorized]]></category>
		<category><![CDATA[wikipedia]]></category>
		<guid isPermaLink="false">https://nerdpress.org/?p=1698</guid>

					<description><![CDATA[<p>&#8230; for whatever reason you might need that ;) For both ways you will need the dumps, you get them here: http://dumps.wikimedia.org/dewiki/latest/ The main dump is the pages-articles.xml.bz2 if you want the categories as well, you need the category.sql.gz and categorylinks.sql.gz too. The pages file is quite huge and will take you probably about 4 &#8230; </p>
<p class="link-more"><a href="https://nerdpress.org/2011/09/28/2-ways-to-your-offline-wikipedia/" class="more-link">Continue reading<span class="screen-reader-text"> "2 ways to your offline wikipedia"</span></a></p>
The post <a href="https://nerdpress.org/2011/09/28/2-ways-to-your-offline-wikipedia/">2 ways to your offline wikipedia</a> first appeared on <a href="https://nerdpress.org">Nerdpress.org</a>.]]></description>
										<content:encoded><![CDATA[<p>&#8230; for whatever reason you might need that ;)</p>
<p>For both ways you will need the dumps, you get them here:<br />
<a href="http://dumps.wikimedia.org/dewiki/latest/">http://dumps.wikimedia.org/dewiki/latest/</a><br />
The main dump is the pages-articles.xml.bz2 if you want the categories as well, you need the<br />
category.sql.gz and categorylinks.sql.gz too.<br />
The pages file is quite huge and will take you probably about 4 hours to download, depending on your connection.</p>
<p>You will also need a fresh and running installation of the <a href="http://www.mediawiki.org/wiki/Main_Page">mediawiki</a> software.<br />
Install it and here we go:<br />
<span id="more-1698"></span></p>
<p>1. way: mirror wikipedia<br />
Hereby you mirror the official Wikipedia Site on your machine.<br />
This is the usual way, but its caveat is: it takes a lot of time to install!<br />
You will need some extra tools that help you to import the huge xml dumps but generally its the normal LAMP stack, which is good.<br />
The main instructions for this are from <a href="http://www.mediawiki.org/wiki/Manual:Importing_XML_dumps">here</a>.</p>
<p>The recommended way is to convert the xml to sql to import it.<br />
Therefor we will need a tool for the conversion, a fast tool written in C: <a href="https://github.com/Tietew/mediawiki-xml2sql">xml2sql</a><br />
On my Ubuntu machine i did so to install:</p>
<pre class="brush: bash; title: ; notranslate">
 git clone https://github.com/Tietew/mediawiki-xml2sql.git
 cd mediawiki-xml2sql/
 ./configure
 make
 make install
</pre>
<p>And now convert!:</p>
<pre class="brush: bash; title: ; notranslate">
bunzip2 -c dewiki-20101013-pages-articles.xml.bz2 | grep -v '    &lt;redirect /&gt;' | xml2sql
</pre>
<p>(The grep command is necessary because the converter cant handle that element and will fail otherwise.)<br />
This will take some time.<br />
In the meantime install the mediawiki.</p>
<p>We also will need some mysql tuning to handle this huge mass import:<br />
So switch on innodb, if not already happened and raise those values in <em>/etc/mysql/my.conf</em>:</p>
<pre class="brush: bash; title: ; notranslate">
 innodb_log_file_size=512M
 max_allowed_packet=32M
</pre>
<p>You best delete all inno logfiles as well, before you restart mysql:</p>
<pre class="brush: bash; title: ; notranslate">
rm /var/log/mysql/inno*
</pre>
<p>Now start the imports:</p>
<pre class="brush: bash; title: ; notranslate">
 mysqlimport -u root -p --local wikipedia `pwd`/{page,revision,text}.txt
</pre>
<p>And do the same for the category dumps.<br />
So this will take up to <strong>30h</strong>! (on a normal machine)<br />
Wow, but after that your wikipedia should be ready to browse.</p>
<p>2. way: use the offline reader extension<br />
This approach will try to avoid the endless SQL import by reading the contents directly from the dump.<br />
The reader comes as an mediawiki extension and uses a very smart mix of console tools.<br />
It will need xapian, bzip2recover and some C++ and python scripts.</p>
<p>Basically it will split the archived xml dump in smaller-&gt;faster chunks, create an index over the chunks and use the index as entry point.</p>
<p><a href="https://nerdpress.org/wp-content/uploads/2011/09/wikisplits.png"><img fetchpriority="high" decoding="async" class="alignnone size-medium wp-image-1778" title="wikisplits" src="https://nerdpress.org/wp-content/uploads/2011/09/wikisplits-300x216.png" alt="" width="300" height="216" srcset="https://nerdpress.org/wp-content/uploads/2011/09/wikisplits-300x216.png 300w, https://nerdpress.org/wp-content/uploads/2011/09/wikisplits.png 766w" sizes="(max-width: 300px) 100vw, 300px" /></a><br />
Read the whole procedere <a href="http://users.softlab.ece.ntua.gr/~ttsiod/buildWikipediaOffline.html">here</a>.</p>
<p>Well i have to admit I didnt finished that approach but it seemed faster and i liked the smart combination of tools.</p>
<p>In Conclusion i can say: the large amount of data makes handling wikipedia dumps quite a challenge.</p>The post <a href="https://nerdpress.org/2011/09/28/2-ways-to-your-offline-wikipedia/">2 ways to your offline wikipedia</a> first appeared on <a href="https://nerdpress.org">Nerdpress.org</a>.]]></content:encoded>
					
					<wfw:commentRss>https://nerdpress.org/2011/09/28/2-ways-to-your-offline-wikipedia/feed/</wfw:commentRss>
			<slash:comments>1</slash:comments>
		
		
			</item>
	</channel>
</rss>
