Comments on: Fast XML parsing with Ruby http://www.ioncannon.net/programming/643/fast-xml-parsing-with-ruby/ Thoughts on Software Development and Engineering Wed, 10 Mar 2010 02:57:29 +0000 hourly 1 By: carson http://www.ioncannon.net/programming/643/fast-xml-parsing-with-ruby/comment-page-1/#comment-156462 carson Thu, 05 Nov 2009 19:18:05 +0000 http://www.ioncannon.net/?p=643#comment-156462 I would try pulling out any processing you are doing for each node, make sure you aren't printing anything to the console and time that. If you aren't even doing anything that would be a mystery. I would try pulling out any processing you are doing for each node, make sure you aren't printing anything to the console and time that. If you aren't even doing anything that would be a mystery.

]]> By: Denis Ivanov http://www.ioncannon.net/programming/643/fast-xml-parsing-with-ruby/comment-page-1/#comment-156449 Denis Ivanov Thu, 05 Nov 2009 16:04:10 +0000 http://www.ioncannon.net/?p=643#comment-156449 Useful article! Got my stuff working just fine, except that libxml-ruby still seems dismally slow. My XML structure is dropdead simple and my file is 200MBs... All nodes are super short with no more than one integer attribute and half a dozen children or so. Yet IM clocking in barely over one node a second on a dualcore 3.0Ghz latest Core 2 Duo with 4GBs of quite speedy DDR2 ram... How can I gain more insight into where the time is being spent... any ideas, guys? Useful article!

Got my stuff working just fine, except that libxml-ruby still seems dismally slow. My XML structure is dropdead simple and my file is 200MBs… All nodes are super short with no more than one integer attribute and half a dozen children or so.

Yet IM clocking in barely over one node a second on a dualcore 3.0Ghz latest Core 2 Duo with 4GBs of quite speedy DDR2 ram…

How can I gain more insight into where the time is being spent… any ideas, guys?

]]> By: How I Used Hpricot and Mechanize in GeeQE | IONCANNON http://www.ioncannon.net/programming/643/fast-xml-parsing-with-ruby/comment-page-1/#comment-152978 How I Used Hpricot and Mechanize in GeeQE | IONCANNON Mon, 28 Sep 2009 10:40:54 +0000 http://www.ioncannon.net/?p=643#comment-152978 [...] IONCANNON What are you building? Skip to content AboutProjectsGoogle Analytics Dashboard Wordpress WidgetHTTP Live Video Stream Segmenter and DistributorGeeQE – Native Stack Overflow iPhone ApplicationBuilding GeeQEContactSitemap « Fast XML parsing with Ruby [...] [...] IONCANNON What are you building? Skip to content AboutProjectsGoogle Analytics Dashboard Wordpress WidgetHTTP Live Video Stream Segmenter and DistributorGeeQE – Native Stack Overflow iPhone ApplicationBuilding GeeQEContactSitemap « Fast XML parsing with Ruby [...]

]]> By: Chuck Fouts http://www.ioncannon.net/programming/643/fast-xml-parsing-with-ruby/comment-page-1/#comment-150249 Chuck Fouts Wed, 09 Sep 2009 15:53:17 +0000 http://www.ioncannon.net/?p=643#comment-150249 I recently needed to parse some crazy nested XML from multiple files into one merged file. I decided that loading the DOM was the only way to accomplish this. I'd read that Nokogiri was faster but didn't think it would be that much faster. I was already familar with REXML so I went with that library. I used TDD while creating the parser and was confident in the code by the time I finished. I started the run to parse the multiple XML files and it was going to take almost 40 minutes. Kept finding corner cases to fix and soon I decided it was taking way too long to get going. Installed Nokogiri and ported the code to use Nokogiri in a couple of hours. Having the unit tests really saved me here. Started the run again and it was amazingly faster. In the end, the entire run with Nokogiri only took around 3 minutes instead of the 40 minutes with REXML. I recently needed to parse some crazy nested XML from multiple files into one merged file. I decided that loading the DOM was the only way to accomplish this. I'd read that Nokogiri was faster but didn't think it would be that much faster. I was already familar with REXML so I went with that library. I used TDD while creating the parser and was confident in the code by the time I finished. I started the run to parse the multiple XML files and it was going to take almost 40 minutes. Kept finding corner cases to fix and soon I decided it was taking way too long to get going. Installed Nokogiri and ported the code to use Nokogiri in a couple of hours. Having the unit tests really saved me here. Started the run again and it was amazingly faster. In the end, the entire run with Nokogiri only took around 3 minutes instead of the 40 minutes with REXML.

]]>