In a recent comparison of different text extraction algorithms, Gravity’s open source project: Goose tied for second place and was even written up over at Read Write Web! I find this very exciting because our project is still quite young and actively in development whereas the algorithms in close standing are mostly well established and semi-finalized. Another interesting point is that most of the competition was built by teams of researchers, you know… Doctors in their fields!

The graph below from Tomaž Kovačič‘s study shows only a small amount of the data he collected in his analysis. If you are curious of how he compared these algorithms, I highly recommend you head over to his post. He does a great job exposing the details behind his analysis.

graph

Goose's standing among other algorithms tested

So what is Goose used for at Gravity and why have we open sourced it?

Goose’s wiki provides a very detailed explanation about what Goose is and how it works, and also touches on the original need we had at Gravity behind its creation. Jim Plush wrote the first version from the ground up on his own and only recently gave me commit access to the repository. By the time I got into the project, it had all the bells and whistles required to compete in the analysis completed by Kovačič. My contributions to Goose have been to extend it to allow for more specific extractions of additional meta data outside of the primary content and have no effect on its standings above.

Such a utility can be applied to a wide variety of web content analysis problems, and I’m really glad Plush decided to share it with the rest of the open source community. At Gravity, we have been building a lot of exciting (to me at least) technology and most of it is held dearly by us and needs to remain a company secret as they make up a large part of our company’s overall value. When it comes to analyzing the content out here on the web, Goose can be looked at as our trusty messenger delivering our system plenty of content to analyze without a lot of the noise that comes along with it on the web pages the content is sourced from.

If you are looking to mine some of the golden nuggets of information that is buried under a ton of ads, peripheral links, site menu structures, and other distracting noise, then why not take a look at what Goose has to offer? If you find anything you think Goose may be lacking or have some ideas on anything else that may be improved, let us know on our Github repository: https://github.com/jiminoc/goose

 
the plight of my klout score

the plight of my klout score

I tend to focus less on my social networks when I have my head down coding. This time however, its effect on my klout score is pretty dramatic. LOL

I guess this is just a consequence of the typical life:work balance, but I wonder how other tech professionals maintain such a high score while also paving new roads in their field.

A couple of examples of the type of tech peeps I’m talking about:

  1. Jeff Atwood – 73: klout.com/codinghorror
  2. Matt Cutts – 73: klout.com/mattcutts

Anyway, I’m not complaining here people. I am just confused about how others manage to keep up their tweeting/blogging while “deep in the cut” of some tech project.

Do any of you have any tips? Please speak up here and let me know.

May 272011
 
Upload progress showing 1,317 of 4,947 songs added...

1,317 of 4,947 songs added... OH HELL YEAH!

I just gained access to Google’s Music Beta and for the first time, I think my personal music library may be smaller than the amout of cloud storage available for free!

It is truly amazing just how far we have come from the the early days of cramming mp3′s into a JPEG image to store on a free image hosting site back in the 90′s. Napster not only broadened what was possible for music online, but also inadvertently set us all back a decade of fighting to truly OWN the music we legally purchase from the big music labels.

Yes, I know there are a lot of you that have second thoughts about giving so much to the Google Collective, and I have no beef with you and your own convictions.

I for one welcome our new online music overlords!

 

This was not an easy task! For one thing, my Community Server (CS) site was not functional, so using RSS / MetaWeblog endpoints were not available options for me. Secondly, I no longer have a Windows development machine. Since CS is built on all Microsoft technologies, I needed to fire up a virtual instance of Windows in order to extract any of the data. If my previous hosting service was able to keep my database online for longer than minutes at a time, I could have run things remotely, but… not the case.

The actual SQL code for extracting all of my blog posts looks surprisingly simple:

SELECT [Subject], PostDate, FormattedBody,
          dbo.old_url(PostDate, [Subject]) AS old_url,
          dbo.make_slug([Subject]) AS slug
FROM dbo.cs_Posts
WHERE UserID = 2102
  AND SectionID = 4
  AND PostLevel = 1
  AND IsApproved = 1
  AND IsLocked = 0

But if you look closely at it, you’ll see that there are to scalar functions in there: ‘dbo.old_url‘ & ‘dbo.make_slug‘. I was surprised to find not find any slugs in the CS DB tables. I assume that all of that logic is being handled from the compiled ASP.NET application itself because there was nothing in the tables, stored procedures, or even functions that did anything related to calculating/parsing URL slugs from post titles. To make matters worse, since my site was not in a running state (due to hosting shenanigans), I had basically just my memory along with the 404 logs on the new WordPress site to help me reverse engineer the rules for converting titles to slugs. This is best represented in my ‘dbo.make_slug’ snippet below:

CREATE FUNCTION [dbo].[make_slug]
(
@post_title nvarchar(256)
)
RETURNS nvarchar(500)
AS
BEGIN
-- Declare the return variable here
DECLARE @slug nvarchar(500)
DECLARE @clean_title nvarchar(500)


SET @clean_title = LOWER(dbo.deDupeSpaces(dbo.removePunctuation(@post_title)))
SET @slug = REPLACE(@clean_title, ' ', '-')

RETURN @slug

END

And that is used by ‘dbo.old_url‘ here:

There are still two more functions remaining (if you have been paying attention) that are used by ‘dbo.make_slug‘ and that is where the real fun comes in. First of these is the simpler ‘dbo.deDupeSpaces‘ which cuts all repeating space characters down to a single space:

CREATE FUNCTION [dbo].[deDupeSpaces]
(
@input nvarchar(500)
)
RETURNS nvarchar(500)
AS
BEGIN
    /**
* Based on Nigel Rivett's SQL script found:
* http://www.nigelrivett.net/SQLTsql/RemoveNonNumericCharacters.html
*/
DECLARE @i int

set @i = patindex('%[ ][ ]%', @input)
while @i > 0
begin
set @input = replace(@input, ' ', ' ')
set @i = patindex('%[ ][ ]%', @input)
end

RETURN @input

END

And the more impressive and pretty much identical to the script I found originally written by Nigel Rivett:

CREATE FUNCTION [dbo].[removePunctuation]
(
@input nvarchar(500)
)
RETURNS nvarchar(500)
AS
BEGIN
/**
* Based on Nigel Rivett's SQL script found:
* http://www.nigelrivett.net/SQLTsql/RemoveNonNumericCharacters.html
*/
DECLARE @i int

set @i = patindex('%[^a-zA-Z0-9 ]%', @input)
while @i > 0
begin
set @input = replace(@input, substring(@input, @i, 1), '')
set @i = patindex('%[^a-zA-Z0-9 ]%', @input)
end

-- Return the result of the function
RETURN @input

END

So all of this so far is just to get my posts out of the CS DB in a format close enough to what I’ll need to stuff into my WordPress DB. In order to continue, I just ran the simple query (snippet at the top) and exported the results to an XML file. Now I could finally shutdown the virtual instance of Windows 7 that was eating up my MacBook’s resources and burning my lap from the CPU pegging. ;-)

The rest is pretty straight forward. I was unable to find any WordPress Plugins so to assist me in this completely custom hackery, so I thought a brute force insert directly into my WordPress mySQL DB was a great idea. I first imported the XML file into a new table that I called cs_posts. This table’s structure is identical to the original query used to export it. Once this was done, I built a basic INSERT INTO …  SELECT query to import these CS posts directly into my WordPress posts table:

INSERT INTO wp_xxxxx_posts
(post_author,
post_date,
post_date_gmt,
post_content,
post_title,
post_status,
post_name,
post_modified,
post_modified_gmt,
guid)
SELECT 2 AS post_author,
cs_posts.PostDate AS post_date,
cs_posts.PostDate AS post_date_gmt,
cs_posts.FormattedBody AS post_content,
cs_posts.Subject AS post_title,
'draft' AS post_status,
cs_posts.slug AS post_name,
cs_posts.PostDate AS post_modified,
cs_posts.PostDate AS post_modified_gmt,
cs_posts.old_url AS guid
FROM cs_posts

From this point, all that was required was for me to correct any permalinks that did not match up to the slug I had calculated. But I also wanted to get 301 redirects in place for all incoming requests looking for /archive/YYYY/MM/DD/some-post-title-slug.aspx to find their way to the new URL /YYYY/MM/some-post-title-slug. This was much easier than I anticipated due to the luxury of John Godley‘s Redirection plugin. This gem of a plugin makes my introduction to the WordPress ecosystem a dream come true. In fact, after I set it up on both this site and my root: robnrob.com site, I was able to populate the redirection item table his plugin uses to skip the need to enter in each post’s specific redirection. The plugin also has an option for regex-ish pattern matching, but a lot of the permalinks I ended up with on WordPress would not directly transpose from the basic:

url pattern: /archive/(d+)/(d+)/(d+)/([a-zA-Z0-9_-]+).aspx
redirect to: http://robbie.robnrob.com/$1/$2/$4

In the end, I lost out on previous comments, categories, and tags, but what I gained was a much more reliable hosting environment and a much more enjoyable platform to hack on. Also, to be honest, I had only a handful of comments anyway. ;-)

 
 

This is the first time that the creator of my programming language of choice sends me an email.

Scala 2.9 was just released this week and the development team at Gravity are working to migrate our code base onto it. Just after our first attempt to run our unit tests, we hit a bug that we could not code around and hit the forums for answers. I found an already reported bug that matched our case as well and jumped on the ticket to receive updates. Later that day I saw a comment on it from Martin Odersky (the original author of Scala) himself. That was exciting enough for me, but the email…. WOW.

…okay, I can go back to my day now.

© 2011 Rob 'n' Rob Suffusion theme by Sayontan Sinha