Searching words of wisdom

I’m currently working on a project that includes a lot of text as part of the data.

Traditionally computer systems and databases don’t like free text; they prefer values that can be added, evaluated and easily stored. Of course there is much to be said for that; assigning a value enables us to clarify our thoughts and opinions. But we can’t communicate everything with numbers and capturing free text provides a great deal of rich understanding. It also enables us to store information which may not be considered relevant right now, but could be valuable in the future or to other people.

Happily, searching text has become a great deal more sophisticated than when I first started working with business systems. Databases such as Microsoft’s SQL Server now have the ability to search for meaning as well as words, to rank search results in terms of likely relevance, and to filter out the “joining words” that don’t add anything to the search process.

When we search for things on the internet we use the same functionality. We expect to key in a search term and find relevant information from a variety of sources. We rarely stop to consider the software technology that is behind making it happen, even though it is hugely powerful and is the piece of the jigsaw that makes all the internet information useful.

Our electronic world has made it so quick and simple to send an email to anywhere in the world, so a huge amount of information is being conveyed in free text. Documents can easily be created and stored on the billions of PCs in businesses everywhere - the issue now is to make the information accessible to the people who need it.